Richard Hamming famously used to ask his colleagues at Bell Labs this question: “What is the most important problem in your field, and why aren’t you working on it?” To which they probably replied, “Look, Dick, it’s eight-thirty in the morning. I haven’t even had my coffee yet. Who starts a conversation like that?

This essay isn’t to explain the solution to the Monty Hall problem—you can look that up anywhere—but to ask a related question: why does it seem to drive some people crazy? Why do they get so attached to their wrong answers, and so upset by the correct answer? That’s weird, right?

We are in a golden age of grift. Where adventurers once flocked to California or the Yukon because “there was gold in them thar hills,” the fastest way to get rich today is by fleecing suckers. We’ve got crypto rug pulls, meme stocks, nutritional supplements, MLMs—anything to make a quick buck.

I bring you news of the single most important intellectual discovery of our generation: the hard problem of human consciousness has been solved. For a long time, the ability to select squares containing traffic lights was our best working definition of what it meant to be truly, deeply, authentically human, but this was never quite satisfactory.

In part I of this two-part series we covered lookup tables and simple devices with at most a handful of moving parts. This time we’ll pick up in the 17th centuries, when computing devices started to became far more complex and the groundwork for later theoretical work began to be laid.

What is a computer, really? Where did it come from? When did we realize we could trick rocks into doing our math homework for us? In this two-part series, I’ll cover the origin and early history of computing and computer science, starting in prehistoric Africa and ending in Victorian-era England.

You might think it’s unlikely for any interesting mathematics to arise from incense appreciation, but that’s only because you’re unfamiliar with the peculiar character of Muromachi (室町) era Japanese nobles. There has never been a group of people, in any time or place, who were so driven to display their sophistication and refinement.

Here’s a fact: GPT-4o charges 170 tokens to process each 512x512 tile used in high-res mode. At ~0.75 tokens/word, this suggests a picture is worth about 227 words—only a factor of four off from the traditional saying. (There’s also an 85 tokens charge for a low-res ‘master thumbnail’ of each picture and higher resolution images are broken into many such 512x512 tiles, but let’s just focus on a single high-res tile.

One thing you may have noticed about the trigonometric functions sine and cosine is that they seem to have no agreed upon definition. Or rather, different authors choose different definitions as the starting point, mainly based on convenience. This isn’t problematic or even particularly unusual in mathematics: as long as we can derive any of the other forms from any starting point, it makes little difference which we start from.

Kaprekar’s routine is a simple arithmetic procedure on four digit numbers which rapidly converges to the fixed point 6174, known as the Kaprekar constant. Unlike other famous iterative procedures such as the Collatz function, the ad hoc nature of the Kaprekar routine doesn’t hint at fundamental mathematical discoveries yet to be made.

In 2020, the Zodiac 340 cipher was finally cracked after more than 50 years of trying by amateur code breakers. While the effort to crack it was extremely impressive, the cipher itself was ultimately disappointing. A homophonic substitution cipher with a minor gimmick of writing diagonally, the main factor that prevented it from being solved much earlier was the several errors the Zodiac killer made when encoding it.

To celebrate the 100th anniversary of the birth of encabulation - dated from Dr. Wolfgang Albrecht Klossner’s first successful run in that historic barn on the outskirts of Eisenhüttenstadt - this article* collects in one place a number of resources that provide, if not a comprehensive history, at least a catalogue of the major milestones and concepts.

In the previous article in this series we distinguished between two kinds of unsupervised learning (cluster analysis and dimensionality reduction) and discussed the former in some detail. In this installment we turn our attention to the later. In dimensionality reduction we seek a function $f : \mathbb{R}^n \mapsto \mathbb{R}^m$ where $n$ is the dimension of the original data $\mathbf{X}$ and $m$ is less than or equal to $n$.

Consider the following motivating dataset: It is apparent that these data have some kind of structure; which is to say, they certainly are not drawn from a uniform or other simple distribution. In particular, there is at least one cluster of data in the lower right which is clearly separate from the rest.

A common example of recursion is the function to calculate the $n$-th Fibonacci number: def naive_fib(n): if n < 2: return n else: return naive_fib(n-1) + naive_fib(n-2) This follows the mathematical definition very closely but it’s performance is terrible: roughly $\mathcal{O}(2^n)$. This is commonly patched up with dynamic programming.

In today’s installment of Machine Learning From Scratch we’ll build on the logistic regression from last time to create a classifier which is able to automatically represent non-linear relationships and interactions between features: the neural network. In particular I want to focus on one central algorithm which allows us to apply gradient descent to deep neural networks: the backpropagation algorithm.

In this second installment of the machine learning from scratch we switch the point of view from regression to classification: instead of estimating a number, we will be trying to guess which of 2 possible classes a given input belongs to. A modern example is looking at a photo and deciding if its a cat or a dog.

To kick off this series, will start with something simple yet foundational: linear regression via ordinary least squares. While not particularly exciting, linear regression finds widespread use both as a standalone learning algorithm and as a building block in more advanced learning algorithms.

Introduction Visualizing the results of a binary classifier is already a challenge, but having more than two classes aggravates the matter considerably. Let’s say we have $k$ classes. Then for each observation, there is one correct prediction and $k-1$ possible incorrect prediction. Instead of a $2 \times 2$ confusion matrix, we have a $k^2$ possibilities.

Craps is a suprisingly fair game. I remember calculating the probability of winning craps for the first time in an undergraduate discrete math class: I went back through my calculations several times, certain there was a mistake somewhere. How could it be closer than $\frac{1}{36}$? (Spoiler Warning If you haven’t calculated these odds for yourself then you may want to do so before reading further.

Last week my boss stopped by and dropped a brand spanking new iPad on my desk. “Make our application work on this,” he commanded. “You have two days before we demo it at the trade show.” Madness? No, these are web apps! You see, for the last couple years we’ve been working exclusively on AJAX applications: web pages stuffed with so much JavaScript they look and feel like desktop apps.

Update 2017-10-23: This article and code library have not kept up with the rapidly changing JavaScript landscape and are now hopelessly out of date. First came non-enumerable properties, and with ES2015 came the introduction of classes, proxies, symbols, and anonymous functions, all of which break the below logic. I'm afraid I no longer know how to fully copy the full menagerie of JavaScript objects correctly.