nathan.rs 1 months ago

Is gzip a language model?

A while back I wrote about language modeling without neural networks , where I generated Shakespeare with an unbounded n-gram model: no weights, no training, just counting. Fortuitously, I came across the paper Language Modeling is Compression , which mentioned the compression–prediction equivalence : every prediction model is inherently a compressor, and all compression algorithms are prediction models . This led to the natural question: can do language modeling? 1 No neural network, no learned parameters, nothing. Just the compressor that ships with your operating system. You prime it with a corpus, and it continues a prompt by searching for the byte sequences that compress best. Here’s some real, unedited output after priming it on tiny Shakespeare:

AI

▲ 0

0 views

nathan.rs 2 months ago

Language Modeling Without Neural Networks

Generating Shakespeare has become the “Hello World” of language models. 1 Recently, I’ve been messing with alternative language models and came across unbounded n-gram models. These models are purely statistical and don’t require optimizing weights or training. A year ago, I read the paper Infini-gram , which scaled an unbounded n-gram model to trillions of tokens. While their model had applications helping guide neural LLMs during generation, standalone language generation was not explored. In this post, I’ll explain how unbounded n-gram models work and how I improved their language generation capabilities.

AI

▲ 0

0 views

nathan.rs 7 months ago

Text Diffusion Models are Faster at Writing Code

In this post, I run small experiments showing that diffusion language models generate code (and other structured text) at a faster rate. Increased stucture tends to correlate with reduced entropy, which leads to higher confident token predictions, which directly means more tokens decoded in parallel per step. 1 In speculative decoding (for autoregressive models), we speed up generation by using a smaller model to generate multiple tokens, which are then verified in parallel by a larger model. The core idea is that most tokens are easily predictable; thus, we should be able to use a smaller and faster model for them. The classic example is the following sentence:

Performance Programming AI

▲ 0

1 views

nathan.rs 8 months ago

The Short Life and Death of a Research Idea

For Cal Hacks 2025 , a few friends and I built Curserve , a fast and scalable server-side engine for agentic coding, which ended up placing for one of the sponsor prizes. We didn’t go to Cal Hacks to try and win, but instead to have a good excuse to work on a potential research idea. It turns out that our idea was a much better hackathon project than it was a research direction!

Programming

▲ 0

0 views

nathan.rs 8 months ago

BERT is just a Single Text Diffusion Step

A while back, Google DeepMind unveiled Gemini Diffusion , an experimental language model that generates text using diffusion. Unlike traditional GPT-style models that generate one word at a time, Gemini Diffusion creates whole blocks of text by refining random noise step-by-step. I read the paper Large Language Diffusion Models and was surprised to find that discrete language diffusion is just a generalization of masked language modeling (MLM), something we’ve been doing since 2018 . The first thought I had was, “can we finetune a BERT-like model to do text generation?” I decided to try a quick proof of concept out of curiosity.

AI Machine Learning

▲ 0

0 views

nathan.rs 9 months ago

Research Log

I’m doing my master’s thesis around distributed low-communication training. Essentially, how can we train large models efficiently across distributed nodes and not be utterly destroyed by network latency and bandwidth? Below is some of what I’ve learned and investigated throughout the days. A desirable problem to solve is being able to use different kinds of hardware for training. Even within the same generation, NVIDIA B300 GPUs are 50% faster than B200s. Companies like Meta have many homogeneous clusters that differ in hardware. It would be ideal to be able to train a model across clusters regardless of the exact underlying hardware used.

Performance AI Hardware

▲ 0

0 views

nathan.rs 1 years ago

Running GPT-2 in WebGL: Rediscovering Classic GPGPU Programming

A few weeks back, I implemented GPT-2 using WebGL and shaders ( Github Repo ) which made the front page of Hacker News ( discussion ). This is a short write-up over what I learned about old-school general-purpose GPU programming over the course of this project. In the early 2000s, NVIDIA introduced programmable shaders with the GeForce 3 (2001) and GeForce FX (2003). Instead of being limited to predetermined transformations and effects of earlier GPUs, developers were now given unprecedented control over the rendering pipeline, enabling much more sophisticated visual effects. These programmable shaders laid the foundation for modern GPU computing.

AI Hardware

▲ 0

0 views

nathan.rs 2 years ago

Mathematical Statistics

My notes over Mark Maxwell’s course, Introduction to Mathematical Statistics, and his textbook, Probability & Statistics with Applications, Second Edition. Normally in a probability experiment, we don’t know the true values of a model’s parameters, and therefore, we must estimate them using random observations. Because the observations are random, our estimates are subject to the vagaries of chance. We find ourselves in a paradoxical situation in which the parameters are fixed, but unknown, while the estimates are random, but observable.

Statistics Data Analysis

▲ 0

0 views

nathan.rs 2 years ago

Common Probability Distributions

An overview of common discrete and continuous distributions found in probability and statistics, from Mark Maxwell’s textbook, Probability & Statistics with Applications, Second Edition. A random variable $X$ is said to have a discrete uniform distribution if its probability function is: $$Pr(X=x)=\frac{1}{n}$$ for $x=1,2,\dots,n$. Expected Value: $$E[X ]=\frac{n+1}{2}$$ Variance: $$Var[X ]= \frac{n^2-1}{12}$$ A Bernoulli trial is an experiment that has two outcomes (true-false; girl-boy, success-fail, in-out, etc). An overview of common discrete and continuous distributions found in probability and statistics, from Mark Maxwell’s textbook, Probability & Statistics with Applications, Second Edition. Common Discrete Distributions # Discrete Uniform # A random variable $X$ is said to have a discrete uniform distribution if its probability function is: $$Pr(X=x)=\frac{1}{n}$$ for $x=1,2,\dots,n$. Main Properties # Expected Value: $$E[X ]=\frac{n+1}{2}$$ Variance: $$Var[X ]= \frac{n^2-1}{12}$$ Median: Same as Expected Value

Statistics

▲ 0

0 views

nathan.rs 2 years ago

How to Fix Hugo's iOS Code-Block Text-Size Rendering Issue

Lately, I’ve been coming across many blogs that have weird font-size rendering issues for code blocks on iOS. Basically, in a code snippet, the text-size would sometimes be much larger for some lines than others. Below is a screenshot of the issue from a website where I’ve seen this occur. As you can see, the text-size isn’t uniform across code block lines. I’ve seen this issue across many blogs that compile markdown files to HTML such as sites built using Hugo, Jekyll, or even custom md-to-html shell scripts .

Shell HTML Mobile Frontend Web

▲ 0

0 views

nathan.rs 2 years ago

Intro to Autograd Engines: Karpathy's Micrograd in Go

For a while, I wanted to build a complete autograd engine. What is an autograd engine, you might ask? To find the answer, we first must know what a neural network is. A neural network can just be seen as a black-box function. We pass in an input into this black box and receive an output. Normally, in a function, we define the rules on how to manipulate the input to get an output. For example, if we want a function that doubles the input, i.e $f(x) = 2x$, then all we would write is:

AI Go

▲ 0

0 views

nathan.rs 2 years ago

Where Rust Shines: Algebraic Types and Match Statements

Recently I was going through Thorsten Ball’s “Writing An Interpreter in Go”. In this book, you create a basic interpreted language and write a lexer, parser, evaluator, and REPL for it. A Lexer takes in source code and turns it into an intermediate representation, usually in the form of a string of tokens. This is called Lexical Analysis. A parser usually takes this stream of tokens and turns it into an Abstract Syntax Tree which is then evaluated and run.

Rust Programming

▲ 0

0 views

nathan.rs 2 years ago

Favorite Quotes

Here are a few of my favorite quotes I’ve liked over the years. “I believe that a man should strive for only one thing in life, and that is to have a touch of greatness” — Félix Martí-Ibáñez “In Three Words, I Can Sum Up Everything I’ve Learned About Life. It Goes On” — Robert Frost “But you see,” said Roark quietly, “I have, let’s say, sixty years to live. Most of that time will be spent working. I’ve chosen the work I want to do. If I find no joy in it, then I’m only condemning myself to sixty years of torture. And I can find the joy only if I do my work in the best way possible to me. But the best is a matter of standards—and I set my own standards. I inherit nothing. I stand at the end of no tradition. I may, perhaps, stand at the beginning of one.” ― Ayn Rand, The Fountainhead

Writing Books AI

▲ 0

0 views

nathan.rs 2 years ago

Favorite Books

Below are all the books I’ve read since middle school, roughly in order. Those highlighted in blue were those I particularly enjoyed :) Willpower - Roy F. Baumeister & John Tierney Deng Xiaoping and the Transformation of China - Ezra F. Vogel

Books

▲ 0

0 views

nathan.rs 2 years ago

Gradient Descent & Optimizers

Theses are some of my over Qiang Liu’s course, Machine Learning II. Gradient Descent is a fundamental, first-order iterative optimization algorithm designed for minimizing a function. The primary objective of Gradient Descent is to find the minimum value of a function by iteratively moving towards the minimum of the gradient. Update Rule: The parameters $ \theta $ are updated as follows in each iteration:

Machine Learning AI

▲ 0

0 views

nathan.rs 2 years ago

Language Modeling: Word Embedings & Architectures

These are a few of my notes from Eunsol Choi’s NLP class at UT Austin. Word embeddings are a type of word representation that captures the semantic meaning of words in a continuous vector space. Unlike one-hot encoding, where each word is represented as a binary vector of all zeros except for a single ‘1’, word embeddings capture much richer information, including semantic relationships, word context, and even aspects of syntax.

AI

▲ 0

0 views

nathan.rs 2 years ago

Neural Networks: RNNs, Seq2Seq, & CNNs

These are a few of my notes from Eunsol Choi’s NLP class at UT Austin. Recurrent Neural Networks (RNNs) are a class of artificial neural networks specifically designed to tackle sequence-based problems. Unlike traditional feedforward neural networks, RNNs possess a memory in the form of a hidden state, enabling them to remember and leverage past information when making decisions. This makes them particularly effective for tasks like language modeling, time-series forecasting, and sentiment analysis.

AI Deep Learning Nlp

▲ 0

0 views

nathan.rs 2 years ago

Classifiers: Generative & Discriminative Models

These are a few of my notes from Eunsol Choi’s NLP class at UT Austin. When it comes to classification, models are broadly categorized into Generative Models and Discriminative Models. In generative models, we aim to model the joint distribution of the data $ p(x, y) $. These models often assume a particular functional form for both $ P(x|y) $ and $ P(y) $. To classify a new data point, we maximize:

Nlp Statistics

▲ 0

0 views

nathan.rs 2 years ago

Probability

My notes over Mark Maxwell’s course, Probability I, and his textbook, Probability & Statistics with Applications, Second Edition. The fundamental theorem of counting is also known as the multiplication principle . Given that there are $N(A)$ outcomes, and for each of these outcomes, there are $N(B)$ outcomes, then the total number of outcomes for the two combined is equal to $N(A)\cdot N(B)$.

Data Analysis Statistics

▲ 0

0 views

nathan.rs 2 years ago

Linear Algebra

These are my notes over my review of Linear Algebra, going through Gilbert Strang’s Introduction To Linear Algebra. The core of linear algebra is vector addition and scalar multiplication. Combining these two operations gives us a set of linear combinations. $$ c\mathbf{v} + d\mathbf{w} = c\begin{bmatrix} 1 \\ 2 \end{bmatrix} + d\begin{bmatrix} 3 \\ 4 \end{bmatrix} = \begin{bmatrix} c + 3d \\ 2c + 4d \end{bmatrix}. $$

Programming

▲ 0

0 views