Library

What Machine Learning Is, and Is Not

The Data Is the Model

Features and Representations

Loss: How a Model Knows It Is Wrong

Gradient Descent

Backpropagation

What a Neural Network Actually Computes

Parameters, Weights, Layers, Activations

Activations: ReLU, GELU, SwiGLU

Training Versus Inference

Self-Supervised Learning

Overfitting and Generalisation

Residual Connections and Normalisation

Why Scale Worked

Why Hand-Built Knowledge Lost

Turning Text Into Numbers

Tokens: How Language Becomes Something a Model Can Process

Byte Pair Encoding

What Tokenisation Breaks

Embeddings: Meaning as Direction

Living in 4,096 Dimensions

Predicting the Next Token

The Recurrent Era and Its Bottleneck

One Word, Many Meanings

Vocabularies, Special Tokens and Chat Templates

Attention, Before the Maths

Queries, Keys and Values

Self-Attention, Step by Step

Multi-Head Attention

The Causal Mask

Position: Attention Has No Sense of Order

Rotary Position Embeddings

Models Without Positional Encoding

The Feedforward Block

The Transformer Block, Assembled

The Residual Stream

Encoder, Decoder, or Both

Why Attention Gets Expensive

FlashAttention

Multi-Query and Grouped-Query Attention

Alternatives to Attention

Logits and the Softmax

Temperature

Top-k, Top-p and Min-p

The Context Window

Long Context Is Not Uniform Attention

The KV Cache

Prefill Versus Decode

Batching, Throughput and Latency

Speculative Decoding

Constrained and Structured Generation

System Prompts and Instruction Hierarchy

Pretraining

What Goes Into the Corpus

Chinchilla and Compute-Optimal Training

Supervised Fine-Tuning

RLHF

DPO and Direct Preference Learning

Training Against Written Principles

LoRA and Parameter-Efficient Fine-Tuning

Fine-Tune, Prompt, or Retrieve

Catastrophic Forgetting

The Emergent Abilities Argument

Chain of Thought

Sampling Many Answers

Test-Time Compute

Reasoning Models

Mixture of Experts

Dense Versus Sparse Models

Distillation

Pruning

Looking Inside the Model

Architectural Limits

Why GPUs

VRAM: The Number That Decides Everything

Bandwidth, Not FLOPs

Unified Memory

Quantisation

GPTQ, AWQ and K-Quants

Model Formats: GGUF, Safetensors, MLX

Running an Open-Weight Model Locally

Why Long Context Eats Your RAM

Splitting a Model Across Machines

A Cluster of Personal Computers

Local Versus Cloud, Honestly

Embedding Models

Vector Search

Retrieval-Augmented Generation

Chunking and Reranking

Tool Use and Function Calling

Agents

Standardised Tool Connections

Multimodal Models

Diffusion and Image Generation

Speech Recognition and Synthesis

Hallucination

Evaluating Models

Bias, Harm and Safety Training

Prompt Injection

Open Weights, Closed Frontiers

Neural Tangent Kernel Analysis

Diffusion Model ODE/SDE Frameworks

Sparse Mixture of Experts Routing

Transformer Neural Machine Translation

JUNE 2017

Vision-Language Navigation

Model-Based Reinforcement Learning with MuZero

BIG-bench Collaborative Benchmark

Mechanistic Probes for Circuit Discovery

Blockwise Quantization-Aware Training

Dataset Distillation

AI and Neuroscience Cross-Pollination

Layer-wise Adaptive Rate Scaling (LARS)

vLLM: PagedAttention for LLM Serving

Unified MultiModal Embedding with ImageBind

Hindsight Experience Replay

Holistic Evaluation of Language Models (HELM)

Sparse Autoencoders for Feature Discovery

Sliding Window Attention for Long Sequences

Web-Scale Text Corpora Curation

Graph Attention Networks

Fourier Features for Positional Encoding

Orca: System for Distributed Inference

Audio-Visual Contrastive Learning

Self-Play with Population-Based Training

Anthropic's Redwood Research Interpretability

TruthfulQA: Measuring Truthfulness

Causal Mediation Analysis in Neural Networks

Activation Sparsity via ReLU Pruning

Crowdsourcing High-Quality Human Feedback

BERT and the Masked Language Modeling Revolution

Stable Diffusion Architecture

Hyperparameter Transfer Learning

TensorRT and Kernel Fusion

NeRF: Neural Radiance Fields

Mixture of Experts Routing

Sparse Transformer with Local Attention

Low-Rank Adaptation (LoRA)

Gradient Checkpointing for Memory-Efficient Training

Model Soups: Weight Averaging for Improved Robustness

Manifold Mixup for Better Representations

Speculative Decoding for Faster LLM Inference

Multimodal Chain-of-Thought Reasoning

Preference Optimization via Nash Learning

Red Teaming with Language Models

Mechanistic Interpretability of Induction Heads

Extreme Low-Bit Post-Training Quantization

Data Selection via Perplexity for Language Model Training

The Deep Learning Hardware Revolution

Vector-Quantized Variational Autoencoders (VQ-VAE)

Gradient-Based Meta-Learning (MAML)

Sharpness-Aware Minimization (SAM)

Paged Attention for Efficient Memory Management

Unified Embedding Space for All Modalities

Multi-Agent Deep Reinforcement Learning with Centralized Critics

Scalable Oversight via Recursive Reward Modeling

Benchmarking Hallucination in Language Models

Activation Compression via Quantized Training

Deduplication of Massive Training Datasets

The Transformer Architecture Paper

Recurrent Independent Mechanisms

Self-Supervised Learning of Visual Features via Contrastive Loss

AdaFactor: Adaptive Learning Rates with Sublinear Memory

Continuous Batching in LLM Serving

Compositional Visual Reasoning with Neural Module Networks

Reward Model Overoptimization

TruthfulQA: Benchmarking Tendency to Generate Falsehoods

Causal Abstraction for Model Verification

Sliding Window Attention for Infinite Context

Model Contamination Detection in Benchmarks

The Bitter Lesson of Compute and Scale

Vision Transformer (ViT) Scaling

Gradient Surgery for Multi-Task Learning

Sparse Mixture of Experts Inference Routing

Neural Tangent Kernel Theory for Wide Networks

Transformer Memory Compression via Sliding Windows

Neural Radiance Fields for 3D Reconstruction

Soft Actor-Critic with Maximum Entropy

Red Teaming for Language Model Safety

Benchmarking Compositional Generalization

Mechanistic Interpretability of Grokking

Speculative Decoding for Faster Inference

Text-to-3D Generation with Score Distillation

Human Feedback from Comparisons

Mixture Density Networks for Uncertainty

Coresets for Efficient Dataset Summarization

Knowledge Distillation from Multiple Teachers

Weight Agnostic Neural Architecture Search

Deep Double Descent: Beyond U-Shaped Risk

Volumetric Rendering with Neural Graphics Primitives

Multi-Task Learning with Task Balancing

Efficient ViT with Shifted Windows

No-Regret Online Learning to Nash Equilibrium

Model Parallelism with Pipeline Bubbles

Video Diffusion with Temporal Consistency

Recursive Reward Modeling for Scalable Oversight

Out-of-Distribution Detection via Likelihood Ratios

Causal Abstraction via Interchange Interventions

Quantization with Learned Rounding

Data Valuation with Shapley Values

Transformer Positional Encoding Alternatives

Memory-Efficient Backpropagation with Reversible Layers

Audio-Driven Talking Head Generation

Constitutional AI for Harmless Assistance

Adversarial Evaluation of Multimodal Reasoning

Sparse Autoencoders for Dictionary Learning

Dynamic Quantization During Serving

Synthetic Data Generation with Differential Privacy

History: The 2012 AlexNet Revolution

Graph Transformers with Structural Encodings

Optimization Theory for Adaptive Gradient Methods

Multi-Task Reinforcement Learning with Shared Representations

Differentiable Architecture Search

Mixture of Experts: Sparse Routing at Scale

Retentive Networks for Long Sequences

State Space Models for Long Sequences

Differential Privacy by Noising Gradients

Progressive Growing of GANs

Meta-Learning with External Memory

Loss Landscapes: Flat Minima and Sharp Ones

The Information Bottleneck

The Lottery Ticket Hypothesis

The Neural Tangent Kernel

Feature Learning: The Lazy-to-Rich Transition

Emergence: Abilities That Appear Only With Scale

KV Cache Compression for Long Context

Predictive Auto-Scaling for Model Serving

Elastic Training: Surviving a Lost Machine

near‑data‑processing-for-embedding-lookup

Contrastive Learning Across Vision, Audio and Text

Vision-Language-Action Models for Robotics

World Models: Planning Inside a Learned Simulator

Imitation Learning from Observation

Model-Based RL and the Limits of a Learned Simulator

Inverse Reinforcement Learning

Trojans and Backdoors: Models with Hidden Triggers

evaluation-with-adversarial-n‑way-forcing

Calibration: When Confidence Matches Correctness

Probing for Linguistic Knowledge

Ablation: Reading Function by Removing Parts

LoRA: Low-Rank Fine-Tuning

Synthetic Data for Data-Scarce Domains

Active Learning: Choosing What to Label

Cleaning Web-Scale Training Data

Connectionism versus Symbolism

The Rise of Self-Supervised Learning

Vision-Transformer Hybrid Architectures

MoE Routing Strategies

Model Merging via Weight Interpolation

Mechanistic Interpretability of In-Context Learning

Retentive Networks (RetNet)

JEPA and I-JEPA

Preference Optimization with DPO

Temporal Difference Learning Theory