YC Paper Club · Pioneer

YC Paper Club: Five AI Papers, From Faster Inference to Infinite Compute

The first YC Paper Club packs five short paper talks into an hour — speculative speculative decoding, diffusion model predictive control, LeWorldModel, why deep learning isn't mysterious, and what pre-training looks like when compute is free but data isn't.

Creator: Y CombinatorRuntime: 67 minVideo published: May 28, 202649K views when summarized

Watch on YouTube

TL;DR

The five papers in one breath

Speculative Speculative Decoding (Kumar, Dao, May): run drafting and verification in parallel on separate hardware instead of in lockstep — hide the draft latency, predict verification outcomes ~80–90% of the time, win on both latency and throughput.
Diffusion Model Predictive Control (D-MPC): use diffusion models for both multi-step action proposals and multi-step dynamics, which cuts compounding error and lets a dead-simple sampling planner adapt to new rewards and broken-robot dynamics at test time.
LeWorldModel (LeCun's group): a JEPA world model whose only new trick is the SIGReg regularizer — keep the latent embeddings Gaussian to dodge representational collapse, in ~15M params on a single card, ~50x faster than competitors.
Andrew Gordon Wilson, 'Deep learning is not so mysterious': overparameterization, benign overfitting, and double descent all fall out of PAC-Bayes plus soft inductive bias — bigger models find flatter, more compressible solutions.
Pre-training under infinite compute (Kim, Kotha, Liang, Hashimoto): when you're data-bound but compute-free, regularize hard and ensemble — clean power laws with measurable asymptotes give a 5x data-efficiency win you can distill back into a tiny model.

01 · The club

Make Pioneer great again

The first YC Paper Club: over a thousand applications, capped at about a hundred people, run out of the Pioneer building in Woodside. The host's pitch is half community, half geography — roughly half the Bay Area's AI talent is down on the Peninsula (Google DeepMind on the corner, Tesla, xAI, Thinking Machines, a wall of Palo Alto startups) and not making the trek into the city for YC. The room skews credentialed: hands stay up through ten thousand citations and past fifty million dollars raised.

FormatFive papers, one presenter each, ~10 minutes apiece. Boba afterward.

02 · Inference

Speculative speculative decoding

Tanishq Kumar (with Tri Dao and Avner May) makes one claim worth keeping: inference is about to stop being a cost-or-convenience lever and start being a capability. If a method's performance scales with how much it thinks, then tokens-per-second is the ceiling on intelligence you can deliver.

Vanilla speculative decoding exchanges flops for latency — a small draft model guesses tokens auto-regressively, the big target model verifies them in one parallel forward pass, and you keep the tokens it would plausibly have generated (plus a free "bonus" token at the rejection point). The bottleneck is the sequential dependency: drafting round t+1 can't start until verification of round t finishes.

SSD's whole idea: stop colocating drafting and verification, run them on separate hardware at the same time. While the big model verifies, the draft model immediately starts anticipating the most likely verification outcomes and drafting on top of them.00:12:28

The hard part is predicting verification outcomes — a number of accepted tokens plus a bonus token drawn from a vocabulary of tens to hundreds of thousands. The trick: the tokens the draft chose not to sample are exactly the plausible bonus-token candidates, so you decode several guessed sequences in parallel over a shared prefix.

You also get the ability, next time you're at a San Francisco house party, to know in the corner what it takes to sample at 300 tokens per second for Llama 3 70B on four H100s.— Tanishq Kumar

Result~80–90% verification-prediction accuracy is enough for big speedups. Unlike normal speculation, it wins on both latency and throughput.

03 · Robotics

Diffusion model predictive control

Stannis (Google DeepMind, now co-leading a world-modeling-for-robotics project) presents earlier work on D-MPC. Model predictive control needs two things to work in practice: a dynamics model accurate enough to avoid compounding error, and a planner strong enough to pick good action sequences. D-MPC uses diffusion models for both — multi-step action proposals and multi-step dynamics — which lets the planner stay almost trivially simple and still beat prior approaches.

The design space of diffusion-based agents, factored as a joint distribution over states and actions: diffusion policy (needs expert demos), Diffuser (joint state-action, implicit world model), decision diffuser (observation-only, learns from video), and D-MPC (runtime adaptation to new rewards and dynamics).00:24:24

The factorization between action proposal and dynamics model is what pays off at test time. Keep the action proposal fixed, swap rewards, and the agent exhibits novel behaviors. Break the robot — a walker with a busted left ankle — and you only need to adapt the dynamics model on a little play data to recover most of the performance.

Why diffusionStronger multimodal modeling capacity is what buys the simpler planner — the model does the heavy lifting so the search doesn't have to.

04 · World models

LeWorldModel and the collapse problem

Isaac R. Ward presents LeWorldModel out of Yann LeCun's group — and frames it as a literal billion-dollar question, since LeCun raised $1.03B in March largely to train world models. A world model predicts how a system changes over time: current observation, an action, predicted next observation. Old idea — he digs up a 1990 Sutton flyer describing exactly a modern world model — newly hot.

Model-free vs. model-based policies. Model-free maps observations straight to actions with no explicit picture of the future; model-based trains an explicit world model and uses it to predict outcomes — more organized, modular, and able to quantify its own modeling error.00:34:02

The core training pain is co-learning a representation of the world and its dynamics, where the optimization landscape is full of trivial-collapse minima ("every state is the same"). Existing methods dodge collapse with explicit heuristics, foundation-model backbones, or privileged data. Ward's honest read: LeWorldModel is "just offering a new kind of trick" — the SIGReg regularizer.

Sketched

one-dimensional passes over high-dimensional latent data

Isotropic

the distribution looks the same sliced in any direction

Gaussian

each sliced curve should be Gaussian-distributed — cheaply checkable, and a healthy latent space means a non-collapsing world model

Why it's nice~15M parameters, runs on a single card under 24GB of VRAM, ~50x faster than competitors because all the work happens in latent space. Loses to Dino-WM on 3D tasks (that one has a big pretrained backbone), but a standout capability is surprise quantification — perturb the input and model error spikes detectably, so the agent knows when its predictions are bad.

05 · Theory

Deep learning is not so mysterious

Akshay Vegesna (Q Labs) walks through Andrew Gordon Wilson's paper. The setup: scaling models improves generalization, but we have no mechanistic story for why, and people point to overparameterization, benign overfitting, and double descent as evidence we can't understand it at all. Wilson's move is to use classical PAC-Bayes — which bounds test loss by training loss plus a compression term — and argue the bounds only looked vacuous because the compression term was being computed wrong.

Benign overfitting via a regularized polynomial model: on random data (panel c) there are enough parameters to fit pure noise, but on structured data the regularization pushes toward low-order terms — flexibility plus a soft inductive bias.00:48:16

On overparameterization: bigger models drive empirical risk down and find more compressible solutions — the volume of flat minima grows exponentially with parameter count, and flat minima compress better than sharp ones. Both terms of the PAC-Bayes bound drop, giving non-vacuous bounds even at billion-parameter scale.

By the no-free-lunch theorem, the only way we get improvements in learning efficiency is through inductive biases.— Akshay Vegesna, on Andrew Gordon Wilson's paper

The takeaway he leaves the room with: if these mysteries are really soft inductive bias plus PAC-Bayes, then finding the right inductive biases is something you can optimize for — and given the sample-efficiency gap between AI and humans, that's a good bet.

06 · Pre-training

Infinite compute, finite data

Konwoo Kim (with Suhas Kotha, Percy Liang, Tatsu Hashimoto) closes on the question the host says he's obsessed with: intelligence per sample. Pre-training keeps improving — in-context learning, alignment, reasoning — but human text on the internet grows ~3% a year while pre-training compute grows 4–5x a year. So the real regime is: data-constrained, compute-unconstrained. What do you do?

Scaling recipes on a fixed 200M-token budget. The standard recipe (red) — epoch the data and scale the model — eventually overfits and turns back up. Aggressive regularization (purple) follows a clean power law with a measurable asymptote: the best loss you could reach with infinite compute.00:56:37

The recipe is to revisit the classical toolkit. Crank weight decay ~30x past compute-optimal and you get a clean power law with an asymptote. Bring back ensembling and it's even more data-efficient — a compute-matched ensemble of small models beats one large model when you're data-bound. Compose the two (the "joint scaling" recipe) and project onto data scaling laws to read off a ~5x data-efficiency win that stays constant out to trillions of tokens.

Distill it downan 8-member ensemble (~2.4B params) distills into a single 300M dense model while keeping ~83% of the loss improvement — data efficiency without the inference cost.
Self-distill for freedistilling a 300M model into a fresh 300M model still improves loss, beating the regularized asymptote — it behaves like implicitly training a 2-ensemble.
It transferstrends hold on held-out downstream benchmarks and on continued pre-training (matching 73B math tokens using only 4B — a ~17x win).

The frameThe hope for these compute asymptotes isn't just ranking known algorithms — it's surfacing new ideas that only make sense once compute is no longer the binding constraint.