From Watch Histories to World Models: A Decade of Forecasting and Simulation

One question, asked many ways

For more than a decade I have worked on a single question: given everything that just happened, what happens next?

For the first half of that decade I predicted the answer: which video a viewer will want, where a person's joints will be in the next frame, how a price will move. For the second half I have been simulating it: generating the next second of a video, an entire interactive world, the flight of a drone through air that does not exist yet.

These look like separate fields. They share two hard constraints. The sequences remember their past for a long time, and the models have to be fast enough to matter. This post traces that path, from YouTube's sequential recommenders, through Kubric and MoveNet, to Veo 3 and Genie 3.

Roots: time series with long memories

My first serious models were for markets. At ABC Arbitrage I studied lead-lag and cointegration in long-range-dependent series. My master's thesis at Polytechnique and Paris VI used Hawkes self-exciting processes for optimal trade execution. At Imperial I modeled gestures with stochastic context-free grammars, on the idea that time series have a syntax.

At UC Berkeley (RISELab and BAIR), advised by Joseph Gonzalez and Alexandre Bayen, my PhD was on alternate representations for scalable analysis and control of heterogeneous time series. Two threads from that work shaped everything after it:

  • Long-range dependence. Many real series forget slowly, with correlations that decay like a power law rather than exponentially. Our AISTATS 2017 paper used Fourier transforms on irregularly sampled data to estimate linear causality at scale without heavy communication between machines.

  • Simulation and control. Traffic flow obeys partial differential equations. I found the equivalent of a Froude number for the ARZ traffic equations (Physics Letters A). With multi-task deep RL, we then reached expert-level control of highway ramp metering without prior knowledge of the dynamics (IEEE Transactions on Intelligent Transportation Systems).

In Fall 2017 I created and taught a new graduate class at Berkeley, Deep Time Series Learning.

YouTube: recommenders that remember

A watch history is a time series, and like language it has long-range dependence: what you watched months ago still shapes what you want tonight. The catch is that a recommender answers an enormous volume of requests under tight latency budgets, so long memory cannot be bought with brute compute. Making sequential recommenders that remember far back and serve fast was my focus at Google Research, first as an intern in 2017 and then full-time from 2018.

  • Low-latency sequence models. As an intern I built the first competitive low-latency CNN for YouTube sequential recommendations, and a Recurrent Mixture-of-Experts that improved on the state of the art in sequential recommendation, language modeling and video classification.

  • Factorized Recurrent Neural Architectures for Longer Range Dependence (AISTATS 2018). We analyzed how gradients propagate through time in GRUs and LSTMs from an information-theoretic angle, then designed a factorized architecture that improved results on YouTube-8M, language modeling and recommendations.

  • Towards Neural Mixture Recommender for Long Range Dependent User Sequences (WWW 2019). A sequential mixture of experts, each specialized in a different range of dependence, combined in a single recommendation model.

  • Quantifying Long Range Dependence in Language and User Behavior to improve RNNs (KDD 2019). We applied spectral estimators of long-range dependence to neural embeddings, then built Evolutive RNNs that predict better at lower computational cost.

  • Top-K Off-Policy Correction for a REINFORCE Recommender System (WSDM 2019). I co-authored this work on training YouTube recommendations with reinforcement learning from logged feedback.

The lesson I took from this period: measure how much memory your data actually has, then spend exactly that much architecture on it.

From predicting users to simulating them

A next-item predictor answers one step at a time. Reinforcement learning needs more: a world it can act in and watch respond. So the forecasting work turned into simulation work.

  • I designed and built the first data-driven, RNN-based user simulator for YouTube, end to end from data gathering to deployment, and the first data-driven sequential user simulator for conversational recommenders.

  • I built a model of the YouTube creator ecosystem, from data mining through statistical modeling to deployment.

  • With colleagues, I generated large, realistic recommendation datasets through randomized fractal expansions (2019), so models could be benchmarked at a scale public data did not offer.

  • I showed that TPUs, built for neural networks, are also fast simulators: Tensor Processing Units for Financial Monte Carlo (SIAM PP 2020) covered option pricing, risk and sensitivity analysis, backed by custom C++ XLA kernels.

The idea I carried forward: a simulator is a forecaster you can run in a loop, and its quality is judged by what you can learn inside it.

Bodies in motion: pose, SimAnim, Kubric and MoveNet

My next sequence was the human body. For fitness and health applications, Google needed pose estimation that runs in real time and holds up on poses people actually make while exercising. Datasets built from shared photos skew toward poses worth photographing; a workout is full of poses nobody photographs. I attacked the problem from two sides.

  • Time. I designed RNNs, graph attention networks and state-space models that filter 2D and 3D pose at more than 60 fps. I also built mixed Euclidean/quaternion state-space models to fuse camera keypoints with IMU data.

  • Synthetic data. I led SimAnim, Google's first procedural, scalable generator of human-centric videos with pixel-perfect annotations. Training on tailored synthetic data cut real-world catastrophic failures on under-represented poses by more than 40%.

That work fed into Kubric (CVPR 2022), an open-source, scalable dataset generator built by a large Google team, on which I was second author. Its pose study improved MoveNet, Google's fast CenterNet-based pose model, by mixing rendered humans into training at an 80/20 real-to-synthetic ratio. Keypoint accuracy rose on COCO, on a proprietary set of diverse poses, and on a hard test set of contorted yoga poses. The model and training procedure did not change at all; only the data did.

Simulation here was not a toy environment. It was a way to put the long tail of reality into the training set on purpose.

Interlude: markets again

From 2022 to 2024 I returned to the problem I started with, as tech lead for sequential deep learning at Squarepoint Capital.

Markets taught me production discipline. A forecast that crashes at the wrong moment is worse than no forecast, and the platform ran for more than a year with zero crashes or performance drops.

Veo 3 and Genie 3: simulation fast enough to step inside

In 2024 I joined Google DeepMind as tech lead for the efficiency of video generation and world models. Video generation is forecasting in pixel space: given a prompt or a first frame, predict the frames that follow. A world model goes one step further. Given the world so far and the user's next action, it must produce the next frame before the user notices the wait.

Diffusion models are excellent at this and expensive at it, because each sample takes many denoising steps through a very large network. My team's job was to remove that cost without losing quality.

  • Veo 1, 2 and 3. Text-to-video and image-to-video generation ran more than an order of magnitude faster at equal compute, with quality and prompt alignment held equal in large-scale human preference studies. Post-training cost was halved. Veo 3 launched at Google I/O in May 2025.

  • Genie 3. Genie 3, announced in August 2025, generates explorable worlds from a text prompt at 720p and 24 fps, consistent over several minutes. It works by autoregressive diffusion, and making that model more than an order of magnitude faster is what unlocked real-time, photorealistic interaction.

  • Evaluation. Fast iteration needs fast judgment, so I also led multimodal-LLM auto-raters for visual and semantic quality, reaching human-level agreement and deployed across several products.

I was a core contributor to both launches, and the work received a Google Tech Impact Award. The lesson echoed YouTube exactly: a model only matters if it fits inside the time budget of the interaction.

What's next: world models that fly

Since October 2025 I have been CTO of a stealth startup, where the forecasting and simulation threads finally meet a physical machine. We build real-time world models for autonomous drones: autoregressive diffusion and JEPA models that render photorealistic scenes and respect aerodynamics. Our lightweight models need 20x fewer model calls and 8x fewer active parameters at equal quality. Multimodal-LLM agents generate procedural environments through Gaussian splats, giving us more than 100x more training data. The goal is a closed loop, where a drone policy can be trained and evaluated inside a world that is fast and faithful enough to stand in for the real sky.

Looking back, four ideas carried across every domain:

  1. Measure memory before you model it. Long-range dependence is a property of the data you can estimate, not a guess about architecture.

  2. Speed is a capability. A 20x speed-up does not just save money; it turns an offline model into an interactive one.

  3. Simulation is forecasting you can act inside. User simulators, synthetic humans and world models are all the same object run in a loop.

  4. Synthetic data works when it targets the gaps. The best simulated data covers exactly what real data under-represents.

From watch histories to flight paths, the question has stayed the same. The answers keep getting faster.

Sources

Written by Frank Belletti, polished by AI, 11/05/2026