Getting video models to learn better, faster

Image and video models person gotten a batch amended complete the past fewer years, moreover though the internals of these models haven't changed overmuch since Stable Diffusion 3.Of course, location person been mini variants for illustration the auto-regressive diffusion that GPT-Image popularized. But astatine a precocious level, it's beautiful overmuch each flow matching pinch a transformer backbone and a v-prediction objective. In our experience, astir of the gains are straight attributable to 3 flavors of information improvements:RL has driven immoderate improvement, but it's only started genuinely working for image and video successful the past fewer months.

1. Data Filtering & Rebalancing: Remove noisy information and resample your information strategically truthful your exemplary learns much effectively
2. Data Annotation: Gather amended annotations for illustration richer captions, bounding boxes, and font specifications truthful that it's easier for your exemplary to disambiguate ocular conceptsLLMs person gotten infinitely amended astatine image captioning successful the past 12 months. Less truthful for video, but that's for different time.
3. Synthetic Data Generation: Finetune an ensemble of existing generative models to create training information for which location is little-to-no people occurring information (e.g. image editing / reference-conditioning for Nano-Banana style models)Often a postulation of brittle LoRAs trained for very circumstantial tasks and an LLM finetuned to select retired bad synthetic data.

A mates of years ago, the prevailing contented crossed each generative models (be it text, image, audio) was to aggregate as overmuch information arsenic humanly possible for pre-training. Luckily, the section has gotten a batch smarter astir this. If you propulsion a bunch of low-quality information (e.g. heavy compressed JPEGs) into pre-training, your exemplary is going to discarded a important magnitude of its capacity learning really to mimic this portion of data. If you select your dataset well, your exemplary will person a batch easier clip learning what you want it to learn.

We cognize this sounds obvious, but it's a batch harder to do successful practice.Then again, each bully proposal should look evident successful retrospect.

Today we're going to locomotion you done really our attack to information filtering has evolved since 2024. And, hopefully we'll prevention you from a mates of headaches if you extremity up training your ain generative models down the line.

[2024] Filtering connected a fund — Traditional CV connected CPUs

On the first spell around, we decided to push our earthy dataset done old-school machine imagination algorithms. This measurement we could get distant pinch a cluster of inexpensive CPU instances alternatively of an unholy number of GPUs moving a multimodal LLM.Or, spending respective cardinal successful GPT-4 tokens.

Scene detection

We request to select down tens of billions of images and videos to create our pre-training dataset. Images don't really require immoderate circumstantial pre-processing, but earthy videos do.When you train a generative video model, you first request to pre-train connected image generation. If the exemplary learns nouns earlier it learns verbs, it tends to converge amended (and faster).

Next clip you watch a tv show aliases movie, way really often the camera cuts. If you're watching thing made successful the past 20 years, much apt than not you'll spot a trim each 5 seconds. When to trim and really to trim is an authorial decision, not thing a generative video exemplary should do arbitrarily. So, we request to portion n' dice our videos connected changeable boundaries into video clips earlier we tin select them down.

With our cheapskate CPU-only agenda, we picked up PySceneDetect. At a precocious level it maintains a rolling model of K-frames and if the K+1 framework has importantly different image statistics, it categorizes the framework arsenic a cut. There's nary underlying instrumentality learning model. It runs really accelerated but struggles pinch communal transitions for illustration dissolves, fades, and jitter cuts (which debased cardinal is simply a immense issue).

Getting to cognize your data

Whenever you get caller data, you should walk a fewer days reviewing random samples, listing what you'd for illustration to support and what you'd for illustration to propulsion out. Ideally, you return the clip to draught an ontology of categories wrong "good" and "bad" and way the comparative sizes of these categories.

At immoderate constituent during the information filtering process, your technologist encephalon will return over, and you'll walk measurement excessively overmuch clip tuning the knobs of your heuristics (or LLMs), chasing that "perfect" determination boundary. These notes are going to prevention you from yourself down the line. They'll springiness you the facts you'll request to talk yourself retired of trying "one much idea", erstwhile the reply is intelligibly "no".

Plus, knowing the style of the information distribution will really thief pinch dataset rebalancing. Certain categories are overrepresented successful the earthy distribution of each videos. We request to subsample and suppress this signal, different it will predominate training and our exemplary will struggle to study the long-tail of people/places/things/actions that we request successful bid to make anything.

Sieving retired the un-captionable

Generative video models are chiefly constricted by what we tin picture correctly and consistently successful words.Or much precisely, what LLMs tin annotate for america cheaply. Text provides a beautiful bully scaffold to understand the ocular world, but it's by nary intends the correct conditioning system for each aspects of video generation. Details for illustration camera trajectories successful space-time and the nuances of an actor's capacity are simply indescribable successful earthy language.Reference-driven video procreation is presently in-vogue. It's the first articulation of a power for these ineffable properties of videos, but we don't deliberation it's a very bully power per se.It's capable for autogenerated and remixed contented for illustration memes and brainrot, but it doesn't let for the level of fine-grained guidance basal to convey authorial intent. It doesn't alteration america to create the compelling stories that this exertion should herald.

For now, we request to select retired clips wherever the superior "thing" that makes the video clip absorbing is un-captionable. Without a crystal clear matter description, it's conscionable sound to our text-to-video model.

Text-heavy

For example, we want to select retired text-heavy videos. It's still difficult for LLMs to caption mobility graphics that are perpetually changing connected screen.Reading the surface isn't the difficult part, it's establishing a clear explanation of erstwhile and really the matter changes connected screen. We don't want to discarded capacity successful our 2B parameter exemplary learning mobility graphics erstwhile it could beryllium allocated alternatively to learning actions.Zooming out, our extremity is imaginative devices for animation. Text and mobility graphics aren't a privilege correct now.

To do this, we sampled frames from each video and ran a mini EAST Detector to extract bounding boxes for text. From there, we filtered retired matter dense videos based connected the percent of the frames that had matter and the percent of each framework covered successful text.

Using a CNN for this task was a bully idea, but the circumstantial prime was wrong. In bid to tally tens of billions of frames connected CPUs, we had to resize the frames aggressively. So, a batch of text-heavy samples pinch mini fonts fell done the cracks.

EAST is simply a beautiful aged exemplary from 2017. It's mini and acold from the authorities of the creation connected matter detection. Getting it to tally efficiently connected CPUs without cache-thrash and thread oversubscription was a challenge. Even aft capacity optimizations, it was still the largest bottleneck for this type of the information pipeline.It took america weeks to tally EAST connected our dataset, crossed a monolithic cluster of CPUs.

Indescribable actions

When there's not overmuch happening connected the surface (e.g. close-up connected a person's face), it's difficult to picture the circumstantial action taking place. If there's excessively overmuch happening (e.g. highly shaky camera, a shot lucifer pinch a bunch of folks moving crossed the transportation astatine once), LLMs struggle to caption the clip correctly. We lumped these categories of videos together arsenic "indescribable action" clips to beryllium thrown out.

Videos are typically serialized connected disk successful a compressed format. Codecs for illustration H.264 trim record size by storing keyframes and mobility vectors that picture really the keyframes alteration complete time, alternatively than RGB values for each pixel complete time.We're eliding a bunch of specifics here. In H.264, the encoder computes the residual betwixt 2 frames (the framework difference) and past applies the DCT (Discrete Cosine Transform) to the residual. This measurement we get a heavy quantized mobility vector that gives america lossy framework reconstructions successful speech for a smaller record size connected disk.

We utilized the mobility vectors stored wrong the mp4 files themselves to isolate and select retired the "indescribable action" videos.In 2024, we besides tried moving accepted optical travel algorithms for illustration Lucas-Kanade connected CPUs. They're highly brittle and didn't transportation good to unfastened world video. Specifically, we utilized mv-extractor and computed 2 heuristics per clip:

• average_frame_energy: L2-norm of each mobility vectors averaged crossed the video
• min(sub_clip_average_frame_energy): Split each clip into a adaptable number of chunks depending connected the video's length, cipher mean framework power for each chunk, and return the minimum crossed these L2-norms

Then came the determination tree:

• average_frame_energy < 0.1: Throw the clip away. These were fundamentally fixed videos (e.g. slideshows, still frames, freeze-frames).
• average_frame_energy > 25: Throw the clip away. The footage was incredibly chaotic.
• min(sub_clip_average_frame_energy) < 0.03: Throw the clip away. A information of the clip has thing happening (e.g. a fade, modulation to a still image successful a documentary).
• Keep everything else.

This useful good arsenic a inexpensive first filter, but it has mediocre callback (i.e., a batch of indescribable action clips are kept successful the dataset).

Subsampling talking caput video clips

From our first reappraisal of the earthy information distribution, it was beautiful evident that talking caput clips wherever folks talk consecutive to camera were dramatically over-represented. If we fto the dataset be, it would person been importantly biased towards this benignant of clip. Our exemplary would get disproportionately bully astatine creating them (likely astatine the disbursal of others), truthful we needed to find them and subsample them.Remember the VEO-3 videos that went viral successful early 2025? A LOT of them were "man connected the street" videos. The truth that it could make these truthful overmuch amended than different styles of videos is suggestive of a important bias successful the underlying training dataset.

Living successful an old-school CV world, we people burrowed deeper down the engineering passageway and introduced additional heuristics. We sampled frames from each video clip and ran a Haar-cascade look detector to extract bounding boxes for faces and calculated 2 numbers:Haar-like features typically neglect to observe faces that are positioned astatine an perspective comparative to the camera. For us, that was a bully thing. This measurement we could abstracted existent talking caput videos from videos of group taking existent actions.

• average_face_frame_energy: L2-norm of mobility vectors wrong look bounding boxes
• average_background_frame_energy: L2-norm of mobility vectors, conscionable successful the corners of the framework (as a proxy for inheritance motion)

And from there, different determination tree:

• Moderate average_frame_energy + average_face_frame_energy >= average_background_frame_energy: Keep the clip. Usually a really bully close-up.
• Low/Moderate average_frame_energy: Subsample these.

[Early 2025] rm -rf — Replacing hand-crafted heuristics pinch finetuned LLMs

We're starting to sketch a alternatively analyzable determination tree. It's afloat of lossy proxies that only benignant of work, and it's very incomplete.We person omitted a bunch of the categories we tried to filter. For example, clips pinch the Ken Burns Effect. Plus, we haven't opened the tin of worms that is quality-based filtering (e.g. place and propulsion retired pixelated content).

This attack simply doesn't scale. Every clip you person a caller thought for a select you person to re-examine really the caller node successful the determination character impacts each the different branches. Everything is intertwined and yet you extremity up pinch a pipeline that's some un-interpretable and uneditable.

When we started successful 2024, we were staring down the tube of tens of billions of samples. Given our constricted budget, our gut was to conception the cheapest filters possible. This was fundamentally wrong. Our video exemplary struggled to study basal motions for illustration guitar strumming aft training for respective weeks connected our 2024 dataset but was capable to study these very actions successful little than 24 hours of training, aft applying our 2025 filters. Instead of looking for the cheapest filters possible, you should optimize for the champion imaginable filters you tin afford.

Migrating to 1000s of GPUs

This brings america to our 2nd takeaway: propulsion distant your "principled" machine imagination techniques and adopt achromatic container neural networks wherever you can. There are patterns that humans simply can't picture well, nary matter really difficult we try. Old schoolhouse CV methods were the champion hand-crafted approximations of their era. They're genuinely awesome feats of engineering, but a well-trained neural web will study a non-linear usability that will triumph connected precision and callback successful 99% of cases.

Once you re-orient yourself astir this reality, your occupation should displacement from crafting inexpensive heuristics to optimizing models for GPU throughput and engineering resilient, parallelizable workloads to tally connected SPOT instances crossed providers.In his YC talk from summertime 2025, Karpathy discusses a akin realization successful his activity astatine Tesla AutoPilot. He describes it arsenic the modulation from "software 1.0" to "software 2.0".

Concretely, we replaced PySceneDetect's heuristics pinch AutoShot and TransNetV2, accelerating conclusion pinch civilization CUDA kernels.These models still struggle pinch capturing 100% of shots. For example, they still miss framework jitter cuts. Good capable for now. But, successful the early we'll person to train these models pinch caller information to capable successful these gaps. We stopped moving the ancient EAST Detector connected CPUs. Instead, we deployed PaddleOCR pinch TensorRT crossed thousands of Nvidia A10Gs and L4s.It only took ~36 hours to rip done tens of billions of samples connected GPUs. Less than $10K USD for faster, amended results. And, we ripped retired the tangled web of machine imagination algorithms (Haar Face Detectors, Lucas-Kanade, monocular extent estimators, etc.), replacing them pinch a group of fine-tuned LLMs.We still usage H.264 mobility vectors arsenic a inexpensive first select for highly debased and precocious mobility clips.

Iterative dataset labeling (aka self-consistency is harder than you think)

Finetuning is beautiful straightforward acknowledgment to the folks astatine Unsloth.Serving these models is somewhat much tricky. We usage vLLM for illustration everyone else. But, it's almost intolerable to guarantee nonstop bit-wise compatibility betwixt Unsloth and vLLM. Every clip we've had to deploy pinch vLLM it's eaten up a week aliases so, tracing done vLLM codification and modding the nonstop ops for image/video patchification (or fixing mini bugs successful vLLM) truthful that we tin lucifer the accuracy of the results we get retired of the container from our LoRAs aliases fine-tuned LLMs. This intends the labeling is the work.

The difficult portion of training LLMs for information filtration is that the categories are ever somewhat fuzzy. You'll person to reply questions like:

• If a sample fits respective categories to different degrees, which explanation do I delegate it?
• Should I simplify my categories, truthful I tin explanation the information much quickly and consistently? Or, do I request to divided my class into pieces to make it clearer?
• How easy is this conception for the LLM to learn? How overmuch information do I request for each category?

More often than not, the biggest problem you'll tally into is 1 of self-consistency. Over the people of labeling a mates 100 samples, it's only earthy that you'll relax your criteria, mislabel samples, and muddy the awesome successful your dataset.

In a CMU study from 2025, researchers hired cinematography experts to annotate camera mobility successful online video clips and train different laypeople to make akin annotations. Even pinch the criteria successful hand, the experts disagreed pinch "ground truth" ~24% of the time. Only done repeated tests were they capable to converge connected 96% agreement. It's achy to walk days labeling and re-labeling a dataset, but them's the breaks. At least, we're fortunate to unrecorded successful an era wherever you only request 2-3K samples to train a bully filter.And, not the early heavy learning days wherever you'd request astatine slightest tens of thousands of samples to train a CNN from scratch.

If you find yourself moving connected information filtration, we'd urge you hack together a elemental labeling instrumentality for illustration the 1 we show below. We conscionable slapped together a ace elemental React app pinch Supabase to shop labels and R2 to shop the samples.

Turning LLMs into categorical classifiers

Ultimately, we supervise-finetuned (SFT) Qwen-2-VL-2B to tag:

• Image Categories: Ugly Product Image, Diagram / Screenshot, Collage, Watermarked, Bad Lighting, Pixelated, Drawing / Illustration, Keep
• Video Categories: Animation, PoV, Bars, Motion Graphics, Ken Burns, Shaky Camera, Little to No Motion, Weird Transition, Keep

We only kept images that our select predicted arsenic Drawing / Illustration aliases Keep. For videos, we retained Animation and Keep clips wholesale, while subsampling PoV.

[Late 2025] Reinforcement Learning pinch Verifiable Rewards (RLVR) for artistic filtering

At the commencement of the information labeling process, we tried to get highly circumstantial astir the properties of the images and videos that divvied up samples into disfigured vs. beautiful (e.g. overexposed lighting, muted colour grades). We thought it would beryllium easier for the LLM to study the precise reasons why we considered an image disfigured than study an arbitrary "ugliness score".

Once again, our first intuition turned retired to beryllium wrong. The properties that make a peculiar sample disfigured thin to beryllium correlated; you extremity up assigning K different artistic tags to the aforesaid sample. And successful turn, this poses 2 important challenges:

• Sparse Data Signal: The combinatorial detonation of tags makes it harder for the exemplary to disentangle the categories, particularly pinch a mini dataset of a fewer 1000 labels.
• Slow, Inconsistent Labeling: It takes a batch longer to explanation samples (and it's a batch harder to beryllium self-consistent) erstwhile you person the cognitive load of weighing respective imaginable tags per sample.

We ended up grading the samples connected a standard from 1 to 4 and keeping the samples that our models branded 3 aliases 4. This isn't an original thought by immoderate means. LAION Aesthetic Predictor was utilized to select information for the original Stable Diffusion. That exemplary took CLIP embeddings and trained a mini linear furniture to foretell an artistic score. While useful, it was beautiful brittle. If you excavation into the predictions, you'll find an abundance of weird ratings. For example, circumstantial types of watercolor illustrations each get a very precocious people sloppy of noticeable JPEG degradation.

Fine-grained artistic scorers

Hang pinch us, arsenic we activity done a short history lesson.

The superior measurement that folks traditionally finetuned LLMs is supervised-finetuning (SFT).Either you tin update each the parameters successful the network, aliases train a LoRA wherever you insert a mini number of parameters passim the web and only train this caller group of parameters. We utilized this attack erstwhile training our categorical classifiers. We fed Qwen-2-VL-2B triplets of (image/video frames, punctual Categorize this arsenic A, B, aliases C, existent label) and utilized the aforesaid nonsubjective from pre-training (next-word-prediction-with-cross-entropy) to update the model. With SFT, you're fundamentally extending pre-training, truthful your exemplary learns your caller task.

At the commencement of 2025, DeepSeek popularized a different finetuning method called "Reinforcement Learning pinch Verifiable Rewards" (RLVR) successful their R1 paper. Instead of utilizing next-word prediction, they utilized reinforcement learning (RL) wherever the model's consequence is graded for accuracy utilizing a rubric.

It turns retired that if your LLM is capable to lick a problem moreover 1 successful 1,000 times, we tin nudge the web towards uncovering this solution much consistently. These web updates are overmuch smaller than those provided during SFT, truthful RLVR allows america to exert overmuch finer grained power connected extending the LLM to our tasks. Bluntly put, SFT is exemplary room pinch a butcher knife, while RLVR is exemplary room pinch a scalpel.

By the clip we had our artistic dataset fresh to go, RLVR was gaining take by the awesome labs; truthful we gave it a spell for our artistic scorers and it smoked its SFT counterparts.If you look astatine each the applicable advancement made successful LLMs successful the past 18 months (Claude Code, cybersecurity, etc.), you tin straight property the gains to RLVR. Math and coding trust heavy connected reasoning to hunt the solution abstraction effectively, earlier issuing an answer. Previously, OpenAI's o-series models were bottlenecked by the magnitude of information they could stitchery from humans. With RLVR, a batch little quality information is necessary. The rubric stands successful for tens of thousands of human-written reasoning traces. It's specified an effective method, that immoderate models are capable to study really to logic pinch zero quality traces. Our punctual explicated the different reasons why we'd people a sample from 1-4, while our rubric penalized the exemplary connected the absolute quality betwixt the crushed truth and predicted labels. Specifically, we utilized Group Sequence Policy Optimization (GSPO) arsenic the RLVR algorithm connected Qwen-2.5-VL-3B.This is conscionable a mini tweak connected the GRPO (Group Relative Policy Optimization) algorithm that DeepSeek projected successful their R1 paper.

Filtering end-to-end

After RLVR, we adopted WAFT (a SOTA optical travel predictor) to select retired moreover much of the agelong tail of low-motion videos. Instead of relying connected heuristics for illustration Haar-cascades, we converted our video captions into tags and utilized these to edit our training information distribution (e.g., subsample talking heads, oversample animal videos).

Nowadays, location is simply a batch of motto being tossed astir the net erstwhile it comes to "pre-training", "mid-training", and "post-training". The beingness of chopped pre-training and mid-training phases is conscionable a usability of the impossibility of creating the cleanable filter.

During pre-training, folks are much permissive. They're unfastened to letting immoderate low-quality information into the stack, truthful that they tin guarantee that they screen each the basal modes of their target information distribution. And by the clip they get to mid-training, they're consenting to tighten the distribution, moreover if it intends throwing distant immoderate bully stuff.

Regardless, information filtering is the azygous biggest lever you tin propulsion extracurricular of scaling to amended the value of your model. If you return 1 point distant from this journey, ne'er skimp connected your information filtering pipeline. These things are what they eat aft all.

Who are we?

We're 2 brothers training text-to-video models from scratch, trying to make animation accessible to everyone. Stay tuned for much blogs connected captioning, synthetic information generation, and our ongoing activity exploring amended manifolds for pixel-space generative models.