🩹 AI Object Removal in Video: Where the Patch Holds

The tell was a shadow. I was cutting a thirty-second spot last spring and the client wanted a light stand gone from three shots of a handheld dolly move. The eraser took it out cleanly, and on the paused frame I would have signed off. Then I scrubbed at quarter speed and watched a soft grey lozenge slide across the concrete where the stand's shadow still lived, perfectly tracked, perfectly wrong. The object was gone. Its evidence was not.

That is the shape of nearly every object-removal failure I have seen on the bench, and it is the shape that the tool pages selling you an eraser never describe. They show you a before and after still. Stills are not the job. Dailies are the job, and dailies run at speed.

By the numbers#

• 80 frames — ProPainter's default subvideo_length, with neighbor_length 10 and ref_stride 10. The model reasons over a bounded chunk, not your whole clip. The docs note that reducing this value "effectively decouples GPU memory costs and video length."
• 81 frames — the training length for Wan VACE video inpainting at 720x480, and the minimum input length the ComfyUI reference workflow expects. Longer clips get tiled across overlapping segments.
• 25 GB VRAM — what ProPainter needs for 1280x720 at 80 frames in fp16; the same job in fp32 runs out of memory. Drop to 720x480 and it is 13 GB (fp32) / 8 GB (fp16); at 320x240, 4 GB / 3 GB. Resolution drives the memory cost, not clip length.
• 5 seconds, 15 credits per second — the input cap and price on Runway's Remove from Video app, published on the product page itself.
• 5 side-effect categories — shadow, reflection, light, translucency, and mirror, as catalogued by ROSE (NeurIPS 2025), which reports that existing removal methods struggle to eliminate them because paired video supervision is scarce.

The seam is temporal, not spatial#

Every ranking page for "ai video editor to remove objects" evaluates the result the way a photo retoucher would: does the patched area blend into its neighbors within this frame. That is a spatial question, and modern inpainting answers it well. The question that decides whether a shot goes to air is whether the patched area agrees with itself over time.

Here is the mechanic nobody names. ProPainter, which is the open reference implementation most people's understanding traces back to, processes video in sub-video chunks with a default length of 80 frames. Inside that chunk it pulls local neighbors (10 frames) and strided global references (every 10th frame) to decide what belongs in the hole. The coherence guarantee lives inside the window. At the boundary, a new window opens with a different reference set, and the reconstruction can shift. On a 24fps timeline, 80 frames is about 3.3 seconds. Wan VACE's generation window is 81 frames; longer clips are handled by sliding overlapping segments across the sequence, which is a mitigation, not a cure.

So the artifact hunt has a schedule. On a six-second shot, mark the timeline at roughly 3.3 seconds and look hard there. That is not the middle of the fill; that is where two independent reconstructions were asked to shake hands.

So what: short shots are structurally safer than long ones. If a removal has to work, and you have any say in the edit, keep the affected shot under one window.

What happens at a cut#

An editor's instinct here is right, and worth stating plainly because no vendor page does. Temporal context does not cross an edit point. When you hand the model a cut sequence, it does not understand that frame 119 and frame 120 are different camera setups in different places. It sees a discontinuity in the pixel stream and re-derives the fill from whatever neighborhood it now has.

The result is the most avoidable failure in this whole discipline: the same removed object patched two different ways on either side of a cut, which the eye reads instantly as a jump. Continuity errors are the ones I cannot forgive, because they are the ones an audience notices without knowing why.

The rule is simple. Split the timeline into individual shots. Export each shot as its own clip. Run removal per shot. Reassemble. It does not cost more, and it eliminates a whole failure class before you spend anything. If you are budgeting a multi-shot job, this also changes your arithmetic; the AI video cost breakdown and the cost calculator are worth a pass before you start, because per-shot billing at 15 credits per second adds up faster than a single-file estimate suggests.

Borrowed pixels versus invented pixels#

The single best predictor of whether a removal will hold has nothing to do with which tool you pick. It is whether the pixels behind the object are ever visible in any frame of the shot.

Two cases. If the camera moves, or the object moves relative to the background, the region behind it gets revealed somewhere in the sequence. The model propagates those real pixels into the hole. Fidelity is high, and the patch tends to sit still because it is made of photographed material.

If both the camera and the object are locked, nothing behind the object is ever revealed. Every pixel in that hole is invented, and invented pixels are re-invented on every window. That is where the breathing texture comes from, the slow shimmer, the wood grain that reorganizes itself over two seconds.

Pre-flight test, thirty seconds on the bench: park the playhead at the first frame, then the last, then two points in between, and ask whether you can see the background behind the object in any of them. If yes, the model is borrowing and you are probably fine. If no, you are asking for a hallucination and should plan accordingly.

So what: a static locked-off shot with a static object is the hardest case, not the easiest, even though it looks simplest.

The footprint problem#

The mask covers the object. The object's effect on the scene is larger than the object.

ROSE names five distinct side effects: shadows, reflections, cast light, translucency, and mirror images. A standing lamp does more than occupy space; it puts light on the wall. A person by a window is also in the window. A car on wet asphalt is also in the puddle, sometimes in several puddles at once. Erase the object with a tight mask and the scene keeps testifying that it was there.

The manual fix is to mask deliberately for the footprint: extend the mask over the cast shadow, over the reflection region, over the pool of light. The tradeoff is honest and unavoidable. A larger mask means more invented pixels, which pushes you back toward the hallucination case above. On a shadow lying on plain concrete that trade is usually worth it. On a shadow falling across a patterned rug, it usually is not.

I have not found a consumer tool that handles side effects automatically with any reliability, and I will not claim otherwise. ROSE's own framing is that this is an open research problem constrained by data.

Backgrounds that break the patch#

PROVE-H, the hard half of PROVE's benchmark, was built from exactly the scenarios that break removal in practice: crowd scenes, moving backgrounds (flowing water, flames, rain, snow), heavily textured backgrounds, complex reflections with intertwined side effects such as multiple puddle reflections, and fast motion. It is 100 videos with no ground truth, which is itself a statement about how hard these cases are to score.

Paired with the artifact each one tends to produce, from my own passes:

So what: read the background before you read the tool comparison. A mediocre model on plain plaster beats the best model on a crowd by a wide margin.

Mask propagation is the actual workload#

Every landing page frames masking as a formality: brush it once, the AI follows it. In practice the mask is where the job is won or lost, and most artifacts blamed on the inpainting model are tracking failures wearing a costume.

Four things break propagation: occlusion (something passes in front of your object), motion blur (the boundary the tracker needs stops existing), rapid scale change (the object rushes toward camera), and the object leaving and re-entering frame (the tracker often re-acquires a different region, or nothing).

Diagnosis is easy once you know to do it. Turn the fill off and inspect the mask alone, stepping frame by frame through the moment where the artifact appears. If the mask slips, lags, or shrinks, you have a tracking problem, and switching inpainting models is wasted money. If the mask sits perfectly on the object and the filled area still misbehaves, the fill is at fault. Fix the right layer. The same discipline applies to any AI process that must hold identity across frames, which is why the reasoning in consistent characters in AI video transfers here almost directly.

Hard limits, stated as numbers#

Cloud: Runway's Remove from Video app accepts input videos up to 5 seconds and charges 15 credits per second. That is a real constraint on how you plan coverage, and I have not seen a single roundup mention it. A ten-second shot is two jobs and two independent reconstructions, which reintroduces the boundary problem by another route. If you are weighing cloud tools generally, best AI video generators and Runway vs Kling cover the surrounding stack.

Local: ProPainter's published memory table is the sizing document nobody reprints. 1280x720 at 80 frames is out of memory in fp32 and needs 25 GB in fp16. 720x480 at 80 frames needs 13 GB / 8 GB. 320x240 needs 4 GB / 3 GB. Resolution is the driver; clip length is handled by chunking. If you are choosing hardware for this kind of work, best GPU for local AI and local vs cloud AI are the relevant reading, and quantization explained covers why fp16 halves the bill.

NLE: DaVinci Resolve's object removal and Magic Mask run on the DaVinci Neural Engine, which ships only in paid Resolve Studio. The free version does not have them. VRAM is typically the limiter, since these tools touch every frame.

QC, and why the scores lie#

PROVE's contribution is uncomfortable for anyone hoping to automate quality control. Its authors state that full-reference metrics "reward copy-paste behaviors over genuine erasure," that no-reference metrics carry systematic biases such as favoring blurry results, and that global temporal metrics are "insensitive to localized artifacts within edited regions." Their proposed replacements are RC-S, which compares features between the masked and background regions with a sliding window, and RC-T, which tracks distributions within shared restored regions across adjacent frames. The paper is accepted to ACM MM 2026.

Until those land in tooling you can run, the substitute is manual and takes about four minutes per shot:

1. Play the patched region at quarter speed, full screen, twice.
2. Watch the mask boundary, not the mask center. Failures announce themselves at the edge first.
3. Step across every window boundary. Mark them at ~3.3-second intervals for an 80-frame model at 24fps.
4. Step across every cut in the assembled sequence, one frame at a time, both directions.
5. Check the footprint: shadow, reflection, cast light.

Triage before you spend#

The honest limitation in all of this: I cannot give you per-tool success rates, because nobody publishes them on comparable footage, and my own numbers come from commercial work I cannot post. What I can give you is the mechanic. A patch is generated inside a window, from a neighborhood, under a mask. Every failure worth knowing traces back to one of those three, and you can read all three off the shot before you spend a credit.