I Let AI Write My Video Using Remotion and Claude — Here’s What I Learned

By Ivana Tilca · January 27, 2026 · 5 min read

The typical text-to-video tools disappointed me — until I tried building videos as code with Remotion and an AI agent. Here's what Remotion is, why AI 'skills' make it click now, and what an honest experiment taught me about where AI video is really headed.

What amazes me the most about AI is video generation — and honestly, for a long time it was also the thing that disappointed me the most. The typical "text-to-video" tools gave me clips that looked impressive for three seconds and then fell apart: inconsistent characters, weird morphing, zero control over the details that actually matter. Then I tried a completely different approach with Remotion, and for the first time it clicked. Not because it's magic, but because it hands you back the one thing every other tool takes away: control.

What is Remotion?

Remotion is an open-source library for creating videos using code. Instead of dragging clips around a timeline in a traditional editor, you build your video the way you'd build a web page — with React components. Every frame is rendered from code, which means everything is programmable: text, shapes, images, transitions, timing.

That "it's just code" part is what makes it click with AI. Because the video is expressed as code, an AI agent like Claude or Codex can actually write and edit it. You describe what you want, the agent generates the components, and Remotion renders the result. It's especially good for the kind of content a lot of us make constantly — shorts, reels, ads, explainers — where you're reusing a format and swapping in new data, text, or timing.

And because it's code, you get to use real programming constructs: loops to repeat elements, conditions to branch layouts, APIs to pull in live data, and JSON to drive the whole thing. A video stops being a one-off artifact and becomes something closer to a template you can generate on demand.

Why Remotion matters now

Remotion first launched back in 2021, with the goal of creating videos programmatically instead of through a traditional editor. That was already a cool idea. But something changed recently that makes it matter far more today: Remotion published guidelines and best practices specifically for AI agents.

Here's why that's a big deal. Video generation with agents was technically possible before, but it was clumsy — the agent didn't really "know" how to work inside a Remotion project, so you spent a lot of time correcting it. Now that there's a defined way for agents to operate, they can handle the full cycle — scaffolding the project, writing the components, wiring up the data, and rendering — without you babysitting every manual step.

It's also genuinely accessible. The free license covers a lot:

Unlimited videos

Commercial use allowed

Self-hosted cloud rendering allowed

For an independent creator, that combination — free, commercial, self-hostable — is rare and worth pausing on.

So what exactly are "SKILLS"?

If you haven't come across the term yet, skills are an open format for giving AI agents new capabilities and expertise. The idea is simple but powerful: instead of hoping the model already knows how to use a tool correctly, you give it procedural knowledge and constraints — a set of rules and best practices for a specific task.

Remotion maintains a list of skills that define best practices for working inside a Remotion project. In practice, that means the agent isn't guessing how to structure a composition or handle timing — it's following the guardrails the Remotion team wrote. Skills are essentially the difference between an agent that can technically touch your project and one that works in it competently.

This is the part I find most interesting, because it generalizes far beyond video. Skills are how we're starting to turn general-purpose agents into reliable specialists.

What the experiment actually taught me

A few honest takeaways from doing this myself, not just reading about it:

The control is the whole point. Being able to say "make this text appear at second 2, hold for 3 seconds, then slide left" and have it actually happen — reliably — is worth more than any flashy generative effect.

It's iterative, not one-shot. The first render is rarely perfect. But because it's code, fixing it is a precise instruction, not a re-roll of a slot machine. That feedback loop is fast and predictable.

You still need taste. The agent handles the mechanics; the timing, pacing, and "does this feel good" judgment are still yours. That's not a limitation — it's the fun part.

The bigger picture

What this experiment made clear to me is that AI video generation isn't about replacing creators — it's about changing how we create. The interesting frontier isn't a button that spits out a finished video; it's a workflow where you direct and the agent executes, inside a system that's transparent and controllable.

We're still early. The tools will improve. The agents will get smarter. But the direction is clear: video is becoming software. And once video is software, everything we've learned about building software — templates, version control, automation, reuse — starts to apply to it too.

That's a shift worth paying attention to. If you make video content regularly, I'd genuinely recommend spending an afternoon with Remotion and an AI agent. Not because it'll replace your editor tomorrow, but because it'll change how you think about what a video even is.