Case study — an AI video production pipeline

The clearest way to judge whether someone can help you put AI to work is to look at something they built that runs every day. So here is one, described honestly — including the parts that are fragile.

What it is: a production system that takes a structured Markdown file and returns a finished vertical video — original music, generated imagery, optional narration, word-accurate captions, and the social copy to publish it with. It runs commercially: my father's business, RanioLab, sells video production built on it. I built the system; he runs the business. That family connection is why I can show you cost figures and failure modes that agencies never publish.

The problem it solves

Producing one short-form video conventionally means coordinating four specialists: a composer, an illustrator, a voice artist, an editor. That is expensive, and worse, it is slow to iterate. Producing thirty of them means doing that thirty times.

The interesting question was never "can AI make a video." It was: which parts of this can be automated end to end without a human checking each intermediate step, and which parts genuinely need judgement?

The answer turned out to be that almost all the production is automatable, and almost none of the authoring is. That split determined the whole architecture.

Architecture

Five sequential stages, each producing assets the next consumes:

              CONTENT AUTHORING (human, assisted)
                            |
                  Structured Markdown file
                            |
        +-------------------+-------------------+
        |                   |                   |
   [1] MUSIC           [2] NARRATION        [3] IMAGERY
     Suno               ElevenLabs          Google Flow
      WAV                  MP3               10x PNG
        |                   |                   |
        +-------------------+-------------------+
                            |
                  [4] COMPOSITION (FFmpeg)
                   Ken Burns + crossfades
                       audio mixing
                            |
                  [5] CAPTIONS (CTC / Whisper)
                    word-level timing
                            |
                    Publish-ready MP4

Stage 1 — Music. Original songs with custom lyrics, generated per item. Two variants come back for every request, so there is always a choice.

Stage 2 — Narration. Optional text-to-speech, multi-language. When narration is present the mix puts voice at full volume and music at 30% underneath.

Stage 3 — Imagery. Ten images per video, composed in 9:16, prompted individually rather than from a template.

Stage 4 — Composition. FFmpeg assembles stills into motion using six rotating Ken Burns effects with one-second crossfades. Image duration is derived from total audio length, so the visuals always fit the track rather than being cut to fit.

Stage 5 — Captions. Word-level timing, exported as ASS subtitles.

The decisions that actually mattered

This is the part worth reading, because it is the part that transfers to a different problem.

Choosing the right model for the actual task

The obvious way to caption a song is to run Whisper over it. It performs poorly — Whisper is trained on speech, and sung vocals with instrumentation behind them are not speech.

But there is a detail that changes the problem entirely: the lyrics are already known. They were written before the song existed. So this is not transcription, it is alignment — matching known text to audio. That is a different task with different tooling, and Meta's MMS CTC forced aligner does it far more accurately.

The pipeline separates the vocal stem from the instrumentation first, then aligns the known lyrics against the isolated vocal.

The transferable lesson: a large fraction of "AI doesn't work for this" is actually "the wrong task was framed." Recognising that transcription and alignment are different problems is worth more than any amount of prompt tuning.

Two paths, chosen automatically

Narration is a genuinely different case — the text may not exactly match what was spoken, so alignment is wrong and transcription is right. The system detects which kind of audio it has and takes the appropriate path: forced alignment for lyrics, Whisper for narration, with silent regions detected and skipped so captions never appear over instrumental sections.

One pipeline, two strategies, no operator decision required.

Automating where no API exists

Two of the three generation services have no usable API for this workflow, so the system drives their web interfaces through Chrome's remote debugging protocol.

That produced a genuinely awkward problem. Suno's interface is built with React controlled components, which ignore synthetic keyboard events — typing into them programmatically does nothing. The fix is to set the underlying input value directly and dispatch the events React itself listens for.

I want to be clear that this is the weakest part of the system. Browser automation against someone else's UI is a dependency on a thing that can change without warning, and when it changes it breaks silently. It is the right call here — the alternative was not building it at all — but it is a liability, and any honest assessment says so.

Making new work require no new code

The Markdown parser maps headers to pipeline fields through a per-project config file, so a new content type — a different subject, a different language, different section names — is a configuration change, not a code change.

This is the difference between a demo and a system. Onboarding a new content line takes minutes.

What it costs

Measured, not estimated:

Fixed subscriptions across all three services come to roughly €22/month, supporting about 300 videos.

The number that matters is the ratio. Human time is 89% of the cost of a finished video. Every AI service combined is under 6%.

That finding is worth more than the pipeline itself, and it generalises. When people plan AI adoption they optimise API spend, because it is the line item with a price attached. Here, halving every model cost would improve unit economics by about three percent. Removing two minutes of human handling would improve them by nearly ninety.

Look for the labour, not the tokens. That is usually where the money is.

Where it breaks

An honest system description includes the failure modes.

• Browser automation is brittle. A UI change upstream breaks a stage, without warning and without an error that says so.
• Music generation is the bottleneck, at four to five minutes per song and sequential. Everything else could run in parallel; the songs cannot.
• Image generators refuse prompts unpredictably, so the pipeline detects rejections rather than assuming a request succeeded.
• Authoring is not automated and should not be. Deciding what a video should say is the part that needs a person. The pipeline removes the assembly, not the thinking.

What this means if you are considering AI for your own work

Four things, all of which came out of building the above rather than reading about it:

1. Measure where the time goes before you buy anything. If humans are 89% of your cost, model pricing is nearly irrelevant to your decision.
2. Frame the task precisely. Transcription versus alignment looked like the same problem and was not. Most disappointing AI results are a framing error, not a capability limit.
3. Build configuration, not features. If a new use case needs code, you will stop at the first one.
4. Know which dependencies can break silently, and decide deliberately whether that risk is acceptable. Sometimes it is.

If you want that kind of assessment applied to your own workflow, the first conversation is free and roughly half of them end with "not yet, and here is what would have to change first" — start here.