AI sound design: how to add voice, music and effects to AI video

Why sound is half of an AI film

This is lesson 5 of the free course Make a short film with AI: your shots are rendered and chosen, and now they need to sound like a film.

Audiences forgive a lot in the picture if the sound is confident. They forgive almost nothing if it is missing or wrong. Sound also does work the picture cannot: it carries continuity across cuts, tells the viewer how to feel before anything happens, and hides the seams between clips that came from different generations. In an AI film, where every shot is a separate render, that glue matters even more.

Film sound is built in four layers. Plan all four for every scene:

Dialogue or voiceover

The words. Always the top priority in the mix; if it competes with anything, the other thing gets quieter.

Music

Emotion and pace. A bed that runs under scenes, plus hits that land on key moments.

Ambience

The sound of the place: wind, traffic, a café murmur, room tone. Quiet, continuous, and the main tool for hiding cuts.

Sound effects

Sounds tied to on-screen action: a pour, a footstep, an engine, porcelain clinking. Sync matters more than volume.

This clip has no soundtrack (apart from one clip later in this lesson, our renders play silently on this site), so use it as an exercise. Sound editors call it a spotting session: watch the cut and list every sound it needs, with a timestamp. For this six-second shot, the list might read:

• 0 to 1 s: teacups sweep past the lens. A soft whoosh for each, a faint porcelain clink.
• From about 1.5 s: tea pours into the cup. A liquid pour that rises in pitch as the cup fills.
• Throughout: open-air ambience with a light breeze, so the endless checkerboard feels like a real place.
• Music: something off-kilter and delicate, like a music-box waltz, entering under the first whoosh.

Four lines, and you already know what to generate, what to prompt for and what to leave out. Do this for the whole cut before you make a single sound.

Native audio vs adding sound after

Several current video models generate sound with the picture. At the time of writing, Veo 3.1, Kling 3.0, Seedance 2.0 and PixVerse V6 all list native audio (dialogue, effects and ambience) on their model pages, usually as a setting that raises the per-second price. The model comparison lesson covers which does what.

Here is what native audio looks like in practice. We re-ran the tea party on Veo 3.1 Fast, kept its native audio, and wrote the sounds from our spotting list into the prompt, after the visuals:

Use the "Sound on" toggle to hear it. The prompt asks for porcelain clinking, tea pouring, a light breeze and a soft music-box melody; listen for how many of those you can pick out, and whether the pour lands when the hare tips the teapot. Two things we could measure: the clip carries a single stereo track with everything already mixed together, and that track is quiet, about -31 LUFS integrated on our meter, far below the delivery target in the mixing section. Even native audio needs leveling in the edit. On price, the same model and length cost us 225 credits with sound and 150 without (the silent tea party above), so the audio setting is a line item in the budget.

Native audio and post-produced audio solve different problems:

A practical split: use native audio for diegetic sound that must be in sync (the clink when a cup lands, a line delivered on camera), and build music, narration and the ambience bed afterward. When you do prompt native audio, describe it after the visuals, tie each sound to an action, and ask the model to leave out music so the score stays yours. Models do not always obey that last part, which is one more reason to keep prompted sounds simple.

How to write a voiceover script for AI text to speech

Text-to-speech models read exactly what you give them, so the script is also the performance direction. Most of the craft is writing for the ear and letting punctuation do the pacing.

1. Budget the words. 150 words per minute is the common planning figure for narration, which is about 2.5 words per second. A five-second shot holds roughly 12 words, and fewer if you want room to breathe. Time the generated audio, not the page.
2. Let punctuation direct. A full stop is a real pause. A comma is a short beat. A dash gives a sharper break. An ellipsis suggests hesitation, though models handle it less predictably.
3. One thought per line. Short sentences give the voice natural places to breathe and give you clean edit points between lines.
4. Write numbers and names as they sound. "Twenty twenty-six", not "2026"; "Ky-OH-toh" if the default pronunciation is wrong. Some models offer a pronunciation dictionary or explicit pause tags, but the syntax differs per model, so check the model page before relying on it.
5. Generate per scene, not per film. Shorter takes are easier to retime, re-record and replace when the edit changes.

How to write a music brief for AI music

"Epic cinematic music" gets you the most average epic cinematic music a model can produce. A music brief works like an image prompt: specific nouns beat adjectives. These are the fields that change the result:

• Genre and era: "1960s French pop", "minimal piano and strings", "lo-fi hip hop". Describe the qualities you want rather than naming a living artist.
• Tempo in BPM: pick it to fit your shot lengths (see below).
• Instrumentation: name three or four instruments. Fewer instruments leave space for a voice.
• Mood arc: how it changes over time. "Sparse and curious, builds from 0:20, resolves warmly at the end."
• Length and the hit: the exact length of the cut and where the big moment lands, in seconds.
• What to leave out: vocals, heavy drums under dialogue, a fade-out if you need a clean ending.

Generate a few seconds longer than the cut and trim in the edit; AI music often ends abruptly or noodles at the end. If the hit lands in the wrong place, move the music rather than the picture.

AI lip sync for talking characters

When a character speaks on camera, you have two routes. Generate the line natively in a video model that supports dialogue, or render the shot first and drive the mouth afterward with a lipsync model that takes a face (image or clip) plus an audio track. The second route lets you lock the voice first, reuse it across shots and change a line without re-rendering the whole scene.

Check the result at full speed and then frame by frame on hard consonants (p, b, m), where the lips should close. If one line fails, regenerate that line only. Mix music and ambience back in after lipsync, never before.

How to mix AI audio: levels, ducking and loudness

Mixing is mostly deciding what the viewer should hear at each moment, then making everything else quieter. Work in this order:

1. 01

   Set the dialogue first

   Voice and on-camera lines sit on top. Everything else is balanced against them.

2. 02

   Duck the music under the voice

   Pull the music down while someone speaks and let it rise in the gaps. If you cannot understand a word with your eyes closed, the music is too loud.

3. 03

   Lay ambience across the cuts

   Keep one continuous ambience bed under each scene, even when the shots came from different models. It is the cheapest way to make separate clips feel like one place.

4. 04

   Place effects on the frame

   Line each sound effect up with the exact frame of the action. Sync sells an effect more than volume does.

5. 05

   Measure loudness on the full mix

   Around -14 LUFS integrated, with true peaks below about -1 dBTP, is the widely cited target for YouTube and most streaming. YouTube turns louder uploads down, so mastering hotter gains nothing. Most editors include a loudness meter; check the whole film, not one clip.

Listen on two systems before you export: headphones and a phone speaker. A mix that only works on headphones will lose its dialogue on the phone, which is where most short films are watched.

Licensing and disclosure for AI music and voices

Commercial rights for generated audio depend on the model provider's license and on your plan, and the terms vary. Some music providers restrict distributing generated tracks to music streaming services or require separate terms for broadcast and cinema. Read the terms for each model you use and keep a record of what you generated: model, prompt or brief, and date. If a track is ever claimed, that record is your evidence.

• Describe, don't imitate. Brief a style by its qualities (tempo, instruments, era), not by the name of a living artist.
• Voices belong to people. Only clone or imitate a real voice with the person's consent.
• Disclose realistic synthetic voices. YouTube's disclosure rules specifically cover synthetically generating a person's voice to narrate a video. The editing lesson covers platform labels in full.

Build the soundtrack in PromptLab Flows

In PromptLab, the whole soundtrack can live on the same node canvas as your shots. Each layer maps to a node:

The "Yacht promo" starter template shows the pattern: a hero image becomes a video, a voiceover and a music bed are generated alongside it, and Composition joins them. "Talking-head onboarding" chains a portrait, a voice and a lipsync node into a single MP4. A sensible starting point in Composition is voice at 0 dB and music around -12 dB, then adjust by ear. Composition sets a level per track; ducking under individual lines, placing a hit on an exact frame and loudness metering belong in an editor, which is where the next lesson picks up.

With picture and sound in place, the last step is the edit: trimming each clip, pacing the cut, matching color and exporting for each platform. Continue to how to edit AI video into a finished short film.

Questions people ask