Music Video Mode

Create music-synchronized video content with lyrics-driven narrative and beat-aligned shots through a 5-phase pipeline.

What is Music Video Mode?

Music Video Mode is a specialized storyboard mode that:

• Analyzes your audio track for structure, BPM, key, and energy
• Extracts and interprets lyrics to drive visual narrative
• Creates a holistic story interpretation of the entire song
• Builds a visual bible for consistent aesthetics
• Generates beat-aligned, lyrics-driven scenes and shots

Enabling Music Video Mode

1. Create a new storyboard project
2. Set the Goal to Music Video
3. Upload your audio track (required)
4. Optionally add a Visual Theme description to guide the overall look
5. Add reference images for character or style guidance
6. Click Generate Storyboard

The music video pipeline (generate-music-video-orchestrated) runs instead of the standard storyboard pipeline.

The 5-Phase Pipeline

Audio Track Upload
     |
+----------------------------+
|  PHASE 1: AUDIO ANALYSIS   |
|  Structure, timing, mood   |
+-------------+--------------+
              |
+----------------------------+
|  PHASE 2: LYRICS EXTRACTION|
|  Primary content driver    |
+-------------+--------------+
              |
+----------------------------+
|  PHASE 3: STORY            |
|  INTERPRETATION            |
|  Holistic song narrative   |
+-------------+--------------+
              |
+----------------------------+
|  PHASE 4: VISUAL BIBLE     |
|  Style, colors, motifs     |
+-------------+--------------+
              |
+----------------------------+
|  PHASE 5: SCENE GENERATION |
|  Story-driven, beat-aligned|
+----------------------------+
              |
       Music Video Storyboard

Phase 1: Audio Analysis

The system analyzes your uploaded audio track to extract structural information.

What Gets Detected

Element	Description
Total Duration	Length of the track in seconds
BPM	Tempo / beats per minute
Key	Musical key (e.g., C minor)
Genre	Detected musical genre
Sections	Structural segments (intro, verse, chorus, bridge, outro, drop, breakdown, buildup)
Overall Mood	General emotional tone
Energy Curve	How energy changes across the track (building, steady, dynamic, fading)

Audio Sections

Each detected section includes:

Field	Description
start / end	Timestamp in seconds
type	Section type (intro, verse, chorus, bridge, outro, drop, breakdown, buildup)
mood	Emotional quality of this section
energy	Intensity level (0-100)
visualSuggestion	Suggested visual treatment

Phase 2: Lyrics Extraction

Lyrics are the primary content driver for visual generation.

What Gets Extracted

Field	Description
Full Lyrics	Complete transcription of the song
Is Instrumental	Whether the track has no vocals
Language	Detected language
Themes	Overall themes identified in the lyrics
Section Lyrics	Per-section lyrics with key imagery and emotional arc

Per-Section Lyrics Data

For each audio section, the system extracts:

• Lyrics: Transcribed text for that section
• Key Imagery: Visual concepts from the lyrics (e.g., "burning bridges", "walking alone")
• Emotional Arc: The emotional interpretation of that section's lyrics

Phase 3: Story Interpretation

This phase creates a holistic understanding of the entire song's narrative before generating any visuals.

Story Interpretation Output

Field	Description
Song Type	narrative, emotional_journey, abstract, performance, or concept
Core Story	Central narrative or message of the song
Protagonist	Description, emotional journey, and visual representation
Narrative Arc	Setup, conflict, and resolution
Visual World	Primary locations, symbolism, and color meaning
Mood Progression	How mood changes across phases with visual tone

Song Types

Type	Description
narrative	Song tells a clear story with characters and events
emotional_journey	Song expresses emotional transformation
abstract	Abstract or poetic content
performance	Performance-focused music video
concept	Concept-driven visual piece

Phase 4: Visual Bible

Informed by the story interpretation, this phase creates a comprehensive visual style guide.

Visual Bible Output

Field	Description
Visual Style	Overall style, color palette, lighting mood, motion style
Recurring Motifs	Visual elements that should repeat throughout
Transition Style	How to handle visual transitions between sections
Performer Directions	Guidance for performer/character depiction (optional)
Forbidden Elements	What to avoid in the visual treatment

Phase 5: Scene Generation

The final phase generates scenes and shots driven by the story interpretation and lyrics, aligned to the audio timeline.

Shot Timing Fields

Music video shots include additional timing fields not present in standard storyboards:

Field	Description
startTime	Absolute position in the timeline (seconds)
audioSectionId	Links the shot to its audio section (e.g., "section_0")
beatAligned	Whether the cut falls on a beat
timingRationale	Why this timing was chosen (e.g., "cuts on chorus drop")

Scene Enhancement

After generation, each shot's prompt is enhanced with the visual bible's style, color palette, and lighting mood to ensure consistency.

UI Features

Audio Timeline

The Audio Timeline component provides a visual representation of the analyzed audio track:

• Color-coded sections (verse = blue, chorus = purple, bridge = green, drop = red, etc.)
• Shot markers overlaid on the timeline showing where each shot falls
• Time markers at regular intervals
• Energy visualization per section

Music Video Analysis Panel

After generation, the Analysis Panel displays the complete analysis results:

• Audio analysis details (BPM, key, genre, sections)
• Lyrics extraction results
• Story interpretation
• Visual bible

Shot Timing Information

In the Shot Details panel, music video shots display:

• Start time position (e.g., "1:15")
• Audio section link (e.g., "chorus")
• Beat alignment indicator
• Timing rationale text

MusicVideoAnalysis Storage

The complete analysis data is stored on the project as musicVideoAnalysis:

musicVideoAnalysis: {
  audioAnalysis    // Full audio analysis with enriched sections
  lyricsExtraction // Lyrics, themes, key imagery
  storyInterpretation // Narrative, protagonist, visual world
  visualBible      // Style guide, motifs, forbidden elements
}

This data is preserved for reference and can be viewed in the Music Video Analysis Panel.

Workflow for Music Videos

Step 1: Prepare

1. Set Goal to Music Video
2. Upload your audio track
3. Add a visual theme description (optional but recommended)
4. Upload reference images for character or style guidance

Step 2: Generate

1. Click Generate Storyboard
2. The 5-phase pipeline runs (takes 1-3 minutes)
3. Review the generated shots in the timeline

Step 3: Review and Refine

1. Check the Audio Timeline for beat alignment
2. Review the Story Interpretation in the Analysis Panel
3. Verify shot timing matches musical sections
4. Regenerate individual shots that don't match your vision

Step 4: Generate Videos

1. Approve hero frames for each shot
2. Generate videos (shots inherit timing from the pipeline)
3. Review sync quality with the audio

Step 5: Assemble

1. Send to Movie Editor for final composition
2. The Movie Editor receives all shots with their timing data
3. Fine-tune the final output

Tips for Great Music Videos

1. Let Lyrics Drive

The lyrics are the primary content driver. The system extracts key imagery from lyrics and translates them into visual concepts. Write a visual theme that complements rather than contradicts the lyrics.

2. Create Section Contrast

Make different sections visually distinct:

• Verse: storytelling, character-focused
• Chorus: high energy, visual payoff
• Bridge: contrast, intimate moments

3. Use Reference Images

Upload reference images that establish the visual world you want. The Visual Bible phase uses these to create a consistent style guide.

4. Repeat Visual Motifs

The Visual Bible defines recurring motifs. These create visual continuity and recognition throughout the video.

5. Trust the Timing

The scene generator calculates shot timing with full audio context. The timingRationale field explains why each cut was placed where it is.

Troubleshooting

Shots Not Aligning to Music

• Verify the audio analysis detected sections correctly (check the Analysis Panel)
• The timing is calculated by the scene generator with full context
• If sections are wrong, try a different audio file format

Visuals Don't Match Song Mood

• Add a more descriptive Visual Theme text
• Upload reference images that match your vision
• Review the Story Interpretation -- if it misread the song, regenerate

Instrumental Track

• For instrumental tracks, isInstrumental is set to true
• The system relies more on audio analysis (mood, energy, genre) rather than lyrics
• Provide a detailed Visual Theme to compensate for the lack of lyrical content