Generation Pipeline

Understanding how DUTO's AI system creates your storyboard through a sequential orchestrator pipeline.

Pipeline Overview

When you generate a storyboard, your concept flows through up to 11 specialized AI workers organized into 4 phases. The orchestrator calls each worker directly in sequence (not via agentic LLM tool-calling), with each worker having its own error handling and sensible fallbacks.

Your Concept + Settings
     |
+----------------------------+
|   PHASE 1: PRE-PRODUCTION  |
|   (conditional workers)    |
+----------------------------+
| 0. Character Reference *   |
| 1. Research *              |
| 2. Reference Digest *      |
+-------------+--------------+
              |
+----------------------------+
|   PHASE 2: STORYBOARD CORE |
|   (required workers)       |
+----------------------------+
| 3. Story Bible             |
| 4. Storyboard Plan         |
| 5. Consistency Validator   |
+-------------+--------------+
              |
+----------------------------+
|  PHASE 3: POST-PRODUCTION  |
|  (required workers)        |
+----------------------------+
| 6. Cinematographer Director|
| 7. Lighting Director       |
| 8. Sound Design            |
| 9. Music Supervisor        |
+-------------+--------------+
              |
+----------------------------+
|   PHASE 4: FINALIZATION    |
+----------------------------+
| 10. Output Storyboard      |
+----------------------------+
              |
       Complete Storyboard

* = Only runs when conditions are met (see below)

LLM Model Selection

The pipeline uses the LLM model selected in your project settings:

Setting	Model Used
Gemini 2.5 Flash	google/gemini-2.5-flash (default)
DeepSeek Chat	deepseek/deepseek-chat
Gemini 2.5 Pro	google/gemini-2.5-pro

The orchestrator also resolves the "auto" style preset based on your goal:

Goal	Default Style	Default Energy	Default Realism
Brand Ad	Commercial	70	60
Short Film	Cinematic	40	70
TikTok/Reel	Ultra Clean	90	50
Explainer	Commercial	50	40
Product Demo	Ultra Clean	60	80

Phase 1: Pre-Production (Conditional)

These workers only run when their conditions are met. If they fail, the pipeline continues without their output.

Worker 0: Character Reference

Triggers when: Your concept contains "looks like X" patterns (e.g., "a character who looks like Beyonce")

What it does:

• Searches for reference portrait photos via Tavily image search
• Extracts the lookalike reference name from your concept text
• Returns reference images and a character description

For each match found:

• Searches: "{lookalike reference} face portrait photo"
• Returns up to 5 reference images

Worker 1: Research

Triggers when: Research is enabled in settings OR your concept contains keywords like "realtime", "current events", "latest", "trending", "news", "today"

What it does:

• Searches the web via Tavily for factual information related to your concept
• Returns facts, source articles, and images
• Supports depth levels: basic (5 results) or advanced (8 results)

Worker 2: Reference Digest

Triggers when: You have uploaded reference images

What it does:

• Analyzes all reference images using multimodal LLM vision
• Extracts structured data from the images

Output (ReferenceDigest):

Field	Description
entities	Array of detected subjects with type (product/character/location), name, and locked traits
style	Extracted palette colors, textures, and observed lighting
risks	Potential consistency issues identified

Phase 2: Storyboard Core (Required)

These are the essential workers that produce the storyboard structure. The Storyboard Plan worker will throw an error if it fails (halting the pipeline).

Worker 3: Story Bible

What it does:

• Creates the narrative and visual framework for the entire storyboard
• Uses your concept, goal, resolved style, reference digest, and research results

Output (StoryBible):

Field	Description
hero_subject	Type (product/character/environment), locked description, and forbidden changes
style_bible	Overall style, lighting design (key, fill, color temp, mood), camera language (energy, rules), effects rules
negative_bible	Things to avoid in generation
continuity_anchors	Elements that must remain consistent across shots

Worker 4: Storyboard Plan

What it does:

• Breaks the narrative into scenes and shots
• Uses all accumulated context: concept, goal, format, duration, style, energy, realism, reference analysis, story bible, cinematography settings, and research results

Output:

• Array of scenes, each containing shots with descriptions, durations, and high-level prompts

Worker 5: Consistency Validator

What it does:

• Reviews all scenes against the story bible
• Checks character consistency, temporal logic, lighting continuity
• Applies fixes to scenes if inconsistencies are found

Output:

• Fixed scenes (if corrections were needed) or validated original scenes

Phase 3: Post-Production Enhancement (Required)

These workers enrich each shot with professional production details. If any fail, the pipeline continues with whatever enhancements were applied.

Worker 6: Cinematographer Director

What it does:

• Designs professional camera work for each shot
• Considers the story bible, resolved style, and project goal

Adds to each shot:

Field	Description
camera_angle	Viewing angle for the shot
camera_motion	Movement type and direction
lens_style	Lens characteristics
framing	Shot composition
motion_intensity	How much movement
vibe_tags	Mood/aesthetic tags
transition_hints	Transition in/out suggestions

Worker 7: Lighting Director

What it does:

• Designs lighting for each shot
• Ensures consistency with time of day and story bible

Adds to each shot:

• Detailed lighting description integrated into cinematography

Worker 8: Sound Design

What it does:

• Plans audio elements for each shot
• Considers the story bible and project goal

Adds to each shot:

• Sound effect descriptions, ambient audio notes

Worker 9: Music Supervisor

What it does:

• Designs musical arc across the storyboard
• Considers duration, goal, and story bible

Adds to each shot:

• Musical direction and timing notes

Phase 4: Finalization

Worker 10: Output Storyboard

What it does:

• Compiles all worker outputs into the final storyboard document
• Enhances shot prompts using the enhanceStoryboardPrompts function
• Produces the final scenes with all production details merged

If this worker fails: The pipeline falls back to using the scenes as they were after Phase 3, along with the story bible.

Final output includes:

• Complete scenes with enriched shots
• Story bible
• Worker log (which workers ran, their status, timestamps)
• Research results (if research was performed)

After the Pipeline

Once the orchestrator returns the storyboard:

1. Scenes and shots are loaded into the storyboard store
2. Story Bible data is stored on the project for future reference
3. Subject Registry is initialized (populated during hero frame generation)
4. Shot Generation Context is extracted and stored for adding shots later
5. If Auto-Generate Images is set to "on_create", hero frame generation begins automatically for each shot

Hero Frame Generation (Post-Pipeline)

Hero frame generation happens client-side after the pipeline completes:

1. Each shot gets its hero frame generated using the assigned image model
2. After a hero frame is generated, the system detects subjects in the image
3. First appearances of subjects are captured in the Subject Registry with frozen traits
4. Subsequent shots reference the registry for visual consistency

Error Handling

The pipeline is designed to be resilient:

Worker	On Failure
Character Reference	Skipped, no character references available
Research	Skipped, no research data available
Reference Digest	Skipped, no reference analysis available
Story Bible	Continues without story bible (worker has internal fallback)
Storyboard Plan	Pipeline halts -- this is the only required-or-fail worker
Consistency Validator	Continues with unvalidated scenes
Cinematographer	Continues without cinematography enhancement
Lighting	Continues without lighting enhancement
Sound	Continues without sound design
Music	Continues without music direction
Output Storyboard	Uses current scenes as fallback

Monitoring Progress

During generation, the UI shows:

• A progress modal with the current worker being processed
• Worker log entries with status (started, completed, failed) and timestamps
• The active worker index indicating which step is currently running
• The generation phase name

Music Video Pipeline

Music video storyboards use a separate orchestrator (generate-music-video-orchestrated) with specialized workers that include:

• Audio analysis and section detection
• Lyrics extraction and interpretation
• Visual bible creation based on song content
• Beat-aligned shot timing

See Music Video Mode for details on the music video pipeline.