Google tests AI agents on short film-making project

Google has detailed an internal experiment in which teams of AI agents produced short films. The project tested whether agents could collaborate outside software work.

The exercise involved 10 crews, each made up of three agents with distinct roles, working through a filmmaking process inside Scion, Google's open-source multi-agent orchestration testbed. Across pilot rounds and the competition, the project produced more than 25 films totaling about 44 minutes, according to Google.

Each crew included an Idea Person, a Technical Lead, and an Editor. A Team Coach agent supervised checkpoints without writing or directing, while a Coordinator agent scheduled the contest in five waves, with two teams running at a time over about 21 hours.

The crews followed a seven-step production process based on traditional filmmaking: concept, beat sheet, character workshop, storyboard, principal photography, assembly, and final render. At each stage, one agent had to check another's work for issues such as timing or resolution.

That control system was added after problems in earlier trials. In one pilot, a team reported that it had finished a film, but the output was a 94-byte placeholder file rather than a completed work.

How Teams Worked

The agents collaborated through messages and shared files rather than direct human management. The Idea Person generated script ideas and defined visual style, the Technical Lead used the media tools, and the Editor handled pacing and final assembly.

The experiment also showed agents making independent editorial decisions by reading shared project files. In one case, an Editor built an eight-second silence around a line of prose and marked it "NON-NEGOTIABLE" in the timeline, while the Technical Lead repeatedly regenerated a shot until a flower separated from a bouquet at the right frame.

Human reviewers fed comments from pilot rounds back into the process. Their feedback covered issues including overlapping audio, character inconsistency, and hard-to-follow stories. Google used it alongside agent-written retrospectives to revise both the production guides and the software toolchain.

That toolchain combined command-line tools written in Go with Python batch automation. The agents used it to call a range of Google AI models for still images, video, music, and speech generation.

Models In Use

For image generation, teams used Gemini image generation, known as Nano Banana, to create character reference sheets, storyboard frames, and scene compositions. To keep characters visually consistent, agents generated headshots first, then body sheets, then scene tests, carrying those references through later prompts.

Video clips were generated with Veo 3.1, typically in segments of four to eight seconds at 720p. The agents switched between text-to-video, image-to-video, and frame interpolation depending on the shot. For longer sequences, they linked clips by feeding the last frame of one into the next.

Veo 3.1 also generated ambient sound and lip-synced dialogue within clips. One team structured its script like a musical score because the timing between generated speech and mouth movement made pauses more effective, Google said.

Lyria 3 was used to generate original music, while Gemini Flash TTS created voices and narration from named personas. Runtime control proved difficult: one team's narrator spoke at 108 words per minute instead of the planned 130, extending the film by a full minute.

A four-minute film typically required more than 40 image generations, more than 25 video clips, several music stems, around a dozen voice recordings, and hundreds of assembly operations, according to Google.

Operational Lessons

The project's strongest lesson, Google said, was that agents worked better through files than through message history. Decisions written into shared documents survived crashes and restarts, while information held only in message threads was often lost.

Scion lets agents run in containerized sandboxes, message one another, spawn additional agents, and work from a shared filesystem. That meant one agent could resume another's work after a failure. In one case, after an Editor crashed during final assembly, the Technical Lead read the Editor's timeline plan and completed the job.

Style choices also mattered, Google said. Teams that chose claymation or silhouette animation were better able to work around weaknesses in generative video, including temporal drift and facial consistency problems. In another case, a safety filter blocked a kiss, so the team replaced it with the shadows of two figures merging on a wall.

Prompt detail also changed the results. General instructions produced conventional output, while tighter constraints on colors, instruments, and unwanted visual effects led to more distinct films.

One supervising agent offered Google's summary of the challenge: "It's a room full of specialists who can each do one thing at superhuman speed, but none of them can taste the soup," the coach agent said.