The workflows built for Star Rifter made this director-first approach work in practice. Every aspect of every shot was driven by human performance input: 3d playblasts for camera and composition, trained AI models for character designs and sets, video recordings that communicated intent visually, directly, and precisely. What would a prompt even be needed for?

Now anyone with some props and an phone camera has the potential to create animated magic.

Performance, Not Prompts

The biggest technical investment in Star Rifter was building workflows that could take raw, imperfect human performance and turn it into polished AI animation while keeping the director’s intent intact at every level.

Colin and his team recorded every aspect they wanted to control. Body movements, facial expressions, eye direction, camera paths. They used whatever was practical: iPhone video for performances, Maya renders for camera and spatial composition. The performers weren’t motion-capture actors in expensive rigs.

They were Colin himself, his friend, his daughter. Sometimes they were acting in the rain. Sometimes the framing was too tight. Sometimes the eye lines were off because someone was reading from a script instead of looking where the other character would be. The tools were everyday and the conditions were imperfect. But the intent was cinematic, and that’s what the pipeline had to capture.

The workflows had to be robust enough to understand inputs that weren't perfect, and still grab the essence of the performance. Not just broad body movement, but the subtlety of a facial expression, the direction of a glance, the timing of a breath between lines.

Colin’s way of thinking about it:
AI is a very high-end rendering process. It handles the clothing, the hair, the simulation, the dust, the effects, the explosions. The grit and the rust and the environmental detail. Those are things AI does exceptionally well. What it doesn’t do well on its own is performance: the eyes, the emotion, the subtle human things that make a character feel alive. That’s what the humans bring. The AI renders. The humans direct.

Prompts optional. Everything is already controlled and directed by the director.

By the end, prompts were barely used across the entire production. Everything was already predesigned: the art style, the composition, the character training data, the models, the body and face and eye performances, the 3D environments, the camera moves. 

There was no room left for a prompt. And there didn’t need to be. It was all the artist and the artist’s intentions.

Open source workflows allow you to create new tools if a solution doesn't exist.

Body Performance

Actor recordings served as the body performance input for each shot. The key problem was that the performer’s physical proportions don’t match the animated character’s. A naive motion transfer would distort the character, conforming their body to the actor’s build instead of just following the movement. The team developed a custom bone re-proportioning workflow that extracted the performance intent (the gesture, the movement, the timing) without imposing the actor’s physical structure onto the character.

Multi million dollar special effects rigs vs. AI and an old TV antenna.

For shots where a complete body performance wasn’t feasible, the team used selective control to isolate the useful parts and discard the rest. Take the shot of Dex hanging off the ship as it flies away. Nobody had expensive stuntmen or rigs to hoist a person into the air. So Colin acted out what he could, the team grabbed the upper body and facial expression, and left the legs behind. The AI filled in the rest within the established character model. That’s the magic of granular workflow control: you don’t need a perfect capture if you can surgically extract the parts that matter.

And then there was the puppet. For some motions that humans simply couldn’t perform as reference (Mooch isn’t exactly human-shaped), Colin built a little puppet and acted out scenes with it. To everyone’s surprise, the Floyo team was able to track the puppet’s motion and translate that silly little puppeteering performance. It’s the kind of moment that reminds you this is still filmmaking, not software engineering.

Capturing the most subtle aspects of human performances is what adds the soul.

Facial Performance and Eye Line

For Colin, the eyes were everything. Eye lines determine whether characters feel present in a scene, whether they’re actually talking to each other or just performing in parallel. Getting eye lines wrong is one of the most visible failures in screen performance, and it was the single highest priority on Star Rifter.