Diversifying AI Generated Characters
Before generating multiple characters, write explicit differentiation specs for each one covering: face shape, age marker, ethnicity/skin tone, distinguishing feature, and one asymmetry or "flaw." Never rely on the model's default interpretation of vague prompts like "a woman" or "a man" repeated across characters — it will regress to the same mean face.
Bad (causes convergence):
Character A: a young woman, brown hair
Character B: a young woman, blonde hair
Character C: a young woman, red hair
Good (forces divergence):
Character A: heart-shaped face, late 20s, olive skin, thick eyebrows,
slight gap in front teeth, hair in a messy low bun
Character B: square jaw, mid-40s, deep brown skin, high forehead,
small scar above left eyebrow, buzzcut
Character C: round face, teens, pale freckled skin, snub nose,
asymmetric undercut, ears slightly protruding
Diffusion and autoregressive image models drift toward a "modal" output — the statistically most common face/pose/composition in their training distribution — whenever prompts are underspecified or repeated in a session. This shows up as:
- Character convergence: multiple distinct characters end up with near-identical faces, only differing in hair color or clothing.
- Session drift: later images in a conversation look more like earlier generated images than like the original prompt intent (model anchors on its own prior outputs, not just the training prior).
- Cardinality collapse: past ~3-4 characters or complex multi-subject scenes, the model loses track of who's who, merges features, or drops characters entirely.
Progress:
- [ ] Step 1: Define a character sheet per subject (not just adjectives in a prompt)
- [ ] Step 2: Force feature divergence across the whole cast
- [ ] Step 3: Generate characters in isolation, not in one crowded prompt
- [ ] Step 4: Cross-check the batch for convergence
- [ ] Step 5: Reset context between unrelated generations
Step 1: Character sheet, not adjective list
For each character, specify independently:
- Bone structure: face shape (oval/square/heart/round/long), jaw width, cheekbone prominence
- Age + era markers: specific age or decade, not "young"/"old"
- Skin/ethnicity: tone, undertone, texture (freckles, weathering, smoothness)
- One signature feature: scar, gap tooth, heterochromia, birthmark, unusual eyebrow shape, prominent nose
- Asymmetry: real faces are asymmetric; explicitly request one (uneven smile, one ear higher, crooked nose) — this alone breaks the "AI-generated symmetric face" look
- Hair as last differentiator only: never let hair color/style be the only distinguishing trait between characters
Step 2: Force divergence across the cast
Before generating, list all characters side by side and check that no two share more than one trait category (face shape, age bracket, skin tone, feature, hair). If two characters both got "oval face, late 20s," change one deliberately. Treat it like assigning traits from a deck you can't reuse.
Step 3: Generate in isolation when possible
If the tool/workflow allows it, generate each character alone with their full character sheet, rather than "three friends standing together" in one prompt. Multi-subject prompts are where models most often blend faces or lose count. Composite afterward (layout, collage, or separate panels) if a group shot is needed.
If a single group image is unavoidable:
- Cap at 3-4 distinctly described subjects per image
- Explicitly assign position: "leftmost," "center," "rightmost" tied to each character sheet
- Repeat each character's signature feature in the same prompt near their positional reference
Step 4: Cross-check the batch
After generating a set, compare them like a lineup: do any two look like the same person with different clothes? If yes, that's convergence — go back to Step 2 and push the diverging trait harder (don't just tweak hair color again, since that's the trait that was already varied and failed).
Step 5: Reset context between unrelated generations
Session drift happens because the model treats recent outputs as informal style anchors. When starting a new, unrelated character or scene:
- Start a new conversation/context rather than continuing a long thread of image generations
- Re-state the full character sheet explicitly rather than saying "make another one like before but different"
- Avoid "more of the same but change X" phrasing — it invites the model to keep everything else identical, including the face
Example 1: Fixing convergence in a 4-character cast
Input (produces convergent faces):
1. A wizard, old, wise
2. A knight, young, brave
3. A rogue, sneaky
4. A healer, kind
Output (differentiated):
1. Wizard: long face, 70s, weathered pale skin, bushy white eyebrows
overtaking deep-set eyes, hooked nose, long grey beard braided once
2. Knight: square jaw, early 20s, tan skin with sunburn on nose,
broken-then-healed nose bridge, buzzed dark hair, one front tooth
slightly chipped
3. Rogue: narrow angular face, 30s, olive skin, thin scar through
right eyebrow, asymmetric smirk, shaved sides with long top hair
4. Healer: round soft face, teens, dark brown skin, round cheeks,
heterochromia (one brown one green eye), tight coily hair in twists
Example 2: Breaking session drift
Input (drift-inducing): "Generate another character, similar style to the last one but a woman this time."
Better: Start fresh, restate full visual spec including face shape, age, skin, signature feature, and asymmetry, without referencing "the last one" as a style anchor.
- Always include at least one asymmetric or "imperfect" trait per character — perfect symmetry is a tell of unmoderated drift toward the model's mean face.
- Vary face geometry first, hair/clothing last — geometry is what actually reads as "different person"; surface styling is not.
- Keep a running character bible (short text block) across a project and paste the relevant character's full spec every time, rather than trusting the model's memory of "the redhead from before."
- For scenes with more than 4 characters, generate in sub-groups or individually and composite, rather than trusting single-shot generation to track everyone.
- If using an iterative chat-based tool, periodically restart the conversation for unrelated characters to avoid latent style bleed from prior turns.
- Relying on hair/clothing alone to differentiate: this is the single most common cause of "same face different wig" syndrome.
- Vague age/ethnicity terms ("young," "exotic," "average"): these collapse to the model's most common training example. Use specific ages and specific descriptors.
- "Same as before but X" prompting: this anchors everything else to the previous output, including facial structure, compounding drift over a session.
- Cramming 5+ named characters into one prompt: expect blending, feature swapping, or dropped characters. Split the generation.
- Not checking output against the rest of the batch: convergence is often invisible looking at one image at a time; always compare the full set side by side.