Why “Pokemon diffusion” worked (LambdaLabs, 2022) — and what we fixed in our SDXL + HiDream + RunPod + ComfyUI tests

Checkpoint fine-tune, not an adapter

The LambdaLabs “text-to-pokemon” release was primarily a full Stable Diffusion fine-tune (SD 1.4 era). That means the entire denoising model learns a Pokémon-biased image prior.

• Effect: you can type ordinary prompts (even just a name) and the checkpoint tends to render in a Pokémon-like style.
• Contrast: a LoRA is a small delta applied on top of a base model; it’s easier to ignore or misapply if your loader expects a different format.

EMA / weight averaging to preserve “world knowledge”

A key trick described in the blog is sampling from EMA weights (think “smoothed average”). This tends to reduce overfit artifacts and keeps the base model’s generality while injecting Pokémon-ness.

References: Justin Pinkney’s write-up and the LambdaLabs diffusers repo: blog, repo.

Known-token prior

Modern base models often already associate the word pokemon with a strong visual manifold. That means a lot of “pokemonization” can come from the base model without any LoRA.

• Great for quick results.
• Bad for “adapter-purity” tests: you can think your LoRA is working when the base model is doing most of the work.

Neologisms are the opposite problem

A made-up token like pk_mn usually has no meaning in the text encoder. UNet-only LoRAs can’t reliably “invent meaning” for a token the encoder treats as near-noise.

If your goal is “LoRA-derived latent space,” the honest test is: does pk_mn do nothing when adapters are off, and do something only when adapters are on?

Full gallery

This is the corrected sweep (40 prompts × 6 scales × 4 images).

Open the 960-image gallery

Note: the older gallery under pig/sdxl-sweep/dcf3d10403dd/ is not used for conclusions.

Selected examples (from the 960)

These captions are prompt strings (not an attempt to identify anyone in the images).

What happened

ComfyUI validates workflows by checking “dropdown” model lists that are built from files found under models/. If a workflow references a file that doesn’t exist on the pod yet, the server rejects the request before it even starts sampling.

Typical symptom: HTTP 400 with prompt_outputs_failed_validation and “value not in list” errors for UNet / VAE / text encoders.

value_not_in_list: unet_name 'hidream_i1_dev_fp8.safetensors' not in []
value_not_in_list: vae_name 'ae.safetensors' not in ['pixel_space']
value_not_in_list: clip_name{1..4} ... not in []

Why it kept “starting fresh”

On RunPod, the container disk is ephemeral, but a Network Volume is persistent. If you don’t mount a Network Volume, your pod starts with empty model folders every time, so you end up re-downloading base models and encoders repeatedly.

Fix: mount a Network Volume and place files into the right ComfyUI model paths (example layout below).

/workspace/models/diffusion_models/hidream_i1_dev_fp8.safetensors
/workspace/models/vae/ae.safetensors
/workspace/models/text_encoders/clip_l_hidream.safetensors
/workspace/models/text_encoders/clip_g_hidream.safetensors
/workspace/models/text_encoders/t5xxl_fp8_e4m3fn_scaled.safetensors
/workspace/models/text_encoders/llama_3.1_8b_instruct_fp8_scaled.safetensors

The causal story (short version)

• Known token prior: the base model already has a strong association for pokemon, so prompts like … pokemon look “good” even with no adapters.
• Neologism problem: pk_mn is near-noise to the text encoders, so a UNet-only LoRA can apply a vague style shift, but it’s an unreliable “handle” for composition.
• What finally worked: train a text-encoder bridge across all 4 encoders (CLIP L + CLIP G + T5 + Llama) so pk_mn behaves like pokemon in the conditioning stack (with low leakage in deterministic tests).

Full experiment log: see lora_test.md in the repo.

Why is Llama even involved?

HiDream uses a multi-encoder conditioning setup. The official workflows load multiple encoders because different encoders contribute different aspects of the prompt embedding (style vs detailed description vs high-level semantics). If you only “bridge” one encoder, the UNet still receives mixed signals.

Practical takeaway: if you’re trying to make a made-up token meaningful on HiDream, bridging only one encoder is rarely enough.

Think in “concept cards”

For your own artwork, you’ll get better leverage by building many small, well-defined concepts (e.g. concrete_home, op_art_drawing, rainy_neon_city) instead of one mega-token that tries to do everything.

• Dataset: 50–300 images per concept, diverse views, consistent aesthetic target.
• Captions: <trigger>, <class noun>, <descriptors> (the class noun anchors the concept).
• Eval: always include “adapters OFF” baselines + fixed seeds so you can see true deltas.

Neologisms: what actually works

If you insist on made-up tokens, you’re asking the system to (a) teach the text encoders a new “word”, and (b) teach the image model what that word implies. UNet-only LoRA training often solves (b) weakly and not (a).

• Easiest: use a rare real phrase + class noun (e.g. op-art-concrete house) instead of a pure neologism.
• Stronger: train text-conditioning (all encoders) or a soft-prompt embedding for the trigger.
• Gold standard: full checkpoint fine-tune (more compute, more risk of overfit/forgetting).

What actually dominates wall-clock time

• Cold start: starting a pod + launching ComfyUI + loading weights into VRAM. HiDream is especially heavy because it loads multiple text encoders.
• Downloads (if not on a Network Volume): fetching base models/encoders again can dwarf inference time.
• Throughput: your total time is roughly images × seconds_per_image plus uploads; “4-up” generation is faster than 1-up because it amortizes model overhead.
• Uploads: pushing hundreds of PNGs (and building galleries) can add noticeable tail latency.

Concrete numbers (from our runs)

These are “warm” (models already present) observed timings — not theoretical best cases:

• HiDream (1024², steps=28, cfg=1.0): ~6–10 seconds per image in our A/B runs once loaded.
• SDXL sweep (1024², steps=30, cfg=7.0): 960 images finished end-to-end in ~52 minutes on H200-class hardware including uploads and orchestration.

If you see “12 images taking 30+ minutes”, it’s almost always cold-start + downloads + waiting on a stuck workflow, not raw sampling speed.