Why “Pokemon diffusion” worked (LambdaLabs, 2022) — and what we fixed in our SDXL + RunPod + ComfyUI tests

This page documents: (1) what LambdaLabs / Justin Pinkney did differently, (2) why “known token” prompts like pokemon behave “magically,” (3) what went wrong in our SDXL LoRA test harness, and (4) the concrete fixes + evidence that the LoRA is now actually applied.
SDXL base 1.0
RunPod (ComfyUI)
LoRA format conversion (Diffusers → Comfy/Kohya)
Evidence: preflight + 960-image sweep
Big picture
LambdaLabs got strong results by shipping a fine-tuned checkpoint (not a LoRA) and sampling from EMA / averaged weights to preserve base knowledge while biasing toward Pokémon-like renders.
Our failure mode
We initially had an SDXL LoRA in Diffusers key format that ComfyUI’s LoraLoader does not load, so a scale sweep can become meaningless if the LoRA is silently ignored.
Where we are now
We added an automatic conversion step and preflight checks. Result: LoRA scale changes output, and the full SDXL sweep is valid.

1) What LambdaLabs did differently (and why it works)

Checkpoint fine-tune, not an adapter

The LambdaLabs “text-to-pokemon” release was primarily a full Stable Diffusion fine-tune (SD 1.4 era). That means the entire denoising model learns a Pokémon-biased image prior.

  • Effect: you can type ordinary prompts (even just a name) and the checkpoint tends to render in a Pokémon-like style.
  • Contrast: a LoRA is a small delta applied on top of a base model; it’s easier to ignore or misapply if your loader expects a different format.

EMA / weight averaging to preserve “world knowledge”

A key trick described in the blog is sampling from EMA weights (think “smoothed average”). This tends to reduce overfit artifacts and keeps the base model’s generality while injecting Pokémon-ness.

References: Justin Pinkney’s write-up and the LambdaLabs diffusers repo: blog, repo.

Concept diagram: checkpoint fine-tune vs LoRA

2) Why the token pokemon “just works”

Known-token prior

Modern base models often already associate the word pokemon with a strong visual manifold. That means a lot of “pokemonization” can come from the base model without any LoRA.

  • Great for quick results.
  • Bad for “adapter-purity” tests: you can think your LoRA is working when the base model is doing most of the work.

Neologisms are the opposite problem

A made-up token like pk_mn usually has no meaning in the text encoder. UNet-only LoRAs can’t reliably “invent meaning” for a token the encoder treats as near-noise.

If your goal is “LoRA-derived latent space,” the honest test is: does pk_mn do nothing when adapters are off, and do something only when adapters are on?

3) The SDXL harness bug we hit (ComfyUI silently ignoring LoRA)

Root cause

Fix: we added automatic conversion + a preflight sanity run that proves scale affects output.

4) Evidence: preflight proving LoRA scale changes output

Same prompt + seed, only scale changes

Prompt: toothbrush pokemon • Seed: 12345 • SDXL: steps=30 cfg=7.0 1024×1024

5) Evidence: full SDXL sweep (960 images)

Full gallery

This is the corrected sweep (40 prompts × 6 scales × 4 images).

Open the 960-image gallery

Note: the older gallery under pig/sdxl-sweep/dcf3d10403dd/ is not used for conclusions.

Selected examples (from the 960)

These captions are prompt strings (not an attempt to identify anyone in the images).

6) The operational “never burn money again” checklist

Safety gate: no runaway pods

source .api-keys\n./.venv/bin/python - <<'PY'\nimport os, asyncio\nfrom app.providers.runpod import RunPodClient\nasync def main():\n  c=RunPodClient(api_key=os.environ[\"RUNPOD_API_KEY\"])\n  pods=await c.list_my_pods()\n  running=[p for p in pods if str((p or {}).get(\"desiredStatus\") or \"\").upper()==\"RUNNING\"]\n  print(\"pods_total\",len(pods),\"pods_running\",len(running))\nasyncio.run(main())\nPY

Run the SDXL sweep (skip training by default)

source .api-keys\n./.venv/bin/python scripts/execute_sdxltest.py \\\n  --volume-id qqx2kgc5v7 \\\n  --ckpt-name sd_xl_base_1.0.safetensors \\\n  --output-name pokemon-sdxl-lora-en-v1 \\\n  --sweep-steps 30 \\\n  --sweep-cfg 7.0 \\\n  --gpu-type-id \"NVIDIA H200\" \\\n  --container-disk-gb 160 \\\n  --skip-train

Internal docs / exact runbook: see restart_plan.md in the repo.

Built for analysis.godwhy.wtf. External references: Justin Pinkney blog, LambdaLabs repo.