I Published My AI Architecture. Have fun with it.

I trained three language models from scratch. Not fine-tuned — from random weights to coherent text. The architecture is called Chimera, the models are called Auron, and as of today, everything is public.

Code · Paper (PDF) · Models on HuggingFace

What Chimera Actually Is

Every transformer has the same problem: deeper = smarter but also = more parameters = more compute = more money. The standard solution is to just throw more hardware at it. That works if you're OpenAI. It doesn't work if you're one person with a DGX Spark on your desk and rented cloud GPUs.

Chimera splits the network into two zones:

• Bottom ("Retina"): Unique layers that parse raw tokens into semantic representations. Like the retina preprocessing before your visual cortex does the actual thinking.
• Top ("Brain"): A small set of shared physical blocks that loop multiple times. Same weights, repeated. Weight sharing as learned recurrence.

The trick: Gated Delta Networks (GDN) handle the recurrent processing — their hidden state naturally evolves across loops, so each pass through the same weights operates on genuinely different input. Standard attention degenerates when you loop it. GDN doesn't. We interleave GDN and attention at a 3:1 ratio.

Result: a 510M parameter model with ~787M virtual parameters. The shared top blocks are only 35MB of bf16 weights — small enough to fit entirely in the H100's 50MB L2 cache. Loops 2 and 3 execute with near-zero memory latency. 50,000 tokens per second on a single H100.

Three Models, Three GPUs

All three are training right now. The 510M is at 103K steps (~3.4 billion tokens seen), val loss 3.16 and still dropping. The 279M runs on my DGX Spark at home. The 510M on a rented H100. The 1.1B on a B200.

What The Models Can Do (And Can't)

These are pretrained base models — no instruction tuning, no RLHF. They're trained on 75% FineWeb-Edu, 18% code, 5% math, and 2% UltraChat dialogue.

That 2% dialogue is doing something unexpected. All three models spontaneously generate ChatML stop tokens (<|im_end|>) and empty <think></think> blocks in assistant-role completions. The 510M correctly parses system/user/assistant roles and retrieves specific information planted in system prompts. At 510M parameters. With 2% dialogue data.

They can't tell jokes. When asked for a joke about a programmer who deleted production, the 510M responded: "Sure! Here's what I'd like to tell you: Hello, programmer! I'm here to talk to you today. I hope that this is helpful!" — Technically not wrong. Not funny either.

Why I'm Publishing This

I grew up watching someone share knowledge freely — hundreds of educational resources, all Creative Commons, used by thousands of people who never knew who made them. No paywalls, no gatekeeping. Just "this is useful, take it."

The paper, the code, the model weights — everything is CC BY 4.0 / Apache 2.0. If it's useful to you, use it. That's the whole point.

What's Next: Chimera-MoE

The paper includes a future work section that I'm genuinely excited about. NVIDIA just published Nemotron-Cascade 2 — a 30B MoE with 3B active parameters that gold-medaled at IMO 2025. Impressive. But it needs multi-GPU.

Chimera-MoE applies Mixture-of-Experts only to the bottom layers. The top shared reasoning layers stay dense — no expert duplication. You get specialization from the experts and depth from the loops, without either technique's downsides.

Projected config: 4.5B total params, 1.5B active, 6B+ virtual. Runs on a single RTX 4090 at 50+ tokens per second. That's the reasoning depth of an 8B model on consumer hardware.

But first — finish the current pretraining, run SFT experiments, prove the base scales. Then we build the router.

Links

• GitHub: Fy-/Auron — code, paper, inference
• HuggingFace: nyxia — model weights
• Paper (PDF)
• Soulkyn — the project this is part of