LazuliQ 1.2 Photon
A 3.1B-parameter model, fine-tuned to see how much reasoning fits into something this small. It won't match the large assistants, and that isn't the point. Small models that run anywhere are where a lot of this is heading.
Early beta. Photon is an experiment and will get things wrong.
What the fine-tune changes
Photon is built on the Qwen2.5-3B-Instruct base. The fine-tune changes how it thinks through an answer rather than how much it knows. The base model's ceiling is still the ceiling.
Steadier step-by-step answers
Trained on reasoning examples so it works through problems in visible steps instead of jumping to an answer. Within its depth, the difference is clear.
Calmer, cleaner tone
Alignment tuning gives replies a more neutral voice and cleaner structure than the raw base model.
More knowledge per parameter
Careful data curation pulls more recall out of a 3.1B footprint than the base model manages, though at this size retrieval matters more than memory.
StartumRAG
A retrieval layer that runs server-side and offline, giving a small model access to facts it was never big enough to hold. No external tool calls, no waiting on the network. It's the most experimental part of Photon, and the most interesting.
Offline and private
Runs inside isolated, internet-free server environments, so nothing leaves the machine. On-device deployment is next.
Ultra-low latency
With no external web APIs in the loop, retrieval adds very little time on top of inference.
Fewer invented answers
Grounding answers in real sources cuts down the guessing a small model would otherwise do when it hits the edge of what it knows.
Under the hood
A modern decoder-only transformer, inherited from the Qwen2.5-3B-Instruct base and built for efficiency over scale.
- Total parameters
- 3.09B
- Context length
- 128K
- Layers
- 36
- Vocabulary
- 151.6K
Grouped-query attention
16 query heads and 2 KV heads. Cuts VRAM usage sharply and speeds up inference, which is what makes a model this size worth running at all.
RoPE embeddings
Rotary positional embeddings for long context. The architecture supports the full 128K window, though quality is strongest well inside it.
SwiGLU and RMSNorm
Standard components in current LLMs. They keep training stable and inference fast.
Tied word embeddings
Keeps the footprint small, at 2.77B non-embedding parameters, without giving up linguistic nuance.
About this release
LazuliQ 1.2 Photon is a student project: a fine-tune of an open-source 3B base model, built and maintained by one developer. It's an experiment in how capable a very small model can be, not a replacement for the large assistants you already use. Expect mistakes: at this size the model can be wrong, invent details, or lose the thread of a long conversation. Everything here is beta, StartumRAG most of all, so check anything important before relying on it.