The Case for Small AI Models

The AI industry's default direction is up and to the right: more parameters, more training compute, bigger context windows, bigger everything. That race makes sense for a model trying to be generally capable at anything a person might ask. It's the wrong race for a model whose entire job is reading one person's journal and generating one honest observation from it. For that job, MirrorNotes uses Gemma 3 at 1 billion parameters — small enough to run entirely on an iPhone's Neural Engine — and after a year of building around it, I'd make the same call again.

Bigger isn't the right axis for a narrow task

A frontier cloud model is trained to be good at code, math, medical questions, creative writing, a hundred languages, and everything in between. Most of that capability is dead weight for "notice a pattern across these twelve journal entries and phrase it as one honest sentence." The task is narrow. It doesn't need general intelligence — it needs consistent, well-scoped judgment applied to a small amount of text, over and over, every day.

Once you accept the task is narrow, model size stops being the main lever. What actually determines whether the output is useful is:

• What context the model gets — feeding it your whole journal history produces mush regardless of model size; feeding it the right slice of recent, relevant entries lets even a 1B model find something real.
• How narrow the prompt is — an open-ended "reflect on my life" invites a small model to wander into generic territory. A specific, scoped question keeps it grounded.
• How much you ask it to do in one pass — one clear task per generation, not five things bundled together.

A 1B model with a well-scoped prompt and the right context beats a 70B model with a sloppy one, for this specific job.

What you get back for accepting a smaller model

Those aren't small conveniences. They change the shape of the product. A cloud-AI journaling app has a marginal cost every time you write, which creates real pressure to ration AI features, gate them behind higher tiers, or throttle usage. An on-device model has no marginal cost per generation — the constraint becomes battery and thermal budget on your phone, not a bill on my end. That's part of why MirrorNotes can offer a real free tier with unlimited writing forever: the free tier was never subsidizing a per-user API cost, because there isn't one.

It also means the product works with zero network connection. Every AI feature — Daily Nudge, mood detection, Ask, the Weekly Digest — runs in airplane mode, on a train underground, mid-flight. A cloud-dependent app simply can't offer that; there's always a "waiting for connection" state somewhere.

What you give up

Being honest about the trade, because I'd rather say it here than let a product page imply otherwise:

• A 1B model will occasionally miss a subtler pattern a frontier model would catch. It's a smaller, more literal reasoner.
• Inference still costs battery and a few seconds of compute on-device — there's no server absorbing that cost for the user.
• It requires hardware capable of running the model reasonably — which shaped MirrorNotes as iOS-first rather than a lowest-common-denominator cross-platform build from day one.

For a general-purpose assistant, I wouldn't make this trade. For something that reads your diary and needs to never leave your phone, the trade is the whole point — not a compromise around it.