Engineering notes
I measured Apple's on-device model so you don't have to
July 28, 2026 · Kaloyan Lachezarov
Deforget turns diary prose into reminders, calendar events, meeting notes, and an index of people - using Apple's on-device foundation model, with no cloud fallback, because nothing leaves the device. That choice has a consequence people underestimate: you cannot pin the model. It belongs to the user's operating system. Whatever iOS ships, my users get, and every OS update can move the ground under the product.
When you depend on a model you can't version, you need instruments, not vibes. So Deforget carries its own evaluation harness: a fixed corpus of diary scenarios with typed expectations - dates, events, reminders, commitments, meeting notes, people, decisions, deduplication, and restraint (emotional prose must extract nothing). It runs on real iPad hardware against every iOS beta, and every case is scored twice: once on the raw model output, and once on the final result after a deterministic repair layer. The gap between those two numbers is the engineering, measured.
The numbers
Method notes that matter: OS comparisons use the same app binary - build once, run on the old OS, update the OS, run again without rebuilding, so nothing moves but the model. Five runs per case, because single runs lie. The corpus grows as field reports arrive, so totals shift between revisions; compare percentages, and only trust a raw-to-raw comparison within one corpus version.
| OS build | raw model | after repair layer | the story |
|---|---|---|---|
| iPadOS 26.5 | 46% (78/170) | 75% (127/170) | Mood prose became junk commitments (restraint passed 3 checks of 25); 9 of 170 model calls died in runaway-generation loops |
| 27.0 beta 3 | 61% (104/170) | 91% (155/170) | Same binary as the row above - only the model changed. Restraint 21/25, zero call failures, meetings and dedupe perfect |
| 27.0 beta 4 | 59% (106/180) | 89% (161/180) | Larger corpus (two new adversarial cases, one deliberately raw-hostile - the raw dip is composition, not regression). Restraint 25/25, dates 25/25, still zero failures |
The number I watch most isn't in any single row: the repair layer's contribution stayed close to thirty points across every build - 29 on iOS 26, 30 on beta 3, 31 on beta 4. Model gains and engineering gains compose instead of cannibalizing each other. When Apple ships a better model, my correctness layer doesn't shrink; it stands on higher ground.
What Apple actually fixed in iOS 27
- Restraint. The worst failure mode on iOS 26 was "I feel really good about how today went" becoming a commitment. Feelings turning into to-dos is the fastest way to make someone turn intelligence off - or stop writing honestly, which is worse. Across the 27 cycle the restraint category went from 12 percent of checks to 84 to 100.
- Reliability. On 26, roughly one call in twenty entered a runaway loop - the model babbling until it blew its own context window. On every 27 beta so far: zero, across hundreds of calls.
- Vocabulary. The decision type effectively didn't exist in 26's output; everything flattened into commitments. 27 emits it unprompted.
- Consistency. 26's failures were flaky - one run in two. 27's failures are consistent, and a consistent failure is something you can engineer around. A flaky one isn't.
What's still broken, and what I do about it
Three failure classes survived beta 4, and each gets a different treatment - that's the point of measuring instead of guessing.
- Type confusion at the commitment/event boundary. "The offsite is on the 14th" comes back as a commitment, not a calendar event - consistently. After the third consecutive beta reproduced it, a pre-agreed deterministic retype rule went in: dated, scheduled phrasing outranks the model's label. That rule was written weeks earlier and sat disabled until the reports met its conditions. Fixes here follow a playbook, not a mood.
- Detached clock times. "Next Thursday's sync at 3:30" once resolved the time separately from the day and produced 3:30 AM. The repair layer now grafts a detached clock onto the sentence's date before it can misfire. Deterministic, tested, no model involved.
- Recall flakes. One decision case sometimes returns nothing at all. The discipline for missing output is different: wait. Across this beta cycle, recall has recovered on its own while precision held - and a silent miss is recoverable in the app, while junk output is trust-fatal. Precision gets engineering; recall gets patience.
The corpus is alive
Every interesting field report becomes a permanent test case. A tester's iPad invented a person named "Writer" for a paragraph that named no one - now there's a sentinel case that fails if any person is attached to text that names nobody. Numbered tasks that a deduplication pass once merged ("finish the Q2 report, start the Q3 plan") are a case. This week the diary learned to index links and photos from rich-text paste, which means bare URLs now reach the model - so the corpus gained three URL cases before any guard was written, including a blog permalink with a date in its path, because a phantom calendar event minted from /2026/03/15/ would land in a real calendar. Measure first; guard only what the measurement says needs guarding.
The lesson, if you're building on this stack
The app's guarantees cannot live in the model, because the model isn't yours. They live in a deterministic layer that repairs dates from the source text, recognizes self-declaring types ("we decided..." is a decision no matter what the label says), titles and links meeting bullets without model involvement, kills anything that can't be grounded in the words the user actually wrote - and knows when to do nothing at all. The model contributes recall. The layer contributes truth. Sixty-ish percent raw becomes ninety-ish percent final, on every build, and the pipeline never asks the model to be something it isn't.
All of this runs on the device - the diary never leaves it. The harness ships inside the app, and every number above comes from reports it saved on an ordinary iPad. When iOS 27 goes public I'll run the same corpus one more time, and that run decides the launch.