<?xml version="1.0" encoding="utf-8" standalone="yes"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:media="http://search.yahoo.com/mrss/">
  <channel>
    <title>Privacy · Jared Lynskey</title>
    <link>https://jared.lynskey.co.nz/en/tags/privacy/</link>
    <description>Short stories of value on software engineering and engineering leadership — performance reviews, hiring and scaling teams, CI/CD, DevOps monitoring and agentic AI. Written from Seoul.</description>
    <generator>Hugo</generator>
    <language>en</language>
    <copyright>© 2026 Jared Lynskey</copyright>
    <lastBuildDate>Sat, 07 Nov 2026 00:00:00 +1300</lastBuildDate>
    <atom:link href="https://jared.lynskey.co.nz/en/tags/privacy/index.xml" rel="self" type="application/rss+xml" />
    <item>
      <title>Self-Hosted Sentry Alternative: Sentry&#39;s SDK, My Own Server</title>
      <link>https://jared.lynskey.co.nz/en/posts/2026/2026-10-09-sentry-sdk-own-server/</link>
      <guid isPermaLink="true">https://jared.lynskey.co.nz/en/posts/2026/2026-10-09-sentry-sdk-own-server/</guid>
      <pubDate>Fri, 09 Oct 2026 00:00:00 +1300</pubDate>
      <dc:creator>Jared Lynskey</dc:creator>
      <category>React Native</category>
      <category>Expo</category>
      <category>Sentry</category>
      <category>Django</category>
      <category>Crash Reporting</category>
      <category>Self-Hosted</category>
      <category>iOS</category>
      <category>Android</category>
      <category>Privacy</category>
      <description>A self-hosted Sentry alternative for React Native crash reporting: keep Sentry&#39;s open-source SDK, point its DSN at your own API, and group, alert on and scrub crashes yourself.</description>
      <content:encoded><![CDATA[<p>If you want self-hosted crash reporting for a React Native app, you don&rsquo;t have to run Sentry&rsquo;s own server or pay for sentry.io. Sentry&rsquo;s SDK is open source and its DSN can point at any host, so I aimed it at an endpoint in my own Django API, which receives the reports, groups them, alerts on them and keeps them, at no cost beyond the API that was already running. The one thing I haven&rsquo;t replaced yet is symbolication, so stack traces don&rsquo;t point to source lines.</p>
<p>Until this week, the only way I found out our app had crashed was when someone told me. Most people don&rsquo;t. They reopen the app, and if it happens again they stop opening it. The API side was covered (every ERROR log already lands in a team chat channel), but a crash on a phone left no trace anywhere I could see.</p>
<p>The app did have one reporter already. When it puts an error message in front of someone, it posts that message to a small endpoint on the API, which logs it at ERROR, so it reaches the chat channel too. I added it after a delete that had <em>worked</em> left the row in the list, and tapping it again showed &ldquo;No Item matches the given query&rdquo;. The server had logged two ordinary 404s and said nothing. That covers errors the app catches and shows. A crash is different: the JavaScript that would send the report is the thing that just died.</p>
<p>The obvious fix is Sentry. I like Sentry&rsquo;s SDK a lot; it&rsquo;s the reason crash reporting on React Native is a solved problem. What I didn&rsquo;t want was another vendor holding user data, another privacy-policy line, and another bill for something that, at our size, is a few reports a day at most. So I kept the SDK and replaced the service: the app&rsquo;s DSN points at our own API, and a small piece of the Django app there receives what it sends, groups it, alerts on it, and keeps it.</p>
<p>This post is how that works, and how I checked that a real native crash actually makes it through.</p>

<h2 class="relative group">The DSN trick
    <div id="the-dsn-trick" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#the-dsn-trick" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>A Sentry DSN looks like <code>https://&lt;public-key&gt;@&lt;host&gt;/&lt;project-id&gt;</code>. The SDK turns that into a POST to <code>https://&lt;host&gt;/api/&lt;project-id&gt;/envelope/</code>, with the key in an <code>X-Sentry-Auth</code> header. Nothing about it is specific to sentry.io. So the DSN baked into the app is:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">https://&lt;CRASH_REPORT_KEY&gt;<span>@</span>api.example.com/1</span></span></code></pre></div></div>
<p>and Django has a route that matches what the SDK will call:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="cl"><span class="n">path</span><span class="p">(</span><span class="s2">&#34;api/&lt;int:project_id&gt;/envelope/&#34;</span><span class="p">,</span> <span class="n">crash_envelope</span><span class="p">,</span> <span class="n">name</span><span class="o">=</span><span class="s2">&#34;crash_envelope&#34;</span><span class="p">),</span></span></span></code></pre></div></div>
<p>The key is in every binary, so it isn&rsquo;t a secret. It&rsquo;s there so that if the endpoint ever starts attracting junk I can change it, and so that an unset key on the server refuses everything rather than accepting the world:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="cl"><span class="k">def</span> <span class="nf">crash_envelope</span><span class="p">(</span><span class="n">request</span><span class="p">,</span> <span class="n">project_id</span><span class="p">):</span>
</span></span><span class="line"><span class="cl">    <span class="n">expected</span> <span class="o">=</span> <span class="n">settings</span><span class="o">.</span><span class="n">CRASH_REPORT_KEY</span>
</span></span><span class="line"><span class="cl">    <span class="k">if</span> <span class="ow">not</span> <span class="n">expected</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">        <span class="k">return</span> <span class="n">Response</span><span class="p">({</span><span class="s2">&#34;detail&#34;</span><span class="p">:</span> <span class="s2">&#34;Crash reporting is not configured&#34;</span><span class="p">},</span> <span class="n">status</span><span class="o">=</span><span class="mi">503</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">    <span class="k">if</span> <span class="ow">not</span> <span class="n">hmac</span><span class="o">.</span><span class="n">compare_digest</span><span class="p">(</span><span class="n">_key</span><span class="p">(</span><span class="n">request</span><span class="p">)</span><span class="o">.</span><span class="n">encode</span><span class="p">(),</span> <span class="n">expected</span><span class="o">.</span><span class="n">encode</span><span class="p">()):</span>
</span></span><span class="line"><span class="cl">        <span class="k">return</span> <span class="n">Response</span><span class="p">({</span><span class="s2">&#34;detail&#34;</span><span class="p">:</span> <span class="s2">&#34;Unknown key&#34;</span><span class="p">},</span> <span class="n">status</span><span class="o">=</span><span class="mi">401</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">    <span class="o">...</span></span></span></code></pre></div></div>
<p><code>_key</code> reads <code>sentry_key=</code> out of the <code>X-Sentry-Auth</code> header, falling back to the query string, which some SDKs use instead.</p>

<h2 class="relative group">What the SDK sends: envelopes
    <div id="what-the-sdk-sends-envelopes" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#what-the-sdk-sends-envelopes" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>Sentry&rsquo;s wire format is the <a href="https://develop.sentry.dev/sdk/foundations/envelopes/"  target="_blank" rel="noreferrer">envelope</a>: one JSON header line, then any number of items, each one a JSON header line followed by a payload. If the item header has a <code>length</code>, the payload is exactly that many bytes; otherwise it runs to the next newline.</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">{&#34;event_id&#34;:&#34;9ec79c33...&#34;,&#34;dsn&#34;:&#34;https://...<span>@</span>api.example.com/1&#34;}
</span></span><span class="line"><span class="cl">{&#34;type&#34;:&#34;event&#34;,&#34;length&#34;:4123}
</span></span><span class="line"><span class="cl">{ ...the event JSON, 4123 bytes... }
</span></span><span class="line"><span class="cl">{&#34;type&#34;:&#34;client_report&#34;}
</span></span><span class="line"><span class="cl">{&#34;timestamp&#34;:1791400000.5,&#34;discarded_events&#34;:[]}</span></span></code></pre></div></div>
<p>Two details caught me out:</p>
<ul>
<li><strong>The native SDKs gzip the body; the JavaScript one doesn&rsquo;t.</strong> A JS error captured by <code>@sentry/react-native</code> arrives as plain text, while a native crash from sentry-cocoa or sentry-android arrives with <code>Content-Encoding: gzip</code>. The decoder handles gzip, zlib-wrapped deflate and raw deflate, and stops decompressing at 1MB, because a crash event is a few kilobytes and anything bigger isn&rsquo;t one.</li>
<li><strong>Length-prefixed payloads can contain newlines.</strong> If you split on <code>\n</code> you&rsquo;ll mangle every event the native SDKs send. You have to honour <code>length</code> when it&rsquo;s there.</li>
</ul>
<p>The parser is short. It only keeps <code>event</code> items; sessions, transactions, attachments and client reports are accepted and dropped.</p>

<h2 class="relative group">Answering in a way the SDK understands
    <div id="answering-in-a-way-the-sdk-understands" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#answering-in-a-way-the-sdk-understands" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>This matters more than I expected. The SDK keeps a queue on disk and retries anything it isn&rsquo;t sure was delivered, so what the server answers decides whether it keeps trying:</p>
<ul>
<li><strong>200</strong> once stored, <em>or deliberately dropped</em>. If I answered a session item with a 4xx, the SDK would keep it and retry forever.</li>
<li><strong>429 with <code>Retry-After</code></strong> when throttled. The endpoint has a per-IP throttle (120 an hour), because a phone in a crash loop sends one report per launch, and the SDK backs off on a 429 the way you&rsquo;d hope.</li>
<li><strong>400</strong> for a malformed envelope, logged as a warning, not an error: a broken report says a client is broken, and the next good one will say what crashed.</li>
</ul>
<p>The SDK also resends the same event if it isn&rsquo;t sure the first send arrived, so <code>event_id</code> has a unique constraint and a duplicate is a quiet no-op.</p>

<h2 class="relative group">Grouping without source maps
    <div id="grouping-without-source-maps" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#grouping-without-source-maps" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>This is where a self-built receiver has to make a choice Sentry makes for you. Sentry groups crashes by stack trace. But a release Hermes bundle is minified, so frame names are single letters and line/column positions change with every build. Group by frames, and the same bug opens a new issue every release.</p>
<p>Without source maps (more on that at the end), the most stable thing in a JavaScript error is its type and message. So the fingerprint is the exception type plus the message, with the parts that differ between sightings masked out:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="cl"><span class="n">_MASKS</span> <span class="o">=</span> <span class="p">[</span>
</span></span><span class="line"><span class="cl">    <span class="p">(</span><span class="n">re</span><span class="o">.</span><span class="n">compile</span><span class="p">(</span><span class="sa">r</span><span class="s2">&#34;0x[0-9a-fA-F]+&#34;</span><span class="p">),</span> <span class="s2">&#34;&lt;addr&gt;&#34;</span><span class="p">),</span>
</span></span><span class="line"><span class="cl">    <span class="p">(</span><span class="n">re</span><span class="o">.</span><span class="n">compile</span><span class="p">(</span><span class="sa">r</span><span class="s2">&#34;\b[0-9a-fA-F]</span><span class="si">{8}</span><span class="s2">-[0-9a-fA-F]</span><span class="si">{4}</span><span class="s2">-[0-9a-fA-F]</span><span class="si">{4}</span><span class="s2">-[0-9a-fA-F]</span><span class="si">{4}</span><span class="s2">-[0-9a-fA-F]</span><span class="si">{12}</span><span class="s2">\b&#34;</span><span class="p">),</span> <span class="s2">&#34;&lt;uuid&gt;&#34;</span><span class="p">),</span>
</span></span><span class="line"><span class="cl">    <span class="p">(</span><span class="n">re</span><span class="o">.</span><span class="n">compile</span><span class="p">(</span><span class="sa">r</span><span class="s2">&#34;\b[0-9a-fA-F]{16,}\b&#34;</span><span class="p">),</span> <span class="s2">&#34;&lt;hex&gt;&#34;</span><span class="p">),</span>
</span></span><span class="line"><span class="cl">    <span class="p">(</span><span class="n">re</span><span class="o">.</span><span class="n">compile</span><span class="p">(</span><span class="sa">r</span><span class="s2">&#34;\d+&#34;</span><span class="p">),</span> <span class="s2">&#34;&lt;n&gt;&#34;</span><span class="p">),</span>
</span></span><span class="line"><span class="cl"><span class="p">]</span></span></span></code></pre></div></div>
<p><code>Cannot read property 'title' of undefined (reading index 3)</code> and <code>(reading index 17)</code> become the same issue. If the SDK supplies its own <code>fingerprint</code> without <code>{{ default }}</code> in it, that wins, so I can force grouping from the client when the default is wrong. An event with no exception falls back to its masked log message, and one with neither falls back to the top in-app frame. For a native signal crash the message is something like <code>Signal 6, Code 0</code>, so those group by signal, which is coarse but honest until there&rsquo;s symbolication.</p>

<h2 class="relative group">Alerting without flooding the channel
    <div id="alerting-without-flooding-the-channel" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#alerting-without-flooding-the-channel" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>Each fingerprint becomes one issue, and each report is stored as an event under it. The issue only logs at ERROR (which is what the chat alerts listen for) in exactly these cases:</p>
<ul>
<li>a <strong>new</strong> issue</li>
<li>a <strong>resolved</strong> issue that <strong>comes back</strong>, which reopens it</li>
<li>the <strong>10th, 100th, 1,000th and 10,000th</strong> sighting</li>
</ul>
<p>An ordinary repeat only increments a counter. An issue marked <em>ignored</em> keeps counting and never alerts. With the chat handler&rsquo;s own rate limit on top, a crash loop on a bad release reads as a handful of messages rather than a wall.</p>
<p>Storage is bounded the same way: each issue keeps its <strong>50 most recent</strong> events and deletes older ones as new ones arrive. That broke my first chart, because 50 events can&rsquo;t tell you how often something happened last week, so each issue also keeps a running count per day for 90 days, updated in the same transaction as the counter. The chart draws from the counts; the events are only for reading.</p>
<p>Two small races are worth handling if you build one of these. When a brand-new crash hits several phones at once, two reports can both try to create the issue, so make the fingerprint unique and have the loser re-read the existing row. And measure the 90-day window from the server&rsquo;s today, not from the newest report, because a phone with its clock set wrong can date a report anywhere, including next year.</p>
<p>Payloads are capped at 128KB. An event over that loses its breadcrumbs and debug metadata first, and if it&rsquo;s still too big only the exception, message, release and platform are kept, with a flag so I know it was cut.</p>

<h2 class="relative group">Scrubbing
    <div id="scrubbing" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#scrubbing" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>The SDK runs with <code>sendDefaultPii: false</code>, so in principle no IP address, cookies or request bodies are sent. The server doesn&rsquo;t rely on that: before storing anything it keeps only <code>id</code> and <code>username</code> from <code>user</code> and drops <code>request</code> entirely, in case a later SDK version or setting starts sending them.</p>
<p>Testing against a real device turned up two more things to strip:</p>
<ul>
<li><strong>Breadcrumbs record full URLs, query strings included.</strong> The app&rsquo;s search requests carry the search term in the query string, so a crash right after a search would have stored what the person searched for. The server now cuts every breadcrumb URL at the <code>?</code>.</li>
<li><strong>Console breadcrumbs carry the raw arguments</strong> that were passed to <code>console.log</code>, which can be any object the app happened to log. The message text stays (it&rsquo;s useful), and the arguments are dropped.</li>
</ul>
<p>I only found those by looking at an actual report, which is the main argument for the next section.</p>

<h2 class="relative group">The client side
    <div id="the-client-side" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#the-client-side" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>On the app, everything goes through one small module, for the same reason I wrap the over-the-air update SDK: an OTA bundle also runs on binaries built before the native module existed. A top-level <code>import * as Sentry from &quot;@sentry/react-native&quot;</code> on a binary without <code>RNSentry</code> throws on import, and that&rsquo;s a blank screen. So the SDK is required lazily, and only after checking the native module is actually there:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-ts" data-lang="ts"><span class="line"><span class="cl"><span class="kr">import</span> <span class="p">{</span> <span class="nx">NativeModules</span><span class="p">,</span> <span class="nx">Platform</span><span class="p">,</span> <span class="nx">TurboModuleRegistry</span> <span class="p">}</span> <span class="kr">from</span> <span class="s2">&#34;react-native&#34;</span><span class="p">;</span>
</span></span><span class="line"><span class="cl"><span class="kr">import</span> <span class="nx">Constants</span> <span class="kr">from</span> <span class="s2">&#34;expo-constants&#34;</span><span class="p">;</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="kr">const</span> <span class="nx">DSN</span> <span class="o">=</span> <span class="nx">process</span><span class="p">.</span><span class="nx">env</span><span class="p">.</span><span class="nx">EXPO_PUBLIC_CRASH_DSN</span><span class="p">;</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="kd">function</span> <span class="nx">nativeModulePresent</span><span class="p">()</span><span class="o">:</span> <span class="kr">boolean</span> <span class="p">{</span>
</span></span><span class="line"><span class="cl">  <span class="k">try</span> <span class="p">{</span>
</span></span><span class="line"><span class="cl">    <span class="k">return</span> <span class="nb">Boolean</span><span class="p">(</span><span class="nx">TurboModuleRegistry</span><span class="p">.</span><span class="kr">get</span><span class="p">(</span><span class="s2">&#34;RNSentry&#34;</span><span class="p">)</span> <span class="o">??</span> <span class="nx">NativeModules</span><span class="p">.</span><span class="nx">RNSentry</span><span class="p">);</span>
</span></span><span class="line"><span class="cl">  <span class="p">}</span> <span class="k">catch</span> <span class="p">{</span>
</span></span><span class="line"><span class="cl">    <span class="k">return</span> <span class="kc">false</span><span class="p">;</span>
</span></span><span class="line"><span class="cl">  <span class="p">}</span>
</span></span><span class="line"><span class="cl"><span class="p">}</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="kr">export</span> <span class="kd">function</span> <span class="nx">initCrashReporting</span><span class="p">()</span><span class="o">:</span> <span class="kr">boolean</span> <span class="p">{</span>
</span></span><span class="line"><span class="cl">  <span class="k">if</span> <span class="p">(</span><span class="o">!</span><span class="nx">DSN</span> <span class="o">||</span> <span class="nx">Platform</span><span class="p">.</span><span class="nx">OS</span> <span class="o">===</span> <span class="s2">&#34;web&#34;</span> <span class="o">||</span> <span class="o">!</span><span class="nx">nativeModulePresent</span><span class="p">())</span> <span class="k">return</span> <span class="kc">false</span><span class="p">;</span>
</span></span><span class="line"><span class="cl">  <span class="kr">const</span> <span class="nx">Sentry</span> <span class="o">=</span> <span class="kr">require</span><span class="p">(</span><span class="s2">&#34;@sentry/react-native&#34;</span><span class="p">);</span>
</span></span><span class="line"><span class="cl">  <span class="nx">Sentry</span><span class="p">.</span><span class="nx">init</span><span class="p">({</span>
</span></span><span class="line"><span class="cl">    <span class="nx">dsn</span>: <span class="kt">DSN</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">    <span class="nx">release</span>: <span class="kt">Constants.expoConfig?.version</span> <span class="o">??</span> <span class="s2">&#34;unknown&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">    <span class="nx">sendDefaultPii</span>: <span class="kt">false</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">    <span class="nx">tracesSampleRate</span>: <span class="kt">0</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">    <span class="nx">enableAutoSessionTracking</span>: <span class="kt">false</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">    <span class="c1">// ...turn off the other tracing options the same way
</span></span></span><span class="line"><span class="cl">    <span class="nx">maxBreadcrumbs</span>: <span class="kt">50</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">    <span class="nx">beforeSend</span><span class="p">(</span><span class="nx">event</span><span class="p">)</span> <span class="p">{</span> <span class="cm">/* tag the OTA bundle */</span> <span class="k">return</span> <span class="nx">event</span><span class="p">;</span> <span class="p">},</span>
</span></span><span class="line"><span class="cl">  <span class="p">});</span>
</span></span><span class="line"><span class="cl">  <span class="k">return</span> <span class="kc">true</span><span class="p">;</span>
</span></span><span class="line"><span class="cl"><span class="p">}</span></span></span></code></pre></div></div>
<p>Errors only. The server doesn&rsquo;t store traces, sessions, profiles or replays, so none of them are switched on. No DSN (development, E2E builds) means the SDK never starts. The build profiles set the DSN and an environment (<code>staging</code> or <code>production</code>), so both kinds of build report to the same endpoint and can be told apart. The signed-in user is set by account ID, never a name or an email.</p>
<p>The error boundaries needed an explicit call. Expo Router&rsquo;s root <code>ErrorBoundary</code> catches a render error and shows its own screen, so that error never reaches the global handler the SDK hooks into. The fix is a small wrapper that calls <code>captureException</code> in a <code>useEffect</code> and then renders the router&rsquo;s boundary as before, used for the root and per-route boundaries alike. It&rsquo;s in an effect rather than in the render so it reports once rather than on every re-render. The root component is also wrapped with <code>Sentry.wrap</code>, and if that throws it falls back to the bare component.</p>
<p>For the web there&rsquo;s a <code>.web.ts</code> version of the module with every function switched off. Metro picks the platform file, so the SDK never ends up in the web bundle at all. The web app has no native module to run on.</p>

<h2 class="relative group">Proving a native crash arrives
    <div id="proving-a-native-crash-arrives" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#proving-a-native-crash-arrives" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>The unit tests post hand-written envelopes, which proves the parser handles what I <em>think</em> the SDK sends. Two days after release, production had zero crash reports. Only three people had updated yet, so zero was plausible, but &ldquo;nobody has crashed&rdquo; and &ldquo;crashes don&rsquo;t arrive&rdquo; look identical from the server. I wanted to see a real one.</p>
<p>So I built a Release simulator binary with the DSN pointed at a local Django on port 8000 and a throwaway key:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl"><span class="nv">EXPO_PUBLIC_CRASH_DSN</span><span class="o">=</span>http://localcrashtestkey0000@localhost:8000/1 <span class="se">\
</span></span></span><span class="line"><span class="cl"><span class="nv">EXPO_PUBLIC_CRASH_ENVIRONMENT</span><span class="o">=</span>local-verify <span class="se">\
</span></span></span><span class="line"><span class="cl">xcodebuild -workspace ios/MyApp.xcworkspace -scheme MyApp -configuration Release <span class="se">\
</span></span></span><span class="line"><span class="cl">  -destination <span class="s1">&#39;platform=iOS Simulator,id=&lt;udid&gt;&#39;</span> -derivedDataPath /tmp/iosbuild</span></span></code></pre></div></div>
<p>(Release, not Debug: Debug builds link the dev launcher and behave differently, and Release is what people run.) Then installed it, launched it, waited a few seconds, and killed it the way a real fault would:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl">xcrun simctl install booted /tmp/iosbuild/Build/Products/Release-iphonesimulator/MyApp.app
</span></span><span class="line"><span class="cl">xcrun simctl launch booted com.example.myapp      <span class="c1"># prints the pid</span>
</span></span><span class="line"><span class="cl"><span class="nb">kill</span> -ABRT &lt;pid&gt;</span></span></code></pre></div></div>
<p>Simulator apps are ordinary macOS processes, so a signal reaches SentryCrash&rsquo;s handler exactly as it would on a device. The handler writes the report to disk; nothing is sent while the process is dying. On the <strong>next launch</strong> the SDK finds it and sends it. I relaunched, and the simulator&rsquo;s network log showed a connection to <code>:8000</code> a second later.</p>
<p>Then nothing in the Django log, which had me confused for a minute: the dev server only logs non-2xx responses at the level I had it on. The database had it:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">SIGABRT: Signal 6, Code 0 | cocoa 1.4.0 iOS 27.1 iPhone19,4 handled=False
</span></span><span class="line"><span class="cl">received 09:19:53, crashed 09:19:38
</span></span><span class="line"><span class="cl">contexts: app, device, os, ota_updates, trace · 24 breadcrumbs</span></span></code></pre></div></div>
<p>Fifteen seconds from crash to stored, most of which was me relaunching. And looking at the real payload is what taught me things the tests couldn&rsquo;t:</p>
<ul>
<li><strong>The user ID wasn&rsquo;t the account ID.</strong> The test build isn&rsquo;t really signed in, so sentry-cocoa fell back to its own per-install ID. Worth knowing, and worth saying in the privacy note: a report carries the account&rsquo;s ID if you&rsquo;re signed in and a random install ID if you&rsquo;re not.</li>
<li><strong>There&rsquo;s no time zone.</strong> I&rsquo;d assumed the culture context would be there. The device context has <code>locale</code>, memory, screen size, low-power mode and the kernel version, but no time zone.</li>
<li><strong>Native crashes skip <code>beforeSend</code>.</strong> My JS <code>beforeSend</code> adds an <code>ota</code> tag with the running OTA bundle, but a native crash is assembled by the native SDK on the next launch and never passes through the JS hook. To tag native crashes, the tag has to be set on the scope with <code>setTag</code> once the bundle is known, so it syncs down to the native layer.</li>
<li><strong>The breadcrumbs showed what the app does.</strong> Console warnings, view-controller lifecycle, and HTTP calls, including the analytics SDK&rsquo;s own requests to its collection endpoint, which is a good reminder of what else the app talks to. And the URLs had query strings, which is how I found the search-term problem above.</li>
</ul>

<h2 class="relative group">Reading them
    <div id="reading-them" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#reading-them" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>The reports are read in the Django admin, on an issue page loosely modelled on Sentry&rsquo;s: counts and Resolve / Ignore / Reopen at the top, a bar per day, the split by release, OS, device and OTA bundle, then one event at a time with its exception, its frames (app frames highlighted, library frames folded away), the breadcrumbs and the device context. It didn&rsquo;t need a new dependency.</p>
<p>One fix came out of building it: for a native crash, the &ldquo;culprit&rdquo; frame was the binary&rsquo;s path on the device, which includes a per-install UUID, so the same crash looked different on every phone. It now shows only the address and the binary name, like <code>0x1000a1b2c (MyApp)</code>.</p>

<h2 class="relative group">The privacy policy, all of it
    <div id="the-privacy-policy-all-of-it" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#the-privacy-policy-all-of-it" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>Crash reports are personal information, near enough: an account ID, a phone model, and what someone did in the minute before the app died. So the privacy policy needed a paragraph. Writing it properly meant saying exactly what a report carries, so I listed it from the payload I&rsquo;d just looked at: the error, app version and OTA update, phone model, OS and language, free memory and screen size, the last few dozen steps (screens, web addresses with anything after a <code>?</code> removed, the app&rsquo;s own log lines), and the account ID or a random install ID. The policy also says the reports come to our server and not to Sentry or any other crash-reporting company. Retention gets its own line in the table: the 50 most recent reports of each problem, plus a daily count for 90 days.</p>
<p>Once I was checking one paragraph against the code, it was hard not to check the rest. The existing policy didn&rsquo;t hold up. The delete-account page said push-notification tokens and passkeys were deleted immediately when you close your account, and that server logs were gone within 90 days. None of that was true. Closing an account unlinks the sign-in methods and anonymises the profile, but the push tokens and passkeys stay until someone deletes them by hand, and the log archive has no expiry yet. So the whole policy was rewritten from an inventory of what the code actually does, and those lines now say &ldquo;not yet removed automatically; email us&rdquo;. That&rsquo;s less reassuring, and it&rsquo;s accurate.</p>
<p>The repo now has a rule for it: the policy describes what the code does, not what we intend, and a change that collects, sends or keeps personal data somewhere new updates it in the same commit. That includes the summary screen in the app, the delete-account page, and the app-store privacy forms, which only live in the store consoles. Crash reporting was the change that made me write that down.</p>

<h2 class="relative group">What&rsquo;s missing
    <div id="whats-missing" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#whats-missing" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p><strong>Symbolication.</strong> This is the real gap, and the one thing sentry.io does that I haven&rsquo;t replaced. Stacks are minified for JavaScript and unsymbolicated for native code:</p>
<ul>
<li><strong>JavaScript:</strong> frames point into the Hermes bundle. Fixing it means keeping the composed Hermes source map for every store build and every OTA bundle, keyed by release and bundle, and applying it to JS frames when the page renders. That&rsquo;s the next piece, because most of our crashes will be JavaScript.</li>
<li><strong>iOS native:</strong> instruction addresses need the build&rsquo;s dSYMs.</li>
<li><strong>Android:</strong> Java frames are obfuscated by R8 when minification is on (they need <code>mapping.txt</code>), and C++ frames need NDK symbols.</li>
</ul>
<p>There&rsquo;s also a smaller gap I found above: native crashes don&rsquo;t carry the OTA tag yet, because it&rsquo;s only set in <code>beforeSend</code>. Setting it on the scope once the running bundle is confirmed is a few lines I haven&rsquo;t written yet.</p>
<p>For now, an issue tells me <em>what</em> crashed, on which release, OTA bundle and device, and what the person was doing just before. It doesn&rsquo;t tell me <em>where</em> in the source. For a small app that ships a few times a week, the type, the message and the breadcrumbs have been enough to find things so far.</p>
<p>The receiver, the admin page and the tests aren&rsquo;t a lot of code, and it costs nothing on top of the API that was already running. The SDK does the hard part, which is catching a crash on a phone and getting the report off it on the next launch.</p>
]]></content:encoded>
      <media:content url="https://jared.lynskey.co.nz/en/posts/2026/2026-10-09-sentry-sdk-own-server/featured.jpg" medium="image" type="image/jpeg" />
    </item>
    <item>
      <title>On-Device LLM in React Native: Running Qwen Offline With llama.rn and llama.cpp</title>
      <link>https://jared.lynskey.co.nz/en/posts/2026/2026-01-30-on-device-ai/</link>
      <guid isPermaLink="true">https://jared.lynskey.co.nz/en/posts/2026/2026-01-30-on-device-ai/</guid>
      <pubDate>Fri, 30 Jan 2026 00:00:00 +0000</pubDate>
      <dc:creator>Jared Lynskey</dc:creator>
      <category>React Native</category>
      <category>AI</category>
      <category>LLM</category>
      <category>llama.cpp</category>
      <category>Qwen</category>
      <category>On-Device AI</category>
      <category>Privacy</category>
      <category>Mobile Development</category>
      <description>How I run a small LLM entirely on the phone in a React Native and Expo app: llama.rn, a Qwen GGUF model at Q4_K_M, streaming, tool calling a small model can manage, and real numbers from an iPhone and a Galaxy S23.</description>
      <content:encoded><![CDATA[<p>The AI assistant in an estimating app I build runs entirely on the phone. Nothing leaves the device. Users can create jobs, search customers, and manage projects by typing plain English, and it all keeps working in airplane mode.</p>
<p>This post walks through how I built it with <strong>llama.rn</strong> (React Native bindings for llama.cpp) and <strong>Qwen 1.7B</strong>, a small model that turned out to be more capable than I expected.</p>
<p>The short version: install llama.rn, download a 4-bit GGUF model (about 1 GB) the first time the app opens, load it with <code>initLlama</code>, and stream completions from it. On an iPhone 14 Pro that runs at about 15 tokens a second with no network at all. The rest of this post is what it took to make that useful: picking the model, getting tool calls out of it reliably, and keeping memory in check.</p>
<blockquote><p><strong>📝 Update:</strong> This implementation now uses <a href="https://huggingface.co/unsloth/Qwen3.5-2B-GGUF"  target="_blank" rel="noreferrer">Qwen3.5-2B-GGUF</a> instead of the original Qwen3-1.7B. I&rsquo;ve also stopped using the <code>/no_think</code> message hack and now disable thinking mode properly via the <code>chat_template_kwargs</code> parameter:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="cl"><span class="n">chat_response</span> <span class="o">=</span> <span class="n">client</span><span class="o">.</span><span class="n">chat</span><span class="o">.</span><span class="n">completions</span><span class="o">.</span><span class="n">create</span><span class="p">(</span>
</span></span><span class="line"><span class="cl">    <span class="n">model</span><span class="o">=</span><span class="s2">&#34;Qwen/Qwen3.5-27B&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">    <span class="n">messages</span><span class="o">=</span><span class="n">messages</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">    <span class="n">max_tokens</span><span class="o">=</span><span class="mi">32768</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">    <span class="n">temperature</span><span class="o">=</span><span class="mf">0.7</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">    <span class="n">top_p</span><span class="o">=</span><span class="mf">0.8</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">    <span class="n">presence_penalty</span><span class="o">=</span><span class="mf">1.5</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">    <span class="n">extra_body</span><span class="o">=</span><span class="p">{</span>
</span></span><span class="line"><span class="cl">        <span class="s2">&#34;top_k&#34;</span><span class="p">:</span> <span class="mi">20</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">        <span class="s2">&#34;chat_template_kwargs&#34;</span><span class="p">:</span> <span class="p">{</span><span class="s2">&#34;enable_thinking&#34;</span><span class="p">:</span> <span class="kc">False</span><span class="p">},</span>
</span></span><span class="line"><span class="cl">    <span class="p">},</span>
</span></span><span class="line"><span class="cl"><span class="p">)</span></span></span></code></pre></div></div>
<p>Much cleaner than smuggling control flags into the prompt.</p>
</blockquote><figure><div class="film-pair">
  <div class="mc-film film-dark">
  <video
    class="my-0 rounded-md"
    controls playsinline preload="none"
    poster="https://jared.lynskey.co.nz/en/posts/2026/2026-01-30-on-device-ai/on-device-ai-film-poster.jpg" style="background:url('data:image/webp;base64,UklGRnAAAABXRUJQVlA4IGQAAACwBACdASogABIAPw1uqUemIyIYBABjEMT0gAFRZzp3IyiSND5CxppNutGcCYkAAP7ixmKvS+0H/fAThx4cbifiH8PzygNYbl6sJuQFubAi+gzkxN3eXPGHfroT5K82AG+x8AAA') center/cover no-repeat"
    width="1280"
    height="720"
    aria-label="A short narrated film of this post">
    <source src="https://jared.lynskey.co.nz/en/posts/2026/2026-01-30-on-device-ai/on-device-ai-film.webm" type="video/webm">
    <track kind="subtitles" src="https://jared.lynskey.co.nz/en/posts/2026/2026-01-30-on-device-ai/on-device-ai-film_en.vtt" srclang="en" label="English">
    <track kind="subtitles" src="https://jared.lynskey.co.nz/en/posts/2026/2026-01-30-on-device-ai/on-device-ai-film_ja.vtt" srclang="ja" label="日本語">
    <track kind="subtitles" src="https://jared.lynskey.co.nz/en/posts/2026/2026-01-30-on-device-ai/on-device-ai-film_ko.vtt" srclang="ko" label="한국어">
    <track kind="subtitles" src="https://jared.lynskey.co.nz/en/posts/2026/2026-01-30-on-device-ai/on-device-ai-film_vi.vtt" srclang="vi" label="Tiếng Việt">
    <track kind="subtitles" src="https://jared.lynskey.co.nz/en/posts/2026/2026-01-30-on-device-ai/on-device-ai-film_zh-cn.vtt" srclang="zh-cn" label="简体中文">
  </video>
  </div>
  <div class="mc-film film-light">
  <video
    class="my-0 rounded-md"
    controls playsinline preload="none"
    poster="https://jared.lynskey.co.nz/en/posts/2026/2026-01-30-on-device-ai/on-device-ai-film-light-poster.jpg" style="background:url('data:image/webp;base64,UklGRoIAAABXRUJQVlA4IHYAAACwBACdASogABIAPw1uqUemIyIYBABjEMT0gAFRZY2mogaLUpH79C93CfzUrxKAAP7vul6EGA0ytD4npxsGq682/6nTvmGx0/XY0S/eONnq1STOt/i7tLHFRM4DTibTbGueLxPoaxVrlM6FoYdfCU8gCNqwAAAA') center/cover no-repeat"
    width="1280"
    height="720"
    aria-label="A short narrated film of this post">
    <source src="https://jared.lynskey.co.nz/en/posts/2026/2026-01-30-on-device-ai/on-device-ai-film-light.webm" type="video/webm">
    <track kind="subtitles" src="https://jared.lynskey.co.nz/en/posts/2026/2026-01-30-on-device-ai/on-device-ai-film_en.vtt" srclang="en" label="English">
    <track kind="subtitles" src="https://jared.lynskey.co.nz/en/posts/2026/2026-01-30-on-device-ai/on-device-ai-film_ja.vtt" srclang="ja" label="日本語">
    <track kind="subtitles" src="https://jared.lynskey.co.nz/en/posts/2026/2026-01-30-on-device-ai/on-device-ai-film_ko.vtt" srclang="ko" label="한국어">
    <track kind="subtitles" src="https://jared.lynskey.co.nz/en/posts/2026/2026-01-30-on-device-ai/on-device-ai-film_vi.vtt" srclang="vi" label="Tiếng Việt">
    <track kind="subtitles" src="https://jared.lynskey.co.nz/en/posts/2026/2026-01-30-on-device-ai/on-device-ai-film_zh-cn.vtt" srclang="zh-cn" label="简体中文">
  </video>
  </div>
  <script>window.mcFilm && window.mcFilm()</script></div><figcaption>This post as a nine-minute film, if you&rsquo;d rather watch than read.</figcaption>
</figure>


<h2 class="relative group">Why I didn&rsquo;t want cloud AI for this
    <div id="why-i-didnt-want-cloud-ai-for-this" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#why-i-didnt-want-cloud-ai-for-this" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>The standard recipe for an AI feature in a mobile app is: user types a message, app sends it to a cloud model provider, response comes back, bill ticks up.</p>
<p>For an app that tradespeople use to run their business, that recipe has problems. Customer names, addresses, and phone numbers go to a third party. Project details, pricing, and notes sit on someone else&rsquo;s servers. There are GDPR and data-residency questions I&rsquo;d rather not spend time on. And the API bill grows with every user.</p>
<p>The alternative is to run the model on the user&rsquo;s phone, so that&rsquo;s what I did.</p>

<h2 class="relative group">Why this is finally practical
    <div id="why-this-is-finally-practical" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#why-this-is-finally-practical" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>A year earlier I wouldn&rsquo;t have bothered, but three things changed roughly at once. Quantised models got small — Qwen 1.7B at Q4_K_M is 1.1 GB, smaller than most games, and it downloads once and lives in app storage. Mobile GPUs got fast — llama.cpp uses Metal on iOS and OpenCL on recent Qualcomm Android GPUs, and on an iPhone 14 Pro I see about 15 tokens a second, plenty for real-time streaming. And small models got genuinely useful: Qwen 1.7B follows structured instructions, parses JSON, and chains multi-step reasoning. That&rsquo;s exactly the profile business logic needs. Nobody&rsquo;s asking it to write poetry.</p>

<h2 class="relative group">Picking a model
    <div id="picking-a-model" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#picking-a-model" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>I tested four small models before settling on Qwen3-1.7B.</p>
<p><strong>TinyLlama 1.1B</strong> (637 MB) was the fastest, but it kept hallucinating customer IDs and dropping required fields — structured output just wasn&rsquo;t reliable. <strong>Phi-3-mini</strong> (1.8 GB) reasons well but wouldn&rsquo;t shut up; simple queries came back as 200-word essays when I needed 20. <strong>Gemma-2B</strong> (1.2 GB) was quick and accurate at classification but weak at function calling — it couldn&rsquo;t consistently produce the <code>&lt;action&gt;</code> tags my tool system needs. <strong>Qwen3-1.7B</strong> (1.1 GB) hit the sweet spot: reliable structured output, precise instruction following, and chain-of-thought support via <code>&lt;think&gt;</code> tags.</p>
<p>Q4_K_M is one of llama.cpp&rsquo;s &ldquo;k-quant&rdquo; formats: weights are stored as 4-bit values in blocks that each carry their own scale, and the M mix keeps a few sensitive tensors at 6-bit. That makes the file about 70% smaller than 16-bit weights for a small quality loss.</p>

<h2 class="relative group">Setting up llama.rn
    <div id="setting-up-llamarn" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#setting-up-llamarn" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>llama.rn wraps llama.cpp for React Native, and installation is the easy part:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl">npm install llama.rn
</span></span><span class="line"><span class="cl"><span class="nb">cd</span> ios <span class="o">&amp;&amp;</span> pod install</span></span></code></pre></div></div>
<p>On Expo, llama.rn is a native module, so it won&rsquo;t run in Expo Go. Add its config plugin and use a development build:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-js" data-lang="js"><span class="line"><span class="cl"><span class="c1">// app.config.js
</span></span></span><span class="line"><span class="cl"><span class="kr">export</span> <span class="k">default</span> <span class="p">{</span>
</span></span><span class="line"><span class="cl">  <span class="nx">expo</span><span class="o">:</span> <span class="p">{</span>
</span></span><span class="line"><span class="cl">    <span class="nx">plugins</span><span class="o">:</span> <span class="p">[</span><span class="s2">&#34;llama.rn&#34;</span><span class="p">],</span>
</span></span><span class="line"><span class="cl">  <span class="p">},</span>
</span></span><span class="line"><span class="cl"><span class="p">};</span></span></span></code></pre></div></div>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl">npx expo install llama.rn
</span></span><span class="line"><span class="cl">npx expo run:ios</span></span></code></pre></div></div>
<p>For production builds, the plugin also adds the two iOS entitlements llama.rn recommends for big models (extended virtual addressing and the increased memory limit).</p>
<p>The model itself downloads on first use:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-typescript" data-lang="typescript"><span class="line"><span class="cl"><span class="kr">const</span> <span class="nx">MODEL_URL</span> <span class="o">=</span> <span class="s2">&#34;https://huggingface.co/unsloth/Qwen3-1.7B-GGUF/resolve/main/Qwen3-1.7B-Q4_K_M.gguf&#34;</span><span class="p">;</span>
</span></span><span class="line"><span class="cl"><span class="kr">const</span> <span class="nx">MODEL_PATH</span> <span class="o">=</span> <span class="nx">FileSystem</span><span class="p">.</span><span class="nx">documentDirectory</span> <span class="o">+</span> <span class="s2">&#34;llama-models/Qwen3-1.7B-Q4_K_M.gguf&#34;</span><span class="p">;</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="kr">const</span> <span class="nx">downloadModel</span> <span class="o">=</span> <span class="kr">async</span> <span class="p">()</span> <span class="o">=&gt;</span> <span class="p">{</span>
</span></span><span class="line"><span class="cl">  <span class="kr">const</span> <span class="nx">downloadResumable</span> <span class="o">=</span> <span class="nx">FileSystem</span><span class="p">.</span><span class="nx">createDownloadResumable</span><span class="p">(</span>
</span></span><span class="line"><span class="cl">    <span class="nx">MODEL_URL</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">    <span class="nx">MODEL_PATH</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">    <span class="p">{},</span>
</span></span><span class="line"><span class="cl">    <span class="p">(</span><span class="nx">progress</span><span class="p">)</span> <span class="o">=&gt;</span> <span class="p">{</span>
</span></span><span class="line"><span class="cl">      <span class="kr">const</span> <span class="nx">pct</span> <span class="o">=</span> <span class="nx">progress</span><span class="p">.</span><span class="nx">totalBytesWritten</span> <span class="o">/</span> <span class="nx">progress</span><span class="p">.</span><span class="nx">totalBytesExpectedToWrite</span><span class="p">;</span>
</span></span><span class="line"><span class="cl">      <span class="nx">setDownloadProgress</span><span class="p">(</span><span class="nx">pct</span><span class="p">);</span>
</span></span><span class="line"><span class="cl">    <span class="p">}</span>
</span></span><span class="line"><span class="cl">  <span class="p">);</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">  <span class="k">await</span> <span class="nx">downloadResumable</span><span class="p">.</span><span class="nx">downloadAsync</span><span class="p">();</span>
</span></span><span class="line"><span class="cl"><span class="p">};</span></span></span></code></pre></div></div>
<p>On WiFi that&rsquo;s 2-3 minutes; on LTE more like 5-8. Once it&rsquo;s on disk, load it with GPU acceleration:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-typescript" data-lang="typescript"><span class="line"><span class="cl"><span class="kr">const</span> <span class="nx">ctx</span> <span class="o">=</span> <span class="k">await</span> <span class="nx">initLlama</span><span class="p">({</span>
</span></span><span class="line"><span class="cl">  <span class="nx">model</span>: <span class="kt">MODEL_PATH</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">  <span class="nx">n_ctx</span>: <span class="kt">8192</span><span class="p">,</span>      <span class="c1">// 8K context window
</span></span></span><span class="line"><span class="cl">  <span class="nx">n_gpu_layers</span>: <span class="kt">99</span><span class="p">,</span> <span class="c1">// Use GPU for all layers
</span></span></span><span class="line"><span class="cl"><span class="p">});</span></span></span></code></pre></div></div>
<p>Don&rsquo;t skip <code>n_gpu_layers: 99</code>. Offloading everything to Metal or OpenCL instead of the CPU is worth about a 5x speedup.</p>

<h2 class="relative group">Streaming inference
    <div id="streaming-inference" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#streaming-inference" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>Nobody wants to stare at a spinner for ten seconds, so responses stream token by token:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-typescript" data-lang="typescript"><span class="line"><span class="cl"><span class="kd">let</span> <span class="nx">fullResponse</span> <span class="o">=</span> <span class="s2">&#34;&#34;</span><span class="p">;</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">await</span> <span class="nx">llamaContext</span><span class="p">.</span><span class="nx">completion</span><span class="p">(</span>
</span></span><span class="line"><span class="cl">  <span class="p">{</span>
</span></span><span class="line"><span class="cl">    <span class="nx">messages</span><span class="o">:</span> <span class="p">[</span>
</span></span><span class="line"><span class="cl">      <span class="p">{</span> <span class="nx">role</span><span class="o">:</span> <span class="s2">&#34;system&#34;</span><span class="p">,</span> <span class="nx">content</span>: <span class="kt">systemPrompt</span> <span class="p">},</span>
</span></span><span class="line"><span class="cl">      <span class="p">{</span> <span class="nx">role</span><span class="o">:</span> <span class="s2">&#34;user&#34;</span><span class="p">,</span> <span class="nx">content</span><span class="o">:</span> <span class="s2">&#34;Create a job for John Smith&#34;</span> <span class="p">}</span>
</span></span><span class="line"><span class="cl">    <span class="p">],</span>
</span></span><span class="line"><span class="cl">    <span class="nx">n_predict</span>: <span class="kt">512</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">    <span class="nx">temperature</span>: <span class="kt">0.7</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">    <span class="nx">top_p</span>: <span class="kt">0.8</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">  <span class="p">},</span>
</span></span><span class="line"><span class="cl">  <span class="p">(</span><span class="nx">data</span><span class="p">)</span> <span class="o">=&gt;</span> <span class="p">{</span>
</span></span><span class="line"><span class="cl">    <span class="c1">// Called for each token
</span></span></span><span class="line"><span class="cl">    <span class="nx">fullResponse</span> <span class="o">+=</span> <span class="nx">data</span><span class="p">.</span><span class="nx">token</span><span class="p">;</span>
</span></span><span class="line"><span class="cl">    <span class="nx">setStreamingText</span><span class="p">(</span><span class="nx">fullResponse</span><span class="p">);</span>
</span></span><span class="line"><span class="cl">  <span class="p">}</span>
</span></span><span class="line"><span class="cl"><span class="p">);</span></span></span></code></pre></div></div>
<p>On modern phones the first token lands in about 200ms (that&rsquo;s prompt processing) and each token after that takes 50-80ms. Compared to anything network-bound, it feels instant.</p>

<h2 class="relative group">Function calling that small models can actually do
    <div id="function-calling-that-small-models-can-actually-do" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#function-calling-that-small-models-can-actually-do" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>The function calling that big cloud models offer leans on JSON schemas, and small models fall over on it — malformed JSON, missing fields, the lot. So I use a simpler XML-based protocol:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-typescript" data-lang="typescript"><span class="line"><span class="cl"><span class="kr">const</span> <span class="nx">systemPrompt</span> <span class="o">=</span> <span class="sb">`You are an AI assistant. When you need to take an action, output:
</span></span></span><span class="line"><span class="cl"><span class="sb">&lt;action&gt;{&#34;type&#34;:&#34;search_customers&#34;,&#34;query&#34;:&#34;Smith&#34;}&lt;/action&gt;
</span></span></span><span class="line"><span class="cl"><span class="sb">
</span></span></span><span class="line"><span class="cl"><span class="sb">Available actions:
</span></span></span><span class="line"><span class="cl"><span class="sb">- search_customers: {&#34;type&#34;:&#34;search_customers&#34;,&#34;query&#34;:&#34;John&#34;}
</span></span></span><span class="line"><span class="cl"><span class="sb">- create_job: {&#34;type&#34;:&#34;create_job&#34;,&#34;customer_id&#34;:5,&#34;alias&#34;:&#34;Living Room&#34;}
</span></span></span><span class="line"><span class="cl"><span class="sb">- update_job: {&#34;type&#34;:&#34;update_job&#34;,&#34;job_id&#34;:&#34;J-0042&#34;,&#34;status&#34;:&#34;quoting&#34;}
</span></span></span><span class="line"><span class="cl"><span class="sb">
</span></span></span><span class="line"><span class="cl"><span class="sb">Rules:
</span></span></span><span class="line"><span class="cl"><span class="sb">1. ONE action tag per response, at the very end
</span></span></span><span class="line"><span class="cl"><span class="sb">2. When searching/creating → end with &lt;action&gt;
</span></span></span><span class="line"><span class="cl"><span class="sb">3. When just chatting → no action tag
</span></span></span><span class="line"><span class="cl"><span class="sb">`</span><span class="p">;</span></span></span></code></pre></div></div>
<p>XML tags won for boring reasons: <code>&lt;action&gt;</code> and <code>&lt;/action&gt;</code> are unambiguous delimiters, a regex is all the parsing you need, the examples in the prompt double as documentation, and it still works when the model wraps the tag in extra chatter.</p>
<p>Parsing is exactly as dumb as it looks:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-typescript" data-lang="typescript"><span class="line"><span class="cl"><span class="kd">function</span> <span class="nx">parseAction</span><span class="p">(</span><span class="nx">text</span>: <span class="kt">string</span><span class="p">)</span><span class="o">:</span> <span class="nx">Action</span> <span class="o">|</span> <span class="kc">null</span> <span class="p">{</span>
</span></span><span class="line"><span class="cl">  <span class="kr">const</span> <span class="nx">match</span> <span class="o">=</span> <span class="nx">text</span><span class="p">.</span><span class="nx">match</span><span class="p">(</span><span class="sr">/&lt;action&gt;([\s\S]*?)&lt;\/action&gt;/</span><span class="p">);</span>
</span></span><span class="line"><span class="cl">  <span class="k">if</span> <span class="p">(</span><span class="o">!</span><span class="nx">match</span><span class="p">)</span> <span class="k">return</span> <span class="kc">null</span><span class="p">;</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">  <span class="k">try</span> <span class="p">{</span>
</span></span><span class="line"><span class="cl">    <span class="k">return</span> <span class="nx">JSON</span><span class="p">.</span><span class="nx">parse</span><span class="p">(</span><span class="nx">match</span><span class="p">[</span><span class="mi">1</span><span class="p">].</span><span class="nx">trim</span><span class="p">());</span>
</span></span><span class="line"><span class="cl">  <span class="p">}</span> <span class="k">catch</span> <span class="p">{</span>
</span></span><span class="line"><span class="cl">    <span class="k">return</span> <span class="kc">null</span><span class="p">;</span>
</span></span><span class="line"><span class="cl">  <span class="p">}</span>
</span></span><span class="line"><span class="cl"><span class="p">}</span></span></span></code></pre></div></div>
<p>When an action comes out, I execute it and feed the result back into the conversation:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-typescript" data-lang="typescript"><span class="line"><span class="cl"><span class="kr">const</span> <span class="nx">action</span> <span class="o">=</span> <span class="nx">parseAction</span><span class="p">(</span><span class="nx">modelResponse</span><span class="p">);</span>
</span></span><span class="line"><span class="cl"><span class="k">if</span> <span class="p">(</span><span class="nx">action</span><span class="p">)</span> <span class="p">{</span>
</span></span><span class="line"><span class="cl">  <span class="kr">const</span> <span class="nx">result</span> <span class="o">=</span> <span class="k">await</span> <span class="nx">executeAction</span><span class="p">(</span><span class="nx">action</span><span class="p">);</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">  <span class="c1">// Inject result as a system message
</span></span></span><span class="line"><span class="cl">  <span class="nx">conversationHistory</span><span class="p">.</span><span class="nx">push</span><span class="p">({</span>
</span></span><span class="line"><span class="cl">    <span class="nx">role</span><span class="o">:</span> <span class="s2">&#34;user&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">    <span class="nx">content</span><span class="o">:</span> <span class="sb">`[TOOL_RESULT] </span><span class="si">${</span><span class="nx">result</span><span class="p">.</span><span class="nx">message</span><span class="si">}</span><span class="err">\</span><span class="sb">n</span><span class="err">\</span><span class="sb">n→ NEXT: Tell the user what happened.`</span>
</span></span><span class="line"><span class="cl">  <span class="p">});</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">  <span class="c1">// Continue generation
</span></span></span><span class="line"><span class="cl">  <span class="k">await</span> <span class="nx">generateNextResponse</span><span class="p">();</span>
</span></span><span class="line"><span class="cl"><span class="p">}</span></span></span></code></pre></div></div>
<p>That&rsquo;s enough for proper multi-step workflows:</p>
<p><strong>User</strong>: &ldquo;Create a job for Smith&rdquo;</p>
<ol>
<li><strong>Model</strong>: <code>&lt;action&gt;{&quot;type&quot;:&quot;search_customers&quot;,&quot;query&quot;:&quot;Smith&quot;}&lt;/action&gt;</code></li>
<li><strong>System</strong>: <code>[TOOL_RESULT] Found: Jane Smith (id:42), Bob Smith (id:89)</code></li>
<li><strong>Model</strong>: &ldquo;I found two Smiths—Jane and Bob. Which one?&rdquo;</li>
<li><strong>User</strong>: &ldquo;Jane&rdquo;</li>
<li><strong>Model</strong>: <code>&lt;action&gt;{&quot;type&quot;:&quot;create_job&quot;,&quot;customer_id&quot;:42}&lt;/action&gt;</code></li>
<li><strong>System</strong>: <code>[TOOL_RESULT] Job J-0073 created</code></li>
<li><strong>Model</strong>: &ldquo;Done! Created job J-0073 for Jane Smith.&rdquo;</li>
</ol>

<h2 class="relative group">Chain-of-thought with &lt;think&gt; tags
    <div id="chain-of-thought-with-think-tags" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#chain-of-thought-with-think-tags" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>Small models do noticeably better when you make them reason out loud. Qwen has a thinking mode built in:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-typescript" data-lang="typescript"><span class="line"><span class="cl"><span class="kr">const</span> <span class="nx">systemPrompt</span> <span class="o">=</span> <span class="sb">`Before each response, wrap your reasoning in &lt;think&gt; tags:
</span></span></span><span class="line"><span class="cl"><span class="sb">
</span></span></span><span class="line"><span class="cl"><span class="sb">&lt;think&gt;
</span></span></span><span class="line"><span class="cl"><span class="sb">INTENT: what does the user want?
</span></span></span><span class="line"><span class="cl"><span class="sb">HAVE: what data do I already have?
</span></span></span><span class="line"><span class="cl"><span class="sb">NEED: what&#39;s still missing?
</span></span></span><span class="line"><span class="cl"><span class="sb">DECISION: call tool | ask user | just respond
</span></span></span><span class="line"><span class="cl"><span class="sb">&lt;/think&gt;
</span></span></span><span class="line"><span class="cl"><span class="sb">
</span></span></span><span class="line"><span class="cl"><span class="sb">Then output your actual response.`</span><span class="p">;</span></span></span></code></pre></div></div>
<p>What the model produces internally looks like this:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">&lt;think&gt;
</span></span><span class="line"><span class="cl">INTENT: create a job
</span></span><span class="line"><span class="cl">HAVE: customer query &#34;Smith&#34;
</span></span><span class="line"><span class="cl">NEED: exact customer_id
</span></span><span class="line"><span class="cl">DECISION: search first
</span></span><span class="line"><span class="cl">&lt;/think&gt;
</span></span><span class="line"><span class="cl">&lt;action&gt;{&#34;type&#34;:&#34;search_customers&#34;,&#34;query&#34;:&#34;Smith&#34;}&lt;/action&gt;</span></span></code></pre></div></div>
<p>I strip the <code>&lt;think&gt;</code> tags before anything hits the UI, but leave them visible during development. The reliability difference is dramatic — the model talks itself through the logic before committing to an action.</p>

<h2 class="relative group">Keeping long conversations inside 8K
    <div id="keeping-long-conversations-inside-8k" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#keeping-long-conversations-inside-8k" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>The 8K context window fills up faster than you&rsquo;d think once tool results start piling in. Around 30 turns, I hit the wall.</p>
<p>My fix is conversation compaction — I ask the model to summarise itself:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-typescript" data-lang="typescript"><span class="line"><span class="cl"><span class="kr">const</span> <span class="nx">compactPrompt</span> <span class="o">=</span> <span class="sb">`Summarize this conversation in under 200 words.
</span></span></span><span class="line"><span class="cl"><span class="sb">Include: user&#39;s goal, customers/jobs created (with IDs), and any unfinished tasks.
</span></span></span><span class="line"><span class="cl"><span class="sb">
</span></span></span><span class="line"><span class="cl"><span class="si">${</span><span class="nx">conversationHistory</span><span class="p">.</span><span class="nx">map</span><span class="p">(</span><span class="nx">m</span> <span class="o">=&gt;</span> <span class="sb">`</span><span class="si">${</span><span class="nx">m</span><span class="p">.</span><span class="nx">role</span><span class="si">}</span><span class="sb">: </span><span class="si">${</span><span class="nx">m</span><span class="p">.</span><span class="nx">content</span><span class="si">}</span><span class="sb">`</span><span class="p">).</span><span class="nx">join</span><span class="p">(</span><span class="s1">&#39;\n\n&#39;</span><span class="p">)</span><span class="si">}</span><span class="sb">`</span><span class="p">;</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="kr">const</span> <span class="nx">summary</span> <span class="o">=</span> <span class="k">await</span> <span class="nx">chat</span><span class="p">([{</span> <span class="nx">role</span><span class="o">:</span> <span class="s2">&#34;user&#34;</span><span class="p">,</span> <span class="nx">content</span>: <span class="kt">compactPrompt</span> <span class="p">}]);</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1">// Replace history with summary
</span></span></span><span class="line"><span class="cl"><span class="nx">conversationHistory</span> <span class="o">=</span> <span class="p">[</span>
</span></span><span class="line"><span class="cl">  <span class="p">{</span> <span class="nx">role</span><span class="o">:</span> <span class="s2">&#34;user&#34;</span><span class="p">,</span> <span class="nx">content</span><span class="o">:</span> <span class="sb">`[CONTEXT SUMMARY]</span><span class="err">\</span><span class="sb">n</span><span class="si">${</span><span class="nx">summary</span><span class="si">}</span><span class="err">\</span><span class="sb">n[END SUMMARY]`</span> <span class="p">}</span>
</span></span><span class="line"><span class="cl"><span class="p">];</span></span></span></code></pre></div></div>
<p>A 50-message conversation squeezes down to about 150 tokens, and the model carries on without losing track of which jobs and customer IDs it created.</p>

<h2 class="relative group">Saving sessions
    <div id="saving-sessions" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#saving-sessions" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>I also save chat sessions to the app&rsquo;s backend. The one idea worth copying is to store every assistant message twice: once cleaned up for display, and once raw, with the <code>&lt;think&gt;</code> and <code>&lt;action&gt;</code> tags left in. History then replays cleanly in the UI, while a resumed session gets the full context back, actions and all. Users switch between past conversations from a history dropdown, the way most chat apps work.</p>

<h2 class="relative group">Benchmarks from production
    <div id="benchmarks-from-production" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#benchmarks-from-production" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p><strong>iPhone 14 Pro</strong> (A16 Bionic, 6GB RAM):</p>
<ul>
<li>Model load: ~3 seconds</li>
<li>First token: 180-220ms</li>
<li>Streaming: 14-16 tokens/sec</li>
<li>Memory: ~1.8 GB</li>
</ul>
<p><strong>Samsung Galaxy S23</strong> (Snapdragon 8 Gen 2, 8GB RAM):</p>
<ul>
<li>Model load: ~4 seconds</li>
<li>First token: 250-300ms</li>
<li>Streaming: 10-12 tokens/sec</li>
<li>Memory: ~2.1 GB</li>
</ul>
<p><strong>iPhone 11</strong> (A13, 4GB RAM):</p>
<ul>
<li>Model load: ~6 seconds</li>
<li>First token: 400-500ms</li>
<li>Streaming: 6-8 tokens/sec</li>
<li>Memory: ~2.2 GB (occasionally crashes on low memory)</li>
</ul>
<p>My rule of thumb: 6GB+ RAM for a smooth experience. It runs on 4GB, but you&rsquo;ll probably need to unload the model when the app backgrounds.</p>

<h2 class="relative group">Memory management
    <div id="memory-management" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#memory-management" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>iOS will happily kill your app for hoarding memory, so I release the model whenever the app leaves the foreground:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-typescript" data-lang="typescript"><span class="line"><span class="cl"><span class="nx">useEffect</span><span class="p">(()</span> <span class="o">=&gt;</span> <span class="p">{</span>
</span></span><span class="line"><span class="cl">  <span class="kr">const</span> <span class="nx">subscription</span> <span class="o">=</span> <span class="nx">AppState</span><span class="p">.</span><span class="nx">addEventListener</span><span class="p">(</span><span class="s2">&#34;change&#34;</span><span class="p">,</span> <span class="p">(</span><span class="nx">nextAppState</span><span class="p">)</span> <span class="o">=&gt;</span> <span class="p">{</span>
</span></span><span class="line"><span class="cl">    <span class="k">if</span> <span class="p">(</span><span class="nx">nextAppState</span> <span class="o">===</span> <span class="s2">&#34;background&#34;</span><span class="p">)</span> <span class="p">{</span>
</span></span><span class="line"><span class="cl">      <span class="nx">llamaContext</span><span class="o">?</span><span class="p">.</span><span class="nx">release</span><span class="p">();</span> <span class="c1">// Free ~2 GB
</span></span></span><span class="line"><span class="cl">    <span class="p">}</span> <span class="k">else</span> <span class="k">if</span> <span class="p">(</span><span class="nx">nextAppState</span> <span class="o">===</span> <span class="s2">&#34;active&#34;</span><span class="p">)</span> <span class="p">{</span>
</span></span><span class="line"><span class="cl">      <span class="nx">loadModel</span><span class="p">();</span> <span class="c1">// Reload when foregrounded
</span></span></span><span class="line"><span class="cl">    <span class="p">}</span>
</span></span><span class="line"><span class="cl">  <span class="p">});</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">  <span class="k">return</span> <span class="p">()</span> <span class="o">=&gt;</span> <span class="nx">subscription</span><span class="p">.</span><span class="nx">remove</span><span class="p">();</span>
</span></span><span class="line"><span class="cl"><span class="p">},</span> <span class="p">[]);</span></span></span></code></pre></div></div>

<h2 class="relative group">Battery impact
    <div id="battery-impact" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#battery-impact" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>On-device inference isn&rsquo;t free, power-wise. From real-world testing:</p>
<table>
	<thead>
			<tr>
					<th>Usage Pattern</th>
					<th>Extra Battery Drain</th>
			</tr>
	</thead>
	<tbody>
			<tr>
					<td>Light (5-10 queries/day)</td>
					<td>&lt;1% per day</td>
			</tr>
			<tr>
					<td>Medium (20-30 queries/day)</td>
					<td>~3-4% per day</td>
			</tr>
			<tr>
					<td>Heavy (50+ queries/day)</td>
					<td>~6-8% per day</td>
			</tr>
	</tbody>
</table>
<p>GPU inference draws more power than CPU but finishes much sooner, and users clearly prefer the instant response, so I&rsquo;ve kept it.</p>

<h2 class="relative group">Two months in, with ~200 beta users
    <div id="two-months-in-with-200-beta-users" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#two-months-in-with-200-beta-users" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>The query breakdown surprised me a little:</p>
<ol>
<li>&ldquo;Create a job for [customer name]&rdquo; - 68% of queries</li>
<li>&ldquo;Find jobs for [customer]&rdquo; - 15%</li>
<li>&ldquo;Update job [ID] to [status]&rdquo; - 9%</li>
<li>General questions about the app - 8%</li>
</ol>
<p>91% of queries succeed on the first try. The failures split into customer not found because the user misspelled the name (4%), context overflow in very long conversations (3%), and the model inventing customer IDs (2%).</p>
<p>That last one used to be much worse. Hallucinated IDs mostly disappeared once I made tool results explicit — returning <code>customer_id=42</code> rather than prose. Small models need that scaffolding spelled out.</p>

<h2 class="relative group">What it costs
    <div id="what-it-costs" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#what-it-costs" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>On-device: $0/month infrastructure, $0 per user beyond the one-time 1.1 GB download, and it scales to as many users as care to install the app.</p>
<p>The cloud version, roughly estimated: an average conversation is ~2,000 tokens, and 200 users at 30 conversations a month is 6,000 conversations — 12M tokens/month. That&rsquo;s about $18/month on a budget cloud model, and $180-360/month on the top-tier ones. Not ruinous at this scale, but the on-device approach paid for itself on day one, and at 10,000 users it would still cost nothing.</p>

<h2 class="relative group">Limitations worth knowing
    <div id="limitations-worth-knowing" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#limitations-worth-knowing" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p><strong>Model capabilities.</strong> Qwen 1.7B is great at structured tasks and terrible at complex multi-hop reasoning, factual recall (it&rsquo;s not a search engine), creative writing, and nuance. Design features around what small models do well and you&rsquo;ll be fine.</p>
<p><strong>Device requirements.</strong> Minimum is 3 GB RAM and 2 GB free storage; realistically you want 6 GB RAM, a 64-bit processor, and GPU support. Older devices like the iPhone 8 or Galaxy S9 struggle — do feature detection and fall back gracefully.</p>
<p><strong>Model updates.</strong> Shipping a new model means users re-download 1.1 GB. I version the model in the storage path:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-typescript" data-lang="typescript"><span class="line"><span class="cl"><span class="kr">const</span> <span class="nx">MODEL_PATH</span> <span class="o">=</span> <span class="sb">`</span><span class="si">${</span><span class="nx">FileSystem</span><span class="p">.</span><span class="nx">documentDirectory</span><span class="si">}</span><span class="sb">llama-models/v2/Qwen3-1.7B-Q4_K_M.gguf`</span><span class="p">;</span></span></span></code></pre></div></div>
<p>Bump the path for each new model version. Old models clean themselves up on uninstall.</p>

<h2 class="relative group">iOS vs Android
    <div id="ios-vs-android" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#ios-vs-android" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>iOS on Metal is around 20% faster than Android, manages memory better, and backs the model up to iCloud by default (disable that with <code>NSURLIsExcludedFromBackupKey</code> unless you want to eat users&rsquo; backup quota). Android varies a lot more across devices — llama.rn&rsquo;s GPU path is OpenCL on Qualcomm Adreno 700-series GPUs, and phones without one run on the CPU — and storage isn&rsquo;t backed up automatically. Test on both; the gap between a good and bad Android device is bigger than the gap between platforms.</p>

<h2 class="relative group">Debugging tips
    <div id="debugging-tips" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#debugging-tips" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p><strong>1. Enable verbose logging</strong></p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-typescript" data-lang="typescript"><span class="line"><span class="cl"><span class="kr">const</span> <span class="nx">ctx</span> <span class="o">=</span> <span class="k">await</span> <span class="nx">initLlama</span><span class="p">({</span>
</span></span><span class="line"><span class="cl">  <span class="nx">model</span>: <span class="kt">MODEL_PATH</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">  <span class="nx">n_ctx</span>: <span class="kt">8192</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">  <span class="nx">n_gpu_layers</span>: <span class="kt">99</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">  <span class="nx">verbose</span>: <span class="kt">true</span><span class="p">,</span> <span class="c1">// Logs every token + timings
</span></span></span><span class="line"><span class="cl"><span class="p">});</span></span></span></code></pre></div></div>
<p><strong>2. Track token counts</strong></p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-typescript" data-lang="typescript"><span class="line"><span class="cl"><span class="kr">const</span> <span class="nx">tokens</span> <span class="o">=</span> <span class="nx">fullResponse</span><span class="p">.</span><span class="nx">split</span><span class="p">(</span><span class="sr">/\s+/</span><span class="p">).</span><span class="nx">length</span> <span class="o">*</span> <span class="mf">1.3</span><span class="p">;</span> <span class="c1">// Rough estimate
</span></span></span><span class="line"><span class="cl"><span class="nx">console</span><span class="p">.</span><span class="nx">log</span><span class="p">(</span><span class="sb">`Generated </span><span class="si">${</span><span class="nx">tokens</span><span class="si">}</span><span class="sb"> tokens in </span><span class="si">${</span><span class="nx">duration</span><span class="si">}</span><span class="sb">ms`</span><span class="p">);</span></span></span></code></pre></div></div>
<p><strong>3. Monitor memory</strong></p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-typescript" data-lang="typescript"><span class="line"><span class="cl"><span class="kr">import</span> <span class="p">{</span> <span class="nx">MemoryInfo</span> <span class="p">}</span> <span class="kr">from</span> <span class="s1">&#39;react-native-device-info&#39;</span><span class="p">;</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="kr">const</span> <span class="nx">memoryUsage</span> <span class="o">=</span> <span class="k">await</span> <span class="nx">MemoryInfo</span><span class="p">.</span><span class="nx">getUsedMemory</span><span class="p">();</span>
</span></span><span class="line"><span class="cl"><span class="nx">console</span><span class="p">.</span><span class="nx">log</span><span class="p">(</span><span class="sb">`Memory: </span><span class="si">${</span><span class="p">(</span><span class="nx">memoryUsage</span> <span class="o">/</span> <span class="mi">1024</span> <span class="o">/</span> <span class="mi">1024</span><span class="p">).</span><span class="nx">toFixed</span><span class="p">(</span><span class="mi">0</span><span class="p">)</span><span class="si">}</span><span class="sb"> MB`</span><span class="p">);</span></span></span></code></pre></div></div>
<p><strong>4. Test offline</strong></p>
<p>Flip on airplane mode and use the app properly. The model should load from cache, inference should complete, and only the backend persistence should fail — gracefully.</p>

<h2 class="relative group">Where I want to take it
    <div id="where-i-want-to-take-it" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#where-i-want-to-take-it" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>llama.cpp already supports vision models, and &ldquo;here&rsquo;s a photo from the site, add it to the job&rdquo; is an obvious fit for this app. I&rsquo;d also like to try fine-tuning a LoRA adapter on past job descriptions — my guess is 20-30% better accuracy on domain queries. Further out: an on-device speech-to-text model for a fully offline voice interface, and maybe federated learning so the model improves across users without any raw data leaving their phones.</p>

<h2 class="relative group">Should you build this?
    <div id="should-you-build-this" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#should-you-build-this" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>If you&rsquo;re building a business app with sensitive data, offline-first workflows, or structured tasks like classification and data entry — and especially if API costs at scale worry you — yes, this works today.</p>
<p>If you need open-ended chat, use one of the big cloud models. Same if the task is knowledge-heavy (small models simply don&rsquo;t know much), if you must support low-end devices, or if your domain knowledge changes faster than you can ship model updates.</p>
<p>The whole thing took me about 12 hours to build. It costs $0 a month to run, works on a plane, and the customer data never leaves the customer&rsquo;s hand. Hard to argue with.</p>
<hr>
<p><em>Questions? Feedback? I&rsquo;m <a href="https://twitter.com/jaredlynskey"  target="_blank" rel="noreferrer">@jaredlynskey</a>. The llama.rn library is at <a href="https://github.com/mybigday/llama.rn"  target="_blank" rel="noreferrer">github.com/mybigday/llama.rn</a>.</em></p>
]]></content:encoded>
      <media:content url="https://jared.lynskey.co.nz/en/posts/2026/2026-01-30-on-device-ai/featured.jpg" medium="image" type="image/jpeg" />
    </item>
  </channel>
</rss>
