<?xml version="1.0" encoding="utf-8" standalone="yes"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:media="http://search.yahoo.com/mrss/">
  <channel>
    <title>Qwen · Jared Lynskey</title>
    <link>https://jared.lynskey.co.nz/en/tags/qwen/</link>
    <description>Short stories of value on software engineering and engineering leadership — performance reviews, hiring and scaling teams, CI/CD, DevOps monitoring and agentic AI. Written from Seoul.</description>
    <generator>Hugo</generator>
    <language>en</language>
    <copyright>© 2026 Jared Lynskey</copyright>
    <lastBuildDate>Tue, 27 Oct 2026 00:00:00 +1300</lastBuildDate>
    <atom:link href="https://jared.lynskey.co.nz/en/tags/qwen/index.xml" rel="self" type="application/rss+xml" />
    <item>
      <title>On-Device LLM in React Native: Running Qwen Offline With llama.rn and llama.cpp</title>
      <link>https://jared.lynskey.co.nz/en/posts/2026/2026-01-30-on-device-ai/</link>
      <guid isPermaLink="true">https://jared.lynskey.co.nz/en/posts/2026/2026-01-30-on-device-ai/</guid>
      <pubDate>Fri, 30 Jan 2026 00:00:00 +0000</pubDate>
      <dc:creator>Jared Lynskey</dc:creator>
      <category>React Native</category>
      <category>AI</category>
      <category>LLM</category>
      <category>llama.cpp</category>
      <category>Qwen</category>
      <category>On-Device AI</category>
      <category>Privacy</category>
      <category>Mobile Development</category>
      <description>How I run a small LLM entirely on the phone in a React Native and Expo app: llama.rn, a Qwen GGUF model at Q4_K_M, streaming, tool calling a small model can manage, and real numbers from an iPhone and a Galaxy S23.</description>
      <content:encoded><![CDATA[<p>The AI assistant in an estimating app I build runs entirely on the phone. Nothing leaves the device. Users can create jobs, search customers, and manage projects by typing plain English, and it all keeps working in airplane mode.</p>
<p>This post walks through how I built it with <strong>llama.rn</strong> (React Native bindings for llama.cpp) and <strong>Qwen 1.7B</strong>, a small model that turned out to be more capable than I expected.</p>
<p>The short version: install llama.rn, download a 4-bit GGUF model (about 1 GB) the first time the app opens, load it with <code>initLlama</code>, and stream completions from it. On an iPhone 14 Pro that runs at about 15 tokens a second with no network at all. The rest of this post is what it took to make that useful: picking the model, getting tool calls out of it reliably, and keeping memory in check.</p>
<blockquote><p><strong>📝 Update:</strong> This implementation now uses <a href="https://huggingface.co/unsloth/Qwen3.5-2B-GGUF"  target="_blank" rel="noreferrer">Qwen3.5-2B-GGUF</a> instead of the original Qwen3-1.7B. I&rsquo;ve also stopped using the <code>/no_think</code> message hack and now disable thinking mode properly via the <code>chat_template_kwargs</code> parameter:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="cl"><span class="n">chat_response</span> <span class="o">=</span> <span class="n">client</span><span class="o">.</span><span class="n">chat</span><span class="o">.</span><span class="n">completions</span><span class="o">.</span><span class="n">create</span><span class="p">(</span>
</span></span><span class="line"><span class="cl">    <span class="n">model</span><span class="o">=</span><span class="s2">&#34;Qwen/Qwen3.5-27B&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">    <span class="n">messages</span><span class="o">=</span><span class="n">messages</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">    <span class="n">max_tokens</span><span class="o">=</span><span class="mi">32768</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">    <span class="n">temperature</span><span class="o">=</span><span class="mf">0.7</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">    <span class="n">top_p</span><span class="o">=</span><span class="mf">0.8</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">    <span class="n">presence_penalty</span><span class="o">=</span><span class="mf">1.5</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">    <span class="n">extra_body</span><span class="o">=</span><span class="p">{</span>
</span></span><span class="line"><span class="cl">        <span class="s2">&#34;top_k&#34;</span><span class="p">:</span> <span class="mi">20</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">        <span class="s2">&#34;chat_template_kwargs&#34;</span><span class="p">:</span> <span class="p">{</span><span class="s2">&#34;enable_thinking&#34;</span><span class="p">:</span> <span class="kc">False</span><span class="p">},</span>
</span></span><span class="line"><span class="cl">    <span class="p">},</span>
</span></span><span class="line"><span class="cl"><span class="p">)</span></span></span></code></pre></div></div>
<p>Much cleaner than smuggling control flags into the prompt.</p>
</blockquote><figure><div class="film-pair">
  <div class="mc-film film-dark">
  <video
    class="my-0 rounded-md"
    controls playsinline preload="none"
    poster="https://jared.lynskey.co.nz/en/posts/2026/2026-01-30-on-device-ai/on-device-ai-film-poster.jpg" style="background:url('data:image/webp;base64,UklGRnAAAABXRUJQVlA4IGQAAACwBACdASogABIAPw1uqUemIyIYBABjEMT0gAFRZzp3IyiSND5CxppNutGcCYkAAP7ixmKvS+0H/fAThx4cbifiH8PzygNYbl6sJuQFubAi+gzkxN3eXPGHfroT5K82AG+x8AAA') center/cover no-repeat"
    width="1280"
    height="720"
    aria-label="A short narrated film of this post">
    <source src="https://jared.lynskey.co.nz/en/posts/2026/2026-01-30-on-device-ai/on-device-ai-film.webm" type="video/webm">
    <track kind="subtitles" src="https://jared.lynskey.co.nz/en/posts/2026/2026-01-30-on-device-ai/on-device-ai-film_en.vtt" srclang="en" label="English">
    <track kind="subtitles" src="https://jared.lynskey.co.nz/en/posts/2026/2026-01-30-on-device-ai/on-device-ai-film_ja.vtt" srclang="ja" label="日本語">
    <track kind="subtitles" src="https://jared.lynskey.co.nz/en/posts/2026/2026-01-30-on-device-ai/on-device-ai-film_ko.vtt" srclang="ko" label="한국어">
    <track kind="subtitles" src="https://jared.lynskey.co.nz/en/posts/2026/2026-01-30-on-device-ai/on-device-ai-film_vi.vtt" srclang="vi" label="Tiếng Việt">
    <track kind="subtitles" src="https://jared.lynskey.co.nz/en/posts/2026/2026-01-30-on-device-ai/on-device-ai-film_zh-cn.vtt" srclang="zh-cn" label="简体中文">
  </video>
  </div>
  <div class="mc-film film-light">
  <video
    class="my-0 rounded-md"
    controls playsinline preload="none"
    poster="https://jared.lynskey.co.nz/en/posts/2026/2026-01-30-on-device-ai/on-device-ai-film-light-poster.jpg" style="background:url('data:image/webp;base64,UklGRoIAAABXRUJQVlA4IHYAAACwBACdASogABIAPw1uqUemIyIYBABjEMT0gAFRZY2mogaLUpH79C93CfzUrxKAAP7vul6EGA0ytD4npxsGq682/6nTvmGx0/XY0S/eONnq1STOt/i7tLHFRM4DTibTbGueLxPoaxVrlM6FoYdfCU8gCNqwAAAA') center/cover no-repeat"
    width="1280"
    height="720"
    aria-label="A short narrated film of this post">
    <source src="https://jared.lynskey.co.nz/en/posts/2026/2026-01-30-on-device-ai/on-device-ai-film-light.webm" type="video/webm">
    <track kind="subtitles" src="https://jared.lynskey.co.nz/en/posts/2026/2026-01-30-on-device-ai/on-device-ai-film_en.vtt" srclang="en" label="English">
    <track kind="subtitles" src="https://jared.lynskey.co.nz/en/posts/2026/2026-01-30-on-device-ai/on-device-ai-film_ja.vtt" srclang="ja" label="日本語">
    <track kind="subtitles" src="https://jared.lynskey.co.nz/en/posts/2026/2026-01-30-on-device-ai/on-device-ai-film_ko.vtt" srclang="ko" label="한국어">
    <track kind="subtitles" src="https://jared.lynskey.co.nz/en/posts/2026/2026-01-30-on-device-ai/on-device-ai-film_vi.vtt" srclang="vi" label="Tiếng Việt">
    <track kind="subtitles" src="https://jared.lynskey.co.nz/en/posts/2026/2026-01-30-on-device-ai/on-device-ai-film_zh-cn.vtt" srclang="zh-cn" label="简体中文">
  </video>
  </div>
  <script>window.mcFilm && window.mcFilm()</script></div><figcaption>This post as a nine-minute film, if you&rsquo;d rather watch than read.</figcaption>
</figure>


<h2 class="relative group">Why I didn&rsquo;t want cloud AI for this
    <div id="why-i-didnt-want-cloud-ai-for-this" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#why-i-didnt-want-cloud-ai-for-this" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>The standard recipe for an AI feature in a mobile app is: user types a message, app sends it to a cloud model provider, response comes back, bill ticks up.</p>
<p>For an app that tradespeople use to run their business, that recipe has problems. Customer names, addresses, and phone numbers go to a third party. Project details, pricing, and notes sit on someone else&rsquo;s servers. There are GDPR and data-residency questions I&rsquo;d rather not spend time on. And the API bill grows with every user.</p>
<p>The alternative is to run the model on the user&rsquo;s phone, so that&rsquo;s what I did.</p>

<h2 class="relative group">Why this is finally practical
    <div id="why-this-is-finally-practical" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#why-this-is-finally-practical" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>A year earlier I wouldn&rsquo;t have bothered, but three things changed roughly at once. Quantised models got small — Qwen 1.7B at Q4_K_M is 1.1 GB, smaller than most games, and it downloads once and lives in app storage. Mobile GPUs got fast — llama.cpp uses Metal on iOS and OpenCL on recent Qualcomm Android GPUs, and on an iPhone 14 Pro I see about 15 tokens a second, plenty for real-time streaming. And small models got genuinely useful: Qwen 1.7B follows structured instructions, parses JSON, and chains multi-step reasoning. That&rsquo;s exactly the profile business logic needs. Nobody&rsquo;s asking it to write poetry.</p>

<h2 class="relative group">Picking a model
    <div id="picking-a-model" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#picking-a-model" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>I tested four small models before settling on Qwen3-1.7B.</p>
<p><strong>TinyLlama 1.1B</strong> (637 MB) was the fastest, but it kept hallucinating customer IDs and dropping required fields — structured output just wasn&rsquo;t reliable. <strong>Phi-3-mini</strong> (1.8 GB) reasons well but wouldn&rsquo;t shut up; simple queries came back as 200-word essays when I needed 20. <strong>Gemma-2B</strong> (1.2 GB) was quick and accurate at classification but weak at function calling — it couldn&rsquo;t consistently produce the <code>&lt;action&gt;</code> tags my tool system needs. <strong>Qwen3-1.7B</strong> (1.1 GB) hit the sweet spot: reliable structured output, precise instruction following, and chain-of-thought support via <code>&lt;think&gt;</code> tags.</p>
<p>Q4_K_M is one of llama.cpp&rsquo;s &ldquo;k-quant&rdquo; formats: weights are stored as 4-bit values in blocks that each carry their own scale, and the M mix keeps a few sensitive tensors at 6-bit. That makes the file about 70% smaller than 16-bit weights for a small quality loss.</p>

<h2 class="relative group">Setting up llama.rn
    <div id="setting-up-llamarn" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#setting-up-llamarn" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>llama.rn wraps llama.cpp for React Native, and installation is the easy part:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl">npm install llama.rn
</span></span><span class="line"><span class="cl"><span class="nb">cd</span> ios <span class="o">&amp;&amp;</span> pod install</span></span></code></pre></div></div>
<p>On Expo, llama.rn is a native module, so it won&rsquo;t run in Expo Go. Add its config plugin and use a development build:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-js" data-lang="js"><span class="line"><span class="cl"><span class="c1">// app.config.js
</span></span></span><span class="line"><span class="cl"><span class="kr">export</span> <span class="k">default</span> <span class="p">{</span>
</span></span><span class="line"><span class="cl">  <span class="nx">expo</span><span class="o">:</span> <span class="p">{</span>
</span></span><span class="line"><span class="cl">    <span class="nx">plugins</span><span class="o">:</span> <span class="p">[</span><span class="s2">&#34;llama.rn&#34;</span><span class="p">],</span>
</span></span><span class="line"><span class="cl">  <span class="p">},</span>
</span></span><span class="line"><span class="cl"><span class="p">};</span></span></span></code></pre></div></div>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl">npx expo install llama.rn
</span></span><span class="line"><span class="cl">npx expo run:ios</span></span></code></pre></div></div>
<p>For production builds, the plugin also adds the two iOS entitlements llama.rn recommends for big models (extended virtual addressing and the increased memory limit).</p>
<p>The model itself downloads on first use:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-typescript" data-lang="typescript"><span class="line"><span class="cl"><span class="kr">const</span> <span class="nx">MODEL_URL</span> <span class="o">=</span> <span class="s2">&#34;https://huggingface.co/unsloth/Qwen3-1.7B-GGUF/resolve/main/Qwen3-1.7B-Q4_K_M.gguf&#34;</span><span class="p">;</span>
</span></span><span class="line"><span class="cl"><span class="kr">const</span> <span class="nx">MODEL_PATH</span> <span class="o">=</span> <span class="nx">FileSystem</span><span class="p">.</span><span class="nx">documentDirectory</span> <span class="o">+</span> <span class="s2">&#34;llama-models/Qwen3-1.7B-Q4_K_M.gguf&#34;</span><span class="p">;</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="kr">const</span> <span class="nx">downloadModel</span> <span class="o">=</span> <span class="kr">async</span> <span class="p">()</span> <span class="o">=&gt;</span> <span class="p">{</span>
</span></span><span class="line"><span class="cl">  <span class="kr">const</span> <span class="nx">downloadResumable</span> <span class="o">=</span> <span class="nx">FileSystem</span><span class="p">.</span><span class="nx">createDownloadResumable</span><span class="p">(</span>
</span></span><span class="line"><span class="cl">    <span class="nx">MODEL_URL</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">    <span class="nx">MODEL_PATH</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">    <span class="p">{},</span>
</span></span><span class="line"><span class="cl">    <span class="p">(</span><span class="nx">progress</span><span class="p">)</span> <span class="o">=&gt;</span> <span class="p">{</span>
</span></span><span class="line"><span class="cl">      <span class="kr">const</span> <span class="nx">pct</span> <span class="o">=</span> <span class="nx">progress</span><span class="p">.</span><span class="nx">totalBytesWritten</span> <span class="o">/</span> <span class="nx">progress</span><span class="p">.</span><span class="nx">totalBytesExpectedToWrite</span><span class="p">;</span>
</span></span><span class="line"><span class="cl">      <span class="nx">setDownloadProgress</span><span class="p">(</span><span class="nx">pct</span><span class="p">);</span>
</span></span><span class="line"><span class="cl">    <span class="p">}</span>
</span></span><span class="line"><span class="cl">  <span class="p">);</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">  <span class="k">await</span> <span class="nx">downloadResumable</span><span class="p">.</span><span class="nx">downloadAsync</span><span class="p">();</span>
</span></span><span class="line"><span class="cl"><span class="p">};</span></span></span></code></pre></div></div>
<p>On WiFi that&rsquo;s 2-3 minutes; on LTE more like 5-8. Once it&rsquo;s on disk, load it with GPU acceleration:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-typescript" data-lang="typescript"><span class="line"><span class="cl"><span class="kr">const</span> <span class="nx">ctx</span> <span class="o">=</span> <span class="k">await</span> <span class="nx">initLlama</span><span class="p">({</span>
</span></span><span class="line"><span class="cl">  <span class="nx">model</span>: <span class="kt">MODEL_PATH</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">  <span class="nx">n_ctx</span>: <span class="kt">8192</span><span class="p">,</span>      <span class="c1">// 8K context window
</span></span></span><span class="line"><span class="cl">  <span class="nx">n_gpu_layers</span>: <span class="kt">99</span><span class="p">,</span> <span class="c1">// Use GPU for all layers
</span></span></span><span class="line"><span class="cl"><span class="p">});</span></span></span></code></pre></div></div>
<p>Don&rsquo;t skip <code>n_gpu_layers: 99</code>. Offloading everything to Metal or OpenCL instead of the CPU is worth about a 5x speedup.</p>

<h2 class="relative group">Streaming inference
    <div id="streaming-inference" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#streaming-inference" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>Nobody wants to stare at a spinner for ten seconds, so responses stream token by token:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-typescript" data-lang="typescript"><span class="line"><span class="cl"><span class="kd">let</span> <span class="nx">fullResponse</span> <span class="o">=</span> <span class="s2">&#34;&#34;</span><span class="p">;</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">await</span> <span class="nx">llamaContext</span><span class="p">.</span><span class="nx">completion</span><span class="p">(</span>
</span></span><span class="line"><span class="cl">  <span class="p">{</span>
</span></span><span class="line"><span class="cl">    <span class="nx">messages</span><span class="o">:</span> <span class="p">[</span>
</span></span><span class="line"><span class="cl">      <span class="p">{</span> <span class="nx">role</span><span class="o">:</span> <span class="s2">&#34;system&#34;</span><span class="p">,</span> <span class="nx">content</span>: <span class="kt">systemPrompt</span> <span class="p">},</span>
</span></span><span class="line"><span class="cl">      <span class="p">{</span> <span class="nx">role</span><span class="o">:</span> <span class="s2">&#34;user&#34;</span><span class="p">,</span> <span class="nx">content</span><span class="o">:</span> <span class="s2">&#34;Create a job for John Smith&#34;</span> <span class="p">}</span>
</span></span><span class="line"><span class="cl">    <span class="p">],</span>
</span></span><span class="line"><span class="cl">    <span class="nx">n_predict</span>: <span class="kt">512</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">    <span class="nx">temperature</span>: <span class="kt">0.7</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">    <span class="nx">top_p</span>: <span class="kt">0.8</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">  <span class="p">},</span>
</span></span><span class="line"><span class="cl">  <span class="p">(</span><span class="nx">data</span><span class="p">)</span> <span class="o">=&gt;</span> <span class="p">{</span>
</span></span><span class="line"><span class="cl">    <span class="c1">// Called for each token
</span></span></span><span class="line"><span class="cl">    <span class="nx">fullResponse</span> <span class="o">+=</span> <span class="nx">data</span><span class="p">.</span><span class="nx">token</span><span class="p">;</span>
</span></span><span class="line"><span class="cl">    <span class="nx">setStreamingText</span><span class="p">(</span><span class="nx">fullResponse</span><span class="p">);</span>
</span></span><span class="line"><span class="cl">  <span class="p">}</span>
</span></span><span class="line"><span class="cl"><span class="p">);</span></span></span></code></pre></div></div>
<p>On modern phones the first token lands in about 200ms (that&rsquo;s prompt processing) and each token after that takes 50-80ms. Compared to anything network-bound, it feels instant.</p>

<h2 class="relative group">Function calling that small models can actually do
    <div id="function-calling-that-small-models-can-actually-do" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#function-calling-that-small-models-can-actually-do" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>The function calling that big cloud models offer leans on JSON schemas, and small models fall over on it — malformed JSON, missing fields, the lot. So I use a simpler XML-based protocol:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-typescript" data-lang="typescript"><span class="line"><span class="cl"><span class="kr">const</span> <span class="nx">systemPrompt</span> <span class="o">=</span> <span class="sb">`You are an AI assistant. When you need to take an action, output:
</span></span></span><span class="line"><span class="cl"><span class="sb">&lt;action&gt;{&#34;type&#34;:&#34;search_customers&#34;,&#34;query&#34;:&#34;Smith&#34;}&lt;/action&gt;
</span></span></span><span class="line"><span class="cl"><span class="sb">
</span></span></span><span class="line"><span class="cl"><span class="sb">Available actions:
</span></span></span><span class="line"><span class="cl"><span class="sb">- search_customers: {&#34;type&#34;:&#34;search_customers&#34;,&#34;query&#34;:&#34;John&#34;}
</span></span></span><span class="line"><span class="cl"><span class="sb">- create_job: {&#34;type&#34;:&#34;create_job&#34;,&#34;customer_id&#34;:5,&#34;alias&#34;:&#34;Living Room&#34;}
</span></span></span><span class="line"><span class="cl"><span class="sb">- update_job: {&#34;type&#34;:&#34;update_job&#34;,&#34;job_id&#34;:&#34;J-0042&#34;,&#34;status&#34;:&#34;quoting&#34;}
</span></span></span><span class="line"><span class="cl"><span class="sb">
</span></span></span><span class="line"><span class="cl"><span class="sb">Rules:
</span></span></span><span class="line"><span class="cl"><span class="sb">1. ONE action tag per response, at the very end
</span></span></span><span class="line"><span class="cl"><span class="sb">2. When searching/creating → end with &lt;action&gt;
</span></span></span><span class="line"><span class="cl"><span class="sb">3. When just chatting → no action tag
</span></span></span><span class="line"><span class="cl"><span class="sb">`</span><span class="p">;</span></span></span></code></pre></div></div>
<p>XML tags won for boring reasons: <code>&lt;action&gt;</code> and <code>&lt;/action&gt;</code> are unambiguous delimiters, a regex is all the parsing you need, the examples in the prompt double as documentation, and it still works when the model wraps the tag in extra chatter.</p>
<p>Parsing is exactly as dumb as it looks:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-typescript" data-lang="typescript"><span class="line"><span class="cl"><span class="kd">function</span> <span class="nx">parseAction</span><span class="p">(</span><span class="nx">text</span>: <span class="kt">string</span><span class="p">)</span><span class="o">:</span> <span class="nx">Action</span> <span class="o">|</span> <span class="kc">null</span> <span class="p">{</span>
</span></span><span class="line"><span class="cl">  <span class="kr">const</span> <span class="nx">match</span> <span class="o">=</span> <span class="nx">text</span><span class="p">.</span><span class="nx">match</span><span class="p">(</span><span class="sr">/&lt;action&gt;([\s\S]*?)&lt;\/action&gt;/</span><span class="p">);</span>
</span></span><span class="line"><span class="cl">  <span class="k">if</span> <span class="p">(</span><span class="o">!</span><span class="nx">match</span><span class="p">)</span> <span class="k">return</span> <span class="kc">null</span><span class="p">;</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">  <span class="k">try</span> <span class="p">{</span>
</span></span><span class="line"><span class="cl">    <span class="k">return</span> <span class="nx">JSON</span><span class="p">.</span><span class="nx">parse</span><span class="p">(</span><span class="nx">match</span><span class="p">[</span><span class="mi">1</span><span class="p">].</span><span class="nx">trim</span><span class="p">());</span>
</span></span><span class="line"><span class="cl">  <span class="p">}</span> <span class="k">catch</span> <span class="p">{</span>
</span></span><span class="line"><span class="cl">    <span class="k">return</span> <span class="kc">null</span><span class="p">;</span>
</span></span><span class="line"><span class="cl">  <span class="p">}</span>
</span></span><span class="line"><span class="cl"><span class="p">}</span></span></span></code></pre></div></div>
<p>When an action comes out, I execute it and feed the result back into the conversation:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-typescript" data-lang="typescript"><span class="line"><span class="cl"><span class="kr">const</span> <span class="nx">action</span> <span class="o">=</span> <span class="nx">parseAction</span><span class="p">(</span><span class="nx">modelResponse</span><span class="p">);</span>
</span></span><span class="line"><span class="cl"><span class="k">if</span> <span class="p">(</span><span class="nx">action</span><span class="p">)</span> <span class="p">{</span>
</span></span><span class="line"><span class="cl">  <span class="kr">const</span> <span class="nx">result</span> <span class="o">=</span> <span class="k">await</span> <span class="nx">executeAction</span><span class="p">(</span><span class="nx">action</span><span class="p">);</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">  <span class="c1">// Inject result as a system message
</span></span></span><span class="line"><span class="cl">  <span class="nx">conversationHistory</span><span class="p">.</span><span class="nx">push</span><span class="p">({</span>
</span></span><span class="line"><span class="cl">    <span class="nx">role</span><span class="o">:</span> <span class="s2">&#34;user&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">    <span class="nx">content</span><span class="o">:</span> <span class="sb">`[TOOL_RESULT] </span><span class="si">${</span><span class="nx">result</span><span class="p">.</span><span class="nx">message</span><span class="si">}</span><span class="err">\</span><span class="sb">n</span><span class="err">\</span><span class="sb">n→ NEXT: Tell the user what happened.`</span>
</span></span><span class="line"><span class="cl">  <span class="p">});</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">  <span class="c1">// Continue generation
</span></span></span><span class="line"><span class="cl">  <span class="k">await</span> <span class="nx">generateNextResponse</span><span class="p">();</span>
</span></span><span class="line"><span class="cl"><span class="p">}</span></span></span></code></pre></div></div>
<p>That&rsquo;s enough for proper multi-step workflows:</p>
<p><strong>User</strong>: &ldquo;Create a job for Smith&rdquo;</p>
<ol>
<li><strong>Model</strong>: <code>&lt;action&gt;{&quot;type&quot;:&quot;search_customers&quot;,&quot;query&quot;:&quot;Smith&quot;}&lt;/action&gt;</code></li>
<li><strong>System</strong>: <code>[TOOL_RESULT] Found: Jane Smith (id:42), Bob Smith (id:89)</code></li>
<li><strong>Model</strong>: &ldquo;I found two Smiths—Jane and Bob. Which one?&rdquo;</li>
<li><strong>User</strong>: &ldquo;Jane&rdquo;</li>
<li><strong>Model</strong>: <code>&lt;action&gt;{&quot;type&quot;:&quot;create_job&quot;,&quot;customer_id&quot;:42}&lt;/action&gt;</code></li>
<li><strong>System</strong>: <code>[TOOL_RESULT] Job J-0073 created</code></li>
<li><strong>Model</strong>: &ldquo;Done! Created job J-0073 for Jane Smith.&rdquo;</li>
</ol>

<h2 class="relative group">Chain-of-thought with &lt;think&gt; tags
    <div id="chain-of-thought-with-think-tags" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#chain-of-thought-with-think-tags" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>Small models do noticeably better when you make them reason out loud. Qwen has a thinking mode built in:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-typescript" data-lang="typescript"><span class="line"><span class="cl"><span class="kr">const</span> <span class="nx">systemPrompt</span> <span class="o">=</span> <span class="sb">`Before each response, wrap your reasoning in &lt;think&gt; tags:
</span></span></span><span class="line"><span class="cl"><span class="sb">
</span></span></span><span class="line"><span class="cl"><span class="sb">&lt;think&gt;
</span></span></span><span class="line"><span class="cl"><span class="sb">INTENT: what does the user want?
</span></span></span><span class="line"><span class="cl"><span class="sb">HAVE: what data do I already have?
</span></span></span><span class="line"><span class="cl"><span class="sb">NEED: what&#39;s still missing?
</span></span></span><span class="line"><span class="cl"><span class="sb">DECISION: call tool | ask user | just respond
</span></span></span><span class="line"><span class="cl"><span class="sb">&lt;/think&gt;
</span></span></span><span class="line"><span class="cl"><span class="sb">
</span></span></span><span class="line"><span class="cl"><span class="sb">Then output your actual response.`</span><span class="p">;</span></span></span></code></pre></div></div>
<p>What the model produces internally looks like this:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">&lt;think&gt;
</span></span><span class="line"><span class="cl">INTENT: create a job
</span></span><span class="line"><span class="cl">HAVE: customer query &#34;Smith&#34;
</span></span><span class="line"><span class="cl">NEED: exact customer_id
</span></span><span class="line"><span class="cl">DECISION: search first
</span></span><span class="line"><span class="cl">&lt;/think&gt;
</span></span><span class="line"><span class="cl">&lt;action&gt;{&#34;type&#34;:&#34;search_customers&#34;,&#34;query&#34;:&#34;Smith&#34;}&lt;/action&gt;</span></span></code></pre></div></div>
<p>I strip the <code>&lt;think&gt;</code> tags before anything hits the UI, but leave them visible during development. The reliability difference is dramatic — the model talks itself through the logic before committing to an action.</p>

<h2 class="relative group">Keeping long conversations inside 8K
    <div id="keeping-long-conversations-inside-8k" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#keeping-long-conversations-inside-8k" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>The 8K context window fills up faster than you&rsquo;d think once tool results start piling in. Around 30 turns, I hit the wall.</p>
<p>My fix is conversation compaction — I ask the model to summarise itself:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-typescript" data-lang="typescript"><span class="line"><span class="cl"><span class="kr">const</span> <span class="nx">compactPrompt</span> <span class="o">=</span> <span class="sb">`Summarize this conversation in under 200 words.
</span></span></span><span class="line"><span class="cl"><span class="sb">Include: user&#39;s goal, customers/jobs created (with IDs), and any unfinished tasks.
</span></span></span><span class="line"><span class="cl"><span class="sb">
</span></span></span><span class="line"><span class="cl"><span class="si">${</span><span class="nx">conversationHistory</span><span class="p">.</span><span class="nx">map</span><span class="p">(</span><span class="nx">m</span> <span class="o">=&gt;</span> <span class="sb">`</span><span class="si">${</span><span class="nx">m</span><span class="p">.</span><span class="nx">role</span><span class="si">}</span><span class="sb">: </span><span class="si">${</span><span class="nx">m</span><span class="p">.</span><span class="nx">content</span><span class="si">}</span><span class="sb">`</span><span class="p">).</span><span class="nx">join</span><span class="p">(</span><span class="s1">&#39;\n\n&#39;</span><span class="p">)</span><span class="si">}</span><span class="sb">`</span><span class="p">;</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="kr">const</span> <span class="nx">summary</span> <span class="o">=</span> <span class="k">await</span> <span class="nx">chat</span><span class="p">([{</span> <span class="nx">role</span><span class="o">:</span> <span class="s2">&#34;user&#34;</span><span class="p">,</span> <span class="nx">content</span>: <span class="kt">compactPrompt</span> <span class="p">}]);</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1">// Replace history with summary
</span></span></span><span class="line"><span class="cl"><span class="nx">conversationHistory</span> <span class="o">=</span> <span class="p">[</span>
</span></span><span class="line"><span class="cl">  <span class="p">{</span> <span class="nx">role</span><span class="o">:</span> <span class="s2">&#34;user&#34;</span><span class="p">,</span> <span class="nx">content</span><span class="o">:</span> <span class="sb">`[CONTEXT SUMMARY]</span><span class="err">\</span><span class="sb">n</span><span class="si">${</span><span class="nx">summary</span><span class="si">}</span><span class="err">\</span><span class="sb">n[END SUMMARY]`</span> <span class="p">}</span>
</span></span><span class="line"><span class="cl"><span class="p">];</span></span></span></code></pre></div></div>
<p>A 50-message conversation squeezes down to about 150 tokens, and the model carries on without losing track of which jobs and customer IDs it created.</p>

<h2 class="relative group">Saving sessions
    <div id="saving-sessions" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#saving-sessions" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>I also save chat sessions to the app&rsquo;s backend. The one idea worth copying is to store every assistant message twice: once cleaned up for display, and once raw, with the <code>&lt;think&gt;</code> and <code>&lt;action&gt;</code> tags left in. History then replays cleanly in the UI, while a resumed session gets the full context back, actions and all. Users switch between past conversations from a history dropdown, the way most chat apps work.</p>

<h2 class="relative group">Benchmarks from production
    <div id="benchmarks-from-production" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#benchmarks-from-production" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p><strong>iPhone 14 Pro</strong> (A16 Bionic, 6GB RAM):</p>
<ul>
<li>Model load: ~3 seconds</li>
<li>First token: 180-220ms</li>
<li>Streaming: 14-16 tokens/sec</li>
<li>Memory: ~1.8 GB</li>
</ul>
<p><strong>Samsung Galaxy S23</strong> (Snapdragon 8 Gen 2, 8GB RAM):</p>
<ul>
<li>Model load: ~4 seconds</li>
<li>First token: 250-300ms</li>
<li>Streaming: 10-12 tokens/sec</li>
<li>Memory: ~2.1 GB</li>
</ul>
<p><strong>iPhone 11</strong> (A13, 4GB RAM):</p>
<ul>
<li>Model load: ~6 seconds</li>
<li>First token: 400-500ms</li>
<li>Streaming: 6-8 tokens/sec</li>
<li>Memory: ~2.2 GB (occasionally crashes on low memory)</li>
</ul>
<p>My rule of thumb: 6GB+ RAM for a smooth experience. It runs on 4GB, but you&rsquo;ll probably need to unload the model when the app backgrounds.</p>

<h2 class="relative group">Memory management
    <div id="memory-management" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#memory-management" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>iOS will happily kill your app for hoarding memory, so I release the model whenever the app leaves the foreground:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-typescript" data-lang="typescript"><span class="line"><span class="cl"><span class="nx">useEffect</span><span class="p">(()</span> <span class="o">=&gt;</span> <span class="p">{</span>
</span></span><span class="line"><span class="cl">  <span class="kr">const</span> <span class="nx">subscription</span> <span class="o">=</span> <span class="nx">AppState</span><span class="p">.</span><span class="nx">addEventListener</span><span class="p">(</span><span class="s2">&#34;change&#34;</span><span class="p">,</span> <span class="p">(</span><span class="nx">nextAppState</span><span class="p">)</span> <span class="o">=&gt;</span> <span class="p">{</span>
</span></span><span class="line"><span class="cl">    <span class="k">if</span> <span class="p">(</span><span class="nx">nextAppState</span> <span class="o">===</span> <span class="s2">&#34;background&#34;</span><span class="p">)</span> <span class="p">{</span>
</span></span><span class="line"><span class="cl">      <span class="nx">llamaContext</span><span class="o">?</span><span class="p">.</span><span class="nx">release</span><span class="p">();</span> <span class="c1">// Free ~2 GB
</span></span></span><span class="line"><span class="cl">    <span class="p">}</span> <span class="k">else</span> <span class="k">if</span> <span class="p">(</span><span class="nx">nextAppState</span> <span class="o">===</span> <span class="s2">&#34;active&#34;</span><span class="p">)</span> <span class="p">{</span>
</span></span><span class="line"><span class="cl">      <span class="nx">loadModel</span><span class="p">();</span> <span class="c1">// Reload when foregrounded
</span></span></span><span class="line"><span class="cl">    <span class="p">}</span>
</span></span><span class="line"><span class="cl">  <span class="p">});</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">  <span class="k">return</span> <span class="p">()</span> <span class="o">=&gt;</span> <span class="nx">subscription</span><span class="p">.</span><span class="nx">remove</span><span class="p">();</span>
</span></span><span class="line"><span class="cl"><span class="p">},</span> <span class="p">[]);</span></span></span></code></pre></div></div>

<h2 class="relative group">Battery impact
    <div id="battery-impact" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#battery-impact" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>On-device inference isn&rsquo;t free, power-wise. From real-world testing:</p>
<table>
	<thead>
			<tr>
					<th>Usage Pattern</th>
					<th>Extra Battery Drain</th>
			</tr>
	</thead>
	<tbody>
			<tr>
					<td>Light (5-10 queries/day)</td>
					<td>&lt;1% per day</td>
			</tr>
			<tr>
					<td>Medium (20-30 queries/day)</td>
					<td>~3-4% per day</td>
			</tr>
			<tr>
					<td>Heavy (50+ queries/day)</td>
					<td>~6-8% per day</td>
			</tr>
	</tbody>
</table>
<p>GPU inference draws more power than CPU but finishes much sooner, and users clearly prefer the instant response, so I&rsquo;ve kept it.</p>

<h2 class="relative group">Two months in, with ~200 beta users
    <div id="two-months-in-with-200-beta-users" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#two-months-in-with-200-beta-users" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>The query breakdown surprised me a little:</p>
<ol>
<li>&ldquo;Create a job for [customer name]&rdquo; - 68% of queries</li>
<li>&ldquo;Find jobs for [customer]&rdquo; - 15%</li>
<li>&ldquo;Update job [ID] to [status]&rdquo; - 9%</li>
<li>General questions about the app - 8%</li>
</ol>
<p>91% of queries succeed on the first try. The failures split into customer not found because the user misspelled the name (4%), context overflow in very long conversations (3%), and the model inventing customer IDs (2%).</p>
<p>That last one used to be much worse. Hallucinated IDs mostly disappeared once I made tool results explicit — returning <code>customer_id=42</code> rather than prose. Small models need that scaffolding spelled out.</p>

<h2 class="relative group">What it costs
    <div id="what-it-costs" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#what-it-costs" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>On-device: $0/month infrastructure, $0 per user beyond the one-time 1.1 GB download, and it scales to as many users as care to install the app.</p>
<p>The cloud version, roughly estimated: an average conversation is ~2,000 tokens, and 200 users at 30 conversations a month is 6,000 conversations — 12M tokens/month. That&rsquo;s about $18/month on a budget cloud model, and $180-360/month on the top-tier ones. Not ruinous at this scale, but the on-device approach paid for itself on day one, and at 10,000 users it would still cost nothing.</p>

<h2 class="relative group">Limitations worth knowing
    <div id="limitations-worth-knowing" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#limitations-worth-knowing" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p><strong>Model capabilities.</strong> Qwen 1.7B is great at structured tasks and terrible at complex multi-hop reasoning, factual recall (it&rsquo;s not a search engine), creative writing, and nuance. Design features around what small models do well and you&rsquo;ll be fine.</p>
<p><strong>Device requirements.</strong> Minimum is 3 GB RAM and 2 GB free storage; realistically you want 6 GB RAM, a 64-bit processor, and GPU support. Older devices like the iPhone 8 or Galaxy S9 struggle — do feature detection and fall back gracefully.</p>
<p><strong>Model updates.</strong> Shipping a new model means users re-download 1.1 GB. I version the model in the storage path:</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-typescript" data-lang="typescript"><span class="line"><span class="cl"><span class="kr">const</span> <span class="nx">MODEL_PATH</span> <span class="o">=</span> <span class="sb">`</span><span class="si">${</span><span class="nx">FileSystem</span><span class="p">.</span><span class="nx">documentDirectory</span><span class="si">}</span><span class="sb">llama-models/v2/Qwen3-1.7B-Q4_K_M.gguf`</span><span class="p">;</span></span></span></code></pre></div></div>
<p>Bump the path for each new model version. Old models clean themselves up on uninstall.</p>

<h2 class="relative group">iOS vs Android
    <div id="ios-vs-android" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#ios-vs-android" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>iOS on Metal is around 20% faster than Android, manages memory better, and backs the model up to iCloud by default (disable that with <code>NSURLIsExcludedFromBackupKey</code> unless you want to eat users&rsquo; backup quota). Android varies a lot more across devices — llama.rn&rsquo;s GPU path is OpenCL on Qualcomm Adreno 700-series GPUs, and phones without one run on the CPU — and storage isn&rsquo;t backed up automatically. Test on both; the gap between a good and bad Android device is bigger than the gap between platforms.</p>

<h2 class="relative group">Debugging tips
    <div id="debugging-tips" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#debugging-tips" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p><strong>1. Enable verbose logging</strong></p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-typescript" data-lang="typescript"><span class="line"><span class="cl"><span class="kr">const</span> <span class="nx">ctx</span> <span class="o">=</span> <span class="k">await</span> <span class="nx">initLlama</span><span class="p">({</span>
</span></span><span class="line"><span class="cl">  <span class="nx">model</span>: <span class="kt">MODEL_PATH</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">  <span class="nx">n_ctx</span>: <span class="kt">8192</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">  <span class="nx">n_gpu_layers</span>: <span class="kt">99</span><span class="p">,</span>
</span></span><span class="line"><span class="cl">  <span class="nx">verbose</span>: <span class="kt">true</span><span class="p">,</span> <span class="c1">// Logs every token + timings
</span></span></span><span class="line"><span class="cl"><span class="p">});</span></span></span></code></pre></div></div>
<p><strong>2. Track token counts</strong></p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-typescript" data-lang="typescript"><span class="line"><span class="cl"><span class="kr">const</span> <span class="nx">tokens</span> <span class="o">=</span> <span class="nx">fullResponse</span><span class="p">.</span><span class="nx">split</span><span class="p">(</span><span class="sr">/\s+/</span><span class="p">).</span><span class="nx">length</span> <span class="o">*</span> <span class="mf">1.3</span><span class="p">;</span> <span class="c1">// Rough estimate
</span></span></span><span class="line"><span class="cl"><span class="nx">console</span><span class="p">.</span><span class="nx">log</span><span class="p">(</span><span class="sb">`Generated </span><span class="si">${</span><span class="nx">tokens</span><span class="si">}</span><span class="sb"> tokens in </span><span class="si">${</span><span class="nx">duration</span><span class="si">}</span><span class="sb">ms`</span><span class="p">);</span></span></span></code></pre></div></div>
<p><strong>3. Monitor memory</strong></p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-typescript" data-lang="typescript"><span class="line"><span class="cl"><span class="kr">import</span> <span class="p">{</span> <span class="nx">MemoryInfo</span> <span class="p">}</span> <span class="kr">from</span> <span class="s1">&#39;react-native-device-info&#39;</span><span class="p">;</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="kr">const</span> <span class="nx">memoryUsage</span> <span class="o">=</span> <span class="k">await</span> <span class="nx">MemoryInfo</span><span class="p">.</span><span class="nx">getUsedMemory</span><span class="p">();</span>
</span></span><span class="line"><span class="cl"><span class="nx">console</span><span class="p">.</span><span class="nx">log</span><span class="p">(</span><span class="sb">`Memory: </span><span class="si">${</span><span class="p">(</span><span class="nx">memoryUsage</span> <span class="o">/</span> <span class="mi">1024</span> <span class="o">/</span> <span class="mi">1024</span><span class="p">).</span><span class="nx">toFixed</span><span class="p">(</span><span class="mi">0</span><span class="p">)</span><span class="si">}</span><span class="sb"> MB`</span><span class="p">);</span></span></span></code></pre></div></div>
<p><strong>4. Test offline</strong></p>
<p>Flip on airplane mode and use the app properly. The model should load from cache, inference should complete, and only the backend persistence should fail — gracefully.</p>

<h2 class="relative group">Where I want to take it
    <div id="where-i-want-to-take-it" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#where-i-want-to-take-it" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>llama.cpp already supports vision models, and &ldquo;here&rsquo;s a photo from the site, add it to the job&rdquo; is an obvious fit for this app. I&rsquo;d also like to try fine-tuning a LoRA adapter on past job descriptions — my guess is 20-30% better accuracy on domain queries. Further out: an on-device speech-to-text model for a fully offline voice interface, and maybe federated learning so the model improves across users without any raw data leaving their phones.</p>

<h2 class="relative group">Should you build this?
    <div id="should-you-build-this" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#should-you-build-this" aria-label="Anchor">#</a>
    </span>
    
</h2>
<p>If you&rsquo;re building a business app with sensitive data, offline-first workflows, or structured tasks like classification and data entry — and especially if API costs at scale worry you — yes, this works today.</p>
<p>If you need open-ended chat, use one of the big cloud models. Same if the task is knowledge-heavy (small models simply don&rsquo;t know much), if you must support low-end devices, or if your domain knowledge changes faster than you can ship model updates.</p>
<p>The whole thing took me about 12 hours to build. It costs $0 a month to run, works on a plane, and the customer data never leaves the customer&rsquo;s hand. Hard to argue with.</p>
<hr>
<p><em>Questions? Feedback? I&rsquo;m <a href="https://twitter.com/jaredlynskey"  target="_blank" rel="noreferrer">@jaredlynskey</a>. The llama.rn library is at <a href="https://github.com/mybigday/llama.rn"  target="_blank" rel="noreferrer">github.com/mybigday/llama.rn</a>.</em></p>
]]></content:encoded>
      <media:content url="https://jared.lynskey.co.nz/en/posts/2026/2026-01-30-on-device-ai/featured.jpg" medium="image" type="image/jpeg" />
    </item>
  </channel>
</rss>
