This post walks through how I built it with llama.rn (React Native bindings for llama.cpp) and Qwen 1.7B, a small model that turned out to be more capable than I expected.

📝 Update: This implementation now uses Qwen3.5-2B-GGUF instead of the original Qwen3-1.7B. I’ve also stopped using the /no_think message hack and now disable thinking mode properly via the chat_template_kwargs parameter:

chat_response = client.chat.completions.create(
    model="Qwen/Qwen3.5-27B",
    messages=messages,
    max_tokens=32768,
    temperature=0.7,
    top_p=0.8,
    presence_penalty=1.5,
    extra_body={
        "top_k": 20,
        "chat_template_kwargs": {"enable_thinking": False},
    },
)

Much cleaner than smuggling control flags into the prompt.

Why I didn’t want cloud AI for this

#

The standard recipe for an AI feature in a mobile app is: user types a message, app sends it to OpenAI or Anthropic or Google, response comes back, bill ticks up.

For an app that curtain installers use to run their business, that recipe has problems. Customer names, addresses, and phone numbers go to a third party. Project details, pricing, and notes sit on someone else’s servers. There are GDPR and data-residency questions I’d rather not spend time on. And the API bill grows with every user.

The alternative is to run the model on the user’s phone, so that’s what I did.

Why this is finally practical

#

A year earlier I wouldn’t have bothered, but three things changed roughly at once. Quantised models got small — Qwen 1.7B at Q4_K_M is 1.1 GB, smaller than most games, and it downloads once and lives in app storage. Mobile GPUs got fast — llama.cpp uses Metal on iOS and Vulkan on Android, and on an iPhone 14 Pro I see about 15 tokens a second, plenty for real-time streaming. And small models got genuinely useful: Qwen 1.7B follows structured instructions, parses JSON, and chains multi-step reasoning. That’s exactly the profile business logic needs. Nobody’s asking it to write poetry.

Picking a model

#

I tested four small models before settling on Qwen3-1.7B.

TinyLlama 1.1B (637 MB) was the fastest, but it kept hallucinating customer IDs and dropping required fields — structured output just wasn’t reliable. Phi-3-mini (1.8 GB) reasons well but wouldn’t shut up; simple queries came back as 200-word essays when I needed 20. Gemma-2B (1.2 GB) was quick and accurate at classification but weak at function calling — it couldn’t consistently produce the <action> tags my tool system needs. Qwen3-1.7B (1.1 GB) hit the sweet spot: reliable structured output, precise instruction following, and chain-of-thought support via <think> tags.

The Q4_K_M quantisation uses 4-bit weights with k-means clustering — about 75% smaller than full precision for roughly 5% quality loss.

Setting up llama.rn

#

llama.rn wraps llama.cpp for React Native, and installation is the easy part:

npm install llama.rn
cd ios && pod install

The model itself downloads on first use:

const MODEL_URL = "https://huggingface.co/unsloth/Qwen3-1.7B-GGUF/resolve/main/Qwen3-1.7B-Q4_K_M.gguf";
const MODEL_PATH = FileSystem.documentDirectory + "llama-models/Qwen3-1.7B-Q4_K_M.gguf";

const downloadModel = async () => {
  const downloadResumable = FileSystem.createDownloadResumable(
    MODEL_URL,
    MODEL_PATH,
    {},
    (progress) => {
      const pct = progress.totalBytesWritten / progress.totalBytesExpectedToWrite;
      setDownloadProgress(pct);
    }
  );

  await downloadResumable.downloadAsync();
};

On WiFi that’s 2-3 minutes; on LTE more like 5-8. Once it’s on disk, load it with GPU acceleration:

const ctx = await initLlama({
  model: MODEL_PATH,
  n_ctx: 8192,      // 8K context window
  n_gpu_layers: 99, // Use GPU for all layers
});

Don’t skip n_gpu_layers: 99. Offloading everything to Metal/Vulkan instead of the CPU is worth about a 5x speedup.

Streaming inference

#

Nobody wants to stare at a spinner for ten seconds, so responses stream token by token:

let fullResponse = "";

await llamaContext.completion(
  {
    messages: [
      { role: "system", content: systemPrompt },
      { role: "user", content: "Create a job for John Smith" }
    ],
    n_predict: 512,
    temperature: 0.7,
    top_p: 0.8,
  },
  (data) => {
    // Called for each token
    fullResponse += data.token;
    setStreamingText(fullResponse);
  }
);

On modern phones the first token lands in about 200ms (that’s prompt processing) and each token after that takes 50-80ms. Compared to anything network-bound, it feels instant.

Function calling that small models can actually do

#

GPT-4-style function calling leans on JSON schemas, and small models fall over on it — malformed JSON, missing fields, the lot. So I use a simpler XML-based protocol:

const systemPrompt = `You are an AI assistant. When you need to take an action, output:
<action>{"type":"search_customers","query":"Smith"}</action>

Available actions:
- search_customers: {"type":"search_customers","query":"John"}
- create_job: {"type":"create_job","customer_id":5,"alias":"Living Room"}
- update_job: {"type":"update_job","job_id":"J-0042","status":"quoting"}

Rules:
1. ONE action tag per response, at the very end
2. When searching/creating → end with <action>
3. When just chatting → no action tag
`;

XML tags won for boring reasons: <action> and </action> are unambiguous delimiters, a regex is all the parsing you need, the examples in the prompt double as documentation, and it still works when the model wraps the tag in extra chatter.

Parsing is exactly as dumb as it looks:

function parseAction(text: string): Action | null {
  const match = text.match(/<action>([\s\S]*?)<\/action>/);
  if (!match) return null;

  try {
    return JSON.parse(match[1].trim());
  } catch {
    return null;
  }
}

When an action comes out, I execute it and feed the result back into the conversation:

const action = parseAction(modelResponse);
if (action) {
  const result = await executeAction(action);

  // Inject result as a system message
  conversationHistory.push({
    role: "user",
    content: `[TOOL_RESULT] ${result.message}\n\n→ NEXT: Tell the user what happened.`
  });

  // Continue generation
  await generateNextResponse();
}

That’s enough for proper multi-step workflows:

User: “Create a job for Smith”

1. Model: <action>{"type":"search_customers","query":"Smith"}</action>
2. System: [TOOL_RESULT] Found: Jane Smith (id:42), Bob Smith (id:89)
3. Model: “I found two Smiths—Jane and Bob. Which one?”
4. User: “Jane”
5. Model: <action>{"type":"create_job","customer_id":42}</action>
6. System: [TOOL_RESULT] Job J-0073 created
7. Model: “Done! Created job J-0073 for Jane Smith.”

Chain-of-thought with <think> tags

#

Small models do noticeably better when you make them reason out loud. Qwen has a thinking mode built in:

const systemPrompt = `Before each response, wrap your reasoning in <think> tags:

<think>
INTENT: what does the user want?
HAVE: what data do I already have?
NEED: what's still missing?
DECISION: call tool | ask user | just respond
</think>

Then output your actual response.`;

What the model produces internally looks like this:

<think>
INTENT: create a job
HAVE: customer query "Smith"
NEED: exact customer_id
DECISION: search first
</think>
<action>{"type":"search_customers","query":"Smith"}</action>

I strip the <think> tags before anything hits the UI, but leave them visible during development. The reliability difference is dramatic — the model talks itself through the logic before committing to an action.

Keeping long conversations inside 8K

#

The 8K context window fills up faster than you’d think once tool results start piling in. Around 30 turns, I hit the wall.

My fix is conversation compaction — I ask the model to summarise itself:

const compactPrompt = `Summarize this conversation in under 200 words.
Include: user's goal, customers/jobs created (with IDs), and any unfinished tasks.

${conversationHistory.map(m => `${m.role}: ${m.content}`).join('\n\n')}`;

const summary = await chat([{ role: "user", content: compactPrompt }]);

// Replace history with summary
conversationHistory = [
  { role: "user", content: `[CONTEXT SUMMARY]\n${summary}\n[END SUMMARY]` }
];

A 50-message conversation squeezes down to about 150 tokens, and the model carries on without losing track of which jobs and customer IDs it created.

Saving sessions

#

Chat sessions get persisted to my Django backend:

interface SessionMessage {
  role: "user" | "assistant";
  content: string;  // Stripped for display
  raw: string;      // Full with <think> and <action> tags
}

await api.createAIChatSession({
  organization_id: orgId,
  title: firstUserMessage.slice(0, 60),
  messages: [
    { role: "user", content: userText, raw: userText },
    { role: "assistant", content: cleanedResponse, raw: fullModelOutput }
  ]
});

Storing both content and raw means chat history replays cleanly in the UI, but a resumed session gets the full context back, actions and all. Users switch between conversations from a history dropdown, much like ChatGPT.

Benchmarks from production

#

iPhone 14 Pro (A16 Bionic, 6GB RAM):

• Model load: ~3 seconds
• First token: 180-220ms
• Streaming: 14-16 tokens/sec
• Memory: ~1.8 GB

Samsung Galaxy S23 (Snapdragon 8 Gen 2, 8GB RAM):

• Model load: ~4 seconds
• First token: 250-300ms
• Streaming: 10-12 tokens/sec
• Memory: ~2.1 GB

iPhone 11 (A13, 4GB RAM):

• Model load: ~6 seconds
• First token: 400-500ms
• Streaming: 6-8 tokens/sec
• Memory: ~2.2 GB (occasionally crashes on low memory)

My rule of thumb: 6GB+ RAM for a smooth experience. It runs on 4GB, but you’ll probably need to unload the model when the app backgrounds.

Memory management

#

iOS will happily kill your app for hoarding memory, so I release the model whenever the app leaves the foreground:

useEffect(() => {
  const subscription = AppState.addEventListener("change", (nextAppState) => {
    if (nextAppState === "background") {
      llamaContext?.release(); // Free ~2 GB
    } else if (nextAppState === "active") {
      loadModel(); // Reload when foregrounded
    }
  });

  return () => subscription.remove();
}, []);

Battery impact

#

On-device inference isn’t free, power-wise. From real-world testing:

GPU inference draws more power than CPU but finishes much sooner, and users clearly prefer the instant response, so I’ve kept it.

Two months in, with ~200 beta users

#

The query breakdown surprised me a little:

1. “Create a job for [customer name]” - 68% of queries
2. “Find jobs for [customer]” - 15%
3. “Update job [ID] to [status]” - 9%
4. General questions about the app - 8%

91% of queries succeed on the first try. The failures split into customer not found because the user misspelled the name (4%), context overflow in very long conversations (3%), and the model inventing customer IDs (2%).

That last one used to be much worse. Hallucinated IDs mostly disappeared once I made tool results explicit — returning customer_id=42 rather than prose. Small models need that scaffolding spelled out.

What it costs

#

On-device: $0/month infrastructure, $0 per user beyond the one-time 1.1 GB download, and it scales to as many users as care to install the app.

The cloud version, roughly estimated: an average conversation is ~2,000 tokens, and 200 users at 30 conversations a month is 6,000 conversations — 12M tokens/month. That’s about $18/month on GPT-3.5, $360/month on GPT-4, or $180/month on Claude. Not ruinous at this scale, but the on-device approach paid for itself on day one, and at 10,000 users it would still cost nothing.

Limitations worth knowing

#

Model capabilities. Qwen 1.7B is great at structured tasks and terrible at complex multi-hop reasoning, factual recall (it’s not a search engine), creative writing, and nuance. Design features around what small models do well and you’ll be fine.

Device requirements. Minimum is 3 GB RAM and 2 GB free storage; realistically you want 6 GB RAM, a 64-bit processor, and GPU support. Older devices like the iPhone 8 or Galaxy S9 struggle — do feature detection and fall back gracefully.

Model updates. Shipping a new model means users re-download 1.1 GB. I version the model in the storage path:

const MODEL_PATH = `${FileSystem.documentDirectory}llama-models/v2/Qwen3-1.7B-Q4_K_M.gguf`;

Bump the path for each new model version. Old models clean themselves up on uninstall.

iOS vs Android

#

iOS on Metal is around 20% faster than Android, manages memory better, and backs the model up to iCloud by default (disable that with NSURLIsExcludedFromBackupKey unless you want to eat users’ backup quota). Android on Vulkan/OpenCL varies a lot more across devices — some older GPUs don’t support Vulkan at all and fall back to CPU — and storage isn’t backed up automatically. Test on both; the gap between a good and bad Android device is bigger than the gap between platforms.

Debugging tips

#

1. Enable verbose logging

const ctx = await initLlama({
  model: MODEL_PATH,
  n_ctx: 8192,
  n_gpu_layers: 99,
  verbose: true, // Logs every token + timings
});

2. Track token counts

const tokens = fullResponse.split(/\s+/).length * 1.3; // Rough estimate
console.log(`Generated ${tokens} tokens in ${duration}ms`);

3. Monitor memory

import { MemoryInfo } from 'react-native-device-info';

const memoryUsage = await MemoryInfo.getUsedMemory();
console.log(`Memory: ${(memoryUsage / 1024 / 1024).toFixed(0)} MB`);

4. Test offline

Flip on airplane mode and use the app properly. The model should load from cache, inference should complete, and only the backend persistence should fail — gracefully.

Where I want to take it

#

llama.cpp already supports vision models (LLaVA, Qwen2-VL), and “here’s a photo of the curtain fabric, add it to the job” is an obvious fit for this app. I’d also like to try fine-tuning a LoRA adapter on past job descriptions — my guess is 20-30% better accuracy on domain queries. Further out: Whisper.cpp for a fully offline voice interface, and maybe federated learning so the model improves across users without any raw data leaving their phones.

Should you build this?

#

If you’re building a business app with sensitive data, offline-first workflows, or structured tasks like classification and data entry — and especially if API costs at scale worry you — yes, this works today.

If you need open-ended chat, use Claude or GPT-4. Same if the task is knowledge-heavy (small models simply don’t know much), if you must support low-end devices, or if your domain knowledge changes faster than you can ship model updates.

The whole thing took me about 12 hours to build. It costs $0 a month to run, works on a plane, and the customer data never leaves the customer’s hand. Hard to argue with.

Questions? Feedback? I’m @jaredlynskey. The llama.rn library is at github.com/mybigday/llama.rn.