Mobile · On-device AI · For beginners

Building a private AI chat app for Android: local models, online models, and shipping to Google Play

You type a message, a few seconds later an AI answers. It feels like magic, but it is a handful of simple ideas stacked together. I built an Android chat app that can answer from a model running on the phone itself, switch to an online model when you want a bigger brain, and quietly fall back to the phone when the internet disappears. This article explains every idea in plain words, with pictures, so you can follow it even if you have never touched AI before.

1. The big picture

From the outside, the app is just a chat screen. Behind it there is a “brain” that turns your message into a reply. The trick is that the brain can come from three different places, and the screen does not need to know which one is in use.

2. How an AI chat actually works

A large language model (LLM) is a program that has read an enormous amount of text and learned its patterns. Give it some text and it predicts what should come next. A chat is built on top of that one ability.

Tokens: the pieces it reads and writes

Models do not read letters or whole words. They read tokens, which are short chunks of text. Roughly, one token is three-quarters of an English word. Speed, memory limits and price are all counted in tokens.

It has no memory, so the app re-sends the chat

This is the idea that surprises most people. A model does not remember your previous message. Each time you press send, the app packs the whole conversation so far into one big block of text and hands it over. The model reads it all and continues it.

The model can only look at a limited amount of text at once, called the context window. It is measured in tokens (for example 4,096 on a phone). When the conversation grows past the window, the oldest messages have to be dropped, and the model effectively forgets them.

The chat template: labels the model understands

A raw model only continues text, so the app marks who is speaking using special labels the model was trained on. Different model families use different labels, and using the wrong ones is the number one reason a small model produces nonsense.

The first part, the system prompt, is a standing instruction such as “be concise” or “you are a code reviewer”. It is just text at the top, which makes it a handy place to slip in anything else the model should know. We will use that for memory.

The dials: why answers differ each time

At every step the model rates every possible next token, and the app has to pick one. A few dials control that pick:

• Temperature: low means safe and repetitive, high means creative and sometimes wrong.
• Top-k / top-p: only consider the most likely few candidates.
• Repeat penalty: nudges the model away from looping.
• Max tokens: a hard cap on how long a reply can be.

Streaming

Because tokens appear one by one, the app shows them as they arrive instead of waiting for the whole answer. That is why replies seem to be “typed” in front of you, both for on-device and online models.

3. How the app is organised

The most useful design decision is to hide where the answer comes from behind one small interface. The screens ask for text and get a stream of text back; they never need to know whether it came from the phone, a server, or a rule list.

In Kotlin that boundary can be tiny:

interface LlmEngine {
    suspend fun load(modelPath: String, config: EngineConfig)
    fun generate(prompt: String, params: GenerationParams): Flow<String>
    suspend fun unload()
}

A Flow<String> is Kotlin's way of saying “a stream of text pieces that arrive over time”. The screen collects them and shows each one as it comes. Cancel the collector and generation stops, which gives you a working Stop button for free. Chats and messages live in Room (a friendly layer over SQLite), so history survives restarts.

4. Adding a model that runs on the phone

What is a model file?

A model is a big collection of numbers, called weights, learned during training. Think of it as a very large recipe card of what the AI “knows”. To fit on a phone you need two tricks:

• Small models: 0.3 to 3 billion parameters instead of hundreds of billions.
• Quantization: storing each number with fewer bits (about 4 instead of 16 or 32). Imagine rounding every price to the nearest dollar: much smaller, slightly less precise. A 4-bit file is roughly 3–4 times smaller with a modest drop in quality.

A handy rule of thumb: a 4-bit model needs about 0.6 GB of storage per billion parameters, and about that much free RAM, plus extra for the conversation.

The GGUF file format

The common format for phone-sized models is GGUF. It is a single file that holds the weights, the tokenizer (the thing that splits text into tokens) and some notes about the model. Tools create it; nobody writes one by hand (more in section 10).

Running it: llama.cpp, a bridge, and Kotlin

llama.cpp is an open-source engine written in C++ that can run GGUF models on an ordinary phone processor. Android apps are written in Kotlin, so we need a bridge between the two languages.

1. Compile the engine with the Android NDK. Add llama.cpp as a git submodule and a CMakeLists.txt that builds it into a shared library for each processor type (arm64-v8a for phones, x86_64 for the emulator).
2. Write a thin bridge with JNI (Java Native Interface). Keep it tiny: load, start a reply, get next piece, stop, free.
3. Wrap it in the same LlmEngine interface, turning “get next piece” into a Flow.

object LlamaBridge {
    init { System.loadLibrary("native_llm") }
    external fun nativeLoad(path: String, contextSize: Int, threads: Int): Long
    external fun nativeStart(handle: Long, prompt: String /* + sampling args */): Int
    external fun nativeNext(handle: Long): ByteArray?   // null when finished
    external fun nativeAbort(handle: Long)
    external fun nativeFree(handle: Long)
}

Two practical rules: run all native calls on one dedicated thread, so freeing the model can never collide with a running reply; and treat native crashes as serious, because there is no Kotlin error message to catch (see section 11).

An alternative is Google's MediaPipe / LiteRT LLM runtime, which uses its own .task files and avoids writing C++. Both can sit behind the same interface.

Getting the model onto the phone

• Download on demand. List a few small models and download the one the user picks. Make downloads resumable (HTTP Range requests), https-only, and cap the allowed size so nobody pulls a model the phone cannot run.
• Import a file with the system file picker and copy it into app-private storage.
• Bundle one inside the app. Put a small model in the app's assets and copy it out on first launch, so a fresh install works with no download. The cost is a bigger download for everyone, plus consequences for Google Play (section 12).

Loading a model reads hundreds of megabytes into memory and takes seconds. Load it lazily on the first message, keep it while the app is open, free it when the app closes, and always show “Loading…” so the first reply does not look frozen.

5. Why the first word takes time (and how to speed it up)

When you press send, two different jobs happen on the phone:

1. Reading the whole prompt (instructions + chat + your message). The model has to process every token once. Developers call this prefill.
2. Writing the answer, one token at a time.

Nothing appears on screen until the reading is done. On a fast phone that takes a blink. On an older phone, with a long prompt, it can take most of a minute. On a test phone from 2018 I measured about 16 tokens per second of reading: a 900-token prompt meant a 55-second wait before the first word, even though the phone was working the whole time.

Fix 1: don't re-read what hasn't changed (the KV cache)

While reading, the model builds notes about the text it has seen, called the KV cache. Next time you send a message, the start of the prompt (instructions and the earlier chat) is identical, so those notes are still valid. The engine keeps them and only reads the new part.

In the bridge this is a small change: remember which tokens are in the cache, find how many match the new prompt from the start, delete the rest, and decode only the difference. If nothing matches (for example old messages were dropped), it falls back to reading everything.

Fix 2: send less

• Cap the prompt on the phone, for example around 800 tokens, and drop the oldest messages first. Offer a small setting (Fast / Balanced / Long) so users choose speed versus memory.
• Add extras only when needed. A recap of other chats is only worth its size when the user actually asks about earlier chats. For a plain “hi” it just slows things down.

Fix 3: pick a file the phone is fast at

Quantized models come in several flavours. Newer phones have special instructions that make the popular “Q4_K_M” type quick; older phones do not, and a plainer type called Q4_0 reads prompts much faster there. Offering both in the download list lets people on older phones have a better time.

6. Adding an online model

An online model does the heavy computing on a server. Many providers, and many self-hosted gateways, speak the same language, the OpenAI-compatible chat completions API. One small client therefore covers a lot of services.

POST {baseUrl}/chat/completions
Authorization: Bearer <api key>

{ "model": "some-model-name", "stream": true,
  "messages": [ {"role": "system", "content": "..."},
                {"role": "user",   "content": "Hello"} ] }

The reply arrives as Server-Sent Events: lines starting with data:, each holding a tiny JSON chunk, ending with data: [DONE].

data: {"choices":[{"delta":{"content":"Hel"}}]}
data: {"choices":[{"delta":{"content":"lo!"}}]}
data: [DONE]

Read each line, pull out choices[0].delta.content, and emit it into the same Flow<String> the local engine uses. To the screen, nothing changed.

Details that matter

• Stopping. A waiting network read cannot be interrupted by cancelling a coroutine. Close the connection from the cancellation callback so Stop works.
• Friendly errors. Turn 401/403 into “the key was rejected” and 429 into “quota reached” so people know what to fix.
• https only. Refuse plain http.
• The API key. A key the user types stays on their phone, but plain preferences are not encrypted; prefer the Android Keystore. Never ship your own key inside the app: anyone can pull it out of the APK. To offer a hosted model yourself, put your own server in the middle to hold the key and limit usage.
• Privacy. Everything in the prompt leaves the phone, including any memory you add. Say so clearly.

7. Offline vs online, heat, and falling back

Neither mode is better. They trade different things.

Why the phone gets warm

Writing each token means doing a huge pile of arithmetic on the processor, using several cores at once, a bit like running a demanding game. It happens in bursts: the phone warms while it writes and cools when idle. If it gets too hot, Android slows the processor down on its own, which makes replies slower but does not damage the phone. Smaller models, fewer threads and shorter replies reduce the heat; Online mode avoids it, because the work happens elsewhere.

The automatic fallback

If the user chose Online but the connection is gone, the app should not just fail. It should quietly use the phone instead.

One detail: check that the network can really reach the internet. Android reports a VALIDATED capability for that; being “connected to Wi-Fi” is not enough.

8. Remembering across chats

Since the model has no memory, “remembering” is something the app does by adding text to the prompt. Two simple mechanisms:

• Saved facts. A short list the user can edit (“I'm vegetarian”, “my dog is called Rex”), or fill by typing “Remember that…”. These go into the system prompt of every chat.
• A recap of other chats. A few recent messages from other conversations (what the user asked and what the model answered), added only when the user asks about earlier chats.

Two cautions. First, keep this extra text small, or it eats the context window of a small model and slows it (section 5). Second, give users control: toggles to switch each part off, a screen to view and delete facts, and an honest note about what an online model would receive. Reopening the last conversation on launch completes the feeling that the app “remembers”.

9. Making it feel like a real chat app

Much of what people like in popular chat apps is not the model. It is how the waiting is handled. An app that goes silent for 30 seconds feels broken, even if it is working hard.

• Show the user's message instantly, before any model work starts.
• Never leave a blank screen. Pulse a dot with “Thinking…”, switch to “Still thinking… 8s” after a few seconds, and say “Loading…” while the model loads.
• Stream the text with a small dot at the end.
• Follow long replies. If the list stays scrolled to the start of a tall reply, the user sees the beginning and thinks it stopped. Scroll to the end as text arrives.
• Offer “Continue”. Replies stop at a length limit, or when the user presses Stop. Mark those replies and show a Continue button that simply asks the model to carry on.
• Handle empty replies. Small models sometimes end their turn immediately. Retry once with a new random seed, and if it is still empty, say so, instead of failing silently.
• Look familiar: plain full-width replies, compact bubbles for the user, a rounded input pill, a flat neutral palette and a side drawer for chats.

10. Making a very simple model

People often ask: can I make my own model? There are two very different answers.

Option A: a rule-based “model” (an afternoon)

If the app only needs a small vocabulary, you do not need a neural network. A list of keywords and canned replies is instant, uses almost no battery, needs no file and behaves the same way every time. It is not AI, but behind the interface from section 3 the screen cannot tell the difference.

private val rules = listOf(
    Rule(setOf("hi", "hello", "hey")) { "Hello! How can I help?" },
    Rule(setOf("time", "clock"))      { "It's ${now.format("h:mm a")}." },
    Rule(setOf("thanks", "thank"))    { "You're welcome!" },
)

fun reply(input: String): String {
    val words = Regex("[\\p{L}\\p{N}']+").findAll(input.lowercase()).map { it.value }.toSet()
    return rules.firstOrNull { r -> r.keywords.any { it in words } }?.reply?.invoke(ctx)
        ?: "I don't know that one yet."
}

It also makes a graceful last resort: a fresh install with no model downloaded can still answer, and can point the user to a real model. Show its text word by word with a short delay so it feels like typing.

Option B: train a real tiny model (a project)

A GGUF file holds learned numbers, not words or rules, which is why it cannot be written by hand. Making one has three stages:

python convert_hf_to_gguf.py ./my-tiny-model --outfile tiny-f16.gguf --outtype f16
llama-quantize tiny-f16.gguf tiny-q4_k_m.gguf Q4_K_M

The result loads through exactly the same code as any downloaded model, so it is a good way to test your loading path. For real quality it is far easier to start from an existing small open model and, if needed, fine-tune it.

11. A pitfall: the crash that only some phones had

The first test build died instantly on one phone with SIGILL, short for “illegal instruction”, deep inside the native library.

To make it worse, the engine chose its fast routines based on how it was compiled, not on what the phone could actually do. The fix was to compile for plain armv8-a, which every 64-bit ARM phone understands. The price is some speed on newer phones; the proper solution is to build several variants and choose one at runtime. The general lesson: native crashes are not ordinary errors, so test on a real, ideally older, phone, and read logcat for the signal name and the top of the stack trace.

12. Publishing on Google Play

Account and format

• Create a Play Console developer account (one-time fee, identity check). New personal accounts have usually had to run a closed test with a minimum number of testers for a couple of weeks before production access. Check the current rule.
• Pick your application ID carefully; it can never change after the first upload.
• Upload an Android App Bundle, not an APK: ./gradlew :app:bundleRelease. Turn on Play App Signing and keep your upload key backed up.
• Raise versionCode for every upload, and keep targetSdk at or above Play's current minimum, which rises every year.

Native code specifics

• Ship only the processor types you need; drop x86_64 in release if it exists just for the emulator.
• Newer Android versions require native libraries to be aligned for 16 KB memory pages. Link with -Wl,-z,max-page-size=16384 and use a current NDK.
• Test the release build on a device. Code shrinking (R8) can remove classes that native code looks up by name, so add keep rules for your JNI class.

The size problem

• Download on first run instead of bundling. Simplest, and people who never need a big model never pay for it.
• Play Asset Delivery: put the model in an install-time asset pack, delivered with the install, which allows far larger files. The app then reads it from the pack and copies it into private storage.
• Bundle a tiny model and offer bigger ones as downloads.

Listing and policy

• App name, descriptions, a 512×512 icon, a feature graphic and phone screenshots.
• A public privacy policy URL.
• The Data safety form. Be precise: offline mode keeps conversations on the device; online mode sends the user's text to the service they configure.
• Generative AI content: Play expects AI apps to give users a way to report offensive or harmful replies, so plan an in-app report option.
• Model licences: open models come with terms. Some require attribution or limit use. Read each one and credit them, for example on an About screen.
• Roll out through internal, then closed testing, before production.

13. Limitations

• Small models are small. Sub-billion-parameter models follow instructions loosely, make facts up and struggle with long or multi-step questions.
• Older phones are slow at reading prompts. The cache, the prompt cap and Q4_0 files help, but a long first prompt still takes time.
• CPU only. No GPU or NPU acceleration, so speed varies a lot between phones.
• Memory is shallow. A short recap is not real long-term recall; the context window is the hard limit.
• Text only. Image input needs a vision model, an extra projector file and more native code.
• Heat and battery in long on-device sessions.
• The online path depends on the provider: availability, cost and what they do with your data.
• A bundled model inflates the download and complicates Play distribution.

14. Takeaways and glossary

1. An AI chat is “re-send the conversation, stream back the continuation”. Learn tokens, the context window and the chat template and the rest follows.
2. Put every source of text behind one interface. Local, online and rule-based answers then become interchangeable, and fallbacks are easy.
3. For local models, use a small quantized GGUF through llama.cpp and the NDK, with a very thin bridge and one native thread.
4. Speed on the phone is mostly about reading the prompt: reuse the cache, send less, and pick a file type the phone is fast at.
5. For online models, an OpenAI-compatible streaming client covers many providers. Treat API keys and user text with care.
6. The experience is mostly about the waiting: instant echo, a thinking dot, streaming, a working Stop button and a Continue button.
7. Plan distribution early: the model's size, native-library requirements and privacy disclosures all shape how you publish.