Local LLMs in Your Pocket: Engineering On-Device Intelligence with MediaPipe and Expo

For the past year, we’ve all been living in the fetch('/api/generate') world. It’s convenient, but it’s a gold-plated cage. Between spiraling token costs, privacy concerns, and the inevitable 'offline' reality of mobile users, relying solely on cloud-based LLMs is a risk I wasn't willing to take for my latest projects.

I wanted to see how far we could push the edge. My goal: Run a quantized LLM directly on-device within an Expo-managed workflow. After some trial and error, I found the sweet spot using Google’s MediaPipe LLM Inference API. Here’s how I bridged the gap between heavy C++ internals and the React Native layer.

The Architecture of the Edge

Running an LLM on a mobile device isn't just about 'installing a library.' It’s about memory management. If you try to load a raw 7B parameter model, the iOS/Android kernel will kill your process before you can say 'Hello World.'

We need three things to make this work in Expo:

1. A Quantized Model: Think Gemma 2B or Phi-2, compressed to 4-bit (int4) to fit within mobile RAM.
2. MediaPipe Runtime: Specifically the experimental LLM Inference task which handles GPU/NPU acceleration.
3. A Custom Expo Module: Since this requires deep native integration, we use expo-modules-sdk to talk to the MediaPipe C++ logic.

Setting the Foundation

First, I had to move away from the standard 'web-first' mindset. We’re working with binary weights (.bin or .task files). In my experience, the best way to handle these in Expo is via a local file URI after a one-time download, as embedding a 1.5GB model in your .ipa or .apk is a recipe for user rejection.

The Native Bridge (Swift/Kotlin)

Using the Expo Module API, I created a wrapper around the MediaPipe task. Here’s a conceptual look at how I initialize the inference engine on the native side:

// Simplified Swift snippet for the Expo Module
import MediaPipeTasksGenAI

public class MediaPipeLLMModule: Module {
  private var llmInference: LlmInference?

  public func definition() -> ModuleDefinition {
    Name("MediaPipeLLM")

    Function("initializeModel") { (modelPath: String) in
      let options = LlmInference.Options(modelPath: modelPath)
      options.maxTokens = 512
      options.topK = 40
      
      self.llmInference = LlmInference(options: options)
    }

    AsyncFunction("generateResponse") { (prompt: String) -> String in
      guard let inference = self.llmInference else {
        throw NSError(domain: "LLM", code: 0, userInfo: [NSLocalizedDescriptionKey: "Model not initialized"])
      }
      return try inference.generateResponse(inputText: prompt)
    }
  }
}

The React Native Implementation

Once the native heavy lifting is done, the TypeScript side becomes surprisingly clean. I found that wrapping the inference in a custom hook provides the best UX, especially when handling the 'loading' states of massive model files.

import { useEffect, useState } from 'react';
import MediaPipeLLM from './modules/MediaPipeLLM';

export function useLocalLLM(modelUri: string) {
  const [isReady, setIsReady] = useState(false);

  useEffect(() => {
    async function setup() {
      // Ensure the model file exists locally
      await MediaPipeLLM.initializeModel(modelUri);
      setIsReady(true);
    }
    setup();
  }, [modelUri]);

  const ask = async (prompt: string) => {
    if (!isReady) return "System booting...";
    return await MediaPipeLLM.generateResponse(prompt);
  };

  return { ask, isReady };
}

Performance Realities: What I Learned

1. The Thermal Throttling is Real: Running a continuous generation loop will heat up an iPhone 15 Pro significantly. I learned to implement 'cool-down' periods and limit max tokens to avoid the OS dimming the screen or killing the app.
2. Quantization is Non-Negotiable: A 4-bit quantized Gemma 2B model is the 'Goldilocks' zone for mobile—small enough to stay under the 2GB memory pressure limit but smart enough to handle structured JSON extraction or basic chat.
3. Async UI: Never run inference on the Main Thread. MediaPipe handles the GPU work, but the bridge overhead can still jank your UI if you aren't careful. Always treat the response as a stream or an async operation.

Why This Matters

By moving inference to the device, we’ve eliminated the $0.01-per-request tax. But more importantly, we’ve created a Tier-1 privacy experience. My users' data never leaves their device. In an era of increasing data scrutiny, that’s not just a technical win—it’s a massive product differentiator.

Engineering local AI in Expo isn't about replacing the cloud; it's about building resilient, private, and cost-effective hybrid systems. The tools are finally here. It's time to stop fetching and start inferring.