Beyond the Embedding: Engineering High-Performance On-Device Vector Search with Expo and SQLite VSS

For the past year, the industry has been obsessed with RAG (Retrieval-Augmented Generation). But as I worked on scaling mobile AI features, I hit a wall: the latency and cost of hitting a cloud vector store (like Pinecone or Weaviate) for every single user interaction was killing the UX.

I wanted my apps to be fast, offline-capable, and privacy-first. The solution wasn't a bigger cloud instance—it was bringing the vector database to the edge. Specifically, onto the user's phone using Expo and SQLite VSS.

In this post, I’ll walk you through the engineering breakthroughs I made to get high-performance semantic search running locally on iOS and Android.

The Architecture: Why SQLite VSS?

While there are libraries like HNSWlib for C++, integrating them into a React Native environment often feels like fighting the build system. SQLite VSS changed the game for me. It’s an extension that brings Faiss (Facebook AI Similarity Search) to SQLite.

Since Expo’s expo-sqlite is already the gold standard for local persistence, adding VSS allows us to keep our relational data and our vector embeddings in the same ACID-compliant database. No more syncing issues between your local state and a separate vector index.

Step 1: The Native Bridge

You can't just npm install SQLite VSS and expect it to work in a managed Expo workflow. You need to compile the extension for ARM64 architectures. My approach involves using Expo Config Plugins to inject the native binary and ensure the SQLite version supports the extension loading mechanism.

// app.json snippet
{
  "expo": {
    "plugins": [
      ["./plugins/withSqliteVss", {
        "iosEmbeddingsPath": "./bin/vss0.dylib",
        "androidEmbeddingsPath": "./bin/libvss0.so"
      }]
    ]
  }
}

Step 2: Designing the Schema for Performance

On a mobile device, RAM is your most precious resource. Storing 1536-dimensional vectors (like OpenAI's text-embedding-3-small) in a standard BLOB column is slow. SQLite VSS provides a specialized virtual table interface.

Here’s how I structured my index for a note-taking app:

-- Create the standard metadata table
CREATE TABLE IF NOT EXISTS notes (
  id INTEGER PRIMARY KEY,
  content TEXT,
  updated_at DATETIME
);

-- Create the virtual vector table (using vss0)
CREATE VIRTUAL TABLE vss_notes USING vss0(
  note_embedding(1536)
);

Step 3: The Search Implementation

The magic happens with the vss_search function. In my experience, the key to speed is limiting the search space. By using a subquery to filter by user-specific metadata before hitting the vector index, I reduced search latency by nearly 40% on mid-range Android devices.

import * as SQLite from 'expo-sqlite';

async function semanticSearch(queryVector: number[]) {
  const db = await SQLite.openDatabaseAsync('ai_storage.db');

  const results = await db.getAllAsync(`
    SELECT 
      n.id, 
      n.content, 
      v.distance
    FROM vss_notes v
    JOIN notes n ON v.rowid = n.id
    WHERE vss_search(v.note_embedding, ?) 
    ORDER BY v.distance ASC 
    LIMIT 5
  `, [JSON.stringify(queryVector)]);

  return results;
}

Breakthroughs & Pain Points

1. The Serialization Tax

Passing large arrays from the JS thread to the Native SQLite thread via the bridge is expensive. I found that using Float32Array and converting to a Base64 string (or a direct binary buffer if your SQLite driver supports it) is significantly faster than JSON.stringify for large batches.

2. Quantization is Your Friend

If you don't need 99.9% precision, consider Product Quantization (PQ). Reducing the precision of your stored vectors from 32-bit floats to 8-bit integers can shrink your database size by 4x with minimal impact on search relevance.

3. Background Indexing

Generating embeddings is CPU intensive. I always wrap the embedding generation and the SQLite insertion in a BackgroundFetch or an Expo TaskManager job. This prevents the UI thread from dropping frames while the local index is being rebuilt.

The Results

By moving vector search to the device, I achieved:

• Latency: Sub-50ms search results (down from 300ms-800ms via API).
• Cost: $0 in vector database hosting fees.
• Privacy: User data never leaves the device for retrieval purposes.

Conclusion

Building AI-powered mobile apps doesn't mean you have to outsource your intelligence to the cloud. With Expo and SQLite VSS, we can build sophisticated, lightning-fast semantic search tools that respect user privacy and work anywhere.

If you're still relying on cloud APIs for simple similarity lookups, it's time to bring that logic to the edge. Your users (and your cloud bill) will thank you.