Postgres to Typesense - 1M records in 3mins

I needed to sync apx 1,000,000 records from Postgres into Typesense, in as little tiem as possible. And I pulled it off: a full backfill of a table in under 3 minutes.

As always, I don’t work alone, even though in this project scope I pulled it off mostly myself. For the sake of completeness and to fill in details in the story, I’ll switch to “we” for the remainder of the post. This

Here’s how we architected it, tuned it, and measured it—all to deliver blazing-fast search experiences to our users.

The Challenge

We had three core demands:

  • Scale & Speed — Sync every 1 million records in under 3 minutes.
  • Search UX expectations — Postgres full-text just can’t meet our scale and latency needs with typo-tolerance and relevance.
  • Freshness without disruption — Data must stay current, syncing continuously without affecting primary workload.
  • Avoid over-engineered pipelines - Polling, triggers, Kafka/Debezium/RabbitMQ or whatever else territory was too brittle or complex for our needs.
  • Avoid constly serverless queues - Saving a buck in the process is interesting for us especially since this is just a PoC for the time being

To meet this, we put Sequin (using Postgres logical replication) in front of a self-hosted Typesense cluster. The result: efficient backfills, real-time change streaming, and a snappy search layer—all within our performance goal.

Architecture Overview

Flow Overview

Postgres (WAL → Sequin) → Sequin (transform + buffer) → Typesense (bulk import) → Search in our web UI*

*Our web UI is built with NextJS and react-instantsearch components, in case anyone is wondering.

  • Sequin hooks into PostgreSQL via logical replication—minimal overhead, exact ordering, and idempotent processing.
  • It supports backfills, transforms, and keeps the data stream consistent and checkpointed.
  • Typesense ingests via efficient bulk import API; tuned batch sizes and concurrency ensure high throughput without throttling search traffic.

Why Sequin?

  • Lightweight & scalable — single Docker container, no Kafka needed.
  • High throughput, low latency — up to 50k ops/sec and ~55ms average end-to-end latency.
  • Production-grade features — backfills, retries, transforms, metrics + observability.

Implementation Details (Template)

sequin.yaml (example)

This is obviously a public example but you will get a better idea of it:

databases:
  - name: prod-db
    username: your_user
    password: your_pass
    hostname: your.postgres.host
    database: your_db
    port: 5432
    slot_name: deepsync_slot
    publication_name: deepsync_pub

sinks:
  - name: typesense-sink
    database: prod-db
    table: public.your_table
    destination:
      type: typesense
      endpoint_url: "http://typesense.local:8108"
      collection_name: your_collection
      api_key: "YOUR_API_KEY"
      batch_size: 1000
      timeout_seconds: 5

Transformation

A supper simple transformation function was used, to flatten out the record and only keep the actual DB record.

def transform(action, record, changes, metadata) do
  record
end

Performance Tuning Tips

Here are the key knobs we tuned for optimal throughput while keeping Typesense responsive:

  • Use the Bulk Import API
    For high-volume writes (>10 ops/sec), stick with the Bulk Import endpoint - it’s far more efficient than single-document writes.

  • Match worker concurrency to CPU capacity
    Limit concurrent bulk write workers to (vCPU count) - 2 to leave headroom for search workloads. For instance, on an 8vCPU server, allow up to 6 parallel workers.

  • Prefer client-side batching over increasing server-side batch_size
    The server’s default batch_size is 40 documents per import call, and increasing it can hurt search performance and raise memory usage. Instead, control write throughput via your own parallel API calls.

  • Extend client-side timeouts
    Bulk imports are synchronous and may take time—set your client/libraries with generous timeouts (up to 60 minutes) to avoid early aborts that might trigger retries.

  • Handle HTTP 503 backpressure gracefully
    If Typesense returns 503 (Not Ready/Lagging), it’s signaling pressure.

    Options:

    1. Scale up CPU cores (min 4 vCPU recommended - I am self-hosting Typesense in a beefy machine and haven’t set CPU limits on its container so that it can consume all it needs).
    2. Increase client timeout to avoid premature retries.
    3. Improve disk I/O (e.g. nVMe SSDs).
    4. Adjust “healthy-write-lag” / “healthy-read-lag” if necessary.

  • Watch for resource exhaustion errors

    • OUT_OF_MEMORY: add RAM to fit your indexed dataset.
    • OUT_OF_DISK: increase disk space (Typesense typically recommends 5× RAM, at least that’s how they configure it on their cloud offering).

Sequin-specific batching control

  • Typesense sink config
    You can tweak batch_size (default: 40, max: 10,000) and timeout_seconds (default: 5s, max: 300s) in sequin.yaml.

  • Tune replication flush behavior
    Sequin buffers messages before sending them downstream. You can tune thresholds using:

    • REPLICATION_FLUSH_MAX_ACCUMULATED_BYTES (default: 100 MB)
    • ..._MESSAGES (default: 100,000 messages)
    • ..._TIME_MS (default: 50 ms)
      Adjusting these can balance throughput and memory utilization, depending on how you have opted to deploy Sequin.

    At least point I should mention, we don’t use their cloud-hosted version, which could be doing a lot of these tunings under the hood.

Infrastructure Setup

  • Both Sequin and Typesense run on a single VM:
    • 2 vCPUs
    • 8 GB RAM
    • Containerized environment with horizontal scaling when throughput demands rise.
  • Every field in the source table is marked searchable in Typesense, so queries can target any attribute instantly.
  • We haven’t yet implemented dedicated metrics, observability, or error-handling layers—this is next on the roadmap.
  • Started with a plain simple docker-compose.yaml and we’ll go as far as possible with this setup. Obviously not production ready so watch out.

What This Means in Practice

  • A single 2-vCPU, 8 GB VM is sufficient to handle backfilling 1 million records in about 3 minutes, while continuing incremental updates.
  • At this modest scale, Typesense’s in-memory design stays well within safe operating RAM:
    • Typical guidance says you’ll need 2×–3× the on-disk size of indexed fields in RAM.
    • Early benchmarks—like indexing 2.2 million recipes in ~3.6 mins using ~900 MB RAM on 4 vCPUs—give us confidence our setup has headroom.

Conclusion

A super successful PoC, which works tirelessly till this day. The same simple workflow works on our stripped-down infrastructure, achieving the target performance of backfilling apx 1M records in ~3-4mins time.