Foodcourt POS: offline-first under a one-week deadline

Executive Summary

Our POS was online-first: every tap hit the API. When a 43-tenant foodcourt signed, many of its tenants sat under two wifi bars and a single order stretched to about 20 seconds of waiting. With one week to make it work, my team batched the existing mutations through a single sync endpoint that the app flushes every 30 seconds. The first version duplicated transactions in the pilot; a per-tenant sync lock ended it, and all 43 tenants went live inside the week.

The Problem & Context

The POS was online-first. Opening a transaction list, creating a transaction, taking payment — every step was an API round trip. On good connections each hit cost 100–150 ms, so nobody noticed. We had fewer than 20 active brands at the time.

Then came the foodcourt: 43 tenants, each one a brand — more than double our brand count. Tenants shared wifi provided by the venue, the mesh coverage was poor, and many sat under two signal bars. In the trial, more than 20 tenants hit waits of up to 3 seconds per API call. One takeaway order cost about 20 seconds of waiting, spread across every tap.

graph TD
    A["Open transaction list"] -->|"3–4s"| B["Tap menus"]
    B --> C["Create transaction"]
    C -->|"3–4s, up to 8s"| D["Back to list"]
    D -->|"3–4s"| E["Create payment"]
    E -->|"3–4s"| F["Back to list"]
    F --> G["≈20s from first tap to ready for the next order"]

• Business constraint: the foodcourt runs direct pay with no table service, so tenants live on the takeaway flow — and the venue’s wifi was not ours to fix.
• Commercial constraint: a sales promise had been made and the client was ready to sign. We had one week to make it work, or the deal was off.

Key Trade-offs Considered

• True offline-first, per table: make every flow read and write locally and sync table by table, with conflict handling for edits and changes arriving mid-sync. Scoped honestly, that was a month-plus of restructuring — we had one week.
• Batch the existing mutations through one endpoint: keep the tables and services as they were, queue mutations on the device, and replay them through a single sync API. Days, not weeks — and reversible per tenant if it failed.

Technical Approach

Mobile pooled mutations locally: creating a transaction or a payment saved it on the device and queued it for sync; the queue entry was removed only after the server confirmed. The app flushed the queue every 30 seconds and stayed usable with no connection at all.

On the backend, one sync endpoint received a batch and dispatched each mutation to the service that already owned it — a switch, not a new write path — inside a single database transaction, so a batch either landed together or not at all.

I ran both tracks in parallel — mobile on the queue, backend on the sync endpoint — directing the work day to day to hold the deadline. The core was working in three days.

The Pilot Incident

We went live for three tenants first, with barely any internal testing — the client was watching and we needed visible results fast. By the next day: almost 100 duplicates from a single day of trading.

The cause: the 30-second flush could fire while the previous sync was still running. Once a batch passed roughly 20 mutations, a sync crossed the 30-second mark, the app sent the next round anyway, and the server replayed mutations that were already applied.

Cleanup came first: a script, per my instruction, that pulled the tenant’s transactions, grouped them by offline id, collected every id with more than one row, and deleted exactly those. We rehearsed it against a backup database before touching production.

The fix: a sync status table per tenant.

A sync that finds the tenant locked returns early; otherwise it sets the lock and proceeds.

graph TD
    A["Sync tick"] --> B{"Sync status"}
    B -->|"1 — already syncing"| C["Return early, keep the queue"]
    B -->|"0"| D["Set status 1"] --> E["Apply the batch in one DB transaction"] --> F["Set status 0"]

One day to build, then three days of repeated test rounds while we extended sync to shift creation and closing. Then we went live for the whole foodcourt.

Sanitized Implementation Example

Simplified, sanitized mock of the sync endpoint — the per-tenant guard plus the dispatch into the existing services. Not production code.

// One sync at a time per tenant. A second round returns early instead
// of replaying mutations while the first sync is still applying them.
func (s *SyncService) Sync(ctx context.Context, brandID string, batch []Mutation) error {
  status, err := s.status.Get(ctx, brandID)
  if err != nil {
    return err
  }
  if status == Syncing {
    return ErrSyncInProgress
  }
  if err := s.status.Set(ctx, brandID, Syncing); err != nil {
    return err
  }
  defer func() { _ = s.status.Set(ctx, brandID, Idle) }()

  return s.db.Transaction(ctx, func(tx *sql.Tx) error {
    for _, m := range batch {
      var err error
      switch m.Kind {
      case CreateTakeaway:
        err = s.orders.CreateTakeaway(ctx, tx, m)
      case CreatePayment:
        err = s.payments.Create(ctx, tx, m)
      }
      if err != nil {
        return err
      }
    }
    return nil
  })
}

Results & Impact

All 43 tenants went live inside the one-week window, and the duplicates did not come back.

Roughly 75% less waiting per order on the same bad wifi — the server now reconciles within the 30-second sync interval instead of the user waiting on it.

A week later the offline port reached table flows as well — editing transactions, moving tables — with split transactions shipped along the way.

What I’d do differently

• Idempotency keys on the sync endpoint. The lock stops overlapping syncs; it does not make a mutation safe to replay twice. Keys would make duplicates impossible even if a lock failed.
• An atomic lock. The guard reads the status, then writes it. A single conditional update would close the gap between the two.
• A staging pass with big batches. The bug only appeared past roughly 20 mutations in a batch; a rehearsed large-batch test would have found it before a pilot tenant did.
• Multi-device brands. A second device for the same brand sees none of the first device’s queued mutations, and nothing pulls the two toward each other. The lock made sync safe; making it distributed is a problem of its own.