The foodcourt deal we lost: conflict rules for a multi-device POS

Executive Summary

A few months after the first foodcourt shipped offline-first, a second one brought a different flow: waiters entered orders on their own devices, one cashier collected the payments, and every device had to see every order. Multi-device was new to a POS built to work disconnected — two devices disagreeing had no answer in the design. Kafka broadcasts shipped in a week; the conflict classes they surfaced took six. We lost the deal in the middle of it. The rules shipped anyway, and the next multi-device client runs on them.

The Problem & Context

The first foodcourt kept moving: quick transaction collapsed submit and payment into one mutation, self-order arrived via table QR, and another team ran a commission-pull project on top of the platform. Then the second foodcourt signed — and its flow was different. Instead of customers walking up to each tenant, floor waiters took orders on their own devices and one cashier handled all payments. Enter an order on one waiter’s device, and the cashier plus every other waiter has to know about it. Deadline: two weeks.

The platform queues mutations on each device and replays them through one sync endpoint. That model answers “how does the server get my changes” — it never had to answer “what happens when two devices change the same order”, because until now one device owned a transaction.

Technical Approach

Broadcast the core mutations — create, edit order, payment, finish — over Kafka, one channel per [brand_id]-[outlet_id]. A Go WebSocket server relays the messages to connected devices, and each device updates from the payload.

graph LR
    A["Device — sync batch"] --> B["Sync endpoint"]
    B -->|"each mutation"| C["Kafka broadcast — [brand_id]-[outlet_id]"]
    C --> D["Go WebSocket relay"]
    D --> E["Other devices apply the payload"]

Kafka was my first call, and I want to be honest about why: the team knew it, we had tried RabbitMQ on an earlier project, and nobody wanted to learn a new broker mid-deadline. It was familiarity, not analysis.

The alpha — setup, research, first implementation — took a week, then two days of testing. The client was testing the live app in parallel the whole time, still on the pre-broadcast behavior, and kept asking why transactions were not appearing between devices.

The Conflicts

QC found the first one on a sync result, and we shortened the sync interval to five seconds just to make it reproduce more often. A waiter’s device had gone offline mid-edit; another device edited the same order with a different quantity, synced, took payment, and finished the transaction. Then the first device came back:

sequenceDiagram
    participant D1 as Device 1 (waiter)
    participant API as Sync API
    participant D2 as Device 2 (waiter)
    D1->>D1: Edits the order — offline
    D2->>API: Edits the same order, different quantity
    API-->>D2: Synced
    D2->>API: Pays and finishes the order
    API-->>D2: Complete — transaction is Paid
    Note over D1,API: Before the fix, this late edit was still applied
    D1->>API: Back online, syncs the old edit
    API-->>D1: Applied — quantities no longer match

The same shape appeared with delete order, refund, and split payment. Then we started walking through the candidate answers:

• Transaction locking — impossible. The whole point of the platform is that devices keep working while disconnected, and you cannot hold a database lock across a network gap.
• Last-write-wins — we built it. Two weeks. Then found its flaw:

sequenceDiagram
    participant D as Device — order X
    participant S as Background sync
    D->>S: Adds item A — saved locally
    S->>S: Round starts, carries order X without item A
    D->>D: Saves the edit
    D->>D: Adds item B, saves again
    S-->>D: Round returns — the whole cart is replaced from the payload
    Note over D: A and B disappear from the screen until the next sync pulls them back

Losing items for ten seconds reads as data loss to the person holding the device, even when the mutations are safe and the next sync restores them.

• Device timestamps — we were already using them for edit order. But they die on ordering: the payment is stamped 10:05:00, the late edit from a disconnected device is stamped 10:06:00. Do we still apply the edit? The clock says yes. The business says absolutely not, the bill is paid.

The Fix

Transaction states decide, not clocks. Once a transaction is Paid or Cancelled, any mutation against it is skipped on the backend, and the device overwrites its local copy from the server state. The gate runs on both the backend and the Android app.

Simplified, sanitized mock of the final-state gate. Not production code.

// Mutations stop at a transaction's final states. A late edit from an
// offline device is skipped, and the device re-reads the server state.
func (s *Transactions) Apply(ctx context.Context, tx *sql.Tx, m Mutation) error {
  state, err := s.states.Get(ctx, tx, m.TransactionID)
  if err != nil {
    return err
  }
  if state == Paid || state == Cancelled {
    return ErrFinalState
  }

  switch m.Kind {
  case EditOrder:
    return s.orders.Edit(ctx, tx, m)
  case AddPayment:
    return s.payments.Add(ctx, tx, m)
  }
  return nil
}

Losing the Deal

The slip became visible two days into testing: the conflict classes were not going to fit inside two weeks. We asked for room and kept the trial running for free while we worked.

Then the human problem landed. Over two days, the client’s cash reconciliation did not match — a shortfall of roughly 600 to 700 thousand rupiah. We pulled the logs and the transactions, session recaps included; nothing was missing on our side. We asked them to review the manual steps and shared the invoice prints to help the audit. It landed as an accusation — their cashier was furious — and neither side moved. The deal ended there.

Meanwhile the engineering room was on fire: one backend developer and one Android developer were allocated to this project, and the trial was also reporting printer and cash-drawer incompatibilities plus a memory leak on devices with 50 or more tables listed. I pulled three dev leads for three days to clear those while the assigned pair stayed on the conflict work. I reviewed every approach myself and kept tight day-to-day direction — closer than I normally run a team.

Results & Impact

The conflict rules shipped to production, gate running on backend and Android. Six weeks, from a two-week estimate — and one deal lost along the way, to the miss and to a disagreement the relationship never recovered from.

The next multi-device client proved the rules out: a two-floor cafe, one waiter device per floor, orders flowing between them on the same broadcast and the same state gate.

What I’d do differently

• Model the conflicts before choosing the transport. A one-page matrix — who can change what, in which state, and what wins — would have shown that the hardest case was an offline edit meeting a paid transaction. We should have spiked that first, before any Kafka plumbing.
• Say the estimate is gone when I believe it, not when the test proves it. Two days of testing made the slip undeniable; it was suspected earlier, and the client conversation should have started at suspicion.
• Broadcast from the commit, not from the service call. Revisiting this design later, I found a flaw the fix never covered: the Kafka messages fired per service call, inside the batch, before it committed — and fired again on retry. A batch that rolled back at mutation 43 had already pinged every device; a retry could echo a quantity older than a newer edit made elsewhere. Transactional outbox, plus a version on each transaction so devices discard anything stale.
• Design the state model with the offline model. The decisions from the first foodcourt — queue everything, replay it — set the shape of every conflict here. That is the lesson I keep: the next case will be designed by the last one unless you design them together.

With the deadline tight, we took the quick fix we hoped would close the deal. Every iteration after paid for it: new features didn’t fit the design, a small addition broke something unrelated, and moving to a better design is a project of its own. Tech debt is real.