Reconciliation, Not Sync: Integrating a System That Also Webhooks You Back

TL;DR — When you integrate with a system that also mutates the shared data and also webhooks you when it changes, you cannot “keep them in sync.” Both sides write, both sides notify, and you get echo loops, out-of-order updates, and fields that quietly diverge. The model that actually works is reconciliation: (1) assign a single owner to every field so a change has an authoritative direction, (2) apply every inbound event idempotently and out-of-order-safely, (3) tag your own writes so you can ignore the webhook they echo back, and (4) run a periodic full sweep that compares both sides and repairs the delta. Live events keep you fresh; the sweep keeps you correct.

The word “sync” is the bug

The ticket says “sync our catalog with the POS.” It sounds symmetric and simple. It is neither, and the word itself is what leads teams into a swamp.

Picture the setup. Your platform has a menu — items, prices, availability. You integrate a point-of-sale system that the restaurant also edits directly. Now:

• The restaurant marks an item out of stock in the POS → the POS webhooks you.
• An operator marks the same item available in your dashboard → you push to the POS.
• The POS applies your push → and webhooks you back that it changed.

Two systems, both authoritative-feeling, both mutating, both notifying. “Sync” implies there is one true state and you’re keeping a mirror of it. But there isn’t one true state — there are two writers racing, and every write generates a notification that can trigger another write. The moment you try to make A always equal B by reacting to change events, you’ve built a distributed feedback loop. Let me show you the three ways it bites, because naming them is most of the cure.

Failure 1: the echo loop

You receive a POS webhook: item 88 is now unavailable. You dutifully write available = false and, because your own writes are supposed to propagate, you push that change… back to the POS. The POS sees a write, and webhooks you: item 88 changed. You apply it, and push again.

  [ POS ] ──"item 88 unavailable"──▶ [ you: write + push back ]
     ▲                                          │
     └──────────"item 88 changed"───────────────┘   ← infinite echo

Nothing is wrong with either system individually. The loop is emergent — it exists only in the seam between them. In production this shows up as a column that flickers, a webhook log that never goes quiet, and rate-limit warnings from the POS at 2 a.m.

The fix is to make your writes attributable. When you apply an inbound change, tag it as originating from the POS and do not echo it back out. Only changes that originate locally get pushed:

async function applyInbound(event) {
  await db.items.update(event.itemId, {
    available:    event.available,
    last_source: 'pos',          // this change came FROM the pos
    pos_version:  event.version, // remember what the pos told us (see Failure 2)
  });
  // note: NO push back to the POS. We do not propagate what the POS just told us.
}

async function applyLocal(itemId, patch) {
  await db.items.update(itemId, { ...patch, last_source: 'local' });
  await pos.push(itemId, patch);  // only LOCAL-origin changes go outbound
}

That single “who caused this change” tag breaks the loop. It’s the integration version of a rule that keeps showing up: know who owns the write.

Failure 2: out-of-order and duplicate events

Webhooks are delivered at-least-once and not necessarily in order — a delivery guarantee most providers, Stripe included, state explicitly in their own documentation. The POS sends you “available → false” then “available → true” a second apart, and the network hands them to you reversed. If you blindly apply the last one you processed, you end up with false — permanently wrong, and nothing looks broken.

You cannot fix this with timestamps you generate on receipt (they reflect arrival, not truth). You need a monotonic version from the source and the rule “never apply an event older than what I’ve already applied”:

async function applyInbound(event) {
  const row = await db.items.findById(event.itemId);

  // Idempotent AND order-safe: only move forward.
  if (row.pos_version != null && event.version <= row.pos_version) {
    return;  // stale or duplicate — we've already seen this or newer. No-op.
  }

  await db.items.update(event.itemId, {
    available:   event.available,
    last_source: 'pos',
    pos_version: event.version,
  });
}

If the source doesn’t give you a version or updated-at you can trust, you’re reduced to reconciliation-only (below) for that field — which is a perfectly honest answer, and better than pretending the event stream is ordered when it isn’t. This is the same shape as making payment webhooks idempotent: the event stream is unreliable by contract, so correctness has to live in how you apply, not in the events arriving perfectly.

Failure 3: the fields don’t mean the same thing

This is the one that burns a week. Two systems rarely model the same concept identically, and the mismatches are silent:

• Your platform has availability (is this sellable right now?). The POS has stock count and a separate tracking mode (is stock even being tracked?). An item with stock = 0 but tracking = off is available, not out of stock. Map “stock 0” straight to “unavailable” and you’ll hide sellable items.
• The POS “deletes” an item, then “revives” it a day later under the same external id. If your import treats delete as terminal, the revive silently does nothing.
• A field you assumed was a boolean is a tri-state; a price you assumed was minor-units is major-units for one region; an id you keyed on gets reused.

There is no clever code for this. The fix is a written field ownership map — for every field, exactly one system is the authority, and you translate meanings explicitly at the boundary:

Field	Owner	Rule
Catalog (name, description, category)	Platform	POS never overrides; local is truth
Price	Platform	pushed to POS; inbound price events ignored
Stock / availability	POS	POS is truth; translate stock+tracking→available
Item existence (create/delete/revive)	POS	delete is soft; a later revive must re-activate

Once ownership is explicit, most “conflicts” evaporate — they were never conflicts, just two systems editing fields the other didn’t own. A change to a platform-owned field ignores whatever the POS says about it, and vice versa. Write this table before the code; it is the actual design.

The part that makes it correct: the reconciliation sweep

Everything above keeps you fresh — reacting to live events. None of it keeps you correct, because events get dropped. At-least-once is a floor: a webhook can fail every retry and vanish, a deploy can drop an in-flight batch, a bug can skip an apply. Over weeks, the two systems drift, and no single event tells you they have.

So the load-bearing piece is a periodic full reconciliation — a scheduled job that ignores the event stream entirely, pulls the full state of both sides, and repairs the delta by ownership:

// Runs on a schedule (this is a perfect job for a Postgres-backed
// outbox/cron worker — see the durable-queue post).
async function reconcile() {
  const ours   = await db.items.allByExternalId();
  const theirs = await pos.listAllItems();          // full snapshot, not events

  for (const [extId, posItem] of theirs) {
    const local = ours.get(extId);

    if (!local)        { await createFromPos(posItem); continue; }   // they have, we don't
    // Repair ONLY fields the other side owns:
    if (local.available !== translateAvailability(posItem)) {
      await db.items.update(local.id, { available: translateAvailability(posItem) });
    }
  }
  for (const [extId, local] of ours) {
    if (!theirs.has(extId) && local.last_source !== 'local') {
      await flagOrphan(local);   // we have it, POS doesn't — human decision, don't guess
    }
  }
}

Run it nightly (or hourly for volatile data). The live webhooks give you low-latency updates; the sweep guarantees that even if every webhook for a day was lost, you converge to correct within one cycle. This is the transactional outbox mindset — durability first, delivery second — applied to a whole integration: don’t trust the stream to be complete — periodically check reality and repair.

Two hard-won rules for the sweep: repair only fields the other side owns (or two sweeps will fight the same field forever), and never auto-delete on “the other side doesn’t have it” — flag it for a human. A dropped event and a real deletion look identical from one snapshot; guessing wrong deletes a live menu item.

Audit everything at the boundary

One operational note that paid for itself ten times over: log every inbound and outbound event, raw, before you interpret it. When a price is wrong or an item vanished, the only way to answer “did they send us bad data, did we misapply good data, or did our push do it?” is to replay the boundary. A tiny append-only integration_events table (direction, external id, raw payload, applied-result, timestamp) turns “no idea what happened” into a five-minute query. Integrations fail in the seam; instrument the seam.

What actually broke in production

• A column-meaning mismatch marked good items unavailable. We mapped the POS stock field to availability without accounting for tracking = off, so every untracked-but-sellable item went dark. Caught only because a restaurant called about missing menu items. Fix: the explicit translation function, plus a reconciliation pass that re-derived availability correctly for the whole catalog.
• delete-then-revive did nothing. Our importer treated a POS delete as final and dropped the row’s linkage; the next day’s revive created no change because our code had stopped tracking the external id. Fix: soft-delete with the external id preserved, and treat a revive as reactivation.
• The echo loop we didn’t notice for days — because each hop was individually valid, only the POS’s rate-limit emails revealed it. Fix: the last_source tag, and a boundary log that made the loop visible at a glance.

What I’d still improve

• Reconciliation is O(catalog) every run. Fine at this scale, wasteful at 10×. I’d move to a hashed/versioned diff — compare a per-item fingerprint and only fetch full detail for mismatches — so the sweep cost tracks the delta, not the total.
• Ownership is still partly in my head. It should be declarative config (a table of field → owner → translator) that both the live-apply path and the sweep read from, so they can never disagree about who owns what. Right now the two paths encode it separately, which is exactly the kind of drift this whole post is about.

Where to read more

The vocabulary here is not mine — eventual consistency and idempotent message handling are long-standing patterns, and it’s worth reading them stated formally once, because the formal version makes clear which guarantees you are not getting.

Neighbouring posts: making payment the source of truth is the single-writer version of this problem, the Postgres outbox is how the outbound half stays durable, and the RBAC model covers who is allowed to trigger a write across the boundary in the first place.

FAQ

What is the difference between syncing and reconciling two systems? Syncing assumes one true state you mirror by reacting to change events. Reconciling accepts that both sides write, assigns every field a single owner, and periodically compares full snapshots to repair the delta. Events keep you fresh; only the sweep makes you correct.

How do you stop a webhook echo loop? Make your writes attributable. Tag each change with where it originated, and only propagate changes that originated locally. An inbound change gets applied and stops there. Without that tag, applying an inbound event triggers an outbound push that triggers another inbound event, forever.

How do you handle out-of-order webhooks? Use a monotonic version supplied by the source and refuse to apply anything at or below the version you have already recorded. Timestamps you generate on receipt reflect arrival, not truth, so they cannot order events for you. Without a trustworthy version, that field is reconciliation-only.

How often should a reconciliation sweep run? Match the cycle to how much staleness the data can tolerate — nightly for a catalog, hourly for volatile availability. The property you want is that even if every event for a full day were lost, you converge to correct within one cycle.

Should reconciliation auto-delete records the other side is missing? No. A dropped event and a genuine deletion look identical in a single snapshot, so guessing wrong removes live data. Flag the discrepancy for a human instead, and only automate the direction where being wrong is recoverable.

What is a field ownership map? A written table naming exactly one authoritative system per field, plus how to translate that field’s meaning across the boundary. Most apparent conflicts are not conflicts at all — they are two systems editing fields the other does not own. Write it before the code.

---

The one idea to take away

You can never sync two systems that both mutate and both notify — the symmetry is a lie that produces echo loops and silent divergence. You can only reconcile: give every field one owner, apply inbound events idempotently and in-order-safely, tag your own writes so you don’t echo them, and run a periodic sweep that treats reality — not the event stream — as the thing to converge to. Events keep you fast. Reconciliation keeps you right.