Every long-lived business ends up with the same archaeology. An ERP that has been the system of record for decades, a storefront that customers actually touch, and between them a strata of integration scripts, each written by someone solving one urgent problem, none of them aware of the others.
We went looking in one such estate recently and found four overlapping attempts at the same integration. Each had its own authentication, its own retry logic, its own idea of state, its own error handling. None of them composed. Adding a fifth flow meant starting again.
The interesting question is not how to write a better script. It is why writing a script was ever the unit of work.
N times M, or N plus M
Point to point integration has a mathematics problem. Every system you connect to every other system is another bespoke link to build and maintain. Five systems is ten links. Add a sixth and you are at fifteen.
A canonical hub changes the arithmetic. Each system speaks once, to a shared model in the middle. Adding a system is one more spoke, not one more link per existing system.
That is not a new idea. Enterprise integration patterns have said it for twenty years. What makes it worth writing about is where you draw the line between the platform and the things plugged into it.
The bet: dumb connectors, smart platform
Here is the design decision everything else hangs off. A connector is allowed to be boring. It knows one external system and one kind of record. It converts that system’s shape into the shared model and back, and it does its own reads and writes. That is all.
Everything hard is written once, in the platform: identity mapping, deduplication, retries, ordering, loop suppression, multi-step orchestration and the audit trail.
The test of that split is simple. If adding a new system requires touching the platform, the split is wrong. If it requires a small adapter, a field mapping file and a registration, the split is right.
The bug hiding in the primary key
One detail from the design work is worth pulling out, because it is the kind of thing that quietly breaks production for years.
The obvious key for a product is its product code. It is human readable, it is in every spreadsheet, everyone in the business says it out loud. It is also mutable. Codes get prefixed when an item is discontinued. They get tidied. They get reused.
Key your integration on the code and every one of those edits looks like a delete followed by the arrival of a brand new product. Catalogue entries vanish. History detaches. Nobody files a bug, because each individual case looks like someone in the warehouse doing something odd.
The canonical model keys on the system’s immutable internal identifier instead, and treats the code as what it actually is: an attribute that changes over time. Unglamorous, and worth more than most features.
At-least-once delivery, exactly-once effect
Distributed messaging gives you at-least-once delivery. Messages get redelivered. Any design that assumes otherwise is wrong in a way that only shows up under load, at the worst moment.
So correctness cannot come from the bus behaving. It comes from the writes being idempotent and from a dedup store keyed on the content, so the same change applied twice is a no-op rather than a duplicate.
Ordering gets the same treatment. Rather than serialising everything, messages are grouped so that changes to a single record are handled in order while unrelated records flow in parallel. You get the safety of a queue per record without the throughput of one queue for everything.
Stopping the echo
Bidirectional sync has a failure mode that delights nobody: you write a change into the storefront, the storefront announces that something changed, your integration picks that up as new news and writes it back, forever.
The design stops it in four layers, deliberately ordered from structural to defensive. First, changes to fields a system does not own are dropped outright. Second, an echo table records the exact result expected from a write-back and suppresses the matching inbound event once. Third, a content hash catches any change whose end state matches what was already propagated. Fourth, idempotency keys catch whatever is left.
Only the first layer is really doing the work. The other three exist because the first one will eventually have a gap, and a loop in production is expensive to notice.
The part we are most pleased with
None of this is built yet. This is a greenfield design, and there is no production code.
What exists instead is a simulation you can run on a laptop. The connector and pipeline code in it is production shaped. Only the message bus, the state store and the two external systems are local stand-ins. One command narrates five scenarios end to end: the happy path, the same event delivered twice, a fan-out that re-resolves many records and pushes only the deltas, a failure that retries and then dead-letters, and a write-back echo that gets suppressed while genuine out-of-band drift is still caught.
Each scenario maps to a section of the design document. So the architecture is demonstrated rather than asserted.
That distinction matters more than it sounds. A design document says the platform will handle redelivery. A simulation delivers the same message twice in front of you and shows the second one changing nothing. One of those survives contact with a sceptical engineer, and it is not the document.
It also changes what a review meeting is. Instead of talking about whether the approach is sound, you run it and watch where it bends.
Where the agents come in
There is a strand of this design we did not expect to enjoy as much as we do.
Not every consumer of a shared data hub wants to copy data somewhere else. Some want to make a judgement: look at what changed, propose an adjustment, and have a person or a policy agree to it before anything is written. Replenishment is the obvious case. So is exception handling.
A sync consumer transforms; its output is a pure function of its input. A deciding consumer manufactures new values, which means its output cannot be trusted by construction. So it does not get to write. It proposes, and its proposal passes an explicit gate before it reaches the same narrow, per-record write allowlist everything else uses.
Two rules keep it honest. The non-determinism stays inside the deciding component, so the platform’s guarantees are untouched: from its point of view this is just another producer of write-back messages, subject to the same allowlist and the same audit. And the approval travels with the write, so the audit trail answers who decided this and why, not merely what was written.
If that sounds familiar, it is the same instinct as the credential broker draft we wrote about: let the autonomous thing propose, but do not let it hold the keys, and keep a record of who agreed.
Why we like this work
Integration is not glamorous. Nobody puts a canonical model on a homepage. But it is where most of the real difficulty lives in a business that has been running long enough to accumulate systems, and it is the difference between a company that can add a channel in a fortnight and one that cannot add one at all.
The satisfying part is that the fix is not heroics. It is drawing one line in the right place, writing the hard parts once, and then proving it runs before anyone commits a year to building it.
We will write again when the first flow is in production. Then it will have earned a green light.