System Design: Building an Attribution Pipeline That Marketers Trust
Event schemas, idempotency, and the boring decisions that keep a martech data pipeline from quietly lying to everyone.
Attribution pipelines fail quietly. Nobody pages you when a campaign is under-reported by 8% — the dashboard just looks slightly wrong forever, and eventually someone makes a budget decision based on bad numbers.
Start from the event, not the dashboard
The instinct is to design backward from the report you want to show. Resist it. Design forward from a single, strict event schema: a stable event name, a deduplication key, a timestamp with a defined source of truth (client-observed vs. server-received), and an explicit schema version. Everything downstream — attribution windows, channel grouping, CRM joins — is a transformation on top of that stream, never a patch to it.
Idempotency is the whole game
Marketing events arrive late, arrive twice, and arrive out of order — retried webhooks, offline mobile queues, ad-platform callbacks that fire on their own schedule. If your ingestion isn't idempotent on the dedup key, every retry becomes a phantom conversion. Concretely: an upsert keyed on (event_id, source) at the ingestion boundary, with the rest of the pipeline treating storage as append-only and immutable.
Attribution windows are a business decision, not a technical one
Last-click, first-click, and multi-touch models will all disagree with each other on the same underlying data. Pick a default model, but keep raw touchpoints intact so you can recompute under a different model later without re-instrumenting anything. That single decision — store raw events, compute attribution as a view — is what separates a pipeline that survives a strategy change from one that requires a rewrite.
What actually breaks in production
In my experience it's rarely the attribution logic itself. It's clock skew between client and server timestamps, PII scrubbing rules changing after launch, and CRM record matching silently degrading as email formats drift. Build monitoring around volume anomalies and match-rate, not just uptime.