Large programs don’t usually fail because someone forgot a Jira ticket.
They fail because everyone is busy doing the right things, while quietly operating on different assumptions and definitions of one.
This is the story of a major system migration where the biggest threat wasn’t the cloud architecture, the tooling choices, or the release plan. It was a single, unspoken belief:
“The data will be there.”
It wasn’t.
The program
The goal was to move a reporting capability off a decades-old, on-premise stack and onto a modern, cloud-native platform fed by dozens of upstream source systems and supporting a large catalogue of datasets used for reporting, reconciliation, and management views. Modernize the engine, improve scalability, move toward cloud-native patterns without breaking a capability the business depended on every month.
The team was busy in the best possible way
For the first few months, the team did what strong teams do:
- Gathered deep detail on the current state
- Built a release plan spanning the next ten to twelve months
- Held architecture discussions, which data-lake table format to standardize on, where AI could accelerate data mapping and the decoding of legacy transformation logic, how to handle extraction and egress
From the outside, everything looked healthy. Inside, we were walking into a trap.
The silent assumption
Somewhere along the way, the team made a reasonable assumption: the new platform would keep receiving feeds from the same upstream systems as before.
Then, in a review with enterprise architects and senior leadership, we got a different directive: source the data from the enterprise data lake instead. Leadership assumed this would be simple — the source systems were “already sending their data to the lake,” so “everything you need is already there.”
It sounded logical. It was also wrong.
Expectation versus ground reality
Once we actually looked into the lake, the picture was very different from the one in the room:
- Only a small fraction of the datasets we needed were actually there.
- A large share of the rest had been deliberately kept out to avoid the storage, transfer, and compute costs of landing them.
- A meaningful chunk didn’t exist in any usable form anywhere, which meant standing up new ingestion pipelines from scratch.
This wasn’t a backlog item. It was a program-level change in reality.
Why it was so dangerous
The risk wasn’t just “we need more data.” It was the cascade:
- Timeline. Waiting on the data-engineering teams to ingest and validate what was missing would stall the build-and-test plan — workstreams idling or thrashing, dependencies stacking up. A three-to-six-month setback, easily.
- Cost. A large portion of the delivery team were contractors. If they kept building against guessed-at fields, development would rest on assumptions, testing would fail late, rework would spike, and carrying costs would climb fast.
- Delivery. The subtle one: a team can produce a lot of output while missing the foundational inputs needed to validate any of it. Velocity without truth.
The real problem wasn’t the data
My role put me across several of the workstreams, and what stood out wasn’t the gap itself — it was the disconnect:
- Leadership’s view: “The lake already has what you need.”
- The team’s view: “We assumed we’d keep the direct feeds. We didn’t know the lake’s history or the directives behind it. We flagged missing data but never escalated it as a program-level risk.”
That’s the actual failure mode:
Not missing data. Missing shared understanding.
What I did: escalate fast, quantify faster
Once the gap was visible, I treated it as a major program risk, not a technical task. I flagged it to the program manager and the delivery lead, and opened cross-team work with data engineering to quantify exactly what was missing, separate what was intentionally excluded from what was genuinely net-new, and get realistic lead times for ingestion and validation.
The goal wasn’t to panic early. It was to avoid the most expensive failure in large programs: discovering the truth too late to change course cheaply.
The lesson: communication isn’t updates, it’s alignment
We had no shortage of communication — meetings, architecture sessions, planning discussions, documentation. What we lacked was alignment on one question:
“Where is the system of record for data sourcing in the target state?”
In a large program, that kind of misalignment is brutally expensive, because by the time it surfaces the contracts are signed, the dates are socialized, the teams are staffed, and the architecture is locked in.
What I’d do differently
- Make assumptions a first-class artifact. Keep an assumptions register: what the team believes, who validated it, what proves it true, and what happens if it’s false.
- Validate critical dependencies before optimizing the solution. Before debating table formats, confirm data availability, lineage, ownership, lead times, and access.
- Don’t confuse “the platform exists” with “your data exists there.” Enterprise architecture says “it’s all in the lake.” Reality is usually: some of it is, some was deliberately left out, and some doesn’t exist anywhere usable.
- Escalate gaps as outcomes, not tasks. Not “Dataset X isn’t available,” but “This blocks integration testing for Release 1 and adds a three-to-six-month delay unless mitigated.” Outcomes move leadership. Task lists don’t.
Closing thought
If you manage large programs, here’s the uncomfortable truth: the biggest risks aren’t in the work you can see. They’re in the assumptions no one is saying out loud.
Effective communication isn’t about frequency. It’s about making sure everyone is aligned on the same outcomes — and the same reality.
The Alignment Gap Checklist
A practical tool to surface silent assumptions before they turn into schedule slips, cost overruns, and rework. Run it at kickoff, before major design decisions, and at every phase gate.
1. Outcomes — Can everyone state the problem in one sentence? Is “success” measurable (accuracy, SLAs, compliance, cutover criteria)? What are the non-negotiables, and what are we explicitly not doing? Where do leadership’s priorities differ from the delivery team’s? (Write both down — the differences are the point.)
2. Source of truth — What is the system of record for each critical input: data, requirements, metrics, definitions? Where will the target solution actually source its data — direct feeds, enterprise lake, or a curated layer? Who owns that platform, and what were its historical constraints (cost controls, intentionally excluded domains)? For each input: is it assumed to exist, or confirmed to exist with evidence?
3. Data readiness — For each critical dataset: available today? Complete enough for build and test? Required attributes present (keys, hierarchies, effective dates, audit fields)? Lineage documented source-to-target? Refresh cadence and latency understood? Access approved and working? If missing — intentionally excluded, or net-new ingestion? Track the top ~20 datasets that drive 80% of scope as a simple red/amber/green list.
4. Dependencies — What are the top dependencies that can delay the program? Which are external to your team? What’s the lead time for each, and is there a committed intake and priority agreement? What happens if one slips four to eight weeks — and what’s the fallback (parallel path, phased scope, temporary feeds, synthetic test data)?
5. Decision rights — Who makes the final call on data-sourcing strategy, platform standards, tooling, compliance exceptions, and release gating? What’s the escalation path and expected turnaround? Which decisions are reversible, and which must be made before build starts?
6. Delivery model — Especially with contractors or partners: who owns requirements and acceptance versus implementation and documentation? Are people building on verified inputs or assumptions? Enforce a definition-of-ready for data-dependent work, and tie staffing ramp-up to milestones so you’re not burning budget while blocked.
7. Risk visibility — Is there a single, shared risk register everyone uses? Do risks describe outcome impact (time, cost, scope), not just tasks? Are probability, impact, owner, mitigation, and date-first-identified all tracked? Are the top risks reviewed with leadership on a fixed cadence — and are “unknown unknowns” being actively hunted?
8. The assumption stress test — the ten-minute exercise that saves months. For each major assumption: What are we assuming? What evidence proves it? Who agrees, by name? If it’s false, what breaks first? How soon will we know? What’s the cheapest way to test it this week?
If you can’t answer those last six questions, it isn’t an assumption. It’s a gamble.