The premise
Store all data in open formats on low-cost object storage; let any engine — BI, SQL, ML, streaming — read one governed copy instead of many siloed extracts.
A data lake stores everything you collect — structured, semi-structured, and raw — in cheap, open cloud storage, so many tools can use one copy of the truth. The lakehouse adds the reliability and governance of a warehouse on top. This briefing lays out the philosophy, the architecture, who's building it, and how it maps to SSM's own enterprise data model.
Store all data in open formats on low-cost object storage; let any engine — BI, SQL, ML, streaming — read one governed copy instead of many siloed extracts.
Cloud object storage plus open table formats (Delta Lake, Apache Iceberg, Hudi) matured into the lakehouse — warehouse reliability on lake economics, without lock-in.
One governed source behind Epic, Databricks, and Mission Intelligence — so Epic reporting and Databricks never tell different stories, and every team works from the same enterprise data model.
The centerpiece · Watch & learn
Pick a talk — it plays right here. Every video below was verified as real and embeddable. These are canonical talks and explainers from the people and platforms that defined the data lake and the lakehouse.
Select a talk
What a "data lake" actually means
A warehouse uses schema-on-write: you model and clean data before it lands, which is reliable but rigid. A lake uses schema-on-read: you store anything cheaply now and impose structure when you query it — flexible, but it pushes the data-quality and governance problem downstream.
The term was coined in 2010 by James Dixon, then CTO of Pentaho, contrasting a tidy data mart with a body of water in its natural state. The danger he and others later named: without governance, a lake silts into a data swamp — full of data nobody trusts. The lakehouse is the answer: warehouse-grade management (transactions, schema, governance) directly on open lake storage.
"If you think of a datamart as a store of bottled water — cleansed and packaged and structured for easy consumption — the data lake is a large body of water in a more natural state." — James Dixon, CTO of Pentaho, the post that coined "data lake," Oct 2010
Structured, modeled, governed before load. Inmon/Kimball heritage.
Any data, raw, in open files on cheap object storage.
Warehouse management on lake storage via open table formats.
Framed for SSM Health
When every store solves supply on its own, you get sprawl and duplication. A shared, governed lake is the central distribution hub — stores still serve their communities, but the supply behind them is finally shared and interchangeable.
The point of consolidating to a lake is that Epic reporting and Databricks tell the same story. Mission Intelligence is the front door; the governed lake is the shared shelf behind it.
Lake vs. swamp is a governance decision, not a technology one. A catalog, ownership, and quality rules are what keep the lake trustworthy at scale.
Land raw once; model it into dimension/fact and enterprise models as use cases demand. Flexibility up front, structure where it pays off.
Open storage and table formats mean one copy of data serves many tools — and the platform choice stays reversible instead of locked in.
"The data lake was a schema-on-read architecture that enabled the agility of storing any data at low cost, but … punted the problem of data quality and governance downstream." — Armbrust, Ghodsi, Xin & Zaharia, Lakehouse, CIDR 2021
The real arc
Every milestone below is dated to its first publication and links to a primary source. This is the genuine lineage of the data lake and the lakehouse — no revisionism.
How it's actually built
The dominant lakehouse pattern refines data in stages — bronze, silver, gold — so quality, structure, and business meaning increase at every hop. It maps almost one-to-one onto SSM's own raw → dimension/fact → enterprise-model path.
Raw data landed in its original form from source systems, message buses (Kafka/Kinesis), and object storage. Appended incrementally; minimal cleanup. The single source of truth for what arrived.
Cleansed, deduplicated, validated, and joined into conformed dimension and fact tables. At least one trustworthy, non-aggregated representation of each record.
Highly refined, aggregated, dimensionally modeled data aligned to business logic — the measures and definitions that power dashboards, executive reporting, and ML.
Source systems → Bronze (raw) → Silver (Dim/Fact) → Gold (enterprise models) → Mission Intelligence & BI
Cheap, near-infinite object storage (S3, ADLS, GCS) is decoupled from elastic compute. You pay for compute only when querying, and many engines read the same copy.
A transaction log over open Parquet files adds ACID transactions, time travel, and schema evolution — turning a pile of files into a reliable, governed table.
A catalog (Unity Catalog, AWS Glue/Lake Formation, Apache Polaris) provides discovery, lineage, and fine-grained access control — the guardrails that keep a lake from becoming a swamp.
One copy of data serves both scheduled batch and near-real-time streams. Lambda splits the two; Kappa unifies them over a replayable log — fewer pipelines, fresher data.
Commodity storage plus elastic compute is dramatically cheaper than duplicating everything into a proprietary warehouse — one open copy, sized to demand.
Snapshots and an audit log let you query data as of a point in time, roll back bad writes, and reproduce a number — the reliability a regulated health system needs.
The three open table formats that made the lakehouse possible
ACID over Parquet via a transaction log; time travel, schema enforcement, MERGE/UPDATE/DELETE, unified batch + streaming. SSM's Databricks lakehouse is built on it.
Hidden partitioning, snapshots, and full schema/partition evolution. Now the cross-vendor interoperability standard; its creators' company, Tabular, was acquired by Databricks in 2024.
"Hadoop Upserts Deletes and Incrementals" — built for fast upserts, change-data-capture, and incremental processing at streaming scale.
Who is building this
A map of where the technology, the open standards, and the ideas are coming from.
Read the primary sources
Papers, founding blog posts, and authoritative vendor docs — each link checked to resolve. Where an official page blocked our automated checker, it is flagged so you can confirm it in a browser.
In their own words
Real, attributed quotes — speaker, venue, and date — each linked to where it can be verified. Nothing here is paraphrased or invented.
Traceability
This was researched before it was written, with the SSM BIA "Slide Library" executive deck as the narrative backbone. Every talk, paper, and quote was traced to a primary source and link-checked. The notes below state exactly what was verified — and what to double-check.
Generated live, by the agent
Anything the embedded agent creates appears here — images (Imagen 4) or short video (Veo). Open the agent (bottom-right) and ask: “generate an image of a governed healthcare data lakehouse,” or “make an AI video for the hero.”