A primer on the modern data estate
What is a data estate?
A data estate is everything an organization uses to hold, move, govern, and use data: the data itself, the systems that store it, the pipelines that move it, and the rules and people that govern it. The clearest way to see one is to follow a single record through it.
At 7:42 on a Tuesday evening, Emily buys a two-person tent on the website of Fern & Pine, a fictional retailer of hiking and camping gear that started in 2004 and now runs fourteen stores. Emily is fictional too, and so is everything that happens to her order. The checkout page writes one row to an orders table. By the next morning, the sales dashboard shows Tuesday’s revenue up by $249. A week later, the website suggests tent pegs to Emily.
One record made all three things happen. To do that, it moved through a web application, a payment provider, a stream, files, a warehouse, cleaning steps, a metric definition, a dashboard, and a machine learning model. That chain, and everything around it, is a data estate.
This primer follows order 48291 from checkout to dashboard to recommendation model. At each stop we name the layer, what it does, and what it costs.
1. What a data estate is
A data estate is everything an organization uses to hold, move, govern, and use data: the data itself, the systems that store it, the pipelines that move it, and the rules and people that govern it. Fern & Pine’s estate includes its checkout database, Stripe’s payment records, a point-of-sale system from 2009, the dashboards, the recommendation model, and the rule that support staff see an order but marketing sees only totals.
Three related terms cause confusion. A data platform is the technology part: the storage, the pipelines, the tools. A data architecture is the design of that platform: which systems exist, how they connect, and why. A data strategy is the plan: which problems data will solve, in which order, with which people and budget.
The estate is the whole thing, and the term refuses to draw a line between the technology and the humans. Most data failures happen on that line.
2. Why the data estate matters now
Four forces push companies to manage their data estate: growth, cloud, regulation, and AI.
Growth. IDC estimates that the world created, captured, replicated, and consumed 181 zettabytes of data in 2025, and projects 394 by 2028 (IDC, via TechTarget, July 2026). In 2018 it was 33 zettabytes (IDC and Seagate, Data Age 2025).
The world’s data creation per year
33
2018
181
2025
394
2028
projected
Sources: 2018: IDC and Seagate, Data Age 2025. 2025 and 2028: IDC Global DataSphere Forecast, as reported by TechTarget, July 2026. The 2018 point comes from an older IDC study than the other two.
The world’s annual data creation in zettabytes, with 2028 a projection.
Cloud. Gartner forecasts worldwide public cloud spending to grow 21.3 percent in 2026 and to reach roughly $1.48 trillion by 2029 (Gartner, September 2025). Cloud services bill by use, and cheap storage changed behavior: “keep the raw data, you might need it,” which leaves more copies than any team can track.
Regulation. Since May 2018 the GDPR has made any organization that handles the personal data of EU residents responsible for it, and regulators have issued more than six billion euros in fines (GDPR Enforcement Tracker). The EU AI Act, in force since August 2024, adds duties for AI systems that touch people. These laws demand that you know what data you hold, where it is, and who may see it.
AI. Gartner predicted in 2024 that at least 30 percent of generative AI projects would be abandoned after the proof of concept by the end of 2025, listing four reasons: poor data quality, weak risk controls, rising costs, and unclear business value (Gartner, July 2024). RAND’s report cites estimates that more than 80 percent of AI projects fail, about twice the rate for ordinary IT projects, and its interviews with 65 practitioners ranked an unclear problem as the top cause, with data problems second (RAND, 2024). A dashboard tolerates a small error in a table. A model learns the error and repeats it.
3. A short history: warehouse, lake, lakehouse
The modern estate grew in three generations, and each solved a real problem while creating the next one.
The data warehouse. In the late 1980s and 1990s, companies wanted answers across the business: sales by region, margin by product. The operational databases that run the business were bad at this. They are tuned for many small transactions, and an analyst’s long query slowed checkout. Bill Inmon and Ralph Kimball shaped the answer: copy data into a separate system, the data warehouse, arranged into tables built for questions.
What it solved: reporting stopped touching the systems that run the business. What it caused: a warehouse demanded a fixed schema, a plan of every table and column, agreed before data could load. Adding a source meant redesign, and the reshaping work, ETL, became the biggest cost. Photos and call recordings stayed out.
The data lake. Around 2010, as companies filled Hadoop clusters with data, James Dixon named the data lake in a blog post. A data lake accepts any data in any format at low cost, and you decide the structure later, when someone reads it.
What it solved: cost and flexibility, with no committee argument about schema first. What it caused: lakes filled with data nobody could find or trust. The lake created the governance problem the term “data estate” now names.
The lakehouse. Around 2020 a paper by Databricks researchers described a third pattern. A lakehouse keeps the lake’s cheap storage but adds warehouse behavior on top: tables with schemas, transactions, and SQL queries. An open table format is a standard that organizes files in object storage into database-style tables, plus a small metadata log. Metadata is data about data, and the log records which files hold which rows. The two main formats are Apache Iceberg, started at Netflix, and Delta Lake, open-sourced by Databricks in 2019.
What it solved: one copy now serves SQL analytics and machine learning for any engine that reads the format. What it caused: formats, catalogs, and engines multiplied. The market is consolidating, not settled: Snowflake released Polaris, a free Iceberg catalog now known as Apache Polaris, in 2024, and Databricks announced native Iceberg support in Unity Catalog in June 2025. Iceberg is becoming the common ground.
Three generations, each solving one problem and creating the next
The data warehouse
late 1980s to 1990s
Solved: reporting no longer touched the systems that run the business.
Created: rigid schemas, slow redesign, ETL as the biggest cost.
The data lake
around 2010
Solved: cheap storage for any data, with structure decided at read time.
Created: data nobody could find or trust.
The lakehouse
around 2020
Solved: one open copy serves SQL analytics and machine learning.
Created: formats and catalogs that someone must own.
Three generations, each solving one problem and creating the next.
Fern & Pine runs all three patterns at once: a warehouse from 2012 in the back office, a lake from 2019, a cloud warehouse from 2023, and a lakehouse it is building.
4. The layers
Now we can draw the estate: a stack of layers through which data flows, and we walk them watching order 48291.
The six layers and the five functions
Governance
Security
Serving
dashboards, models, reverse ETL, data products
Semantic layer
one agreed definition per metric
Transformation
ELT, and the bronze, silver, gold tables
Storage
S3 raw zone, cloud warehouse, lakehouse
Ingestion
batch loads, CDC, streaming
Sources
checkout, Stripe, point of sale, scanners, clickstream
Monitoring
Orchestration
Cost control
The six layers data flows through, wrapped by the five functions that serve them all.
Sources
A source is any system that first creates or captures data. Five kinds cover most estates.
Operational databases run the business itself: orders, customers, stock, staff. Fern & Pine’s checkout service writes order 48291 as a row in its PostgreSQL database. The order is born there.
SaaS applications are software your company rents, such as Stripe for payments or Salesforce for sales. You reach their data through an API, a documented set of requests a program can make. Stripe holds Emily’s payment.
Event streams are live feeds of small records called events. Every page Emily visited emitted a clickstream event, and a message broker such as Apache Kafka holds them until consumers read them.
Files still matter: the point-of-sale system writes a daily sales file, and the shipping partner drops a daily file of returns. Files break pipelines when someone renames a column.
Devices create the last kind: barcode scanners in the stores, wired into that point-of-sale system from 2009.
You control your own systems. You do not control the vendor’s API or the partner’s file, so ingestion must capture data exactly as it arrives.
Ingestion
Ingestion copies data from sources into the estate’s storage. Three patterns cover most cases.
A batch load copies a set of records on a schedule, usually nightly. Batch is simple and cheap. Its cost is freshness: the data is a day old before anyone sees it.
Change data capture, CDC, watches a database’s own log and copies only the rows that changed. Most operational databases keep a write-ahead log, a journal of every change. A tool such as Debezium reads that log. When Emily’s order is saved in PostgreSQL, the log records it, and Debezium writes an order_placed event to Kafka in seconds. Its cost is operations: someone must run and watch it.
Streaming moves events one at a time as they happen. Emily’s clickstream flows to Kafka continuously, giving fresh data for models and alerts. Its cost is complexity: duplicates, late events, and ordering become engineering problems.
The choice is freshness against complexity. Fern & Pine streams what must be fresh and batches what is fine overnight: Stripe’s export, the returns file, and the point-of-sale daily file.
Batch and streaming
Batch
order
order
return
return
payment
collected all day, delivered together at 2 a.m.
Cost: the data is a day old before anyone sees it.
Streaming
each one delivered the moment it happens
Cost: duplicates, late events, and ordering are yours to handle.
Batch lands records in one scheduled group; streaming arrives one record at a time.
Storage
Storage is where data rests and where most queries run. Modern estates use three kinds, often side by side.
Object storage holds files cheaply and durably: Amazon S3, Azure Blob Storage, or Google Cloud Storage. A data lake is object storage used for analytics. Fern & Pine’s raw zone is a set of S3 folders where every ingested event lands as a Parquet file. Parquet is a columnar format: it stores data column by column, which makes queries fast and files small. From there, loads copy the raw files into the cloud warehouse, where bronze, silver, and gold are built.
A data warehouse is a managed system built for SQL analytics: Snowflake, Google BigQuery, Amazon Redshift. It holds its own copy, tuned for fast aggregation, the summing of many rows into totals, and you pay for the compute it uses.
A lakehouse removes the copy. Open table formats such as Iceberg and Delta Lake organize files in object storage into proper tables with schemas and version history. Any engine that reads the format can query the same files. Fern & Pine is moving this way: one table of orders, held as Iceberg files in S3, queried by warehouse and machine learning jobs. The 2012 system still runs the Monday report in the back office.
The choice is where you pay: object storage is cheap but you bring your own engine, a warehouse is convenient but compute bills for every query, and a lakehouse is open but you run the catalogs.
Transformation
Raw data is hard to use: sources disagree on names, formats, and duplicates. Transformation turns raw data into reliable tables, and two design choices live here.
The first is ETL or ELT. ETL, extract, transform, load, cleans data on separate machinery, then loads finished tables. ELT, extract, load, transform, lands raw data first, then transforms it inside the data warehouse with SQL, using a tool such as dbt. Cloud warehouses made ELT the default. The cost moves from ETL machines to warehouse compute.
The second is the medallion pattern, named for bronze, silver, and gold. Bronze tables hold raw data exactly as it arrived. Silver tables are cleaned, deduplicated, and joined, one row per real thing. Gold tables hold totals and metrics ready for dashboards. Each layer costs storage and compute. What it buys is traceability: when a number looks wrong, you walk back down the layers and see where it broke.
Order 48291 passes through all three. In bronze, its raw event sits in orders_raw, untouched: a timestamp in local time, an amount stored as text. In silver, orders_clean joins the order to the customer table, converts the amount to one currency, and drops a duplicate event. In gold, daily_sales adds the order’s $249 to the row for Tuesday, store 0, tent category. Store 0 is the online store; the fourteen real stores’ rows come from the point-of-sale daily file.
Order 48291 in bronze, silver, and gold
Bronze · orders_raw
order_id
amount
ordered_at
48291
“249”
19:42:03
48291
“249” (retry)
19:42:05
Exactly as it arrived: amount as text, local time, duplicate kept.
clean · join · dedupe
Silver · orders_clean
order_id
amount
customer_id
currency
48291
249.00
8217
USD
One row per order: duplicate dropped, customer joined, amount fixed.
aggregate
Gold · daily_sales
day
store
category
sales
Tue
0 (online)
tents
12,483.50
Tuesday’s total for online tent sales, with order 48291’s $249 inside, summed with the stores’ rows from the point-of-sale daily file.
The same order row at each stage of the medallion pattern, ending inside a day total.
Semantic layer
Above the tables sits the semantic layer, where a company defines its metrics once and every tool reads that definition. A metric is a business measurement with an agreed formula. At Fern & Pine, net revenue means gross sales minus returns minus discounts, excluding tax and shipping. The formula lives in a semantic layer tool, such as dbt’s semantic layer, Cube, or Looker’s LookML.
The cost is an up-front argument: finance, marketing, and support must agree on the formula. The payoff: the formula is code, and it changes the same way everywhere.
Serving
Serving is how data reaches the person or program that uses it.
Dashboards and BI. Tools such as Power BI, Tableau, and Looker query the gold tables and draw charts. Emily’s order sits in the Tuesday bar on the sales dashboard.
Machine learning. A model is trained on historical data, then run on new data. Fern & Pine’s recommendation model trains nightly on silver features, the input columns a model learns from. When Emily returns, it suggests tent pegs.
AI applications. Assistants, search, and agents get their own section below.
Reverse ETL copies warehouse results back into operational systems. Fern & Pine syncs a segment called “bought a tent this month” into its email tool, using a tool such as Hightouch or Census, now part of Fivetran. Emily gets a peg offer. The cost: each flow is a live pipe that can fail silently.
Data products package datasets for reuse: a schema, a named owner, quality rules, and a data contract, a written promise about shape and quality. Fern & Pine’s orders dataset serves finance and the recommendation team.
Order 48291’s path from checkout to dashboard and model
5. The functions that touch every layer
Five functions run across all the layers: governance, security, monitoring, orchestration, and cost control.
Governance is the system of rules and decisions about data: who owns it, who may see it, how good it is, how long it stays. A data catalog is a searchable inventory of datasets, and each entry names an owner. Fern & Pine’s catalog says orders_clean is owned by the checkout team. The owner answers when the table is wrong, late, or misunderstood.
Data lineage is the recorded history of where data came from and how it was changed. For order 48291, the chain runs from checkout and Stripe, through Kafka, bronze, silver, and gold, to the dashboard.
The lineage behind net revenue and the recommendation model
Data quality means rules and checks: every order must have a customer ID, and yesterday’s daily_sales must exist by 8 a.m. Checks fail loudly and alert the table’s owner.
Access control decides who may see what, through roles. Fern & Pine defines three: support agents open order 48291 to help Emily, marketing analysts query totals but never names, and finance reads the revenue tables.
Privacy and retention. Laws such as the GDPR limit what you may keep. Fern & Pine masks card numbers before bronze, the one change raw data gets before it lands. It keeps order records for seven years because tax rules require it, and raw clickstream for thirty days, because longer retention adds risk without benefit.
Security is the technical side: encryption in transit and at rest, managed credentials, audited access. A weak control on raw clickstream leaks customer data.
Monitoring watches the estate: are pipelines running, is data fresh, has a quality check failed? The freshness check pages the owning team if daily_sales is missing at 8 a.m.
Orchestration is the machinery that runs pipelines. An orchestrator such as Airflow, Dagster, or Prefect schedules jobs, retries failures, and records what ran when. Fern & Pine’s orchestrator starts the 2 a.m. batch and retries the returns-file load.
Cost control. In the cloud everything bills by use, and spend hides in one monthly bill. Watching cloud cost has its own name, FinOps. Fern & Pine tags every pipeline and table with a cost center and reviews the bill monthly.
6. Operating models
Who owns the data? Three models answer.
The central data team builds and runs the whole estate: ingestion, storage, transformation, serving. Business teams send requests. This is where Fern & Pine started, with three data engineers and most of its systems inherited. Its cost arrives with growth: the team becomes a bottleneck, requests queue, and business teams start building private copies.
The data mesh. Zhamak Dehghani proposed it in 2019 as an answer to that bottleneck. In a mesh, each domain, such as orders, inventory, or marketing, owns its data as a data product with a contract. A small platform team builds shared tooling. The mesh fixes ownership: the team that understands the data is the team responsible for it. Its cost: every domain needs data engineering skill, overlapping domains build the same dataset twice, and it all needs mature platform tooling.
The data fabric. Vendors and Gartner popularized “data fabric” as automated, AI-assisted integration. No two vendors mean the same thing, so treat it as capabilities, not an architecture.
Most companies run a hybrid: a central platform team for the shared pipes, plus domain teams that own the datasets they know best. Fern & Pine runs that hybrid today: its three engineers build the pipes, the checkout team owns the web orders, and the retail operations team owns the store sales. The 8 a.m. page goes to whichever owner’s rows are missing. The honest question in any model is the same: if this table is wrong, who gets paged? If nobody, the model is decoration.
Three ways to divide ownership
Central data team
Business teams
requests
One data team builds and runs everything
Cost: the team becomes a bottleneck, and business teams start private copies.
Data mesh
Orders domain owns its data product
Inventory domain owns its data product
Marketing domain owns its data product
Platform team: shared tooling
Cost: every domain needs data engineering skill; overlapping domains build the same data twice.
Hybrid (Fern & Pine)
Three engineers build the shared pipes
Checkout team owns the web orders
Retail operations team owns the store sales
The 8 a.m. page goes to whichever owner’s rows are missing.
Three ways to divide ownership, including the hybrid Fern & Pine runs.
7. The data estate and AI
AI changed what an estate must hold. Classical analytics wants rows and columns. Modern AI also wants unstructured data: reviews, support chats, returns notes, call transcripts. Until recently most of it sat unused.
An embedding is a list of numbers that represents a text or image by its meaning, so similar things get similar numbers. A vector database stores embeddings and answers “which stored texts are most like this query?” pgvector and Pinecone are two, and that search is vector search. A large language model is an AI system trained on vast text to predict and generate language.
Together these enable retrieval-augmented generation, or RAG: the system retrieves the most relevant chunks of the company’s data and hands question plus chunks to the model.
A few days later, Emily asks the support assistant, Trail Guide, whether the tent from order 48291 will survive a rainy coast. Trail Guide retrieves reviews and returns notes for it and answers from them. The retrieval step obeys the dashboard’s access rules: Emily may read her own order and public reviews, but not another customer’s name or cost prices. A RAG system that skips the check leaks data.
AI agents are the next step: large language models that call tools, query databases, and act on the answers. An agent is only as good as the catalog it can read. If the estate is undocumented, the agent guesses.
Gartner listed poor data quality among four reasons projects die after the proof of concept. RAND’s interviews ranked data problems as the second most common cause of failure, and its report recommends investing in data infrastructure and governance (RAND, 2024). Both studies point at data. Neither studied assistants like Trail Guide, which use a ready-made model. We think the point is stronger there, because a RAG assistant is a retrieval system first and a language model second. When the documents underneath are duplicated, stale, or unowned, even a strong model answers from bad data. Gartner estimates that using an existing model through an API costs about $100,000 to $200,000 upfront, while fine-tuning or building one costs $5 million to $20 million (Gartner, July 2024). Spend either sum on an ungoverned estate and the model may still fail.
Fix the estate, then add the model.
8. Common failures
These are the five failures seen most often. Fern & Pine made each of them before it fixed them.
Separate copies. Marketing once exported orders from the old warehouse into its own spreadsheet and built its own report. Two numbers claimed to be “orders,” and each team trusted its own. The fix: one governed table that both teams read.
Data with no owner. Nobody owned the customer table. Marketing and support each defined “active customer” differently, and nobody could settle it, so both definitions reached the dashboard. The fix: one named owner per table.
Unrecorded pipelines. A cron job written in 2019 loaded store sales files. Its author left. It broke silently, and the dashboard went stale for three weeks. The orchestrator now runs that load, and a catalog lists every pipeline with an owner and an alert.
Unpredicted costs. A new dashboard scanned raw JSON files every minute, and the monthly cloud bill rose by a third. Cost tags and a monthly review fixed it.
Two definitions of one metric. Finance reported “revenue” as recognized sales. Marketing reported “revenue” as gross merchandise value. Both appeared on the same dashboard page, and the quarter-end meeting burned a day reconciling them. The semantic layer fixed it: one formula, read everywhere.
Not one of the five is a technology problem. Each is an ownership, documentation, or agreement problem.
9. How to assess your own data estate
You do not need an audit. Ask these questions Monday morning and write down the answers. Where the answer is “nobody knows,” you have found the gap.
Which system first creates our most important record, such as an order, a claim, or a policy?
Can anyone draw the path that record takes from creation to our main dashboard? Who actually can?
How old is the data on our main dashboard right now? How old is too old?
Do we keep raw data from each source, or do we transform it into its final shape as soon as it lands?
Can a new hire find every table we own without asking a colleague?
Who owns the customer table? Who decides when two teams disagree about it?
If the sales number on our main dashboard looks wrong, can we trace it to its source in under an hour?
Do two teams define revenue, active customer, or churn differently? Do they know that they do?
What happens to the spreadsheets and scheduled jobs of an employee who leaves?
Do all our scheduled jobs run inside the orchestrator, or do some run outside it?
How much do our data systems cost each month, and can we tie that cost to teams or projects?
Which data must we keep by law, for how long, and do we delete the rest?
If we started an AI project next month, which data would feed it, and do we trust that data?
Who may see customer data today, and could we prove that list is enforced?
Which pipeline would stop working if its author left tomorrow?
The answers are the first draft of your estate diagram.
10. Where Dataring fits
All five failures in section 8 were problems of ownership, documentation, and agreement, and the first step toward fixing each one is a map of what exists and what depends on what.
Dataring draws that map: every source, pipeline, table, and dashboard, with lineage end to end. That answers three of the questions in section 9: drawing the path a record takes, finding every table without asking a colleague, and tracing a wrong number back to its source. Dataring also shows which tables the business actually depends on, worked out from how the data is used.
It runs in a cloud account you own and reads metadata and access logs, never row data or query text.
11. Glossary
Access control: the rules and mechanisms that decide who may see or change which data.
Access logs: the records a system keeps of who read which data and when.
Aggregation: combining many rows into summaries such as totals and averages.
AI agent: a large language model system that can call tools, query data sources, and act on the results.
Apache Iceberg: an open table format, started at Netflix, that organizes files in object storage into database-style tables.
Apache Kafka: a widely used message broker that receives events and holds them until consumers read them.
API (application programming interface): a documented set of requests a program can make to a system.
Batch load: copying a set of records on a fixed schedule, usually nightly.
BI (business intelligence): the tools and screens that turn data into charts and reports for people to read.
Bronze, silver, gold: the medallion pattern, which stores raw data, then cleaned data, then business-ready tables.
CDC (change data capture): copying only the rows that changed, by reading a database’s own log.
Clickstream: the events a website records as a visitor moves through it.
Columnar format: storing data column by column rather than row by row, which speeds up analytical queries.
Cron job: a scheduled task on a server that runs at fixed times.
Data architecture: the design of a data platform: which systems exist, how they connect, and why.
Data catalog: a searchable inventory of datasets, with owners, descriptions, and schedules.
Data contract: a written promise about a dataset’s shape, meaning, and quality.
Data estate: everything an organization uses to hold, move, govern, and use data: the data, the systems, the pipelines, and the rules and people that govern it.
Data fabric: a vendor term for automated, AI-assisted integration over existing systems; no two vendors mean the same thing by it.
Data governance: the system of rules and decisions about data: ownership, access, quality, and retention.
Data lake: object storage used for analytics, holding raw data in any format.
Data lakehouse: a pattern that adds warehouse-style tables, schemas, and SQL queries to a data lake, using open table formats.
Data lineage: the recorded history of where data came from and how it was changed.
Data mesh: an operating model in which each business domain owns its data as a data product.
Data platform: the technology part of a data estate: storage, pipelines, and tools.
Data product: a dataset packaged for reuse, with a schema, an owner, quality rules, and a contract.
Data quality: rules and automated checks that data meets agreed standards.
Data strategy: the plan for which problems data will solve, in which order, with which people and budget.
Data warehouse: a separate, managed system that holds a copy of data arranged into tables built for analysis.
dbt: a widely used tool that runs SQL transformations inside a data warehouse, mainly for ELT.
Debezium: an open-source CDC tool that reads database logs and emits change events.
Delta Lake: an open table format, open-sourced by Databricks in 2019; Databricks now also supports Apache Iceberg.
Domain: a business area, such as orders or marketing, that owns its data in a data mesh.
ELT: extract, load, transform; landing raw data first and transforming it inside the data warehouse.
Embedding: a list of numbers that represents a text or image by its meaning, so similar items get similar numbers.
ETL: extract, transform, load; cleaning and reshaping data on separate machinery before loading finished tables.
EU AI Act: a European law, in force since August 2024, that sets duties for AI systems that affect people.
Feature: an input column a machine learning model learns from, such as “categories bought this month.”
FinOps: the practice of watching and attributing cloud costs.
GDPR: the European data protection law in force since May 2018.
Generative AI: AI systems that produce new text, images, or other content.
Hadoop: an early open-source system for storing and processing large data sets on many machines.
JSON: a common text format for storing and exchanging data.
Large language model: an AI system trained on vast amounts of text to predict and generate language.
Machine learning: building models that learn patterns from data instead of following fixed rules.
Medallion pattern: see bronze, silver, gold.
Message broker: a system that receives events and holds them until consumers read them.
Metadata: data about data, such as names, owners, schemas, and lineage.
Object storage: a service that holds files cheaply and durably, such as Amazon S3, Azure Blob Storage, or Google Cloud Storage.
Open table format: a standard that organizes files in object storage so they behave like database tables, such as Iceberg or Delta Lake.
Operational database: the database a business system uses to run the business itself.
Orchestration: the machinery that schedules and runs pipelines, retries failures, and records what ran when.
Parquet: a columnar file format used widely in data lakes and lakehouses.
Pipeline: an automated series of steps that moves and processes data from one system to another.
PostgreSQL: a widely used open-source operational database; Fern & Pine’s checkout database.
Proof of concept: a small trial build of a system, made to test whether it works.
RAG (retrieval-augmented generation): answering a question by retrieving relevant chunks of the company’s data and handing them to a large language model.
Retention: the rule for how long each dataset is kept and when it is deleted.
Reverse ETL: copying warehouse results back into operational systems, such as syncing a customer segment to an email tool.
SaaS (software as a service): software a company rents, such as Stripe or Salesforce.
Schema: the formal plan of a table or database: its columns, their types, and their meanings.
Segment: a group of customers defined by shared traits or behavior, such as “bought a tent this month.”
Semantic layer: the place where a company defines its metrics once, in code, so every tool reads the same definition.
Serving: the ways data reaches a person or program that uses it, such as dashboards, models, and reverse ETL.
SQL (Structured Query Language): the standard language for asking questions of databases.
Streaming: moving events one at a time as they happen, with no schedule.
Structured data: data in rows and columns, such as a sales table.
Unstructured data: data without rows and columns, such as text, images, and call recordings.
Vector database: a database that stores embeddings and finds the ones most similar to a query.
Vector search: finding stored texts or images by similarity of meaning, using embeddings.
Write-ahead log: the journal most operational databases keep of each change, used for crash recovery and for CDC.
Zettabyte: a trillion gigabytes.