The situation
The company grew faster than its data team. Sales wanted the CRM in the warehouse, so someone set up a sync service. Finance wanted margins by product, so the analytics team started a dbt project. Marketing bought its own BI tool, then product bought a different one. Each decision made sense on the day. Nobody was asked to keep a map, because there was no map to keep.
Four years on, the estate is BigQuery with hundreds of tables, a few dozen scheduled jobs in Airflow, several dbt projects of different vintages, a sync service pulling from more sources than anyone can list from memory, and three BI tools reading from all of it. The engineers who built the early pipelines have mostly left. Their pipelines still run every night.
What it looked like day to day
A product manager asks who owns a table. The question goes to a channel. Three people reply that it isn’t them.
A pipeline fails at 3 a.m. The on-call engineer fixes it, then spends the morning working out whether anyone would have noticed if she hadn’t.
The platform lead wants to turn off a job that has run for two years. Nobody can say what reads its output, so it keeps running.
An analyst finds several tables named some variant of “customers” and picks one. A week later the numbers don’t match the finance deck.
A new hire opens the catalog, finds the table they need, and reads a description that says “TODO”.
Why the usual fixes didn’t hold
They tried the right things, and each one stopped short. The catalog came first. It pulled in every table and the lineage inside the warehouse, and the search alone was worth it. But a catalog records what people write into it. Descriptions, tags and owners stayed blank, because filling them in was nobody’s job, and the lineage thinned out at the edges of the warehouse: before the sync service, after the dashboards.
Then came a tagging sprint. For two weeks the team documented tables, assigned owners and tagged the ones that mattered. It worked, for the tables that existed that quarter. By the next year many entries were stale, and the people who had done the tagging had moved to other projects or other companies. A snapshot taken by hand ages from the day it is taken.
Cost dashboards showed which jobs were expensive. They could not show which jobs were useful, and the expensive ones were often the ones nobody dared touch. A warehouse bill tells you what you spent, not why.
None of these tools were wrong. Each one depended on a human to supply the thing the team was shortest of: time to describe the estate.
What changed with Dataring
Dataring started from the catalog the team already opens and a read-only account for the warehouse’s access logs. That was the whole setup. No rows read, no query text.
Five tools, no map
First it mapped. Every source, pipeline, table and dashboard went onto one map, with lineage end to end: from the sync service, through the dbt models and Airflow jobs, into the warehouse tables, out to the dashboards in all three BI tools. Where the catalog already knew something, Dataring kept it. Where it didn’t, Dataring filled the gap from what actually runs.
One lineage path, end to end
Then it understood. The access logs show who reads each table and how often. So the map doesn’t just say that a table feeds a dashboard. It says which dashboard, and who opens it. The two-year-old job turned out to feed one table that one dashboard reads, and that dashboard is opened every Monday by the finance team.
Then its agents proposed, and the team accepted. They drafted the descriptions nobody had time for, from the models and jobs that produce each table. They proposed owners. They proposed tags and a medallion layout, so raw, cleaned and reporting tables stopped living in one flat list. What the team accepted was written back into the catalog, where the team already looks, and every change was logged with what was done and why.
What Dataring reads and what it writes back
The decisions were the team’s. Nothing was written until a person accepted it. In the first weeks that meant reading every proposed description and owner before accepting it. Later the descriptions needed a glance and the ownership changes still got a careful read. Dataring works on any warehouse and is strongest on BigQuery.
What the team has now
The team can answer the two questions that used to take a week. “What breaks if we turn this off?” is a lineage view: every table and dashboard downstream, with who reads it. “Who reads this table?” is on the table’s page in the catalog, kept current from the access logs rather than from memory.
The catalog is no longer a list of empty forms. Descriptions say what each table is and where it comes from. Owners are named. Tags reflect what the estate does now, not what it did during a sprint last year. New pipelines show up on the map at the next scan, so the map doesn’t decay the way the tagging sprint did.
And the engineer who fixes a pipeline at 3 a.m. knows by 3:05 whether it mattered.
Try it on your estate
A Dataring POC runs for 30 days in your own cloud, on your catalog and your access logs. It takes about two hours of one engineer’s time to connect, on a call with us, then a 30-minute call a week.