The situation
The company shipped its first AI feature and the estate changed shape. A support copilot needed ticket history embedded and refreshed. A recommendation model needed feature tables built from orders, sessions and catalog data, rebuilt nightly. Internal agents for sales and finance each wanted their own curated tables. Every one of them arrived with evaluation sets, a few experiment branches and the pipelines that keep all of it fresh, on top of BigQuery, dbt and Airflow that were already busy.
The data team did not grow. The ML engineers built what they needed and moved to the next model. The experiment tables stayed. Describing, owning, scheduling and paying for the new assets by hand had been hard when the estate was dashboards and marts. With a new cluster of tables per launch, it stopped being possible.
What it looked like day to day
Three feature tables named some variant of “user_features_v2” refresh nightly. One feeds production. Nobody is sure which.
An embedding pipeline reprocesses the full ticket history every night for a copilot that reads it a few times a day. It is the most expensive job in the warehouse and nobody owns it.
A model is retrained on a new feature set. The old feature set, its evaluation tables and its pipeline keep running, because turning them off was not on the launch checklist.
The platform lead opens the catalog to tag the AI assets and finds they were never registered. They exist in the warehouse, the orchestrator and a notebook, and nowhere in one place.
Finance asks what the AI features cost to run. The answer is a guess, because feature pipelines look exactly like every other pipeline on the bill.
Why the usual fixes didn’t hold
Each fix assumed the estate was growing at a pace a team could keep up with. The catalog could hold the new assets, and it was good at it once they were in. But getting them in was a manual step the ML engineers skipped under deadline, and lineage from a source table to the model that reads it was not something the catalog could see on its own.
The tagging sprint had worked once for the warehouse. The second sprint, for the AI assets, was out of date before it ended, because two more experiments launched during the two weeks it ran. A sprint is a snapshot. The AI estate was changing weekly.
A retention timer was the obvious next idea: drop any table not written to in a set number of days. It nearly deleted a live feature table that is written monthly and read every hour. Writes are a poor signal for what matters, and a timer cannot tell an abandoned experiment from a slow-changing production asset.
Cost dashboards showed the embedding job at the top of the list. They could not say whether it needed to run nightly, or whether anything read the output. The team could see what the AI estate cost. They could not see what it was for.
Each launch adds a cluster, the team stays the same
What changed with Dataring
Dataring connected to the catalog the team already opens and to a read-only account for the warehouse’s access logs. No rows, no query text, and nothing installed alongside the models.
First it mapped. The feature tables, embedding tables, evaluation sets and their pipelines went onto the same lineage map as the marts and dashboards, from source tables through the pipelines to the model, copilot or agent that reads the result. Assets that were never registered appeared on the map from what actually runs, and were written into the catalog with the descriptions and owners nobody had time for.
From source to model, with the live path visible
Then it understood. The access logs show which of those paths are live. The feature table behind the production recommendation model is read every hour by the serving job. The feature set from the previous training run has not been read since the model moved on. Dataring worked out which assets the business depends on from that use, and gave the live ones the standing they earn.
Live experiment, abandoned experiment
Then its agents proposed, and the team accepted. Pipelines feeding live models and copilots got priority in the scheduler and first place in line after an outage. The embedding job’s refresh was wound down to match how often the copilot actually read it, and Dataring would propose bringing it back up if reads increased. Experiment pipelines nobody read were wound down, then proposed for retirement, a separate action with its own sign-off. New launches were mapped at the next scan, so the next copilot did not need a sprint. Every accepted change was logged with what was done and why.
The decisions were the team’s. Every description and schedule change was accepted by a person before it was applied; retirement always got its own review, and the ML engineers were named on every retirement proposal that touched their work. Dataring works on any warehouse and is strongest on BigQuery.
What the team has now
The AI estate is on the same map as everything else. Every feature table, embedding table and evaluation set has a description, an owner and a line of lineage to the model or app that reads it, kept current from what runs rather than from a registration step someone might skip.
Launches no longer leave a trail. The feature set behind the live model is protected. The one behind last quarter’s model is wound down and waiting for a sign-off to retire. The next launch shows up on the map at the next scan, with nobody registering it.
Finance can see what the AI features cost to keep fresh, scoped to refresh compute, rebuilds and storage, in the team’s own account. And the data team is the same size it was, which is the point.
Try it on your estate
A Dataring POC runs for 30 days in your own cloud, on your catalog and your access logs. It takes about two hours of one engineer’s time to connect, on a call with us, then a 30-minute call a week.