/

Data

How to prioritize data pipelines by what the business depends on

How to prioritize data pipelines by what the business depends on

Data

Data

Mahesh Chandran

Mahesh Chandran

CEO

CEO

Guide cover: How to prioritize data pipelines. Run first what the business depends on most.

When the warehouse is busy at 6 a.m., which job waits? In many setups, schedule order and queue position decide it, and neither knows what each job feeds.

That is how the table behind the invoice run ends up queued behind a backfill for an experiment. Pipeline prioritization fixes it by ranking pipelines by what the business depends on, then using that ranking to decide run order, compute and recovery order after a failure.

This guide shows how to prioritize data pipelines that way. It covers how to tell whether contention is hurting you, how to rank every pipeline from what it feeds, how to protect compute for the most important ones in Snowflake and BigQuery, how to set run order in Airflow and dbt, what to do with lower-priority work, and how to recover in the right order after an outage. It includes the queries, a script and a worked example.

The short answer

Rank each pipeline by the highest tier of the data it feeds, including everything downstream. Give Tier 1 pipelines their own compute, so other work cannot slow them. Order the shared queue so Tier 1 runs before Tier 2, and move Tier 3 work outside the busy hours. Write down the recovery order and restore Tier 1 first after an outage. Retire pipelines that feed nothing anyone reads, because every run they skip frees capacity for the rest. Check the results with queue times and freshness hit rates.

Key terms

  • Pipeline priority: the rank that decides which pipeline gets compute and runs first when there is not enough for all of them.

  • Contention: the point at which more work is waiting than the available compute can run at once.

  • Queue time: how long a job or query waits for compute before it starts running.

  • Pool: a limit on how many tasks of a kind can run at once in an orchestrator, used to protect a shared resource.

  • Critical window: the hours in which Tier 1 data must be built to meet its ready-by times, often early morning.

  • Dataset tier: a rank from 1 to 3 that records how much the business depends on a dataset. Tier 1 is the highest.

The method at a glance

  1. Measure queue times and late Tier 1 datasets to see where contention hurts.

  2. Give every pipeline the highest tier of anything downstream of it.

  3. Record the tier where the orchestrator and the warehouse can read it.

  4. Move Tier 1 pipelines onto compute that other work cannot use.

  5. Set run order in the orchestrator so higher tiers start first.

  6. Schedule Tier 3 work outside the critical window.

  7. Write down the order for restoring pipelines after an outage.

  8. Retire pipelines that feed nothing anyone reads.

Why does default scheduling ignore importance?

Orchestrators and warehouses schedule work by the rules they are given, and the default rules know nothing about the business. Jobs start when their schedule fires, tasks take slots in the order they become ready, and warehouses queue queries as they arrive. When there is spare capacity, none of that matters. When there is not, such as at the start of the working day, at month-end or after an outage, the order is close to arbitrary.

Two things make it worse. Large, low-value jobs such as backfills and experiments are often scheduled overnight alongside the jobs that feed the morning’s reports. And recovery after a failure tends to rerun everything at once, so the pipelines that matter compete with all the others for the same capacity.

How do you know contention is a problem?

Look for these signs before changing anything:

  • Tier 1 data is late on busy days. The same pipeline meets its ready-by time on quiet days and misses it on Mondays or at month-end. How to set data freshness SLAs shows how to record those misses.

  • Jobs wait before they run. Run time looks normal, but the gap between the scheduled start and the actual start grows.

  • Recovery takes most of a morning. After an outage, the important tables are among the last to come back.

In Snowflake, the QUERY_HISTORY view records how long each query waited in a warehouse queue because of overload, in milliseconds, for the last 365 days. It can lag by up to 45 minutes. This query shows queue time by warehouse and hour over the last two weeks:

SELECT
  warehouse_name,
  HOUR(start_time) AS hour_of_day,
  COUNT(*) AS queries,
  ROUND(SUM(queued_overload_time) / 1000 / 60, 1) AS queued_minutes,
  ROUND(MAX(queued_overload_time) / 1000, 0) AS longest_wait_seconds
FROM snowflake.account_usage.query_history
WHERE start_time >= DATEADD('day', -14, CURRENT_TIMESTAMP())
  AND warehouse_name IS NOT NULL
GROUP BY warehouse_name, hour_of_day
ORDER BY queued_minutes DESC
LIMIT 20

SELECT
  warehouse_name,
  HOUR(start_time) AS hour_of_day,
  COUNT(*) AS queries,
  ROUND(SUM(queued_overload_time) / 1000 / 60, 1) AS queued_minutes,
  ROUND(MAX(queued_overload_time) / 1000, 0) AS longest_wait_seconds
FROM snowflake.account_usage.query_history
WHERE start_time >= DATEADD('day', -14, CURRENT_TIMESTAMP())
  AND warehouse_name IS NOT NULL
GROUP BY warehouse_name, hour_of_day
ORDER BY queued_minutes DESC
LIMIT 20

SELECT
  warehouse_name,
  HOUR(start_time) AS hour_of_day,
  COUNT(*) AS queries,
  ROUND(SUM(queued_overload_time) / 1000 / 60, 1) AS queued_minutes,
  ROUND(MAX(queued_overload_time) / 1000, 0) AS longest_wait_seconds
FROM snowflake.account_usage.query_history
WHERE start_time >= DATEADD('day', -14, CURRENT_TIMESTAMP())
  AND warehouse_name IS NOT NULL
GROUP BY warehouse_name, hour_of_day
ORDER BY queued_minutes DESC
LIMIT 20

Hours with high queued minutes on the warehouse that builds Tier 1 data are where prioritization will help most. In Airflow, the same pattern shows up as tasks sitting in the queued state while worker slots are full.

How do you rank pipelines?

How to decide which datasets your business depends on ranks datasets into tiers by where their data lands. A pipeline takes the highest tier of anything it feeds, directly or downstream. A staging job that nobody queries is Tier 1 if the invoice table is built from it.

In a dbt project, the tier can be worked out from the lineage. Every run writes a manifest.json file that includes a parent_map listing the direct parents of each resource. Mark the tier on the final models and exposures, then pass it up to every parent with this script:

import json

manifest = json.load(open("target/manifest.json"))
resources = {**manifest["nodes"], **manifest["exposures"]}

tier = {}
for uid, node in resources.items():
    meta = node.get("config", {}).get("meta", {}) or node.get("meta", {})
    if "tier" in meta:
        tier[uid] = int(meta["tier"])

# Give every parent the highest tier (lowest number) of its children.
changed = True
while changed:
    changed = False
    for child, parents in manifest["parent_map"].items():
        if child not in tier:
            continue
        for parent in parents:
            if tier.get(parent, 4) > tier[child]:
                tier[parent] = tier[child]
                changed = True

for uid, t in sorted(tier.items(), key=lambda item: item[1]):
    if uid.startswith("model."):
        print(t, uid)
import json

manifest = json.load(open("target/manifest.json"))
resources = {**manifest["nodes"], **manifest["exposures"]}

tier = {}
for uid, node in resources.items():
    meta = node.get("config", {}).get("meta", {}) or node.get("meta", {})
    if "tier" in meta:
        tier[uid] = int(meta["tier"])

# Give every parent the highest tier (lowest number) of its children.
changed = True
while changed:
    changed = False
    for child, parents in manifest["parent_map"].items():
        if child not in tier:
            continue
        for parent in parents:
            if tier.get(parent, 4) > tier[child]:
                tier[parent] = tier[child]
                changed = True

for uid, t in sorted(tier.items(), key=lambda item: item[1]):
    if uid.startswith("model."):
        print(t, uid)
import json

manifest = json.load(open("target/manifest.json"))
resources = {**manifest["nodes"], **manifest["exposures"]}

tier = {}
for uid, node in resources.items():
    meta = node.get("config", {}).get("meta", {}) or node.get("meta", {})
    if "tier" in meta:
        tier[uid] = int(meta["tier"])

# Give every parent the highest tier (lowest number) of its children.
changed = True
while changed:
    changed = False
    for child, parents in manifest["parent_map"].items():
        if child not in tier:
            continue
        for parent in parents:
            if tier.get(parent, 4) > tier[child]:
                tier[parent] = tier[child]
                changed = True

for uid, t in sorted(tier.items(), key=lambda item: item[1]):
    if uid.startswith("model."):
        print(t, uid)

Models with no tier after the script runs feed nothing marked as important. They are Tier 3 by default, and candidates for a retirement review.

Write the tier where the orchestrator and the warehouse can read it, such as a tag on the dbt model, a tag on the orchestrator job and a tag or label on the warehouse table. A tier recorded only in a spreadsheet will not change how anything runs.

Pipelines ranked by tier: Tier 1 runs first on its own compute, Tier 2 runs next on shared compute, Tier 3 runs outside the busy hours.

How do you get agreement on the ranking?

Every team believes its pipelines matter, so a ranking set by the data team alone will be argued over. Base it on evidence everyone can check.

  • Rank by destination. Show each team what its pipelines feed. A pipeline that feeds invoices outranks one that feeds an internal experiment, and most people accept that once they see it written down.

  • Let business owners confirm Tier 1. The head of billing confirms that the invoice run is Tier 1, and the head of finance confirms the close reports. The data team does not have to make the call alone.

  • Publish the list. Put the ranking where everyone can read it, with the reason for each Tier 1 entry. Disputes get raised early, in the open.

  • Agree how to change it. A team that thinks a pipeline is ranked too low brings evidence of what it feeds to the quarterly review.

How do you handle shared upstream tables?

A shared table such as customers or orders often feeds Tier 1 and Tier 3 destinations at once. It takes the highest tier of anything it feeds, so it is built in the Tier 1 window on Tier 1 compute.

That can make the Tier 1 window crowded with large shared tables. Two changes help. Build the shared table incrementally, so the Tier 1 run processes only new data. And move expensive transformations that only Tier 3 readers need into separate downstream models, so they run later on lower-tier compute.

How do you protect Tier 1 compute?

The most reliable protection is separation. When Tier 1 pipelines run on compute that nothing else uses, other work cannot slow them down.

  • In Snowflake, give Tier 1 transformations their own warehouse. Its credits show up separately in the metering views, so the cost of protecting Tier 1 stays visible. How to reduce Snowflake costs covers sizing and auto-suspend for each warehouse.

  • In BigQuery, capacity pricing groups slots into reservations, which can separate workloads such as production and testing. Assign the projects that build Tier 1 data to their own reservation, so their slots are not shared with ad hoc queries.

  • In the orchestrator, give Tier 1 tasks their own pool or queue, so a large batch of lower-tier tasks cannot take every worker.

Separate compute costs money even when it sits idle, so size it for the critical window. A Tier 1 warehouse that suspends when the morning build finishes costs little outside those hours.

How do you set run order?

Within shared capacity, tell the orchestrator which tasks to start first.

In Airflow

Each Airflow task has a priority_weight that sets its place in the executor queue. Larger numbers mean higher priority, and the default weight is 1. The default weight rule, downstream, adds up the weights of a task’s downstream tasks, so long chains rise in priority whatever they feed. Setting the weight rule to absolute makes the number mean exactly what is set, which lets the tier decide.

Airflow pools limit how many tasks of a kind run at once. Tasks without a pool go to default_pool, which starts with 128 slots. When a pool is full, runnable tasks wait in the queue, and as slots free up they start according to their priority weights. A dedicated Tier 1 pool keeps lower-tier tasks from filling every slot.

TIER_WEIGHT = {1: 100, 2: 10, 3: 1}

build_invoice_lines = BashOperator(
    task_id="build_invoice_lines",
    bash_command="dbt build --select invoice_lines+",
    priority_weight=TIER_WEIGHT[1],
    weight_rule="absolute",
    pool="tier1",
)

build_experiment_q2 = BashOperator(
    task_id="build_experiment_q2",
    bash_command="dbt build --select experiment_q2_scratch",
    priority_weight=TIER_WEIGHT[3],
    weight_rule="absolute",
)
TIER_WEIGHT = {1: 100, 2: 10, 3: 1}

build_invoice_lines = BashOperator(
    task_id="build_invoice_lines",
    bash_command="dbt build --select invoice_lines+",
    priority_weight=TIER_WEIGHT[1],
    weight_rule="absolute",
    pool="tier1",
)

build_experiment_q2 = BashOperator(
    task_id="build_experiment_q2",
    bash_command="dbt build --select experiment_q2_scratch",
    priority_weight=TIER_WEIGHT[3],
    weight_rule="absolute",
)
TIER_WEIGHT = {1: 100, 2: 10, 3: 1}

build_invoice_lines = BashOperator(
    task_id="build_invoice_lines",
    bash_command="dbt build --select invoice_lines+",
    priority_weight=TIER_WEIGHT[1],
    weight_rule="absolute",
    pool="tier1",
)

build_experiment_q2 = BashOperator(
    task_id="build_experiment_q2",
    bash_command="dbt build --select experiment_q2_scratch",
    priority_weight=TIER_WEIGHT[3],
    weight_rule="absolute",
)

In dbt

A single dbt build runs models in lineage order, with no notion of priority. Splitting the build by tier gives the orchestrator something to order. Tag models with their tier, then select them with the tag method. The graph operators add everything upstream with a plus sign in front of the selector, so a Tier 1 job can build its models and their parents in one command.

# Tier 1 job, run first in the critical window
dbt build --select "+tag:tier1"

# Tier 2 job, run after Tier 1 finishes
dbt build --select "tag:tier2"

# Tier 3 job, run outside the critical window
dbt build --select "tag:tier3"
# Tier 1 job, run first in the critical window
dbt build --select "+tag:tier1"

# Tier 2 job, run after Tier 1 finishes
dbt build --select "tag:tier2"

# Tier 3 job, run outside the critical window
dbt build --select "tag:tier3"
# Tier 1 job, run first in the critical window
dbt build --select "+tag:tier1"

# Tier 2 job, run after Tier 1 finishes
dbt build --select "tag:tier2"

# Tier 3 job, run outside the critical window
dbt build --select "tag:tier3"

Other orchestrators have equivalent settings, such as run priorities and concurrency limits per queue. The principle is the same. The number that decides the order should come from the tier.

What about ad hoc queries and BI tools?

Pipelines are only part of the load on a warehouse. Analysts exploring data and BI dashboards refreshing on a timer compete for the same compute, often in the same morning hours.

  • Give people their own compute. A separate warehouse or reservation for analysts keeps a heavy exploratory query from slowing the Tier 1 build.

  • Give BI tools their own service account and compute. This makes their load visible and lets you size it on its own.

  • Match dashboard refreshes to the data. A dashboard that refreshes every few minutes from a table rebuilt once a day adds load without showing anything new.

When should lower tiers run?

Move Tier 3 work, and Tier 2 work that is read later in the day, away from the hours when Tier 1 is building. Backfills, experiments and full refreshes are the usual candidates. Running them after the morning’s Tier 1 targets are met costs nothing in freshness for the data people use first, and it removes their load from the critical window.

Check each pipeline’s schedule against when its data is read. A pipeline that runs every hour for data read once a day can usually run once a day. A pipeline that rebuilds a large table in full can often build only the newest data incrementally.

Large one-off jobs need a rule of their own. Ask whoever runs a backfill to schedule it outside the critical window, run it on Tier 3 compute, and tell the owners of Tier 1 data before it starts.

What order should pipelines recover in?

After a warehouse outage or a failed source load, many pipelines need to run at once. Write down the recovery order in advance, so the person on call does not have to work it out at 3 a.m.

  • Tier 1 first, in lineage order, from sources to the tables that feed invoices, finance and customer-facing products.

  • Tier 2 next, once Tier 1 is confirmed correct.

  • Tier 3 last, or at the next scheduled run.

A recovery runbook for each Tier 1 dataset can fit on one page:

  • The command or job that rebuilds it, with its upstream sources.

  • How to confirm the rebuilt data is correct, such as row counts against the previous day.

  • Who to tell when it is restored, usually the business owner.

  • Which lower-tier schedules to pause until it is restored.

Hold Tier 3 schedules during recovery, so they do not compete with the work being restored.

How do you roll this out?

Prioritization changes how production runs, so introduce it in steps that can each be checked:

  1. Measure. Record queue times, Tier 1 freshness hit rates and recovery times for two weeks, so you have a baseline.

  2. Rank. Run the tier script, agree Tier 1 with business owners and tag the models.

  3. Separate Tier 1 compute. Move Tier 1 builds to their own warehouse, reservation or pool, and watch their queue times.

  4. Reorder the rest. Set priority weights, split the dbt build by tier and move Tier 3 schedules out of the critical window.

  5. Write the recovery runbooks. Do this for each Tier 1 dataset, then test one during a quiet period.

  6. Compare. Check the same measures against the baseline after a month, and share the result with the business owners.

A worked example

Here is how the method plays out for one morning. The details are invented to show the steps.

A company’s invoice_lines table must be ready by 6 a.m. Eastern. Its freshness log shows misses on most Mondays and on the first working day of each month. The Snowflake queue query shows that the shared transformation warehouse queues heavily between 4 and 6 a.m. on those days. Two large jobs run in the same window: a weekly full refresh of a marketing attribution model, and a backfill an analyst started for an experiment.

The team runs the tier script and finds 40 models that feed Tier 1 destinations. They tag those models tier1, move them to a dedicated warehouse that resumes at 3:30 a.m. and suspends when idle, and put them in a Tier 1 Airflow pool with an absolute priority weight. The attribution refresh moves to 10 a.m., after the readers who need it have started work. Backfills get a written rule to run after 8 a.m. on Tier 3 compute.

The following month, the freshness log shows invoice_lines met its target every working day. The Tier 1 warehouse’s credits are reported separately, so the team can show what the protection costs next to what it achieved.

How do you check that prioritization is working?

Track a few measures before and after the change:

  • Tier 1 freshness hit rate: the share of days each Tier 1 dataset met its ready-by time.

  • Queue time on Tier 1 compute: which should be close to zero in the critical window.

  • Cost of Tier 1 compute: reported on its own, so the price of protection is visible.

  • Recovery time after incidents: how long Tier 1 data took to come back after the last outage.

What about pipelines nobody needs?

The cheapest way to make room for Tier 1 is to stop running work that feeds nothing. How to find and retire unused data pipelines shows how to find tables that are written but never read, and how to retire them in stages. Every retired pipeline frees its compute and its place in the queue.

What mistakes should you avoid?

  • Ranking by the pipeline’s own name or team. A pipeline’s tier comes from what it feeds. A finance team’s scratch model is not Tier 1 because finance owns it.

  • Forgetting upstream jobs. Protecting the final model while its staging models wait in the shared queue leaves Tier 1 just as late.

  • Making everything Tier 1. If most pipelines are Tier 1, the ranking decides nothing. Keep Tier 1 to the data behind money, customers and regulators.

  • Leaving default weight rules in place. Airflow’s downstream rule rewards long chains. Use the absolute rule when weights come from tiers.

  • Hiding the cost of protection. Dedicated compute that nobody reports on gets cut in the next budget review. Report it next to the hit rate it buys.

Questions people ask

Does a bigger warehouse remove the need to prioritize?

Only while there is spare capacity. Busy periods, month-end and recovery after an outage still create contention, and a bigger warehouse costs more for every hour it runs. Prioritization decides who waits when contention happens.

Should Tier 1 pipelines run more often?

They should run as often as their readers need, and no more. A Tier 1 table read once a day needs one reliable run before its ready-by time. Hourly runs add cost without making it more useful.

Who decides a pipeline’s tier?

The tier comes from the data the pipeline feeds, so it follows the dataset ranking agreed with business owners. How to assign data ownership covers who those owners are.

How often should priorities be reviewed?

Every quarter, and whenever a dataset starts feeding a new high-stakes destination, such as billing or a production model.

What if two Tier 1 pipelines compete with each other?

Order them by ready-by time, earliest first. If both are due at the same time and cannot both finish, the Tier 1 compute is too small for the critical window, and that is a sizing decision to take with the business owners.

Does this work without an orchestrator?

Partly. Separate warehouses or reservations and moving low-tier schedules out of the critical window work with any scheduler, including cron and dbt’s own job scheduler. Fine-grained run order needs an orchestrator that supports priorities.

Does prioritization cost more?

Dedicated Tier 1 compute adds some cost. Moving Tier 3 work out of busy hours, slowing over-frequent schedules and retiring unused pipelines usually take cost away. Report both, so the net change is visible.

How many datasets should be Tier 1?

As few as the business can justify. Tier 1 should hold the data behind payments, financial reporting, customer-facing products and anything regulators see. When a new candidate comes up, ask what would happen if it were a few hours late. If the answer is an inconvenience, it is Tier 2.

What if a business owner disagrees with a tier?

Ask them to name what the data feeds and what a late or wrong number would cost. If the evidence supports a higher tier, change it and record the reason. If it does not, record the decision and revisit it at the next quarterly review.

Where Dataring fits

We built Dataring to rank every dataset by how much the business depends on it, then act on that ranking. Pipelines behind billing, finance and customer-facing products get the resources and run order they need to stay reliable, and pipelines nobody depends on are retired within limits each company sets, under its supervision.

Book a free call to see it on your data estate. For the wider process this guide fits into, read How to audit your data estate in seven steps.

Newsletter

Product news by email. We use your address for nothing else.

© 2026 ShareData Inc.Last updated October 29, 2026