6sense HQ6sense
6sense HQ6sense

What are the best practices for oil and gas data quality?

Oil and gas data quality best practices: the six quality dimensions, rules for wells, units, and volumes, placeholder traps, reconciliation, and ownership.

Written by: AKM Ahsan Created on: 23 Oct 202612 min to read

Illustration of an oil pumpjack and wells connected by data lines to a dashboard with charts, a checkmark shield, and a magnifying glass highlighting a flagged data value

Good oil and gas data quality comes from checking six dimensions on every load: accuracy, completeness, uniqueness, consistency, timeliness, and validity. In practice, that means rules for well identifiers, units, dates, volumes, and locations, plus catching placeholder values, reconciling against independent sources, and naming an owner for each dataset. Royalty reporting errors alone have cost operators millions in penalties.

This guide is for data leads, analysts, and engineering managers at oil and gas operators, mineral and royalty companies, and oil and gas data businesses. You probably already have data flowing. The problem is trusting it: production totals that don't tie out, wells that appear twice, or a report a regulator questions before your team notices.

If you are still building the pipeline itself, start with our guide to oil and gas ETL pipelines, which covers source formats, well identifiers, datums, and revisions. This article covers how to keep the data that comes through those pipelines trustworthy.

Why does data quality matter more in oil and gas?

Photo by Zbynek Burival on Unsplash
Photo by Zbynek Burival on Unsplash

Data quality matters more in oil and gas because the numbers drive royalty payments, regulatory filings, and emissions reports, and errors in those carry legal and financial consequences, not just bad dashboards.

Three public examples show the stakes:

  • Royalty reporting. In a 2017 case, the Interior Board of Land Appeals upheld $3,217,250 in civil penalties against an operator. The operator had used erroneous gas prices on federal royalty reports and took 8 to 22 months to correct them. The penalty accrued at $25 per day across 229 violations, according to Welborn Sullivan Meck & Tooley. The underpayment itself was $120,242.
  • Flaring. An Environmental Defense Fund analysis of NOAA satellite data estimated that Permian operators flared about 104 billion cubic feet of gas in 2017. Operators reported 55 billion cubic feet to the Texas Railroad Commission.
  • Methane. A 2018 study in Science, led by EDF researchers, estimated US oil and gas methane emissions at nearly 60% more than the EPA inventory at the time, according to the University of Michigan.

The penalty case is the most instructive. The penalty was more than 25 times the underpayment, and most of it came from how long the correction took. Fast detection is worth as much as accuracy.

What are the dimensions of oil and gas data quality?

The most common framework uses six dimensions, as defined by DAMA UK and summarized by the UK Government Data Quality Hub. Each one translates into specific checks for oil and gas data.

DimensionMeaningExample oil and gas check
AccuracyData reflects realityReported volumes reconcile with the regulator's published totals
CompletenessRequired fields are presentEvery producing well has an identifier, operator, and location
UniquenessEach real thing appears onceNo well appears twice under different identifier formats
ConsistencyValues agree within and across datasetsA well marked "plugged" has no production after its plug date
TimelinessData arrives when expectedEach state's monthly production file lands within its usual window
ValidityValues fit the expected format and rangeAPI numbers are 10, 12, or 14 digits; volumes are not negative

Completeness is about critical fields, not every field. A well header missing a spud date may be fine for a production dashboard and unacceptable for a decline-curve model. Define completeness per use.

Best practice 1: Should data quality rules live in code?

Photo by Rob Wingate on Unsplash
Photo by Rob Wingate on Unsplash

Yes. Data quality rules should be written down as executable checks, not held in the heads of the people who know the data. Rules that live only in people's memory leave with them.

Two existing resources make this easier:

  • The PPDM Rules Library. The PPDM Association's Rules Library holds hundreds of data rules contributed by members. Each is written as a logical statement of an expected business condition, independent of any data model. It's a good starting point before writing your own.
  • Test frameworks. Tools such as dbt data tests express each rule as a query that returns failing rows. Zero rows means the check passed. Built-in tests cover uniqueness, null values, accepted values, and relationships between tables.

Give every rule a severity. A negative oil volume should stop a load. A missing county name might only warrant a warning. dbt, for example, lets each test either error or warn.

Best practice 2: How do you catch placeholder values?

Catch placeholder values with explicit rules for each known default, because placeholders pass format checks and then quietly corrupt analysis.

The USGS's 2023 analysis of documented orphaned wells shows how common this is. Across 117,672 wells from 27 states:

  • 11,866 wells had a completion year of 1801, a placeholder for an unknown date.
  • 1,983 Pennsylvania wells used 1901 as their placeholder.
  • About 6% of wells had no API number, including every orphaned well in the Kansas and Indiana files.

A date of 1801 is a valid date, so a format check won't catch it. If it isn't flagged, an average well age, a vintage analysis, or a decline curve built on it is wrong.

Rules to add:

  1. Flag dates before the first commercial oil well in each state, and known placeholders such as 1801 and 1901.
  2. Flag zero coordinates (0, 0) and coordinates outside the state boundary.
  3. Flag depths of 0 or 99999 and other repeated "magic numbers."
  4. Store a separate "unknown" flag instead of overwriting a placeholder with a guess.

Best practice 3: How should you handle units and conversions?

Store every volume in its original unit, label the unit explicitly, and calculate conversions in a separate field rather than replacing the original value.

Oil is reported in barrels, gas in thousand cubic feet (Mcf), and water in barrels. Combined figures usually use barrels of oil equivalent (BOE). According to the SPE's PetroWiki, one BOE is roughly 6 Mcf of gas, and the ratio is approximate because energy content varies.

Because of that variability, never store only BOE. If a source switches gas reporting from Mcf to MMcf, or a buyer changes its BOE convention, you can only recover if the original units were kept.

Rules to add:

  • Every volume column has a unit column, or a unit fixed in its name (gas_mcf, not gas).
  • Month-over-month changes above a threshold for a single well are flagged for review. A sudden thousand-fold jump usually means a unit change, not a new well.
  • The BOE conversion ratio is stored as a parameter, not hard-coded.

Best practice 4: Why do well identity and grain need quality rules?

Well identity and grain need quality rules because a duplicate or mismatched well corrupts every total built on top of it. A well recorded once with a 10-digit API number and once with a 14-digit one doubles its production in any sum.

The rules for this are covered in detail in our ETL pipeline guide. In short:

  • Normalize identifiers to one format.
  • Keep a crosswalk to each source's raw identifier.
  • Be explicit about whether each table is at well, wellbore, completion, lease, or unit level. PPDM's US Well Number Standard, the official successor to the API number, is the reference for well origins and wellbores.

Add a uniqueness test on the normalized identifier and period for every production table. It is the single check that catches the most expensive errors.

Best practice 5: Can you compare data across states directly?

No. Data from different states often can't be compared directly, because each state defines and thresholds the same event differently. Comparing raw counts produces confident, wrong conclusions.

Spill data is a clear example. A 2017 study in Environmental Science & Technology, reported by Inside Energy, counted 6,647 spills from fracked wells in four states between 2005 and 2014. North Dakota accounted for 67% of them. But North Dakota requires reporting for spills of 42 gallons or more, while Colorado's threshold was 210 gallons. Part of the difference is the rule, not the operators.

Rules to add:

  • Store each source's definitions and reporting thresholds as metadata alongside the data.
  • When combining states, filter to the strictest common threshold, or label comparisons as not like-for-like.
  • Map state-specific status codes (such as "shut-in", "temporarily abandoned", "inactive") to a documented common set, and keep the original code.

Best practice 6: How do you check accuracy when you can't see the field?

Photo by Buddy AN on Unsplash
Photo by Buddy AN on Unsplash

Check accuracy by reconciling against an independent source that measures the same thing a different way. Internal consistency checks can't catch a number that is consistently wrong.

The flaring example above is a reconciliation in practice: satellite observations compared against self-reported volumes. Most teams have simpler independent sources available:

Your dataIndependent check
Well-level production you aggregatedThe regulator's own published state or county totals
State production totalsEIA's monthly state estimates
Royalty volumesOperator check-stub volumes, or ONRR federal data for federal leases
Well locationsThe regulator's GIS layer, or satellite imagery for a sample
Well statusPlugging records and inspection reports

Set a tolerance for each reconciliation, and alert when it is exceeded. Expect small differences from timing and revisions. EIA, for example, re-estimates the prior two months with every release under its EIA-914 revision policy.

Best practice 7: How do you validate allocated production?

Validate allocated production by checking that well-level volumes add back up to the measured total they were allocated from. Most well-level production in shared facilities is allocated, not measured directly at each well.

The American Petroleum Institute's Manual of Petroleum Measurement Standards, Chapter 20.1, covers allocation measurement of oil and natural gas, including metering, sampling, and calculation procedures. Whatever method a facility uses, a data team can test the output:

  1. Closure: the sum of allocated well volumes equals the measured facility or lease volume for the same period, within a tolerance.
  2. Non-negative allocation: no well receives a negative share.
  3. Active wells only: no volume is allocated to a well that was shut in or plugged during the period.
  4. Stable factors: allocation factors don't change sharply between months without a recorded reason, such as a well test.

Best practice 8: Is late data a quality problem?

Yes. Late or revised data is a timeliness problem, and it should be monitored like any other quality failure. A file that arrives a week late, or a month revised after you loaded it, makes everything downstream stale without raising an error.

For each source, record its expected arrival window and how it revises history. Alert when a file is late, and reload a rolling window of recent periods so revisions replace old values. The ETL guide linked above covers the reload mechanics.

Best practice 9: Who should own oil and gas data quality?

Photo by Vitaly Gariev on Unsplash
Photo by Vitaly Gariev on Unsplash

Every dataset should have a named owner: someone accountable for its rules, its exceptions, and fixing problems at the source. Without one, quality issues get noticed by whoever consumes the data, usually too late.

That is the common outcome today. In Monte Carlo's 2023 survey of 200 data decision-makers, 74% said business stakeholders found data problems first all or most of the time. Gartner estimated in 2020 that poor data quality costs organizations at least $12.9 million a year on average.

A workable ownership model:

  • A data owner per domain (production, wells, land, emissions) decides what "correct" means.
  • A steward per source handles exceptions and contacts the source when data looks wrong.
  • Engineering maintains the checks and the pipeline, but doesn't decide business rules alone.

Best practice 10: How do you measure data quality over time?

Photo by Hasnain Ayaz on Unsplash
Photo by Hasnain Ayaz on Unsplash

Measure data quality with a scorecard per source and per dimension, tracked over time, so you can see whether quality is improving and where problems concentrate.

A simple scorecard tracks, for each source and month:

  • The percentage of records passing each rule
  • The number of records quarantined or flagged
  • Reconciliation differences against the independent source
  • Arrival delay versus the expected window
  • Open issues and their age

For master data such as well headers, ISO 8000 provides a formal framework for master data quality and machine-checkable requirements when data is exchanged between organizations. Most teams don't need certification, but its approach of building quality from individual property values upward is useful.

Proof: what Peak ETL's 50-state platform required

Peak ETL, an oil and gas data business in Colorado Springs, sells data that landmen and speculators rely on for decisions. 6sense HQ built its Spud Wire platform to unify well, production, land, and regulatory data from all 50 US states, ingested through SFTP, scraping, Excel, and CSV sources.

When every state publishes in a different structure, quality can't be an afterthought. Each source had to be absorbed into one model without rebuilding the system for every new state. That is the same discipline this guide describes: source-specific handling at the edge, shared rules in the core.

Jason Holden, Owner of Peak ETL, said: "6sense HQ delivered the working platform and stayed responsive as requirements changed."

When is this too much data quality work?

Not every team needs all ten practices. Scale the effort to the decisions the data supports.

  • If you consume a commercial dataset and don't modify it, focus on reconciliation and on rules at the points where you join it to your own data. The provider owns the rest.
  • If the data drives royalties, filings, or emissions reports, apply all ten. The penalty example shows why.
  • Don't block every load on warnings. Stop on errors that would corrupt totals, quarantine the failing records, and let the rest flow.
  • Don't build a quality platform before you have rules. Write the ten most important rules first. Tooling comes second.

When you shouldn't work with 6sense HQ: if your data quality problems are mainly about business process, such as who enters what, a data governance consultant may help more than engineering. If your problem is pipelines that break, data that doesn't reconcile, or checks that don't exist yet, that's our work. We start with one source or one data domain, not a rebuild.

The bottom line

Oil and gas data quality is not one check but a discipline. Measure six dimensions, write rules as code, catch placeholder values, keep original units, test well identity and allocation closure, respect state-by-state definitions, and reconcile against independent sources. Above all, name an owner for each dataset and detect problems fast, because in oil and gas the cost of an error grows with every day it goes uncorrected.

About the author: AKM Ahsan is CTO of 6sense HQ, a software engineering partner for energy, mobility, and operations-heavy companies. 6sense HQ built the 50-state data platform behind Peak ETL's Spud Wire. LinkedIn

Data that doesn't reconcile, or checks that don't exist yet? Book a 30-minute technical review with a senior engineer and get a straight read on what to fix first, whether or not you work with us. See how we approach energy data engineering and the build vs. buy decision for energy data.

Share this article

Frequently asked questions

More on Blog