
How do you integrate oil and gas regulatory data from multiple US states?
How to integrate oil and gas regulatory data across US states: who publishes it, how to access it, lease vs. well reporting, operator matching, federal records.
10 min read
Oil and gas data quality best practices: the six quality dimensions, rules for wells, units, and volumes, placeholder traps, reconciliation, and ownership.
Written by: AKM Ahsan Created on: 23 Oct 202612 min to read

Good oil and gas data quality comes from checking six dimensions on every load: accuracy, completeness, uniqueness, consistency, timeliness, and validity. In practice, that means rules for well identifiers, units, dates, volumes, and locations, plus catching placeholder values, reconciling against independent sources, and naming an owner for each dataset. Royalty reporting errors alone have cost operators millions in penalties.
This guide is for data leads, analysts, and engineering managers at oil and gas operators, mineral and royalty companies, and oil and gas data businesses. You probably already have data flowing. The problem is trusting it: production totals that don't tie out, wells that appear twice, or a report a regulator questions before your team notices.
If you are still building the pipeline itself, start with our guide to oil and gas ETL pipelines, which covers source formats, well identifiers, datums, and revisions. This article covers how to keep the data that comes through those pipelines trustworthy.

Data quality matters more in oil and gas because the numbers drive royalty payments, regulatory filings, and emissions reports, and errors in those carry legal and financial consequences, not just bad dashboards.
Three public examples show the stakes:
The penalty case is the most instructive. The penalty was more than 25 times the underpayment, and most of it came from how long the correction took. Fast detection is worth as much as accuracy.
The most common framework uses six dimensions, as defined by DAMA UK and summarized by the UK Government Data Quality Hub. Each one translates into specific checks for oil and gas data.
| Dimension | Meaning | Example oil and gas check |
| Accuracy | Data reflects reality | Reported volumes reconcile with the regulator's published totals |
| Completeness | Required fields are present | Every producing well has an identifier, operator, and location |
| Uniqueness | Each real thing appears once | No well appears twice under different identifier formats |
| Consistency | Values agree within and across datasets | A well marked "plugged" has no production after its plug date |
| Timeliness | Data arrives when expected | Each state's monthly production file lands within its usual window |
| Validity | Values fit the expected format and range | API numbers are 10, 12, or 14 digits; volumes are not negative |
Completeness is about critical fields, not every field. A well header missing a spud date may be fine for a production dashboard and unacceptable for a decline-curve model. Define completeness per use.

Yes. Data quality rules should be written down as executable checks, not held in the heads of the people who know the data. Rules that live only in people's memory leave with them.
Two existing resources make this easier:
Give every rule a severity. A negative oil volume should stop a load. A missing county name might only warrant a warning. dbt, for example, lets each test either error or warn.
Catch placeholder values with explicit rules for each known default, because placeholders pass format checks and then quietly corrupt analysis.
The USGS's 2023 analysis of documented orphaned wells shows how common this is. Across 117,672 wells from 27 states:
A date of 1801 is a valid date, so a format check won't catch it. If it isn't flagged, an average well age, a vintage analysis, or a decline curve built on it is wrong.
Rules to add:
Store every volume in its original unit, label the unit explicitly, and calculate conversions in a separate field rather than replacing the original value.
Oil is reported in barrels, gas in thousand cubic feet (Mcf), and water in barrels. Combined figures usually use barrels of oil equivalent (BOE). According to the SPE's PetroWiki, one BOE is roughly 6 Mcf of gas, and the ratio is approximate because energy content varies.
Because of that variability, never store only BOE. If a source switches gas reporting from Mcf to MMcf, or a buyer changes its BOE convention, you can only recover if the original units were kept.
Rules to add:
gas_mcf, not gas).Well identity and grain need quality rules because a duplicate or mismatched well corrupts every total built on top of it. A well recorded once with a 10-digit API number and once with a 14-digit one doubles its production in any sum.
The rules for this are covered in detail in our ETL pipeline guide. In short:
Add a uniqueness test on the normalized identifier and period for every production table. It is the single check that catches the most expensive errors.
No. Data from different states often can't be compared directly, because each state defines and thresholds the same event differently. Comparing raw counts produces confident, wrong conclusions.
Spill data is a clear example. A 2017 study in Environmental Science & Technology, reported by Inside Energy, counted 6,647 spills from fracked wells in four states between 2005 and 2014. North Dakota accounted for 67% of them. But North Dakota requires reporting for spills of 42 gallons or more, while Colorado's threshold was 210 gallons. Part of the difference is the rule, not the operators.
Rules to add:

Check accuracy by reconciling against an independent source that measures the same thing a different way. Internal consistency checks can't catch a number that is consistently wrong.
The flaring example above is a reconciliation in practice: satellite observations compared against self-reported volumes. Most teams have simpler independent sources available:
| Your data | Independent check |
| Well-level production you aggregated | The regulator's own published state or county totals |
| State production totals | EIA's monthly state estimates |
| Royalty volumes | Operator check-stub volumes, or ONRR federal data for federal leases |
| Well locations | The regulator's GIS layer, or satellite imagery for a sample |
| Well status | Plugging records and inspection reports |
Set a tolerance for each reconciliation, and alert when it is exceeded. Expect small differences from timing and revisions. EIA, for example, re-estimates the prior two months with every release under its EIA-914 revision policy.
Validate allocated production by checking that well-level volumes add back up to the measured total they were allocated from. Most well-level production in shared facilities is allocated, not measured directly at each well.
The American Petroleum Institute's Manual of Petroleum Measurement Standards, Chapter 20.1, covers allocation measurement of oil and natural gas, including metering, sampling, and calculation procedures. Whatever method a facility uses, a data team can test the output:
Yes. Late or revised data is a timeliness problem, and it should be monitored like any other quality failure. A file that arrives a week late, or a month revised after you loaded it, makes everything downstream stale without raising an error.
For each source, record its expected arrival window and how it revises history. Alert when a file is late, and reload a rolling window of recent periods so revisions replace old values. The ETL guide linked above covers the reload mechanics.

Every dataset should have a named owner: someone accountable for its rules, its exceptions, and fixing problems at the source. Without one, quality issues get noticed by whoever consumes the data, usually too late.
That is the common outcome today. In Monte Carlo's 2023 survey of 200 data decision-makers, 74% said business stakeholders found data problems first all or most of the time. Gartner estimated in 2020 that poor data quality costs organizations at least $12.9 million a year on average.
A workable ownership model:

Measure data quality with a scorecard per source and per dimension, tracked over time, so you can see whether quality is improving and where problems concentrate.
A simple scorecard tracks, for each source and month:
For master data such as well headers, ISO 8000 provides a formal framework for master data quality and machine-checkable requirements when data is exchanged between organizations. Most teams don't need certification, but its approach of building quality from individual property values upward is useful.
Peak ETL, an oil and gas data business in Colorado Springs, sells data that landmen and speculators rely on for decisions. 6sense HQ built its Spud Wire platform to unify well, production, land, and regulatory data from all 50 US states, ingested through SFTP, scraping, Excel, and CSV sources.
When every state publishes in a different structure, quality can't be an afterthought. Each source had to be absorbed into one model without rebuilding the system for every new state. That is the same discipline this guide describes: source-specific handling at the edge, shared rules in the core.
Jason Holden, Owner of Peak ETL, said: "6sense HQ delivered the working platform and stayed responsive as requirements changed."
Not every team needs all ten practices. Scale the effort to the decisions the data supports.
When you shouldn't work with 6sense HQ: if your data quality problems are mainly about business process, such as who enters what, a data governance consultant may help more than engineering. If your problem is pipelines that break, data that doesn't reconcile, or checks that don't exist yet, that's our work. We start with one source or one data domain, not a rebuild.
Oil and gas data quality is not one check but a discipline. Measure six dimensions, write rules as code, catch placeholder values, keep original units, test well identity and allocation closure, respect state-by-state definitions, and reconcile against independent sources. Above all, name an owner for each dataset and detect problems fast, because in oil and gas the cost of an error grows with every day it goes uncorrected.
About the author: AKM Ahsan is CTO of 6sense HQ, a software engineering partner for energy, mobility, and operations-heavy companies. 6sense HQ built the 50-state data platform behind Peak ETL's Spud Wire. LinkedIn
Data that doesn't reconcile, or checks that don't exist yet? Book a 30-minute technical review with a senior engineer and get a straight read on what to fix first, whether or not you work with us. See how we approach energy data engineering and the build vs. buy decision for energy data.

How to integrate oil and gas regulatory data across US states: who publishes it, how to access it, lease vs. well reporting, operator matching, federal records.
10 min read

An oil and gas ETL pipeline pulls well, production, land, and regulatory data from state regulators, federal sources, and operators, then converts many file formats into one model keyed on the well identifier. The transformation code is rarely the hard part. The hard parts are format sprawl, well identity, coordinate datums, and sources that revise their own history.
11 min read

Buy energy data software when the job is standard and a vendor already handles your source formats, such as meter data management or Green Button usage feeds. Build when your sources, reconciliation rules, or reporting logic are what make your operation different. Most energy companies end up doing both: they buy the platforms and build the pipelines between them.
11 min read