Data Quality Flags and Correction Records

Data Quality Flags and Correction Records

A groundwater or pumping-test dataset needs a defensible way to distinguish observations that are usable, uncertain, corrected or unsuitable for a particular calculation. Quality flags provide that distinction. A correction record explains what changed and why. Together they allow the project to improve its analysis without erasing the evidence of how the original measurements were collected.

The objective is not to make every graph look smooth. It is to preserve enough information for another competent person to judge the result. An unusual reading may represent a real event, an instrument problem or an unresolved condition. Removing it because it does not fit the expected trend can eliminate the very evidence the investigation needs.

Separate quality status from physical condition

A quality flag describes the reliability or treatment of the record for a stated purpose. It does not directly describe the health of the well. A low water level may be a valid and important observation. A reassuring level may be invalid if the sensor is faulty. Keep the physical value and its quality status as separate fields.

Also distinguish data quality from operating alarms. An alarm can indicate that an approved operating condition has been exceeded, while a quality flag can indicate that the measurement itself is questionable. The operating response to uncertain measurement must be defined by the installation’s control procedure. Do not suppress a protective response merely by relabelling the data as uncertain.

Establish a small controlled vocabulary

Use a manageable set of codes with clear definitions. For example, a project may distinguish unreviewed, accepted, suspect, corrected, estimated and excluded-from-analysis records. These are suggested categories for a project dictionary, not a mandatory universal coding system. Define which calculations may use each status and who may change it.

Avoid a single “bad” flag that hides every reason for qualification. Preserve a separate reason code or explanatory record. Instrument drift, clock uncertainty, reference ambiguity and interrupted pumping have different consequences. A reviewer should be able to understand why the observation is qualified without reading every comment in the project archive.

Keep raw observations immutable

Preserve the original logger export, field sheet or source record. Perform corrections in a separate, versioned layer. The raw record may contain an error, but it remains evidence of what was recorded. Replacing it with a corrected value destroys the ability to audit the correction or reconsider it later.

USGS data-quality guidance treats quality management as a lifecycle activity covering collection, processing, use and maintenance. The project implementation should therefore include checks before collection, during field work and after processing. A final spreadsheet review cannot compensate for an undocumented measuring point or an instrument configuration that was never recorded.

Apply automated checks as questions rather than verdicts

Range, rate-of-change, duplicate and timestamp checks can identify records requiring attention. They do not automatically prove that a measurement is wrong. A rapid level change may correspond to a pump event. A repeated value may be a real stable reading or a frozen sensor. Review the event log and measurement context before choosing a disposition.

Document each check, its threshold and its intended purpose. If a threshold is adjusted, retain the earlier rule and explain the effect on the dataset. Do not tune validation limits solely to make a desired proportion of records pass. The rule should reflect the measurement and analysis requirements, and exceptions should remain visible.

Use manual checks where context matters

Compare automatic records with suitable independent observations where available. Check the measurement reference, units, instrument identity and collection time. Review abrupt changes around maintenance, power interruptions and instrument replacements. A discrepancy can identify a data problem, but it can also reveal a previously undocumented physical change.

When the evidence is insufficient, retain a qualified status. “Unresolved” is a legitimate outcome when it accurately describes the record. It is preferable to an invented correction. Identify the decisions affected by the uncertainty and determine whether the available data remain sufficient for those decisions.

Write a complete correction record

A correction record should identify the affected observations, original values, corrected values, units, reason, evidence and method. Include the author, review status and effective dataset version. If one rule affects many records, specify the selection criteria and retain a reproducible transformation rather than manually editing isolated cells without a log.

Corrections can be legitimate without being certain. State any residual uncertainty and whether the correction is suitable for every intended use. A clock adjustment may improve alignment for a daily volume comparison while remaining insufficient for a rapid transient analysis. The correction’s purpose and limits belong in the record.

Handle a measurement-reference correction

Suppose a fictional field record reports depth below a measuring point that is later confirmed to differ from the analysis reference by 0.30 m. Before applying any adjustment, define both references and the sign convention. Derive the conversion explicitly from their geometry. Do not apply a memorised plus or minus sign without checking which point is higher and what the stored value represents.

Preserve the original depths and issue a converted field with a clear name. Check at least one known observation by hand. Then identify calculations and plots that used the earlier reference. A constant reference shift may affect absolute levels while leaving some differences unchanged, but that consequence must be checked for the actual calculation rather than assumed for the whole project.

Handle time corrections cautiously

A constant clock offset and a drifting clock require different treatment. Record the evidence used to establish the correction, such as a documented synchronization event. If the offset is uncertain, preserve that uncertainty. Do not align curves solely by moving them until they appear to match an expected physical response.

Keep original and corrected timestamps available. Recalculate elapsed time and interval integration where the correction changes them. If several instruments were corrected independently, review their relative timing. An analysis that depends on cause and response can be sensitive to a timing error even when the total daily volume is largely unaffected.

Distinguish interpolation from observation

Interpolation supplies an estimated value between observations under a stated method. It does not recover an unmeasured event. Identify the interval and method, and avoid using interpolation across a known change in operating state without an appropriate basis. A pump stop or rate adjustment can invalidate a simple straight-line assumption.

For reporting, show the proportion and location of estimated data where it affects interpretation. A monthly total with a small estimated interval and an event diagnosis based almost entirely on estimated data have different evidential strength. The same completed-looking spreadsheet should not conceal that difference.

Review the effect on decisions

After correcting data, assess whether the correction changes a material result or its confidence. Compare the previous and revised calculation using the same defined method where possible. Record whether the operating decision remains valid, requires qualification or must be revised. A correction that leaves a headline number unchanged may still matter if it alters the interpretation of a critical event.

Notify the responsible decision owner through the project’s agreed process. Preserve previous issued reports and mark their relationship to the correction. Do not silently update a dataset while leaving a report that cites the old version as though nothing changed.

Fictional quality review

A logger reports repeated identical water-level values for several hours while flow and pump-state records change. The reviewer flags the interval as suspect and checks the instrument and field notes. A later manual reading confirms that the logger had stopped updating. The team retains the frozen values in the raw file, excludes them from the specified response analysis and records the evidence.

If an estimated daily summary is still required, it is issued separately with a stated method and limitation. The estimated series is not used to claim that the water level remained stable during the missing interval. This distinction protects both operational reporting and technical interpretation without pretending the lost measurements can be reconstructed exactly.

Design a quality review worksheet

The worksheet should identify the dataset version, intended analysis and review rules before listing exceptions. Include the number of observations reviewed, the intervals affected and the disposition of each concern. Counts are useful for reconciliation, but a high acceptance percentage does not establish that the critical interval is usable. One missing event can matter more than many ordinary accepted readings.

Give each exception a stable identifier and connect it to the source observations. Record the initial flag, investigation evidence, reviewer and final treatment. This prevents comments from becoming detached when rows are sorted or files are exported. A separate exception table can be easier to maintain than long comments embedded in thousands of cells.

Conclude with the dataset’s suitability for the intended use. A dataset may support monthly volume accounting while being unsuitable for detailed transient interpretation. The review should say which use was assessed. Avoid a blanket “quality passed” label that can be carried into an analysis with different requirements.

Distinguish accuracy precision and completeness

Accuracy concerns agreement with the relevant true or reference value within the measurement context. Precision concerns repeatability or resolution in a defined sense. Completeness concerns whether the required observations are present. These characteristics are related but not interchangeable. A very repeatable instrument can retain a systematic offset, and a complete dataset can consistently use the wrong unit.

The review should identify which characteristic each check addresses. Repeating a manual observation can reveal inconsistency but may not detect a shared reference error. Comparing against an independent reference can reveal an offset but does not automatically establish performance across the whole operating range. State the scope of each verification.

Do not convert display resolution into a claimed uncertainty. A logger showing three decimal places does not prove that every digit is accurate. Preserve the instrument and method information needed for an appropriate uncertainty assessment, and avoid reporting results more confidently than that evidence supports.

Investigate impossible values without losing the source

An impossible or implausible value should trigger a check of units, scaling, reference, instrument state and collection method. Preserve the raw value even when it is excluded from an analysis. The error pattern may reveal a recurring configuration problem or a specific event that affects neighbouring records.

If the cause is established, apply a documented correction where justified. If it is not established, retain a suspect or excluded status for the stated use. Do not replace the value with a nearby average solely because the replacement looks reasonable. That creates an estimate and must be labelled accordingly.

Review whether the issue affects a single observation or a whole interval. A scaling error usually has a different scope from a transcription error in one field entry. The correction record should explain how the affected set was identified, allowing another reviewer to test the same selection.

Reconcile duplicate observations

Duplicates can arise from repeated exports, repeated measurements or different instruments observing the same event. These are not the same condition. Compare source identity, timestamp, instrument and value before choosing a treatment. A duplicate timestamp alone is insufficient reason to delete a record.

For repeated exports, preserve one analysis representation while retaining the source files and deduplication rule. For repeated measurements, preserve both observations and their sequence where meaningful. For multiple instruments, retain instrument identity and assess agreement rather than collapsing them into an unexplained average.

Document how duplicates affect integration or summary calculations. Counting an exported record twice can inflate volume or distort a mean. Deleting a genuine second observation can remove evidence of repeatability or instability. The treatment must follow the record’s meaning.

Review quality around interventions

Instrument work, controller updates, power interruptions and reference changes deserve focused review because they can create discontinuities. Compare the as-found and as-left configuration and identify the effective time. Check whether the data path resumed correctly and whether units, scaling or timestamps changed.

A discontinuity after intervention can be physical, instrumental or both. Do not correct it away before investigating. The event log and independent observations help distinguish the possibilities. Preserve the decision and evidence even when the final conclusion is that no numerical correction is needed.

If the exact change time is uncertain, qualify the affected interval. A precise-looking boundary chosen for convenience can create false confidence in subsequent calculations. Identify the uncertainty and assess whether it materially affects the intended result.

Perform a correction impact calculation

For a fictional volume record, suppose a documented scaling correction changes an interval total from 48 to 52 m³. The correction is +4 m³ for that interval. If the monthly total previously included 1,000 m³, the revised arithmetic total becomes 1,004 m³, assuming no other changes. The percentage change is 0.4% relative to the previous total.

That small monthly effect does not establish that every decision is unaffected. If the corrected interval was the basis for a critical rate comparison, its local impact may be material. Review both aggregate and decision-specific consequences. A correction can be negligible for one report and important for another.

Record the previous and revised results, the comparison basis and the decision disposition. This creates evidence for retaining or revising the conclusion instead of relying on a vague statement that the correction was “minor.”

Define a defensible estimation record

When an estimate is necessary, identify its purpose, method, inputs, affected interval and uncertainty. State whether the estimate uses neighbouring observations, operating logs or another source. Explain why that method is suitable for the limited use. Do not present an estimated reconstruction as the original observation.

Keep the estimate reversible in the analysis workflow. A recipient should be able to exclude estimated values or replace them when better evidence becomes available. This is easier when the estimate has its own status and source reference rather than overwriting the missing field.

Review the influence of estimates on the conclusion. If a claimed improvement depends mainly on estimated data, the evidence may be insufficient for a verified claim.

Close the quality review with a usable disposition

The final quality note should state the dataset and version, intended use, checks performed, material corrections, estimated or excluded intervals and remaining limitations. It should identify the reviewed output that consumes the data. This gives future users enough context to decide whether the same dataset suits another purpose.

Retain the raw evidence and review history with the issued dataset. A clean analysis file is useful, but it should not become the only surviving record. The combination of usable data and preserved provenance supports both efficient work and defensible reconsideration when new information appears.

Questions about quality control

Should outliers always be removed

No. Investigate the measurement and operating context. A genuine event may be the most important observation in the dataset. Exclusion needs a purpose-specific reason and a preserved source record.

Can corrected data be called measured data

Describe them as corrected measurements where appropriate and retain the correction basis. Estimated values require a separate designation. The terminology should let the reader understand the evidence, not conceal processing.

Does acceptance mean the value is exact

No. Acceptance means the record is suitable under the stated review and use criteria. Measurement uncertainty and model limitations can remain.

What should be handed over with corrected records

Provide the raw source, correction log, transformation method, quality-code dictionary, reviewed dataset version and the effect on issued conclusions. This allows future users to reproduce and, if necessary, reconsider the treatment.

Source notes

Content-structure method attribution: Eugen Ullrich, eullrich.com, GEO Content Engineering, CC BY 4.0. No ranking or AI citation guarantee is implied.