Quality and Rules

How trustworthy an extracted value is, and the rules you can add to check it: grounding, confidence, schema validation and business rules.

Scores and rule outcomes come back with every extraction (Running extractions). Every field: API reference.

Quality Scoring

Every text extraction result includes quality scores that help you assess how trustworthy the extracted data is. These scores are purely informational — structuredOutput is never modified.

Grounding vs Confidence

2kw.ai distinguishes between two types of quality signals:

Grounding measures whether an extracted value has evidence in the source text. If your schema extracts a sourceFile field and the value "gear.geo" appears in the input, that's grounded. If "phantom.geo" doesn't appear anywhere in the input, it was likely hallucinated by the model. Grounding is deterministic, free (no extra LLM calls), and available on every extraction.

Confidence combines the grounding score with the other evidence 2kw.ai already has about a value: the schema violations recorded for it and the verdicts of your business rules. It is arithmetic over those signals. No model is asked how sure it is, no extra call is made, and the same result always produces the same score. See Composite Confidence for how the number is built.

The scoring.methods array tells you which methods ran: grounding, and confidence when the composite was computed.

How Grounding Works

For each extracted field, 2kw.ai searches the original input text for the extracted value:

Per-item scores are weighted averages of their fields, with key fields (identifiers like filenames, IDs, names) weighted 3x higher than regular fields. This means a hallucinated filename dominates the item score even if other fields look fine. Unverifiable fields (numbers, booleans, short strings) are excluded from this average — they don't drag the score down or inflate it.

Composite Confidence

Every text extraction that is scored carries a composite confidence per scored field: one number in [0, 1], built from the field's grounding score, its schema violations and its business-rule verdicts. Like the other quality signals it is recorded only. Nothing is blocked, no status changes and nothing is retried.

The scores are not calibrated

resultMetadata.scoring.confidence.calibrated is false, and it is the most important field in this section. The numbers are an ordering, not a probability: 0.7 does not mean "70% likely to be correct". The constants below have not been fitted against labelled outcomes. Use the scores to rank fields against each other within one schema version, derive any threshold from your own observed results, and do not show a confidence number to an end user as a probability.

How the score is computed

The grounding score is the base. Negative evidence attributed to the field either deducts from it or caps it. Nothing raises it: a rule that passed adds to the field's provenance, never to its score.

base    = grounding score
penalty = sum of the deductions above
cap     = the smallest applicable cap, if any
score   = clamp(min(round(base − penalty, 2 decimals), cap), 0, 1)

severity and automationBlocker act independently. A blocking WARNING failure deducts 0.2 and caps at 0.1; when several caps apply, the smallest wins. Deductions add up without limit (three failed WARNING rules deduct 0.6) and the result is clamped at 0. Caps beat deductions because the two meet in a min. The SKIPPED and ERROR verdicts do not count: the input was absent or the rule could not run, and neither says anything about the value. Scores are rounded to two decimals, the same grid as the grounding scores.

The basis

confidence.basis names the one cause that decided the number. It is always one of six values:

The order of the table is the priority order. A cap claims the basis only when it binds, that is, when it is at or below the score the deductions produced. A field grounded at 0.1 that also has a schema violation reads grounded at 0.1, because the 0.2 cap would have raised it. On an exact tie the cap takes the label. Read the basis as "what decided this number", not as "everything wrong with this field": the violation is still listed in resultMetadata.schemaValidation.

low_information is not a middle score

Grounding checks a value by searching for it in the source text. That cannot work for numbers, booleans, objects, strings shorter than three characters, or arrays with nothing searchable in them. Without a field hint they score 0.5 with grounding basis unverifiable, because nothing was checked. When such a field has no violation and no failed rule, its composite stays 0.5 with basis low_information: this number tells you nothing about this field.

• A 0.5 that means "unverified" and a 0.5 that came out of a real comparison are different claims. Only the basis tells them apart.
• Averaging a document's scores without separating out low_information mostly measures how many numeric fields the schema has.
• Do not infer the basis from a field's type. On array-item fields a field hint is applied first (system_assigned scores 1.0 without looking at the document), so read the basis.

Negative evidence still applies: a low_information field that violates its schema is capped like any other, and its basis changes.

Provenance

provenance lists the checks that looked at the value and did not object. It is not a second score.

Entries appear in that order. history and human are reserved and not written yet. system_assigned is the one entry that is not a check: the value was trusted because of the field's configuration, not compared against the input.

An empty list is an answer: the composite was computed and nothing vouches for the value. It does not move with the score. A field can score 0.5 with [] (nothing was checked) or 0.2 with ["grounding"] (the text was found, then a schema violation capped it). The list is null only when confidence is null.

Thresholds and flags

Two independent thresholds can be configured per property on the schema version (see Extraction Configuration), and they judge different numbers:

Both are keyed by property name: a root field's own name, an array property's name (which covers every scored field of every element), or $root for a schema whose root is an array.

A score strictly below its threshold records a flag; a score exactly at the threshold passes. Meeting a composite threshold also adds threshold provenance. One field can raise both flags in the same extraction. A flag is an observation only: nothing is blocked, retried or changed. At most 100 flags are recorded per result. flagCount is the true total and truncated says whether the list was cut, so read the count, not the length of the list.

The composite threshold takes a plain JSON number in [0, 1]. Anything else (a quoted "0.8", a boolean, a number out of range) counts as not configured, without a warning. If a composite threshold never flags anything, check its type first, then the property name, then whether the field was scored at all.

Which field a signal belongs to

Schema violations and rule outcomes both carry a JSON pointer. Each is attributed to the deepest scored field whose pointer is a prefix of it: /vendor/vatId lands on the root field vendor, and /parts/0/price on the item field parts[0].price.

A signal that matches no scored field lowers no score; it stays where it was recorded. The common case is a root field the model omitted or returned as null: it has no score, so the required violation that names it has nothing to lower. Rule outcomes count only when they name one concrete location. An outcome with a wildcard pointer or the empty pointer is not attributed to any field.

Known limitations:

• A signal on an array element as a whole (/parts/0 rather than /parts/0/price) lowers nothing, because only an element's fields are scored.
• Flags name fields flatly (parts[0].price). A root field literally named parts[0].price, or $root, shares that flag key with the item field. Their scores stay separate everywhere else in the result.

When confidence is null

A null confidence never means "not confident". It means the number was not computed:

1. Vision extractions carry no grounding, so they have no scoring at all: resultMetadata.scoring is null.
2. Validation is switched off for the schema version (validation.enabled: false). That also switches off grounding, so there is nothing to build on.
3. Nothing was scored. Either there was no source text, or no property of the output could be scored. In the second case the scoring block can still show methods: ["grounding"] and a grounding.score of 1.0 (the average of nothing) while fields and items are empty. Read those two maps, not the presence of the block.
4. The extraction was scored before composite confidence existed.
5. The confidence step could not complete for this extraction. The grounding scores are left as they were. If this happens on every extraction for one schema version, contact support.

The quickest positive check is "confidence" in scoring.methods. Before you gate anything on a score, check that resultMetadata.scoring.confidence is present and treat its absence as "unknown", never as "clean". A missing summary and a summary with flagCount: 0 look nothing alike, and only the second is a statement about your data.

Reading the payload

Confidence sits inside resultMetadata.scoring, next to the grounding scores it is built from:

{
  "methods": ["grounding", "confidence"],
  "grounding": { "score": 0.95, "basis": null },
  "thresholds": { "parts": 0.5 },
  "fields": {
    "invoiceNumber": {
      "grounding": { "score": 1.0, "basis": "exact_match" },
      "confidence": { "score": 0.2, "basis": "schema_capped" },
      "provenance": ["grounding"]
    },
    "total": {
      "grounding": { "score": 0.5, "basis": "unverifiable" },
      "confidence": { "score": 0.5, "basis": "low_information" },
      "provenance": []
    }
  },
  "items": {
    "parts[0]": {
      "grounding": { "score": 0.9, "basis": "normalized_match" },
      "fields": {
        "price": {
          "grounding": { "score": 0.9, "basis": "normalized_match" },
          "confidence": { "score": 0.7, "basis": "rule_penalized" },
          "provenance": ["grounding"]
        }
      }
    }
  },
  "confidence": {
    "calibrated": false,
    "flags": [
      { "field": "invoiceNumber", "confidence": 0.2, "threshold": 0.8, "source": "COMPOSITE" }
    ],
    "flagCount": 1,
    "truncated": false
  }
}

• fields holds root-level properties and items holds array elements, keyed parts[0] (or [0] when the schema's root is an array). A property the model omitted or returned as null appears in neither.
• An item's own grounding.basis names the basis of its weakest scored field, the one that pulled the item score down. It reads unverifiable when none of the item's fields could be checked. The top-level grounding.basis is always null.
• flags[].field uses the same keys: invoiceNumber, parts[0].price, [2].name.
• flags[].confidence is the grounding score for LEGACY_GROUNDING flags and the composite for COMPOSITE flags. Always read it together with source. A field that appears twice, once per source, is two signals, not a duplicate.
• thresholds is a hint map for the console. It lists only array properties and $root, and flags do not depend on it: a grounding threshold on a root field raises flags without ever appearing here.

Schema Validation

Every extraction output is checked against its schema version before it is scored. The check only records: violations are listed on the result, nothing is blocked, the status does not change and nothing is retried. For chunked extractions only the final merged output is checked, never the individual chunks. If you want non-conforming output to fail hard, enforce it in your own code from this object.

The result is in resultMetadata.schemaValidation:

{
  "applied": true,
  "violationCount": 2,
  "truncated": false,
  "violations": [
    {
      "fieldPath": "/total",
      "keyword": "type",
      "message": "string found, number expected",
      "source": "MODEL",
      "property": null
    },
    {
      "fieldPath": "/invoiceNumber",
      "keyword": "type",
      "message": "null found, string expected",
      "source": "RESTORED_NULL",
      "property": null
    }
  ]
}

A required violation is anchored on the containing object, not on the missing field. A top-level required field the model left out entirely reports {"fieldPath": "", "keyword": "required", "property": "invoiceNumber"}, so read the field's name from property, never from message.

source: RESTORED_NULL means the field was missing after the chunks were merged and was filled with null by post-merge validation. That is a deliberate absence signal: the input most likely does not contain the value. Treat it as "not found", not as a model error. Only chunked extractions produce it.

At most 100 violations are recorded. violationCount is always the true total, and truncated is true when more were found.

applied: false never means "passed"; a clean output is applied: true with an empty violations list. It means the check did not run, either because validation is switched off for the schema version (validation.enabled: false, which also switches off grounding and confidence scoring) or because it could not run for this extraction.

Absence at a glance:

A field that keeps coming back as RESTORED_NULL points at the schema or the input, not at the model: either the value is not in the documents, or the field should not be required.

Extraction Configuration

Each schema version can carry an extractionConfig that tunes per-property scoring, dedup, thresholds and business rules. It is stored alongside the JSON Schema but never sent to the model; it controls pipeline behavior only.

The configuration is set on the schema version by support. The API does not accept it when you create a version (POST /v1/schemas/{schemaId}/versions takes only jsonSchema and changeDescription), but you can read it back as extractionConfig on the schema version. To change it, contact support with the schema, the version and the settings you want.

For a schema whose parts property is an array of items with sourceFile, partNumber, material and quantity, a configuration looks like this:

{
  "validation": {
    "enabled": true,
    "properties": {
      "parts": {
        "confidenceThreshold": 0.5,
        "keyFields": ["sourceFile"],
        "fieldHints": {
          "partNumber": { "scoring": "system_assigned" },
          "quantity": { "scoring": "default_absent" }
        }
      }
    }
  },
  "confidence": {
    "properties": {
      "parts": { "threshold": 0.6 }
    }
  }
}

Available field hints:

Business Rules

Business rules are deterministic checks over an extraction's output: a VAT ID that matches a pattern, a total that equals the sum of its line items, a date in ISO form. They need no model call and no calibration data. The same output always gets the same verdict.

Rules are part of the schema version's extraction configuration, in a top-level rules array, and are set by support like the rest of it. They run on every extraction of that version, right after schema validation and before scoring. validation.enabled does not switch them off; each rule has its own enabled flag.

Rules only record. Every outcome, passes included, is written to resultMetadata.rules; nothing is blocked, the status does not change and nothing is retried.

{
  "validation": { "enabled": true },
  "rules": [
    {
      "id": "vat-id-format",
      "type": "REGEX",
      "field": "/supplier/vatId",
      "pattern": "^DE[0-9]{9}$",
      "severity": "ERROR",
      "automationBlocker": true,
      "onFail": "FLAG"
    }
  ]
}

Rule fields

Every rule accepts these fields, whatever its type:

The type-specific fields (field, pattern, min, max, …) sit next to these.

Enum values are case-sensitive: "severity": "error" is not ERROR, so that rule is invalid and reports an ERROR verdict. An explicit null for severity or onFail also makes the rule invalid, while a null for a boolean field falls back to its default. When two rules share an id, the first one is evaluated and every later one reports ERROR with duplicate rule id.

Field pointers

Rules address values with RFC 6901 JSON pointers, the same format as fieldPath in schema validation. The empty pointer is the document root.

A pointer may contain at most one * segment, which expands over an array. The rule then produces one outcome per location that resolves, each with its own concrete pointer. /items/*/price over three items, of which only the first two have a price, produces two outcomes (/items/0/price and /items/1/price) and nothing for the third item. Two wildcards, or a non-empty pointer that does not start with /, is a configuration error.

A pointer into something the document does not have (an absent key, an index out of range, a wildcard over a non-array) is a miss, not an error. When the pointer as a whole resolves to nothing, the rule reports one SKIPPED (or FAIL with failOnMissing) at the pointer as configured, wildcard included. An explicit null at a location that did resolve reports SKIPPED (or FAIL) at that location.

An object key literally named * cannot be addressed, because * is always the wildcard.

Rule types

Values are read at their JSON type and never converted. A number written as a string, or a string where a number is expected, reports ERROR. Numbers are compared as exact decimals, so 1.50 equals 1.5 and 0.1 + 0.2 is exactly 0.3.

REGEX

{
  "id": "iban-shape",
  "type": "REGEX",
  "field": "/supplier/iban",
  "pattern": "^[A-Z]{2}[0-9]{2}[A-Z0-9]{11,30}$"
}

The pattern must match the whole value, not just occur in it, so "AT123 DE123456789 xx" does not satisfy a VAT ID pattern. The syntax is Java's regular expression syntax. Strings only: a number reports ERROR, as does a missing or invalid pattern.

RANGE

{
  "id": "vat-rate-plausible",
  "type": "RANGE",
  "field": "/taxRate",
  "min": 0,
  "max": 0.27
}

Both bounds are inclusive and at least one is required. A non-numeric bound, or a min above max, reports ERROR. So does a non-numeric value at field.

ENUM

{
  "id": "currency-supported",
  "type": "ENUM",
  "field": "/items/*/currency",
  "values": ["EUR", "CHF", "GBP"]
}

Values are compared by their text form, case-sensitively, so "de" does not satisfy ["DE"]. Two numbers compare as decimals, so 1.0 satisfies [1]. Mixed types fall back to text: the string "1" satisfies [1], but the number 1.0 does not satisfy ["1"]. An object or array at field reports ERROR. A missing or empty values array reports ERROR.

ARITHMETIC

{
  "id": "total-equals-line-items",
  "type": "ARITHMETIC",
  "field": "/totals/net",
  "equals": { "op": "SUM", "operands": ["/items/*/lineNet"] },
  "tolerance": 0.01
}

op is SUM; any other operator reports ERROR. tolerance is an absolute amount, 0 by default, and a difference exactly equal to it passes. Operands may use a wildcard, which is how a line-item sum is written; field may not.

An ARITHMETIC rule always produces one outcome. Its actual text shows both sides and the number of addends, for example 119.05 vs 119 (2 operands). Operands that resolve only partly are summed over what exists, and a null operand is not counted. The rule reports SKIPPED when field is absent or when no operand resolves; the two outcomes look identical, so check the output to see which side was missing.

CROSS_FIELD

{
  "id": "net-not-above-gross",
  "type": "CROSS_FIELD",
  "left": "/totals/net",
  "operator": "LTE",
  "right": "/totals/gross"
}

operator is EQ, NE, LT, LTE, GT or GTE. Two numbers compare as exact decimals. Two strings can only be compared with EQ and NE; ordering strings reports ERROR. Any other pairing (a number against a string, booleans, objects) reports ERROR. Neither side may use a wildcard. An absent or null side reports SKIPPED, and the outcome names the pointer of the missing side.

FORMAT

{
  "id": "invoice-date-iso",
  "type": "FORMAT",
  "field": "/invoiceDate",
  "format": "DATE_ISO"
}

An unknown format name reports ERROR. Strings only: a date extracted as a number reports ERROR.

Verdicts

Every outcome has exactly one verdict, and the report counts each:

Watch failCount for data quality, errorCount for rules that need fixing and skippedCount for coverage: a rule that is always SKIPPED is not checking anything. No verdict ever fails the extraction. See Rule Verdict Precedence for which verdict wins when several apply.

Severity vs automationBlocker

The two are independent. severity says how much a failure matters: ERROR for a defect in the document, WARNING for something that needs human attention, INFO for observation only. automationBlocker says whether a failure should stop straight-through processing. A rule can be INFO and blocking, or ERROR and non-blocking.

Both are copied onto every outcome of the rule, including passes: they describe the rule, the verdict describes what happened. Both also feed into composite confidence.

onFail is recorded, not enforced

onFail states what should happen when a rule fails. Today all four actions behave as WARN: the action is validated and recorded on every outcome, and the extraction result is returned unchanged whatever the verdict.

If a failing rule should stop your workflow now, enforce it in your own code from resultMetadata.rules.

Reading Rule Outcomes

The report is in resultMetadata.rules:

{
  "applied": true,
  "passCount": 1,
  "failCount": 1,
  "skippedCount": 1,
  "errorCount": 1,
  "truncated": false,
  "outcomes": [
    {
      "ruleId": "gross-equals-net-plus-vat",
      "field": "/totals/gross",
      "verdict": "FAIL",
      "expected": "sum(/totals/net, /totals/vat) ± 0.01",
      "actual": "119.05 vs 119 (2 operands)",
      "severity": "ERROR",
      "automationBlocker": true,
      "action": "REJECT"
    },
    {
      "ruleId": "vat-id-format",
      "field": "/supplier/vatId",
      "verdict": "PASS",
      "expected": "^DE[0-9]{9}$",
      "actual": "DE123456789",
      "severity": "ERROR",
      "automationBlocker": true,
      "action": "FLAG"
    },
    {
      "ruleId": "invoice-date-iso",
      "field": "/invoiceDate",
      "verdict": "SKIPPED",
      "expected": "value present",
      "actual": "absent",
      "severity": "WARNING",
      "automationBlocker": false,
      "action": "WARN"
    },
    {
      "ruleId": "tax-rate-plausible",
      "field": "/taxRate",
      "verdict": "ERROR",
      "expected": "numeric 'min'",
      "actual": "non-numeric 'min': ten",
      "severity": "WARNING",
      "automationBlocker": false,
      "action": "WARN"
    }
  ]
}

At most 200 outcomes are recorded, because passes are recorded too and a rule multiplies over array items. The four counts are always the true totals, and truncated is true when the list was cut. Read the counts, not the length of the list. expected and actual are cut at 500 characters and marked with a trailing ….

applied: false never means "all rules passed"; that is applied: true with failCount: 0. It means nothing was evaluated: the version has no rules, every rule is disabled, or rule evaluation could not run for this extraction. The three look identical in the result. validation.enabled: false does not cause it.

A rule that is always SKIPPED usually has a pointer that does not match the real shape of the output. Compare it with an output you got back, not with the schema. Set failOnMissing: true if absence is itself the problem, but note that array elements without the field never produce an outcome, so failOnMissing cannot flag them; require the field in the schema instead. A rule that is always ERROR needs fixing: read actual for the reason.

Rule Verdict Precedence

When more than one business rule verdict could apply, these decide:

• A broken rule beats an absent field. A rule's configuration is checked before the document is read. A rule with an invalid pattern, an empty values array or a min above max reports ERROR even when its field is also absent, so it counts in errorCount, never in skippedCount. A rule that can never work says so instead of reporting SKIPPED forever.
• A broken rule is one ERROR, not one per location. A configuration problem on a rule whose pointer covers fifty array items produces a single ERROR, because the problem belongs to the rule. A value-level ERROR, such as a wrong type at one location, is still reported per location, because it belongs to the document.
• failOnMissing turns SKIPPED into FAIL and nothing else. It does not affect ERROR. It reaches a pointer that resolved to nothing (one FAIL at the configured pointer) and an explicit null at a resolved location (one FAIL there).
• A rule that crashes is that rule's ERROR. It never suppresses the rules around it and never fails the extraction. The exception is a pattern so pathological that evaluation cannot continue at all; the whole report is then recorded as applied: false.