Handling Low-Confidence spaCy Predictions in Maintenance Intent Routing

A correctly designed NLP intent classification stage never dispatches a request the model is unsure about — it diverts it to a human. This page documents a regression where that guardrail quietly disappeared: a code path took the raw argmax of doc.cats and routed on it regardless of how thin the winning margin was, so a request the model was genuinely guessing on got stamped with a confident priority and escalation_tier and sent straight to dispatch. The fix restores the abstain path, adds a margin check the original threshold alone did not cover, and makes the review queue’s provisional tier deliberately conservative rather than a second guess.

Incident Profile

The request text was ambiguous by any human reading — a tenant complaint that mentioned both a tripped breaker and a warm room, phrased in a way that split the model’s attention almost evenly across two unrelated trades. The classifier’s own output made the uncertainty obvious in doc.cats, but nothing downstream looked at more than the single winning label before building a work order.

2026-06-11 03:22:17,004 INFO  [intent_router] Scoring WO-inbound-55210
2026-06-11 03:22:17,041 DEBUG [intent_router] doc.cats: {'general_request': 0.34, 'electrical_emergency': 0.31, 'hvac_repair': 0.22, 'plumbing_cm': 0.13}
2026-06-11 03:22:17,042 DEBUG [intent_router] argmax label='general_request' confidence=0.34
2026-06-11 03:22:17,043 INFO  [intent_router] Routed WO-55210 -> queue=general priority=standard escalation_tier=0
2026-06-11 03:22:17,044 INFO  [cmms_dispatch] POST /api/v1/work-orders 201 work_order_id=WO-55210
2026-06-11 07:48:52,930 ERROR [facility_ops] WO-55210 escalated manually — breaker panel arcing, tenant re-called twice, no technician assigned for 4h26m

The tell is not an exception; the pipeline never raised one. general_request won with 0.34, comfortably below any threshold that would inspire confidence, and the runner-up electrical_emergency trailed by only 0.03. A one-in-three prediction, effectively tied with a safety-relevant alternative, was treated as ground truth and mapped straight to the lowest-priority queue. The work order that should have carried escalation_tier=3 and a thirty-minute response window instead carried escalation_tier=0 and sat in the standard backlog for hours.

Root Cause Analysis

Nothing about the training data or the model architecture is at fault; doc.cats reported exactly what an honest, uncertain model should report. Two decisions upstream of the classifier produced the outcome.

A throughput fix silently removed the abstain path. The version of route_request that shipped to production had accumulated a growing human-review backlog, and a later patch aimed at “reducing manual work” changed the confident/triage branch into an always-dispatch branch, treating requires_human_review as advisory logging rather than a hard gate. The threshold check still ran and still logged a warning, but nothing stopped the dispatch call that followed it.
No check ever compared the top two scores. Even in the original, unmodified pipeline, the gate only asked whether the winning score cleared an absolute threshold. A distribution like {'general_request': 0.34, 'electrical_emergency': 0.31, ...} can sit entirely below a 0.75 threshold and still get caught by that check — but a distribution where the top label barely edges out a safety-relevant runner-up needs its own signal, because a near-tie is a stronger uncertainty indicator than the absolute score alone. Two intents effectively tied at one-third confidence each is not a case where “widen the model’s training data” fixes anything quickly; it is a case the routing layer must recognize and defer, structurally, every time it recurs.
The escalation tier was derived, not defaulted. Because the routing matrix maps a label directly to escalation_tier, an uncertain label produces a confident-looking tier with no marker that it came from a guess. Nothing downstream — dispatch, SLA timers, audit logs — can distinguish “the model is certain this is low priority” from “the model picked the least-bad option and rounded down.”

This is the same category of fragility the domain guide for Work Order Ingestion & Parsing Pipelines warns about at every stage boundary: a component that looks like it succeeded (HTTP 201, no exception, a valid payload) can still have smuggled an unjustified decision past the point where anyone can catch it. The classifier itself — built as an exclusive single-label model per training spaCy models for maintenance intent routing — was never the defect; the defect was trusting its winning label without asking how convincingly it won.

Resolution

The fix restores two checks as a single gate the routing code cannot bypass, and it changes what happens on failure from “log and continue” to “return a different, smaller object that dispatch cannot act on directly.”

Before — argmax with no threshold, no margin, no abstain path:

def classify_intent(nlp, text: str) -> dict:
    """Regression: takes the winning label unconditionally."""
    doc = nlp(text)
    cats = doc.cats
    best_label = max(cats, key=cats.get)
    # No threshold check. No margin check. No abstain path.
    # A 0.34-confidence guess is indistinguishable from a 0.98-confidence certainty.
    return {"label": best_label, "confidence": cats[best_label]}

After — full distribution read, threshold and margin both enforced, explicit abstain:

from dataclasses import dataclass, field
from typing import Dict, Optional


@dataclass
class IntentPrediction:
    """Full result of a classification pass — never just the winning label."""
    label: str
    confidence: float
    runner_up_label: Optional[str]
    margin: float
    distribution: Dict[str, float]
    confident: bool


def classify_intent(
    nlp,
    text: str,
    confidence_threshold: float = 0.75,
    margin_threshold: float = 0.15,
) -> IntentPrediction:
    """Read the whole label distribution and gate on both threshold and margin."""
    doc = nlp(text)
    distribution = dict(doc.cats)  # never discard the non-winning scores
    ranked = sorted(distribution.items(), key=lambda kv: kv[1], reverse=True)

    top_label, top_score = ranked[0]
    runner_up_label, runner_up_score = (ranked[1] if len(ranked) > 1 else (None, 0.0))
    margin = top_score - runner_up_score

    # Both conditions must hold: the winner must be strong on its own,
    # AND it must beat its closest rival by a real gap.
    confident = top_score >= confidence_threshold and margin >= margin_threshold

    return IntentPrediction(
        label=top_label,
        confidence=top_score,
        runner_up_label=runner_up_label,
        margin=margin,
        distribution=distribution,
        confident=confident,
    )

The margin check is what catches the incident case directly: general_request at 0.34 against electrical_emergency at 0.31 yields a margin of 0.03, far under a 0.15 margin threshold, so confident is False even though a naive implementation might someday lower the absolute threshold far enough to let 0.34 through on its own. Keeping both checks independent means fixing one blind spot never reopens the other.

Minimal Reproducible Pipeline

This script trains a tiny exclusive-label model, reproduces the ambiguous request from the incident, and shows both branches of the gate: a confident prediction resolves into a routed WorkOrderPayload with matched SLA fields, and the low-confidence prediction resolves into a ReviewQueueItem with a conservative provisional tier instead of a guess. Run it as-is.

from dataclasses import dataclass, field
from datetime import datetime, timedelta, timezone
from enum import Enum
from typing import Dict, List, Optional, Union

import spacy
from spacy.training import Example


class Priority(str, Enum):
    CRITICAL = "critical"
    HIGH = "high"
    STANDARD = "standard"
    PLANNED = "planned"


@dataclass
class WorkOrderPayload:
    """Canonical CMMS work order — SLA fields are mandatory site-wide."""
    work_order_id: str
    asset_id: str
    part_skus: List[str]
    required_quantities: Dict[str, int]
    priority: Priority = Priority.STANDARD
    requested_completion: Optional[datetime] = None
    escalation_tier: int = 0
    status: str = "open"
    created_at: datetime = field(default_factory=lambda: datetime.now(timezone.utc))


@dataclass
class ReviewQueueItem:
    """A request the classifier could not confidently resolve — never dispatched directly."""
    work_order_id: str
    asset_id: str
    raw_text: str
    top_label: str
    top_confidence: float
    runner_up_label: Optional[str]
    margin: float
    distribution: Dict[str, float]
    provisional_priority: Priority = Priority.HIGH
    provisional_escalation_tier: int = 1
    status: str = "pending_human_review"


@dataclass
class IntentPrediction:
    label: str
    confidence: float
    runner_up_label: Optional[str]
    margin: float
    distribution: Dict[str, float]
    confident: bool


ROUTING_MATRIX = {
    "general_request":     {"priority": Priority.STANDARD, "hours": 72, "tier": 0},
    "electrical_emergency": {"priority": Priority.CRITICAL, "hours": 0.5, "tier": 3},
    "hvac_repair":         {"priority": Priority.STANDARD, "hours": 8,  "tier": 0},
    "plumbing_cm":         {"priority": Priority.HIGH,     "hours": 4,  "tier": 1},
}

TRAIN_DATA = [
    ("please update the tenant contact record",
     {"cats": {"general_request": 1.0, "electrical_emergency": 0.0, "hvac_repair": 0.0, "plumbing_cm": 0.0}}),
    ("breaker panel arcing and sparking in unit 12",
     {"cats": {"general_request": 0.0, "electrical_emergency": 1.0, "hvac_repair": 0.0, "plumbing_cm": 0.0}}),
    ("thermostat unresponsive, room running warm all week",
     {"cats": {"general_request": 0.0, "electrical_emergency": 0.0, "hvac_repair": 1.0, "plumbing_cm": 0.0}}),
    ("slow drain backing up in the break room sink",
     {"cats": {"general_request": 0.0, "electrical_emergency": 0.0, "hvac_repair": 0.0, "plumbing_cm": 1.0}}),
]


def build_model() -> spacy.Language:
    nlp = spacy.blank("en")
    textcat = nlp.add_pipe("textcat")  # exclusive single-label head
    for label in ROUTING_MATRIX:
        textcat.add_label(label)

    optimizer = nlp.initialize()
    for _ in range(25):
        losses: Dict[str, float] = {}
        for text, annotations in TRAIN_DATA:
            example = Example.from_dict(nlp.make_doc(text), annotations)
            nlp.update([example], sgd=optimizer, losses=losses)
    return nlp


def classify_intent(
    nlp,
    text: str,
    confidence_threshold: float = 0.75,
    margin_threshold: float = 0.15,
) -> IntentPrediction:
    """Read the whole label distribution and gate on both threshold and margin."""
    doc = nlp(text)
    distribution = dict(doc.cats)
    ranked = sorted(distribution.items(), key=lambda kv: kv[1], reverse=True)

    top_label, top_score = ranked[0]
    runner_up_label, runner_up_score = (ranked[1] if len(ranked) > 1 else (None, 0.0))
    margin = top_score - runner_up_score
    confident = top_score >= confidence_threshold and margin >= margin_threshold

    return IntentPrediction(
        label=top_label,
        confidence=top_score,
        runner_up_label=runner_up_label,
        margin=margin,
        distribution=distribution,
        confident=confident,
    )


def route(
    prediction: IntentPrediction, work_order_id: str, asset_id: str, raw_text: str
) -> Union[WorkOrderPayload, ReviewQueueItem]:
    """Confident predictions dispatch; everything else abstains to a conservative review item."""
    if prediction.confident:
        sla = ROUTING_MATRIX[prediction.label]
        return WorkOrderPayload(
            work_order_id=work_order_id,
            asset_id=asset_id,
            part_skus=[],
            required_quantities={},
            priority=sla["priority"],
            requested_completion=datetime.now(timezone.utc) + timedelta(hours=sla["hours"]),
            escalation_tier=sla["tier"],
        )

    # Low confidence or thin margin: never guess a tier from an uncertain label.
    # Default to a conservative HIGH/tier-1 provisional so an ambiguous emergency
    # is never buried in the standard backlog while a human reviews it.
    return ReviewQueueItem(
        work_order_id=work_order_id,
        asset_id=asset_id,
        raw_text=raw_text,
        top_label=prediction.label,
        top_confidence=prediction.confidence,
        runner_up_label=prediction.runner_up_label,
        margin=prediction.margin,
        distribution=prediction.distribution,
    )


if __name__ == "__main__":
    model = build_model()

    # The incident case: two intents nearly tied, both well under a comfortable margin.
    ambiguous_text = "breaker keeps tripping and the room feels warm, not sure who to call"
    prediction = classify_intent(model, ambiguous_text)
    result = route(prediction, "WO-55210", "ASSET-7702", ambiguous_text)

    print(f"confident={prediction.confidence:.2f} margin={prediction.margin:.2f}")
    if isinstance(result, ReviewQueueItem):
        print(f"-> {result.status}: provisional {result.provisional_priority}, tier {result.provisional_escalation_tier}")
    else:
        print(f"-> dispatched: priority={result.priority}, tier={result.escalation_tier}")

    # A clear-cut case for contrast: high confidence, wide margin, dispatches directly.
    clear_text = "breaker panel arcing and sparking, smell of smoke"
    prediction2 = classify_intent(model, clear_text)
    result2 = route(prediction2, "WO-55211", "ASSET-7703", clear_text)
    print(f"confident={prediction2.confidence:.2f} margin={prediction2.margin:.2f}")
    if isinstance(result2, WorkOrderPayload):
        print(f"-> dispatched: priority={result2.priority}, tier={result2.escalation_tier}")

The confidence and margin values will vary slightly with this toy training set’s randomness, but the structural guarantee holds regardless: the ambiguous text either fails the threshold, fails the margin, or both, and comes back as a ReviewQueueItem with provisional_escalation_tier=1 — never as a silently mis-prioritized dispatch. That ReviewQueueItem still needs the same SLA tier and escalation machinery as a routed work order, just started conservatively instead of guessed low, so a human reviewer’s decision only ever tightens the tier, never loosens a request that was already sitting exposed. Whatever the reviewer confirms should still be written back into the same WorkOrderPayload shape defined in work order schema standards, so the rest of the pipeline never has to special-case a human-resolved ticket differently from a model-resolved one.

Prevention Checklist

Read the full doc.cats distribution on every prediction, never only the argmax label and its score.
Enforce both a confidence threshold and a top-2 margin check as one gate; a near-tie must abstain even if the winning score alone would pass.
Treat requires_human_review (or its equivalent) as a hard branch in code, not a log line a later refactor can silently detach from the dispatch call.
Default the review queue’s provisional escalation_tier and priority to a conservative, elevated value — never derive them from the low-confidence label itself.
Add a regression test that asserts a near-tied distribution (like {0.34, 0.31, ...}) always produces a ReviewQueueItem, so the abstain path cannot be quietly removed again.

FAQ

Why check the margin between the top two labels instead of just raising the confidence threshold?

Raising the threshold alone cannot catch every dangerous case. A distribution can have its winner sit right at the threshold while a safety-relevant runner-up trails by only a hair — the absolute score looks acceptable, but the model was effectively guessing between two options. The margin check catches that near-tie directly regardless of where the threshold happens to be set.

Why does the review queue’s provisional tier default high instead of matching the winning label?

Because the winning label is exactly what the model was not confident about. Defaulting to the label’s own tier can silently reproduce the original bug — an uncertain general_request guess still stamps a low tier. A conservative default means the worst outcome of an abstain is a technician arriving slightly sooner than strictly necessary, not a genuine emergency sitting in the standard backlog for hours.

Doesn’t gating on both threshold and margin send more requests to human review?

Yes, and that is the intended trade-off — a higher triage rate is a visible, measurable cost, while a mis-prioritized auto-dispatch is invisible until it causes an incident. Track the review queue’s depth and the confidence distribution over time; a persistently high abstain rate is a signal to retrain per training spaCy models for maintenance intent routing, not a reason to loosen the gate.

Can this same gate be reused for other classification stages in the pipeline?

Yes — the IntentPrediction shape and the classify_intent gate are general to any exclusive-label textcat model in this pipeline. The same threshold-plus-margin pattern applies wherever a single low-confidence label would otherwise be trusted as if it were certain.

This page hardens the guardrail defined in NLP intent classification against a specific regression; the model itself is built following training spaCy models for maintenance intent routing, the resolved work order must still satisfy work order schema standards, a deferred or reviewed item still runs against SLA timer escalation, and high-volume scoring belongs inside async batch processing.

Part of: NLP Intent Classification.