Training spaCy Models for Maintenance Intent Routing: Fixing Multi-Label Collisions

A custom spaCy text-classification model trains cleanly, ships without raising an exception, and then quietly routes a single maintenance request into two or three downstream queues at once. The model is not broken — it is doing exactly what its architecture was configured to do: emit independent per-label probabilities instead of one mutually exclusive decision. This is the most common production defect when teams add a trained classifier to the NLP intent classification stage of their Work Order Ingestion & Parsing Pipelines, and it surfaces as duplicate tickets, double dispatch, and broken single-ticket-per-request SLAs. The procedure below isolates the misconfiguration, switches the model to an exclusive routing head, retrains, and hardens inference so every work order resolves to exactly one trade.

Incident Profile

The failure appears during batch scoring, when the ingestion service reads doc.cats and finds confidence spread across two semantically adjacent intents rather than concentrated on one. Two intents clear the routing threshold, two tickets are created, and the deduplication layer rejects the overlapping asset tags:

2024-05-12 09:14:22,110 | INFO  | cmms_router   | Routing payload: WO-88421
2024-05-12 09:14:22,115 | DEBUG | nlp_inference | doc.cats: {'hvac_pm': 0.48, 'electrical_cm': 0.46, 'plumbing_cm': 0.41, 'general_request': 0.07}
2024-05-12 09:14:22,118 | WARN  | cmms_router   | Multi-intent collision: hvac_pm and electrical_cm both exceed route threshold
2024-05-12 09:14:22,120 | ERROR | cmms_router   | Duplicate ticket rejected; overlapping asset_id ASSET-3391 in two queues

The tell is in the numbers. With a healthy single-label model the probabilities form a distribution that sums to 1.0, so a confident prediction looks like {'hvac_pm': 0.91, ...} with the rest near zero. Here the four values sum to 1.42. Each category is being scored on its own, independent of the others — the signature of a multi-label classifier wired into a single-label routing decision.

One-line diagnostic. Score a validation batch and sum the category probabilities. If the sum drifts above 1.0, the pipeline is in multi-label mode:

import spacy

nlp = spacy.load("./cmms_intent_router")
suspect = "AHU-04 filter overdue and breaker keeps tripping"
doc = nlp(suspect)
print(round(sum(doc.cats.values()), 3))  # exclusive model -> 1.0; multi-label -> > 1.0

Root Cause Analysis

The defect lives in the pipeline architecture, not in the training data. spaCy v3 ships two text-classification components, and they are easy to swap by accident:

  • textcat is the single-label component. Its output head applies a softmax, so the predicted categories are mutually exclusive and the probabilities sum to 1.0. This is what intent routing needs.
  • textcat_multilabel is the multi-label component. Its head applies a per-class sigmoid, so each category is an independent yes/no decision and the probabilities do not sum to 1.0.

A pipeline lands in the broken state in one of two ways. Either someone called nlp.add_pipe("textcat_multilabel") because the name sounded like the safer, more flexible choice, or a hand-edited config.cfg left the textcat model with exclusive_classes = false, which forces the supposedly single-label component to use a sigmoid head anyway. Both produce the same doc.cats that sums above 1.0.

Multi-label scoring is correct when annotations genuinely overlap — flagging a request as both safety_hazard and electrical_cm is legitimate. It is wrong for trade routing, where a work order must map to exactly one crew. A sigmoid head has no mechanism to make categories compete, so when two intents share vocabulary (“breaker,” “panel,” “overheating”), both light up and both clear the threshold. The routing layer then does precisely what the model told it to: it opens a ticket per qualifying intent.

Sigmoid multi-label head versus softmax exclusive head on the same work order Two panels score the identical request "AHU filter overdue and breaker keeps tripping". The left panel, textcat_multilabel with a sigmoid head, scores each intent independently: hvac_pm 0.48, electrical_cm 0.46, plumbing_cm 0.41 and general_request 0.07, which sum to 1.42. Both hvac_pm and electrical_cm cross the 0.45 routing threshold, so two tickets are created and the deduplication layer flags a collision on the shared asset. The right panel, textcat with exclusive_classes set to true and a softmax head, makes the intents compete: hvac_pm wins at 0.91 while electrical_cm 0.05, plumbing_cm 0.03 and general_request 0.01 collapse, summing to 1.00. Only hvac_pm clears the threshold, so exactly one ticket is routed. Same request, two output heads — “AHU filter overdue and breaker keeps tripping” textcat_multilabel · sigmoid head each intent scored independently — no competition route ≥ 0.45 hvac_pm 0.48 electrical_cm 0.46 plumbing_cm 0.41 general_request 0.07 Σ = 1.42 → 2 intents clear threshold Ticket A · hvac_pm asset ASSET-3391 Ticket B · electrical_cm asset ASSET-3391 duplicate dispatch — collision rejected textcat · softmax head exclusive_classes = true — intents compete route ≥ 0.45 hvac_pm 0.91 electrical_cm 0.05 plumbing_cm 0.03 general_request 0.01 Σ = 1.00 → 1 intent clears threshold Ticket · hvac_pm asset ASSET-3391 exactly one trade — clean dispatch

Resolution

The fix is to score intents against one another with a softmax head. Below is the before/after for both ways the bug appears.

Before — independent per-class scoring (collides):

import spacy

nlp = spacy.blank("en")
# Multi-label component: sigmoid head, probabilities do NOT compete.
textcat = nlp.add_pipe("textcat_multilabel")
for label in ("hvac_pm", "electrical_cm", "plumbing_cm", "general_request"):
    textcat.add_label(label)

After — mutually exclusive scoring (one trade wins):

import spacy

nlp = spacy.blank("en")
# Single-label component: softmax head, probabilities compete and sum to 1.0.
textcat = nlp.add_pipe("textcat")  # NOT textcat_multilabel
for label in ("hvac_pm", "electrical_cm", "plumbing_cm", "general_request"):
    textcat.add_label(label)

If you train from a config.cfg rather than building the pipeline in code, the same correction is one key in the [components.textcat.model] block. Patch it before training so the architecture is locked at the source instead of relying on whoever last edited the file:

from pathlib import Path
from spacy.util import load_config_from_disk


def patch_textcat_exclusive(config_path: Path, output_path: Path) -> None:
    """Force the single-label (softmax) routing head in a spaCy v3 config."""
    config = load_config_from_disk(config_path)

    # The single-label component must be named `textcat`, not `textcat_multilabel`.
    if "textcat_multilabel" in config["components"]:
        raise ValueError(
            "Config uses textcat_multilabel; rebuild the config with `textcat` "
            "for exclusive intent routing."
        )

    # exclusive_classes = True selects the softmax head over the sigmoid head.
    config["components"]["textcat"]["model"]["exclusive_classes"] = True
    config.to_disk(output_path)


patch_textcat_exclusive(Path("base_config.cfg"), Path("cmms_exclusive_config.cfg"))

The two changes are equivalent: both guarantee that the predicted categories compete, so the winning intent absorbs probability mass from its rivals and the routing layer sees a single, decisive label.

Minimal Reproducible Pipeline

The script below trains a corrected model, scores a request, and turns the winning intent into the canonical WorkOrderPayload used everywhere on this site. It is self-contained — paste it into a file and run it. The routing matrix populates the SLA fields (priority, requested_completion, escalation_tier) from the matched intent, exactly as the production stage does before handing off to dispatch.

from dataclasses import dataclass, field
from datetime import datetime, timedelta, timezone
from enum import Enum
from typing import Dict, List, Optional

import spacy
from spacy.training import Example


class Priority(str, Enum):
    CRITICAL = "critical"
    HIGH = "high"
    STANDARD = "standard"
    PLANNED = "planned"


@dataclass
class WorkOrderPayload:
    """Canonical CMMS work order — SLA fields are mandatory site-wide."""
    work_order_id: str
    asset_id: str
    part_skus: List[str]
    required_quantities: Dict[str, int]
    priority: Priority = Priority.STANDARD
    requested_completion: Optional[datetime] = None
    escalation_tier: int = 0
    status: str = "open"
    created_at: datetime = field(default_factory=lambda: datetime.now(timezone.utc))


# Intent -> SLA mapping. Keep this matrix and the model's label set under one review.
ROUTING_MATRIX = {
    "hvac_pm":        {"priority": Priority.PLANNED,  "hours": 168, "tier": 0},
    "electrical_cm":  {"priority": Priority.HIGH,     "hours": 8,   "tier": 1},
    "plumbing_cm":    {"priority": Priority.CRITICAL, "hours": 4,   "tier": 2},
    "general_request": {"priority": Priority.STANDARD, "hours": 72, "tier": 0},
}

TRAIN_DATA = [
    ("AHU-04 filter replacement overdue",
     {"cats": {"hvac_pm": 1.0, "electrical_cm": 0.0, "plumbing_cm": 0.0, "general_request": 0.0}}),
    ("Breaker panel tripping in Bldg 3",
     {"cats": {"hvac_pm": 0.0, "electrical_cm": 1.0, "plumbing_cm": 0.0, "general_request": 0.0}}),
    ("Emergency leak in mechanical room 2B",
     {"cats": {"hvac_pm": 0.0, "electrical_cm": 0.0, "plumbing_cm": 1.0, "general_request": 0.0}}),
    ("Please update the asset photo on record",
     {"cats": {"hvac_pm": 0.0, "electrical_cm": 0.0, "plumbing_cm": 0.0, "general_request": 1.0}}),
]


def build_model() -> spacy.Language:
    nlp = spacy.blank("en")
    textcat = nlp.add_pipe("textcat")  # exclusive softmax head
    for label in ROUTING_MATRIX:
        textcat.add_label(label)

    optimizer = nlp.initialize()
    for _ in range(20):
        losses: Dict[str, float] = {}
        for text, annotations in TRAIN_DATA:
            example = Example.from_dict(nlp.make_doc(text), annotations)
            nlp.update([example], sgd=optimizer, losses=losses)
    return nlp


def route(nlp: spacy.Language, work_order_id: str, asset_id: str,
          text: str, threshold: float = 0.65) -> WorkOrderPayload:
    cats = nlp(text).cats

    # Exclusive selection: argmax over a distribution that sums to 1.0.
    intent = max(cats, key=cats.get)
    confidence = cats[intent]

    # Guardrail: a genuinely ambiguous request goes to triage, not a guessed trade.
    if confidence < threshold:
        intent = "general_request"

    sla = ROUTING_MATRIX[intent]
    return WorkOrderPayload(
        work_order_id=work_order_id,
        asset_id=asset_id,
        part_skus=[],
        required_quantities={},
        priority=sla["priority"],
        requested_completion=datetime.now(timezone.utc) + timedelta(hours=sla["hours"]),
        escalation_tier=sla["tier"],
    )


if __name__ == "__main__":
    model = build_model()

    sample = model("Emergency leak in mechanical room 2B")
    assert abs(sum(sample.cats.values()) - 1.0) < 1e-4, "Model is still multi-label!"

    payload = route(model, "WO-88421", "ASSET-3391", "Emergency leak in mechanical room 2B")
    print(payload.priority, payload.escalation_tier)  # Priority.CRITICAL 2

The assert is the regression guard: an exclusive model always satisfies it, and a model that slipped back to multi-label mode fails loudly before it can ever construct a payload. Once a request is routed, the resolved intent and SLA fields flow downstream into async batch processing for dispatch, and parts referenced on the work order are resolved through parts availability checks.

Prevention Checklist

Frequently Asked Questions

Why does doc.cats sum to more than 1.0?

The model is using a sigmoid output head, which scores each category independently. That happens when the pipeline was built with textcat_multilabel, or when a textcat config left exclusive_classes = false. Switch to the single-label textcat component with exclusive_classes = true and the softmax head forces the categories to compete, so the probabilities sum to 1.0.

Is textcat_multilabel ever the right choice for routing?

Only when a single request can legitimately belong to more than one trade at once and you want to open multiple work orders deliberately. For standard one-ticket-per-request routing it is the wrong component, because it gives you no mutually exclusive decision and pushes the disambiguation burden onto brittle threshold tuning downstream.

Do I have to retrain after switching the head?

Yes. The output layer changes from sigmoid to softmax, so the learned weights for the head are no longer valid. Retrain from your single-label annotations — the same {"cats": {...}} data works for both components — and validate that the probability sum collapses to 1.0 before promoting the model.

How does this connect to the upstream pipeline?

This stage assumes clean text. Capture happens earlier in email intake configuration, attachments are read by PDF parsing with Python, and the fields this model emits must conform to work order schema standards before dispatch.

Return to NLP intent classification for the full routing stage, scale the corrected model inside async batch processing, and compare debugging patterns with extracting tables from PDF work orders using pdfplumber and configuring IMAP polling for maintenance email queues.

Part of: Work Order Ingestion & Parsing Pipelines.