Debugging MTBF-to-PM Interval Miscalculations in CMMS Routing Pipelines
Maintenance engineers report the same symptom again and again: immediately after a fresh mean-time-between-failures (MTBF) load, preventive maintenance (PM) work orders stop honoring their quarterly or monthly cadence and start firing daily — or, worse, with a next_due_date that lands before the last completion. The damage is downstream. Technician queues flood, the routing engine rejects payloads, and trust in the schedule evaporates. The fault almost never lives in the failure data itself; it lives in the PM interval calculation stage, where unfiltered downtime states, timezone drift, and a bare division turn clean reliability numbers into invalid due dates. This page isolates the exact failure and ships a calibrated fix.
Incident Profile
The Python aggregation worker degrades quietly. Nothing surfaces until the routing scheduler tries to resolve next_due_date against the asset’s last completion and the CMMS API rejects the write. The log trace at the point of failure is distinctive:
[2026-06-14T08:12:03Z] INFO ingesting MTBF metrics | asset_group=HVAC-CHILLERS
[2026-06-14T08:12:04Z] DEBUG raw MTBF values: [142.5, 0.0, 89.2, -12.4, 320.1]
[2026-06-14T08:12:04Z] WARN division by zero in interval_normalizer.py:47
[2026-06-14T08:12:04Z] ERROR ValueError: invalid interval resolution | pm_interval_days=-0.34
[2026-06-14T08:12:05Z] CRIT CMMS routing API rejected payload | HTTP 422: next_due_date < last_completion_date
Three tells in that excerpt point straight at the root cause. The raw values include 0.0 and -12.4 — planned downtime and a manual reset that were never filtered out. The division-by-zero warning means a 0.0 reached the denominator path. And the final HTTP 422 is the routing engine refusing a due date that violates monotonic progression. Any one of these is enough to corrupt the schedule; together they produce the negative interval seen on line four.
Root Cause Analysis
The pipeline assumes every failure_timestamp record represents an unplanned breakdown. Production CMMS exports never honor that assumption — they interleave corrective maintenance, planned shutdowns, sensor calibration windows, and manual resets. When the aggregation query sums operational hours without filtering event_type != 'PLANNED', the observed MTBF collapses toward zero. The worker then runs the offending line, pm_interval_days = mtbf_hours / 24, with no minimum boundary and no safety margin, so a depressed MTBF yields a sub-daily trigger and a 0.0 yields the division-by-zero you saw logged.
A second, quieter defect compounds it. UTC ingestion timestamps are applied directly to local maintenance windows with no offset correction, so last_failure_date drifts by the local UTC offset across daylight-saving boundaries. That drift can push the projected next_due_date behind the previous completion, which violates the work order state machine defined under the CMMS Architecture & Maintenance Taxonomy — it requires strict monotonic progression from COMPLETED to SCHEDULED. The routing engine enforces that rule with the HTTP 422 rejection, and a naive retry loop simply replays the same invalid payload.
Resolution: Before and After
The fix is a state-aware normalizer that filters non-failure events, applies a safety margin, clamps the interval to the routing schema’s accepted range, and resolves the due date in the facility’s own timezone before guarding monotonic progression. Compare the original logic against the hardened version.
# BEFORE — naive normalizer: ships sub-daily, zero, and negative intervals
from datetime import timedelta
def mtbf_to_due_date(raw_mtbf_hours, last_completion):
mtbf = sum(raw_mtbf_hours) / len(raw_mtbf_hours) # 0.0 and negatives included
interval_days = mtbf / 24 # no floor, no safety margin
return last_completion + timedelta(days=interval_days) # may precede last_completion
# AFTER — state-aware normalizer aligned with CMMS routing constraints
import logging
from datetime import datetime, timedelta
from zoneinfo import ZoneInfo
from typing import List, Optional
logger = logging.getLogger(__name__)
MIN_INTERVAL_DAYS = 7.0 # routing schema floor: nothing schedules sub-weekly
MAX_INTERVAL_DAYS = 365.0 # routing schema ceiling: forces an annual touch
SAFETY_FACTOR = 0.85 # land PM before the failure mean, not on it
def normalize_mtbf_to_due_date(
raw_mtbf_hours: List[float],
last_completion_utc: str,
facility_tz: str = "America/New_York",
min_days: float = MIN_INTERVAL_DAYS,
max_days: float = MAX_INTERVAL_DAYS,
) -> Optional[datetime]:
"""Resolve a valid PM next_due_date from raw MTBF data, or None to fall back."""
# 1. Drop planned downtime, sensor noise, and negative logs before any math.
valid_mtbf = [h for h in raw_mtbf_hours if h > 0.0]
if not valid_mtbf:
logger.warning("no valid MTBF records — caller should use default schedule")
return None # never divide by an empty/zero population
# 2. Derate the observed mean so PM lands ahead of the failure point.
avg_mtbf_hours = sum(valid_mtbf) / len(valid_mtbf)
interval_days = (avg_mtbf_hours * SAFETY_FACTOR) / 24.0
# 3. Clamp to the routing schema range — kills sub-daily and runaway intervals.
interval_days = max(min_days, min(interval_days, max_days))
# 4. Resolve the due date in the facility timezone so DST cannot drift it.
last_completed = datetime.fromisoformat(last_completion_utc).replace(
tzinfo=ZoneInfo("UTC")
).astimezone(ZoneInfo(facility_tz))
next_due = last_completed + timedelta(days=interval_days)
# 5. Monotonic guard: next_due must always follow last_completed.
if next_due <= last_completed:
logger.error("monotonic progression violated — forcing minimum interval")
next_due = last_completed + timedelta(days=min_days)
logger.info("resolved PM interval: %.2f days | due=%s",
interval_days, next_due.isoformat())
return next_due
Each numbered change maps to one symptom in the trace. Step 1 removes the 0.0 and -12.4 that caused the division-by-zero and artificial MTBF compression. Step 2 keeps PM ahead of the failure mean instead of scheduling exactly on it. Step 3 is the clamp that makes a -0.34 impossible. Step 4 replaces the bare UTC arithmetic with zoneinfo, so DST transitions no longer shift the anchor. Step 5 is the last line of defense against the HTTP 422 — it guarantees the payload the routing engine receives already satisfies monotonic progression.
Minimal Reproducible Pipeline
This script runs end to end with the standard library plus the canonical work order model. Drop it into a file and execute it — it reproduces the bad input, applies the fix, and emits a schema-valid PM work order. The WorkOrderPayload is the same shape used across the pipeline, carrying the SLA fields (priority, requested_completion, escalation_tier) that the routing engine reads; keeping its definitions identical to work order schema standards means the output drops straight into validation without redefinition errors.
import logging
from dataclasses import dataclass, field
from datetime import datetime, timedelta, timezone
from enum import Enum
from typing import Dict, List, Optional
from zoneinfo import ZoneInfo
logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)
MIN_INTERVAL_DAYS = 7.0
MAX_INTERVAL_DAYS = 365.0
SAFETY_FACTOR = 0.85
class Priority(str, Enum):
CRITICAL = "critical"
HIGH = "high"
STANDARD = "standard"
PLANNED = "planned"
@dataclass
class WorkOrderPayload:
"""Canonical CMMS work order — SLA fields are mandatory site-wide."""
work_order_id: str
asset_id: str
part_skus: List[str]
required_quantities: Dict[str, int]
priority: Priority = Priority.STANDARD
requested_completion: Optional[datetime] = None
escalation_tier: int = 0
status: str = "open"
created_at: datetime = field(default_factory=lambda: datetime.now(timezone.utc))
def normalize_mtbf_to_due_date(
raw_mtbf_hours: List[float],
last_completion_utc: str,
facility_tz: str = "America/New_York",
) -> Optional[datetime]:
valid_mtbf = [h for h in raw_mtbf_hours if h > 0.0]
if not valid_mtbf:
logger.warning("no valid MTBF records — using default schedule")
return None
interval_days = (sum(valid_mtbf) / len(valid_mtbf) * SAFETY_FACTOR) / 24.0
interval_days = max(MIN_INTERVAL_DAYS, min(interval_days, MAX_INTERVAL_DAYS))
last_completed = datetime.fromisoformat(last_completion_utc).replace(
tzinfo=ZoneInfo("UTC")
).astimezone(ZoneInfo(facility_tz))
next_due = last_completed + timedelta(days=interval_days)
if next_due <= last_completed:
next_due = last_completed + timedelta(days=MIN_INTERVAL_DAYS)
return next_due
def build_pm_work_order(asset_id: str, raw_mtbf_hours: List[float],
last_completion_utc: str) -> Optional[WorkOrderPayload]:
due = normalize_mtbf_to_due_date(raw_mtbf_hours, last_completion_utc)
if due is None:
return None # caller falls back to manufacturer-recommended schedule
return WorkOrderPayload(
work_order_id=f"PM-{asset_id}-{due:%Y%m%d}",
asset_id=asset_id,
part_skus=["FILTER-HVAC-24X24"],
required_quantities={"FILTER-HVAC-24X24": 2},
priority=Priority.PLANNED,
requested_completion=due,
escalation_tier=0,
)
if __name__ == "__main__":
# The exact corrupting input from the incident log.
raw = [142.5, 0.0, 89.2, -12.4, 320.1]
wo = build_pm_work_order("HVAC-CHILLER-07", raw, "2026-06-14T08:12:03")
assert wo is not None
assert wo.requested_completion > datetime.fromisoformat(
"2026-06-14T08:12:03"
).replace(tzinfo=ZoneInfo("UTC"))
print(f"{wo.work_order_id} due {wo.requested_completion:%Y-%m-%d} "
f"priority={wo.priority.value}")
Running it prints a single line such as PM-HVAC-CHILLER-07 due 2026-06-21 priority=planned. The interval clamps to the seven-day floor because the surviving MTBF samples derate below it — exactly the behavior that prevents the original daily storm. Swap the floor, feed a richer history, or change the timezone and re-run to confirm the output stays monotonic and schema-valid.
Prevention Checklist
Work through these before re-enabling production routing on any asset group whose MTBF feed has changed.
Frequently Asked Questions
Why does an MTBF of zero crash the worker instead of just scheduling nothing?
A zero or empty sample reaches the bare mtbf / 24 division and raises before any guard runs. Filtering non-positive values first, then short-circuiting to a None fallback when nothing survives, turns the crash into a clean handoff to the default schedule. The hardened normalizer does both in steps one and two.
How many failure samples do I need before trusting the MTBF interval?
Treat fewer than five observed intervals as low confidence and keep the manufacturer baseline as the fallback until the sample grows. A sparse history yields a high-variance mean that swings the interval every cycle; the deeper statistical treatment and confidence gating live in the parent PM interval calculation guide.
Does this belong in the synchronous worker or behind the broker?
Run it on every validated telemetry snapshot, driven by the same broker that handles async batch processing, and keep the math idempotent so redundant recalculations are cheap. A nightly reconciliation pass then catches any asset whose telemetry was delayed or dropped.
Related
Ground the statistical method and configuration reference in the parent PM interval calculation guide, keep the emitted payload conformant with JSON Schema validation for work order payloads, make recalculation safe under retries with implementing Celery for async work order batching, risk-weight intervals from the equipment tree via asset hierarchy design, and confirm spares before a generated PM dispatches through parts availability checks.
Part of: CMMS Architecture & Maintenance Taxonomy.