Debugging RBAC 403 Denials in CMMS Preventive Maintenance Routing
Automated preventive maintenance (PM) routing pipelines fail at dispatch when role-based access control (RBAC) boundaries do not line up with the asset tree the work order targets. The classic symptom: a Python integration service account that creates work orders all day suddenly receives a 403 Forbidden the moment it tries to route a PM across a site boundary. The root cause almost always lives in the security and access boundaries stage of the CMMS Architecture & Maintenance Taxonomy layer, where a role grant fails to inherit down to the exact node the routing action executes on. This guide isolates that failure, fixes it, and gives you a runnable simulator to prove the fix before it touches production.
Incident Profile
When the routing engine evaluates a PM schedule, it checks the caller’s role against the target asset’s security group at the precise node named in the payload — not at the PM schedule’s origin node. If the service account lacks the routing privilege at that node, the API rejects the dispatch. The following trace captures the exact failure during a scheduled PM run:
2026-05-14T08:12:03Z [CMMS-API] POST /api/v1/workorders/route
2026-05-14T08:12:03Z [AUTH] Token validated: svc_pm_router@prod
2026-05-14T08:12:03Z [RBAC] Evaluating role: integration_pm_dispatcher
2026-05-14T08:12:03Z [RBAC] Checking scope: pm:route:execute on asset_id: HVAC-CH-04
2026-05-14T08:12:03Z [RBAC] Scope DENIED. Missing inheritance from parent site: SITE-BLDG-01
2026-05-14T08:12:03Z [RESPONSE] HTTP 403 {"error": "insufficient_permissions", "required_scope": "pm:route:execute", "context": "cross_hierarchy_routing"}The give-away is the pairing of required_scope: pm:route:execute with context: cross_hierarchy_routing. The token authenticated cleanly and the role resolved — the denial is authorization, not authentication. The integration_pm_dispatcher role holds workorder:create but not the routing privilege needed to assign labor and push a PM down to a child asset, and the scope it does need is not inherited from the parent site SITE-BLDG-01.
Root Cause Analysis
CMMS platforms enforce RBAC at three layers: the user or service account, the role definition, and the asset security group. A routing action only succeeds when the required scope is present on the role and that scope resolves on the target node’s security boundary. Three independent gaps produce the same 403:
- Token claim gap. The service account token never carried the
pm:route:executeclaim — usually a stale OAuth2 scope set or an identity-provider mapping that grantsworkorder:createbut not routing. - Role-to-security-group decoupling. The role exists but is not bound to the parent site’s
Maintenance_Routingsecurity group, so the inheritance chain that the canonical nodes from asset hierarchy design rely on never reaches the child asset. - Inheritance disabled on the action. Asset-hierarchy inheritance is explicitly turned off for
pm:routein the policy engine to block privilege escalation across tenant or site boundaries — a safe default that silently denies legitimate cross-site PM routing.
Integration pipelines routinely assume top-down inheritance is automatic. Enterprise CMMS platforms evaluate the exact asset_id in the payload, so a PM authored at the system level but dispatched to a site-specific asset is checked at the child node, where the grant was never propagated. Shared assets, dynamically provisioned equipment, and child nodes with overridden local policies all amplify this.
Resolution
The fix has two halves: grant the missing scope and let it inherit to the target node. The before/after below is the routing pre-flight check that the pipeline runs before calling the API. The “before” version trusts that creation rights imply routing rights; the “after” version checks the actual routing scope at the resolved node and refuses to dispatch when it is absent — failing closed rather than firing a request guaranteed to 403.
# BEFORE — assumes workorder:create implies routing, never inspects the target node
def can_route(role_scopes: set[str]) -> bool:
# Bug: routing is gated on creation, and the asset node is never consulted,
# so a cross-site PM passes this check and then 403s at the API.
return "workorder:create" in role_scopes# AFTER — checks the routing scope AND that it resolves on the target node
def can_route(
role_scopes: set[str],
resolved_scopes_at_node: set[str], # scopes that actually inherit to asset_id
) -> bool:
# 1. The role itself must carry the routing privilege, not just creation rights.
if "pm:route:execute" not in role_scopes:
return False
# 2. That privilege must inherit all the way down to the node being routed to;
# a grant stranded at the parent site does not authorize the child asset.
return "pm:route:execute" in resolved_scopes_at_nodeTo make resolved_scopes_at_node actually contain the scope, apply three configuration changes on the CMMS side:
- Issue the scope. Update the identity provider so the client-credentials flow assigns
pm:route:execute(alongsidepm:dispatch) to the service account. - Bind the role to the parent security group. Attach
integration_pm_dispatcherto theSITE-BLDG-01Maintenance_Routinggroup and enable “propagate to child assets.” - Enable inheritance for the action. Turn on hierarchy inheritance for
pm:routeandpm:dispatchso the grant resolves atHVAC-CH-04, not just at the site root.
Minimal Reproducible Pipeline
The script below reproduces the 403 and then verifies the fix without touching a live CMMS. It carries the canonical WorkOrderPayload — including the SLA fields priority, requested_completion, and escalation_tier — so it drops straight into an existing pipeline without redefining the contract, and models the policy engine’s node-level scope resolution exactly. Run it as-is: the first evaluation denies, the second permits.
from __future__ import annotations
from dataclasses import dataclass, field
from datetime import datetime, timezone
from enum import Enum
from typing import FrozenSet, Optional
class MaintenanceScope(str, Enum):
CORRECTIVE = "corrective"
PREVENTIVE = "preventive"
@dataclass(frozen=True)
class WorkOrderPayload:
work_order_id: str
asset_id: str
requested_role: str
zone: str
scope: MaintenanceScope
priority: int # 1 (highest) .. 5 (lowest)
requested_completion: datetime # SLA due timestamp, timezone-aware UTC
escalation_tier: int = 0 # 0 = none .. 5 = executive
certifications: FrozenSet[str] = field(default_factory=frozenset)
pm_trigger_source: Optional[str] = None
class RoutingDenied(Exception):
"""Raised when the routing scope does not resolve on the target node."""
# Asset path from site root down to the equipment being routed to.
ASSET_PATH = ["SITE-BLDG-01", "BLDG-A", "HVAC-System", "HVAC-CH-04"]
def resolve_scopes_at_node(
asset_id: str,
grants: dict[str, set[str]],
inheritable_actions: set[str],
) -> set[str]:
"""Mirror the CMMS policy engine: walk the path to asset_id and accumulate
only the grants whose action is flagged inheritable along the way."""
resolved: set[str] = set()
for node in ASSET_PATH:
for scope in grants.get(node, set()):
action = scope.rsplit(":", 1)[0] # 'pm:route' from 'pm:route:execute'
if node == asset_id or action in inheritable_actions:
resolved.add(scope)
if node == asset_id:
break
return resolved
def route_pm(
payload: WorkOrderPayload,
role_scopes: set[str],
grants: dict[str, set[str]],
inheritable_actions: set[str],
) -> str:
resolved = resolve_scopes_at_node(payload.asset_id, grants, inheritable_actions)
if "pm:route:execute" not in role_scopes:
raise RoutingDenied("role missing pm:route:execute scope")
if "pm:route:execute" not in resolved:
raise RoutingDenied(
f"pm:route:execute not inherited at {payload.asset_id} "
f"(context=cross_hierarchy_routing)"
)
return f"routed {payload.work_order_id} -> {payload.asset_id}"
pm = WorkOrderPayload(
work_order_id="WO-PM-8842",
asset_id="HVAC-CH-04",
requested_role="integration_pm_dispatcher",
zone="ZONE-A",
scope=MaintenanceScope.PREVENTIVE,
priority=3,
requested_completion=datetime(2026, 6, 30, tzinfo=timezone.utc),
escalation_tier=0,
pm_trigger_source="PM-2026-05A",
)
# --- Reproduce the 403: role has the scope, granted at the site, but pm:route
# is not inheritable, so it never resolves at the child node ---
broken_role = {"workorder:create", "pm:route:execute"}
broken_grants = {"SITE-BLDG-01": {"pm:route:execute"}}
try:
route_pm(pm, broken_role, broken_grants, inheritable_actions=set())
except RoutingDenied as exc:
print("DENIED (expected):", exc)
# --- Apply the fix: role carries the scope AND pm:route is inheritable ---
fixed_role = {"workorder:create", "pm:route:execute", "pm:dispatch"}
fixed_grants = {"SITE-BLDG-01": {"pm:route:execute", "pm:dispatch"}}
print(route_pm(pm, fixed_role, fixed_grants, inheritable_actions={"pm:route", "pm:dispatch"}))Expected output:
DENIED (expected): pm:route:execute not inherited at HVAC-CH-04 (context=cross_hierarchy_routing)
routed WO-PM-8842 -> HVAC-CH-04The first call fails exactly as production does — the grant is stranded at the site root because pm:route is not inheritable. The second call succeeds once the role carries the scope and the action is allowed to propagate to the child node.
Prevention Checklist
Work through these before the next scheduling window to keep cross-site PM routing from regressing.
Frequently Asked Questions
Why does work order creation succeed but routing return 403?
Creation and routing are separate scopes. workorder:create lets the account write a work order record; pm:route:execute lets it assign labor and push the job to a specific asset. A role can hold the first without the second, which is exactly the gap this incident exposes. Treat routing as its own privilege and check it independently.
Why is the scope present at the site but still denied at the asset?
The policy engine evaluates the exact asset_id in the payload, not the PM schedule’s origin. A grant attached to SITE-BLDG-01 only reaches HVAC-CH-04 if pm:route is flagged inheritable and the role is bound to a security group that propagates to child assets. Without inheritance the grant is stranded at the parent and the child node denies.
Should I just enable inheritance for every action to avoid 403s?
No. Inheritance is disabled on pm:route by default specifically to stop privilege escalation across site or tenant boundaries. Enable it only for the routing and dispatch actions on the specific groups that need it, and keep destructive or administrative scopes non-inheritable. Pair the change with the audit trail described in security and access boundaries so every grant is reviewable.
How is this different from a schema or asset-resolution failure?
A 403 with insufficient_permissions is authorization — the payload was well-formed and the asset resolved, but the caller lacked the right. A malformed payload surfaces earlier as a 400 from work order schema standards, and a mis-parented node surfaces as the gate pointing at the wrong boundary. Read the error code and context field first to route your debugging.
Related
Set the runtime authorization model this debugging targets in security and access boundaries, key inheritance on the canonical nodes from asset hierarchy design, validate the payload upstream with JSON schema validation for work order payloads, and align the schedules being routed with PM interval calculation.