Configuring IMAP Polling for Maintenance Email Queues: Resolving Duplicate Work Order Ingestion via Flag Synchronization

Duplicate work order creation is the most common failure surfaced when a maintenance team first wires up email intake configuration against a busy shared mailbox. The same maintenance request lands as two, three, or four tickets; preventive maintenance (PM) routing matrices fragment; and false-positive SLA alerts fire against work that was only ever requested once. The root cause is almost always an IMAP polling race condition between the SEARCH UNSEEN query and \Seen flag synchronization. When a queue exceeds roughly 50 messages per polling interval, naive imaplib loops re-ingest identical payloads before the pipeline commits state. This page isolates the exact race, shows the before/after configuration fix, and gives a runnable script you can drop into a CMMS ingestion service to guarantee idempotency.

Incident Profile: Duplicate Tickets During High-Volume Polling

The symptom is consistent: technicians see two or more identical work orders for one inbound email, each with the same sender, subject, and attachment. The duplicates appear within a single polling window, not across days, which rules out the operator re-sending the request. The defining evidence is an interleaved poller/pipeline trace where the same UID is fetched twice before its flag is written:

[2024-05-12 08:14:01] IMAP_POLL: SELECT INBOX -> OK [READ-WRITE]
[2024-05-12 08:14:01] IMAP_POLL: SEARCH UNSEEN -> 1042 1043 1044
[2024-05-12 08:14:02] PIPELINE: FETCH 1042 (BODY[HEADER.FIELDS (FROM SUBJECT)])
[2024-05-12 08:14:03] PIPELINE: ATTACHMENT_PARSE: maintenance_request_1042.pdf -> QUEUED
[2024-05-12 08:14:04] IMAP_POLL: SEARCH UNSEEN -> 1042 1043 1044 1045
[2024-05-12 08:14:04] PIPELINE: FETCH 1042 (RFC822) -> WORK_ORDER_CREATED (again)
[2024-05-12 08:14:05] IMAP_POLL: STORE 1042 +FLAGS (\Seen) -> OK

When duplicate work orders spike, run this targeted diagnostic before correlating full logs:

  1. Verify UID overlap in polling windows. Grep ingestion logs for identical IMAP UID or Message-ID values processed inside the same interval. Overlapping timestamps confirm a concurrent SEARCH.
  2. Inspect the FLAGS state. Fetch the message flags directly (FETCH <UID> FLAGS). If \Seen is absent despite a created ticket, the flag commit is lagging the fetch.
  3. Measure pipeline commit latency. Time the delta between FETCH RFC822 completion and the database INSERT. Deltas above ~2 seconds open a window where the next poll re-discovers unflagged messages.
  4. Apply immediate mitigation. Temporarily raise the polling interval to 120 seconds, drop to a single poller thread, and enable a Message-ID idempotency check until the atomic fix below is deployed.

Root Cause: The UNSEEN Flag Race Condition

IMAP servers maintain per-connection message state, and a standard polling loop executes SELECT INBOX, SEARCH UNSEEN, FETCH, then STORE +FLAGS (\Seen). If the pipeline parses attachments, runs downstream validation, or maps custom CMMS fields before committing the \Seen flag, a second poll fired during that processing window issues its own SEARCH UNSEEN and gets the same UIDs back. Per RFC 3501 §2.3.2, SEARCH UNSEEN evaluates mailbox state at query execution time, not at fetch time — so a message that has been fetched but not yet flagged is still “unseen” to the next query. Under load, state drift is not a possibility; it is guaranteed.

The deeper issue is that discovery and processing are coupled. The flag is treated as a “done” marker written at the end of processing, when it must instead be a “claimed” marker written the instant a UID is discovered. The longer the gap between claim and flag, the wider the duplication window.

Sequence diagram of the SEARCH UNSEEN / \Seen flag race between two pollers Poller A selects the inbox and runs SEARCH UNSEEN, which returns UIDs 1042, 1043 and 1044. A fetches 1042 and begins a slow attachment parse. Because the \Seen flag for 1042 has not yet been written, poller B runs its own SEARCH UNSEEN during that parse and the mailbox returns 1042, 1043, 1044 and a new 1045. Poller B fetches 1042 and creates a work order for it. Poller A then finishes parsing and also creates a work order for 1042, producing a duplicate. Only afterwards does A write STORE +FLAGS (\Seen) for 1042 — too late to stop B. The shaded band marks the race window in which 1042 is fetched but not yet flagged. Two pollers, one unflagged window — how UID 1042 is ingested twice RACE WINDOW · 1042 fetched, not yet \Seen Poller A IMAP mailbox (INBOX) Poller B SEARCH UNSEEN returns [1042 1043 1044] slow attachment parse FETCH 1042 (RFC822) SEARCH UNSEEN (fires mid-parse) returns [1042 1043 1044 1045] FETCH 1042 (RFC822) WORK_ORDER_CREATED 1042 WORK_ORDER_CREATED 1042 (again) STORE 1042 +FLAGS (\Seen) — too late = duplicate ticket

Resolution: Atomic Flag Synchronization (Before / After)

The fix decouples message discovery from payload processing: the poller marks discovered UIDs \Seen immediately, before any parsing, so a concurrent poll can never re-claim them. The imapclient library gives robust UID handling and clean flag semantics.

Before — flag written after processing (race-prone):

import imapclient

def poll_broken(host, user, password):
    with imapclient.IMAPClient(host, ssl=True) as client:
        client.login(user, password)
        client.select_folder("INBOX", readonly=False)
        for uid in client.search(["UNSEEN"]):
            raw = client.fetch([uid], ["RFC822"])[uid][b"RFC822"]
            create_work_order(raw)            # slow: attachment + field mapping
            client.add_flags([uid], [imapclient.SEEN])  # flag set TOO LATE

After — flag written on discovery, then fetch (idempotent):

import logging
import imapclient
import imapclient.exceptions
from typing import List, Tuple

logger = logging.getLogger("cmms_imap_poller")

def poll_maintenance_queue_atomic(
    host: str,
    user: str,
    password: str,
    ssl: bool = True,
    batch_size: int = 100,
) -> List[Tuple[int, bytes]]:
    """Flag messages the instant they are discovered, then fetch.

    Returns (UID, raw_message_bytes) tuples for downstream CMMS ingestion.
    """
    processed: List[Tuple[int, bytes]] = []
    try:
        with imapclient.IMAPClient(host, ssl=ssl, timeout=30) as client:
            client.login(user, password)
            # readonly=False is mandatory — read-only mode silently drops flag writes.
            client.select_folder("INBOX", readonly=False)

            # Step 1: discover unprocessed messages.
            unseen = client.search(["UNSEEN"])
            if not unseen:
                logger.debug("No unseen messages in maintenance queue.")
                return processed

            # Cap the batch so attachment parsing cannot exhaust memory.
            target = unseen[:batch_size]

            # Step 2: claim the UIDs by flagging BEFORE any processing.
            # A concurrent SEARCH UNSEEN can no longer return these UIDs.
            client.add_flags(target, [imapclient.SEEN])
            logger.info("Atomically claimed %d messages.", len(target))

            # Step 3: fetch payloads only after the claim is committed.
            for uid, data in client.fetch(target, ["RFC822"]).items():
                raw = data.get(b"RFC822")
                if raw:
                    processed.append((uid, raw))
            return processed
    except imapclient.exceptions.IMAPClientError as exc:
        logger.error("IMAP poll failed: %s", exc)
        raise

The single load-bearing change is ordering: add_flags() runs before fetch(). Even if the worker crashes mid-parse, the \Seen flag persists and the message is never re-claimed. Three configuration details make the difference durable:

  • readonly=False on select_folder() is non-negotiable — read-only mode suppresses flag writes and fails the synchronization silently.
  • imapclient.SEEN is the correct constant; never pass the raw string "\\Seen".
  • batch_size capping prevents memory bloat when a single email carries large schematics or multi-page work orders.

Minimal Reproducible Pipeline

This end-to-end script wires the atomic poller to a UID dedup cache and the canonical WorkOrderPayload, including the SLA fields every work order schema on this site carries. It runs against any IMAP mailbox; the dedup cache is an in-memory set here, but swap in Redis or a IMAP_UID -> CMMS_WO_ID table for production.

from dataclasses import dataclass, field
from datetime import datetime, timezone
from email import message_from_bytes
from typing import Optional, Set

import imapclient


@dataclass
class WorkOrderPayload:
    """Canonical work order envelope shared across the ingestion pipeline."""
    source_uid: int                       # IMAP UID — the idempotency anchor
    message_id: str                       # RFC 5322 Message-ID header
    asset_tag: str
    description: str
    priority: str = "medium"              # SLA: low | medium | high | critical
    requested_completion: Optional[datetime] = None  # SLA: due date
    escalation_tier: int = 1              # SLA: 1=standard, 2=supervisor, 3=manager
    received_at: datetime = field(default_factory=lambda: datetime.now(timezone.utc))


SEEN_UIDS: Set[int] = set()  # replace with Redis/Postgres in production


def to_work_order(uid: int, raw: bytes) -> WorkOrderPayload:
    msg = message_from_bytes(raw)
    return WorkOrderPayload(
        source_uid=uid,
        message_id=msg.get("Message-ID", f"<no-id-{uid}>"),
        asset_tag=msg.get("X-Asset-Tag", "UNASSIGNED"),
        description=(msg.get("Subject") or "").strip(),
        priority="high" if "URGENT" in (msg.get("Subject") or "").upper() else "medium",
    )


def ingest_once(host: str, user: str, password: str) -> list[WorkOrderPayload]:
    created: list[WorkOrderPayload] = []
    with imapclient.IMAPClient(host, ssl=True, timeout=30) as client:
        client.login(user, password)
        client.select_folder("INBOX", readonly=False)

        unseen = client.search(["UNSEEN"])
        if not unseen:
            return created

        # Claim before processing — this is what kills the race.
        client.add_flags(unseen, [imapclient.SEEN])

        for uid, data in client.fetch(unseen, ["RFC822"]).items():
            if uid in SEEN_UIDS:           # second safeguard: UID-level dedup
                continue
            raw = data.get(b"RFC822")
            if not raw:
                continue
            wo = to_work_order(uid, raw)
            SEEN_UIDS.add(uid)
            created.append(wo)
    return created


if __name__ == "__main__":
    orders = ingest_once("mail.example.com", "[email protected]", "app-password")
    for o in orders:
        print(o.source_uid, o.priority, o.escalation_tier, o.description)

Run it twice back to back against the same mailbox: the first run prints the new orders, the second prints nothing because every UID is already flagged and cached. That is the verification signal for a correct fix.

CMMS Routing and Preventive Maintenance Edge Cases

Duplicate ingestion does not stop at the ticket table — it propagates into routing and inventory:

  • PM schedule fragmentation. Preventive maintenance tasks split across duplicate tickets produce partial completion states and skew MTBF calculations, which feed back into asset hierarchy design and interval planning.
  • Inventory allocation conflicts. Automated reservation logic deducts stock twice, tripping false low-stock alerts during parts availability checks and delaying procurement.
  • Technician dispatch collisions. Routing engines assign conflicting copies to different crews, creating redundant site visits and avoidable SLA breaches.

To contain these, keep UID-level idempotency at the ingestion boundary rather than relying on downstream deduplication. Maintain a lightweight IMAP_UID -> CMMS_WO_ID cache; before routing, look up the UID, and if it exists, attach the payload to the existing work order as a comment instead of spawning a new ticket. This preserves the audit trail while eliminating duplicate dispatches. Where the same logic must absorb bursty volume, hand the flagged batch to async batch processing so parsing never blocks the poller, and route the parsed subject line through NLP intent classification to set priority and escalation tier consistently.

Validation and Monitoring

Deploy the atomic poller alongside structured telemetry:

  1. Flag commit verification. Log client.get_flags(uids) after fetch to confirm \Seen status matches the claimed list.
  2. Duplicate-rate tracking. Emit a duplicate_uid_detected metric and alert if it exceeds 0.1% of polled volume.
  3. Connection-pool health. Watch IMAP CAPABILITY responses and IDLE timeouts; high-volume queues benefit from pooling to avoid NO [LIMIT] rejections.
  4. Fallback idempotency. Hash Message-ID plus Date and store the hash in a 7-day TTL cache; reject matches before insertion.

Prevention Checklist

Frequently Asked Questions

Why not just delete messages after processing instead of flagging them?

Deletion (STORE +FLAGS (\Deleted) followed by EXPUNGE) removes the audit trail and is irreversible if the downstream insert fails. Flagging \Seen on discovery gives the same race protection while keeping the original email recoverable for reprocessing or compliance review.

Does switching to IMAP IDLE eliminate the race instead?

IDLE reduces polling frequency but does not remove the race on its own. You still issue SEARCH UNSEEN on each wake, and a slow parse between discovery and flag write reopens the same window. Claim-on-discovery flagging is required regardless of whether you poll on an interval or use IDLE.

What if two workers must share one mailbox?

Flag-on-discovery makes the add_flags call the claim point, but two workers can still call SEARCH UNSEEN simultaneously before either flags. Serialize the search-and-claim step with a short-lived distributed lock (for example a Redis SET NX key per mailbox), then let parsing run in parallel after the claim.

Continue with the parent email intake configuration guide, then compare the broker-side equivalent of this race in implementing Celery for async work order batching and the field-extraction stage in training spaCy models for maintenance intent routing.

Part of: Work Order Ingestion & Parsing Pipelines