Fixing IMAP IDLE Disconnects in Long-Running Work Order Pollers

A poller built on the IMAP IDLE command is supposed to be the quiet, efficient alternative to interval polling: open one connection, tell the server “push me anything new,” and block until it does. In production it fails in a way that is worse than the noisy interval-polling problems it was meant to fix — it stops receiving anything at all, without an exception, a log line, or a process crash. New maintenance requests land in the mailbox, the poller sits there believing it is still listening, and nobody notices until the backlog is large enough to trip an unrelated alert. This page isolates the exact mechanism (a server-side inactivity timeout the client never re-negotiates), and gives a runnable fix that refreshes the IDLE command inside the timeout window, detects a socket that has already gone dead, and reconnects from the last processed UID rather than the mailbox’s unreliable \Seen state.

Incident Profile

The on-call engineer for a regional facilities team was paged not by the ingestion pipeline but by a site manager asking why a compressor work order submitted that morning still had no technician assigned seven hours later. The poller process was running, its supervisor showed no restarts, and CPU/memory graphs were flat — every signal said “healthy.” The only evidence of the actual failure was in the poller’s own log, which simply stopped after issuing its one and only IDLE:

2026-07-14 09:02:11 INFO  [imap_poller] Connected to mail.example.com:993 (TLS)
2026-07-14 09:02:11 INFO  [imap_poller] SELECT INBOX -> OK [READ-WRITE] 1 EXISTS
2026-07-14 09:02:11 INFO  [imap_poller] IDLE issued, waiting for server push
2026-07-14 09:02:11 DEBUG [imap_poller] + idling
2026-07-14 14:47:33 WARNING [ops_oncall] 41 unprocessed maintenance emails found in INBOX (oldest: 2026-07-14 09:31:02)
2026-07-14 14:47:34 INFO  [imap_poller] Manual restart triggered by on-call
2026-07-14 14:47:35 INFO  [imap_poller] Connected to mail.example.com:993 (TLS)
2026-07-14 14:47:35 INFO  [imap_poller] SELECT INBOX -> OK [READ-WRITE] 42 EXISTS
2026-07-14 14:47:36 INFO  [imap_poller] IDLE issued, waiting for server push

Between 09:02:11 and 14:47:33 the process wrote nothing at all — not a heartbeat, not a warning, not a stack trace. The gap is the incident. The symptom that eventually surfaced it was a manual audit of the mailbox after a site manager complained, which found 41 unread maintenance emails, the oldest one 41 minutes into the poller’s supposed “connected and listening” state. The connection had gone quiet within the first hour of the run; the discovery happened almost six hours later, purely because a human went looking.

Root Cause Analysis

Nothing about the mailbox, the network path, or the CMMS ingestion logic was broken. The IDLE command itself was used correctly for exactly one call, and that is the defect: RFC 2177 defines IDLE as a single, time-bounded grant, not an indefinite subscription. Most IMAP servers enforce an inactivity autologout — commonly implemented around the 29-minute mark, just under the 30-minute minimum RFC 3501 requires servers to honor — and expect a well-behaved client to terminate the current IDLE with DONE and immediately issue a new one before that window closes. A client that issues IDLE once and then blocks on the response forever is relying on the server to hold the connection open indefinitely, which no compliant server promises to do.

Three gaps compounded into the six-hour blind spot:

  1. No refresh timer. The poller called the equivalent of “wait for a push, no timeout” and never re-armed IDLE. Once the server’s autologout fired, there was no second command to renegotiate the session.
  2. No liveness detection. When the server ends the session at its inactivity limit, it does not always send a clean BYE or close the TCP socket with a FIN — a NAT gateway or firewall sitting between the poller and the mail server can silently age out the connection’s state table entry first, so the client’s socket looks perfectly open with no data ever arriving on it again. A blocking call with no read timeout has no way to distinguish “quiet mailbox” from “dead peer.”
  3. No reconnect-and-resume path. Even if the disconnect had been detected, the poller had no logic to reopen the session and pick up from where it left off — it would either crash-loop from the top of the mailbox (reprocessing everything) or, as happened here, simply never come back on its own.

This is a different failure mode from the discovery-side race condition covered in configuring IMAP polling for maintenance email queues — that page’s SEARCH UNSEEN race assumes the connection itself stays alive between polls. Here the connection dies invisibly, so there is no poll to race against; every candidate message the work order ingestion and parsing pipelines rely on simply accumulates unseen in the mailbox until someone restarts the process.

Connection lifecycle: silent timeout versus refreshed IDLE with reconnect A timeline from 0 to 60 elapsed minutes with a dashed marker at the roughly 29 minute server inactivity timeout. In the broken path, IDLE is issued once at minute 0 and the connection silently dies at the 29 minute mark, opening a shaded gap with no data and no error that lasts until a backlog is discovered hours later. In the fixed path, IDLE is issued at minute 0 and refreshed with DONE plus a new IDLE at minute 25, safely inside the timeout window. A dead-peer drop is detected at minute 33 through a NOOP liveness probe, the client reconnects with backoff and resumes from the last seen UID at minute 42, and normal refresh cycles continue at minute 50. One IDLE, no refresh, versus refreshed IDLE with reconnect 0 min 15 min 45 min 60 min elapsed minutes since connect ~29 min server inactivity timeout BROKEN — single IDLE, no refresh IDLE issued SILENT GAP — no data, no error socket looks open; peer is already gone Backlog discovered (hours later, by manual audit) FIXED — IDLE refreshed <29 min, reconnect on drop IDLE issued refresh: DONE + re-IDLE NOOP fails: dead peer detected reconnect + backoff: resume from last_seen_uid refresh: DONE + re-IDLE cycle repeats indefinitely

Resolution

The fix has three independent parts, and all three matter: refresh IDLE before the server’s own inactivity clock expires, probe for liveness so a silently dead socket cannot masquerade as a quiet mailbox, and reconnect with backoff from a tracked UID rather than trusting \Seen state that may itself be stale after a gap.

Before — one IDLE, no timeout, no liveness check:

import imapclient


def poll_idle_broken(host: str, user: str, password: str):
    client = imapclient.IMAPClient(host, ssl=True)
    client.login(user, password)
    client.select_folder("INBOX", readonly=False)

    client.idle()
    # No timeout -> this blocks until the server pushes something.
    # If the server already dropped the session ~29 minutes ago,
    # this call never returns and never raises an exception.
    responses = client.idle_check(timeout=None)
    client.idle_done()
    return responses

After — bounded refresh, liveness probe, reconnect with backoff:

import logging
import socket
import time
from typing import Optional

import imapclient
import imapclient.exceptions

logger = logging.getLogger("cmms_idle_poller")

IDLE_REFRESH_SECONDS = 24 * 60   # refresh well inside the ~29 min server timeout
SOCKET_TIMEOUT_SECONDS = 30      # bound every blocking socket call
MAX_BACKOFF_SECONDS = 60


def connect(host: str, user: str, password: str) -> imapclient.IMAPClient:
    client = imapclient.IMAPClient(host, ssl=True, timeout=SOCKET_TIMEOUT_SECONDS)
    client.login(user, password)
    client.select_folder("INBOX", readonly=False)
    return client


def idle_refresh_cycle(client: imapclient.IMAPClient) -> bool:
    """Run one bounded IDLE window. Returns True if the server pushed
    an update, False if the refresh window simply elapsed."""
    client.idle()
    try:
        # Bounded wait: returns [] on timeout instead of blocking forever,
        # so the loop always regains control before the server's own clock.
        responses = client.idle_check(timeout=IDLE_REFRESH_SECONDS)
    finally:
        client.idle_done()

    # Liveness probe: a drop that never sends FIN/RST (a NAT or firewall
    # aging out the connection) leaves idle_check() reporting a clean
    # timeout even though the peer is gone. NOOP forces a round trip and
    # raises on a genuinely dead socket instead of staying silent.
    client.noop()
    return bool(responses)

The single load-bearing change is that idle_check() now always returns control to the loop — either because the server pushed something, or because the bounded timeout elapsed — and noop() converts a dead socket from silent nothing into a raised exception the caller can catch. Three configuration details make this durable in production: IDLE_REFRESH_SECONDS must stay comfortably under the server’s own timeout (24 minutes gives a 5-minute margin against a stricter server); the timeout= on the IMAPClient constructor bounds noop(), search(), and fetch() too, so a wedged connection cannot hang the process outside of IDLE; and idle_done() runs in a finally block so a mid-window exception never leaves the session stuck in IDLE state on the next call.

Minimal Reproducible Pipeline

This script runs end-to-end against any IMAP mailbox. It combines the refresh-and-reconnect loop above with UID tracking that survives a process restart, and maps each new message to the canonical WorkOrderPayload with the site-wide SLA fields. Point it at a maintenance mailbox and let it run; kill the network to the mail server mid-run and it reconnects on its own once the path is restored.

import json
import logging
import os
import socket
import time
from dataclasses import dataclass, field
from datetime import datetime, timezone
from email import message_from_bytes
from enum import Enum
from typing import Dict, Iterator, List, Optional, Tuple

import imapclient
import imapclient.exceptions

logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s [%(name)s] %(message)s")
logger = logging.getLogger("cmms_idle_poller")

IDLE_REFRESH_SECONDS = 24 * 60
SOCKET_TIMEOUT_SECONDS = 30
MAX_BACKOFF_SECONDS = 60
STATE_PATH = "imap_poller_state.json"


class Priority(str, Enum):
    CRITICAL = "critical"
    HIGH = "high"
    STANDARD = "standard"
    PLANNED = "planned"


@dataclass
class WorkOrderPayload:
    """Canonical CMMS work order — SLA fields are mandatory site-wide."""
    work_order_id: str
    asset_id: str
    part_skus: List[str]
    required_quantities: Dict[str, int]
    location_id: str
    priority: Priority = Priority.STANDARD
    requested_completion: Optional[datetime] = None
    escalation_tier: int = 0
    status: str = "open"
    created_at: datetime = field(default_factory=lambda: datetime.now(timezone.utc))


def load_last_seen_uid() -> int:
    if os.path.exists(STATE_PATH):
        with open(STATE_PATH) as fh:
            return json.load(fh).get("last_seen_uid", 0)
    return 0


def save_last_seen_uid(uid: int) -> None:
    with open(STATE_PATH, "w") as fh:
        json.dump({"last_seen_uid": uid}, fh)


def to_work_order(uid: int, raw: bytes) -> WorkOrderPayload:
    msg = message_from_bytes(raw)
    subject = (msg.get("Subject") or "").strip()
    return WorkOrderPayload(
        work_order_id=f"WO-IMAP-{uid}",
        asset_id=msg.get("X-Asset-Tag", "UNASSIGNED"),
        part_skus=[],
        required_quantities={},
        location_id=msg.get("X-Location-Id", "UNKNOWN"),
        priority=Priority.HIGH if "URGENT" in subject.upper() else Priority.STANDARD,
        escalation_tier=1 if "URGENT" in subject.upper() else 0,
    )


def connect(host: str, user: str, password: str) -> imapclient.IMAPClient:
    client = imapclient.IMAPClient(host, ssl=True, timeout=SOCKET_TIMEOUT_SECONDS)
    client.login(user, password)
    client.select_folder("INBOX", readonly=False)
    return client


def fetch_new_since(client: imapclient.IMAPClient, last_seen_uid: int) -> List[Tuple[int, bytes]]:
    uids = client.search([f"UID {last_seen_uid + 1}:*"]) if last_seen_uid else client.search(["UNSEEN"])
    uids = [u for u in uids if u > last_seen_uid]
    if not uids:
        return []
    fetched = client.fetch(uids, ["RFC822"])
    return [(uid, data[b"RFC822"]) for uid, data in fetched.items() if b"RFC822" in data]


def idle_poll_forever(host: str, user: str, password: str) -> Iterator[WorkOrderPayload]:
    last_seen_uid = load_last_seen_uid()
    backoff = 1.0
    client: Optional[imapclient.IMAPClient] = None

    while True:
        try:
            if client is None:
                client = connect(host, user, password)
                logger.info("Connected; resuming from UID %d", last_seen_uid)
                backoff = 1.0  # reset only after a clean (re)connect

            client.idle()
            try:
                responses = client.idle_check(timeout=IDLE_REFRESH_SECONDS)
            finally:
                client.idle_done()

            client.noop()  # liveness probe: raises if the peer is already gone

            if responses:
                for uid, raw in fetch_new_since(client, last_seen_uid):
                    wo = to_work_order(uid, raw)
                    last_seen_uid = max(last_seen_uid, uid)
                    save_last_seen_uid(last_seen_uid)
                    yield wo
            else:
                logger.debug("IDLE refresh (no push within %ss)", IDLE_REFRESH_SECONDS)

        except (imapclient.exceptions.IMAPClientError, socket.error, OSError, EOFError) as exc:
            logger.warning("IDLE connection lost (%s); reconnecting in %.1fs", exc, backoff)
            try:
                if client is not None:
                    client.shutdown()
            except Exception:
                pass
            client = None
            time.sleep(backoff)
            backoff = min(backoff * 2, MAX_BACKOFF_SECONDS)


if __name__ == "__main__":
    for work_order in idle_poll_forever("mail.example.com", "[email protected]", "app-password"):
        logger.info(
            "New work order %s priority=%s escalation_tier=%d",
            work_order.work_order_id, work_order.priority, work_order.escalation_tier,
        )

fetch_new_since() deliberately prefers a UID range query over SEARCH UNSEEN once a last_seen_uid exists — after a reconnect, \Seen flags may be inconsistent with what was actually delivered downstream, but the UID watermark is durable local state the poller controls. Messages this loop discovers are the same shape the async batch processing queue expects on the other end, and the payload fields line up with the work order schema standards every downstream consumer validates against.

Prevention Checklist

FAQ

Why not just lower the refresh interval to something extreme, like every minute?

It works, but at the cost of a full DONE/IDLE round trip every minute for no benefit — most servers hold the connection open for the better part of 30 minutes, and hammering the session that often adds load and log noise without closing any additional gap. Staying just under the real timeout, with a safety margin of a few minutes, gets the same guarantee at a fraction of the traffic.

Doesn’t imaplib’s built-in socket timeout already catch this?

Only if you set one, and only for calls that are actually reading from the socket. The default imaplib/imapclient connections have no timeout unless you pass one explicitly, and even with a timeout set, a bounded idle_check() call returning an empty list on schedule looks identical to a genuinely quiet mailbox — it does not by itself prove the peer is still there. That is why the liveness probe after each window matters as a separate step, not a substitute for the timeout.

We already fixed the SEARCH UNSEEN duplicate-ticket race — is this the same bug?

No, they sit on opposite sides of the same pipeline. The duplicate-ticket race in configuring IMAP polling for maintenance email queues happens while the connection is alive and polling normally; this bug happens because the connection is dead and nothing is polling at all. Both fixes should run together — atomic flagging on discovery, plus refresh-and-reconnect on the IDLE session that discovers messages in the first place.

How do I reproduce the 29-minute drop without waiting 29 minutes?

Point the poller at a mailbox behind a firewall or proxy you control and lower its idle connection-tracking timeout to something short, like 60 seconds, or simply kill the TCP connection at the network layer (a firewall rule, or tc/iptables on a test host) a few minutes into the IDLE window. The noop() liveness probe should raise on the very next refresh cycle, and the log should show a reconnect with backoff rather than silence.

This fix sits inside email intake configuration alongside the discovery-side race handled in configuring IMAP polling for maintenance email queues; newly discovered messages flow onward into async batch processing, and every payload this poller emits should validate against the work order schema standards shared across work order ingestion and parsing pipelines.

Part of: Email Intake Configuration.