Designing CRM and CTI integrations for retries, duplicates and outages

Define record and event ownership first

A CRM and telephony integration can fail without a dramatic outage: the same interaction is logged twice, a delayed update reopens a completed activity, or an agent is told that notes were saved when their write remains uncertain. Begin with an ownership contract rather than a retry library.

• Interaction authority: which platform defines the interaction identifier and lifecycle? How are transfers or related contacts represented?
• CRM authority: which system owns the customer, case relationship, human notes and disposition? Which fields may the integration update?
• Operation identity: what makes a particular create, update or association the same business operation when delivered again?
• Recovery authority: who may reconcile, correct or replay the operation, under which current access and retention rules?

Keep a source event identifier, a transport receipt identifier, an interaction identifier and a CRM record identifier distinct. A redelivery may have a new transport receipt; different events may refer to the same interaction. Scope keys to the authenticated tenant and source instance so that two customers' identical-looking identifiers do not collide. This is key design, not a substitute for authorisation.

Specify field-level ownership and conflict rules. A telephony status update should not overwrite an agent's edited notes merely because its payload arrived later. Missing fields need explicit semantics: absence may mean no change rather than erase. Treat a snapshot and a delta differently; a later snapshot may supersede an older state, but a skipped delta may contain information you still need.

Separate the source contract from downstream delivery

AWS's current Connect Customer contact-event documentation says events are not ordered and are emitted on a best-effort basis. It also distinguishes DISCONNECTED from COMPLETED: completion can include after-contact work. Do not assume arrival order is lifecycle order, that a disconnection means all agent work is complete, or that every expected event must arrive. The current documentation uses Connect Customer branding for the contact-centre product. [1]

EventBridge's delivery-level reference makes an important distinction: best-effort service-to-bus delivery can miss an event, whereas an event that reaches the bus is then handled under its target retry policy. Target retries cannot reconstruct an event that never reached the bus. Establish an authorised reconciliation source and coverage checks if the workflow needs to detect such gaps. [2]

For retriable target-delivery errors, EventBridge documents a default of 24 hours and up to 185 attempts with exponential backoff and jitter, configurable at the target. After exhausted delivery attempts, an event is dropped unless the appropriate dead-letter handling captures it. These are that service's documented target-delivery settings—not a recommendation for an agent's waiting time, a promise of CRM completion, or the retry policy for every downstream worker. Inspect the actual configuration and each acknowledgement boundary. [3]

For a CRM API example, Dataverse documents Web API service-protection responses as HTTP 429 and provides a Retry-After duration. Its limits consider request count, combined execution time and concurrency, evaluated per authenticated user and enforced per web server. Published defaults can vary or change; batching and parallel requests are not universal ways around them. Check applicable entitlements as well as service protection, and use actual response guidance rather than treating a published number as guaranteed throughput. [4]

Make the business operation idempotent—not just the message handler

Idempotency means repeating the same operation does not produce an additional unintended effect within the scope you designed. Record the stable operation identity, permitted payload or fingerprint, processing state and confirmed target outcome. The same identity with a different payload is a conflict to investigate, not permission to replace whichever value arrived first.

A production consumer needs durable coordination: claim work safely across workers, track pending versus confirmed outcomes, and recover abandoned claims. An in-memory seen set disappears on restart and cannot arbitrate concurrent processes. Marking a message complete before its business effect is confirmed risks losing work; marking it only afterwards leaves a crash window where the effect may have happened but the acknowledgement did not.

Address that window at the target or reconciliation boundary. Use a supported idempotency mechanism, appropriately unique external key or conditional operation where available, and verify what it actually protects. Dataverse's Upsert documentation describes finding a row through its key and creating or updating it, with different standard and elastic-table behaviour. It does not make every plug-in, notification or other downstream effect execute exactly once. Review those side effects separately. [5]

A timeout after a write is an unknown outcome, not proof of failure. Where an authorised read can confirm the specific operation marker and intended result, reconcile before issuing another write. A matching-looking status alone may be insufficient if another operation could have produced it. If the outcome cannot be established safely, hold it for controlled recovery instead of retrying a non-idempotent side effect blindly.

Handle late events without inventing a universal ordering rule

Where the source provides a documented comparable revision or sequence, record it at the correct scope and reject obsolete updates without discarding unrelated required effects. Timestamps are useful evidence, but equal timestamps, clock differences and late observations make them a poor substitute for a guaranteed total order. Event IDs and receipt IDs are not automatically sequence numbers.

If the source has no authoritative ordering token, choose an explicit alternative: retrieve a current authoritative state, preserve separate lifecycle facts, or flag a conflict for reconciliation. Do not assume that a hard-coded ranking such as connected before closed works for every transfer, resume or correction. The numbered revisions in the example below are invented fixture inputs, not fields promised by Amazon Connect events.

Protect concurrent CRM edits using supported conditional writes where appropriate. Dataverse documents optimistic concurrency through ETags and warns that ETag values are opaque: equality can be meaningful, numerical magnitude is not a source revision. A failed precondition calls for re-reading and applying ownership/conflict rules, not simply removing the condition. Confirm the target table's supported concurrency behaviour. [6]

Bound retries by attempts, elapsed time and safe effect

Classify the provider's error code, operation and commit uncertainty before retrying. A transient unavailable response may be retryable; malformed input, denied permission or a conflicting operation needs correction or review. Some failures have operation-specific semantics, so a generic retry-every-error rule is unsafe.

Set a maximum attempt count, total operation age, per-request deadline and concurrency limit. Include queue waiting, request execution and backoff in the real time budget. Use capped exponential backoff with jitter to spread retries, and honour a valid provider Retry-After as a minimum wait where applicable. If that wait exceeds the operation's remaining budget, defer or escalate: do not cap it downward and retry too early. [4]

Coordinate retry layers. A browser, SDK, worker and delivery service can each retry, multiplying load while the CRM is already struggling. Assign ownership of each retry boundary, preserve the same business operation identity, limit in-flight work and use a pause/circuit mechanism where appropriate. Releasing a backlog requires controlled concurrency, not every failed item starting together.

Handle credentials separately. If the documented grant supports refreshing an expired token, permit a bounded, coordinated refresh and retry with current authorisation. Repeated failure, revoked credentials or changed permissions should stop that path and alert the responsible operator. Do not retry indefinitely, rotate through privileged identities, expose tokens in diagnostics or treat permission denial as a reason to bypass policy.

An original failure-mode matrix

Separate the agent's fallback from background recovery

An agent may need a decision in the current interaction while background delivery continues on a different timescale. Define whether call control, customer lookup and CRM writeback can degrade independently. Do not hold an agent indefinitely because a remote write might eventually succeed.

Use precise states such as pending, confirmed, unavailable, unknown outcome and needs review. Only say saved when the workflow's defined persistence boundary has been confirmed, and explain which boundary that is. If the integration cannot retain an entry safely, do not imply that it has a recoverable local copy. Any temporary handling of notes or customer data needs a separately approved access, storage and retention design.

Manual continuation must not create a second uncontrolled writer. Define how an agent's authorised manual action is correlated with a pending automated operation, how later recovery detects it, and which owner resolves a conflict. Recovered data must not navigate an old customer record into a new interaction's tab.

Replay a synthetic event sequence offline

The following self-contained JavaScript runs with Node.js: save the block as reliability-example.mjs and run node reliability-example.mjs. It makes no network requests, reads no credentials and uses only invented identifiers. It models a single serial consumer of complete status snapshots, with a trustworthy revision assumed solely for the fixture. It does not implement an Amazon Connect payload or a Dataverse adapter.

The three-attempt, 900-millisecond logical budget and mock Retry-After millisecond values are arbitrary teaching inputs—not provider limits, a recommended policy or wire-format headers. Waits advance a logical counter instead of sleeping; request/queue durations are not simulated. Fixed jitter fractions make the trace repeatable. A production design must use validated response parsing, real elapsed-time accounting, suitable jitter and durable, concurrent coordination.

const event = (id, rev, state, plan = [200], tenant = 'SYN-A', contact = 'SYN-C1') =>
  ({ id, rev, state, plan, tenant, contact, source: 'SYN-SOURCE' });
const input = [
  event('E1', 1, 'open'),
  event('E1', 1, 'open'),
  event('E3', 3, 'closed', [429, 503, 200]),
  event('E2', 2, 'connected'),
  event('E3', 3, 'open'),
  event('E3b', 3, 'closed'),
  event('E1', 1, 'open', [200], 'SYN-B'),
  event('E4', 4, 'closed', ['committed-timeout']),
  event('E5', 5, 'closed', [503, 503, 503]),
  event('E6', 1, 'open', [403], 'SYN-A', 'SYN-C2'),
  event('E5', 5, 'closed'),
  event('E7', 1, 'open', ['long-throttle'], 'SYN-A', 'SYN-C3')
];
const records = new Map(), ledger = new Map(), trace = [];
let targetWrites = 0;
for (const e of input) {
  const recordKey = JSON.stringify([e.tenant, e.source, e.contact]);
  const eventKey = JSON.stringify([e.tenant, e.source, e.id]);
  const digest = JSON.stringify([recordKey, e.rev, e.state]);
  const operation = JSON.stringify([recordKey, e.rev]);
  const previous = ledger.get(eventKey);
  const row = { id: e.id, tenant: e.tenant, result: '', attempts: 0, waits: [] };
  trace.push(row);
  if (previous) {
    row.result = previous.digest !== digest ? 'conflict'
      : ['applied', 'reconciled', 'stale', 'no-op'].includes(previous.result) ? 'duplicate' : 'held';
    continue;
  }
  const current = records.get(recordKey);
  if (current && e.rev <= current.rev) {
    row.result = e.rev < current.rev ? 'stale' : e.state === current.state ? 'no-op' : 'conflict';
    ledger.set(eventKey, { digest, result: row.result });
    continue;
  }
  row.result = 'needs-review';
  ledger.set(eventKey, { digest, result: 'pending' });
  let elapsed = 0;
  for (let attempt = 0; attempt < 3; attempt++) {
    row.attempts++;
    const response = e.plan[attempt] ?? 503;
    if (response === 200 || response === 'committed-timeout') {
      records.set(recordKey, { tenant: e.tenant, contact: e.contact, rev: e.rev, state: e.state, operation });
      targetWrites++;
      row.result = 'applied';
      if (response === 'committed-timeout') {
        // Mock authorised readback confirms this exact operation; no second write.
        row.result = records.get(recordKey).operation === operation ? 'reconciled' : 'unknown';
      }
      break;
    }
    if (![429, 503, 'long-throttle'].includes(response)) { row.result = 'quarantined'; break; }
    if (attempt === 2) break;
    const hint = response === 429 ? 400 : response === 'long-throttle' ? 1000 : 0;
    const jitter = Math.floor(Math.min(800, 100 * 2 ** attempt) * [0.25, 0.75][attempt]);
    const wait = Math.max(hint, jitter);
    if (elapsed + wait >= 900) break; // do not shorten the provider hint
    elapsed += wait;
    row.waits.push(wait);
  }
  ledger.set(eventKey, { digest, result: row.result });
}
console.log(JSON.stringify({ trace, records: [...records.values()], targetWrites }, null, 2));

The model ends with separate Tenant A and Tenant B records; Tenant A's first contact is at revision 4, not regressed by revision 2 or advanced by the failed revision 5. Its four simulated target writes are a fixture count, not a customer performance metric. The mock readback always succeeds in the committed-timeout case; if a real read is unavailable or inconclusive, preserve an unknown state for controlled recovery. Restarting this program loses its maps. It proves neither durable deduplication nor crash/concurrency safety, and it does not simulate authentication, missing-source reconciliation, CRM plug-ins or external side effects.

Give failed work a recovery owner—not just a queue

A dead-letter queue or failed-work store is a place to retain eligible unresolved work, not a repair mechanism by itself. Name the owner, access controls, alert route, inspection procedure and retention boundary. Distinguish delivery failure, application-processing failure and an unknown business outcome; they may require different evidence.

1. Triage the failure class and establish whether any target effect is already confirmed, absent or unknown.
2. Check the current authoritative record, permissions, intended deletion/retention state and any human correction before replay.
3. Correct the cause under an approved change process; preserve the stable operation identity and safe provenance.
4. Replay only eligible operations at controlled concurrency with the relevant idempotency/conflict protections intact.
5. Verify the business result and record closure; do not mark a queue item resolved solely because it was resent.

Do not replay a stale payload over newer agent work or recreate a deliberately deleted record without authority. Dataverse documents using If-Match: * to prevent creation during an update-style upsert, alongside separate ETag-based concurrency controls. Select the correct operation for the owner's intent rather than removing conditions to make a retry pass. [6]

Observe safe states and test the recovery boundaries

Useful operational signals include pending-work age, retry exhaustion, throttling, reconciliation gaps, denied access and unresolved outcomes. Assign an owner and response expectation to each meaningful alert. A large queue is not necessarily failed work, and a successful HTTP delivery counter is not a confirmed CRM business outcome.

Approve diagnostic fields and retention with the responsible data/security owners. Tenant, interaction and record identifiers can be sensitive too. Avoid raw transcripts, caller numbers, credentials and full CRM payloads in general logs; use access-controlled correlation and minimal error classifications. This guide adds no monitoring, logging destination or website tracking.

• Duplicate and conflict: repeat identical input and reuse an identity with a changed payload; verify the intended effect and conflict handling independently.
• Crash/concurrency: exercise a crash after the target commit but before acknowledgement, abandoned work claims and simultaneous workers in an approved harness—not just the serial example.
• Ordering/gaps: deliver obsolete state, skip an input and present conflicting snapshots; verify the actual source ordering contract and reconciliation coverage.
• Throttling/outage: include Retry-After longer than the budget, SDK retries, shared credentials, queue age and controlled backlog release.
• Access/credentials: test bounded supported refresh, revoked permission, tenant boundaries and a permission change before a recovery action.
• Agent experience: confirm accurate pending/unknown states, accessible manual continuation and no late update to another interaction's workspace.
• Recovery: demonstrate owner-approved inspection, correction, replay and closure without resurrecting deleted data or repeating external side effects.

Review this evidence with an integration/reliability specialist before choosing a production design. The goal is explicit, testable behaviour at each boundary—not a claim that retries make every distributed business operation happen exactly once.

Resources

• Software development and integration capabilities →
• DevOps and delivery capabilities →
• Discuss integration recovery requirements →