Mobile Money Webhook Reliability: Handling Carrier Timeouts and Idempotency Without Double-Charging

"In mobile money, network drops are not exceptions; they are the baseline operational reality. If your webhook handler does not treat every incoming request as potentially duplicated, you are giving away free inventory."

1. The Telco Reality in East Africa

Integrating payments across East Africa means interfacing with mobile network operators: Vodacom (M-Pesa), Airtel Money, Yas / Tigo Pesa, and aggregators like Snippe or Selcom. While the client-side USSD push (STK Push) experience feels instantaneous to the customer, the server-to-server callback infrastructure operates under extreme asynchronous unpredictability.

Carrier SMS gateways and billing servers enforce aggressive HTTP client timeout thresholds—often as low as 1,500ms to 3,000ms. If your webhook endpoint takes 2,400ms to verify an order, write logs, update database rows, and invoke a third-party SMS confirmation, the carrier gateway considers the delivery failed. It immediately schedules an exponential retry.

Here is where disaster strikes: the retry often arrives within 400 milliseconds of the original request. If your application handles requests concurrently across multiple worker threads, both threads will read the order status as PENDING at the exact same millisecond. Both will mark it PAID, both will add funds to the customer's wallet, and your finance team will spend days untangling balance discrepancies.

2. The Core Architecture: Three Invariant Safeguards

To eliminate race conditions and achieve five-nines payment reliability, our production systems enforce three non-negotiable architectural layers:

1. Cryptographic Signature Verification: Validating incoming HMAC-SHA256 signatures before reading the JSON body into application state.
2. Database-Level Unique Idempotency Keys: Relying on the relational database engine, not application memory or Redis, as the ultimate arbiter of uniqueness.
3. Pessimistic Row-Level Locking: Executing SELECT ... FOR UPDATE inside strict ACID transactions.

3. Cryptographic Signature Verification

Never trust an unauthenticated callback. An attacker scanning IP ranges can forge a {"status": "SUCCESS", "amount": 500000} payload and trigger unauthorized ledger credits. Always compute the keyed HMAC hash using constant-time string comparison to prevent timing attacks:

// Verifying HMAC-SHA256 signature in constant time
function verifyWebhookSignature(rawBody, signatureHeader, secretKey) {
  if (!signatureHeader || !secretKey) return false;
  
  const hmac = crypto.createHmac('sha256', secretKey);
  hmac.update(rawBody, 'utf8');
  const expectedSignature = hmac.digest('hex');

  // Constant-time comparison prevents timing side-channel attacks
  return crypto.timingSafeEqual(
    Buffer.from(signatureHeader, 'hex'),
    Buffer.from(expectedSignature, 'hex')
  );
}

4. Database-Level Unique Idempotency Constraint

Application-level checks like if (order.status === 'PAID') return; will always fail under high-concurrency race conditions. The only deterministic defense is a composite unique index on the primary database engine:

-- Relational schema enforcing carrier transaction uniqueness
CREATE TABLE payment_transactions (
    id BIGINT UNSIGNED AUTO_INCREMENT PRIMARY KEY,
    order_id VARCHAR(64) NOT NULL,
    provider ENUM('mpesa', 'tigopesa', 'airtelmoney', 'selcom', 'snippe') NOT NULL,
    carrier_tx_id VARCHAR(128) NOT NULL,
    amount_tzs DECIMAL(14, 2) NOT NULL,
    status ENUM('pending', 'completed', 'reversed', 'failed') NOT NULL DEFAULT 'pending',
    idempotency_hash CHAR(64) NOT NULL,
    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
    updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
    
    -- Deterministic barrier against duplicate concurrent carrier retries:
    UNIQUE KEY uq_carrier_receipt (provider, carrier_tx_id),
    UNIQUE KEY uq_idempotency (idempotency_hash),
    INDEX idx_order_lookup (order_id, status)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4;

5. The Atomic Processing Pattern

When the webhook arrives, process the ledger update using pessimistic row locking. If a duplicate thread attempts to update the same record simultaneously, InnoDB forces it to wait for the first transaction to complete:

// Atomic ledger credit with pessimistic row lock
public function processCarrierCallback(array $payload): bool 
{
    $db = Database::getConnection();
    $db->beginTransaction();

    try {
        // 1. Lock the order row exclusively
        $stmt = $db->prepare("SELECT id, status, total_tzs FROM orders WHERE order_id = :id FOR UPDATE");
        $stmt->execute([':id' => $payload['order_id']]);
        $order = $stmt->fetch(PDO::FETCH_ASSOC);

        if (!$order) {
            $db->rollBack();
            return false;
        }

        // 2. Check if already settled
        if ($order['status'] === 'completed') {
            $db->rollBack();
            return true; // Return true to send HTTP 200 to carrier and stop retries
        }

        // 3. Insert transaction record (fails on duplicate carrier_tx_id)
        $txStmt = $db->prepare("
            INSERT INTO payment_transactions 
                (order_id, provider, carrier_tx_id, amount_tzs, status, idempotency_hash)
            VALUES 
                (:order_id, :provider, :tx_id, :amount, 'completed', :hash)
        ");
        $txStmt->execute([
            ':order_id' => $order['id'],
            ':provider' => $payload['provider'],
            ':tx_id'    => $payload['carrier_reference'],
            ':amount'   => $payload['amount'],
            ':hash'     => hash('sha256', $payload['provider'] . $payload['carrier_reference'])
        ]);

        // 4. Update order status and credit wallet
        $updateStmt = $db->prepare("UPDATE orders SET status = 'completed', paid_at = NOW() WHERE id = :id");
        $updateStmt->execute([':id' => $order['id']]);

        $db->commit();
        return true;
    } catch (\PDOException $e) {
        $db->rollBack();
        // Duplicate key error (MySQL 23000 / 1062) means another thread already settled it
        if ($e->getCode() == 23000) {
            return true; // Safely acknowledge
        }
        throw $e;
    }
}

6. Automated Daily Telco Reconciliation

Even with rigorous webhook idempotency, network interruptions can cause the initial webhook to never arrive (for example, if the telco's callback worker exhausts its retry queue during an unannounced maintenance window). For this reason, a production mobile money architecture must include an automated daily reconciliation cron job.

Every night at 02:00 EAT, the system queries each carrier API for all transactions marked PENDING that are older than 30 minutes, reconciling discrepancies against the carrier's master ledger before releasing merchant settlement payouts.