Monitoring and Alerting on Paystack Webhook Failures

A webhook handler that crashes is easy to detect. It returns a 500, Paystack retries, your error tracking service (Sentry, Bugsnag, Rollbar) fires an alert, and someone investigates. Loud failures get fixed.

Silent failures are different. Your handler receives the webhook, verifies the signature, returns 200, pushes the event to a queue, and the queue worker processes it. Everything looks fine in the logs. But the order update query silently matched zero rows because the payment reference had a typo. Or the idempotency check matched a previous event when it should not have. Or the database write succeeded but a subsequent step (granting access, sending a confirmation) failed and nobody noticed.

The customer paid. Paystack recorded the payment. Your database shows the order as "pending." The customer waits, then contacts support, then requests a chargeback. By the time you investigate, hours or days have passed.

Monitoring closes this gap. You track not just "did the webhook arrive?" but "did the expected outcome happen?" This is the difference between delivery monitoring and outcome monitoring. You need both.

This article is part of the Paystack webhooks engineering guide.

1. Delivery success rate

Count the webhooks that arrive at your endpoint (any event type) and the ones that pass signature verification. If the delivery rate drops, either Paystack cannot reach your server or your signature verification is broken.

// Track at the webhook endpoint level
app.post('/webhooks/paystack', express.raw({ type: 'application/json' }), async (req, res) => {
  metrics.increment('webhook.received');

  const isValid = verifySignature(req.body, req.headers['x-paystack-signature']);

  if (!isValid) {
    metrics.increment('webhook.signature_failed');
    return res.status(400).send('Invalid signature');
  }

  metrics.increment('webhook.signature_valid');
  res.status(200).send('OK');

  // Queue for processing...
});

2. Processing lag

Measure the time between when you receive the webhook and when the business logic completes (order updated, access granted, email sent). This is different from response time. You respond in milliseconds, but the actual processing might take seconds or minutes if your queue is backed up.

// In your queue worker
async function processWebhookEvent(job) {
  const receivedAt = new Date(job.data.receivedAt);
  const processingStart = Date.now();

  try {
    await handleEvent(job.data.eventType, job.data.payload);

    const lagMs = Date.now() - receivedAt.getTime();
    const processingMs = Date.now() - processingStart;

    metrics.histogram('webhook.processing_lag_ms', lagMs, {
      event_type: job.data.eventType,
    });
    metrics.histogram('webhook.processing_duration_ms', processingMs, {
      event_type: job.data.eventType,
    });
  } catch (err) {
    metrics.increment('webhook.processing_failed', {
      event_type: job.data.eventType,
    });
    throw err;
  }
}

3. Queue depth

How many webhook events are sitting in the queue waiting to be processed? A growing queue means your workers cannot keep up. This is often the first sign of trouble.

// Check queue depth every 30 seconds
setInterval(async () => {
  const waiting = await webhookQueue.getWaitingCount();
  const active = await webhookQueue.getActiveCount();
  const delayed = await webhookQueue.getDelayedCount();
  const failed = await webhookQueue.getFailedCount();

  metrics.gauge('webhook.queue.waiting', waiting);
  metrics.gauge('webhook.queue.active', active);
  metrics.gauge('webhook.queue.delayed', delayed);
  metrics.gauge('webhook.queue.failed', failed);
}, 30000);

4. Expected vs. actual outcomes

For every charge.success you receive, a corresponding order should move to "paid" status. For every transfer.success, a payout record should move to "success." Track the gap between events received and outcomes achieved.

Plain text logs (console.log('Received charge.success for ref-001')) are hard to query and aggregate. Use structured logging (JSON) so you can filter, count, and alert on specific fields.

// Use a structured logger (pino, winston, or similar)
const pino = require('pino');
const logger = pino({ level: 'info' });

// When a webhook arrives
function logWebhookReceived(eventType, reference, domain) {
  logger.info({
    msg: 'webhook_received',
    event_type: eventType,
    reference: reference,
    domain: domain,
    timestamp: new Date().toISOString(),
  });
}

// When processing completes
function logWebhookProcessed(eventType, reference, durationMs, outcome) {
  logger.info({
    msg: 'webhook_processed',
    event_type: eventType,
    reference: reference,
    duration_ms: durationMs,
    outcome: outcome, // 'success', 'skipped_duplicate', 'failed'
    timestamp: new Date().toISOString(),
  });
}

// When processing fails
function logWebhookFailed(eventType, reference, error) {
  logger.error({
    msg: 'webhook_processing_failed',
    event_type: eventType,
    reference: reference,
    error: error.message,
    stack: error.stack,
    timestamp: new Date().toISOString(),
  });
}

With structured logs, you can query your logging platform (Datadog, Grafana Loki, CloudWatch, Elastic) with queries like:

• "Show me all webhook_processing_failed events in the last hour."
• "Count webhook_received events grouped by event_type for the last 24 hours."
• "Find all charge.success events with duration_ms > 5000."
• "Show me events where outcome is failed and event_type is charge.success."

If you do not have a logging platform, even writing structured JSON to a file is better than plain text. You can grep for specific fields and pipe through jq for analysis.

In addition to logs, store webhook events in a database table for querying and reconciliation. This is different from the processed_events table used for idempotency. This table records every event you received, regardless of whether it was processed, skipped, or failed.

CREATE TABLE webhook_events (
  id BIGSERIAL PRIMARY KEY,
  event_type VARCHAR(100) NOT NULL,
  reference VARCHAR(255),
  idempotency_key VARCHAR(255),
  payload JSONB NOT NULL,
  received_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
  processed_at TIMESTAMPTZ,
  processing_status VARCHAR(20) NOT NULL DEFAULT 'pending',
  -- pending, processing, completed, failed, skipped
  error_message TEXT,
  processing_duration_ms INTEGER
);

CREATE INDEX idx_webhook_events_status ON webhook_events(processing_status);
CREATE INDEX idx_webhook_events_type ON webhook_events(event_type);
CREATE INDEX idx_webhook_events_received ON webhook_events(received_at);
CREATE INDEX idx_webhook_events_reference ON webhook_events(reference);

Update this table as events flow through your pipeline:

// When webhook arrives (before returning 200)
async function recordWebhookReceived(eventType, reference, payload) {
  const result = await db.query(
    'INSERT INTO webhook_events (event_type, reference, payload, received_at) ' +
    'VALUES ($1, $2, $3, NOW()) RETURNING id',
    [eventType, reference, JSON.stringify(payload)]
  );
  return result.rows[0].id;
}

// When processing starts
async function markProcessing(eventId) {
  await db.query(
    'UPDATE webhook_events SET processing_status = $1 WHERE id = $2',
    ['processing', eventId]
  );
}

// When processing completes
async function markCompleted(eventId, durationMs) {
  await db.query(
    'UPDATE webhook_events SET processing_status = $1, ' +
    'processed_at = NOW(), processing_duration_ms = $2 WHERE id = $3',
    ['completed', durationMs, eventId]
  );
}

// When processing fails
async function markFailed(eventId, errorMessage) {
  await db.query(
    'UPDATE webhook_events SET processing_status = $1, ' +
    'error_message = $2, processed_at = NOW() WHERE id = $3',
    ['failed', errorMessage, eventId]
  );
}

This table gives you full visibility into your webhook pipeline. You can query it for failure rates, processing times, and stuck events. For storage management, archive or delete events older than 90 days.

Not every metric needs an alert. Alert fatigue kills response quality. Focus on alerts that indicate real problems requiring human intervention.

Alert 1: No webhooks received in 30 minutes (during business hours)

If your app normally receives at least a few webhooks per hour during business hours and suddenly goes silent, something is wrong. Either Paystack cannot reach your server, your server is down, or your DNS changed.

-- Query: any webhooks received in the last 30 minutes?
SELECT COUNT(*) as recent_count
FROM webhook_events
WHERE received_at > NOW() - INTERVAL '30 minutes';

Alert 2: Failed events exceed 5% in a rolling hour

A few failures are normal (network blips, transient database errors). A sustained failure rate above 5% means something systemic is wrong.

-- Query: failure rate in the last hour
SELECT
  COUNT(*) FILTER (WHERE processing_status = 'failed') as failed_count,
  COUNT(*) as total_count,
  ROUND(
    100.0 * COUNT(*) FILTER (WHERE processing_status = 'failed') / NULLIF(COUNT(*), 0),
    2
  ) as failure_percentage
FROM webhook_events
WHERE received_at > NOW() - INTERVAL '1 hour';

Alert 3: Processing lag exceeds 60 seconds

If events are sitting in the queue for more than a minute, your workers are overwhelmed or stuck.

-- Query: events received but not processed within 60 seconds
SELECT COUNT(*) as stale_count
FROM webhook_events
WHERE processing_status = 'pending'
AND received_at < NOW() - INTERVAL '60 seconds';

Alert 4: Unprocessed charge.success events

This is the most critical alert. A charge.success event that has not been processed means a customer paid but was not credited.

-- Query: charge.success events not completed within 5 minutes
SELECT reference, received_at, processing_status
FROM webhook_events
WHERE event_type = 'charge.success'
AND processing_status != 'completed'
AND processing_status != 'skipped'
AND received_at < NOW() - INTERVAL '5 minutes'
AND received_at > NOW() - INTERVAL '1 hour'
ORDER BY received_at DESC;

Send this alert to a channel that gets immediate attention (PagerDuty, Slack with a loud notification, SMS). A missed payment is a support ticket, a chargeback risk, and a trust problem.

Alerts catch problems as they happen. Reconciliation catches problems that slipped through. Run a reconciliation query periodically (every hour or every day) that compares your webhook events with your business records.

-- Find charge.success events where the order was not updated to 'paid'
SELECT
  we.reference,
  we.received_at,
  we.processing_status as webhook_status,
  o.status as order_status,
  (we.payload->'data'->>'amount')::int as webhook_amount,
  o.amount_paid
FROM webhook_events we
LEFT JOIN orders o ON o.payment_reference = we.reference
WHERE we.event_type = 'charge.success'
AND we.received_at > NOW() - INTERVAL '24 hours'
AND (
  o.status IS NULL                  -- No matching order at all
  OR o.status != 'paid'             -- Order exists but not marked paid
  OR o.amount_paid IS NULL          -- Order marked paid but amount not set
)
ORDER BY we.received_at DESC;

This query surfaces three types of problems:

1. No matching order: The webhook arrived for a reference that does not exist in your orders table. This might be a test transaction, a reference format mismatch, or a legitimate missed order creation.
2. Order not paid: The order exists but is still "pending" or "failed." The webhook was received but processing did not complete.
3. Amount mismatch: The order is marked "paid" but the amount_paid was not set. Partial processing.

For high-value applications, also reconcile against the Paystack API. Fetch all successful transactions from Paystack for the day and compare with your records:

// Fetch transactions from Paystack API for reconciliation
async function fetchPaystackTransactions(fromDate, toDate) {
  const transactions = [];
  let page = 1;
  let hasMore = true;

  while (hasMore) {
    const response = await fetch(
      'https://api.paystack.co/transaction?' +
      'from=' + fromDate + '&to=' + toDate +
      '&status=success&perPage=100&page=' + page,
      {
        headers: {
          'Authorization': 'Bearer ' + process.env.PAYSTACK_SECRET_KEY,
        },
      }
    );

    const data = await response.json();
    transactions.push(...data.data);

    if (data.data.length < 100) {
      hasMore = false;
    } else {
      page++;
    }
  }

  return transactions;
}

// Compare with your records
async function reconcile() {
  const yesterday = new Date();
  yesterday.setDate(yesterday.getDate() - 1);

  const fromDate = yesterday.toISOString().split('T')[0];
  const toDate = new Date().toISOString().split('T')[0];

  const paystackTxns = await fetchPaystackTransactions(fromDate, toDate);

  for (const txn of paystackTxns) {
    const order = await db.query(
      'SELECT status FROM orders WHERE payment_reference = $1',
      [txn.reference]
    );

    if (order.rows.length === 0 || order.rows[0].status !== 'paid') {
      console.error(
        'RECONCILIATION MISMATCH: Paystack reference ' + txn.reference +
        ' amount ' + txn.amount +
        ' is successful but order is ' +
        (order.rows.length === 0 ? 'missing' : order.rows[0].status)
      );
    }
  }
}

Schedule this with a cron job or a scheduled task. Fix mismatches immediately.

If you use Grafana, Datadog, or a similar tool, build a dashboard with these panels:

Panel 1: Webhooks per minute (time series)

Shows the rate of incoming webhooks. A sudden drop means Paystack cannot reach you. A sudden spike might indicate a replay or a burst of activity.

Panel 2: Processing status breakdown (stacked bar or pie)

Shows the proportion of completed, failed, skipped, and pending events. A growing "failed" slice requires investigation.

Panel 3: Processing lag p95 (time series)

The 95th percentile processing lag. Normal is under 5 seconds. Anything above 30 seconds means your queue is backing up.

Panel 4: Failed events table (latest failures)

A live table showing the most recent failed events with their error messages. This is the first thing you check during an incident.

Panel 5: Unprocessed events count (single stat)

The number of events received more than 5 minutes ago that are still not completed. This should be zero or close to it. If it grows, something is stuck.

If you do not have a dashboarding tool, a simple cron script that runs the SQL queries from the alerting section and sends results to Slack or email gives you 80% of the value at 10% of the effort.

For storing raw payloads for audit purposes, see storing raw webhook payloads. For handling events that permanently fail processing, see building a dead letter queue.

Not every team has Datadog or Grafana. Here is a lightweight monitoring approach using only your database, a cron job, and Slack (or email).

// scripts/webhook-health-check.js
// Run every 15 minutes via cron:
// */15 * * * * node /path/to/scripts/webhook-health-check.js

const { Pool } = require('pg');
const pool = new Pool({ connectionString: process.env.DATABASE_URL });

async function checkHealth() {
  const alerts = [];

  // Check 1: Any failed events in the last 15 minutes?
  const failed = await pool.query(
    'SELECT COUNT(*)::int as count FROM webhook_events ' +
    'WHERE processing_status = $1 AND received_at > NOW() - INTERVAL $2',
    ['failed', '15 minutes']
  );
  if (failed.rows[0].count > 0) {
    alerts.push('FAILED EVENTS: ' + failed.rows[0].count + ' in last 15 min');
  }

  // Check 2: Any stuck events (pending for more than 5 minutes)?
  const stuck = await pool.query(
    'SELECT COUNT(*)::int as count FROM webhook_events ' +
    'WHERE processing_status = $1 AND received_at < NOW() - INTERVAL $2',
    ['pending', '5 minutes']
  );
  if (stuck.rows[0].count > 0) {
    alerts.push('STUCK EVENTS: ' + stuck.rows[0].count + ' pending > 5 min');
  }

  // Check 3: Any unprocessed charge.success?
  const missed = await pool.query(
    'SELECT COUNT(*)::int as count FROM webhook_events ' +
    'WHERE event_type = $1 AND processing_status != $2 ' +
    'AND processing_status != $3 AND received_at < NOW() - INTERVAL $4 ' +
    'AND received_at > NOW() - INTERVAL $5',
    ['charge.success', 'completed', 'skipped', '5 minutes', '1 hour']
  );
  if (missed.rows[0].count > 0) {
    alerts.push(
      'MISSED PAYMENTS: ' + missed.rows[0].count +
      ' charge.success events not processed'
    );
  }

  if (alerts.length > 0) {
    await sendSlackAlert(alerts.join('
'));
  }

  await pool.end();
}

async function sendSlackAlert(message) {
  await fetch(process.env.SLACK_WEBHOOK_URL, {
    method: 'POST',
    headers: { 'Content-Type': 'application/json' },
    body: JSON.stringify({
      text: 'Paystack Webhook Alert:
' + message,
    }),
  });
}

checkHealth().catch(function(err) {
  console.error('Health check failed: ' + err.message);
  process.exit(1);
});

This script runs every 15 minutes, checks three conditions, and sends a Slack message if anything is wrong. It uses no paid services. It runs anywhere you can run a cron job.

For the complete webhook reliability picture, see the Paystack webhooks engineering guide.