Payment Reconciliation Failure Modes and How to Test for Them
The six reconciliation failure modes are: (1) missed webhooks where no fallback verify call exists, (2) duplicate charge.success events processed twice, (3) amount mismatch between what you charged and what Paystack reports, (4) order status desync when DB writes fail after webhook receipt, (5) timezone drift causing next-billing-date comparisons to be off by a day, and (6) reference collisions when two orders share a reference. Test each with dedicated unit tests and a reconciliation script that runs nightly.
The Six Reconciliation Failure Modes
Each of these failures has a distinct signature. Knowing the pattern tells you how to test for it.
1. Missed Webhooks
Paystack delivers webhooks on a best-effort basis. Network blips, server restarts, and deployments can all cause missed deliveries. An order stays in pending forever.
Fix: A nightly job that calls GET /transaction/verify/:reference for every order older than 15 minutes in pending status.
2. Duplicate charge.success Events
Paystack retries webhooks if your endpoint is slow to respond or returns a non-200. You process the same event twice and fulfill an order twice.
Fix: Check a webhook_events table before processing. If event.data.id is already there, skip and return 200.
3. Amount Mismatch
The user modified the DOM price, a currency conversion rounded differently, or an old order had a different price. The charged amount differs from what you expected.
Fix: Compare txData.amount / 100 against your stored order.expected_amount. Reject if the difference exceeds 0.01.
4. Status Desync After DB Failure
Webhook fires, you return 200, but the database write throws an exception. Order is marked paid by Paystack, but your DB still shows pending.
Fix: Use a database transaction or outbox pattern. Log the webhook payload before updating the order. Replay from logs on failure.
5. Timezone Drift
Paystack returns timestamps in UTC (2026-07-22T09:00:00Z). Your code compares them using local server time (EAT = UTC+3). A subscription renews at 11pm UTC but your server sees it as next day, marking it as early payment.
Fix: Parse all Paystack timestamps with explicit UTC: new Date(txData.paid_at + 'Z') or use a date library with timezone support.
6. Reference Collisions
Two requests hit your initialize endpoint simultaneously and generate the same reference (e.g., based on Date.now() with no randomness). One payment overwrites the other in your DB.
Fix: Use a UUID-based reference: ref_ + crypto.randomUUID(). Make reference a unique index in your database.
Testing Each Failure Mode
// Jest tests for reconciliation failure modes
describe('Reconciliation failure modes', () => {
// 1. Missed webhook — nightly verify job
it('marks pending orders as paid when Paystack says success', async () => {
await db.orders.create({ reference: 'ref_001', status: 'pending' });
jest.spyOn(paystackClient, 'verify').mockResolvedValue({
status: 'success', amount: 500000, currency: 'NGN'
});
await runNightlyReconciliation();
var order = await db.orders.findByReference('ref_001');
expect(order.status).toBe('paid');
});
// 2. Duplicate webhook
it('skips duplicate charge.success events', async () => {
await db.webhookEvents.create({ event_id: 'evt_001' });
var markPaid = jest.spyOn(orderService, 'markAsPaid');
await handleWebhook({ event: 'charge.success', data: { id: 'evt_001', reference: 'ref_001' } });
expect(markPaid).not.toHaveBeenCalled();
});
// 3. Amount mismatch
it('rejects payment when amount does not match expected', async () => {
await db.orders.create({ reference: 'ref_002', expected_amount: 5000 });
var result = await verifyAndFulfill('ref_002', { amount: 490000 }); // 4900 NGN
expect(result.error).toMatch(/amount mismatch/i);
var order = await db.orders.findByReference('ref_002');
expect(order.status).toBe('pending');
});
// 5. Timezone drift
it('parses Paystack paid_at timestamp as UTC', () => {
// Simulate a payment at 22:30 UTC on July 22
var paidAt = '2026-07-22T22:30:00.000Z';
var parsedDate = new Date(paidAt);
expect(parsedDate.getUTCDate()).toBe(22);
expect(parsedDate.getUTCHours()).toBe(22);
});
// 6. Reference collision
it('rejects duplicate references at the database level', async () => {
await db.orders.create({ reference: 'ref_003', status: 'pending' });
await expect(
db.orders.create({ reference: 'ref_003', status: 'pending' })
).rejects.toThrow(/unique/i);
});
});
Nightly Reconciliation Script
// reconcile.ts — run nightly via cron or scheduled function
async function runNightlyReconciliation() {
var cutoff = new Date(Date.now() - 15 * 60 * 1000); // 15 min ago
var pendingOrders = await db.orders.findAll({
where: { status: 'pending', created_at: { lt: cutoff } }
});
var results = { fixed: 0, stillPending: 0, failed: 0 };
for (var order of pendingOrders) {
try {
var response = await fetch(
'https://api.paystack.co/transaction/verify/' + order.reference,
{ headers: { Authorization: 'Bearer ' + process.env.PAYSTACK_SECRET_KEY } }
);
var data = await response.json();
if (data.data?.status === 'success') {
var paidAmount = data.data.amount / 100;
if (Math.abs(paidAmount - order.expected_amount) > 0.01) {
console.error('Amount mismatch on ' + order.reference, { paidAmount, expected: order.expected_amount });
results.failed++;
continue;
}
await db.orders.update({ status: 'paid', reconciled_at: new Date() }, { where: { id: order.id } });
results.fixed++;
} else {
results.stillPending++;
}
} catch (e) {
console.error('Reconciliation error for', order.reference, e);
results.failed++;
}
}
console.log('Reconciliation complete:', results);
}
Learn More
This guide is part of the Paystack payment verification and reconciliation series.
Key Takeaways
- ✓Missed webhooks are the most common failure — always have a nightly verify job that re-checks pending orders.
- ✓Duplicate charge.success events happen when Paystack retries and your handler processes both. Use an idempotency key.
- ✓Amount mismatch: compare txData.amount / 100 against your order.amount_expected — reject if they differ by more than floating point tolerance.
- ✓Status desync happens when a webhook fires but the DB write fails. Wrap verify + DB update in a transaction or use an outbox pattern.
- ✓Timezone drift hits bank transfer orders: Paystack timestamps are UTC, your server may be in EAT (+3). Always parse timestamps with explicit timezone handling.
- ✓Test each failure mode with a specific unit test. Write a reconciliation script that runs every night and alerts on mismatches.
Frequently Asked Questions
- How often should I run the nightly reconciliation job?
- At minimum, once a day during off-peak hours. For high-volume apps, run it every hour. The job should only process orders older than 15 minutes in pending status — this avoids racing with in-flight webhooks.
- What is the best database index for reconciliation queries?
- Index (status, created_at) together. This lets the reconciliation query scan only pending orders within a date range without a full table scan. Also add a unique index on reference to prevent collision bugs.
- Should I alert on amount mismatches or silently reject them?
- Alert on them. An amount mismatch almost always indicates either a bug in your price calculation or an active attempt at price manipulation. Log it, do not fulfill the order, and have an alert ping your team immediately.
- Can I replay missed webhooks from Paystack?
- No — Paystack does not provide a webhook replay API. You verify status directly via the transaction API. This is why the nightly reconciliation job is essential: it fills the gap that missed webhooks leave.
Ready to build real-world apps?
Join the McTaba Labs full-stack marathon (4 months full-time · 6 months part-time). Learn M-Pesa, USSD, and WhatsApp engineering while shipping 8 production apps.
Apply to the McTaba Marathon