Overnight batch jobs are comfortable until the business outgrows them. Here is what actually broke when we moved a bank’s reconciliation process to near real time, and what we would do differently.
Batch reconciliation works well until the business changes around it. For one of our banking clients, the trigger was mobile wallet volume: customers expected to see their transactions immediately, but they were appearing the next morning.
The problem was not the batch job
Our first instinct was to run the batch job more often. That is usually the wrong answer, and it was wrong here. The overnight job existed because several upstream systems could only produce end-of-day files. Running it every hour simply meant it failed every hour.
The real work was upstream: getting a stream of events out of systems that were designed to produce files. Two of the four sources could send events with modest changes. One needed a change-data-capture layer. The fourth was a vendor product we could not modify, so we kept a daily reconciliation for that source and were open with the business about it.
What broke
- Duplicate handling. A nightly job that reads a file once is easy to protect against duplicates. A stream is not. We spent more time on deduplication keys than on the ledger itself.
- Assumptions about order. Downstream reports quietly assumed that transactions arrived in time order. They no longer did.
- Working habits. The operations team had years of routines built around a morning report. The technology change turned out to be the easy half.
What we would do differently
We would run the old and new systems side by side for longer. We planned for four weeks and needed nine. Running systems in parallel is the cheapest insurance available for this kind of migration, and we did not buy enough of it.
We would also involve the operations team in the first week rather than the sixth. In a single afternoon, they found three edge cases we had not considered at all.