Forward to next post: On the prospect of being left behind on a ice floe
Was it only a month ago that The Economist did a “Capitalism! Whisky! Sexy!” post about the coming revolution in ‘fusty old retail banking’? How time flies.
[Banks] will cut costs by closing many of their branches. Banks will also tap into new sources of revenue by mining their enormous troves of customer data. A bank that knows what you have just bought or where you have booked a holiday will be able to offer real-time discounts on related products (much as Google targets advertising at people based on their searches). The retail revolution will also offer the best banks the opportunity to gain new economies of scale through their IT platforms.
That was published on May 19, 2012.
A software update at the Royal Bank of Scotland Group on on June 19, 2012—precisely a month later—seems to have led to corruption in the payments feed. The overnight batch on Wednesday, June 20, failed to update a large number of customer accounts with incoming payments. Millions of accounts were affected at RBS and its subsidaries, National Westminster Bank and UlsterBank. Millions of people woke up on Thursday morning without access to their money. It’s been a nightmare.
As a result, some customers were blocked from taking money out of cash machines, while others had internet supermarket food deliveries stopped after payments were rejected.[…]Some people could not use debit cards at tills, including hotel check-out desks, airports and petrol stations.
The bank is still unwinding the problems, and there are still people without access to their money. For weekly-paid employees, people on benefits, and pensioners, the lack of payments is a real hardship. Not everyone has a cushion to tide them over. Some of the very poor, who frequently have coin-operated meters, expected to spent the weekend without electricity.
British techie site The Register has a few guesses about what went wrong in the update. Former RBSG staff have written in to suggest that it was a bug in the bank’s CA-7 system, which schedules the jobs for the overnight batch. They blame the recent outsourcing effort that has moved the support of the bank’s systems to India, a move that was undertaken to cut costs. There might have been plans to gain new economies of scale through the IT platform as well. I wouldn’t bet against it.
I suspect that The Reg’s diagnosis is fair—not because the work was sent to India per se, but because turning over your systems support staff in a wave of redundancies is not the best way to manage the transfer of knowledge. Not everyone who worked the batch at RBSG even knew what it is they knew; how, then, could they explain it to people who didn’t know there was knowledge to acquire? Outsourcing the work from Edinburgh to Aberdeen and sacking the staff would have exposed them to the same risks.
Meanwhile, the only way for customers to get cash has been to go into those much-abused, unsexy bank branches and talk to staff members. The bank extended branch opening hours, even opening on Sunday, to try to meet the demand. It’s going to take some time to get everyone back to where they should have been, and even longer to compensate people for late fees, overdraft charges, and other ancillary costs. I suspect the branch network is going to be busy for some time to come.
It’s the worst IT disaster in British banking history. That Economist article doesn’t come across in quite the same way in the light of it.
More personally: I Y2K tested one of the batch feeder systems at RBS from 1997 - 1998, and managed acceptance testing in payments processing systems from 1999 - 2001. I was one of the people who watched over the first batch of the millennium instead of going to a party. I was part of the project that moved the National Westminster batch onto the RBS software without a single failure. I haven’t worked for the bank for five years, and I am surprised at how personally affronted I am that they let that batch fail. But I shouldn’t be. Protectiveness of the batch was the defining characteristic of our community. We were proud of how well that complex structure of disparate components hummed along.
It was a thing of beauty, of art and craft, and they dropped it all over the floor. Sheesh.