Oh but there's more. The £59.5 million fine levied by the British FSA (the rest is US regulators') *goes back to the banks*! Because the FSA is paid for by the banks it regulates, and it explicitly uses any fines it levies to reduce next year's fees! It's as if your parking fines were deducted from your next road tax bill.

(chapter and verse, for the wonks amongst us, on my Tiintax blog)

British banks are not just failing to be boring this week—they’re being actively interesting

In the next issue of "Thrilling Banking Stories", Captain Futures faces the return of the Creative Accountant!

That being said, whenever our team makes *any* change to autosys job flow, I generate a script that'd recreate the whole thing if somethign happened to our unix server. Not sure I could do that with our CA-7 job flow on the mainframe.

IJWTS that Abi's recent couple of posts on the crimes of British banks have been fantastically interesting to me, because Abi knows what she's talking about in that way that one-time front-line middle-level employees tend to do. It's a kind of knowledge that can't be substituted for by more abstract varieties.

Also, this: The Barclays scandal is not 'wholly inappropriate'. It's a crime.

Also, this.

Years from now, they will wonder why we were so supine.

I'm afraid to watch any video coverage of British scandals, for fear that they will disrupt my conviction that the English upper classes consist entirely of portly white-haired fellows in waistcoats and elaborate facial hair going "harrrumph-a-rumph" all the time. (Am I maybe thinking of Parliament?)

Every normal man must be tempted at times to spit on his hands, hoist the black flag, and begin to slit throats. -HL Puffinstuff, er Menken.

It's fun to charter an accountant
And sail the wide accountancy,
To find, explore the funds offshore
And skirt the shoals of bankruptcy!

It can be manly in insurance.
We'll up your premium semi-annually.
It's all tax deductible.
We're fairly incorruptible,
We're sailing on the wide accountancy!

Mencken, not Menken.

I can't be the only person who's looked at the history of events leading up to the Great Depression of the 1930s, and thought "this looks oddly familiar", now can I?

Then again, as someone who watched a lot of ABC television in Australia during the 1970s and the 1980s, I'm very used to re-runs.

It is so easy to skim these articles about financial scandals and not make the connections. Thank you for connecting the dots.

And thank you for the explaination of LIBOR. I knew what the acronym stood for, but had no idea how it was calculated.

Doing some research on the 1920s, as you do when when you want to pitch a vixen^[1] of breeding against a bunch of kidnappers^[2], I was struck by how much today's mess is a re-run of that past.

Not just banking (it is a recurring theme that the "City" fails to invest in UK manufacturing, and we almost lost two world wars because of it). but the politics. The constant thread is the Conservative Party, with the liberal opposition to that rapacious strand of British politics continually being reinvented and renamed.

This is a rerun of the ancient "I'm all right, Jack!" political routine.

The sad thing is that it doesn't work. We know that. We have evidence. We know how countries have worked their way out of an economic crisis, and how they have not.

That is not why I made Lady Helen something of an Anarchist. I thought she had seen the idiocy of the old style politicians, expressed through the Great War during her teenage years. But it fits. And her father, still alive, might be a Duke, but he is neither Conservative nor rapacious. He's just an old fox who can think well ahead. I have a feeling that he would find he has a lot in common with Seebohm Rowntree. And he is in a position to do something, so he does.

I think we could do with a few more people like Gonville Todd in politics.

[1] Of course she's a vixen, it's a furry setting. And David Lloyd George really is a Welsh goat.

[2] Unfortunately, Lady Helen does not recall exactly what she did to the kidnappers: concussion after she fell off her motor-bicycle. The local Coroner's Court is having a busy day...

Now would be a good time for Colonel Rainborough, Major Wildman, and Captain Lilburne to make an appearance.

Just a note for Wendy at #1: The VED is actually not road tax. It's a vehicle tax, assessed on emissions (and there are low-emitting cars for which the rate is £0), and goes into the general taxation pot. It's not earmarked for roads.

Important distinction because there is a widespread belief that the VED is specifically for road upkeep, and this stokes aggression toward cyclists (who don't pay VED) in the media and on the roads.

Chris at #11 - [laughing out loud] because I thought I was pushing it with the tax-and-regulation wonkery. Well spotted, that man!

Fragano Ledgister at #10 absolutely! Although I chiefly remember Colonel Rainborough for the dry comment of "quite" written in the margin of the A level essay in which I had written that "Colonel Rainborough's death was probably fatal" [I meant, to the Leveller cause]

And, Patrick at #3, if you have access to the BBC Parliament channel, Bob Diamond is testifying in front of the House of Commons Committee *right now*.

Wendy: I'm afraid I'm a one-trick pony where taxation is concerned.

Also, I ought to apologize: I should have pointed out at the time that I found the substance of your comment very illuminating.

Fragano @10: Except that (as I was told it), even the Levellers didn't want suffrage for the completely unpropertied. I suspect that what we really need is Gerard Winstanley.

(HLN: area man finds himself unexpectedly in Edinburgh for the evening en route to a philosophy conference. Other things that are currently insufficiently boring here include the weather - we've just had a thunder storm which has pretty much turned Prince's Street into a river.)

BTW - does anyone know any good but quietish pubs in the vicinity.

Wendy Bradley @12:

That's a delightfully wonkish blog you're writing there. Sounds like a fun project.

And thanks for the information about where the fines go in your comment at 1. I should have expected that there'd be some cozy arrangement.

By the way, contra Patrick @3, my perspectives on the City, finance, and investment banking don't come from my solid, reliable years at RBS. That gave me the jargon and argot for the first half of this post, but the second half dates back to my sordid past as a trainee chartered accountant. I specialized in the audit of unit trusts for the investment arms of life assurance companies, and got a crash course in City culture as I read up on all of the assets I was dealing with. The slightly cynical spin on matters came with the territory, and I have never lost it. I don't expect to.

Abi @ 15... "...my sordid past as a trainee chartered accountant..."

Children, let that be a lesson to *you*!

praisegod barebones @14:
does anyone know any good but quietish pubs in the vicinity.

Straw poll says the Guildford Arms on West Register Street, which is certainly what I would have recommended five years ago. It's set back from Princes Street on the east end -- it's behind the Burger King across from the Balmoral Hotel.

Also, I'm vexed that I'm no longer in Embra.

What's especially crazy is they weren't twiddling some minor tranche of loans or even a whole subsector, as they've been wont to do in the past. The LIBOR is the whole damn system. Unless you have access to the Fed window, LIBOR is The Price Of Money.

abi @ 17: thanks! It seems that despite my years in Turkey, I can still sniff out a good pub, as I'd already picked out the Guildford Arms as a likely spot. (Nice ceilings, too).

Ahh, banking. Banking and Ireland. More than just the bread in the Dutch Sandwich, we appear to be uniquely, spectacularly bad at High Finance, so bad we've achieved escape velocity from the normally endless cycle of good -> bad -> so bad it's good. Where else would take a patch of derelict inner city wasteland, and turn it into, well, Mos Eisely isn't a bad comparison, but there's usually a bit of life in most Mos Eisleys - the IFSC is dead after 6pm.

When it comes to retail banking, there were the Big Four: Allied Irish Bank, Bank of Ireland, Ulster Bank, and Northern Bank.
Northern Bank was bought by National Australia Bank, and then Danske, and have just (June 28th) announced they're closing all of their branches now that everyone can use internet banking. Good timing on that one.
AIB is now owned by the government.
Bank of Ireland is ticking over, having lost ~99% of its value and accepted bailouts up to the edge of nationalisation, but no further.
And now Ulster Bank has not functioned as a retail bank since June 19th, and won't be up and running until "the week of July 16th". The expectation is that as soon as they get back on their feet, they'll dump their bad loans and flee, having lost quite a few customer accounts.

Because Ireland is so small, banking and politics are quite heavily intertwined, what with loans to politicians having very flexible repayment terms. The small size also means that Facebook posts from recruiters like "'Anyone know anyone in Carrigaline Cork, Ballincollog (sic) Cork, Cavan, Gorey, Monaghan and Mullingar looking for temping data input work???? Working for Henry the Hippo !!" make the rounds, and posts like this one make you wonder how far back the manual reconciliation has to go.

There are moments of light relief, though: has the saga of the Quinn group made it past these shores yet? The latest entry comes with the standard Daily Mail health warning, but is a good synopsis.

I shall enquire after their relatives in Zurich, and will promise to bring Barrys Tea and soda bread, and a few packs of Tayto while I rest.
(Please let it be the Daily Mail link at fault!)

More on what Libor is from before the scandal broke.

Mike McHugh @ #20:

Interesting. I knew banks bought each other about as frequently as Swedish communist parties splintered, but...

I do however have a slight semantic quibble. I think that would be "... then Danske Bank" ('Danske' is Danish for "Danish", I suspect calling Ulster, Ireland and Northern just that would be borderline confusing, at best).

Ingvar M @ 23... I knew banks bought each other about as frequently as Swedish communist parties splintered, but...

"Excuse me. Are you the Judean People's Front?"
"F*ck off! We're the People's Front of Judea."

Ingvar M @ 23: Noted - I suspect that my fingers couldn't quite believe I meant to write Bank so often in one post :)

Here's an interesting opinion piece: I didn't know about that cyber raid, sensationalised or not.

Mike McHugh @ #25:

I half suspected bank-fatigue was involved, but I didn't want to bank on it.

At least none of the Banks were on the Banks of a river (and if that was the Thames, they might've been near Bank, as well).

Ingvar M @ 26... if that was the Thames, they might've been near Bank

Reading Iain Banks?

BSD:

Actually, it *wasn't* the price of money--that's the scandal!

As I understand it, the banks' traders were influencing it to improve the prospects of their investments, offering to influence it in exchange for business from important customers, and misreporting their own borrowing costs (without any particular interest in changing the reported LIBOR) in order to pretend they weren't paying high interest rates lest someone take that as a sign that they were risky. The claim is that the Bank of England and maybe other governemnt/regulator types may have known and approved of that last bit. (That would be of a piece with the way the bailout was done in the US, where all kinds of efforts were expended to keep things secret lest investors and lenders find out that bank X was weaker than the rest, and thus stop lending to it, downgrade its bonds triggering the CDS version of a margin call, etc.)

I'd heard of LIBOR years ago in school, but I didn't know how it was set. The fact that this was self-report data says to me that it has probably been being manipulated like this forever, but that over time it probably became more brazen, and it was a scandal that could be found once people started looking. The reporting I've read on this also says that this manipulation was an open secret--I read quotes from a sales guy talking about how he was spinning it to big-investor customers of the bank's services.

I suspect there is a maxim in here, related to the "you get more of what you measure and reward." When lots of regulations and contracts reference some piece of data (like LIBOR, the Fed funds rate, the reported inflation rate, or the rating on a security), there is a huge incentive to game that data somehow. You can end up with a potemkin village of made-up and tweaked numbers which make the world look very different from reality.

Another example of this, depressingly, is in crime statistics. Big cities have apparently been manipulating those numbers for years, because the police and mayor are judged by those numbers. Yet another example is high-stakes tests, where teachers have an incentive to game the numbers used to judge their performance (and there have been big scandals involving systematic cheating on NCLB tests, as well as counterproductive teaching to the test), and students have an incentive to game the numbers on tests that get them into college or qualify them for a professional degree (and there have been scandals about cheating on those tests, as well as a big test prep industry involved in a vast zero-sum game with roughly the productivity of shoveling hundred dollar bills into a fire.)

A pessimist would probably wonder how many officially reported numbers we assume reflect reality are really BS, and what happens when more and more of them turn out to be so. Certainly, nobody really important will get into serious trouble for doing so, though perhaps a few low-level nobodies at the level of enlisted men from West Virginia will be sent to jail.

SFnal speculation (I've commented along these lines before):

It seems clear that much of the world we live in is built on deception at a certain level. In order to get everyone working in the same direction, propaganda, spin, and BS for the rubes is necessary and widely deployed. Perhaps it is true that few or no really large organizations can survive without it--once you get a decent sized hierarchy and investments and physical plant and such, you routinely do stuff that would shock your employees, taxpayers, voters, parishoners, members, investors, customers, etc., and only by keeping them from seeing this do you make the continued existence of the organization possible. Perhaps it is merely that our organizations, having evolved in a low-transparency environment, cannot survive too much sunlight.

The internet and surrounding technology seems to be making radical transparency possible. This is having effects everywhere--it's disrupting the propaganda messaging of people at the top, showing disturbing pictures and videos that make it harder to keep running our wars and foreign policy, making it harder to keep secrets by threats and bribes and connections and calling in favors.

It seems possible to me that, if transparency keeps increasing, it will have an effect on our biggest institutions like sun pouring in the windows of a room full of vampires. Perhaps the financial system as it exists now cannot survive serious transparency. Perhaps government as we are used to it in the US cannot survive serious transparency. Or police work. Or courts. Or medicine. Or journalism.

That's very speculative, but if true, it would explain why the Obama administration has hammered so hard on whistleblowers--that could be a fear response, responding to the increasing difficulty of keeping the level of secrecy from the American people of what we're doing that is necessary to be allowed to keep doing it.

Oh you hit a pet peeve of mine there! Trying to use the "transparency" of government data on consultations [ http://www.hm-treasury.gov.uk/d/tax_consultation_tracker.pdf] when they share it as a PDF (so you* can't DO anything useful with it) and update it without so much as a tweet to TELL you it's been updated...

(*...well, ok then, maybe you could, but I can't do anything useful with it!)

Awesome post; awesome enough I don't have anything to add. (Although if auditing the weird assets in life insurance is sordid, I hate to think what adjective my job would get; "the regulators make us hold fake liabilities so we are in the business of producing fake assets" was more honest than normally put, but that's a normal level of cynicism.)

albatross @ 29 makes me link to one of the most thought-provoking finance posts I've read in the last year--Interfluidity on opacity in banking. (floowup post)

tl;dr: The complexity of bank finance is the way we make it opaque enough to enable socially-beneficial risk taking.

An awful lot of rates depend on LIBOR. It's not just what banks charge each other. In the commodities trading world, all of the discount factors (short version: a dollar today is worth more than a dollar in the future because you can invest it for profit) are calculated off of LIBOR.

This ripples through the entire system.

Wendy @30:
at the risk of mansplaining, some ways to monitor online pdfs for changes..

@1: goodness gracious. They're worse than you think, even when you think you've thought the worst..

Mike McHugh @20:

There but for fortune go Scotland's banks as well, you know. The two regional banking systems orbit a larger one, after all: the Phobos and Deimos of the English-and-Welsh Mars.

Scotland had three large native banks when I moved there in 1993. The English banks and the building societies clouded the issue, but it's easy to tell a major Scottish bank: if the money in your wallet could come from them, they're a biggie.

The three are:

The Bank of Scotland, which is the second-oldest bank in the UK (a year younger than the Bank of England). Unlike the BofE, it's always been a commercial bank. Visitors are often surprised to see ATMs on its headquarters on the Mound in Edinburgh.

The Bank of Scotland was in the bidding for NatWest, up against RBS, and lost*. Shortly afterward, it merged with the Halifax, an English ex-building society that had demutualized in the Great Carpetbagger Wave of the 1990's. Because the Bank of Scotland was founded by an Act of (the Scottish) Parliament, it took an Act of (the Westminster) Parliament to allow the merged group (HBOS) to reorganize itself.

But all the jiggery-pokery and asset-shuffling couldn't save them, and HBOS was swallowed by Lloyds TSB (another bank-building society centaur) in 2009.

The Royal Bank of Scotland, not to be confused with the Other Lot. It's "Royal" because the Bank of Scotland was associated with the Jacobites back when that mattered. When I joined it in 1997, it was a reasonably solid regional bank, a little on the adventurous side. It had a few London branches from the purchase of Williams & Glyn in the 1980's, and had partnered with Banco Santander in Spain back when sticking a British card into a Spanish ATM and getting pesetas out was a startling concept. (Also, back when there were pesetas.)

I've talked a little about its subsequent growth, overgrowth, and die-back before. The relevant bit here is that RBS got Ulster Bank as part of NatWest. But NatWest had never integrated UB into their systems, so adding that set of accounts to the RBS batch was done later. I get the feeling the integration process was more difficult than anticipated, and might not have been quite as smoothly finished off as the high-visibility work of adding the NatWest accounts to the batch.

The third, and smallest, major Scottish bank is Clydesdale Bank, whose banknotes are the least common†. They're part of the National Australia Bank group, like Northern Bank was. Unlike the Northern Bank, it's still part of NABG. In 2001, it subsumed its fellow NABer, Yorkshire bank.

Really, had RBS not done its minnow-swallowing-a-whale act on NatWest, the Scots would be looking at the same combination of train-wreck and colonial empire on their banking high street that the Northern Irish are. And considering what happened to RBS after that, and what it's done to the economy as a whole, one almost kinda wishes they had.

Something in me still protests: "We were good, once. We were awesome, technically, and our board was no more foolish than any board." I want to blame bad influences (the Threadneedle Street gang and the gateway drugs of M&A) and early trauma (repeating the failures the Darien scheme the way children repeat their parents' mistakes). But I know it was all foolishness, chosen and directed by fools.

Basically, I want it to have been different.

-----
* Both bidders were partly interested in NatWest as a poison pill, to reduce their appeal as a takeover target. It worked for RBS.

† Tourists sometimes take a while to figure out that the non-Bank of England notes they get handed in Scotland are still good money. Then they tend to be comfortable with RBS and BoS notes, but still balk at Clydesdale ones. This makes the sign Clydesdale had at Edinburgh Airport a little extra-funny. Against a background of some industrial-looking metalwork, Clydesdale Bank claimed it was "Forging a New Scotland."

(One day, when all this banking gossip grows old, we will talk about Scottish life assurance companies. Money and history: always a fun combination.)

Further details emerged today at an Oireachtas Finance Committee meeting (Irish Parliamentary system). While the majority of the ire focussed on the whole "that's my bonus, and I may have earned it" approach of the CEO (since defused), the thing that stuck out like a sore thumb was this bit:

"16.15 – Shane Ross is now asking about a back up plan for the bank. Chris Sullivan says he thinks the back up plan was last tested six months ago. Jim Brown says: “I don’t know when it was last tested or when it was last reviewed. Shane Ross is incredulous."

Granted, they're CxO level, but you'd imagine they'd've been briefed on the backup strategy. Unless there are no valid backups more recent than 6 months, which would feed into the speculation about how far back the manual reconciliations have to go.

abi @ 34: the individual banks issuing their own bank notes always raised issues: it was next to impossible to actually use Bank of Ireland sterling notes in London. I seem to recall one of the regulars here once expanded on that, wrt Cheltenham, petrol stations, and lots of cash.

albatross @29: I think a lot of the hope around "new internet transparency" relies on the good guys making better use of the technology than the bad guys. Sure, there's lots more data becoming available on-line, but much of it is keyed by company. We already know the familiar pattern where one company owns the wells, another does the drilling, and another employs the drillers, in order to make it harder for humans using pencil and paper to figure out who's liable for what. What happens when bureaucracy gets more automated, so it's easy to to have a different company owning each well, and a different company employing each employee?

I've already seen an analogous situation with internet domain name registrars. You know about companies like Go Daddy, TuCows, eNom, etc. who will register domain names for their customers. As .com registrars, they can also participate in "the drop", meaning they can queue up to buy previously-registered domain names when their registration terms expire (and turn around and auction them off to the public). However, since positions in the queue are handed out in round-robin fashion to any registrar who applies, some of these companies now own tens or hundreds of registrars.

albatross, Jeremy Leader.

If you haven't already, you'd probably be interested to read the current discussions on Open Data over at Crooked Timber.

Mike McHugh @36:

Shane Ross is now asking about a back up plan for the bank. Chris Sullivan says he thinks the back up plan was last tested six months ago. Jim Brown says: “I don’t know when it was last tested or when it was last reviewed. Shane Ross is incredulous."

Granted, they're CxO level, but you'd imagine they'd've been briefed on the backup strategy. Unless there are no valid backups more recent than 6 months, which would feed into the speculation about how far back the manual reconciliations have to go.

They're not talking about data backups, which will be taken via live disk-mirroring between the main and backup datacenters, then archived for a set number of days. It's reasonable to assume that someone saved a copy of the key night's data, so it's not been overwritten by later backups. So the transaction information will be available, as will be the account balances as at the night of the disaster. The trick is that the process and order of applying the former to the latter is pretty complicated, even before you add in interest calculations and interactions with the rest of the global financial network.

What these guys are talking about the action of running the batch in the backup datacenter, which is what should have happened when the batch failed. Take the same input files (mirrored, remember) and run them through clean machines using backed-up CA-7 that didn't get trashed.

Back in my day, we did a full disaster recovery exercise every six months. The base scenario was "the main datacenter is now a smoking hole in the ground. Can we run the batch?" But of course, the batch never failed. The data center never did become a smoking hole in the ground. So, you see, all of those expensive drills were clearly a waste of money.

The thing is, a disaster recovery plan has to be more complicated than just a six-monthly drill. Somewhere, on the night in question, a bunch of people made a bunch of judgment calls: do we proceed with the batch in the main datacenter? Can we get the backup datacenter to run it too? Which copy of CA-7 is clean? Can we roll back the last update to it? What are the risks? What are the consequences? How long will it take to test that option?

You can't plan for every scenario, but having a team of smart, experienced people who have thought about this stuff before it's critical helps an awful lot. My suspicion is that that team no longer exists at RBSG—or at many of its competitors, either. My added suspicion is that that fact is going to change soon.

(I was never part of the DR team, but they're the ones that moved the backup datacenter from Edinburgh to London over a weekend. Powered it down on Saturday morning after the Thursday night batch, loaded the mainframes onto trucks, drove it down the country, set it up, and had it powered up and tested in time for the Sunday night batch. Since we ran a five-batch process, that means they moved the backup facilities 400 miles without disrupting our batch recovery capability.

That team is long broken up and scattered to the four winds now, of course. But I, and everyone from those days, knows exactly who RBSG should call up and offer a job to. At whatever rate of pay he asks.)

Abi @ #39:
You can't plan for every scenario, but having a team of smart, experienced people who have thought about this stuff before it's critical helps an awful lot.

And that, everyone, is why you run regular disaster drills. It's a bit of a pain, but actually exercising things on some sort of basis is a Good Thing.

I caught the news the other day and heard the headline that the CEO of Barclays had resigned due to a banking scandal and I realised I'd lost track of which scandal it was*.

David Weingart @32 - Oh yes. Back when I worked in insurance LIBOR turned up in contracts and hence pricing models all the time**.

As for the coziness of the fines reducing the amount the banks have to pay the FSA... well someone needs to look into that. Designing this kind of thing to punish the guilty and reward the virtuous is not as trivial as it seems, based on some things I remember.

* "It's not going to be easy to get a replacement CEO," I said.
"They'll have to offer him, I don't know, a really big bonus," said my brother.
"And as he knows there's a good chance the board might have to fire him, he'll want a really good golden parachute. And since they want to keep him, some really good golden handcuffs as well."

** During a contract negotiation the Finance Director was unhappy with the predicted future interest rates which lead me to remark to my boss that "If he really thinks we can predict the future LIBOR, he's not paying us nearly enough."

There's a story going around that there is an Android botnet out there, which is potentially nasty. There's is some reason to be sceptical: the source is Microsoft, and the evidence for it being Android-related is one of the easily-spoofable email headers.

Still, it raises the question: can you be sure of yout apps? I'm undecided if it is allegation or scare story, but the suggestion is of smartphones getting their apps from an unreliable source.

I am inclined to the feeling that, with so many Android things not being supplied with access to Google Play for apps, source reliability could be a big issue. But what does Google do to protect you?

I'm inclined to be sceptical about an actual botnet, but the questions don't go away.

Neil W @ 41... he'll want a really good golden parachute

...or an anvil. Those always worked well for Wyle E. Coyote.

"Klytus, I'm libored."
- Ming the Merciless

"No! Not the libor worm!"
- Aura, daughter of Ming the Merciless

Dave Bell @42:

Thanks for that. Pulled it to the front page, because it bounces off of something else that's going on and into somewhere interesting.

Dave Bell #42: Well, that's the tradeoff against the limits of Apple's "walled garden" approach.

Serge Broom #44:

Clearly, the international economic system is experiencing LIBOR pains. It's about to lay an egg.

abi @39: Thanks for the explanation: it cleared up a lot.

Another article that sheds further light appeared in the Irish Times today. It seems that the Ulster Bank transactions were bolted on to the batch system in 2006, and by default those transactions did not include the date (unlike the other transactions), but did include the dual-currency requirement.

I could be wrong, but it looks like they're saying that as part of the recovery process "the processing centre subsequently tried to run the scheduled batch of transactions for June 20th in parallel with the batch of June 19th" which included two sets of transactions form Ulster Bank which couldn't be differentiated from each other: Job N.Date A is distinct from Job N.Date B, but Job N on its own isn't. I wonder if that could be the root cause of the whole snafu?

Fragano @ 47... LIBOR pains. It's about to lay an egg.

Love's LIBOR's Lost?

Mike McHugh @48:

Thanks for that link. The description of the CA-7 problem turning into a flush of the scheduler data matches what I've heard here and there.

That was the point where they were screwed. What baffles me is why they didn't start again in the backup datacenter in parallel. I suspect we'll never know the whole story.

I hope the right people inside the bank learn it. I'm just a rubbernecker here, but there are people who need to make it their jobs to truly understand the problem, then prevent it from happening again.

Not every institution has the moral courage to really do that. Lines of investigation cross the paths of powerful people, and suddenly get stopped or redirected. Solutions get hung up on culture wars. Management loses interest partway through the process and things never get finished.

Also, from the article, as clear an example why we ported the NatWest batch to RBS software as I ever did see:

THE WAY DAILY transactions are processed within the RBS group mirrors how the group was built up in the first place.

RBS bought NatWest in 2000 and integrated its system into RBS that same year. Ulster Bank was integrated into the group systems on top of NatWest in 2006.

I was working off-host (not on mainframe systems) when the decision was made to merge systems, rather than merge data, with Ulster Bank's stuff. So I don't know the basis of that decision.

There may have been a technical reason why the systems had to be merged. But I also do wonder whether management was tired of this expensive conversion and testing process and wanted to be off doing something more exciting to the shareholders. I have no inside knowledge, but I know how bank management works, and six years is a long time to be seen to be doing the same project.

This meant that the IT problem had to be fixed for RBS and NatWest first before the Ulster Bank transactions could be processed. This happened within days of the computer glitch on June 19th.

Plausible. They were running in parallel through well-understood, bedded-in processes. CA-7 broke, but the rest of the process clearly wasn't that fragile.

The nature of the Ulster Bank computer systems posed more complicated problems, which caused delays in the repair work.

The Irish bank is in effect made up of two systems – Ulster Bank and First Active. The Ulster Bank system also has to deal in two currencies – sterling in Northern Ireland and euro in the Republic.

This last point may be the reason that the Ulster Bank data was not run purely on RBS processing. Like most British banks, RBS runs on one currency. Swapping to another is one (huge) piece of work, one that pretty much every British bank has long since scoped out and planned for. But running two in parallel is a much more substantial systems change, and I don't know that the RBS management wanted to invest in that kind of change to core systems.

Betcha anything that First Active and Ulster Bank people made redundant after the merger with RBS are getting some very good day rates right now. I also bet that their knowledge was sorely missed in the first hours after the disaster.

A major technical difference between the NatWest/RBS and Ulster Bank systems is that the UK system is “date-specific”, said RBS executive Chris Sullivan.

This means that if it is running scheduled transactions for June 19th, any other day’s transactions will automatically be excluded until the June 19th batch is completed.

Yep.

“The Ulster Bank system is agnostic of date,” said Sullivan. “It will not automatically kick out the wrong date, so they have to be manually extracted. The minute you get into manual processing, it gets extremely difficult.”

"Manually" here means "using hand-carved filtering routines", by the way, not playing sic-et-non with the delete key. We did a bunch of that in prepping for the NatWest systems merger. I wonder if the descendants of that toolkit were in use that night. Entirely possible that they were.

Also, is everyone clear why we have such an intimate understanding of the date processing routines of all of these banks? Yes, I thought you would be.

abi, from the "Holding Vulnerability" thread:

They removed the keyboard, mouse and network cables, then locked the door and went to lunch.

That's one way to secure a system. I think you can see the flaws in it.

Unfortunately, that's the only way to entirely secure a system. And trying to get as close to that as possible is why every financially-critical system I'm familiar with runs mirror-and-batch, rather than having a live connection to the outside world. (And then, when the batch breaks, the abi's and Sam's of the world look on in fascinated horror.)

I am finding this whole discussion fascinating--it's just at that point where I don't know this stuff, but I know enough about related stuff that it makes sense.

Chance that lessons will be learned from this: 100%.
Chance that the right lessons will be learned from this: 60%.
Chance that those lessons will be forgotten in 5 years, at least by those who can't see why they spend so much money and staff on local IT: 90%.

What, cynical, me?

Serge Berowne #49:

Some things are just too abhominable.

Countrywide issued hundreds of VIP loans to buy influence, report says

Countrywide Financial, once the nation's largest mortgage lender, bought influence on Capitol Hill by issuing hundreds of sweetheart loans for members of Congress, their staffs and other government employees, according to a report issued by the House of Representatives Thursday.

The three-year investigation, which was led by House Oversight and Government Reform Committee Chairman Darrell Issa, revealed the extent to which Countrywide was offering discounted loans and other perks to Congress members and other officials in order to benefit its business interests.

Mycroft W @ 52:

Chance that senior technical personnel who remember the lessons haven't lost their jobs to outsourcing in 5 years: 10%.

Fragano Ledgister #47

Looks like reforming the banking system would be on the scale of the LIBORs of Herakles!!!

#56: A certain stable-cleaning feat comes to mind.

54
Gallegly and McKeon both represent the area where Countrywide had its main office. So I'm not surprised they were getting special deals.

Greenwald today has a wonderful take on this stuff. So long as the people at the top know that breaking the law will have few consequences, there is no reason at all to think they won't continue doing it.

If tomorrow, the prosecutors all throughout Maryland announce that they will no longer prosecute insurance-fraud arsons, a strange physical transition will take place across the state, and houses and buildings (especially ones underwater on their mortgages) will magically become more flamable, somehow. If they announce that they're done prosecuting embezzling cases, then by some inexplicable change in the laws of nature, the rate at which money evaporates from cash drawers and poorly-watched accounts will increase dramatically.

We have done more-or-less this to our bankers and top spies and government officials. We've done the equivalent of announcing that car thefts will no longer be prosecuted, and now cars keep disappearing from their lots. And this is making us all poorer and worse off, but the car theives are too well-connected and have coopted too much of the respectable media to allow much of a movement to re-criminalize grand theft auto.

Greenwald today has a wonderful take on this stuff. So long as the people at the top know that breaking the law will have few consequences, there is no reason at all to think they won't continue doing it.

If tomorrow, the prosecutors all throughout Maryland announce that they will no longer prosecute insurance-fraud arsons, a strange physical transition will take place across the state, and houses and buildings (especially ones underwater on their mortgages) will magically become more flamable, somehow. If they announce that they're done prosecuting embezzling cases, then by some inexplicable change in the laws of nature, the rate at which money evaporates from cash drawers and poorly-watched accounts will increase dramatically.

We have done more-or-less this to our bankers and top spies and government officials. We've done the equivalent of announcing that car thefts will no longer be prosecuted, and now cars keep disappearing from their lots. And this is making us all poorer and worse off, but the car theives are too well-connected and have coopted too much of the respectable media to allow much of a movement to re-criminalize grand theft auto.

Stefan Jones #57
Diverting the Thames through the boardroom *is* tempting, but I dread to think of the pollution it would cause downstream...

Wow. Greenwald thinks that the City of London is a shining example of making executives take the consequences of their actions?

Wow. I'm...speechless.

@55 Bruce Cohen (StM): I believe that your line and my last line are functionally identical (more specifically, an excellent example of DeMorgan's theorems). It is clear, indeed, that you have S to M.

Re: Greenwald: while what he is saying is correct (and so I agree with albatross), "Losing your job" is suitable punishment? Allow me two years of their salary and bonus, and you can "make me resign" for cause all you like. One of my colleagues will hire me within a year, I bet, but even if I never work again... (and so I agree with abi).

OK, this is more like it.

Vince Cable, Secretary of State for Business, Innovation and Skills (aka Trade Secretary), gave an interview yesterday in which he said some pointed and sensible things.

"The real problem at the moment is that the banks – because of their existing culture, which is frankly anti-business, obsession with short-term trading profits, not focusing on the long term – are throttling the recovery of British industry."

and

"Laissez-faire just doesn't work. When you are making big long-term investments you have to have the government and the private sector working together. It has been a great success story in the car industry, similarly in aerospace and life sciences and others."

Awww. A LibDem who didn't sell his spine to get into the coalition government. Admittedly, he's kicking the banks when they're down, but that's when they're in foot range.

My reading of Private EYe over the years has made me think that the UK is pathetic at prosecuting white collar financial crime, with cases collapsing all over the place and fines in the mere tens of millions, rather than hundreds of millions and actual jail time seen in the USA.

I'd like this not to be the case, but the Serious fraud office doesn't seem to have a good track record.

abi @64

There's plenty of evidence from the last century that British banking has been consistently anti-business.

And Vince Cable has only just noticed?

Abi #64 - or Cable can take a tip from my local MP - stick the heid in...
Although obviously it would be better to attack using strings of words thought up by using your brain than by actually using your skull to hit them.

Dave Bell #66 - yes and no, insofar as there has been* mutual incomprehension between business and banks for centuries. I think anti-business is perhaps pushing it a bit too far, rather, "pro-profit and anti-risk (Except when they can't see the risk or ignore it)", so if you don't look like you may generate much of a return they will ignore you. This has probably been exacerbated by the consolidation in the banking sector and the financial 'reforms' which allow banks etc to roam free, making money in any way they please. By contrast, doing the hard work of understanding a business, why it needs money and whether it will pay it back or not is simply not worth it. In a world run by detached MBA holding senior management, no manager gets to stay anywhere long enough to actually build a proper relationship and they certainly don't get to risk the banks money on company X who needs a good facility because of the volatility of their business.

*Based on my readings of various books over the last 15 years or so since I became an adult, not based on personal experience.

Hmmm...just seen on the InLinked site that the DR guy I was thinking of is actually available for contract work at the moment.

Now we see whether anyone has any damn sense in RBSG.

Either that, or someone for whom English is not their primary language who doesn't know how blogs work.