BOI having ATM network issues. Don't be left stuck this evening. - Mod note post #1

Kensington · 07-12-2010 06:02PM

Cavehill Red wrote: »

Big coincidence that they suffer an unprecedented IT breakdown, including all support systems and online, on the very day of the international bank run campaign.

A pal of mine used to work in the ATM IT system for another Irish bank. It's highly unusual for every single system to go down at once. One might say, almost unbelievably unusual.

I do hope they're not lying and can resolve their IT difficulty ASAP.

Is it actually that uncommon? My days of retail would be filled with terror knowing the AIB card authorisations system would crash, without fail, during the christmas rush. Card authorisations would go down, then their atm's would go down due to the extra demand from people unable to use their cards and then their manual payment processing would go down. It'd take hours before they came back online fully.

Monty Burnz · 07-12-2010 06:04PM

Kensington wrote: »

Is it actually that uncommon? My days of retail would be filled with terror knowing the AIB card authorisations system would crash, without fail, during the christmas rush. Card authorisations would go down, then their atm's would go down due to the extra demand from people unable to use their cards and then their manual payment processing would go down. It'd take hours before they came back online fully.

I can never remember that happening in the last 20 years. Not to say that it hasn't, but I certainly never noticed it.

dr.bollocko · 07-12-2010 06:05PM

Cavehill Red wrote: »

True. What I find interesting about that report is the fact that the banks issued denials. They're hardly going to say, yeah, we're losing deposits. People should stop.

If there was a bank run, you're not going to hear about it on the news until it's too late.

We've made it to the wall street journal.
"There is no unusual behavior at bank ATMs," the spokeswoman said. "All ATMs are operating normally but with a restricted cash service for Bank of Ireland cardholders," the bank said. "Some customers may be unable to conduct point-of-sale transactions but Bank of Ireland credit card transactions are unaffected."
http://online.wsj.com/article/BT-CO-20101207-708963.html
Transactions restricted due to issue checking balances.

marco_polo · 07-12-2010 06:07PM

ch750536 wrote: »

Because that is in hindsight. If people were following his words it is too late, they needed to act before it was too late.

It would have made a bit more sense to shut it down their systems before opening for business then, rather than late morning .

Cavehill Red · 07-12-2010 06:07PM

Kensington wrote: »

Is it actually that uncommon? My days of retail would be filled with terror knowing the AIB card authorisations system would crash, without fail, during the christmas rush. Card authorisations would go down, then their atm's would go down due to the extra demand from people unable to use their cards and then their manual payment processing would go down. It'd take hours before they came back online fully.

A single system, especially the remote card authorisations system, can go down regularly. Card authorisations especially, as they are reliant on phone lines too. They can go down in one location and not next door.
For a systematic IT snafu to happen though, is to my knowledge pretty much unprecedented.

FreudianSlippers · 07-12-2010 06:11PM

marco_polo wrote: »

It would have made a bit more sense to shut it down their systems before opening for business then, rather than late morning .

Maybe they didn't notice at first until ppl started getting free cash

Ziycon · 07-12-2010 06:11PM

DISCLAIMER: I'm not saying it is, just my observations of the media coverage.

From that press releases and what we know on what has happened there are similarities as to what happens in big organisations when they are hacked, they will pull the plug on all mission critical systems until they can say they are 100% safe. It doesn't matter how much redundancy they have it will all be taken offline.(Which I wouldn't think is likely for a bank in this day and age)

The other possibility is that due to the legacy systems that banks have there could be a simple issue with the middle-ware failing which would cause havoc with all 'new' systems and retrieving account information/balances.

Or it could be nether...;)

marco_polo · 07-12-2010 06:13PM

OisinT wrote: »

Maybe they didn't notice at first until ppl started getting free cash

Pity it only happened after I had tranferred my rent

marco_polo · 07-12-2010 06:17PM

Ziycon wrote: »

DISCLAIMER: I'm not saying it is, just my observations of the media coverage.

From that press releases and what we know on what has happened there are similarities as to what happens in big organisations when they are hacked, they will pull the plug on all mission critical systems until they can say they are 100% safe. It doesn't matter how much redundancy they have it will all be taken offline.(Which I wouldn't think is likely for a bank in this day and age)

The other possibility is that due to the legacy systems that banks have there could be a simple issue with the middle-ware failing which would cause havoc with all 'new' systems and retrieving account information/balances.

Or it could be nether...;)

Two sounds more likely, and the delay is probably because they are trying to dig up and trying to reanimate the corpses of long dead Mainframe and/or Cobol programmers. :pac:

Justin Collery · 07-12-2010 06:46PM

It's probably just a computer fault, but I would be very, very interested to know what type of fault it was.

The first items to go down were the branch network, ATM's and personal online banking. This was followed a couple of hours later by the business banking section. I know this as I was checking both.

Many moons ago I worked in the BoI ITD. The mainframe at the time as I remember stored personal and business customers side by side. Each customer type was flagged appropriately, but both were store on the same machine.

A failure that effects just personal customers first, and then business customers could happen. You could have an update to the mainframe software that effects only personal customers. Something happens during this update that invalidates the data for personal customers (or at least makes it look invalid), business customers continue as before without problems. In order to fix the problem, the entire system needs to be taken down to roll back any changes, meaning after the fault has been identified, all customers, including business customers are effected.

I'm guessing something like I described is what has happened. Wouldn't like to be the guy or gal in charge of upgrading the production system, or the testing before production, or the update, I'd say some choice language is being used.

I wouldn't get too worked up about it. One banks IT failing, thats just unlucky. Now two banks IT failing at the same time, well, that would be just careless

FreudianSlippers · 07-12-2010 06:47PM

jcollery wrote: »

It's probably just a computer fault, but I would be very, very interested to know what type of fault it was.

mad hax lolz

soden12 · 07-12-2010 06:48PM

It could of course be another example of incompetence from the bank.

Monty Burnz · 07-12-2010 06:49PM

Have they fixed this yet?

CabinteelyGuy · 07-12-2010 06:52PM

Monty Burnz wrote: »

Have they fixed this yet?

From what i know not till tomorrow.

AnonoBoy · 07-12-2010 06:52PM

"Well Mr. Bond I expect you're wondering why I shut down the entire ATM system of the Bank of Ireland. Let me explain it to you while this high-powered death laser slowly makes it way towards your crotch......"

K-9 · 07-12-2010 06:56PM

Monty Burnz wrote: »

Have they fixed this yet?

Suppose we'll know if it's a bug or not by tomorrow.

Anyway, plenty of vested interests who'd love a bank run, currency speculators etc.

wobblyknees · 07-12-2010 06:59PM

Ziycon wrote: »

DISCLAIMER: I'm not saying it is, just my observations of the media coverage.

From that press releases and what we know on what has happened there are similarities as to what happens in big organisations when they are hacked, they will pull the plug on all mission critical systems until they can say they are 100% safe. It doesn't matter how much redundancy they have it will all be taken offline.(Which I wouldn't think is likely for a bank in this day and age)

The other possibility is that due to the legacy systems that banks have there could be a simple issue with the middle-ware failing which would cause havoc with all 'new' systems and retrieving account information/balances.

Or it could be nether...;)

Do you enjoy working for BOI?

Monty Burnz · 07-12-2010 07:00PM

K-9 wrote: »

Suppose we'll know if it's a bug or not by tomorrow.

Anyway, plenty of vested interests who'd love a bank run, currency speculators etc.

I don't know if we could move the euro to an extent that speculators could profit from, not without them making a HUGE bet on it. And it would be a bet.

Slogger Jogger · 07-12-2010 07:00PM

Monty Burnz wrote: »

Have they fixed this yet?

365online was back when I tried it a few mins ago.

Cavehill Red · 07-12-2010 07:01PM

Someone suggested the phone banking system in BoI is now up again? Any confirmation of that?

srsly78 · 07-12-2010 07:03PM

BOI business banking is also offline today wtf?

Novella · 07-12-2010 07:46PM

Just been in my local shop and their BOI atm is off, as is the one on the main street just up from the shop. One woman was actually shouting at a member of staff at the till, crazy lady.

Canonfan · 07-12-2010 07:51PM

I heard people were taking money from the ATM with 0 banlance. Something wrong for sure.

Resi12 · 07-12-2010 07:56PM

Someone I know just took out 500 quid! I'm off for a walk

Gamesnash.ie: Pat · 07-12-2010 07:57PM

Slogger Jogger wrote: »

365online was back when I tried it a few mins ago.

Did you actually log in though ? The main login page is up but it comes up with a "failed to deliver request" error message when actually logging in.

TheSpecialOne · 07-12-2010 07:59PM

People are withdrawing money they dont have! something is going on!

Slogger Jogger · 07-12-2010 08:00PM

Gamesnash.ie: Pat wrote: »

Did you actually log in though ? The main login page is up but it comes up with a "failed to deliver request" error message when actually logging in.

Unless it was just temporarily back, but I got my balances etc. Around 5ish I used a hole in the wall and did a laser txn so your bread and butter services still work. Didn't try to withdraw 500 like others though :rolleyes:

Slogger Jogger · 07-12-2010 08:00PM

TheSpecialOne wrote: »

People are withdrawing money they dont have! something is going on!

Its not like the Bank won't know who they are...

hmmm · 07-12-2010 08:02PM

TheSpecialOne wrote: »

People are withdrawing money they dont have! something is going on!

What may be happening is that BOI are allowing small withdrawals to be made to a certain limit until their main system comes back online. When the system returns, all those people will be charged healthy overdraft fees if appropriate. Good, because they are gob****es if they think they can get free money.

tbh · 07-12-2010 08:38PM

Cavehill Red wrote: »

For a systematic IT snafu to happen though, is to my knowledge pretty much unprecedented.

You've clearly never worked in IT, with respect. A complex multilayer system designed and produced by multiple vendors all trying to underbid each other failing on a large scale is the most precedented thing in the world. What did you think snafu stood for? Situation normal, all fckd up. To paint an it failure as something to be suspicious about is hilarious.

BOI having ATM network issues. Don't be left stuck this evening. - Mod note post #1

Comments