Query on Disaster Recovery- Personal data in BOI

northwestramble

Hi
In light of the recent problems with Ulster Bank, it got me thinking about how other Irish banks that I do my day to day banking with, are set up to cope of recovering from major problems. It would be great if you could answer the following questions and put my mind at ease that all is correct and to help me decide if I should remain with Bank of Ireland. If there is a question you cannot answer, could you explain why you are not in a position to provide an answer.

1. How often you run the disaster recovery tests and when was the last time this was done.

2. Are all of your IT systems maintained within Ireland, or have aspects of it been outsourced. If outsources, are they within or external to Ireland/EU.

3. For any outsourced, outside of the EU, is access to all of my personal data restricted to comply with Irish/EU data protection laws.

4. Are all of Bank Of Ireland systems now fully integrated across the IT section, or are there a number of systems that are independent and not part of a Disaster recovery process.

Bank of Ireland: Linda

Hi northwestramble,

Thank you very much for your questions. We appreciate that customers do have concerns due to the recent issues with Ulster Bank.

For reason of confidentiality and security, I cannot go into detail of our disaster recovery processes. I would like to assure you that Bank of Ireland is fully committed to ensuring continuity of our services to customers.

We can however confirm that we operate a comprehensive policy across the Group, which complies with our regulatory obligations, and encompasses stringent planning with regular and extensive testing.

Thanks
Linda

northwestramble

Hi Linda
Thank you for the quick reply, I am not quiet sure why there would be any security or confidentiality around letting people know about how often you run a DR Test. That does not give anything way, for example if you were to say you run it once a year or 3 times a year. I do not see how that can be a security risk or breach confidentially. Maybe I am missing something ?

In relation to possible personal data been held on systems outside of the EU. e.g on cloud systems hosted in the US or outsourced to other parts of the world. Can you confirm that all personal data is strictly controlled and meets all Irish/EU data protection requirements, and that any access to view or access the data is fully logged.

I can understand you might not want to say if you outsource things or not, but again I do not see any security risk in knowing this or any confidentially issue. I am not asking about the provider or location. Again, maybe I am missing something?

Bank of Ireland: Billy

Hi northwestramble,

Thanks for your questions.

At Bank of Ireland, privacy and data protection rights are very important to us.

Bank of Ireland is registered under the Data Protection Act 1998 - 2003 as a data controller and data processor and all personal data will be maintained in accordance with the obligations of that Act.

While our detailed operational plans are confidential, Bank of Ireland operates a comprehensive policy across the Group, which complies with our regulatory obligations, and encompasses stringent planning with regular and extensive testing.

Thanks for getting in touch.

Billy

northwestramble

Hi Billy
Thanks for the update. As a customer I find it a little strange that you cannot really offer any form of comment on something this critical, other than to say "you meet all regulatory obligations" As a bank I would expect that at a minimum.
However, if you can point in the direction to where I can get a copy of these requirements, that would be very helpful. From that, I can see what Bank of Ireland is committed to providing to their customers, to ensure they have full recovery in the event of a disaster.

MadsL

There are published standards for Business Continuity - BS25999, which will be replaced in Nov 2012 by the International Standard, ISO 22301.

Compliance with such published standards would allow banks and other services to the public to be independently audited. However a lot of banks cling steadfastly to "banking standards" which means they get to decide what is acceptable.

A lot of the Credit Unions have gone for ISO 27001, a standard for information security - basically providing reassurance over the safety of personal information. Again, independently audited.

The Revenue Comissioners hold both ISO 27001 and BS25999. The banks really ought to catch up and stop with the meaningless platitudes.

Random

can the same sh1te that happened ulster bank recently happen you or have you robust measures in place to resolve issues over hours rather than weeks?

Tow

Random wrote: »

can the same sh1te that happened ulster bank recently happen you

It can happen to any bank, but you can be sure the RBS/UB disaster which I believe is rated as the worse banking IT disaster in the world, has made them triple check and update any lacking procedures.

Bank of Ireland: Billy

Hi All,

In relation to your questions on disaster recovery, I can fully appreciate your concerns.

I can again confirm that Bank of Ireland is fully compliant with regulatory obligations.

Although we cannot discuss business sensitive information, I can assure you that we operate a comprehensive policy across the Group which complies with our regulatory obligations and encompasses stringent planning with regular and extensive testing.

Thanks

Billy

northwestramble

Hi Billy
I was wondering if you could point me in the direction of the regulatory obligations. I guess these are publicly available ? Would be great to see what the regulator feels are the correct protections Irish Banks should meet, or it an EU obligation that BOI meets.
John

Bank of Ireland: Graham

Hi northwestramble,

We are unaware of where this information is publicly available. However, if you contact the Financial Regulator, they should be able to steer you in the right direction.

Thanks,
Graham

northwestramble

Hi Graham
Maybe it is just me been silly, but I am a little confused. You state that you meet the regulatory requirements yet
1. You cannot provide me with any details on these requirements.
2. You have no information on where these requirements can be located.

I would have assumed that an organisation that meets standards would have a copy of these standard close to hand and know where to get the latest version to be compliant, or am I missing something ?

Could you so find out from your compliance/auditing section within the bank (or similar named section)
1. What version of the regulatory requirements you use.
2. The exact title of the documents.

This will then help me to make sure I get the correct version of the document from the Regulator and that I am then looking as the same version as you use in there.

You might also be able to find out, the date you last ran the compliance check. I assume there is nothing confidential in this ?

John

Tow

A quick Google brings up this document as the second link. (The company in question may need to have a quick read of BOI's current terms and conditions for suppliers). It basically says they have implemented a SAN, replicated across two data centres. Which as we all know is not much good when you wipe the scripts/programs, corrupt the data and don't have proper backups.

northwestramble

I like the phrase "support its retail banking operations" I would have assumed it was critical to the retail banking process. Bank of Ireland encourages its customers to use more online services, which I have to say are great and I use them a lot.

As a key part of my life now depends on access to these online services, I find it hard to understand why Bank of Ireland is so reluctant to be open and honest about the services they have put in place to protect their customers.

Hopefully now at least I get an answer to my query on the exact document title and version that they use to be compliant and I can then follow up with the Financial Regulator.

Bank of Ireland: Linda

Hi northwestrambler,

We have commented as much as we can here. Thank you for your comments and your concerns have been raised with the various departments which assure us the correct checks are in place.

Thanks
Linda

northwestramble

Hi Linda,
Your collegue asked me to contact the financial regulator to get a copy of the document. I just was looking for the name of the document you use.
I simply want to get the same version the bank uses, as you know yourself, documents can have many versions.
What is the reason you cannot provide that, it cannot be anyway private if your collegue asked me to get it from the regulator.
John

Bank of Ireland: Linda wrote: »

Hi northwestrambler,

We have commented as much as we can here. Thank you for your comments and your concerns have been raised with the various departments which assure us the correct checks are in place.

Thanks
Linda

northwestramble

Hi Linda
I am sorry to hear that you are not in a position to answer any of the questions I have asked here. I understand you have some concerns, but can you explain to me, why Bank of Ireland, cannot tell me what regulations you are compliant with and the version of the document. If you know you are compliant, then it stands to reason that you must know what you are compliant with. Is it that you are not allowed tell me ?

I am sure you must understand my point of view in that I cannot take at face value something the bank says, without
having a basis to compare it with.

Sadly, if Bank of Ireland cannot answer a few simple questions around the security of my account, then I will have to leave the bank after 15 years of doing business.

Irish Steve

Time for a reality check here.

Theories are wonderful, actual practise is another matter, and can come home to haunt you big time.

I've been either involved with or done 2 disaster recoveries, and instigated a third.

The first was a very long time ago, a large poultry processor in the UK had their entire accounts system (visible record computers, yes that long ago) wiped out by a flash flood, and replacing those machines was not possible. By a stroke of luck, and a few other favourable factors, I had a small multi screen mini computer doing the basics in just under a week, but they were 18 hour days, and the full system wasn't operational for nearer 3 months. They survived, but it was close. They were £15 million per annum turnover in the mid 80's and employed 500 people, and disaster recovery as a concept didn't exist in those days for that size of a company. Many with similar problems didn't survive, there was no way back, they were lucky, it was only the machines, they didn't lose the data.

A site in Belgium were running a large mini, and unknown to them, the backup software they were using was leaving a "hole" in the data every time they changed tape reels, so 5 reels of tape per disc meant 5 holes per disc on the recovery. When one of their discs crashed, after recovering the data, and checking the integrity of the database, they found there was a problem, and the problem went right back through their database archives, so it had to be fixed in other ways, there was no clean restore path.

A team of specialists, (myself included) were on site for 10 days, working 18 hour days, going through every transaction that had been processed in the week prior to the failure to extract any updates that were relevant to the corrupt data sets, so that we could update the file with those records. We managed to get all the records, and once that was done, and the database updated, they were live again, having been in inquire only mode for 10 days. This was a large insurance company, and every office in the country was affected. The direct cost of that downtime was massive.

A few years ago, I had a problem with E-bay not working correctly. After a lot of digging, it turned out that Eircom's server system was caching a lot of the E-Bay static content here in Ireland to reduce transatlantic traffic, and the server that provided that data was severely compromised. It took me a number of days to work out what was wrong, doing wireshark data captures, and comparing them to similar captures from another country, and once we found it, it then took 2 days to get through to the right level in Eircom to report it, and once they acknowleged the problem, another 4 days to rebuild the server that had failed.

So, where disaster recovery is concerned, been there, done that, shredded the T shirt several times, and the more I see of "we are prepared", the less I believe it, in that taking regular backups and having a properly organised archive is only the tip of the iceberg, if the backups are not checked far more comprehensively than just that the data can be read from the media, there's NO guarantee that it's going to work when it's needed.

We already know that the regulator has been found wanting in recent times, in many areas, and I'll put money on it that thre are very few IT specialists in the regulators office with IN DEPTH practical experience of disaster recovery.

The added complication is that a lot of the BOI software is legacy, and running on old platforms, and now outsourced, and I'll also suggest that the people that knew the legacy software best, and how to fix it quickest, are no longer working for BOI and didn't move when it was outsourced, as they will have been the most expensive people on the payroll.

And no I'm not taking a dig at BOI here, I suspect that most large multinationals in Ireland would be in the same boat if the fertiliser hit the fan, too many of the day to day activiities of most companies are now controlled by beancounters, and it's a rare beancounter that is prepared to listen to "what if" arguments that are based on possible scenarios that have not happened for a long time, if ever.

The same concepts have taken over the airlines. Read the report on the Air France Airbus 330 crash a couple of years ago, and weep. An incompetent crew killed themselves and their passengers because they didn't know how to deal with the situation they were in. If you dig into why, you will find that somewhere, a bean counter decided that the risk was so rare, it wasn't appropriate to spend that much money on training for such an unlikely event. Maybe if the bean counters had to appear at the airport to comfort distressed relatives and explain their policy then, they might make different decisions.

Banking is no longer about customer service, if the banks could close the branches, they'd do it tomorrow. As it is, they don't open till 10, or later, close for lunch, and then close earlier than any other full time business, and the in branch services are being reduced almost on a weekly basis.

In the old days, the reason for that was so they could process the work on the accounting machines of the day, which were slow, batch, and needed time to get it right. Now, it's mostly computerised, so the time required to "end the day" is a LOT less than it used to be. Now, it's down to they don't want to employ staff to provide the service, they want the customer to queue at automatic machines, to lodge cheques or notes, NO more coin, and certainly not on busy days.

I wonder what the reaction in the branch would be if I presented an invoice for €25 every time I make a lodgement? The reality of changed opening hours is that's the minimum it costs me to lodge 1 cheque, as I have to hang around until they condescend to open at 10, (ish) or even 10 30, if they are doing "staff training", which too often is an excuse for the reduced time they work now. No way to post a cheque in the door for lodging, no late evenings, or weekends, so a direct cost to lodge a cheque that way exceeds the value.

So, to go back to the original theme, I would love to be a fly on the wall at a disaster recovery exercise, with the ability to intervene at a specific point and say "that tape is invalid, and cannot be used, now continue the recovery". In more than a few cases, I suspect the recovery would fail, if for no other reason than it would take time that's not available to get the alternate version from secure storage, and then validate it.

I could probably come up with plenty of other ways to invalidate most exercises, and that's what the real concern is, not that someone has done a pre scripted test of the "standard" get out of jail recovery, but has anyone done a real recovery with unplanned wrinkles thrown in along the way that could invalidate the whole scenario.

That's also why the companies don't want to reveal their test plans, in that if someone wants to really screw them up, all they have to do is to make sure that something "simple" goes wrong with their recovery plan at just the right moment, and there are people out there that would love to do things like that.

Any more comfortable. Probably not, but for most of us, there's damn all we can do about it, and if we try, the pain of trying is way beyond comfort levels.