Ulster Bank Systems are down part 3 *READ* First post

A mere 2 odd weeks after this debacle began - an e-mail from UB :eek:

We are working hard to put things right

Many of our customers should see an improved position on their accounts over the next few days, with account transactions continuing to update as we work through the backlog. Once again we would like to apologise to our customers for the significant disruption this has caused, in the meantime, we continue to:

Provide access to cash through any Ulster Bank ATM. If the balance does not reflect up to date payments due into your account, including salary payments, please come to any Ulster Bank branch with your photo ID, account details and payslip.

Waive the cash advance fee and interest for Ulster Bank credit card holders who use their credit card to access cash during the current incident.

Treble our call centre staff to assist you with customer queries through our helpline on 1800 205 100 which is open from 8am until 10pm. If you are calling from abroad please call +353 9156 2910.

Open our Saturday opening branches today from 10am until 3pm and we will also open our main branches on Sunday from 10am until 1pm to help as many customers as possible. A full list of these branches is available on www.ulsterbank.ie/help.

If you are elderly or have a disability and are unable to access one of our branches, we would ask you to phone your nearest branch so that our staff can put an arrangement in place to support you.

We promise that no customer will be permanently out of pocket as a result of this problem.

Our staff are ready to help and answer any questions you may have.

Do they seriously think sending an e-mail at this late stage will restore confidence/goodwill ?

Long statement by Hester the CEO of RBS issued yesterday at 17.35 just after the stock markets close. It is in the form of a letter to the chairman of the UK Treasury Select Committee. Worth reading, in particular the paragraphs relating to "making sure our customers aren't out of pocket" and "compo for consequential loss"

http://www.rbs.com/news/2012/07/chief-executive-response-chairman-treasury-select-committee-update.html

So no iPads folks ! In a nutshell it looks like they will attempt to automatically reverse any charges and interest, and after that put you through every available customer service and complaints hoop before you get another red cent. 

The overall tone of the article is also very interesting. It continues the tone seen previously, bit of a glitch, nearly fixed, all's well, back to business as usual any day now. Oh and we're very sorry so we are. No realisation that they are responsible for the most catastrophic operational disaster in modern banking history. Have they any grasp of corporate governance whatsoever?

Just had my first communication (an e-mail) from UB since this started. That’s one group e mail in three and a bit weeks.


.

Full text of the statement/letter, link is in my previous post


Chief Executive response to chairman of the Treasury Select Committee - update
6th July 2012

Dear Andrew

Following my initial response to your letter of 27 June, I am now writing to address your additional questions, and welcome the further opportunity to update the Committee on the actions we are taking.

We remain deeply sorry for the disruption our customers encountered, and I would like to acknowledge their understanding and patience. The number one job for everyone at RBS as we resolve the incident is to put our customers’ interests first. As stated in my first response, the initial technical issue has been resolved in RBS and NatWest. There is continuing backlog in our service in Ulster Bank though this too is substantially improving each day. Our focus is now on completing the recovery in Ulster Bank, resolving the remaining issues faced by our customers, and on remediation and redress.

As you know a full and detailed investigation is underway into the causes of the problem, overseen by independent experts and reporting to the Board Risk Committee. It will consider both the Group’s own operations, as well as the role of third parties in the context of the incident. It will establish a full account of what happened, an assessment of how the Group responded and a thorough review of root cause. It would be wrong for me to second-guess its findings. Therefore, some of the responses below must be viewed as preliminary in nature.

Overview

The main RBS Group (RBS) brands affected have been NatWest, RBS and Ulster Bank Group (UBG) and the customers primarily affected are in the UK, Ireland and the Crown Dependencies. At UBG, progress in restoring normal services has been the slowest. We continue to work to accelerate the processing of the backlog in Ireland, which is reducing. We now expect that the coming week will be the final week of any significant delays for Ulster Bank customers, although there is likely to be some further tidy up work required thereafter. In the meantime, we continue to publish daily updates on our progress and maintain extended opening hours in our branches with additional resource in our call centres, to ensure we can help as many of our affected customers as possible.

The precise number and type of customers adversely affected and the losses they may have suffered are under urgent investigation. Detailed impact assessments have already started. These should give us a fuller picture of customer impact and help us implement customer redress.

Root cause and incident response

Preliminary investigations would indicate that the root cause was a routine software upgrade managed and operated by our team in Edinburgh, which caused the automated batch processing software to malfunction. The immediate software issue was promptly identified and rectified. Despite this, significant manual intervention in a highly automated and complex batch processing was required. This resulted in a significant backlog of daily data and information processing. The consequential IT problems and backlog have taken time to resolve. However, I would like to emphasise that at no point was any customer data lost or destroyed.

Everyone at RBS is deeply frustrated by the scale of disruption experienced by our customers and, especially the time being taken to restore normal service at Ulster Bank. The investigation will address the effectiveness of our risk management systems, including the identification of low probability/high impact events and their mitigation. It will also assess contingency planning and business resilience i.e. whether other systems within RBS Group are at risk of similar incidents. The strategic and operational response to the incident will also be assessed, as will the impact on business resilience of any cost saving measures. A substantial programme of activity will be implemented to reduce the potential of comparable incidents recurring. This programme will upgrade resilience, capability and governance in the relevant areas to the level that would be expected of us. It will be responsible for closing any identified gaps quickly and also for longer term upgrades to our capability.

Customers

My senior management team and I are completely focused on fixing the remaining processing difficulties we face and making good on our own customers issues and those of other banks affected by our systems failure. We have already managed to resolve a very large number of difficulties experienced by customers through our branches and call centres. There have been 21,000 other problems or complaints raised that required a greater degree of attention and we have so far resolved over 90% of these.

The day to day impact on our operations during the delay period included: reduced functionality of some back office systems; extended opening hours for branches and call centres; the implementation of manual "fixes" to deal with our customers' needs (e.g., increasing customer credit limits, providing emergency cash to customers etc.); and the re-allocation of resources from "business as usual" issues to fixing the problem and mitigating its impact on customers.

The incident did affect the Government Banking Service (GBS) system. Most of the impacts, such as the inability to view balances and delayed clearing of cheques and cash in and out during 20-22 June, were quickly resolved. BACS payments for GBS – the main part of our service contract - were actioned as normal throughout the period. There continue to be some delays, however, in some overseas payments to GBS via Ulster Bank.

As regards customer switching, whilst we experienced a decline in incoming switching volumes during the incident, these have now begun to recover. Outbound switching increased temporarily during the incident but has now returned to pre-incident levels.

Credit reports

I have already said that no customer should lose out because of our IT problems and this will of course include taking action to ensure the credit reference data of our customers are not unduly affected.

All June data for our customers who have had late incoming or internal payments to any of their accounts, will be reviewed for potential errors before being sent to the credit reference agencies (CRA). We are working with the CRAs and in turn the Information Commissioner Office (ICO) to agree an appropriate treatment for corrections made to these files. In addition, concerned consumer and commercial customers that want to check their credit reports will be provided with details of how to do this and given free access for a period of time.

We will also write to both consumer and commercial customers whose outgoing payments to third parties (e.g. other banks, utility companies etc.) have been delayed, to provide them with details as to how they can likewise check their credit reports, with free access for a period of time. For non-RBS customers whose expected payments have been delayed because of our processing issues (e.g. late salary payments), a process is being established to identify where issues may have occurred. Where our error has adversely affected any accounts held with another organisation we will provide an explanation of this for the other organisation so they can amend their records. The Group is working with the credit references agencies and their members to put in place an appropriate correction process.

Redress

As noted above, we stand by our commitment to ensure that if possible, affected customers are restored to the position they would have been in had the incident not occurred. We are also conscious that there are customers of other banks who have been affected by this incident. Since I last wrote, we have now finalised a Customer Redress Strategy outlining the approach to be taken for both RBS customers and customers of other banks.

The principles at the heart of this strategy are to put RBS and non-RBS affected customers back to the position they would have been in had the incident not occurred and to ensure consistent treatment of customers across the various customer segments and jurisdictions which have been affected – although some variations will be necessary depending on the precise customer impact.

The primary criteria for those eligible for redress are those RBS and non-RBS customers who have been adversely affected by the incident. Specific customer groups that may have been adversely affected will be identified to offer proactive redress wherever possible.

In addition to the proactive work, there are a range of types of consequential loss that may lead to compensation. We are assessing complaints/claims received from adversely affected customers through our customer complaints and redress processes, whose capacity we will expand as necessary to support the processing of such claims. Such claims will be handled on a case by case basis, as quickly as possible.

Our plans with respect to compensation for RBS customers include the following elements:

We will proactively refund (i) fees/charges incurred due to systems outage, including any interest on those charges and debtor interest, and (ii) any loss of credit interest due to delayed credits.
We will proactively identify cohorts of customers where we can reasonably assume the incident will have had an adverse impact, to contact them and offer suitable redress.
All claims for any additional losses in excess of the pro-active redress identified above will be dealt with through our standard complaints handling processes, with capacity expanded as necessary to support the prompt processing of such claims.
With respect to non-RBS customers, we have an agreement in principle with other banks, building societies, credit card companies and other financial institutions, via the Payments Council and Irish Payment Services Organisation, to ensure that any non-RBS customers impacted by this incident should not be left out of pocket. This will primarily involve:

The reversal / adjustment of debit interest and/or other account charges incurred as a result of the incident.
Those financial institutions dealing with any claims for reimbursement of other categories of loss through their own complaints handling processes.
Any redress and processing costs incurred by other financial institutions in respect of the incident will be reimbursed by RBS.
A dedicated unit is being established within RBS to provide a ‘direct-to-RBS’ route for exceptional non-RBS customer claims.
Conclusion

I am deeply sorry for the disruption experienced by our customers during the systems delays. We are completely focused on restoring normal levels of services, where these remain disrupted, and on re-gaining the confidence of our customers. We will also continue to explain what went wrong, what we are doing (or have done) to fix things and how we are putting our customers first. We are also determined to learn the lessons of this disruption and we will provide your Committee with the relevant findings from our investigation in due course.

Our staff continue to work incredibly hard during this episode and have dedicated themselves to fixing the problems and helping customers.

One thing is already evident. However radical our agenda was to change this bank in 2009 – and we have changed many things for the better – it is clear that we have more work to do. I am determined that we should drive further change to make sure this bank is better for its customers and becomes an appropriate model for how a bank should relate to society.

With best regards,

Stephen

Hibernicis wrote: »

Full text of the statement/letter, link is in my previous post


Chief Executive response to chairman of the Treasury Select Committee - update
6th July 2012

Root cause and incident response

Preliminary investigations would indicate that the root cause was a routine software upgrade managed and operated by our team in Edinburgh, which caused the automated batch processing software to malfunction.

Infrastructure managed from Edinburgh. Software upgrades and production support done in Chennai in India. Someone in India was tasked with backing out the upgrade, and wiped out the current schedule, in backing out the CA-7 Scheduler upgrade. India was then in charge of trying to run the overnight batch, when the overnight batch processing had malfunctioned. If this all occured in Edinburgh, why have RBS chiefs being having daily conference calls with India ? Hester is trying to be clever with the way he phrases things. The words "managed and operated by our team in Edinburgh" is carefully selected to keep India out of the picture

Geri Male wrote: »

Well that makes sense.

If you had a UB account and were due to be paid by the HSE on say 25 June then you wouldn't see that money until the batch transactions due for 25 June were processed (which ended up being 2/3 July)

So where's my(and other HSE employees)salary due on June 21st(and July 5th)???

filmbuffboy wrote: »

will disputing these transactions as duplicates be hard to do?

like, i have several 20 euro transactions on the same day when i know i only made one.

Very difficult to answer this from outside. If it is the case that whole batches have been duplicated, then it should be relatively simple to identify these batches and reverse them in their entirety. If however the batches have been partially duplicated, on top of all the other chaos, this will be require a transaction by transaction audit and reversal which is likely to be very considerably more complicated.

And I think you need to bear in mind that the UB systems and databases are now in anything but a normal state having been severly traumatised over the last few weeks with many things happening out of sequence, jobs failing and being re-run, manual interventions, the inevitable mistakes. So corrections that would in normal times be routine and risk free are likely to carry all sorts of complications and risks.

Shamrok wrote: »

Meh, I don't see how further concentrating the banking market here is a good thing.

Here, here. Ulster Bank are very much needed in Ireland. Its already very uncompetitive as we are. Just need to invest more cash in IT

Islander13 wrote: »

Here, here. Ulster Bank are very much needed in Ireland. Its already very uncompetitive as we are. Just need to invest more cash in IT

It's penny pinching and trying to save money in their IT that caused this problem. They sent all the IT work to India three years ago. Not only that but they reduced the number of production support staff from 60 to 30 in doing so. I'd say they'll stick a plaster on the open wound by bumping up their India support staff. They're too stupid to bring the IT work back to Ireland and the UK

My acc now up to date fully since lunchtime today

Sonnenblumen wrote: »

Reminds me of the unfortunate ostrich with its head stuck in deep sand? If the downturn/financial crisis has taught us anything, is that this country does not require any dysfunctional financial institution and certainly not one which is clearly in breach of fundamental industry regulations.


RBS has no interest in Irish retail banking, so get rid of their UB dysfuntional subsidiary and avoid spreading any idiotic stupidity.

Why anyone would want to be a UB customer especially after this catastrophe is baffling.

Well said my friend.