Advertisement
If you have a new account but are having problems posting or verifying your account, please email us on hello@boards.ie for help. Thanks :)
Hello all! Please ensure that you are posting a new thread or question in the appropriate forum. The Feedback forum is overwhelmed with questions that are having to be moved elsewhere. If you need help to verify your account contact hello@boards.ie

Ulster Bank Fiasco

  • 22-06-2012 6:09pm
    #1
    Closed Accounts Posts: 2,117 ✭✭✭


    What could have caused the disaster in the Ulster Bank systems? I've read reports of problems caused by a patch applied by NatWest to their group banking systems but there are also rumours of hacks and liquidity problems.

    More likely just poor testing and failover procedures?


«1

Comments

  • Moderators, Politics Moderators Posts: 38,406 Mod ✭✭✭✭Seth Brundle


    Possibly a case of "Hello, IT. Have you tried turning it off and on again?" staff!


  • Registered Users Posts: 579 ✭✭✭Mad Benny


    Was just about to post in here and ask the same question.

    If it was a patch then surely they could roll it back and then resubmit the transactions. I worked in banks for a few years and there is always very intensive testing.

    It's not hardware anyway. The banks have redundancy to cope with major events/disasters.


  • Closed Accounts Posts: 2,497 ✭✭✭omahaid


    Somebody forgot where they put the back up tapes? :D


  • Registered Users Posts: 579 ✭✭✭Mad Benny


    I'm amazed that the real story hasn't leaked out. It's very odd. I would prefer to hear that they messed up with a software change and didn't test it properly.

    I'm not surprised that there is rumour of a hack.


  • Registered Users Posts: 167 ✭✭Saggitarius


    kbannon wrote: »
    Possibly a case of "Hello, IT. Have you tried turning it off and on again?" staff!

    You have partially right only, but a big system, like that isn't simply just shut down and restart because of complexity and loads of servers work together as a redundant system. That means, if any unit(or units) failing, you can replace them without any glitch, because of faulty unit role automatically replaced by or shifted to an another, in other words there is more option for 24/7 continuous safe work. I'm not really believe that the full system just failing, I mean this big black-out can caused by a full server room fire or something like that. Until that this kind a system can "self healing" up to a certain level with a slow-down (but not this kinda slow-down)


  • Advertisement
  • Registered Users Posts: 167 ✭✭Saggitarius


    Mad Benny wrote: »
    Was just about to post in here and ask the same question.

    If it was a patch then surely they could roll it back and then resubmit the transactions. I worked in banks for a few years and there is always very intensive testing.

    It's not hardware anyway. The banks have redundancy to cope with major events/disasters.

    Yeah, and for this reason they should have special IT guys with disaster recovery skill.:rolleyes:


  • Moderators, Business & Finance Moderators Posts: 9,921 Mod ✭✭✭✭Jim2007


    Mad Benny wrote: »
    Was just about to post in here and ask the same question.

    If it was a patch then surely they could roll it back and then resubmit the transactions. I worked in banks for a few years and there is always very intensive testing.

    It's not hardware anyway. The banks have redundancy to cope with major events/disasters.

    Well after say 5 years of cut backs, upgrade postponements and so on, you eventually get to something like this! I fully expect well see more of this kind of thing in the coming months/years...


  • Registered Users Posts: 9,218 ✭✭✭markpb


    My gut feeling is a dodgy upgrade to a database which would be almost impossible to roll back. The delay processing batches is probably deliberate while they try to fix both the software problem and the dodgy data problem - there's no point having more transactions flooding in if you know they'll be handled incorrectly too.

    I'd imagine having to keep most of the system live while fixing the broken bits is complicating it even further but it's unavoidable in their business.


  • Registered Users Posts: 23,212 ✭✭✭✭Tom Dunne


    markpb wrote: »
    My gut feeling is a dodgy upgrade to a database which would be almost impossible to roll back.

    I am not so sure about that. In all the years I spent upgrading mission critical databases (both the actual schemas, and the RDBMS software itself), there has always been a backout.

    In my experience, especially in 24x7, large heterogeneous systems as they would have in a bank, problems like this are usually caused by an upgrade/human intervention. Things don't just break.


  • Moderators, Society & Culture Moderators Posts: 9,689 Mod ✭✭✭✭stevenmu


    It does appear to have been a software update gone awry. From talking to people familiar with similar systems (altough not the RBS ones in particular), it looks like updates would have been applied across a number of their backend systems. This would include the systems and data stores responsible for maintaining the actual accounts,fund transfers,encryption key management etc. These would be a number of heterogeneous systems, mainframes, AS400s based on pretty old software platforms. Not at all like the windows/Linux systems running SQL server or MySQL that most of use are used to where its easy to just copy a few dll/jar files, run a few SQL statements and do a few reboots to see if it works.

    The people I've spoken to think its quite plausible to take a couple of days to roll back the updates.

    It's also quite possible that the system are already fixed but there are a ton of transactions that need to be reentered to make sure everything is consistent.


  • Advertisement
  • Registered Users Posts: 1,533 ✭✭✭Colonialboy


    outsourcing.....


  • Registered Users Posts: 5,246 ✭✭✭conor.hogan.2


    While Sonys fiasco was hugely different they both share one thing, they both had recently cut back on IT staff.


  • Registered Users Posts: 9,153 ✭✭✭everdead.ie


    Hard to believe they actually cut that many IT staff and didn't expect some kind of draw back.

    This situation has all the Hall marks of an Oracl DB upgrade from 10G to 11G well known to cause issues as there is a lot of difference between the two.


  • Registered Users Posts: 1,533 ✭✭✭Colonialboy


    I think one poster on the register put it best.

    There are two trains of thought.
    Train A : Our IT systems work great because we have a 1000 well paid experienced people
    Train B : Our IT systems work well, why do we need a 1000 well paid experienced people.
    Unfortunately there is standing room only in the carriages on Train B and Train A has been replaced with a bus service.

    This whole fiasco does everyone a disservice , the RBS experienced people, the technology, the vendors, even the workers in india.

    Its obvious to me that the RBS bankers who have already proven they couldnt manage running the financial side of the banks buisness when told to start running a sound buisness, turned their attention to cost savings and have now demonstrated they cant manage the IT side of a business either.

    Pity the head of the Computer Science dept in one of Irelands top universitys didnt listen to me a while back when I tolfd him there was an opening to train up Irish graduates in mainframe skills to replace the workforce and to take over the millions of lines of code and major systems still running mainframes and advanced mainframe technologies.

    btw its wasnt Oracle, its wasnt even a db problem, it was a bad 'job scheduler' upgrade they waited too long to back out.


  • Registered Users Posts: 9,153 ✭✭✭everdead.ie


    I think one poster on the register put it best.

    There are two trains of thought.
    Train A : Our IT systems work great because we have a 1000 well paid experienced people
    Train B : Our IT systems work well, why do we need a 1000 well paid experienced people.
    Unfortunately there is standing room only in the carriages on Train B and Train A has been replaced with a bus service.

    This whole fiasco does everyone a disservice , the RBS experienced people, the technology, the vendors, even the workers in india.

    Its obvious to me that the RBS bankers who have already proven they couldnt manage running the financial side of the banks buisness when told to start running a sound buisness, turned their attention to cost savings and have now demonstrated they cant manage the IT side of a business either.

    Pity the head of the Computer Science dept in one of Irelands top universitys didnt listen to me a while back when I tolfd him there was an opening to train up Irish graduates in mainframe skills to replace the workforce and to take over the millions of lines of code and major systems still running mainframes and advanced mainframe technologies.

    btw its wasnt Oracle, its wasnt even a db problem, it was a bad 'job scheduler' upgrade they waited too long to back out.
    I have yet to see a business that see technology as a business enabler and not a cost.


  • Closed Accounts Posts: 899 ✭✭✭djk1000


    I have yet to see a business that see technology as a business enabler and not a cost.

    Too true.


  • Closed Accounts Posts: 2,828 ✭✭✭Reamer Fanny


    I reckon they were running Windows NT


  • Closed Accounts Posts: 5,430 ✭✭✭testicle


    justryan wrote: »
    I reckon they were running Windows NT

    Incorrect. IBM zSeries Mainframe running CA-7


  • Closed Accounts Posts: 2,828 ✭✭✭Reamer Fanny


    Smarty pants :p


  • Advertisement
  • Registered Users Posts: 1,533 ✭✭✭Colonialboy


    testicle wrote: »
    Incorrect. IBM zSeries Mainframe running CA-7
    correct... and just in case people think this is old technology cos it mentions the 'M' word ... these machiines are the most advanced out there in terms of high-street computing and demand for them is growing not shrinking

    but like all technology its only as good as the people you outsource to.

    http://www-03.ibm.com/press/us/en/pressrelease/32414.wss
    This world record-breaking speed is necessary for businesses managing huge workloads, such as banks and retailers, especially as the world becomes increasingly more inter-connected, data has grown beyond the world's storage capacity, and business transactions continue to skyrocket. The new zEnterprise technology is the result of an investment of more than $1.5 billion in IBM research and development in the zEnterprise line, as well as more than three years of collaboration with some of IBM's top clients around the world.

    Revenues from System z mainframe server products increased 41 percent compared with the year-ago period.
    Total delivery of System z computing power, as measured in MIPS (millions of instructions per second), increased 34 percent.
    IBM REPORTS 2011 FIRST-QUARTER RESULTS
    http://www.ibm.com/investor/1q11/press.phtml


  • Moderators, Sports Moderators, Regional Abroad Moderators Posts: 2,634 Mod ✭✭✭✭TrueDub


    It's been confirmed (well, on the Register anyway), that it was a major CA7 blunder that caused the outage.

    http://www.theregister.co.uk/2012/06/26/rbs_natwest_ca_technologies_outsourcing/

    I can vouch for the fact that batch schedules are immensely complex and would be difficult to recreate from scratch. I would have expected them to have it backed up though.


  • Registered Users Posts: 2,144 ✭✭✭dazberry


    justryan wrote: »
    I reckon they were running Windows NT

    I worked in an RBS company from 2004 to 2011, we used NT until 2007, whereby we were migrated to.... Windows.... Vista. In IBM parlance we were blue rinsed.

    D.


  • Registered Users Posts: 45 irishd


    My understanding is that they switched to FINO scheduling, using temporary job caching based on Signetics 25120 Memory. This was backed up onto S4 storage for DR.

    Been known to cause data retrieval issues in the past :D


  • Moderators, Sports Moderators, Regional Abroad Moderators Posts: 2,634 Mod ✭✭✭✭TrueDub


    irishd wrote: »
    My understanding is that they switched to FINO scheduling, using temporary job caching based on Signetics 25120 Memory. This was backed up onto S4 storage for DR.

    Been known to cause data retrieval issues in the past :D

    Had to google it, but I get it now! :D

    I was in IT in RBS until 2010, our network was still NT. However the mainframe processing, which is what runs the heart of the bank, was bang up to date and well-maintained. Things have changed.


  • Moderators, Society & Culture Moderators Posts: 9,689 Mod ✭✭✭✭stevenmu


    correct... and just in case people think this is old technology cos it mentions the 'M' word ... these machiines are the most advanced out there in terms of high-street computing and demand for them is growing not shrinking

    They might be very powerful, but do they support modern development methods such as:

    - "sh1t, it's not working, can we just copy the old DLL back again?"

    or

    - "crap, that didn't do it, quick do a reboot"

    and not forgetting the always reliable

    - "fcuk it, it's not audited anyway, just blame the guys the india"


  • Moderators, Sports Moderators, Regional Abroad Moderators Posts: 2,634 Mod ✭✭✭✭TrueDub


    stevenmu wrote: »
    They might be very powerful, but do they support modern development methods such as:

    - "sh1t, it's not working, can we just copy the old DLL back again?"

    Yes, in their own fashion. It's called a backout.
    stevenmu wrote: »
    or

    - "crap, that didn't do it, quick do a reboot"

    Less easy in a system designed for 100% uptime. It's called an IPL, and you don't do it lightly.
    stevenmu wrote: »
    and not forgetting the always reliable

    - "fcuk it, it's not audited anyway, just blame the guys the india"

    Blaming the offshore team is a regular way out, but when you've only got an offshore team...


  • Registered Users Posts: 2,164 ✭✭✭hobochris


    I heard rumours it was an untested patch that bit them in the ass. :D


  • Moderators, Sports Moderators, Regional Abroad Moderators Posts: 2,634 Mod ✭✭✭✭TrueDub


    hobochris wrote: »
    I heard rumours it was an untested patch that bit them in the ass. :D

    lots of rumours flying - the most believable one I've heard consists of a failed backout of a change which wiped their batch schedule.


  • Advertisement
  • Closed Accounts Posts: 2,930 ✭✭✭COYW


    Pity the head of the Computer Science dept in one of Irelands top universitys didnt listen to me a while back when I tolfd him there was an opening to train up Irish graduates in mainframe skills to replace the workforce and to take over the millions of lines of code and major systems still running mainframes and advanced mainframe technologies.

    I have just finished up in a company which is heavily reliant on mainframe technologies. There wasn't one intermediate or senior person under the age of 40 working on that side of things. The major players were 45 year old plus who haven't updated their skills in decades, were getting paid a fortune and were passing on their same tired knowledge to the younger members coming along. I made some suggestions on improving coding and was royally shot down for doing so. It is an area with massive potential for change. If I was in uni now, I would focus on the area.
    Yes, in their own fashion. It's called a backout.

    Do most places not require you to do a backout plan and backout test plan before getting approval for changes to a live system. I thought this practice was standard in the corporate world for IT rollouts.


Advertisement