Government IT projects that went wrong

The_Minister

I was wondering about this after having a conversation with someone about this.

We both agreed that the government was useless at IT, using PPARS as the example (over €100,000,000 wasted IMO) and we remembered that there was something about the FAS website, but when I Googled just now nothing came up.

Have I been really unfair on the government, and actually Ireland has only had one mess up with IT or are there other that I've forgotten?

P. Breathnach

I don't think that you can generalise so simply as to say that the "government" is useless at IT.

Yes, PPARS was a huge mess, and the PULSE system used by the Garda also has had a lot of problems. And these are big projects, involving big money, so they matter.

On the other hand, the Revenue Commissioners seem to have one of the best IT setups anywhere, and very many other public-sector IT services perform sweetly.

It's a mixed bag.

youcancallmeal

Sure aren't all these IT projects outsourced to so called experts like Accenture and Deloitte who then in turn hire somebody else who then in turn hire somebody else etc etc. All the government do is put these projects out to tender and then pick someone so all you can really argue is that they didn't pick the right people? The whole PPARS fiasco was a disgrace though

jmayo

P. Breathnach wrote: »

I don't think that you can generalise so simply as to say that the "government" is useless at IT.

Yes, PPARS was a huge mess, and the PULSE system used by the Garda also has had a lot of problems. And these are big projects, involving big money, so they matter.

On the other hand, the Revenue Commissioners seem to have one of the best IT setups anywhere, and very many other public-sector IT services perform sweetly.

It's a mixed bag.

They were two very expensive cockups that would not be stomached in a private sector company.

And yes we know that some projects have been cockups in private sector companies, but the difference is they pull the plug when it is going chronically over budget & overtime and then someone can be canned.
What happened with PPARS was grand larceny of taxpayers money and AFAIK no one got their ar**s kicked.

How a project could go on for years costing over a hundred million and still not deliver and then we had an explanation from the CEO, in this case taoiseach, that it would only take another 20 odd million to get it right.

BTW people must have short memories when they don't recall e-voting, as in essense it was another IT project failure, albeit due to political interference me thinks :rolleyes:

Scofflaw

As far as I recall, the Forestry Service built (at the cost of several millions) an expert system, just before they decentralised to Wexford and lost all the staff who understood how to use it.

Not strictly an IT issue, except in the sense that most "IT failures" occur at the interface of the system and the business.

cordially,
Scofflaw

jmayo

youcancallme>al wrote: »

Sure aren't all these IT projects outsourced to so called experts like Accenture and Deloitte who then in turn hire somebody else who then in turn hire somebody else etc etc. All the government do is put these projects out to tender and then pick someone so all you can really argue is that they didn't pick the right people? The whole PPARS fiasco was a disgrace though

It doesn't matter who they were outsourced to, as it is up to the project management team within the purchaser to ensure they are getting bang for their buck.
In the case of PPARS no one within Dept of Health bothered their ar** to shout stop and demand the contractors deliver.

Porjects normally have defined goals, defined timeframes, defined budgets and if the providers fail to deliver the latter two they are penalised not effing rewarded by continous payment.

Private sector entities do the same hiring in contractors and consultants with the requsite experience and know how, but they don't allow projects drag on for years, all the while costing millions and still getting no working product.

OhNoYouDidn't

jmayo wrote: »

They were two very expensive cockups that would not be stomached in a private sector company.

Which begs the question why private companies in the form of consultants propose these systems to the public sector.

If anything thats an argument to allow the public sector get on with it without these parasitic consultants ballsing it up on them.

OhNoYouDidn't

jmayo wrote: »

Private sector entities do the same hiring in contractors and consultants with the requsite experience and know how, but they don't allow projects drag on for years, all the while costing millions and still getting no working product.

Thats actually not true. The difference is private companies can hush op their mistakes, public bodies cannot.

Scofflaw

jmayo wrote:

Private sector entities do the same hiring in contractors and consultants with the requsite experience and know how, but they don't allow projects drag on for years, all the while costing millions and still getting no working product.

That's so untrue as to have made me laugh! I can cite you the case of a knowledge management system built for a major Irish legal firm with a budget of, if I recall correctly, about €50m. They'd spent the whole €50m and a little bit more besides building the system - and then they came to the point where they needed to get the knowledge into the system, which involved getting the solicitors and lawyers to review their old cases. At that point, they discovered something they should have known all along - the legal "employees" are actually partners, and charge the firm for their time. The estimated cost of getting the knowledge into the €50m system was about €200m. Project abandoned.

And the things that go on in major retailers, insurance firms, finance houses...most private companies only don't waste millions because they're small, so they only waste thousands - and for someone to honestly believe that private companies simply don't make this kind of mistake! God bless your innocence and preserve it!

deeply amused,
Scofflaw

OhNoYouDidn't

Scofflaw wrote: »

That's so untrue as to have made me laugh! I can cite you the case of a knowledge management system built for a major Irish legal firm with a budget of, if I recall correctly, about €50m. They'd spent the whole €50m and a little bit more besides building the system - and then they came to the point where they needed to get the knowledge into the system, which involved getting the solicitors and lawyers to review their old cases. At that point, they discovered something they should have known all along - the legal "employees" are actually partners, and charge the firm for their time. The estimated cost of getting the knowledge into the €50m system was about €200m. Project abandoned.

And the things that go on in major retailers, insurance firms, finance houses...most private companies only don't waste millions because they're small, so they only waste thousands - and for someone to honestly believe that private companies simply don't make this kind of mistake! God bless your innocence and preserve it!

deeply amused,
Scofflaw

Could give you 100 similar examples in banks.

The only difference is as you say scale and the ability to bury the mistakes.

paddyland

I have no doubt that mistakes are made on a grand scale on every level, private and public. That's how we learn, after all.

I do object, however, when PUBLIC money is wasted on daft schemes that have no mandate while that money is required desperately for grossly underfunded public services.

If the government want to fund extravagant experiments and flag flying exercises, why don't they petition the public for extra funding specifically for that, and see how many rush to take them up on it?

dvpower

paddyland wrote: »

I have no doubt that mistakes are made on a grand scale on every level, private and public. That's how we learn, after all.

I do object, however, when PUBLIC money is wasted on daft schemes that have no mandate while that money is required desperately for grossly underfunded public services.

If the government want to fund extravagant experiments and flag flying exercises, why don't they petition the public for extra funding specifically for that, and see how many rush to take them up on it?

That's exactly what we need... more bureaucracy

Scofflaw

paddyland wrote: »

I have no doubt that mistakes are made on a grand scale on every level, private and public. That's how we learn, after all.

I do object, however, when PUBLIC money is wasted on daft schemes that have no mandate while that money is required desperately for grossly underfunded public services.

If the government want to fund extravagant experiments and flag flying exercises, why don't they petition the public for extra funding specifically for that, and see how many rush to take them up on it?

To be fair, though, the idea behind most of these projects is to offer a better service or to do it more cheaply or more quickly. They're not generally done just for fun.

It's probably also worth pointing out that most of these projects involve private sector contractors. The in-house projects that I know of tended to go better.

cordially,
Scofflaw

OhNoYouDidn't

paddyland wrote: »

I have no doubt that mistakes are made on a grand scale on every level, private and public. That's how we learn, after all.

I do object, however, when PUBLIC money is wasted on daft schemes that have no mandate while that money is required desperately for grossly underfunded public services.

If the government want to fund extravagant experiments and flag flying exercises, why don't they petition the public for extra funding specifically for that, and see how many rush to take them up on it?

The intent is not to waste money, its to improve things. The mandate is most certanly there. Unfortunatly in Ireland project management as a discipline is quite poor, but the point is this is across the board.

mickoneill30

I used to work for one of the big IT consulting companies. All the projects we wanted were put out to tender. We didn't get some of them because we'd bid a proper price to do the job. Even with costs pared back as much as we could our company often lost out to another major company (that has a bad reputation in the industry, they're cheap but crap).

Do the public departments have to go with the cheapest bidder? There was one project we bid on that was a huge mess by the time the winning bidder was finished with it.

jmayo

OhNoYouDidn't wrote: »

Which begs the question why private companies in the form of consultants propose these systems to the public sector.

If anything thats an argument to allow the public sector get on with it without these parasitic consultants ballsing it up on them.

The private companies do not propose these projects. :rolleyes:
It is someone within the public sector, as you admit later in another post, that come up with ideas for the projects and they are most likely good valid ideas.
It is the execution that is cock eyed and cack handled.
Yes the parasitic consultants keep things going, but they know they are onto a good thing when no one in the public sector, the actual project executive and end user staekholders shouts stop and enough is enough.

OhNoYouDidn't wrote: »

Thats actually not true. The difference is private companies can hush op their mistakes, public bodies cannot.

Exccept somewhere along the line someone is usually responsible and it looks damm dodgy in year end accounts that project x that is still ongoing cost y millions over budget.

Of course you keep telling yourself that proivate companies, especially smaller ones, are still in business even though they have had disasterous project delivery that they have well hidden.

Scofflaw wrote: »

That's so untrue as to have made me laugh! I can cite you the case of a knowledge management system built for a major Irish legal firm with a budget of, if I recall correctly, about €50m. They'd spent the whole €50m and a little bit more besides building the system - and then they came to the point where they needed to get the knowledge into the system, which involved getting the solicitors and lawyers to review their old cases. At that point, they discovered something they should have known all along - the legal "employees" are actually partners, and charge the firm for their time. The estimated cost of getting the knowledge into the €50m system was about €200m. Project abandoned.

That is my f***ing point, they called a halt rather than keep bloody well going no matter what.
The project was not well planned from the start as you admit, becasue they never envisaged the cost of inputting the required information.

BTW what was the original budget for the development of the project ?

IMHO it sounds like one very expensive system from the start.

Scofflaw wrote: »

And the things that go on in major retailers, insurance firms, finance houses...most private companies only don't waste millions because they're small, so they only waste thousands - and for someone to honestly believe that private companies simply don't make this kind of mistake! God bless your innocence and preserve it!

deeply amused,
Scofflaw

I know damm well that there are project cockups in private sector companies.
I have seen them, badly planned projects, poor project management, people trying to change goalposts, delays, etc.
I have seen mid size company drop a SAP project because after 5 million down the swany they reckoned is what time to cut and run.

Difference is it would never get so out of hand as in case of PPARS and at the end of the day someone would get their ar** kicked.

BTW I am so happy that I can amuse you. It makes my day :rolleyes:

The_Minister

Was the FAS website overpaid for?
Or was it another website they overpaid for?

Nodin

The_Minister wrote: »

We both agreed that the government was useless at IT, using PPARS as the example (over €100,000,000 wasted IMO) and we remembered that there was something about the FAS website, but when I Googled just now nothing came up.

FAS, which employs 2,200 people, has been rocked by the disclosure of wastage such as the establishment of a €1.7m Jobs Ireland website at a time when FAS had its own jobs website. It ultimately had to be scrapped at a cost of €1m.

http://www.independent.ie/national-news/fas-seeks-outside-auditors-in-wake-of-expenses-row-1859853.html

The_Minister

Nodin wrote: »

http://www.independent.ie/national-news/fas-seeks-outside-auditors-in-wake-of-expenses-row-1859853.html

€1million! Thats worse than I remembered.

The Corinthian

Scofflaw wrote: »

That's so untrue as to have made me laugh! I can cite you the case of a knowledge management system built for a major Irish legal firm with a budget of, if I recall correctly, about €50m. They'd spent the whole €50m and a little bit more besides building the system - and then they came to the point where they needed to get the knowledge into the system, which involved getting the solicitors and lawyers to review their old cases. At that point, they discovered something they should have known all along - the legal "employees" are actually partners, and charge the firm for their time. The estimated cost of getting the knowledge into the €50m system was about €200m. Project abandoned.

I would be horrified if this issue was not raised by a BA at the design / requirements stage of the project.

Scofflaw

mickoneill30 wrote: »

I used to work for one of the big IT consulting companies. All the projects we wanted were put out to tender. We didn't get some of them because we'd bid a proper price to do the job. Even with costs pared back as much as we could our company often lost out to another major company (that has a bad reputation in the industry, they're cheap but crap).

Do the public departments have to go with the cheapest bidder? There was one project we bid on that was a huge mess by the time the winning bidder was finished with it.

There's usually a set of criteria, but price is often the most important. Unfortunately, the requirement for 'transparency and accountability' in the tendering process has had the side-effect of making it rigid and often sub-optimal.

cordially,
Scofflaw

Scofflaw

The Corinthian wrote: »

I would be horrified if this issue was not raised by a BA at the design / requirements stage of the project.

To my knowledge it wasn't, because the project originated with the IT people, and they didn't get the appropriate stakeholders on board.

cordially,
Scofflaw

jimmycrackcorm

As long as these IT projects are awarded to the likes of Accenture then PPARS will be repeated over and over. They have a complete consultancy culture that is world apart from software product development.

Much of the blame though has to lie with the public service that is procuring and managing these projects as there is no accountability.

Einhard

I think it's unfair to focus on the public sector in this regard. As other posters have pointed out, private firms make similar cock ups all the time, but they are rarely reported on. I remember during the whole PPARS imbroglio there was a parallel story involving the Credit Union, and an IT system that had run massively over budget and still didn't provide the expected return. I googled it and came up with this story from 2000: http://www.independent.ie/national-news/rift-grows-over-credit--union-pound25m-overhaul-363025.html. It would appear though, to be too early to have coincided with PPARS, and so it's possible that there was another such costly fiasco involving the CU later on in the decade. Either way, whether there were just the one failure or two, it dispels the notion that only state agencies make such costly and embarassing mistakes.

spider guardian

What about the Dublin transport smart card? 9 years late and a €55 million cost. I would dearly love to know why there was such delays and such a huge bill, how difficult can it be to program a back end system and deploy terminals across the board? The core system could not be that complex, i would imagine that scale would be the biggest challenge.

I would have expected more from a company like IBM but they must've seen an opportunity to take the government for a ride. Joe Public simply shrugs his shoulders, well used to this type of carry on.

OhNoYouDidn't

jimmycrackcorm wrote: »

As long as these IT projects are awarded to the likes of Accenture then PPARS will be repeated over and over. They have a complete consultancy culture that is world apart from software product development.

Much of the blame though has to lie with the public service that is procuring and managing these projects as there is no accountability.

So how do we even know about them then?

LookBehindYou

Has anyone been sacked over the waste ? or have they been promoted ?

Scofflaw

LookBehindYou wrote: »

Has anyone been sacked over the waste ? or have they been promoted ?

In none of the failed projects I can think of was anyone sacked, either in the public or the private sector. Sacking would mean that everyone accepted publicly that the project was a failure, and anyone with an ounce of sense gets buy-in from enough senior people to ensure that that doesn't happen.

cordially,
Scofflaw

Einhard

OhNoYouDidn't wrote: »

So how do we even know about them then?

There's a big difference between knowledge being made public, and someone actually being held to account for their (in)actions.

OhNoYouDidn't

Einhard wrote: »

There's a big difference between knowledge being made public, and someone actually being held to account for their (in)actions.

Agreed, but there is transparancy there, if not the level of accountability we would prefer. See Scofflaw's post on why.

But also, if every public servant knew they would be fired if a project overspent, they wouldn't take any on. At all. Its a fine line between accountability and a lynchmob.

thebman

I think it fundamentally comes down to Irish business both private and public not giving project management the importance it deserves and not co-ordinating across their businesses effectively.

I'm sure its the same in many other countries too but I can see it here quite often in companies with these failures that proper management would have stopped the problems.

Unqualified or ill qualified people given the job because of who they know managing projects.

Einhard

OhNoYouDidn't wrote: »

But also, if every public servant knew they would be fired if a project overspent, they wouldn't take any on. At all. Its a fine line between accountability and a lynchmob.

Yes but we could at least make an attempt to thread that fine line.

Nijmegen

If we talk about IT projects alone - accepting, perhaps, my view that we have a state wide culture of poor management controls at the centre of much of our problems - then I would say this:

An IT project requires very good planning to pull off successfully. You need to have a clear idea of what you want, to define this at an early stage and to lock it down in place when you bring in the right crowd to fulfill the order.

In Ireland we usually have poorly defined projects set by people who do not really understand what they are trying to do or how they may accomplish this, and the death-knell to all IT projects is introduced - feature creep. New features, modifications, the lot, tend to kill or run over budget projects and produce substandard results.

For a successful IT project you need clear and detailed planning outlining every nut and bolt and what happens and how piece A affects piece D and what happens to pieces X, Y and Z when scenario 1 or 2 occurs.

The Irish government, and most organisations in Ireland private and public, lack the levels of organisational control and excellence to pull this sort of thing off.

The easiest thing in the world is to say "Okay, I want a website where Irish citizens can get all the information they need about interacting with the state, social welfare, everything, personalised to them." Figuring out how that works is a NASA level project. But hey, NASA does it all the time. So too HP. Microsoft. Etc etc.

Revenue has always been the best and most efficient branch of the state, some might observe for cynical reasons as to why.

OhNoYouDidn't

Einhard wrote: »

Yes but we could at least make an attempt to thread that fine line.

I'm not sure is naming and shaming mid level civil servants in the IT departments of various government departments going to achieve a whole lot. If the bullet was in the post because through no fault of your own a project assigned to you was open to political/management interference or scope creep, why in the name of thundering jaysis would you take one on. Or any of your line managers? Thats in the public and private sphere.

The issue, yet again, is at senior management level. As it always is in Ireland.

Liberalbrehon

IT project failure isn't just an Irish issue, it's across the world. The problem is senior management/politicans thinking you can just automate a manual process that deals with lots of exceptions, domain knowledge transferred people to people over years, ad-hoc rules, undefined rules in the manual process that can just get buried. The best systems work when creating a new process from nothing, attempting to automate an existing system is horrible. Defining those existing requirements is a minefield and something is always not thought off, usually the non-technical stuff that is left till last, instead of it being a pre-requiste. Communication issues, politics, technical challenges, change mgmt, etc. I'm not surprised €100m was wasted. It was a hugely ambitious project.

The other problem is the consultancy firms. They operate in a strange world of cost accounting. Everything has to be billed to the hour. People with no passion or interest in the project are contracted in. Team efficiencies are rare, due to people moving all the time. The contract rates are horrendously high. The IT big consultancies really know how to charge rates. At the end of the day the people tendering and procuring for IT systems haven't the foggiest what they are tendering for. They don't know what criteria to use so just go for price. Consultancy firms like to deliver big bang but it hardly ever works. Small iterations that expand slowly is the best approach.

So don't expect this issue to go away. I have a lot of experience in this field.

kmick

Having done some work as a technical consultant in one of the better government projects I would say the problems are
1. No one buys off the shelf products. Everyone thinks they need bespoke software. This is the first and most deadly mistake you can make.
2. No one says I want a billing system and my budget is 500k and thats it. If after 500k my billing system does not work you will lose the contract and your reputation.
3. Most government IT departments do not have people with the skills to design, build and deploy a semi complex software application. This could be said of a lot of companies so its common.

Anecdote - The reason the PULSE system was considered a failure was that to save costs it ran over the lotto network (unbelieveable yes). When the lotto line was busy it was taking 30 minutes to log in to the system. Therefore the Gardai were very reluctant to use it.

Sizzler

Could give countless examples of wasteful project spend but even the basic opex costs are being pi**ed up against the wall.

A govt agency ordered a mirrored frame relay network (not essential infrastructure) with a certain telco even though it wasnt required, why? "Because if we dont spend the budget it will be taken off us next year"

Thats taxpayers cash being absolutely WASTED :mad:

kennethsmyth

With regards to the ppars systems and its failures, something that I've seen but not on as big a scale is that you can't always blame the IT system. If the physical types of data to be input / worked on all use different methods and systems and you are asked to merge into one system you must firstly ask for changes to the physical system as too many methods destroy the fundamental idea of one IT system

Eg there was many different pay systems, pay methods, pay formulas and one size cannot fit all. You must consolidate first and simplfy. Then you can automate.;)

Scofflaw

Liberalbrehon wrote: »

IT project failure isn't just an Irish issue, it's across the world. The problem is senior management/politicans thinking you can just automate a manual process that deals with lots of exceptions, domain knowledge transferred people to people over years, ad-hoc rules, undefined rules in the manual process that can just get buried. The best systems work when creating a new process from nothing, attempting to automate an existing system is horrible. Defining those existing requirements is a minefield and something is always not thought off, usually the non-technical stuff that is left till last, instead of it being a pre-requiste. Communication issues, politics, technical challenges, change mgmt, etc. I'm not surprised €100m was wasted. It was a hugely ambitious project.

The other problem is the consultancy firms. They operate in a strange world of cost accounting. Everything has to be billed to the hour. People with no passion or interest in the project are contracted in. Team efficiencies are rare, due to people moving all the time. The contract rates are horrendously high. The IT big consultancies really know how to charge rates. At the end of the day the people tendering and procuring for IT systems haven't the foggiest what they are tendering for. They don't know what criteria to use so just go for price. Consultancy firms like to deliver big bang but it hardly ever works. Small iterations that expand slowly is the best approach.

So don't expect this issue to go away. I have a lot of experience in this field.

That's well put. Unfortunately, it's really hard to get through the tender system when what you're saying is "I'm not sure what you've put in your scoping document is either what you want or likely to work, but I can see what you probably need in there. How about we start with a basic system and then iterate once people are happy?".

That is, on the other hand, why small company projects are successful - they initially do the minimum for a usually obvious and measurable gain, iterate on top of success, and are small enough not to run into complexity issues, plus everybody who has a stake is usually easy to assemble, and the money is something that can be argued about as the project evolves. Unfortunately, you can't easily replicate that in a large organisation with hundreds or thousands of users.

cordially,
Scofflaw

cyclopath2001

kmick wrote: »

3. Most government IT departments do not have people with the skills to design, build and deploy a semi complex software application.

Didn't they try to move all the IT departments out of Dublin back in 2003?

RATM

kmick wrote: »

Anecdote - The reason the PULSE system was considered a failure was that to save costs it ran over the lotto network (unbelieveable yes). When the lotto line was busy it was taking 30 minutes to log in to the system. Therefore the Gardai were very reluctant to use it.

haha you couldn't make that up. Gardai couldn't access the Pulse system every Wednesday and Saturday from 6-7.30pm because the nation was busy gambling on the Lotto

kmick

cyclopath2001 wrote: »

Didn't they try to move all the IT departments out of Dublin back in 2003?

They were going to but it was going to be so complex and expensive that in reality it was never going to happen.

Liberalbrehon

This is probably one area where private business and government are equally poor. After saying that, I'd like to get the figures on projects that went well, just to put it into perspective. how much is spent on IT systems annually, maintenance, new projects etc. How much is paid in license fees, etc It's too easier to make headlines with failures but its important to put into context.
Is failure rate 5%, 25% 75%? A 5% failure rate would probably be acceptable/

P. Breathnach

Liberalbrehon wrote: »

... Is failure rate 5%, 25% 75%? A 5% failure rate would probably be acceptable/

I have no idea what the failure rate is, so I can't help you on advancing a case.

There is the question of what constitutes failure. My understanding is that a lot of the PPARs package is usable, and used, but that, taken as a whole, the project was a disaster. I also understand that PULSE was a "less bad" project, and mostly remediable. How do you put a numerical value on things like that?

Scofflaw

Liberalbrehon wrote: »

This is probably one area where private business and government are equally poor. After saying that, I'd like to get the figures on projects that went well, just to put it into perspective. how much is spent on IT systems annually, maintenance, new projects etc. How much is paid in license fees, etc It's too easier to make headlines with failures but its important to put into context.
Is failure rate 5%, 25% 75%? A 5% failure rate would probably be acceptable/

Hey! We could build a PS-wide IT system to track such things...

brightly,
Scofflaw

The Corinthian

Scofflaw wrote: »

Hey! We could build a PS-wide IT system to track such things...

...I'm already replying to the RFT...

sfakiaman

As a firearms owner I have a Pulse number and all my firearms and personal data are on computer (somewhere). On the recent firearms information day I was phoned by my local Garda to ask whether I had renewed my licences. The licences had been renewed last year and I am the only person with my surname in my area. Does Pulse work? go figure!