P-Pars and other large state IT projects - a little insight

DublinWriter

"No-one gets sacked for hiring IBM"

Here’s a little insight as to why P-PARS is the debacle it eventually become.

Early last year, the EU Fisheries Commission issued a tender looking for ‘Web Development Resources’.

A friend of mine with a small software consultancy in Dublin applied and received back the tender documentation from the Commission.

The tender documentation that came from the Commission would have put Soviet Russia to shame. It constituted 10 very large questionnaire documents.

My friend decided not to bid for this business as I would have took him, or an employee, about two to three man weeks to actually prepare the response to the RFT (Request for Tender).

Now, here’s roughly how it would work in the private sector. Middle/Senior management decides that needs web development resources on a temporary basis. Contacts agency. Agency provides a couple of bodies for a few hundred Euro per day, job done within about nine months.

A year later my friend receives an email from the Fisheries Commission. It turns out that out of the 147 companies that initially expressed an interest, only 10 submitted for the job and none were selected. The Commission are obviously conducting a post-mortem into what went wrong. So what do they do? Submit 3 equally-long questionnaire documents to the 137 IT companies that dropped out at stage to find out what went wrong. According to my friend, the three documents would have taken at least a week to complete.

What has this got to do with the P-PARS debacle I hear you ask? Three words – ‘EU Procurement Guidelines’.

All public bodies follow EU Procurement Guidelines. This means that only the largest, most expensive and prestigious consultancies can afford to bid for public sector work in the first place because of the huge amount of Red Tape involved. All of the ‘Big Six’ companies employ specialist departments whose sole job it is to prepare tender documents for EU Public Sector work.

The above, coupled with the CYA (Cover Your Ass) non-risk taking attitude pervasive in management strata of the Civil Service means that only the big multi-national consultancies will always receive the work.

In the private sector there is a saying - “No one gets sacked for hiring IBM’, however in the public sector, no one gets sacked at all.


“Masses of Asses”

In the late 80’s while IBM and Microsoft were still on speaking terms, they agreed to jointly develop the 32-bit successor to DOS, a project that would eventually become OS/2.

Development was agreed to take place at Microsoft’s Redmond HQ. Microsoft had initially assigned a half-a-dozen of their programmers to the project. IBM bussed in 50 development staff in the first week and another 50 in the second week.

Back then, Microsoft favoured a ‘lean-fighting-team’ approach to development, as the more bodies there were on a project the harder it was to manage the project. They termed IBM’s approach ‘masses-of-asses’.

Eventually the cultural differences between the two organisations led to a split. IBM went on to develop OS/2 on their own and Microsoft eventually developed NT.

The ‘masses of asses’ approach is endemic in big consulting companies. This is typically because a) they get paid per consultant, so the more bodies they can throw at a project the better for them and b) senior management in consultancies are usually so ignorant to the technicalities of a project that they feel the more ‘brains’ they can throw at a project the better that projects chance of success is.

Compare and contrast with Ryanair’s approach to developing their website. At the time, Aer Lingus spent just over a million punts developing a static website that couldn’t take on-line bookings. Ryanair employed two young programmers for a little under a year and delivered their website for what I’d term ‘lunch-money’.

“The fighting unit”

In my own personal experience in IT, the two most successful groups of people to implement IT projects are a) the military and b) the Japanese.

The reason this is the high culture of self-reliance favoured by both cultures. Rarely does either engage consultancy companies as they always consider that solutions should be developed in-house and that expertise should always be fostered from within.

Japanese organisations especially have a culture that says that no-one understands a companies problems more than the company itself.

On Questions and Answers last night I heard Minister Willie O’Dea say that most IT people within the Civil Service get snapped up by the private sector for three-times the salary. What he forgot to add was that usually the Public Sector then might re-employ the same person from the Private-Sector as a consultant for ten times their original salary.

The solution is obvious. There needs to be a Shared-Service IT organisation within the Civil Service that pays private sector salaries and works to a private sector ethos. This organisation, rather than consultancies, should be the first point of call for any large scale state IT-projects such as PULSE or P-PARS.

The BBC in the UK successfully took this approach. In the mid-1990’s the BBC, a vast and nebulous civil-service organisation, set up a shared-service IT function that not only catered for the vast majority of the organisation’s IT needs, but was also allowed to bid for outside work in the private sector.

tba

I agree, however a shared IT service should only be for new systems, not for maintaining existing or future ones.

Any SITS can be in danger of becoming removed from the body they serve. In an organisation with a limitless number of branchs and disiplines local IT mantiance staff are required to solve problems.

i71jskz5xu42pb

I agree with a lot of what you are saying here - just two points

DublinWriter wrote:

Now, here’s roughly how it would work in the private sector. Middle/Senior management decides that needs web development resources on a temporary basis. Contacts agency. Agency provides a couple of bodies for a few hundred Euro per day, job done within about nine months.

I will give you an alternate view of how it works in the private sector. You have some work that needs done so you farm it out to people you have worked with before or former colleagues you trust. You do not open the process up to everybody (even if you go the agency route you will only use a few agencies) and you do not have a highly rigorous process to decide who gets the contract. As such, there is a good chance that, while you will get your contractor, it may not be a fair and open process to all applicant contractors. The EU/government does not have this luxury - they need to make sure that the process has the appearance of fairness to all (not that dodgy deals are not done sometimes).

DublinWriter wrote:

Compare and contrast with Ryanair’s approach to developing their website. At the time, Aer Lingus spent just over a million punts developing a static website that couldn’t take on-line bookings. Ryanair employed two young programmers for a little under a year and delivered their website for what I’d term ‘lunch-money’.

This is a bit of an urban myth I'm afraid. Ryanair did indeed hire some young programmers to develop their website i.e. the pretty HTML on and around the front page. However to the best of my knowledge the booking portion of this site was provided by OpenSkies (as it is to this day). Openskies were a company formed by <queue baddie music> the the type big bad consultancy companies you so deride: Accenture and HP.

That said Ryanair did still probably get a better deal. They were buying a total off the self reservations system from Openskies as opposed to Aer Lingus buy a custom web only front end to their existing system.

Hobbes

Actually the saying is "No-one ever got fired for buying IBM " and refers back to the 70's. IBM has only redefined itself as a global services in the last 5 years of so (TMK and MOPO).

bonkey

I fail to see how a private-sector-ethos-and-pay company which automatically gets awarded a contract can consistently beat out an approach of tendering to private-sector companies and picking the best.

What you're proposing is little different to saying "we will award all contracts to Accenture" (or other company). You're removing competition from the tendering process, which is the only thing which can offer you a better deal.

At the very least, an internal group - regardless of payscale and the rest of it - should be willing to put its bid up against private-sector in a tendering process. If its really better, it should win anyway. If its not better, it shouldn't get the contract.

In my own personal experience in IT, the two most successful groups of people to implement IT projects are a) the military and b) the Japanese

Keywords here being "personal experience". The Swiss army just had an unbelievable overrun in a new personnel management system. It was years late, massively over-budget, and ultimately scaled-back in functionality.

The solution is obvious.

I couldn't disagree more. The highly-paid external consultants are the type of people you're saying should be kept for this internal company. If they're that good, then how did they get it so wrong on this project? Why would they do a better job working for an internal development group than an external one? What reason would an internal group who is automatically awarded contracts have to keep its standards up, care about getting value-for-money on its employees, or any of the rest of it.

Ask yourself how the final hardware/software purchase/licensing costs alone came to more than the original budget. Perhaps the developers screwed the development (in which case it wouldn't matter if they worked for Accenture or an internal group), or maybe something was seriously underestimated at design time (in which case it wouldn't matter if they worked for Accenture or an internal group), or it could have been that there was some massive discrepancy between the details in the RFT and the reality of what was actually sought (in which case it wouldn't matter...you get the idea).

Without knowing the details of where the order-of-magnitude budget overrun came about, suggesting that the solution is obvious strikes me as little more than armchair quarterbacking.

Regardless, reducing the tendering from "something only the Big6 can afford to do" to "we'll award it always to one company which is matching/beating the Big6 for salaries etc." is hardly a magic bullet.

I always remember an idea an old boss of mine put to me once. He told me that bridges are (or used to be) built in two stages. First, you tender out the design. Then, when the system is designed, you tender out the construction. Here's the kicker - the designers cannot tender for the construction. Personally, I think there's the core of a good idea there.

I have never once seen an RFT from a public body without one of the following being true:

1) Design was part of what was being tendered, making it impossible to accurately predict how expensive construction would be
or
2) Design was included so (notionally) only construction was needed, but the design was not done by experienced IT people, meaning that even though proper design wasn't part of the tender, it had to be subsequently carried out anyway.

Lastly, the real shame about this project is that no-one noticed it was a failure until we were 15X over-budget....or if they noticed, nothing was done. Obviously, failure and success conditions were not specified (correctly, or perhaps at all) at the outset - which is not a failing of the developers.

jc

DublinWriter

bonkey wrote:

I fail to see how a private-sector-ethos-and-pay company which automatically gets awarded a contract can consistently beat out an approach of tendering to private-sector companies and picking the best.

Well, that's what happened with P-PARS.

bonkey wrote:

What you're proposing is little different to saying "we will award all contracts to Accenture" (or other company). You're removing competition from the tendering process, which is the only thing which can offer you a better deal.

That's really the classic insourcing vs. outsourcing arguement. What I'm proposing would combine the best of both worlds. However, in an in-sourcing arrangement, you do need proper governance and evaluation processes.

bonkey wrote:

Keywords here being "personal experience". The Swiss army just had an unbelievable overrun in a new personnel management system. It was years late, massively over-budget, and ultimately scaled-back in functionality.

Maybe I should have said professional experience. I've studied how a lot of Japenese corporations work, especially in terms of IT investment. I know a couple of Lieutenants in the IT section in the Irish army and have evaluated military projects in other countries.

The Swiss army project is only one example you're offering. I could list 100's of failed IT projects in the non-military public/private sector.

bonkey wrote:

What reason would an internal group who is automatically awarded contracts have to keep its standards up, care about getting value-for-money on its employees, or any of the rest of it.

You mean like for example, the sales function of an organisation? I don't see any companies beating a path to outsource that particular area. A sales function is always kept insourced because it's performance is very transparent. Same should be true of any IT activity.

bonkey wrote:

Without knowing the details of where the order-of-magnitude budget overrun came about, suggesting that the solution is obvious strikes me as little more than armchair quarterbacking.

Well, I think D&T getting €60 million for putting bodies on-site is a wee-bit of a clue.

bonkey wrote:

Regardless, reducing the tendering from "something only the Big6 can afford to do" to "we'll award it always to one company which is matching/beating the Big6 for salaries etc." is hardly a magic bullet.

Basically, what I'm advocating is a return to in-sourcing with proper controls , evaluation processes and governance.

bonkey

Well, that's what happened with P-PARS.

No, its not. PPARS was an example of the tendering process failnig to produce a success.
That by no means suggests that removing teh tendering process and giving the project to an internal company would have yielded a success.

That's really the classic insourcing vs. outsourcing arguement.

Sort of, and its based on the fact that experience shows that once a system gets "settled" and competition disappears, ineffeciency sets in.

People started outsourcing because internal groups were massively inefficient and extrenal companies willing to fight over teh contract produced better results.

Now, outsourcing is limited to a small number of players who are more-or-less slicing up a pie between them without real competition, so inefficiency has set in.

What I'm proposing would combine the best of both worlds.

Only if you still allow external companies to bid, and simply take the internal-paying-market-rates as just another tender. If it really is better, it will win out. If its not, then granting it the tender when someone else is making a better offer is not the best path.

However, in an in-sourcing arrangement, you do need proper governance and evaluation processes.

Yup. The two things that are missing from the design-phase and customer involvement of outsourced projects. Maybe thats where the flaw lies, rather than in who actually does the work.

Maybe I should have said professional experience.

Its still your personal experience, whether it be professional or not. There is nothing yet offered on this thread to suggest that your experience matches the market reality other than your apparent suggestion that it does.

Incidentally, I've worked on several projects for public-sector in Ireland. Every single one was delivered on-time, on-budget, and met with custmoer approval. So I could argue that my professional experience of the Irish public sector is that the PPARS project disaster is an exception, not a rule. I wouldn't do that, but what I'm pointing out that its still just experience/opinion and without something to show it reflects the market reality it cannot be taken as anything more.

The Swiss army project is only one example you're offering.

Can you explain how your "military do it right" can gel with even one example of gross failure? OK - if you want to limit it to the Irish military, maybe you ahve a point, but I'm not aware that they have a long history of large IT development. The US military, certainly, have had no end of expensive failures, as well as overly-expensive completions, and the occasional great success.

I could list 100's of failed IT projects in the non-military public/private sector.

And you could list 100's successful IT projects in the non-military public / private sector than in the miltary...assuming you knew the markets as well.

Such numbers don't prove points - they are simply chosen to support already-chosen stances.

Percentages are far more significant. What percentage are successful. What percentage overrun budget/time but still meet requirements? What percentage are cut back on requirements to save time/money? What percentage are canned, and at what point, and for what reason?

Well, I think D&T getting €60 million for putting bodies on-site is a wee-bit of a clue.

Afraid not.

It still doesn't explain to me how 20 million was spent on hardware and licensing costs for a project with an initial budget of 9 million. While this might seem like a small point because its "only" 20 million, it is still an overrun of the same order of magnitude, or at least in the same ballpark.

Also, if I know that an external developer will cost 1,500 per day, and I budget 1,500 per day, it doesn't explain the 15x overrun. If I had that developer as an internal resource, they might cost 300 per day, and be budgeted at 300 per day....so my staffing budget would be 5 times lower....but that still doesn't explain the 15x overrun, it simply means that my initial and final staffing budgets would be 5x lower.

Basically, what I'm advocating is a return to in-sourcing with proper controls , evaluation processes and governance.

If you had proper controls, evaluation processes and governance, whether or not the project was in- or out-sourced would, in all probability, cease to be a factor in terms of success, and it would become a simple question of whether or not it was cost-effective to maintain the full-time staff. If you don't have ongoing development of a fairly-constant nature, on a fairly-constant platform, then its simply not practical to have a permanent staff of developers.

When I worked in IT training, I used to occasionally train Microsoft employees on how to use Microsoft products. Why did MS do this? Because it was cheaper for them to buy in training when needed, rather than maintain a training capability over the full range of products/platforms they needed. The quality of training was a seperate issue, and rightly so.

In short, I agree with what your'e looking for, except for the return to in-sourcing. There should be a small contingent of in-house IT - enough to develop small projects, and to supply the necessary knowledge to enable the quality-control of externals. After that, its a question of practicalities. It is not cost-effective to try and get yourself a 30-man SAP team as permanent employees for a one-off SAP development no matter what way you look at it.

jc

DublinWriter

bonkey wrote:

It is not cost-effective to try and get yourself a 30-man SAP team as permanent employees for a one-off SAP development no matter what way you look at it.

No - you completely misunderstand my arguement. I'm suggesting a state shared-services IT function.

NewDubliner

DublinWriter wrote:

No - you completely misunderstand my arguement. I'm suggesting a state shared-services IT function.

This would start with two major handicaps:

1: 85% of the existing experienced IT staff are due to be moved out of ICT work to make way for 'decentralisation'.

2: The new body will be located in 'Parlon Country'.

jd

NewDubliner wrote:

This would start with two major handicaps:

1: 85% of the existing experienced IT staff are due to be moved out of ICT work to make way for 'decentralisation'.

So are they going to recruit (at graduate lvel) to replace?
Or are most IT jobs at clerical office level?!? ( I think I read that somewhere), so they will "train up" admin staff?

NewDubliner

jd wrote:

So are they going to recruit (at graduate lvel) to replace?
Or are most IT jobs at clerical office level?!? ( I think I read that somewhere), so they will "train up" admin staff?

The plan developed by Finance is something like this:

1: Ask the existing staff to leave Dublin.

2: If that doesn't work, offer the jobs to non IT staff & get the existing staff to train them while they're still doing their own jobs. Once they've finished, they'll be dropped in a black hole somewhere.The existing staff will be motivated by the knowledge that no matter how well they do, they'll never have a chance of promotion & that their efforts will greatly assist the re-election of the FF/PD alliance.

3: If that doesn't work, offer promotions to junior staff.

4: If that doesn't work, recruit experienced IT graduates at the HEO grade, but without increasing overall civil service numbers.

The DIG has asked all the IT departments to present detailed plans by the end of the month.

is_that_so

Ultimately a project falls or stands based on how the requirements are gathered and the potential merits of the solution stand up to it.
In my own experience there can be too many yes men out to prove themselves tied up in projects. They also seem to have an inability to apply the KISS principle. Software solutions cannot overcome poor business practices and invariably fail. The HSE as it stands needs a lot more than a sofware systesm to fix it.

democrates

bonkey wrote:

The Swiss army just had an unbelievable overrun in a new personnel management system. It was years late, massively over-budget, and ultimately scaled-back in functionality.

Maybe the swiss army had too many attachments on their project (look, a tweezers!). Occams razor might be more fitting in future.

Hobbes

Probably looking at this the wrong way but... What if a companies that tendered the jobs were given tax credits as opposed to money. it would be technically spending money but wouldn't be taking the money from the tax payers. Or am I just over simplifying?

democrates

Hobbes wrote:

Probably looking at this the wrong way but... What if a companies that tendered the jobs were given tax credits as opposed to money. it would be technically spending money but wouldn't be taking the money from the tax payers. Or am I just over simplifying?

The net result of the transactions would be the same ok, but I'm thinking that companies like to have a growth story to tell around revenue and turnover, and governments like to have a growth story to tell around the 'size' of the economy.

Basically the more buying and selling occurring the better your economy looks so mechanisms that mask economic activity may be seen as second best.

bonkey

DublinWriter wrote:

No - you completely misunderstand my arguement. I'm suggesting a state shared-services IT function.

I'm not misunderstanding at all.

Perhaps you could explain to me what SAP project is due to start anywhere in government when PPARS is over?
If there isn't one, then explain to me what your government-exployed SAP engineers will be developing after this PPARS project? If there is one, will there a third following the second? Then a fourth?

Sooner or later, your SAP development needs will dry up, but you have a team of SAP developers. Whatchya gonna do?

Now, the same applies for every language, platform, etc. in use by our governemnt. You obviously have experience in the industry, so I assume I don't need to waste time arguing that trying to explain why you'll need developers who are specialists in (or at least focussed on) each language/technology/platform used.

Or are you going to just suggest that an "obvious" prerequisite (as well as good governance, oversight, and all of the rest of the things that have nothing to do with whether something is in- or out-sourced but which are currently missing) would also be to standardise on a small number of platforms? So no SAP in your world, perhaps, but just .Net or J2EE, or whatever. If you do take this stance, could you comment on how it will be cost effective to scrap all existing software which is not in these new standards of yours and redevelop them to these standards? Or how it would be cost effective to develop something in-house from scratch because you couldn't outsource to the guys who are offering to customise their already-written-but-in-the-wrong-language system for a pittance?

The solution, as I said at the outset, is far from obvious....especially when your obvious solution still relies on having oversight, control, culpability, and all of the other things who's lack is one of the real core reasons that projects are going disastrously wrong.

There is no magic bullet. There never has been in IT, but every so often, something is sold as one. A new language, a new OS, new hardware, outsourcing to consultancies, outsourcing to India, insourcing.....its all been tried, and you know what people eventually conclude? Each has their place, each can be beneficial when used correctly, and none are a panacea.

jc

RainyDay

DublinWriter wrote:

The solution is obvious. There needs to be a Shared-Service IT organisation within the Civil Service that pays private sector salaries and works to a private sector ethos. This organisation, rather than consultancies, should be the first point of call for any large scale state IT-projects such as PULSE or P-PARS.

You mean something like the CITS organisation in the Dept of Finance (mentioned here).

DublinWriter

bonkey wrote:

I'm not misunderstanding at all.

Perhaps you could explain to me what SAP project is due to start anywhere in government when PPARS is over?
If there isn't one, then explain to me what your government-exployed SAP engineers will be developing after this PPARS project? If there is one, will there a third following the second? Then a fourth?

You're failing to see the wider picture and how IT project teams are composed.

Do you think a shared-service IT function such as the one I proposed would consist totally of programmers?

The type of employees of such a function would consist of Business Analysts, Project Managers, etc, etc.

The technical employees would have experience in technologies used commonly across the public sector - Oracle, VB, MS SQL Server.

Plus SAP, or any other big enterprise software, isn't a 'one-hit' solution. Developement resources are always required after the initial implementation as functional needs change.

Specific vertical skills such as SAP (ABAP, implementation of, etc) would be contracted in *on an individual basis* as required through agencies and not big-6 consultancies.

RainyDay

DublinWriter wrote:

Specific vertical skills such as SAP (ABAP, implementation of, etc) would be contracted in *on an individual basis* as required through agencies and not big-6 consultancies.

This isn't necessarily a route to save money. A properly structured fixed price contract for specific deliverables may well be cheaper in the long run than employing a load of agency contractors on an hourly rate.