Tesco Mobile

lil5

Many thanks to Paul from me too for providing the updates throughout the day.

Also well done on the engineers for their hard work and getting a lot of the service restored. Will keep fingers crossed that rest will get sorted soon.


P.S. If Paul says there was a leak then there was a leak

Tesco Mobile: Paul

Hi All,
Looks like Texting and Data are both now back. Waiting on official confirmation.

regards Paul

2012IE

Tesco Mobile: Paul wrote: »

Hi All,
Looks like Texting and Data are both now back. Waiting on official confirmation.

regards Paul

hi Paul, what about people who were porting away from 48 / tesco mobile when this problem happened? i imagine there are a good few messed up ports, and people with zero service due to messed up ports...

Tesco Mobile: Paul

2012IE wrote: »

hi Paul, what about people who were porting away from 48 / tesco mobile when this problem happened? i imagine there are a good few messed up ports, and people with zero service due to messed up ports...

Yes, team are still looking at this...

regards
Paul

krudler

manutd83 wrote: »

i dont think its ridiculous to expect that when you pay for a service that you get the service you paying for,iv never seen a network down for as long,well other than eircoms broadband

less than a day? it happens, no network gives 100% service at all times and this is something out of peoples control. why not get a sim from another network to use as a spare number if it ever happens?

digitaldr

Everything's working fine now - reckon you deserve a pint Paul - have agood weekend

Stavro Mueller

I was with O2 for 14 years. Never had an outage like this. I hope it was a once-off because now that I've moved once, I can move again.

Paul. Thanks for your help and updates.

vicM

A hard days work it seems Paul, go on and have one down the pub :-)

The outage affected people in different ways I guess, as for me, the gates to our apartment complex is operated via gsm!!

jluv

Personally I had a day of peace and quiet! Thank you Tesco Mobile;)
But to be fair I have been with then 2 years now.Was unsure at first but have had no problems except for the 2 mentioned here. The difference in what I pay monthly though is fantastic.About a quarter of what I used to pay. We're also lucky to have Paul on here as i have witnessed him deal with peoples querys in an efficient and timely fashion.Thakfully have not needed his help yet..

lir6777

This was a pain in the backside (especially, I'm sure, for people using their phones for work) but having experienced crap customer support from another phone company for years, I have to thank Paul for his regular updates. I followed them all day and it was great to know what was happening- thanks Paul!

For the nuisance it was for us I can only imagine what a s**ty start to the bank holiday weekend it was for all at Tesco, who had to unexpectedly work overtime when they surely had better plans!

So long as this doesn't happen again too soon it's actually served to make me more satisfied with my move to Tesco Mobile- cheers again Paul and happy Paddy's Day!

DRose1

Been a very frustrating day, but all seems to be working again now.

Tesco Mobile: Paul

Hi All

Thank you for your patience on this. I know myself how frustrating it is to be without your phone. My wife says I should have married my iPhone!

This was not a good day for us. There are still some stability issues being worked on (topup is not back yet). I do though have to compliment our team of engineers in our partner companies who are dealing with this extreme issue with professionalism and dedication.

They will still be there for quite a while yet resolving topup and a few other back office remaining issues.

Again, we do sincerely apologise for the inconvenience here.

I will sign off now but will update in the am again.

regards
Paul

dohouch

Hi
Any reason why a new Tesco sim won't let me save any numbers to the sim with my old Nokia 6230i, but will with a newer Nokia N82

Tesco Mobile: Paul

One last update

Topup and porting are now working also

regards
Paul

tall chapy

Thanks for the updates Paul, I know what is like to have one of these events on a bank holiday weekend. Murphy's law....Go grab yourself a pint or two or many.. and charge it back to tesco mobile..you deserve it.

Tesco Mobile: Paul

Tesco Mobile: Paul wrote: »

One last update

Topup and porting are now working also

regards
Paul

wow paul sorry to hear about what happened no wonder i couldnt send txts or make a call

everything fine now good ot hear

thanks

TinyExplosions

Paulw wrote: »

But, there hasn't been a full outage for 9 hours, it's actually a partial outage, from what posters have been saying (on another site too). I don't use Tesco mobile myself, so can't confirm, but my mother uses it and she hasn't had much of a problem (I phoned her on her mobile to check). A few call failures, but a retry gets through. She doesn't really text much anyway, so no idea about that.

Similar has happened before in Ireland. I remember Eircom being offline twice, where the whole of Dublin was without service (1999 and 2002, I think). It has also happened in the UK (to two major mobile operators since 2000).

This could happen to any phone operator. As I said, due to the complexity of telecoms, it is almost impossible to have a fully redundant network operation.

As for your contract, you couldn't just walk away. There are actually provisions in the contract to protect the operator. You should check the fine print.

And, just for the record, I don't work in a data centre, and don't work for Tesco mobile.

I had no ability to make calls, send texts, or get online from 10am this morning, data and texts only came back at about 9, so that's an 11 hour outage of the things I use most, and it was 5 or 6 before I could make a call. To me, that is an unacceptably long period of time to be without service (especially as I tend to make very few calls, so lack of text/data is the same to me as a total outage).

Yes, it could happen to any network, and if it had happened to any other one, I'd be similarly critical, but I simply don't agree that it's too complex to have some redundancy in place -at it's simplest you can have an exact copy of your systems in another data centre, and it's a question of flipping a switch and awaiting propagation at worst to be back up and running -just because it's not done doesn't mean it can't/shouldn't be.

As for contracts, if they can't provide you with the service you have agreed upon, they are breaking their obligations for the provision of service, and as such effectively render their contract void -no amount of small print can completely cover them from that -yes it'd be down to an interpretation of what is unacceptable, but you'd certainly be able to get out of a contract if you argued enough.

Bad Panda wrote: »

I work in the industry. You clearly don't. You've no idea the amount of work involved and money isn't simply going to fix it! Cop on !

Really ridiculous what people expect, especially given that they've no idea what they're talking about.

I don't think it's ridiculous to expect a company to not have a single point of failure -maybe if we'd had floods like earlier in the year and the data centre was completely submerged people would be more understanding, but a mildly damp day and a single hole in a roof shouldn't cause a major outage, regardless of what industry you are in.

It's clear that there was no redundancy in place, and no backup plan at all meaning that a mad panic/scramble happened when things went down. All I can hope is that they learn from this, and put some backups in place!

Kudos must go to Paul though -I think everyone here knows it's not his fault, and he's done trojan work keeping in touch with all the grumpy boardsies!

ShaneU

Was wondering why I couldn't top-up earlier

byrnefm

It was definitely inconvenient not to have much of a service yesterday but I really did appreciate seeing such regular updates on what was happening from Tesco Mobile Paul - for most services you see go unavailable in other environments, the best you can usually hope for is 'technical issue - we're working on it' without any details whatsoever as to what went wrong.

Glad it's back now though

Paulw

TinyExplosions wrote: »

I had no ability to make calls, send texts, or get online from 10am this morning, data and texts only came back at about 9, so that's an 11 hour outage of the things I use most, and it was 5 or 6 before I could make a call.

Which was not a full outage, but partial outage, and not everyone was effected the same, but yeah, I believe text was down most of the day. Just because you use text most doesn't mean that they should prioritise text service, in fact, voice calls are much more important.

TinyExplosions wrote: »

I'd be similarly critical, but I simply don't agree that it's too complex to have some redundancy in place -at it's simplest you can have an exact copy of your systems in another data centre, and it's a question of flipping a switch and awaiting propagation at worst to be back up and running -just because it's not done doesn't mean it can't/shouldn't be.

Again, you don't understand a telecoms network. There is plenty of redundancy in place. "At it's simplest" is not reality, and is not practical. I don't know any organisation that would flip a switch (within a business day) to move everything to a different site. The normal plan of things is to give it at least 24 hours to recover a site, before moving to an alternate site, depending on the level of the fault, the estimated damage and estimated time for recovery. I've worked on disaster planning for large organisations before, and know what is and isn't likely. Many high-end systems can take a number of hours to restore from backups and get a system running, assuming all the hardware, infrastructure, etc is in place. 12 hours is totally an unrealistic timeframe to switch to an alternate site like that.

TinyExplosions wrote: »

As for contracts, if they can't provide you with the service you have agreed upon, they are breaking their obligations for the provision of service, and as such effectively render their contract void -no amount of small print can completely cover them from that -yes it'd be down to an interpretation of what is unacceptable, but you'd certainly be able to get out of a contract if you argued enough.

Not for a partial outage of service for a few hours. While you were unable to make voice calls before 5pm, many were able to make voice calls all day long. So, again, it wasn't a full outage at all.

TinyExplosions wrote: »

It's clear that there was no redundancy in place, and no backup plan at all meaning that a mad panic/scramble happened when things went down. All I can hope is that they learn from this, and put some backups in place!

Clear to you maybe. But, since I have worked on such disaster planning, I'm sure that there is a plan in place, there is redundancy, but whatever happened made the redundancy fail. I doubt there was panic, and I'm sure the on-site engineers did all they could to restore service as quickly as they could.

Telecoms networks strive for a 99.9% uptime of service. Even with a 12 hour outage (which it wasn't), that still puts them at 99.86% uptime for the year.

People rely on their mobile phones too much, and expect miracles when things go wrong. Every phone network, everywhere in the world, fails from time to time. Some have larger outages than others. Telecom networks are designed with no single point of failure, but, if multiple systems are hit, then there can be an outage, such as this.

I can only assume that the water damage here didn't just effect one machine, but rather there was water damage to a multiple number of machines. The effect of multiple nodes being hit had the net effect to customers of an outage. It's almost impossible to design a network to function given the vast number of possible faults and problems that could potentially hit, and I think Tesco Mobile, and their suppliers did the very best they could to bring back service as quick as possible today, and fair play to them. I'd love to see how some of the posters in here would work under that kind of pressure. :rolleyes:

Johnny Storm

To put it in context, only a few short years ago, Vodafone's network regularly used to collapse on a Friday afternoon while everyone was trying to organise their weekends.

2012IE

Johnny Storm wrote: »

To put it in context, only a few short years ago, Vodafone's network regularly used to collapse on a Friday afternoon while everyone was trying to organise their weekends.

to put it in context i am an o2 customer now without service for over 24 hours now, and i am being told that it will be a few days to fix. its a joke..

why isnt someone in working to fix it now..

Paulw

2012IE wrote: »

to put it in context i am an o2 customer now without service for over 24 hours now, and i am being told that it will be a few days to fix. its a joke..

why isnt someone in working to fix it now..

To put it in context, and I don't mean to sound crass, but you are just one customer. I'm sure someone will fix your individual issue when they can.

Might be no harm in you calling O2 customer support and asking them to look in to it.

Also, Tesco Mobile aren't O2, it's only run on the O2 network. Not sure why you're posting about an o2 issue in a Tesco Mobile thread.

banjacksed

5 People in the house on tesco, no one can put credit on their phones, Just not good enough on a bank holiday weekend, You would think that tesco mobile would have backup plans in such event, Even the car has a spare wheel.
Were now into sunday and still cant topup, wont hold my breath for 2moro, bank holiday,

Mowgli597

Paulw wrote: »

Again, you don't understand a telecoms network. There is plenty of redundancy in place. "At it's simplest" is not reality, and is not practical. I don't know any organisation that would flip a switch (within a business day) to move everything to a different site.

Of course there are such organisations. One which immediately springs to mind is an air-traffic control centre. Aircraft have triple redundancy systems, and we saw too clearly what happened in Chernobyl and Fukushima when standby systems failed and there was no disaster recovery plan to handle the inconceivable - because it WILL happen.

Paulw wrote: »

Telecoms networks strive for a 99.9% uptime of service. Even with a 12 hour outage (which it wasn't), that still puts them at 99.86% uptime for the year.

I thought we'd moved from percentage figures for quality of service a long time ago. 99.9% of service means that we accept 1 baby in 1000 will die on delivery because of the incompetence of hospital staff or systems, 1 aircraft in 1000 crashes and all aboard die etc..

Paulw wrote: »

People rely on their mobile phones too much, and expect miracles when things go wrong.

It's not up to any service provider to determine whether people rely on their phones too much. People pay for a service and expect it to be there when they want it. If they think people use their service too much then they should get out of the business.

I'm afraid this is a prime example of the Irish tradition of "Ah sure, it's grand." It's why we have such poor service in all areas - because we'll put up with things that other people would not tolerate.

Paulw wrote: »

I can only assume that the water damage here didn't just effect one machine, but rather there was water damage to a multiple number of machines.

With respect this is exactly what some of us have been saying. To design a system whereby a leak (for that's what we're told it was) can bring down a complete network for many hours is bad planning. It wasn't even a flood of the data centre, was it?

What would have been wrong with placing some of the machines in a different room, on a different power supply, preferably in a different building? Split the network and design redundancy into it.

Tesco Mobile: Paul

banjacksed wrote: »

5 People in the house on tesco, no one can put credit on their phones, Just not good enough on a bank holiday weekend, You would think that tesco mobile would have backup plans in such event, Even the car has a spare wheel.
Were now into sunday and still cant topup, wont hold my breath for 2moro, bank holiday,

Hi there

Systems are fully back since Fruday night at 10pm. Please call care between 8am and 8pm on 0894200000 as your problem is unrelated to the outage

Regards
Paul

Paulw

Mowgli597 wrote: »

Of course there are such organisations. One which immediately springs to mind is an air-traffic control centre. Aircraft have triple redundancy systems, and we saw too clearly what happened in Chernobyl and Fukushima when standby systems failed and there was no disaster recovery plan to handle the inconceivable - because it WILL happen.

I thought we'd moved from percentage figures for quality of service a long time ago. 99.9% of service means that we accept 1 baby in 1000 will die on delivery because of the incompetence of hospital staff or systems, 1 aircraft in 1000 crashes and all aboard die etc..

You're comparing a phone system to life and death? That's going a bit far, I think. :rolleyes:

I wonder what the actual percentage of babies die on delivery because of faults, or what percentage of aircraft crash??

Even ATC have outages, that can take a number of hours to resolve or switch over, during which time airlines go to their backup plan.

Mowgli597 wrote: »

It's not up to any service provider to determine whether people rely on their phones too much.

I was giving my own personal view that people rely on their phones too much. I guess telecoms providers would hope people use their phones more and more.

Mowgli597 wrote: »

I'm afraid this is a prime example of the Irish tradition of "Ah sure, it's grand." It's why we have such poor service in all areas - because we'll put up with things that other people would not tolerate.

I think from the Tesco Mobile updates, people were working as hard as they could, and it was the very opposite of the attitude you believe.

Mowgli597 wrote: »

With respect this is exactly what some of us have been saying. To design a system whereby a leak (for that's what we're told it was) can bring down a complete network for many hours is bad planning. It wasn't even a flood of the data centre, was it?

I can only go on what was said here too, about the flooding. No idea how bad or limited it was.

The complete network wasn't down from reports. Parts of the network were. Some could make calls, some couldn't. Some could receive calls all the time. SMS seems to have been down, as was topups, from the posts here. That's not a complete network down though.

Mowgli597 wrote: »

What would have been wrong with placing some of the machines in a different room, on a different power supply, preferably in a different building? Split the network and design redundancy into it.

Well, when you know more about the design of telecoms networks, maybe you can come back, get a job with Tesco Mobile, and show them how to do it?

Every network has redundancy, resiliance, but again, taking out a number of systems can cause problems, depending on what systems and the level of damage.

Maybe the power to the whole building went down? That happened to Eircom before. The whole building had no power, due to a flooding (on a day with no rain), and the whole of Dublin city had no phone service.

I guess we won't know on here what exactly the fault was, how many systems were damaged, how bad the flooding was, etc. But, clearly, from the updates, those involved were doing all they could to get everything back working as quickly as possible. Fair play to them for that.

Tesco Mobile: Paul

banjacksed wrote: »

5 People in the house on tesco, no one can put credit on their phones, Just not good enough on a bank holiday weekend, You would think that tesco mobile would have backup plans in such event, Even the car has a spare wheel.
Were now into sunday and still cant topup, wont hold my breath for 2moro, bank holiday,

Just checked the service reports there and there is an intermittent topup issue. If you retry it should go through

Regards
Paul

JTMan

There have been some interesting posts here about the data center that Tesco/Telefonica use. I have been reading a lot about this and many of your concerns mentioned here are echoed elsewhere. In fact, I was surprised about the sheer amount of information on this online.

The outage pulled down GiffGaff as well as 48 and Tesco Mobile Ireland and some other MVNO's.

The below might be of interest to some of the posters here ...

El Reg: GiffGaff goes titsup yet again in 'leccy cable gaffe

http://www.theregister.co.uk/2012/03/16/giffgaff_down_again/

From the comments section:

The root problem here is not the burst water pipe/power outage.
It is the complete lack of business continuity management and service delivery ethos that is plainly missing.

This is not rocket science. I was doing this 15years ago at BAT, 10 years ago at Global Crossing, and more recently for SME's. Most people will tolerate an outage, as they understand that things break. But they only understand when you communicate with them, and keep them informed. Giffgaff have been less then proactive in this case. It was my comment on the GG forums that provoked the response quoted earlier re a message on the GG landing page. At midday there were over 7000 people on the site, in all likelihood seeking an update or logging an issue. I would not be surprised to find the server has since buckled under the load, leaving everyone in the dark.

In terms of BCM, it is shocking that they use a single location to hold the access and accounting servers that control network access. This was evident when trying to obtain an account update and logging a service call because one could not use the network. Good network design would have had multiple AAA servers, for load sharing as well as resiliency. One wonders how many other SPFs exist in GGs infrastructure.

This is not 'new'. 9/11, anybody? Why does an inherently fragile technology market insist on recruiting the under-30's, who have no memory/experience of major outages or service delivery, and whom thus blithely assume that "because it hasn't happened... it won't", as opposed to the >40's who have been there, seen it, done it, recovered from it, and mitigated against it :-)

Someone in GiffGaff deserves a rollicking - and then being given responsibility for BCM going forwards, as you can be sure they won't make this mistake again.

As I mentioned earlier in this thread, this may end in compensation. For some users it will. Maybe Telefonica will replicate this compensation with their other MVNO brands ...

Telefonica to pay compensation for Network Outage

http://www.gadgetsandgizmos.org/giffgaff-set-to-compensate-users-for-downtime/

Users of the budget mobile network Giffgaff have been facing some issues today as services where wiped out by a burst water pipe. The saga began at around 11:30am on 16th March and for some users the problems aren’t over.

We’ve recently received an update via our customer email which sheds a little light on why Giffgaff went down and what steps the operators have taken to get services back online. The letter begins by explaining the problem was caused by a burst water pipe which has since been fixed. Their engineers have been working on this all day

Plenty more interesting articles on Google News about the outage.

Oceans12

i am on o2 and cannot register on o2.ie, aparrently some o2 customers are having this issue too, is there a common thread here?