Tesco Mobile

BuzzG

As a previous poster said have been with Tesco Mobile about a month and this is the 2nd outage in that time with similar end user impact so there must be a very leaky roof in that data centre. Will give the benefit of the doubt for now but if these occurrences continue happening I for one will be taking my business else where rather sharpish. Don't have a bill pay account to have this level of service.

Tesco Mobile: Paul

BuzzG wrote: »

As a previous poster said have been with Tesco Mobile about a month and this is the 2nd outage in that time with similar end user impact so there must be a very leaky roof in that data centre. Will give the benefit of the doubt for now but if these occurrences continue happening I for one will be taking my business else where rather sharpish. Don't have a bill pay account to have this level of service.

Hi there

This is not good I agree but just to be clear, the previous outage was totally unrelated - that was a hardware failure.

regards
Paul

SMI

Tesco Mobile: Paul wrote: »

Hi SMI

As unbelievable a reason as it may seem its absolutely true. There was some roof work being performed at the data centre and the roof was pierced and water flooded in over our cage in the data centre. There are engineers from our managed service partners out there presently doing 2 things - 1 trying to restore service ASAP for customers and 2 trying to assess the damage and medium term work needed.

Obviously we are not happy that this could happen at all but post mortems and lessons are for later. For now we are focussed on minimising impact to our customers and getting service in place.

regards
Paul

Hi Paul,

Thanks for getting back to me so promptly. I don't want to bang the drum, but any chance of a bit of evidence i.e. pictures?
Do you think there will be some sort of concessions offered to customers affected once you get the problems sorted?
Still no service for me at the moment, couldn't make work related calls and it is pretty inconvenient when you want to be making plans for paddy's weekend also...

manutd83

This is not good and i cannot get picture messaging working either with my phone,the picture messaging is a problem since i joined this network

Merch

Can receive calls now, its from a tesco number so thats something, i could not send a text up till a few mins ago, so that appears to be still down.

While I was a bit unhappy about the communication of their being an issue initially, it did seem tesco were doing what was needed to fix the issue, some people are going a bit mad about something that could happen to any network.

shamwari

SMI wrote: »

Hi Paul,

I don't want to bang the drum, but any chance of a bit of evidence i.e. pictures?

That's a bit OTT, is it not? Do you not accept / trust what they are saying?

du Maurier

shamwari wrote: »

That's a bit OTT, is it not? Do you not accept / trust what they are saying?

It really is indeed. Interminable and stuffy-natured. Real Meldrew stuff:pac:

Tesco Mobile: Paul

SMI wrote: »

Hi Paul,

Thanks for getting back to me so promptly. I don't want to bang the drum, but any chance of a bit of evidence i.e. pictures?
Do you think there will be some sort of concessions offered to customers affected once you get the problems sorted?
Still no service for me at the moment, couldn't make work related calls and it is pretty inconvenient when you want to be making plans for paddy's weekend also...

Hi SMI

Yes, I apologise for the incovenience this causes.You should now be able to make calls at least most of the time - retry if you are having issues. I wont be asking the engineers to take pictures as they have more important activties underway.

Focus is still on getting service in place. I have no information beyond this.

regards
Paul

SMI

shamwari wrote: »

That's a bit OTT, is it not? Do you not accept / trust what they are saying?

Not to offend Paul but I can't say I fully do. My sister is also on their rolling monthly bill pay service, with a bigger collective bundle than I have. It would take her a Herculean effort to get near or even surpass her normal bundle but she has been getting some very strange estimates (as I mentioned in a previous post, when my sister rang up to inquire about it she was told that her bill for that month had reached €400...). I have my concerns that the issues they have may be ongoing teething problems, I am sorry to say that I do not fully accept/trust what they are saying.

I'm not angry or looking to vent my frustration at an individual, I just want to have my mind set at ease.
I need the phone for business, I've been put out today. I would like to know that I can rely on Tesco mobile in the future. Thats why I thought it would be a good idea for them to take a quick pic of the damage with a phone and upload it here or on facebook, a fairly easy task and not time consuming.
It would put my mind at ease and potentially save the operator losing 2 customers.
I have no shame in putting that out there, no offense to anyone connected with the operator but I'm paying for a service that has let me, and someone else I know, down badly. Asking for the above under that context doesn't hit me as OTT at all.
You should demand the best out of what you pay for.

SMI

And it still isn't working for me....

i can make calls now yeah but not txts

shamwari

SMI wrote: »

Not to offend Paul but I can't say I fully do. My sister is also on their rolling monthly bill pay service, with a bigger collective bundle than I have. It would take her a Herculean effort to get near or even surpass her normal bundle but she has been getting some very strange estimates (as I mentioned in a previous post, when my sister rang up to inquire about it she was told that her bill for that month had reached €400...). I have my concerns that the issues they have may be ongoing teething problems, I am sorry to say that I do not fully accept/trust what they are saying.

I'm not angry or looking to vent my frustration at an individual, I just want to have my mind set at ease.
I need the phone for business, I've been put out today. I would like to know that I can rely on Tesco mobile in the future. Thats why I thought it would be a good idea for them to take a quick pic of the damage with a phone and upload it here or on facebook, a fairly easy task and not time consuming.
It would put my mind at ease and potentially save the operator losing 2 customers.
I have no shame in putting that out there, no offense to anyone connected with the operator but I'm paying for a service that has let me, and someone else I know, down badly. Asking for the above under that context doesn't hit me as OTT at all.
You should demand the best out of what you pay for.

I don't doubt for one minute that gravity of the situation that you and your sister find yourselves in with Tesco's service here. Certainly with regard to your own plight and the frustration and annoyance this outages are causing, I empathise fully with you because I'm more or less in the same boat. I've got a very ill family member in hospital at the mo and I need to be able to make and receive calls.

However if Tesco are saying that they had a leak, then they had a leak. Sight or otherwise of photo's top prove it or the scale of damage thereof makes no difference to me. I accept their word.

mathepac

Paul, thanks for the timely updates. I can make and receive calls and receive texts but not send them.

Best of luck to the guys / gals cleaning up the mess at the data-centre.

rougerie

Joined tescomobile yesterday, just as well I didn't bet on Cheltenham today! Waiting to see how this develops. Must say I was impressed yesterday by TM customer service staff and by Paul's heroic efforts to communicate via Boards.ie.e.

Tesco Mobile: Paul

Hi All

Just to give you a further update.

The team are reporting that outbound and inbound calls should work 100% now. If you are still having an issue with this please restart your phone.

Texting is still not working outbound and is working inbound

Data and MMS are not working yet either.

Topup and My Tesco Mobile account access should be working now also.

Will update more shortly.

regards
Paul

manutd83

my calls still isnt working

Mowgli597

I too am a new customer of Tesco Mobile, having migrated from O2.

I have to say the migration was not the easiest exercie - I have an iPhone4 and require a microSIM. Although Tesco Mobile mailed me two of these I ended up having to cut down a full-size SIM in order to get connected. Anyone want a spare microSIM

As regards the current frustrating outage, I agree totally with the posters who found difficulty in finding out what was going on - until I discovered this thread. Initially I thought it was part of the problems I had when migrating until a call to a colleague who is also on Tesco Mobile confirmed it was a general fault.

Having managed a data centre for a major mainframe manufacturer many, many moons ago I have to say that this outage is totally unacceptable. Any data centre worth its salt has a disaster recovery plan - and a flooded data centre is one of the standard contingencies that a good manager has to consider (along with an aircraft landing on the centre if you're located near an airport, or, my favourite, a mad axeman getting through security and starting to smash up everything!).

There was (is) a standard time in which you must get the centre back up and running, and there are companies out there who specialise in doing this.

Add to that that any work on a roof over any equipment is an obvious risk and so steps should be taken to relocate the equipment or, failing that, protect it from exactly what Paul is saying happened (it's not hard - sheets of plywood and plastic sheeting).

Having had to deal with a flooded data centre in Ohio one Christmas when a plant shut down but didn't consider the fact that the (water based) sprinkler system was liable to freeze and burst (which it did, covering their mainframe in dissolved plaster ceiling tiles), the golden rule is if you think it can't happen then it will.

Sorry - rant over :eek:

shamwari

Mowgli597 wrote: »

I too am a new customer of Tesco Mobile, having migrated from O2.

I have to say the migration was not the easiest exercie - I have an iPhone4 and require a microSIM. Although Tesco Mobile mailed me two of these I ended up having to cut down a full-size SIM in order to get connected. Anyone want a spare microSIM

As regards the current frustrating outage, I agree totally with the posters who found difficulty in finding out what was going on - until I discovered this thread. Initially I thought it was part of the problems I had when migrating until a call to a colleague who is also on Tesco Mobile confirmed it was a general fault.

Having managed a data centre for a major mainframe manufacturer many, many moons ago I have to say that this outage is totally unacceptable. Any data centre worth its salt has a disaster recovery plan - and a flooded data centre is one of the standard contingencies that a good manager has to consider (along with an aircraft landing on the centre if you're located near an airport, or, my favourite, a mad axeman getting through security and starting to smash up everything!).

There was (is) a standard time in which you must get the centre back up and running, and there are companies out there who specialise in doing this.

Add to that that any work on a roof over any equipment is an obvious risk and so steps should be taken to relocate the equipment or, failing that, protect it from exactly what Paul is saying happened (it's not hard - sheets of plywood and plastic sheeting).

Having had to deal with a flooded data centre in Ohio one Christmas when a plant shut down but didn't consider the fact that the (water based) sprinkler system was liable to freeze and burst (which it did, covering their mainframe in dissolved plaster ceiling tiles), the golden rule is if you think it can't happen then it will.

Sorry - rant over :eek:

There was also the story about the firm that had its production data centre in the North Tower at the World Trade centre. The disaster recovery site was in the south tower.....:eek:

Your comment above about the "golden rule" is quite correct..

chris2008x

I just thought it was a regional thing in my area because our phones couldn't make/receive calls earlier. I'm billpay and my mum is prepay and my dad who was in Thurles at the time is prepay and he could receive calls when I phoned him on the landline which we couldn't from.

I was just talking to Comreg they said they were never notified about the service disruption. I wasn't complaining about it btw, I just called to try get a landline number off them for customer care and they said you don't have one and were shocked to hear about the outage.

Tesco Mobile: Paul

Hi All

Just in case you are trying to topup by our topup IVR 1740 it is rediecting to O2 presently (one of the side effects of the workarounds to get calls working).

If you need to topup a voucher you can do so by using *101*Voucher No# and press call

regards
Paul

SMI

Tesco Mobile: Paul wrote: »

Hi SMI
I wont be asking the engineers to take pictures as they have more important activties underway.

Focus is still on getting service in place. I have no information beyond this.

regards
Paul

Hi Paul,

Just occurred to me that you mentioned the engineers were assessing the damage and were considering what to do in the medium term.
Surely an insurance claim will be made by someone in Tesco in consultation with the engineers, no doubt accompanied by pictures of the damage, a standard enough step, no?
Any chance you could request one or two and present them. After reading the a number of the more recent posts on this thread I feel fairly vindicated in repeating my request. My service still hasn't returned to normall, that's an entire working day. Extremely disappointing to say the least.

Paulw

Mowgli597 wrote: »

Having managed a data centre for a major mainframe manufacturer many, many moons ago I have to say that this outage is totally unacceptable. Any data centre worth its salt has a disaster recovery plan - and a flooded data centre is one of the standard contingencies that a good manager has to consider.

There was (is) a standard time in which you must get the centre back up and running, and there are companies out there who specialise in doing this.

So, since you managed a data centre, what is the "standard time" to get the centre back operational? Hours, days, weeks?? :rolleyes:

As someone who works in IT, a system site disaster. with flooding and power outage, it is something you can plan for, but reality is that when it happens, it can take a good number of hours, at best, to get systems working again. And that assumes that the IT systems weren't fried by the water and power, which could mean days/weeks before the system was running again.

This is certainly not the first time a telecoms company has had an outage due to flooding, and it won't be the last. The level of damage determines the recovery time.

Give the guys a chance, I'm sure everyone is doing everything they can to get systems back operational.

Tesco Mobile: Paul

Hi All

Hopeful of a major update in the next 30-45 minutes

regards
Paul

Mowgli597

Paulw wrote: »

So, since you managed a data centre, what is the "standard time" to get the centre back operational? Hours, days, weeks?? :rolleyes:

Depends on the urgency of your operation (i.e. how long can you afford to be down).

Paulw wrote: »

As someone who works in IT, a system site disaster. with flooding and power outage, it is something you can plan for, but reality is that when it happens, it can take a good number of hours, at best, to get systems working again. And that assumes that the IT systems weren't fried by the water and power, which could mean days/weeks before the system was running again.

If your plan doesn't cover off-site operations then it's hardly worth having. Weeks to get a data centre runniing again would mean most companies would go out of business - especially if their business is communications. If your hardware is fried then you take your operation somewhere else, load up your (backup) data (stored off-site, of course), and get back on the air.

But you should never be configured such that ALL your hardware fries in one incident. You have to have standby equipment, preferably in hot standby mode, so that it kicks in immediately.

Paulw wrote: »

This is certainly not the first time a telecoms company has had an outage due to flooding, and it won't be the last. The level of damage determines the recovery time.

And your plan should allow for that. One option is to port your data to a back-up site somewhere and get up and running (as a previous poster said unfortunately one user's backup site was the other tower in the Twin Towers disaster).

Another option we had (as I say many, many moons ago when we were running plants on mainframes) was that a whole new data centre arrived on the back of artics and set up in the car-park (a container for the hardware, an air-conditioning unit to cool it, power generators to run it). Bring in new hardware, configure and load it up and start running. 24 hours max.

Paulw wrote: »

Give the guys a chance, I'm sure everyone is doing everything they can to get systems back operational.

I agree, and having been one of those support engineers working all over the world I know just what it's like. I was even "trapped" in Medellin, Colombia (drug capital of the world) for two weeks until I restored a mainframe.

In my case the operation was a computer manufacturing plant. If "my" data centre went down for whatever reason, the net result was a plant of almost 1000 people became idle and production stopped. That could not, and thanks be to God never did, happen.

wmits

Here here. Give them a chance. I also work in IT and it's no bloody easy dealing with something like this.

The fact that services are coming back is some sign that there was a plan in place. As for taking pictures: I give less than a fiddler's for photographic evidence. If taking a photo would make my phone work, then by all means snap away. If not, then please get on with your work and leave the amateur photography for a later stage.

Fact is, as someone who got shafted for years on rolling 18 month contracts at exorbitant prices for a service that I wasn't getting full use of, I can put up with a few inconveniences for Tesco prices and conditions. There may be a few hiccups from time to time, but try getting someone from a major carrier to hang around and take the flack, while still addressing all questions (no matter him inane) in a helpful and courteous manner.

Good on you Paul.

Oh. .. and fix my phone will you ... It's a real pain :-)

Paulw

Mowgli597 wrote: »

If your plan doesn't cover off-site operations then it's hardly worth having. Weeks to get a data centre runniing again would mean most companies would go out of business - especially if their business is communications. If your hardware is fried then you take your operation somewhere else, load up your (backup) data (stored off-site, of course), and get back on the air.

But you should never be configured such that ALL your hardware fries in one incident. You have to have standby equipment, preferably in hot standby mode, so that it kicks in immediately.

While, in theory, your ideas are good, in practice, they're unrealistic, mainly due to cost. You obviously have no understanding of a telecoms network, the number of nodes involved, the number of systems, the configuration and complexity of it all.

Even with the best designed telecoms networks, there are always limited points of potential failure. You can only have so much configuration that is georedundant, especially considering the number of systems involved. A single voice call can need 8 or more systems just to enable the call to connect, never mind all the patch panels, switches, routers, firewalls, etc, and if one is off the air, then a call will fail.

And the cost of having a standby site/system, in telecoms, is almost impractical, due to connectivity, infrastructure, configuration, hardware, and the cost of millions for it all.

Thankfully, such telecoms outages such as this one seldom happen. People need to be realistic.

Stavro Mueller

Have you any plans to send a text message to all customers, letting them know what has happened? Not all your customers use the internet and will know what's going on.

Tesco Mobile: Paul

cymbaline wrote: »

Have you any plans to send a text message to all customers, letting them know what has happened? Not all your customers use the internet and will know what's going on.

Hi Cymbaline,

Yes we will do this on Tuesday most likely. We actually cant text them at the moment for the same reason they cant text out

regards
Paul

TinyExplosions

Paulw wrote: »

While, in theory, your ideas are good, in practice, they're unrealistic, mainly due to cost. You obviously have no understanding of a telecoms network, the number of nodes involved, the number of systems, the configuration and complexity of it all.

Even with the best designed telecoms networks, there are always limited points of potential failure. You can only have so much configuration that is georedundant, especially considering the number of systems involved. A single voice call can need 8 or more systems just to enable the call to connect, never mind all the patch panels, switches, routers, firewalls, etc, and if one is off the air, then a call will fail.

Limited points of potential failure are all very well, but losing entire service because a numpty puts a hole in the ceiling is not really a limited point of failure is it, it's a large, gaping flaw in how the system is set up of all eggs are in one basket (or cage in this case).

If your disaster recovery plan is "panic, then try and rebuild everything, because it'd be too expensive to have redundant backups", then frankly you don't have a recovery plan. The loss of service for an entire day (and it still not being fully back yet) is simply not acceptable in any shape or form - blips in service I can understand and live with, but 9 hrs and counting is beyond a joke imo.

TBH, if it was with one of the larger providers, at this stage I'd be looking to move away from them, and would expect that my contract would be cancelled due to their incompetence, but I'm willing to put this down to getting what I pay for, though if it continues much longer, or happens again I don't think I'll be hanging around.

tall chapy

Paul, I do not doubt your bona fides as all you have done on Boards.ie is help people.
The people who manage your data centre or those responsible for the exterior of the building need to reflect on their operation there.
Roof works on a day it was to rain (above a data centre) !!!!
A contractor piercing the roof is a big no-no and for so much rain to come in through, maybe it was an open upstand issue..
Obviously, the initial piercing was undetected, as it would have been patched/fixed immediately, so the damage was undetected until the sh1t hit the fan and for it to fall on the essential piece of kit that shuts the operation down was unfortunate! Was there no type of mirrored site, that could have been used as a back up. Then again maybe it is not possible...

Texting still not working