Tesco Mobile

Tesco Mobile: Paul

Hi All, the technology team are close to resolution but it could be an hour. I am going to move base from the office to home so will be offline for the next hour while I drive home. Will update again at 8pm when hopefully there will be better news. regards Paul

shamwari

Tesco Mobile: Paul wrote: »

Hi All, the technology team are close to resolution but it could be an hour. I am going to move base from the office to home so will be offline for the next hour while I drive home. Will update again at 8pm when hopefully there will be better news. regards Paul

Safe home and have a good weekend. You and the "rescue team" deserve one..;)

Tesco Mobile: Paul

tall chapy wrote: »

Paul, I do not doubt your bona fides as all you have done on Boards.ie is help people.
The people who manage your data centre or those responsible for the exterior of the building need to reflect on their operation there.
Roof works on a day it was to rain (above a data centre) !!!!
A contractor piercing the roof is a big no-no and for so much rain to come in through, maybe it was an open upstand issue..
Obviously, the initial piercing was undetected, as it would have been patched/fixed immediately, so the damage was undetected until the sh1t hit the fan and for it to fall on the essential piece of kit that shuts the operation down was unfortunate! Was there no type of mirrored site, that could have been used as a back up. Then again maybe it is not possible...

Texting still not working

Regarding the works in the data centre that caused this its not something I can really go into in public but yes there will be many questions when we get to root cause analysis.

At the moment the focus is 100% on restoring service.

Paulw

TinyExplosions wrote: »

Limited points of potential failure are all very well, but losing entire service because a numpty puts a hole in the ceiling is not really a limited point of failure is it, it's a large, gaping flaw in how the system is set up of all eggs are in one basket (or cage in this case).

The loss of service for an entire day (and it still not being fully back yet) is simply not acceptable in any shape or form - blips in service I can understand and live with, but 9 hrs and counting is beyond a joke imo.

TBH, if it was with one of the larger providers, at this stage I'd be looking to move away from them, and would expect that my contract would be cancelled due to their incompetence, but I'm willing to put this down to getting what I pay for, though if it continues much longer, or happens again I don't think I'll be hanging around.

But, there hasn't been a full outage for 9 hours, it's actually a partial outage, from what posters have been saying (on another site too). I don't use Tesco mobile myself, so can't confirm, but my mother uses it and she hasn't had much of a problem (I phoned her on her mobile to check). A few call failures, but a retry gets through. She doesn't really text much anyway, so no idea about that.

Similar has happened before in Ireland. I remember Eircom being offline twice, where the whole of Dublin was without service (1999 and 2002, I think). It has also happened in the UK (to two major mobile operators since 2000).

This could happen to any phone operator. As I said, due to the complexity of telecoms, it is almost impossible to have a fully redundant network operation.

As for your contract, you couldn't just walk away. There are actually provisions in the contract to protect the operator. You should check the fine print.

And, just for the record, I don't work in a data centre, and don't work for Tesco mobile.

JTMan

Back many years ago, when Meteor went down, in it's early days, they gave a 'goodwill gesture' of free phone credit to users.

A similar gesture, by 48 & Tesco, might be an expensive move, but might placate some users, and prevent a spike in porting.

At the end of the day, it will stick in customers minds that 48/Tesco effected the start of their Paddys weekend, customers will tell mates in the pub tomorrow. Vodafone customers will then brag back that they got free calls for Paddys day and no outage. The story needs a silver lining for Tesco and 48.

chris2008x

Tesco Mobile: Paul wrote: »

Hi Cymbaline,

Yes we will do this on Tuesday most likely. We actually cant text them at the moment for the same reason they cant text out

regards
Paul

So it will be Tuesday before I can text? That is what I use my phone for 90% of the time. I have to use my other phone now which has an expensive price plan.

Aren't you obliged to notify customers by any means calls/texts (well obviously not texts at the moment) if the service is down for 48 hours?

Also the network 48 go conqour is down.

Stavro Mueller

Tuesday???? I appreciate that your network isn't working properly at the moment but why do your customers who don't know what's wrong have to wait til then for an explanation?

And if I can't text til Tuesday, I'll be taking my business elsewhere :mad::mad::mad:

JTMan

Tuesday clearly refers to the day they are back in their office after Paddys weekend. I assume it does not refer to an estimation as to when the network will be fully restored. That been said, surely, the text message to customers could be sent sooner than Tuesday.

chris2008x

They are mobile network. They don't take the weekend off before St Patrick's day. If texting is back tomorrow then surely someone will be there to do it.

manutd83

cant believe the service still hasnt been fully restored,ridiculous

krudler

manutd83 wrote: »

cant believe the service still hasnt been fully restored,ridiculous

you read the reason why its out right?

manutd83

yeah i have and its not as if they dont have the money to get as many people as they need to have it back asap,

manutd83

oh and im not sure i believe 100% what has happened

krudler

manutd83 wrote: »

yeah i have and its not as if they dont have the money to get as many people as they need to have it back asap,

yeah they have a fleet of magicians on staff for this

Bad Panda

manutd83 wrote: »

yeah i have and its not as if they dont have the money to get as many people as they need to have it back asap,

I work in the industry. You clearly don't. You've no idea the amount of work involved and money isn't simply going to fix it! Cop on !

Really ridiculous what people expect, especially given that they've no idea what they're talking about.

JTMan

Some interesting postings on Twitter about this.

Very frequent stream of updates from 48 on the issue here:
https://twitter.com/#!/48GoConquer

48 have said:

Guys we’re still working on it, we won’t be leaving until it’s fixed ~Gareth at 48

Meanwhile Tesco on Twitter:
https://twitter.com/#!/tescomobileire

Tuesday next we'll be live on Twitter ..... Enjoy the long weekend!

Good god. :rolleyes:

Tesco need to learn from 48 on how to handle this on Twitter. Should be going live now and not on Tuesday.

manutd83

Bad Panda wrote: »

I work in the industry. You clearly don't. You've no idea the amount of work involved and money isn't simply going to fix it! Cop on !

Really ridiculous what people expect, especially given that they've no idea what they're talking about.

i dont think its ridiculous to expect that when you pay for a service that you get the service you paying for,iv never seen a network down for as long,well other than eircoms broadband

2012IE

just to add another perspective, i was porting from 48 to O2 at around 1300
port went through very quickly, now i am numberless, calls and texts to my number i ported to o2 from 48 dont work, when someone tries to call my number despite signal on the o2 sim, they get the message that the "phone is off", and no inbound texts are received.


nice one 48 / o2 / tesco mobile, phone is down since 1300 with no service of any kind...

go conquer me ********. :mad:

try to port away and my weekend is fecked up. :eek:

when will this issue be resolved? staff should be working flat out until all the issues are resolved, considering o2 are behind all the networks involved!

JTMan

Latest post/updated post from 48:

Guys, the latest update is that our suppliers and engineers are still working to fix these problems and we are in constant communication with them. It’s Friday night, it’s St Paddy’s weekend and we know it’s not good enough. We’ll continue to keep you updated and we’re working flat out to get this sorted. Just to remind you that you can make and receive calls and also receive SMS.

Paulw

2012IE wrote: »

when will this issue be resolved? staff should be working flat out until all the issues are resolved, considering o2 are behind all the networks involved!

I'm sure the right staff ARE working flat out to get full service restored.

Then again, maybe they're all sitting around having a pint in the local on a Fri evening, and don't care. :rolleyes:

It will be resolved when the right people can do their job and restore service. I'm sure it will be done as quickly as they possibly can, and I'm sure it will be worked on continuously until then. They're much more likely to be working on fixing the problem than on here posting updates every few mins saying that they are working on it.

2012IE

Paulw wrote: »

I'm sure the right staff ARE working flat out to get full service restored.

Then again, maybe they're all sitting around having a pint in the local on a Fri evening, and don't care. :rolleyes:

It will be resolved when the right people can do their job and restore service. I'm sure it will be done as quickly as they possibly can, and I'm sure it will be worked on continuously until then. They're much more likely to be working on fixing the problem than on here posting updates every few mins saying that they are working on it.

o2 are telling me it will be tomorrow again someone looks at it,

Tesco Mobile: Paul

chris2008x wrote: »

So it will be Tuesday before I can text? That is what I use my phone for 90% of the time. I have to use my other phone now which has an expensive price plan.

Aren't you obliged to notify customers by any means calls/texts (well obviously not texts at the moment) if the service is down for 48 hours?

Also the network 48 go conqour is down.

Sorry that is not what I meant. I meant we likely wont text the entire base on the issue until Tuesday

The plan is to resolve the issues this evening and restore service to everyone.

Not sure what obligation you refer to.

regards
Paul

Tesco Mobile: Paul

cymbaline wrote: »

Tuesday???? I appreciate that your network isn't working properly at the moment but why do your customers who don't know what's wrong have to wait til then for an explanation?

And if I can't text til Tuesday, I'll be taking my business elsewhere :mad::mad::mad:

Not what was meant or even said.

Tesco Mobile: Paul

Fungus wrote: »

Some interesting postings on Twitter about this.

Very frequent stream of updates from 48 on the issue here:
https://twitter.com/#!/48GoConquer

48 have said:


Meanwhile Tesco on Twitter:
https://twitter.com/#!/tescomobileire



Good god. :rolleyes:

Tesco need to learn from 48 on how to handle this on Twitter. Should be going live now and not on Tuesday.

In all fairness Fungus, our Twitter service was planned to be launched next Tuesday and the resources are scheduled for that. Then today happens which was obviously unexpected.

We dont have the followers on Twitter yet anyway until we launch so we would have been shouting into the dark.

I am doing my best to update here, on Facebook and on our Website. These are the three digital locations we have with Twitter on the way next week (manned by customer care in office hours) and Talk To Boards on the way in a couple of weeks hopefully.

regards
Pau

Tesco Mobile: Paul

Paulw wrote: »

I'm sure the right staff ARE working flat out to get full service restored.

Then again, maybe they're all sitting around having a pint in the local on a Fri evening, and don't care. :rolleyes:

It will be resolved when the right people can do their job and restore service. I'm sure it will be done as quickly as they possibly can, and I'm sure it will be worked on continuously until then. They're much more likely to be working on fixing the problem than on here posting updates every few mins saying that they are working on it.

There are a team of engineers in the data centre working on this. I will saty online to update everyone as long as there are updates to give.

regards
Paul

Tesco Mobile: Paul

2012IE wrote: »

o2 are telling me it will be tomorrow again someone looks at it,

They are referrring to your port and not our major network incident.

As you were porting from 48 to O2 you need to speak with those operators.

regards
Paul

Cromwelly

You do a really great job Paul, but you seem to be the only person in Tesco Mobile Ireland providing support, often out of hours.

The powers that be need to get you some help.

They also need a DR plan and a second data centre for failover, but they just figured that one out I guess.

You might update Facebook when you get a sec...

Tesco Mobile: Paul

Data is coming back online now. You may need to go to airplane mode and back.

regards
Paul

wmits

Tesco Mobile: Paul wrote: »

Data is coming back online now. You may need to go to airplane mode and back.

regards
Paul

Thanks all

Cromwelly

SMS send is working