Blueface Off

Freddie59

Blueface one all day here in Waterford (City Wide - others affected). Anyone else out?

GerardKeating

Freddie59 wrote:

Blueface one all day here in Waterford (City Wide - others affected). Anyone else out?

A comment on the blueface forum indicates an upgrade is in progress

bhickey

Still out this morning in Galway. They seemed to go down about 13:35 yesterday.

dubadubdub

Line is still nor working (Dublin number), I'm beginning to see why Blueface calls are cheap; you can't make them! Crazy. :mad:

bhickey

Seems to be okay now again.

bhickey

Uh-oh! Spoke too soon. They're gone again now. This doesn't look good.

sirlinux

in the few minutes it was up looked like all the voice mailboxes are gone/wiped

dubadubdub

Does anyone have a service? Blueface support claims that some people do.

I've posted the following on Blueface's own forum but I won't hold my breath waiting for a reply!


Blueface's inability to give even a rough guess at when the system will be back up and running is very worrying. It's beginning to look like they have a major problem or else they don't know what they're doing; take your pick.

This is from their 'terms of service' :

"Blue Face will endeavour to ensure that the service has a high degree of reliability, although no guarantee is made that service will be available in the event of power outages or other unforeseen events beyond the control of Blue Face".

But it seems like this problem was caused by a badly planned upgrade.

Goodwill is hard won but easily lost. Some hard information about the problem would go someway towards easing my mind about using Blueface in the future.

fatherdougalmag

They claim that the outtage and the uprgade were purely coincidental. I, too, am a little suspicious of the claim that only some people are affected. Can't even get throug to their 436 1700 number. I'm starting to evaluate alternatives.

olearydc

Shame

I like to think that this company are on the ball
Have had a few people go to them on my recommedation

Been using Blueface for over 6 months...;) Pretty good to that point

lets hope that things are sorted soon....

and a explanation of what happened (maybe a email to people to let them know and to for-warn them next time!)

paulm17781

I've been with them for nearly a year and a half and generally am very happy. I am a little bit annoyed by the lack of information coming from them though. Working with hardware I know the kind of unforseen problems that can occur but I think I, like everyone else, would like a decent explanation of the problem.

dubadubdub

FYI:

"under the terms of our customer service guarantee it is our aim to clear all faults within 2 working days, if we fail to meet this deadline you can claim the equivalent of two months line rental".

That's from Eircom's terms of service. Seems better that Blueface's; which seems to be 'keep turning it on and off and if that doesn't work then wait'.

msmx5

dubadubdub wrote:

That's from Eircom's terms of service. Seems better that Blueface's; which seems to be 'keep turning it on and off and if that doesn't work then wait'.

In fairness to Blueface, you get what you pay for.

We all knew before we signed up (or should have) that there are far too many variables in the pot to expect anywhere near carrier class service from a VOIP provider over the public internet. Local power, ISP connection, ISP Bandwith, Local network activity, contention ratios, . . the list goes on.

We know that resource wise Blueface is quite small, that’s how they can offer the cheap rates that they do, I'm sure they don't have a vast technical department with 24hr availability at their beck and call.

Personally I'd prefer hearing nothing or "as soon as possible" rather than Blueface giving a definite time for restoration of service when they have nothing to base that on.

By the sounds of it they [Blueface] are fire fighting to get service restored. I'm sure they have all hands on deck - hence the lack of CS information. Personally if I was in Blueface's shoes - (after this episode maybe they should rebrand to Redface :mad:) - I too would put restoration of the service higher up the agenda than constant customer service updates.

It is possible to forecast and notify customers of outage times for planned upgrades and preventitave maintenance, but it is next to impossible to forecast how long it will take to track down a fault, isolate the hardware, obtain and configure suitable replacements to restore service when the faults relate to serious or intermittent h/w issues.

This outage certainly has damaged Blueface's reputation with both existing and potential customers. How they deal with the aftermath and what redundancy and safeguards them implement in their network to avoid it happening again will be key to their survival as a reputable VOIP provider, I think.

If they had any disaster recovery plan in place it has proved to be patently inadequate! Maybe its the case that their customer base has expanded more quickly than their infrastructure was dimensioned for, but I honestly can't see that having such sudden and catastrophic affects.

Service is up and down like a wh...'s knickers still, most of the time incoming calls aren't even diverted to voicemail. Sometime, when they are diverted it happens without ringing and the caller hears a generic announcement with the mailbox/blueface extension number, which obviously means nothing to the caller expecting my personalised announcement.

Things are stablising as the day goes on but Blueface are far for being able to state that even near normal service has been resumed..... the missus in on the blower to me regularly and is putting alot of pressure on me to cancel and revert to a traditional fixed line, running up her mobile bill in the process!

If only their Netgear ATA allowed for a 2nd VOIP provider I'd at least have a fallback for time like this!

shltter

Thankfully the linksys does allow for another service provider

Anyone know the procedure for moving your phone number to another VOIP operator

paulm17781

msmx5 wrote:

We know that resource wise Blueface is quite small, that’s how they can offer the cheap rates that they do, I'm sure they don't have a vast technical department with 24hr availability at their beck and call.

I work for a small company, people are often on 24hour support and often will come into work the next day after being up most of the night.

Perhaps Blueface are doing this, I don't know. I just want my phone back.

dubadubdub

shltter wrote:

Thankfully the linksys does allow for another service provider

Anyone know the procedure for moving your phone number to another VOIP operator

If it's a geographic number then you have to port back to eircom and then port to the new VOIP operator. This is what passes as deregulation in this country! :eek:

I'm a Blueface customer too but I'm not too peed. I expect any computer-related technology to give grief like this eventually, I know all too well from my former job as a repair tech. I think even digital television has its woes in the same way.

I wouldn't recommend anyone to permanently switch to VoIP if they're serious about their phones, the technlogy just isn't stable enough.

fatherdougalmag

I think the people who will be most peed are (a) business owners who run their company's phone lines with VoIP or (b) techno-slut husbands who love to be on the cutting edge but can't seem to get their wife to see things the same way.

I fall into (b).

unkel

It's been on and off for me since yesterday afternoon. 95% off though...

fatherdougalmag

http://forums.blueface.ie/showthread.php?t=236

Freddie59

Everyone - let's not get carried away with the criticism. This is the first real problem I've had in 8 months. And remember - in the last 12 months two high profile national mobile networks have suffered outages of a day or more.....and Blueface don't charge anything remotely like they do. Just a thought.;)

paulm17781

Freddie59 wrote:

Everyone - let's not get carried away with the criticism. This is the first real problem I've had in 8 months. And remember - in the last 12 months two high profile national mobile networks have suffered outages of a day or more.....and Blueface don't charge anything remotely like they do. Just a thought.;)

I'd have to agree Freddie. Some people will just complain when they get angry and some are just bitches. Sadly the bitches will always be loudest most profile and that is why they are bitches.

The downtime was annoying but I doubt Blueface said "Lets take our network down for two days so we get no traffic, lose revenue and piss off customers. That'll show them." However I'm sure people will still bitch for another few months about this.

ps. I don't now and never did work for Blueface

Blaster99

Freddie59 wrote:

Everyone - let's not get carried away with the criticism. This is the first real problem I've had in 8 months. And remember - in the last 12 months two high profile national mobile networks have suffered outages of a day or more.....and Blueface don't charge anything remotely like they do. Just a thought.;)

Mobile networks might just be a tad bit more complicated and expensive to keep running than a couple of servers in a data centre. Just a thought.

It doesn't look like any compensation is forthcoming so I'm cancelling. Plenty of alternatives out there.

JimmySmith

I called a 2 friends on blueface and got their voicemail, which is the bog standard blueface message.
Have contacted noth since and they tell me that their recorded greetings are gone.
Just lettng people know to check theirs.

paulm17781

Blaster99 wrote:

Mobile networks might just be a tad bit more complicated and expensive to keep running than a couple of servers in a data centre. Just a thought.

It must be great to always be right. However mobile networks aren't a whole hell of a lot more complex than that, I work with them.

Secondly, does it matter? An outage is an outage.

shltter

paulm17781 wrote:

I'd have to agree Freddie. Some people will just complain when they get angry and some are just bitches. Sadly the bitches will always be loudest most profile and that is why they are bitches.

The downtime was annoying but I doubt Blueface said "Lets take our network down for two days so we get no traffic, lose revenue and piss off customers. That'll show them." However I'm sure people will still bitch for another few months about this.

ps. I don't now and never did work for Blueface

On the other hand they only lose revenue from PAYG customers those of us on a subscription service have paid for a service that Blueface was unable to provide for 2 days this month.
They are not offering any rebate or compensation for the lack of service why should Blueface expect to keep money for a service that they were unable to provide the reason for the failure is irrelevant.
I know it is only a small ammount of money but that is not the point it is the gesture more than the actual ammount of money.

Maxminon

I just like to say that I think the whole issue with the loss of the Blueface service for 2 full days should be a good lesson for Blueface.That is they have seen their business grow over the past few months and this has been purely from word of mouth, (tell a friend ect)Now they can see what it is really like to run a business and provide a service that people subscribe to.
If you take away a service that customers are paying for you have to be prepared to take the complaints that follow.I also think that if they had been more up front and honest with people about this breakdown and I say again breakdown not an upgrade as they were saying from the start their customers would not have been pisssssssss off as they were.I have added a copy of an apology from Aaron Clauson the Chief Technical Office at Blueface explaining all in relation to this breakdown and I say again breakdown not upgrade.
All I have to say to Aaron honesty is the best policy by far in the end.
Quote by Aaron.
Hello
First we would like to apologise on behalf of Blue Face for our recent downtime and to any customers that have been unhappy with the information that they received on the cause of this downtime.

Our main systems are now mostly fully operational and we anticipate things will shortly be fully stable and back to normal. There are a few areas we are still working to bring back online such as voicemail and a fix for a registration drop some customer's are experiencing but the main call services are now fully functional. We anticipate all outstanding systems will be back up by Friday morning and hopefully well before.

We would also like to assure customers that this incident has been worked on around the clock by Blue Face staff and was not the result of a scheduled systems upgrade.

We will be providing a full fault report on this incident to any concerned customers after the all clear is given. We realise that the fault report does not excuse the systems failure and once we have systems running normally will be undertaking a full analysis to ensure the possibility of a similar failure in the future is minimised.

A brief synopsis of the fault is:

- Disk failure and irretreivable system crash on main Blue Face web server on Monday the 14th of August,

- The result of the web server failure and rebuild was an increase in load on our two main database servers as systems were re-deployed to cope with the web server crash,

- Around Friday the 18th of August some erratic behaviour was logged on our main Asterisk and SIP Proxy servers,

- Erratic behaviour continued and was traced to a degradation in performance of one of our two main databases,

- Measures to improve the database performance were not successful and on Tuesday the 29th of August an emergency software upgrade was introduced to attempt to lessen the database load and keep critical systems operational,

- In parallel with this activity our recently purchased Sun Database with significantly superior performance was accelerated into production,

- At approx. 4am on the 30th of August the Sun Database was bought fully into service and took over the load of our main Asterisk and SIP systems and system functions were largely returned to normal. This operation was complicated by the database being located in our second data center location which was being bought online for increased fault tolerance,

- At approx. 1130 on the 30th of August there was another malfunction and it was traced to erratic performance on the Sun Database, this was followed by a full system crash which lasted for approx. 2 hours. During this time no call services were available,

- At approx 1330 on the 30th of August the Sun Database problem was resolved and traced to what appears to be a faulty hot swap disk. This machine is fully redundant on all major hardware sub-systems and our hypothesis is that the recent server move may have left the Hot Swap disk incorrectly seated so that it somehow affected the server's operation,

- Subsequent to 1330 on the 30th of August call services were returned to normal and work has continued on optimising our SIP Registrar to ensure registrations are not dropped. The registrar is required to be located near the authentication database and this required the server be moved to the second data center.

Due to the rapid changes in the situation over the 29th and 30th of August it was not possible to keep all our support staff fully abreast of the situation at all times and we regret if inadequate information was provided to customers.

We would like to thank all customers for their patience and understanding as we deal with this issue.

Thanks,

Aaron Clauson
Chief Technical Office
Blue Face
__________________
"And if you gaze for long into an abyss, the abyss gazes also into you." Friedrich Nietzsche

bhickey

Service is down again this morning for me. If it's because they're doing planned maintenance work at this time on a Saturday morning then I'm almost happy in a way if it means that from next Monday things could be back to normal.

Maxminon

Just to let ye know that it has been reported back to me that some Eircom customers in Waterford City Dunmore Road area have had no phone service this morning and the service was still off up to 5.13pm today.There was also problems with their service on Friday.Good to know that it is not just Blueface that are having problems providing a service.

Maxminon

Maxminon wrote:

Just to let ye know that it has been reported back to me that some Eircom customers in Waterford City Dunmore Road area have had no phone service this morning and the service was still off up to 5.13pm today.There was also problems with their service on Friday.Good to know that it is not just Blueface that are having problems providing a service.

Now that you say that it reminds me of something that happened here about 9 years ago.

A tractor overturned on a road about a half mile from my house, tearing down the overhead phone line in the process (the driver was fine though). Anyway, it took the then TÉ four days to repair it during which we had no phone service at all.

msmx5

Karsini wrote:

........
Anyway, it took the then TÉ four days to repair it during which we had no phone service at all.

Those were the days when our internet access depended on out phone line! Now its the other way around so when our VOIP provider has an outgage we can log on and bitch about it! :-) Luckily we can also use a backup VOIP provider for outgoing calls!

Apart from some flakey voice quality and a few failed registrations just after Blueface service was restored, I've had no problems at all.

Now do I re-record my voicemail announcemnts or do I hold on in the hope of them being restored automaticlly?

bhickey

Incoming calls are down for a lot of Blueface users this morning. There's nothing from Blueface on their forums and no response from support.

Cabaal

bhickey wrote:

Incoming calls are down for a lot of Blueface users this morning. There's nothing from Blueface on their forums and no response from support.

Posted 20min ago
http://forums.blueface.ie/showthread.php?t=266