UPC’s flaky network down again today, in Cork

Impetus

All services were affected by an outage on UPS’s network serving Cork city today from 13.30 to 15.30. Does this company have any backup for its core infrastructure, one might ask?

UPC’s contention ratio is appallingly high, leading to slow screen fills.

This company is taking advantage of the user, thanks to its monopoly position in "fat pipe infrastructure".

It is time to unbundle broadband services to allow multiple service providers operate over monopoly pipelines (eg cable and fibre optic).

moonboy52

The network was down for 2 hours and you call it flaky?

I find UPC's network to have at least a 99% uptime. It is superb on my end for the last 5 years.

Outkast_IRE

Impetus wrote: »

All services were affected by an outage on UPS’s network serving Cork city today from 13.30 to 15.30. Does this company have any backup for its core infrastructure, one might ask?

UPC’s contention ratio is appallingly high, leading to slow screen fills.

This company is taking advantage of the user, thanks to its monopoly position in "fat pipe infrastructure".

It is time to unbundle broadband services to allow multiple service providers operate over monopoly pipelines (eg cable and fibre optic).

Did you really sign up to just moan about upc.

I am with them about 7 years now and in terms of uptime its about 99.999% .
In 7 years I only recollect it being down 4 or 5 times for a few hours at most.

UPC made the investment in the infrastructure, they can do with it as they like, to force them to share the infrastructure would be madness.

Impetus

“Loyal customers” / employees even... Talking.


The telecommunications industry worldwide works to a “five nines” 99.999% uptime.


Which equates to 5 minutes of failure per annum.


Not 2 hours.


And not an entire city blacked out for two hours.


I have never heard of a two hour city wide failure affecting eircom, Deutsche Telekom, Swisscom, Telefonica, etc.


Hopefully the newly proposed EU telco passport system will be realised soon to bring serious competition to the dozy Irish broadband market. A market where the publicans dilute the whiskey with 20 to 40 x water (the broadband contention ratios).

Outkast_IRE

Impetus wrote: »

“Loyal customers” / employees even... Talking.


The telecommunications industry worldwide works to a “five nines” 99.999% uptime.


Which equates to 5 minutes of failure per annum.


Not 2 hours.


And not an entire city blacked out for two hours.


I have never heard of a two hour city wide failure affecting eircom, Deutsche Telekom, Swisscom, Telefonica, etc.


Hopefully the newly proposed EU telco passport system will be realised soon to bring serious competition to the dozy Irish broadband market. A market where the publicans dilute the whiskey with 20 to 40 x water (the broadband contention ratios).

I am in cork , I wasn't affected on the day your talking about.

And how would I be an employee, anyone with any bit of sense can see from my post history what I am.

Impetus

Were you using the UPC network between 13.30 and 15.30 on 11.09.2013?

When I called UPC to report the issue, there was an RVA stating that ALL services were down in "the Cork area" during that period. The outage is by UPC's own admission.

(Of course the "Cork area" is as vague as mud -- there is no postcode system where they could specify a range of affected postcodes - eg 2000 to 2050 range. Yet another infrastructural deficit in Ireland's communications platform).

Maybe you were lucky, but judging by the postings to this threat it seems to me that everybody seems happy to "roll in the mud" of a second rate service economy. Which is probably why the quality of services of all types from buying a cup of coffee, to the service one gets from a financial institution is generally very poor, with a few notable exceptions (in the coffee space).

keithclancy

Impetus wrote: »

“Loyal customers” / employees even... Talking.


The telecommunications industry worldwide works to a “five nines” 99.999% uptime.


Which equates to 5 minutes of failure per annum.


Not 2 hours.


And not an entire city blacked out for two hours.


I have never heard of a two hour city wide failure affecting eircom, Deutsche Telekom, Swisscom, Telefonica, etc.


Hopefully the newly proposed EU telco passport system will be realised soon to bring serious competition to the dozy Irish broadband market. A market where the publicans dilute the whiskey with 20 to 40 x water (the broadband contention ratios).

UPC are excellent in regards to uptime, have them here in the Netherlands and same in Ireland, simply because they have their own cabling.

DSL in comparison is terribly flaky.

If anything I would say their only fault is billing and customer service, technically they are superb.

I think your expecting business service from a consumer product.

Then again we have had outages on business lines from Deutsche Telecom (Dusseldorf)

2 hours outage on a consumer line .. lol .. if you cannot live with that then get an Eircom Business line in as you have no SLA with the product you are currently using.

Impetus

I have used DSL in Ireland, and I have never had a breakdown that I can recall. I have cable and fibre in other locations too, so I think I am in a position to compare QoS issues.

Have you as an entity or has the entire ort/city of Düsseldorf (PLZ 40210 > 40629) had a citywide telecom failure from DT? If so, when please?

I have my own backup. The key point I was trying to make is that the UPC network covering Cork city was down (for all services). A properly managed phone company would have backup infrastructure to ensure that wide-spread outages do not happen.

Also UPC has not told us what element of the network fabric failed leading to the outage in question.

roast

Impetus wrote: »

A properly managed phone company would have backup infrastructure to ensure that wide-spread outages do not happen.

Out of interest, can you name a service provider in Ireland that has a redundant network in place as a backup?

Paulw

Eircom have had major outages in the past. Full telecoms outage in Dublin (phone/DSL). They have also had international outages (no in/out traffic for the whole country).

And since almost every other DSL provider uses the Eircom network, it effects them too.

UPC have an excellent record. I have used them for the last 7 years and have only had one short outage.

Impetus

Paulw wrote: »

Eircom have had major outages in the past. Full telecoms outage in Dublin (phone/DSL). They have also had international outages (no in/out traffic for the whole country).

And since almost every other DSL provider uses the Eircom network, it effects them too.

UPC have an excellent record. I have used them for the last 7 years and have only had one short outage.

When was this outage on eircom please? Did it affect the entire city or a single switching entity? I did a quick google and can't find anything, except for a DNS server problem (which one could fix by using an alternative such as opendns), and a fault affecting approx. 22,000 PSTN channels in the SE.

dazdrog

Paulw wrote: »

Eircom have had major outages in the past. Full telecoms outage in Dublin (phone/DSL). They have also had international outages (no in/out traffic for the whole country).

And since almost every other DSL provider uses the Eircom network, it effects them too.

UPC have an excellent record. I have used them for the last 7 years and have only had one short outage.

yeah and im sure everyone remembers eircoms near constant DNS issues at one stage

allen175

Been with UPC/NTL for years for broadband and like everyone else never had any serious downtime or intermittent issues, only the odd outage now and again which is fine, its to be expected, no ISP is perfect.

No ISP has a backup infrastructure, there was a huge outage for the south-east about 2 weeks ago that was caused by a eircom fibre cable being dug up, nothing could be done about that other than to repair the cable, think it was out for about 12 hours.

roast

Impetus wrote: »

When was this outage on eircom please? Did it affect the entire city or a single switching entity? I did a quick google and can't find anything, except for a DNS server problem (which one could fix by using an alternative such as opendns), and a fault affecting approx. 22,000 PSTN channels in the SE.

Oh buddy....

This kind of info isn't publicly available, but the amount of outages and the affected footprint in the last year alone would make your toes curl.

Paulw

Impetus wrote: »

When was this outage on eircom please? Did it affect the entire city or a single switching entity?

You won't find it on google.

It took out the majority of Dublin (that's Dublin 1-24 areas).

I am not at liberty to give specific dates, but I am sure it's in the archive of boards.ie if you can find it. There were a few large threads at the time.

Impetus

roast wrote: »

Oh buddy....

This kind of info isn't publicly available, but the amount of outages and the affected footprint in the last year alone would make your toes curl.

Perhaps this information should be available publicly, along with the contention ratios. In the interest of creating an informed marketplace. A job for ComReg. It would lead to an improvement in the QoS for customers, and enable customers to go with the provider that has the lowest contention and breakdown ratios.

When one buys food, the label has to show one nutritional information - saturated fats, carbohydrates, sodium etc in g/100.

A city with say 50,000 customers offline for 2h = 100,000 outage units. 300 customers offline for 7h = 2100 units, etc.

Improvement starts with measurement.

roast

Impetus wrote: »

Perhaps this information should be available publicly, along with the contention ratios. In the interest of creating an informed marketplace. A job for ComReg. It would lead to an improvement in the QoS for customers, and enable customers to go with the provider that has the lowest contention and breakdown ratios.

When one buys food, the label has to show one nutritional information - saturated fats, carbohydrates, sodium etc in g/100.

A city with say 50,000 customers offline for 2h = 100,000 outage units. 300 customers offline for 7h = 2100 units, etc.

Improvement starts with measurement.

The majority of outages are due to faults, issues that are not anticipated.
If it is anticipated that routine maintenance or upgrades may affect a customers connection, they are notified in advance. If an error develops over the course of scheduled maintenance or an upgrade, it is declared a fault/outage and thus considered as such.

If an outage were declared publicly, it would be after the fact.
Yes, it would be a measurement, but I fail to see how it would be an improvement, if the issue has already occurred?

Contention ratios, while not always advertised, are advised if questioned.

Impetus

roast wrote: »

The majority of outages are due to faults, issues that are not anticipated.
If it is anticipated that routine maintenance or upgrades may affect a customers connection, they are notified in advance. If an error develops over the course of scheduled maintenance or an upgrade, it is declared a fault/outage and thus considered as such.

If an outage were declared publicly, it would be after the fact.
Yes, it would be a measurement, but I fail to see how it would be an improvement, if the issue has already occurred?

Contention ratios, while not always advertised, are advised if questioned.

I suspect that most if not all outages are unanticipated. They can be minimised or even eliminated by redundancy in the core network. I am not expecting a telco to provide 100% uptime to an individual subscriber, unless they themselves are paying for that last km of redundancy.

However if a company strings up a "network" and makes no provision for outage in the form of redundant paths and switches, it would be asking for trouble. And imposing trouble on the customer.

Which is why I say that outage performance should be published in a standardised basis routinely. Similarly for contention. It is not true to suggest that contention ratios will be advised to a potential customer on request, in my experience. Eircom for one, does not appear to have provided this information to its sales or support staff. Another favourite lie seems to be "we have no contention".

SpaceTime

Bear in mind that the cable network uses shared trunks that run along buildings / underground from a fibre-supplied local street cabinet. Every house on a given route connects to the same cable. They are not individual cables connected back to a central point like a telephone network.

A problem with one of those trunks can knock out a street or a problem with a cabinet can knock out several streets.

The problem seems to be that UPC gave a "in the Cork area" service message because of that usual Irish mentality that there's "Dublin" which has lots of areas and then other cities/major towns are just one big area to be lumped together when speaking about them.

The UPC service in Cork was not down, the UPC service in a small area of Cork may well have been.

Also, UPC's network's being progressively rebuilt, so some areas of Cork may have much better service levels than others depending on whether the local infrastructure has been replaced or not.

In most of these networks the core is pretty secure and redundant, the problem will usually be some piece of local infrastructure - typically a cable.

roast

Impetus wrote: »

I suspect that most if not all outages are unanticipated. They can be minimised or even eliminated by redundancy in the core network. I am not expecting a telco to provide 100% uptime to an individual subscriber, unless they themselves are paying for that last km of redundancy.

However if a company strings up a "network" and makes no provision for outage in the form of redundant paths and switches, it would be asking for trouble. And imposing trouble on the customer.

Which is why I say that outage performance should be published in a standardised basis routinely. Similarly for contention. It is not true to suggest that contention ratios will be advised to a potential customer on request, in my experience. Eircom for one, does not appear to have provided this information to its sales or support staff. Another favourite lie seems to be "we have no contention".

There is a certain amount of redundancy in place for the majority of networks.

However, you have to take into account the feasibility of switching to a workaround or (for example) a redundant fibre line. Will it take longer to switch routing through a different node than it will to fix the initial problem?

For example - consumer lines. Lets say, a PSTN line. Eircom have an SLA (by standard) of 5 working days to repair a "standard fault". In the event of an outage that's affecting 5,000 PSTN lines, is it quicker to focus on resolving the initial issue, than it is to have half the manpower working on the resolution and half switching everyone to a workaround? Yes, absolutely, if it's just a case that the exchange has been isolated from the core network.

In the case of an outage due to backhaul (be it a fibre feed to an exchange, or a wireless P2P link) in the majority of cases, regardless of redundancy, the issue is unavoidable.

Furthermore, on business/corporate lines, (while it's always advisable for a backup to be in place anyway) in the event of a fault, there may be a shorter SLA in place, depending on the tariff. Regardless, the same problem applies. An advanced SLA is what, two hours? The outage would be resolved within SLA anyway.

Impetus

Surely a modern network is built with re-routing already planned in the system fabric. If one path fails, traffic is automatically re-routed over a working path, until the fault is fixed. Re-routing should involve little or no time or human effort. Terms like "self-healing" and "fault tolerant" come to mind.

roast

Impetus wrote: »

Surely a modern network is built with re-routing already planned in the system fabric. If one path fails, traffic is automatically re-routed over a working path, until the fault is fixed. Re-routing should involve little or no time or human effort. Terms like "self-healing" and "fault tolerant" come to mind.

Easy to do on a small, local network. When you're talking WAN infrastructure, the game changes.

SpaceTime

d

Impetus wrote: »

Surely a modern network is built with re-routing already planned in the system fabric. If one path fails, traffic is automatically re-routed over a working path, until the fault is fixed. Re-routing should involve little or no time or human effort. Terms like "self-healing" and "fault tolerant" come to mind.

Not if the network element that has broken down is a cable running across the eves of 10 houses somewhere in Cork City.

They're hardly going to install two working lines to everyone's house just in case one fails.

The outage you're describing sounds like a very localised cable fault. There were no core network issues on UPC's network in recent months.

The most likely problem there is a broken coaxial cable in the local distribution network, not the core UPC fibre network which would generally speaking be very secure and have a lot of built-in redundancy.

In most networks in Ireland there would be multiple routes from one node to another, and if there was a fault on one of those links then traffic would flow the other way (albeit with possibly reduced capacity) but, on local lines to houses / groups of houses there's very little you can do other than repair the fault within a reasonable time.

....

You should report the fault to UPC and keep reporting it if it reoccurs or if you're getting speed issues.
They may need to replace a faulty cable or a tap or amp somewhere and sometimes these faults can only be identified by complaints.

You can check for explanations to faults here : http://www.upc.ie/customer_support/service_status/

roast

On a side note...
The below is a report from a server I use on a residential UPC connection over the period of a year. 2hrs, 55mins of downtime in total. The majority of this is due to server restarts and maintenance.

Seems good enough to me...

keithclancy

Impetus wrote: »

I suspect that most if not all outages are unanticipated. They can be minimised or even eliminated by redundancy in the core network. I am not expecting a telco to provide 100% uptime to an individual subscriber, unless they themselves are paying for that last km of redundancy.

However if a company strings up a "network" and makes no provision for outage in the form of redundant paths and switches, it would be asking for trouble. And imposing trouble on the customer.

Which is why I say that outage performance should be published in a standardised basis routinely. Similarly for contention. It is not true to suggest that contention ratios will be advised to a potential customer on request, in my experience. Eircom for one, does not appear to have provided this information to its sales or support staff. Another favourite lie seems to be "we have no contention".

If you want to be protected from something like that then get a business line.

You are using a consumer product.

Impetus

roast wrote: »

Easy to do on a small, local network. When you're talking WAN infrastructure, the game changes.

Rubbish! When one is running a big network, there is plenty of money to pay for alternative routing strategies. If UPC or eircom or anybody else was a tiny village network, your "excuse" might be valid.

roast

Impetus wrote: »

Rubbish! When one is running a big network, there is plenty of money to pay for alternative routing strategies. If UPC or eircom or anybody else was a tiny village network, your "excuse" might be valid.

Again, read my previous responses. As rubbish as it may seem to you, these are facts.

Impetus

SpaceTime wrote: »

d

Not if the network element that has broken down is a cable running across the eves of 10 houses somewhere in Cork City.

They're hardly going to install two working lines to everyone's house just in case one fails.

The outage you're describing sounds like a very localised cable fault. There were no core network issues on UPC's network in recent months.

The most likely problem there is a broken coaxial cable in the local distribution network, not the core UPC fibre network which would generally speaking be very secure and have a lot of built-in redundancy.

In most networks in Ireland there would be multiple routes from one node to another, and if there was a fault on one of those links then traffic would flow the other way (albeit with possibly reduced capacity) but, on local lines to houses / groups of houses there's very little you can do other than repair the fault within a reasonable time.

....

You should report the fault to UPC and keep reporting it if it reoccurs or if you're getting speed issues.
They may need to replace a faulty cable or a tap or amp somewhere and sometimes these faults can only be identified by complaints.

You can check for explanations to faults here : http://www.upc.ie/customer_support/service_status/

Read the thread.... UPC admitted that the fault was service-wide and city-wide. Not local.

Impetus

roast wrote: »

Again, read my previous responses. As rubbish as it may seem to you, these are facts.

http://www.open-mpi.org/papers/dapsys-2006-self-healing-network/dapsys-2006-self-healing-network.pdf

Impetus

keithclancy wrote: »

If you want to be protected from something like that then get a business line.

You are using a consumer product.

Consumer or business user, one is entitled to a quality of service. Hence the need for publicly available service reliability and contention ratio statistics for all, from ComReg. Give people the information they should have to make an informed choice.

cali_eire

Impetus wrote: »

“Loyal customers” / employees even... Talking.


The telecommunications industry worldwide works to a “five nines” 99.999% uptime.


Which equates to 5 minutes of failure per annum.


Not 2 hours.


And not an entire city blacked out for two hours.


I have never heard of a two hour city wide failure affecting eircom, Deutsche Telekom, Swisscom, Telefonica, etc.


Hopefully the newly proposed EU telco passport system will be realised soon to bring serious competition to the dozy Irish broadband market. A market where the publicans dilute the whiskey with 20 to 40 x water (the broadband contention ratios).

To be fair I just moved back from Silicon Valley in California and our UPC equivalent there which is called Comcast regularly had outages so I wouldn't be so quick to think far fields are greener.

roast

Impetus wrote: »

http://www.open-mpi.org/papers/dapsys-2006-self-healing-network/dapsys-2006-self-healing-network.pdf

Thanks for the link. Don't worry, I understand the concept. :pac:

Impetus

cali_eire wrote: »

To be fair I just moved back from Silicon Valley in California and our UPC equivalent there which is called Comcast regularly had outages so I wouldn't be so quick to think far fields are greener.

Anyone who has their ear to the ground knows that US telecommunications stink to high heaven. Mobile is extortionately priced, as is cable TV. And there is zero competition. When an incumbent's copper pair faces competition, they cut the copper for good. Leaving the victim to the mercy of Comcast or similar.

Watch: live.twit.tv or on demand at www.twit.tv See all the problems they (and the co-hosts situated all over) have with Skype - even though they have a dedicated "business class" fiber connection from the carrier exchange almost next door to their Brickhouse studio.

So yes, IRL could be in a worse position, a la USA. But that is no excuse. Many parts of Asia have outpaced us on internet services, and they are heading to outpace Europe and the US on virtually every front - cars, telecommunications, financial services, environmental systems, transportation infrastructure, etc. And most of the people posting to this thread seem to be either industry insiders or fat and happy morons, loyal customer types... Ugh.... What hope for the future?

Impetus

cali_eire wrote: »

To be fair I just moved back from Silicon Valley in California and our UPC equivalent there which is called Comcast regularly had outages so I wouldn't be so quick to think far fields are greener.

PS one neglected to mention that UPC's parent co is based at 12300 Liberty Blvd. Englewood CO 80112. Not CA, but not far away.

One wonders what Edward Snowden has in his files on UPC?

roast

Impetus wrote: »

And most of the people posting to this thread seem to be either industry insiders or fat and happy morons, loyal customer types... Ugh.... What hope for the future?

I work for a company which is in no way affiliated to UPC whatsoever. Your "fat and happy moron" comment is a little offensive to those who have taken the time out to reply to your topic, in the spirit of the debate. However your replies don't seem to discuss the points raised properly, and just come across as whining. :rolleyes:

Can I ask, what industry do you work in?

dub45

SpaceTime wrote: »

d

Not if the network element that has broken down is a cable running across the eves of 10 houses somewhere in Cork City..............................................


You can check for explanations to faults here : http://www.upc.ie/customer_support/service_status/

Unfortunately that is not fully true as the on line status checker is not updated just when it is needed most! UPC have acknowledged that it is not updated over night despite encouraging people to believe that it is " live" on their website.

This means for example that during recent lengthy faults in Dublin people were being given wrong information with the consequences that some people thinking that there is no fault will then begin to "mess" with their settings etc.

http://www.boards.ie/ttfthread/2057030795

dub45

roast wrote: »

On a side note...
The below is a report from a server I use on a residential UPC connection over the period of a year. 2hrs, 55mins of downtime in total. The majority of this is due to server restarts and maintenance.

Seems good enough to me...

There were three lengthy UPC outages in areas of Dublin in recent times:

Two on successive nights:

18/19 June approx 10 hours - Midnight to 10 am

19/20 June approx 12.5 hours - 23.30 to 12 Noon

27/28 August approx 9.5 hours - 23.30 - 9 am

No causes given and UPC refuse to say when repairs start in the event of such faults.

keithclancy

Impetus wrote: »

Consumer or business user, one is entitled to a quality of service. Hence the need for publicly available service reliability and contention ratio statistics for all, from ComReg. Give people the information they should have to make an informed choice.

You are 'entitled' to what you agreed to in your contract.

Perhaps change provider if you are unhappy with the service.

Consumer product .... if you are losing money in some way due to an outage then you should have an SLA in place backed up with a contract.

If what you ask existed then the price of the product would rise considerably.

Impetus

keithclancy wrote: »

You are 'entitled' to what you agreed to in your contract.

Perhaps change provider if you are unhappy with the service.

Consumer product .... if you are losing money in some way due to an outage then you should have an SLA in place backed up with a contract.

If what you ask existed then the price of the product would rise considerably.

There is no competition for "fat pipe" internet, for the most part, unless one lives in an apartment on top of a telephone switch. UPC has a monopoly, and is getting away with abusing this monopoly. In a properly regulated market, the fat pipe, (ideally fiber) would be open to all service providers, and a customer could then change easily, if s/he was dissatisfied.

[Eircom's eFibre service is a mis-nomer as it relies on copper for the often/usually long, bendy trip from the VDSL2 cabinet to the point of use, the distance of which kills the performance of VDSL in all but lucky cases].

In other countries the telcos often offer the end-user FTTP from the cabinet as an extra cost option, or provide FTTP to everybody - replacing their copper network entirely (the lowest medium and long term cost option because one is not running and maintaining two network infrastructures). Example Andorra - andorratelecom - which is a state monopoly - fiber coverage is 100% of the country, including to mountain locations 20km from the nearest switch - a far longer distance than any flavour of DSL could perform in.

If there was fault reporting and contention ratio transparency as I have advocated I don't believe it would increase the cost. Rather it would help to eliminate the cowboys who make exaggerated claims for the speed or other characteristics of their "service". It would tend to replace these "rubbish claims" with more substantial claims of who has the less faults and the lowest contention ratio (diluting the whisk(e)y with water) - forcing the industry to compete in this direction - ie delivering real QoS.

Impetus

cali_eire wrote: »

To be fair I just moved back from Silicon Valley in California and our UPC equivalent there which is called Comcast regularly had outages so I wouldn't be so quick to think far fields are greener.

Adam Curry (in Austin) and John C Dvorak (in Silicon Valley) do a bi-weekly netcast, and below is a link to their show that aired on 8.9.2013 – listen to all the breaks in their internet connection….. Many of these breaks are probably eliminated from the on demand download link below. ASFAIK Dvorak uses Comcast – don’t know which cable company Curry uses – though I suspect it is Comcast too.

I gave up listening to that Sunday show live, and emailed them suggesting that they use ISDN instead. Dvorak emailed me back a story on how expensive ISDN is – about $450 per month for each of them in subscription charges – which is prohibitive. In Europe one can get 2 channel ISDN for around 25 to 30 € with 64k bandwidth guaranteed per channel. Not so in the US.

Download audio: http://mp3s.nashownotes.com/NA-546-2013-09-08-Final.mp3

Listen live on Sunday and Thursday afternoons (about 18:00 IST). http://noagendastream.com

Show archive: http://feed.nashownotes.com/rss.xml