Advertisement
If you have a new account but are having problems posting or verifying your account, please email us on hello@boards.ie for help. Thanks :)
Hello all! Please ensure that you are posting a new thread or question in the appropriate forum. The Feedback forum is overwhelmed with questions that are having to be moved elsewhere. If you need help to verify your account contact hello@boards.ie
Hi there,
There is an issue with role permissions that is being worked on at the moment.
If you are having trouble with access or permissions on regional forums please post here to get access: https://www.boards.ie/discussion/2058365403/you-do-not-have-permission-for-that#latest

Shannon Radar down

  • 02-10-2018 7:32pm
    #1
    Registered Users, Registered Users 2 Posts: 10,296 ✭✭✭✭


    Just after 7pm Shannon's main radar went tech. As a result no movements into and out of Shannon and knock until at least 10pm. Aer Lingus from Knock to Gatwick was on the runway and had to return to stand. Looking at Flight radar its like Xmas day currently


Comments

  • Registered Users, Registered Users 2 Posts: 352 ✭✭jasT1981


    Looks like no flights over all of ROI right now.
    LIVE: The Shannon FIR region EISN has suffered a complete radar data failure, with no flights allowed over Southern Ireland ('zero rate'). EuroControl states an up-time of 2200UTC (subject to change). Listen


  • Posts: 0 [Deleted User]


    Dublin is in the republic as well.


  • Registered Users, Registered Users 2 Posts: 6,562 ✭✭✭kub


    Just after 7pm Shannon's main radar went tech. As a result no movements into and out of Shannon and knock until at least 10pm. Aer Lingus from Knock to Gatwick was on the runway and had to return to stand. Looking at Flight radar its like Xmas day currently


    Is this affecting Cork as well? Just inbound and outbound from there is controlled by Shannon also.


  • Registered Users, Registered Users 2 Posts: 6,562 ✭✭✭kub


    Dublin is in the republic as well.


    On the FR App planes can be seen over Dublin and a little south of it over The Irish Sea. Nothing to the Wast of Dublin Airport.


  • Registered Users, Registered Users 2 Posts: 10,296 ✭✭✭✭Oscar Bravo


    kub wrote: »
    Is this affecting Cork as well? Just inbound and outbound from there is controlled by Shannon also.

    Yeah, EI724 to LHR cancelled.


  • Advertisement
  • Closed Accounts Posts: 1,112 ✭✭✭notharrypotter


    Just after 7pm Shannon's main radar went tech. As a result no movements into and out of Shannon and knock until at least 10pm. Aer Lingus from Knock to Gatwick was on the runway and had to return to stand. Looking at Flight radar its like Xmas day currently

    SHANNON FIR covers Donegal, Sligo, Knock, Shannon, Kerry, Cork and Waterford.
    So no arrivals/departures from those airport's untill the system is restored.
    A clear the skies protocol operates in the Shannon FIR at present.

    Dublin CTA is a separate entity per say.


  • Moderators, Recreation & Hobbies Moderators Posts: 11,906 Mod ✭✭✭✭igCorcaigh




  • Registered Users, Registered Users 2 Posts: 3,163 ✭✭✭EchoIndia


    Must say I'm surprised that procedural (i.e non-radar) control procedures have not been adopted. Dublin used to do this in the old days when the primary radar sat down. Modern aircraft are well able to navigate accurately without ATC radar having to monitor them.


  • Registered Users, Registered Users 2 Posts: 6,562 ✭✭✭kub


    I just got an alert from my RTE App, it is saying all flights to Shannon and Cork are suspended.


  • Closed Accounts Posts: 1,112 ✭✭✭notharrypotter


    EchoIndia wrote: »
    Must say I'm surprised that procedural (i.e non-radar) control procedures have not been adopted. Dublin used to do this in the old days when the primary radar sat down. Modern aircraft are well able to navigate accurately without ATC radar having to monitor them.

    There are no area rated procedural ATCO's in the IAA.

    Radar is used to keep aircraft apart not just monitor flight path adhearance.


  • Advertisement
  • Registered Users, Registered Users 2 Posts: 3,163 ✭✭✭EchoIndia


    I know that, but as you will know there are large areas of the Earth where separation is achieved without radar. I accept that people here are not trained/ rated for such a rare eventuality.


  • Posts: 0 [Deleted User]


    This will be an expensive evening.

    How can the whole radar go down? Is there no redundancy in the system?


  • Closed Accounts Posts: 1,112 ✭✭✭notharrypotter


    EchoIndia wrote: »
    I know that, but as you will know there are large areas of the Earth where separation is achieved without radar. I accept that people here are not trained/ rated for such a rare eventuality.

    They do not have a license to provide a procedural area service. To attempt to do so is illegal.


  • Registered Users, Registered Users 2 Posts: 9,605 ✭✭✭gctest50


    This will be an expensive evening.

    How can the whole radar go down? Is there no redundancy in the system?

    Stuff just fails sometimes

    bpc0Wtf.jpg


  • Registered Users, Registered Users 2 Posts: 10,296 ✭✭✭✭Oscar Bravo


    Shannon starting departures now


  • Closed Accounts Posts: 689 ✭✭✭nim1bdeh38l2cw


    Normal service has resumed


  • Registered Users, Registered Users 2 Posts: 6,562 ✭✭✭kub


    Shannon starting departures now


    FR 3326 from Krakow to Shannon timed it perfectly, 5 mins out from SNN now.


  • Registered Users, Registered Users 2 Posts: 130 ✭✭mikel97


    4th world atc


  • Moderators, Society & Culture Moderators Posts: 4,217 Mod ✭✭✭✭Locker10a


    I was on the EI912 we were at the threshold lined up to go. Cancelled....


  • Registered Users, Registered Users 2 Posts: 814 ✭✭✭Macdarack


    gctest50 wrote: »
    Stuff just fails sometimes

    Rotation bearing on ballycasey radar?


  • Advertisement
  • Registered Users, Registered Users 2 Posts: 36,170 ✭✭✭✭ED E


    Macdarack wrote: »
    Rotation bearing on ballycasey radar?

    That picture is from 2011.


  • Registered Users, Registered Users 2 Posts: 28,404 ✭✭✭✭vicwatson


    kub wrote: »
    FR 3326 from Krakow to Shannon timed it perfectly, 5 mins out from SNN now.

    Otherwise a few hours bus ride


  • Registered Users, Registered Users 2 Posts: 814 ✭✭✭Macdarack


    ED E wrote: »
    That picture is from 2011.

    Obviously, but was it from ballycasey radar?


  • Registered Users, Registered Users 2 Posts: 9,223 ✭✭✭Tow


    This will be an expensive evening.

    How can the whole radar go down? Is there no redundancy in the system?

    A few years ago the Dublin radar went down several times. It was eventually traced to a faulty network card on a PC.

    When is the money (including lost growth) Michael Noonan took in the Pension Levy going to be paid back?



  • Registered Users, Registered Users 2 Posts: 7,559 ✭✭✭plodder


    This will be an expensive evening.

    How can the whole radar go down? Is there no redundancy in the system?
    Good question. The Dublin radar has (at least) two complete units as far as I know. There could still be single points of failure possible but you would expect those components to be replaceable quickly. It was interesting how few flights were affected though. Maybe that is the reason.

    Actually it seems a backup system exists and is still in use. So does that mean the shutdown was caused by a delay in switching over. ?


  • Closed Accounts Posts: 1,112 ✭✭✭notharrypotter


    plodder wrote: »
    Good question. The Dublin radar has (at least) two complete units as far as I know. There could still be single points of failure possible but you would expect those components to be replaceable quickly. It was interesting how few flights were affected though. Maybe that is the reason.

    Actually it seems a backup system exists and is still in use. So does that mean the shutdown was caused by a delay in switching over. ?


    First thing is to safely manage current traffic while dealing with the failure.
    This requires halting departures so you are not adding more aircraft while dealing with all all aircraft currently in the air.

    Then assess what is the root cause (s).

    There is no point switching over if you have no idea what caused the problem in the first place.
    If the main system cannot be relied on then the backup must be brought on-line.

    This will be done in a manged fashion.
    Safety first.


  • Registered Users, Registered Users 2 Posts: 36,170 ✭✭✭✭ED E


    Macdarack wrote: »
    Obviously, but was it from ballycasey radar?

    Nope, South Africa. And its from a radio telescope not a radar.


  • Registered Users, Registered Users 2 Posts: 2,090 ✭✭✭RadioRetro


    It was quite freaky last night, nothing over the majority of the island.


  • Registered Users, Registered Users 2 Posts: 8,644 ✭✭✭cml387


    It's unlikely to be an actual radar head failure. More likely a problem in the software that combines the plots into a single screen.


  • Advertisement
  • Registered Users, Registered Users 2 Posts: 645 ✭✭✭faoiarvok


    cml387 wrote: »
    It's unlikely to be an actual radar head failure. More likely a problem in the software that combines the plots into a single screen.

    Yes, Eurocontrol Network Operations Portal reported it as a Flight Data Processing System failure


  • Closed Accounts Posts: 624 ✭✭✭arccosh


    that's what I was going to mention .... plenty of overlapping coverage between Mt Gabriel, Shannon and that radar in Mayo.. Even Dublin would cover some of the enroute stuff .. to about Kilkenny if not further south.

    Network failure or Radar Data Processing issue I'd say.

    Edit: didn't see faoiarvok's post, was replying to the previous one


  • Registered Users, Registered Users 2 Posts: 7,559 ✭✭✭plodder


    Software doesn't usually just fail randomly like that. It's most likely to fail just after being updated though. If it was truly a random happening, then my money is on some hardware component that has failed.

    Apparently, they still don't know the cause of the problem, reported on radio this morning.

    Also, some interesting details here:

    https://www.irishexaminer.com/breakingnews/ireland/chaos-in-the-air-as-radar-fails-873174.html
    One source in ATC told the Irish Examiner staff at Shannon had been “repeatedly told” that the system “could never fail”.

    “We were told this could never happen,” the source said.
    Naive to make a claim like that, and to believe it. I don't know how new the system is, because new systems would be more vulnerable to software failures as well.

    More info in this RTE report which suggests the system is new and "frequently updated". In that case, it is likely that the cause is an inadequately tested software update. One wonders why is it being updated so regularly?

    https://www.rte.ie/news/2018/1003/1000587-irish-air-space/


  • Registered Users, Registered Users 2 Posts: 645 ✭✭✭faoiarvok


    plodder wrote: »
    Apparently, they still don't know the cause of the problem, reported on radio this morning.

    What station/programme?


  • Registered Users, Registered Users 2 Posts: 7,559 ✭✭✭plodder


    faoiarvok wrote: »
    What station/programme?
    Morning Ireland. (RTE 1)


  • Registered Users, Registered Users 2 Posts: 7,559 ✭✭✭plodder


    Updated statement from the IAA

    https://www.iaa.ie/news/2018/10/04/irish-aviation-authority-update-on-technical-issue-11-30---04-october

    I see the backup system is a replica of the primary system. I hope it's not running identical software though. Or at least, they wait a considerable time before updating the backup after updating the primary..


  • Advertisement
  • Registered Users, Registered Users 2 Posts: 9,223 ✭✭✭Tow


    That are the odds it was a Microsoft Windows Update...

    When is the money (including lost growth) Michael Noonan took in the Pension Levy going to be paid back?



  • Registered Users, Registered Users 2 Posts: 645 ✭✭✭faoiarvok


    Tow wrote: »
    That are the odds it was a Microsoft Windows Update...

    The system doesn't run on Windows.


  • Registered Users, Registered Users 2 Posts: 523 ✭✭✭ShRT


    plodder wrote: »
    Updated statement from the IAA

    https://www.iaa.ie/news/2018/10/04/irish-aviation-authority-update-on-technical-issue-11-30---04-october

    I see the backup system is a replica of the primary system. I hope it's not running identical software though. Or at least, they wait a considerable time before updating the backup after updating the primary..

    They are actually running a newer version of the software.

    They have other redundancy systems if the normal backups fail and that's what they went to initially when live and main backup went down.


  • Registered Users, Registered Users 2 Posts: 7,559 ✭✭✭plodder


    ShRT wrote: »
    They are actually running a newer version of the software.

    They have other redundancy systems if the normal backups fail and that's what they went to initially when live and main backup went down.
    Ouch. If that was the case, it's easier to understand why airspace was shutdown for a number of hours.


  • Closed Accounts Posts: 689 ✭✭✭nim1bdeh38l2cw


    Tow wrote: »
    That are the odds it was a Microsoft Windows Update...

    And what do you base that on?


  • Advertisement
  • Closed Accounts Posts: 624 ✭✭✭arccosh


    patch Tuesday is next week :-D

    the report doesn't specifty which system... ARTAS ? FPDS? both?


  • Registered Users, Registered Users 2 Posts: 7,559 ✭✭✭plodder


    arccosh wrote: »
    patch Tuesday is next week :-D
    shudder..

    whatever platform it's built on (eg Linux, IBM etc), it's very unlikely that operating system updates are just installed directly from the internet :pac:

    I was reading a bit about COOPANS which is the same system used in five different countries including Ireland. They say the same build is used across all centres, but I guess they don't update them all at the same time. Does Shannon get it first maybe?


  • Closed Accounts Posts: 624 ✭✭✭arccosh


    judging by the timing, I'm going to guess it wasn't an update, as they are usually done after midnight ... or if it was, it was a bug introduced from the previous night.


  • Registered Users, Registered Users 2 Posts: 9,223 ✭✭✭Tow


    And what do you base that on?

    I believe CAIRDE 2000 is Linux based, but it is basically a computer network with many different interconnected systems. They don't know the cause (or claim not to), but it would be a typical failure if the likes of a Windows based DHCP Server decided to load a large update, and another critical component decided to renew it's lease during the same period. This is just speculation, it fits the symptoms of the published information.

    When is the money (including lost growth) Michael Noonan took in the Pension Levy going to be paid back?



  • Closed Accounts Posts: 1,112 ✭✭✭notharrypotter


    Tow wrote: »
    CAIRDE 2000
    That's a blast from the past.


  • Registered Users, Registered Users 2 Posts: 645 ✭✭✭faoiarvok


    Tow wrote: »
    I believe CAIRDE 2000 is Linux based, but it is basically a computer network with many different interconnected systems. They don't know the cause (or claim not to), but it would be a typical failure if the likes of a Windows based DHCP Server decided to load a large update, and another critical component decided to renew it's lease during the same period. This is just speculation, it fits the symptoms of the published information.

    COOPANS is the system in use, and the IAA statement above confirms the issue has been identified.


  • Closed Accounts Posts: 689 ✭✭✭nim1bdeh38l2cw


    Tow wrote: »
    but it would be a typical failure if the likes of a Windows based DHCP Server decided to load a large update, and another critical component decided to renew it's lease during the same period.

    You have no idea how DHCP works if you think that'l a plausible explanation.
    Tow wrote: »
    This is just speculation,

    Ah, now I understand... :rolleyes:
    Tow wrote: »
    it fits the symptoms of the published information.

    No it doesn't.


  • Registered Users, Registered Users 2 Posts: 883 ✭✭✭Keplar240B


    Any update on root cause of this incident?


Advertisement