Redbrick Down?

spockety

Anyone know what the craic is with Redbrick being down?

Amz

Haven't a clue what's wrong with it. Been down since yesterday afternoon and no idea when it'll be back up.

I'll consult with my spies.

Ste.phen

I gather their gibson was haxxored. (websites, including the main one, defaced).
In the absence of any evidence that it was someone who was supposed to have root access just doing something they shouldn't have, they've pulled the servers offline to rebuild them*.

* - they may not be completely rebuilding them, but if someone really did root one/all of the boxes, they can't be trusted really.

Will

the plot of gaybrick thickens :pac:

phaxx

RedBrick was compromised recently, and websites were defaced yesterday morning.

Everything has been shut down until we're certain we know the extent of the damage, and that we can secure everything again.

We haven't lost any user data - all your email, databases, websites, etc is safe, and there are multiple (incremental) backups of all that.

The RedBrick admins worked all through yesterday until early in the morning to get email receiving services back up, so no members are losing new email.

They'll be working on it over the next few days to slowly bring all the services back online, but as you can imagine it's a big job. Please be patient, they're working as fast as they can, and all are juggling other commitments, such as their degrees!

As for whodunnit, we're pretty sure we know who the culprit is, but we're waiting until we can be certain.

Will

what action if any will be taken?

phaxx

It's far too early to speculate about that. The focus at the moment is on getting our services back online, while keeping user data safe.

The basic services should, I hope, be coming online one by one over the weekend, but we have no estimate for when everything will be back to normal.

Amz

What can they do other than disuser them?

Will

just realised, i was working on a site for my cousin saving stuff on the server... and not saving em locally on my laptop ;_; argh

dregin

Amz wrote: »

What can they do other than disuser them?

Who says it's a user??

Ste.phen

Amz wrote: »

What can they do other than disuser them?

I'd be fairly optimistic about the fact that whoever dunnit did something illegal

dregin

It would have been nice if the email sent out to users had contained something similar to the info in Charlie's post instead of "unscheduled maintenance blah blah blaaahhh...."

phaxx

What email was that? I didn't get one...

As far as I'm aware, we haven't had time (nor, currently, the infrastructure) to email all our members about this.

The priority is getting services back up safely. Everything else, including notification, is secondary.

Ste.phen

phaxx wrote: »

What email was that? I didn't get one...

I think the point was that one wasn't sent out.
(Or else it was and it's in your redbrick email)

dregin

mail to redbricksocATlist.dcu.ie wrote:

There will be down time for unplanned maintenance of the network. We'll keep
you up to date with a dramatic blow by blow dialog to let you all know whats
happening! We're working on it and hopefully will not take too long! In the
meanwhile stretch out your legs get some coffee, go for a walk, smell a flower
or something,get high on life! And we'll be in touch!

That.

phaxx

Aah, that was sent to the DCU society mailing list, not the main RedBrick list. That would be why I didn't get it.

There'll be a proper mail from the admins when we've got most of the services back up and the whole story is clear.

public_enemy

In what way was the website defaced? Anyone take a screenshot or anything before it went down?

dregin

public_enemy

spockety

Amz wrote: »

What can they do other than disuser them?

If it was a student at the very very least they could/should be hauled on
front of the University disciplinary committee.. they would probably be
expelled from DCU if they were found to have done it.

Also should be passed onto the Gardai to deal with.

Will

dregin wrote: »

haha that is hilarious. Although I'm annoyed having to wait to get my crap off the server its still funny.

rugbug86

spockety wrote: »

If it was a student at the very very least they could/should be hauled on
front of the University disciplinary committee.. they would probably be
expelled from DCU if they were found to have done it.

Also should be passed onto the Gardai to deal with.

The life.dcu.ie website was hacked into last year and afaik nothing was done to the student(s) involved, even though their identity was known.

dregin

rugbug86 wrote: »

The life.dcu.ie website was hacked into last year and afaik nothing was done to the student(s) involved, even though their identity was known.

You clearly need to go do a little research on the meaning of the word "hack". People gaining access to a forum due to stupid retards using moronic passwords for life.dcu.ie is not "hacking". I wouldn't call saving and then hosting that image any sort of evidence that that user had anything to do with it.

Will

Ah my apologies there.

lil_cain

rugbug86 wrote: »

The life.dcu.ie website was hacked into last year and afaik nothing was done to the student(s) involved, even though their identity was known.

That could be because no actual damage was done to life, and the first thing that the students who got in did was to tell the administrators of the system. Plus, as dregin said, guessing someone's *incredibly* moronic password is not hacking

Mad_Max

Definately the last straw, i'm going to try and get my club's website off redbrick.

I know this one isn't rb's fault but our site is down nearly on a weekly basis. We had no way of contacting our members this weekend as we manage it through the site.

Matthewthebig

I doubt your site beign down is redbricks fault. It's rarely if ever down.

andrew163

Actually, compared to the experience I've had with a number of commercial hosts, redbrick's uptime is better than most. Also, before becoming a rootholder myself, it was nice to be able to actually get the attention of somebody when something needed to be done/changed/fixed, instead of just $random_support_drone. There was downtime a weekend or two ago, but that was announced well in advance via internal boards, login message, IRC, and on the front page of the website. I think it was mentioned in a couple of emails too.

Anyway. We just spent the last 11 hours in the server room. It's not looking likely that login services will be back by the end of the weekend. It's not an ideal situation but we have a choice between either downtime, or potentially bringing insecure systems back on line, and letting somebody untrustworthy have access to your accounts.

(This is werdz on redbrick)

nemo!

Afaik there's never been one single conviction for hacking in Ireland. Cheers for working through the weekend btw.

Cid-Highwind

Mad_Max wrote: »

Definately the last straw, i'm going to try and get my club's website off redbrick.

I know this one isn't rb's fault but our site is down nearly on a weekly basis. We had no way of contacting our members this weekend as we manage it through the site.

The last unplanned downtime I can remember is last year when stuff went on fire in CSD. Commercial hosts require money and skills, something most clubs and societies don't have. RedBrick provides an excellent service at no charge, something we don't have to do.

Dregin: Apologies for that email that went out. Dano was asked to send out a mail, but he didn't really have any information to work with. (the email only went to our csd lit, cause all redbrick stuff was offline.

I'll be sending out a mail tomorrow evening when we should have a better idea of when stuff will be working. I've also got a placeholder page catching all website access, and I'll be keeping that up to date.

If anyone has files they need urgently, or desperately need their redbrick mail forwarded to an alternate address then please mail me (receive AT redbrick.dcu.ie) and I'll do my best to sort something out.

Mad_Max

Matthewthebig wrote: »

I doubt your site beign down is redbricks fault. It's rarely if ever down.

So it goes down why? It was an over-exageration saying its down on a weekly basis, but whenever its down its usually at an important time for us and its happened too often this year.

Not saying anything bad about rb, they do a great job for a society but us personally just need a bit more reliability.

gamblitis

Mad_Max wrote: »

So it goes down why? It was an over-exageration saying its down on a weekly basis, but whenever its down its usually at an important time for us and its happened too often this year.

Not saying anything bad about rb, they do a great job for a society but us personally just need a bit more reliability.

Finished complaining yet??Anything else ya wanna get off your chest??Give it a rest already.

RB's reliability cannot be slated due to someone hacking into the system. I remember plenty of warnings before previous downtimes so you had plenty of time to contact your members. If they didn't find out about something then its your fault not RB's!

And kudos to the admin team and everyone else involved in trying to get it back online.I'm sure after a weeks work they had better plans for the weekend than sitting in a cold server room reinstalling everything.FOR NOTHING!!

So how about a little more support and a little less whining! If you think you can make the system or any system more reliable for that matter go right ahead and we'll see how you find it.

Sarunas

Mad_Max wrote: »

So it goes down why? It was an over-exageration saying its down on a weekly basis, but whenever its down its usually at an important time for us and its happened too often this year.

Not saying anything bad about rb, they do a great job for a society but us personally just need a bit more reliability.

During this year there was new server(s) set up and few major changes which caused some problems. All/most scheduled outages where given at least few days notice.

As for reliability, first thing you need to remember is Redbrick's services are free for clubs and societies; the servers are maintained by various people who give as much time as possible, sacrificing college work, work and sleep in order for everything to work as smoothly as possible, All for free. Its not run by a team of people which get big fat cheques every week for the job they do.

Current and past admins staid in the server room on Friday till early hours in the morning, same for yesterday. I have no doubt the admins will be working on restoring all the services till wee hours again tonight, basically till they're completely exhausted and cannot work for any longer. Is that not good enough for you?

**** ups happen with commercial hosting too. Do us a favour and just stop bitching about, if you're not happy with the reliability there is nothing forcing your society to use Redbrick.

You can just use DCUs mailing lists to contact your members too.

andrew163

I think this is getting a little bitchy. Mad_Max just said he wanted more reliability, and while looking for that in a commercial host may seem futile (most of the ones I've had dealings with are worse than redbrick), that's his problem.

On another note, as Sarunas has said, it's not just current admins working on getting this fixed. Past admins, both in DCU and a few who have graduated, are working hard trying to get services back too... so send them chocolates on sysadmin appreciation day too

To confirm what gav mentioned, we are completely rebuilding all of our machines from scratch. The only thing that we're not deleting and starting again with is user data. This is why it's taking so long..

Joe5ie

how did they get in then?

how did ye notice?

dregin

I'd say y'on image on everyone's homepage was a bit of a giveaway.

*newb*

Joe5ie

dregin wrote: »

I'd say y'on image on everyone's homepage was a bit of a giveaway.

*newb*

which image?

gamblitis

dregin wrote: »

This one

andrew163

Joe5ie wrote: »

how did they get in then?

We're not releasing this information until we're absolutely certain of it

how did ye notice?

See epic photoshop skillz shown above :pac:

tnkrbell

Well college work is being done cause of no Rebrick!

Fair play to all the admins that are working to get RB back.Probably not what any of the admins that are in college wanted especially when exams start in 2 weeks time!

AlanSparrowhawk

I thought Redbrick couldn't be hacked??????

Tonnes and tonnes of admin and ex-admin working around the clock to fix something they could have taken much more efficient ways of fixing. Poor guys

dregin

Whoever rooted redbrick could have very easily, once they had root, installed their own little backdoor that could have taken weeks to find. Re-installing everything on multiple servers to guard against the possibility of this will inevitably take a while and having a few admins working on it will help get the job done quicker. You also have to take into account that some of the servers were installed and configured before the current admins were on the scene, so it makes sense to have the admins who did the job in the first place there to oversee proceedings.

























No?

AlanSparrowhawk

dregin wrote: »

Whoever rooted redbrick could have very easily, once they had root, installed their own little backdoor that could have taken weeks to find. Re-installing everything on multiple servers to guard against the possibility of this will inevitably take a while and having a few admins working on it will help get the job done quicker. You also have to take into account that some of the servers were installed and configured before the current admins were on the scene, so it makes sense to have the admins who did the job in the first place there to oversee proceedings. No?

Absolutely. But if you look at it from the point of view that a friendly face performed this tomfoolery, I think friendly conversation from peer-to-peer would have quickly confirmed Code 11 measures weren't necessary.

dregin

I think the fact that they'd do something like this in the first place nullifies that argument already. I'm sure whoever did it realises full well that CSD do not look on security breaches in their network with any sort of rosey affection and an incident like this could put Redbrick's relationship with CSD on the ropes. It is true to say that NO system running any service is completely secure, so you can't guard against all possibilities. Only do your best. I think redbrick admins over the years have done an outstanding job and you only have to look at their current career paths to see that redbrick isn't run by mindless idiots.

To come along and first of all claim that you thought Redbrick was secure and then call this a poor show from them is nothing more than the kneejerk reaction of a spoilt child.

nemo!

AlanSparrowhawk wrote: »

Absolutely. But if you look at it from the point of view that a friendly face performed this tomfoolery, I think friendly conversation from peer-to-peer would have quickly confirmed Code 11 measures weren't necessary.

That seems a bit naive.

What dregin said.

AlanSparrowhawk

Well I think it was more naive for a boastful admin to make audacious claims. I also think it was a far more severe knee-jerk reaction to go all out on a rebuild. One could also argue it's slightly more spoilt-childish not to listen to helpful advice in times of need.

Mad_Max

gamblitis wrote: »

Finished complaining yet??Anything else ya wanna get off your chest??Give it a rest already.

RB's reliability cannot be slated due to someone hacking into the system. I remember plenty of warnings before previous downtimes so you had plenty of time to contact your members. If they didn't find out about something then its your fault not RB's!

And kudos to the admin team and everyone else involved in trying to get it back online.I'm sure after a weeks work they had better plans for the weekend than sitting in a cold server room reinstalling everything.FOR NOTHING!!

So how about a little more support and a little less whining! If you think you can make the system or any system more reliable for that matter go right ahead and we'll see how you find it.

Jaysus, relax. I never said anything bad about the guys who run rb. I simply said i'd like our site to be moved. I'm well aware of their effort for the pittance they get.

I didn't think any of my posts where rants, well they werent meant to be anyway. My initial post was out of frustration because of the timing. Also a very significant proportion of our members are either non-dcu students, st.pats, or alumni who dont use their dcu addresses anymore which causes our problem.

Again wasn't a swipe at rb guys so chillax

spockety

AlanSparrowhawk wrote: »

Absolutely. But if you look at it from the point of view that a friendly face performed this tomfoolery, I think friendly conversation from peer-to-peer would have quickly confirmed Code 11 measures weren't necessary.

Rubbish, it sounds like they have taken the right course of action.

If the knob jockey who performed the deed had gone to them BEFORE showing his skills and said "lads this is what I've found" then maybe you can work from a position of trust. He/They didn't, and I hope to god that the trouble they have caused comes with a wedgeload of retribution in the form of disciplinary proceedings.

What you reap is what you sow.

AlanSparrowhawk

spockety wrote: »

wedgeload of retribution in the form of disciplinary proceedings.

ha. Like sent to the principle's office? I think redbrick have made a mountain out of a mole hill and all this "let's stay around all day and night rebuilding" although sounds noble enough sounds is just masochism.

spockety

AlanSparrowhawk wrote: »

ha. Like sent to the principle's office? I think redbrick have made a mountain out of a mole hill and all this "let's stay around all day and night rebuilding" although sounds noble enough sounds is just masochism.

Can I ask if you have a technical background at all?

If the facts of the matter are that a student compromised Redbrick to gain root access to one or more servers, and then used that access to deface web sites or interfere in any way with user data, then yes they could find themselves in a lot of trouble. You might laugh, as obviously there is no 'principle' in DCU, but there is a disciplinary board, and if the facts are as above then the person involved could very easily find themselves kicked out of the university for it.

Thousands of people rely on redbrick directly or indirectly at this stage. Redbrick provides invaluable services that are used 24/7 on a 365 day a year basis, and for someone to interfere intentionally with that, it is a big deal no matter what kind of mole hill you think it represents.

From the sounds of it, the admins are dealing with the technical side of the problem appropriately, now I would hope that they can prove who it was that did this, so they can deal with the non-technical aspect of the problem correctly.



"# It is not acceptable to engage in any activity which:

1. Disrupts the intended use of the resources.
2. Wastes resources(people, capacity, computer, network, data etc.).
3. Compromises the legal rights of others.
4. Modifies, damages or destroys computing resources or the data on them.

# Disciplinary action and/or criminal proceedings may result from any breach of this Code of Conduct. "


"All Members of the University are expected to ensure that those on Campus who infringe the Code of Discipline are identified and dealt with according to the regulations of the University."

belleray

Got to say I've never found redbrick anything but amazingly helpful in hosting the society website (<3 receive)

Well done to all who are trying to sort it out

Now to wait patiently till its sorted.