And you thought your Monday was bad.....

Graham

Being an optimist you might say that accidentally wiping 1000+ customer servers on a Monday morning means your week can only get better from there on. I'm not convinced.

From a post on ServerFault:

I run a small hosting provider with more or less 1535 customers and I use Ansible to automate some operations to be run on all servers. Last night I accidentally ran, on all servers, a Bash script with a rm -rf {foo}/{bar} with those variables undefined due to a bug in the code above this line.

All servers got deleted and the offsite backups too because the remote storage was mounted just before............

http://serverfault.com/questions/769357/recovering-from-a-rm-rf

Sparks

Yeah, it was a hoax for marketing purposes.

http://www.repubblica.it/tecnologia/2016/04/15/news/cancella_l_azienda_per_sbaglio_la_disavventura_tecnologica_di_marco_marsala-137693154/

It is a guerrilla marketing operation, a 'faux pas' invented out of whole cloth just '' to advertise my startup where we offer outsourced server management services '', this is the author's version that reveals: '' I am also writing a book on Unix for Dummies Horror Stories and anyway that fact really happened to someone I know, but years ago, there was an article in the newspaper ''. If it really happened, happened '' before 2006 ' ', explains Marsala.

Graham

Sparks wrote: »

Yeah, it was a hoax for marketing purposes.

http://www.repubblica.it/tecnologia/2016/04/15/news/cancella_l_azienda_per_sbaglio_la_disavventura_tecnologica_di_marco_marsala-137693154/

I hope for their sake Karma isn't real

Along similar lines, it looks like 123-reg did it for real last weekend accidentally mass-deleting a pile of VPSs.

This script is run to show us the number of machines active against the master database. An error on the script showed 'zero-records' response from the database for some live VPS. For those customers, this created a 'failure' scenario – showing no VMs and effectively deleting what was on the host. As a result of our team's investigations, we can conclude that the issues faced having resulted in some data loss for some customers. Our teams have been and continue to work to restore.

http://www.theregister.co.uk/2016/04/18/123_reg_mass_deletion_woes_continue/

opus

One of the people at work came up to me & said they'd deleted all their files 'by accident', this was on a PC running Linux. No problem I thought as we diligently backup everything at night & also take a snapshot at lunchtime every day.

However this person decided a good place to keep their 'important files' (their words) was on the local drive of the PC rather than in their home directory :rolleyes: In the end I got everything back using this bit of s/w in conjunction with an exteral drive caddy but tbh I kind of hoped it wouldn't work!

Graham

opus wrote: »

However this person decided a good place to keep their 'important files' (their words) was on the local drive of the PC rather than in their home directory :rolleyes:

I've seen dumber. What do you mean you kept your work in progress in Trash!

opus

Graham wrote: »

I've seen dumber. What do you mean you kept your work in progress in Trash!

I'd believe it! One of my colleagues told me they came across someone who had set up a whole folder structure in the 'Deleted Items' folder in Outlook :eek:

jamesd

I once deleted 10000 mails from a users deleted items and he freaked out, I asked him does he keep things in his rubbish bin under his desk and he said it was completely different.

vibe666

just had a user delete a couple of hundred gb off a server share. no problem, I'll just do a restore, we have a large local tape library as well as offsites, so we can restore anything from the previous 30 days without needing a recall request.

i check the backups, no backuyp for the last 6 weeks (on a file server).

that can't be right, we get alerts for failed backup jobs. check the logs and the backup hadn't actually failed, it had been continuing a full backup job, but had backed up 0 bytes in the 6 weeks since it crapped itself, so there's not even any VSS to fall back on as it was in the middle of the backup job.

jamesd

vibe666 wrote: »

just had a user delete a couple of hundred gb off a server share. no problem, I'll just do a restore, we have a large local tape library as well as offsites, so we can restore anything from the previous 30 days without needing a recall request.

i check the backups, no backuyp for the last 6 weeks (on a file server).

that can't be right, we get alerts for failed backup jobs. check the logs and the backup hadn't actually failed, it had been continuing a full backup job, but had backed up 0 bytes in the 6 weeks since it crapped itself, so there's not even any VSS to fall back on as it was in the middle of the backup job.

Crap not a nice situation, note for future get backup success emails sent daily to 2 people.
I have 4 people that get our success/fail emails and I have them all told its the end of the world if they get a fail and dont ring me.

vibe666

yeah, i'm only here a month just on a short term contract to provide extra cover while there's some project work going on, i did the restore yesterday when we finally got the tapes back and micraculously nothing seems to be missing, so it was likely old static data that they disappeared, which was very lucky.

jamesd

vibe666 wrote: »

yeah, i'm only here a month just on a short term contract to provide extra cover while there's some project work going on, i did the restore yesterday when we finally got the tapes back and micraculously nothing seems to be missing, so it was likely old static data that they disappeared, which was very lucky.

Now make sure to get the backup emails going to a few people so that cant happen again.