Data Storage/Backup Solution

Mickah

I'm looking to mirror our my data onto an physically separate disk that can be removed easily and taken off-site each day.

Initially I thought a NAS was the solution, with 2 drives (500 GB each) running RAID 1 (mirror) and simply hot-swap the backup out. However, I've been told that RAID will rebuild the backup drive from scratch when I plug it back in.

I need:
1) live data to be mirrored to a separate drive.
2) the backup drive can be removed without upsetting the live drive.
3) the backup drive will be updated with any changes when I plug it back in.
4) a solution that's scalable (4-6 drives).

Any ideas?

moridin

How much data are you adding to the drive (which doesn't get swapped in and out) every day?

Are you looking for a hardware solution or a software one?

Mickah

Anywhere between 50 to 200 GB per day onto the live drive.

We're open to all options Hardware or software.

zod

what kind of server is the data on?

consider syncing change only to an online backup location with an SDSL connection or using a cloud service ( http://mozy.com/pro/pricing )

raid is quick but expensive for version control and raid migrations are usually destructive if you want scalability.

Mickah

Online backup is not an option. Our internet connection wouldn't handle the data amounts.

The biggest stumbling block at the moment is creating a mirror of our data onto a drive that can physically be taken off-site daily. When that drive is plugged back in it doesn't mirror from scratch but simply updates with any changes made to live data.

As I said I thought a €500 5-bay NAS was the solution with RAID 1, but it re-builds the backup drive from scratch on re-plug.

Data is on a HP proliant SCSI 300GB at the moment with tape backup and drive image backups. I.E. a pain in the arse.

Naz_st

Have you looked into DFS-R (a pretty recent addition to the MS windows server environment)? See here. We use this for drive syncronisation across multiple servers. Perhaps in conjunction with a hot-swappable HDD if bandwidth is a real issue?

Alternatively there is a management suite for Windows servers called the Data Protection Manager that might do the job also. See here.

unnameduser

does the likes of novasoft or syncbackpro backup software complete incremental backups. i.e compare files and simply update if there are differences.

moridin

I was gonna suggest creating some kind of incremental backup system for yourself, in that you take a fingerprint of all the files on your source drive, and copy them to the second disk. Then, the next time you want to run some updates you take another fingerprint of the files and compare the md5s - if they match then you don't copy anything, but if they're different you copy the files to the second drive.

However, with the data sizes you're quoting - 50 to 200GB - it sounds like you'd be better off looking into some kind of block level replication rather than file level? Are there a lot of files involved, or a few files which are a large size? I know that Storage Foundation can mirror one partition to another, and you can decouple that mirroring once the sync is complete, but I'm not sure if you can re-add the same partition as a mirror again and have it just sync the changed data.

Mickah

Thanks for all the help lads.

Most days it'll be small updates, 5GB-20GB at most. However there are odd days where we recieve huge amounts of data.

The data is mostly thousands of small files < 1MB. Largest files might be at most 100MB, but these a few and far between.

I'm trying to find NAS solutions with incremental sync software on them at the moment, still open to suggestions.

Naz_st

Maybe something like this then would do the job:
QNAP-409, and here.

Or from Netgear (pricier):
ReadyNAS NV+
CNET review

PaddyTheNth

Not as neat a solution as a hot-swappable NAS but but in a very limited budget situation you could probably hack a solution using something like Acronis TI.

I know that it can handle incremental backups and I'm pretty sure you can specify directories instead of doing the whole disk.

Dunno if it would fit the needs of your environment.

You could certainly schedule the backups and you might be able to set up a trigger too.

If you have the funds then Naz_st's suggestions look like the way to go.