[woolymammoth]

anyone have good experience with it? Have a large number of files, not a very large amount of data, that i'm trying to migrate from a directly attached storage array to a NAS. Was hoping it would finish overnight but i'm getting appalling transfer rates of about 1.2MB/s.. Server 2003 btw. Have tried mapping the drive to a 2008 box in an effort to use the multithreading function but it doesn't make any difference. So naturally i assumed it's the network or the disk read/write of the SA or NAS, but every time it encounters a larger file the transfer speed picks up considerably, so it's not likely the network, and only possibly the disks. At this rate i'm looking at a week to move less than half a TB! the NAS is unix based and has it's own tool which is even slower am i doomed by the sheer number of smaller files?

Any thoughts or recommendations appreciated. winner gets a free pint

[bhickey]

Originally Posted by woolymammoth

... the NAS is unix based ....

Maybe if you tar gzip it first and then copy it across as a big lump and extract it again?

[BaconZombie]

I know we have to use RichCopy when migrating to Server 2008 x64 due to similar issues with speed and ACL's not been copied correctly.

I can see if I can dig up the scripts we used if you want.

[woolymammoth]

Originally Posted by bhickey

Maybe if you tar gzip it first and then copy it across as a big lump and extract it again?

not at your suggestion it would be a great way out for me, but need to preserve everything, and, well, there's about 2.7 million files would probably take as long to zip & unzip!

Originally Posted by BaconZombie

I know we have to use RichCopy when migrating to Server 2008 x64 due to similar issues with speed and ACL's not been copied correctly.

I can see if I can dig up the scripts we used if you want.

that sir, that would be fantastic.

[bhickey]

Originally Posted by woolymammoth

.....there's about 2.7 million files would probably take as long to zip & unzip!

Maybe. Try it on a chunk and see, I still think it would be much quicker than copying files individually. When you say "preserve everything", are you talking about permissions etc.? Depending on how complex the permissions are, maybe you could even reapply them after the transfer?

[woolymammoth]

Originally Posted by bhickey

When you say "preserve everything", are you talking about permissions etc.? Depending on how complex the permissions are, maybe you could even reapply them after the transfer?

ya, the security, acl's, the lot. unfortunately for me the directory structure is deep and complet in places, with different permissions everywhere. Luckily there's only a small number of shares which i can recreate, but it's really the number of files and permissions that's the reall killer here.

[brianwalshcork]

It's the number and size of small files that is killing you but the problem could be on the NAS end rather than the DAS end.... i.e. it the NAS could be far slower at seeks and writes then than the DAS is at seeks and reads - asumming that the das is something decent like a SCSI based disk array not talking a USB drive.

Have you tried copying the files from the DAS to sonething else besides the NAS?

If the NAS is bad at writing small files, then it's probably not going to be much better at reading them, so I wouldn't be too confident about putting 2.7 million files on it. Even if you're only using it for backup, at some point you will want them off of it, and you probably won't want to wait a week to get them back.

[woolymammoth]

Originally Posted by brianwalshcork

asumming that the das is something decent like a SCSI based disk array not talking a USB drive.

the das is scsi alright

Originally Posted by brianwalshcork

Have you tried copying the files from the DAS to sonething else besides the NAS?

umm.. no, but.. I ran a number of tests on different directories beforehand, from das to nas, was getting transfer rates up to 30-40MB/s.

Originally Posted by brianwalshcork

If the NAS is bad at writing small files, then it's probably not going to be much better at reading them, so I wouldn't be too confident about putting 2.7 million files on it.

the NAS isn't too bad actually, but ultimately, the company gets what the company wants. Or what they're prepared to pay for at least

[woolymammoth]

so it would seem that it was just more to do with the sheer number of small files. We jumped from an estimated 140hrs to completed in under 24. Picked up after getting to the larger files