Boards.ie uses cookies. By continuing to browse this site you are agreeing to our use of cookies. Click here to find out more x
Post Reply  
 
Thread Tools Search this Thread
07-12-2010, 23:48   #1
woolymammoth
Registered User
 
woolymammoth's Avatar
 
Join Date: Feb 2003
Location: Adelaide
Posts: 1,650
file migration with robocopy

anyone have good experience with it? Have a large number of files, not a very large amount of data, that i'm trying to migrate from a directly attached storage array to a NAS. Was hoping it would finish overnight but i'm getting appalling transfer rates of about 1.2MB/s.. Server 2003 btw. Have tried mapping the drive to a 2008 box in an effort to use the multithreading function but it doesn't make any difference. So naturally i assumed it's the network or the disk read/write of the SA or NAS, but every time it encounters a larger file the transfer speed picks up considerably, so it's not likely the network, and only possibly the disks. At this rate i'm looking at a week to move less than half a TB! the NAS is unix based and has it's own tool which is even slower am i doomed by the sheer number of smaller files?

Any thoughts or recommendations appreciated. winner gets a free pint
woolymammoth is offline  
Advertisement
08-12-2010, 10:11   #2
bhickey
Registered User
 
Join Date: May 2005
Location: Athenry
Posts: 1,339
Send a message via MSN to bhickey
Quote:
Originally Posted by woolymammoth View Post
... the NAS is unix based ....
Maybe if you tar gzip it first and then copy it across as a big lump and extract it again?
bhickey is offline  
Thanks from:
08-12-2010, 10:26   #3
BaconZombie
/dev/random
 
BaconZombie's Avatar
 
Join Date: Jan 2007
Posts: 9,919
I know we have to use RichCopy when migrating to Server 2008 x64 due to similar issues with speed and ACL's not been copied correctly.

I can see if I can dig up the scripts we used if you want.
BaconZombie is offline  
Thanks from:
08-12-2010, 11:37   #4
woolymammoth
Registered User
 
woolymammoth's Avatar
 
Join Date: Feb 2003
Location: Adelaide
Posts: 1,650
Quote:
Originally Posted by bhickey View Post
Maybe if you tar gzip it first and then copy it across as a big lump and extract it again?


not at your suggestion it would be a great way out for me, but need to preserve everything, and, well, there's about 2.7 million files would probably take as long to zip & unzip!

Quote:
Originally Posted by BaconZombie View Post
I know we have to use RichCopy when migrating to Server 2008 x64 due to similar issues with speed and ACL's not been copied correctly.

I can see if I can dig up the scripts we used if you want.
that sir, that would be fantastic.
woolymammoth is offline  
08-12-2010, 11:47   #5
bhickey
Registered User
 
Join Date: May 2005
Location: Athenry
Posts: 1,339
Send a message via MSN to bhickey
Quote:
Originally Posted by woolymammoth View Post
.....there's about 2.7 million files would probably take as long to zip & unzip!
Maybe. Try it on a chunk and see, I still think it would be much quicker than copying files individually. When you say "preserve everything", are you talking about permissions etc.? Depending on how complex the permissions are, maybe you could even reapply them after the transfer?
bhickey is offline  
Advertisement
08-12-2010, 11:52   #6
woolymammoth
Registered User
 
woolymammoth's Avatar
 
Join Date: Feb 2003
Location: Adelaide
Posts: 1,650
Quote:
Originally Posted by bhickey View Post
When you say "preserve everything", are you talking about permissions etc.? Depending on how complex the permissions are, maybe you could even reapply them after the transfer?
ya, the security, acl's, the lot. unfortunately for me the directory structure is deep and complet in places, with different permissions everywhere. Luckily there's only a small number of shares which i can recreate, but it's really the number of files and permissions that's the reall killer here.
woolymammoth is offline  
08-12-2010, 11:53   #7
brianwalshcork
Registered User
 
Join Date: Oct 2007
Posts: 403
It's the number and size of small files that is killing you but the problem could be on the NAS end rather than the DAS end.... i.e. it the NAS could be far slower at seeks and writes then than the DAS is at seeks and reads - asumming that the das is something decent like a SCSI based disk array not talking a USB drive.

Have you tried copying the files from the DAS to sonething else besides the NAS?

If the NAS is bad at writing small files, then it's probably not going to be much better at reading them, so I wouldn't be too confident about putting 2.7 million files on it. Even if you're only using it for backup, at some point you will want them off of it, and you probably won't want to wait a week to get them back.
brianwalshcork is offline  
08-12-2010, 12:14   #8
woolymammoth
Registered User
 
woolymammoth's Avatar
 
Join Date: Feb 2003
Location: Adelaide
Posts: 1,650
Quote:
Originally Posted by brianwalshcork View Post
asumming that the das is something decent like a SCSI based disk array not talking a USB drive.
the das is scsi alright

Quote:
Originally Posted by brianwalshcork View Post
Have you tried copying the files from the DAS to sonething else besides the NAS?
umm.. no, but.. I ran a number of tests on different directories beforehand, from das to nas, was getting transfer rates up to 30-40MB/s.

Quote:
Originally Posted by brianwalshcork View Post
If the NAS is bad at writing small files, then it's probably not going to be much better at reading them, so I wouldn't be too confident about putting 2.7 million files on it.
the NAS isn't too bad actually, but ultimately, the company gets what the company wants. Or what they're prepared to pay for at least
woolymammoth is offline  
09-12-2010, 10:10   #9
woolymammoth
Registered User
 
woolymammoth's Avatar
 
Join Date: Feb 2003
Location: Adelaide
Posts: 1,650
so it would seem that it was just more to do with the sheer number of small files. We jumped from an estimated 140hrs to completed in under 24. Picked up after getting to the larger files
woolymammoth is offline  
Post Reply

Quick Reply
Message:
Remove Text Formatting
Bold
Italic
Underline

Insert Image
Wrap [QUOTE] tags around selected text
 
Decrease Size
Increase Size
Please sign up or log in to join the discussion

Thread Tools Search this Thread
Search this Thread:

Advanced Search



Share Tweet