Advertisement
If you have a new account but are having problems posting or verifying your account, please email us on hello@boards.ie for help. Thanks :)
Hello all! Please ensure that you are posting a new thread or question in the appropriate forum. The Feedback forum is overwhelmed with questions that are having to be moved elsewhere. If you need help to verify your account contact hello@boards.ie
Hi there,
There is an issue with role permissions that is being worked on at the moment.
If you are having trouble with access or permissions on regional forums please post here to get access: https://www.boards.ie/discussion/2058365403/you-do-not-have-permission-for-that#latest

file migration with robocopy

  • 07-12-2010 11:48pm
    #1
    woolymammoth
    Registered Users, Registered Users 2 Posts: 1,772 ✭✭✭


    anyone have good experience with it? Have a large number of files, not a very large amount of data, that i'm trying to migrate from a directly attached storage array to a NAS. Was hoping it would finish overnight but i'm getting appalling transfer rates of about 1.2MB/s.. Server 2003 btw. Have tried mapping the drive to a 2008 box in an effort to use the multithreading function but it doesn't make any difference. So naturally i assumed it's the network or the disk read/write of the SA or NAS, but every time it encounters a larger file the transfer speed picks up considerably, so it's not likely the network, and only possibly the disks. At this rate i'm looking at a week to move less than half a TB! :( the NAS is unix based and has it's own tool which is even slower :eek: am i doomed by the sheer number of smaller files?

    Any thoughts or recommendations appreciated. winner gets a free pint :pac:


Comments

  • #2
    bhickey
    Registered Users, Registered Users 2 Posts: 1,340 ✭✭✭bhickey


    woolymammoth wrote: »
    ... the NAS is unix based ....

    Maybe if you tar gzip it first and then copy it across as a big lump and extract it again?


  • #3
    BaconZombie
    Registered Users, Registered Users 2 Posts: 8,813 ✭✭✭BaconZombie


    I know we have to use RichCopy when migrating to Server 2008 x64 due to similar issues with speed and ACL's not been copied correctly.

    I can see if I can dig up the scripts we used if you want.


  • #4
    woolymammoth
    Registered Users, Registered Users 2 Posts: 1,772 ✭✭✭woolymammoth


    bhickey wrote: »
    Maybe if you tar gzip it first and then copy it across as a big lump and extract it again?
    :eek:

    not at your suggestion :p it would be a great way out for me, but need to preserve everything, and, well, there's about 2.7 million files :( would probably take as long to zip & unzip!
    BaconZombie wrote: »
    I know we have to use RichCopy when migrating to Server 2008 x64 due to similar issues with speed and ACL's not been copied correctly.

    I can see if I can dig up the scripts we used if you want.

    that sir, that would be fantastic.


  • #5
    bhickey
    Registered Users, Registered Users 2 Posts: 1,340 ✭✭✭bhickey


    woolymammoth wrote: »
    .....there's about 2.7 million files :( would probably take as long to zip & unzip!

    Maybe. Try it on a chunk and see, I still think it would be much quicker than copying files individually. When you say "preserve everything", are you talking about permissions etc.? Depending on how complex the permissions are, maybe you could even reapply them after the transfer?


  • #6
    woolymammoth
    Registered Users, Registered Users 2 Posts: 1,772 ✭✭✭woolymammoth


    bhickey wrote: »
    When you say "preserve everything", are you talking about permissions etc.? Depending on how complex the permissions are, maybe you could even reapply them after the transfer?
    ya, the security, acl's, the lot. unfortunately for me the directory structure is deep and complet in places, with different permissions everywhere. Luckily there's only a small number of shares which i can recreate, but it's really the number of files and permissions that's the reall killer here.


  • Advertisement
  • #7
    brianwalshcork
    Registered Users, Registered Users 2 Posts: 611 ✭✭✭brianwalshcork


    It's the number and size of small files that is killing you but the problem could be on the NAS end rather than the DAS end.... i.e. it the NAS could be far slower at seeks and writes then than the DAS is at seeks and reads - asumming that the das is something decent like a SCSI based disk array not talking a USB drive.

    Have you tried copying the files from the DAS to sonething else besides the NAS?

    If the NAS is bad at writing small files, then it's probably not going to be much better at reading them, so I wouldn't be too confident about putting 2.7 million files on it. Even if you're only using it for backup, at some point you will want them off of it, and you probably won't want to wait a week to get them back.


  • #8
    woolymammoth
    Registered Users, Registered Users 2 Posts: 1,772 ✭✭✭woolymammoth


    brianwalshcork wrote: »
    asumming that the das is something decent like a SCSI based disk array not talking a USB drive.
    :) the das is scsi alright
    brianwalshcork wrote: »
    Have you tried copying the files from the DAS to sonething else besides the NAS?
    umm.. no, but.. I ran a number of tests on different directories beforehand, from das to nas, was getting transfer rates up to 30-40MB/s.
    brianwalshcork wrote: »
    If the NAS is bad at writing small files, then it's probably not going to be much better at reading them, so I wouldn't be too confident about putting 2.7 million files on it.
    the NAS isn't too bad actually, but ultimately, the company gets what the company wants. Or what they're prepared to pay for at least :p


  • #9
    woolymammoth
    Registered Users, Registered Users 2 Posts: 1,772 ✭✭✭woolymammoth


    so it would seem that it was just more to do with the sheer number of small files. We jumped from an estimated 140hrs :eek: to completed in under 24. Picked up after getting to the larger files :)


Advertisement