Hi Jonathan, i would be very interested in seeing your scripts when they are posted. Let me know where to get them!
Thanks a bunch! Andi Christiansen > On 11/18/2020 12:48 PM Jonathan Buzzard <jonathan.buzz...@strath.ac.uk> wrote: > > > On 17/11/2020 23:17, Chris Schlipalius wrote: > > So at my last job we used to rsync data between isilons across campus, > > and isilon to Windows File Cluster (and back). > > > > I recommend using dry run to generate a list of files and then use this > > to run with rysnc. > > > > This allows you also to be able to break up the transfer into batches, > > and check if files have changed before sync (say if your isilon files > > are not RO. > > > > Also ensure you have a recent version of rsync that preserves extended > > attributes and check your ACLS. > > > > A dry run example: > > > > https://unix.stackexchange.com/a/261372 > > > > I always felt more comfortable having a list of files before a sync…. > > > > I would counsel in the strongest possible terms against that approach. > > Basically you have to be assured that none of your file names have > "wacky" characters in them, because handling "wacky" characters in file > names is exceedingly difficult. I cannot stress how hard it is and the > above example does not handle all "wacky" characters in file names. > > So what do I mean by "wacky" characters. Well remember a file name can > have just about anything in it on Linux with the exception of '/', and > users especially when using a GUI, and even more so if they are Mac > users can and do use what I will call "wacky" characters in their file > names. > > The obvious ones are spaces, but it's not just ASCII 0x20, but tabs too. > Then there is the use of the wildcard characters, especially '?' but > also '*'. > > Not too difficult to handle you might say. Right now deal with a file > name with a newline character in it :-) Don't ask me how or why you even > do that but let me assure you that I have seen them on more than one > occasion. And now your dry run list is broken... > > Not only that if you have a few hundred million files to move a list > just becomes unwieldy anyway. > > One thing I didn't mention is that I would run anything with in a screen > (or tmux if that is your poison) and turn on logging. > > For those interested I am in the process of cleaning up the script a bit > and will post it somewhere in due course. > > > JAB. > > -- > Jonathan A. Buzzard Tel: +44141-5483420 > HPC System Administrator, ARCHIE-WeSt. > University of Strathclyde, John Anderson Building, Glasgow. G4 0NG > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss
