Re: rsync keeps writing files over

[Steven Levine]

In , on 06/02/16
   at 10:42 PM, "McDowell, Blake"  said:

Hi Blake,

>The storage is just an regular HDD in a mac pro tower. I can t imagine
>why it wouldn t handle timestamps. Also of note - this problem doesn t
>exist for every file, just the vast majority. So, that just makes it more
>confusing. 

Are the file systems the same on the source and the destination
partitions?

Check out

  --modify-window

in the help.  If the source and destination file systems have different
timestamp precision, this is the usual solution.

You can also try

  rsync /path-to-foo

where path-to-foo is a directory or file.  This will list the file
timestamps as rsync interprets them.

BTW, what version of rsync are you running?  It might matter.

Steven

-- 
--
"Steven Levine"   Warp/DIY/BlueLion etc.
www.scoug.com www.arcanoae.com www.warpcave.com
--


-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: rsync keeps writing files over

[Kevin Korb]

Rsync only cares about the modification time.  The ls command usually
abbreviates the timestamp so it is better to use the stat command on one
of the problem files to see the full thing.

On 06/02/2016 06:42 PM, McDowell, Blake wrote:
> Cool Thanks!
> 
> Specifically, the timestamps on both  and  match for "ls -l"
> but do not match for "ls -lu" or "ls -lc”
> 
> The storage is just an regular HDD in a mac pro tower. I can’t imagine why
> it wouldn’t handle timestamps. Also of note - this problem doesn’t exist
> for every file, just the vast majority. So, that just makes it more
> confusing. 
> 
> Yes, I imagine rsync is not the best for linear tape but give the choice
> between cp (which is faster and causes less problems but offers almost
> zero verbosity) and rsync, I’ll choose rsync. If people know of other
> options, I’d be very happy to know of them.
> 
> On 6/2/16, 6:33 PM, "Kevin Korb"  wrote:
> 
>> The man page has a section on what all the itemize-changes flags do.
>>
>> There is a --ignore-times but the result is what you have now, re-copy
>> everything even if the timestamp matches.
>>
>> The best you can really do with storage that can't handle timestamps is
>> to use --update.  But if you do that you need to get rid of --partial
>> (part of -P) or else rsync will never complete a file that was partially
>> copied since the partial copy is newer that the source file.
>>
>> OTOH, are you using rsync to copy files to a tape drive?  If so then I
>> would think rsync is not the right tool for dealing with linear storage.
>>
>> On 06/02/2016 06:25 PM, McDowell, Blake wrote:
>>> OK. Thanks. Where can I find information regarding how to interpret
>>> —itemize-changes?
>>>
>>> The timestamps aren’t changing, so the target must not be storing them,
>>> which I have no idea why. The directory I’m writing to is 777.
>>>
>>> What is the flag to tell rsync to ignore the timestamps?
>>>
>>> Thanks,
>>> Blake
>>>
>>> On 6/2/16, 6:18 PM, "rsync on behalf of Kevin Korb"
>>>  wrote:
>>>
 It is saying the timestamp is wrong and that it is copying the file and
 changing the timestamp.  If it does that every time then either the
 timestamps are changing on the source or the target isn't storing them.

 On 06/02/2016 06:13 PM, McDowell, Blake wrote:
> Thanks Kevin! I¹m unclear how to read the ‹itemize-changes output. Can
> you
> provide some insight?
>
> This is a local transfer from an external drive to an internal drive
> all
> attached to one computer.
>
>
> rsync -aPh --itemize-changes -n
> /Volumes/shuttle_05/2012_79_1_14_1__1199_Workprint
> /Volumes/3TB_LTO/LT003A/
> sending incremental file list
>> f..t... 
>>
>>
>> 2012_79_1_14_1__1199_Workprint/2012_79_1_14_1__1199_Workprint__Derivat
>> iv
>> es
>> /2012_79_1_14_1_DER_01.mov
>> f..t... 
>>
>>
>> 2012_79_1_14_1__1199_Workprint/2012_79_1_14_1__1199_Workprint__Derivat
>> iv
>> es
>> /2012_79_1_14_1_DER_02.mov
>> f..t... 
>>
>>
>> 2012_79_1_14_1__1199_Workprint/2012_79_1_14_1__1199_Workprint__UHD_DPX
>> /1
>> 19
>> 9WP_00086400.dpx
>> f..t... 
>>
>>
>> 2012_79_1_14_1__1199_Workprint/2012_79_1_14_1__1199_Workprint__UHD_DPX
>> /1
>> 19
>> 9WP_00086401.dpx
>> f..t... 
>>
>>
>> 2012_79_1_14_1__1199_Workprint/2012_79_1_14_1__1199_Workprint__UHD_DPX
>> /1
>> 19
>> 9WP_00086402.dpx
>> f..t... 
>>
>>
>> 2012_79_1_14_1__1199_Workprint/2012_79_1_14_1__1199_Workprint__UHD_DPX
>> /1
>> 19
>> 9WP_00086403.dpx
>> f..t... 
>>
>>
>> 2012_79_1_14_1__1199_Workprint/2012_79_1_14_1__1199_Workprint__UHD_DPX
>> /1
>> 19
>> 9WP_00086404.dpx
>> f..t... 
>>
>>
>> 2012_79_1_14_1__1199_Workprint/2012_79_1_14_1__1199_Workprint__UHD_DPX
>> /1
>> 19
>> 9WP_00086405.dpx
>> f..t... 
>>
>>
>> 2012_79_1_14_1__1199_Workprint/2012_79_1_14_1__1199_Workprint__UHD_DPX
>> /1
>> 19
>> 9WP_00086406.dpx
>
>
> ~Blake
>
>
>
> On 6/2/16, 5:44 PM, "rsync on behalf of Kevin Korb"
>  wrote:
>
>> Instead of the second -v (or even the first) add --itemize-changes.
>> It
>> will tell you why each file is being copied.  If the file timestamps
>> are
>> not correct then perhaps the underlying storage doesn't allow
>> arbitrary
>> mtime changes.
>>
>> On 06/02/2016 05:23 PM, McDowell, Blake wrote:
>>> Hi,
>>>
>>>  
>>>
>>> At my work we use rsync to move files between drives and to LTO
>>> among
>>> other things.
>>>
>>>  
>>>
>>> I¹m having an issue using rsync to move material between and
>>> 

Re: rsync keeps writing files over

[McDowell, Blake]

Cool Thanks!

Specifically, the timestamps on both  and  match for "ls -l"
but do not match for "ls -lu" or "ls -lc”

The storage is just an regular HDD in a mac pro tower. I can’t imagine why
it wouldn’t handle timestamps. Also of note - this problem doesn’t exist
for every file, just the vast majority. So, that just makes it more
confusing. 

Yes, I imagine rsync is not the best for linear tape but give the choice
between cp (which is faster and causes less problems but offers almost
zero verbosity) and rsync, I’ll choose rsync. If people know of other
options, I’d be very happy to know of them.

On 6/2/16, 6:33 PM, "Kevin Korb"  wrote:

>The man page has a section on what all the itemize-changes flags do.
>
>There is a --ignore-times but the result is what you have now, re-copy
>everything even if the timestamp matches.
>
>The best you can really do with storage that can't handle timestamps is
>to use --update.  But if you do that you need to get rid of --partial
>(part of -P) or else rsync will never complete a file that was partially
>copied since the partial copy is newer that the source file.
>
>OTOH, are you using rsync to copy files to a tape drive?  If so then I
>would think rsync is not the right tool for dealing with linear storage.
>
>On 06/02/2016 06:25 PM, McDowell, Blake wrote:
>> OK. Thanks. Where can I find information regarding how to interpret
>> —itemize-changes?
>> 
>> The timestamps aren’t changing, so the target must not be storing them,
>> which I have no idea why. The directory I’m writing to is 777.
>> 
>> What is the flag to tell rsync to ignore the timestamps?
>> 
>> Thanks,
>> Blake
>> 
>> On 6/2/16, 6:18 PM, "rsync on behalf of Kevin Korb"
>>  wrote:
>> 
>>> It is saying the timestamp is wrong and that it is copying the file and
>>> changing the timestamp.  If it does that every time then either the
>>> timestamps are changing on the source or the target isn't storing them.
>>>
>>> On 06/02/2016 06:13 PM, McDowell, Blake wrote:
 Thanks Kevin! I¹m unclear how to read the ‹itemize-changes output. Can
 you
 provide some insight?

 This is a local transfer from an external drive to an internal drive
all
 attached to one computer.


 rsync -aPh --itemize-changes -n
 /Volumes/shuttle_05/2012_79_1_14_1__1199_Workprint
 /Volumes/3TB_LTO/LT003A/
 sending incremental file list
> f..t... 
>
> 
>2012_79_1_14_1__1199_Workprint/2012_79_1_14_1__1199_Workprint__Derivat
>iv
> es
> /2012_79_1_14_1_DER_01.mov
> f..t... 
>
> 
>2012_79_1_14_1__1199_Workprint/2012_79_1_14_1__1199_Workprint__Derivat
>iv
> es
> /2012_79_1_14_1_DER_02.mov
> f..t... 
>
> 
>2012_79_1_14_1__1199_Workprint/2012_79_1_14_1__1199_Workprint__UHD_DPX
>/1
> 19
> 9WP_00086400.dpx
> f..t... 
>
> 
>2012_79_1_14_1__1199_Workprint/2012_79_1_14_1__1199_Workprint__UHD_DPX
>/1
> 19
> 9WP_00086401.dpx
> f..t... 
>
> 
>2012_79_1_14_1__1199_Workprint/2012_79_1_14_1__1199_Workprint__UHD_DPX
>/1
> 19
> 9WP_00086402.dpx
> f..t... 
>
> 
>2012_79_1_14_1__1199_Workprint/2012_79_1_14_1__1199_Workprint__UHD_DPX
>/1
> 19
> 9WP_00086403.dpx
> f..t... 
>
> 
>2012_79_1_14_1__1199_Workprint/2012_79_1_14_1__1199_Workprint__UHD_DPX
>/1
> 19
> 9WP_00086404.dpx
> f..t... 
>
> 
>2012_79_1_14_1__1199_Workprint/2012_79_1_14_1__1199_Workprint__UHD_DPX
>/1
> 19
> 9WP_00086405.dpx
> f..t... 
>
> 
>2012_79_1_14_1__1199_Workprint/2012_79_1_14_1__1199_Workprint__UHD_DPX
>/1
> 19
> 9WP_00086406.dpx


 ~Blake



 On 6/2/16, 5:44 PM, "rsync on behalf of Kevin Korb"
  wrote:

> Instead of the second -v (or even the first) add --itemize-changes.
>It
> will tell you why each file is being copied.  If the file timestamps
> are
> not correct then perhaps the underlying storage doesn't allow
>arbitrary
> mtime changes.
>
> On 06/02/2016 05:23 PM, McDowell, Blake wrote:
>> Hi,
>>
>>  
>>
>> At my work we use rsync to move files between drives and to LTO
>>among
>> other things.
>>
>>  
>>
>> I¹m having an issue using rsync to move material between and
>>external
>> drive and an internal drive.
>>
>>  
>>
>> We run ³rsync ­avvPh  ² and most of the files keep
>>writing
>> every time I run this. It appears that the modification times are
>>not
>> being carried through to the destination resulting in the files
>> continually wanting to re-write.
>>
>>  
>>
>> I¹m hoping someone can help me figure this out. Any information 

Re: rsync keeps writing files over

[Kevin Korb]

The man page has a section on what all the itemize-changes flags do.

There is a --ignore-times but the result is what you have now, re-copy
everything even if the timestamp matches.

The best you can really do with storage that can't handle timestamps is
to use --update.  But if you do that you need to get rid of --partial
(part of -P) or else rsync will never complete a file that was partially
copied since the partial copy is newer that the source file.

OTOH, are you using rsync to copy files to a tape drive?  If so then I
would think rsync is not the right tool for dealing with linear storage.

On 06/02/2016 06:25 PM, McDowell, Blake wrote:
> OK. Thanks. Where can I find information regarding how to interpret
> —itemize-changes?
> 
> The timestamps aren’t changing, so the target must not be storing them,
> which I have no idea why. The directory I’m writing to is 777.
> 
> What is the flag to tell rsync to ignore the timestamps?
> 
> Thanks,
> Blake
> 
> On 6/2/16, 6:18 PM, "rsync on behalf of Kevin Korb"
>  wrote:
> 
>> It is saying the timestamp is wrong and that it is copying the file and
>> changing the timestamp.  If it does that every time then either the
>> timestamps are changing on the source or the target isn't storing them.
>>
>> On 06/02/2016 06:13 PM, McDowell, Blake wrote:
>>> Thanks Kevin! I¹m unclear how to read the ‹itemize-changes output. Can
>>> you
>>> provide some insight?
>>>
>>> This is a local transfer from an external drive to an internal drive all
>>> attached to one computer.
>>>
>>>
>>> rsync -aPh --itemize-changes -n
>>> /Volumes/shuttle_05/2012_79_1_14_1__1199_Workprint
>>> /Volumes/3TB_LTO/LT003A/
>>> sending incremental file list
 f..t... 

 2012_79_1_14_1__1199_Workprint/2012_79_1_14_1__1199_Workprint__Derivativ
 es
 /2012_79_1_14_1_DER_01.mov
 f..t... 

 2012_79_1_14_1__1199_Workprint/2012_79_1_14_1__1199_Workprint__Derivativ
 es
 /2012_79_1_14_1_DER_02.mov
 f..t... 

 2012_79_1_14_1__1199_Workprint/2012_79_1_14_1__1199_Workprint__UHD_DPX/1
 19
 9WP_00086400.dpx
 f..t... 

 2012_79_1_14_1__1199_Workprint/2012_79_1_14_1__1199_Workprint__UHD_DPX/1
 19
 9WP_00086401.dpx
 f..t... 

 2012_79_1_14_1__1199_Workprint/2012_79_1_14_1__1199_Workprint__UHD_DPX/1
 19
 9WP_00086402.dpx
 f..t... 

 2012_79_1_14_1__1199_Workprint/2012_79_1_14_1__1199_Workprint__UHD_DPX/1
 19
 9WP_00086403.dpx
 f..t... 

 2012_79_1_14_1__1199_Workprint/2012_79_1_14_1__1199_Workprint__UHD_DPX/1
 19
 9WP_00086404.dpx
 f..t... 

 2012_79_1_14_1__1199_Workprint/2012_79_1_14_1__1199_Workprint__UHD_DPX/1
 19
 9WP_00086405.dpx
 f..t... 

 2012_79_1_14_1__1199_Workprint/2012_79_1_14_1__1199_Workprint__UHD_DPX/1
 19
 9WP_00086406.dpx
>>>
>>>
>>> ~Blake
>>>
>>>
>>>
>>> On 6/2/16, 5:44 PM, "rsync on behalf of Kevin Korb"
>>>  wrote:
>>>
 Instead of the second -v (or even the first) add --itemize-changes.  It
 will tell you why each file is being copied.  If the file timestamps
 are
 not correct then perhaps the underlying storage doesn't allow arbitrary
 mtime changes.

 On 06/02/2016 05:23 PM, McDowell, Blake wrote:
> Hi,
>
>  
>
> At my work we use rsync to move files between drives and to LTO among
> other things.
>
>  
>
> I¹m having an issue using rsync to move material between and external
> drive and an internal drive.
>
>  
>
> We run ³rsync ­avvPh  ² and most of the files keep writing
> every time I run this. It appears that the modification times are not
> being carried through to the destination resulting in the files
> continually wanting to re-write.
>
>  
>
> I¹m hoping someone can help me figure this out. Any information you
> need
> from me (logs, etc) I¹m happy to try and provide.
>
>  
>
> Many thanks,
>
> Blake  
>
>
>

 -- 

 ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
Kevin Korb  Phone:(407) 252-6853
Systems Administrator   Internet:
FutureQuest, Inc.   ke...@futurequest.net  (work)
Orlando, Floridak...@sanitarium.net (personal)
Web page:   http://www.sanitarium.net/
PGP public key available on web site.

 ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,

>>
>> -- 
>> ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
>>  Kevin Korb  Phone:(407) 252-6853
>>  Systems Administrator   Internet:
>>  FutureQuest, Inc.   

Re: rsync keeps writing files over

[McDowell, Blake]

OK. Thanks. Where can I find information regarding how to interpret
—itemize-changes?

The timestamps aren’t changing, so the target must not be storing them,
which I have no idea why. The directory I’m writing to is 777.

What is the flag to tell rsync to ignore the timestamps?

Thanks,
Blake

On 6/2/16, 6:18 PM, "rsync on behalf of Kevin Korb"
 wrote:

>It is saying the timestamp is wrong and that it is copying the file and
>changing the timestamp.  If it does that every time then either the
>timestamps are changing on the source or the target isn't storing them.
>
>On 06/02/2016 06:13 PM, McDowell, Blake wrote:
>> Thanks Kevin! I¹m unclear how to read the ‹itemize-changes output. Can
>>you
>> provide some insight?
>> 
>> This is a local transfer from an external drive to an internal drive all
>> attached to one computer.
>> 
>> 
>> rsync -aPh --itemize-changes -n
>> /Volumes/shuttle_05/2012_79_1_14_1__1199_Workprint
>>/Volumes/3TB_LTO/LT003A/
>> sending incremental file list
>>> f..t... 
>>> 
>>>2012_79_1_14_1__1199_Workprint/2012_79_1_14_1__1199_Workprint__Derivativ
>>>es
>>> /2012_79_1_14_1_DER_01.mov
>>> f..t... 
>>> 
>>>2012_79_1_14_1__1199_Workprint/2012_79_1_14_1__1199_Workprint__Derivativ
>>>es
>>> /2012_79_1_14_1_DER_02.mov
>>> f..t... 
>>> 
>>>2012_79_1_14_1__1199_Workprint/2012_79_1_14_1__1199_Workprint__UHD_DPX/1
>>>19
>>> 9WP_00086400.dpx
>>> f..t... 
>>> 
>>>2012_79_1_14_1__1199_Workprint/2012_79_1_14_1__1199_Workprint__UHD_DPX/1
>>>19
>>> 9WP_00086401.dpx
>>> f..t... 
>>> 
>>>2012_79_1_14_1__1199_Workprint/2012_79_1_14_1__1199_Workprint__UHD_DPX/1
>>>19
>>> 9WP_00086402.dpx
>>> f..t... 
>>> 
>>>2012_79_1_14_1__1199_Workprint/2012_79_1_14_1__1199_Workprint__UHD_DPX/1
>>>19
>>> 9WP_00086403.dpx
>>> f..t... 
>>> 
>>>2012_79_1_14_1__1199_Workprint/2012_79_1_14_1__1199_Workprint__UHD_DPX/1
>>>19
>>> 9WP_00086404.dpx
>>> f..t... 
>>> 
>>>2012_79_1_14_1__1199_Workprint/2012_79_1_14_1__1199_Workprint__UHD_DPX/1
>>>19
>>> 9WP_00086405.dpx
>>> f..t... 
>>> 
>>>2012_79_1_14_1__1199_Workprint/2012_79_1_14_1__1199_Workprint__UHD_DPX/1
>>>19
>>> 9WP_00086406.dpx
>> 
>> 
>> ~Blake
>> 
>> 
>> 
>> On 6/2/16, 5:44 PM, "rsync on behalf of Kevin Korb"
>>  wrote:
>> 
>>> Instead of the second -v (or even the first) add --itemize-changes.  It
>>> will tell you why each file is being copied.  If the file timestamps
>>>are
>>> not correct then perhaps the underlying storage doesn't allow arbitrary
>>> mtime changes.
>>>
>>> On 06/02/2016 05:23 PM, McDowell, Blake wrote:
 Hi,

  

 At my work we use rsync to move files between drives and to LTO among
 other things.

  

 I¹m having an issue using rsync to move material between and external
 drive and an internal drive.

  

 We run ³rsync ­avvPh  ² and most of the files keep writing
 every time I run this. It appears that the modification times are not
 being carried through to the destination resulting in the files
 continually wanting to re-write.

  

 I¹m hoping someone can help me figure this out. Any information you
need
 from me (logs, etc) I¹m happy to try and provide.

  

 Many thanks,

 Blake  



>>>
>>> -- 
>>> 
>>>~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
>>> Kevin Korb  Phone:(407) 252-6853
>>> Systems Administrator   Internet:
>>> FutureQuest, Inc.   ke...@futurequest.net  (work)
>>> Orlando, Floridak...@sanitarium.net (personal)
>>> Web page:   http://www.sanitarium.net/
>>> PGP public key available on web site.
>>> 
>>>~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
>>>
>
>-- 
>~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
>   Kevin Korb  Phone:(407) 252-6853
>   Systems Administrator   Internet:
>   FutureQuest, Inc.   ke...@futurequest.net  (work)
>   Orlando, Floridak...@sanitarium.net (personal)
>   Web page:   http://www.sanitarium.net/
>   PGP public key available on web site.
>~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
>

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: rsync keeps writing files over

[Kevin Korb]

It is saying the timestamp is wrong and that it is copying the file and
changing the timestamp.  If it does that every time then either the
timestamps are changing on the source or the target isn't storing them.

On 06/02/2016 06:13 PM, McDowell, Blake wrote:
> Thanks Kevin! I¹m unclear how to read the ‹itemize-changes output. Can you
> provide some insight?
> 
> This is a local transfer from an external drive to an internal drive all
> attached to one computer.
> 
> 
> rsync -aPh --itemize-changes -n
> /Volumes/shuttle_05/2012_79_1_14_1__1199_Workprint /Volumes/3TB_LTO/LT003A/
> sending incremental file list
>> f..t... 
>> 2012_79_1_14_1__1199_Workprint/2012_79_1_14_1__1199_Workprint__Derivatives
>> /2012_79_1_14_1_DER_01.mov
>> f..t... 
>> 2012_79_1_14_1__1199_Workprint/2012_79_1_14_1__1199_Workprint__Derivatives
>> /2012_79_1_14_1_DER_02.mov
>> f..t... 
>> 2012_79_1_14_1__1199_Workprint/2012_79_1_14_1__1199_Workprint__UHD_DPX/119
>> 9WP_00086400.dpx
>> f..t... 
>> 2012_79_1_14_1__1199_Workprint/2012_79_1_14_1__1199_Workprint__UHD_DPX/119
>> 9WP_00086401.dpx
>> f..t... 
>> 2012_79_1_14_1__1199_Workprint/2012_79_1_14_1__1199_Workprint__UHD_DPX/119
>> 9WP_00086402.dpx
>> f..t... 
>> 2012_79_1_14_1__1199_Workprint/2012_79_1_14_1__1199_Workprint__UHD_DPX/119
>> 9WP_00086403.dpx
>> f..t... 
>> 2012_79_1_14_1__1199_Workprint/2012_79_1_14_1__1199_Workprint__UHD_DPX/119
>> 9WP_00086404.dpx
>> f..t... 
>> 2012_79_1_14_1__1199_Workprint/2012_79_1_14_1__1199_Workprint__UHD_DPX/119
>> 9WP_00086405.dpx
>> f..t... 
>> 2012_79_1_14_1__1199_Workprint/2012_79_1_14_1__1199_Workprint__UHD_DPX/119
>> 9WP_00086406.dpx
> 
> 
> ~Blake
> 
> 
> 
> On 6/2/16, 5:44 PM, "rsync on behalf of Kevin Korb"
>  wrote:
> 
>> Instead of the second -v (or even the first) add --itemize-changes.  It
>> will tell you why each file is being copied.  If the file timestamps are
>> not correct then perhaps the underlying storage doesn't allow arbitrary
>> mtime changes.
>>
>> On 06/02/2016 05:23 PM, McDowell, Blake wrote:
>>> Hi,
>>>
>>>  
>>>
>>> At my work we use rsync to move files between drives and to LTO among
>>> other things.
>>>
>>>  
>>>
>>> I¹m having an issue using rsync to move material between and external
>>> drive and an internal drive.
>>>
>>>  
>>>
>>> We run ³rsync ­avvPh  ² and most of the files keep writing
>>> every time I run this. It appears that the modification times are not
>>> being carried through to the destination resulting in the files
>>> continually wanting to re-write.
>>>
>>>  
>>>
>>> I¹m hoping someone can help me figure this out. Any information you need
>>> from me (logs, etc) I¹m happy to try and provide.
>>>
>>>  
>>>
>>> Many thanks,
>>>
>>> Blake  
>>>
>>>
>>>
>>
>> -- 
>> ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
>>  Kevin Korb  Phone:(407) 252-6853
>>  Systems Administrator   Internet:
>>  FutureQuest, Inc.   ke...@futurequest.net  (work)
>>  Orlando, Floridak...@sanitarium.net (personal)
>>  Web page:   http://www.sanitarium.net/
>>  PGP public key available on web site.
>> ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
>>

-- 
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
Kevin Korb  Phone:(407) 252-6853
Systems Administrator   Internet:
FutureQuest, Inc.   ke...@futurequest.net  (work)
Orlando, Floridak...@sanitarium.net (personal)
Web page:   http://www.sanitarium.net/
PGP public key available on web site.
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,



signature.asc
Description: OpenPGP digital signature
-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: rsync keeps writing files over

[McDowell, Blake]

Thanks Kevin! I¹m unclear how to read the ‹itemize-changes output. Can you
provide some insight?

This is a local transfer from an external drive to an internal drive all
attached to one computer.


rsync -aPh --itemize-changes -n
/Volumes/shuttle_05/2012_79_1_14_1__1199_Workprint /Volumes/3TB_LTO/LT003A/
sending incremental file list
>f..t... 
>2012_79_1_14_1__1199_Workprint/2012_79_1_14_1__1199_Workprint__Derivatives
>/2012_79_1_14_1_DER_01.mov
>f..t... 
>2012_79_1_14_1__1199_Workprint/2012_79_1_14_1__1199_Workprint__Derivatives
>/2012_79_1_14_1_DER_02.mov
>f..t... 
>2012_79_1_14_1__1199_Workprint/2012_79_1_14_1__1199_Workprint__UHD_DPX/119
>9WP_00086400.dpx
>f..t... 
>2012_79_1_14_1__1199_Workprint/2012_79_1_14_1__1199_Workprint__UHD_DPX/119
>9WP_00086401.dpx
>f..t... 
>2012_79_1_14_1__1199_Workprint/2012_79_1_14_1__1199_Workprint__UHD_DPX/119
>9WP_00086402.dpx
>f..t... 
>2012_79_1_14_1__1199_Workprint/2012_79_1_14_1__1199_Workprint__UHD_DPX/119
>9WP_00086403.dpx
>f..t... 
>2012_79_1_14_1__1199_Workprint/2012_79_1_14_1__1199_Workprint__UHD_DPX/119
>9WP_00086404.dpx
>f..t... 
>2012_79_1_14_1__1199_Workprint/2012_79_1_14_1__1199_Workprint__UHD_DPX/119
>9WP_00086405.dpx
>f..t... 
>2012_79_1_14_1__1199_Workprint/2012_79_1_14_1__1199_Workprint__UHD_DPX/119
>9WP_00086406.dpx


~Blake



On 6/2/16, 5:44 PM, "rsync on behalf of Kevin Korb"
 wrote:

>Instead of the second -v (or even the first) add --itemize-changes.  It
>will tell you why each file is being copied.  If the file timestamps are
>not correct then perhaps the underlying storage doesn't allow arbitrary
>mtime changes.
>
>On 06/02/2016 05:23 PM, McDowell, Blake wrote:
>> Hi,
>> 
>>  
>> 
>> At my work we use rsync to move files between drives and to LTO among
>> other things.
>> 
>>  
>> 
>> I¹m having an issue using rsync to move material between and external
>> drive and an internal drive.
>> 
>>  
>> 
>> We run ³rsync ­avvPh  ² and most of the files keep writing
>> every time I run this. It appears that the modification times are not
>> being carried through to the destination resulting in the files
>> continually wanting to re-write.
>> 
>>  
>> 
>> I¹m hoping someone can help me figure this out. Any information you need
>> from me (logs, etc) I¹m happy to try and provide.
>> 
>>  
>> 
>> Many thanks,
>> 
>> Blake  
>> 
>> 
>> 
>
>-- 
>~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
>   Kevin Korb  Phone:(407) 252-6853
>   Systems Administrator   Internet:
>   FutureQuest, Inc.   ke...@futurequest.net  (work)
>   Orlando, Floridak...@sanitarium.net (personal)
>   Web page:   http://www.sanitarium.net/
>   PGP public key available on web site.
>~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
>


-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

rsync keeps writing files over

[McDowell, Blake]

Hi,

At my work we use rsync to move files between drives and to LTO among other 
things.

I'm having an issue using rsync to move material between and external drive and 
an internal drive.

We run "rsync -avvPh  " and most of the files keep writing every 
time I run this. It appears that the modification times are not being carried 
through to the destination resulting in the files continually wanting to 
re-write.

I'm hoping someone can help me figure this out. Any information you need from 
me (logs, etc) I'm happy to try and provide.

Many thanks,
Blake
-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: rsync keeps writing files over

[Kevin Korb]

Instead of the second -v (or even the first) add --itemize-changes.  It
will tell you why each file is being copied.  If the file timestamps are
not correct then perhaps the underlying storage doesn't allow arbitrary
mtime changes.

On 06/02/2016 05:23 PM, McDowell, Blake wrote:
> Hi,
> 
>  
> 
> At my work we use rsync to move files between drives and to LTO among
> other things.
> 
>  
> 
> I’m having an issue using rsync to move material between and external
> drive and an internal drive.
> 
>  
> 
> We run “rsync –avvPh  ” and most of the files keep writing
> every time I run this. It appears that the modification times are not
> being carried through to the destination resulting in the files
> continually wanting to re-write.
> 
>  
> 
> I’m hoping someone can help me figure this out. Any information you need
> from me (logs, etc) I’m happy to try and provide.
> 
>  
> 
> Many thanks,
> 
> Blake  
> 
> 
> 

-- 
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
Kevin Korb  Phone:(407) 252-6853
Systems Administrator   Internet:
FutureQuest, Inc.   ke...@futurequest.net  (work)
Orlando, Floridak...@sanitarium.net (personal)
Web page:   http://www.sanitarium.net/
PGP public key available on web site.
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,



signature.asc
Description: OpenPGP digital signature
-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Rsync unexpectedly removing ALL files

[Koehn, Troy N.]

I am using rsync with --files-from option with a file named 'rsync.in' that 
contains 37K+ directories that are queried from a database. One of the 
directories looked like this:

/home/usr/directory/./

I know the directory should NOT end with a './' but this was stored in the 
database that way by another process. When performing the rsync, this one 
directory caused all files from the root '/' directory on down to be deleted. 
However, if ran without the --files-from option it works as expected. It is 
because of the './' on the end and using the 'files-from' option but I don't 
know why rsync would do that.

Obviously not allowing these directories to be stored in the database in the 
first place is the thing to do, but I just want to understand why rsync would 
do this so I can ensure I prevent things like this in the future. Here's the 
command (with the dry-run option for safety).

rsync --dry-run --perms --delete --compress --dirs --links --times 
--keep-dirlinks --files-from=/usr/home/rsync.in /home/src/shared/directory/ /

BTW, rsync  version 3.1.0  protocol version 31
-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Delete remote folder

[Steven Levine]

In <20160602082722.8F11829233B7@macserver.private>, on 06/02/16
   at 10:27 AM, Fabian Cenedese  said:

Hi Fabian,

>rsync -a --delete empty_local_folder remote_folder_to_delete

>to get rid of the contents. However the folder itself still remains. So
>after a while I'd have a lot of empty folders.

This comes up on the list every now and then.

Here's a snip from an archive message:

  Here is how you would write an "rsyncrm" script that can be invoked like
  "rsyncrm -ni remote::volume/directory".  Note that this reveals the
  attributes of the current directory to the server.

  #!/bin/bash
  victim="${@: -1:1}"
  rsync -d --del --filter="R /$(basename -- "$victim")" --filter='- /*' \
  "${@:1:$#-1}" . "$(dirname -- "$victim")"

This works because it deletes the unwanted directory from the parent
directory.

Steven

-- 
--
"Steven Levine"   Warp/DIY/BlueLion etc.
www.scoug.com www.arcanoae.com www.warpcave.com
--


-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Delete remote folder

[Fabian Cenedese]

At 13:22 02.06.2016, you wrote:
>Content-Transfer-Encoding: 7bit
>
>Fabian Cenedese  wrote:
>
>> This script is bash and also uses the "remote shell hacks" using SSH.
>> As I want to run it also from Windows I'm looking for a rsync solution.
>
>Assuming you have control of the server, can you do a bit of semaphore ?
>
>Eg, do your backup with rsync, then when it's complete, touch a file (or put 
>something in it) and rsync that to the server. The server then runs a process 
>that does some action.
>
>Eg, you touch a file "/etc/ivedoneabackup" and sync it, when the server sees 
>that, it runs a process which links/moves/whatever stuff around to suit.
>The client side can be scripted in whatever language is available on that 
>platform.

There are several ways to solve it on the server, from simple cron jobs
to do the cleanup to more sophisticated solutions. But what if I don't
have a server but e.g. just a USB disk? Then it has to be done from
the client. Of course now you have full access to the backups and can
do whatever you want. However that depends again on the operating
system and maybe even file system or so. I don't want to (but maybe
have to) implement various solutions depending on local/remote,
source Windows/Linux, destination Windows/Linux etc.

But thanks for your inputs, I may have to do some more thinking.

bye  Fabi


-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Delete remote folder

[Simon Hobson]

Fabian Cenedese  wrote:

> This script is bash and also uses the "remote shell hacks" using SSH.
> As I want to run it also from Windows I'm looking for a rsync solution.

Assuming you have control of the server, can you do a bit of semaphore ?

Eg, do your backup with rsync, then when it's complete, touch a file (or put 
something in it) and rsync that to the server. The server then runs a process 
that does some action.

Eg, you touch a file "/etc/ivedoneabackup" and sync it, when the server sees 
that, it runs a process which links/moves/whatever stuff around to suit.
The client side can be scripted in whatever language is available on that 
platform.


-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Delete remote folder

[Simon Hobson]

Fabian Cenedese  wrote:

> Are there pure rsync ways to solve these two problems?

Short answer - no I don't think there is.
My feeling is that rsync (at least, rsync on it's own) isn't the right tool for 
the job.

One think I would comment on though is that, IMO, making backup policies under 
the control of the client is "asking for trouble" for the general case. Your 
use case may well be different, but IME you can't trust clients to do the right 
thing - unless they are completely under your control, in which case there's 
little difference between client or server controlled policies.

Personally I use a combination of rsync to maintain a clone of each client onto 
one server, and then StoreBackup (on that one server) to manage multiple 
generations of backups with file-level dedupe etc.


-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Delete remote folder

[Fabian Cenedese]

At 10:52 02.06.2016, Hans-Christian Jehg wrote:
>Content-Transfer-Encoding: 7bit
>
>Try having a look at this, for inspiration.
>
>It does require the capability of hard-links on the remote end though... and 
>it does not resolve your "delete target directory" either, but it might save 
>you some fiddeling around :-)
>
>https://blog.interlinked.org/tutorials/rsync_time_machine.html

Thanks, I already saw that before. I looked for existing backup programs
but none really did what I wanted that's why I'm trying to do it myself.

This script is bash and also uses the "remote shell hacks" using SSH.
As I want to run it also from Windows I'm looking for a rsync solution.

bye  Fabi


-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Delete remote folder

[Hans-Christian Jehg]

Try having a look at this, for inspiration.

It does require the capability of hard-links on the remote end though... 
and it does not resolve your "delete target directory" either, but it 
might save you some fiddeling around :-)


https://blog.interlinked.org/tutorials/rsync_time_machine.html

HC

On 02-06-2016 10:27, Fabian Cenedese wrote:

Hi

I'm trying to build a backup system based on rsync. It should
work with different destinations (local, share, remote etc) that's
why I'm looking for ways how every operation can be controlled
from the client side.

I want to create full backups and incremental backups. For the
incremental backups I use --compare-dest lastfullbackup. This
works nicely except that the whole directory tree is created
even if no files are copied. This is mentioned several times
on the Internet and the usual answer is that there's no way
around it because of the way that rsync works. I don't mind
that they are created if there's a (rsync-)way to delete them
again afterwards, something like

rsync -a --delete --prune-empty-dirs dest dest

Unfortunately at least one place has to be local, doing a
remote-remote is not possible.

The second problem is that I want to delete older backups.
I can use

rsync -a --delete empty_local_folder remote_folder_to_delete

to get rid of the contents. However the folder itself still remains.
So after a while I'd have a lot of empty folders.

As I want this to work on various systems I can't (and don't
want to) use remote shell hacks like "do_something && rsync ".
Also the policy of what and when to delete should be configurable
from the client.

Are there pure rsync ways to solve these two problems?

Thanks

bye  Fabi





--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Delete remote folder

[Fabian Cenedese]

Hi

I'm trying to build a backup system based on rsync. It should
work with different destinations (local, share, remote etc) that's
why I'm looking for ways how every operation can be controlled
from the client side.

I want to create full backups and incremental backups. For the
incremental backups I use --compare-dest lastfullbackup. This
works nicely except that the whole directory tree is created
even if no files are copied. This is mentioned several times
on the Internet and the usual answer is that there's no way
around it because of the way that rsync works. I don't mind
that they are created if there's a (rsync-)way to delete them
again afterwards, something like

rsync -a --delete --prune-empty-dirs dest dest

Unfortunately at least one place has to be local, doing a
remote-remote is not possible.

The second problem is that I want to delete older backups.
I can use

rsync -a --delete empty_local_folder remote_folder_to_delete

to get rid of the contents. However the folder itself still remains.
So after a while I'd have a lot of empty folders.

As I want this to work on various systems I can't (and don't
want to) use remote shell hacks like "do_something && rsync ".
Also the policy of what and when to delete should be configurable
from the client.

Are there pure rsync ways to solve these two problems?

Thanks

bye  Fabi


-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html