propagate atimes with rsync-2.6.6

[Christian Konz T3]

Dear Wayne.

Thank you very much for your extensive comments on the our patch. I will
try and take them into account and mend our patch accordingly.
Unfortunately, I can only do this during my leisure time which is already
very limited so it might take some time.
But it would be very nice if this feature could make it into the reference
version. We consider it as very useful if not necessary for back-ups of
systems where the access to files by employees is monitored. That's
especially relevant if the usage of a database should be assessed. We also
thought of extending the changes such that rsync would also transfer
atimes if the modification times are the same but we dropped that for the
moment since the hardlink-feature allows this already.
I have received an email from Emile LeBlanc who supports the development
of these features very much. The initiative for our changes came from a
company with daily back-ups of their database. Maybe, this feature may be
useful to other people too.
Maybe, we could merge the two patches to create a nice new feature.
That would be great.
Thank you very much,

Christian


-- 
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

propagate atimes with rsync-2.6.6 (fwd)

[Christian Konz T3]

Dear Martin Pool.

We regularly use rsync for making backups of our file systems but we have
noticed that the atimes are not transferred with the files and are also
always updated on the sender's side. Therefore, we have created a modified
version of rsync based on rsync-2.6.6 protocol version 29 which transfers
the access times with the transferred files and also allows to preserve
the access times on the sending side. The transfer of the atimes is done
by default while the preservation of the source atimes is switched on by
an additional option (-X --atime-preserve). To guarantee compatibility
with previous protocol versions, we have introduced a new protocol version
(30). If the remote protocol version is older than 30, rsync behaves as
before. The changes made to the reference version are small and kept in
accordance with the original style. The affected files are:
flist.c
generator.c
options.c
proto.h
rsync.1
rsync.c
rsync.h
rsync3.txt
sender.c
and util.c
Please find the corresponding patch attached to this email.
We have thouroughly tested our new protocol under Linux and also compiled
it under Unix (Aix). It works as expected under Linux while some of the
changes were redundant under Unix.
We consider the preservation of the source access times as useful since
this information might be worth being backed-up as well.
We hope our approach will find your attention and would be glad if it were
included in the next update of rsync.
Please let us know about your suggestions for improvements.
Yours truly,

Christian Konz
diff -uNr rsync-2.6.6/flist.c rsync-2.6.6_patch/flist.c
--- rsync-2.6.6/flist.c 2005-07-07 21:49:14.0 +0200
+++ rsync-2.6.6_patch/flist.c   2006-01-07 14:04:38.0 +0100
@@ -54,6 +54,7 @@
 extern int implied_dirs;
 extern int copy_links;
 extern int copy_unsafe_links;
+extern int remote_protocol;
 extern int protocol_version;
 extern int sanitize_paths;
 extern int orig_umask;
@@ -314,6 +315,7 @@
 {
unsigned short flags;
static time_t modtime;
+   static time_t acctime;
static mode_t mode;
static int64 dev;
static dev_t rdev;
@@ -329,7 +331,7 @@
 
if (!file) {
write_byte(f, 0);
-   modtime = 0, mode = 0;
+   modtime = 0, acctime = 0, mode = 0;
dev = 0, rdev = makedev(0, 0);
rdev_major = 0;
uid = 0, gid = 0;
@@ -379,6 +381,8 @@
else
modtime = file->modtime;
 
+acctime = file->acctime;
+
 #ifdef SUPPORT_HARD_LINKS
if (file->link_u.idev) {
if (file->F_DEV == dev) {
@@ -431,6 +435,8 @@
write_longint(f, file->length);
if (!(flags & XMIT_SAME_TIME))
write_int(f, modtime);
+   if (remote_protocol >= 30)
+   write_int(f, acctime);
if (!(flags & XMIT_SAME_MODE))
write_int(f, to_wire_mode(mode));
if (preserve_uid && !(flags & XMIT_SAME_UID)) {
@@ -506,6 +512,7 @@
  unsigned short flags, int f)
 {
static time_t modtime;
+   static time_t acctime;
static mode_t mode;
static int64 dev;
static dev_t rdev;
@@ -524,7 +531,7 @@
struct file_struct *file;
 
if (!flist) {
-   modtime = 0, mode = 0;
+   modtime = 0, acctime = 0, mode = 0;
dev = 0, rdev = makedev(0, 0);
rdev_major = 0;
uid = 0, gid = 0;
@@ -578,6 +585,10 @@
file_length = read_longint(f);
if (!(flags & XMIT_SAME_TIME))
modtime = (time_t)read_int(f);
+   if (remote_protocol >= 30)
+   acctime = (time_t)read_int(f);
+   else
+   acctime = time(NULL);
if (!(flags & XMIT_SAME_MODE))
mode = from_wire_mode(read_int(f));
 
@@ -630,6 +641,7 @@
 
file->flags = 0;
file->modtime = modtime;
+   file->acctime = acctime;
file->length = file_length;
file->mode = mode;
file->uid = uid;
@@ -882,6 +894,7 @@
 
file->flags = flags;
file->modtime = st.st_mtime;
+   file->acctime = st.st_atime;
file->length = st.st_size;
file->mode = st.st_mode;
file->uid = st.st_uid;
diff -uNr rsync-2.6.6/generator.c rsync-2.6.6_patch/generator.c
--- rsync-2.6.6/generator.c 2005-07-28 21:06:03.0 +0200
+++ rsync-2.6.6_patch/generator.c   2005-11-25 11:33:33.0 +0100
@@ -917,6 +917,13 @@
  fnamecmpbuf, 1,
  itemizing && verbose > 1,
  code) == 0) {
+   if (set_modtime(fname,file->modtime,
+   file->acctime) != 0) {
+  

Re: propagate atimes with rsync-2.6.6

[Matt McCutchen]

On Thu, 2006-01-26 at 01:12 -0800, Wayne Davison wrote:
> On Tue, Jan 24, 2006 at 05:46:36PM +0100, Christian Konz T3 wrote:
> > Therefore, we have created a modified version of rsync based on
> > rsync-2.6.6 protocol version 29 which transfers the access times with
> > the transferred files and also allows to preserve the access times on
> > the sending side.
>
> - Your patch's optional preservation of the access times on the
>   sending side is done by forcing both the access time and the modify
>   time to the values that were acquired at the start of a file's
>   transfer.[...]

My personal stance on file times is as follows.  A file's mtime is a
useful counterpart to its data because it identifies the version of the
file; two files with identical names and mtimes are very likely to have
identical data (as rsync's quick check assumes).  Thus, mtimes should
usually be kept with data and preserved when a file's data is copied.
On the other hand, atime and ctime have hardly any use to normal users
and should be available only to the owner of the filesystem for forensic
purposes, if at all.  I especially have trouble with atime because
"reading shouldn't write".  Thus, all my computer's filesystems are
mounted noatime, and all my files on my computer and others' computers
carry the ext2 attribute A.

Enough digression.  While I don't have any desire to copy atimes to my
backups, others might want this feature, so we might as well add an
option to rsync.  On the other hand, the issue of changing back or
leaving undisturbed the atimes of the source files has been discussed
extensively on the tar mailing list.  The thread starts here:
   http://lists.gnu.org/archive/html/bug-tar/2005-09/msg00035.html

Summary: For a while, tar has supported setting atimes back with
utime(2).  The new O_NOATIME option to open(2) provides a better way:
one can open a file one owns without disturbing its atime.  However,
O_NOATIME only works on some kernels; on others, it is silently ignored.
I think tar can now be told to use one method or the other or to try and
figure out whether O_NOATIME works and then act accordingly.
-- 
Matt McCutchen, ``hashproduct''
[EMAIL PROTECTED] -- http://mysite.verizon.net/hashproduct/

-- 
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: propagate atimes with rsync-2.6.6 (fwd)

[Wayne Davison]

On Tue, Jan 24, 2006 at 05:46:36PM +0100, Christian Konz T3 wrote:
> Therefore, we have created a modified version of rsync based on
> rsync-2.6.6 protocol version 29 which transfers the access times with
> the transferred files and also allows to preserve the access times on
> the sending side.

First -- thanks for the patch.  You've unfortunately duplicated some
work that was already done (see the file patches/atimes.diff in the
rsync source), but your version does do things a little differently:

- The atimes.diff patch does not require a protocol bump because the
  new behavior is not enabled by default; it is turned on via the
  --atimes (-U) option.  (I personally don't think that preserving
  the atime should happen by default, not even with -a.)

- The atimes.diff patch omits directories from its access-time
  preservation (since it is very likely that the atime on directories
  will change outside of rsync's control and some systems don't like
  the atime to be forced on a directory).

- The atimes.diff patch adds a new XMIT_SAME_ATIME flag that avoids
  sending the same atime value twice in a row (just as the regular
  code uses XMIT_SAME_TIME for the mtime value).

- The atimes.diff patch changes the output of --itemize-changes when
  --atimes is specified to indicate which files needed an access-time
  change.

- Your patch has an option for leaving the access time unchanged on
  the source where the atimes.diff patch does not (though see below
  for my comments on this).

The main reason that the --atimes option has not made it into the main
source is that I was not willing to increase the amount of per-file
memory needed to make room for a value that most people didn't need
(since the preservation of access times is not commonly required).
However, I recently modified the atimes.diff to make it only use extra
memory when the --atimes option was actually specified, so I am
considering adding the option to the next version (its inclusion now
mainly depends on if I consider this memory-saving design too much of
a kluge or not).

Here are some comments on your patch:

- Your code should not be checking the remote_protocol variable, but
  the protocol_version variable.  This is because the negotiated
  protocol version can differ from the maximum remote value (mainly
  by the use of the --protocol option).

- You have code that sets the atime on a file that was hard-linked
  via --link-dest.  This is probably the right choice (and one that
  the atimes.diff patch fails to consider):  it can mean that older
  version(s) of identical files can get the access time updated, but
  it's probably not an important enough of a difference to force a
  new version to be copied into the destination (rather than
  hard-linked to a --link-dest version).
  
- Your change in set_perms() only sets the atime if the mtime differs.

- Your patch's optional preservation of the access times on the
  sending side is done by forcing both the access time and the modify
  time to the values that were acquired at the start of a file's
  transfer.  This can be unsafe if someone managed to change the file
  during the time that the file was being sent.  You should at least
  re-stat the file at the end of the transfer to make sure that the
  modify time hasn't changed (and skip the update if it has).  Even
  then, I'd be uneasy about this method of trying to leave the access
  time unchanged on the source because there would still be a chance
  that it would obliterate an updated mtime.

Overall you did a very nice job of changing the code.  Thanks again
for helping out.

..wayne..
-- 
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html