Re: Rsync deletes too much?

[Marco Nicosia]

Yes, that was it. I'd forgotten that /usr was a separately mounted FS.

Thank you.

-- Marco

John Van Essen ([EMAIL PROTECTED]) wrote: 
> On Sun, 12 Oct 2003, Marco Nicosia <[EMAIL PROTECTED]> wrote:
> >
> >Hello. I'm having a problem using rsync to copy my root partition to a
> >new disk. This disk has a previous copy on it, and rsync seems determined
> >to (unnecessarily) delete files from it. Worse, despite the fact that the
> >files DO exist on the source, it does not update the destination, leaving
> >me with less than I had before?!
> >
> >Here's my usage:
> >
> >] server# /usr/local/bin/rsync -naxv --delete --progress --stats --exclude '/dev/' 
> >--exclude '/tmp/' --exclude '/usr/var/tmp/' --exclude '/usr/var/run/' / /tmp/bkup/ 
> >| grep -c deleting
> >] 105668
> 
> You are using -x, which will not include the contents of any mounted
> partitions.  105,668 deletes and no update of the destination makes me
> think that /usr on the source is a mounted partition, which would
> explain both symptoms.
> -- 
> John Van Essen  Univ of MN Alumnus  <[EMAIL PROTECTED]>
> 
___
Marco E. Nicosia  |  http://www.escape.org/~marco/  |  [EMAIL PROTECTED]
-- 
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Unable to rsync files of size 30 GB

[nitin.agarwal]

Hi..
I m using rsync on AIX 4.3 for backing up a huge database. Rsync gives error on 
bringing files greater than 2 GB. I tried the 64 bit version of Rsync 2.5.6 (on 64bit 
file system) also, but it didnt worked.

Please advice... 
Thanks in Advance
Nitin Agarwal

-- 
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Unable to rsync files of size 30 GB

[Andy Henson]

Rsync maps files into memory, so it cannot handle files larger than the 
program's address space.  On 32 bit machines this is likely to be less 
than 2GB.

You don't say what processor you are running on?  Can it map files of 
bigger than 2GB into memory?

Andy Henson
-- 
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Adding support for versioned files in rsync

[Jason M. Felice]

Hi!

Below is a link to a proposal I'm writing for two clients of ours who want an
Internet-based backup solution.  I propose eleven "objectives" in it,
most of which are modifications to rsync.  I'd like to contribute
these changes back where possible, and so I'm posting this here for review.

The nuts and bolts of it is the ability to keep multiple copies of files
(think VAX file system, or maybe CVS) and the ability to restore from a
particular date or version.

Feel free to rip it to shreds.  I know there are a couple of minor
issues with it already.  I've poked around in the source to research
this, but if I'm missing any major sticking-points that someone knows
about off the top of their head, that's something I'd like to know, too.

All feedback is appreciated,
-Jay 'Eraserhead' Felice


 SafeSync Backup System

   Project Definition

   Prepared By: Jason M. Felice

   Date: October 11, 2003

Project overview

   Inline Technologies, Inc and Gem have approached Cronosys, LLC to discuss
   the implementation of an easy-to-use backup mechanism which would meet the
   following criteria:

1. Would keep multiple versions of backed-up files and allow several
   different versions to be restored (think of Netware's "salvage"
   command).

2. Would use as little bandwidth as possible.

3. Would back up to a central server.

4. Would run unattended and grab new revisions of files.

   Cronosys researched several possibilities for implementing this system.
   The bandwidth requirements suggest the rsync algorithm, so all
   possibilities investigated would use the rsync algorithm for transferring
   files. The methods investigated were:

 * Writing a new protocol which supported the additional features
   required.

   Some general designs were scratched out, the rsync source code was
   inspected for possible traps, and a general design of a new protocol
   was worked out. Evaluation of it indicated that this option would be
   fairly labor-intensive.

 * Using the rsync algorithm on top of another protocol.

   XML-RPC over HTTP was investigated as a possible transport for the
   system. Several fatal flaws were discovered with this. First, XML's
   language encoding prevents sending binary data directly, and XML-RPC's
   base64 encoding for binary data inflates the size of the data
   unacceptably. Second, file transfers to store to the server would have
   to be made in two requests (one to retreive the old file information
   and one to post the reconstruction instructions), which becomes more
   complicated since XML-RPC is essentially stateless. Third, this
   introduces a number of software dependencies which would have to be
   managed (web server, HTTP client library, XML-RPC library, SSL
   library).

 * Extending the existing rsync with these capabilities.

   This was decided as the best method for implementation. There are
   drawbacks, most notebly that rsync was not designed to handle multiple
   versions and there will be some difficulties in retrofitting the
   internal structures and the protocol to handle versions.

   On the other hand, this has several benefits:

  * The code base is already very well tested, and any code which is
integrated back into the project will be very well tested.

  * Contributing back to this project is inline with our corporate
philosophy.

  * The possibility of collaboration and support from developers
which require similar features.

   Project objectives

   The SafeSync Backup System project will meet the following objectives:

1. Modify the rsync client and server to support SSL.

   The modified client will use SSL for encryption of the protocol
   stream. Existing clients can use an external shell program such as SSH
   to provide encryption, but this is not portable and it is difficult to
   manage.

   An "--ssl" option will be added to the rsync program to enable this
   feature. This option will be accepted in both client and daemon mode.

2. Write a Windows backup service.

   This service will be a wrapper for the rsync program. It will read its
   configuration from the registry, then loop forever attempting to back
   up configured directories to the server then sleeping a configurable
   interval.

   The backup service will be aware of changes to its configuration and
   adjust its operation appropriately.

3. Write a configuration GUI for the Windows backup service.

   This will be a simple program which maintains the registry settings.
   It can be used to set the server, authentication information, and
   select which directories should be backed up to the server.

4. Mak

Re: Adding support for versioned files in rsync

[Andy Henson]

I agree version support should be added to rsync, indeed I was about to 
propose an extension, but I start from a very different point of view.

We already use versioned files.  For example, the rsync download is called 
rsync-2.5.6.tar.gz and the previous one is rsync-2.5.5.tar.gz.

I agree some systems may prefer a ;1 syntax for versioning, but rsync can 
(should?) treat this as just part of the file name, an underlying 
versioning file system can do the rest.

What we do need to do is provide a version of rsync which permits the 
following efficiently using the rsync protocol:

$ ls rsync*
rsync-2.5.5.tar.gz
$ rsync remote.host:rsync-2.5.6.tar.gz .

Now at present this will transfer the entire file, not just the 
differences.  Now I can write clever code to overcome this, indeed I do 
use a script with the following when transferring CD ISOs:

BASE=rsync-2.5.5.tar.gz
NEW=rsync-2.5.6.tar.gz
if [ ! -f $NEW ]; then
echo copying...
cp -l $BASE $NEW
elif [ `stat -t $NEW | cut -d' ' -f2` -lt 600334000 ]; then
echo appending...
cat $BASE >>$NEW
fi
rsync -vvtPB remote.host:$NEW .

But that's not very good or friendly - I'm creating a rogue copy of the 
old version under the new version's name which I hope will soon go.
If it partially copies and is restarted, say 300MB into the copy, it 
appends the old to the partial new file the second time around (if >600MB 
it might be all transferred ok so it doesn't alter it).

But this doesn't work if uploading to a machine which I have no shell 
access to.  It really should be in rsync itself.

I propose a new option:
--comp-file=

allowing:
rsync -vvtPB --comp-file=$BASE remote.host:$NEW .

Then the receiving rsync will generate checksums on *both* the old BASE 
file *and* the any partially transferred NEW file to match against the NEW 
file on the sending end.  Thus a 600MB ISO image which is 95% unchanged, 
and a 300MB partial transfer can be both used as match material for the 
retransfer.

I'm highly tempted to permit the default if not -comp-file is specified to 
be "look for a similarly named file with just a number changed", but 
perhaps we need an option for that too :)

I've had a read through the relevant source code and I am confident I 
could add this functionality.  I was waiting to see what others are doing 
before starting anything...

Andy Henson
-- 
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Unable to rsync files of size 30 GB

[jw schultz]

On Mon, Oct 13, 2003 at 02:34:00PM +0100, Andy Henson wrote:
> Rsync maps files into memory, so it cannot handle files larger than the 
> program's address space.  On 32 bit machines this is likely to be less 
> than 2GB.
> 
> You don't say what processor you are running on?  Can it map files of 
> bigger than 2GB into memory?

Not true.  It uses a rolling mapping that is actually done
by read not mmap for portability reasons.

The most likely cause of his problems is an incorrect build
but since the error messages, version number and
command-lines were not provided i have no way to know why he
is having this problem.

-- 

J.W. SchultzPegasystems Technologies
email address:  [EMAIL PROTECTED]

Remember Cernan and Schmitt
-- 
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Adding support for versioned files in rsync

[jw schultz]

On Mon, Oct 13, 2003 at 10:57:36AM -0400, Jason M. Felice wrote:
> Hi!
> 
> Below is a link to a proposal I'm writing for two clients of ours who want an
> Internet-based backup solution.  I propose eleven "objectives" in it,
> most of which are modifications to rsync.  I'd like to contribute
> these changes back where possible, and so I'm posting this here for review.
> 
> The nuts and bolts of it is the ability to keep multiple copies of files
> (think VAX file system, or maybe CVS) and the ability to restore from a
> particular date or version.
> 
> Feel free to rip it to shreds.  I know there are a couple of minor
> issues with it already.  I've poked around in the source to research
> this, but if I'm missing any major sticking-points that someone knows
> about off the top of their head, that's something I'd like to know, too.
> 
> All feedback is appreciated,
> -Jay 'Eraserhead' Felice
> 
> 
>  SafeSync Backup System
> 
>Project Definition
> 
>Prepared By: Jason M. Felice
> 
>Date: October 11, 2003
> 
> Project overview
> 
>Inline Technologies, Inc and Gem have approached Cronosys, LLC to discuss
>the implementation of an easy-to-use backup mechanism which would meet the
>following criteria:
> 
> 1. Would keep multiple versions of backed-up files and allow several
>different versions to be restored (think of Netware's "salvage"
>command).
> 
> 2. Would use as little bandwidth as possible.
> 
> 3. Would back up to a central server.
> 
> 4. Would run unattended and grab new revisions of files.

Try dirvish or one of the other backup systems already out
there.

>Cronosys researched several possibilities for implementing this system.
>The bandwidth requirements suggest the rsync algorithm, so all
>possibilities investigated would use the rsync algorithm for transferring
>files. The methods investigated were:
> 
>  * Writing a new protocol which supported the additional features
>required.
> 
>Some general designs were scratched out, the rsync source code was
>inspected for possible traps, and a general design of a new protocol
>was worked out. Evaluation of it indicated that this option would be
>fairly labor-intensive.

I don't say the rsync protocol is a good one but you don't
say why you need your own protocol.  Of course if you wish
to pay developers for the next three years...

>  * Using the rsync algorithm on top of another protocol.
> 
>XML-RPC over HTTP was investigated as a possible transport for the
>system. Several fatal flaws were discovered with this. First, XML's
>language encoding prevents sending binary data directly, and XML-RPC's
>base64 encoding for binary data inflates the size of the data
>unacceptably. Second, file transfers to store to the server would have
>to be made in two requests (one to retreive the old file information
>and one to post the reconstruction instructions), which becomes more
>complicated since XML-RPC is essentially stateless. Third, this
>introduces a number of software dependencies which would have to be
>managed (web server, HTTP client library, XML-RPC library, SSL
>library).
> 
>  * Extending the existing rsync with these capabilities.
> 
>This was decided as the best method for implementation. There are
>drawbacks, most notebly that rsync was not designed to handle multiple
>versions and there will be some difficulties in retrofitting the
>internal structures and the protocol to handle versions.
> 
>On the other hand, this has several benefits:
> 
>   * The code base is already very well tested, and any code which is
> integrated back into the project will be very well tested.
> 
>   * Contributing back to this project is inline with our corporate
> philosophy.

We will be appreciative of bug fixes and enhancements that
are of general value.  We do not care for enhancements just
with the sole value of supporting a commercial application.

>   * The possibility of collaboration and support from developers
> which require similar features.

s/possibility/requirement/  if you want the patches
accepted.

> 
>Project objectives
> 
>The SafeSync Backup System project will meet the following objectives:
> 
> 1. Modify the rsync client and server to support SSL.
> 
>The modified client will use SSL for encryption of the protocol
>stream. Existing clients can use an external shell program such as SSH
>to provide encryption, but this is not portable and it is difficult to
>manage.
> 
>An "--ssl" option will be added to the rsync program to enable this
>feature. This option will be accep

Re: Adding support for versioned files in rsync

[Jason M. Felice]

On Mon, Oct 13, 2003 at 10:44:52AM -0700, jw schultz wrote:
> Try dirvish or one of the other backup systems already out
> there.

dirvish looks interesting.  One of the requirements that I now realize I
didn't write into the proposal was the ability to store only a single
copy of duplicate files... duplicate as determined by file contents, not
naming or inode or anything.  (Rationale: will likely need to back up
large numbers of clients with lots of files in common.)

Dirvish does not appear to have that capability, but it *does* appear to
solve the right problem.  Extending Dirvish to make hard links from the
image into a directory tree based on SHA1 hash might work to implement
this.

[snip]
> Ever hear of vfs?  I have considered pluggability and might
> consider it for the next version of rsync (3.x or perhaps 2.6)

Sounds interesting.  I remember the VFS from mc, and there is now some
gnome-vfs stuff.  I'm not sure if you are referring to one of these or
something else.  It definitely sounds like something to investigate,
could you send a link?

[snip]
> Forget it.  Version awareness will just bog things down.
> They did it on VMS and it has mainly served to be a
> full-employment feature for admins.  If you want to use
> version aware filesystems fine, rsync doesn't need to be
> aware of that.  At most rsync might want to be able to
> detect renames or, perhaps through plugin, be able to select
> whether to see all versions with versioning disabled or just
> latest.

I'm pondering these ideas of plugins plus the ideas Andy put in his
post.  I might be able to make the system out of it with less
hacking-and-slashing in code.

[snip] 
> Forgive me if i have abused and mocked your proposal.

I invited it in the first paragraphs of my post 

> You have outlined an ambitious project.  Some of the ideas have
> merit.  One or two would be nice to see in rsync.  The rsync
> team are a few unpaid volunteers.  It sounds to me like you
> propose creating a monstrosity out of rsync for the benefit
> of one piece of vaporware.  Doing so on the backs of unpaid
> volunteers rankles as would hijacking rsync.

I'm not trying to force patches on you guys (not that I even could).
I basically have to solve a problem for a client and would like to do so
in a way that I can most benefit the rsync project (or another project
if rsync isn't the "right" project).  You seem to be unduly afraid that I
have some sort of power here, and really all I can do is take my marbles and
go home.  Well, perhaps I could spam you with awful patches, too ;-)

-- 
 Jason M. Felice
 Cronosys, LLC 
 216.221.4600 x302
-- 
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Adding support for versioned files in rsync

[Jason M. Felice]

On Mon, Oct 13, 2003 at 05:18:00PM +0100, Andy Henson wrote:
> I propose a new option:
> --comp-file=
> 
> allowing:
> rsync -vvtPB --comp-file=$BASE remote.host:$NEW .
> 

Specifying this for each file would be murder for my purposes, but I
like the concept.  I think there needs to be some sort of configurable
logic for determining a "fallback" file when one doesn't exist.  In my
case, it might be nice to have a "fallback directory" and if the file doesn't
exist here, check in the fallback directory.

That doesn't help with ISO images, though.

It would be nice if some generic method that solves all these problems
could be devised.  In my case, I would need to figure out the
highest-versioned file to use if the file we were syncing didn't already
exist.  I can't think of a simple way to make that work.

It will never help with rsync-2.5.6.tar.gz vs. rsync-2.5.5.tar.gz
anyway, since any small change early in the stream will cause the entire
file to be encoded differently (side affect of how compression
algorithms work).

-- 
 Jason M. Felice
 Cronosys, LLC 
 216.221.4600 x302
-- 
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Adding support for versioned files in rsync

[jw schultz]

On Mon, Oct 13, 2003 at 03:57:37PM -0400, Jason M. Felice wrote:
> On Mon, Oct 13, 2003 at 05:18:00PM +0100, Andy Henson wrote:
> > I propose a new option:
> > --comp-file=
> > 
> > allowing:
> > rsync -vvtPB --comp-file=$BASE remote.host:$NEW .
> > 
> 
> Specifying this for each file would be murder for my purposes, but I
> like the concept.  I think there needs to be some sort of configurable
> logic for determining a "fallback" file when one doesn't exist.  In my
> case, it might be nice to have a "fallback directory" and if the file doesn't
> exist here, check in the fallback directory.

See --compare-dest and --link-dest.

> That doesn't help with ISO images, though.
> 
> It would be nice if some generic method that solves all these problems
> could be devised.  In my case, I would need to figure out the
> highest-versioned file to use if the file we were syncing didn't already
> exist.  I can't think of a simple way to make that work.
> 
> It will never help with rsync-2.5.6.tar.gz vs. rsync-2.5.5.tar.gz
> anyway, since any small change early in the stream will cause the entire
> file to be encoded differently (side affect of how compression
> algorithms work).

See the rsyncable patch for gzip which can solve this
problem.


-- 

J.W. SchultzPegasystems Technologies
email address:  [EMAIL PROTECTED]

Remember Cernan and Schmitt
-- 
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Adding support for versioned files in rsync

[Ben Escoto]

On Mon, 13 Oct 2003 15:43:35 -0400
"Jason M. Felice" <[EMAIL PROTECTED]> wrote:
> dirvish looks interesting.  One of the requirements that I now realize I
> didn't write into the proposal was the ability to store only a single
> copy of duplicate files... duplicate as determined by file contents, not
> naming or inode or anything.  (Rationale: will likely need to back up
> large numbers of clients with lots of files in common.)

Check out Craig Barrett's BackupPC at

http://backuppc.sourceforge.net/

It has this feature and I have heard good things about it.  And since
I'm posting this I might as well mention my project at
http://rdiff-backup.stanford.edu.  It does versions and uses the rsync
algorithm, but doesn't notice duplicate files.


-- 
Ben Escoto


pgp0.pgp
Description: PGP signature
-- 
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Adding support for versioned files in rsync

[jw schultz]

On Mon, Oct 13, 2003 at 02:30:32PM -0700, Ben Escoto wrote:
> On Mon, 13 Oct 2003 15:43:35 -0400
> "Jason M. Felice" <[EMAIL PROTECTED]> wrote:
> > dirvish looks interesting.  One of the requirements that I now realize I
> > didn't write into the proposal was the ability to store only a single
> > copy of duplicate files... duplicate as determined by file contents, not
> > naming or inode or anything.  (Rationale: will likely need to back up
> > large numbers of clients with lots of files in common.)

A re-read of this paragraph highlighted the multiple-client
issue which dirvish does address rather well through branches.
Mail me off-list if you need specifics of how to set that
up.

The more general problem of coalescing duplicate files is a
real performance pig.  To do that what must be done is to
hash(checksum) every single file, store the files based on
that hash and then use a database to associate the paths
with the files with a separate handling of hardlinks.
Then while doing all of that you somehow want to be able to
use the rsync algorithm between versions of the same file.
Doing that is far outside the scope of rsync.

> 
> Check out Craig Barrett's BackupPC at
> 
> http://backuppc.sourceforge.net/
> 
> It has this feature and I have heard good things about it.  And since
> I'm posting this I might as well mention my project at
> http://rdiff-backup.stanford.edu.  It does versions and uses the rsync
> algorithm, but doesn't notice duplicate files.

I generally don't wish to appear to be pitching my own
project (dirvish) which is why i don't often provide a link
(google does that for me) and generally mention it in the
context of other tools.  But the other tools list is
starting to get a bit long as there are so many permutations
of the wheel.



-- 

J.W. SchultzPegasystems Technologies
email address:  [EMAIL PROTECTED]

Remember Cernan and Schmitt
-- 
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Token.c appears to have a bug.

[John E. Malmberg]

[A repost under a new topic since the other seems to have been lost in 
the noise]

I am basically down to one unresolved compilier diagnostic.

The HP/COMPAQ/DEC C compiler is concerned about this line in TOKEN.C

   4   22136   temp_byte = (char) n >> 8;
  1
   %CC-I-RIGHTSHIFTOVR, (1) In this statement, the right shift count
   "8" is greater than or equal to the size of the unpromoted operand
"(char)n".
If I am interpreting this right, the value n is an integer, and is being 
cast to be a 8 bit size.  And this results in all the higher bits being 
discarded.  So the >> should shift out all 8 remaining significant bits.

Since this is a signed value, it appears to leave what should be the 
high byte of a word either set to zero, or with possibly all bits set to 
the value of the sign bit for the character that resulted from the cast.

Is this really what is intended?  Or should there be parenthesis around 
(n >> 8) to make sure that it happens before the most significant part 
of "n" is discarded?

The actual assembly code generated is:

   LDL R10, n   ; R10, 16(FP)
   SLL R10, 56, R10
   SRA R10, 63, R10
   STQ R10, temp_byte   ; R10, 8(FP)
-John
[EMAIL PROTECTED]
Personal Opinion Only
--
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Token.c appears to have a bug.

[Wayne Davison]

On Mon, Oct 13, 2003 at 07:29:24PM -0400, John E. Malmberg wrote:
> Or should there be parenthesis around (n >> 8) to make sure that it
> happens before the most significant part of "n" is discarded?

Yes, there should be parens there.  Note that this bug only affects
batch mode (a fairly rare option).  Thanks for pointing this out.
I'll check in a fix in a moment.

..wayne..
-- 
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Token.c appears to have a bug.

[jw schultz]

On Mon, Oct 13, 2003 at 04:43:45PM -0700, Wayne Davison wrote:
> On Mon, Oct 13, 2003 at 07:29:24PM -0400, John E. Malmberg wrote:
> > Or should there be parenthesis around (n >> 8) to make sure that it
> > happens before the most significant part of "n" is discarded?
> 
> Yes, there should be parens there.  Note that this bug only affects
> batch mode (a fairly rare option).  Thanks for pointing this out.
> I'll check in a fix in a moment.

Having looked at that bit of code now.  I am a bit concerned
about the use of shift operators on signed integers here.
Habit also makes me shy of relying on char being 8 bits but
that is probably overcautious.



Index: token.c
===
RCS file: /data/cvs/rsync/token.c,v
retrieving revision 1.27
diff -u -r1.27 token.c
--- token.c 11 Sep 2003 04:48:09 -  1.27
+++ token.c 13 Oct 2003 23:47:50 -
@@ -160,7 +160,7 @@
int n, r;
static int init_done, flush_pending;
extern int write_batch;
-   char temp_byte;
+   unsigned char temp_byte;
 
if (last_token == -1) {
/* initialization */
@@ -211,7 +211,7 @@
write_byte(f, n >> 8);
if (write_batch) {
write_batch_delta_file((char *)&n,sizeof(char));
-   temp_byte = (char) n >> 8;
+   temp_byte = (unsigned char) (n >> 8 & 0xFF);
write_batch_delta_file(&temp_byte,sizeof(temp_byte));
}
}

-- 

J.W. SchultzPegasystems Technologies
email address:  [EMAIL PROTECTED]

Remember Cernan and Schmitt
-- 
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Token.c appears to have a bug.

[Wayne Davison]

On Mon, Oct 13, 2003 at 04:54:20PM -0700, jw schultz wrote:
> Having looked at that bit of code now.  I am a bit concerned about the
> use of shift operators on signed integers here.

I don't see the problem with regard to this code because we're only
looking at bits that where known to exist in the var before the shift
(i.e. no sign-extended bits have any effect).  Also, your proposed
change doesn't affect the shifting at all -- you'd have to cast the "n"
to unsigned before shifting it, not cast the result to unsigned after
shifting it.  Since the code we're calling is expecting a "char *", I
think we should leave the type of temp_byte unchanged.

> Habit also makes me shy of relying on char being 8 bits but that is
> probably overcautious.

I think a lot of things would fail in the code if "char" wasn't 8 bits.

..wayne..
-- 
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: rsync on OpenVMS

[Donovan Baarda]

On Mon, 2003-10-13 at 13:00, John E. Malmberg wrote:
> jw schultz wrote:
> > On Sun, Oct 12, 2003 at 12:38:40AM -0400, John E. Malmberg wrote:
[...]
> > I do not know but if OpenVMS support is a problem for rsync
> > proper you might wish to look at pysync or unison which
> > might meet your immediate needs.
> 
> I have not heard of unison.  I have heard that pysync was successful in 
> a limited test on OpenVMS.  As near as I can tell though the librsync it 

cool... someone playing with pysync :-)

> is based on is a bit out of date.

pysync includes a pure-python implementation that will run on anything
that can run python, as well as a basic wrapper for librsync. It also
includes extension modules for md4sum and rollsum that speed it up a
fair bit.

Something possibly worth trying on it is psyco... it compiles python to
native code on the fly using a simple "import psyco". Pure python is a
bit slow compared to native C implementations, but psyco could help
close the gap a bit.

librsync is still under active development by myself and others, and
recently had a new release. 

I haven't touched pysync for a while, but it should still work with the
latest librsync as the API hasn't changed. If there are any problems,
please let me know. I believe rdiff-backup also has a python wrapper for
librsync that might be more advanced than the one in pysync.

I have plans for both pysync and librsync, but I haven't worked on them
much lately. I find I am mostly motivated by feedback from others when
funding is not available :-)

-- 
Donovan Baarda <[EMAIL PROTECTED]>
http://minkirri.apana.org.au/~abo/

-- 
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Token.c appears to have a bug.

[jw schultz]

On Mon, Oct 13, 2003 at 05:41:56PM -0700, Wayne Davison wrote:
> On Mon, Oct 13, 2003 at 04:54:20PM -0700, jw schultz wrote:
> > Having looked at that bit of code now.  I am a bit concerned about the
> > use of shift operators on signed integers here.
> 
> I don't see the problem with regard to this code because we're only
> looking at bits that where known to exist in the var before the shift
> (i.e. no sign-extended bits have any effect).  Also, your proposed
> change doesn't affect the shifting at all -- you'd have to cast the "n"
> to unsigned before shifting it, not cast the result to unsigned after
> shifting it.  Since the code we're calling is expecting a "char *", I
> think we should leave the type of temp_byte unchanged.

Sorry, the full extent of the signedness (in particular n)
didn't hit me until after i had already created the
half-a**ed patch.  I've little doubt that the sign-extension
won't come into play simply because we never touch the bits
that would have resulted.  "concerned" is perhaps an
overstatement but i'm not sure quite what term to use,
discomfited perhaps.  The fact that it works is primary.

> > Habit also makes me shy of relying on char being 8 bits but that is
> > probably overcautious.
> 
> I think a lot of things would fail in the code if "char" wasn't 8 bits.

Too true.  I'm not aware of any modern platforms where char
isn't 8 bits.  But i do recall one or two old architectures
where char was 12 or 9 bits and nothing says it cannot be 16
bits or even larger.


-- 

J.W. SchultzPegasystems Technologies
email address:  [EMAIL PROTECTED]

Remember Cernan and Schmitt
-- 
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: rsync on OpenVMS

[Donovan Baarda]

On Mon, 2003-10-13 at 13:00, John E. Malmberg wrote:
> jw schultz wrote:
> > On Sun, Oct 12, 2003 at 12:38:40AM -0400, John E. Malmberg wrote:
[...]
> > I do not know but if OpenVMS support is a problem for rsync
> > proper you might wish to look at pysync or unison which
> > might meet your immediate needs.
> 
> I have not heard of unison.  I have heard that pysync was successful in 
> a limited test on OpenVMS.  As near as I can tell though the librsync it 

cool... someone playing with pysync :-)

> is based on is a bit out of date.

pysync includes a pure-python implementation that will run on anything
that can run python, as well as a basic wrapper for librsync. It also
includes extension modules for md4sum and rollsum that speed it up a
fair bit.

Something possibly worth trying on it is psyco... it compiles python to
native code on the fly using a simple "import psyco". Pure python is a
bit slow compared to native C implementations, but psyco could help
close the gap a bit.

librsync is still under active development by myself and others, and
recently had a new release. 

I haven't touched pysync for a while, but it should still work with the
latest librsync as the API hasn't changed. If there are any problems,
please let me know. I believe rdiff-backup also has a python wrapper for
librsync that might be more advanced than the one in pysync.

I have plans for both pysync and librsync, but I haven't worked on them
much lately. I find I am mostly motivated by feedback from others when
funding is not available :-)

-- 
Donovan Baarda <[EMAIL PROTECTED]>
http://minkirri.apana.org.au/~abo/

-- 
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Adding support for versioned files in rsync

[Christopher Vance]

On Mon, Oct 13, 2003 at 01:20:48PM -0700, jw schultz wrote:
It will never help with rsync-2.5.6.tar.gz vs. rsync-2.5.5.tar.gz
anyway, since any small change early in the stream will cause the entire
file to be encoded differently (side affect of how compression
algorithms work).
See the rsyncable patch for gzip which can solve this
problem.
Do you know at which version this was included in the gzip release?

--
Christopher Vance
--
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Adding support for versioned files in rsync

[jw schultz]

On Tue, Oct 14, 2003 at 11:31:36AM +1000, Christopher Vance wrote:
> On Mon, Oct 13, 2003 at 01:20:48PM -0700, jw schultz wrote:
> >>It will never help with rsync-2.5.6.tar.gz vs. rsync-2.5.5.tar.gz
> >>anyway, since any small change early in the stream will cause the entire
> >>file to be encoded differently (side affect of how compression
> >>algorithms work).
> >
> >See the rsyncable patch for gzip which can solve this
> >problem.
> 
> Do you know at which version this was included in the gzip release?

It isn't yet.  The debian version of gzip might have the
patch applied because it is an issue for the debian mirror
sites but i don't know.  I believe the gzip developers are
considering applying the patch.

Please note, there is no such patch for gtar so making
.tar.gz files rsyncable will require doing the gzip as a
seperate step.


-- 

J.W. SchultzPegasystems Technologies
email address:  [EMAIL PROTECTED]

Remember Cernan and Schmitt
-- 
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Adding support for versioned files in rsync

[Christopher Vance]

On Mon, Oct 13, 2003 at 06:54:02PM -0700, jw schultz wrote:
It isn't yet.  The debian version of gzip might have the
patch applied because it is an issue for the debian mirror
sites but i don't know.  I believe the gzip developers are
considering applying the patch.
Oh.  That makes it mostly not-yet-useful for those most of us who
don't control the public rsync servers we'd like to get stuff from.
:-(
--
Christopher Vance
--
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Adding support for versioned files in rsync

[jw schultz]

On Tue, Oct 14, 2003 at 12:03:28PM +1000, Christopher Vance wrote:
> On Mon, Oct 13, 2003 at 06:54:02PM -0700, jw schultz wrote:
> >It isn't yet.  The debian version of gzip might have the
> >patch applied because it is an issue for the debian mirror
> >sites but i don't know.  I believe the gzip developers are
> >considering applying the patch.
> 
> Oh.  That makes it mostly not-yet-useful for those most of us who
> don't control the public rsync servers we'd like to get stuff from.
> :-(

And of high import to those who do control those public
rsync servers.

-- 

J.W. SchultzPegasystems Technologies
email address:  [EMAIL PROTECTED]

Remember Cernan and Schmitt
-- 
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Adding support for versioned files in rsync

[jw schultz]

On Mon, Oct 13, 2003 at 03:43:35PM -0400, Jason M. Felice wrote:
> On Mon, Oct 13, 2003 at 10:44:52AM -0700, jw schultz wrote:
> > Try dirvish or one of the other backup systems already out
> > there.
> 
> dirvish looks interesting.  One of the requirements that I now realize I
> didn't write into the proposal was the ability to store only a single
> copy of duplicate files... duplicate as determined by file contents, not
> naming or inode or anything.  (Rationale: will likely need to back up
> large numbers of clients with lots of files in common.)
> 
> Dirvish does not appear to have that capability, but it *does* appear to
> solve the right problem.  Extending Dirvish to make hard links from the
> image into a directory tree based on SHA1 hash might work to implement
> this.

That would be better accomplished by adding an extended
rename detection to rsync.  I say extended because a normal
rename detection would depend on leveraging deletes.

The question that has t be addressed is whether or not the
added complexity is worthwhile for the space and bandwidth
efficiency.  I'm not yet convinced although directory
rename which is quite different from file-rename might be
worth it if we could reliably detect it.

> [snip]
> > Ever hear of vfs?  I have considered pluggability and might
> > consider it for the next version of rsync (3.x or perhaps 2.6)
> 
> Sounds interesting.  I remember the VFS from mc, and there is now some
> gnome-vfs stuff.  I'm not sure if you are referring to one of these or
> something else.  It definitely sounds like something to investigate,
> could you send a link?

I don't have any links but the vfs i'm referring to is the
adding of a layer between the physical filesystem and the
user-mode system calls.  This is often used for compression
and encryption and would be a good candidate for versioning.
One of the things this allows is different mount points
having different views on the same underlying filesystem
Historically this has also been done using NFS.

> 
> [snip]
> > Forget it.  Version awareness will just bog things down.
> > They did it on VMS and it has mainly served to be a
> > full-employment feature for admins.  If you want to use
> > version aware filesystems fine, rsync doesn't need to be
> > aware of that.  At most rsync might want to be able to
> > detect renames or, perhaps through plugin, be able to select
> > whether to see all versions with versioning disabled or just
> > latest.
> 
> I'm pondering these ideas of plugins plus the ideas Andy put in his
> post.  I might be able to make the system out of it with less
> hacking-and-slashing in code.

Because of how rsync does file i/o the plugin approach has
at least a chance but it may have some reliability and
portability issues.  A vfs approach may actually be better
suited for your purposes.

> 
> [snip] 
[snip]
> 
> > You have outlined an ambitious project.  Some of the ideas have
> > merit.  One or two would be nice to see in rsync.  The rsync
> > team are a few unpaid volunteers.  It sounds to me like you
> > propose creating a monstrosity out of rsync for the benefit
> > of one piece of vaporware.  Doing so on the backs of unpaid
> > volunteers rankles as would hijacking rsync.
> 
> I'm not trying to force patches on you guys (not that I even could).
> I basically have to solve a problem for a client and would like to do so
> in a way that I can most benefit the rsync project (or another project
> if rsync isn't the "right" project).  You seem to be unduly afraid that I
> have some sort of power here, and really all I can do is take my marbles and
> go home.  Well, perhaps I could spam you with awful patches, too ;-)

That differs somewhat from my impression of your
proposal/presentation.  My error?  It sounded like you were
aiming to create a shrink-wrap application and were
proposing to do so dependant on significant functional
changes to rsync that i recognised would alter its codebase
structuraly.

As you rightly recognise, you couldn't force patches on us.
You could fork the project but that has its own drawbacks,
political as well as practical.  I merely was trying to
point out that we have our on agenda(s) which do not align
well with some of the specific proposals and (rather
bluntly) remind you that it would be good to respect the
volunteer nature of the project.  That isn't to say that
some commercial involvement would not be accepted, only that
there are ways that would be welcomed more than others.

-- 

J.W. SchultzPegasystems Technologies
email address:  [EMAIL PROTECTED]

Remember Cernan and Schmitt
-- 
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html