Jeff,
I've pushed fixes to backuppc-xs, rsync-bpc 3.0.9 and rsync-bpc 3.1.2 for
making mtime signed and correctly writing and reading negative mtimes.
That should fix things for v4. However, I haven't released new versions
yet, since I haven't decided whether (or how) to deal with the issue in v3.
Craig
On Wed, Apr 10, 2019 at 8:49 PM <backu...@kosowsky.org> wrote:
> The following patch seems to fix bpc_attrib.c
>
> -- libbackuppc-xs-perl-0.59.orig/bpc_attrib.c
> +++ libbackuppc-xs-perl-0.59/bpc_attrib.c
> @@ -567,10 +567,7 @@ static void setVarInt(uchar **bufPP, uch
> {
> uchar *bufP = *bufPP;
>
> - if ( value < 0 ) {
> - bpc_logErrf("setVarInt botch: got negative argument %ld;
> - setting to 0\n", (long int)value);
> - value = 0;
> - }
> + value = (unsigned) value;
> do {
> uchar c = value & 0x7f;
>
> NOTE: if you don't convert to 'unsigned', then the setVarInt routine
> hangs. This is probably why you originally tested for a negative
> number.
>
>
> With this patch the new attrib file stores the negative mtime
> similarly as an unsigned 32 bit integer.
>
> Original v3 attrib entry
> 'MacWrite 5.0Scrap' => {
> 'uid' => 501,
> 'mode' => 33188,
> 'sizeDiv4GB' => 0,
> 'size' => 0,
> 'type' => 0,
> 'gid' => 501,
> 'sizeMod4GB' => 0,
> 'mtime' => 2212140496
> },
>
> v4 attrib entry after migrating v3 to v4:
> 'MacWrite 5.0Scrap' => {
> 'compress' => 3,
> 'digest' => 'd41d8cd98f00b204e9800998ecf8427e',
> 'gid' => 501,
> 'inode' => 12458250,
> 'mode' => 33188,
> 'mtime' => 2212140496,
> 'name' => 'MacWrite 5.0Scrap',
> 'nlinks' => 0,
> 'size' => 0,
> 'type' => 0,
> 'uid' => 501
> }, },
>
> Now, all that needs to be done is to change the attrib read functions
> for v3 and v4 to read back a signed integer from the packed unsigned
> integer that is stored.
>
>
> backu...@kosowsky.org wrote at about 14:00:48 -0400 on Wednesday, April
> 10, 2019:
> > Thanks Craig.
> > I think the bugs affect multiple routines:
> > 1. BackupPC_attribPrint print the mtime of v3 stored attribs as
> > unsigned integers so that the 2's complement is displayed for
> > negative mtimes (32-bit for v3, 64-bit for v4)
> > i.e this bug affects both v3 and v4
> >
> > 2. BackupPC_migrateV3toV4, sets the mtime to 0 for negative mtime
> > files
> >
> > 3. For V3, BackupPC_tarCreate (and presumably BackupPC_restore) sets
> the mtime
> > of restored files to the unsigned value. Meaning that originally
> negative
> > mtime's are set to the future!
> >
> > Interestingly, this bug is thus both a v3 and v4 bug.
> > Surprising perhaps that it was never caught in v3 - i.e. that nobody
> ever had
> > a negative mtime file (especially given that both macs and windows
> > have an epoch before 1/1/1970)
> >
> > Jeff
> >
> > Craig Barratt wrote at about 23:30:34 -0700 on Tuesday, April 9, 2019:
> > > Jeff,
> > >
> > > Thanks for the detective work and proposed fix. Yes, I assumed
> mtime was
> > > unsigned, when in fact it might not be.
> > >
> > > Let me check into the various scenarios of v3/v4 read/write.
> Hopefully
> > > I'll have time to push fixes this weekend.
> > >
> > > Craig
> > >
> > > On Mon, Apr 8, 2019 at 7:38 PM <backu...@kosowsky.org> wrote:
> > >
> > > > So, the problem seems to be that backuppc treats mtime as an
> unsigned
> > > > integer -- meaning that any file created before Jan 1, 1970 is not
> > > > representable.
> > > >
> > > > v3 represents treats the number as an unsigned integer and
> basically
> > > > shows it's 2's complement (in 32 bits) - i.e. it's packed as an
> > > > unsigned 32 bit integer ('N' type in Perl)
> > > >
> > > > v4 assumes a negative number is an error, sets it to zero, then
> > > > defeating the purpose displays it as the 2's complement of 0! v4
> > > > stores the number as a variable length unsigned integer.
> > > >
> > > > According to Wikipedia,
> > > > The Unix time_t data type that represents a point in
> time is, on
> > > > many
> > > > platforms, a signed integer, traditionally of 32 bits
> (but see
> > > > below),
> > > > directly encoding the Unix time number as described in
> the
> > > > preceding
> > > > section. Being 32 bits means that it covers a range of
> about 136
> > > > years
> > > > in total. The minimum representable date is Friday
> 1901-12-13,
> > > > and the
> > > > maximum representable date is Tuesday 2038-01-19. One
> second
> > > > after
> > > > 03:14:07 UTC 2038-01-19 this representation will
> overflow. This
> > > > milestone is anticipated with a mixture of amusement and
> > > > dread—see
> > > > year 2038 problem.
> > > >
> > > > In some newer operating systems, time_t has been widened
> to 64
> > > > bits. This expands the times representable by
> approximately 293
> > > > billion years in both directions, which is over twenty
> times the
> > > > present age of the universe per direction.
> > > >
> > > > There was originally some controversy over whether the
> Unix
> > > > time_t
> > > > should be signed or unsigned. If unsigned, its range in
> the
> > > > future
> > > > would be doubled, postponing the 32-bit overflow (by 68
> > > > years). However, it would then be incapable of
> representing times
> > > > prior to the epoch. The consensus is for time_t to be
> signed,
> > > > and this
> > > > is the usual practice.
> > > >
> > > > Seems to me that would be best to continue to represent time as a
> 32 bit
> > > > unsigned
> > > > integer in v3 attrib files, represented using 2's complements, so
> that
> > > > any mtime > 2^31 (about 68 years) would be considered as a negative
> > > > time -- this seems to be the current implied convention.
> > > >
> > > > In v4, time would be written as 64 bit unsigned integer using
> > > > SetVarInt where any mtime > 2^63 (about 292 billion years from now)
> > > > would be considered a negative number written in 2's complements
> > > >
> > > > Specifically, rather than testing for the sign of 'value' in
> > > > setVarInt, it might be better just to cast value to an unsigned
> 64-bit
> > > > integer that is then variable-length encoded as is done currently.
> > > >
> > > > I am rusty on my 'c' but maybe just:
> > > >
> > > > static void setVarInt(uchar **bufPP, uchar *bufEnd, int64 value)
> > > > {
> > > > + unsigned int64 uvalue = value;
> > > > uchar *bufP = *bufPP;
> > > >
> > > > - if ( value < 0 ) {
> > > > - bpc_logErrf("setVarInt botch: got negative argument %ld;
> setting
> > > > to 0\n", (long int)value);
> > > > - value = 0;
> > > > - }
> > > > do {
> > > > - uchar c = value & 0x7f;
> > > > + uchar c = uvalue & 0x7f;
> > > > value >>= 7;
> > > > + uvalue >>= 7;
> > > > - if ( value ) c |= 0x80;
> > > > + if ( uvalue ) c |= 0x80;
> > > > if ( bufP < bufEnd ) {
> > > > *bufP++ = c;
> > > > } else {
> > > > bufP++;
> > > > }
> > > > - } while ( value );
> > > > + } while ( uvalue );
> > > > *bufPP = bufP;
> > > > }
> > > >
> > > >
> > > > Both v3 and V4 versions of BackupPC_attribPrint then should
> > > > additionally be "fixed" to print mtime as a signed integer
> converting
> > > > from a 32 or 64 bit unsigned integer depending on whether the
> attrib
> > > > file was created with v3 or v4.
> > > >
> > > > backu...@kosowsky.org wrote at about 21:56:08 -0400 on Monday,
> April 8,
> > > > 2019:
> > > > > I should have said that the v3 case is 2's complement for 32 bit
> > > > > integers while the v4 case is 2's complement for 64 bit
> integers.
> > > > >
> > > > > Also, the internal representation is clearly right in that both
> v3 and
> > > > > v4 are retrieving the right number, just displaying it
> incorrectly.
> > > > >
> > > > > - v3 is showing it as an unsigned rather than signed integer.
> > > > > - v4 is realizing it is an unsigned integer but rejecting it as
> negative
> > > > > and then erroneously setting it to zero (and topping on the
> cake
> > > > > recording it as the 64 bit 2's complement of zero)
> > > > >
> > > > > backu...@kosowsky.org wrote at about 21:51:27 -0400 on Monday,
> April
> > > > 8, 2019:
> > > > > > OK, figured it out... VERY INTERESTING.
> > > > > >
> > > > > > The original file has an mtime of -2082826800 which
> corresponds to
> > > > 5AM
> > > > > > Friday, Jan 1, 1904.
> > > > > > The the error message in question is actually *right* and
> not an
> > > > > > error:
> > > > > > setVarInt botch: got negative argument -2082826800;
> setting to 0
> > > > > >
> > > > > > i.e. it truly is a negative mtime.
> > > > > >
> > > > > > The reason for this odd time is that the file was created on
> a Mac128
> > > > > > (circa 1985) and apparently, on old Macs (pre-PowerPC), time
> starts
> > > > with midnight Jan
> > > > > > 1, 1904 and runs until 2040. Now 5AM GMT is actually 12AM
> EST (my
> > > > home
> > > > > > timezone), so the file shows a 'default' old Mac start date
> of
> > > > > > midnight Jan 1, 1904.
> > > > > >
> > > > > > Now for the interesting part.
> > > > > > Both the v3 and v4 versions of BackupPC_attribPrint are
> actually
> > > > wrong! 2 (different) bugs!!!!
> > > > > >
> > > > > > 1. v3 displays a positive mtime of 2212140496 which is is
> right only
> > > > > > in the sense that it is the decimal equivalent of the 2's
> > > > > > complement representation of -2082826800
> > > > > >
> > > > > > 2. v4 properly catches that the mtime is negative, but
> signals an
> > > > > > error rather than writing it. it then proceeds however to
> record a
> > > > > > wrong (and very large) mtime of 18446744071626724816,
> rather than
> > > > > > 0. The number however is 2^64 which is the 2's complement
> of 0 for
> > > > > > 8 byte integers
> > > > > >
> > > > > >
> > > > > > Fascinating. I learned a lot from nailing this one down.
> > > > > > Now both the v3 and v4 bugs should presumably be corrected :)
> > > > > >
> > > > > > "" wrote at about 20:23:18 -0400 on Monday, April 8, 2019:
> > > > > > > "" wrote at about 20:13:33 -0400 on Monday, April 8, 2019:
> > > > > > >
> > > > > > > Once again seems to be a problem with bpc_attrib.c:
> > > > > > > BackupPC_attribPrint
> > > > > > >
> > > > > > > Using v3:
> > > > > > > 'MacWrite 5.0Scrap' => {
> > > > > > > 'uid' => 501,
> > > > > > > 'mode' => 33188,
> > > > > > > 'sizeDiv4GB' => 0,
> > > > > > > 'size' => 0,
> > > > > > > 'type' => 0,
> > > > > > > 'gid' => 501,
> > > > > > > 'sizeMod4GB' => 0,
> > > > > > > 'mtime' => 2212140496
> > > > > > > },
> > > > > > >
> > > > > > > Using v4:
> > > > > > >
> > > > > > > 'MacWrite 5.0Scrap' => {
> > > > > > > 'compress' => 3,
> > > > > > > 'digest' => '',
> > > > > > > 'gid' => 501,
> > > > > > > 'inode' => 0,
> > > > > > > 'mode' => 33188,
> > > > > > > 'mtime' => '18446744071626724816',
> > > > > > > 'name' => 'MacWrite 5.0Scrap',
> > > > > > > 'nlinks' => 0,
> > > > > > > 'size' => 0,
> > > > > > > 'type' => 0,
> > > > > > > 'uid' => 501
> > > > > > > },
> > > > > > >
> > > > > > > Clearly, v4 is messing up on reading 'mtime' which
> presumably is
> > > > what
> > > > > > > is causing the problem, consistent with the verbose
> output of
> > > > > > > BackupPC_migrateV3toV4 (as more fully excertped below)
> > > > > > >
> > > > > > > > setVarInt botch: got negative argument -2082826800;
> setting to
> > > > 0
> > > > > > > > Wrote file MacWrite 5.0Scrap: type = 0, mode = 0100644,
> > > > uid/gid = 501/501, size = 0, inode = 11616435, nlinks = 0, digest =
> > > > 0xd41d8c..., bufUsed = 925
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > >
> > > > > > > > Craig Barratt wrote at about 23:39:50 -0700 on Sunday,
> April
> > > > 7, 2019:
> > > > > > > > > Jeff,
> > > > > > > > >
> > > > > > > > > Please send me an attrib file that triggers that
> error.
> > > > > > > > >
> > > > > > > > > Craig
> > > > > > > > >
> > > > > > > >
> > > > > > > > Here is the relevant verbose output from
> BackupPc_migrateV3toV4
> > > > > > > > starting with the attrib file:
> > > > > > > >
> > > > > > > >
> bpc_poolRefDeltaUpdate(5a3631521955e47e179ea20d8a84ac5a, 1),
> > > > count now 1
> > > > > > > >
> > > >
> bpc_attrib_dirWrite(/var/lib/backuppc/pc/consult/13.v4/./f%2f/fvar/fopt/fexecutor/fshare/fhome/fSystem
> > > > Folder/attrib): dirPath =
> > > >
> /var/lib/backuppc/pc/consult/13.v4/./f%2f/fvar/fopt/fexecutor/fshare/fhome/fSystem
> > > > Folder, attribFileName = attrib, baseAttribFileName = attrib
> > > > > > > >
> > > >
> bpc_path_create(/var/lib/backuppc/pc/consult/13.v4/./f%2f/fvar/fopt/fexecutor/fshare/fhome/fSystem
> > > > Folder)
> > > > > > > > Wrote file Extensions: type = 5, mode = 040771,
> uid/gid =
> > > > 0/1002, size = 4096, inode = 11616442, nlinks = 0, digest =
> 0x000000...,
> > > > bufUsed = 4
> > > > > > > > Wrote file %mac.rsrc: type = 0, mode = 0100660,
> uid/gid =
> > > > 0/1002, size = 6753, inode = 11616437, nlinks = 0, digest =
> 0xc874b4...,
> > > > bufUsed = 37
> > > > > > > > Wrote file system.ard: type = 0, mode = 0100660,
> uid/gid =
> > > > 0/1002, size = 0, inode = 11616455, nlinks = 0, digest =
> 0xd41d8c...,
> > > > bufUsed = 85
> > > > > > > > Wrote file %MacWrite 5.0Scrap: type = 0, mode =
> 0100644,
> > > > uid/gid = 501/501, size = 198, inode = 11616432, nlinks = 0,
> digest =
> > > > 0x85e52a..., bufUsed = 133
> > > > > > > > Wrote file Excel Toolbars: type = 0, mode = 0100644,
> uid/gid =
> > > > 501/501, size = 267, inode = 11616447, nlinks = 0, digest =
> 0x619077...,
> > > > bufUsed = 191
> > > > > > > > Wrote file %ParamRAM: type = 0, mode = 0100660,
> uid/gid =
> > > > 0/1002, size = 198, inode = 11616457, nlinks = 0, digest =
> 0x5a3631...,
> > > > bufUsed = 245
> > > > > > > > Wrote file Printer: type = 0, mode = 0100640, uid/gid =
> > > > 0/1002, size = 0, inode = 11616441, nlinks = 0, digest =
> 0xd41d8c...,
> > > > bufUsed = 293
> > > > > > > > Wrote file %system.ard: type = 0, mode = 0100660,
> uid/gid =
> > > > 0/1002, size = 798, inode = 11616445, nlinks = 0, digest =
> 0xdcd6d0...,
> > > > bufUsed = 338
> > > > > > > > Wrote file %System: type = 0, mode = 0100660, uid/gid =
> > > > 0/1002, size = 487415, inode = 11616434, nlinks = 0, digest =
> 0xa9c2a3...,
> > > > bufUsed = 388
> > > > > > > > Wrote file System: type = 0, mode = 0100660, uid/gid =
> 0/1002,
> > > > size = 0, inode = 11616433, nlinks = 0, digest = 0xd41d8c...,
> bufUsed = 435
> > > > > > > > Wrote file Browser: type = 0, mode = 0100640, uid/gid =
> > > > 0/1002, size = 0, inode = 11616450, nlinks = 0, digest =
> 0xd41d8c...,
> > > > bufUsed = 479
> > > > > > > > Wrote file ParamRAM: type = 0, mode = 0100660, uid/gid
> =
> > > > 0/1002, size = 20, inode = 11616443, nlinks = 0, digest =
> 0xcef2a5...,
> > > > bufUsed = 524
> > > > > > > > Wrote file %Printer: type = 0, mode = 0100640, uid/gid
> =
> > > > 0/1002, size = 2279, inode = 11616438, nlinks = 0, digest =
> 0x07c53e...,
> > > > bufUsed = 570
> > > > > > > > Wrote file %Browser: type = 0, mode = 0100640, uid/gid
> =
> > > > 0/1002, size = 119268, inode = 11616448, nlinks = 0, digest =
> 0x9fb0e6...,
> > > > bufUsed = 617
> > > > > > > > Wrote file %windows.rsrc: type = 0, mode = 0100660,
> uid/gid =
> > > > 0/1002, size = 15917, inode = 11616451, nlinks = 0, digest =
> 0xd30945...,
> > > > bufUsed = 665
> > > > > > > > Wrote file %Excel Toolbars: type = 0, mode = 0100644,
> uid/gid
> > > > = 501/501, size = 198, inode = 11616439, nlinks = 0, digest =
> 0x7c1511...,
> > > > bufUsed = 717
> > > > > > > > Wrote file windows.rsrc: type = 0, mode = 0100660,
> uid/gid =
> > > > 0/1002, size = 0, inode = 11616453, nlinks = 0, digest =
> 0xd41d8c...,
> > > > bufUsed = 772
> > > > > > > > Wrote file mac.rsrc: type = 0, mode = 0100660, uid/gid
> =
> > > > 0/1002, size = 0, inode = 11616440, nlinks = 0, digest =
> 0xd41d8c...,
> > > > bufUsed = 822
> > > > > > > > Wrote file Excel Settings (4): type = 0, mode =
> 0100644,
> > > > uid/gid = 501/501, size = 0, inode = 11616436, nlinks = 0, digest =
> > > > 0xd41d8c..., bufUsed = 868
> > > > > > > > setVarInt botch: got negative argument -2082826800;
> setting to
> > > > 0
> > > > > > > > Wrote file MacWrite 5.0Scrap: type = 0, mode = 0100644,
> > > > uid/gid = 501/501, size = 0, inode = 11616435, nlinks = 0, digest =
> > > > 0xd41d8c..., bufUsed = 925
> > > > > > > > Wrote file %godata.sav: type = 0, mode = 0100644,
> uid/gid =
> > > > 501/501, size = 198, inode = 11616452, nlinks = 0, digest =
> 0x39c291...,
> > > > bufUsed = 977
> > > > > > > > Wrote file %Preferences: type = 0, mode = 0100660,
> uid/gid =
> > > > 0/1002, size = 198, inode = 11616454, nlinks = 0, digest =
> 0x087760...,
> > > > bufUsed = 1028
> > > > > > > > Wrote file godata.sav: type = 0, mode = 0100644,
> uid/gid =
> > > > 501/501, size = 1100, inode = 11616456, nlinks = 0, digest =
> 0xb247c3...,
> > > > bufUsed = 1079
> > > > > > > > Wrote file MacWrite 5.0Undo: type = 0, mode = 0100644,
> uid/gid
> > > > = 501/501, size = 32, inode = 11616446, nlinks = 0, digest =
> 0x70bc8f...,
> > > > bufUsed = 1129
> > > > > > > > Wrote file %MacWrite 5.0Undo: type = 0, mode = 0100644,
> > > > uid/gid = 501/501, size = 198, inode = 11616449, nlinks = 0,
> digest =
> > > > 0xb5c763..., bufUsed = 1184
> > > > > > > > Wrote file Preferences: type = 5, mode = 040771,
> uid/gid =
> > > > 0/1002, size = 4096, inode = 11616431, nlinks = 0, digest =
> 0x000000...,
> > > > bufUsed = 1241
> > > > > > > >
> > > > > > > >
> > > > > > > > I am also attaching the relevant attrib file
> > > > > > > > [DELETED ATTACHMENT attrib, Untyped binary data]
> > > > > >
> > > > > >
> > > > > > _______________________________________________
> > > > > > BackupPC-users mailing list
> > > > > > BackupPC-users@lists.sourceforge.net
> > > > > > List:
> https://lists.sourceforge.net/lists/listinfo/backuppc-users
> > > > > > Wiki: http://backuppc.wiki.sourceforge.net
> > > > > > Project: http://backuppc.sourceforge.net/
> > > > >
> > > > >
> > > > > _______________________________________________
> > > > > BackupPC-users mailing list
> > > > > BackupPC-users@lists.sourceforge.net
> > > > > List:
> https://lists.sourceforge.net/lists/listinfo/backuppc-users
> > > > > Wiki: http://backuppc.wiki.sourceforge.net
> > > > > Project: http://backuppc.sourceforge.net/
> > > >
> >
> >
> > _______________________________________________
> > BackupPC-users mailing list
> > BackupPC-users@lists.sourceforge.net
> > List: https://lists.sourceforge.net/lists/listinfo/backuppc-users
> > Wiki: http://backuppc.wiki.sourceforge.net
> > Project: http://backuppc.sourceforge.net/
>
_______________________________________________
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List: https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki: http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/
