[Christian]
thanks Tim, for the partial enlightenment. :) Unfortunately my
BTree/C-wisdom is much much smaller than yours, so I got to check a
couple of assumptions over here. :)
Yup, it really helps if you have a 64-bit box to check them on.
[Christian]
Hah. Looks like BTrees can accept 2**31 on machines where maxint is a
larger ...
[Tim]
Note that "accept" in this case means "silently loses the
most-significant bits", and that's a BTree bug on platforms where
sizeof(long) > sizeof(int):
http://www.zope.org/Collectors/Zope/1592
I was able to reproduce this on an Intel 64-Bit machine (EM64T) running
Linux and gcc.
And I expect the same will happen on any 64-bit platform other than
Win64 (which is unique, AFAICT, in leaving sizeof(long) == 4 instead
of boosting it to 8).
For one: I didn't see any compiler warnings. That sounds bad, right?
The underlying BTree bug is assignments of the form (usually "hidden"
in macro expansions):
some_C_int = some_C_long;
When sizeof(int) < sizeof(long), that can silently lose information.
I was really hoping that major compilers on boxes where sizeof(int) <
sizeof(long) would warn about that. Oh well.
The problem arises because what _Python_ calls "int" is what C calls
"long", but the I-flavor BTree code stores C "int", and C "int"
doesn't correspond to any Python type (except "by accident" on 32-bit
boxes). C "int" is 4 bytes on all known current 32- and 64-bit
platforms, but the size of what Python calls "int" varies. The BTree
code isn't aware of the possible mismatch, storing Python "int" (C
"long") into C "int" without any checking.
2**31 doesn't actually lose any bits when you store it, but it will
probably misinterpret the high-order data bit as the sign bit when you
fetch it again, magically changing it into -2**31.
I can store 2**31 in the BTree, but the keys() method will tell you that
it actually stored -2**31.
Right. If it's not clear, this is because the _bit_ pattern
0x80000000
is 2**31 when viewed as an 8-byte C long, but is -2**31 when viewed as
a 4-byte C long. If you had, e.g., stored 2**32 instead, you would
have gotten 0 back when you fetched it (the top 32 bits are simply
thrown away).
...
Did you see my simpler suggestion for fixing the underlying bug (it
was a one-liner change to the original code)? When you get tired of
fighting the 64-bit BTree bug here (it will be a minor miracle if the
test actually passes now on a 64-bit box, despite all you've tried
...), look that up ;-)
Nope, I didn't find your one-liner. :) Can you post it explicitly for my
blind eyes?
Here; it was a reply to one of the checkin messages:
http://mail.zope.org/pipermail/checkins/2006-June/002395.html
The key problem in the original intid code is that it used randint()
instead of randrange(), theoretically allowing 2**31 to be a return
value. It's _almost_ enough just to use randrange() instead.
Unfortunately, that's not quite enough; see the msg for what is.
It _could_ be fixed by adding 2 characters to the original, changing
2**31 to 2**31-2 (or to 0x7ffffffe), but that would leave it pretty
mysterious ;-)
I think I could have made up something that made it work, but I started
looking into making the BTree behave sanely.
My idea was, to modify the BTree code in a way that it actually checks
after the type cast whether the casted value is equal to the requested
key, or alternatively try making the CHECK_KEY macro do an "exact type
match" instead of allowing subclasses. But that wouldn't work either as
2**31 is still an int on those platforms. I'm a bit puzzled now. :)
There's potential silent information loss for both Ix-flavor BTree
keys and xI-flavor BTree values. There are two robust ways to check
("robust" means that Python's C code has used these ways at various
times for many years now without problems). Complain if:
some_C_long < INT_MIN || some_C_long > INT_MAX
or complain if (this sounds close to what you had in mind above, and
is my favorite):
(long)(int)some_C_long != some_C_long
Because the problem is due to bogus assumptions about the relationship
between C types, it's not going to help to examine Python's idea of
type.
Checking isn't needed (can't fail) if SIZEOF_INT == SIZEOF_LONG
(Python.h supplies definitions for those macros), so there's some
worth to skipping checks when that's not true. Unfortunately, C
doesn't allow "#if" preprocessor statements _inside_ macro expansions,
so the best way to do that isn't immediately clear.
In short, irritiating little issues abound :-(. That's why I couldn't
make time to fix it (relatively high cost with no benefit on most Zope
platforms).
Note that if ZODB moves to 64-bit Ix/xI BTrees on all boxes (IIRC, Jim
and Fred were agitating in that direction, but suffered massive
short-sighted ;-) opposition), the BTree problem would go away by
magic (C "int" would no longer be the type of Ix keys or xI values).
_______________________________________________
Zope3-dev mailing list
Zope3-dev@zope.org
Unsub: http://mail.zope.org/mailman/options/zope3-dev/archive%40mail-archive.com
