Hi Francesc,
On Thu, 2009-12-31 at 13:17 +0100, Francesc Alted wrote:
> Hi David,
>
> Yes, this is mostly a Python issue in 32-bit platforms. Look at this:
>
> >>> import array # The native array Python module
> >>> b = array.array('i')
> >>> b.append(1)
> >>> b.append(1L)
> >>> b.append(2**32-999)
> Traceback (most recent call last):
> File "<stdin>", line 1, in ?
> OverflowError: long int too large to convert to int
>
> and NumPy behaves exactly the same:
>
> >>> import numpy as np
> >>> a = np.array([1], dtype='i4')
> >>> a[0] = 2
> >>> a[0] = 2L
> >>> a[0] = 2**32-999
> Traceback (most recent call last):
> File "<stdin>", line 1, in ?
> OverflowError: long int too large to convert to int
>
> PyTables inherits this behaviour from NumPy, although the error raised
> is a `TypeError` instead of an `OverflowError`. This is due to the
> fact that error handling is not very sophisticated in the `Row`
> accessor (speed is favoured instead).
I tried these examples both on my 32-bit laptop and our 64-bit server,
with the same results on 32-bit and the strange type conversion on
64-bit. I have:
>>> import numpy as np
>>> a = np.array([1], dtype='i4')
>>> a[0] = 2**32-999
>>> a
array([-999], dtype=int32)
So, NumPy is inconsistent here...
> So, as you can see, performing this operation (i.e. interpreting
> signed 32-bit integers as unsigned ones) is not supported by Python in
> 32-bit platforms. So I suggest you to find another way to do what you
> are after (perhaps you can use an UInt32Col and do conversions
> afterwards).
Well, what I'm after is nothing like type conversion, but rather
consistency. I don't want type conversion, but since it happened on
64-bit, it masked a bug I have in our client-side scripts. It didn't
happen on 32-bit, but raised an exception on my development machine
(i.e. netbook). I'd rather have this raise an exception on 64-bit as
well.
Thanks for your explanation!
Best regards,
David
>
> Hope that helps,
>
> Francesc
>
>
> 2009/12/29, David Fokkema <[email protected]>:
> > Hi list,
> >
> > PyTables 2.1.2 with HDF5 1.6.6 and both NumPy 1.3.0 and 1.4.0 on 32-bit
> > vs
> > PyTables 2.1.2 with HDF5 1.8.3 and NumPy 1.4.0rc1 on 64-bit
> >
> > I have lots of clients interpreting detector data and sending pickled
> > objects over http to a server, which uses PyTables to store it (yes, I'm
> > very happy now that everything's running; it works very well!). I have a
> > bug on the client which I didn't notice until I had a crash on my dev
> > machine. I ultimately found an inconsistency in PyTables.
> >
> > On the client, I'm interpreting signed 32-bit integers as unsigned ones
> > (ouch). So, a value of -999 (error flag) is interpreted as 2**32 - 999 =
> > 4294966297L. This value is received by the server, which tries to store
> > it in a Int32Col.
> >
> > On 64-bit: no problem! The stored value is actually -999, so some
> > conversion takes place. This is why I didn't notice.
> >
> > On 32-bit: in tables.tableExtension.Row.__setitem__(): invalid type
> > (<type 'long'>) for column ``col``
> >
> > On 32-bit, I tested with the distributions NumPy (1.3.0) and PyPI's
> > NumPy (1.4.0), both with the same result.
> >
> > This shouldn't be, right? I'm not sure if it is NumPy or PyTables...
> >
> > Thanks,
> >
> > David
> >
> >
> >>>> import tables
> >>>> class MyTable(tables.IsDescription):
> > ... col = tables.Int32Col()
> > ...
> >>>> data = tables.openFile('/tmp/test.h5', 'w')
> >>>> data.createTable('/', 'test', MyTable)
> > /test (Table(0,)) ''
> > description := {
> > "col": Int32Col(shape=(), dflt=0, pos=0)}
> > byteorder := 'little'
> > chunkshape := (2048,)
> >>>> row = data.root.test.row
> >>>> row['col'] = 2**32 - 999
> > ------------------------------------------------------------
> > Traceback (most recent call last):
> > File "<ipython console>", line 1, in <module>
> > File "tableExtension.pyx", line 1309, in
> > tables.tableExtension.Row.__setitem__
> > TypeError: invalid type (<type 'long'>) for column ``col``
> >
> >
> >
> > ------------------------------------------------------------------------------
> > This SF.Net email is sponsored by the Verizon Developer Community
> > Take advantage of Verizon's best-in-class app development support
> > A streamlined, 14 day to market process makes app distribution fast and easy
> > Join now and get one step closer to millions of Verizon customers
> > http://p.sf.net/sfu/verizon-dev2dev
> > _______________________________________________
> > Pytables-users mailing list
> > [email protected]
> > https://lists.sourceforge.net/lists/listinfo/pytables-users
> >
>
>
------------------------------------------------------------------------------
This SF.Net email is sponsored by the Verizon Developer Community
Take advantage of Verizon's best-in-class app development support
A streamlined, 14 day to market process makes app distribution fast and easy
Join now and get one step closer to millions of Verizon customers
http://p.sf.net/sfu/verizon-dev2dev
_______________________________________________
Pytables-users mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/pytables-users
