Re: [Numpy-discussion] numpy pickling problem - python 2 vs. python 3

2015-03-06 Thread Julian Taylor
On 07.03.2015 00:20, Pauli Virtanen wrote:
 06.03.2015, 22:43, Eric Firing kirjoitti:
 On 2015/03/06 10:23 AM, Pauli Virtanen wrote:
 06.03.2015, 20:00, Benjamin Root kirjoitti:
 A slightly different way to look at this is one of sharing data. If I am
 working on a system with 3.4 and I want to share data with others who may
 be using a mix of 2.7 and 3.3 systems, this problem makes npz format much
 less attractive.

 pickle is used in npy files only if there are object arrays in them.
 Of course, savez could just decline saving object arrays.

 Or issue a prominent warning.
 
 https://github.com/numpy/numpy/pull/5641
 

I think the ship for a warning has long sailed. At this point its
probably more an annoyance for python3 users and will not prevent many
more python2 users from saving files that can't be loaded into python3.
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion


Re: [Numpy-discussion] numpy pickling problem - python 2 vs. python 3

2015-03-06 Thread Pauli Virtanen
06.03.2015, 22:23, Pauli Virtanen kirjoitti:
 06.03.2015, 20:00, Benjamin Root kirjoitti:
 A slightly different way to look at this is one of sharing data. If I am
 working on a system with 3.4 and I want to share data with others who may
 be using a mix of 2.7 and 3.3 systems, this problem makes npz format much
 less attractive.
 
 pickle is used in npy files only if there are object arrays in them.
 Of course, savez could just decline saving object arrays.

np.load is missing the Py2-3 workaround flags that pickle.load has,
probably could be added:

https://github.com/numpy/numpy/pull/5640


___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion


Re: [Numpy-discussion] numpy pickling problem - python 2 vs. python 3

2015-03-06 Thread Pauli Virtanen
06.03.2015, 22:43, Eric Firing kirjoitti:
 On 2015/03/06 10:23 AM, Pauli Virtanen wrote:
 06.03.2015, 20:00, Benjamin Root kirjoitti:
 A slightly different way to look at this is one of sharing data. If I am
 working on a system with 3.4 and I want to share data with others who may
 be using a mix of 2.7 and 3.3 systems, this problem makes npz format much
 less attractive.

 pickle is used in npy files only if there are object arrays in them.
 Of course, savez could just decline saving object arrays.
 
 Or issue a prominent warning.

https://github.com/numpy/numpy/pull/5641

___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion


Re: [Numpy-discussion] numpy pickling problem - python 2 vs. python 3

2015-03-06 Thread Eric Firing
On 2015/03/06 1:29 PM, Julian Taylor wrote:
 I think the ship for a warning has long sailed. At this point its
 probably more an annoyance for python3 users and will not prevent many
 more python2 users from saving files that can't be loaded into python3.

The point of a warning is that anything that relies on pickles is 
fundamentally unreliable in the long term.  It's potentially a surprise 
that the npz format relies on pickles.
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion


[Numpy-discussion] Numpy 1.10

2015-03-06 Thread Charles R Harris
Hi All,

Time to start thinking about numpy 1.10. At the moment there are 21
blockers and 93 PRs. it would be good if we could prioritize the blockers
and the PRs. It might be a good idea to also look at closing some of the
aging PRs. These days it seems to be normal to have 20 or so PRs in
progress, it it would be nice if we could get the total down around 40-50.

I don't see a numpy 1.10 before July, but it is going to take a few months
to get things in order, so now is a good time to start.

Chuck
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion


Re: [Numpy-discussion] Adding keyword to asarray and asanyarray.

2015-03-06 Thread Benjamin Root
On Fri, Mar 6, 2015 at 7:59 AM, Charles R Harris charlesr.har...@gmail.com
wrote:

 Datetime64 seems to use the highest precision

 In [12]: result_type(ones(1, dtype='datetime64[D]'), 'datetime64[us]')
 Out[12]: dtype('M8[us]')

 In [13]: result_type(ones(1, dtype='datetime64[D]'), 'datetime64[Y]')
 Out[13]: dtype('M8[D]')



Ah, yes, that's what I'm looking for. +1 from me to have this in
asarray/asanyarray. Of course, there is always the usual caveats about
converting your datetime data in this manner, but this would be helpful in
many situations in writing functions that expect to deal with temporal data
at the resolution of minutes or somesuch.

Cheers!
Ben Root
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion


Re: [Numpy-discussion] numpy pickling problem - python 2 vs. python 3

2015-03-06 Thread Ryan Nelson
Arnd,

I can see where this is an issue. If you are trying to update your code for
Py3, I still think that it would really help to add a version attribute of
some sort to your new HDF files. You can then write a little check in your
access code that looks for this variable. If it is not present, you know
that it is an old file, and you can use the trick that I gave you.
Otherwise, it will process the file as normal. It could even throw a little
error saying that the file is outdated. You could write a small conversion
script that could run through old files and reprocess them into the new
format. Fortunately, Python is pretty good at automating tasks, even for
hundreds of files :)
It might be informative to ask at the PyTables list to see what they've
done. The Pandas folks also do a lot with HDF files, and they have
certainly worked their way through the Py2-3 transition. Also, because this
is an issue with Python pickle, a quick note on SO might get some hits. I
tried your script using a lists of list, rather than a list of arrays, and
the same problem still persists, so as Pauli notes this is going to be a
problem regardless of the type of attributes you set, I think your just
going to have to hard code some kind of check in your code to switch
behavior. I recently switched to using Py3 exclusively, and although it was
painful at first, I'm quite happy with Py3 overall. I also use the Anaconda
Python distribution, which makes it very easy to have Py2 and Py3
environments if you need to switch back and forth.
Sorry if that doesn't help much. Just some thoughts from my recent
conversion experiences.

Ryan



On Fri, Mar 6, 2015 at 9:48 AM, Arnd Baecker arnd.baec...@web.de wrote:

 On Fri, 6 Mar 2015, Pauli Virtanen wrote:

  Arnd Baecker arnd.baecker at web.de writes:
  [clip]
  Still I would have thought that this should be working out-of-the box,
  i.e. without the pickle.loads trick?
 
  Pickle files should be considered incompatible between Python 2 and
 Python 3.
 
  Python 3 interprets all bytes objects saved by Python 2 as str and
 attempts
  to decode them under some unicode locale. The default locale is ASCII,
 so it
  will simply just fail in most cases if the files contain any binary data.
 
  Failing by default is also the right thing to do, since the saved bytes
  objects might actually represent strings in some locale, and ASCII is the
  safest guess.
 
  This behavior is that of Python's pickle module, and does not depend on
 Numpy.

 Thank's a lot for the explanation!

 So what is then the recommded way to save data under python 2 so that
 they can still be loaded under python 3?

 For example using np.save with a list of arrays works fine
 either on python 2 or on python 3.
 However it does not work if one tries to open under python 3
 a file generated before on python 2.
 (Again, because pickle is involved internally
python3.4/site-packages/numpy/lib/npyio.py,
line 393, in load  return format.read_array(fid)
File python34/lib/python3.4/site-packages/numpy/lib/format.py,
line 602, in read_array  array = pickle.load(fp)
UnicodeDecodeError: 'ascii' codec can't decode byte 0xf0  ...

 Just to be clear: I don't want to beat a dead horse here - for my usage
 via pytables I was able to solve the loading of old files following
 Ryan's solutions. Personally I don't use .npy files.
 Maybe saving a list containing arrays is an unusual example ...

 Still, I am a little bit worried about backwards-compatibility:
 being able to load old data files is an important issue
 as by this it is possible to check whether current code still
 reproduces previously obtained (maybe also published) results.

 Best, Arnd

 ___
 NumPy-Discussion mailing list
 NumPy-Discussion@scipy.org
 http://mail.scipy.org/mailman/listinfo/numpy-discussion

___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion


Re: [Numpy-discussion] numpy pickling problem - python 2 vs. python 3

2015-03-06 Thread Arnd Baecker
On Fri, 6 Mar 2015, Pauli Virtanen wrote:

 Arnd Baecker arnd.baecker at web.de writes:
 [clip]
 Still I would have thought that this should be working out-of-the box,
 i.e. without the pickle.loads trick?

 Pickle files should be considered incompatible between Python 2 and Python 3.

 Python 3 interprets all bytes objects saved by Python 2 as str and attempts
 to decode them under some unicode locale. The default locale is ASCII, so it
 will simply just fail in most cases if the files contain any binary data.

 Failing by default is also the right thing to do, since the saved bytes
 objects might actually represent strings in some locale, and ASCII is the
 safest guess.

 This behavior is that of Python's pickle module, and does not depend on Numpy.

Thank's a lot for the explanation!

So what is then the recommded way to save data under python 2 so that
they can still be loaded under python 3?

For example using np.save with a list of arrays works fine 
either on python 2 or on python 3.
However it does not work if one tries to open under python 3
a file generated before on python 2.
(Again, because pickle is involved internally
   python3.4/site-packages/numpy/lib/npyio.py,
   line 393, in load  return format.read_array(fid)
   File python34/lib/python3.4/site-packages/numpy/lib/format.py,
   line 602, in read_array  array = pickle.load(fp)
   UnicodeDecodeError: 'ascii' codec can't decode byte 0xf0  ...

Just to be clear: I don't want to beat a dead horse here - for my usage
via pytables I was able to solve the loading of old files following
Ryan's solutions. Personally I don't use .npy files.
Maybe saving a list containing arrays is an unusual example ...

Still, I am a little bit worried about backwards-compatibility:
being able to load old data files is an important issue
as by this it is possible to check whether current code still
reproduces previously obtained (maybe also published) results.

Best, Arnd

___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion


Re: [Numpy-discussion] numpy pickling problem - python 2 vs. python 3

2015-03-06 Thread Pauli Virtanen
Arnd Baecker arnd.baecker at web.de writes:
[clip]
 Still I would have thought that this should be working out-of-the box,
 i.e. without the pickle.loads trick?

Pickle files should be considered incompatible between Python 2 and Python 3.

Python 3 interprets all bytes objects saved by Python 2 as str and attempts
to decode them under some unicode locale. The default locale is ASCII, so it
will simply just fail in most cases if the files contain any binary data.

Failing by default is also the right thing to do, since the saved bytes
objects might actually represent strings in some locale, and ASCII is the
safest guess.

This behavior is that of Python's pickle module, and does not depend on Numpy.

___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion


Re: [Numpy-discussion] Adding keyword to asarray and asanyarray.

2015-03-06 Thread Charles R Harris
On Thu, Mar 5, 2015 at 10:02 PM, josef.p...@gmail.com wrote:

 On Thu, Mar 5, 2015 at 12:33 PM, Charles R Harris
 charlesr.har...@gmail.com wrote:
 
 
  On Thu, Mar 5, 2015 at 10:04 AM, Chris Barker chris.bar...@noaa.gov
 wrote:
 
  On Thu, Mar 5, 2015 at 8:42 AM, Benjamin Root ben.r...@ou.edu wrote:
 
  dare I say... datetime64/timedelta64 support?
 
 
  well, the precision of those is 64 bits, yes? so if you asked for less
  than that, you'd still get a dt64. If you asked for 64 bits, you'd get
 it,
  if you asked for datetime128  -- what would you get???
 
  a 128 bit integer? or an Exception, because there is no 128bit datetime
  dtype.
 
  But I think this is the same problem with any dtype -- if you ask for a
  precision that doesn't exist, you're going to get an error.
 
  Is there a more detailed description of the proposed feature anywhere?
 Do
  you specify a dtype as a precision? or jsut the precision, and let the
 dtype
  figure it out for itself, i.e.:
 
  precision=64
 
  would give you a float64 if the passed in array was a float type, but a
  int64 if the passed in array was an int type, or a uint64 if the passed
 in
  array was a unsigned int type, etc.
 
  But in the end,  I wonder about the use case. I generaly use asarray one
  of two ways:
 
  Without a dtype -- to simple make sure I've got an ndarray of SOME
 dtype.
 
  or
 
  With a dtype - because I really care about the dtype -- usually because
 I
  need to pass it on to C code or something.
 
  I don't think I'd ever need at least some precision, but not care if I
 got
  more than that...
 
 
  The main use that I want to cover is that float64 and complex128 have the
  same precision and it would be good if either is acceptable.  Also, one
  might just want either float32 or float64, not just one of the two.
 Another
  intent is to make the fewest possible copies. The determination of the
  resulting type is made using the result_type function.


 How does this work for object arrays, or datetime?

 Can I specify at least float32 or float64, and it raises an exception
 if it cannot be converted?

 The problem we have in statsmodels is that pandas frequently uses
 object arrays and it messes up patsy or statsmodels if it's not
 explicitly converted.


Object arrays go to object arrays, datetime64 depends.

In [10]: result_type(ones(1, dtype=object_), float32)
Out[10]: dtype('O')


Datetime64 seems to use the highest precision

In [12]: result_type(ones(1, dtype='datetime64[D]'), 'datetime64[us]')
Out[12]: dtype('M8[us]')

In [13]: result_type(ones(1, dtype='datetime64[D]'), 'datetime64[Y]')
Out[13]: dtype('M8[D]')

but doesn't convert to float

In [11]: result_type(ones(1, dtype='datetime64[D]'), float32)
---
TypeError Traceback (most recent call last)
ipython-input-11-e1a09e933dc7 in module()
 1 result_type(ones(1, dtype='datetime64[D]'), float32)

TypeError: invalid type promotion

What would you like it to do?

Chuck
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion


Re: [Numpy-discussion] Adding keyword to asarray and asanyarray.

2015-03-06 Thread josef.pktd
On Fri, Mar 6, 2015 at 7:59 AM, Charles R Harris
charlesr.har...@gmail.com wrote:


 On Thu, Mar 5, 2015 at 10:02 PM, josef.p...@gmail.com wrote:

 On Thu, Mar 5, 2015 at 12:33 PM, Charles R Harris
 charlesr.har...@gmail.com wrote:
 
 
  On Thu, Mar 5, 2015 at 10:04 AM, Chris Barker chris.bar...@noaa.gov
  wrote:
 
  On Thu, Mar 5, 2015 at 8:42 AM, Benjamin Root ben.r...@ou.edu wrote:
 
  dare I say... datetime64/timedelta64 support?
 
 
  well, the precision of those is 64 bits, yes? so if you asked for less
  than that, you'd still get a dt64. If you asked for 64 bits, you'd get
  it,
  if you asked for datetime128  -- what would you get???
 
  a 128 bit integer? or an Exception, because there is no 128bit datetime
  dtype.
 
  But I think this is the same problem with any dtype -- if you ask for a
  precision that doesn't exist, you're going to get an error.
 
  Is there a more detailed description of the proposed feature anywhere?
  Do
  you specify a dtype as a precision? or jsut the precision, and let the
  dtype
  figure it out for itself, i.e.:
 
  precision=64
 
  would give you a float64 if the passed in array was a float type, but a
  int64 if the passed in array was an int type, or a uint64 if the passed
  in
  array was a unsigned int type, etc.
 
  But in the end,  I wonder about the use case. I generaly use asarray
  one
  of two ways:
 
  Without a dtype -- to simple make sure I've got an ndarray of SOME
  dtype.
 
  or
 
  With a dtype - because I really care about the dtype -- usually because
  I
  need to pass it on to C code or something.
 
  I don't think I'd ever need at least some precision, but not care if I
  got
  more than that...
 
 
  The main use that I want to cover is that float64 and complex128 have
  the
  same precision and it would be good if either is acceptable.  Also, one
  might just want either float32 or float64, not just one of the two.
  Another
  intent is to make the fewest possible copies. The determination of the
  resulting type is made using the result_type function.


 How does this work for object arrays, or datetime?

 Can I specify at least float32 or float64, and it raises an exception
 if it cannot be converted?

 The problem we have in statsmodels is that pandas frequently uses
 object arrays and it messes up patsy or statsmodels if it's not
 explicitly converted.


 Object arrays go to object arrays, datetime64 depends.

 In [10]: result_type(ones(1, dtype=object_), float32)
 Out[10]: dtype('O')


 Datetime64 seems to use the highest precision

 In [12]: result_type(ones(1, dtype='datetime64[D]'), 'datetime64[us]')
 Out[12]: dtype('M8[us]')

 In [13]: result_type(ones(1, dtype='datetime64[D]'), 'datetime64[Y]')
 Out[13]: dtype('M8[D]')

 but doesn't convert to float

 In [11]: result_type(ones(1, dtype='datetime64[D]'), float32)
 ---
 TypeError Traceback (most recent call last)
 ipython-input-11-e1a09e933dc7 in module()
  1 result_type(ones(1, dtype='datetime64[D]'), float32)

 TypeError: invalid type promotion

 What would you like it to do?

Note: the dtype handling in statsmodels is still a mess, and we just
plugged some of the worst cases.


What we would need is asarray with at least a minimum precision (e.g.
float32) and raise an exception if it's not numeric, like string,
object, custom dtypes ...

However, we need custom dtype handling in statsmodels anyway, so the
enhancement to asarray with exceptions would mainly be convenient to
get something to work with because pandas and numpy as now object
array friendly.

I assume scipy also has insufficient checks for non-numeric dtypes, AFAIR.


Josef



 Chuck

 ___
 NumPy-Discussion mailing list
 NumPy-Discussion@scipy.org
 http://mail.scipy.org/mailman/listinfo/numpy-discussion

___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion


Re: [Numpy-discussion] numpy pickling problem - python 2 vs. python 3

2015-03-06 Thread Sebastian

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256

Hi all,

As this also affects .npy files, which uses pickle internally, why can't
this be done by Numpy itself? This breaks backwards compatibility in a
very bad way in my opinion.

The company I worked for uses Numpy and consorts a lot and also has many
data in .npy and pickle files. They currently work with 2.7, but I also
tried to develop my programs to be compatible with Py 3. But this was
not possible when it came to the point of dumping and loading npy files.
I think this will be major reason why people won't take the step forward
to Py3 and Numpy is not considered to be compatible to Python 3.

just my 5 cents,
Sebastian

On 03/06/2015 04:37 PM, Ryan Nelson wrote:
 Arnd,

 I can see where this is an issue. If you are trying to update your
code for Py3, I still think that it would really help to add a version
attribute of some sort to your new HDF files. You can then write a
little check in your access code that looks for this variable. If it is
not present, you know that it is an old file, and you can use the trick
that I gave you. Otherwise, it will process the file as normal. It could
even throw a little error saying that the file is outdated. You could
write a small conversion script that could run through old files and
reprocess them into the new format. Fortunately, Python is pretty good
at automating tasks, even for hundreds of files :)
 It might be informative to ask at the PyTables list to see what
they've done. The Pandas folks also do a lot with HDF files, and they
have certainly worked their way through the Py2-3 transition. Also,
because this is an issue with Python pickle, a quick note on SO might
get some hits. I tried your script using a lists of list, rather than a
list of arrays, and the same problem still persists, so as Pauli notes
this is going to be a problem regardless of the type of attributes you
set, I think your just going to have to hard code some kind of check in
your code to switch behavior. I recently switched to using Py3
exclusively, and although it was painful at first, I'm quite happy with
Py3 overall. I also use the Anaconda Python distribution, which makes it
very easy to have Py2 and Py3 environments if you need to switch back
and forth.
 Sorry if that doesn't help much. Just some thoughts from my recent
conversion experiences.

 Ryan



 On Fri, Mar 6, 2015 at 9:48 AM, Arnd Baecker arnd.baec...@web.de
mailto:arnd.baec...@web.de wrote:

 On Fri, 6 Mar 2015, Pauli Virtanen wrote:

  Arnd Baecker arnd.baecker at web.de http://web.de writes:
  [clip]
  Still I would have thought that this should be working
out-of-the box,
  i.e. without the pickle.loads trick?
 
  Pickle files should be considered incompatible between Python 2
and Python 3.
 
  Python 3 interprets all bytes objects saved by Python 2 as str
and attempts
  to decode them under some unicode locale. The default locale is
ASCII, so it
  will simply just fail in most cases if the files contain any
binary data.
 
  Failing by default is also the right thing to do, since the
saved bytes
  objects might actually represent strings in some locale, and
ASCII is the
  safest guess.
 
  This behavior is that of Python's pickle module, and does not
depend on Numpy.

 Thank's a lot for the explanation!

 So what is then the recommded way to save data under python 2 so that
 they can still be loaded under python 3?

 For example using np.save with a list of arrays works fine
 either on python 2 or on python 3.
 However it does not work if one tries to open under python 3
 a file generated before on python 2.
 (Again, because pickle is involved internally
python3.4/site-packages/numpy/lib/npyio.py,
line 393, in load  return format.read_array(fid)
File python34/lib/python3.4/site-packages/numpy/lib/format.py,
line 602, in read_array  array = pickle.load(fp)
UnicodeDecodeError: 'ascii' codec can't decode byte 0xf0  ...

 Just to be clear: I don't want to beat a dead horse here - for my
usage
 via pytables I was able to solve the loading of old files following
 Ryan's solutions. Personally I don't use .npy files.
 Maybe saving a list containing arrays is an unusual example ...

 Still, I am a little bit worried about backwards-compatibility:
 being able to load old data files is an important issue
 as by this it is possible to check whether current code still
 reproduces previously obtained (maybe also published) results.

 Best, Arnd

 ___
 NumPy-Discussion mailing list
 NumPy-Discussion@scipy.org mailto:NumPy-Discussion@scipy.org
 http://mail.scipy.org/mailman/listinfo/numpy-discussion




 ___
 NumPy-Discussion mailing list
 NumPy-Discussion@scipy.org
 

Re: [Numpy-discussion] numpy pickling problem - python 2 vs. python 3

2015-03-06 Thread Benjamin Root
A slightly different way to look at this is one of sharing data. If I am
working on a system with 3.4 and I want to share data with others who may
be using a mix of 2.7 and 3.3 systems, this problem makes npz format much
less attractive.

Ben Root

On Fri, Mar 6, 2015 at 12:51 PM, Charles R Harris charlesr.har...@gmail.com
 wrote:



 On Fri, Mar 6, 2015 at 10:34 AM, Sebastian se...@sebix.at wrote:


 -BEGIN PGP SIGNED MESSAGE-
 Hash: SHA256

 Hi all,

 As this also affects .npy files, which uses pickle internally, why can't
 this be done by Numpy itself? This breaks backwards compatibility in a
 very bad way in my opinion.

 The company I worked for uses Numpy and consorts a lot and also has many
 data in .npy and pickle files. They currently work with 2.7, but I also
 tried to develop my programs to be compatible with Py 3. But this was
 not possible when it came to the point of dumping and loading npy files.
 I think this will be major reason why people won't take the step forward
 to Py3 and Numpy is not considered to be compatible to Python 3.


 Are you suggesting adding a flag to the files to mark the python version
 in which they were created? The *.npy format is versioned, so something
 could probably be done with that.

 Chuck

 ___
 NumPy-Discussion mailing list
 NumPy-Discussion@scipy.org
 http://mail.scipy.org/mailman/listinfo/numpy-discussion


___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion


Re: [Numpy-discussion] numpy pickling problem - python 2 vs. python 3

2015-03-06 Thread Charles R Harris
On Fri, Mar 6, 2015 at 10:34 AM, Sebastian se...@sebix.at wrote:


 -BEGIN PGP SIGNED MESSAGE-
 Hash: SHA256

 Hi all,

 As this also affects .npy files, which uses pickle internally, why can't
 this be done by Numpy itself? This breaks backwards compatibility in a
 very bad way in my opinion.

 The company I worked for uses Numpy and consorts a lot and also has many
 data in .npy and pickle files. They currently work with 2.7, but I also
 tried to develop my programs to be compatible with Py 3. But this was
 not possible when it came to the point of dumping and loading npy files.
 I think this will be major reason why people won't take the step forward
 to Py3 and Numpy is not considered to be compatible to Python 3.


Are you suggesting adding a flag to the files to mark the python version in
which they were created? The *.npy format is versioned, so something could
probably be done with that.

Chuck
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion


Re: [Numpy-discussion] numpy pickling problem - python 2 vs. python 3

2015-03-06 Thread Pauli Virtanen
06.03.2015, 20:00, Benjamin Root kirjoitti:
 A slightly different way to look at this is one of sharing data. If I am
 working on a system with 3.4 and I want to share data with others who may
 be using a mix of 2.7 and 3.3 systems, this problem makes npz format much
 less attractive.

pickle is used in npy files only if there are object arrays in them.
Of course, savez could just decline saving object arrays.


___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion


Re: [Numpy-discussion] appveyor CI

2015-03-06 Thread Chris Barker
On Thu, Mar 5, 2015 at 5:07 PM, Charles R Harris charlesr.har...@gmail.com
wrote:

 Do line endings in the scripts matter?


I have no idea if powershell cares about line endings, but if you are using
git, then you'll want to make sure that your repo is properly configured to
normalize line endings -- then there should be no problems. And you realy
want that anyway for any multi-platform project.

-CHB

-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/ORR(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion


Re: [Numpy-discussion] numpy pickling problem - python 2 vs. python 3

2015-03-06 Thread Eric Firing
On 2015/03/06 10:23 AM, Pauli Virtanen wrote:
 06.03.2015, 20:00, Benjamin Root kirjoitti:
 A slightly different way to look at this is one of sharing data. If I am
 working on a system with 3.4 and I want to share data with others who may
 be using a mix of 2.7 and 3.3 systems, this problem makes npz format much
 less attractive.

 pickle is used in npy files only if there are object arrays in them.
 Of course, savez could just decline saving object arrays.

Or issue a prominent warning.

Eric

___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion