[Numpy-discussion] problem in running test(nose) with numpy/scipy
hi, i have installed numpy (1.6.0) and scipy (0.9). , nose version is 1.0 i have intel cluster toolkit installed on my system. (11/069 version and mkl 10.3). i have machine having intel xeon processor and rhel 5.2 x86_64 platform. i have installed it with intel compilers. when i execute numpy.test and scipy.test, it hangs . numpy.test(verbose=3) Running unit tests for numpy NumPy version 1.6.0 NumPy is installed in /home/akshar/Python-2.6/lib/python2.6/site-packages/numpy Python version 2.6 (r26:66714, May 29 2011, 15:10:47) [GCC 4.1.2 20071124 (Red Hat 4.1.2-42)] nose version 1.0.0 nose.config: INFO: Excluding tests matching ['f2py_ext', 'f2py_f90_ext', 'gen_ext', 'pyrex_ext', 'swig_ext'] nose.selector: INFO: /home/akshar/Python-2.6/lib/ python2.6/site-packages/numpy/core/multiarray.so is executable; skipped nose.selector: INFO: /home/akshar/Python-2.6/lib/python2.6/site-packages/numpy/core/scalarmath.so is executable; skipped nose.selector: INFO: /home/akshar/Python-2.6/lib/python2.6/site-packages/numpy/core/umath.so is executable; skipped nose.selector: INFO: /home/akshar/Python-2.6/lib/python2.6/site-packages/numpy/core/multiarray_tests.so is executable; skipped nose.selector: INFO: /home/akshar/Python-2.6/lib/python2.6/site-packages/numpy/core/umath_tests.so is executable; skipped nose.selector: INFO: /home/akshar/Python-2.6/lib/python2.6/site-packages/numpy/fft/fftpack_lite.so is executable; skipped nose.selector: INFO: /home/akshar/Python-2.6/lib/python2.6/site-packages/numpy/linalg/lapack_lite.so is executable; skipped nose.selector: INFO: /home/akshar/Python-2.6/lib/python2.6/site-packages/numpy/random/mtrand.so is executable; skipped test_api.test_fastCopyAndTranspose ... ok test_arrayprint.TestArrayRepr.test_nan_inf ... ok test_str (test_arrayprint.TestComplexArray) ... ok Ticket 844. ... ok test_blasdot.test_blasdot_used ... ok test_blasdot.test_dot_2args ... ok test_blasdot.test_dot_3args ... ok test_blasdot.test_dot_3args_errors ... ok test_creation (test_datetime.TestDateTime) ... ok test_creation_overflow (test_datetime.TestDateTime) ... ok test_divisor_conversion_as (test_datetime.TestDateTime) ... ok test_divisor_conversion_bday (test_datetime.TestDateTime) ... ok test_divisor_conversion_day (test_datetime.TestDateTime) ... ok test_divisor_conversion_fs (test_datetime.TestDateTime) ... ok test_divisor_conversion_hour (test_datetime.TestDateTime) ... ok test_divisor_conversion_minute (test_datetime.TestDateTime) ... ok test_divisor_conversion_month (test_datetime.TestDateTime) ... ok test_divisor_conversion_second (test_datetime.TestDateTime) ... ok test_divisor_conversion_week (test_datetime.TestDateTime) ... ok test_divisor_conversion_year (test_datetime.TestDateTime) ... ok test_hours (test_datetime.TestDateTime) ... ok test_from_object_array (test_defchararray.TestBasic) ... ok test_from_object_array_unicode (test_defchararray.TestBasic) ... ok test_from_string (test_defchararray.TestBasic) ... ok test_from_string_array (test_defchararray.TestBasic) ... ok test_from_unicode (test_defchararray.TestBasic) ... ok test_from_unicode_array (test_defchararray.TestBasic) ... ok test_unicode_upconvert (test_defchararray.TestBasic) ... ok test_it (test_defchararray.TestChar) ... ok test_equal (test_defchararray.TestComparisons) ... ok test_greater (test_defchararray.TestComparisons) ... ok test_greater_equal (test_defchararray.TestComparisons) ... ok test_less (test_defchararray.TestComparisons) ... ok test_less_equal (test_defchararray.TestComparisons) ... ok test_not_equal (test_defchararray.TestComparisons) ... ok test_equal (test_defchararray.TestComparisonsMixed1) ... ok test_greater (test_defchararray.TestComparisonsMixed1) ... ok test_greater_equal (test_defchararray.TestComparisonsMixed1) ... ok test_less (test_defchararray.TestComparisonsMixed1) ... ok test_less_equal (test_defchararray.TestComparisonsMixed1) ... ok test_not_equal (test_defchararray.TestComparisonsMixed1) ... ok test_equal (test_defchararray.TestComparisonsMixed2) ... ok test_greater (test_defchararray.TestComparisonsMixed2) ... ok test_greater_equal (test_defchararray.TestComparisonsMixed2) ... ok test_less (test_defchararray.TestComparisonsMixed2) ... ok test_less_equal (test_defchararray.TestComparisonsMixed2) ... ok test_not_equal (test_defchararray.TestComparisonsMixed2) ... ok test_count (test_defchararray.TestInformation) ... ok test_endswith (test_defchararray.TestInformation) ... ok test_find (test_defchararray.TestInformation) ... ok test_index (test_defchararray.TestInformation) ... ok test_isalnum (test_defchararray.TestInformation) ... ok test_isalpha (test_defchararray.TestInformation) ... ok test_isdigit (test_defchararray.TestInformation) ... ok test_islower (test_defchararray.TestInformation) ... ok test_isspace (test_defchararray.TestInformation) ... ok test_istitle (test_defchararray.TestInformation) ... ok test_isupper (test_defchararray.TestInformation) ... ok test_len (test_defchararray.TestInfor
Re: [Numpy-discussion] Moving to gcc 4.* for win32 installers ?
On Sun, Oct 30, 2011 at 7:18 AM, David Cournapeau wrote: > On Thu, Oct 27, 2011 at 5:19 PM, Ralf Gommers > wrote: >> Hi David, >> >> On Thu, Oct 27, 2011 at 3:02 PM, David Cournapeau >> wrote: >>> >>> Hi, >>> >>> I was wondering if we could finally move to a more recent version of >>> compilers for official win32 installers. This would of course concern >>> the next release cycle, not the ones where beta/rc are already in >>> progress. >>> >>> Basically, the pros: >>> - we will have to move at some point >>> - gcc 4.* seem less buggy, especially C++ and fortran. >>> - no need to maintain msvcr90 vodoo >>> The cons: >>> - it will most likely break the ABI >>> - we need to recompile atlas (but I can take care of it) >>> - the biggest: it is difficult to combine gfortran with visual >>> studio (more exactly you cannot link gfortran runtime to a visual >>> studio executable). The only solution I could think of would be to >>> recompile the gfortran runtime with Visual Studio, which for some >>> reason does not sound very appealing :) >> >> To get the datetime changes to work with MinGW, we already concluded that >> building with 4.x is more or less required (without recognizing some of the >> points you list above). Changes to mingw32ccompiler to fix compilation with >> 4.x went in in https://github.com/numpy/numpy/pull/156. It would be good if >> you could check those. > > I will look into it more carefully, but overall, it seems that > building atlas 3.8.4, numpy and scipy with gcc 4.x works quite well. > The main issue is that gcc 4.* adds some dependencies on mingw dlls. > There are two options: > - adding the dlls in the installers > - statically linking those, which seems to be a bad idea > (generalizing the dll boundaries problem to exception and things we > would rather not care about: > http://cygwin.com/ml/cygwin/2007-06/msg00332.html). > >> It probably makes sense make this move for numpy 1.7. If this breaks the ABI >> then it would be easiest to make numpy 1.7 the minimum required version for >> scipy 0.11. > > My thinking as well. It looks like it's really time to upgrade. pythonxy comes with gfortran and a new MingW, and I cannot build scipy anymore. I don't find an installer for the old MingW 3.5 anymore for my new computer. (I haven't seen any problems mixing gcc 4.4 with C/C++ code like scikits.learn against the official numpy/scipy installer versions.) I can volunteer for testing, since it looks like I'm set up for gfortran. Josef > > cheers, > > David > ___ > NumPy-Discussion mailing list > NumPy-Discussion@scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] NumPy nogil API
30.10.2011 21:48, mark florisson kirjoitti: > First, I'd like to report a bug. It seems ndarray does not implement > tp_traverse or tp_clear, so if you have a reference cycle in an > ndarray with dtype object none of those objects will ever be > collected. Indeed, this is missing. http://projects.scipy.org/numpy/ticket/1003 If I recall correctly, there was something painful in implementing this for Numpy arrays, though... > Secondly, please bear with me, I'm not a NumPy expert, but would it be > possible to have NumPy export an API that can be called without the > GIL? In the upcoming Cython release 0.16 (soon to be released) we will > have what is called typed memoryviews [1], which allow you to obtain a > typed view on any PEP 3118 buffer, and it allows you to do a lot more > things without the GIL. e.g. it allows you to pass these views around, > transpose them, slice them (in the same way as in NumPy but slightly > more restricted, it doesn't support masks and such), index them etc > without the GIL. The closest thing to making this to happen is the work made on porting Numpy to IronPython. Basically, a major part of that involved ripping the innards of Numpy out to a more reusable C library. It's been in a merge-limbo for some time now, however. -- Pauli Virtanen ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] NumPy nogil API
Hello, First, I'd like to report a bug. It seems ndarray does not implement tp_traverse or tp_clear, so if you have a reference cycle in an ndarray with dtype object none of those objects will ever be collected. Secondly, please bear with me, I'm not a NumPy expert, but would it be possible to have NumPy export an API that can be called without the GIL? In the upcoming Cython release 0.16 (soon to be released) we will have what is called typed memoryviews [1], which allow you to obtain a typed view on any PEP 3118 buffer, and it allows you to do a lot more things without the GIL. e.g. it allows you to pass these views around, transpose them, slice them (in the same way as in NumPy but slightly more restricted, it doesn't support masks and such), index them etc without the GIL. However, there is a lot of good functionality in NumPy that is simply not accessible without going through a Python and GIL layer, only to have NumPy (possibly) release the GIL. So instead we could have an API that takes a C-level ndarray view descriptor (shape/strides/ndim (possibly suboffsets), base type description) that would do the actual work (and perhaps doesn't even need to know anything about Python) and won't need the GIL (this wouldn't work for the dtype object, of course). The Py_buffer struct comes to mind, but format strings are hard to deal with. It would be the caller's responsibility to do any necessary synchronization. This API would simply be wrapped by a Python API that gets this "view" from the PyArrayObject, and which may or may not decide to release the GIL, and call the nogil API. This wrapping API could even be written easily in Cython. As for exceptions and error handling, there are many ways we can think of doing this without requiring the GIL. One of the reasons that I think this is important is that when you're using cython.parallel [2] you don't want to hold the gil, but you do want your NumPy goodies. Cython re-implements a very small subset of NumPy to get you the core functionality, but to get back to NumPy world you have to acquire the GIL, convert your memoryview slice to an ndarray (using the buffer interface through numpy.asarray()) and then have NumPy operate on that. It's a pain to write and it's terrible for performance. Even if you forget the GIL part, there's still the (expensive and explicit) conversion. In general I think there might be many advantages to such functionality other than for Cython. There shouldn't really be a reason to tie NumPy only to the CPython platform. Anyway, what do you guys think, does this make any sense? Mark [1]: https://sage.math.washington.edu:8091/hudson/job/cython-docs/doclinks/1/src/userguide/memoryviews.html [2]: http://docs.cython.org/src/userguide/parallelism.html ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] consensus (was: NA masks in the next numpy release?)
Hi, On Sun, Oct 30, 2011 at 12:24 PM, Ralf Gommers wrote: > > > On Sat, Oct 29, 2011 at 11:55 PM, Matthew Brett > wrote: >> >> Hi, >> >> On Sat, Oct 29, 2011 at 2:48 PM, Ralf Gommers >> wrote: >> > >> > >> > On Sat, Oct 29, 2011 at 11:36 PM, Matthew Brett >> > >> > wrote: >> >> >> >> Hi, >> >> >> >> On Sat, Oct 29, 2011 at 1:48 PM, Matthew Brett >> >> >> >> wrote: >> >> > Hi, >> >> > >> >> > On Sat, Oct 29, 2011 at 1:44 PM, Ralf Gommers >> >> > wrote: >> >> >> >> >> >> >> >> >> On Sat, Oct 29, 2011 at 9:04 PM, Matthew Brett >> >> >> >> >> >> wrote: >> >> >>> >> >> >>> Hi, >> >> >>> >> >> >>> On Sat, Oct 29, 2011 at 3:26 AM, Ralf Gommers >> >> >>> wrote: >> >> >>> > >> >> >>> > >> >> >>> > On Sat, Oct 29, 2011 at 1:37 AM, Matthew Brett >> >> >>> > >> >> >>> > wrote: >> >> >>> >> >> >> >>> >> Hi, >> >> >>> >> >> >> >>> >> On Fri, Oct 28, 2011 at 4:21 PM, Ralf Gommers >> >> >>> >> wrote: >> >> >>> >> > >> >> >>> >> > >> >> >>> >> > On Sat, Oct 29, 2011 at 12:37 AM, Matthew Brett >> >> >>> >> > >> >> >>> >> > wrote: >> >> >>> >> >> >> >> >>> >> >> Hi, >> >> >>> >> >> >> >> >>> >> >> On Fri, Oct 28, 2011 at 3:14 PM, Charles R Harris >> >> >>> >> >> wrote: >> >> >>> >> >> >> >> >> >>> >> >> >> >> >>> >> >> No, that's not what Nathaniel and I are saying at all. >> >> >>> >> >> Nathaniel >> >> >>> >> >> was >> >> >>> >> >> pointing to links for projects that care that everyone agrees >> >> >>> >> >> before >> >> >>> >> >> they go ahead. >> >> >>> >> > >> >> >>> >> > It looked to me like there was a serious intent to come to an >> >> >>> >> > agreement, >> >> >>> >> > or >> >> >>> >> > at least closer together. The discussion in the summer was >> >> >>> >> > going >> >> >>> >> > around >> >> >>> >> > in >> >> >>> >> > circles though, and was too abstract and complex to follow. >> >> >>> >> > Therefore >> >> >>> >> > Mark's >> >> >>> >> > choice of implementing something and then asking for feedback >> >> >>> >> > made >> >> >>> >> > sense >> >> >>> >> > to >> >> >>> >> > me. >> >> >>> >> >> >> >>> >> I should point out that the implementation hasn't - as far as I >> >> >>> >> can >> >> >>> >> see - changed the discussion. The discussion was about the API. >> >> >>> >> >> >> >>> >> Implementations are useful for agreed APIs because they can >> >> >>> >> point >> >> >>> >> out >> >> >>> >> where the API does not make sense or cannot be implemented. In >> >> >>> >> this >> >> >>> >> case, the API Mark said he was going to implement - he did >> >> >>> >> implement - >> >> >>> >> at least as far as I can see. Again, I'm happy to be corrected. >> >> >>> > >> >> >>> > Implementations can also help the discussion along, by allowing >> >> >>> > people >> >> >>> > to >> >> >>> > try out some of the proposed changes. It also allows to construct >> >> >>> > examples >> >> >>> > that show weaknesses, possibly to be solved by an alternative >> >> >>> > API. >> >> >>> > Maybe >> >> >>> > you >> >> >>> > can hold the complete history of this topic in your head and >> >> >>> > comprehend >> >> >>> > it, >> >> >>> > but for me it would be very helpful if someone said: >> >> >>> > - here's my dataset >> >> >>> > - this is what I want to do with it >> >> >>> > - this is the best I can do with the current implementation >> >> >>> > - here's how API X would allow me to solve this better or simpler >> >> >>> > This can be done much better with actual data and an actual >> >> >>> > implementation >> >> >>> > than with a design proposal. You seem to disagree with this >> >> >>> > statement. >> >> >>> > That's fine. I would hope though that you recognize that concrete >> >> >>> > examples >> >> >>> > help people like me, and construct one or two to help us out. >> >> >>> That's what use-cases are for in designing APIs. There are >> >> >>> examples >> >> >>> of use in the NEP: >> >> >>> >> >> >>> >> >> >>> https://github.com/numpy/numpy/blob/master/doc/neps/missing-data.rst >> >> >>> >> >> >>> the alterNEP: >> >> >>> >> >> >>> https://gist.github.com/1056379 >> >> >>> >> >> >>> and my longer email to Travis: >> >> >>> >> >> >>> >> >> >>> >> >> >>> >> >> >>> http://article.gmane.org/gmane.comp.python.numeric.general/46544/match=ignored >> >> >>> >> >> >>> Mark has done a nice job of documentation: >> >> >>> >> >> >>> http://docs.scipy.org/doc/numpy/reference/arrays.maskna.html >> >> >>> >> >> >>> If you want to understand what the alterNEP case is, I'd suggest >> >> >>> the >> >> >>> email, just because it's the most recent and I think the >> >> >>> terminology >> >> >>> is slightly clearer. >> >> >>> >> >> >>> Doing the same examples on a larger array won't make the point >> >> >>> easier >> >> >>> to understand. The discussion is about what the right concepts >> >> >>> are, >> >> >>> and you can help by looking at the snippets of code in those >> >> >>> documents, and deciding for yourself whether you think the current >> >> >>> masking / NA implementation seems natural and easy to explain, or >> >> >>> rather forced and d
Re: [Numpy-discussion] consensus
Hi, On Sun, Oct 30, 2011 at 11:37 AM, Chris Barker wrote: > On 10/29/11 2:59 PM, Charles R Harris wrote: > >> I'm much opposed to ripping the current code out. It isn't like it is >> (known to be) buggy, nor has anyone made the case that it isn't a basis >> on which build other options. It also smacks of gratuitous violence >> committed by someone yet to make a positive contribution to the project. > > 1) contributing to the discussion IS a positive contribution to the project. Yes, but, personally I'd rather the discussion was not about who was saying something, but what they were saying. That is, if someone proposes something, or offers a discussion, we don't first ask 'who are you', but try and engage with the substance of the argument. Best, Matthew ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] consensus (was: NA masks in the next numpy release?)
On Sat, Oct 29, 2011 at 11:55 PM, Matthew Brett wrote: > Hi, > > On Sat, Oct 29, 2011 at 2:48 PM, Ralf Gommers > wrote: > > > > > > On Sat, Oct 29, 2011 at 11:36 PM, Matthew Brett > > > wrote: > >> > >> Hi, > >> > >> On Sat, Oct 29, 2011 at 1:48 PM, Matthew Brett > > >> wrote: > >> > Hi, > >> > > >> > On Sat, Oct 29, 2011 at 1:44 PM, Ralf Gommers > >> > wrote: > >> >> > >> >> > >> >> On Sat, Oct 29, 2011 at 9:04 PM, Matthew Brett > >> >> > >> >> wrote: > >> >>> > >> >>> Hi, > >> >>> > >> >>> On Sat, Oct 29, 2011 at 3:26 AM, Ralf Gommers > >> >>> wrote: > >> >>> > > >> >>> > > >> >>> > On Sat, Oct 29, 2011 at 1:37 AM, Matthew Brett > >> >>> > > >> >>> > wrote: > >> >>> >> > >> >>> >> Hi, > >> >>> >> > >> >>> >> On Fri, Oct 28, 2011 at 4:21 PM, Ralf Gommers > >> >>> >> wrote: > >> >>> >> > > >> >>> >> > > >> >>> >> > On Sat, Oct 29, 2011 at 12:37 AM, Matthew Brett > >> >>> >> > > >> >>> >> > wrote: > >> >>> >> >> > >> >>> >> >> Hi, > >> >>> >> >> > >> >>> >> >> On Fri, Oct 28, 2011 at 3:14 PM, Charles R Harris > >> >>> >> >> wrote: > >> >>> >> >> >> > >> >>> >> >> > >> >>> >> >> No, that's not what Nathaniel and I are saying at all. > Nathaniel > >> >>> >> >> was > >> >>> >> >> pointing to links for projects that care that everyone agrees > >> >>> >> >> before > >> >>> >> >> they go ahead. > >> >>> >> > > >> >>> >> > It looked to me like there was a serious intent to come to an > >> >>> >> > agreement, > >> >>> >> > or > >> >>> >> > at least closer together. The discussion in the summer was > going > >> >>> >> > around > >> >>> >> > in > >> >>> >> > circles though, and was too abstract and complex to follow. > >> >>> >> > Therefore > >> >>> >> > Mark's > >> >>> >> > choice of implementing something and then asking for feedback > >> >>> >> > made > >> >>> >> > sense > >> >>> >> > to > >> >>> >> > me. > >> >>> >> > >> >>> >> I should point out that the implementation hasn't - as far as I > can > >> >>> >> see - changed the discussion. The discussion was about the API. > >> >>> >> > >> >>> >> Implementations are useful for agreed APIs because they can point > >> >>> >> out > >> >>> >> where the API does not make sense or cannot be implemented. In > >> >>> >> this > >> >>> >> case, the API Mark said he was going to implement - he did > >> >>> >> implement - > >> >>> >> at least as far as I can see. Again, I'm happy to be corrected. > >> >>> > > >> >>> > Implementations can also help the discussion along, by allowing > >> >>> > people > >> >>> > to > >> >>> > try out some of the proposed changes. It also allows to construct > >> >>> > examples > >> >>> > that show weaknesses, possibly to be solved by an alternative API. > >> >>> > Maybe > >> >>> > you > >> >>> > can hold the complete history of this topic in your head and > >> >>> > comprehend > >> >>> > it, > >> >>> > but for me it would be very helpful if someone said: > >> >>> > - here's my dataset > >> >>> > - this is what I want to do with it > >> >>> > - this is the best I can do with the current implementation > >> >>> > - here's how API X would allow me to solve this better or simpler > >> >>> > This can be done much better with actual data and an actual > >> >>> > implementation > >> >>> > than with a design proposal. You seem to disagree with this > >> >>> > statement. > >> >>> > That's fine. I would hope though that you recognize that concrete > >> >>> > examples > >> >>> > help people like me, and construct one or two to help us out. > >> >>> That's what use-cases are for in designing APIs. There are examples > >> >>> of use in the NEP: > >> >>> > >> >>> > https://github.com/numpy/numpy/blob/master/doc/neps/missing-data.rst > >> >>> > >> >>> the alterNEP: > >> >>> > >> >>> https://gist.github.com/1056379 > >> >>> > >> >>> and my longer email to Travis: > >> >>> > >> >>> > >> >>> > >> >>> > http://article.gmane.org/gmane.comp.python.numeric.general/46544/match=ignored > >> >>> > >> >>> Mark has done a nice job of documentation: > >> >>> > >> >>> http://docs.scipy.org/doc/numpy/reference/arrays.maskna.html > >> >>> > >> >>> If you want to understand what the alterNEP case is, I'd suggest the > >> >>> email, just because it's the most recent and I think the terminology > >> >>> is slightly clearer. > >> >>> > >> >>> Doing the same examples on a larger array won't make the point > easier > >> >>> to understand. The discussion is about what the right concepts are, > >> >>> and you can help by looking at the snippets of code in those > >> >>> documents, and deciding for yourself whether you think the current > >> >>> masking / NA implementation seems natural and easy to explain, or > >> >>> rather forced and difficult to explain, and then email back trying > to > >> >>> explain your impression (which is not always easy). > >> >> > >> >> If you seriously believe that looking at a few snippets is as helpful > >> >> and > >> >> instructive as being able to play around with them in IPython and > >> >> modify > >> >> them, then I guess we won't m
Re: [Numpy-discussion] consensus
On 10/29/11 2:59 PM, Charles R Harris wrote: > I'm much opposed to ripping the current code out. It isn't like it is > (known to be) buggy, nor has anyone made the case that it isn't a basis > on which build other options. It also smacks of gratuitous violence > committed by someone yet to make a positive contribution to the project. 1) contributing to the discussion IS a positive contribution to the project. 2) If we use the term "ripping out" it does "smacks of gratuitous violence" -- if we use the term "roll back", maybe not so much -- it's not like the code couldn't be put back in. That being said, I like the idea of it being easy and accessible for not-very-familiar-with-git folks to test -- so I'd like to see it left there for now at least. On 10/29/11 3:47 PM, Eric Firing wrote: > Similarly, in Marks implementation, 7 bits are available for a payload > to describe what kind of masking is meant. This seems more consistent > with True as masked (or NA) than with False as masked. +1 -- we've got 8 bits, nice to be able to use them On 10/29/11 3:57 PM, Charles R Harris wrote: > I wouldn't rely on the 7 bits yet. Mark left them available to keep open > possible future use, but didn't implement anything using them yet. If > memory use turns out to exclude whole sectors of application we will > have to go to bit masks. would there have to be only one type of mask available? -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R(206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception chris.bar...@noaa.gov ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] consensus
On 10/29/11 2:48 PM, Ralf Gommers wrote: > That's true, but I am hoping that the difference between - say: > > a[0:2] = np.NA > > and > > a.mask[0:2] = False > > would be easy enough to imagine. > > > It is in this case. I agree the explicit ``a.mask`` is clearer. Interesting -- I suspect I'm mirror's Pandas' users here: a[0:2] = np.NA is simpler and easier to me -- I'm avoiding the word "clearer" because I m not sure what it means -- if we thin it's important for the user to understand that the NA value is implemented with a mask, then setting the mask explicitly is certainly clearer -- but I don't think that's important. Indeed, I still like the idea that for "casual" use, NA could be a special value, and could be a mask, and that the user does not need to know the difference. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R(206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception chris.bar...@noaa.gov ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Is distributing GPL + exception dll in the windows installer ok
On Sun, Oct 30, 2011 at 11:38 AM, Matthieu Brucher wrote: > Hi David, > > Is every GPL part GCC related? If yes, GCC has a licence that allows to > redistribute its runtime in any program (meaning the program's licence is > not relevant). Good point, I should have specified the dll in question: - libgcc - libstdc++ - libgfortran As far as I know, all those fall under the license you mention (GPL + exception), but it was not entirely clear to me. cheers, David >> >> While testing the mingw gcc 3.x -> 4.x migration, I realized that some >> technical requirements in gcc 4.x have potential license implications. >> In short, it is more difficult now than before to statically link >> gcc-related runtimes into numpy/scipy. I think using the DLL is safer >> and better, but it means the windows installers will contain GPL code. >> My understanding is that this is OK because the code in question is >> GPL + exception, meaning the usual GPL requirements only apply to >> those runtimes, and that's ok ? >> >> cheers, >> >> David >> ___ >> NumPy-Discussion mailing list >> NumPy-Discussion@scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > > -- > Information System Engineer, Ph.D. > Blog: http://matt.eifelle.com > LinkedIn: http://www.linkedin.com/in/matthieubrucher > > ___ > NumPy-Discussion mailing list > NumPy-Discussion@scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Is distributing GPL + exception dll in the windows installer ok
Hi David, Is every GPL part GCC related? If yes, GCC has a licence that allows to redistribute its runtime in any program (meaning the program's licence is not relevant). Cheers, Matthieu 2011/10/30 David Cournapeau > Hi, > > While testing the mingw gcc 3.x -> 4.x migration, I realized that some > technical requirements in gcc 4.x have potential license implications. > In short, it is more difficult now than before to statically link > gcc-related runtimes into numpy/scipy. I think using the DLL is safer > and better, but it means the windows installers will contain GPL code. > My understanding is that this is OK because the code in question is > GPL + exception, meaning the usual GPL requirements only apply to > those runtimes, and that's ok ? > > cheers, > > David > ___ > NumPy-Discussion mailing list > NumPy-Discussion@scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -- Information System Engineer, Ph.D. Blog: http://matt.eifelle.com LinkedIn: http://www.linkedin.com/in/matthieubrucher ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] Is distributing GPL + exception dll in the windows installer ok
Hi, While testing the mingw gcc 3.x -> 4.x migration, I realized that some technical requirements in gcc 4.x have potential license implications. In short, it is more difficult now than before to statically link gcc-related runtimes into numpy/scipy. I think using the DLL is safer and better, but it means the windows installers will contain GPL code. My understanding is that this is OK because the code in question is GPL + exception, meaning the usual GPL requirements only apply to those runtimes, and that's ok ? cheers, David ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Moving to gcc 4.* for win32 installers ?
On Thu, Oct 27, 2011 at 5:19 PM, Ralf Gommers wrote: > Hi David, > > On Thu, Oct 27, 2011 at 3:02 PM, David Cournapeau > wrote: >> >> Hi, >> >> I was wondering if we could finally move to a more recent version of >> compilers for official win32 installers. This would of course concern >> the next release cycle, not the ones where beta/rc are already in >> progress. >> >> Basically, the pros: >> - we will have to move at some point >> - gcc 4.* seem less buggy, especially C++ and fortran. >> - no need to maintain msvcr90 vodoo >> The cons: >> - it will most likely break the ABI >> - we need to recompile atlas (but I can take care of it) >> - the biggest: it is difficult to combine gfortran with visual >> studio (more exactly you cannot link gfortran runtime to a visual >> studio executable). The only solution I could think of would be to >> recompile the gfortran runtime with Visual Studio, which for some >> reason does not sound very appealing :) > > To get the datetime changes to work with MinGW, we already concluded that > building with 4.x is more or less required (without recognizing some of the > points you list above). Changes to mingw32ccompiler to fix compilation with > 4.x went in in https://github.com/numpy/numpy/pull/156. It would be good if > you could check those. I will look into it more carefully, but overall, it seems that building atlas 3.8.4, numpy and scipy with gcc 4.x works quite well. The main issue is that gcc 4.* adds some dependencies on mingw dlls. There are two options: - adding the dlls in the installers - statically linking those, which seems to be a bad idea (generalizing the dll boundaries problem to exception and things we would rather not care about: http://cygwin.com/ml/cygwin/2007-06/msg00332.html). > It probably makes sense make this move for numpy 1.7. If this breaks the ABI > then it would be easiest to make numpy 1.7 the minimum required version for > scipy 0.11. My thinking as well. cheers, David ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Large numbers into float128
Hi, On Sun, Oct 30, 2011 at 2:38 AM, Berthold Höllmann wrote: > Matthew Brett writes: > >> Hi, >> >> Can anyone think of a good way to set a float128 value to an >> arbitrarily large number? >> >> As in >> >> v = int_to_float128(some_value) >> >> ? >> >> I'm trying things like >> >> v = np.float128(2**64+2) >> >> but, because (in other threads) the float128 seems to be going through >> float64 on assignment, this loses precision, so although 2**64+2 is >> representable in float128, in fact I get: >> >> In [35]: np.float128(2**64+2) >> Out[35]: 18446744073709551616.0 >> >> In [36]: 2**64+2 >> Out[36]: 18446744073709551618L >> >> So - can anyone think of another way to assign values to float128 that >> will keep the precision? > > Just use float128 all the was through, and avoid casting to float in > between: > > .>>> "%20.1f"%float(2**64+2) > '18446744073709551616.0' > .>>> np.float128(np.float128(2)**64+2) > 18446744073709551618.0 Ah yes - sorry - that would work in this example where I know the component parts of the number, but I was thinking in the general case where I have been given any int. I think my code works for that, by casting to float64 to break up the number into parts: In [35]: def int_to_float128(val): :f64 = np.float64(val) :res = val - int(f64) :return np.float128(f64) + np.float128(res) : In [36]: int_to_float128(2**64) Out[36]: 18446744073709551616.0 In [37]: int_to_float128(2**64+2) Out[37]: 18446744073709551618.0 Thanks, Matthew ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Large numbers into float128
Matthew Brett writes: > Hi, > > Can anyone think of a good way to set a float128 value to an > arbitrarily large number? > > As in > > v = int_to_float128(some_value) > > ? > > I'm trying things like > > v = np.float128(2**64+2) > > but, because (in other threads) the float128 seems to be going through > float64 on assignment, this loses precision, so although 2**64+2 is > representable in float128, in fact I get: > > In [35]: np.float128(2**64+2) > Out[35]: 18446744073709551616.0 > > In [36]: 2**64+2 > Out[36]: 18446744073709551618L > > So - can anyone think of another way to assign values to float128 that > will keep the precision? Just use float128 all the was through, and avoid casting to float in between: .>>> "%20.1f"%float(2**64+2) '18446744073709551616.0' .>>> np.float128(np.float128(2)**64+2) 18446744073709551618.0 Regards Berthold > > Thanks a lot, > > Matthew -- A: Weil es die Lesbarkeit des Textes verschlechtert. F: Warum ist TOFU so schlimm? A: TOFU F: Was ist das größte Ärgernis im Usenet? pgpgCtTQeLJUE.pgp Description: PGP signature ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Large numbers into float128
A quick and dirty cython code is attached
Use:
>> import Float128
>> a = Float128.Float128('1E500')
array([ 1e+500], dtype=float128)
or
>> b = np.float128(1.34) * np.float128(10)**2500
>> b
1.3400779e+2500
Maybe there is also a way to do it in a pure python code via ctypes?
Nadav
From: numpy-discussion-boun...@scipy.org [numpy-discussion-boun...@scipy.org]
On Behalf Of Charles R Harris [charlesr.har...@gmail.com]
Sent: 30 October 2011 05:02
To: Discussion of Numerical Python
Subject: Re: [Numpy-discussion] Large numbers into float128
On Sat, Oct 29, 2011 at 8:49 PM, Matthew Brett
mailto:matthew.br...@gmail.com>> wrote:
Hi,
On Sat, Oct 29, 2011 at 3:55 PM, Matthew Brett
mailto:matthew.br...@gmail.com>> wrote:
> Hi,
>
> Can anyone think of a good way to set a float128 value to an
> arbitrarily large number?
>
> As in
>
> v = int_to_float128(some_value)
>
> ?
>
> I'm trying things like
>
> v = np.float128(2**64+2)
>
> but, because (in other threads) the float128 seems to be going through
> float64 on assignment, this loses precision, so although 2**64+2 is
> representable in float128, in fact I get:
>
> In [35]: np.float128(2**64+2)
> Out[35]: 18446744073709551616.0
>
> In [36]: 2**64+2
> Out[36]: 18446744073709551618L
>
> So - can anyone think of another way to assign values to float128 that
> will keep the precision?
To answer my own question - I found an unpleasant way of doing this.
Basically it is this:
def int_to_float128(val):
f64 = np.float64(val)
res = val - int(f64)
return np.float128(f64) + np.float128(res)
Used in various places here:
https://github.com/matthew-brett/nibabel/blob/e18e94c5b0f54775c46b1c690491b8bd6f07eb49/nibabel/floating.py
Best,
It might be useful to look into mpmath. I didn't see any way to export mp
values into long double, but they do offer a number of resources for working
with arbitrary precision. We could maybe even borrow some of their stuff for
parsing values from strings
Chuck
from cython import *
import numpy as np
cimport numpy as np
cdef extern from "stdlib.h":
long double strtold(char* number, char** endptr)
def Float128( char *number):
cdef long double num
cdef np.ndarray[np.longdouble_t, ndim=1] output = np.empty(shape=1,
dtype=np.float128)
num = strtold(number, NULL)
output[0] = num
return output
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] consensus (was: NA masks in the next numpy release?)
Hi, On Sat, Oct 29, 2011 at 11:19 PM, Travis Oliphant wrote: Thanks again for your email, I'm sure I'm not the only one who breathes a deep sigh of relief when I see your posts. > I appreciate Nathaniel's idea to pull the changes and I can respect his > desire to do that. It seemed like there was a lot more heat than light in > the discussion this summer. The differences seemed to be enflamed by the > discussion instead of illuminated by it. Perhaps, that is why Nathaniel felt > like merging Mark's pull request was too strong-armed and not a proper > resolution. > > However, I did not interpret Matthew or Nathaniel's explanations of their > position as manipulative or inappropriate. Nonetheless, I don't think > removing Mark's changes are a productive direction to take at this point. I > agree, it would have been much better to reach a rough consensus before the > code was committed. At least, those who felt like their ideas where not > accounted for should have felt like there was some plan to either accommodate > them, or some explanation of why that was not a good idea. The only thing I > recall being said was that there was nobody to implement their ideas. I > wish that weren't the case. I think we can still continue to discuss their > concerns and look for ways to reasonably incorporate their use-cases if > possible. > > I have probably contributed in the past to the idea that "he who writes the > code gets the final say". In early-stage efforts, this is approximately > right, but success of anything relies on satisfied users and as projects > mature the voice of users becomes more relevant than the voice of > contributors in my mind. I've certainly had to learn that in terms of ABI > changes to NumPy. I think that's right though - that the person who wrote the code has the final say. But that's the final say. The question I wanted to ask was the one Nathaniel brought up at the beginning of the thread, which is, before the final say, how hard do we try for consensus? Is that - the numpy way? Here Chuck was saying 'I listen to you in proportion to your code contribution' (I hope I'm not misrepresenting him). I think that's different way of working than the consensus building that Karl Fogel describes. But maybe that is just the numpy way. I would feel happier to know what that way is. Then, when we get into this kind of dispute Chuck can say 'Matthew, change the numpy constitution or accept the situation because that's how we've agreed to work'. And I'll say - 'OK - I don't like it, but I agree those are the rules'. And we'll get on with it. But at the moment, it feels as if it isn't clear, and, as Ben pointed out, that means we are having a discussion and a discussion about the discussion at the same time. See you, Matthew ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
