Re: [Numpy-discussion] performance matrix multiplication vs. matlab
2010/1/17 David Cournapeau : > On Sun, Jan 17, 2010 at 2:20 PM, Benoit Jacob > wrote: > >> Couldn't you simply: >> - either add LGPL-licensed code to a third_party subdirectory not >> subject to the NumPy license, and just use it? This is common >> practice, see e.g. how Qt puts a copy of WebKit in a third_party >> subdirectory. >> - or use LGPL-licensed code as an external dependency? > Thanks for the reply! First of all I should say that I was only talking about the raised licensing issue, I'm not saying that you _should_ use eigen from a technical point of view. > There are several issues with eigen2 for NumPy usage: > - using it as a default implementation does not make much sense IMHO, > as it would make distributed binaries non 100 % BSD. But the LGPL doesn't impose restrictions on the usage of binaries, so how does it matter? The LGPL and the BSD licenses are similar as far as the binaries are concerned (unless perhaps one starts disassembling them). The big difference between LGPL and BSD is at the level of source code, not binary code: one modifies LGPL-based source code and distributes a binary form of it, then one has to release the modified source code as well. Since NumPy's users are presumably not interested in modifying _Eigen_ itself, I don't think that matters. I understand that they may want to modify NumPy's source code without releasing their modified source code, so the BSD license is important for NumPy, but having Eigen in a third_party directory wouldn't affect that. > - to my knowledge, eigen2 does not have a BLAS API, so we would have > to write specific wrappers for eigen2, which is undesirable. That's true. FYI, a BLAS API is coming in Eigen 3, https://bitbucket.org/eigen/eigen/src/tip/blas/ > - eigen2 is C++, and it is a stated goal to make numpy depend only on > a C compiler (it may optionally uses fortran to link against > blas/lapack, though). Ah OK. Well, once the Eigen BLAS is implemented, it will be usable by a C compiler. > As I see it, people would be able to easily use eigen2 if there was a > BLAS API for it. We still would not distribute binaries built with > eigen2, but it means people who don't care about using GPL code could > use it. I see. I'd quite like to see this happening! Maybe, just give a look at where Eigen is in 1 year from now, the BLAS should be ready for that. > > Independently of NumPy, I think a BLAS API for eigen2 would be very > beneficial for eigen2 if you care about the numerical scientific > community. So do we, that's why we're doing it ;) see above. Benoit ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] Structured array sorting
I am having trouble sorting a structured array - in the example below, sorting by the first column (col1) seems to work, but not sorting by the second column (col2). Is this a bug? I am using numpy svn r8071 on MacOS 10.6. Thanks for any help, Thomas Python 2.6.1 (r261:67515, Jul 7 2009, 23:51:51) [GCC 4.2.1 (Apple Inc. build 5646)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> import numpy as np >>> data = np.array([('a ', 2.), ('b', 4.), ('d', 3.), ('c', 1.)], ... dtype=[('col1', '|S5'), ('col2', '>f8')]) >>> >>> data array([('a ', 2.0), ('b', 4.0), ('d', 3.0), ('c', 1.0)], dtype=[('col1', '|S5'), ('col2', '>f8')]) >>> data.sort(order=['col1']) >>> data array([('a ', 2.0), ('b', 4.0), ('c', 1.0), ('d', 3.0)], dtype=[('col1', '|S5'), ('col2', '>f8')]) >>> data.sort(order=['col2']) >>> data array([('a ', 2.0), ('d', 3.0), ('b', 4.0), ('c', 1.0)], dtype=[('col1', '|S5'), ('col2', '>f8')]) ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] performance matrix multiplication vs. matlab
On Sun, Jan 17, 2010 at 08:52, Benoit Jacob wrote: > 2010/1/17 David Cournapeau : >> There are several issues with eigen2 for NumPy usage: >> - using it as a default implementation does not make much sense IMHO, >> as it would make distributed binaries non 100 % BSD. > > But the LGPL doesn't impose restrictions on the usage of binaries, so > how does it matter? The LGPL and the BSD licenses are similar as far > as the binaries are concerned (unless perhaps one starts disassembling > them). > > The big difference between LGPL and BSD is at the level of source > code, not binary code: one modifies LGPL-based source code and > distributes a binary form of it, then one has to release the modified > source code as well. This is not true. Binaries that contain LGPLed code must be able to be relinked with a modified version of the LGPLed component. This is technically non-trivial. In addition, binaries containing an LGPLed component must still come with the source of the LGPLed component (or come with a written offer to distribute via the same mechanism ... yada yada yada). These are non-trivial restrictions above and beyond the BSD license that we, as a matter of policy, do not wish to impose on numpy users. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] Module Index for numpy?
I was just looking at the (Win) Python documentation via the Help on IDLE, and a Global Module Index. Does anything like that exist for numpy, matplotlib, scipy? -- Wayne Watson (Watson Adventures, Prop., Nevada City, CA) (121.015 Deg. W, 39.262 Deg. N) GMT-8 hr std. time) Obz Site: 39° 15' 7" N, 121° 2' 32" W, 2700 feet "I was thinking about how people seem to read the Bible a whole lot more as they get older; then it dawned on me . . they're cramming for their final exam." -- George Carlin Web Page: ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] performance matrix multiplication vs. matlab
2010/1/17 Robert Kern : > On Sun, Jan 17, 2010 at 08:52, Benoit Jacob wrote: >> 2010/1/17 David Cournapeau : > >>> There are several issues with eigen2 for NumPy usage: >>> - using it as a default implementation does not make much sense IMHO, >>> as it would make distributed binaries non 100 % BSD. >> >> But the LGPL doesn't impose restrictions on the usage of binaries, so >> how does it matter? The LGPL and the BSD licenses are similar as far >> as the binaries are concerned (unless perhaps one starts disassembling >> them). >> >> The big difference between LGPL and BSD is at the level of source >> code, not binary code: one modifies LGPL-based source code and >> distributes a binary form of it, then one has to release the modified >> source code as well. > > This is not true. Binaries that contain LGPLed code must be able to be > relinked with a modified version of the LGPLed component. This doesn't apply to Eigen which is a header-only pure template library, hence can't be 'linked' to. Actually you seem to be referring to Section 4 of the LGPL3, we have already asked the FSF about this and their reply was that it just doesn't apply in the case of Eigen: http://listengine.tuxfamily.org/lists.tuxfamily.org/eigen/2009/01/msg00083.html In your case, what matters is Section 5. >In addition, binaries containing an LGPLed > component must still come with the source of the LGPLed component (or > come with a written offer to distribute via the same mechanism ... > yada yada yada). Since you would presumably be using vanilla Eigen without changes of your own, it would be enough to just give the link to the Eigen website, that's all. Just one line, and it doesn't have to be in a very prominent place, it just has to be reasonably easy to find for someone looking for it. > These are non-trivial restrictions above and beyond > the BSD license that we, as a matter of policy, do not wish to impose > on numpy users. The only thing you'd be imposing on NumPy users would be that somewhere at the bottom of, say, your README file, there would be a link to Eigen's website. Then who am I to discuss your policies ;) Finally let me just give an example why this is moot. You are using GCC, right? So you use the GNU libc (their standard C library)? It is LGPL ;) It's just that nobody cares to put a link to the GNU libc homepage, which is understandable ;) Cheers, Benoit ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Structured array sorting
Looks like 'sort' is not handling endianess of the column data correctly. If you change the type of the floating point data to 'i2')]) In [141]: z.sort(order='num') In [142]: z Out[142]: array([(255,), (0,), (256,), (1,), (258,)], dtype=[('num', '>i2')]) In [143]: np.__version__ Out[143]: '1.3.0' - Sorting works as expected with a simple array of short ints: - In [152]: w = np.array([0,258, 3, 255], dtype=' I am having trouble sorting a structured array - in the example below, > sorting by the first column (col1) seems to work, but not sorting by the > second column (col2). Is this a bug? > > I am using numpy svn r8071 on MacOS 10.6. > > Thanks for any help, > > Thomas > > Python 2.6.1 (r261:67515, Jul 7 2009, 23:51:51) > [GCC 4.2.1 (Apple Inc. build 5646)] on darwin > Type "help", "copyright", "credits" or "license" for more information. > import numpy as np data = np.array([('a ', 2.), ('b', 4.), ('d', 3.), ('c', 1.)], > ... dtype=[('col1', '|S5'), ('col2', '>f8')]) > data > array([('a ', 2.0), ('b', 4.0), ('d', 3.0), ('c', 1.0)], > dtype=[('col1', '|S5'), ('col2', '>f8')]) > data.sort(order=['col1']) data > array([('a ', 2.0), ('b', 4.0), ('c', 1.0), ('d', 3.0)], > dtype=[('col1', '|S5'), ('col2', '>f8')]) > data.sort(order=['col2']) data > array([('a ', 2.0), ('d', 3.0), ('b', 4.0), ('c', 1.0)], > dtype=[('col1', '|S5'), ('col2', '>f8')]) > > > ___ > NumPy-Discussion mailing list > NumPy-Discussion@scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] performance matrix multiplication vs. matlab
On Sun, Jan 17, 2010 at 12:11, Benoit Jacob wrote: > 2010/1/17 Robert Kern : >> On Sun, Jan 17, 2010 at 08:52, Benoit Jacob wrote: >>> 2010/1/17 David Cournapeau : >> There are several issues with eigen2 for NumPy usage: - using it as a default implementation does not make much sense IMHO, as it would make distributed binaries non 100 % BSD. >>> >>> But the LGPL doesn't impose restrictions on the usage of binaries, so >>> how does it matter? The LGPL and the BSD licenses are similar as far >>> as the binaries are concerned (unless perhaps one starts disassembling >>> them). >>> >>> The big difference between LGPL and BSD is at the level of source >>> code, not binary code: one modifies LGPL-based source code and >>> distributes a binary form of it, then one has to release the modified >>> source code as well. >> >> This is not true. Binaries that contain LGPLed code must be able to be >> relinked with a modified version of the LGPLed component. > > This doesn't apply to Eigen which is a header-only pure template > library, hence can't be 'linked' to. > > Actually you seem to be referring to Section 4 of the LGPL3, we have > already asked the FSF about this and their reply was that it just > doesn't apply in the case of Eigen: > > http://listengine.tuxfamily.org/lists.tuxfamily.org/eigen/2009/01/msg00083.html > > In your case, what matters is Section 5. You mean Section 3. Good. I admit to being less up on the details of LGPLv3 than I was of LGPLv2 which had a problem with C++ header templates. That said, we will not be using the C++ templates directly in numpy for technical reasons (not least that we do not want to require a C++ compiler for the default build). At best, we would be using a BLAS interface which requires linking of objects, not just header templates. That *would* impose the Section 4 requirements. Furthermore, we would still prefer not to have any LGPL code in the official numpy sources or binaries, regardless of how minimal the real requirements are. Licensing is confusing enough that being able to say "numpy is BSD licensed" without qualification is quite important. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] performance matrix multiplication vs. matlab
2010/1/17 Robert Kern : > On Sun, Jan 17, 2010 at 12:11, Benoit Jacob wrote: >> 2010/1/17 Robert Kern : >>> On Sun, Jan 17, 2010 at 08:52, Benoit Jacob >>> wrote: 2010/1/17 David Cournapeau : >>> > There are several issues with eigen2 for NumPy usage: > - using it as a default implementation does not make much sense IMHO, > as it would make distributed binaries non 100 % BSD. But the LGPL doesn't impose restrictions on the usage of binaries, so how does it matter? The LGPL and the BSD licenses are similar as far as the binaries are concerned (unless perhaps one starts disassembling them). The big difference between LGPL and BSD is at the level of source code, not binary code: one modifies LGPL-based source code and distributes a binary form of it, then one has to release the modified source code as well. >>> >>> This is not true. Binaries that contain LGPLed code must be able to be >>> relinked with a modified version of the LGPLed component. >> >> This doesn't apply to Eigen which is a header-only pure template >> library, hence can't be 'linked' to. >> >> Actually you seem to be referring to Section 4 of the LGPL3, we have >> already asked the FSF about this and their reply was that it just >> doesn't apply in the case of Eigen: >> >> http://listengine.tuxfamily.org/lists.tuxfamily.org/eigen/2009/01/msg00083.html >> >> In your case, what matters is Section 5. > > You mean Section 3. Good. Section 3 is for using Eigen directly in a C++ program, yes, but I got a bit ahead of myself there: see below > I admit to being less up on the details of > LGPLv3 than I was of LGPLv2 which had a problem with C++ header > templates. Indeed, it did, that's why we don't use it. > > That said, we will not be using the C++ templates directly in numpy > for technical reasons (not least that we do not want to require a C++ > compiler for the default build). At best, we would be using a BLAS > interface which requires linking of objects, not just header > templates. That *would* impose the Section 4 requirements. ... or rather Section 5: that is what I was having in mind: " 5. Combined Libraries. " I have to admit that I don't understand what 5.a) means. > Furthermore, we would still prefer not to have any LGPL code in the > official numpy sources or binaries, regardless of how minimal the real > requirements are. Licensing is confusing enough that being able to say > "numpy is BSD licensed" without qualification is quite important. I hear you, in the same way we definitely care about being able to say "Eigen is LGPL licensed". So it's a hard problem. I think that this is the only real issue here, but I definitely agree that it is a real one. Large projects (such as Qt) that have a third_party subdirectory have to find a wording to explain that their license doesn't cover it. Benoit > > -- > Robert Kern > > "I have come to believe that the whole world is an enigma, a harmless > enigma that is made terrible by our own mad attempt to interpret it as > though it had an underlying truth." > -- Umberto Eco > ___ > NumPy-Discussion mailing list > NumPy-Discussion@scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Module Index for numpy?
Hi, Wayne. They're not nearly as structured, but for the time being (indefinitely? unless a volunteer steps forward to build something for us more closely resembling the GMI), you could use the numpy and scipy doc Wiki Milestones pages: http://docs.scipy.org/numpy/Milestones/ http://docs.scipy.org/scipy/Milestones/ in this fashion. DG On Sun, Jan 17, 2010 at 9:57 AM, Wayne Watson wrote: > I was just looking at the (Win) Python documentation via the Help on > IDLE, and a Global Module Index. Does anything like that exist for > numpy, matplotlib, scipy? > > -- > Wayne Watson (Watson Adventures, Prop., Nevada City, CA) > > (121.015 Deg. W, 39.262 Deg. N) GMT-8 hr std. time) > Obz Site: 39° 15' 7" N, 121° 2' 32" W, 2700 feet > > "I was thinking about how people seem to read the Bible > a whole lot more as they get older; then it dawned on > me . . they're cramming for their final exam." > -- George Carlin > > Web Page: > > ___ > NumPy-Discussion mailing list > NumPy-Discussion@scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Module Index for numpy?
On Sun, Jan 17, 2010 at 2:19 PM, David Goldsmith wrote: > Hi, Wayne. > > They're not nearly as structured, but for the time being > (indefinitely? unless a volunteer steps forward to build something for > us more closely resembling the GMI), you could use the numpy and scipy > doc Wiki Milestones pages: > > http://docs.scipy.org/numpy/Milestones/ > > http://docs.scipy.org/scipy/Milestones/ > > in this fashion. > > DG > > On Sun, Jan 17, 2010 at 9:57 AM, Wayne Watson > wrote: >> I was just looking at the (Win) Python documentation via the Help on >> IDLE, and a Global Module Index. Does anything like that exist for >> numpy, matplotlib, scipy? >> >> -- >> Wayne Watson (Watson Adventures, Prop., Nevada City, CA) >> >> (121.015 Deg. W, 39.262 Deg. N) GMT-8 hr std. time) >> Obz Site: 39° 15' 7" N, 121° 2' 32" W, 2700 feet >> >> "I was thinking about how people seem to read the Bible >> a whole lot more as they get older; then it dawned on >> me . . they're cramming for their final exam." >> -- George Carlin >> >> Web Page: >> >> ___ >> NumPy-Discussion mailing list >> NumPy-Discussion@scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> > ___ > NumPy-Discussion mailing list > NumPy-Discussion@scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > the autogenerated indices are here and also in the htmlhelp http://docs.scipy.org/doc/numpy/modindex.html http://docs.scipy.org/doc/numpy/genindex.html However, because of the package structure of numpy the modindex is not as useful as the one for python. I find the structure of routines, and the index search in the htmlhelp more useful. Josef ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] performance matrix multiplication vs. matlab
On Sun, Jan 17, 2010 at 13:18, Benoit Jacob wrote: > 2010/1/17 Robert Kern : >> On Sun, Jan 17, 2010 at 12:11, Benoit Jacob wrote: >>> 2010/1/17 Robert Kern : On Sun, Jan 17, 2010 at 08:52, Benoit Jacob wrote: > 2010/1/17 David Cournapeau : >> There are several issues with eigen2 for NumPy usage: >> - using it as a default implementation does not make much sense IMHO, >> as it would make distributed binaries non 100 % BSD. > > But the LGPL doesn't impose restrictions on the usage of binaries, so > how does it matter? The LGPL and the BSD licenses are similar as far > as the binaries are concerned (unless perhaps one starts disassembling > them). > > The big difference between LGPL and BSD is at the level of source > code, not binary code: one modifies LGPL-based source code and > distributes a binary form of it, then one has to release the modified > source code as well. This is not true. Binaries that contain LGPLed code must be able to be relinked with a modified version of the LGPLed component. >>> >>> This doesn't apply to Eigen which is a header-only pure template >>> library, hence can't be 'linked' to. >>> >>> Actually you seem to be referring to Section 4 of the LGPL3, we have >>> already asked the FSF about this and their reply was that it just >>> doesn't apply in the case of Eigen: >>> >>> http://listengine.tuxfamily.org/lists.tuxfamily.org/eigen/2009/01/msg00083.html >>> >>> In your case, what matters is Section 5. >> >> You mean Section 3. Good. > > Section 3 is for using Eigen directly in a C++ program, yes, but I got > a bit ahead of myself there: see below > >> I admit to being less up on the details of >> LGPLv3 than I was of LGPLv2 which had a problem with C++ header >> templates. > > Indeed, it did, that's why we don't use it. > >> >> That said, we will not be using the C++ templates directly in numpy >> for technical reasons (not least that we do not want to require a C++ >> compiler for the default build). At best, we would be using a BLAS >> interface which requires linking of objects, not just header >> templates. That *would* impose the Section 4 requirements. > > ... or rather Section 5: that is what I was having in mind: > " 5. Combined Libraries. " > > I have to admit that I don't understand what 5.a) means. I don't think it applies. Let's say I write some routines that use an LGPLed Library (let's call them Routines A). I can include those routines in a larger library with routines that do not use the LGPLed library (Routines B). The Routines B can be under whatever license you like. However, one must make a library containing only Routines A and the LGPLed Library and release that under the LGPLv3, distribute it along with the combined work, and give notice about how to obtain Routines A+Library separate from Routines B. Basically, it's another exception for needing to be able to relink object code in a particular technical use case. This cannot apply to numpy because we cannot break out numpy.linalg from the rest of numpy. Even if we could, we do not wish to make numpy.linalg itself LGPLed. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] performance matrix multiplication vs. matlab
2010/1/17 Robert Kern : > On Sun, Jan 17, 2010 at 13:18, Benoit Jacob wrote: >> 2010/1/17 Robert Kern : >>> On Sun, Jan 17, 2010 at 12:11, Benoit Jacob >>> wrote: 2010/1/17 Robert Kern : > On Sun, Jan 17, 2010 at 08:52, Benoit Jacob > wrote: >> 2010/1/17 David Cournapeau : > >>> There are several issues with eigen2 for NumPy usage: >>> - using it as a default implementation does not make much sense IMHO, >>> as it would make distributed binaries non 100 % BSD. >> >> But the LGPL doesn't impose restrictions on the usage of binaries, so >> how does it matter? The LGPL and the BSD licenses are similar as far >> as the binaries are concerned (unless perhaps one starts disassembling >> them). >> >> The big difference between LGPL and BSD is at the level of source >> code, not binary code: one modifies LGPL-based source code and >> distributes a binary form of it, then one has to release the modified >> source code as well. > > This is not true. Binaries that contain LGPLed code must be able to be > relinked with a modified version of the LGPLed component. This doesn't apply to Eigen which is a header-only pure template library, hence can't be 'linked' to. Actually you seem to be referring to Section 4 of the LGPL3, we have already asked the FSF about this and their reply was that it just doesn't apply in the case of Eigen: http://listengine.tuxfamily.org/lists.tuxfamily.org/eigen/2009/01/msg00083.html In your case, what matters is Section 5. >>> >>> You mean Section 3. Good. >> >> Section 3 is for using Eigen directly in a C++ program, yes, but I got >> a bit ahead of myself there: see below >> >>> I admit to being less up on the details of >>> LGPLv3 than I was of LGPLv2 which had a problem with C++ header >>> templates. >> >> Indeed, it did, that's why we don't use it. >> >>> >>> That said, we will not be using the C++ templates directly in numpy >>> for technical reasons (not least that we do not want to require a C++ >>> compiler for the default build). At best, we would be using a BLAS >>> interface which requires linking of objects, not just header >>> templates. That *would* impose the Section 4 requirements. >> >> ... or rather Section 5: that is what I was having in mind: >> " 5. Combined Libraries. " >> >> I have to admit that I don't understand what 5.a) means. > > I don't think it applies. Let's say I write some routines that use an > LGPLed Library (let's call them Routines A). I can include those > routines in a larger library with routines that do not use the LGPLed > library (Routines B). The Routines B can be under whatever license you > like. However, one must make a library containing only Routines A and > the LGPLed Library and release that under the LGPLv3, distribute it > along with the combined work, and give notice about how to obtain > Routines A+Library separate from Routines B. Basically, it's another > exception for needing to be able to relink object code in a particular > technical use case. > > This cannot apply to numpy because we cannot break out numpy.linalg > from the rest of numpy. Even if we could, we do not wish to make > numpy.linalg itself LGPLed. Indeed, that seems very cumbersome. I will ask the FSF about this, as this is definitely not something that we want to impose on Eigen users. Benoit > > -- > Robert Kern > > "I have come to believe that the whole world is an enigma, a harmless > enigma that is made terrible by our own mad attempt to interpret it as > though it had an underlying truth." > -- Umberto Eco > ___ > NumPy-Discussion mailing list > NumPy-Discussion@scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Waf or scons/numscons for a C/Fortran/Cython/Python project -- what's your recommendation?
On Sat, Jan 16, 2010 at 10:04 PM, David Cournapeau wrote: > On Sun, Jan 17, 2010 at 4:12 AM, Kurt Smith wrote: >> My questions here concern those familiar with configure/build/install >> systems such as distutils, setuptools, scons/numscons or waf >> (particularly David Cournapeau). >> >> I'm creating a tool known as 'fwrap' that has a component that needs >> to do essentially what f2py does now -- take fortran source code and >> compile it into a python extension module. It uses Cython to create >> the extension module, and the current configure/build/install system >> is a very kludgy monkeypatched Cython.distutils and numpy.distutils >> setup.py script. The setup.py script works for testing on my system >> here, but for going prime time, I dread using it. David has made his >> critiques of distutils known for scientific software, and I agree. >> What's the best alternative? > > The best alternative in the short term is no alternative: making sure > everything you need is incorporated in numpy.distutils. Otherwise, you > will have to recreate everything that distutils is doing: you will > have people who will demand egg, mac os x .mpkg, windows installers, > etc.. Basically what I am trying to do with toydist now - I don't mind > getting help there, though :) If you ignore the installation phase and focus on just the configure/build phases (see below), what would you say then? How much work needs to go into the install step as compared to the configure/build steps for waf or scons? > > I promised to add decent cython support in numpy.distutils for 1.5.0, > maybe we should see what we can do for fwrap at the same time. > > I am also a bit unclear about what is needed exactly, and what would > be the workflow: I don't understand why fwrap should care about > packaging/deployment at all, for example. I should have emphasized that fwrap primarily needs a good configure/build system **for the projects that fwrap wraps.** The original post probably didn't make this clear. The workflow is this, assuming fwrap is installed appropriately on the platform: * fwrap is called on a bunch of fortran source files. * fwrap generates fortran wrappers, consisting of fortran source files and Cython source files. * If the user wants fwrap to create an extension module then and there: * The configure/build system (waf, scons, toydist, or a setup.py script) kicks in. It... * Configures the build, getting appropriate compilers, making sure Cython is installed, sorting out fortran <-> C type sizes, etc. * Builds the code appropriately, in the right order, etc. (requires fortran, Cython & C compilation and linking, with scanning and dependency injection). * Puts the extension *.so file in the current working directory by default. So the build tool's installation stage is much less important. It seems from these discussions that its the installation step that is particularly nasty. Fwrap would leave the installation of the extension module to the user. >> >> More specifically: what are the pros/cons between waf and >> scons/numscons for configure/build/install of a >> Fortran-C-Cython-Python project? > > waf has no fortran support whatsoever, so you would need to add it. > Waf codebase is much better than scons, but there lacks some internal > documentation. There were some things that I did not manage to do in > waf, because the internal API for scanning/dependency injection was > not very clear to me (by scanning I mean the ability to scan the > source code to look for dependency, e.g. fortran modules, and by > dependency injection, I mean adding new targets to the DAG of > dependencies at runtime - again needed for fortran modules). I'll likely need both scanning and Dependency Injection in fwrap. The complete lack of fortran support is an issue, although not a deal breaker. I like that waf is designed to be small, self-contained and not installed system-wide. I could distribute waf with fwrap and users wouldn't have to have yet another external dependency for fwrap to work. > > Basic handling of fortran compilation and fortran detection was > relatively easy in comparison. > > The biggest drawback I see with waf is the lack of users: the only > significant project I know which uses waf is Ardour. OTOH, I believe > scons has deep structural problems, and only a few people can change > some significant parts of the code. Yes, the lack of adoption is an issue. Would scons' structural problems affect a project like fwrap, in your view? You have a fair amount of experience dealing with Fortran/C hybrid programming -- is scons still flexible enough to handle it? > >> >> Is scons capable of handling the configure and install stages, or is >> it only a build system? As I understand it, numscons is called from >> distutils; distutils handles the configure/install stages. > > Distutils only handles the installation - everything done within the > build_* co
[Numpy-discussion] Pandas LongPanel/WidePanel for 3d timeseries?
Hello, I am successfully using the new Pandas library for series and matrix analysis using 2 dimensional arrays. I am using the "fromDict" method which works well, and I am creating 2-dimensional arrays where each axis is indexed by FX currency names. So for example pandas DataFrame called aa: >>> aa.columns Index([AUDUSD, EURCHF, EURCZK, EURHUF, EURPLN, EURSEK, EURUSD, GBPUSD, NZDUSD, USDBRL, USDCAD, USDCLP, USDILS, USDJPY, USDKRW, USDMXN, USDRUB, USDSGD, USDTRY, USDTWD, USDZAR], dtype=object) >>> aa.rows Index: 21 entries, AUDUSD to USDZAR Data columns: AUDUSD20 non-null values EURCHF20 non-null values EURCZK20 non-null values EURHUF20 non-null values EURPLN20 non-null values EURSEK20 non-null values EURUSD20 non-null values GBPUSD20 non-null values NZDUSD20 non-null values USDBRL20 non-null values USDCAD20 non-null values USDCLP20 non-null values USDILS20 non-null values USDJPY20 non-null values USDKRW20 non-null values USDMXN20 non-null values USDRUB20 non-null values USDSGD20 non-null values USDTRY20 non-null values USDTWD20 non-null values USDZAR20 non-null values > >>> aa['USDZAR']['USDTWD'] 1.2711725043942563 >>> (each cell contains the number of standard errors of today's prices in the linear regression of the two currency pairs). Now I want to create a 3 dimensional stack of these aa-style matrices, where the z axis is indexed by historical dates. IE one matrix for each date, from today, going back 2 years. What is the best pandas function for doing this? Is it pandas.WidePanel or pandas.LongPanel, and how do I use these functions to construct this 3d stack? (I would ideally like to append each 2d matrix to the 3d stack as I create each one). Thanks for the help.. unfortunately can't find this in the online docs. Tom ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Pandas LongPanel/WidePanel for 3d timeseries?
Apologies - too quick to ask the list without thoroughly checking the online docs. I have found the answer (fromDict method takes DataFrame objects): http://pandas.sourceforge.net/generated/pandas.WidePanel.html#pandas.WidePanel Still would like to know how to append 2d matrices one-by-one though. On 17 Jan 2010, at 23:43, totalb...@mac.com wrote: > Hello, > > I am successfully using the new Pandas library for series and matrix analysis > using 2 dimensional arrays. I am using the "fromDict" method which works > well, and I am creating 2-dimensional arrays where each axis is indexed by FX > currency names. So for example pandas DataFrame called aa: > aa.columns > Index([AUDUSD, EURCHF, EURCZK, EURHUF, EURPLN, EURSEK, EURUSD, GBPUSD, > NZDUSD, USDBRL, USDCAD, USDCLP, USDILS, USDJPY, USDKRW, USDMXN, > USDRUB, USDSGD, USDTRY, USDTWD, USDZAR], dtype=object) aa.rows > > Index: 21 entries, AUDUSD to USDZAR > Data columns: > AUDUSD20 non-null values > EURCHF20 non-null values > EURCZK20 non-null values > EURHUF20 non-null values > EURPLN20 non-null values > EURSEK20 non-null values > EURUSD20 non-null values > GBPUSD20 non-null values > NZDUSD20 non-null values > USDBRL20 non-null values > USDCAD20 non-null values > USDCLP20 non-null values > USDILS20 non-null values > USDJPY20 non-null values > USDKRW20 non-null values > USDMXN20 non-null values > USDRUB20 non-null values > USDSGD20 non-null values > USDTRY20 non-null values > USDTWD20 non-null values > USDZAR20 non-null values >> aa['USDZAR']['USDTWD'] > 1.2711725043942563 > > (each cell contains the number of standard errors of today's prices in the > linear regression of the two currency pairs). Now I want to create a 3 > dimensional stack of these aa-style matrices, where the z axis is indexed by > historical dates. IE one matrix for each date, from today, going back 2 > years. What is the best pandas function for doing this? Is it > pandas.WidePanel or pandas.LongPanel, and how do I use these functions to > construct this 3d stack? (I would ideally like to append each 2d matrix to > the 3d stack as I create each one). > > Thanks for the help.. unfortunately can't find this in the online docs. > > Tom > ___ > NumPy-Discussion mailing list > NumPy-Discussion@scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Pandas LongPanel/WidePanel for 3d timeseries?
On Sun, Jan 17, 2010 at 3:49 PM, wrote: > Apologies - too quick to ask the list without thoroughly checking the online > docs. I have found the answer (fromDict method takes DataFrame objects): > > http://pandas.sourceforge.net/generated/pandas.WidePanel.html#pandas.WidePanel > > Still would like to know how to append 2d matrices one-by-one though. Here's the best place to ask questions about pandas: http://groups.google.ca/group/pystatsmodels ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Pandas LongPanel/WidePanel for 3d timeseries?
On Jan 17, 2010, at 9:21 PM, Keith Goodman wrote: > On Sun, Jan 17, 2010 at 3:49 PM, wrote: >> Apologies - too quick to ask the list without thoroughly checking >> the online docs. I have found the answer (fromDict method takes >> DataFrame objects): >> >> http://pandas.sourceforge.net/generated/pandas.WidePanel.html#pandas.WidePanel >> >> Still would like to know how to append 2d matrices one-by-one though. > > Here's the best place to ask questions about pandas: > > http://groups.google.ca/group/pystatsmodels > ___ > NumPy-Discussion mailing list > NumPy-Discussion@scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion pystatsmodels is the best place to ask questions. You will want to use WidePanel.fromDict. Additional items can be added easily like so: wp[x] = frame I apologize for the lack of documentation-- that should change in the next month or so. ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] Numpy.dot segmentation fault
Hello, I've encountered a segfault in numpy when trying to compute a dot product for two arrays - see code below. The problem only seems to occur when the arrays reach a certain size. I'm using Numpy version 1.3.0, installed via macports, on a 2.33 GHz Intel Core2 Duo Macbook Pro running Leopard (OSX 10.5.8). I've posted this as a bug on the numpy page, where I was told it might have to do with my ATLAS installation (version 3.8.3_1, also installed via macports). Has anyone run into anything like this before? Cheers, Mark Example code: import numpy as N print 'Demonstration of Numpy Bug:' print 'loading X (random numbers)' SzList = [10,20,30,40,50,60,70,80,90,100] for Sz in SzList: print 'X size = %d,%d'%(300,Sz) X = N.random.rand(300,Sz) Y = N.random.rand(300,3) print 'Attempting dot product of X and Y' N.dot(X.T,Y) print 'Finished without bug.' Result (run through gdb): (There were a number of warnings like this - so many that they went off the top of the screen and I couldn't copy them all. This was typical of the warnings.) Reading symbols for shared libraries warning: Could not find object file "/opt/local/var/macports/ build_opt_local_var_macports_sources_rsync .macports.org_release_ports_lang_python26/work/Python-2.6.4/build/ temp.macosx-10.5-i386-2.6/opt/local/var/macports/ build_opt_local_var_macports_sources_rsync.macports.org_release _ports_lang_python26/work/Python-2.6.4/Modules/_collectionsmodule.o" - no debug information available for "/opt/local/var/macports/build/ _opt_local_var_macports_sources_rsync .macports.org_release_ports_lang_python26/work/Python-2.6.4/Modules/ _collectionsmodule.c". . done Demonstration of Numpy Bug: loading X (random numbers) X size = 300,10 Attempting dot product of X and Y Finished without bug. X size = 300,20 Attempting dot product of X and Y Finished without bug. X size = 300,30 Attempting dot product of X and Y Finished without bug. X size = 300,40 Attempting dot product of X and Y Finished without bug. X size = 300,50 Attempting dot product of X and Y Finished without bug. X size = 300,60 Attempting dot product of X and Y Program received signal EXC_BAD_ACCESS, Could not access memory. Reason: 13 at address: 0x [Switching to process 56005 thread 0x117] 0x01038884 in ATL_dupMBmm0_2_0_b0 () ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Numpy.dot segmentation fault
Mark Lescroart wrote: > Hello, > > I've encountered a segfault in numpy when trying to compute a dot > product for two arrays - see code below. The problem only seems to occur > when the arrays reach a certain size. Your atlas is most likely broken. You will have to double-check how you built it, and maybe run the whole test suite (as indicated in the ATLAS installation notes). Note that you can use the Accelerate framework on mac os x, this is much easier to get numpy working on mac, cheers, David ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion