Re: [Numpy-discussion] Nasty bug using pre-initialized arrays
I wasn't at all arguing that having complex data chopped and downcast into an int or float container was the right thing to do. indeed, it is an clearly bad thing to do -- but a bug magnet? I'm not so sure, surely, anyone that had any idea at all what they were doing with complex numbers would notice it right away. To the OP: Did you waste any time thinking this was working right? Or was your time spent figuring out why numpy wold do such a thing? which is wasted time none the less. Thanks for the question. I spent about 1/2 hour looking at my other code trying to figure out why I was getting strange results. Of course, I suspected my own code to be the culpret, since NumPy is a mature package, right?. Then, when I looked closely at the return array given by NumPy, I noticed that it was real, when I was working with complex numbers. I said to myself WTF?. Then a little more investigating revealed that NumPy was silently truncating my complex array to real. I respectfully submit that silently chopping the imaginary part *is* a magnet for bugs, since many dumb NumPy users (like me) aren't necessarily aware of the typecasting rules. We're only thinking about doing math, after all! Stuart Brorson Interactive Supercomputing, inc. 135 Beaver Street | Waltham | MA | 02452 | USA http://www.interactivesupercomputing.com/ ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Nasty bug using pre-initialized arrays
There are two related hierarchies of datatypes: different kinds (integer, floating point, complex floating point) and different precisions within a given kind (int8, int16, int32, int64). The term downcasting should probably be reserved for the latter only. It seems to me that Zach and Scott are defending downcasting of precisions within a given kind. It does not necessarily follow that the behavior we choose for dtypes within a given kind should be the behavior when we are dealing with dtypes across different kinds. We can keep the precision downcasting behavior that you want while raising an error when one attempts to assign a complex number into a floating point array. Actually, my points were explicitly about cross-kind conversions! A lot of image-processing code features heavy use of converting integer images to float for intermediate calculations, and then rescaling and down-converting back to the original type for display, etc. In fact, scipy.ndimage makes use of this, allowing for operations with output into user-specified arrays, or arrays of user-specified dtype, while (I believe) carrying out some of the intermediate calculations at higher precision. As such, pretty much any code that takes a user-specified array and assigns to it the result of a (potentially) mixed-mode calculation would need to be changed from: A[i] = calculate(B) to A[i] = calculate(B).astype(A.dtype) with the proliferation of temp arrays that that entails. I think, but am not sure, that there is a lot of code out there that does this, intentionally, which would be broken by this change. Explicit is indeed better than implicit, but in this case, finding all of the places where mixed-mode conversions happen and tracking them down could be a pain on the same scale as const chasing in C++, where fixing the error in one place makes it reappear elsewhere in the chain of usage, leading to long, painful, and often totally pointless debugging sessions. In the specific case of converting from complex to anything else, I can see the desire for a warning to prevent data loss. But converting from float to integer is a lot more routine and is used a lot in the kind of work that I do, at least. Zach ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Error importing from numpy.matlib
On Jan 3, 2008, at 18:29 , Steve Lianoglou wrote: Anyway, somewhere in my codebase (for a long time now) I'm doing: from numpy.matlib import * Now, when I try to use this code, or just type that in the interpreter, I get this message: AttributeError: 'module' object has no attribute 'pkgload' This doesn't happen when I do: import numpy.matlib as M Anyway, can anyone shed some light on this? This is the behaviour you'd get if the module's __all__ attribute lists objects which don't exist in the module. Looks like a regression in r4659; fixed in SVN now as r4674. -- ||\/| /--\ |David M. Cooke http://arbutus.physics.mcmaster.ca/dmc/ |[EMAIL PROTECTED] ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Moving away from svn ?
On Jan 4, 2008, at 13:58 , Fernando Perez wrote: My vote so far is for hg, for performance reasons but also partly because sage and sympy already use it, two projects I'm likely to interact a lot with and that are squarely in line with the ipython/numpy/scipy/matplotlib world. Since they went first and made the choice, I'm happy to let that be a factor in my decision. I'd rather use a tool that others in the same community are also using, especially when the choice is a sound one on technical merit alone. Just my 1e-2... +1 on mercurial. It's what I use these days (previously, I used darcs, which I still like for its patch-handling semantics, but its dependence on Haskell, and the dreaded exponential-time merge are a bit of a pain). One thing that can help is an official Mercurial mirror of the subversion repository. IIRC, sharing changegroups or pulling patches between hg repos requires that they have a common ancestor repo (as opposed to two developers independently converting the svn repo). -- ||\/| /--\ |David M. Cooke http://arbutus.physics.mcmaster.ca/dmc/ |[EMAIL PROTECTED] ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Moving away from svn ?
On Jan 5, 2008 12:08 PM, David M. Cooke [EMAIL PROTECTED] wrote: On Jan 4, 2008, at 13:58 , Fernando Perez wrote: My vote so far is for hg, for performance reasons but also partly because sage and sympy already use it, two projects I'm likely to interact a lot with and that are squarely in line with the ipython/numpy/scipy/matplotlib world. Since they went first and made the choice, I'm happy to let that be a factor in my decision. I'd rather use a tool that others in the same community are also using, especially when the choice is a sound one on technical merit alone. Just my 1e-2... +1 on mercurial. It's what I use these days (previously, I used darcs, which I still like for its patch-handling semantics, but its dependence on Haskell, and the dreaded exponential-time merge are a bit of a pain). Regarding the 'record' capapbilities of darcs which were indeed very nice, here's something that was recently mentioned on the sage list: I noticed that Mercurial 0.9.5 has a record extension that mimics the darcs record functionality of interactively asking what changes you want to commit out of a file. I know there was discussion of this a while ago. Reference: http://www.selenic.com/pipermail/mercurial/2007-October/015150.html under the New extensions heading. See also http://www.selenic.com/mercurial/wiki/index.cgi/RecordExtension Anyways, I'm just posting this as an FYI. It might be nice to expose this functionality to sage, if we haven't already. Thanks, Jason Cheers, f ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Moving away from svn ?
On Jan 5, 2008 8:15 PM, Fernando Perez [EMAIL PROTECTED] wrote: On Jan 5, 2008 12:08 PM, David M. Cooke [EMAIL PROTECTED] wrote: On Jan 4, 2008, at 13:58 , Fernando Perez wrote: My vote so far is for hg, for performance reasons but also partly because sage and sympy already use it, two projects I'm likely to interact a lot with and that are squarely in line with the ipython/numpy/scipy/matplotlib world. Since they went first and made the choice, I'm happy to let that be a factor in my decision. I'd rather use a tool that others in the same community are also using, especially when the choice is a sound one on technical merit alone. Just my 1e-2... +1 on mercurial. It's what I use these days (previously, I used darcs, which I still like for its patch-handling semantics, but its dependence on Haskell, and the dreaded exponential-time merge are a bit of a pain). Regarding the 'record' capapbilities of darcs which were indeed very nice, here's something that was recently mentioned on the sage list: I noticed that Mercurial 0.9.5 has a record extension that mimics the darcs record functionality of interactively asking what changes you want to commit out of a file. I know there was discussion of this a while ago. Reference: http://www.selenic.com/pipermail/mercurial/2007-October/015150.html under the New extensions heading. See also http://www.selenic.com/mercurial/wiki/index.cgi/RecordExtension Anyways, I'm just posting this as an FYI. It might be nice to expose this functionality to sage, if we haven't already. Thanks, Jason Kirill (a sympy developer) has also sent patches for qrecord (record for mercurial queues) http://www.selenic.com/pipermail/mercurial-devel/2007-December/003953.html Ondrej ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Moving away from svn ?
David M. Cooke wrote: On Jan 4, 2008, at 13:58 , Fernando Perez wrote: My vote so far is for hg, for performance reasons but also partly because sage and sympy already use it, two projects I'm likely to interact a lot with and that are squarely in line with the ipython/numpy/scipy/matplotlib world. Since they went first and made the choice, I'm happy to let that be a factor in my decision. I'd rather use a tool that others in the same community are also using, especially when the choice is a sound one on technical merit alone. Just my 1e-2... +1 on mercurial. It's what I use these days (previously, I used darcs, which I still like for its patch-handling semantics, but its dependence on Haskell, and the dreaded exponential-time merge are a bit of a pain). I don't think it is time to move wholesale to something like Mercurial or bzr. I would prefer it if all of the Enthought-hosted projects moved to the (new) system at once, which is not going to happen in the short term (but long term of course it's an open question). But, having an official mirror is a useful thing to explore. I suspect there are others with serious reservations about jumping off of SVN just now (just when a lot of people have finally figured out how to use it). If there is an Hg mirror that allows other to use mercurial at the same time then that sounds like a good idea to me. -Travis O. ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Error importing from numpy.matlib
This is the behaviour you'd get if the module's __all__ attribute lists objects which don't exist in the module. Looks like a regression in r4659; fixed in SVN now as r4674. Thanks! -steve ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Moving away from svn ?
On Sat, Jan 05, 2008 at 03:00:21PM -0600, Travis E. Oliphant wrote: I suspect there are others with serious reservations about jumping off of SVN just now (just when a lot of people have finally figured out how to use it). I recall something you said to David last week, regarding merges with SVN: that a person never knows how to do it until *after* you've done it! We often make branches in scipy and numpy, and stand a lot to gain from a distributed RCS. Once a person knows how to use SVN, it doesn't take much effort at all to learn bzr or hg (even the commands are often the same). The main change is a mind-shift: that branches are now a lot friendlier, and that they are accessable to everybody. At the end of 2005, back when I was still working with Octave, we had a discussion on the merits of switching over to Subversion. That conversation never went anywhere, which is the reason you can still obtain Octave today using cvs -d :ext:[EMAIL PROTECTED]:/cvs I know there are reservations about doing the switch *right now*, which is fine -- we must just not wait too long. Regards Stéfan ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Moving away from svn ?
I'd like to briefly provide a different perspective on this question, which is not a technical one but a more social/process one. It seems to me (but I could be wrong; this is opinion, not research!) that a DVCS encourages a more open participation model for newcomers. Since anyone with a checkout has the same tree, there is no more 'us vs. them' in the sense of 'developers vs users'. Yes, with SVN anyone can track trunk or branches and submit a patch, but there's a distinct asymmetry in the process that DVCS remove (bviously even with a DVCS model there always be a canonical repository that is considered official, and to which only a group with commit rights can push changes). In addition, DVCS allow more easily the creation of subgroups of parallel developers who share their branches and explore ideas, subprojects, optimizations, etc. With a DVCS, anyone can join such a subgroup, contribute, and if that idea bears fruit, it's easy to fold it back into the official trunk. SVN doesn't really lend itself well at all to this type of approach, and I think it therefore tends to lower the amount of intellectual exploration a project is likely to do during its lifetime. So I'd venture that a DVCS can benefit a project in the long run by lowering the tunneling energy required to make the user-developer transition. Given how users who make this transition are the life and blood of any open source project, I'd argue that anything that helps this is worth considering. Obviously the above is not an argument for doing anything *now*, as for many reasons now may not be the right time. But it is to me a compelling argument for taking the step, leaving only the when and which specific tool as decisions to be appropriately determined. Of course, I could be fully wrong, since the above is little more than common-sense-sounding speculation. Cheers, f ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Moving away from svn ?
On Jan 6, 2008 8:25 AM, Stefan van der Walt [EMAIL PROTECTED] wrote: I recall something you said to David last week, regarding merges with SVN: that a person never knows how to do it until *after* you've done it! We often make branches in scipy and numpy, and stand a lot to gain from a distributed RCS. Once a person knows how to use SVN, it doesn't take much effort at all to learn bzr or hg (even the commands are often the same). The main change is a mind-shift: that branches are now a lot friendlier, and that they are accessable to everybody. I understand that DVCS's do merging better. But what I don't really understand is why this is an inherent advantage of DVCS. Isnt it just a matter of the current crop of DVCS's implementing a better merge algorithm than SVN? The SVN guys seem to be competent, so if you just give them time surely they will eventually incorporate these better merging algorithms into SVN. Who wouldn't want better merging? --bb ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] branches/maskedarray patch + lost password
Hi Pierre On Fri, Jan 04, 2008 at 04:30:47PM -0500, Pierre GM wrote: Here's a patch to the current version of svn/numpy/branches/maskedarray, that corrects several bugs. Those tests at the end of core.py should be included in the test suite before we commit (otherwise they won't be ran by numpy.test()). Regards Stéfan ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Moving away from svn ?
Fernando Perez fperez.net at gmail.com writes: Incidentally, the emacs guys seem to be worrying about the same thing: http://thread.gmane.org/gmane.emacs.devel/85893 If they actually do the work of comparing tools, that work may be useful for us. I'm pretty sure that any tool that can handle the entire history of emacs can chew on numpy/scipy/ipython/matplotlib *combined* for breakfast. The discussions on emacs-devel are interesting indeed. I've been keeping my own local Emacs patches (to make it build on FreeBSD as a port) in a Mercurial repository for a fair amount of time now, and it seems to work ok. One interesting datapoint is that the entire history of the HEAD branch of the repository of Emacs can fit in 141 MB, which is smaller than a full checkout of Emacs plus object code after a build, and it takes less than 2 seconds to look at the log of the first commit done in CVS ever: kobe % du -sk ~/tmp/emacs-src 187436 /home/keramida/tmp/emacs-src kobe % pwd /home/keramida/hg/emacs/gnu kobe % du -sk .hg 141620 .hg kobe % /usr/bin/time hg log -r0 changeset: 0:c67c006134ec user:jimb date:Thu Apr 18 00:48:29 1985 + summary: entered into RCS 1.72 real 1.40 user 0.29 sys kobe % I'm sure that this is far from a killer feature, but at least it shows that there's no huge obstacle in using Hg to check-out and browse the history of a repository the size of Emacs' CVS tree. I don't know if Emacs will use Mercurial or some other DVCS, but it should be certainly do-able. Just my two cents, Giorgos ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Moving away from svn ?
On Jan 5, 2008 4:38 PM, Robert Kern [EMAIL PROTECTED] wrote: Chris Barker wrote: hmmm. Everyone posting so far seems to be positive on this idea, but I'm not so sure. A few thoughts: 1) change is bad. It may be worth it, but this decision needs to be made very differently than if we were starting from scratch. 2) apparently svn merge sucks compared to other merge technology. svn (and cvs before it) is the only system I'm used, and yes, merging is painful, but I have to say that it appeared to be painful because it's a very hard problem. Can anyone comment on why these other systems seem so much better? Does it have anything to do with Centralized vs. Distributed at all? Tangentially, yes. DVCSes need to keep track of more information in order to be distributed. That information is extremely useful for managing merges properly. Centralized systems could track this information, but they don't *need* to in order to be functional, so they mostly haven't, yet. For each revision, the DVCS knows what revisions it derives from. SVN does not keep this information. SVN primarily just knows the text diffs from revision to revision. In particular, if I have a long-lived branch, I am going to merge in changes from the trunk while I'm working on it. When I go to merge the branch back into the trunk, I need to know which trunk-revisions I've already merged into the branch. SVN does not track this information. Tools like svnmerge.py track some of this information at the expense of some added clumsiness. It's worth noting that SVN 1.5 will be tracking such information. But that release is a long ways off. My understanding, but I do not follow svn much, is that in 1.5, you will only get what svnmerge gives you today. David ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Moving away from svn ?
Travis E. Oliphant wrote: David M. Cooke wrote: On Jan 4, 2008, at 13:58 , Fernando Perez wrote: My vote so far is for hg, for performance reasons but also partly because sage and sympy already use it, two projects I'm likely to interact a lot with and that are squarely in line with the ipython/numpy/scipy/matplotlib world. Since they went first and made the choice, I'm happy to let that be a factor in my decision. I'd rather use a tool that others in the same community are also using, especially when the choice is a sound one on technical merit alone. Just my 1e-2... +1 on mercurial. It's what I use these days (previously, I used darcs, which I still like for its patch-handling semantics, but its dependence on Haskell, and the dreaded exponential-time merge are a bit of a pain). I don't think it is time to move wholesale to something like Mercurial or bzr. I would prefer it if all of the Enthought-hosted projects moved to the (new) system at once, which is not going to happen in the short term (but long term of course it's an open question). How would you define short term and long term ? I understand the need to plan carefully things, and this takes time. But I am not sure I understand why not moving now, in the sense what will change in the future which will make the change better then than now ? I thought about targeting the change for 1.1, because it gives one precise time target, and gives use around 6 months, but this can be later if it takes more time. I have not thought about all the details, but I had something like the following plan in mind: - first, importing the different trunks into the different contenders - making read-only mirrors of those import so that people can try it out - making tutorials so that people who do not know about the tools can start in a few minutes - Having a clear idea on the difference between the tools, and makes a list of the different requirements (speed, availability on platforms, 3-party tools, GUI, etc...). It seems mercurial is the tool of choice for almost everybody; I don't have any problem with that, except that I would like to see more arguments than just I am using mercurial and it works (the fact that most contributors knows mercurial certainly is an argument in favor of it, though). Having a list of requirements and how each tool fulfill each of them would be helpful, no ? I am certainly willing to do all the above for bzr, and it seems doing it for mercurial won't be any problem since so many people already know it. But, having an official mirror is a useful thing to explore. I suspect there are others with serious reservations about jumping off of SVN just now (just when a lot of people have finally figured out how to use it). Using bzr is easier than svn IMHO, and knowing bzr, I knew how to use mercurial for basic things in 5 minutes (by basic I mean checking out code, branching, committing and getting patches): bzr co http://bzr.scipy.org/bzr/numpy - get the code bzr st - see the changes bzr diff - get a patch bzr ci - commit Basically, you have to change svn to bzr :) And this is really similar in mercurial. Things like getting through the history, merging is a bit more complicated because there is no notion of global revision anymore, but this won't matter to most people ? If there is an Hg mirror that allows other to use mercurial at the same time then that sounds like a good idea to me. Is it possible to get the svn dump for people willing to do the import (doing it from the import is not the only way, but is certainly the fastest) ? I can do the import for bzr; I can do it in mercurial too if nobody else jumps in, but since I am less familiar with mercurial, it would be better for someone else to do it. cheers, David ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
