Re: [Numpy-discussion] BOF notes: Fernando's proposal: NumPy ndarray with named axes
Not really. 1-D structured arrays can and do work well for the very common case where one has unlabeled rows and labeled columns. They are also a little bit more flexible in that the columns can be heterogeneous in dtype, as columns are wont to do. May I politely suggest that, just as some people did not do a sufficient job of reading the datarray proposal to understand how they differ from structured arrays, you do not know as much about structured arrays to understand the ways in which they are similar to labeled arrays? Understanding both the similarities and differences is important because both are going to be living in the same ecosystem with overlapping niches. All right, you got me there. I had no idea that record arrays had that functionality. In the end, we are making the same point. Datarrays and record arrays take different approaches to similar problems, and can easily coexist, and you could even make a datarray of records if you wanted. -- Rob ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] numpy installation question (IronPython)
Hello, I simply installed numpy in my Python26 installation, and then copied the numpy directory to my site-packages folder of my IronPython installation. Did I miss any installation steps in doing so? The multiarray module could not be found using IronPython. Thanks. ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] numpy installation question (IronPython)
On Tue, Jul 13, 2010 at 2:20 PM, William Johnston willi...@tenbase2.com wrote: Hello, I simply installed numpy in my Python26 installation, and then copied the numpy directory to my site-packages folder of my IronPython installation. Did I miss any installation steps in doing so? The multiarray module could not be found using IronPython. Hi William, Why do you think that numpy works in IronPython ? I thought most Python modules work only with standard (C) Python Numpy depends heavily on C implementations for most of its functionality. Regards, Sebastian Haase ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] numpy installation question (IronPython)
On Tue, Jul 13, 2010 at 8:26 AM, Sebastian Haase seb.ha...@gmail.com wrote: On Tue, Jul 13, 2010 at 2:20 PM, William Johnston willi...@tenbase2.com wrote: Hello, I simply installed numpy in my Python26 installation, and then copied the numpy directory to my site-packages folder of my IronPython installation. Did I miss any installation steps in doing so? The multiarray module could not be found using IronPython. Hi William, Why do you think that numpy works in IronPython ? I thought most Python modules work only with standard (C) Python Numpy depends heavily on C implementations for most of its functionality. Regards, Sebastian Haase ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion William-- I think you will want to look here to be able to use NumPy in IronPython for the time being: http://code.google.com/p/ironclad/ ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] BOF notes: Fernando's proposal: NumPy?ndarray with named axes
Ryan May writes: On Mon, Jul 12, 2010 at 8:04 AM, Neil Crighton neilcrigh...@gmail.com wrote: Gael Varoquaux gael.varoquaux at normalesup.org writes: I do such manipulation all the time, and keeping track of which axis is what is fairly tedious and error prone. It would be much nicer to be able to write: data.ax_day.mean(axis=0) data.ax_hour.mean(axis=0) Thanks, that's a really nice description. Instead of data.ax_day.mean(axis=0) I think it would be clearer to do something like data.mean(axis='day') IIRC somewhere, at least at the BOF, this exact syntax was intended to be supported. Yes, this is already implemented on datarray. Even if the 'axis' argument were not automatically translated from an axis name to the axis index (i.e., an extension on top of scipy, as the current datarray implementation patches functions one-by-one), you could still be getting the index through the name with something similar to: arr.mean(axis=arr.axis['day'].index) Note that this arr.axis['day'] is still not implemented, but I wrote on the wiki the possibility of providing it (as well as someone else commented on the list). This is: * arr.axis give me an accessor the the 'Axis' objects in 'arr' * ['day'] give me the 'Axis' named 'day' * .index give me the index of this 'Axis' in 'arr' (the position on the list of dimensions/axes) Lluis -- And it's much the same thing with knowledge, for whenever you learn something new, the whole world becomes that much richer. -- The Princess of Pure Reason, as told by Norton Juster in The Phantom Tollbooth ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Memory leak found in ndarray (I think)?
On Mon, Jul 12, 2010 at 6:44 PM, Nathaniel Peterson nathanielpeterso...@gmail.com wrote: Wes McKinney wesmck...@gmail.com writes: Did you mean to post a different link? That's the ticket I just created :) How silly of me! I meant http://projects.scipy.org/numpy/ticket/1427 ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion Looks to be caused by the same bug. I would suggest this should get bumped up in priority for a fix? I am going to have a poke around the C codebase but I don't feel competent enough to patch it myself. ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] A release for python 2.7?
On Sat, Jul 10, 2010 at 7:51 PM, Peter numpy-discuss...@maubp.freeserve.co.uk wrote: On Fri, Jul 9, 2010 at 4:27 PM, Christoph Gohlke cgoh...@uci.edu wrote: On 7/9/2010 7:11 AM, Peter wrote: I was going to ask if someone could build Windows installers for NumPy 1.4.1 on Python 2.7 (to facilitate 3rd party packages which use NumPy and want to support Python 2.7 on Windows). To build numpy 1.4.1 for Python 2.7 you need at least to apply changeset 7926 [1] and followups. Releasing 3rd party extensions compiled against this version of numpy is not a good idea because those binaries will likely be incompatible with any later official release of numpy for Python 2.7, which will use PyCapsule instead of PyCObject [2]. [1] http://projects.scipy.org/numpy/changeset/7926 [2] http://mail.scipy.org/pipermail/numpy-discussion/2010-June/051239.html Thanks for the info - I was perhaps unclear, I'm waiting for an official NumPy windows installers for Python 2.7, and agree we shouldn't release binaries dependent on a self-compiled pre-release. Numpy 1.5 should work with Python 2.7 and 3.1 and not be too far off. In August hopefully. Cheers, Ralf ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] numpy installation question (IronPython)
On Tue, Jul 13, 2010 at 4:31 AM, Wes McKinney wesmck...@gmail.com wrote: On Tue, Jul 13, 2010 at 8:26 AM, Sebastian Haase seb.ha...@gmail.com wrote: On Tue, Jul 13, 2010 at 2:20 PM, William Johnston willi...@tenbase2.com wrote: Hello, I simply installed numpy in my Python26 installation, and then copied the numpy directory to my site-packages folder of my IronPython installation. Did I miss any installation steps in doing so? The multiarray module could not be found using IronPython. Hi William, Why do you think that numpy works in IronPython ? I thought most Python modules work only with standard (C) Python Numpy depends heavily on C implementations for most of its functionality. Regards, Sebastian Haase ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion William-- I think you will want to look here to be able to use NumPy in IronPython for the time being: http://code.google.com/p/ironclad/ ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion Also, http://www.enthought.com/media/SciPyNumPyDotNet.pdf . SOON right? I've never actually used ironpython (or pypy or, well, anything not cpython). :/ --Josh ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] Difference between shape=() and shape=(1,)
Hi, I have some arrays of various shapes in which I need to set any NaNs to 0. I have been doing the following: a[numpy.where(numpy.isnan(a)] = 0. as you can see here: In [20]: a=numpy.ones(2) In [21]: a[1]=numpy.log(-1) In [22]: a Out[22]: array([ 1., NaN]) In [23]: a[numpy.where(numpy.isnan(a))]=0. In [24]: a Out[24]: array([ 1., 0.]) Unfortunately, I've just discovered that when a.shape == () this doesn't work at all. For example: In [41]: a=numpy.array((1.)) In [42]: a.shape Out[42]: () In [43]: a[numpy.where(numpy.isnan(a))]=0. In [44]: a Out[44]: array(0.0) but if the shape is (1,), everything is ok: In [47]: a=numpy.ones(1) In [48]: a.shape Out[48]: (1,) In [49]: a[numpy.where(numpy.isnan(a))]=0. In [50]: a Out[50]: array([ 1.]) What's the difference between the 2 arrays with different shapes? If I pass a scalar into numpy.asarray() why do I get an array of shape () back? In my case this has caused a subtle bug. Is there a better way to set NaNs in an array to 0? Thanks for any tips, John. ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Difference between shape=() and shape=(1,)
On Tue, Jul 13, 2010 at 9:54 AM, John Reid j.r...@mail.cryst.bbk.ac.uk wrote: Hi, I have some arrays of various shapes in which I need to set any NaNs to 0. I have been doing the following: a[numpy.where(numpy.isnan(a)] = 0. as you can see here: In [20]: a=numpy.ones(2) In [21]: a[1]=numpy.log(-1) In [22]: a Out[22]: array([ 1., NaN]) In [23]: a[numpy.where(numpy.isnan(a))]=0. In [24]: a Out[24]: array([ 1., 0.]) Unfortunately, I've just discovered that when a.shape == () this doesn't work at all. For example: In [41]: a=numpy.array((1.)) In [42]: a.shape Out[42]: () In [43]: a[numpy.where(numpy.isnan(a))]=0. In [44]: a Out[44]: array(0.0) No need to use where. You can just do a[np.isnan(a)] = 0. But you do have to watch out for 0d arrays, can't index into those. How about: def nan_replace(a, fill=0): : a = a.copy() : if a.ndim == 0: : return a : a[np.isnan(a)] = fill : return a : a = np.array(9) nan_replace(a, 0) array(9) a = np.array([9, np.nan]) nan_replace(a, 0) array([ 9., 0.]) Oh, I guess a[np.isnan(a)] = fill makes a copy so the a.copy() can be moved inside the if statement. ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Difference between shape=() and shape=(1,)
ti, 2010-07-13 kello 10:06 -0700, Keith Goodman kirjoitti: No need to use where. You can just do a[np.isnan(a)] = 0. But you do have to watch out for 0d arrays, can't index into those. You can, but the index must be appropriate: x = np.array(4) x[()] = 3 x array(3) ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Difference between shape=() and shape=(1,)
On Tue, Jul 13, 2010 at 11:54 AM, John Reid j.r...@mail.cryst.bbk.ac.uk wrote: Hi, I have some arrays of various shapes in which I need to set any NaNs to 0. I have been doing the following: a[numpy.where(numpy.isnan(a)] = 0. as you can see here: In [20]: a=numpy.ones(2) In [21]: a[1]=numpy.log(-1) In [22]: a Out[22]: array([ 1., NaN]) In [23]: a[numpy.where(numpy.isnan(a))]=0. In [24]: a Out[24]: array([ 1., 0.]) Unfortunately, I've just discovered that when a.shape == () this doesn't work at all. For example: In [41]: a=numpy.array((1.)) In [42]: a.shape Out[42]: () In [43]: a[numpy.where(numpy.isnan(a))]=0. In [44]: a Out[44]: array(0.0) but if the shape is (1,), everything is ok: In [47]: a=numpy.ones(1) In [48]: a.shape Out[48]: (1,) In [49]: a[numpy.where(numpy.isnan(a))]=0. In [50]: a Out[50]: array([ 1.]) What's the difference between the 2 arrays with different shapes? If I pass a scalar into numpy.asarray() why do I get an array of shape () back? In my case this has caused a subtle bug. Is there a better way to set NaNs in an array to 0? You could make use of np.atleast_1d, and then everything would be canonicalized: In [33]: a = np.array(np.nan) In [34]: a Out[34]: array(nan) In [35]: a1d = np.atleast_1d(a) In [36]: a1d Out[36]: array([ NaN]) In [37]: a Out[37]: array(nan) In [38]: a1d.base is a Out[38]: True In [39]: a1d[np.isnan(a1d)] = 0. In [40]: a1d Out[40]: array([ 0.]) In [41]: a Out[41]: array(0.0) So Keith's nan_replace would be: In [42]: def nan_replace(a, fill=0.0): : a_ = np.atleast_1d(a) : a_[np.isnan(a_)] = fill : Thanks for any tips, John. ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Difference between shape=() and shape=(1,)
On Tue, Jul 13, 2010 at 10:45 AM, Kurt Smith kwmsm...@gmail.com wrote: You could make use of np.atleast_1d, and then everything would be canonicalized: In [33]: a = np.array(np.nan) In [34]: a Out[34]: array(nan) In [35]: a1d = np.atleast_1d(a) In [36]: a1d Out[36]: array([ NaN]) In [37]: a Out[37]: array(nan) In [38]: a1d.base is a Out[38]: True In [39]: a1d[np.isnan(a1d)] = 0. In [40]: a1d Out[40]: array([ 0.]) In [41]: a Out[41]: array(0.0) So Keith's nan_replace would be: In [42]: def nan_replace(a, fill=0.0): : a_ = np.atleast_1d(a) : a_[np.isnan(a_)] = fill : Neat. The docstring for atleast_1d says Copies are made only if necessary. I don't know when a copy is necessay, but watch out for it. ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Difference between shape=() and shape=(1,)
On Tue, Jul 13, 2010 at 10:36 AM, Pauli Virtanen p...@iki.fi wrote: ti, 2010-07-13 kello 10:06 -0700, Keith Goodman kirjoitti: No need to use where. You can just do a[np.isnan(a)] = 0. But you do have to watch out for 0d arrays, can't index into those. You can, but the index must be appropriate: x = np.array(4) x[()] = 3 x array(3) Then should this error message be changed? a = np.array(4) a[1] snip IndexError: 0-d arrays can't be indexed ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] HOWTO build NumPy without external math libs
Stefan, I don't believe that there are any such options, but is there a particular reason why you *don't* want to use such external libs if you have them? I don't think anyone has considered a use-case where one would want to ignore such libraries. Ben Root On Sat, Jul 10, 2010 at 11:15 AM, Stefan Seefeld seef...@sympatico.cawrote: Hello, the NumPy docs (at http://docs.scipy.org/doc/numpy/user/install.html) suggest that NumPy doesn't require external linear algebra libraries (such as ATLAS). How can I compiler NumPy without using them, even if they are detected ? Are there any undocumented options I can pass to setup.py ? (`setup.py build --help` didn't report anything.) Thanks, Stefan -- ...ich hab' noch einen Koffer in Berlin... ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] HOWTO build NumPy without external math libs
On Tue, Jul 13, 2010 at 14:39, Benjamin Root ben.r...@ou.edu wrote: Stefan, I don't believe that there are any such options, but is there a particular reason why you *don't* want to use such external libs if you have them? I don't think anyone has considered a use-case where one would want to ignore such libraries. Building redistributable binaries. -- Robert Kern I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth. -- Umberto Eco ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] HOWTO build NumPy without external math libs
Robert Kern wrote: I don't believe that there are any such options, but is there a particular reason why you *don't* want to use such external libs if you have them? I don't think anyone has considered a use-case where one would want to ignore such libraries. Building redistributable binaries. indeed -- and it is quite possible. It looks like things have changed since I last did it, but take a look at setup.cfg.example, you should be able to figure it out. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/ORR(206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception chris.bar...@noaa.gov ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] HOWTO build NumPy without external math libs
On Tue, Jul 13, 2010 at 3:05 PM, Christopher Barker chris.bar...@noaa.govwrote: Robert Kern wrote: I don't believe that there are any such options, but is there a particular reason why you *don't* want to use such external libs if you have them? I don't think anyone has considered a use-case where one would want to ignore such libraries. Building redistributable binaries. indeed -- and it is quite possible. It looks like things have changed since I last did it, but take a look at setup.cfg.example, you should be able to figure it out. -Chris Ah, I see I am getting stuck in my developer mentality... Ben Root ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Difference between shape=() and shape=(1,)
On Tue, Jul 13, 2010 at 12:45 PM, Kurt Smith kwmsm...@gmail.com wrote: On Tue, Jul 13, 2010 at 11:54 AM, John Reid j.r...@mail.cryst.bbk.ac.uk wrote: Hi, I have some arrays of various shapes in which I need to set any NaNs to 0. I have been doing the following: a[numpy.where(numpy.isnan(a)] = 0. as you can see here: In [20]: a=numpy.ones(2) In [21]: a[1]=numpy.log(-1) In [22]: a Out[22]: array([ 1., NaN]) In [23]: a[numpy.where(numpy.isnan(a))]=0. In [24]: a Out[24]: array([ 1., 0.]) Unfortunately, I've just discovered that when a.shape == () this doesn't work at all. For example: In [41]: a=numpy.array((1.)) In [42]: a.shape Out[42]: () In [43]: a[numpy.where(numpy.isnan(a))]=0. In [44]: a Out[44]: array(0.0) but if the shape is (1,), everything is ok: In [47]: a=numpy.ones(1) In [48]: a.shape Out[48]: (1,) In [49]: a[numpy.where(numpy.isnan(a))]=0. In [50]: a Out[50]: array([ 1.]) What's the difference between the 2 arrays with different shapes? If I pass a scalar into numpy.asarray() why do I get an array of shape () back? In my case this has caused a subtle bug. Is there a better way to set NaNs in an array to 0? You could make use of np.atleast_1d, and then everything would be canonicalized: In [33]: a = np.array(np.nan) In [34]: a Out[34]: array(nan) In [35]: a1d = np.atleast_1d(a) In [36]: a1d Out[36]: array([ NaN]) In [37]: a Out[37]: array(nan) In [38]: a1d.base is a Out[38]: True In [39]: a1d[np.isnan(a1d)] = 0. In [40]: a1d Out[40]: array([ 0.]) In [41]: a Out[41]: array(0.0) So Keith's nan_replace would be: In [42]: def nan_replace(a, fill=0.0): : a_ = np.atleast_1d(a) : a_[np.isnan(a_)] = fill : Maybe I am missing something subtle, but what about numpy's nan_to_num() function? Ben Root ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] update on the transition to git/github
Hello all, On May 26th, I sent an email titled curious about how people would feel about moving to github. While there were a few concerns raised, everyone was generally positive and were mainly concerned that this transition would need to be done carefully with clear workflow instructions and an clearly marked master repository. Since then David Cournapeau has done a lot of work looking into how to make the transition go as smoothly as possible. He has written a script to convert the svn repository to a git repository. And I've registered a numpy organization account on github. Over the next week, David, Stefan, and I will set things up so that everyone can see how things will work after the transition to git. David will convert the current trunk to a git repository and put it on the github site. We will write up instructions on how to use git and the github site. Everyone who wants to can get a github account and test out the workflow. At that point everyone can provide feedback and we can decide if we are ready to move forward. If we are ready to move forward, we will set up a date for the transition. On that date, we would turn off the old subversion account and create a new git repository which would from the point forward be the new master branch. If the transition to git and github for numpy goes smoothly, we will turn our attention to scipy. During the SciPy conference in Austin, we had a Birds-of-a-Feather to discuss the transition to git and github. [[ Here is a picture of the git/github BOF (several people joined the discussion later including Travis Oliphant and Fernando Perez): http://www.flickr.com/photos/irees/4750650877/sizes/l/in/set-72157624272131693/ ]] At the end of the discussion there was a general consensus that it was time to make the move. Several questions and concerns were raised: 1. Since there are many possible workflows, it is important to clearly document are proposed workflow. This document should provide simple cut-and-paste commands necessary to get developers up and running with git. 2. The question was raised about how to handle bug reports. It was pointed out that while our current trac bug report system isn't perfect, it does work and people are used to it. We decided to keep our existing trac instance and integrate it with the github site. Potentially moving from trac to another system like redmine was brought up, but most people felt it was better to only change one thing at a time (and besides that no one volunteered to do the work necessary to move to a new bug tracking system). 3. Since Ralf Gommers is in the middle of making a release, did he want us to delay any transition preparation until he was finished. This is Ralf's response when Stefan van der Walt contacted him: Thanks for asking! For me the sooner the better, I do everything with git and haven't touched svn since I discovered Mercurial while writing my thesis (and that feels like a long long time ago). When Stefan contacted Ralf, Ralf raised the following additional concern: 4. The two things I haven't seen a good solution for are the svn/externals in scipy which pulls in doc/sphinxext from numpy, and the vendor branch. In response, Stefan asked whether submodules would provide a solution. David Cournapeau responded to Stefan stating submodules are very awkward to use. Then David added in response to Ralf's original query: For vendor, it will be a separate repo, and there is no need for synchronization, so that's easy to deal with. For the sphinx extension, I would just merge with the subtree strategy from time to time from a separate repository. That's what I do for bento + yaku: yaku has its own repo, and when I update the copy in bento, I just use git merge -s subtree --no-squash, so everything is updated in one commit. The big advantage is that there is no need to even be aware of the second repo for users (git clone will bring everything), and there is no chance of screwing things up: http://book.git-scm.com/5_advanced_branching_and_merging.html; Best, Jarrod ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] update on the transition to git/github
This is awesome! I love github. I really wanted to champion for its use at the BoF but unfortunately missed it. --Josh On Tue, Jul 13, 2010 at 6:20 PM, Jarrod Millman mill...@berkeley.edu wrote: Hello all, On May 26th, I sent an email titled curious about how people would feel about moving to github. While there were a few concerns raised, everyone was generally positive and were mainly concerned that this transition would need to be done carefully with clear workflow instructions and an clearly marked master repository. Since then David Cournapeau has done a lot of work looking into how to make the transition go as smoothly as possible. He has written a script to convert the svn repository to a git repository. And I've registered a numpy organization account on github. Over the next week, David, Stefan, and I will set things up so that everyone can see how things will work after the transition to git. David will convert the current trunk to a git repository and put it on the github site. We will write up instructions on how to use git and the github site. Everyone who wants to can get a github account and test out the workflow. At that point everyone can provide feedback and we can decide if we are ready to move forward. If we are ready to move forward, we will set up a date for the transition. On that date, we would turn off the old subversion account and create a new git repository which would from the point forward be the new master branch. If the transition to git and github for numpy goes smoothly, we will turn our attention to scipy. During the SciPy conference in Austin, we had a Birds-of-a-Feather to discuss the transition to git and github. [[ Here is a picture of the git/github BOF (several people joined the discussion later including Travis Oliphant and Fernando Perez): http://www.flickr.com/photos/irees/4750650877/sizes/l/in/set-72157624272131693/ ]] At the end of the discussion there was a general consensus that it was time to make the move. Several questions and concerns were raised: 1. Since there are many possible workflows, it is important to clearly document are proposed workflow. This document should provide simple cut-and-paste commands necessary to get developers up and running with git. 2. The question was raised about how to handle bug reports. It was pointed out that while our current trac bug report system isn't perfect, it does work and people are used to it. We decided to keep our existing trac instance and integrate it with the github site. Potentially moving from trac to another system like redmine was brought up, but most people felt it was better to only change one thing at a time (and besides that no one volunteered to do the work necessary to move to a new bug tracking system). 3. Since Ralf Gommers is in the middle of making a release, did he want us to delay any transition preparation until he was finished. This is Ralf's response when Stefan van der Walt contacted him: Thanks for asking! For me the sooner the better, I do everything with git and haven't touched svn since I discovered Mercurial while writing my thesis (and that feels like a long long time ago). When Stefan contacted Ralf, Ralf raised the following additional concern: 4. The two things I haven't seen a good solution for are the svn/externals in scipy which pulls in doc/sphinxext from numpy, and the vendor branch. In response, Stefan asked whether submodules would provide a solution. David Cournapeau responded to Stefan stating submodules are very awkward to use. Then David added in response to Ralf's original query: For vendor, it will be a separate repo, and there is no need for synchronization, so that's easy to deal with. For the sphinx extension, I would just merge with the subtree strategy from time to time from a separate repository. That's what I do for bento + yaku: yaku has its own repo, and when I update the copy in bento, I just use git merge -s subtree --no-squash, so everything is updated in one commit. The big advantage is that there is no need to even be aware of the second repo for users (git clone will bring everything), and there is no chance of screwing things up: http://book.git-scm.com/5_advanced_branching_and_merging.html; Best, Jarrod ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] update on the transition to git/github
On Tue, Jul 13, 2010 at 8:20 PM, Jarrod Millman mill...@berkeley.eduwrote: Hello all, On May 26th, I sent an email titled curious about how people would feel about moving to github. While there were a few concerns raised, everyone was generally positive and were mainly concerned that this transition would need to be done carefully with clear workflow instructions and an clearly marked master repository. Since then David Cournapeau has done a lot of work looking into how to make the transition go as smoothly as possible. He has written a script to convert the svn repository to a git repository. And I've registered a numpy organization account on github. Over the next week, David, Stefan, and I will set things up so that everyone can see how things will work after the transition to git. David will convert the current trunk to a git repository and put it on the github site. We will write up instructions on how to use git and the github site. Everyone who wants to can get a github account and test out the workflow. At that point everyone can provide feedback and we can decide if we are ready to move forward. If we are ready to move forward, we will set up a date for the transition. On that date, we would turn off the old subversion account and create a new git repository which would from the point forward be the new master branch. If the transition to git and github for numpy goes smoothly, we will turn our attention to scipy. During the SciPy conference in Austin, we had a Birds-of-a-Feather to discuss the transition to git and github. [[ Here is a picture of the git/github BOF (several people joined the discussion later including Travis Oliphant and Fernando Perez): http://www.flickr.com/photos/irees/4750650877/sizes/l/in/set-72157624272131693/ ]] At the end of the discussion there was a general consensus that it was time to make the move. Several questions and concerns were raised: 1. Since there are many possible workflows, it is important to clearly document are proposed workflow. This document should provide simple cut-and-paste commands necessary to get developers up and running with git. 2. The question was raised about how to handle bug reports. It was pointed out that while our current trac bug report system isn't perfect, it does work and people are used to it. We decided to keep our existing trac instance and integrate it with the github site. Potentially moving from trac to another system like redmine was brought up, but most people felt it was better to only change one thing at a time (and besides that no one volunteered to do the work necessary to move to a new bug tracking system). 3. Since Ralf Gommers is in the middle of making a release, did he want us to delay any transition preparation until he was finished. This is Ralf's response when Stefan van der Walt contacted him: Thanks for asking! For me the sooner the better, I do everything with git and haven't touched svn since I discovered Mercurial while writing my thesis (and that feels like a long long time ago). When Stefan contacted Ralf, Ralf raised the following additional concern: 4. The two things I haven't seen a good solution for are the svn/externals in scipy which pulls in doc/sphinxext from numpy, and the vendor branch. In response, Stefan asked whether submodules would provide a solution. David Cournapeau responded to Stefan stating submodules are very awkward to use. Then David added in response to Ralf's original query: For vendor, it will be a separate repo, and there is no need for synchronization, so that's easy to deal with. For the sphinx extension, I would just merge with the subtree strategy from time to time from a separate repository. That's what I do for bento + yaku: yaku has its own repo, and when I update the copy in bento, I just use git merge -s subtree --no-squash, so everything is updated in one commit. The big advantage is that there is no need to even be aware of the second repo for users (git clone will bring everything), and there is no chance of screwing things up: http://book.git-scm.com/5_advanced_branching_and_merging.html; One thing that we need to be clear about is handling line endings in text, i.e., linefeeds only. I have read tales of folks on windows finding many of their files showing up as modified on that account. There are ways to deal with it -- the details escape me at the moment -- but we need instructions on setting up git on windows so there aren't any problems. How should we handle commits during the next week or two? I have a few things I want to get in before 1.5 is tagged. Chuck ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] update on the transition to git/github
On Tue, Jul 13, 2010 at 8:15 PM, Charles R Harris charlesr.har...@gmail.com wrote: How should we handle commits during the next week or two? I have a few things I want to get in before 1.5 is tagged. Just keep committing as normal using svn for now. Once we are ready to make the transition (which includes having documentation for windows' users posted and having received positive feedback during the testing phase), we will announce a commit freeze and make sure the time is OK for everyone. During the testing phase and up until we've actually made the transition, the only place where committed code counts is in the svn repo. Once the testing phase has successfully been completed, the testing git repo will be deleted. Then when the commit freeze starts we will make the svn repo read only. At that point David will create a new git repo from the svn repo and host it on github. At that point forward, the svn repo will remain read only so that no one can mistakenly write to it. I hope that makes sense (I just got back from the Euroscipy conference and didn't get any sleep on the plane). Jarrod ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
