Re: [Numpy-discussion] Development workflow (not git tutorial)
15.08.2015, 01:44, Chris Barker kirjoitti: [clip] > numpy doesn't use namespace packages, so develop mode works there. The develop mode is mainly useful with a virtualenv. Otherwise, you install work-in-progress development version into your ~/.local which then breaks everything else. In addition to this, "python setupegg.py develop --uninstall" says "Note: you must uninstall or replace scripts manually!", and since the scripts end up with dev version requirement hardcoded, and you have to delete the scripts manually. Virtualenvs are annoying to manage, and at least for me personally it's easier to just deal with pythonpath, especially as runtests.py manages that. Anyway, TIMTOWTDI ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Development workflow (not git tutorial)
On Fri, Aug 14, 2015 at 10:15 AM, Stefan van der Walt wrote: > Perhaps mpl_toolkits should think of > becoming mpl_3d, mpl_basemaps, etc.? > namespace packages are a fine idea, but the implementation(s) are just one big kludge... I think so, but we're getting off-topic here. numpy doesn't use namespace packages, so develop mode works there. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R(206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception chris.bar...@noaa.gov ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Problems using add_npy_pkg_config
Dear Ralf, > I stared at it for a while, and can't figure it out despite you following > the example in the add_npy_pkg_config docstring pretty much to the letter. > When you see that the error is generated in a function that starts with ``# > XXX: another ugly workaround to circumvent distutils brain damage.``, > you're usually in trouble. what a pity... do you have an alternative suggestion? Is there a good alternative, e.g. using cmake, to distribute python modules? Ciao Christian ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Development workflow (not git tutorial)
On 2015-08-14 10:08:11, Benjamin Root wrote: > I should also note that there is currently an open issue with > "pip install -e" and namespace packages. This has been reported > to matplotlib with regards to mpl_toolkits. Essentially, if you > have namespace packages, it doesn't get installed correctly in > this mode, and python won't find them. There are lots of issues with namespace packages, which is why what used to be scikits.learn and scikits.image are now all standalone packages. Perhaps mpl_toolkits should think of becoming mpl_3d, mpl_basemaps, etc.? Stéfan ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Development workflow (not git tutorial)
I think it's clear that develop/-e does not work well together with namespace packages. As noted on the relevant matplotlib issue https://github.com/matplotlib/matplotlib/issues/4907 I think the issue with namespace packages is essentially this well known one https://github.com/pypa/pip/issues/3 which I think I agree with Chris is enough to drop namespace packages if possible. >From the output of pip install -e I would say that it clear that it calls develop. Since pip install -e and and pip install uses fundamentally different ways of manage namespace packages they can't work together. In the case of matplotlib issue #4907 basemap is probably installed into the namespace with pip install while matplotlib is installed with pip install -e which clearly triggers the issue in https://github.com/pypa/pip/issues/3 best Jens fre. 14. aug. 2015 kl. 21.21 skrev Chris Barker : > On Fri, Aug 14, 2015 at 10:08 AM, Benjamin Root wrote: > >> I used to be a huge advocate for the "develop" mode, but not anymore. I >> have run into way too many Heisenbugs that would clear up if I nuked my >> source tree and re-clone. >> > > well, you do need to remember to clean out once in a while, when somethign > weird is happening... > > But I prefer that to the other options, which are: > > * re-builda nd re-install with every frikin' change > > * do sys.path manipulations, which is ugly,, error prone, and has the same > problems as develop mode anyway > > * rely on relative imports for all your tests and the like -- error prone > and ugly -- oh, and you still have the problems above... > > > I should also note that there is currently an open issue with "pip install >> -e" and namespace packages. >> > > yeah, I actually gave up on namespace packages due to them not working > right with develop mode. > > (I'm not sure if -e and develop mode are exactly the same or not...) > > -CHB > > > > -- > > Christopher Barker, Ph.D. > Oceanographer > > Emergency Response Division > NOAA/NOS/OR&R(206) 526-6959 voice > 7600 Sand Point Way NE (206) 526-6329 fax > Seattle, WA 98115 (206) 526-6317 main reception > > chris.bar...@noaa.gov > ___ > NumPy-Discussion mailing list > NumPy-Discussion@scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Development workflow (not git tutorial)
On Fri, Aug 14, 2015 at 10:08 AM, Benjamin Root wrote: > I used to be a huge advocate for the "develop" mode, but not anymore. I > have run into way too many Heisenbugs that would clear up if I nuked my > source tree and re-clone. > well, you do need to remember to clean out once in a while, when somethign weird is happening... But I prefer that to the other options, which are: * re-builda nd re-install with every frikin' change * do sys.path manipulations, which is ugly,, error prone, and has the same problems as develop mode anyway * rely on relative imports for all your tests and the like -- error prone and ugly -- oh, and you still have the problems above... I should also note that there is currently an open issue with "pip install > -e" and namespace packages. > yeah, I actually gave up on namespace packages due to them not working right with develop mode. (I'm not sure if -e and develop mode are exactly the same or not...) -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R(206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception chris.bar...@noaa.gov ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Development workflow (not git tutorial)
On Aug 14, 2015 09:16, "Chris Barker" wrote: > > On Thu, Aug 13, 2015 at 11:25 AM, Stefan van der Walt < stef...@berkeley.edu> wrote: >> >> >(for >> > example "python setup.py develop", although suggested by >> > setup.py itself, claims that "develop" is not a command). > > > develop is a command provided by setuptools, not distutils itself. > > I find it absolutely invaluable -- it is THE way to go when actively working on any package under development. > > if numpy doesn't currently use setuptools, it probably should (though maybe it's gets messy with numpy's distutils extensions...) Regarding using setuptools by default, one problem is that it actually acts rather differently from distutils by default. See https://bitbucket.org/pypa/setuptools/issues/371/setuptools-and-state-of-pep-376 >> Nowadays, you can use >> >> pip install -e . > > > pip "injects" setuptools into the mix -- so this may be develope mode with a different name. but yes, a fine option for a package that doesn't use setuptools out of the box. The version of setuptools that pip injects is also monkeypatched by pip to fix some of setuptools more obnoxious defaults. (The ones described in that bug report.) Using pip is also is the only way to reliably install all the right metadata needed to avoid problems later -- in particular pip will record the information needed to do uninstall/upgrades correctly, which neither distutils nor setuptools will do if you run setup.py directly. Basically this means running 'setup.py install' is always broken, for all projects and no matter how setup.py is written, and you should always run a pip command instead, even when building from the source tree. This is true for every python package, though, not just numpy. So setuptools doesn't provide much that's compelling for us... I believe if you really want it, though, you can run numpy's setupegg.py, which is the same as setup.py but using setuptools. Or something like that? I share Benjamin's doubts about the whole 'develop' approach, though, however accessed. For pure python packages, just importing from the source tree directly works fine and is way less error prone. For non-pure packages, I don't trust develop much anyway... build_ext --inplace can work nicely, or for numpy in particular runtests solves all my problems. (Though even then I still sometimes need to nuke the build directory or run clean manually.) -n ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Development workflow (not git tutorial)
On 08/14/2015 01:52 PM, Pauli Virtanen wrote: > 14.08.2015, 20:45, Allan Haldane kirjoitti: > [clip] >> Related to this, does anyone know how to debug numpy in gdb with proper >> symbols/source lines, like I can do with other C extensions? I've tried >> modifying numpy distutils to try to add the right compiler/linker flags, >> without success. > > runtests.py --help > > gdb --args python runtests.py -g --python script.py > > grep env runtests.py Oh! Thank you, I missed that. ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Development workflow (not git tutorial)
14.08.2015, 20:45, Allan Haldane kirjoitti: [clip] > Related to this, does anyone know how to debug numpy in gdb with proper > symbols/source lines, like I can do with other C extensions? I've tried > modifying numpy distutils to try to add the right compiler/linker flags, > without success. runtests.py --help gdb --args python runtests.py -g --python script.py grep env runtests.py ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Development workflow (not git tutorial)
On 08/13/2015 11:52 AM, Anne Archibald wrote: > Hi, > > What is a sensible way to work on (modify, compile, and test) numpy? > > There is documentation about "contributing to numpy" at: > http://docs.scipy.org/doc/numpy-dev/dev/index.html > and: > http://docs.scipy.org/doc/numpy-dev/dev/gitwash/development_workflow.html > but these are entirely focused on using git. I have no problem with that > aspect. It is building and testing that I am looking for the Right Way > to do. Related to this, does anyone know how to debug numpy in gdb with proper symbols/source lines, like I can do with other C extensions? I've tried modifying numpy distutils to try to add the right compiler/linker flags, without success. Allan ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Development workflow (not git tutorial)
I used to be a huge advocate for the "develop" mode, but not anymore. I have run into way too many Heisenbugs that would clear up if I nuked my source tree and re-clone. I should also note that there is currently an open issue with "pip install -e" and namespace packages. This has been reported to matplotlib with regards to mpl_toolkits. Essentially, if you have namespace packages, it doesn't get installed correctly in this mode, and python won't find them. On Fri, Aug 14, 2015 at 12:12 PM, Chris Barker wrote: > On Thu, Aug 13, 2015 at 11:25 AM, Stefan van der Walt < > stef...@berkeley.edu> wrote: > >> >(for >> > example "python setup.py develop", although suggested by >> > setup.py itself, claims that "develop" is not a command). > > > develop is a command provided by setuptools, not distutils itself. > > I find it absolutely invaluable -- it is THE way to go when actively > working on any package under development. > > if numpy doesn't currently use setuptools, it probably should (though > maybe it's gets messy with numpy's distutils extensions...) > > Nowadays, you can use >> >> pip install -e . >> > > pip "injects" setuptools into the mix -- so this may be develope mode with > a different name. but yes, a fine option for a package that doesn't use > setuptools out of the box. > > -Chris > > -- > > Christopher Barker, Ph.D. > Oceanographer > > Emergency Response Division > NOAA/NOS/OR&R(206) 526-6959 voice > 7600 Sand Point Way NE (206) 526-6329 fax > Seattle, WA 98115 (206) 526-6317 main reception > > chris.bar...@noaa.gov > > ___ > NumPy-Discussion mailing list > NumPy-Discussion@scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Development workflow (not git tutorial)
On Thu, Aug 13, 2015 at 11:25 AM, Stefan van der Walt wrote: > >(for > > example "python setup.py develop", although suggested by > > setup.py itself, claims that "develop" is not a command). develop is a command provided by setuptools, not distutils itself. I find it absolutely invaluable -- it is THE way to go when actively working on any package under development. if numpy doesn't currently use setuptools, it probably should (though maybe it's gets messy with numpy's distutils extensions...) Nowadays, you can use > > pip install -e . > pip "injects" setuptools into the mix -- so this may be develope mode with a different name. but yes, a fine option for a package that doesn't use setuptools out of the box. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R(206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception chris.bar...@noaa.gov ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Changes to np.digitize since NumPy 1.9?
For what it's worth, also from my astropy perspective I think hat any index array should be a base ndarray! -- Marten On Fri, Aug 14, 2015 at 7:11 AM, Jaime Fernández del Río < jaime.f...@gmail.com> wrote: > On Thu, Aug 13, 2015 at 9:57 AM, Jaime Fernández del Río < > jaime.f...@gmail.com> wrote: > >> On Thu, Aug 13, 2015 at 7:59 AM, Nathan Goldbaum >> wrote: >> >>> >>> >>> On Thu, Aug 13, 2015 at 9:44 AM, Charles R Harris < >>> charlesr.har...@gmail.com> wrote: >>> On Thu, Aug 13, 2015 at 12:09 AM, Jaime Fernández del Río < jaime.f...@gmail.com> wrote: > On Wed, Aug 12, 2015 at 2:03 PM, Nathan Goldbaum < > nathan12...@gmail.com> wrote: > >> Hi all, >> >> I've been testing the package I spend most of my time on, yt, under >> numpy 1.10b1 since the announcement went out. >> >> I think I've narrowed down and fixed all of the test failures that >> cropped up except for one last issue. It seems that the behavior of >> np.digitize with respect to ndarray subclasses has changed since the >> NumPy >> 1.9 series. Consider the following test script: >> >> ```python >> import numpy as np >> >> >> class MyArray(np.ndarray): >> def __new__(cls, *args, **kwargs): >> return np.ndarray.__new__(cls, *args, **kwargs) >> >> data = np.arange(100) >> >> bins = np.arange(100) + 0.5 >> >> data = data.view(MyArray) >> >> bins = bins.view(MyArray) >> >> digits = np.digitize(data, bins) >> >> print type(digits) >> ``` >> >> Under NumPy 1.9.2, this prints "", but under >> the 1.10 beta, it prints "" >> >> I'm curious why this change was made. Since digitize outputs index >> arrays, it doesn't make sense to me why it should return anything but a >> plain ndarray. I see in the release notes that digitize now uses >> searchsorted under the hood. Is this related? >> > > It is indeed searchsorted's fault, as it returns an object of the same > type as the needle (the items to search for): > > >>> import numpy as np > >>> class A(np.ndarray): pass > >>> class B(np.ndarray): pass > >>> np.arange(10).view(A).searchsorted(np.arange(5).view(B)) > B([0, 1, 2, 3, 4]) > > I am all for making index-returning functions always return a base > ndarray, and will be more than happy to send a PR fixing this if there is > some agreement. > I think that is the right thing to do. >>> >>> Awesome, I'd appreciate having a PR to fix this. Arguably the return >>> type *could* be the same type as the inputs, but given that it's a behavior >>> change I agree that it's best to add a patch so the output of serachsorted >>> is "sanitized" to be an ndarray before it's returned by digitize. >>> >> >> It is relatively simple to do, just replace Py_TYPE(ap2) with >> &PyArray_Type in this line: >> >> >> https://github.com/numpy/numpy/blob/maintenance/1.10.x/numpy/core/src/multiarray/item_selection.c#L1725 >> >> Then fix all the tests that are expecting searchsorted to return >> something else than a base ndarray. We already have modified nonzero to >> return base ndarray's in this release, see the release notes, so it will go >> with the same theme. >> >> For 1.11 I think we should try to extend this "if it returns an index, it >> will be a base ndarray" to all other functions that don't right now. Then >> sit back and watch AstroPy come down in flames... ;-))) >> >> Seriously, I think this makes a lot of sense, and should be documented as >> the way NumPy handles index arrays. >> >> Anyway, I will try to find time tonight to put this PR together, unless >> someone beats me to it, which I would be totally fine with. >> > > PR #6206 it is: https://github.com/numpy/numpy/pull/6206 > > Jaime > > -- > (\__/) > ( O.o) > ( > <) Este es Conejo. Copia a Conejo en tu firma y ayúdale en sus planes > de dominación mundial. > > ___ > NumPy-Discussion mailing list > NumPy-Discussion@scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Help in understanding
names == 'Bob' returns a boolean array [True, False, False, True, False, False, False], and data[boolean_array] returns all those elements of data where the boolean array is True. data is a list of 7 lists, so the two lists corresponding to True values are returned. Read the Numpy basics and Advanced Numpy chapters from Python for Data Analysis by Wes McKinney, it is available ol. On Fri, Aug 14, 2015 at 10:31 AM, subro wrote: > Hi, > > I am new to NumPy, Can someone help me in understanding below code. > > >>> names = np.array(['Bob', 'Joe', 'Will', 'Bob', 'Will', 'Joe', 'Joe']) > > >>> data = np.random.random((7,4)) > > >>> print data > [[ 0.85402649 0.12827655 0.580 0.86288236] > [ 0.30162683 0.45269508 0.98098039 0.1291469 ] > [ 0.21229924 0.37497112 0.57367496 0.08607771] > [ 0.3028660.42160468 0.26879288 0.68032467] > [ 0.60612492 0.35210577 0.91355096 0.57872181] > [ 0.11583826 0.8192 0.39214077 0.51377566] > [ 0.03767641 0.1920532 0.24872009 0.36068313]] > > >>> data[names == 'Bob'] > > array([[ 0.85402649, 0.12827655, 0.580 , 0.86288236], >[ 0.302866 , 0.42160468, 0.26879288, 0.68032467]]) > > Also, can someone help me where and how to practice NumPy? > > > > -- > View this message in context: > http://numpy-discussion.10968.n7.nabble.com/Help-in-understanding-tp40827.html > Sent from the Numpy-discussion mailing list archive at Nabble.com. > ___ > NumPy-Discussion mailing list > NumPy-Discussion@scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
