Re: [Numpy-discussion] Custom dtypes without C -- or, a standard ndarray-like type

2014-09-24 Thread Chris Barker
On Tue, Sep 23, 2014 at 4:40 AM, Eric Moore e...@redtetrahedron.org wrote: Improving the dtype system requires working on c code. yes -- it sure does. But I think that is a bit of a Red Herring. I'm barely competent in C, and don't like it much, but the real barrier to entry for me is not

Re: [Numpy-discussion] Custom dtypes without C -- or, a standard ndarray-like type

2014-09-24 Thread Travis Oliphant
This could actually be done by using the structured dtype pretty easily. The hard work would be improving the ufunc and generalized ufunc mechanism to handle structured data-types. Numba actually provides some of this already, so if you have NumPy + Numba you can do this sort of thing now.

Re: [Numpy-discussion] Custom dtypes without C -- or, a standard ndarray-like type

2014-09-23 Thread Stephan Hoyer
On Sun, Sep 21, 2014 at 8:31 PM, Nathaniel Smith n...@pobox.com wrote: For cases where people genuinely want to implement a new array-like types (e.g. DataFrame or scipy.sparse) then numpy provides a fair amount of support for this already (e.g., the various hooks that allow things like

Re: [Numpy-discussion] Custom dtypes without C -- or, a standard ndarray-like type

2014-09-23 Thread David Cournapeau
On Mon, Sep 22, 2014 at 4:31 AM, Nathaniel Smith n...@pobox.com wrote: On Sun, Sep 21, 2014 at 7:50 PM, Stephan Hoyer sho...@gmail.com wrote: pandas has some hacks to support custom types of data for which numpy can't handle well enough or at all. Examples include datetime and Categorical

Re: [Numpy-discussion] Custom dtypes without C -- or, a standard ndarray-like type

2014-09-23 Thread Todd
On Mon, Sep 22, 2014 at 5:31 AM, Nathaniel Smith n...@pobox.com wrote: On Sun, Sep 21, 2014 at 7:50 PM, Stephan Hoyer sho...@gmail.com wrote: My feeling though is that in most of the cases you mention, implementing a new array-like type is huge overkill. ndarray's interface is vast and

Re: [Numpy-discussion] Custom dtypes without C -- or, a standard ndarray-like type

2014-09-23 Thread Eric Moore
On Tuesday, September 23, 2014, Todd toddr...@gmail.com wrote: On Mon, Sep 22, 2014 at 5:31 AM, Nathaniel Smith n...@pobox.com javascript:_e(%7B%7D,'cvml','n...@pobox.com'); wrote: On Sun, Sep 21, 2014 at 7:50 PM, Stephan Hoyer sho...@gmail.com

Re: [Numpy-discussion] Custom dtypes without C -- or, a standard ndarray-like type

2014-09-23 Thread Travis Oliphant
On Sun, Sep 21, 2014 at 6:50 PM, Stephan Hoyer sho...@gmail.com wrote: pandas has some hacks to support custom types of data for which numpy can't handle well enough or at all. Examples include datetime and Categorical [1], and others like GeoArray [2] that haven't make it into pandas yet.

Re: [Numpy-discussion] Custom dtypes without C -- or, a standard ndarray-like type

2014-09-23 Thread Benjamin Root
Travis, Thank you for your perspective on this issue. Such input is always valuable in helping us see where we came from and where we might go. My perspective on NumPy is fairly different, having come into Python right after the whole Numeric/NumArray transition to NumPy. One of the things that

Re: [Numpy-discussion] Custom dtypes without C -- or, a standard ndarray-like type

2014-09-22 Thread Jeff Reback
Hopefully this is not TL;DR! Their are 3 'dtype' likes that exist in pandas that could in theory mostly be migrated back to numpy. These currently exist as the .values in-other-words the object to which pandas defers data storage and computation for some/most of operations. 1) SparseArray: This

[Numpy-discussion] Custom dtypes without C -- or, a standard ndarray-like type

2014-09-21 Thread Stephan Hoyer
pandas has some hacks to support custom types of data for which numpy can't handle well enough or at all. Examples include datetime and Categorical [1], and others like GeoArray [2] that haven't make it into pandas yet. Most of these look like numpy arrays but with custom dtypes and type specific

Re: [Numpy-discussion] Custom dtypes without C -- or, a standard ndarray-like type

2014-09-21 Thread Charles R Harris
On Sun, Sep 21, 2014 at 5:50 PM, Stephan Hoyer sho...@gmail.com wrote: pandas has some hacks to support custom types of data for which numpy can't handle well enough or at all. Examples include datetime and Categorical [1], and others like GeoArray [2] that haven't make it into pandas yet.

Re: [Numpy-discussion] Custom dtypes without C -- or, a standard ndarray-like type

2014-09-21 Thread Nathaniel Smith
On Sun, Sep 21, 2014 at 7:50 PM, Stephan Hoyer sho...@gmail.com wrote: pandas has some hacks to support custom types of data for which numpy can't handle well enough or at all. Examples include datetime and Categorical [1], and others like GeoArray [2] that haven't make it into pandas yet.