[Numpy-discussion] ANN: pandas 0.11.0 released!

2013-04-23 Thread Wes McKinney
hat's new: http://pandas.pydata.org/pandas-docs/stable/whatsnew.html Installers: http://pypi.python.org/pypi/pandas $ git log v0.10.1..v0.11.0 --pretty=format:%aN | sort | uniq -c | sort -rn 308 y-p 279 jreback 85 Vytautas Jancauskas 74 Wes McKinney 25 Stephen Lin 22

[Numpy-discussion] ANN: pandas 0.10.1 is released

2013-01-22 Thread Wes McKinney
lers: http://pypi.python.org/pypi/pandas $ git log v0.10.0..v0.10.1 --pretty=format:%aN | sort | uniq -c | sort -rn 66 jreback 59 Wes McKinney 43 Chang She 12 y-p 5 Vincent Arel-Bundock 4 Damien Garaud 3 Christopher Whelan 3 Andy Hayden 2 Jay Parlar 2

[Numpy-discussion] ANN: pandas 0.10.0 released

2012-12-17 Thread Wes McKinney
pypi.python.org/pypi/pandas $ git log v0.9.1..v0.10.0 --pretty=format:%aN | sort | uniq -c | sort -rn 246 Wes McKinney 140 y-p 99 Chang She 45 jreback 18 Abraham Flaxman 17 Jeff Reback 14 locojaydev 11 Keith Hughitt 5 Adam Obeng 2 Dieter Vandenbussc

Re: [Numpy-discussion] memory-efficient loadtxt

2012-10-03 Thread Wes McKinney
On Monday, October 1, 2012, Chris Barker wrote: > Paul, > > Nice to see someone working on these issues, but: > > I'm not sure the problem you are trying to solve -- accumulating in a > list is pretty efficient anyway -- not a whole lot overhead. > > But if you do want to improve that, it may be b

Re: [Numpy-discussion] NumPy 1.7 release delays

2012-06-26 Thread Wes McKinney
On Tue, Jun 26, 2012 at 8:15 PM, Skipper Seabold wrote: > On Tue, Jun 26, 2012 at 7:59 PM, Fernando Perez wrote: >> On Tue, Jun 26, 2012 at 1:10 PM, Travis Oliphant wrote: >>> One issues is the one that Sage identified about the array interface >>> regression as noted by Jason.    Any other regr

Re: [Numpy-discussion] Created NumPy 1.7.x branch

2012-06-21 Thread Wes McKinney
On Thu, Jun 21, 2012 at 2:49 PM, Ralf Gommers wrote: > > > On Thu, Jun 21, 2012 at 5:25 PM, Travis Oliphant > wrote: >> >> I thought it was clear we were doing a 1.7 release before SciPy.   It >> seems pretty urgent that we get something out sooner than later.      I know >> there is never enough

Re: [Numpy-discussion] Enum/Factor NEP (now with code)

2012-06-17 Thread Wes McKinney
On Sun, Jun 17, 2012 at 6:10 AM, Nathaniel Smith wrote: > On Wed, Jun 13, 2012 at 7:54 PM, Wes McKinney wrote: >> It looks like the levels can only be strings. This is too limited for >> my needs. Why not support all possible NumPy dtypes? In pandas world, >> the levels ca

Re: [Numpy-discussion] Enum/Factor NEP (now with code)

2012-06-13 Thread Wes McKinney
On Wed, Jun 13, 2012 at 5:19 PM, Bryan Van de Ven wrote: > On 6/13/12 1:54 PM, Wes McKinney wrote: >> OK, I need to spend some time on this as it will directly impact me. >> Random thoughts here. >> >> It looks like the levels can only be strings. This is too limit

Re: [Numpy-discussion] Enum/Factor NEP (now with code)

2012-06-13 Thread Wes McKinney
On Wed, Jun 13, 2012 at 2:12 PM, Nathaniel Smith wrote: > On Wed, Jun 13, 2012 at 5:44 PM, Bryan Van de Ven wrote: >> On 6/13/12 8:33 AM, Nathaniel Smith wrote: >>> Hi Bryan, >>> >>> I skimmed over the diff: >>>     https://github.com/bryevdv/numpy/compare/master...enum >>> It was a bit hard to r

Re: [Numpy-discussion] Status of np.bincount

2012-05-03 Thread Wes McKinney
On Thu, May 3, 2012 at 12:51 PM, Tony Yu wrote: > > > On Thu, May 3, 2012 at 9:57 AM, Robert Kern wrote: >> >> On Thu, May 3, 2012 at 2:50 PM, Robert Elsner wrote: >> > >> > Am 03.05.2012 15:45, schrieb Robert Kern: >> >> On Thu, May 3, 2012 at 2:24 PM, Robert Elsner >> >> wrote: >> >>> Hello E

Re: [Numpy-discussion] Issue Tracking

2012-05-02 Thread Wes McKinney
On Wed, May 2, 2012 at 9:48 AM, Charles R Harris wrote: > > > On Tue, May 1, 2012 at 11:47 PM, Ralf Gommers > wrote: >> >> >> >> On Wed, May 2, 2012 at 1:48 AM, Pauli Virtanen wrote: >>> >>> 01.05.2012 21:34, Ralf Gommers kirjoitti: >>> [clip] >>> > At this point it's probably good to look again

Re: [Numpy-discussion] datetime dtype possible regression

2012-04-29 Thread Wes McKinney
On Sat, Apr 28, 2012 at 11:18 AM, Charles R Harris wrote: > > > On Sat, Apr 28, 2012 at 9:13 AM, Wes McKinney wrote: >> >> On Fri, Apr 27, 2012 at 4:57 PM, Robert Kern >> wrote: >> > On Fri, Apr 27, 2012 at 21:52, Travis Vaught wrote: >> >> With

Re: [Numpy-discussion] datetime dtype possible regression

2012-04-28 Thread Wes McKinney
On Fri, Apr 27, 2012 at 4:57 PM, Robert Kern wrote: > On Fri, Apr 27, 2012 at 21:52, Travis Vaught wrote: >> With NumPy 1.6.1 (from EPD 7.2-2) I get this behavior: >> >> >> ~ >> >> In [1]: import numpy as np >> >> In [2]: schema = np.dtype({'names':['symbol', 'date

Re: [Numpy-discussion] Improving NumPy's indexing / subsetting / fancy indexing implementation

2012-04-07 Thread Wes McKinney
#x27;t know if it affects >> anything you're talking about here, but just as a heads up, you might >> want to benchmark master, since it may have a different performance >> profile. >> >> -- Nathaniel >> >> On Fri, Apr 6, 2012 at 4:04 AM, Wes McKinney wr

[Numpy-discussion] Improving NumPy's indexing / subsetting / fancy indexing implementation

2012-04-05 Thread Wes McKinney
dear all, I've routinely found that: 1) ndarray.take is up to 1 order of magnitude faster than fancy indexing 2) Hand-coded Cython boolean indexing is many times faster than ndarray indexing 3) putmask is significantly faster than ndarray indexing For example, I stumbled on this tonight: strai

Re: [Numpy-discussion] draft enum NEP

2012-03-11 Thread Wes McKinney
On Fri, Mar 9, 2012 at 5:48 PM, David Gowers (kampu) <00a...@gmail.com> wrote: > Hi, > > On Sat, Mar 10, 2012 at 3:25 AM, Bryan Van de Ven wrote: >> Hi all, >> >> I have started working on a NEP for adding an enumerated type to NumPy. >> It is on my GitHub: >> >>     https://github.com/bryevdv/num

Re: [Numpy-discussion] Migrating issues to GitHub

2012-02-26 Thread Wes McKinney
On Thu, Feb 16, 2012 at 4:32 PM, Ralf Gommers wrote: > > > On Thu, Feb 16, 2012 at 10:20 PM, Thouis (Ray) Jones > wrote: >> >> On Thu, Feb 16, 2012 at 19:25, Ralf Gommers >> wrote: >> > In another thread Jira was proposed as an alternative to Trac. Can you >> > point >> > out some of its strengt

Re: [Numpy-discussion] Possible roadmap addendum: building better text file readers

2012-02-25 Thread Wes McKinney
On Fri, Feb 24, 2012 at 9:07 AM, Erin Sheldon wrote: > Excerpts from Travis Oliphant's message of Thu Feb 23 15:08:52 -0500 2012: >> This is actually on my short-list as well --- it just didn't make it to the >> list. >> >> In fact, we have someone starting work on it this week.  It is his >> fir

Re: [Numpy-discussion] Possible roadmap addendum: building better text file readers

2012-02-23 Thread Wes McKinney
On Thu, Feb 23, 2012 at 4:20 PM, Erin Sheldon wrote: > Excerpts from Wes McKinney's message of Thu Feb 23 16:07:04 -0500 2012: >> That's pretty good. That's faster than pandas's csv-module+Cython >> approach almost certainly (but I haven't run your code to get a read >> on how much my hardware mak

Re: [Numpy-discussion] Possible roadmap addendum: building better text file readers

2012-02-23 Thread Wes McKinney
On Thu, Feb 23, 2012 at 3:55 PM, Erin Sheldon wrote: > Excerpts from Wes McKinney's message of Thu Feb 23 15:45:18 -0500 2012: >> Reasonably wide CSV files with hundreds of thousands to millions of >> rows. I have a separate interest in JSON handling but that is a >> different kind of problem, and

Re: [Numpy-discussion] Possible roadmap addendum: building better text file readers

2012-02-23 Thread Wes McKinney
On Thu, Feb 23, 2012 at 3:31 PM, Éric Depagne wrote: > Le jeudi 23 février 2012 21:24:28, Wes McKinney a écrit : >> > That would indeed be great. Reading large files is a real pain whatever the > python method used. > > BTW, could you tell us what you mean by large file

Re: [Numpy-discussion] Possible roadmap addendum: building better text file readers

2012-02-23 Thread Wes McKinney
On Thu, Feb 23, 2012 at 3:23 PM, Erin Sheldon wrote: > Wes - > > I designed the recfile package to fill this need.  It might be a start. > > Some features: > >    - the ability to efficiently read any subset of the data without >      loading the whole file. >    - reads directly into a recarray,

Re: [Numpy-discussion] Possible roadmap addendum: building better text file readers

2012-02-23 Thread Wes McKinney
realistic performance test cases that we can monitor via vbench (wesm/vbench) while we're work on the project. - W > >> >> >> On Feb 23, 2012, at 1:53 PM, Pauli Virtanen wrote: >> >> > Hi, >> > >> > 23.02.2012 20:32, Wes McKinney kirjoi

Re: [Numpy-discussion] Possible roadmap addendum: building better text file readers

2012-02-23 Thread Wes McKinney
unique string value occurring) where structured arrays cannot, so you get much better performance and memory usage that way. That's soon to change, though, I gather, at which point I'll almost definitely (!) move to pointer arrays instead of dtype=object arrays. - Wes > > >

[Numpy-discussion] Possible roadmap addendum: building better text file readers

2012-02-23 Thread Wes McKinney
dear all, I haven't read all 180 e-mails, but I didn't see this on Travis's initial list. All of the existing flat file reading solutions I have seen are not suitable for many applications, and they compare very unfavorably to tools present in other languages, like R. Here are some of the main is

Re: [Numpy-discussion] Index Array Performance

2012-02-14 Thread Wes McKinney
On Tue, Feb 14, 2012 at 4:03 AM, Francesc Alted wrote: > On Feb 14, 2012, at 1:50 AM, Wes McKinney wrote: > [clip] >> But: >> >> In [40]: timeit hist[i, j] >> 1 loops, best of 3: 32 us per loop >> >> So that's roughly 7-8x slower than a simple Cy

Re: [Numpy-discussion] Index Array Performance

2012-02-13 Thread Wes McKinney
On Mon, Feb 13, 2012 at 7:48 PM, Wes McKinney wrote: > On Mon, Feb 13, 2012 at 7:46 PM, Wes McKinney wrote: >> On Mon, Feb 13, 2012 at 7:30 PM, Nathaniel Smith wrote: >>> How would you fix it? I shouldn't speculate without profiling, but I'll be >>> na

Re: [Numpy-discussion] Index Array Performance

2012-02-13 Thread Wes McKinney
On Mon, Feb 13, 2012 at 7:46 PM, Wes McKinney wrote: > On Mon, Feb 13, 2012 at 7:30 PM, Nathaniel Smith wrote: >> How would you fix it? I shouldn't speculate without profiling, but I'll be >> naughty. Presumably the problem is that python turns that into something >&

Re: [Numpy-discussion] Index Array Performance

2012-02-13 Thread Wes McKinney
buy it: In [33]: timeit arr.__iadd__(1) 1000 loops, best of 3: 1.13 ms per loop In [37]: timeit arr[:] += 1 1000 loops, best of 3: 1.13 ms per loop - Wes > On Feb 14, 2012 12:18 AM, "Wes McKinney" wrote: >> >> On Mon, Feb 13, 2012 at 6:23 PM, Marcel Oliver >> wro

Re: [Numpy-discussion] Index Array Performance

2012-02-13 Thread Wes McKinney
On Mon, Feb 13, 2012 at 6:23 PM, Marcel Oliver wrote: > Hi, > > I have a short piece of code where the use of an index array "feels > right", but incurs a severe performance penalty: It's about an order > of magnitude slower than all other operations with arrays of that > size. > > It comes up in

Re: [Numpy-discussion] fast method to to count a particular value in a large matrix

2012-02-05 Thread Wes McKinney
On Sun, Feb 5, 2012 at 7:02 PM, David Verelst wrote: > Just out of curiosity, what speed-up factor did you achieve? > > Regards, > David > > On 04/02/12 22:20, Naresh wrote: >> Warren Weckesser  enthought.com>  writes: >> >>> >>> On Sat, Feb 4, 2012 at 2:35 PM, Benjamin Root  ou.edu>  wrote: >>> >

Re: [Numpy-discussion] Numpy 'groupby'

2012-01-11 Thread Wes McKinney
On Wed, Jan 11, 2012 at 7:05 AM, Neal Becker wrote: > Michael Hull wrote: > >> Hi Everyone, >> First off, thanks for all your hard work on numpy, its a really great help! >> I was wondering if there was a standard 'groupby' in numpy, that >> similar to that in itertools. >> I know its not hard to

Re: [Numpy-discussion] PyInt and Numpy's int64 conversion

2012-01-06 Thread Wes McKinney
On Wed, Jan 4, 2012 at 5:22 AM, xantares 09 wrote: > > >> From: wesmck...@gmail.com >> Date: Sat, 24 Dec 2011 19:51:06 -0500 > >> To: numpy-discussion@scipy.org >> Subject: Re: [Numpy-discussion] PyInt and Numpy's int64 conversion >> >> On Sat, Dec 24, 2011 at 3:11 AM, xantares 09 >> wrote: >> >

Re: [Numpy-discussion] Enum type

2012-01-03 Thread Wes McKinney
On Tue, Jan 3, 2012 at 1:06 PM, Jim Vickroy wrote: > On 1/3/2012 10:46 AM, Ognen Duzlevski wrote: >> Hello, >> >> I am playing with adding an enum dtype to numpy (to get my feet wet in >> numpy really). I have looked at the >> https://github.com/martinling/numpy_quaternion and I feel comfortable >

Re: [Numpy-discussion] Enum type

2012-01-03 Thread Wes McKinney
On Tue, Jan 3, 2012 at 12:46 PM, Ognen Duzlevski wrote: > Hello, > > I am playing with adding an enum dtype to numpy (to get my feet wet in > numpy really). I have looked at the > https://github.com/martinling/numpy_quaternion and I feel comfortable > with my understanding of adding a simple type

Re: [Numpy-discussion] GSOC

2011-12-29 Thread Wes McKinney
On Thu, Dec 29, 2011 at 4:36 PM, Ralf Gommers wrote: > > > On Thu, Dec 29, 2011 at 9:50 PM, Charles R Harris > wrote: >> >> Hi All, >> >> I thought I'd raise this topic just to get some ideas out there. At the >> moment I see two areas that I'd like to see addressed. >> >> Documentation editor. T

Re: [Numpy-discussion] PyInt and Numpy's int64 conversion

2011-12-24 Thread Wes McKinney
On Sat, Dec 24, 2011 at 3:11 AM, xantares 09 wrote: > > >> From: wesmck...@gmail.com >> Date: Fri, 23 Dec 2011 12:31:45 -0500 >> To: numpy-discussion@scipy.org >> Subject: Re: [Numpy-discussion] PyInt and Numpy's int64 conversion > >> >> On Fri, Dec 23, 2011 at 4:37 AM, xantares 09 >> wrote: >> >

Re: [Numpy-discussion] PyInt and Numpy's int64 conversion

2011-12-23 Thread Wes McKinney
On Fri, Dec 23, 2011 at 4:37 AM, xantares 09 wrote: > Hi, > > I'm using Numpy from the C python api side while tweaking my SWIG interface > to work with numpy array types. > I want to convert a numpy array of integers (whose elements are numpy's > 'int64') > The problem is that it this int64 type

Re: [Numpy-discussion] Fast Reading of ASCII files

2011-12-13 Thread Wes McKinney
On Mon, Dec 12, 2011 at 12:34 PM, Warren Weckesser wrote: > > > On Mon, Dec 12, 2011 at 10:22 AM, Chris.Barker > wrote: >> >> On 12/11/11 8:40 AM, Ralf Gommers wrote: >> > On Wed, Dec 7, 2011 at 7:50 PM, Chris.Barker > >     * If we have a good, fast ascii (or unicode?) to array reader, >> > hope

Re: [Numpy-discussion] numpy 1.7.0 release?

2011-12-06 Thread Wes McKinney
On Tue, Dec 6, 2011 at 4:11 PM, Ralf Gommers wrote: > > > On Mon, Dec 5, 2011 at 8:43 PM, Ralf Gommers > wrote: >> >> Hi all, >> >> It's been a little over 6 months since the release of 1.6.0 and the NA >> debate has quieted down, so I'd like to ask your opinion on the timing of >> 1.7.0. It look

Re: [Numpy-discussion] consensus (was: NA masks in the next numpy release?)

2011-10-29 Thread Wes McKinney
On Fri, Oct 28, 2011 at 9:32 PM, Charles R Harris wrote: > > > On Fri, Oct 28, 2011 at 6:45 PM, Wes McKinney wrote: >> >> On Fri, Oct 28, 2011 at 7:53 PM, Benjamin Root wrote: >> > >> > >> > On Friday, October 28, 2011, Matthew Brett >> &g

Re: [Numpy-discussion] consensus (was: NA masks in the next numpy release?)

2011-10-28 Thread Wes McKinney
On Fri, Oct 28, 2011 at 7:53 PM, Benjamin Root wrote: > > > On Friday, October 28, 2011, Matthew Brett wrote: >> Hi, >> >> On Fri, Oct 28, 2011 at 4:21 PM, Ralf Gommers >> wrote: >>> >>> >>> On Sat, Oct 29, 2011 at 12:37 AM, Matthew Brett >>> wrote: Hi, On Fri, Oct 28, 2011

Re: [Numpy-discussion] NA masks in the next numpy release?

2011-10-24 Thread Wes McKinney
On Mon, Oct 24, 2011 at 10:54 AM, Charles R Harris wrote: > > > On Mon, Oct 24, 2011 at 8:40 AM, Charles R Harris > wrote: >> >> >> On Sun, Oct 23, 2011 at 11:23 PM, Wes McKinney >> wrote: >>> >>> On Sun, Oct 23, 2011 at 8:07 PM, Eric Firing

Re: [Numpy-discussion] NA masks in the next numpy release?

2011-10-23 Thread Wes McKinney
On Sun, Oct 23, 2011 at 8:07 PM, Eric Firing wrote: > On 10/23/2011 12:34 PM, Nathaniel Smith wrote: > >> like. And in this case I do think we can come up with an API that will >> make everyone happy, but that Mark's current API probably can't be >> incrementally evolved to become that API.) >> >

Re: [Numpy-discussion] Preventing an ndarray subclass from returning new subclass instances for std(), etc

2011-09-18 Thread Wes McKinney
On Sun, Sep 18, 2011 at 4:25 PM, Pierre GM wrote: > > On Sep 18, 2011, at 21:25 , Stéfan van der Walt wrote: > >> On Sun, Sep 18, 2011 at 9:48 AM, Keith Hughitt >> wrote: >>> Interesting. It works as expected when called as a method: >>> >>> In [10]: x = np.ma.array([[1,2,3]]) >>> In [11]: x.std

Re: [Numpy-discussion] Bug in numpy std, etc. with other data structures?

2011-09-17 Thread Wes McKinney
On Sat, Sep 17, 2011 at 10:50 PM, Bruce Southey wrote: > On Sat, Sep 17, 2011 at 4:12 PM, Wes McKinney wrote: >> On Sat, Sep 17, 2011 at 4:48 PM, Skipper Seabold wrote: >>> Just ran into this. Any objections for having numpy.std and other >>> functions in core/fro

Re: [Numpy-discussion] Bug in numpy std, etc. with other data structures?

2011-09-17 Thread Wes McKinney
On Sat, Sep 17, 2011 at 8:36 PM, wrote: > On Sat, Sep 17, 2011 at 5:12 PM, Wes McKinney wrote: >> On Sat, Sep 17, 2011 at 4:48 PM, Skipper Seabold wrote: >>> Just ran into this. Any objections for having numpy.std and other >>> functions in core/fromnumeric.py call

Re: [Numpy-discussion] Bug in numpy std, etc. with other data structures?

2011-09-17 Thread Wes McKinney
On Sat, Sep 17, 2011 at 4:48 PM, Skipper Seabold wrote: > Just ran into this. Any objections for having numpy.std and other > functions in core/fromnumeric.py call asanyarray before trying to use > the array's method? Other data structures like pandas and larry define > their own std method, for i

Re: [Numpy-discussion] NA masks for NumPy are ready to test

2011-08-24 Thread Wes McKinney
On Wed, Aug 24, 2011 at 8:19 PM, Mark Wiebe wrote: > On Fri, Aug 19, 2011 at 11:37 AM, Bruce Southey wrote: >> >> Hi, >> >> >> 2) Can the 'skipna' flag be added to the methods? >> >>> a.sum(skipna=True) >> Traceback (most recent call last): >>  File "", line 1, in >> TypeError: 'skipna' is an i

Re: [Numpy-discussion] Questionable reduceat behavior

2011-08-16 Thread Wes McKinney
On Sun, Aug 14, 2011 at 11:58 AM, Wes McKinney wrote: > On Sat, Aug 13, 2011 at 8:06 PM, Mark Wiebe wrote: >> Looks like this is the second-oldest open bug in the bug tracker. >> http://projects.scipy.org/numpy/ticket/236 >> For what it's worth, I'm in favour of ch

Re: [Numpy-discussion] Questionable reduceat behavior

2011-08-14 Thread Wes McKinney
et. > -Mark > > On Thu, Aug 11, 2011 at 11:25 AM, Wes McKinney wrote: >> >> I'm a little perplexed why reduceat was made to behave like this: >> >> In [26]: arr = np.ones((10, 4), dtype=bool) >> >> In [27]: arr >> Out[27]: >> array([

[Numpy-discussion] Questionable reduceat behavior

2011-08-11 Thread Wes McKinney
I'm a little perplexed why reduceat was made to behave like this: In [26]: arr = np.ones((10, 4), dtype=bool) In [27]: arr Out[27]: array([[ True, True, True, True], [ True, True, True, True], [ True, True, True, True], [ True, True, True, True], [ True, T

Re: [Numpy-discussion] Warning: invalid value encountered in divide

2011-08-08 Thread Wes McKinney
On Mon, Aug 8, 2011 at 7:01 PM, Neal Becker wrote: > Warning: invalid value encountered in divide > > No traceback.  How can I get more info on this?  Can this warning be converted > to an exception so I can get a trace? > > ___ > NumPy-Discussion mailin

Re: [Numpy-discussion] inconsistent semantics for double-slicing

2011-07-27 Thread Wes McKinney
On Wed, Jul 27, 2011 at 5:36 PM, Alex Flint wrote: > When applying two different slicing operations in succession (e.g. select a > sub-range, then select using a binary mask) it seems that numpy arrays can > be inconsistent with respect to assignment: > For example, in this case an array is modifi

Re: [Numpy-discussion] named ndarray axes

2011-07-13 Thread Wes McKinney
On Wed, Jul 13, 2011 at 7:15 PM, Craig Yoshioka wrote: > Yup exactly.  To enable this sort of tracking I needed to explicitly > reverse-engineer the effects of indexing on axes.  I figure overriding > indexing catches most cases that modify axes, but other holes need to be > plugged as well...

Re: [Numpy-discussion] Concepts for masked/missing data

2011-06-25 Thread Wes McKinney
On Sat, Jun 25, 2011 at 3:51 PM, Nathaniel Smith wrote: > On Sat, Jun 25, 2011 at 11:32 AM, Benjamin Root wrote: >> On Sat, Jun 25, 2011 at 12:05 PM, Nathaniel Smith wrote: >>> I guess that is a difference, but I'm trying to get at something more >>> fundamental -- not just what operations are a

Re: [Numpy-discussion] Concepts for masked/missing data

2011-06-25 Thread Wes McKinney
On Sat, Jun 25, 2011 at 1:05 PM, Nathaniel Smith wrote: > On Sat, Jun 25, 2011 at 9:26 AM, Matthew Brett > wrote: >> So far I see the difference between 1) and 2) being that you cannot >> unmask.  So, if you didn't even know you could unmask data, then it >> would not matter that 1) was being im

Re: [Numpy-discussion] feedback request: proposal to add masks to the core ndarray

2011-06-25 Thread Wes McKinney
On Sat, Jun 25, 2011 at 10:25 AM, Charles R Harris wrote: > > > On Sat, Jun 25, 2011 at 8:14 AM, Wes McKinney wrote: >> >> On Sat, Jun 25, 2011 at 12:42 AM, Charles R Harris >> wrote: >> > >> > >> > On Fri, Jun 24, 2011 at 10:06 PM, Wes McK

Re: [Numpy-discussion] feedback request: proposal to add masks to the core ndarray

2011-06-25 Thread Wes McKinney
On Sat, Jun 25, 2011 at 12:42 AM, Charles R Harris wrote: > > > On Fri, Jun 24, 2011 at 10:06 PM, Wes McKinney wrote: >> >> On Fri, Jun 24, 2011 at 11:59 PM, Nathaniel Smith wrote: >> > On Fri, Jun 24, 2011 at 6:57 PM, Benjamin Root wrote: >> >> On Fri

Re: [Numpy-discussion] feedback request: proposal to add masks to the core ndarray

2011-06-24 Thread Wes McKinney
On Fri, Jun 24, 2011 at 11:59 PM, Nathaniel Smith wrote: > On Fri, Jun 24, 2011 at 6:57 PM, Benjamin Root wrote: >> On Fri, Jun 24, 2011 at 8:11 PM, Nathaniel Smith wrote: >>> This is a situation where I would just... use an array and a mask, >>> rather than a masked array. Then lots of things -

Re: [Numpy-discussion] feedback request: proposal to add masks to the core ndarray

2011-06-24 Thread Wes McKinney
On Fri, Jun 24, 2011 at 8:02 PM, Charles R Harris wrote: > > > On Fri, Jun 24, 2011 at 5:22 PM, Wes McKinney wrote: >> >> On Fri, Jun 24, 2011 at 7:10 PM, Charles R Harris >> wrote: >> > >> > >> > On Fri, Jun 24, 2011 at 4:21 PM, Matthew

Re: [Numpy-discussion] feedback request: proposal to add masks to the core ndarray

2011-06-24 Thread Wes McKinney
On Fri, Jun 24, 2011 at 7:10 PM, Charles R Harris wrote: > > > On Fri, Jun 24, 2011 at 4:21 PM, Matthew Brett > wrote: >> >> Hi, >> >> On Fri, Jun 24, 2011 at 10:09 PM, Benjamin Root wrote: >> ... >> > Again, there are pros and cons either way and I see them very orthogonal >> > and >> > complem

Re: [Numpy-discussion] feedback request: proposal to add masks to the core ndarray

2011-06-24 Thread Wes McKinney
On Fri, Jun 24, 2011 at 12:33 PM, Mark Wiebe wrote: > On Thu, Jun 23, 2011 at 8:32 PM, Nathaniel Smith wrote: >> >> On Thu, Jun 23, 2011 at 5:21 PM, Mark Wiebe wrote: >> > On Thu, Jun 23, 2011 at 7:00 PM, Nathaniel Smith wrote: >> >> It's should also be possible to accomplish a general solution

Re: [Numpy-discussion] fixing up datetime

2011-06-09 Thread Wes McKinney
On Wed, Jun 8, 2011 at 8:53 PM, Mark Wiebe wrote: > On Wed, Jun 8, 2011 at 4:57 AM, Wes McKinney wrote: >> >> >> >> >> So in summary, w.r.t. time series data and datetime, the only things I >> care about from a datetime / pandas point of view: >> &g

Re: [Numpy-discussion] fixing up datetime

2011-06-08 Thread Wes McKinney
On Wed, Jun 8, 2011 at 6:37 AM, Dave Hirschfeld wrote: > Wes McKinney gmail.com> writes: > >> >> > >> > - Fundamental need to be able to work with multiple time series, >> > especially performing operations involving cross-sectional data >> > -

Re: [Numpy-discussion] fixing up datetime

2011-06-08 Thread Wes McKinney
On Wed, Jun 8, 2011 at 11:57 AM, Wes McKinney wrote: > On Wed, Jun 8, 2011 at 7:36 AM, Chris Barker wrote: >> On 6/7/11 4:53 PM, Pierre GM wrote: >>>  Anyhow, each time yo >>> read 'frequency' in scikits.timeseries, think 'unit'. >> >&g

Re: [Numpy-discussion] fixing up datetime

2011-06-08 Thread Wes McKinney
On Wed, Jun 8, 2011 at 7:36 AM, Chris Barker wrote: > On 6/7/11 4:53 PM, Pierre GM wrote: >>  Anyhow, each time yo >> read 'frequency' in scikits.timeseries, think 'unit'. > > or maybe "precision" -- when I think if unit, I think of something that > can be represented as a floating point value --

Re: [Numpy-discussion] fixing up datetime

2011-06-06 Thread Wes McKinney
On Mon, Jun 6, 2011 at 8:16 AM, Mark Dickinson wrote: > On Thu, Jun 2, 2011 at 5:42 PM, Mark Wiebe wrote: >> Leap years are easy compared with leap seconds. Leap seconds involve a >> hardcoded table of particular leap-seconds that are added or subtracted, and >> are specified roughly 6 months in

Re: [Numpy-discussion] fixing up datetime

2011-06-01 Thread Wes McKinney
On Wed, Jun 1, 2011 at 10:29 PM, Charles R Harris wrote: > > > On Wed, Jun 1, 2011 at 3:16 PM, Mark Wiebe wrote: >> >> On Wed, Jun 1, 2011 at 3:52 PM, Charles R Harris >> wrote: >>> >>> >>> Just a quick comment, as this really needs more thought, but time is a >>> bag of worms. >> >> Certainly

Re: [Numpy-discussion] labeled axes

2011-05-26 Thread Wes McKinney
On Tue, May 24, 2011 at 6:39 PM, Craig Yoshioka wrote: > Hi all, > I've read some discussions about adding labeled axes, and even ticks, to > numpy arrays  (such as in Luis' dataarray). > I have recently found that the ability to label axes would be very helpful > to me, but I'd like to keep the i

Re: [Numpy-discussion] datarray -- a couple of questions

2011-05-22 Thread Wes McKinney
On Sun, May 22, 2011 at 10:49 AM, Ben Walsh wrote: > Hi > > I've been looking at various labelled-array packages, including Pandas > (https://github.com/wesm/pandas) and datarray > (https://github.com/fperez/datarray). > > I was quite interested in the design discussion for datarray, and I had > a

[Numpy-discussion] Bug? NumPy types circumvent some overloaded methods in ndarray subclasses

2011-05-07 Thread Wes McKinney
This strikes me as a bug-- haven't checked NumPy 1.6 yet but this happens in 1.5.1. Here's a toy example: class MyNdarray(np.ndarray): def __new__(cls, data): subarr = np.array(data, dtype=np.float64).view(cls) return subarr def __radd__(self, other): print 'hello

Re: [Numpy-discussion] Data standardizing

2011-04-13 Thread Wes McKinney
On Wed, Apr 13, 2011 at 9:50 AM, Jonathan Rocher wrote: > Hi, > > I assume you have this data in a txt file, correct? You can load up all of > it in a numpy array using > import numpy as np > data = np.loadtxt("climat_file.txt", skiprows = 1) > > Then you can compute the mean you want by taking it

Re: [Numpy-discussion] Remove deprecated skiprows

2011-04-05 Thread Wes McKinney
On Tue, Apr 5, 2011 at 7:32 PM, Robert Kern wrote: > On Tue, Apr 5, 2011 at 18:20, Wes McKinney wrote: > >> FYI: >> >> http://docs.python.org/library/warnings.html >> >> "DeprecationWarning     Base category for warnings about deprecated >&

Re: [Numpy-discussion] Remove deprecated skiprows

2011-04-05 Thread Wes McKinney
On Tue, Apr 5, 2011 at 5:52 PM, Ralf Gommers wrote: > On Tue, Apr 5, 2011 at 11:45 PM, Skipper Seabold wrote: >> On Sun, Apr 3, 2011 at 8:20 PM, Charles R Harris >> wrote: >>> Should skiprows be removed? >>> >>>     if skiprows: >>>     warnings.warn(\ >>>     "The use of `skiprows`

Re: [Numpy-discussion] [Numpy-svn] aada93: ENH: Add 'subok' parameter to PyArray_NewLikeArray...

2011-03-15 Thread Wes McKinney
On Tue, Mar 15, 2011 at 3:25 PM, Sebastian Haase wrote: > On Tue, Mar 15, 2011 at 7:22 PM,   wrote: >> Branch: refs/heads/master >> Home:   https://github.com/numpy/numpy >> >> Commit: aada93306acfb4e2eb816faf32652edf8825cf45 >>     >> https://github.com/numpy/numpy/commit/aada93306acfb4e2eb816faf

Re: [Numpy-discussion] broadcasting behavior for 1.6 (was: Numpy 1.6 schedule)

2011-03-11 Thread Wes McKinney
On Fri, Mar 11, 2011 at 9:57 AM, Charles R Harris wrote: > > > On Fri, Mar 11, 2011 at 7:42 AM, Charles R Harris > wrote: >> >> >> On Fri, Mar 11, 2011 at 2:01 AM, Ralf Gommers >> wrote: >>> >>> I'm just going through the very long 1.6 schedule thread to see what >>> is still on the TODO list be

Re: [Numpy-discussion] {zeros, empty, ...}_like functions behavior with subclasses

2011-03-01 Thread Wes McKinney
On Mon, Feb 28, 2011 at 10:52 PM, Wes McKinney wrote: > On Mon, Feb 28, 2011 at 7:24 PM, Pierre GM wrote: >> >> On Mar 1, 2011, at 1:05 AM, Bruce Southey wrote: >> >>> On Mon, Feb 28, 2011 at 4:52 PM, Wes McKinney wrote: >>>> I'm having some

Re: [Numpy-discussion] {zeros, empty, ...}_like functions behavior with subclasses

2011-02-28 Thread Wes McKinney
On Mon, Feb 28, 2011 at 7:24 PM, Pierre GM wrote: > > On Mar 1, 2011, at 1:05 AM, Bruce Southey wrote: > >> On Mon, Feb 28, 2011 at 4:52 PM, Wes McKinney wrote: >>> I'm having some trouble with the zeros_like function via np.fix: >>> >>> def

[Numpy-discussion] {zeros, empty, ...}_like functions behavior with subclasses

2011-02-28 Thread Wes McKinney
I'm having some trouble with the zeros_like function via np.fix: def zeros_like(a): if isinstance(a, ndarray): res = ndarray.__new__(type(a), a.shape, a.dtype, order=a.flags.fnc) res.fill(0) return res try: wrap = a.__array_wrap__ except AttributeError:

Re: [Numpy-discussion] OT: performance in C extension; OpenMP, or SSE ?

2011-02-15 Thread Wes McKinney
tian. > > > On Tue, Feb 15, 2011 at 5:26 PM, Wes McKinney wrote: >> On Tue, Feb 15, 2011 at 11:25 AM, Matthieu Brucher >> wrote: >>> Use directly restrict in C99 mode (__restrict does not have exactly the same >>> semantics). >>> For a valgrind pro

Re: [Numpy-discussion] OT: performance in C extension; OpenMP, or SSE ?

2011-02-15 Thread Wes McKinney
On Tue, Feb 15, 2011 at 11:25 AM, Matthieu Brucher wrote: > Use directly restrict in C99 mode (__restrict does not have exactly the same > semantics). > For a valgrind profil, you can check my blog > (http://matt.eifelle.com/2009/04/07/profiling-with-valgrind/) > Basically, if you have a python sc

Re: [Numpy-discussion] relational join

2011-02-02 Thread Wes McKinney
On Wed, Feb 2, 2011 at 4:46 PM, Robert Kern wrote: > On Wed, Feb 2, 2011 at 21:42, Ilya Shlyakhter wrote: >> Does numpy have a relational join operation for joining recordarrays? > > [~] > |1> from numpy.lib import recfunctions > > [~] > |2> recfunctions.join_by? > Type:           function > Base

Re: [Numpy-discussion] [ANN] Nanny, faster NaN functions

2010-11-21 Thread Wes McKinney
On Sun, Nov 21, 2010 at 6:37 PM, Keith Goodman wrote: > On Sun, Nov 21, 2010 at 3:16 PM, Wes McKinney wrote: > >> What would you say to a single package that contains: >> >> - NaN-aware NumPy and SciPy functions (nanmean, nanmin, etc.) > > I'd say yes. >

Re: [Numpy-discussion] [ANN] Nanny, faster NaN functions

2010-11-21 Thread Wes McKinney
On Sun, Nov 21, 2010 at 6:02 PM, wrote: > On Sun, Nov 21, 2010 at 5:09 PM, Keith Goodman wrote: >> On Sun, Nov 21, 2010 at 12:30 PM,   wrote: >>> On Sun, Nov 21, 2010 at 2:48 PM, Keith Goodman wrote: >>>> On Sun, Nov 21, 2010 at 10:25 AM, Wes McKinney wrote: >

Re: [Numpy-discussion] [ANN] Nanny, faster NaN functions

2010-11-21 Thread Wes McKinney
On Sat, Nov 20, 2010 at 7:24 PM, Keith Goodman wrote: > On Sat, Nov 20, 2010 at 3:54 PM, Wes McKinney wrote: > >> Keith (and others), >> >> What would you think about creating a library of mostly Cython-based >> "domain specific functions"? So stuff

Re: [Numpy-discussion] [ANN] Nanny, faster NaN functions

2010-11-20 Thread Wes McKinney
On Sat, Nov 20, 2010 at 6:54 PM, Wes McKinney wrote: > On Sat, Nov 20, 2010 at 6:39 PM, Keith Goodman wrote: >> On Fri, Nov 19, 2010 at 7:42 PM, Keith Goodman wrote: >>> I should make a benchmark suite. >> >>>> ny.benchit(verbose=False) >> Nanny

Re: [Numpy-discussion] [ANN] Nanny, faster NaN functions

2010-11-20 Thread Wes McKinney
On Sat, Nov 20, 2010 at 6:39 PM, Keith Goodman wrote: > On Fri, Nov 19, 2010 at 7:42 PM, Keith Goodman wrote: >> I should make a benchmark suite. > >>> ny.benchit(verbose=False) > Nanny performance benchmark >    Nanny 0.0.1dev >    Numpy 1.4.1 >    Speed is numpy time divided by nanny time >    

Re: [Numpy-discussion] Anyone with Core i7 and Ubuntu 10.04?

2010-11-08 Thread Wes McKinney
On Mon, Nov 8, 2010 at 11:33 PM, David Warde-Farley wrote: > On 2010-11-08, at 8:52 PM, David wrote: > >> Please tell us what error you got - saying that something did not >> working is really not useful to help you. You need to say exactly what >> fails, and which steps you followed before that f

Re: [Numpy-discussion] Financial TS models

2010-09-18 Thread Wes McKinney
On Sat, Sep 18, 2010 at 10:33 AM, wrote: > On Sat, Sep 18, 2010 at 10:13 AM,   wrote: >> On Sat, Sep 18, 2010 at 8:09 AM, Virgil Stokes wrote: >>>  I am considering the development of an all Python package (with numpy and >>> matplotlib) for the modeling and analysis of financial time series. >>

Re: [Numpy-discussion] Memory leak found in ndarray (I think)?

2010-07-13 Thread Wes McKinney
On Mon, Jul 12, 2010 at 6:44 PM, Nathaniel Peterson wrote: > Wes McKinney writes: > >> Did you mean to post a different link? That's the ticket I just created :) > > How silly of me! I meant http://projects.scipy.org/numpy/ticket/1427 > _

Re: [Numpy-discussion] numpy installation question (IronPython)

2010-07-13 Thread Wes McKinney
On Tue, Jul 13, 2010 at 8:26 AM, Sebastian Haase wrote: > On Tue, Jul 13, 2010 at 2:20 PM, William Johnston > wrote: >> Hello, >> >> I simply installed numpy in my Python26 installation, and then copied the >> numpy directory to my site-packages folder of my IronPython installation. >> >> Did I

Re: [Numpy-discussion] Memory leak found in ndarray (I think)?

2010-07-12 Thread Wes McKinney
On Mon, Jul 12, 2010 at 3:39 PM, Nathaniel Peterson wrote: > This memory leak may be related: http://projects.scipy.org/numpy/ticket/1542 > It shows what appears to be a memory leak when calling astype('float') > on an array of dtype 'object'. > ___ > Nu

Re: [Numpy-discussion] Memory leak found in ndarray (I think)?

2010-07-12 Thread Wes McKinney
On Mon, Jul 12, 2010 at 2:22 PM, Wes McKinney wrote: > This one was quite a bear to track down, starting from the of course > very high level observation of "why is my application leaking memory". > I've reproduced it on Windows XP using NumPy 1.3.0 on Python 2.5 and &g

[Numpy-discussion] Memory leak found in ndarray (I think)?

2010-07-12 Thread Wes McKinney
This one was quite a bear to track down, starting from the of course very high level observation of "why is my application leaking memory". I've reproduced it on Windows XP using NumPy 1.3.0 on Python 2.5 and 1.4.1 on Python 2.6 (EPD). Basically it seems that calling .astype(bool) on an ndarray sli

Re: [Numpy-discussion] BOF notes: Fernando's proposal: NumPy ndarray with named axes

2010-07-08 Thread Wes McKinney
On Thu, Jul 8, 2010 at 1:35 PM, Rob Speer wrote: >> Forgive me if this is has already been addressed, but my question is >> what happens when we have more than one "label" (not as in a labeled >> axis but an observation label -- but not a tick because they're not >> unique!) per say row axis and h

Re: [Numpy-discussion] Memory usage of numpy-arrays

2010-07-08 Thread Wes McKinney
On Thu, Jul 8, 2010 at 9:26 AM, Hannes Bretschneider wrote: > Dear NumPy developers, > > I have to process some big data files with high-frequency > financial data. I am trying to load a delimited text file having > ~700 MB with ~ 10 million lines using numpy.genfromtxt(). The > machine is a Debia

Re: [Numpy-discussion] BOF notes: Fernando's proposal: NumPy ndarray with named axes

2010-07-06 Thread Wes McKinney
On Tue, Jul 6, 2010 at 12:56 PM, Keith Goodman wrote: > On Tue, Jul 6, 2010 at 9:52 AM, Joshua Holbrook > wrote: >> On Tue, Jul 6, 2010 at 8:42 AM, Skipper Seabold wrote: >>> On Tue, Jul 6, 2010 at 12:36 PM, Joshua Holbrook >>> wrote: I'm kinda-sorta still getting around to building/readi

Re: [Numpy-discussion] indexing question

2010-06-21 Thread Wes McKinney
On Mon, Jun 21, 2010 at 7:10 PM, Robert Kern wrote: > On Mon, Jun 21, 2010 at 17:42, Neal Becker wrote: >> Robert Kern wrote: >> >>> On Mon, Jun 21, 2010 at 14:01, Neal Becker wrote: Can I find an efficient way to do this? I have a 2d array, A, 80 rows by 880 columns. I

Re: [Numpy-discussion] Tools / data structures for statistical analysis and related applications

2010-06-11 Thread Wes McKinney
On Fri, Jun 11, 2010 at 9:46 AM, Bruce Southey wrote: > On 06/09/2010 03:40 PM, Wes McKinney wrote: >> Dear all, >> >> We've been having discussions on the pystatsmodels mailing list >> recently regarding data structures and other tools for statistics / >> ot

  1   2   >