Re: [Numpy-discussion] 2d binning and linear regression

2010-06-23 Thread Bruce Southey
On 06/22/2010 02:58 PM, josef.p...@gmail.com wrote: On Tue, Jun 22, 2010 at 10:09 AM, Tom Durrantthdurr...@gmail.com wrote: the basic idea is in polyfit on multiple data points on numpy-disscusion mailing list April 2009 In this case, calculations have to be done by groups subtract

Re: [Numpy-discussion] 2d binning and linear regression

2010-06-22 Thread Tom Durrant
the basic idea is in polyfit on multiple data points on numpy-disscusion mailing list April 2009 In this case, calculations have to be done by groups subtract mean (this needs to be replaced by group demeaning) modeldm = model - model.mean() obsdm = obs - obs.mean() xx =

Re: [Numpy-discussion] 2d binning and linear regression

2010-06-22 Thread Tom Durrant
What exactly are trying to fit because it is rather bad practice to fit a model to some summarized data as you lose the uncertainty in the original data? If you define your boxes, you can loop through directly on each box and even fit the equation: model=mu +beta1*obs The extension is

Re: [Numpy-discussion] 2d binning and linear regression

2010-06-22 Thread Bruce Southey
On 06/22/2010 09:13 AM, Tom Durrant wrote: What exactly are trying to fit because it is rather bad practice to fit a model to some summarized data as you lose the uncertainty in the original data? If you define your boxes, you can loop through directly on each box

Re: [Numpy-discussion] 2d binning and linear regression

2010-06-22 Thread josef . pktd
On Tue, Jun 22, 2010 at 10:09 AM, Tom Durrant thdurr...@gmail.com wrote: the basic idea is in polyfit  on multiple data points on numpy-disscusion mailing list April 2009 In this case, calculations have to be done by groups subtract mean (this needs to be replaced by group demeaning)

Re: [Numpy-discussion] 2d binning and linear regression

2010-06-21 Thread josef . pktd
On Sun, Jun 20, 2010 at 10:57 PM, Tom Durrant thdurr...@gmail.com wrote: are you doing something like np.polyfit(model, obs, 1) ? If you are using polyfit with deg=1, i.e. fitting a straight line, then this could be also calculated using the weights in histogram2d. histogram2d

Re: [Numpy-discussion] 2d binning and linear regression

2010-06-21 Thread Bruce Southey
On 06/20/2010 03:24 AM, Tom Durrant wrote: Hi All, I have a problem involving lat/lon data. Basically, I am evaluating numerical weather model data against satellite data, and trying to produce gridded plots of various statistics. There are various steps involved with this, but

[Numpy-discussion] 2d binning and linear regression

2010-06-20 Thread Tom Durrant
Hi All, I have a problem involving lat/lon data. Basically, I am evaluating numerical weather model data against satellite data, and trying to produce gridded plots of various statistics. There are various steps involved with this, but basically, I get to the point where I have four arrays of

Re: [Numpy-discussion] 2d binning and linear regression

2010-06-20 Thread josef . pktd
On Sun, Jun 20, 2010 at 4:24 AM, Tom Durrant thdurr...@gmail.com wrote: Hi All, I have a problem involving lat/lon data.  Basically, I am evaluating numerical weather model data against satellite data, and trying to produce gridded plots of various statistics.  There are various steps involved

Re: [Numpy-discussion] 2d binning and linear regression

2010-06-20 Thread Tom Durrant
are you doing something like np.polyfit(model, obs, 1) ? If you are using polyfit with deg=1, i.e. fitting a straight line, then this could be also calculated using the weights in histogram2d. histogram2d (histogramdd) uses np.digitize and np.bincount, so I'm surprised if the

Re: [Numpy-discussion] 2d binning on regular grid

2010-06-03 Thread Friedrich Romstedt
Hello Andreas, please see this as a side remark. A colleague of mine made me aware of a very beautiful thing about covering spheres by evenly spaced points: http://healpix.jpl.nasa.gov/ Since you want to calculate mean and stddev, to my understanding a grid in longitude/latitude is without

Re: [Numpy-discussion] 2d binning on regular grid

2010-06-03 Thread josef . pktd
On Wed, Jun 2, 2010 at 11:40 AM, Andreas Hilboll li...@hilboll.de wrote: Hi there, I'm interested in the solution to a special case of the parallel thread '2D binning', which is going on at the moment. My data is on a fine global grid, say .125x.125 degrees. I'm looking for a way to do

Re: [Numpy-discussion] 2D binning

2010-06-02 Thread Vincent Schut
On 06/02/2010 04:52 AM, josef.p...@gmail.com wrote: On Tue, Jun 1, 2010 at 9:57 PM, Zachary Pincuszachary.pin...@yale.edu wrote: I guess it's as fast as I'm going to get. I don't really see any other way. BTW, the lat/lons are integers) You could (in c or cython) try a brain-dead hashtable

Re: [Numpy-discussion] 2D binning

2010-06-02 Thread Benjamin Root
Why not simply use a set? uniquePoints = set(zip(lats, lons)) Ben Root On Wed, Jun 2, 2010 at 2:41 AM, Vincent Schut sc...@sarvision.nl wrote: On 06/02/2010 04:52 AM, josef.p...@gmail.com wrote: On Tue, Jun 1, 2010 at 9:57 PM, Zachary Pincuszachary.pin...@yale.edu wrote: I guess it's as

[Numpy-discussion] 2d binning on regular grid

2010-06-02 Thread Andreas Hilboll
Hi there, I'm interested in the solution to a special case of the parallel thread '2D binning', which is going on at the moment. My data is on a fine global grid, say .125x.125 degrees. I'm looking for a way to do calculations on coarser grids, e.g. * calculate means() * calculate std() * ...

Re: [Numpy-discussion] 2D binning

2010-06-02 Thread Wes McKinney
On Wed, Jun 2, 2010 at 3:41 AM, Vincent Schut sc...@sarvision.nl wrote: On 06/02/2010 04:52 AM, josef.p...@gmail.com wrote: On Tue, Jun 1, 2010 at 9:57 PM, Zachary Pincuszachary.pin...@yale.edu   wrote: I guess it's as fast as I'm going to get. I don't really see any other way. BTW, the

Re: [Numpy-discussion] 2D binning

2010-06-02 Thread Mathew Yeates
thanks. I am also getting an error in ndi.mean Were you getting the error RuntimeError: data type not supported? -Mathew On Wed, Jun 2, 2010 at 9:40 AM, Wes McKinney wesmck...@gmail.com wrote: On Wed, Jun 2, 2010 at 3:41 AM, Vincent Schut sc...@sarvision.nl wrote: On 06/02/2010 04:52 AM,

Re: [Numpy-discussion] 2D binning

2010-06-02 Thread Wes McKinney
On Wed, Jun 2, 2010 at 1:23 PM, Mathew Yeates mat.yea...@gmail.com wrote: thanks. I am also getting an error in ndi.mean Were you getting the error RuntimeError: data type not supported? -Mathew On Wed, Jun 2, 2010 at 9:40 AM, Wes McKinney wesmck...@gmail.com wrote: On Wed, Jun 2, 2010 at

Re: [Numpy-discussion] 2D binning

2010-06-02 Thread Mathew Yeates
I'm on Windows, using a precompiled binary. I never built numpy/scipy on Windows. On Wed, Jun 2, 2010 at 10:45 AM, Wes McKinney wesmck...@gmail.com wrote: On Wed, Jun 2, 2010 at 1:23 PM, Mathew Yeates mat.yea...@gmail.com wrote: thanks. I am also getting an error in ndi.mean Were you

Re: [Numpy-discussion] 2D binning

2010-06-02 Thread josef . pktd
On Wed, Jun 2, 2010 at 2:09 PM, Mathew Yeates mat.yea...@gmail.com wrote: I'm on Windows, using a precompiled binary. I never built numpy/scipy on Windows. ndimage measurements has been recently rewritten. ndimage is very fast but (the old version) has insufficient type checking and may crash

Re: [Numpy-discussion] 2D binning

2010-06-02 Thread Wes McKinney
On Wed, Jun 2, 2010 at 2:26 PM, josef.p...@gmail.com wrote: On Wed, Jun 2, 2010 at 2:09 PM, Mathew Yeates mat.yea...@gmail.com wrote: I'm on Windows, using a precompiled binary. I never built numpy/scipy on Windows. ndimage measurements has been recently rewritten. ndimage is very fast but

Re: [Numpy-discussion] 2D binning

2010-06-02 Thread Mathew Yeates
Nope. This version didn't work either. If you're on Python 2.6 the binary on here might work for you: http://www.lfd.uci.edu/~gohlke/pythonlibs/ It looks recent enough to have the rewritten ndimage ___ NumPy-Discussion mailing list

Re: [Numpy-discussion] 2D binning

2010-06-02 Thread Christoph Gohlke
On 6/2/2010 2:32 PM, Mathew Yeates wrote: Nope. This version didn't work either. If you're on Python 2.6 the binary on here might work for you: http://www.lfd.uci.edu/~gohlke/pythonlibs/ It looks recent enough to have the rewritten ndimage On 6/2/2010 2:32 PM, Mathew

Re: [Numpy-discussion] 2D binning

2010-06-02 Thread Stephen Simmons
On 1/06/2010 10:51 PM, Wes McKinney wrote: snip This is a pretty good example of the group-by problem that will hopefully work its way into a future edition of NumPy. Wes (or anyone else), please can you elaborate on any plans for groupby? I've made my own modification to numpy.bincount

Re: [Numpy-discussion] 2D binning

2010-06-02 Thread Wes McKinney
On Wed, Jun 2, 2010 at 6:18 PM, Stephen Simmons m...@stevesimmons.com wrote: On 1/06/2010 10:51 PM, Wes McKinney wrote:   snip   This is a pretty good example of the group-by problem that will   hopefully work its way into a future edition of NumPy. Wes (or anyone else), please can you

[Numpy-discussion] 2D binning

2010-06-01 Thread Mathew Yeates
Hi Can anyone think of a clever (non-lopping) solution to the following? A have a list of latitudes, a list of longitudes, and list of data values. All lists are the same length. I want to compute an average of data values for each lat/lon pair. e.g. if lat[1001] lon[1001] = lat[2001] [lon

Re: [Numpy-discussion] 2D binning

2010-06-01 Thread Keith Goodman
On Tue, Jun 1, 2010 at 1:07 PM, Mathew Yeates mat.yea...@gmail.com wrote: Hi Can anyone think of a clever (non-lopping) solution to the following? A have a list of latitudes, a list of longitudes, and list of data values. All lists are the same length. I want to compute an average  of data

Re: [Numpy-discussion] 2D binning

2010-06-01 Thread Zachary Pincus
Hi Can anyone think of a clever (non-lopping) solution to the following? A have a list of latitudes, a list of longitudes, and list of data values. All lists are the same length. I want to compute an average of data values for each lat/lon pair. e.g. if lat[1001] lon[1001] = lat[2001]

Re: [Numpy-discussion] 2D binning

2010-06-01 Thread Wes McKinney
On Tue, Jun 1, 2010 at 4:49 PM, Zachary Pincus zachary.pin...@yale.edu wrote: Hi Can anyone think of a clever (non-lopping) solution to the following? A have a list of latitudes, a list of longitudes, and list of data values. All lists are the same length. I want to compute an average  of

Re: [Numpy-discussion] 2D binning

2010-06-01 Thread Mathew Yeates
I guess it's as fast as I'm going to get. I don't really see any other way. BTW, the lat/lons are integers) -Mathew On Tue, Jun 1, 2010 at 1:49 PM, Zachary Pincus zachary.pin...@yale.eduwrote: Hi Can anyone think of a clever (non-lopping) solution to the following? A have a list of

Re: [Numpy-discussion] 2D binning

2010-06-01 Thread Zachary Pincus
I guess it's as fast as I'm going to get. I don't really see any other way. BTW, the lat/lons are integers) You could (in c or cython) try a brain-dead hashtable with no collision detection: for lat, long, data in dataset: bin = (lat ^ long) % num_bins hashtable[bin] =

Re: [Numpy-discussion] 2D binning

2010-06-01 Thread josef . pktd
On Tue, Jun 1, 2010 at 9:57 PM, Zachary Pincus zachary.pin...@yale.edu wrote: I guess it's as fast as I'm going to get. I don't really see any other way. BTW, the lat/lons are integers) You could (in c or cython) try a brain-dead hashtable with no collision detection: for lat, long, data in

Re: [Numpy-discussion] 2D binning

2010-06-01 Thread Brent Pedersen
On Tue, Jun 1, 2010 at 1:51 PM, Wes McKinney wesmck...@gmail.com wrote: On Tue, Jun 1, 2010 at 4:49 PM, Zachary Pincus zachary.pin...@yale.edu wrote: Hi Can anyone think of a clever (non-lopping) solution to the following? A have a list of latitudes, a list of longitudes, and list of data