[Numpy-discussion] 2d binning and linear regression

2010-06-20 Thread Tom Durrant
Hi All, I have a problem involving lat/lon data. Basically, I am evaluating numerical weather model data against satellite data, and trying to produce gridded plots of various statistics. There are various steps involved with this, but basically, I get to the point where I have four arrays of th

Re: [Numpy-discussion] 2d binning and linear regression

2010-06-20 Thread josef . pktd
On Sun, Jun 20, 2010 at 4:24 AM, Tom Durrant wrote: > Hi All, > I have a problem involving lat/lon data.  Basically, I am evaluating > numerical weather model data against satellite data, and trying to produce > gridded plots of various statistics.  There are various steps involved with > this, bu

Re: [Numpy-discussion] 2d binning and linear regression

2010-06-20 Thread Tom Durrant
> > are you doing something like np.polyfit(model, obs, 1) ? > > If you are using polyfit with deg=1, i.e. fitting a straight line, > then this could be also calculated using the weights in histogram2d. > > histogram2d (histogramdd) uses np.digitize and np.bincount, so I'm > surprised if the hi

Re: [Numpy-discussion] 2d binning and linear regression

2010-06-21 Thread josef . pktd
On Sun, Jun 20, 2010 at 10:57 PM, Tom Durrant wrote: > >> >> are you doing something like np.polyfit(model, obs, 1) ? >> >> If you are using polyfit with deg=1, i.e. fitting a straight line, >> then this could be also calculated using the weights in histogram2d. >> >> histogram2d (histogramdd) use

Re: [Numpy-discussion] 2d binning and linear regression

2010-06-21 Thread Bruce Southey
On 06/20/2010 03:24 AM, Tom Durrant wrote: > Hi All, > > I have a problem involving lat/lon data. Basically, I am evaluating > numerical weather model data against satellite data, and trying to > produce gridded plots of various statistics. There are various steps > involved with this, but bas

Re: [Numpy-discussion] 2d binning and linear regression

2010-06-22 Thread Tom Durrant
> > > the basic idea is in "polyfit on multiple data points" on > numpy-disscusion mailing list April 2009 > > In this case, calculations have to be done by groups > > subtract mean (this needs to be replaced by group demeaning) > modeldm = model - model.mean() > obsdm = obs - obs.mean() > > xx =

Re: [Numpy-discussion] 2d binning and linear regression

2010-06-22 Thread Tom Durrant
> > > > > What exactly are trying to fit because it is rather bad practice to fit > a model to some summarized data as you lose the uncertainty in the > original data? > If you define your boxes, you can loop through directly on each box and > even fit the equation: > > model=mu +beta1*obs > > The

Re: [Numpy-discussion] 2d binning and linear regression

2010-06-22 Thread Bruce Southey
On 06/22/2010 09:13 AM, Tom Durrant wrote: > What exactly are trying to fit because it is rather bad practice to fit a model to some summarized data as you lose the uncertainty in the original data? If you define your boxes, you can loop through directly on each box

Re: [Numpy-discussion] 2d binning and linear regression

2010-06-22 Thread josef . pktd
On Tue, Jun 22, 2010 at 10:09 AM, Tom Durrant wrote: >> >> the basic idea is in "polyfit  on multiple data points" on >> numpy-disscusion mailing list April 2009 >> >> In this case, calculations have to be done by groups >> >> subtract mean (this needs to be replaced by group demeaning) >> modeldm

Re: [Numpy-discussion] 2d binning and linear regression

2010-06-23 Thread Bruce Southey
On 06/22/2010 02:58 PM, josef.p...@gmail.com wrote: > On Tue, Jun 22, 2010 at 10:09 AM, Tom Durrant wrote: > >>> the basic idea is in "polyfit on multiple data points" on >>> numpy-disscusion mailing list April 2009 >>> >>> In this case, calculations have to be done by groups >>> >>> subtract