Re: [Numpy-discussion] invalid correlation coefficient from np.ma.corrcoef

2013-09-26 Thread josef . pktd
On Thu, Sep 26, 2013 at 6:42 PM, Nathaniel Smith wrote: > On 26 Sep 2013 21:59, "Faraz Mirzaei" wrote: >> >> Thanks Josef and Nathaniel for your responses. >> >> In the application that I have, I don't use the correlation coefficient >> matrix as a whole (so I don't care if it is PSD or not). I s

Re: [Numpy-discussion] invalid correlation coefficient from np.ma.corrcoef

2013-09-26 Thread Nathaniel Smith
On 26 Sep 2013 21:59, "Faraz Mirzaei" wrote: > > Thanks Josef and Nathaniel for your responses. > > In the application that I have, I don't use the correlation coefficient matrix as a whole (so I don't care if it is PSD or not). I simply read the off-diagonal elements for pair-wise correlation coe

Re: [Numpy-discussion] invalid correlation coefficient from np.ma.corrcoef

2013-09-26 Thread Faraz Mirzaei
Thanks Josef and Nathaniel for your responses. In the application that I have, I don't use the correlation coefficient matrix as a whole (so I don't care if it is PSD or not). I simply read the off-diagonal elements for pair-wise correlation coefficients. I use the pairwise correlation coefficient

Re: [Numpy-discussion] invalid correlation coefficient from np.ma.corrcoef

2013-09-26 Thread Nathaniel Smith
On 26 Sep 2013 17:32, wrote: > > On Thu, Sep 26, 2013 at 7:35 AM, Nathaniel Smith wrote: > > By textbook I mean, users expect corrcoef to use this formula, which > > is printed in every textbook: > > https://en.wikipedia.org/wiki/Pearson_product-moment_correlation_coefficient#For_a_sample > > The

Re: [Numpy-discussion] invalid correlation coefficient from np.ma.corrcoef

2013-09-26 Thread josef . pktd
On Thu, Sep 26, 2013 at 7:35 AM, Nathaniel Smith wrote: > On Thu, Sep 26, 2013 at 11:51 AM, wrote: >> On Thu, Sep 26, 2013 at 4:21 AM, Nathaniel Smith wrote: >>> If you want a proper self-consistent correlation/covariance matrix, then >>> pairwise deletion just makes no sense period, I don't se

Re: [Numpy-discussion] invalid correlation coefficient from np.ma.corrcoef

2013-09-26 Thread Nathaniel Smith
On Thu, Sep 26, 2013 at 11:51 AM, wrote: > On Thu, Sep 26, 2013 at 4:21 AM, Nathaniel Smith wrote: >> If you want a proper self-consistent correlation/covariance matrix, then >> pairwise deletion just makes no sense period, I don't see how postprocessing >> can help. > > clipping to [-1, 1] and

Re: [Numpy-discussion] invalid correlation coefficient from np.ma.corrcoef

2013-09-26 Thread josef . pktd
On Thu, Sep 26, 2013 at 6:51 AM, wrote: > On Thu, Sep 26, 2013 at 4:21 AM, Nathaniel Smith wrote: >> If you want a proper self-consistent correlation/covariance matrix, then >> pairwise deletion just makes no sense period, I don't see how postprocessing >> can help. > > clipping to [-1, 1] and f

Re: [Numpy-discussion] invalid correlation coefficient from np.ma.corrcoef

2013-09-26 Thread josef . pktd
On Thu, Sep 26, 2013 at 4:21 AM, Nathaniel Smith wrote: > If you want a proper self-consistent correlation/covariance matrix, then > pairwise deletion just makes no sense period, I don't see how postprocessing > can help. clipping to [-1, 1] and finding the nearest positive semi-definite matrix.

Re: [Numpy-discussion] invalid correlation coefficient from np.ma.corrcoef

2013-09-26 Thread Nathaniel Smith
If you want a proper self-consistent correlation/covariance matrix, then pairwise deletion just makes no sense period, I don't see how postprocessing can help. If you want a matrix of correlations, then pairwise deletion makes sense. It's an interesting point that arguably the current ma.corrcoef

Re: [Numpy-discussion] invalid correlation coefficient from np.ma.corrcoef

2013-09-25 Thread josef . pktd
On Wed, Sep 25, 2013 at 11:05 PM, wrote: > On Wed, Sep 25, 2013 at 8:26 PM, Faraz Mirzaei wrote: >> Hi everyone, >> >> I'm using np.ma.corrcoef to compute the correlation coefficients among rows >> of a masked matrix, where the masked elements are the missing data. I've >> observed that in some

Re: [Numpy-discussion] invalid correlation coefficient from np.ma.corrcoef

2013-09-25 Thread josef . pktd
On Wed, Sep 25, 2013 at 8:26 PM, Faraz Mirzaei wrote: > Hi everyone, > > I'm using np.ma.corrcoef to compute the correlation coefficients among rows > of a masked matrix, where the masked elements are the missing data. I've > observed that in some cases, the np.ma.corrcoef gives invalid coefficien

[Numpy-discussion] invalid correlation coefficient from np.ma.corrcoef

2013-09-25 Thread Faraz Mirzaei
Hi everyone, I'm using np.ma.corrcoef to compute the correlation coefficients among rows of a masked matrix, where the masked elements are the missing data. I've observed that in some cases, the np.ma.corrcoef gives invalid coefficients that are greater than 1 or less than -1. Here's an example: