Re: [Numpy-discussion] Problem with correlate

[Ryan May]

On Thu, Jun 4, 2009 at 5:14 AM, David Cournapeau  wrote:

> On Tue, Jun 2, 2009 at 10:56 PM, Ryan May  wrote:
> > On Tue, Jun 2, 2009 at 5:59 AM, David Cournapeau
> >  wrote:
> >>
> >> Robin wrote:
> >> > On Tue, Jun 2, 2009 at 11:36 AM, David Cournapeau  >
> >> > wrote:
> >> >
> >> >> Done in r7031 - correlate/PyArray_Correlate should be unchanged, and
> >> >> acorrelate/PyArray_Acorrelate implement the conventional definitions,
> >> >>
> >> >
> >> > I don't know if it's been discussed before but while people are
> >> > thinking about/changing correlate I thought I'd like to request as a
> >> > user a matlab style xcorr function (basically with the functionality
> >> > of the matlab version).
> >> >
> >> > I don't know if this is a deliberate emission, but it is often one of
> >> > the first things my colleagues try when I get them using Python, and
> >> > as far as I know there isn't really a good answer. There is xcorr in
> >> > pylab, but it isn't vectorised like xcorr from matlab...
> >> >
> >>
> >> There is one in the talkbox scikit:
> >>
> >>
> >>
> http://github.com/cournape/talkbox/blob/202135a9d848931ebd036b97302f1e10d7488c63/scikits/talkbox/tools/correlations.py
> >>
> >> It uses the fft, and bonus point, the file is independent of the rest of
> >> toolbox. There is another version which uses direct implementation (this
> >> is faster if you need only a few lags, and it takes less memory too).
> >
> > I'd be +1 on including something like this (provided it expanded to
> include
> > complex-valued data).  I think it's a real need, since everyone seems to
> > keep rolling their own.  I had to write my own just so that I can
> calculate
> > a few lags in a vectorized fashion.
>
> The code in talkbox is not good enough for scipy. I made an attempt
> for scipy.signal here:
>
>
> http://github.com/cournape/scipy3/blob/b004d17d824f1c03921d9663207ee40adadc5762/scipy/signal/correlations.py
>
> It is reasonably fast when only a few lags are needed, both double and
> complex double are supported, and it works on arbitrary axis and lags.
> Other precisions should be easy to add, but I think I need to extend
> the numpy code generators to support cython sources to avoid code
> duplication.
>
> Does that fill your need ?


It would fill mine.  Would it make sense to make y default to x, so that you
can use xcorr to do the autocorrelation as:

xcorr(x)

?

Ryan

-- 
Ryan May
Graduate Research Assistant
School of Meteorology
University of Oklahoma
___
Numpy-discussion mailing list
Numpy-discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Problem with correlate

[David Cournapeau]

On Tue, Jun 2, 2009 at 10:56 PM, Ryan May  wrote:
> On Tue, Jun 2, 2009 at 5:59 AM, David Cournapeau
>  wrote:
>>
>> Robin wrote:
>> > On Tue, Jun 2, 2009 at 11:36 AM, David Cournapeau 
>> > wrote:
>> >
>> >> Done in r7031 - correlate/PyArray_Correlate should be unchanged, and
>> >> acorrelate/PyArray_Acorrelate implement the conventional definitions,
>> >>
>> >
>> > I don't know if it's been discussed before but while people are
>> > thinking about/changing correlate I thought I'd like to request as a
>> > user a matlab style xcorr function (basically with the functionality
>> > of the matlab version).
>> >
>> > I don't know if this is a deliberate emission, but it is often one of
>> > the first things my colleagues try when I get them using Python, and
>> > as far as I know there isn't really a good answer. There is xcorr in
>> > pylab, but it isn't vectorised like xcorr from matlab...
>> >
>>
>> There is one in the talkbox scikit:
>>
>>
>> http://github.com/cournape/talkbox/blob/202135a9d848931ebd036b97302f1e10d7488c63/scikits/talkbox/tools/correlations.py
>>
>> It uses the fft, and bonus point, the file is independent of the rest of
>> toolbox. There is another version which uses direct implementation (this
>> is faster if you need only a few lags, and it takes less memory too).
>
> I'd be +1 on including something like this (provided it expanded to include
> complex-valued data).  I think it's a real need, since everyone seems to
> keep rolling their own.  I had to write my own just so that I can calculate
> a few lags in a vectorized fashion.

The code in talkbox is not good enough for scipy. I made an attempt
for scipy.signal here:

http://github.com/cournape/scipy3/blob/b004d17d824f1c03921d9663207ee40adadc5762/scipy/signal/correlations.py

It is reasonably fast when only a few lags are needed, both double and
complex double are supported, and it works on arbitrary axis and lags.
Other precisions should be easy to add, but I think I need to extend
the numpy code generators to support cython sources to avoid code
duplication.

Does that fill your need ?

cheers,

David
___
Numpy-discussion mailing list
Numpy-discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Problem with correlate

[rob steed]

> I wonder if xcorrelate would be a better name than acorrelate?

I think it would.


  
___
Numpy-discussion mailing list
Numpy-discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Problem with correlate

[David Cournapeau]

Charles R Harris wrote:
>
>
>
> I also think that having weighting options would be good.  I now
> understand the complexities of the various
> weightings that can be applied to the correlation i.e. biased vs
> unbiased but I think that having correlate
> include these options might prompt users to investigate which one
> they really needed. Correlate seemed
> so simple when I first used it but it took me ages to realise that
> these are choices to be made.
>
>
> I wonder if xcorrelate would be a better name than acorrelate?

It may be confusing, because what acorrelate currently does is not what
xcorr does under matlab. Under matlab, xcorr computes the 1d cross
correlation column-wise (would be axis wise under numpy). If I get the
time today, I will try to implement a first version using generalized
ufunc - that would be a good occasion to learn generalized ufunc.

If that works well, maybe there is no need for the current acorrelate at
all (but we should still keep PyArray_Acorrelate, I think).

cheers,

David
___
Numpy-discussion mailing list
Numpy-discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Problem with correlate

[Charles R Harris]

On Tue, Jun 2, 2009 at 1:16 PM, rob steed  wrote:

>
> I also think that the conjugate should be taken. I spent the last few weeks
> using correlate to experiment with
> signal processing and I got strange results until I realised that I had to
> manually take the conjugate. It
> would also be good if the function did it since applying the conjugate to
> the wrong sequence yields the
> complex conjugate of the correlation.
>
> Who would want to use the correlation without the conjugate, if someone is
> only using real values it won't
> affect them, if they are using complex values they probably want to
> conjugate.
>
> One function that does depend on correlate though is convolution! Changes
> made to correlate will
> affect it! but I have understand that a new function acorrelate is being
> created instead of changing
> correlate?
>
> Otherwise I've never used matlab but it does seem like xcorr has some good
> features. The modes
> 'same' and 'valid' were initially quite confusing especially as the default
> is 'valid', meaning that autocorrelations
> lead to a single value by default!
>
> I also think that having weighting options would be good.  I now understand
> the complexities of the various
> weightings that can be applied to the correlation i.e. biased vs unbiased
> but I think that having correlate
> include these options might prompt users to investigate which one they
> really needed. Correlate seemed
> so simple when I first used it but it took me ages to realise that these
> are choices to be made.
>

I wonder if xcorrelate would be a better name than acorrelate?

Chuck
___
Numpy-discussion mailing list
Numpy-discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Problem with correlate

[rob steed]

I also think that the conjugate should be taken. I spent the last few weeks 
using correlate to experiment with 
signal processing and I got strange results until I realised that I had to 
manually take the conjugate. It 
would also be good if the function did it since applying the conjugate to the 
wrong sequence yields the 
complex conjugate of the correlation.

Who would want to use the correlation without the conjugate, if someone is only 
using real values it won't 
affect them, if they are using complex values they probably want to conjugate.

One function that does depend on correlate though is convolution! Changes made 
to correlate will
affect it! but I have understand that a new function acorrelate is being 
created instead of changing
correlate?

Otherwise I've never used matlab but it does seem like xcorr has some good 
features. The modes
'same' and 'valid' were initially quite confusing especially as the default is 
'valid', meaning that autocorrelations 
lead to a single value by default!

I also think that having weighting options would be good.  I now understand the 
complexities of the various
weightings that can be applied to the correlation i.e. biased vs unbiased but I 
think that having correlate
include these options might prompt users to investigate which one they really 
needed. Correlate seemed 
so simple when I first used it but it took me ages to realise that these are 
choices to be made. 

regards

Rob


  
___
Numpy-discussion mailing list
Numpy-discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Problem with correlate

[Ryan May]

On Tue, Jun 2, 2009 at 5:59 AM, David Cournapeau <
da...@ar.media.kyoto-u.ac.jp> wrote:

> Robin wrote:
> > On Tue, Jun 2, 2009 at 11:36 AM, David Cournapeau 
> wrote:
> >
> >> Done in r7031 - correlate/PyArray_Correlate should be unchanged, and
> >> acorrelate/PyArray_Acorrelate implement the conventional definitions,
> >>
> >
> > I don't know if it's been discussed before but while people are
> > thinking about/changing correlate I thought I'd like to request as a
> > user a matlab style xcorr function (basically with the functionality
> > of the matlab version).
> >
> > I don't know if this is a deliberate emission, but it is often one of
> > the first things my colleagues try when I get them using Python, and
> > as far as I know there isn't really a good answer. There is xcorr in
> > pylab, but it isn't vectorised like xcorr from matlab...
> >
>
> There is one in the talkbox scikit:
>
>
> http://github.com/cournape/talkbox/blob/202135a9d848931ebd036b97302f1e10d7488c63/scikits/talkbox/tools/correlations.py
>
> It uses the fft, and bonus point, the file is independent of the rest of
> toolbox. There is another version which uses direct implementation (this
> is faster if you need only a few lags, and it takes less memory too).


I'd be +1 on including something like this (provided it expanded to include
complex-valued data).  I think it's a real need, since everyone seems to
keep rolling their own.  I had to write my own just so that I can calculate
a few lags in a vectorized fashion.

Ryan

-- 
Ryan May
Graduate Research Assistant
School of Meteorology
University of Oklahoma
___
Numpy-discussion mailing list
Numpy-discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Problem with correlate

[David Cournapeau]

Robin wrote:
> On Tue, Jun 2, 2009 at 11:36 AM, David Cournapeau  wrote:
>   
>> Done in r7031 - correlate/PyArray_Correlate should be unchanged, and
>> acorrelate/PyArray_Acorrelate implement the conventional definitions,
>> 
>
> I don't know if it's been discussed before but while people are
> thinking about/changing correlate I thought I'd like to request as a
> user a matlab style xcorr function (basically with the functionality
> of the matlab version).
>
> I don't know if this is a deliberate emission, but it is often one of
> the first things my colleagues try when I get them using Python, and
> as far as I know there isn't really a good answer. There is xcorr in
> pylab, but it isn't vectorised like xcorr from matlab...
>   

There is one in the talkbox scikit:

http://github.com/cournape/talkbox/blob/202135a9d848931ebd036b97302f1e10d7488c63/scikits/talkbox/tools/correlations.py

It uses the fft, and bonus point, the file is independent of the rest of
toolbox. There is another version which uses direct implementation (this
is faster if you need only a few lags, and it takes less memory too).

David
___
Numpy-discussion mailing list
Numpy-discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Problem with correlate

[Robin]

On Tue, Jun 2, 2009 at 11:36 AM, David Cournapeau  wrote:
>
> Done in r7031 - correlate/PyArray_Correlate should be unchanged, and
> acorrelate/PyArray_Acorrelate implement the conventional definitions,

I don't know if it's been discussed before but while people are
thinking about/changing correlate I thought I'd like to request as a
user a matlab style xcorr function (basically with the functionality
of the matlab version).

I don't know if this is a deliberate emission, but it is often one of
the first things my colleagues try when I get them using Python, and
as far as I know there isn't really a good answer. There is xcorr in
pylab, but it isn't vectorised like xcorr from matlab...

Cheers

Robin
___
Numpy-discussion mailing list
Numpy-discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Problem with correlate

[David Cournapeau]

On Tue, Jun 2, 2009 at 12:37 PM, David Cournapeau
 wrote:
> Robert Kern wrote:
>>> This does not solve the C function problem (PyArray_Correlate). The easy
>>> solution would be to keep the current C version, deal with the problem
>>> in python for acorrelate for the time being, and replace the old C
>>> function with the 'correct' one once we remove the deprecated correlate ?
>>>
>>
>> No, you do the same thing at the C level.
>>
>

Done in r7031 - correlate/PyArray_Correlate should be unchanged, and
acorrelate/PyArray_Acorrelate implement the conventional definitions,

David
___
Numpy-discussion mailing list
Numpy-discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Problem with correlate

[David Cournapeau]

Robert Kern wrote:
>> This does not solve the C function problem (PyArray_Correlate). The easy
>> solution would be to keep the current C version, deal with the problem
>> in python for acorrelate for the time being, and replace the old C
>> function with the 'correct' one once we remove the deprecated correlate ?
>> 
>
> No, you do the same thing at the C level.
>   

Ok,

David

___
Numpy-discussion mailing list
Numpy-discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Problem with correlate

[Robert Kern]

On Mon, Jun 1, 2009 at 22:33, David Cournapeau
 wrote:
> Charles R Harris wrote:
>>
>>
>> On Mon, Jun 1, 2009 at 11:48 AM, Charles R Harris
>> mailto:charlesr.har...@gmail.com>> wrote:
>>
>>
>>
>>     On Mon, Jun 1, 2009 at 10:35 AM, Robert Kern
>>     mailto:robert.k...@gmail.com>> wrote:
>>
>>         On Mon, Jun 1, 2009 at 00:05, David Cournapeau
>>         >         > wrote:
>>
>>         > I think we should just fix it to use conjugate - I will do
>>         this in the
>>         > branch, and I will integrate it in the trunk later unless
>>         someone stands
>>         > up vehemently against the change. I opened up a ticket to
>>         track this,
>>         > though,
>>
>>         It breaks everyone's code that works around the current behavior.
>>
>>
>>     Maybe we need a new function. But what to call it?
>>
>>
>> How about introducing acorrelate and deprecating the old version?
>
> This does not solve the C function problem (PyArray_Correlate). The easy
> solution would be to keep the current C version, deal with the problem
> in python for acorrelate for the time being, and replace the old C
> function with the 'correct' one once we remove the deprecated correlate ?

No, you do the same thing at the C level.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco
___
Numpy-discussion mailing list
Numpy-discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Problem with correlate

[David Cournapeau]

Charles R Harris wrote:
>
>
> On Mon, Jun 1, 2009 at 11:48 AM, Charles R Harris
> mailto:charlesr.har...@gmail.com>> wrote:
>
>
>
> On Mon, Jun 1, 2009 at 10:35 AM, Robert Kern
> mailto:robert.k...@gmail.com>> wrote:
>
> On Mon, Jun 1, 2009 at 00:05, David Cournapeau
>  > wrote:
>
> > I think we should just fix it to use conjugate - I will do
> this in the
> > branch, and I will integrate it in the trunk later unless
> someone stands
> > up vehemently against the change. I opened up a ticket to
> track this,
> > though,
>
> It breaks everyone's code that works around the current behavior.
>
>
> Maybe we need a new function. But what to call it?
>
>
> How about introducing acorrelate and deprecating the old version?

This does not solve the C function problem (PyArray_Correlate). The easy
solution would be to keep the current C version, deal with the problem
in python for acorrelate for the time being, and replace the old C
function with the 'correct' one once we remove the deprecated correlate ?

David
___
Numpy-discussion mailing list
Numpy-discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Problem with correlate

[Robert Kern]

On Mon, Jun 1, 2009 at 13:44, Charles R Harris
 wrote:
>
>
> On Mon, Jun 1, 2009 at 11:48 AM, Charles R Harris
>  wrote:
>>
>>
>> On Mon, Jun 1, 2009 at 10:35 AM, Robert Kern 
>> wrote:
>>>
>>> On Mon, Jun 1, 2009 at 00:05, David Cournapeau
>>>  wrote:
>>>
>>> > I think we should just fix it to use conjugate - I will do this in the
>>> > branch, and I will integrate it in the trunk later unless someone
>>> > stands
>>> > up vehemently against the change. I opened up a ticket to track this,
>>> > though,
>>>
>>> It breaks everyone's code that works around the current behavior.
>>
>> Maybe we need a new function. But what to call it?
>
> How about introducing acorrelate and deprecating the old version?

Sure.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco
___
Numpy-discussion mailing list
Numpy-discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Problem with correlate

[Charles R Harris]

On Mon, Jun 1, 2009 at 11:48 AM, Charles R Harris  wrote:

>
>
> On Mon, Jun 1, 2009 at 10:35 AM, Robert Kern wrote:
>
>> On Mon, Jun 1, 2009 at 00:05, David Cournapeau
>>  wrote:
>>
>> > I think we should just fix it to use conjugate - I will do this in the
>> > branch, and I will integrate it in the trunk later unless someone stands
>> > up vehemently against the change. I opened up a ticket to track this,
>> > though,
>>
>> It breaks everyone's code that works around the current behavior.
>>
>
> Maybe we need a new function. But what to call it?
>

How about introducing acorrelate and deprecating the old version?

Chuck
___
Numpy-discussion mailing list
Numpy-discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Problem with correlate

[Charles R Harris]

On Mon, Jun 1, 2009 at 10:35 AM, Robert Kern  wrote:

> On Mon, Jun 1, 2009 at 00:05, David Cournapeau
>  wrote:
>
> > I think we should just fix it to use conjugate - I will do this in the
> > branch, and I will integrate it in the trunk later unless someone stands
> > up vehemently against the change. I opened up a ticket to track this,
> > though,
>
> It breaks everyone's code that works around the current behavior.
>

Maybe we need a new function. But what to call it?

Chuck
___
Numpy-discussion mailing list
Numpy-discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Problem with correlate

[Robert Kern]

On Mon, Jun 1, 2009 at 00:05, David Cournapeau
 wrote:

> I think we should just fix it to use conjugate - I will do this in the
> branch, and I will integrate it in the trunk later unless someone stands
> up vehemently against the change. I opened up a ticket to track this,
> though,

It breaks everyone's code that works around the current behavior.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco
___
Numpy-discussion mailing list
Numpy-discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Problem with correlate

[David Cournapeau]

Charles R Harris wrote:
>
>
> On Sun, May 31, 2009 at 9:08 PM, David Cournapeau
> mailto:da...@ar.media.kyoto-u.ac.jp>>
> wrote:
>
> Charles R Harris wrote:
> >
> >
> > On Sun, May 31, 2009 at 7:18 PM, David Cournapeau
> >  
>  >>
> > wrote:
> >
> > Charles R Harris wrote:
> > >
> > >
> > > On Sun, May 31, 2009 at 11:54 AM, rob steed
> mailto:rjst...@talk21.com>
> > >
> > > 
>  > >
> > >
> > > Hi,
> > > After my previous email, I have opened a ticket #1117
> (correlate
> > > not order dependent)
> > >
> > > I have found that the correlate function is defined in
> > > multiarraymodule.c and
> > > that inputs are being swapped using the following code
> > >
> > >n1 = ap1->dimensions[0];
> > >n2 = ap2->dimensions[0];
> > >if (n1 < n2) {
> > >ret = ap1;
> > >ap1 = ap2;
> > >ap2 = ret;
> > >ret = NULL;
> > >i = n1;
> > >n1 = n2;
> > >n2 = i;
> > >}
> > >
> > > I do not know the code well enough to see whether this
> could
> > just
> > > be removed (I don't know c either).
> > > Maybe the algorithmn requires the inputs to be length
> ordered? I
> > > will try to work it out.
> > >
> > >
> > > If the correlation algorithm doesn't use an fft and is done
> > > explicitly, then the maximum overlap for any shift is the
> length of
> > > the shortest input.  Swapping the arrays makes that logic
> easier to
> > > implement, but it isn't necessary.
> >
> > But this logic is also wrong if the swapping is not taken into
> > account -
> > as the OP mentioned, correlate(a, b) is not equal to
> correlate(b,
> > a) in
> > the general case. The output is reversed in the second case
> > compared to
> > the first case.
> >
> >
> > I didn't say it was *correctly* implemented ;)
>
> :) So I gave it a shot
>
> http://github.com/cournape/numpy/commits/fix_correlate
>
> (It took me a while to realize that PyArray_ISFLEXIBLE returns
> false for
> array object. Is this expected ? The documentation concerning copyswap
> says that it is necessary for flexible arrays, but I think it is
> necessary  for object arrays as well).
>
>
> Don't know. PyArray_ISFLEXIBLE looks like a macro... 
>
> #define PyArray_ISFLEXIBLE(obj) PyTypeNum_ISFLEXIBLE(PyArray_TYPE(obj))
>
> #define PyTypeNum_ISFLEXIBLE(type) (((type) >=NPY_STRING) &&  \
> ((type) <=NPY_VOID))
>
> And the typecodes are '?bhilqpBHILQPfdgFDGSUVO'. So 'SUV' are flexible
> and O is not.

I re-read the copyswap documentation, and realized I did not read it
correctly. Now, I am not sure when to use copyswap vs memcpy (memcpy
should be much faster on basic types, as memcpy should be inlined
generally, whereas I doubt copyswap can).

> I'm not clear on how correlate should apply to any of 'SUV' but it
> might be worth having it work for objects.

It already does (I added a couple of unit tests in the branch, since
there were no test for correlate, and one is for Decimal object arrays).

>
>
> It still bothers me that correlate does not conjugate the second
> argument for complex arrays...
>
>
> It bothers me also...

I think we should just fix it to use conjugate - I will do this in the
branch, and I will integrate it in the trunk later unless someone stands
up vehemently against the change. I opened up a ticket to track this,
though,

cheers,

David
___
Numpy-discussion mailing list
Numpy-discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Problem with correlate

[Charles R Harris]

On Sun, May 31, 2009 at 9:08 PM, David Cournapeau <
da...@ar.media.kyoto-u.ac.jp> wrote:

> Charles R Harris wrote:
> >
> >
> > On Sun, May 31, 2009 at 7:18 PM, David Cournapeau
> > mailto:da...@ar.media.kyoto-u.ac.jp>>
> > wrote:
> >
> > Charles R Harris wrote:
> > >
> > >
> > > On Sun, May 31, 2009 at 11:54 AM, rob steed  > 
> > > >> wrote:
> > >
> > >
> > > Hi,
> > > After my previous email, I have opened a ticket #1117
> (correlate
> > > not order dependent)
> > >
> > > I have found that the correlate function is defined in
> > > multiarraymodule.c and
> > > that inputs are being swapped using the following code
> > >
> > >n1 = ap1->dimensions[0];
> > >n2 = ap2->dimensions[0];
> > >if (n1 < n2) {
> > >ret = ap1;
> > >ap1 = ap2;
> > >ap2 = ret;
> > >ret = NULL;
> > >i = n1;
> > >n1 = n2;
> > >n2 = i;
> > >}
> > >
> > > I do not know the code well enough to see whether this could
> > just
> > > be removed (I don't know c either).
> > > Maybe the algorithmn requires the inputs to be length ordered?
> I
> > > will try to work it out.
> > >
> > >
> > > If the correlation algorithm doesn't use an fft and is done
> > > explicitly, then the maximum overlap for any shift is the length of
> > > the shortest input.  Swapping the arrays makes that logic easier to
> > > implement, but it isn't necessary.
> >
> > But this logic is also wrong if the swapping is not taken into
> > account -
> > as the OP mentioned, correlate(a, b) is not equal to correlate(b,
> > a) in
> > the general case. The output is reversed in the second case
> > compared to
> > the first case.
> >
> >
> > I didn't say it was *correctly* implemented ;)
>
> :) So I gave it a shot
>
> http://github.com/cournape/numpy/commits/fix_correlate
>
> (It took me a while to realize that PyArray_ISFLEXIBLE returns false for
> array object. Is this expected ? The documentation concerning copyswap
> says that it is necessary for flexible arrays, but I think it is
> necessary  for object arrays as well).
>

Don't know. PyArray_ISFLEXIBLE looks like a macro...

#define PyArray_ISFLEXIBLE(obj) PyTypeNum_ISFLEXIBLE(PyArray_TYPE(obj))

#define PyTypeNum_ISFLEXIBLE(type) (((type) >=NPY_STRING) &&  \
((type) <=NPY_VOID))

And the typecodes are '?bhilqpBHILQPfdgFDGSUVO'. So 'SUV' are flexible and O
is not. I'm not clear on how correlate should apply to any of 'SUV' but it
might be worth having it work for objects.


> It still bothers me that correlate does not conjugate the second
> argument for complex arrays...
>

It bothers me also... Chuck
___
Numpy-discussion mailing list
Numpy-discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Problem with correlate

[David Cournapeau]

Charles R Harris wrote:
>
>
> On Sun, May 31, 2009 at 7:18 PM, David Cournapeau
> mailto:da...@ar.media.kyoto-u.ac.jp>>
> wrote:
>
> Charles R Harris wrote:
> >
> >
> > On Sun, May 31, 2009 at 11:54 AM, rob steed  
> > >> wrote:
> >
> >
> > Hi,
> > After my previous email, I have opened a ticket #1117 (correlate
> > not order dependent)
> >
> > I have found that the correlate function is defined in
> > multiarraymodule.c and
> > that inputs are being swapped using the following code
> >
> >n1 = ap1->dimensions[0];
> >n2 = ap2->dimensions[0];
> >if (n1 < n2) {
> >ret = ap1;
> >ap1 = ap2;
> >ap2 = ret;
> >ret = NULL;
> >i = n1;
> >n1 = n2;
> >n2 = i;
> >}
> >
> > I do not know the code well enough to see whether this could
> just
> > be removed (I don't know c either).
> > Maybe the algorithmn requires the inputs to be length ordered? I
> > will try to work it out.
> >
> >
> > If the correlation algorithm doesn't use an fft and is done
> > explicitly, then the maximum overlap for any shift is the length of
> > the shortest input.  Swapping the arrays makes that logic easier to
> > implement, but it isn't necessary.
>
> But this logic is also wrong if the swapping is not taken into
> account -
> as the OP mentioned, correlate(a, b) is not equal to correlate(b,
> a) in
> the general case. The output is reversed in the second case
> compared to
> the first case.
>
>
> I didn't say it was *correctly* implemented ;)

:) So I gave it a shot

http://github.com/cournape/numpy/commits/fix_correlate

(It took me a while to realize that PyArray_ISFLEXIBLE returns false for
array object. Is this expected ? The documentation concerning copyswap
says that it is necessary for flexible arrays, but I think it is
necessary  for object arrays as well).

It still bothers me that correlate does not conjugate the second
argument for complex arrays...

cheers,

David
___
Numpy-discussion mailing list
Numpy-discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Problem with correlate

[Charles R Harris]

On Sun, May 31, 2009 at 7:18 PM, David Cournapeau <
da...@ar.media.kyoto-u.ac.jp> wrote:

> Charles R Harris wrote:
> >
> >
> > On Sun, May 31, 2009 at 11:54 AM, rob steed  > > wrote:
> >
> >
> > Hi,
> > After my previous email, I have opened a ticket #1117 (correlate
> > not order dependent)
> >
> > I have found that the correlate function is defined in
> > multiarraymodule.c and
> > that inputs are being swapped using the following code
> >
> >n1 = ap1->dimensions[0];
> >n2 = ap2->dimensions[0];
> >if (n1 < n2) {
> >ret = ap1;
> >ap1 = ap2;
> >ap2 = ret;
> >ret = NULL;
> >i = n1;
> >n1 = n2;
> >n2 = i;
> >}
> >
> > I do not know the code well enough to see whether this could just
> > be removed (I don't know c either).
> > Maybe the algorithmn requires the inputs to be length ordered? I
> > will try to work it out.
> >
> >
> > If the correlation algorithm doesn't use an fft and is done
> > explicitly, then the maximum overlap for any shift is the length of
> > the shortest input.  Swapping the arrays makes that logic easier to
> > implement, but it isn't necessary.
>
> But this logic is also wrong if the swapping is not taken into account -
> as the OP mentioned, correlate(a, b) is not equal to correlate(b, a) in
> the general case. The output is reversed in the second case compared to
> the first case.
>

I didn't say it was *correctly* implemented ;)

Chuck
___
Numpy-discussion mailing list
Numpy-discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Problem with correlate

[David Cournapeau]

Charles R Harris wrote:
>
>
> On Sun, May 31, 2009 at 11:54 AM, rob steed  > wrote:
>
>
> Hi,
> After my previous email, I have opened a ticket #1117 (correlate
> not order dependent)
>
> I have found that the correlate function is defined in
> multiarraymodule.c and
> that inputs are being swapped using the following code
>
>n1 = ap1->dimensions[0];
>n2 = ap2->dimensions[0];
>if (n1 < n2) {
>ret = ap1;
>ap1 = ap2;
>ap2 = ret;
>ret = NULL;
>i = n1;
>n1 = n2;
>n2 = i;
>}
>
> I do not know the code well enough to see whether this could just
> be removed (I don't know c either).
> Maybe the algorithmn requires the inputs to be length ordered? I
> will try to work it out.
>
>
> If the correlation algorithm doesn't use an fft and is done
> explicitly, then the maximum overlap for any shift is the length of
> the shortest input.  Swapping the arrays makes that logic easier to
> implement, but it isn't necessary.

But this logic is also wrong if the swapping is not taken into account -
as the OP mentioned, correlate(a, b) is not equal to correlate(b, a) in
the general case. The output is reversed in the second case compared to
the first case.

cheers,

David
___
Numpy-discussion mailing list
Numpy-discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Problem with correlate

[Charles R Harris]

On Sun, May 31, 2009 at 11:54 AM, rob steed  wrote:

>
> Hi,
> After my previous email, I have opened a ticket #1117 (correlate not order
> dependent)
>
> I have found that the correlate function is defined in multiarraymodule.c
> and
> that inputs are being swapped using the following code
>
>n1 = ap1->dimensions[0];
>n2 = ap2->dimensions[0];
>if (n1 < n2) {
>ret = ap1;
>ap1 = ap2;
>ap2 = ret;
>ret = NULL;
>i = n1;
>n1 = n2;
>n2 = i;
>}
>
> I do not know the code well enough to see whether this could just be
> removed (I don't know c either).
> Maybe the algorithmn requires the inputs to be length ordered? I will try
> to work it out.
>

If the correlation algorithm doesn't use an fft and is done explicitly, then
the maximum overlap for any shift is the length of the shortest input.
Swapping the arrays makes that logic easier to implement, but it isn't
necessary.

Chuck
___
Numpy-discussion mailing list
Numpy-discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

[Numpy-discussion] Problem with correlate

[rob steed]

Hi,
After my previous email, I have opened a ticket #1117 (correlate not order 
dependent)

I have found that the correlate function is defined in multiarraymodule.c and
that inputs are being swapped using the following code

n1 = ap1->dimensions[0];
n2 = ap2->dimensions[0];
if (n1 < n2) {
ret = ap1;
ap1 = ap2;
ap2 = ret;
ret = NULL;
i = n1;
n1 = n2;
n2 = i;
}

I do not know the code well enough to see whether this could just be removed (I 
don't know c either).
Maybe the algorithmn requires the inputs to be length ordered? I will try to 
work it out.


Regards

Rob



  
___
Numpy-discussion mailing list
Numpy-discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Problem with correlate

[josef . pktd]

2009/5/18 Stéfan van der Walt :
> 2009/5/18 rob steed :
>> This works fine. However, if the arrays have different lengths, we get a 
>> problem.
>>
> y2=N.array([0,0,0,1])
> N.correlate(x,y2,'full')
>
> This looks like a bug to me.
>
> In [54]: N.correlate([1, 0, 0, 0], [0, 0, 0, 1],'full')
> Out[54]: array([1, 0, 0, 0, 0, 0, 0])
>
> In [55]: N.correlate([1, 0, 0, 0, 0], [0, 0, 0, 1],'full')
> Out[55]: array([1, 0, 0, 0, 0, 0, 0, 0])
>
> In [56]: N.correlate([1, 0, 0, 0, 0], [0, 0, 0, 0, 1],'full')
> Out[56]: array([1, 0, 0, 0, 0, 0, 0, 0, 0])
>
> In [57]: N.correlate([1, 0, 0, 0, 0], [0, 0, 0, 0, 0, 1],'full')
> Out[57]: array([0, 0, 0, 0, 0, 0, 0, 0, 0, 1])
>

comparing with scipy:
  signal.correlate behaves the same "flipping" way as np.correlate,
  ndimage.correlate keeps the orientation.

>>> np.correlate([1, 2, 0, 0, 0], [0, 0, 1, 0, 0,0],'same')
array([0, 0, 0, 2, 1, 0])
>>> np.correlate([1, 2, 0, 0, 0], [0, 0, 1, 0, 0],'same')
array([1, 2, 0, 0, 0])
>>> np.correlate([1, 2, 0, 0, 0], [0, 0, 1, 0, 0, 0],'full')
array([0, 0, 0, 0, 0, 2, 1, 0, 0, 0])
>>> np.correlate([1, 2, 0, 0, 0], [0, 0, 1, 0, 0],'full')
array([0, 0, 1, 2, 0, 0, 0, 0, 0])
>>>
>>> signal.correlate([1, 2, 0, 0, 0], [0, 0, 1, 0, 0, 0])
array([0, 0, 0, 0, 0, 2, 1, 0, 0, 0])
>>> signal.correlate([1, 2, 0, 0, 0], [0, 0, 1, 0, 0])
array([0, 0, 1, 2, 0, 0, 0, 0, 0])
>>> ndimage.filters.correlate([1, 2, 0, 0, 0], [0, 0, 1, 0, 0, 
>>> 0],mode='constant')
array([0, 1, 2, 0, 0])
>>> ndimage.filters.correlate([1, 2, 0, 0, 0], [0, 0, 1, 0, 0],mode='constant')
array([1, 2, 0, 0, 0])

Josef
___
Numpy-discussion mailing list
Numpy-discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Problem with correlate

[Stéfan van der Walt]

2009/5/18 rob steed :
> This works fine. However, if the arrays have different lengths, we get a 
> problem.
>
 y2=N.array([0,0,0,1])
 N.correlate(x,y2,'full')

This looks like a bug to me.

In [54]: N.correlate([1, 0, 0, 0], [0, 0, 0, 1],'full')
Out[54]: array([1, 0, 0, 0, 0, 0, 0])

In [55]: N.correlate([1, 0, 0, 0, 0], [0, 0, 0, 1],'full')
Out[55]: array([1, 0, 0, 0, 0, 0, 0, 0])

In [56]: N.correlate([1, 0, 0, 0, 0], [0, 0, 0, 0, 1],'full')
Out[56]: array([1, 0, 0, 0, 0, 0, 0, 0, 0])

In [57]: N.correlate([1, 0, 0, 0, 0], [0, 0, 0, 0, 0, 1],'full')
Out[57]: array([0, 0, 0, 0, 0, 0, 0, 0, 0, 1])

Regards
Stéfan
___
Numpy-discussion mailing list
Numpy-discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

[Numpy-discussion] Problem with correlate

[rob steed]

Hi all,

I have been using numpy.correlate and was finding something weird. I now think 
that there might be a bug.

Correlations should be order dependent eg. correlate(x,y) != correlate(y,x) in 
general (whereas convolutions are symmetric)

>>> import numpy as N
>>> x = N.array([1,0,0])
>>> y = N.array([0,0,1])

>>> N.correlate(x,y,'full')
array([1, 0, 0, 0, 0])
>>> N.correlate(y,x,'full')
array([0, 0, 0, 0, 1])

This works fine. However, if the arrays have different lengths, we get a 
problem.

>>> y2=N.array([0,0,0,1])
>>> N.correlate(x,y2,'full')
array([0, 0, 0, 0, 0, 1])
>>> N.correlate(y2,x,'full')
array([0, 0, 0, 0, 0, 1])

I believe that somewhere in the code, the arrays are re-ordered by their 
length. Initially I thought that this was because
correlate was deriving from convolution but looking at numpy.core, I can see 
that in fact convolution derives from correlate. 
After that, it becomes C code which I haven't managed to look at yet.

Am I correct, is this a bug? 

regards

Rob Steed



  
___
Numpy-discussion mailing list
Numpy-discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Problem with correlate?

[Hanno Klemm]

Hi Stefan,

I checked it with numpy version 1.1.1 just now and the result is the same:

>>> x = N.array([0.,0,1,0,0])
>>> y1 = N.array([1.,0,0,0,0])
>>> N.correlate(x,y1, mode='full')
array([ 0.,  0.,  0.,  0.,  0.,  0.,  1.,  0.,  0.])
>>> y2 = N.array([1.,0,0,0,0,0,0])
>>> N.correlate(x,y2, mode='full')
array([ 0.,  0.,  1.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.])
>>> N.__version__
'1.1.1'
>>>

Best regards,
Hanno

Stéfan van der Walt <[EMAIL PROTECTED]> said:

> Hi Hanno
> 
> 2008/8/22 Hanno Klemm <[EMAIL PROTECTED]>:
> > yes, indeed, that's what I thought. This result is odd. Has correlate
> > been changed since version 1.0.4, or should I submit this as a bug?
> 
> Is there any way that you could try out the latest release on your
> machine and see if it solves your problem?  We probably won't be
> releasing bug-fixes on 1.0.4, but if it exists in 1.1 we'll still
> address it.  I'm not aware of any changes, but I may simply have
> missed it.
> 
> Regards
> Stéfan
> ___
> Numpy-discussion mailing list
> Numpy-discussion@scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
> 



-- 
Hanno Klemm
[EMAIL PROTECTED]


___
Numpy-discussion mailing list
Numpy-discussion@scipy.org
http://projects.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Problem with correlate?

[Stéfan van der Walt]

Hi Hanno

2008/8/22 Hanno Klemm <[EMAIL PROTECTED]>:
> yes, indeed, that's what I thought. This result is odd. Has correlate
> been changed since version 1.0.4, or should I submit this as a bug?

Is there any way that you could try out the latest release on your
machine and see if it solves your problem?  We probably won't be
releasing bug-fixes on 1.0.4, but if it exists in 1.1 we'll still
address it.  I'm not aware of any changes, but I may simply have
missed it.

Regards
Stéfan
___
Numpy-discussion mailing list
Numpy-discussion@scipy.org
http://projects.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Problem with correlate?

[Hanno Klemm]

Hi Stefan,

yes, indeed, that's what I thought. This result is odd. Has correlate
been changed since version 1.0.4, or should I submit this as a bug?

Best regards,
Hanno

Stéfan van der Walt <[EMAIL PROTECTED]> said:

> --=_Part_25307_10322093.1219268954678
> Content-Type: text/plain; charset=ISO-8859-1
> Content-Transfer-Encoding: quoted-printable
> Content-Disposition: inline
> 
> 2008/8/20 Hanno Klemm <[EMAIL PROTECTED]>:
> > In [29]: x =3D array([0.,0.,1, 0, 0])
> > In [35]: y1 =3D array([1,0,0,0,0])
> >
> > In [36]: correlate(x,y1,mode=3D'full')
> > Out[36]: array([ 0.,  0.,  0.,  0.,  0.,  0.,  1.,  0.,  0.])
> 
> That doesn't look right.  Under r5661:
> 
> In [60]: np.convolve([0, 0, 1, 0, 0], [1, 0, 0, 0, 0], mode=3D'f')
> Out[60]: array([0, 0, 1, 0, 0, 0, 0, 0, 0])
> 
> Regards
> St=E9fan
> 
> --=_Part_25307_10322093.1219268954678
> Content-Type: text/html; charset=ISO-8859-1
> Content-Transfer-Encoding: quoted-printable
> Content-Disposition: inline
> 
> 2008/8/20 Hanno Klemm  z.ch">[EMAIL PROTECTED]>:> In [29]: x =3D
array([0.,0.,1, 0,=
>  0])> In [35]: y1 =3D array([1,0,0,0,0])>> In
[36]: co=
> rrelate(x,y1,mode=3D'full')
> > Out[36]: array([ 0.,  0.,  0.,  0.,  0.,
 0.,=
>   1.,  0.,  0.])That doesn't look
right.  U=
> nder r5661:In [60]: np.convolve([0, 0, 1, 0, 0], [1, 0, 0,
0, 0], m=
> ode=3D'f')Out[60]: array([0, 0, 1, 0, 0, 0, 0, 0, 0])
> RegardsSt=E9fan
> 
> --=_Part_25307_10322093.1219268954678--
> 



-- 
Hanno Klemm
[EMAIL PROTECTED]


___
Numpy-discussion mailing list
Numpy-discussion@scipy.org
http://projects.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Problem with correlate?

[Stéfan van der Walt]

2008/8/20 Hanno Klemm <[EMAIL PROTECTED]>:
> In [29]: x = array([0.,0.,1, 0, 0])
> In [35]: y1 = array([1,0,0,0,0])
>
> In [36]: correlate(x,y1,mode='full')
> Out[36]: array([ 0.,  0.,  0.,  0.,  0.,  0.,  1.,  0.,  0.])

That doesn't look right.  Under r5661:

In [60]: np.convolve([0, 0, 1, 0, 0], [1, 0, 0, 0, 0], mode='f')
Out[60]: array([0, 0, 1, 0, 0, 0, 0, 0, 0])

Regards
Stéfan
___
Numpy-discussion mailing list
Numpy-discussion@scipy.org
http://projects.scipy.org/mailman/listinfo/numpy-discussion

[Numpy-discussion] Problem with correlate?

[Hanno Klemm]

Hi All,

after the discussion on numpy.correlate some time ago, regarding
complex conjugation, etc. I today was pointed to yet another oddity,
which I hope somebody could explain to me, as to why that's a feature,
rather than a bug. I'm thoroughly confused by the following behaviour: 
 

In [29]: x = array([0.,0.,1, 0, 0])

In [30]: y = array([0, 0, 0, 1., 0, 0, 0])

In [31]: correlate(x,y)
Out[31]: array([ 0.,  1.,  0.])

In [32]: correlate(x,y, mode='full')
Out[32]: array([ 0.,  0.,  0.,  0.,  0.,  1.,  0.,  0.,  0.,  0.,  0.])

In [33]: y = array([0,0,1.,0,0])

In [34]: correlate(x,y,mode='full')
Out[34]: array([ 0.,  0.,  0.,  0.,  1.,  0.,  0.,  0.,  0.])

So far, everything is allright and as expected. Now:

In [35]: y1 = array([1,0,0,0,0])

In [36]: correlate(x,y1,mode='full')
Out[36]: array([ 0.,  0.,  0.,  0.,  0.,  0.,  1.,  0.,  0.])

In [37]: y2 = array([1,0,0,0,0,0,0])

In [38]: correlate(x,y2,mode='full')
Out[38]: array([ 0.,  0.,  1.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.])

Could somebody please help me understanding, why the one switches
places? Furthermore, this behaviour does not seem to be very
consistent, if we switch the x and y argument:

In [39]: correlate(y2,x,mode='full')
Out[39]: array([ 0.,  0.,  1.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.])

In [40]: correlate(y1,x,mode='full')
Out[40]: array([ 0.,  0.,  1.,  0.,  0.,  0.,  0.,  0.,  0.])

The answer remains the same. Any help would be appreciated, the numpy
version is 1.0.4.

Best regards,
Hanno




-- 
Hanno Klemm
[EMAIL PROTECTED]


___
Numpy-discussion mailing list
Numpy-discussion@scipy.org
http://projects.scipy.org/mailman/listinfo/numpy-discussion