Re: [Numpy-discussion] convert strides/shape/offset into nd index?

[Anne Archibald]

2009/11/30 James Bergstra :
> Your question involves a few concepts:
>
> - an integer vector describing the position of an element
>
> - the logical shape (another int vector)
>
> - the physical strides (another int vector)
>
> Ignoring the case of negative offsets, a physical offset is the inner
> product of the physical strides with the position vector.
>
> In these terms, you are asking how to solve the inner-product equation
> for the position vector.  There can be many possible solutions (like,
> if there is a stride of 1, then you can make that dimension account
> for the entire offset.  This is often not the solution you want.).
> For valid ndarrays though, there is at most one solution though with
> the property that every position element is less than the shape.
>
> You will also need to take into account that for certain stride
> vectors, there is no way to get certain offsets.  Imagine all the
> strides were even, and you needed to get at an odd offset... it would
> be impossible.  It would even be impossible if there were a dimension
> with stride 1 but it had shape of 1 too.
>
> I can't think of an algorithm off the top of my head that would do
> this in a quick and elegant way.

Not to be a downer, but this problem is technically NP-complete. The
so-called "knapsack problem" is to find a subset of a collection of
numbers that adds up to the specified number, and it is NP-complete.
Unfortunately, it is exactly what you need to do to find the indices
to a particular memory location in an array of shape (2,2,...,2).

What that means in practice is that either you have to allow
potentially very slow algorithms (though you know that there will
never be more than 32 different values in the knapsack, which might or
might not be enough to keep things tractable) or use heuristic
algorithms that don't always work. There are probably fairly good
heuristics, particularly if the array elements are all at distinct
memory locations (arrays with overlapping elements can arise from
broadcasting and other slightly more arcane operations).

Anne

>
> James
>
> On Sun, Nov 29, 2009 at 10:36 AM, Zachary Pincus
>  wrote:
>> Hi all,
>>
>> I'm curious as to what the most straightforward way is to convert an
>> offset into a memory buffer representing an arbitrarily strided array
>> into the nd index into that array. (Let's assume for simplicity that
>> each element is one byte...)
>>
>> Does sorting the strides from largest to smallest and then using
>> integer division and mod (in the obvious way) always work? (I can't
>> seem to find a counterexample, but I am not used to thinking too
>> deeply about bizarrely-strided configurations.) Is there a method that
>> doesn't involve sorting?
>>
>> Thanks,
>> Zach
>> ___
>> NumPy-Discussion mailing list
>> NumPy-Discussion@scipy.org
>> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>>
>
>
>
> --
> http://www-etud.iro.umontreal.ca/~bergstrj
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

[Numpy-discussion] Numpy 1.4.0 rc1 released

[David Cournapeau]

Hi,

The first release candidate for 1.4.0 has been released. The sources,
as well as mac and windows installers may be found here:

https://sourceforge.net/projects/numpy/files/

The main improvements compared to 1.3.0 are:

* Faster import time
* Extended array wrapping mechanism for ufuncs
* New Neighborhood iterator (C-level only)
* C99-like complex functions in npymath

As well as more than 50 bug fixes. The detailed list of changes may be
found on trac:

http://projects.scipy.org/numpy/roadmap

cheers,

David
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] convert strides/shape/offset into nd index?

[James Bergstra]

Your question involves a few concepts:

- an integer vector describing the position of an element

- the logical shape (another int vector)

- the physical strides (another int vector)

Ignoring the case of negative offsets, a physical offset is the inner
product of the physical strides with the position vector.

In these terms, you are asking how to solve the inner-product equation
for the position vector.  There can be many possible solutions (like,
if there is a stride of 1, then you can make that dimension account
for the entire offset.  This is often not the solution you want.).
For valid ndarrays though, there is at most one solution though with
the property that every position element is less than the shape.

You will also need to take into account that for certain stride
vectors, there is no way to get certain offsets.  Imagine all the
strides were even, and you needed to get at an odd offset... it would
be impossible.  It would even be impossible if there were a dimension
with stride 1 but it had shape of 1 too.

I can't think of an algorithm off the top of my head that would do
this in a quick and elegant way.

James

On Sun, Nov 29, 2009 at 10:36 AM, Zachary Pincus
 wrote:
> Hi all,
>
> I'm curious as to what the most straightforward way is to convert an
> offset into a memory buffer representing an arbitrarily strided array
> into the nd index into that array. (Let's assume for simplicity that
> each element is one byte...)
>
> Does sorting the strides from largest to smallest and then using
> integer division and mod (in the obvious way) always work? (I can't
> seem to find a counterexample, but I am not used to thinking too
> deeply about bizarrely-strided configurations.) Is there a method that
> doesn't involve sorting?
>
> Thanks,
> Zach
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>



-- 
http://www-etud.iro.umontreal.ca/~bergstrj
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Packaging numpy with py2app v0.4.3

[Christopher Barker]

Donovan Parks wrote:

I think this may have nothing at all to do with numpy. Have you tried a 
simple script that doesn't use numpy?
> OS X Leopard
> MacPort

I use neither Leopard nor macports, so it may be an issue there, but 
with the python.org python25 and numpy1.3, I have no problems.

> When I run 'python setup.py py2app' I get the following error after
> py2app starts stripping out debug information:
> 
> /usr/bin/strip: for architecture i386 object:
> /Users/dparks/test/dist/numpyTest.app/Contents/Frameworks/libgcc_s.1.dylib
> truncated or malformed object (LC_SEGMENT command 2 fileoff field plus

note that this is a libgcc error -- you wouldn't need libgcc if you used 
the python.org build...

> ZipImportError: can't decompress data; zlib not available

> I tried --no-strip which does cause a clean 'build' from py2app,

I don't think this has anything to do with the stripping (or numpy). 
py2app, by default, puts all the modules you need into a zipped archive, 
so any import you do is going to be a problem if zlib is not available.

I suspect this may be a macports/py2app incompatibility -- with the 
python.org python, it probably uses a system zlib, so it hasn't bundled 
up zlib from macports for you.

I'd try asking on the pythonmac list.

-Chris

-- 
Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

[Numpy-discussion] Packaging numpy with py2app v0.4.3

[Donovan Parks]

Hello,

I am trying to package a simple python script which uses numpy using
py2app. There was a lengthy discussion about this back in 2006 on this
list, but it doesn't seem relevant anymore given the changes to numpy
and py2app over the last few years. I am hoping someone on this list
has experience with py2app and can give me a hand.

My system is as follows:

OS X Leopard
MacPort
Python 2.6.4 (installed via MacPort two days ago)
py26-py2app 0.4.3 (installed via MacPort two days ago)
py26-numpy 1.3.0 (installed via MacPort two days ago)


The minimal numpy script I've been working with is:

from numpy import array

a = array( [ 10,20,30,40 ] )
print a


I've been trying to package this with numpy using the following setup.py file:

from setuptools import setup

APP = ['numpyTest.py']
OPTIONS = {'argv_emulation': True}
setup(
 app=APP,
 options={'py2app': OPTIONS},
 setup_requires=['py2app'],
)


When I run 'python setup.py py2app' I get the following error after
py2app starts stripping out debug information:

/usr/bin/strip: for architecture i386 object:
/Users/dparks/test/dist/numpyTest.app/Contents/Frameworks/libgcc_s.1.dylib
truncated or malformed object (LC_SEGMENT command 2 fileoff field plus
filesize field extends past the end of the file)
stripping saved 2592052 bytes (32431544 / 35023596)

A numpyTest.app file is created in the dist directory. However, when I
try to run this file I get the following error:

ZipImportError: can't decompress data; zlib not available

I tried --no-strip which does cause a clean 'build' from py2app, but I
still get the same error message about zlib. Explicitly adding zlib to
setup.py doesn't change anything. I've also tried going into
numpyTest.app and unzipping the site-packages.zip file, but again this
makes no difference. I'm at a bit of a lost at what to try next since
my understanding is that py2app has a built-in recipe for numpy so
should more-or-less work out of the box. Any insight into what the
stripping error relates to?

Perhaps I am missing something simple here. Does one need to explictly
tell py2app what recipes to use (the documentation doesn't make it
sound that way)? Do I need to explicitly indicate I have a 32-bit or
64-bit system (actually not sure what it is at the moment)?

Thanks for any and all help.

Regards,
Donovan
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

[Numpy-discussion] Python 3K merge

[Charles R Harris]

Hi Pauli,

It looks like you doing great stuff with the py3k transition. Do you and
David have any sort of merge schedule in mind?

Chuck
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Is anyone knowledgeable about dll deployment on windows ?

[Christoph Gohlke]

The most popular/simple way to deal with the VC90.CRT dependency issue 
is to have the user install the runtime redistributable on their system. 
If you don't want to put that burden on the user, which I understand, 
you have to make adjustments to the assembly manifests. This is not 
unofficial or unsupported. It is a bug in Python that it embeds the 
assemblyIdentity for VC90.CRT in all extensions build with 
distutils/msvc9compiler.py. In fact, the *.pyd distributed with Python 
2.6.3+ don't have that problem. Maybe you can raise your concerns about 
future compatibility at .

Christoph

On 11/30/2009 1:11 AM, Eloi Gaudry wrote:
> Christoph, thanks for pointing this discussion. That's a perfect match.
>
> If the workaround provided offers a solution to the current
> redistribution issue, I'm wondering if it will still be the case when an
> update to the assembly check function will be activated/implemented
> (within Windows).
> The manifest edition (removing the "assemblyIdentity" tag) doesn't seem
> to be a popular/official/supported way of dealing with the whole runtime
> libraries issue. Don't you think ?
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Is anyone knowledgeable about dll deployment on windows ?

[Sturla Molden]

David Cournapeau skrev:
> We are talking about the numpy extensions here, which are not
> installed through the install_data command. The problem is about how
> windows looks for dll with the manifest mechanism, and how to
> build/install extensions when the C runtime (or any other "system"
> dll) is not installed in the SxS cache.
>   
Is COM (aka ActiveX) out of the question?


___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Is anyone knowledgeable about dll deployment on windows ?

[Sturla Molden]

David Cournapeau skrev:
> We are talking about the numpy extensions here, which are not
> installed through the install_data command. The problem is about how
> windows looks for dll with the manifest mechanism, and how to
> build/install extensions when the C runtime (or any other "system"
> dll) is not installed in the SxS cache.
>   
Do we need to support Windows 2000?

S.M.

___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Is anyone knowledgeable about dll deployment on windows ?

[David Cournapeau]

On Mon, Nov 30, 2009 at 10:48 PM, Sturla Molden  wrote:
> David Cournapeau skrev:
>> If every python package starts to put its extensions (*.pyd) into a
>> directory, what happens when two different packages have an extension
>> with the same name (e.g. package foo has a package multiarray.pyd) ? I
>> would also be really annoyed if a 3rd party extension starts polluting
>> my C:\Python26.
>>
>>
> In disutils, data_files can install DLLs where Python wants them. Just
> pass 'DLLs' as path, and the rest it up to distutils. If anyone pollutes
> your c:\Python26 it is distutils, not a third party extension. This is
> not any different from installing other data files.

We are talking about the numpy extensions here, which are not
installed through the install_data command. The problem is about how
windows looks for dll with the manifest mechanism, and how to
build/install extensions when the C runtime (or any other "system"
dll) is not installed in the SxS cache.

David
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Is anyone knowledgeable about dll deployment on windows ?

[Sturla Molden]

David Cournapeau skrev:
> If every python package starts to put its extensions (*.pyd) into a
> directory, what happens when two different packages have an extension
> with the same name (e.g. package foo has a package multiarray.pyd) ? I
> would also be really annoyed if a 3rd party extension starts polluting
> my C:\Python26.
>
>   
In disutils, data_files can install DLLs where Python wants them. Just 
pass 'DLLs' as path, and the rest it up to distutils. If anyone pollutes 
your c:\Python26 it is distutils, not a third party extension. This is 
not any different from installing other data files.

Sturla

 





___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] non-standard standard deviation

[Sturla Molden]

Colin J. Williams skrev:
> Where the distribution of a variate is not known a priori, then I 
> believe that it can be shown
> that the n-1 divisor provides the best estimate of the variance.
>   
Have you ever been shooting with a rifle?

What would you rather do:

- Hit 9 or 10, with a bias to the right.

- Hit 7 or better, with no bias.

Do you think it can be shown that the latter option is the better?

No?


Sturla Molden
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Is anyone knowledgeable about dll deployment on windows ?

[David Cournapeau]

On Mon, Nov 30, 2009 at 8:52 PM, Eloi Gaudry  wrote:

>
> Well, I wasn't aware of Microsoft willing to giving up the whole
> SxS/manifest thing. Is there any MSDN information available?

I have seen this mentioned for the first time on the python-dev ML:

http://aspn.activestate.com/ASPN/Mail/Message/python-dev/3764855

The mention of including the version in the dll file, if true, is
tragically comic. Maybe in 20 years windows will be able to have a
system which exists for more than a decade on conventional unix... The
link given by M.A Lemburg has changed since, though, as the
description is nowhere to be found in the link. I think I have read
that VS 2010 will never install the runtime in the SxS configuration,
but I of course cannot find this information anymore. Maybe it is not
true anymore, VS 2010 has not yet been released.

You can also find useful manifest troubleshooting information there:

http://blogs.msdn.com/junfeng/archive/2006/04/14/576314.aspx

cheers,

David
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Is anyone knowledgeable about dll deployment on windows ?

[Eloi Gaudry]

David Cournapeau wrote:
> Eloi Gaudry wrote:
>   
>> David, I think that what Christoph pointed here applies to the msvc 
>> built libraries only (I need to build the whole python/extensions using 
>> msvc).
>>   
>> 
>
> Ok, I did not understand that point. Do you know if the patch discussed
> in 4120 has been applied to 2.6.4 ? 
>   
I don't think so... I'm using 2.6.4 and still wondering how to 
circumvent the runtime redistribution issue.
>   
>> May I ask you what re the "clashes between extensions" you were 
>> referring to in your previous answer ? Have you already encountered such 
>> issues ?
>>   
>> 
>
> If every python package starts to put its extensions (*.pyd) into a
> directory, what happens when two different packages have an extension
> with the same name (e.g. package foo has a package multiarray.pyd) ? I
> would also be really annoyed if a 3rd party extension starts polluting
> my C:\Python26.
>   
You made your point.
>   
>> I'm wondering how it could break things asp roviding libraries in an 
>> unique location is a known and common practice.
>> 
>
> The problem is that numpy does not "own" C:\Python*, and it is not
> expected that projects start dumping their stuff there. It usually
> "works" on windows because each applications is independent of each
> other. But here, numpy extends python, so we cannot just do as we please
> - especially to deal with this whole manifest stupidity. Given that even
> MS has implicitly recognized how broken manifests were (they are giving
> it up for VS 2010), I don't think it worths spending time on this issue
> which may well be unsolvable in a reliable way anyway. AFAIK, the
> related issue of broken "install for me" python installers on vista and
> windows 7 has not been solved yet.
>   
Well, I wasn't aware of Microsoft willing to giving up the whole 
SxS/manifest thing. Is there any MSDN information available?
Thanks for your input David,
Eloi
> cheers,
>
> David
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>   


-- 


Eloi Gaudry

Free Field Technologies
Axis Park Louvain-la-Neuve
Rue Emile Francqui, 1
B-1435 Mont-Saint Guibert
BELGIUM

Company Phone: +32 10 487 959
Company Fax:   +32 10 454 626

___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Is anyone knowledgeable about dll deployment on windows ?

[David Cournapeau]

Eloi Gaudry wrote:
> David, I think that what Christoph pointed here applies to the msvc 
> built libraries only (I need to build the whole python/extensions using 
> msvc).
>   

Ok, I did not understand that point. Do you know if the patch discussed
in 4120 has been applied to 2.6.4 ? 

> May I ask you what re the "clashes between extensions" you were 
> referring to in your previous answer ? Have you already encountered such 
> issues ?
>   

If every python package starts to put its extensions (*.pyd) into a
directory, what happens when two different packages have an extension
with the same name (e.g. package foo has a package multiarray.pyd) ? I
would also be really annoyed if a 3rd party extension starts polluting
my C:\Python26.

> I'm wondering how it could break things asp roviding libraries in an 
> unique location is a known and common practice.

The problem is that numpy does not "own" C:\Python*, and it is not
expected that projects start dumping their stuff there. It usually
"works" on windows because each applications is independent of each
other. But here, numpy extends python, so we cannot just do as we please
- especially to deal with this whole manifest stupidity. Given that even
MS has implicitly recognized how broken manifests were (they are giving
it up for VS 2010), I don't think it worths spending time on this issue
which may well be unsolvable in a reliable way anyway. AFAIK, the
related issue of broken "install for me" python installers on vista and
windows 7 has not been solved yet.

cheers,

David
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Is anyone knowledgeable about dll deployment on windows ?

[Eloi Gaudry]

David, I think that what Christoph pointed here applies to the msvc 
built libraries only (I need to build the whole python/extensions using 
msvc).

May I ask you what re the "clashes between extensions" you were 
referring to in your previous answer ? Have you already encountered such 
issues ?
I'm wondering how it could break things asp roviding libraries in an 
unique location is a known and common practice.

Eloi


David Cournapeau wrote:
> On Mon, Nov 30, 2009 at 4:25 AM, Christoph Gohlke  wrote:
>   
>> Eloi: take a look at , which contains
>> a discussion on, and a patch for, not embedding manifest files in *.pyd
>> when compiling with MSVC.
>> 
>
> Note that the official binaries are built with mingw, and thus do not
> embed any manifest in the .pyd.
>
> David
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>   


-- 


Eloi Gaudry

Free Field Technologies
Axis Park Louvain-la-Neuve
Rue Emile Francqui, 1
B-1435 Mont-Saint Guibert
BELGIUM

Company Phone: +32 10 487 959
Company Fax:   +32 10 454 626

___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Is anyone knowledgeable about dll deployment on windows ?

[Eloi Gaudry]

Christoph, thanks for pointing this discussion. That's a perfect match.

If the workaround provided offers a solution to the current 
redistribution issue, I'm wondering if it will still be the case when an 
update to the assembly check function will be activated/implemented 
(within Windows).
The manifest edition (removing the "assemblyIdentity" tag) doesn't seem 
to be a popular/official/supported way of dealing with the whole runtime 
libraries issue. Don't you think ?

Christoph Gohlke wrote:
> Eloi: take a look at , which contains 
> a discussion on, and a patch for, not embedding manifest files in *.pyd 
> when compiling with MSVC.
>
> You could recompile the PYD/DLL files that depend on the VC90.CRT in 
> winSXS with the patch applied, or manually remove the manifests from 
> those PYD/DLL files using a HEX editor or script. Also make sure that 
> all DLLs that your PYD files depend on are in the Windows search PATH.
>
> Christoph
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>   


-- 


Eloi Gaudry

Free Field Technologies
Axis Park Louvain-la-Neuve
Rue Emile Francqui, 1
B-1435 Mont-Saint Guibert
BELGIUM

Company Phone: +32 10 487 959
Company Fax:   +32 10 454 626

___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion