Re: [Numpy-discussion] how to compile Fortran using setup.py

[Ondrej Certik]

On Thu, Mar 10, 2011 at 8:25 PM, Robert Kern  wrote:
> On Thu, Mar 10, 2011 at 19:58, Ondrej Certik  wrote:
>> Hi,
>>
>> I spent about an hour googling and didn't figure this out. Here is my 
>> setup.py:
>>
>> setup(
>>    name = "libqsnake",
>>    cmdclass = {'build_ext': build_ext},
>>    version = "0.1",
>>    packages = [
>>        'qsnake',
>>        'qsnake.calculators',
>>        'qsnake.calculators.tests',
>>        'qsnake.data',
>>        'qsnake.mesh2d',
>>        'qsnake.tests',
>>        ],
>>    package_data = {
>>        'qsnake.tests': ['phaml_data/domain.*'],
>>        },
>>    include_dirs=[numpy.get_include()],
>>    ext_modules = [Extension("qsnake.cmesh", [
>>        "qsnake/cmesh.pyx",
>>        "qsnake/fmesh.f90",
>>        ])],
>>    description = "Qsnake standard library",
>>    license = "BSD",
>> )
>>
>> The qsnake.cmesh extension needs to compile .pyx into .c and later to
>> .o, it needs to use gfortran to compile fmesh.f90 to fmesh.o, and then
>> link both things. That's it. In other words, this is what I want
>> distutils to do:
>>
>> $ cython cmesh.pyx
>> $ gcc -fPIC -o cmesh.o -c cmesh.c -I$SPKG_LOCAL/include/python2.6
>> -I$SPKG_LOCAL/lib/python2.6/site-packages/numpy/core/include
>> $ gfortran -fPIC -o fmesh.o -c fmesh.f90
>> $ gcc -shared -o cmesh.so cmesh.o fmesh.o
>
> Difficult if sticking with distutils of any flavor. You would have
> probably to replace the build_ext command to handle both the Fortran
> and the Cython. You may want to give David Cournapeau's Bento a try:
>
>  http://pypi.python.org/pypi/bento/

Thanks Robert. I burned most of my today's evening on this, trying to
replace the command, but so far I didn't figure it out. It is indeed
difficult, but I wasn't sure, whether it is because I am not so
familiar with distutils.

I looked at bento, but I'll simply stick to cmake. I thought that for
a Python package (that uses cython + C or fortran, which I thought is
a pretty standard configuration for scientific computing), I would use
distutils, just like other Python packages.

With cmake, I have already figured out how to mix fotran, c, c++,
Cython and Python and everything works nicely together, very robustly.
So if I have to hack distutils anyway, I would write a simple
distutils -> cmake bridge.

Ondrej
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] C API freeze and review

[Ralf Gommers]

On Thu, Mar 10, 2011 at 10:10 PM, Charles R Harris
 wrote:
>
>
> On Thu, Mar 10, 2011 at 12:25 AM, Charles R Harris
>  wrote:
>>
>>
>> On Thu, Mar 10, 2011 at 12:15 AM, Ralf Gommers
>>  wrote:
>>>
>>> Hi all,
>>>
>>> In preparation for making a 1.6.x branch, I just updated the C API
>>> version. Please do not add any more functions before the branch is
>>> created.
>>>
>>> Over 70 new functions and types were added, we should also review if
>>> all those are necessary and if they are documented. Below is a list of
>>> all new objects (generated with attached script), and an indication of
>>> those items being documented.
>>>
>>> Most function are from the new iterator, are they all necessary?
>>>
>>> And can the authors of the items with missing docs please write some?
>>>
>>> Cheers,
>>> Ralf
>>>
>>>
>>> NpyIter_Type
>>> PyDatetimeArrType_Type
>>> PyHalfArrType_Type
>>> PyTimeIntegerArrType_Type
>>> PyTimedeltaArrType_Type
>>>
>>> NpyIter_Copy                            YES
>>> NpyIter_CreateCompatibleStrides
>>> NpyIter_Deallocate                      YES
>>> NpyIter_DebugPrint
>>> NpyIter_GetAxisStrideArray              YES
>>> NpyIter_GetBufferSize                   YES
>>> NpyIter_GetDataPtrArray                 YES
>>> NpyIter_GetDescrArray                   YES
>>> NpyIter_GetGetCoords                    YES
>>> NpyIter_GetIndexPtr                     YES
>>> NpyIter_GetInitialDataPtrArray
>>> NpyIter_GetInnerFixedStrideArray        YES
>>> NpyIter_GetInnerLoopSizePtr             YES
>>> NpyIter_GetInnerStrideArray             YES
>>> NpyIter_GetIterIndex                    YES
>>> NpyIter_GetIterIndexRange               YES
>>> NpyIter_GetIterNext                     YES
>>> NpyIter_GetIterSize                     YES
>>> NpyIter_GetIterView                     YES
>>> NpyIter_GetNDim                         YES
>>> NpyIter_GetNIter                        YES
>>> NpyIter_GetOperandArray                 YES
>>> NpyIter_GetReadFlags                    YES
>>> NpyIter_GetShape                        YES
>>> NpyIter_GetWriteFlags                   YES
>>> NpyIter_GotoCoords                      YES
>>
>> I'd like to rename this something like  NpyIter_GotoMultiIndex,
>> NpyIter_GotoPolyIndex, or some such. We need a name for extended indices.
>> Suggestions?
>>
>
> Maybe the following terms?
>
> Index -- single index
> FlatIndex -- single index into raveled array
> FullIndex -- [1,2,3]
> FancyIndex -- fancy index

Thanks to Mark for quickly adding docs and renaming some of the above
items. The undocumented items left are:

NpyIter_Type
NpyIter_CreateCompatibleStrides
NpyIter_DebugPrint# does this need to be exposed?

PyDatetimeArrType_Type
PyTimeIntegerArrType_Type
PyTimedeltaArrType_Type
PyArray_DatetimeStructToDatetime
PyArray_DatetimeToDatetimeStruct
PyArray_SetDatetimeParseFunction
PyArray_TimedeltaStructToTimedelta
PyArray_TimedeltaToTimedeltaStruct

PyHalfArrType_Type
PyArray_ConvertClipmodeSequence


I know the datetime authors are quite busy, but it would really be
helpful if they would make a small update to the NEPs to indicate what
is implemented, still to come, or will not be done, and copy some of
the applicable documentation from the NEP to the C API docs and
relevant docstrings.

Thanks,
Ralf
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] how to compile Fortran using setup.py

[Robert Kern]

On Thu, Mar 10, 2011 at 19:58, Ondrej Certik  wrote:
> Hi,
>
> I spent about an hour googling and didn't figure this out. Here is my 
> setup.py:
>
> setup(
>    name = "libqsnake",
>    cmdclass = {'build_ext': build_ext},
>    version = "0.1",
>    packages = [
>        'qsnake',
>        'qsnake.calculators',
>        'qsnake.calculators.tests',
>        'qsnake.data',
>        'qsnake.mesh2d',
>        'qsnake.tests',
>        ],
>    package_data = {
>        'qsnake.tests': ['phaml_data/domain.*'],
>        },
>    include_dirs=[numpy.get_include()],
>    ext_modules = [Extension("qsnake.cmesh", [
>        "qsnake/cmesh.pyx",
>        "qsnake/fmesh.f90",
>        ])],
>    description = "Qsnake standard library",
>    license = "BSD",
> )
>
> The qsnake.cmesh extension needs to compile .pyx into .c and later to
> .o, it needs to use gfortran to compile fmesh.f90 to fmesh.o, and then
> link both things. That's it. In other words, this is what I want
> distutils to do:
>
> $ cython cmesh.pyx
> $ gcc -fPIC -o cmesh.o -c cmesh.c -I$SPKG_LOCAL/include/python2.6
> -I$SPKG_LOCAL/lib/python2.6/site-packages/numpy/core/include
> $ gfortran -fPIC -o fmesh.o -c fmesh.f90
> $ gcc -shared -o cmesh.so cmesh.o fmesh.o

Difficult if sticking with distutils of any flavor. You would have
probably to replace the build_ext command to handle both the Fortran
and the Cython. You may want to give David Cournapeau's Bento a try:

  http://pypi.python.org/pypi/bento/

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

[Numpy-discussion] how to compile Fortran using setup.py

[Ondrej Certik]

Hi,

I spent about an hour googling and didn't figure this out. Here is my setup.py:

setup(
name = "libqsnake",
cmdclass = {'build_ext': build_ext},
version = "0.1",
packages = [
'qsnake',
'qsnake.calculators',
'qsnake.calculators.tests',
'qsnake.data',
'qsnake.mesh2d',
'qsnake.tests',
],
package_data = {
'qsnake.tests': ['phaml_data/domain.*'],
},
include_dirs=[numpy.get_include()],
ext_modules = [Extension("qsnake.cmesh", [
"qsnake/cmesh.pyx",
"qsnake/fmesh.f90",
])],
description = "Qsnake standard library",
license = "BSD",
)

The qsnake.cmesh extension needs to compile .pyx into .c and later to
.o, it needs to use gfortran to compile fmesh.f90 to fmesh.o, and then
link both things. That's it. In other words, this is what I want
distutils to do:

$ cython cmesh.pyx
$ gcc -fPIC -o cmesh.o -c cmesh.c -I$SPKG_LOCAL/include/python2.6
-I$SPKG_LOCAL/lib/python2.6/site-packages/numpy/core/include
$ gfortran -fPIC -o fmesh.o -c fmesh.f90
$ gcc -shared -o cmesh.so cmesh.o fmesh.o


If I import the relevant commands above by:

from numpy.distutils.core import setup
from numpy.distutils.extension import Extension
from Cython.Distutils import build_ext
import numpy

then it gives me:

running install
running build
running config_cc
unifing config_cc, config, build_clib, build_ext, build commands
--compiler options
running config_fc
unifing config_fc, config, build_clib, build_ext, build commands
--fcompiler options
running build_src
build_src
building extension "qsnake.cmesh" sources
f2py options: []
f2py:> build/src.linux-x86_64-2.6/qsnake/cmeshmodule.c
creating build
creating build/src.linux-x86_64-2.6
creating build/src.linux-x86_64-2.6/qsnake
Reading fortran codes...
Reading file 'qsnake/fmesh.f90' (format:free)
Traceback (most recent call last):
  File "setup.py", line 29, in 
license = "BSD",
  File 
"/home/certik1/repos/qsnake/local/lib/python2.6/site-packages/numpy/distutils/core.py",
line 186, in setup
return old_setup(**new_attr)
  File "/home/certik1/repos/qsnake/local/lib/python/distutils/core.py",
line 152, in setup
dist.run_commands()
  File "/home/certik1/repos/qsnake/local/lib/python/distutils/dist.py",
line 975, in run_commands
self.run_command(cmd)
  File "/home/certik1/repos/qsnake/local/lib/python/distutils/dist.py",
line 995, in run_command
cmd_obj.run()
  File 
"/home/certik1/repos/qsnake/local/lib/python2.6/site-packages/numpy/distutils/command/install.py",
line 55, in run
r = old_install.run(self)
  File 
"/home/certik1/repos/qsnake/local/lib/python/distutils/command/install.py",
line 577, in run
self.run_command('build')
  File "/home/certik1/repos/qsnake/local/lib/python/distutils/cmd.py",
line 333, in run_command
self.distribution.run_command(command)
  File "/home/certik1/repos/qsnake/local/lib/python/distutils/dist.py",
line 995, in run_command
cmd_obj.run()
  File 
"/home/certik1/repos/qsnake/local/lib/python2.6/site-packages/numpy/distutils/command/build.py",
line 37, in run
old_build.run(self)
  File "/home/certik1/repos/qsnake/local/lib/python/distutils/command/build.py",
line 134, in run
self.run_command(cmd_name)
  File "/home/certik1/repos/qsnake/local/lib/python/distutils/cmd.py",
line 333, in run_command
self.distribution.run_command(command)
  File "/home/certik1/repos/qsnake/local/lib/python/distutils/dist.py",
line 995, in run_command
cmd_obj.run()
  File 
"/home/certik1/repos/qsnake/local/lib/python2.6/site-packages/numpy/distutils/command/build_src.py",
line 152, in run
self.build_sources()
  File 
"/home/certik1/repos/qsnake/local/lib/python2.6/site-packages/numpy/distutils/command/build_src.py",
line 169, in build_sources
self.build_extension_sources(ext)
  File 
"/home/certik1/repos/qsnake/local/lib/python2.6/site-packages/numpy/distutils/command/build_src.py",
line 334, in build_extension_sources
sources = self.f2py_sources(sources, ext)
  File 
"/home/certik1/repos/qsnake/local/lib/python2.6/site-packages/numpy/distutils/command/build_src.py",
line 593, in f2py_sources
['-m',ext_name]+f_sources)
  File 
"/home/certik1/repos/qsnake/local/lib/python2.6/site-packages/numpy/f2py/f2py2e.py",
line 342, in run_main
postlist=callcrackfortran(files,options)
  File 
"/home/certik1/repos/qsnake/local/lib/python2.6/site-packages/numpy/f2py/f2py2e.py",
line 276, in callcrackfortran
postlist=crackfortran.crackfortran(files)
  File 
"/home/certik1/repos/qsnake/local/lib/python2.6/site-packages/numpy/f2py/crackfortran.py",
line 2683, in crackfortran
readfortrancode(files,crackline)
  File 
"/home/certik1/repos/qsnake/local/lib/python2.6/site-packages/numpy/f2py/crackfortran.py",
line 420, in readfortrancode
dowithline(finalline)
  File 
"/home/certik1/repos/qsnake/local/lib/python2.6/site-packages/numpy/f2py/crackfortran.py",
line 624, in crackline
analyzelin

[Numpy-discussion] Intel C Compiler users: please test

[Ralf Gommers]

Hi,

In commit dede2691 I've committed a change to the 32-bit Intel C
compiler (adding -fPIC) and added a 64-bit "intelem" compiler. Please
test, if there's any issue the 32-bit flag change can be reverted.

Thanks,
Ralf
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Request for a bit more info on structured arrays in the "basics" page

[Russell E. Owen]

In article 
,
 Skipper Seabold  wrote:

>> The page 
>>
>> gives a good introduction to structured arrays. However, it says 
nothing
>> about how to set a particular element (all fields at once) from a
>> collection of data.
> ...
> I added a bit at the end here, though it is mentioned briefly above.
> Feel free to expand. It's a wiki. You just need edit rights.
> 
> http://docs.scipy.org/numpy/docs/numpy.doc.structured_arrays/

Thank you very much. I had already tried to update the page

but it's auto-generated from docs and I had no idea where to find those 
docs.

-- Russell

___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Anybody going to PyCon?

[Peter Wang]

I will be there tonight through Sunday night.

I posted on Convore about a Visualization BoF, which may be of
interest to folks in the numpy/scipy crowd:
https://convore.com/pycon-2011/python-visualization-bof/

See you all there!

-Peter

On Mon, Mar 7, 2011 at 3:53 AM, Gökhan Sever  wrote:
> Hello,
>
> I am going to the PyCon this week. I am presenting a poster about an
> atmospheric sciences related project -- the most active development
> from my coding site over at http://code.google.com/p/ccnworks/
>
> Is there anybody in the community participating there as well? Any
> plans for sprinting or similar activities?
>
> See you at PyCon.
>
> --
> Gökhan
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Anybody going to PyCon?

[Gökhan Sever]

Yung-Yu,

We are advertised on this blog
http://pycon.blogspot.com/2011/03/pycon-2011-outside-talks-poster-session.html

I will be in the conference venue by tomorrow morning. There are many
interesting talks and posters that I look forward seeing plus meeting
those presenters.

See you in Atlanta.

On 3/9/11, Yung-Yu Chen  wrote:
> I will be there tomorrow, and giving a talk about solving PDEs on Saturday
> morning (plus a poster).  Looking forward to meet you guys and to learn more
> about contribution to the community.
>
> with regards,
> Yung-Yu Chen
>
> --
> Yung-Yu Chen
> PhD candidate of Mechanical Engineering
> The Ohio State University, Columbus, Ohio
> +1 (614) 859 2436
> http://solvcon.net/yyc/
>

-- 
Gökhan
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] ragged array implimentation

[Francesc Alted]

A Thursday 10 March 2011 20:29:00 Christopher Barker escrigué:
> I don't think so -- my approach is a lot simpler that I think carray
> is:
> 
> it acts much like a python list:
> 
> 1) it pre-allocates extra space when an array is created.

same for carray

> 2) when you append, it uses that extra space

ditto for carray

> 3) when the extra space is used up, it re-allocates the entire array,
> with some more extra room

again, carray works exactly the same: the extra room is just a new chunk

> And to keep it really simple, I'm using a numpy array internally, and
> np.ndarray.resize to grow it. It can only be grown on the first axis
> (C-order).

uh, very similar, except that due to the chunked nature of carray, it 
does not use `ndarray.resize`, but simply allocates a new chunk.

> > Perhaps it is a good time to join efforts and create a nice
> > 'growable' array package?
> 
> Sure, but I don't know that I have much to offer, except maybe some
> test and timing code.

Well, that sounds like it could be a start.  Your experience with your 
package would be a nice thing to have.  BTW, can you remember me where 
it is your code?  Maybe I can get some ideas by looking at it.

> By the way, it would be great to have a growable array that could be
> efficiently used in Cython code -- so you could accumulate a bunch of
> native C datatype data easily.

As you noticed, chunks and carrays are both native Cython extensions, so 
it should be pretty easy to call them from Cython directly, avoiding the 
Python overhead.

-- 
Francesc Alted
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] ragged array implimentation

[Christopher Barker]

On 3/10/11 11:29 AM, Christopher Barker wrote:
> By the way, it would be great to have a growable array that could be
> efficiently used in Cython code -- so you could accumulate a bunch of
> native C datatype data easily.

I just noticed that carray is written in Cython, so that part should be 
easy!

-Chris


-- 
Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] ragged array implimentation

[Christopher Barker]

On 3/10/11 9:51 AM, Francesc Alted wrote:
> A Thursday 10 March 2011 18:05:11 Christopher Barker escrigué:
>> NOTE: this looks like it could use a "growable" numpy array, much
>> like one I've written before -- maybe it's time to revive that
>> project and use it here, fixing some performance issues while I'm at
>> it.
>
> A growable array would be indeed very useful in many situations.  My
> carray project [1] is another attempt to create such an object.  It
> grows (or shrink) arrays by adding (removing) chunks, so they are fast
> beasts for performing these operations.  I suppose this is more or less
> like your approach.

I don't think so -- my approach is a lot simpler that I think carray is:

it acts much like a python list:

1) it pre-allocates extra space when an array is created.
2) when you append, it uses that extra space
3) when the extra space is used up, it re-allocates the entire array, 
with some more extra room

And to keep it really simple, I'm using a numpy array internally, and 
np.ndarray.resize to grow it. It can only be grown on the first axis 
(C-order).

IIUC, carray is doing a lot more smarts about keeping the array in 
chunks that can be individually manipulated, which is very cool.NOt to 
mention the compression!

> I also thought about implementing ragged arrays in carray, like those
> you are suggesting, but not sure when they will be ready for public
> consumption.

That would be very cool, and I think you are right, the facilities in 
carray could make a really nice ragged array.

> Perhaps it is a good time to join efforts and create a nice 'growable'
> array package?

Sure, but I don't know that I have much to offer, except maybe some test 
and timing code.

By the way, it would be great to have a growable array that could be 
efficiently used in Cython code -- so you could accumulate a bunch of 
native C datatype data easily.

-Chris



-- 
Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Crash with nested recarray on Python 3.x

[Christoph Gohlke]

On 3/9/2011 10:29 PM, Charles R Harris wrote:
>
>
> On Wed, Mar 9, 2011 at 11:20 PM, Christoph Gohlke  > wrote:
>
> Hello,
>
> the following code crashes in the last line when using numpy 1.5.1 on
> Python 3.1 and 3.2, 32 and 64 bit, for Windows. It works with Python
> 2.x. Can anyone confirm the crash on other platforms?
>
> import numpy
> RECORD1 = [('i', 'i4')]
> RECORD2 = [('j', RECORD1, 2)]
> a = numpy.recarray((1,), RECORD2)
> a.data
>
>
> Don't see it here.
>
> Python 3.1.2 (r312:79147, Sep  8 2010, 23:02:57)
> [GCC 4.5.1 20100812 (Red Hat 4.5.1-1)] on linux2
> Type "help", "copyright", "credits" or "license" for more information.
>> >> import numpy
>> >> RECORD1 = [('i', 'i4')]
>> >> RECORD2 = [('j', RECORD1, 2)]
>> >> a = numpy.recarray((1,), RECORD2)
>> >> a.data
> 
>> >> numpy.__version__
> '1.6.0.dev-3f0f12f'
>
> OTOH, the values look suspicious
>
>> >> a
> rec.array([(array([(-1770425816,), (62,)],
>dtype=[('i', 'dtype=[('j', [('i', '
> Hmm...
>
> Chuck
>

This is apparently fixed in numpy 1.6.0.dev.

I opened a ticket for numpy 1.5.x at 
.

Christoph

___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] ragged array implimentation

[Francesc Alted]

A Thursday 10 March 2011 18:05:11 Christopher Barker escrigué:
> NOTE: this looks like it could use a "growable" numpy array, much
> like one I've written before -- maybe it's time to revive that
> project and use it here, fixing some performance issues while I'm at
> it.

A growable array would be indeed very useful in many situations.  My 
carray project [1] is another attempt to create such an object.  It 
grows (or shrink) arrays by adding (removing) chunks, so they are fast 
beasts for performing these operations.  I suppose this is more or less 
like your approach.

I also thought about implementing ragged arrays in carray, like those 
you are suggesting, but not sure when they will be ready for public 
consumption.

Perhaps it is a good time to join efforts and create a nice 'growable' 
array package?

[1] https://github.com/FrancescAlted/carray

-- 
Francesc Alted
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] ragged array implimentation

[Christopher Barker]

On 3/7/11 5:51 PM, Sturla Molden wrote:
> Den 07.03.2011 18:28, skrev Christopher Barker:
>> 1, 2, 3, 4
>> 5, 6
>> 7, 8, 9, 10, 11, 12
>> 13, 14, 15
>> ...
>>

> A ragged array, as implemented in C++, Java or C# is just an array of
> arrays (or 'a pointer to an array of pointers').

Sure, but as a rule I don't find direct translation of C++ or Java code 
to Pyton the best approach ;-)


> Basically, that is an
> ndarray of ndarrays (or a list of ndarrays, whatever you prefer).
>
>   >>>  ra = np.zeros(4, dtype=np.ndarray)
>   >>>  ra[0] = np.array([1,2,3,4])
>   >>>  ra[1] = np.array([5,6])
>   >>>  ra[2] = np.array([7,8,9,10,11,12])
>   >>>  ra[3] = np.array([13,14,15])
>   >>>  ra
> array([[1 2 3 4], [5 6], [ 7  8  9 10 11 12], [13 14 15]], dtype=object)
>   >>>  ra[1][1]
> 6
>   >>>  ra[2][:]
> array([ 7,  8,  9, 10, 11, 12])

yup -- or I could use a list to store the rows, which would add the 
ability to append rows.

> Slicing in two dimensions does not work as some might expect:
>
>   >>>  ra[:2][:2]
> array([[1 2 3 4], [5 6]], dtype=object)

yup -- might want to overload indexing to do something smarter about 
that, though in myuse-case, slicing "vertically" isn't really useful 
anyway -- the nth element in one row doesn't neccessarily have anything 
to do with the nth element in another row.

However, asside from the slicing syntax issue, what I lose with the 
approach is the ability to get reasonable performance on operations on 
the entire array:

ra *= 3.3

I"d like that to be numpy-efficient.

What I need to grapple with is:

1) Is there a point to trying to build a general purpose ragged array? 
Or should I jsut build something that satisfies my use-case at hand?

2) What's the balance I need between performance and flexibility? 
putting the rows in a list give a lot more flexibility, putting it all 
in one 1-d numpy array could give better performance.

NOTE: this looks like it could use a "growable" numpy array, much like 
one I've written before -- maybe it's time to revive that project and 
use it here, fixing some performance issues while I'm at it.

Thanks for all your ideas,

-Chris


-- 
Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Is this the optimal way to take index along a single axis?

[Jonathan Taylor]

I see.

Should functionality like this be included in numpy?

Jon.


On Tue, Mar 8, 2011 at 3:39 PM,   wrote:
> On Tue, Mar 8, 2011 at 3:03 PM, Jonathan Taylor
>  wrote:
>> I am wanting to use an array b to index into an array x with dimension
>> bigger by 1 where the element of b indicates what value to extract
>> along a certain direction.  For example, b = x.argmin(axis=1).
>> Perhaps I want to use b to create x.min(axis=1) but also to index
>> perhaps another array of the same size.
>>
>> I had a difficult time finding a way to do this with np.take easily
>> and even with fancy indexing the resulting line is very complicated:
>>
>> In [322]: x.shape
>> Out[322]: (2, 3, 4)
>>
>> In [323]: x.min(axis=1)
>> Out[323]:
>> array([[ 2,  1,  7,  4],
>>       [ 8,  0, 15, 12]])
>>
>> In [324]: x[np.arange(x.shape[0])[:,np.newaxis,np.newaxis],
>> idx[:,np.newaxis,:], np.arange(x.shape[2])]
>> Out[324]:
>> array([[[ 2,  1,  7,  4]],
>>
>>       [[ 8,  0, 15, 12]]])
>>
>> In any case I wrote myself my own function for doing this (below) and
>> am wondering if this is the best way to do this or if there is
>> something else in numpy that I should be using? -- I figure that this
>> is a relatively common usecase.
>>
>> Thanks,
>> Jon.
>>
>> def mytake(A, b, axis):
>>    assert len(A.shape) == len(b.shape)+1
>>
>>    idx = []
>>    for i in range(len(A.shape)):
>>        if i == axis:
>>            temp = b.copy()
>>            shapey = list(temp.shape)
>>            shapey.insert(i,1)
>>        else:
>>            temp = np.arange(A.shape[i])
>>            shapey = [1]*len(b.shape)
>>            shapey.insert(i,A.shape[i])
>>        shapey = tuple(shapey)
>>        temp = temp.reshape(shapey)
>>        idx += [temp]
>>
>>    return A[tuple(idx)].squeeze()
>>
>>
>> In [319]: util.mytake(x,x.argmin(axis=1), 1)
>> Out[319]:
>> array([[ 2,  1,  7,  4],
>>       [ 8,  0, 15, 12]])
>>
>> In [320]: x.min(axis=1)
>> Out[320]:
>> array([[ 2,  1,  7,  4],
>>       [ 8,  0, 15, 12]])
>
> fewer lines but essentially the same thing and no shortcuts, I think
>
 x= np.random.randint(5, size=(2, 3, 4))
 x
> array([[[3, 1, 0, 1],
>        [4, 2, 2, 1],
>        [2, 3, 2, 2]],
>
>       [[2, 1, 1, 1],
>        [0, 2, 0, 3],
>        [2, 3, 3, 1]]])
>
 idx = [np.arange(i) for i in x.shape]
 idx = list(np.ix_(*idx))
 idx[axis]=np.expand_dims(x.argmin(axis),axis)
 x[idx]
> array([[[2, 1, 0, 1]],
>
>       [[0, 1, 0, 1]]])
>
 np.squeeze(x[idx])
> array([[2, 1, 0, 1],
>       [0, 1, 0, 1]])
>
 mytake(x,x.argmin(axis=1), 1)
> array([[2, 1, 0, 1],
>       [0, 1, 0, 1]])
>
> Josef
>
>> ___
>> NumPy-Discussion mailing list
>> NumPy-Discussion@scipy.org
>> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>>
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] problem with array/ndarray/list parameter to function

[Benjamin Root]

On Thu, Mar 10, 2011 at 4:04 AM, Mic J  wrote:

> Hi i have a problem with passing a parameter to a function
>
>
> xtickvals = np.arange(0, total_project_days+5,  5)
>
> its for setting ticks on an axis, the x-axis
>
> Ex1: This works
> xticks = pylab.setp(ax2,
> xticklabels=['0','10','20','30','40','50','60','70','80','90','100'])
>
> Ex2: This doesnt work
> xticklabels=['0', '5', '10', '15', '20', '25', '30', '35', '40', '45',
> '50',
> '55', '60', '65']
> xticks = pylab.setp(ax2, xticklabels)
>
> I want to calculate how many tick is needed on that axis and then pass it
> to
> function.
> So that why i need something like the second example (or another way?)
>

Mic,

The problem is that there is a difference between "positional" arguments and
"keyword" arguments.  In the first example, the second argument is passed as
a keyword argument (due to the equal sign).  This allows python to know
which argument (by name) you want that value to be associated with.
However, in the second example, the second argument is given as a positional
argument (due to the lack of an equal sign).  Python knows which argument
(by position) you want that value associated with.

So, to make your second example work, just do this:

xticklabels=['0', '5', '10', '15', '20', '25', '30', '35', '40', '45', '50',
'55', '60', '65']
xticks = pylab.setp(ax2, xticklabels=xticklabels)

That's it!

I hope this helps!
Ben Root
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] C API freeze and review

[Charles R Harris]

On Thu, Mar 10, 2011 at 12:25 AM, Charles R Harris <
charlesr.har...@gmail.com> wrote:

>
>
> On Thu, Mar 10, 2011 at 12:15 AM, Ralf Gommers <
> ralf.gomm...@googlemail.com> wrote:
>
>> Hi all,
>>
>> In preparation for making a 1.6.x branch, I just updated the C API
>> version. Please do not add any more functions before the branch is
>> created.
>>
>> Over 70 new functions and types were added, we should also review if
>> all those are necessary and if they are documented. Below is a list of
>> all new objects (generated with attached script), and an indication of
>> those items being documented.
>>
>> Most function are from the new iterator, are they all necessary?
>>
>> And can the authors of the items with missing docs please write some?
>>
>> Cheers,
>> Ralf
>>
>>
>> NpyIter_Type
>> PyDatetimeArrType_Type
>> PyHalfArrType_Type
>> PyTimeIntegerArrType_Type
>> PyTimedeltaArrType_Type
>>
>> NpyIter_CopyYES
>> NpyIter_CreateCompatibleStrides
>> NpyIter_Deallocate  YES
>> NpyIter_DebugPrint
>> NpyIter_GetAxisStrideArray  YES
>> NpyIter_GetBufferSize   YES
>> NpyIter_GetDataPtrArray YES
>> NpyIter_GetDescrArray   YES
>> NpyIter_GetGetCoordsYES
>> NpyIter_GetIndexPtr YES
>> NpyIter_GetInitialDataPtrArray
>> NpyIter_GetInnerFixedStrideArrayYES
>> NpyIter_GetInnerLoopSizePtr YES
>> NpyIter_GetInnerStrideArray YES
>> NpyIter_GetIterIndexYES
>> NpyIter_GetIterIndexRange   YES
>> NpyIter_GetIterNext YES
>> NpyIter_GetIterSize YES
>> NpyIter_GetIterView YES
>> NpyIter_GetNDim YES
>> NpyIter_GetNIterYES
>> NpyIter_GetOperandArray YES
>> NpyIter_GetReadFlagsYES
>> NpyIter_GetShapeYES
>> NpyIter_GetWriteFlags   YES
>> NpyIter_GotoCoords  YES
>>
>
> I'd like to rename this something like  NpyIter_GotoMultiIndex,
> NpyIter_GotoPolyIndex, or some such. We need a name for extended indices.
> Suggestions?
>
>
Maybe the following terms?

Index -- single index
FlatIndex -- single index into raveled array
FullIndex -- [1,2,3]
FancyIndex -- fancy index



Chuck
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

[Numpy-discussion] problem with array/ndarray/list parameter to function

[Mic J]

Hi i have a problem with passing a parameter to a function


xtickvals = np.arange(0, total_project_days+5,  5)

its for setting ticks on an axis, the x-axis

Ex1: This works
xticks = pylab.setp(ax2,
xticklabels=['0','10','20','30','40','50','60','70','80','90','100'])

Ex2: This doesnt work
xticklabels=['0', '5', '10', '15', '20', '25', '30', '35', '40', '45', '50',
'55', '60', '65']
xticks = pylab.setp(ax2, xticklabels)

I want to calculate how many tick is needed on that axis and then pass it to
function.
So that why i need something like the second example (or another way?)
--
I couldnt se code tags?

Code to calculate ticks and steps for output
--
#xticksval = np.arange(0, total_project_days+stpsz=5,  stepsz)
xtickvals = np.arange(0, total_project_days+5,  5)


# why do we need to define it here, instead of it being created in the for
loop
xticklabels = []

for i in xtickvals:
 xticklabels.append(str(i))

xticks = pylab.setp(ax2, xticklabels)
-

The above code gives the same problem as example2

regards mic
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Request for a bit more info on structured arrays in the "basics" page

[Pauli Virtanen]

Thu, 10 Mar 2011 15:23:59 +0800, Ralf Gommers wrote:
[clip]
>> x = np.array([1.5])
>> x.view(int)
>> array([4609434218613702656])

Yes, `view` is meant to do exactly that. Use `astype` if you want a type 
cast.

Pauli


___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion