Re: [Numpy-discussion] Need more comments from scientific community on python-dev

[Sasha]

On 11/3/06, Torgil Svensson <[EMAIL PROTECTED]> wrote:
> class struct_type(Structure):
> _fields_ = []
>
> ...
> ... which is somewhat static in nature. How do you create "structures"
> dynamically?
>

You can put the above in a function that takes fields as an argument, or

type('struct_type', (Structure,), {'_fields_':[('a', c_int), ('b',
c_long)]})

-
Using Tomcat but need to do more? Need to support web services, security?
Get stuff done quickly with pre-integrated technology to make your job easier
Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
___
Numpy-discussion mailing list
Numpy-discussion@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/numpy-discussion

Re: [Numpy-discussion] Need more comments from scientific community on python-dev

[Alexander Belopolsky]

On 11/3/06, Torgil Svensson <[EMAIL PROTECTED]> wrote:
> class struct_type(Structure):
> _fields_ = []
>
> ...
> ... which is somewhat static in nature. How do you create "structures"
> dynamically?
>

You can put the above in a function that takes fields as an argument, or

type('struct_type', (Structure,), {'_fields_':[('a', c_int), ('b',
c_long)]})

-
Using Tomcat but need to do more? Need to support web services, security?
Get stuff done quickly with pre-integrated technology to make your job easier
Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
___
Numpy-discussion mailing list
Numpy-discussion@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/numpy-discussion

Re: [Numpy-discussion] Need more comments from scientific community on python-dev

[Torgil Svensson]

> > It seems to me that at the python level, there's not much reason to
> > choose the dtypes proposal over ctypes.

I disagree. I can see a point in unifying ctypes and dtypes but in my
mindset they've got two different scopes.

ctypes is an interface to the c language
dtype is a data description for the python language

The biggest advantages python has over c in my mind (except the script
vs compile) is that it's dynamic and expressive. I think basing dtype
on ctypes is in that mind is simply destructive and against what I
like best with python.

ctypes mimics c's static behaviour:

class struct_type(Structure):
_fields_ = []

...

... which is somewhat static in nature. How do you create "structures"
dynamically?

How do you do things like this in ctypes:

mytype=filter(lambda x: is_little_endian(x),map(lambda x: x==int32 and
int64 or x, old_type))
b=['gg',[1,2,3]]
mytype=dtype([{str: uint64 , list:another_type }[type(x)] for x in b])

I'm sure it's possibl but I'm also suspecting it not at all as clean
and expressive (my second argument for choosing python over c).

Since we creating something new in python isn't it best we do
something the python way and later see if we can adapt the c structure
to that rather than the other way around?  If we can't adapt ctypes to
this new thing i'm perfectly happy with leaving ctypes as is.

ctype is doing a great job on the c interaction level but is plain
ugly on the python level.


On 11/1/06, Bill Baxter <[EMAIL PROTECTED]> wrote:
>
> On 11/2/06, A. M. Archibald <[EMAIL PROTECTED]> wrote:
> > On 01/11/06, Travis Oliphant <[EMAIL PROTECTED]> wrote:
> >
> > > And it may be a good idea to also have a get_ctype method or some-such
> > > on the ctypes attribute so that  one could get a "ctypes description"
> > > from the NumPy data-type.
> >
> > It seems to me that at the python level, there's not much reason to
> > choose the dtypes proposal over ctypes. There is one at the C level,
> > it seems (though I, like perhaps most of the people on python-dev,
> > have never actually tried using either). So perhaps, to persuade the
> > python-dev folks, what is needed is a comparison of what has to be
> > done at the C level. What would it take to rewrite numpy to use
> > ctypes? There seems to be some problem with extending the type objects
> > used by ctypes, but it's not very clear to me what that problem is
> > (what the extensions are supposed to do).
>
>
> I posted a message to the thread trying to prod things in that direction.
> I.e. can we see a simple concrete example of the complications involved in
> using ctypes interface code, vs the presumably much nicer
> numpy/data-descriptor code.  I too think that would help convince people.
> Just saying it's more compilcated, trust me, doesn't help when most people
> reading the list have never had to write any sort of C extension.
>
> --bb
>
>
> -
> Using Tomcat but need to do more? Need to support web services, security?
> Get stuff done quickly with pre-integrated technology to make your job
> easier
> Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
> http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
>
> ___
> Numpy-discussion mailing list
> Numpy-discussion@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/numpy-discussion
>
>
>

-
Using Tomcat but need to do more? Need to support web services, security?
Get stuff done quickly with pre-integrated technology to make your job easier
Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
___
Numpy-discussion mailing list
Numpy-discussion@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/numpy-discussion

Re: [Numpy-discussion] Need more comments from scientific community on python-dev

[Bill Baxter]

On 11/2/06, A. M. Archibald <[EMAIL PROTECTED]> wrote:
On 01/11/06, Travis Oliphant <[EMAIL PROTECTED]> wrote:> And it may be a good idea to also have a get_ctype method or some-such> on the ctypes attribute so that  one could get a "ctypes description"
> from the NumPy data-type.It seems to me that at the python level, there's not much reason tochoose the dtypes proposal over ctypes. There is one at the C level,it seems (though I, like perhaps most of the people on python-dev,
have never actually tried using either). So perhaps, to persuade thepython-dev folks, what is needed is a comparison of what has to bedone at the C level. What would it take to rewrite numpy to usectypes? There seems to be some problem with extending the type objects
used by ctypes, but it's not very clear to me what that problem is(what the extensions are supposed to do).I posted a message to the thread trying to prod things in that direction.  I.e. can we see a simple concrete example of the complications involved in using ctypes interface code, vs the presumably much nicer numpy/data-descriptor code.  I too think that would help convince people.  Just saying it's more compilcated, trust me, doesn't help when most people reading the list have never had to write any sort of C extension.
--bb
-
Using Tomcat but need to do more? Need to support web services, security?
Get stuff done quickly with pre-integrated technology to make your job easier
Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642___
Numpy-discussion mailing list
Numpy-discussion@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/numpy-discussion

Re: [Numpy-discussion] Need more comments from scientific community on python-dev

[A. M. Archibald]

On 01/11/06, Travis Oliphant <[EMAIL PROTECTED]> wrote:

> And it may be a good idea to also have a get_ctype method or some-such
> on the ctypes attribute so that  one could get a "ctypes description"
> from the NumPy data-type.

It seems to me that at the python level, there's not much reason to
choose the dtypes proposal over ctypes. There is one at the C level,
it seems (though I, like perhaps most of the people on python-dev,
have never actually tried using either). So perhaps, to persuade the
python-dev folks, what is needed is a comparison of what has to be
done at the C level. What would it take to rewrite numpy to use
ctypes? There seems to be some problem with extending the type objects
used by ctypes, but it's not very clear to me what that problem is
(what the extensions are supposed to do).

The argument that *some* datatypes format should become standard is much easier.

A. M. Archibald

-
Using Tomcat but need to do more? Need to support web services, security?
Get stuff done quickly with pre-integrated technology to make your job easier
Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
___
Numpy-discussion mailing list
Numpy-discussion@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/numpy-discussion

Re: [Numpy-discussion] Need more comments from scientific community on python-dev

[Travis Oliphant]

Alexander Belopolsky wrote:
> On 10/31/06, Travis Oliphant <[EMAIL PROTECTED]> wrote:
> ...
>   
>> Please understand what I meant.  I meant putting effort into getting
>> PyArray_DescrConverter to allow ctypes inputs and convert to the
>> appropriate PyArray_Descr * structure.   I already understand ctypes
>> objects.  I want the dtype() command to also understand them.
>>
>> 
>
> I think I am starting to understand. Forgive me for being slow.
>   

And I think I'm beginning to understand why some people are resistant to 
what I'm saying.   I have no problem converting from ctypes types to the 
underlying PyArray_Descr structure.  I don't necessarily want that to be 
the only way to describe new types but I sure don't mind people using 
that approach.

I suspect it's the concern about "how to describe data-formats on the 
Python level" that have people concerned.  

> Is it correct that you don't mind writing c_int * 10 instead of
> dtype((' PyTypeObject at the C level?
That is correct. 

>  If this is the case, I would suggest to
> merge ctypes syntax with your implementation.  
> This may help to make
> the case for the python-dev crowd.
Thanks for this suggestion.  You could very probably be right.
>   I believe very few people
> understand the subtle problems in inheriting from PyTypeObject and
> resist your proposal simply because they like c_int * 10  better than
> dtype(('
> There is no reason why ctypes should be implemented the way it is. It
> is not necessary that type(c_int()) is c_int.  If a type with almost
> the same properties gets into the core, ctypes developers may see an
> advantage of simply inheriting from it and adding a factory __call__
> method.  Meanwhile, users familiar with ctypes will not need to learn
> yet another type specification syntax.
>
>   
It would actually be "easier" for NumPy to understand ctypes syntax if 
something like a data-type object were.  As it is, the converter to a 
PyArray_Descr structure will probably have to check the tp_name of the 
ctypes.  Maybe this will help with the PEP to describe the difficulty of 
using ctypes constructs inside of other programs.

And it may be a good idea to also have a get_ctype method or some-such 
on the ctypes attribute so that  one could get a "ctypes description" 
from the NumPy data-type.

-Travis


-
Using Tomcat but need to do more? Need to support web services, security?
Get stuff done quickly with pre-integrated technology to make your job easier
Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
___
Numpy-discussion mailing list
Numpy-discussion@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/numpy-discussion

Re: [Numpy-discussion] Need more comments from scientific community on python-dev

[Lisandro Dalcin]

On 10/31/06, Travis Oliphant <[EMAIL PROTECTED]> wrote:
>
> I'm recruiting more comments on python-dev regarding my two proposals
> for improving Python's native ability to share ndarray-like information.
>

I believe there are another really important reason to support
Travis's proposal.

Python should have a general, simple, powerful and standard way to
associate type metadata to binary streams. This enables efficient
share or memory, but also can help a lot to achieve inter-language
capabilities.

As an example, I would to make some comments about MPI, the "de facto"
standard for message-passing in parallel computing. When MPI
specification was being developed, there were many different,
incompatible libraries for message passing, and many of them (ie. PVM)
usually communicated binary data following a pack/unpack approach. The
MPI Forum decided to add an alternative approach, adding a class
'MPI_Datatype', and many predefined instances of this class (MPI_INT,
MPI_FLOAT, etc.) in order to represent the basic datatypes (integers,
floats, strings, complex numbers) in C, Fortran, and C++. Predefined
MPI datatypes can be used as building blocks for describing very
complicated memory layouts and derived,nested structures. This way,
any implementations of this standard can support interprocess
communication in a multi-language, muti-architecture approach. MPI
datatypes can also be "decoded" in order to get the basic, predefined
types contained in a user-defined datatype.

I think the MPI example, despite being taken from a very specific
domain with particular needs, has a strong connection with Travis's
proposal. Pack/unpack degrades performance, but if you want to
efficiently share binary data (with other languages/architecture),
then a general and extensible mechanism for attaching type metadata to
binary streams must me defined.

I would like to know your opinions about this. Travis's datatypes in
Python should be more that something to share data, they should be
able to **describe** data. And I am not sure if ctypes can be be the
way. Can Jython take advantage of ctypes??? Or in the near future
CPython/Jython will have a 'jtypes' module? I forget! Travis/Pearu,
why don't you develop a 'ftypes' for Fortran binary data, and next ask
Python core developers to include it in standard library (a joke, of
course)?

Travis, if you think my previous example about MPI can help, I can
resend this post to Python-Dev. I ask you this because I really want
to help and not to make noise.

Regars,


-- 
Lisandro Dalcín
---
Centro Internacional de Métodos Computacionales en Ingeniería (CIMEC)
Instituto de Desarrollo Tecnológico para la Industria Química (INTEC)
Consejo Nacional de Investigaciones Científicas y Técnicas (CONICET)
PTLC - Güemes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

-
Using Tomcat but need to do more? Need to support web services, security?
Get stuff done quickly with pre-integrated technology to make your job easier
Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
___
Numpy-discussion mailing list
Numpy-discussion@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/numpy-discussion

Re: [Numpy-discussion] Need more comments from scientific community on python-dev

[Sasha]

On 10/31/06, Travis Oliphant <[EMAIL PROTECTED]> wrote:
> No, it's not that simple.  We have a headache whenever we want to do
> something like I just did and separate out the concepts of what makes a
> Python Object a Python object.  Now, we don't just modify a simple
> C-structure (PyArray_Descr *), we have to modify a "meta-type" or a
> altered dictionary and get that change put in to ctypes.

I think I am starting to understand. Forgive me for being slow.

Is it correct that you don't mind writing c_int * 10 instead of
dtype(('http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
___
Numpy-discussion mailing list
Numpy-discussion@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/numpy-discussion

Re: [Numpy-discussion] Need more comments from scientific community on python-dev

[Bill Baxter]

Grr, I wrote something up (it was wonderfully eloquent and convincing --  trust me!), but the gmane interface seems to have lost it (I posted two things before responding to the two "authorization requested" emails, and only one of the two things ever got posted it seems.)
--bbOn 11/2/06, Lisandro Dalcin <[EMAIL PROTECTED]> wrote:
On 10/31/06, Bill Baxter <[EMAIL PROTECTED]> wrote:> One thing I see with PIL and Matplotlib and PyOpenGL is that they're having> to add specific support for Numpy and numeric and numarray and PIL and
> ctypes etc.This is a real problem, and a waste of developing time, because welack a **general** and **standard** mechanism.> would the proposed extensions make it> possible for all these different packages to just have to support "buffer
> protocol" and magically be able to work with numpy or numeric without having> to introduce explicit dependencies and support code for all those other> packages?I think this is the actual reason of Travis's proposal, and of all
other people like me supporting him.> If I've got the facts straight then I'd be happy to go and say> that on comp.lang.python.>Please, go on!
-
Using Tomcat but need to do more? Need to support web services, security?
Get stuff done quickly with pre-integrated technology to make your job easier
Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642___
Numpy-discussion mailing list
Numpy-discussion@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/numpy-discussion

Re: [Numpy-discussion] Need more comments from scientific community on python-dev

[Lisandro Dalcin]

On 10/31/06, Bill Baxter <[EMAIL PROTECTED]> wrote:
> One thing I see with PIL and Matplotlib and PyOpenGL is that they're having
> to add specific support for Numpy and numeric and numarray and PIL and
> ctypes etc.

This is a real problem, and a waste of developing time, because we
lack a **general** and **standard** mechanism.

> would the proposed extensions make it
> possible for all these different packages to just have to support "buffer
> protocol" and magically be able to work with numpy or numeric without having
> to introduce explicit dependencies and support code for all those other
> packages?

I think this is the actual reason of Travis's proposal, and of all
other people like me supporting him.

> If I've got the facts straight then I'd be happy to go and say
> that on comp.lang.python.
>

Please, go on!



-- 
Lisandro Dalcín
---
Centro Internacional de Métodos Computacionales en Ingeniería (CIMEC)
Instituto de Desarrollo Tecnológico para la Industria Química (INTEC)
Consejo Nacional de Investigaciones Científicas y Técnicas (CONICET)
PTLC - Güemes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

-
Using Tomcat but need to do more? Need to support web services, security?
Get stuff done quickly with pre-integrated technology to make your job easier
Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
___
Numpy-discussion mailing list
Numpy-discussion@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/numpy-discussion

Re: [Numpy-discussion] Need more comments from scientific community on python-dev

[Bill Baxter]

One thing I see with PIL and Matplotlib and PyOpenGL is that they're having to add specific support for Numpy and numeric and numarray and PIL and ctypes etc.I'm also in the camp that doesn't really understand this stuff well enough to really argue in favor of it, but would the proposed extensions make it possible for all these different packages to just have to support "buffer protocol" and magically be able to work with numpy or numeric without having to introduce explicit dependencies and support code for all those other packages?  If I've got the facts straight then I'd be happy to go and say that on 
comp.lang.python.--bbOn 10/31/06, Travis Oliphant <[EMAIL PROTECTED]> wrote:
I'm recruiting more comments on python-dev regarding my two proposalsfor improving Python's native ability to share ndarray-like information.
There is a dearth of scientific-computing and number-crunching-awarepeople on python-dev.  The result is that I sound like a lone voicearguing for something that nobody cares about.  When, I don't think that
is true.  Please, please.  If you want Python to grow support for thearray interface (or something like it), then please speak up onpython-dev.Even something as simple as I really see the need for a way to exchange
data-format information between two objects sharing the buffer protocolcan be helpful.You can post through the gmane newsgroup interface:gmane.comp.python.develFind any of the posts on the PEP's I've introduced.  Thanks for your help.
-Travis-Using Tomcat but need to do more? Need to support web services, security?Get stuff done quickly with pre-integrated technology to make your job easier
Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimohttp://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
___Numpy-discussion mailing listNumpy-discussion@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/numpy-discussion
-
Using Tomcat but need to do more? Need to support web services, security?
Get stuff done quickly with pre-integrated technology to make your job easier
Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642___
Numpy-discussion mailing list
Numpy-discussion@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/numpy-discussion

Re: [Numpy-discussion] Need more comments from scientific community on python-dev

[Alexander Belopolsky]

On 10/31/06, Travis Oliphant <[EMAIL PROTECTED]> wrote:
...
> Please understand what I meant.  I meant putting effort into getting
> PyArray_DescrConverter to allow ctypes inputs and convert to the
> appropriate PyArray_Descr * structure.   I already understand ctypes
> objects.  I want the dtype() command to also understand them.
>

I think I am starting to understand. Forgive me for being slow.

Is it correct that you don't mind writing c_int * 10 instead of
dtype(('http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
___
Numpy-discussion mailing list
Numpy-discussion@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/numpy-discussion

Re: [Numpy-discussion] Need more comments from scientific community on python-dev

[Josh Marshall]

On 11/31/06, Fernando Perez <[EMAIL PROTECTED]> wrote:

> Fernando Perez wrote:
> ps - one more thing.  This guy:
>
> http://blog.vrplumber.com/
>
> has been rewriting the OpenGL bindings using ctypes, and I've seen
> posts from him about numpy (in his blog).  He might be able to
> contribute something...

I've been working on the OSX port and extensions for OpenGL-ctypes.  
(now released as PyOpenGL-3.00a, please test!). What Mike has done is  
define setuptools plugins to interface to different kinds of array  
data. The default type is ctypes arrays, or numpy arrays if numpy is  
installed. The data types handled are ctypes sized arrays, ctypes  
pointers, strings (read-only), and Python lists. (and old Numeric/ 
numarray via a non-default build.)

Summaries of development and usage, resp, are found:
http://pyopengl.sourceforge.net/ctypes/development.html
http://pyopengl.sourceforge.net/ctypes/using.html

Notes on array handling from the above:

> Perhaps the most complex mechanisms in OpenGL-ctypes are those  
> which implement the array-based operations which allow for using  
> low-level blocks of formatted data to communicate with the OpenGL  
> implementation.  OpenGL-ctypes preferred basic array implementation  
> is the (new) numpy reimplementation of the original Numeric Python.
>
> The array handling functionality provided within OpenGL-ctypes is  
> localised to the OpenGL.arrays sub-package.  Within the package,  
> there are two major classes, one (the FormatHandler) which  
> implements an interface to a way of storing data in Python, and  
> another (the ArrayDatatype) which models an OpenGL array format.   
> The ArrayDatatype classes use FormatHandlers to manipulate array- 
> compatible objects for use in the system.
and on FormatHandlers, for each of the datatypes I mentioned earlier:

> Each format handler is responsible for implementing an API that  
> ArrayDatatypes can use to work with the Python data-format.  Data- 
> formats can support a subset of the API, they only need to support  
> those aspects of the data-format which make sense.
Now, I haven't spent much time looking at these parts of OpenGL- 
ctypes, as they have just worked for me. I would think that it would  
be trivial to write a FormatHandler which uses the ndarray interface  
and data type description to use any object implenting it as an input  
for OpenGL. This would include things such as PIL images.

Mike, can you give us your opinion on how a standardised data type  
descriptor would be helpful for PyOpenGL? The PEP and some  
information about it can be found here:
http://www.scipy.org/ArrayInterfacePEP

Cheers,
Josh



-
Using Tomcat but need to do more? Need to support web services, security?
Get stuff done quickly with pre-integrated technology to make your job easier
Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
___
Numpy-discussion mailing list
Numpy-discussion@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/numpy-discussion

Re: [Numpy-discussion] Need more comments from scientific community on python-dev

[Travis Oliphant]

Sasha wrote:

>On 10/31/06, Travis Oliphant <[EMAIL PROTECTED]> wrote:
>
>  
>
>>Please read my posts about the Python type-object verses normal Python
>>Object situation.  That really is the crux of the matter.
>>
>>
>>
>I read the whole python-dev thread before replying.  I may be a little
>biased because I never liked somewhat cryptic letter codes in Numeric
>and the change of codes from Numeric to numpy did not contribute to my
>sympathy particularly when the convert script changed all unrelated
>instances of 'b' in my code to something else.  
>

The letter codes are not data-type objects.  The letter codes are only 
there for historical reasons (and they are in the struct and array 
modules too so blame Python...)

>I am also not a big
>fan of record arrays.  I believe that numeric data should be stored in
>"inverted tables," where columns of homogeneous data are stored
>contiguously. 
>
Sure.  I understand this.  This was my argument too when Numarray was 
first proposed.  But, how do you memory-map a record where the data is 
actually stored differently?  Numeric's answer is "you don't" but that 
is un-acceptable.  This is why record-arrays were created and the whole 
pandora's box of data-types was opened.

>  
>
>>Ctypes uses a Python type object for every data-format.
>>NumPy uses an instance of a data-type object for every data-format.
>>
>>
>>
>Yes, but AFAIK this is a recent innovation.  Numarray used type
>objects and Numeric simply used letter codes.
>
>  
>
Recent if 1 year is recent, and only if you worry precisely about when 
they were Python objects. Numeric always used a PyArray_Descr * 
structure (which was a PyObject_HEAD away from being a Python object) to 
describe data.   The letter codes were just simple ways to "represent" 
those underlying structures (which could have been exposed to Python as 
"data-types" from the very beginning.

>>What advantage do we gain by making every instance of a data-type object
>>*also* a Python type object?
>>
>>
>
>I think the main advantage is that you can have instances:
>  
>
c_int(42)


>c_int(42)
>
>Of course, numpy has scalars for that, but ctypes also has
>fixed-length arrays, that are somewhat different from ndarrays:
>  
>

Sure, but you don't "need" an instance for every memory-layout 
description.   If you want one, then great, ctypes gives it to you.  But 
"requiring" data-type to be encoded in a type object is over-kill.


>> We get a lot of head-ache.  Have you seen
>>what ctypes had to do?  It had to define a new Dictionary object so it
>>could attach it to the tp_dict parameter because you can't just inherit
>>from the PyTypeObject and add the fields you want to the structure.
>>This is my argument.
>>
>>
>>
>
>But, isn't this someone else's head-ache?  Someone has already gone
>through all these contortions, why not reuse the effort?  
>

No, it's not that simple.  We have a headache whenever we want to do 
something like I just did and separate out the concepts of what makes a 
Python Object a Python object.  Now, we don't just modify a simple 
C-structure (PyArray_Descr *), we have to modify a "meta-type" or a 
altered dictionary and get that change put in to ctypes.

Inheriting from Python type objects is harder.   People who have 
apparently never tried seem to think it's not but it is. 

>Others on
>python-dev described some specific needs of ctypes that your datatype
>object does not address.  Your point seems to be that numpy does not
>share these needs and could use a much simpler approach.
>  
>
No, that's not the point.  The data-type object could easily be extended 
to accomodate those needs.  The point is using data-types as instances 
of a regular Python object or data-types as instances of a "type" object.

>>I for one am not going to put any effort in that direction.  People are
>>free to do it, if they want, of course.  But, it's no small change.
>>
>>
>
>Can you present some use cases that illustrate the advantage of your
>datatype approach best?
>  
>
>I will try to implement them with ctypes.
>  
>
This is missing the point.  I have no doubt ctypes "could" be used on 
the Python side to describe a data-type.  But, we need the PyArray_Descr 
* structure in NumPy.  What are you going to replace the PyArray_Descr * 
structure with?  That is the point. 

>> I would, however, put effort into "undertstanding ctypes objects" as
>>data-type objects.
>>
>>
>
>Yes, this is the key.  I think we should better understand ctypes
>limitations before proposing an alternative.  
>
Please understand what I meant.  I meant putting effort into getting 
PyArray_DescrConverter to allow ctypes inputs and convert to the 
appropriate PyArray_Descr * structure.   I already understand ctypes 
objects.  I want the dtype() command to also understand them.

-Travis




-
Using Tomcat but need to do more? Need to support web ser

Re: [Numpy-discussion] Need more comments from scientific community on python-dev

[Travis Oliphant]

Sasha wrote:

>On 10/31/06, Travis Oliphant <[EMAIL PROTECTED]> wrote:
>  
>
>Yes, this is the key.  I think we should better understand ctypes
>limitations before proposing an alternative. 
>

I already understand it's practical limitations --- type objects as 
data-type instances is too bulky and too restrictive.  You have to "be a 
Python type object" in order to be a data-type object. 

I wish others would actually understand the difference.

-Travis


-
Using Tomcat but need to do more? Need to support web services, security?
Get stuff done quickly with pre-integrated technology to make your job easier
Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
___
Numpy-discussion mailing list
Numpy-discussion@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/numpy-discussion

Re: [Numpy-discussion] Need more comments from scientific community on python-dev

[Sasha]

On 10/31/06, Travis Oliphant <[EMAIL PROTECTED]> wrote:

> Please read my posts about the Python type-object verses normal Python
> Object situation.  That really is the crux of the matter.
>
I read the whole python-dev thread before replying.  I may be a little
biased because I never liked somewhat cryptic letter codes in Numeric
and the change of codes from Numeric to numpy did not contribute to my
sympathy particularly when the convert script changed all unrelated
instances of 'b' in my code to something else.  I am also not a big
fan of record arrays.  I believe that numeric data should be stored in
"inverted tables," where columns of homogeneous data are stored
contiguously.  With this disclaimer, I will address a few issues
below.

> Ctypes uses a Python type object for every data-format.
> NumPy uses an instance of a data-type object for every data-format.
>
Yes, but AFAIK this is a recent innovation.  Numarray used type
objects and Numeric simply used letter codes.


> What advantage do we gain by making every instance of a data-type object
> *also* a Python type object?

I think the main advantage is that you can have instances:
>>> c_int(42)
c_int(42)

Of course, numpy has scalars for that, but ctypes also has
fixed-length arrays, that are somewhat different from ndarrays:
>>> a10 = c_int*10
>>> a10()
<__main__.c_int_Array_10 object at 0x2a95816958>


>  We get a lot of head-ache.  Have you seen
> what ctypes had to do?  It had to define a new Dictionary object so it
> could attach it to the tp_dict parameter because you can't just inherit
> from the PyTypeObject and add the fields you want to the structure.
> This is my argument.
>

But, isn't this someone else's head-ache?  Someone has already gone
through all these contortions, why not reuse the effort?  Others on
python-dev described some specific needs of ctypes that your datatype
object does not address.  Your point seems to be that numpy does not
share these needs and could use a much simpler approach.

> I for one am not going to put any effort in that direction.  People are
> free to do it, if they want, of course.  But, it's no small change.

Can you present some use cases that illustrate the advantage of your
datatype approach best?
I will try to implement them with ctypes.

>  I would, however, put effort into "undertstanding ctypes objects" as
> data-type objects.

Yes, this is the key.  I think we should better understand ctypes
limitations before proposing an alternative.  At the end of the day,
it is better to have buffer protocol that describes the data using
ctypes types than to have no standard type information at all.

-
Using Tomcat but need to do more? Need to support web services, security?
Get stuff done quickly with pre-integrated technology to make your job easier
Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
___
Numpy-discussion mailing list
Numpy-discussion@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/numpy-discussion

Re: [Numpy-discussion] Need more comments from scientific community on python-dev

[Travis Oliphant]

Sasha wrote:

>On 10/31/06, Travis Oliphant <[EMAIL PROTECTED]> wrote:
>  
>
>>I'm recruiting more comments on python-dev regarding my two proposals
>>for improving Python's native ability to share ndarray-like information.
>>
>>
>>
>
>I would love to help, but I feel that I will be on the other side of
>the disagreement.  (That's why I reply here rather than on python-dev
>first.)
>  
>

There's actually two issues here as well.

1) The extended buffer protocol

2) How to exchange data-format information through it.



-Travis


-
Using Tomcat but need to do more? Need to support web services, security?
Get stuff done quickly with pre-integrated technology to make your job easier
Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
___
Numpy-discussion mailing list
Numpy-discussion@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/numpy-discussion

Re: [Numpy-discussion] Need more comments from scientific community on python-dev

[Travis Oliphant]

Sasha wrote:

>On 10/31/06, Travis Oliphant <[EMAIL PROTECTED]> wrote:
>  
>
>>I'm recruiting more comments on python-dev regarding my two proposals
>>for improving Python's native ability to share ndarray-like information.
>>
>>
>>
>
>I would love to help, but I feel that I will be on the other side of
>the disagreement.  (That's why I reply here rather than on python-dev
>first.)
>  
>

Please read my posts about the Python type-object verses normal Python 
Object situation.  That really is the crux of the matter.

Ctypes uses a Python type object for every data-format.
NumPy uses an instance of a data-type object for every data-format.

What advantage do we gain by making every instance of a data-type object 
*also* a Python type object?  We get a lot of head-ache.  Have you seen 
what ctypes had to do?  It had to define a new Dictionary object so it 
could attach it to the tp_dict parameter because you can't just inherit 
from the PyTypeObject and add the fields you want to the structure.
This is my argument.

I for one am not going to put any effort in that direction.  People are 
free to do it, if they want, of course.  But, it's no small change.   I 
would, however, put effort into "undertstanding ctypes objects" as 
data-type objects.

-Travis




-
Using Tomcat but need to do more? Need to support web services, security?
Get stuff done quickly with pre-integrated technology to make your job easier
Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
___
Numpy-discussion mailing list
Numpy-discussion@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/numpy-discussion

Re: [Numpy-discussion] Need more comments from scientific community on python-dev

[Alan G Isaac]

On Tue, 31 Oct 2006, Sasha apparently wrote: 
> I think we should attempt to implement a ctypes approach 

You are probably aware that Travis has addressed this at some length on
comp.python.devel (for example
http://article.gmane.org/gmane.comp.python.devel/84661 
http://article.gmane.org/gmane.comp.python.devel/84670 )
but just in case ...

Cheers,
Alan Isaac




-
Using Tomcat but need to do more? Need to support web services, security?
Get stuff done quickly with pre-integrated technology to make your job easier
Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
___
Numpy-discussion mailing list
Numpy-discussion@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/numpy-discussion

Re: [Numpy-discussion] Need more comments from scientific community on python-dev

[Sasha]

On 10/31/06, Travis Oliphant <[EMAIL PROTECTED]> wrote:
>
> I'm recruiting more comments on python-dev regarding my two proposals
> for improving Python's native ability to share ndarray-like information.
>

I would love to help, but I feel that I will be on the other side of
the disagreement.  (That's why I reply here rather than on python-dev
first.)

I've suggested that numpy could use ctypes way to describe binary data
a long time ago:

http://sourceforge.net/mailarchive/message.php?msg_id=15395017

I agree that it would not work out of the box, but I don't see any
major obstacles to adding the missing features.

I think we should attempt to implement a ctypes approach and compare
the result to the dtype object approach.  If we discover any
unimplementable features along the way, this will be a strong argument
to abandon ctypes as a standard.

The last time I looked at the issue, the only feature that ctypes were
lacking for me was the itemsize information (endianness is probably
lacking as well, but I did not need it.)  I am not sure, however that
this information belongs to the type description object rather than
carried separately.

Note that it is trivial to add an itemsize attribute to ctypes because
each primitive ctype has a fixed itemsize and Structure can easily
compute its itemsize.  Doubling the number of ctypes to allow for
different endianness is a more troubling proposition.

In any case, I think  we should seriously discuss the relative merits
of the two approaches on this list and present our view to python-dev
when we reach some consensus.

-
Using Tomcat but need to do more? Need to support web services, security?
Get stuff done quickly with pre-integrated technology to make your job easier
Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
___
Numpy-discussion mailing list
Numpy-discussion@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/numpy-discussion

Re: [Numpy-discussion] Need more comments from scientific community on python-dev

[David L Goldsmith]

Perhaps you could please elaborate on what you regard as "the issue(s)" 
and "what's at stake"?  I'm still comparatively new (been using python 
and numpy circa 8-9 months), but so far I haven't encountered any 
problems I'd attribute to "Python's ... [in]ability to share 
ndarray-like information".  (The most vexing problems I've had have all 
been with matplotlib, its dual-mode API, and its rather clumsy and 
haphazard documentation.)  I'd like to be supportive, but not blindly-so.

DG

Travis Oliphant wrote:
> I'm recruiting more comments on python-dev regarding my two proposals 
> for improving Python's native ability to share ndarray-like information.
>
> There is a dearth of scientific-computing and number-crunching-aware 
> people on python-dev.  The result is that I sound like a lone voice 
> arguing for something that nobody cares about.  When, I don't think that 
> is true.  Please, please.  If you want Python to grow support for the 
> array interface (or something like it), then please speak up on 
> python-dev. 
>
> Even something as simple as I really see the need for a way to exchange 
> data-format information between two objects sharing the buffer protocol 
> can be helpful.
>
> You can post through the gmane newsgroup interface: 
>
> gmane.comp.python.devel
>
> Find any of the posts on the PEP's I've introduced.  Thanks for your help.
>
> -Travis
>
>
> -
> Using Tomcat but need to do more? Need to support web services, security?
> Get stuff done quickly with pre-integrated technology to make your job easier
> Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
> http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
> ___
> Numpy-discussion mailing list
> Numpy-discussion@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/numpy-discussion
>   


-- 
HMRD/ORR/NOS/NOAA 

-
Using Tomcat but need to do more? Need to support web services, security?
Get stuff done quickly with pre-integrated technology to make your job easier
Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
___
Numpy-discussion mailing list
Numpy-discussion@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/numpy-discussion

Re: [Numpy-discussion] Need more comments from scientific community on python-dev

[Alan Isaac]

On Tue, 31 Oct 2006, Travis Oliphant wrote: 
> Please, please.  If you want Python to grow support for 
> the array interface (or something like it), then please 
> speak up on python-dev. 

The easiest access to this discussion for me was
http://news.gmane.org/group/gmane.comp.python.devel/
I cannot add to this discussion, but I REALLY hope others 
will help Travis out here.  (A few have.)  He is fielding a lot 
of questions, some of which look to me to be from 
individuals who are ready to have fairly strong opinions 
without really understanding the "why" of his proposals.

The good news is, there seems to be (on my naive reading) 
some sympathy for what Travis is trying to do.  I think more 
motivating examples would prove helpful in swinging things.

Cheers,
Alan Isaac




-
Using Tomcat but need to do more? Need to support web services, security?
Get stuff done quickly with pre-integrated technology to make your job easier
Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
___
Numpy-discussion mailing list
Numpy-discussion@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/numpy-discussion