Re: [pypy-dev] which xml libraries? was (Re: PyPy 1.4 released)

[Maciej Fijalkowski]

On Wed, Dec 1, 2010 at 9:48 AM, Stefan Behnel  wrote:
> Paolo Giarrusso, 01.12.2010 00:34:
>> Anyway, this does not interact with benchmarks above - Stefan, I still
>> don't get why you complained that pyexpat is slow by showing
>> benchmarks for another module, I guess I do not understand your email,
>> but it asks "reasonable?" after Amaury talks about pyexpat.
>
> Well, in CPython, I can see little to no reason why anyone would go as
> low-level as pyexpat when you can use cElementTree. So IMHO the level to
> compare is what people would normally use rather than what people could
> potentially use if they were beaten hard enough.
>
> Stefan
>

Hey.

Sure, makes sense :-) I think one of the reasons for some slowdown is
that calls from C are not jitted if they don't contain loops
themselves. This doesn't explain the whole thing obviously, because
there is something really wrong going on looking at numbers.
___
pypy-dev@codespeak.net
http://codespeak.net/mailman/listinfo/pypy-dev

Re: [pypy-dev] a possible leak in the object namespace...

[Alex A. Naanou]

On Tue, Nov 30, 2010 at 20:33, Carl Friedrich Bolz  wrote:
> Hi Alex,
>
> On 11/29/2010 09:02 PM, Alex A. Naanou wrote:
>>
>> On Mon, Nov 29, 2010 at 21:46, Carl Friedrich Bolz  wrote:
>
> [snip]
>>>
>>>  a) lots of attributes, which we expect to be rare
>>>  b) access them with setattr, which is a lot slower than a fixed
>>> attribute
>>>  c) access a different attribute every loop iteration, which means the
>>> compiler has to produce one bit of code for every attribute
>>
>> This is intentional (all three points), I did not want the jit to
>> factor out the loop -- I wanted to time the initial attribute
>> creation...
>
> Yes, I fear initial attribute creation is never going to be very efficient.
>
> [snip]
>>
>> But quadratic behaviour + a three orders of magnitude increase in time
>> it takes to create an attr is scarry... but you are right, how often
>> does that usecase happen? :)
>
> I improved some things so that setattr/getattr is a bit faster. Should now
> only be one or two orders of magnitude :-). If you run it tomorrow with the
> nightly build from tonight, you should see an improvement.

That's good news!
It is not uncommon, at least in my work, to write generic code that
does not know of attribute names it uses, thus getattr/setattr
functions are unavoidable.


> The quadraticness is not easily fixable without giving up all optimizations
> with instances of more than X (maybe 1000?) attributes. Again, I don't think
> this is common. And I don't want to chose an X.

I'd even argue that most of the cases were the number of attributes
exceeds a certain "sane" N would and should be better implemented via
a container...
(in spite of the how I wrote the code in my first mail ;) )

I've been planning of moving most of my commercial projects to PyPy as
soon as it is stable enough -- I'm tired of fighting the CPython
implementation for most of the time -- we will see how it goes... too
bad some of the code will be PyPy specific due to the way CPython
works.


Thanks!


>
> Cheers,
>
> Carl Friedrich
>



-- 
Alex.
___
pypy-dev@codespeak.net
http://codespeak.net/mailman/listinfo/pypy-dev

[pypy-dev] (no subject)

[renaud blanch]

hi,

i'm trying to make some pyopengl [0] -based code [1] run on top of pypy.
this is partially successful, but i need some advice to progress further.
pyopengl 3.x makes use of ctypes to provide the opengl binding, and it
works out of the box for simple functions (those that do not takes
c-pointer to buffers of data as arguments).
for the rest, the first issue is that pyopengl use two functions from
the ctypes.pythonapi lib, namely PyString_AsString and
PyBuffer_FromMemory.
any advice on how to replace those functions to make them compatible with pypy?

Mike Fletcher (pyopengl author) gave me some hints about that point:
> For the first issue, those are going to require some reworking, in
> essence those are "C" implemented code that happens to use Python/ctypes
> as the implementation language and makes assumptions about the
> data-storage for the objects (e.g. that a string is internally a
> contiguous series of bytes, which is *not necessarily* true in PyPy).
> We'd need to find a mechanism in PyPy that would give us that direct
> memory-pointer access to be able to use it.  Note: a compacting garbage
> collector (or anything else that can move memory locations) will cause
> problems there, so we may need to find a way to signal PyPy not to move
> a given object, and to use contiguous data-arrays for their storage.

thanks a lot for any advice,

renaud

0. PyOpenGL 3.x / The Python OpenGL® Binding


1. opengl-programmable / a short step by step tutorial to OpenGL
programmable pipeline

___
pypy-dev@codespeak.net
http://codespeak.net/mailman/listinfo/pypy-dev

[pypy-dev] pyopengl on pypy

[renaud blanch]

sorry, i forgot to put a subject.
renaud
___
pypy-dev@codespeak.net
http://codespeak.net/mailman/listinfo/pypy-dev

[pypy-dev] pyopengl on pypy

[renaud blanch]

this time with a subject, the message, and my apologies for the noise,
renaud

On 12/1/10, renaud blanch  wrote:
> hi,
>
> i'm trying to make some pyopengl [0] -based code [1] run on top of pypy.
> this is partially successful, but i need some advice to progress further.
> pyopengl 3.x makes use of ctypes to provide the opengl binding, and it
> works out of the box for simple functions (those that do not takes
> c-pointer to buffers of data as arguments).
> for the rest, the first issue is that pyopengl use two functions from
> the ctypes.pythonapi lib, namely PyString_AsString and
> PyBuffer_FromMemory.
> any advice on how to replace those functions to make them compatible with
> pypy?
>
> Mike Fletcher (pyopengl author) gave me some hints about that point:
>> For the first issue, those are going to require some reworking, in
>> essence those are "C" implemented code that happens to use Python/ctypes
>> as the implementation language and makes assumptions about the
>> data-storage for the objects (e.g. that a string is internally a
>> contiguous series of bytes, which is *not necessarily* true in PyPy).
>> We'd need to find a mechanism in PyPy that would give us that direct
>> memory-pointer access to be able to use it.  Note: a compacting garbage
>> collector (or anything else that can move memory locations) will cause
>> problems there, so we may need to find a way to signal PyPy not to move
>> a given object, and to use contiguous data-arrays for their storage.
>
> thanks a lot for any advice,
>
> renaud
>
> 0. PyOpenGL 3.x / The Python OpenGL® Binding
> 
>
> 1. opengl-programmable / a short step by step tutorial to OpenGL
> programmable pipeline
> 
>
___
pypy-dev@codespeak.net
http://codespeak.net/mailman/listinfo/pypy-dev

Re: [pypy-dev] (no subject)

[Maciej Fijalkowski]

Hey.

I would suggest using array module. It allocates memory in a
non-moving location and it's address can be found using buffer_info
method.

Cheers,
fijal

On Wed, Dec 1, 2010 at 3:18 PM, renaud blanch  wrote:
> hi,
>
> i'm trying to make some pyopengl [0] -based code [1] run on top of pypy.
> this is partially successful, but i need some advice to progress further.
> pyopengl 3.x makes use of ctypes to provide the opengl binding, and it
> works out of the box for simple functions (those that do not takes
> c-pointer to buffers of data as arguments).
> for the rest, the first issue is that pyopengl use two functions from
> the ctypes.pythonapi lib, namely PyString_AsString and
> PyBuffer_FromMemory.
> any advice on how to replace those functions to make them compatible with 
> pypy?
>
> Mike Fletcher (pyopengl author) gave me some hints about that point:
>> For the first issue, those are going to require some reworking, in
>> essence those are "C" implemented code that happens to use Python/ctypes
>> as the implementation language and makes assumptions about the
>> data-storage for the objects (e.g. that a string is internally a
>> contiguous series of bytes, which is *not necessarily* true in PyPy).
>> We'd need to find a mechanism in PyPy that would give us that direct
>> memory-pointer access to be able to use it.  Note: a compacting garbage
>> collector (or anything else that can move memory locations) will cause
>> problems there, so we may need to find a way to signal PyPy not to move
>> a given object, and to use contiguous data-arrays for their storage.
>
> thanks a lot for any advice,
>
> renaud
>
> 0. PyOpenGL 3.x / The Python OpenGL® Binding
> 
>
> 1. opengl-programmable / a short step by step tutorial to OpenGL
> programmable pipeline
> 
> ___
> pypy-dev@codespeak.net
> http://codespeak.net/mailman/listinfo/pypy-dev
>
___
pypy-dev@codespeak.net
http://codespeak.net/mailman/listinfo/pypy-dev

Re: [pypy-dev] pyopengl on pypy

[Armin Rigo]

Hi Renaud,

On Wed, Dec 1, 2010 at 2:21 PM, renaud blanch  wrote:
>>> We'd need to find a mechanism in PyPy that would give us that direct
>>> memory-pointer access to be able to use it.  Note: a compacting garbage
>>> collector (or anything else that can move memory locations) will cause
>>> problems there, so we may need to find a way to signal PyPy not to move
>>> a given object, and to use contiguous data-arrays for their storage.

Indeed, this part from Mike Fletcher got the cause right, but the
solution we recommend is different.  It's kind of impossible in PyPy
to mark objects as non-moving ("pin" them, as the garbage collection
language goes).  Instead you need to just use the official ctypes API,
create_string_buffer().  It's equivalent, but requires one extra copy
of the data.


A bientôt,

Armin.
___
pypy-dev@codespeak.net
http://codespeak.net/mailman/listinfo/pypy-dev

Re: [pypy-dev] pyopengl on pypy

[Amaury Forgeot d'Arc]

Hi,

2010/12/1 renaud blanch 

> > i'm trying to make some pyopengl [0] -based code [1] run on top of pypy.
> > this is partially successful, but i need some advice to progress further.
> > pyopengl 3.x makes use of ctypes to provide the opengl binding, and it
> > works out of the box for simple functions (those that do not takes
> > c-pointer to buffers of data as arguments).
> > for the rest, the first issue is that pyopengl use two functions from
> > the ctypes.pythonapi lib, namely PyString_AsString and
> > PyBuffer_FromMemory.
> > any advice on how to replace those functions to make them compatible with
> > pypy?
> >
> > Mike Fletcher (pyopengl author) gave me some hints about that point:
> >> For the first issue, those are going to require some reworking, in
> >> essence those are "C" implemented code that happens to use Python/ctypes
> >> as the implementation language and makes assumptions about the
> >> data-storage for the objects (e.g. that a string is internally a
> >> contiguous series of bytes, which is *not necessarily* true in PyPy).
> >> We'd need to find a mechanism in PyPy that would give us that direct
> >> memory-pointer access to be able to use it.  Note: a compacting garbage
> >> collector (or anything else that can move memory locations) will cause
> >> problems there, so we may need to find a way to signal PyPy not to move
> >> a given object, and to use contiguous data-arrays for their storage.
>

Don't worry, with PyPy's ctypes you'll never get the address of a moving
object.
c_char_p makes a non-moving copy of the string.

But IMO the call to pythonapi.PyString_AsString could be removed in
OpenGL/arrays/strings.py:

def dataPointer(value):
return ctypes.cast(ctypes.c_char_p(value), ctypes.c_void_p).value

I don't know about the other function, though.

-- 
Amaury Forgeot d'Arc
___
pypy-dev@codespeak.net
http://codespeak.net/mailman/listinfo/pypy-dev

Re: [pypy-dev] pyopengl on pypy

[rndblnch]

hi,

Amaury Forgeot d'Arc  gmail.com> writes:
> But IMO the call to pythonapi.PyString_AsString could be removed in
> OpenGL/arrays/strings.py:
> 
> def dataPointer(value):
>     return ctypes.cast(ctypes.c_char_p(value), ctypes.c_void_p).value

great, this replacement (and also the one suggested by Armin with
create_string_buffer instead of c_char_p) seams to works (i.e., runs fine 
on cpython).

> I don't know about the other function, though.
this one is only in experimental code of pyopengl, so i won't bother much 
for now.


the next issue is now that the array data types used by pyopengl are not
compatible with pypy ctypes: they are lacking (at least) _CData_input as shown
in the stack trace below.
any advice on that one?
should i try to fix PyOpenGL ArrayDatatype, or try to patch pypy's ctypes
function.py to work around that one?

thanks again,
renaud

% pypy 01-direct.py
Traceback (most recent call last):
  File "app_main.py", line 33, in run_toplevel
  File "01-direct.py", line 268, in 
sys.exit(main())
  File "01-direct.py", line 262, in main
init_texture()
  File "01-direct.py", line 48, in init_texture
"".join(pixel(i, j, k) for i in range(width)
  File "/Users/Shared/src/pypy-1.4-osx/site-packages/OpenGL/latebind.py", line
45, in __call__
return self._finalCall( *args, **named )
  File "/Users/Shared/src/pypy-1.4-osx/site-packages/OpenGL/wrapper.py", line
784, in wrapperCall
result = self.wrappedOperation( *cArguments )
  File
"/Users/Shared/src/pypy-1.4-osx/site-packages/OpenGL/platform/baseplatform.py",
line 335, in __call__
return self( *args, **named )
  File "/Users/Shared/src/pypy-1.4-osx/lib_pypy/_ctypes/function.py", line 166,
in __call__
argtypes, argsandobjs = self._wrap_args(argtypes, args)
  File "/Users/Shared/src/pypy-1.4-osx/lib_pypy/_ctypes/function.py", line 281,
in _wrap_args
wrapped = argtype._CData_input(arg)
AttributeError: type object 'ArrayDatatype' has no attribute '_CData_input'


___
pypy-dev@codespeak.net
http://codespeak.net/mailman/listinfo/pypy-dev

Re: [pypy-dev] [PyPy Status Blog] New comment on PyPy 1.4: Ouroboros in practice.

[Antonio Cuni]

On 01/12/10 23:21, Amaury Forgeot d'Arc wrote:
[cut]
> There is already an ongoing effort to port PyPy to Python 2.7.
> 
> But we need some help! It's a good way to become a PyPy developer.
> And no, you don't have to be a JIT expert to implement itertools.combinations
> or asian codecs.

Nice comment Amaury :-).

What about writing a blog post explicitly asking for help? I also agree that
the fast-forward branch is a good way to enter pypy, moreover we should
exploit the hype we seem to have at the moment :-).

ciao,
Anto
___
pypy-dev@codespeak.net
http://codespeak.net/mailman/listinfo/pypy-dev