[pypy-dev] Can't pickle a numpy object?

2011-10-05 Thread mike c
Hi there,
I'm new to pypy and trying it out on some numerical projects and the speed
of pypy+numpy is about 4x faster than cpython+numpy. Pretty impressive!

However, I want to pickle some of my numpy objects and I get the
error: TypeError: can't pickle numarray objects  (full error is included
below)

I realise that the numpy implementation in pypy is currently a
proof-of-concept, and so I was wondering what I would have to change to get
numarray's to be pickle-able.  As simple as adding a something like a
__pickle__ method to numarray?  Or is the problem deeper than that?

Mike.



 import pickle, numpy
 a = numpy.array([1,2,3])
 pickle.dumps(a)
Traceback (most recent call last):
  File "", line 1, in 
  File "/Users/mike/pypy-1.6/lib-python/modified-2.7/pickle.py", line 1423,
in dumps
Pickler(file, protocol).dump(obj)
  File "/Users/mike/pypy-1.6/lib-python/modified-2.7/pickle.py", line 224,
in dump
self.save(obj)
  File "/Users/mike/pypy-1.6/lib-python/modified-2.7/pickle.py", line 306,
in save
rv = reduce(self.proto)
  File "/Users/mike/pypy-1.6/lib-python/2.7/copy_reg.py", line 70, in
_reduce_ex
raise TypeError, "can't pickle %s objects" % base.__name__
TypeError: can't pickle numarray objects
___
pypy-dev mailing list
pypy-dev@python.org
http://mail.python.org/mailman/listinfo/pypy-dev


Re: [pypy-dev] PyPy 1.6 not working on Windows XP

2011-10-05 Thread Ram Rachum
Can you fill in something for "#Do something that uses lots of RAM"? Because
I'm not sure I'll get it right.

On Thu, Oct 6, 2011 at 12:02 AM, Alex Pyattaev wrote:

> I've had a very similar stuff, as in something crashing only when run many
> times when I had a bug in a container type implemented in C. Basically, I
> had
> wrong refcount for the objets, which caused them to be freed by garbage
> collectore while they have been still used. Maybe something similar happens
> in
> the code that wraps windows API that handles file opening. That would
> explain
> why the bug never happens on linux. A good candidate would be incorrect
> refcount for the return value if the file does not  exist. Try something
> like
> this:
> s="some_file"
> rets=[]
> for i in range(1000):
>rets.append(os.stat(s))
>gc.collect()
>#Do something that uses lots of RAM (but a random amount, preferably
> in
> small blocks)
>print rets
> if it crashes then you have exactly that issue. 1000 might be not enough to
> toggle the crash though, as you need the OS to actually allocate different
> segments of memory for this to work. The more RAM you have the more cycles
> you
> need to toggle the crash. At least this approach helped me to debug
> extension
> modules written in C.
>
> BTW, for me on VM the test case does not crash. But I have SP2 windows
> there.
> On Wednesday 05 October 2011 19:37:07 Ram Rachum wrote:
> > On Wed, Oct 5, 2011 at 6:51 PM, Amaury Forgeot d'Arc
> wrote:
> > > 2011/10/5 Ram Rachum :
> > > > Okay, I've spent a few hours print-debugging, and I think I've
> > > > almost got it.
> > > >
> > > > The crash happens on a line:
> > > > st = os.stat(s)
> > > >
> > > > inside `os.path.isdir`, where `s` is a string 'C:\\Documents and
> > > > Settings\\User\\My Documents\\Python
> > > > Projects\\GarlicSim\\garlicsim\\src' This is a directory that
> > > > happens not to exist, but of course this is not>
> > > a
> > >
> > > > good reason to crash.
> > > > I have tried running `os.stat(s)` in the PyPy shell with that same
> > > > `s`,
> > >
> > > but
> > >
> > > > didn't get a crash there. I don't know why it's crashing in Nose but
> > > > not>
> > > in
> > >
> > > > the shell.
> > > >
> > > > Does anyone have a clue?
> > >
> > > it's possible that it's a RPython-level exception, or a bad handle
> > > because too many files wait for the garbage collector to close them.
> > >
> > > Can you give more information about the crash itself?
> > > - What are the last lines printed in the console? Try to disable
> > > "stdout capture" in Nose, by passing the -s option.
> >
> > This is the entire output:
> >
> > Preparing to run tests using Python 2.7.1 (080f42d5c4b4, Aug 23 2011,
> > 11:41:11)
> > [PyPy 1.6.0 with MSC v.1500 32 bit]
> > Running tests directly from GarlicSim repo.
> > Pypy doesn't have wxPython, not loading `garlicsim_wx` tests.
> > nose.config: INFO: Set working dir to C:\Documents and Settings\User\My
> > Documents\Python Projects\GarlicSim
> > nose.config: INFO: Ignoring files matching ['^\\.', '^_', '^setup\\.py$']
> > nose.plugins.cover: INFO: Coverage report will include only packages:
> > ['garlicsim', 'garlicsim_lib', 'garlicsim_wx', 'test_garlicsim',
> > 'test_garlicsim_lib', 'test_garlicsim_wx', 'garlicsim', 'garlicsim_lib',
> > 'garlicsim_wx', 'test_garlicsim', 'test_garlicsim_lib',
> 'test_garlicsim_wx',
> > 'garlicsim', 'garlicsim_lib', 'garlicsim_wx', 'test_garlicsim',
> > 'test_garlicsim_lib', 'test_garlicsim_wx']
> >
> >
> >
> >
> > - after the pypy process has exited, type "echo %ERRORLEVEL%" in the
> >
> > > same console, to print the exit code
> > > of the last process. Which number is it?
> >
> > -1073741819
> >
> > > --
> > > Amaury Forgeot d'Arc
>
___
pypy-dev mailing list
pypy-dev@python.org
http://mail.python.org/mailman/listinfo/pypy-dev


Re: [pypy-dev] PyPQ: a dbapi 2 PostgreSQL driver working with pypy

2011-10-05 Thread Maciej Fijalkowski
On Wed, Oct 5, 2011 at 12:59 PM, Igor Katson  wrote:
> So I've launched a few tests for all PostgreSQL python drivers
> out there, and optimized PyPq a bit along the way.
>
> *IntergerInsert* test does this 3 times (i wanted more, but pg8000
> bloats
> postgres memory usage to 1,5 gigabytes on this somewhy, so i lowered the
> amount
> of queries to 3)
>
>    cursor.executemany('insert into test_table values(%s, %s, %s, %s)',
> [(1,2,3,4)] * 3)
>
> *IntegerSelect* selects this data back into python
>
> *VariableDataInsert* does the same as IntegerInsert, but inserts a string, a
> datetime, a date and a timestamp into the database (except for pg8000, which
> told
> me that it did not support timestamps)
>
> *VariableDataSelect* selects this data back into python
>
> cPython 2.7.2 (32-bit, archlinux latest build), 3 inserts
>
> Psycopg2IntegerInsert.test_insert took 1.78s
> .Psycopg2IntegerSelect.test_select took 0.06s
> .Psycopg2VariableDataInsert.test_insert took 2.57s
> .Psycopg2VariableDataSelect.test_select took 0.25s
>
> .Psycopg2ctIntegerInsert.test_insert took 4.46s
> .Psycopg2ctIntegerSelect.test_select took 1.62s
> .Psycopg2ctVariableDataInsert.test_insert took 6.00s
> .Psycopg2ctVariableDataSelect.test_select took 3.31s
>
> .PyPQIntegerInsertTest.test_insert took 3.41s
> .PyPQIntegerSelectTest.test_select took 0.84s
> .PyPQVariableDataInsertTest.test_insert took 4.07s
> .PyPQVariableDataSelectTest.test_select took 3.70s
>
> pg8000IntegerInsert.test_insert took 16.20s
> .pg8000IntegerSelect.test_select took 1.43s
> .pg8000VariableDataInsert.test_insert took 18.00s
> .pg8000VariableDataSelect.test_select took 2.17s
>
> PyPy 1.6.0 (32-bit, archlinux latest build), 3 inserts
>
> Psycopg2ctIntegerInsert.test_insert took 2.69s
> .Psycopg2ctIntegerSelect.test_select took 0.63s
> .Psycopg2ctVariableDataInsert.test_insert took 4.53s
> .Psycopg2ctVariableDataSelect.test_select took 1.36s
>
> .PyPQIntegerInsertTest.test_insert took 4.61s
> .PyPQIntegerSelectTest.test_select took 0.37s
> .PyPQVariableDataInsertTest.test_insert took 4.48s
> .PyPQVariableDataSelectTest.test_select took 1.58s
>
> pg8000IntegerInsert.test_insert took 8.34s
> .pg8000IntegerSelect.test_select took 0.60s
> .pg8000VariableDataInsert.test_insert took 9.15s
> .pg8000VariableDataSelect.test_select took 1.64s
>
> As we can see, pg8000 is slow on inserts, and as i've said, it does some
> strange
> things to my postgres, bloating the postgres memory usage to 1.5 gigabytes
> (i tried
> to insert 10 records with executemany)
>
> On cPython, pypq is faster than psycopg2ct and pg8000, except for
> VariableDataSelect
> test.
> On PyPy, all of them get faster, except pypq, though it is still a bit
> faster
> than psycopg2ct in 2 tests.
>
> Next, I tested pypq side by side to see the difference more clearly.
>
> Here are the results.
>
> cPython 2.7.2 (32-bit, archlinux latest build), 20 inserts
>
> Psycopg2IntegerInsert.test_insert took 12.22s
> .Psycopg2IntegerSelect.test_select took 0.39s
> .Psycopg2VariableDataInsert.test_insert took 17.30s
> .Psycopg2VariableDataSelect.test_select took 1.71s
>
> .Psycopg2ctIntegerInsert.test_insert took 28.56s
> .Psycopg2ctIntegerSelect.test_select took 10.48s
> .Psycopg2ctVariableDataInsert.test_insert took 38.53s
> .Psycopg2ctVariableDataSelect.test_select took 21.67s
>
> .PyPQIntegerInsertTest.test_insert took 22.53s
> .PyPQIntegerSelectTest.test_select took 5.59s
> .PyPQVariableDataInsertTest.test_insert took 26.86s
> .PyPQVariableDataSelectTest.test_select took 24.84s
>
> PyPy 1.6.0 (32-bit, archlinux latest build), 20 inserts
>
> Psycopg2ctIntegerInsert.test_insert took 14.11s
> .Psycopg2ctIntegerSelect.test_select took 3.18s
> .Psycopg2ctVariableDataInsert.test_insert took 29.36s
> .Psycopg2ctVariableDataSelect.test_select took 7.78s
>
> .PyPQIntegerInsertTest.test_insert took 25.91s
> .PyPQIntegerSelectTest.test_select took 1.92s
> .PyPQVariableDataInsertTest.test_insert took 30.31s
> .PyPQVariableDataSelectTest.test_select took 8.73s
>
> On 10/05/2011 01:38 AM, Maciej Fijalkowski wrote:
>>
>> On Tue, Oct 4, 2011 at 4:51 PM, Igor Katson  wrote:
>>>
>>> Hi, Dan,
>>> before answering I'll describe the situation a bit.
>>>
>>> there was a question today, if I know about pg8000 or psycopg2ct.
>>>
>>> As for pg8000, pure python should be slower than ctypes, anyway, so I
>>> don't
>>> think these two should be compared.
>>
>> [citation needed]
>
>

Great data!

It probably does make sense to run each of pypy tests twice to see how
much time is spent warming up the JIT, although definitely the data is
very interesting already.

Cheers,
fijal
___
pypy-dev mailing list
pypy-dev@python.org
http://mail.python.org/mailman/listinfo/pypy-dev


Re: [pypy-dev] PyPy 1.6 not working on Windows XP

2011-10-05 Thread Alex Pyattaev
I've had a very similar stuff, as in something crashing only when run many 
times when I had a bug in a container type implemented in C. Basically, I had 
wrong refcount for the objets, which caused them to be freed by garbage 
collectore while they have been still used. Maybe something similar happens in 
the code that wraps windows API that handles file opening. That would explain 
why the bug never happens on linux. A good candidate would be incorrect 
refcount for the return value if the file does not  exist. Try something like 
this:
s="some_file"
rets=[]
for i in range(1000):
rets.append(os.stat(s))
gc.collect()
#Do something that uses lots of RAM (but a random amount, preferably in 
small blocks)
print rets
if it crashes then you have exactly that issue. 1000 might be not enough to 
toggle the crash though, as you need the OS to actually allocate different 
segments of memory for this to work. The more RAM you have the more cycles you 
need to toggle the crash. At least this approach helped me to debug extension 
modules written in C.

BTW, for me on VM the test case does not crash. But I have SP2 windows there.
On Wednesday 05 October 2011 19:37:07 Ram Rachum wrote:
> On Wed, Oct 5, 2011 at 6:51 PM, Amaury Forgeot d'Arc 
wrote:
> > 2011/10/5 Ram Rachum :
> > > Okay, I've spent a few hours print-debugging, and I think I've
> > > almost got it.
> > > 
> > > The crash happens on a line:
> > > st = os.stat(s)
> > > 
> > > inside `os.path.isdir`, where `s` is a string 'C:\\Documents and
> > > Settings\\User\\My Documents\\Python
> > > Projects\\GarlicSim\\garlicsim\\src' This is a directory that
> > > happens not to exist, but of course this is not> 
> > a
> > 
> > > good reason to crash.
> > > I have tried running `os.stat(s)` in the PyPy shell with that same
> > > `s`,
> > 
> > but
> > 
> > > didn't get a crash there. I don't know why it's crashing in Nose but
> > > not> 
> > in
> > 
> > > the shell.
> > > 
> > > Does anyone have a clue?
> > 
> > it's possible that it's a RPython-level exception, or a bad handle
> > because too many files wait for the garbage collector to close them.
> > 
> > Can you give more information about the crash itself?
> > - What are the last lines printed in the console? Try to disable
> > "stdout capture" in Nose, by passing the -s option.
> 
> This is the entire output:
> 
> Preparing to run tests using Python 2.7.1 (080f42d5c4b4, Aug 23 2011,
> 11:41:11)
> [PyPy 1.6.0 with MSC v.1500 32 bit]
> Running tests directly from GarlicSim repo.
> Pypy doesn't have wxPython, not loading `garlicsim_wx` tests.
> nose.config: INFO: Set working dir to C:\Documents and Settings\User\My
> Documents\Python Projects\GarlicSim
> nose.config: INFO: Ignoring files matching ['^\\.', '^_', '^setup\\.py$']
> nose.plugins.cover: INFO: Coverage report will include only packages:
> ['garlicsim', 'garlicsim_lib', 'garlicsim_wx', 'test_garlicsim',
> 'test_garlicsim_lib', 'test_garlicsim_wx', 'garlicsim', 'garlicsim_lib',
> 'garlicsim_wx', 'test_garlicsim', 'test_garlicsim_lib', 'test_garlicsim_wx',
> 'garlicsim', 'garlicsim_lib', 'garlicsim_wx', 'test_garlicsim',
> 'test_garlicsim_lib', 'test_garlicsim_wx']
> 
> 
> 
> 
> - after the pypy process has exited, type "echo %ERRORLEVEL%" in the
> 
> > same console, to print the exit code
> > of the last process. Which number is it?
> 
> -1073741819
> 
> > --
> > Amaury Forgeot d'Arc
___
pypy-dev mailing list
pypy-dev@python.org
http://mail.python.org/mailman/listinfo/pypy-dev


Re: [pypy-dev] gettrace?

2011-10-05 Thread Amaury Forgeot d'Arc
2011/10/5 Armin Rigo :
> Indeed, sys.gettrace() and sys.getprofile() were added in Python 2.6,
> with proper unit tests in test_sys_settrace.py.  We should try to
> understand why this test didn't fail so far on PyPy...  Do we run it
> at all?  If not, it's kind of bad.

test_sys_settrace.py runs correctly,
but the function is named "set_and_retrieve_func"...
without the test_ prefix!

Fixed in revision 383ca802ba07, will fix in cpython as well.

-- 
Amaury Forgeot d'Arc
___
pypy-dev mailing list
pypy-dev@python.org
http://mail.python.org/mailman/listinfo/pypy-dev


Re: [pypy-dev] gettrace?

2011-10-05 Thread Armin Rigo
Hi Amaury,

On Wed, Oct 5, 2011 at 21:54, Amaury Forgeot d'Arc  wrote:
>> Is there any other way to do the equivalent of `sys.gettrace` in PyPy?
>
> Hum, it was probably ovelooked.
> Seems trivial to implement, will give a try.

Indeed, sys.gettrace() and sys.getprofile() were added in Python 2.6,
with proper unit tests in test_sys_settrace.py.  We should try to
understand why this test didn't fail so far on PyPy...  Do we run it
at all?  If not, it's kind of bad.


Armin
___
pypy-dev mailing list
pypy-dev@python.org
http://mail.python.org/mailman/listinfo/pypy-dev


Re: [pypy-dev] gettrace?

2011-10-05 Thread Amaury Forgeot d'Arc
2011/10/5 Ram Rachum :
> Hey guys,
> I notice that PyPy doesn't offer a `sys.gettrace` function, given that it is
> an implementation detail.
> Is there any other way to do the equivalent of `sys.gettrace` in PyPy?

Hum, it was probably ovelooked.
Seems trivial to implement, will give a try.

-- 
Amaury Forgeot d'Arc
___
pypy-dev mailing list
pypy-dev@python.org
http://mail.python.org/mailman/listinfo/pypy-dev


[pypy-dev] gettrace?

2011-10-05 Thread Ram Rachum
Hey guys,

I notice that PyPy doesn't offer a `sys.gettrace` function, given that it is
an implementation detail.

Is there any other way to do the equivalent of `sys.gettrace` in PyPy?


Thanks,
Ram.
___
pypy-dev mailing list
pypy-dev@python.org
http://mail.python.org/mailman/listinfo/pypy-dev


Re: [pypy-dev] strange error in urlunparse

2011-10-05 Thread Max Lavrenov
Hi Antonio
You were right, with --jit off project works just fine.
I will post simple project to reproduce this bug tomorrow.

Best regards,
Max

On Wed, Oct 5, 2011 at 22:36, Antonio Cuni  wrote:

> On 05/10/11 16:30, Amaury Forgeot d'Arc wrote:
>
>> 2011/10/5 Max Lavrenov
>> >:
>>
>>> after ~500 successfull responses i am starting to get error on line "if
>>> params"
>>>
>>
>> Looks like a JIT error to me, which has a default threshold of 1000
>> iterations.
>> Can you try again with
>> pypy --jit threshold=-1
>> to completely disable the JIT?
>>
>
> --jit threshold=-1 only disables the jit for loops, but it will still
> compile functions.
>
> To completely disable the jit you should use --jit off.
>
> Max, could you please try again with --jit off?
> Your problems reminds me of a bug that I encountered ~1 year ago:
> UnboundLocalError which disappears if I print it.
>
> I don't remember the details, but in that case it was a bug in the JIT.
>
> ciao,
> Anto
>
___
pypy-dev mailing list
pypy-dev@python.org
http://mail.python.org/mailman/listinfo/pypy-dev


Re: [pypy-dev] strange error in urlunparse

2011-10-05 Thread Antonio Cuni

On 05/10/11 16:30, Amaury Forgeot d'Arc wrote:

2011/10/5 Max Lavrenov:

after ~500 successfull responses i am starting to get error on line "if
params"


Looks like a JIT error to me, which has a default threshold of 1000 iterations.
Can you try again with
 pypy --jit threshold=-1
to completely disable the JIT?


--jit threshold=-1 only disables the jit for loops, but it will still compile 
functions.


To completely disable the jit you should use --jit off.

Max, could you please try again with --jit off?
Your problems reminds me of a bug that I encountered ~1 year ago: 
UnboundLocalError which disappears if I print it.


I don't remember the details, but in that case it was a bug in the JIT.

ciao,
Anto
___
pypy-dev mailing list
pypy-dev@python.org
http://mail.python.org/mailman/listinfo/pypy-dev


Re: [pypy-dev] PyPy 1.6 not working on Windows XP

2011-10-05 Thread Ram Rachum
On Wed, Oct 5, 2011 at 6:51 PM, Amaury Forgeot d'Arc wrote:

> 2011/10/5 Ram Rachum :
> > Okay, I've spent a few hours print-debugging, and I think I've almost got
> > it.
> > The crash happens on a line:
> > st = os.stat(s)
> > inside `os.path.isdir`, where `s` is a string 'C:\\Documents and
> > Settings\\User\\My Documents\\Python Projects\\GarlicSim\\garlicsim\\src'
> > This is a directory that happens not to exist, but of course this is not
> a
> > good reason to crash.
> > I have tried running `os.stat(s)` in the PyPy shell with that same `s`,
> but
> > didn't get a crash there. I don't know why it's crashing in Nose but not
> in
> > the shell.
> >
> > Does anyone have a clue?
>
> it's possible that it's a RPython-level exception, or a bad handle because
> too many files wait for the garbage collector to close them.
>
> Can you give more information about the crash itself?
> - What are the last lines printed in the console? Try to disable
> "stdout capture" in Nose, by passing the -s option.
>

This is the entire output:

Preparing to run tests using Python 2.7.1 (080f42d5c4b4, Aug 23 2011,
11:41:11)
[PyPy 1.6.0 with MSC v.1500 32 bit]
Running tests directly from GarlicSim repo.
Pypy doesn't have wxPython, not loading `garlicsim_wx` tests.
nose.config: INFO: Set working dir to C:\Documents and Settings\User\My
Documents\Python Projects\GarlicSim
nose.config: INFO: Ignoring files matching ['^\\.', '^_', '^setup\\.py$']
nose.plugins.cover: INFO: Coverage report will include only packages:
['garlicsim', 'garlicsim_lib', 'garlicsim_wx', 'test_garlicsim',
'test_garlicsim_lib', 'test_garlicsim_wx', 'garlicsim', 'garlicsim_lib',
'garlicsim_wx', 'test_garlicsim', 'test_garlicsim_lib', 'test_garlicsim_wx',
'garlicsim', 'garlicsim_lib', 'garlicsim_wx', 'test_garlicsim',
'test_garlicsim_lib', 'test_garlicsim_wx']




- after the pypy process has exited, type "echo %ERRORLEVEL%" in the
> same console, to print the exit code
> of the last process. Which number is it?
>

-1073741819


>
> --
> Amaury Forgeot d'Arc
>
___
pypy-dev mailing list
pypy-dev@python.org
http://mail.python.org/mailman/listinfo/pypy-dev


Re: [pypy-dev] PyPy 1.6 not working on Windows XP

2011-10-05 Thread Amaury Forgeot d'Arc
2011/10/5 Ram Rachum :
> Okay, I've spent a few hours print-debugging, and I think I've almost got
> it.
> The crash happens on a line:
>     st = os.stat(s)
> inside `os.path.isdir`, where `s` is a string 'C:\\Documents and
> Settings\\User\\My Documents\\Python Projects\\GarlicSim\\garlicsim\\src'
> This is a directory that happens not to exist, but of course this is not a
> good reason to crash.
> I have tried running `os.stat(s)` in the PyPy shell with that same `s`, but
> didn't get a crash there. I don't know why it's crashing in Nose but not in
> the shell.
>
> Does anyone have a clue?

it's possible that it's a RPython-level exception, or a bad handle because
too many files wait for the garbage collector to close them.

Can you give more information about the crash itself?
- What are the last lines printed in the console? Try to disable
"stdout capture" in Nose, by passing the -s option.
- after the pypy process has exited, type "echo %ERRORLEVEL%" in the
same console, to print the exit code
of the last process. Which number is it?

-- 
Amaury Forgeot d'Arc
___
pypy-dev mailing list
pypy-dev@python.org
http://mail.python.org/mailman/listinfo/pypy-dev


Re: [pypy-dev] PyPy 1.6 not working on Windows XP

2011-10-05 Thread Ram Rachum
On Wed, Oct 5, 2011 at 4:11 PM, Ram Rachum  wrote:

> On Wed, Oct 5, 2011 at 2:23 PM, Alex Pyattaev wrote:
>
>> Another option is to edit the sources of the test suite adding print
>> statements incrementally until you spot the place where it crashes. It is
>> a
>> slow, but very reliable way. That is of course if it is a particular
>> segment
>> of python code that crashes it.
>>
>
> I'll try, thanks.
>
>
Okay, I've spent a few hours print-debugging, and I think I've almost got
it.

The crash happens on a line:

st = os.stat(s)

inside `os.path.isdir`, where `s` is a string 'C:\\Documents and
Settings\\User\\My Documents\\Python Projects\\GarlicSim\\garlicsim\\src'

This is a directory that happens not to exist, but of course this is not a
good reason to crash.

I have tried running `os.stat(s)` in the PyPy shell with that same `s`, but
didn't get a crash there. I don't know why it's crashing in Nose but not in
the shell.


Does anyone have a clue?


Ram.
___
pypy-dev mailing list
pypy-dev@python.org
http://mail.python.org/mailman/listinfo/pypy-dev


Re: [pypy-dev] strange error in urlunparse

2011-10-05 Thread Amaury Forgeot d'Arc
2011/10/5 Max Lavrenov :
> I could try writing small application for reproduce this error.  It happens
> in twisted function twisted.web.client.getPage
> Or may be i shoud try another version of pypy or twisted ( i use trunk
> version of twisted ) ?

Yes, please try to find a minimal example that shows the issue.

-- 
Amaury Forgeot d'Arc
___
pypy-dev mailing list
pypy-dev@python.org
http://mail.python.org/mailman/listinfo/pypy-dev


Re: [pypy-dev] strange error in urlunparse

2011-10-05 Thread Max Lavrenov
Hello Amaury
Thanks for response.

No, i got same error with this line
python --jit threshold=-1  /home/e-max/.virtualenvs/pypy/bin/twistd -n
dalight

But If i change urlunparse function and try printing date variable before,
all starts work correctly.

def urlunparse(data):
"""Put a parsed URL back together again.  This may result in a
slightly different, but equivalent URL, if the URL that was parsed
originally had redundant delimiters, e.g. a ? with an empty query
(the draft states that these are equivalent)."""
print data
scheme, netloc, url, params, query, fragment = data
if params:
url = "%s;%s" % (url, params)
return urlunsplit((scheme, netloc, url, query, fragment))


On Wed, Oct 5, 2011 at 18:30, Amaury Forgeot d'Arc wrote:

> 2011/10/5 Max Lavrenov :
> > after ~500 successfull responses i am starting to get error on line "if
> > params"
>
> Looks like a JIT error to me, which has a default threshold of 1000
> iterations.
> Can you try again with
>pypy --jit threshold=-1
> to completely disable the JIT?
>
> --
> Amaury Forgeot d'Arc
>
___
pypy-dev mailing list
pypy-dev@python.org
http://mail.python.org/mailman/listinfo/pypy-dev


Re: [pypy-dev] strange error in urlunparse

2011-10-05 Thread Max Lavrenov
I could try writing small application for reproduce this error.  It happens
in twisted function twisted.web.client.getPage
Or may be i shoud try another version of pypy or twisted ( i use trunk
version of twisted ) ?

On Wed, Oct 5, 2011 at 18:57, Max Lavrenov  wrote:

> Hello Amaury
> Thanks for response.
>
> No, i got same error with this line
> python --jit threshold=-1  /home/e-max/.virtualenvs/pypy/bin/twistd -n
> dalight
>
> But If i change urlunparse function and try printing date variable before,
> all starts work correctly.
>
> def urlunparse(data):
> """Put a parsed URL back together again.  This may result in a
> slightly different, but equivalent URL, if the URL that was parsed
> originally had redundant delimiters, e.g. a ? with an empty query
> (the draft states that these are equivalent)."""
> print data
>
> scheme, netloc, url, params, query, fragment = data
> if params:
> url = "%s;%s" % (url, params)
> return urlunsplit((scheme, netloc, url, query, fragment))
>
>
> On Wed, Oct 5, 2011 at 18:30, Amaury Forgeot d'Arc wrote:
>
>> 2011/10/5 Max Lavrenov :
>> > after ~500 successfull responses i am starting to get error on line "if
>> > params"
>>
>> Looks like a JIT error to me, which has a default threshold of 1000
>> iterations.
>> Can you try again with
>>pypy --jit threshold=-1
>> to completely disable the JIT?
>>
>> --
>> Amaury Forgeot d'Arc
>>
>
>
___
pypy-dev mailing list
pypy-dev@python.org
http://mail.python.org/mailman/listinfo/pypy-dev


Re: [pypy-dev] strange error in urlunparse

2011-10-05 Thread Amaury Forgeot d'Arc
2011/10/5 Max Lavrenov :
> after ~500 successfull responses i am starting to get error on line "if
> params"

Looks like a JIT error to me, which has a default threshold of 1000 iterations.
Can you try again with
pypy --jit threshold=-1
to completely disable the JIT?

-- 
Amaury Forgeot d'Arc
___
pypy-dev mailing list
pypy-dev@python.org
http://mail.python.org/mailman/listinfo/pypy-dev


Re: [pypy-dev] PyPy 1.6 not working on Windows XP

2011-10-05 Thread Amaury Forgeot d'Arc
2011/10/5 Brian Curtin :
> On Wed, Oct 5, 2011 at 08:22, Amaury Forgeot d'Arc  wrote:
>> 2011/10/5 Michael Foord :
>>> Well, in recent versions it gained the faulthandler module... :-)
>>>
>>> http://pypi.python.org/pypi/faulthandler/
>>
>> Yes, and this was a *great* improvement!
>> I hope we will be able to adapt it to pypy.
>
> It may be early to start bringing this up, but I recently created an
> extension to catch crashes in C code and write minidumps:
> https://bitbucket.org/briancurtin/minidumper. It's 3.x only at the
> moment, although I'll be backporting within the next few days. I'll
> have a look at how/if it works on PyPy once I get there.

The code is quite simple, and should probably compile as is with pypy
(except for the PyModuleDef of course)
but it would be as simple to rewrite it in RPython.

btw there is a suspect thing in your code:
"app_name = pyname" will store the char* buffer into a static variable,
but nothing guarantees that the PyObject passed will stay alive!

-- 
Amaury Forgeot d'Arc
___
pypy-dev mailing list
pypy-dev@python.org
http://mail.python.org/mailman/listinfo/pypy-dev


Re: [pypy-dev] PyPy 1.6 not working on Windows XP

2011-10-05 Thread Ram Rachum
On Wed, Oct 5, 2011 at 2:23 PM, Alex Pyattaev wrote:

> Generally, any binary-level debugger such as gdb or MSVC should work with
> pypy. At the very least you will find which operation crashed.
>

As I said to Amaury, I don't know how to use those... Python is the only
language I program in.



> If it is something really specific, for example
> sin/log/sign, it might be quite easy to map it back to python code. If it
> is
> not, it will be nearly impossible to find the original source line (at
> least
> I've failed when I tried).
>
> Another option is to edit the sources of the test suite adding print
> statements incrementally until you spot the place where it crashes. It is a
> slow, but very reliable way. That is of course if it is a particular
> segment
> of python code that crashes it.
>

I'll try, thanks.


>
> Also, could you send your exact environtment specs? I'll try to replicate
> it
> on a VM and see if it crashes for me too. I have an XP VM somewhere.
>

What specs do you mean? It's just the recent PyPy 1.6 on a Windows XP SP3 32
bit machine with minimal packages installed. (distribute, pip, nosetests.)

Let me know if you need any more data.


>
> PS: Sorry for my stupid joke about switching to linux. It was meant to
> cheer
> you up a bit.
>

Forgiven :)



> Alex.
>
> On Wednesday 05 October 2011 14:18:08 Ram Rachum wrote:
> > On Wed, Oct 5, 2011 at 2:11 PM, Amaury Forgeot d'Arc
> wrote:
> > > 2011/10/5 Ram Rachum :
> > > > I have hundreds of tests, and PyPy fails before a single one begins.
> > > > It
> > > > seems that PyPy crashes somewhere in nose's initialization.
> > > > Isn't there a way to find the last Python line run before the crash
> > >
> > > without
> > >
> > > > stepping with a finer granularity every time?
> > >
> > > Not without a debugger, I'm afraid
> > >
> > > --
> > > Amaury Forgeot d'Arc
> >
> > How do I run the Nose test suite on Pypy with a debugger? I usually use
> Wing
> > IDE, but it doesn't support PyPy. I'm also aware of Nose's `--pdb` flag
> > which drops you into the debugger after an error, but it doesn't work
> here
> > because this crash seems to be happening at a lower level. So I don't
> know
> > how to start this in a debugger.
> >
> >
> > Ram.
> ___
> pypy-dev mailing list
> pypy-dev@python.org
> http://mail.python.org/mailman/listinfo/pypy-dev
>
___
pypy-dev mailing list
pypy-dev@python.org
http://mail.python.org/mailman/listinfo/pypy-dev


Re: [pypy-dev] PyPy 1.6 not working on Windows XP

2011-10-05 Thread Brian Curtin
On Wed, Oct 5, 2011 at 08:22, Amaury Forgeot d'Arc  wrote:
> 2011/10/5 Michael Foord :
>> Well, in recent versions it gained the faulthandler module... :-)
>>
>> http://pypi.python.org/pypi/faulthandler/
>
> Yes, and this was a *great* improvement!
> I hope we will be able to adapt it to pypy.

It may be early to start bringing this up, but I recently created an
extension to catch crashes in C code and write minidumps:
https://bitbucket.org/briancurtin/minidumper. It's 3.x only at the
moment, although I'll be backporting within the next few days. I'll
have a look at how/if it works on PyPy once I get there.

(minidumper will likely be absorbed into faulthandler once it becomes
more complete, which is why I mention it)
___
pypy-dev mailing list
pypy-dev@python.org
http://mail.python.org/mailman/listinfo/pypy-dev


[pypy-dev] strange error in urlunparse

2011-10-05 Thread Max Lavrenov
Hello all.

I make some experiments with pypy in our twisted project and get strange
error in standard urlparse.urlunparse function

def urlunparse(data):
"""Put a parsed URL back together again.  This may result in a
slightly different, but equivalent URL, if the URL that was parsed
originally had redundant delimiters, e.g. a ? with an empty query
(the draft states that these are equivalent)."""
scheme, netloc, url, params, query, fragment = data
if params:
url = "%s;%s" % (url, params)
return urlunsplit((scheme, netloc, url, query, fragment))

If i try using apache benchmark , after ~500 successfull responses i am
starting to get error on line "if params":   (216 line in real urlparse.py
file)  with message "local variable 'params' referenced before assignment".
  How it's possible?

My pypy version is "Python 2.7.1 (7acf2b8fcafd, Sep 26 2011, 11:38:29) [PyPy
1.6.1-dev0 with GCC 4.6.1] on linux2"

Best regards,
Max Lavrenov
___
pypy-dev mailing list
pypy-dev@python.org
http://mail.python.org/mailman/listinfo/pypy-dev


Re: [pypy-dev] PyPy 1.6 not working on Windows XP

2011-10-05 Thread Amaury Forgeot d'Arc
2011/10/5 Michael Foord :
> Well, in recent versions it gained the faulthandler module... :-)
>
> http://pypi.python.org/pypi/faulthandler/

Yes, and this was a *great* improvement!
I hope we will be able to adapt it to pypy.

-- 
Amaury Forgeot d'Arc
___
pypy-dev mailing list
pypy-dev@python.org
http://mail.python.org/mailman/listinfo/pypy-dev


Re: [pypy-dev] PyPy 1.6 not working on Windows XP

2011-10-05 Thread Michael Foord
On 5 October 2011 13:50, Amaury Forgeot d'Arc  wrote:

> 2011/10/5 Alex Pyattaev :
> > In Cpython it works
> > somewhat better, but not in PYPY
>
> How is CPython behavior better with segfaults?
>
>
Well, in recent versions it gained the faulthandler module... :-)

http://pypi.python.org/pypi/faulthandler/

Michael



> --
> Amaury Forgeot d'Arc
> ___
> pypy-dev mailing list
> pypy-dev@python.org
> http://mail.python.org/mailman/listinfo/pypy-dev
>



-- 

http://www.voidspace.org.uk/

May you do good and not evil
May you find forgiveness for yourself and forgive others
May you share freely, never taking more than you give.
-- the sqlite blessing http://www.sqlite.org/different.html
___
pypy-dev mailing list
pypy-dev@python.org
http://mail.python.org/mailman/listinfo/pypy-dev


Re: [pypy-dev] PyPy 1.6 not working on Windows XP

2011-10-05 Thread Amaury Forgeot d'Arc
2011/10/5 Alex Pyattaev :
> In Cpython it works
> somewhat better, but not in PYPY

How is CPython behavior better with segfaults?

-- 
Amaury Forgeot d'Arc
___
pypy-dev mailing list
pypy-dev@python.org
http://mail.python.org/mailman/listinfo/pypy-dev


Re: [pypy-dev] PyPy 1.6 not working on Windows XP

2011-10-05 Thread Alex Pyattaev
> If I use a Python debugger, can't I just step forward line by line, see
> where I get the crash, and then isolate the offending line?
The way pypy works - no you can not really do that. In Cpython it works 
somewhat better, but not in PYPY. Basically you have to use C debugger to 
locate the crash point, because the program that crashes can not really 
identify where it crashed precisely (well it can but pypy detects crash points 
very unprecisely). So in order to figure where exactly it crashes you have to 
use C debugger. OR, as I have said, add print statements until you localize 
the bug. *This may not work in some cases, as adding print's actually modifies 
the program=> it might not crash at all*
Alex.
On Wednesday 05 October 2011 14:27:45 Ram Rachum wrote:

___
pypy-dev mailing list
pypy-dev@python.org
http://mail.python.org/mailman/listinfo/pypy-dev


Re: [pypy-dev] PyPy 1.6 not working on Windows XP

2011-10-05 Thread Ram Rachum
On Wed, Oct 5, 2011 at 2:22 PM, Amaury Forgeot d'Arc wrote:

> 2011/10/5 Ram Rachum :
> > How do I run the Nose test suite on Pypy with a debugger? I usually use
> Wing
> > IDE, but it doesn't support PyPy. I'm also aware of Nose's `--pdb` flag
> > which drops you into the debugger after an error, but it doesn't work
> here
> > because this crash seems to be happening at a lower level. So I don't
> know
> > how to start this in a debugger.
>
> A Python debugger won't help, since it runs in the same (segfaulting)
> process.
> You need a C-level debugger, i.e. Visual Studio.
>
> --
> Amaury Forgeot d'Arc
>

I don't know how to use that... I don't program C at all, only Python.

If I use a Python debugger, can't I just step forward line by line, see
where I get the crash, and then isolate the offending line?


Ram.
___
pypy-dev mailing list
pypy-dev@python.org
http://mail.python.org/mailman/listinfo/pypy-dev


Re: [pypy-dev] PyPy 1.6 not working on Windows XP

2011-10-05 Thread Alex Pyattaev
Generally, any binary-level debugger such as gdb or MSVC should work with 
pypy. At the very least you will find which operation crashed. 

If it is something really specific, for example 
sin/log/sign, it might be quite easy to map it back to python code. If it is 
not, it will be nearly impossible to find the original source line (at least 
I've failed when I tried). 

Another option is to edit the sources of the test suite adding print 
statements incrementally until you spot the place where it crashes. It is a 
slow, but very reliable way. That is of course if it is a particular segment 
of python code that crashes it.

Also, could you send your exact environtment specs? I'll try to replicate it 
on a VM and see if it crashes for me too. I have an XP VM somewhere.

PS: Sorry for my stupid joke about switching to linux. It was meant to cheer 
you up a bit.
Alex.

On Wednesday 05 October 2011 14:18:08 Ram Rachum wrote:
> On Wed, Oct 5, 2011 at 2:11 PM, Amaury Forgeot d'Arc 
wrote:
> > 2011/10/5 Ram Rachum :
> > > I have hundreds of tests, and PyPy fails before a single one begins.
> > > It
> > > seems that PyPy crashes somewhere in nose's initialization.
> > > Isn't there a way to find the last Python line run before the crash
> > 
> > without
> > 
> > > stepping with a finer granularity every time?
> > 
> > Not without a debugger, I'm afraid
> > 
> > --
> > Amaury Forgeot d'Arc
> 
> How do I run the Nose test suite on Pypy with a debugger? I usually use Wing
> IDE, but it doesn't support PyPy. I'm also aware of Nose's `--pdb` flag
> which drops you into the debugger after an error, but it doesn't work here
> because this crash seems to be happening at a lower level. So I don't know
> how to start this in a debugger.
> 
> 
> Ram.
___
pypy-dev mailing list
pypy-dev@python.org
http://mail.python.org/mailman/listinfo/pypy-dev


Re: [pypy-dev] PyPy 1.6 not working on Windows XP

2011-10-05 Thread Amaury Forgeot d'Arc
2011/10/5 Ram Rachum :
> How do I run the Nose test suite on Pypy with a debugger? I usually use Wing
> IDE, but it doesn't support PyPy. I'm also aware of Nose's `--pdb` flag
> which drops you into the debugger after an error, but it doesn't work here
> because this crash seems to be happening at a lower level. So I don't know
> how to start this in a debugger.

A Python debugger won't help, since it runs in the same (segfaulting) process.
You need a C-level debugger, i.e. Visual Studio.

-- 
Amaury Forgeot d'Arc
___
pypy-dev mailing list
pypy-dev@python.org
http://mail.python.org/mailman/listinfo/pypy-dev


Re: [pypy-dev] PyPy 1.6 not working on Windows XP

2011-10-05 Thread Ram Rachum
On Wed, Oct 5, 2011 at 2:11 PM, Amaury Forgeot d'Arc wrote:

> 2011/10/5 Ram Rachum :
> > I have hundreds of tests, and PyPy fails before a single one begins. It
> > seems that PyPy crashes somewhere in nose's initialization.
> > Isn't there a way to find the last Python line run before the crash
> without
> > stepping with a finer granularity every time?
>
> Not without a debugger, I'm afraid
>
> --
> Amaury Forgeot d'Arc
>

How do I run the Nose test suite on Pypy with a debugger? I usually use Wing
IDE, but it doesn't support PyPy. I'm also aware of Nose's `--pdb` flag
which drops you into the debugger after an error, but it doesn't work here
because this crash seems to be happening at a lower level. So I don't know
how to start this in a debugger.


Ram.
___
pypy-dev mailing list
pypy-dev@python.org
http://mail.python.org/mailman/listinfo/pypy-dev


Re: [pypy-dev] PyPy 1.6 not working on Windows XP

2011-10-05 Thread Amaury Forgeot d'Arc
2011/10/5 Ram Rachum :
> I have hundreds of tests, and PyPy fails before a single one begins. It
> seems that PyPy crashes somewhere in nose's initialization.
> Isn't there a way to find the last Python line run before the crash without
> stepping with a finer granularity every time?

Not without a debugger, I'm afraid

-- 
Amaury Forgeot d'Arc
___
pypy-dev mailing list
pypy-dev@python.org
http://mail.python.org/mailman/listinfo/pypy-dev


Re: [pypy-dev] PyPy 1.6 not working on Windows XP

2011-10-05 Thread Ram Rachum
On Wed, Oct 5, 2011 at 2:07 PM, Amaury Forgeot d'Arc wrote:

> 2011/10/5 Ram Rachum :
> > I am running PyPy from terminal, bash provided by msys. The error still
> > comes up in a dialog and the shell contains only the output from `nose`
> up
> > to the failure, with no word on the failure.
>
> Can you still see which test fails?
> and then add print statements to determine the exact location of the crash?
>
> --
> Amaury Forgeot d'Arc
>


I have hundreds of tests, and PyPy fails before a single one begins. It
seems that PyPy crashes somewhere in nose's initialization.

Isn't there a way to find the last Python line run before the crash without
stepping with a finer granularity every time?


Ram.
___
pypy-dev mailing list
pypy-dev@python.org
http://mail.python.org/mailman/listinfo/pypy-dev


Re: [pypy-dev] PyPy 1.6 not working on Windows XP

2011-10-05 Thread Amaury Forgeot d'Arc
2011/10/5 Ram Rachum :
> I am running PyPy from terminal, bash provided by msys. The error still
> comes up in a dialog and the shell contains only the output from `nose` up
> to the failure, with no word on the failure.

Can you still see which test fails?
and then add print statements to determine the exact location of the crash?

-- 
Amaury Forgeot d'Arc
___
pypy-dev mailing list
pypy-dev@python.org
http://mail.python.org/mailman/listinfo/pypy-dev


Re: [pypy-dev] PyPy 1.6 not working on Windows XP

2011-10-05 Thread Ram Rachum
On Wed, Oct 5, 2011 at 1:49 PM, Alex Pyattaev wrote:

> 1. Switch to linux. It helps.
>

Not funny.


2. To get a meaningful error log try to run the pypy from terminal. To do
> copy-paste you will need 3-rd party terminal, i.e. power shell. Then you'll
> be
> able to copy the error messages. Without them it is pretty much impossible
> to
> identify the problem.
>

I am running PyPy from terminal, bash provided by msys. The error still
comes up in a dialog and the shell contains only the output from `nose` up
to the failure, with no word on the failure.


3. Another option is to use debugger to run the suite step-by-step and see
> what happens.
>

I'll give that a try.


>
> Choose your poison.
> Alex.
> On Wednesday 05 October 2011 13:44:14 Ram Rachum wrote:
> > Hey guys,
> >
> > I upgraded to PyPy 1.6 on my 2 Windows XP 32 bit machines. It crashes on
> > both system when running the GarlicSim test suite.
> >
> > It shows a Windows error dialog saying "pypy.exe has encountered a
> problem
> > and needs to close.  We are sorry for the inconvenience." and giving this
> > data:
> >
> > AppName: pypy.exe AppVer: 0.0.0.0 ModName: kernel32.dll
> > ModVer: 5.1.2600.5512 Offset: 00040b0d
> >
> >
> > I can also open a dialog with a lot of data on the error (don't know how
> > useful it is) but Windows won't let me Ctrl-C it.
> >
> >
> > What can I do?
> >
> >
> > Thanks,
> > Ram.
> ___
> pypy-dev mailing list
> pypy-dev@python.org
> http://mail.python.org/mailman/listinfo/pypy-dev
>
___
pypy-dev mailing list
pypy-dev@python.org
http://mail.python.org/mailman/listinfo/pypy-dev


Re: [pypy-dev] PyPy 1.6 not working on Windows XP

2011-10-05 Thread Amaury Forgeot d'Arc
2011/10/5 Ram Rachum :
> Hey guys,
> I upgraded to PyPy 1.6 on my 2 Windows XP 32 bit machines. It crashes on
> both system when running the GarlicSim test suite.
> It shows a Windows error dialog saying "pypy.exe has encountered a problem
> and needs to close.  We are sorry for the inconvenience." and giving this
> data:
>
> AppName: pypy.exe AppVer: 0.0.0.0 ModName: kernel32.dll
> ModVer: 5.1.2600.5512 Offset: 00040b0d
>
> I can also open a dialog with a lot of data on the error (don't know how
> useful it is) but Windows won't let me Ctrl-C it.
>
> What can I do?

pypy is a console application, i.e. it opens a Console window.
You should be able to "Select all" and "Copy" all the content (click
on the top-left icon)

-- 
Amaury Forgeot d'Arc
___
pypy-dev mailing list
pypy-dev@python.org
http://mail.python.org/mailman/listinfo/pypy-dev


Re: [pypy-dev] PyPy 1.6 not working on Windows XP

2011-10-05 Thread Alex Pyattaev
1. Switch to linux. It helps.

2. To get a meaningful error log try to run the pypy from terminal. To do 
copy-paste you will need 3-rd party terminal, i.e. power shell. Then you'll be 
able to copy the error messages. Without them it is pretty much impossible to 
identify the problem.

3. Another option is to use debugger to run the suite step-by-step and see 
what happens.

Choose your poison.
Alex.
On Wednesday 05 October 2011 13:44:14 Ram Rachum wrote:
> Hey guys,
> 
> I upgraded to PyPy 1.6 on my 2 Windows XP 32 bit machines. It crashes on
> both system when running the GarlicSim test suite.
> 
> It shows a Windows error dialog saying "pypy.exe has encountered a problem
> and needs to close.  We are sorry for the inconvenience." and giving this
> data:
> 
> AppName: pypy.exe AppVer: 0.0.0.0 ModName: kernel32.dll
> ModVer: 5.1.2600.5512 Offset: 00040b0d
> 
> 
> I can also open a dialog with a lot of data on the error (don't know how
> useful it is) but Windows won't let me Ctrl-C it.
> 
> 
> What can I do?
> 
> 
> Thanks,
> Ram.
___
pypy-dev mailing list
pypy-dev@python.org
http://mail.python.org/mailman/listinfo/pypy-dev


[pypy-dev] PyPy 1.6 not working on Windows XP

2011-10-05 Thread Ram Rachum
Hey guys,

I upgraded to PyPy 1.6 on my 2 Windows XP 32 bit machines. It crashes on
both system when running the GarlicSim test suite.

It shows a Windows error dialog saying "pypy.exe has encountered a problem
and needs to close.  We are sorry for the inconvenience." and giving this
data:

AppName: pypy.exe AppVer: 0.0.0.0 ModName: kernel32.dll
ModVer: 5.1.2600.5512 Offset: 00040b0d


I can also open a dialog with a lot of data on the error (don't know how
useful it is) but Windows won't let me Ctrl-C it.


What can I do?


Thanks,
Ram.
___
pypy-dev mailing list
pypy-dev@python.org
http://mail.python.org/mailman/listinfo/pypy-dev


Re: [pypy-dev] PyPQ: a dbapi 2 PostgreSQL driver working with pypy

2011-10-05 Thread Igor Katson

So I've launched a few tests for all PostgreSQL python drivers
out there, and optimized PyPq a bit along the way.

*IntergerInsert* test does this 3 times (i wanted more, but pg8000 
bloats
postgres memory usage to 1,5 gigabytes on this somewhy, so i lowered the 
amount

of queries to 3)

cursor.executemany('insert into test_table values(%s, %s, %s, %s)', 
[(1,2,3,4)] * 3)


*IntegerSelect* selects this data back into python

*VariableDataInsert* does the same as IntegerInsert, but inserts a string, a
datetime, a date and a timestamp into the database (except for pg8000, 
which told

me that it did not support timestamps)

*VariableDataSelect* selects this data back into python

cPython 2.7.2 (32-bit, archlinux latest build), 3 inserts

Psycopg2IntegerInsert.test_insert took 1.78s
.Psycopg2IntegerSelect.test_select took 0.06s
.Psycopg2VariableDataInsert.test_insert took 2.57s
.Psycopg2VariableDataSelect.test_select took 0.25s

.Psycopg2ctIntegerInsert.test_insert took 4.46s
.Psycopg2ctIntegerSelect.test_select took 1.62s
.Psycopg2ctVariableDataInsert.test_insert took 6.00s
.Psycopg2ctVariableDataSelect.test_select took 3.31s

.PyPQIntegerInsertTest.test_insert took 3.41s
.PyPQIntegerSelectTest.test_select took 0.84s
.PyPQVariableDataInsertTest.test_insert took 4.07s
.PyPQVariableDataSelectTest.test_select took 3.70s

pg8000IntegerInsert.test_insert took 16.20s
.pg8000IntegerSelect.test_select took 1.43s
.pg8000VariableDataInsert.test_insert took 18.00s
.pg8000VariableDataSelect.test_select took 2.17s

PyPy 1.6.0 (32-bit, archlinux latest build), 3 inserts

Psycopg2ctIntegerInsert.test_insert took 2.69s
.Psycopg2ctIntegerSelect.test_select took 0.63s
.Psycopg2ctVariableDataInsert.test_insert took 4.53s
.Psycopg2ctVariableDataSelect.test_select took 1.36s

.PyPQIntegerInsertTest.test_insert took 4.61s
.PyPQIntegerSelectTest.test_select took 0.37s
.PyPQVariableDataInsertTest.test_insert took 4.48s
.PyPQVariableDataSelectTest.test_select took 1.58s

pg8000IntegerInsert.test_insert took 8.34s
.pg8000IntegerSelect.test_select took 0.60s
.pg8000VariableDataInsert.test_insert took 9.15s
.pg8000VariableDataSelect.test_select took 1.64s

As we can see, pg8000 is slow on inserts, and as i've said, it does some 
strange
things to my postgres, bloating the postgres memory usage to 1.5 
gigabytes (i tried

to insert 10 records with executemany)

On cPython, pypq is faster than psycopg2ct and pg8000, except for 
VariableDataSelect

test.
On PyPy, all of them get faster, except pypq, though it is still a bit 
faster

than psycopg2ct in 2 tests.

Next, I tested pypq side by side to see the difference more clearly.

Here are the results.

cPython 2.7.2 (32-bit, archlinux latest build), 20 inserts

Psycopg2IntegerInsert.test_insert took 12.22s
.Psycopg2IntegerSelect.test_select took 0.39s
.Psycopg2VariableDataInsert.test_insert took 17.30s
.Psycopg2VariableDataSelect.test_select took 1.71s

.Psycopg2ctIntegerInsert.test_insert took 28.56s
.Psycopg2ctIntegerSelect.test_select took 10.48s
.Psycopg2ctVariableDataInsert.test_insert took 38.53s
.Psycopg2ctVariableDataSelect.test_select took 21.67s

.PyPQIntegerInsertTest.test_insert took 22.53s
.PyPQIntegerSelectTest.test_select took 5.59s
.PyPQVariableDataInsertTest.test_insert took 26.86s
.PyPQVariableDataSelectTest.test_select took 24.84s

PyPy 1.6.0 (32-bit, archlinux latest build), 20 inserts

Psycopg2ctIntegerInsert.test_insert took 14.11s
.Psycopg2ctIntegerSelect.test_select took 3.18s
.Psycopg2ctVariableDataInsert.test_insert took 29.36s
.Psycopg2ctVariableDataSelect.test_select took 7.78s

.PyPQIntegerInsertTest.test_insert took 25.91s
.PyPQIntegerSelectTest.test_select took 1.92s
.PyPQVariableDataInsertTest.test_insert took 30.31s
.PyPQVariableDataSelectTest.test_select took 8.73s

On 10/05/2011 01:38 AM, Maciej Fijalkowski wrote:

On Tue, Oct 4, 2011 at 4:51 PM, Igor Katson  wrote:

Hi, Dan,
before answering I'll describe the situation a bit.

there was a question today, if I know about pg8000 or psycopg2ct.

As for pg8000, pure python should be slower than ctypes, anyway, so I don't
think these two should be compared.

[citation needed]


___
pypy-dev mailing list
pypy-dev@python.org
http://mail.python.org/mailman/listinfo/pypy-dev