Re: [pypy-dev] Slow sqlite user defined functions with pypy.
Thanks to all for your answers. I took some time to think some more about the results and: For the simple function (which returns 1), CPython roughly takes 1 sec and pypy 13 secs. IMHO, this case reveals pypy's callback overhead. For the complex function case, CPython roughly takes 6 secs and pypy 15 secs. If we assume that the overhead will be the same as in the simple function case, and subtract it, then CPython roughly took 5 secs to execute the complex function and pypy 2 secs. From the above, i think that my jit guess was wrong (as Alex said), and jit indeed works, but the overhead of the pypy callbacks is indeed quite large. I know that callbacks from C code aren't so frequent in Python programs, but my project: http://code.google.com/p/madis/ lives on them. So i would be glad if it could be solved (and madis's data processing would become a lot faster). Alternatively, is there something that i could do in my code that would overcome this overhead? Like signalling to the jit engine that this function should be always jitted and to do not waste time updating statistics (or other tracking information) on it? thanks lefteris. On 17/11/11 10:42, Antonio Cuni wrote: On 11/17/2011 02:56 AM, Alex Gaynor wrote: The JIT compiles functions without loops too now, so this should be jitted. ctypes callbacks still go through the old _rawffi, so it's possible that this introduces some unneeded overhead. ciao, Anto ___ pypy-dev mailing list pypy-dev@python.org http://mail.python.org/mailman/listinfo/pypy-dev ___ pypy-dev mailing list pypy-dev@python.org http://mail.python.org/mailman/listinfo/pypy-dev
Re: [pypy-dev] Slow sqlite user defined functions with pypy.
On 11/17/2011 02:56 AM, Alex Gaynor wrote: > The JIT compiles functions without loops too now, so this should be jitted. ctypes callbacks still go through the old _rawffi, so it's possible that this introduces some unneeded overhead. ciao, Anto ___ pypy-dev mailing list pypy-dev@python.org http://mail.python.org/mailman/listinfo/pypy-dev
Re: [pypy-dev] Slow sqlite user defined functions with pypy.
On Wed, Nov 16, 2011 at 8:24 PM, William ML Leslie < william.leslie@gmail.com> wrote: > Ack. > > On 17 November 2011 12:23, William ML Leslie > wrote: > > On 17 November 2011 12:13, Elefterios Stamatogiannakis > wrote: > >> Pypy seems to not jit at all when a (pypy) Python function is called > from C. > > > > Calls to native functions must be residualised, as there is no way to > > tell what state gives rise to the call to the UDF. > > > > If there was a loop inside the UDF, that would still be compiled. But > > the loop that occurs in the query must just call the C function in the > > usual way, as the JIT has no idea what it might do. > > -- > William Leslie > ___ > pypy-dev mailing list > pypy-dev@python.org > http://mail.python.org/mailman/listinfo/pypy-dev > The JIT compiles functions without loops too now, so this should be jitted. Alex -- "I disapprove of what you say, but I will defend to the death your right to say it." -- Evelyn Beatrice Hall (summarizing Voltaire) "The people's good is the highest law." -- Cicero ___ pypy-dev mailing list pypy-dev@python.org http://mail.python.org/mailman/listinfo/pypy-dev
Re: [pypy-dev] Slow sqlite user defined functions with pypy.
Ack. On 17 November 2011 12:23, William ML Leslie wrote: > On 17 November 2011 12:13, Elefterios Stamatogiannakis > wrote: >> Pypy seems to not jit at all when a (pypy) Python function is called from C. > > Calls to native functions must be residualised, as there is no way to > tell what state gives rise to the call to the UDF. > > If there was a loop inside the UDF, that would still be compiled. But > the loop that occurs in the query must just call the C function in the > usual way, as the JIT has no idea what it might do. -- William Leslie ___ pypy-dev mailing list pypy-dev@python.org http://mail.python.org/mailman/listinfo/pypy-dev
[pypy-dev] Slow sqlite user defined functions with pypy.
The following code is a lot slower with pypy as compared to CPython. The
code mainly measures the time taken to execute a simple SQLite user
defined function (UDF) and a more complex one, 100 times each.
Execution time for both queries is:
CPython 2.7: 7 sec 489 msec
Pypy nightly build: 28 sec 753 msec
Execution time for simple query only is:
CPython 2.7: 1 sec 39 msec
Pypy nightly build: 13 sec 645 msec
Pypy seems to not jit at all when a (pypy) Python function is called from C.
Also based on the simple query times, pypy seems to have massive
overhead (nearly 13 times slower) when executing callbacks from C code.
lefteris.
--- code --
import sqlite3
import datetime
# Simple function
def testfun(*args):
return 1
# Complex function
def sectohuman(*args):
secs=int(args[0])
h=''
days=secs/86400
if days > 0:
h+=str(days)+' day'
if days > 1:
h+='s'
h+=' '
secs=secs % 86400
hours=secs/3600
if hours > 0:
h+=str(hours)+' hour'
if hours > 1:
h+='s'
h+=' '
secs=secs % 3600
mins=secs/60
if mins > 0:
h+=str(mins)+' min '
secs=secs % 60
if secs > 0:
h+=str(secs)+' sec'
return h
con=sqlite3.Connection('')
con.create_function('testfun', -1, testfun)
con.create_function('sectohuman', -1, sectohuman)
cur=con.cursor()
cur.execute('create table l(a);')
cur.execute('begin;')
for i in xrange(100):
cur.execute('insert into l values(?)',(i,))
before=datetime.datetime.now()
# Simple query
a=list(cur.execute('select sum(testfun(a)) as s from l'))
# Complex query
a=list(cur.execute('select sum(length(sectohuman(a))) as s from l'))
after=datetime.datetime.now()
tmdiff=after-before
print "Execution time is %s min. %s sec %s msec"
%((int(tmdiff.days)*24*60+(int(tmdiff.seconds)/60),(int(tmdiff.seconds)%60),(int(tmdiff.microseconds)/1000)))
___
pypy-dev mailing list
pypy-dev@python.org
http://mail.python.org/mailman/listinfo/pypy-dev
