Re: [pypy-dev] docs on doc.pypy.org

[Antonio Cuni]

On 03/05/11 11:55, holger krekel wrote:
> Hi all,
> 
> i created a CNAME for our use of pypy.readthedocs.org and so you can now
> reach pypy docs under the frontend URL of
> 
> http://doc.pypy.org
> 
> It might be better to promote this link also from pypy.org so
> that URLs have a higher chance to remain valid even if we
> decide to move away from readthedocs some day.

yes, good idea.
Moreover, I think we should change the link from "Dev Documentation" to just
"Documentation", because it contains docs which are generally useful also for
users (e.g., the getting started, the papers, etc.)

Alternatively, we could have a heavier refactoring of the documentation and
have clearly separated sections for "user docs" and "dev docs".

> Moreover, we could see to have (another) sphinx setup for sphinx 
> setup for our current pypy.org so we can host it there as well, maybe
> under www.pypy.org -- we need to have a subdomain/CNAME as we cannot
> redirect the root domain to readthedocs.

I'm not a sphinx expert, but I don't think that the default layout (i.e., with
a sidebar on the left) is the best for the home page of pypy.org.

ciao,
Anto
___
pypy-dev@codespeak.net
http://codespeak.net/mailman/listinfo/pypy-dev

Re: [pypy-dev] cpyext: Detecting pypy and other issues

[Antonio Cuni]

Hello Roger,
I'm not "the" cpyext expert here, so I'll let the others to answer your
specific questions. However:

On 02/05/11 06:55, Roger Binns wrote:

> Unfortunately disabling the JIT wasn't much help.  My goal is to produce the
> best bug report possible and use the least amount of the pypy team's time.
> (It is a lot easier to fix and understand things starting from a good bug
> report.)  What would be most helpful is if the cpyext page gave some
> instructions to follow in particular making an appropriate pypy and using
> valgrind.  (Usually making valgrind work well requires extra options above
> and beyond regular debugging builds.)

you are right, we lack such a document. Do you feel like writing it? :-)
(seriously: you are by far in a better position than us for knowing what could
or could not help a newcomer.  Even if you don't know the answer of all the
questions, having a document with XXX that we can fix would be valuable).

ciao,
Anto
___
pypy-dev@codespeak.net
http://codespeak.net/mailman/listinfo/pypy-dev

Re: [pypy-dev] [pypy-svn] pypy default: fixed test_circular

[Antonio Cuni]

On 15/04/11 10:50, Hakan Ardo wrote:
> Hi,
> the point here is that we want max(a,b) to be turned into a single
> guard while we dont want max(*range(300)) and max(range(300)) to blow
> up into 300 guards, since that might lead to 2**300 different traces.
> I'm not sure how to best test this...

can't we just check that the loop contains a residual call to min_max_loop?
___
pypy-dev@codespeak.net
http://codespeak.net/mailman/listinfo/pypy-dev

Re: [pypy-dev] [pypy-svn] pypy default: fixed test_circular

[Antonio Cuni]

> Right. My point was that since we dont care if they are there or not
> the test should not test that they are there and fail if they are not.
> So if there is an easy way to ignore them in this new test_pypy_c
> framework (which is very cool by the way), we should. If it's not easy
> I'm fine with keeping the test as it is. My main motivation here is to
> learn about the new framework :)

ah, I understand now.
No, ignoring all force_tokens at once is not possible at the moment,
but I agree that it would be a nice feature, I think I'll implement it
later.

Btw, I fear I need more of your help with test_silly_max and
test_iter_max (see 2e5bd737be0c): what do we want to check there?
___
pypy-dev@codespeak.net
http://codespeak.net/mailman/listinfo/pypy-dev

Re: [pypy-dev] [pypy-svn] pypy default: fixed test_circular

[Antonio Cuni]

Hi Hakan,
thanks for the commits

> +# We want to check that the array bound checks are removed,
> +# so it's this part of the trace. However we dont care about
> +# the force_token()'s. Can they be ignored?

yes, I think they can be just ignored, because AFAIK operations without side
effects and whose result is unused, are removed by the backend regalloc.

ciao,
Anto
___
pypy-dev@codespeak.net
http://codespeak.net/mailman/listinfo/pypy-dev

Re: [pypy-dev] [pypy-svn] pypy default: port test_intbound_addsub_ge to test_pypy_c_new

[Antonio Cuni]

On 14/04/11 16:44, Hakan Ardo wrote:
> Second though, that will recompile on every call, but if we cache the
> promote functions:
> 
> def main(n, promoters={}):
> i, a = 0, 0
> if n not in promoters:
> exec """def promote(n):
> assert n==%d""" % n
> promoters[n] = promote
> else:
> promote = promoters[n]
> while i < n:
> promote(n)
> a += i+5
> i += 1
> return a
> 
> we will actualy get rid of the extra ptr_eq and guard_false too...

wow, that's advanced... and without knowing the internals of the JIT, it
really looks like black magic.

While we are at it and if you have time/feel like, could you please have a
look to test_zeropadded and test_circular in test_pypy_c_new? It's not clear
to me what they are supposed to check (note that it's fine if you say "they
just check that the program works correctly" :-)).

___
pypy-dev@codespeak.net
http://codespeak.net/mailman/listinfo/pypy-dev

Re: [pypy-dev] [pypy-svn] pypy default: port test_intbound_addsub_ge to test_pypy_c_new

[Antonio Cuni]

Hi Hakan,

On 14/04/11 14:53, Hakan Ardo wrote:

>> +def test_intbound_addsub_ge(self):
>> +def main(n):
>> +i, a, b = 0, 0, 0
>> +while i < n:
>> +if i + 5 >= 5:
>> +a += 1
>> +if i - 1 >= -1:
>> +b += 1
>> +i += 1
>> +return (a, b)
>> +#
>> +log = self.run(main, [300], threshold=200)
>> +assert log.result == (300, 300)
>> +loop, = log.loops_by_filename(self.filepath)
>> +assert loop.match("""
>> +i10 = int_lt(i8, i9)
>> +guard_true(i10, descr=...)
>> +# XXX: why do we need ovf check here? If we put a literal "300"
>> +# instead of "n", it disappears
> 
> With n==sys.maxint, the operation i+5 would be the one overflowing.

yes, you are right of course.  I was just thinking nonsense, but I realized
only after I asked the question :-).

Of course the ovf check needs to be there because we don't specialize the loop
on the value of n. Although it might be cool to be able to do promotion at
applevel, for those who really want :-)

ciao,
Anto
___
pypy-dev@codespeak.net
http://codespeak.net/mailman/listinfo/pypy-dev

Re: [pypy-dev] Possible sprint in Genova before/after Europython

[Antonio Cuni]

On 13/04/11 14:53, Jacob Hallén wrote:
> Wednesday 13 April 2011 you wrote:
>> In a message of Wed, 13 Apr 2011 12:20:26 +0200, Antonio Cuni writes:
>>> Also, would you prefer to do it before or after europython?
>>
>> I am coming, and both times are fine for me.  I am going to be
>> spending the week after Europython in Italy somewhere anyway, prior
>> to going kayaking in Sicily.
> 
> Corsica, not Sicily.

uhm, how do you go there? If by boat, it's very likely that you'll start by
genova or savona (which is also close, ~50 km)
___
pypy-dev@codespeak.net
http://codespeak.net/mailman/listinfo/pypy-dev

Re: [pypy-dev] Possible sprint in Genova before/after Europython

[Antonio Cuni]

On 13/04/11 12:27, Maciej Fijalkowski wrote:

> Not that I can't google myself, but those links are broken

uhm, indeed. These seems to work :-)

http://bit.ly/e6vHkh
http://bit.ly/fkHwMu
___
pypy-dev@codespeak.net
http://codespeak.net/mailman/listinfo/pypy-dev

[pypy-dev] Possible sprint in Genova before/after Europython

[Antonio Cuni]

Hi all,

as we were discussing yesterday in another thread, the post-europython sprint
will be only two days long, and so we might want to have a longer one either
before or after europython.

I am considering organizing one in my place.  It could be either in Genova or
more preferably in some other town in the nearby of the italian riviera.
The place is ~3hr away from Florence by train.

Googling images for "pegli" or "arenzano" (i.e., two of the towns we could go
to) shows pictures which (I think) should encourage people to come :-)
http://tinyurl.com/655p2at
http://tinyurl.com/67q67r3

My idea is to find a hotel that gives us a sprinting room and internet in
exchange of N people sleeping there, as we do in leysin.

But before asking them, I need to have a rough idea about which value of "N"
we are talking about (of course the higher the more interesting for them it is).

Thus, I ask everybody who is potentially/likely interested in coming to let me
know.

Also, would you prefer to do it before or after europython?

ciao,
Anto
___
pypy-dev@codespeak.net
http://codespeak.net/mailman/listinfo/pypy-dev

Re: [pypy-dev] thinking about the EuroPython sprint

[Antonio Cuni]

On 12/04/11 08:52, Armin Rigo wrote:

> Then what about a short week's sprint before the conference, plus the
> two days after?

if people are interested, I might try to organize a sprint in Genova or
surrounding (which is ~3h away from florence by train).

However, I cannot assure that I'll be able to find a suitable place, because I
see only two options:

- ask the uni (but there are not really many rooms suitable for a sprint --
just one or two, and I think it'll be hard to reserve them for a whole week)

- find a place "a la leysin", i.e. a hotel which gives also us a room for the
sprint.  However, middle of june is already vacation time here, so it is
possible that hotels are full anyway and prefer to have "normal" visitors
instead of nerds which stay there all the time :-)

ciao,
Anto
___
pypy-dev@codespeak.net
http://codespeak.net/mailman/listinfo/pypy-dev

Re: [pypy-dev] The JVM backend and Jython

[Antonio Cuni]

Hi Frank,

On 30/03/11 04:40, fwierzbi...@gmail.com wrote:
[cut]
> So to my question - just how broken is the JVM backend? Are there
> workarounds that would allow the Java code to get generated? 

so, now the jvm (and cli) translation works again. You can just type
./translate.py -b jvm, and the fish the relevant .class/.j files from
/tmp/usession-default-*/pypy.

ciao,
Anto
___
pypy-dev@codespeak.net
http://codespeak.net/mailman/listinfo/pypy-dev

Re: [pypy-dev] Pre sprint beer session Sunday 24th of April

[Antonio Cuni]

On 11/04/11 14:47, Bea During wrote:

> I still would like to meet up so I invite you to a pre sprint beer 
> session at Bishops Arms pub in central Gothenburg, Sunday 24th of April, 
> starting from 18:00.
[cut]
> p.s. Anto - will miss you there d.s

yeah, unfortunately I won't be able to make it :-(
Have fun!

ciao,
Anto
___
pypy-dev@codespeak.net
http://codespeak.net/mailman/listinfo/pypy-dev

Re: [pypy-dev] Pypy custom interpreter JIT question

[Antonio Cuni]

On 04/04/11 19:46, Andrew Brown wrote:

> 1) do you know about the existence of rlib.streamio? It's is part of the
> "RPython standard library" and it allows you to read/write files in a 
> higher
> level way than file descriptors
> 
> No, I didn't. That's good to know. I don't think it's worth updating the
> examples though, so unless you disagree, I'll just add a note about this
> module's existence.

sure, I think that for this example, using fd is fine.

Btw, in case you want to do more with pypy, having a look to rlib might be a
good idea, there is useful stuff there :)

ciao,
Anto
___
pypy-dev@codespeak.net
http://codespeak.net/mailman/listinfo/pypy-dev

Re: [pypy-dev] Pypy custom interpreter JIT question

[Antonio Cuni]

On 04/04/11 17:43, Andrew Brown wrote:

> Sure! What needs to be done to turn it into a blog post and get it posted? I
> assume there are format considerations, but I'm also open to any content
> suggestions and feedback before it "goes live".

Hello Andrew,
thanks for the tutorial, it's really well written and easy to read.

Two notes:

1) do you know about the existence of rlib.streamio? It's is part of the
"RPython standard library" and it allows you to read/write files in a higher
level way than file descriptors

2) Maybe the tutorial is a bit too long to fit in just one post; what about
splitting it into two parts? (e.g., one until "Adding JIT" and one after).

ciao,
Anto
___
pypy-dev@codespeak.net
http://codespeak.net/mailman/listinfo/pypy-dev

Re: [pypy-dev] [PATCH] fix executing a file under sandbox by implementing fstat

[Antonio Cuni]

Hello Seth,
thank you for your patch!

On 01/04/11 02:27, Seth de l'Isle wrote:
[cut]
> The following patch changes the code so that it tracks the virtual
> file system nodes that correspond to each virtual file descriptor so
> that the node.stat() function can
> be used for fstat the same way it is used for stat.

could you please write the corresponding test in test_sandlib.py please?


> +def do_ll_os__ll_os_fstat(self, fd):
> +node = self.fd_to_node[fd]
> +return node.stat()

Also, you probably need to handle the case in which we call fstat with a fd
which doesn't exist (and write the corresponding test :-))

ciao,
Anto
___
pypy-dev@codespeak.net
http://codespeak.net/mailman/listinfo/pypy-dev

Re: [pypy-dev] The JVM backend and Jython

[Antonio Cuni]

On 31/03/11 21:57, Maciej Fijalkowski wrote:

>> Ok, so if Ademan tells me that he's not going to work on the 
>> ootype-virtualref
>> branch, I'll try to finish the work so you can start playing with it.
> 
> Note to frank: this is kind of cool but only needed for the JIT,
> otherwise it's a normal reference.

well, no. Virtualrefs were introduced for the JIT, but they also need to be
supported by normal backends.  This is why translation is broken at the moment.

It is true that the implementation is straightforward, though (I suppose this
is what you meant originally :-))

ciao,
Anto
___
pypy-dev@codespeak.net
http://codespeak.net/mailman/listinfo/pypy-dev

Re: [pypy-dev] Pypy custom interpreter JIT question

[Antonio Cuni]

On 31/03/11 22:05, Andrew Brown wrote:

> python double-interpreted: > 78m (did not finish)
> pypy-c (with jit) double-interpreted: 41m 34.528s

this is interesting. We are beating cpython by more than 2x even in a "worst
case" scenario, because interpreters in theory are not a very good target for
tracing JITs.
However, it's not the first time that we experience this, so it might be that
this interpreter/tracing JIT thing is just a legend :-)

> translated interpreter no jit: 45s
> translated interpreter jit: 7.5s
> translated direct to C, gcc -O0
>   translate: 0.2s
>   compile: 0.4s
>   run: 18.5s
> translated direct to C, gcc -O1
>   translate: 0.2s
>   compile: 0.85s
>   run: 1.28s
> translated direct to C, gcc -O2
>   translate: 0.2s
>   compile: 2.0s
>   run: 1.34s

these are cool as well. We are 3x faster than gcc -O0 and ~3x slower than -O1
and -O2.  Pretty good, I'd say :-)

ciao,
anto
___
pypy-dev@codespeak.net
http://codespeak.net/mailman/listinfo/pypy-dev

Re: [pypy-dev] Pypy custom interpreter JIT question

[Antonio Cuni]

On 31/03/11 14:28, Andrew Brown wrote:
> In any case, I'm satisfied with the speed. It's still beaten by a BF to C
> translator combined with gcc -O2 though, that'd be a tough case to beat. =)

what happens if you combine the BF to C with gcc -O0 or -O1?

Anyway, I think that if you feel like writing a post explaining your
experience with using pypy and its jit for writing an interpreter, we could
publish it on our blog.  I suppose it would be useful/interesting for other
people as well.

What do the others think?
___
pypy-dev@codespeak.net
http://codespeak.net/mailman/listinfo/pypy-dev

Re: [pypy-dev] The JVM backend and Jython

[Antonio Cuni]

On 30/03/11 19:37, fwierzbi...@gmail.com wrote:

> My thoughts here are taking a very primitive step - that is run the
> JVM translation and look at the generated Java - then see what needs
> to be modified so that I could use the generated Java parser from
> Jython. At this stage I would be using PyPy exactly the way I use
> ANTLR now - as a parser generator. There wouldn't be any need at all
> for calling into Java code (as far as I can think of). 

yes, I think it makes sense.
Actually, as Leonardo says we don't generate java code but assembler which is
converted to .class by jasmin. However, it should not change anything.

> I think if we
> Jython developers get some experience with PyPY - we might be able to
> help with the task of calling into Java from PyPy - since we know a
> bit about that :)

that would be extremely cool :-)

Ok, so if Ademan tells me that he's not going to work on the ootype-virtualref
branch, I'll try to finish the work so you can start playing with it.

ciao,
Anto
___
pypy-dev@codespeak.net
http://codespeak.net/mailman/listinfo/pypy-dev

Re: [pypy-dev] The JVM backend and Jython

[Antonio Cuni]

Hi Frank,

On 30/03/11 04:40, fwierzbi...@gmail.com wrote:
> Hi all,
> 
> It was nice meeting up with many of you at PyCon!
> 
> I've been thinking about the first steps towards collaboration between
> the Jython project and the PyPy project. It looks like it isn't going
> to be too long before we are all (CPython, PyPy, IronPython, Jython,
> etc) working on a single shared repository for all of our standard
> library .py code. In my ideal world there would come a day when there
> is also no standalone Java code in the Jython project: that is the
> shared standard library would contain all of Jython's .py files, and
> all of the Java would be generated from PyPy and Jython as a
> standalone project would disappear. It is possible that this is too
> ambitious, but big goals are more fun, right? In reality even if this
> where to get going, I imagine it would be a 10+ year plan :)


wow, that's definitely a nice (and big) plan :-)

> So to my question - just how broken is the JVM backend? Are there
> workarounds that would allow the Java code to get generated? 

"not much broken".
Last time I tried, the only broken thing was "virtualrefs", which is something
needed for the jit, but that at the moment is not supported at all by ootype
and thus it blocks the translation.

However, I think that fixing it is probably very easy: there is a branch for
this which was started by Ademan:
https://bitbucket.org/pypy/pypy/src/ootype-virtualrefs

Dan, do you plan to finish the work on it? Else, I can just do it probably.

> I ask
> because I would like to evaluate the generated Java parser as a
> potential replacement for our current ANTLR based parser. It seems
> like a nice baby step towards real collaboration since it seems like a
> relatively easy place to start. Clearly it would need adjustments to
> actually work for Jython - but I'd be able to look into that part. I
> don't think I have the time to try to unbreak the translation
> though...

I have to warn you that at the moment, you cannot invoke any Java code from
RPython.  Implementing it has been on my todo list for years now :-(, but I
never managed to find the time and the motivation to do it.  However, for
using the PyPy parser inside Jython it should be enough to do the other way
around, i.e. call RPython code from Java, which should be possible.

ciao,
Anto
___
pypy-dev@codespeak.net
http://codespeak.net/mailman/listinfo/pypy-dev

[pypy-dev] Hacking dropbox

[Antonio Cuni]

Hi,

I don't have any news from the dropbox side, but as an aside note, I found a
project which is open source and implements the very same technique I used for
dropbox, and additionally it also decompiles back to the source code:

http://code.google.com/p/pyretic/

ciao,
Anto
___
pypy-dev@codespeak.net
http://codespeak.net/mailman/listinfo/pypy-dev

Re: [pypy-dev] When translating pypy, how do I notify it of library locations?

[Antonio Cuni]

Hi Alexander,

On 23/03/11 16:48, Alexander Golec wrote:
> I'm building pypy with the default JIT configuration, and I'm on a machine 
> where I do not have root access. I've installed the libraries that I need in 
> my ~/lib folder, but I can't get pypy to see them. I can neither see the 
> library files, nor the headers. 

which libraries are you talking about? Pure python modules/packages (i.e., the
ones contained in lib-python and lib_pypy) or shared libraries?

ciao,
Anto
___
pypy-dev@codespeak.net
http://codespeak.net/mailman/listinfo/pypy-dev

Re: [pypy-dev] status of the graphviz server on codespeak?

[Antonio Cuni]

On 23/03/11 02:36, Maciej Fijalkowski wrote:
> On Tue, Mar 22, 2011 at 7:17 PM, Laura Creighton  wrote:
>> getting-started-dev says:
>>
>> Download and install `Dot Graphviz`_ (optional if you have an internet
>>connection: the flowgraph viewer then connects to
>>codespeak.net and lets it convert the flowgraph by a graphviz server).
>>
>> What's going to happen to that when codespeak goes away?
> 
> This one is defunct for ages now I think.

no, I think it's still up:
http://codespeak.net/pypy/convertdot.cgi

we can either migrate it to some other machine (tannit?) or discontinue the
service. Nowadays, it's only useful for us developers, and (I think) we all
have graphviz installed locally.

ciao,
Anto
___
pypy-dev@codespeak.net
http://codespeak.net/mailman/listinfo/pypy-dev

Re: [pypy-dev] [pypy-svn] pypy fold_intadd: Changed test to reflect my optimizeopt's decision to emit int_sub(iX, -x) when x < 0

[Antonio Cuni]

Hi Daniel,

On 19/03/11 04:45, ademan wrote:
> Author: Daniel Roberts 
> Branch: fold_intadd
> Changeset: r42802:386510d0fb45
> Date: 2011-03-18 20:41 -0700
> http://bitbucket.org/pypy/pypy/changeset/386510d0fb45/
> 
> Log:  Changed test to reflect my optimizeopt's decision to emit
>   int_sub(iX, -x) when x < 0

what happens when you are on 32 bit and see int_add(i0, -2147483648)?

ciao,
Anto
___
pypy-dev@codespeak.net
http://codespeak.net/mailman/listinfo/pypy-dev

Re: [pypy-dev] ideas for Google Summer of code

[Antonio Cuni]

Hi Wim,

On 17/03/11 19:33, wlavrij...@lbl.gov wrote:
> Hi Anto,
> 
>> -  :-)
> 
> the Reflex work has at one time before been proposed as a GSoC. Now that
> a prototype is in place, there are several minor tasks that can be done,
> which can lead to bigger/more research tasks as desired/appropriate.

good idea!

> How does this work anyway, do you need to come with your own student?

not necessarily. At this stage, the goal is to collect ideas which are
reasonable, then publish it and see which students are interested in them.
However, nothing stops you to suggest a student of course (especially if you
already know him and you are confident that can do the job well).

ciao,
Anto
___
pypy-dev@codespeak.net
http://codespeak.net/mailman/listinfo/pypy-dev

Re: [pypy-dev] ideas for Google Summer of code

[Antonio Cuni]

On 17/03/11 21:25, Dan Stromberg wrote:
> 
> On Thu, Mar 17, 2011 at 10:23 AM, Leonardo Santagada  > wrote:
> 
> 7 - make pypy on .net jit work (or on java)

this is probably a too large task for GSoc. For one, before working on the JIT
it is necessary to make normal translation working, because right now ootype
backends are broken.  IIRC, it's not too hard because it consists of porting
virtualrefs to ootype, which should be simple. But it is possible that there
are other issues after that, since ootype translations have not run since a
long time (more than 1 year, I think).

About the JIT: JIT on .NET is not going to be any fast.  I did it for my
thesis when the JIT was more .NET friendly (no virtualrefs) and results were
interesting as a research project, but not good enough to be used in production.

The JIT for pypy-jvm is an open topic: the JVM has the potential to do a much
better job than the CLI, but we cannot know until we really try.

Having a working pypy-jvm-jit is a lot of work, though. It consists of:

1) make pypy-jvm (without jit) working again

2) design and implement a way to use Java objects at RPython level: this is
needed to write the backend

3) port the JIT to ootype again. Should not be too hard, but there are going
to be issues, because the codewriter has been heavily refactored since last
year, when the JIT on ootype worked well

4) write the backend

All together, it's probably huge for GSoc. But e.g 1+2 could fit it; we
already had a GSoc on this topic few years ago, but it didn't work well.

> 
> This reminds me: it might be good to make the JVM PyPy be able to call native
> Java code - on a typical JRE, and on Android.  Last

yes, that's another interesting topic. It requires both points 1 and 2 above,
though. Once we have that, it should not be too hard, as it has already been
implemented for .NET and could use the same techniques (and probably reuse
also most of the code).

ciao,
Anto
___
pypy-dev@codespeak.net
http://codespeak.net/mailman/listinfo/pypy-dev

[pypy-dev] ideas for Google Summer of code

[Antonio Cuni]

Hi all,

the deadlines for GSoc are approaching, and at some point we should probably
make a blog post about that.

But first, we need to 1) collect ideas for possible tasks and 2) find
potential mentors.

Two ideas that just came to my mind:

- "general work" on speed.pypy.org (we need to define better what we want, of
course)

- improving the jitviewer, maybe integrating it with the profiler (when we'll
have one :-))

-  :-)

ciao,
Anto
___
pypy-dev@codespeak.net
http://codespeak.net/mailman/listinfo/pypy-dev

Re: [pypy-dev] problem after merging of jit-virtual_state

[Antonio Cuni]

Hi Hakan,
thank you for the deep explanation. Now I understand what's going on :-)

So, I changed test_pypy_c_new to add a sys.setcheckinterval(some-huge-number),
so that the bridge from the signal/thread counter is never created and we can
forget about it.

Now, if I understand correctly, the two remaining loops are one for the case
"i non virtual" and the other for the case "i virtual", although both lead to
the same operations. I think this is the expected behavior in this case, so
are you ok if I just fix test_f1 to expect two loops?

ciao,
Anto

On 13/03/11 11:12, Hakan Ardo wrote:
> Hi,
> this is what happens here:
> 
> 1. The inner loop is traced and Loop0 is produced with preamble Loop1
> 
> 2. A bridge from Guard3 (the test in the while) back to Loop0 is
> traced (i.e the remaining parts of the outer loop)
> 
> 3. At the end of this bridge the VirtualState does not match the
> VirtualState of Loop0, so the loop is retraced
> 
> 4. The VirtualState of the newly traced version of the loop does not
> match the VirtualState at the end of the bridge so the bridge has to
> jump to the preamble instead of jumping to the new specialized version
> of the loop.
> 
> 5. A bridge from Guard6 (signal/thread counter) is traced and the same
> thing happens for this bridge.
> 
> This means that the additional two versions of the loop will never be
> used and should hopefully be reomved by the gc...
> 
> So there are two issues:
> 
> A. The additional specialized versions created does not become usable.
> This is the issue I'm working on in the jit-usable_retrace branch. The
> idea there is to have the retrace inherit the OptValue's of the
> jumpargs at the end of the bridge. This will become a fairly large
> change functionality wise...
> 
> B. The VirtualStates' s differs in the first place forcing a retrace.
> This is probably fixable by introducing some more cases in
> NotVirtualInfo._generate_guards(). The jit-usable_retrace branch
> contains more cases than trunk, don't know if those are enough for
> this test though...
> 
> Note however that
> jit/metainterp/test/test_nested_loops_discovered_by_bridge in
> test_loop_unroll.py, which conatins the same loop for a simple
> interpreter, does work nicely, wihtout the issues above.
> 
> On Sat, Mar 12, 2011 at 10:59 PM, Hakan Ardo  wrote:
>> On Sat, Mar 12, 2011 at 8:34 PM, Antonio Cuni  wrote:
>>> Hi Hakan,
>>>
>>> On 12/03/11 19:25, Hakan Ardo wrote:
>>>> Yes, this is probably the VirtualState checking. It will retrace a
>>>> loop whenever the VirtualState at the end of a bridge differs from the
>>>> VirtualState at the beginning of the compiled trace (any of the
>>>> compiled traces). This might indeed produce an identical trace if we
>>>> are unlucky, but the idea is that this should only happen rarely.
>>>
>>> ok, that's clear. So, hopefully this particular example looks a bit bad, but
>>> in general it should not be an issue. It'd be nice to have a way to check 
>>> this
>>> thesis, but I agree that it's a bit hard.
>>
>> We should probably log the VirtualState together with the produced
>> loops and bridges. That would allow us to see how they differ when a
>> new version of a loop is traced. There are __repr__ methods I've been
>> using for that while debugging. They might need some rpythonizing to
>> translate though---
>>
>>>
>>>> This is because the VirtualState  at the beginning of a trace is the
>>>> state of all the OptValue of the inputargs produced during the
>>>> optimization of the trace. This does not have to be the most general
>>>> state for which the trace is usable (which would be hard to calculate
>>>> I'm afraid).
>>>
>>> so, if I understand correctly, this is what happens:
>>>
>>> 1. we trace, optimize and compile loop A
>>>
>>> 2. after a while, we trace, optimize a compile a bridge B which then jumps
>>> back to A; by chance, the bridge looks the same as the loop
>>>
>>> Am I right?
>>
>> Maybe, I've not had the chance to look into any details yet. I'll do
>> that tomorrow...
>>
>>>
>>>> A few cases that would (most likely) result in identical traces are
>>>> salvaged in NotVirtualInfo._generate_guards by producing some extra
>>>> gurads at the end of a bridge to make the VirtualState there match the
>>>> VirtualState of a compiled trace. This is however only done if the
>>>> guards would (most likely) not fail for t

Re: [pypy-dev] problem after merging of jit-virtual_state

[Antonio Cuni]

Hi Hakan,

On 12/03/11 19:25, Hakan Ardo wrote:
> Yes, this is probably the VirtualState checking. It will retrace a
> loop whenever the VirtualState at the end of a bridge differs from the
> VirtualState at the beginning of the compiled trace (any of the
> compiled traces). This might indeed produce an identical trace if we
> are unlucky, but the idea is that this should only happen rarely.

ok, that's clear. So, hopefully this particular example looks a bit bad, but
in general it should not be an issue. It'd be nice to have a way to check this
thesis, but I agree that it's a bit hard.

> This is because the VirtualState  at the beginning of a trace is the
> state of all the OptValue of the inputargs produced during the
> optimization of the trace. This does not have to be the most general
> state for which the trace is usable (which would be hard to calculate
> I'm afraid).

so, if I understand correctly, this is what happens:

1. we trace, optimize and compile loop A

2. after a while, we trace, optimize a compile a bridge B which then jumps
back to A; by chance, the bridge looks the same as the loop

Am I right?

> A few cases that would (most likely) result in identical traces are
> salvaged in NotVirtualInfo._generate_guards by producing some extra
> gurads at the end of a bridge to make the VirtualState there match the
> VirtualState of a compiled trace. This is however only done if the
> guards would (most likely) not fail for the traced iteration.
> 
> I'll look into what's happening in this particular test...

I just did a quick check because I'm in a hurry, but from what I see we get
three actual *loops*, not bridges.

ciao,
Anto
___
pypy-dev@codespeak.net
http://codespeak.net/mailman/listinfo/pypy-dev

Re: [pypy-dev] problem after merging of jit-virtual_state

[Antonio Cuni]

On 10/03/11 16:21, Antonio Cuni wrote:

> I did a bit of manual binary search and discovered that the test started
> failing between r42273:b8bb5d085684 and r42305:3fe93dd500bd. Looking at the
> commit, it seems that the only one relevant to the problem is
> r42305:3fe93dd500bd, i.e. the merging of jit-virtual_state.

I confirm that the problem is the merging of jit-virtual_state: I just tried
the revision just before the merge, and the test passes.

ciao,
Anto
___
pypy-dev@codespeak.net
http://codespeak.net/mailman/listinfo/pypy-dev

[pypy-dev] problem after merging of jit-virtual_state

[Antonio Cuni]

Hi Hakan, hi all,

as you might have noticed, there is test_pypy_c_new.test_f1 which is currently
failing:

http://buildbot.pypy.org/summary/longrepr?testname=TestPyPyCNew.%28%29.test_f1&builder=pypy-c-jit-linux-x86-32&build=699&mod=pypy.module.pypyjit.test_pypy_c.test_pypy_c_new

buildbot did not show it earlier because before yesterday it did not run the
test at all. As you can see from the traceback, the problem is that running f1
makes three loops instead of one as expected.

I did a bit of manual binary search and discovered that the test started
failing between r42273:b8bb5d085684 and r42305:3fe93dd500bd. Looking at the
commit, it seems that the only one relevant to the problem is
r42305:3fe93dd500bd, i.e. the merging of jit-virtual_state.

Do you have any idea why jit-virtual_state causes this problem? (if it's a
problem at all, but I think so because the three loops contain the very same
operations, so I don't see why we need to compile the more than once).

ciao,
Anto
___
pypy-dev@codespeak.net
http://codespeak.net/mailman/listinfo/pypy-dev

Re: [pypy-dev] [PATCH] Fix segmentation fault on parsing empty list-assignments

[Antonio Cuni]

On 09/03/11 13:29, Greg Price wrote:
> This same patch is on bitbucket at https://bitbucket.org/price/pypy,
> where I've sent a pull request. Holger Krekel suggested on IRC that I
> send mail here. If others have different preferences for how to submit
> a patch, let me know.

Hi Greg,
I have pushed your commit upstream, thanks for help!

ciao,
anto
___
pypy-dev@codespeak.net
http://codespeak.net/mailman/listinfo/pypy-dev

Re: [pypy-dev] we need to fix the links on our blog posts.

[Antonio Cuni]

you can get a link to "whatever the latest revision is" on bitbucket
by manually substituting the mercurial id with "default" inside the
link.

On 3/3/11, Armin Rigo  wrote:
> Hi Laura,
>
> On Thu, Mar 3, 2011 at 8:37 AM, Laura Creighton  wrote:
>> Do we need dcolish's bouncer to work for what used to be people's svn
>> directories on codespeak?
>
> Actually, does bitbucket serve files straight from repositories, like
> http://codespeak.net/svn/some/path did?  I tried a bit, but seem to
> only get links to specific revisions of files, not "whatever the
> latest revision is".
>
>
> A bientôt,
>
> Armin.
> ___
> pypy-dev@codespeak.net
> http://codespeak.net/mailman/listinfo/pypy-dev
>
___
pypy-dev@codespeak.net
http://codespeak.net/mailman/listinfo/pypy-dev

Re: [pypy-dev] implementing the additional repo migrations

[Antonio Cuni]

+1 on keeping the pdf files. It happens quite often not to have all
the needed latex packages

On 2/26/11, Carl Friedrich Bolz  wrote:
> On 02/26/2011 01:03 PM, Armin Rigo wrote:
>> Hi Laura,
>>
>> On Sat, Feb 26, 2011 at 10:59 AM, Laura Creighton  wrote:
>>> I don't care about the old versions of binary files.
>>
>> That was the only thing we talked about -- as far as I understood, it
>> was never suggested that we should stop tracking revisions of .txt or
>> .tex files.  I don't know the BigfilesExtension either, but it looks
>> to me like we can achieve some more precise result manually.
>> Something along the lines of: the .pdf's built from .tex's are not
>> checked in, but they are in some standardized place on
>> http://pypy.org, where we can fetch them, update them (via ssh), or
>> point people to (via their url).  This can be easily done with a
>> script independent from Mercurial.  (The point is of course that
>> tracking revisions is a bit useless, because we can always go back in
>> time and re-run latex2pdf.)
>
> Not necessarily, it's always possible that whatever latex packages were
> needed to compile the pdf are no longer around or a big hassle to
> install. This can make regeneration impractical. So I am in favor of
> keeping the PDFs in the repo.
>
> Carl Friedrich
> ___
> pypy-dev@codespeak.net
> http://codespeak.net/mailman/listinfo/pypy-dev
>
___
pypy-dev@codespeak.net
http://codespeak.net/mailman/listinfo/pypy-dev

Re: [pypy-dev] further pypy repo migrations

[Antonio Cuni]

On 17/02/11 15:23, holger krekel wrote:

> Thanks to Ronny Pfannschmidt, sponsorship from merlinux (my company), and work
> from Armin, Anto and others the main PyPy development repository has been
> migrated to mercurial on Bitbucket.  Btw, we just asked and Bitbucket nicely
> upgraded the PyPy project account to become an unlimited one, thanks!

just to clarify: "unlimited account" means that we can have private
repositories with as many users as we want (which is what we needed for
pypy-z). Thanks to Bitbucket also from my side :-)


> Any more pypy related repositories that need migration
> or any notes/comments?

currently, pypy has three svn subrepos which live on codespeak:

greenlet = [svn]http://codespeak.net/svn/greenlet/trunk/c
testrunner = [svn]http://codespeak.net/svn/pypy/build/testrunner
lib_pypy/pyrepl = [svn]http://codespeak.net/svn/pyrepl/trunk/pyrepl/pyrepl

we will need to migrate those as well at some point, and probably make them
mercurial subrepositories instead of svn ones.

ciao,
Anto
___
pypy-dev@codespeak.net
http://codespeak.net/mailman/listinfo/pypy-dev

Re: [pypy-dev] European sprints?

[Antonio Cuni]

On 16/02/11 12:01, Bea During wrote:

> Here is a suggestion of places and dates based on Lauras, Carl Friedrich and
> Antos
> input:
> 
> - Gothenburg sprint: 25th of April to 1st of May
> - Europython sprint/Florence: 25th of June to 26th of June (EP2011 official
> sprint dates)
> - Düsseldorf sprint: 22nd of August to 28th of August (fits the plan of the
> funded PyJIT project which ends end August)
> 
> What do you think about these dates - would they work?

for me they "mostly" work, although I might arrive one day later or depart one
day earlier for the Gothenburg and Duesseldorf ones.

ciao,
Anto
___
pypy-dev@codespeak.net
http://codespeak.net/mailman/listinfo/pypy-dev

Re: [pypy-dev] European sprints?

[Antonio Cuni]

On 15/02/11 15:02, Carl Friedrich Bolz wrote:

> Also I guess it is likely that we will have another one in Düsseldorf 
> later in the year.

and probably one after europython, as usual?

ciao,
Anto
___
pypy-dev@codespeak.net
http://codespeak.net/mailman/listinfo/pypy-dev

Re: [pypy-dev] google should build a pypy phone or someone

[Antonio Cuni]

On 15/02/11 11:52, Massa, Harald Armin wrote:
> Antonio,
> 
> 
> Anyway, back to the original topic: what would python offer more than 
> java for
> android?
> 
> a much less challenging licence-situation. 

I'm not completely sure.
AFAIK, the Oracle-Google lawsuit is about patents, not licences: in
particular, they are complaining about how Dalvik is implemented, not about
the fact that it implements Java.

>From what I could read at least a couple of those patents apply to PyPy as
well (and to whatever language with JIT, fwiw), e.g. IIRC there is one about
"compiling an optimized version of code that executes often" or something
similar (whether this patent is valid is another topic, of course).

ciao,
Anto
___
pypy-dev@codespeak.net
http://codespeak.net/mailman/listinfo/pypy-dev

Re: [pypy-dev] google should build a pypy phone or someone

[Antonio Cuni]

On 15/02/11 11:25, holger krekel wrote:
> Hi all, 
> 
> what do you think or know why Google choose Java over Python for
> its Android system?  Surely, Python doesn't have a big track record
> in terms of targetting mobile devices.  Also CPython is limited
> in how it can change and adapt i guess.  I imagine PyPy could do much
> better.  Wouldn't it be cool to have a totally python-based 
> state-of-the-art phone? :)

yes: this way, when my friends ask me what the hell I'm working on, I could
just show them the phone :-)

More seriously, I think that nowadays one of the most important points for a
mobile platform is to have a huge selection of 3rd party apps: in this sense,
the popularity of Java and the big number of developers (especially mediocre
and cheap developers, which is what you need to develop most of the apps out
there) is a big win for android.

On the other hand, iphone shows that you can do that even if your primary
language is Objective-C, but it AFAIK objc was already used quite a lot in the
apple ecosystem.

Anyway, back to the original topic: what would python offer more than java for
android?

ciao,
Anto
___
pypy-dev@codespeak.net
http://codespeak.net/mailman/listinfo/pypy-dev

Re: [pypy-dev] Separate compilation and friends

[Antonio Cuni]

On 15/02/11 03:41, Dima Tisnek wrote:
> On a related note, how hard is it to "freeze" the translator/compiler
> state of a given pypy version just before it begins to read extension
> modules and distribute that, it would speed up module development a
> lot.
> It would be a quick equivalent of distributing headers for a C library.

this is hard, because the compilation of the modules is intermixed with the
compilation of the rest of the interpreter for each phase: we have (roughly)
something like:

- annotation of the interpreter
- annotation of the modules
- rtyping of the interpreter
- rtyping of the modules
- etc. etc.

ciao,
Anto
___
pypy-dev@codespeak.net
http://codespeak.net/mailman/listinfo/pypy-dev

Re: [pypy-dev] testing floating point

[Antonio Cuni]

On 03/02/11 13:13, Maciej Fijalkowski wrote:

> I'm not sure about warnings module. How about
> --jit warnings=1
> ?
> 
> That would fit with other jit options. 

not really. The other jit options really belongs to the JIT engine, while this
one is dependent on the Python interpreter

ciao,
Anto
___
pypy-dev@codespeak.net
http://codespeak.net/mailman/listinfo/pypy-dev

Re: [pypy-dev] New version of Codespeed (0.7) for speed.pypy.org

[Antonio Cuni]

On 21/01/11 08:49, Miquel Torres wrote:
> @Anto
>
> Yes, branches are a pending item that has been requested a couple of times 
> now.

yes, I think most of the requests has been by me :)

> The current solution is actually not to abuse an environment like you
> say, but to create a new project for a branch. That way it gets
> cleanly separated, and in a way branches are like different projects.
> But it is of course not optimal. Technically it is very easy to come
> up with several solutions (add a branch dimension, for example), but
> interface-wise it is not easy to find something that doesn't clutter
> things.

Uhm, I don't think that using a different project is a good idea. For 
branches, we are usually not much interested in the branch history, but in the 
comparison with trunk (ideally, with trunk at the point we created the branch, 
or at the point of the last merge from trunk).

As for visualize changes, I think that we don't need anything fancy, for 
example it would be already immensely useful to have the possibility of 
displaying the Changes page in a way that it compares the performances of the 
branch against trunk.

ciao,
Anto
___
pypy-dev@codespeak.net
http://codespeak.net/mailman/listinfo/pypy-dev

Re: [pypy-dev] New version of Codespeed (0.7) for speed.pypy.org

[Antonio Cuni]

Hi Miquel

On 20/01/11 22:00, Miquel Torres wrote:
> Hi all,
>
> I want to announce the release of Codespeed 0.7, the version which is
> now powering speed.pypy.org

wow, that's very nice, thank you once more :-).

 From my side, there is only one feature that I miss a lot, which is the 
possibility to benchmark a branch.

IIRC, at the moment if you just run benchmarks on a branch, they are appended 
to the list of results, which can be a bit confusing especially if you look at 
them weeks later.

I think that maybe it's possible to do it by (ab)using an additional 
environment, but probably a more builtin solution is better. Do you have any 
idea/opinion/suggestion on this?

ciao,
Anto
___
pypy-dev@codespeak.net
http://codespeak.net/mailman/listinfo/pypy-dev

Re: [pypy-dev] 1.4.1 threading

[Antonio Cuni]

On 27/12/10 09:31, Carl Friedrich Bolz wrote:

> The only case where improving dicts would help is for user-defined 
> dictionaries, i.e. when you write dicts with curly braces.

then it's easy to optimize, you just write "dict()" instead of "{}".

(sorry, I could not resist :-))
___
pypy-dev@codespeak.net
http://codespeak.net/mailman/listinfo/pypy-dev

Re: [pypy-dev] long.__itemsize__

[Antonio Cuni]

On 21/12/10 12:05, Maciej Fijalkowski wrote:

>> __itemsize__ - in bytes, corresponds to item size field in the types
>> definition structure.
>>
>> It's a field for types.
>> See:
>>http://docs.python.org/c-api/typeobj.html#tp_itemsize
>>
> 
> Well... Those are docs for C API. It doesn't say it's exposed at
> applevel nor since which version. (Also, to be precise, C API is known
> to be implementation specific)

Moreover, I don't think we could give it a sane semantics on PyPy, given that
the same applevel type can be potentially implemented by many different low
level structures with different sizes.

ciao,
Anto
___
pypy-dev@codespeak.net
http://codespeak.net/mailman/listinfo/pypy-dev

Re: [pypy-dev] playing with fast-forward

[Antonio Cuni]

On 18/12/10 23:58, Gary Robinson wrote:
> I'm experimenting with the fast-forward branch. I'm actually not sure about 
> the proper way to get it. (I have the main branch working fine.)
> 
> I downloaded a nojit version from 
> http://buildbot.pypy.org/nightly/branch/fast-forward/ since I didn't see any 
> jit versions listed there. I was happy to see that the method I care most 
> about, multiprocessing.Pool.imap_unordered, seemed to work. But I want to 
> test it with the jit.

consider that we don't have automatic nightly builds for fast-forward, so the
ones you find on that page are manually triggered, and possibly outdated.

> The page (https://bitbucket.org/pypy/pypy/src/021b219e0aef) appears to be for 
> the branch, but the mercurial URL shown on that page appears to be for the 
> main project. 

yes, if you clone the mercurial repo you get all the branches together.

> I tried the downloads under the Source pop-up menu. For some reason, I only 
> sporadically am able to get a complete .gz file.** But I did get one. 
> 
> I was able to run:
> 
>   python translate.py -Ojit
> 
> successfully -- or at least it appeared so. But I couldn't find a bin/pypy to 
> run!?

as suggested by Alex, you probably have a pypy-c binary in
pypy/translator/goal (I think that the translate script even says so, but I
agree that it produces so much output that it might be hard to stop the
message :-))

> Any suggestions how to run fast-forward with jit?

I don't know how the "download source" button of bitbucket works, so I'm not
even sure that you downloaded the fast-forward branch instead of the default 
one.

The best way is to do this:

$ hg clone http://bitbucket.org/pypy/pypy
$ cd pypy
$ hg up -r fast-forward

Now you can go to pypy/translator/goal and run translate.py again.

ciao,
Anto
___
pypy-dev@codespeak.net
http://codespeak.net/mailman/listinfo/pypy-dev

Re: [pypy-dev] speed.pypy.org and mercurial

[Antonio Cuni]

Hi Miquel,

On 16/12/10 09:29, Miquel Torres wrote:
> Hi Anto,
> 
> yes, that is expected, but no problem. Codespeed is designed in such a
> way that it can support different version control systems. It is just
> that there is only support for svn now ;-)
> 
> So for mercurial support, I need to implement a function that takes a
> changeset hash, connects to the hg repo and returns a dict with
> 'date', 'user' and 'description' or 'summary' info. Anybody knows what
> library I could use for that? or do I have to parse local 'hg log'
> output?

I think you have several options to do that. For the particular pypy case,
maybe the simplest is to directly ask bitbucket for the changeset:
https://bitbucket.org/pypy/pypy/changeset/c55286db0781/raw/

this has the advantage that you don't need any library but just an http
request, but of course it works only for bitbucket repos.

Alternatively, if you install mercurial you can then "import
mercurial.commands" and use its public API from Python. Or as you said you can
just execute hg log and parse the output: in this case you might be interested
in the --template option, which allows you to format things exactly as you
want, e.g.:

$ hg log -r c55286db0781 --template '{author}\n{date|isodate}\n{desc}\n'
Armin Rigo 
2010-12-16 10:01 +0100
Document this publicly.

see this page for more details:
http://hgbook.red-bean.com/read/customizing-the-output-of-mercurial.html

ciao,
Anto
___
pypy-dev@codespeak.net
http://codespeak.net/mailman/listinfo/pypy-dev

[pypy-dev] speed.pypy.org and mercurial

[Antonio Cuni]

Hi Miquel, hi all,

as you probably have noticed, we have recently migrated the main repo to
mercurial.  Now speed.pypy.org receives a revision number in the form
"40046:2088ce763fc2", but of course it can no longer fetches the commit logs
from the svn server.

Would it be possible to fetch the commits from the bitbucket repo, please?

ciao,
Anto
___
pypy-dev@codespeak.net
http://codespeak.net/mailman/listinfo/pypy-dev

Re: [pypy-dev] [pypy-svn] pypy commit 7b8aa74da1fb: Handle output on standard error when invoking Mercurial.

[Antonio Cuni]

On 14/12/10 20:28, Dan Villiom Podlaski Christiansen wrote:
> On 14 Dec 2010, at 20:01, Maciej Fijalkowski wrote:
> 
>> This commit contains no tests whatsoever, it would be cooler if it did.
> 
> Antonio kindly wrote a test for the functionality in an earlier changeset.
> Ironically, I broke it that change, but that's fixed now :)

anyway, maciek on the channel mentioned a way to test it without messing with
the hg repo: you can provide a custom hgexe script which returns whatever you
want.

Or even, restructure the code in a way that the running of the hg command is
decoupled by the parsing of its results, so that you can at least test the 
latter.

Sorry, I should have pointed this out when you asked me to review the patches,
but today I was fried and had thousands of things to look at :-/

ciao,
Anto
___
pypy-dev@codespeak.net
http://codespeak.net/mailman/listinfo/pypy-dev

Re: [pypy-dev] Migration to mercurial

[Antonio Cuni]

On 14/12/10 00:02, Antonio Cuni wrote:
> Hi all,
> 
> finally, it's happening :-).
> Thanks to Ronny's work, we are going to complete the migration to mercurial
> very soon.

so, the migration is done!
The svn repo is readonly, and from now the official pypy repo is this one:
http://bitbucket.org/pypy/pypy

There are still some rough edges to fix on buildbot, but for the rest
everything seems to work fine.

I would like to thank everyone who was involved in the migration, in
particular Ronny who did most of the dirty work :-)

ciao,
Anto
___
pypy-dev@codespeak.net
http://codespeak.net/mailman/listinfo/pypy-dev

Re: [pypy-dev] Migration to mercurial

[Antonio Cuni]

On 14/12/10 01:57, Benjamin Peterson wrote:

> What will be happening to the other repos? ie. extradoc

it's open to discussion, but I suppose that it's fine to migrate them as well.
However, there is no hurry on this, we can do it one by one.

ciao,
Anto
___
pypy-dev@codespeak.net
http://codespeak.net/mailman/listinfo/pypy-dev

[pypy-dev] Migration to mercurial

[Antonio Cuni]

Hi all,

finally, it's happening :-).
Thanks to Ronny's work, we are going to complete the migration to mercurial
very soon.

The pypy buildbot is already configured to pull from the bitbucket repository:
http://bitbucket.org/pypy/pypy

I have tried to run a test build, and it seems to work ok. We will see
tomorrow morning how the nightly runs went.

If everything goes right, tomorrow we will declare the hg repo as the
"official" one: the svn repo will be made read-only (as soon as armin does it
:-)), the mercurial one will be synchronized to include the latest svn
commits, and from that point everyone should start to commit to mercurial.  If
you don't have write access to the bitbucket repo, please ask me or anyone
else who is an admin.

A note for people working on active branches: before you can work on those,
you need to perform two simple manual steps, as detailed in this howto:
http://codespeak.net/svn/pypy/extradoc/planning/hg-migration/active-branches-howto.txt

ciao,
Anto
___
pypy-dev@codespeak.net
http://codespeak.net/mailman/listinfo/pypy-dev

Re: [pypy-dev] setrecursionlimit deprecation?

[Antonio Cuni]

On 11/12/10 18:32, Armin Rigo wrote:

> But then people are going to complain that their app seems to hang up
> on pypy, and neither they nor we have any clue what is going on ---
> until we figure out that they used sys.setrecursionlimit() to put a
> bound on recursion and catch the RuntimeError.  That's at least the
> reason for which I suggested that calling sys.setrecursionlimit()
> should at least print a warning.  Now I agree that maybe the message
> of the warning is not the clearest one.

I was about to propose to change the message into something like
"sys.setrecursionlimit is ignored by PyPy".  I think this is already an
improvement over the current message, but has the drawback than then people
will complain that pypy might run out of stack (which is false, but not
apparent by the warning message).

Also, we should maybe change the class of the Warning. IIRC,
DeprecationWarning is going to be ignored by default on python2.7, which means
that most people won't see the warning at all once we merge fast-forward.

ciao,
Anto
___
pypy-dev@codespeak.net
http://codespeak.net/mailman/listinfo/pypy-dev

Re: [pypy-dev] [PyPy Status Blog] New comment on PyPy 1.4: Ouroboros in practice.

[Antonio Cuni]

On 01/12/10 23:21, Amaury Forgeot d'Arc wrote:
[cut]
> There is already an ongoing effort to port PyPy to Python 2.7.
> 
> But we need some help! It's a good way to become a PyPy developer.
> And no, you don't have to be a JIT expert to implement itertools.combinations
> or asian codecs.

Nice comment Amaury :-).

What about writing a blog post explicitly asking for help? I also agree that
the fast-forward branch is a good way to enter pypy, moreover we should
exploit the hype we seem to have at the moment :-).

ciao,
Anto
___
pypy-dev@codespeak.net
http://codespeak.net/mailman/listinfo/pypy-dev

Re: [pypy-dev] gdbm

[Antonio Cuni]

On 30/11/10 02:28, Dan Stromberg wrote:

> Agreed.
> 
> I should have time for this sometime this week or the next.

thank you! :-)
___
pypy-dev@codespeak.net
http://codespeak.net/mailman/listinfo/pypy-dev

Re: [pypy-dev] gdbm

[Antonio Cuni]

On 27/11/10 04:07, Dan Stromberg wrote:

> This module can be used with the “classic” ndbm interface, the BSD DB
> compatibility interface, or the GNU GDBM compatibility interface. On Unix, the
> *configure* script will attempt to locate the appropriate header file to
> simplify building this module.
> 
> I suppose that means that if it can't find ndbm (which at one time was hard
> due to licensing, but last I heard it'd become readily available), it's free
> to pretend it has ndbm using something else.
> 
> I'd call that puzzlingly worded - it's not the interface that's changing, but
> the backend implementation.  But perhaps dbm.py is free to use Berkeley DB if
> it prefers to pretend it can never find ndbm.  And perhaps I shouldn't have
> skimmed that page so quickly ^_^
> 
> CPython 2.7's configure script has:
> 
>   --with-dbmliborder=db1:db2:...
>   order to check db backends for dbm. Valid value is a
>   colon separated string with the backend names
>   `ndbm', `gdbm' and `bdb'.
> 
> 

so, having a dbm.py which links to libdb is fine, and it's also what you get
with cpython on ubuntu.  There is still the issue of how to find the correct
library name, as it seems it can vary (it was db-4.5 when the module was
written, it's db-4.8 nowadays), but this is a bit orthogonal to what we are
discussing here.

To summarize, I think we should keep the current dbm.py to link against libdb,
and integrate your gdbm.py to link against libgdbm.  But before merging it to
trunk, I'd like to solve the issue of code duplication between the two modules.

ciao,
Anto
___
pypy-dev@codespeak.net
http://codespeak.net/mailman/listinfo/pypy-dev

Re: [pypy-dev] PyPy 1.4 released

[Antonio Cuni]

On 28/11/10 10:48, Maciej Fijalkowski wrote:
>> > i got python-magic working , after i installed without easy_install
>> > (easy_install fail because it tried to install ctypes).

> great


well, that's still a bit strange. Why does easy_install try to install ctypes,
given that it's in the stdlib?
___
pypy-dev@codespeak.net
http://codespeak.net/mailman/listinfo/pypy-dev

Re: [pypy-dev] PyPy 1.4 released

[Antonio Cuni]

On 27/11/10 03:09, Phyo Arkar wrote:
> libmagic python fails to work on pypy (python-magic)
>
> it uses ctypes and easy-install try to download and instaill it , but it 
> fails.
>
> how to enable ctypes on pypy?

Hi Phyo,
ctypes *is* enabled on pypy by default.

If python-magic does not work, it can either:

1) be a pypy bug: please report it as an issue (possibly with a simple failing 
test and the full traceack)

2) a python-magic issue, e.g. if it plays dirty ctypes trick that cannot 
really be supported by pypy due to the internal differences with CPython

ciao,
Anto
___
pypy-dev@codespeak.net
http://codespeak.net/mailman/listinfo/pypy-dev

Re: [pypy-dev] gdbm

[Antonio Cuni]

On 16/11/10 04:30, Dan Stromberg wrote:

> BTW, it might cause confusion down the road to call something that is
> basically like cpython's bsddb (Berkeley DB) by the name "dbm" in pypy's
> library.  In the cpython standard library, "dbm" is an interface to ndbm
> databases.  These all provide the same dictionary-like interface to Python
> programs, but have somewhat different API's to C, and pretty different,
> incompatible on-disk representations.

Hi Dan,
I played a bit (vry quickly) with dbm on both pypy and cpython, and I'm 
not sure I get what you mean when you say that our dbm.py is equivalent to 
cpython's bsddb. E.g., I can create a db on cpython and open it from pypy, so 
it seems that the two modules are compatible.

Moreover, I checked which libraries the links to. On CPython, it links to 
libdb-4.8.so:

viper2 ~ $ ldd /usr/lib/python2.6/lib-dynload/dbm.so
 linux-gate.so.1 =>  (0x00884000)
 libdb-4.8.so => /usr/lib/libdb-4.8.so (0x0011)
 libpthread.so.0 => /lib/libpthread.so.0 (0x003de000)
 libc.so.6 => /lib/libc.so.6 (0x003f8000)
 /lib/ld-linux.so.2 (0x002e)

the pypy version first tries to open libdb.so, then libdb-4.5.so. I had to 
manually modify it to open version 4.8 (I agree that we should find a more 
general way to find it), but apart from that what I can see is that it uses 
the same underlying wrapper as CPython.

So, to summarise: could you elaborate a bit more why we should delete dbm.py 
from pypy?

ciao,
Anto
___
pypy-dev@codespeak.net
http://codespeak.net/mailman/listinfo/pypy-dev

Re: [pypy-dev] ype object 'BlackholeInterpreter' has no attribute 'bhimpl_direct_ptradd'

[Antonio Cuni]

On 25/11/10 17:46, wlavrij...@lbl.gov wrote:

> yes, but the fast path was disabled (commented out) when updating the branch
> with trunk (I needed a few casts in the JIT that were in trunk already, but
> not in the branch). It should thus be in addition when it's working again.
> (And to be sure, using a float argument makes no difference for the numbers.)
>
> Myself, I'm just working on functionality right now. A factor of almost 10
> is already good enough to start working on a functional demo of real code,
> given that most of our analysis is I/O bound.

wow, if you did not use the fast path, then the 10x factor it's even more 
impressive :-).

I think that yesterday Carl Friedrich added support to cppyy for calling the 
function through the new JIT-friendly "rlib.libffi" module, so now you should 
get the "fast-path" performance for most of the calls.  Carl, could you post 
the benchmarks you did please?

ciao,
Anto
___
pypy-dev@codespeak.net
http://codespeak.net/mailman/listinfo/pypy-dev

Re: [pypy-dev] ype object 'BlackholeInterpreter' has no attribute 'bhimpl_direct_ptradd'

[Antonio Cuni]

Hi Wim,

On 25/11/10 01:32, wlavrij...@lbl.gov wrote:
> Hi,
>
> so I'm revising my numbers after finding out that I was using a debug version
> of ROOT/Reflex ...
>
> PyROOT:   48.6
> PyCintex: 50.2
> pypy-c:5.5
> C++:   0.05
>

wow, impressive numbers :-).

You said that the benchmark you used is a loop calling a c++ function that 
does nothing. What is the signature of that function?
If you remember, in interp_cppyy there is a fast path that gives huge speedups 
when calling a c++ function which takes an integer and returns an integer, so 
depending on the signature you get very different numbers today.

The good news is that nowadays the JIT has got the capability to optimise 
external calls in a general way: this means that as soon as we integrate it 
with cppyy we can have all calls as fast as the ones that go through the 
current fast path.

ciao,
Anto
___
pypy-dev@codespeak.net
http://codespeak.net/mailman/listinfo/pypy-dev

Re: [pypy-dev] [pypy-svn] r79480 - pypy/trunk/pypy/jit/tool

[Antonio Cuni]

On 25/11/10 07:40, Maciej Fijalkowski wrote:
> Is pypy repository really a place for such files? Maybe we should keep
> it somewhere else?

well, it's the memory usage of cpython when translating pypy. Why the pypy 
repo should not be the place for a pypy-related file?

> On Wed, Nov 24, 2010 at 6:39 PM,  wrote:
>> Author: antocuni
>> Date: Wed Nov 24 17:39:03 2010
>> New Revision: 79480
>>
>> Added:
>>pypy/trunk/pypy/jit/tool/cpython.vmrss
>> Log:
>> add the vmrss file produced by armin's memusage.py when running 
>> ./translate.py -Ojit at rev 79456

___
pypy-dev@codespeak.net
http://codespeak.net/mailman/listinfo/pypy-dev

Re: [pypy-dev] [pypy-svn] r79382 - pypy/trunk/pypy/config

[Antonio Cuni]

On 23/11/10 08:18, fi...@codespeak.net wrote:
> Author: fijal
> Date: Tue Nov 23 08:18:57 2010
> New Revision: 79382
>
> Modified:
> pypy/trunk/pypy/config/translationoption.py
> Log:
> Disable jit debugging by default
>
>
> Modified: pypy/trunk/pypy/config/translationoption.py
> ==
> --- pypy/trunk/pypy/config/translationoption.py   (original)
> +++ pypy/trunk/pypy/config/translationoption.py   Tue Nov 23 08:18:57 2010
> @@ -115,7 +115,7 @@
>default="auto", cmdline="--jit-backend"),
>   ChoiceOption("jit_debug", "the amount of debugging dumps made by the 
> JIT",
>["off", "profile", "steps", "detailed"],
> - default="profile",  # XXX for now
> + default="off",
>cmdline="--jit-debug"),
>   ChoiceOption("jit_profiler", "integrate profiler support into the JIT",
>["off", "oprofile"],

Not sure it's a good idea. It's useful to see the statistics at the end of the 
run.  I know that you can turn it on explicitly, but it's easy to forget.

What about having an env variable that automatically turns jit debug on?

ciao,
Anto
___
pypy-dev@codespeak.net
http://codespeak.net/mailman/listinfo/pypy-dev

Re: [pypy-dev] gdbm

[Antonio Cuni]

On 18/11/10 01:17, Dan Stromberg wrote:

Hi Dan,

[cut]
> Given #1 and #2 above, anydbm should continue working, due to the presence of
> gdbm and dumbdbm.
>
> I guess I think that if someone has a need for bsddb (and it's assorted
> interfaces), they probably should work on that.
>
> Sound reasonable?

Yes. To me it sounds reasonable. I don't know if it's correct because I never 
used any of those modules, but it's surely reasoable :-).

I think that the best would be if you get an account on codespeak and do that 
in a branch: this way we can follow your commits and the if everything is fine 
we merge it.

I personally cannot create new accounts on codespeak, so you should ask 
someone else for this (e.g. Armin Rigo): the easiest way is if you just come 
on #pypy on freenode.net and talk with us "real-time".

ciao,
Anto
___
pypy-dev@codespeak.net
http://codespeak.net/mailman/listinfo/pypy-dev

Re: [pypy-dev] gdbm

[Antonio Cuni]

Hi Dan,
first: thanks for your help :-)

On 16/11/10 03:17, Dan Stromberg wrote:
>
> Yes, the dbm module in pypy is basically like the bsddb module in cpython.
>
> cpython includes modules for bsddb, gdbm, and more.
>
> I tend to prefer gdbm over bsddb, because I've seen bsddb databases get
> corrupt too many times - EG, when a filesystem overflows.  bsddb might be a
> little faster though; I've never compared their performance.

So, if I understand correctly you are saying that we should rename our dbm.py 
to bsdb.py, and write a new dbm.py which can use either bsdb or gdbm?
Sounds fine, do you feel like implementing it? :-)

Moreover, I also agree with amaury that your code is very similar to the one 
in the current dbm.py, so we should maybe try to refactor things to share 
common parts between the twos.

ciao,
Anto
___
pypy-dev@codespeak.net
http://codespeak.net/mailman/listinfo/pypy-dev

[pypy-dev] Migration to mercurial

[Antonio Cuni]

Hi all,

as some of you might have noticed by watching the IRC channel, we are finally
migrating to mercurial.  The actual conversion of the repository will happen
at some point during next week (or maybe the week after).

One issue that we have to care about is how to convert the author names: in
the current svn repository each author is simply indicated by its codespeak
username, while with mercurial this is no longer possible as there is no
longer a "centralized" server where everyone has an account.

In mercurial an author is just a string, but the standard way is put both the
real name and a valid email address to uniquely identify the author.  This is
exploited by websites like e.g. bitbucket to link the author of the commit to
the bitbucket user.

For doing the conversion, we will use the following usermap:
http://codespeak.net/svn/pypy/extradoc/planning/hg-migration/usermap.txt

By default, we map each username to its real name, as found in the /etc/passwd
file on codespeak.net.  Because of the reasons I explained above, I strongly
recommend everyone to put its email address there, so that it will be there in
the final mercurial repository.

If for any reason you don't want your real name to be in the mercurial
repository, please kill the corresponding line in the file.

Note that it won't be possible to change the info after the conversion has
been made.

ciao,
Anto
___
pypy-dev@codespeak.net
http://codespeak.net/mailman/listinfo/pypy-dev

Re: [pypy-dev] [pypy-svn] r77589 - pypy/trunk/pypy/jit/metainterp

[Antonio Cuni]

On 04/10/10 23:32, fi...@codespeak.net wrote:

>   def transform(op):
>   from pypy.jit.metainterp.history import AbstractDescr
> -# Rename CALL_PURE to CALL.
> +# Rename CALL_PURE and CALL_INVARIANT to CALL.
>   # Simplify the VIRTUAL_REF_* so that they don't show up in the backend.
> -if op.getopnum() == rop.CALL_PURE:
> +if (op.getopnum() == rop.CALL_PURE or
> +op.getopnum() == rop.CALL_LOOPINVARIANT):
>   op = ResOperation(rop.CALL, op.getarglist()[1:], op.result,
> op.getdescr())
>   elif op.getopnum() == rop.VIRTUAL_REF:

Hi,
I just wanted to tell that since the resoperation-refactoring, there is now a 
nice "copy_and_change" method to be used exactly for the case where you want 
to replace an operation with a "similar" one.

So, the code above should be written like:

op = op.copy_and_change(rop.CALL, args=op.getarglist()[1:])

ciao,
Anto
___
pypy-dev@codespeak.net
http://codespeak.net/mailman/listinfo/pypy-dev

Re: [pypy-dev] Question on the future of RPython

[Antonio Cuni]

On 29/09/10 22:40, Terrence Cole wrote:

>> then, mylog contains all the loops and bridges produced by the jit. The
>> interesting point is that there are also special operations called
>> "debug_merge_point" that are emitted for each python bytecode, so you can
>> easily map the low-level jit instructions back to the original python source.
>
> I think that 'easily' in that last sentence is missing scare-quotes. :-)

well, it's easy as long as you have a bytecode-compiled version around. With 
only the AST I agree that it might be a bit trickier.

[cut]
> My first inclination would be to continue this chain and add a bytecode
> compiler on top of the ast builder.  This would keep ast node references
> in the instructions it creates.  If the algorithms don't diverge too
> much, I think this would allow the debug output to be mapped all the way
> back to the source chars with minimal effort.  I'm not terrifically
> familiar with the specifics of how python emits bytecode from an ast, so
> I'd appreciate any feedback if you think this is crazy-talk.

Are you using your custom-made AST or the one from the standard library? In 
the latter case, you can just pass the ast to the compile() builtin function 
to get the corresponding bytecode.

___
pypy-dev@codespeak.net
http://codespeak.net/mailman/listinfo/pypy-dev

Re: [pypy-dev] Question on the future of RPython

[Antonio Cuni]

Hi Terrence, hi all

On 28/09/10 22:33, Terrence Cole wrote:
> I assume workflow would go like this:  1) run pypy on a bunch of code in
> profiling mode, 2) pypy spits out lots of data about what happened in
> the jit when the program exits, 3) start up external analysis program
> pointing it at this data, 4) browse the python source with the data from
> the jit overlayed as color, formatting, etc on top of the source.

You can already to it (partially) by using the PYPYLOG environment variable 
like this:

PYPYLOG=jit-log-opt:mylog ./pypy -m test.pystone

then, mylog contains all the loops and bridges produced by the jit. The 
interesting point is that there are also special operations called 
"debug_merge_point" that are emitted for each python bytecode, so you can 
easily map the low-level jit instructions back to the original python source.

E.g., take lines 214 of pystone:
 Array1Par[IntLoc+30] = IntLoc

The corresponding python bytecode is this:
214  38 LOAD_FAST4 (IntLoc)
  41 LOAD_FAST0 (Array1Par)
  44 LOAD_FAST4 (IntLoc)
  47 LOAD_CONST   3 (30)
  50 BINARY_ADD
  51 STORE_SUBSCR


By searching in the logs, you find the following (I edited it a bit to improve 
readability):

debug_merge_point(' #38 LOAD_FAST')
debug_merge_point(' #41 LOAD_FAST')
debug_merge_point(' #44 LOAD_FAST')
debug_merge_point(' #47 LOAD_CONST')
debug_merge_point(' #50 BINARY_ADD')
debug_merge_point(' #51 STORE_SUBSCR')
p345 = new_with_vtable(ConstClass(W_IntObject))
setfield_gc(p345, 8, descr=)
call(ConstClass(ll_setitem__dum_checkidxConst_listPtr_Signed_objectPtr),
  p333, 38, p345, descr=)
guard_no_exception(, descr=) [p1, p0, p71, p345, p312, p3, p4, p6, 
p308, p315, p335, p12, p13, p14, p15, p16, p18, p19, p178, p26, p320, p328, 
i124, p25, i329]

Here, you can see that most opcodes are "empty" (i.e., no operations between 
one debug_merge_point and the next). In general, all the opcodes that 
manipulate the python stack are optimized away by the jit, because all the 
python variables on the stack become "local variables" in the assembler.

Moreover, you can see that BINARY_ADD is also empty: this probably means that 
the loop was specialized for the specific value of IntLoc, so the addition has 
been constant-folded away.  Indeed, the only opcode that do real work is 
STORE_SUBSCR.  What it does it to allocate a new W_IntObject whose value is 8 
(i.e., boxing IntLoc on the fly, because it's escaping), and store it into the 
element 38 of the list stored in p333.

Finally, we check that no exception was raised.

Obviously, when presenting these information to the user you must consider 
that there is not a 1-to-1 mapping from python source to jit loops.  In the 
example above, the very same opcodes are compiled also in another loop (which 
by chance it has the same jit-operations, but they might also be very 
different, depending on the cases).

As you can see, there is already lot of information that can be useful to the 
user.  However, don't ask me how to present it visually :-)

ciao,
anto
___
pypy-dev@codespeak.net
http://codespeak.net/mailman/listinfo/pypy-dev

Re: [pypy-dev] [pypy-svn] r77101 - in pypy/trunk/pypy: jit/tl module/__builtin__ module/__builtin__/test module/pypyjit/test

[Antonio Cuni]

Hi,

On 16/09/10 07:27, hakana...@codespeak.net wrote:

> Log:
> Allow jit to unroll calls to max() and min() with more than one argument.
>
[cut]
> +...@unroll_safe
>   @specialize.arg(2)
>   def min_max(space, args, implementation_of):
>   if implementation_of == "max":
>   compare = space.gt
>   else:
>   compare = space.lt
> +
> +args_w = args.arguments_w
> +if len(args_w)>  1 and not args.keywords: # Unrollable case
> +w_max_item = None
> +for w_item in args_w:
> +if w_max_item is None or \
> +   space.is_true(compare(w_item, w_max_item)):
> +w_max_item = w_item
> +return w_max_item
> +else:
> +return min_max_loop(space, args, implementation_of)


I don't think it's a good idea. What happens if I call max() over a list of 1 
million of elements? We obviously don't want the jit to unroll 1 million of 
iterations. Or am I missing something?

ciao,
Anto
___
pypy-dev@codespeak.net
http://codespeak.net/mailman/listinfo/pypy-dev

Re: [pypy-dev] [pypy-svn] r77083 - pypy/branch/jitffi

[Antonio Cuni]

Hi,

On 15/09/10 18:18, Maciej Fijalkowski wrote:
> Hey anto.
>
> There was a SoC about that, I guess it would be good to chat about it
> at least (personally I think jitting rlib/libffi is exactly bad layer
> to be jitted and some experiments were done).

yes, I read the code in the fast-ctypes branch but I wanted to take another 
(simpler) approach.  Note that my goal is not only to speed up ctypes, but 
also to provide a useful building block for cppyy (the module to call c++ 
functions that we started at the cern sprint).

My basic idea was to mark libffi.FuncPtr.{push_arg,call} in a special way, so 
that the backend can recognize the pattern (i.e. push* + call) and emit a 
single assembler call.

I even started to write a bit of code, but then I realized that libffi.FuncPtr 
is not used at all, as _rawffi uses RawFuncPtr: the bad news is that 
RawFuncPtr uses a different interface, as it does not have push_arg but passes 
the arguments already packed in a list, so my easy solution above cannot work. 
  Note however that doing it at the level of FuncPtr might still be useful for 
cppyy.

Question: why does _rawffi use RawFuncPtr instead of FuncPtr? Would it be 
possible/easy/hard/whatever to switch to FuncPtr?

ciao,
Anto
___
pypy-dev@codespeak.net
http://codespeak.net/mailman/listinfo/pypy-dev

Re: [pypy-dev] External RPython mailing list

[Antonio Cuni]

On 13/09/10 10:27, Maciej Fijalkowski wrote:

> Is it really about interpreters? (what's interpreter-specific after
> all in RPython) or is it just that it's hard to use and does not
> integrate with CPython well?

my point if that it's definitely good enough for writing interpreters. For the
rest, it's a bit unknown (in the sense that nobody has ever tried), and we
don't care about knowing :-)
___
pypy-dev@codespeak.net
http://codespeak.net/mailman/listinfo/pypy-dev

Re: [pypy-dev] External RPython mailing list

[Antonio Cuni]

On 13/09/10 10:14, Armin Rigo wrote:
> Hi,
> 
> On Mon, Sep 13, 2010 at 10:11 AM, Maciej Fijalkowski  wrote:
>> I don't think it's hideable.
> 
> Sorry, I wasn't clear.  I'm not really trying to hide it.  But I'm
> also not really trying to push it forward (which seems to be what
> creating a website for it would do).

well, I don't think that hiding it or pushing it backward is a good idea.  In
theory, we would like if other people start to use rpython to write 
interpreters.

What we don't like is to use rpython as a general purpose language, but that's
a slightly different issue, IMHO.

ciao,
Anto
___
pypy-dev@codespeak.net
http://codespeak.net/mailman/listinfo/pypy-dev

Re: [pypy-dev] PyPy JIT & C extensions, greenlet

[Antonio Cuni]

On 13/09/10 10:10, Armin Rigo wrote:
> Hi Andy,
> 
> On Mon, Sep 13, 2010 at 7:21 AM, Andy  wrote:
>> Does that mean PyPy will not work with greenlet/gevent/etc?
> 
> Sorry if I wasn't clear.  PyPy contains greenlet support (since
> 2005-6).  It's part of the same package that we call "pypy-stackless".

yes, but it must also be said that at the moment, pypy-stackless and pypy-jit
do not work together.

ciao,
Anto
___
pypy-dev@codespeak.net
http://codespeak.net/mailman/listinfo/pypy-dev

Re: [pypy-dev] PyPy JIT and Django

[Antonio Cuni]

On 09/09/10 20:32, Andy wrote:
> Would this replace my existing Python interpretor with PyPy? I want to keep 
> my existing Python.

well, no: if you run pypy setup.py install on whatever package, what happens
is that you install this package in pypy's site-package directory instead of
cpython's one.

> I'd like to have the option to choose either the standard CPython or PyPy. I 
> could set up 2 vrtualenvs I suppose - 1 for CPython and 1 for PyPy. Does PyPy 
> work with virtualenv?

yes, pypy supports virtualenv but:

1) you need a pypy newer than pypy 1.3: you can build one by yourself from
svn, or download one of our nightly builds:
http://buildbot.pypy.org/nightly/trunk/

note that you probably want the one with -jit and -linux in it, which is a 32
bit executable. There is no pypy-jit for linux 64 bit yet (but it might be
there soon).

2) you need a recent version of virtualenv, as pypy support has been added
only recently (http://bitbucket.org/ianb/virtualenv/changeset/a03cb042dd81).
AFAIK, no released version supports it, so you need to install/run it from the
mercurial repository.

ciao,
Anto
___
pypy-dev@codespeak.net
http://codespeak.net/mailman/listinfo/pypy-dev

Re: [pypy-dev] [pypy-svn] r76608 - in pypy/branch/jit-bounds/pypy/jit/metainterp: . test

[Antonio Cuni]

On 12/08/10 19:02, hakana...@codespeak.net wrote:

> +def boundint_gt(self, val):
> +if val is None: return
> +self.minint = val + 1

what happens if val == sys.maxint?

ciao,
Anto
___
pypy-dev@codespeak.net
http://codespeak.net/mailman/listinfo/pypy-dev

Re: [pypy-dev] rstruct where is pack?

[Antonio Cuni]

On 10/08/10 15:14, Benjamin Peterson wrote:
> 2010/8/10 Hart's Antler :
>> Seems like struct.pack is not RPython?  I see the examples for unpack in the 
>> tests folder, but not for packing.
> 
> struct.pack() is implemented in pypy/module/rstruct/.

I suppose you mean pypy/module/struct.

But if the OP is looking for an rpython lib to use in his rpython program,
this is not exactly what he looks for, although I agree it could be adapted
and ported to rlib.

ciao,
Anto
___
pypy-dev@codespeak.net
http://codespeak.net/mailman/listinfo/pypy-dev

Re: [pypy-dev] array performace?

[Antonio Cuni]

On 03/07/10 19:22, Paolo Giarrusso wrote:
> I had 3 colleague students who implemented, for instance, a
> Python-to-JVM bytecode compiler which was way faster than Jython.
> Which was the trick?
[cut]

I'm ready to bet that they did not implement a Python compiler, but a 
simil-Python language that implements 80/90/95% of Python features.  The web 
is full of projects like this.

I'm not saying that the techniques used for that project are not worth of 
attention, just that probably "the trick" was not to support the features of 
Python that are hardest to implement efficiently.

ciao,
Anto
___
pypy-dev@codespeak.net
http://codespeak.net/mailman/listinfo/pypy-dev

Re: [pypy-dev] Interpreter level array implementation

[Antonio Cuni]

On 03/07/10 08:14, Hakan Ardo wrote:

> What is a bridge?

you might be interested to read the chapter of my PhD thesis which explains 
exactly that, with diagrams:
http://codespeak.net/svn/user/antocuni/phd/thesis/thesis.pdf

In particular, section 6.4 explains the difference between loops, bridges and 
entry bridges.

ciao,
Anto
___
pypy-dev@codespeak.net
http://codespeak.net/mailman/listinfo/pypy-dev

Re: [pypy-dev] [pypy-svn] r75683 - in pypy/trunk: include lib-python/modified-2.5.2/distutils lib-python/modified-2.5.2/distutils/command pypy/_interfaces pypy/module/cpyext pypy/module/cpyext/test

[Antonio Cuni]

On 02/07/10 09:28, Maciej Fijalkowski wrote:

> Fine by me. Can you fix test_package then? It assumes there is
> Python.h in include (which might not be there).

ah right... because when we run own-test translation didn't happen, so .h are 
not there. Ok, I'll fix it later.

ciao,
Anto
___
pypy-dev@codespeak.net
http://codespeak.net/mailman/listinfo/pypy-dev

Re: [pypy-dev] [pypy-svn] r75683 - in pypy/trunk: include lib-python/modified-2.5.2/distutils lib-python/modified-2.5.2/distutils/command pypy/_interfaces pypy/module/cpyext pypy/module/cpyext/test

[Antonio Cuni]

On 02/07/10 08:45, Maciej Fijalkowski wrote:
> Hey.
>
> Any reason why we should copy .h files during translation and can't
> just have them there?
>

I talked with Amaury and he told me that he prefers to keep all the 
cpyext-related files together, which I think makes sense.  Moreover, we need 
to generate© pypy_decl.h and pypy_macros.h anyway, so we can copy the 
others as well while we are at it.

ciao,
Anto
___
pypy-dev@codespeak.net
http://codespeak.net/mailman/listinfo/pypy-dev

Re: [pypy-dev] virtualenv support and directory hierarchy

[Antonio Cuni]

On 19/06/10 11:00, Amaury Forgeot d'Arc wrote:

>> The drawback is that it's a bit non-standard on unix; moreover, if we install
>> pypy in say /opt/pypy1.2, it would be hard to put a binary in /usr/bin 
>> without
>> hardcoding the path to pypy1.2 somewhere.
>
> Is it possible to just put a symlink, or a small script in /usr/bin?

yes, the symlink should be possible, as armin also points out. I already 
thought about it, but I was not sure that distributions like ubuntu allows to 
put a symlink in /usr/bin to something external.  But indeed, looking at my 
/usr/bin it seems that symlinks are used a lot, so it should be fine.

So, does this mean that we are going for solution number 2?



___
pypy-dev@codespeak.net
http://codespeak.net/mailman/listinfo/pypy-dev

Re: [pypy-dev] virtualenv support and directory hierarchy

[Antonio Cuni]

Hi,

I also don't like too much to have a hardcoded version number, but when I 
asked for alternatives nobody suggested anything.

On IRC, amaury suggested to install the whole pypy distribution in its 
self-contained directory, more or less as cpython does on windows.
I didn't think about this solution, but now I see that it might be a good one, 
as it would allow to have the same hierarchy on svn checkouts, user-specific 
installations and system-wide installations.

The drawback is that it's a bit non-standard on unix; moreover, if we install 
pypy in say /opt/pypy1.2, it would be hard to put a binary in /usr/bin without 
hardcoding the path to pypy1.2 somewhere.

So, for me there are four possible solutions:

1) leave things as they are on branch/sys-prefix and have the version number 
hardcoded in svn

2) put the whole distribution in its own directory, as jython or cpython on 
windows.  This has the open problem of determining where the directory is, as 
described above

3) don't hardcode the version number in svn, but add a special case to 
virtualenv to detect if we are inside a pypy checkout to handle it specially

4) don't care about running virtualenv inside a pypy checkout

Personally, I would exclude (4): I think that it would be very cool for people 
to try pypy in a sandboxed virtualenv without having to install it, and it 
would be useful to us too.

So, before I do more work, I'd like to hear what people think and which of the 
alternatives they prefer.

ciao,
Anto

On 18/06/10 19:01, Maciej Fijalkowski wrote:
> Hey anto.
>
> I expressed my concerns already, but I'm going to express them once
> again, since amaury also said that on IRC.
>
> I really dislike having version number hardcoded in source checkout
> for a whole variety of reasons. I don't think need to have virtualenv
> working from source checkout is enough to push that through.
>
> How about install script that does that maybe?
>
> Cheers,
> fijal

___
pypy-dev@codespeak.net
http://codespeak.net/mailman/listinfo/pypy-dev

Re: [pypy-dev] [pypy-svn] r75220 - in pypy/trunk/pypy: annotation annotation/test jit/backend jit/backend/llgraph jit/backend/llgraph/test jit/backend/llsupport jit/backend/llsupport/test jit/backend/

[Antonio Cuni]

On 13/06/10 08:46, Armin Rigo wrote:

> I'm a bit confused, btw: I thought that ootype did not need ConstAddr at
> all, because it used ConstObj for all pointer-ish things.

ah sorry. You wrote ConstAddr but I actually read ConstPtr (i.e., ConstObj for 
ootype). Indeed, ConstAddr is not used at all by ootype, so there should be no 
problem (at least in this respect :-)).

ciao,
Anto
___
pypy-dev@codespeak.net
http://codespeak.net/mailman/listinfo/pypy-dev

Re: [pypy-dev] [pypy-svn] r75220 - in pypy/trunk/pypy: annotation annotation/test jit/backend jit/backend/llgraph jit/backend/llgraph/test jit/backend/llsupport jit/backend/llsupport/test jit/backend/

[Antonio Cuni]

On 08/06/10 23:42, ar...@codespeak.net wrote:
> The number of changes is a bit huge though.  The
> format of the static bytecodes used by the jit
> changed completely, and codewriter.py is now split
> among many files in the new directory pypy.jit.codewriter.
> There is also no longer ConstAddr, only ConstInt: prebuilt
> addresses are now turned into integers (which are
> symbolic, with the new class AddressAsInt, so they can
> be rendered by the C translation backend).  Various
> related changes occurred here and there.

I didn't look at the branch deeply, but the last sentence looks suspiciously 
hard/impossible to implement in ootype.  Could you explain why ConstAdrr were 
bad please?

ciao,
Anto
___
pypy-dev@codespeak.net
http://codespeak.net/mailman/listinfo/pypy-dev

Re: [pypy-dev] [pypy-svn] r74976 - in pypy/branch/sys-prefix: lib/pypy1.2/lib_pypy/ctypes_config_cache pypy/interpreter/test pypy/rlib pypy/tool pypy/tool/test pypy/translator/goal pypy/translator/san

[Antonio Cuni]

On 01/06/10 06:00, Maciej Fijalkowski wrote:
> A bit about directory structure:

I think I have explained everything in my original email to pypy-dev:
http://codespeak.net/pipermail/pypy-dev/2010q2/005854.html

> Can you explain to me a bit?
> 
> What's in lib except pypy1.2?

nothing. It really plays the same role as /usr/lib. We need it because in this
way we can have sys.prefix == '/path/to/pypy-trunk' and still have the lib in
join(sys.prefix, 'lib', 'pypy%d.%d')

> Why pypy1.2? Do we have any reason?

yes. When we install pypy system wide, we really want to have a version number
in the directory that contains the stdlib; also, it is consistent with
cpython, which puts it into e.g. /usr/lib/python2.6

> Do we ever need to keep more than one?

no. We will rename it every time we do a new release.

> What's in pypy1.2 except lib_pypy?

there will be also lib-python, although I've not moved it yet.

ciao,
Anto
___
pypy-dev@codespeak.net
http://codespeak.net/mailman/listinfo/pypy-dev

Re: [pypy-dev] [pypy-svn] r74850 - in pypy/branch/sys-prefix/lib: . pypy1.2 pypy1.2/lib_pypy pypy1.2/lib_pypy/app_test pypy1.2/lib_pypy/ctypes_config_cache pypy1.2/lib_pypy/test2

[Antonio Cuni]

On 30/05/10 00:08, Maciej Fijalkowski wrote:
> My concern is mostly that we should not over engineer things and be
> smarter than CPython (at least in this respect). 

well, if we want sys.prefix we *need* to be smarter than CPython, as we don't
have any ./configure where take it from. The alternative would be to pass
--prefix to translate.py: do you *really* want to do it?

> There is a whole
> variety of reasons why it would not work in the end.

like?

> Point 2) talks about superset of CPython funcionality. How for example
> this would work with compiled cpython extensions that has setup.py
> install run?

if you use virtualenv, there should be no problem, as it recreates the whole
environment needed.
If you are talking about running ./translator/goal/pypy-c
/my/extension/setup.py install, well, in that case I guess that what happens
is just that your extension will be installed inside
pypy-trunk/lib/pypy1.2/site-packages or some path like this.  Well, too bad
for you, I would say.

> On smaller issue, I don't like pypy1.2. Both because it contains a dot
> and because it contains a revision number. Why we need that?

CPython has the standard lib in $PREFIX/lib/python2.6, so for consistency we
want ours to reside in $PREFIX/lib/pypy1.2.  I know, the drawback of this is
that we need to manually rename it every time we change version number, but
it's also true that it does not happens often.

If you have an alternative solution, I'd like to hear that.  Note that I don't
think that "don't care about using virtualenv from trunk" is a good
alternative solution :-).

ciao,
Anto
___
pypy-dev@codespeak.net
http://codespeak.net/mailman/listinfo/pypy-dev

Re: [pypy-dev] [pypy-svn] r74850 - in pypy/branch/sys-prefix/lib: . pypy1.2 pypy1.2/lib_pypy pypy1.2/lib_pypy/app_test pypy1.2/lib_pypy/ctypes_config_cache pypy1.2/lib_pypy/test2

[Antonio Cuni]

On 30/05/10 00:43, holger krekel wrote:

>> pyrolog for example doesn't use lib_pypy or pypy/module, does it? 
> 
> sorry, i meant prolog, gameboy, etc ... i.e. the projects in pypy/lang

yes, indeed. That's why I think it's a good idea to move pypy/lib outside pypy.
___
pypy-dev@codespeak.net
http://codespeak.net/mailman/listinfo/pypy-dev

Re: [pypy-dev] [pypy-svn] r74850 - in pypy/branch/sys-prefix/lib: . pypy1.2 pypy1.2/lib_pypy pypy1.2/lib_pypy/app_test pypy1.2/lib_pypy/ctypes_config_cache pypy1.2/lib_pypy/test2

[Antonio Cuni]

Hi holger,

On 29/05/10 22:25, holger krekel wrote:

> I think the idea is to make sys.prefix (and thus virtualenv) work 
> even with a translation in a checkout, i.e. not forcing to copy 
> things to another location (which virtualenv partly does on its own).  

yes

> Moreover, keeping app-level modules (and maybe pypy/module at some point)
> outside the main (interpreter, objspaces, translation and JIT) PyPy tree 
> makes sense to me.  e.g. pypy/lang would probably not need to access anything
> outside such a pypy tree, for example, or am i mistaken? 

I agree that keeping app-level modules out of pypy is a good idea (this is
what I'm doing it, of course :-)), but I don't get what does the reference to
pypy/lang mean in this context.

ciao,
Anto
___
pypy-dev@codespeak.net
http://codespeak.net/mailman/listinfo/pypy-dev

Re: [pypy-dev] [pypy-svn] r74850 - in pypy/branch/sys-prefix/lib: . pypy1.2 pypy1.2/lib_pypy pypy1.2/lib_pypy/app_test pypy1.2/lib_pypy/ctypes_config_cache pypy1.2/lib_pypy/test2

[Antonio Cuni]

Hi Maciek,

On 29/05/10 19:32, Maciej Fijalkowski wrote:
> Hm.
> 
> I might be missing something, but I thought sys.prefix is only meant
> for stuff after installation. If this is true (my CPython trunk build
> has sys.prefix == '/usr/local'), then modifying source checkout does
> not make any sense, since it's only about installation (which we don't
> really support anyway).

you are partly right. In CPython, sys.prefix is just an hard-coded string that
represent what was passed to ./configure; tools like virtualenv use it to find
the directories containing the stdlib, include files, etc. on the filesystem
(I'm not sure it's an entirely good idea, but this is the state of the things
and we have to face it).

However, nothing prevents the interpreter to set sys.prefix dynamically, e.g.
by searching for the directory that contains the stdlib in some location
relative to the pypy-c binary; this is what pypy-c does it already to set
sys.path.

The point of this branch is both:

1) to make it easier to install pypy system-wide: it will be enough to copy
pypy-c in /usr/bin and lib/pypy1.2 in /usr/lib (and of course we can automate
this with a script, if we want)

2) as holger pointed out, to make it possible to use virtualenv *without*
having to install pypy system-wide (which you cannot do with cpython, for
example): this will be possible because the directory hierarchy of the svn
checkout is designed in a way that setting sys.prefix to the the root of the
svn checkout will "just work"

ciao,
Anto
___
pypy-dev@codespeak.net
http://codespeak.net/mailman/listinfo/pypy-dev

[pypy-dev] virtualenv support and directory hierarchy

[Antonio Cuni]

Hi all,

I am investigating how to make virtualenv working on pypy and I'm running into 
a couple of issues: the most important one is that virtualenv relies on 
sys.prefix (which does not exists in pypy) to find the standard library, and 
the other is that the standard library of pypy is supposed to be put in 
/usr/share instead of /usr/lib (or /usr/local/*).

Currently, a pypy installation is supposed to have this structure:
 /usr/bin/pypy-c
 /usr/share/pypy-1.2/pypy/lib/
 /usr/share/pypy-1.2/lib-python/modified-2.5.2
 /usr/share/pypy-1.2/lib-python/2.5.2

In such a situation, sys.pypy_prefix is set to '/usr/share/pypy-1.2'.

I propose to change it in this way:
 /usr/bin/pypy-c
 /usr/lib/pypy1.2/lib-pypy/
 /usr/lib/pypy1.2/lib-python/modified-2.5.2
 /usr/lib/pypy1.2/lib-python/2.5.2

where lib-pypy contains what it's now in pypy/lib.
In such a situation, sys.prefix would be set to '/usr', in a similar way as 
cpython.  Also, we should also add a sys.exec_prefix which is meant to be 
always equal to sys.prefix (at least for now).
(I removed the dash in pypy-1.2 for consistency with cpython, which uses 
something like lib/python2.6).


Moreover, I would also like virtualenv to work from an svn checkout/source 
tarball of pypy, without any needing of installing it system-wide. To do so, 
we need to find a sensible value for sys.prefix, considering that tools like 
virtualenv suppose to find the stdlib under sys.prefix+'lib/'+something_else.

So, the proposed new structure is this:

 /path/to/pypy-trunk/
 /path/to/pypy-trunk/lib/pypy1.2/{lib-pypy,modified-2.5.2,2.5.2}
 sys.prefix == '/path/to/pypy-trunk'

The drawback is that before getting to the real files you have to walk a lot 
of empty directories, and that we should manually change the name of pypy1.2 
each time we increase the version number (not that it happens very often :-)).
One side-advantage is that in this way we would move pypy/lib outside the main 
pypy package, which is good because it's not really a part of it.

Finally, we should probably think of where to put the include/ directory (plus 
other that might be needed to build extension), but I'll let cpyext experts to 
say what it's better :-)

What do you think? Any comment/suggestion/problem that I overlooked?

ciao,
Anto
___
pypy-dev@codespeak.net
http://codespeak.net/mailman/listinfo/pypy-dev

Re: [pypy-dev] [pypy-svn] r73628 - in pypy/branch/cpython-extension/pypy/module/cpyext: . test

[Antonio Cuni]

Maciej Fijalkowski wrote:
> I probably have missed some part of discussion, but - shouldn't we
> have it present the same interface as lib/array.py? If lib/array.py is
> not needed, since we use the C version, how about killing lib/array?

lib/array is needed by OO backends, which can't use the C version
___
pypy-dev@codespeak.net
http://codespeak.net/mailman/listinfo/pypy-dev

Re: [pypy-dev] Branches

[Antonio Cuni]

On Tue, Mar 23, 2010 at 10:43 PM, Armin Rigo  wrote:

>    * 55751  eval-loop-experiments (antocuni)

"maybe one day"

In this branch, I made some changes to the main eval loop and in some
cases I observed speedups up to 20% (which are less relevant now that
we have a jit, but still), but I never managed to reproduce them
consistently.  I'd like to keep it around to investigate if the
changes are worth of or not.

ciao,
Anto
___
pypy-dev@codespeak.net
http://codespeak.net/mailman/listinfo/pypy-dev

Re: [pypy-dev] Nightly builds are unavailable

[Antonio Cuni]

On Tue, Mar 23, 2010 at 7:43 PM, Gasper Zejn  wrote:

> Nope, not working. The "directory listing" works, but not the links. Did you
> click on any of the links provided on that url?

Ah no, sorry. My fault, the links indeed don't work.

ciao,
Anto
___
pypy-dev@codespeak.net
http://codespeak.net/mailman/listinfo/pypy-dev

Re: [pypy-dev] Nightly builds are unavailable

[Antonio Cuni]

On Tue, Mar 23, 2010 at 6:02 PM, Gasper Zejn  wrote:
> The links at the URL http://buildbot.pypy.org/nightly/ are failing with
> "resource unavailable"; misconfiguration?
>
> Just to make sure it's a known problem.

It's working for me right now. Maybe it has just been down for a while?
___
pypy-dev@codespeak.net
http://codespeak.net/mailman/listinfo/pypy-dev

Re: [pypy-dev] autonosiness in roundup

[Antonio Cuni]

Benjamin Peterson wrote:
> 2010/3/16 Maciej Fijalkowski :
>> Anyone knows how to set up autonosy on roundup? I only get information
>> about new tickets and then I don't get followups.
> 
> When that's figured out, I'd also like to be on autonosy.

me too :-)
___
pypy-dev@codespeak.net
http://codespeak.net/mailman/listinfo/pypy-dev

Re: [pypy-dev] speed.pypy.org launched

[Antonio Cuni]

Carl Friedrich Bolz wrote:

> I disagree with this, please read the following paper:
> 
> http://buytaert.net/files/oopsla07-georges.pdf

+1 for this paper. I was about to link it as well, but cf has been faster.
I looked at the unladen swallow runner when I was writing my thesis, and I can 
confirm that it does the "statistically right" thing as described in the paper 
to compute error bars.

ciao,
Anto
___
pypy-dev@codespeak.net
http://codespeak.net/mailman/listinfo/pypy-dev

Re: [pypy-dev] pypy-jvm doesn't work?

[Antonio Cuni]

Hi Olli,

Olli Wang wrote:
> Hi, I tried to execute the compiled Python interpreter with jvm backend 
> on both Gentoo Linux(amd64) and Snow Leopard. 
> But unfortunately both of them failed to start. I simply ran 
> "./translate.py --backend=jvm targetpypystandalone.py" and "pypy-jvm" on 
> both computer. Do I miss something? Thanks.
> 
> ==
> Error messages on Gentoo Linux (amd64)
> 
> Exception in thread "main" java.lang.VerifyError: (class: 
> pypy/ConstantInit_0, method: constant_init signature: ()V) Expecting to 
> find long on stack
> at pypy.ConstantInit.init(ConstantInit.j:6)
> at pypy.Main.(Main.j:26)
> Could not find the main class: pypy.Main. Program will exit.

I suppose it's a 32/64 bit issue again: the problem is that when you run 
./translate.py with a 64 bit python, it assumes that 'int' variables are 64 
bits, but on the JVM they are always 32 bit. For what I know, the CLI backend 
has exactly the same problem.

As a workaround, you can try to run translate.py under a 32bit chroot (which 
works for sure, as I use it daily to develop pypy) or with a 32bit python 
(which should work, but I've never tried).

ciao,
Anto
___
pypy-dev@codespeak.net
http://codespeak.net/mailman/listinfo/pypy-dev

Re: [pypy-dev] Parallel translation?

[Antonio Cuni]

Benjamin Peterson wrote:

> No, that's not planned at the moment. The only time we could easily
> use multiple CPU at the moment is compiling the final C sources. You
> can even do this manually with the make file in the temp source
> directory.

I agree that at this point in time we cannot or don't want to make 
annotation/rtyping/backend parallelizable, but it should definitely be 
possible to just pass the -j flag to 'make' in an automatic way.

Anyone feeling like to implement it? :-)
___
pypy-dev@codespeak.net
http://codespeak.net/mailman/listinfo/pypy-dev