Re: [pypy-dev] docs on doc.pypy.org
On 03/05/11 11:55, holger krekel wrote: Hi all, i created a CNAME for our use of pypy.readthedocs.org and so you can now reach pypy docs under the frontend URL of http://doc.pypy.org It might be better to promote this link also from pypy.org so that URLs have a higher chance to remain valid even if we decide to move away from readthedocs some day. yes, good idea. Moreover, I think we should change the link from Dev Documentation to just Documentation, because it contains docs which are generally useful also for users (e.g., the getting started, the papers, etc.) Alternatively, we could have a heavier refactoring of the documentation and have clearly separated sections for user docs and dev docs. Moreover, we could see to have (another) sphinx setup for sphinx setup for our current pypy.org so we can host it there as well, maybe under www.pypy.org -- we need to have a subdomain/CNAME as we cannot redirect the root domain to readthedocs. I'm not a sphinx expert, but I don't think that the default layout (i.e., with a sidebar on the left) is the best for the home page of pypy.org. ciao, Anto ___ pypy-dev@codespeak.net http://codespeak.net/mailman/listinfo/pypy-dev
Re: [pypy-dev] cpyext: Detecting pypy and other issues
Hello Roger, I'm not the cpyext expert here, so I'll let the others to answer your specific questions. However: On 02/05/11 06:55, Roger Binns wrote: Unfortunately disabling the JIT wasn't much help. My goal is to produce the best bug report possible and use the least amount of the pypy team's time. (It is a lot easier to fix and understand things starting from a good bug report.) What would be most helpful is if the cpyext page gave some instructions to follow in particular making an appropriate pypy and using valgrind. (Usually making valgrind work well requires extra options above and beyond regular debugging builds.) you are right, we lack such a document. Do you feel like writing it? :-) (seriously: you are by far in a better position than us for knowing what could or could not help a newcomer. Even if you don't know the answer of all the questions, having a document with XXX that we can fix would be valuable). ciao, Anto ___ pypy-dev@codespeak.net http://codespeak.net/mailman/listinfo/pypy-dev
Re: [pypy-dev] [pypy-svn] pypy default: fixed test_circular
Hi Hakan, thanks for the commits +# We want to check that the array bound checks are removed, +# so it's this part of the trace. However we dont care about +# the force_token()'s. Can they be ignored? yes, I think they can be just ignored, because AFAIK operations without side effects and whose result is unused, are removed by the backend regalloc. ciao, Anto ___ pypy-dev@codespeak.net http://codespeak.net/mailman/listinfo/pypy-dev
Re: [pypy-dev] [pypy-svn] pypy default: fixed test_circular
Right. My point was that since we dont care if they are there or not the test should not test that they are there and fail if they are not. So if there is an easy way to ignore them in this new test_pypy_c framework (which is very cool by the way), we should. If it's not easy I'm fine with keeping the test as it is. My main motivation here is to learn about the new framework :) ah, I understand now. No, ignoring all force_tokens at once is not possible at the moment, but I agree that it would be a nice feature, I think I'll implement it later. Btw, I fear I need more of your help with test_silly_max and test_iter_max (see 2e5bd737be0c): what do we want to check there? ___ pypy-dev@codespeak.net http://codespeak.net/mailman/listinfo/pypy-dev
Re: [pypy-dev] [pypy-svn] pypy default: fixed test_circular
On 15/04/11 10:50, Hakan Ardo wrote: Hi, the point here is that we want max(a,b) to be turned into a single guard while we dont want max(*range(300)) and max(range(300)) to blow up into 300 guards, since that might lead to 2**300 different traces. I'm not sure how to best test this... can't we just check that the loop contains a residual call to min_max_loop? ___ pypy-dev@codespeak.net http://codespeak.net/mailman/listinfo/pypy-dev
Re: [pypy-dev] [pypy-svn] pypy default: port test_intbound_addsub_ge to test_pypy_c_new
Hi Hakan, On 14/04/11 14:53, Hakan Ardo wrote: +def test_intbound_addsub_ge(self): +def main(n): +i, a, b = 0, 0, 0 +while i n: +if i + 5 = 5: +a += 1 +if i - 1 = -1: +b += 1 +i += 1 +return (a, b) +# +log = self.run(main, [300], threshold=200) +assert log.result == (300, 300) +loop, = log.loops_by_filename(self.filepath) +assert loop.match( +i10 = int_lt(i8, i9) +guard_true(i10, descr=...) +# XXX: why do we need ovf check here? If we put a literal 300 +# instead of n, it disappears With n==sys.maxint, the operation i+5 would be the one overflowing. yes, you are right of course. I was just thinking nonsense, but I realized only after I asked the question :-). Of course the ovf check needs to be there because we don't specialize the loop on the value of n. Although it might be cool to be able to do promotion at applevel, for those who really want :-) ciao, Anto ___ pypy-dev@codespeak.net http://codespeak.net/mailman/listinfo/pypy-dev
Re: [pypy-dev] [pypy-svn] pypy default: port test_intbound_addsub_ge to test_pypy_c_new
On 14/04/11 16:44, Hakan Ardo wrote:
Second though, that will recompile on every call, but if we cache the
promote functions:
def main(n, promoters={}):
i, a = 0, 0
if n not in promoters:
exec def promote(n):
assert n==%d % n
promoters[n] = promote
else:
promote = promoters[n]
while i n:
promote(n)
a += i+5
i += 1
return a
we will actualy get rid of the extra ptr_eq and guard_false too...
wow, that's advanced... and without knowing the internals of the JIT, it
really looks like black magic.
While we are at it and if you have time/feel like, could you please have a
look to test_zeropadded and test_circular in test_pypy_c_new? It's not clear
to me what they are supposed to check (note that it's fine if you say they
just check that the program works correctly :-)).
___
pypy-dev@codespeak.net
http://codespeak.net/mailman/listinfo/pypy-dev
[pypy-dev] Possible sprint in Genova before/after Europython
Hi all, as we were discussing yesterday in another thread, the post-europython sprint will be only two days long, and so we might want to have a longer one either before or after europython. I am considering organizing one in my place. It could be either in Genova or more preferably in some other town in the nearby of the italian riviera. The place is ~3hr away from Florence by train. Googling images for pegli or arenzano (i.e., two of the towns we could go to) shows pictures which (I think) should encourage people to come :-) http://tinyurl.com/655p2at http://tinyurl.com/67q67r3 My idea is to find a hotel that gives us a sprinting room and internet in exchange of N people sleeping there, as we do in leysin. But before asking them, I need to have a rough idea about which value of N we are talking about (of course the higher the more interesting for them it is). Thus, I ask everybody who is potentially/likely interested in coming to let me know. Also, would you prefer to do it before or after europython? ciao, Anto ___ pypy-dev@codespeak.net http://codespeak.net/mailman/listinfo/pypy-dev
Re: [pypy-dev] Possible sprint in Genova before/after Europython
On 13/04/11 12:27, Maciej Fijalkowski wrote: Not that I can't google myself, but those links are broken uhm, indeed. These seems to work :-) http://bit.ly/e6vHkh http://bit.ly/fkHwMu ___ pypy-dev@codespeak.net http://codespeak.net/mailman/listinfo/pypy-dev
Re: [pypy-dev] Possible sprint in Genova before/after Europython
On 13/04/11 14:53, Jacob Hallén wrote: Wednesday 13 April 2011 you wrote: In a message of Wed, 13 Apr 2011 12:20:26 +0200, Antonio Cuni writes: Also, would you prefer to do it before or after europython? I am coming, and both times are fine for me. I am going to be spending the week after Europython in Italy somewhere anyway, prior to going kayaking in Sicily. Corsica, not Sicily. uhm, how do you go there? If by boat, it's very likely that you'll start by genova or savona (which is also close, ~50 km) ___ pypy-dev@codespeak.net http://codespeak.net/mailman/listinfo/pypy-dev
Re: [pypy-dev] thinking about the EuroPython sprint
On 12/04/11 08:52, Armin Rigo wrote: Then what about a short week's sprint before the conference, plus the two days after? if people are interested, I might try to organize a sprint in Genova or surrounding (which is ~3h away from florence by train). However, I cannot assure that I'll be able to find a suitable place, because I see only two options: - ask the uni (but there are not really many rooms suitable for a sprint -- just one or two, and I think it'll be hard to reserve them for a whole week) - find a place a la leysin, i.e. a hotel which gives also us a room for the sprint. However, middle of june is already vacation time here, so it is possible that hotels are full anyway and prefer to have normal visitors instead of nerds which stay there all the time :-) ciao, Anto ___ pypy-dev@codespeak.net http://codespeak.net/mailman/listinfo/pypy-dev
Re: [pypy-dev] Pre sprint beer session Sunday 24th of April
On 11/04/11 14:47, Bea During wrote: I still would like to meet up so I invite you to a pre sprint beer session at Bishops Arms pub in central Gothenburg, Sunday 24th of April, starting from 18:00. [cut] p.s. Anto - will miss you there d.s yeah, unfortunately I won't be able to make it :-( Have fun! ciao, Anto ___ pypy-dev@codespeak.net http://codespeak.net/mailman/listinfo/pypy-dev
Re: [pypy-dev] The JVM backend and Jython
Hi Frank, On 30/03/11 04:40, fwierzbi...@gmail.com wrote: [cut] So to my question - just how broken is the JVM backend? Are there workarounds that would allow the Java code to get generated? so, now the jvm (and cli) translation works again. You can just type ./translate.py -b jvm, and the fish the relevant .class/.j files from /tmp/usession-default-*/pypy. ciao, Anto ___ pypy-dev@codespeak.net http://codespeak.net/mailman/listinfo/pypy-dev
Re: [pypy-dev] Pypy custom interpreter JIT question
On 04/04/11 17:43, Andrew Brown wrote: Sure! What needs to be done to turn it into a blog post and get it posted? I assume there are format considerations, but I'm also open to any content suggestions and feedback before it goes live. Hello Andrew, thanks for the tutorial, it's really well written and easy to read. Two notes: 1) do you know about the existence of rlib.streamio? It's is part of the RPython standard library and it allows you to read/write files in a higher level way than file descriptors 2) Maybe the tutorial is a bit too long to fit in just one post; what about splitting it into two parts? (e.g., one until Adding JIT and one after). ciao, Anto ___ pypy-dev@codespeak.net http://codespeak.net/mailman/listinfo/pypy-dev
Re: [pypy-dev] Pypy custom interpreter JIT question
On 04/04/11 19:46, Andrew Brown wrote: 1) do you know about the existence of rlib.streamio? It's is part of the RPython standard library and it allows you to read/write files in a higher level way than file descriptors No, I didn't. That's good to know. I don't think it's worth updating the examples though, so unless you disagree, I'll just add a note about this module's existence. sure, I think that for this example, using fd is fine. Btw, in case you want to do more with pypy, having a look to rlib might be a good idea, there is useful stuff there :) ciao, Anto ___ pypy-dev@codespeak.net http://codespeak.net/mailman/listinfo/pypy-dev
Re: [pypy-dev] Pypy custom interpreter JIT question
On 31/03/11 14:28, Andrew Brown wrote: In any case, I'm satisfied with the speed. It's still beaten by a BF to C translator combined with gcc -O2 though, that'd be a tough case to beat. =) what happens if you combine the BF to C with gcc -O0 or -O1? Anyway, I think that if you feel like writing a post explaining your experience with using pypy and its jit for writing an interpreter, we could publish it on our blog. I suppose it would be useful/interesting for other people as well. What do the others think? ___ pypy-dev@codespeak.net http://codespeak.net/mailman/listinfo/pypy-dev
Re: [pypy-dev] Pypy custom interpreter JIT question
On 31/03/11 22:05, Andrew Brown wrote: python double-interpreted: 78m (did not finish) pypy-c (with jit) double-interpreted: 41m 34.528s this is interesting. We are beating cpython by more than 2x even in a worst case scenario, because interpreters in theory are not a very good target for tracing JITs. However, it's not the first time that we experience this, so it might be that this interpreter/tracing JIT thing is just a legend :-) translated interpreter no jit: 45s translated interpreter jit: 7.5s translated direct to C, gcc -O0 translate: 0.2s compile: 0.4s run: 18.5s translated direct to C, gcc -O1 translate: 0.2s compile: 0.85s run: 1.28s translated direct to C, gcc -O2 translate: 0.2s compile: 2.0s run: 1.34s these are cool as well. We are 3x faster than gcc -O0 and ~3x slower than -O1 and -O2. Pretty good, I'd say :-) ciao, anto ___ pypy-dev@codespeak.net http://codespeak.net/mailman/listinfo/pypy-dev
Re: [pypy-dev] The JVM backend and Jython
On 31/03/11 21:57, Maciej Fijalkowski wrote: Ok, so if Ademan tells me that he's not going to work on the ootype-virtualref branch, I'll try to finish the work so you can start playing with it. Note to frank: this is kind of cool but only needed for the JIT, otherwise it's a normal reference. well, no. Virtualrefs were introduced for the JIT, but they also need to be supported by normal backends. This is why translation is broken at the moment. It is true that the implementation is straightforward, though (I suppose this is what you meant originally :-)) ciao, Anto ___ pypy-dev@codespeak.net http://codespeak.net/mailman/listinfo/pypy-dev
Re: [pypy-dev] The JVM backend and Jython
Hi Frank, On 30/03/11 04:40, fwierzbi...@gmail.com wrote: Hi all, It was nice meeting up with many of you at PyCon! I've been thinking about the first steps towards collaboration between the Jython project and the PyPy project. It looks like it isn't going to be too long before we are all (CPython, PyPy, IronPython, Jython, etc) working on a single shared repository for all of our standard library .py code. In my ideal world there would come a day when there is also no standalone Java code in the Jython project: that is the shared standard library would contain all of Jython's .py files, and all of the Java would be generated from PyPy and Jython as a standalone project would disappear. It is possible that this is too ambitious, but big goals are more fun, right? In reality even if this where to get going, I imagine it would be a 10+ year plan :) wow, that's definitely a nice (and big) plan :-) So to my question - just how broken is the JVM backend? Are there workarounds that would allow the Java code to get generated? not much broken. Last time I tried, the only broken thing was virtualrefs, which is something needed for the jit, but that at the moment is not supported at all by ootype and thus it blocks the translation. However, I think that fixing it is probably very easy: there is a branch for this which was started by Ademan: https://bitbucket.org/pypy/pypy/src/ootype-virtualrefs Dan, do you plan to finish the work on it? Else, I can just do it probably. I ask because I would like to evaluate the generated Java parser as a potential replacement for our current ANTLR based parser. It seems like a nice baby step towards real collaboration since it seems like a relatively easy place to start. Clearly it would need adjustments to actually work for Jython - but I'd be able to look into that part. I don't think I have the time to try to unbreak the translation though... I have to warn you that at the moment, you cannot invoke any Java code from RPython. Implementing it has been on my todo list for years now :-(, but I never managed to find the time and the motivation to do it. However, for using the PyPy parser inside Jython it should be enough to do the other way around, i.e. call RPython code from Java, which should be possible. ciao, Anto ___ pypy-dev@codespeak.net http://codespeak.net/mailman/listinfo/pypy-dev
Re: [pypy-dev] The JVM backend and Jython
On 30/03/11 19:37, fwierzbi...@gmail.com wrote: My thoughts here are taking a very primitive step - that is run the JVM translation and look at the generated Java - then see what needs to be modified so that I could use the generated Java parser from Jython. At this stage I would be using PyPy exactly the way I use ANTLR now - as a parser generator. There wouldn't be any need at all for calling into Java code (as far as I can think of). yes, I think it makes sense. Actually, as Leonardo says we don't generate java code but assembler which is converted to .class by jasmin. However, it should not change anything. I think if we Jython developers get some experience with PyPY - we might be able to help with the task of calling into Java from PyPy - since we know a bit about that :) that would be extremely cool :-) Ok, so if Ademan tells me that he's not going to work on the ootype-virtualref branch, I'll try to finish the work so you can start playing with it. ciao, Anto ___ pypy-dev@codespeak.net http://codespeak.net/mailman/listinfo/pypy-dev
[pypy-dev] Hacking dropbox
Hi, I don't have any news from the dropbox side, but as an aside note, I found a project which is open source and implements the very same technique I used for dropbox, and additionally it also decompiles back to the source code: http://code.google.com/p/pyretic/ ciao, Anto ___ pypy-dev@codespeak.net http://codespeak.net/mailman/listinfo/pypy-dev
Re: [pypy-dev] status of the graphviz server on codespeak?
On 23/03/11 02:36, Maciej Fijalkowski wrote: On Tue, Mar 22, 2011 at 7:17 PM, Laura Creighton l...@openend.se wrote: getting-started-dev says: Download and install `Dot Graphviz`_ (optional if you have an internet connection: the flowgraph viewer then connects to codespeak.net and lets it convert the flowgraph by a graphviz server). What's going to happen to that when codespeak goes away? This one is defunct for ages now I think. no, I think it's still up: http://codespeak.net/pypy/convertdot.cgi we can either migrate it to some other machine (tannit?) or discontinue the service. Nowadays, it's only useful for us developers, and (I think) we all have graphviz installed locally. ciao, Anto ___ pypy-dev@codespeak.net http://codespeak.net/mailman/listinfo/pypy-dev
Re: [pypy-dev] [pypy-svn] pypy fold_intadd: Changed test to reflect my optimizeopt's decision to emit int_sub(iX, -x) when x 0
Hi Daniel, On 19/03/11 04:45, ademan wrote: Author: Daniel Roberts ademan...@gmail.com Branch: fold_intadd Changeset: r42802:386510d0fb45 Date: 2011-03-18 20:41 -0700 http://bitbucket.org/pypy/pypy/changeset/386510d0fb45/ Log: Changed test to reflect my optimizeopt's decision to emit int_sub(iX, -x) when x 0 what happens when you are on 32 bit and see int_add(i0, -2147483648)? ciao, Anto ___ pypy-dev@codespeak.net http://codespeak.net/mailman/listinfo/pypy-dev
[pypy-dev] ideas for Google Summer of code
Hi all, the deadlines for GSoc are approaching, and at some point we should probably make a blog post about that. But first, we need to 1) collect ideas for possible tasks and 2) find potential mentors. Two ideas that just came to my mind: - general work on speed.pypy.org (we need to define better what we want, of course) - improving the jitviewer, maybe integrating it with the profiler (when we'll have one :-)) - insert-your-idea-here :-) ciao, Anto ___ pypy-dev@codespeak.net http://codespeak.net/mailman/listinfo/pypy-dev
Re: [pypy-dev] ideas for Google Summer of code
On 17/03/11 21:25, Dan Stromberg wrote: On Thu, Mar 17, 2011 at 10:23 AM, Leonardo Santagada santag...@gmail.com mailto:santag...@gmail.com wrote: 7 - make pypy on .net jit work (or on java) this is probably a too large task for GSoc. For one, before working on the JIT it is necessary to make normal translation working, because right now ootype backends are broken. IIRC, it's not too hard because it consists of porting virtualrefs to ootype, which should be simple. But it is possible that there are other issues after that, since ootype translations have not run since a long time (more than 1 year, I think). About the JIT: JIT on .NET is not going to be any fast. I did it for my thesis when the JIT was more .NET friendly (no virtualrefs) and results were interesting as a research project, but not good enough to be used in production. The JIT for pypy-jvm is an open topic: the JVM has the potential to do a much better job than the CLI, but we cannot know until we really try. Having a working pypy-jvm-jit is a lot of work, though. It consists of: 1) make pypy-jvm (without jit) working again 2) design and implement a way to use Java objects at RPython level: this is needed to write the backend 3) port the JIT to ootype again. Should not be too hard, but there are going to be issues, because the codewriter has been heavily refactored since last year, when the JIT on ootype worked well 4) write the backend All together, it's probably huge for GSoc. But e.g 1+2 could fit it; we already had a GSoc on this topic few years ago, but it didn't work well. This reminds me: it might be good to make the JVM PyPy be able to call native Java code - on a typical JRE, and on Android. Last yes, that's another interesting topic. It requires both points 1 and 2 above, though. Once we have that, it should not be too hard, as it has already been implemented for .NET and could use the same techniques (and probably reuse also most of the code). ciao, Anto ___ pypy-dev@codespeak.net http://codespeak.net/mailman/listinfo/pypy-dev
Re: [pypy-dev] ideas for Google Summer of code
Hi Wim, On 17/03/11 19:33, wlavrij...@lbl.gov wrote: Hi Anto, - insert-your-idea-here :-) the Reflex work has at one time before been proposed as a GSoC. Now that a prototype is in place, there are several minor tasks that can be done, which can lead to bigger/more research tasks as desired/appropriate. good idea! How does this work anyway, do you need to come with your own student? not necessarily. At this stage, the goal is to collect ideas which are reasonable, then publish it and see which students are interested in them. However, nothing stops you to suggest a student of course (especially if you already know him and you are confident that can do the job well). ciao, Anto ___ pypy-dev@codespeak.net http://codespeak.net/mailman/listinfo/pypy-dev
Re: [pypy-dev] problem after merging of jit-virtual_state
Hi Hakan, thank you for the deep explanation. Now I understand what's going on :-) So, I changed test_pypy_c_new to add a sys.setcheckinterval(some-huge-number), so that the bridge from the signal/thread counter is never created and we can forget about it. Now, if I understand correctly, the two remaining loops are one for the case i non virtual and the other for the case i virtual, although both lead to the same operations. I think this is the expected behavior in this case, so are you ok if I just fix test_f1 to expect two loops? ciao, Anto On 13/03/11 11:12, Hakan Ardo wrote: Hi, this is what happens here: 1. The inner loop is traced and Loop0 is produced with preamble Loop1 2. A bridge from Guard3 (the test in the while) back to Loop0 is traced (i.e the remaining parts of the outer loop) 3. At the end of this bridge the VirtualState does not match the VirtualState of Loop0, so the loop is retraced 4. The VirtualState of the newly traced version of the loop does not match the VirtualState at the end of the bridge so the bridge has to jump to the preamble instead of jumping to the new specialized version of the loop. 5. A bridge from Guard6 (signal/thread counter) is traced and the same thing happens for this bridge. This means that the additional two versions of the loop will never be used and should hopefully be reomved by the gc... So there are two issues: A. The additional specialized versions created does not become usable. This is the issue I'm working on in the jit-usable_retrace branch. The idea there is to have the retrace inherit the OptValue's of the jumpargs at the end of the bridge. This will become a fairly large change functionality wise... B. The VirtualStates' s differs in the first place forcing a retrace. This is probably fixable by introducing some more cases in NotVirtualInfo._generate_guards(). The jit-usable_retrace branch contains more cases than trunk, don't know if those are enough for this test though... Note however that jit/metainterp/test/test_nested_loops_discovered_by_bridge in test_loop_unroll.py, which conatins the same loop for a simple interpreter, does work nicely, wihtout the issues above. On Sat, Mar 12, 2011 at 10:59 PM, Hakan Ardo ha...@debian.org wrote: On Sat, Mar 12, 2011 at 8:34 PM, Antonio Cuni anto.c...@gmail.com wrote: Hi Hakan, On 12/03/11 19:25, Hakan Ardo wrote: Yes, this is probably the VirtualState checking. It will retrace a loop whenever the VirtualState at the end of a bridge differs from the VirtualState at the beginning of the compiled trace (any of the compiled traces). This might indeed produce an identical trace if we are unlucky, but the idea is that this should only happen rarely. ok, that's clear. So, hopefully this particular example looks a bit bad, but in general it should not be an issue. It'd be nice to have a way to check this thesis, but I agree that it's a bit hard. We should probably log the VirtualState together with the produced loops and bridges. That would allow us to see how they differ when a new version of a loop is traced. There are __repr__ methods I've been using for that while debugging. They might need some rpythonizing to translate though--- This is because the VirtualState at the beginning of a trace is the state of all the OptValue of the inputargs produced during the optimization of the trace. This does not have to be the most general state for which the trace is usable (which would be hard to calculate I'm afraid). so, if I understand correctly, this is what happens: 1. we trace, optimize and compile loop A 2. after a while, we trace, optimize a compile a bridge B which then jumps back to A; by chance, the bridge looks the same as the loop Am I right? Maybe, I've not had the chance to look into any details yet. I'll do that tomorrow... A few cases that would (most likely) result in identical traces are salvaged in NotVirtualInfo._generate_guards by producing some extra gurads at the end of a bridge to make the VirtualState there match the VirtualState of a compiled trace. This is however only done if the guards would (most likely) not fail for the traced iteration. I'll look into what's happening in this particular test... I just did a quick check because I'm in a hurry, but from what I see we get three actual *loops*, not bridges. So if it's the same loop traced several times they should all have the same preamble, and the preamble would have two bridges leading to the two second versions of the loop. The preamble and it's two bridges should end with different VirtualState's. The loops should be specialized to the different VirtualState's, but if the VirtualState's are similar enough (but not equal) they might consist of the exact same operations. If there are 3 preambles for the same loop, there is a bug somewhere... -- Håkan Ardö ___ pypy
Re: [pypy-dev] problem after merging of jit-virtual_state
Hi Hakan, On 12/03/11 19:25, Hakan Ardo wrote: Yes, this is probably the VirtualState checking. It will retrace a loop whenever the VirtualState at the end of a bridge differs from the VirtualState at the beginning of the compiled trace (any of the compiled traces). This might indeed produce an identical trace if we are unlucky, but the idea is that this should only happen rarely. ok, that's clear. So, hopefully this particular example looks a bit bad, but in general it should not be an issue. It'd be nice to have a way to check this thesis, but I agree that it's a bit hard. This is because the VirtualState at the beginning of a trace is the state of all the OptValue of the inputargs produced during the optimization of the trace. This does not have to be the most general state for which the trace is usable (which would be hard to calculate I'm afraid). so, if I understand correctly, this is what happens: 1. we trace, optimize and compile loop A 2. after a while, we trace, optimize a compile a bridge B which then jumps back to A; by chance, the bridge looks the same as the loop Am I right? A few cases that would (most likely) result in identical traces are salvaged in NotVirtualInfo._generate_guards by producing some extra gurads at the end of a bridge to make the VirtualState there match the VirtualState of a compiled trace. This is however only done if the guards would (most likely) not fail for the traced iteration. I'll look into what's happening in this particular test... I just did a quick check because I'm in a hurry, but from what I see we get three actual *loops*, not bridges. ciao, Anto ___ pypy-dev@codespeak.net http://codespeak.net/mailman/listinfo/pypy-dev
[pypy-dev] problem after merging of jit-virtual_state
Hi Hakan, hi all, as you might have noticed, there is test_pypy_c_new.test_f1 which is currently failing: http://buildbot.pypy.org/summary/longrepr?testname=TestPyPyCNew.%28%29.test_f1builder=pypy-c-jit-linux-x86-32build=699mod=pypy.module.pypyjit.test_pypy_c.test_pypy_c_new buildbot did not show it earlier because before yesterday it did not run the test at all. As you can see from the traceback, the problem is that running f1 makes three loops instead of one as expected. I did a bit of manual binary search and discovered that the test started failing between r42273:b8bb5d085684 and r42305:3fe93dd500bd. Looking at the commit, it seems that the only one relevant to the problem is r42305:3fe93dd500bd, i.e. the merging of jit-virtual_state. Do you have any idea why jit-virtual_state causes this problem? (if it's a problem at all, but I think so because the three loops contain the very same operations, so I don't see why we need to compile the more than once). ciao, Anto ___ pypy-dev@codespeak.net http://codespeak.net/mailman/listinfo/pypy-dev
Re: [pypy-dev] problem after merging of jit-virtual_state
On 10/03/11 16:21, Antonio Cuni wrote: I did a bit of manual binary search and discovered that the test started failing between r42273:b8bb5d085684 and r42305:3fe93dd500bd. Looking at the commit, it seems that the only one relevant to the problem is r42305:3fe93dd500bd, i.e. the merging of jit-virtual_state. I confirm that the problem is the merging of jit-virtual_state: I just tried the revision just before the merge, and the test passes. ciao, Anto ___ pypy-dev@codespeak.net http://codespeak.net/mailman/listinfo/pypy-dev
Re: [pypy-dev] [PATCH] Fix segmentation fault on parsing empty list-assignments
On 09/03/11 13:29, Greg Price wrote: This same patch is on bitbucket at https://bitbucket.org/price/pypy, where I've sent a pull request. Holger Krekel suggested on IRC that I send mail here. If others have different preferences for how to submit a patch, let me know. Hi Greg, I have pushed your commit upstream, thanks for help! ciao, anto ___ pypy-dev@codespeak.net http://codespeak.net/mailman/listinfo/pypy-dev
Re: [pypy-dev] we need to fix the links on our blog posts.
you can get a link to whatever the latest revision is on bitbucket by manually substituting the mercurial id with default inside the link. On 3/3/11, Armin Rigo ar...@tunes.org wrote: Hi Laura, On Thu, Mar 3, 2011 at 8:37 AM, Laura Creighton l...@openend.se wrote: Do we need dcolish's bouncer to work for what used to be people's svn directories on codespeak? Actually, does bitbucket serve files straight from repositories, like http://codespeak.net/svn/some/path did? I tried a bit, but seem to only get links to specific revisions of files, not whatever the latest revision is. A bientôt, Armin. ___ pypy-dev@codespeak.net http://codespeak.net/mailman/listinfo/pypy-dev ___ pypy-dev@codespeak.net http://codespeak.net/mailman/listinfo/pypy-dev
Re: [pypy-dev] further pypy repo migrations
On 17/02/11 15:23, holger krekel wrote: Thanks to Ronny Pfannschmidt, sponsorship from merlinux (my company), and work from Armin, Anto and others the main PyPy development repository has been migrated to mercurial on Bitbucket. Btw, we just asked and Bitbucket nicely upgraded the PyPy project account to become an unlimited one, thanks! just to clarify: unlimited account means that we can have private repositories with as many users as we want (which is what we needed for pypy-z). Thanks to Bitbucket also from my side :-) Any more pypy related repositories that need migration or any notes/comments? currently, pypy has three svn subrepos which live on codespeak: greenlet = [svn]http://codespeak.net/svn/greenlet/trunk/c testrunner = [svn]http://codespeak.net/svn/pypy/build/testrunner lib_pypy/pyrepl = [svn]http://codespeak.net/svn/pyrepl/trunk/pyrepl/pyrepl we will need to migrate those as well at some point, and probably make them mercurial subrepositories instead of svn ones. ciao, Anto ___ pypy-dev@codespeak.net http://codespeak.net/mailman/listinfo/pypy-dev
Re: [pypy-dev] European sprints?
On 16/02/11 12:01, Bea During wrote: Here is a suggestion of places and dates based on Lauras, Carl Friedrich and Antos input: - Gothenburg sprint: 25th of April to 1st of May - Europython sprint/Florence: 25th of June to 26th of June (EP2011 official sprint dates) - Düsseldorf sprint: 22nd of August to 28th of August (fits the plan of the funded PyJIT project which ends end August) What do you think about these dates - would they work? for me they mostly work, although I might arrive one day later or depart one day earlier for the Gothenburg and Duesseldorf ones. ciao, Anto ___ pypy-dev@codespeak.net http://codespeak.net/mailman/listinfo/pypy-dev
Re: [pypy-dev] Separate compilation and friends
On 15/02/11 03:41, Dima Tisnek wrote: On a related note, how hard is it to freeze the translator/compiler state of a given pypy version just before it begins to read extension modules and distribute that, it would speed up module development a lot. It would be a quick equivalent of distributing headers for a C library. this is hard, because the compilation of the modules is intermixed with the compilation of the rest of the interpreter for each phase: we have (roughly) something like: - annotation of the interpreter - annotation of the modules - rtyping of the interpreter - rtyping of the modules - etc. etc. ciao, Anto ___ pypy-dev@codespeak.net http://codespeak.net/mailman/listinfo/pypy-dev
Re: [pypy-dev] google should build a pypy phone or someone
On 15/02/11 11:52, Massa, Harald Armin wrote: Antonio, Anyway, back to the original topic: what would python offer more than java for android? a much less challenging licence-situation. I'm not completely sure. AFAIK, the Oracle-Google lawsuit is about patents, not licences: in particular, they are complaining about how Dalvik is implemented, not about the fact that it implements Java. From what I could read at least a couple of those patents apply to PyPy as well (and to whatever language with JIT, fwiw), e.g. IIRC there is one about compiling an optimized version of code that executes often or something similar (whether this patent is valid is another topic, of course). ciao, Anto ___ pypy-dev@codespeak.net http://codespeak.net/mailman/listinfo/pypy-dev
Re: [pypy-dev] European sprints?
On 15/02/11 15:02, Carl Friedrich Bolz wrote: Also I guess it is likely that we will have another one in Düsseldorf later in the year. and probably one after europython, as usual? ciao, Anto ___ pypy-dev@codespeak.net http://codespeak.net/mailman/listinfo/pypy-dev
Re: [pypy-dev] testing floating point
On 03/02/11 13:13, Maciej Fijalkowski wrote: I'm not sure about warnings module. How about --jit warnings=1 ? That would fit with other jit options. not really. The other jit options really belongs to the JIT engine, while this one is dependent on the Python interpreter ciao, Anto ___ pypy-dev@codespeak.net http://codespeak.net/mailman/listinfo/pypy-dev
Re: [pypy-dev] New version of Codespeed (0.7) for speed.pypy.org
On 21/01/11 08:49, Miquel Torres wrote: @Anto Yes, branches are a pending item that has been requested a couple of times now. yes, I think most of the requests has been by me :) The current solution is actually not to abuse an environment like you say, but to create a new project for a branch. That way it gets cleanly separated, and in a way branches are like different projects. But it is of course not optimal. Technically it is very easy to come up with several solutions (add a branch dimension, for example), but interface-wise it is not easy to find something that doesn't clutter things. Uhm, I don't think that using a different project is a good idea. For branches, we are usually not much interested in the branch history, but in the comparison with trunk (ideally, with trunk at the point we created the branch, or at the point of the last merge from trunk). As for visualize changes, I think that we don't need anything fancy, for example it would be already immensely useful to have the possibility of displaying the Changes page in a way that it compares the performances of the branch against trunk. ciao, Anto ___ pypy-dev@codespeak.net http://codespeak.net/mailman/listinfo/pypy-dev
Re: [pypy-dev] New version of Codespeed (0.7) for speed.pypy.org
Hi Miquel On 20/01/11 22:00, Miquel Torres wrote: Hi all, I want to announce the release of Codespeed 0.7, the version which is now powering speed.pypy.org wow, that's very nice, thank you once more :-). From my side, there is only one feature that I miss a lot, which is the possibility to benchmark a branch. IIRC, at the moment if you just run benchmarks on a branch, they are appended to the list of results, which can be a bit confusing especially if you look at them weeks later. I think that maybe it's possible to do it by (ab)using an additional environment, but probably a more builtin solution is better. Do you have any idea/opinion/suggestion on this? ciao, Anto ___ pypy-dev@codespeak.net http://codespeak.net/mailman/listinfo/pypy-dev
Re: [pypy-dev] 1.4.1 threading
On 27/12/10 09:31, Carl Friedrich Bolz wrote:
The only case where improving dicts would help is for user-defined
dictionaries, i.e. when you write dicts with curly braces.
then it's easy to optimize, you just write dict() instead of {}.
(sorry, I could not resist :-))
___
pypy-dev@codespeak.net
http://codespeak.net/mailman/listinfo/pypy-dev
Re: [pypy-dev] long.__itemsize__
On 21/12/10 12:05, Maciej Fijalkowski wrote: __itemsize__ - in bytes, corresponds to item size field in the types definition structure. It's a field for types. See: http://docs.python.org/c-api/typeobj.html#tp_itemsize Well... Those are docs for C API. It doesn't say it's exposed at applevel nor since which version. (Also, to be precise, C API is known to be implementation specific) Moreover, I don't think we could give it a sane semantics on PyPy, given that the same applevel type can be potentially implemented by many different low level structures with different sizes. ciao, Anto ___ pypy-dev@codespeak.net http://codespeak.net/mailman/listinfo/pypy-dev
Re: [pypy-dev] playing with fast-forward
On 18/12/10 23:58, Gary Robinson wrote: I'm experimenting with the fast-forward branch. I'm actually not sure about the proper way to get it. (I have the main branch working fine.) I downloaded a nojit version from http://buildbot.pypy.org/nightly/branch/fast-forward/ since I didn't see any jit versions listed there. I was happy to see that the method I care most about, multiprocessing.Pool.imap_unordered, seemed to work. But I want to test it with the jit. consider that we don't have automatic nightly builds for fast-forward, so the ones you find on that page are manually triggered, and possibly outdated. The page (https://bitbucket.org/pypy/pypy/src/021b219e0aef) appears to be for the branch, but the mercurial URL shown on that page appears to be for the main project. yes, if you clone the mercurial repo you get all the branches together. I tried the downloads under the Source pop-up menu. For some reason, I only sporadically am able to get a complete .gz file.** But I did get one. I was able to run: python translate.py -Ojit successfully -- or at least it appeared so. But I couldn't find a bin/pypy to run!? as suggested by Alex, you probably have a pypy-c binary in pypy/translator/goal (I think that the translate script even says so, but I agree that it produces so much output that it might be hard to stop the message :-)) Any suggestions how to run fast-forward with jit? I don't know how the download source button of bitbucket works, so I'm not even sure that you downloaded the fast-forward branch instead of the default one. The best way is to do this: $ hg clone http://bitbucket.org/pypy/pypy $ cd pypy $ hg up -r fast-forward Now you can go to pypy/translator/goal and run translate.py again. ciao, Anto ___ pypy-dev@codespeak.net http://codespeak.net/mailman/listinfo/pypy-dev
Re: [pypy-dev] speed.pypy.org and mercurial
Hi Miquel,
On 16/12/10 09:29, Miquel Torres wrote:
Hi Anto,
yes, that is expected, but no problem. Codespeed is designed in such a
way that it can support different version control systems. It is just
that there is only support for svn now ;-)
So for mercurial support, I need to implement a function that takes a
changeset hash, connects to the hg repo and returns a dict with
'date', 'user' and 'description' or 'summary' info. Anybody knows what
library I could use for that? or do I have to parse local 'hg log'
output?
I think you have several options to do that. For the particular pypy case,
maybe the simplest is to directly ask bitbucket for the changeset:
https://bitbucket.org/pypy/pypy/changeset/c55286db0781/raw/
this has the advantage that you don't need any library but just an http
request, but of course it works only for bitbucket repos.
Alternatively, if you install mercurial you can then import
mercurial.commands and use its public API from Python. Or as you said you can
just execute hg log and parse the output: in this case you might be interested
in the --template option, which allows you to format things exactly as you
want, e.g.:
$ hg log -r c55286db0781 --template '{author}\n{date|isodate}\n{desc}\n'
Armin Rigo ar...@tunes.org
2010-12-16 10:01 +0100
Document this publicly.
see this page for more details:
http://hgbook.red-bean.com/read/customizing-the-output-of-mercurial.html
ciao,
Anto
___
pypy-dev@codespeak.net
http://codespeak.net/mailman/listinfo/pypy-dev
[pypy-dev] speed.pypy.org and mercurial
Hi Miquel, hi all, as you probably have noticed, we have recently migrated the main repo to mercurial. Now speed.pypy.org receives a revision number in the form 40046:2088ce763fc2, but of course it can no longer fetches the commit logs from the svn server. Would it be possible to fetch the commits from the bitbucket repo, please? ciao, Anto ___ pypy-dev@codespeak.net http://codespeak.net/mailman/listinfo/pypy-dev
Re: [pypy-dev] Migration to mercurial
On 14/12/10 00:02, Antonio Cuni wrote: Hi all, finally, it's happening :-). Thanks to Ronny's work, we are going to complete the migration to mercurial very soon. so, the migration is done! The svn repo is readonly, and from now the official pypy repo is this one: http://bitbucket.org/pypy/pypy There are still some rough edges to fix on buildbot, but for the rest everything seems to work fine. I would like to thank everyone who was involved in the migration, in particular Ronny who did most of the dirty work :-) ciao, Anto ___ pypy-dev@codespeak.net http://codespeak.net/mailman/listinfo/pypy-dev
Re: [pypy-dev] [pypy-svn] pypy commit 7b8aa74da1fb: Handle output on standard error when invoking Mercurial.
On 14/12/10 20:28, Dan Villiom Podlaski Christiansen wrote: On 14 Dec 2010, at 20:01, Maciej Fijalkowski wrote: This commit contains no tests whatsoever, it would be cooler if it did. Antonio kindly wrote a test for the functionality in an earlier changeset. Ironically, I broke it that change, but that's fixed now :) anyway, maciek on the channel mentioned a way to test it without messing with the hg repo: you can provide a custom hgexe script which returns whatever you want. Or even, restructure the code in a way that the running of the hg command is decoupled by the parsing of its results, so that you can at least test the latter. Sorry, I should have pointed this out when you asked me to review the patches, but today I was fried and had thousands of things to look at :-/ ciao, Anto ___ pypy-dev@codespeak.net http://codespeak.net/mailman/listinfo/pypy-dev
[pypy-dev] Migration to mercurial
Hi all, finally, it's happening :-). Thanks to Ronny's work, we are going to complete the migration to mercurial very soon. The pypy buildbot is already configured to pull from the bitbucket repository: http://bitbucket.org/pypy/pypy I have tried to run a test build, and it seems to work ok. We will see tomorrow morning how the nightly runs went. If everything goes right, tomorrow we will declare the hg repo as the official one: the svn repo will be made read-only (as soon as armin does it :-)), the mercurial one will be synchronized to include the latest svn commits, and from that point everyone should start to commit to mercurial. If you don't have write access to the bitbucket repo, please ask me or anyone else who is an admin. A note for people working on active branches: before you can work on those, you need to perform two simple manual steps, as detailed in this howto: http://codespeak.net/svn/pypy/extradoc/planning/hg-migration/active-branches-howto.txt ciao, Anto ___ pypy-dev@codespeak.net http://codespeak.net/mailman/listinfo/pypy-dev
Re: [pypy-dev] setrecursionlimit deprecation?
On 11/12/10 18:32, Armin Rigo wrote: But then people are going to complain that their app seems to hang up on pypy, and neither they nor we have any clue what is going on --- until we figure out that they used sys.setrecursionlimit() to put a bound on recursion and catch the RuntimeError. That's at least the reason for which I suggested that calling sys.setrecursionlimit() should at least print a warning. Now I agree that maybe the message of the warning is not the clearest one. I was about to propose to change the message into something like sys.setrecursionlimit is ignored by PyPy. I think this is already an improvement over the current message, but has the drawback than then people will complain that pypy might run out of stack (which is false, but not apparent by the warning message). Also, we should maybe change the class of the Warning. IIRC, DeprecationWarning is going to be ignored by default on python2.7, which means that most people won't see the warning at all once we merge fast-forward. ciao, Anto ___ pypy-dev@codespeak.net http://codespeak.net/mailman/listinfo/pypy-dev
Re: [pypy-dev] [PyPy Status Blog] New comment on PyPy 1.4: Ouroboros in practice.
On 01/12/10 23:21, Amaury Forgeot d'Arc wrote: [cut] There is already an ongoing effort to port PyPy to Python 2.7. But we need some help! It's a good way to become a PyPy developer. And no, you don't have to be a JIT expert to implement itertools.combinations or asian codecs. Nice comment Amaury :-). What about writing a blog post explicitly asking for help? I also agree that the fast-forward branch is a good way to enter pypy, moreover we should exploit the hype we seem to have at the moment :-). ciao, Anto ___ pypy-dev@codespeak.net http://codespeak.net/mailman/listinfo/pypy-dev
Re: [pypy-dev] gdbm
On 27/11/10 04:07, Dan Stromberg wrote: This module can be used with the “classic” ndbm interface, the BSD DB compatibility interface, or the GNU GDBM compatibility interface. On Unix, the *configure* script will attempt to locate the appropriate header file to simplify building this module. I suppose that means that if it can't find ndbm (which at one time was hard due to licensing, but last I heard it'd become readily available), it's free to pretend it has ndbm using something else. I'd call that puzzlingly worded - it's not the interface that's changing, but the backend implementation. But perhaps dbm.py is free to use Berkeley DB if it prefers to pretend it can never find ndbm. And perhaps I shouldn't have skimmed that page so quickly ^_^ CPython 2.7's configure script has: --with-dbmliborder=db1:db2:... order to check db backends for dbm. Valid value is a colon separated string with the backend names `ndbm', `gdbm' and `bdb'. so, having a dbm.py which links to libdb is fine, and it's also what you get with cpython on ubuntu. There is still the issue of how to find the correct library name, as it seems it can vary (it was db-4.5 when the module was written, it's db-4.8 nowadays), but this is a bit orthogonal to what we are discussing here. To summarize, I think we should keep the current dbm.py to link against libdb, and integrate your gdbm.py to link against libgdbm. But before merging it to trunk, I'd like to solve the issue of code duplication between the two modules. ciao, Anto ___ pypy-dev@codespeak.net http://codespeak.net/mailman/listinfo/pypy-dev
Re: [pypy-dev] gdbm
On 30/11/10 02:28, Dan Stromberg wrote: Agreed. I should have time for this sometime this week or the next. thank you! :-) ___ pypy-dev@codespeak.net http://codespeak.net/mailman/listinfo/pypy-dev
Re: [pypy-dev] PyPy 1.4 released
On 28/11/10 10:48, Maciej Fijalkowski wrote: i got python-magic working , after i installed without easy_install (easy_install fail because it tried to install ctypes). great well, that's still a bit strange. Why does easy_install try to install ctypes, given that it's in the stdlib? ___ pypy-dev@codespeak.net http://codespeak.net/mailman/listinfo/pypy-dev
Re: [pypy-dev] ype object 'BlackholeInterpreter' has no attribute 'bhimpl_direct_ptradd'
On 25/11/10 17:46, wlavrij...@lbl.gov wrote: yes, but the fast path was disabled (commented out) when updating the branch with trunk (I needed a few casts in the JIT that were in trunk already, but not in the branch). It should thus be in addition when it's working again. (And to be sure, using a float argument makes no difference for the numbers.) Myself, I'm just working on functionality right now. A factor of almost 10 is already good enough to start working on a functional demo of real code, given that most of our analysis is I/O bound. wow, if you did not use the fast path, then the 10x factor it's even more impressive :-). I think that yesterday Carl Friedrich added support to cppyy for calling the function through the new JIT-friendly rlib.libffi module, so now you should get the fast-path performance for most of the calls. Carl, could you post the benchmarks you did please? ciao, Anto ___ pypy-dev@codespeak.net http://codespeak.net/mailman/listinfo/pypy-dev
Re: [pypy-dev] gdbm
On 16/11/10 04:30, Dan Stromberg wrote: BTW, it might cause confusion down the road to call something that is basically like cpython's bsddb (Berkeley DB) by the name dbm in pypy's library. In the cpython standard library, dbm is an interface to ndbm databases. These all provide the same dictionary-like interface to Python programs, but have somewhat different API's to C, and pretty different, incompatible on-disk representations. Hi Dan, I played a bit (vry quickly) with dbm on both pypy and cpython, and I'm not sure I get what you mean when you say that our dbm.py is equivalent to cpython's bsddb. E.g., I can create a db on cpython and open it from pypy, so it seems that the two modules are compatible. Moreover, I checked which libraries the links to. On CPython, it links to libdb-4.8.so: viper2 ~ $ ldd /usr/lib/python2.6/lib-dynload/dbm.so linux-gate.so.1 = (0x00884000) libdb-4.8.so = /usr/lib/libdb-4.8.so (0x0011) libpthread.so.0 = /lib/libpthread.so.0 (0x003de000) libc.so.6 = /lib/libc.so.6 (0x003f8000) /lib/ld-linux.so.2 (0x002e) the pypy version first tries to open libdb.so, then libdb-4.5.so. I had to manually modify it to open version 4.8 (I agree that we should find a more general way to find it), but apart from that what I can see is that it uses the same underlying wrapper as CPython. So, to summarise: could you elaborate a bit more why we should delete dbm.py from pypy? ciao, Anto ___ pypy-dev@codespeak.net http://codespeak.net/mailman/listinfo/pypy-dev
Re: [pypy-dev] PyPy 1.4 released
On 27/11/10 03:09, Phyo Arkar wrote: libmagic python fails to work on pypy (python-magic) it uses ctypes and easy-install try to download and instaill it , but it fails. how to enable ctypes on pypy? Hi Phyo, ctypes *is* enabled on pypy by default. If python-magic does not work, it can either: 1) be a pypy bug: please report it as an issue (possibly with a simple failing test and the full traceack) 2) a python-magic issue, e.g. if it plays dirty ctypes trick that cannot really be supported by pypy due to the internal differences with CPython ciao, Anto ___ pypy-dev@codespeak.net http://codespeak.net/mailman/listinfo/pypy-dev
Re: [pypy-dev] [pypy-svn] r79480 - pypy/trunk/pypy/jit/tool
On 25/11/10 07:40, Maciej Fijalkowski wrote: Is pypy repository really a place for such files? Maybe we should keep it somewhere else? well, it's the memory usage of cpython when translating pypy. Why the pypy repo should not be the place for a pypy-related file? On Wed, Nov 24, 2010 at 6:39 PM,antoc...@codespeak.net wrote: Author: antocuni Date: Wed Nov 24 17:39:03 2010 New Revision: 79480 Added: pypy/trunk/pypy/jit/tool/cpython.vmrss Log: add the vmrss file produced by armin's memusage.py when running ./translate.py -Ojit at rev 79456 ___ pypy-dev@codespeak.net http://codespeak.net/mailman/listinfo/pypy-dev
Re: [pypy-dev] ype object 'BlackholeInterpreter' has no attribute 'bhimpl_direct_ptradd'
Hi Wim, On 25/11/10 01:32, wlavrij...@lbl.gov wrote: Hi, so I'm revising my numbers after finding out that I was using a debug version of ROOT/Reflex ... PyROOT: 48.6 PyCintex: 50.2 pypy-c:5.5 C++: 0.05 wow, impressive numbers :-). You said that the benchmark you used is a loop calling a c++ function that does nothing. What is the signature of that function? If you remember, in interp_cppyy there is a fast path that gives huge speedups when calling a c++ function which takes an integer and returns an integer, so depending on the signature you get very different numbers today. The good news is that nowadays the JIT has got the capability to optimise external calls in a general way: this means that as soon as we integrate it with cppyy we can have all calls as fast as the ones that go through the current fast path. ciao, Anto ___ pypy-dev@codespeak.net http://codespeak.net/mailman/listinfo/pypy-dev
Re: [pypy-dev] [pypy-svn] r79382 - pypy/trunk/pypy/config
On 23/11/10 08:18, fi...@codespeak.net wrote: Author: fijal Date: Tue Nov 23 08:18:57 2010 New Revision: 79382 Modified: pypy/trunk/pypy/config/translationoption.py Log: Disable jit debugging by default Modified: pypy/trunk/pypy/config/translationoption.py == --- pypy/trunk/pypy/config/translationoption.py (original) +++ pypy/trunk/pypy/config/translationoption.py Tue Nov 23 08:18:57 2010 @@ -115,7 +115,7 @@ default=auto, cmdline=--jit-backend), ChoiceOption(jit_debug, the amount of debugging dumps made by the JIT, [off, profile, steps, detailed], - default=profile, # XXX for now + default=off, cmdline=--jit-debug), ChoiceOption(jit_profiler, integrate profiler support into the JIT, [off, oprofile], Not sure it's a good idea. It's useful to see the statistics at the end of the run. I know that you can turn it on explicitly, but it's easy to forget. What about having an env variable that automatically turns jit debug on? ciao, Anto ___ pypy-dev@codespeak.net http://codespeak.net/mailman/listinfo/pypy-dev
Re: [pypy-dev] gdbm
Hi Dan, first: thanks for your help :-) On 16/11/10 03:17, Dan Stromberg wrote: Yes, the dbm module in pypy is basically like the bsddb module in cpython. cpython includes modules for bsddb, gdbm, and more. I tend to prefer gdbm over bsddb, because I've seen bsddb databases get corrupt too many times - EG, when a filesystem overflows. bsddb might be a little faster though; I've never compared their performance. So, if I understand correctly you are saying that we should rename our dbm.py to bsdb.py, and write a new dbm.py which can use either bsdb or gdbm? Sounds fine, do you feel like implementing it? :-) Moreover, I also agree with amaury that your code is very similar to the one in the current dbm.py, so we should maybe try to refactor things to share common parts between the twos. ciao, Anto ___ pypy-dev@codespeak.net http://codespeak.net/mailman/listinfo/pypy-dev
[pypy-dev] Migration to mercurial
Hi all, as some of you might have noticed by watching the IRC channel, we are finally migrating to mercurial. The actual conversion of the repository will happen at some point during next week (or maybe the week after). One issue that we have to care about is how to convert the author names: in the current svn repository each author is simply indicated by its codespeak username, while with mercurial this is no longer possible as there is no longer a centralized server where everyone has an account. In mercurial an author is just a string, but the standard way is put both the real name and a valid email address to uniquely identify the author. This is exploited by websites like e.g. bitbucket to link the author of the commit to the bitbucket user. For doing the conversion, we will use the following usermap: http://codespeak.net/svn/pypy/extradoc/planning/hg-migration/usermap.txt By default, we map each username to its real name, as found in the /etc/passwd file on codespeak.net. Because of the reasons I explained above, I strongly recommend everyone to put its email address there, so that it will be there in the final mercurial repository. If for any reason you don't want your real name to be in the mercurial repository, please kill the corresponding line in the file. Note that it won't be possible to change the info after the conversion has been made. ciao, Anto ___ pypy-dev@codespeak.net http://codespeak.net/mailman/listinfo/pypy-dev
Re: [pypy-dev] Question on the future of RPython
On 29/09/10 22:40, Terrence Cole wrote: then, mylog contains all the loops and bridges produced by the jit. The interesting point is that there are also special operations called debug_merge_point that are emitted for each python bytecode, so you can easily map the low-level jit instructions back to the original python source. I think that 'easily' in that last sentence is missing scare-quotes. :-) well, it's easy as long as you have a bytecode-compiled version around. With only the AST I agree that it might be a bit trickier. [cut] My first inclination would be to continue this chain and add a bytecode compiler on top of the ast builder. This would keep ast node references in the instructions it creates. If the algorithms don't diverge too much, I think this would allow the debug output to be mapped all the way back to the source chars with minimal effort. I'm not terrifically familiar with the specifics of how python emits bytecode from an ast, so I'd appreciate any feedback if you think this is crazy-talk. Are you using your custom-made AST or the one from the standard library? In the latter case, you can just pass the ast to the compile() builtin function to get the corresponding bytecode. ___ pypy-dev@codespeak.net http://codespeak.net/mailman/listinfo/pypy-dev
Re: [pypy-dev] [pypy-svn] r77083 - pypy/branch/jitffi
Hi,
On 15/09/10 18:18, Maciej Fijalkowski wrote:
Hey anto.
There was a SoC about that, I guess it would be good to chat about it
at least (personally I think jitting rlib/libffi is exactly bad layer
to be jitted and some experiments were done).
yes, I read the code in the fast-ctypes branch but I wanted to take another
(simpler) approach. Note that my goal is not only to speed up ctypes, but
also to provide a useful building block for cppyy (the module to call c++
functions that we started at the cern sprint).
My basic idea was to mark libffi.FuncPtr.{push_arg,call} in a special way, so
that the backend can recognize the pattern (i.e. push* + call) and emit a
single assembler call.
I even started to write a bit of code, but then I realized that libffi.FuncPtr
is not used at all, as _rawffi uses RawFuncPtr: the bad news is that
RawFuncPtr uses a different interface, as it does not have push_arg but passes
the arguments already packed in a list, so my easy solution above cannot work.
Note however that doing it at the level of FuncPtr might still be useful for
cppyy.
Question: why does _rawffi use RawFuncPtr instead of FuncPtr? Would it be
possible/easy/hard/whatever to switch to FuncPtr?
ciao,
Anto
___
pypy-dev@codespeak.net
http://codespeak.net/mailman/listinfo/pypy-dev
Re: [pypy-dev] [pypy-svn] r77101 - in pypy/trunk/pypy: jit/tl module/__builtin__ module/__builtin__/test module/pypyjit/test
Hi, On 16/09/10 07:27, hakana...@codespeak.net wrote: Log: Allow jit to unroll calls to max() and min() with more than one argument. [cut] +...@unroll_safe @specialize.arg(2) def min_max(space, args, implementation_of): if implementation_of == max: compare = space.gt else: compare = space.lt + +args_w = args.arguments_w +if len(args_w) 1 and not args.keywords: # Unrollable case +w_max_item = None +for w_item in args_w: +if w_max_item is None or \ + space.is_true(compare(w_item, w_max_item)): +w_max_item = w_item +return w_max_item +else: +return min_max_loop(space, args, implementation_of) I don't think it's a good idea. What happens if I call max() over a list of 1 million of elements? We obviously don't want the jit to unroll 1 million of iterations. Or am I missing something? ciao, Anto ___ pypy-dev@codespeak.net http://codespeak.net/mailman/listinfo/pypy-dev
Re: [pypy-dev] PyPy JIT C extensions, greenlet
On 13/09/10 10:10, Armin Rigo wrote: Hi Andy, On Mon, Sep 13, 2010 at 7:21 AM, Andy angelf...@yahoo.com wrote: Does that mean PyPy will not work with greenlet/gevent/etc? Sorry if I wasn't clear. PyPy contains greenlet support (since 2005-6). It's part of the same package that we call pypy-stackless. yes, but it must also be said that at the moment, pypy-stackless and pypy-jit do not work together. ciao, Anto ___ pypy-dev@codespeak.net http://codespeak.net/mailman/listinfo/pypy-dev
Re: [pypy-dev] External RPython mailing list
On 13/09/10 10:27, Maciej Fijalkowski wrote: Is it really about interpreters? (what's interpreter-specific after all in RPython) or is it just that it's hard to use and does not integrate with CPython well? my point if that it's definitely good enough for writing interpreters. For the rest, it's a bit unknown (in the sense that nobody has ever tried), and we don't care about knowing :-) ___ pypy-dev@codespeak.net http://codespeak.net/mailman/listinfo/pypy-dev
Re: [pypy-dev] [pypy-svn] r76608 - in pypy/branch/jit-bounds/pypy/jit/metainterp: . test
On 12/08/10 19:02, hakana...@codespeak.net wrote: +def boundint_gt(self, val): +if val is None: return +self.minint = val + 1 what happens if val == sys.maxint? ciao, Anto ___ pypy-dev@codespeak.net http://codespeak.net/mailman/listinfo/pypy-dev
Re: [pypy-dev] rstruct where is pack?
On 10/08/10 15:14, Benjamin Peterson wrote: 2010/8/10 Hart's Antler bhart...@yahoo.com: Seems like struct.pack is not RPython? I see the examples for unpack in the tests folder, but not for packing. struct.pack() is implemented in pypy/module/rstruct/. I suppose you mean pypy/module/struct. But if the OP is looking for an rpython lib to use in his rpython program, this is not exactly what he looks for, although I agree it could be adapted and ported to rlib. ciao, Anto ___ pypy-dev@codespeak.net http://codespeak.net/mailman/listinfo/pypy-dev
Re: [pypy-dev] Interpreter level array implementation
On 03/07/10 08:14, Hakan Ardo wrote: What is a bridge? you might be interested to read the chapter of my PhD thesis which explains exactly that, with diagrams: http://codespeak.net/svn/user/antocuni/phd/thesis/thesis.pdf In particular, section 6.4 explains the difference between loops, bridges and entry bridges. ciao, Anto ___ pypy-dev@codespeak.net http://codespeak.net/mailman/listinfo/pypy-dev
Re: [pypy-dev] [pypy-svn] r75683 - in pypy/trunk: include lib-python/modified-2.5.2/distutils lib-python/modified-2.5.2/distutils/command pypy/_interfaces pypy/module/cpyext pypy/module/cpyext/test
On 02/07/10 08:45, Maciej Fijalkowski wrote: Hey. Any reason why we should copy .h files during translation and can't just have them there? I talked with Amaury and he told me that he prefers to keep all the cpyext-related files together, which I think makes sense. Moreover, we need to generatecopy pypy_decl.h and pypy_macros.h anyway, so we can copy the others as well while we are at it. ciao, Anto ___ pypy-dev@codespeak.net http://codespeak.net/mailman/listinfo/pypy-dev
Re: [pypy-dev] [pypy-svn] r75683 - in pypy/trunk: include lib-python/modified-2.5.2/distutils lib-python/modified-2.5.2/distutils/command pypy/_interfaces pypy/module/cpyext pypy/module/cpyext/test
On 02/07/10 09:28, Maciej Fijalkowski wrote: Fine by me. Can you fix test_package then? It assumes there is Python.h in include (which might not be there). ah right... because when we run own-test translation didn't happen, so .h are not there. Ok, I'll fix it later. ciao, Anto ___ pypy-dev@codespeak.net http://codespeak.net/mailman/listinfo/pypy-dev
Re: [pypy-dev] virtualenv support and directory hierarchy
Hi, I also don't like too much to have a hardcoded version number, but when I asked for alternatives nobody suggested anything. On IRC, amaury suggested to install the whole pypy distribution in its self-contained directory, more or less as cpython does on windows. I didn't think about this solution, but now I see that it might be a good one, as it would allow to have the same hierarchy on svn checkouts, user-specific installations and system-wide installations. The drawback is that it's a bit non-standard on unix; moreover, if we install pypy in say /opt/pypy1.2, it would be hard to put a binary in /usr/bin without hardcoding the path to pypy1.2 somewhere. So, for me there are four possible solutions: 1) leave things as they are on branch/sys-prefix and have the version number hardcoded in svn 2) put the whole distribution in its own directory, as jython or cpython on windows. This has the open problem of determining where the directory is, as described above 3) don't hardcode the version number in svn, but add a special case to virtualenv to detect if we are inside a pypy checkout to handle it specially 4) don't care about running virtualenv inside a pypy checkout Personally, I would exclude (4): I think that it would be very cool for people to try pypy in a sandboxed virtualenv without having to install it, and it would be useful to us too. So, before I do more work, I'd like to hear what people think and which of the alternatives they prefer. ciao, Anto On 18/06/10 19:01, Maciej Fijalkowski wrote: Hey anto. I expressed my concerns already, but I'm going to express them once again, since amaury also said that on IRC. I really dislike having version number hardcoded in source checkout for a whole variety of reasons. I don't think need to have virtualenv working from source checkout is enough to push that through. How about install script that does that maybe? Cheers, fijal ___ pypy-dev@codespeak.net http://codespeak.net/mailman/listinfo/pypy-dev
Re: [pypy-dev] virtualenv support and directory hierarchy
On 19/06/10 11:00, Amaury Forgeot d'Arc wrote: The drawback is that it's a bit non-standard on unix; moreover, if we install pypy in say /opt/pypy1.2, it would be hard to put a binary in /usr/bin without hardcoding the path to pypy1.2 somewhere. Is it possible to just put a symlink, or a small script in /usr/bin? yes, the symlink should be possible, as armin also points out. I already thought about it, but I was not sure that distributions like ubuntu allows to put a symlink in /usr/bin to something external. But indeed, looking at my /usr/bin it seems that symlinks are used a lot, so it should be fine. So, does this mean that we are going for solution number 2? ___ pypy-dev@codespeak.net http://codespeak.net/mailman/listinfo/pypy-dev
Re: [pypy-dev] [pypy-svn] r75220 - in pypy/trunk/pypy: annotation annotation/test jit/backend jit/backend/llgraph jit/backend/llgraph/test jit/backend/llsupport jit/backend/llsupport/test jit/backend/
On 13/06/10 08:46, Armin Rigo wrote: I'm a bit confused, btw: I thought that ootype did not need ConstAddr at all, because it used ConstObj for all pointer-ish things. ah sorry. You wrote ConstAddr but I actually read ConstPtr (i.e., ConstObj for ootype). Indeed, ConstAddr is not used at all by ootype, so there should be no problem (at least in this respect :-)). ciao, Anto ___ pypy-dev@codespeak.net http://codespeak.net/mailman/listinfo/pypy-dev
Re: [pypy-dev] [pypy-svn] r75220 - in pypy/trunk/pypy: annotation annotation/test jit/backend jit/backend/llgraph jit/backend/llgraph/test jit/backend/llsupport jit/backend/llsupport/test jit/backend/
On 08/06/10 23:42, ar...@codespeak.net wrote: The number of changes is a bit huge though. The format of the static bytecodes used by the jit changed completely, and codewriter.py is now split among many files in the new directory pypy.jit.codewriter. There is also no longer ConstAddr, only ConstInt: prebuilt addresses are now turned into integers (which are symbolic, with the new class AddressAsInt, so they can be rendered by the C translation backend). Various related changes occurred here and there. I didn't look at the branch deeply, but the last sentence looks suspiciously hard/impossible to implement in ootype. Could you explain why ConstAdrr were bad please? ciao, Anto ___ pypy-dev@codespeak.net http://codespeak.net/mailman/listinfo/pypy-dev
Re: [pypy-dev] [pypy-svn] r74976 - in pypy/branch/sys-prefix: lib/pypy1.2/lib_pypy/ctypes_config_cache pypy/interpreter/test pypy/rlib pypy/tool pypy/tool/test pypy/translator/goal pypy/translator/san
On 01/06/10 06:00, Maciej Fijalkowski wrote: A bit about directory structure: I think I have explained everything in my original email to pypy-dev: http://codespeak.net/pipermail/pypy-dev/2010q2/005854.html Can you explain to me a bit? What's in lib except pypy1.2? nothing. It really plays the same role as /usr/lib. We need it because in this way we can have sys.prefix == '/path/to/pypy-trunk' and still have the lib in join(sys.prefix, 'lib', 'pypy%d.%d') Why pypy1.2? Do we have any reason? yes. When we install pypy system wide, we really want to have a version number in the directory that contains the stdlib; also, it is consistent with cpython, which puts it into e.g. /usr/lib/python2.6 Do we ever need to keep more than one? no. We will rename it every time we do a new release. What's in pypy1.2 except lib_pypy? there will be also lib-python, although I've not moved it yet. ciao, Anto ___ pypy-dev@codespeak.net http://codespeak.net/mailman/listinfo/pypy-dev
Re: [pypy-dev] [pypy-svn] r74850 - in pypy/branch/sys-prefix/lib: . pypy1.2 pypy1.2/lib_pypy pypy1.2/lib_pypy/app_test pypy1.2/lib_pypy/ctypes_config_cache pypy1.2/lib_pypy/test2
On 30/05/10 00:43, holger krekel wrote: pyrolog for example doesn't use lib_pypy or pypy/module, does it? sorry, i meant prolog, gameboy, etc ... i.e. the projects in pypy/lang yes, indeed. That's why I think it's a good idea to move pypy/lib outside pypy. ___ pypy-dev@codespeak.net http://codespeak.net/mailman/listinfo/pypy-dev
Re: [pypy-dev] [pypy-svn] r74850 - in pypy/branch/sys-prefix/lib: . pypy1.2 pypy1.2/lib_pypy pypy1.2/lib_pypy/app_test pypy1.2/lib_pypy/ctypes_config_cache pypy1.2/lib_pypy/test2
On 30/05/10 00:08, Maciej Fijalkowski wrote: My concern is mostly that we should not over engineer things and be smarter than CPython (at least in this respect). well, if we want sys.prefix we *need* to be smarter than CPython, as we don't have any ./configure where take it from. The alternative would be to pass --prefix to translate.py: do you *really* want to do it? There is a whole variety of reasons why it would not work in the end. like? Point 2) talks about superset of CPython funcionality. How for example this would work with compiled cpython extensions that has setup.py install run? if you use virtualenv, there should be no problem, as it recreates the whole environment needed. If you are talking about running ./translator/goal/pypy-c /my/extension/setup.py install, well, in that case I guess that what happens is just that your extension will be installed inside pypy-trunk/lib/pypy1.2/site-packages or some path like this. Well, too bad for you, I would say. On smaller issue, I don't like pypy1.2. Both because it contains a dot and because it contains a revision number. Why we need that? CPython has the standard lib in $PREFIX/lib/python2.6, so for consistency we want ours to reside in $PREFIX/lib/pypy1.2. I know, the drawback of this is that we need to manually rename it every time we change version number, but it's also true that it does not happens often. If you have an alternative solution, I'd like to hear that. Note that I don't think that don't care about using virtualenv from trunk is a good alternative solution :-). ciao, Anto ___ pypy-dev@codespeak.net http://codespeak.net/mailman/listinfo/pypy-dev
Re: [pypy-dev] [pypy-svn] r74850 - in pypy/branch/sys-prefix/lib: . pypy1.2 pypy1.2/lib_pypy pypy1.2/lib_pypy/app_test pypy1.2/lib_pypy/ctypes_config_cache pypy1.2/lib_pypy/test2
Hi holger, On 29/05/10 22:25, holger krekel wrote: I think the idea is to make sys.prefix (and thus virtualenv) work even with a translation in a checkout, i.e. not forcing to copy things to another location (which virtualenv partly does on its own). yes Moreover, keeping app-level modules (and maybe pypy/module at some point) outside the main (interpreter, objspaces, translation and JIT) PyPy tree makes sense to me. e.g. pypy/lang would probably not need to access anything outside such a pypy tree, for example, or am i mistaken? I agree that keeping app-level modules out of pypy is a good idea (this is what I'm doing it, of course :-)), but I don't get what does the reference to pypy/lang mean in this context. ciao, Anto ___ pypy-dev@codespeak.net http://codespeak.net/mailman/listinfo/pypy-dev
Re: [pypy-dev] Nightly builds are unavailable
On Tue, Mar 23, 2010 at 7:43 PM, Gasper Zejn z...@kiberpipa.org wrote: Nope, not working. The directory listing works, but not the links. Did you click on any of the links provided on that url? Ah no, sorry. My fault, the links indeed don't work. ciao, Anto ___ pypy-dev@codespeak.net http://codespeak.net/mailman/listinfo/pypy-dev
Re: [pypy-dev] Nightly builds are unavailable
On Tue, Mar 23, 2010 at 6:02 PM, Gasper Zejn z...@kiberpipa.org wrote: The links at the URL http://buildbot.pypy.org/nightly/ are failing with resource unavailable; misconfiguration? Just to make sure it's a known problem. It's working for me right now. Maybe it has just been down for a while? ___ pypy-dev@codespeak.net http://codespeak.net/mailman/listinfo/pypy-dev
Re: [pypy-dev] autonosiness in roundup
Benjamin Peterson wrote: 2010/3/16 Maciej Fijalkowski fij...@gmail.com: Anyone knows how to set up autonosy on roundup? I only get information about new tickets and then I don't get followups. When that's figured out, I'd also like to be on autonosy. me too :-) ___ pypy-dev@codespeak.net http://codespeak.net/mailman/listinfo/pypy-dev
Re: [pypy-dev] speed.pypy.org launched
Carl Friedrich Bolz wrote: I disagree with this, please read the following paper: http://buytaert.net/files/oopsla07-georges.pdf +1 for this paper. I was about to link it as well, but cf has been faster. I looked at the unladen swallow runner when I was writing my thesis, and I can confirm that it does the statistically right thing as described in the paper to compute error bars. ciao, Anto ___ pypy-dev@codespeak.net http://codespeak.net/mailman/listinfo/pypy-dev
Re: [pypy-dev] pypy-jvm doesn't work?
Hi Olli, Olli Wang wrote: Hi, I tried to execute the compiled Python interpreter with jvm backend on both Gentoo Linux(amd64) and Snow Leopard. But unfortunately both of them failed to start. I simply ran ./translate.py --backend=jvm targetpypystandalone.py and pypy-jvm on both computer. Do I miss something? Thanks. == Error messages on Gentoo Linux (amd64) Exception in thread main java.lang.VerifyError: (class: pypy/ConstantInit_0, method: constant_init signature: ()V) Expecting to find long on stack at pypy.ConstantInit.init(ConstantInit.j:6) at pypy.Main.clinit(Main.j:26) Could not find the main class: pypy.Main. Program will exit. I suppose it's a 32/64 bit issue again: the problem is that when you run ./translate.py with a 64 bit python, it assumes that 'int' variables are 64 bits, but on the JVM they are always 32 bit. For what I know, the CLI backend has exactly the same problem. As a workaround, you can try to run translate.py under a 32bit chroot (which works for sure, as I use it daily to develop pypy) or with a 32bit python (which should work, but I've never tried). ciao, Anto ___ pypy-dev@codespeak.net http://codespeak.net/mailman/listinfo/pypy-dev
Re: [pypy-dev] Parallel translation?
Benjamin Peterson wrote: No, that's not planned at the moment. The only time we could easily use multiple CPU at the moment is compiling the final C sources. You can even do this manually with the make file in the temp source directory. I agree that at this point in time we cannot or don't want to make annotation/rtyping/backend parallelizable, but it should definitely be possible to just pass the -j flag to 'make' in an automatic way. Anyone feeling like to implement it? :-) ___ pypy-dev@codespeak.net http://codespeak.net/mailman/listinfo/pypy-dev
Re: [pypy-dev] [pypy-svn] r69710 - in pypy/trunk/pypy/module/oracle: . test
a...@codespeak.net wrote: Log: Now the cx_Oracle module translates, compiles, and works! Tested on Windows. great work! I think it's worth a blog post. ciao, Anto ___ pypy-dev@codespeak.net http://codespeak.net/mailman/listinfo/pypy-dev
Re: [pypy-dev] Why isn't the PyPy logo Ouroboros(Snake biting its tail)?
On Tue, Oct 20, 2009 at 11:42 AM, holger krekel hol...@merlinux.eu wrote: - vmdjinn nice - i like the djinn meme - so also throw djynn - which VM do you want to generate today? I think that both vmdjinn and djynn are almost impossible to pronunciate. Btw, I think I miss what they mean and/or refer to. Are they acronyms? ___ pypy-dev@codespeak.net http://codespeak.net/mailman/listinfo/pypy-dev
Re: [pypy-dev] Why isn't the PyPy logo Ouroboros(Snake biting its tail)?
R. David Murray wrote: http://en.wikipedia.org/wiki/Djinn Once you know how to pronounce the reference, the proposed names become easier to pronounce. ahah, now everything it's much clearer, thanks! And yes, I like it :-) ___ pypy-dev@codespeak.net http://codespeak.net/mailman/listinfo/pypy-dev
Re: [pypy-dev] Why isn't the PyPy logo Ouroboros(Snake biting its tail)?
Carl Friedrich wrote: While I agree, to do this we would have to come up with a name for the translation toolchain part. And coming up with names is even harder than coming up with logos. it's also unclear how to split the sources: e.g., pypy/interpreter/ belongs to both. ___ pypy-dev@codespeak.net http://codespeak.net/mailman/listinfo/pypy-dev
Re: [pypy-dev] Why isn't the PyPy logo Ouroboros(Snake biting its tail)?
holger krekel wrote: the interpreter is needed for abstract interpretation, true - but does it maybe make sense to eventually decouple these rpython analysis capabilities from how/which Python version/bytecodes are implemented for our Python interpreter offering? it's probably possible, but I don't think it's easy unless you want to duplicate a lot of code ___ pypy-dev@codespeak.net http://codespeak.net/mailman/listinfo/pypy-dev
Re: [pypy-dev] Why isn't the PyPy logo Ouroboros(Snake biting its tail)?
holger krekel wrote: yip. so we need a name for our super-powered ultra-flexible translator. dynajite: - it's for dynamic languages - it provides you a jit - it makes your head explode :-) ___ pypy-dev@codespeak.net http://codespeak.net/mailman/listinfo/pypy-dev
Re: [pypy-dev] Why isn't the PyPy logo Ouroboros(Snake biting its tail)?
Carl Friedrich wrote: dynajite: - it's for dynamic languages - it provides you a jit - it makes your head explode :-) Funny, but not really a name I would seriously consider. Sounds too much like shite... uhm, my italian mind would never pronounce dynajite like that, but maybe native speakers of other languages think differently, I don't know. To mitigate this, we could simply drop the 'e' at the end: 'dynajit', but I agree that it's much more boring. About other names, the most obvious and boring alternative is rpyc (for rpython compiler), but I don't like it too much. ciao, Anto ___ pypy-dev@codespeak.net http://codespeak.net/mailman/listinfo/pypy-dev
Re: [pypy-dev] Help with Exception: unexpected prebuilt constant
Hi Philip, Philip Guo wrote: Sorry for another newbish question, but I'm having trouble getting PyPy translated into C code. I added some calls to open() throughout the interpreter codebase (to log some info to files), but it failed translation with the following error: [translation:ERROR] Exception: unexpected prebuilt constant: built-in function open I then tried doing 'import os' and then using os.fdopen, but with the same error: [translation:ERROR] Exception: unexpected prebuilt constant: built-in function fdopen as you have noticed, neither os.open nor os.fdopen are supported by RPython. The only low-level supported way to deal with files is using os.open/os.read/os.write. However, you might want to have a look to rlib.streamio, that provides a higher level API to deal with files, offering features such as buffering, CR/LF conversions etc. In particular, you can use streamio.open_file_as_stream to get a file like object similar to the builtin python ones. In general, what Python standard library calls can I make from my modified PyPy code and still have it translate properly? Specifically, how do I read/write files from within PyPy? see above :-) ciao, Anto ___ pypy-dev@codespeak.net http://codespeak.net/mailman/listinfo/pypy-dev
Re: [pypy-dev] Help with Exception: unexpected prebuilt constant
Philip Guo wrote: Thanks for the informative reply, Antonio! My (probably dumb) follow-up question is: Is pickle (or cPickle) supported by RPython? The bottom line is that I need to serialize some data structures from my modified PyPy interpreter to disk so that data can persist across successive interpreter runs. Pickling is the most obvious way to do so, but it requires a Python built-in file object. pickle and cpickle are not supported by RPython. If you need to serialize interp-level objects, I fear the only way is to manually write the code to write them to disk and read back. You might want to have a look at rlib.rmarshal, which can be used to serialize simple objects such as numbers, lists and tuples. On the other hand, remember that you have the whole PyPy python interpreter in your hands: depending on what you are trying to do, you could use it to interact with the fully working cPickle modules. But please notice that this solution works only if you want to serialize app-level objects (i.e. all the objects that are called as w_* inside the pypy sources). If pickling doesn't work, what other way of persisting data on disk do you suggest? (e.g., printing it to a file as plaintext and then eval-ing it again? does eval() even work in RPython?) no, eval() doesn't work in RPython. ciao, Anto ___ pypy-dev@codespeak.net http://codespeak.net/mailman/listinfo/pypy-dev
Re: [pypy-dev] Does PyPy supports Java ME?
Jan Wedel wrote: Hi Anto! Thanks for your answer. Hi Jan, (I'm cc-ing pypy-dev again) Although it's now made by Sun, it shows the key points: - No class loading - No JNI - No reflection Which means you cannot load dynamic code during run-time. On top, there are a lot of libraries missing (XML parsing, Regex, logging...) so far pypy-jvm doesn't use any of those, so it should not be a problem. However, in the future reflection will be surely needed to access java libraries (but you don't seem to be interested in it) and class loading for the jit. THAT might be a problem since we are working on memory constrained systems with a maximum of a few MB of flash and approx. 200kB RAM. Are there any estimations about the RAM usage of PyPy using classes and other memory-consuming features? I think that 200kB are definitely not enough for pypy-jvm. I don't know for sure how to measure memory usage of java programs, but if I just launch pypy-jvm and measure the amount of memory used on linux, I get something like ~400 MB: this also includes the overhead of the jvm itself, but it's still a lot. And is this 8-10MB including only the core interpreter? Or are there any ways to strip it down by excluding unnecessary libs? it includes the core interpreter and the builtin modules; some of them can be stripped down, but I don't know how much you can save. I didn't make myself clear. I meant python libs/modules. E.g. sys, socket, sre etc... Because if these modules are supported I won't need a lot of Java Libraries. As I explained, I already ported PyMite to Java and I discovered that numerous python modules have a C counterpart such as _socket, _sre etc. So If you want to run these libs, you need to translate these C files. ah, now I understand. This is the complete list of builtin modules that are enabled for pypy-jvm: [translation] [usemodules] [translation] __pypy__ = True [translation] _ast = True [translation] _codecs = True [translation] _pickle_support = True [translation] _random = True [translation] _sre = True [translation] _testing = True [translation] _weakref = True [translation] cStringIO = True [translation] errno = True [translation] gc = True [translation] itertools = True [translation] marshal = True [translation] math = True [translation] md5 = True [translation] operator = True [translation] parser = True [translation] posix = True [translation] rbench = True [translation] sha = True [translation] symbol = True [translation] time = True [translation] token = True As you can see, _socket is missing. It's surely possible to write a version of it that works for the jvm, but as usual it takes some time and nobody cared enough to implement it. ciao, Anto ___ pypy-dev@codespeak.net http://codespeak.net/mailman/listinfo/pypy-dev
Re: [pypy-dev] Does PyPy supports Java ME?
Hi Jan, Jan Wedel wrote: 1.) Does PyPy generate both C and Java Bytecode for the Interpreter out of the box or with low effort? yes 2.) Is the java code compatible with Java ME or is it possible to modify the PyPy-code/Interpreter-code so that Java ME is supported? I don't know what are the requirements/restrictions for Java ME. If the restrictions are only about the library, it's possible that it works out of the box as pypy-jvm doesn't use many external libraries. However, note that you might have memory problems because the binary it's kind of big (the JAR file is ~8-10 MB, IIRC). 3.) What standard libraries are supported? I coulnd't find a list. Especially the communication (such as socket) functionality is important. unfortunately, you cannot access any external java library from pypy-jvm so far. I agree that it makes it a bit pointless, but basically nobody ever cared enough/had time to add support for it. I plan to do it at some point, but surely not before I finish my PhD (which is supposed to finish at the end of next april). 4.) Do you think PyPy is stable enough for commercial use? The MIT license allows commercial use as far as I read. The Python interpreter in PyPy is quite stable and compliant with CPython. However, the resulting pypy-jvm has never been extensively tested as far as I know, so it's likely to have undiscovered bugs (but usually they are easy to fix). So, to sum up: I'm sorry but pypy-jvm is not usable for production right now; it could be made useful with relatively low effort if someone invested time and/or money on it, but I don't think it's going to happen very soon. ciao, Anto ___ pypy-dev@codespeak.net http://codespeak.net/mailman/listinfo/pypy-dev
Re: [pypy-dev] Problems translating PyPy to Java
Jan Wedel wrote: c:\dokumente und einstellungen\jan wedel\eigene dateien\repo\trunk\rialto\tool s\pypy-1.1.0\pypy\rpython\typesystem.py(153)deref() - assert isinstance(ootype.typeOf(obj), ootype.OOType) (Pdb+) exit uhm, that's weird. I've just tried a jvm translation on a linux machine and it works fine. I don't have any windows machine with JDK to try it by myself, but I'm downloading it right now. In the meantime: could you tell us what is the result of typing print obj and print ootype.typeOf(obj) at the pdb prompt? Thanks, Anto ___ pypy-dev@codespeak.net http://codespeak.net/mailman/listinfo/pypy-dev
Re: [pypy-dev] Problems translating PyPy to Java
Amaury Forgeot d'Arc wrote: Jan Wedel wrote: c:\dokumente und einstellungen\jan wedel\eigene dateien\repo\trunk\rialto\tool s\pypy-1.1.0\pypy\rpython\typesystem.py(153)deref() - assert isinstance(ootype.typeOf(obj), ootype.OOType) (Pdb+) exit Hey, do you use trunk version, or some release branch? Recent development is done on trunk... ah, indeed, I didn't notice the 1.1.0. I remember that at post-europython sprint (which was after the 1.1.0 release) I fixed a couple of issues that prevented pypy-jvm to be translated on windows, so if you shoul really use trunk for this. ___ pypy-dev@codespeak.net http://codespeak.net/mailman/listinfo/pypy-dev
Re: [pypy-dev] benchmarking input
Anders Hammarquist wrote: Hi all, Hi Iko, hi all [cut] So benchmarks will obviously have to specify what interpreter(s) they should be run by somehow. I think this is a vital requirement. I can imagine various scenarios where you want to specify which interpreters to benchmark, e.g.: 1) benchmarking pypy-cli vs IronPython or pypy-jvm vs Jython 2) benchmarking pypy-cs at different svn revisions 3) benchmarking pypy-c-trunk vs pypy-c-some-branch (maybe with the possibility of specifying pypy-c-trunk-at-the-revision-where-the-branch-was-created, to avoid noise) 4) benchmarking pypy-cs with different build options 5) bencharmking with profiling enabled (I'd say that profiling should be off by default) The bigger question is how to get those interpreters. Should running the benchmarks also trigger building one (or more) pypy interpreters according to specs in the benchmarking framework? (but then if you only want it to run one benchmark, you may have to wait for all the interpreters to build) Perhaps each benchmark should build its own interpreter (though this seems slow, given that most benchmarks can probably run on an identically built interpreter). Or maybe the installed infrastructure should only care about history, and if you want to run a single benchmark, you do that on your own. Conceptually, I would say that you need to rebuild the required pypys every time you run the benchmarks. Concretely, we can think of putting them into a cache, so that if you need a pypy-c that for some reason has already been built, you just reuse it. Moreover, it could be nice if you could select the pypy to benchmark from a list of already built pypys, if you want to same time. Also, we may need to think how to deal with excessive loading: if everyone of us tries to run his own set of benchmark, the benchmarking machine could become too overloaded to be useful in any sense. ciao, Anto ___ pypy-dev@codespeak.net http://codespeak.net/mailman/listinfo/pypy-dev
Re: [pypy-dev] [pypy-svn] r66656 - pypy/branch/pyjitpl5/pypy/jit/metainterp
ar...@codespeak.net wrote: Author: arigo Date: Mon Jul 27 21:07:20 2009 New Revision: 66656 Modified: pypy/branch/pyjitpl5/pypy/jit/metainterp/history.py pypy/branch/pyjitpl5/pypy/jit/metainterp/optimize.py Log: An RPython-friendly way to dump a loop. in case you don't know, there is already code that serves this purpose, in backend.support.AbstractLogger (used by both x86 and cli backend); you can just set the PYPYJITLOG env variable and you get a .ops file containing a textual representation of the loop (btw, it would be nice to have a graphical viewer for it... hint hint :-)). ___ pypy-dev@codespeak.net http://codespeak.net/mailman/listinfo/pypy-dev
