Re: [pypy-dev] Contributing Polyhedral Optimisations in PyPy
On Fri, 18 Dec 2020 at 05:14, muke101 via pypy-dev wrote: > > I'm doing a computer science masters and am looking for an appropriate > project to take on for a dissertation related to Polyhedral optimisations. > Talking to my professor, we both think trying to implement the model and it's > loop transformations in PyPy's JIT optimiser could be a good project to > pursue, but before committing to anything I wanted to run this idea by the > devs here who might be able to point out any hurdles I'd be likely to quickly > come across that could prove difficult to solve at just a masters level, or > whether or not these optimisations are actually already implemented in the > first place (I have tried to google if this is the case and hadn't found > anything, but can't be sure). I think this could have some good real world > impact too as a lot of scientific code is written in Python and run on PyPy, > and the Polyhedral model can offer substantial performance improvements in > the form of auto-parallelization for these types of codes, which is why I'm > interested in workin g on this for PyPy rather than CPython, although if anyone has good reason that I might want to look at CPython for this over PyPy please let me know. > > Appreciate any and all advice, thanks. Hi! That's a great topic. The challenge with implementing this in the pypy JIT at this point is that the JIT only sees one control flow path. That is, one loop, and the branches taken within that loop. It does not find out about the outer loop usually until later, and may not ever find out about the content of other control flow paths if they aren't taken. This narrows the amount of information available about effects and possible aliases quite a bit, making semantic-preserving cross-loop transformations difficult in many cases. On the other hand, since you can deal with precise types in the JIT, it's possible to narrow down the domain of discourse, which might make it possible to rule out problematic side-effects. Nevertheless, please dig and experiment. You might find that a combination of custom annotations and JIT work get you what you need. -- William Leslie Q: What is your boss's password? A: "Authentication", clearly Notice: Likely much of this email is, by the nature of copyright, covered under copyright law. You absolutely MAY reproduce any part of it in accordance with the copyright law of the nation you are reading this in. Any attempt to DENY YOU THOSE RIGHTS would be illegal without prior contractual agreement. ___ pypy-dev mailing list pypy-dev@python.org https://mail.python.org/mailman/listinfo/pypy-dev
Re: [pypy-dev] windows 64 bit
On Tue, 14 Apr 2020, 4:35 pm Steven Mathews, wrote: > I saw pypy is windows 32bit. > > Does that mean my code won't be able to use 32GB of RAM, only 4GB ? > Correct. For more detail, please see the FAQ entry here: https://doc.pypy.org/en/latest/faq.html#what-is-needed-for-windows-64-support-of-pypy ___ pypy-dev mailing list pypy-dev@python.org https://mail.python.org/mailman/listinfo/pypy-dev
Re: [pypy-dev] Python 2 vs Python 3 again, and a 2.x-only dependency
On Tue., 16 Jul. 2019, 2:34 pm Ryan Gonzalez, wrote: > I'm actually largely wondering if RPython is going to eventually move to > 3... > >> >> Significant effort, for what benefit exactly? ___ pypy-dev mailing list pypy-dev@python.org https://mail.python.org/mailman/listinfo/pypy-dev
Re: [pypy-dev] let's clean up open branches
On Tue., 12 Feb. 2019, 2:53 am Carl Friedrich Bolz-Tereick Hi all, > > Armin and I briefly looked at the long list of our open branches today. > Could everybody take a look which of "their" branches could be closed, > or maybe merged? see list below, ordered by last committer on the > branch, and date. > > William ML Leslie 2017-01-24 12:54 +1100 > taskengine-sorted-optionals > William ML Leslie 2017-01-24 18:55 +1100 > inline-taskengine > Sorry, i closed one branch in this series and forgot the rest. Feel free to delete these. ___ pypy-dev mailing list pypy-dev@python.org https://mail.python.org/mailman/listinfo/pypy-dev
Re: [pypy-dev] A quick question for you!
On 18 June 2018 at 22:18, Etienne Robillard wrote: > Hi, > > Quick question: Does anyone of you know what is the effect of enabling > gc.enable() in sitecustomize.py when using PyPy? Can it reduce latency for > long-lived WSGI applications? > gc is enabled by default. you only need to use gc.enable() if you have earlier run gc.disable(). -- William Leslie Notice: Likely much of this email is, by the nature of copyright, covered under copyright law. You absolutely MAY reproduce any part of it in accordance with the copyright law of the nation you are reading this in. Any attempt to DENY YOU THOSE RIGHTS would be illegal without prior contractual agreement. ___ pypy-dev mailing list pypy-dev@python.org https://mail.python.org/mailman/listinfo/pypy-dev
[pypy-dev] [CFP] Workshop on Virtual Machines and Language Implementations (VMIL’18)
Stefan Marr had sent this on some JVM lists; but it's super relevant for those of us doing research on or with pypy, too. Call for Papers Workshop on Virtual Machines and Language Implementations (VMIL’18) Co-located with SPLASH 2018 November 4, 2018, Boston, USA https://2018.splashcon.org/track/vmil-2018 Follow us on twitter @vmil18 The concept of virtual machines is pervasive in the design and implementation of programming systems. Virtual machines and the languages they implement are crucial in the specification, implementation and/or user-facing deployment of most programming technologies. The VMIL workshop is a forum for researchers and cutting-edge practitioners in language virtual machines, the intermediate languages they use, and related issues. The workshop is intended to be welcoming to a wide range of topics and perspectives, covering all areas relevant to the workshop’s theme. Aspects of interest include, but are not limited to: - design issues in VMs and IRs (e.g. IR design, VM modularity, polyglotism) - compilation (static and dynamic compilation strategies, optimizations, data representations) - memory management - concurrency (both internal and user-facing) - tool support and related infrastructure (profiling, debugging, liveness, persistence) - the experience of VM development (use of high-level languages, bootstrapping and self-hosting, reusability, portability, developer tooling, etc) - empirical studies on related topics, such as usage patterns, the usability of languages or tools, experimental methodology, or benchmark design Submission Guidelines We invite high-quality papers in the following two categories: Research and experience papers: These submissions should describe work that advances the current state of the art in the above or related areas. The suggested length of these submissions is 6-10 pages (maximum 10pp). Work-in-progress or position papers: These papers should document ongoing efforts in an area of interest which have not yet yielded final results, and/or should present and defend the authors' position on a topic related to the broad area of the workshop. The suggested length of these submissions is 4-6 pages (maximum 6pp). For the first submission deadline, all paper types are considered for publication in the ACM Digital Library, except if the authors prefer not to be included. Publication of work-in-progress and position papers at VMIL is not intended to preclude later publication elsewhere. Submissions will be judged on novelty, clarity, timeliness, relevance, and potential to stimulate discussion during the workshop. For the second deadline, we will consider only work-in-progress and position papers. These will not be published in the ACM DL, and will only appear on the web site. The address of the submission site is: https://vmil18.hotcrp.com/ Important Dates All deadlines are Anywhere on Earth (AoE), i.e. GMT/UTC−12:00 hour Abstract Submission 07 August 2018 Paper Submission17 August 2018 Paper Notification 14 September 2018 ACM Camera Ready Deadline 28 September 2018 “Late submission” deadline 07 September 2018 (WIP/position papers only) “Late submission” notification 30 September 2018 Workshop Date 04 November 2018 Format Instructions Please use the SIGPLAN acmart style for all papers: http://www.sigplan.org/Resources/Author/. The provided double-column template is available for Latex and Word. Program Committee Stephen Kell, University of Cambridge Stefan Marr, University of Kent Leif Andersen, Northeastern University Steve Blackburn, Australian National University Stephen Dolan, University of Cambridge Apala Guha, Simon Fraser University Christine H. Flood, Red Hat Roberto Ierusalimschy, PUC-Rio Tomas Kalibera, Czech Technical University Christos Kotselidis, The University of Manchester Doug Lea, State University of New York (SUNY) Oswego Sanhong Li, Alibaba Inc. Mark Marron, Microsoft Research Erez Petrank, Technion Julien Ponge, INSA Lyon, CITI Laboratory / Red Hat Richard Roberts, Victoria University of Wellington Jeremy Singer, University of Glasgow Sam Tobin-Hochstadt, Indiana University -- Stefan Marr School of Computing, University of Kent http://stefan-marr.de/research/ ___ mlvm-dev mailing list mlvm-...@openjdk.java.net http://mail.openjdk.java.net/mailman/listinfo/mlvm-dev -- William Leslie Notice: Likely much of this email is, by the nature of copyright, covered under copyright law. You absolutely MAY reproduce any part of it in accordance with the copyright law of the nation you are reading this in. Any attempt to DENY YOU THOS
Re: [pypy-dev] Mysterious IndexError in service running with PyPy
Where is the code that changes the size of self.heap? How do we know that size(self.heap) is constant? My guess is that some thread changes this; but l is not recomputed. On 18 Dec 2017 6:59 PM, "hubo" wrote: > I'm reporting this issue in this mail group, though I don't know if it is > related with PyPy, because it is really strange, and is not able to > reproduce stably. But I hope someone may know the reason or have some > points. > > I'm running some SDN services written in Python with PyPy 5.3.0. The > related code is here: > > https://github.com/hubo1016/vlcp/blob/8022e3a3c67cf4305af503d507640a730ca3949d/vlcp/utils/indexedheap.py#L100 > > The full code is also in the repo, but may be too complex to describe. But > this related piece is quite simple: > > def _siftdown(self, pos): > temp = self.heap[pos] > l = len(self.heap) > while pos * 2 + 1 < l: > cindex = pos * 2 + 1 > pt = self.heap[cindex] > if cindex + 1 < l and self.heap[cindex+1][0] < pt[0]: > cindex = cindex + 1 > pt = self.heap[cindex] > if pt[0] < temp[0]: > self.heap[pos] = pt > self.index[pt[1]] = pos > else: > break > pos = cindex > self.heap[pos] = temp > self.index[temp[1]] = pos > It is a simple heap operation. The service uses a heap to process timers. > When the service is not busy, it usually runs this piece of code several > times per minute. > > I have 32 servers running this service. They are quite stable in about > three months, but one day one of the services crashes on line 100 reporting > IndexError: > pt = self.heap[cindex] > > As you can see, cindex = pos * 2 + 1, which is tested by the while > pre-conditon just two lines before. And there is not any multi-threading > issues here because this piece of code always runs in the same thread. So > it is not possible in theory for this to happen. > > Only the following facts are known about this issue: > >1. It reproduces - through quite rarely. I've met this kind of >crashes 4 times, each with different machine, so it should not be related >to hardware issuess. Since I've got 32 servers, it might take more than one >year to reproduce with a single server. >2. It seems not to be related to pressures. All of the crashes happens >at night, when there are little requests. Only some cleanup tasks are >running in fixed interval. > > The services are running with PyPy 5.3.0. I've upgraded a few of them to > 5.9, but it will take a long time to validate whether this still happens. > And It is not validated on CPython too. I'm also trying to collect more > debugging information for this issue, but it is very hard since it rarely > reproduces. > > It is not a serious issue. It could be workarounded with a auto-restart, > but I'm searching the cause. > > > 2017-12-18 > -- > hubo > > ___ > pypy-dev mailing list > pypy-dev@python.org > https://mail.python.org/mailman/listinfo/pypy-dev > > ___ pypy-dev mailing list pypy-dev@python.org https://mail.python.org/mailman/listinfo/pypy-dev
Re: [pypy-dev] Static bytecode instruction optimization and pypy (JIT?) optimization
On 11 July 2017 at 19:28, Carl Friedrich Bolz wrote: > On 11/07/17 10:37, Rocky Bernstein wrote: >> Sure but that's a straw argument and has a lot of packed opinions in it. >> Things like "run my program exactly like I expect" is loaded with >> opinions. As has already been noted, Python removes "if 0:" by >> default. When I first saw that, I thought it was cool, but honestly it >> wasn't "exactly as I expect" because I wanted my decompiler to recreate >> the source code as written which helps me in testing and lo and behold I >> was getting something different :-) You know what? I got use to it. I >> had to change my opinion of what "exactly as I expect" meant slightly. > > Actually, to me this would be an argument to remove the "if 0:" > optimization ;-). > Not least of all because `if False:` is more ideomatic, but False can be bound to a value that is True, so the statement can't be folded! The lengths the compiler goes to in order to do what you told it (: and yet we still get bug reports that our built-in objects have methods that cpython's built-ins don't have and this makes some popular library give the wrong result. Oblig: https://twitter.com/shit_jit_says/status/446914805191172096 https://twitter.com/fijall/status/446913102190505984 -- William Leslie Notice: Likely much of this email is, by the nature of copyright, covered under copyright law. You absolutely MAY reproduce any part of it in accordance with the copyright law of the nation you are reading this in. Any attempt to DENY YOU THOSE RIGHTS would be illegal without prior contractual agreement. ___ pypy-dev mailing list pypy-dev@python.org https://mail.python.org/mailman/listinfo/pypy-dev
Re: [pypy-dev] Static bytecode instruction optimization and pypy (JIT?) optimization
TLDR: Doing these sort of optimisations on /python/ requires you understand the evaluation model. You can't just do store-sinking like descriptors aren't a thing, and you can't go inlining functions like module globals aren't mutable bindings. #XXTiredCompilation On 11 July 2017 at 18:53, Rocky Bernstein wrote: > > > On Tue, Jul 11, 2017 at 4:35 AM, William ML Leslie > wrote: >> >> On 11 July 2017 at 18:22, Rocky Bernstein wrote: >> > There's too much generalization and abstraction in the discussion at >> > least >> > for me. So let me try to make this a little more concrete with real >> > examples. First note that, although Python, yes, does "optimize" "if 0:" >> > away, as soon as >> > you do something as simple as "debug = 0; if debug:" it is confounded. >> > And >> > in my own experience I have done this. >> > >> >> Typically DEBUG is a global in the module, which means it can be set >> from /outside/ the module, so it's not constant as far as the runtime >> is concerned. Within a function it's a bit wacky sure, and is only >> kept around for interactive debugging reasons afaict. > > > You presume to know much more about the programmer's intentions and behavior > than I would care to venture into, or how and why such things could arise. > If the programmer assigns a module-level variable, then the variable should appear to be set in that module. What other intention do you suppose to give this operation? > >> >> Here's a possible type for `self.a` which illustrates why >> CSE/Store-Sinking on a STORE_ATTR is not valid for the example >> function `foo`. >> >> class A(object): >> def set_b(self, value): >> print value >> self._b = value >> b = property(set_b) >> >> class Example(object): >> def __init__(self): >> self.a = A() >> >> def foo(self): >> self.a.b = 5 >> x = 0 >> if x: >> self.a.b = 2 >> else: >> return >> >> Example().foo() > > > > Absolutely one can create pathological cases like this. I hope though this > isn't your normal mode of coding though. > Descriptors - including properties - are a language feature that people should be able to expect to work. You can't just compile their invocation out because you think it will make their program faster. > I suspect that this kind of thing doesn't occur often. I don't think you'll find many in industry that agree with you. Have you *seen* the level of magic behind django, flask, pandas? Have you ever had people bring their super-obscure bugs to you when they run their program with your runtime? > You know, I > encountered something analogous when I first started using Python's code > coverage tool, coverage, and the Python style checker flake8. As per Mark > Pilgram's original version of Dive into Python I used: > > if __name__ == '__main__': > > And I was annoyed that older versions of code coverage dinged me because > that code was untested code unless I took special care to run it in testing. > (Which sometimes I would do). > PSA: don't run your tests directly, use a test runner, and say goodbye to all the awful hacks around getting your path setup correctly and bugs when the tests run against the installed version rather than the version you are editing. -- William Leslie Notice: Likely much of this email is, by the nature of copyright, covered under copyright law. You absolutely MAY reproduce any part of it in accordance with the copyright law of the nation you are reading this in. Any attempt to DENY YOU THOSE RIGHTS would be illegal without prior contractual agreement. ___ pypy-dev mailing list pypy-dev@python.org https://mail.python.org/mailman/listinfo/pypy-dev
Re: [pypy-dev] Static bytecode instruction optimization and pypy (JIT?) optimization
On 11 July 2017 at 18:22, Rocky Bernstein wrote: > There's too much generalization and abstraction in the discussion at least > for me. So let me try to make this a little more concrete with real > examples. First note that, although Python, yes, does "optimize" "if 0:" > away, as soon as > you do something as simple as "debug = 0; if debug:" it is confounded. And > in my own experience I have done this. > Typically DEBUG is a global in the module, which means it can be set from /outside/ the module, so it's not constant as far as the runtime is concerned. Within a function it's a bit wacky sure, and is only kept around for interactive debugging reasons afaict. Here's a possible type for `self.a` which illustrates why CSE/Store-Sinking on a STORE_ATTR is not valid for the example function `foo`. class A(object): def set_b(self, value): print value self._b = value b = property(set_b) class Example(object): def __init__(self): self.a = A() def foo(self): self.a.b = 5 x = 0 if x: self.a.b = 2 else: return Example().foo() -- William Leslie Notice: Likely much of this email is, by the nature of copyright, covered under copyright law. You absolutely MAY reproduce any part of it in accordance with the copyright law of the nation you are reading this in. Any attempt to DENY YOU THOSE RIGHTS would be illegal without prior contractual agreement. ___ pypy-dev mailing list pypy-dev@python.org https://mail.python.org/mailman/listinfo/pypy-dev
Re: [pypy-dev] Static bytecode instruction optimization and pypy (JIT?) optimization
On 10 July 2017 at 11:10, Rocky Bernstein wrote: > In looking at Python bytecode over the years, I've noticed that it does very > little to no traditional static-compiler optimization. Specifically: > > > Yes, over the years more compiler optimization has been done but it's still > pretty sparse. > > The little that I've looked at pypy, it is pretty much the same thing at the > Python bytecode level. That's why a python decompiler for say 2.7 will work > largely unmodified for he corresponding pypy 2.7 version. Same is true 3.2 > versus pypy 3.2. > > I understand though that pypy can do and probably does some of the > optimization when it JITs. But my question is: if traditional compiler > optimization were done would this hinder, be neutral, or help pypy's > optimization. > It'd likely be neutral with respect to the JIT optimisation, which operates on traces (what actually happens to the rpython objects) rather than bytecode. Within a trace, which is a fairly closed universe, all of those optimisations are very easy to make. As Ryan has mentioned, the bigger concern is going to be around the expectations that people have on how python code will behave, and a secondary concern is going to be people relying on optimisations that only work on pypy. It's one thing for people to tune their programs for pypy, and another for people to rely on certain optimisations being performed for correct execution. If you attempt this, here's a few things to keep in mind: > * Dead code elmination (most of the time) Mostly a code size optimisation and not so key for performance, but it is very strange that python generates different code for `return` at the end of a function and `else: return` also at the end of a function. On the other hand, `if 0:` statements are removed, which is handy. > * Jumps to Jumps or Jumps to the next instruction SGTM > * Constant propagation (most of the time) This is an interesting one. Consider beta reducing xs in this code (assume xs is otherwise unused): \[ xs = [1, 7, 2] if x in xs: \] Doing so would allow the list to be converted to a tuple and stored as a constant against the code object (mostly because the value is never used in an identity-dependent way). However, people can use the debugger to set a breakpoint on the assignment to xs, which they can't do if no evaluation happens on this line. They can even set xs to be a different value in the debugger before continuing. I'm a big fan of prebuilding objects at compile time, it's something that the pypy team wished the JVM could do back when the Java backend was written because loading time was awful. Even now the JVM is making changes to the constant pool layout that improve that possibility. > * Common subexpression elimination This is not widely applicable in python, in fact the only subexpressions that can be eliminated are a small group of expressions that apply to objects constructed from literals. A notable example is that you can't remove a duplicated `x = 1 + y` because the object that `y` refers to may have overridden __radd__ and that method may even have side-effects. There can be some success when either whole-program optimisation, encoding preconditions into optimistically compiled module code, or intraprocedural effect analysis is performed, but there isn't much precedent for the first two in pypy and the third is the subject of ongoing research. > * Tail recursion For functions that the compiler has shown are total and have a reasonable and static bound on memory allocation and time? Any situation where asyncs need to be suspended, lest anyone hit C-c and generate a traceback / kill the operation because it is taking too long, need to be very tightly time bounded. > * Code hoisting/sinking Again, without a really solid type or effect system it's really hard to know you're not changing the behaviour of the program when performing such code movement, because almost all operations can delegate to user-defined methods. > Of course, if there were static compiler optimization of the type described > above, that might be a win when JITing doesn't kick in. (And perhaps then > who cares) > > But I'm interested in the other situation where both are in play. > A direction that might be interesting is seeing how far you can get by making reasonable assumptions (module bindings for certain functions have their original values, this value has this exact code object as its special method `__radd__` or method `append`, this module `foo` imported has this exact hash), checking them and branching to optimised code, which may or may not be pypy bytecode. The preconditions themselves can't be written in pypy bytecode - you don't want to be executing side effects when pre-emptively *looking up* methods, which can happen due to descriptors, so you'd need to go behind the scenes. Nevertheless, they can be platform-independent and could live in the .pyc for a first cut. At the moment, it's necess
Re: [pypy-dev] Asking for help
On 3 July 2017 at 15:47, Meide Zhao wrote: > Thanks a lot Leslie. > > Can I use the newer version of Ubuntu like 16.04 or 17.04? Which Linux and > which version do you use? > Tannit (which does the automatic linux-64 builds) has GCC 4.8.2, so even something that old should work. I'm not sure if it differs from 12.04 in any other way. I build pypy regularly on Ubuntu 16.04 LTS (has gcc 5.4.0) as well as Debian Jessie (with gcc 4.9.2). -- William Leslie Notice: Likely much of this email is, by the nature of copyright, covered under copyright law. You absolutely MAY reproduce any part of it in accordance with the copyright law of the nation you are reading this in. Any attempt to DENY YOU THOSE RIGHTS would be illegal without prior contractual agreement. ___ pypy-dev mailing list pypy-dev@python.org https://mail.python.org/mailman/listinfo/pypy-dev
Re: [pypy-dev] Asking for help
Also: Please don't run compilers with sudo. On 3 July 2017 at 11:16, Meide Zhao wrote: > (4)build pypy > cd ~/pypy/pypy/pypy/goal/ > sudo ../../rpython/bin/rpython --opt=jit targetpypystandalone.py > > -- William Leslie Notice: Likely much of this email is, by the nature of copyright, covered under copyright law. You absolutely MAY reproduce any part of it in accordance with the copyright law of the nation you are reading this in. Any attempt to DENY YOU THOSE RIGHTS would be illegal without prior contractual agreement. ___ pypy-dev mailing list pypy-dev@python.org https://mail.python.org/mailman/listinfo/pypy-dev
Re: [pypy-dev] Asking for help
On 3 July 2017 at 11:16, Meide Zhao wrote:
> Hi all,
>
> I'm trying to build pypy from source on ubuntu 12.04 LTS but can't get it
> to work (see error messages below). Can someone help me and let me know the
> correct steps since I'm new to pypy?
>
Hi! 12.04 reached End Of Life at the end of April. It has GCC 4.6.3 so
I'm not surprised pypy isn't building on it.
At a minimum you'll need a more recent GCC, but it would probably be best
to update to an LTS version that still has support.
>
> Thanks,
> Meide
>
> (1) The details for the ubuntu in virtualbox are as follows
> cat /etc/*-release
>DISTRIB_ID=Ubuntu
>DISTRIB_RELEASE=12.04
>DISTRIB_CODENAME=precise
>DISTRIB_DESCRIPTION="Ubuntu 12.04.3 LTS"
>NAME="Ubuntu"
>VERSION="12.04.3 LTS, Precise Pangolin"
>ID=ubuntu
>ID_LIKE=debian
>PRETTY_NAME="Ubuntu precise (12.04.3 LTS)"
>VERSION_ID="12.04"
> uname -a
>Linux ubuntu 3.8.0-29-generic #42~precise1-Ubuntu SMP Wed Aug 14
> 16:19:23 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux
> python -V
> Python 2.7.13
>
> (2) install pypy dependencies via command:
> sudo apt-get install gcc make libffi-dev pkg-config zlib1g-dev
> libbz2-dev libsqlite3-dev libncurses5-dev libexpat1-dev libssl-dev
> libgdbm-dev tk-dev libgc-dev liblzma-dev mercurial
>
> (3) check out pypy source code:
> cd ~/pypy
> hg clone http://bitbucket.org/pypy/pypy pypy
>
> (4)build pypy
> cd ~/pypy/pypy/pypy/goal/
> sudo ../../rpython/bin/rpython --opt=jit targetpypystandalone.py
>
>
> [platform:execute] gcc -c -O3 -pthread -fomit-frame-pointer -Wall -Wno-unused
> -Wno-address -DRPYTHON_VMPROF -O3 -DVMPROF_UNIX -DVMPROF_LINUX
> -DRPYTHON_LL2CTYPES -I/home/meide/pypy/pypy/rpython/rlib/rvmprof/src
> -I/home/meide/pypy/pypy/rpython/rlib/rvmprof/src/shared
> -I/home/meide/pypy/pypy/rpython/rlib/rvmprof/src/shared/libbacktrace
> /home/meide/pypy/pypy/rpython/rlib/rvmprof/src/shared/libbacktrace/elf.c -o
> /tmp/usession-default-0/rpython/rlib/rvmprof/src/shared/libbacktrace/elf.o
> [translation:info] Error:
>File "/home/meide/pypy/pypy/rpython/translator/goal/translate.py", line
> 284, in main
> default_goal='compile')
>File "/home/meide/pypy/pypy/rpython/translator/driver.py", line 566, in
> from_targetspec
> spec = target(driver, args)
>File "targetpypystandalone.py", line 337, in target
> return self.get_entry_point(config)
>File "targetpypystandalone.py", line 368, in get_entry_point
> space = make_objspace(config)
>File "/home/meide/pypy/pypy/pypy/tool/option.py", line 35, in make_objspace
> return Space(config)
>File "/home/meide/pypy/pypy/pypy/interpreter/baseobjspace.py", line 441,
> in __init__
> self.initialize()
>File "/home/meide/pypy/pypy/pypy/objspace/std/objspace.py", line 105, in
> initialize
> self.make_builtins()
>File "/home/meide/pypy/pypy/pypy/interpreter/baseobjspace.py", line 637,
> in make_builtins
> self.install_mixedmodule(mixedname, installed_builtin_modules)
>File "/home/meide/pypy/pypy/pypy/interpreter/baseobjspace.py", line 668,
> in install_mixedmodule
> modname = self.setbuiltinmodule(mixedname)
>File "/home/meide/pypy/pypy/pypy/interpreter/baseobjspace.py", line 507,
> in setbuiltinmodule
> None, None, ["Module"]).Module
>File "/home/meide/pypy/pypy/pypy/module/_vmprof/__init__.py", line 30, in
>
> import pypy.module._vmprof.interp_vmprof
>File "/home/meide/pypy/pypy/pypy/module/_vmprof/interp_vmprof.py", line
> 14, in
> my_execute_frame = _decorator(PyFrame.execute_frame)
>File "/home/meide/pypy/pypy/rpython/rlib/rvmprof/rvmprof.py", line 196, in
> decorate
> _get_vmprof()
>File "/home/meide/pypy/pypy/rpython/rlib/rvmprof/rvmprof.py", line 243, in
> _get_vmprof
> _vmprof_instance = VMProf()
>File "/home/meide/pypy/pypy/rpython/rlib/rvmprof/rvmprof.py", line 49, in
> __init__
> self.cintf = cintf.setup()
>File "/home/meide/pypy/pypy/rpython/rlib/rvmprof/cintf.py", line 82, in
> setup
> **eci_kwds))
>File "/home/meide/pypy/pypy/rpython/rtyper/tool/rffi_platform.py", line
> 94, in verify_eci
> configure(CConfig)
>File "/home/meide/pypy/pypy/rpython/rtyper/tool/rffi_platform.py", line
> 223, in configure
> res[key] = value.question(writer.ask_gcc)
>File "/home/meide/pypy/pypy/rpython/rtyper/tool/rffi_platform.py", line
> 555, in question
> ask_gcc("")
>File "/home/meide/pypy/pypy/rpython/rtyper/tool/rffi_platform.py", line
> 191, in ask_gcc
> try_compile_cache([self.path], self.eci)
>File "/home/meide/pypy/pypy/rpython/tool/gcc_cache.py", line 71, in
> try_compile_cache
> platform.compile(c_files, eci)
>File "/home/meide/pypy/pypy/rpython/translator/platform/__init__.py", line
> 54, in compile
> ofiles = self._compile_o_files(cfiles, eci, standalone)
>File "/home/meide/pypy/pypy/rpython/translat
[pypy-dev] Fwd: [CfP][Meta'17] Workshop on Meta-Programming Techniques and Reflection
-- Forwarded message -- From: Stefan Marr Date: 29 June 2017 at 23:22 Subject: [CfP][Meta'17] Workshop on Meta-Programming Techniques and Reflection To: mlvm-...@openjdk.java.net Call for Papers: Meta’17 Workshop on Meta-Programming Techniques and Reflection Co-located with SPLASH 2017 October 22, 2017, Vancouver, Canada Twitter @MetaAtSPLASH http://2017.splashcon.org/track/meta-2017 The heterogeneity of mobile computing, cloud applications, multicore architectures, and other systems leads to increasing complexity of software and requires new approaches to programming languages and software engineering tools. To manage the complexity, we require generic solutions that can be adapted to specific application domains or use cases, making metaprogramming an important topic of research once more. However, the challenges with metaprogramming are still manifold. They start with fundamental issues such as typing of reflective programs, continue with practical concerns such as performance and tooling, and reach into the empirical field to understand how metaprogramming is used and how it affects software maintainability. Thus, while industry accepted metaprogramming on a wide scale with Ruby, Scala, JavaScript and others, academia still needs to answer a wide range of questions to bring it to the same level of convenience, tooling, and programming styles to cope with the increasing complexity of software systems. This workshop aims to explore meta-level technologies that help tackling the heterogeneity, scalability and openness requirements of emerging computations platforms. ### Topics of Interest The workshop is a venue for all approaches that embrace metaprogramming: - from static to dynamic techniques - reflection, meta-level architectures, staging, open language runtimes applications to middleware, frameworks, and DSLs - optimization techniques to minimize runtime overhead - contract systems, or typing of reflective programs reflection and metaobject protocols to enable tooling - case studies and evaluation of such techniques, e.g., to build applications, language extensions, or tools - empirical evaluation of metaprogramming solutions - security in reflective systems and capability-based designs - meta-level architectures and reflective middleware for modern runtime platforms (e.g. IoT, cyber-physical systems, mobile/cloud/grid computing, etc) - surveys, conceptualization, and taxonomization of existing approaches In short, we invite contributions to the workshop on a wide range of topics related to design, implementation, and application of reflective APIs and meta-programming techniques, as well as empirical studies and typing for such systems and languages. ### Workshop Format and Submissions This workshop welcomes the presentation of new ideas and emerging problems as well as mature work as part of a mini-conference format. Furthermore, we plan interactive brainstorming and demonstration sessions between the formal presentations to enable an active exchange of ideas. The workshop papers will be published in the ACM DL, if not requested otherwise by the authors. Thus, they will be part of SPLASH workshop proceedings. Therefore, papers are to be submitted using the SIGPLAN acmart style: http://www.sigplan.org/Resources/Author/. Please use the provided double-column templates for Latex http://www.sigplan.org/sites/default/files/acmart/current/ acmart-sigplanproc-template.tex) or Word http://www.acm.org/publications/proceedings-template. - technical paper: max. 8 pages, excluding references - position and work-in-progress paper: 1-4 pages, excluding references - technology demos or a posters: 1-page abstract Demos, posters, position and work-in-progress papers can be submitted on a second, later deadline to discuss the latest results and current work. For the submission, please use the submission system at: https://meta17.hotcrp.com/ ### Important Dates Abstract Submission: 07 August 2017 Paper Submission: 14 August 2017 Author Notification: 06 September 2017 Position/WIP Paper Deadline: 08 September 2017 Camera Ready Deadline:18 September 2017 Position/WIP Notification:21 September 2017 ### Program Committee The program committee consists of the organizers and the following reviewers: Anya Helen Bagge, University of Bergen, Norway Daniele Bonetta, Oracle Labs, Austria Nicolas Cardozo, Universidad de los Andes, Colombia Sebastian Erdweg, TU Delf, The Nederlands Robert Hirschfeld, HPI, Germany Roberto Ierusalimschy, PUC-Rio, Brazil Pablo Inostroza, CWI, The Nederlands Kim Mens, Universite Catholique de Louvain, Belgium Cyrus Omar, Carnegie Mellon University, USA Guillermo Polito, CNRS, France Tiark R
Re: [pypy-dev] Integer division
On 1 June 2017 at 20:53, Armin Rigo wrote: > Hi, > > On 31 May 2017 at 17:11, Tuom Larsen wrote: > > # k = i//j # 2.12 seconds > > k = int(i/j) # 0.98 seconds > > Note first that if you don't do anything with 'k', it might be optimized > away. > > I just wrote a pure C example doing the same thing, and indeed > converting the integers to float, dividing, and then converting back > to integer... is 2.2x times faster there too. > > Go figure it out. I have no idea why the CPU behaves like that. > Maybe Neal can provide a clue. > I took a look at this earlier today. It turns out that, on Skylake (and I think it's similar on other recent x86_64 implementations), 80-bit floating point division has a latency of 14 cycles, where 32 bit integer division has a latency of 26 cycles. I expect this is because there are only two hardware division units, and both are on the floating point path. -- William Leslie Notice: Likely much of this email is, by the nature of copyright, covered under copyright law. You absolutely MAY reproduce any part of it in accordance with the copyright law of the nation you are reading this in. Any attempt to DENY YOU THOSE RIGHTS would be illegal without prior contractual agreement. ___ pypy-dev mailing list pypy-dev@python.org https://mail.python.org/mailman/listinfo/pypy-dev
Re: [pypy-dev] What is RuntimeTypeInfo?
On 16 March 2017 at 11:47, John Zhang wrote: > Hi Carl, Armin, William, > I have thought about modifying the JIT code instruction set, descriptors and > runtime rewrite etc. to encode the MuTyped CFG (which is a further type and > ops specialisation towards the Mu MicroVM) for Mu back-end. But I presume > this will involve a large amount of work? Would this be the case? And thus > this wouldn’t be a good idea, right? > The more of the metainterp you can make use of the better. I would start by trying to push your graphs through the codewriter and seeing how/if that fails. LLTyped graphs have gone to the effort to encode the types involved into the identifier of the operation, eg, you get float_abs and llong_rshift. If you are not able to do something like that, you can still pass constants as arguments to the operation, and then ignore them in the backend (or use them to generate a particular operation) and have the codewriter preserve them for analysis later. They will get recorded in the constant table on the jitcodes. For an example of this, note that indirect_call operations maintain a set of possible targets as their last argument; it might be illustrative to search for that name. I guess it is worth asking: What other operations are you finding difficult to type? I get that the lltype specialisation of new_ is one, and that address manipulation with composite offsets is another (though it turns out to always be well-typed in practice). -- William Leslie Notice: Likely much of this email is, by the nature of copyright, covered under copyright law. You absolutely MAY reproduce any part of it in accordance with the copyright law of the nation you are reading this in. Any attempt to DENY YOU THOSE RIGHTS would be illegal without prior contractual agreement. ___ pypy-dev mailing list pypy-dev@python.org https://mail.python.org/mailman/listinfo/pypy-dev
Re: [pypy-dev] What is RuntimeTypeInfo?
On 14 March 2017 at 11:17, John Zhang wrote: > Hi all, > Can anyone does the favour for me by explaining the story of RuntimeTypeInfo > in RPython? Hi John! The RTTI are a hook that the backend can implement, there is a fair bit of flexibility in what values they can take. The hooks in the C backend live here: https://bitbucket.org/pypy/pypy/src/699382943bd73bf19565e996d2042d54e7569e31/rpython/translator/c/gc.py?at=default&fileviewer=file-view-default#gc.py-145 This class is one example of a node for an RTTI value. In this one, the RTTI value is a function that can statically deallocate structs of this type. There are more in this file, for example the RTTI for a struct in a framework GC is just an identifier iiuc. I think you want to translate your opaque types yourself, possibly > I couldn’t quite understand the description on the online documentation > (http://rpython.readthedocs.io/en/latest/rtyper.html#opaque-types). From > looking at the source code, and inspecting the generated C source, RTTI > doesn’t seem to be used at all at runtime. The generated C source seems to > just return an uninitialised function pointer on the stack when compiling > the following code: > > from rpython.rtyper.rclass import OBJECTPTR > class A: > pass > > class B(A): > pass > > def f(a): > obj = rffi.cast(OBJECTPTR, a) > return lltype.runtime_type_info(obj) > > t = Translation(f, [A], backend='c') > t.backendopt(mallocs=True) > t.view() > lib = t.compile() > This example (which uses the refcount GC) grabs a function from the type OBJECTPTR that can de-allocate an OBJECTPTR (that is, it can decrement the refcount of the object `a`). I haven't had a look at why it would be uninitialised. > We were looking at RTTI in finding a solution to the loss of type > information problem at JIT encoding. My collaborators are trying to develop > a JIT back-end targeting a micro virtual machine (Mu). It seems that by > default the JIT transformer throws away the actual type information > (GcStruct etc., which is not representable under RPython, I know) and only > keeps the size. However, in Mu, memory allocation requires specific type > information. Thus, among other ways, we are trying to see how much we can > recover this object layout/type information. RTTI seems promising based on > the description on the documentation, but I can’t picture what it looks like > at run time. > Can anyone provide some insight on this? > The low-level allocation operations specify size as an operand - it might be better for you to translate the various new_ operations into a form you can make use of long before jitcode generation. You'll need to extend the codewriter to allow for those operations, too. -- William Leslie Notice: Likely much of this email is, by the nature of copyright, covered under copyright law. You absolutely MAY reproduce any part of it in accordance with the copyright law of the nation you are reading this in. Any attempt to DENY YOU THOSE RIGHTS would be illegal without prior contractual agreement. ___ pypy-dev mailing list pypy-dev@python.org https://mail.python.org/mailman/listinfo/pypy-dev
[pypy-dev] Fwd: [MoreVMs'17] Call for Contributions: 1st Workshop on Modern Language Runtimes, Ecosystems, and VMs at 2017
Hello pypy contributors and rpython users, In case any of you missed this CFP this looks like an exciting workshop! -- Forwarded message -- From: Stefan Marr Date: 6 December 2016 at 00:56 Subject: [MoreVMs'17] Call for Contributions: 1st Workshop on Modern Language Runtimes, Ecosystems, and VMs at 2017 To: mlvm-...@openjdk.java.net Call for Papers: MoreVMs’17 1st Workshop on Modern Language Runtimes, Ecosystems, and VMs Co-located with 2017 April, 2017, Brussels, Belgium http://2017.programming-conference.org/track/MoreVMs-2017-papers The MoreVMs'17 workshop aims bring together programmers from industry and academy to discuss the design, implementation, and usage of modern languages and runtimes. This includes aspects such as reuse of language runtimes, modular implementation, or design and compilation strategies to target existing runtimes. The main goals of the workshop is to bring together both researchers and practitioners and facilitate effective sharing of their respective experiences and ideas on how languages and runtimes are utilized and where they need to improve further. We welcome presentation proposals in the form of extended abstracts discussing experiences, work-in-progress, as well as future visions from the academic as well as industrial perspective. Relevant topics include, but are definitely not limited to, the following: - extensible VM design (compiler- or interpreter-based VMs) - reusable runtime components (e.g. interpreters, garbage collectors, intermediate representations) - static and dynamic compiler techniques - techniques for compilation to high-level languages such as JavaScript - runtimes and mechanisms for interoperability between languages - tooling support (e.g. debugging, profiling, etc.) - programming language development environments and virtual machines - case studies of existing language implementations, virtual machines, and runtime components (e.g. design choices, tradeoffs, etc.) - language implementation challenges and trade-offs (e.g. performance, completeness, etc.) - surveys and applications usage reports to understand runtime usage in the wild - surveys on frameworks and their impact on runtime usage - new research ideas on how we want to build languages in the future ### Workshop Format and Submissions This workshop welcomes the presentation and discussion of new ideas and emerging problems to facilitate interaction among workshop participants and exchange of ideas. We accept presentation proposals in the form of extended abstracts (1-2 pages). Accepted abstracts will be published on the workshop's website before the workshop date. For preparing your abstract, please use the provided author kit: https://github.com/smarr/morevms17-author-kit. It is based on the ACM SIGPLAN Conference Format with 10 point font, and includes a Creative Commons License, which will allow us to publish the abstract on the workshop web site. Please submit abstracts through http://ssw.jku.at/morevms17/ ### Important Dates Abstract submission: 15 February 2017 Author notification: 01 March 2017 Workshop: 3 April 2017 All deadlines are Anywhere on Earth (AoE), i.e. GMT/UTC−12:00 hour ### Program Committee Matthis Grimmer, Oracle Labs Christine H. Flood, Red Hat Tony Hosking, Australian National University Hannes Payer, Purdue University Jeremy Singer, University of Glasgow Mark Stoodley, IBM Canada Sam Tobin-Hochstadt, Indiana University ### Workshop Organizers Laurence Tratt, King's College London, United Kingdom Adam Welc, Oracle Labs, United States Stefan Marr, Johannes Kepler University Linz, Austria -- Stefan Marr Johannes Kepler Universität Linz http://stefan-marr.de/research/ ___ mlvm-dev mailing list mlvm-...@openjdk.java.net http://mail.openjdk.java.net/mailman/listinfo/mlvm-dev -- William Leslie Notice: Likely much of this email is, by the nature of copyright, covered under copyright law. You absolutely MAY reproduce any part of it in accordance with the copyright law of the nation you are reading this in. Any attempt to DENY YOU THOSE RIGHTS would be illegal without prior contractual agreement. ___ pypy-dev mailing list pypy-dev@python.org https://mail.python.org/mailman/listinfo/pypy-dev
Re: [pypy-dev] cppyy fails to build on gcc 5 and clang
On 18 January 2017 at 21:02, Antonio Cuni wrote: > Hi Tobias, > > Note also that capnpy is serialization only: there is no support for RPC > stuff. > Although if you want to work on it, I've got some preliminary support for it in my fork that I mostly ripped from an internal CapTP implementation I wrote at work. I wasn't going to look at it in the short term, but if there's interest I can get back to it, fix interface generation and rewrite it to be SansIO but otherwise feel free to use whatever code you like from it. https://bitbucket.org/william_ml_leslie/capnpy I'd love too see CapnProto hooked up to Autobahn/WS on Twisted too! I don't think there's a good client-side implementation of CapnProto though, so there's still a missing part of that pipeline. -- William Leslie Notice: Likely much of this email is, by the nature of copyright, covered under copyright law. You absolutely MAY reproduce any part of it in accordance with the copyright law of the nation you are reading this in. Any attempt to DENY YOU THOSE RIGHTS would be illegal without prior contractual agreement. ___ pypy-dev mailing list pypy-dev@python.org https://mail.python.org/mailman/listinfo/pypy-dev
Re: [pypy-dev] Fwd: Re: Adding init/variables to W_Root
On 12 January 2017 at 09:08, Frank Wang wrote: > def binaryoperation(operationname): > """NOT_RPYTHON""" > def opimpl(self, *ignored): > operation = getattr(self.space, operationname) > w_2 = self.popvalue() > w_1 = self.popvalue() > w_result = operation(w_1, w_2) > > # union flags in w_1 and w_2 and propagate to result > w_1_rbflags = w_1.get_rbflags() > w_2_rbflags = w_2.get_rbflags() > w_1_rbflags.update(w_2_rbflags) > w_result.set_rbflags(w_1_rbflags) > > self.pushvalue(w_result) It looks like, if an app-level exception is in progress, w_result may be None. Try checking for that before setting the flags. Also: did you mean to alter w_1's rbflags? Seems strange that you'd alter w_1 but not w_2. -- William Leslie Notice: Likely much of this email is, by the nature of copyright, covered under copyright law. You absolutely MAY reproduce any part of it in accordance with the copyright law of the nation you are reading this in. Any attempt to DENY YOU THOSE RIGHTS would be illegal without prior contractual agreement. ___ pypy-dev mailing list pypy-dev@python.org https://mail.python.org/mailman/listinfo/pypy-dev
Re: [pypy-dev] Object pinning
On 22 December 2016 at 15:56, Kunshan Wang wrote:
> Hi folks,
Hi Kunshan!
>
> I have a question regarding object pinning in RPython.
>
> Consider the following snippet from rpython/rtyper/lltypesystem/rstr.py
>
> @jit.oopspec('stroruni.copy_contents(src, dst, srcstart, dststart,
> length)')
> @signature(types.any(), types.any(), types.int(), types.int(),
> types.int(), returns=types.none())
> def copy_string_contents(src, dst, srcstart, dststart, length):
> """Copies 'length' characters from the 'src' string to the 'dst'
> string, starting at position 'srcstart' and 'dststart'."""
> # xxx Warning: don't try to do this at home. It relies on a lot
> # of details to be sure that it works correctly in all cases.
> # Notably: no GC operation at all from the first cast_ptr_to_adr()
> # because it might move the strings. The keepalive_until_here()
> # are obscurely essential to make sure that the strings stay alive
> # longer than the raw_memcopy().
> assert length >= 0
> ll_assert(srcstart >= 0, "copystrc: negative srcstart")
> ll_assert(srcstart + length <= len(src.chars), "copystrc: src ovf")
> ll_assert(dststart >= 0, "copystrc: negative dststart")
> ll_assert(dststart + length <= len(dst.chars), "copystrc: dst ovf")
> # from here, no GC operations can happen
> asrc = _get_raw_buf(SRC_TP, src, srcstart)
> adst = _get_raw_buf(DST_TP, dst, dststart)
> llmemory.raw_memcopy(asrc, adst, llmemory.sizeof(CHAR_TP) * length)
> # end of "no GC" section
> keepalive_until_here(src)
> keepalive_until_here(dst)
> copy_string_contents._always_inline_ = True
> copy_string_contents = func_with_new_name(copy_string_contents,
> 'copy_%s_contents' % name)
>
> There is a region where heap objects in the RPython heap is accessed
> externally by native programs. I understand that GC must neither
> recycle the object nor move it in the memory. But I have two questions
> about how object pinning is done in RPython:
>
> (1) From the perspective of the RPython user (e.g. high-level language
> implementer, interpreter writer, library writer, ...), what is the
> "protocol" to follow when interacting with native programs (such as
> passing a buffer to the `read` syscall)? I have seen idiomatic use of
> `cast_ptr_to_adr` followed by `keepalive_until_here`. But there is also
> `pin` and `unpin` functions in the rpython/rlib/rgc.py module. What is
> the expected way for *the user* to pin objects for native access?
>
The general practice so far has been not to expose
interpreter-allocated objects to native code at all, but to give
native code either a raw malloc'd handle (a-la
https://bitbucket.org/pypy/pypy/src/29df4aac463be1c3508b47754014844edb765bfa/pypy/module/cpyext/pyobject.py?at=default&fileviewer=file-view-default#pyobject.py-52
) or require the native side to do the allocation (eg
https://bitbucket.org/pypy/pypy/src/29df4aac463be1c3508b47754014844edb765bfa/pypy/module/_cffi_backend/cbuffer.py?at=default&fileviewer=file-view-default
). The aim was to significantly reduce the amount of memory that
needed pinning.
To the extent that there is an interface, it's the rpython.rlib.rgc
module. A fair bit of lltypesystem code makes use of the assumption
that no other code is touching interpreter objects and the
implementation detail that if no heap memory is allocated in some
dynamic extent, the GC cannot run and move objects. Seems a bit
disquieting.
TAN: you might have better primitives for dealing with strings in your
target, in which case avoiding much of the lltypesystem might be a
better deal for you.
> (2) From the perspective of the RPython developer (those who develop the
> translation from RTyped CFGs to C source code, assembly code and machine
> code), how does the C backend actually enforce the no-GC policy between
> "from here, no GC operations can happen" and "end of 'no GC' section"?
> As I observed, keepalive_until_here essentially generates a no-op inline
> assembly that "uses" the variable so that the C compiler keeps that
> variable alive. But what is preventing GC from happening?
>
There are no instructions that may allocate within that dynamic
section. It's not machine verified at all. It could be.
>
> Some background: We are building a new backend for RPython on the Mu
> micro virtual machine (https://gitlab.anu.edu.au/mu/mu-client-pypy).
> This VM has built-in GC and exception handling, so they don't need to be
> injected after RTyper. But the micro VM also keeps the representation
> of object references opaque. The only way to "cast" references to
> addresses is using the "object pinning" operation which returns its
> address. The idiom in Mu when passing a buffer to native programs is
> that you "pin" the object, give the pointer to the native functions
> (such as `memcpy
Re: [pypy-dev] PYPY:why withprebuiltint option is off default? The speed is slower after I open it.
On 14 December 2016 at 15:19, 张耀 wrote: > Hi, every experts: >Recently I'm trying to move my old project from python to pypy. In the > past, I have do some optimizations for python interpreter. One Important > thing is modify python default integer object cache range(default is from -5 > to 257), my config is -100 to 1024 * 1024, this range covers most of the > my app logics. The benchmark shows that it saves a lot of memory , and > speedup the program(mainly benefits from avoiding large amount of creating > intobject and releasing intobject) . > > BUT, in pypy, I found the withprebuiltint option is set off default. While > I tried open this option and add translateoption : --prebuiltintfrom -5 > --prebuiltintfrom 1048576, the benchmark shows that it saves memory(OK, it's > expected), BUT the speed is SLOWER. > Allocating a new object in PyPy is faster to start with, especially with the default GC, and because of the GC releasing short-lived intobject costs nothing. Additionally, there are lots of places where ints do not need an object. PyPy's specialised dictionary for laying out class instances (mapdict) leaves only a platform integer in many cases. Similarly for lists, lists of just integers have a special compact representation in a default PyPy build. Hot loops also get compiled, and any integers that don't escape the loop are unboxed and need to object to represent them. Prebuilt ints also have another cost that works against the layouts and compiled code above: everywhere an operation could produce a prebuilt intobject, an extra conditional check has to be performed. Having prebuilt integers turned off removes this condition from warm code. > I check the pypy sourcecode, but I have not find out the problem so far. > So, my doubt is: why withprebuiltint option is off default? Maybe the > performance is a problem as known? > Yes. Prebuilt ints are only really an optimisation on CPython, PyPy has better optimisations that work well with GC. -- William Leslie Notice: Likely much of this email is, by the nature of copyright, covered under copyright law. You absolutely MAY reproduce any part of it in accordance with the copyright law of the nation you are reading this in. Any attempt to DENY YOU THOSE RIGHTS would be illegal without prior contractual agreement. ___ pypy-dev mailing list pypy-dev@python.org https://mail.python.org/mailman/listinfo/pypy-dev
Re: [pypy-dev] RFC: draft idea for making for loops automatically close iterators
A quick definition: A resource is something, typically represented as a program object, whose timely disposal is relevant for the correct operation of the program. Typically, it can't be closed too late (in the case of files, the process may end before all writes are flushed), but it often can't be closed too soon. This has a few obvious corallaries: * If the correct functioning of the program relies on prompt disposal, then something must be responsible for that disposal. If you place it in a container and hand that container off to a library without linearity or borrow checking, you're saying you have no interest in its disposal. * The property of "being a resource" is transitive over ownership (Hopwood's Corallary?): if X requires explicit disposal, and that disposal is managed by Y, then Y is also a resource (proof of this is easy via contradiction). Typically, there is no special magic for Y; the documentation for Y must either describe how to dispose of it, or whatever code constructed Y must also maintain control over the lifetime of X. * Are files/sockets open for reading a resource? They typically aren't, especially on GNU and BSD derived systems. Some programs will never run out of them because they will never open enough. At the same time, given that "the correct operation of the program" is specific to the actual program and its usage, the answer isn't typically clear. The first point we should notice is that explicit lifetime management is /required/ - it's a direct consequence of the definition of resource! On 22 October 2016 at 01:20, Alex S. wrote: > That’s a good point, as it means there’s probably no safe & portable way to > ensure that kind of stuff. «Trying to collect» something doesn’t really fall > short of an actual collection, I believe (finding referers is hard). > But I believe iterclose() defined appropriately on derived iterators would > solve that?.. That now places the onus on every user of iterators to ensure that their iterators are either consumed or closed; but many of the iterators we create that are resources are then handed to library functions which just can't be relied upon to maintain resource transitivity. The second point, then, is that requiring everyone to rewrite any code they have that makes or consumes generators or iterables is probably not tractible. Even if everyone decided that would be better than fixing their use of files, you'd still stumble across library code that didn't make the effort. We might have been able to do something like this if we'd had something like (dynamic-wind) when generators first arrived in python, but it's probably beyond us now without a good helping of SA and/or ugly magic. So it really is easier to rewrite file usage than it is to rewrite iterator usage, especially if we can only detect and fix a handful of obvious cases in the runtime. -- William Leslie Notice: Likely much of this email is, by the nature of copyright, covered under copyright law. You absolutely MAY reproduce any part of it in accordance with the copyright law of the nation you are reading this in. Any attempt to DENY YOU THOSE RIGHTS would be illegal without prior contractual agreement. ___ pypy-dev mailing list pypy-dev@python.org https://mail.python.org/mailman/listinfo/pypy-dev
Re: [pypy-dev] RFC: draft idea for making for loops automatically close iterators
*shrug* The real solution to relying on implementation-defined behaviour isn't to emulate that behavour through even more pervasive user-visible (and user-breakable) generator madness, but to use static analysis tools as part of the development process that can detect and report common violations of linearity (especially *local* violations, which seem to be very common and easy to detect). Do any of the existing tools do such checks? On 22 October 2016 at 01:13, hubo wrote: > Well I'm really shocked to find out what I thought was a "automatic close" > is really the ref-couting GC of CPython, means that a lot of my code breaks > in PyPy... > It really becomes a big problem after iterators heavily used in Python > nowadays. Some builtin functions like zip, map, filter return iterators in > Python 3 instead of lists in Python 2, means invisible bugs for code ported > from Python 2, like zip(my_generator(), my_other_generator()) may leave the > iterators open if exited from a for loop. Even in Python 2, functions in > itertools may create these bugs. > In CPython, this kind of code will work because of the ref-counting GC, so > it is not obvious in CPython, but they break in PyPy. > > I'm wondering since a ref-counting GC implemention is not possible for PyPy, > is it possible to hack on the for loop to make it "try to" collect the > generator? That may really save a lot of lives. If the generator is still > referenced after the for loop, it may be the programmer's fault for not > calling close(), but loop through a returned value is something different - > sometimes you even do not know if it is a generator. > > 2016-10-21 > > hubo > > > 发件人：Armin Rigo > 发送时间：2016-10-18 16:01 > 主题：Re: [pypy-dev] RFC: draft idea for making for loops automatically close > iterators > 收件人："Nathaniel Smith" > 抄送："PyPy Developer Mailing List" > > Hi, > > On 17 October 2016 at 10:08, Nathaniel Smith wrote: >> thought I'd send around a draft to see what you think. (E.g., would >> this be something that makes your life easier?) > > As a general rule, PyPy's GC behavior is similar to CPython's if we > tweak the program to start a chain of references at a self-referential > object. So for example, consider that the outermost loop of a program > takes the objects like the async generators, and stores them inside > such an object: > > class A: > def __init__(self, ref): > self.ref = ref > self.myself = self > > and then immediately forget that A instance. Then both this A > instance and everything it refers to is kept alive until the next > cyclic GC occurs. PyPy just always exhibits that behavior instead of > only when you start with reference cycles. > > So the real issue should not be "how to so something that will make > PyPy happy", or not only---it should be "how to do something that will > make CPython happy even in case of reference cycles". If you don't, > then arguably CPython is slightly broken. > > Yes, anything that can reduce file descriptor leaks in Python sounds good to > me. > > > A bientôt, > > Armin. > ___ > pypy-dev mailing list > pypy-dev@python.org > https://mail.python.org/mailman/listinfo/pypy-dev > > > ___ > pypy-dev mailing list > pypy-dev@python.org > https://mail.python.org/mailman/listinfo/pypy-dev > -- William Leslie Notice: Likely much of this email is, by the nature of copyright, covered under copyright law. You absolutely MAY reproduce any part of it in accordance with the copyright law of the nation you are reading this in. Any attempt to DENY YOU THOSE RIGHTS would be illegal without prior contractual agreement. ___ pypy-dev mailing list pypy-dev@python.org https://mail.python.org/mailman/listinfo/pypy-dev
Re: [pypy-dev] RFC: draft idea for making for loops automatically close iterators
Have you considered Custodians, a-la racket? I suspect that adding resources to and finalising custodians requires less defensiveness than marking all iterables as resources, but I've yet to see someone implement them in python. https://docs.racket-lang.org/reference/custodians.html -- William Leslie Notice: Likely much of this email is, by the nature of copyright, covered under copyright law. You absolutely MAY reproduce any part of it in accordance with the copyright law of the nation you are reading this in. Any attempt to DENY YOU THOSE RIGHTS would be illegal without prior contractual agreement. ___ pypy-dev mailing list pypy-dev@python.org https://mail.python.org/mailman/listinfo/pypy-dev
Re: [pypy-dev] Build Pypy with different Interpreters
On 8 September 2016 at 22:38, Jan Brohl wrote: > Sorry, for the typo - I was asking if it is possible to build pypy *with* > different interpreters instead of just cpython and pypy. > > (using eg "ipy64" instead of "pypy" or "python" in the translation step > described at > http://doc.pypy.org/en/latest/build.html#run-the-translation ) > > I didn't say anything about Stackless; maybe I should. Stackless is CPython with a few extra features. Translation probably works on it, but given that Stackless has a GIL, and that rpython doesn't make use of any Stackless APIs, it's probably just the same as translating with CPython. -- William Leslie Notice: Likely much of this email is, by the nature of copyright, covered under copyright law. You absolutely MAY reproduce any part of it in accordance with the copyright law of the nation you are reading this in. Any attempt to DENY YOU THOSE RIGHTS would be illegal without prior contractual agreement. ___ pypy-dev mailing list pypy-dev@python.org https://mail.python.org/mailman/listinfo/pypy-dev
Re: [pypy-dev] Build Pypy with different Interpreters
On 8 September 2016 at 22:38, Jan Brohl wrote: > Sorry, for the typo - I was asking if it is possible to build pypy *with* > different interpreters instead of just cpython and pypy. > That makes sense, it fits with your other question (: The translator operates on functions that have been byte-compiled by the host compiler. IronPython and Jython compile directly to their corresponding virtual machine, so there's no cpython or pypy bytecode available. > (using eg "ipy64" instead of "pypy" or "python" in the translation step > described at > http://doc.pypy.org/en/latest/build.html#run-the-translation ) > > -- William Leslie Notice: Likely much of this email is, by the nature of copyright, covered under copyright law. You absolutely MAY reproduce any part of it in accordance with the copyright law of the nation you are reading this in. Any attempt to DENY YOU THOSE RIGHTS would be illegal without prior contractual agreement. ___ pypy-dev mailing list pypy-dev@python.org https://mail.python.org/mailman/listinfo/pypy-dev
Re: [pypy-dev] Build Pypy with different Interpreters
On 8 September 2016 at 19:40, Jan Brohl wrote: > Is it possible to build different interpreters like Stackless, IronPython > or Jython? > That was actually the original motivation for creating pypy - maintaining all those different python implementations was a lot of unnecessary work. Stackless support is enabled by default. Support for translating to CLI and the JVM was eventually dropped for lack of interest. If someone wanted to re-add that support, they could learn from the mistakes that the previous implementation used. http://doc.pypy.org/en/latest/stackless.html > > If not - why? > > If yes - is it (in theory) possible to gain a speedup on those without > GIL? (Is there multithreading at all the in translation process?) > Translation can't be done concurrently at the moment. I probably should have expanded upon this in my previous email, and maybe I will; there are a number of global structures, registries, and work lists that would need to be refactored before the translation work could be distributed. If that's the route the pypy team go, we will consider it after pypy itself supports parallelism. There's another route, which is to support separate compilation, and then to hand off the translation of built-in modules to different executors. This is itself quite a bit of work due to some inherent properties of rpython. -- William Leslie Notice: Likely much of this email is, by the nature of copyright, covered under copyright law. You absolutely MAY reproduce any part of it in accordance with the copyright law of the nation you are reading this in. Any attempt to DENY YOU THOSE RIGHTS would be illegal without prior contractual agreement. ___ pypy-dev mailing list pypy-dev@python.org https://mail.python.org/mailman/listinfo/pypy-dev
Re: [pypy-dev] Any known tricks or effort made to make the PyPy source code build faster?
On 30/08/2016 9:53 am, "Wang, Peter Xihong" wrote: > > Hi, > > > > By default, it appears most of the time during the build/compile process, only 1 single CPU core is busy, signaling missing of parallel compiling. Is there any best known practice to make it faster? > > Not at the moment. The STM work that Armin is doing may give us an interpreter that can make better use of extra cores during translation, but it is still experimental. > > Thanks, > > > > Peter > > > > > > > ___ > pypy-dev mailing list > pypy-dev@python.org > https://mail.python.org/mailman/listinfo/pypy-dev > ___ pypy-dev mailing list pypy-dev@python.org https://mail.python.org/mailman/listinfo/pypy-dev
Re: [pypy-dev] Django DateField default value and migrations fail
On 17 August 2016 at 19:53, Yury V. Zaytsev wrote: > On Wed, 17 Aug 2016, Sergey Kurdakov wrote: > >> class TestClass(models.Model): >> >> start_date = models.DateField( >> verbose_name=u'start date', >> default=today, >> ) >> >> so I just wrap required function call into function. > > > You can, of course, use a lambda, i.e. `default=lambda: date.today()` which > is to the same effect, but a bit shorter. It doesn't work with a lambda; the serialiser writes code that looks up the function by name. -- William Leslie Notice: Likely much of this email is, by the nature of copyright, covered under copyright law. You absolutely MAY reproduce any part of it in accordance with the copyright law of the nation you are reading this in. Any attempt to DENY YOU THOSE RIGHTS would be illegal without prior contractual agreement. ___ pypy-dev mailing list pypy-dev@python.org https://mail.python.org/mailman/listinfo/pypy-dev
Re: [pypy-dev] Django DateField default value and migrations fail
I took a look at the Django source. There's code for serialising methods in Django that checks `if isinstance(value, (types.FunctionType, types.BuiltinFunctionType))` which succeeds on cpython because datetime.date.today is a BuiltinFunctionType, wheras it's a types.MethodType on pypy. Maybe that check could be expanded to include MethodType. Here's the code at master: https://github.com/django/django/blob/3b383085fb89a48e756383e7cd5d3bd867353ba1/django/db/migrations/serializer.py#L379 On 17 August 2016 at 17:27, Sergey Kurdakov wrote: > Hi William, > > here is my traceback in console > > Traceback (most recent call last): > File "manage.py", line 10, in > execute_from_command_line(sys.argv) > File > "/usr/local/lib/pypy2.7/dist-packages/django/core/management/__init__.py", > line 353, in execute_from_command_line > utility.execute() > File > "/usr/local/lib/pypy2.7/dist-packages/django/core/management/__init__.py", > line 345, in execute > self.fetch_command(subcommand).run_from_argv(self.argv) > File > "/usr/local/lib/pypy2.7/dist-packages/django/core/management/base.py", line > 348, in run_from_argv > self.execute(*args, **cmd_options) > File > "/usr/local/lib/pypy2.7/dist-packages/django/core/management/base.py", line > 399, in execute > output = self.handle(*args, **options) > File > "/usr/local/lib/pypy2.7/dist-packages/django/core/management/commands/makemigrations.py", > line 150, in handle > self.write_migration_files(changes) > File > "/usr/local/lib/pypy2.7/dist-packages/django/core/management/commands/makemigrations.py", > line 178, in write_migration_files > migration_string = writer.as_string() > File > "/usr/local/lib/pypy2.7/dist-packages/django/db/migrations/writer.py", line > 167, in as_string > operation_string, operation_imports = > OperationWriter(operation).serialize() > File > "/usr/local/lib/pypy2.7/dist-packages/django/db/migrations/writer.py", line > 124, in serialize > _write(arg_name, arg_value) > File > "/usr/local/lib/pypy2.7/dist-packages/django/db/migrations/writer.py", line > 76, in _write > arg_string, arg_imports = MigrationWriter.serialize(item) > File > "/usr/local/lib/pypy2.7/dist-packages/django/db/migrations/writer.py", line > 357, in serialize > item_string, item_imports = cls.serialize(item) > File > "/usr/local/lib/pypy2.7/dist-packages/django/db/migrations/writer.py", line > 433, in serialize > return cls.serialize_deconstructed(path, args, kwargs) > File > "/usr/local/lib/pypy2.7/dist-packages/django/db/migrations/writer.py", line > 318, in serialize_deconstructed > arg_string, arg_imports = cls.serialize(arg) > File > "/usr/local/lib/pypy2.7/dist-packages/django/db/migrations/writer.py", line > 540, in serialize > "topics/migrations/#migration-serializing" % (value, get_docs_version()) > ValueError: Cannot serialize: 'datetime.date'>> > > > > Regards > Sergey > > > On Wed, Aug 17, 2016 at 10:21 AM, William ML Leslie > wrote: >> >> On 17 August 2016 at 16:50, Sergey Kurdakov >> wrote: >> > >> > still causes error of the kind: >> > >> > topics/migrations/#migration-serializing" % (value, get_docs_version()) >> > ValueError: Cannot serialize: > > 'datetime.date' >> >> Do you get a full traceback? That would be much more useful for >> tracking down the problem. >> >> -- >> William Leslie >> >> Notice: >> Likely much of this email is, by the nature of copyright, covered >> under copyright law. You absolutely MAY reproduce any part of it in >> accordance with the copyright law of the nation you are reading this >> in. Any attempt to DENY YOU THOSE RIGHTS would be illegal without >> prior contractual agreement. > > -- William Leslie Notice: Likely much of this email is, by the nature of copyright, covered under copyright law. You absolutely MAY reproduce any part of it in accordance with the copyright law of the nation you are reading this in. Any attempt to DENY YOU THOSE RIGHTS would be illegal without prior contractual agreement. ___ pypy-dev mailing list pypy-dev@python.org https://mail.python.org/mailman/listinfo/pypy-dev
Re: [pypy-dev] Django DateField default value and migrations fail
Darn reply-to not being set. Sorry Sergey. On 17 August 2016 at 17:21, William ML Leslie wrote: > On 17 August 2016 at 16:50, Sergey Kurdakov wrote: >> >> still causes error of the kind: >> >> topics/migrations/#migration-serializing" % (value, get_docs_version()) >> ValueError: Cannot serialize: > 'datetime.date' > > Do you get a full traceback? That would be much more useful for > tracking down the problem. > > -- > William Leslie > > Notice: > Likely much of this email is, by the nature of copyright, covered > under copyright law. You absolutely MAY reproduce any part of it in > accordance with the copyright law of the nation you are reading this > in. Any attempt to DENY YOU THOSE RIGHTS would be illegal without > prior contractual agreement. -- William Leslie Notice: Likely much of this email is, by the nature of copyright, covered under copyright law. You absolutely MAY reproduce any part of it in accordance with the copyright law of the nation you are reading this in. Any attempt to DENY YOU THOSE RIGHTS would be illegal without prior contractual agreement. ___ pypy-dev mailing list pypy-dev@python.org https://mail.python.org/mailman/listinfo/pypy-dev
Re: [pypy-dev] pickle numpy array from pypy to cpython?
On 24 June 2016 at 12:14, Eli Stevens (Gmail) wrote: > I'm trying to construct some data that includes numpy arrays in pypy, > pickle it, then unpickle it in cpython (to use some > non-pypy-compatible libs). > > However, the actual class of the pickled array is _numpypy.multiarray, > which cpython doesn't have. > > Any suggestions? Have you considered the tofile method and fromfile function? http://docs.scipy.org/doc/numpy/reference/generated/numpy.ndarray.tofile.html#numpy.ndarray.tofile http://docs.scipy.org/doc/numpy/reference/generated/numpy.fromfile.html#numpy.fromfile -- William Leslie Notice: Likely much of this email is, by the nature of copyright, covered under copyright law. You absolutely MAY reproduce any part of it in accordance with the copyright law of the nation you are reading this in. Any attempt to DENY YOU THOSE RIGHTS would be illegal without prior contractual agreement. ___ pypy-dev mailing list pypy-dev@python.org https://mail.python.org/mailman/listinfo/pypy-dev
Re: [pypy-dev] fix libpypy-c.so needing certein versions
On 22 June 2016 at 18:19, Armin Rigo wrote: > http://pypy.org/download.html#linux-binaries-and-common-distributions The portable builds linked from that page work really well if you're on a gnu/linux (thanks, squeaky!) - try those out first. -- William Leslie Notice: Likely much of this email is, by the nature of copyright, covered under copyright law. You absolutely MAY reproduce any part of it in accordance with the copyright law of the nation you are reading this in. Any attempt to DENY YOU THOSE RIGHTS would be illegal without prior contractual agreement. ___ pypy-dev mailing list pypy-dev@python.org https://mail.python.org/mailman/listinfo/pypy-dev
Re: [pypy-dev] Question about "__extend__" hacks (in pypyjit module)
On 9 February 2016 at 10:24, Maciej Fijalkowski wrote: > __extend__ hacks add extra methods to classes See rpython.tool.pairtype and rpython.tool.test.test_pairtype for more information on this pattern. -- William Leslie Notice: Likely much of this email is, by the nature of copyright, covered under copyright law. You absolutely MAY reproduce any part of it in accordance with the copyright law of the nation you are reading this in. Any attempt to DENY YOU THOSE RIGHTS would be illegal without prior contractual agreement. ___ pypy-dev mailing list pypy-dev@python.org https://mail.python.org/mailman/listinfo/pypy-dev
Re: [pypy-dev] PyPy doesn't implement the most important module!
On 22 October 2015 at 19:14, Maciej Fijalkowski wrote: > pypy will not automatically pick up packages from site-packages of > cpython. you need to install them separately, preferably in a > virtualenv. It comes with cpython, as part of the 'frozen' module system. https://docs.python.org/2/library/imp.html#imp.PY_FROZEN -- William Leslie Notice: Likely much of this email is, by the nature of copyright, covered under copyright law. You absolutely MAY reproduce any part of it in accordance with the copyright law of the nation you are reading this in. Any attempt to DENY YOU THOSE RIGHTS would be illegal without prior contractual agreement. ___ pypy-dev mailing list pypy-dev@python.org https://mail.python.org/mailman/listinfo/pypy-dev
Re: [pypy-dev] FIY: Basic Block Versioning JIT
On 29 September 2015 at 21:18, Carl Friedrich Bolz wrote: > One thing that's very cool in BBV that I'm not aware of any other JIT > doing is to have type-specialized entry points for uninlinable methods. > Ie if a method is not inlined then the caller calls a type specialized > entry point with all the local type knowledge taken into account. In > addition, the result type is propagated back out from the uninlined > method to the calling function. I couldn't help but think when I read this post last week that it's like a dynamic equivalent of CFA2; which is exciting because CFA2 is such a dramatic improvement over k-CFA for intraprocedural analysis. It's good to see how well such a simple principle about specialising type information can produce such good results. -- William Leslie Notice: Likely much of this email is, by the nature of copyright, covered under copyright law. You absolutely MAY reproduce any part of it in accordance with the copyright law of the nation you are reading this in. Any attempt to DENY YOU THOSE RIGHTS would be illegal without prior contractual agreement. ___ pypy-dev mailing list pypy-dev@python.org https://mail.python.org/mailman/listinfo/pypy-dev
Re: [pypy-dev] type inference error
On 28 August 2015 at 13:34, William ML Leslie wrote: > def entry(x, y): > m = mtype() > return mmm(m) > er I meant: def entry(): m = mtype() return mmm(m) t = Translation(entry, []) -- William Leslie Notice: Likely much of this email is, by the nature of copyright, covered under copyright law. You absolutely MAY reproduce any part of it in accordance with the copyright law of the nation you are reading this in. Any attempt to DENY YOU THOSE RIGHTS would be illegal without prior contractual agreement. ___ pypy-dev mailing list pypy-dev@python.org https://mail.python.org/mailman/listinfo/pypy-dev
Re: [pypy-dev] type inference error
On 28 August 2015 at 04:23, Andrey Ayupov wrote: > Hello, > > Hopefully a very simple question. I am trying out RPython toolchain. pypy > 2.6, python 2.7. I am using the translator shell. > > Here's my simple example: > > python bin/translatorshell.py > > class mtype(): > def __init__(self): > self.x = 0; > > def mmm(x): > z = x.x + 1 > return z > > t = Translation(mmm, [mtype]) > > t.annotate() > > and it gives me the following. if you put mtype class and mmm function > definitions in a separate module, then you can see it complains about this > line: z = x.x + 1. Why can't it infer type for all arguments and the > result here? I know i am missing something basic. It never saw the creation of an mytpe, so it doesn't know that mtypes have an x attribute, let alone know what type it has. Try a bigger entry point: def entry(x, y): m = mtype() return mmm(m) -- William Leslie Notice: Likely much of this email is, by the nature of copyright, covered under copyright law. You absolutely MAY reproduce any part of it in accordance with the copyright law of the nation you are reading this in. Any attempt to DENY YOU THOSE RIGHTS would be illegal without prior contractual agreement. ___ pypy-dev mailing list pypy-dev@python.org https://mail.python.org/mailman/listinfo/pypy-dev
Re: [pypy-dev] is the handling of python signals a language implementation detail?
On 8 August 2015 at 19:32, Laura Creighton wrote: > I am reading 18.8.1.1 and 18.8.1.2 in > https://docs.python.org/3/library/signal.html which clearly is written > from the CPython point of view. > > A Python signal handler does not get executed inside the low-level > (C) signal handler. Instead, the low-level signal handler sets a flag > which tells the virtual machine to execute the corresponding Python > signal handler at a later point(for example at the next bytecode > instruction) ... Python signal handlers are always executed in the > main Python thread, even if the signal was received in another thread ... > > But is this part of the language definition? Or an implementation detail? Since Python does not define a concept of signal-safe, I guess in practice the handler must be observed to run at a later point. The thread that a handler runs in can be semantically significant; I think you should be able to rely on the behaviour of the signal module in this regard. Different APIs and semantic-preserving optimisations aside. -- William Leslie Notice: Likely much of this email is, by the nature of copyright, covered under copyright law. You absolutely MAY reproduce any part of it in accordance with the copyright law of the nation you are reading this in. Any attempt to DENY YOU THOSE RIGHTS would be illegal without prior contractual agreement. ___ pypy-dev mailing list pypy-dev@python.org https://mail.python.org/mailman/listinfo/pypy-dev
Re: [pypy-dev] How to use interperter level code from a module for app level code in another module
On 9 June 2015 at 19:39, Amaury Forgeot d'Arc wrote: > 2015-06-09 11:34 GMT+02:00 Yicong Huang : > >> No, I' am trying to call interp code from app code, and found the module >> could not be imported: >> >> import select => ImportError: No module named select >> import pypy.module.select =>ImportError: No module named pypy >> > > "import select" is the one which should work at applevel. > But for this, the PyPy object space must have the option "spaceconfig = > dict(usemodules=['select'])", or be translated with --withmod-select. > In which context are you running the code above? > Are you using a pypy sandbox Huang? If so, many built-in modules are disabled by default, you have to enable them during translation. -- William Leslie Notice: Likely much of this email is, by the nature of copyright, covered under copyright law. You absolutely MAY reproduce any part of it in accordance with the copyright law of the nation you are reading this in. Any attempt to DENY YOU THOSE RIGHTS would be illegal without prior contractual agreement. ___ pypy-dev mailing list pypy-dev@python.org https://mail.python.org/mailman/listinfo/pypy-dev
Re: [pypy-dev] How to use interperter level code from a module for app level code in another module
On 8 June 2015 at 21:22, Yicong Huang wrote:
> By reading modules in pypy/module, finally we found out the workaround
> method.
>
> For most of modules in 'pypy/module', their interp code could not be
> imported or used by other module's app code.
> But there is special module '__pypy__', it exposed several interp
> functions, and those code are used by other module's app code.
>
There's nothing special about __pypy__ for exposing interp-level code, the
modules in pypy/modules usually include interp level code.
For example, the select module exposes the interp-level function
select_interp.select. Here's how it does so:
https://bitbucket.org/pypy/pypy/src/b0c01840baea472c834635de627bb596c05d3bd5/pypy/module/select/__init__.py?at=default
The interpreter gateway has more options for exposing interp-level
functions and types to app-level.
The workaround solution is simple: add one more interface in __pypy__
> module's __init__.py
>
You should make a new module rather than adding to one existing one.
> PYC_MAGIC = get_pyc_magic(self.space)
self.extra_interpdef('PYC_MAGIC', 'space.wrap(%d)' % PYC_MAGIC)
> Are the above code do the magic thing to expose interp code?
>
They are exposing an integer value called PYC_MAGIC, which is the value
for determining what python the .pyc file was compiled for.
This is about exposing an integer rather than a function, it's probably
not relevant to you.
--
William Leslie
Notice:
Likely much of this email is, by the nature of copyright, covered under
copyright law. You absolutely MAY reproduce any part of it in accordance
with the copyright law of the nation you are reading this in. Any attempt
to DENY YOU THOSE RIGHTS would be illegal without prior contractual
agreement.
___
pypy-dev mailing list
pypy-dev@python.org
https://mail.python.org/mailman/listinfo/pypy-dev
Re: [pypy-dev] mix-and-match approach to implementation decisions
I guess it's reasonable to ask how true this is of rpython. What options are there, and how do they exclude / depend on one another? In the l*o*p problem, it now ignores all p but one, and rpython doesn't concern itself with l. Maybe it's the gc * o problem now? -- William Leslie Notice: Likely much of this email is, by the nature of copyright, covered under copyright law. You absolutely MAY reproduce any part of it in accordance with the copyright law of the nation you are reading this in. Any attempt to DENY YOU THOSE RIGHTS would be illegal without prior contractual agreement. ___ pypy-dev mailing list pypy-dev@python.org https://mail.python.org/mailman/listinfo/pypy-dev
Re: [pypy-dev] mix-and-match approach to implementation decisions
On 14 February 2015 at 19:48, anatoly techtonik wrote: > Hi, > > Reading http://rpython.readthedocs.org/en/latest/ to see if I can > understand the text without a formal CS education. What does > "mix-and-match approach to implementation decisions" mean? > It means that making one decision - like which GC to use - does not have a great impact on other decisions, like whether you can have a JIT. Pypy itself is a great representation of this philosophy, where there are a large number of 'strategies' that optimise built-in objects for different use cases, which are enabled or disabled at translation time. -- William Leslie Notice: Likely much of this email is, by the nature of copyright, covered under copyright law. You absolutely MAY reproduce any part of it in accordance with the copyright law of the nation you are reading this in. Any attempt to DENY YOU THOSE RIGHTS would be illegal without prior contractual agreement. ___ pypy-dev mailing list pypy-dev@python.org https://mail.python.org/mailman/listinfo/pypy-dev
Re: [pypy-dev] PyPy for analysis?
Ugh, thanks Gmail (: On 8 September 2014 01:33, William ML Leslie wrote: > On 7 September 2014 21:42, Scott West > wrote: > >> Hello all, >> >> I was looking recently into trying to do some simple static analysis of >> Python programs (to experiment with what is possible), and came across >> PyPy. Reading some of the documentation it seems that PyPy forms a control >> flow graph and does some abstract interpretation, which seems to indicate >> that it could be used for other forms of static analysis! >> > > Static analysis of python programs is not difficult, weather you use ast > or code objects. The flow object space really only works on rpython > though, using it to analyse python code generally is not sensible. > > I do think pypy is a great platform for associating axioms with builtin > functions and bytecodes (and sometimes other weird stuff like exception > handling, depending on your analysis) which you would otherwise have to > write by hand. If you want to do that, target the llgraph stage of the > compiler, as there is nothing 'outside' it that you need to rely upon. You > can possibly even generate the app-level analyser from the interpreter, but > in order to do this I had to rewrite the eval loop - I could never figure > out how to extract semantics from the frame object. > > I did at one point have something like a points-to analysis for > interp-level code working (a region polymorphic effects analysis), but it > has bitrotten beyond repair. I would still say pypy is a great place to > start, however. > > >> >> I had a couple questions: >> 1) Would PyPy (seem) to be a good place to start for a general analysis >> framework? If so, would it be of general interest? >> 2) If one doesn't care about code generation, can the build time be >> reduced? I tried just using pyinteractive.py and that _seemed_ to work, >> though even that compiles a few things to start. >> > > By about 40% if you target llgraph by my last count. You could avoid > building the jit, since it should not have any semantic impact. > > > >> 3) What is a good way to get into modifying the internals (adding >> analyses, skipping code gen.)? I have read the chapter of the Open Source >> Architecture book, and some of the documentation pages. >> >> >> > 0) Write lots of tests. > > The existing tests make for good starting points. > > 1) > Start with a very small interpreter, not the full pypy interpreter. IIRC > there was a language called TL which was for testing the JIT, use it as the > target until everything works. > > 2) Dumping the llgraph to disk somewhere - perhaps into a logic or > relational database - will make little experiments easier. > > > > >> >> >> >> I w >> >> >> ould be most grateful if anyone could provide any comments on these >> issues, or pointers to other similar works. >> > > Unfortunately I have no idea what state my work was in when I left it, I > don't even have it on my current box; I would like to get back to it at > some point but Armin suggested I look at better low level languages to > target and I'm still working in that space. > > -- > William Leslie > > Notice: > Likely much of this email is, by the nature of copyright, covered under > copyright law. You absolutely MAY reproduce any part of it in accordance > with the copyright law of the nation you are reading this in. Any attempt > to DENY YOU THOSE RIGHTS would be illegal without prior contractual > agreement. > -- William Leslie Notice: Likely much of this email is, by the nature of copyright, covered under copyright law. You absolutely MAY reproduce any part of it in accordance with the copyright law of the nation you are reading this in. Any attempt to DENY YOU THOSE RIGHTS would be illegal without prior contractual agreement. ___ pypy-dev mailing list pypy-dev@python.org https://mail.python.org/mailman/listinfo/pypy-dev
[pypy-dev] Fwd: Access the name of variable that is being assigned
Ack, resend because pypy-dev does not set reply-to ... On 16 July 2014 03:55, anatoly techtonik wrote: > On Tue, Jul 15, 2014 at 12:50 PM, Yichao Yu wrote: >> On Tue, Jul 15, 2014 at 5:05 PM, anatoly techtonik >> wrote: >> I guess it would be better if you can describe what you really want to do. > > I described. Or you need use case or user story? I think I want to link > object instances to variable names without to print those names > (if possible) in __repr__ and debug messages to save time on > troubleshooting. The most common practice is to pass some kind of debugging info to the constructor, otherwise, setting debug info on the instance is ok, too. my_object = SomeClass() my_object.debug_info = 'This is the typical case' You can also use the inspect module to grab the source line of the caller: http://codepad.org/YvStcEMv -- William Leslie Notice: Likely much of this email is, by the nature of copyright, covered under copyright law. You absolutely MAY reproduce any part of it in accordance with the copyright law of the nation you are reading this in. Any attempt to DENY YOU THOSE RIGHTS would be illegal without prior contractual agreement. ___ pypy-dev mailing list pypy-dev@python.org https://mail.python.org/mailman/listinfo/pypy-dev
Re: [pypy-dev] More strategies
On 8 November 2013 09:18, Laurence Tratt wrote: > Looking up floats and longs is > now much faster; for many other types (e.g. strings or user objects whose > class does not override __eq__) these return immediately. I wonder a bit if it is worth introducing additional fast paths for clearly nonsense (eg, string) cases, when it's the sensible case (user-provided types that provide __eq__) that we should be optimising for. Did you menchbark the code without such cases? -- William Leslie Notice: Likely much of this email is, by the nature of copyright, covered under copyright law. You absolutely may reproduce any part of it in accordance with the copyright law of the nation you are reading this in. Any attempt to deny you those rights would be illegal without prior contractual agreement. ___ pypy-dev mailing list pypy-dev@python.org https://mail.python.org/mailman/listinfo/pypy-dev
Re: [pypy-dev] pypy about c-extension
The blog post describes how to produce a cpython module .so for your schema. Needless to say that is pointless on pypy. On 04/11/2013 7:09 PM, "Armin Rigo" wrote: > Hi, > > On Mon, Nov 4, 2013 at 9:04 AM, KaShining wrote: > > import podpbpypy > > ImportError: No module named podpbpypy > > > > seen can't find podpbpypy.so > > There is no point repeating what you already said. I'm asking, how > did you build the .so in the first place. > > > Armin > ___ > pypy-dev mailing list > pypy-dev@python.org > https://mail.python.org/mailman/listinfo/pypy-dev > ___ pypy-dev mailing list pypy-dev@python.org https://mail.python.org/mailman/listinfo/pypy-dev
Re: [pypy-dev] The fate of gc-del
On 12 August 2013 17:38, Armin Rigo wrote: > The advantage of this approach is that it's done without RPython > changes, just by tweaks in the GC. Do you know what changes to the GC interface you expect to make, if any? -- William Leslie Notice: Likely much of this email is, by the nature of copyright, covered under copyright law. You absolutely may reproduce any part of it in accordance with the copyright law of the nation you are reading this in. Any attempt to deny you those rights would be illegal without prior contractual agreement. ___ pypy-dev mailing list pypy-dev@python.org http://mail.python.org/mailman/listinfo/pypy-dev
Re: [pypy-dev] parallel building
On 22 July 2013 16:18, Nathan Hurst wrote: > On Sun, Jul 21, 2013 at 08:25:46PM -0700, Alex Gaynor wrote: >> No, there currently isn't a way to parallelize building. > > Ok. Is it hard or just low priority? It's mostly pretty hard, although it's not the same reason through each of the stages involved - the short answer is that translation mostly involves walking over graphs, including the inter-procedural call graph, and propagating information. In one stage the graph is known mostly ahead of time, but otherwise it's being built as it goes, which means the job list probably stays pretty small. If we managed to make that stage parallel, we'd probably lose out on the fact that several of the steps work by mutating the model, so we'd either need a concurrency model that dealt with that or separate compilation to work. -- William Leslie Notice: Likely much of this email is, by the nature of copyright, covered under copyright law. You absolutely may reproduce any part of it in accordance with the copyright law of the nation you are reading this in. Any attempt to deny you those rights would be illegal without prior contractual agreement. ___ pypy-dev mailing list pypy-dev@python.org http://mail.python.org/mailman/listinfo/pypy-dev
Re: [pypy-dev] comparing strings using is does not works
On 6 June 2013 16:09, Amaury Forgeot d'Arc wrote: > Don't use "is" with immutable objects (except with the singletons: None, > True, False) Never with True or False either. It's required to work by the language definition, but it's still nonsense. -- William Leslie Notice: Likely much of this email is, by the nature of copyright, covered under copyright law. You absolutely may reproduce any part of it in accordance with the copyright law of the nation you are reading this in. Any attempt to deny you those rights would be illegal without prior contractual agreement. ___ pypy-dev mailing list pypy-dev@python.org http://mail.python.org/mailman/listinfo/pypy-dev
Re: [pypy-dev] Killing OOType? (was Re: Translating pypy on FreeBSD with CLI backend)
On 22 May 2013 16:11, James Lan wrote: > I really wish pypy is able to support jvm, cli and dalvik, so that it is > possible to write core business logic in python which runs everywhere. > > Is there any plan to implement a better ootype as well as OO backends? Nobody is working on any of the OO backends at the moment. It might be worthwhile to document what we mean by 'from scratch', too. Graphs that haven't been rtyped actually have most of the information that oo-style or untyped backends might care about, the most significant things missing probably involve dealing with constants as well as the extregistry. -- William Leslie Notice: Likely much of this email is, by the nature of copyright, covered under copyright law. You absolutely may reproduce any part of it in accordance with the copyright law of the nation you are reading this in. Any attempt to deny you those rights would be illegal without prior contractual agreement. ___ pypy-dev mailing list pypy-dev@python.org http://mail.python.org/mailman/listinfo/pypy-dev
Re: [pypy-dev] x is y <=> id(x)==id(y)
On 6 May 2013 17:03, Amaury Forgeot d'Arc wrote: > 2013/5/6 Armin Rigo >> >> On Sun, May 5, 2013 at 10:40 PM, Michael Hudson-Doyle >> wrote: >> > I want to say something about negative zeroes here >> >> Right: on floats it's not actually the usual equality, but equality of >> the bit pattern (using float2longlong). > > > Except for NaN... It's perfectly acceptable for NaN to `is` on their bit pattern. -- William Leslie Notice: Likely much of this email is, by the nature of copyright, covered under copyright law. You absolutely may reproduce any part of it in accordance with the copyright law of the nation you are reading this in. Any attempt to deny you those rights would be illegal without prior contractual agreement. ___ pypy-dev mailing list pypy-dev@python.org http://mail.python.org/mailman/listinfo/pypy-dev
Re: [pypy-dev] PyPy in getting started
Ugh, hit the wrong reply button. On 24 December 2012 19:24, William ML Leslie wrote: > On 24 December 2012 19:03, anatoly techtonik wrote: >> Hi, >> >> The PyPy description in getting started is still too scientific. I attach a >> patch with accessibility fix. =) But it is still not completely clear if >> PyPy's Python implementation is written in Python or in RPython? > > They aren't mutually exclusive; all rpython is python. > >> '''And the second is one particular implementation that is so generated – an >> implementation of the Python programming language written in Python >> itself.''' > > Maybe we could be even clearer - in my mind, implementations aren't > generated, they do have to be written (: > >> If Python implementation is written in Python, and not in RPython then there >> is the missing link between the framework and implementation, which is >> confusing. >> >> http://doc.pypy.org/en/latest/getting-started.html#what-is-pypy >> -- >> anatoly t. > > -- > William Leslie -- William Leslie ___ pypy-dev mailing list pypy-dev@python.org http://mail.python.org/mailman/listinfo/pypy-dev
Re: [pypy-dev] pypq in RPython
On 13/11/2012, Nick Fankhauser wrote: > Hi! > > I'm trying to compile one of my python applications using "translate.py". > Because the application needs access to postgreSQL, it's using pypq. The > application runs fine in python, and pypy as well. > Using the error messages from translate.py, I made a lot of changes to > my program to make it as much RPython as I can make it. > > But even after changing the code many, many times, I never get past the > following error message. It's from pypq and I'm not sure what command in > my application triggers the error, but by looking at the pypq source, I > assume it's from a pypq-"commit" command. The pypy translation toolchain has never been intended to support translation of end-user programs. If at all possible, use the pypy python interpreter. It's often faster than rpython for real-world code, and a lot easier to use, too. > My questions are: > - Is it even possible to use pypq in RPython? No, it uses ctypes, which is only available at app-level. > - If not, is there any other postgreSQL adapter that works? No, you will have to use a low-level C interface; there are examples of this in pypy/modules. (I think this answers the remaining questions.) -- William Leslie ___ pypy-dev mailing list pypy-dev@python.org http://mail.python.org/mailman/listinfo/pypy-dev
Re: [pypy-dev] Dependence Graphs in Pypy?
On 09/08/2012, Tim Henderson wrote: > Fortunately, I don't need a complete call graph. To get started I really > only need procedure level dependency graphs (with control and > data-dependencies). This graph is essentially a CFG with extra edges for > different types of data dependencies. Thanks to Alex's previous reply I am > taking a look at the CPython bytecode compiler to see if I can extract the > local CFG's it produces. Although pypy's own compiler creates a block structure and allows you to do post-order traversal of the blocks, I've found the easiest way to get the CFG of a single code object to be to use the logic found within the dis module. Here's an example of some old code that does that, somewhat stripped down and provided without tests I'm afraid because I kind of got bored of it. It's slightly more involved than dis.findlabels, part of a large piece of nonsense for deriving dataflow equations from python at app-level. https://gist.github.com/3300866 Functions OP_FN are not hard to specify but I've got them intermingled with other, much more complicated code here, they use the EffectContext to store the residual behaviour. Stack depth is completely statically determinable, and in 2.5 or later, it's determinable without looking at other code objects. I think other people have managed to do this, by the way - I have the vague recollection that Prof. Yanhong Liu at Stony Brook wrote a tool to extract datalog facts from python, although with hand-coded flow for built-ins; if you're only interested in intraprocedural analysis (often next-to-useless in python) I guess that would be fine. But I can't seem to find that, now. -- William Leslie ___ pypy-dev mailing list pypy-dev@python.org http://mail.python.org/mailman/listinfo/pypy-dev
Re: [pypy-dev] change of strategy for the py3k branch?
On 31 May 2012 04:42, Maciej Fijalkowski wrote: > On Wed, May 30, 2012 at 7:24 PM, Martijn Faassen > wrote: >> >> Hi there, >> >> Just throwing in my little bit: any change that is made that would >> make it easier to run Python 2 and Python 3 interpretors in the same >> process would interesting, as I'm still vaguely dreaming (nothing >> more) of a combined interpreter that can run both Python 2 and Python >> 3 code. >> >> Regards, >> >> Martijn > > > Hi Martijn. > > Can you describe what sort of semantics you have in mind? Would you like to > have two copies of builtin modules? How about namespaces? What about objects > being passed from one interpreter to the another? Would they magically > change or would they be "py2k dict" and "py3k dict"? If you can describe the > semantics of a proposed beast I'm willing to answer how likely it is to > happen I think we already discussed this at one point, here is what I remember getting out of it: * Any such language integration that we do encourages people to write pypy-only programs. There was a question as to whether this was a good idea. I think someone suggested it could go further than python 2/3 and allow interaction with scheme or prolog or javascript since we are already there. * There probably are arguments around semantics, but any solution is better than no solution. This is a good topic for further research imao. * It is worthwhile considering the effect it has on python 3 uptake and porting. If pypy gave people an easy way out, it could have made quite a mess. I don't think this is as significant a problem now as it has been. And of course if you don't want to do language integration, but just add eg a command line switch, you're not getting much out of it but the cost is significant, it means users have to co-ordinate the upgrades of two languages, it increases translation and testing time, etc. By the way, I like solution 1; it's a bit closer to the way pypy/lang was done. I get the cases for moving the other languages away, but python 3 is different because so much of the existing code can be re-used. -- William Leslie ___ pypy-dev mailing list pypy-dev@python.org http://mail.python.org/mailman/listinfo/pypy-dev
Re: [pypy-dev] change of strategy for the py3k branch?
On 31 May 2012 15:20, Justin Bogner wrote: > The other solution is to split the current pypy tree in two. Having a > translator and an interpreter as separate repositories makes the > translator more accessible as a tool, and projects to implement other > languages' interpreters need only depend on it. This is the prettiest > solution architecturally, but adds the burden on developers to match > translator with compatible interpreters, which may or may not end up > being a pain point. Do remember that the translator actually requires the python 2 interpreter, that is a fundamental part of the way it works. So moving the translator into a different repository now also means maintaining two python 2 interpreters. -- William Leslie ___ pypy-dev mailing list pypy-dev@python.org http://mail.python.org/mailman/listinfo/pypy-dev
Re: [pypy-dev] libgmp
On 24 February 2012 13:05, William ML Leslie wrote: > On 24 February 2012 12:56, Timothy Baldridge wrote: >> For a project I'm working on, I'd like to have support for gmp in >> pypy. I have a ctypes pypy module, but from what I understand, pypy's >> ctypes are a bit slow compared to CPython. > > On the contrary, ctypes is particularly fast on pypy for common usage. > Does your benchmark get to the point of JITting it? Sorry, forgot to reply-all. -- William Leslie ___ pypy-dev mailing list pypy-dev@python.org http://mail.python.org/mailman/listinfo/pypy-dev
Re: [pypy-dev] Global executable naming policy
On 17 February 2012 16:21, Aaron DeVore wrote: > Gentoo's CPython package gives python, python2, python3, python2.x, > and python3.x. Gentoo's PyPy package allows parallel PyPy versions by > appending a version number. The names I listed would bring PyPy up to > the level of CPython. Sure, but the pypy equivalent of cpython is pypy-c. You would probably not want to have cpython and jython with the same name by default, nor would you probably want to have pypy-c and pypy-jvm with the same name. The reason it matters is that applications may depend on particular integration features, or not, in which case whatever backend is available is fine. -- William Leslie ___ pypy-dev mailing list pypy-dev@python.org http://mail.python.org/mailman/listinfo/pypy-dev
Re: [pypy-dev] Global executable naming policy
On 17 February 2012 13:41, Aaron DeVore wrote: > There currently isn't an official naming scheme for the PyPy > executable for system installations. Of the 3 setups I've seen, the > official build uses pypy, Arch Linux uses pypy, and Gentoo Linux uses > pypy-c. I brought up the similar issue of how to > differentiate between Python 2 and 3 a while ago, but the conversation > sputtered out. On a related note, can we see situations where people only want to specify the language level, only want to specify the pypy version, or only want to specify the backend? Of course it is easy to specify /a specific installation/, and I suppose people want to be able to specify 'the default pypy' too. There is also the question of how much of this should be provided, and how much should be left to package maintainers or custom configuration; for Windows, 'pypy' and 'pypy3' are probably sufficient. -- William Leslie ___ pypy-dev mailing list pypy-dev@python.org http://mail.python.org/mailman/listinfo/pypy-dev
Re: [pypy-dev] STM
On 5 January 2012 10:30, Armin Rigo wrote: > Hi all, > > (Andrew: I'll answer your mail too, but this is independent) > > Sorry if this mail is more general than just pypy-dev, but I'll post > it here at least at first. When thinking more about STM (and also > after reading a paper that cfbolz pointed me to), I would like to > argue that the currently accepted way to add STM to languages is > slightly bogus. > > So far, the approach is to take an existing language, like Python or > C, and add a keyword like "transaction" that delimits a nested scope. > You end up with syntax like this (example in Python): > > def foo(): > before() > with transaction: > during1() > during2() > after() > > In this example, "during1(); during2()" is the content of the > transaction. But the issue with this approach is that there is no way > to structure the transactions differently. What if I want a > transaction that starts somewhere, and ends at some unrelated place? This is the way it has been described, and how most common usages will probably look. But I don't think there has ever been any suggestion that dynamic extent is the scope at which transactions *should* be implemented, any more than context managers are the the be-all-and-end-all solution for resource management. It's a convenience thing. In the case of open files, for example, "with" has lower syntactic overhead than the equivalent try/finally; but file.close() still needs to exist for more advanced usage patterns. > Following that reasoning, I no longer want to go for a PyPy version exposing a > "with transaction" context manager. Instead, it would rather be > something like a function to call at any point to "yield" control --- > or, in the idea that it's better to avoid having yields at random > unexpected points, maybe an alternative solution that looks more > promising: to start a new thread, the programmer calls a function > "start_transactional_thread(g)" where g is a generator; when g yields, > it marks the end of a transaction and the start of the next one. (The > exact yielded value should be None.) The difference with the previous > solution is that you cannot yield from a random function; your > function has to have some structure, namely be a generator, in order > to use this yield. Here you propose two solutions. I'll consider the generator one first: The requirement to be a generator is clever, because it enables you to know that some random function you call won't attempt to commit your current transaction and start a new one. Yet, this is also slightly ugly, because it has similar composition-related issues to context managers - the user is still limited to a single dynamic extent. It's not clear to me what this means in the presence of eg. stackless, either, as you say. Nevertheless, you can implement this generator model using a context manager, if you don't care about protecting the loop header and creation of the context manager (and I don't see why you would): def start_transactional_thread(g): stm.start_thread(_transactional_thread, g) def _transactional_thread(g): iterator = g() while True: with stm.transaction: try: iterator.next() except StopIteration: return The other case was a function to call to commit the transaction (and start a new one?). I would like to think that you shouldn't be able to commit a transaction that you don't know about (following capability discipline), and that concept is easier to represent as a method of some transaction object rather than a function call. This approach is strictly more general than the generator concept and the one that makes the most sense to me. It also more easily extends to distributed transaction management &c. I suspect that *even if you don't allow nesting of transactions*, this model will suit you better. Consider what happens when a JITted loop (which has its own transaction) makes a residual call. If you have the ability to pass around the transaction object you want to talk about, you can commit the existing transaction and create a new one. When you return into the loop, you can create a new transaction and store that somewhere, this transaction becoming the current transaction for the remainder of the loop. The reason I bring this up is that even though you implement transaction handling with its own special llop, you'd never sensibly model this with a generator. If you were limited to generators, you'd not be able to implement this in the llinterp, or the blackhole interpreter either without sufficient magic. -- William Leslie ___ pypy-dev mailing list pypy-dev@python.org http://mail.python.org/mailman/listinfo/pypy-dev
Re: [pypy-dev] OS
On 11/12/2011, Rinu Boney wrote: > which are the languages suited for it other than c/c++ ? Of the safe languages that I know have been used for operating systems, there have been C# (Singularity, Windows 8?), Java (JNode), and Haskell (fillet-o-fish); but there are languages that are perhaps better suited to it, such as cyclone, bitc, and ATS. Occasionally you need to be able to specify the layout of structures in memory to interact with hardware, and it is particularly useful to have safe handling of unions with custom descriminators. BitC in particular was designed explicitly for this purpose (writing a kernel with verified memory safety, among other things), although I don't know if anyone has used it for that purpose yet. Think too that when the OS starts, there isn't even a heap yet, and you don't know what space you can use to prepare one. > wouldn't it be fun and readable and 'fast development' if written in python > ? There's very little a kernel has to do, and what it must do, it must normally do with particular time bounds. Implementing your kernel in a language that relies on GC means you must be careful about how much allocation can occur on each system call. > can't we have a single language suited for everything ? I can't really answer that one. Part of me hopes so. But the thought of writing a kernel in rpython is not a pleasant one for me. > cant we write it in python to quickly enter the market ? Well, you don't spend much time writing a kernel anyway - use an existing kernel and then run python in userspace. It's pretty unusual to need your code colocated with the kernel, but it would be easier to do with a runtime like pypy (because of memory safety, you can guarantee that code will never do anything that traps). I don't know what you would do about interrupt handlers and GC. > doesn't pypy convert the python code to optimised c code ( am i > wrong ? ) It does, yes. But there is much work to do to remove the dependence on ANSI C functionality; things like memory mapping and allocation, file descriptors, etc. -- William Leslie ___ pypy-dev mailing list pypy-dev@python.org http://mail.python.org/mailman/listinfo/pypy-dev
Re: [pypy-dev] OS
With a little work it would be possible to target ring0 with rpython, but the real question is why you would want to. There are many other languages better suited to the task. On 11/12/2011 1:01 PM, "Rinu Boney" wrote: can RPython be converted to low level code as to create something like a kernel or something low level as that ? what about an OS with assembly lang , little C/C++ and RPython ? can an OS like android be developed using RPython ? ( by RPython i also mean using the PyPy tool chain ) ___ pypy-dev mailing list pypy-dev@python.org http://mail.python.org/mailman/listinfo/pypy-dev ___ pypy-dev mailing list pypy-dev@python.org http://mail.python.org/mailman/listinfo/pypy-dev
Re: [pypy-dev] Slow sqlite user defined functions with pypy.
Ack. On 17 November 2011 12:23, William ML Leslie wrote: > On 17 November 2011 12:13, Elefterios Stamatogiannakis > wrote: >> Pypy seems to not jit at all when a (pypy) Python function is called from C. > > Calls to native functions must be residualised, as there is no way to > tell what state gives rise to the call to the UDF. > > If there was a loop inside the UDF, that would still be compiled. But > the loop that occurs in the query must just call the C function in the > usual way, as the JIT has no idea what it might do. -- William Leslie ___ pypy-dev mailing list pypy-dev@python.org http://mail.python.org/mailman/listinfo/pypy-dev
Re: [pypy-dev] Maintaining an exhaustive list of supported and non-supported packages
On 24 October 2011 18:27, Jonathan Livni wrote: > Would it be possible to maintain a user-generated exhaustive list of > supported and non-supported packages? Not only would it be possible, it is currently done! https://bitbucket.org/pypy/compatibility/wiki/Home We should probably link here from the 'compatability' page on the user-facing website. -- William Leslie ___ pypy-dev mailing list pypy-dev@python.org http://mail.python.org/mailman/listinfo/pypy-dev
Re: [pypy-dev] How to organize the py3k branch (was: [pypy-commit] pypy py3k: Remove print statement)
On 13 October 2011 19:55, Antonio Cuni wrote: > - support for py2 and py3 in the same branch, with minimal duplication of > code. This would mean that e.g. in ast.py you would have tons of "if py2: > enable_print_stmt()", etc. Personally, I think that the codebase would > become too cluttered. In order to support __future__.print_function, you already need this case. Do we have a good idea of where we expect the biggest divergence? Is it things that now return iterables or views, or is it IO? Is it new-style defaults, or unicode field names? > - support for py2 and py3 in the same branch, with some duplication of code; > e.g., we could copy the existing interpreter/ into interpreter/py3k and > modify it there. While we are at it, we should try hard to minimize the > code duplication, but then it's up to us to decide when it's better to > duplicate or when it's better to share the code between the twos. An adaptive approach sounds very sensible. -- William Leslie ___ pypy-dev mailing list pypy-dev@python.org http://mail.python.org/mailman/listinfo/pypy-dev
Re: [pypy-dev] Realtime communication and webserver to use with pypy?
On 30 September 2011 13:39, John Anderson wrote: > In cpython I deploy using gevent or gunicorn for high performance/low memory > usage with the ability to be non-blocking for realtime communication using > socket.io. > If I want to move to using PyPy... what are my options for this type of > setup? Is there a non-blocking webserver in python that works well with > PyPy? Twisted has worked well for some time. Gevent is written in cython, which is currently not supported. Not sure about Gunicorn, it seems to be able to sit on top of several different workers. -- William Leslie ___ pypy-dev mailing list pypy-dev@python.org http://mail.python.org/mailman/listinfo/pypy-dev
Re: [pypy-dev] djangobench performance
On 1 September 2011 15:29, Fenn Bailey wrote: > The results were a little surprising (and not in a good way): > http://pastie.org/2463906 ... > Any ideas as to why the performance drop-off would be so significant? N = 200 means most of the benchmarks probably won't even JIT, so that might be a start. The threshold in the released pypy is N = 1000. But even without JIT, 20+ fold slowdowns are very interesting: 10n_render, query_all and query_raw. I wonder if anyone has benchmarked sqlite under pypy - that would have the most dramatic effect here. -- William Leslie ___ pypy-dev mailing list pypy-dev@python.org http://mail.python.org/mailman/listinfo/pypy-dev
Re: [pypy-dev] Object identity and dict strategies
On 12 July 2011 00:21, William ML Leslie wrote: > Referential equivalence, which is a slightly more complicated (yet > much better defined) idea says that x and y are equivalent when no > operation can tell the difference between the two objects. Ack, sorry. I meant Referential Transparency; which is much more googleable! -- William Leslie ___ pypy-dev mailing list pypy-dev@python.org http://mail.python.org/mailman/listinfo/pypy-dev
Re: [pypy-dev] Object identity and dict strategies
On 11 July 2011 23:21, Bengt Richter wrote: > On 07/11/2011 01:36 PM William ML Leslie wrote: >> >> On 11 July 2011 20:29, Bengt Richter wrote: >>> >>> On 07/10/2011 09:13 PM Laura Creighton wrote: >>>> >>>> What do we want to happen when somebody -- say in a C extension -- takes >>>> the id of an object >>>> that is scheduled to be removed when the gc next runs? >>> >>> IMO taking the id should increment the object ref counter >>> and prevent the garbage collection, until the id value itself is garbage >>> collected. >> >> This significantly changes the meaning of id() in a way that will >> break existing code. >> > Do you have an example of existing code that depends on the integer-cast > value of a dangling pointer?? I mean that id creating a reference will break existing code. id() has always returned an integer, and the existence of some integer in some python code has never prevented some otherwise unrelated object from being collected. Existing code will not make sure that it cleans up the return value of id(), as nowhere has id() ever kept a reference to the object passed in. I know that you are suggesting that id returns something that is /not/ an integer, but that is also a language change. People have always been able to assume that they can % format ids as decimals or hexadecimals. > Or do you mean that id's must be allowed to be compared == to integers, > which my example prohibits? (I didn't define __cmp__, BTW, just lazy ;-) Good, __cmp__ has been deprecated for over 10 years now. >> If you want an object reference, just use one. If you want them to be >> persistent, build a dictionary from id to object. > > Yes, dictionary is one way to bind an object and thus make sure its id is > valid. > > But it would be overkill to use a dictionary to guarantee object id > persistence > just for the duration of an expression such as id(x.a) == id(y.a) But id is not about persistence. The lack of persistence is one of its key features. That said, I do think id()'s current behaviour is overkill. I just don't think we can change it in a way that will fit existing usage. And cleaning it up properly is far too much work. >> You can already do >> this yourself in pure python, and it doesn't have the side-effect of >> bloating id(). > > My examples *are* in pure python ;-) As is copy.py. We've seen several examples on this thread where you can build additional features on top of what id() gives you without changing id(). So we've no need to break id() in any of the ways that have been suggested here. >> Otherwise, such a suggestion should go through the usual process for >> such a significant change to a language primitive. >> > Sure, but I only really want to understand the real (well, *intended* ;-) > meaning of the id function, so I am putting forth illustrative examples > to identify aspects of its current and possible behavior. The notion of identity is important in any stateful language. Referential equivalence, which is a slightly more complicated (yet much better defined) idea says that x and y are equivalent when no operation can tell the difference between the two objects. 'is' is an approximation that is at least accurate for mutability of python objects. In order for x to "be" y, assignments like x.__class__ = Foo must have exactly the same effect as y.__class__ = Foo. You could presumably write a type in the implementation language that was in no way discernable from the real x, but if x is y, you *know* there is no difference. What id() does is it attempts to distil 'the thing compared' when 'is' is used. On cpython, it just returned the integer value of the pointer to the object, because on cpython that is cheap and does the job (and hey, it *is* the thing compared when you do 'is' on cpython). On pypy, things are slightly more complicated. Pypy is written in python, which has no concept of pointers. It translates to the JVM and the (safe) CLI, neither of which have a direct analogue of the pointer. And even when C or LLVM is used as the backend, objects may move around in memory. Having id() return different values after a collection cycle would be very confusing. So, pypy implements its own, quite clever mechanism for creating ids. It is described in a blog post, if you'd like to read it. The definition of id(), according to docs.python.org, is: Return the “identity” of an object. This is an integer (or long integer) which is guaranteed to be unique and constant for this object during its lifetime. Two objects with non-overlapping lifetimes may have the same id() value. > Also, a new id could live alongside the old ;-) It's j
Re: [pypy-dev] Object identity and dict strategies
On 11 July 2011 20:29, Bengt Richter wrote: > On 07/10/2011 09:13 PM Laura Creighton wrote: >> >> What do we want to happen when somebody -- say in a C extension -- takes >> the id of an object >> that is scheduled to be removed when the gc next runs? > > IMO taking the id should increment the object ref counter > and prevent the garbage collection, until the id value itself is garbage > collected. This significantly changes the meaning of id() in a way that will break existing code. If you want an object reference, just use one. If you want them to be persistent, build a dictionary from id to object. You can already do this yourself in pure python, and it doesn't have the side-effect of bloating id(). Otherwise, such a suggestion should go through the usual process for such a significant change to a language primitive. -- William Leslie ___ pypy-dev mailing list pypy-dev@python.org http://mail.python.org/mailman/listinfo/pypy-dev
Re: [pypy-dev] Object identity and dict strategies
On 9 July 2011 18:17, Armin Rigo wrote: > Hi, > > On Sat, Jul 9, 2011 at 5:20 AM, William ML Leslie > wrote: >> My point about small integers (...) > > I think that your point about small integers is broken (even assuming > that smallints are enabled by default, which is not the case). It > means that we'd get an id() function which "behaves" as long as taking > the id() of a 31-bit signed integer, and then doesn't behave as > expected neither for full 32-bit integers, nor for longs, nor for > floats. > >> ... > > I don't understand these two sentences, because they seem to say the > exact opposite of each other... For a non-smallint integer or float bound to x, "x is x" is tautological by virtue of x being represented by the same instance. There may be other objects with the same value, but I don't see why that must imply that they be the same object - why x == y must imply x is y for x and y of the same immutable type. It might make the identity dict in copy.py use slightly less memory, but it would make *much* more sense to optimise the specific use case according to the defined semantics of id(). copy.py is already effectively "broken" by the behaviour of non-cached ints on cpython; so copy.py is no excuse to break id() in pypy, which is a much more fundamental concept. >> That does mean that id() is no longer word-sized, but it does not make it >> unbounded. > > The "unbounded" part in my e-mail was about longs. Obviously if you > are computing id(x) where x is in some finite set (like ints or > floats), then the results are in some finite set too. I'm not disagreeing with you there, but id has a *universally* finite range already, and imao, to change this is to make an annoying change to the language. >> Speaking of, maybe it'd be easier to attempt to get the identity dict >> into the language proper. > > We tried at some point, but python-dev refused the idea. Maybe the > idea has more chances for approval now that we can really show with > performance numbers that it's a deep issue, as opposed to just wave > our hands. Feel free to try again. In the meantime I've decided, at > least myself, to stick with the approach that Python is whatever is in > 2.7, and that we have to work around such issues instead of fixing > them properly in the language. Ok. It would be little help to existing code. -- William Leslie ___ pypy-dev mailing list pypy-dev@python.org http://mail.python.org/mailman/listinfo/pypy-dev
Re: [pypy-dev] Object identity and dict strategies
On 8 July 2011 19:52, Armin Rigo wrote: > Hi William, Hi Armin, everybody, > On Fri, Jul 8, 2011 at 10:31 AM, William ML Leslie > wrote: >> On another note: what Alex talks about as being two different cases >> are just one with the small int optimisation - all references can be >> compared by value in the C backend with small ints enabled, if the >> object space doesn't provide alternative behaviour. > > No, because e.g. of longs. You don't have the same issue if you use > longs instead of ints in this particular example, but more generally > the issue exists too. > > A first note is that it's impossible to satisfy all three of Alex's > criteria in general: we would like id(x) to be a unique word-sized > number determined only by the value of 'x', and different for every > value of 'x' and for every object of a different type too; but if the > possible 'x'es are all possible word-sized integers, then it's > impossible to satisfy this, just because there are too many of them. > The problem only gets "more impossible" if we include all long objects > in 'x'. That id(x) be a word-sized value uniquely determined by the value of x is impossible, yes, as the following program should demonstrate: while True: id([]) We create an infinite number of objects here, and if each one had to have a unique word-sized id, we'd exhaust that space pretty quickly. What id() does provide is that as long as there is a *reference* to the object, it returns a constant and unique integer (which we are assuming should be a word-sized integer). As later emails on this list suggested, it would be better if the semantics were relaxed to "the id should be preserved", meaning that placement of an integer or string into a container should allow it to have the same id on retrieval. New objects resulting from operations such as multiplication or string concatenation need not have the same id as other objects with the same value - if we did that for strings, it would have serious consequences for parallel code. What I was suggesting is that, since every live object must be encoded in an object reference somewhere, that object references should be at least good enough to suggest an id. My point about small integers wasn't really about word-sized ints, I was talking about the smallint optimisation, which as I understood it, boxes small app-level integers in the C backend. It does this by shifting integers right and bitwise-oring with 1; encoding the integer into the reference. "By definition", as Carl put it, there are never more objects represented in this way than we can fit in a reasonable, bounded id size. The suggestion then is to use the value of the object reference cast as a word-sized integer as the id of the object for integers so encoded, and calculate the id of other objects in the usual fashion. This happens to work for larger integers and floats, because the id will be preserved as long as a reference to them exists by their boxedness. Floats could use a similar mechanism to integers, eg, their bit representation shifted left two and bitwise-ored with 2. That does mean that id() is no longer word-sized, but it does not make it unbounded. > The problem is not new, it is just a bit more apparent. For example, > already in pypy 1.5 we have: > >>>>> a = A() >>>>> d = a.__dict__ >>>>> s = 'foobar' >>>>> d[s] = 5 >>>>> id(s) > 163588812 >>>>> id(d.keys()[0]) > 163609508 >>>>> id(d.keys()[0]) > 163609520 What is keeping us from using the underlying rpython string as the basis for id? I guess there is nowhere obvious to store the id or the need to store the id in the gc copy phase. In that way, it'd make for an ugly special case. On 9 July 2011 00:07, Armin Rigo wrote: > After some discussion with Carl Friedrich, here is the best we could > think of. Say that "a is b" is redefined as being True for two equal > immutable objects of the same type. Then we want the following > property to always hold: "a is b" if and only if "id(a) == id(b)". I would prefer to weaken this just a little: x is y iff id(x) == id(y) WHEN x and y are live for the duration of the equality. This counters cases such as: id([]) == id([]) which can certainly happen under any reasonable definition of id. > We > can do that by having id(x) return either a regular integer or a long. > Let's call for the rest of this discussion an "immutable object" an > object of type int, long or float. If 'x' is not an immutable object, > then id(x) can return the same value as it does now. If 'x' is an > immutable object, then we comput
Re: [pypy-dev] Object identity and dict strategies
On 8 July 2011 17:58, holger krekel wrote:
> IOW, i think the issue here is that iterating over keys of a dict usually
> gives the exact same ("is") objects in CPython whereas pypy trunk does not
> provide that at least for ints.
I couldn't find anything precise in the official documentation on the
meaning of 'is'. I think that the general understanding is that it
makes no sense whatsoever on immutable objects (as in, it isn't
guaranteed to do so).
Consequently, a python implementation could also cache tuples.
Re-using tuples might sound unusual, but there are special cases that
start to sound reasonable, such as caching the empty tuple, or
copy-propogating a tuple unpack & repack. The language spec is very
light on what is allowed to be a 'different object', and what is
*suggested* by cpython's int caching behaviour is that the behaviour
of 'is' for language-provided immutable objects can't be relied upon
in any way, shape or form.
Pypy hasn't matched cpython's behaviour with ints here in a long time,
so it obviously doesn't matter.
On another note: what Alex talks about as being two different cases
are just one with the small int optimisation - all references can be
compared by value in the C backend with small ints enabled, if the
object space doesn't provide alternative behaviour.
--
William Leslie
___
pypy-dev mailing list
pypy-dev@python.org
http://mail.python.org/mailman/listinfo/pypy-dev
Re: [pypy-dev] SmallTuples
On 26 May 2011 16:00, Amaury Forgeot d'Arc wrote: > 2011/5/26 William ML Leslie : >> The python tuple is intended to be a heterogeneous data structure, as >> they are in RPython. The length of a tuple is conceptually part of the >> type, which means that tuple length will be static in sane code >> anyway. > > But does this apply to the *args tuple? Good point - you mean "sometimes, frequently not". In the context we are considering, namely promotion of tuple length for small tuples, I want to pretend the answer is yes. It means that we get call-site specialisation for heterogeneous usage, and ignore it for most homogeneous usage. Heterogeneous *args usage is probably common enough - for example, in wrapping decorators, it is common to accept and pass *args and **kwargs to the wrapped function, and if the length of the tuple were promoted, the pack and unpack could be simplified. I suspect that limiting the discussion to /small/ tuples will take care of the outliers, but benchmarks are a good idea. -- William Leslie ___ pypy-dev mailing list pypy-dev@python.org http://mail.python.org/mailman/listinfo/pypy-dev
Re: [pypy-dev] SmallTuples
On 26 May 2011 14:23, William ML Leslie wrote: > On 26 May 2011 14:15, Alex Gaynor wrote: >> Well, IMO the issue is you can end up with a ton of guard_class that are >> more precise than you need, plus you have to manually specialize for each >> length, whereas with the other approach you can just automatically apply it >> to all tuples. > > With the class approach, you can store the length on the class, saving > some space for each object. Also possibly relevant: The python tuple is intended to be a heterogeneous data structure, as they are in RPython. The length of a tuple is conceptually part of the type, which means that tuple length will be static in sane code anyway. I don't think this is over-specialisation, because for homogeneous usage of tuples we usually won't be talking about SmallTuples. I guess that it does have potential to blow up, and wonder if we already have a system for deoptimising the truly polymorphic case. -- William Leslie ___ pypy-dev mailing list pypy-dev@python.org http://mail.python.org/mailman/listinfo/pypy-dev
Re: [pypy-dev] SmallTuples
On 26 May 2011 14:15, Alex Gaynor wrote: > Well, IMO the issue is you can end up with a ton of guard_class that are > more precise than you need, plus you have to manually specialize for each > length, whereas with the other approach you can just automatically apply it > to all tuples. With the class approach, you can store the length on the class, saving some space for each object. -- William Leslie ___ pypy-dev mailing list pypy-dev@python.org http://mail.python.org/mailman/listinfo/pypy-dev
Re: [pypy-dev] cpyext - compilation, and swig?
On 14 May 2011 06:27, Amaury Forgeot d'Arc wrote: > Also, there are probably other directions that we have not yet explored: > - implement another C API better suited to pypy > - a SWIG backend which emits ctypes code or similar > And probably others... Not to mention cppyy, which might be a good fit for SWIG. Has anyone tried to use cppyy recently? -- William Leslie ___ pypy-dev mailing list pypy-dev@python.org http://mail.python.org/mailman/listinfo/pypy-dev
