Re: [pypy-dev] pypy 4.0.0 "Fatal RPython error: AssertionError"
Nice work on deducing the issue from the limited information provided! I've downloaded the referenced nightly build and tested today. It looks like this does indeed fix the issue. I've run through the section of python code about 30 times so far without issue where it was failing about 1 in 4 times previously. -Andrew On Sun, Nov 8, 2015 at 7:23 AM, Armin Rigo wrote: > Hi Andrew, > > On Sun, Nov 8, 2015 at 9:52 AM, Armin Rigo wrote: > > I'm still going with checking the gctype of the object. It should > > work in the same cases (and theoretically more but well). > > Not done yet, but I think I fixed Andrew's original problem. Andrew, > could you try again with the latest build? > http://buildbot.pypy.org/nightly/trunk/pypy-c-jit-latest-linux64.tar.bz2 > > > A bientôt, > > Armin. > -- Andrew Mann DivvyCloud Inc. www.divvycloud.com ___ pypy-dev mailing list pypy-dev@python.org https://mail.python.org/mailman/listinfo/pypy-dev
Re: [pypy-dev] pypy 4.0.0 "Fatal RPython error: AssertionError"
Hi Andrew, On Sun, Nov 8, 2015 at 9:52 AM, Armin Rigo wrote: > I'm still going with checking the gctype of the object. It should > work in the same cases (and theoretically more but well). Not done yet, but I think I fixed Andrew's original problem. Andrew, could you try again with the latest build? http://buildbot.pypy.org/nightly/trunk/pypy-c-jit-latest-linux64.tar.bz2 A bientôt, Armin. ___ pypy-dev mailing list pypy-dev@python.org https://mail.python.org/mailman/listinfo/pypy-dev
Re: [pypy-dev] pypy 4.0.0 "Fatal RPython error: AssertionError"
On Sun, Nov 8, 2015 at 9:52 AM, Armin Rigo wrote: > > Ah, that's a good plan too. It needs to have a way to record what we > saw during tracing (preferably not in some huge dict), and we decided > long ago that the optimizer should not essentially look at the content > of boxes (apart to guide some optimizations). For example, CALL_PURE > uses a big dict. The problem is that such a dict would be much bigger > for all GETFIELD_PURE. > Why not store the observed results on the resops (maybe using some new OpWithObservedResult mixin to not get an extra field on all of them)? > > I'm still going with checking the gctype of the object. It should > work in the same cases (and theoretically more but well). > Yes, makes sens. Another thing that strikes me is that those crazy cases does probably not benefit much from unrolling, and the only result of unrolling them might be to duplicate a lot of guards, which leads to more tracing of bridges. Maybe it would be beneficial to compare the constant inputargs of the peeled loop with the observed inputargs of the preamble and if they don't match (differ enough?) skip unrolling for that loop. Alternatively demote the non-matching constants non-constants (as they are obviously not loop invariant constants) to make the peeled loop a more general target for bridges to jump directly to. -- Håkan Ardö ___ pypy-dev mailing list pypy-dev@python.org https://mail.python.org/mailman/listinfo/pypy-dev
Re: [pypy-dev] pypy 4.0.0 "Fatal RPython error: AssertionError"
Hi Hakan, On Sun, Nov 8, 2015 at 9:39 AM, Hakan Ardo wrote: > A small adjustment to your approach that would cover those cases is to > compare the values of the constants with the values observed during tracing. > If they are the same it should be safe to perform the read, right? Ah, that's a good plan too. It needs to have a way to record what we saw during tracing (preferably not in some huge dict), and we decided long ago that the optimizer should not essentially look at the content of boxes (apart to guide some optimizations). For example, CALL_PURE uses a big dict. The problem is that such a dict would be much bigger for all GETFIELD_PURE. I'm still going with checking the gctype of the object. It should work in the same cases (and theoretically more but well). A bientôt, Armin. ___ pypy-dev mailing list pypy-dev@python.org https://mail.python.org/mailman/listinfo/pypy-dev
Re: [pypy-dev] pypy 4.0.0 "Fatal RPython error: AssertionError"
On Sat, Nov 7, 2015 at 2:40 PM, Armin Rigo wrote: > > Would that miss an important case? > I would guess that the interesting cases are when we have the same constant in both iterations. In that case what we would miss is reads performed on the constant pointer before it is proven to be a constant. If these are numerous enough to care about I don't know, but I would not be surprised. A small adjustment to your approach that would cover those cases is to compare the values of the constants with the values observed during tracing. If they are the same it should be safe to perform the read, right? -- Håkan Ardö ___ pypy-dev mailing list pypy-dev@python.org https://mail.python.org/mailman/listinfo/pypy-dev
Re: [pypy-dev] pypy 4.0.0 "Fatal RPython error: AssertionError"
Hi,
On Sat, Nov 7, 2015 at 2:40 PM, Armin Rigo wrote:
> Would that miss an important case?
Fijal shed doubt on that, so I'm going with this solution:
For GC objects, check the gctype or correct subclass when doing an
access from the optimizer. (We can't do that with Boehm, but then we
never unroll with Boehm anyway, because the same checks are needed for
short preambles.)
For RAW pointers, kill the resoperations
GET{FIELD,ARRAYITEM}_RAW_PURE, but *not* the jitcodes
get{field,arrayitem}_raw_*_pure. The latter are used for example in
all RPython classes' vtable accesses, and so are essential. But in
all cases I could find, they'd be constant-folded during tracing
already (e.g. a previous guard_class always promotes the class to a
tracing-time constant). For the rare case where they are not, we can
simply emit a regular GET{FIELD,ARRAYITEM}_RAW.
A bientôt,
Armin.
___
pypy-dev mailing list
pypy-dev@python.org
https://mail.python.org/mailman/listinfo/pypy-dev
Re: [pypy-dev] pypy 4.0.0 "Fatal RPython error: AssertionError"
ReHi, On Sat, Nov 7, 2015 at 2:25 PM, Armin Rigo wrote: > More thoughts needed. One solution would be to use two modes when optimizing the two unrolled versions of the loop. For the 1st version we do like now. But the 2nd version is run "speculatively" and so more care is needed: we should not do any actual memory read (from either GC or raw pointers). Instead, we record the reads done during the 1st version's optimization, and we replay them during the 2nd version. The common case should be that there are reads from the same locations anyway. But if an attempted read occurs *only* during the 2nd version, like in these strange examples, then we don't do any constant-folding. Would that miss an important case? A bientôt, Armin. ___ pypy-dev mailing list pypy-dev@python.org https://mail.python.org/mailman/listinfo/pypy-dev
Re: [pypy-dev] pypy 4.0.0 "Fatal RPython error: AssertionError"
Hi Hakan, On Sat, Nov 7, 2015 at 12:11 PM, Hakan Ardo wrote: > Also maybe for an opaque pointer to an array that was pointing some instance > in the first iteration? Yes, anything dereferencing a pointer basically---during optimization. It should only matter for Consts. I realized it is annoying: in theory, we can try to unroll this loop: [i0, i1] i2 = int_gt(i1, 0) guard_true(i2) [] i4 = getarrayitem_raw_pure_i(i0, 0, descr=rawarraydescr) i3 = int_sub(i1, 1) jump(12345, i3) The "12345" is a nonsensical constant raw pointer. Again, this only makes sense if it is called with i1 = 0 or 1, but unrolling will really try to perform the getarrayitem_raw_pure(12345, 0) during optimization. More thoughts needed. A bientôt, Armin. ___ pypy-dev mailing list pypy-dev@python.org https://mail.python.org/mailman/listinfo/pypy-dev
Re: [pypy-dev] pypy 4.0.0 "Fatal RPython error: AssertionError"
On Sat, Nov 7, 2015 at 11:27 AM, Armin Rigo wrote: > > This crash occurs when trying to optimize read/write fields on NULLs. > The same problem can occur too (even more rarely I suppose) if the > constant is not NULL, but is an instance of the wrong class. > Also maybe for an opaque pointer to an array that was pointing some instance in the first iteration? -- Håkan Ardö ___ pypy-dev mailing list pypy-dev@python.org https://mail.python.org/mailman/listinfo/pypy-dev
Re: [pypy-dev] pypy 4.0.0 "Fatal RPython error: AssertionError"
Hi Andrew, On Sat, Nov 7, 2015 at 9:43 AM, Maciej Fijalkowski wrote: > This looks fairly serious, we would love to have a way to reproduce > it. For what is worth, you don't have to minimize the example very > much - a program we can run, even if it has serious dependecies, is > ok, but we NEED to be able to reproduce it. We managed to figure it out just by discussion, for once. It seems that this time we won't need to reproduce it. The problem is that in some cases, loop unrolling can create a second iteration of a loop in which what used to be a variable argument is now a constant (like NULL). But the loop can contain operations to read/write fields on that variable. In this case, the second iteration would never be executed in practice (as it would segfault), but the optimizer doesn't realize that. This crash occurs when trying to optimize read/write fields on NULLs. The same problem can occur too (even more rarely I suppose) if the constant is not NULL, but is an instance of the wrong class. A bientôt, Armin. ___ pypy-dev mailing list pypy-dev@python.org https://mail.python.org/mailman/listinfo/pypy-dev
Re: [pypy-dev] pypy 4.0.0 "Fatal RPython error: AssertionError"
Hi Andrew This looks fairly serious, we would love to have a way to reproduce it. For what is worth, you don't have to minimize the example very much - a program we can run, even if it has serious dependecies, is ok, but we NEED to be able to reproduce it. On Sat, Nov 7, 2015 at 4:23 AM, Andrew Mann wrote: > Hi everyone, > > I'm testing our software on PyPy 4.0.0, and things generally work well, but > occasionally I encounter the following RPython traceback and subsequent core > dump: > > RPython traceback: > File "rpython_jit_metainterp_optimizeopt_unroll.c", line 9591, in > UnrollOptimizer_optimize_bridge > File "rpython_jit_metainterp_optimizeopt_unroll.c", line 17372, in > UnrollOptimizer_jump_to_existing_trace > File "rpython_jit_metainterp_optimizeopt_unroll.c", line 23101, in > UnrollOptimizer_inline_short_preamble > File "rpython_jit_metainterp_optimizeopt_intbounds.c", line 162, in > OptIntBounds__optimize_guard_true_false_value > File "rpython_jit_metainterp_optimizeopt_optimizer.c", line 18252, in > Optimizer_make_constant > File "rpython_jit_metainterp_optimizeopt_info.c", line 21500, in > AbstractStructPtrInfo_copy_fields_to_const > File "rpython_jit_metainterp_optimizeopt_info.c", line 12522, in > ConstPtrInfo__get_info > File "rpython_rtyper_lltypesystem_rordereddict_4.c", line 36646, in > ll_dict_get__dicttablePtr_GCREFPtr_rpython_jit_m > Fatal RPython error: AssertionError > > I don't want to waste anyone's time diagnosing the issue if it's a) known or > b) probably not anything the pypy devs care about. However, if it's > something that seems worth investigating, I'm happy to provide whatever > information is useful or debug as needed. I'm not really familiar with the > pypy architecture, but I know my way around debuggers and the various > languages in use here. > > Details: > > This is a flask based web application wrapper around sql alchemy access to a > mysql database. The operation that triggers this is a retrieval of about > 20,000 database rows with a total of maybe 5MB of data - so nothing too > huge. > > I also have cProfiler active and recording data. At the moment I'm not sure > if the problem happens with profiling enabled; I haven't tested. > > The same section of python code with the same data set (perhaps with > differences of timestamps) runs successfully most of the time. It seems to > fail about 1 in 4 passes through this section of code. > > This is run on an Ubuntu 14.04 VM with the x64 binaries of pypy 4.0.0 > retrieved from the public pypy web site downloads. > > Does this seem worth investigating further? > > -- > Andrew Mann > DivvyCloud Inc. > www.divvycloud.com > > ___ > pypy-dev mailing list > pypy-dev@python.org > https://mail.python.org/mailman/listinfo/pypy-dev > ___ pypy-dev mailing list pypy-dev@python.org https://mail.python.org/mailman/listinfo/pypy-dev
