Re: [Python-Dev] Obtaining stack-frames from co-routine objects
On Thu, Jun 11, 2015 at 11:38 PM, Ben Leslie be...@benno.id.au wrote: On 2 June 2015 at 14:39, Yury Selivanov yselivanov...@gmail.com wrote: Hi Ben, On 2015-05-31 8:35 AM, Ben Leslie wrote: Hi Yury, I'm just starting my exploration into using async/await; all my 'real-world' scenarios are currently hypothetical. One such hypothetical scenario however is that if I have a server process running, with some set of concurrent connections, each managed by a co-routine. Each co-routine is of some arbitrary complexity e.g: some combination of reading files, reading from database, reading from peripherals. If I notice one of those co-routines appears stuck and not making progress, I'd very much like to debug that, and preferably in a way that doesn't necessarily stop the rest of the server (or even the co-routine that appears stuck). The problem with the if debug: log(...) approach is that you need foreknowledge of the fault state occurring; on a busy server you don't want to just be logging every 'switch()'. I guess you could do something like switch_state[outer_coro] = get_current_stack_frames() on each switch. To me double book-keeping something that the interpreter already knows seems somewhat wasteful but maybe it isn't really too bad. I guess it all depends on how switching is organized in your framework of choice. In asyncio, for instance, all the code that knows about coroutines is in tasks.py. `Task` class is responsible for running coroutines, and it's the single place where you would need to put the if debug: ... line for debugging slow Futures-- the only thing that coroutines can stuck with (the other thing is accidentally calling blocking code, but your proposal wouldn't help with that). I suspect that I haven't properly explained the motivating case. My motivating case is being able to debug a relatively large, complex system. If the system crashes (through an exception), or in some other manner enters an unexpected state (co-routines blocked for too long) it would be very nice to be able to debug an arbitrary co-routine, not necessarily the one indicating a bad system state, to see exactly what it is/was doing at the time the anomalous behavior occurs. So, this is a case of trying to analyse some system wide behavior than necessarily one particular task. So until you start analysing the rest of the system you don't know which co-routines you want to analyse. My motivation for this is primarily avoiding double book-keeping. I assume that the framework has organised things so that there is some data structure to find all the co-routines (or some other object wrapping the co-routines) and that all the switching occurs in one place. With that in mind I can have some code that works something like: @coroutine def switch(): coro_stacks[current_coro] = inspect.getouterframes(inspect.currentframe()) I think something like this is probably the best approach to achieve my desired goals with currently available APIs. However I really don't like it as it required this double book-keeping. I'm manually retaining this trace back for each coro, which seems like a waste of memory, considering the interpreter already has this information stored, just unexposed. I feel it is desirable, and in-line with existing Python patterns to expose the interpreter data structures, rather than making the user do extra work to access the same information. Ben, I suspect that this final paragraph is actually the crux to your request. You need to understand what the interpreter is doing before you can propose an API to its data structures. The particular thing to understand about coroutines is that a coroutine which is suspended at yield or yield from has a frame but no stack -- the frame holds the locals and the suspension point, but it is not connected to any other frames. Its f_back pointer is literally NULL. (Perhaps you are more used to threads, where a suspended thread still has a stack.) Moreover, the interpreter has no bookkeeping that keeps track of suspended frames. So I'm not sure exactly what information you think the interpreter has stored but does not expose. Even asyncio/tasks.py does not have this bookkeeping -- it keeps track of Tasks, which are an asyncio-specific class that wraps certain coroutines, but not every coroutine is wrapped by a Task (and this is intentional, as a coroutine is a much more lightweight data structure than a Task instance). IOW I don't think that the problem here is that you haven't sufficiently motivated your use case -- you are asking for information that just isn't available. (Which is actually where you started the thread -- you can get to the frame of the coroutine but there's nowhere to go from that frame.) -- --Guido van Rossum (python.org/~guido) ___ Python-Dev mailing list Python-Dev@python.org
[Python-Dev] Summary of Python tracker Issues
ACTIVITY SUMMARY (2015-06-05 - 2015-06-12) Python tracker at http://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open4884 (+31) closed 31317 (+22) total 36201 (+53) Open issues with patches: 2239 Issues opened (40) == #24390: Python 3.4.3 64 bits is not high dpi aware http://bugs.python.org/issue24390 opened by ivb #24391: Better repr for threading objects http://bugs.python.org/issue24391 opened by serhiy.storchaka #24393: Test urllib2_localnet fails depending on host proxy configurat http://bugs.python.org/issue24393 opened by eacb #24395: webbrowser.py update to use argparse.py http://bugs.python.org/issue24395 opened by Hasan Diwan #24396: Provide convenience function for paths relative to the current http://bugs.python.org/issue24396 opened by madison.may #24397: Test asynchat makes wrong assumptions http://bugs.python.org/issue24397 opened by eacb #24398: Update test_capi to use test.support.script_helper http://bugs.python.org/issue24398 opened by bobcatfish #24400: Awaitable ABC incompatible with functools.singledispatch http://bugs.python.org/issue24400 opened by Ben.Darnell #24401: Windows 8.1 install gives DLL required to complete could not r http://bugs.python.org/issue24401 opened by lac #24402: input() uses sys.__stdout__ instead of sys.stdout for prompt http://bugs.python.org/issue24402 opened by Keita Kita #24403: Missing fixer for changed round() behavior http://bugs.python.org/issue24403 opened by priska #24405: Missing code markup in Expressions documentation http://bugs.python.org/issue24405 opened by Gareth.Rees #24406: Built-in Types documentation doesn't explain how dictionarie http://bugs.python.org/issue24406 opened by Gareth.Rees #24407: Use after free in PyDict_merge http://bugs.python.org/issue24407 opened by pkt #24408: tkinter.font.Font.measure() broken in 3.5 http://bugs.python.org/issue24408 opened by vadmium #24412: setUpClass equivalent for addCleanup http://bugs.python.org/issue24412 opened by r.david.murray #24413: Inconsistent behavior between set and dict_keys/dict_items: fo http://bugs.python.org/issue24413 opened by canjo #24414: MACOSX_DEPLOYMENT_TARGET set incorrectly by configure http://bugs.python.org/issue24414 opened by debohman #24415: SIGINT always reset to SIG_DFL by Py_Finalize() http://bugs.python.org/issue24415 opened by ABalmosan #24416: Return a namedtuple from date.isocalendar() http://bugs.python.org/issue24416 opened by bmispelon #24417: Type-specific documentation for __format__ methods http://bugs.python.org/issue24417 opened by skip.montanaro #24418: make install will not install pip if already present in user http://bugs.python.org/issue24418 opened by pitrou #24419: In argparse action append_const doesn't work for positional ar http://bugs.python.org/issue24419 opened by py.user #24420: Documentation regressions from adding subprocess.run() http://bugs.python.org/issue24420 opened by vadmium #24421: Race condition compiling Modules/_math.c http://bugs.python.org/issue24421 opened by vadmium #24424: xml.dom.minidom: performance issue with Node.insertBefore() http://bugs.python.org/issue24424 opened by Robert Haschke #24425: Installer Vender Issue http://bugs.python.org/issue24425 opened by Hayden Young #24426: re.split performance degraded significantly by capturing group http://bugs.python.org/issue24426 opened by Patrick Maupin #24427: subclass of multiprocessing Connection segfault upon attribute http://bugs.python.org/issue24427 opened by neologix #24429: msvcrt error when embedded http://bugs.python.org/issue24429 opened by erik flister #24430: ZipFile.read() cannot decrypt multiple members from Windows 7z http://bugs.python.org/issue24430 opened by era #24431: StreamWriter.drain is not callable concurrently http://bugs.python.org/issue24431 opened by Martin.Teichmann #24432: Upgrade windows builds to use OpenSSL 1.0.2b http://bugs.python.org/issue24432 opened by alex #24434: ItemsView.__contains__ does not mimic dict_items http://bugs.python.org/issue24434 opened by clevy #24435: Grammar/Grammar points to outdated guide http://bugs.python.org/issue24435 opened by Rosuav #24436: _PyTraceback_Add has no const qualifier for its char * argumen http://bugs.python.org/issue24436 opened by mic-e #24437: Add information about the buildbot console view and irc notice http://bugs.python.org/issue24437 opened by r.david.murray #24438: Strange behaviour when multiplying imaginary inf by 1 http://bugs.python.org/issue24438 opened by Alex Monk #24439: Feedback for awaitable coroutine documentation http://bugs.python.org/issue24439 opened by vadmium #24440: Move the buildslave setup information from the wiki to the dev http://bugs.python.org/issue24440 opened by r.david.murray Most recent 15 issues with no replies (15)
Re: [Python-Dev] Obtaining stack-frames from co-routine objects
On 2 June 2015 at 14:39, Yury Selivanov yselivanov...@gmail.com wrote: Hi Ben, On 2015-05-31 8:35 AM, Ben Leslie wrote: Hi Yury, I'm just starting my exploration into using async/await; all my 'real-world' scenarios are currently hypothetical. One such hypothetical scenario however is that if I have a server process running, with some set of concurrent connections, each managed by a co-routine. Each co-routine is of some arbitrary complexity e.g: some combination of reading files, reading from database, reading from peripherals. If I notice one of those co-routines appears stuck and not making progress, I'd very much like to debug that, and preferably in a way that doesn't necessarily stop the rest of the server (or even the co-routine that appears stuck). The problem with the if debug: log(...) approach is that you need foreknowledge of the fault state occurring; on a busy server you don't want to just be logging every 'switch()'. I guess you could do something like switch_state[outer_coro] = get_current_stack_frames() on each switch. To me double book-keeping something that the interpreter already knows seems somewhat wasteful but maybe it isn't really too bad. I guess it all depends on how switching is organized in your framework of choice. In asyncio, for instance, all the code that knows about coroutines is in tasks.py. `Task` class is responsible for running coroutines, and it's the single place where you would need to put the if debug: ... line for debugging slow Futures-- the only thing that coroutines can stuck with (the other thing is accidentally calling blocking code, but your proposal wouldn't help with that). I suspect that I haven't properly explained the motivating case. My motivating case is being able to debug a relatively large, complex system. If the system crashes (through an exception), or in some other manner enters an unexpected state (co-routines blocked for too long) it would be very nice to be able to debug an arbitrary co-routine, not necessarily the one indicating a bad system state, to see exactly what it is/was doing at the time the anomalous behavior occurs. So, this is a case of trying to analyse some system wide behavior than necessarily one particular task. So until you start analysing the rest of the system you don't know which co-routines you want to analyse. My motivation for this is primarily avoiding double book-keeping. I assume that the framework has organised things so that there is some data structure to find all the co-routines (or some other object wrapping the co-routines) and that all the switching occurs in one place. With that in mind I can have some code that works something like: @coroutine def switch(): coro_stacks[current_coro] = inspect.getouterframes(inspect.currentframe()) I think something like this is probably the best approach to achieve my desired goals with currently available APIs. However I really don't like it as it required this double book-keeping. I'm manually retaining this trace back for each coro, which seems like a waste of memory, considering the interpreter already has this information stored, just unexposed. I feel it is desirable, and in-line with existing Python patterns to expose the interpreter data structures, rather than making the user do extra work to access the same information. Cheers, Ben ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
