[Python-Dev] Re: Deadlock when interrupting interpreter initialisation with ptrace?
> Don't you have the backtrace from libunwind that you could save from austinp > itself? Unfortunately no as the "deadlock" happens before any samples have a chance to be collected. Upon further investigation, it seems that trying to resume a thread over and over when ptrace fails takes quite "some" time (in fact, more than I'd have hoped). Playing with a larger wait timeout (100 ms, but the largest number I've seen so far on my machine is 4 ms, which is still an eternity compared to a sensible sampling interval of 10 ms) seems to "cure" the problem, which I've only seen during interpreter initialisation. So perhaps Python itself is off the hook! On Mon, 6 Jun 2022 at 19:20, Barry Scott wrote: > > > > On 6 Jun 2022, at 17:52, Gabriele wrote: > > I've found it hard to give an answer to this question. Because austinp > is already tracing the interpreter, I cannot use, e.g., gdb to dump a > backtrace. > > > Don't you have the backtrace from libunwind that you could save from austinp > itself? > > Barry > -- "Egli è scritto in lingua matematica, e i caratteri son triangoli, cerchi, ed altre figure geometriche, senza i quali mezzi è impossibile a intenderne umanamente parola; senza questi è un aggirarsi vanamente per un oscuro laberinto." -- G. Galilei, Il saggiatore. ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/Y2GHXXQGJ6CNHHIO6DVERKWKDRUCOQIR/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Deadlock when interrupting interpreter initialisation with ptrace?
> Do you know what is involved in the deadlock (as in, what the threads are > waiting on)? I've found it hard to give an answer to this question. Because austinp is already tracing the interpreter, I cannot use, e.g., gdb to dump a backtrace. The event is also quite rare and it seems to happen before austinp has the chance to capture any samples. With the new support for 3.11 I might be able to see if I come across the same issue with the latest beta. I was hoping that the description of the issue could have rung a bell for anybody more familiar than me with all the locking going on during imports. The logs from austinp seem to suggest that the thread fails to resume after being interrupted, so something for me to explore is whether attempting to resume the thread more times before giving up is the actual solution in this case. > How do you spawn a new process? I should have clarified that this is just a plain fork/exec from C: https://github.com/P403n1x87/austin/blob/e3d79ddc9f9737a791362e6962b5cac25a4e3dc2/src/py_proc.c#L972-L1010 Cheers, Gabriele On Mon, 6 Jun 2022 at 16:30, Victor Stinner wrote: > > On Mon, Jun 6, 2022 at 4:35 PM Gabriele wrote: > > The austinp variant is a variant of Austin > > (https://github.com/P403n1x87/austin) for Linux that uses ptrace to > > seize and interrupt/continue threads to capture native stack traces > > using libunwind. During testing, I have discovered that there are good > > chances of causing what looks like a deadlock in Python if the seizing > > and interrupting of threads happen very early when spawning a Python > > subprocess from austinp. > > Do you have a backtrace of the Python main thread when the hang > happens? How do you spawn a new process? With the Python subprocess > module? > > Victor > -- > Night gathers, and now my watch begins. It shall not end until my death. -- "Egli è scritto in lingua matematica, e i caratteri son triangoli, cerchi, ed altre figure geometriche, senza i quali mezzi è impossibile a intenderne umanamente parola; senza questi è un aggirarsi vanamente per un oscuro laberinto." -- G. Galilei, Il saggiatore. ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/DHRBK2WNXQJNKKQDHTGNSKRQ32MO365H/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Deadlock when interrupting interpreter initialisation with ptrace?
Hi there I hope you don't mind me sharing my experience with testing the austinp variant of Austin with Python >=2.7,<3.11. The austinp variant is a variant of Austin (https://github.com/P403n1x87/austin) for Linux that uses ptrace to seize and interrupt/continue threads to capture native stack traces using libunwind. During testing, I have discovered that there are good chances of causing what looks like a deadlock in Python if the seizing and interrupting of threads happen very early when spawning a Python subprocess from austinp. This seems to coincide with the initialisation of the interpreter when modules are being loaded. To avoid interfering so destructively with Python, I have added a sleep of about 0.5s on fork to prevent sampling during this initialisation phase, which has helped significantly. However, I think this poses one question: is this behaviour from Python to be expected or is it perhaps an indication of a potential bug? Whilst I find it conceivable that something like this could happen, given the locking that happens around imports, is it acceptable that the pausing and resuming of the execution of a thread lead to a potential deadlock? Cheers, Gabriele ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/EWE5IK53IAME7ODZOGCQGSSP4YBE37YX/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: code.replace() and Python 3.11 exception table
Does this mean that this line in the bytecode library is likely to fail
with 3.11, with no way to fix it?
https://github.com/MatthieuDartiailh/bytecode/blob/7b0423234b0e999b45a4eb0c58115b284314f46b/bytecode/concrete.py#L398
On Fri, 1 Apr 2022, 10:40 Victor Stinner, wrote:
> I created https://bugs.python.org/issue47185 to discuss this issue:
> either recompute automatically co_exceptiontable, or at least document
> the change.
>
> Victor
>
> On Fri, Apr 1, 2022 at 11:21 AM Victor Stinner
> wrote:
> >
> > ("Re: C API: Move PEP 523 "Adding a frame evaluation API to CPython"
> > private C API to the internal C API")
> >
> > On Fri, Apr 1, 2022 at 11:01 AM Chris Angelico wrote:
> > >
> > > On Fri, 1 Apr 2022 at 19:51, Victor Stinner
> wrote:
> > > > In Python, sadly the types.CodeType type also has a public
> constructor
> > > > and many projects break at each Python release because the API
> > > > changes. Hopefully, it seems like the new CodeType.replace() method
> > > > added to Python 3.8 mitigated the issue. IMO CodeType.replace() is a
> > > > better abstraction and closer to what developers need in practice.
> > >
> > > It certainly has been for me. When I want to do bytecode hackery, I
> > > usually start by creating a function with def/lambda, then construct a
> > > modified function using f.__code__.replace(). It's the easiest way to
> > > ensure that all the little details are correct.
> >
> > Python 3.11 added the concept of "exception table"
> > (code.co_exceptiontable). You have to build this table, otherwise
> > Python can no longer catch exceptions :-)
> >
> > I don't know how to build this exception table. It seems like
> > currently there is no Python function in the stdlib to build this
> > table.
> >
> > Example:
> > ---
> > def f():
> > try:
> > print("raise")
> > raise ValueError
> > except ValueError:
> > print("except")
> > else:
> > print("else")
> > print("exit func")
> >
> > def g(): pass
> >
> > if 1:
> > code = f.__code__
> > g.__code__ = g.__code__.replace(
> > co_code=code.co_code,
> > co_consts=code.co_consts,
> > co_names=code.co_names,
> > co_flags=code.co_flags,
> > co_stacksize=code.co_stacksize)
> > else:
> > g.__code__ = f.__code__ # this code path works on Python 3.11
> >
> > g()
> > ---
> >
> > Output with Python 3.10 (ok):
> > ---
> > raise
> > except
> > exit func
> > ---
> >
> > Output with Python 3.11 (oops):
> > ---
> > raise
> > Traceback (most recent call last):
> > ...
> > ValueError
> > ---
> >
> > By the way, this change is not documented at all:
> >
> > * https://docs.python.org/dev/library/types.html#types.CodeType
> > * https://docs.python.org/dev/whatsnew/3.11.html
> >
> > I understand that these changes come from the "Zero cost exception
> > handling" change:
> > https://bugs.python.org/issue40222
> >
> > Victor
> > --
> > Night gathers, and now my watch begins. It shall not end until my death.
>
>
>
> --
> Night gathers, and now my watch begins. It shall not end until my death.
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/6N6DX3JT4XQ7LOGCYM7WJCI3RYVW2VGV/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at
https://mail.python.org/archives/list/python-dev@python.org/message/ARKTYXZMYEW3JHNTR7RNLIDDIIOMU7B6/
Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Request for feedback on new features for 3.11
Hi there I hope you will indulge me while I request some feedback on a couple of open PRs that I would like to be considered for 3.11. https://github.com/python/cpython/pull/31150 https://github.com/python/cpython/pull/31142 The first one should be pretty easy to review as it simply makes use of the new co_qualname attribute of code objects to enrich the traceback output with qualified names. Provided it gets accepted, similar changes could also be introduced in other areas, like profiling, debugging, IDLE etc... The second one might not be as easy to review. As pointed out in the attached bpo, a similar issue was already considered back in 2012, but the proposed solution there has been "opposed" because of backwards-compatibility concerns. The change I am proposing would expose thread names via the ABI (albeit still in the form of a Python object) in a backwards-compatible way. However, I am not very satisfied with the way I have implemented this for now, nor can I think of a better solution. Cheers, Gabriele ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/EKSD6MXK2FZBV5FG6P2NENK6RJZ7HYQK/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Compiling of ast.Module in Python 3.10 and co_firstlineno behavior
Hi Fabio Does the actual function object get re-created as well during the recompilation process that you have described? Perhaps it might help to note that the __code__ attribute of a function object f can be mutated and that f is hashable? Cheers, Gabriele On Thu, 17 Feb 2022 at 19:33, Fabio Zadrozny wrote: > > > Em qui., 17 de fev. de 2022 às 16:05, Mark Shannon escreveu: >> >> Hi Fabio, >> >> This happened as part of implementing PEP 626. >> The previous behavior isn't very robust w.r.t doc strings and >> compiler optimizations. >> >> OOI, why would you want to revert to the old behavior? >> > > Hi Mark, > > The issue I'm facing is that ipython uses an approach of obtaining the ast > for a function to be executed and then it goes on node by node executing it. > > When running in the debugger, the debugger caches some information based on > (co_firstlineno, co_name, co_filename) to have information saved across > multiple calls to the same function, which works in general because each > function in a given python file would have its own co_firstlineno, but in > this specific case here it gets a single function and then recompiles it > expression by expression -- so, it'll have the same co_filename () and > the same co_name (), but then the co_firstlineno would be different > (because the statement resides in a different line), but with Python 3.10 > this assumption fails as even the co_firstlineno will be the same... > > You can see the actual issues at: > https://github.com/microsoft/vscode-jupyter/issues/8803 / > https://github.com/ipython/ipykernel/issues/841/ > https://github.com/microsoft/debugpy/issues/844 > > After thinkering a bit it seems it's possible to create a new code object > based on an existing code object with `code.replace` (re-assembling the > co_lnotab/co_firstlineno), so, I'm going to propose that as a fix to ipython, > but I found it really strange that this did change in Python 3.10 in the > first place as the old behavior seemed reasonable for me (i.e.: with the new > behavior it's a bit strange that the user is compiling something with a > single statement on line 99 and yet the resulting code object will have the > co_firstlineno == 1). > > -- note: I also couldn't find any mention of this in the changelog, so, I > thought this could've happened by mistake. > > Best regards, > > Fabio > ___ > Python-Dev mailing list -- python-dev@python.org > To unsubscribe send an email to python-dev-le...@python.org > https://mail.python.org/mailman3/lists/python-dev.python.org/ > Message archived at > https://mail.python.org/archives/list/python-dev@python.org/message/DVP4VK3BY4XDC6B6HSVPLJTPCQKISAPC/ > Code of Conduct: http://python.org/psf/codeofconduct/ -- "Egli è scritto in lingua matematica, e i caratteri son triangoli, cerchi, ed altre figure geometriche, senza i quali mezzi è impossibile a intenderne umanamente parola; senza questi è un aggirarsi vanamente per un oscuro laberinto." -- G. Galilei, Il saggiatore. ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/O5EEGEHE7G6UFTYO4UX7Y7QHZXA4ACYG/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: About the new CFrame structure
> That's already the case for 3.11 Ah, that's awesome news! Like with the rest, I'll wait and see what shape this ends up taking :). Cheers, Gab ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/7VJRYSKXWCQ67EUFJRKPJKEPZ7JB2A7F/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: About the new CFrame structure
Brandt, Guido, Pablo Many thanks for your helpful answers. Indeed I'm asking because I just finished working on some improvements to Austin and got back to looking into what was coming in order to add support for 3.11 (plus make use of some of the changes that I recently contributed, like PyThreadState.native_thread_id, Py_Version and code.co_qualname). Pablo's suggestion of waiting until 3.11 becomes more stable is a sensible one, but in the meantime I wanted to understand where the frame stack management is heading to at least have an idea of what's in store for Austin :). I'm very much satisfied by all the details in your replies as they answer all my questions, for now, so thanks a lot for that. A final thought: I have discussed some of the technical details of the recent improvements to Austin in a blog post ( https://p403n1x87.github.io/increasing-austin-accuracy-with-a-dobule-heap-trick.html). My experiments seem to suggest that, the less sparse the frame objects are in memory, the more accurate tools like Austin can be. So this makes me wonder if it would make sense for CPython to ensure frame objects are created in a contiguous block of memory (perhaps there could be benefits from the locality of reference, although it's not obvious to me why this would be the case at the moment). Best, Gabriele On Mon, 20 Dec 2021 at 20:16, Pablo Galindo Salgado wrote: > Hi Gabriele, > > In addition to what Guido and Brandt have already said, I can help to you > adapting Austin to 3.11 as I reviewed or authored some of these changes and > I have already been helping some projects do the relevant changes as well > as in my own tools. > > What you want to do si the following for unwinding: > > * Go from _PyRuntime -> PyThreadState -> CFrame -> current_frame > > This will lead you to s PyInterpreterFrame that you should use for > unwinding the entire thread stack. The difference is that cframe->previous > will skip you several frames as it points to the previous CFrame, but there > are a one to many relationships between CFrame and interpreter frames > because several python functions can now reuse the same evaluation loop. > > Also, I would recommend waiting until beta freeze to start adapting > anything as things can still massively change until then for 3.11. > > If you have any questions or you need help, feel free to ping me in GitHub > if you want. > > Regards from rainy London, > Pablo Galindo Salgado > > > > On Mon, 20 Dec 2021, 18:27 Gabriele, wrote: > >> Hi there >> >> I hope you would indulge me in asking for some details about the new >> CFrame structure, even in the form of existing literature (e.g. PEP) >> where the idea behind it is explained. >> >> Also, I'd like to a quick question, if I may. There now appear to be >> two ways of unwinding the frame stack: either iterate over >> CFrame.previous, or the more traditional PyFrameObject.f_back. I >> suspect there are reasons why these are perhaps not actually >> equivalent, and indeed this is mainly what I'd like to read in the >> literature I've requested above. >> >> Cheers, >> Gabriele >> >> -- >> "Egli è scritto in lingua matematica, e i caratteri son triangoli, >> cerchi, ed altre figure >> geometriche, senza i quali mezzi è impossibile a intenderne umanamente >> parola; >> senza questi è un aggirarsi vanamente per un oscuro laberinto." >> >> -- G. Galilei, Il saggiatore. >> ___ >> Python-Dev mailing list -- python-dev@python.org >> To unsubscribe send an email to python-dev-le...@python.org >> https://mail.python.org/mailman3/lists/python-dev.python.org/ >> Message archived at >> https://mail.python.org/archives/list/python-dev@python.org/message/KQOQTLR5IXMJXYZGPDHWR32I2Z53UVBL/ >> Code of Conduct: http://python.org/psf/codeofconduct/ >> > -- *"Egli è scritto in lingua matematica, e i caratteri son triangoli, cerchi, ed altre figuregeometriche, senza i quali mezzi è impossibile a intenderne umanamente parola;senza questi è un aggirarsi vanamente per un oscuro laberinto."* *-- *G. Galilei*, Il saggiatore.* ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/YBAMU2YESAL66DB7CPG6JXJWQMFO2HNN/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] About the new CFrame structure
Hi there I hope you would indulge me in asking for some details about the new CFrame structure, even in the form of existing literature (e.g. PEP) where the idea behind it is explained. Also, I'd like to a quick question, if I may. There now appear to be two ways of unwinding the frame stack: either iterate over CFrame.previous, or the more traditional PyFrameObject.f_back. I suspect there are reasons why these are perhaps not actually equivalent, and indeed this is mainly what I'd like to read in the literature I've requested above. Cheers, Gabriele -- "Egli è scritto in lingua matematica, e i caratteri son triangoli, cerchi, ed altre figure geometriche, senza i quali mezzi è impossibile a intenderne umanamente parola; senza questi è un aggirarsi vanamente per un oscuro laberinto." -- G. Galilei, Il saggiatore. ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/KQOQTLR5IXMJXYZGPDHWR32I2Z53UVBL/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Question regarding the value of PyThreadState.thread_id
Hi Victor
Thanks for your reply. Indeed, upon further investigation, I have realised
changing the value of thread_id is not wise as there are other methods that
expect this field to be a pthread_t. I have opened issue
https://bugs.python.org/issue43879https://bugs.python.org/issue43879 to
have the new field added. I'm happy to do the work myself if it gets
accepted.
Cheers,
Gabriele
On Sat, 17 Apr 2021, 11:07 Victor Stinner, wrote:
> Hi,
>
> There are two reasons:
>
> * PyThread_get_thread_native_id() was only added recently (Python 3.8)
> * PyThread_get_thread_native_id() is not portable and not available on
> all platforms: the function is only declared if the
> PY_HAVE_THREAD_NATIVE_ID macro is defined.
>
> Maybe PyThreadState.thread_id could use
> PyThread_get_thread_native_id() if avaialble, or
> PyThread_get_thread_ident() otherwise. But that sounds like an
> incompatible change.
>
> Another approach would be to add a *new* structure member, like
> thread_native_id. It would not exist if the PY_HAVE_THREAD_NATIVE_ID
> macro is not defined.
>
>
> Include/pythread.h:
>
> #if defined(__APPLE__) || defined(__linux__) || defined(__FreeBSD__)
> || defined(__OpenBSD__) || defined(__NetBSD__) || defined(_WIN32) ||
> defined(_AIX)
> #define PY_HAVE_THREAD_NATIVE_ID
> PyAPI_FUNC(unsigned long) PyThread_get_thread_native_id(void);
> #endif
>
>
> thread_pthread.h implementation:
>
> unsigned long
> PyThread_get_thread_native_id(void)
> {
> if (!initialized)
> PyThread_init_thread();
> #ifdef __APPLE__
> uint64_t native_id;
> (void) pthread_threadid_np(NULL, &native_id);
> #elif defined(__linux__)
> pid_t native_id;
> native_id = syscall(SYS_gettid);
> #elif defined(__FreeBSD__)
> int native_id;
> native_id = pthread_getthreadid_np();
> #elif defined(__OpenBSD__)
> pid_t native_id;
> native_id = getthrid();
> #elif defined(_AIX)
> tid_t native_id;
> native_id = thread_self();
> #elif defined(__NetBSD__)
> lwpid_t native_id;
> native_id = _lwp_self();
> #endif
> return (unsigned long) native_id;
> }
>
> Victor
> --
> Night gathers, and now my watch begins. It shall not end until my death.
>
>
> On Fri, Apr 16, 2021 at 10:47 PM Gabriele wrote:
> >
> > Hi all.
> >
> > My apologies if this is a topic that's been discussed already, but I
> > wasn't able to locate anything in the archive on the subject. I was
> > wondering if there's a fundamental reason for using
> > PyThread_get_thread_ident instead of PyThread_get_thread_native_id for
> > the value of the thread_id field of the PyThreadState object.
> >
> > The reason why I'm asking is that I would like to have easy access to
> > the native TID on Linux to identify the /proc/stat file associated
> > with the thread from an external process. At the moment I have to
> > resort to calling process_vm_readv to copy the struct pthread over to
> > local VM and then guess where the tid field might be. So, if there's
> > no fundamental reason for thread_id to be PyThread_get_thread_ident, I
> > would like to propose to change it to PyThread_get_thread_native_id
> > instead.
> >
> > Thanks,
> > Gabriele
> >
> > --
> > "Egli è scritto in lingua matematica, e i caratteri son triangoli,
> > cerchi, ed altre figure
> > geometriche, senza i quali mezzi è impossibile a intenderne umanamente
> parola;
> > senza questi è un aggirarsi vanamente per un oscuro laberinto."
> >
> > -- G. Galilei, Il saggiatore.
> > ___
> > Python-Dev mailing list -- python-dev@python.org
> > To unsubscribe send an email to python-dev-le...@python.org
> > https://mail.python.org/mailman3/lists/python-dev.python.org/
> > Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/3DOB6VUTAIJKK4SUGBYWL4QPWI2E5Q2T/
> > Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at
https://mail.python.org/archives/list/python-dev@python.org/message/ZSXLNWY3HIWYEHV7Z2UKHHSUEVTKYBQY/
Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Question regarding the value of PyThreadState.thread_id
Hi all. My apologies if this is a topic that's been discussed already, but I wasn't able to locate anything in the archive on the subject. I was wondering if there's a fundamental reason for using PyThread_get_thread_ident instead of PyThread_get_thread_native_id for the value of the thread_id field of the PyThreadState object. The reason why I'm asking is that I would like to have easy access to the native TID on Linux to identify the /proc/stat file associated with the thread from an external process. At the moment I have to resort to calling process_vm_readv to copy the struct pthread over to local VM and then guess where the tid field might be. So, if there's no fundamental reason for thread_id to be PyThread_get_thread_ident, I would like to propose to change it to PyThread_get_thread_native_id instead. Thanks, Gabriele -- "Egli è scritto in lingua matematica, e i caratteri son triangoli, cerchi, ed altre figure geometriche, senza i quali mezzi è impossibile a intenderne umanamente parola; senza questi è un aggirarsi vanamente per un oscuro laberinto." -- G. Galilei, Il saggiatore. ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/3DOB6VUTAIJKK4SUGBYWL4QPWI2E5Q2T/ Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-Dev] What is the purpose of the _PyThreadState_Current symbol in Python 3?
Ah ok, this might be related to Victor's observation based on the latest sources. I haven't tested 3.7 yet, but if _PyRuntime is available from dynsym then this is already enough. Thanks, Gabriele On Sat, 29 Sep 2018 at 11:00, Nathaniel Smith wrote: > > On Fri, Sep 28, 2018 at 3:29 PM, Gabriele wrote: > > On Fri, 28 Sep 2018 at 23:12, Nathaniel Smith wrote: > >> What information do you wish the interpreter provided, that would make > >> your program simpler and more reliable? > > > > An exported global variable that points to the head of the > > PyInterpreterState linked list (i.e. the return value of > > PyInterpreterState_Head). This way my program could just look this up > > from the dynsym section instead of scanning a dump of the bss section > > in memory to find a possible candidate. > > Hmm, it looks like in 3.7+, _PyRuntime is marked PyAPI_DATA, which I > think should make it exported from dynsym? > > https://github.com/python/cpython/blob/4b430e5f6954ef4b248e95bfb4087635dcdefc6d/Include/internal/pystate.h#L206 > > And PyInterpreterState_Head is just _PyRuntime.interpreters.head. So > maybe this is already done... > > -n > > -- > Nathaniel J. Smith -- https://vorpus.org -- "Egli è scritto in lingua matematica, e i caratteri son triangoli, cerchi, ed altre figure geometriche, senza i quali mezzi è impossibile a intenderne umanamente parola; senza questi è un aggirarsi vanamente per un oscuro laberinto." -- G. Galilei, Il saggiatore. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] What is the purpose of the _PyThreadState_Current symbol in Python 3?
On Fri, 28 Sep 2018 at 23:12, Nathaniel Smith wrote: > What information do you wish the interpreter provided, that would make your > program simpler and more reliable? An exported global variable that points to the head of the PyInterpreterState linked list (i.e. the return value of PyInterpreterState_Head). This way my program could just look this up from the dynsym section instead of scanning a dump of the bss section in memory to find a possible candidate. It would be grand if also the string in the rodata section that gives the Python version could be dereferenced from dynsym, but that's a different question. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] What is the purpose of the _PyThreadState_Current symbol in Python 3?
Hi Victor, > I understand that you are writing a debugger and you can only *read* > modify, not execute code, right? I'm working on a frame stack sampler that runs independently from the Python process. The project is "Austin" (https://github.com/P403n1x87/austin). Whilst I could, in principle, execute code with other system calls, I prefer not to in this case. > In the master branch, it's now _PyRuntime.gilstate.tstate_current. If > you run time.sleep(3600) and look into > _PyRuntime.gilstate.tstate_current using gdb, you can a NULL pointer > (tstate_current=0) because Python releases the GIL.. I would like my application to make as few assumptions as possible. The _PyRuntime symbol might not be available if all the symbols have been stripped out of the binaries. That's why I was trying to rely on _PyThreadState_Current, which is in the .dynsym section. Judging by the output of nm -D `which python3` (I'm on Python 3.6.6 at the moment) I cannot see anything more useful than that. My current strategy is to try and make something out of this symbol and then fall back to a brute force approach to scan the .bss section for valid PyInterpreterState instances (which works reliably well and is quite fast too, but a bit ugly). > There is also _PyGILState_GetInterpreterStateUnsafe() which gives > access to the current Python interpreter: > _PyRuntime.gilstate.autoInterpreterState. From the interpreter, you > can use the linked list of thread states from interp->tstate_head. > > I hope that I helped :-) Yes thanks! Your comment made me realise why I can use PyThreadState_Current at the very beginning, and it is because Python is going through the intensive startup process, which involves, among other things, the loading of frozen modules (I can clearly see most if not all the steps in the output of Austin, as mentioned in the repo's README). During this phase, the main (and only thread) holds the GIL and is quite busy doing stuff. The long-running applications that I was trying to attach to have very long wait periods where they sit idle waiting for a timer to trigger the next operations, that fire very quickly and put the threads back to sleep again. If this is what the _PyThreadState_Current is designed for, then I guess I cannot really rely on it, especially when attaching Austin to another process. Best regards, Gabriele ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] What is the purpose of the _PyThreadState_Current symbol in Python 3?
In trying to find the location of a valid instance of PyInterpreterState in the virtual memory of a running Python (3.6) application (using process_vm_read on Linux), I have noticed that I can only rely on _PyThreadState_Current.interp at the very beginning of the execution. If I try to attach to a running Python process, then _PythreadState_Current.interp doesn't seem to point to anything useful to derive the currently running threads and the frame stacks for each of them. This makes me wonder about the purpose of this symbol in the .dynsym section. Apart from a brute force approach for finding a valid PyInterpreterState, is there a more reliable approach for the version of Python that I'm targeting? Thanks, Gabriele ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Doc-SIG] The docs, reloaded
On 5/19/07, Georg Brandl <[EMAIL PROTECTED]> wrote: > > [snip] > > Waiting for comments! Awesome, Georg! Wow. Nice work. Seems like this has been a long time comin', and I bet others have been working away "in secret" on similar projects. I hope you keep running with it until it gets hijacked into being the "official" versions. :) I'm bookmarking it as "python docs" in my browser. BTW, would like to see a little blurb of your own on that page about how the docs were converted, rendered, and their new source format. Thanks much, ---John P.S. -- funny sig, btw. :) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
