Script 'mail_helper' called by obssrc
Hello community,
here is the log from the commit of package python-greenlet for openSUSE:Factory
checked in at 2021-10-20 20:23:26
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Comparing /work/SRC/openSUSE:Factory/python-greenlet (Old)
and /work/SRC/openSUSE:Factory/.python-greenlet.new.1890 (New)
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Package is "python-greenlet"
Wed Oct 20 20:23:26 2021 rev:38 rq:925731 version:1.1.2
Changes:
--------
--- /work/SRC/openSUSE:Factory/python-greenlet/python-greenlet.changes
2021-09-03 21:26:48.274216348 +0200
+++
/work/SRC/openSUSE:Factory/.python-greenlet.new.1890/python-greenlet.changes
2021-10-20 20:24:11.245374487 +0200
@@ -1,0 +2,19 @@
+Sat Oct 16 19:07:41 UTC 2021 - Dirk M??ller <dmuel...@suse.com>
+
+- update to 1.1.2:
+ - Fix a potential crash due to a reference counting error when Python
+ subclasses of ``greenlet.greenlet`` were deallocated. The crash
+ became more common on Python 3.10; on earlier versions, silent
+ memory corruption could result.
+ - Fix a leak of a list object when the last reference to a greenlet
+ was deleted from some other thread than the one to which it
+ belonged. For this to work correctly, you must call a greenlet API
+ like ``getcurrent()`` before the thread owning the greenlet exits:
+ this is a long-standing limitation that can also lead to the leak of
+ a thread's main greenlet if not called; we hope to lift this
+ limitation. Note that in some cases this may also fix leaks of
+ greenlet objects themselves. See `issue 251
+ - Python 3.10: Tracing or profiling into a spawned greenlet didn't
+ work as expected. See `issue 256
+
+-------------------------------------------------------------------
Old:
----
greenlet-1.1.0.tar.gz
New:
----
greenlet-1.1.2.tar.gz
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Other differences:
------------------
++++++ python-greenlet.spec ++++++
--- /var/tmp/diff_new_pack.hnmjPy/_old 2021-10-20 20:24:11.709374774 +0200
+++ /var/tmp/diff_new_pack.hnmjPy/_new 2021-10-20 20:24:11.713374776 +0200
@@ -19,7 +19,7 @@
%{?!python_module:%define python_module() python-%{**} python3-%{**}}
Name: python-greenlet
-Version: 1.1.0
+Version: 1.1.2
Release: 0
Summary: Lightweight in-process concurrent programming
License: MIT
++++++ greenlet-1.1.0.tar.gz -> greenlet-1.1.2.tar.gz ++++++
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn'
'--exclude=.svnignore' old/greenlet-1.1.0/.github/workflows/tests.yml
new/greenlet-1.1.2/.github/workflows/tests.yml
--- old/greenlet-1.1.0/.github/workflows/tests.yml 2021-05-06
16:53:06.000000000 +0200
+++ new/greenlet-1.1.2/.github/workflows/tests.yml 2021-09-29
12:35:47.000000000 +0200
@@ -7,6 +7,7 @@
ZOPE_INTERFACE_STRICT_IRO: 1
PYTHONUNBUFFERED: 1
PYTHONDONTWRITEBYTECODE: 1
+ PYTHONDEVMODE: 1
PIP_UPGRADE_STRATEGY: eager
# Don't get warnings about Python 2 support being deprecated. We
# know. The env var works for pip 20.
@@ -23,7 +24,7 @@
runs-on: ${{ matrix.os }}
strategy:
matrix:
- python-version: [2.7, 3.5, 3.6, 3.7, 3.8, 3.9, 3.10.0-beta.1]
+ python-version: [2.7, 3.5, 3.6, 3.7, 3.8, 3.9, 3.10.0-rc.1]
os: [ubuntu-latest, macos-latest]
steps:
- uses: actions/checkout@v2
@@ -58,6 +59,11 @@
- name: Test
run: |
python -m unittest discover -v greenlet.tests
+ - name: Doctest
+ # FIXME: This conditional can go away when a Sphinx greater than 4.1.2
+ # is released. See https://github.com/sphinx-doc/sphinx/issues/9512
+ if: matrix.python-version != '3.10.0-rc.1'
+ run: |
sphinx-build -b doctest -d docs/_build/doctrees2 docs
docs/_build/doctest2
- name: Publish package to PyPI (mac)
# We cannot 'uses: pypa/gh-action-pypi-publish@v1.4.1' because
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn'
'--exclude=.svnignore' old/greenlet-1.1.0/CHANGES.rst
new/greenlet-1.1.2/CHANGES.rst
--- old/greenlet-1.1.0/CHANGES.rst 2021-05-06 16:53:06.000000000 +0200
+++ new/greenlet-1.1.2/CHANGES.rst 2021-09-29 12:35:47.000000000 +0200
@@ -2,6 +2,38 @@
Changes
=========
+1.1.2 (2021-09-29)
+==================
+
+- Fix a potential crash due to a reference counting error when Python
+ subclasses of ``greenlet.greenlet`` were deallocated. The crash
+ became more common on Python 3.10; on earlier versions, silent
+ memory corruption could result. See `issue 245
+ <https://github.com/python-greenlet/greenlet/issues/245>`_. Patch by
+ fygao-wish.
+- Fix a leak of a list object when the last reference to a greenlet
+ was deleted from some other thread than the one to which it
+ belonged. For this to work correctly, you must call a greenlet API
+ like ``getcurrent()`` before the thread owning the greenlet exits:
+ this is a long-standing limitation that can also lead to the leak of
+ a thread's main greenlet if not called; we hope to lift this
+ limitation. Note that in some cases this may also fix leaks of
+ greenlet objects themselves. See `issue 251
+ <https://github.com/python-greenlet/greenlet/issues/251>`_.
+- Python 3.10: Tracing or profiling into a spawned greenlet didn't
+ work as expected. See `issue 256
+ <https://github.com/python-greenlet/greenlet/issues/256>`_, reported
+ by Joe Rickerby.
+
+1.1.1 (2021-08-06)
+==================
+
+- Provide Windows binary wheels for Python 3.10 (64-bit only).
+
+- Update Python 3.10 wheels to be built against 3.10rc1, where
+ applicable.
+
+
1.1.0 (2021-05-06)
==================
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn'
'--exclude=.svnignore' old/greenlet-1.1.0/PKG-INFO new/greenlet-1.1.2/PKG-INFO
--- old/greenlet-1.1.0/PKG-INFO 2021-05-06 16:53:07.788602400 +0200
+++ new/greenlet-1.1.2/PKG-INFO 2021-09-29 12:35:47.783776500 +0200
@@ -1,8 +1,12 @@
Metadata-Version: 2.1
Name: greenlet
-Version: 1.1.0
+Version: 1.1.2
Summary: Lightweight in-process concurrent programming
Home-page: https://greenlet.readthedocs.io/
+Author: Alexey Borzenkov
+Author-email: sna...@gmail.com
+Maintainer: Jason Madden
+Maintainer-email: ja...@nextthought.com
License: MIT License
Project-URL: Bug Tracker, https://github.com/python-greenlet/greenlet/issues
Project-URL: Source Code, https://github.com/python-greenlet/greenlet/
@@ -69,7 +73,9 @@
Documentation is available on readthedocs.org:
https://greenlet.readthedocs.io
+Keywords: greenlet coroutine concurrency threads cooperative
Platform: any
+Classifier: Development Status :: 5 - Production/Stable
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Natural Language :: English
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn'
'--exclude=.svnignore' old/greenlet-1.1.0/appveyor.yml
new/greenlet-1.1.2/appveyor.yml
--- old/greenlet-1.1.0/appveyor.yml 2021-05-06 16:53:06.000000000 +0200
+++ new/greenlet-1.1.2/appveyor.yml 2021-09-29 12:35:47.000000000 +0200
@@ -19,6 +19,7 @@
CMD_IN_ENV: "cmd /E:ON /V:ON /C .\\appveyor\\run_with_env.cmd"
# Use a fixed hash seed for reproducability
PYTHONHASHSEED: 8675309
+ PYTHONDEVMODE: 1
# Don't get warnings about Python 2 support being deprecated. We
# know.
PIP_NO_PYTHON_VERSION_WARNING: 1
@@ -33,6 +34,11 @@
matrix:
# http://www.appveyor.com/docs/installed-software#python
+ - PYTHON: "C:\\Python310-x64"
+ PYTHON_VERSION: "3.10.0rc2"
+ PYTHON_ARCH: "64"
+ PYTHON_EXE: python
+ APPVEYOR_BUILD_WORKER_IMAGE: Visual Studio 2019
- PYTHON: "C:\\Python39-x64"
PYTHON_ARCH: "64"
@@ -161,7 +167,8 @@
test_script:
- "%CMD_IN_ENV% python -m unittest discover -v greenlet.tests"
- - "%CMD_IN_ENV% python -m sphinx -b doctest -d docs/_build/doctrees docs
docs/_build/doctest"
+# XXX: Doctest disabled pending sphinx release for 3.10; see tests.yml.
+# - "%CMD_IN_ENV% python -m sphinx -b doctest -d docs/_build/doctrees docs
docs/_build/doctest"
after_test:
- "%CMD_IN_ENV% python setup.py bdist_wheel"
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn'
'--exclude=.svnignore' old/greenlet-1.1.0/setup.py new/greenlet-1.1.2/setup.py
--- old/greenlet-1.1.0/setup.py 2021-05-06 16:53:06.000000000 +0200
+++ new/greenlet-1.1.2/setup.py 2021-09-29 12:35:47.000000000 +0200
@@ -112,6 +112,11 @@
long_description=readfile("README.rst"),
long_description_content_type="text/x-rst",
url="https://greenlet.readthedocs.io/";,
+ keywords="greenlet coroutine concurrency threads cooperative",
+ author="Alexey Borzenkov",
+ author_email="sna...@gmail.com",
+ maintainer='Jason Madden',
+ maintainer_email='ja...@nextthought.com',
project_urls={
'Bug Tracker': 'https://github.com/python-greenlet/greenlet/issues',
'Source Code': 'https://github.com/python-greenlet/greenlet/',
@@ -125,6 +130,7 @@
headers=headers,
ext_modules=ext_modules,
classifiers=[
+ "Development Status :: 5 - Production/Stable",
'Intended Audience :: Developers',
'License :: OSI Approved :: MIT License',
'Natural Language :: English',
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn'
'--exclude=.svnignore' old/greenlet-1.1.0/src/greenlet/__init__.py
new/greenlet-1.1.2/src/greenlet/__init__.py
--- old/greenlet-1.1.0/src/greenlet/__init__.py 2021-05-06 16:53:06.000000000
+0200
+++ new/greenlet-1.1.2/src/greenlet/__init__.py 2021-09-29 12:35:47.000000000
+0200
@@ -25,7 +25,7 @@
###
# Metadata
###
-__version__ = '1.1.0'
+__version__ = '1.1.2'
from ._greenlet import _C_API # pylint:disable=no-name-in-module
###
@@ -48,7 +48,8 @@
from ._greenlet import settrace
except ImportError:
# Tracing wasn't supported.
- # TODO: Remove the option to disable it.
+ # XXX: The option to disable it was removed in 1.0,
+ # so this branch should be dead code.
pass
###
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn'
'--exclude=.svnignore' old/greenlet-1.1.0/src/greenlet/greenlet.c
new/greenlet-1.1.2/src/greenlet/greenlet.c
--- old/greenlet-1.1.0/src/greenlet/greenlet.c 2021-05-06 16:53:06.000000000
+0200
+++ new/greenlet-1.1.2/src/greenlet/greenlet.c 2021-09-29 12:35:47.000000000
+0200
@@ -136,6 +136,11 @@
static PyObject* volatile ts_passaround_args = NULL;
static PyObject* volatile ts_passaround_kwargs = NULL;
+/* Used internally in ``g_switchstack()`` */
+#if GREENLET_USE_CFRAME
+static int volatile ts__g_switchstack_use_tracing = 0;
+#endif
+
/***********************************************************/
/* Thread-aware routines, switching global variables when needed */
@@ -195,6 +200,7 @@
}
gmain->stack_start = (char*)1;
gmain->stack_stop = (char*)-1;
+ /* GetDict() returns a borrowed reference. Make it strong. */
gmain->run_info = dict;
Py_INCREF(dict);
return gmain;
@@ -254,6 +260,11 @@
it stores them in the thread dict; delete them now. */
deleteme = PyDict_GetItem(tstate->dict, ts_delkey);
if (deleteme != NULL) {
+ /* The only reference to these greenlets should be in this list, so
+ clearing the list should let them be deleted again, triggering
+ calls to green_dealloc() in the correct thread. This may run
+ arbitrary Python code?
+ */
PyList_SetSlice(deleteme, 0, INT_MAX, NULL);
}
@@ -267,7 +278,6 @@
/* release an extra reference */
Py_DECREF(current);
-
/* restore current exception */
PyErr_Restore(exc, val, tb);
@@ -276,7 +286,6 @@
if (ts_current->run_info != tstate->dict) {
goto green_updatecurrent_restart;
}
-
return 0;
}
@@ -481,26 +490,37 @@
return 0;
}
+/**
+ Perform a stack switch according to some global variables
+ that must be set before calling this function. Those variables
+ are:
+
+ - ts_current: current greenlet (holds a reference)
+ - ts_target: greenlet to switch to (weak reference)
+ - ts_passaround_args: NULL if PyErr_Occurred(),
+ else a tuple of args sent to ts_target (holds a reference)
+ - ts_passaround_kwargs: switch kwargs (holds a reference)
+
+ Because the stack switch happens in this function, this function can't use
+ its own stack (local) variables, set before the switch, and then accessed
after the
+ switch. Global variables beginning with ``ts__g_switchstack`` are used
+ internally instead.
+
+ On return results are passed via global variables as well:
+
+ - ts_origin: originating greenlet (holds a reference)
+ - ts_current: current greenlet (holds a reference)
+ - ts_passaround_args: NULL if PyErr_Occurred(),
+ else a tuple of args sent to ts_current (holds a reference)
+ - ts_passaround_kwargs: switch kwargs (holds a reference)
+
+ It is very important that stack switch is 'atomic', i.e. no
+ calls into other Python code allowed (except very few that
+ are safe), because global variables are very fragile.
+*/
static int
g_switchstack(void)
{
- /* Perform a stack switch according to some global variables
- that must be set before:
- - ts_current: current greenlet (holds a reference)
- - ts_target: greenlet to switch to (weak reference)
- - ts_passaround_args: NULL if PyErr_Occurred(),
- else a tuple of args sent to ts_target (holds a reference)
- - ts_passaround_kwargs: switch kwargs (holds a reference)
- On return results are passed via global variables as well:
- - ts_origin: originating greenlet (holds a reference)
- - ts_current: current greenlet (holds a reference)
- - ts_passaround_args: NULL if PyErr_Occurred(),
- else a tuple of args sent to ts_current (holds a reference)
- - ts_passaround_kwargs: switch kwargs (holds a reference)
- It is very important that stack switch is 'atomic', i.e. no
- calls into other Python code allowed (except very few that
- are safe), because global variables are very fragile.
- */
int err;
{ /* save state */
PyGreenlet* current = ts_current;
@@ -519,10 +539,23 @@
current->exc_traceback = tstate->exc_traceback;
#endif
#if GREENLET_USE_CFRAME
+ /*
+ IMPORTANT: ``cframe`` is a pointer into the STACK.
+ Thus, because the call to ``slp_switch()``
+ changes the contents of the stack, you cannot read from
+ ``ts_current->cframe`` after that call and necessarily
+ get the same values you get from reading it here. Anything
+ you need to restore from now to then must be saved
+ in a global variable (because we can't use stack variables
+ here either).
+ */
current->cframe = tstate->cframe;
+ ts__g_switchstack_use_tracing = tstate->cframe->use_tracing;
#endif
}
+
err = slp_switch();
+
if (err < 0) { /* error */
PyGreenlet* current = ts_current;
current->top_frame = NULL;
@@ -566,6 +599,13 @@
#if GREENLET_USE_CFRAME
tstate->cframe = target->cframe;
+ /*
+ If we were tracing, we need to keep tracing.
+ There should never be the possibility of hitting the
+ root_cframe here. See note above about why we can't
+ just copy this from ``origin->cframe->use_tracing``.
+ */
+ tstate->cframe->use_tracing = ts__g_switchstack_use_tracing;
#endif
assert(ts_origin == NULL);
@@ -678,10 +718,8 @@
PyObject* tracefunc;
origin = ts_origin;
ts_origin = NULL;
-
current = ts_current;
- if ((tracefunc = PyDict_GetItem(current->run_info, ts_tracekey)) !=
- NULL) {
+ if ((tracefunc = PyDict_GetItem(current->run_info, ts_tracekey)) !=
NULL) {
Py_INCREF(tracefunc);
if (g_calltrace(tracefunc,
args ? ts_event_switch : ts_event_throw,
@@ -765,7 +803,15 @@
PyGreenlet* self = ts_target;
PyObject* args = ts_passaround_args;
PyObject* kwargs = ts_passaround_kwargs;
-
+#if GREENLET_USE_CFRAME
+ /*
+ See green_new(). This is a stack-allocated variable used
+ while *self* is in PyObject_Call().
+ We want to defer copying the state info until we're sure
+ we need it and are in a stable place to do so.
+ */
+ CFrame trace_info;
+#endif
/* save exception in case getattr clears it */
PyErr_Fetch(&exc, &val, &tb);
/* self.run is the object to call in the new greenlet */
@@ -806,6 +852,17 @@
return 1;
}
+#if GREENLET_USE_CFRAME
+ /* OK, we need it, we're about to switch greenlets, save the state. */
+ trace_info = *PyThreadState_GET()->cframe;
+ /* Make the target greenlet refer to the stack value. */
+ self->cframe = &trace_info;
+ /*
+ And restore the link to the previous frame so this one gets
+ unliked appropriately.
+ */
+ self->cframe->previous = &PyThreadState_GET()->root_cframe;
+#endif
/* start the greenlet */
self->stack_start = NULL;
self->stack_stop = (char*)mark;
@@ -829,8 +886,8 @@
err = g_switchstack();
/* returns twice!
- The 1st time with err=1: we are in the new greenlet
- The 2nd time with err=0: back in the caller's greenlet
+ The 1st time with ``err == 1``: we are in the new greenlet
+ The 2nd time with ``err <= 0``: back in the caller's greenlet
*/
if (err == 1) {
/* in the new greenlet */
@@ -850,8 +907,7 @@
Py_INCREF(self->run_info);
Py_XDECREF(o);
- if ((tracefunc = PyDict_GetItem(self->run_info, ts_tracekey)) !=
- NULL) {
+ if ((tracefunc = PyDict_GetItem(self->run_info, ts_tracekey)) != NULL)
{
Py_INCREF(tracefunc);
if (g_calltrace(tracefunc,
args ? ts_event_switch : ts_event_throw,
@@ -918,6 +974,47 @@
Py_INCREF(ts_current);
((PyGreenlet*)o)->parent = ts_current;
#if GREENLET_USE_CFRAME
+ /*
+ The PyThreadState->cframe pointer usually points to memory on the
+ stack, alloceted in a call into PyEval_EvalFrameDefault.
+
+ Initially, before any evaluation begins, it points to the initial
+ PyThreadState object's ``root_cframe`` object, which is statically
+ allocated for the lifetime of the thread.
+
+ A greenlet can last for longer than a call to
+ PyEval_EvalFrameDefault, so we can't set its ``cframe`` pointer to
+ be the current ``PyThreadState->cframe``; nor could we use one from
+ the greenlet parent for the same reason. Yet a further no: we can't
+ allocate one scoped to the greenlet and then destroy it when the
+ greenlet is deallocated, because inside the interpreter the CFrame
+ objects form a linked list, and that too can result in accessing
+ memory beyond its dynamic lifetime (if the greenlet doesn't actually
+ finish before it dies, its entry could still be in the list).
+
+ Using the ``root_cframe`` is problematic, though, because its
+ members are never modified by the interpreter and are set to 0,
+ meaning that its ``use_tracing`` flag is never updated. We don't
+ want to modify that value in the ``root_cframe`` ourself: it
+ *shouldn't* matter much because we should probably never get back to
+ the point where that's the only cframe on the stack; even if it did
+ matter, the major consequence of an incorrect value for
+ ``use_tracing`` is that if its true the interpreter does some extra
+ work --- however, it's just good code hygiene.
+
+ Our solution: before a greenlet runs, after its initial creation,
+ it uses the ``root_cframe`` just to have something to put there.
+ However, once the greenlet is actually switched to for the first
+ time, ``g_initialstub`` (which doesn't actually "return" while the
+ greenlet is running) stores a new CFrame on its local stack, and
+ copies the appropriate values from the currently running CFrame;
+ this is then made the CFrame for the newly-minted greenlet.
+ ``g_initialstub`` then proceeds to call ``glet.run()``, which
+ results in ``PyEval_...`` adding the CFrame to the list. Switches
+ continue as normal. Finally, when the greenlet finishes, the call to
+ ``glet.run()`` returns and the CFrame is taken out of the linked
+ list and the stack value is now unused and free to expire.
+ */
((PyGreenlet*)o)->cframe = &PyThreadState_GET()->root_cframe;
#endif
}
@@ -988,10 +1085,16 @@
lst = PyDict_GetItem(self->run_info, ts_delkey);
if (lst == NULL) {
lst = PyList_New(0);
- if (lst == NULL ||
- PyDict_SetItem(self->run_info, ts_delkey, lst) < 0) {
+ if (lst == NULL
+ || PyDict_SetItem(self->run_info, ts_delkey, lst) < 0) {
return -1;
}
+ /* PyDict_SetItem now holds a strong reference. PyList_New also
+ returned a fresh reference. We need to DECREF it now and let
+ the dictionary keep sole ownership. Frow now on, we're working
+ with a borrowed reference that will go away when the thread
+ dies. */
+ Py_DECREF(lst);
}
if (PyList_Append(lst, (PyObject*)self) < 0) {
return -1;
@@ -1116,6 +1219,20 @@
/* Resurrected! */
_Py_NewReference((PyObject*)self);
Py_SET_REFCNT(self, refcnt);
+ /* Better to use tp_finalizer slot (PEP 442)
+ * and call ``PyObject_CallFinalizerFromDealloc``,
+ * but that's only supported in Python 3.4+; see
+ * Modules/_io/iobase.c for an example.
+ *
+ * The following approach is copied from iobase.c in CPython 2.7.
+ * (along with much of this function in general). Here's their
+ * comment:
+ *
+ * When called from a heap type's dealloc, the type will be
+ * decref'ed on return (see e.g. subtype_dealloc in typeobject.c).
*/
+ if (PyType_HasFeature(Py_TYPE(self), Py_TPFLAGS_HEAPTYPE)) {
+ Py_INCREF(Py_TYPE(self));
+ }
PyObject_GC_Track((PyObject*)self);
@@ -1567,6 +1684,13 @@
#endif
if (_green_not_dead(self)) {
+ /* XXX: The otid= is almost useless becasue you can't correlate it to
+ any thread identifier exposed to Python. We could use
+ PyThreadState_GET()->thread_id, but we'd need to save that in the
+ greenlet, or save the whole PyThreadState object itself.
+
+ As it stands, its only useful for identifying greenlets from the same
thread.
+ */
result = GNative_FromFormat(
"<%s object at %p (otid=%p)%s%s%s%s>",
Py_TYPE(self)->tp_name,
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn'
'--exclude=.svnignore' old/greenlet-1.1.0/src/greenlet/tests/test_greenlet.py
new/greenlet-1.1.2/src/greenlet/tests/test_greenlet.py
--- old/greenlet-1.1.0/src/greenlet/tests/test_greenlet.py 2021-05-06
16:53:06.000000000 +0200
+++ new/greenlet-1.1.2/src/greenlet/tests/test_greenlet.py 2021-09-29
12:35:47.000000000 +0200
@@ -7,6 +7,8 @@
from greenlet import greenlet
+# We manually manage locks in many tests
+# pylint:disable=consider-using-with
class SomeError(Exception):
pass
@@ -30,7 +32,7 @@
g1.switch(exc)
-class GreenletTests(unittest.TestCase):
+class TestGreenlet(unittest.TestCase):
def test_simple(self):
lst = []
@@ -549,6 +551,105 @@
if attempt():
break
+ def test_issue_245_reference_counting_subclass_no_threads(self):
+ # https://github.com/python-greenlet/greenlet/issues/245
+ # Before the fix, this crashed pretty reliably on
+ # Python 3.10, at least on macOS; but much less reliably on other
+ # interpreters (memory layout must have changed).
+ # The threaded test crashed more reliably on more interpreters.
+ from greenlet import getcurrent
+ from greenlet import GreenletExit
+
+ class Greenlet(greenlet):
+ pass
+
+ initial_refs = sys.getrefcount(Greenlet)
+ # This has to be an instance variable because
+ # Python 2 raises a SyntaxError if we delete a local
+ # variable referenced in an inner scope.
+ self.glets = [] # pylint:disable=attribute-defined-outside-init
+
+ def greenlet_main():
+ try:
+ getcurrent().parent.switch()
+ except GreenletExit:
+ self.glets.append(getcurrent())
+
+ # Before the
+ for _ in range(10):
+ Greenlet(greenlet_main).switch()
+
+ del self.glets
+ self.assertEqual(sys.getrefcount(Greenlet), initial_refs)
+
+ def test_issue_245_reference_counting_subclass_threads(self):
+ # https://github.com/python-greenlet/greenlet/issues/245
+ from threading import Thread
+ from threading import Event
+
+ from greenlet import getcurrent
+
+ class MyGreenlet(greenlet):
+ pass
+
+ glets = []
+ ref_cleared = Event()
+
+ def greenlet_main():
+ getcurrent().parent.switch()
+
+ def thread_main(greenlet_running_event):
+ mine = MyGreenlet(greenlet_main)
+ glets.append(mine)
+ # The greenlets being deleted must be active
+ mine.switch()
+ # Don't keep any reference to it in this thread
+ del mine
+ # Let main know we published our greenlet.
+ greenlet_running_event.set()
+ # Wait for main to let us know the references are
+ # gone and the greenlet objects no longer reachable
+ ref_cleared.wait()
+ # The creating thread must call getcurrent() (or a few other
+ # greenlet APIs) because that's when the thread-local list of dead
+ # greenlets gets cleared.
+ getcurrent()
+
+ # We start with 3 references to the subclass:
+ # - This module
+ # - Its __mro__
+ # - The __subclassess__ attribute of greenlet
+ # - (If we call gc.get_referents(), we find four entries, including
+ # some other tuple ``(greenlet)`` that I'm not sure about but must
be part
+ # of the machinery.)
+ #
+ # On Python 3.10 it's often enough to just run 3 threads; on Python
2.7,
+ # more threads are needed, and the results are still
+ # non-deterministic. Presumably the memory layouts are different
+ initial_refs = sys.getrefcount(MyGreenlet)
+ thread_ready_events = []
+ for _ in range(
+ initial_refs + 45
+ ):
+ event = Event()
+ thread = Thread(target=thread_main, args=(event,))
+ thread_ready_events.append(event)
+ thread.start()
+
+
+ for done_event in thread_ready_events:
+ done_event.wait()
+
+
+ del glets[:]
+ ref_cleared.set()
+ # Let any other thread run; it will crash the interpreter
+ # if not fixed (or silently corrupt memory and we possibly crash
+ # later).
+ time.sleep(1)
+ self.assertEqual(sys.getrefcount(MyGreenlet), initial_refs)
+
+
class TestRepr(unittest.TestCase):
def assertEndsWith(self, got, suffix):
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn'
'--exclude=.svnignore' old/greenlet-1.1.0/src/greenlet/tests/test_leaks.py
new/greenlet-1.1.2/src/greenlet/tests/test_leaks.py
--- old/greenlet-1.1.0/src/greenlet/tests/test_leaks.py 2021-05-06
16:53:06.000000000 +0200
+++ new/greenlet-1.1.2/src/greenlet/tests/test_leaks.py 2021-09-29
12:35:47.000000000 +0200
@@ -4,82 +4,175 @@
import time
import weakref
-import greenlet
import threading
+import greenlet
+
+class TestLeaks(unittest.TestCase):
-class ArgRefcountTests(unittest.TestCase):
def test_arg_refs(self):
args = ('a', 'b', 'c')
refcount_before = sys.getrefcount(args)
+ # pylint:disable=unnecessary-lambda
g = greenlet.greenlet(
lambda *args: greenlet.getcurrent().parent.switch(*args))
- for i in range(100):
+ for _ in range(100):
g.switch(*args)
self.assertEqual(sys.getrefcount(args), refcount_before)
def test_kwarg_refs(self):
kwargs = {}
+ # pylint:disable=unnecessary-lambda
g = greenlet.greenlet(
lambda **kwargs: greenlet.getcurrent().parent.switch(**kwargs))
- for i in range(100):
+ for _ in range(100):
g.switch(**kwargs)
self.assertEqual(sys.getrefcount(kwargs), 2)
- if greenlet.GREENLET_USE_GC:
- # These only work with greenlet gc support
+ assert greenlet.GREENLET_USE_GC # Option to disable this was removed in 1.0
- def recycle_threads(self):
- # By introducing a thread that does sleep we allow other threads,
- # that have triggered their __block condition, but did not have a
- # chance to deallocate their thread state yet, to finally do so.
- # The way it works is by requiring a GIL switch (different thread),
- # which does a GIL release (sleep), which might do a GIL switch
- # to finished threads and allow them to clean up.
- def worker():
- time.sleep(0.001)
+ def recycle_threads(self):
+ # By introducing a thread that does sleep we allow other threads,
+ # that have triggered their __block condition, but did not have a
+ # chance to deallocate their thread state yet, to finally do so.
+ # The way it works is by requiring a GIL switch (different thread),
+ # which does a GIL release (sleep), which might do a GIL switch
+ # to finished threads and allow them to clean up.
+ def worker():
+ time.sleep(0.001)
+ t = threading.Thread(target=worker)
+ t.start()
+ time.sleep(0.001)
+ t.join()
+
+ def test_threaded_leak(self):
+ gg = []
+ def worker():
+ # only main greenlet present
+ gg.append(weakref.ref(greenlet.getcurrent()))
+ for _ in range(2):
t = threading.Thread(target=worker)
t.start()
- time.sleep(0.001)
t.join()
-
- def test_threaded_leak(self):
- gg = []
- def worker():
- # only main greenlet present
- gg.append(weakref.ref(greenlet.getcurrent()))
- for i in range(2):
- t = threading.Thread(target=worker)
- t.start()
- t.join()
- del t
- greenlet.getcurrent() # update ts_current
- self.recycle_threads()
- greenlet.getcurrent() # update ts_current
- gc.collect()
- greenlet.getcurrent() # update ts_current
- for g in gg:
- self.assertTrue(g() is None)
-
- def test_threaded_adv_leak(self):
- gg = []
- def worker():
- # main and additional *finished* greenlets
- ll = greenlet.getcurrent().ll = []
- def additional():
- ll.append(greenlet.getcurrent())
- for i in range(2):
- greenlet.greenlet(additional).switch()
- gg.append(weakref.ref(greenlet.getcurrent()))
- for i in range(2):
- t = threading.Thread(target=worker)
- t.start()
- t.join()
- del t
- greenlet.getcurrent() # update ts_current
- self.recycle_threads()
- greenlet.getcurrent() # update ts_current
+ del t
+ greenlet.getcurrent() # update ts_current
+ self.recycle_threads()
+ greenlet.getcurrent() # update ts_current
+ gc.collect()
+ greenlet.getcurrent() # update ts_current
+ for g in gg:
+ self.assertIsNone(g())
+
+ def test_threaded_adv_leak(self):
+ gg = []
+ def worker():
+ # main and additional *finished* greenlets
+ ll = greenlet.getcurrent().ll = []
+ def additional():
+ ll.append(greenlet.getcurrent())
+ for _ in range(2):
+ greenlet.greenlet(additional).switch()
+ gg.append(weakref.ref(greenlet.getcurrent()))
+ for _ in range(2):
+ t = threading.Thread(target=worker)
+ t.start()
+ t.join()
+ del t
+ greenlet.getcurrent() # update ts_current
+ self.recycle_threads()
+ greenlet.getcurrent() # update ts_current
+ gc.collect()
+ greenlet.getcurrent() # update ts_current
+ for g in gg:
+ self.assertIsNone(g())
+
+ def test_issue251_killing_cross_thread_leaks_list(self,
manually_collect_background=True):
+ # See https://github.com/python-greenlet/greenlet/issues/251
+ # Killing a greenlet (probably not the main one)
+ # in one thread from another thread would
+ # result in leaking a list (the ts_delkey list).
+
+ # For the test to be valid, even empty lists have to be tracked by the
+ # GC
+ assert gc.is_tracked([])
+
+ def count_objects(kind=list):
+ # pylint:disable=unidiomatic-typecheck
+ # Collect the garbage.
+ for _ in range(3):
+ gc.collect()
gc.collect()
- greenlet.getcurrent() # update ts_current
- for g in gg:
- self.assertTrue(g() is None)
+ return sum(
+ 1
+ for x in gc.get_objects()
+ if type(x) is kind
+ )
+
+ # XXX: The main greenlet of a dead thread is only released
+ # when one of the proper greenlet APIs is used from a different
+ # running thread. See #252
(https://github.com/python-greenlet/greenlet/issues/252)
+ greenlet.getcurrent()
+ greenlets_before = count_objects(greenlet.greenlet)
+
+ background_glet_running = threading.Event()
+ background_glet_killed = threading.Event()
+ background_greenlets = []
+ def background_greenlet():
+ # Throw control back to the main greenlet.
+ greenlet.getcurrent().parent.switch()
+
+ def background_thread():
+ glet = greenlet.greenlet(background_greenlet)
+ background_greenlets.append(glet)
+ glet.switch() # Be sure it's active.
+ # Control is ours again.
+ del glet # Delete one reference from the thread it runs in.
+ background_glet_running.set()
+ background_glet_killed.wait()
+ # To trigger the background collection of the dead
+ # greenlet, thus clearing out the contents of the list, we
+ # need to run some APIs. See issue 252.
+ if manually_collect_background:
+ greenlet.getcurrent()
+
+
+ t = threading.Thread(target=background_thread)
+ t.start()
+ background_glet_running.wait()
+
+ lists_before = count_objects()
+
+ assert len(background_greenlets) == 1
+ self.assertFalse(background_greenlets[0].dead)
+ # Delete the last reference to the background greenlet
+ # from a different thread. This puts it in the background thread's
+ # ts_delkey list.
+ del background_greenlets[:]
+ background_glet_killed.set()
+
+ # Now wait for the background thread to die.
+ t.join(10)
+ del t
+
+ # Free the background main greenlet by forcing greenlet to notice a
difference.
+ greenlet.getcurrent()
+ greenlets_after = count_objects(greenlet.greenlet)
+
+ lists_after = count_objects()
+ # On 2.7, we observe that lists_after is smaller than
+ # lists_before. No idea what lists got cleaned up. All the
+ # Python 3 versions match exactly.
+ self.assertLessEqual(lists_after, lists_before)
+
+ self.assertEqual(greenlets_before, greenlets_after)
+
+ @unittest.expectedFailure
+ def test_issue251_issue252_need_to_collect_in_background(self):
+ # This still fails because the leak of the list
+ # still exists when we don't call a greenlet API before exiting the
+ # thread. The proximate cause is that neither of the two greenlets
+ # from the background thread are actually being destroyed, even though
+ # the GC is in fact visiting both objects.
+ # It's not clear where that leak is? For some reason the thread-local
dict
+ # holding it isn't being cleaned up.
+
self.test_issue251_killing_cross_thread_leaks_list(manually_collect_background=False)
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn'
'--exclude=.svnignore' old/greenlet-1.1.0/src/greenlet/tests/test_tracing.py
new/greenlet-1.1.2/src/greenlet/tests/test_tracing.py
--- old/greenlet-1.1.0/src/greenlet/tests/test_tracing.py 2021-05-06
16:53:06.000000000 +0200
+++ new/greenlet-1.1.2/src/greenlet/tests/test_tracing.py 2021-09-29
12:35:47.000000000 +0200
@@ -1,52 +1,267 @@
+import sys
import unittest
-import threading
import greenlet
class SomeError(Exception):
pass
-class TracingTests(unittest.TestCase):
- if greenlet.GREENLET_USE_TRACING:
- def test_greenlet_tracing(self):
- main = greenlet.getcurrent()
- actions = []
- def trace(*args):
- actions.append(args)
- def dummy():
- pass
- def dummyexc():
- raise SomeError()
- oldtrace = greenlet.settrace(trace)
- try:
- g1 = greenlet.greenlet(dummy)
- g1.switch()
- g2 = greenlet.greenlet(dummyexc)
- self.assertRaises(SomeError, g2.switch)
- finally:
- greenlet.settrace(oldtrace)
- self.assertEqual(actions, [
- ('switch', (main, g1)),
- ('switch', (g1, main)),
- ('switch', (main, g2)),
- ('throw', (g2, main)),
- ])
-
- def test_exception_disables_tracing(self):
- main = greenlet.getcurrent()
- actions = []
- def trace(*args):
- actions.append(args)
- raise SomeError()
- def dummy():
- main.switch()
- g = greenlet.greenlet(dummy)
- g.switch()
- oldtrace = greenlet.settrace(trace)
- try:
- self.assertRaises(SomeError, g.switch)
- self.assertEqual(greenlet.gettrace(), None)
- finally:
- greenlet.settrace(oldtrace)
- self.assertEqual(actions, [
- ('switch', (main, g)),
- ])
+class GreenletTracer(object):
+ oldtrace = None
+
+ def __init__(self, error_on_trace=False):
+ self.actions = []
+ self.error_on_trace = error_on_trace
+
+ def __call__(self, *args):
+ self.actions.append(args)
+ if self.error_on_trace:
+ raise SomeError
+
+ def __enter__(self):
+ self.oldtrace = greenlet.settrace(self)
+ return self.actions
+
+ def __exit__(self, *args):
+ greenlet.settrace(self.oldtrace)
+
+
+class TestGreenletTracing(unittest.TestCase):
+ """
+ Tests of ``greenlet.settrace()``
+ """
+
+ def test_greenlet_tracing(self):
+ main = greenlet.getcurrent()
+ def dummy():
+ pass
+ def dummyexc():
+ raise SomeError()
+
+ with GreenletTracer() as actions:
+ g1 = greenlet.greenlet(dummy)
+ g1.switch()
+ g2 = greenlet.greenlet(dummyexc)
+ self.assertRaises(SomeError, g2.switch)
+
+ self.assertEqual(actions, [
+ ('switch', (main, g1)),
+ ('switch', (g1, main)),
+ ('switch', (main, g2)),
+ ('throw', (g2, main)),
+ ])
+
+ def test_exception_disables_tracing(self):
+ main = greenlet.getcurrent()
+ def dummy():
+ main.switch()
+ g = greenlet.greenlet(dummy)
+ g.switch()
+ with GreenletTracer(error_on_trace=True) as actions:
+ self.assertRaises(SomeError, g.switch)
+ self.assertEqual(greenlet.gettrace(), None)
+
+ self.assertEqual(actions, [
+ ('switch', (main, g)),
+ ])
+
+
+class PythonTracer(object):
+ oldtrace = None
+
+ def __init__(self):
+ self.actions = []
+
+ def __call__(self, frame, event, arg):
+ # Record the co_name so we have an idea what function we're in.
+ self.actions.append((event, frame.f_code.co_name))
+
+ def __enter__(self):
+ self.oldtrace = sys.setprofile(self)
+ return self.actions
+
+ def __exit__(self, *args):
+ sys.setprofile(self.oldtrace)
+
+def tpt_callback():
+ return 42
+
+class TestPythonTracing(unittest.TestCase):
+ """
+ Tests of the interaction of ``sys.settrace()``
+ with greenlet facilities.
+
+ NOTE: Most of this is probably CPython specific.
+ """
+
+ maxDiff = None
+
+ def test_trace_events_trivial(self):
+ with PythonTracer() as actions:
+ tpt_callback()
+ # If we use the sys.settrace instead of setprofile, we get
+ # this:
+
+ # self.assertEqual(actions, [
+ # ('call', 'tpt_callback'),
+ # ('call', '__exit__'),
+ # ])
+
+ self.assertEqual(actions, [
+ ('return', '__enter__'),
+ ('call', 'tpt_callback'),
+ ('return', 'tpt_callback'),
+ ('call', '__exit__'),
+ ('c_call', '__exit__'),
+ ])
+
+ def _trace_switch(self, glet):
+ with PythonTracer() as actions:
+ glet.switch()
+ return actions
+
+ def _check_trace_events_func_already_set(self, glet):
+ actions = self._trace_switch(glet)
+ self.assertEqual(actions, [
+ ('return', '__enter__'),
+ ('c_call', '_trace_switch'),
+ ('call', 'run'),
+ ('call', 'tpt_callback'),
+ ('return', 'tpt_callback'),
+ ('return', 'run'),
+ ('c_return', '_trace_switch'),
+ ('call', '__exit__'),
+ ('c_call', '__exit__'),
+ ])
+
+ def test_trace_events_into_greenlet_func_already_set(self):
+ def run():
+ return tpt_callback()
+
+ self._check_trace_events_func_already_set(greenlet.greenlet(run))
+
+ def test_trace_events_into_greenlet_subclass_already_set(self):
+ class X(greenlet.greenlet):
+ def run(self):
+ return tpt_callback()
+ self._check_trace_events_func_already_set(X())
+
+ def _check_trace_events_from_greenlet_sets_profiler(self, g, tracer):
+ g.switch()
+ tpt_callback()
+ tracer.__exit__()
+ self.assertEqual(tracer.actions, [
+ ('return', '__enter__'),
+ ('call', 'tpt_callback'),
+ ('return', 'tpt_callback'),
+ ('return', 'run'),
+ ('call', 'tpt_callback'),
+ ('return', 'tpt_callback'),
+ ('call', '__exit__'),
+ ('c_call', '__exit__'),
+ ])
+
+
+ def test_trace_events_from_greenlet_func_sets_profiler(self):
+ tracer = PythonTracer()
+ def run():
+ tracer.__enter__()
+ return tpt_callback()
+
+
self._check_trace_events_from_greenlet_sets_profiler(greenlet.greenlet(run),
+ tracer)
+
+ def test_trace_events_from_greenlet_subclass_sets_profiler(self):
+ tracer = PythonTracer()
+ class X(greenlet.greenlet):
+ def run(self):
+ tracer.__enter__()
+ return tpt_callback()
+
+ self._check_trace_events_from_greenlet_sets_profiler(X(), tracer)
+
+
+ def test_trace_events_multiple_greenlets_switching(self):
+ tracer = PythonTracer()
+
+ g1 = None
+ g2 = None
+
+ def g1_run():
+ tracer.__enter__()
+ tpt_callback()
+ g2.switch()
+ tpt_callback()
+ return 42
+
+ def g2_run():
+ tpt_callback()
+ tracer.__exit__()
+ tpt_callback()
+ g1.switch()
+
+ g1 = greenlet.greenlet(g1_run)
+ g2 = greenlet.greenlet(g2_run)
+
+ x = g1.switch()
+ self.assertEqual(x, 42)
+ tpt_callback() # ensure not in the trace
+ self.assertEqual(tracer.actions, [
+ ('return', '__enter__'),
+ ('call', 'tpt_callback'),
+ ('return', 'tpt_callback'),
+ ('c_call', 'g1_run'),
+ ('call', 'g2_run'),
+ ('call', 'tpt_callback'),
+ ('return', 'tpt_callback'),
+ ('call', '__exit__'),
+ ('c_call', '__exit__'),
+ ])
+
+ def test_trace_events_multiple_greenlets_switching_siblings(self):
+ # Like the first version, but get both greenlets running first
+ # as "siblings" and then establish the tracing.
+ tracer = PythonTracer()
+
+ g1 = None
+ g2 = None
+
+ def g1_run():
+ greenlet.getcurrent().parent.switch()
+ tracer.__enter__()
+ tpt_callback()
+ g2.switch()
+ tpt_callback()
+ return 42
+
+ def g2_run():
+ greenlet.getcurrent().parent.switch()
+
+ tpt_callback()
+ tracer.__exit__()
+ tpt_callback()
+ g1.switch()
+
+ g1 = greenlet.greenlet(g1_run)
+ g2 = greenlet.greenlet(g2_run)
+
+ # Start g1
+ g1.switch()
+ # And it immediately returns control to us.
+ # Start g2
+ g2.switch()
+ # Which also returns. Now kick of the real part of the
+ # test.
+ x = g1.switch()
+ self.assertEqual(x, 42)
+
+ tpt_callback() # ensure not in the trace
+ self.assertEqual(tracer.actions, [
+ ('return', '__enter__'),
+ ('call', 'tpt_callback'),
+ ('return', 'tpt_callback'),
+ ('c_call', 'g1_run'),
+ ('call', 'tpt_callback'),
+ ('return', 'tpt_callback'),
+ ('call', '__exit__'),
+ ('c_call', '__exit__'),
+ ])
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn'
'--exclude=.svnignore' old/greenlet-1.1.0/src/greenlet.egg-info/PKG-INFO
new/greenlet-1.1.2/src/greenlet.egg-info/PKG-INFO
--- old/greenlet-1.1.0/src/greenlet.egg-info/PKG-INFO 2021-05-06
16:53:07.000000000 +0200
+++ new/greenlet-1.1.2/src/greenlet.egg-info/PKG-INFO 2021-09-29
12:35:47.000000000 +0200
@@ -1,8 +1,12 @@
Metadata-Version: 2.1
Name: greenlet
-Version: 1.1.0
+Version: 1.1.2
Summary: Lightweight in-process concurrent programming
Home-page: https://greenlet.readthedocs.io/
+Author: Alexey Borzenkov
+Author-email: sna...@gmail.com
+Maintainer: Jason Madden
+Maintainer-email: ja...@nextthought.com
License: MIT License
Project-URL: Bug Tracker, https://github.com/python-greenlet/greenlet/issues
Project-URL: Source Code, https://github.com/python-greenlet/greenlet/
@@ -69,7 +73,9 @@
Documentation is available on readthedocs.org:
https://greenlet.readthedocs.io
+Keywords: greenlet coroutine concurrency threads cooperative
Platform: any
+Classifier: Development Status :: 5 - Production/Stable
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Natural Language :: English
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn'
'--exclude=.svnignore' old/greenlet-1.1.0/tox.ini new/greenlet-1.1.2/tox.ini
--- old/greenlet-1.1.0/tox.ini 2021-05-06 16:53:06.000000000 +0200
+++ new/greenlet-1.1.2/tox.ini 2021-09-29 12:35:47.000000000 +0200
@@ -11,12 +11,18 @@
test
docs
+[testenv:py310]
+# See tests.yml
+commands =
+ python -m unittest discover -v greenlet.tests
+
[testenv:docs]
# usedevelop to save rebuilding the extension
usedevelop = true
+# We can't use Python 3.10 yet, so specify fully what we need.
basepython =
- python3
+ python3.9
commands =
sphinx-build -b html -d docs/_build/doctrees docs docs/_build/html
sphinx-build -b doctest -d docs/_build/doctrees docs docs/_build/doctest
