Script 'mail_helper' called by obssrc Hello community, here is the log from the commit of package python-ipykernel for openSUSE:Factory checked in at 2022-03-20 20:55:10 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Comparing /work/SRC/openSUSE:Factory/python-ipykernel (Old) and /work/SRC/openSUSE:Factory/.python-ipykernel.new.25692 (New) ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Package is "python-ipykernel" Sun Mar 20 20:55:10 2022 rev:22 rq:962924 version:6.9.2 Changes: -------- --- /work/SRC/openSUSE:Factory/python-ipykernel/python-ipykernel.changes 2022-02-21 17:45:51.531580789 +0100 +++ /work/SRC/openSUSE:Factory/.python-ipykernel.new.25692/python-ipykernel.changes 2022-03-20 20:55:17.406504147 +0100 @@ -1,0 +2,20 @@ +Fri Mar 18 18:23:12 UTC 2022 - Ben Greiner <c...@bnavigator.de> + +- Skip the new shutdown test: too flaky for obs + +------------------------------------------------------------------- +Fri Mar 18 17:19:26 UTC 2022 - Arun Persaud <a...@gmx.de> + +- specfile: + * require psutil + +- update to version 6.9.2: + * Bugs fixed + + Catch error when shutting down kernel from the control channel + #877 (@ccordoba12) + + Only kill children in process group at shutdown #874 (@minrk) + + BUG: Kill subprocesses on shutdown. #869 (@Carreau) + * Maintenance and upkeep improvements + + Clean up CI #871 (@blink1073) + +------------------------------------------------------------------- Old: ---- ipykernel-6.9.1.tar.gz New: ---- ipykernel-6.9.2.tar.gz ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Other differences: ------------------ ++++++ python-ipykernel.spec ++++++ --- /var/tmp/diff_new_pack.7pUTaW/_old 2022-03-20 20:55:18.206505298 +0100 +++ /var/tmp/diff_new_pack.7pUTaW/_new 2022-03-20 20:55:18.214505309 +0100 @@ -19,7 +19,7 @@ %{?!python_module:%define python_module() python3-%{**}} %define skip_python2 1 Name: python-ipykernel -Version: 6.9.1 +Version: 6.9.2 Release: 0 Summary: IPython Kernel for Jupyter License: BSD-3-Clause @@ -37,6 +37,7 @@ Requires: python-jupyter-client Requires: python-jupyter-core Requires: python-matplotlib-inline >= 0.1 +Requires: python-psutil Requires: python-tornado >= 4.2 Requires: python-traitlets >= 5.1.0 Provides: python-jupyter_ipykernel = %{version} @@ -59,6 +60,7 @@ BuildRequires: %{python_module jupyter-client} BuildRequires: %{python_module jupyter-core} BuildRequires: %{python_module matplotlib-inline >= 0.1} +BuildRequires: %{python_module psutil} BuildRequires: %{python_module pytest} BuildRequires: %{python_module tornado >= 4.2} BuildRequires: %{python_module traitlets >= 5.1.0} @@ -99,7 +101,9 @@ %fdupes %{buildroot}%{_jupyter_kernel_dir} %check -%pytest ipykernel +# flaky: bad timings in obs often cause this to fail +donttest="test_shutdown_subprocesses" +%pytest ipykernel -k "not ($donttest)" %files %{python_files} %doc README.md ++++++ ipykernel-6.9.1.tar.gz -> ipykernel-6.9.2.tar.gz ++++++ diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/ipykernel-6.9.1/CHANGELOG.md new/ipykernel-6.9.2/CHANGELOG.md --- old/ipykernel-6.9.1/CHANGELOG.md 2022-02-15 16:52:14.000000000 +0100 +++ new/ipykernel-6.9.2/CHANGELOG.md 2022-03-14 12:21:01.000000000 +0100 @@ -2,6 +2,28 @@ <!-- <START NEW CHANGELOG ENTRY> --> +## 6.9.2 + +([Full Changelog](https://github.com/ipython/ipykernel/compare/v6.9.1...d6744f9e423dacc6b317b1d31805304e89cbec5d)) + +### Bugs fixed + +- Catch error when shutting down kernel from the control channel [#877](https://github.com/ipython/ipykernel/pull/877) ([@ccordoba12](https://github.com/ccordoba12)) +- Only kill children in process group at shutdown [#874](https://github.com/ipython/ipykernel/pull/874) ([@minrk](https://github.com/minrk)) +- BUG: Kill subprocesses on shutdown. [#869](https://github.com/ipython/ipykernel/pull/869) ([@Carreau](https://github.com/Carreau)) + +### Maintenance and upkeep improvements + +- Clean up CI [#871](https://github.com/ipython/ipykernel/pull/871) ([@blink1073](https://github.com/blink1073)) + +### Contributors to this release + +([GitHub contributors page for this release](https://github.com/ipython/ipykernel/graphs/contributors?from=2022-02-15&to=2022-03-14&type=c)) + +[@blink1073](https://github.com/search?q=repo%3Aipython%2Fipykernel+involves%3Ablink1073+updated%3A2022-02-15..2022-03-14&type=Issues) | [@Carreau](https://github.com/search?q=repo%3Aipython%2Fipykernel+involves%3ACarreau+updated%3A2022-02-15..2022-03-14&type=Issues) | [@ccordoba12](https://github.com/search?q=repo%3Aipython%2Fipykernel+involves%3Accordoba12+updated%3A2022-02-15..2022-03-14&type=Issues) | [@echarles](https://github.com/search?q=repo%3Aipython%2Fipykernel+involves%3Aecharles+updated%3A2022-02-15..2022-03-14&type=Issues) | [@fabioz](https://github.com/search?q=repo%3Aipython%2Fipykernel+involves%3Afabioz+updated%3A2022-02-15..2022-03-14&type=Issues) | [@minrk](https://github.com/search?q=repo%3Aipython%2Fipykernel+involves%3Aminrk+updated%3A2022-02-15..2022-03-14&type=Issues) | [@vidartf](https://github.com/search?q=repo%3Aipython%2Fipykernel+involves%3Avidartf+updated%3A2022-02-15..2022-03-14&type=Issues) + +<!-- <END NEW CHANGELOG ENTRY> --> + ## 6.9.1 ([Full Changelog](https://github.com/ipython/ipykernel/compare/v6.9.0...c27e5b95c3d104d9fb6cae3375aec0e98974dcff)) @@ -18,8 +40,6 @@ [@blink1073](https://github.com/search?q=repo%3Aipython%2Fipykernel+involves%3Ablink1073+updated%3A2022-02-07..2022-02-15&type=Issues) | [@echarles](https://github.com/search?q=repo%3Aipython%2Fipykernel+involves%3Aecharles+updated%3A2022-02-07..2022-02-15&type=Issues) | [@minrk](https://github.com/search?q=repo%3Aipython%2Fipykernel+involves%3Aminrk+updated%3A2022-02-07..2022-02-15&type=Issues) -<!-- <END NEW CHANGELOG ENTRY> --> - ## 6.9.0 ([Full Changelog](https://github.com/ipython/ipykernel/compare/v6.8.0...7a229c6c83d44d315f637ef63159a43c64ec73d6)) diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/ipykernel-6.9.1/PKG-INFO new/ipykernel-6.9.2/PKG-INFO --- old/ipykernel-6.9.1/PKG-INFO 2022-02-15 16:52:53.615234000 +0100 +++ new/ipykernel-6.9.2/PKG-INFO 2022-03-14 12:21:48.905651300 +0100 @@ -1,6 +1,6 @@ Metadata-Version: 2.1 Name: ipykernel -Version: 6.9.1 +Version: 6.9.2 Summary: IPython Kernel for Jupyter Home-page: https://ipython.org Author: IPython Development Team diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/ipykernel-6.9.1/ipykernel/_version.py new/ipykernel-6.9.2/ipykernel/_version.py --- old/ipykernel-6.9.1/ipykernel/_version.py 2022-02-15 16:52:35.000000000 +0100 +++ new/ipykernel-6.9.2/ipykernel/_version.py 2022-03-14 12:21:26.000000000 +0100 @@ -4,7 +4,7 @@ import re # Version string must appear intact for tbump versioning -__version__ = '6.9.1' +__version__ = '6.9.2' # Build up version_info tuple for backwards compatibility pattern = r'(?P<major>\d+).(?P<minor>\d+).(?P<patch>\d+)(?P<rest>.*)' diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/ipykernel-6.9.1/ipykernel/debugger.py new/ipykernel-6.9.2/ipykernel/debugger.py --- old/ipykernel-6.9.1/ipykernel/debugger.py 2022-02-15 16:52:14.000000000 +0100 +++ new/ipykernel-6.9.2/ipykernel/debugger.py 2022-03-14 12:21:01.000000000 +0100 @@ -19,10 +19,14 @@ from .compiler import (get_file_name, get_tmp_directory, get_tmp_hash_seed) -# This import is required to have the next ones working... -from debugpy.server import api # noqa -from _pydevd_bundle import pydevd_frame_utils -from _pydevd_bundle.pydevd_suspended_frames import SuspendedFramesManager, _FramesTracker +try: + # This import is required to have the next ones working... + from debugpy.server import api # noqa + from _pydevd_bundle import pydevd_frame_utils + from _pydevd_bundle.pydevd_suspended_frames import SuspendedFramesManager, _FramesTracker + _is_debugpy_available = True +except ImportError: + _is_debugpy_available = False # Required for backwards compatiblity ROUTING_ID = getattr(zmq, 'ROUTING_ID', None) or zmq.IDENTITY diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/ipykernel-6.9.1/ipykernel/eventloops.py new/ipykernel-6.9.2/ipykernel/eventloops.py --- old/ipykernel-6.9.1/ipykernel/eventloops.py 2022-02-15 16:52:14.000000000 +0100 +++ new/ipykernel-6.9.2/ipykernel/eventloops.py 2022-03-14 12:21:01.000000000 +0100 @@ -285,7 +285,10 @@ @loop_tk.exit def loop_tk_exit(kernel): - kernel.app_wrapper.app.destroy() + try: + kernel.app_wrapper.app.destroy() + except RuntimeError: + pass @register_integration('gtk') diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/ipykernel-6.9.1/ipykernel/ipkernel.py new/ipykernel-6.9.2/ipykernel/ipkernel.py --- old/ipykernel-6.9.1/ipykernel/ipkernel.py 2022-02-15 16:52:14.000000000 +0100 +++ new/ipykernel-6.9.2/ipykernel/ipkernel.py 2022-03-14 12:21:01.000000000 +0100 @@ -18,6 +18,7 @@ from .zmqshell import ZMQInteractiveShell from .eventloops import _use_appnope from .compiler import XCachingCompiler +from .debugger import Debugger, _is_debugpy_available try: from IPython.core.interactiveshell import _asyncio_runner @@ -33,12 +34,6 @@ except ImportError: _use_experimental_60_completion = False -try: - import debugpy - from .debugger import Debugger - _is_debugpy_available = True -except ImportError: - _is_debugpy_available = False _EXPERIMENTAL_KEY_NAME = '_jupyter_types_experimental' diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/ipykernel-6.9.1/ipykernel/kernelbase.py new/ipykernel-6.9.2/ipykernel/kernelbase.py --- old/ipykernel-6.9.1/ipykernel/kernelbase.py 2022-02-15 16:52:14.000000000 +0100 +++ new/ipykernel-6.9.2/ipykernel/kernelbase.py 2022-03-14 12:21:01.000000000 +0100 @@ -5,22 +5,26 @@ import asyncio import concurrent.futures -from datetime import datetime -from functools import partial +import inspect import itertools import logging -import inspect import os -from signal import signal, default_int_handler, SIGINT -import sys import socket +import sys import time import uuid import warnings -try: - import psutil -except ImportError: - psutil = None +from datetime import datetime +from functools import partial +from signal import SIGINT, SIGTERM, Signals, default_int_handler, signal + +if sys.platform != "win32": + from signal import SIGKILL +else: + SIGKILL = "windown-SIGKILL-sentinel" + + + try: # jupyter_client >= 5, use tz-aware now @@ -29,20 +33,18 @@ # jupyter_client < 5, use local now() now = datetime.now +import psutil +import zmq +from IPython.core.error import StdinNotImplementedError +from jupyter_client.session import Session from tornado import ioloop from tornado.queues import Queue, QueueEmpty -import zmq +from traitlets import (Any, Bool, Dict, Float, Instance, Integer, List, Set, + Unicode, default, observe) +from traitlets.config.configurable import SingletonConfigurable from zmq.eventloop.zmqstream import ZMQStream -from traitlets.config.configurable import SingletonConfigurable -from IPython.core.error import StdinNotImplementedError from ipykernel.jsonutil import json_clean -from traitlets import ( - Any, Instance, Float, Dict, List, Set, Integer, Unicode, Bool, - observe, default -) - -from jupyter_client.session import Session from ._version import kernel_protocol_version @@ -796,16 +798,15 @@ reply_content, parent, ident) self.log.debug("%s", msg) - async def interrupt_request(self, stream, ident, parent): - pid = os.getpid() - pgid = os.getpgid(pid) - + def _send_interupt_children(self): if os.name == "nt": self.log.error("Interrupt message not supported on Windows") - else: + pid = os.getpid() + pgid = os.getpgid(pid) # Prefer process-group over process - if pgid and hasattr(os, "killpg"): + # but only if the kernel is the leader of the process group + if pgid and pgid == pid and hasattr(os, "killpg"): try: os.killpg(pgid, SIGINT) return @@ -816,6 +817,8 @@ except OSError: pass + async def interrupt_request(self, stream, ident, parent): + self._send_interupt_children() content = parent['content'] self.session.send(stream, 'interrupt_reply', content, parent, ident=ident) return @@ -830,7 +833,7 @@ content, parent ) - self._at_shutdown() + await self._at_shutdown() self.log.debug('Stopping control ioloop') control_io_loop = self.control_stream.io_loop @@ -892,8 +895,6 @@ reply_content = { 'hostname': socket.gethostname() } - if psutil is None: - return reply_content current_process = psutil.Process() all_processes = [current_process] + current_process.children(recursive=True) process_metric_value = self.get_process_metric_value @@ -1131,10 +1132,81 @@ raise EOFError return value - def _at_shutdown(self): + def _signal_children(self, signum): + """ + Send a signal to all our children + + Like `killpg`, but does not include the current process + (or possible parents). + """ + for p in self._process_children(): + self.log.debug(f"Sending {Signals(signum)!r} to subprocess {p}") + try: + if signum == SIGTERM: + p.terminate() + elif signum == SIGKILL: + p.kill() + else: + p.send_signal(signum) + except psutil.NoSuchProcess: + pass + + def _process_children(self): + """Retrieve child processes in the kernel's process group + + Avoids: + - including parents and self with killpg + - including all children that may have forked-off a new group + """ + kernel_process = psutil.Process() + all_children = kernel_process.children(recursive=True) + if os.name == "nt": + return all_children + kernel_pgid = os.getpgrp() + process_group_children = [] + for child in all_children: + try: + child_pgid = os.getpgid(child.pid) + except OSError: + pass + else: + if child_pgid == kernel_pgid: + process_group_children.append(child) + return process_group_children + + async def _progressively_terminate_all_children(self): + sleeps = (0.01, 0.03, 0.1, 0.3, 1, 3, 10) + if not self._process_children(): + self.log.debug("Kernel has no children.") + return + + for signum in (SIGTERM, SIGKILL): + for delay in sleeps: + children = self._process_children() + if not children: + self.log.debug("No more children, continuing shutdown routine.") + return + # signals only children, not current process + self._signal_children(signum) + self.log.debug( + f"Will sleep {delay}s before checking for children and retrying. {children}" + ) + await asyncio.sleep(delay) + + async def _at_shutdown(self): """Actions taken at shutdown by the kernel, called by python's atexit. """ - if self._shutdown_message is not None: - self.session.send(self.iopub_socket, self._shutdown_message, ident=self._topic('shutdown')) - self.log.debug("%s", self._shutdown_message) - self.control_stream.flush(zmq.POLLOUT) + try: + await self._progressively_terminate_all_children() + except Exception as e: + self.log.exception("Exception during subprocesses termination %s", e) + + finally: + if self._shutdown_message is not None: + self.session.send( + self.iopub_socket, + self._shutdown_message, + ident=self._topic("shutdown"), + ) + self.log.debug("%s", self._shutdown_message) + self.control_stream.flush(zmq.POLLOUT) diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/ipykernel-6.9.1/ipykernel/kernelspec.py new/ipykernel-6.9.2/ipykernel/kernelspec.py --- old/ipykernel-6.9.1/ipykernel/kernelspec.py 2022-02-15 16:52:14.000000000 +0100 +++ new/ipykernel-6.9.2/ipykernel/kernelspec.py 2022-03-14 12:21:01.000000000 +0100 @@ -13,7 +13,7 @@ from jupyter_client.kernelspec import KernelSpecManager -from .ipkernel import _is_debugpy_available +from .debugger import _is_debugpy_available pjoin = os.path.join diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/ipykernel-6.9.1/ipykernel/tests/test_kernel.py new/ipykernel-6.9.2/ipykernel/tests/test_kernel.py --- old/ipykernel-6.9.1/ipykernel/tests/test_kernel.py 2022-02-15 16:52:14.000000000 +0100 +++ new/ipykernel-6.9.2/ipykernel/tests/test_kernel.py 2022-03-14 12:21:01.000000000 +0100 @@ -6,14 +6,16 @@ import ast import os.path import platform +import signal import subprocess import sys import time +from subprocess import Popen from tempfile import TemporaryDirectory from flaky import flaky +import psutil import pytest -from packaging import version import IPython from IPython.paths import locate_profile @@ -496,3 +498,75 @@ # comparing first to last ought to be enough, since queues preserve order # use <= in case of very-fast handling and/or low resolution timers assert control_dates[-1] <= shell_dates[0] + + +def _child(): + print("in child", os.getpid()) + + def _print_and_exit(sig, frame): + print(f"Received signal {sig}") + # take some time so retries are triggered + time.sleep(0.5) + sys.exit(-sig) + + signal.signal(signal.SIGTERM, _print_and_exit) + time.sleep(30) + + +def _start_children(): + ip = IPython.get_ipython() + ns = ip.user_ns + + cmd = [sys.executable, "-c", f"from {__name__} import _child; _child()"] + child_pg = Popen(cmd, start_new_session=False) + child_newpg = Popen(cmd, start_new_session=True) + ns["pid"] = os.getpid() + ns["child_pg"] = child_pg.pid + ns["child_newpg"] = child_newpg.pid + # give them time to start up and register signal handlers + time.sleep(1) + + +@pytest.mark.skipif( + platform.python_implementation() == "PyPy", + reason="does not work on PyPy", +) +def test_shutdown_subprocesses(): + """Kernel exits after polite shutdown_request""" + with new_kernel() as kc: + km = kc.parent + msg_id, reply = execute( + f"from {__name__} import _start_children\n_start_children()", + kc=kc, + user_expressions={ + "pid": "pid", + "child_pg": "child_pg", + "child_newpg": "child_newpg", + }, + ) + print(reply) + expressions = reply["user_expressions"] + kernel_process = psutil.Process(int(expressions["pid"]["data"]["text/plain"])) + child_pg = psutil.Process(int(expressions["child_pg"]["data"]["text/plain"])) + child_newpg = psutil.Process( + int(expressions["child_newpg"]["data"]["text/plain"]) + ) + wait_for_idle(kc) + + kc.shutdown() + for i in range(300): # 30s timeout + if km.is_alive(): + time.sleep(0.1) + else: + break + assert not km.is_alive() + assert not kernel_process.is_running() + # child in the process group shut down + assert not child_pg.is_running() + # child outside the process group was not shut down (unix only) + if os.name != 'nt': + assert child_newpg.is_running() + try: + child_newpg.terminate() + except psutil.NoSuchProcess: + pass diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/ipykernel-6.9.1/ipykernel.egg-info/PKG-INFO new/ipykernel-6.9.2/ipykernel.egg-info/PKG-INFO --- old/ipykernel-6.9.1/ipykernel.egg-info/PKG-INFO 2022-02-15 16:52:53.000000000 +0100 +++ new/ipykernel-6.9.2/ipykernel.egg-info/PKG-INFO 2022-03-14 12:21:48.000000000 +0100 @@ -1,6 +1,6 @@ Metadata-Version: 2.1 Name: ipykernel -Version: 6.9.1 +Version: 6.9.2 Summary: IPython Kernel for Jupyter Home-page: https://ipython.org Author: IPython Development Team diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/ipykernel-6.9.1/ipykernel.egg-info/requires.txt new/ipykernel-6.9.2/ipykernel.egg-info/requires.txt --- old/ipykernel-6.9.1/ipykernel.egg-info/requires.txt 2022-02-15 16:52:53.000000000 +0100 +++ new/ipykernel-6.9.2/ipykernel.egg-info/requires.txt 2022-03-14 12:21:48.000000000 +0100 @@ -4,6 +4,7 @@ jupyter_client<8.0 tornado<7.0,>=4.2 matplotlib-inline<0.2.0,>=0.1.0 +psutil nest_asyncio [:platform_system == "Darwin"] diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/ipykernel-6.9.1/pyproject.toml new/ipykernel-6.9.2/pyproject.toml --- old/ipykernel-6.9.1/pyproject.toml 2022-02-15 16:52:35.000000000 +0100 +++ new/ipykernel-6.9.2/pyproject.toml 2022-03-14 12:21:26.000000000 +0100 @@ -16,7 +16,7 @@ skip = ["check-links"] [tool.tbump.version] -current = "6.9.1" +current = "6.9.2" regex = ''' (?P<major>\d+)\.(?P<minor>\d+)\.(?P<patch>\d+) ((?P<channel>a|b|rc|.dev)(?P<release>\d+))? diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/ipykernel-6.9.1/setup.py new/ipykernel-6.9.2/setup.py --- old/ipykernel-6.9.1/setup.py 2022-02-15 16:52:14.000000000 +0100 +++ new/ipykernel-6.9.2/setup.py 2022-03-14 12:21:01.000000000 +0100 @@ -68,6 +68,7 @@ 'tornado>=4.2,<7.0', 'matplotlib-inline>=0.1.0,<0.2.0', 'appnope;platform_system=="Darwin"', + 'psutil', 'nest_asyncio', ], extras_require={