Re: [PATCH] Use debianbts instead of SOAPpy
On Tue, 07 Nov 2017, Ville Skyttä wrote: > Attached is a small patch on top of the previous one: 'fixed' is not a > documented value for a pending bug, 'done' is. Just happened to > notice, but did not see any effect in cases I tested with. Thanks, I merged your two patches as I got python-debianbts installed on the production server. Cheers, -- Raphaël Hertzog ◈ Debian Developer Support Debian LTS: https://www.freexian.com/services/debian-lts.html Learn to master Debian: https://debian-handbook.info/get/
Re: [PATCH] Use debianbts instead of SOAPpy
On Wed, Nov 1, 2017 at 10:51 PM, Ville Skyttä wrote: > On Wed, Nov 1, 2017 at 12:52 PM, Raphael Hertzog wrote: >> On Tue, 31 Oct 2017, Ville Skyttä wrote: >>> This switches to using debianbts instead of SOAPpy. To be applied over >>> the LDAP->nm.d.o REST patch in https://bugs.debian.org/797223#10 >> >> Thanks for this. Right now tracker.debian.org still runs jessie. Do >> you know if your code works with python-debianbts 1.12 from jessie or if >> I need to request the installation from jessie-backports ? (which has >> 2.6.1~bpo8+1) > > I don't know; while doing the port I skimmed through old > python-debianbts tags at GitHub, and IIRC all needed functions have > always been there, which is why I didn't add any version to the > dependency. All the testing I've done has been with the 2.6.1 shipping > with Ubuntu 17.10. Tested locally using jessie docker image + django 1.8 from backports (so this is with debianbts 1.12): success, works. Attached is a small patch on top of the previous one: 'fixed' is not a documented value for a pending bug, 'done' is. Just happened to notice, but did not see any effect in cases I tested with. From baaa91765aab70e6bfbd6bf1bc125dc67816829e Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ville=20Skytt=C3=A4?= Date: Tue, 7 Nov 2017 22:49:34 +0200 Subject: [PATCH 2/2] BTS: Check for pending == 'done' instead of 'fixed' https://wiki.debian.org/DebbugsSoapInterface#line-55 --- distro_tracker/vendor/debian/tracker_tasks.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/distro_tracker/vendor/debian/tracker_tasks.py b/distro_tracker/vendor/debian/tracker_tasks.py index 3478c06..3dc8d3b 100644 --- a/distro_tracker/vendor/debian/tracker_tasks.py +++ b/distro_tracker/vendor/debian/tracker_tasks.py @@ -247,7 +247,7 @@ class UpdatePackageBugStats(BaseTask): bug_stats = {} bugs = debianbts.get_status(*bug_numbers) for bug in bugs: -if bug.done or bug.fixed_versions or bug.pending == 'fixed': +if bug.done or bug.fixed_versions or bug.pending == 'done': continue bug_stats.setdefault(bug.package, 0) -- 2.14.1
Re: [PATCH] Use debianbts instead of SOAPpy
On Wed, 01 Nov 2017, Mattia Rizzolo wrote: > On Wed, Nov 01, 2017 at 11:52:16AM +0100, Raphael Hertzog wrote: > > Thanks for this. Right now tracker.debian.org still runs jessie. > > DSA is actively looking to upgrade as many hosts as possible, and I know > a DSA member asked around about who to contact about upgrading ticharich > ("is it buxy only or somebody else as well?" kind of questions). If you > believe tracker.d.o would do great with the host on stretch, you should > definitely send DSA a word about it. We should certainly upgrade to stretch but I would like to switch to Django 1.11 and Python 3 at the same time. Thus I'm not fully ready yet. Cheers, -- Raphaël Hertzog ◈ Debian Developer Support Debian LTS: https://www.freexian.com/services/debian-lts.html Learn to master Debian: https://debian-handbook.info/get/
Re: [PATCH] Use debianbts instead of SOAPpy
On Wed, Nov 01, 2017 at 11:52:16AM +0100, Raphael Hertzog wrote: > Thanks for this. Right now tracker.debian.org still runs jessie. DSA is actively looking to upgrade as many hosts as possible, and I know a DSA member asked around about who to contact about upgrading ticharich ("is it buxy only or somebody else as well?" kind of questions). If you believe tracker.d.o would do great with the host on stretch, you should definitely send DSA a word about it. -- regards, Mattia Rizzolo GPG Key: 66AE 2B4A FCCF 3F52 DA18 4D18 4B04 3FCD B944 4540 .''`. more about me: https://mapreri.org : :' : Launchpad user: https://launchpad.net/~mapreri `. `'` Debian QA page: https://qa.debian.org/developer.php?login=mattia `-
Re: [PATCH] Use debianbts instead of SOAPpy
On Wed, Nov 1, 2017 at 12:52 PM, Raphael Hertzog wrote: > On Tue, 31 Oct 2017, Ville Skyttä wrote: >> This switches to using debianbts instead of SOAPpy. To be applied over >> the LDAP->nm.d.o REST patch in https://bugs.debian.org/797223#10 > > Thanks for this. Right now tracker.debian.org still runs jessie. Do > you know if your code works with python-debianbts 1.12 from jessie or if > I need to request the installation from jessie-backports ? (which has > 2.6.1~bpo8+1) I don't know; while doing the port I skimmed through old python-debianbts tags at GitHub, and IIRC all needed functions have always been there, which is why I didn't add any version to the dependency. All the testing I've done has been with the 2.6.1 shipping with Ubuntu 17.10. Attached is a revised version of the patch: adds tox.ini modifications, no other changes. From 91a7d107f3a0f3949a1f4b8c03db6dad193db2e4 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ville=20Skytt=C3=A4?= Date: Tue, 31 Oct 2017 08:44:42 +0200 Subject: [PATCH] Use debianbts instead of SOAPpy --- TODO | 3 --- debian/control| 4 ++-- distro_tracker/vendor/debian/tracker_tasks.py | 30 ++- docs/setup/setup.rst | 4 ++-- tox.ini | 2 +- 5 files changed, 16 insertions(+), 27 deletions(-) diff --git a/TODO b/TODO index 87d41fd..d5d2851 100644 --- a/TODO +++ b/TODO @@ -16,9 +16,6 @@ Misc possible enhancements: * Add a subscriber count on each package page (and put subscription-related actions in the same panel). * Paul Wise's input: https://lists.debian.org/debian-qa/2014/07/msg00092.html -* Codebase is Python 3 compatible but some optional dependencies need to - be ported: - - python-soappy => switch to pysimplesoap * Re-enable a smarter clickjacking protection: only activate it when the page is rendered for a logged in user and allow callers to pass a "anonymous=1" parameter that will force render the web page as if diff --git a/debian/control b/debian/control index a5189e6..3a7373c 100644 --- a/debian/control +++ b/debian/control @@ -12,13 +12,13 @@ Build-Depends: debhelper (>= 9), python-apt, python-bs4, python-debian, +python-debianbts, python-django (>= 1.8), python-django-jsonfield, python-django-captcha, python-gpgme, python-lzma, python-requests (>= 2), -python-soappy, python-yaml, python-pyinotify, Standards-Version: 3.9.5 @@ -59,12 +59,12 @@ Depends: ${python:Depends}, python-apt, python-beautifulsoup, python-debian, +python-debianbts, python-django (>= 1.8), python-django-jsonfield, python-gpgme, python-lzma, python-requests (>= 2), -python-soappy, python-yaml, python-pyinotify, ${misc:Depends} diff --git a/distro_tracker/vendor/debian/tracker_tasks.py b/distro_tracker/vendor/debian/tracker_tasks.py index 0a7b185..bf4d4b8 100644 --- a/distro_tracker/vendor/debian/tracker_tasks.py +++ b/distro_tracker/vendor/debian/tracker_tasks.py @@ -56,7 +56,7 @@ from bs4 import BeautifulSoup as soup import yaml try: -import SOAPpy +import debianbts except ImportError: pass @@ -222,7 +222,7 @@ class UpdatePackageBugStats(BaseTask): def _get_tagged_bug_stats(self, tag, user=None): """ -Using the BTS SOAP interface, retrieves the statistics of bugs with a +Using the BTS interface, retrieves the statistics of bugs with a particular tag. :param tag: The tag for which the statistics are required. @@ -237,29 +237,21 @@ class UpdatePackageBugStats(BaseTask): debian_ca_bundle = '/etc/ssl/ca-debian/ca-certificates.crt' if os.path.exists(debian_ca_bundle): os.environ['SSL_CERT_FILE'] = debian_ca_bundle -url = 'https://bugs.debian.org/cgi-bin/soap.cgi' -namespace = 'Debbugs/SOAP' -server = SOAPpy.SOAPProxy(url, namespace) if user: -bugs = server.get_usertag(user, tag) -bugs = bugs[0] +bug_numbers = debianbts.get_usertag(user, tag).values() else: -bugs = server.get_bugs('tag', tag) +bug_numbers = debianbts.get_bugs('tag', tag) # Match each retrieved bug ID to a package and then find the aggregate # count for each package. bug_stats = {} -statuses = server.get_status(bugs) -statuses = statuses[0] -for status in statuses: -status = status['value'] -if status['done'] or status['fixed'] or \ -status['pending'] == 'fixed': +bugs = debianbts.get_status(*bug_numbers) +for bug
Re: [PATCH] Use debianbts instead of SOAPpy
On Tue, 31 Oct 2017, Ville Skyttä wrote: > This switches to using debianbts instead of SOAPpy. To be applied over > the LDAP->nm.d.o REST patch in https://bugs.debian.org/797223#10 Thanks for this. Right now tracker.debian.org still runs jessie. Do you know if your code works with python-debianbts 1.12 from jessie or if I need to request the installation from jessie-backports ? (which has 2.6.1~bpo8+1) Cheers, -- Raphaël Hertzog ◈ Debian Developer Support Debian LTS: https://www.freexian.com/services/debian-lts.html Learn to master Debian: https://debian-handbook.info/get/
[PATCH] Use debianbts instead of SOAPpy
This switches to using debianbts instead of SOAPpy. To be applied over the LDAP->nm.d.o REST patch in https://bugs.debian.org/797223#10
[PATCH] Use debianbts instead of SOAPpy
--- TODO | 3 --- debian/control| 4 ++-- distro_tracker/vendor/debian/tracker_tasks.py | 30 ++- docs/setup/setup.rst | 4 ++-- 4 files changed, 15 insertions(+), 26 deletions(-) diff --git a/TODO b/TODO index 87d41fd..d5d2851 100644 --- a/TODO +++ b/TODO @@ -16,9 +16,6 @@ Misc possible enhancements: * Add a subscriber count on each package page (and put subscription-related actions in the same panel). * Paul Wise's input: https://lists.debian.org/debian-qa/2014/07/msg00092.html -* Codebase is Python 3 compatible but some optional dependencies need to - be ported: - - python-soappy => switch to pysimplesoap * Re-enable a smarter clickjacking protection: only activate it when the page is rendered for a logged in user and allow callers to pass a "anonymous=1" parameter that will force render the web page as if diff --git a/debian/control b/debian/control index a5189e6..3a7373c 100644 --- a/debian/control +++ b/debian/control @@ -12,13 +12,13 @@ Build-Depends: debhelper (>= 9), python-apt, python-bs4, python-debian, +python-debianbts, python-django (>= 1.8), python-django-jsonfield, python-django-captcha, python-gpgme, python-lzma, python-requests (>= 2), -python-soappy, python-yaml, python-pyinotify, Standards-Version: 3.9.5 @@ -59,12 +59,12 @@ Depends: ${python:Depends}, python-apt, python-beautifulsoup, python-debian, +python-debianbts, python-django (>= 1.8), python-django-jsonfield, python-gpgme, python-lzma, python-requests (>= 2), -python-soappy, python-yaml, python-pyinotify, ${misc:Depends} diff --git a/distro_tracker/vendor/debian/tracker_tasks.py b/distro_tracker/vendor/debian/tracker_tasks.py index 0a7b185..bf4d4b8 100644 --- a/distro_tracker/vendor/debian/tracker_tasks.py +++ b/distro_tracker/vendor/debian/tracker_tasks.py @@ -56,7 +56,7 @@ from bs4 import BeautifulSoup as soup import yaml try: -import SOAPpy +import debianbts except ImportError: pass @@ -222,7 +222,7 @@ class UpdatePackageBugStats(BaseTask): def _get_tagged_bug_stats(self, tag, user=None): """ -Using the BTS SOAP interface, retrieves the statistics of bugs with a +Using the BTS interface, retrieves the statistics of bugs with a particular tag. :param tag: The tag for which the statistics are required. @@ -237,29 +237,21 @@ class UpdatePackageBugStats(BaseTask): debian_ca_bundle = '/etc/ssl/ca-debian/ca-certificates.crt' if os.path.exists(debian_ca_bundle): os.environ['SSL_CERT_FILE'] = debian_ca_bundle -url = 'https://bugs.debian.org/cgi-bin/soap.cgi' -namespace = 'Debbugs/SOAP' -server = SOAPpy.SOAPProxy(url, namespace) if user: -bugs = server.get_usertag(user, tag) -bugs = bugs[0] +bug_numbers = debianbts.get_usertag(user, tag).values() else: -bugs = server.get_bugs('tag', tag) +bug_numbers = debianbts.get_bugs('tag', tag) # Match each retrieved bug ID to a package and then find the aggregate # count for each package. bug_stats = {} -statuses = server.get_status(bugs) -statuses = statuses[0] -for status in statuses: -status = status['value'] -if status['done'] or status['fixed'] or \ -status['pending'] == 'fixed': +bugs = debianbts.get_status(*bug_numbers) +for bug in bugs: +if bug.done or bug.fixed_versions or bug.pending == 'fixed': continue -package_name = status['package'] -bug_stats.setdefault(package_name, 0) -bug_stats[package_name] += 1 +bug_stats.setdefault(bug.package, 0) +bug_stats[bug.package] += 1 return bug_stats @@ -465,14 +457,14 @@ class UpdatePackageBugStats(BaseTask): if not bug_stats: bug_stats = {} -# Add in help bugs from the BTS SOAP interface +# Add in help bugs from the BTS interface try: help_bugs = self._get_tagged_bug_stats('help') self._extend_bug_stats(bug_stats, help_bugs, 'help') except: logger.exception("Could not get bugs tagged help") -# Add in newcomer bugs from the BTS SOAP interface +# Add in newcomer bugs from the BTS interface try: newcomer_bugs = self._get_tagged_bug_stats('newcomer') self._extend_bug_stats(bug_stats, newcomer_bugs, 'newcomer') diff --git a/docs/setup/setup.rst b/docs/setup/setup.rst index cd95ba6..d4c62f8 100644 --- a/docs/setup/setup.rst +++ b/docs/setup/setup.rst @@ -16,12 +16,12 @@ Distro Tracker currently depends on the following Debian p