Hello Christos,
Attached a new patch version. It contains the following changes:
* Rebase to current origin/master with cannonical module paths.
* Rename debian/modules/README.Modules-version to debian/modules/control.
* Convert the uscan to a more generic command named ngxmod with uscan as
a subcommand.
* Auto bump module version in debian/modules/control after an upgrade.
* Auto commit the module source changes after an upgrade.
On 08/31/2017 04:08 PM, Christos Trochalakis wrote:
On Sun, Jul 23, 2017 at 09:42:47PM +0300, Mpampis Kostas wrote:
Package: nginx
Severity: wishlist
Tags: patch
This patch introduces the debian/modules/uscan script and some helper
files which can be used to automate the modules watch & upgrade process.
The main script is executed as '$ debian/modules/uscan' without any
arguments from
the root of the debianized nginx source tree where you see the
debian/ directory.
It runs uscan for each nginx module listed in the newly deb822 formatted
debian/modules/README.Modules-version using the watchfiles in
debian/modules/watch.
If a new version is available, it will ask you if you want to upgrade.
If you agree, it will download the tarball, place it in the parent
directory
and upgrade the module's source.
After the upgrade, it will remove the files listed in the
Files-Excluded module field.
Example output:
$ debian/modules/uscan
Thanks a lot Mpampis, great patch, it's a time saver!
Since the module config is now a proper deb822 file, we can rename it to
debian/modules/control. As a final step, this might be a good time to
cannonicalize the module paths to exactly match the package naming. I
believe this should happen first to keep the diffs as simple as
possible.
Regarding the script itself, I believe we can make it more generic
(ngxmod?) and add uscan as a subcommand, we can the add more commands
like list, add, etc.
>From e7c547062e2b79c0a66fc62c615de3523bea9173 Mon Sep 17 00:00:00 2001
From: Mpampis Kostas <charkost...@gmail.com>
Date: Sat, 22 Jul 2017 16:26:21 +0300
Subject: [PATCH] Automate modules watch & upgrade process
This patch introduces the debian/ngxmod script and some helper
files which can be used to automate the modules watch & upgrade process.
The only subcommand for now is `uscan` and is called as '$ ./ngixmod uscan'.
The uscan subcommand runs uscan for each nginx module listed in
the newly deb822 formatted debian/modules/control using the watchfiles
in debian/modules/watch.
If a new version is available, it will ask you if you want to upgrade.
If you agree, it will download the tarball, place it in the nginx
source's parent directory and upgrade the module's source.
After the upgrade, it will remove the files listed in the
Files-Excluded module field and finally it will commit the changes.
---
debian/modules/README.Modules-versions | 69 ------------
debian/modules/control | 71 ++++++++++++
debian/modules/uupdate | 10 ++
debian/modules/watch/http-auth-pam | 4 +
debian/modules/watch/http-cache-purge | 4 +
debian/modules/watch/http-dav-ext | 4 +
debian/modules/watch/http-echo | 4 +
debian/modules/watch/http-fancyindex | 4 +
debian/modules/watch/http-headers-more-filter | 4 +
debian/modules/watch/http-lua | 4 +
debian/modules/watch/http-ndk | 4 +
debian/modules/watch/http-subs-filter | 4 +
debian/modules/watch/http-uploadprogress | 4 +
debian/modules/watch/nchan | 4 +
debian/modules/watch/rtmp | 4 +
debian/ngxmod | 156 ++++++++++++++++++++++++++
16 files changed, 285 insertions(+), 69 deletions(-)
delete mode 100644 debian/modules/README.Modules-versions
create mode 100644 debian/modules/control
create mode 100755 debian/modules/uupdate
create mode 100644 debian/modules/watch/http-auth-pam
create mode 100644 debian/modules/watch/http-cache-purge
create mode 100644 debian/modules/watch/http-dav-ext
create mode 100644 debian/modules/watch/http-echo
create mode 100644 debian/modules/watch/http-fancyindex
create mode 100644 debian/modules/watch/http-headers-more-filter
create mode 100644 debian/modules/watch/http-lua
create mode 100644 debian/modules/watch/http-ndk
create mode 100644 debian/modules/watch/http-subs-filter
create mode 100644 debian/modules/watch/http-uploadprogress
create mode 100644 debian/modules/watch/nchan
create mode 100644 debian/modules/watch/rtmp
create mode 100755 debian/ngxmod
diff --git a/debian/modules/README.Modules-versions b/debian/modules/README.Modules-versions
deleted file mode 100644
index ff5f63b..0000000
--- a/debian/modules/README.Modules-versions
+++ /dev/null
@@ -1,69 +0,0 @@
-README for Modules versions
----------------------------
-
- This file lists third party modules built with nginx in Debian, homepage and
- version.
-
- headers-more-nginx-module
- Homepage: https://github.com/agentzh/headers-more-nginx-module
- Version: v0.32
-
- nginx-development-kit
- Homepage: https://github.com/simpl/ngx_devel_kit/
- Version: v0.3.0
-
- nginx-auth-pam
- Homepage: https://github.com/stogh/ngx_http_auth_pam_module
- Version: 1.5.1
-
- nginx-echo
- Homepage: https://github.com/agentzh/echo-nginx-module
- Version: v0.60
- Patch: build-nginx-1.11.11.patch
-
- nginx-lua
- Homepage: https://github.com/openresty/lua-nginx-module
- Version: v0.10.10
- Patch: openssl-1.1.0.patch
- Patch: discover-luajit-2.1.patch
-
- nginx-upstream-fair
- Homepage: https://github.com/gnosek/nginx-upstream-fair
- Version: a18b409
- Patch: dynamic-module.patch
- Patch: openssl-1.1.0.patch
- Patch: drop-default-port.patch
-
- nchan
- Homepage: https://github.com/slact/nchan
- Version: 1.0.8
-
- nginx-upload-progress
- Homepage: https://github.com/masterzen/nginx-upload-progress-module
- rm -r debian/nginx-upload-progress/test
- Version: v0.9.2
-
- nginx-cache-purge
- Homepage: https://github.com/FRiCKLE/ngx_cache_purge/
- Version: 2.3
- Patch: dynamic-module.patch
- Patch: segfault-1.11.6.patch
-
- nginx-dav-ext-module
- Homepage: https://github.com/arut/nginx-dav-ext-module
- Version: v0.0.3
- Patch: dynamic-module.patch
-
- ngx-fancyindex
- Homepage: https://github.com/aperezdc/ngx-fancyindex
- Version: v0.4.1
-
- ngx_http_substitutions_filter_module
- Homepage: https://github.com/yaoweibin/ngx_http_substitutions_filter_module
- Version: v0.6.4
- Patch: dynamic-module.patch
-
- nginx-rtmp
- Homepage: https://github.com/arut/nginx-rtmp-module
- rm -r debian/modules/nginx-rtmp-module/test
- Version: v1.1.11
diff --git a/debian/modules/control b/debian/modules/control
new file mode 100644
index 0000000..5f67e38
--- /dev/null
+++ b/debian/modules/control
@@ -0,0 +1,71 @@
+Module: http-headers-more-filter
+Homepage: https://github.com/agentzh/headers-more-nginx-module
+Version: v0.32
+Files-Excluded: .gitignore .gitattributes .travis.yml
+
+Module: http-ndk
+Homepage: https://github.com/simpl/ngx_devel_kit/
+Version: v0.3.0
+
+Module: http-auth-pam
+Homepage: https://github.com/stogh/ngx_http_auth_pam_module
+Version: v1.5.1
+
+Module: http-echo
+Homepage: https://github.com/agentzh/echo-nginx-module
+Version: v0.60
+Files-Excluded: .gitignore .gitattributes .travis.yml
+Patch: build-nginx-1.11.11.patch
+
+Module: http-lua
+Homepage: https://github.com/openresty/lua-nginx-module
+Version: v0.10.10
+Patch:
+ openssl-1.1.0.patch
+ discover-luajit-2.1.patch
+Files-Excluded: .gitignore .gitattributes .travis.yml .github
+
+Module: http-upstream-fair
+Homepage: https://github.com/gnosek/nginx-upstream-fair
+Version: a18b409
+Patch:
+ dynamic-module.patch
+ openssl-1.1.0.patch
+ drop-default-port.patch
+
+Module: nchan
+Homepage: https://github.com/slact/nchan
+Version: v1.0.8
+
+Module: http-uploadprogress
+Homepage: https://github.com/masterzen/nginx-upload-progress-module
+Files-Excluded: test
+Version: v0.9.2
+
+Module: http-cache-purge
+Homepage: https://github.com/FRiCKLE/ngx_cache_purge/
+Version: 2.3
+Patch:
+ dynamic-module.patch
+ segfault-1.11.6.patch
+
+Module: http-dav-ext
+Homepage: https://github.com/arut/nginx-dav-ext-module
+Version: v0.0.3
+Patch: dynamic-module.patch
+
+Module: http-fancyindex
+Homepage: https://github.com/aperezdc/ngx-fancyindex
+Version: v0.4.1
+Files-Excluded: .gitignore .travis.yml
+
+Module: http-subs-filter
+Homepage: https://github.com/yaoweibin/ngx_http_substitutions_filter_module
+Version: v0.6.4
+Patch: dynamic-module.patch
+
+Module: rtmp
+Homepage: https://github.com/arut/nginx-rtmp-module
+Files-Excluded: test
+Version: v1.1.11
+
diff --git a/debian/modules/uupdate b/debian/modules/uupdate
new file mode 100755
index 0000000..98a5f8b
--- /dev/null
+++ b/debian/modules/uupdate
@@ -0,0 +1,10 @@
+#!/bin/bash
+
+# debian/modules/uupdate - simple tar wrapper used by uscan to upgrade modules source
+
+MOD_TAR_NAME=$2
+MOD_TAR_VER=$4
+DESTDIR=debian/modules/$1
+
+tar --strip-components=1 -xf ../$MOD_TAR_NAME-$MOD_TAR_VER.tar.gz -C $DESTDIR
+echo $MOD_TAR_VER > /tmp/ngx_uupdate_version_$1
diff --git a/debian/modules/watch/http-auth-pam b/debian/modules/watch/http-auth-pam
new file mode 100644
index 0000000..eac2f73
--- /dev/null
+++ b/debian/modules/watch/http-auth-pam
@@ -0,0 +1,4 @@
+version=4
+opts="dversionmangle=s/v//,filenamemangle=s%(?:.*?)?v?(\d[\d.]*)\.tar\.gz%ngx-mod-http-auth-pam-$1.tar.gz%" \
+ https://github.com/stogh/ngx_http_auth_pam_module/tags \
+ (?:.*?/)?v?(\d[\d.]*)\.tar\.gz debian debian/modules/uupdate http-auth-pam ngx-mod-http-auth-pam
diff --git a/debian/modules/watch/http-cache-purge b/debian/modules/watch/http-cache-purge
new file mode 100644
index 0000000..2566801
--- /dev/null
+++ b/debian/modules/watch/http-cache-purge
@@ -0,0 +1,4 @@
+version=4
+opts="dversionmangle=s/v//,filenamemangle=s%(?:.*?)?v?(\d[\d.]*)\.tar\.gz%ngx-mod-cache-purge-$1.tar.gz%" \
+ https://github.com/FRiCKLE/ngx_cache_purge/tags \
+ (?:.*?/)?v?(\d[\d.]*)\.tar\.gz debian debian/modules/uupdate http-cache-purge ngx-mod-cache-purge
diff --git a/debian/modules/watch/http-dav-ext b/debian/modules/watch/http-dav-ext
new file mode 100644
index 0000000..d704cfc
--- /dev/null
+++ b/debian/modules/watch/http-dav-ext
@@ -0,0 +1,4 @@
+version=4
+opts="dversionmangle=s/v//,filenamemangle=s%(?:.*?)?v?(\d[\d.]*)\.tar\.gz%ngx-mod-dav-ext-$1.tar.gz%" \
+ https://github.com/arut/nginx-dav-ext-module/tags \
+ (?:.*?/)?v?(\d[\d.]*)\.tar\.gz debian debian/modules/uupdate http-dav-ext ngx-mod-dav-ext
diff --git a/debian/modules/watch/http-echo b/debian/modules/watch/http-echo
new file mode 100644
index 0000000..615532b
--- /dev/null
+++ b/debian/modules/watch/http-echo
@@ -0,0 +1,4 @@
+version=4
+opts="dversionmangle=s/v//,filenamemangle=s%(?:.*?)?v?(\d[\d.]*)\.tar\.gz%ngx-mod-echo-$1.tar.gz%" \
+ https://github.com/agentzh/echo-nginx-module/tags \
+ (?:.*?/)?v?(\d[\d.]*)\.tar\.gz debian debian/modules/uupdate http-echo ngx-mod-echo
diff --git a/debian/modules/watch/http-fancyindex b/debian/modules/watch/http-fancyindex
new file mode 100644
index 0000000..72435fd
--- /dev/null
+++ b/debian/modules/watch/http-fancyindex
@@ -0,0 +1,4 @@
+version=4
+opts="dversionmangle=s/v//,filenamemangle=s%(?:.*?)?v?(\d[\d.]*)\.tar\.gz%ngx-mod-fancyindex-$1.tar.gz%" \
+ https://github.com/aperezdc/ngx-fancyindex/tags \
+ (?:.*?/)?v?(\d[\d.]*)\.tar\.gz debian debian/modules/uupdate http-fancyindex ngx-mod-fancyindex
diff --git a/debian/modules/watch/http-headers-more-filter b/debian/modules/watch/http-headers-more-filter
new file mode 100644
index 0000000..cfe5fc9
--- /dev/null
+++ b/debian/modules/watch/http-headers-more-filter
@@ -0,0 +1,4 @@
+version=4
+opts="uversionmangle=s/0.261/0.26.1/,dversionmangle=s/v//,filenamemangle=s%(?:.*?)?v?(\d[\d.]*)\.tar\.gz%ngx-mod-headers-more-$1.tar.gz%" \
+ https://github.com/agentzh/headers-more-nginx-module/tags \
+ (?:.*?/)?v?(\d[\d.]*)\.tar\.gz debian debian/modules/uupdate http-headers-more-filter ngx-mod-headers-more
diff --git a/debian/modules/watch/http-lua b/debian/modules/watch/http-lua
new file mode 100644
index 0000000..7f1718f
--- /dev/null
+++ b/debian/modules/watch/http-lua
@@ -0,0 +1,4 @@
+version=4
+opts="dversionmangle=s/v//,filenamemangle=s%(?:.*?)?v?(\d[\d.]*)\.tar\.gz%lua-nginx-module-$1.tar.gz%" \
+ https://github.com/openresty/lua-nginx-module/tags \
+ (?:.*?/)?v?(\d[\d.]*)\.tar\.gz debian debian/modules/uupdate http-lua lua-nginx-module
diff --git a/debian/modules/watch/http-ndk b/debian/modules/watch/http-ndk
new file mode 100644
index 0000000..43f0cbd
--- /dev/null
+++ b/debian/modules/watch/http-ndk
@@ -0,0 +1,4 @@
+version=4
+opts="dversionmangle=s/v//,filenamemangle=s%(?:.*?)?v?(\d[\d.]*)\.tar\.gz%ngx-mod-devel-kit-$1.tar.gz%" \
+ https://github.com/simpl/ngx_devel_kit/tags \
+ (?:.*?/)?v?(\d[\d.]*)\.tar\.gz debian debian/modules/uupdate http-ndk ngx-mod-devel-kit
diff --git a/debian/modules/watch/http-subs-filter b/debian/modules/watch/http-subs-filter
new file mode 100644
index 0000000..cca3e05
--- /dev/null
+++ b/debian/modules/watch/http-subs-filter
@@ -0,0 +1,4 @@
+version=4
+opts="dversionmangle=s/v//,filenamemangle=s%(?:.*?)?v?(\d[\d.]*)\.tar\.gz%ngx-mod-http-substitutions-filter-$1.tar.gz%" \
+ https://github.com/yaoweibin/ngx_http_substitutions_filter_module/tags \
+ (?:.*?/)?v?(\d[\d.]*)\.tar\.gz debian debian/modules/uupdate http-subs-filter ngx-mod-http-substitutions-filter
diff --git a/debian/modules/watch/http-uploadprogress b/debian/modules/watch/http-uploadprogress
new file mode 100644
index 0000000..7c0b7cf
--- /dev/null
+++ b/debian/modules/watch/http-uploadprogress
@@ -0,0 +1,4 @@
+version=4
+opts="dversionmangle=s/v//,filenamemangle=s%(?:.*?)?v?(\d[\d.]*)\.tar\.gz%ngx-mod-upload-progress-$1.tar.gz%" \
+ https://github.com/masterzen/nginx-upload-progress-module/tags \
+ (?:.*?/)?v?(\d[\d.]*)\.tar\.gz debian debian/modules/uupdate http-uploadprogress ngx-mod-upload-progress
diff --git a/debian/modules/watch/nchan b/debian/modules/watch/nchan
new file mode 100644
index 0000000..6b95884
--- /dev/null
+++ b/debian/modules/watch/nchan
@@ -0,0 +1,4 @@
+version=4
+opts="dversionmangle=s/v//,filenamemangle=s%(?:.*?)?v?(\d[\d.]*)\.tar\.gz%ngx-mod-nchan-$1.tar.gz%" \
+ https://github.com/slact/nchan/tags \
+ (?:.*?/)?v?(\d[\d.]*)\.tar\.gz debian debian/modules/uupdate nchan ngx-mod-nchan
diff --git a/debian/modules/watch/rtmp b/debian/modules/watch/rtmp
new file mode 100644
index 0000000..f8137a2
--- /dev/null
+++ b/debian/modules/watch/rtmp
@@ -0,0 +1,4 @@
+version=4
+opts="dversionmangle=s/v//,filenamemangle=s%(?:.*?)?v?(\d[\d.]*)\.tar\.gz%ngx-mod-rtmp-$1.tar.gz%" \
+ https://github.com/arut/nginx-rtmp-module/tags \
+ (?:.*?/)?v?(\d[\d.]*)\.tar\.gz debian debian/modules/uupdate rtmp ngx-mod-rtmp
diff --git a/debian/ngxmod b/debian/ngxmod
new file mode 100755
index 0000000..f8180d1
--- /dev/null
+++ b/debian/ngxmod
@@ -0,0 +1,156 @@
+#!/usr/bin/env python
+'''
+ngxmod - nginx modules management helper script
+
+Usage:
+ ngxmod <subcommand>
+
+Subcommands:
+
+ uscan - scan/watch upstream nginx module sources for new releases
+
+ It runs uscan for each nginx module listed in debian/modules/control.
+
+ If a new version is available, it will ask you if you want to upgrade.
+ If you agree it will use uscan to download the tarball, place it in the nginx
+ source's parent directory and unpack it using modules/uupdate to upgrade
+ the module's source.
+
+ After the upgrade, it will remove the files listed in the Files-Excluded
+ field and finally it will commit the changes.
+'''
+
+import deb822
+import os
+import sys
+import re
+from subprocess import call
+
+MODULES_PATH = 'modules/'
+MODULES_CTRL = os.path.join(MODULES_PATH, 'control')
+
+def prompt(query):
+ '''
+ Ask the given query and wait for an y/N answer.
+
+ :query: The query
+ :returns: The yes (True) or no (False) answer
+ '''
+ sys.stdout.write('%s [y/N]: ' % query)
+ choice = raw_input().lower()
+
+ return choice == 'y'
+
+
+def upgrade_module(mi, watchfile):
+ '''
+ Upgrade the given module using uscan and the custom modules/uupdate.
+ After the upgrade, remove the Files-Excluded.
+
+ :param mi: modules/control info for the module
+ :param watchfile: The watchfile of the module
+ :returns: The new module version
+ '''
+ uscan_upgrade_cmd = [
+ 'uscan', '--upstream-version', mi['Version'],
+ '--watchfile', 'debian/%s' % watchfile, '--no-symlink'
+ ]
+ upgrade_ret = call(uscan_upgrade_cmd, stdout=open(os.devnull, 'w'))
+
+ if 'Files-Excluded' in mi and upgrade_ret == 0:
+ print 'Removing Files-Excluded: %s' % mi['Files-Excluded']
+
+ files = re.split('\s+', mi['Files-Excluded'])
+ files = filter(None, files)
+ abs_files = ['%s%s/%s' % (MODULES_PATH, mi['Module'], f) for f in files]
+
+ rm_cmd = ['rm', '-r']
+ rm_cmd.extend(abs_files)
+
+ call(rm_cmd)
+
+ return open('/tmp/ngx_uupdate_version_%s' % mi['Module']).read().rstrip()
+
+
+def update_module_version(mi, new_ver):
+ '''
+ Update the given module's version in modules/control.
+
+ :param mi: modules/control info for the module
+ :param new_ver: The new module version
+ '''
+ modules_info = deb822.Deb822.iter_paragraphs(file(MODULES_CTRL))
+ modules_ctrl_new = '%s%s' % (MODULES_CTRL, '.new')
+ with open(modules_ctrl_new, 'w') as c:
+ for m in modules_info:
+ if mi['Module'] == m['Module']:
+ m['Version'] = re.sub('[^v]+', new_ver, m['Version'])
+
+ c.write(m.dump() + '\n')
+ c.closed
+ os.rename(modules_ctrl_new, MODULES_CTRL)
+
+
+def commit_module(mi, new_ver):
+ '''
+ Git-commit the given module's upgraded source.
+
+ :param mi: modules/control info for the module
+ :param new_ver: The new module version
+ '''
+ module_name = mi['Module']
+ module_path = os.path.join(MODULES_PATH, module_name)
+
+ git_add_cmd = ['git', 'add', module_path, MODULES_CTRL]
+ call(git_add_cmd)
+
+ commit_msg = 'Update %s to v%s' % (module_name, new_ver)
+ git_commit_cmd = ['git', 'commit', '-m', commit_msg]
+ call(git_commit_cmd)
+
+
+def usage():
+ print("""
+Usage:
+ ngxmod <subcommand>
+
+Subcommands:
+
+ uscan - scan/watch upstream nginx module sources for new releases
+""")
+
+
+def main():
+ modules_info = deb822.Deb822.iter_paragraphs(file(MODULES_CTRL))
+
+ for mi in modules_info:
+ sys.stdout.write('Uscanning %s...' % mi['Module'])
+ sys.stdout.flush()
+
+ watchfile = os.path.join(MODULES_PATH, 'watch', mi['Module'])
+
+ if not os.path.isfile(watchfile):
+ print 'no watchfile available.'
+ continue
+
+ uscan_check_cmd = [
+ 'uscan', '--upstream-version', mi['Version'],
+ '--watchfile', watchfile, '--package',
+ mi['Module']
+ ]
+ check_ret = call(uscan_check_cmd)
+
+ if check_ret == 0:
+ if prompt('Do you want to upgrade %s?' % mi['Module']):
+ new_module_ver = upgrade_module(mi, watchfile)
+ update_module_version(mi, new_module_ver)
+ commit_module(mi, new_module_ver)
+ else:
+ print 'up-to-date.'
+
+
+if __name__ == '__main__':
+ if len(sys.argv) == 2 and sys.argv[1] == 'uscan':
+ main()
+ else:
+ usage()
--
2.11.0
