As promised on IRC last Friday, here is a POC implementation of the
"generate changelog from commit messages" functionality, added to
release.py (I'm not very experienced in Python; I mainly depend on
google, SO, copy-paste, ... so don't focus on coding style etc).
This patch is intended to enable further discussion of this idea
(which was already discussed on this list 4 years ago [1]).
The idea is that we agree on some structured syntax for adding
(optionally) a changelog text to commit messages, to make it easier
for RMs to generate the text for CHANGES (and push the responsibility
for writing a good CHANGES entry first and foremost to the original
committer; and keep the relevant info coupled with the commit). RMs
can use the output of write-changelog as a draft, they can still edit
it, summarize items, shuffle things around, ... But it gives them a
rough version to start from.
Proposal for changelog syntax in commit messages (as implemented in this patch):
[[[
Changelog lines are lines with the following format:
'['<visibility>:<section>']' <message>
where <visibility> = U (User-visible) or D (Developer-visible)
<section> =
general|major|minor|client|server|client-server|other|api|bindings
(section is treated case-insensitively)
<message> = the actual text for CHANGES
Examples:
[U:major] Better interactive conflict resolution for tree conflicts
[U:minor] ra_serf: Adjustments for serf versions with HTTP/2 support
[U:client] Fix 'svn diff URL@REV WC' wrongly looks up URL@HEAD (issue #4597)
[U:client-server] Fix bug with canonicalizing Window-specific
drive-relative URL
[D:api] New svn_ra_list() API function
[D:bindings] JavaHL: Allow access to constructors of a couple JavaHL classes
Q: Shorter prefix syntax ([U:C], [U:CS], [U:MJ], ...) to keep lines
short, or longer (and put message on next line) to make it more
human-readable? While making it easily rememberable for devs so they
don't have to look it up all the time when they just want to commit
...
Q: What to do with merged revisions? Use 'log -g', or make sure
relevant changelog entry is part of the commit message of the merge? I
vote for the latter, it's simpler and has less surprises. In case of
feature branches, a generic "Add feature foo" message on the
reintegrate commit usually suffices. In case of backports perhaps our
backport script can scrape the relevant changelog entries from the
revisions-to-be-merged and add them to the commit message of the
merge.
]]]
To get a rough idea: since we don't have any commit messages
containing such lines in our history, I've added a --pocfirstlines
option, which just takes the first line of every log message (ignoring
lines with 'STATUS', 'CHANGES', 'bump', or starting with '*'), putting
them in the "User -> General" section.
Here is the usage output:
[[[
$ ./release.py write-changelog -h
usage: release.py write-changelog [-h] [--pocfirstlines] branch previous
positional arguments:
branch The branch (or tag or trunk), relative to ^/subversion/, of
which to generate the changelog, when compared to
"previous".
previous The "previous" branch or tag, relative to ^/subversion/, to
compare "branch" against.
optional arguments:
-h, --help show this help message and exit
--pocfirstlines Proof of concept: just take the first line of each relevant
commit messages (except if it contains 'STATUS', 'CHANGES'
or 'bump' or starts with '*'), and put it in User:General.
]]]
Example output:
[[[
$ ./release.py write-changelog --pocfirstlines branches/1.9.x tags/1.9.7
User-visible changes:
- General:
* Merge r1804691 and r1804692 from trunk: (r1804698)
- Client-side bugfixes:
(none)
- Server-side bugfixes:
(none)
- Bindings bugfixes:
(none)
Developer-visible changes:
- General:
(none)
- API changes:
(none)
$ ./release.py write-changelog --pocfirstlines branches/1.8.x tags/1.8.19
User-visible changes:
- General:
* Merge r1804691 from trunk: (r1804723)
* On the 1.8.x branch: Merge r1804692 from trunk. (r1804737)
- Client-side bugfixes:
(none)
- Server-side bugfixes:
(none)
- Bindings bugfixes:
(none)
Developer-visible changes:
- General:
(none)
- API changes:
(none)
$ ./release.py write-changelog --pocfirstlines trunk tags/1.9.7
User-visible changes:
- General:
* A bug fix and minor tweaks in 'svnmover'. (r1715781)
* A cosmetic tweak: add a final comma to lists of tests in a few
test files (r1706965)
* A few FSFS-only tests apply to FSX just as well. So, run them
for (r1667524)
* A follow-up to r1715354. (r1715358)
* A minor code cleanup in FSFS. (r1740722)
* A minor tweak in 'svnmover'. (r1717793)
* A small step towards making 'svnmover merge' operate into a new
temporary (r1717957)
* Abbreviate the potentially rather long list of revisions shown
for tree (r1736063)
* Actually use some helpful error messaging that we bother to
create in (r1683161)
* Add "merge_" prefix to the names of conflict resolver merge test
sandboxes. (r1749457)
* Add '--include' and '--exclude' options to 'svnadmin dump'. (r1811992)
* Add '--search' option support to 'svnbench null-list'. (r1767202)
* Add 'http-compression=auto' mode on the client, now used by
default. (r1803899)
...
]]]
Thoughts?
[1]
https://lists.apache.org/thread.html/c80dd19a7bbafc4f535382b3f361f76ba6535ab3d447a8b988594bfc@1377814810@%3Cdev.subversion.apache.org%3E
--
Johan
Index: release.py
===================================================================
--- release.py (revision 1817073)
+++ release.py (working copy)
@@ -1174,6 +1174,124 @@
fd.seek(0)
subprocess.check_call(['gpg', '--import'], stdin=fd)
+def add_to_changes_dict(changes_dict, section, change, revision):
+ if section in changes_dict:
+ changes = changes_dict[section]
+ if change in changes:
+ revset = changes[change]
+ revset.add(revision)
+ else:
+ changes[change] = set([revision])
+ else:
+ changes_dict[section] = dict()
+ changes_dict[section][change] = set([revision])
+
+def print_section(changes_dict, section, title, mandatory=False):
+ if mandatory or (section in changes_dict):
+ print(' - %s:' % title)
+
+ if section in changes_dict:
+ print_changes(changes_dict[section])
+ elif mandatory:
+ print(' (none)')
+
+def print_changes(changes):
+ # Print in alphabetical order, so entries with the same prefix are together
+ for change in sorted(changes):
+ revs = changes[change]
+ rev_string = 'r' + str(min(revs)) + (' et al' if len(revs) > 1 else '')
+ print(' * %s (%s)' % (change, rev_string))
+
+def write_changelog(args):
+ 'Write changelog, parsed from commit messages'
+ branch = secure_repos + '/' + args.branch
+ previous = secure_repos + '/' + args.previous
+ poc = args.pocfirstlines
+
+ mergeinfo = subprocess.check_output(['svn', 'mergeinfo', '--show-revs',
+ 'eligible', '--log', branch, previous]).splitlines()
+
+ separator_pattern = re.compile('^-{72}$')
+ revline_pattern = re.compile('^r(\d+) \| \w+ \| [^\|]+ \| \d+ lines?$')
+ # Changelog lines are lines with the following format:
+ # '['<visibility>:<section>']' <message>
+ # where <visibility> = U (User-visible) or D (Developer-visible)
+ # <section> =
general|major|minor|client|server|client-server|other|api|bindings
+ # (section is treated case-insensitively)
+ # <message> = the actual text for CHANGES
+ #
+ # Examples:
+ # [U:major] Better interactive conflict resolution for tree conflicts
+ # [U:minor] ra_serf: Adjustments for serf versions with HTTP/2 support
+ # [U:client] Fix 'svn diff URL@REV WC' wrongly looks up URL@HEAD (issue
#4597)
+ # [U:client-server] Fix bug with canonicalizing Window-specific
drive-relative URL
+ # [D:api] New svn_ra_list() API function
+ # [D:bindings] JavaHL: Allow access to constructors of a couple JavaHL
classes
+ #
+ ### TODO: Support continuation of changelog message on multiple lines
+ ### TODO: Shorter prefix syntax ([U:C], [U:CS], [U:MJ], ...) to keep lines
short,
+ ### or longer (and put message on next line) to make it more
human-readable?
+ ### While making it easily rememberable for devs so they don't have
to look
+ ### it up all the time when they just want to commit ...
+ changelog_pattern = re.compile('^\[(U|D):([^\]]+)\](.*)$')
+
+ user_changes = dict() # section -> (change -> set(revision))
+ dev_changes = dict() # section -> (change -> set(revision))
+ revision = -1
+ poc_get_nextline = False
+
+ for line in mergeinfo:
+ if separator_pattern.match(line):
+ revision = -1
+ continue
+
+ if line == '':
+ continue
+
+ if poc_get_nextline:
+ poc_get_nextline = False
+ if not re.search('status|changes|bump|^\*', line, re.IGNORECASE):
+ add_to_changes_dict(user_changes, 'general', line, revision)
+ continue
+
+ revmatch = revline_pattern.match(line)
+ if revmatch != None and revision == -1:
+ # A revision line: get the revision number; reset changelog_lines
+ revision = int(revmatch.group(1))
+ logging.debug('Changelog processing revision r%d' % revision)
+ if poc:
+ poc_get_nextline = True
+ continue
+
+ logmatch = changelog_pattern.match(line)
+ if logmatch != None:
+ # A changelog line: get visibility, section and rest of the line.
+ visibility = logmatch.group(1).upper()
+ section = logmatch.group(2).lower()
+ change = logmatch.group(3).strip()
+ if visibility == 'U':
+ add_to_changes_dict(user_changes, section, change, revision)
+ if visibility == 'D':
+ add_to_changes_dict(dev_changes, section, change, revision)
+
+ # Output the sorted changelog entries
+ # 1) User-visible changes
+ print(' User-visible changes:')
+ print_section(user_changes, 'general', 'General')
+ print_section(user_changes, 'major', 'Major new features')
+ print_section(user_changes, 'minor', 'Minor new features and improvements')
+ print_section(user_changes, 'client', 'Client-side bugfixes',
mandatory=True)
+ print_section(user_changes, 'server', 'Server-side bugfixes',
mandatory=True)
+ print_section(user_changes, 'client-server', 'Client-side and server-side
bugfixes')
+ print_section(user_changes, 'other', 'Other tool improvements and
bugfixes')
+ print_section(user_changes, 'bindings', 'Bindings bugfixes',
mandatory=True)
+ print
+ # 2) Developer-visible changes
+ print(' Developer-visible changes:')
+ print_section(dev_changes, 'general', 'General', mandatory=True)
+ print_section(dev_changes, 'api', 'API changes', mandatory=True)
+ print_section(dev_changes, 'bindings', 'Bindings')
+
#----------------------------------------------------------------------
# Main entry point for argument parsing and handling
@@ -1338,6 +1456,24 @@
separate subcommand.''')
subparser.set_defaults(func=cleanup)
+ # write-changelog
+ subparser = subparsers.add_parser('write-changelog',
+ help='''Output to stdout changelog entries parsed from
+ commit messages.''')
+ subparser.set_defaults(func=write_changelog)
+ subparser.add_argument('branch',
+ help='''The branch (or tag or trunk), relative to
+ ^/subversion/, of which to generate the
+ changelog, when compared to "previous".''')
+ subparser.add_argument('previous',
+ help='''The "previous" branch or tag, relative to
+ ^/subversion/, to compare "branch" against.''')
+ subparser.add_argument('--pocfirstlines', action='store_true',
default=False,
+ help='''Proof of concept: just take the first line of
+ each relevant commit messages (except if it
+ contains 'STATUS', 'CHANGES' or 'bump' or starts
+ with '*'), and put it in User:General.''')
+
# Parse the arguments
args = parser.parse_args()
