Re: GSoC 17: Release notes extension

[Rishabh Madan]

On Sun, Mar 26, 2017 at 10:55 PM, Gregory Szorc 
wrote:

> On Thu, Mar 23, 2017 at 10:02 PM, Martin von Zweigbergk <
> martinv...@google.com> wrote:
>
>> As an alternative to compiling the release notes from commit messages, we
>> could do what Git does, which is to keep a file per release
>> (Documentation/Relnotes/.txt in their case) that we update
>> every now and then. I think Junio (the maintainer) generally updates that
>> file. We don't have a single maintainer, but we could accept patches to
>> from anyone of course.
>>
>
> Yes, I think having release notes files in the repo is a good idea. If we
> did this, we could start suggesting people update the release notes when
> writing patches.
>
> Of course, there would likely be merge conflicts for whoever queues
> things. There is a "union" merge tool suitable for this case. And there is
> a [merge-patterns] that can be used to specify which tool to use for which
> filenames. But I feel there is a missing feature for a .hgmergepatterns
> file or some such.
>

  I think there is provision for configuring [merge-patterns]. See
https://www.mercurial-scm.org/wiki/MergeToolConfiguration

>
>
>>
>> We could also keep the current process, which is to keep the release
>> notes on the wiki (I just started on https://www.mercurial-scm.org/
>> wiki/Release4.2 yesterday).
>>
>> I did like Greg's extension, but I wonder if it's worth the trouble of
>> having each contributor write part of the release notes. I wonder if we
>> won't want to restructure it enough afterwards anyway that I'm not sure the
>> automated step is worth the trouble. I understand that it helps us not
>> forget to include things in the release notes, but maybe it would be as
>> effective to add a new commit message tag like (RN) in addition to the (BC)
>> and (API) we already have.
>>
>
> This is the path I initially went down. I quickly realized that commit
> message summary lines weren't sufficient for release notes. They are often
> just not appropriate for end users or not detailed enough. This
> necessitates some kind of content elsewhere in the commit to denote release
> notes. This is how the use of ReST admonitions was arrived at.
>
> Regarding restructuring afterwards, my thoughts were the workflow would
> look something like this:
>
> 1. Changeset author adds release notes fragment to commit message
> 2. Person queuing changesets runs `hg releasenotes` to update a release
> notes file in repo
> 3. Later, someone tweaks wording in release notes file
> 4. Subsequent `hg releasenotes` invocations automatically reconcile minor
> differences between commit message and file and give precedence to newer
> content in file.
>

This approach to form release notes seems nice to me.

>
>>
>> On Thu, Mar 23, 2017 at 9:14 PM, Gregory Szorc 
>> wrote:
>>
>>> On Mon, Mar 20, 2017 at 6:04 PM, Rishabh Madan >> > wrote:
>>>
 Hello,

 I am Rishabh Madan and I will be a GSoC applicant this year under
 Mercurial.

 I wanted to discuss about the Release notes extension project.[1]
 
 I have gone through the mailing list discussions. I am currently
 understanding the intial patch sent by Gregory Szorc, for the releasenotes
 extension.

>>>
>>> I hacked on this for an hour or two several days ago. A newer version is
>>> at https://hg.mozilla.org/users/gszorc_mozilla.com/hg/rev/releasenotes.
>>> Not much has fundamentally changed: I was just trying to make commit
>>> message parsing more robust.
>>>
>>>

 I would like to get a few inputs for this project:

 1. Commit message parser
 (i) I feel this is the core of the idea. From what I understand, we
 will have to keep a specific format for the commit messages that will be
 parsed and output will be stored in a changelog. But, this is a extension
 that would help other organizations as well. Do we expect them to abide by
 certain format for the commit message?

>>>
>>> For this idea to work, you have to standardize on *something*. I chose
>>> to use ReStructured Text admonitions (".. directive:: ...") because a)
>>> Mercurial already has a basic ReST parser and it was easy to leverage ReST
>>> b) ReST is extensible and therefore is a natural fit for this use case.
>>> There's no reason other formats couldn't be leveraged. But for a minimal
>>> viable product, I'd focus on ReST.
>>>
>>> The code as currently written converts extracted release notes fragments
>>> to Python data structures. So it should be possible to plug in other
>>> formats/parsers easily enough.
>>>
>>>
 (ii) Is this going to be a hybrid process? I mean, will the notes once
 created be edited by maintainers at a later stage?

>>>
>>> This isn't implemented in my proof-of-concept yet, but yes, we
>>> absolutely need to support edits. The way I envision this being 

Re: GSoC 17: Release notes extension

[Gregory Szorc]

On Thu, Mar 23, 2017 at 10:02 PM, Martin von Zweigbergk <
martinv...@google.com> wrote:

> As an alternative to compiling the release notes from commit messages, we
> could do what Git does, which is to keep a file per release
> (Documentation/Relnotes/.txt in their case) that we update every
> now and then. I think Junio (the maintainer) generally updates that file.
> We don't have a single maintainer, but we could accept patches to from
> anyone of course.
>

Yes, I think having release notes files in the repo is a good idea. If we
did this, we could start suggesting people update the release notes when
writing patches.

Of course, there would likely be merge conflicts for whoever queues things.
There is a "union" merge tool suitable for this case. And there is a
[merge-patterns] that can be used to specify which tool to use for which
filenames. But I feel there is a missing feature for a .hgmergepatterns
file or some such.


>
> We could also keep the current process, which is to keep the release notes
> on the wiki (I just started on https://www.mercurial-scm.org/
> wiki/Release4.2 yesterday).
>
> I did like Greg's extension, but I wonder if it's worth the trouble of
> having each contributor write part of the release notes. I wonder if we
> won't want to restructure it enough afterwards anyway that I'm not sure the
> automated step is worth the trouble. I understand that it helps us not
> forget to include things in the release notes, but maybe it would be as
> effective to add a new commit message tag like (RN) in addition to the (BC)
> and (API) we already have.
>

This is the path I initially went down. I quickly realized that commit
message summary lines weren't sufficient for release notes. They are often
just not appropriate for end users or not detailed enough. This
necessitates some kind of content elsewhere in the commit to denote release
notes. This is how the use of ReST admonitions was arrived at.

Regarding restructuring afterwards, my thoughts were the workflow would
look something like this:

1. Changeset author adds release notes fragment to commit message
2. Person queuing changesets runs `hg releasenotes` to update a release
notes file in repo
3. Later, someone tweaks wording in release notes file
4. Subsequent `hg releasenotes` invocations automatically reconcile minor
differences between commit message and file and give precedence to newer
content in file.


>
>
> On Thu, Mar 23, 2017 at 9:14 PM, Gregory Szorc 
> wrote:
>
>> On Mon, Mar 20, 2017 at 6:04 PM, Rishabh Madan 
>> wrote:
>>
>>> Hello,
>>>
>>> I am Rishabh Madan and I will be a GSoC applicant this year under
>>> Mercurial.
>>>
>>> I wanted to discuss about the Release notes extension project.[1]
>>> 
>>> I have gone through the mailing list discussions. I am currently
>>> understanding the intial patch sent by Gregory Szorc, for the releasenotes
>>> extension.
>>>
>>
>> I hacked on this for an hour or two several days ago. A newer version is
>> at https://hg.mozilla.org/users/gszorc_mozilla.com/hg/rev/releasenotes.
>> Not much has fundamentally changed: I was just trying to make commit
>> message parsing more robust.
>>
>>
>>>
>>> I would like to get a few inputs for this project:
>>>
>>> 1. Commit message parser
>>> (i) I feel this is the core of the idea. From what I understand, we will
>>> have to keep a specific format for the commit messages that will be parsed
>>> and output will be stored in a changelog. But, this is a extension that
>>> would help other organizations as well. Do we expect them to abide by
>>> certain format for the commit message?
>>>
>>
>> For this idea to work, you have to standardize on *something*. I chose to
>> use ReStructured Text admonitions (".. directive:: ...") because a)
>> Mercurial already has a basic ReST parser and it was easy to leverage ReST
>> b) ReST is extensible and therefore is a natural fit for this use case.
>> There's no reason other formats couldn't be leveraged. But for a minimal
>> viable product, I'd focus on ReST.
>>
>> The code as currently written converts extracted release notes fragments
>> to Python data structures. So it should be possible to plug in other
>> formats/parsers easily enough.
>>
>>
>>> (ii) Is this going to be a hybrid process? I mean, will the notes once
>>> created be edited by maintainers at a later stage?
>>>
>>
>> This isn't implemented in my proof-of-concept yet, but yes, we absolutely
>> need to support edits. The way I envision this being implemented is that
>> there is some kind of similarity detection to reconcile differences between
>> release notes fragments in commit messages and release notes files.
>> Currently, the extension parses commit messages and a release notes file to
>> a common data structure type, merges/unions the data, and emits a new
>> release notes file. We need to add a step that attempts to match edited
>> 

Re: GSoC 17: Release notes extension

[Rishabh Madan]

On Fri, Mar 24, 2017 at 10:32 AM, Martin von Zweigbergk <
martinv...@google.com> wrote:

> As an alternative to compiling the release notes from commit messages, we
> could do what Git does, which is to keep a file per release
> (Documentation/Relnotes/.txt in their case) that we update every
> now and then. I think Junio (the maintainer) generally updates that file.
> We don't have a single maintainer, but we could accept patches to from
> anyone of course.
>
> I think that is an organisation specific practice. We should keep in mind
that this extension if shipped with core would not only be specific to
Mercurial but other people using mercurial might also use this. Also, I
feel that having an automated (accompanied with copyediting) process for
generating release notes would definitely improve the quality of it since,
a lot of things get left out when writing these notes. For example, some
things might have been left out in the relnotes for 4.2 considering the
time gap between release of 4.1 and 4.2.

We could also keep the current process, which is to keep the release notes
> on the wiki (I just started on https://www.mercurial-scm.org/
> wiki/Release4.2 yesterday).
>
> I did like Greg's extension, but I wonder if it's worth the trouble of
> having each contributor write part of the release notes. I wonder if we
> won't want to restructure it enough afterwards anyway that I'm not sure the
> automated step is worth the trouble. I understand that it helps us not
> forget to include things in the release notes, but maybe it would be as
> effective to add a new commit message tag like (RN) in addition to the (BC)
> and (API) we already have.
>


> On Thu, Mar 23, 2017 at 9:14 PM, Gregory Szorc 
> wrote:
>
>> On Mon, Mar 20, 2017 at 6:04 PM, Rishabh Madan 
>> wrote:
>>
>>> Hello,
>>>
>>> I am Rishabh Madan and I will be a GSoC applicant this year under
>>> Mercurial.
>>>
>>> I wanted to discuss about the Release notes extension project.[1]
>>> 
>>> I have gone through the mailing list discussions. I am currently
>>> understanding the intial patch sent by Gregory Szorc, for the releasenotes
>>> extension.
>>>
>>
>> I hacked on this for an hour or two several days ago. A newer version is
>> at https://hg.mozilla.org/users/gszorc_mozilla.com/hg/rev/releasenotes.
>> Not much has fundamentally changed: I was just trying to make commit
>> message parsing more robust.
>>
>>
>>>
>>> I would like to get a few inputs for this project:
>>>
>>> 1. Commit message parser
>>> (i) I feel this is the core of the idea. From what I understand, we will
>>> have to keep a specific format for the commit messages that will be parsed
>>> and output will be stored in a changelog. But, this is a extension that
>>> would help other organizations as well. Do we expect them to abide by
>>> certain format for the commit message?
>>>
>>
>> For this idea to work, you have to standardize on *something*. I chose to
>> use ReStructured Text admonitions (".. directive:: ...") because a)
>> Mercurial already has a basic ReST parser and it was easy to leverage ReST
>> b) ReST is extensible and therefore is a natural fit for this use case.
>> There's no reason other formats couldn't be leveraged. But for a minimal
>> viable product, I'd focus on ReST.
>>
>> The code as currently written converts extracted release notes fragments
>> to Python data structures. So it should be possible to plug in other
>> formats/parsers easily enough.
>>
>>
>>> (ii) Is this going to be a hybrid process? I mean, will the notes once
>>> created be edited by maintainers at a later stage?
>>>
>>
>> This isn't implemented in my proof-of-concept yet, but yes, we absolutely
>> need to support edits. The way I envision this being implemented is that
>> there is some kind of similarity detection to reconcile differences between
>> release notes fragments in commit messages and release notes files.
>> Currently, the extension parses commit messages and a release notes file to
>> a common data structure type, merges/unions the data, and emits a new
>> release notes file. We need to add a step that attempts to match edited
>> release notes file entries to their original commit message fragments so
>> duplicates aren't produced. This can likely utilize the APIs already in
>> Mercurial for identifying similar content.
>>
>>
>>> (iii) The extension shouldn't add extremely small changesets to the
>>> changelog. I think this can be done by adding some keyword to the commit
>>> message (like [IGNORERST]).
>>>
>>
>> As currently implemented, a release notes entry is only produced if the
>> commit message contains special syntax - a ReST admonition. This avoids the
>> problem of "extremely small changesets" polluting the release notes. But it
>> has the drawback of requiring changeset authors to remember to write a
>> release notes admonition. This trade-off is arguably 

Re: GSoC 17: Release notes extension

[Rishabh Madan]

>
> For now, I'd assume we'd implement our own parser and if others want
> other formats, they'd need to send us patches.


Yes, we can do that, or once we have a workable parser ready we can also
have some default templates that the user can configure/set in the
.hgrc file.

 Another interesting approach is towncrier:
> https://github.com/hawkowl/towncrier though I'm not in love with the
> approach of commiting lots of short-lived files to the repository (I
> much prefer having it in the commit message).


Thank you for this reference. I had a look at this project and I think we
can get some good ideas from here, to begin with,
ᐧ
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

Re: GSoC 17: Release notes extension

[Martin von Zweigbergk via Mercurial-devel]

As an alternative to compiling the release notes from commit messages, we
could do what Git does, which is to keep a file per release
(Documentation/Relnotes/.txt in their case) that we update every
now and then. I think Junio (the maintainer) generally updates that file.
We don't have a single maintainer, but we could accept patches to from
anyone of course.

We could also keep the current process, which is to keep the release notes
on the wiki (I just started on https://www.mercurial-scm.org/wiki/Release4.2
yesterday).

I did like Greg's extension, but I wonder if it's worth the trouble of
having each contributor write part of the release notes. I wonder if we
won't want to restructure it enough afterwards anyway that I'm not sure the
automated step is worth the trouble. I understand that it helps us not
forget to include things in the release notes, but maybe it would be as
effective to add a new commit message tag like (RN) in addition to the (BC)
and (API) we already have.


On Thu, Mar 23, 2017 at 9:14 PM, Gregory Szorc 
wrote:

> On Mon, Mar 20, 2017 at 6:04 PM, Rishabh Madan 
> wrote:
>
>> Hello,
>>
>> I am Rishabh Madan and I will be a GSoC applicant this year under
>> Mercurial.
>>
>> I wanted to discuss about the Release notes extension project.[1]
>> 
>> I have gone through the mailing list discussions. I am currently
>> understanding the intial patch sent by Gregory Szorc, for the releasenotes
>> extension.
>>
>
> I hacked on this for an hour or two several days ago. A newer version is
> at https://hg.mozilla.org/users/gszorc_mozilla.com/hg/rev/releasenotes.
> Not much has fundamentally changed: I was just trying to make commit
> message parsing more robust.
>
>
>>
>> I would like to get a few inputs for this project:
>>
>> 1. Commit message parser
>> (i) I feel this is the core of the idea. From what I understand, we will
>> have to keep a specific format for the commit messages that will be parsed
>> and output will be stored in a changelog. But, this is a extension that
>> would help other organizations as well. Do we expect them to abide by
>> certain format for the commit message?
>>
>
> For this idea to work, you have to standardize on *something*. I chose to
> use ReStructured Text admonitions (".. directive:: ...") because a)
> Mercurial already has a basic ReST parser and it was easy to leverage ReST
> b) ReST is extensible and therefore is a natural fit for this use case.
> There's no reason other formats couldn't be leveraged. But for a minimal
> viable product, I'd focus on ReST.
>
> The code as currently written converts extracted release notes fragments
> to Python data structures. So it should be possible to plug in other
> formats/parsers easily enough.
>
>
>> (ii) Is this going to be a hybrid process? I mean, will the notes once
>> created be edited by maintainers at a later stage?
>>
>
> This isn't implemented in my proof-of-concept yet, but yes, we absolutely
> need to support edits. The way I envision this being implemented is that
> there is some kind of similarity detection to reconcile differences between
> release notes fragments in commit messages and release notes files.
> Currently, the extension parses commit messages and a release notes file to
> a common data structure type, merges/unions the data, and emits a new
> release notes file. We need to add a step that attempts to match edited
> release notes file entries to their original commit message fragments so
> duplicates aren't produced. This can likely utilize the APIs already in
> Mercurial for identifying similar content.
>
>
>> (iii) The extension shouldn't add extremely small changesets to the
>> changelog. I think this can be done by adding some keyword to the commit
>> message (like [IGNORERST]).
>>
>
> As currently implemented, a release notes entry is only produced if the
> commit message contains special syntax - a ReST admonition. This avoids the
> problem of "extremely small changesets" polluting the release notes. But it
> has the drawback of requiring changeset authors to remember to write a
> release notes admonition. This trade-off is arguably tailored towards
> Mercurial's workflow, which focuses on writing many, smaller changesets
> (which results in many changesets being irrelevant to release notes). Since
> this extension is being authored to benefit the Mercurial project and since
> many of us feel passionately that our "microcommit" approach is superior to
> other approaches, we should probably focus on this (as opposed to merely
> extracting commit message summary lines to release notes files) :)
>
>
>>
>> 2. Linking hgweb to release notes
>> I found this idea quite intriguing. Can someone please elaborate a bit
>> more on what exactly we're trying to accomplish with this?
>>
>
> I think the idea here is for pages that reference a series or range of
> changesets (such as a tag) to 

Re: GSoC 17: Release notes extension

[Gregory Szorc]

On Mon, Mar 20, 2017 at 6:04 PM, Rishabh Madan 
wrote:

> Hello,
>
> I am Rishabh Madan and I will be a GSoC applicant this year under
> Mercurial.
>
> I wanted to discuss about the Release notes extension project.[1]
> 
> I have gone through the mailing list discussions. I am currently
> understanding the intial patch sent by Gregory Szorc, for the releasenotes
> extension.
>

I hacked on this for an hour or two several days ago. A newer version is at
https://hg.mozilla.org/users/gszorc_mozilla.com/hg/rev/releasenotes. Not
much has fundamentally changed: I was just trying to make commit message
parsing more robust.


>
> I would like to get a few inputs for this project:
>
> 1. Commit message parser
> (i) I feel this is the core of the idea. From what I understand, we will
> have to keep a specific format for the commit messages that will be parsed
> and output will be stored in a changelog. But, this is a extension that
> would help other organizations as well. Do we expect them to abide by
> certain format for the commit message?
>

For this idea to work, you have to standardize on *something*. I chose to
use ReStructured Text admonitions (".. directive:: ...") because a)
Mercurial already has a basic ReST parser and it was easy to leverage ReST
b) ReST is extensible and therefore is a natural fit for this use case.
There's no reason other formats couldn't be leveraged. But for a minimal
viable product, I'd focus on ReST.

The code as currently written converts extracted release notes fragments to
Python data structures. So it should be possible to plug in other
formats/parsers easily enough.


> (ii) Is this going to be a hybrid process? I mean, will the notes once
> created be edited by maintainers at a later stage?
>

This isn't implemented in my proof-of-concept yet, but yes, we absolutely
need to support edits. The way I envision this being implemented is that
there is some kind of similarity detection to reconcile differences between
release notes fragments in commit messages and release notes files.
Currently, the extension parses commit messages and a release notes file to
a common data structure type, merges/unions the data, and emits a new
release notes file. We need to add a step that attempts to match edited
release notes file entries to their original commit message fragments so
duplicates aren't produced. This can likely utilize the APIs already in
Mercurial for identifying similar content.


> (iii) The extension shouldn't add extremely small changesets to the
> changelog. I think this can be done by adding some keyword to the commit
> message (like [IGNORERST]).
>

As currently implemented, a release notes entry is only produced if the
commit message contains special syntax - a ReST admonition. This avoids the
problem of "extremely small changesets" polluting the release notes. But it
has the drawback of requiring changeset authors to remember to write a
release notes admonition. This trade-off is arguably tailored towards
Mercurial's workflow, which focuses on writing many, smaller changesets
(which results in many changesets being irrelevant to release notes). Since
this extension is being authored to benefit the Mercurial project and since
many of us feel passionately that our "microcommit" approach is superior to
other approaches, we should probably focus on this (as opposed to merely
extracting commit message summary lines to release notes files) :)


>
> 2. Linking hgweb to release notes
> I found this idea quite intriguing. Can someone please elaborate a bit
> more on what exactly we're trying to accomplish with this?
>

I think the idea here is for pages that reference a series or range of
changesets (such as a tag) to automatically extract release notes from
changesets in that range. For example, if you load the "4.2" tag page, you
would have a link to or inline rendering of all release notes relevant to
changesets between 4.1 and 4.2.

There is a lot of ambiguity in this feature. For example, how do you handle
edits to release notes files? This would require the server to define where
release notes files are stored in the repo. Otherwise, the hgweb view would
only render commit message content, which may not be complete! For this
reason, I think hgweb support should be a secondary goal of the project.


>
> I also found some open source implementations like [2]
>  similar to this
> idea that might help in implementing this project.
>

Yes, there is a lot of prior art in this topic. We can and should look to
others for inspiration for good ideas.

One thing to keep in mind is that many of these solutions simply assemble
*all* changes. For projects with ~1000 commits every release (like
Mercurial), listing every change is too noisy: you need to curate the
information in release notes or people will get lost in the noise. This is
why the 

GSoC 17: Release notes extension

[Rishabh Madan]

Hello,

I am Rishabh Madan and I will be a GSoC applicant this year under Mercurial.

I wanted to discuss about the Release notes extension project.[1]

I have gone through the mailing list discussions. I am currently
understanding the intial patch sent by Gregory Szorc, for the releasenotes
extension.

I would like to get a few inputs for this project:

1. Commit message parser
(i) I feel this is the core of the idea. From what I understand, we will
have to keep a specific format for the commit messages that will be parsed
and output will be stored in a changelog. But, this is a extension that
would help other organizations as well. Do we expect them to abide by
certain format for the commit message?
(ii) Is this going to be a hybrid process? I mean, will the notes once
created be edited by maintainers at a later stage?
(iii) The extension shouldn't add extremely small changesets to the
changelog. I think this can be done by adding some keyword to the commit
message (like [IGNORERST]).

2. Linking hgweb to release notes
I found this idea quite intriguing. Can someone please elaborate a bit more
on what exactly we're trying to accomplish with this?

I also found some open source implementations like [2]
 similar to this
idea that might help in implementing this project.


I would really appreciate if someone can point out the issues and features
I may have missed. Also, any relevant resources or links for further
readings would help too.

Thanks.
Rishabh Madan
ᐧ
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel