Re: Moving Haddock *development* out of GHC tree
Hello Herbert, I think the pre-commit hook needs to be adjusted; I used to have push rights on master, but I cannot seem to push to ghc-head. Thanks, Edward Excerpts from Herbert Valerio Riedel's message of 2014-08-16 16:34:46 +0100: On 2014-08-16 at 16:59:51 +0200, Mateusz Kowalczyk wrote: [...] Herbert kindly updated the sync-all script that defaults to the new branch so I think we're covered. Minor correction: I did not touch the sync-all script at all. I merely declared a default branch in the .gitmodules file: http://git.haskell.org/ghc.git/commitdiff/03a8003e5d3aec97b3a14b2d3c774aad43e0456e ___ ghc-devs mailing list ghc-devs@haskell.org http://www.haskell.org/mailman/listinfo/ghc-devs
Re: Moving Haddock *development* out of GHC tree
On 08/25/2014 01:21 PM, Alan Kim Zimmerman wrote: What happens in the case of a change to the dev branch of ghc that requires a patch to haddock as well, how does that patch get added to phabricator, or is there a separate process? A case in point is https://phabricator.haskell.org/D157 with matching change at https://github.com/alanz/haddock/tree/wip/landmine-param-family Regards Alan You need to push the patch against the Haddock ghc-head branch and update the submodule reference to point at your patch. I don't think that you need to do anything special for Phabricator unless it does some weird checking out instead of using whatever references GHC points to. -- Mateusz K. ___ ghc-devs mailing list ghc-devs@haskell.org http://www.haskell.org/mailman/listinfo/ghc-devs
Re: Moving Haddock *development* out of GHC tree
Ok thanks. I am travelling at the moment, will try this in a few days. Alan On 26 Aug 2014 11:23 AM, Mateusz Kowalczyk fuuze...@fuuzetsu.co.uk wrote: On 08/25/2014 01:21 PM, Alan Kim Zimmerman wrote: What happens in the case of a change to the dev branch of ghc that requires a patch to haddock as well, how does that patch get added to phabricator, or is there a separate process? A case in point is https://phabricator.haskell.org/D157 with matching change at https://github.com/alanz/haddock/tree/wip/landmine-param-family Regards Alan You need to push the patch against the Haddock ghc-head branch and update the submodule reference to point at your patch. I don't think that you need to do anything special for Phabricator unless it does some weird checking out instead of using whatever references GHC points to. -- Mateusz K. ___ ghc-devs mailing list ghc-devs@haskell.org http://www.haskell.org/mailman/listinfo/ghc-devs ___ ghc-devs mailing list ghc-devs@haskell.org http://www.haskell.org/mailman/listinfo/ghc-devs
Re: Moving Haddock *development* out of GHC tree
On 08/15/2014 04:32 PM, Simon Peyton Jones wrote: Great. Please can what you do be documented clearly somewhere, with a link to that documentation from here https://ghc.haskell.org/trac/ghc/wiki/Repositories, and/or somewhere else suitable? Thanks Simon Nothing on that page needs to change. The only thing that needs documenting is than any GHC dev pushing to Haddock needs to do so on the ‘ghc-head’ branch. I have made a change to the table at [1] and added a note but perhaps there's another place that I need to make a change at that's not immediately obvious. Herbert kindly updated the sync-all script that defaults to the new branch so I think we're covered. Please don't hesitate to ask if you (plural) need help with something here. [1]: https://ghc.haskell.org/trac/ghc/wiki/WorkingConventions/Git/Submodules -- Mateusz K. ___ ghc-devs mailing list ghc-devs@haskell.org http://www.haskell.org/mailman/listinfo/ghc-devs
Re: Moving Haddock *development* out of GHC tree
On 2014-08-16 at 16:59:51 +0200, Mateusz Kowalczyk wrote: [...] Herbert kindly updated the sync-all script that defaults to the new branch so I think we're covered. Minor correction: I did not touch the sync-all script at all. I merely declared a default branch in the .gitmodules file: http://git.haskell.org/ghc.git/commitdiff/03a8003e5d3aec97b3a14b2d3c774aad43e0456e ___ ghc-devs mailing list ghc-devs@haskell.org http://www.haskell.org/mailman/listinfo/ghc-devs
RE: Moving Haddock *development* out of GHC tree
Great. Please can what you do be documented clearly somewhere, with a link to that documentation from here https://ghc.haskell.org/trac/ghc/wiki/Repositories, and/or somewhere else suitable? Thanks Simon | -Original Message- | From: ghc-devs [mailto:ghc-devs-boun...@haskell.org] On Behalf Of | Mateusz Kowalczyk | Sent: 13 August 2014 23:10 | To: ghc-devs@haskell.org | Subject: Re: Moving Haddock *development* out of GHC tree | | On 08/08/2014 06:25 AM, Mateusz Kowalczyk wrote: | Hello, | | [snip] | | Transition from current setup: | If I receive some patches I was promised then I will then make a | 2.14.4 bugfix/compat release make sure that master is up to date and | then create something like GHC-tracking branch from master and track | that. I will then abandon that branch and not push to it unless it is | GHC release time. The next commit in master will bring Haddock to a | state where it works with 7.8.3: yes, this means removing all new API | stuff until 7.10 or 7.8.4 or whatever. GHC API changes go onto | GHC-tracking while all the stuff I write goes master. When GHC makes | a | release or is about to, I make master work with that and make | GHC-tracking point to that instead. | | | Thanks! | | | So it is now close to a week gone and I have received many positive | replies and no negative ones. I will probably execute what I stated | initially at about this time tomorrow. | | To reiterate in short: | | 1. I make sure what we have now compiles with GHC HEAD and I stick it | in separate branch which GHC folk will now track and apply any API | patches to. Unless something changes by tomorrow, this will most likely | be what master is at right now, perhaps with a single change to the | version in cabal file. | | 2. I make the master branch work with 7.8.3 (and possibly 7.8.x) and do | development without worrying about any API changes in HEAD, releasing | as often as I need to. | | 3. At GHC release time, I update master with API changes so that up-to- | date Haddock is ready to be used to generate the docs and ship with the | compiler. | | I don't know what the GHC branch name will be yet. 'ghc-head' makes | most sense but IIRC Herbert had some objections as it had been used in | the past for something else, but maybe he can pitch in. | | The only thing I require from GHC folk is to simply use that branch and | not push/pull to/from master unless contributing feature patches or | trying to port some fixes into HEAD version for whatever reason. | | Thanks! | | -- | Mateusz K. | ___ | ghc-devs mailing list | ghc-devs@haskell.org | http://www.haskell.org/mailman/listinfo/ghc-devs ___ ghc-devs mailing list ghc-devs@haskell.org http://www.haskell.org/mailman/listinfo/ghc-devs
Re: Moving Haddock *development* out of GHC tree
On 08/14/2014 01:43 AM, Carter Schonwald wrote: one thing I wonder about is how should we approach noting theres a new language constructor, we should figure out a good way to present it in haddock in this work flow? because the initial haddocks presentation might just be a strawman till someone thinks about it carefully right? On Wed, Aug 13, 2014 at 6:30 PM, Herbert Valerio Riedel hvrie...@gmail.com wrote: On 2014-08-14 at 00:09:40 +0200, Mateusz Kowalczyk wrote: [...] I don't know what the GHC branch name will be yet. ‘ghc-head’ makes most sense but IIRC Herbert had some objections as it had been used in the past for something else, but maybe he can pitch in. I had no objections at all to that name, 'ghc-head' is fine with me :-) ___ ghc-devs mailing list ghc-devs@haskell.org http://www.haskell.org/mailman/listinfo/ghc-devs If there's more than one reasonable way then there's going to be strawman along the way somewhere anyway but we can at least delegate that until later. As I mention in the OP, there's at least no need for me to worry about it until it's finished on the GHC side although I'll no doubt be aware of it sooner than that. The PatternSynonyms stuff is an example where the implementor also stepped up to putting in support into Haddock for rendering. At the same time, the implementation has changed multiple times along the way creating hassle for both parties so perhaps in the future it's better to simply make sure Haddock still compiles and works but perhaps delegate everything else to closer to the release. In the end, it does not matter if Haddock can't display a bleeding edge feature until it's going out as a release. -- Mateusz K. ___ ghc-devs mailing list ghc-devs@haskell.org http://www.haskell.org/mailman/listinfo/ghc-devs
Re: Moving Haddock *development* out of GHC tree
good points by all :) On Thu, Aug 14, 2014 at 12:48 PM, Mateusz Kowalczyk fuuze...@fuuzetsu.co.uk wrote: On 08/14/2014 01:43 AM, Carter Schonwald wrote: one thing I wonder about is how should we approach noting theres a new language constructor, we should figure out a good way to present it in haddock in this work flow? because the initial haddocks presentation might just be a strawman till someone thinks about it carefully right? On Wed, Aug 13, 2014 at 6:30 PM, Herbert Valerio Riedel hvrie...@gmail.com wrote: On 2014-08-14 at 00:09:40 +0200, Mateusz Kowalczyk wrote: [...] I don't know what the GHC branch name will be yet. ‘ghc-head’ makes most sense but IIRC Herbert had some objections as it had been used in the past for something else, but maybe he can pitch in. I had no objections at all to that name, 'ghc-head' is fine with me :-) ___ ghc-devs mailing list ghc-devs@haskell.org http://www.haskell.org/mailman/listinfo/ghc-devs If there's more than one reasonable way then there's going to be strawman along the way somewhere anyway but we can at least delegate that until later. As I mention in the OP, there's at least no need for me to worry about it until it's finished on the GHC side although I'll no doubt be aware of it sooner than that. The PatternSynonyms stuff is an example where the implementor also stepped up to putting in support into Haddock for rendering. At the same time, the implementation has changed multiple times along the way creating hassle for both parties so perhaps in the future it's better to simply make sure Haddock still compiles and works but perhaps delegate everything else to closer to the release. In the end, it does not matter if Haddock can't display a bleeding edge feature until it's going out as a release. -- Mateusz K. ___ ghc-devs mailing list ghc-devs@haskell.org http://www.haskell.org/mailman/listinfo/ghc-devs ___ ghc-devs mailing list ghc-devs@haskell.org http://www.haskell.org/mailman/listinfo/ghc-devs
Re: Moving Haddock *development* out of GHC tree
On 08/13/2014 11:09 PM, Mateusz Kowalczyk wrote: On 08/08/2014 06:25 AM, Mateusz Kowalczyk wrote: Hello, [snip] Transition from current setup: If I receive some patches I was promised then I will then make a 2.14.4 bugfix/compat release make sure that master is up to date and then create something like GHC-tracking branch from master and track that. I will then abandon that branch and not push to it unless it is GHC release time. The next commit in master will bring Haddock to a state where it works with 7.8.3: yes, this means removing all new API stuff until 7.10 or 7.8.4 or whatever. GHC API changes go onto GHC-tracking while all the stuff I write goes master. When GHC makes a release or is about to, I make master work with that and make GHC-tracking point to that instead. Thanks! So it is now close to a week gone and I have received many positive replies and no negative ones. I will probably execute what I stated initially at about this time tomorrow. To reiterate in short: 1. I make sure what we have now compiles with GHC HEAD and I stick it in separate branch which GHC folk will now track and apply any API patches to. Unless something changes by tomorrow, this will most likely be what master is at right now, perhaps with a single change to the version in cabal file. 2. I make the master branch work with 7.8.3 (and possibly 7.8.x) and do development without worrying about any API changes in HEAD, releasing as often as I need to. 3. At GHC release time, I update master with API changes so that up-to-date Haddock is ready to be used to generate the docs and ship with the compiler. I don't know what the GHC branch name will be yet. ‘ghc-head’ makes most sense but IIRC Herbert had some objections as it had been used in the past for something else, but maybe he can pitch in. The only thing I require from GHC folk is to simply use that branch and not push/pull to/from master unless contributing feature patches or trying to port some fixes into HEAD version for whatever reason. Thanks! The deed is done, the branch to pull/push to/from if you're doing GHC API work is ‘ghc-head’. ‘master’ is now a development branch against 7.8.3. When the time comes for 7.10, I can simply re-apply the fixes + anything from ‘ghc-head’ at that time. You only need to concern yourself with this if you ever push to Haddock. -- Mateusz K. ___ ghc-devs mailing list ghc-devs@haskell.org http://www.haskell.org/mailman/listinfo/ghc-devs
Re: Moving Haddock *development* out of GHC tree
On 08/08/2014 06:25 AM, Mateusz Kowalczyk wrote: Hello, [snip] Transition from current setup: If I receive some patches I was promised then I will then make a 2.14.4 bugfix/compat release make sure that master is up to date and then create something like GHC-tracking branch from master and track that. I will then abandon that branch and not push to it unless it is GHC release time. The next commit in master will bring Haddock to a state where it works with 7.8.3: yes, this means removing all new API stuff until 7.10 or 7.8.4 or whatever. GHC API changes go onto GHC-tracking while all the stuff I write goes master. When GHC makes a release or is about to, I make master work with that and make GHC-tracking point to that instead. Thanks! So it is now close to a week gone and I have received many positive replies and no negative ones. I will probably execute what I stated initially at about this time tomorrow. To reiterate in short: 1. I make sure what we have now compiles with GHC HEAD and I stick it in separate branch which GHC folk will now track and apply any API patches to. Unless something changes by tomorrow, this will most likely be what master is at right now, perhaps with a single change to the version in cabal file. 2. I make the master branch work with 7.8.3 (and possibly 7.8.x) and do development without worrying about any API changes in HEAD, releasing as often as I need to. 3. At GHC release time, I update master with API changes so that up-to-date Haddock is ready to be used to generate the docs and ship with the compiler. I don't know what the GHC branch name will be yet. ‘ghc-head’ makes most sense but IIRC Herbert had some objections as it had been used in the past for something else, but maybe he can pitch in. The only thing I require from GHC folk is to simply use that branch and not push/pull to/from master unless contributing feature patches or trying to port some fixes into HEAD version for whatever reason. Thanks! -- Mateusz K. ___ ghc-devs mailing list ghc-devs@haskell.org http://www.haskell.org/mailman/listinfo/ghc-devs
Re: Moving Haddock *development* out of GHC tree
one thing I wonder about is how should we approach noting theres a new language constructor, we should figure out a good way to present it in haddock in this work flow? because the initial haddocks presentation might just be a strawman till someone thinks about it carefully right? On Wed, Aug 13, 2014 at 6:30 PM, Herbert Valerio Riedel hvrie...@gmail.com wrote: On 2014-08-14 at 00:09:40 +0200, Mateusz Kowalczyk wrote: [...] I don't know what the GHC branch name will be yet. ‘ghc-head’ makes most sense but IIRC Herbert had some objections as it had been used in the past for something else, but maybe he can pitch in. I had no objections at all to that name, 'ghc-head' is fine with me :-) ___ ghc-devs mailing list ghc-devs@haskell.org http://www.haskell.org/mailman/listinfo/ghc-devs ___ ghc-devs mailing list ghc-devs@haskell.org http://www.haskell.org/mailman/listinfo/ghc-devs
Re: Moving Haddock *development* out of GHC tree
Hi Mateusz, I'm mostly interested in understanding the Git-level/workflow changes, so here's a few questions to improve my understanding of what's changing related to Git: On 2014-08-08 at 07:25:01 +0200, Mateusz Kowalczyk wrote: [...] I do all the work and I think it's a 1 line change in sync-all when transition is ready. What change in ./sync-all are you thinking about specifically? (or alternatively: what about those not using ./sync-all anymore?) [...] * GHC itself checks out Haddock as a submodule as it does now. The only difference is that it points at whatever commit worked last. Let us assume it is the Haddock 2.14.3 release commit. The vital difference from current state is that GHC will no longer track changes in master branch. * Now when GHC API changes things proceed as they normally do: whoever is responsible for the changes, pops into the Haddock submodule applies the patches necessary for Haddock to build with HEAD and everyone is happy. What does *not* happen is these patches don't go into master: I ignore them and keep working with 7.8.3. Just to clarify, as the last sentence contains a double-negation: GHC devs continue pushing to github.com/haddock.git's `master` branch to keep Haddock building with GHC HEAD? It's just that the Haddock development proper happens in a branch other than `master` from now on? If I get this right, there will be a branch (`master`?) that's kept compatible with GHC HEAD, then there's a branch where new Haddock features are implemented (name?), and then there are stable branches for past releases (in the spirit of the current `v2.14`) So the only new thing would be a new `haddock-next` (or whatever you'd call that) branch, and `master` will just be on life-support for GHC HEAD until the next major GHC release is around the corner? [...] If I receive some patches I was promised then I will then make a 2.14.4 bugfix/compat release make sure that master is up to date and then create something like GHC-tracking branch from master and track that. I will then abandon that branch and not push to it unless it is GHC release time. The next commit in master will bring Haddock to a state where it works with 7.8.3: yes, this means removing all new API stuff until 7.10 or 7.8.4 or whatever. GHC API changes go onto GHC-tracking while all the stuff I write goes master. When GHC makes a release or is about to, I make master work with that and make GHC-tracking point to that instead. This paragraph confuses me a bit about which haddock branch is used for what. Can you maybe enumerate all haddock branches in the new scheme with their purpose? Cheers, hvr ___ ghc-devs mailing list ghc-devs@haskell.org http://www.haskell.org/mailman/listinfo/ghc-devs
Re: Moving Haddock *development* out of GHC tree
On Fri, Aug 08, 2014 at 09:00:21AM +0200, Herbert Valerio Riedel wrote: Just to clarify, as the last sentence contains a double-negation: GHC devs continue pushing to github.com/haddock.git's `master` branch to keep Haddock building with GHC HEAD? It's just that the Haddock development proper happens in a branch other than `master` from now on? From my perspective I would prefer to use `master` for Haddock development and use a branch with some other name for GHC development. My main motivation here is that as a contributor to Haddock I expect the latest code to be on `master`, and I would use it as a base when developing new features. Alternatively, maybe use `master` for both Haddock and GHC development, but push to different remotes (say use http://git.haskell.org/haddock.git for GHC development and https://github.com/haskell/haddock for Haddock development). I think this is what we already do for e.g. `containers`. Cheers, Simon ___ ghc-devs mailing list ghc-devs@haskell.org http://www.haskell.org/mailman/listinfo/ghc-devs
RE: Moving Haddock *development* out of GHC tree
Mateusz What you say makes sense to me. For me, the big thing is that we can make, and push, changes to Haddock in the GHC private branch, without having to negotiate. (Haddock reaches very deep into GHC's internals, so many many changes to GHC have some knock-on effect in Haddock.) You seem OK with this, so I am too. One concern: if you and Simon pay no attention to the GHC HEAD fork of Haddock, there is no guarantee that it works at all. Presumably it compiles (because GHC's build system will build it, forcing us to fix type errors) but it might not actually work! So it would probably pay for you to watch what is happening, to ensure that the patch-ups that ignorant GHC developers apply to Haddock do indeed have the desired effect. Some of these patch-ups might even be panics --- I don't know how to make Haddock render new construct foobar. That might be quite reasonable. But in general, thumbs up from me Simon | -Original Message- | From: ghc-devs [mailto:ghc-devs-boun...@haskell.org] On Behalf Of | Mateusz Kowalczyk | Sent: 08 August 2014 06:25 | To: ghc-devs@haskell.org | Cc: Simon Hengel | Subject: Moving Haddock *development* out of GHC tree | | Hello, | | A slightly long e-mail but I ask that you voice your opinion if you | ever changed GHC API. You can skim over the details, simply know that | it saves me vast amount of time, allows me to try and find contributors | and doesn't impact GHC negatively. It seems like a win-win scenario for | GHC and Haddock. GHC team's workflow does not change and will not | require any new commitment: I do all the work and I think it's a 1 line | change in sync-all when transition is ready. Here it is: | | | It is no secret that many core Haskell projects lack developer hands | and Haddock is no exception: the current maintainers are Simon Hengel | and myself. Simon does not have much time so currently all the issues | and updates are up to me. Ideally I would like if some more people | could come and hack on Haddock but there are a couple of problems with | trying to recruit folk for this: | | 1. Interacting with GHC API is not the easiest thing. This is Haddock's | problem but I thought I'd mention it here. | | 2. Haddock resides directly in the GHC tree and it is currently | *required* that it compiles with GHC HEAD. This is a huge barrier of | entry for anyone: today I wanted to make a fairly simple change but it | still took me 3 validate runs to be at least somewhat confident that I | didn't break much in GHC. On top of this I had help from Edward Z. Yang | on IRC and information from him on what the issue exactly was. If I was | to do everything alone it would have taken even more validates. A | validate is not fast on machine by any means, it takes an hour or two. | | Here is what I want to do unless there are major objections: I want to | move the active development away from GHC tree. Below is how it would | work. For simplicity please imagine that we have *just* released 7.8.3. | | * Haddock development would concentrate on supporting the last public | release of GHC: I stop developing against GHC HEAD and currently would | develop against 7.8.3. | | * GHC itself checks out Haddock as a submodule as it does now. The only | difference is that it points at whatever commit worked last. Let us | assume it is the Haddock 2.14.3 release commit. The vital difference | from current state is that GHC will no longer track changes in master | branch. | | * Now when GHC API changes things proceed as they normally do: whoever | is responsible for the changes, pops into the Haddock submodule applies | the patches necessary for Haddock to build with HEAD and everyone is | happy. What does *not* happen is these patches don't go into master: I | ignore them and keep working with 7.8.3. | | * When a GHC release rolls around, I update Haddock to work with the | new API so that people with new release can still use it. Once it works | against new API, GHC can start tracking from that commit onwards and | proceed as usual. | | Here are the advantages: | | * I don't have to work against GHC HEAD. This means I don't have to | build GHC HEAD and I don't need to worry about GHC API changes. I don't | waste 2-4 hours building before hacking and validating after hacking to | make any minor changes and to make sure I haven't broken anything. | | * More importantly, anyone who wants to write a patch for Haddock can | now do so easily, all they need is recent compiler rather than being | forced to build HEAD. Building and validating against HEAD is a | **huge** barrier of entry. | | * I only have to care about GHC API changes once a release and not | twice a week. I think PatternSynonyms have changed 4 times in a month | but the end result at release time is the same and that's what people | care about. | | * It is less work for anyone changing GHC API: they only have to deal | with their own changes and not my changes which add features
Re: Moving Haddock *development* out of GHC tree
The biggest disadvantage in my mind is that you're setting yourself up for a potentially huge merge just before the GHC release and might block the GHC release until that merge is done (assuming that haddock is still shipped with GHC). ___ ghc-devs mailing list ghc-devs@haskell.org http://www.haskell.org/mailman/listinfo/ghc-devs
RE: Moving Haddock *development* out of GHC tree
The biggest disadvantage in my mind is that you're setting yourself up for a potentially huge merge just before the GHC release and might block the GHC release until that merge is done (assuming that haddock is still shipped with GHC). Excellent point. The merge shouldn’t block the release, though. In extremis, I guess we could always release the GHC fork of Haddock if the tip of Haddock wasn’t merged to match GHC! But I doubt it’ll come to that Simon From: ghc-devs [mailto:ghc-devs-boun...@haskell.org] On Behalf Of Johan Tibell Sent: 08 August 2014 09:07 To: Mateusz Kowalczyk Cc: ghc-devs@haskell.org; Simon Hengel Subject: Re: Moving Haddock *development* out of GHC tree The biggest disadvantage in my mind is that you're setting yourself up for a potentially huge merge just before the GHC release and might block the GHC release until that merge is done (assuming that haddock is still shipped with GHC). ___ ghc-devs mailing list ghc-devs@haskell.org http://www.haskell.org/mailman/listinfo/ghc-devs
Re: Moving Haddock *development* out of GHC tree
On Fri, Aug 8, 2014 at 10:11 AM, Simon Peyton Jones simo...@microsoft.com wrote: The biggest disadvantage in my mind is that you're setting yourself up for a potentially huge merge just before the GHC release and might block the GHC release until that merge is done (assuming that haddock is still shipped with GHC). Excellent point. The merge shouldn’t block the release, though. In extremis, I guess we could always release the GHC fork of Haddock if the tip of Haddock wasn’t merged to match GHC! But I doubt it’ll come to that But as you mentioned the GHC fork of Haddock might not work (it might just type check) so at the very least Mateusz is signing up for validating that it indeed works before a GHC release. That's of course fine, I just want people to understand what we're signing up for. ___ ghc-devs mailing list ghc-devs@haskell.org http://www.haskell.org/mailman/listinfo/ghc-devs
Re: Moving Haddock *development* out of GHC tree
I thought this was what you were already doing :-) Anyway, this is more or less the setup we had in mind when Haddock was added to the GHC tree. The only question is which branches are used for GHC and for regular development, and where they live. As long as that's clear for everyone (both Haddock and GHC developers), then this should be fine. The GHC release engineer will need to give the Haddock maintainers plenty of heads-up time before a release so that the merge can be done - Austin could you add that to the release checklist? Cheers, Simon On 08/08/2014 06:25, Mateusz Kowalczyk wrote: Hello, A slightly long e-mail but I ask that you voice your opinion if you ever changed GHC API. You can skim over the details, simply know that it saves me vast amount of time, allows me to try and find contributors and doesn't impact GHC negatively. It seems like a win-win scenario for GHC and Haddock. GHC team's workflow does not change and will not require any new commitment: I do all the work and I think it's a 1 line change in sync-all when transition is ready. Here it is: It is no secret that many core Haskell projects lack developer hands and Haddock is no exception: the current maintainers are Simon Hengel and myself. Simon does not have much time so currently all the issues and updates are up to me. Ideally I would like if some more people could come and hack on Haddock but there are a couple of problems with trying to recruit folk for this: 1. Interacting with GHC API is not the easiest thing. This is Haddock's problem but I thought I'd mention it here. 2. Haddock resides directly in the GHC tree and it is currently *required* that it compiles with GHC HEAD. This is a huge barrier of entry for anyone: today I wanted to make a fairly simple change but it still took me 3 validate runs to be at least somewhat confident that I didn't break much in GHC. On top of this I had help from Edward Z. Yang on IRC and information from him on what the issue exactly was. If I was to do everything alone it would have taken even more validates. A validate is not fast on machine by any means, it takes an hour or two. Here is what I want to do unless there are major objections: I want to move the active development away from GHC tree. Below is how it would work. For simplicity please imagine that we have *just* released 7.8.3. * Haddock development would concentrate on supporting the last public release of GHC: I stop developing against GHC HEAD and currently would develop against 7.8.3. * GHC itself checks out Haddock as a submodule as it does now. The only difference is that it points at whatever commit worked last. Let us assume it is the Haddock 2.14.3 release commit. The vital difference from current state is that GHC will no longer track changes in master branch. * Now when GHC API changes things proceed as they normally do: whoever is responsible for the changes, pops into the Haddock submodule applies the patches necessary for Haddock to build with HEAD and everyone is happy. What does *not* happen is these patches don't go into master: I ignore them and keep working with 7.8.3. * When a GHC release rolls around, I update Haddock to work with the new API so that people with new release can still use it. Once it works against new API, GHC can start tracking from that commit onwards and proceed as usual. Here are the advantages: * I don't have to work against GHC HEAD. This means I don't have to build GHC HEAD and I don't need to worry about GHC API changes. I don't waste 2-4 hours building before hacking and validating after hacking to make any minor changes and to make sure I haven't broken anything. * More importantly, anyone who wants to write a patch for Haddock can now do so easily, all they need is recent compiler rather than being forced to build HEAD. Building and validating against HEAD is a **huge** barrier of entry. * I only have to care about GHC API changes once a release and not twice a week. I think PatternSynonyms have changed 4 times in a month but the end result at release time is the same and that's what people care about. * It is less work for anyone changing GHC API: they only have to deal with their own changes and not my changes which add features or whatever. * If I break something in Haddock HEAD, GHC is not affected. * If Haddock's binary interface doesn't change, we may even allow more versions of GHC be compatible through CPP and other such trickery. If we were to do it today, it would be an increased burden on the GHC team to deal with those. * I can release as often as I want against the same compiler version. Currently doing this requires backporting features (see v2.14 branch) which is a massive pain. I no longer have to tell the users ‘yes, your bug is fixed but to get it you need to compile GHC HEAD or wait 6-12 months until next GHC release’. I have to do this a lot. Here are the disadvantages and why I think they don't make a big difference: * GHC
Re: Moving Haddock *development* out of GHC tree
On 2014-08-08 at 09:42:14 +0200, Simon Hengel wrote: On Fri, Aug 08, 2014 at 09:00:21AM +0200, Herbert Valerio Riedel wrote: Just to clarify, as the last sentence contains a double-negation: GHC devs continue pushing to github.com/haddock.git's `master` branch to keep Haddock building with GHC HEAD? It's just that the Haddock development proper happens in a branch other than `master` from now on? From my perspective I would prefer to use `master` for Haddock development and use a branch with some other name for GHC development. My main motivation here is that as a contributor to Haddock I expect the latest code to be on `master`, and I would use it as a base when developing new features. Just a minor nitpick (but I agree with having `master` used for hosting active Haddock development): latest code might not be a canonical concept, as there will be latest code that works with GHC HEAD, and latest code that works with last released GHC Alternatively, maybe use `master` for both Haddock and GHC development, but push to different remotes (say use http://git.haskell.org/haddock.git for GHC development and https://github.com/haskell/haddock for Haddock development). I think this is what we already do for e.g. `containers`. I'd rather reduce the number of doubled repositories (not the least to simplify the mirroring setup) to avoid confusion about where things live/need to be pushed to. If this is just an alpha-conversion modulo thing, then let's just call the new branch for GHC HEAD simply `ghc-head` (or something like that) and keep hosting it in github.com/haskell/haddock.git, and have GHC HEAD developers push to that instead (fwiw, you can specify the default branch in .gitmodules, which some few Git tools honor). Cheers, hvr ___ ghc-devs mailing list ghc-devs@haskell.org http://www.haskell.org/mailman/listinfo/ghc-devs
Re: Moving Haddock *development* out of GHC tree
I'm also in favour of a more decoupled development/release process. I'd like to change a few things in haddock to make it more suitable for use as a library, so that I can set up a haddock for GHCJS without duplicating the whole package (it needs a custom platform setup and some changes in file name handling). It'd be great if such a release could be made independently of GHC and changes like this could be made without requiring the user to update their GHC. Also I'd be happy to do some of the work for the out-of-tree change, for example backporting fixes to the 2.14 branch or updating the 2.15 branch to work with the 7.8 api (but I can't promise more, GHCJS is taking enough of my time, and I'm not sure how much time I can afford to keep spending on it, so I'd like to minimize my other maintenance tasks as much as possible). luite On Fri, Aug 8, 2014 at 10:18 AM, Simon Marlow marlo...@gmail.com wrote: I thought this was what you were already doing :-) Anyway, this is more or less the setup we had in mind when Haddock was added to the GHC tree. The only question is which branches are used for GHC and for regular development, and where they live. As long as that's clear for everyone (both Haddock and GHC developers), then this should be fine. The GHC release engineer will need to give the Haddock maintainers plenty of heads-up time before a release so that the merge can be done - Austin could you add that to the release checklist? Cheers, Simon On 08/08/2014 06:25, Mateusz Kowalczyk wrote: Hello, A slightly long e-mail but I ask that you voice your opinion if you ever changed GHC API. You can skim over the details, simply know that it saves me vast amount of time, allows me to try and find contributors and doesn't impact GHC negatively. It seems like a win-win scenario for GHC and Haddock. GHC team's workflow does not change and will not require any new commitment: I do all the work and I think it's a 1 line change in sync-all when transition is ready. Here it is: It is no secret that many core Haskell projects lack developer hands and Haddock is no exception: the current maintainers are Simon Hengel and myself. Simon does not have much time so currently all the issues and updates are up to me. Ideally I would like if some more people could come and hack on Haddock but there are a couple of problems with trying to recruit folk for this: 1. Interacting with GHC API is not the easiest thing. This is Haddock's problem but I thought I'd mention it here. 2. Haddock resides directly in the GHC tree and it is currently *required* that it compiles with GHC HEAD. This is a huge barrier of entry for anyone: today I wanted to make a fairly simple change but it still took me 3 validate runs to be at least somewhat confident that I didn't break much in GHC. On top of this I had help from Edward Z. Yang on IRC and information from him on what the issue exactly was. If I was to do everything alone it would have taken even more validates. A validate is not fast on machine by any means, it takes an hour or two. Here is what I want to do unless there are major objections: I want to move the active development away from GHC tree. Below is how it would work. For simplicity please imagine that we have *just* released 7.8.3. * Haddock development would concentrate on supporting the last public release of GHC: I stop developing against GHC HEAD and currently would develop against 7.8.3. * GHC itself checks out Haddock as a submodule as it does now. The only difference is that it points at whatever commit worked last. Let us assume it is the Haddock 2.14.3 release commit. The vital difference from current state is that GHC will no longer track changes in master branch. * Now when GHC API changes things proceed as they normally do: whoever is responsible for the changes, pops into the Haddock submodule applies the patches necessary for Haddock to build with HEAD and everyone is happy. What does *not* happen is these patches don't go into master: I ignore them and keep working with 7.8.3. * When a GHC release rolls around, I update Haddock to work with the new API so that people with new release can still use it. Once it works against new API, GHC can start tracking from that commit onwards and proceed as usual. Here are the advantages: * I don't have to work against GHC HEAD. This means I don't have to build GHC HEAD and I don't need to worry about GHC API changes. I don't waste 2-4 hours building before hacking and validating after hacking to make any minor changes and to make sure I haven't broken anything. * More importantly, anyone who wants to write a patch for Haddock can now do so easily, all they need is recent compiler rather than being forced to build HEAD. Building and validating against HEAD is a **huge** barrier of entry. * I only have to care about GHC API changes once a release and not twice a week. I think PatternSynonyms
Re: Moving Haddock *development* out of GHC tree
On Fri, Aug 08, 2014 at 10:35:44AM +0200, Herbert Valerio Riedel wrote: If this is just an alpha-conversion modulo thing, then let's just call the new branch for GHC HEAD simply `ghc-head` (or something like that) and keep hosting it in github.com/haskell/haddock.git, and have GHC HEAD developers push to that instead (fwiw, you can specify the default branch in .gitmodules, which some few Git tools honor). Ok, cool, that would work for me. Cheers. ___ ghc-devs mailing list ghc-devs@haskell.org http://www.haskell.org/mailman/listinfo/ghc-devs
Re: Moving Haddock *development* out of GHC tree
On 08/08/2014 10:35 AM, Herbert Valerio Riedel wrote: On 2014-08-08 at 09:42:14 +0200, Simon Hengel wrote: On Fri, Aug 08, 2014 at 09:00:21AM +0200, Herbert Valerio Riedel wrote: Just to clarify, as the last sentence contains a double-negation: GHC devs continue pushing to github.com/haddock.git's `master` branch to keep Haddock building with GHC HEAD? It's just that the Haddock development proper happens in a branch other than `master` from now on? From my perspective I would prefer to use `master` for Haddock development and use a branch with some other name for GHC development. My main motivation here is that as a contributor to Haddock I expect the latest code to be on `master`, and I would use it as a base when developing new features. Just a minor nitpick (but I agree with having `master` used for hosting active Haddock development): latest code might not be a canonical concept, as there will be latest code that works with GHC HEAD, and latest code that works with last released GHC Alternatively, maybe use `master` for both Haddock and GHC development, but push to different remotes (say use http://git.haskell.org/haddock.git for GHC development and https://github.com/haskell/haddock for Haddock development). I think this is what we already do for e.g. `containers`. I'd rather reduce the number of doubled repositories (not the least to simplify the mirroring setup) to avoid confusion about where things live/need to be pushed to. If this is just an alpha-conversion modulo thing, then let's just call the new branch for GHC HEAD simply `ghc-head` (or something like that) and keep hosting it in github.com/haskell/haddock.git, and have GHC HEAD developers push to that instead (fwiw, you can specify the default branch in .gitmodules, which some few Git tools honor). Cheers, hvr Hi, Here is what my plan was Haddock branches would be: master – Haddock devs push here, the fixes go here GHC-tracking – GHC team pushes here At GHC release master would be brought up to a state where it works with current GHC API. GHC-tracking then would be reset to master. The change in sync-all I was referring to is that ./sync-all get ./sync-all pull would not end up pointing at a master branch but after some sleep I realise that's probably not the case anyway. We simply need GHC team to push to their own branch. If I get this right, there will be a branch (`master`?) that's kept compatible with GHC HEAD, then there's a branch where new Haddock features are implemented (name?), The other way around, master is for Haddock while the other branch is for GHC. -- Mateusz K. ___ ghc-devs mailing list ghc-devs@haskell.org http://www.haskell.org/mailman/listinfo/ghc-devs
Re: Moving Haddock *development* out of GHC tree
On 08/08/2014 10:15 AM, Johan Tibell wrote: On Fri, Aug 8, 2014 at 10:11 AM, Simon Peyton Jones simo...@microsoft.com wrote: The biggest disadvantage in my mind is that you're setting yourself up for a potentially huge merge just before the GHC release and might block the GHC release until that merge is done (assuming that haddock is still shipped with GHC). Excellent point. The merge shouldn’t block the release, though. In extremis, I guess we could always release the GHC fork of Haddock if the tip of Haddock wasn’t merged to match GHC! But I doubt it’ll come to that But as you mentioned the GHC fork of Haddock might not work (it might just type check) so at the very least Mateusz is signing up for validating that it indeed works before a GHC release. That's of course fine, I just want people to understand what we're signing up for. Well, I stick around and am usually aware of GHC release early. In the usual case Haddock will be fixed up before the actual GHC release. I don't think API changes were ever drastic enough to provide major problems especially seeing as I'll be able to refer to the GHC-tracked branch to see what patches were applied there. However let's consider I can't make it for the release because I'm not available around that time or otherwise. This should still not hold up GHC release. I would expect the GHC team to release Haddock + their fixes, it would simply be like an existing release with some patches on top to have it work with new GHC. I can then come around and once I apply any API patches, I make an actual Haddock release. People can then simply cabal install haddock and use what they get here rather than what came with GHC. Does this make sense? -- Mateusz K. ___ ghc-devs mailing list ghc-devs@haskell.org http://www.haskell.org/mailman/listinfo/ghc-devs
Re: Moving Haddock *development* out of GHC tree
On 08/08/2014 10:18 AM, Simon Marlow wrote: I thought this was what you were already doing :-) Anyway, this is more or less the setup we had in mind when Haddock was added to the GHC tree. The only question is which branches are used for GHC and for regular development, and where they live. As long as that's clear for everyone (both Haddock and GHC developers), then this should be fine. I think there is no problem if they both live in the existing repository (github.com/haskell/haddock) or whatever the submodule refers to today. The GHC release engineer will need to give the Haddock maintainers plenty of heads-up time before a release so that the merge can be done - Austin could you add that to the release checklist? Right, although I don't exactly plan to abandon any of the GHC information channels I'm on today: I tend to be well aware of a release coming. Cheers, Simon -- Mateusz K. ___ ghc-devs mailing list ghc-devs@haskell.org http://www.haskell.org/mailman/listinfo/ghc-devs
Moving Haddock *development* out of GHC tree
Hello, A slightly long e-mail but I ask that you voice your opinion if you ever changed GHC API. You can skim over the details, simply know that it saves me vast amount of time, allows me to try and find contributors and doesn't impact GHC negatively. It seems like a win-win scenario for GHC and Haddock. GHC team's workflow does not change and will not require any new commitment: I do all the work and I think it's a 1 line change in sync-all when transition is ready. Here it is: It is no secret that many core Haskell projects lack developer hands and Haddock is no exception: the current maintainers are Simon Hengel and myself. Simon does not have much time so currently all the issues and updates are up to me. Ideally I would like if some more people could come and hack on Haddock but there are a couple of problems with trying to recruit folk for this: 1. Interacting with GHC API is not the easiest thing. This is Haddock's problem but I thought I'd mention it here. 2. Haddock resides directly in the GHC tree and it is currently *required* that it compiles with GHC HEAD. This is a huge barrier of entry for anyone: today I wanted to make a fairly simple change but it still took me 3 validate runs to be at least somewhat confident that I didn't break much in GHC. On top of this I had help from Edward Z. Yang on IRC and information from him on what the issue exactly was. If I was to do everything alone it would have taken even more validates. A validate is not fast on machine by any means, it takes an hour or two. Here is what I want to do unless there are major objections: I want to move the active development away from GHC tree. Below is how it would work. For simplicity please imagine that we have *just* released 7.8.3. * Haddock development would concentrate on supporting the last public release of GHC: I stop developing against GHC HEAD and currently would develop against 7.8.3. * GHC itself checks out Haddock as a submodule as it does now. The only difference is that it points at whatever commit worked last. Let us assume it is the Haddock 2.14.3 release commit. The vital difference from current state is that GHC will no longer track changes in master branch. * Now when GHC API changes things proceed as they normally do: whoever is responsible for the changes, pops into the Haddock submodule applies the patches necessary for Haddock to build with HEAD and everyone is happy. What does *not* happen is these patches don't go into master: I ignore them and keep working with 7.8.3. * When a GHC release rolls around, I update Haddock to work with the new API so that people with new release can still use it. Once it works against new API, GHC can start tracking from that commit onwards and proceed as usual. Here are the advantages: * I don't have to work against GHC HEAD. This means I don't have to build GHC HEAD and I don't need to worry about GHC API changes. I don't waste 2-4 hours building before hacking and validating after hacking to make any minor changes and to make sure I haven't broken anything. * More importantly, anyone who wants to write a patch for Haddock can now do so easily, all they need is recent compiler rather than being forced to build HEAD. Building and validating against HEAD is a **huge** barrier of entry. * I only have to care about GHC API changes once a release and not twice a week. I think PatternSynonyms have changed 4 times in a month but the end result at release time is the same and that's what people care about. * It is less work for anyone changing GHC API: they only have to deal with their own changes and not my changes which add features or whatever. * If I break something in Haddock HEAD, GHC is not affected. * If Haddock's binary interface doesn't change, we may even allow more versions of GHC be compatible through CPP and other such trickery. If we were to do it today, it would be an increased burden on the GHC team to deal with those. * I can release as often as I want against the same compiler version. Currently doing this requires backporting features (see v2.14 branch) which is a massive pain. I no longer have to tell the users ‘yes, your bug is fixed but to get it you need to compile GHC HEAD or wait 6-12 months until next GHC release’. I have to do this a lot. Here are the disadvantages and why I think they don't make a big difference: * GHC HEAD doesn't get any new-and-cool features that we might implement. I say this doesn't matter because no one uses varying GHC HEAD versions to develop actual software, documentation and all. What I mean to say is that the only user of the Haddock that's developed in GHC tree is GHC itself. The only case where GHC actually used in-tree Haddock was when Herbert generated documentation for base-4.7 early for me to eye before the release. Even this doesn't matter because so close to the release I'll already have the existing GHC API integrated anyway. Again, it does not matter if GHC HEAD itself doesn't get
