Bug#845297: Migration to Salsa - reorganization of webmaster-team and repos
Hello again I have moved (transferred in gitlab words) some repos. The current structure is: https://salsa.debian.org/webmaster-team/ https://salsa.debian.org/webmaster-team/cgi (cgi.debian.org) https://salsa.debian.org/webmaster-team/cron (cron scripts) https://salsa.debian.org/webmaster-team/theming_debbugs (theming for debbugs) https://salsa.debian.org/webmaster-team/theming_gitweb (theming for gitweb) https://salsa.debian.org/webmaster-team/theming_ikiwiki (theming for ikiwiki) https://salsa.debian.org/webmaster-team/theming_planet (theming for planet) Note that I removed the empty debwww repo, and renamed the theming repos so it is more clear for everybody. I will update the alioth settings of the old repos to refer to their new location. I didn't move the test_webwml* repos up, because I missed my own deadline and other people are working on them. So they are still here: https://salsa.debian.org/webmaster-team/webwml/test_webwml_cvs2git (test) https://salsa.debian.org/webmaster-team/webwml/test_webwml (test) I think we can leave them like this for now, and remove both the test repos and the webwml subgroup when we do the final migration. Regards -- Laura Arjona Reina https://wiki.debian.org/LauraArjona
Bug#845297: Migration to Salsa - reorganization of webmaster-team and repos
On Sat, May 12, 2018 at 1:35 AM, Laura Arjona Reina wrote: > After some interchange of ideas in #debian-www, looks like it's simpler > not to have the webwml subgroup in Salsa. We can set members/permissions > per repo, so we can put all the different git repos at the same level, > as "direct children" of webmaster-team. Good plan! > If you have any comment or better suggestion, please tell. Thanks for dealing with the migration, it is much appreciated. -- bye, pabs https://wiki.debian.org/PaulWise
Bug#845297: Migration to Salsa - reorganization of webmaster-team and repos
Hello all El 26/04/18 a las 14:39, Laura Arjona Reina escribió: > Hello > > I have migrated to Salsa the following repos: > > webwml/debbugstheming for bugs.debian.org > Now it lives in https://salsa.debian.org/webmaster-team/webwml/debbugs > > webwml/gitweb theming for gitweb > Now it lives in https://salsa.debian.org/webmaster-team/webwml/gitweb > > webwml/ikiwikitheming for ikiwiki > Now it lives in https://salsa.debian.org/webmaster-team/webwml/ikiwiki > > webwml/planet theming for planet > Now it lives in https://salsa.debian.org/webmaster-team/webwml/planet > After some interchange of ideas in #debian-www, looks like it's simpler not to have the webwml subgroup in Salsa. We can set members/permissions per repo, so we can put all the different git repos at the same level, as "direct children" of webmaster-team. Thus if there are no objections, next week (let's say Monday 14 May) during the UTC evening I will move the debbugs, gitweb, ikiwiki, planet, test_webwml and test_webwml_cvs2git one level up, with their corresponding members and permissions, and I will remove the webmaster-team/webwml subgroup. When everything is in the new place I will send another mail so people know that they should update their "git remotes", if they were working in any of the salsa repositories. The proposed resulting structure is: https://salsa.debian.org/webmaster-team/ https://salsa.debian.org/webmaster-team/cgi (cgi.debian.org) https://salsa.debian.org/webmaster-team/cron (cron scripts) https://salsa.debian.org/webmaster-team/debbugs (theming for debbugs) https://salsa.debian.org/webmaster-team/gitweb (theming for gitweb) https://salsa.debian.org/webmaster-team/ikiwiki (theming for ikiwiki) https://salsa.debian.org/webmaster-team/planet (theming for planet) https://salsa.debian.org/webmaster-team/test_webwml_cvs2git (test) https://salsa.debian.org/webmaster-team/test_webwml (test) https://salsa.debian.org/webmaster-team/debwww is an empty repo with no git log history so I will remove it Future repos (to be migrated soon) will be: https://salsa.debian.org/webmaster-team/packages (packages.debian.org) https://salsa.debian.org/webmaster-team/webwml (www.debian.org) (Each repo would start with the same members and permissions that had in alioth) If you have any comment or better suggestion, please tell. Cheers -- Laura Arjona Reina https://wiki.debian.org/LauraArjona
Bug#845297: Migration to Salsa
On Mon, Apr 30, 2018 at 12:29 AM, Alex Muntada wrote:
> Thanks for pointing that out. I've been studying smart_change.pl
> and I couldn't find where that commit you mention is happening.
The commit is done after smart_change.pl is run, by the person who is
making the change. So if we use git commit ids in the headers that
smart_change.pl writes, they need to refer to a future commit that
hasn't happened yet, whose date is unknown and therefore whose SHA-1
hash is unknowable because it depends on the date.
> Could you elaborate a bit more on the git file+metafile details?
> I'm not following you, sorry.
Each file in a git tree is associated with a hash of some data that is
the file contents prefixed with a header containing the object type
('blob' for files), a space, the file size and a NUL character.
Storing these blob object hashes in the translation-check headers
would allow us to calculate them before commits (as smart_change.pl
does with CVS revision numbers). The machinery behind the translation
comparison stuff can then easily do a git diff between the two object
hashes. Since these git object hashes refer to individual files not
trees, they work more like CVS revision numbers than git commit hashes
do. The main question is, are they compatible with all of the VCS
operations used by the translation stuff (I think they are)?
https://git-scm.com/book/en/v2/Git-Internals-Git-Objects
The script converting all existing translation-check headers from CVS
revision numbers to git object hashes would also be more complicated,
since the cvs-revisions file produced by the git conversion contains
filenames, CVS revision IDs and corresponding git commits. So, we have
to run `git ls-tree` for each line in cvs-revisions to get the object
hashes to put into translation-check headers.
git ls-tree -zr 7c95dc979cd7184ec4f20b0dd37e73e001a22d4f
TRANSLATING.pages | cut -zf1 | cut -z -d' ' -f3
--
bye,
pabs
https://wiki.debian.org/PaulWise
Bug#845297: Migration to Salsa
Hi Paul, > Please note that smart_change.pl needs to work with the new workflow. > This means that we can't use git commit IDs in translations headers, > unless we want smart_change.pl to make two commits instead of one. Thanks for pointing that out. I've been studying smart_change.pl and I couldn't find where that commit you mention is happening. > The alternative is to use git file+metadata hashes, which can be > calculated before a commit happens, placed into translations headers > and used to diff between two versions of a file. I'm not sure how well > this will fit with the existing webwml VCS modules though. Could you elaborate a bit more on the git file+metafile details? I'm not following you, sorry. Thanks! Alex signature.asc Description: PGP signature
Bug#845297: Migration to Salsa
Hello again El 29/04/18 a las 15:01, Paul Wise escribió: > On Sun, 2018-04-29 at 13:49 +0200, Alex Muntada wrote: > >> Therefore, it seems that we'll need to translate CVS revisions to >> commit IDs at least once after the repository has been migrated >> finally to git and before any translation is performed. > Please note that smart_change.pl needs to work with the new workflow. > This means that we can't use git commit IDs in translations headers, > unless we want smart_change.pl to make two commits instead of one. I'm not an expert but getting smart_change.pl doing 2 commits looks like easy to understand (and maybe to implement?). First commit for the english folder, and then other commit updating translations. > The alternative is to use git file+metadata hashes, which can be > calculated before a commit happens, placed into translations headers > and used to diff between two versions of a file. I'm not sure how well > this will fit with the existing webwml VCS modules though. > No idea, either :/ Cheers -- Laura Arjona Reina https://wiki.debian.org/LauraArjona
Bug#845297: Migration to Salsa
Hi Note: you can drop me in the CC, I'm following the bug report via the debian-www mailing list. El 29/04/18 a las 13:49, Alex Muntada escribió: > Laura explained the translation workflow to me, and I thought > that we wouldn't need to keep track of current CVS revisions > because using the last commit id from a file and checking if a > commit id is ancestor to another should be enough: No, as you note below. > > $ git log -1 --oneline english/license.wml > 8e8136309fa use https for the link to www.spi-inc.org > $ git log -1 --oneline catalan/license.wml > 8e8136309fa use https for the link to www.spi-inc.org > $ git merge-base --is-ancestor 8e8136309fa 8e8136309fa > $ echo $? > 0 > > However, I didn't take into account that sometimes there are > changes applied to several files at the same time that have > nothing to do with translations (as showed in the example above). > > Therefore, it seems that we'll need to translate CVS revisions to > commit IDs at least once after the repository has been migrated > finally to git and before any translation is performed. This can be done in the meanwhile in the test_cvs2git repo: https://salsa.debian.org/webmaster-team/webwml/test_webwml_cvs2git I've just added there the file cvs-revisions which was generated by the cvs2git script during the conversion. > Since we can't be sure that people translating will stop during > the migration, maybe we can add a kill-switch in the scripts so > we can enable it before migrating and removing it to enable the > translation work after everything is setup correctly in git. > > Something like this makes sense? > > 1. People translate on CVS. > 2. Add a kill-switch to translations scripts. > 3. Let people translate on CVS until migration date. > 4. Enable kill-switch so people cannot translate. > 5. Migrate repo to git. > 6. Setup new workflow. > 7. Make changes to translations docs. > 8. Remove kill-switch. > 9. Let people translate on git. > > Steps 5-7 should take the shortest time possible to avoid impact, > of course. > > The kill-switch should be very easy to implement, e.g.: > > my $kill_switch = 0; > die "Sorry! webwml migration in progress. Check the wiki.\n" > if $kill_switch; I have no opinion on this. Frankly I don't know what to do if we cannot get the new workflow working when Alioth is shutdown or the cvs service stopped. Maybe we can try to setup a CVS repo in other place, or ask translators to manually keep track of the translation-check line (even when the dashboards are not working). > > Since I'm not very familiar with the translation workflow yet, > I'd like to work on the Perl scripts that will deal with the > files in git, etc. But I'm fine doing something else if someone > prefers to do that. I think the next steps are: 1.- Writing a script to get all the files with translation-check line to be updated using the corresponding git hash. (This can be done in the test_webwml_cvs2git repo, and when we do the authentic migration we run it again) 2.- Get the copypage.pl and the smart_change.pl scripts working with git hashes. Thanks! -- Laura Arjona Reina https://wiki.debian.org/LauraArjona
Bug#845297: Migration to Salsa
On Sun, 2018-04-29 at 13:49 +0200, Alex Muntada wrote: > Therefore, it seems that we'll need to translate CVS revisions to > commit IDs at least once after the repository has been migrated > finally to git and before any translation is performed. Please note that smart_change.pl needs to work with the new workflow. This means that we can't use git commit IDs in translations headers, unless we want smart_change.pl to make two commits instead of one. The alternative is to use git file+metadata hashes, which can be calculated before a commit happens, placed into translations headers and used to diff between two versions of a file. I'm not sure how well this will fit with the existing webwml VCS modules though. -- bye, pabs https://wiki.debian.org/PaulWise signature.asc Description: This is a digitally signed message part
Bug#845297: Migration to Salsa
Hi Steve, > I'm thinking of taking some vacation (a week or so) from my day > job during May to get stuck in here. When are other people > available? I'm attending a conference from May 8 to 10, so I guess I'll be more available than usual in working hours the whole week. > What can I help with? I'm OK-ish in perl, so I'm thinking of > looking at the translation workflow stuff, unless that's > already in hand. Laura explained the translation workflow to me, and I thought that we wouldn't need to keep track of current CVS revisions because using the last commit id from a file and checking if a commit id is ancestor to another should be enough: $ git log -1 --oneline english/license.wml 8e8136309fa use https for the link to www.spi-inc.org $ git log -1 --oneline catalan/license.wml 8e8136309fa use https for the link to www.spi-inc.org $ git merge-base --is-ancestor 8e8136309fa 8e8136309fa $ echo $? 0 However, I didn't take into account that sometimes there are changes applied to several files at the same time that have nothing to do with translations (as showed in the example above). Therefore, it seems that we'll need to translate CVS revisions to commit IDs at least once after the repository has been migrated finally to git and before any translation is performed. Since we can't be sure that people translating will stop during the migration, maybe we can add a kill-switch in the scripts so we can enable it before migrating and removing it to enable the translation work after everything is setup correctly in git. Something like this makes sense? 1. People translate on CVS. 2. Add a kill-switch to translations scripts. 3. Let people translate on CVS until migration date. 4. Enable kill-switch so people cannot translate. 5. Migrate repo to git. 6. Setup new workflow. 7. Make changes to translations docs. 8. Remove kill-switch. 9. Let people translate on git. Steps 5-7 should take the shortest time possible to avoid impact, of course. The kill-switch should be very easy to implement, e.g.: my $kill_switch = 0; die "Sorry! webwml migration in progress. Check the wiki.\n" if $kill_switch; Since I'm not very familiar with the translation workflow yet, I'd like to work on the Perl scripts that will deal with the files in git, etc. But I'm fine doing something else if someone prefers to do that. Cheers! Alex signature.asc Description: PGP signature
Bug#845297: Migration to Salsa
Hello El 27/04/18 a las 17:43, Steve McIntyre escribió: > On Wed, Apr 18, 2018 at 11:01:03AM +0200, Laura Arjona Reina wrote: >> Hello again >> >> El 18/04/18 a las 01:33, Laura Arjona Reina escribió: >> >>> >>> * I will copy the migrated-to-git webwml repos (the tests) to >>> https://salsa.debian.org/groups/webmaster-team/webwml/ labelling them as >>> tests so people don't get confused. If you want to hack on those and you >>> were not added as "developer" of the group yet, please request join the >>> group and me or other admin will grant permissions. >> >> This is done now: >> >> https://salsa.debian.org/webmaster-team/webwml/test_webwml_cvs2git >> https://salsa.debian.org/webmaster-team/webwml/test_webwml >> >> I will remove the "test repos" in my personal space in Salsa, and in Alioth. > > Cool. > > So... Laura and I were talking on IRC yesterday. I've been swamped > with a range of family things that removed my "free" time over the > last few months, so I've been unable to really help much so far. Laura > has been talking about running a sprint to work on the migration. I > think that sounds like a splendid idea, but I can't really travel much > at the moment. However, I'm thinking of taking some vacation (a week > or so) from my day job during May to get stuck in here. When are other > people available? > For now I have May free so I am available mostly European evenings/nights but probably at other times too. I cannot travel much at this moment either but if we decide 2-3 days to do a "distributed" or "remote" sprint I would do my best to be more available (remotely) for website work in those days. > What can I help with? I'm OK-ish in perl, so I'm thinking of looking > at the translation workflow stuff, unless that's already in hand. Yes please! I would propose to work in this repo: https://salsa.debian.org/webmaster-team/webwml/test_webwml_cvs2git and when we have ready the new scripts or adapted the current ones, we can make patches of those commits and apply them in the CVS webwml repo, and then when we run again the cvs2git script for the final migration, the resultant repo will contain the good scripts. Is > https://wiki.debian.org/WebsiteGitTransition still the canonical place > to track the planning? > I have just reviewed and updated it with the current status of things. So as for today, I think that reading the wiki page (alone) should give an idea of what has been done, the remaining work, and the other ideas/approaches that we could try. Cheers -- Laura Arjona Reina https://wiki.debian.org/LauraArjona
Bug#845297: Migration to Salsa
On Wed, Apr 18, 2018 at 11:01:03AM +0200, Laura Arjona Reina wrote: >Hello again > >El 18/04/18 a las 01:33, Laura Arjona Reina escribió: > >> >> * I will copy the migrated-to-git webwml repos (the tests) to >> https://salsa.debian.org/groups/webmaster-team/webwml/ labelling them as >> tests so people don't get confused. If you want to hack on those and you >> were not added as "developer" of the group yet, please request join the >> group and me or other admin will grant permissions. > >This is done now: > >https://salsa.debian.org/webmaster-team/webwml/test_webwml_cvs2git >https://salsa.debian.org/webmaster-team/webwml/test_webwml > >I will remove the "test repos" in my personal space in Salsa, and in Alioth. Cool. So... Laura and I were talking on IRC yesterday. I've been swamped with a range of family things that removed my "free" time over the last few months, so I've been unable to really help much so far. Laura has been talking about running a sprint to work on the migration. I think that sounds like a splendid idea, but I can't really travel much at the moment. However, I'm thinking of taking some vacation (a week or so) from my day job during May to get stuck in here. When are other people available? What can I help with? I'm OK-ish in perl, so I'm thinking of looking at the translation workflow stuff, unless that's already in hand. Is https://wiki.debian.org/WebsiteGitTransition still the canonical place to track the planning? -- Steve McIntyre, Cambridge, UK.st...@einval.com "I suspect most samba developers are already technically insane... Of course, since many of them are Australians, you can't tell." -- Linus Torvalds
Bug#845297: Migration to Salsa
Hello again El 18/04/18 a las 01:33, Laura Arjona Reina escribió: > > * I will copy the migrated-to-git webwml repos (the tests) to > https://salsa.debian.org/groups/webmaster-team/webwml/ labelling them as > tests so people don't get confused. If you want to hack on those and you > were not added as "developer" of the group yet, please request join the > group and me or other admin will grant permissions. This is done now: https://salsa.debian.org/webmaster-team/webwml/test_webwml_cvs2git https://salsa.debian.org/webmaster-team/webwml/test_webwml I will remove the "test repos" in my personal space in Salsa, and in Alioth. Regards -- Laura Arjona Reina https://wiki.debian.org/LauraArjona
