Re: [gdal-dev] Trac to GitHub

[Mateusz Loskot]

On 22 March 2018 at 17:17, Even Rouault  wrote:
>
> I've prototyped what should be the new repository (baring any new commits in 
> the meantime) and
> put it temporarily at https://github.com/rouault/gdal_wip
>
> You can check if branches, tags etc look good. I've removed a few useless 
> branches and tags.
> Branches are named release/{major}.{minor} and tags v{major}.{minor}.{micro}


Looks good!

For a few branches, I've checked that
in release/X.Y branch
number in VERSION file
matches latest vX.Y.Z tag

Once again, big thank you Even!

Best regards,
-- 
Mateusz Loskot, http://mateusz.loskot.net
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] Trac to GitHub

[Even Rouault]

Hi,

I've prototyped what should be the new repository (baring any new commits in 
the meantime) and
put it temporarily at https://github.com/rouault/gdal_wip

You can check if branches, tags etc look good. I've removed a few useless 
branches and tags.
Branches are named release/{major}.{minor} and tags v{major}.{minor}.{micro}

The procedure I applied:

git clone --mirror git://github.com/OSGeo/gdal.git gdal/.git
cd gdal
git config --unset core.bare
git config receive.denyCurrentBranch updateInstead
git branch -D asi-20021114-branch
git branch -D asi-20030922-branch 
git branch -D asi-rel-1-6-patches 
git branch -D SRRefCount 
git branch -D tags/asi-rel-1-6
git branch -D tags/asi-rel-1-82
git branch -D tags/1.11.0beta1
git branch -D tags/1.11.1@27202
git branch -D tags/1.11.1@27733
git branch -D tags/1.11.4@27202
git branch -D tags/1.11.4@33077
git branch -D tags/1.11beta1
git branch -m trunk master
git reset --hard HEAD
git checkout master
git filter-branch --msg-filter 'python 
/home/even/gdal/historyrewrite/rewrite.py' -- --all
for i in `git branch | grep tags | grep gdal_ | sed "s/  tags\///"`; do j=`echo 
v$i | sed "s/gdal_//" | sed "s/_/./g"`; git checkout tags/$i; git tag -a $j -m 
"Create tag $j"; done
git checkout master
for i in `git branch | grep tags | grep -v gdal_ | sed "s/  tags\///"`; do 
j="v$i"; git checkout tags/$i; git tag -a $j -m "Create tag $j"; done
git checkout master
for i in `git branch | grep tags | sed "s/  tags\///"`; do git branch -D 
tags/$i; done
for i in `git branch | grep "\." | sed "s/  //"`; do git branch -m $i 
"release/$i"; done
git branch -m "release/1.4-esri" "vendor/1.4-esri"


git push gdal_wip --all
git push gdal_wip --tags

Even

-- 
Spatialys - Geospatial professional services
http://www.spatialys.com
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] Trac to GitHub

[Ben Elliston]

On 20/03/18 00:25, Even Rouault wrote:

> A more complicated version of the above is that we would migrate only the 
> open 
> Trac tickets to github (so < 600 instead of 6000). And we would add in each 
> open Trac ticket a message like "This ticket has been migated to https://
> github.com/OSGeo/gdal/issue/4567", and close it. But that requires still some 
> migration effort and deal with github API.

I think it makes sense to migrate the open tickets. Users go to the
trouble of reporting problems and they should all be investigated (at
some stage).

To my mind, there is an issued about what to do about closed tickets.
It's convenient to not have to search for bu reports in two different
places when trying to search for the history of a bug. So I think there
may also be justification for migtating the closed tickets.

It all comes down to how difficult the migration process is.

Cheers, Ben
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] Trac to GitHub

[Ben Elliston]

On 20/03/18 00:25, Even Rouault wrote:

> I actually discussed about that with Howard yesterday, and he suggested an 
> even easier and least-effort solution. Do we actually need to migrate the 
> existing Trac ticket database to github ?

This brings up another thing I have been wondering about: when will the
GDAL source tree be moved from Subversion to git?  It's problematic
working with an SVN mirror.

Cheers, Ben
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] Trac to GitHub

[Mateusz Loskot]

On 19 March 2018 at 18:14, Even Rouault  wrote:
>>> 3) Close all Trac tickets with assignment to a "closed-before-github-
>>> migration" milestone, and a message "Issue reporting has now been migrated 
>>> >to
>> https://github.com/OSGeo/gdal/issues...;
>>
>>I guess, you mean manual action
>
> Manual action, but lukily to be done just once for all tickets (batch action)
>
> Basically open
> https://trac.osgeo.org/gdal/query?status=assigned=new=reopened=1000
>
> and if you have the sufficient rights, click on the top-left checkbox to 
> select all the tickets,
> go to the bottom to the "Batch modify" tab, and define a milestone, add a 
> comment and click on "Change tickets"
> So a 1-minute action.

THAT, I can do ;)


>> > git filter-branch -f --msg-filter 'python /home/even/rewrite.py'  -- trunk
>> > [...]
>>
>> You lost me shortly after the while True, I suck at advanced git terribly :)
>
> Huh that's just my-not-so-idiomatic Python hacking to parse the old commit 
> message and change it ;-)

Actually, it was `git filter-branch` that already sent me off :)


>> The main goal is to apply semantic to the raw numbers, even fi the URL>> 
>> gets broken
>> in future (eg. read-only GDAL Trac moves to some archives server).
>
> Yes the point is to make it obvious that those are references to Trac and not 
> github issues.
>
> I'm preparing a RFC to summarize various aspects and have some official plan 
> that can be discussed.

Thank you!

Best regards,
-- 
Mateusz Loskot, http://mateusz.loskot.net
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] Trac to GitHub

[Even Rouault]

Mateuze,

Answering several of your points of your last emails;

>> 3) Close all Trac tickets with assignment to a "closed-before-github-
>> migration" milestone, and a message "Issue reporting has now been migrated 
>> >to
> https://github.com/OSGeo/gdal/issues...;
>
>I guess, you mean manual action

Manual action, but lukily to be done just once for all tickets (batch action)

Basically open
https://trac.osgeo.org/gdal/query?status=assigned=new=reopened=1000

and if you have the sufficient rights, click on the top-left checkbox to select 
all the tickets,
go to the bottom to the "Batch modify" tab, and define a milestone, add a 
comment and click on "Change tickets"
So a 1-minute action.

> > 
> > git filter-branch -f --msg-filter 'python /home/even/rewrite.py'  -- trunk
> > [...]
> 
> You lost me shortly after the while True, I suck at advanced git terribly :)

Huh that's just my-not-so-idiomatic Python hacking to parse the old commit 
message and change it ;-)

> > I'v uploaded the log of the changes (the content of /tmp/log.txt) in
> > http://even.rouault.free.fr/log.txt It shows the original and modified
> > commit messages (just for commits for which a modification was done) In
> > case you have the motivation to actually review it, let me know if you
> > see some weird things in it...
> Slight chance, I suspect, there are numbers that denote reference to
> ticket in the old GDAL Bugzilla,
> but this shold not be an issue if it gets replaced with the Trac URL.

I did try to avoid rewriting those commit messages that refered to the old (and 
now disappeared) bugzilla

> The main goal is to apply semantic to the raw numbers, even fi the URL
> gets broken
> in future (eg. read-only GDAL Trac moves to some archives server).

Yes the point is to make it obvious that those are references to Trac and not 
github issues.

I'm preparing a RFC to summarize various aspects and have some official plan 
that can be discussed.

-- 
Spatialys - Geospatial professional services
http://www.spatialys.com
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] Trac to GitHub

[Mateusz Loskot]

On 19 March 2018 at 17:42, Even Rouault  wrote:
> Regarding my point 1), I've experimented locally on my git clone to rewrite 
> references
> to Trac tickets like "fix #1234" to become "fix 
> https://trac.osgeo.org/gdal/ticket/1234;

Looks good.

> with
>
> git filter-branch -f --msg-filter 'python /home/even/rewrite.py'  -- trunk
> [...]

You lost me shortly after the while True, I suck at advanced git terribly :)

> I'v uploaded the log of the changes (the content of /tmp/log.txt) in 
> http://even.rouault.free.fr/log.txt
> It shows the original and modified commit messages (just for commits for 
> which a modification was done)
> In case you have the motivation to actually review it, let me know if you see 
> some weird things in it...

Slight chance, I suspect, there are numbers that denote reference to
ticket in the old GDAL Bugzilla,
but this shold not be an issue if it gets replaced with the Trac URL.
The main goal is to apply semantic to the raw numbers, even fi the URL
gets broken
in future (eg. read-only GDAL Trac moves to some archives server).

Best regards,
-- 
Mateusz Loskot, http://mateusz.loskot.net
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] Trac to GitHub

[Mateusz Loskot]

On 19 March 2018 at 14:25, Even Rouault  wrote:
> I actually discussed about that with Howard yesterday, and he suggested an
> even easier and least-effort solution. Do we actually need to migrate the
> existing Trac ticket database to github ?

I'm glad Howard came up with suggestion to simplify :-)

I think this will help us avoid issues with GitHub API,
or artificial owner of issues (a bot user) and some others that
Paul Ramsey encountered (there are some discussed in IRC logs).

> So to sum up my thoughts would be to:
> 1) Rewrite the github history (still need to figure out how to automate that)
> to fix references to Trac ticket, and force-push the result to github.com/
> OSGeo/gdal. Note: that would invalidate current forks, so people with active
> work would probably have to rebase or to export as a patch and re-apply on top
> of updated Git repository.
> 2) Open github for filing tickets
> 3) Close all Trac tickets with assignment to a "closed-before-github-
> migration" milestone, and a message "Issue reporting has now been migrated to
> https://github.com/OSGeo/gdal/issues...;

I guess, you mean manual action, if you need help, I can do this monkey
job in the evening(s). Let me know.

> Does that sound good ?

To me, it does.

Best regards,
-- 
Mateusz Loskot, http://mateusz.loskot.net
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] Trac to GitHub

[Mateusz Loskot]

On 19 March 2018 at 15:15, Even Rouault  wrote:
> On lundi 19 mars 2018 14:55:14 CET Tamas Szekeres wrote:
>> I don't think the trac tickets should be closed automatically. The ticket
>> owners should decide either to close (with a meaningful comment) or copy
>> it's variant to github if necessary.
>> I'm fine with option #2 and #4 and preserving/updating the history would be
>> a plus, but not a necessary requirement.
>
> Yeah, we could still allow modifying existing tickets in Trac, but disable
> creation of new ones. If fixing a Trac ticket in git, the committer should
> make sure to add a manual message to the Trac ticket referencing the commit,
> but that's acceptable I guess.
>
> The advantage of 3) (close all tickets) is that it would be a good way to get
> rid of a lot of historical tickets still opened but about which nobody really
> takes care about.

Even,

If you see such puring as beneficial yourself, as the principal maintainer,
I see no reason to object.
If OP of a closed ticked does not come back, it could be interpreted as
either an issue is no longer an issue or OP is no longer interested in
solveing it.

Switching Trac to read-only, completely, would stop any activities
there (eg. commenting)
what I think would be good to kickstart actual switch to GitHub.

Best regards,
-- 
Mateusz Loskot, http://mateusz.loskot.net
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] Trac to GitHub

[Even Rouault]

Regarding my point 1), I've experimented locally on my git clone to rewrite 
references
to Trac tickets like "fix #1234" to become "fix 
https://trac.osgeo.org/gdal/ticket/1234;

with

git filter-branch -f --msg-filter 'python /home/even/rewrite.py'  -- trunk

where rewrite.py is

{{{
import sys
msg = sys.stdin.read()

# Don't touch messages that reference other databases
if msg.find('MITAB bug ') >= 0 or msg.find('Safe bug ') >= 0 or 
msg.find('bugzilla') >= 0 or msg.find('Bugzilla') >= 0:
sys.stdout.write(msg)
sys.exit(0)

oldpos = 0
old_msg = msg
while True:

# We already have reference to github pull requests written like
# 'github #1234', so skip them
newpos = msg.find('github #', oldpos)
if newpos >= 0:
oldpos = newpos + len('github #')
continue

# Exception...
newpos = msg.find('patch #1 ', oldpos)
if newpos >= 0:
oldpos = newpos + len('patch #1 ')
continue

# Exception...
newpos = msg.find('bug 1 ', oldpos)
if newpos >= 0:
oldpos = newpos + len('bug 1 ')
continue

# Fix 'bug 1234'
newpos = msg.find('bug ', oldpos)
if newpos >= 0:
if newpos == len(msg) - 4:
break
if not(msg[newpos+4] >= '1' and msg[newpos+4] <= '9'):
oldpos = newpos + 4
continue

msg = msg[0:newpos] + 'https://trac.osgeo.org/gdal/ticket/' + 
msg[newpos+4:]
oldpos = newpos
continue

# Fix 'ticket 1234'
newpos = msg.find('ticket ', oldpos)
if newpos >= 0:
if newpos == len(msg) - 7:
break
if not(msg[newpos+7] >= '1' and msg[newpos+7] <= '9'):
oldpos = newpos + 7
continue

msg = msg[0:newpos] + 'https://trac.osgeo.org/gdal/ticket/' + 
msg[newpos+7:]
oldpos = newpos
continue

# Fix '#1234'
newpos = msg.find('#', oldpos)
if newpos >= 0:
if newpos == len(msg) - 1:
break
if not(msg[newpos+1] >= '1' and msg[newpos+1] <= '9'):
oldpos = newpos + 1
continue

# Skip stacktraces like '#1 0xdeadbeef'
space_pos = msg.find(' ', newpos)
if space_pos > 0 and space_pos + 2 < len(msg) and msg[space_pos + 1] == 
'0' and msg[space_pos + 2] == 'x':
oldpos = newpos + 1
continue

msg = msg[0:newpos] + 'https://trac.osgeo.org/gdal/ticket/' + 
msg[newpos+1:]
oldpos = newpos
continue

break

if msg != old_msg:
f = open('/tmp/log.txt', 'a')
f.write('Old message was:\n' + old_msg + 'New message is:\n' + msg + '\n')
f.close()

sys.stdout.write(msg)
}}}


I'v uploaded the log of the changes (the content of /tmp/log.txt) in 
http://even.rouault.free.fr/log.txt
It shows the original and modified commit messages (just for commits for which 
a modification was done)
In case you have the motivation to actually review it, let me know if you see 
some weird things in it...

Even

-- 
Spatialys - Geospatial professional services
http://www.spatialys.com
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] Trac to GitHub

[Even Rouault]

On lundi 19 mars 2018 14:55:14 CET Tamas Szekeres wrote:
> I don't think the trac tickets should be closed automatically. The ticket
> owners should decide either to close (with a meaningful comment) or copy
> it's variant to github if necessary.
> I'm fine with option #2 and #4 and preserving/updating the history would be
> a plus, but not a necessary requirement.

Yeah, we could still allow modifying existing tickets in Trac, but disable 
creation of new ones. If fixing a Trac ticket in git, the committer should 
make sure to add a manual message to the Trac ticket referencing the commit, 
but that's acceptable I guess.

The advantage of 3) (close all tickets) is that it would be a good way to get 
rid of a lot of historical tickets still opened but about which nobody really 
takes care about.

Even

-- 
Spatialys - Geospatial professional services
http://www.spatialys.com
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] Trac to GitHub

[Tamas Szekeres]

I don't think the trac tickets should be closed automatically. The ticket
owners should decide either to close (with a meaningful comment) or copy
it's variant to github if necessary.
I'm fine with option #2 and #4 and preserving/updating the history would be
a plus, but not a necessary requirement.

Best regards,

Tamas



2018-03-19 14:25 GMT+01:00 Even Rouault :

> Hi,
>
> Adding gdal-dev into the loop to get more feedback.
>
> I actually discussed about that with Howard yesterday, and he suggested an
> even easier and least-effort solution. Do we actually need to migrate the
> existing Trac ticket database to github ?
>
> If not, we could just freeze Trac as read-only and decide that we just open
> the github repo for tickets...
> What would be nice to do is to rewrite a bit the git history of the current
> mirror so that commit messages like "Fix blabla (fixes #1234)" as
> rewritten as
> "Fix blabla (fixes https://trac.osgeo.org/gdal/ticket/1234)"
>
> A more complicated version of the above is that we would migrate only the
> open
> Trac tickets to github (so < 600 instead of 6000). And we would add in each
> open Trac ticket a message like "This ticket has been migated to https://
> github.com/OSGeo/gdal/issue/4567", and close it. But that requires still
> some
> migration effort and deal with github API.
> A simpler variant of the above would be just close all those open tickets,
> assigning them to some milestone "closed-before-github-migration" with a
> message "Issue reporting has now been migrated to
> https://github.com/OSGeo/
> gdal/issues. If that issue is still valid, please file a ticket over
> there".
> That can be done simply in a few seconds from Trac UI with a "batch
> modifiy"
> action.
>
> So to sum up my thoughts would be to:
> 1) Rewrite the github history (still need to figure out how to automate
> that)
> to fix references to Trac ticket, and force-push the result to github.com/
> OSGeo/gdal. Note: that would invalidate current forks, so people with
> active
> work would probably have to rebase or to export as a patch and re-apply on
> top
> of updated Git repository.
> 2) Open github for filing tickets
> 3) Close all Trac tickets with assignment to a "closed-before-github-
> migration" milestone, and a message "Issue reporting has now been migrated
> to
> https://github.com/OSGeo/gdal/issues...;
> 4) Remove TICKET_CREATE rights to authenticated users of Trac
>
> Does that sound good ?
>
> Even
>
>
> On lundi 19 mars 2018 11:48:15 CET Mateusz Loskot wrote:
> > Hi Even,
> [...]
> > I've just pushed some basic stab at exporting Trac to GitHub
> > which I started prototyping in the beginning of October last year
> > https://github.com/mloskot/trac-to-github
> >
> > In October, Paul Ramsey released his bunch of scripts
> > https://github.com/pramsey/postgis-to-github/
> > which, I think, cover the procedure more completely
> > and it's also based on GitHub API.
> > Shortly, instead of continue my development, Paul's solution may be
> > the way to go.
> > I haven't tried to run Paul's scripts, so I don't know what technically
> > is stopping the PostGIS migration, if anything.
> >
> > Generally, I think GitHub API approach is the way to go.
> > Annoyingly, the rate limits seem to lead to strange issues that I
> > experienced (eg. adding or adding 30 labels, some are left over).
> > This confirms what Thomas Bonfort was warning about and suggesting
> > to reach out to GitHub support stating you are running a batch import.
> >
> > I don't if Paul reached to GitHub support before performing PostGIS test
> > import https://github.com/pramsey/postgis-gh/issues
> > but it looks that a few thousands of tickets made it through.
> >
> > I've taken the liberty to CC to Paul perhaps he could shed more light.
> [...]
> >
> > Best regards,
>
>
> --
> Spatialys - Geospatial professional services
> http://www.spatialys.com
> ___
> gdal-dev mailing list
> gdal-dev@lists.osgeo.org
> https://lists.osgeo.org/mailman/listinfo/gdal-dev
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] Trac to GitHub

[Even Rouault]

Hi,

Adding gdal-dev into the loop to get more feedback.

I actually discussed about that with Howard yesterday, and he suggested an 
even easier and least-effort solution. Do we actually need to migrate the 
existing Trac ticket database to github ?

If not, we could just freeze Trac as read-only and decide that we just open 
the github repo for tickets...
What would be nice to do is to rewrite a bit the git history of the current 
mirror so that commit messages like "Fix blabla (fixes #1234)" as rewritten as 
"Fix blabla (fixes https://trac.osgeo.org/gdal/ticket/1234)"

A more complicated version of the above is that we would migrate only the open 
Trac tickets to github (so < 600 instead of 6000). And we would add in each 
open Trac ticket a message like "This ticket has been migated to https://
github.com/OSGeo/gdal/issue/4567", and close it. But that requires still some 
migration effort and deal with github API.
A simpler variant of the above would be just close all those open tickets, 
assigning them to some milestone "closed-before-github-migration" with a 
message "Issue reporting has now been migrated to https://github.com/OSGeo/
gdal/issues. If that issue is still valid, please file a ticket over there". 
That can be done simply in a few seconds from Trac UI with a "batch modifiy" 
action.

So to sum up my thoughts would be to:
1) Rewrite the github history (still need to figure out how to automate that) 
to fix references to Trac ticket, and force-push the result to github.com/
OSGeo/gdal. Note: that would invalidate current forks, so people with active 
work would probably have to rebase or to export as a patch and re-apply on top 
of updated Git repository.
2) Open github for filing tickets
3) Close all Trac tickets with assignment to a "closed-before-github-
migration" milestone, and a message "Issue reporting has now been migrated to 
https://github.com/OSGeo/gdal/issues...;
4) Remove TICKET_CREATE rights to authenticated users of Trac

Does that sound good ?

Even


On lundi 19 mars 2018 11:48:15 CET Mateusz Loskot wrote:
> Hi Even,
[...]
> I've just pushed some basic stab at exporting Trac to GitHub
> which I started prototyping in the beginning of October last year
> https://github.com/mloskot/trac-to-github
> 
> In October, Paul Ramsey released his bunch of scripts
> https://github.com/pramsey/postgis-to-github/
> which, I think, cover the procedure more completely
> and it's also based on GitHub API.
> Shortly, instead of continue my development, Paul's solution may be
> the way to go.
> I haven't tried to run Paul's scripts, so I don't know what technically
> is stopping the PostGIS migration, if anything.
> 
> Generally, I think GitHub API approach is the way to go.
> Annoyingly, the rate limits seem to lead to strange issues that I
> experienced (eg. adding or adding 30 labels, some are left over).
> This confirms what Thomas Bonfort was warning about and suggesting
> to reach out to GitHub support stating you are running a batch import.
> 
> I don't if Paul reached to GitHub support before performing PostGIS test
> import https://github.com/pramsey/postgis-gh/issues
> but it looks that a few thousands of tickets made it through.
> 
> I've taken the liberty to CC to Paul perhaps he could shed more light.
[...]
> 
> Best regards,


-- 
Spatialys - Geospatial professional services
http://www.spatialys.com
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev