Re: [osmosis-dev] Version Control Full History and GIT Test
Another couple of updates. On Mon, Sep 26, 2011 at 11:23 PM, Brett Henderson br...@bretth.com wrote: I'm using my own repo for now. Given the nature of GIT it shouldn't be too limiting. It's fairly simple to create a Jenkins job for anybody that has a public GIT repo. But we could also use a central shared repo if that's the wish. A new repository has now been setup under the openstreetmap organisation on github. The source can be accessed from: https://github.com/openstreetmap/osmosis I've renamed my existing brettch/osmosis repo to osmosis-old and forked off the openstreetmap repository. I'll delete my osmosis-old repo once I'm sure everything is working. I'm not sure what to do about SVN. I'll probably rename the trunk so that nobody accidentally commits to it. I'll also update the wiki accordingly. I've renamed the trunk to trunk.orig, and created a new trunk with a README explaining how to find the latest version of source code. Brett ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] Version Control Full History and GIT Test
Hi All, Some further updates. On Sat, Sep 24, 2011 at 10:52 PM, Brett Henderson br...@bretth.com wrote: Hi All, I've uploaded a new git migration test repository: https://github.com/brettch/osmosis-test The repository has been renamed. https://github.com/brettch/osmosis It now contains full Osmosis history since day one, a complete set of annotated git tags for each SVN tag, and contains history from all branches that have contributed to the current codebase. It took a while to piece together all the pieces of history and fix the cases where the git svn command was confused. I'm very happy with the result. If anybody sees any issues, please let me know. There's still a few things to do: - Incorporate Igor's suggestion to make the git commands work under windows. Should be very straightforward. Done. Actually, Igor beat me to it and sent me a pull request with the fix. - Make unit tests run. They're failing due to GIT not supporting the creation of empty directories. I've fixed most of it already, but need to get a full local test environment running before I can complete it (I've lost my existing PostgreSQL test databases after a db version upgrade). Done. Available in the GIT repo only, I've stopped committing to SVN. - Update the Hudson server to support GIT builds. I'm not sure how much work is involved here. Done. The main SNAPSHOT build is running against my personal repo for now. If anybody wants a Jenkins job for their own repository please let me know. - Figure out which repo to treat as authoritative for builds. I'll probably just use mine for now because it's easy (for me at least ;-). It should be easy to move at any time. I'm using my own repo for now. Given the nature of GIT it shouldn't be too limiting. It's fairly simple to create a Jenkins job for anybody that has a public GIT repo. But we could also use a central shared repo if that's the wish. I'm not sure what to do about SVN. I'll probably rename the trunk so that nobody accidentally commits to it. I'll also update the wiki accordingly. Let me know if anybody runs into issues. Cheers, Brett ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] Version Control Full History and GIT Test
Hi All, I've uploaded a new git migration test repository: https://github.com/brettch/osmosis-test It now contains full Osmosis history since day one, a complete set of annotated git tags for each SVN tag, and contains history from all branches that have contributed to the current codebase. It took a while to piece together all the pieces of history and fix the cases where the git svn command was confused. I'm very happy with the result. If anybody sees any issues, please let me know. There's still a few things to do: - Incorporate Igor's suggestion to make the git commands work under windows. Should be very straightforward. - Make unit tests run. They're failing due to GIT not supporting the creation of empty directories. I've fixed most of it already, but need to get a full local test environment running before I can complete it (I've lost my existing PostgreSQL test databases after a db version upgrade). - Update the Hudson server to support GIT builds. I'm not sure how much work is involved here. - Figure out which repo to treat as authoritative for builds. I'll probably just use mine for now because it's easy (for me at least ;-). It should be easy to move at any time. Cheers, Brett ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] Version Control Full History and GIT Test
Hi Igor,
On Tue, Sep 20, 2011 at 1:12 AM, Igor Podolskiy
igor.podols...@vwi-stuttgart.de wrote:
Hi Brett,
I'm a bit confused :-) What does the openstreetmap organisation provide?
Can't I just upload the repo under my own user id and let people fork
from there as they see fit?
Sure you can do that. As far as I understand, the organization gives you
the benefit of having multiple users have write access to a repository - in
your repository, only you are able to push.
If you add it to an organization, it will be more or less like Subversion
now: all members of the organization would be able to directly push to that
repository, i.e. there will be Official OSM Developers' Osmosis tree and
other trees. If you just upload it to brettch/osmosis, it will be
Official Brett's Osmosis tree and other trees. So in the end, it's just
about how you label the official sources, how important this official
label is for you and who can push to it.
Okay, thanks. Now I understand. I'm not sure which way to go to be
honest. On the one hand using individual repositories makes me less central
to the project because everybody is a peer, but on the other hand having a
single central repository with multiple committers also makes me less
central because the repository I publish won't have my name on it :-)
I'm still leaning towards invididual repositories. I like the anarchist
approach where anybody can pick up and run in a new direction without being
a non-official fork. The biggest downside I can see is that individual
repositories are not as discoverable as a bunch of repositories under the
openstreetmap organisation.
The Hudson server will require some work also. I'm not sure whether it
should create tags based on an automated build (as it currently does) or
whether it should simply build off pre-existing tags. The former seems
better, but is more complicated.
I've uploaded a test repository to github. It contains full history from
day one (April 4th, 2007) including all tags that were ever created. Let
me know if you see any issues with it.
https://github.com/brettch/**osmosis-testhttps://github.com/brettch/osmosis-test
Looks good to me, just checked it out and played around with it, the tags
worked fine, the history too. It did build (ant publish) like a charm,
also.
I also tested your last commit with the git support for version string
generation (not yet in the osmosis-test on GitHub). Unfortunately, it
doesn't work on Windows (MSysGit) in its current state.
The reason for this: The git binary is actually a batch script called
git.cmd in msysgit. git works fine on the command line, but for some
reason, when Ant does an exec executable=git / it fails with a file not
found error even if it is in your PATH. If you exec
executable=git.cmd/, it works. My solution for this was to add a
condition to my ant script like this:
condition property=git.command value=git.cmd
os family=windows /
/condition
condition property=git.command value=git
notos family=windows //not
/condition
exec executable=${git.command} ... /
I would agree that this is somewhat ugly, but it worked for me[TM].
Oops. I used your ant scripts as an example but forgot to include that
bit. I'll take a look at it soon.
I'm going to re-implement the migration scripts. My existing approach lost
branch information from Subversion and trying to fix svn:mergeinfo tags to
re-create it will be very difficult. I've also realised that my old scripts
didn't create annotated tags, just lightweight tags (I didn't realise there
was a difference). I've created a second set of scripts that use GIT
features to re-build the history. It has preserved branches correctly, but
I haven't obtained the tags yet.
I'm no longer using my SVN repo for hosting the migration scripts. They're
here:
https://github.com/brettch/osmosis-svn-to-git
I'll post a new migrated repo over the next week or two when I finally get
around to finishing it.
Brett
___
osmosis-dev mailing list
osmosis-dev@openstreetmap.org
http://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] Version Control Full History and GIT Test
Hi Igor, On Thu, Sep 8, 2011 at 1:01 AM, Igor Podolskiy igor.podols...@vwi-stuttgart.de wrote: Hi Brett, • I'm pondering a move to GIT. Primarily because it gives people a better opportunity to experiment with the codebase. Secondly because I am experimenting with GIT anyway and wanted a real-world task to work with. [...] Any thoughts re a possible GIT move? that would be _really_ cool. In fact, I've been running more or less of a mirror of the latest revisions of Osmosis trunk on Github [1]. It is really much simpler to work and try things out and keep track of upstream and so on than with SVN. There's even a place for a GIT repository already, the Openstreetmap organization on GitHub [2]. I'll check those links out. I know very little about GitHub (or Git for that matter) so I don't know how users, organisations, etc work. The only thing we'd need to change would be the build script for the snapshot version string - currently it uses the SVN revision. In osmosis-fastused, I already made a similar Ant build script which uses the git revision (or tag, if available), to be found here [3]. Cool, thanks for that. I still have some work to do before I have a complete migration tool. I think my re-built SVN repo is looking good but I need to do all the GIT merge steps now including preservation of existing tags. As soon as I get that working I'll incorporate your build changes. Brett ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] Version Control Full History and GIT Test
On Thu, Sep 8, 2011 at 8:36 PM, Brett Henderson br...@bretth.com wrote: Hi Igor, On Thu, Sep 8, 2011 at 1:01 AM, Igor Podolskiy igor.podols...@vwi-stuttgart.de wrote: Hi Brett, • I'm pondering a move to GIT. Primarily because it gives people a better opportunity to experiment with the codebase. Secondly because I am experimenting with GIT anyway and wanted a real-world task to work with. [...] Any thoughts re a possible GIT move? that would be _really_ cool. In fact, I've been running more or less of a mirror of the latest revisions of Osmosis trunk on Github [1]. It is really much simpler to work and try things out and keep track of upstream and so on than with SVN. There's even a place for a GIT repository already, the Openstreetmap organization on GitHub [2]. I'll check those links out. I know very little about GitHub (or Git for that matter) so I don't know how users, organisations, etc work. I'm a bit confused :-) What does the openstreetmap organisation provide? Can't I just upload the repo under my own user id and let people fork from there as they see fit? The only thing we'd need to change would be the build script for the snapshot version string - currently it uses the SVN revision. In osmosis-fastused, I already made a similar Ant build script which uses the git revision (or tag, if available), to be found here [3]. Cool, thanks for that. I still have some work to do before I have a complete migration tool. I think my re-built SVN repo is looking good but I need to do all the GIT merge steps now including preservation of existing tags. As soon as I get that working I'll incorporate your build changes. I've uploaded a test repository to github. It contains full history from day one (April 4th, 2007) including all tags that were ever created. Let me know if you see any issues with it. https://github.com/brettch/osmosis-test One thing I'm unsure about is the list of email addresses in the history. Once I've done the migration I can't change them. This is my current list. Getting them right gives the advantage of allowing github to match commits to github users. https://www.bretth.com/repos/main/osmosis-svn-to-git/users.txt I've used up my Osmosis time quota for now, but I'll take a look at the ant build over the coming days. Brett ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
[osmosis-dev] Version Control Full History and GIT Test
Hi All, I've been playing in a couple of areas with regards to Osmosis version control. - I'm pondering a move to GIT. Primarily because it gives people a better opportunity to experiment with the codebase. Secondly because I am experimenting with GIT anyway and wanted a real-world task to work with. - The svn history is disjointed due to the early history being hosted on my server, and subsequent history being on the osm svn server. In addition there have been some renames and re-organisation that are confusing and break potential migrations to other tools (ie. GIT). I decided I'd try to fix the SVN history and produce a single repo with complete history. What started out as a simple shell script with some grep and sed statements quickly got out of hand and became a full-blown java-based SVN dump file re-writer that looks quite a lot like Osmosis itself. Perhaps I should have written it as an Osmosis plugin :-) Anyway, it does the following major operations. - Creates a new empty repo with standard trunk/tags/branches structure at the root. - Extracts history from http://www.bretth.com/repos/main from revision 417-471 which was the initial period of Osmosis when it was called conduit. - Replaces the conduit path with trunk. - Removes the initial creation of conduit directory which isn't necessary as data will be imported into existing trunk. - Renumbers the revisions from 1 upwards. - Skips revision 472 which was a rename from conduit to osmosis. We don't need this because the project name isn't reflected in the new repo, the project is stored in the root. - Extracts history from revision 473-474 which are the only two revisions with code stored in the root of the osmosis project. - Replaces the osmosis path with trunk. - Renumbers the revisions. - Skips revision 475 which was a re-organisation into a trunk/tags/branches structure. This is already done in the new repository. - Extracts all remaining revisions in my repository from 475-HEAD. - Removes the osmosis prefix on paths. - Renumbers the revisions. - Skips 4742 because that's the original import which is already done in the new repo. - Extracts history from http://svn.openstreetmap.org/applications/utils/osmosis from 4743-12410. - Replaces the applications/utils/osmosis path with trunk. - Renumbers the revisions. - Skips 12411 which was a re-organisation into a trunk/tags/branches structure. - Extracts remaining history from 12412-HEAD. - Replaces a copyfrom path for creation of 0.28 and 0.29 tags which refer to revisions prior to the repo re-structure and thus have incorrect path structures in the dump file. - Removes the applications/utils/osmosis prefix on paths. - Renumbers the revisions. It seems to have all worked well. It provides a single repo with full history from day 1, gives a consistent directory layout for the entire history, and includes all tags that were ever created. I got part way there with shell scripts, but the real trick was the revision renumbering which had to be done because there are numerous occasions where revisions copy files from significantly older revisions where the main svn repo moved on between me copying a latest version file and actually doing a commit. Renumbering required knowledge of entire history and called for a better language. One thing I haven't fixed is the svn:mergeinfo tags (it's hard), so I'll probably remove them completely to avoid any confusion. If you want to try it out for yourself, the migration app is here: https://www.bretth.com/repos/main/osmosis-svn-to-git If you load it up in Eclipse, you can simply run the DumpFileMigrator class. From a command line you'd need to compile the app and run it with the shell scripts in your working directory. Note that it will take a *very* long time the first time you run it because it generates a complete snapshot of the main osm repository locally using svnsync. If you already have a replica you can copy it in because it continues to update the one it has each time you run it. Any thoughts re a possible GIT move? Brett ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] Version Control Full History and GIT Test
Hi Brett, • I'm pondering a move to GIT. Primarily because it gives people a better opportunity to experiment with the codebase. Secondly because I am experimenting with GIT anyway and wanted a real-world task to work with. [...] Any thoughts re a possible GIT move? that would be _really_ cool. In fact, I've been running more or less of a mirror of the latest revisions of Osmosis trunk on Github [1]. It is really much simpler to work and try things out and keep track of upstream and so on than with SVN. There's even a place for a GIT repository already, the Openstreetmap organization on GitHub [2]. The only thing we'd need to change would be the build script for the snapshot version string - currently it uses the SVN revision. In osmosis-fastused, I already made a similar Ant build script which uses the git revision (or tag, if available), to be found here [3]. Greetings from Stuttgart, Igor [1] https://github.com/podolsir/osmosis-trunk-mirror [2] https://github.com/openstreetmap [3] https://github.com/podolsir/osmosis-fastused/blob/master/build.xml#L35 ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
