This is a good point. I somehow believed that releases/ only has the releases (in the _fcs branches,etc) while the main development is still in mail-silver (and the team repositories).
Only yesterday I noticed that releases/ seems to have in the default brach recent commits. Except it didn't occur to me I could just pull from releases/ into a main-silver clone! I will try it out. --emi On Wed, Dec 21, 2016 at 1:01 PM, Jaroslav Tulach <[email protected] > wrote: > On úterý 20. prosince 2016 15:47:08 CET Emilian Bold wrote: > > You guys weren't very clear which repository are reviewing for the > donation. > > > > Initially I converted releases/ but then I released I have some commits > of > > mine in there so I couldn't push it to github. > > > > So I did a fresh conversion for main-silver/ which I assumed is the > > repository Oracle will donate. > > > > I will re-do releases/ but it will take a bit. > > Hello Emilian, > I want to point out that main-silver and releases aren't that different. In > fact every commit inside of main-silver (like http://hg.netbeans.org/main- > silver/rev/f966be3cb73a) is also available in releases (at http:// > hg.netbeans.org/releases/rev/f966be3cb73a). > > However releases is more important, as it contains more. Especially the > actual > sources used for individual NetBeans releases - like > http://hg.netbeans.org/ > releases/rev/release82_fcs - those aren't in main-silver. > > As these repos are 90% the same, I thought it is enough to keep the > repository > you already have (https://github.com/emilianbold/main-silver) and just > migrate > rest of releases on top of it. > -jt > > > On Tue, Dec 20, 2016 at 3:31 PM, Jaroslav Tulach < > [email protected] > > > wrote: > > > > > > On pátek 9. prosince 2016 19:05:48 CET Emilian Bold wrote: > > > > Martin, I have just pushed https://github.com/ > emilianbold/main-silver > > > > > > You > > > > > > > may experiment with that. > > > > > > Hello Emilian, > > > I managed to fork & use your repository and everything seems great. I > have > > > a > > > functional job that executes > > > > > > $ ant build-platform > > > > > > for each pull request. I plan to add a call to "ant test-platform" > once it > > > is > > > stable enough[1]. Great work! I believe we shall use your Git > repository > > > as a > > > base (somehow) when donating code to Apache once my Oracle peers finish > > > review > > > of the code to donate. > > > > > > > To https://github.com/emilianbold/main-silver.git > > > > > > > > * [new branch] master -> master > > > > > > > > Branch master set up to track remote branch master from origin. > > > > > > Could you synchronize https://hg.netbeans.org/releases/ instead? It > > > contains > > > history of all the NetBeans releases (in branches like release82, etc.) > > > and it > > > is the repository that is currently under the review. Btw. The releases > > > repository contains everything that is available in the main-silver - > just > > > more. > > > > > > It would be fantastic, if you could create a complete mirror of the > > > releases > > > repository. Thanks again for your great work! > > > > > > -jt > > > > > > [1] http://deadlock.netbeans.org/job/prototypes-MavenDownload269264/ - > > > still > > > 12 test failures remaining > > > > > > > git push -u origin master > > > > Counting objects: 3951610, done. > > > > Delta compression using up to 8 threads. > > > > Compressing objects: 100% (732965/732965), done. > > > > Writing objects: 100% (3951610/3951610), 674.94 MiB | 717.00 KiB/s, > > > > done. > > > > Total 3951610 (delta 2068729), reused 3951610 (delta 2068729) > > > > remote: Resolving deltas: 100% (2068729/2068729), done. > > > > remote: Checking connectivity: 3951610, done. > > > > remote: warning: GH001: Large files detected. You may want to try Git > > > > > > Large > > > > > > > File Storage - https://git-lfs.github.com. > > > > remote: warning: See http://git.io/iEPt8g for more information. > > > > remote: warning: File dlight.util/test/manual/ > DLight_Simple_Tests/core > > > > > > is > > > > > > > 51.88 MB; this is larger than GitHub's recommended maximum file size > of > > > > 50.00 MB > > > > To https://github.com/emilianbold/main-silver.git > > > > > > > > * [new branch] master -> master > > > > > > > > Branch master set up to track remote branch master from origin. > > > > > > > > > > > > --emi > > > > > > > > On Wed, Dec 7, 2016 at 4:42 PM, Martin Balin < > [email protected]> > > > > > > > > wrote: > > > > > Hello Emilian, > > > > > I'm working at Oracle on NetBeans development and we would like to > > > > > > start > > > > > > > > fixing build scripts to use Git instead of HG. > > > > > This could be done earlier on your Git repo if you agree to as it > will > > > > > take time. Does not need to wait for final official donation of > > > > > > sources. > > > > > > > > Can you please send me the URL,... > > > > > Thank you Martin Balin > > > > > > > > > > On 24.11.2016 20:07, Emilian Bold wrote: > > > > >> At under 1GB the repository size is not an issue anymore. > > > > >> > > > > >> It's sad to see we will still have migration problems due to legal > > > > >> considerations. > > > > >> > > > > >> Could you provide an estimate how long it would take to verify and > > > > >> whitelist the entire codebase Oracle plans on donating? > > > > >> > > > > >> It's unclear to me how history would be preserved with an > incremental > > > > >> approach. > > > > >> > > > > >> I would prefer we migrate the whole thing in one piece with > history > > > > > > and > > > > > > > >> all. > > > > >> > > > > >> > > > > >> --emi > > > > >> > > > > >> On Thu, Nov 24, 2016 at 5:22 PM, Jaroslav Tulach < > > > > >> [email protected] > > > > >> > > > > >>> wrote: > > > > >>> Emilian, Jan, Mark, great work. > > > > >>> > > > > >>> Smooth migration from Hg to Git is essential for successful > > > > > > migration to > > > > > > > >>> Apache. Thanks a lot for investigating how to do that. > > > > >>> > > > > >>> My plan (as described in another email) is to prepare the code > > > > > > donation > > > > > > > >>> in > > > > >>> Hg > > > > >>> and update it incrementally with code integrated into Hg. > > > > >>> > > > > >>> Are your conversions methods ready for incremental updates or do > > > > >>> they > > > > >>> only > > > > >>> work as a one-time batch conversion? > > > > >>> > > > > >>> -jt > > > > >>> > > > > >>> On čtvrtek 24. listopadu 2016 10:41:50 CET Jan Lahoda wrote: > > > > >>>> Interesting. I tried "git gc --aggressive" on the Mark's > converted > > > > >>>> repository, and the result is: > > > > >>>> netbeans-import/.git$ du -hs . > > > > >>>> 792M . > > > > >>>> > > > > >>>> The original was: > > > > >>>> netbeans-import.git $ du -hs . > > > > >>>> 3,5G . > > > > >>>> > > > > >>>> (IIRC Mark was converting http://hg.netbeans.org/main, not > > > > > > releases, so > > > > > > > >>> the > > > > >>> > > > > >>>> repository is a little bit smaller than the releases one.) > > > > >>>> > > > > >>>> I tried: > > > > >>>> $ git log -p | sha1sum > > > > >>>> > > > > >>>> on both repositories, and the hashes appear to be the same. I > also > > > > >>>> tried > > > > >>> > > > > >>> to > > > > >>> > > > > >>>> clone the gc-ed repository using git clone --bare --no-local, > and > > > > > > the > > > > > > > >>>> resulting repository is still about the same size. So, this > seems > > > > > > good > > > > > > > >>>> to > > > > >>>> me, unless there is some downside I don't know about. > > > > >>>> > > > > >>>> Jan > > > > >>>> > > > > >>>> > > > > >>>> On Wed, Nov 23, 2016 at 8:26 PM, Emilian Bold < > > > > > > [email protected]> > > > > > > > >>>> wrote: > > > > >>>>> Actually I don't believe the data loss is that large. (There > may > > > > > > also > > > > > > > >>>> be > > > > >>>> > > > > >>>> mercurial commits that are intentionally ignored by the > conversion > > > > >>>> > > > > >>>> script, > > > > >>>> > > > > >>>> like commits that only add tags?) > > > > >>>> > > > > >>>>> hg log | grep '^changeset:' | wc -l > > > > >>>>> > > > > >>>>> 313209 > > > > >>>>> > > > > >>>>> git log | grep '^commit ' | wc -l > > > > >>>>> > > > > >>>>> 301478 > > > > >>>>> > > > > >>>>> So there is a difference of 11731 commits (about 4%) but those > > > > >>>>> couldn't > > > > >>>>> have such a large impact on repository size. > > > > >>>>> > > > > >>>>> I hope somebody else is willing to work with me on this so we > > > > > > document > > > > > > > >>>>> everything and do a reproducible repository conversion. > > > > >>>>> > > > > >>>>> > > > > >>>>> > > > > >>>>> --emi > > > > >>>>> > > > > >>>>> On Wed, Nov 23, 2016 at 9:10 PM, Emilian Bold < > > > > > > [email protected]> > > > > > > > >>>>> wrote: > > > > >>>>>> Well, I dunno what black magic `gc --aggressive` does but the > > > > >>>>> > > > > >>>>> repository > > > > >>>> > > > > >>>> is 0.85GB now! > > > > >>>> > > > > >>>>>> I also ran `git reflog expire` first but it didn't change the > > > > > > size at > > > > > > > >>>>> all. > > > > >>>>> > > > > >>>>> One thing to keep in mind is that I used --force although I > had 6 > > > > >>>>> > > > > >>>>>> commits > > > > >>>>>> with the warning "repository has at least one unnamed head". > > > > >>>>>> Which > > > > >>>>> > > > > >>>>> were > > > > >>>> > > > > >>>> probably all close branch commits (hg commit --close-branch). > > > > >>>> > > > > >>>>>> So I might have have data loss(!) since I believe I read > > > > >>>>> > > > > >>>>> hg-fast-export.sh > > > > >>>>> > > > > >>>>> picks only one unnamed head as the migration winner. I wonder > if > > > > > > the > > > > > > > >>>>> gc > > > > >>>> > > > > >>>> command didn't just purge a lot of valid commits from such an > > > > > > unnamed > > > > > > > >>>>> head > > > > >>>>> > > > > >>>>> and that's why the repository became so small. > > > > >>>>> > > > > >>>>>> Could somebody else try a test repository conversion and > validate > > > > > > my > > > > > > > >>>>>> numbers? > > > > >>>>>> > > > > >>>>>> git gc --aggressive --prune=now > > > > >>>>>> Counting objects: 4085031, done. > > > > >>>>>> Delta compression using up to 8 threads. > > > > >>>>>> Compressing objects: 100% (2909203/2909203), done. > > > > >>>>>> Writing objects: 100% (4085031/4085031), done. > > > > >>>>>> Total 4085031 (delta 2150468), reused 1585934 (delta 0) > > > > >>>>>> Checking connectivity: 4085031, done. > > > > >>>>>> > > > > >>>>>> > > > > >>>>>> > > > > >>>>>> --emi > > > > >>>>>> > > > > >>>>>> On Wed, Nov 23, 2016 at 7:59 PM, Paul Merlin < > > > > > > [email protected]> > > > > > > > >>>>>> wrote: > > > > >>>>>>> Hi Emilian, > > > > >>>>>>> > > > > >>>>>>> I see hg-fast-export.sh finished at some point. > > > > >>>>>>> > > > > >>>>>>>> As expected though, git does not have any of the disk space > > > > > > gains. > > > > > > > >>>>>>>> The > > > > >>>>>>>> converted git releases/ repository is 3.6GB. > > > > >>>>>>> > > > > >>>>>>> Just a thought. > > > > >>>>>>> Did you try some git cleanups after the conversion? > > > > >>>>>>> > > > > >>>>>>> git reflog expire --expire=now --all > > > > >>>>>>> git gc --aggressive --prune=now > > > > >>>>>>> > > > > >>>>>>> Cheers > > > > >>>>>>> > > > > >>>>>>> In case these statistics mean something: > > > > >>>>>>>> git-fast-import statistics: > > > > >>>>>>>> ------------------------------ > ------------------------------ > > > > >>>>>>> > > > > >>>>>>> --------- > > > > >>>> > > > > >>>> Alloc'd objects: 4090000 > > > > >>>> > > > > >>>>>>>> Total objects: 4085509 ( 40220100 duplicates > > > > >>>>>>>> > > > > >>>>>>> ) > > > > >>>>>>> > > > > >>>> blobs : 1036365 ( 28386238 duplicates 858087 > > > > >>>>>>> > > > > >>>>>>> deltas > > > > >>>> > > > > >>>> of > > > > >>>> > > > > >>>>> 969684 attempts) > > > > >>>>> > > > > >>>>>>>> trees : 2735935 ( 11833862 duplicates > 1370606 > > > > >>>>>>> > > > > >>>>>>> deltas > > > > >>>> > > > > >>>> of > > > > >>>> > > > > >>>>> 2613480 attempts) > > > > >>>>> > > > > >>>>>>>> commits: 313209 ( 0 duplicates > 0 > > > > >>>>>>> > > > > >>>>>>> deltas > > > > >>>> > > > > >>>> of > > > > >>>> > > > > >>>>> 0 attempts) > > > > >>>>> > > > > >>>>>>>> tags : 0 ( 0 duplicates > 0 > > > > >>>>>>> > > > > >>>>>>> deltas > > > > >>>> > > > > >>>> of > > > > >>>> > > > > >>>>> 0 attempts) > > > > >>>>>>>> > > > > >>>>>>>> Total branches: 1283 ( 346 loads ) > > > > >>>>>>>> > > > > >>>>>>>> marks: 1048576 ( 313209 unique ) > > > > >>>>>>>> atoms: 124011 > > > > >>>>>>>> > > > > >>>>>>>> Memory total: 218429 KiB > > > > >>>>>>>> > > > > >>>>>>>> pools: 26711 KiB > > > > >>>>>>>> > > > > >>>>>>>> objects: 191718 KiB > > > > >>>>>>>> > > > > >>>>>>>> ------------------------------ > ------------------------------ > > > > >>>>>>> > > > > >>>>>>> --------- > > > > >>>> > > > > >>>> pack_report: getpagesize() = 4096 > > > > >>>> > > > > >>>>>>>> pack_report: core.packedGitWindowSize = 1073741824 > > > > >>>>>>>> pack_report: core.packedGitLimit = 8589934592 > > > > >>>>>>>> pack_report: pack_used_ctr = 39000045 > > > > >>>>>>>> pack_report: pack_mmap_calls = 733040 > > > > >>>>>>>> pack_report: pack_open_windows = 4 / > 7 > > > > >>>>>>>> pack_report: pack_mapped = 4280730006 / > 6950823920 > > > > >>>>>>>> ------------------------------ > ------------------------------ > > > > >>>>>>> > > > > >>>>>>> --------- > > > > >>>>>>> > > > > >>>>>>>> --emi > > > > >>>>>>>> > > > > >>>>>>>> On Fri, Nov 18, 2016 at 1:32 PM, Emilian Bold < > > > > >>>>>>> > > > > >>>>>>> [email protected] > > > > >>>> > > > > >>>> wrote: > > > > >>>>>>>>> A releases/ clone which on my system takes 3.8GB is > reduced to > > > > >>>>>>>> > > > > >>>>>>>> 1.6GB > > > > >>>> > > > > >>>> with > > > > >>>> > > > > >>>>>>> the generaldelta and aggressivemergedeltas flags (took about > 14 > > > > >>>>>>> > > > > >>>>>>>> hours). > > > > >>>>> > > > > >>>>> Pretty impressive! > > > > >>>>> > > > > >>>>>>>>> Converting to git with hg-fast-export.sh complains that > > > > >>>>>>>> > > > > >>>>>>>> "repository > > > > >>>> > > > > >>>> has at > > > > >>>> > > > > >>>>>>> least one unnamed head" for about 6 revisions. With --force > I'm > > > > >>>>>>> > > > > >>>>>>>> able > > > > >>>> > > > > >>>> to > > > > >>>> > > > > >>>>> start the conversion but it hasn't finished yet. > > > > >>>>> > > > > >>>>>>>>> The git conversion is about 35% done and already using > 1.3GB. > > > > >>>>>>>>> > > > > >>>>>>>>> So... I assume it's going to need just like the original > > > > >>>>>>>> > > > > >>>>>>>> repository > > > > >>>> > > > > >>>> about > > > > >>>> > > > > >>>>>>> 3.8GB. > > > > >>>>>>> > > > > >>>>>>>>> I wonder if git has similar space-saving tricks? > > > > >>>>>>>>> > > > > >>>>>>>>> > > > > >>>>>>>>> > > > > >>>>>>>>> --emi > > > > >>>>>>>>> > > > > >>>>>>>>> On Thu, Nov 17, 2016 at 8:46 AM, Emilian Bold < > > > > >>>>>>>> > > > > >>>>>>>> [email protected]> > > > > >>>>> > > > > >>>>> wrote: > > > > >>>>>>>>>> Forgot about this. I've just started the Mercurial > repository > > > > >>>>>>>>> > > > > >>>>>>>>> conversion > > > > >>>>>>> > > > > >>>>>>> which will take a few hours. > > > > >>>>>>> > > > > >>>>>>>>>> Will report tomorrow or when it's done. > > > > >>>>>>>>>> > > > > >>>>>>>>>> > > > > >>>>>>>>>> --emi > > > > >>>>>>>>>> > > > > >>>>>>>>>> On Wed, Nov 16, 2016 at 11:18 PM, cowwoc < > > > > >>>>>>>>> > > > > >>>>>>>>> [email protected]> > > > > >>>> > > > > >>>> wrote: > > > > >>>>>>>> Hi Emilian, > > > > >>>>>>>> > > > > >>>>>>>>>>> Any update on this? > > > > >>>>>>>>>>> > > > > >>>>>>>>>>> Thanks, > > > > >>>>>>>>>>> Gili > > > > >>>>>>>>>>> > > > > >>>>>>>>>>> On 2016-11-11 01:33 (-0500), Emilian Bold < > [email protected]> > > > > >>>>>>>>>> > > > > >>>>>>>>>> wrote: > > > > >>>> Thank you for following through with this after we talked on > > > > >>>> > > > > >>>>>>>>>>>> IRC.> > > > > >>>>>>>>>>>> > > > > >>>>>>>>>>>> I will check later the size reduction for the releases/ > > > > > > repo.> > > >
