Re: Draft daily (hmm) cloud builds of Android toolchain from bzr
Hello James, On Thu, 16 Jun 2011 16:43:57 -0400 James Westby james.wes...@canonical.com wrote: On Thu, 16 Jun 2011 14:47:59 +0300, Paul Sokolovsky paul.sokolov...@linaro.org wrote: As for our mirror service, adding such support in maintainable manner would require its complete rewrite (which will also allow to implement other functionality and improvements). Adding support for what would require a complete rewrite? Adding support for general/flexible mode of operation, and extending scope. In particular, following things were discussed/under consideration: 1. Implement generic git mirror. 2. Implement repo mirroring in terms of the above. 3. Implement more flexible mirror pre/postprocessing, to allow for improvements to areas like high-availability and updates scheduling. 4. The above would require either good rehashing of current Twisted control flow, or getting rif of Twisted. 5. Add HTTP and bzr mirroring. Thanks, James -- Best Regards, Paul ___ linaro-dev mailing list linaro-dev@lists.linaro.org http://lists.linaro.org/mailman/listinfo/linaro-dev
Re: Draft daily (hmm) cloud builds of Android toolchain from bzr
On Thu, 16 Jun 2011 14:47:59 +0300, Paul Sokolovsky paul.sokolov...@linaro.org wrote: As for our mirror service, adding such support in maintainable manner would require its complete rewrite (which will also allow to implement other functionality and improvements). Adding support for what would require a complete rewrite? Thanks, James ___ linaro-dev mailing list linaro-dev@lists.linaro.org http://lists.linaro.org/mailman/listinfo/linaro-dev
Re: Draft daily (hmm) cloud builds of Android toolchain from bzr
On Mon, Jun 13, 2011 at 2:41 PM, Paul Sokolovsky paul.sokolov...@linaro.org wrote: So, where do we go from here? Unless there's tip export feature directly out of remote repo for bzr (bzr export seem to requite local working copy), we'd need to play mirror, this time bzr, again %). For the reason that it is trivial to do release manifests (pinned) I would really love to see bzr feature (as well as static tarball feature) to be available in repo and then use our repo mirror to do the mirroring in the cloud for that as well. Could you check how much effort that is to add support to repo and to extend our cloud mirror to also understand bzr/tarballs? Based on that we could decide whether we want to have a quick hack to get down the branching time quickly or if we wait till we can do this right using repo all together. -- - Alexander ___ linaro-dev mailing list linaro-dev@lists.linaro.org http://lists.linaro.org/mailman/listinfo/linaro-dev
Re: Draft daily (hmm) cloud builds of Android toolchain from bzr
Hello Michael, On Tue, 14 Jun 2011 08:41:21 +1200 Michael Hope michael.h...@linaro.org wrote: [] Note that it now expects bzr branch with explicit version, e.g. lp:gcc-linaro/4.5 or lp:gcc-linaro/4.6 . Minor nit: these are called 'series'. I understand that in Launchpad's terms it's series, but there's doesn't seem to be such notion on bzr level, so how they're represented in bzr - are they 2 separate repositories, or 2 branches of the same repo? Well, now the issue: the builds above are not quite ready to be daily. That's because bzr checkout is slow, and is also quite big. Normal bzr clone takes 1.1Gb and cloud build takes 1hr instead of 30min with a tarball. I tried to play with bzr checkout --lightweight, but ctrl+c'ed it when it showed that it dealt with 1.3Gb of stuff (surprisingly, after break, I saw almost empty local empty structure, so I wonder if that wasn't d/l counter, but even then it's awfully slow). So, where do we go from here? Unless there's tip export feature directly out of remote repo for bzr (bzr export seem to requite local working copy), we'd need to play mirror, this time bzr, again %). Make an initial shared repository, take an initial branch, tar it up, store it somewhere in the same zone, and use that to seed the checkout. That should cut the checkout to minutes and ~10 MB. Yes, I was thinking about some adhoc mirroring using cron, but this seems like even better idea, thanks. See: https://wiki.linaro.org/WorkingGroups/ToolChain/BzrTips -- Michael -- Best Regards, Paul ___ linaro-dev mailing list linaro-dev@lists.linaro.org http://lists.linaro.org/mailman/listinfo/linaro-dev
Re: Draft daily (hmm) cloud builds of Android toolchain from bzr
On Wed, Jun 15, 2011 at 12:16 AM, Paul Sokolovsky paul.sokolov...@linaro.org wrote: Hello Michael, On Tue, 14 Jun 2011 08:41:21 +1200 Michael Hope michael.h...@linaro.org wrote: [] Note that it now expects bzr branch with explicit version, e.g. lp:gcc-linaro/4.5 or lp:gcc-linaro/4.6 . Minor nit: these are called 'series'. I understand that in Launchpad's terms it's series, but there's doesn't seem to be such notion on bzr level, so how they're represented in bzr - are they 2 separate repositories, or 2 branches of the same repo? Interesting question. Here's my understanding: * They are both branches * The could be stored as different repos * Both are stacked on top of another branch to reduce the download and storage cost * Both are originally forked from different places of a different branch How this ties in with the repo implementation I'm not sure. I think the technical history of the gcc-linaro branch is: * lp:gcc is a automatic import from GCC SVN * The lp:gcc-linaro stacked branch was created from the lp:gcc/4.5 branch * The lp:gcc-linaro/4.5 was branched off lp:gcc/4.5 and stacked on top of gcc-linaro * The lp:gcc-linaro/4.6 was branched off lp:gcc/4.6 and stacked on top of the 4.5 derrived lp:gcc-linaro -- Michael ___ linaro-dev mailing list linaro-dev@lists.linaro.org http://lists.linaro.org/mailman/listinfo/linaro-dev
Draft daily (hmm) cloud builds of Android toolchain from bzr
Hello, Alexander, I've set up builds of Android toolchain from bzr: https://android-build.linaro.org/builds/~pfalcon/linaro-toolchain-4.5-bzr/ https://android-build.linaro.org/builds/~pfalcon/linaro-toolchain-4.6-bzr/ Jim, I had to revamp bzr build in linaro-build.sh: http://git.linaro.org/gitweb?p=people/pfalcon/android/toolchain/build.git;a=shortlog;h=refs/heads/pfalcon Note that it now expects bzr branch with explicit version, e.g. lp:gcc-linaro/4.5 or lp:gcc-linaro/4.6 . Well, now the issue: the builds above are not quite ready to be daily. That's because bzr checkout is slow, and is also quite big. Normal bzr clone takes 1.1Gb and cloud build takes 1hr instead of 30min with a tarball. I tried to play with bzr checkout --lightweight, but ctrl+c'ed it when it showed that it dealt with 1.3Gb of stuff (surprisingly, after break, I saw almost empty local empty structure, so I wonder if that wasn't d/l counter, but even then it's awfully slow). So, where do we go from here? Unless there's tip export feature directly out of remote repo for bzr (bzr export seem to requite local working copy), we'd need to play mirror, this time bzr, again %). -- Best Regards, Paul ___ linaro-dev mailing list linaro-dev@lists.linaro.org http://lists.linaro.org/mailman/listinfo/linaro-dev
Re: Draft daily (hmm) cloud builds of Android toolchain from bzr
On Mon, 13 Jun 2011 15:41:33 +0300, Paul Sokolovsky paul.sokolov...@linaro.org wrote: So, where do we go from here? Unless there's tip export feature directly out of remote repo for bzr (bzr export seem to requite local working copy), we'd need to play mirror, this time bzr, again %). You can export remotely, but it probably has the same performance as checkout --lightweight The mirror service may be the best way to deal with this. Thanks, James ___ linaro-dev mailing list linaro-dev@lists.linaro.org http://lists.linaro.org/mailman/listinfo/linaro-dev
Re: Draft daily (hmm) cloud builds of Android toolchain from bzr
On Tue, Jun 14, 2011 at 12:41 AM, Paul Sokolovsky paul.sokolov...@linaro.org wrote: Hello, Alexander, I've set up builds of Android toolchain from bzr: https://android-build.linaro.org/builds/~pfalcon/linaro-toolchain-4.5-bzr/ https://android-build.linaro.org/builds/~pfalcon/linaro-toolchain-4.6-bzr/ Jim, I had to revamp bzr build in linaro-build.sh: http://git.linaro.org/gitweb?p=people/pfalcon/android/toolchain/build.git;a=shortlog;h=refs/heads/pfalcon Note that it now expects bzr branch with explicit version, e.g. lp:gcc-linaro/4.5 or lp:gcc-linaro/4.6 . Minor nit: these are called 'series'. Well, now the issue: the builds above are not quite ready to be daily. That's because bzr checkout is slow, and is also quite big. Normal bzr clone takes 1.1Gb and cloud build takes 1hr instead of 30min with a tarball. I tried to play with bzr checkout --lightweight, but ctrl+c'ed it when it showed that it dealt with 1.3Gb of stuff (surprisingly, after break, I saw almost empty local empty structure, so I wonder if that wasn't d/l counter, but even then it's awfully slow). So, where do we go from here? Unless there's tip export feature directly out of remote repo for bzr (bzr export seem to requite local working copy), we'd need to play mirror, this time bzr, again %). Make an initial shared repository, take an initial branch, tar it up, store it somewhere in the same zone, and use that to seed the checkout. That should cut the checkout to minutes and ~10 MB. See: https://wiki.linaro.org/WorkingGroups/ToolChain/BzrTips -- Michael ___ linaro-dev mailing list linaro-dev@lists.linaro.org http://lists.linaro.org/mailman/listinfo/linaro-dev
