Re: Moz2D Repository Creation
> And, Git subtree is essentially the strategy m-c uses today: one large > repository holding the history (albeit sometimes squashed landings) of > many external ones. however it has the possible advantage that the VCS has some idea what is happening, and you have the history for the external project. > If we switched to Mercurial subrepos and wished for Git to flatten these > subrepos into 1 large one, this is a feature hg-git would need to grow. > It's doable, sure. However, going from squashed Git back to Mercurial > subrepos could be problematic to say the least. I'm not sure it could be > done reliably. This may kill our idea to have a *fully* writable Git > server someday (we may need to prevent writes to directories that are > Mercurial subrepos). http://khuey.pastebin.mozilla.org/2253745 seems to demonstrate that at least in theory it is possible to extract a history for the sub project and push just that history somewhere, but the integration foo seems non trivial. > At the point you are squashing repos into 1 to convenience Git, you > might as well do the same to Mercurial to facilitate easier conversion. having useful blaim in the main repo may or not be a significant advantage Trev > That leaves us pretty much with where we are today. > ___ > dev-platform mailing list > dev-platform@lists.mozilla.org > https://lists.mozilla.org/listinfo/dev-platform ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Moz2D Repository Creation
On 3/28/2013 8:28 AM, Milan Sreckovic wrote: > Anyone considering git submodules should definitely take a look at git > subtrees and see if that solution is more appropriate for the situation in > question. > > In our case, it may be that we want one approach for something like Moz2D, > and another one for something like skis, cairo, etc. > For those not aware of git subtrees, it's essentially the merging of a foreign repository into a directory (subtree) of a larger repository. There are no externals/submodules. The foreign repository is integrated into a larger repository, history and all (unless you squash the history). It essentially provides a simple-to-use way to replay commits from a foreign repository onto your local repository. It even allows you to extract your changes in the larger repository back into the independent one! And, Git subtree is essentially the strategy m-c uses today: one large repository holding the history (albeit sometimes squashed landings) of many external ones. If we switched to Mercurial subrepos and wished for Git to flatten these subrepos into 1 large one, this is a feature hg-git would need to grow. It's doable, sure. However, going from squashed Git back to Mercurial subrepos could be problematic to say the least. I'm not sure it could be done reliably. This may kill our idea to have a *fully* writable Git server someday (we may need to prevent writes to directories that are Mercurial subrepos). At the point you are squashing repos into 1 to convenience Git, you might as well do the same to Mercurial to facilitate easier conversion. That leaves us pretty much with where we are today. ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Moz2D Repository Creation
Anyone considering git submodules should definitely take a look at git subtrees and see if that solution is more appropriate for the situation in question. In our case, it may be that we want one approach for something like Moz2D, and another one for something like skis, cairo, etc. Milan On 2013-03-27, at 7:08 PM, George Wright wrote: > On 03/27/2013 07:37 PM, Robert O'Callahan wrote: >> I predict that eventually we'll want to switch mozilla-central to git. (I'm >> not in favour of it, but hg is not the DVCS of the future.) So, git users >> not liking git's subrepositories gives me pause and I think it's imperative >> to consider the situation of git users now and after a hypothetical >> mozilla-central switch to git. > > I've investigated submodules before to try to solve this exact problem > and came away pretty unimpressed. This was nearly 3 or 4 years ago, but > I doubt things have changed much. > > A quick search suggests that other people are also similarly unimpressed > with submodules, such as the post at > http://codingkilledthecat.wordpress.com/2012/04/28/why-your-company-shouldnt-use-git-submodules/ > > I suspect the answer here would be to choose a solution that is easily > convertible to a repo-style setup. We already have experience in-house > with repo due to Android and B2G and it seems to mostly do what we would > be aiming for with this project. > > On the main note (which I already brought up on IRC): I would be very > interested in seeing whether this experiment succeeds or not, as I'd > like to get to the stage where the entire Skia source tree inside > mozilla-central is just a checkout of upstream Skia, whether that's via > submodules or repo or whatever. > > George > ___ > dev-platform mailing list > dev-platform@lists.mozilla.org > https://lists.mozilla.org/listinfo/dev-platform ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Moz2D Repository Creation
On 3/27/2013 7:37 PM, Robert O'Callahan wrote: Adding a subrepository just for Moz2D seems like it could add global complexity for the benefit of just a small set of people. So it seems much more worthwhile if we regard it as an experiment to blaze a trail other modules would follow if the experiment succeeds, or be backed out if the experiment fails (costs are found to exceed benefits). I'm still skeptical the experiment will succeed but I'd support trying it. We definitely have this problem in multiple places already. We include many external projects already, including NSPR, NSS, zlib, libbz2, freetype2, libevent, pymake, skia, and I'm pretty sure there are quite a few others. Some of these we import unmodified from tags; others we modify, and if we're good about it we check in the patches against named tags or revisions to make future updating easy. In at least some of these cases, we use the root client.py script to actually perform the update, e.g. `python client.py update_nspr TAGNAME`. Although reading client.py, I see that we must not be using it any more, since it still tries to pull from CVS, and NSPR has moved to mercurial! If submodules/subrepos were proved to be a workable solution, we would definitely consider moving some or all of these imports to use them. Given the discussion I've seen so far, I think that mercurial subrepos are probably pretty satisfactory. But I think it's essential that we have a pretty good solution for git as well, and git subrepos are not nearly as good. I have personally experienced the confusion about having to run `git submodule update` at non-obvious times when hacking shumway. I expect that we'd be introducing additional developer confusion. I haven't experienced significant problems with the current import system, and I propose that we try that for moz2d as well. It isn't currently designed to be run daily, but I don't see any particular reason it couldn't scale to daily imports, driven either manually by the moz2d developers or automatically by some continuous integration system. Are any of the original goal/problem statements left unsolved by this approach? --BDS ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Moz2D Repository Creation
On 03/27/13 06:06 PM, Justin Lebar wrote: >> hg-git (the tool we use to synchronize Mercurial and Git repos) supports >> subrepos. Although, I'm not sure how well it works. > > Well, we should definitely figure this out before we move forward with > this plan. Hal Wine (hw...@mozilla.com) should be included in this. ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Moz2D Repository Creation
On 03/27/13 06:38 PM, Joshua Cranmer 🐧 wrote: > On 3/27/2013 5:25 PM, Bas Schouten wrote: >> I would argue this is probably true. As I've talked to a developer >> from a large third party where the discussion of if they would try to >> adjust their work to Moz2D literally ended at: 'I have to check out >> -all- of firefox? How big is that 2 Gig or something?' I think it's >> certainly the largest practical and mental barrier. > > How hard would it be to add support in mercurial for checking out a > specific directory within a repository? http://mercurial.selenic.com/wiki/ShallowClone ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Moz2D Repository Creation
Fwiw I decided to take a more detailed look at the quoted article and there's a couple of things I noticed: 'Don’t blink…' The answer to this point is simple. The reason we'd like this is we don't expect people pulling in new versions of the subrepo regularly. We expect that to happen on a regular basis in an automated way, and only occasionally by developers. Mercurial subrepositories also solve the latter part of the problems mentioned here by making an hg up update both your parent and subrepository, preventing the 'accidental' reverts mentioned in the article. 'Merging? Ha!' Again, we don't expect to be 'merging' changes to the subrepository changeset we're linked to. In the sense that unless well coordinated (in which case you know what you're doing and it seems the concerns in the article would not really apply) you're not really merging very often here. You might have branches on the subrepository in some cases where you're fixing an aurora bug for example, but I don't foresee serious problems there. That's only really a small amount of extra work for a graphics developer that I think is perfectly acceptable. I'd say we've had a lot more trouble in the past merging libraries we manually pulled into our tree. 'Oh, were you using that?' Although I find this bit of concern is valid, I think it's a problem that really only occurs when there's a lot of interdependencies where you're editing both the parent and subrepo at once. It seems to be this problem would not occur if you landed changes to your subrepository always to the tip of that subrepository, and not push them directly from your parent repository. For example if you were working on a new feature, even if you were doing so while working with all of mozilla central, you'd update the subrepo, develop your new code, push that to the subrepository, but not necessarily push the updated tag to mozilla-central. This workflow is a little different, but I don't think it's problematic. Finally there's an interesting one: 'What am I supposed to do with this?' 'Push changes from the submodule and not the parent repository? No one knows to use your new submodule changes.' Well, this sounds a lot like just using submodules in a situation where there isn't really a distinct project that you're using the submodule for, when you push changes to a submodule it's not that 'noone knows to use it'. It means new features/bugfixes have become available in your dependency project. This is fine, they'll be picked up in time on an automated update, or when they're required earlier. 'Push changes from the parent repository and not the submodule? Congratulations, no one can use your new commits because they don’t have the right submodule commit available to check out.' Mercurial actually doesn't seem to allow this as far as I've been tell (i.e. your parent push will also ensure your subrepos are pushed). However it is again an indication that there's not actually a distinct development process going on here. The default process here should be to add a new feature, tests, etc. Push them to the subrepo, ensure all tests are passing everywhere and that everything is ready, and then start using it in the parent when it becomes available. In the rare cases of API changes which are incompatible, a little more care is required, but when making changes like that I'd say it's actually -good- to be aware of that, especially when we're in a situation where multiple projects (i.e. Gecko and Servo) use the API. And if you're consciously making such a change making sure you push your subrepo change first and then update your parent repository with the API changes and the tag update seems like a very easy thing. All in all having read, and thought for a while, about this particular post of concerns has not really made me feel like it's something we don't want to do. Although there seems to be some minor things which are a little better handled with mercurial subrepositories. I wonder if it would be possible to make sure git's improved in that area before we switch mozilla-central to it, if the experiment goes through and succeeds. Bas - Original Message - From: "George Wright" To: dev-platform@lists.mozilla.org Sent: Thursday, March 28, 2013 2:08:45 AM Subject: Re: Moz2D Repository Creation On 03/27/2013 07:37 PM, Robert O'Callahan wrote: > I predict that eventually we'll want to switch mozilla-central to git. (I'm > not in favour of it, but hg is not the DVCS of the future.) So, git users > not liking git's subrepositories gives me pause and I think it's imperative > to consider the situation of git
Re: Moz2D Repository Creation
On 03/27/2013 07:37 PM, Robert O'Callahan wrote: > I predict that eventually we'll want to switch mozilla-central to git. (I'm > not in favour of it, but hg is not the DVCS of the future.) So, git users > not liking git's subrepositories gives me pause and I think it's imperative > to consider the situation of git users now and after a hypothetical > mozilla-central switch to git. I've investigated submodules before to try to solve this exact problem and came away pretty unimpressed. This was nearly 3 or 4 years ago, but I doubt things have changed much. A quick search suggests that other people are also similarly unimpressed with submodules, such as the post at http://codingkilledthecat.wordpress.com/2012/04/28/why-your-company-shouldnt-use-git-submodules/ I suspect the answer here would be to choose a solution that is easily convertible to a repo-style setup. We already have experience in-house with repo due to Android and B2G and it seems to mostly do what we would be aiming for with this project. On the main note (which I already brought up on IRC): I would be very interested in seeing whether this experiment succeeds or not, as I'd like to get to the stage where the entire Skia source tree inside mozilla-central is just a checkout of upstream Skia, whether that's via submodules or repo or whatever. George ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Moz2D Repository Creation
On 13-03-27 4:00 PM, Robert O'Callahan wrote: > And you probably won't do it at a conference. If I heard about Moz2D at a conference, I'd certainly want to clone the repo and have a quick look at it. This is quite relevant to the goal of reducing contributor barriers. However, I agree with your larger point: this isn't a great trade off compared to some of the other issues. Import scripts (automating 'cp' from an upstream checkout) have worked well for the media libraries I've helped maintain, and unfortunately remain the best option for incorporating external repos in our monolithic build. But those tend to have very stable interfaces, which probably isn't the case for Moz2D development. -r ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Moz2D Repository Creation
Summary of my thoughts after discussing on IRC: Adding a subrepository just for Moz2D seems like it could add global complexity for the benefit of just a small set of people. So it seems much more worthwhile if we regard it as an experiment to blaze a trail other modules would follow if the experiment succeeds, or be backed out if the experiment fails (costs are found to exceed benefits). I'm still skeptical the experiment will succeed but I'd support trying it. This means that the setup we use for Moz2D should scale to a larger number of subrepositories. In particular solutions for automatic regression finding, updating Aurora/Beta/Release branches, etc, need to be able to scale, i.e. not add new manual steps per subrepository. I predict that eventually we'll want to switch mozilla-central to git. (I'm not in favour of it, but hg is not the DVCS of the future.) So, git users not liking git's subrepositories gives me pause and I think it's imperative to consider the situation of git users now and after a hypothetical mozilla-central switch to git. Rob -- Wrfhf pnyyrq gurz gbtrgure naq fnvq, “Lbh xabj gung gur ehyref bs gur Tragvyrf ybeq vg bire gurz, naq gurve uvtu bssvpvnyf rkrepvfr nhgubevgl bire gurz. Abg fb jvgu lbh. Vafgrnq, jubrire jnagf gb orpbzr terng nzbat lbh zhfg or lbhe freinag, naq jubrire jnagf gb or svefg zhfg or lbhe fynir — whfg nf gur Fba bs Zna qvq abg pbzr gb or freirq, ohg gb freir, naq gb tvir uvf yvsr nf n enafbz sbe znal.” [Znggurj 20:25-28] ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Moz2D Repository Creation
Thanks a lot for thinking along here from the RelEng side of things! - Original Message - > From: "Alex Keybl" > To: "Bas Schouten" > Cc: "dev-platform" > Sent: Wednesday, March 27, 2013 11:25:14 PM > Subject: Re: Moz2D Repository Creation > > > Can you clarify the main motivation behind using a subrepo over a > Tier 1 dev branch like m-i or services-central? Mimicking what we > already do elsewhere would have less process/infra change overhead, > and my intuition tells me that per-checkin builds/tests could be > configured specifically for that repo. Perhaps we should be focusing > mostly on your point about " Improv[ing] sharing between > Servo/Gecko" though. The main motivations would be: - Reduced tree size (for clone/pull/update/etc.) - Easier bisection within the specific Moz2D code. - Cleaner merging system (i.e. rather than merging, it would just be a revision tag update in m-c) > > > Assuming we move forward with a subrepo, a few thoughts: > > > * Whatever the branching model becomes, it sounds like it'll end up > being around merges. Just let us know what you'd like us to do at > m-c->m-a and upon subsequent merges. No problem there. Nothing should change for that, m-c -> m-a would just take the currently tagged revision along with it, and that would remain the revision used. For upstreaming patches the process gets a tiny bit different, see also your point below this one. I've added some of my own thoughts to the wiki page. > * We'll need to decide how to handle approvals - whether to gate at > landing to a branched subrepo, or updating the changeset in the main > repo. Just requires some discussion. > * Somebody (engineer, sheriff) would need to make sure that after > landing to a subrepo branch, a version of Firefox is marked as fixed > once it the changeset starts being referenced from the main repo. Yeah, this is an interesting point. The question is if it's defined fixed by being fixed on Moz2D or in Firefox, probably the latter is best. > > > As for impact to QA and investigations, I agree that the change > should be fairly minimal. Since we'd still like the help of > contributors and QA in bisecting a day's worth of Moz2D changes, new > documentation may be necessary. Indeed, a day (or a week) worth of Moz2D changes will be a lot easier to bisect though! :-) And could even be done within m-c where you'd for example only have to rebuild gfx/2d and layout/media, as long as your bisection doesn't include API changes. This would be one of the advantages of the separation listed above. > > > > -Alex Bas ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Moz2D Repository Creation
Can you clarify the main motivation behind using a subrepo over a Tier 1 dev branch like m-i or services-central? Mimicking what we already do elsewhere would have less process/infra change overhead, and my intuition tells me that per-checkin builds/tests could be configured specifically for that repo. Perhaps we should be focusing mostly on your point about "Improv[ing] sharing between Servo/Gecko" though. Assuming we move forward with a subrepo, a few thoughts: * Whatever the branching model becomes, it sounds like it'll end up being around merges. Just let us know what you'd like us to do at m-c->m-a and upon subsequent merges. No problem there. * We'll need to decide how to handle approvals - whether to gate at landing to a branched subrepo, or updating the changeset in the main repo. Just requires some discussion. * Somebody (engineer, sheriff) would need to make sure that after landing to a subrepo branch, a version of Firefox is marked as fixed once it the changeset starts being referenced from the main repo. As for impact to QA and investigations, I agree that the change should be fairly minimal. Since we'd still like the help of contributors and QA in bisecting a day's worth of Moz2D changes, new documentation may be necessary. -Alex On Mar 27, 2013, at 1:42 PM, Bas Schouten wrote: > Hi all, > > Over the past year we've increased our dependencies on Moz2D (formerly known > under the codename Azure), our new 2D rendering API. Currently we're using it > for canvas drawing on all platforms and content drawing on Windows where > using Direct2D, and in the near future we will be moving to using it for > content on all platforms. > > From the very beginning of Moz2D development we've taken care to make sure it > can be built outside of the rest of Gecko, having only dependencies on > several MFBT headers. This was done to reduce the barrier for external usage > and development, we've since seen this benefit us for example in Servo using > Moz2D as well. In addition to that it has helped development and bugfixing by > having a much faster workflow for building/testing with which bugs could be > located and fixes created. > > Going forward we're expanding on that by ensuring the stand-alone builds and > works on all platforms and becomes the defacto way of the 'first level' of > testing when implementing new features or fixing bugs in Moz2D. In addition > to that we're expanding on our existing stand-alone unittesting suite as well > as adding a peformance testing suite, which will include microbenchmarking > and support measuring performance on Moz2D drawing recordings (single-page > recording support has just landed on mozilla-inbound). > > When moving forward we have several goals for Moz2D when it comes to the > workflow: > > - Improve Moz2D development workflow by having faster turnaround time on > builds and tests (both local and Try) > - Lower the barrier for external contributors, some people have already > expressed the desire to work on potential backends we do not wish to invest > in ourselves, but have been deterred by the complexity of the > checkout/building process. > - Improve sharing between Servo/Gecko. > - Reduce load on our infrastructure by building and testing only Moz2D > stand-alone when a change affects only Moz2D, both on regular builds and try. > > As the next step in moving towards these goals and optimally supporting them > the proposal is to move Moz2D into its own repository. We would use hg > subrepos for this purpose, you can read more about this here > (http://mercurial.selenic.com/wiki/Subrepository). The intention is that this > will be done in such a way that the workflow for developers not involved in > Moz2D development would essentially not change. > > A more detailed description of the proposal and the workflows involved can be > found here (https://wiki.mozilla.org/Platform/GFX/Moz2DSubrepository) for > those interested in the details. Since we believe if we go through with this > it would be the first time we use a true subrepository system for a component > used in mozilla-central, we'd very much appreciate any thoughts or feedback > people might have on the idea. > > Best regards, > Bas > ___ > dev-platform mailing list > dev-platform@lists.mozilla.org > https://lists.mozilla.org/listinfo/dev-platform ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Moz2D Repository Creation
On Thu, Mar 28, 2013 at 12:13 PM, Bas Schouten wrote: > This is true, and it even already more or less works (although the > Makefiles aren't in the tree, the VS2010 files are, although the lack of > automated testing means I've not gone through the trouble of putting all > the newer tests to m-c) but it would be complicated to integrate this into > our infrastructure, in such a way that a try push to Moz2D code only runs > the Moz2D compile and Moz2D tests. > I think it's worth considering how hard this would be. Integrating new repos and tests into our infrastructure already seems complicated. Rob -- Wrfhf pnyyrq gurz gbtrgure naq fnvq, “Lbh xabj gung gur ehyref bs gur Tragvyrf ybeq vg bire gurz, naq gurve uvtu bssvpvnyf rkrepvfr nhgubevgl bire gurz. Abg fb jvgu lbh. Vafgrnq, jubrire jnagf gb orpbzr terng nzbat lbh zhfg or lbhe freinag, naq jubrire jnagf gb or svefg zhfg or lbhe fynir — whfg nf gur Fba bs Zna qvq abg pbzr gb or freirq, ohg gb freir, naq gb tvir uvf yvsr nf n enafbz sbe znal.” [Znggurj 20:25-28] ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Moz2D Repository Creation
- Original Message - > From: "Robert O'Callahan" > To: "Bas Schouten" > Cc: "dev-platform" > Sent: Wednesday, March 27, 2013 11:00:01 PM > Subject: Re: Moz2D Repository Creation > > On Thu, Mar 28, 2013 at 11:45 AM, Bas Schouten < > bschou...@mozilla.com > wrote: > > > > I don't think it works that way. At least it doesn't for me, these > time issues don't work that way. There's a context switch involved, > those are expensive and there's the feeling of pulling in 2.5 gigs+ > onto your hard drive to do something small, for someone else really, > not because you get paid for it. Especially when you're for example > on a conference with a poor network connection. As another example, > currently pushing to try is a very conscious decision for me. Is > this test result worth it if it's going to take me 4 hours to get > the results on all platforms? I'll have to look at it again > tomorrow, maybe I should look a little more if I've done everything > to make sure it works. Because I know that by pushing that try run > I'm going to put a lot of burden on our infrastructure to see if a > small change compiles on all platforms. > > > But pushing to try is something you do a lot. Cloning Moz2D is > something you probably only do once. And you probably won't do it at > a conference. This would is true for me, for mozilla-central and Moz2D. It's a lot less true I believe for projects you work on sporadically. If I look at myself, some of the other things I occasionally play with that I don't use in my day-to-day job I work on them quite infrequently. Which means I frequently clear them out, or have to pull long periods of updates, here the cost of a large repository becomes a lot larger again. > > Additionally all -other- operations on a larger repository are a lot > slower, things like top-level diffs, updates, etc. They take seconds > or less on small repository, every step of your workflow is more > efficient in a more isolated environment. Bisecting is only a small > example of this, bisecting inside a small repository with a fast > build/test is a process that's done very quickly and is very easy to > justify timewise. Bisecting mozilla-central is a hard task that > takes a lot of time. > > Mainly because you have to build all of mozilla-central. Standalone > builds and tests of Moz2D would be great and we can do that without > putting it in its own repository. This is true, and it even already more or less works (although the Makefiles aren't in the tree, the VS2010 files are, although the lack of automated testing means I've not gone through the trouble of putting all the newer tests to m-c) but it would be complicated to integrate this into our infrastructure, in such a way that a try push to Moz2D code only runs the Moz2D compile and Moz2D tests. Bas ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Moz2D Repository Creation
On 3/27/13 11:38 PM, Joshua Cranmer 🐧 wrote: > On 3/27/2013 5:25 PM, Bas Schouten wrote: >> I would argue this is probably true. As I've talked to a developer >> from a large third party where the discussion of if they would try to >> adjust their work to Moz2D literally ended at: 'I have to check out >> -all- of firefox? How big is that 2 Gig or something?' I think it's >> certainly the largest practical and mental barrier. > > How hard would it be to add support in mercurial for checking out a > specific directory within a repository? > I personally prefer this solution. comm-central does it successfully with its client.py and it hasn't been annoying for me. Philipp ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Moz2D Repository Creation
On Thu, Mar 28, 2013 at 11:45 AM, Bas Schouten wrote: > I don't think it works that way. At least it doesn't for me, these time > issues don't work that way. There's a context switch involved, those are > expensive and there's the feeling of pulling in 2.5 gigs+ onto your hard > drive to do something small, for someone else really, not because you get > paid for it. Especially when you're for example on a conference with a poor > network connection. As another example, currently pushing to try is a very > conscious decision for me. Is this test result worth it if it's going to > take me 4 hours to get the results on all platforms? I'll have to look at > it again tomorrow, maybe I should look a little more if I've done > everything to make sure it works. Because I know that by pushing that try > run I'm going to put a lot of burden on our infrastructure to see if a > small change compiles on all platforms. > But pushing to try is something you do a lot. Cloning Moz2D is something you probably only do once. And you probably won't do it at a conference. Additionally all -other- operations on a larger repository are a lot > slower, things like top-level diffs, updates, etc. They take seconds or > less on small repository, every step of your workflow is more efficient in > a more isolated environment. Bisecting is only a small example of this, > bisecting inside a small repository with a fast build/test is a process > that's done very quickly and is very easy to justify timewise. Bisecting > mozilla-central is a hard task that takes a lot of time. > Mainly because you have to build all of mozilla-central. Standalone builds and tests of Moz2D would be great and we can do that without putting it in its own repository. Rob -- Wrfhf pnyyrq gurz gbtrgure naq fnvq, “Lbh xabj gung gur ehyref bs gur Tragvyrf ybeq vg bire gurz, naq gurve uvtu bssvpvnyf rkrepvfr nhgubevgl bire gurz. Abg fb jvgu lbh. Vafgrnq, jubrire jnagf gb orpbzr terng nzbat lbh zhfg or lbhe freinag, naq jubrire jnagf gb or svefg zhfg or lbhe fynir — whfg nf gur Fba bs Zna qvq abg pbzr gb or freirq, ohg gb freir, naq gb tvir uvf yvsr nf n enafbz sbe znal.” [Znggurj 20:25-28] ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Moz2D Repository Creation
On Wed 27 Mar 2013 03:25:20 PM PDT, Bas Schouten wrote: > I would argue this is probably true. As I've talked to a developer from a > large third party where the discussion of if they would try to adjust their > work to Moz2D literally ended at: 'I have to check out -all- of firefox? How > big is that 2 Gig or something?' I think it's certainly the largest practical > and mental barrier. You only have to require checking out all of Firefox for those who want to modify Moz2D and submit patches back. That's how the JS engine works. It's a regular part of the full mozilla-central tree, but we produce periodic "releases" that are basically tarballs of the js/src subtree. (It's not really nearly that simple, and our releases are very irregularly spaced in time, but let's play pretend for now.) Embedders who just want to use Spidermonkey can download the release tarball, which has its own configure script and everything. The Spidermonkey-only build procedure feels "normal"; you don't have to know about the Mozilla build system or anything. (gps has thankfully preserved this characteristic with recent build changes.) I don't know if this changes the fundamental equation or not. ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Moz2D Repository Creation
I don't think it works that way. At least it doesn't for me, these time issues don't work that way. There's a context switch involved, those are expensive and there's the feeling of pulling in 2.5 gigs+ onto your hard drive to do something small, for someone else really, not because you get paid for it. Especially when you're for example on a conference with a poor network connection. As another example, currently pushing to try is a very conscious decision for me. Is this test result worth it if it's going to take me 4 hours to get the results on all platforms? I'll have to look at it again tomorrow, maybe I should look a little more if I've done everything to make sure it works. Because I know that by pushing that try run I'm going to put a lot of burden on our infrastructure to see if a small change compiles on all platforms. Additionally all -other- operations on a larger repository are a lot slower, things like top-level diffs, updates, etc. They take seconds or less on small repository, every step of your workflow is more efficient in a more isolated environment. Bisecting is only a small example of this, bisecting inside a small repository with a fast build/test is a process that's done very quickly and is very easy to justify timewise. Bisecting mozilla-central is a hard task that takes a lot of time. Bas - Original Message - From: "Robert O'Callahan" To: "Bas Schouten" Cc: "dev-platform" Sent: Wednesday, March 27, 2013 10:36:34 PM Subject: Re: Moz2D Repository Creation On Thu, Mar 28, 2013 at 11:25 AM, Bas Schouten < bschou...@mozilla.com > wrote: I would argue this is probably true. As I've talked to a developer from a large third party where the discussion of if they would try to adjust their work to Moz2D literally ended at: 'I have to check out -all- of firefox? How big is that 2 Gig or something?' I think it's certainly the largest practical and mental barrier. I find that difficult to understand. Mostly that just means you go to lunch while your "hg clone" runs. Maybe it's a mental thing but I don't think that should be a driver of our strategy hear. Rob -- Wrfhf pnyyrq gurz gbtrgure naq fnvq, “Lbh xabj gung gur ehyref bs gur Tragvyrf ybeq vg bire gurz, naq gurve uvtu bssvpvnyf rkrepvfr nhgubevgl bire gurz. Abg fb jvgu lbh. Vafgrnq, jubrire jnagf gb orpbzr terng nzbat lbh zhfg or lbhe freinag, naq jubrire jnagf gb or svefg zhfg or lbhe fynir — whfg nf gur Fba bs Zna qvq abg pbzr gb or freirq, ohg gb freir, naq gb tvir uvf yvsr nf n enafbz sbe znal.” [Znggurj 20:25-28] ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Moz2D Repository Creation
On 3/27/2013 5:25 PM, Bas Schouten wrote: I would argue this is probably true. As I've talked to a developer from a large third party where the discussion of if they would try to adjust their work to Moz2D literally ended at: 'I have to check out -all- of firefox? How big is that 2 Gig or something?' I think it's certainly the largest practical and mental barrier. How hard would it be to add support in mercurial for checking out a specific directory within a repository? -- Joshua Cranmer Thunderbird and DXR developer Source code archæologist ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Moz2D Repository Creation
On Thu, Mar 28, 2013 at 11:25 AM, Bas Schouten wrote: > I would argue this is probably true. As I've talked to a developer from a > large third party where the discussion of if they would try to adjust their > work to Moz2D literally ended at: 'I have to check out -all- of firefox? > How big is that 2 Gig or something?' I think it's certainly the largest > practical and mental barrier. > I find that difficult to understand. Mostly that just means you go to lunch while your "hg clone" runs. Maybe it's a mental thing but I don't think that should be a driver of our strategy hear. Rob -- Wrfhf pnyyrq gurz gbtrgure naq fnvq, “Lbh xabj gung gur ehyref bs gur Tragvyrf ybeq vg bire gurz, naq gurve uvtu bssvpvnyf rkrepvfr nhgubevgl bire gurz. Abg fb jvgu lbh. Vafgrnq, jubrire jnagf gb orpbzr terng nzbat lbh zhfg or lbhe freinag, naq jubrire jnagf gb or svefg zhfg or lbhe fynir — whfg nf gur Fba bs Zna qvq abg pbzr gb or freirq, ohg gb freir, naq gb tvir uvf yvsr nf n enafbz sbe znal.” [Znggurj 20:25-28] ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Moz2D Repository Creation
I would argue this is probably true. As I've talked to a developer from a large third party where the discussion of if they would try to adjust their work to Moz2D literally ended at: 'I have to check out -all- of firefox? How big is that 2 Gig or something?' I think it's certainly the largest practical and mental barrier. Best regards, Bas - Original Message - From: "Robert O'Callahan" To: "Bas Schouten" Cc: "dev-platform" Sent: Wednesday, March 27, 2013 10:14:35 PM Subject: Re: Moz2D Repository Creation On Thu, Mar 28, 2013 at 9:42 AM, Bas Schouten < bschou...@mozilla.com > wrote: - Improve Moz2D development workflow by having faster turnaround time on builds and tests (both local and Try) - Lower the barrier for external contributors, some people have already expressed the desire to work on potential backends we do not wish to invest in ourselves, but have been deterred by the complexity of the checkout/building process. - Improve sharing between Servo/Gecko. - Reduce load on our infrastructure by building and testing only Moz2D stand-alone when a change affects only Moz2D, both on regular builds and try. Seems to me that of those goals, only "lower the barrier for external contributors" actually *requires* Moz2D to be in a separate repository. Is that right? Rob -- Wrfhf pnyyrq gurz gbtrgure naq fnvq, “Lbh xabj gung gur ehyref bs gur Tragvyrf ybeq vg bire gurz, naq gurve uvtu bssvpvnyf rkrepvfr nhgubevgl bire gurz. Abg fb jvgu lbh. Vafgrnq, jubrire jnagf gb orpbzr terng nzbat lbh zhfg or lbhe freinag, naq jubrire jnagf gb or svefg zhfg or lbhe fynir — whfg nf gur Fba bs Zna qvq abg pbzr gb or freirq, ohg gb freir, naq gb tvir uvf yvsr nf n enafbz sbe znal.” [Znggurj 20:25-28] ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Moz2D Repository Creation
We also want the ability to make some returns and run the tests without them ever showing up in m-c. So to speak. Especially when it comes to having to back things out. That could perhaps be accomplished with a branch, but you start getting into the situation where others could be updating the code directly in m-c and all that goes with it. But, it does introduce a, perhaps uncomfortable, feeling of "one of these things is not like the other, and begs the "why don't we do this with other pieces as well". Milan On 2013-03-27, at 3:14 PM, "Robert O'Callahan" wrote: > On Thu, Mar 28, 2013 at 9:42 AM, Bas Schouten wrote: > >> - Improve Moz2D development workflow by having faster turnaround time on >> builds and tests (both local and Try) >> - Lower the barrier for external contributors, some people have already >> expressed the desire to work on potential backends we do not wish to invest >> in ourselves, but have been deterred by the complexity of the >> checkout/building process. >> - Improve sharing between Servo/Gecko. >> - Reduce load on our infrastructure by building and testing only Moz2D >> stand-alone when a change affects only Moz2D, both on regular builds and >> try. >> > > Seems to me that of those goals, only "lower the barrier for external > contributors" actually *requires* Moz2D to be in a separate repository. Is > that right? > > Rob > -- > Wrfhf pnyyrq gurz gbtrgure naq fnvq, “Lbh xabj gung gur ehyref bs gur > Tragvyrf ybeq vg bire gurz, naq gurve uvtu bssvpvnyf rkrepvfr nhgubevgl > bire gurz. Abg fb jvgu lbh. Vafgrnq, jubrire jnagf gb orpbzr terng nzbat > lbh zhfg or lbhe freinag, naq jubrire jnagf gb or svefg zhfg or lbhe fynir > — whfg nf gur Fba bs Zna qvq abg pbzr gb or freirq, ohg gb freir, naq gb > tvir uvf yvsr nf n enafbz sbe znal.” [Znggurj 20:25-28] > ___ > dev-platform mailing list > dev-platform@lists.mozilla.org > https://lists.mozilla.org/listinfo/dev-platform ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Moz2D Repository Creation
Personally I don't care if we use Git or Mercurial, what I do feel fairly strongly about is that whatever we do for Moz2D runs on our own infrastructure. If that infrastructure supports git I have no issues with it being in git. Although I've done my testing with hg subrepos only. For what it's worth I personally find our mix of mercurial and git extremely disturbing, and this once again goes to show the amount of problems that come from such an architecture. I'd find it quite sad if the fact we're in this situation would block progress on this front. As to subrepository functionality, they seem to work well for our purposes, it's true that some complexities arise if you would like to do certain operations across repository boundaries but I didn't see a lot of cases where we'd actually want that. For what it's worth, I've worked with SVN externals before and found them quite nice to work with. But I must admit my experience with hg subrepos is limited to the testing I did. I've heard stories that mercurial subrepos work a lot better than git submodules, however I have even less experience with the latter. One argument for hg subrepos is that their type of functionality is pretty much -exactly- what we want/need for a great experience. This could be true for git submodules as well, but I have no idea if it is. Bas - Original Message - From: "Justin Lebar" To: "Gregory Szorc" Cc: "Bas Schouten" , "dev-platform" Sent: Wednesday, March 27, 2013 10:06:56 PM Subject: Re: Moz2D Repository Creation > hg-git (the tool we use to synchronize Mercurial and Git repos) supports > subrepos. Although, I'm not sure how well it works. Well, we should definitely figure this out before we move forward with this plan. If the hg support for git repos is decent, that might be a better way to go, since then we'd have one fewer repo we needed to mirror for B2G's purposes. Assuming that hg-git is happy with git submodules. :) > I haven't met many Git users who enjoy submodules, myself included. Git submodules suck, but so does importing library code via cp, as we do now. :-/ ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Moz2D Repository Creation
On Thu, Mar 28, 2013 at 9:42 AM, Bas Schouten wrote: > - Improve Moz2D development workflow by having faster turnaround time on > builds and tests (both local and Try) > - Lower the barrier for external contributors, some people have already > expressed the desire to work on potential backends we do not wish to invest > in ourselves, but have been deterred by the complexity of the > checkout/building process. > - Improve sharing between Servo/Gecko. > - Reduce load on our infrastructure by building and testing only Moz2D > stand-alone when a change affects only Moz2D, both on regular builds and > try. > Seems to me that of those goals, only "lower the barrier for external contributors" actually *requires* Moz2D to be in a separate repository. Is that right? Rob -- Wrfhf pnyyrq gurz gbtrgure naq fnvq, “Lbh xabj gung gur ehyref bs gur Tragvyrf ybeq vg bire gurz, naq gurve uvtu bssvpvnyf rkrepvfr nhgubevgl bire gurz. Abg fb jvgu lbh. Vafgrnq, jubrire jnagf gb orpbzr terng nzbat lbh zhfg or lbhe freinag, naq jubrire jnagf gb or svefg zhfg or lbhe fynir — whfg nf gur Fba bs Zna qvq abg pbzr gb or freirq, ohg gb freir, naq gb tvir uvf yvsr nf n enafbz sbe znal.” [Znggurj 20:25-28] ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Moz2D Repository Creation
> hg-git (the tool we use to synchronize Mercurial and Git repos) supports > subrepos. Although, I'm not sure how well it works. Well, we should definitely figure this out before we move forward with this plan. If the hg support for git repos is decent, that might be a better way to go, since then we'd have one fewer repo we needed to mirror for B2G's purposes. Assuming that hg-git is happy with git submodules. :) > I haven't met many Git users who enjoy submodules, myself included. Git submodules suck, but so does importing library code via cp, as we do now. :-/ ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Moz2D Repository Creation
On Wed, Mar 27, 2013 at 9:42 PM, Bas Schouten wrote: > As the next step in moving towards these goals and optimally supporting them > the proposal is to move Moz2D into its own repository. We would use hg > subrepos for this purpose, you can read more about this here > (http://mercurial.selenic.com/wiki/Subrepository). The intention is that this > will be done in such a way that the workflow for developers not involved in > Moz2D development would essentially not change. Did you take note of the warnings and recommendations in the wiki regarding subrepository usage? I used them for a while at work, some time ago, and it's... different. Theoretically it's all simple and nice, but there end up being corner case that don't work so well. And what with the git conversions that some people seem to prefer, that would all be even more complex. I'd recommend thinking this over a bit, and maybe just using the NSS model instead, or some method to pull in a fixed revision/tarball via a build system or similar feature. Cheers, Dirkjan ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Moz2D Repository Creation
On 3/27/13 2:03 PM, Justin Lebar wrote: Since we believe if we go through with this it would be the first time we use a true subrepository system for a component used in mozilla-central, we'd very much appreciate any thoughts or feedback people might have on the idea. Have you thought about how this hg subrepo will interact with the git mirrors of m-c? Using m-c git isn't just for contrarians these days: B2G relies heavily on git clones of m-c. So we need to make sure this will work properly. hg-git (the tool we use to synchronize Mercurial and Git repos) supports subrepos. Although, I'm not sure how well it works. And, the side-effect there is that you are forcing submodules on Git users. I haven't met many Git users who enjoy submodules, myself included. submodules are so bad that a whole class of wrapping tools (like repo [1]) have sprung up to plug the deficiencies. [1] https://code.google.com/p/git-repo/ ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Moz2D Repository Creation
On 28/03/13 10:03, Justin Lebar wrote: >> Since we believe if we go through with this it would be the first time we >> use a true subrepository system for a component >> used in mozilla-central, we'd very much appreciate any thoughts or feedback >> people might have on the idea. > > Have you thought about how this hg subrepo will interact with the git > mirrors of m-c? > > Using m-c git isn't just for contrarians these days: B2G relies > heavily on git clones of m-c. So we need to make sure this will work > properly. Why not just put Moz2D in a git repo? http://mercurial.selenic.com/wiki/Subrepository#Git_subrepositories ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Moz2D Repository Creation
On Wed, Mar 27, 2013 at 01:42:31PM -0700, Bas Schouten wrote: > A more detailed description of the proposal and the workflows involved > can be found here > (https://wiki.mozilla.org/Platform/GFX/Moz2DSubrepository) for those > interested in the details. Since we believe if we go through with this > it would be the first time we use a true subrepository system for a > component used in mozilla-central, we'd very much appreciate any > thoughts or feedback people might have on the idea. Relatedly, https://bugzilla.mozilla.org/show_bug.cgi?id=853749#c11 Mike ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Moz2D Repository Creation
> Since we believe if we go through with this it would be the first time we use > a true subrepository system for a component > used in mozilla-central, we'd very much appreciate any thoughts or feedback > people might have on the idea. Have you thought about how this hg subrepo will interact with the git mirrors of m-c? Using m-c git isn't just for contrarians these days: B2G relies heavily on git clones of m-c. So we need to make sure this will work properly. -Justin On Wed, Mar 27, 2013 at 4:42 PM, Bas Schouten wrote: > Hi all, > > Over the past year we've increased our dependencies on Moz2D (formerly known > under the codename Azure), our new 2D rendering API. Currently we're using it > for canvas drawing on all platforms and content drawing on Windows where > using Direct2D, and in the near future we will be moving to using it for > content on all platforms. > > From the very beginning of Moz2D development we've taken care to make sure it > can be built outside of the rest of Gecko, having only dependencies on > several MFBT headers. This was done to reduce the barrier for external usage > and development, we've since seen this benefit us for example in Servo using > Moz2D as well. In addition to that it has helped development and bugfixing by > having a much faster workflow for building/testing with which bugs could be > located and fixes created. > > Going forward we're expanding on that by ensuring the stand-alone builds and > works on all platforms and becomes the defacto way of the 'first level' of > testing when implementing new features or fixing bugs in Moz2D. In addition > to that we're expanding on our existing stand-alone unittesting suite as well > as adding a peformance testing suite, which will include microbenchmarking > and support measuring performance on Moz2D drawing recordings (single-page > recording support has just landed on mozilla-inbound). > > When moving forward we have several goals for Moz2D when it comes to the > workflow: > > - Improve Moz2D development workflow by having faster turnaround time on > builds and tests (both local and Try) > - Lower the barrier for external contributors, some people have already > expressed the desire to work on potential backends we do not wish to invest > in ourselves, but have been deterred by the complexity of the > checkout/building process. > - Improve sharing between Servo/Gecko. > - Reduce load on our infrastructure by building and testing only Moz2D > stand-alone when a change affects only Moz2D, both on regular builds and try. > > As the next step in moving towards these goals and optimally supporting them > the proposal is to move Moz2D into its own repository. We would use hg > subrepos for this purpose, you can read more about this here > (http://mercurial.selenic.com/wiki/Subrepository). The intention is that this > will be done in such a way that the workflow for developers not involved in > Moz2D development would essentially not change. > > A more detailed description of the proposal and the workflows involved can be > found here (https://wiki.mozilla.org/Platform/GFX/Moz2DSubrepository) for > those interested in the details. Since we believe if we go through with this > it would be the first time we use a true subrepository system for a component > used in mozilla-central, we'd very much appreciate any thoughts or feedback > people might have on the idea. > > Best regards, > Bas > ___ > dev-platform mailing list > dev-platform@lists.mozilla.org > https://lists.mozilla.org/listinfo/dev-platform ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Moz2D Repository Creation
Hi all, Over the past year we've increased our dependencies on Moz2D (formerly known under the codename Azure), our new 2D rendering API. Currently we're using it for canvas drawing on all platforms and content drawing on Windows where using Direct2D, and in the near future we will be moving to using it for content on all platforms. >From the very beginning of Moz2D development we've taken care to make sure it >can be built outside of the rest of Gecko, having only dependencies on several >MFBT headers. This was done to reduce the barrier for external usage and >development, we've since seen this benefit us for example in Servo using Moz2D >as well. In addition to that it has helped development and bugfixing by having >a much faster workflow for building/testing with which bugs could be located >and fixes created. Going forward we're expanding on that by ensuring the stand-alone builds and works on all platforms and becomes the defacto way of the 'first level' of testing when implementing new features or fixing bugs in Moz2D. In addition to that we're expanding on our existing stand-alone unittesting suite as well as adding a peformance testing suite, which will include microbenchmarking and support measuring performance on Moz2D drawing recordings (single-page recording support has just landed on mozilla-inbound). When moving forward we have several goals for Moz2D when it comes to the workflow: - Improve Moz2D development workflow by having faster turnaround time on builds and tests (both local and Try) - Lower the barrier for external contributors, some people have already expressed the desire to work on potential backends we do not wish to invest in ourselves, but have been deterred by the complexity of the checkout/building process. - Improve sharing between Servo/Gecko. - Reduce load on our infrastructure by building and testing only Moz2D stand-alone when a change affects only Moz2D, both on regular builds and try. As the next step in moving towards these goals and optimally supporting them the proposal is to move Moz2D into its own repository. We would use hg subrepos for this purpose, you can read more about this here (http://mercurial.selenic.com/wiki/Subrepository). The intention is that this will be done in such a way that the workflow for developers not involved in Moz2D development would essentially not change. A more detailed description of the proposal and the workflows involved can be found here (https://wiki.mozilla.org/Platform/GFX/Moz2DSubrepository) for those interested in the details. Since we believe if we go through with this it would be the first time we use a true subrepository system for a component used in mozilla-central, we'd very much appreciate any thoughts or feedback people might have on the idea. Best regards, Bas ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
