Re: Rethinking build defaults

[Justin Lebar]

> - Remove all conditional feature configuration from configure. WebRTC et al 
> are always
>   on. Features should be disabled dynamically (prefs), if at all.

Note that Fabrice has seen sizable memory-usage wins on B2G by
disabling non-web features such as printing and XUL.  That alone is,
in my mind, a very strong argument for keeping at least some of these
configure options around.

On Fri, Aug 16, 2013 at 2:43 AM, Andreas Gal  wrote:
>
> First of all, thanks for raising this. Its definitely a problem that needs
> fixing.
>
> I am not convinced by your approach though. In a few months from now
> disabling WebRTC is like calling for the DOM or JS or CSS to be disabled in
> local developer builds. It will become a natural part of the core Web
> platform.
>
> I would like to propose the opposite approach:
>
> - Remove all conditional feature configuration from configure. WebRTC et al
> are always on. Features should be disabled dynamically (prefs), if at all.
> - Reduce configure settings to choice of OS and release or developer.
> - Require triple super-reviews (hand signed, in blood) for any changes to
> configure.
> - Make parts of the code base more modular and avoid super include files
> cross modules (hello LayoutUtils.h).
>
> Rationale:
>
> Its not slow for you to build WebRTC. Its slow for you to have it build over
> and over. Almost every time I pull from mozilla-central, someone touched
> configure and I have to rebuild from scratch, which is infuriating (argh!).
> Minimizing changes to configure and banning static defines for feature
> management would solve that. If we make sure major subsystems like WebRTC
> can stand on their own, you will take a hit building it one time, and then
> occasionally as the team lands changes. Its a pretty small team, so the
> amount of code they can possibly check in is actually pretty small. You will
> see churn all over the three when you pull, but you won't have to rebuild
> the entire Universe every time you pull.
>
> What do you think?
>
> Andreas
>
>
> Mike Hommey wrote:
>>
>> Hi everyone,
>>
>> There's been a lot of traction recently about our builds getting slower
>> and what we could do about it, and what not.
>>
>> Starting with bug 904979, I would like to change the way we're thinking
>> about default flags and options. And I am therefore opening a discussion
>> about it.
>>
>> The main thing bug 904979 does is to make release engineering builds (as
>> well as linux distros, palemoon, icecat, you name it) use a special
>> --enable-release configure flag to use flags that we deem necessary for
>> a build of Firefox, the product. The flip side is that builds without
>> this flag, which matches builds from every developer, obviously, would
>> use flags that make the build faster. For the moment, on Linux systems,
>> this means disabling identical code folding and dead code removal (which,
>> while they make the binary size smaller, increase link time), and
>> forcing the use of the gold linker when it's available but is not system
>> default. With bug 905646, it will mean enabling -gsplit-dwarf when it's
>> available, which make link time on linux really very much faster (<4s
>> on my machine instead of 30s). We could and should do the same kind
>> of things for other platforms, with the goal of making linking
>> libxul.so/xul.dll/XUL faster, making edit-compile-edit cycles faster.
>> If that works reliably, for instance, we should for instance use
>> incremental linking. Please feel free to file Core::Build Config bugs
>> for what you think would help on your favorite build platform (and if
>> you do, for better tracking, make them depend on bug 904979).
>>
>> That being said, this is not the discussion I want to have here, that
>> was merely an introduction.
>>
>> The web has grown in the past few years, and so has our code base, to
>> support new technologies. As Nathan noted on his blog[1] disabling
>> webrtc calls for great build time improvements. And I think it's
>> something we should address by a switch in strategy.
>>
>> - How many people are working on webrtc code?
>> - How many people are working on peripheral code that may affect webrtc?
>> - How many people are building webrtc code they're not working on and
>>not using?
>>
>> I'm fairly certain the answer to the above is that the latter population
>> is much bigger than the other two, by probably more than an order of
>> magnitude.
>>
>> So here's the discussion opener: why not make things like webrtc (I'm
>> sure we can find many more[2]) opt-in instead of opt-out, for local,
>> developer builds? What do you think are good candidates for such a
>> switch?
>>
>> Mike
>>
>> 1.
>> https://blog.mozilla.org/nfroyd/2013/08/15/better-build-times-through-configury/
>> 2. and we can already start with ICU, because it's built and not even
>> used. And to add injury to pain, it's currently built without
>> parallelism (and the patch to make it not do so was backed out).
>> __

Re: Flash and e10s

[Justin Lebar]

On Tue, Aug 6, 2013 at 5:46 PM, Robert O'Callahan  wrote:
> I was talking to people about plans for Flash on e10s.
>
> Full support for windowed Flash on e10s is possible but would be a ton of
> work. Flash is on a downward trajectory and it would be a shame to do a ton
> of work to support something that may not be relevant for much longer.

Just to be clear about our assumptions here: You think it would be a
lot of work to do anything with Flash and content processes, including
running Flash within the content process itself?

> One idea I had is this: suppose, independently of e10s, we make Flash
> click-to-play. (I understand this is already a goal, or at least a wish.)
> Then suppose we allowed click-to-play to reload the page. We would then be
> able to ensure that any page where Flash is enabled is loaded directly in
> the master process and everything would just work. That's not ideal, but
> it's a fine stop-gap approach IMHO.
>
> As Shumway matures we could whitelist common sites where Shumway is known
> to work, so those sites wouldn't need to be hoisted to the master process.
>
> One problem with these ideas is H.264 video sites on Windows XP. We're
> stuck with using Flash there for now. We might need to impose a different
> policy on Windows XP, backing off e10s more there perhaps.
>
> Rob
> --
> Jtehsauts  tshaei dS,o n" Wohfy  Mdaon  yhoaus  eanuttehrotraiitny  eovni
> le atrhtohu gthot sf oirng iyvoeu rs ihnesa.r"t sS?o  Whhei csha iids  teoa
> stiheer :p atroa lsyazye,d  'mYaonu,r  "sGients  uapr,e  tfaokreg iyvoeunr,
> 'm aotr  atnod  sgaoy ,h o'mGee.t"  uTph eann dt hwea lmka'n?  gBoutt  uIp
> waanndt  wyeonut  thoo mken.o w  *
> *
> ___
> dev-platform mailing list
> dev-platform@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-platform
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: On builds getting slower

[Justin Lebar]

Nick, when you made changes to the JS engine's #includes, did you
observe a change in build times?

On Sat, Aug 3, 2013 at 1:14 AM, Nicholas Nethercote
 wrote:
> On Sat, Aug 3, 2013 at 5:47 PM, Mike Hommey  wrote:
>>
>> One piece of the puzzle, at least in Mozilla code, is the tendency to
>> #include "Foo.h" when class Bar contains a field of type Foo*, instead of
>> leaving the include to Bar.cpp and forward declare in Bar.h. This
>> certainly contributes to the conflation of the number of includes (most
>> if not all includers of Bar.h don't need Foo.h, and that chains up
>> pretty easily).
>>
>> I don't remember if IWYU tells about those. Does it?
>
> It does!  In the "file X should add the following:" parts it lists
> both #includes and forward declarations.
>
> Despite its imperfections[*] it's a pretty good tool, and you can get
> much further, much faster with it than you can manually.  If anyone
> wants to try it out, I'm happy to help with setting it up and all
> that.
>
> Nick
>
> [*] One imperfection I forgot to mention is that, although it gives me
> info on every .cpp file in SpiderMonkey, for some reason it doesn't
> give me info on every .h file, and I haven't worked out why.  This is
> frustrating, since minimizing #include statements in .h files is
> likely to have bigger benefits than minimizing them in .cpp files.
> ___
> dev-platform mailing list
> dev-platform@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-platform
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: reminder: content processes (e10s) are now used by desktop Firefox

[Justin Lebar]

On Sun, Aug 4, 2013 at 8:15 AM, Robert Kaiser  wrote:
> Mark Hammond schrieb:
>
>> We ask the docShell to not allow plugins or media
>
> So that means that for any page with a video or a big Flash/Java thing on
> it, I would get a completely wrong thumbnail? That's unfortunate.

It's a lot better than the page

a) playing audio,
b) spinning your cpu, or
a) pwning you.

All of which can happen if we run flash/java.  :)

Still, if this is a problem (there /are/ a lot of websites which are
just one big flash object), I wonder if we could detect it.
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Standard C/C++ and Mozilla

[Justin Lebar]

> I agree that iostream-based logging would be safer.  If we had it I
> wouldn't have had to work on this one:
>
> https://bugzilla.mozilla.org/show_bug.cgi?id=855335

I can't access that bug, but maybe you mean
https://bugzilla.mozilla.org/show_bug.cgi?id=onelogger ?

I feel like the goals there are orthogonal to NSPR vs iostream.

I haven't had a chance to work on this lately, but I do intend to land
something when I can.

On Fri, Aug 2, 2013 at 2:41 PM, Ethan Hugg  wrote:
>>It is very useful for building a logging interface that is safer and more
>>convenient than NSPR's printf-style logging. Note that, again, Eric
>>Rescorla already built an (partial) iostream-based wrapper around NSPR for
>>WebRTC. I would say that, if there is no additional overhead, then we
>>should consider making iostream-based logging the default way of doing
>>things in Gecko because it is so much less error-prone.
>
> I found this comment interesting.  It wasn't that long ago I was instructed
> to get rid of all iostream-based logging from media/webrtc/signaling and
> media/mtransport if it we wanted the logging to appear in opt builds.
>
> https://bugzilla.mozilla.org/show_bug.cgi?id=795126
> https://bugzilla.mozilla.org/show_bug.cgi?id=841899
>
> I agree that iostream-based logging would be safer.  If we had it I
> wouldn't have had to work on this one:
>
> https://bugzilla.mozilla.org/show_bug.cgi?id=855335
>
> Can we now use iostreams throughout this code?
>
> -EH
>
>
>
>
> On Fri, Aug 2, 2013 at 2:21 PM, Brian Smith  wrote:
>
>> On Wed, Jul 31, 2013 at 7:41 PM, Joshua Cranmer 🐧 > >wrote:
>>
>> > implementation, libc++, libstdc++, and stlport. Since most nice charts of
>> > C++11 compatibility focus on what the compiler needs to do, I've put
>> > together a high-level overview of the major additions to the standard
>> > library [3]:
>> > * std::function/std::bind -- Generalization of function pointers
>> >
>>
>> Note that Eric Rescorla implemented his own std::bind polyfill when he was
>> working on WebRTC. I also have some new code I am working on where
>> std::bind is extremely helpful.
>>
>>
>> > Now that you have the background for what is or will be in standard C++,
>> > let me discuss the real question I want to discuss: how much of this
>> should
>> > we be using in Mozilla?
>> >
>>
>>
>> > For purposes of discussion, I think it's worth breaking down the C++ (and
>> > C) standard library into the following components:
>> > * Containers--vector, map, etc.
>> > * Strings
>> > * I/O
>> > * Platform support (threading, networking, filesystems, locales)
>> > * Other helpful utilities (std::random, std::tuple, etc.)
>> >
>> > The iostream library has some issues with using (particularly static
>> > constructors IIRC), and is not so usable for most of the things that
>> Gecko
>> > needs to do.
>>
>>
>> It is very useful for building a logging interface that is safer and more
>> convenient than NSPR's printf-style logging. Note that, again, Eric
>> Rescorla already built an (partial) iostream-based wrapper around NSPR for
>> WebRTC. I would say that, if there is no additional overhead, then we
>> should consider making iostream-based logging the default way of doing
>> things in Gecko because it is so much less error-prone.
>>
>>
>> > Even if fully using the standard library is untenable from a performance
>> > perspective, usability may be enhanced if we align some of our APIs which
>> > mimic STL functionality with the actual STL APIs. For example, we could
>> add
>> > begin()/end()/push_back()/etc. methods to nsTArray to make it a fairly
>> > drop-in replacement for std::vector, or at least close enough to one that
>> > it could be used in other STL APIs (like std::sort, std::find, etc.).
>> > However, this does create massive incongruities in our API, since the
>> > standard library prefers naming stuff with this_kind_of_convention
>> whereas
>> > most Mozilla style guides prefer ThisKindOfConvention.
>> >
>>
>> Perhaps a more annoying issue--though not a showstoper--is that
>> unique_ptr::release() means something quite different than
>> nsXXXPtr::Release() means.
>>
>>
>> > With all of that stated, the questions I want to pose to the community at
>> > large are as follows:
>> > 1. How much, and where, should we be using standard C++ library
>> > functionality in Mozilla code?
>> >
>>
>> We should definitely prefer using the standard C++ library over writing any
>> new code for MFBT, *unless* there is consensus that the new thing we'd do
>> in MFBT is substantially clearer. (For example, I think some people
>> successfully argued that we should have our own atomic types because our
>> interface is clearly better than std::atomic.)
>>
>> Even in the case where MFBT or XPCOM stuff is generally better, We should
>> *allow* using the standard C++ library anywhere that has additional
>> constraints that warrant a different tradeoff; e.g. needing to be built
>> separately from Gecko and/or otherwise needing to minimize Gecko

Re: reminder: content processes (e10s) are now used by desktop Firefox

[Justin Lebar]

On Thu, Aug 1, 2013 at 6:50 PM, Nicholas Nethercote
 wrote:
> On Thu, Aug 1, 2013 at 6:29 PM, Gavin Sharp  wrote:
>>
>> Do you have specific issues you're worried about, or are you just speaking
>> about issues in general?
>
> This AdBlock issue worries me specifically.  And the fact that there's
> breakage with our #1 add-on makes me worry in general.

To be clear, the issue is that thumbnails don't have ABP enabled, not
that ABP falls over in general.

>> In general, I don't see us resolving e10s issues until we have some
>> incentive to do so, and get some sense of their relative severity.
>
> Huh?  This sentence seems entirely antithetical to our standard
> operating procedure.  I.e. backing out known regressions, etc.

It's a trade-off, right?  You can have thumbnails with ABP that jank
the browser, or you can have thumbnails without ABP that don't.  (AIUI
the oop business also fixes the issue where thumbnails showed private
data, like bank account info.  There are probably other things it
fixes too.)

Showing banner ads in the screencaps doesn't seem like a big deal to
me compared to what we get in return...  I guess the biggest issue
from my perspective might be that we're allowing advertisers to track
us, by making HTTP requests we otherwise wouldn't.  Hopefully we send
the cookies to /dev/null, but even still they get our IP.
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Rethinking separate Mercurial repositories

[Justin Lebar]

> Sadly, mercurial doesn't support having multiple working directories
> from a single clone, which would be useful to avoid wasting so much disk
> space on .hg.

I'm not usually one to defend hg, but hg does have the |relink|
command, which gets you most of the way there in terms of saving disk
space.

On Wed, Jul 31, 2013 at 5:57 PM, Mike Hommey  wrote:
> On Wed, Jul 31, 2013 at 10:28:38AM -0700, Justin Lebar wrote:
>> >> Wouldn't switching branches in the same repo clone touch many files
>> >> and trigger unfortunately clobber builds? Even with ccache and
>> >> separate per-branch objdirs, this seems like a problem.
>> >
>> > Yes.
>>
>> Nothing about this proposal forces you to have only one clone and
>> switch back and forth between aurora and central, thus clobbering your
>> objdir.  You could still have two trees, one used for aurora, and one
>> used for central, if you prefer that.
>
> Sadly, mercurial doesn't support having multiple working directories
> from a single clone, which would be useful to avoid wasting so much disk
> space on .hg. But git does ;) (albeit, with a script in contrib/)
> I guess someone could patch mercurial to support such setups, which,
> independently of what we do on the servers, could be useful on the
> clients (pull different branches in the same local clone, use different
> working directories for each).
>
> Mike
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Standard C/C++ and Mozilla

[Justin Lebar]

> 1. How much, and where, should we be using standard C++ library
> functionality in Mozilla code?

We've tuned tarray, nsthashtable, strings, etc. to meet our precise
needs, and the implementations are consistent across all platforms.
I can imagine things becoming quite messy we had three or four
different implementations of these classes (in the different stdlibs),
each with their own quirks, and if we couldn't change the
implementations to meet our needs.

I definitely think that some of our APIs could use some love, but it
seems unlikely to me that replacing a complex class like nsTArray with
std::vector would be a net win for us, as compared to simply improving
nsTArray's interface.  Even performing this experiment would be an
expensive endeavor.

By way of contrast, I think it's great that we're using simple
classes, functions, and types from stdlib, such as static_assert and
stdint.  The downside here is much smaller, since we don't have to
worry about the quirks of the different implementations, and since
there's nothing we might want to tune.

-Justin

On Wed, Jul 31, 2013 at 12:08 PM, Ehsan Akhgari  wrote:
> On 2013-07-31 1:41 PM, Joshua Cranmer 🐧 wrote:
>>
>> With all of that stated, the questions I want to pose to the community
>> at large are as follows:
>> 1. How much, and where, should we be using standard C++ library
>> functionality in Mozilla code?
>
>
> I'm not sure if it's easy to have this discussion in general without talking
> about a specific standard library feature.
>
>
>> 2. To what degree should our custom ADTs (like nsTArray) be
>> interoperable with the C++ standard library?
>
>
> I think some people would like to make the code more understandable to newer
> contributors, and some people would prefer to keep existing convention
> intact.
>
>
>> 3. How should we handle bridge support for standardized features not yet
>> universally-implemented?
>
>
> I think MFBT has been working fine so far.
>
>
>> 4. When should we prefer our own implementations to standard library
>> implementations?
>
>
> Usually doing our own implementation is faster, but there is a significant
> lag from contributing something upstream (if that's even possible) until
> that gets released in a toolchain that people use.  So perhaps we should do
> both in parallel when there is the option.
>
>
>> 5. To what degree should our platform-bridging libraries
>> (xpcom/mfbt/necko/nspr) use or align with the C++ standard library?
>
>
> This is also hard to talk about without having a concrete thing under
> consideration.
>
>
>> 6. Where support for an API we wish to use is not universal, what is the
>> preferred way to mock that support?
>
>
> MFBT, I believe.
>
> Cheers,
> Ehsan
>
> ___
> dev-platform mailing list
> dev-platform@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-platform
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Rethinking separate Mercurial repositories

[Justin Lebar]

>> Wouldn't switching branches in the same repo clone touch many files and
>> trigger unfortunately clobber builds? Even with ccache and separate
>> per-branch objdirs, this seems like a problem.
>
> Yes.

Nothing about this proposal forces you to have only one clone and
switch back and forth between aurora and central, thus clobbering your
objdir.  You could still have two trees, one used for aurora, and one
used for central, if you prefer that.

I'm concerned about this argument mainly because I don't want people
to apply it to git.  :)
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Is it necessary to remove message listeners?

[Justin Lebar]

> Just to be clear though, if I find they are *not* all being removed, I
> should open a bug on that rather than just removing the listeners myself and
> calling it done?  ie, is it accurate to say that it *should* not be
> necessary to remove these handlers (and, if I verify that is true, that I
> could explicitly add a note to this effect on the relevant MDN pages?)

You'd have to ask smaug to be sure, but judging from what I looked at,
I'd think that we are trying to remove the event listeners when the
relevant window and/or goes away.  So I'd file a bug; we can always
resolve it as invalid.

>> [1] https://bug893242.bugzilla.mozilla.org/attachment.cgi?id=774978
>>
>> On Thu, Jul 25, 2013 at 6:51 PM, Mark Hammond 
>> wrote:
>>>
>>> Felipe and I were having a discussion around a patch that uses
>>> nsIMessageManager.  Specifically, we create a  element, then
>>> call
>>> browser.messageManager.addMessageListener() with the requirement that the
>>> listener live for as long as the browser element itself.
>>>
>>> The question we had was whether it was necessary to explicitly call
>>> removeMessageListener, or whether we can rely on automatic cleanup when
>>> the
>>> browser element dies?  It seems obvious to us that it *should* be safe to
>>> rely on automatic cleanup, but searching both docs and mxr didn't make it
>>> clear, so I figured it was better to ask rather than to cargo-cult the
>>> addition of explicit cleanup code that wasn't necessary.
>>>
>>> Thanks,
>>>
>>> Mark
>>> ___
>>> dev-platform mailing list
>>> dev-platform@lists.mozilla.org
>>> https://lists.mozilla.org/listinfo/dev-platform
>
>
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Is it necessary to remove message listeners?

[Justin Lebar]

Thanks for asking about this; we have a lot of unnecessary unlinking
code in our JS,

Let me share how I investigated your question.

$ git grep -i addmessagelistener -- '*.cpp'
content/base/src/nsFrameMessageManager.cpp:nsFrameMessageManager::AddMessageListener(const
nsAString& aMessage,

Only one hit; that's easy.

Looking at the function, message listeners are stored in
nsFrameMessageManager::mListeners.  So what's the lifetime of that
array?

The array is cleared in a few places:

1) nsFrameMessageManager::Disconnect(),
2) MMListenerRemover::~MMListenerRemover
3) nsFrameMessageManager's CC unlink function,

It looks like (2) is just a way of delaying Disconnect()'s removal
until message handling is complete.

So when do we call Disconnect()?

http://dxr.mozilla.org/mozilla-central/search?q=%2Bcallers%3AnsFrameMessageManager%3A%3ADisconnect%28_Bool%29

At this point it's clear that we /attempt/ to clear the message
listeners when the message manager's window goes away.  That's
probably as far as you'd get by asking someone.

Whether or not we totally succeed in this endeavor is another question
entirely.  You could instrument your build to count the number of live
nsFrameMessageManager objects and report the number of message
listeners in each -- one way to do this would be with a patch very
similar to this one [1].

[1] https://bug893242.bugzilla.mozilla.org/attachment.cgi?id=774978

On Thu, Jul 25, 2013 at 6:51 PM, Mark Hammond  wrote:
> Felipe and I were having a discussion around a patch that uses
> nsIMessageManager.  Specifically, we create a  element, then call
> browser.messageManager.addMessageListener() with the requirement that the
> listener live for as long as the browser element itself.
>
> The question we had was whether it was necessary to explicitly call
> removeMessageListener, or whether we can rely on automatic cleanup when the
> browser element dies?  It seems obvious to us that it *should* be safe to
> rely on automatic cleanup, but searching both docs and mxr didn't make it
> clear, so I figured it was better to ask rather than to cargo-cult the
> addition of explicit cleanup code that wasn't necessary.
>
> Thanks,
>
> Mark
> ___
> dev-platform mailing list
> dev-platform@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-platform
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Breaking the docshell hierarchy for an iframe to enable indexeddb

[Justin Lebar]

> As I understand it this is the case. From Mark's original post:

Ah, thanks.  I missed that whole paragraph.

With some simple Gecko patches, we could pretty easily control which
process an iframe gets allocated into, so going down that route sounds
sane to me.

Alternatively, if you made the frame directly inside the 
chrome, then I think that would probably also solve the problem.

-Justin
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Breaking the docshell hierarchy for an iframe to enable indexeddb

[Justin Lebar]

> Poking around, I discovered bug 806127, where jlebar struck the same basic 
> problem for b2g, which gave me the
> clue for a work-around.  If I grab the docShell for this frame and execute:
>
>  docShell.setIsBrowserInsideApp(Ci.nsIScriptSecurityManager.NO_APP_ID);

You should check, if you haven't already, that if you do this, the
frame gets the set of cookies you want.  I'm not totally clear on how
Necko handles this, but it's possible that this would cause the social
API window not to share cookies with the rest of the browser.  That's
probably not what you want.

I'm actually surprised that  inside the hidden
window does not count as top-level.  This window should be the
top-level window in its process.

Unless you have an  inside the , and it's
that  that you were setting as an app?  If that's the case,
could you merge the  and ?  (You could
either use a  or an .)

Feel free to find me on IRC if you want to talk about this synchronously.

-Justin

On Wed, Jul 24, 2013 at 10:50 PM, Mark Hammond
 wrote:
> As a bit of background, I'm working on an enhancement to our "Social API".
> We currently have the concept of a "FrameWorker", which is basically a
> simulation of a SharedWorker implemented by hosting an  in the
> hiddenDOMWindow. This iframe loads content from a (real) site, and needs
> access to IndexedDB - so far, so good - it all works fine.
>
> The enhancement is that we'd like to host this in a  remote="true"/>, so the content lives in its own process.  Everything works
> fine *except* IndexedDB.  I've narrowed this down to the fact that IndexedDB
> is not available in "3rd party" iframes (by design - bug 595307).  It does
> work fine when the iframe is hosted in the hidden window - the iframe is not
> considered 3rd party (jst tells me on #developers that the hidden window
> gets some special handling so this is all as expected.)
>
> My problem arises when I host the iframe in the  element - this
> special-case for the hidden window no longer applies.  The parent browser is
> about:blank and the iframe is on a real site, so suddenly the iframe is
> considered 3rd party and IndexedDB access is blocked.  The problem would
> probably (ie, I haven't checked) go away if we just hosted the content
> directly in the  element (ie, dropped the iframe), but we'd prefer
> to avoid that currently if possible (mainly so we can host multiple of these
> iframes in a single remote process if desired)
>
> Poking around, I discovered bug 806127, where jlebar struck the same basic
> problem for b2g, which gave me the clue for a work-around.  If I grab the
> docShell for this frame and execute:
>
>   docShell.setIsBrowserInsideApp(Ci.nsIScriptSecurityManager.NO_APP_ID);
>
> then everything works fine - the 3rd-party checking code sees the frame as
> an "app boundary" and allows access to indexeddb - and life is good - except
> for this hack I need to explain to a reviewer ;)
>
> (FYI, docShell.setIsApp(Ci.nsIScriptSecurityManager.NO_APP_ID) does *not*
> work, even though it looked like it might, and, at face value, sounds more
> appropriate.  It would work if I supplied a real appId.)
>
> So, my questions are:
>
> * Does anyone know a better work-around or solution to this?
> * If not, is my work-around a safe and reasonable thing to be doing?
>
> Thanks,
>
> Mark
> ___
> dev-platform mailing list
> dev-platform@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-platform
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Using C++0x auto

[Justin Lebar]

> It seems really dangerous that there is an implicit conversion from a strong
> ref ptr to a weak pointer. With C++11, you can thankfully require this
> conversion to be explicit which should alleviate your concern.

Wouldn't disallowing this implicit conversion break code which does

  void UseFoo(nsIFoo* foo);

  nsCOMPtr foo;
  UseFoo(foo);

?  That is an extremely common idiom in our code.

Like I say, maybe there's a way to allow lvalue nsRefPtr to convert
to T*, but disallow the conversion for rvalue nsRefPtr.  That would
be reasonable, I think.
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Using C++0x auto

[Justin Lebar]

AIUI the new constructor would be something like

  nsRefPtr(nsRefPtr&& aOther)

where "&&" means "r-value", which means "temporary", so "moveable".

But I'm not totally sure about being able to return nsRefPtr.

Right now, if I do

  already_AddRefed GetFoo();
  Foo* foo = GetFoo();

that's a compile error.  In order to leak the reference, I'd have to
do foo = GetFoo().forget();, which is relatively easy for reviewers to
catch.

But if we return nsRefPtr, then if I have

  nsRefPtr GetFoo();
  Foo* = GetFoo();

that compiles fine (I think?) because nsRefPtr has an implicit
conversion to T*.  I'd wager that this is probably pretty hard for
reviewers to catch; at least, I don't have confidence I'd catch this
mistake.

Maybe there's a way to delete the nsRefPtr conversion to T* if the
nsRefPtr is an r-value.

-Justin

On Sun, Jul 21, 2013 at 9:08 PM, Robert O'Callahan  wrote:
> On Mon, Jul 22, 2013 at 3:58 PM, L. David Baron  wrote:
>
>> Is the idea here that nsRefPtr/nsCOMPtr/etc. would have move
>> constructors, and we'd just return them, and the move constructors
>> plus return value optimizations would take care of avoiding excess
>> reference counting?
>>
>
> That, plus forget() returns a nsRefPtr.
>
> Rob
> --
> Jtehsauts  tshaei dS,o n" Wohfy  Mdaon  yhoaus  eanuttehrotraiitny  eovni
> le atrhtohu gthot sf oirng iyvoeu rs ihnesa.r"t sS?o  Whhei csha iids  teoa
> stiheer :p atroa lsyazye,d  'mYaonu,r  "sGients  uapr,e  tfaokreg iyvoeunr,
> 'm aotr  atnod  sgaoy ,h o'mGee.t"  uTph eann dt hwea lmka'n?  gBoutt  uIp
> waanndt  wyeonut  thoo mken.o w  *
> *
> ___
> dev-platform mailing list
> dev-platform@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-platform
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Using C++0x auto

[Justin Lebar]

>> Maybe we should call ours mozilla::move and mozilla::forward so that we
>> can change to std::move and std::forward with minimal pain?
>>
> Won't that cause confusion if someone accidentally has both using namespace
> mozilla; and using namespace std; at the same time?

That's a fair point.  Maybe we should call ours mozilla::Move and
mozilla::Forward instead.  That way we can still easily change them to
std::move and std::forward, but there's no possibility of a collision.
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Using C++0x auto

[Justin Lebar]

Maybe we should call ours mozilla::move and mozilla::forward so that we can
change to std::move and std::forward with minimal pain?
On Jul 19, 2013 4:36 PM, "Ehsan Akhgari"  wrote:

> On 2013-07-19 7:04 PM, Mike Hommey wrote:
>
>> On Fri, Jul 19, 2013 at 03:45:13PM -0400, Ehsan Akhgari wrote:
>>
>>> On 2013-07-19 3:26 PM, Ehsan Akhgari wrote:
>>>
 On 2013-07-18 10:46 PM, Mike Hommey wrote:

> On Thu, Jul 18, 2013 at 08:54:02PM -0500, Joshua Cranmer ? wrote:
>
>> On 7/18/2013 7:15 PM, Robert O'Callahan wrote:
>>
>>> On Fri, Jul 19, 2013 at 3:34 AM, Ehsan Akhgari
>>> **wrote:
>>>
>>>  On 2013-07-18 5:48 AM, mscl...@googlemail.com wrote:

   r-value references  4.3@10.0!   Yes
 This is very useful.  I believe the JS engine already rolls their
 own tricks to implement this semantics.

  With this we can get rid of already_AddRefed and just pass
>>> nsRefPtr/nsCOMPtr/RefPtr around, right?
>>>
>> I believe so. We can also add a non-broken variant of nsAutoPtr
>> modeled after std::unique_ptr (allows moves but not copies).
>>
>
> Note that STL is another story. We're not using libstdc++ that comes
> with the compiler on android and b2g. We use STLport instead, and
> STLport has, afaik, no support for C++11 STL types. So, while we can
> now fix nsAutoPtr to use move semantics instead of copy semantics,
> we can't use std::unique_ptr.
>

 Does this mean that we can't use std::move and std::forward either?

>>>
>> Indeed
>>
>
> Sadface.  I filed bug 896100 about this, and I'll write a patch for it
> soon.
>
>  In that case, we need MFBT versions of those, because working with
 rvalue references without those two functions is a pain!

 Good news is that implementing these functions should be really easy.

>>>
>>> Also, is there a reason that we don't build on Android with the
>>> libstdc++ that ships with the toolchain?
>>>
>>
>> For one, the fact that it actually doesn't work currently (at least, it
>> didn't last i heard someone tried) doesn't help.
>>
>
> Heh, yeah it doesn't, indeed!
>
>  And I think that when stlport was chosen, using the GNU stl made the
>> android package significantly larger. But then, we weren't statically
>> linking stlport, so this may have changed.
>> Also note that because we're building for specific targets that are not
>> exactly those for which we have prebuilt libraries coming in the NDK,
>> we're actually building stlport ourselves. We'd have to do the same for
>> the GNU stl if we chose to use it, and we have no current support for
>> that in the build system (the build system only knows how to build
>> against the prebuilt one, and that's what doesn't even work). I don't
>> know how easy it is to compile the GNU stl outside of building gcc.
>>
>
> Ah, I have  feeling that doing that is actually very hard...  Let's
> declare defeat on this for now (unless people want to do all of this work!)
> and move on with bug 896100.  Once we have that, I will try to find some
> time to see if we can kill already_AddRefed.  That is one class I won't
> miss.  :-)
>
> Ehsan
> __**_
> dev-platform mailing list
> dev-platform@lists.mozilla.org
> https://lists.mozilla.org/**listinfo/dev-platform
>
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Proposal: requiring build peer review for Makefile.in changes

[Justin Lebar]

The flip side of this, of course, is that build peers need to ensure
that they are not the long pole in reviews.  But I presume you guys
are prepared to turn around these additional reviews quickly,
otherwise you wouldn't have asked for the extra load.

On Wed, Jul 17, 2013 at 5:00 PM, Gregory Szorc  wrote:
> Traditionally, it's been very difficult for the build peers to keep on
> top of changes in the build config because changes are occurring on bug
> components we don't follow, are touching files all over the tree, and
> because build peers aren't always asked for review.
>
> The potential for "sneaking" things past build peer review has resulted
> in a number of self-inflicted wounds, the Fennec build rules probably
> being the best example (the dependencies are all wrong and no-op builds
> take ~16s when they should take 1 or 2s).
>
> This history contributed to us implementing a more strict "sandbox" in
> moz.build files: take away as many footguns as possible and there will
> be less self-inflicted wounds.
>
> Unfortunately, the moz.build conversion isn't finished and it will drag
> on for a while. There will still be Makefile.in in the tree and that
> leaves the door open for new badness.
>
> I would like to reinforce the existing policy: *if you are changing a
> Makefile.in, please ask a build peer for review unless the change is
> just adding or removing to an existing list.*
>
> For the most part, people have been abiding by this policy. However,
> things are still creeping through. Unfortunately, some of them wouldn't
> get r+ from a build peer.
>
> Since new Makefile.in badness makes people's lives harder (especially
> when it makes the build slower), I would like to propose a more strict
> policy around Makefile.in changes: *if a non-list change in a
> Makefile.in isn't reviewed by a build peer, it doesn't land or gets
> backed out.* This could potentially be enforced with repository push hooks.
>
> I /think/ this proposal is supported by our module governance system
> since Makefile.in are the purview of the build config module. But I
> wanted to run the proposal by people to make sure it is generally
> well-received and there aren't any major concerns.
>
> Gregory
> ___
> dev-platform mailing list
> dev-platform@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-platform
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Using C++0x auto

[Justin Lebar]

> We can't require any c++11 feature until we drop support for gcc 4.4.
> [...] there are problems in the gcc 4.4 system headers that make using c++11 
> mode impossible (except on b2g/android).

Is there any reason to support gcc 4.4 outside of B2G/Android?

If we dropped support for gcc 4.4 on desktop, would we get auto (and
potentially other C++11 niceties)?  That seems like a pretty easy
win...

On Sat, Jul 13, 2013 at 5:00 PM, Mike Hommey  wrote:
> On Sat, Jul 13, 2013 at 01:15:31PM -0700, Kyle Huey wrote:
>> We've dropped support for versions of MSVC prior to 2010, and we're
>> requiring at least GCC 4.4.  According to [0] that means we should be able
>> to use *auto*.  Anybody know any reasons why we can't start using it?
>
> We can't require any c++11 feature until we drop support for gcc 4.4. Or
> remove support for non c++11. Because there are problems in the gcc 4.4
> system headers that make using c++11 mode impossible (except on
> b2g/android). I'm happy with removing support for non-c++11 mode now
> that m-c is past next esr.
> But starting to use C++11 features can be dangerous, as it will make
> backporting harder.
>
> Mike
> ___
> dev-platform mailing list
> dev-platform@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-platform
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Embracing git usage for Firefox/Gecko development?

[Justin Lebar]

> I may still be missing something, but afaict mq < git rebase -i < hg
> qcrecord (from the crecord extension.) This is speaking as someone who
> hasn't used git rebase -i much, but people who have seem to agree with me
> after seeing a qcrecord/qcrefresh demo.

qcrecord is, as far as I'm aware (it's been a while), much more like
git add -i than git rebase -i.

I would definitely be interested in having a patch-off with you sometime.  :)
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Embracing git usage for Firefox/Gecko development?

[Justin Lebar]

> Can you describe your "patch queue"-like workflow using git? git rebase -i
> lets you reorder commits, but how do you quickly do the equivalent of hg
> qpopping and qpushing a bunch of commits?

qpop && qpush is a nop, of course; what do you want to in-between
those two commands?

I have a |git qrebase| command which does |git rebase -i| on my whole
"patch queue".  The "patch queue" is defined as the set of commits in
the range $(git qparent)..HEAD, where |git qparent| is the first
common ancestor between HEAD and |git tracks| (i.e., |git merge-base
$(git tracks) HEAD|).  |git tracks| is defined as the upstream branch
that the current branch tracks (as set by |git branch -u| in newer
versions of git).

All of these commands are defined in my git-tools repository.

https://github.com/jlebar/moz-git-tools

As an example, to qpop your patch queue so that commit X is the new
head, then add a new patch on top of X, then re-apply your patch
queue, you'd do |git qrebase|, then mark commit X as "edit", then |git
commit| your new patch, then |git rebase --continue|.

This is better than hg in a number of ways, but among them is the fact
that the old state of your patch queue is not lost for 90 days or so.
If you discover that you screwed up and want to go back to the old
version of the patch queue, you merely need to look through |git
reflog| and then |git reset --hard Z| where Z is the commit you were
at before you rebased.  This is in contrast to hg, where |hg qref| is
destructive and can cause you to lose work (unless you're versioning
your patch queue, which is a whole other can of worms).

On Wed, Jul 10, 2013 at 6:49 PM, Chris Peterson  wrote:
> On 7/10/13 3:01 PM, Justin Lebar wrote:
>>>
>>> I can't see how they are a good alternative. With patch queues, I can
>>> maintain a complex refactoring in a patch queue
>>> containing dozens of smallish patches. In particular, I can easily
>>> realize I made a mistake in patch 3 while working on patch
>>> 21 and make sure that the fix ends up in patch 3; I don't see how that is
>>> easily achievable in git branches.
>>
>>
>> This is far easier to achieve in git than with mq.
>
>
> Can you describe your "patch queue"-like workflow using git? git rebase -i
> lets you reorder commits, but how do you quickly do the equivalent of hg
> qpopping and qpushing a bunch of commits?

I use
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: review stop-energy (was 24hour review)

[Justin Lebar]

One thing I've been thinking about is /why/ people are slow at reviews.

Someone who usually has a long review queue has told me that he "hates"
reviewing code.  I realized that we don't really have a place at Mozilla for
experienced hackers who don't want to do reviews.  Should we?  Could we do this
without violating people's sense of fairness?

As another data point, someone told me recently that he's trying not to become
a peer of the modules he's working on, because he doesn't want to review code.
Maybe we're not incentivizing people enough to be reviewers.

Roc said that we've spoken with people who are slow at reviews.  Did we learn
anything from that?  I'd expect that people are slow for different reasons.

It seems backward to me to focus on a solution without first understanding the
causes.

-Justin

On Thu, Jul 11, 2013 at 3:08 AM, Robert O'Callahan  wrote:
> On Wed, Jul 10, 2013 at 3:26 PM, Justin Lebar 
> wrote:
>>
>> If I can propose something that's perhaps different:
>>
>> 1) Write software to figure out who's "slow with reviews".
>> 2) We talk to those people.
>
>
> We've done this before too.
>
> But we should just do it again --- the "definition of insanity" aphorism is
> nonsense. Repetition helps people learn.
>
> Rob
> --
> Jtehsauts  tshaei dS,o n" Wohfy  Mdaon  yhoaus  eanuttehrotraiitny  eovni le
> atrhtohu gthot sf oirng iyvoeu rs ihnesa.r"t sS?o  Whhei csha iids  teoa
> stiheer :p atroa lsyazye,d  'mYaonu,r  "sGients  uapr,e  tfaokreg iyvoeunr,
> 'm aotr  atnod  sgaoy ,h o'mGee.t"  uTph eann dt hwea lmka'n?  gBoutt  uIp
> waanndt  wyeonut  thoo mken.o w
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: review stop-energy (was 24hour review)

[Justin Lebar]

One definition of insanity is doing the same thing twice and expecting
different results.

I recall that Taras has written basically this same e-mail before.  We
seem to have this conversation every six months or so.  Why do we
expect different results this time?

If I can propose something that's perhaps different:

1) Write software to figure out who's "slow with reviews".
2) We talk to those people.

This is basically the same approach as we had with the tryserver
highscore board.  It seems to have helped (?).

I think Bugzilla archaeology was intended to help with [1], but that
site is not useful for this purpose, IMO.  But I don't see anything
stopping us from actually writing this software.

[1] https://metrics.mozilla.com/bugzilla-analysis/ReviewHistory.html

On Tue, Jul 9, 2013 at 3:14 PM, Taras Glek  wrote:
> Hi,
> Browsers are a competitive field. We need to move faster. Eliminating review
> lag is an obvious step in the right direction.
>
> I believe good code review is essential for shipping a good browser.
>
> Conversely, poor code review practices hold us back. I am really frustrated
> with how many excellent developers are held back by poor review practices.
> IMHO the single worst practice is not communicating with patch author as to
> when the patch will get reviewed.
>
> Anecdotal evidence suggests that we do best at reviews where the patch in
> question lines up with reviewer's current project The worst thing that
> happens there is rubber-stamping (eg reviewing non-trivial 60KB+ patches in
> 30min).
>
> Anecdotally, latency correlates inversely with how close the reviewer is to
> patch author, eg:
>
> project > same team > same part of organization > org-wide > random
> community member
>
> I think we need change a couple things*:
>
> a) Realize that reviewing code is more valuable than writing code as it
> results in higher overall project activity. If you find you can't write code
> anymore due to prioritizing reviews over coding, grow more reviewers.
>
> b) Communicate better. If you are an active contributor, you should not
> leave r? patches sitting in your queue without feedback. "I will review this
> next week because I'm (busy reviewing ___ this week|away at conference). I
> think bugzilla could use some improvements there. If you think a patch is
> lower priority than your other work communicate that.
>
> c) If you think saying nothing is better than admitting than you wont get to
> the patch for a while**, that's passive aggressiveness
> (https://en.wikipedia.org/wiki/Passive-aggressive_behavior). This is not a
> good way to build a happy coding community. Managers, look for instances of
> this on your team.
>
> In my experience the main cause of review stop-energy is lack of will to
> inconvenience own projects by switching gears to go through another person's
> work.
>
> I've seen too many amazing, productive people get discouraged by poor review
> throughput. Most of these people would rather not create even more tension
> by complaining about this...that's what managers are for :)
>
> Does anyone disagree with my 3 points above? Can we make some derivative of
> these rules into a formal policy(some sort of code of developer conduct)?
>
> Taras
>
> * There obvious exceptions to above guidelines (eg deadlines).
> ** Holding back bad code is a feature, not a bug, do it politely.
> ___
> dev-platform mailing list
> dev-platform@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-platform
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Embracing git usage for Firefox/Gecko development?

[Justin Lebar]

On Fri, May 31, 2013 at 4:07 PM, Matt Brubeck  wrote:
> On 5/31/2013 12:32 PM, Boris Zbarsky wrote:
>>
>> On 5/31/13 3:20 PM, Matt Brubeck wrote:
>>>
>>> blame mobile/android/chrome/content/browser.xul:
>>>git 1.015s
>>>hg  0.830s
>>
>>
>> Was this a git blame -C (which would be more similar to hg blame), or
>> just a git blame?

The git repository contains CVS history, so is often slower than hg blame.
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Embracing git usage for Firefox/Gecko development?

[Justin Lebar]

> I can't see how they are a good alternative. With patch queues, I can 
> maintain a complex refactoring in a patch queue
> containing dozens of smallish patches. In particular, I can easily realize I 
> made a mistake in patch 3 while working on patch
> 21 and make sure that the fix ends up in patch 3; I don't see how that is 
> easily achievable in git branches.

This is far easier to achieve in git than with mq.  I this to a room
of experienced hg developers a few days ago and got a loud chorus of
"wow"s.

But git isn't easy to use, and the fact that you think git can't do
this is certainly evidence for that.

>>* Developers can benefit from the better merge algorithms
>>  used by git.
>
> I don't know what you mean by that. Note that git's octopus merges are not
> directly realizable in hg, so an hg-git bidirectional mirror may need to ban
> such things in the incoming git.

An octopus merge is not a merge algorithm.

The merge algorithm is the thing that decides /how/ to merge two (or N) commits.

> Mercurial also treats history as immutable, so you can't push history 
> revisions.

History in git is no more or less immutable than in hg.

Every time you hg qpop or qref, you're mutating history.

On Thu, May 30, 2013 at 11:04 PM, Joshua Cranmer 🐧  wrote:
> On 5/30/2013 7:56 PM, Johnny Stenback wrote:
>>
>> [TL;DR, I think we need to embrace git in addition to hg for
>> Firefox/Gecko hacking, what do you think?]
>
>
> Personally, the more I use git, the more I hate it. But I also see that
> other people love it...
>
>
>>* Developers can use git branches. They just work,
>>  and they're a good alternative to patch queues.
>
>
> I can't see how they are a good alternative. With patch queues, I can
> maintain a complex refactoring in a patch queue containing dozens of
> smallish patches. In particular, I can easily realize I made a mistake in
> patch 3 while working on patch 21 and make sure that the fix ends up in
> patch 3; I don't see how that is easily achievable in git branches.
>
>
>>* Developers can benefit from the better merge algorithms
>>  used by git.
>
> I don't know what you mean by that. Note that git's octopus merges are not
> directly realizable in hg, so an hg-git bidirectional mirror may need to ban
> such things in the incoming git.
>
>
>>* Git works well with Github, even though we're not switching
>>  to Github as the ultimate source of truth (more on that below).
>
>
> Outside of the community size, I personally can't see any benefits to Github
> for Mozilla; if all you want is community outreach, you pretty much just
> need a read-only git mirror on Github, which we already have.
>
>
>> Some of the known issues with embracing git are:
>>
>>* Performance of git on windows is sub-optimal (we're
>>  already working on it).
>>* Infrastructure changes needed...
>
> * Retraining Mercurial developers to use git (if we went git-only)
> * Porting all of our Mercurial customizations to git for a check-in head
> * Updating all of our developer build documentation
> * Adapting build processes to adjust to the git way of doing things
>
>
>
>> Option 1 is where I personally think it's worth investing effort. It
>> means we'd need to set up an atomic bidirectional bridge between hg and
>> git (which I'm told is doable, and there are even commercial solutions
>> for this out there that may solve this for us).
>
>
> This is harder than it sounds: hg and git are subtly different in ways that
> can seriously frustrate users if you don't pay attention. For example (one
> that bit me very hard): hg records file copy/moves as part of the changeset,
> while git attempts to reverse engineer them when it needs to do so. So my
> attempt to merge my changes with a remote git repository via hg-git failed
> miserably since the hg local repo didn't catch the git moved repo. Mercurial
> also treats history as immutable, so you can't push history revisions.
>
> If all you need is a read-only mirror, these kind of changes aren't as
> important (you're not going to damage official records, just inconvenience
> some developers); for a two-way mirror, they are very much worth
> considering.
>
> --
> Joshua Cranmer
> Thunderbird and DXR developer
> Source code archæologist
>
>
> ___
> dev-platform mailing list
> dev-platform@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-platform
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Embracing git usage for Firefox/Gecko development?

[Justin Lebar]

On Wed, Jun 5, 2013 at 3:25 AM, Philipp Kewisch  wrote:
> I also agree to Randell and Joshua. I've been using both lately and there
> are just a few things missing in git that I am used to in hg.
>
> Mercurial Queues is the most prominent. I am used to switching the order of
> patches in my queue, which seems like a pain to me in git. Or maybe I
> haven't quite found out how to do it reliably.

git rebase --interactive.  It is /far/ more powerful than mq for this use-case.

I even have a |git qrebase| alias for this.

https://github.com/jlebar/moz-git-tools

> Yes, somehow rebasing seems to be the solution here, but with git its quite
> common that you push your changes to your own remote and then do pull
> requests. This again will require me to do push -f almost always, since I
> often change the order of patches. This doesn't sound ideal to me.

I don't understand the objection here.  Yes, if you're changing remote
history (e.g. updating a pull request after you re-ordered patches),
you have to push -f.

What's the problem with this?
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: MOZ_NOT_REACHED and JS_NOT_REACHED are now MOZ_ASSUME_UNREACHABLE, and other changes.

[Justin Lebar]

> #3 from the tldr and #2 from the main post contradict each other, no?

I was out to lunch and didn't realize that I was tossing around two
macro names.  There's only one.

The name we liked was MOZ_ASSUME_UNREACHABLE; MOZ_ASSUME_NOT_REACHED
won't exist in a minute, after I push.  There's no difference between
them.

Sorry for the confusion!

On Fri, Jun 28, 2013 at 7:07 PM, Kyle Huey  wrote:
> On Fri, Jun 28, 2013 at 7:04 PM, Justin Lebar 
> wrote:
>>
>> tl;dr - Changes from bug 820686:
>>
>>  1. We renamed MOZ_NOT_REACHED to MOZ_ASSUME_UNREACHABLE.
>>  2. In Gecko, please use MOZ_CRASH instead of MOZ_NOT_REACHED unless you
>> care
>> about code size or perf.
>>  3. In JS, we removed JS_NOT_REACHED in favor of MOZ_ASSUME_UNREACHABLE.
>>  4. Don't put code after MOZ_CRASH() or MOZ_ASSUME_UNREACHABLE(); it just
>> gives
>> a false sense of security.
>>
>> Dear all,
>>
>> Assuming bug 820686 sticks, we've made the following changes to the
>> MOZ_NOT_REACHED macro.
>>
>> 1) MOZ_NOT_REACHED is now called MOZ_ASSUME_UNREACHABLE.  It still does
>> the
>> same thing it used to: It informs the compiler that the given line cannot
>> be
>> reached.  If execution does reach that line, the program's behavior is
>> undefined.  (The program will likely crash, but maybe not.)
>>
>> 2) JS_NOT_REACHED is now replaced by MOZ_NOT_REACHED.  There's no change
>> in
>> behavior.
>>
>> 3) In Gecko, we replaced all MOZ_NOT_REACHED's with MOZ_CRASH.  This /is/
>> a
>> change in behavior: We replaced undefined behavior (whatever the compiler
>> felt
>> like doing) with defined behavior (a crash).
>>
>> I hope this change sets a precedent that Gecko hackers should use
>> MOZ_ASSUME_UNREACHABLE() only where they care about code size or
>> performance.
>> In general, I think it's better to have defined behavior, which you can
>> get
>> with MOZ_CRASH() (crash in all builds) or MOZ_ASSERT(false) (crash in
>> debug
>> builds only).
>>
>> For example, the new precedent I hope to set is that we should use
>> MOZ_CRASH()
>> or MOZ_ASSERT(false) in the default branch of a switch statement, except
>> where
>> we care about size or perf.  Adding in even a small chance of undefined
>> behavior when we don't care about size or perf is just a premature
>> optimization, IMO.
>>
>> JS hackers can continue using MOZ_ASSUME_UNREACHABLE as they have been.
>>
>> 4) We removed code which comes after MOZ_CRASH() AND
>> MOZ_ASSUME_UNREACHABLE().
>> Code like
>>
>>   MOZ_ASSUME_UNREACHABLE();
>>   return false;
>>
>> just gives us a false sense of security; there is no guarantee that the
>> |return
>> false| will be hit.  It's not necessary to add a return statement to
>> placate
>> the compiler; all of the compilers we care about understand that
>> MOZ_ASSUME_UNREACHABLE() and MOZ_CRASH() are noreturn.
>>
>> Happy hacking,
>> -Justin
>> ___
>> dev-platform mailing list
>> dev-platform@lists.mozilla.org
>> https://lists.mozilla.org/listinfo/dev-platform
>
>
> #3 from the tldr and #2 from the main post contradict each other, no?
>
> - Kyle
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

MOZ_NOT_REACHED and JS_NOT_REACHED are now MOZ_ASSUME_UNREACHABLE, and other changes.

[Justin Lebar]

tl;dr - Changes from bug 820686:

 1. We renamed MOZ_NOT_REACHED to MOZ_ASSUME_UNREACHABLE.
 2. In Gecko, please use MOZ_CRASH instead of MOZ_NOT_REACHED unless you care
about code size or perf.
 3. In JS, we removed JS_NOT_REACHED in favor of MOZ_ASSUME_UNREACHABLE.
 4. Don't put code after MOZ_CRASH() or MOZ_ASSUME_UNREACHABLE(); it just gives
a false sense of security.

Dear all,

Assuming bug 820686 sticks, we've made the following changes to the
MOZ_NOT_REACHED macro.

1) MOZ_NOT_REACHED is now called MOZ_ASSUME_UNREACHABLE.  It still does the
same thing it used to: It informs the compiler that the given line cannot be
reached.  If execution does reach that line, the program's behavior is
undefined.  (The program will likely crash, but maybe not.)

2) JS_NOT_REACHED is now replaced by MOZ_NOT_REACHED.  There's no change in
behavior.

3) In Gecko, we replaced all MOZ_NOT_REACHED's with MOZ_CRASH.  This /is/ a
change in behavior: We replaced undefined behavior (whatever the compiler felt
like doing) with defined behavior (a crash).

I hope this change sets a precedent that Gecko hackers should use
MOZ_ASSUME_UNREACHABLE() only where they care about code size or performance.
In general, I think it's better to have defined behavior, which you can get
with MOZ_CRASH() (crash in all builds) or MOZ_ASSERT(false) (crash in debug
builds only).

For example, the new precedent I hope to set is that we should use MOZ_CRASH()
or MOZ_ASSERT(false) in the default branch of a switch statement, except where
we care about size or perf.  Adding in even a small chance of undefined
behavior when we don't care about size or perf is just a premature
optimization, IMO.

JS hackers can continue using MOZ_ASSUME_UNREACHABLE as they have been.

4) We removed code which comes after MOZ_CRASH() AND MOZ_ASSUME_UNREACHABLE().
Code like

  MOZ_ASSUME_UNREACHABLE();
  return false;

just gives us a false sense of security; there is no guarantee that the |return
false| will be hit.  It's not necessary to add a return statement to placate
the compiler; all of the compilers we care about understand that
MOZ_ASSUME_UNREACHABLE() and MOZ_CRASH() are noreturn.

Happy hacking,
-Justin
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Code coverage take 2, and other code hygiene tools

[Justin Lebar]

Under what circumstances would you expect the code coverage build to break
but all our other builds to remain green?
On Jun 24, 2013 6:51 PM, "Clint Talbert"  wrote:

> Decoder and Jcranmer got code coverage working on Try[1]. They'd like to
> expand this into something that runs automatically, generating results over
> time so that we can actually know what our code coverage status is with our
> major run-on-checkin test harnesses.  While both Joduinn and I are happy to
> turn this on, we have been down this road before. We got code coverage
> stood up in 2008, ran it for a while, but when it became unusable and fell
> apart, we were left with no options but to turn it off.
>
> Now, Jcranmer and Decoder's work is of far higher quality than that old
> run, but before we invest the work in automating it, I want to know if this
> is going to be useful and whether or not I can depend on the community of
> platform developers to address inevitable issues where some checkin,
> somewhere breaks the code coverage build.  Do we have your support?  Will
> you find the generated data useful?  I know I certainly would, but I need
> more buy-in than that (I can just use try if I'm the only one concerned
> about it). Let me know your thoughts on measuring code coverage and owning
> breakages to the code coverage builds.
>
> Also, what do people think about standing up JSLint as well (in a separate
> automation job)?  We should treat these as two entirely separate things,
> but if that would be useful, we can look into that as well.  We can
> configure the rules around JSLint to be amenable to our practices and
> simply enforce against specific errors we don't want in our JS code.  If
> the JS style flamewars start-up, I'll split this question into its own
> thread because they are irrelevant to my objective here.  I want to know if
> it would be useful to have something like this for JS or not.  If we do
> decide to use something like JSLint, then I will be happy to facilitate
> JS-Style flamewars because they will then be relevant to defining what we
> want Lint to do but until that decision is made, let's hold them in check.
>
> So, the key things I want to know:
> * Will you support code coverage? Would it be useful to your work to have
> a regularly scheduled code coverage build & test run?
> * Would you want to additionally consider using something like JS-Lint for
> our codebase?
>
> Let me know,
>
> Clint
>
> [1] https://developer.mozilla.org/**en-US/docs/Measuring_Code_**
> Coverage_on_Firefox
> __**_
> dev-platform mailing list
> dev-platform@lists.mozilla.org
> https://lists.mozilla.org/**listinfo/dev-platform
>
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Embracing git usage for Firefox/Gecko development?

[Justin Lebar]

> On a different gear, would we want to keep m-c/m-i/projects etc. as separate 
> git repos, or could we make them branches in a merged repo?

Many hours of sleep have been lost fighting over the answer to this
question -- the debate lasted for almost a year., with strong
advocates on both sides.  We've pretty solidly decided that it's
better to have just one git repo.  Without rehashing a yearlong
discussion: One repo is more git-like.

If releng wants to have a mirror of the main repo that contains only
certain branches, that's fine.

> Also, note that a historical weakness of our hg usage has been onboarding 
> education.

I've been working on-and-off on a "git for mozilla hackers" blog post,
and I hear that gps is too.

That's not a magic wand, but it's something.

On Tue, Jun 11, 2013 at 6:32 PM, Steve Fink  wrote:
> For me, the most appealing reason to switch to git is so that we can get
> back to having a single revision control system, but I think the benefit of
> that is probably outweighed by the churn. We'd need to rewrite a lot of
> stuff simultaneously, more or less.
>
> As for supporting both, I have a hard time forming an opinion on it without
> having a clue what the cost would be. Clearly, it would involve a lot of
> effort to support git's idiosyncrasies and failure modes (even if git ends
> up being better overall for our needs than hg.) What's the opportunity cost?
> What would get stalled, postponed, or scrapped? And what status quo-ish
> alternative are we talking about, given that we have to gradually support
> git in more and more places due to existing usage and external dependencies?
>
> In short, it's hard to have an opinion about a decision when it's unclear
> what the decision means. Is it that git becomes "tier 1" and anything you
> cannot do with git that you can do with hg automatically becomes a blocker?
> Or is it just a request "please prioritize things that will improve git
> support a little higher than you have in the past"? You've laid out some
> concrete milestones in your original email, but I don't know how those
> milestones would be prioritized relative to the one or two other little
> things that releng and IT are working on already.
>
> On a different gear, would we want to keep m-c/m-i/projects etc. as separate
> git repos, or could we make them branches in a merged repo? Separate repos
> feel like a mercurial-induced nuisance to me. (I'd prefer
> aurora/beta/release to be in the same repo too, not that I truly understand
> the repercussions.)
>
> Also, note that a historical weakness of our hg usage has been onboarding
> education. Adding git as an alternative would improve this for people
> already accustomed to git, but would make the situation worse for anyone who
> isn't already comfortable with the more advanced usage of either git or
> mercurial. (I assert that nontrivial git is harder to learn, and we're
> already not doing a good job with nontrivial hg education.) It's somewhat
> tangential, but it would be cool if somebody could wave a magic wand and
> address education at the same time -- if only to prevent a bad situation
> from becoming worse.
>
>
> ___
> dev-platform mailing list
> dev-platform@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-platform
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Need for unit testing changing the code?

[Justin Lebar]

> For example, a public method (which we want to test in the unit test) has a 
> number of side effects, but we don't have the public accessors to examine all 
> of those private side effects/state.

I had this problem with the B2G process priority tests.

>From a mochitest, I wanted to create a variety of circumstances which
would cause a side-effect in hal.  But hal doesn't keep track of this
state -- it sends it off to the OS and forgets it -- and it provides
no way of reading the state.

What I did was create a pref which, when set, causes my code to fire
observer notifications when it calls into hal.  Then I can listen to
these notifications from my mochitest.  Effectively, we're "mocking"
the hal interface.

I'm not sure this is a good solution everywhere, but it has worked
well for my needs in this case.

The tests are in dom/browser-element/mochitest/priority, and the code
is in ProcessPriorityManager.cpp (search for "testonly"), if you want
to have a look.

-Justin

> Add public accessors, even if they're (currently) only used by the unit tests.
> Make the private members protected, derive a class inside the unit test code 
> and put the public accessors there.
> Make the unit test class a friend of the original class so that it can access 
> private members.
> Extend the signature of the method in question to allow the "return" of all 
> the side effects.
> Take the fact that you can't see side effects as a sign of bad design, 
> because that means the non-unit test code cannot check if the right thing 
> happened. (I imagine that would lead to #1 or #4 or some combination of 
> those.)
>
> Thoughts or precedents?
>
> Milan
>
> ___
> dev-platform mailing list
> dev-platform@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-platform
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Code Review Session

[Justin Lebar]

This is only tangentially on topic, but I have a git pre-commit hook
which detects .orig files and trailing whitespace.  It's saved me a
lot of embarrassment.

I also have a git tool which will fix trailing whitespace in your patch.

https://github.com/jlebar/moz-git-tools#pre-commit
https://github.com/jlebar/moz-git-tools#git-fix-whitespace

On Mon, May 27, 2013 at 10:54 PM, Anthony Jones  wrote:
> On 25/05/13 04:16, Ehsan Akhgari wrote:
>> On 2013-05-24 11:46 AM, Benoit Girard wrote:
>> Another option is to use clang-format, which can lexically parse diff
>> files.
>
> A pre-upload check would give the fastest feedback.
>
> It would help me (and those who review my code) if there is an easy way
> to check my patches from the command line before doing hg bzexport. Even
> if it is only for white space. What we need is a way to specify which
> file paths the standard formatting rules apply to. I just need to figure
> out how to install clang-format.
>
> If this proves to be successful then we could consider putting effort
> into other style conventions.
>
> Anthony
>
> ___
> dev-platform mailing list
> dev-platform@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-platform
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Code Review Session

[Justin Lebar]

> How about integrating it into BugzillaJS?

I don't think we should require a Firefox add-on to use our bug
tracker effectively.

We're a web company.  We make a web browser.  Let's write a webpage.

If a browser extension is the only way we can provide a good user
experience in our bug tracker and review tool, we have surely failed.

On Fri, May 24, 2013 at 10:59 AM, Mike de Boer  wrote:
> How about integrating it into BugzillaJS?
>
> I'm working on it quite a lot now
> (https://github.com/gkoberger/BugzillaJS/pull/77 and
> https://github.com/gkoberger/omnium/pull/3) to make some improvements to
> Bugzilla 'core'.
>
> I think an add-on that eventually is good enough to use for Mozillians would
> be very beneficial. I think it might even go as far as the add-on being the
> frontend and Bugzilla itself being regarded as the backend/ datastore.
>
> Regardless, doing Github integration in BugzillaJS should be trivial.
>
> Mike.
>
> On May 24, 2013, at 4:50 PM, Justin Lebar  wrote:
>
> * I think we should experiment (again) with real pull-request integration
> into bugzilla.
>
>
> I'm totally in favor of better tools and real pull requests, and of
> course the PRs need to be linked to bugzilla /somehow/.
>
> But I want to qualify "integration into bugzilla": I explicitly do not
> want a tool that is tightly coupled to bugzilla.  In fact, I want a
> tool that has as little to do with bugzilla as feasible.
>
> I mean no disrespect to our bugzilla maintainers, who have an
> impossible and largely thankless job, but bugzilla has so much baggage
> from the '90s, my experience is that it ruins everything it touches.
> Consider for example how much better harthur's fileit and dashboard
> tools [1] [2] are than bugzilla's built-in equivalents.
>
> We shouldn't conflate owning the PR data with integrating the PR tool
> into bugzilla.  If we do, we risk ending up with yet another crappy
> non-solution to a real problem (see bugzilla interdiff, splinter
> integration, and so on).
>
> -Justin
>
> [1] http://harthur.github.com/fileit/
> [2] http://harthur.github.io/bugzilla-todos/
> ___
> dev-platform mailing list
> dev-platform@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-platform
>
>
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Code Review Session

[Justin Lebar]

> * I think we should experiment (again) with real pull-request integration
> into bugzilla.

I'm totally in favor of better tools and real pull requests, and of
course the PRs need to be linked to bugzilla /somehow/.

But I want to qualify "integration into bugzilla": I explicitly do not
want a tool that is tightly coupled to bugzilla.  In fact, I want a
tool that has as little to do with bugzilla as feasible.

I mean no disrespect to our bugzilla maintainers, who have an
impossible and largely thankless job, but bugzilla has so much baggage
from the '90s, my experience is that it ruins everything it touches.
Consider for example how much better harthur's fileit and dashboard
tools [1] [2] are than bugzilla's built-in equivalents.

We shouldn't conflate owning the PR data with integrating the PR tool
into bugzilla.  If we do, we risk ending up with yet another crappy
non-solution to a real problem (see bugzilla interdiff, splinter
integration, and so on).

-Justin

[1] http://harthur.github.com/fileit/
[2] http://harthur.github.io/bugzilla-todos/
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Ordering shutdown observers?

[Justin Lebar]

> I believe roc proposed just having an explicit hard coded list of things
> to start up a while ago, and I'm tempted to say that's what we should do for
> shutdown too.  So just add an explicit call to some os.file thing
> followd by a call to a telemetry function after profile-before-change
> but not after anything else.

Indeed, to implement ClearOnShutdown, I added an explicit call to
KillClearOnShutdown during the XPCOM shutdown process, precisely
because I needed to guarantee a particular ordering wrt shutdown
observers.
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Help with migrating to clang 3.3

[Justin Lebar]

If we wanted to check out the RC from svn, is that trunk, or is it one
of the branches?  It's not really clear how the SVN repo is set up, or
how to check out the right rev for llvm + clang.

http://llvm.org/svn/llvm-project/llvm
http://llvm.org/svn/llvm-project/cfe

On Fri, May 10, 2013 at 12:51 PM, Ehsan Akhgari  wrote:
> Hi all,
>
> clang 3.3 is in RC stage right now, and it would be really great if we can
> start testing it to ensure that we can migrate our Mac builds to 3.3. once
> it's released, and also that clang 3.3 doesn't contain any regressions that
> will prevent us from building Firefox with it.
>
> Rail has prepared the required manifests in bug 870173, but we currently
> don't have anybody who can push it to try and look at test results and
> performance numbers, etc.  It would be great if somebody could volunteer to
> do that.
>
> Please get in touch with me or Rail about this.
>
> Thanks!
> Ehsan
> ___
> dev-platform mailing list
> dev-platform@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-platform
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: We should drop MathML

[Justin Lebar]

Four points here.

1. We're assuming that MathJax is as good with MathML as it is without
it, but perhaps we could ask the MathJax folks to comment on whether
this is true.  I'd certainly be a lot more comfortable dropping MathML
if the MathJax folks said there was no point.

2.

> A suitable subset of TeX (not
> the entirety of TeX, as that is a huge, single-implementation technology
> that reputedly only Knuth ever fully understood) was the right choice all
> along

Jonathan Kew is a much better person to comment on this, but in my
relatively limited experience typesetting documents in TeX, I've had
to use various LaTeX packages (particularly amsmath and amssymb) in
order to get all of the symbols and so on that I needed.  I suspect
that "heavy" users of TeX frequently need more than these two
packages.

The point being, "a subset of TeX" isn't necessarily sufficient.

3. It's not clear to me why we should go through all the work of
rewriting MathML into this TeX thing unless we thought that the new
thing would see more enthusiastic adoption.  It sounds like you would
probably agree on this point.

4.

> 2.2. High-quality mathematical typography in browsers is now possible,
> without using MathML. Examples include MathJax ( http://www.mathjax.org/ ),
> which happily takes either TeX or MathML input and renders it without
> specific browser support, and of course PDF.js which is theoretically able
> to render all PDFs including those generated by pdftex. Both approaches
> give far higher quality output than what any current MathML browser
> implementation offers.

Could you elaborate on how MathML is inferior to MathJax's HTML+CSS
rendering?  MathJax has a page where you can switch between different
rendering modes, and to my eyes, the two modes are almost identical.
The only difference I see is that the HTML+CSS mode is better at
correctly sizing large parentheses and radicals, but I wouldn't call
this "far higher quality."

http://www.mathjax.org/demos/mathml-samples/

On Sun, May 5, 2013 at 11:38 AM, Benoit Jacob  wrote:
> Hi,
>
> Summary: MathML is a vestigial remnant of the XML-everything era, and we
> should drop it.
>
> ***
>
> 1. Reasons why I believe that MathML never was a good idea. Summary:
> over-specialized and uniformly inferior to the pre-existing,
> well-established standard, TeX.
>
>1.1. MathML is too specialized: we should be reluctant to have a
> separate spec for every kind of specialized typography. What if musicians
> wanted their own MusicML too?
>
>1.2. MathML reinvents the wheel, poorly. A suitable subset of TeX (not
> the entirety of TeX, as that is a huge, single-implementation technology
> that reputedly only Knuth ever fully understood) was the right choice all
> along, because:
>
>   1.2.1. TeX is already the universally adopted standard --- and
> already was long before MathML was invented. Check for yourself on
> http://arxiv.org/ , where most new math papers are uploaded --- pick any
> article, then "other" formats, then "Source": you can then download TeX
> sources for almost every article.
>
>   1.2.2. TeX is very friendly to manual writing, being concise and
> close to natural notation, with limited overhead (some backslashes and
> curly braces), while MathML is as tedious to handwrite as any other
> XML-based format. An example is worked out at
> http://en.wikipedia.org/wiki/MathML#Example_and_comparison_to_other_formats,
> where the solution to the quadratic equation is one line of TeX versus
> 30
> lines of MathML!
>
>   1.2.3. An important corollary of being very close to natural notation
> is that TeX can be nearly trivially "read aloud". That means that it offers
> a particularly easy accessibility story. No matter what mechanism is used
> to graphically display equations, providing the TeX source (similarly to
> images alt text) would allow anyone to quickly read it themselves without
> any kind of software support; and screen reading software could properly
> read equations with minimal TeX-specific support code. For example, TeX
> code such as "\int_0^1 x^2 dx" can be readily understood by any human with
> basic TeX exposure (which is nearly 100% of mathematicians) and can be
> easily handled by any screen reader that knows that \int should be read as
> "integral" and that immediately after it, _ and ^ should be read as "from"
> and "to" respectively.
>
> ***
>
> 2. Reasons why even if MathML had ever been a decent idea, now would be the
> right time to drop it. Summary: never really got traction, and the same
> rendering can now be achieved without MathML support.
>
>2.1. MathML never saw much traction outside of Mozilla, despite having
> been around for a decade. WebKit only got a very limited partial
> implementation recently, and Google removed it from Blink. The fact that it
> was just dropped from Blink says much about how little it's used: Google
> wouldn't have disabled a feature that's needed to render web pages in the
> real w

PSA: make -C toolkit/library does not currently work on trunk; use make libs -C toolkit/library instead

[Justin Lebar]

See https://bugzilla.mozilla.org/show_bug.cgi?id=809430#c39 for details.

As roc points out, this has broken |mach build |.  Stay tuned in
the bug if you're interested in whether we resolve this by backing out
the change or fixing mach.

-Justin
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Proposal for an inbound2 branch

[Justin Lebar]

> Given the whole point of this thread is about how unreliable inbound is, why
> are people trying to develop against it?

You still need a copy of inbound to rebase your patches against when pushing.

Whatever your personal opinions about git happen to be, I don't think
a "git doesn't need a copy of inbound" approach is going to fly.
We're simply too invested in git at this point to be OK adding more
circumstances under which we have to say "well, if you want to do
/that/, you need to use hg."
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Proposal for an inbound2 branch

[Justin Lebar]

> Is there sanity to this proposal or am I still crazy?

If we had a lot more project branches, wouldn't that increase the load
on infra dramatically, because we'd have less coalescing?

This is of course a solvable problem, but the fact that the problem
exists suggests to me that your proposal here should be considered
separately from RyanVM's tightly-scoped proposal here.

On Tue, Apr 30, 2013 at 2:46 AM, Gregory Szorc  wrote:
> On 4/26/2013 12:17 PM, Ryan VanderMeulen wrote:
>> Specific goals:
>> -Offer an alternative branch for developers to push to during extended
>> inbound closures
>> -Avoid patch pile-up after inbound re-opens from a long closure
>>
>> Specific non-goals:
>> -Reducing infrastructure load
>> -Changing pushing strategies from the widely-accepted status quo (i.e.
>> multi-headed approach)
>> -Creating multiple integration branches that allow for simultaneous
>> pushing (i.e. inbound-b2g, inbound-gfx, etc)
>>
>> My proposal:
>> -Create an inbound2 branch identically configured to mozilla-inbound.
>> -Under normal circumstances (i.e. m-i open), inbound2 will be CLOSED.
>> -In the event of a long tree closure, the last green changeset from
>> m-i will be merged to inbound2 and inbound2 will be opened for checkins.
>> ---It will be a judgment call for sheriffs as to how long of a closure
>> will suffice for opening inbound2.
>> -When the bustage on m-i is resolved and it is again passing tests,
>> inbound2 will be closed again.
>> -When all pending jobs on inbound2 are completed, it will be merged to
>> m-i.
>> -Except under extraordinary circumstances, all merges to
>> mozilla-central will continue to come from m-i ONLY.
>> -If bustage lands on inbound2, then both trees will be closed until
>> resolved. Tough. We apparently can't always have nice things.
>
> If you consider that every repository is essentially a clone of
> mozilla-central, what we have *now* is effectively equivalent to a
> single repository with multiple heads/branches/bookmarks. However, the
> different heads/branches/bookmarks differ in:
>
> * How much attention sheriffs give them.
> * The automation configuration (coalescing, priority, etc).
> * Policies around landing.
> * How developers use it.
>
> These are all knobs in our control.
>
> When we say "create an inbound2," we're essentially establishing a new
> head/branch/bookmark that behaves much like "inbound1" with a slightly
> different landing policy. If that's what we want to do, sure. I think
> it's better than a single, frequently closed inbound.
>
> Anyway, no matter how much I think about this proposal, I keep coming
> back to the question of "why don't we use project branches more?"
> Instead of everyone and her brother landing on inbound, what if more
> landings were performed on {fx-team, services-central,  twig>, etc}? I /think/ the worst that can happen is merge conflicts and
> bit rot. And, we can abate that through intelligent grouping of related
> commits in the same repository, frequent merges, and maybe even better
> communication (perhaps even automatically with tools that alert
> developers to potential conflicts - wouldn't it be cool if you updated a
> patch and Mercurial was like "o hai - Ehsan recently pushed a Try push
> that conflicts with your change: you two should talk.").
>
> As a counter-proposal, I propose that we start shifting landings to
> project branches/twigs. We should aim for a small and well-defined set
> of repositories (say 3 to 5) sharing similar automation configuration
> and sheriff love. By keeping the number small, it's easy to figure out
> where something should land and it's not too much of an extra burden on
> sheriffs. We can still keep inbound, but it should only be reserved for
> major, cross-repository landings with multi-module impact (e.g. build
> system changes), merges from the main landing repositories (unless we
> merge straight to central), and possibly as a backup in case one of the
> landing repositories is closed.
>
> We can test this today with very little effort: we figure out how to
> bucket commits, re-purpose existing repositories/twigs, and see what
> happens. If it works: great - we've just validated that distributed
> version control works for Firefox development (as opposed to the
> CVS/Subversion workflow we're currently using with inbound). If not, we
> can try variations and/or the inbound2 idea.
>
> Is there sanity to this proposal or am I still crazy?
> ___
> dev-platform mailing list
> dev-platform@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-platform
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Proposal for an inbound2 branch

[Justin Lebar]

> 3/ Being a git guy, I prefer having a try-like server where you don't get
> push contention or closed tree, because we are creating a new head
> every-time, and let the sheriffs cherry-pick the good changes which are not
> source of conflicts.  And ask developers to rebase their changes otherwise.

I prefer this too, and I've proposed trying this.

But kats convinced me that this isn't a decent option right now,
because this would disable a lot of coalescing, and we don't have the
capacity to handle this.  It's possible that we'd have enough
capacity, but end up shifting load to the time when the sun is over
the Pacific ocean, but even that isn't great, since it significantly
increases the turn-around time for landing patches.

Anyway, RyanVM wanted us to limit discussion here to a yes/no vote on
his proposal, and I think we should respect that.
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Proposal for an inbound2 branch

[Justin Lebar]

> If I have a patch ready to land when inbound closes, what would be the
> sequence of steps that I need to do to land it on inbound2?  Would I
> need to have an up-to-date inbound2 clone and transplant the patch
> across?

Yes, I think so.

I think mbrubeck or someone knows how to maintain multiple hg branches
in one repo, but I've never figured that out...
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Proposal for an inbound2 branch

[Justin Lebar]

I like that inbound2 would be open only when inbound is closed.  That
way you don't have to make a decision wrt which tree to push to.

sgtm.

On Fri, Apr 26, 2013 at 3:17 PM, Ryan VanderMeulen  wrote:
> As has been discussed at length in the various infrastructure meetings, one
> common point of frustration for developers is frequent tree closures due to
> bustage on inbound. While there are other issues and ideas for how to
> improve the inbound bustage situation, one problem I'm seeing is that
> multiple different issues with different solutions are being lumped together
> into one discussion, which makes it hard to gain traction on getting
> anything done. For that reason, I would like to specifically separate out
> the specific issue of inbound closures negatively affecting developer
> productivity and offer a more fleshed-out solution that can be implemented
> now independent of any other ideas on the table.
>
> Specific goals:
> -Offer an alternative branch for developers to push to during extended
> inbound closures
> -Avoid patch pile-up after inbound re-opens from a long closure
>
> Specific non-goals:
> -Reducing infrastructure load
> -Changing pushing strategies from the widely-accepted status quo (i.e.
> multi-headed approach)
> -Creating multiple integration branches that allow for simultaneous pushing
> (i.e. inbound-b2g, inbound-gfx, etc)
>
> My proposal:
> -Create an inbound2 branch identically configured to mozilla-inbound.
> -Under normal circumstances (i.e. m-i open), inbound2 will be CLOSED.
> -In the event of a long tree closure, the last green changeset from m-i will
> be merged to inbound2 and inbound2 will be opened for checkins.
> ---It will be a judgment call for sheriffs as to how long of a closure will
> suffice for opening inbound2.
> -When the bustage on m-i is resolved and it is again passing tests, inbound2
> will be closed again.
> -When all pending jobs on inbound2 are completed, it will be merged to m-i.
> -Except under extraordinary circumstances, all merges to mozilla-central
> will continue to come from m-i ONLY.
> -If bustage lands on inbound2, then both trees will be closed until
> resolved. Tough. We apparently can't always have nice things.
>
> As stated above, I believe that this will solve one of the biggest
> painpoints of long tree closures without adding tons of extra complexity to
> what we're already doing and what developers are used to. The affect on
> infrastructure load should be close to neutral since at any given time,
> patches will only be getting checked into one branch. This proposal also has
> the advantage of being easy to implement since it's simply a clone of an
> existing repo, with a little extra sheriff overhead. It also helps to
> mitigate the pile-up of pushes we tend to see after a long closure which
> increase the likelihood of another long closure in the event of any bustage
> due to a high level of coalescing.
>
> To be clear, this proposal is NOT intended to solve all of the various
> problems that have been raised with respect to infrastructure load, good
> coding practices, bustage minimization, good Try usage, etc. This is only
> looking to reduce the impact such issues have on developer workflow and make
> sheriffing easier after the tree reopens.
>
> Feedback?
> ___
> dev-platform mailing list
> dev-platform@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-platform
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Storage in Gecko

[Justin Lebar]

> The current level of flakiness in the IndexedDB test suite (especially on
> OSX) makes me concerned about what to expect if it starts getting heavier
> use across the various platforms.

Is that just in the OOP tests, or everywhere?
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Improving Mac OS X 10.6 test wait times by reducing 10.7 load

[Justin Lebar]

I'd be happy if we did the 10.7 and 10.6 tests only when we trigger
Windows PGO builds.  That seems like a good compromise here, since the
sheriffs have to wait for PGO coverage on m-i before merging anyway.

On Fri, Apr 26, 2013 at 2:34 PM, Armen Zambrano G.  wrote:
> After re-reading, I'm happy to disable just m-i/try for now.
>
> Modifying to trigger *some* jobs on m-i through would be some decent amount
> of work (adding Mac pgo builders) but still different than normal operations
> and increase the 10.6/10.8 test load.
>
>
> On 2013-04-26 1:31 PM, Justin Lebar wrote:
>>
>> I don't think I'm comfortable disabling this platform across the
>> board, or even disabling debug-only runs across the board.
>>
>> As jmaher pointed out, there are platform differences here.  If we
>> disable this platform entirely, we lose visibility into rare but, we
>> seem to believe, possible events.
>>
>> It seems like the only reason to disable everywhere instead of only on
>> m-i/try (or running less frequently on m-i, like we do with PGO) is
>> that the former is easier to implement.  It seems like we're proposing
>> taking a lot of risk here to work around our own failings...
>>
>> On Fri, Apr 26, 2013 at 1:03 PM, Armen Zambrano G. 
>> wrote:
>>>
>>>
>>> On 2013-04-26 12:14 PM, Justin Lebar wrote:
>>>>>
>>>>>
>>>>> Would we be able to go back to where we disabled 10.7 altogether?
>>>>
>>>>
>>>>
>>>> On m-i and try only, or everywhere?
>>>
>>>
>>>
>>> The initial proposal was for disabling everywhere.
>>>
>>> We could leave 10.7 opt jobs running everywhere as a compromise and
>>> re-visit
>>> after I re-purpose the first batch of machines.
>>>
>>> best regards,
>>> Armen
>>>
>>>
>>>>
>>>> On Fri, Apr 26, 2013 at 12:10 PM, Armen Zambrano G.
>>>> 
>>>> wrote:
>>>>>
>>>>>
>>>>> Just disabling debug and talos jobs for 10.7 should reduce more than
>>>>> 50%
>>>>> of
>>>>> the load on 10.7. That might be sufficient for now.
>>>>>
>>>>> Any objections on this plan?
>>>>> We can re-visit later on if we need more disabled.
>>>>>
>>>>> cheers,
>>>>> Armen
>>>>>
>>>>>
>>>>> On 2013-04-26 11:50 AM, Armen Zambrano G. wrote:
>>>>>>
>>>>>>
>>>>>>
>>>>>> Would we be able to go back to where we disabled 10.7 altogether?
>>>>>> Product (Asa in separate thread) and release drivers (Akeybl) were OK
>>>>>> to
>>>>>> the compromise of version specific test coverage being removed
>>>>>> completely.
>>>>>>
>>>>>> Side note: adding Mac PGO would increase the build load (Besides this
>>>>>> we
>>>>>> have to do a large PO as we expect Mac wait times to be showing up as
>>>>>> general load increases).
>>>>>>
>>>>>> Not all reducing load approaches are easy to implement (due to the way
>>>>>> that buildbot is designed) and it does not ensure that we would reduce
>>>>>> it enough. It's expensive enough to support 3 different versions of
>>>>>> Mac
>>>>>> as is without bringing 10.9 into the table. We have to cut things at
>>>>>> times.
>>>>>>
>>>>>> One compromise that would be easy to implement and *might* reduce the
>>>>>> load is to disable all debug jobs for 10.7.
>>>>>>
>>>>>> cheers,
>>>>>> Armen
>>>>>>
>>>>>> On 2013-04-26 11:29 AM, Justin Lebar wrote:
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> As a compromise, how hard would it be to run the Mac 10.6 and 10.7
>>>>>>> tests on m-i occasionally, like we run the PGO tests?  (Maybe we
>>>>>>> could
>>>>>>> trigger them on the same csets as we run PGO; it seems like that
>>>>>>> would
>>>>>>> be useful.)
>>>>>>>
>>>>>>> On Fri, Apr 26, 2013 at 11:19 AM, Ryan VanderMeulen
>>>>>>> 
>>>>

Re: Improving Mac OS X 10.6 test wait times by reducing 10.7 load

[Justin Lebar]

I don't think I'm comfortable disabling this platform across the
board, or even disabling debug-only runs across the board.

As jmaher pointed out, there are platform differences here.  If we
disable this platform entirely, we lose visibility into rare but, we
seem to believe, possible events.

It seems like the only reason to disable everywhere instead of only on
m-i/try (or running less frequently on m-i, like we do with PGO) is
that the former is easier to implement.  It seems like we're proposing
taking a lot of risk here to work around our own failings...

On Fri, Apr 26, 2013 at 1:03 PM, Armen Zambrano G.  wrote:
>
> On 2013-04-26 12:14 PM, Justin Lebar wrote:
>>>
>>> Would we be able to go back to where we disabled 10.7 altogether?
>>
>>
>> On m-i and try only, or everywhere?
>
>
> The initial proposal was for disabling everywhere.
>
> We could leave 10.7 opt jobs running everywhere as a compromise and re-visit
> after I re-purpose the first batch of machines.
>
> best regards,
> Armen
>
>
>>
>> On Fri, Apr 26, 2013 at 12:10 PM, Armen Zambrano G. 
>> wrote:
>>>
>>> Just disabling debug and talos jobs for 10.7 should reduce more than 50%
>>> of
>>> the load on 10.7. That might be sufficient for now.
>>>
>>> Any objections on this plan?
>>> We can re-visit later on if we need more disabled.
>>>
>>> cheers,
>>> Armen
>>>
>>>
>>> On 2013-04-26 11:50 AM, Armen Zambrano G. wrote:
>>>>
>>>>
>>>> Would we be able to go back to where we disabled 10.7 altogether?
>>>> Product (Asa in separate thread) and release drivers (Akeybl) were OK to
>>>> the compromise of version specific test coverage being removed
>>>> completely.
>>>>
>>>> Side note: adding Mac PGO would increase the build load (Besides this we
>>>> have to do a large PO as we expect Mac wait times to be showing up as
>>>> general load increases).
>>>>
>>>> Not all reducing load approaches are easy to implement (due to the way
>>>> that buildbot is designed) and it does not ensure that we would reduce
>>>> it enough. It's expensive enough to support 3 different versions of Mac
>>>> as is without bringing 10.9 into the table. We have to cut things at
>>>> times.
>>>>
>>>> One compromise that would be easy to implement and *might* reduce the
>>>> load is to disable all debug jobs for 10.7.
>>>>
>>>> cheers,
>>>> Armen
>>>>
>>>> On 2013-04-26 11:29 AM, Justin Lebar wrote:
>>>>>
>>>>>
>>>>> As a compromise, how hard would it be to run the Mac 10.6 and 10.7
>>>>> tests on m-i occasionally, like we run the PGO tests?  (Maybe we could
>>>>> trigger them on the same csets as we run PGO; it seems like that would
>>>>> be useful.)
>>>>>
>>>>> On Fri, Apr 26, 2013 at 11:19 AM, Ryan VanderMeulen 
>>>>> wrote:
>>>>>>
>>>>>>
>>>>>> On 4/26/2013 11:11 AM, Justin Lebar wrote:
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> So what we're saying is that we are going to completely reverse our
>>>>>>>> previous tree management policy?
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Basically, yes.
>>>>>>>
>>>>>>> Although, due to coalescing, do you always have a full run of tests
>>>>>>> on
>>>>>>> the tip of m-i before merging to m-c?
>>>>>>>
>>>>>>
>>>>>> Yes. Note that we generally aren't merging inbound tip to m-c - we're
>>>>>> taking
>>>>>> a known-green cset (including PGO tests).
>>>>>>
>>>>>> ___
>>>>>> dev-platform mailing list
>>>>>> dev-platform@lists.mozilla.org
>>>>>> https://lists.mozilla.org/listinfo/dev-platform
>>>>
>>>>
>>>>
>>>
>>> ___
>>> dev-platform mailing list
>>> dev-platform@lists.mozilla.org
>>> https://lists.mozilla.org/listinfo/dev-platform
>
>
> ___
> dev-platform mailing list
> dev-platform@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-platform
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Improving Mac OS X 10.6 test wait times by reducing 10.7 load

[Justin Lebar]

> Would we be able to go back to where we disabled 10.7 altogether?

On m-i and try only, or everywhere?

On Fri, Apr 26, 2013 at 12:10 PM, Armen Zambrano G.  wrote:
> Just disabling debug and talos jobs for 10.7 should reduce more than 50% of
> the load on 10.7. That might be sufficient for now.
>
> Any objections on this plan?
> We can re-visit later on if we need more disabled.
>
> cheers,
> Armen
>
>
> On 2013-04-26 11:50 AM, Armen Zambrano G. wrote:
>>
>> Would we be able to go back to where we disabled 10.7 altogether?
>> Product (Asa in separate thread) and release drivers (Akeybl) were OK to
>> the compromise of version specific test coverage being removed completely.
>>
>> Side note: adding Mac PGO would increase the build load (Besides this we
>> have to do a large PO as we expect Mac wait times to be showing up as
>> general load increases).
>>
>> Not all reducing load approaches are easy to implement (due to the way
>> that buildbot is designed) and it does not ensure that we would reduce
>> it enough. It's expensive enough to support 3 different versions of Mac
>> as is without bringing 10.9 into the table. We have to cut things at
>> times.
>>
>> One compromise that would be easy to implement and *might* reduce the
>> load is to disable all debug jobs for 10.7.
>>
>> cheers,
>> Armen
>>
>> On 2013-04-26 11:29 AM, Justin Lebar wrote:
>>>
>>> As a compromise, how hard would it be to run the Mac 10.6 and 10.7
>>> tests on m-i occasionally, like we run the PGO tests?  (Maybe we could
>>> trigger them on the same csets as we run PGO; it seems like that would
>>> be useful.)
>>>
>>> On Fri, Apr 26, 2013 at 11:19 AM, Ryan VanderMeulen 
>>> wrote:
>>>>
>>>> On 4/26/2013 11:11 AM, Justin Lebar wrote:
>>>>>>
>>>>>>
>>>>>> So what we're saying is that we are going to completely reverse our
>>>>>> previous tree management policy?
>>>>>
>>>>>
>>>>>
>>>>> Basically, yes.
>>>>>
>>>>> Although, due to coalescing, do you always have a full run of tests on
>>>>> the tip of m-i before merging to m-c?
>>>>>
>>>>
>>>> Yes. Note that we generally aren't merging inbound tip to m-c - we're
>>>> taking
>>>> a known-green cset (including PGO tests).
>>>>
>>>> ___
>>>> dev-platform mailing list
>>>> dev-platform@lists.mozilla.org
>>>> https://lists.mozilla.org/listinfo/dev-platform
>>
>>
>
> ___
> dev-platform mailing list
> dev-platform@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-platform
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Improving Mac OS X 10.6 test wait times by reducing 10.7 load

[Justin Lebar]

As a compromise, how hard would it be to run the Mac 10.6 and 10.7
tests on m-i occasionally, like we run the PGO tests?  (Maybe we could
trigger them on the same csets as we run PGO; it seems like that would
be useful.)

On Fri, Apr 26, 2013 at 11:19 AM, Ryan VanderMeulen  wrote:
> On 4/26/2013 11:11 AM, Justin Lebar wrote:
>>>
>>> So what we're saying is that we are going to completely reverse our
>>> previous tree management policy?
>>
>>
>> Basically, yes.
>>
>> Although, due to coalescing, do you always have a full run of tests on
>> the tip of m-i before merging to m-c?
>>
>
> Yes. Note that we generally aren't merging inbound tip to m-c - we're taking
> a known-green cset (including PGO tests).
>
> ___
> dev-platform mailing list
> dev-platform@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-platform
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Improving Mac OS X 10.6 test wait times by reducing 10.7 load

[Justin Lebar]

> So what we're saying is that we are going to completely reverse our
> previous tree management policy?

Basically, yes.

Although, due to coalescing, do you always have a full run of tests on
the tip of m-i before merging to m-c?

A better solution would be to let you trigger a full set of tests (w/o
coalescing) on m-i before merging to m-c.  We've been asking for a
similar feature for tryserver (let us add new jobs to my push) for a
long time.  Perhaps if we made this change, we could get releng to
implement that feature sooner rather than later, particularly if this
change caused pain to other teams who pull from a broken m-c.

I am not above effecting a sense of urgency in order to get bugs fixed.  :)

> Currently, m-c is supposed to be the tree that's safely unbroken, and we
> know it's unbroken because the tests that we run on it have already been
> run on the tree that merged into it, and you should almost never push
> directly to it unless you're in a desperate hurry to hit a nightly.
>
> This change would mean that we expect to have merges of hundreds of
> csets from inbound sometimes break m-c with no idea which one broke it,
> that we expect to sometimes have permaorange on it for days, and that
> it's better to push your widget/cocoa/ pushes directly to m-c than to
> inbound.
> ___
> dev-platform mailing list
> dev-platform@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-platform
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: packaged apps and origins

[Justin Lebar]

> If apps are served from and signed by the marketplace, then any origin is 
> okay (after
> review.)

I know that we rely on code review for a lot of security assurance
questions, but it seems to me that allowing /any origin/ opens us up
to attacks needlessly.

Could we allow any out of a whitelist of origins specified in the
manifest, instead?

On Thu, Apr 25, 2013 at 8:34 PM, Ben Adida  wrote:
>
> Hi folks,
>
> I want to raise what I believe is a relatively urgent issue with packaged
> apps and web origins:
>
>   https://bugzilla.mozilla.org/show_bug.cgi?id=852720
>
> Currently, packaged apps run in an origin that is newly minted for each
> device installation, effectively a GUID that differs from device to device.
> This works up until the point where the rest of the Web expects a stable
> origin across devices, e.g. OAuth and OpenID flows, and Persona. Since
> origins are so critical to the Web, I expect to see many more failures over
> time.
>
> Can we fix this?
>
> Potch has a great proposal: let apps declare a marketplace in their
> manifest. If apps are served from and signed by the marketplace, then any
> origin is okay (after review.) If apps are self-hosted, then the only origin
> allowed is that of the hosting site.
>
> I suggested a tweak to this: if a packaged app is served from
> https://example.com, then it can set an origin of app://example.com, so that
> it is stable but also different from the actual hosted origin.
>
> Can we converge on a solution here ASAP? This is now holding up making
> Marketplace a packaged app, and I suspect it will bite us again soon.
>
> -Ben
> ___
> dev-platform mailing list
> dev-platform@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-platform
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Increasing the platform meeting's relevance for engineers

[Justin Lebar]

Sorry, I must have misunderstood what you meant.

If all you're saying is that sometimes, it's good to call a meeting to
make a decision, I don't think we disagree.

On Thu, Apr 25, 2013 at 4:56 PM, Milan Sreckovic  wrote:
>
>
> On 2013-04-25, at 2:07 PM, Justin Lebar  wrote:
>
>>> Justin pointed out his earlier post and the apparent disagreement I had 
>>> with it with the
>>> "pick a long thread topic" example - and he has a point.  I meant it as an 
>>> example, and
>>> didn't say as much, and I meant more to focus on decision making, and 
>>> didn't say that
>>> either.
>>
>> I also meant "pick a long newsgroup thread" as an example.  I still
>> disagree with using the platform meeting for decision-making.
>>
>
>
> The reason the whole thread started is because Lawrence is looking for 
> suggestion whether we need changes, and if so, what they are.  I'm pretty 
> sure that a quick vote right now would give us a great majority that thinks 
> that the days of all of platform engineers in a single meeting are long gone.
>
>
>>
>>> However, if a particular topic is announced as "we've exchanged e-mails 
>>> over two weeks
>>> on the topic X and we think we should now make a decision", then only the 
>>> people
>>> interested would show up.
>>
>> The idea that the people in timezones which allow them to attend the
>> meeting are privileged to participate directly in decisions made
>> there, and anyone else who wants to participate in these decisions
>> needs a proxy, is antithetical to how Mozilla works and ought to work,
>> I think.
>
> OK.  Yet we have meetings.
>
>>
>>> So, just by showing up, you're claiming to have interest in the particular 
>>> topic.
>>
>> Well, you're claiming to have interest in /a/ particular topic, which
>> might be just one of many discussed at the meeting.  Even in the event
>> that the agenda is announced ahead of time (good luck with that), you
>> may have to wait through an hour-long meeting just to talk for sixty
>> seconds.
>
> There would be one topic per meeting.  One decision.
>
>>
>> Maybe we're solving a problem here that we don't have.  I think we're
>> perfectly capable of debating and deciding things over e-mail.  Do you
>> disagree, and do you have evidence for this?
>
> Do I have evidence that synchronous communication is useful at times?  Are 
> you really suggesting that all the decision making is always best done 
> asynchronously?
>
>>
>> It seems to me that decisions that need in-depth, live debate are
>> poorly suited for a meeting that nominally involves all of
>> engineering.  I have difficulty fathoming a decision which is simple
>> enough to be decided in a few minutes on a conference call involving
>> hundreds of engineers, but is so complex that we can't decide it
>> effectively over e-mail.
>
> I agree with "all of engineering".  I think that topic is dead.  There is no 
> meeting useful for all of platform engineering, other than the "announcement" 
> ones.
>
>>
>> Perhaps you could provide an example of something that you think would
>> have been decided much better in the paradigm that you're suggesting.
>
> An e-mail topic that takes 20 replies is often a waste of time.  A small 
> number of people was interesting in this particular topic to start, and at 
> this point, an even smaller number cares.  So, instead of the five that do 
> care getting on the phone for 15 minutes, we are going to spend two hours 
> typing and reading the e-mails.
> Yes, sending everything to everybody is open, and that's great.  At some 
> point, with the company 10x the size it was, there is going to be so much 
> stuff flying around that we won't be able to get to the stuff we care about.
>
> A much simpler example is the bug triage.  Takes a lot less time to reach a 
> decision with everybody involved in the room.
>
>>
>>> The time zone mess is, well, a mess.  Not much that can be done about it in 
>>> general.
>>
>> This is begging the question.  Of course something can be done about
>> the timezone mess: Asynchronous communication works around the problem
>> entirely.
>
> True.  And yet, we do have meetings.  And yet we do have work weeks and can't 
> wait to have them, and rave about how useful they are.
>
>
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Improving Mac OS X 10.6 test wait times by reducing 10.7 load

[Justin Lebar]

> Is this what you're saying?
> * 10.6 opt tests - per-checkin (no change)
> * 10.6 debug tests- reduced
> * 10.7 opt tests - reduced
> * 10.7 debug tests - reduced
>
> * reduced --> m-c, m-a, m-b, m-r, esr17

Yes.

Now that I think about this more, maybe we should go big or go home:
change 10.6 opt tests to reduced as well, and see how it goes.  We can
always change it back.

If it goes well, we can try to do the same thing with the Windows tests.

We should get the sheriffs to sign off.

On Thu, Apr 25, 2013 at 2:47 PM, Armen Zambrano Gasparnian
 wrote:
> On 2013-04-25 2:39 PM, Justin Lebar wrote:
>>>>
>>>> We could come to the compromise of running them on m-c, m-a, m-b and
>>>> m-r. Only this would help a lot since most of the load comes from m-i and
>>>> try. We could make it a non-by-default platform on try.
>>
>> I wonder if we should do the same for debug 10.6 tests (and maybe builds).
>
> Is this what you're saying?
> * 10.6 opt tests - per-checkin (no change)
> * 10.6 debug tests- reduced
> * 10.7 opt tests - reduced
> * 10.7 debug tests - reduced
>
> * reduced --> m-c, m-a, m-b, m-r, esr17
>
>
>>
>> The fact of the matter is that coalescing reduces our test coverage on
>> m-i as it is; so long as we run these tests on central and we're OK
>> with occasional bustage there, this seems pretty reasonable to me.
>
> Great!
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Improving Mac OS X 10.6 test wait times by reducing 10.7 load

[Justin Lebar]

>> We could come to the compromise of running them on m-c, m-a, m-b and m-r. 
>> Only this would help a lot since most of the load comes from m-i and try. We 
>> could make it a non-by-default platform on try.

I wonder if we should do the same for debug 10.6 tests (and maybe builds).

The fact of the matter is that coalescing reduces our test coverage on
m-i as it is; so long as we run these tests on central and we're OK
with occasional bustage there, this seems pretty reasonable to me.

On Thu, Apr 25, 2013 at 2:35 PM, Alex Keybl  wrote:
>> We could come to the compromise of running them on m-c, m-a, m-b and m-r. 
>> Only this would help a lot since most of the load comes from m-i and try. We 
>> could make it a non-by-default platform on try.
>
> This strategy would prevent any holes in our coverage, but accomplish the 
> goal of reducing load. Seems very reasonable, given how infrequently I've 
> seen tests fail for one OS X version but not another.
>
> -Alex
>
> On Apr 25, 2013, at 11:02 AM, "Armen Zambrano G."  wrote:
>
>> On 2013-04-25 1:40 PM, Andreas Gal wrote:>
>> > How many 10.7 machines do we operate in that pool?
>> >
>> > Andreas
>> 84 of them are 10.6
>> 86 of them are 10.7
>>
>> Unfortunately, we have a lot of them down (maybe a dozen) trying to fix them 
>> (broken hard drives, bad memory, NIC). They don't have warranty.
>>
>> On 2013-04-25 1:55 PM, Justin Lebar wrote:
>>> It would be nice if we had data indicating how often tests fail on
>>> just one version of MacOS, so we didn't have guess how useful having
>>> 10.6, 10.7, and 10.8 tests are.  That's bug 860870.  It's currently
>>> blocked on treeherder, but maybe it should be re-prioritized, since we
>>> keep running into cases where this data would be helpful.
>>>
>> It would be nice indeed.
>>
>>> Anyway, disabling the 10.7 tests sounds reasonable to me given no
>>> data, but maybe we continue running these tests on m-c?  Maybe we also
>>> deprecate the 10.7 tests on tryserver, so you only get the tests if
>>> you really really want them?
>>>
>> We could come to the compromise of running them on m-c, m-a, m-b and m-r. 
>> Only this would help a lot since most of the load comes from m-i and try. We 
>> could make it a non-by-default platform on try.
>> I assume that the wait times for 10.6 should be good enough but we should be 
>> willing to revisit later down the road if they get bad again.
>>
>> We can start with decreasing the load and visit again down the road.
>>
>> Sounds good?
>>
>> cheers,
>> Armen
>> ___
>> dev-platform mailing list
>> dev-platform@lists.mozilla.org
>> https://lists.mozilla.org/listinfo/dev-platform
>
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Increasing the platform meeting's relevance for engineers

[Justin Lebar]

> Justin pointed out his earlier post and the apparent disagreement I had with 
> it with the
> "pick a long thread topic" example - and he has a point.  I meant it as an 
> example, and
> didn't say as much, and I meant more to focus on decision making, and didn't 
> say that
> either.

I also meant "pick a long newsgroup thread" as an example.  I still
disagree with using the platform meeting for decision-making.

> However, if a particular topic is announced as "we've exchanged e-mails over 
> two weeks
> on the topic X and we think we should now make a decision", then only the 
> people
> interested would show up.

The idea that the people in timezones which allow them to attend the
meeting are privileged to participate directly in decisions made
there, and anyone else who wants to participate in these decisions
needs a proxy, is antithetical to how Mozilla works and ought to work,
I think.

> So, just by showing up, you're claiming to have interest in the particular 
> topic.

Well, you're claiming to have interest in /a/ particular topic, which
might be just one of many discussed at the meeting.  Even in the event
that the agenda is announced ahead of time (good luck with that), you
may have to wait through an hour-long meeting just to talk for sixty
seconds.

Maybe we're solving a problem here that we don't have.  I think we're
perfectly capable of debating and deciding things over e-mail.  Do you
disagree, and do you have evidence for this?

It seems to me that decisions that need in-depth, live debate are
poorly suited for a meeting that nominally involves all of
engineering.  I have difficulty fathoming a decision which is simple
enough to be decided in a few minutes on a conference call involving
hundreds of engineers, but is so complex that we can't decide it
effectively over e-mail.

Perhaps you could provide an example of something that you think would
have been decided much better in the paradigm that you're suggesting.

> The time zone mess is, well, a mess.  Not much that can be done about it in 
> general.

This is begging the question.  Of course something can be done about
the timezone mess: Asynchronous communication works around the problem
entirely.

On Thu, Apr 25, 2013 at 1:54 PM, Milan Sreckovic  wrote:
> Justin pointed out his earlier post and the apparent disagreement I had with 
> it with the "pick a long thread topic" example - and he has a point.  I meant 
> it as an example, and didn't say as much, and I meant more to focus on 
> decision making, and didn't say that either.
>
> I agree the general engineering meeting is a good place to discuss particular 
> issues, as they may not be of interest to everybody.  That would be like 
> telling everybody to show up for a movie at 2pm, and not actually tell them 
> what movie is being shown.  However, if a particular topic is announced as 
> "we've exchanged e-mails over two weeks on the topic X and we think we should 
> now make a decision", then only the people interested would show up.  So, 
> just by showing up, you're claiming to have interest in the particular topic.
>
> The time zone mess is, well, a mess.  Not much that can be done about it in 
> general.  When there is a decision making meeting that you can't attend, get 
> yourself a proxy, make sure somebody understands your stand, and hopefully 
> they can make the meeting and represent you.
>
> Not that in person meetings solve all the problems, but at least you know who 
> was there when a decision was made.
>
>
> On 2013-04-24, at 3:13 PM, Justin Lebar  wrote:
>
>> ...
>> On a related note, I think the engineering meeting is a bad place for
>> having discussions or debating decisions.  Inevitably, many of the
>> people in attendance won't care about this particular issue, so we're
>> just wasting their time.  And similarly, at our current numeric and
>> geographic scale it's inevitable that people who do care about the
>> issue won't be in attendance at the meeting and thus won't be able to
>> participate.  I think therefore that discussions / debates are
>> better-suited for our newsgroups or for smaller meetings.
>>
>> -Justin
>
>
>
> Milan
>
> On 2013-04-25, at 1:03 PM, Milan Sreckovic  wrote:
>
>>
>> Every good meeting needs a conflict  - a meeting about whether we should 
>> have the platform meeting would be a great meeting. If we are too large to 
>> actually have a meeting where  something could be argued or decided, we 
>> probably don't need that meeting.
>>
>> Status meetings are useful, but as was pointed out, reading the notes is a 
>> good way to ge

Re: Improving Mac OS X 10.6 test wait times by reducing 10.7 load

[Justin Lebar]

It would be nice if we had data indicating how often tests fail on
just one version of MacOS, so we didn't have guess how useful having
10.6, 10.7, and 10.8 tests are.  That's bug 860870.  It's currently
blocked on treeherder, but maybe it should be re-prioritized, since we
keep running into cases where this data would be helpful.

Anyway, disabling the 10.7 tests sounds reasonable to me given no
data, but maybe we continue running these tests on m-c?  Maybe we also
deprecate the 10.7 tests on tryserver, so you only get the tests if
you really really want them?

On Thu, Apr 25, 2013 at 1:40 PM, Andreas Gal  wrote:
>
> How many 10.7 machines do we operate in that pool?
>
> Andreas
>
> On Apr 25, 2013, at 10:30 AM, "Armen Zambrano G."  wrote:
>
>> (please follow up through mozilla.dev.planning)
>>
>> Hello all,
>> I have recently been looking into our Mac OS X test wait times which have 
>> been bad for many months and progressively getting worst.
>> Less than 80% of test jobs on OS X 10.6 and 10.7 are able to start
>> within 15 minutes of being requested.
>> This slows down getting tests results for OS X and makes tree closures 
>> longer if we have Mac OS X test back logs.
>> Unfortunately, we can't buy any more revision 4 Mac minis (they're not sold 
>> anymore) as Apple discontinues old hardware as new ones comes out.
>>
>> In order to improve the turnaround time for Mac testing, we have to look 
>> into reducing our test load in one of these two OSes (both of them run on 
>> revision 4 minis).
>> We have over a third of our OS X users running 10.6. Eventually, down the 
>> road, we could drop 10.6 but we still have a significant amount of our users 
>> there; even though Mac stopped serving them major updates since July 2011 
>> [1].
>>
>> Our current Mac OS X distribution looks like this:
>> * 10.6 - 43%
>> * 10.7 - 30%
>> * 10.8 - 27%
>> OS X 10.8 is the only version that is growing.
>>
>> In order to improve our wait times, I propose that we stop testing on tbpl 
>> per-checkin [2] on OS X 10.7 and re-purpose the 10.7 machines as 10.6 to 
>> increase our capacity.
>>
>> Please let us know if this plan is unacceptable and needs further discussion.
>>
>> best regards,
>> Armen Zambrano - Mozilla's Release Engineering
>> ___
>> dev-platform mailing list
>> dev-platform@lists.mozilla.org
>> https://lists.mozilla.org/listinfo/dev-platform
>
> ___
> dev-platform mailing list
> dev-platform@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-platform
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Some data on mozilla-inbound

[Justin Lebar]

> Which specific proposals should we start with? As you say, there are dozens
> of ideas out there, none with any kind of consensus behind them.

If a preponderance of options is actually the only thing standing in
the way of serious and timely work being done by releng here, I would
be more than happy to assist you guys in narrowing things down to
fewer choices.

On Thu, Apr 25, 2013 at 10:27 AM, Chris AtLee  wrote:
> On 01:48, Thu, 25 Apr, Justin Lebar wrote:
>>>
>>> One idea might be to give developers feedback on the consequences of a
>>> particular push, e.g. the AWS cost, a proxy for "time during which
>>> developers couldn't push" or some other measurable metric.  Right now
>>> each push probably "feels" as "expensive" as every other.
>>
>>
>> For tryserver, I proposed bug 848589 to do just this.  I think it's
>> worth trying, but someone needs to implement it.
>>
>>> Nobody's blaming the user.  We should just empower them to make better
>>> choices.
>>
>>
>> Okay.
>>
>> I guess what's frustrating to me is that we have this problem and
>> essentially our only option to solve it is to change users' behavior.
>> I totally believe that some people could use resources much more
>> efficiently, but it's frustrating if changing user behavior is our
>> only tool.
>>
>> We keep talking about this every few weeks, as though there's some
>> hidden solution that will emerge only after ten newsgroup threads.  In
>> actuality, we very likely will need to do a bunch of different things,
>> each having a small impact.  And in particular, I don't think we'll
>> solve this problem without significant work from release engineering.
>> If that work isn't forthcoming, I don't think we're going to make a
>> significant dent in this.
>
>
> I would say without significant work from *everyone*. There's only so much
> releng can do here.
>
> Which specific proposals should we start with? As you say, there are dozens
> of ideas out there, none with any kind of consensus behind them.
> We're already adding capacity as quickly as we can, and as Ehsan implied, I
> don't think we're ever going to have enough capacity. We're always adding
> more, but developer activity and load generated per developer has always
> increased faster than we have been able to add more capacity. I can't see
> that changing anytime soon. New mobile test platforms are particularly
> challenging to bring online, as anybody from releng/ateam or IT will tell
> you.
>
> As we work on adding more capacity, we also need to look at how to be
> smarter with our existing resources, which comes down to doing less work
> over all, or being able to smooth out the peaks in our load without
> impacting developers.
>
> I would *love* some help from developers help make tests run more
> efficiently. At last week's infra load meeting, gps mentioned that there are
> plenty of tests that could be parallelized. There are also bugs like
> https://bugzilla.mozilla.org/show_bug.cgi?id=864085 where windows debug
> browser chrome test times have been steadily increasing over the past year.
> Can we find other cases like this?
>
> Also, there are plenty of jobs that are run per push, but are hidden on
> inbound and other branches. I would love to be able to disable those from
> running per push and save everybody from paying the cost of running them.
>
> Cheers,
> Chris
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Some data on mozilla-inbound

[Justin Lebar]

> One idea might be to give developers feedback on the consequences of a
> particular push, e.g. the AWS cost, a proxy for "time during which
> developers couldn't push" or some other measurable metric.  Right now
> each push probably "feels" as "expensive" as every other.

For tryserver, I proposed bug 848589 to do just this.  I think it's
worth trying, but someone needs to implement it.

> Nobody's blaming the user.  We should just empower them to make better 
> choices.

Okay.

I guess what's frustrating to me is that we have this problem and
essentially our only option to solve it is to change users' behavior.
I totally believe that some people could use resources much more
efficiently, but it's frustrating if changing user behavior is our
only tool.

We keep talking about this every few weeks, as though there's some
hidden solution that will emerge only after ten newsgroup threads.  In
actuality, we very likely will need to do a bunch of different things,
each having a small impact.  And in particular, I don't think we'll
solve this problem without significant work from release engineering.
If that work isn't forthcoming, I don't think we're going to make a
significant dent in this.

On Thu, Apr 25, 2013 at 1:20 AM, Ehsan Akhgari  wrote:
> On 2013-04-25 1:02 AM, David Ascher wrote:
>>
>>
>> The messaging around this should not be to tell people "always test
>> on try".  It should be to help them figure out how to make better
>> judgement calls on this.  This is a skill that people develop and
>> are not born with, and without data it's hard an an individual to
>> judge how good I'm at that.
>>
>>
>> One idea might be to give developers feedback on the consequences of a
>> particular push, e.g. the AWS cost, a proxy for "time during which
>> developers couldn't push" or some other measurable metric.  Right now
>> each push probably "feels" as "expensive" as every other.
>
>
> The AWS cost would be the wrong measure, since it doesn't account for the
> amount of time that 100 other people spent grinding their teeth because they
> could not push.  :-)  But yeah, I agree with the general idea of a cost
> measure, I just can't think of what a good one would be (well, one better
> than the wall-clock time...)
>
> Ehsan
>
> ___
> dev-platform mailing list
> dev-platform@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-platform
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Increasing the platform meeting's relevance for engineers

[Justin Lebar]

One thing I love about the MoCo meetings is that if I don't go, I
don't miss anything except the chance to ask questions: mbrubeck &co
create detailed minutes (really, transcripts) of every meeting, which
I can read on my schedule.  He then e-mails the transcript out to
everyone, so I don't even have to remember to go looking for it.

Since in-person attendance at the MoCo meetings is non-zero, it seems
clear that some people prefer being in live attendance.  That's
totally fine.  But at the very least, I think it's useful to recognize
that not everyone is willing or able to attend the meetings live, and
if we think what's going on there is important, we should make an
effort to broadcast it to a larger audience.

Last time I checked the wiki isn't a canonical record of the
engineering meeting's contents, and last time I checked there's no way
to get notified when a meeting's notes are up (via RSS or e-mail or
whatever).

On a related note, I think the engineering meeting is a bad place for
having discussions or debating decisions.  Inevitably, many of the
people in attendance won't care about this particular issue, so we're
just wasting their time.  And similarly, at our current numeric and
geographic scale it's inevitable that people who do care about the
issue won't be in attendance at the meeting and thus won't be able to
participate.  I think therefore that discussions / debates are
better-suited for our newsgroups or for smaller meetings.

-Justin

On Wed, Apr 24, 2013 at 2:26 PM, Lawrence Mandel  wrote:
> tl;dr
> I would like to make the platform meeting more relevant for engineers. I have 
> already made some changes (see below) and am interested in your feedback on 
> what you would like to get out of this meeting. If you don't currently 
> attend, what would make this meeting relevant for you?
>
> ---
> Since taking over management of the Platform meeting in February, my goal has 
> been to make it more engineer driven and more relevant to the day-to-day 
> activities of engineers. I think that there is value in having a forum for 
> Mozilla's engineers to speak with one another on a regular basis. However, 
> with the size of Mozilla's engineering base, creating a useful forum is a 
> challenge.
>
> Some of the changes that we've made are:
>
> - Engineers are once again doing the talking - Many of the technical updates 
> were being given by project managers. No more. Updates for engineers, by 
> engineers. So far, I think this has been successful in changing the tone of 
> the meeting.
>
> - More talk about work in progress - Reporting about completed work is great. 
> However, we are now also talking about active projects. I think work that 
> touches common sections of the code base or work that is likely to impact 
> other teams is especially useful to flag.
>
> - Identify topics that are stalled / need more attention - Over the last 
> month I have seen a new focus on stability and orange factor issues. I have 
> also seen a number of calls for help in identifying the source of issues.
>
> - Invite other teams to speak with engineering - I invited Sheppy to join us 
> to discuss documentation. I would encourage the invitation of other teams 
> with which we collaborate to join the meeting for specific topics.
>
> - Flag important discussions from across Mozilla's mailing lists - A number 
> of people have commented to me that there are too many lists to track. This 
> meeting is an opportunity to surface key online discussions to ensure they 
> have the participation of the right people and reach a conclusion.
>
> Some other suggestions are:
>
> - A review of best practices and anti-pattens in order to build the technical 
> vitality of the Mozilla engineering organization.
>
> - More involvement by more people. The product and project sections of the 
> agenda do not have a specific owner. You can bring up any related work 
> wherever you think it makes sense. There are also the catchall sections "Key 
> Issues" and "Roundtable" if you are not sure where your update belongs.
>
> I want this meeting to be relevant for you. However, I am just a steward for 
> this meeting. I need your help. Why do you attend the platform meeting? If 
> you previously attended the meeting, why did you stop? Do you have ideas to 
> make the Platform meeting more useful?
>
> Thanks,
>
> Lawrence
> ___
> dev-platform mailing list
> dev-platform@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-platform
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Rethinking the amount of system JS we use in Gecko on B2G

[Justin Lebar]

To close the loop on this thread, the consensus here seems to be that

1. We should continue to make JS slimmer.  This is a high priority for
B2G even setting aside the memory consumption of B2G's chrome JS,
since of course B2G runs plenty of content JS.

The memory profile of B2G is different from desktop -- small overheads
matter more on B2G, and the consequences of using too much memory are
more drastic.  We should keep this in mind when working on JS in the
future.

2. We should fix bug 829482 (run more shrinking GCs in workers).  This
will get us an easy 4-5mb in the main B2G process.  This is a
MemShrink:P1 and has been open for a while; I'd love some assistance
with it.

3. We should rewrite these main-process workers in C++, if and when we
have manpower.  Even with bug 829482 fixed, the workers will still be
some of the largest individual consumers of JS memory in B2G, and all
of the JS folks I spoke with said that they thought they'd be unable
to reduce the memory overhead of a worker significantly in the medium
term.

I filed bugs:

https://bugzilla.mozilla.org/show_bug.cgi?id=864927
https://bugzilla.mozilla.org/show_bug.cgi?id=864931
https://bugzilla.mozilla.org/show_bug.cgi?id=864932

4. It's worthwhile to at least look carefully at the biggest B2G
chrome compartments and see whether we can reduce their size one way
or another.  I filed a metabug:
https://bugzilla.mozilla.org/show_bug.cgi?id=864943

5. When writing new APIs, we should at least consider writing them in
C++.  JS should not be the default.  Where things are super-easy in JS
and super-annoying in C++, we should consider investing in our C++
infrastructure to make it more pleasant.

Since not everyone reads this newsgroup, I'd appreciate assistance
disseminating (5) in bugs.  At the very least, we should ask patch
authors to consider the alternatives before creating new JS modules
that are enabled on B2G.

I'm also going to post this summary to dev-b2g with a pointer back to
this newsgroup.

Thanks for your thoughts, everyone.

-Justin

On Mon, Apr 22, 2013 at 9:46 PM, Nicholas Nethercote
 wrote:
> On Mon, Apr 22, 2013 at 6:35 PM, Justin Dolske  wrote:
>>
>> That said, I think it's critically important that we're working to make JS a
>> acceptable -- nay, _excellent_ -- language/runtime for application
>> development for the long run. We can't tell a credible story about why
>> people should write HTML5 apps, if we're tearing out swaths of JS in our own
>> products. Sometimes dogfooding is unpleasant or hard, but that's the whole
>> point.
>
> There's a big difference between apps on Firefox OS, which are likely
> to have relatively short lifetimes and can be killed if they take up
> too much memory, and the main process.  Bad memory behaviour in the
> main process is a much bigger deal, and it's something that's
> happening right now with some frequency.
>
> Nick
> ___
> dev-platform mailing list
> dev-platform@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-platform
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Some data on mozilla-inbound

[Justin Lebar]

>> The ratio of things landed on inbound which turn out to busted is really
>> worrying

On the one hand, we're told not to push to try too much, because that
wastes resources.

On the other hand, we're told not to burn m-i, because that wastes resources.

Should we be surprised when people don't get this right 100% of the time?

Instead of considering how to get people people to strike a better
balance between wasting infra resources and burning inbound, I think
we need to consider what we can do to increase the acceptable margin
of error.

Note that we don't have enough capacity to turn around current try
requests within a reasonable amount of time.  Pushing to inbound is
the only way to get quick feedback on whether your patch works, these
days.  As I've said before, I'd love to see releng report on try
turnaround times, so we can hold someone accountable.  The data is
there; we just need to process it.

If we can't increase the amount of infra capacity we have, perhaps we
could use it more effectively.  We've discussed lots of ways we might
accomplish this on this newsgroup, and I've seen very few of them
tried.  Perhaps an important part of the problem is that we're not
able to innovate quickly enough on this front.

People are always going to make mistakes, and the purpose of processes
is to minimize the harm caused by those mistakes, not to embarrass or
cajole people into behaving better in the future.  As Jono would say,
it's not the user's fault.

On Tue, Apr 23, 2013 at 12:50 AM, Justin Lebar  wrote:
>> The ratio of things landed on inbound which turn out to busted is really
>> worrying
>
>> * 116 of the 916 changesets (12.66%) were backed out
>
> If 13% is "really worrying", what do you think our goal should be?
>
> On Tue, Apr 23, 2013 at 12:39 AM, Ehsan Akhgari  
> wrote:
>> This was a fantastic read, it almost made me shed happy tears!  Thanks a lot
>> kats for doing this.
>>
>> The ratio of things landed on inbound which turn out to busted is really
>> worrying, and it might be an indicator that (some?) developers have a poor
>> judgement on how safe their patches are.  How hard would it be to gather a
>> list of the total number of patches being backed out plus the amount of time
>> that we spent building/testing those, hopefully in a style similar to
>> <http://people.mozilla.org/~catlee/highscores/highscores.html>?  If we had
>> such a list, perhaps we could reach out to the high offenders there and let
>> them know about the problem, and see if that changes these stats a couple of
>> weeks from now?
>>
>> Thanks!
>> Ehsan
>>
>>
>> On 2013-04-22 3:54 PM, Kartikaya Gupta wrote:
>>>
>>> TL;DR:
>>> * Inbound is closed 25% of the time
>>> * Turning off coalescing could increase resource usage by up to 60% (but
>>> probably less than this).
>>> * We spend 24% of our machine resources on changes that are later backed
>>> out, or changes that are doing the backout
>>> * The vast majority of changesets that are backed out from inbound are
>>> detectable on a try push
>>>
>>> Because of the large effect from coalescing, any changes to the current
>>> process must not require running the full set of tests on every push.
>>> (In my proposal this is easily accomplished with trychooser syntax, but
>>> other proposals include rotating through T-runs on pushes, etc.).
>>>
>>> --- Long verion below ---
>>>
>>> Following up from the infra load meeting we had last week, I spent some
>>> time this weekend crunching various pieces of data on mozilla-inbound to
>>> get a sense of how much coalescing actually helps us, how much backouts
>>> hurt us, and generally to get some data on the impact of my previous
>>> proposal for using a multi-headed tree. I didn't get all the data that I
>>> wanted but as I probably won't get back to this for a bit, I thought I'd
>>> share what I found so far and see if anybody has other specific pieces
>>> of data they would like to see gathered.
>>>
>>> -- Inbound uptime --
>>>
>>> I looked at a ~9 day period from April 7th to April 16th. During this
>>> time:
>>> * inbound was closed for 24.9587% of the total time
>>> * inbound was closed for 15.3068% of the total time due to "bustage".
>>> * inbound was closed for 11.2059% of the total time due to "infra".
>>>
>>> Notes:
>>> 1) "bustage" and "infra" were determined by grep -i on the data from
>>> treestatus.mozil

Re: Some data on mozilla-inbound

[Justin Lebar]

> The ratio of things landed on inbound which turn out to busted is really
> worrying

> * 116 of the 916 changesets (12.66%) were backed out

If 13% is "really worrying", what do you think our goal should be?

On Tue, Apr 23, 2013 at 12:39 AM, Ehsan Akhgari  wrote:
> This was a fantastic read, it almost made me shed happy tears!  Thanks a lot
> kats for doing this.
>
> The ratio of things landed on inbound which turn out to busted is really
> worrying, and it might be an indicator that (some?) developers have a poor
> judgement on how safe their patches are.  How hard would it be to gather a
> list of the total number of patches being backed out plus the amount of time
> that we spent building/testing those, hopefully in a style similar to
> ?  If we had
> such a list, perhaps we could reach out to the high offenders there and let
> them know about the problem, and see if that changes these stats a couple of
> weeks from now?
>
> Thanks!
> Ehsan
>
>
> On 2013-04-22 3:54 PM, Kartikaya Gupta wrote:
>>
>> TL;DR:
>> * Inbound is closed 25% of the time
>> * Turning off coalescing could increase resource usage by up to 60% (but
>> probably less than this).
>> * We spend 24% of our machine resources on changes that are later backed
>> out, or changes that are doing the backout
>> * The vast majority of changesets that are backed out from inbound are
>> detectable on a try push
>>
>> Because of the large effect from coalescing, any changes to the current
>> process must not require running the full set of tests on every push.
>> (In my proposal this is easily accomplished with trychooser syntax, but
>> other proposals include rotating through T-runs on pushes, etc.).
>>
>> --- Long verion below ---
>>
>> Following up from the infra load meeting we had last week, I spent some
>> time this weekend crunching various pieces of data on mozilla-inbound to
>> get a sense of how much coalescing actually helps us, how much backouts
>> hurt us, and generally to get some data on the impact of my previous
>> proposal for using a multi-headed tree. I didn't get all the data that I
>> wanted but as I probably won't get back to this for a bit, I thought I'd
>> share what I found so far and see if anybody has other specific pieces
>> of data they would like to see gathered.
>>
>> -- Inbound uptime --
>>
>> I looked at a ~9 day period from April 7th to April 16th. During this
>> time:
>> * inbound was closed for 24.9587% of the total time
>> * inbound was closed for 15.3068% of the total time due to "bustage".
>> * inbound was closed for 11.2059% of the total time due to "infra".
>>
>> Notes:
>> 1) "bustage" and "infra" were determined by grep -i on the data from
>> treestatus.mozilla.org.
>> 2) There is some overlap so bustage + infra != total.
>> 3) I also weighted the downtime using checkins-per-hour histogram from
>> joduinn's blog at [1], but this didn't have a significant impact: the
>> total, bustage, and infra downtime percentages moved to 25.5392%,
>> 15.7285%, and 11.3748% respectively.
>>
>> -- Backout changes --
>>
>> Next I did an analysis of the changes that landed on inbound during that
>> time period. The exact pushlog that I looked at (corresponding to the
>> same April 7 - April 16 time period) is at [2]. I removed all of the
>> merge changesets from this range, since I wanted to look at inbound in
>> as much isolation as possible.
>>
>> In this range:
>> * there were a total of 916 changesets
>> * there were a total of 553 "pushes"
>> * 74 of the 916 changesets (8.07%) were backout changesets
>> * 116 of the 916 changesets (12.66%) were backed out
>> * removing all backouts and changes backed out removed 114 pushes (20.6%)
>>
>> Of the 116 changesets that were backed out:
>> * 37 belonged to single-changeset pushes
>> * 65 belonged to multi-changeset pushes where the entire pushed was
>> backed out
>> * 14 belonged to multi-changeset pushes where the changesets were
>> selectively backed out
>>
>> Of the 74 backout changesets:
>> * 4 were for commit message problems
>> * 25 were for build failures
>> * 36 were for test failures
>> * 5 were for leaks/talos regressions
>> * 1 was for premature landing
>> * 3 were for unknown reasons
>>
>> Notes:
>> 1) There were actually 79 backouts, but I ignored 5 of them because they
>> backed out changes that happened prior to the start of my range).
>> 2) Additional changes at the end of my range may have been backed out,
>> but the backouts were not in my range so I didn't include them in my
>> analysis.
>> 3) The 14 csets that were selectively backed out is interesting to me
>> because it implies that somebody did some work to identify which changes
>> in the push were bad, and this naturally means that there is room to
>> save on doing that work.
>>
>> -- Merge conflicts --
>>
>> I also wanted to determine how many of these changes conflicted with
>> each other, and how far away the conflicting changes were. I 

Re: Rethinking the amount of system JS we use in Gecko on B2G

[Justin Lebar]

> This is all great stuff, but as mentioned elsewhere, B2G branched at
> version 18 and so they need improvements that that can land quickly on
> the relevant branches.

Well, to be clear, it would be great if we could land some
improvements for v1.1 (which is based off version 18), but we're
locking the tree down pretty hard already, so I suspect that e.g. a
wifi worker rewrite is off the table for that version.  Hopefully v1.1
is the last b2g version that will be based off b2g18.

v1.1 is also, I found out last week, targeting a 512mb device, so
memory usage probably isn't as critical there as it will be in future
releases which target 256mb devices.
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Rethinking the amount of system JS we use in Gecko on B2G

[Justin Lebar]

> There are a few things we're working on in SpiderMonkey that should improve
> this situation quite a bit:

Thanks, but I again need to emphasize that these are large, long-term
plans.  Terrence tells me that GGC is planned for "sometime this
year".  Lazy bytecode generation has been on the roadmap for a long
time as well.

I understand that these are large projects, and I don't mean to
suggest that you guys aren't doing a good job with them, but I do not
think that the right solution for b2g system compartments / workers is
"JS should be able to meet your requirements one day," and "wait until
things get better."  In the meantime we continue to dig ourselves in a
hole by implementing new DOM APIs in JS.

We of course still want these engine improvements for reducing the
memory usage of web content.

Again, in the past, we've had the most success dealing with the JS
engine as it is today.  Then the engine optimizations, when they come,
are gravy.  That's all I'm suggesting we do here.

On Mon, Apr 22, 2013 at 4:25 PM, Till Schneidereit
 wrote:
> There are a few things we're working on in SpiderMonkey that should improve
> this situation quite a bit:
>
> Terrence already mentioned generational GC, which certainly is the largest
> piece by far. Getting rid of all or almost all memory lost to fragmentation
> makes the tradeoff a considerably different one, I'd say.
>
> Additionally, the work on making bytecode generation lazy[1] should vastly
> reduce the memory used for scripts. Based on that, I'm investigating
> several more options to reduce script memory:
> - re-lazyfication of JSScripts, reducing memory usage by removing the
> parsed represantation of scripts that are only run once or very rarely.
> - lazy cloning of JSScripts with the same source from other compartments
> containing the same script. Probably to go hand-in-hand with
> re-lazyfication.
>
> Combined, I'd very vaguely say that these measures should reduce the memory
> usage by an additional 20 or 30 percentage points. (Based on the current
> usage as 100%, and no guarantees, of course).
>
>
>
>
> On Mon, Apr 22, 2013 at 8:44 PM, Terrence Cole  wrote:
>
>> On 04/22/2013 12:12 PM, Jeff Muizelaar wrote:
>> > On 2013-04-22, at 2:15 PM, Bill McCloskey wrote:
>> >
>> >> I can't agree with you more, Justin. I think Boris is right that we
>> should make these decisions on a case-by-case basis. But in the case of
>> these workers, it seems clear that converting them to C++ is the way to go,
>> assuming we have the resources to do so.
>> > So a specific case that I ran into during the Performance Workshop is
>> RILContentHelper.js. During the startup of the settings app
>> > we jank for 53ms while initializing the RILContentHelper.js:
>> >
>> >
>> http://people.mozilla.com/~bgirard/cleopatra/#report=bf7077c6552fe2bc015d7074a338b673911f3ce8&search=Mobile
>>
>> That link gives me this: "Error fetching profile :(. URL:
>> '
>> http://profile-store.commondatastorage.googleapis.com/bf7077c6552fe2bc015d7074a338b673911f3ce8
>> '.
>> Did you set the CORS headers?"
>>
>> >
>> > There doesn't seem to be anything specific taking that much time in the
>> profile, just general JS overhead. In this case RILContentHelper.js is
>> wrapped by by C++ code in dom/network/src/MobileConnection.cpp and so we
>> end up spending a fair amount of time transitioning from JS to C++ to JS to
>> C++.
>>
>> That seems like the sort of thing that SpiderMonkey may be able to
>> address in the short term, depending on what exactly it turns out to be.
>> Is there a bug on file somewhere to coordinate the investigation?
>>
>> -Terrence
>>
>> > -Jeff
>> > ___
>> > dev-platform mailing list
>> > dev-platform@lists.mozilla.org
>> > https://lists.mozilla.org/listinfo/dev-platform
>>
>> ___
>> dev-platform mailing list
>> dev-platform@lists.mozilla.org
>> https://lists.mozilla.org/listinfo/dev-platform
>>
> ___
> dev-platform mailing list
> dev-platform@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-platform
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Rethinking the amount of system JS we use in Gecko on B2G

[Justin Lebar]

> I can't really agree or disagree without knowing why they use "too much"
> memory.

At the risk of sounding like a broken record, it's all in the memory
reports.  You probably understand this data better than I do.  Extract
and load in about:memory (button is at the bottom).

http://people.mozilla.org/~jlebar/downloads/merged.json.xz
http://people.mozilla.org/~jlebar/downloads/unmerged.json.xz

As I said earlier, if the JS team wants to own B2G memory usage and
commit to getting chrome JS memory usage down to C++ levels within a
timeframe that's acceptable to the B2G team, that's fantastic.

If on the other hand the JS team is not ready to commit to getting
this work done on B2G's schedule, then by construction "wait for JS to
get better" is not a solution that works for us.

Given how long some of this prerequisite work (e.g. generational
garbage collection) has been ongoing for, I'm highly dubious of the
claim that our JS engine will improve at the requisite rate.  Where
we've had success reducing our JS memory in the past (e.g. bug
798491), it's been by working within the current reality of the JS
engine, instead of twiddling our thumbs waiting for the Right Fix
(e.g. bug 759585, which did not come in time to be useful for B2G
1.x).

Please don't take this as a suggestion that I think you guys are doing
a bad job -- I continue to characterize the JS team's work as heroic.
I just think that there's a limit to how much we ought to expect from
the JS folks, particularly given how many other high-priority projects
you have.

On Mon, Apr 22, 2013 at 1:36 PM, Terrence Cole  wrote:
> On 04/21/2013 04:51 PM, Justin Lebar wrote:
>> I think we should consider using much less JS in the parts of Gecko that are
>> used in B2G.  I'd like us to consider writing new modules in C++ where
>> possible, and I'd like us to consider rewriting existing modules in C++.
>>
>> I'm only proposing a change for modules which are enabled for B2G.  For 
>> modules
>> which aren't enabled on B2G, I'm not proposing any change.
>>
>> What I'd like to come out of this thread is a consensus one way or another as
>> to whether we continue along our current path of writing many features that 
>> are
>> enabled on B2G in JS, or whether we change course.
>>
>> Since most of these features implemented in JS seem to be DOM features, I'm
>> particularly interested in the opinions of the DOM folks.  I'm also 
>> interested
>> in the opinions of JS folks, particularly those who know about the memory 
>> usage
>> of our new JITs.
>>
>> In the remainder of this e-mail I'll first explain where our JS memory is
>> going.  Then I'll address two arguments that might be made against my 
>> proposal
>> to use more C++.  Finally, I'll conclude by suggesting a plan of action.
>>
>> === Data ===
>>
>> Right now about 50% (16mb) of the memory used by the B2G main process
>> immediately after rebooting is JS.   It is my hypothesis that we could 
>> greatly
>> reduce this by converting modules to C++.
>>
>> On our 256mb devices, we have about 120mb available to Gecko, so this 16mb
>> represents 13% of all memory available to B2G.
>>
>> To break down the 16mb of JS memory, 8mb is from four workers: ril_worker,
>> net_worker, wifi_worker (x2).  5mb of the 8mb is under "unused-arenas"; this 
>> is
>> fragmentation in the JS heap.  Based on my experience tackling fragmentation 
>> in
>> the jemalloc heap, I suspect reducing this would be difficult.  But even if 
>> we
>> eliminated all of the fragmentation, we'd still be spending 3mb on these four
>> workers, which I think is likely far more than we need.
>
> Once exact rooting of the browser is complete we can implement heap
> defragmentation easily. Generational GC should help as well here.
>
>> The other 8mb is everything else in the system compartment (all our JSMs,
>> XPCOM components, etc).  In a default B2G build you don't get a lot of 
>> insight
>> into this, because most of the system compartments are squished together to 
>> save
>> memory (bug 798491).  If I set jsloader.reuseGlobal to false, the amount of
>> memory used increases from 8mb to 15mb, but now we can see where it's going.
>>
>> The list of worst offenders follows, but because this data was collected with
>> reuseGlobal turned off, apply generous salt.
>>
>>   0.74 MB modules/Webapps.jsm
>>   0.59 MB anonymous sandbox from devtools/dbg-server.jsm:41
>>   0.53 MB components/SettingsManager.js
>>   0.53 MB chrome://

Re: Rethinking the amount of system JS we use in Gecko on B2G

[Justin Lebar]

> So to the extent that b2g is in a "not enough hands" situation, implementing 
> in JS
> makes sense.

I don't feel like b2g is in this position.  We don't have a lot of
Gecko hackers with 2+ years of experience, but we do have a lot of
hackers.  Whether or not we have a lot of C++ versus JS hackers, I
can't say.

> To the extent that we have code whose performance we want to maximize or whose
> memory footprint we need to minimize, a JS implementation right now is not a 
> good
> idea.

My contention is that /everything that runs on B2G/ is code whose
memory footprint we need to minimize.  Fitting inside 120mb leaves
very little room for waste.

But it doesn't sound like you disagree too much with the idea of
re-implementing the worst offenders in C++, as opposed to sitting back
and hoping for heroics from the JS team.  (To be clear, if I had to
choose a team to deliver heroics, I'd choose the JS team in a
heartbeat.)

It seems foolhardy to me to on the one hand do the work of
re-implementing these workers in C++, and on the other hand continue
implementing DOM features in JS that eat up tens or hundreds of KB of
memory each.  But I suppose that's better than doing nothing, and if
we all can agree on doing that much, I'd be happy.

On Mon, Apr 22, 2013 at 10:40 AM, Boris Zbarsky  wrote:
> On 4/21/13 7:51 PM, Justin Lebar wrote:
>>
>> Since most of these features implemented in JS seem to be DOM features,
>> I'm
>> particularly interested in the opinions of the DOM folks.
>
>
> Opinions differ.  ;)
>
> I think for DOM features that are not invoked on most pages, implementing
> them in JS seems like a reasonable choice.  We are actively working on
> making this easier to do than it is now.
>
> This is only a viable implementation strategy for objects you don't expect
> to have lots of around, since there is a lot more per-object overhead for a
> JS-implemented DOM object.  But lots of things that are currently
> implemented in JS are per-page singletons, so that's ok.
>
> I'm very sympathetic to Andreas' points about memory-safety (especially for
> hard-to-fuzz things) and ease of prototyping and implementation. But on the
> flip side, there are security bugs that JS implementations are subject to
> (involving content passing in unexpected objects and whatnot) that a C++
> implementation simply never has to deal with.  See
> https://bugzilla.mozilla.org/show_bug.cgi?id=856042 for those with the
> access...  I'm hoping that putting a WebIDL layer in front of the JS
> implementations will help, but will of course add to the per-object
> overhead.
>
> One other thing to keep in mind is that doing an implementation in JS does
> not involve having to figure out cycle collection, etc, boilerplate...
>
> So to the extent that b2g is in a "not enough hands" situation, implementing
> in JS makes sense.
>
> To the extent that we have code whose performance we want to maximize or
> whose memory footprint we need to minimize, a JS implementation right now is
> not a good idea.
>
> I'm afraid in practice the right decision should probably be made on a
> per-component basis, and the hard part with that is making sure informed
> decisions are made.  :(
>
> -Boris
>
> ___
> dev-platform mailing list
> dev-platform@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-platform
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Rethinking the amount of system JS we use in Gecko on B2G

[Justin Lebar]

> How about making the JS engine use jemalloc instead of its own
> allocator? Does anything actually rely on the arenas being independent
> at the paging level?

My understanding, which may be wrong, is that the JS engine needs to
be able to quickly map an object's address to a compartment.  They do
this by keeping a map (in the runtime?) of chunk addresses (1mb
aligned) to compartments.  Given an address, you can easily find its
chunk (zero the low-order bits), and then it's a quick lookup to find
its compartment.

We could imagine shrinking chunks to 4kb, but that doesn't solve the
fragmentation problem here; the fragmentation in "unused-arenas" is
/within/ a page.  (We also have a bunch of decommitted memory in these
workers due to fragmentation between pages, but that's not a problem.)
 What you really need, I guess, is a moving GC.

But I emphasize again that reducing the unused-arenas does not solve
all of our problems, only the single largest.

On Mon, Apr 22, 2013 at 11:05 AM, Mike Hommey  wrote:
> On Mon, Apr 22, 2013 at 10:53:40AM -0400, Justin Lebar wrote:
>> > How about pre-compiling JS in JITed form?
>>
>> While significant, it seems that memory used for script source isn't
>> the biggest offender.
>>
>> Full details are in the about:memory reports,
>>
>> http://people.mozilla.org/~jlebar/downloads/merged.json.xz
>> http://people.mozilla.org/~jlebar/downloads/unmerged.json.xz
>>
>> but here's a teaser.
>>
>> This is the wifi worker.  I think "script-sources" is code.  Note that
>> fragmentation (unused-arenas) is way too high, but even despite this
>> the worker uses too much memory.
>
> How about making the JS engine use jemalloc instead of its own
> allocator? Does anything actually rely on the arenas being independent
> at the paging level?
>
> Mike
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Rethinking the amount of system JS we use in Gecko on B2G

[Justin Lebar]

> How about pre-compiling JS in JITed form?

While significant, it seems that memory used for script source isn't
the biggest offender.

Full details are in the about:memory reports,

http://people.mozilla.org/~jlebar/downloads/merged.json.xz
http://people.mozilla.org/~jlebar/downloads/unmerged.json.xz

but here's a teaser.

This is the wifi worker.  I think "script-sources" is code.  Note that
fragmentation (unused-arenas) is way too high, but even despite this
the worker uses too much memory.

> 2.38 MB (05.13%) -- worker(resource://gre/modules/wifi_worker.js, 0x45584800)
> ├──1.77 MB (03.81%) -- gc-heap
> │  ├──1.74 MB (03.74%) ── unused-arenas
> │  └──0.03 MB (00.07%) -- (2 tiny)
> │ ├──0.03 MB (00.07%) ── chunk-admin
> │ └──0.00 MB (00.00%) ── unused-chunks
> └──0.61 MB (01.32%) -- (3 tiny)
>├──0.26 MB (00.57%) -- runtime
>│  ├──0.13 MB (00.27%) ── gc-marker
>│  ├──0.06 MB (00.12%) ── jaeger-code
>│  ├──0.04 MB (00.08%) ── runtime-object
>│  ├──0.02 MB (00.03%) ── atoms-table
>│  ├──0.01 MB (00.03%) ── script-sources
>│  ├──0.01 MB (00.01%) ── unused-code
>│  ├──0.00 MB (00.01%) ── dtoa
>│  ├──0.00 MB (00.01%) ── stack
>│  ├──0.00 MB (00.01%) ── temporary
>│  ├──0.00 MB (00.00%) ── contexts
>│  ├──0.00 MB (00.00%) ── script-filenames
>│  ├──0.00 MB (00.00%) ── ion-code
>│  ├──0.00 MB (00.00%) ── math-cache
>│  └──0.00 MB (00.00%) ── regexp-code
>├──0.25 MB (00.54%) -- compartment(web-worker)
>│  ├──0.13 MB (00.29%) -- gc-heap
>│  │  ├──0.07 MB (00.15%) ── unused-gc-things [2]
>│  │  ├──0.03 MB (00.06%) -- objects
>│  │  │  ├──0.02 MB (00.04%) ── non-function
>│  │  │  └──0.01 MB (00.02%) ── function
>│  │  ├──0.02 MB (00.04%) ── shapes/tree
>│  │  └──0.02 MB (00.03%) ── sundries [2]
>│  ├──0.06 MB (00.13%) ── analysis-temporary
>│  ├──0.03 MB (00.06%) ── other-sundries [2]
>│  ├──0.02 MB (00.04%) ── objects-extra/slots
>│  └──0.01 MB (00.02%) ── script-data
>└──0.10 MB (00.22%) -- compartment(web-worker-atoms)
>   ├──0.09 MB (00.19%) -- gc-heap
>   │  ├──0.09 MB (00.18%) ── strings
>   │  └──0.00 MB (00.01%) ── sundries
>   └──0.01 MB (00.02%) ── other-sundries

Here's the worst-offending compartment (webapps.js) with
jsloader.reuseGlobal turned off.  Recall that flipping this switch
expands memory usage by a factor of 2, and not all pieces of the
compartment are affected equally, so take this with salt.

> 0.74 MB (01.60%) -- compartment([System Principal], 
> resource://gre/modules/Webapps.jsm)
> ├──0.28 MB (00.61%) -- gc-heap
> │  ├──0.10 MB (00.21%) ── unused-gc-things
> │  ├──0.08 MB (00.17%) -- objects
> │  │  ├──0.06 MB (00.12%) ── non-function
> │  │  └──0.02 MB (00.04%) ── function
> │  ├──0.05 MB (00.11%) ── strings
> │  ├──0.03 MB (00.05%) ── shapes/tree
> │  ├──0.02 MB (00.04%) ── scripts
> │  └──0.01 MB (00.02%) ── sundries
> ├──0.17 MB (00.36%) -- string-chars
> │  ├──0.09 MB (00.20%) ── non-huge
> │  └──0.08 MB (00.17%) -- huge
> │ ├──0.04 MB (00.08%) ── string(length=9114, 
> "data:image//png;base64,iVBORw0KG...") [2]
> │ └──0.04 MB (00.08%) ── string(length=9646, 
> "data:image//png;base64,iVBORw0KG...") [2]
> ├──0.11 MB (00.23%) -- objects-extra
> │  ├──0.09 MB (00.20%) ── slots
> │  └──0.02 MB (00.03%) ── property-iterator-data
> ├──0.07 MB (00.16%) ── script-data
> ├──0.06 MB (00.13%) ── analysis-temporary
> ├──0.02 MB (00.04%) ── shapes-extra/tree-tables
> ├──0.02 MB (00.04%) ── other-sundries
> └──0.02 MB (00.03%) ── cross-compartment-wrappers

To me, this seems like death by a thousand cuts; there are /lots/ of
little things that we'd need to improve.

On Mon, Apr 22, 2013 at 10:31 AM, Mike Hommey  wrote:
> On Sun, Apr 21, 2013 at 07:51:18PM -0400, Justin Lebar wrote:
>> I think we should consider using much less JS in the parts of Gecko that are
>> used in B2G.  I'd like us to consider writing new modules in C++ where
>> possible, and I'd like us to consider rewriting existing modules in C++.
>>
>> I'm only proposing a change for modules which are enabled for B2G.  For 
>> modules
>> which aren't enabled on B2G, I'm not proposing any change.
>>
>> What I'd like to come out of this thread is a consensus one way or another as
>> to whether we continue along our current path of writing many features that 
>> are
>> enabled on B2G in JS, or whether we change course.
>>
>> Since most of these features implemented in JS seem to be DOM features, I'm
>> particularly interested in the opinions of the DOM folks.  I'm also 
>> interested
>> in the opinions o

Re: Rethinking the amount of system JS we use in Gecko on B2G

[Justin Lebar]

> I think I would like to hear from the JS team on reducing memory use of JS and
> disabling any compilation for infrequently running code before we give
> up on it.

The issue isn't compilation of code; that doesn't stick out in the
memory reports.  The issue seems to be mostly the overhead of the JS
engine for each file (even if the file stores very little data, as
BrowserElementParent does) and also the unavoidable inefficiency
associated with assigning each worker its own runtime (in particular,
the fact that this greatly increases fragmentation).

> Using JS for infrequently executing code should be a memory win.

If the JS team wants to own B2G memory usage from here on out and can
commit the necessary resources to re-optimizing the JS engine
specifically for B2G, then that's great, I'd love to see us test this
hypothesis.

Otherwise, we're leaving a huge amount of memory for B2G on the table
when we have a solution in hand that requires zero heroics.  I think
that's folly.

On Mon, Apr 22, 2013 at 10:11 AM, Andreas Gal  wrote:
> JS is a big advantage for rapid implementation of features and it's
> easier to avoid exploitable mistakes. Also, in many cases JS code
> (bytecode, not data) should be slimmer than C++. Using JS for
> infrequently executing code should be a memory win. I think I would
> like to hear from the JS team on reducing memory use of JS and
> disabling any compilation for infrequently running code before we give
> up on it.
>
> Andreas
>
> Sent from Mobile.
>
> On Apr 22, 2013, at 7:05, Justin Lebar  wrote:
>
>> Of course attachments don't work great on newsgroups.  I've uploaded
>> the about:memory dumps I tried to attach to people.m.o:
>>
>> http://people.mozilla.org/~jlebar/downloads/merged.json.xz
>> http://people.mozilla.org/~jlebar/downloads/unmerged.json.xz
>>
>> On Sun, Apr 21, 2013 at 7:51 PM, Justin Lebar  wrote:
>>> I think we should consider using much less JS in the parts of Gecko that are
>>> used in B2G.  I'd like us to consider writing new modules in C++ where
>>> possible, and I'd like us to consider rewriting existing modules in C++.
>>>
>>> I'm only proposing a change for modules which are enabled for B2G.  For 
>>> modules
>>> which aren't enabled on B2G, I'm not proposing any change.
>>>
>>> What I'd like to come out of this thread is a consensus one way or another 
>>> as
>>> to whether we continue along our current path of writing many features that 
>>> are
>>> enabled on B2G in JS, or whether we change course.
>>>
>>> Since most of these features implemented in JS seem to be DOM features, I'm
>>> particularly interested in the opinions of the DOM folks.  I'm also 
>>> interested
>>> in the opinions of JS folks, particularly those who know about the memory 
>>> usage
>>> of our new JITs.
>>>
>>> In the remainder of this e-mail I'll first explain where our JS memory is
>>> going.  Then I'll address two arguments that might be made against my 
>>> proposal
>>> to use more C++.  Finally, I'll conclude by suggesting a plan of action.
>>>
>>> === Data ===
>>>
>>> Right now about 50% (16mb) of the memory used by the B2G main process
>>> immediately after rebooting is JS.   It is my hypothesis that we could 
>>> greatly
>>> reduce this by converting modules to C++.
>>>
>>> On our 256mb devices, we have about 120mb available to Gecko, so this 16mb
>>> represents 13% of all memory available to B2G.
>>>
>>> To break down the 16mb of JS memory, 8mb is from four workers: ril_worker,
>>> net_worker, wifi_worker (x2).  5mb of the 8mb is under "unused-arenas"; 
>>> this is
>>> fragmentation in the JS heap.  Based on my experience tackling 
>>> fragmentation in
>>> the jemalloc heap, I suspect reducing this would be difficult.  But even if 
>>> we
>>> eliminated all of the fragmentation, we'd still be spending 3mb on these 
>>> four
>>> workers, which I think is likely far more than we need.
>>>
>>> The other 8mb is everything else in the system compartment (all our JSMs,
>>> XPCOM components, etc).  In a default B2G build you don't get a lot of 
>>> insight
>>> into this, because most of the system compartments are squished together to 
>>> save
>>> memory (bug 798491).  If I set jsloader.reuseGlobal to false, the amount of
>>> memory used increases from 8mb to 15mb, but now we c

Re: Rethinking the amount of system JS we use in Gecko on B2G

[Justin Lebar]

Of course attachments don't work great on newsgroups.  I've uploaded
the about:memory dumps I tried to attach to people.m.o:

http://people.mozilla.org/~jlebar/downloads/merged.json.xz
http://people.mozilla.org/~jlebar/downloads/unmerged.json.xz

On Sun, Apr 21, 2013 at 7:51 PM, Justin Lebar  wrote:
> I think we should consider using much less JS in the parts of Gecko that are
> used in B2G.  I'd like us to consider writing new modules in C++ where
> possible, and I'd like us to consider rewriting existing modules in C++.
>
> I'm only proposing a change for modules which are enabled for B2G.  For 
> modules
> which aren't enabled on B2G, I'm not proposing any change.
>
> What I'd like to come out of this thread is a consensus one way or another as
> to whether we continue along our current path of writing many features that 
> are
> enabled on B2G in JS, or whether we change course.
>
> Since most of these features implemented in JS seem to be DOM features, I'm
> particularly interested in the opinions of the DOM folks.  I'm also interested
> in the opinions of JS folks, particularly those who know about the memory 
> usage
> of our new JITs.
>
> In the remainder of this e-mail I'll first explain where our JS memory is
> going.  Then I'll address two arguments that might be made against my proposal
> to use more C++.  Finally, I'll conclude by suggesting a plan of action.
>
> === Data ===
>
> Right now about 50% (16mb) of the memory used by the B2G main process
> immediately after rebooting is JS.   It is my hypothesis that we could greatly
> reduce this by converting modules to C++.
>
> On our 256mb devices, we have about 120mb available to Gecko, so this 16mb
> represents 13% of all memory available to B2G.
>
> To break down the 16mb of JS memory, 8mb is from four workers: ril_worker,
> net_worker, wifi_worker (x2).  5mb of the 8mb is under "unused-arenas"; this 
> is
> fragmentation in the JS heap.  Based on my experience tackling fragmentation 
> in
> the jemalloc heap, I suspect reducing this would be difficult.  But even if we
> eliminated all of the fragmentation, we'd still be spending 3mb on these four
> workers, which I think is likely far more than we need.
>
> The other 8mb is everything else in the system compartment (all our JSMs,
> XPCOM components, etc).  In a default B2G build you don't get a lot of insight
> into this, because most of the system compartments are squished together to 
> save
> memory (bug 798491).  If I set jsloader.reuseGlobal to false, the amount of
> memory used increases from 8mb to 15mb, but now we can see where it's going.
>
> The list of worst offenders follows, but because this data was collected with
> reuseGlobal turned off, apply generous salt.
>
>   0.74 MB modules/Webapps.jsm
>   0.59 MB anonymous sandbox from devtools/dbg-server.jsm:41
>   0.53 MB components/SettingsManager.js
>   0.53 MB chrome://browser/content/shell.xul
>   0.49 MB components/WifiWorker.js
>   0.43 MB modules/DOMRequestHelper.jsm
>   0.38 MB modules/XPCOMUtils.jsm
>   0.34 MB RadioInterfaceLayer.js
>   0.31 MB AppsUtils.jsm
>   0.27 MB Webapps.js
>   0.22 MB BrowserElementParent.jsm
>   0.21 MB app://system.gaiamobile.org/index.html
>
> Many (but certainly not all) of these modules could be rewritten in C++.
>
> Beyond this list, it's death by a thousand cuts; there are 100 compartments in
> there, and they each cost a small amount.
>
> I've attached two about:memory dumps collected on my hamachi device soon after
> reboot, so you can examine the situation more closely, if you like.
> merged.json was collected with the default config, and unmerged.json was
> collected with jsloader.reuseGlobal set to false.
>
> Download and extract these files and then open them with the button at
> the bottom
> of about:memory in Nightly.
>
> (Before you ask: Most of the heap-unclassified in these dumps is
> graphics memory,
> allocated in drivers.)
>
> === Should we use JS because it's nicer than C++? ===
>
> I recognize that in many ways JS is a more convenient language than C++.  But
> that's besides the point here.  The point is that in the environment we're
> targeting, our implementation of JS is too heavyweight.  We can either fix our
> implementation or use less JS, but we can't continue using as much JS as we
> like without doing one of these two things.
>
> === Why not just make JS slimmer? ===
>
> It's been suggested to me that instead of converting existing and future JS
> code to C++, we should focus on making our JS engine slimmer.  Such changes
> would of course have the advantage o

Rethinking the amount of system JS we use in Gecko on B2G

[Justin Lebar]

I think we should consider using much less JS in the parts of Gecko that are
used in B2G.  I'd like us to consider writing new modules in C++ where
possible, and I'd like us to consider rewriting existing modules in C++.

I'm only proposing a change for modules which are enabled for B2G.  For modules
which aren't enabled on B2G, I'm not proposing any change.

What I'd like to come out of this thread is a consensus one way or another as
to whether we continue along our current path of writing many features that are
enabled on B2G in JS, or whether we change course.

Since most of these features implemented in JS seem to be DOM features, I'm
particularly interested in the opinions of the DOM folks.  I'm also interested
in the opinions of JS folks, particularly those who know about the memory usage
of our new JITs.

In the remainder of this e-mail I'll first explain where our JS memory is
going.  Then I'll address two arguments that might be made against my proposal
to use more C++.  Finally, I'll conclude by suggesting a plan of action.

=== Data ===

Right now about 50% (16mb) of the memory used by the B2G main process
immediately after rebooting is JS.   It is my hypothesis that we could greatly
reduce this by converting modules to C++.

On our 256mb devices, we have about 120mb available to Gecko, so this 16mb
represents 13% of all memory available to B2G.

To break down the 16mb of JS memory, 8mb is from four workers: ril_worker,
net_worker, wifi_worker (x2).  5mb of the 8mb is under "unused-arenas"; this is
fragmentation in the JS heap.  Based on my experience tackling fragmentation in
the jemalloc heap, I suspect reducing this would be difficult.  But even if we
eliminated all of the fragmentation, we'd still be spending 3mb on these four
workers, which I think is likely far more than we need.

The other 8mb is everything else in the system compartment (all our JSMs,
XPCOM components, etc).  In a default B2G build you don't get a lot of insight
into this, because most of the system compartments are squished together to save
memory (bug 798491).  If I set jsloader.reuseGlobal to false, the amount of
memory used increases from 8mb to 15mb, but now we can see where it's going.

The list of worst offenders follows, but because this data was collected with
reuseGlobal turned off, apply generous salt.

  0.74 MB modules/Webapps.jsm
  0.59 MB anonymous sandbox from devtools/dbg-server.jsm:41
  0.53 MB components/SettingsManager.js
  0.53 MB chrome://browser/content/shell.xul
  0.49 MB components/WifiWorker.js
  0.43 MB modules/DOMRequestHelper.jsm
  0.38 MB modules/XPCOMUtils.jsm
  0.34 MB RadioInterfaceLayer.js
  0.31 MB AppsUtils.jsm
  0.27 MB Webapps.js
  0.22 MB BrowserElementParent.jsm
  0.21 MB app://system.gaiamobile.org/index.html

Many (but certainly not all) of these modules could be rewritten in C++.

Beyond this list, it's death by a thousand cuts; there are 100 compartments in
there, and they each cost a small amount.

I've attached two about:memory dumps collected on my hamachi device soon after
reboot, so you can examine the situation more closely, if you like.
merged.json was collected with the default config, and unmerged.json was
collected with jsloader.reuseGlobal set to false.

Download and extract these files and then open them with the button at
the bottom
of about:memory in Nightly.

(Before you ask: Most of the heap-unclassified in these dumps is
graphics memory,
allocated in drivers.)

=== Should we use JS because it's nicer than C++? ===

I recognize that in many ways JS is a more convenient language than C++.  But
that's besides the point here.  The point is that in the environment we're
targeting, our implementation of JS is too heavyweight.  We can either fix our
implementation or use less JS, but we can't continue using as much JS as we
like without doing one of these two things.

=== Why not just make JS slimmer? ===

It's been suggested to me that instead of converting existing and future JS
code to C++, we should focus on making our JS engine slimmer.  Such changes
would of course have the advantage of improving our handling of web content on
B2G.

I'm absolutely in favor of reducing JS memory usage, but I see this effort as
orthogonal to the question of rewriting our current code and writing our future
code in C++, for a few reasons.

1. Content JS does not run in the B2G main process, where the impact of high
memory usage is strongest.  We can probably tolerate higher memory usage for
content JS than we can for main-process code.  I think it makes sense for our
JS team to focus their effort on optimizing for content JS, since that's far
more widespread.

2. We have a large team of B2G engineers, some of whom could work exclusively
on converting components from JS to C++.  In contrast, we have a relatively
small team of JS engineers, few of whom can work exclusively on optimizing the
JS engine for B2G's use-cases.

3. I know people get in trouble at Mozilla for suggesting tha

Re: Automatic tree clobbering is coming

[Justin Lebar]

> Could the
> anti-auto-clobberers please explain where this analogy falls apart for your
> workflow?

As a simple example, I often will cd objdir and then do
dist/bin/firefox | tee out, to save stdout to a file.  I could do "tee
../out", but then my log file is not associated with the objdir, which
I might want.

If the objdir were a hidden directory managed entirely by the build
system, deleting it automatically would be one thing.  But it's not --
it's a directory that developers interact with all the time.

On Thu, Apr 18, 2013 at 1:12 AM, Gregory Szorc  wrote:
> I agree that we should consider a compromise regarding the UI/UX of auto
> clobber. I have filed bug 863091.
>
> I would like to say that I view the object directory as a cache of the
> output of the build system. Since it's a cache, cache rules apply and data
> may disappear at any time. This analogy works well for my developer workflow
> - I never put anything not derived from the build system in my object
> directory. But, I don't know what other people are doing. Could the
> anti-auto-clobberers please explain where this analogy falls apart for your
> workflow?
>
>
> On 4/17/13 3:36 PM, Justin Lebar wrote:
>>
>> I think the possibility of deleting user data should be taken
>> seriously.  Exactly who is doing the deletion (configure vs. make) is
>> immaterial.  It's also not right to argue that since a majority of
>> users don't expect to lose data, it's OK to silently delete data for a
>> minority of them.
>>
>> I think we should either opt in to auto-clobbering or prompt to
>> clobber by default and allow users to opt out of the prompt.
>>
>> On Thu, Apr 18, 2013 at 12:18 AM, Ralph Giles  wrote:
>>>
>>> On 13-04-17 12:36 PM, Gregory Szorc wrote:
>>>
>>>> It /could/, sure. However, I consider auto clobbering a core build
>>>> system feature (sheriffs were very vocal about wanting it). As such, it
>>>> needs to be part of client.mk. (Please correct me if I am wrong.)
>>>
>>>
>>> Ok. A makefile deleting things is less surprising that an a configure
>>> script doing so.
>>>
>>>   -r
>>> ___
>>> dev-platform mailing list
>>> dev-platform@lists.mozilla.org
>>> https://lists.mozilla.org/listinfo/dev-platform
>
>
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Automatic tree clobbering is coming

[Justin Lebar]

I think the possibility of deleting user data should be taken
seriously.  Exactly who is doing the deletion (configure vs. make) is
immaterial.  It's also not right to argue that since a majority of
users don't expect to lose data, it's OK to silently delete data for a
minority of them.

I think we should either opt in to auto-clobbering or prompt to
clobber by default and allow users to opt out of the prompt.

On Thu, Apr 18, 2013 at 12:18 AM, Ralph Giles  wrote:
> On 13-04-17 12:36 PM, Gregory Szorc wrote:
>
>> It /could/, sure. However, I consider auto clobbering a core build
>> system feature (sheriffs were very vocal about wanting it). As such, it
>> needs to be part of client.mk. (Please correct me if I am wrong.)
>
> Ok. A makefile deleting things is less surprising that an a configure
> script doing so.
>
>  -r
> ___
> dev-platform mailing list
> dev-platform@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-platform
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Annotating Commits

[Justin Lebar]

> It bothers me that we're cluttering up our commit messages with
> ephemeral data unrelated to the actual change. DONTBUILD and CLOSED
> TREE are the worst offenders.

What if we asked people to put DONTBUILD / CLOSED TREE in a new line
at the bottom of their commit message?  Most of the time we look at
only the first line of the commit message, so it seems to me this
would eliminate most of the aesthetic offense, without requiring us to
develop entirely new tools for interacting with our SCM.

This would have the added benefit of teaching people that they can
write multiline commit messages, which is something I hope we can all
agree on.  :)

On Thu, Apr 11, 2013 at 12:48 PM, Steve Fink  wrote:
> It bothers me that we're cluttering up our commit messages with
> ephemeral data unrelated to the actual change. DONTBUILD and CLOSED
> TREE are the worst offenders. I would also like to have
> machine-readable tags for regular push vs bustage fix vs backout vs
> merge, because they would be useful for computing statistics that could
> then drive automation optimizations. But I think using push metadata
> for all of this is a non-starter, both for new and existing
> contributors.
>
> If I were to propose something, it would probably be (1) define a
> format for metadata that could be placed after the first line of a
> commit message, (2) make the automation check push metadata (the
> built-in kind) for DONTBUILD and CLOSED TREE, and (3) make tbpl expose
> DONTBUILD/CLOSED TREE and perhaps backout metadata in the UI. The value
> of #2 is debatable, since you can already put it after the first line
> and it's fairly easily ignorable.
>
> If you wanted to ensure that #1 serves its purpose, you might also want
> to implement a push hook that requires that if the metadata is present,
> then it must be properly formatted (and semantically valid, to the
> extend that it's automatically checkable.)
>
> I can also see the argument for externalizing this metadata, since
> there are some related things that are not known at the time of commit
> -- whether it crashed and burned on landing, for example.
>
> ___
> dev-platform mailing list
> dev-platform@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-platform
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Landing on Trunk - Target Milestone vs. Status Flag Usage vs. Tracking for Future Releases

[Justin Lebar]

Right now the status and tracking flags for a version get hidden when
that version becomes old.  If we switched away from using
target-milestone, we'd need to prevent this from happening.

On Wed, Apr 10, 2013 at 4:53 PM, Alex Keybl  wrote:
>> * The need for a particular team to track the concept of "We would
>>   like to get this fixed in this Firefox release." Some teams I've
>>   worked with have considered using the target milestone field here,
>>   but that collides with the trunk management definition, which often
>>   causes contention during the landing or post landing process.
>
> We may want to go into more detail here and see if we can change the trunk 
> management details in such a way that teams own the target milestone until 
> the bug is resolved/fixed.
>
>> * The need to be able to query the status-firefox flags to determine
>>   when a patch lands on a per branch basis even if has landed on
>>   trunk. This helps for those tracking fixes to use one universal flag
>>   to query on to determine such things such as:
>
> This does add one extra step, since the person querying needs to include 
> target milestone and bug resolution in their query. I'd like to hear if this 
> has been a major pain point for others before reacting.
>
>> Knowing this, why not consider just using the status-flags purely to track 
>> landings and let the team determine how to use target milestone? Also, why 
>> not set the status-flag in general for the appropriate Firefox release when 
>> a patch lands on trunk?
>
> The difference between version-specific status flags (fixed) and target 
> milestone (along with resolved/fixed) is subtle. Status flags being set to 
> fixed mean this bug has been fixed in this particular version of Firefox. The 
> combination of a target milestone and resolved/fixed means that a bug is 
> fixed in all Firefox versions after the one specified. That may prove a 
> little difficult to untangle.
>
> -Alex
>
> On Apr 10, 2013, at 1:38 PM, Jason Smith  wrote:
>
>> Hi Everyone,
>>
>> Right now, when a landing occurs, my understanding is that we're typically 
>> setting the target milestone field to what Firefox release the code lands on 
>> if it lands on trunk. If a patch is uplifted, then the status flag is set 
>> appropriately for the target Firefox release.
>>
>> However, after working with three distinct teams, talking with Ed, and 
>> analyzing status flag usage, I have seen these contention points dependent 
>> on a team's method for tracking that might happen:
>>
>> * The need for a particular team to track the concept of "We would
>>   like to get this fixed in this Firefox release." Some teams I've
>>   worked with have considered using the target milestone field here,
>>   but that collides with the trunk management definition, which often
>>   causes contention during the landing or post landing process.
>> * The need to be able to query the status-firefox flags to determine
>>   when a patch lands on a per branch basis even if has landed on
>>   trunk. This helps for those tracking fixes to use one universal flag
>>   to query on to determine such things such as:
>> o What bugs that are features are worth verifying on X branch?
>> o What bugs that defects are worth verifying on X branch?
>>
>> Knowing this, why not consider just using the status-flags purely to track 
>> landings and let the team determine how to use target milestone? Also, why 
>> not set the status-flag in general for the appropriate Firefox release when 
>> a patch lands on trunk?
>>
>> --
>> Sincerely,
>> Jason Smith
>>
>> Desktop QA Engineer
>> Mozilla Corporation
>> https://quality.mozilla.com
>>
>> ___
>> dev-platform mailing list
>> dev-platform@lists.mozilla.org
>> https://lists.mozilla.org/listinfo/dev-platform
>
> ___
> dev-platform mailing list
> dev-platform@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-platform
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Virtual Memory fragmentation issues

[Justin Lebar]

> I see, so the hypothesis is that 100% of the fragmentation is coming from
> VirtualAlloc/MapViewOfFile, not from allocations in general?

jemalloc does not make 4kb allocations, I think ever.  So yes.

On Tue, Apr 9, 2013 at 9:23 AM, Kevin Gadd  wrote:
> I see, so the hypothesis is that 100% of the fragmentation is coming from
> VirtualAlloc/MapViewOfFile, not from allocations in general? I believe the
> kernel instrumentation tools could give you data on this but it would
> probably require modifications to the profiler tool, so maybe it can't be
> used to answer questions about 'who allocated this'.
>
> It does appear that MS offers a utility for this, though, called LeakDiag:
> http://blogs.jetbrains.com/yole/archives/34.html
>
> It appears to instrument VirtualAlloc and record tracebacks. Maybe worth a
> look.
>
>
> On Tue, Apr 9, 2013 at 6:07 AM, Benjamin Smedberg 
> wrote:
>>
>> On 4/9/13 7:12 AM, Kevin Gadd wrote:
>>>
>>> This is one of the problems Heap Profiler was written to solve:
>>>
>>> https://github.com/kevingadd/HeapProfiler
>>>
>>> It enables stack trace captures at allocation time, so at any given point
>>> in the capture you can open the heap and see who allocated a given block
>>> of
>>> memory. It can also display a fragmentation summary.
>>>
>>> It would probably be trivial to extend it to do a fragmentation analysis
>>> and generate a summary of who's causing the most fragmentation (by module
>>> or function, most likely).
>>
>> As far as I understand it, this tool profiles heap allocations like
>> "malloc". I am not concerned with those allocations, but only with page
>> allocations. Can this tool intercept calls to VirtualAlloc and especially
>> MapViewOfFile so that we can see the stack at the point the large memory
>> mappings are created?
>>
>> Note that although it's interesting that we're fragmenting the page
>> because of the 4k/64k behavior, I am much less concerned with that behavior
>> than I am with the leaked memory mappings.
>>
>> --BDS
>>
>
>
>
> --
> -kg
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Virtual Memory fragmentation issues

[Justin Lebar]

> AIUI, on Windows the smallest block you can ask for with VirtualAlloc
> is 4 KiB.  However, no more than one VirtualAlloc block can exist per
> 64 KiB chunk.  So if you ask for 4 KiB you'll end up wasting the
> remaining 60 KiB of address space in the 64 KiB chunk.

Awesome memory, Nick.

MSDN seems to confirm this.

"The second parameter [to VirtualAlloc] indicates the range of
addresses the function should allocate. This value can be anywhere
from one page to 2 GB in size, but VirtualAlloc is actually
constrained to a smaller range than that. The minimum size that can be
reserved is 64K, and the maximum that can be reserved is the largest
contiguous range of free addresses in the process. Requesting one page
of reserved addresses results in a 64K address range."

http://msdn.microsoft.com/en-us/library/ms810627.aspx

We have the ability to intercept VirtualAlloc calls on Windows, so
assuming that works for these particular VirtualAlloc calls (it
should?) perhaps we can take backtraces and write them to a file.


On Mon, Apr 8, 2013 at 7:57 PM, Nicholas Nethercote
 wrote:
> On Mon, Apr 8, 2013 at 4:46 PM, Benjamin Smedberg  
> wrote:
>>
>> 1) early in the memory info, there appears to be a common pattern of a
>> committed block and then 15 free blocks.
>
> AIUI, on Windows the smallest block you can ask for with VirtualAlloc
> is 4 KiB.  However, no more than one VirtualAlloc block can exist per
> 64 KiB chunk.  So if you ask for 4 KiB you'll end up wasting the
> remaining 60 KiB of address space in the 64 KiB chunk.
>
> Assuming I'm right about this (I'm fairly certain, but not 100%), it
> sounds like some code (be it Mozilla or driver code) is
> VirtualAlloc'ing lots of 4 KiB blocks, and thus wasting lots of
> address space.
>
> Nick
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Proposal for using a multi-headed tree instead of inbound

[Justin Lebar]

> But can we do this with rebased changesets instead of "trivial" merge 
> changesets?

Personally, I think merges are a Very Good idea because they bake into
the tree information that is currently contained only in the pushlog.
This improves bisect from my perspective, although I'll grant it might
not improve other tools.  Merges also preserve the differences between
the code I wrote and the code that eventually got merged in after
conflicts were resolved, perhaps by the sheriff.

But perhaps we should table this discussion (and some of the other
ones in this thread) until after we decide that this is a good idea.

On Wed, Apr 3, 2013 at 7:49 PM, Jesse Ruderman  wrote:
> +1.
>
> But can we do this with rebased changesets instead of "trivial" merge
> changesets?  While the core of hg can handle merges, pretty much none of the
> tools we rely on for understanding history (hg {log, grep, diff, bisect})
> handle them well.
>
> ___
> dev-platform mailing list
> dev-platform@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-platform
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Proposal for using a multi-headed tree instead of inbound

[Justin Lebar]

If anything this should improve the experience of bisecting, because
you'll be able to bisect known-good csets on m-c and only at the end
step in to the merge csets which may or may not be good.

Right now we say that when people push a patch queue to m-c every
patch should be green, but in practice that's often not the case.
Right now when you bisect you have no way that I'm aware of to say
"bisect only the tips of pushes", because hg doesn't store that
information, so you end up bisecting through many bad csets.

On Wed, Apr 3, 2013 at 6:43 PM, Gary Kwong  wrote:
>> Another potential problem with this approach is that we will have more
>> merge changes in m-c, which generally screws with hg bisect. Personally
>> I already have enough trouble with hg bisect to the point where I don't
>> use it because I can't trust it. This may be a legitimate problem for
>> some, but it's not for me.
>
>
> autoBisect relies heavily on `hg bisect` and has been working well with
> JSBugMon especially in Spidermonkey for the better part of ~ 4 years now.
>
> -Gary
>
> ___
> dev-platform mailing list
> dev-platform@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-platform
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Proposal for using a multi-headed tree instead of inbound

[Justin Lebar]

This is a really interesting idea.

git is smart enough to let you pull only the csets that end up getting
merged into master.  I don't know how multiple heads works with hg,
but I wonder if we could make hg smart enough to do the same thing.

Otherwise, maybe it's time to switch to git.  

On Wed, Apr 3, 2013 at 6:24 PM, Gregory Szorc  wrote:
> On 4/3/13 2:31 PM, Kartikaya Gupta wrote:
>
> Excellent write-up! I think a re-examination of our tree management is long
> overdue, especially with all the recent closures on inbound.
>
>
>> My suggested process *requires* a tree which allows multiple heads,
>> which is why I suggest "try" instead of "inbound". It has been suggested
>> that hg does not deal well with tree with large numbers of heads, which
>> is why try is reset every so often. In my proposal, we can still reset
>> try periodically, as long as the changes have been merged to m-c. M-c
>> itself will always have only one head.
>
>
> We could consider Mercurial bookmarks for this. (I believe bookmarks weren't
> ready for prime time when we established our current tree management
> policy.)
>
> There are some interesting things you could do with bookmarks. e.g. you
> could have the convention that all pushed bookmarks following a specific
> naming convention (e.g. autoland/bug123456) are automatically considered for
> "cherry-picking" to mainline.
>
> While I concede that Mercurial's UI around bookmarks isn't as good as say
> Git branches (e.g. |hg push| by default pushes all bookmarks and AFAIK there
> is no config option to disable that - you need to know to specify -r
> ), one of the things that makes Mercurial special is that it is
> highly extensible. We could consider maintaining a Mercurial extension that
> forces proper use of our ordained tree policy. Combine this with some push
> hooks that reject invalid pushes and we should be set!
>
> ___
> dev-platform mailing list
> dev-platform@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-platform
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Unable to run TPS tests

[Justin Lebar]

I don't know, actually.  You can ask on #developers, but I'd just run
'em all.  :)

On Wed, Apr 3, 2013 at 12:54 PM, Raymond Lee
 wrote:
> Thanks Justin!  Can you suggest what try syntax I can use please?  I don't
> see a TPS option in the try syntax builder page.
> http://trychooser.pub.build.mozilla.org/
>
> On 3 Apr, 2013, at 11:47 PM, Justin Lebar  wrote:
>
> In general you'll have much more success running these benchmarks on
> tryserver rather than trying to run them locally.  Even if you got the
> test working, there's no guarantee that your local benchmark results
> will have any bearing on the benchmark results on our servers.  (In
> particular, the servers are configured to reduce the noise in these
> results.)
>
> On Wed, Apr 3, 2013 at 3:15 AM,   wrote:
>
> Hi all
>
> I am trying to run TPS to ensure my patch works for bug 852041
> (https://bugzilla.mozilla.org/show_bug.cgi?id=852041)
>
> However, I got some errors when I ran the following.  Could someone give me
> some suggestions how to fix it please?
>
> 1. source /Users/raymond/Documents/virtualenv/bin/activate
> 2. runtps --binary=/Users/raymond/Documents/mozilla-central/obj-ff-dbg/
>
> == START ==
> using result file tps_result.json
> ['/Users/raymondlee/Documents/appcoast/mozilla-central2/obj-ff-dbg/',
> '-profile',
> '/var/folders/43/d4b7hbz56jn2gvhmdhtrytxwgn/T/tmpozAsVT.mozrunner',
> '-tps', '/var/folders/43/d4b7hbz56jn2gvhmdhtrytxwgn/T/tps_test_Uwxl4Y',
> '-tpsphase', u'1', '-tpslogfile',
> '/Users/raymondlee/Documents/appcoast/mozilla-central2/obj-ff-dbg/dist/tps.log']
> Traceback (most recent call last):
>  File
> "/Users/raymondlee/Documents/appcoast/virtualenv/lib/python2.7/site-packages/tps-0.4-py2.7.egg/tps/testrunner.py",
> line 331, in run_tests
>self.run_test_group()
>  File
> "/Users/raymondlee/Documents/appcoast/virtualenv/lib/python2.7/site-packages/tps-0.4-py2.7.egg/tps/testrunner.py",
> line 405, in run_test_group
>result = self.run_single_test(testdir, test)
>  File
> "/Users/raymondlee/Documents/appcoast/virtualenv/lib/python2.7/site-packages/tps-0.4-py2.7.egg/tps/testrunner.py",
> line 223, in run_single_test
>phase.run()
>  File
> "/Users/raymondlee/Documents/appcoast/virtualenv/lib/python2.7/site-packages/tps-0.4-py2.7.egg/tps/phase.py",
> line 49, in run
>profile=self.profile)
>  File
> "/Users/raymondlee/Documents/appcoast/virtualenv/lib/python2.7/site-packages/tps-0.4-py2.7.egg/tps/firefoxrunner.py",
> line 90, in run
>self.runner.start()
>  File
> "/Users/raymondlee/Documents/appcoast/virtualenv/lib/python2.7/site-packages/mozrunner-5.15-py2.7.egg/mozrunner/runner.py",
> line 183, in start
>self.process_handler.run(timeout, outputTimeout)
>  File
> "/Users/raymondlee/Documents/appcoast/virtualenv/lib/python2.7/site-packages/mozprocess-0.9-py2.7.egg/mozprocess/processhandler.py",
> line 621, in run
>self.proc = self.Process(self.cmd, **args)
>  File
> "/Users/raymondlee/Documents/appcoast/virtualenv/lib/python2.7/site-packages/mozprocess-0.9-py2.7.egg/mozprocess/processhandler.py",
> line 76, in __init__
>universal_newlines, startupinfo, creationflags)
>  File
> "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py",
> line 679, in __init__
>errread, errwrite)
>  File
> "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py",
> line 1249, in _execute_child
>raise child_exception
> OSError: [Errno 13] Permission denied
> Test Summary
>
> Exception AttributeError: "'ProcessHandler' object has no attribute 'proc'"
> in  object at 0x1088e4e90>> ignored
>
> == END ==
>
> Thanks
> Raymond
> ___
> dev-platform mailing list
> dev-platform@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-platform
>
>
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Unable to run TPS tests

[Justin Lebar]

In general you'll have much more success running these benchmarks on
tryserver rather than trying to run them locally.  Even if you got the
test working, there's no guarantee that your local benchmark results
will have any bearing on the benchmark results on our servers.  (In
particular, the servers are configured to reduce the noise in these
results.)

On Wed, Apr 3, 2013 at 3:15 AM,   wrote:
> Hi all
>
> I am trying to run TPS to ensure my patch works for bug 852041 
> (https://bugzilla.mozilla.org/show_bug.cgi?id=852041)
>
> However, I got some errors when I ran the following.  Could someone give me 
> some suggestions how to fix it please?
>
> 1. source /Users/raymond/Documents/virtualenv/bin/activate
> 2. runtps --binary=/Users/raymond/Documents/mozilla-central/obj-ff-dbg/
>
> == START ==
> using result file tps_result.json
> ['/Users/raymondlee/Documents/appcoast/mozilla-central2/obj-ff-dbg/', 
> '-profile', 
> '/var/folders/43/d4b7hbz56jn2gvhmdhtrytxwgn/T/tmpozAsVT.mozrunner', 
> '-tps', '/var/folders/43/d4b7hbz56jn2gvhmdhtrytxwgn/T/tps_test_Uwxl4Y', 
> '-tpsphase', u'1', '-tpslogfile', 
> '/Users/raymondlee/Documents/appcoast/mozilla-central2/obj-ff-dbg/dist/tps.log']
> Traceback (most recent call last):
>   File 
> "/Users/raymondlee/Documents/appcoast/virtualenv/lib/python2.7/site-packages/tps-0.4-py2.7.egg/tps/testrunner.py",
>  line 331, in run_tests
> self.run_test_group()
>   File 
> "/Users/raymondlee/Documents/appcoast/virtualenv/lib/python2.7/site-packages/tps-0.4-py2.7.egg/tps/testrunner.py",
>  line 405, in run_test_group
> result = self.run_single_test(testdir, test)
>   File 
> "/Users/raymondlee/Documents/appcoast/virtualenv/lib/python2.7/site-packages/tps-0.4-py2.7.egg/tps/testrunner.py",
>  line 223, in run_single_test
> phase.run()
>   File 
> "/Users/raymondlee/Documents/appcoast/virtualenv/lib/python2.7/site-packages/tps-0.4-py2.7.egg/tps/phase.py",
>  line 49, in run
> profile=self.profile)
>   File 
> "/Users/raymondlee/Documents/appcoast/virtualenv/lib/python2.7/site-packages/tps-0.4-py2.7.egg/tps/firefoxrunner.py",
>  line 90, in run
> self.runner.start()
>   File 
> "/Users/raymondlee/Documents/appcoast/virtualenv/lib/python2.7/site-packages/mozrunner-5.15-py2.7.egg/mozrunner/runner.py",
>  line 183, in start
> self.process_handler.run(timeout, outputTimeout)
>   File 
> "/Users/raymondlee/Documents/appcoast/virtualenv/lib/python2.7/site-packages/mozprocess-0.9-py2.7.egg/mozprocess/processhandler.py",
>  line 621, in run
> self.proc = self.Process(self.cmd, **args)
>   File 
> "/Users/raymondlee/Documents/appcoast/virtualenv/lib/python2.7/site-packages/mozprocess-0.9-py2.7.egg/mozprocess/processhandler.py",
>  line 76, in __init__
> universal_newlines, startupinfo, creationflags)
>   File 
> "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py",
>  line 679, in __init__
> errread, errwrite)
>   File 
> "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py",
>  line 1249, in _execute_child
> raise child_exception
> OSError: [Errno 13] Permission denied
> Test Summary
>
> Exception AttributeError: "'ProcessHandler' object has no attribute 'proc'" 
> in  object at 0x1088e4e90>> ignored
>
> == END ==
>
> Thanks
> Raymond
> ___
> dev-platform mailing list
> dev-platform@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-platform
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Awesome "Quantile" Telemetry Plots on metrics.mozilla.com

[Justin Lebar]

https://bugzilla.mozilla.org/show_bug.cgi?id=699670

On Tue, Apr 2, 2013 at 11:58 AM, Patrick McManus  wrote:
> Today I noticed some (relatively) new CDF plots of telemetry histogram
> data on metrics.mozilla.com. Maybe in the last week or so?
>
> This makes it much easier to determine medians and 90th percentiles -
> which is a very common use case for me. If you haven't seen it I
> recommend checking it out.
>
> If, dear reader, you are responsible then thank you! I didn't know who to 
> thank.
>
> -Patrick
> ___
> dev-platform mailing list
> dev-platform@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-platform
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: T pushes to try

[Justin Lebar]

This is awesome, Steve.

> Adding either a link to that, or examples directly, to
> http://trychooser.pub.build.mozilla.org/ would be awesome...

https://bugzilla.mozilla.org/show_bug.cgi?id=855591

On Wed, Mar 27, 2013 at 9:23 PM, Boris Zbarsky  wrote:
> On 3/27/13 8:08 PM, Steve Fink wrote:
>>
>> Yes, it does depend on a mostly undocumented try syntax feature from bug
>> 802937. I've at least added these example pushes to
>> .
>
>
> Adding either a link to that, or examples directly, to
> http://trychooser.pub.build.mozilla.org/ would be awesome...
>
> -Boris
>
> ___
> dev-platform mailing list
> dev-platform@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-platform
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Moz2D Repository Creation

[Justin Lebar]

> hg-git (the tool we use to synchronize Mercurial and Git repos) supports
> subrepos. Although, I'm not sure how well it works.

Well, we should definitely figure this out before we move forward with
this plan.

If the hg support for git repos is decent, that might be a better way
to go, since then we'd have one fewer repo we needed to mirror for
B2G's purposes.  Assuming that hg-git is happy with git submodules.
:)

> I haven't met many Git users who enjoy submodules, myself included.

Git submodules suck, but so does importing library code via cp, as we
do now.  :-/
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Moz2D Repository Creation

[Justin Lebar]

> Since we believe if we go through with this it would be the first time we use 
> a true subrepository system for a component
> used in mozilla-central, we'd very much appreciate any thoughts or feedback 
> people might have on the idea.

Have you thought about how this hg subrepo will interact with the git
mirrors of m-c?

Using m-c git isn't just for contrarians these days: B2G relies
heavily on git clones of m-c.  So we need to make sure this will work
properly.

-Justin

On Wed, Mar 27, 2013 at 4:42 PM, Bas Schouten  wrote:
> Hi all,
>
> Over the past year we've increased our dependencies on Moz2D (formerly known 
> under the codename Azure), our new 2D rendering API. Currently we're using it 
> for canvas drawing on all platforms and content drawing on Windows where 
> using Direct2D, and in the near future we will be moving to using it for 
> content on all platforms.
>
> From the very beginning of Moz2D development we've taken care to make sure it 
> can be built outside of the rest of Gecko, having only dependencies on 
> several MFBT headers. This was done to reduce the barrier for external usage 
> and development, we've since seen this benefit us for example in Servo using 
> Moz2D as well. In addition to that it has helped development and bugfixing by 
> having a much faster workflow for building/testing with which bugs could be 
> located and fixes created.
>
> Going forward we're expanding on that by ensuring the stand-alone builds and 
> works on all platforms and becomes the defacto way of the 'first level' of 
> testing when implementing new features or fixing bugs in Moz2D. In addition 
> to that we're expanding on our existing stand-alone unittesting suite as well 
> as adding a peformance testing suite, which will include microbenchmarking 
> and support measuring performance on Moz2D drawing recordings (single-page 
> recording support has just landed on mozilla-inbound).
>
> When moving forward we have several goals for Moz2D when it comes to the 
> workflow:
>
> - Improve Moz2D development workflow by having faster turnaround time on 
> builds and tests (both local and Try)
> - Lower the barrier for external contributors, some people have already 
> expressed the desire to work on potential backends we do not wish to invest 
> in ourselves, but have been deterred by the complexity of the 
> checkout/building process.
> - Improve sharing between Servo/Gecko.
> - Reduce load on our infrastructure by building and testing only Moz2D 
> stand-alone when a change affects only Moz2D, both on regular builds and try.
>
> As the next step in moving towards these goals and optimally supporting them 
> the proposal is to move Moz2D into its own repository. We would use hg 
> subrepos for this purpose, you can read more about this here 
> (http://mercurial.selenic.com/wiki/Subrepository). The intention is that this 
> will be done in such a way that the workflow for developers not involved in 
> Moz2D development would essentially not change.
>
> A more detailed description of the proposal and the workflows involved can be 
> found here (https://wiki.mozilla.org/Platform/GFX/Moz2DSubrepository) for 
> those interested in the details. Since we believe if we go through with this 
> it would be the first time we use a true subrepository system for a component 
> used in mozilla-central, we'd very much appreciate any thoughts or feedback 
> people might have on the idea.
>
> Best regards,
> Bas
> ___
> dev-platform mailing list
> dev-platform@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-platform
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: TBPL job visibility policy - now documented

[Justin Lebar]

> 4) Runs on every push

I thought many tests don't run on every push to m-i, due to
coalescing.  (Just to be clear, I wish we could disable coalescing on
m-i!)

But also, PGO builds/tests are not run on every push to m-c.

-Justin

On Tue, Mar 26, 2013 at 10:10 AM, Ed Morley  wrote:
> (Please reply to dev.tree-management)
>
> Until now, the requirements for a new platform/test-suite to be shown in the
> default TBPL view have been scattered across many newsgroup discussions,
> bugs & IRC conversations, which understandably leads to surprise when
> developers working on bringing a new job type into our buildbot automation
> are told that it does not yet meet them.
>
> In order to make the existing criteria more discoverable, I have documented
> them at:
> https://wiki.mozilla.org/Sheriffing/Job_Visibility_Policy
>
> The new page also includes:
> * A bug template for requesting changes in visibility.
> * Tips for tracking a job if it is not shown in the default view.
> * A rough overview of planned tooling improvements to give us more
> flexibility in the future.
>
> If you spot any omissions or any of the requirements need rewording to be
> more clear, please let me know :-)
>
> Best wishes,
>
> Ed
> ___
> dev-platform mailing list
> dev-platform@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-platform
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Decoupling the build config via moz.build files (nuking all.js)

[Justin Lebar]

> 1) Preferences and all.js. We currently define most of the default
> preferences in /modules/libpref/src/init/all.js. There are things in there
> related to the browser, Necko, gfx, dom, etc. Prety much the kitchen sink.

> 2) Telemetry histograms. They are all defined in
> /toolkit/components/telemetry/Histograms.json. Same deal as all.js: bits from
> multiple modules all exist here.

If the proposal is that each module could have its own prefs and histograms
files, I am not convinced this would be an improvement over the current
situation.  At the very least, we should consider the downsides to splitting
these files up.

The putative advantages to splitting these files up are

> * including increased utility of |hg log|
> * easier code comprehension (since everything is self-contained).
> * faster builds (from splitting up Histograms.json, bug 831104).

I don't think splitting up the files would result in a big improvement in any
of these areas.  To address each in turn:

> * including increased utility of |hg log|

Splitting up the files would also decrease the utility of hg/git log.  Right
now it's easy to say "show me all pref changes in all.js."  But if we split
all.js up into many separate files, I'd have to do

  |git log foo/bar/prefs.js foo/baz/prefs.js ...|

Module boundaries are always blurry, so "show me all image pref changes"
wouldn't necessarily be as simple as |git log image/prefs.js|.  For example,
the pref for image locking is "content.image.allow_locking".  I think this is
an image pref, but obviously whoever wrote it thought it was a content pref.

> * easier code comprehension (since everything is self-contained).

I don't think this is a big advantage.  Modules are so large, the only way to
navigate around one is by searching.  Does the fact that I have to use my
editor's find command to navigate all.js really stand in the way of
comprehension?

Again splitting the files also makes code comprehension more difficult in some
circumstances.  Currently, if I want to find out a pref's value in b2g, I have
search through two files.  Under the proposed scheme, I'd have to search
through N files.  This is a net loss in my ability to comprehend the code.

> * faster builds (from splitting up Histograms.json, bug 831104).

We could accomplish these build speedups by putting the split-up
probe-definition files all in one directory (instead of spreading them around
the tree).  That would be fine with me, if the build speedups here are
something that we really want.

In practice, we have lots of headers that are included all over the tree.  Some
of them (e.g. nsContentUtils.h) are modified quite often.  I'd rather focus on
making "rebuild-the-world" builds faster (e.g. with PCH) than try to fix each
of these cases piecemeal.

In addition to not having large benefits, splitting up prefs and telemetry
would have considerable costs.  The two big ones which come to mind are:

* Increased difficulty figuring out "what's the default value of pref X on
  product Y?" because given a pref name it's not clear where it lives in the
  tree.  (Establishing a simple function mapping pref names to tree locations
  would require mass-modifying a lot of pref names, which would break users who
  have those prefs set to custom values, so I don't think we can do that.)

* What happens when two files define the same pref?  In order to be sane, we'd
  have to make that a build error.  But then because some modules can be
  disabled conditionally, some pref files will be disabled conditionally.  This
  introduces a whole new combinatorial way to break the build with weird
  configurations.

-Justin

On Sat, Mar 16, 2013 at 3:33 AM, Gregory Szorc  wrote:
> Our build config has a number of areas where metadata is centrally defined
> and a module or component's "configuration" is fragmented in the source
> tree. Here are some examples:
>
> 1) Preferences and all.js. We currently define most of the default
> preferences in /modules/libpref/src/init/all.js. There are things in there
> related to the browser, Necko, gfx, dom, etc. Prety much the kitchen sink.
>
> 2) Telemetry histograms. They are all defined in
> /toolkit/components/telemetry/Histograms.json. Same deal as all.js: bits
> from multiple modules all exist here.
>
> 3) xpcshell.ini master manifest. /testing/xpcshell/xpcshell.ini defines
> every xpcshell.ini that exists in the tree.
>
> I think all of these (and more) are sub-optimal because it creates a strong
> and possibly unwanted coupling between modules while fragmenting a
> module/component's code. In my ideal world, I'd like to see as much of the
> code as possible for a module/component be self-contained and loosely
> coupled from the rest of the tree. There are a number of reasons for this,
> including increased utility of |hg log| and easier code comprehension (since
> everything is self-contained). The latter likely results in less tree cruft
> over time. In case of Telemetry histograms, decoupling c

Re: proposal: replace talos with inline tests

[Justin Lebar]

> 1) something checked into mc anyone can easily author or run (for tracking 
> down regressions) without having to checkout a separate repo, or setup and 
> run a custom perf test framework.

I don't oppose the gist of what you're suggesting here, but please
keep in mind that small perf changes are often very difficult to track
down locally.  Small changes in system and toolchain configuration can
have large effects on average build speed and its variance.  For
example, I've found observable performance differences between Try and
m-c/m-i builds in the past (bug 653961), despite their build configs
being nearly identical.

In my experience, we spend the majority of our time trying to track
down small perf changes, so a change which makes it easier to track
down the source of large perf changes might not have an outsize
effect.

> 3) no releng overhead for setup of new perf tests. something that is built 
> into the test framework / infrastructure we set up.

If we did this, we'd need to figure out how and when to promote
benchmarks to "we care about them" status.

We already don't back back out changes for regressing a benchmark like
we back them  out for regressing tests.  I think this is at least
partially because a general sentiment that not all of our benchmarks
correlate strongly to what they're trying to measure.

I suspect if anyone could check in a benchmark, the average quality of
benchmarks would likely stay roughly the same, but the number of
benchmarks would increase.  In that case we'd have even more
benchmarks with spurious regressions to deal with.

-Justin
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: improving access to telemetry data

[Justin Lebar]

It sounds to me like people want both

1) Easier access to aggregated data so they can build their own
dashboards roughly comparable in features to the current dashboards.

2) Easier access to raw databases so that people can build up more
complex analyses, either by exporting the raw data from the db, or by
analyzing it in the db.

That is, I don't think we can or should export JSON with all the data
in our databases.  That is a lot of data.

On Thu, Feb 28, 2013 at 12:08 PM, Benjamin Smedberg
 wrote:
> On 2/28/2013 10:59 AM, Benoit Jacob wrote:
>>>
>>> Because the raw crash files do not include new metadata fields, this has
>>> led to weird engineering practices like shoving interesting metadata into
>>> the freeform app notes field, and then parsing that data back out later.
>>> I'm worried about perpetuating this kind of behavior, which is hard on
>>> the
>>> database and leads to very arcane queries in many cases.
>>>
>> I don't agree with the notion that freeform fields are bad. freeform plain
>> text is an amazing file format. It allows to add any kind of data without
>> administrative overhead and is still easy to parse (if the data was that
>> was added was formatted with easy parsing in mind).
>
> The obvious disadvantage is that it is much more difficult to
> machine-process. For example elasticsearch can't index on it (at least not
> without lots of custom parsing), and in general you can't ask tools like
> hbase or elasticsearch to filter on that without a user defined function.
> (Regexes might work for some kinds of text processing.)
>
>>
>> But if one considers it a bad thing that people use it, then one should
>> address the issues that are causing people to use it. As you mention, raw
>> crash files may not include newer metadata fields. So maybe that can be
>> fixed by making it easier or even automatable to include new fields in raw
>> crash files?
>
> Yes, that is all filed. We can't automatically include the field, because we
> don't know whether they are supposed to be public or private, but we should
> soon be able to have a dynamically updateable list.
>
> Note that if mcmanus is correct, we're going to be dealing with 1M fields
> per day here. That's a lot more than the 250k from crash-stats, especially
> because the payload is bigger. I believe that the flat files from
> crash-stats are a really useful kludge because we couldn't figure out a
> better way to expose the raw data. But that kludge will start to fall over
> pretty quickly, and perhaps we should just expose a better way to do queries
> using the databases, which are surprisingly good at doing these kinds of
> queries efficiently.
>
>
> --BDS
>
> ___
> dev-platform mailing list
> dev-platform@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-platform
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Please upgrade to at least Mercurial 2.5.1

[Justin Lebar]

> Can guidance be
> provided as to where to get such things for commonly-run versions of Linux?

It's also easy to install hg using pip or easy_install.  You can get
either of these tools using your distro's package manager.
(easy_install is usually called python-setuptools or something like
that.)  I think pip is the preferred method these days.

The advantage of using pip/easy_install is that you'll always get the
latest version of hg.  The disadvantage, of course, is that you're
using an additional package manager.

P.S. I'm working on a tree hook which will detect these busted csets.
Follow along at home at

https://bugzilla.mozilla.org/show_bug.cgi?id=843081

-Justin

On Thu, Feb 21, 2013 at 6:36 AM, Gervase Markham  wrote:
> On 20/02/13 16:06, Justin Lebar wrote:
>> The client bug that's fixed with the new version of hg is slowly and
>> irreversibly ruining our blame, so I don't think we should wait before
>> upgrading clients.
>
> The Mercurial download page:
> http://mercurial.selenic.com/downloads/
> offers 2.5.1 for Mac and Windows, but no Linux packages. Can guidance be
> provided as to where to get such things for commonly-run versions of Linux?
>
> Gerv
>
> ___
> dev-platform mailing list
> dev-platform@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-platform
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Please upgrade to at least Mercurial 2.5.1

[Justin Lebar]

> Don't we need to update our servers first due to phases? But either way,
> yes, this should be a bigger priority.

The client bug that's fixed with the new version of hg is slowly and
irreversibly ruining our blame, so I don't think we should wait before
upgrading clients.

Based on experience, blocking on IT rolling out an hg upgrade is
probably not a good idea if we want to go quickly here.  :-/

https://bugzilla.mozilla.org/show_bug.cgi?id=741353
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Increase in memory utilization on Mac OSX 10.7+ due to history swipe animations

[Justin Lebar]

This is kind of OT, but

> OTOH, if the computer has memory available, we should be using *all* of it 
> any place we can trade
> memory for speed.

We considered this idea in MemShrink some months ago, and we mostly
dropped it.  There are two essential problems that we weren't able to
overcome.

1) It's not always simple to determine how much memory the system has
available.  This is particularly true on Mac, as it turns out; bug
789975 says we can't even accurately measure our own memory usage on
Mac.

The issue is that the OS keeps some types of free'd pages in a
process's address space and only releases them when the system is very
low on memory, but it doesn't give us any way (that I'm aware of) to
measure how many of these pages there are.

2) More importantly, using as much RAM is available but no more makes
Firefox the nicest process on the system.  We can get into a situation
where, because some other program is hogging memory, the user's swipe
animations stop working as she expects.  That may look like a bug in
Firefox, and in any case it may not be what the user wants.

Doing this makes Firefox change its behavior based upon what other
processes on the system are doing, and I'm not convinced that's a safe
thing to do in general.

But this is a better discussion to have in the context of DDD than in
the context of this bug.

-Justin

On Wed, Feb 13, 2013 at 9:13 AM, Benjamin Smedberg
 wrote:
> On 2/13/2013 3:12 AM, Justin Lebar wrote:
>>
>> c) Consider adaptive techniques so that users who use this feature
>> heavily will store more screenshots (at the cost of more memory),
>> while those who don't use it won't pay a price.
>
> Apart from the other solution of re-rendering directly from bfcache, this
> seems like the obvious longer-term solution, and it ties in with Josh's
> proposal about "ddd". I don't think there's any way to make absolute
> judgements about whether 2.5MB or 5MB or 10MB is "reasonable" for a feature,
> and there's a lot of room for adaptive data-driven behavior here. If the
> computer is memory-constrained, then we should be evicting all sorts of
> memory-based caches pretty aggressively, and this is one of them. OTOH, if
> the computer has memory available, we should be using *all* of it any place
> we can trade memory for speed.
>
> Other possible technical tradeoffs that might be possible:
>
> * render from bfcache if its present
> * when we evict from bfcache, take a high-res snapshot
> * when things go back far enough, compress or resample the image to a
> low-res snapshot.
>
> Is there a video/demo/feature page which explains what this feature actually
> is? I've seen status updates about a new cool swipe thing, but I don't
> actually know what problem its solving. Is this feature mac-only, or is it
> starting out on mac and we'll be porting it back to other platforms? Or is
> it touchscreen-only?
>
> --BDS
>
>
> ___
> dev-platform mailing list
> dev-platform@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-platform
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Increase in memory utilization on Mac OSX 10.7+ due to history swipe animations

[Justin Lebar]

Now that I've looked more closely, I take back my earlier "What Ed
said."  The issue is more nuanced than I originally thought.

> To save everyone having to look at the graph - the initial landing showed a
> consistent 20% regression in trace malloc maxheap. If this were a 1-5%
> regression, then I think it would be worth discussing the trade-off. At 20%,
> I really don't see how we can take this, sorry! :-(

As I read the graph, the 20% max-heap regression corresponds to 10mb,
which I understand is the expected memory-usage increase here.  So it
looks like this feature is behaving as intended.

To be clear, this regression isn't a 20% regression of memory usage
overall; it's pushing the maximum amount of memory allocated on the
C++ heap up by 10mb, which corresponds to a 20% regression of maximum
heap size in our particular test.

I'd like to put the 10mb in perspective based on our telemetry data.

Unfortunately we don't have good RSS telemetry numbers on Mac (a
direct effect of fixing bug 789975, with no known work-around), so I
can't say precisely how much memory the average Mac user uses.  I also
don't have a CDF in the telemetry front-end (bug 723604), so I can't
interpret the data I do have except by eyeballing it.

But very roughly I'd guess that the bottom 10% of users are somewhere
around 300mb of memory used.  A 10mb regression represents a ~1.05x
increase in memory usage for these users, which doesn't seem like the
end of the world to me.

Still my preference would be to be conservative and decrease this down
to 5 or 10 screenshots (2.5 - 5mb), which I'd bet most of us could
agree is acceptable in terms of memory usage.  I'd prefer going down
to 5 or 10 screenshots largely because of the precedent it sets:
Multiply 10mb of memory times five must-have features and you're
talking serious money.

After configuring this feature to keep fewer screenshots, we could do
the following in parallel:

a) Collect data to determine how many screenshots we "need to" keep
around, as Asa suggested.  Offhand, I'd prefer that we used telemetry
instead of test pilot for this, since presumably the usage of this
feature will change over time, as people discover it.

b) Try to do clever things such as re-rendering bfcached pages, as
Andreas suggested.

c) Consider adaptive techniques so that users who use this feature
heavily will store more screenshots (at the cost of more memory),
while those who don't use it won't pay a price.

I prefer this "start small" approach of limiting us to 5 or 10
screenshots for four reasons.

0) It lets us land this feature now, rather than after we've tried to
reduce its footprint by being clever.  Perfect is the enemy of the
good and all that.

1) It minimizes the impact on users, particularly those who don't use
the feature.

2) It gives the team in charge of this feature an incentive to do
(a)-(c) above, all of which would be beneficial even if we were
storing 50 screenshots.

3) Most importantly, it sets a precedent that we will configure
features to use as little memory as feasible until we have data
demonstrating that additional memory would be beneficial, and that if
you want to use more than the bare minimum necessary, you need to be
clever and avoid making users who don't use the feature pay.

-Justin

On Tue, Feb 12, 2013 at 6:08 PM, Ed Morley  wrote:
> On 12 February 2013 22:11:12, Stephen Pohl wrote:
>>
>> I wanted to give a heads up that we're in the process of finalizing
>> the patch for bug 678392 which will give us history swipe animations
>> on Mac OSX 10.7+. Since we will be taking snapshots of the 20
>> most-recently visited pages, this will undoubtedly lead to an increase
>> in memory utilization on these platforms.
>
>
> To save everyone having to look at the graph - the initial landing showed a
> consistent 20% regression in trace malloc maxheap. If this were a 1-5%
> regression, then I think it would be worth discussing the trade-off. At 20%,
> I really don't see how we can take this, sorry! :-(
>
> Ed
>
> ___
> dev-platform mailing list
> dev-platform@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-platform
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Increase in memory utilization on Mac OSX 10.7+ due to history swipe animations

[Justin Lebar]

> To save everyone having to look at the graph - the initial landing showed a
> consistent 20% regression in trace malloc maxheap. If this were a 1-5%
> regression, then I think it would be worth discussing the trade-off. At 20%,
> I really don't see how we can take this, sorry! :-(

I hope it's not adding noise to say, "what Ed said."

-Justin
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Effects of JSM Compartments on Performance and Development Practices

[Justin Lebar]

We've had bug 764220 open on the CPG memory regression for six months
now.  Although it's a MemShrink:P1, it hasn't gotten the attention it
deserves.

FWIW, I think part of this stems from the fact that the MemShrink team
is often ineffective at getting others to fix bugs that we think are a
priority, perhaps because we don't have an Important Person prodding
other managers to worry about our bugs.  So when we can't fix a bug
ourselves, it often languishes.  (Not to dismiss the heroic efforts of
many people outside the core MemShrink group who have fixed memory
bugs!)  The CPG memory regression and bug 689623 are good examples of
this.

There are a lot of different approaches we could take to reducing the
overhead from CPG.  I'd think that a first step would be trying to
apply bug 798491 (the B2G patch to load most chrome code into one
compartment) to desktop Firefox, but I don't fully understand the
trade-offs here.  CPG resulted in a memory regression for both content
and chrome JS, and we may need different techniques to address them.

But ultimately, I think this is an eminently solvable problem; we just
need to convince someone to look at it.  And as engineers, the main
way we have to effect that is to bring it up, like you did.  So
thanks.  :)

-Justin

On Sun, Jan 20, 2013 at 2:37 PM, Gregory Szorc  wrote:
> I'm writing to the Platform list as a developer that maintains a significant
> amount of JavaScript for Firefox. Much of the JavaScript I maintain is for
> large backend features, such as Sync and Firefox Health Report. These
> features are thousands of lines of code (mostly JavaScript) and interact
> with many systems in Firefox. These backend systems tend to perform many of
> the same tasks, so there is code shared between them in the form of JSMs. To
> reduce redundant code (therefore bugs) and development costs, there is
> obviously the incentive to share as much code as possible.
>
> I'd like to start a discussion about an issue that has been troubling me for
> a while: the performance overhead of separate compartments per JSM. I've
> previously brought this up in bugs and with a few individuals. I wanted to
> raise awareness by posting here. I apologize in advance if I make a few
> semantic mistakes on things like "globals" and "compartments" and the
> overhead of cross-compartment operations: I'm not a core platform hacker!
>
> Ever since Compartment per Global (CPG) landed, JS modules imported with
> Cu.import now get their own compartment (at least on desktop Firefox - I'm
> told they share a compartment on B2G for performance reasons). This is all
> nice in theory. You even get a line item in about:memory to see if your
> module is doing anything wonky!
>
> Unfortunately, this change has a few unfortunate side-effects:
>
> a) Memory overhead for each compartment
> b) Perf losses due to crossing compartments
>
> Assuming the about:memory compartment numbers are correct, each imported JS
> module seems to consume at *least* ~55kb (numbers are similar on both OS X
> and Windows Nightly builds).
>
> Furthermore, there appear to be some scenarios where crossing compartments
> has a horrible effect on performance. For example, it is my understanding
> that crossing compartments can sometimes copy data. I believe this always
> holds for strings. See bug 806087 for example. Hundreds of MBs of
> "string-chars/huge" are seemingly being copied between JSMs/compartments.
> While this specific bug number is resolved, the underlying copying issue
> remains AFAIK.
>
> Sadly, these two issues have a negative influence on how I write JavaScript
> code.
>
> I'm of the camp that tends to write many, small, standalone JSMs rather than
> large and monolithic ones. I find the resulting code is easier to comprehend
> and test. As an added bonus, anybody can come along later and reuse an
> individual module. This includes add-ons. Yay code reuse! The end result is
> less overall code being written and debugged and higher test coverage. I
> like to think this means the quality of the code base is higher and new
> features can be rolled out quicker (due to code reuse). I hold strong
> convictions that this code "architecture" is superior.
>
> With separate compartments per module, I've been put in a tough position. I
> have to weigh the code maintainability advantages of multiple modules
> against memory usage and performance overhead. If I create dozens of small,
> reusable, and specifically-targeted JSMs, I've got the Perf team giving me
> the stink eye for increasing memory usage and cross-compartment overhead. If
> I write JSMs that are thousands of lines long, I'm creating technical debt
> and increasing the barrier to change.
>
> This is a Kobayashi Maru (no win) scenario. Firefox and other Gecko apps
> suffer with both ends of the spectrum. Finding a compromise between the two
> extremes is difficult if not impossible in some scenarios. Furthermore, code
> is always changing thus the conditions influenci

Re: The state of the Aurora branch

[Justin Lebar]

> I was trying to suggest that we usually close trees for
> build/test bustage, not for there being regressions there, so I don't see a
> reason to close beta.  I don't understand whether you're arguing that we
> should close beta or are you just pointing out a problem in what I said.

I was more trying to point out that I don't think you addressed
dbaron's argument.  I happen to agree with him, although that wasn't
really what I was getting at.

To restate dbaron's argument in my own words:

 1. There is a known issue affecting both beta and aurora nightly builds.

 2. Either the issue is or isn't serious enough to warrant closing the
aurora tree.

 3. If it is serious enough to warrant closing the aurora tree, it
seems unlikely to me that the mere fact that we don't run these tests
on beta nightlies means that it is not serious enough to warrant
closing the beta tree.

 4. If on the other hand it's not serious enough to warrant closing
the beta tree, that indicates we're willing to ship with these
failures, which indicates that perhaps the Aurora tree should not
remain closed.

 5. Therefore we should probably either close both Aurora and Beta or
close neither, unless something other than the fact that we don't run
the relevant tests on Beta mitigates the issue's impact there.

The key point to this argument is that the fact of whether we do or
don't run a given set of tests on the beta tree does not affect the
seriousness of the issue on that tree.

-Justin

On Fri, Jan 18, 2013 at 11:17 AM, Ehsan Akhgari  wrote:
> On 2013-01-18 11:03 AM, Justin Lebar wrote:
>>
>>   Fri, Jan 18, 2013 at 10:35 AM, Ehsan Akhgari 
>> wrote:
>>>
>>> On Fri, Jan 18, 2013 at 5:39 AM, L. David Baron 
>>> wrote:
>>>
>>>> So given that this is a regression in Firefox 19 (which is now on
>>>> beta), and the only reason we're not seeing this permaorange on beta
>>>> is because we don't generate non-debug nightly builds on beta (and I
>>>> don't think we run tests on any of our debug nightlies), it seems
>>>> odd to close only Aurora for this.  It seems like depending on what
>>>> we think of its seriousness, we should either close both aurora and
>>>> beta, or we should close neither.
>>>>
>>>
>>> I don't think we've ever closed a tree for test failures which _would_
>>> show
>>> up there if we ran tests there but don't because we don't do that...
>>
>>
>> This is an is/ought fallacy: dbaron is answering the question "what
>> ought we to do?", while the response above is an answer to the
>> question "what /do/ we do?".
>>
>> See also http://en.wikipedia.org/wiki/Is%E2%80%93ought_problem
>>
>> There may be a good reason not to close beta, but "we haven't done so
>> in the past" isn't particularly compelling.
>
>
> I'm not sure where you're going with this, Justin.  My intention was not to
> present a fallacy.  I was trying to suggest that we usually close trees for
> build/test bustage, not for there being regressions there, so I don't see a
> reason to close beta.  I don't understand whether you're arguing that we
> should close beta or are you just pointing out a problem in what I said.  In
> the latter case, I stand corrected and apologies for not getting my sentence
> quite right.  In the former case, you need to have a better argument I
> think.
>
> Ehsan
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: The state of the Aurora branch

[Justin Lebar]

 Fri, Jan 18, 2013 at 10:35 AM, Ehsan Akhgari  wrote:
> On Fri, Jan 18, 2013 at 5:39 AM, L. David Baron  wrote:
>
>> So given that this is a regression in Firefox 19 (which is now on
>> beta), and the only reason we're not seeing this permaorange on beta
>> is because we don't generate non-debug nightly builds on beta (and I
>> don't think we run tests on any of our debug nightlies), it seems
>> odd to close only Aurora for this.  It seems like depending on what
>> we think of its seriousness, we should either close both aurora and
>> beta, or we should close neither.
>>
>
> I don't think we've ever closed a tree for test failures which _would_ show
> up there if we ran tests there but don't because we don't do that...

This is an is/ought fallacy: dbaron is answering the question "what
ought we to do?", while the response above is an answer to the
question "what /do/ we do?".

See also http://en.wikipedia.org/wiki/Is%E2%80%93ought_problem

There may be a good reason not to close beta, but "we haven't done so
in the past" isn't particularly compelling.

-Justin
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Action required from Linux package maintainers building with GCC 4.4 and earlier: JS engine miscompilation caused by unsafe compiler flags

[Justin Lebar]

Due to bug 821502 [1], if you build Firefox 18 for Linux/Android with
GCC 4.4 (and perhaps earlier), GCC will miscompile the JS engine.
These miscompilations are known to cause test failures [2] and crashes
[3].

If you don't build on Linux/Android, don't care about building Firefox
18 (the current release), don't build with GCC, or build with a
version of GCC newer than 4.4, this issue does not affect you.

Note that some versions of the Android NDK include GCC 4.4.  If you
build Firefox 18 with such an NDK, your builds likely /are/ affected
by this bug.

This issue has been fixed on Beta (FF19), Aurora (FF20), and trunk
(FF21).  We know that the issue affects the current release (FF18),
and I expect it affects ESR10 and ESR17, although I haven't checked.

If you're a package maintainer and you build Firefox with GCC 4.4 or
older, you may wish to respin your FF18 release and include the fix
from bug 821502.  If you're a developer, you may simply wish to
upgrade to a newer GCC.

If Mozilla releases a version 18 chemspil (i.e., 18.0.1), we may
include the fix for bug 821502 in that release; this hasn't been
decided yet (and anyway, we don't know if we'll have a chemspill).

I'm very sorry for the inconvenience to you (and your users, if you're
a package maintainer).  Many thanks go to Chris Coulson for figuring
out that this issue was not benign.

Please let me know if I can provide further clarification, if I can
provide assistance in determining whether your builds are affected, or
if I can help backport this fix to your packages.

Regards,
-Justin

[1] https://bugzilla.mozilla.org/show_bug.cgi?id=821502
[2] https://bugzilla.mozilla.org/show_bug.cgi?id=821502#c18
[3] https://bugzilla.mozilla.org/show_bug.cgi?id=828205
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: region of zeroes in Gecko address space

[Justin Lebar]

Aren't anonymous mmap'ed pages automatically zeroed for you?  (It has
to be this way for security reasons.)  So I'd guess you could just
make an anonymous mmap (or the Windows equivalent) and you'd get what
you want.

Of course, I imagine your goal is not to pull in pages in RAM for
these zero pages.  I suspect mmap may do what you want on Linux and
Mac, so long as you don't write to the pages.  I don't know about
Windows, though.

On Sun, Jan 13, 2013 at 5:43 PM, Robert O'Callahan  wrote:
> I need a big read-only buffer full of zeroes. On Linux I could mmap
> /dev/zero read-only, and something similar on Windows/Mac I'm sure, but do
> we already have code for that, or better yet something like that already
> mapped into memory?
>
> Rob
> --
> Jesus called them together and said, “You know that the rulers of the
> Gentiles lord it over them, and their high officials exercise authority
> over them. Not so with you. Instead, whoever wants to become great among
> you must be your servant, and whoever wants to be first must be your
> slave — just
> as the Son of Man did not come to be served, but to serve, and to give his
> life as a ransom for many.” [Matthew 20:25-28]
> ___
> dev-platform mailing list
> dev-platform@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-platform
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Emacs and vim modelines

[Justin Lebar]

> Are there any common installations of vi/vim that actually honor
> modelines anymore?  The ones on the Linux distros that I used
> stopped doing so as a security measure.

:help modeline says

> No other commands than "set" are supported, for security reasons (somebody
> might create a Trojan horse text file with modelines).  And not all options
> can be set.  For some options a flag is set, so that when it's used the
> |sandbox| is effective.  Still, there is always a small risk that a modeline
> causes trouble.  E.g., when some joker sets 'textwidth' to 5 all your lines
> are wrapped unexpectedly.  So disable modelines before editing untrusted text.

So it seems like they have the security angle covered by default, or
at least they intend to?  I don't know if modelines are enabled by
default in distros, though.

If we were to set tabstop=8, I think we'd want to set softtabstop=2
(or 4, depending on the file), so the "tab" key would continue to
shift 2 spaces.

-Justin
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform