Re: Moving Away from Makefile's
On Wed, Aug 22, 2012 at 1:36 AM, Gregory Szorc wrote: > Up until now, the focus has been on making Makefile.in's themselves generic > and data-driven [1]. We would use pymake's API to parse, load, and extract > data from Makefile.in's to construct the build definition. In the long run, > we'd realize that using make files for data definition was silly (and a > large foot gun) and thus we would switch to something else. Can you expand on that? From the discussion so far, JSON is not expressive enough and Python is too expressive. There are some very understandable reservations about inventing a new language, as well as the desire for a "clean slate" (admittedly attractive). Why doesn't it make sense to use a very-restricted dialect of Makefiles? pymake already has a parser, the dumbing down of which to disallow arbitrary shell expressions would supposedly be fairly straightforward. No one would have to learn a new language, and you can start with the current files, duplicate the parser inside pymake, then start to dumb it down as you weed the complexity out of the Makefile.ins. Cheers, Dirkjan ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Moving Away from Makefile's
On 08/22/2012 11:19 PM, Mike Hommey wrote: On Wed, Aug 22, 2012 at 01:38:22PM -0700, Gregory Szorc wrote: On 8/22/12 1:00 PM, Ben Hearsum wrote: On 08/22/12 12:43 PM, Jeff Hammel wrote: If we do go with python, it would be nice to keep the configuration files as much configuration as possible. The reason I question having any "full" language as a configuration language is that in practice I see a lot of logic creep in to "configuration" files and then they become very difficult for a consumer to parse/use as configuration. While a bit of an unfair example, our buildbot-configs fall into this category. A million times this. Putting our configs in Python has let to terrible, terrible hacks - to the point where we don't even know how to do things like "turn off leopard tests for some branches" anymore (https://bugzilla.mozilla.org/show_bug.cgi?id=773120). I strongly discourage using Python when you don't need a fully featured language. You will do bad things. You will regret it. There are a lot of bullets in "batteries included" languages. The Python foot gun worries me greatly. We even have this problem in make, and that is a much simpler language! As I said on irc, i think the "language" for build manifests should be something between .ini and jar.mn. Because they're simple, straightforward, and are well known. And simple. I understand why people are eager to use python or lua, or other scripting languages. But the fact is you really only need scripting in very rare cases. The vast majority of our build is "take this file, put it there", "take this file, preprocess it, put it there", "compile this" or "link that". And occasionally, we need to add flags for specific files. The build manifests should make defining those extremely simple. There's also the case of conditionals. But i think the vast majority of the conditionals we use are of the type "do this only on windows", or "do this only on arm" The few remaining things can have python scripts, but let's worry about them later. Mike Maybe the thing to do is to stub this out for a few existing cases in various formats and see how they look. It seems like the consensus is that the format should be - as simple as possible - self-evident: easy to write, easy to edit, preferably something Mozillians are familiar with - able to do the simple things simply - able to handle complex specialization for the cases that require it But its hard (for me, anyway) to visualize what this would look like without seeing more concrete proposals. If .ini suffices, I am all for it. Jeff P.S. In the past, I was quite fond of buildit: http://agendaless.com/Members/chrism/software/buildit . AFAIK, it is not actively maintained, but it could be a source of inspiration. ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Moving Away from Makefile's
On Wed, Aug 22, 2012 at 01:38:22PM -0700, Gregory Szorc wrote: > On 8/22/12 1:00 PM, Ben Hearsum wrote: > >On 08/22/12 12:43 PM, Jeff Hammel wrote: > >>If we do go with python, it would be nice to keep the configuration > >>files as much configuration as possible. The reason I question having > >>any "full" language as a configuration language is that in practice I > >>see a lot of logic creep in to "configuration" files and then they > >>become very difficult for a consumer to parse/use as configuration. > >>While a bit of an unfair example, our buildbot-configs fall into this > >>category. > > > >A million times this. Putting our configs in Python has let to terrible, > >terrible hacks - to the point where we don't even know how to do things > >like "turn off leopard tests for some branches" anymore > >(https://bugzilla.mozilla.org/show_bug.cgi?id=773120). I strongly > >discourage using Python when you don't need a fully featured language. > >You will do bad things. You will regret it. > > There are a lot of bullets in "batteries included" languages. The > Python foot gun worries me greatly. We even have this problem in > make, and that is a much simpler language! As I said on irc, i think the "language" for build manifests should be something between .ini and jar.mn. Because they're simple, straightforward, and are well known. And simple. I understand why people are eager to use python or lua, or other scripting languages. But the fact is you really only need scripting in very rare cases. The vast majority of our build is "take this file, put it there", "take this file, preprocess it, put it there", "compile this" or "link that". And occasionally, we need to add flags for specific files. The build manifests should make defining those extremely simple. There's also the case of conditionals. But i think the vast majority of the conditionals we use are of the type "do this only on windows", or "do this only on arm" The few remaining things can have python scripts, but let's worry about them later. Mike ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Gecko switched to use standard integer types
On Wed, Aug 22, 2012 at 04:12:30PM -0700, Justin Lebar wrote:
> bholley and I have a script for doing this in git. With thanks to
> glandium for telling us how to do it:
>
> 0. Fetch the prtypes change, and merge it into your local master branch.
>
> 1. Let your git checkout be directory |src|.
>
> 2. Save the script at the end of this message as src/../convert.sh.
>
> 3. For each branch B that you want to rebase over this change, do:
>
> a. $ git checkout B. Maybe make a note of the hash, so you can
> reset --hard in case something goes wrong.
If something goes wrong, you can check the reflog (git reflog show B)
and reset --hard to the old version of B that is not broken, most likely
B@{1}.
> b. Let P be the parent changeset of B. This is the result of |git
> qparent|, if you have moz-git-tools [1] installed. Otherwise it's
> |git merge-base B origin/master|, assuming that your branch is
> upstream from origin/master.
>
> c. $ git filter-branch --tree-filter /absolute/path/to/convert.sh P^..
>
> Note that it's "P^.." and not "P..". This is because we want to
> convert P and also all of the commits in B (and the rev range P.. does
> not include P). We're effectively applying the pr-types change to P
> and all changesets in B.
>
> d. Let P' be the new parent changeset of B. (The hash has changed,
> since we rewrote this commit.)
>
> e. $ git rebase --onto master P'
>
> Note that |git rebase master| will not do what you want, because
> that will try to move P' onto master. But P' is already in master!
> If in doubt, do rebase --interactive, so you can see what git is going
> to try to do.
>
>
> The git apologist in me feels compelled to admit that this is, in
> fact, much more complicated than the equivalent task in hg. :)
If you want to do the equivalent task from hg, you can:
a) git format-patch -o /some/where P..
b) patch the patches
c) git checkout master
d) git am /some/where/*
Mike
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform
Re: Moving Away from Makefile's
So... Data-only formats such as JSON are too weak -- no comments, no conditionals. Full languages (e.g. Python, Lua) are too powerful, which means it's too tempting to use the full language features. Should you invent something new? If it's just lists, maps, comments and conditionals, that should be easy. And looking at Gregory's initial list of eight requirements, it sounds like they could be satisfied nicely. Just a thought :) Nick ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Moving Away from Makefile's
On Thu, Aug 23, 2012 at 8:40 AM, Ben Hearsum wrote: > On 08/22/12 04:38 PM, Gregory Szorc wrote: > > Let's think of what can be done to secure/limit Python. Disabling import > > has already been mentioned. That's a start. > > I think it's worth noting that even if you *do* limit what you can do > through some technical means, you still have the option to change that > later, disable it some places, etc. It's really easy to get into that > game when you're fixing blockers or working on chemspills, too. > If someone is that desperate, what would you have them do instead of hack the configuration file? Aren't they likely to respond by doing some even worse hack that gets the job done? I think it makes a ton of sense to use automation to stop developers accidentally doing something they shouldn't. But if someone's desperate enough to disable the automation, and can get a review for it, then I don't think it makes sense to try to stop them. Rob -- “You have heard that it was said, ‘Love your neighbor and hate your enemy.’ But I tell you, love your enemies and pray for those who persecute you, that you may be children of your Father in heaven. ... If you love those who love you, what reward will you get? Are not even the tax collectors doing that? And if you greet only your own people, what are you doing more than others?" [Matthew 5:43-47] ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Gecko switched to use standard integer types
What is going to happen to PRTime? Is that going away too? Philipp ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Gecko switched to use standard integer types
bholley and I have a script for doing this in git. With thanks to
glandium for telling us how to do it:
0. Fetch the prtypes change, and merge it into your local master branch.
1. Let your git checkout be directory |src|.
2. Save the script at the end of this message as src/../convert.sh.
3. For each branch B that you want to rebase over this change, do:
a. $ git checkout B. Maybe make a note of the hash, so you can
reset --hard in case something goes wrong.
b. Let P be the parent changeset of B. This is the result of |git
qparent|, if you have moz-git-tools [1] installed. Otherwise it's
|git merge-base B origin/master|, assuming that your branch is
upstream from origin/master.
c. $ git filter-branch --tree-filter /absolute/path/to/convert.sh P^..
Note that it's "P^.." and not "P..". This is because we want to
convert P and also all of the commits in B (and the rev range P.. does
not include P). We're effectively applying the pr-types change to P
and all changesets in B.
d. Let P' be the new parent changeset of B. (The hash has changed,
since we rewrote this commit.)
e. $ git rebase --onto master P'
Note that |git rebase master| will not do what you want, because
that will try to move P' onto master. But P' is already in master!
If in doubt, do rebase --interactive, so you can see what git is going
to try to do.
The git apologist in me feels compelled to admit that this is, in
fact, much more complicated than the equivalent task in hg. :)
Good luck!
[1] https://github.com/jlebar/moz-git-tools
== convert.sh ==
#!/bin/bash
# This could be faster, but maybe it's good enough.
function convert() {
git ls-files | grep '\.cpp$\|\.idl$\|\.h$' | xargs grep -l $1 |
xargs -n1 sed -i '' -e "s/$1/$2/g"
}
convert PRInt8 int8_t
convert PRUint8 uint8_t
convert PRInt16 int16_t
convert PRUint16 uint16_t
convert PRInt32 int32_t
convert PRUint32 uint32_t
convert PRInt64 int64_t
convert PRUint64 uint64_t
convert PRIntn int32_t
convert PRUintn uint32_t
convert PRSize size_t
convert PROffset32 int32_t
convert PROffset64 int64_t
convert PRPtrdiff ptrdiff_t
convert PRFloat64 double
On Wed, Aug 22, 2012 at 12:25 PM, Ehsan Akhgari wrote:
> On 12-08-22 3:18 PM, Jonathan Kew wrote:
>>
>> On 22/8/12 17:35, Ehsan Akhgari wrote:
>>>
>>> I just landed the patches in bug 579517 which switch all of the code in
>>> mozilla-central and comm-central [1] to use the standard integer types
>>> as opposed to NSPR integer types. Here's what this means to you:
>>>
>>> * If you're a developer, this will most likely bitrot your patches. You
>>> can use this script [2] in order to unbitrot your patches if you use
>>> mercurial queues.
>>
>>
>>> [2] https://bugzilla.mozilla.org/attachment.cgi?id=650572
>>
>>
>> Beware that this script is slightly obsolete, inasmuch as it converts
>> PRIntn/PRUintn to int32_t/uint32_t (should be int/unsigned).
>
>
> Good point. But you should've called the cops if anyone used those types in
> their patches in the first place. ;-)
>
> Ehsan
>
> ___
> dev-platform mailing list
> dev-platform@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-platform
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform
Re: Moving Away from Makefile's
On Wed, Aug 22, 2012 at 3:38 PM, Gregory Szorc wrote: > Let's think of what can be done to secure/limit Python. Disabling import > has already been mentioned. That's a start. > > What about the ast module [1]? I /think/ it could be used to ensure parsed > code conforms to whatever we allow. e.g. no import statements, no function > definitions, no classes, etc. Actually, it may be better to think of it in > terms of what would be allowed. Only declaring variables that are strings, > lists, or dicts. Only calling pre-defined functions "exported" by the build > system, etc. We start with the minimum and open it up as we need to. > Instead of disallowing imports, you could replace the importer (during build file evaluation) with something that allows only whitelisted modules, and potentially only allows subsets of those modules. For instance, you might want os.environ, but not os.open. You might also want to do something like replace os.environ with a wrapper that tracks access, so you can detect the environmental dependencies. I don't think functions and classes can make things too much more complicated, so long as the build files can't share those definitions outside the file (which could lead to an ad hoc and hidden library). Something like open() seems perhaps more dangerous in terms of creating hard-to-understand dependencies. Doing strict type checks on the resulting data would be a good check. E.g., you don't want a file to return a clever str subclass that overrides equality. ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Moving Away from Makefile's
On 08/22/12 04:38 PM, Gregory Szorc wrote: > Let's think of what can be done to secure/limit Python. Disabling import > has already been mentioned. That's a start. I think it's worth noting that even if you *do* limit what you can do through some technical means, you still have the option to change that later, disable it some places, etc. It's really easy to get into that game when you're fixing blockers or working on chemspills, too. ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Moving Away from Makefile's
On 8/22/12 1:00 PM, Ben Hearsum wrote: On 08/22/12 12:43 PM, Jeff Hammel wrote: If we do go with python, it would be nice to keep the configuration files as much configuration as possible. The reason I question having any "full" language as a configuration language is that in practice I see a lot of logic creep in to "configuration" files and then they become very difficult for a consumer to parse/use as configuration. While a bit of an unfair example, our buildbot-configs fall into this category. A million times this. Putting our configs in Python has let to terrible, terrible hacks - to the point where we don't even know how to do things like "turn off leopard tests for some branches" anymore (https://bugzilla.mozilla.org/show_bug.cgi?id=773120). I strongly discourage using Python when you don't need a fully featured language. You will do bad things. You will regret it. There are a lot of bullets in "batteries included" languages. The Python foot gun worries me greatly. We even have this problem in make, and that is a much simpler language! Let's think of what can be done to secure/limit Python. Disabling import has already been mentioned. That's a start. What about the ast module [1]? I /think/ it could be used to ensure parsed code conforms to whatever we allow. e.g. no import statements, no function definitions, no classes, etc. Actually, it may be better to think of it in terms of what would be allowed. Only declaring variables that are strings, lists, or dicts. Only calling pre-defined functions "exported" by the build system, etc. We start with the minimum and open it up as we need to. Is there anyone out there with experience with this module? What's performance like for walking the AST? Is this something we could do at manifest read/eval time without incurring too much of a penalty? (If not, there's always automated auditing, which could even be part of the test suite so trees would go red if someone checked in something in violation.) Are there any caveats to the low-level nature of this module? i.e. is there a burden to supporting ast functionality on multiple versions of Python (we'll eventually need to handle the 2.7 -> 3.x transition). For those who don't know Lua, the main reason I said it was perfect in the original post is it does all of this automatically! It's designed to be embeddable. A newly-created Lua context/interpreter can do very little. You need to explicitly load the standard library from the C API! A new context doesn't even have a string module nor the ability to load modules: These must be selectively enabled through the C API by the process creating the context instance! It's still a full programming language. But, it's so constricted without any external modules or module loading support that you almost don't have to worry. It is a more natural fit for the task at hand. But, given Mozilla's requirements (extra build dependency, maintaining the Lua bridge code - I know the Lua C API, does anyone else?), I think shoehorning Python would be preferred over Lua. [1] http://docs.python.org/library/ast.html ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Moving Away from Makefile's
On 08/22/2012 12:29 PM, Gregory Szorc wrote: Switching to something else also has another advantage: the opportunity for a clean slate. I'm hoping we'll use the opportunity to scrape away 10+ years of cruft. I support the sentiment, though this reasoning always makes me cautious, as it often translates to (unintentionally) eliminating the old system (cruft + accrued knowledge), replacing with a new system that initially feels clean but once all the features have added and it needs to be extended is just different cruft, and cruft that is no longer standard. Don't get me wrong: I don't think you would do this and I am very grateful to see you tackle making the build system better. I just wanted to point out to the list that this is a "WARNING! Big change! Proceed with caution!" scenario. ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Moving Away from Makefile's
On 08/22/12 12:43 PM, Jeff Hammel wrote: > If we do go with python, it would be nice to keep the configuration > files as much configuration as possible. The reason I question having > any "full" language as a configuration language is that in practice I > see a lot of logic creep in to "configuration" files and then they > become very difficult for a consumer to parse/use as configuration. > While a bit of an unfair example, our buildbot-configs fall into this > category. A million times this. Putting our configs in Python has let to terrible, terrible hacks - to the point where we don't even know how to do things like "turn off leopard tests for some branches" anymore (https://bugzilla.mozilla.org/show_bug.cgi?id=773120). I strongly discourage using Python when you don't need a fully featured language. You will do bad things. You will regret it. ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Moving Away from Makefile's
Jeff Hammel wrote: While a bit of an unfair example, our buildbot-configs fall into this category. IMO not unfair at all. (p.s. to stay on topic, +1 to all else you said) ~Justin Wood (Callek) ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Moving Away from Makefile's
On 8/22/12 2:02 AM, Neil wrote: Gregory Szorc wrote: Up until now, the focus has been on making Makefile.in's themselves generic and data-driven. We would use pymake's API to parse, load, and extract data from Makefile.in's to construct the build definition. In the long run, we'd realize that using make files for data definition was silly (and a large foot gun) and thus we would switch to something else. Switch to something else for all Makefile.in's or just the generic ones? The plan would be to have no Makefile.in's, as the presence of a Makefile implies usage of make, which may not always be the case. We want to support building with alternate build backends and that is difficult if large parts involve one-off usages of make. We may use make for some specific tasks (because it is a decent dependency-action DSL). But, for building the core of mozilla-central, use of Makefile's as the *definition* of the build config will likely be marginalized. On 8/22/12 6:10 AM, Robert Kaiser wrote:> Gregory Szorc schrieb: >> We could go the route of GYP and shoehorn conditionals into a static >> document (JSON) [3]. > > JSON is a good format for data for the most part, but IMHO we *really* > want comments in those files, and unfortunately JSON doesn't have those > and therefore probably must be thrown out of the equation. :( > > Actually, I know a format that allows everything we need: Makefile! :p I actually agree with you! This was our initial plan: load Makefiles into pymake and evaluate the values of specific variables, like CPPSRCS. Unfortunately, there are some deal-breakers. The evaluation model of make is... complicated. Lazy evaluation, rules that must be executed before variables can be read, side-effects from evaluating variables, etc. There are a *lot* of pitfalls. Switching to something else also has another advantage: the opportunity for a clean slate. I'm hoping we'll use the opportunity to scrape away 10+ years of cruft. > Seriously, this is hard, and needing to parse even more files to build > doesn't sound faster and cleaner, but rather slower and more complex, > esp. given that basic file I/O is often costly (from watching my CPU > usage, a lot of the build time is spent in I/O wait when using spinning > disks - SSDs improve that hugely). On my browser build, we place 19000+ files in the objdir. That's not counting all the files in the srcdir that need to be stat() or read during builds. I think the overhead of a few hundred build manifest files won't be too bad. It will also likely be mitigated through caching. Yes, I/O wait can significantly reduce build times. I recommended at [1] that aside from a modern CPU, investing in an SSD is the best thing you can do for build times. If a build machine doesn't have an SSD, it's effectively throwing away CPU cycles in I/O wait. It's true that the build system today wastes a lot of CPU cycles due to inefficient use of all available cores (lack of wide parallelism due to recursive make). But, the efficiency should rise drastically with our build system improvements. That will make the impact of an SSD even more pronounced. On my MBP (with 8GB), building with an SSD reduced overall build time by a few minutes, with libxul linking going from ~55s to ~5s. That's without any build system changes! There is no doubt in my mind that SSDs are worth the investment. [1] http://gregoryszorc.com/blog/2012/07/29/mozilla-central-build-times/ ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Gecko switched to use standard integer types
On 12-08-22 3:18 PM, Jonathan Kew wrote: On 22/8/12 17:35, Ehsan Akhgari wrote: I just landed the patches in bug 579517 which switch all of the code in mozilla-central and comm-central [1] to use the standard integer types as opposed to NSPR integer types. Here's what this means to you: * If you're a developer, this will most likely bitrot your patches. You can use this script [2] in order to unbitrot your patches if you use mercurial queues. [2] https://bugzilla.mozilla.org/attachment.cgi?id=650572 Beware that this script is slightly obsolete, inasmuch as it converts PRIntn/PRUintn to int32_t/uint32_t (should be int/unsigned). Good point. But you should've called the cops if anyone used those types in their patches in the first place. ;-) Ehsan ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Gecko switched to use standard integer types
On 22/8/12 17:35, Ehsan Akhgari wrote: I just landed the patches in bug 579517 which switch all of the code in mozilla-central and comm-central [1] to use the standard integer types as opposed to NSPR integer types. Here's what this means to you: * If you're a developer, this will most likely bitrot your patches. You can use this script [2] in order to unbitrot your patches if you use mercurial queues. [2] https://bugzilla.mozilla.org/attachment.cgi?id=650572 Beware that this script is slightly obsolete, inasmuch as it converts PRIntn/PRUintn to int32_t/uint32_t (should be int/unsigned). JK ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Moving Away from Makefile's
On 12-08-22 12:43 PM, Jeff Hammel wrote: TL;DR - python if we want/need a full language, JSON if we can get away with POD I think JSON is the wrong choice here. It will not satisfy the need for conditionals, which causes us to invent terrible hacks inside the JSON like we had to do with the xpcshell manifest format, and it's not really easy to be consumed or edited by humans. Python is the clear winner here, IMO. Ehsan ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Moving Away from Makefile's
On 08/22/2012 06:50 AM, Benjamin Smedberg wrote: On 8/21/2012 7:36 PM, Gregory Szorc wrote: On the other end of the spectrum, we could have the build manifest files be Python "scripts." This solves a lot of problems around needing functionality in the manifest files. But, it would be a potential foot gun. See requirement #7. I don't think this would be a big deal. We could just enforce no side effects at review, or with a small bit of python we could enforce some basic restraints in code: * allow imports from only a small whitelist of known-safe modules: perhaps just 're', or disallow imports altogether (by modifying the globals in the execution environment) and pre-import the safe modules * "fix" output variables to ensure that they are the expected string/list-of-strings/whatever types I really think that python manifests are the best choice here, since python is already optimized to parse them efficiently, the control structures are fairly straightforward, and most people (both regular engineers and build-sytem hackers) are going to know enough to get started without confusion. One thing that wasn't clear to me from the original post is whether we are planning on automatically transforming some parts of the existing makefiles (e.g. DIRS, EXPORTS, preprocessor stuff) into the new manifest format, or whether the plan is to just migrate by hand. Or is that still TBD? --BDS I don't have a magic bullet here. My inclination is to not introduce new technology/languages/syntax to keep the learning curve lower/comparable. If we're going to go for a whole language, I am +1 on python. We already use it...well, all over the place. It is well known, solid, and not "yet another thing to learn or syntax switch" to. Unless a very compelling argument argument could be made for the functionality of other languages vs python for this problem space, python seems a natural solution and can build on our existing python ecosystem (which likewise can grow driven by needs here). My actual inclination is to use a pure data format. I'm not sure if .ini is sufficient: guessing not. My inclination here is JSON. That said, it won't have conditionals or logic, so as :gps puts it, these would have to be shoehorned in. To expand on that a bit, the different platforms/configurations/states will have to be recorded and the runner would have to have the logic to decide what to use. I haven't looked at this problem in depth, so I don't know how good/bad/horrible this would be in practice. If we do go with python, it would be nice to keep the configuration files as much configuration as possible. The reason I question having any "full" language as a configuration language is that in practice I see a lot of logic creep in to "configuration" files and then they become very difficult for a consumer to parse/use as configuration. While a bit of an unfair example, our buildbot-configs fall into this category. TL;DR - python if we want/need a full language, JSON if we can get away with POD Jeff ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Gecko switched to use standard integer types
I just landed the patches in bug 579517 which switch all of the code in mozilla-central and comm-central [1] to use the standard integer types as opposed to NSPR integer types. Here's what this means to you: * If you're a developer, this will most likely bitrot your patches. You can use this script [2] in order to unbitrot your patches if you use mercurial queues. Also, please get into the habit of using stdint types (such as int16_t and uint32_t) as opposed to NSPR numeric types (such as PRInt16 and PRUint32). * If you maintain a branch which merges against mozilla-central, you probably want to merge mozilla-central into it as soon as possible. There will probably be merge conflicts. The easiest way to resolve them is to take your branch's version in the conflicts and then run the script that I used for the conversion again so that you make sure you're not adding more NSPR numeric type usages in your branch. * If you want to back out a patch which landed before this, there's probably a bunch of manual conflict resolution that you need to do. * If you talk to new Mozilla contributors regularly, you probably won't need to explain to them what these NSPR types are any more. :-) * If you're not in any of the above categories, this work will probably not affect you. Cheers, Ehsan [1] (except for NSPR and NSS) [2] https://bugzilla.mozilla.org/attachment.cgi?id=650572 [3] https://bugzilla.mozilla.org/attachment.cgi?id=653946 ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Moving Away from Makefile's
On Thu, Aug 23, 2012 at 01:07:34AM +1200, Blair McBride wrote: > On 22/08/2012 5:58 p.m., Mike Hommey wrote: > >>+1 for Lua - it seems perfect for this. For #1, I find it far easier > >>>to read (and write) than Gyp, when it comes to things like > >>>conditionals. For #8, we could just ship the entire runtime in the > >>>tree for Tier 1 platforms (its small enough!), then its only an > >>>additional dependency for the 0.01%. > >But then, you have a chicken and egg kind of problem, where your build > >system depends on building something... > > When I say "runtime" I mean the actual pre-built runtime, not the source. Which you then need to ship as a windows binary, an osx binary, and a linux binary. And that other platforms need to build or install separately. Mike ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Moving Away from Makefile's
On 8/21/2012 7:36 PM, Gregory Szorc wrote: On the other end of the spectrum, we could have the build manifest files be Python "scripts." This solves a lot of problems around needing functionality in the manifest files. But, it would be a potential foot gun. See requirement #7. I don't think this would be a big deal. We could just enforce no side effects at review, or with a small bit of python we could enforce some basic restraints in code: * allow imports from only a small whitelist of known-safe modules: perhaps just 're', or disallow imports altogether (by modifying the globals in the execution environment) and pre-import the safe modules * "fix" output variables to ensure that they are the expected string/list-of-strings/whatever types I really think that python manifests are the best choice here, since python is already optimized to parse them efficiently, the control structures are fairly straightforward, and most people (both regular engineers and build-sytem hackers) are going to know enough to get started without confusion. One thing that wasn't clear to me from the original post is whether we are planning on automatically transforming some parts of the existing makefiles (e.g. DIRS, EXPORTS, preprocessor stuff) into the new manifest format, or whether the plan is to just migrate by hand. Or is that still TBD? --BDS ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: LinuxGL widget backend for Mozilla
Hi, Am 17.08.2012 20:03, schrieb romaxa: > Some people already expressed interest in working with me on this so > I'm trying to figure out what is the best place to park the code we > are going to work on. > > Would it make sense to land initial build/run configuration support > into mozilla-central now and work on it in main tree. Alternatively > we could maintain a branch outside (github or similar) but that would > mean frequent rebases. I'm one of the people that plan to work on this with Oleg, and I'd much prefer to work on m-c. I think the required reviews would actually be a pretty good thing to get more input on the basic ideas before it's too late. Philipp ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Moving Away from Makefile's
Gregory Szorc schrieb: We could go the route of GYP and shoehorn conditionals into a static document (JSON) [3]. JSON is a good format for data for the most part, but IMHO we *really* want comments in those files, and unfortunately JSON doesn't have those and therefore probably must be thrown out of the equation. :( Actually, I know a format that allows everything we need: Makefile! :p Seriously, this is hard, and needing to parse even more files to build doesn't sound faster and cleaner, but rather slower and more complex, esp. given that basic file I/O is often costly (from watching my CPU usage, a lot of the build time is spent in I/O wait when using spinning disks - SSDs improve that hugely). Robert Kaiser ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Moving Away from Makefile's
On 22/08/2012 5:58 p.m., Mike Hommey wrote: +1 for Lua - it seems perfect for this. For #1, I find it far easier >to read (and write) than Gyp, when it comes to things like >conditionals. For #8, we could just ship the entire runtime in the >tree for Tier 1 platforms (its small enough!), then its only an >additional dependency for the 0.01%. But then, you have a chicken and egg kind of problem, where your build system depends on building something... When I say "runtime" I mean the actual pre-built runtime, not the source. - Blair ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: LinuxGL widget backend for Mozilla
Ehsan Akhgari schrieb: I suggest that the best place to develop this would be on mozilla-central, especially since most of the work involved would probably not be part of the build of any Tier 1 platform. I agree on the m-c part, but given that a lot of the Linux world is looking at maybe getting rid of X completely and using Wayland as the main graphics output base layer, and Wayland being all about OpenGL, some of this work may even have tier-1 impact in the longer run. :) Robert Kaiser ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Proposed policy change: reusability of tests by other browsers
Chris Hofmann schrieb: Yeah, this is not for the other browser vendors or users, but is mostly a benefit for web developers that want to count on certain behaviors to work across browsers effectively and reliably every release of every browser. And as web developers write websites for users, this is for users. And as we want to serve the web and users, it's a benefit for us. Robert Kaiser ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: LinuxGL widget backend for Mozilla
On Fri, Aug 17, 2012 at 1:48 PM, romaxa wrote: > Hi, > > Recently I have been working on B2G port to mobile Linux which would be > widget toolkit-independent (qt/gtk/cocoa/etc). Some time ago Chris Lord did work on a widget toolkit independent backend that had a similar focus[1]. His work was targeted at being the renderer for the Clutter toolkit, which was used in Moblin. I found his work really useful because it was possible to use it without an X server present. Does your work require the presence of OpenGL to render, or can it render without a display as well? I've seen a number of other people interested in the ability to render without a display, and we've never managed to meet that need in the core Mozilla code. -Ted 1. http://hg.mozilla.org/incubator/offscreen ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Moving Away from Makefile's
Gregory Szorc wrote: Up until now, the focus has been on making Makefile.in's themselves generic and data-driven. We would use pymake's API to parse, load, and extract data from Makefile.in's to construct the build definition. In the long run, we'd realize that using make files for data definition was silly (and a large foot gun) and thus we would switch to something else. Switch to something else for all Makefile.in's or just the generic ones? -- Warning: May contain traces of nuts. ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
