[chromium-dev] waterfall and memory tests

[Nicolas Sylvain]

Hi,

The number of memory tests bots has been increasing rapidly recently, and
since the main buildbot console view
was getting too big and slow, I decided to move the memory tests in their
own buildbot master. HOWEVER, this does
not indicate in any ways that they are less important.

On http://build.chromium.org you can now see a fourth line at the top
showing the status of the memory tests. Any redness there
should be considered as bad as any other redness on the main waterfall, and
the sheriff should act on it.

The memory test console is at this url :
http://build.chromium.org/buildbot/memory/console

Please take a look at this page when you commit to make sure you did not
break anything.  Since these bots are slow
to cycle (some take more than 1 hour), I would suggest that you monitor
closely the main console at build.chromium.org
when you commit a change, and when you see that everything is turning green,
you should also take a look at the memory
console to make sure you did not regress this one either.

This is clearly not the most user-friendly way of doing it, but it was
essential at this point to relieve the pressure on our
main buildbot master.  We are currently working on the design of a new
implementation of the console view that will
let us merge multiple buildbot masters together. This should fix that
problem. ETA unknown.

Thank you,

Nicolas
-- 
Chromium Developers mailing list: chromium-dev@googlegroups.com 
View archives, change email options, or unsubscribe: 
http://groups.google.com/group/chromium-dev

Re: [chromium-dev] Red Tree 2010/1/6

[Nicolas Sylvain]

On Thu, Jan 7, 2010 at 11:22 AM, James Robinson  wrote:

> Thanks a lot, Chase.
>
> When will it become feasible to run the perf tests as part of the WebKit
> canary process?  We seem to have all the code necessary to catch these
> regressions at the exact WebKit revision that causes them.
>

We've been doing that for months now.

Nicolas


>
> - James
>
> On Thu, Jan 7, 2010 at 11:13 AM, Chase Phillips wrote:
>
>> I'll take the bug for now to identify the WebKit rev that caused the
>> regression.
>>
>> On Wed, Jan 6, 2010 at 20:10, Robert Sesek  wrote:
>>
>>> Good evening,
>>>
>>> At 9pm ET, the tree was significantly red and Erik Kay closed the tree:
>>> Win browser_tests, Win & Mac perf, Linux views, and Win Webkit all were red.
>>> I cleaned up the Win browser_tests, Linux Views, and Win WebKit redness, but
>>> the Win & Mac perf regressions are still in play. mpcompete, jamesr, and I
>>> debated about what to do with this since it originated from a fairly
>>> significant WebKit roll:
>>> http://src.chromium.org/viewvc/chrome?view=rev&revision=35646. Backing
>>> out didn't seem like the best option because it was large jump and another
>>> roll happened later at r35669. The tree *is still closed* as of 11pm ET
>>> because http://crbug.com/31698 (tracking bug for this perf regression)
>>> does not yet have an owner and it desperately needs one! If in the morning
>>> someone more knowledgeable feels like we can open, please do.
>>>
>>> Also, I didn't have time to track it down tonight, but Windows UI
>>> Interactive (dbg) has been red on and off for the past couple of cycles,
>>> starting at
>>> http://build.chromium.org/buildbot/waterfall/builders/Interactive%20Tests%20(dbg)/builds/19501.
>>> I'm hesitant to mark it as flaky without knowing a little more, but I backed
>>> out oshima's change (the likely candidate) and the redness continued. Help
>>> appreciated :).
>>>
>>> Thanks and good night!
>>>
>>> rsesek / @chromium.org
>>>
>>> --
>>> Chromium Developers mailing list: chromium-dev@googlegroups.com
>>> View archives, change email options, or unsubscribe:
>>>http://groups.google.com/group/chromium-dev
>>>
>>
>>
>
> --
> Chromium Developers mailing list: chromium-dev@googlegroups.com
> View archives, change email options, or unsubscribe:
>http://groups.google.com/group/chromium-dev
>
-- 
Chromium Developers mailing list: chromium-dev@googlegroups.com 
View archives, change email options, or unsubscribe: 
http://groups.google.com/group/chromium-dev

Re: [chromium-dev] LATEST symlink missing for win builds

[Nicolas Sylvain]

On Wed, Jan 6, 2010 at 3:32 PM, Kenneth Russell  wrote:

> It looks like the LATEST symlink was deleted out of the directory
> http://build.chromium.org/buildbot/continuous/win/ . It's there for
> Mac and Linux. Could someone please take a look?
>

It should be fixed now. Feel free to send me a private email if you notice
that the link goes down again.

Thank you,

Nicolas

>
> Thanks,
>
> -Ken
>
> --
> Chromium Developers mailing list: chromium-dev@googlegroups.com
> View archives, change email options, or unsubscribe:
>http://groups.google.com/group/chromium-dev
>
-- 
Chromium Developers mailing list: chromium-dev@googlegroups.com 
View archives, change email options, or unsubscribe: 
http://groups.google.com/group/chromium-dev

Re: [chromium-dev] buildbot cycle time dashboard?

[Nicolas Sylvain]

On Tue, Dec 8, 2009 at 3:00 PM, James Hawkins  wrote:

> Thanks for the link Eric!  Can someone explain what the "Percentage of
> failures" graph is representing?
>

# failed build / # total builds.  (over the last 300 builds).  if it's 10,
then that
means that in the last 300 builds, 10% of them were red.

Nicolas


>
> Thanks,
> James Hawkins
>
> On Tue, Dec 8, 2009 at 2:56 PM, Eric Seidel  wrote:
> > http://build.chromium.org/buildbot/waterfall/stats
> >
> > On Tue, Dec 8, 2009 at 2:51 PM, Dan Kegel  wrote:
> >> Hey folks,
> >> I seem to recall we have a dashboard somewhere showing
> >> the cycle time of each of our build/test bots, but I can't
> >> find it.  Can somebody point me to it?
> >>
> >> If it doesn't exist, maybe I'll scrape one together.
> >>
> >> Thanks...
> >>
> >> --
> >> Chromium Developers mailing list: chromium-dev@googlegroups.com
> >> View archives, change email options, or unsubscribe:
> >>http://groups.google.com/group/chromium-dev
> >>
> >
> > --
> > Chromium Developers mailing list: chromium-dev@googlegroups.com
> > View archives, change email options, or unsubscribe:
> >http://groups.google.com/group/chromium-dev
> >
>
> --
> Chromium Developers mailing list: chromium-dev@googlegroups.com
> View archives, change email options, or unsubscribe:
>http://groups.google.com/group/chromium-dev
>

-- 
Chromium Developers mailing list: chromium-dev@googlegroups.com 
View archives, change email options, or unsubscribe: 
http://groups.google.com/group/chromium-dev

Re: [chromium-dev] gcl commit throwing SVN server error

[Nicolas Sylvain]

On Thu, Dec 3, 2009 at 4:03 PM, Dan Kegel  wrote:

> If you're a committer, you should have recieved instructions on how
> to set up your client (it will have an svn rather than an http url)
> when you got your commit password.
>

Or for new people to the team, your mentor should guide you through the
process.

Nicolas


>
> On Thu, Dec 3, 2009 at 3:51 PM, Zelidrag Hornung 
> wrote:
> > How do I know if the repo is read-only? This is how my .gclient looks
> like
> > solutions = [
> >   { "name": "src",
> > "url" : "http://src.chromium.org/svn/trunk/src";,
> > "custom_deps" : {
> >   # To use the trunk of a component instead of what's in DEPS:
> >   #"component": "https://svnserver/component/trunk/";,
> >   # To exclude a component from your working copy:
> >   #"data/really_large_component": None,
> > },
> > "safesync_url": "http://chromium-status.appspot.com/lkgr";
> >   },
> > ]
> >
> >
> > On Thu, Dec 3, 2009 at 3:48 PM, Dan Kegel  wrote:
> >>
> >> Sounds like you're trying to commit to the read-only repo?
> >>
> >> On Thu, Dec 3, 2009 at 3:31 PM, Zelidrag Hornung  >
> >> wrote:
> >> > When I try to commit my change, I am getting following error there -
> any
> >> > ideas what might be causing this?
> >> >>gcl commit my_change
> >> > Presubmit checks took 1.7s to calculate.
> >> > Loaded authentication cookies from
> >> > C:\Users\zelidrag/.codereview_upload_cookies
> >> > svn: Commit failed (details follow):
> >> > svn: Server sent unexpected return value (500 Internal Server Error)
> in
> >> > response to MKACTIVITY request for
> >> > '/svn/!svn/act/aa57d512-f59f-9147-897e-0c6aef5f25fc'
> >> >
> >> > --
> >> > Chromium Developers mailing list: chromium-dev@googlegroups.com
> >> > View archives, change email options, or unsubscribe:
> >> > http://groups.google.com/group/chromium-dev
> >
> >
>
> --
> Chromium Developers mailing list: chromium-dev@googlegroups.com
> View archives, change email options, or unsubscribe:
>http://groups.google.com/group/chromium-dev
>

-- 
Chromium Developers mailing list: chromium-dev@googlegroups.com 
View archives, change email options, or unsubscribe: 
http://groups.google.com/group/chromium-dev

[chromium-dev] Re: Something smells weird on the buildbot

[Nicolas Sylvain]

On Thu, Dec 3, 2009 at 10:25 AM, Ojan Vafai  wrote:

> On Thu, Dec 3, 2009 at 10:21 AM, Dimitri Glazkov 
>  wrote:
>
> How about we turn red for unexpected crashiness?
>>
>
> Makes sense to me. We can just not retry tests that unexpectedly crash.
>
> On Thu, Dec 3, 2009 at 10:09 AM, Dirk Pranke  wrote:
>
>> On Thu, Dec 3, 2009 at 10:07 AM, Ojan Vafai  wrote:
>> > The test is consistently crashing when run with all the other tests, but
>> > passing when we retry it in isolation. Note that the test is listed as
>> an
>> > unexpected flaky test on the waterfall. This is one of the downsides of
>> > retrying failing tests. We can't distinguish flakiness from this case.
>> We
>> > just need to careful to not ignore unexpected flakiness on the
>> waterfall.
>> > Note that the dashboard only shows the result from the first run.
>> Including
>> > the retry results from the bots seems like more trouble than it's worth.
>>
>> Agreed. However, why aren't the webkit bots orange in the main waterfall?
>>
>
> They are orange. We don't show orange in the console view or summary view
> at the top part of the waterfall though.
>
I'm not really sure why. Nicolas, you know?
>
No idea, did you not write this? :)


> I know in the console view we use orange to mean something else, but maybe
> we should not overload the meaning of orange so much. :)
>
I think it's the goal of the console view to try hard not to attribute
flakiness to a change that most likely did not cause it.
The flakiness dashboard does a good job tracking this already.

That said, if someone can find a good color for "fail again", then I'd use
it and use orange like on the waterfall.

Nicolas

-- 
Chromium Developers mailing list: chromium-dev@googlegroups.com 
View archives, change email options, or unsubscribe: 
http://groups.google.com/group/chromium-dev

[chromium-dev] Re: Something smells weird on the buildbot

[Nicolas Sylvain]

On Thu, Dec 3, 2009 at 10:09 AM, Dimitri Glazkov wrote:

> On Thu, Dec 3, 2009 at 10:07 AM, Ojan Vafai  wrote:
> > +chromium-dev as others who look at the waterfall might also be confused.
> > On Thu, Dec 3, 2009 at 8:50 AM, Dimitri Glazkov 
> wrote:
> >>
> >> Sending out random people, because it's early :)
> >>
> >> There's a couple of things I see on the bot this morning:
> >>
> >> 1) There's a crashing test on all bots -- and the tree is still green!
> >>
> >>
> http://src.chromium.org/viewvc/chrome/trunk/src/webkit/tools/layout_tests/flakiness_dashboard.html#tests=LayoutTests/plugins/embed-attributes-setting.html
> >
> >
> > The test is consistently crashing when run with all the other tests, but
> > passing when we retry it in isolation. Note that the test is listed as an
> > unexpected flaky test on the waterfall. This is one of the downsides of
> > retrying failing tests. We can't distinguish flakiness from this case. We
> > just need to careful to not ignore unexpected flakiness on the waterfall.
> > Note that the dashboard only shows the result from the first run.
> Including
> > the retry results from the bots seems like more trouble than it's worth.
>
> Should unexpected flakiness turn the bot red?
>
If it turns the bot red, then it defeats the purpose of that code. Might as
well not retry and mark it as FAIL. (which turns the tree red).

Nicolas


> :DG<
>

-- 
Chromium Developers mailing list: chromium-dev@googlegroups.com 
View archives, change email options, or unsubscribe: 
http://groups.google.com/group/chromium-dev

[chromium-dev] GTTF: results. WAS: revert now, ask questions later? WAS: Reverting a change, the fast way

[Nicolas Sylvain]

Hi,

It looks like our discussion here helped a lot. The tree has never been that
green.

In the past 6 months, we averaged at about 60-65% tree open during work
hours (pacific time). But in
the last month, we are at about 85-90%.  This is really great! Thanks to all
the sheriffs!

I attached a graph of the % open over the last 6 months if you want to see
the progression.
You can also see the raw data at
http://chromium-status.appspot.com/status_viewer.html?curView=peak&startTime=TODAY&numDays=180

There is also a lot more green on
http://build.chromium.org/buildbot/waterfall/console. Thanks to
Ojan for taking care of a lot of webkit flakiness! And also everyone else
who helped fix bugs and
make the bots faster.

Nicolas



On Tue, Nov 3, 2009 at 9:18 PM, Marc-Antoine Ruel 
> wrote:
> > On Tue, Nov 3, 2009 at 9:16 PM, John Abd-El-Malek 
> wrote:
> >>
> >>
> >> On Tue, Nov 3, 2009 at 6:08 PM, Ben Goodger (Google) 
> >> wrote:
> >>>
> >>> I am supportive of auto-revert as long as we apply it universally. So
> >>> many times the tree has been busted forever because of a vacuum of
> action by
> >>> the sheriff.
> >>> Also FYI - the trybots never work for me on my home system. No idea
> why.
> >>
> >> That sounds like a bug, because external contributors are able to use
> it.
> >>  Marc-Antoine?
> >
> > Hadn't had reports before, can look tomorrow. Automagical "protocol"
> > selection must be broken somehow.
> > M-A
>

-- 
Chromium Developers mailing list: chromium-dev@googlegroups.com 
View archives, change email options, or unsubscribe: 
http://groups.google.com/group/chromium-dev<>

Re: [chromium-dev] [GTTF] running tests that fail most often early in the queue

[Nicolas Sylvain]

On Thu, Nov 19, 2009 at 10:48 AM, Paweł Hajdan jr wrote:

> Nicolas, what do you think about applying the reordering to our bots?

we should do it.

>
>
> On Thu, Nov 12, 2009 at 20:10, Pam Greene  wrote:
>
>> On Thu, Nov 12, 2009 at 1:52 AM, Paweł Hajdan Jr.
>>>  wrote:
>>> > I was just looking at the buildbot cycle stats
>>> > at http://build.chromium.org/buildbot/waterfall/stats and realized
>>> that on
>>> > many bots the most frequently failing tests are browser_tests and
>>> ui_tests.
>>> > Then I checked how early they are run by each bot (the earlier we know
>>> about
>>> > the failure, the earlier we can react). So, for example Chromium XP
>>> runs a
>>> > lot of slow page cycler tests before browser_tests, and then another
>>> bunch
>>> > of page cycler tests. page_cycler tests don't fail so frequently, and
>>> when
>>> > they fail, it's frequently some evil flakiness. When browser_tests do
>>> fail
>>> > however, it may indicate something more serious.
>>> > A similar thing is with XP Tests: we're running mini_installer_tests
>>> (which
>>> > take about 2 minutes), and then some other things which rarely fail,
>>> then UI
>>> > tests (which fail frequently), and browser_tests at the end!
>>> > I know that some of these cases are just flaky failures. But by knowing
>>> > earlier about a failure, we'd have more time to decide what to do with
>>> it.
>>>
>>

-- 
Chromium Developers mailing list: chromium-dev@googlegroups.com 
View archives, change email options, or unsubscribe: 
http://groups.google.com/group/chromium-dev

Re: [chromium-dev] Status of Linux 64 Memory Tests?

[Nicolas Sylvain]

>From irc:
  I've only narrowed the webkit roll down by half, compile problems
I have to look at more closely - badness is after 50610 and before 50633

Steve: Maybe we should disable the test while you work on it?

Nicolas

On Wed, Nov 11, 2009 at 8:49 AM, Anthony LaForge  wrote:

> It's seems the memory test on the Linux 64 chromium builder has been red
> for quite a while.  Does anyone know the status of this builder, and if
> anything is underway to address it?
>
>
> http://build.chromium.org/buildbot/waterfall/builders/Chromium%20Linux%20x64
>
> Kind Regards,
>
> Anthony Laforge
> Technical Program Manager
> Mountain View, CA
>
> --
> Chromium Developers mailing list: chromium-dev@googlegroups.com
> View archives, change email options, or unsubscribe:
> http://groups.google.com/group/chromium-dev

-- 
Chromium Developers mailing list: chromium-dev@googlegroups.com 
View archives, change email options, or unsubscribe: 
http://groups.google.com/group/chromium-dev

[chromium-dev] Re: revert now, ask questions later? WAS: Reverting a change, the fast way

[Nicolas Sylvain]

On Wed, Nov 4, 2009 at 12:16 PM, Peter Kasting  wrote:

> On Wed, Nov 4, 2009 at 12:08 PM, James Robinson  wrote:
>
>> I don't see who this benefits - assuming that a given patch is broken and
>> needs a small delta to be correct, it's just as easy to submit a patch with
>> a small delta as it is to submit the small delta. Leaving the broken patch
>> in the tree for any period of time after it's known to be bad is a waste of
>> everybody's time.
>>
>
> For one, the patch isn't always broken.  I've seen a number of cases as
> both sheriff and committer where something "breaks" because the bots needed
> a clobber, not because the patch was wrong.  Or when a test started failing,
> but it was due to some other problem than the patch at hand.  The author is
> frequently in the best position to determine if this is the case.
>
> For another, because if the period of time between breaking and fixing is,
> say, two minutes, then the inconvenience to the rest of the team
> is minuscule (based on our commit frequency, it's likely to be zero),
> whereas the inconvenience to the author of reverting, retesting, reapplying
> is not.  This equation reverses extremely rapidly, which is why I am OK with
> reverts for anything that aren't trivial, immediate fixes.
>
> Finally, because I have seen many cases where the additional cycle time of
> the trivial patch-to-fix was lower than the cycle time of the revert.  The
> revert itself may be fast to perform, but if it touches some core header
> file and causes the bots to rebuild half the world, you're not actually
> saving time in the end.
>
> But I think all this could be summed up in "Try to balance courtesy to the
> author and courtesy to the team; judgment calls are better than unilateral
> statements".
>
I agree.  But for someone who is new to the team and is sheriffing for the
first time, it might be hard to apply this rule.
I think we should still try to have a specific rule, and make it clear that
good judgement is always more important.

Nicolas


> Not that this bikeshed debate matters anyway.  Whoever is sheriff runs the
> tree how they want to, in the end.
>
> PK
>

--~--~-~--~~~---~--~~
Chromium Developers mailing list: chromium-dev@googlegroups.com 
View archives, change email options, or unsubscribe: 
http://groups.google.com/group/chromium-dev
-~--~~~~--~~--~--~---

[chromium-dev] Re: revert now, ask questions later? WAS: Reverting a change, the fast way

[Nicolas Sylvain]

On Wed, Nov 4, 2009 at 11:40 AM, Peter Kasting  wrote:

> On Tue, Nov 3, 2009 at 6:05 PM, John Abd-El-Malek wrote:
>
>> But this means that the person didn't use the trybot.
>>
>> I think we need to be harsher on people who commit with changes that
>> didn't complete or failed on the trybot.
>>
>
> There are a large number of reasons why the trybots can have false
> negatives or false positives, or why it's easy to break things even when
> you're trying to make use of them.  Trybots are great, they're not a
> panacea.
>
> I strongly oppose any kind of "immediate auto-revert" policy.
>
I don't think anyone suggested "immediate auto revert".

Try bots are not perfect. They won't get all the failures. But even if it's
not entirely your fault, it does not mean that your change deserves to be in
the tree.

It sucks to have your change reverted, and it will add a 10-minute overhead
for you to un-revert the change on your working copy, but at least it won't
keep the tree closed until you try to figure out what the problem is.

And you can blame this 10-minute time wasted on the try bots. We always keep
adding new things to it... but unless we get thousands of machines (which we
don't have the capacity of handling right now), we won't be able to test
everything and build all possible configurations.

Nicolas


> PK
>

--~--~-~--~~~---~--~~
Chromium Developers mailing list: chromium-dev@googlegroups.com 
View archives, change email options, or unsubscribe: 
http://groups.google.com/group/chromium-dev
-~--~~~~--~~--~--~---

[chromium-dev] Re: Reverting a change, the fast way

[Nicolas Sylvain]

On Tue, Nov 3, 2009 at 10:50 PM, Darin Fisher  wrote:

> Yeah, does anyone know why SVN does that?  It seems like it could easily
> inspect the revision to see what files changed, and then only merge those
> files :-/
>
> depot_tools/revert also works this way.  Should we remove
> depot_tools/revert, or re-implement it on top of drover?
>
I think we should. Or just make "gcl revert XXX" call "drover --revert "
 and delete revert completely.  I also think
it would make sense to add a "--local" flag to that changes the local
workspace instead of doing it in a temporary
directory.

Nicolas


> -Darin
>
>
> On Tue, Nov 3, 2009 at 4:34 PM, Anthony LaForge wrote:
>
>> Drover is actually an order of magnitude faster than a standard svn
>> merge/reverse, merge since it only affects the files that are involved with
>> the CL (as opposed to trying to descending the entire trunk).  A normal
>> merge would take in the space of 8 minutes on my box (which required an
>> existing working copy).  In the case of drover, the same merge/revert can be
>> performed in approximately 30 seconds to 1 minute (w/ out a pre existing
>> working copy).
>>
>> Kind Regards,
>>
>> Anthony Laforge
>> Technical Program Manager
>> Mountain View, CA
>>
>>
>>
>> On Tue, Nov 3, 2009 at 2:01 PM, Eric Seidel  wrote:
>>
>>>
>>> Interesting.  WebKit has similar functionality as:
>>> bugzilla-tool rollout 12345
>>>
>>> We've found that git reverts are at least an order of magnitude faster
>>> than SVN reverse merges.
>>>
>>> I wonder what bugzilla-tool/drover can learn from one another.
>>>
>>> -eric
>>>
>>> On Tue, Nov 3, 2009 at 1:53 PM, Nicolas Sylvain 
>>> wrote:
>>> > Hi
>>> > One of the goal of the Green Tree Task Force was to make reverting a
>>> change
>>> > easy and fast.
>>> > This is really important to keep the tree green and open. The
>>> old saying
>>> > is "Revert now, think later"... If a change broke the build and the fix
>>> > would
>>> > take more than 1 or 2 minutes to be committed, or if the committer is
>>> not
>>> > answering pings, then the change needs to be reverted asap.
>>> > Turns out it was already easy and fast with Anthony's drover tool.
>>> > If you have depot_tools in your path, just type :
>>> > drover --revert 
>>> > where  is the revision you want to revert.
>>> > Since the tool creates temporary files, I suggest you go to the temp
>>> > directory first.
>>> > It should take only a few seconds, even for a really big change. You
>>> don't
>>> > even need
>>> > to have a chrome checkout on your machine.
>>> > This tool should work on all platforms. You need svn 1.5 or above in
>>> your
>>> > path.
>>> > If you run into any issues with the tool, please let me know.
>>> > Thank you,
>>> > Nicolas
>>> >
>>> > 
>>> > If the tool does not work, you can revert the long way by doing:
>>> > cd src/
>>> > svn update
>>> > svn merge -c - .
>>> > (the dot at the end of the previous line is important)
>>> >
>>> > >
>>> >
>>>
>>>
>>>
>>
>> >>
>>
>

--~--~-~--~~~---~--~~
Chromium Developers mailing list: chromium-dev@googlegroups.com 
View archives, change email options, or unsubscribe: 
http://groups.google.com/group/chromium-dev
-~--~~~~--~~--~--~---

[chromium-dev] Re: revert now, ask questions later? WAS: Reverting a change, the fast way

[Nicolas Sylvain]

On Tue, Nov 3, 2009 at 10:38 PM, Drew Wilson  wrote:

> Do the trybots build the release version? Because I had a build break last
> week that passed the 3 basic trybots, but failed to compile on the release
> buildbots because of a missing include which was apparently pulled in
> through other means in the debug version.

No, they do not currently build the release version.

Nicolas


>
> -atw
>
>
> On Tue, Nov 3, 2009 at 7:30 PM, Nicolas Sylvain wrote:
>
>>
>>
>> On Tue, Nov 3, 2009 at 7:46 PM, Kenneth Russell  wrote:
>>
>>> On Tue, Nov 3, 2009 at 6:05 PM, John Abd-El-Malek 
>>> wrote:
>>> > But this means that the person didn't use the trybot.
>>> > I think we need to be harsher on people who commit with changes that
>>> didn't
>>> > complete or failed on the trybot.  They need to have a really good
>>> reason as
>>> > to why they want to try their change on the buildbot and possibly delay
>>> many
>>> > other engineers.
>>>
>>> For the record, I completely support immediate backouts of changes
>>> that break the tree, and agree that all changes should go through the
>>> trybots -- but sometimes the trybots don't work. I don't know anything
>>> about the architectural differences between the trybots and buildbots,
>>> but from recent experience I think the trybots are trying to do
>>> incremental builds, when that isn't guaranteed to always work.
>>>
>> even the bots on the main waterfall do incremental builds (except some of
>> them).
>> If the change requires a clobber, use "gcl try CHANGENAME -c" to run the
>> code
>> on the try bot doing a full build.
>>
>>>
>>> If it's just a matter of throwing hardware at the problem of making
>>> the trybots nearly 100% reliable I think we should make that
>>> investment.
>>
>>
>>> -Ken
>>>
>>> > On Tue, Nov 3, 2009 at 3:11 PM, Ben Goodger (Google) >> >
>>> > wrote:
>>> >>
>>> >> The most common case of "< 5 minute" bustage fix is "file was omitted
>>> >> from changelist".
>>> >>
>>> >> -Ben
>>> >>
>>> >> On Tue, Nov 3, 2009 at 2:34 PM, Peter Kasting 
>>> wrote:
>>> >> > On Tue, Nov 3, 2009 at 2:08 PM, Ojan Vafai 
>>> wrote:
>>> >> >>
>>> >> >> To be clear, here's the proposed policy: Any change that would
>>> close
>>> >> >> the
>>> >> >> tree can be reverted if it can't be fixed in <2 minutes.
>>> >> >
>>> >> > How about:
>>> >> > If a change closes the tree, the change author has 1 or 2 minutes to
>>> >> > respond
>>> >> > to a ping.  The change should be reverted if the author doesn't
>>> respond,
>>> >> > if
>>> >> > he says to revert, or if he does not say he has a fix within the
>>> next 5
>>> >> > minutes.
>>> >> > I can't fix _any_ problem in 2 minutes.  But I can fix most of them
>>> in
>>> >> > 5.
>>> >> >  The goal is to allow the author a reasonable chance to fix trivial
>>> >> > problems
>>> >> > before we revert.  And I think the tree should go ahead and close
>>> during
>>> >> > that interval.
>>> >> > PK
>>> >> > >
>>> >> >
>>> >>
>>> >>
>>> >
>>> >
>>> > >
>>> >
>>>
>>
>>
>> >>
>>
>

--~--~-~--~~~---~--~~
Chromium Developers mailing list: chromium-dev@googlegroups.com 
View archives, change email options, or unsubscribe: 
http://groups.google.com/group/chromium-dev
-~--~~~~--~~--~--~---

[chromium-dev] Re: revert now, ask questions later? WAS: Reverting a change, the fast way

[Nicolas Sylvain]

On Tue, Nov 3, 2009 at 7:46 PM, Kenneth Russell  wrote:

> On Tue, Nov 3, 2009 at 6:05 PM, John Abd-El-Malek 
> wrote:
> > But this means that the person didn't use the trybot.
> > I think we need to be harsher on people who commit with changes that
> didn't
> > complete or failed on the trybot.  They need to have a really good reason
> as
> > to why they want to try their change on the buildbot and possibly delay
> many
> > other engineers.
>
> For the record, I completely support immediate backouts of changes
> that break the tree, and agree that all changes should go through the
> trybots -- but sometimes the trybots don't work. I don't know anything
> about the architectural differences between the trybots and buildbots,
> but from recent experience I think the trybots are trying to do
> incremental builds, when that isn't guaranteed to always work.
>
even the bots on the main waterfall do incremental builds (except some of
them).
If the change requires a clobber, use "gcl try CHANGENAME -c" to run the
code
on the try bot doing a full build.

>
> If it's just a matter of throwing hardware at the problem of making
> the trybots nearly 100% reliable I think we should make that
> investment.


> -Ken
>
> > On Tue, Nov 3, 2009 at 3:11 PM, Ben Goodger (Google) 
> > wrote:
> >>
> >> The most common case of "< 5 minute" bustage fix is "file was omitted
> >> from changelist".
> >>
> >> -Ben
> >>
> >> On Tue, Nov 3, 2009 at 2:34 PM, Peter Kasting 
> wrote:
> >> > On Tue, Nov 3, 2009 at 2:08 PM, Ojan Vafai  wrote:
> >> >>
> >> >> To be clear, here's the proposed policy: Any change that would close
> >> >> the
> >> >> tree can be reverted if it can't be fixed in <2 minutes.
> >> >
> >> > How about:
> >> > If a change closes the tree, the change author has 1 or 2 minutes to
> >> > respond
> >> > to a ping.  The change should be reverted if the author doesn't
> respond,
> >> > if
> >> > he says to revert, or if he does not say he has a fix within the next
> 5
> >> > minutes.
> >> > I can't fix _any_ problem in 2 minutes.  But I can fix most of them in
> >> > 5.
> >> >  The goal is to allow the author a reasonable chance to fix trivial
> >> > problems
> >> > before we revert.  And I think the tree should go ahead and close
> during
> >> > that interval.
> >> > PK
> >> > >
> >> >
> >>
> >>
> >
> >
> > > >
> >
>

--~--~-~--~~~---~--~~
Chromium Developers mailing list: chromium-dev@googlegroups.com 
View archives, change email options, or unsubscribe: 
http://groups.google.com/group/chromium-dev
-~--~~~~--~~--~--~---

[chromium-dev] Re: revert now, ask questions later? WAS: Reverting a change, the fast way

[Nicolas Sylvain]

+1

On Tue, Nov 3, 2009 at 3:38 PM, Avi Drissman  wrote:

> I'm OK with that.
>
> Just make it clear that the sheriff does have authority. One time when I
> was sheriff I wanted to revert a broken patch. The author insisted on
> patching it over and over. He finally got it working about about seven
> patches and nearly three hours or so, when I was insisting on backing it out
> after the first 30m.
>
Yes, this is exactly what we want to avoid.

The 2-minute rule usually includes:
"Oops, I forgot to commit a file"
"Let me disable the test I just added, it clearly does not work"
"Oops, before committing I renamed a variable and forgot to change it at one
place"

It also use to mean:
"Oops, I forgot an include". But this one has been biting us to much in the
past, so I leave it at the discretion of the sheriff.

I think people need to use their good judgement too. The length of a minute
should be inversely proportional to the number of people trying to commit
during this time of the day.

Nicolas


> Avi
>
> On Tue, Nov 3, 2009 at 5:34 PM, Peter Kasting  wrote:
>
>> On Tue, Nov 3, 2009 at 2:08 PM, Ojan Vafai  wrote:
>>
>>> To be clear, here's the proposed policy: Any change that would close the
>>> tree can be reverted if it can't be fixed in <2 minutes.
>>>
>>
>> How about:
>>
>> If a change closes the tree, the change author has 1 or 2 minutes to
>> respond to a ping.  The change should be reverted if the author doesn't
>> respond, if he says to revert, or if he does not say he has a fix within the
>> next 5 minutes.
>>
>> I can't fix _any_ problem in 2 minutes.  But I can fix most of them in 5.
>>  The goal is to allow the author a reasonable chance to fix trivial problems
>> before we revert.  And I think the tree should go ahead and close during
>> that interval.
>>
>> PK
>>
>> >>
>>
>

--~--~-~--~~~---~--~~
Chromium Developers mailing list: chromium-dev@googlegroups.com 
View archives, change email options, or unsubscribe: 
http://groups.google.com/group/chromium-dev
-~--~~~~--~~--~--~---

[chromium-dev] Re: revert now, ask questions later? WAS: Reverting a change, the fast way

[Nicolas Sylvain]

On Tue, Nov 3, 2009 at 4:03 PM, Eric Seidel  wrote:

>
> Could we just automate rollouts and this "5-minute timer"?  If we have
> the tools to do automated rollouts, would it be reasonable to add them
> as a phase in the buildbot?
>
I looked into this, and implemented a proof of concept, but ditched it
because I don't think there was enough benefit.

It is so easy to revert with drover than everyone can do it really fast, and
if
we automate it, we'll still need someone to sign off on it, because maybe
the
compile failure was a bot problem, or maybe it needed a clobber.  We can't
just
blindly revert changes.

Nicolas

>
> On Tue, Nov 3, 2009 at 2:52 PM, Nicolas Sylvain 
> wrote:
> > +1
> >
> > On Tue, Nov 3, 2009 at 3:38 PM, Avi Drissman  wrote:
> >>
> >> I'm OK with that.
> >>
> >> Just make it clear that the sheriff does have authority. One time when I
> >> was sheriff I wanted to revert a broken patch. The author insisted on
> >> patching it over and over. He finally got it working about about seven
> >> patches and nearly three hours or so, when I was insisting on backing it
> out
> >> after the first 30m.
> >
> > Yes, this is exactly what we want to avoid.
> > The 2-minute rule usually includes:
> > "Oops, I forgot to commit a file"
> > "Let me disable the test I just added, it clearly does not work"
> > "Oops, before committing I renamed a variable and forgot to change it at
> one
> > place"
> > It also use to mean:
> > "Oops, I forgot an include". But this one has been biting us to much in
> the
> > past, so I leave it at the discretion of the sheriff.
> > I think people need to use their good judgement too. The length of a
> minute
> > should be inversely proportional to the number of people trying to commit
> > during this time of the day.
> > Nicolas
> >>
> >> Avi
> >>
> >> On Tue, Nov 3, 2009 at 5:34 PM, Peter Kasting 
> wrote:
> >>>
> >>> On Tue, Nov 3, 2009 at 2:08 PM, Ojan Vafai  wrote:
> >>>>
> >>>> To be clear, here's the proposed policy: Any change that would close
> the
> >>>> tree can be reverted if it can't be fixed in <2 minutes.
> >>>
> >>> How about:
> >>> If a change closes the tree, the change author has 1 or 2 minutes to
> >>> respond to a ping.  The change should be reverted if the author doesn't
> >>> respond, if he says to revert, or if he does not say he has a fix
> within the
> >>> next 5 minutes.
> >>> I can't fix _any_ problem in 2 minutes.  But I can fix most of them in
> 5.
> >>>  The goal is to allow the author a reasonable chance to fix trivial
> problems
> >>> before we revert.  And I think the tree should go ahead and close
> during
> >>> that interval.
> >>> PK
> >>>
> >>
> >
> >
> > >
> >
>
> >
>

--~--~-~--~~~---~--~~
Chromium Developers mailing list: chromium-dev@googlegroups.com 
View archives, change email options, or unsubscribe: 
http://groups.google.com/group/chromium-dev
-~--~~~~--~~--~--~---

[chromium-dev] Reverting a change, the fast way

[Nicolas Sylvain]

Hi

One of the goal of the Green Tree Task Force was to make reverting a change
easy and fast.

This is really important to keep the tree green and open. The old saying
is "Revert now, think later"... If a change broke the build and the fix
would
take more than 1 or 2 minutes to be committed, or if the committer is not
answering pings, then the change needs to be reverted asap.

Turns out it was already easy and fast with Anthony's drover tool.

If you have depot_tools in your path, just type :

*drover --revert *

where  is the revision you want to revert.

Since the tool creates temporary files, I suggest you go to the temp
directory first.

It should take only a few seconds, even for a really big change. You don't
even need
to have a chrome checkout on your machine.

This tool should work on all platforms. You need svn 1.5 or above in your
path.

If you run into any issues with the tool, please let me know.

Thank you,

Nicolas




If the tool does not work, you can revert the long way by doing:
cd src/
svn update
svn merge -c - .
(the dot at the end of the previous line is important)

--~--~-~--~~~---~--~~
Chromium Developers mailing list: chromium-dev@googlegroups.com 
View archives, change email options, or unsubscribe: 
http://groups.google.com/group/chromium-dev
-~--~~~~--~~--~--~---

[chromium-dev] Re: revert now, ask questions later? WAS: Reverting a change, the fast way

[Nicolas Sylvain]

On Tue, Nov 3, 2009 at 7:08 PM, Ben Goodger (Google) wrote:

> I am supportive of auto-revert as long as we apply it universally. So many
> times the tree has been busted forever because of a vacuum of action by the
> sheriff.
>
> Also FYI - the trybots never work for me on my home system. No idea why.
>
>From home you to type something like :

gcl try CHANGENAME --use_svn --svn_repo=svn://svn.chromium.org/chrome-try/try


Nicolas

>
> -Ben
>
>
> On Tue, Nov 3, 2009 at 7:05 PM, John Abd-El-Malek wrote:
>
>> But this means that the person didn't use the trybot.
>>
>> I think we need to be harsher on people who commit with changes that
>> didn't complete or failed on the trybot.  They need to have a really good
>> reason as to why they want to try their change on the buildbot and possibly
>> delay many other engineers.
>>
>>
>> On Tue, Nov 3, 2009 at 3:11 PM, Ben Goodger (Google) 
>> wrote:
>>
>>>
>>> The most common case of "< 5 minute" bustage fix is "file was omitted
>>> from changelist".
>>>
>>> -Ben
>>>
>>> On Tue, Nov 3, 2009 at 2:34 PM, Peter Kasting 
>>> wrote:
>>> > On Tue, Nov 3, 2009 at 2:08 PM, Ojan Vafai  wrote:
>>> >>
>>> >> To be clear, here's the proposed policy: Any change that would close
>>> the
>>> >> tree can be reverted if it can't be fixed in <2 minutes.
>>> >
>>> > How about:
>>> > If a change closes the tree, the change author has 1 or 2 minutes to
>>> respond
>>> > to a ping.  The change should be reverted if the author doesn't
>>> respond, if
>>> > he says to revert, or if he does not say he has a fix within the next 5
>>> > minutes.
>>> > I can't fix _any_ problem in 2 minutes.  But I can fix most of them in
>>> 5.
>>> >  The goal is to allow the author a reasonable chance to fix trivial
>>> problems
>>> > before we revert.  And I think the tree should go ahead and close
>>> during
>>> > that interval.
>>> > PK
>>> > >
>>> >
>>>
>>> >>>
>>>
>>
>

--~--~-~--~~~---~--~~
Chromium Developers mailing list: chromium-dev@googlegroups.com 
View archives, change email options, or unsubscribe: 
http://groups.google.com/group/chromium-dev
-~--~~~~--~~--~--~---

[chromium-dev] Re: Reverting a change, the fast way

[Nicolas Sylvain]

On Tue, Nov 3, 2009 at 3:01 PM, Nico Weber  wrote:

> I tried that a few days ago, and drover died with me with something
> along the lines of "Can't talk to chrome-svn" (which as far as I
> understand is some internal svn repo?). A quick glance at the source
> confirms that this is still the case.
>

Good point, it should have a way to specify which url it should use to
checkout from.

by default it uses the internal url. I'll try to find a way to fix that.

Nicolas



>
> On Tue, Nov 3, 2009 at 1:53 PM, Nicolas Sylvain 
> wrote:
> > Hi
> > One of the goal of the Green Tree Task Force was to make reverting a
> change
> > easy and fast.
> > This is really important to keep the tree green and open. The old saying
> > is "Revert now, think later"... If a change broke the build and the fix
> > would
> > take more than 1 or 2 minutes to be committed, or if the committer is not
> > answering pings, then the change needs to be reverted asap.
> > Turns out it was already easy and fast with Anthony's drover tool.
> > If you have depot_tools in your path, just type :
> > drover --revert 
> > where  is the revision you want to revert.
> > Since the tool creates temporary files, I suggest you go to the temp
> > directory first.
> > It should take only a few seconds, even for a really big change. You
> don't
> > even need
> > to have a chrome checkout on your machine.
> > This tool should work on all platforms. You need svn 1.5 or above in your
> > path.
> > If you run into any issues with the tool, please let me know.
> > Thank you,
> > Nicolas
> >
> > 
> > If the tool does not work, you can revert the long way by doing:
> > cd src/
> > svn update
> > svn merge -c - .
> > (the dot at the end of the previous line is important)
> >
> > > >
> >
>

--~--~-~--~~~---~--~~
Chromium Developers mailing list: chromium-dev@googlegroups.com 
View archives, change email options, or unsubscribe: 
http://groups.google.com/group/chromium-dev
-~--~~~~--~~--~--~---

[chromium-dev] Re: Flaky layout tests and WebKit Linux (dbg)(3)

[Nicolas Sylvain]

On Fri, Oct 23, 2009 at 2:16 PM, Andrew Scherkus wrote:

> On Fri, Oct 23, 2009 at 12:28 PM, Nicolas Sylvain 
> wrote:
>
>>
>>
>> On Fri, Oct 23, 2009 at 12:21 PM, Andrew Scherkus 
>> wrote:
>>
>>> I've never witnessed these tests taking an extra 10-20 seconds on my
>>> local machine, no.
>>>
>>> I don't doubt that some of the tests might be flaky themselves, but that
>>> machine does run tests slower.  Take a look at the SVG tests, for example:
>>>
>>> http://src.chromium.org/viewvc/chrome/trunk/src/webkit/tools/layout_tests/flakiness_dashboard.html#tests=LayoutTests%2Fsvg
>>>
>>> Are there other tricks I do on my local machine to simulate running on
>>> the bots?  I usually try to test for these things by maxing out my CPU then
>>> running layout tests but even then they run smoothly.
>>>
>>
>> This machine is one of the oldest linux machine we have in the lab. I'll
>> recreate it, make sure it run fast, and see if it helps.
>>
>
> Thanks Nicolas.
>

I changed the machine, and so far so good. The time to run the layout tests
on this machine went from 11minutes to 5 minutes. Looks like
something was wrong on the machine after all.  There are still some
flakiness... but at least they are not TIMEOUT.

Thanks

Nicolas


>
> I'm wondering if it's slow disk access... know of any simple commands I can
> use to simulate disk thrashing?
>
>
>>
>> Nicolas
>>
>>
>>> Andrew
>>>
>>> On Fri, Oct 23, 2009 at 12:02 PM, Nicolas Sylvain >> > wrote:
>>>
>>>>
>>>>
>>>> On Fri, Oct 23, 2009 at 11:59 AM, Andrew Scherkus <
>>>> scher...@chromium.org> wrote:
>>>>
>>>>> I've been trying to get the media layout tests passing consistently,
>>>>> but WebKit Linux (dbg)(3) takes an absurdly longer time to run tests and I
>>>>> don't know why.
>>>>>
>>>>> For example:
>>>>>
>>>>> http://src.chromium.org/viewvc/chrome/trunk/src/webkit/tools/layout_tests/flakiness_dashboard.html#tests=video-played
>>>>>
>>>>> To keep the tree green (and collect data), I've marked all media layout
>>>>> tests on Linux debug as pass/fail/timeout.  My hope is if the bot was less
>>>>> bogged down, it would lead to faster build times (GTTF) and less flaky
>>>>> results/timeouts (LTTF).
>>>>>
>>>>> This machine is supposed to be fast.
>>>>
>>>> Are you saying that this flakiness never happens on your machine?
>>>>
>>>> Are you sure the bot is really to blame here?
>>>>
>>>> Nicolas
>>>>
>>>> Any ideas?
>>>>> Andrew
>>>>>
>>>>>
>>>>> >>>>>
>>>>>
>>>>
>>>
>>
>

--~--~-~--~~~---~--~~
Chromium Developers mailing list: chromium-dev@googlegroups.com 
View archives, change email options, or unsubscribe: 
http://groups.google.com/group/chromium-dev
-~--~~~~--~~--~--~---

[chromium-dev] Re: revising the output from run_webkit_tests

[Nicolas Sylvain]

On Fri, Oct 23, 2009 at 3:43 PM, Dirk Pranke  wrote:

>
> If you've never run run_webkit_tests to run the layout test
> regression, or don't care about it, you can stop reading ...
>
> If you have run it, and you're like me, you've probably wondered a lot
> about the output ... questions like:
>
> 1) what do the numbers printed at the beginning of the test mean?
> 2) what do all of these "test failed" messages mean, and are they bad?
> 3) what do the numbers printed at the end of the test mean?
> 4) why are the numbers at the end different from the numbers at the
> beginning?
> 5) did my regression run cleanly, or not?
>
> You may have also wondered a couple of other things:
> 6) What do we expect this test to do?
> 7) Where is the baseline for this test?
> 8) What is the baseline search path for this test?
>
> Having just spent a week trying (again), to reconcile the numbers I'm
> getting on the LTTF dashboard with what we print out in the test, I'm
> thinking about drastically revising the output from the script,
> roughly as follows:
>
> * print the information needed to reproduce the test and look at the
> results
> * print the expected results in summary form (roughly the expanded
> version of the first table in the dashboard - # of tests by
> (wontfix/fix/defer x pass/fail/are flaky).
> * don't print out failure text to the screen during the run
> * print out any *unexpected* results at the end (like we do today)
>
> The goal would be that if all of your tests pass, you get less than a
> small screenful of output from running the tests.
>
> In addition, we would record a full log of (test,expectation,result)
> to the results directory (and this would also be available onscreen
> with --verbose)
>
> Lastly, I'll add a flag to re-run the tests that just failed, so it's
> easy to test if the failures were flaky.
>
This would be nice for the buildbots. We would also need to add a new
section
in the results for Unexpected Flaky Tests (failed then passed).

Nicolas


>
> Then I'll rip out as much of the set logic in test_expectations.py as
> we can possibly get away with, so that no one has to spend the week I
> just did again. I'll probably replace it with much of the logic I use
> to generate the dashboard, which is much more flexible in terms of
> extracting different types of queries and numbers.
>
> I think the net result will be the same level of information that we
> get today, just in much more meaningful form.
>
> Thoughts? Comments? Is anyone particularly wedded to the existing
> output, or worried about losing a particular piece of info?
>
> -- Dirk
>
> >
>

--~--~-~--~~~---~--~~
Chromium Developers mailing list: chromium-dev@googlegroups.com 
View archives, change email options, or unsubscribe: 
http://groups.google.com/group/chromium-dev
-~--~~~~--~~--~--~---

[chromium-dev] Re: Flaky layout tests and WebKit Linux (dbg)(3)

[Nicolas Sylvain]

On Fri, Oct 23, 2009 at 12:21 PM, Andrew Scherkus wrote:

> I've never witnessed these tests taking an extra 10-20 seconds on my local
> machine, no.
>
> I don't doubt that some of the tests might be flaky themselves, but that
> machine does run tests slower.  Take a look at the SVG tests, for example:
>
> http://src.chromium.org/viewvc/chrome/trunk/src/webkit/tools/layout_tests/flakiness_dashboard.html#tests=LayoutTests%2Fsvg
>
> Are there other tricks I do on my local machine to simulate running on the
> bots?  I usually try to test for these things by maxing out my CPU then
> running layout tests but even then they run smoothly.
>

This machine is one of the oldest linux machine we have in the lab. I'll
recreate it, make sure it run fast, and see if it helps.

Nicolas


> Andrew
>
> On Fri, Oct 23, 2009 at 12:02 PM, Nicolas Sylvain 
> wrote:
>
>>
>>
>> On Fri, Oct 23, 2009 at 11:59 AM, Andrew Scherkus 
>> wrote:
>>
>>> I've been trying to get the media layout tests passing consistently, but
>>> WebKit Linux (dbg)(3) takes an absurdly longer time to run tests and I don't
>>> know why.
>>> For example:
>>>
>>> http://src.chromium.org/viewvc/chrome/trunk/src/webkit/tools/layout_tests/flakiness_dashboard.html#tests=video-played
>>>
>>> To keep the tree green (and collect data), I've marked all media layout
>>> tests on Linux debug as pass/fail/timeout.  My hope is if the bot was less
>>> bogged down, it would lead to faster build times (GTTF) and less flaky
>>> results/timeouts (LTTF).
>>>
>>> This machine is supposed to be fast.
>>
>> Are you saying that this flakiness never happens on your machine?
>>
>> Are you sure the bot is really to blame here?
>>
>> Nicolas
>>
>> Any ideas?
>>> Andrew
>>>
>>>
>>> >>>
>>>
>>
>

--~--~-~--~~~---~--~~
Chromium Developers mailing list: chromium-dev@googlegroups.com 
View archives, change email options, or unsubscribe: 
http://groups.google.com/group/chromium-dev
-~--~~~~--~~--~--~---

[chromium-dev] Re: Flaky layout tests and WebKit Linux (dbg)(3)

[Nicolas Sylvain]

On Fri, Oct 23, 2009 at 11:59 AM, Andrew Scherkus wrote:

> I've been trying to get the media layout tests passing consistently, but
> WebKit Linux (dbg)(3) takes an absurdly longer time to run tests and I don't
> know why.
> For example:
>
> http://src.chromium.org/viewvc/chrome/trunk/src/webkit/tools/layout_tests/flakiness_dashboard.html#tests=video-played
>
> To keep the tree green (and collect data), I've marked all media layout
> tests on Linux debug as pass/fail/timeout.  My hope is if the bot was less
> bogged down, it would lead to faster build times (GTTF) and less flaky
> results/timeouts (LTTF).
>
> This machine is supposed to be fast.

Are you saying that this flakiness never happens on your machine?

Are you sure the bot is really to blame here?

Nicolas

Any ideas?
> Andrew
>
>
> >
>

--~--~-~--~~~---~--~~
Chromium Developers mailing list: chromium-dev@googlegroups.com 
View archives, change email options, or unsubscribe: 
http://groups.google.com/group/chromium-dev
-~--~~~~--~~--~--~---

[chromium-dev] Re: New Tab Cold 20% perf regression on xp-release

[Nicolas Sylvain]

On Fri, Oct 16, 2009 at 2:35 PM, Erik Arvidsson  wrote:

>
> The performance for New Tab Cold has regressed without about 20%
> sometime before rev28918.
>
This seems to be when i changed the hardware where this test is running.
Where is
the reference build?

Nicolas


>
>
> http://build.chromium.org/buildbot/perf/xp-release/new-tab-ui-cold/report.html?history=150
>
> The graph tells us that it is something in the rendering path. My main
> suspect atm is the change to use system colors on windows since
> neither Linux nor Mac was affected by this.
>
> --
> erik
>
> >
>

--~--~-~--~~~---~--~~
Chromium Developers mailing list: chromium-dev@googlegroups.com 
View archives, change email options, or unsubscribe: 
http://groups.google.com/group/chromium-dev
-~--~~~~--~~--~--~---

[chromium-dev] Re: layout test dashboard goofup

[Nicolas Sylvain]

On Wed, Oct 14, 2009 at 4:53 PM, Ojan Vafai  wrote:

> The data is stored in a single file per bot. For example, the webkit
> release bot's results are at
> http://build.chromium.org/buildbot/layout_test_results/webkit-rel/results.json.
>  That
> file holds all the historical data for that bot and is copied over during
> the archive step of each run. We intentionally limit the number of results
> we keep in that file to 750 runs to keep filesize down. In my accidental
> checking, I changed 750 to 9. :(

A little bit unrelated: This data, along with all the data on
build.chromium.org, is replicated on at least 4 machines. It would be easy
to recover the data if the server dies for example.  We are also planning to
do daily backups, but the data is huge.  For example, we archive 25GB of new
layout test results every day.

Nicolas


> A trivial to implement "backup" would be to also copy the file to the
> archive location for just that run (same place as where we copy
> layout_test_results.zip), e.g. also copy it to
> http://build.chromium.org/buildbot/layout_test_results/webkit-rel/29056/.
> The downside is that this uses up disk space (e.g. the largest results.json
> file was 25mb before being clobbered).
>
> Another problem with backing up is that you'd also have to find a way to
> restore from backup that didn't lose data from runs that happened since the
> problem occurred. Merging the two files results.json should be pretty
> relatively trivial code, but it's all code that someone would need to write
> and test.
>
> While it sucks, I don't think backing up this data is worth the effort.
> It's a temporary productivity hit for the team, but we get enough new data
> to make reasonable decisions relatively quickly. Mistakes like this are very
> rare. It will become even more rare as coding work on the dashboard winds
> down.
>
> Feel free to have at it if you disagree.
>
> Creates the results.json file and it's content:
> trunk/src/webkit/tools/layout_tests/layout_package/json_results_generator.py
>  Copies the results.json file to the right
> place: 
> trunk/tools/buildbot/scripts/slave/chromium/archive_layout_test_results.py
>
> Ojan
>
>
> On Wed, Oct 14, 2009 at 4:24 PM, Jeremy Orlow  wrote:
>
>> I haven't actually gotten anything done on LocalStorage this week because
>> I've been doing so many small side projects like this.but if it's a
>> priority, sure.
>> How about a cron job on some machine that ssh's via a cert into whatever
>> machines the data is stored on, pulls it back, and dumps it into some dir?
>>  When we start filling up the hard drive, we can look at doing something
>> smarter, deleting old data, or putting it somewhere like GFS.
>>
>> What server can I use and where's the data stored?
>>
>> On Wed, Oct 14, 2009 at 4:17 PM, Evan Martin  wrote:
>>
>>> Sounds like we've got a volunteer!  :D :D :D
>>>
>>> On Wed, Oct 14, 2009 at 4:15 PM, Jeremy Orlow 
>>> wrote:
>>> > I assume we're going to start backing this data up from now on?
>>> >
>>> > On Wed, Oct 14, 2009 at 4:13 PM, Peter Kasting 
>>> wrote:
>>> >>
>>> >> On Wed, Oct 14, 2009 at 3:58 PM, Ojan Vafai  wrote:
>>> >>>
>>> >>> I accidentally checked in some test code (one number was wrong!) and
>>> >>> clobbered all but 10 of the runs of data for each builder. There's no
>>> way to
>>> >>> recover it.
>>> >>
>>> >> Do you moonlight for the Danger team at Microsoft?
>>> >> PK
>>> >>
>>> >
>>> >
>>> > >
>>> >
>>>
>>
>>
>
> >
>

--~--~-~--~~~---~--~~
Chromium Developers mailing list: chromium-dev@googlegroups.com 
View archives, change email options, or unsubscribe: 
http://groups.google.com/group/chromium-dev
-~--~~~~--~~--~--~---

[chromium-dev] Re: [LTTF][WebKit Gardening]: Keeping up with the weeds.

[Nicolas Sylvain]

On Tue, Oct 13, 2009 at 10:45 AM, Peter Kasting  wrote:

> On Tue, Oct 13, 2009 at 10:43 AM, Dimitri Glazkov 
> wrote:
>
>> Let's not conflate the two. There are flakes, and there are clearly,
>> consistently failing tests, arriving in chunks every day via WebKit
>> rolls.
>
>
> OK, I'm just saying that my observations are that the obvious, consistent
> failing tests that arrive are like 0.1% compared with the flaky tests.  I
> added many dozens of flaky test lines in my last sheriffing stint and the
> number of additional failures from WebKit updates was like 3.  So I don't
> agree with your assessment of the problem space.
>
It's because the sheriff don't notice the new failing tests, because most of
the time the gardener does a good job of updating the list at the same time
as the merge, so the tree stays mostly green.
See http://src.chromium.org/viewvc/chrome?view=rev&revision=28727

What sheriffs see is leftovers and flakiness introduced by the merge.

Nicolas



Nicolas


> PK
>
> >
>

--~--~-~--~~~---~--~~
Chromium Developers mailing list: chromium-dev@googlegroups.com 
View archives, change email options, or unsubscribe: 
http://groups.google.com/group/chromium-dev
-~--~~~~--~~--~--~---

[chromium-dev] Re: Buildbot upgrade

[Nicolas Sylvain]

On Mon, Oct 12, 2009 at 7:58 AM, Thomas Van Lenten wrote:

>
>
> On Mon, Oct 12, 2009 at 10:45 AM, Nicolas Sylvain 
> wrote:
>
>>
>>
>> On Sun, Oct 11, 2009 at 6:50 PM, Thomas Van Lenten > > wrote:
>>
>>> Looks like the failures link needs to be updated in waterfall header.
>>
>> I'm pretty sure I did, can you show me which one is not updated?
>>
>
> http://build.chromium.org/buildbot/waterfall/console, Navigate section in
> the top left has a failures links.
>
fixed, thanks


>
> fyi - I updated the Sheriff wiki page on dev.chromium.org.
>
> TVL
>
>
>
>>
>>> TVL
>>>
>>>
>>> On Sat, Oct 10, 2009 at 9:25 PM, Nicolas Sylvain 
>>> wrote:
>>>
>>>> Hello,
>>>> Today I upgraded buildbot to the latest version.
>>>>
>>>> If you have a bookmark for the "failures only" waterfall, you will need
>>>> to change it. Previously it
>>>> was "failures=1" and it is now "show_events=true&failures_only=true"
>>>>
>>>> Other than that, nothing should have changed. If you see any issues with
>>>> the new version, please
>>>> let me know!
>>>>
>>>> Thank you,
>>>>
>>>> Nicolas
>>>>
>>>>
>>>>
>>>>
>>>>
>>>
>>
>> >>
>>
>

--~--~-~--~~~---~--~~
Chromium Developers mailing list: chromium-dev@googlegroups.com 
View archives, change email options, or unsubscribe: 
http://groups.google.com/group/chromium-dev
-~--~~~~--~~--~--~---

[chromium-dev] Re: Buildbot upgrade

[Nicolas Sylvain]

On Sun, Oct 11, 2009 at 6:50 PM, Thomas Van Lenten wrote:

> Looks like the failures link needs to be updated in waterfall header.

I'm pretty sure I did, can you show me which one is not updated?

>
> TVL
>
>
> On Sat, Oct 10, 2009 at 9:25 PM, Nicolas Sylvain wrote:
>
>> Hello,
>> Today I upgraded buildbot to the latest version.
>>
>> If you have a bookmark for the "failures only" waterfall, you will need to
>> change it. Previously it
>> was "failures=1" and it is now "show_events=true&failures_only=true"
>>
>> Other than that, nothing should have changed. If you see any issues with
>> the new version, please
>> let me know!
>>
>> Thank you,
>>
>> Nicolas
>>
>>
>>
>> >>
>>
>

--~--~-~--~~~---~--~~
Chromium Developers mailing list: chromium-dev@googlegroups.com 
View archives, change email options, or unsubscribe: 
http://groups.google.com/group/chromium-dev
-~--~~~~--~~--~--~---

[chromium-dev] Re: Buildbot upgrade

[Nicolas Sylvain]

On Sun, Oct 11, 2009 at 6:41 PM, Anthony LaForge  wrote:

> Hey man, this url doesn't seem to be resolving anymore:
>
> Ooops. I have a fix and will restart the master in a few minutes. Sorry
about that.

Nicolas


> http://chrome-buildbot.corp.google.com:8010/bot_status.json?json=213
>
> Kind Regards,
>
> Anthony Laforge
> Technical Program Manager
> Mountain View, CA
>
>
> On Sat, Oct 10, 2009 at 6:25 PM, Nicolas Sylvain wrote:
>
>> Hello,
>> Today I upgraded buildbot to the latest version.
>>
>> If you have a bookmark for the "failures only" waterfall, you will need to
>> change it. Previously it
>> was "failures=1" and it is now "show_events=true&failures_only=true"
>>
>> Other than that, nothing should have changed. If you see any issues with
>> the new version, please
>> let me know!
>>
>> Thank you,
>>
>> Nicolas
>>
>>
>>
>> >>
>>
>

--~--~-~--~~~---~--~~
Chromium Developers mailing list: chromium-dev@googlegroups.com 
View archives, change email options, or unsubscribe: 
http://groups.google.com/group/chromium-dev
-~--~~~~--~~--~--~---

[chromium-dev] Re: Buildbot upgrade

[Nicolas Sylvain]

On Sun, Oct 11, 2009 at 11:33 AM, Jeremy Orlow  wrote:

> Is there documentation anywhere for all the parameters you can feed into
> the buildbot webpage?  If not, a cheat sheet would be really helpful.
>

the help link at the bottom has most of them
http://build.chromium.org/buildbot/waterfall/waterfall/help

<http://build.chromium.org/buildbot/waterfall/waterfall/help>Otherwise the
buildbot documentation may help:
http://djmitche.github.com/buildbot/docs/0.7.11/#Buildbot-Web-Resources


<http://djmitche.github.com/buildbot/docs/0.7.11/#Buildbot-Web-Resources>
Nicolas

>
> On Sat, Oct 10, 2009 at 6:25 PM, Nicolas Sylvain wrote:
>
>> Hello,
>> Today I upgraded buildbot to the latest version.
>>
>> If you have a bookmark for the "failures only" waterfall, you will need to
>> change it. Previously it
>> was "failures=1" and it is now "show_events=true&failures_only=true"
>>
>> Other than that, nothing should have changed. If you see any issues with
>> the new version, please
>> let me know!
>>
>> Thank you,
>>
>> Nicolas
>>
>>
>>
>> >>
>>
>

--~--~-~--~~~---~--~~
Chromium Developers mailing list: chromium-dev@googlegroups.com 
View archives, change email options, or unsubscribe: 
http://groups.google.com/group/chromium-dev
-~--~~~~--~~--~--~---

[chromium-dev] Buildbot upgrade

[Nicolas Sylvain]

Hello,
Today I upgraded buildbot to the latest version.

If you have a bookmark for the "failures only" waterfall, you will need to
change it. Previously it
was "failures=1" and it is now "show_events=true&failures_only=true"

Other than that, nothing should have changed. If you see any issues with the
new version, please
let me know!

Thank you,

Nicolas

--~--~-~--~~~---~--~~
Chromium Developers mailing list: chromium-dev@googlegroups.com 
View archives, change email options, or unsubscribe: 
http://groups.google.com/group/chromium-dev
-~--~~~~--~~--~--~---

[chromium-dev] Re: How to deal with flaky unit tests

[Nicolas Sylvain]

On Tue, Oct 6, 2009 at 10:52 PM, Darin Fisher  wrote:

> If a test sometimes crashes or hangs, it'll still be disabled, right?


Yes.

But if it's a ui_tests that crashes chrome.exe (and not ui_tests.exe), we
can still mark it as flaky.

Nicolas


> -darin
>
> On Tue, Oct 6, 2009 at 5:02 PM, Nicolas Sylvain wrote:
>
>> Hello,
>> We currently have more than 50 unit tests that are disabled. Most of them
>> because they were flaky.
>>
>> Disabling tests is bad because we lose complete coverage on them, so I
>> implemented a way to mark
>> tests as "flaky".
>>
>> The same way you disable a test with DISABLED_ at the beginning of its
>> name, you can now mark
>> is as flaky with FLAKY_.  The behavior is exactly the same as any other
>> running tests. You will still
>> be able to see when it fails (and why).  The only difference is that if
>> only FLAKY_ tests failed, the
>> buildbot/trybots won't consider it as a failure. On the waterfall, it will
>> show the box as orange with the
>> list of all flaky tests that failed (pending one more buildbot restart).
>> On the console view it will stay
>> green.
>>
>> But.. this is not a toy. Flaky tests are bad. We should mark tests flaky
>> only if we really have to, and
>> if you do, please make sure to file a P1 bug. Set the owner of the bug to
>> whoever regressed the test.
>> If you can't find who regressed the test, assign it to the person who
>> originally wrote the test.
>>
>> Once we start tagging the flaky tests, we will monitor the flakiness
>> dashboard and make sure
>> that a test that is no longer flaky has its FLAKY_ tag removed.
>>
>> Let me know if you have questions.
>>
>> Thanks
>>
>> Nicolas
>>
>>
>> >>
>>
>

--~--~-~--~~~---~--~~
Chromium Developers mailing list: chromium-dev@googlegroups.com 
View archives, change email options, or unsubscribe: 
http://groups.google.com/group/chromium-dev
-~--~~~~--~~--~--~---

[chromium-dev] How to deal with flaky unit tests

[Nicolas Sylvain]

Hello,
We currently have more than 50 unit tests that are disabled. Most of them
because they were flaky.

Disabling tests is bad because we lose complete coverage on them, so I
implemented a way to mark
tests as "flaky".

The same way you disable a test with DISABLED_ at the beginning of its name,
you can now mark
is as flaky with FLAKY_.  The behavior is exactly the same as any other
running tests. You will still
be able to see when it fails (and why).  The only difference is that if only
FLAKY_ tests failed, the
buildbot/trybots won't consider it as a failure. On the waterfall, it will
show the box as orange with the
list of all flaky tests that failed (pending one more buildbot restart). On
the console view it will stay
green.

But.. this is not a toy. Flaky tests are bad. We should mark tests flaky
only if we really have to, and
if you do, please make sure to file a P1 bug. Set the owner of the bug to
whoever regressed the test.
If you can't find who regressed the test, assign it to the person who
originally wrote the test.

Once we start tagging the flaky tests, we will monitor the flakiness
dashboard and make sure
that a test that is no longer flaky has its FLAKY_ tag removed.

Let me know if you have questions.

Thanks

Nicolas

--~--~-~--~~~---~--~~
Chromium Developers mailing list: chromium-dev@googlegroups.com 
View archives, change email options, or unsubscribe: 
http://groups.google.com/group/chromium-dev
-~--~~~~--~~--~--~---

[chromium-dev] Re: gclient fails to update gyp files

[Nicolas Sylvain]

On Tue, Sep 29, 2009 at 11:37 AM, Chris Guillory  wrote:

> Dominic and I starting running into this today also. We both don't have the
> src/native_client directory. And I don't see it listed in the directory list
> at http://src.chromium.org/svn/trunk/src/.

gclient update will fetch it since it's in the DEPS file.

If you downloaded the tarball, we had the .gclient file set up to explicitly
ignore this dependency.

This should now be fixed for people who are downloading the tarball today.

Nicolas




>
> On Tue, Sep 29, 2009 at 10:53 AM, Mark Mentovai  wrote:
>
>>
>> gclient may have gotten confused in a previous run.  Try "gclient sync
>> --force" and let us know if you still have a problem with that.
>>
>> Mark
>>
>>
>>
>
> >
>

--~--~-~--~~~---~--~~
Chromium Developers mailing list: chromium-dev@googlegroups.com 
View archives, change email options, or unsubscribe: 
http://groups.google.com/group/chromium-dev
-~--~~~~--~~--~--~---

[chromium-dev] Re: gclient fails to update gyp files

[Nicolas Sylvain]

you need to edit your .gclient and remove the line that says:
"src/native_client":
None

Is native_client really required? Why? We don't want to build this by
default, do we?

it's too big and it does not fit in the tarball, so it has been excluded
there.

Nicolas



On Tue, Sep 29, 2009 at 12:40 PM, Thomas Van Lenten
wrote:

> It's not in src.chromium.org, it comes in via DEPS:
>   "src/native_client":
> "http://nativeclient.googlecode.com/svn/trunk/src/native_cli...@797";,
>
> TVL
>
>
> On Tue, Sep 29, 2009 at 3:32 PM, Nebojša Ćirić  wrote:
>
>> Hi Nick, As Chris said, src/native_client is missing from the trunk (and
>> I don't have it in my client).
>>
>> Cira
>>
>>
>> On Tue, Sep 29, 2009 at 12:28 PM, Nick Carter  wrote:
>>
>>> FWIW, on my (healthy) client, "svn info" inside of the native_client dir
>>> looks like:
>>>
>>> ncarter /cygdrive/d/src/crgit/src/native_client/src
>>> $ svn info
>>> Path: .
>>> URL: http://nativeclient.googlecode.com/svn/trunk/src/native_client/src
>>> Repository Root: http://nativeclient.googlecode.com/svn
>>> Repository UUID: fcba33aa-ac0c-11dd-b9e7-8d5594d729c2
>>> Revision: 730
>>> Node Kind: directory
>>> Schedule: normal
>>> Last Changed Author: kschi...@google.com
>>> Last Changed Rev: 728
>>> Last Changed Date: 2009-09-16 14:58:52 -0700 (Wed, 16 Sep 2009)
>>>
>>> How does that compare with your checkout?
>>>
>>>  - nick
>>>
>>> On Tue, Sep 29, 2009 at 12:23 PM, Nebojša Ćirić wrote:
>>>
  Still the same message (after gclient sync --force).


 On Tue, Sep 29, 2009 at 10:53 AM, Mark Mentovai wrote:

> gclient may have gotten confused in a previous run.  Try "gclient sync
> --force" and let us know if you still have a problem with that.
>
> Mark
>




>>>
>>
>
> >
>

--~--~-~--~~~---~--~~
Chromium Developers mailing list: chromium-dev@googlegroups.com 
View archives, change email options, or unsubscribe: 
http://groups.google.com/group/chromium-dev
-~--~~~~--~~--~--~---

[chromium-dev] Re: src.chromium.org is down

[Nicolas Sylvain]

The server has been stable enough in the last 20 minutes, so I think it is
fixed now.
If you have problems with the server today, please let me know.

Thanks

Nicolas


On Tue, Sep 29, 2009 at 7:45 AM, Nicolas Sylvain wrote:

> src.chromium.org is down for a few minutes... while we try to fix the disk
> error.
> Sorry for the inconvenience,
>
> Nicolas
>

--~--~-~--~~~---~--~~
Chromium Developers mailing list: chromium-dev@googlegroups.com 
View archives, change email options, or unsubscribe: 
http://groups.google.com/group/chromium-dev
-~--~~~~--~~--~--~---

[chromium-dev] src.chromium.org is down

[Nicolas Sylvain]

src.chromium.org is down for a few minutes... while we try to fix the disk
error.
Sorry for the inconvenience,

Nicolas

--~--~-~--~~~---~--~~
Chromium Developers mailing list: chromium-dev@googlegroups.com 
View archives, change email options, or unsubscribe: 
http://groups.google.com/group/chromium-dev
-~--~~~~--~~--~--~---

[chromium-dev] Re: Webkit merges and tree closures

[Nicolas Sylvain]

On Tue, Sep 22, 2009 at 10:47 PM, David Levin  wrote:

> Actionable items for keeping the tree green (in addition to blaming the
> WebKit gardener for [insert action here]):
>
>- *Get people putting in chromium patches upstream to run their changes
>through trybots, etc*. imo, patches from @chromium folks cause well
>over 50% of the grief with WebKit rolls (and usually the worst issues like
>today).
>- As Adam suggested, *make changes to the WebKit commit queue to run
>items through the chromium try bots*.
>- *WebKit gardeners should be able to submit high priority jobs to the
>try bots*. These jobs should job to the front of any queues so that
>they run asap.
>Why? Time is critical when doing the gardening because if you find a
>breakage upstream, you need to check in a fix and then roll to that fix. 
> The
>longer the delay, the more likely another breakage will be checked in.
>
> I haven't seen a queue on the try bots in weeks now. If you do see one, and
your change is not tested on all platforms as soon as you upload it, please
let me know and we'll allocate more resources.

Nicolas

>
>- *Consider auto-rolling WebKit deps*. Have the something like a
>parallel buildbot that runs against tip of tree WebKit. If everything 
> passes
>except layout tests, add any layout test failures to test_expectations.txt
>(if there are less than 15) and roll DEPS on passing. If things fail, then
>turn red.
>- *Make it easier/faster to disable tests and file bugs about them *(using
>the last person in the "blame/annotate" for the test as the initial 
> assignee
>or auto-assign it to the sheriff so (s)he can assign it to the right 
> person)
>* *because issues will slip from WebKit rolls even though the gardeners
>try to be thorough. Also, this should help with turning the tree green in
>other cases as well.
>
> The sheriff (and everyone on the chromium team) should care about the
> WebKit roll as this is critical to the success of this project. Frequent
> rolls, should isolate issues and hopefully help to keep the tree green.
> *
> *
> To help shed some light on why WebKit gardening is more painful than
> sheriffing:
>
>1. WebKit gardeners are all alone in trying to deal with things.
>2. When things go bad on the canary, no one shuts down the tree for you
>and any changes to help with merging are not given priority (if the tree is
>red and you have an innocuous change to fix the webkit merge, you won't get
>it in.)
>3. When tests fail anywhere (on the canary, when committing the roll,
>etc.), you have to figure out why, typically for a lot of changes that you
>know nothing about.
>4. Two days of gardening -- multiple days of clean up afterward.
>5. WebKit gardening occurs more often than sheriff duties.
>6. afaik all WebKit gardeners also have sheriff duties.
>
> Net: chromium sheriffs please be willing to give a little extra help to the
> WebKit gardener. Remember that their hair is turning white as they try to
> run in front of a locomotive.
>
> Dave
>
>

--~--~-~--~~~---~--~~
Chromium Developers mailing list: chromium-dev@googlegroups.com 
View archives, change email options, or unsubscribe: 
http://groups.google.com/group/chromium-dev
-~--~~~~--~~--~--~---

[chromium-dev] Re: WebKit Hygiene: Please be kind. Test your patches. Warn about two-sided patches.

[Nicolas Sylvain]

On Wed, Sep 23, 2009 at 8:55 AM, Paweł Hajdan Jr.
wrote:

> On Tue, Sep 22, 2009 at 18:06, Dimitri Glazkov wrote:
>
>> Today wasn't a happy day for p...@. He did a seemingly innocuous roll
>> that broke the world: selenium, ui tests, layout tests. I am sure it
>> was stressful and probably added unnecessary gray to his hair.
>
>
> How about running ui and selenium tests on the canary bot?
>
We run selenium already. And UI is covered by try server,
but I'll make the change to also run the ui tests on the canary.

Nicolas


> >
>

--~--~-~--~~~---~--~~
Chromium Developers mailing list: chromium-dev@googlegroups.com 
View archives, change email options, or unsubscribe: 
http://groups.google.com/group/chromium-dev
-~--~~~~--~~--~--~---

[chromium-dev] Re: WebKit Hygiene: Please be kind. Test your patches. Warn about two-sided patches.

[Nicolas Sylvain]

On Tue, Sep 22, 2009 at 6:35 PM, Adam Barth  wrote:

>
> On Tue, Sep 22, 2009 at 6:25 PM, John Abd-El-Malek 
> wrote:
> > Is this even possible?  i.e. I had uploaded a WebKit patch on codereview
> but
> > none of the patchsets got run on the try server
> > http://codereview.chromium.org/178030/show
>
> It is possible:
>
> aba...@zenque:~/svn/kr/src/third_party/WebKit$ gcl try scriptcontext
> --use_svn --svn_repo=svn://svn.chromium.org/chrome-try/try --bot
> layout_win,layout_mac,layout_linux --root src/third_party
>

And if you use git, there was another thread today about this on
chromium-dev :
http://groups.google.com/group/chromium-dev/browse_thread/thread/c481ecc1842ddfe4#


Nicolas


>
> >
>

--~--~-~--~~~---~--~~
Chromium Developers mailing list: chromium-dev@googlegroups.com 
View archives, change email options, or unsubscribe: 
http://groups.google.com/group/chromium-dev
-~--~~~~--~~--~--~---

[chromium-dev] Re: Webkit merges and tree closures

[Nicolas Sylvain]

On Tue, Sep 22, 2009 at 6:10 PM, Jeremy Orlow  wrote:

> There are 2 major issues here (besides leaving things for the Sheriff to
> clean up):
> 1) a lot of the gardeners are inexperienced and drop the ball.  This has
> bitten us many times.  The last time we had a big string of problems related
> to this, I meant to send out an email giving people advice on what to expect
> and how to prepare.  I will do it this time.  Hopefully people listen.
>
> 2) we don't have enough tools for gardeners to do their jobs.  As I
> mentioned in another thread you started the other day, we really need more
> try bots and/or canaries that run memory tests, our full suite of tests,
> etc.  Without that, things are not going to get better.
>
> Just to be clear, on bad days, gardening is WAY harder than sheriffing by
> yourself.  Like you mentioned, reverting a merge is pretty much not an
> option because you only get further behind, which makes things harder.  If
> several hour tree closures once or twice a week are not an option, then we
> need more bots.
>
We are creating bots as fast as we can. We delivered almost (all?) of them
you asked for. You have purify canary, valgrind canary, perf tests canary,
and selenium tests canary.  You also have the layout tests try slaves, and
the valgrind try slave. I guess the one that remains in ui tests? We can
work on that and make it happen. (which you have a try server for in the
mean time).

Nicolas


> J
>
> On Tue, Sep 22, 2009 at 5:50 PM, Paweł Hajdan Jr.  > wrote:
>
>> On Tue, Sep 22, 2009 at 17:40, Nicolas Sylvain wrote:
>>
>>> 3 PM : two failing ui tests are disabled by the webkit sheriff
>>>
>>
>> I was looking at the UI tests and it wasn't immediately obvious that a
>> webkit update might break them. Can we run all the UI tests on the webkit
>> canary bot? If that takes too long, we can selectively exclude the slow
>> tests, based on http://build.chromium.org/buildbot/slowness-report/
>>
>> By the way, the layout test failures... can we catch them on the webkit
>> canary bot(s)?
>>
>> >>
>>
>

--~--~-~--~~~---~--~~
Chromium Developers mailing list: chromium-dev@googlegroups.com 
View archives, change email options, or unsubscribe: 
http://groups.google.com/group/chromium-dev
-~--~~~~--~~--~--~---

[chromium-dev] Re: Webkit merges and tree closures

[Nicolas Sylvain]

On Tue, Sep 22, 2009 at 6:08 PM, Adam Barth  wrote:

> On Tue, Sep 22, 2009 at 5:40 PM, Nicolas Sylvain 
> wrote:
> > If this is an issue, I am proposing that Webkit merges be done outside
> peak
> > hours (11am-5pm pacific).
>
> This seems backwards.  Don't we want to integrate more often so we can
> catch and fix these issue faster?  Ideally, we'll be able to merge
> with every webkit.org commit once we get the WebKit API in place.
>

I really want to stress the first part of my sentence: "if this is an issue"
.

For small webkit merges, I really don't expect this to be an issue.

Once the API is in place, I'm pretty sure we won't have this problem. Or at
least not as often.

Nicolas


> Adam
>

--~--~-~--~~~---~--~~
Chromium Developers mailing list: chromium-dev@googlegroups.com 
View archives, change email options, or unsubscribe: 
http://groups.google.com/group/chromium-dev
-~--~~~~--~~--~--~---

[chromium-dev] Webkit merges and tree closures

[Nicolas Sylvain]

Hi,
In the last few weeks I've been trying to be aware as much as possible about
the reasons we close the tree, and
my gut feeling seems to match what I'm seeing: Webkit merges is the main
cause.

Now, I understand that Webkit merges are not easy, and really, kudos to the
team for keeping up with this and
doing this constant work, but I think there are some ways we can make it a
little bit more seamless for the rest
of us.

Today is a good example, the tree was red for 3 hours, from 2:30PM pacific
to 5:30PM. This also coincide with
the peak hours where chromium developers work and need the tree to be open.
 And as I said, today is just
an example, and I don't want to sound rude to whoever did the merge, this is
not his fault.

Overview of the day:

2 PM : webkit merge committed
2:30 : a bunch of bots turn red
3 PM : two failing ui tests are disabled by the webkit sheriff
3:41 : 21 layout tests are disabled by the webkit sheriff
4 PM : 2 new valgrind errors are disabled by the sheriff (not webkit
sheriff)
4:36 : 3 new purify UMR are disable by the sheriff (not webkit sheriff)
4:38: 1 more layout test is disabled by the sheriff (not webkit sheriff)
5:04: 3 new selenium tests are disabled by the sheriff (not webkit sheriff)
5:30: Tree is open


There has to be a better way to handle this.

Bad webkit rolls happen. There is not much we can't do, but we can try to
limit the damage.

There are some tools we developed to help make your job easier please
use them.

1. Try servers. (this should have caught the ui tests failures)
2. On demand try server for the layout tests (this should have caught a
bunch of other layout tests failures)

And, as with other changes, the current rules apply:

Monitor the tree after your change. You don't have anything better to do
than to make it green. Seriously.

If there are failures, add them to the expectation list right away (or
revert, but with webkit merges this is not
really an option).  When the tree is red, this is not a good time to start
investigate! You can do that later.

If you did cause the tree to be red, please stick around until it's green.
Maybe you fix, or test
disabling will uncover yet another failing test. You might also check back
again ~30 minutes to an
hour after to make sure some flakiness was not introduced.

purify and valgrind redness count. Treat them the same way as the other
tests.

Don't expect the sheriff to hold your hand.  In the timeline above, in the
last hour the sheriff had to clean up
after the webkit merge. This is not his job. (But he has to do it when you
don't!)

In the green tree task force we have an OKR to enforce that people take care
of their failures within
10 minutes after the tree turns red. Webkit merges are not going to be an
exception.

If this is an issue, I am proposing that Webkit merges be done outside peak
hours (11am-5pm pacific).

Oh, and, if your change turned the tree red for 3 hours, don't be mad at the
sheriff when he pings
you repeatedly about the status of the fix.  His job is to keep the tree
green and running. He does
not care about your change.

Thoughts?

Nicolas

--~--~-~--~~~---~--~~
Chromium Developers mailing list: chromium-dev@googlegroups.com 
View archives, change email options, or unsubscribe: 
http://groups.google.com/group/chromium-dev
-~--~~~~--~~--~--~---

[chromium-dev] Re: debugging the browser started by UI tests

[Nicolas Sylvain]

On windows just use windbg, and tell it to attach to child processes.
I can show you if you want.

Nicolas


On Tue, Sep 22, 2009 at 4:10 PM, Scott Violet  wrote:

>
> WAIT_FOR_DEBUGGER_ON_OPEN predates the Linux port. It may work on
> Linux, I just haven't tried it.
>
>  -Scott
>
> On Tue, Sep 22, 2009 at 4:06 PM, Evan Martin  wrote:
> > Both of these should go to the "ui tests" section of the debugging
> > wiki, which is where I turned in attempting to answer Pawel's
> > question:
> > http://code.google.com/p/chromium/wiki/LinuxDebugging#UI_tests
> >
> > On Tue, Sep 22, 2009 at 4:03 PM, Scott Violet  wrote:
> >>
> >> If you uncomment WAIT_FOR_DEBUGGER_ON_OPEN on ui_test you'll be
> >> prompted. We really need to convert this into a comment line option.
> >>
> >>  -Scott
> >>
> >> On Tue, Sep 22, 2009 at 3:00 PM, Paweł Hajdan Jr.
> >>  wrote:
> >>> What's the best way to attach the debugger to a browser started by a UI
> >>> test? How about doing that only in case of a crash?
> >>> I'm looking for solution both for Windows and Linux, so if you have
> good
> >>> techniques, it'd be really nice. I can even document them on the wiki,
> but
> >>> currently I'm using LOG statements when debugging the browser (I know
> it's
> >>> not the optimal and kind of sucks, but I couldn't find a good way to
> attach
> >>> the debugger).
> >>> >
> >>>
> >>
> >> >>
> >>
> >
>
> >
>

--~--~-~--~~~---~--~~
Chromium Developers mailing list: chromium-dev@googlegroups.com 
View archives, change email options, or unsubscribe: 
http://groups.google.com/group/chromium-dev
-~--~~~~--~~--~--~---

[chromium-dev] Re: [Green Tree] Week 1

[Nicolas Sylvain]

On Mon, Sep 21, 2009 at 10:53 AM, Antony Sargent wrote:

> Do we have anything running which monitors disk free space? It seems like
> in a couple of cases over the last few months getting email alerts when a
> bot's disk is 90% full might have helped alert Sheriffs/Troopers to a
> problem earlier and possibly prevent a tree closure.


At this point the problem is that the build got bigger, and we can't fit 2
checkouts at the same time on the same machine. We are currently slowly
replacing all the old 30GB VM with new one that has 70GB.

Eventually we should try to implement some alert mechanism.

Nicolas


>
> On Mon, Sep 21, 2009 at 10:31 AM, Jeremy Orlow wrote:
>
>> On Mon, Sep 21, 2009 at 8:25 AM, Nicolas Sylvain 
>> wrote:
>>
>>> Hi chromium-dev,
>>>   A small group of us joined forces to create a "Green Tree" task force.
>>> The goal of this task
>>> force is to make sure the tree stays green most of the time.  The 2 main
>>> pain points that
>>> we are attacking at this time are "reducing the buildbot cycle time", to
>>> catch errors earlier, and
>>> "getting rid of the flakiness", to make sure the tree does not turn red
>>> for no reason.
>>>
>>>   I'll be prepending "[Green Tree]" to the emails I send related to the
>>> task force.
>>>
>>>   You can also follow the progress and our tasks there:
>>> http://code.google.com/p/chromium/issues/list?q=label:GreenTreeTaskForce
>>>
>>> For those interested, these are the highlights of the last week:
>>>
>>> - Make sure all the tasks have bugs associated with them (pamg)
>>> - Make sure VMWare Tools is installed on all the slaves (bev / nsylvain)
>>> - Disable all services that we don't need on the slaves (bev)
>>> - Split the windows chromium tests in 3 slaves (maruel)
>>> - Change the gatekeeper to close the tree on more failures (maruel)
>>>  - Change LKGR to care about more tests, and make it cycle faster
>>> (maruel)
>>> - Write a status page to see the cycle speed on the slaves (nsylvain)
>>> - Make sure we build only what we need on Mac (thomasvl)
>>> - Add more try bots (linux views, valgrind) (maruel)
>>> - Refactor Linux Valgrind buildbots into builder/testers. (mmoss)
>>> - Create a dashboard to see the slowest tests (phajdan)
>>> - Speed up the transfer of builds between builders/testers by reducing
>>> the compression (mmoss)
>>>
>>>   I'm sure I forgot some, feel free to append to this list.
>>>
>>>   Despite our efforts, this was one of the worse week we've seen in a
>>> long time in term of tree closure. This
>>> was caused by 5 main events:
>>>
>>>  - Buildbot maintenance went wrong. By changing a mounted drive on the
>>> buildbot master, the mount table got corrupted, and we had to reboot the
>>> main server. We started the maintenance at 7:30AM (pacific) and we got the
>>> buildbot back online shortly after 10AM. It had to cycle a little, so it was
>>> closed for almost 3 hours
>>>  - A webkit merge left some failures in the tree. And it looks like
>>> everyone left without fixing it, so it was closed overnight. We fixed it in
>>> the morning, but before reopening we let another webkit merge go by, and it
>>> also broke the tree, requiring a change on webkit.org to fix the
>>> reliability tests (IIRC). Total closure time: 20 hours.
>>>
>>
>> The more try bots we get, the better this will get.  At the very least,
>> when we check in something that upsets bots covered by try bots, we can
>> always roll back out and triage without the tree closed.  Maybe we should
>> have one try bot for each different type of build bot?
>>
>> Btw, in case anyone is wondering what makes WebKit gardening special:
>> WebKit is a freight train that we can't stop.  And so, if we get behind by
>> even a day, it has a serious impact on Chromium developers' ability to do a
>> lot of stuff (especially 2 sided patches).  I'm not trying to condone the 20
>> hour closure (I don't know the details), but if we can't figure out what's
>> wrong quickly (when it breaks our stuff) we can get into a pretty bad
>> situation pretty quickly.
>>
>>
>>>  - A bad gclient change got checked in. Some machines stopped running
>>> "runhooks" and some bots got confused. The damage seems to have been
>>> limited.
>>>  - A second bad gclient change got checked in. This time

[chromium-dev] Re: [Green Tree] Week 1

[Nicolas Sylvain]

On Mon, Sep 21, 2009 at 6:24 PM, Lei Zhang  wrote:

> Of the three slowest tests as of last week, two has been fixed (thanks
> to evan and jcampan) and the third one has a fix being reviewed.
>
> The Linux trybots build almost as fast as the Mac trybots now, thanks
> to shared ccache with a ~80% cache hit rate. Windows trybots are still
> struggling - taking 3X as long to build vs. Linux/Mac.
>
We finally figured out how to give 4 cpus to the windows try bots instead of
2. This
should add a little bit of speed up! Bev is working on this.

Nicolas


>
> On Mon, Sep 21, 2009 at 8:25 AM, Nicolas Sylvain 
> wrote:
> > Hi chromium-dev,
> >   A small group of us joined forces to create a "Green Tree" task force.
> The
> > goal of this task
> > force is to make sure the tree stays green most of the time.  The 2 main
> > pain points that
> > we are attacking at this time are "reducing the buildbot cycle time", to
> > catch errors earlier, and
> > "getting rid of the flakiness", to make sure the tree does not turn red
> for
> > no reason.
> >   I'll be prepending "[Green Tree]" to the emails I send related to the
> task
> > force.
> >   You can also follow the progress and our tasks
> > there:
> http://code.google.com/p/chromium/issues/list?q=label:GreenTreeTaskForce
> > For those interested, these are the highlights of the last week:
> > - Make sure all the tasks have bugs associated with them (pamg)
> > - Make sure VMWare Tools is installed on all the slaves (bev / nsylvain)
> > - Disable all services that we don't need on the slaves (bev)
> > - Split the windows chromium tests in 3 slaves (maruel)
> > - Change the gatekeeper to close the tree on more failures (maruel)
> > - Change LKGR to care about more tests, and make it cycle faster (maruel)
> > - Write a status page to see the cycle speed on the slaves (nsylvain)
> > - Make sure we build only what we need on Mac (thomasvl)
> > - Add more try bots (linux views, valgrind) (maruel)
> > - Refactor Linux Valgrind buildbots into builder/testers. (mmoss)
> > - Create a dashboard to see the slowest tests (phajdan)
> > - Speed up the transfer of builds between builders/testers by reducing
> the
> > compression (mmoss)
> >   I'm sure I forgot some, feel free to append to this list.
> >   Despite our efforts, this was one of the worse week we've seen in a
> long
> > time in term of tree closure. This
> > was caused by 5 main events:
> >  - Buildbot maintenance went wrong. By changing a mounted drive on the
> > buildbot master, the mount table got corrupted, and we had to reboot the
> > main server. We started the maintenance at 7:30AM (pacific) and we got
> the
> > buildbot back online shortly after 10AM. It had to cycle a little, so it
> was
> > closed for almost 3 hours
> >  - A webkit merge left some failures in the tree. And it looks like
> everyone
> > left without fixing it, so it was closed overnight. We fixed it in the
> > morning, but before reopening we let another webkit merge go by, and it
> also
> > broke the tree, requiring a change on webkit.org to fix the reliability
> > tests (IIRC). Total closure time: 20 hours.
> >  - A bad gclient change got checked in. Some machines stopped running
> > "runhooks" and some bots got confused. The damage seems to have been
> > limited.
> >  - A second bad gclient change got checked in. This time causing all the
> > bots to throw away their checkouts. Almost each slaves had to do a full
> > checkout (which takes an hour or so), and some of them ran out of disk
> > space, so we had to manually fix them. The tree was closed for another
> > couple of hours.
> >  - A bad DEPS file got checked in. Causing again a bunch of slaves to
> throw
> > away their checkout. It was closed for another hour or two.
> > Nicolas
> > > >
> >
>

--~--~-~--~~~---~--~~
Chromium Developers mailing list: chromium-dev@googlegroups.com 
View archives, change email options, or unsubscribe: 
http://groups.google.com/group/chromium-dev
-~--~~~~--~~--~--~---

[chromium-dev] [Green Tree] Week 1

[Nicolas Sylvain]

Hi chromium-dev,
  A small group of us joined forces to create a "Green Tree" task force. The
goal of this task
force is to make sure the tree stays green most of the time.  The 2 main
pain points that
we are attacking at this time are "reducing the buildbot cycle time", to
catch errors earlier, and
"getting rid of the flakiness", to make sure the tree does not turn red for
no reason.

  I'll be prepending "[Green Tree]" to the emails I send related to the task
force.

  You can also follow the progress and our tasks there:
http://code.google.com/p/chromium/issues/list?q=label:GreenTreeTaskForce

For those interested, these are the highlights of the last week:

- Make sure all the tasks have bugs associated with them (pamg)
- Make sure VMWare Tools is installed on all the slaves (bev / nsylvain)
- Disable all services that we don't need on the slaves (bev)
- Split the windows chromium tests in 3 slaves (maruel)
- Change the gatekeeper to close the tree on more failures (maruel)
 - Change LKGR to care about more tests, and make it cycle faster (maruel)
- Write a status page to see the cycle speed on the slaves (nsylvain)
- Make sure we build only what we need on Mac (thomasvl)
- Add more try bots (linux views, valgrind) (maruel)
- Refactor Linux Valgrind buildbots into builder/testers. (mmoss)
- Create a dashboard to see the slowest tests (phajdan)
- Speed up the transfer of builds between builders/testers by reducing the
compression (mmoss)

  I'm sure I forgot some, feel free to append to this list.

  Despite our efforts, this was one of the worse week we've seen in a long
time in term of tree closure. This
was caused by 5 main events:

 - Buildbot maintenance went wrong. By changing a mounted drive on the
buildbot master, the mount table got corrupted, and we had to reboot the
main server. We started the maintenance at 7:30AM (pacific) and we got the
buildbot back online shortly after 10AM. It had to cycle a little, so it was
closed for almost 3 hours
 - A webkit merge left some failures in the tree. And it looks like everyone
left without fixing it, so it was closed overnight. We fixed it in the
morning, but before reopening we let another webkit merge go by, and it also
broke the tree, requiring a change on webkit.org to fix the reliability
tests (IIRC). Total closure time: 20 hours.
 - A bad gclient change got checked in. Some machines stopped running
"runhooks" and some bots got confused. The damage seems to have been
limited.
 - A second bad gclient change got checked in. This time causing all the
bots to throw away their checkouts. Almost each slaves had to do a full
checkout (which takes an hour or so), and some of them ran out of disk
space, so we had to manually fix them. The tree was closed for another
couple of hours.
 - A bad DEPS file got checked in. Causing again a bunch of slaves to throw
away their checkout. It was closed for another hour or two.

Nicolas

--~--~-~--~~~---~--~~
Chromium Developers mailing list: chromium-dev@googlegroups.com 
View archives, change email options, or unsubscribe: 
http://groups.google.com/group/chromium-dev
-~--~~~~--~~--~--~---

[chromium-dev] Re: Is it possible to run Chromium in a Windows Job?

[Nicolas Sylvain]

On Thu, Sep 17, 2009 at 6:25 PM, Marc-Antoine Ruel wrote:

>
> There are issues with the sandbox because it is creating the renderer
> and plugin processes in a job object.
>
> I recall runas has issues too because of the CREATE_BREAKAWAY_FROM_JOB
> flag is not working.
>
This is right.

When you create your job you need to set the basic limit flag
JOB_OBJECT_LIMIT_BREAKAWAY_OK.

The renderer processes won't be in your job (since it's not possible to be
in 2 jobs at the time, and the sandbox needs to control its own job), but
the browser process and the plugin processes will be in your job.

If it does not work, feel free to post what kind of restrictions you are
using on your job, and we'll take a look.

Nicolas


>
> You may want to try to disable the job object functionality of the
> sandbox but I'm not sure this is what you want either.
>
> M-A
>
> On Thu, Sep 17, 2009 at 9:11 PM, Daniel Cowx 
> wrote:
> >
> > To be clear, I don't want to run within a job **only** so I can be
> > notified of exit (obviously I could do this with a handle), so please
> > don't suggest that I do that instead . What I am looking for is
> > a solution to how I can run not only the sandboxed target processes,
> > but also the main broker process within a job.
> >
> > Thanks,
> > Daniel
> >
> > On Sep 17, 5:58 pm, Daniel Cowx  wrote:
> >> I'd like to run chromium (the broker) within a Windows job so that I
> >> can be notified when it exits. Does anyone know if this is possible?
> >> My preliminary testing (with a job that imposes no limits whatsoever)
> >> is causing problems unless I use the "no-sandbox" or "single-process"
> >> flags; which is not what I want to do. Thoughts?
> > >
> >
>
> >
>

--~--~-~--~~~---~--~~
Chromium Developers mailing list: chromium-dev@googlegroups.com 
View archives, change email options, or unsubscribe: 
http://groups.google.com/group/chromium-dev
-~--~~~~--~~--~--~---

[chromium-dev] Re: Missing symbols for Chrome 2.0.172.43

[Nicolas Sylvain]

Symbols are being copied.
If it still does not work for you, let us know.

Thanks

Nicolas

On Mon, Sep 14, 2009 at 10:00 AM, Huan Ren  wrote:

> On Sat, Sep 12, 2009 at 5:49 PM, yuhong  wrote:
> >
> > Yes, notice the symbol path that begins with
> http://build.chromium.org/buildbot/symsrv/.
> >
> > On Sep 12, 4:00 pm, Amit Joshi  wrote:
> >> Are you sure you have added chromium debug symbol server to the symbol
> path?
> >> At least from the following output, I don't see an attempt to connect
> >> to it.More
> >> information at:
> http://dev.chromium.org/developers/how-tos/debugging#TOC-Debugging-wi...
> >> <
> http://dev.chromium.org/developers/how-tos/debugging#TOC-Debugging-wi...>
> >>
> >>
> >>
> >> On Sat, Sep 12, 2009 at 1:56 PM, yuhong 
> wrote:
> >>
> >> > SYMCHK output:
> >> > DBGHELP: c:\windows\symbols\chrome_dll.pdb - file not found
> >> > DBGHELP: c:\windows\symbols\dll\chrome_dll.pdb - file not found
> >> > DBGHELP: c:\windows\symbols\symbols\dll\chrome_dll.pdb - file not
> >> > found
> >> > SYMSRV:  c:\downloads\symbols\chrome_dll.pdb
> >> > \B9CAB1AA626A4C32BD2E681FDD12495D2\chrome_dll.pdb not found
> >> > SYMSRV:
> >> >http://msdl.microsoft.com/download/symbols/chrome_dll.pdb/B9CAB1AA626.
> ..
> >> > not found
> >> > SYMSRV:  c:\downloads\symbols\chrome_dll.pdb
> >> > \B9CAB1AA626A4C32BD2E681FDD12495D2\chrome_dll.pdb not found
> >> > SYMSRV:
> >> >http://build.chromium.org/buildbot/symsrv/chrome_dll.pdb/B9CAB1AA626A.
> ..
> >> > not found
> >> > SYMSRV:  c:\downloads\symbols\chrome_dll.pdb
> >> > \B9CAB1AA626A4C32BD2E681FDD12495D2\chrome_dll.pdb not found
> >> > SYMSRV:
> >> >http://symbols.mozilla.org/firefox/chrome_dll.pdb/B9CAB1AA626A4C32BD2.
> ..
> >> > not found
> >> > DBGHELP: C:\Users\bob\AppData\Local\Google\Chrome\Application
> >> > \2.0.172.43\chrome_dll.pdb - file not found
> >> > DBGHELP: C:\b\slave\chrome-official-2\build\src\chrome\Release
> >> > \chrome_dll.pdb - file not found
> >> > *** ERROR: Symbol file could not be found.  Defaulted to export
> >> > symbols for C:\Users\bob\AppData\Local\Google\Chrome\Application
> >> > \2.0.172.43\chrome.dll -
> >> > DBGHELP: chrome_5c34 - export symbols
> >> > DBGHELP: c:\windows\symbols\chrome_exe.pdb - file not found
> >> > DBGHELP: c:\windows\symbols\exe\chrome_exe.pdb - file not found
> >> > DBGHELP: c:\windows\symbols\symbols\exe\chrome_exe.pdb - file not
> >> > found
> >> > SYMSRV:  c:\downloads\symbols\chrome_exe.pdb
> >> > \BA04199F772D437C8BE4076986127B4E2\chrome_exe.pdb not found
> >> > SYMSRV:
> >> >http://msdl.microsoft.com/download/symbols/chrome_exe.pdb/BA04199F772.
> ..
> >> > not found
> >> > SYMSRV:  c:\downloads\symbols\chrome_exe.pdb
> >> > \BA04199F772D437C8BE4076986127B4E2\chrome_exe.pdb not found
> >> > SYMSRV:
> >> >http://build.chromium.org/buildbot/symsrv/chrome_exe.pdb/BA04199F772D.
> ..
> >> > not found
> >> > SYMSRV:  c:\downloads\symbols\chrome_exe.pdb
> >> > \BA04199F772D437C8BE4076986127B4E2\chrome_exe.pdb not found
> >> > SYMSRV:
> >> >http://symbols.mozilla.org/firefox/chrome_exe.pdb/BA04199F772D437C8BE.
> ..
> >> > not found
> >> > DBGHELP: C:\Users\bob\AppData\Local\Google\Chrome\Application
> >> > \chrome_exe.pdb - file not found
> >> > DBGHELP: C:\b\slave\chrome-official-2\build\src\chrome\Release
> >> > \chrome_exe.pdb - file not found
> >> > *** ERROR: Symbol file could not be found.  Defaulted to export
> >> > symbols for C:\Users\bob\AppData\Local\Google\Chrome\Application
> >> > \chrome.exe -
> >> > DBGHELP: chrome - export symbols
> > > >
> >
>

--~--~-~--~~~---~--~~
Chromium Developers mailing list: chromium-dev@googlegroups.com 
View archives, change email options, or unsubscribe: 
http://groups.google.com/group/chromium-dev
-~--~~~~--~~--~--~---

[chromium-dev] Re: Argh: Rebuilding installer_util on every F5

[Nicolas Sylvain]

+maruel, who changed this code yesterday
Nicolas

On Wed, Sep 9, 2009 at 5:29 PM, Peter Kasting  wrote:

> Somehow, starting today, I'm now rebuilding installer_util (recompiling a
> half dozen files, which then causes me to relink chrome.dll) every time I
> hit F5.  Deleting chrome.{ncb,suo} and rm -rfing Debug/ and Release/ do not
> help.
> Did someone goof with these recently?  Did the ICU change somehow touch
> this?  It makes my "retest" cycle a lot slower :(
>
> PK
>
> >
>

--~--~-~--~~~---~--~~
Chromium Developers mailing list: chromium-dev@googlegroups.com 
View archives, change email options, or unsubscribe: 
http://groups.google.com/group/chromium-dev
-~--~~~~--~~--~--~---

[chromium-dev] Re: linux startup regression

[Nicolas Sylvain]

[from my @chromium.org this time]

In theory this is not required, because a more recent (or older)
startup_test binary should be able to drive
any chrome binary. I think it's fair to assume that the regression is not in
startup_test directly, but in
chrome.

On Thu, Sep 3, 2009 at 9:50 AM, Michael Moss  wrote:

> Should we be archiving perf tests so we can backtrack these
> regressions more easily? For instance, Windows archives some tests
> (e.g.
> http://build.chromium.org/buildbot/continuous/LATEST/chrome-win32.test/),
> though startup isn't one of them.
>
> Michael
>
> On Thu, Sep 3, 2009 at 9:47 AM, Michael Moss wrote:
> > The bot that builds and runs these tests is a dedicated 32-bit machine.
> >
> > On Thu, Sep 3, 2009 at 9:38 AM, Evan Martin wrote:
> >>
> >> On Thu, Sep 3, 2009 at 9:33 AM, Brett Wilson
> wrote:
> >>> On Thu, Sep 3, 2009 at 9:26 AM, Evan Martin wrote:
>  Ok, I'll mark that I own this here and unassign it if I can't figure
> it out:
>  http://code.google.com/p/chromium/issues/detail?id=20991
> 
>  On Thu, Sep 3, 2009 at 2:41 AM, Dean McNamee
> wrote:
> > Ug, I can't reproduce this on my desktop. If I had to take a guess, I
> > would guess Brett's gfx::Font change.
> >>>
> >>> It could be my font change. It does measure some extra text on
> >>> startup. But I would have thought this was nothing compared to the
> >>> text we already measure.
> >>
> >> Random idea: did someone switch our builders to build 64-bit binaries
> now?
> >>
> >> > >>
> >>
> >
>

--~--~-~--~~~---~--~~
Chromium Developers mailing list: chromium-dev@googlegroups.com 
View archives, change email options, or unsubscribe: 
http://groups.google.com/group/chromium-dev
-~--~~~~--~~--~--~---

[chromium-dev] Re: The webkit canary bots, now with performance tests!

[Nicolas Sylvain]

And now it is also running test_shell_tests in purify and valgrind:
http://build.chromium.org/buildbot/waterfall.fyi/waterfall?branch=&builder=Webkit+(purify+webkit.org)&builder=Webkit+Linux+(valgrind+webkit.org)


Nicolas


On Wed, Aug 26, 2009 at 11:53 PM, Adam Barth  wrote:

>
> Yay!  This is fantastic!
>
> Adam
>
>
> On Wed, Aug 26, 2009 at 11:47 PM, Darin Fisher wrote:
> > Thanks to bevc and nsylvain, we now have performance bots testing
> chromium
> > with the very latest webkit code:
> >
> http://build.chromium.org/buildbot/waterfall.fyi/waterfall?branch=&builder=XP+Perf+(webkit.org)&builder=Linux+Perf+(webkit.org)&reload=none
> > (no mac bot yet)
> > The dashboard is here:
> > http://build.chromium.org/buildbot/perf/dashboard/overview-canary.html
> > Now, we can see with much finer granularity the impact of webkit changes
> on
> > the chromium performance tests!  No more guessing as to what will happen
> to
> > performance when we roll deps ;-)  Marvelous!
> > -Darin
> > >
> >
>
> >
>

--~--~-~--~~~---~--~~
Chromium Developers mailing list: chromium-dev@googlegroups.com 
View archives, change email options, or unsubscribe: 
http://groups.google.com/group/chromium-dev
-~--~~~~--~~--~--~---

[chromium-dev] Re: Problem with try servers?

[Nicolas Sylvain]

On Thu, Aug 27, 2009 at 3:17 AM, Ben Laurie  wrote:

>
> If I do a gcl try, I get prompted for a password for 'ben', which is
> my login name on my FreeBSD machine. Whatever I enter, the script then
> hangs - investigation shows that this is because it is prompting for a
> username, but gcl is swallowing the output.
>
> If I then enter b...@chromium.org, I get prompted for a password
> again. At that point, neither my googlecode password nor my gaia
> password work.
>
> Am I doing something wrong? How is this supposed to work?


I talked to benl offline, and he actually does not have a svn password yet.

He will get one once he gets committer access.

Nicolas


>
>
> >
>

--~--~-~--~~~---~--~~
Chromium Developers mailing list: chromium-dev@googlegroups.com 
View archives, change email options, or unsubscribe: 
http://groups.google.com/group/chromium-dev
-~--~~~~--~~--~--~---

[chromium-dev] Splitting up chrome.dll

[Nicolas Sylvain]

Hello,
In the past a lot of you have asked to split up chrome.dll in multiple DLLs.
 For this quarter, I had a goal to see how feasible this is.

Background information:

Breaking up chrome.dll would make linking time faster and use less memory.
It would also enforce a cleaner cut between our modules. Eventually
it might make some of the modules be more easily reusable or swappable.

This is likely not suitable for release or official builds, because adding
more dlls means slower startup time. And the penalty we would get from
loading 2 or 3 more dlls won't be acceptable.

This is why the scope of this change is Debug mode only.  (Actually, it
would be possible to enable this with a gyp define for release too, but it
would
be enabled by default only in debug). Oh, and this is Windows only for now
too. (Linux already supports that AFAIK)

The results:

The good news is that it is feasible. The bad news is that the cost
associated with it might be too high.
This is what we need to decide here.

I started by looking at base and net. Right now I have a checkout that can
build the full solution with a base.dll and a net.dll. (205 modified files)

The initial goal was to export only what was needed, but since the unit
tests do a good job at testing almost everything, then
we were exporting almost everything, so I am now exporting full classes
instead. This is a lot less intrusive
and the code is cleaner, even though the export table can look a little bit
ugly.

Some of the problems I am seeing:

- ICU needs to be registered per module. Right now since the Initialize
function is in base.dll, it always get initialized there, even though it's
called
by other dlls. We would need to move to a more central dll icu mode. I think
it's already supported.

- Some classes in base were clearly designed to be in a lib. One example is
PathService. Right now when I call PathService::Get(FILE_MODULE), I
always get "base.dll". There are some similar problems with the VersionInfo
classes. The solution here would be to create a new project called
base_lib, which is always a lib. Unfortunately, PathService depends on a lot
of other base classes, so base.lib would need to depend on base.dll, and
base.dll would need to depend on base.lib (since it does use PathService).
So a major cleanup would be required here. I'm sure we can find a better
solution though.

- Global variables are not global for the app, they are global per module.
So if you use a lib that has a global variables, and this is used
from multiple dlls (base, net, chrome, etc), then they will have different
values.  The solution here would be to move this lib to a dll too.

- Singleton. They are not too much of an issue right now because they seem
to be all registered by base.dll, but there is a risk that you would
have multiple instances of a singleton. (one per module).

Eventually we would like to be able to split webkit in it's own dll too. I
heard to it's not possible right now due to some inter-dependency problems,
but people are working on that. Webkit was designed to be in its own dll, so
it should be less of a problem.

My next step is to back off a little bit from base and net, and go look at
the other more self contained modules, like libxml and the rest of the
third_party libs we use.

Depending on the feedback on this thread, I will trash the base/net split,
or finish it.

Let me know what you think. I want to know if you have huge concerns about
this, or if you think this would result in too much code
change for benefit we would get out of it.

Thanks

Nicolas

[Oh, and right now i'm using the DLL CRT, not tcmalloc. This is something
else I will need to get working before we can roll this out]

--~--~-~--~~~---~--~~
Chromium Developers mailing list: chromium-dev@googlegroups.com 
View archives, change email options, or unsubscribe: 
http://groups.google.com/group/chromium-dev
-~--~~~~--~~--~--~---

[chromium-dev] Re: fighting the test flakiness

[Nicolas Sylvain]

phajdan: Feel free to add a link at the top of the waterfall. Maybe beside
"perf" at the top left corner.
You just need to edit
http://src.chromium.org/viewvc/chrome/trunk/tools/buildbot/master.chromium/public_html/announce.html?view=markup

You
can send me the review.

Nicolas


On Tue, Aug 18, 2009 at 1:19 PM, Scott Violet  wrote:

>
> This is very useful. How about a link to this some where on the main
> builder page.
>
>  -Scott
>
> On Tue, Jul 28, 2009 at 9:52 AM, Paweł Hajdan
> Jr. wrote:
> > So, the flakiness dashboard is now public and updated daily,
> > at http://build.chromium.org/buildbot/flakiness-report/ . It displays
> top 15
> > flaky test groups and tests, and relevant parts of the logs ("toggle
> > details" javascript link).
> > It has dates of the first and last flip of each test or group. It's a
> > feature that was frequently asked for, and if you have any more feedback,
> I
> > will be glad to hear it.
> > One thing it currently does not track correctly are test crashes/timeouts
> I
> > think. I'm working on that. It also doesn't track browser_tests yet.
> > There is a label in the bug tracker "FlakyTest" - please use it when
> > reporting flaky tests. One of my goals is to fix those bugs.
> > >
> >
>
> >
>

--~--~-~--~~~---~--~~
Chromium Developers mailing list: chromium-dev@googlegroups.com 
View archives, change email options, or unsubscribe: 
http://groups.google.com/group/chromium-dev
-~--~~~~--~~--~--~---

[chromium-dev] Re: Stack traces on layout test crashes

[Nicolas Sylvain]

On Mon, Aug 10, 2009 at 10:35 AM, Jeremy Orlow  wrote:

> On Mon, Aug 10, 2009 at 9:59 AM, Ojan Vafai  wrote:
>
>> On Mon, Aug 10, 2009 at 3:12 AM, Jeremy Orlow wrote:
>>
>>>  On Sun, Aug 9, 2009 at 9:07 AM, Nicolas Sylvain 
>>> wrote:
>>>
>>>> 2. The show stopper for any implementation of this feature is that the
>>>> machines running the layout tests don't have the pdbs for test_shell. Since
>>>> the binary is built on another machine, it was too slow to copy the pdbs
>>>> from one machine to another. If you guys think it's important, and can take
>>>> the ~30-60 more seconds to cycle the bots, then we can copy them too, and
>>>> the feature would work.
>>>>
>>>
>>> I'm not really sure what the right answer is.  It sure would be nice if
>>> we didn't need to use a tool to decode the call stacks of crashed layout
>>> tests.  I kind of feel like an additional 30-60 seconds isn't going to be a
>>> big deal on these bots, but maybe it is.  What do people think?
>>>
>>
>> A minute is a really long amount of time to add for the bots to cycle. Our
>> goal historically has been 5 minutes for a full cycle. The tree is generally
>> so much greener and easier to keep that way when we have fast cycle times.
>> If there is a way to do it that doesn't require hitting the critical path, I
>> think we should seriously consider it.
>>
>
> K.  I'll assume that's not an option when I look into this, then.
>
We might be able to speed it up. We can try it and see how much slower it
gets.

see test_shell_switches.cc
There is already a :
const wchar_t kCrashDumps[] = L"crash-dumps";  // Enable crash dumps

It is using breakpad. I think it should work.

Nicolas

--~--~-~--~~~---~--~~
Chromium Developers mailing list: chromium-dev@googlegroups.com 
View archives, change email options, or unsubscribe: 
http://groups.google.com/group/chromium-dev
-~--~~~~--~~--~--~---

[chromium-dev] Re: Stack traces on layout test crashes

[Nicolas Sylvain]

Some things to consider:
1. On windows, breakpad used to be wired in test_shell. And I'm pretty sure
we used to archive crash dumps for the layout tests too.  It should not be
hard to do that again. Huan also write a nice script to dump to stdio the
crashing stacks of all crashes that happened in a buildbot run.  It only
runs on "chromium xp" for now, but there is no reason why we could not make
it work for the layout tests. See
http://build.chromium.org/buildbot/waterfall/builders/Chromium%20XP/builds/6491/steps/Process%20Dumps/logs/stdio
for
example.

2. The show stopper for any implementation of this feature is that the
machines running the layout tests don't have the pdbs for test_shell. Since
the binary is built on another machine, it was too slow to copy the pdbs
from one machine to another. If you guys think it's important, and can take
the ~30-60 more seconds to cycle the bots, then we can copy them too, and
the feature would work.

Nicolas

On Sat, Aug 8, 2009 at 11:42 AM, Albert J. Wong (王重傑)
wrote:

> FYI, the code in debug_util.h will generate a stack trace, but symbol
> resolution doesn't work on mac.  Last I messed with it (~4 months ago), mac
> didn't work because most of the symbols are private.  Mark Mentovai
> suggested trying to reimplement dladdr, but I could never get it working.
> Here's the uploaded code if anyone to mess with it:
>
>http://codereview.chromium.org/164228
>
> On windows and linux, assuming you actually have symbols generated (which I
> don't think you do for windows on the build bots), getting a trace should be
> as simple as creating one of those StackTrace objects in debug_util.h, and
> calling PrintBacktrace or OutputToStream on it.  The hard part is knowing
> when to create the object, and making sure you're on the right thread.
> Also, these functions do some heap allocations so using them in a crash
> handler might be a bit unsafe...but if it's crashing, and there's already no
> way to get a core or something, making it crash harder isn't going to
> matter.
>
> -Albert
>
>
>
> On Sat, Aug 8, 2009 at 11:12 AM, Jeremy Orlow  wrote:
>
>> I'll take a look.
>> If anyone has ideas on how to implement this (beyond looking at
>> base/debug_util.h) please let me know! The last time I messed around with
>> programatic stack traces was 5+ years ago. :-)
>>
>>
>> On Sat, Aug 8, 2009 at 7:53 AM, Dimitri Glazkov wrote:
>>
>>> Somebody please run with this! :)
>>>
>>> :DG<
>>>
>>> On Fri, Aug 7, 2009 at 8:45 PM, Darin Fisher wrote:
>>> > On Fri, Aug 7, 2009 at 8:39 PM, Ojan Vafai  wrote:
>>> >>
>>> >> On Fri, Aug 7, 2009 at 8:45 PM, Jeremy Orlow 
>>> wrote:
>>> >>>
>>> >>> Has anyone ever looked into printing out stack traces when layout
>>> tests
>>> >>> crash?  Of the couple layout test crashes I've investigated, I think
>>> most
>>> >>> could have been solved just by having a stack trace.  I'm not really
>>> sure
>>> >>> what's involved or if anyone's looked into this, which is why I'm
>>> asking.
>>> >>>  This could be especially helpful for flaky tests that crash.
>>> >>
>>> >> I don't remember anyone having looked into this. I agree that this
>>> would
>>> >> make tracking down and fixing these issues immensely easier,
>>> especially for
>>> >> tests that only sometimes crash.
>>> >> Ojan
>>> >
>>> > I've wanted this for a very long time.  I think there might be a bug on
>>> it.
>>> >  At any rate, we now have a nice utility in base/debug_util.h that can
>>> > provide a stack trace.  I would love to see crash stacks on the
>>> buildbot.
>>> >  It would probably help us eliminate a lot of flakiness!
>>> > -Darin
>>> > >
>>> >
>>>
>>
>>
>>
>>
>
> >
>

--~--~-~--~~~---~--~~
Chromium Developers mailing list: chromium-dev@googlegroups.com 
View archives, change email options, or unsubscribe: 
http://groups.google.com/group/chromium-dev
-~--~~~~--~~--~--~---

[chromium-dev] Re: flaky compilation of sandbox on Vista

[Nicolas Sylvain]

On Fri, Aug 7, 2009 at 11:02 AM, Paweł Hajdan Jr.
wrote:

> Example:
> http://build.chromium.org/buildbot/waterfall/builders/Modules%20Vista%20(dbg)/builds/13568/steps/compile_3/logs/stdio
> C:\Windows\system32\cmd.exe /c src\third_party\python_24\python_slave.exe
> ..\..\..\scripts\slave\compile.py --solution sandbox.sln --target Debug
> --build-dir src/sandbox in dir c:\b\slave\sub-dbg-vista\build (timeout 600
> secs) watching logfiles {} argv: ['C:\\Windows\\system32\\cmd.exe', '/c',
> 'src\\third_party\\python_24\\python_slave.exe',
> '..\\..\\..\\scripts\\slave\\compile.py', '--solution', 'sandbox.sln',
> '--target', 'Debug', '--build-dir', 'src/sandbox'] environment:
> APPDATA=C:\Users\chrome-bot\AppData\Roaming CHROME_HEADLESS=1
> COMSPEC=C:\Windows\system32\cmd.exe GYP_MSVS_VERSION=2005
> HOMEPATH=\Users\chrome-bot LOCALAPPDATA=C:\Users\chrome-bot\AppData\Local
> LOGNAME=chrome-bot NUMBER_OF_PROCESSORS=2 OS=Windows_NT
> PATH=c:\b\depot_tools;c:\b\depot_tools\python_bin;C:\Windows\system32;C:\Windows\system32\WBEM
> PATHEXT=.COM;.EXE;.BAT;.CMD;.VBS;.VBE;.JS;.JSE;.WSF;.WSH;.MSC
> PROGRAMFILES=C:\Program Files
> PYTHONPATH=c:\b\scripts\private;c:\b\scripts\common;c:\b\scripts\slave;c:\b\pylibs;c:\b\symsrc
> SYSTEMDRIVE=C: SYSTEMROOT=C:\Windows
> TEMP=C:\Users\CHROME~1\AppData\Local\Temp
> TMP=C:\Users\CHROME~1\AppData\Local\Temp USERNAME=chrome-bot
> USERPROFILE=C:\Users\chrome-bot WINDIR=C:\Windows closing stdin using PTY:
> False command timed out: 600 seconds without output, killing pid 79348
> program finished with exit code 1
> I don't understand why it happens, but seen it a few times, and it's
> definitely a flaky issue. :-/
>

If you notice it before it ends (if it has been "building" for too long and
still has not produced output), let me know and I'll connect to the machine
to see what's going on.

Nicolas


>
> >
>

--~--~-~--~~~---~--~~
Chromium Developers mailing list: chromium-dev@googlegroups.com 
View archives, change email options, or unsubscribe: 
http://groups.google.com/group/chromium-dev
-~--~~~~--~~--~--~---

[chromium-dev] Re: Running UI test in parallel (experimental)

[Nicolas Sylvain]

On Thu, Aug 6, 2009 at 1:06 PM, John Abd-El-Malek  wrote:

> This is very cool, but I ran into a few problems when I tried to run it:
> a:\chrome2\src\chrome>tools\test\smoketests.py --tests=ui
>
> >> You must have your local path of trunk/src/tools/python added to your
> PYTHONPATH.<<
>
> Traceback (most recent call last):
>   File "A:\chrome2\src\chrome\tools\test\smoketests.py", line 32, in
> 
> import google.httpd_utils
> ImportError: No module named google.httpd_utils
>

You get this error when you use a python that is different from the one in
src\third_party\python_24\python.exe


>
>
> hmmm, never needed PYTHONPATH set before.  Can't this script set it itself?
>  Setting it manually will fail when I move depot locations etc..  But
> anyways, I set it and then
>
> a:\chrome2\src\chrome>set PYTHONPATH=a:\chrome\src\tools\python
>
> a:\chrome2\src\chrome>tools\test\smoketests.py --tests=ui
> Traceback (most recent call last):
>   File "A:\chrome2\src\chrome\tools\test\smoketests.py", line 274, in
> 
> result = main(options, args)
>   File "A:\chrome2\src\chrome\tools\test\smoketests.py", line 155, in main
> sys.path.insert(0, _BuildbotScriptPath('slave'))
>   File "A:\chrome2\src\chrome\tools\test\smoketests.py", line 84, in
> _BuildbotScriptPath
> 'scripts', sub_dir)
>   File "a:\chrome\src\tools\python\google\path_utils.py", line 72, in
> FindUpward
> parent = FindUpwardParent(start_dir, *desired_list)
>   File "a:\chrome\src\tools\python\google\path_utils.py", line 55, in
> FindUpwardParent
> (desired_path, start_dir))
> google.path_utils.PathNotFound: Unable to find tools\buildbot\scripts\slave
> above A:\chrome2\src\chrome\tools\test
>
>
>
> tools\buildbot isn't in the public tree I think, since I don't find it
> here: http://src.chromium.org/viewvc/chrome/trunk/src/chrome/tools/.  So
> external contributors can't use this?  Also, why should this script depend
> on the buildbot scripts?  I don't have them so I can't try this out.
>
http://src.chromium.org/viewvc/chrome/trunk/tools/buildbot/

Not sure why it needs it though.

>
> Can't we just have a minimal python script that runs ui_tests in parallel?
>

Nicolas

>
> On Wed, Jul 29, 2009 at 3:28 PM, Huan Ren  wrote:
>
>>
>> I just checked in a change to run ui_tests in parallel based on
>> sharding mechanism provided by GTest. Each ui_tests instance has its
>> own user data dir, and the number of ui_tests instances is
>> NUMBER_OF_PROCESSORS. I have updated
>> src/chrome/tools/test/smoketests.py so you can run it through command
>> line:
>>
>> python.exe smoketests.py --tests=ui [--verbose]
>>
>> Running ui_tests.exe directly is still the old behavior of
>> sequentially running. On my 4 core machine, the running time has been
>> reduced by half, from 832 secs to 443 secs. But I need to make sure
>> all tests can run reliably in this parallel fashion. So if you try it
>> out, I will be interested to know how fast the performance is improved
>> and what additional tests are failing.
>>
>> Huan
>>
>> P.S. this change is for Windows platform as I think Linux/Mac is
>> already using GTest sharding.
>>
>>
>>
>
> >
>

--~--~-~--~~~---~--~~
Chromium Developers mailing list: chromium-dev@googlegroups.com 
View archives, change email options, or unsubscribe: 
http://groups.google.com/group/chromium-dev
-~--~~~~--~~--~--~---

[chromium-dev] Re: Flakiness. Please help.

[Nicolas Sylvain]

On Thu, Aug 6, 2009 at 1:05 PM, Scott Violet  wrote:

>
> Not sure, perhaps Huan could answer that. That said, --enable-dcheck
> certainly works on the Chromium release builds from the buildbot:
> http://build.chromium.org/buildbot/continuous/LATEST/ .


Yes, --enable-dcheck is supposed to be disabled in Google Chrome build.

Nicolas


>
>
>  -Scott
>
> On Thu, Aug 6, 2009 at 12:59 PM, Antony Sargent
> wrote:
> > To clarify, doesn't --enable-dcheck only work on chromium release builds
> you
> > built yourself and not official builds of Google Chrome?
> >
> > On Thu, Aug 6, 2009 at 10:15 AM, Scott Violet  wrote:
> >>
> >> One easy suggestion in helping catch bugs is to run Chrome with
> >> --enable-dcheck . This'll prompt if you hit a DCHECK in release builds
> >> and hopefully help isolate crashes before the fact.
> >>
> >>  -Scott
> >>
> >> On Wed, Aug 5, 2009 at 9:44 PM, Peter Kasting
> wrote:
> >> > THIS MAIL APPLIES TO YOU
> >> > Flakiness is growing.  Smash it before it gets bigger, and keep it
> >> > smashed.
> >> > ***
> >> > The MOST IMPORTANT section in this gigantic mail:
> >> > PLEASE spend some of every workday (or each week at least, if you
> can't
> >> > spare time each day) looking at test failures, flakiness,
> >> > valgrind/purify/coverity bugs, crashes, and/or memory bugs.  Make it a
> >> > goal
> >> > to get an average of one line in the test-expectations file removed
> each
> >> > day.  If you're a Googler, put it on your OKRs (now, not sometime
> >> > tomorrow).
> >> > * DON'T wait for someone to assign bugs to you or ask for your help
> >> > * DON'T wait for a team fixit week (those haven't worked)
> >> > * DON'T wait for someone else to solve the problems
> >> > * DON'T wait until after your current project is finished
> >> > * DON'T wait until you have worked on WebKit
> >> > HELP, even if it's just a little, even if it's not your core
> competence.
> >> >  We
> >> > currently have hundreds upon hundreds of failing or flaky tests.  We
> can
> >> > dramatically reduce this quickly but ONLY IF YOU HELP.  This is an
> >> > investment not only in the quality of Chrome but in the team's ability
> >> > to
> >> > move fast, so help here doesn't just improve the quality of Chrome,
> but
> >> > also
> >> > the derivative of the quality :)
> >> > (If you do not know how to do anything above and need handholding,
> >> > e-mail me
> >> > and I will help you.  It's OK to be ignorant.)
> >> > ***
> >> > Next, how you should help keep the tree green at all times:
> >> > * If you ever look at the buildbot and see red, and there's no
> >> > explanation
> >> > in the build status, ask what's going on on #chromium.  Ping the
> >> > sheriffs
> >> > specifically (they're listed in the upper-right corner).  If you do
> not
> >> > get
> >> > an answer about ownership within a few minutes, close the tree (if you
> >> > have
> >> > the rights to) or ask someone to close it.  THE TREE SHOULD NOT BE
> OPEN
> >> > WITH
> >> > RED THAT NO ONE OWNS.  Help the sheriffs out with this -- they can't
> >> > watch
> >> > every second.  Closed trees suck; unowned bustage sucks more.  Be
> >> > hard-nosed.
> >> > * Yes, even purify, valgrind, and reliability bot redness.  If you
> can't
> >> > figure out what to do with these, try pinging erikkay for purify
> issues
> >> > and
> >> > huanr for reliability issues.  (Not sure who a good general valgrind
> >> > contact
> >> > is.)
> >> > * If you ever look at the buildbot and see orange ("unexpected pass"),
> >> > especially in the WebKit LayoutTest bots, ping the WebKit sheriff (the
> >> > calendar is linked from the top
> >> > of http://dev.chromium.org/developers/how-tos/webkit-merge-1 ; I
> don't
> >> > know
> >> > whether it's world-readable).  If he wasn't aware of it, agree between
> >> > you
> >> > on who will deal with it.  Orange alone is not reason to close the
> tree,
> >> > but
> >> > it should NOT be ignored.
> >> > * DON'T IGNORE TESTS BECAUSE THEY WENT GREEN ON THE NEXT CYCLE.  If
> >> > they're
> >> > really fixed by someone's commit, that should be easy to determine.
> >> >  Otherwise, they're flaky, and we NEED to mark them as such, not just
> >> > leave
> >> > them.
> >> > ***
> >> > Finally, how to help if the LayoutTest bots are red or orange:
> >> > (1) Try and determine if the test(s) are consistently passing/failing
> >> > unexpectedly, or if they're flaky.  Make sure you look at all the
> >> > different
> >> > bots to see which OSes are affected.
> >> > (2) Update src/webkit/tools/layout_tests/test-expectations.txt.  Look
> >> > for
> >> > the test(s) in question.  Often, flaky tests will already be in there
> as
> >> > failing or flaky for one OS, and need to have more added; or they will
> >> > be
> >> > marked flaky ("FAIL PASS") and need "CRASH" added.  If they're not
> >> > there,
> >> > add a line.
> >> > (3) Ensure the test(s) have a bug on file.  Note the bug on the
> >> > expectation.
> >> > (4) If any tests are crashing (flaky or not), t

[chromium-dev] Re: fighting the flakiness - resource bundle issues?

[Nicolas Sylvain]

On Tue, Jul 28, 2009 at 4:37 PM, Evan Martin  wrote:

>
> On Tue, Jul 28, 2009 at 4:35 PM, Paweł Hajdan
> Jr. wrote:
> > On Tue, Jul 28, 2009 at 16:30, Evan Martin  wrote:
> >>
> >> This happens semi-frequently.  It's related to the grit resources
> >> rebuilding.  Something like: not remapping IDR_THROBBER_WAITING to the
> >> right place, or remapping it to the right place but not rebuilding
> >> this file, or something like that.
> >
> > Will the workaround for the original problem (clobbering headers
> generated
> > by .grd files) work also for this issue? If not, do you have any idea for
> a
> > fix or workaround?
> > Or maybe... why not set the resources projects to always-build, even just
> > for the buildbots? lastchange works this way, so it is possible. I think
> it
> > is not more or less crude than the clobbering workaround, and is simpler
> to
> > implement. Increase in the build time should not be significant (on the
> > bots; doing that on developers' machine probably isn't a good idea).
>
> I like the idea of always-building the resources project.
> Though note it can cause a lot of code to rebuild: it generates
> headers, which are included via includes via includes...


I still vote for the original idea of clobbering the .h files if the .grd
file is changed. I think it should be easy to implement with the DEPS file.

The faster the incremental build is on the bots, the faster they cycle. They
are already too slow, that'd be sad to make it even slower.

Nicolas


>
>
> >
>

--~--~-~--~~~---~--~~
Chromium Developers mailing list: chromium-dev@googlegroups.com 
View archives, change email options, or unsubscribe: 
http://groups.google.com/group/chromium-dev
-~--~~~~--~~--~--~---

[chromium-dev] Re: fighting the flakiness - resource bundle issues?

[Nicolas Sylvain]

On Fri, Jul 24, 2009 at 11:10 AM, Tony Chang  wrote:

> I haven't seen this error in VS for probably a year.  It's true that
> changing a resource (e.g., a png file) would not pick up the change
> since we don't specify dependencies on the actual data files, but
> changes to grd files should trigger the right things to rebuild.
>
> When we see the error on the buildbots, what's normally happening is
> that a new resource is added which causes IDs after the new resource
> to shift.  The header is updated with the new values and new .rc files
> are generated.  The .rc files properly get rebuilt into the locale dll
> since the .rc files changed.  However, some of the .cc files don't
> recompile so they are referencing the old IDs and get the wrong
> resource (e.g., we would get the wrong title for a tab).
>
> Is there a recent build log where this failed?  I think we'll be able
> to determine more from that.


the build never actually fails, but the test fails, and it's exactly what
you said, it expects STRINGX, but gets STRINGY (which happened to have
the same ID that STRINGX had before).

I still think this is expected. Modifying a rc file should not force every
files which depends on it to rebuild. Since when you use the VS UI, it
will never change the order, so the rebuild will be useless. (And if you
add or delete a string, you'll have the cc files containing them modified
anyway, so
it will rebuild correctly that part).

I guess someone should try it ;)

Nicolas



>
> On Fri, Jul 24, 2009 at 11:00 AM, Nicolas Sylvain
> wrote:
> > Tony, just to make sure, are you certain that this is a IB only issue?
> > I used to be convinced that it was happening with visual studio too.
> > If this is a IB only problem, we should create a repro case and send it
> to
> > them, they
> > are usually really responsive.
> > [For some reasons, I thought it was because it was generating the rc
> files
> > and the ID in them were not static. I.e. you add a string in the middle,
> and
> > it's shifting the string IDs of all the strings after the addition.
> Visual
> > Studio does not support that. You can't change string ids in a RC file
> > without having the clobber the source. The normal use case is always to
> take
> > the next available ID (which is why the rc file always keep track of the
> > next available ID at the bottom).]
> > Nicolas
> >
> > On Thu, Jul 23, 2009 at 5:54 PM, Tony Chang  wrote:
> >>
> >> Look at how the current gyp hook works.  It looks for changes to .gyp
> >> files and only runs if a .gyp (and maybe gypi?) file has changed.
> >>
> >> You can find what header it generates by opening the grd file and
> >> parsing the XML (the XML lists the output files).  You'll need to
> >> build the base directory (e.g., Debug/obj/global_intermediate/ and the
> >> Release version), I would just check for the Windows versions since we
> >> don't seem to have this problem with the mac or linux buildbots.
> >>
> >>
> >>
> >> On Thu, Jul 23, 2009 at 5:36 PM, Paweł Hajdan
> >> Jr. wrote:
> >> > I think that this workaround may be worth it. I'm not familiar with
> the
> >> > IncrediBuild, but I can help making the hook (and we can run it only
> on
> >> > Windows).
> >> > How do I make a hook know which grd files changed? And also know which
> >> > headers it generates? Alternatively, maybe this Windows-only hook
> would
> >> > just
> >> > delete all generated headers (with a hardcoded list)? Generation seems
> >> > to be
> >> > cheap, and such hook seems trivial to write.
> >> > So, yes, this hook is kludgey. But we are aware of its limitations,
> and
> >> > it
> >> > would eliminate one kind of build mysteries. What do you think?
> >> >
> >> > On Thu, Jul 23, 2009 at 17:30, Tony Chang  wrote:
> >> >>
> >> >> Here's a crappy work around:
> >> >> Add a gclient hook that checks for grd file changes.  When a grd file
> >> >> changes, force delete the header it would generate.  I'm pretty sure
> >> >> this would prevent bad builds from IncrediBuild.
> >> >>
> >> >> Alternately, maybe we can make a reduced test case and file a bug
> >> >> against IncrediBuild.
> >> >>
> >> >> On Thu, Jul 23, 2009 at 4:44 PM, Paweł Hajdan
> >> >> Jr. wrote:
> >> >> > Is it possible to force it to rebuild some files, or... I don'

[chromium-dev] Re: fighting the flakiness - resource bundle issues?

[Nicolas Sylvain]

Tony, just to make sure, are you certain that this is a IB only issue?
I used to be convinced that it was happening with visual studio too.

If this is a IB only problem, we should create a repro case and send it to
them, they
are usually really responsive.

[For some reasons, I thought it was because it was generating the rc files
and the ID in them were not static. I.e. you add a string in the middle, and
it's shifting the string IDs of all the strings after the addition. Visual
Studio does not support that. You can't change string ids in a RC file
without having the clobber the source. The normal use case is always to take
the next available ID (which is why the rc file always keep track of the
next available ID at the bottom).]

Nicolas

On Thu, Jul 23, 2009 at 5:54 PM, Tony Chang  wrote:

>
> Look at how the current gyp hook works.  It looks for changes to .gyp
> files and only runs if a .gyp (and maybe gypi?) file has changed.
>
> You can find what header it generates by opening the grd file and
> parsing the XML (the XML lists the output files).  You'll need to
> build the base directory (e.g., Debug/obj/global_intermediate/ and the
> Release version), I would just check for the Windows versions since we
> don't seem to have this problem with the mac or linux buildbots.
>
>
>
> On Thu, Jul 23, 2009 at 5:36 PM, Paweł Hajdan
> Jr. wrote:
> > I think that this workaround may be worth it. I'm not familiar with the
> > IncrediBuild, but I can help making the hook (and we can run it only on
> > Windows).
> > How do I make a hook know which grd files changed? And also know which
> > headers it generates? Alternatively, maybe this Windows-only hook would
> just
> > delete all generated headers (with a hardcoded list)? Generation seems to
> be
> > cheap, and such hook seems trivial to write.
> > So, yes, this hook is kludgey. But we are aware of its limitations, and
> it
> > would eliminate one kind of build mysteries. What do you think?
> >
> > On Thu, Jul 23, 2009 at 17:30, Tony Chang  wrote:
> >>
> >> Here's a crappy work around:
> >> Add a gclient hook that checks for grd file changes.  When a grd file
> >> changes, force delete the header it would generate.  I'm pretty sure
> >> this would prevent bad builds from IncrediBuild.
> >>
> >> Alternately, maybe we can make a reduced test case and file a bug
> >> against IncrediBuild.
> >>
> >> On Thu, Jul 23, 2009 at 4:44 PM, Paweł Hajdan
> >> Jr. wrote:
> >> > Is it possible to force it to rebuild some files, or... I don't know,
> do
> >> > you
> >> > see some real way to fix this problem?
> >> >
> >> > On Thu, Jul 23, 2009 at 16:41, Tony Chang  wrote:
> >> >>
> >> >> To elaborate on Peter's comment.  IncrediBuild (which the buildbots
> >> >> use) get confused by changes to our grd files.  Our grd files
> generate
> >> >> headers, which should then cause lots of cc files to get rebuilt.
> >> >> Visual Studio seems to always get this right, but IncrediBuild gets
> >> >> this wrong and cc files don't get rebuilt.  I imagine IncrediBuild is
> >> >> checking the timestamp of the file before the headers are
> re-generated
> >> >> and doesn't realize it needs to rebuild.
> >> >>
> >> >> On Thu, Jul 23, 2009 at 4:38 PM, Peter Kasting
> >> >> wrote:
> >> >> > On Thu, Jul 23, 2009 at 4:31 PM, Paweł Hajdan Jr.
> >> >> > 
> >> >> > wrote:
> >> >> >>
> >> >> >> Some of the flaky failures are caused by resource bundle issues.
> If
> >> >> >> you
> >> >> >> are familiar with the build process, or the resource bundle,
> please
> >> >> >> take a
> >> >> >> look.
> >> >> >
> >> >> > It looks like something needed a manual clobber and didn't get it.
> >> >> > PK
> >> >> > >> >> >
> >> >> >
> >> >
> >> >
> >
> >
>
> >
>

--~--~-~--~~~---~--~~
Chromium Developers mailing list: chromium-dev@googlegroups.com 
View archives, change email options, or unsubscribe: 
http://groups.google.com/group/chromium-dev
-~--~~~~--~~--~--~---

[chromium-dev] Re: Generating files in source tree considered harmful

[Nicolas Sylvain]

On Thu, Jul 23, 2009 at 8:53 AM, Evan Martin  wrote:

> On Thu, Jul 23, 2009 at 7:47 AM, Nicolas Sylvain
> wrote:
> > gclient has nothing to do with this case. "svn update src/" was trying to
> > add a directory called "src/bleh", but "src/bleh" already existed, so
> "svn
> > update" failed.
>
> Sorry to back-seat drive, but can't you do something like
>  "svn status | xargs rm -rf"
> before syncing?


svn status does not show files that are svn:ignore'd, like all generated
files should be. But, that's a good point,
the directory might not be ignored, so that can work. We already have code
to do that and we do it on the
try bots. This is really slow though, can take 1-2 minutes to run before
each sync.

>
>
> (In git it's a builtin: "git clean -f" to delete all files that aren't
> officially part of the repository.)
>
> > The only thing gclient might want to do, is clobber your tree, and try
> > again.  We used to have stuff like that, but they are all disabled AFAIK,
> I
> > lost too many important files because gclient wanted to be nice and clean
> up
> > my machine.
>
> I don't understand this paragraph, which might explain why my above
> proposal is dumb.


An example that bit me in the past: You have a DEPS file that fetches
src/blah.  you add new code in src/blah, but
before you commit, someone deletes the line "src/blah" in the DEPS file. At
the time the default behavior was :
"rm -rf src/blah", which killed all my new code in there.

Maybe the non-versionned files are important. We can't kill them.  (We could
have a mode for buildbot only
that is more aggressive though, since we are not likely to have important
non-versionned files there).

Nicolas

--~--~-~--~~~---~--~~
Chromium Developers mailing list: chromium-dev@googlegroups.com 
View archives, change email options, or unsubscribe: 
http://groups.google.com/group/chromium-dev
-~--~~~~--~~--~--~---

[chromium-dev] Re: Generating files in source tree considered harmful

[Nicolas Sylvain]

On Wed, Jul 22, 2009 at 9:15 PM, Mark Larson (Google) wrote:

>
>
> On Wed, Jul 22, 2009 at 20:27, Dan Kegel  wrote:
>
>>
>> Stop me if you've heard this one before.
>>
>> Today, a new directory was added to the source tree, and shortly
>> thereafter was reverted.
>> Should have been no problem, but... because the new directory
>> contained a gyp file, a file was generated in that directory,
>> and svn couldn't delete the directory when the revert landed.
>> This caused a build breakage, and I gather from nsylvain's
>> comments that this wasn't the first time this has happened.
>>
>> At some point soon, it'd be good to teach gyp not to generate
>> files in the source tree.
>
>
> Or maybe teach gclient how to deal forcefully with directories with no
> files under version control. Generating files in the source tree is kinda
> the point of gyp.
>

gclient has nothing to do with this case. "svn update src/" was trying to
add a directory called "src/bleh", but "src/bleh" already existed, so "svn
update" failed.

The only thing gclient might want to do, is clobber your tree, and try
again.  We used to have stuff like that, but they are all disabled AFAIK, I
lost too many important files because gclient wanted to be nice and clean up
my machine.

What we currently do, on the buildbot side, is clobber the source tree when
something goes wrong with gclient. In this case it mostly worked, except on
about 8 machines, where a file was locked. (No idea what it was, procexp was
not helpful, closing all apps either. I had to reboot the machine)

To answer dirk : Well, it needs manual cleanup. Ask a trooper to connect to
each machine and fix them. (Or ask on irc, at this point about half of the
team knows the passwords to the machines :| ).  I guess we should formalize
it more : "When in doubt, find someone to reboot the machine". It does not
need any special skills, the machine will auto login and restart all the
buildbot stuff automatically.

Nicolas

>
>
>>
>>
>>
>>
>
> >
>

--~--~-~--~~~---~--~~
Chromium Developers mailing list: chromium-dev@googlegroups.com 
View archives, change email options, or unsubscribe: 
http://groups.google.com/group/chromium-dev
-~--~~~~--~~--~--~---

[chromium-dev] Re: FYI: Upcoming O3D integration changes.

[Nicolas Sylvain]

On Wed, Jul 22, 2009 at 11:47 AM, Greg Spencer  wrote:

>
>
> On Wed, Jul 22, 2009 at 11:27 AM, Nicolas Sylvain 
> wrote:
>
>>   "src/third_party/nixysa/files":
>>>
>> why in a subdir called "files"?
>>
>
> A leftover from converting from p4/scons -- I'll remove it.
>
>   # NACL has to be in this weird directory because it looks for
>>>   # googleclient two levels above it.
>>>   "src/third_party/native_client/googleclient/native_client":
>>>
>> Looks like they should change their code to make it work without assuming
>> all these directories.
>>
>> and this code is already fetched in src/native_client, we don't want it
>> twice.
>>
>
> Yes, that just happened recently -- I'll try to switch to using their new
> GYP build.
>
> For those who are curious :
>>
>>  $ du -h --max-depth=1 .
>> 4.1M./gflags
>> 34M ./native_client
>> 1.3M./nixysa
>> 251K./npapi
>> 2.1M./ply-3.1
>> 24M ./vectormath
>> 64M .
>>
>
>
> And the additional native_client will disappear, so more like 30M.  (and
> these numbers include all the .svn dirs, so the real code is half that).
> The docs in the vectormath library are 17M, so we could probably delete
> those from the repo if size is an issue, and make it 13M total.
>
That would be great!

Thanks

Nicolas

>
>
> -Greg.
>

--~--~-~--~~~---~--~~
Chromium Developers mailing list: chromium-dev@googlegroups.com 
View archives, change email options, or unsubscribe: 
http://groups.google.com/group/chromium-dev
-~--~~~~--~~--~--~---

[chromium-dev] Re: FYI: Upcoming O3D integration changes.

[Nicolas Sylvain]

On Tue, Jul 21, 2009 at 4:30 PM, Greg Spencer  wrote:

> Hello Chromium Devs,
> The O3D team is working on getting O3D integrated into the Chromium build,
> and we're close to being able to complete our first step towards
> integration:  To build the O3D plugin as part of the Chromium code base, and
> link it into Chromium DLL.  It will still be a "plugin", but will be added
> to the internal plugins list so that its entry points are found in the
> Chromium DLL, instead of loading a separate plugin DLL.  This is a small
> step, and makes little practical difference except that 1) it will be
> bundled with Chromium, and 2) it opens the door to start integrating more
> fully.
>
> To achieve this first step, we have converted the O3D plugin to build using
> GYP, and have migrated to using all of the third_party libraries that we
> have in common at the same revision levels as Chromium is using.
>
> The next thing to do is to change Chromium's DEPS file to include the few
> remaining necessary dependencies that O3D uses and Chromium does not.
>
> These are:  A vector math library (vectormath), Nixysa: an NPAPI IDL
> generator, gflags (the python parts, for nixysa), and native client (for the
> nacl imc libraries).
>
Overall it looks good to me, with some comments/questions:

>
> I'm planning to map these like this:
> --
>   "src/third_party/vectormath":
> "
> http://o3d.googlecode.com/svn/trunk/googleclient/third_party/vectormath@";
> + Var("o3d_code_rev"),
>
>   "src/third_party/nixysa/files":
>
why in a subdir called "files"?

> "http://nixysa.googlecode.com/svn/trunk/nixysa@"; + Var("nixysa_rev"),
>
>   "src/third_party/npapi/files":
>
same here?

> "http://nixysa.googlecode.com/svn/trunk/third_party/npapi@"; +
> Var("nixysa_rev"),
>
>   "src/third_party/ply/files":
>
same here?

> "http://nixysa.googlecode.com/svn/trunk/third_party/ply-3.1@"; +
> Var("nixysa_rev"),
>
>   # NACL has to be in this weird directory because it looks for
>   # googleclient two levels above it.
>   "src/third_party/native_client/googleclient/native_client":
>
Looks like they should change their code to make it work without assuming
all these directories.

and this code is already fetched in src/native_client, we don't want it
twice.


> "
> http://nativeclient.googlecode.com/svn/trunk/nacl/googleclient/native_cli...@188
> ",
>
And if we really can't, then this path does not exist on head for this
repo. Any change we can sync to a later revision so we don't have to change
the path later on?


>
>   "src/third_party/gflags/":
> "http://google-gflags.googlecode.com/svn/tr...@30";,
> --
>
> In addition, I'll be making the Windows build of Chromium be dependent upon
> building O3D as part of the build process.
>
> This is really just a "heads up!" announcement to prompt heated discussions
> about the ensuing calamities that this will cause, so please feel free to
> comment.
>

For those who are curious :

$ du -h --max-depth=1 .
4.1M./gflags
34M ./native_client
1.3M./nixysa
251K./npapi
2.1M./ply-3.1
24M ./vectormath
64M .


Nicolas

>
> -Greg.
>
> >
>

--~--~-~--~~~---~--~~
Chromium Developers mailing list: chromium-dev@googlegroups.com 
View archives, change email options, or unsubscribe: 
http://groups.google.com/group/chromium-dev
-~--~~~~--~~--~--~---

[chromium-dev] Re: No more buildbot auto-refresh - Update your bookmark!

[Nicolas Sylvain]

On Tue, Jul 21, 2009 at 12:18 PM, Erik Kay  wrote:

> On Tue, Jul 21, 2009 at 11:53 AM, Nicolas Sylvain 
> wrote:
>
>> On Tue, Jul 21, 2009 at 11:37 AM, Aaron Boodman  wrote:
>>
>>> Nicolas,
>>>
>>> Are you going to be turning back on the extension support at any
>>> point? If not, just let us know so that we can remove the feature from
>>> the sample. Right now it looks broken.
>>
>>
>> Is the reload in the extension still 30secs?  If it can be changed to 2 or
>> 5 minutes I will reenable it.
>>
>
> We can fix it, but until extension autoupdate gets enabled, existing users
> with it installed will still have it on a 30 second timer.  I'll let you
> know when we've pushed out the change, and we'll send a PSA asking people to
> update manually.
>

Awesome, thanks.


>
> Erik
>
>
>
>
>>
>> Nicolas
>>
>>
>>>
>>>
>>> - a
>>>
>>> On Tue, Jul 21, 2009 at 8:27 AM, Nicolas Sylvain
>>> wrote:
>>> > Hi,
>>> > Last week a lot of people on this list said that they like the idea of
>>> > disabling the auto-reload for
>>> > our buildbot waterfall and console view. This is now implemented.
>>> > If you do want to keep the page auto-refreshing, please update your
>>> > bookmark. just add the
>>> > query parameter "reload=X", where X is the number of second to wait
>>> between
>>> > each reloads.
>>> > X used to be 90. Depending on your needs, you can increase or decrease
>>> this
>>> > number.
>>> > Examples:
>>> > console view:
>>> > http://build.chromium.org/buildbot/waterfall/console?reload=180
>>> > waterfall
>>> > : http://build.chromium.org/buildbot/waterfall/waterfall?reload=180
>>> > For sheriffs:
>>> >
>>> http://build.chromium.org/buildbot/waterfall/waterfall?failures=1&reload=30
>>> > Thanks
>>> > Nicolas
>>> > >
>>> >
>>>
>>
>>
>>
>>
>
> >
>

--~--~-~--~~~---~--~~
Chromium Developers mailing list: chromium-dev@googlegroups.com 
View archives, change email options, or unsubscribe: 
http://groups.google.com/group/chromium-dev
-~--~~~~--~~--~--~---

[chromium-dev] Re: No more buildbot auto-refresh - Update your bookmark!

[Nicolas Sylvain]

On Tue, Jul 21, 2009 at 11:37 AM, Aaron Boodman  wrote:

> Nicolas,
>
> Are you going to be turning back on the extension support at any
> point? If not, just let us know so that we can remove the feature from
> the sample. Right now it looks broken.


Is the reload in the extension still 30secs?  If it can be changed to 2 or 5
minutes I will reenable it.

Nicolas


>
>
> - a
>
> On Tue, Jul 21, 2009 at 8:27 AM, Nicolas Sylvain
> wrote:
> > Hi,
> > Last week a lot of people on this list said that they like the idea of
> > disabling the auto-reload for
> > our buildbot waterfall and console view. This is now implemented.
> > If you do want to keep the page auto-refreshing, please update your
> > bookmark. just add the
> > query parameter "reload=X", where X is the number of second to wait
> between
> > each reloads.
> > X used to be 90. Depending on your needs, you can increase or decrease
> this
> > number.
> > Examples:
> > console view:
> > http://build.chromium.org/buildbot/waterfall/console?reload=180
> > waterfall
> > : http://build.chromium.org/buildbot/waterfall/waterfall?reload=180
> > For sheriffs:
> >
> http://build.chromium.org/buildbot/waterfall/waterfall?failures=1&reload=30
> > Thanks
> > Nicolas
> > > >
> >
>

--~--~-~--~~~---~--~~
Chromium Developers mailing list: chromium-dev@googlegroups.com 
View archives, change email options, or unsubscribe: 
http://groups.google.com/group/chromium-dev
-~--~~~~--~~--~--~---

[chromium-dev] Re: Please keep TOOLKIT_VIEWS green!

[Nicolas Sylvain]

On Tue, Jul 21, 2009 at 8:31 AM, Paweł Hajdan Jr.
wrote:

> Can we have a trybot with that configuration, which would just compile the
> code? I think it would really save people's time. I never build with
> TOOLKIT_VIEWS, and in case of breakage I would have to immediately revert
> and then investigate. If I got a trybot failure, I would know much more
> before committing.


Ben: How likely is this to break?

I'm reluctant to add yet another try bot because they do take a lot of
resources.  There are maybe 200-300 try changes sent every day. That means
we need enough resources to build this configuration that many times, and it
needs to answer fast enough.

I think we should create this try bot, but make it "on demand" only. We are
already planning to do this for valgrind, purify and layout tests. That
won't help much catching random errors at the first place, but at least
you'll be able to see if you fixed it correctly. And if you think your code
might be breaking it, you can always pre-emptively trigger a try run on this
try bot.

I'm not sure yet when this kind of infrastructure is going to be ready. I
think it's on maruel's plate.

Nicolas

>
> On Mon, Jul 20, 2009 at 17:50, Ben Goodger (Google) wrote:
>
>>
>> This mostly applies to people working on UI.
>>
>> Sometime tonight or tomorrow we'll be moving the TOOLKIT_VIEWS builder
>> to the front page of the waterfall. Right now it's building just the
>> chrome target, and running no tests, but in time we'll build this out.
>> This is an experimental build distinct from the standard Chromium
>> linux which we will ship but a codepath which we will support as first
>> class for other purposes. So please keep it green! If it goes red, the
>> same rules that apply to any other builder on the front page will
>> apply.
>>
>> To build,
>>
>> export GYP_DEFINES="toolkit_views=1"
>> gclient runhooks --force
>> hammer chrome
>>
>> Thanks,
>>
>> -Ben
>>
>>
>>
>
> >
>

--~--~-~--~~~---~--~~
Chromium Developers mailing list: chromium-dev@googlegroups.com 
View archives, change email options, or unsubscribe: 
http://groups.google.com/group/chromium-dev
-~--~~~~--~~--~--~---

[chromium-dev] Re: Please keep TOOLKIT_VIEWS green!

[Nicolas Sylvain]

On Mon, Jul 20, 2009 at 5:50 PM, Ben Goodger (Google) wrote:

>
> This mostly applies to people working on UI.
>
> Sometime tonight or tomorrow we'll be moving the TOOLKIT_VIEWS builder
> to the front page of the waterfall. Right now it's building just the
> chrome target, and running no tests, but in time we'll build this out.
> This is an experimental build distinct from the standard Chromium
> linux which we will ship but a codepath which we will support as first
> class for other purposes. So please keep it green! If it goes red, the
> same rules that apply to any other builder on the front page will
> apply.


Now available at  :
http://build.chromium.org/buildbot/waterfall/waterfall?builder=Linux%20Builder%20(Toolkit%20dbg)


>
>
> To build,
>
> export GYP_DEFINES="toolkit_views=1"
> gclient runhooks --force
> hammer chrome
>
> Thanks,
>
> -Ben
>
> >
>

--~--~-~--~~~---~--~~
Chromium Developers mailing list: chromium-dev@googlegroups.com 
View archives, change email options, or unsubscribe: 
http://groups.google.com/group/chromium-dev
-~--~~~~--~~--~--~---

[chromium-dev] No more buildbot auto-refresh - Update your bookmark!

[Nicolas Sylvain]

Hi,
Last week a lot of people on this list said that they like the idea of
disabling the auto-reload for
our buildbot waterfall and console view. This is now implemented.

If you do want to keep the page auto-refreshing, please update your
bookmark. just add the
query parameter "reload=X", where X is the number of second to wait between
each reloads.

X used to be 90. Depending on your needs, you can increase or decrease this
number.

Examples:

console view:
http://build.chromium.org/buildbot/waterfall/console?reload=180
waterfall :
http://build.chromium.org/buildbot/waterfall/waterfall?reload=180

For sheriffs:
http://build.chromium.org/buildbot/waterfall/waterfall?failures=1&reload=30

Thanks

Nicolas

--~--~-~--~~~---~--~~
Chromium Developers mailing list: chromium-dev@googlegroups.com 
View archives, change email options, or unsubscribe: 
http://groups.google.com/group/chromium-dev
-~--~~~~--~~--~--~---

[chromium-dev] Re: Unable to download Chrome Source Code

[Nicolas Sylvain]

On Sat, Jul 18, 2009 at 1:56 AM, Pradeep  wrote:

>
> Hi All,
>
> From past several days I am trying to download the source code, but
> for some reason the download quits after completion of 17 - 18 MB.


what is the URL of the file you are downloading?

Have you tried other browsers? Does it work in firefox?

Thanks

Nicolas


>
>
> I need assistance in this matter.
>
> Thanks & Regards
> Pradeep Chandra
>
> >
>

--~--~-~--~~~---~--~~
Chromium Developers mailing list: chromium-dev@googlegroups.com 
View archives, change email options, or unsubscribe: 
http://groups.google.com/group/chromium-dev
-~--~~~~--~~--~--~---

[chromium-dev] Re: Code coverage

[Nicolas Sylvain]

+cc randall, the coverage guru

On Fri, Jul 17, 2009 at 10:20 AM, Scott Violet  wrote:

> Thanks!
>
> Personally I think the overall numbers are the most important, rather
> than what they are for a build. Any reason we don't drop you into the
> subdirectory view from the get go?
>
>  -Scott
>
> On Fri, Jul 17, 2009 at 9:37 AM, Nicolas Sylvain
> wrote:
> >
> >
> > On Fri, Jul 17, 2009 at 8:39 AM, Scott Violet  wrote:
> >>
> >> I'm interested in seeing code coverage. We have a code coverage page
> >> (http://build.chromium.org/buildbot/perf/dashboard/coverage.html), but
> >> I can't make heads or tails of this. Is there a page that shows
> >> coverage per file, so that we know what areas to add tests for?
> >
> > I agree that this is not easy to find. Click on the "view coverage" link
> > on the coverage bot on the experimental (FYI) waterfall.
> > It brings you to the detailed result for a given build.
> >
> http://build.chromium.org/buildbot/coverage/linux-debug/20957/CHROMIUM/index.html
> > Nicolas
> >
> >>
> >> Thanks,
> >>
> >>  -Scott
> >>
> >> > >>
> >
> >
>

--~--~-~--~~~---~--~~
Chromium Developers mailing list: chromium-dev@googlegroups.com 
View archives, change email options, or unsubscribe: 
http://groups.google.com/group/chromium-dev
-~--~~~~--~~--~--~---

[chromium-dev] Re: Code coverage

[Nicolas Sylvain]

On Fri, Jul 17, 2009 at 8:39 AM, Scott Violet  wrote:

>
> I'm interested in seeing code coverage. We have a code coverage page
> (http://build.chromium.org/buildbot/perf/dashboard/coverage.html), but
> I can't make heads or tails of this. Is there a page that shows
> coverage per file, so that we know what areas to add tests for?


I agree that this is not easy to find. Click on the "view coverage" link
on the coverage bot on the experimental (FYI) waterfall.

It brings you to the detailed result for a given build.

http://build.chromium.org/buildbot/coverage/linux-debug/20957/CHROMIUM/index.html


Nicolas


>
>
> Thanks,
>
>  -Scott
>
> >
>

--~--~-~--~~~---~--~~
Chromium Developers mailing list: chromium-dev@googlegroups.com 
View archives, change email options, or unsubscribe: 
http://groups.google.com/group/chromium-dev
-~--~~~~--~~--~--~---

[chromium-dev] Re: Mac Valgrind Bots

[Nicolas Sylvain]

On Fri, Jul 17, 2009 at 7:40 AM, Dan Kegel  wrote:

> On Fri, Jul 17, 2009 at 2:34 PM, Thomas Van Lenten
> wrote:
> >> We're already running at -O1 on Linux, so there isn't much improvement
> >> left to be had, I suspect.   It might be worth it, dunno.
> >
> > There is a lot of code cut out by being NDEBUG instead of DEBUG, that's
> > where I think the mac is getting its speed up.  Yesterdays mac valgrind
> runs
> > on the main waterfall were actually -O0 while we waited for the v8 issue
> to
> > be fixed and they saw that speed up.
>
> FWIW, I saw a noticable speedup from goint to -O1.  Glad to
> hear there's more to be had.  I guess I'll try it.


On Windows we also use a release build without optimization. Purify can
barely run with a debug build.

Nicolas

--~--~-~--~~~---~--~~
Chromium Developers mailing list: chromium-dev@googlegroups.com 
View archives, change email options, or unsubscribe: 
http://groups.google.com/group/chromium-dev
-~--~~~~--~~--~--~---

[chromium-dev] Re: Mac Valgrind Bots

[Nicolas Sylvain]

On Fri, Jul 17, 2009 at 6:48 AM, Thomas Van Lenten wrote:

> We've finished the shuffle of the Mac Valgrind bots over to release builds.
>  It looks like some of the slower tests now run 2x as fast, so the cycles
> types seem to be a lot better (fewer cls per cycle).


Great! Should we do the same for linux too?

Nicolas


>
>  There are two bots on the main waterfall, and a few more on the fyi
> waterfall. Nirnimesh & Stuart have been doing a great job working through
> the data on the FYI so we can get those green and move them over where folks
> will see them.
>
> With the faster cycle times, please make sure you help keep an eye on them
> as cls land.  We're still finding some leaks in the System Libs that we have
> to add suppressions for, but we're also find valid leaks in the core code
> too.
>
> TVL
>
>
> >
>

--~--~-~--~~~---~--~~
Chromium Developers mailing list: chromium-dev@googlegroups.com 
View archives, change email options, or unsubscribe: 
http://groups.google.com/group/chromium-dev
-~--~~~~--~~--~--~---

[chromium-dev] Re: Purify - instrumentation fails on XP Unit (purify) bot

[Nicolas Sylvain]

On Thu, Jul 16, 2009 at 1:04 PM, Sverrir Á. Berg wrote:

>
> http://build.chromium.org/buildbot/waterfall/waterfall?builder=XP%20Unit%20(purify)


I cleaned up the machine and restarted it. Sometimes it's enough to make it
work.
Nicolas


> Any insights would be appreciated...
>
> Sverrir (sheriff)
>
> >
>

--~--~-~--~~~---~--~~
Chromium Developers mailing list: chromium-dev@googlegroups.com 
View archives, change email options, or unsubscribe: 
http://groups.google.com/group/chromium-dev
-~--~~~~--~~--~--~---

[chromium-dev] Re: Buildbot performance issue.

[Nicolas Sylvain]

On Tue, Jul 14, 2009 at 2:25 PM, Evan Martin  wrote:

> On Tue, Jul 14, 2009 at 2:18 PM, Nicolas Sylvain
> wrote:
> >   The underlying problem with buildbot is the database format, which is
> just
> > hundred of
> > thousand of files on the harddrive, with no "seek" capability, and the
> fact
> > that the
> > webserver itself is single threaded.
> >   We currently have 63 slaves on our main waterfall. I think this is too
> > many for what
> > buildbot can really support. We would ideally need to split it.
>
> Can we get upstream buildbot devs involved in this discussion?  It
> seems they ought to be able to scale better than they have.


I talked to them a little. They do plan some of that for their 1.0 release,
but they
said that it was not on the radar until then.


>
>
> It seems to me a caching layer that only ever hit the "backend" every
> ten seconds would allow it ten seconds to grind through its
> computations, which should be more than sufficient, without any extra
> splitting up required.  That is, we should (a) fix the proxy and (b)
> make every use the proxy.


That makes a lot of sense. I agree that we should fix the proxy, and
make more people use it. Some direct buildbot access would still be
required internally to force a build and stuff like that.


>
> >   Q3: What kind of auto-refresh do we need?
> >   We used to be at 60 secs for a long time, and I changed it a couple of
> > weeks ago to 90 secs.
> > No one complained, so I guess this is good. Should we go even higher than
> > that?
>
> I personally hate auto-refresh.  We should make it opt-in since I
> doubt most users need it and it adds load.  I expect many people
> (myself included) leave the buildbot page open in a background tab and
> have it continually refreshing despite not looking at it.
>
> (My other common use case: the tree is red, I start scrolling down to
> see what's gone wrong, and then the page refreshes out from under me
> and I lose where I was looking.)


Yep, a lot of people told me the same thing. Some other told me they would
like
a faster reload.  Now i'm tempted to let the user control the reload and not
give
a default one.



>
> > - Get a better machine. It's already running on a dedicated dual quad
> core
> > nehalem server
> >   with 24gb of RAM and 15k rpm drives.
>
> This is absurdly powerful!  It should have all the data necessary to
> generate the page in RAM already, no need for even touching the disks
> (?).


Yeah, i'm not too sure how to debug this. When I strace the process I only
see file reads, scrolling, like crazy, all the time.

Nicolas

--~--~-~--~~~---~--~~
Chromium Developers mailing list: chromium-dev@googlegroups.com 
View archives, change email options, or unsubscribe: 
http://groups.google.com/group/chromium-dev
-~--~~~~--~~--~--~---

[chromium-dev] Buildbot performance issue.

[Nicolas Sylvain]

Hello,
  We reached a point where we have too many build slaves and too many users
to get good performance/latency out of our current buildbot waterfall and
console view page.

  We've been tweaking the page a lot lately to make it faster to load, but
the number
of new slaves every day, and the number of new users make it going slower
faster
than we can address the problem. This morning buildbot was completely
unresponsive
because it was not able to keep up with the demand.

  To make it come back to life, I disabled two features, which account for
about 50%
of all traffics : The buildbot chrome extensions, and the top 3 overview
bars at the top
of the waterfall.  This will be able to make it stay online for a while, but
this is not ideal.
It's time to think of a better solution.

  The underlying problem with buildbot is the database format, which is just
hundred of
thousand of files on the harddrive, with no "seek" capability, and the fact
that the
webserver itself is single threaded.

  We currently have 63 slaves on our main waterfall. I think this is too
many for what
buildbot can really support. We would ideally need to split it.

  Q1: Want kind of split would you prefer? mac/linux/windows or
chromium/webkit/modules
or full/windows/mac/linux/memory, etc?

  the main buildbot page would most likely become a bunch of iframe to
display all the
slaves at the same time. The console view integration might be a little bit
less nice. If there is
anyone with web devel experience who wants to help, we could modify the
current waterfall
to fetch only json data from the buildbot, and merge them together, client
side, to get a
combined view.

  Q2: How many changes do we need to display on the console view?

  We are currently displaying the last 50 changes. Which is usually
half-day. If people don't
mind about this, we could scale back to 30. This would make it a little
faster to load.

  Q3: What kind of auto-refresh do we need?

  We used to be at 60 secs for a long time, and I changed it a couple of
weeks ago to 90 secs.
No one complained, so I guess this is good. Should we go even higher than
that?

  Q4: How much build history do we need?

  Right now stdio log are kept for 3 weeks and build results (green, red)
are kept for 1 month. Older
build results are archived but can't be accessed directly by the buildbot.

  If you have any other suggestions, please let me know!

Some things that we can't do:
- Get a better machine. It's already running on a dedicated dual quad core
nehalem server
  with 24gb of RAM and 15k rpm drives.
- Change buildbot to use non-single threaded web server. This is way too
much involved.

*WHAT I NEED YOU HELP WITH :*

1. No more scraping of the waterfall please! If you need to crawl the logs,
let me know and I can
run your script on the database directly.
2. If you know about apache mod_cache / mod_proxy, and wants to help, please
let me know.
build.chromium.org is a proxied cache of the real buildbot server, and
the cache does not work
well. This contribute to another got 25/30% of the overall load on the
buildbot.

Thanks!

Nicolas

--~--~-~--~~~---~--~~
Chromium Developers mailing list: chromium-dev@googlegroups.com 
View archives, change email options, or unsubscribe: 
http://groups.google.com/group/chromium-dev
-~--~~~~--~~--~--~---

[chromium-dev] Re: [chromium-checkins] r20238 - trunk/tools/buildbot/config/master.chromium.fyi

[Nicolas Sylvain]

linux2 there is the platform, as in linux with kernel 2.*.
>>> import sys
>>> print sys.platform
linux2

Nicolas


On Thu, Jul 9, 2009 at 12:34 AM, PhistucK  wrote:

> Should this not be changed also?" m_webkit_linux_v8_latest =
> chromium_factory.ChromiumFactory('src/build',
> 'linux2')"
> (I also marked it in the changes below)
>
> ☆PhistucK
>
>
>
> On Thu, Jul 9, 2009 at 06:24,  wrote:
>
>>
>> Author: nsylv...@chromium.org
>> Date: Wed Jul  8 20:24:38 2009
>> New Revision: 20238
>>
>> Log:
>> Rename linux2 to chromeos, and add a new slave for marshall's chromium
>> embedded project.
>>
>> Review URL: http://codereview.chromium.org/155268
>>
>> Modified:
>>   trunk/tools/buildbot/config/master.chromium.fyi/master.cfg
>>
>> Modified: trunk/tools/buildbot/config/master.chromium.fyi/master.cfg
>>
>> ==
>> --- trunk/tools/buildbot/config/master.chromium.fyi/master.cfg  (original)
>> +++ trunk/tools/buildbot/config/master.chromium.fyi/master.cfg  Wed Jul  8
>> 20:24:38 2009
>> @@ -164,12 +164,13 @@
>>  'Webkit Linux (valgrind)',
>>  'Webkit Linux (valgrind layout)',
>>  'Linux Builder (Toolkit dbg)',
>> - 'Linux Builder (linux2)',
>> + 'Linux Builder (ChromeOS)',
>>  'XP Coverage (dbg)',
>>  'Mac Coverage (dbg)',
>>  'Linux Coverage (dbg)',
>> 'Google Chrome XP',
>> 'Google Chrome Linux',
>> +'Chromium Embedded',
>>  ])
>>
>>  # Scheduler to trigger when v8 bleeding edge is updated.'
>> @@ -222,6 +223,7 @@
>>  m_webkit_mac_v8_latest = chromium_factory.ChromiumFactory('src/build',
>> 'darwin')
>> *  m_webkit_linux_v8_latest = chromium_factory.**
>> ChromiumFactory('src/build',
>> 'linux2')*
>> +m_win_cef = chromium_factory.ChromiumFactory('src/cef', 'win32')
>>
>>  # The identifier of the factory is the build configuration. If two
>> factories
>>  # are using the same build configuration, they should have the same
>> identifier.
>> @@ -387,12 +389,12 @@
>> tests=[],
>> properties={'gclient_env': { 'GYP_DEFINES':'toolkit_views=1'}})
>>
>> -f_chromium_rel_linux_linux2 = m_linux.ChromiumFactory(
>> -'chromium-rel-linux-linux2',
>> +f_chromium_rel_linux_chromeos = m_linux.ChromiumFactory(
>> +'chromium-rel-linux-chromeos',
>> tests=[],
>> options=['chrome'],
>> properties={'archive_build': True,
>> -'gclient_env': { 'GYP_DEFINES':'linux2=1
>> branding=Chrome'}})
>> +'gclient_env': { 'GYP_DEFINES':'chromeos=1
>> branding=Chrome'}})
>>
>>  f_google_chrome_rel_xp = m_win.ChromiumFactory(
>> 'google-chrome-rel-xp',
>> @@ -410,6 +412,9 @@
>> tests=['ui', 'unit'],
>> properties={'gclient_env': { 'GYP_DEFINES' : "branding=Chrome"}})
>>
>> +f_chromium_rel_xp_cef = m_win_cef.CEFFactory(
>> +'chromium-rel-cef', tests=[])
>> +
>>
>>  #
>> 
>>  # BUILDER DEFINITIONS
>> @@ -603,10 +608,10 @@
>>   'category': '4linux|builders_compile',
>>  }
>>
>> -b_chromium_rel_linux_linux2 = {'name': 'Linux Builder (linux2)',
>> +b_chromium_rel_linux_chromeos = {'name': 'Linux Builder (ChromeOS)',
>>   'slavename': 'codf189',
>> -  'builddir': 'chromium-rel-linux-linux2',
>> -  'factory': f_chromium_rel_linux_linux2,
>> +  'builddir': 'chromium-rel-linux-chromeos',
>> +  'factory': f_chromium_rel_linux_chromeos,
>>   'category': '4linux|builders_compile',
>>  }
>>
>> @@ -622,6 +627,11 @@
>>   'factory': f_google_chrome_rel_linux,
>>  }
>>
>> +b_chromium_rel_xp_cef = {'name': 'Chromium Embedded',
>> +  'slavename': 'codf205',
>> +  'builddir': 'chromium-rel-xp-cef',
>> +  'factory': f_chromium_rel_xp_cef,
>> +}
>>
>>  c['builders'] = [
>>   b_google_chrome_rel_xp,
>> @@ -650,10 +660,11 @@
>>   b_webkit_dbg_linux_valgrind,
>>   b_webkit_dbg_linux_valgrind_layout,
>>   b_chromium_dbg_linux_toolkit,
>> -  b_chromium_rel_linux_linux2,
>> +  b_chromium_rel_linux_chromeos,
>>   b_coverage_dbg_mac,
>>   b_coverage_dbg_linux,
>>   b_coverage_dbg_xp,
>> +  b_chromium_rel_xp_cef,
>>  ]
>>
>>  ### STATUS TARGETS
>>
>> >>
>>
>

--~--~-~--~~~---~--~~
Chromium Developers mailing list: chromium-dev@googlegroups.com 
View archives, change email options, or unsubscribe: 
http://groups.google.com/group/chromium-dev
-~--~~~~--~~--~--~---

[chromium-dev] Re: running browser_tests on more bots

[Nicolas Sylvain]

On Wed, Jul 8, 2009 at 9:42 AM, Paweł Hajdan Jr. wrote:

>
> I noticed that browser_tests run on the "Chromium XP" buildbot, but I
> couldn't see them on other official (non-FYI) buildbots, and they
> don't run on trybots. What's blocking us from enabling these tests on
> more buildbots?


Hmmm. They do run on try bots. They also do run in debug and release on both
vista and xp too on the main watefall.

Not sure where you would like to see them. (Except maybe linux and mac...
and I need to work on that this week).

Nicolas


>
>
> >
>

--~--~-~--~~~---~--~~
Chromium Developers mailing list: chromium-dev@googlegroups.com 
View archives, change email options, or unsubscribe: 
http://groups.google.com/group/chromium-dev
-~--~~~~--~~--~--~---

[chromium-dev] Re: Clobber third_party/ffmpeg/binaries before syncing

[Nicolas Sylvain]

On Tue, Jul 7, 2009 at 9:05 AM, Erik Kay  wrote:

> With a gclient sync this morning, it seems very confused about
> ffmpeg/binaries.  First the sync failed saying that ffmpeg/binaries was from
> a different repository.  Then I removed the directory and did a sync where
> it pulled down ffmpeg/binaries again and told me that ffmpeg/binaries was no
> longer part of the client and I should delete it.  It's now telling me this
> on ever sync regardless of whether I delete it or not.


You also need to remove the line from .gclient_entries

Nicolas


>
> Erik
>
>
> On Mon, Jul 6, 2009 at 2:04 PM, Evan Martin  wrote:
>
>>
>> On Mon, Jul 6, 2009 at 2:02 PM, Andrew Scherkus
>> wrote:
>> > On Mon, Jul 6, 2009 at 1:54 PM, Evan Martin  wrote:
>> >>
>> >> third_party/ffmpeg/binaries is an extra 6mb of download for our source
>> >> checkout, even on platforms that can't use these binaries.  If you
>> >> multiply by 3 for Mac/Linux it'll become 18mb.
>> >>
>> >> In the past we've used conditional DEPS entries to pull these
>> >> depending on platform.  (See the references to cygwin in src/DEPS, for
>> >> example.)  Does that make sense here?
>> >
>> > Makes sense to me.  I guess I'd have to move them outside of /src
>> though...
>> > maybe to /deps
>>
>> Yes, that is what I was suggesting.
>>
>>
>>
>
> >
>

--~--~-~--~~~---~--~~
Chromium Developers mailing list: chromium-dev@googlegroups.com 
View archives, change email options, or unsubscribe: 
http://groups.google.com/group/chromium-dev
-~--~~~~--~~--~--~---

[chromium-dev] Re: Rev 19669 and WebFrame::GetPrintPageShrink()

[Nicolas Sylvain]

On Tue, Jul 7, 2009 at 7:25 AM, Marc-Antoine Ruel wrote:

> On Mon, Jul 6, 2009 at 6:28 PM, Marshall Greenblatt <
> magreenbl...@gmail.com> wrote:
>
>> On Mon, Jul 6, 2009 at 6:09 PM, Darin Fisher  wrote:
>>
>>> I'm not sure what would be best.  I was just describing the problem.  It
>>> might help to setup a buildbot for chromiumembedded on the fyi page:
>>> http://build.chromium.org/buildbot/waterfall.fyi/waterfall
>>>
>>
>> That sounds like a good first step.  Who should I talk to about getting
>> this set up?
>>
>
> Me or Nicolas. We could probably share it on a underused slave. Do you have
> an OS preference?
>

We talked offline, and I'm planning to set it up on a unused windows slave.

Nicolas

--~--~-~--~~~---~--~~
Chromium Developers mailing list: chromium-dev@googlegroups.com 
View archives, change email options, or unsubscribe: 
http://groups.google.com/group/chromium-dev
-~--~~~~--~~--~--~---

[chromium-dev] Re: Trac

[Nicolas Sylvain]

On Wed, Jul 1, 2009 at 7:40 PM, Jeremy Orlow  wrote:

> Can I do something like this with ViewVC?
> http://trac.webkit.org/changeset?new=45...@trunk&old=45...@trunk
>  Also,
> is there any reason we can't implement multiple tools?  I assume the upkeep
> isn't that high.
>

There is no reason. We could implement both, if it's easy to install. (i.e.
no sql backend of stuff like that. Viewvc can read the subversion repo
directly and it's just a single python file).

If you have experience with installing trac and want to help, you can ping
me and we can see how hard it would be. Otherwise you can file a Feature
request and assign it to bevc and I and we'll see what we can do.

Nicolas


>
>
> On Wed, Jul 1, 2009 at 7:31 PM, Peter Kasting wrote:
>
>> On Wed, Jul 1, 2009 at 7:27 PM, Jeremy Orlow  wrote:
>>
>>> Is there any reason we use ViewVC rather than something like trac (for
>>> example, trac.webkit.org)?  I wouldn't exactly call Trac amazing, but
>>> there are a few cool features that I definitely missin ViewVC.
>>
>>
>> I sincerely hope we do not switch from ViewVC to Trac, which has a far
>> more annoying interface for viewing source IMO (especially version
>> annotation UI).  I am sad every time I have to view WebKit source (which is
>> frequently).
>>
>> Now if we could just turn off the pagination in ViewVC (why was that ever
>> added?!)...
>>
>> PK
>>
>
>
> >
>

--~--~-~--~~~---~--~~
Chromium Developers mailing list: chromium-dev@googlegroups.com 
View archives, change email options, or unsubscribe: 
http://groups.google.com/group/chromium-dev
-~--~~~~--~~--~--~---

[chromium-dev] Re: WebKit Sheriffing Documentation Update

[Nicolas Sylvain]

On Fri, Jun 26, 2009 at 12:10 PM, Amanda Walker  wrote:

> On Fri, Jun 26, 2009 at 2:42 PM, Ojan Vafai wrote:
> > For the layout tests steps, why do you we need to build test_shell
> locally,
> > rebaseline and then add the other platforms to tests_fixable? Can't we
> just
> > use the rebaseline tool and point it at the integration bots?
>
> Agreed.  Because of the way baselines are organized, the usual case is
> that of a new test arriving with a baseline for the Mac but not for
> Windows and Linux.  Using the rebaseline tool (or even rework it a
> little so that it knows how to check for new tests) would be a lot
> more useful than having to generate new baselines locally.
>
> > Speaking of the integration bots, should we set one up for Linux as well?
>
> At this point, it would be nice to have one for each platform.


Mac already has one.

I can add a linux webkit-latest bot too.  (Might take a couple of days,
since i'm not sure I have enough hardware.. will check)

Nicolas

>
>
> --Amanda
>

--~--~-~--~~~---~--~~
Chromium Developers mailing list: chromium-dev@googlegroups.com 
View archives, change email options, or unsubscribe: 
http://groups.google.com/group/chromium-dev
-~--~~~~--~~--~--~---

[chromium-dev] New waterfall mode for sheriffs

[Nicolas Sylvain]

Hello,
The console view is not the best for sheriffs since it does not show in
details the current state of the tree. The waterfall has always been better
for that.

But since our waterfall is really big, it can be hard to scroll all the time
and keep track of all the failures.

For that, I modified the current waterfall to have a mode to hide all the
green slaves. Only the slaves with failures are shown.

Link: http://build.chromium.org/buildbot/waterfall/waterfall?failures=1

This should help sheriffs to quickly see what builders they need to be
monitoring, and remind them that some of them are red, even though
they are usually completely
hidden on the far right of the waterfall.

A slave is present on this waterfall if :
- The last finished build was red.
- A step of the build in progress is red (This will eventually turn the
build red)
- The slave is offline.

Let me know if you have any comments/suggestions about this.

Thanks

Nicolas

--~--~-~--~~~---~--~~
Chromium Developers mailing list: chromium-dev@googlegroups.com 
View archives, change email options, or unsubscribe: 
http://groups.google.com/group/chromium-dev
-~--~~~~--~~--~--~---

[chromium-dev] Re: Linux buildbots, which locale?

[Nicolas Sylvain]

On Thu, Jun 25, 2009 at 7:03 AM, Evan Martin  wrote:

> On Thu, Jun 25, 2009 at 6:27 AM, Nicolas Sylvain
> wrote:
> >
> >
> > On Thu, Jun 25, 2009 at 5:27 AM, Dean McNamee 
> wrote:
> >>
> >> I just took a look at what was happening on one of the Linux
> >> buildbots.  It looks like we completely clear the env and just
> >> explicitly set a few things.  We need to add LANG=en_US.UTF-8 to that
> >> list of things.  Any takers?
> >
> > I can add LANG to the list of whitelisted env var to keep.  Would that
> work?
>
> If any of our tests depend on a particular locale, we should have them
> set that locale first.
>  man setlocale   (and man 7 locale)
> Be sure to reset the locale in the tear-down.
>

I made the bot at least keep the current lang, but I agree that it should be
set by the test if it needs to be something specific.

Nicolas

--~--~-~--~~~---~--~~
Chromium Developers mailing list: chromium-dev@googlegroups.com 
View archives, change email options, or unsubscribe: 
http://groups.google.com/group/chromium-dev
-~--~~~~--~~--~--~---

[chromium-dev] Re: Linux buildbots, which locale?

[Nicolas Sylvain]

On Thu, Jun 25, 2009 at 5:27 AM, Dean McNamee  wrote:

>
> I just took a look at what was happening on one of the Linux
> buildbots.  It looks like we completely clear the env and just
> explicitly set a few things.  We need to add LANG=en_US.UTF-8 to that
> list of things.  Any takers?


I can add LANG to the list of whitelisted env var to keep.  Would that work?

Nicolas

>
>
> Thanks
>
> On Thu, Jun 25, 2009 at 2:12 PM, Dean McNamee wrote:
> >
> http://build.chromium.org/buildbot/waterfall/builders/Modules%20Linux%20(dbg)/builds/9486/steps/base_unittests/logs/stdio
> >
> http://build.chromium.org/buildbot/waterfall/builders/Modules%20Linux%20(dbg)/builds/9486/steps/net_unittests/logs/stdio
> >
> > I just committed a change that converted our
> > sys_string_conversion_linux from using ICU UTF-8 (assuming that the
> > system locale was always UTF-8) to calling the system APIs that will
> > vary depending on the locale.  I believe this is technically what we
> > want, we want these APIs to do the conversion to whatever locale your
> > system is set to.
> >
> > On my machine these all pass fine.  I don't know what locale we have
> > set on the buildbots, but it all seems to fail.  It is also kind of
> > curious that some of our net/file tests depend on locale, but I
> > suppose that makes sense.
> >
> > For now I'll pull out my changes until we can get the buildbots on a
> > utf-8 locale (like en_US.UTF-8).
> >
> > -- dean
> >
>
> >
>

--~--~-~--~~~---~--~~
Chromium Developers mailing list: chromium-dev@googlegroups.com 
View archives, change email options, or unsubscribe: 
http://groups.google.com/group/chromium-dev
-~--~~~~--~~--~--~---

[chromium-dev] Re: Is it possible to do a gclient sync without running hooks?

[Nicolas Sylvain]

On Tue, Jun 23, 2009 at 5:53 PM, Daniel Cowx  wrote:

>
> When I run "gclient sync", it automatically runs the hooks; which
> causes various non-versioned files to get generated within my tree
> (most notably *.vcproj and *.sln files, but there may be others). I'd
> like to be able to do a sync *without* generating any files (i.e. so
> that if I do a "svn status" immediately after a "gclient sync", I see
> a pristine unmodified tree). Short of going into src/DEPS and
> commenting out the "hooks" section, can this be done?


I don't think we have this capability at this time.

But in theory if you "gclient sync ; svn status" you should not see
anything, since all the
generated files should be in the svn:ignore. If they are not, then someone
forgot to add them,
and we should fix that.

Nicolas


> >
>

--~--~-~--~~~---~--~~
Chromium Developers mailing list: chromium-dev@googlegroups.com 
View archives, change email options, or unsubscribe: 
http://groups.google.com/group/chromium-dev
-~--~~~~--~~--~--~---

[chromium-dev] Re: Conventions and patterns for multi-platform development

[Nicolas Sylvain]

On Fri, Jun 19, 2009 at 11:15 AM, Mike Pinkerton wrote:

> On Fri, Jun 19, 2009 at 2:11 PM, Peter Kasting wrote:
> > On Fri, Jun 19, 2009 at 11:07 AM, Nicolas Sylvain  >
> > wrote:
> >>
> >> We should say that we prefer  "#if defined(OS_WIN)" over "#ifdef
> OS_WIN".
> >
> > In general, you should always prefer "#if defined(FOO)" over "#ifdef
> FOO",
> > e.g. because you can add "|| defined(BAR)" to the former later.
> > PK
>
> This is already documented in our Chromium coding style guide. Do we
> need to replicate it?


Oops. It was not there when I look at the coding style a long time ago..
Nevermind!

Thanks

Nicolas


>
>
> --
> Mike Pinkerton
> Mac Weenie
> pinker...@google.com
>

--~--~-~--~~~---~--~~
Chromium Developers mailing list: chromium-dev@googlegroups.com 
View archives, change email options, or unsubscribe: 
http://groups.google.com/group/chromium-dev
-~--~~~~--~~--~--~---

[chromium-dev] Re: Conventions and patterns for multi-platform development

[Nicolas Sylvain]

On Fri, Jun 19, 2009 at 9:42 AM, Brett Wilson  wrote:

>
> Our team has had somewhat of an ad-hoc approach to organizing code
> that's different across platforms. In many cases our approach has been
> quite good. In others, less so, and there have also been questions
> about what the preferred method for writing a certain component in a
> cross-platform way.
>
> Last night Ben and I wrote a document that tries to clarify this. It's
> a combination of what we're doing now that works well, and what we
> probably should be doing:
>
> http://dev.chromium.org/developers/design-documents/conventions-and-patterns-for-multi-platform-development
> This is linked to from the "Engineering design documents" page.
>
> If you're starting a new component or reorganizing an existing one,
> try to follow these guidelines. It can't possibly cover every case, so
> you'll have to use your best judgement.
>
> Feedback from people who have done a lot of this stuff would be great.
> Ideally it would be easy to follow and cover most cases for everybody.

I have a feedback from someone who did not do a lot of this stuff.

We should say that we prefer  "#if defined(OS_WIN)" over "#ifdef OS_WIN". I
always ask myself before doing it, and get confused by seeing the two used
in our code base (and even in the same file, like canvas.h)

Nicolas


>
>
> Thanks!
> Brett
>
> >
>

--~--~-~--~~~---~--~~
Chromium Developers mailing list: chromium-dev@googlegroups.com 
View archives, change email options, or unsubscribe: 
http://groups.google.com/group/chromium-dev
-~--~~~~--~~--~--~---

[chromium-dev] Re: Viewvc on src.chromium.org is down

[Nicolas Sylvain]

back online
thanks

Nicolas


On Thu, Jun 18, 2009 at 10:56 AM, Nicolas Sylvain wrote:

> Hello,
> we've had a couple of problems with viewvc on src.chromium.org in the last
> day, so I turned it off for now.
>
> I'll try to fix it in the next hours.
>
> in the mean time you can still use src.chromium.org to fetch your files,
> and http://src.chromium.org/svn to browse the repo.
>
> Thanks
>
> Nicolas
>

--~--~-~--~~~---~--~~
Chromium Developers mailing list: chromium-dev@googlegroups.com 
View archives, change email options, or unsubscribe: 
http://groups.google.com/group/chromium-dev
-~--~~~~--~~--~--~---

[chromium-dev] Viewvc on src.chromium.org is down

[Nicolas Sylvain]

Hello,
we've had a couple of problems with viewvc on src.chromium.org in the last
day, so I turned it off for now.

I'll try to fix it in the next hours.

in the mean time you can still use src.chromium.org to fetch your files, and
http://src.chromium.org/svn to browse the repo.

Thanks

Nicolas

--~--~-~--~~~---~--~~
Chromium Developers mailing list: chromium-dev@googlegroups.com 
View archives, change email options, or unsubscribe: 
http://groups.google.com/group/chromium-dev
-~--~~~~--~~--~--~---

[chromium-dev] Re: What's the TabRestoreUITest bug 1215881?

[Nicolas Sylvain]

I filed this bug with this comment:--
TabRestoreUITest.RestoreToDifferentWindow fails on win2k debug. I disabled
it.

This is not reproducible outside the buildbot environment.

The problem seems to be that chrome cannot access a font. I was not able to
determine what the font was.
---

Later on I fixed it, but forgot to remove the comment.

This bug was only for debug mode, it should not matter for release mode.

Nicolas


On Wed, Jun 17, 2009 at 2:31 PM, pi  wrote:

>
> I'm trying to find out if it's possible to patch chrome to run on my
> old windows 2000 machine.
> When I debugging chrome 2.0.172.28, the
> Browser::CreateTabContentsForURL sometimes will raise exception.
> According to those comments in the source files,  the TabUI indeed has
> some incompatibility on windows 2000.
> What's it? Can it be overcomed?
> The internal bug reports may be helpful.
>
> Dan Kegel wrote:
> > On Wed, Jun 17, 2009 at 1:07 PM, pi wrote:
> > > In the latest revision 18442,  there are comments such as "Bug
> > > 1230446", "Bug 1204135".
> > > Is there any means to view these internal bug reports?
> >
> > Not directly.  What are you trying to do?
> >
>

--~--~-~--~~~---~--~~
Chromium Developers mailing list: chromium-dev@googlegroups.com 
View archives, change email options, or unsubscribe: 
http://groups.google.com/group/chromium-dev
-~--~~~~--~~--~--~---

[chromium-dev] Re: FYI: Layout test failures for and if you don't have FFmpeg dlls

[Nicolas Sylvain]

On Fri, Jun 12, 2009 at 4:43 PM, Alpha Lam  wrote:

> (This only applies to Windows, if you are not developing on Windows and you
> don't care about layout tests, you can stop reading now.)
>
> Hi all,
> I enabled about 150 layout tests for media today, they are mostly
> sitting inside:
> webkit/data/layout_tests/LayoutTests/media
> webkit/data/layout_tests/LayoutTests/fast/media
> webkit/data/layout_tests/LayoutTests/http/tests/media
>
> test_shell.exe requires the following DLLs to enable  and
> :
> avcodec-52.dll
> avformat-52.dll
> avutil-50.dll
>
> If you don't have all of the above DLLs sitting next to test_shell.exe
> you will expect  and  layout tests to all fail. If you don't
> care about  and  you can safely ignore all these failures.
>

Where are those dlls?

Usually tests should be self contained.  Does that mean we need to update
all the build slaves we create to have these dlls?

Nicolas




>
> Alpha
>
> >
>

--~--~-~--~~~---~--~~
Chromium Developers mailing list: chromium-dev@googlegroups.com 
View archives, change email options, or unsubscribe: 
http://groups.google.com/group/chromium-dev
-~--~~~~--~~--~--~---

[chromium-dev] Re: Can I have sandbox/linux?

[Nicolas Sylvain]

On Fri, Jun 12, 2009 at 2:48 PM, Adam Langley  wrote:

> On Fri, Jun 12, 2009 at 2:40 PM, Nicolas Sylvain
> wrote:
> > In theory sandbox/ should be fetched from DEPS pointing to
> > http://code.google.com/p/rollcage , but since we don't really have a lot
> of
> > customer, we did it the other way around.
> > in general sandbox/ has no chrome specific code, and I would prefer if we
> > keep it that way.   Can your linux sandbox be make generic and work with
> > other apps? (If so we should really promote it part of the rollcage
> > sandbox).
>
> Ok, thanks. I'll put the generic stuff in sandbox/linux and the Chrome
> specific stuff (which will just be canonical AppArmor and SELinux
> configs) in chrome/app/linux.


Thanks

fyi: the chrome specific sandbox code on windows is
in src\chrome\browser\sandbox_policy.cc/.h.

Nicolas

>
>
>
> AGL
>

--~--~-~--~~~---~--~~
Chromium Developers mailing list: chromium-dev@googlegroups.com 
View archives, change email options, or unsubscribe: 
http://groups.google.com/group/chromium-dev
-~--~~~~--~~--~--~---

[chromium-dev] Re: Can I have sandbox/linux?

[Nicolas Sylvain]

On Fri, Jun 12, 2009 at 2:35 PM, Adam Langley  wrote:

>
> sandbox/ is contains the Windows sandbox. However, we'll soon have
> need for somewhere to put Linux sandboxing code. sandbox/linux is what
> I'm using in my branch at the moment, but I'm open to suggestions.


In theory sandbox/ should be fetched from DEPS pointing to
http://code.google.com/p/rollcage , but since we don't really have a lot of
customer, we did it the other way around.

in general sandbox/ has no chrome specific code, and I would prefer if we
keep it that way.   Can your linux sandbox be make generic and work with
other apps? (If so we should really promote it part of the rollcage
sandbox).

If not, then we should try to find a better place.  I don't want to slow you
down though, having the sandbox on linux is more important than naming
conventions, so if you land it there for now, I won't mind.

Nicolas


>
>
>
> Cheers
>
> AGL
>
> >
>

--~--~-~--~~~---~--~~
Chromium Developers mailing list: chromium-dev@googlegroups.com 
View archives, change email options, or unsubscribe: 
http://groups.google.com/group/chromium-dev
-~--~~~~--~~--~--~---

[chromium-dev] Re: No symbols for Google Chrome 2.0.172.30

[Nicolas Sylvain]

Looks like it did not.
I'll investigate why.

thanks

Nicolas


On Wed, Jun 10, 2009 at 1:00 AM, Dean McNamee  wrote:

> Hey Nicolas,
>
> Did these symbols ever get uploaded?
>
> Thanks
>
> On Sun, Jun 7, 2009 at 2:55 AM, yuhong wrote:
> >
> > SYMSRV:
> http://build.chromium.org/buildbot/symsrv/chrome_exe.pdb/0C57BF1B1D3D48
> > 49B3037D4FE9424CCC2/chrome_exe.pdb not found
> > DBGHELP: chrome - no symbols loaded
> > [SYMCHK] MODULE64 Info --
> > [SYMCHK] Struct size: 1672 bytes
> > [SYMCHK] Base: 0x0040
> > [SYMCHK] Image size: 782336 bytes
> > [SYMCHK] Date: 0x4a128615
> > [SYMCHK] Checksum: 0x000c0d3d
> > [SYMCHK] NumSyms: 0
> > [SYMCHK] SymType: SymNone
> > [SYMCHK] ModName: chrome
> > [SYMCHK] ImageName: C:\Users\bob\AppData\Local\Google\Chrome
> > \Application\chrome.
> > exe
> > [SYMCHK] LoadedImage: C:\Users\bob\AppData\Local\Google\Chrome
> > \Application\chrom
> > e.exe
> > [SYMCHK] PDB: ""
> > [SYMCHK] CV: RSDS
> > [SYMCHK] CV DWORD: 0x53445352
> > [SYMCHK] CV Data:  C:\b\slave\chrome-official-2\build\src\chrome
> > \Release\chrome_
> > exe.pdb
> > [SYMCHK] PDB Sig:  0
> > [SYMCHK] PDB7 Sig: {----}
> > [SYMCHK] Age: 0
> > [SYMCHK] PDB Matched:  TRUE
> > [SYMCHK] DBG Matched:  TRUE
> > [SYMCHK] Line nubmers: FALSE
> > [SYMCHK] Global syms:  FALSE
> > [SYMCHK] Type Info:FALSE
> > [SYMCHK] 
> > SymbolCheckVersion  0x0002
> > Result  0x00010001
> > DbgFilename chrome.dbg
> > DbgTimeDateStamp0x
> > DbgSizeOfImage  0x
> > DbgChecksum 0x
> > PdbFilename C:\b\slave\chrome-official-2\build\src\chrome
> > \Release\chrome
> > _exe.pdb
> > PdbSignature{0C57BF1B-1D3D-4849-B303-7D4FE9424CCC}
> > PdbDbiAge   0x0002
> >
> > > >
> >
>

--~--~-~--~~~---~--~~
Chromium Developers mailing list: chromium-dev@googlegroups.com 
View archives, change email options, or unsubscribe: 
http://groups.google.com/group/chromium-dev
-~--~~~~--~~--~--~---