[chromium-dev] UI Jank Task Force Status Update
UI Jank Task Force Status Update Last meeting until 2010. *Status* Carlos - Keep trying to build PGO. Trying to build just a part of Chrome (instrument just the ‘browser’ project) to get around the problem. Chase - Tab switching tests problem resolved; re-enabled. Cause is unclear (interactive perf), bug filed. - Patch for reference build for tab switching tests in the works, will send to John. John - Fixing the whole browser hang with Target’s flyer site. Turned out to be an evangelism issue -- reported upstream. Tony - Profiling renderer startup; most slowness is when renderer is waiting from the browser. Not much to be done here. - Will try to move the visited database loading on the file thread. (Will still block UI thread if it takes a long time to load) - Writes may be on IO thread, will look into moving them too. Evan - Looking at tabstrip on Linux, found an improvement in tab-dragging. - Working on extensions, bugs. -- Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev
[chromium-dev] UI Jank Task Force Status Update
* UI Jank Task Force Status Update The UI Jank Task Force is out to fix UI Jank and slowness in the browser. A list of open Jank bugs is here: http://code.google.com/ p/chromium/issues/list?q=label:Jank (feel free to take one!) Status John - Chasing down browser hang from Target's website (Flash fetching many URLs repeatedly) Carlos - Fighting with PGO build, several steps failing. - A NaCl linking step fails - NaCl team looking into it with VS2008. - Linker crashes. Will need to file a ticket upstream. - Will try to build with LKGR today. - Will give a build to John if he can get it building again. - Darin landed a change to improve how we paint small updates (fast scrolling rects). Chase - Infrastructure changes; changing how we gather perf test outputs. - Has a patch that will fix orange on DHTML page cyclers. Will send the CL to Carlos. Evan - Triaging Linux Jank bugs. Probably not many that are actionable right now, though. Tony - Profiling renderer startup. Faster renderers - faster new tabs. * -- Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev
[chromium-dev] UI Jank Task Force Status Update
*UI Jank Task Force Status Update* The UI Jank Task Force is out to fix UI Jank and slowness in the browser. A list of open Jank bugs is here: http://code.google.com/ p/chromium/issues/list?q=label:Jank (feel free to take one!) *Updates* John - Bug fixing - Last shutdown file Lookup on startup to different thread - Looking at remaining Jank bugs this week Carlos - Sick, OOO last week - Pointer to an API to detect if memory is paged out. Still not sure it is possible given the API, but investigating further. - Ricardo's experiments indicate that backing stores may not be paged out after sleep? - AI: John Glenn to follow up with Ricardo - Another contributor working on PGO, but hasn't been able to get it working either (VS2010 may be better than VS2005.) - AI: Carlos to file a ticket about linker limits (only 5 linkers spawned) - AI: John to run startup perf tests with PGO build. Chase - Tools work, perf tests, reference builds last week. Evan - No jank updates. Tony - Added reference builds for new tab tests. - Investigating thread contention for new tab creation. - Elliot changing how themes are stored on disk, should make theme startup faster (cram all files into one.) -- Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev
[chromium-dev] UI Jank Task Force Status Update
*UI Jank Task Force Status Update* * * *Agenda* * * Excellent progress last week - Spellchecking now in done in the renderer; moved a lot of disk access off I/O thread. - Launching child processes now done in background thread; off of UI / IO threads. - 30% improvement on Linux new tab creation histograms. - Memory purger now available. - First automatic purge candidate: System memory pressure notification? - Hibernate may not be the cause of thrashing. Investigation ongoing. - Tab switching tests now live on Windows bots. - Cookies no longer loading on startup. New investigations - Investigating using a flat file for safebrowsing instead of SQLite (need to recreate bloom filter and watch disk I/O) * * *Goal review* The team reviewed progress towards its current goals: Measure and speed up different browser UI operations: - First input event delta after tab switch + waking up after sleep - Have histogram for input event delta, but don't have it for this specific 'wake' event. - Omnibox responsiveness - No progress. - Tab switching when process is paged out - Added tab switching tests and measurements. - Whiteout duration (for the 95th percentile) -- tab switch paint duration? - Have histograms for whiteout, but it's a red herring. Will keep it around, but not as deterministic as onPaint roundtrip. - Startup - Added child process background thread changes. Need a perf tests for n-page session restore to prevent regressions. - Shutdown - No progress. - New tab creation - Background thread process creation...possibly a big improvement on Linux. - Tab close - May already have histograms, fast shutdown path (onUnload handlers), but didn't measure the before after. Create a Jank bot that simulates a heavily overloaded system for running Jank tests. - Single core perf bots turned into 'jank bots' w/ lower memory. Also added tab switching tests (XP interactive perf.) No Disk I/O on UI, I/O threads. - Added run-time check to prevent regressions. - Moved spellchecking into renderer. - Cookie disk I/O moved out of startup. Choose whether to optimize PGO build for Renderer or Browser. - PGO build working, Carlos is investigating better optimizations. Also added many other UI Jank tweaks as well. -- Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev
[chromium-dev] Re: UI Jank Task Force Status Update
Sorry, two corrections: - The promo cookie was removed from the new tab page startup, but local tests show cookies are still being loaded. Investigation is ongoing. - The disk access run time check is actually a script that uses Windbg to detect these types of accesses. The next step is to turn this into an automated regression check. On Mon, Nov 23, 2009 at 2:58 PM, Glenn Wilson gwil...@chromium.org wrote: *UI Jank Task Force Status Update* * * *Agenda* * * Excellent progress last week - Spellchecking now in done in the renderer; moved a lot of disk access off I/O thread. - Launching child processes now done in background thread; off of UI / IO threads. - 30% improvement on Linux new tab creation histograms. - Memory purger now available. - First automatic purge candidate: System memory pressure notification? - Hibernate may not be the cause of thrashing. Investigation ongoing. - Tab switching tests now live on Windows bots. - Cookies no longer loading on startup. New investigations - Investigating using a flat file for safebrowsing instead of SQLite (need to recreate bloom filter and watch disk I/O) * * *Goal review* The team reviewed progress towards its current goals: Measure and speed up different browser UI operations: - First input event delta after tab switch + waking up after sleep - Have histogram for input event delta, but don't have it for this specific 'wake' event. - Omnibox responsiveness - No progress. - Tab switching when process is paged out - Added tab switching tests and measurements. - Whiteout duration (for the 95th percentile) -- tab switch paint duration? - Have histograms for whiteout, but it's a red herring. Will keep it around, but not as deterministic as onPaint roundtrip. - Startup - Added child process background thread changes. Need a perf tests for n-page session restore to prevent regressions. - Shutdown - No progress. - New tab creation - Background thread process creation...possibly a big improvement on Linux. - Tab close - May already have histograms, fast shutdown path (onUnload handlers), but didn't measure the before after. Create a Jank bot that simulates a heavily overloaded system for running Jank tests. - Single core perf bots turned into 'jank bots' w/ lower memory. Also added tab switching tests (XP interactive perf.) No Disk I/O on UI, I/O threads. - Added run-time check to prevent regressions. - Moved spellchecking into renderer. - Cookie disk I/O moved out of startup. Choose whether to optimize PGO build for Renderer or Browser. - PGO build working, Carlos is investigating better optimizations. Also added many other UI Jank tweaks as well. -- Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev
[chromium-dev] UI Jank Task Force Status Update
*UI Jank Task Force Status Updates* The UI Jank Task Force is out to fix UI Jank and slowness in the browser. A list of open Jank bugs is here: http://code.google.com/p/chromium/issues/list?q=label:Jank (feel free to take one!) *Updates* Chase - Added tab switch paint histogram. - Found a deeper problem on xp perf bots, working with Nicolas to resolve. - Will have tab-switching test results this week. John - Moved launching of renderer process off UI thread. - On POSIX, up to 2s of sleep when killing off renderer process. - Change going out today. - Will work on moving creation of other types of processes off UI thread. Carlos - Built PGO built on a bot with 24GB! - Not measuring any improvement. Training set was opening a bunch of pages (10MB). - Results about PGO's efficacy are inconclusive. - Using time of UI test runs...still investigating a better measurement and better training sets. Evan - Landed patch to move spellchecker to renderer on Windows, wrote it on Mac. - Caused reliability regressions, memory corruption. - Posting Task when creating spellcheck host. Investigating now. Tony - Tracking down crashers for Mstone 4. -- Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev
[chromium-dev] Chrome UI Jank Task Force Status Update
*Chrome UI Jank Task Force Update* The Chrome UI Jank Task Force is out to fix UI Jank and slowness in the browser. A list of open Jank bugs is here: http://code.google.com/p/chromium/issues/list?q=label:Jank (feel free to take one!) *Updates* John - Finished up Mstone-4 bugs - All structures of Refcounted objects private -- cleaned up some things, and found some bugs. - Looked into how long it takes to create a render process from cold start (see multi-process histograms) - Win: 120ms first on average, 50ms subsequently - Linux: 1.2s! (3ms after that) But NTP load time is less than that? Maybe Zygote-related. - Something seems to be off, investigating further. - John to cc agl and Dan K about this issue (they know Zygote) Chase - Cookies on startup (cookies loading from disk for the NTP?). Understanding pipeline webkit - renderer - browser - Tab switching test only measuring whiteout duration, want to expand it to measure more. - Working with John on adding more measurements to this. Tony - Theme.dll merged into chrome.dll (caused issues with icon for the app) -- didn't seem to affect startup time. - Working on mstone-4 crashers. Evan - Landed the spellchecker - renderer move on Linux. This may have made memory tests flaky. - Should be minimal effort to move Win Mac. (Windows doesn't like to open files in the renderer - have to pass a handle) - Working on search engine prepopulate data Linux bugs. --~--~-~--~~~---~--~~ Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev -~--~~~~--~~--~--~---
[chromium-dev] Chrome UI Jank Task Force Status Update
*Chrome UI Jank Task Force Status* The Chrome UI Jank Task Force is out to fix UI Jank and slowness in the browser. A list of open Jank bugs is here: http://code.google.com/p/chromium/issues/list?q=label:Jank (feel free to take one!) *Updates* * * Chase: working on disk access on startup, has a log of all disk accesses happening. - Cookie reads happening on UI thread? - Investigating more, will sync with Brett. - updated the windows reference build. - Enabled new tab warm load perf tests. Carlos: re-writing chrome.exe from scratch (lots of cruft.) - Some reading of DLL before DLL loads - rewrite will eliminate some I/Os. - Need another builder box for PGO on official builds (takes a long time.) - No good way to know the improvement without doing the build. - Need to choose how we want to optimize - can only optimize for one type of process...and some tests won't work. John: done with patch to simplify threading in browser process. - Simplifies moving things to different threads (won't have to hop threads to see if things are still alive.) Tony: killing off the theme.dll, may make cold startup faster. Evan: started moving spellchecker to renderer, lots of work to be done, in progress. --~--~-~--~~~---~--~~ Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev -~--~~~~--~~--~--~---
[chromium-dev] Chrome UI Jank Task Force status update
*Chrome Jank Task Force Status Update* The UI Jank task force is focused on improving UI 'jankiness'/slowness throughout the browser. These are status updates of team members working on the task force. * * * Current Status* Chase - Working on reference build (hasn't been updated in ~7 months) to get our tab switching tests rolling. - Mozilla using Consume tool to load down a system? May want to investigate using the same tool. Carlos - Gardening on Jank bots. John - ImageUnloadHandler work, passing along the work to Nate. - Found that thread 'jumping' could possibly lead to deadlocks w/ UI thread; investigating. - Investigating moving spellchecking to file thread - Evan to move spellchecking to renderer process (estimate: this quarter) Evan - Fixing Chrome 4 bugs. - Tackling moving spellchecking to renderer this quarter. Tony - Fixing Chrome 4 bugs / Webkit merges. - Wrote change to delete unused themes on shutdown. - Slows down shutdown but speeds up startup (+/- 500ms?) - Maybe delete the themes as soon as 'undo' bar goes away? - Need to eliminate shutdown slowdown (John filed a bug on this) Peter - Added more plumbing into memory purger (probably can't purge web database based on architecture SQLite) - Still not plumbed but need investigation: Appcache, Database, Safebrowsing (probably?) - Would probably only purge the bloom filter -- may be expensive to rebuild. - Safebrowsing seems to be hitting the HD pretty hard on wake, not clear why this is happening. (bug filed on this) - Renderer is slow after waking up - affected by safebrowsing? - Do we vacuum the SB SQLite database? (data indicates it may not make a big difference) - Maybe add histograms for whiteout duration / etc. with SB on or off -- may already have this data and we just need to correlate. --~--~-~--~~~---~--~~ Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev -~--~~~~--~~--~--~---
[chromium-dev] Canary bot, now with formatted layout test results
Hi Webkit gardeners, The Windows canary bot now generates formatted layout test results on each build that had unexpected failing layout tests, which means you can quickly see layout test failures, diffs, and upstream baselines without running the tool manually. The goal of this output is to help gardeners assess what may have changed, and how to best handle the failure. For example, here are the failures for build 13359: http://build.chromium.org/buildbot/layout_test_results/webkit-rel-webkit-org/29119/index-13359.html To see the formatted layout test failures for a specific canary build, just click on the layout test results link in the archive webkit test results step, then click on the index-.html file. There's still more work to do, including getting the formatter working properly on the Mac Linux canaries, adding flakiness data, adding a link directly to the waterfall display, and more. Many thanks go to Ojan, Nicolas and all others who endured my many code reviews, and Eric Roman and Dimitri for the original spec and design. Please let me know what you think, and what changes you'd like to see. Glenn --~--~-~--~~~---~--~~ Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev -~--~~~~--~~--~--~---
[chromium-dev] Re: Canary bot, now with formatted layout test results
Yes, possibly. The tool is written explicitly for the Chromium builders'
output, so it would be plausible but non-trivial to generalize it.
It may not be as useful since I don't think they have a concept of an
'upstream' baseline. Those of you who do a lot of work upstream, would it
be worth upstreaming this?
Glenn
On Thu, Oct 15, 2009 at 12:44 PM, Marc-Antoine Ruel mar...@chromium.orgwrote:
Actually, can you package this so it could be upstreamed to webkit.org?
On Thu, Oct 15, 2009 at 1:09 PM, Glenn Wilson gwil...@chromium.orgwrote:
The tool lives in
src/webkit/tools/layout_tests/test_output_formatter{.bat,.sh}, and it can
actually be run against any builder, not just the canaries. Adding better
documentation is on the TODO list :)
(Also, re: adding link to waterfall, I think we can do this -- I'm
investigating now.)
On Thu, Oct 15, 2009 at 10:05 AM, Yaar Schnitman y...@chromium.orgwrote:
Glenn, this tool is amazing! It really helped me yesterday on my 1st
gardening shift.
Re: upstream baselines without running the tool manually
How do I do that?
On Thu, Oct 15, 2009 at 10:01 AM, Jeremy Orlow jor...@chromium.orgwrote:
Christmas came early for anyone working on WebKit! Thanks Glen. :-)
On Thu, Oct 15, 2009 at 9:17 AM, Erik Corry erik.co...@gmail.comwrote:
This makes me very happy!
2009/10/15 Glenn Wilson gwil...@chromium.org:
Hi Webkit gardeners,
The Windows canary bot now generates formatted layout test results on
each
build that had unexpected failing layout tests, which means you can
quickly
see layout test failures, diffs, and upstream baselines without
running the
tool manually. The goal of this output is to help gardeners assess
what may
have changed, and how to best handle the failure.
For example, here are the failures for build 13359:
http://build.chromium.org/buildbot/layout_test_results/webkit-rel-webkit-org/29119/index-13359.html
To see the formatted layout test failures for a specific canary
build, just
click on the layout test results link in the archive webkit test
results
step, then click on the index-.html file.
There's still more work to do, including getting the formatter
working
properly on the Mac Linux canaries, adding flakiness data, adding a
link
directly to the waterfall display, and more. Many thanks go to Ojan,
Nicolas and all others who endured my many code reviews, and Eric
Roman and
Dimitri for the original spec and design.
Please let me know what you think, and what changes you'd like to
see.
Glenn
--
Erik Corry, Software Engineer
Google Denmark ApS. CVR nr. 28 86 69 84
c/o Philip Partners, 7 Vognmagergade, P.O. Box 2227, DK-1018
Copenhagen K, Denmark.
--~--~-~--~~~---~--~~
Chromium Developers mailing list: chromium-dev@googlegroups.com
View archives, change email options, or unsubscribe:
http://groups.google.com/group/chromium-dev
-~--~~~~--~~--~--~---
[chromium-dev] Re: Webkit merges and tree closures
I'll take the action item to remove anyone in the rotation who is not actively working on Webkit / is not a committer. In the past, I've floated the idea in the past of having a Webkit deputy who helps the gardener keep the canary green. I'm not sure if that would help, though. Regards, Glenn On Wed, Sep 23, 2009 at 8:40 AM, David Levin le...@chromium.org wrote: I find that being a WebKit gardener is always dancing on a minefield regardless of my familiarity with the WebKit code base. Look at yesterday for an example. In addition, we have several gardeners who are not actively working on WebKit (amanda@ has 0 WebKit commits, pinkerton@ has a few all in 2008). Lastly, can we add anyone to the rotation who has ideas how WebKit gardeners should/can do their job better? (This will help them in getting more ideas.) Dave On Wed, Sep 23, 2009 at 7:53 AM, Dimitri Glazkov dglaz...@chromium.orgwrote: On Wed, Sep 23, 2009 at 7:46 AM, Mike Pinkerton pinker...@chromium.org wrote: It is hard to be a WebKit gardener if you do not have WebKit commit access. Sometimes the gardener has to commit a quick bustage fix upstream or roll back a fellow Chromium committers change to WebKit. -Darin Correct, it is hard, but many/most of us who are webkit sheriffs/gardeners are not webkit committers (for example, the entire mac group). I don't understand your point. Are you saying that only webkit committers should be on the webkit sheriff rotation? In my experience, better familiarity with WebKit code base is a huge advantage for a gardener. I am almost tempted to say that if you're not actively working on WebKit, being a gardener will be a foreign and dancing-on-a-minefield-type task. :DG --~--~-~--~~~---~--~~ Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev -~--~~~~--~~--~--~---
[chromium-dev] Re: Creating bugs from test_expectations.txt
Yes, the intention is to have something maintainable that gets run as part of the regular testing/merging/etc. process -- which means eventually rewriting in python. I wrote this in Java because we had a java lib for automatically creating bugs in the issue tracker, and I wanted to get it up and running quickly. We should have a python API for doing the same thing. I can easily change the script so it marks DEFERs as P3s. However, currently, the script doesn't alter any entries in the file that already have bugs -- meaning something like BUG1234 DEFER : ... or WONTFIX DEFER : ... won't change. Is this the correct behavior? I'm planning on running the script later today with the most recent version of test_expectations.txt -- there will be 200+ new bugs. Let me know if I should hold off on doing so. Ojan, I'll send you the review for test_expecations.txt. Regards, Glenn On Fri, Apr 10, 2009 at 4:36 PM, Ojan Vafai o...@google.com wrote: I hate to ask this, but any chance we could rewrite it in python? It would be a lot less code in python (one script file) and would match the rest of our codebase (which currently does all scripting in python). I'm mainly worried about usability and maintainability here. That all said, I think you should go ahead and run this script as a one off so we can get the test lists having bug IDs ASAP as that's kind of urgent to us being able to track the current state of layout tests. Then we can worry about checking in a python version later if we decide it's necessary. Pam, Darin, Jon, Mark: You might want to confirm that my request below makes sense with respect to future handling/triaging of layout tests bugs. Also for the initial one-off version, can you make any tests that are currently marked as DEFER be P3s? Then when you spit out the results to the test_expectations file, I don't think there's any need to include the DEFER anymore as we'll rely entirely on bug priorities to decide which tests need fixing for the next release (i.e. all P1 tests and maybe some P2 tests). That way there will be less initial triaging to do in order to make sure we keep the tree shippable. Eventually we'll have to go through all the P3s in the Untriaged state and up some of them to P2s, but there is no pressing need to do so right away. Ojan On Fri, Apr 10, 2009 at 3:42 PM, Glenn Wilson gwil...@chromium.orgwrote: (Ack...resending) Ok, I have the CL ready, if anyone with Java readability would be willing to do a review, please let me know. more inline... On Fri, Apr 10, 2009 at 2:07 PM, Pam Greene p...@chromium.org wrote: On Thu, Apr 9, 2009 at 6:49 PM, Ojan Vafai o...@google.com wrote: At a quick glance, this looks great. I didn't look over every bug, but the ones I did look at look good. Yep. You'll want some sort of default description for the ones that have none. I ran the script on a small test file...an example bug is here: http://code.google.com/p/chromium/issues/detail?id=9991 There's at least a small bit of indication of where it came from. I also added the line numbers from test_expectations.txt that generated the bug. Additionally, I also wrote the script to then add the created bug numbers to the file. This ends up re-arranging some of the flags (DEBUG DEFER ... etc. -- DEFER DEBUG ...), but all data is retained. 200+ bugs is certainly too many, but that's no reason not to file them. (Sorry, couldn't resist. Seriously, yes, definitely file them. Better in the issue tracker than getting lost.) There are probably some improvements to be had to better group some of the bugs, but it's probably not an order of magnitude improvement. I just didn't want to create tons of bugs that were not useful. It would be great to check in a version of this script that we could run when a number of tests fail (e.g. when doing a bad webkit merge). That way, we can add them all to the local test_expectations.txt file and have it spit out the new results. Fixing the file we have and moving forward are slightly different use cases, but yes. In the long run, we shouldn't need any script to fix an existing bug-less expectations file, only the part that adds bugs for newly added tests. I'm not sure whether that part should be controlled by adding the tests to the file and having the script file bugs, or making it a fully interactive app: give it a list of files and a description, and it both changes the file and creates a bug. There may be a short-term solution and a long-term solution. The short term solution seems to be for someone (assumedly the merger) to add the failing tests to the file, run the script, then commit the file. In the long term, this could be one step, or perhaps be driven right from a roll DEPS to new version of WebKit script...right? Really, it would be great if the script filed bugs and then just modified test_expectations.txt directly (without committing it). Also
[chromium-dev] Re: Creating bugs from test_expectations.txt
Ok, I ran the script and created 211 bugs (in the ~10270 - 10500 range). As far as I can tell, all the bugs were created successfullythe script now picks up 0 tests without bugs. Ojan, I sent you the review of test_expectations: http://codereview.chromium.org/73026/show Two things I noticed: 1. The script didn't recognize layout tests in chrome/ and not in LayoutTests/ -- it's only a handful, so updating those by hand shouldn't take very long. 2. The script re-arranged some of the flags, as expected. If it is important that we keep the prior ordering of the flags, let me know. Also, I'd love to get this script reviewed and checked in, if anyone would be willing to do a Java review. Regards, Glenn On Mon, Apr 13, 2009 at 10:19 AM, Ojan Vafai o...@google.com wrote: On Mon, Apr 13, 2009 at 9:35 AM, Pam Greene p...@chromium.org wrote: On Mon, Apr 13, 2009 at 9:09 AM, Glenn Wilson gwil...@chromium.orgwrote: Yes, the intention is to have something maintainable that gets run as part of the regular testing/merging/etc. process -- which means eventually rewriting in python. I wrote this in Java because we had a java lib for automatically creating bugs in the issue tracker, and I wanted to get it up and running quickly. We should have a python API for doing the same thing. I can easily change the script so it marks DEFERs as P3s. However, currently, the script doesn't alter any entries in the file that already have bugs -- meaning something like BUG1234 DEFER : ... or WONTFIX DEFER : ... won't change. Is this the correct behavior? The end goal is to take DEFER out and let the bug priorities run the show. But if we're not entirely confident that this will work (both the script and the human tracking process afterward), it's easy enough to strip the DEFER modifiers later. Makes sense to me. Lets leave in the defer modifiers, but mark any *new* bugs for deferred tests as P3s. The current behavior of leaving existing bugs unaltered seems correct to me. BTW, doing it in Java for expediency sake is totally reasonable. Ojan - Pam I'm planning on running the script later today with the most recent version of test_expectations.txt -- there will be 200+ new bugs. Let me know if I should hold off on doing so. Ojan, I'll send you the review for test_expecations.txt. Regards, Glenn On Fri, Apr 10, 2009 at 4:36 PM, Ojan Vafai o...@google.com wrote: I hate to ask this, but any chance we could rewrite it in python? It would be a lot less code in python (one script file) and would match the rest of our codebase (which currently does all scripting in python). I'm mainly worried about usability and maintainability here. That all said, I think you should go ahead and run this script as a one off so we can get the test lists having bug IDs ASAP as that's kind of urgent to us being able to track the current state of layout tests. Then we can worry about checking in a python version later if we decide it's necessary. Pam, Darin, Jon, Mark: You might want to confirm that my request below makes sense with respect to future handling/triaging of layout tests bugs. Also for the initial one-off version, can you make any tests that are currently marked as DEFER be P3s? Then when you spit out the results to the test_expectations file, I don't think there's any need to include the DEFER anymore as we'll rely entirely on bug priorities to decide which tests need fixing for the next release (i.e. all P1 tests and maybe some P2 tests). That way there will be less initial triaging to do in order to make sure we keep the tree shippable. Eventually we'll have to go through all the P3s in the Untriaged state and up some of them to P2s, but there is no pressing need to do so right away. Ojan On Fri, Apr 10, 2009 at 3:42 PM, Glenn Wilson gwil...@chromium.orgwrote: (Ack...resending) Ok, I have the CL ready, if anyone with Java readability would be willing to do a review, please let me know. more inline... On Fri, Apr 10, 2009 at 2:07 PM, Pam Greene p...@chromium.org wrote: On Thu, Apr 9, 2009 at 6:49 PM, Ojan Vafai o...@google.com wrote: At a quick glance, this looks great. I didn't look over every bug, but the ones I did look at look good. Yep. You'll want some sort of default description for the ones that have none. I ran the script on a small test file...an example bug is here: http://code.google.com/p/chromium/issues/detail?id=9991 There's at least a small bit of indication of where it came from. I also added the line numbers from test_expectations.txt that generated the bug. Additionally, I also wrote the script to then add the created bug numbers to the file. This ends up re-arranging some of the flags (DEBUG DEFER ... etc. -- DEFER DEBUG ...), but all data is retained. 200+ bugs is certainly too many, but that's no reason not to file them. (Sorry, couldn't resist. Seriously, yes, definitely file them. Better
[chromium-dev] Re: Creating bugs from test_expectations.txt
(Ack...resending) Ok, I have the CL ready, if anyone with Java readability would be willing to do a review, please let me know. more inline... On Fri, Apr 10, 2009 at 2:07 PM, Pam Greene p...@chromium.org wrote: On Thu, Apr 9, 2009 at 6:49 PM, Ojan Vafai o...@google.com wrote: At a quick glance, this looks great. I didn't look over every bug, but the ones I did look at look good. Yep. You'll want some sort of default description for the ones that have none. I ran the script on a small test file...an example bug is here: http://code.google.com/p/chromium/issues/detail?id=9991 There's at least a small bit of indication of where it came from. I also added the line numbers from test_expectations.txt that generated the bug. Additionally, I also wrote the script to then add the created bug numbers to the file. This ends up re-arranging some of the flags (DEBUG DEFER ... etc. -- DEFER DEBUG ...), but all data is retained. 200+ bugs is certainly too many, but that's no reason not to file them. (Sorry, couldn't resist. Seriously, yes, definitely file them. Better in the issue tracker than getting lost.) There are probably some improvements to be had to better group some of the bugs, but it's probably not an order of magnitude improvement. I just didn't want to create tons of bugs that were not useful. It would be great to check in a version of this script that we could run when a number of tests fail (e.g. when doing a bad webkit merge). That way, we can add them all to the local test_expectations.txt file and have it spit out the new results. Fixing the file we have and moving forward are slightly different use cases, but yes. In the long run, we shouldn't need any script to fix an existing bug-less expectations file, only the part that adds bugs for newly added tests. I'm not sure whether that part should be controlled by adding the tests to the file and having the script file bugs, or making it a fully interactive app: give it a list of files and a description, and it both changes the file and creates a bug. There may be a short-term solution and a long-term solution. The short term solution seems to be for someone (assumedly the merger) to add the failing tests to the file, run the script, then commit the file. In the long term, this could be one step, or perhaps be driven right from a roll DEPS to new version of WebKit script...right? Really, it would be great if the script filed bugs and then just modified test_expectations.txt directly (without committing it). Also, the script should remove any comments it moves into bug descriptions. We should get to a point where all the comments about layout tests are in the bug descriptions themselves instead of in this file. I disagree with this last part. I'd prefer a brief description to remain in the file, with any details in the bug. Certainly that's needed for WONTFIX bugs, where we may not have a bug since there's no work to be tracked, but it's helpful for others too. I've found it frustrating and time-consuming to track down when I see big blocks of failures with no explanations at all. Think of it like the svn checkin comment: enough to have an idea what's going on right there where you need it, with more detail in the bug for when you're really digging. Yep, I modified the script so that it will change test_expectations.txt to have the bug number (you'd just need to commit it afterwards). It's pretty easy to for the script to clip out the comments, but I'm reluctant to, because the script may be over/under zealous in what comments it associates with a layout test. Maybe I can add that behavior as another command-line flag. - Pam I think it would be good to get the script checked in first and then run it on the existing test_expectations.txt file. Ojan On Thu, Apr 9, 2009 at 6:42 PM, Glenn Wilson gwil...@google.com wrote: Hi Pam Ojan, I wrote a script that would extract all of the layout tests from test_expectations.txt that we haven't marked as WONTFIX and don't have a bug number. I also tried a simple heuristic to get the context of the layout test via nearby commentsit's not perfect, and I'll have to change some of them by hand, but many of the merge comments are getting picked up. I've also hooked up our library for creating demetrius bugs, so getting bugs made for these should be a matter of running the script with different arguments (I hope). What are your thoughts? Is these as descriptive/accurate as we need? Is 200+ bugs too many? Thanks! Glenn --~--~-~--~~~---~--~~ Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev -~--~~~~--~~--~--~---
