On Mon, Apr 13, 2009 at 9:35 AM, Pam Greene <p...@chromium.org> wrote:
> On Mon, Apr 13, 2009 at 9:09 AM, Glenn Wilson <gwil...@chromium.org>wrote: > >> Yes, the intention is to have something maintainable that gets run as part >> of the regular testing/merging/etc. process -- which means eventually >> rewriting in python. I wrote this in Java because we had a java lib for >> automatically creating bugs in the issue tracker, and I wanted to get it up >> and running quickly. We should have a python API for doing the same thing. >> >> I can easily change the script so it marks DEFERs as P3s. However, >> currently, the script doesn't alter any entries in the file that already >> have bugs -- meaning something like "BUG1234 DEFER : ..." or "WONTFIX DEFER >> : ..." won't change. Is this the correct behavior? >> > > The end goal is to take DEFER out and let the bug priorities run the show. > But if we're not entirely confident that this will work (both the script and > the human tracking process afterward), it's easy enough to strip the "DEFER" > modifiers later. > Makes sense to me. Lets leave in the defer modifiers, but mark any *new* bugs for deferred tests as P3s. The current behavior of leaving existing bugs unaltered seems correct to me. BTW, doing it in Java for expediency sake is totally reasonable. Ojan > > - Pam > > >> >> I'm planning on running the script later today with the most recent >> version of test_expectations.txt -- there will be 200+ new bugs. Let me >> know if I should hold off on doing so. Ojan, I'll send you the review for >> test_expecations.txt. >> >> Regards, >> Glenn >> >> >> >> On Fri, Apr 10, 2009 at 4:36 PM, Ojan Vafai <o...@google.com> wrote: >> >>> I hate to ask this, but any chance we could rewrite it in python? It >>> would be a lot less code in python (one script file) and would match the >>> rest of our codebase (which currently does all scripting in python). I'm >>> mainly worried about usability and maintainability here. >>> That all said, I think you should go ahead and run this script as a one >>> off so we can get the test lists having bug IDs ASAP as that's kind of >>> urgent to us being able to track the current state of layout tests. Then we >>> can worry about checking in a python version later if we decide it's >>> necessary. >>> >>> Pam, Darin, Jon, Mark: You might want to confirm that my request below >>> makes sense with respect to future handling/triaging of layout tests bugs. >>> >>> Also for the initial one-off version, can you make any tests that are >>> currently marked as DEFER be P3s? Then when you spit out the results to the >>> test_expectations file, I don't think there's any need to include the DEFER >>> anymore as we'll rely entirely on bug priorities to decide which tests need >>> fixing for the next release (i.e. all P1 tests and maybe some P2 tests). >>> That way there will be less initial triaging to do in order to make sure we >>> keep the tree shippable. Eventually we'll have to go through all the P3s in >>> the "Untriaged" state and up some of them to P2s, but there is no pressing >>> need to do so right away. >>> >>> Ojan >>> >>> On Fri, Apr 10, 2009 at 3:42 PM, Glenn Wilson <gwil...@chromium.org>wrote: >>> >>>> (Ack...resending) >>>> Ok, I have the CL ready, if anyone with Java readability would be >>>> willing to do a review, please let me know. >>>> more inline... >>>> >>>> On Fri, Apr 10, 2009 at 2:07 PM, Pam Greene <p...@chromium.org> wrote: >>>> >>>>> >>>>> >>>>> On Thu, Apr 9, 2009 at 6:49 PM, Ojan Vafai <o...@google.com> wrote: >>>>> >>>>>> At a quick glance, this looks great. I didn't look over every bug, but >>>>>> the ones I did look at look good. >>>>> >>>>> >>>>> Yep. You'll want some sort of default description for the ones that >>>>> have none. >>>>> >>>> >>>> I ran the script on a small test file...an example bug is here: >>>> http://code.google.com/p/chromium/issues/detail?id=9991 >>>> >>>> There's at least a small bit of indication of where it came from. I >>>> also added the line numbers from test_expectations.txt that generated the >>>> bug. Additionally, I also wrote the script to then add the created bug >>>> numbers to the file. This ends up re-arranging some of the flags (DEBUG >>>> DEFER ... etc. --> DEFER DEBUG ...), but all data is retained. >>>> >>>> >>>>> 200+ bugs is certainly too many, but that's no reason not to file them. >>>>> (Sorry, couldn't resist. Seriously, yes, definitely file them. Better in >>>>> the >>>>> issue tracker than getting lost.) >>>>> >>>> >>>> There are probably some improvements to be had to better group some of >>>> the bugs, but it's probably not an order of magnitude improvement. I just >>>> didn't want to create tons of bugs that were not useful. >>>> >>>> >>>>> >>>>> >>>>>> It would be great to check in a version of this script that we could >>>>>> run when a number of tests fail (e.g. when doing a bad webkit merge). >>>>>> That >>>>>> way, we can add them all to the local test_expectations.txt file and >>>>>> have it >>>>>> spit out the new results. >>>>>> >>>>> >>>>> Fixing the file we have and moving forward are slightly different use >>>>> cases, but yes. In the long run, we shouldn't need any script to fix an >>>>> existing bug-less expectations file, only the part that adds bugs for >>>>> newly >>>>> added tests. I'm not sure whether that part should be controlled by adding >>>>> the tests to the file and having the script file bugs, or making it a >>>>> fully >>>>> interactive "app": give it a list of files and a description, and it both >>>>> changes the file and creates a bug. >>>>> >>>> >>>> There may be a short-term solution and a long-term solution. The short >>>> term solution seems to be for someone (assumedly the merger) to add the >>>> failing tests to the file, run the script, then commit the file. In the >>>> long term, this could be one step, or perhaps be driven right from a "roll >>>> DEPS to new version of WebKit" script...right? >>>> >>>> >>>>> >>>>>> Really, it would be great if the script filed bugs and then just >>>>>> modified test_expectations.txt directly (without committing it). Also, >>>>>> the >>>>>> script should remove any comments it moves into bug descriptions. We >>>>>> should >>>>>> get to a point where all the comments about layout tests are in the bug >>>>>> descriptions themselves instead of in this file. >>>>>> >>>>> >>>>> I disagree with this last part. I'd prefer a brief description to >>>>> remain in the file, with any details in the bug. Certainly that's needed >>>>> for >>>>> WONTFIX bugs, where we may not have a bug since there's no work to be >>>>> tracked, but it's helpful for others too. I've found it frustrating and >>>>> time-consuming to track down when I see big blocks of failures with no >>>>> explanations at all. Think of it like the svn checkin comment: enough to >>>>> have an idea what's going on right there where you need it, with more >>>>> detail >>>>> in the bug for when you're really digging. >>>>> >>>> >>>> Yep, I modified the script so that it will change test_expectations.txt >>>> to have the bug number (you'd just need to commit it afterwards). It's >>>> pretty easy to for the script to clip out the comments, but I'm reluctant >>>> to, because the script may be over/under zealous in what comments it >>>> associates with a layout test. Maybe I can add that behavior as another >>>> command-line flag. >>>> >>>> >>>>> >>>>> - Pam >>>>> >>>>> >>>>>> I think it would be good to get the script checked in first and then >>>>>> run it on the existing test_expectations.txt file. >>>>>> >>>>>> Ojan >>>>>> >>>>>> >>>>>> On Thu, Apr 9, 2009 at 6:42 PM, Glenn Wilson <gwil...@google.com> >>>>>> wrote: >>>>>> >>>>>>> Hi Pam & Ojan, >>>>>>> I wrote a script that would extract all of the layout tests from >>>>>>> test_expectations.txt that we haven't marked as WONTFIX and don't have >>>>>>> a bug >>>>>>> number. I also tried a simple heuristic to get the context of the >>>>>>> layout >>>>>>> test via nearby comments....it's not perfect, and I'll have to change >>>>>>> some >>>>>>> of them by hand, but many of the merge comments are getting picked up. >>>>>>> >>>>>>> I've also hooked up our library for creating demetrius bugs, so >>>>>>> getting bugs made for these should be a matter of running the script >>>>>>> with >>>>>>> different arguments (I hope). >>>>>>> >>>>>>> What are your thoughts? Is these as descriptive/accurate as we need? >>>>>>> Is 200+ bugs too many? >>>>>>> >>>>>>> Thanks! >>>>>>> Glenn >>>>>>> >>>>>>> >>>>>>> >>>>>> >>>> >>> >> > > > > --~--~---------~--~----~------------~-------~--~----~ Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev -~----------~----~----~----~------~----~------~--~---
