Hi, I'd like this idea basically, it would be great if you or somebody else could work on it.
One of my former collegue started to investigate the same thing 1.5 years before, but it was lost among many tasks with higher priorities. Feel free to reuse these bugs and WIP patches: (or file new ones) - https://bugs.webkit.org/show_bug.cgi?id=117831 - and bugs in its "depends on" list In my opinion red tree and flakey tets could cause many problems, because EWS is designed primarily for green tree. But unfortunately only Mac (and now Win release) bots are green almost all the time. I can't remember if EFL or GTK bots were green in the last 1-2 years ever. :-/ Now the tester EWS bots (Mac WK1 and WK2) run all tests. If all tests pass, there is no problem. But if there is only one failure, it tries to build and run tests without the patch. If all tests pass, the patch caused the regression. If there are exactly the same failures on the trunk and with the patch, the bubble becomes green. But flakey failures on the trunk can make EWS crazy. I remember that EWS wasn't able to process any patch for days because of 1-2 flakey failures on the trunk. If we enabled full or partial testing on unstable EWSs, they would become very slow, unstable and we would lost the very quick and stable builder EWSs. Maybe we should have separated builder and builder-and-tester EWS bots to avoid this kind of problems, but we would need more hardware for it. Anyway it would be great if EFL and GTK maintainers could keep the tester bots green almost all the time. I know it isn't an easy task and would need more resource. I remember from the QtWebKit era that 1-2 full-time gardener can be enough to keep the tree green, but it means only minor buildfixes, expected updates, new bug reports and skipping/marking new failures time to time. To increase the stabilility, fix regressions, decrease the number of skipped tests would need much more resource. Of course, I willingly help you, but unfortunately I don't have too much time for this topic nowadays. But feel free to cc me to related bugs and I'll try to help if I can. br, Ossy youenn fablet írta:
Hi, Related to the webkit contributor meeting discussion related to ports, I would find it useful if EWS bots (gtk, efl, win, ios) were running the tests that are modified/created by a patch. The idea would be to turn yellow the port bubble whenever one of these tests do not pass. Results would be uploaded to bugzilla. This would give an incentive for patch developers to try fixing the tests on these ports. That may reduce (slightly? noticeably?) port maintainers gardening effort. That would also be valuable when importing test suites. Any potential issue? objection to move that forward? Anyone willing to help? Thoughts? Thanks, Youenn
_______________________________________________ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev