On Thu, May 18, 2017 at 5:59 AM, Anatol Belski <weltl...@outlook.de> wrote: > > Hi Sammy, > > > -----Original Message----- > > From: sam...@sammykmedia.com [mailto:sam...@sammykmedia.com] On > > Behalf Of Sammy Kaye Powers > > Sent: Wednesday, May 17, 2017 4:29 PM > > To: PHP Internals <internals@lists.php.net> > > Subject: [PHP-DEV] Implement formal process for run-tests.php refactor > > > > Hello internals folks! > > > > As you may already know, run-tests.php is an old legacy app that is in > > serious > > need of a refactor. Some things that are badly needed amongst other things > > are: > > > > - Better maintainability > > - Unit tests > > - Concurrency > > - Prettier output > > > > I have seen numerous attempts of rewriting run-tests from scratch after some > > rallying at a UG or PHP conference only to see the initial excitement die > > off. The > > general advice for refactoring legacy apps is to not start with a clean > > slate, > > rather refactor a little bit at a time. We should treat run-tests as we > > would any > > other legacy app. > > > > We need a structured process to allow the run-tests app to get refactored a > > little bit at a time. I propose the following (open to discussion of > > course). > > > > 1) Create a new run-tests karma with access to php-src.git/run-tests.php > > (and > > hopefully the soon-to-be created php-src.git/run-tests folder). > > > > 2) Elect 2-3 people and grant the new run-tests karma to them. They will > > champion the PR's that incrementally improve run-tests. > > > > 3) The new run-tests champions (we'll call them "champions" because, why > > not?) will run the entire test suite ensuring that the refactor didn't > > inadvertently > > break the functionality and if all good, merge. (We might be able to get a > > solid > > Docker container or an automated CI process for an end-to-end test - > > thoughts?) > > > > 4) The run-tests champions could be elected for a year-long term with the > > possibility of being reelected for another year commitment. This would help > > prevent run-test champion drop-off (because sometimes, life happens). > > > > Once we get a process in place, I've drafted up a possible proposal of next > > steps > > to get run-tests refactoring kicked off: > > > > https://github.com/SammyK/run-tests#proposed-refactoring-path > > > > It's time to circle the wagons on run-tests! :) > > > That's a fair amount on pre evaluation already 😉. > > AFM, the parallelism is the most wanted feature that would improve QA a lot, > so that is what should stand in the foreground. Having a better structured > app might help on it, yes, but having the parallelism to be targeted "some > when" is probably not a good sign. Otherwise, an nice app with colors and > styles, etc. would be indeed nice but less helpful, IMHO, as the current > run-tests.php is great for simplicity and still maintainable well. > > I would suggest therefore, to put the harder goals into the foreground, as > that would require and warrant a certain rewrite. It's not for nothing, that > previous rewrite attempts targeting similar things was abandoned previously, > and it's probably not wise to allow for the same mistakes. I'd see as first > step an app that > > - is a very minimal rewrite of run-tests.php, made OO and whatsoever > - mimics all the existing features and options, but only them > - has as less dependencies as possible, be it PHP extensions or PHP code > itself, best requires pure core only > - is compatible with the current test format > - is able to run tests in parallel > - some simple tests are ported for parallel runs already > > I'd see such a bare app first developed as a pull request, agreed upon so > then run-tests.php is completely replaced by it. Then where the main work can > begin - the most of it will be anyway not the app itself, but the porting of > the existing tests. The required karma would include the app itself, > ext/*/tests, Zend/tests, tests, and other related areas. Perhaps, the tests > able to run in parallel can be marked as such with a new section or alike. > There are probably thousands of tests that have to be touched to run in > parallel otherwise, be it DB tests utilizing same table name or built-in > server based tests that want to occupy same port. If there were such an app > merged into master, that supports current test format and can run a mix of > tests in old/new ways, it'd allow an organic integration and step-by-step > improvement of the test suite. Then also, the other features from your > evaluation could be developed while the test porting work is being done. > > Thanks > > Anatol > >
Hey Anatol! I agree that parallelism should be the first and primary goal of the refactor. I just mentioned maintainability and unit tests first since I foresee those things happing as we refactor to parallelism. :) I'm OK doing a very minimal "proof-of-concept" project that would showcase a possible concurrency model so that we could discuss the tradeoffs of the implementation details in more depth. But after we agree on an implementation, I don't think we need to merge it directly, instead we should make small refactorings of the live run-tests.php that would eventually guide it toward the agreed-upon proof-of-concept implementation. As far as tests that can't be run in parallel, we could look to HHVM which checks for an empty file with the same name as the test file with the ".serial" extension. If it finds that, the test is put into a special "run in serial" bucket. I know make has a ".NOTPARALLEL" target to specify that things should be run serially. We could also have a new section in the .phpt file "--NOTPARALLEL--" or a tag similar to ===DONE=== so it's not parsed as a section: "===NOTPARALLEL===". At any rate, we'll need to decide on something but I think an empty ".serial" file makes a lot of sense. :) What would you say would be good next steps? Should I draft up a more formal process in the WIKI so we can discuss specifics further? https://wiki.php.net/qa/runtests -- PHP Internals - PHP Runtime Development Mailing List To unsubscribe, visit: http://www.php.net/unsub.php
