Hi Sammy, > -----Original Message----- > From: sam...@sammykmedia.com [mailto:sam...@sammykmedia.com] On > > I agree that parallelism should be the first and primary goal of the > refactor. I just > mentioned maintainability and unit tests first since I foresee those things > happing as we refactor to parallelism. :) > ACK. > I'm OK doing a very minimal "proof-of-concept" project that would showcase a > possible concurrency model so that we could discuss the tradeoffs of the > implementation details in more depth. But after we agree on an > implementation, I don't think we need to merge it directly, instead we should > make small refactorings of the live run-tests.php that would eventually guide > it > toward the agreed-upon proof-of-concept implementation. > From your first message, it looked to me like you had intention to rewrite the most of the tool. My suggestion was based on that. The PR I mentioned were to see the initial work in that case. As that were the base for the sustainable improvement, which could be continued directly in the core. Say, a tool that can run both the existing tests, even unchanged, and some tests also in parallel by whatsoever new approach. OFC run-tests.php should stay, literally, just it'd have a new engine under the hood. Once that initial work would exist, it could be merged and more detailed incremental process would take place, that would include also the aspects you mentioned in the first mail and on github. The plus I'd see to proceed this way were to stay compatible, while having the base for the further improvements.
> As far as tests that can't be run in parallel, we could look to HHVM which > checks > for an empty file with the same name as the test file with the ".serial" > extension. > If it finds that, the test is put into a special "run in serial" bucket. I > know make > has a ".NOTPARALLEL" > target to specify that things should be run serially. We could also have a new > section in the .phpt file "--NOTPARALLEL--" or a tag similar to ===DONE=== so > it's not parsed as a section: > "===NOTPARALLEL===". At any rate, we'll need to decide on something but I > think an empty ".serial" file makes a lot of sense. :) > Some extra file to mark parallelism looks excessive for me, it should be only one test file with different sections. So either way, --SERIAL-- or --PARALLEL-- section would be enough to tell the runner about parallelism, depends on what makes sense to mark by default. As for me, it looks like we should expect tests to be not parallelism capable by default, at least at the start - in that case less things can go wrong. Only those that are explicitly marked as parallel, would be executed that way, otherwise the runner would act same way as before. It looks like the idea to split some tests into sub folders might simplify things a lot, as noted by Andrea and Rasmus. With the subdir approach - yeah, there can be also a huge simplification, also with regard to tests sensitive to the run order. I recall at least ext/pgsql is based on that approach. But for sure, fe one can run tests from different extension folders in parallel, for example ext/pgsql and ext/mysqli should not collide, many others won't, too, as there are also lots of exts not using any resources at all. It'd be indeed a fast way to achieve *some* parallelism without big effort. To call some - ext/pcre, ext/date. Issues are to expect - DB extensions for same database, fe ext/mysqli and ext/pdo_mysqli - built-in server based tests - tests working on same test files - creating/deleting/etc. - any other tests using same resources - slow tests possibly to occupy execution slots Even then, there are quite some tests located in the same dir, that still can be run in parallel. For directories with a big amount of tests, so the subdir approach might be less usefyl. Thus, having support for the exact sections still should be implemented, AFM. > What would you say would be good next steps? Should I draft up a more formal > process in the WIKI so we can discuss specifics further? > https://wiki.php.net/qa/runtests That's a good idea, yep. Especially to articulate the minimum required and the pieces of work to be done. As you told, there are already some people interested in this work, so everyone could fork and update the wiki with the part of the taken work. That'd be helpful to both track the progress and issues and to avoid duplicate effort. We've used similar approach to track the work on the 64-bit branch https://wiki.php.net/rfc/string-size_t/progress . Regards Anatol
