> I wonder how possible it would be to transcode the old tests to Swat? Swat has some architectural specifics. As it simply makes http requests ( by curl ) and validate server response ( by swat DSL ) it is considered as black box testing tools , there is no way to interact with httpd internal structure. So probably most of old tests could be easily ported but some probably not ( which implies something more complicated ).
Basically swat could cover simple and average use cases, everything that could expressed in terms of make http requests and then analyzing an output , see examples for some issues I have expressed in swat scenarios at https://github.com/melezhik/apache-swat Also alternatively we may run 2 frameworks in parallel. Apache-swat is intended for developers usage first and is able to run per-issues test. This how I see possible workflow: * developer accept a bug request * developer ( or some else ) create swat test for given bug/feature * so we have `test first` approach - everyone involved in bug/feature resolution has possibility to verify changes by simply running : swat -t $issue-id 2016-02-05 17:08 GMT+03:00 Alexey Melezhik <melez...@gmail.com>: > 2016-02-05 17:01 GMT+03:00 Jim Jagielski <j...@jagunet.com>: >> Personally, I like the idea of having another framework; >> the current one is OK but somewhat "painful" to update. > > Hi! What kind of possible issues you here ? when saying "painful" to > update, please explain. > Thanks. > > Also , please consider this link - https://github.com/melezhik/apache-swat and > this post - > http://mail-archives.apache.org/mod_mbox/httpd-dev/201602.mbox/%3CCAL7UUk-zqb1=7wDvCe8E-Ku=U=ay0mdj_-jgcjvzzdvaran...@mail.gmail.com%3E > as latest version of this idea .... > > > > > >> >> I wonder how possible it would be to transcode the old tests >> to Swat? We could then provide for 2 testing frameworks, one >> developed by the ASF and the other external and 3rd party. >> >> >>> On Feb 1, 2016, at 5:23 AM, Alexey Melezhik <melez...@gmail.com> wrote: >>> >>> Here the list of existed issues I was able to automate tests for: >>> >>> vagrant@Debian-jessie-amd64-netboot:~/my/apache-swat$ ls -1 | grep -P '\d' >>> 44221 >>> 46751 >>> 58789 >>> 58828 >>> 58854 >>> >>> >>> I have informed developers at https://bz.apache.org/bugzilla/ about >>> these, so they rely upon them. Or you have some pre-release testing, >>> so run all of these in a whole chunk? >>> >>> Regards >>> >>> Alexey >>> >>> PS chaining issues / tests listing as always - >>> https://github.com/melezhik/apache-swat >>> >>> PS2 it'd be could to verify tests as well against "bleeding edge" >>> apache version gets installed from SCM, but I don't know how to do >>> this. >>> Currently I use http://httpd.apache.org/download.cgi ( last stable release ) >>> >>> >>> >>> 2016-01-30 21:36 GMT+03:00 Alexey Melezhik <melez...@gmail.com>: >>>> Hi Bill! >>>> >>>> I have started to assemble swat tests for apache2 issues coming from >>>> https://bz.apache.org/bugzilla >>>> >>>> And this the first one - >>>> https://bz.apache.org/bugzilla/show_bug.cgi?id=44221 , >>>> >>>> Test suite is failed for the moment : >>>> >>>> >>>> vagrant@Debian-jessie-amd64-netboot:~/my/apache-swat$ swat -t 44221 >>>> /home/vagrant/.swat/.cache/384/prove/44221/foo.baz/00.GET.t ... >>>> ok 1 - GET 127.0.0.1/44221/foo.baz succeeded >>>> # http headers saved to /home/vagrant/.swat/.cache/384/prove/52jLAo76YZ.hdr >>>> # body saved to /home/vagrant/.swat/.cache/384/prove/52jLAo76YZ >>>> ok 2 - output match '200 OK' >>>> 1..2 >>>> ok >>>> /home/vagrant/.swat/.cache/384/prove/44221/FOO.bar/00.GET.t ... >>>> ok 1 - GET 127.0.0.1/44221/FOO.bar succeeded >>>> # http headers saved to /home/vagrant/.swat/.cache/384/prove/DvEZnO6ALe.hdr >>>> # body saved to /home/vagrant/.swat/.cache/384/prove/DvEZnO6ALe >>>> ok 2 - output match '200 OK' >>>> ok 3 - output match /Location: \S+/ >>>> ok 4 - 'Location: http://127.0.0.1/44221/foo.bar' match 'foo.bar' >>>> 1..4 >>>> ok >>>> /home/vagrant/.swat/.cache/384/prove/44221/foo.BAR/00.GET.t ... >>>> ok 1 - GET 127.0.0.1/44221/foo.BAR succeeded >>>> # http headers saved to /home/vagrant/.swat/.cache/384/prove/bWQ1Sbl0YZ.hdr >>>> # body saved to /home/vagrant/.swat/.cache/384/prove/bWQ1Sbl0YZ >>>> ok 2 - output match '200 OK' >>>> ok 3 - output match /Location: \S+/ >>>> ok 4 - 'Location: http://127.0.0.1/44221/foo.bar' match 'foo.bar' >>>> 1..4 >>>> ok >>>> /home/vagrant/.swat/.cache/384/prove/44221/foo.bar/00.GET.t ... >>>> ok 1 - GET 127.0.0.1/44221/foo.bar succeeded >>>> # http headers saved to /home/vagrant/.swat/.cache/384/prove/Jd_hlo7Qwv.hdr >>>> # body saved to /home/vagrant/.swat/.cache/384/prove/Jd_hlo7Qwv >>>> ok 2 - output match '200 OK' >>>> 1..2 >>>> ok >>>> /home/vagrant/.swat/.cache/384/prove/44221/foo.html/00.GET.t .. >>>> ok 1 - GET 127.0.0.1/44221/foo.html succeeded >>>> # http headers saved to /home/vagrant/.swat/.cache/384/prove/tW2L511eym.hdr >>>> # body saved to /home/vagrant/.swat/.cache/384/prove/tW2L511eym >>>> ok 2 - output match /HTTP\/(\S+) (\d+) \S+/ >>>> not ok 3 - 'HTTP/1.1 300 Multiple Choices' match /404 / >>>> >>>> # Failed test ''HTTP/1.1 300 Multiple Choices' match /404 /' >>>> # at /usr/local/share/perl/5.20.2/swat.pm line 218. >>>> not ok 4 - output match 'Not Found' >>>> >>>> # Failed test 'output match 'Not Found'' >>>> # at /usr/local/share/perl/5.20.2/swat.pm line 218. >>>> 1..4 >>>> # Looks like you failed 2 tests of 4. >>>> Dubious, test returned 2 (wstat 512, 0x200) >>>> Failed 2/4 subtests >>>> >>>> Test Summary Report >>>> ------------------- >>>> /home/vagrant/.swat/.cache/384/prove/44221/foo.html/00.GET.t (Wstat: >>>> 512 Tests: 4 Failed: 2) >>>> Failed tests: 3-4 >>>> Non-zero exit status: 2 >>>> Files=5, Tests=16, 0 wallclock secs ( 0.03 usr 0.00 sys + 0.27 cusr >>>> 0.00 csys = 0.30 CPU) >>>> Result: FAIL >>>> >>>> ... >>>> >>>> Ok, this is just the beginning ;-))) , watch soon updates at - >>>> https://github.com/melezhik/apache-swat >>>> >>>> 2016-01-29 20:44 GMT+03:00 William A Rowe Jr <wr...@rowe-clan.net>: >>>>> On Fri, Jan 29, 2016 at 6:35 AM, Alexey Melezhik <melez...@gmail.com> >>>>> wrote: >>>>>> >>>>>> Hi Bill! >>>>>> >>>>>> Any news? ( Please see my previous reply ...) >>>>>> Intr >>>>> >>>>> >>>>> Intrigued :) But my responses will be delayed, I personally >>>>> won't have time to look further myself until other backlogged >>>>> commitments to httpd are caught up some more. >>>>> >>>>> Be aware many of us have only a few hours to participate in the >>>>> httpd project itself, so it will take several days or more to gather >>>>> feedback on your interest and perhaps a champion to drive such >>>>> an effort if it is actively driven from this side. Of course that >>>>> should not stop you from assembling some ideas at the vcs of >>>>> your choice, and sharing a link for those who want to look at an >>>>> early draft of such an effort! >>>>> >>>>> Cheers, >>>>> >>>>> Bill >>
