> On Feb 5, 2016, at 9:26 AM, Alexey Melezhik <melez...@gmail.com> wrote: > >> I wonder how possible it would be to transcode the old tests to Swat? > > Swat has some architectural specifics. As it simply makes http > requests ( by curl ) and validate server response ( by swat DSL ) it > is considered as black box testing tools , there is no way to interact > with httpd internal structure. So probably most of old tests could be > easily ported but some probably not ( which implies something more > complicated ). >
Most of the other tests w/ the old framework is the exact same... requests sent and responses verified against what is expected. > Basically swat could cover simple and average use cases, everything > that could expressed in terms of make http requests and then > analyzing an output , see examples for some issues I have expressed in > swat scenarios at https://github.com/melezhik/apache-swat > > > Also alternatively we may run 2 frameworks in parallel. Apache-swat is > intended for developers usage first and is able to run per-issues > test. > > This how I see possible workflow: > > * developer accept a bug request > * developer ( or some else ) create swat test for given bug/feature > * so we have `test first` approach - everyone involved in bug/feature > resolution has possibility to verify changes by simply running : > > swat -t $issue-id > > > > > > > 2016-02-05 17:08 GMT+03:00 Alexey Melezhik <melez...@gmail.com>: >> 2016-02-05 17:01 GMT+03:00 Jim Jagielski <j...@jagunet.com>: >>> Personally, I like the idea of having another framework; >>> the current one is OK but somewhat "painful" to update. >> >> Hi! What kind of possible issues you here ? when saying "painful" to >> update, please explain. >> Thanks. >> >> Also , please consider this link - https://github.com/melezhik/apache-swat >> and >> this post - >> http://mail-archives.apache.org/mod_mbox/httpd-dev/201602.mbox/%3CCAL7UUk-zqb1=7wDvCe8E-Ku=U=ay0mdj_-jgcjvzzdvaran...@mail.gmail.com%3E >> as latest version of this idea .... >> >> >> >> >> >>> >>> I wonder how possible it would be to transcode the old tests >>> to Swat? We could then provide for 2 testing frameworks, one >>> developed by the ASF and the other external and 3rd party. >>> >>> >>>> On Feb 1, 2016, at 5:23 AM, Alexey Melezhik <melez...@gmail.com> wrote: >>>> >>>> Here the list of existed issues I was able to automate tests for: >>>> >>>> vagrant@Debian-jessie-amd64-netboot:~/my/apache-swat$ ls -1 | grep -P '\d' >>>> 44221 >>>> 46751 >>>> 58789 >>>> 58828 >>>> 58854 >>>> >>>> >>>> I have informed developers at https://bz.apache.org/bugzilla/ about >>>> these, so they rely upon them. Or you have some pre-release testing, >>>> so run all of these in a whole chunk? >>>> >>>> Regards >>>> >>>> Alexey >>>> >>>> PS chaining issues / tests listing as always - >>>> https://github.com/melezhik/apache-swat >>>> >>>> PS2 it'd be could to verify tests as well against "bleeding edge" >>>> apache version gets installed from SCM, but I don't know how to do >>>> this. >>>> Currently I use http://httpd.apache.org/download.cgi ( last stable release >>>> ) >>>> >>>> >>>> >>>> 2016-01-30 21:36 GMT+03:00 Alexey Melezhik <melez...@gmail.com>: >>>>> Hi Bill! >>>>> >>>>> I have started to assemble swat tests for apache2 issues coming from >>>>> https://bz.apache.org/bugzilla >>>>> >>>>> And this the first one - >>>>> https://bz.apache.org/bugzilla/show_bug.cgi?id=44221 , >>>>> >>>>> Test suite is failed for the moment : >>>>> >>>>> >>>>> vagrant@Debian-jessie-amd64-netboot:~/my/apache-swat$ swat -t 44221 >>>>> /home/vagrant/.swat/.cache/384/prove/44221/foo.baz/00.GET.t ... >>>>> ok 1 - GET 127.0.0.1/44221/foo.baz succeeded >>>>> # http headers saved to >>>>> /home/vagrant/.swat/.cache/384/prove/52jLAo76YZ.hdr >>>>> # body saved to /home/vagrant/.swat/.cache/384/prove/52jLAo76YZ >>>>> ok 2 - output match '200 OK' >>>>> 1..2 >>>>> ok >>>>> /home/vagrant/.swat/.cache/384/prove/44221/FOO.bar/00.GET.t ... >>>>> ok 1 - GET 127.0.0.1/44221/FOO.bar succeeded >>>>> # http headers saved to >>>>> /home/vagrant/.swat/.cache/384/prove/DvEZnO6ALe.hdr >>>>> # body saved to /home/vagrant/.swat/.cache/384/prove/DvEZnO6ALe >>>>> ok 2 - output match '200 OK' >>>>> ok 3 - output match /Location: \S+/ >>>>> ok 4 - 'Location: http://127.0.0.1/44221/foo.bar' match 'foo.bar' >>>>> 1..4 >>>>> ok >>>>> /home/vagrant/.swat/.cache/384/prove/44221/foo.BAR/00.GET.t ... >>>>> ok 1 - GET 127.0.0.1/44221/foo.BAR succeeded >>>>> # http headers saved to >>>>> /home/vagrant/.swat/.cache/384/prove/bWQ1Sbl0YZ.hdr >>>>> # body saved to /home/vagrant/.swat/.cache/384/prove/bWQ1Sbl0YZ >>>>> ok 2 - output match '200 OK' >>>>> ok 3 - output match /Location: \S+/ >>>>> ok 4 - 'Location: http://127.0.0.1/44221/foo.bar' match 'foo.bar' >>>>> 1..4 >>>>> ok >>>>> /home/vagrant/.swat/.cache/384/prove/44221/foo.bar/00.GET.t ... >>>>> ok 1 - GET 127.0.0.1/44221/foo.bar succeeded >>>>> # http headers saved to >>>>> /home/vagrant/.swat/.cache/384/prove/Jd_hlo7Qwv.hdr >>>>> # body saved to /home/vagrant/.swat/.cache/384/prove/Jd_hlo7Qwv >>>>> ok 2 - output match '200 OK' >>>>> 1..2 >>>>> ok >>>>> /home/vagrant/.swat/.cache/384/prove/44221/foo.html/00.GET.t .. >>>>> ok 1 - GET 127.0.0.1/44221/foo.html succeeded >>>>> # http headers saved to >>>>> /home/vagrant/.swat/.cache/384/prove/tW2L511eym.hdr >>>>> # body saved to /home/vagrant/.swat/.cache/384/prove/tW2L511eym >>>>> ok 2 - output match /HTTP\/(\S+) (\d+) \S+/ >>>>> not ok 3 - 'HTTP/1.1 300 Multiple Choices' match /404 / >>>>> >>>>> # Failed test ''HTTP/1.1 300 Multiple Choices' match /404 /' >>>>> # at /usr/local/share/perl/5.20.2/swat.pm line 218. >>>>> not ok 4 - output match 'Not Found' >>>>> >>>>> # Failed test 'output match 'Not Found'' >>>>> # at /usr/local/share/perl/5.20.2/swat.pm line 218. >>>>> 1..4 >>>>> # Looks like you failed 2 tests of 4. >>>>> Dubious, test returned 2 (wstat 512, 0x200) >>>>> Failed 2/4 subtests >>>>> >>>>> Test Summary Report >>>>> ------------------- >>>>> /home/vagrant/.swat/.cache/384/prove/44221/foo.html/00.GET.t (Wstat: >>>>> 512 Tests: 4 Failed: 2) >>>>> Failed tests: 3-4 >>>>> Non-zero exit status: 2 >>>>> Files=5, Tests=16, 0 wallclock secs ( 0.03 usr 0.00 sys + 0.27 cusr >>>>> 0.00 csys = 0.30 CPU) >>>>> Result: FAIL >>>>> >>>>> ... >>>>> >>>>> Ok, this is just the beginning ;-))) , watch soon updates at - >>>>> https://github.com/melezhik/apache-swat >>>>> >>>>> 2016-01-29 20:44 GMT+03:00 William A Rowe Jr <wr...@rowe-clan.net>: >>>>>> On Fri, Jan 29, 2016 at 6:35 AM, Alexey Melezhik <melez...@gmail.com> >>>>>> wrote: >>>>>>> >>>>>>> Hi Bill! >>>>>>> >>>>>>> Any news? ( Please see my previous reply ...) >>>>>>> Intr >>>>>> >>>>>> >>>>>> Intrigued :) But my responses will be delayed, I personally >>>>>> won't have time to look further myself until other backlogged >>>>>> commitments to httpd are caught up some more. >>>>>> >>>>>> Be aware many of us have only a few hours to participate in the >>>>>> httpd project itself, so it will take several days or more to gather >>>>>> feedback on your interest and perhaps a champion to drive such >>>>>> an effort if it is actively driven from this side. Of course that >>>>>> should not stop you from assembling some ideas at the vcs of >>>>>> your choice, and sharing a link for those who want to look at an >>>>>> early draft of such an effort! >>>>>> >>>>>> Cheers, >>>>>> >>>>>> Bill >>>
