Re: Gherkin DSL for testcase description and automation
Hi, Are there any tools available that can parse the Cucumber/Gherkin DSL code and produce the more human readable list oriented test cases? It sounds like it should be theoritically possible to write the test cases in Gherkin code and then produce both the automated test cases and the manual test cases. David On Mon, Dec 12, 2011 at 8:17 AM, Руаньяк wrote: > Hi guys, > > I guess, this idea will be a good decision for a small project on its > early days of development - in that case, all features can easily be > defined and QA will have some scenarios to test. > > As Gema and Charlie have shown, this will not work for manual > testcases for Ubuntu on the whole. However, Gherkin scenarios seem to > be a better choice for a small project than python unittests, > especially for newcomers. > > Thanks everyone for such a productive discussion, I hope some people > have learned about basic possibilities of BDD and will use it for some > projects (maybe, for Mago automation?) > -- > Vadim Rutkovsky > > 2011/12/10 Alex Lourie : >> First, let me just say wow. Thank you roignac for the idea; Gema, Brandon >> and Charlie for comments. This is a great discussion, and I would like to >> have much more of these in the future. >> >> To sum up what we have now: >> >> Roignac - Thanks a lot!! I think that this is a viable idea on its own, and >> we may continue discussing it in some form. >> Others (including Gema, Brendan, Charlie and myself) are not very fond of >> it. Not because of its merit, but because we feel it will make the work >> harder for manual testers. >> >> I would currently recommend the following course of action: continue with >> the rewriting test cases in ISTQB format, and, maybe, add future plans for >> exploring the possibilities for automation of those test cases, including >> Gherking format. I don't think that it's necessarily mutually exclusive to >> have both manual and automatic test cases. But I believe that we need to >> concentrate on just one format currently (I prefer the one which is easier >> for manual testers), and look for the possible expansions later. >> >> Great to have this discussion, it's refreshing after long period of low >> traffic in QA. >> >> Thanks all. >> >> -- >> Alex Lourie >> >> -- >> Ubuntu-qa mailing list >> Ubuntu-qa@lists.ubuntu.com >> Modify settings or unsubscribe at: >> https://lists.ubuntu.com/mailman/listinfo/ubuntu-qa >> > > -- > Ubuntu-qa mailing list > Ubuntu-qa@lists.ubuntu.com > Modify settings or unsubscribe at: > https://lists.ubuntu.com/mailman/listinfo/ubuntu-qa -- Ubuntu-qa mailing list Ubuntu-qa@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-qa
Re: Gherkin DSL for testcase description and automation
Hi guys, I guess, this idea will be a good decision for a small project on its early days of development - in that case, all features can easily be defined and QA will have some scenarios to test. As Gema and Charlie have shown, this will not work for manual testcases for Ubuntu on the whole. However, Gherkin scenarios seem to be a better choice for a small project than python unittests, especially for newcomers. Thanks everyone for such a productive discussion, I hope some people have learned about basic possibilities of BDD and will use it for some projects (maybe, for Mago automation?) -- Vadim Rutkovsky 2011/12/10 Alex Lourie : > First, let me just say wow. Thank you roignac for the idea; Gema, Brandon > and Charlie for comments. This is a great discussion, and I would like to > have much more of these in the future. > > To sum up what we have now: > > Roignac - Thanks a lot!! I think that this is a viable idea on its own, and > we may continue discussing it in some form. > Others (including Gema, Brendan, Charlie and myself) are not very fond of > it. Not because of its merit, but because we feel it will make the work > harder for manual testers. > > I would currently recommend the following course of action: continue with > the rewriting test cases in ISTQB format, and, maybe, add future plans for > exploring the possibilities for automation of those test cases, including > Gherking format. I don't think that it's necessarily mutually exclusive to > have both manual and automatic test cases. But I believe that we need to > concentrate on just one format currently (I prefer the one which is easier > for manual testers), and look for the possible expansions later. > > Great to have this discussion, it's refreshing after long period of low > traffic in QA. > > Thanks all. > > -- > Alex Lourie > > -- > Ubuntu-qa mailing list > Ubuntu-qa@lists.ubuntu.com > Modify settings or unsubscribe at: > https://lists.ubuntu.com/mailman/listinfo/ubuntu-qa > -- Ubuntu-qa mailing list Ubuntu-qa@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-qa
Re: Gherkin DSL for testcase description and automation
On Sat, Dec 10, 2011 at 7:56 PM, Gema Gomez wrote: > Hi, > > I agree with Charlie, but I think it is worth discussing this idea and > the reasons why (imo) this way of developing test cases wouldn't work > in our case. > > When I was reading about this method in this thread it altogether > reminded me of the days when we were trying to move the development of > Symbian OS to agile methodology, we were asked to plan in a way that > tasks had this format (also called stories in SCRUM): > > > As a I want so that (see > > http://www.thedailyscrum.co.uk/post/86171940/effective-story-writing-in-scrum > ) > > We underwent 2 years of intensive work trying to figure out how to do > testing in such environment and trying to figure out how system > testing, integration testing and unit testing fitted in this world. > And believe you me, the Ubuntu dev world is much more complex than > Symbian ever was. > > The first problem was that SCRUM doesn't really care about integration > and system testing, it cares only about unit testing. The only reason > I have been able to come up with for this is that SCRUM was initially > developed for small projects where developers could talk to each other > on daily basis plus the QA role wasn't a separate one. Only unit > testing was required. > > I learned that for a big project this is not enough and you still need > QA people and testers so that you get the integration of the product > and the packaging of it right. > > But Agile was cool and we were forced to move to it. We did, and we > implemented the lower layers of it, we wrote stories, tracked backlogs > and did daily stand ups like champions. When it came to write test > cases, ISTQB format still made sense for the whole company and was > widely used. Even for unit testing we were using that when tests were > written in something more formal than lines of code. > > The Gherkin method seems to be an evolution of Xtreme Programming or > Agile (en.wikipedia.org/wiki/Behavior_Driven_Development). It is used > for doing Behaviour Driven Development, i.e. the developer writes such > test case, then implements the test code or generates it with a > cucumber (I haven't figured out this bit but as funny as it sounds, it > seems to be the way... a code generator from stories, call me old > school, but I still prefer to write code with an editor) and then, > when the test case fails, they go to the code and create the piece of > software that will make that test case pass. That is unit testing, > doesn't matter which language you use to write the test case. > > As we've seen with the example that roignac put together (thanks!), > the language doesn't scale very well in terms of readability. Despite > making the test case logically very accurate, it doesn't really make > the manual tester's life easier, and at the end of the day, we are > trying to attract people to do more manual testing. Irrespective of > the subject having or not having a degree. It probably works for some > programmers, but definitely doesn't work for testers (imho). > > In a diverse environment like ours where developers use different > development life cycles (even though they all try to fit in the 6 > month bound releases), I am not sure how we would be able to push this > forward nor sure if we would want to do that if it is going to be more > complicated than just getting the wording of the test cases right and > accurate enough so that in front of the same behavior of the system, > two different people fail the test in the same way (consistency). > > Those are my thoughts. More ideas? > > Thanks, > Gema > > On 10/12/2011 14:43, Charlie Kravetz wrote: > > On Sat, 10 Dec 2011 12:35:21 +0200 Руаньяк > > wrote: > > > >> Hi, > > > >> 2011/12/10 Alex Lourie : > >>> I have the following questions: > >>> > >>> 1. How long does it take to write something like that for > >>> someone who's not a programmer and have no idea about Cucumber > >>> or unit testing in general? > >> I guess, it should not take much time. The only rule to be > >> followed is 'Action after When, Expected result after Then, > >> Concatenation is And'. Maybe, someone inexperienced in Gherkin > >> could measure the time and post results here? For instance, this > >> conversion of DesktopWhole case took about 10 min. for me. > > > > As someone completely ignorant of Gherkin, it took me over 10 > > minutes to read through this test case. In reading through, I > > attempted to picture most of the steps, and found the flow doesn't > > work right here. I test a lot of images, and I test images daily. > > This test is much too specific. Also, it does not read well, from > > an english language user. It reads much like something written by > > lawyers reads to the lay person. > > > > I don't think I could actually write such a test case, in less than > > at least an hour, and perhaps that is not enough enough time. > > > > > >>> 2. How would one execute this in LiveCD environment? > >> Manual teste
Re: Gherkin DSL for testcase description and automation
Hi, I agree with Charlie, but I think it is worth discussing this idea and the reasons why (imo) this way of developing test cases wouldn't work in our case. When I was reading about this method in this thread it altogether reminded me of the days when we were trying to move the development of Symbian OS to agile methodology, we were asked to plan in a way that tasks had this format (also called stories in SCRUM): > As a I want so that (see http://www.thedailyscrum.co.uk/post/86171940/effective-story-writing-in-scrum) We underwent 2 years of intensive work trying to figure out how to do testing in such environment and trying to figure out how system testing, integration testing and unit testing fitted in this world. And believe you me, the Ubuntu dev world is much more complex than Symbian ever was. The first problem was that SCRUM doesn't really care about integration and system testing, it cares only about unit testing. The only reason I have been able to come up with for this is that SCRUM was initially developed for small projects where developers could talk to each other on daily basis plus the QA role wasn't a separate one. Only unit testing was required. I learned that for a big project this is not enough and you still need QA people and testers so that you get the integration of the product and the packaging of it right. But Agile was cool and we were forced to move to it. We did, and we implemented the lower layers of it, we wrote stories, tracked backlogs and did daily stand ups like champions. When it came to write test cases, ISTQB format still made sense for the whole company and was widely used. Even for unit testing we were using that when tests were written in something more formal than lines of code. The Gherkin method seems to be an evolution of Xtreme Programming or Agile (en.wikipedia.org/wiki/Behavior_Driven_Development). It is used for doing Behaviour Driven Development, i.e. the developer writes such test case, then implements the test code or generates it with a cucumber (I haven't figured out this bit but as funny as it sounds, it seems to be the way... a code generator from stories, call me old school, but I still prefer to write code with an editor) and then, when the test case fails, they go to the code and create the piece of software that will make that test case pass. That is unit testing, doesn't matter which language you use to write the test case. As we've seen with the example that roignac put together (thanks!), the language doesn't scale very well in terms of readability. Despite making the test case logically very accurate, it doesn't really make the manual tester's life easier, and at the end of the day, we are trying to attract people to do more manual testing. Irrespective of the subject having or not having a degree. It probably works for some programmers, but definitely doesn't work for testers (imho). In a diverse environment like ours where developers use different development life cycles (even though they all try to fit in the 6 month bound releases), I am not sure how we would be able to push this forward nor sure if we would want to do that if it is going to be more complicated than just getting the wording of the test cases right and accurate enough so that in front of the same behavior of the system, two different people fail the test in the same way (consistency). Those are my thoughts. More ideas? Thanks, Gema On 10/12/2011 14:43, Charlie Kravetz wrote: > On Sat, 10 Dec 2011 12:35:21 +0200 Руаньяк > wrote: > >> Hi, > >> 2011/12/10 Alex Lourie : >>> I have the following questions: >>> >>> 1. How long does it take to write something like that for >>> someone who's not a programmer and have no idea about Cucumber >>> or unit testing in general? >> I guess, it should not take much time. The only rule to be >> followed is 'Action after When, Expected result after Then, >> Concatenation is And'. Maybe, someone inexperienced in Gherkin >> could measure the time and post results here? For instance, this >> conversion of DesktopWhole case took about 10 min. for me. > > As someone completely ignorant of Gherkin, it took me over 10 > minutes to read through this test case. In reading through, I > attempted to picture most of the steps, and found the flow doesn't > work right here. I test a lot of images, and I test images daily. > This test is much too specific. Also, it does not read well, from > an english language user. It reads much like something written by > lawyers reads to the lay person. > > I don't think I could actually write such a test case, in less than > at least an hour, and perhaps that is not enough enough time. > > >>> 2. How would one execute this in LiveCD environment? >> Manual testers may use this as a usual test case. Whenever bug >> appears, the tester may break the instruction, e.g. --- When I >> double click 'Install Ubuntu' icon Then Ubuquity starts #here we >> can check main window etc. > >> Result: Ubiquity cr
Re: Gherkin DSL for testcase description and automation
On 10/12/11 13:43, Charlie Kravetz wrote:
> On Sat, 10 Dec 2011 12:35:21 +0200
> Руаньяк wrote:
>
> > Hi,
>
> > 2011/12/10 Alex Lourie :
> >> I have the following questions:
> >>
> >> 1. How long does it take to write something like that for someone
> who's not
> >> a programmer and have no idea about Cucumber or unit testing in
> general?
> > I guess, it should not take much time. The only rule to be followed is
> > 'Action after When, Expected result after Then, Concatenation is And'.
> > Maybe, someone inexperienced in Gherkin could measure the time and
> > post results here? For instance, this conversion of DesktopWhole case
> > took about 10 min. for me.
>
> As someone completely ignorant of Gherkin, it took me over 10 minutes
> to read through this test case. In reading through, I attempted to
> picture most of the steps, and found the flow doesn't work right here.
> I test a lot of images, and I test images daily. This test is much too
> specific. Also, it does not read well, from an english language user.
> It reads much like something written by lawyers reads to the lay person.
I think this a perfect analogy. Yes, it is still English and yes, if you
have certain background then they are perfectly readable.
Let me just say thanks to Vadim for introducing us all to Gherkin - I
know that it's something I'll be taking a closer look at in future, but
I think the prerogative for the near future has to be to fix the flaws
in the current test cases, those being that they have out-of-date
instructions and often lack any sort of expected result, as well as a
clear objective - and the fact is that apart from these problems the
current test cases are pretty much in ISTQB form so the most
straightforward way to complete this exercise is going to be to continue
with that.
>
> I don't think I could actually write such a test case, in less than at
> least an hour, and perhaps that is not enough enough time.
>
>
> >> 2. How would one execute this in LiveCD environment?
> > Manual testers may use this as a usual test case. Whenever bug
> > appears, the tester may break the instruction, e.g.
> > ---
> > When I double click 'Install Ubuntu' icon
> > Then Ubuquity starts #here we can check main window etc.
>
> > Result: Ubiquity crashes with error 'ImportError:..."
> > ---
> >> 3. Is it possible to run something like that in automated VM
> environment?
> > Yes, as there is nose framework plugin Freshen (see
> > https://github.com/rlisagor/freshen), which generates xUnit reports
> > from feature files. Another option is Lettuce (see
> > https://github.com/gabrielfalcao/lettuce)
>
> > All steps should be defined (using ldtp) in definition file(s), which
> > contain the actual automation code for each step, something like this:
> > -- from steps.py --
> > @step("I double click 'Install Ubuntu' icon"):
> > def double_click_on_ubuntu_icon:
> > ltdp.double_click("btnInstallUbuntu");
>
> > Then, using steps definitions and scenarios, automation suite will do
> > the following:
> > a) create a virtual machine
> > b) start up from live cd
> > c) fire up LDTP server on virtual machine
> > d) connect to LDTP server and execute steps from scenarios
> > e) collect xUnit results and post them to CI (Jenkins)
>
> > I haven't tried this at home, so we should contact automation guys, as
> > they are using something similar for automation daily precise images
> > --
> > Vadim Rutkovsky
>
>
> While I realize that many of you are degree holders, community members
> as a rule are users who decided to get involved in helping their
> favorite software. For some, this is a way to give back for the ability
> to use the software.
>
> For the common user, these types of tests are not actually something
> they can follow easily. The test cases as written, using steps, are
> usable by all of us.
>
>
--
Ubuntu-qa mailing list
Ubuntu-qa@lists.ubuntu.com
Modify settings or unsubscribe at:
https://lists.ubuntu.com/mailman/listinfo/ubuntu-qa
Re: Gherkin DSL for testcase description and automation
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1
On Sat, 10 Dec 2011 12:35:21 +0200
Руаньяк wrote:
> Hi,
>
> 2011/12/10 Alex Lourie :
> > I have the following questions:
> >
> > 1. How long does it take to write something like that for someone who's not
> > a programmer and have no idea about Cucumber or unit testing in general?
> I guess, it should not take much time. The only rule to be followed is
> 'Action after When, Expected result after Then, Concatenation is And'.
> Maybe, someone inexperienced in Gherkin could measure the time and
> post results here? For instance, this conversion of DesktopWhole case
> took about 10 min. for me.
As someone completely ignorant of Gherkin, it took me over 10 minutes
to read through this test case. In reading through, I attempted to
picture most of the steps, and found the flow doesn't work right here.
I test a lot of images, and I test images daily. This test is much too
specific. Also, it does not read well, from an english language user.
It reads much like something written by lawyers reads to the lay person.
I don't think I could actually write such a test case, in less than at
least an hour, and perhaps that is not enough enough time.
>
> > 2. How would one execute this in LiveCD environment?
> Manual testers may use this as a usual test case. Whenever bug
> appears, the tester may break the instruction, e.g.
> ---
> When I double click 'Install Ubuntu' icon
> Then Ubuquity starts #here we can check main window etc.
>
> Result: Ubiquity crashes with error 'ImportError:..."
> ---
> > 3. Is it possible to run something like that in automated VM environment?
> Yes, as there is nose framework plugin Freshen (see
> https://github.com/rlisagor/freshen), which generates xUnit reports
> from feature files. Another option is Lettuce (see
> https://github.com/gabrielfalcao/lettuce)
>
> All steps should be defined (using ldtp) in definition file(s), which
> contain the actual automation code for each step, something like this:
> -- from steps.py --
> @step("I double click 'Install Ubuntu' icon"):
> def double_click_on_ubuntu_icon:
> ltdp.double_click("btnInstallUbuntu");
>
> Then, using steps definitions and scenarios, automation suite will do
> the following:
> a) create a virtual machine
> b) start up from live cd
> c) fire up LDTP server on virtual machine
> d) connect to LDTP server and execute steps from scenarios
> e) collect xUnit results and post them to CI (Jenkins)
>
> I haven't tried this at home, so we should contact automation guys, as
> they are using something similar for automation daily precise images
> --
> Vadim Rutkovsky
>
While I realize that many of you are degree holders, community members
as a rule are users who decided to get involved in helping their
favorite software. For some, this is a way to give back for the ability
to use the software.
For the common user, these types of tests are not actually something
they can follow easily. The test cases as written, using steps, are
usable by all of us.
- --
Charlie Kravetz
Linux Registered User Number 425914 [http://counter.li.org/]
Never let anyone steal your DREAM. [http://keepingdreams.com]
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.11 (GNU/Linux)
iQEcBAEBAgAGBQJO42HwAAoJEFNEIRz9dxbAhPkH/iML4W2sztkjnrw9poOR85sK
vy4rRZkErffZAcu7oGL30s1dZWHy1C/stXsKjov2eg9/CLOWQxC40S7h5MbKz25B
hIZ2nrGl5Ow2BiddVchTD3o3bfrxyP9WWIKw9+sZSN+MEEaz3lm969Ux717DCxkq
jGK3ShxhyUMyJ6lyh4/S9Pe+Npkp+xVGjCfAUr323v9UiR5c8uVPx3kqxgInqyxa
rWTuZRIv2nYo7YcbqOu21oUewqx/89SP+0nO5+HnQhErLq53t+/+5RmkQ9n4WV37
bIt7rRPbppGDt5UwASguinGPQivKhA0eyUskwe/upOSLVdolENhJOdHe3nY6XBQ=
=cbTu
-END PGP SIGNATURE-
--
Ubuntu-qa mailing list
Ubuntu-qa@lists.ubuntu.com
Modify settings or unsubscribe at:
https://lists.ubuntu.com/mailman/listinfo/ubuntu-qa
Re: Gherkin DSL for testcase description and automation
Hi,
2011/12/10 Alex Lourie :
> I have the following questions:
>
> 1. How long does it take to write something like that for someone who's not
> a programmer and have no idea about Cucumber or unit testing in general?
I guess, it should not take much time. The only rule to be followed is
'Action after When, Expected result after Then, Concatenation is And'.
Maybe, someone inexperienced in Gherkin could measure the time and
post results here? For instance, this conversion of DesktopWhole case
took about 10 min. for me.
> 2. How would one execute this in LiveCD environment?
Manual testers may use this as a usual test case. Whenever bug
appears, the tester may break the instruction, e.g.
---
When I double click 'Install Ubuntu' icon
Then Ubuquity starts #here we can check main window etc.
Result: Ubiquity crashes with error 'ImportError:..."
---
> 3. Is it possible to run something like that in automated VM environment?
Yes, as there is nose framework plugin Freshen (see
https://github.com/rlisagor/freshen), which generates xUnit reports
from feature files. Another option is Lettuce (see
https://github.com/gabrielfalcao/lettuce)
All steps should be defined (using ldtp) in definition file(s), which
contain the actual automation code for each step, something like this:
-- from steps.py --
@step("I double click 'Install Ubuntu' icon"):
def double_click_on_ubuntu_icon:
ltdp.double_click("btnInstallUbuntu");
Then, using steps definitions and scenarios, automation suite will do
the following:
a) create a virtual machine
b) start up from live cd
c) fire up LDTP server on virtual machine
d) connect to LDTP server and execute steps from scenarios
e) collect xUnit results and post them to CI (Jenkins)
I haven't tried this at home, so we should contact automation guys, as
they are using something similar for automation daily precise images
--
Vadim Rutkovsky
--
Ubuntu-qa mailing list
Ubuntu-qa@lists.ubuntu.com
Modify settings or unsubscribe at:
https://lists.ubuntu.com/mailman/listinfo/ubuntu-qa
Re: Gherkin DSL for testcase description and automation
On Sat, Dec 10, 2011 at 11:44 AM, Руаньяк wrote: > 2011/12/9 Gema Gomez : > > +1 to brendand for putting it so clearly. > > > > On the other hand, I am happy to do the intellectual exercise of trying > > to imagine what a long test case would look like with Gherkin method, > > and figure out if it would work. Can someone have a go at, let's say.. > > http://testcases.qa.ubuntu.com/Install/DesktopWhole > Here is a quick and dirty example. I marked comments with #, however, > they should be removed for automation suite. > > -- > Feature: Ubuntu Full Install > > Scenario: Default installation > > When I boot with Live CD > Then I see a desktop > And 'Install Ubuntu' icon is present > > When I double click 'Install Ubuntu' icon > Then Ubuquity starts #here we can check main window etc. > > When Welcome screen appears# add steps to check welcome screen > And I click 'Forward' button > > When 'Preparing to install Ubuntu' screen appears > Then Checkmark 'Your system have at least the amount a space indicated' is > set > And Checkmark 'Is plugged to a power source' is set > And Checkmark 'Is connected to the Internet' is set > > When I click 'Forward' > Then 'Allocate drive space' screen appears > > When I select 'Erase Disk and install Ubuntu' > Then option is selected # rewrite me, i look stupid! > > When I click 'Forward' > Then 'Erase disk and install Ubuntu' screen appears > And Selected item in 'Select drive' list is the drive on the chart > And Full drive space is allocated > > When I click 'Forward' button > Then 'Where are you?' screen appears > And Current city is set to 'Minsk' #Use variables here! > And Current timezone is set to 'GMT+3' #And here! > > When I click on 'London' city on the map > Then Current timezone is set to 'GMT' > And Current city is set to 'London' > > When I input 'Par' in city textbox > Then Suggestions dropdown list is opened > And Suggestion list contains 'Paris' item > > When I select 'Paris' from suggestion list > Then Current city is set to 'Paris' > And Current timezone is set to 'GMT+1' #Is it? Correct if it is not > > When I click 'Forward' button > Then 'Keyboard layout' screen appears > > When I select 'EN' layout #review me! > And I select 'EN-GB' variant #and me too, please! > And I press 't' button > Then 't' symbol appears in verification textbox #rewrite, looks ugly, huh? > > When I click 'Detect Keyboard Layout' button > And I answer all questions #expand me! > Then Proposed keyboard layout is 'En, En-gb' > > When I click 'Forward' button > Then 'Who are you?' screen appears > > When I input user details#expand - I input 'name' > as name etc. > And I click 'Forward' > Then Installation screen appears > > When I click left arrow on the side of the slide > Then Next slide is displayed > > When I click right arrow on the side of the slide > Then Previous slide is displayed > > When I wait for 1 minute > Then Next slide is displayed > > When Installer has finished its tasks > And I click 'Restart now' option > Then 'something about remove disk' dialog appears #oh my, its been > long since I installed Ubuntu! > > When I remove disk > And I press Enter > Then machine reboots > > When Grub is displayed > And Plymouth displays fancy animation > Then Login prompt is displayed > > When I input required credentials# expand - I input '' as login > etc. > Then User is logged in > > -- > Vadim Rutkovsky > > -- > Ubuntu-qa mailing list > Ubuntu-qa@lists.ubuntu.com > Modify settings or unsubscribe at: > https://lists.ubuntu.com/mailman/listinfo/ubuntu-qa > Hm, this seems to be quite interesting. I have the following questions: 1. How long does it take to write something like that for someone who's not a programmer and have no idea about Cucumber or unit testing in general? 2. How would one execute this in LiveCD environment? 3. Is it possible to run something like that in automated VM environment? Thanks. -- Alex Lourie -- Ubuntu-qa mailing list Ubuntu-qa@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-qa
Re: Gherkin DSL for testcase description and automation
2011/12/9 Gema Gomez : > +1 to brendand for putting it so clearly. > > On the other hand, I am happy to do the intellectual exercise of trying > to imagine what a long test case would look like with Gherkin method, > and figure out if it would work. Can someone have a go at, let's say.. > http://testcases.qa.ubuntu.com/Install/DesktopWhole Here is a quick and dirty example. I marked comments with #, however, they should be removed for automation suite. -- Feature: Ubuntu Full Install Scenario: Default installation When I boot with Live CD Then I see a desktop And 'Install Ubuntu' icon is present When I double click 'Install Ubuntu' icon Then Ubuquity starts #here we can check main window etc. When Welcome screen appears# add steps to check welcome screen And I click 'Forward' button When 'Preparing to install Ubuntu' screen appears Then Checkmark 'Your system have at least the amount a space indicated' is set And Checkmark 'Is plugged to a power source' is set And Checkmark 'Is connected to the Internet' is set When I click 'Forward' Then 'Allocate drive space' screen appears When I select 'Erase Disk and install Ubuntu' Then option is selected # rewrite me, i look stupid! When I click 'Forward' Then 'Erase disk and install Ubuntu' screen appears And Selected item in 'Select drive' list is the drive on the chart And Full drive space is allocated When I click 'Forward' button Then 'Where are you?' screen appears And Current city is set to 'Minsk' #Use variables here! And Current timezone is set to 'GMT+3' #And here! When I click on 'London' city on the map Then Current timezone is set to 'GMT' And Current city is set to 'London' When I input 'Par' in city textbox Then Suggestions dropdown list is opened And Suggestion list contains 'Paris' item When I select 'Paris' from suggestion list Then Current city is set to 'Paris' And Current timezone is set to 'GMT+1' #Is it? Correct if it is not When I click 'Forward' button Then 'Keyboard layout' screen appears When I select 'EN' layout #review me! And I select 'EN-GB' variant #and me too, please! And I press 't' button Then 't' symbol appears in verification textbox #rewrite, looks ugly, huh? When I click 'Detect Keyboard Layout' button And I answer all questions #expand me! Then Proposed keyboard layout is 'En, En-gb' When I click 'Forward' button Then 'Who are you?' screen appears When I input user details#expand - I input 'name' as name etc. And I click 'Forward' Then Installation screen appears When I click left arrow on the side of the slide Then Next slide is displayed When I click right arrow on the side of the slide Then Previous slide is displayed When I wait for 1 minute Then Next slide is displayed When Installer has finished its tasks And I click 'Restart now' option Then 'something about remove disk' dialog appears #oh my, its been long since I installed Ubuntu! When I remove disk And I press Enter Then machine reboots When Grub is displayed And Plymouth displays fancy animation Then Login prompt is displayed When I input required credentials# expand - I input '' as login etc. Then User is logged in -- Vadim Rutkovsky -- Ubuntu-qa mailing list Ubuntu-qa@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-qa
Re: Gherkin DSL for testcase description and automation
+1 to brendand for putting it so clearly.
I am not very keen on rewriting everything twice, we should be creating
test cases that whether manual or automated, are reusable and not need
changing unless the behavior of Ubuntu changes. Otherwise, we are
shooting ourselves on the foot... we will always be catching up.
On the other hand, I am happy to do the intellectual exercise of trying
to imagine what a long test case would look like with Gherkin method,
and figure out if it would work. Can someone have a go at, let's say..
http://testcases.qa.ubuntu.com/Install/DesktopWhole
I think we need to ask ourselves at which level are we creating test
cases here. My take, is that we are trying to do system testing, because
we test on Alpha and Beta releases, and it is too late already for any
other kind of testing. So we are targeting Ubuntu as a whole, either
Desktop or Server. Our test cases should run on any architecture, though
(i386, AMD64, ARM Server whenever it comes along?).
I am happy to contribute to Alex's wiki about how to write test cases
whenever it is in place.
Gherkin.. worth considering, but it doesn't look very promising for the
problem at hand.
Thanks,
Gema
On 09/12/2011 13:51, Руаньяк wrote:
> Hi all,
>
> 2011/12/9 Alex Lourie :
>>
>>
>> On Fri, Dec 9, 2011 at 12:41 PM, Brendan Donegan
>> wrote:
>>>
>>> Not sure I buy some of the criticisms of the ISTQB definitions, but it's
>>> certainly true that Gherkin encodes the tests in such a way as to make
>>> it possible for tools to parse them, whereas perhaps ISTQB definitions
>>> don't have that. I do think the language ends up looking slightly weird
>>> for manual testers who may not have that much experience with the
>>> software though. Here's a weakness, at least in the definitions given
>>> below:
>>>
>>> Scenario: TC_CAL-002: Calculate sums
>>> Given Calculator is in Basic mode
>>> When I input "1+1"
>>>And I calculate result <- How?
>>> Then Result is "2" <- Where?
>>>And No error occurs <- Where does the error not occur?
> Yes, a poorly written scenario here. Anyway, these steps can be
> re-written in any possible way.
> The most important parts are keywords - Given, When, Then, And, But
> etc. - and step consistency (mostly for automation).
> We can even use bzr+launchpad to store testcases and Launchpad Code
> Merge for testcase review and merge. That would be great to
> find/develop a system, which will present the tests in Gherkin in cool
> sexy-looking way, which will be at the same time a) easy for manual
> testers, b) useful for automated testing team.
>
>>> I will concede that those things are probably rectifiable though. We
>>> just need to acknowledge that given a.) time and resource constraints
>>> and b.) the very nature of certain tests, we won't be able to automate
>>> everything and whereas ISTQB provides a definition which is probably
>>> easier to understand for the manual tester but not so much for tools,
>>> Gherkin DSL provides one that is easier for tools to understand but
>>> maybe vulnerable to providing definitions that are more difficult for
>>> manual testers.
>>>
>>
>> So, in fact, we kind of need to decide who should we cater more with the
>> test cases - manual
>> testers or automatic ones? I think we need to handle manual ones now, and
>> try to move as much
>> as possible for automation later. So ISTQB definition would get my +1 for
>> the nearest future.
> I guess, Gherkin tests are not that hard to understand and they have
> another feature - they are much easier to create new ones (simply as
> they use less symbols ;)
> If we use Gherkin DSL for both manual and automated tests, we will
> avoid converstion from manual tests to automated unit-test - and this
> takes a lot of time, as I've been developing some tests for Mago
> recently.
> Simply, automation team just have to write actions for each step, for
> instance:
> @Given('Calculator is in (.+) mode')
> def switch_to_mode(mode):
>
> @When('I click "(.+)" button')
> def click_button(button):
>
> etc. etc.
>
>>
>>>
>>> It also seems that it's background is unit test generation and I'd be
>>> curious to see what a larger test case written in this syntax would look
>>> like.
>>>
>>
>> And so would I :-)
> I havn't written many tests using Gherkin, here are some Mago tests
> rewritten - lp:~roignac/+junk/lettuce_experiment
> --
> Vadim Rutkovsky
>
--
Ubuntu-qa mailing list
Ubuntu-qa@lists.ubuntu.com
Modify settings or unsubscribe at:
https://lists.ubuntu.com/mailman/listinfo/ubuntu-qa
Re: Gherkin DSL for testcase description and automation
Hi all,
2011/12/9 Alex Lourie :
>
>
> On Fri, Dec 9, 2011 at 12:41 PM, Brendan Donegan
> wrote:
>>
>> Not sure I buy some of the criticisms of the ISTQB definitions, but it's
>> certainly true that Gherkin encodes the tests in such a way as to make
>> it possible for tools to parse them, whereas perhaps ISTQB definitions
>> don't have that. I do think the language ends up looking slightly weird
>> for manual testers who may not have that much experience with the
>> software though. Here's a weakness, at least in the definitions given
>> below:
>>
>> Scenario: TC_CAL-002: Calculate sums
>> Given Calculator is in Basic mode
>> When I input "1+1"
>> And I calculate result <- How?
>> Then Result is "2" <- Where?
>> And No error occurs <- Where does the error not occur?
Yes, a poorly written scenario here. Anyway, these steps can be
re-written in any possible way.
The most important parts are keywords - Given, When, Then, And, But
etc. - and step consistency (mostly for automation).
We can even use bzr+launchpad to store testcases and Launchpad Code
Merge for testcase review and merge. That would be great to
find/develop a system, which will present the tests in Gherkin in cool
sexy-looking way, which will be at the same time a) easy for manual
testers, b) useful for automated testing team.
>> I will concede that those things are probably rectifiable though. We
>> just need to acknowledge that given a.) time and resource constraints
>> and b.) the very nature of certain tests, we won't be able to automate
>> everything and whereas ISTQB provides a definition which is probably
>> easier to understand for the manual tester but not so much for tools,
>> Gherkin DSL provides one that is easier for tools to understand but
>> maybe vulnerable to providing definitions that are more difficult for
>> manual testers.
>>
>
> So, in fact, we kind of need to decide who should we cater more with the
> test cases - manual
> testers or automatic ones? I think we need to handle manual ones now, and
> try to move as much
> as possible for automation later. So ISTQB definition would get my +1 for
> the nearest future.
I guess, Gherkin tests are not that hard to understand and they have
another feature - they are much easier to create new ones (simply as
they use less symbols ;)
If we use Gherkin DSL for both manual and automated tests, we will
avoid converstion from manual tests to automated unit-test - and this
takes a lot of time, as I've been developing some tests for Mago
recently.
Simply, automation team just have to write actions for each step, for instance:
@Given('Calculator is in (.+) mode')
def switch_to_mode(mode):
@When('I click "(.+)" button')
def click_button(button):
etc. etc.
>
>>
>> It also seems that it's background is unit test generation and I'd be
>> curious to see what a larger test case written in this syntax would look
>> like.
>>
>
> And so would I :-)
I havn't written many tests using Gherkin, here are some Mago tests
rewritten - lp:~roignac/+junk/lettuce_experiment
--
Vadim Rutkovsky
--
Ubuntu-qa mailing list
Ubuntu-qa@lists.ubuntu.com
Modify settings or unsubscribe at:
https://lists.ubuntu.com/mailman/listinfo/ubuntu-qa
Re: Gherkin DSL for testcase description and automation
On Fri, Dec 9, 2011 at 12:41 PM, Brendan Donegan < brendan.done...@canonical.com> wrote: > Not sure I buy some of the criticisms of the ISTQB definitions, but it's > certainly true that Gherkin encodes the tests in such a way as to make > it possible for tools to parse them, whereas perhaps ISTQB definitions > don't have that. I do think the language ends up looking slightly weird > for manual testers who may not have that much experience with the > software though. Here's a weakness, at least in the definitions given > below: > > Scenario: TC_CAL-002: Calculate sums > Given Calculator is in Basic mode > When I input "1+1" > And I calculate result <- How? > Then Result is "2" <- Where? >And No error occurs <- Where does the error not occur? > > > I will concede that those things are probably rectifiable though. We > just need to acknowledge that given a.) time and resource constraints > and b.) the very nature of certain tests, we won't be able to automate > everything and whereas ISTQB provides a definition which is probably > easier to understand for the manual tester but not so much for tools, > Gherkin DSL provides one that is easier for tools to understand but > maybe vulnerable to providing definitions that are more difficult for > manual testers. > > So, in fact, we kind of need to decide who should we cater more with the test cases - manual testers or automatic ones? I think we need to handle manual ones now, and try to move as much as possible for automation later. So ISTQB definition would get my +1 for the nearest future. > It also seems that it's background is unit test generation and I'd be > curious to see what a larger test case written in this syntax would look > like. > > And so would I :-) On 09/12/11 10:05, Руаньяк wrote: > > Hi all, > > > > After some hot debate in the mailing list about re-writing and > > categorizing testcases, I've decided to propose an adoption of another > > testcase DSL: Gherkin. > > Current ISTQB definition doesn't seem to be elegant enough to quickly > > describe the expected behavior of the system. As for me, these tests > > with actions separated from expected results are not very readable, as > > the tester has to do too many action while performing the test: a) > > perform the action, b) look down for expected result, c) verify the > > result, d) look up for another action and so on. > > Also, current testcases can be used for manual testing only and it is > > not directly suitable for Mago or any other automation framework. > > > > There is also another way to describe the behavior of the system, > > developed to be readable and directly used for automation: Gherkin > > DSL. See more info on the links sections > > > > This is a sample test case written using ISTQD definition for Gnome > Calculator: > > > > --- > > Test Case ID: TC-CAL-002 > > Test Case Description: Check that sums are calculated > > Assumptions: Gnome Calculator is started, mode is switched to the Basic > mode > > > > Actions: > > 1. I input "1+1" > > 2. I click '=' button > > Expected Results: > > 1. '1+1' appears in the calculator textbox > > 2. Result is 2 > > 3. No error appears > > --- > > As you may see, there several flaws in this testcase description: > > 1. As we are using lists in testcase description, it won't be easy to > > add more steps in the middle of the scenario, especially if testcase > > is long > > 2. We have to use 2 expected results (result is 2 and no error > > appears) for one action (I click '=' button), as there two things to > > be checked after this action. > > > > And here is a sample in Gherkin DSL: > > --- > > Feature: Basic Mode > > > > Scenario: TC_CAL-002: Calculate sums > >Given Calculator is in Basic mode > >When I input "1+1" > > And I calculate result > >Then Result is "2" > > And No error occurs > > --- > > > > This description is much more readable and can be easily copy-pasted > > in bugreport, as it doesn't have lost of unnecessary info, lists and > > can be broken anywhere with actual error description. > > > > Note, that automation suites can easily use the same scenarios. The > > only thing implemented is just actions for steps, which are basically > > procedures (using i.e ldtp), which use regexp to parse steps. If you > > guys need more info on this, I will write a separate post on > > automation with Gherkin DSL. > > > > There also some more interesting stuff in Gherkin - such as tagging, > > step argument transform and step reusing - I guess, I can describe > > them in other posts, if this whole propasl will be interesting. > > > > Hope to hear for more ideas and thoughts. > > Vadim Rutkovsky > > > > - > > Links: > > Gherkin DSL examples and structure: > > https://github.com/cucumber/cucumber/wiki/Gherkin > > http://code.google.com/p/spectacular/wiki/WritingBDDTests > > > http://en.wikipedia.org/wiki/Behavior_Driven_Development#Application_examples_in_the_Gherkin_language > > > > Sample automatio
Re: Gherkin DSL for testcase description and automation
Not sure I buy some of the criticisms of the ISTQB definitions, but it's certainly true that Gherkin encodes the tests in such a way as to make it possible for tools to parse them, whereas perhaps ISTQB definitions don't have that. I do think the language ends up looking slightly weird for manual testers who may not have that much experience with the software though. Here's a weakness, at least in the definitions given below: Scenario: TC_CAL-002: Calculate sums Given Calculator is in Basic mode When I input "1+1" And I calculate result <- How? Then Result is "2" <- Where? And No error occurs <- Where does the error not occur? I will concede that those things are probably rectifiable though. We just need to acknowledge that given a.) time and resource constraints and b.) the very nature of certain tests, we won't be able to automate everything and whereas ISTQB provides a definition which is probably easier to understand for the manual tester but not so much for tools, Gherkin DSL provides one that is easier for tools to understand but maybe vulnerable to providing definitions that are more difficult for manual testers. It also seems that it's background is unit test generation and I'd be curious to see what a larger test case written in this syntax would look like. On 09/12/11 10:05, Руаньяк wrote: > Hi all, > > After some hot debate in the mailing list about re-writing and > categorizing testcases, I've decided to propose an adoption of another > testcase DSL: Gherkin. > Current ISTQB definition doesn't seem to be elegant enough to quickly > describe the expected behavior of the system. As for me, these tests > with actions separated from expected results are not very readable, as > the tester has to do too many action while performing the test: a) > perform the action, b) look down for expected result, c) verify the > result, d) look up for another action and so on. > Also, current testcases can be used for manual testing only and it is > not directly suitable for Mago or any other automation framework. > > There is also another way to describe the behavior of the system, > developed to be readable and directly used for automation: Gherkin > DSL. See more info on the links sections > > This is a sample test case written using ISTQD definition for Gnome > Calculator: > > --- > Test Case ID: TC-CAL-002 > Test Case Description: Check that sums are calculated > Assumptions: Gnome Calculator is started, mode is switched to the Basic mode > > Actions: > 1. I input "1+1" > 2. I click '=' button > Expected Results: > 1. '1+1' appears in the calculator textbox > 2. Result is 2 > 3. No error appears > --- > As you may see, there several flaws in this testcase description: > 1. As we are using lists in testcase description, it won't be easy to > add more steps in the middle of the scenario, especially if testcase > is long > 2. We have to use 2 expected results (result is 2 and no error > appears) for one action (I click '=' button), as there two things to > be checked after this action. > > And here is a sample in Gherkin DSL: > --- > Feature: Basic Mode > > Scenario: TC_CAL-002: Calculate sums >Given Calculator is in Basic mode >When I input "1+1" > And I calculate result >Then Result is "2" > And No error occurs > --- > > This description is much more readable and can be easily copy-pasted > in bugreport, as it doesn't have lost of unnecessary info, lists and > can be broken anywhere with actual error description. > > Note, that automation suites can easily use the same scenarios. The > only thing implemented is just actions for steps, which are basically > procedures (using i.e ldtp), which use regexp to parse steps. If you > guys need more info on this, I will write a separate post on > automation with Gherkin DSL. > > There also some more interesting stuff in Gherkin - such as tagging, > step argument transform and step reusing - I guess, I can describe > them in other posts, if this whole propasl will be interesting. > > Hope to hear for more ideas and thoughts. > Vadim Rutkovsky > > - > Links: > Gherkin DSL examples and structure: > https://github.com/cucumber/cucumber/wiki/Gherkin > http://code.google.com/p/spectacular/wiki/WritingBDDTests > > http://en.wikipedia.org/wiki/Behavior_Driven_Development#Application_examples_in_the_Gherkin_language > > Sample automation code for GCalc: > https://code.launchpad.net/~roignac/+junk/freshen_mago > -- Ubuntu-qa mailing list Ubuntu-qa@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-qa
