Re: [openstack-dev] [Nova] Remove duplicate code using Data Driven Tests (DDT)
On 25 July 2016 at 13:56, Bhor, Dinesh <dinesh.b...@nttdata.com> wrote: > > > -Original Message- > From: Sean Dague [mailto:s...@dague.net] > Sent: Monday, July 25, 2016 5:53 PM > To: openstack-dev@lists.openstack.org > Subject: Re: [openstack-dev] [Nova] Remove duplicate code using Data Driven > Tests (DDT) > > On 07/25/2016 08:05 AM, Daniel P. Berrange wrote: >> On Mon, Jul 25, 2016 at 07:57:08AM -0400, Sean Dague wrote: >>> On 07/22/2016 11:30 AM, Daniel P. Berrange wrote: >>>> On Thu, Jul 21, 2016 at 07:03:53AM -0700, Matt Riedemann wrote: >>>>> On 7/21/2016 2:03 AM, Bhor, Dinesh wrote: >>>>> >>>>> I agree that it's not a bug. I also agree that it helps in some >>>>> specific types of tests which are doing some kind of input >>>>> validation (like the patch you've proposed) or are simply iterating >>>>> over some list of values (status values on a server instance for example). >>>>> >>>>> Using DDT in Nova has come up before and one of the concerns was >>>>> hiding details in how the tests are run with a library, and if >>>>> there would be a learning curve. Depending on the usage, I >>>>> personally don't have a problem with it. When I used it in manila >>>>> it took a little getting used to but I was basically just looking >>>>> at existing tests and figuring out what they were doing when adding new >>>>> ones. >>>> >>>> I don't think there's significant learning curve there - the way it >>>> lets you annotate the test methods is pretty easy to understand and >>>> the ddt docs spell it out clearly for newbies. We've far worse >>>> things in our code that create a hard learning curve which people >>>> will hit first :-) >>>> >>>> People have essentially been re-inventing ddt in nova tests already >>>> by defining one helper method and them having multiple tests methods >>>> all calling the same helper with a different dataset. So ddt is just >>>> formalizing what we're already doing in many places, with less code >>>> and greater clarity. >>>> >>>>> I definitely think DDT is easier to use/understand than something >>>>> like testscenarios, which we're already using in Nova. >>>> >>>> Yeah, testscenarios feels little over-engineered for what we want >>>> most of the time. >>> >>> Except, DDT is way less clear (and deterministic) about what's going >>> on with the test name munging. Which means failures are harder to >>> track back to individual tests and data load. So debugging the failures is >>> harder. >> >> I'm not sure what you think is unclear - given an annotated test: >> >>@ddt.data({"foo": "test", "availability_zone": "nova1"}, >> {"name": " test ", "availability_zone": "nova1"}, >> {"name": "", "availability_zone": "nova1"}, >> {"name": "x" * 256, "availability_zone": "nova1"}, >> {"name": "test", "availability_zone": "x" * 256}, >> {"name": "test", "availability_zone": " nova1 "}, >> {"name": "test", "availability_zone": ""}, >> {"name": "test", "availability_zone": "nova1", "foo": "bar"}) >> def test_create_invalid_create_aggregate_data(self, value): >> >> It is generated one test for each data item: >> >> test_create_invalid_create_aggregate_data_1 >> test_create_invalid_create_aggregate_data_2 >> test_create_invalid_create_aggregate_data_3 >> test_create_invalid_create_aggregate_data_4 >> test_create_invalid_create_aggregate_data_5 >> test_create_invalid_create_aggregate_data_6 >> test_create_invalid_create_aggregate_data_7 >> test_create_invalid_create_aggregate_data_8 >> >> This seems about as obvious as you can possibly get > > At least when this was attempted to be introduced into Tempest, the naming > was a lot less clear, maybe it got better. But I still think milestone 3 > isn't the time to start a thing like this. > > -Sean > > Hi Sean, > > IMO it is possibl
Re: [openstack-dev] [Nova] Remove duplicate code using Data Driven Tests (DDT)
On Mon, Jul 25, 2016 at 08:22:52AM -0400, Sean Dague wrote: > On 07/25/2016 08:05 AM, Daniel P. Berrange wrote: > > On Mon, Jul 25, 2016 at 07:57:08AM -0400, Sean Dague wrote: > >> On 07/22/2016 11:30 AM, Daniel P. Berrange wrote: > >>> On Thu, Jul 21, 2016 at 07:03:53AM -0700, Matt Riedemann wrote: > On 7/21/2016 2:03 AM, Bhor, Dinesh wrote: > > I agree that it's not a bug. I also agree that it helps in some specific > types of tests which are doing some kind of input validation (like the > patch > you've proposed) or are simply iterating over some list of values (status > values on a server instance for example). > > Using DDT in Nova has come up before and one of the concerns was hiding > details in how the tests are run with a library, and if there would be a > learning curve. Depending on the usage, I personally don't have a problem > with it. When I used it in manila it took a little getting used to but I > was > basically just looking at existing tests and figuring out what they were > doing when adding new ones. > >>> > >>> I don't think there's significant learning curve there - the way it > >>> lets you annotate the test methods is pretty easy to understand and > >>> the ddt docs spell it out clearly for newbies. We've far worse things > >>> in our code that create a hard learning curve which people will hit > >>> first :-) > >>> > >>> People have essentially been re-inventing ddt in nova tests already > >>> by defining one helper method and them having multiple tests methods > >>> all calling the same helper with a different dataset. So ddt is just > >>> formalizing what we're already doing in many places, with less code > >>> and greater clarity. > >>> > I definitely think DDT is easier to use/understand than something like > testscenarios, which we're already using in Nova. > >>> > >>> Yeah, testscenarios feels little over-engineered for what we want most > >>> of the time. > >> > >> Except, DDT is way less clear (and deterministic) about what's going on > >> with the test name munging. Which means failures are harder to track > >> back to individual tests and data load. So debugging the failures is > >> harder. > > > > I'm not sure what you think is unclear - given an annotated test: > > > >@ddt.data({"foo": "test", "availability_zone": "nova1"}, > > {"name": " test ", "availability_zone": "nova1"}, > > {"name": "", "availability_zone": "nova1"}, > > {"name": "x" * 256, "availability_zone": "nova1"}, > > {"name": "test", "availability_zone": "x" * 256}, > > {"name": "test", "availability_zone": " nova1 "}, > > {"name": "test", "availability_zone": ""}, > > {"name": "test", "availability_zone": "nova1", "foo": "bar"}) > > def test_create_invalid_create_aggregate_data(self, value): > > > > It is generated one test for each data item: > > > > test_create_invalid_create_aggregate_data_1 > > test_create_invalid_create_aggregate_data_2 > > test_create_invalid_create_aggregate_data_3 > > test_create_invalid_create_aggregate_data_4 > > test_create_invalid_create_aggregate_data_5 > > test_create_invalid_create_aggregate_data_6 > > test_create_invalid_create_aggregate_data_7 > > test_create_invalid_create_aggregate_data_8 > > > > This seems about as obvious as you can possibly get > > At least when this was attempted to be introduced into Tempest, the > naming was a lot less clear, maybe it got better. But I still think > milestone 3 isn't the time to start a thing like this. Historically we've allowed patches that improve / adapt unit tests to be merged at any time that we're not in final bug-fix only freeze periods. So on this basis, I'm happy to see this accepted now, especially since the module is already in global requirements, so not a new thing from an openstack POV Regards, Daniel -- |: http://berrange.com -o-http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :| __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Nova] Remove duplicate code using Data Driven Tests (DDT)
-Original Message- From: Sean Dague [mailto:s...@dague.net] Sent: Monday, July 25, 2016 5:53 PM To: openstack-dev@lists.openstack.org Subject: Re: [openstack-dev] [Nova] Remove duplicate code using Data Driven Tests (DDT) On 07/25/2016 08:05 AM, Daniel P. Berrange wrote: > On Mon, Jul 25, 2016 at 07:57:08AM -0400, Sean Dague wrote: >> On 07/22/2016 11:30 AM, Daniel P. Berrange wrote: >>> On Thu, Jul 21, 2016 at 07:03:53AM -0700, Matt Riedemann wrote: >>>> On 7/21/2016 2:03 AM, Bhor, Dinesh wrote: >>>> >>>> I agree that it's not a bug. I also agree that it helps in some >>>> specific types of tests which are doing some kind of input >>>> validation (like the patch you've proposed) or are simply iterating >>>> over some list of values (status values on a server instance for example). >>>> >>>> Using DDT in Nova has come up before and one of the concerns was >>>> hiding details in how the tests are run with a library, and if >>>> there would be a learning curve. Depending on the usage, I >>>> personally don't have a problem with it. When I used it in manila >>>> it took a little getting used to but I was basically just looking >>>> at existing tests and figuring out what they were doing when adding new >>>> ones. >>> >>> I don't think there's significant learning curve there - the way it >>> lets you annotate the test methods is pretty easy to understand and >>> the ddt docs spell it out clearly for newbies. We've far worse >>> things in our code that create a hard learning curve which people >>> will hit first :-) >>> >>> People have essentially been re-inventing ddt in nova tests already >>> by defining one helper method and them having multiple tests methods >>> all calling the same helper with a different dataset. So ddt is just >>> formalizing what we're already doing in many places, with less code >>> and greater clarity. >>> >>>> I definitely think DDT is easier to use/understand than something >>>> like testscenarios, which we're already using in Nova. >>> >>> Yeah, testscenarios feels little over-engineered for what we want >>> most of the time. >> >> Except, DDT is way less clear (and deterministic) about what's going >> on with the test name munging. Which means failures are harder to >> track back to individual tests and data load. So debugging the failures is >> harder. > > I'm not sure what you think is unclear - given an annotated test: > >@ddt.data({"foo": "test", "availability_zone": "nova1"}, > {"name": " test ", "availability_zone": "nova1"}, > {"name": "", "availability_zone": "nova1"}, > {"name": "x" * 256, "availability_zone": "nova1"}, > {"name": "test", "availability_zone": "x" * 256}, > {"name": "test", "availability_zone": " nova1 "}, > {"name": "test", "availability_zone": ""}, > {"name": "test", "availability_zone": "nova1", "foo": "bar"}) > def test_create_invalid_create_aggregate_data(self, value): > > It is generated one test for each data item: > > test_create_invalid_create_aggregate_data_1 > test_create_invalid_create_aggregate_data_2 > test_create_invalid_create_aggregate_data_3 > test_create_invalid_create_aggregate_data_4 > test_create_invalid_create_aggregate_data_5 > test_create_invalid_create_aggregate_data_6 > test_create_invalid_create_aggregate_data_7 > test_create_invalid_create_aggregate_data_8 > > This seems about as obvious as you can possibly get At least when this was attempted to be introduced into Tempest, the naming was a lot less clear, maybe it got better. But I still think milestone 3 isn't the time to start a thing like this. -Sean Hi Sean, IMO it is possible to have a descriptive name to test cases using DDT. For ex., @ddt.data(annotated('missing_name', {"foo": "test", "availability_zone": "nova1"}), annotated('name_greater_than_255_characters', {"name": "x" * 256, "availability_zone": "nova1"})) def test_create_invalid_aggregate_data(self, value):
Re: [openstack-dev] [Nova] Remove duplicate code using Data Driven Tests (DDT)
On 07/25/2016 08:05 AM, Daniel P. Berrange wrote: > On Mon, Jul 25, 2016 at 07:57:08AM -0400, Sean Dague wrote: >> On 07/22/2016 11:30 AM, Daniel P. Berrange wrote: >>> On Thu, Jul 21, 2016 at 07:03:53AM -0700, Matt Riedemann wrote: On 7/21/2016 2:03 AM, Bhor, Dinesh wrote: I agree that it's not a bug. I also agree that it helps in some specific types of tests which are doing some kind of input validation (like the patch you've proposed) or are simply iterating over some list of values (status values on a server instance for example). Using DDT in Nova has come up before and one of the concerns was hiding details in how the tests are run with a library, and if there would be a learning curve. Depending on the usage, I personally don't have a problem with it. When I used it in manila it took a little getting used to but I was basically just looking at existing tests and figuring out what they were doing when adding new ones. >>> >>> I don't think there's significant learning curve there - the way it >>> lets you annotate the test methods is pretty easy to understand and >>> the ddt docs spell it out clearly for newbies. We've far worse things >>> in our code that create a hard learning curve which people will hit >>> first :-) >>> >>> People have essentially been re-inventing ddt in nova tests already >>> by defining one helper method and them having multiple tests methods >>> all calling the same helper with a different dataset. So ddt is just >>> formalizing what we're already doing in many places, with less code >>> and greater clarity. >>> I definitely think DDT is easier to use/understand than something like testscenarios, which we're already using in Nova. >>> >>> Yeah, testscenarios feels little over-engineered for what we want most >>> of the time. >> >> Except, DDT is way less clear (and deterministic) about what's going on >> with the test name munging. Which means failures are harder to track >> back to individual tests and data load. So debugging the failures is harder. > > I'm not sure what you think is unclear - given an annotated test: > >@ddt.data({"foo": "test", "availability_zone": "nova1"}, > {"name": " test ", "availability_zone": "nova1"}, > {"name": "", "availability_zone": "nova1"}, > {"name": "x" * 256, "availability_zone": "nova1"}, > {"name": "test", "availability_zone": "x" * 256}, > {"name": "test", "availability_zone": " nova1 "}, > {"name": "test", "availability_zone": ""}, > {"name": "test", "availability_zone": "nova1", "foo": "bar"}) > def test_create_invalid_create_aggregate_data(self, value): > > It is generated one test for each data item: > > test_create_invalid_create_aggregate_data_1 > test_create_invalid_create_aggregate_data_2 > test_create_invalid_create_aggregate_data_3 > test_create_invalid_create_aggregate_data_4 > test_create_invalid_create_aggregate_data_5 > test_create_invalid_create_aggregate_data_6 > test_create_invalid_create_aggregate_data_7 > test_create_invalid_create_aggregate_data_8 > > This seems about as obvious as you can possibly get At least when this was attempted to be introduced into Tempest, the naming was a lot less clear, maybe it got better. But I still think milestone 3 isn't the time to start a thing like this. -Sean -- Sean Dague http://dague.net __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Nova] Remove duplicate code using Data Driven Tests (DDT)
On Mon, Jul 25, 2016 at 07:57:08AM -0400, Sean Dague wrote: > On 07/22/2016 11:30 AM, Daniel P. Berrange wrote: > > On Thu, Jul 21, 2016 at 07:03:53AM -0700, Matt Riedemann wrote: > >> On 7/21/2016 2:03 AM, Bhor, Dinesh wrote: > >> > >> I agree that it's not a bug. I also agree that it helps in some specific > >> types of tests which are doing some kind of input validation (like the > >> patch > >> you've proposed) or are simply iterating over some list of values (status > >> values on a server instance for example). > >> > >> Using DDT in Nova has come up before and one of the concerns was hiding > >> details in how the tests are run with a library, and if there would be a > >> learning curve. Depending on the usage, I personally don't have a problem > >> with it. When I used it in manila it took a little getting used to but I > >> was > >> basically just looking at existing tests and figuring out what they were > >> doing when adding new ones. > > > > I don't think there's significant learning curve there - the way it > > lets you annotate the test methods is pretty easy to understand and > > the ddt docs spell it out clearly for newbies. We've far worse things > > in our code that create a hard learning curve which people will hit > > first :-) > > > > People have essentially been re-inventing ddt in nova tests already > > by defining one helper method and them having multiple tests methods > > all calling the same helper with a different dataset. So ddt is just > > formalizing what we're already doing in many places, with less code > > and greater clarity. > > > >> I definitely think DDT is easier to use/understand than something like > >> testscenarios, which we're already using in Nova. > > > > Yeah, testscenarios feels little over-engineered for what we want most > > of the time. > > Except, DDT is way less clear (and deterministic) about what's going on > with the test name munging. Which means failures are harder to track > back to individual tests and data load. So debugging the failures is harder. I'm not sure what you think is unclear - given an annotated test: @ddt.data({"foo": "test", "availability_zone": "nova1"}, {"name": " test ", "availability_zone": "nova1"}, {"name": "", "availability_zone": "nova1"}, {"name": "x" * 256, "availability_zone": "nova1"}, {"name": "test", "availability_zone": "x" * 256}, {"name": "test", "availability_zone": " nova1 "}, {"name": "test", "availability_zone": ""}, {"name": "test", "availability_zone": "nova1", "foo": "bar"}) def test_create_invalid_create_aggregate_data(self, value): It is generated one test for each data item: test_create_invalid_create_aggregate_data_1 test_create_invalid_create_aggregate_data_2 test_create_invalid_create_aggregate_data_3 test_create_invalid_create_aggregate_data_4 test_create_invalid_create_aggregate_data_5 test_create_invalid_create_aggregate_data_6 test_create_invalid_create_aggregate_data_7 test_create_invalid_create_aggregate_data_8 This seems about as obvious as you can possibly get Regards, Daniel -- |: http://berrange.com -o-http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :| __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Nova] Remove duplicate code using Data Driven Tests (DDT)
On 07/25/2016 07:57 AM, Sean Dague wrote: On 07/22/2016 11:30 AM, Daniel P. Berrange wrote: On Thu, Jul 21, 2016 at 07:03:53AM -0700, Matt Riedemann wrote: On 7/21/2016 2:03 AM, Bhor, Dinesh wrote: I agree that it's not a bug. I also agree that it helps in some specific types of tests which are doing some kind of input validation (like the patch you've proposed) or are simply iterating over some list of values (status values on a server instance for example). Using DDT in Nova has come up before and one of the concerns was hiding details in how the tests are run with a library, and if there would be a learning curve. Depending on the usage, I personally don't have a problem with it. When I used it in manila it took a little getting used to but I was basically just looking at existing tests and figuring out what they were doing when adding new ones. I don't think there's significant learning curve there - the way it lets you annotate the test methods is pretty easy to understand and the ddt docs spell it out clearly for newbies. We've far worse things in our code that create a hard learning curve which people will hit first :-) People have essentially been re-inventing ddt in nova tests already by defining one helper method and them having multiple tests methods all calling the same helper with a different dataset. So ddt is just formalizing what we're already doing in many places, with less code and greater clarity. I definitely think DDT is easier to use/understand than something like testscenarios, which we're already using in Nova. Yeah, testscenarios feels little over-engineered for what we want most of the time. Except, DDT is way less clear (and deterministic) about what's going on with the test name munging. Which means failures are harder to track back to individual tests and data load. So debugging the failures is harder. I agree with have a lot of bad patterns in the tests. But I also don't think that embedding another pattern during milestone 3 is the right time to do it. At least lets hold until next cycle opens up when there is more time to actually look at trade offs here. +1 Also, I actually don't see how testscenarios won't/can't work for everything DDT is doing. Sounds a bit like the "why can't we use pytest instead of testr?" thing again. Best, -jay __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Nova] Remove duplicate code using Data Driven Tests (DDT)
On 07/22/2016 11:30 AM, Daniel P. Berrange wrote: > On Thu, Jul 21, 2016 at 07:03:53AM -0700, Matt Riedemann wrote: >> On 7/21/2016 2:03 AM, Bhor, Dinesh wrote: >> >> I agree that it's not a bug. I also agree that it helps in some specific >> types of tests which are doing some kind of input validation (like the patch >> you've proposed) or are simply iterating over some list of values (status >> values on a server instance for example). >> >> Using DDT in Nova has come up before and one of the concerns was hiding >> details in how the tests are run with a library, and if there would be a >> learning curve. Depending on the usage, I personally don't have a problem >> with it. When I used it in manila it took a little getting used to but I was >> basically just looking at existing tests and figuring out what they were >> doing when adding new ones. > > I don't think there's significant learning curve there - the way it > lets you annotate the test methods is pretty easy to understand and > the ddt docs spell it out clearly for newbies. We've far worse things > in our code that create a hard learning curve which people will hit > first :-) > > People have essentially been re-inventing ddt in nova tests already > by defining one helper method and them having multiple tests methods > all calling the same helper with a different dataset. So ddt is just > formalizing what we're already doing in many places, with less code > and greater clarity. > >> I definitely think DDT is easier to use/understand than something like >> testscenarios, which we're already using in Nova. > > Yeah, testscenarios feels little over-engineered for what we want most > of the time. Except, DDT is way less clear (and deterministic) about what's going on with the test name munging. Which means failures are harder to track back to individual tests and data load. So debugging the failures is harder. I agree with have a lot of bad patterns in the tests. But I also don't think that embedding another pattern during milestone 3 is the right time to do it. At least lets hold until next cycle opens up when there is more time to actually look at trade offs here. -Sean -- Sean Dague http://dague.net __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Nova] Remove duplicate code using Data Driven Tests (DDT)
On Thu, Jul 21, 2016 at 07:03:53AM -0700, Matt Riedemann wrote: > On 7/21/2016 2:03 AM, Bhor, Dinesh wrote: > > Hi Nova Devs, > > > > > > > > Many times, there are a number of data sets that we have to run the same > > tests on. > > > > And, to create a different test for each data set values is > > time-consuming and inefficient. > > > > > > > > Data Driven Testing [1] overcomes this issue. Data-driven testing (DDT) > > is taking a test, > > > > parameterizing it and then running that test with varying data. This > > allows you to run the > > > > same test case with many varying inputs, therefore increasing coverage > > from a single test, > > > > reduces code duplication and can ease up error tracing as well. > > > > > > > > DDT is a third party library needs to be installed separately and invoke the > > > > module when writing the tests. At present DDT is used in cinder and rally. > > There are several projects using it: > > http://codesearch.openstack.org/?q=ddt%3E%3D1.0.1=nope== > > I first came across it when working a little in manila. > > > > > > > > > To start with, I have reported this as a bug [2] and added initial patch > > [3] for the same, > > > > but couple of reviewers has suggested to discuss about this on ML as > > this is not a real bug. > > > > IMO this is not a feature implementation and it’s just a effort for > > simplifying our tests, > > > > so a blueprint will be sufficient to track its progress. > > > > > > > > So please let me know whether I can file a new blueprint or nova-specs > > to proceed with this. > > > > > > > > [1] http://ddt.readthedocs.io/en/latest/index.html > > > > [2] https://bugs.launchpad.net/nova/+bug/1604798 > > > > [3] https://review.openstack.org/#/c/344820/ > > > > > > > > Thank you, > > > > Dinesh Bhor > > > > > > __ > > Disclaimer: This email and any attachments are sent in strictest confidence > > for the sole use of the addressee and may contain legally privileged, > > confidential, and proprietary data. If you are not the intended recipient, > > please advise the sender by replying promptly to this email and then delete > > and destroy this email and any attachments without any further use, copying > > or forwarding. > > > > > > __ > > OpenStack Development Mailing List (not for usage questions) > > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > > > I agree that it's not a bug. I also agree that it helps in some specific > types of tests which are doing some kind of input validation (like the patch > you've proposed) or are simply iterating over some list of values (status > values on a server instance for example). > > Using DDT in Nova has come up before and one of the concerns was hiding > details in how the tests are run with a library, and if there would be a > learning curve. Depending on the usage, I personally don't have a problem > with it. When I used it in manila it took a little getting used to but I was > basically just looking at existing tests and figuring out what they were > doing when adding new ones. I don't think there's significant learning curve there - the way it lets you annotate the test methods is pretty easy to understand and the ddt docs spell it out clearly for newbies. We've far worse things in our code that create a hard learning curve which people will hit first :-) People have essentially been re-inventing ddt in nova tests already by defining one helper method and them having multiple tests methods all calling the same helper with a different dataset. So ddt is just formalizing what we're already doing in many places, with less code and greater clarity. > I definitely think DDT is easier to use/understand than something like > testscenarios, which we're already using in Nova. Yeah, testscenarios feels little over-engineered for what we want most of the time. Regards, Daniel -- |: http://berrange.com -o-http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :| __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Nova] Remove duplicate code using Data Driven Tests (DDT)
On 7/21/2016 2:03 AM, Bhor, Dinesh wrote: Hi Nova Devs, Many times, there are a number of data sets that we have to run the same tests on. And, to create a different test for each data set values is time-consuming and inefficient. Data Driven Testing [1] overcomes this issue. Data-driven testing (DDT) is taking a test, parameterizing it and then running that test with varying data. This allows you to run the same test case with many varying inputs, therefore increasing coverage from a single test, reduces code duplication and can ease up error tracing as well. DDT is a third party library needs to be installed separately and invoke the module when writing the tests. At present DDT is used in cinder and rally. There are several projects using it: http://codesearch.openstack.org/?q=ddt%3E%3D1.0.1=nope== I first came across it when working a little in manila. To start with, I have reported this as a bug [2] and added initial patch [3] for the same, but couple of reviewers has suggested to discuss about this on ML as this is not a real bug. IMO this is not a feature implementation and it’s just a effort for simplifying our tests, so a blueprint will be sufficient to track its progress. So please let me know whether I can file a new blueprint or nova-specs to proceed with this. [1] http://ddt.readthedocs.io/en/latest/index.html [2] https://bugs.launchpad.net/nova/+bug/1604798 [3] https://review.openstack.org/#/c/344820/ Thank you, Dinesh Bhor __ Disclaimer: This email and any attachments are sent in strictest confidence for the sole use of the addressee and may contain legally privileged, confidential, and proprietary data. If you are not the intended recipient, please advise the sender by replying promptly to this email and then delete and destroy this email and any attachments without any further use, copying or forwarding. __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev I agree that it's not a bug. I also agree that it helps in some specific types of tests which are doing some kind of input validation (like the patch you've proposed) or are simply iterating over some list of values (status values on a server instance for example). Using DDT in Nova has come up before and one of the concerns was hiding details in how the tests are run with a library, and if there would be a learning curve. Depending on the usage, I personally don't have a problem with it. When I used it in manila it took a little getting used to but I was basically just looking at existing tests and figuring out what they were doing when adding new ones. I definitely think DDT is easier to use/understand than something like testscenarios, which we're already using in Nova. -- Thanks, Matt Riedemann __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Nova] Remove duplicate code using Data Driven Tests (DDT)
Hi Dinesh Bhor, I could see a lot of projects have used this. Seems interesting. You can refer Cinder, for example. I think we should not raise bug for this. For me it's invalid. You can treat this as improvement(just to reduce test code blocks), it seems ok to go ahead even without any BP or spec. Best Regards, Sheel Rana On Thu, Jul 21, 2016 at 2:33 PM, Bhor, Dineshwrote: > Hi Nova Devs, > > > > Many times, there are a number of data sets that we have to run the same > tests on. > > And, to create a different test for each data set values is time-consuming > and inefficient. > > > > Data Driven Testing [1] overcomes this issue. Data-driven testing (DDT) is > taking a test, > > parameterizing it and then running that test with varying data. This > allows you to run the > > same test case with many varying inputs, therefore increasing coverage > from a single test, > > reduces code duplication and can ease up error tracing as well. > > > > DDT is a third party library needs to be installed separately and invoke > the > > module when writing the tests. At present DDT is used in cinder and rally. > > > > To start with, I have reported this as a bug [2] and added initial patch > [3] for the same, > > but couple of reviewers has suggested to discuss about this on ML as this > is not a real bug. > > IMO this is not a feature implementation and it’s just a effort for > simplifying our tests, > > so a blueprint will be sufficient to track its progress. > > > > So please let me know whether I can file a new blueprint or nova-specs to > proceed with this. > > > > [1] http://ddt.readthedocs.io/en/latest/index.html > > [2] https://bugs.launchpad.net/nova/+bug/1604798 > > [3] https://review.openstack.org/#/c/344820/ > > > > Thank you, > > Dinesh Bhor > > __ > Disclaimer: This email and any attachments are sent in strictest confidence > for the sole use of the addressee and may contain legally privileged, > confidential, and proprietary data. If you are not the intended recipient, > please advise the sender by replying promptly to this email and then delete > and destroy this email and any attachments without any further use, copying > or forwarding. > > __ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [Nova] Remove duplicate code using Data Driven Tests (DDT)
Hi Nova Devs, Many times, there are a number of data sets that we have to run the same tests on. And, to create a different test for each data set values is time-consuming and inefficient. Data Driven Testing [1] overcomes this issue. Data-driven testing (DDT) is taking a test, parameterizing it and then running that test with varying data. This allows you to run the same test case with many varying inputs, therefore increasing coverage from a single test, reduces code duplication and can ease up error tracing as well. DDT is a third party library needs to be installed separately and invoke the module when writing the tests. At present DDT is used in cinder and rally. To start with, I have reported this as a bug [2] and added initial patch [3] for the same, but couple of reviewers has suggested to discuss about this on ML as this is not a real bug. IMO this is not a feature implementation and it's just a effort for simplifying our tests, so a blueprint will be sufficient to track its progress. So please let me know whether I can file a new blueprint or nova-specs to proceed with this. [1] http://ddt.readthedocs.io/en/latest/index.html [2] https://bugs.launchpad.net/nova/+bug/1604798 [3] https://review.openstack.org/#/c/344820/ Thank you, Dinesh Bhor __ Disclaimer: This email and any attachments are sent in strictest confidence for the sole use of the addressee and may contain legally privileged, confidential, and proprietary data. If you are not the intended recipient, please advise the sender by replying promptly to this email and then delete and destroy this email and any attachments without any further use, copying or forwarding.__ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev