Re: [openstack-dev] [nova] novaclient functional test guidelines
First pass at trying to capture this thread into a README: https://review.openstack.org/162334 On Tue, Feb 24, 2015 at 2:07 PM, Joe Gordon joe.gord...@gmail.com wrote: On Tue, Feb 24, 2015 at 1:18 PM, melanie witt melwi...@gmail.com wrote: On Feb 24, 2015, at 9:47, Sean Dague s...@dague.net wrote: I'm happy if there are other theories about how we do these things, being the first functional test in the python-novaclient tree that creates and destroys real resources, there isn't an established pattern yet. But I think doing all CLI calls in CLI tests is actually really cumbersome, especially in the amount of output parsing code needed if you are going to setup any complicated resource structure. I think I'm in agreement with the pattern you describe. I imagine having a set of functional tests for the API, that don't do any CLI calls at all. With that we test that the API works properly. Then have a separate set for the CLI, which only calls CLI for the command being tested, everything else to set up and tear down the test done by API calls. This would be done with the rationale that because the entire API functionality is tested separately, we can safely use it for setup/teardown with the intent to isolate the CLI test to the command being tested and avoid introducing side effects from the CLI commands. But I suppose one could make the same argument for using CLI everywhere (if they are all tested, they can also be trusted not to introduce side effects). I tend to favor using the API because it's the most bare bones setup/teardown we could use. At the same time I understand the idea of performing an entire test using the CLI, as a way of replicating the experience a real user might have using the CLI, from start to end. I don't think I feel strongly either way. I guess its time to revisit the actual status of novaclient and if we want to actively move away from it to openstacksdk/OSC as well. If we are actively trying to move away from novaclient, using the python API as much as possible makes a lot of sense. For the --poll stuff, I agree the API should have it and the CLI uses it. And with and without poll functionality should be tested separately, API and CLI. melanie (melwitt) __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova] novaclient functional test guidelines
On Feb 24, 2015, at 9:47, Sean Dague s...@dague.net wrote: I'm happy if there are other theories about how we do these things, being the first functional test in the python-novaclient tree that creates and destroys real resources, there isn't an established pattern yet. But I think doing all CLI calls in CLI tests is actually really cumbersome, especially in the amount of output parsing code needed if you are going to setup any complicated resource structure. I think I'm in agreement with the pattern you describe. I imagine having a set of functional tests for the API, that don't do any CLI calls at all. With that we test that the API works properly. Then have a separate set for the CLI, which only calls CLI for the command being tested, everything else to set up and tear down the test done by API calls. This would be done with the rationale that because the entire API functionality is tested separately, we can safely use it for setup/teardown with the intent to isolate the CLI test to the command being tested and avoid introducing side effects from the CLI commands. But I suppose one could make the same argument for using CLI everywhere (if they are all tested, they can also be trusted not to introduce side effects). I tend to favor using the API because it's the most bare bones setup/teardown we could use. At the same time I understand the idea of performing an entire test using the CLI, as a way of replicating the experience a real user might have using the CLI, from start to end. I don't think I feel strongly either way. For the --poll stuff, I agree the API should have it and the CLI uses it. And with and without poll functionality should be tested separately, API and CLI. melanie (melwitt) signature.asc Description: Message signed with OpenPGP using GPGMail __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova] novaclient functional test guidelines
On Feb 24, 2015, at 2:30 PM, Sean Dague s...@dague.net wrote: Right, I think to some degree novaclient is legacy code, and we should focus on specific regressions and bugs without doing to much code change. The future should be more focussed on openstacksdk and openstackclient. IMO, openstackclient has an impossible task: taking the varied (and flawed) CLI clients and unite them under a single CLI interface. It is better to create the clean separation of the API wrapping into a Python library, and keep the CLI completely separate. -- Ed Leafe signature.asc Description: Message signed with OpenPGP using GPGMail __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova] novaclient functional test guidelines
On 02/24/2015 03:28 PM, Ed Leafe wrote: On Feb 24, 2015, at 1:49 PM, Sean Dague s...@dague.net wrote: IMHO the CLI should have an option to returned raw JSON back instead of pretty tabled results as well. Um... isn't that just the API calls? I'm not sure creating a 3rd functional surface is really the answer here, because we still need to actually test the CLI / pretty table output. The python-openstacksdk project was originally envisioned to wrap the API calls and return usable Python objects. The nova client CLI (or any other CLI, for that matter) would then just provide the command line input parsing and output presentation. It's been a while since I was involved with that project, but it seems that decoupling the command line interface from the Python API wrapper would make testing much, much easier. Right, I think to some degree novaclient is legacy code, and we should focus on specific regressions and bugs without doing to much code change. The future should be more focussed on openstacksdk and openstackclient. -Sean -- Sean Dague http://dague.net signature.asc Description: OpenPGP digital signature __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova] novaclient functional test guidelines
On 02/24/2015 01:47 PM, Joe Gordon wrote: On Tue, Feb 24, 2015 at 9:47 AM, Sean Dague s...@dague.net mailto:s...@dague.net wrote: Towards the end of merging the regression test for the nova volume-attach bug - https://review.openstack.org/#/c/157959/ there was a discussion around what style the functional tests should take. Especially as that had a mix of CLI and API calls in it. Thanks for starting this thread. Once we reach general agreement lets put this in a in tree README for record keeping. Absolutely. Here are my thoughts for why that test ended up that way: 1) All resource setup that is table stakes for the test should be done via the API, regardless if it's a CLI or API test. The reason for this is that structured data is returned, which removes one possible error in the tests by parsing incorrectly. The API objects returned also include things like .delete methods in most cases, which means that addCleanup is a little more clean. IMHO the CLI should have an option to returned raw JSON back instead of pretty tabled results as well. Um... isn't that just the API calls? I'm not sure creating a 3rd functional surface is really the answer here, because we still need to actually test the CLI / pretty table output. 2) Main logic should touch which ever interface you are trying to test. This was demonstrating a CLI regression, so the volume-attach call was important to be done over the CLI. Now... here's where theory runs into issues. #1 - nova boot is table stakes. Under the above guidelines it should be called via API. However --poll is a CLI construct and so saved a custom wait loop here. We should implement that custom wait loop down the road and make that an API call https://github.com/openstack/python-novaclient/blob/master/novaclient/tests/functional/test_instances.py#L116 This issue is from a real short coming in the python client. So this isn't really an issue with your theory, just an issue with novaclient. Sure, though, if we support the async form in the API we need to test it. So while adding poll support is a good UX add, it doesn't actually mean we get to not test the async versions. It just adds another feature set we need to test. #2 - the volume create command is table stakes. It should be an API call. However, it can't be because the service catalog redirection only works at the CLI layer. This is actually also the crux of bug #1423695. The same reason the completion cache code failed is the reason we can't use the API for that. https://github.com/openstack/python-novaclient/blob/master/novaclient/tests/functional/test_instances.py#L129 Issues like this are why I wrote the read only nova CLI tests in the first place. Unit testing the python API is doable, but unit testing the CLI is a little bit more tricky. So I expect issues like this to come up over and over again. So this is actually an issue with the API code. We have a chunk of code that's directly callable by consumers, but if they do, it will error. #3 - the cleanup of the volume should have been API call. See reason for #2. https://github.com/openstack/python-novaclient/blob/master/novaclient/tests/functional/test_instances.py#L131 #4 - the cleanup of the attachment should be an addCleanup via the API. See reason for #2 why it's not. https://github.com/openstack/python-novaclient/blob/master/novaclient/tests/functional/test_instances.py#L155 I'm happy if there are other theories about how we do these things, being the first functional test in the python-novaclient tree that creates and destroys real resources, there isn't an established pattern yet. But I think doing all CLI calls in CLI tests is actually really cumbersome, especially in the amount of output parsing code needed if you are going to setup any complicated resource structure. Here is an alternate theory: We should have both python API and CLI functional tests. But they should be kept separate. This is to help us make sure both the CLI and python API are actually usable interfaces. As the exercise above has shown, they both have really major short comings. I think having in tree functional testing covering both the CLI and python API will make it easier for us to review new client features in terms of usability. Here is a very rough proof of concept patch showing the same tests: https://review.openstack.org/#/c/157974/2/novaclient/tests/functional/test_volumes.py,cm No matter how we define this functional testing model, I think its clear novaclient needs a decent amount of work before it can really be usable. Agreed, but I think we should focus on testing the code we actually have to prevent regressions. I think functional changes to put polling throughout are nice UX, but
Re: [openstack-dev] [nova] novaclient functional test guidelines
On Feb 24, 2015, at 1:49 PM, Sean Dague s...@dague.net wrote: IMHO the CLI should have an option to returned raw JSON back instead of pretty tabled results as well. Um... isn't that just the API calls? I'm not sure creating a 3rd functional surface is really the answer here, because we still need to actually test the CLI / pretty table output. The python-openstacksdk project was originally envisioned to wrap the API calls and return usable Python objects. The nova client CLI (or any other CLI, for that matter) would then just provide the command line input parsing and output presentation. It's been a while since I was involved with that project, but it seems that decoupling the command line interface from the Python API wrapper would make testing much, much easier. -- Ed Leafe signature.asc Description: Message signed with OpenPGP using GPGMail __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova] novaclient functional test guidelines
On Tue, Feb 24, 2015 at 9:47 AM, Sean Dague s...@dague.net wrote: Towards the end of merging the regression test for the nova volume-attach bug - https://review.openstack.org/#/c/157959/ there was a discussion around what style the functional tests should take. Especially as that had a mix of CLI and API calls in it. Thanks for starting this thread. Once we reach general agreement lets put this in a in tree README for record keeping. Here are my thoughts for why that test ended up that way: 1) All resource setup that is table stakes for the test should be done via the API, regardless if it's a CLI or API test. The reason for this is that structured data is returned, which removes one possible error in the tests by parsing incorrectly. The API objects returned also include things like .delete methods in most cases, which means that addCleanup is a little more clean. IMHO the CLI should have an option to returned raw JSON back instead of pretty tabled results as well. 2) Main logic should touch which ever interface you are trying to test. This was demonstrating a CLI regression, so the volume-attach call was important to be done over the CLI. Now... here's where theory runs into issues. #1 - nova boot is table stakes. Under the above guidelines it should be called via API. However --poll is a CLI construct and so saved a custom wait loop here. We should implement that custom wait loop down the road and make that an API call https://github.com/openstack/python-novaclient/blob/master/novaclient/tests/functional/test_instances.py#L116 This issue is from a real short coming in the python client. So this isn't really an issue with your theory, just an issue with novaclient. #2 - the volume create command is table stakes. It should be an API call. However, it can't be because the service catalog redirection only works at the CLI layer. This is actually also the crux of bug #1423695. The same reason the completion cache code failed is the reason we can't use the API for that. https://github.com/openstack/python-novaclient/blob/master/novaclient/tests/functional/test_instances.py#L129 Issues like this are why I wrote the read only nova CLI tests in the first place. Unit testing the python API is doable, but unit testing the CLI is a little bit more tricky. So I expect issues like this to come up over and over again. #3 - the cleanup of the volume should have been API call. See reason for #2. https://github.com/openstack/python-novaclient/blob/master/novaclient/tests/functional/test_instances.py#L131 #4 - the cleanup of the attachment should be an addCleanup via the API. See reason for #2 why it's not. https://github.com/openstack/python-novaclient/blob/master/novaclient/tests/functional/test_instances.py#L155 I'm happy if there are other theories about how we do these things, being the first functional test in the python-novaclient tree that creates and destroys real resources, there isn't an established pattern yet. But I think doing all CLI calls in CLI tests is actually really cumbersome, especially in the amount of output parsing code needed if you are going to setup any complicated resource structure. Here is an alternate theory: We should have both python API and CLI functional tests. But they should be kept separate. This is to help us make sure both the CLI and python API are actually usable interfaces. As the exercise above has shown, they both have really major short comings. I think having in tree functional testing covering both the CLI and python API will make it easier for us to review new client features in terms of usability. Here is a very rough proof of concept patch showing the same tests: https://review.openstack.org/#/c/157974/2/novaclient/tests/functional/test_volumes.py,cm No matter how we define this functional testing model, I think its clear novaclient needs a decent amount of work before it can really be usable. -Sean -- Sean Dague http://dague.net __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [nova] novaclient functional test guidelines
Towards the end of merging the regression test for the nova volume-attach bug - https://review.openstack.org/#/c/157959/ there was a discussion around what style the functional tests should take. Especially as that had a mix of CLI and API calls in it. Here are my thoughts for why that test ended up that way: 1) All resource setup that is table stakes for the test should be done via the API, regardless if it's a CLI or API test. The reason for this is that structured data is returned, which removes one possible error in the tests by parsing incorrectly. The API objects returned also include things like .delete methods in most cases, which means that addCleanup is a little more clean. 2) Main logic should touch which ever interface you are trying to test. This was demonstrating a CLI regression, so the volume-attach call was important to be done over the CLI. Now... here's where theory runs into issues. #1 - nova boot is table stakes. Under the above guidelines it should be called via API. However --poll is a CLI construct and so saved a custom wait loop here. We should implement that custom wait loop down the road and make that an API call https://github.com/openstack/python-novaclient/blob/master/novaclient/tests/functional/test_instances.py#L116 #2 - the volume create command is table stakes. It should be an API call. However, it can't be because the service catalog redirection only works at the CLI layer. This is actually also the crux of bug #1423695. The same reason the completion cache code failed is the reason we can't use the API for that. https://github.com/openstack/python-novaclient/blob/master/novaclient/tests/functional/test_instances.py#L129 #3 - the cleanup of the volume should have been API call. See reason for #2. https://github.com/openstack/python-novaclient/blob/master/novaclient/tests/functional/test_instances.py#L131 #4 - the cleanup of the attachment should be an addCleanup via the API. See reason for #2 why it's not. https://github.com/openstack/python-novaclient/blob/master/novaclient/tests/functional/test_instances.py#L155 I'm happy if there are other theories about how we do these things, being the first functional test in the python-novaclient tree that creates and destroys real resources, there isn't an established pattern yet. But I think doing all CLI calls in CLI tests is actually really cumbersome, especially in the amount of output parsing code needed if you are going to setup any complicated resource structure. -Sean -- Sean Dague http://dague.net __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova] novaclient functional test guidelines
On Tue, Feb 24, 2015 at 12:30 PM, Sean Dague s...@dague.net wrote: On 02/24/2015 03:28 PM, Ed Leafe wrote: On Feb 24, 2015, at 1:49 PM, Sean Dague s...@dague.net wrote: IMHO the CLI should have an option to returned raw JSON back instead of pretty tabled results as well. Um... isn't that just the API calls? I'm not sure creating a 3rd functional surface is really the answer here, because we still need to actually test the CLI / pretty table output. The python-openstacksdk project was originally envisioned to wrap the API calls and return usable Python objects. The nova client CLI (or any other CLI, for that matter) would then just provide the command line input parsing and output presentation. It's been a while since I was involved with that project, but it seems that decoupling the command line interface from the Python API wrapper would make testing much, much easier. Right, I think to some degree novaclient is legacy code, and we should focus on specific regressions and bugs without doing to much code change. The future should be more focussed on openstacksdk and openstackclient. While I don't disagree with this, it seems like we have been waiting for openstackclient and openstacksdk for a while now, do we even have a timeline for moving off of novaclient? -Sean -- Sean Dague http://dague.net __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova] novaclient functional test guidelines
On Tue, Feb 24, 2015 at 1:18 PM, melanie witt melwi...@gmail.com wrote: On Feb 24, 2015, at 9:47, Sean Dague s...@dague.net wrote: I'm happy if there are other theories about how we do these things, being the first functional test in the python-novaclient tree that creates and destroys real resources, there isn't an established pattern yet. But I think doing all CLI calls in CLI tests is actually really cumbersome, especially in the amount of output parsing code needed if you are going to setup any complicated resource structure. I think I'm in agreement with the pattern you describe. I imagine having a set of functional tests for the API, that don't do any CLI calls at all. With that we test that the API works properly. Then have a separate set for the CLI, which only calls CLI for the command being tested, everything else to set up and tear down the test done by API calls. This would be done with the rationale that because the entire API functionality is tested separately, we can safely use it for setup/teardown with the intent to isolate the CLI test to the command being tested and avoid introducing side effects from the CLI commands. But I suppose one could make the same argument for using CLI everywhere (if they are all tested, they can also be trusted not to introduce side effects). I tend to favor using the API because it's the most bare bones setup/teardown we could use. At the same time I understand the idea of performing an entire test using the CLI, as a way of replicating the experience a real user might have using the CLI, from start to end. I don't think I feel strongly either way. I guess its time to revisit the actual status of novaclient and if we want to actively move away from it to openstacksdk/OSC as well. If we are actively trying to move away from novaclient, using the python API as much as possible makes a lot of sense. For the --poll stuff, I agree the API should have it and the CLI uses it. And with and without poll functionality should be tested separately, API and CLI. melanie (melwitt) __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
