[openstack-dev] [Tempest][qa] : backporting Test Security Groups Basic Ops in stable/havana ?
Hi everyone, I'm interested in having the Tempest scenario Test Security Groups Basic Ops available in the stable/Havana branch . This scenario is nice for acceptance tests and it's running fine with an Havana deployment. Can someone in the Stable-maint team can backport it from master to stable/Havana ? If it's OK for you of course :) Best Regards, Julien LELOUP julien.lel...@3ds.com This email and any attachments are intended solely for the use of the individual or entity to whom it is addressed and may be confidential and/or privileged. If you are not one of the named recipients or have received this email in error, (i) you should not read, disclose, or copy it, (ii) please notify sender of your receipt by reply email and delete this email and all attachments, (iii) Dassault Systemes does not accept or assume any liability or responsibility for any use of or reliance on this email. For other languages, go to http://www.3ds.com/terms/email-disclaimer ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Tempest - Stress Test][qa] : implement a full SSH connection on ssh_floating.py and improve it
I mean we still have the tempest/stress/actions/ and we could put something like that in there. In general I would like to discuss this topic in the next QA meeting.. @Julien: are you able to join the next meeting? It would be 22:00 UTC. @Marc : so next Thrusday (3/6/2014) ? Yes I can be there. I have a last minute hindrance preventing me to attend to the next QA meeting. I will see to attend the one at 1700 UTC Thursday 13 march. Best Regards, Julien LELOUP julien.lel...@3ds.com This email and any attachments are intended solely for the use of the individual or entity to whom it is addressed and may be confidential and/or privileged. If you are not one of the named recipients or have received this email in error, (i) you should not read, disclose, or copy it, (ii) please notify sender of your receipt by reply email and delete this email and all attachments, (iii) Dassault Systemes does not accept or assume any liability or responsibility for any use of or reliance on this email. For other languages, go to http://www.3ds.com/terms/email-disclaimer ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Tempest - Stress Test][qa] : implement a full SSH connection on ssh_floating.py and improve it
From: Koderer, Marc [mailto:m.kode...@telekom.de] Sent: Tuesday, March 04, 2014 10:41 AM To: LELOUP Julien; OpenStack Development Mailing List (not for usage questions) Subject: AW: [Tempest - Stress Test][qa] : implement a full SSH connection on ssh_floating.py and improve it So where should I put this test ? I'm thinking of creating a new scenario file inheriting of the class specified in the large_ops file (TestLargeOpsScenario) in order to avoid duplicating the nova_boot method. Is it OK for you ? Is there a better place where I should put my test ? I mean we still have the tempest/stress/actions/ and we could put something like that in there. In general I would like to discuss this topic in the next QA meeting.. @Julien: are you able to join the next meeting? It would be 22:00 UTC. @Marc : so next Thrusday (3/6/2014) ? Yes I can be there. Best Regards, Julien LELOUP julien.lel...@3ds.com Regards, Marc -Original Message- From: LELOUP Julien [mailto:julien.lel...@3ds.com] Sent: Monday, February 17, 2014 5:02 PM To: OpenStack Development Mailing List (not for usage questions); Koderer, Marc Subject: Re: [openstack-dev] [qa] [Tempest - Stress Test] : implement a full SSH connection on ssh_floating.py and improve it Hello Marc, I'm raising this subject again since I have pushed a patch set implementing this SSH stress test in Tempest. As you suggested, I wrote it as a scenario test : I'm currently using it to check the ability of my stack to launch x servers at the same time and having them fully available to use. As a reminder, here is the Blueprint : https://blueprints.launchpad.net/tempest/+spec/stress-test-ssh-floatin g-ip Plese feel free to check it and make feedback :) Best Regards, Julien LELOUP julien.lel...@3ds.com -Original Message- From: LELOUP Julien [mailto:julien.lel...@3ds.com] Sent: Monday, January 20, 2014 11:55 AM To: Koderer, Marc Cc: openstack-dev@lists.openstack.org Subject: Re: [openstack-dev] [qa] [Tempest - Stress Test] : implement a full SSH connection on ssh_floating.py and improve it Hello Marc, Thanks again for your help, your blog post is helpful. So I will start writing a new scenario test to get this full SSH stress test on newly created VM. I will put more details about it in the blueprint I created for this : https://blueprints.launchpad.net/tempest/+spec/stress-test-ssh-floatin g-ip Best Regards, Julien LELOUP julien.lel...@3ds.com -Original Message- From: Koderer, Marc [mailto:m.kode...@telekom.de] Sent: Saturday, January 18, 2014 10:11 AM To: LELOUP Julien Cc: openstack-dev@lists.openstack.org Subject: RE: [qa] [Tempest - Stress Test] : implement a full SSH connection on ssh_floating.py and improve it Hello Julien, maybe my blog post helps you with some more details: http://telekomcloud.github.io/2013/09/11/new-ways-of-tempest-stress- testing.html You can run single test if you add a new json file with the test function you want to test. Like: https://github.com/openstack/tempest/blob/master/tempest/stress/etc/sa mple -unit-test.json With that you can launch them with the parameters you already described. Regards, Marc From: LELOUP Julien [julien.lel...@3ds.com] Sent: Friday, January 17, 2014 3:49 PM To: Koderer, Marc Cc: openstack-dev@lists.openstack.org Subject: RE: [Tempest - Stress Test] : implement a full SSH connection on ssh_floating.py and improve it Hi Marc, The Etherpad you provided was helpful to know the current state of the stress tests. I admit that I have some difficulties to understand how I can run a single test built with the @stresstest decorator (even not a beginner in Python, I still have things to learn on this technology and a lot more on OpenStack/Tempest :) ). I used to run my test using ./run_stress.py -t JSON configuration pointing at my action/.py script -d duration, which allowed me to run only one test with a dedicated configuration (number of threads, ...) For what I understand now in Tempest, I only managed to run all tests, using ./run_tests.sh and the only configuration I found related to stress tests was the [stress] section in tempest.conf. For example : let say I ported my SSH stress test as a scenario test with the @stresstest decorator. How can I launch this test (and only this one) and use a dedicated configuration file like ones we can found in tempest/stress/etc ? Another question I have now : in the case that I have to use run_test.sh and not run_stress.py anymore, how do I get the test runs statistics I used to have, and where should I put some code to improve them ? When I will have cleared my mind with all these kinds of practical details, maybe I should add some content on the Wiki about stress tests in Tempest. Best Regards, Julien LELOUP julien.lel...@3ds.com -Original Message- From: Koderer, Marc
[openstack-dev] [qa] : error during gate-tempest-dsvm-neutron-large-ops when attaching floating IP
Hello everyone, I'm currently pushing a stress test in Tempest which at some point tries to attach a floating IP to newly started servers . The change I'm talking about is available here : https://review.openstack.org/#/c/74067/ . This test runs fine on my local devstack, but I have an issue during Jenkins tests, on the gate 'gate-tempest-dsvm-neutron-large-ops'. The error happen when my test wants to attach a floating IP using the Neutron API : * Last week, the error was about the fact that no ports was attached to my servers : I wasn't specifying any nics during the server creation, but since there was only one network available from the nova point of view, I believed it was automatically attached to my server, thus creating a port for it. This explains why this code is working on my local devstack and openstack deployment. * In order to resolve this I'm currently forcing the 'nics' parameter during the server creation in order to have my server attached to the 'private' network and so have a port to use later during floating IP attachment. This triggers a new error : I can request Neutron to get the UUID of the network named 'private', but when I'm specifying this UUID in the 'nics' parameter, I get an error saying that this network is not found. I don't know if the issue is related to my code, the tempest configuration used in the gate or the devstack deployed for the gate. Can someone give me some insights about this gate ? Is there something peculiar in the devstack deployment used for this gate that can explain the errors I'm having ? Thanks in advance for your time :) Best Regards, Julien LELOUP julien.lel...@3ds.com This email and any attachments are intended solely for the use of the individual or entity to whom it is addressed and may be confidential and/or privileged. If you are not one of the named recipients or have received this email in error, (i) you should not read, disclose, or copy it, (ii) please notify sender of your receipt by reply email and delete this email and all attachments, (iii) Dassault Systemes does not accept or assume any liability or responsibility for any use of or reliance on this email. For other languages, go to http://www.3ds.com/terms/email-disclaimer ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [qa] [Tempest - Stress Test] : implement a full SSH connection on ssh_floating.py and improve it
Hello Marc, I'm raising this subject again since I have pushed a patch set implementing this SSH stress test in Tempest. As you suggested, I wrote it as a scenario test : I'm currently using it to check the ability of my stack to launch x servers at the same time and having them fully available to use. As a reminder, here is the Blueprint : https://blueprints.launchpad.net/tempest/+spec/stress-test-ssh-floating-ip Plese feel free to check it and make feedback :) Best Regards, Julien LELOUP julien.lel...@3ds.com -Original Message- From: LELOUP Julien [mailto:julien.lel...@3ds.com] Sent: Monday, January 20, 2014 11:55 AM To: Koderer, Marc Cc: openstack-dev@lists.openstack.org Subject: Re: [openstack-dev] [qa] [Tempest - Stress Test] : implement a full SSH connection on ssh_floating.py and improve it Hello Marc, Thanks again for your help, your blog post is helpful. So I will start writing a new scenario test to get this full SSH stress test on newly created VM. I will put more details about it in the blueprint I created for this : https://blueprints.launchpad.net/tempest/+spec/stress-test-ssh-floating-ip Best Regards, Julien LELOUP julien.lel...@3ds.com -Original Message- From: Koderer, Marc [mailto:m.kode...@telekom.de] Sent: Saturday, January 18, 2014 10:11 AM To: LELOUP Julien Cc: openstack-dev@lists.openstack.org Subject: RE: [qa] [Tempest - Stress Test] : implement a full SSH connection on ssh_floating.py and improve it Hello Julien, maybe my blog post helps you with some more details: http://telekomcloud.github.io/2013/09/11/new-ways-of-tempest-stress-testing.html You can run single test if you add a new json file with the test function you want to test. Like: https://github.com/openstack/tempest/blob/master/tempest/stress/etc/sample-unit-test.json With that you can launch them with the parameters you already described. Regards, Marc From: LELOUP Julien [julien.lel...@3ds.com] Sent: Friday, January 17, 2014 3:49 PM To: Koderer, Marc Cc: openstack-dev@lists.openstack.org Subject: RE: [Tempest - Stress Test] : implement a full SSH connection on ssh_floating.py and improve it Hi Marc, The Etherpad you provided was helpful to know the current state of the stress tests. I admit that I have some difficulties to understand how I can run a single test built with the @stresstest decorator (even not a beginner in Python, I still have things to learn on this technology and a lot more on OpenStack/Tempest :) ). I used to run my test using ./run_stress.py -t JSON configuration pointing at my action/.py script -d duration, which allowed me to run only one test with a dedicated configuration (number of threads, ...) For what I understand now in Tempest, I only managed to run all tests, using ./run_tests.sh and the only configuration I found related to stress tests was the [stress] section in tempest.conf. For example : let say I ported my SSH stress test as a scenario test with the @stresstest decorator. How can I launch this test (and only this one) and use a dedicated configuration file like ones we can found in tempest/stress/etc ? Another question I have now : in the case that I have to use run_test.sh and not run_stress.py anymore, how do I get the test runs statistics I used to have, and where should I put some code to improve them ? When I will have cleared my mind with all these kinds of practical details, maybe I should add some content on the Wiki about stress tests in Tempest. Best Regards, Julien LELOUP julien.lel...@3ds.com -Original Message- From: Koderer, Marc [mailto:m.kode...@telekom.de] Sent: Friday, January 17, 2014 3:07 PM To: LELOUP Julien Cc: openstack-dev@lists.openstack.org Subject: RE: [qa] RE: [Tempest - Stress Test] : implement a full SSH connection on ssh_floating.py and improve it Hi Julien, most of the cases in tempest/stress are already covered by exiting tests in /api or /scenario. The only thing that is missing is the decorator on them. BTW here is the Etherpad from the summit talk that we had: https://etherpad.openstack.org/p/icehouse-summit-qa-stress-tests It possible help to understand the state. I didn't managed to work on the action items that are left. Your suggestions sound good. So I'd happy so see some patches :) Regards Marc From: LELOUP Julien [julien.lel...@3ds.com] Sent: Friday, January 17, 2014 11:52 AM To: Koderer, Marc Cc: openstack-dev@lists.openstack.org Subject: RE: [qa] RE: [Tempest - Stress Test] : implement a full SSH connection on ssh_floating.py and improve it Hello Marc, Thanks for your answer. At the moment I'm willing to spend some time on this kind of scenario so I will see how to use the stress decorator inside a scenario test. Does this means that all stress tests available in tempest/stress should be ported as scenario tests with this decorator ? I do have some ideas about features
Re: [openstack-dev] [Tempest - Stress test] : cleanup() removing resources for all tenants with an admin_manager
Hello Boris, Rally seemes to be a really interesting tool and I will eventually using it for full scale benchmarking. However I will focus on tempest based tests for the time being in order to have these tests replayed regularly during the integration process. I believe that my needs are covered by both Tempest and Rally and if time allow me to do so, I will submit the same kind of scenario for full scale benchmarking. Best Regards, Julien LELOUP julien.lel...@3ds.com From: LELOUP Julien [mailto:julien.lel...@3ds.com] Sent: Tuesday, January 21, 2014 11:34 AM To: OpenStack Development Mailing List (not for usage questions) Subject: Re: [openstack-dev] [Tempest - Stress test] : cleanup() removing resources for all tenants with an admin_manager Hello Boris, I'll check Rally in order to see what tool is the best for my tests. Best Regards, Julien LELOUP julien.lel...@3ds.commailto:julien.lel...@3ds.com From: Boris Pavlovic [mailto:bpavlo...@mirantis.com] Sent: Monday, January 20, 2014 5:52 PM To: OpenStack Development Mailing List (not for usage questions) Subject: Re: [openstack-dev] [Tempest - Stress test] : cleanup() removing resources for all tenants with an admin_manager Julien, Probably you should try to use Rally for benchmarking. https://wiki.openstack.org/wiki/Rally There is already working generic cleanup... There is already implemented framework that allows parametrized benchmarks: https://github.com/stackforge/rally/blob/master/rally/benchmark/scenarios/nova/servers.py#L32-L39 Simple way to configure load using json (load will be created from real users, no admin, that will be pre created for each benchmark): https://github.com/stackforge/rally/blob/master/doc/samples/tasks/nova/boot-and-delete.json And simple CLI interface (now we are working around Web UI) Best regards, Boris Pavlovic On Mon, Jan 20, 2014 at 8:32 PM, LELOUP Julien julien.lel...@3ds.commailto:julien.lel...@3ds.com wrote: Hi everyone, I'm forwarding my own email previously posted on the QA list. I would like to discuss about the cleanup() process used right after a stress test run in Tempest. For what I see now by using it and by reading the code, the cleanup() seems a bit rough since it is using an admin_manager in order to get all kind of test resources actually available : servers, key pairs, volumes, .etc... More precisely, when it comes to clean servers, it is searching for servers on all tenants. I find this behavior a little rough since it will blow all objects on the target OpenStack, even object unrelated to the stress tests that just ran. Actually before reading the cleanup() I had a problem when one of my stress test erased all the servers and volumes on another tenant, which impaired other people working on our OpenStack. I can imagine that for some scenarios, using an admin user to deeply clean an OpenStack is required, but I believe that most of the time the cleanup() process should focus only on the tenant used during the stress test and leave the other tenants alone. Am I doing something wrong ? Is there a way to restrain the cleanup() process ? If no parameters or configuration allows me to do so, should I improve the cleanup() code in order to allow it to remove only the test resources created for the test? I do not wish to make this kind of code if the OpenStack community believe that the present behavior is totally intended and should not be modified. Best Regards, Julien LELOUP julien.lel...@3ds.commailto:julien.lel...@3ds.com This email and any attachments are intended solely for the use of the individual or entity to whom it is addressed and may be confidential and/or privileged. If you are not one of the named recipients or have received this email in error, (i) you should not read, disclose, or copy it, (ii) please notify sender of your receipt by reply email and delete this email and all attachments, (iii) Dassault Systemes does not accept or assume any liability or responsibility for any use of or reliance on this email. For other languages, go to http://www.3ds.com/terms/email-disclaimer ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.orgmailto:OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev This email and any attachments are intended solely for the use of the individual or entity to whom it is addressed and may be confidential and/or privileged. If you are not one of the named recipients or have received this email in error, (i) you should not read, disclose, or copy it, (ii) please notify sender of your receipt by reply email and delete this email and all attachments, (iii) Dassault Systemes does not accept or assume any liability or responsibility for any use of or reliance on this email. For other languages, go to http://www.3ds.com/terms/email-disclaimer This email and any attachments are intended solely for the use
[openstack-dev] [Tempest - Stress Test] : some improvements / issues on the stress test framework
Hi everyone, I would like to discuss some ideas / behavior that seems broken in the stress test part of Tempest. I opened some tickets in Launchpad and I would like to get the feedback of the community on these ideas / issues : - Provide kwargs from UnitTest to stress test scenarios (https://bugs.launchpad.net/tempest/+bug/1273186 ) - Stress Test - tearDown() not called if an exception occurred (https://bugs.launchpad.net/tempest/+bug/1273245 ) - Stress Test - cleanUp() removing all test resources as an admin (https://bugs.launchpad.net/tempest/+bug/1273254 ) Best Regards, Julien LELOUP julien.lel...@3ds.com This email and any attachments are intended solely for the use of the individual or entity to whom it is addressed and may be confidential and/or privileged. If you are not one of the named recipients or have received this email in error, (i) you should not read, disclose, or copy it, (ii) please notify sender of your receipt by reply email and delete this email and all attachments, (iii) Dassault Systemes does not accept or assume any liability or responsibility for any use of or reliance on this email. For other languages, go to http://www.3ds.com/terms/email-disclaimer ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Tempest - Stress test] : cleanup() removing resources for all tenants with an admin_manager
Hello Boris, I'll check Rally in order to see what tool is the best for my tests. Best Regards, Julien LELOUP julien.lel...@3ds.com From: Boris Pavlovic [mailto:bpavlo...@mirantis.com] Sent: Monday, January 20, 2014 5:52 PM To: OpenStack Development Mailing List (not for usage questions) Subject: Re: [openstack-dev] [Tempest - Stress test] : cleanup() removing resources for all tenants with an admin_manager Julien, Probably you should try to use Rally for benchmarking. https://wiki.openstack.org/wiki/Rally There is already working generic cleanup... There is already implemented framework that allows parametrized benchmarks: https://github.com/stackforge/rally/blob/master/rally/benchmark/scenarios/nova/servers.py#L32-L39 Simple way to configure load using json (load will be created from real users, no admin, that will be pre created for each benchmark): https://github.com/stackforge/rally/blob/master/doc/samples/tasks/nova/boot-and-delete.json And simple CLI interface (now we are working around Web UI) Best regards, Boris Pavlovic On Mon, Jan 20, 2014 at 8:32 PM, LELOUP Julien julien.lel...@3ds.commailto:julien.lel...@3ds.com wrote: Hi everyone, I'm forwarding my own email previously posted on the QA list. I would like to discuss about the cleanup() process used right after a stress test run in Tempest. For what I see now by using it and by reading the code, the cleanup() seems a bit rough since it is using an admin_manager in order to get all kind of test resources actually available : servers, key pairs, volumes, .etc... More precisely, when it comes to clean servers, it is searching for servers on all tenants. I find this behavior a little rough since it will blow all objects on the target OpenStack, even object unrelated to the stress tests that just ran. Actually before reading the cleanup() I had a problem when one of my stress test erased all the servers and volumes on another tenant, which impaired other people working on our OpenStack. I can imagine that for some scenarios, using an admin user to deeply clean an OpenStack is required, but I believe that most of the time the cleanup() process should focus only on the tenant used during the stress test and leave the other tenants alone. Am I doing something wrong ? Is there a way to restrain the cleanup() process ? If no parameters or configuration allows me to do so, should I improve the cleanup() code in order to allow it to remove only the test resources created for the test? I do not wish to make this kind of code if the OpenStack community believe that the present behavior is totally intended and should not be modified. Best Regards, Julien LELOUP julien.lel...@3ds.commailto:julien.lel...@3ds.com This email and any attachments are intended solely for the use of the individual or entity to whom it is addressed and may be confidential and/or privileged. If you are not one of the named recipients or have received this email in error, (i) you should not read, disclose, or copy it, (ii) please notify sender of your receipt by reply email and delete this email and all attachments, (iii) Dassault Systemes does not accept or assume any liability or responsibility for any use of or reliance on this email. For other languages, go to http://www.3ds.com/terms/email-disclaimer ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.orgmailto:OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev This email and any attachments are intended solely for the use of the individual or entity to whom it is addressed and may be confidential and/or privileged. If you are not one of the named recipients or have received this email in error, (i) you should not read, disclose, or copy it, (ii) please notify sender of your receipt by reply email and delete this email and all attachments, (iii) Dassault Systemes does not accept or assume any liability or responsibility for any use of or reliance on this email. For other languages, go to http://www.3ds.com/terms/email-disclaimer ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [qa] [Tempest - Stress Test] : implement a full SSH connection on ssh_floating.py and improve it
Hello Marc, Thanks again for your help, your blog post is helpful. So I will start writing a new scenario test to get this full SSH stress test on newly created VM. I will put more details about it in the blueprint I created for this : https://blueprints.launchpad.net/tempest/+spec/stress-test-ssh-floating-ip Best Regards, Julien LELOUP julien.lel...@3ds.com -Original Message- From: Koderer, Marc [mailto:m.kode...@telekom.de] Sent: Saturday, January 18, 2014 10:11 AM To: LELOUP Julien Cc: openstack-dev@lists.openstack.org Subject: RE: [qa] [Tempest - Stress Test] : implement a full SSH connection on ssh_floating.py and improve it Hello Julien, maybe my blog post helps you with some more details: http://telekomcloud.github.io/2013/09/11/new-ways-of-tempest-stress-testing.html You can run single test if you add a new json file with the test function you want to test. Like: https://github.com/openstack/tempest/blob/master/tempest/stress/etc/sample-unit-test.json With that you can launch them with the parameters you already described. Regards, Marc From: LELOUP Julien [julien.lel...@3ds.com] Sent: Friday, January 17, 2014 3:49 PM To: Koderer, Marc Cc: openstack-dev@lists.openstack.org Subject: RE: [Tempest - Stress Test] : implement a full SSH connection on ssh_floating.py and improve it Hi Marc, The Etherpad you provided was helpful to know the current state of the stress tests. I admit that I have some difficulties to understand how I can run a single test built with the @stresstest decorator (even not a beginner in Python, I still have things to learn on this technology and a lot more on OpenStack/Tempest :) ). I used to run my test using ./run_stress.py -t JSON configuration pointing at my action/.py script -d duration, which allowed me to run only one test with a dedicated configuration (number of threads, ...) For what I understand now in Tempest, I only managed to run all tests, using ./run_tests.sh and the only configuration I found related to stress tests was the [stress] section in tempest.conf. For example : let say I ported my SSH stress test as a scenario test with the @stresstest decorator. How can I launch this test (and only this one) and use a dedicated configuration file like ones we can found in tempest/stress/etc ? Another question I have now : in the case that I have to use run_test.sh and not run_stress.py anymore, how do I get the test runs statistics I used to have, and where should I put some code to improve them ? When I will have cleared my mind with all these kinds of practical details, maybe I should add some content on the Wiki about stress tests in Tempest. Best Regards, Julien LELOUP julien.lel...@3ds.com -Original Message- From: Koderer, Marc [mailto:m.kode...@telekom.de] Sent: Friday, January 17, 2014 3:07 PM To: LELOUP Julien Cc: openstack-dev@lists.openstack.org Subject: RE: [qa] RE: [Tempest - Stress Test] : implement a full SSH connection on ssh_floating.py and improve it Hi Julien, most of the cases in tempest/stress are already covered by exiting tests in /api or /scenario. The only thing that is missing is the decorator on them. BTW here is the Etherpad from the summit talk that we had: https://etherpad.openstack.org/p/icehouse-summit-qa-stress-tests It possible help to understand the state. I didn't managed to work on the action items that are left. Your suggestions sound good. So I'd happy so see some patches :) Regards Marc From: LELOUP Julien [julien.lel...@3ds.com] Sent: Friday, January 17, 2014 11:52 AM To: Koderer, Marc Cc: openstack-dev@lists.openstack.org Subject: RE: [qa] RE: [Tempest - Stress Test] : implement a full SSH connection on ssh_floating.py and improve it Hello Marc, Thanks for your answer. At the moment I'm willing to spend some time on this kind of scenario so I will see how to use the stress decorator inside a scenario test. Does this means that all stress tests available in tempest/stress should be ported as scenario tests with this decorator ? I do have some ideas about features on stress test that I find useful for my own use case : like adding more statistics on stress test runs in order to use them as benchmarks. I don't know if this kind of feature was already discussed in the OpenStack community but since stress tests are a bit deprecated now, maybe there is some room for this kind of improvement on fresh stress tests. Best Regards, Julien LELOUP -Original Message- From: Koderer, Marc [mailto:m.kode...@telekom.de] Sent: Friday, January 17, 2014 9:45 AM To: LELOUP Julien Cc: openstack-dev@lists.openstack.org Subject: [qa] RE: [Tempest - Stress Test] : implement a full SSH connection on ssh_floating.py and improve it Hello Juilen, I forwarded your mail to the correct mailing list. Please do not use the qa list any longer. I am happy that you are interested in stress
[openstack-dev] [Tempest - Stress test] : cleanup() removing resources for all tenants with an admin_manager
Hi everyone, I'm forwarding my own email previously posted on the QA list. I would like to discuss about the cleanup() process used right after a stress test run in Tempest. For what I see now by using it and by reading the code, the cleanup() seems a bit rough since it is using an admin_manager in order to get all kind of test resources actually available : servers, key pairs, volumes, .etc... More precisely, when it comes to clean servers, it is searching for servers on all tenants. I find this behavior a little rough since it will blow all objects on the target OpenStack, even object unrelated to the stress tests that just ran. Actually before reading the cleanup() I had a problem when one of my stress test erased all the servers and volumes on another tenant, which impaired other people working on our OpenStack. I can imagine that for some scenarios, using an admin user to deeply clean an OpenStack is required, but I believe that most of the time the cleanup() process should focus only on the tenant used during the stress test and leave the other tenants alone. Am I doing something wrong ? Is there a way to restrain the cleanup() process ? If no parameters or configuration allows me to do so, should I improve the cleanup() code in order to allow it to remove only the test resources created for the test? I do not wish to make this kind of code if the OpenStack community believe that the present behavior is totally intended and should not be modified. Best Regards, Julien LELOUP julien.lel...@3ds.com This email and any attachments are intended solely for the use of the individual or entity to whom it is addressed and may be confidential and/or privileged. If you are not one of the named recipients or have received this email in error, (i) you should not read, disclose, or copy it, (ii) please notify sender of your receipt by reply email and delete this email and all attachments, (iii) Dassault Systemes does not accept or assume any liability or responsibility for any use of or reliance on this email. For other languages, go to http://www.3ds.com/terms/email-disclaimer ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [qa] RE: [Tempest - Stress Test] : implement a full SSH connection on ssh_floating.py and improve it
Hello Marc, Thanks for your answer. At the moment I'm willing to spend some time on this kind of scenario so I will see how to use the stress decorator inside a scenario test. Does this means that all stress tests available in tempest/stress should be ported as scenario tests with this decorator ? I do have some ideas about features on stress test that I find useful for my own use case : like adding more statistics on stress test runs in order to use them as benchmarks. I don't know if this kind of feature was already discussed in the OpenStack community but since stress tests are a bit deprecated now, maybe there is some room for this kind of improvement on fresh stress tests. Best Regards, Julien LELOUP -Original Message- From: Koderer, Marc [mailto:m.kode...@telekom.de] Sent: Friday, January 17, 2014 9:45 AM To: LELOUP Julien Cc: openstack-dev@lists.openstack.org Subject: [qa] RE: [Tempest - Stress Test] : implement a full SSH connection on ssh_floating.py and improve it Hello Juilen, I forwarded your mail to the correct mailing list. Please do not use the qa list any longer. I am happy that you are interested in stress tests. All the tests in tempest/stress/actions are more or less deprecated. So what you should use instead is the stress decorator (e.g. https://github.com/openstack/tempest/blob/master/tempest/api/volume/test_volumes_actions.py#L55). Unfortunately it's not yet used for scenarios like you describe. I'd suggest to build a scenario test in tempest/scenario and use this decorator on it. Any patch like that is welcome on gerrit. If you are planning to work in that area for more than just a patch, a blueprint would be nice. A good way to coordinate your efforts is also in the QA meeting (https://wiki.openstack.org/wiki/Meetings/QATeamMeeting) Regards Marc From: LELOUP Julien [julien.lel...@3ds.com] Sent: Wednesday, January 15, 2014 5:57 PM To: openstack...@lists.openstack.org Subject: [openstack-qa] [Tempest - Stress Test] : implement a full SSH connection on ssh_floating.py and improve it Hi everyone, I’m quite new on OpenStack / Tempest and I’m actually working on stress tests. I want to suggest a new feature in a currently available stress test. Not sure if this email should be posted on the QA mailing list or the dev mailing list, but I give it a try here since it is about a Tempest stress test ☺ At the moment the “ssh_floating.py” stress test seems really interesting but I would like to improve it a bit. By now this script is simulating an SSH connection by binding a TCP socket on the newly created instance. But this test don’t allow us to check if this instance is really available. I’m mostly thinking about the metadata service unable to provide the SSH key pair to the instance, but surely other scenarios can lead to an instance considered “ACTIVE” but actually unusable. So I’m implementing a full SSH connection test using the “paramiko” SSH library and a key pair generated in the same way the other test resources are managed in this script : either one SSH key pair for every test runs or a new key pair for each run (depends on the JSON configuration file). I don’t plan to remove the old test (TCP socket binding), rather move this one on a separate test function and put the full SSH connection test code instead. Is this feature interesting for the OpenStack community ? Should I create a blueprint on the Tempest project on Launchpad in order to provide my code through Gerrit ? On a second time, I plan to overhaul improve this “ssh_floating.py” script by clean the code a little bit, and add more cleaning code in order to avoid leaving instances/security groups/floating IP behind : I do have this kind of behavior right now and I already improved the teardown() on this way. Should I consider this code as a new functionality (thus create a blueprint) or should I create a defect and assign it to myself ? Cordialement / Best Regards, Julien LELOUP RD 3DExperience Platform IaaS Factory Technology Engineer julien.lel...@3ds.commailto:julien.lel...@3ds.com [cid:image003.gif@01CF1216.D43ECE20] 3DS.COMhttp://www.3ds.com/ Dassault Systèmes | 10 rue Marcel Dassault, CS 40501 | 78946 Vélizy-Villacoublay Cedex | France This email and any attachments are intended solely for the use of the individual or entity to whom it is addressed and may be confidential and/or privileged. If you are not one of the named recipients or have received this email in error, (i) you should not read, disclose, or copy it, (ii) please notify sender of your receipt by reply email and delete this email and all attachments, (iii) Dassault Systemes does not accept or assume any liability or responsibility for any use of or reliance on this email. For other languages, go to http://www.3ds.com/terms/email-disclaimer This email and any attachments are intended solely for the use of the individual or entity
Re: [openstack-dev] [Tempest - Stress Test] : implement a full SSH connection on ssh_floating.py and improve it
Hi Marc, The Etherpad you provided was helpful to know the current state of the stress tests. I admit that I have some difficulties to understand how I can run a single test built with the @stresstest decorator (even not a beginner in Python, I still have things to learn on this technology and a lot more on OpenStack/Tempest :) ). I used to run my test using ./run_stress.py -t JSON configuration pointing at my action/.py script -d duration, which allowed me to run only one test with a dedicated configuration (number of threads, ...) For what I understand now in Tempest, I only managed to run all tests, using ./run_tests.sh and the only configuration I found related to stress tests was the [stress] section in tempest.conf. For example : let say I ported my SSH stress test as a scenario test with the @stresstest decorator. How can I launch this test (and only this one) and use a dedicated configuration file like ones we can found in tempest/stress/etc ? Another question I have now : in the case that I have to use run_test.sh and not run_stress.py anymore, how do I get the test runs statistics I used to have, and where should I put some code to improve them ? When I will have cleared my mind with all these kinds of practical details, maybe I should add some content on the Wiki about stress tests in Tempest. Best Regards, Julien LELOUP julien.lel...@3ds.com -Original Message- From: Koderer, Marc [mailto:m.kode...@telekom.de] Sent: Friday, January 17, 2014 3:07 PM To: LELOUP Julien Cc: openstack-dev@lists.openstack.org Subject: RE: [qa] RE: [Tempest - Stress Test] : implement a full SSH connection on ssh_floating.py and improve it Hi Julien, most of the cases in tempest/stress are already covered by exiting tests in /api or /scenario. The only thing that is missing is the decorator on them. BTW here is the Etherpad from the summit talk that we had: https://etherpad.openstack.org/p/icehouse-summit-qa-stress-tests It possible help to understand the state. I didn't managed to work on the action items that are left. Your suggestions sound good. So I'd happy so see some patches :) Regards Marc From: LELOUP Julien [julien.lel...@3ds.com] Sent: Friday, January 17, 2014 11:52 AM To: Koderer, Marc Cc: openstack-dev@lists.openstack.org Subject: RE: [qa] RE: [Tempest - Stress Test] : implement a full SSH connection on ssh_floating.py and improve it Hello Marc, Thanks for your answer. At the moment I'm willing to spend some time on this kind of scenario so I will see how to use the stress decorator inside a scenario test. Does this means that all stress tests available in tempest/stress should be ported as scenario tests with this decorator ? I do have some ideas about features on stress test that I find useful for my own use case : like adding more statistics on stress test runs in order to use them as benchmarks. I don't know if this kind of feature was already discussed in the OpenStack community but since stress tests are a bit deprecated now, maybe there is some room for this kind of improvement on fresh stress tests. Best Regards, Julien LELOUP -Original Message- From: Koderer, Marc [mailto:m.kode...@telekom.de] Sent: Friday, January 17, 2014 9:45 AM To: LELOUP Julien Cc: openstack-dev@lists.openstack.org Subject: [qa] RE: [Tempest - Stress Test] : implement a full SSH connection on ssh_floating.py and improve it Hello Juilen, I forwarded your mail to the correct mailing list. Please do not use the qa list any longer. I am happy that you are interested in stress tests. All the tests in tempest/stress/actions are more or less deprecated. So what you should use instead is the stress decorator (e.g. https://github.com/openstack/tempest/blob/master/tempest/api/volume/test_volumes_actions.py#L55). Unfortunately it's not yet used for scenarios like you describe. I'd suggest to build a scenario test in tempest/scenario and use this decorator on it. Any patch like that is welcome on gerrit. If you are planning to work in that area for more than just a patch, a blueprint would be nice. A good way to coordinate your efforts is also in the QA meeting (https://wiki.openstack.org/wiki/Meetings/QATeamMeeting) Regards Marc From: LELOUP Julien [julien.lel...@3ds.com] Sent: Wednesday, January 15, 2014 5:57 PM To: openstack...@lists.openstack.org Subject: [openstack-qa] [Tempest - Stress Test] : implement a full SSH connection on ssh_floating.py and improve it Hi everyone, I’m quite new on OpenStack / Tempest and I’m actually working on stress tests. I want to suggest a new feature in a currently available stress test. Not sure if this email should be posted on the QA mailing list or the dev mailing list, but I give it a try here since it is about a Tempest stress test ☺ At the moment the “ssh_floating.py” stress test seems really interesting but I would like to improve
