Re: [openstack-dev] [qa] [Tempest - Stress Test] : implement a full SSH connection on ssh_floating.py and improve it
Hello Marc, I'm raising this subject again since I have pushed a patch set implementing this SSH stress test in Tempest. As you suggested, I wrote it as a scenario test : I'm currently using it to check the ability of my stack to launch x servers at the same time and having them fully available to use. As a reminder, here is the Blueprint : https://blueprints.launchpad.net/tempest/+spec/stress-test-ssh-floating-ip Plese feel free to check it and make feedback :) Best Regards, Julien LELOUP julien.lel...@3ds.com -Original Message- From: LELOUP Julien [mailto:julien.lel...@3ds.com] Sent: Monday, January 20, 2014 11:55 AM To: Koderer, Marc Cc: openstack-dev@lists.openstack.org Subject: Re: [openstack-dev] [qa] [Tempest - Stress Test] : implement a full SSH connection on ssh_floating.py and improve it Hello Marc, Thanks again for your help, your blog post is helpful. So I will start writing a new scenario test to get this full SSH stress test on newly created VM. I will put more details about it in the blueprint I created for this : https://blueprints.launchpad.net/tempest/+spec/stress-test-ssh-floating-ip Best Regards, Julien LELOUP julien.lel...@3ds.com -Original Message- From: Koderer, Marc [mailto:m.kode...@telekom.de] Sent: Saturday, January 18, 2014 10:11 AM To: LELOUP Julien Cc: openstack-dev@lists.openstack.org Subject: RE: [qa] [Tempest - Stress Test] : implement a full SSH connection on ssh_floating.py and improve it Hello Julien, maybe my blog post helps you with some more details: http://telekomcloud.github.io/2013/09/11/new-ways-of-tempest-stress-testing.html You can run single test if you add a new json file with the test function you want to test. Like: https://github.com/openstack/tempest/blob/master/tempest/stress/etc/sample-unit-test.json With that you can launch them with the parameters you already described. Regards, Marc From: LELOUP Julien [julien.lel...@3ds.com] Sent: Friday, January 17, 2014 3:49 PM To: Koderer, Marc Cc: openstack-dev@lists.openstack.org Subject: RE: [Tempest - Stress Test] : implement a full SSH connection on ssh_floating.py and improve it Hi Marc, The Etherpad you provided was helpful to know the current state of the stress tests. I admit that I have some difficulties to understand how I can run a single test built with the @stresstest decorator (even not a beginner in Python, I still have things to learn on this technology and a lot more on OpenStack/Tempest :) ). I used to run my test using ./run_stress.py -t JSON configuration pointing at my action/.py script -d duration, which allowed me to run only one test with a dedicated configuration (number of threads, ...) For what I understand now in Tempest, I only managed to run all tests, using ./run_tests.sh and the only configuration I found related to stress tests was the [stress] section in tempest.conf. For example : let say I ported my SSH stress test as a scenario test with the @stresstest decorator. How can I launch this test (and only this one) and use a dedicated configuration file like ones we can found in tempest/stress/etc ? Another question I have now : in the case that I have to use run_test.sh and not run_stress.py anymore, how do I get the test runs statistics I used to have, and where should I put some code to improve them ? When I will have cleared my mind with all these kinds of practical details, maybe I should add some content on the Wiki about stress tests in Tempest. Best Regards, Julien LELOUP julien.lel...@3ds.com -Original Message- From: Koderer, Marc [mailto:m.kode...@telekom.de] Sent: Friday, January 17, 2014 3:07 PM To: LELOUP Julien Cc: openstack-dev@lists.openstack.org Subject: RE: [qa] RE: [Tempest - Stress Test] : implement a full SSH connection on ssh_floating.py and improve it Hi Julien, most of the cases in tempest/stress are already covered by exiting tests in /api or /scenario. The only thing that is missing is the decorator on them. BTW here is the Etherpad from the summit talk that we had: https://etherpad.openstack.org/p/icehouse-summit-qa-stress-tests It possible help to understand the state. I didn't managed to work on the action items that are left. Your suggestions sound good. So I'd happy so see some patches :) Regards Marc From: LELOUP Julien [julien.lel...@3ds.com] Sent: Friday, January 17, 2014 11:52 AM To: Koderer, Marc Cc: openstack-dev@lists.openstack.org Subject: RE: [qa] RE: [Tempest - Stress Test] : implement a full SSH connection on ssh_floating.py and improve it Hello Marc, Thanks for your answer. At the moment I'm willing to spend some time on this kind of scenario so I will see how to use the stress decorator inside a scenario test. Does this means that all stress tests available in tempest/stress should be ported as scenario tests with this decorator ? I do have some ideas about features
Re: [openstack-dev] [qa] [Tempest - Stress Test] : implement a full SSH connection on ssh_floating.py and improve it
Hello Marc, Thanks again for your help, your blog post is helpful. So I will start writing a new scenario test to get this full SSH stress test on newly created VM. I will put more details about it in the blueprint I created for this : https://blueprints.launchpad.net/tempest/+spec/stress-test-ssh-floating-ip Best Regards, Julien LELOUP julien.lel...@3ds.com -Original Message- From: Koderer, Marc [mailto:m.kode...@telekom.de] Sent: Saturday, January 18, 2014 10:11 AM To: LELOUP Julien Cc: openstack-dev@lists.openstack.org Subject: RE: [qa] [Tempest - Stress Test] : implement a full SSH connection on ssh_floating.py and improve it Hello Julien, maybe my blog post helps you with some more details: http://telekomcloud.github.io/2013/09/11/new-ways-of-tempest-stress-testing.html You can run single test if you add a new json file with the test function you want to test. Like: https://github.com/openstack/tempest/blob/master/tempest/stress/etc/sample-unit-test.json With that you can launch them with the parameters you already described. Regards, Marc From: LELOUP Julien [julien.lel...@3ds.com] Sent: Friday, January 17, 2014 3:49 PM To: Koderer, Marc Cc: openstack-dev@lists.openstack.org Subject: RE: [Tempest - Stress Test] : implement a full SSH connection on ssh_floating.py and improve it Hi Marc, The Etherpad you provided was helpful to know the current state of the stress tests. I admit that I have some difficulties to understand how I can run a single test built with the @stresstest decorator (even not a beginner in Python, I still have things to learn on this technology and a lot more on OpenStack/Tempest :) ). I used to run my test using ./run_stress.py -t JSON configuration pointing at my action/.py script -d duration, which allowed me to run only one test with a dedicated configuration (number of threads, ...) For what I understand now in Tempest, I only managed to run all tests, using ./run_tests.sh and the only configuration I found related to stress tests was the [stress] section in tempest.conf. For example : let say I ported my SSH stress test as a scenario test with the @stresstest decorator. How can I launch this test (and only this one) and use a dedicated configuration file like ones we can found in tempest/stress/etc ? Another question I have now : in the case that I have to use run_test.sh and not run_stress.py anymore, how do I get the test runs statistics I used to have, and where should I put some code to improve them ? When I will have cleared my mind with all these kinds of practical details, maybe I should add some content on the Wiki about stress tests in Tempest. Best Regards, Julien LELOUP julien.lel...@3ds.com -Original Message- From: Koderer, Marc [mailto:m.kode...@telekom.de] Sent: Friday, January 17, 2014 3:07 PM To: LELOUP Julien Cc: openstack-dev@lists.openstack.org Subject: RE: [qa] RE: [Tempest - Stress Test] : implement a full SSH connection on ssh_floating.py and improve it Hi Julien, most of the cases in tempest/stress are already covered by exiting tests in /api or /scenario. The only thing that is missing is the decorator on them. BTW here is the Etherpad from the summit talk that we had: https://etherpad.openstack.org/p/icehouse-summit-qa-stress-tests It possible help to understand the state. I didn't managed to work on the action items that are left. Your suggestions sound good. So I'd happy so see some patches :) Regards Marc From: LELOUP Julien [julien.lel...@3ds.com] Sent: Friday, January 17, 2014 11:52 AM To: Koderer, Marc Cc: openstack-dev@lists.openstack.org Subject: RE: [qa] RE: [Tempest - Stress Test] : implement a full SSH connection on ssh_floating.py and improve it Hello Marc, Thanks for your answer. At the moment I'm willing to spend some time on this kind of scenario so I will see how to use the stress decorator inside a scenario test. Does this means that all stress tests available in tempest/stress should be ported as scenario tests with this decorator ? I do have some ideas about features on stress test that I find useful for my own use case : like adding more statistics on stress test runs in order to use them as benchmarks. I don't know if this kind of feature was already discussed in the OpenStack community but since stress tests are a bit deprecated now, maybe there is some room for this kind of improvement on fresh stress tests. Best Regards, Julien LELOUP -Original Message- From: Koderer, Marc [mailto:m.kode...@telekom.de] Sent: Friday, January 17, 2014 9:45 AM To: LELOUP Julien Cc: openstack-dev@lists.openstack.org Subject: [qa] RE: [Tempest - Stress Test] : implement a full SSH connection on ssh_floating.py and improve it Hello Juilen, I forwarded your mail to the correct mailing list. Please do not use the qa list any longer. I am happy that you are interested in stress
Re: [openstack-dev] [qa] [Tempest - Stress Test] : implement a full SSH connection on ssh_floating.py and improve it
Hello Julien, maybe my blog post helps you with some more details: http://telekomcloud.github.io/2013/09/11/new-ways-of-tempest-stress-testing.html You can run single test if you add a new json file with the test function you want to test. Like: https://github.com/openstack/tempest/blob/master/tempest/stress/etc/sample-unit-test.json With that you can launch them with the parameters you already described. Regards, Marc From: LELOUP Julien [julien.lel...@3ds.com] Sent: Friday, January 17, 2014 3:49 PM To: Koderer, Marc Cc: openstack-dev@lists.openstack.org Subject: RE: [Tempest - Stress Test] : implement a full SSH connection on ssh_floating.py and improve it Hi Marc, The Etherpad you provided was helpful to know the current state of the stress tests. I admit that I have some difficulties to understand how I can run a single test built with the @stresstest decorator (even not a beginner in Python, I still have things to learn on this technology and a lot more on OpenStack/Tempest :) ). I used to run my test using ./run_stress.py -t JSON configuration pointing at my action/.py script -d duration, which allowed me to run only one test with a dedicated configuration (number of threads, ...) For what I understand now in Tempest, I only managed to run all tests, using ./run_tests.sh and the only configuration I found related to stress tests was the [stress] section in tempest.conf. For example : let say I ported my SSH stress test as a scenario test with the @stresstest decorator. How can I launch this test (and only this one) and use a dedicated configuration file like ones we can found in tempest/stress/etc ? Another question I have now : in the case that I have to use run_test.sh and not run_stress.py anymore, how do I get the test runs statistics I used to have, and where should I put some code to improve them ? When I will have cleared my mind with all these kinds of practical details, maybe I should add some content on the Wiki about stress tests in Tempest. Best Regards, Julien LELOUP julien.lel...@3ds.com -Original Message- From: Koderer, Marc [mailto:m.kode...@telekom.de] Sent: Friday, January 17, 2014 3:07 PM To: LELOUP Julien Cc: openstack-dev@lists.openstack.org Subject: RE: [qa] RE: [Tempest - Stress Test] : implement a full SSH connection on ssh_floating.py and improve it Hi Julien, most of the cases in tempest/stress are already covered by exiting tests in /api or /scenario. The only thing that is missing is the decorator on them. BTW here is the Etherpad from the summit talk that we had: https://etherpad.openstack.org/p/icehouse-summit-qa-stress-tests It possible help to understand the state. I didn't managed to work on the action items that are left. Your suggestions sound good. So I'd happy so see some patches :) Regards Marc From: LELOUP Julien [julien.lel...@3ds.com] Sent: Friday, January 17, 2014 11:52 AM To: Koderer, Marc Cc: openstack-dev@lists.openstack.org Subject: RE: [qa] RE: [Tempest - Stress Test] : implement a full SSH connection on ssh_floating.py and improve it Hello Marc, Thanks for your answer. At the moment I'm willing to spend some time on this kind of scenario so I will see how to use the stress decorator inside a scenario test. Does this means that all stress tests available in tempest/stress should be ported as scenario tests with this decorator ? I do have some ideas about features on stress test that I find useful for my own use case : like adding more statistics on stress test runs in order to use them as benchmarks. I don't know if this kind of feature was already discussed in the OpenStack community but since stress tests are a bit deprecated now, maybe there is some room for this kind of improvement on fresh stress tests. Best Regards, Julien LELOUP -Original Message- From: Koderer, Marc [mailto:m.kode...@telekom.de] Sent: Friday, January 17, 2014 9:45 AM To: LELOUP Julien Cc: openstack-dev@lists.openstack.org Subject: [qa] RE: [Tempest - Stress Test] : implement a full SSH connection on ssh_floating.py and improve it Hello Juilen, I forwarded your mail to the correct mailing list. Please do not use the qa list any longer. I am happy that you are interested in stress tests. All the tests in tempest/stress/actions are more or less deprecated. So what you should use instead is the stress decorator (e.g. https://github.com/openstack/tempest/blob/master/tempest/api/volume/test_volumes_actions.py#L55). Unfortunately it's not yet used for scenarios like you describe. I'd suggest to build a scenario test in tempest/scenario and use this decorator on it. Any patch like that is welcome on gerrit. If you are planning to work in that area for more than just a patch, a blueprint would be nice. A good way to coordinate your efforts is also in the QA meeting (https://wiki.openstack.org/wiki/Meetings/QATeamMeeting)
