Re: [openstack-dev] [neutron] [third-party] Current status of Neutron 3rd Party CI and how to move forward
I will probably be unable, as usual, to attend today's CI meeting (falls right around my dinner time). I think it's a good idea to starting keeping track of the status of the various CI systems, but I feel the etherpad will not work very well in the long term. However, it would be great if we could start devising a solution for having health reports from the various CI systems. This report should report the following kind of information: - timestamp of last run - timestamp of last vote (a system might start job which then get aborted for CI infra problems) - % of success vs failures (not sure how important is that one but provides a metric of comparison with upstream jenkins) - % of disagreements with jenkins (this might allow us to quickly spot those CI systems which are randomly -1'ing patches) The percentage metrics might be taken over a 48 hours or 7 days interval, or both. Does this idea sound reasonable? Also, regarding [1]. I agree that more is always better... but I would like a minimum required set of tests to be enforced. Is this something that can be achieved? Salvatore [1] https://wiki.openstack.org/wiki/NeutronThirdPartyTesting On 16 June 2014 07:07, YAMAMOTO Takashi yamam...@valinux.co.jp wrote: hi, My initial analysis of Neutron 3rd Party CI is here [1]. This was somewhat correlated with information from DriverLog [2], which was helpful to put this together. i updated the etherpad for ofagent. currently a single CI system is running tests for both of ofagent and ryu. is it ok? YAMAMOTO Takashi ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [neutron] [third-party] Current status of Neutron 3rd Party CI and how to move forward
On Mon, Jun 16, 2014 at 8:52 AM, Salvatore Orlando sorla...@nicira.com wrote: I will probably be unable, as usual, to attend today's CI meeting (falls right around my dinner time). I think it's a good idea to starting keeping track of the status of the various CI systems, but I feel the etherpad will not work very well in the long term. Agreed. The etherpad was a starting point, I'll move this information to a wiki page later today. However, it would be great if we could start devising a solution for having health reports from the various CI systems. This report should report the following kind of information: - timestamp of last run - timestamp of last vote (a system might start job which then get aborted for CI infra problems) - % of success vs failures (not sure how important is that one but provides a metric of comparison with upstream jenkins) - % of disagreements with jenkins (this might allow us to quickly spot those CI systems which are randomly -1'ing patches) The percentage metrics might be taken over a 48 hours or 7 days interval, or both. Does this idea sound reasonable? This sounds like a very good idea. Now we just need to find someone with the time to write this. :) Also, regarding [1]. I agree that more is always better... but I would like a minimum required set of tests to be enforced. Is this something that can be achieved? I think the tests that are in there are the minimum tests I'd like to see run. I'll clarify the language on the wiki page a bit. Salvatore [1] https://wiki.openstack.org/wiki/NeutronThirdPartyTesting On 16 June 2014 07:07, YAMAMOTO Takashi yamam...@valinux.co.jp wrote: hi, My initial analysis of Neutron 3rd Party CI is here [1]. This was somewhat correlated with information from DriverLog [2], which was helpful to put this together. i updated the etherpad for ofagent. currently a single CI system is running tests for both of ofagent and ryu. is it ok? YAMAMOTO Takashi ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [neutron] [third-party] Current status of Neutron 3rd Party CI and how to move forward
On 16 June 2014 15:58, Kyle Mestery mest...@noironetworks.com wrote: On Mon, Jun 16, 2014 at 8:52 AM, Salvatore Orlando sorla...@nicira.com wrote: I will probably be unable, as usual, to attend today's CI meeting (falls right around my dinner time). I think it's a good idea to starting keeping track of the status of the various CI systems, but I feel the etherpad will not work very well in the long term. Agreed. The etherpad was a starting point, I'll move this information to a wiki page later today. However, it would be great if we could start devising a solution for having health reports from the various CI systems. This report should report the following kind of information: - timestamp of last run - timestamp of last vote (a system might start job which then get aborted for CI infra problems) - % of success vs failures (not sure how important is that one but provides a metric of comparison with upstream jenkins) - % of disagreements with jenkins (this might allow us to quickly spot those CI systems which are randomly -1'ing patches) The percentage metrics might be taken over a 48 hours or 7 days interval, or both. Does this idea sound reasonable? This sounds like a very good idea. Now we just need to find someone with the time to write this. :) Also, regarding [1]. I agree that more is always better... but I would like a minimum required set of tests to be enforced. Is this something that can be achieved? I think the tests that are in there are the minimum tests I'd like to see run. I'll clarify the language on the wiki page a bit. If the intention is to have the minimum set of test we'd like to see run then it's perfect! I was trying to say we should impose that set as a minimum requirement... Salvatore [1] https://wiki.openstack.org/wiki/NeutronThirdPartyTesting On 16 June 2014 07:07, YAMAMOTO Takashi yamam...@valinux.co.jp wrote: hi, My initial analysis of Neutron 3rd Party CI is here [1]. This was somewhat correlated with information from DriverLog [2], which was helpful to put this together. i updated the etherpad for ofagent. currently a single CI system is running tests for both of ofagent and ryu. is it ok? YAMAMOTO Takashi ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [neutron] [third-party] Current status of Neutron 3rd Party CI and how to move forward
However, it would be great if we could start devising a solution for having health reports from the various CI systems. This report should report the following kind of information: - timestamp of last run - timestamp of last vote (a system might start job which then get aborted for CI infra problems) - % of success vs failures (not sure how important is that one but provides a metric of comparison with upstream jenkins) - % of disagreements with jenkins (this might allow us to quickly spot those CI systems which are randomly -1'ing patches) The percentage metrics might be taken over a 48 hours or 7 days interval, or both. Does this idea sound reasonable? This sounds like a very good idea. Now we just need to find someone with the time to write this. :) That's exactly what Stackalytics/DriverLog may do! It already collects the latest CI votes and it wouldn't be hard to calculate metrics based on them. Ilya 2014-06-16 18:11 GMT+04:00 Salvatore Orlando sorla...@nicira.com: On 16 June 2014 15:58, Kyle Mestery mest...@noironetworks.com wrote: On Mon, Jun 16, 2014 at 8:52 AM, Salvatore Orlando sorla...@nicira.com wrote: I will probably be unable, as usual, to attend today's CI meeting (falls right around my dinner time). I think it's a good idea to starting keeping track of the status of the various CI systems, but I feel the etherpad will not work very well in the long term. Agreed. The etherpad was a starting point, I'll move this information to a wiki page later today. However, it would be great if we could start devising a solution for having health reports from the various CI systems. This report should report the following kind of information: - timestamp of last run - timestamp of last vote (a system might start job which then get aborted for CI infra problems) - % of success vs failures (not sure how important is that one but provides a metric of comparison with upstream jenkins) - % of disagreements with jenkins (this might allow us to quickly spot those CI systems which are randomly -1'ing patches) The percentage metrics might be taken over a 48 hours or 7 days interval, or both. Does this idea sound reasonable? This sounds like a very good idea. Now we just need to find someone with the time to write this. :) Also, regarding [1]. I agree that more is always better... but I would like a minimum required set of tests to be enforced. Is this something that can be achieved? I think the tests that are in there are the minimum tests I'd like to see run. I'll clarify the language on the wiki page a bit. If the intention is to have the minimum set of test we'd like to see run then it's perfect! I was trying to say we should impose that set as a minimum requirement... Salvatore [1] https://wiki.openstack.org/wiki/NeutronThirdPartyTesting On 16 June 2014 07:07, YAMAMOTO Takashi yamam...@valinux.co.jp wrote: hi, My initial analysis of Neutron 3rd Party CI is here [1]. This was somewhat correlated with information from DriverLog [2], which was helpful to put this together. i updated the etherpad for ofagent. currently a single CI system is running tests for both of ofagent and ryu. is it ok? YAMAMOTO Takashi ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [neutron] [third-party] Current status of Neutron 3rd Party CI and how to move forward
On 06/16/2014 10:25 AM, Ilya Shakhat wrote: However, it would be great if we could start devising a solution for having health reports from the various CI systems. This report should report the following kind of information: - timestamp of last run - timestamp of last vote (a system might start job which then get aborted for CI infra problems) - % of success vs failures (not sure how important is that one but provides a metric of comparison with upstream jenkins) - % of disagreements with jenkins (this might allow us to quickly spot those CI systems which are randomly -1'ing patches) The percentage metrics might be taken over a 48 hours or 7 days interval, or both. Does this idea sound reasonable? This sounds like a very good idea. Now we just need to find someone with the time to write this. :) That's exactly what Stackalytics/DriverLog may do! It already collects the latest CI votes and it wouldn't be hard to calculate metrics based on them. Ilya Hi Ilya: Can you add an agenda item to today's Third Party meeting agenda so that you can outline how you address this currently, any bugs you are aware of and your current direction of development. Thanks, Anita. 2014-06-16 18:11 GMT+04:00 Salvatore Orlando sorla...@nicira.com: On 16 June 2014 15:58, Kyle Mestery mest...@noironetworks.com wrote: On Mon, Jun 16, 2014 at 8:52 AM, Salvatore Orlando sorla...@nicira.com wrote: I will probably be unable, as usual, to attend today's CI meeting (falls right around my dinner time). I think it's a good idea to starting keeping track of the status of the various CI systems, but I feel the etherpad will not work very well in the long term. Agreed. The etherpad was a starting point, I'll move this information to a wiki page later today. However, it would be great if we could start devising a solution for having health reports from the various CI systems. This report should report the following kind of information: - timestamp of last run - timestamp of last vote (a system might start job which then get aborted for CI infra problems) - % of success vs failures (not sure how important is that one but provides a metric of comparison with upstream jenkins) - % of disagreements with jenkins (this might allow us to quickly spot those CI systems which are randomly -1'ing patches) The percentage metrics might be taken over a 48 hours or 7 days interval, or both. Does this idea sound reasonable? This sounds like a very good idea. Now we just need to find someone with the time to write this. :) Also, regarding [1]. I agree that more is always better... but I would like a minimum required set of tests to be enforced. Is this something that can be achieved? I think the tests that are in there are the minimum tests I'd like to see run. I'll clarify the language on the wiki page a bit. If the intention is to have the minimum set of test we'd like to see run then it's perfect! I was trying to say we should impose that set as a minimum requirement... Salvatore [1] https://wiki.openstack.org/wiki/NeutronThirdPartyTesting On 16 June 2014 07:07, YAMAMOTO Takashi yamam...@valinux.co.jp wrote: hi, My initial analysis of Neutron 3rd Party CI is here [1]. This was somewhat correlated with information from DriverLog [2], which was helpful to put this together. i updated the etherpad for ofagent. currently a single CI system is running tests for both of ofagent and ryu. is it ok? YAMAMOTO Takashi ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [neutron] [third-party] Current status of Neutron 3rd Party CI and how to move forward
On 06/16/2014 09:52 AM, Salvatore Orlando wrote: I will probably be unable, as usual, to attend today's CI meeting (falls right around my dinner time). I'm sorry to hear that, since I value your input in these matters. Your CI system is doing an excellent job and I use it as an example for others. Maybe we need to get you speaking somewhere so you can be in a different time zone for the meeting? :D Thanks Salvatore, Anita. I think it's a good idea to starting keeping track of the status of the various CI systems, but I feel the etherpad will not work very well in the long term. However, it would be great if we could start devising a solution for having health reports from the various CI systems. This report should report the following kind of information: - timestamp of last run - timestamp of last vote (a system might start job which then get aborted for CI infra problems) - % of success vs failures (not sure how important is that one but provides a metric of comparison with upstream jenkins) - % of disagreements with jenkins (this might allow us to quickly spot those CI systems which are randomly -1'ing patches) The percentage metrics might be taken over a 48 hours or 7 days interval, or both. Does this idea sound reasonable? Also, regarding [1]. I agree that more is always better... but I would like a minimum required set of tests to be enforced. Is this something that can be achieved? Salvatore [1] https://wiki.openstack.org/wiki/NeutronThirdPartyTesting On 16 June 2014 07:07, YAMAMOTO Takashi yamam...@valinux.co.jp wrote: hi, My initial analysis of Neutron 3rd Party CI is here [1]. This was somewhat correlated with information from DriverLog [2], which was helpful to put this together. i updated the etherpad for ofagent. currently a single CI system is running tests for both of ofagent and ryu. is it ok? YAMAMOTO Takashi ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [neutron] [third-party] Current status of Neutron 3rd Party CI and how to move forward
hi, My initial analysis of Neutron 3rd Party CI is here [1]. This was somewhat correlated with information from DriverLog [2], which was helpful to put this together. i updated the etherpad for ofagent. currently a single CI system is running tests for both of ofagent and ryu. is it ok? YAMAMOTO Takashi ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [neutron] [third-party] Current status of Neutron 3rd Party CI and how to move forward
Hi Kyle, Arista CI has been voting +1 for success and comments in case of Failures. Are the CI's now allowed to post -1 for failures? I have to make a minor change to start voting -1. Please advise. -Sukhdev On Fri, Jun 13, 2014 at 10:07 AM, Kyle Mestery mest...@noironetworks.com wrote: I've spent some time doing some initial analysis of 3rd Party CI in Neutron. The tl;dr is that it's a mess, and it needs fixing. And I'm setting a deadline of Juno-2 for people to address their CI systems and get them in shape or we will remove plugins and drivers in Juno-3 which do not meet the expectations set out below. My initial analysis of Neutron 3rd Party CI is here [1]. This was somewhat correlated with information from DriverLog [2], which was helpful to put this together. As you can see from the list, there are a lot of CI systems which are broken right now. Some have just recently started working again. Others are working great, and some are in the middle somewhere. The overall state isn't that great. I'm sending this email to openstack-dev and BCC;ing CI owners to raise awareness of this issue. If I have incorrectly labeled your CI, please update the etherpad and include links to the latest voting/comments your CI system has done upstream and reply to this thread. I have documented the 3rd Party CI requirements for Neutron here [3]. I expect people to be following these guidelines for their CI systems. If there are questions on the guidelines or expectations, please reply to this thread or reach out to myself in #openstack-neutron on Freenode. There is also a third-party meeting [4] which is a great place to ask questions and share your experience setting up a 3rd party CI system. The infra team has done a great job sponsoring and running this meeting (thanks Anita!), so please both take advantage of it and also contribute to it so we can all share knowledge and help each other. Owners of plugins/drivers should ensure their CI is matching the requirements set forth by both infra and Neutron when running tests and posting results. Like I indicated earlier, we will look at removing code for drivers which are not meeting these requirements as set forth in the wiki pages. The goal of this effort is to ensure consistency across testing platforms, making it easier for developers to diagnose issues when third party CI systems fail, and to ensure these drivers are tested since they are part of the integrated releases we perform. We used to require a core team member to sponsor a plugin/driver, but we moved to the 3rd party CI system in Icehouse instead. Ensuring these systems are running and properly working is the only way we can ensure code is working when it's part of the integrated release. Thanks, Kyle [1] https://etherpad.openstack.org/p/ZLp9Ow3tNq [2] http://www.stackalytics.com/driverlog/?project_id=openstack%2Fneutronvendor=release_id= [3] https://wiki.openstack.org/wiki/NeutronThirdPartyTesting [4] https://wiki.openstack.org/wiki/Meetings/ThirdParty ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [neutron] [third-party] Current status of Neutron 3rd Party CI and how to move forward
I've spent some time doing some initial analysis of 3rd Party CI in Neutron. The tl;dr is that it's a mess, and it needs fixing. And I'm setting a deadline of Juno-2 for people to address their CI systems and get them in shape or we will remove plugins and drivers in Juno-3 which do not meet the expectations set out below. My initial analysis of Neutron 3rd Party CI is here [1]. This was somewhat correlated with information from DriverLog [2], which was helpful to put this together. As you can see from the list, there are a lot of CI systems which are broken right now. Some have just recently started working again. Others are working great, and some are in the middle somewhere. The overall state isn't that great. I'm sending this email to openstack-dev and BCC;ing CI owners to raise awareness of this issue. If I have incorrectly labeled your CI, please update the etherpad and include links to the latest voting/comments your CI system has done upstream and reply to this thread. I have documented the 3rd Party CI requirements for Neutron here [3]. I expect people to be following these guidelines for their CI systems. If there are questions on the guidelines or expectations, please reply to this thread or reach out to myself in #openstack-neutron on Freenode. There is also a third-party meeting [4] which is a great place to ask questions and share your experience setting up a 3rd party CI system. The infra team has done a great job sponsoring and running this meeting (thanks Anita!), so please both take advantage of it and also contribute to it so we can all share knowledge and help each other. Owners of plugins/drivers should ensure their CI is matching the requirements set forth by both infra and Neutron when running tests and posting results. Like I indicated earlier, we will look at removing code for drivers which are not meeting these requirements as set forth in the wiki pages. The goal of this effort is to ensure consistency across testing platforms, making it easier for developers to diagnose issues when third party CI systems fail, and to ensure these drivers are tested since they are part of the integrated releases we perform. We used to require a core team member to sponsor a plugin/driver, but we moved to the 3rd party CI system in Icehouse instead. Ensuring these systems are running and properly working is the only way we can ensure code is working when it's part of the integrated release. Thanks, Kyle [1] https://etherpad.openstack.org/p/ZLp9Ow3tNq [2] http://www.stackalytics.com/driverlog/?project_id=openstack%2Fneutronvendor=release_id= [3] https://wiki.openstack.org/wiki/NeutronThirdPartyTesting [4] https://wiki.openstack.org/wiki/Meetings/ThirdParty ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [neutron] [third-party] Current status of Neutron 3rd Party CI and how to move forward
Hi Kyle, Embrane's CI was blocked by some nasty bugs affecting the testing environment. It resumed yesterday (6/12) [0]. Unfortunately it's still non voting (only commenting so far). Not sure if this is a requirement or not, but it should be able to put +1/-1 immediately after the voting right is granted. I'll keep an eye on it to make sure it is stable again. Thanks sorting out the CI situation :D! [0] https://review.openstack.org/#/c/98813/ On Fri, Jun 13, 2014 at 10:07 AM, Kyle Mestery mest...@noironetworks.com wrote: I've spent some time doing some initial analysis of 3rd Party CI in Neutron. The tl;dr is that it's a mess, and it needs fixing. And I'm setting a deadline of Juno-2 for people to address their CI systems and get them in shape or we will remove plugins and drivers in Juno-3 which do not meet the expectations set out below. My initial analysis of Neutron 3rd Party CI is here [1]. This was somewhat correlated with information from DriverLog [2], which was helpful to put this together. As you can see from the list, there are a lot of CI systems which are broken right now. Some have just recently started working again. Others are working great, and some are in the middle somewhere. The overall state isn't that great. I'm sending this email to openstack-dev and BCC;ing CI owners to raise awareness of this issue. If I have incorrectly labeled your CI, please update the etherpad and include links to the latest voting/comments your CI system has done upstream and reply to this thread. I have documented the 3rd Party CI requirements for Neutron here [3]. I expect people to be following these guidelines for their CI systems. If there are questions on the guidelines or expectations, please reply to this thread or reach out to myself in #openstack-neutron on Freenode. There is also a third-party meeting [4] which is a great place to ask questions and share your experience setting up a 3rd party CI system. The infra team has done a great job sponsoring and running this meeting (thanks Anita!), so please both take advantage of it and also contribute to it so we can all share knowledge and help each other. Owners of plugins/drivers should ensure their CI is matching the requirements set forth by both infra and Neutron when running tests and posting results. Like I indicated earlier, we will look at removing code for drivers which are not meeting these requirements as set forth in the wiki pages. The goal of this effort is to ensure consistency across testing platforms, making it easier for developers to diagnose issues when third party CI systems fail, and to ensure these drivers are tested since they are part of the integrated releases we perform. We used to require a core team member to sponsor a plugin/driver, but we moved to the 3rd party CI system in Icehouse instead. Ensuring these systems are running and properly working is the only way we can ensure code is working when it's part of the integrated release. Thanks, Kyle [1] https://etherpad.openstack.org/p/ZLp9Ow3tNq [2] http://www.stackalytics.com/driverlog/?project_id=openstack%2Fneutronvendor=release_id= [3] https://wiki.openstack.org/wiki/NeutronThirdPartyTesting [4] https://wiki.openstack.org/wiki/Meetings/ThirdParty ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
