Re: [openstack-dev] [Infra][Neutron] Request voting for Tail-f CI account
Thanks everybody! Onward :-) On 24 July 2014 19:41, Anita Kuno ante...@anteaya.info wrote: On 07/24/2014 01:18 PM, Kyle Mestery wrote: On Thu, Jul 24, 2014 at 12:03 PM, Collins, Sean sean_colli...@cable.comcast.com wrote: On Wed, Jul 23, 2014 at 11:19:13AM EDT, Luke Gorrie wrote: Tail-f NCS: I want to keep this feature well maintained and compliant with all the rules. I am the person who wrote this driver originally, I have been the responsible person for 90% of its lifetime, I am the person who setup the current CI, and I am the one responsible for smooth operation of that CI. I am reviewing its results with my morning coffee and have been doing so for the past 6 weeks. I would like to have it start voting and I believe that it and I are ready for that. I am responsive to email, I am usually on IRC (lukego), and in case of emergency you can SMS/call my mobile on +41 79 244 32 17. So... Let's be friends again? (and do ever cooler stuff in Kilo?) Luke was kind enough to reach out to me, and we had a discussion in order to bury the hatchet. Posting his contact details and being available to discuss things has put my mind at ease, I am ready to move forward. +1 He also reached out to me, so I'm also happy to add this back and move forward with burying the hatchet. I'm all for second chances in general, and Luke's gone out of his way to work with people upstream in a much more efficient and effective manner. Thanks, Kyle Well done, Luke. It takes a lot of work to dig oneself out of a hole and create good relationships where there need to be some. It is a tough job and not everyone chooses to do it. You chose to and you succeeded. I commend your work. I'm glad we have a good resolution in this space. Thanks to all involved for their persistence and hard work. Well done, Anita. -- Sean M. Collins ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Infra][Neutron] Request voting for Tail-f CI account
On Wed, Jul 23, 2014 at 11:19:13AM EDT, Luke Gorrie wrote: Tail-f NCS: I want to keep this feature well maintained and compliant with all the rules. I am the person who wrote this driver originally, I have been the responsible person for 90% of its lifetime, I am the person who setup the current CI, and I am the one responsible for smooth operation of that CI. I am reviewing its results with my morning coffee and have been doing so for the past 6 weeks. I would like to have it start voting and I believe that it and I are ready for that. I am responsive to email, I am usually on IRC (lukego), and in case of emergency you can SMS/call my mobile on +41 79 244 32 17. So... Let's be friends again? (and do ever cooler stuff in Kilo?) Luke was kind enough to reach out to me, and we had a discussion in order to bury the hatchet. Posting his contact details and being available to discuss things has put my mind at ease, I am ready to move forward. -- Sean M. Collins ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Infra][Neutron] Request voting for Tail-f CI account
On Thu, Jul 24, 2014 at 12:03 PM, Collins, Sean sean_colli...@cable.comcast.com wrote: On Wed, Jul 23, 2014 at 11:19:13AM EDT, Luke Gorrie wrote: Tail-f NCS: I want to keep this feature well maintained and compliant with all the rules. I am the person who wrote this driver originally, I have been the responsible person for 90% of its lifetime, I am the person who setup the current CI, and I am the one responsible for smooth operation of that CI. I am reviewing its results with my morning coffee and have been doing so for the past 6 weeks. I would like to have it start voting and I believe that it and I are ready for that. I am responsive to email, I am usually on IRC (lukego), and in case of emergency you can SMS/call my mobile on +41 79 244 32 17. So... Let's be friends again? (and do ever cooler stuff in Kilo?) Luke was kind enough to reach out to me, and we had a discussion in order to bury the hatchet. Posting his contact details and being available to discuss things has put my mind at ease, I am ready to move forward. +1 He also reached out to me, so I'm also happy to add this back and move forward with burying the hatchet. I'm all for second chances in general, and Luke's gone out of his way to work with people upstream in a much more efficient and effective manner. Thanks, Kyle -- Sean M. Collins ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Infra][Neutron] Request voting for Tail-f CI account
On 07/24/2014 01:18 PM, Kyle Mestery wrote: On Thu, Jul 24, 2014 at 12:03 PM, Collins, Sean sean_colli...@cable.comcast.com wrote: On Wed, Jul 23, 2014 at 11:19:13AM EDT, Luke Gorrie wrote: Tail-f NCS: I want to keep this feature well maintained and compliant with all the rules. I am the person who wrote this driver originally, I have been the responsible person for 90% of its lifetime, I am the person who setup the current CI, and I am the one responsible for smooth operation of that CI. I am reviewing its results with my morning coffee and have been doing so for the past 6 weeks. I would like to have it start voting and I believe that it and I are ready for that. I am responsive to email, I am usually on IRC (lukego), and in case of emergency you can SMS/call my mobile on +41 79 244 32 17. So... Let's be friends again? (and do ever cooler stuff in Kilo?) Luke was kind enough to reach out to me, and we had a discussion in order to bury the hatchet. Posting his contact details and being available to discuss things has put my mind at ease, I am ready to move forward. +1 He also reached out to me, so I'm also happy to add this back and move forward with burying the hatchet. I'm all for second chances in general, and Luke's gone out of his way to work with people upstream in a much more efficient and effective manner. Thanks, Kyle Well done, Luke. It takes a lot of work to dig oneself out of a hole and create good relationships where there need to be some. It is a tough job and not everyone chooses to do it. You chose to and you succeeded. I commend your work. I'm glad we have a good resolution in this space. Thanks to all involved for their persistence and hard work. Well done, Anita. -- Sean M. Collins ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Infra][Neutron] Request voting for Tail-f CI account
On 22 July 2014 11:06, Luke Gorrie l...@tail-f.com wrote: End of Part One. Let's skip Part Two. That is just more frustration. Let's talk about Part Three in which we all do awesome CI hacking in Juno together :-). Here is what I want to achieve in Juno: NFV CI: Myself and my colleagues are developing the open source Neutron networking for Deutsche Telekom's TeraStream project (and we want to bring up a CI that tests this configuration. That will exercise new and exciting-for-NFV features of QEMU, Libvirt, Nova, and Neutron. This should serve several purposes: making TeraStream a success story for OpenStack and Neutron, making the whole design easy to replicate for other users (it's already open source), and providing test coverage for more OpenStack features. (Good stuff for everybody, I hope! More info: http://blog.ipspace.net/2013/11/deutsche-telekom-terastream-designed.html) People: I want to onboard great new open source hackers into the OpenStack community and get them contributing to CI. I am right now bringing new people up to speed on OpenStack development and working with them on bringing up our NFV CI this month. shellci: I want to make shellci a practical alternative for CI operators whose style is more screen+bash+awk than jenkins+zuul+nodepool. The development is already done, and it works great in my own tests, so now we plan to battle test it on the NFV CI. (link: https://github.com/SnabbCo/shellci) Tail-f NCS: I want to keep this feature well maintained and compliant with all the rules. I am the person who wrote this driver originally, I have been the responsible person for 90% of its lifetime, I am the person who setup the current CI, and I am the one responsible for smooth operation of that CI. I am reviewing its results with my morning coffee and have been doing so for the past 6 weeks. I would like to have it start voting and I believe that it and I are ready for that. I am responsive to email, I am usually on IRC (lukego), and in case of emergency you can SMS/call my mobile on +41 79 244 32 17. So... Let's be friends again? (and do ever cooler stuff in Kilo?) Cheers! -Luke ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Infra][Neutron] Request voting for Tail-f CI account
Hi Sean, On 21 July 2014 22:53, Collins, Sean sean_colli...@cable.comcast.com wrote: The fact that I tried to reach out to the person who was listed as the contact back in November to try and resolve the –1 that this CI system gave, and never received a response until the public mailing list thread about revoking voting rights for Tail-F, makes me believe that the Tail-F CI system is still not ready to have that kind of privilege. Especially if the account was idle from around February, until June – that is a huge gap, if I understand correctly? I understand your frustration. It seems like the experience of bringing up our CI has been miserable for all concerned. I am sad about that. It does not seem that it should have worked out this way, since everybody concerned is a competent person and acting in good faith. I hope we can finally clear this up and then continue with contributing to OpenStack on good terms with everybody. Back in November we were feeling eager to be good citizens and we wanted to be amongst the first to setup a 3rd party CI for Neutron. We were trying to be proactive: our driver was already in Havana and the deadlines for us to setup the CI were far in the future. My colleague Tobbe was also planning to take the lead on development of our OpenStack code from me and we thought the perfect first step would be to setup our CI system, since that would get him familiar with the code and since neither of us had prior experience operating an OpenStack CI. We read through the 3rd Party CI setup instructions and created a CI. Our initial setup ran Jenkins and would use a custom script to create a one-shot VM and inside that it would run the Neutron unit tests together with a patch that made our driver talk to our real external system. This got quite good test coverage because the unit tests really exercise the ML2 interface quite well. (Likely we should have used Tempest instead, as everybody does nowadays include us, but we didn't know that back then.) This seemed to work well and so we let it run. Honestly, we did not really know what would happen with our results after they were posted, and we did not have a definite goal for what service level we should uphold. That was surely naive, but I think understandable. We were relatively new and minor contributors to OpenStack and we were amongst the first wave of Neutron people to setup a CI. We hadn't yet had the opportunity to learn from the mistakes of others or see how reviews are used by the upstream people and systems. We were also perhaps a little too relaxed because our total contribution was around 150 lines of code that only run when explicitly enabled, and we had our own test procedure in place separately from OpenStack CI that we had been using since Havana, so it did not feel like we had much potential to impact other OpenStack users and developers with our code. Anyway. The test runs started to fail unexpectedly, for a boring kind of reason like that OpenStack needed a newer version of a library and our CI script lacked a pip upgrade command that would pick it up, so all tests would fail until manual intervention. So what happens when the CI falls down and needs help to come back up? First of all, it creates a big problem for upstream developers and slows down work on OpenStack (ouch). Second, you poor guys who are having problems try to contact the person responsible, but all you have is one work email address and IRC nick. In that case, you guys did not get a response. I think that was for the very pedestrian reason that my colleague who was responsible was on vacation and didn't appreciate that an operational issue with our CI would create an urgent problem for other people and must be attended to at all times. This must have been bad for you guys since you were stuck waiting on us and couldn't fix the problem on your side. I was also contacted by email, as the previous contact person for that driver, but the message simply asked me to confirm my colleague's email address and did not tell me that there was a problem that we had to resolve. So eventually the problem boiled over and when we started getting publicly flamed on the mailing list then I finally saw that there was an issue and called up my colleague directly who *then* jumped into account to sort it out (logging into gerrit and reversing old negative votes, and so on). So what do we take away from this first experience? To me it just looks like processes to fix: people operating 3rd party CIs need to better understand the required service level, there should be multiple contact points to deal with mundane stuff like vacations and illness, and that people should operate their CI successfully for a while before voting is enabled. It sucks that work was interrupted and people got mad, but at the end of the day this happened with everybody acting in good faith, and it shows us what kind of problems to prevent in the future. This is where it became a bit
Re: [openstack-dev] [Infra][Neutron] Request voting for Tail-f CI account
On 22 July 2014 11:06, Luke Gorrie l...@tail-f.com wrote: This must have been bad for you guys since you were stuck waiting on us and couldn't fix the problem on your side. I was also contacted by email, as the previous contact person for that driver, but the message simply asked me to confirm my colleague's email address and did not tell me that there was a problem that we had to resolve. (I checked and that is not true: actually it did tell me that there was a problem, and I just didn't get that it was urgent. This narrative is a little clouded with emotion at this point I must admit :-)). ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Infra][Neutron] Request voting for Tail-f CI account
From: Luke Gorrie l...@snabb.comailto:l...@snabb.co Reply-To: OpenStack Development Mailing List (not for usage questions) openstack-dev@lists.openstack.orgmailto:openstack-dev@lists.openstack.org Date: Monday, July 21, 2014 3:22 PM To: OpenStack Development Mailing List openstack-dev@lists.openstack.orgmailto:openstack-dev@lists.openstack.org Subject: [openstack-dev] [Infra][Neutron] Request voting for Tail-f CI account The account has a rich history :-). Initially we brought it online back around Nov 2013 early in the Icehouse cycle. That didn't work out so well: we had a bunch of operational issues and as OpenStack newbies we were oblivious to the impact they had on other people's workflows -- we were mortified to learn that we had created a disruption. Since then we have been more conservative which is why the account was mostly idle until June. The fact that I tried to reach out to the person who was listed as the contact back in November to try and resolve the –1 that this CI system gave, and never received a response until the public mailing list thread about revoking voting rights for Tail-F, makes me believe that the Tail-F CI system is still not ready to have that kind of privilege. Especially if the account was idle from around February, until June – that is a huge gap, if I understand correctly? -- Sean M. Collins ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
