Re: [openstack-dev] [all] Improving Vendor Driver Discoverability
On 20/01/17 01:40, Mike Perez wrote: > On 17:38 Jan 18, Morales, Victor wrote: >> Just a FYI, Ankur have been working on have a Feature Classification Matrix >> in Neutron[1] which collects some of this information >> >> [1] https://review.openstack.org/#/c/318192/ > > I actually didn't know Nova also generated this with a script and ini file. > Perhaps this would be a better approach than a giant JSON file like driver log > is today. I could then have the marketplace parse these ini files using the > common script. What do others think? > FWIW Designate also does this - [0] is generated by [1] and a modified version of the Nova script. If there is a common way, we will use that, but I like our current implementation. 0 - http://docs.openstack.org/developer/designate/support-matrix.html 1 - https://git.openstack.org/cgit/openstack/designate/tree/doc/source/support-matrix.ini __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] Improving Vendor Driver Discoverability
On 2017-01-17 02:08 AM, Isaac Beckman wrote: I think that it would also be a good idea to have the option to let the CI maintainers add some useful information on the current status. It is very helpful to know that the CI system is under maintenance which is the reason why it hasn't been reporting for the last week or so... Isaac Beckman Office: +972-3-6897874 Fax: +972-3-6897755 Mobile: +972-50-2680180 Email: isa...@il.ibm.com IBM XIV, Cloud Storage Solutions (previously HSG) www.ibm.com/storage/disk/xiv From: "Jay S. Bryant" <jsbry...@electronicjungle.net> To: "OpenStack Development Mailing List (not for usage questions)" <openstack-dev@lists.openstack.org> Date: 16/01/2017 21:56 Subject: Re: [openstack-dev] [all] Improving Vendor Driver Discoverability On 01/16/2017 12:19 PM, Jonathan Bryce wrote: On Jan 16, 2017, at 11:58 AM, Jay S. Bryant <jsbry...@electronicjungle.net> wrote: On 01/13/2017 10:29 PM, Mike Perez wrote: The way validation works is completely up to the project team. In my research as shown in the Summit etherpad [5] there's a clear trend in projects doing continuous integration for validation. If we wanted to we could also have the marketplace give the current CI results, which was also requested in the feedback from driver maintainers. Having the CI results reported would be an interesting experiment. I wonder if having the results even more publicly reported would result in more stable CI's. It is a dual edged sword however. Given the instability of many CI's it could make OpenStack look bad to customers who don't understand what they are looking at. Just my thoughts on that idea. That?s very useful feedback. Having that kind of background upfront is really helpful. As we make updates on the display side, we can take into account if certain attributes are potentially unreliable or at a higher risk of showing instability and have the interface better support that without it looking like everything is failing and a river of red X?s. Are there other things that might be similar? Jonathan Jonathan, Glad to be of assistance. I think reporting some percentage of success might be the most accurate way to report the CI results. Not necessarily flagging it good or bed but leave it for the consumers to see and compare. Also combine that with Anita's idea of when the CI last successfully reported and I think it could give a good barometer for consumers. Our systems all have their rough times so we need to avoid a 'snapshot in time' view and provide more of a 'activity over time' view. Third party CI is a good barometer of community activity and attention, but not always 100% accurate. Obviously there will need to be some information included with the results explaining what they are and helping guide interpretations. Jay Since the information about system details (contact information, current status - with the option to fill in as many details as you like on your individual wikipage) already exists here: https://wiki.openstack.org/wiki/ThirdPartySystems I think it would be easy to add a link to this wikipage. Thanks, Anita. __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] Improving Vendor Driver Discoverability
Mike, Similar the evil driver support matrix that Cinder has had forever [1]. So, there is a precedent and this looks like it would be better than something that has to be manually updated. No initial objections. :-) Jay [1] https://wiki.openstack.org/wiki/CinderSupportMatrix On 01/19/2017 07:37 PM, Mike Perez wrote: On 17:38 Jan 18, Morales, Victor wrote: Just a FYI, Ankur have been working on have a Feature Classification Matrix in Neutron[1] which collects some of this information [1] https://review.openstack.org/#/c/318192/ I actually didn't know Nova also generated this with a script and ini file. Perhaps this would be a better approach than a giant JSON file like driver log is today. I could then have the marketplace parse these ini files using the common script. What do others think? __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] Improving Vendor Driver Discoverability
On 2017-01-19 17:37:13 -0800 (-0800), Mike Perez wrote: > I actually didn't know Nova also generated this with a script and > ini file. Perhaps this would be a better approach than a giant > JSON file like driver log is today. I could then have the > marketplace parse these ini files using the common script. What do > others think? With at least two existing instances, we can assert this is an emerging common practice. That seems far more palatable than having some new solution pushed at various projects from scratch and insisting that those who have already solved it give up what they're using successfully. -- Jeremy Stanley __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] Improving Vendor Driver Discoverability
On 17:38 Jan 18, Morales, Victor wrote: > Just a FYI, Ankur have been working on have a Feature Classification Matrix > in Neutron[1] which collects some of this information > > [1] https://review.openstack.org/#/c/318192/ I actually didn't know Nova also generated this with a script and ini file. Perhaps this would be a better approach than a giant JSON file like driver log is today. I could then have the marketplace parse these ini files using the common script. What do others think? -- Mike Perez signature.asc Description: PGP signature __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] Improving Vendor Driver Discoverability
Just a FYI, Ankur have been working on have a Feature Classification Matrix in Neutron[1] which collects some of this information [1] https://review.openstack.org/#/c/318192/ Regards/Saludos Victor Morales Irc: electrocucaracha On 1/13/17, 10:29 PM, "Mike Perez"wrote: >Hello all, > >In the spirit of recent Technical Committee discussions I would like to bring >focus on how we're doing vendor driver discoverability. Today we're doing this >with the OpenStack Foundation marketplace [1] which is powered by the driverlog >project. In a nutshell, it is a big JSON file [2] that has information on which >vendor solutions are supported by which projects in which releases. This >information is then parsed to generate the marketplace so that users can >discover them. As discussed in previous TC meetings [3] we need to recognize >vendors that are trying to make great products work in OpenStack so that they >can be successful, which allows our community to be successful and healthy. > >In the feedback I have received from various people in the community, some >didn’t know how it worked, and were unhappy that the projects themselves >weren’t owning this. I totally agree that project teams should own this and >should be encouraged to be involved in the reviews. Today that’s not happening. >I’d like to propose we come up with a way for the marketplace to be more >community-driven by the projects that are validating these solutions. > >At the Barcelona Summit [4] we discussed ways to improve driverlog. Projects >like Nova have a support matrix of hypervisors in their in-tree documentation. >Various members of the Cinder project also expressed interest in using this >solution. It was suggested in the session that the marketplace should just link >to the projects' appropriate documentation. The problem with this solution is >the information is not presented in a consistent way across projects, as >driverlog does it today. We could accomplish this instead by using a parsable >format that is stored in each appropriate project's git repository. I'm >thinking of pretty much how driverlog works today, but broken up into >individual projects. > >The marketplace can parse this information and present it in one place >consistently. Projects may also continue to parse this information in their own >documentation, and we can even write a common tool to do this. The way a vendor >is listed here is based on being validated by the project team itself. Keeping >things in the marketplace would also address the suggestions that came out of >the recent feedback we received from various driver maintainers [4]. > >The way validation works is completely up to the project team. In my research >as shown in the Summit etherpad [5] there's a clear trend in projects doing >continuous integration for validation. If we wanted to we could also have the >marketplace give the current CI results, which was also requested in the >feedback from driver maintainers. > >I would like to volunteer in creating the initial files for each project with >what the marketplace says today. > >[1] - https://www.openstack.org/marketplace/drivers/ >[2] - >http://git.openstack.org/cgit/openstack/driverlog/tree/etc/default_data.json >[3] - >http://eavesdrop.openstack.org/meetings/tc/2017/tc.2017-01-10-20.01.log.html#l-106 >[4] - >http://lists.openstack.org/pipermail/openstack-dev/2017-January/109855.html >[5] - https://etherpad.openstack.org/p/driverlog-validation > >-- >Mike Perez > >__ >OpenStack Development Mailing List (not for usage questions) >Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe >http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] Improving Vendor Driver Discoverability
I think that it would also be a good idea to have the option to let the CI maintainers add some useful information on the current status. It is very helpful to know that the CI system is under maintenance which is the reason why it hasn't been reporting for the last week or so... Isaac Beckman Office: +972-3-6897874 Fax: +972-3-6897755 Mobile: +972-50-2680180 Email: isa...@il.ibm.com IBM XIV, Cloud Storage Solutions (previously HSG) www.ibm.com/storage/disk/xiv From: "Jay S. Bryant" <jsbry...@electronicjungle.net> To: "OpenStack Development Mailing List (not for usage questions)" <openstack-dev@lists.openstack.org> Date: 16/01/2017 21:56 Subject: Re: [openstack-dev] [all] Improving Vendor Driver Discoverability On 01/16/2017 12:19 PM, Jonathan Bryce wrote: >> On Jan 16, 2017, at 11:58 AM, Jay S. Bryant <jsbry...@electronicjungle.net> wrote: >> >> On 01/13/2017 10:29 PM, Mike Perez wrote: >>> The way validation works is completely up to the project team. In my research >>> as shown in the Summit etherpad [5] there's a clear trend in projects doing >>> continuous integration for validation. If we wanted to we could also have the >>> marketplace give the current CI results, which was also requested in the >>> feedback from driver maintainers. >> Having the CI results reported would be an interesting experiment. I wonder if having the results even more publicly reported would result in more stable CI's. It is a dual edged sword however. Given the instability of many CI's it could make OpenStack look bad to customers who don't understand what they are looking at. Just my thoughts on that idea. > That?s very useful feedback. Having that kind of background upfront is really helpful. As we make updates on the display side, we can take into account if certain attributes are potentially unreliable or at a higher risk of showing instability and have the interface better support that without it looking like everything is failing and a river of red X?s. Are there other things that might be similar? > > Jonathan > Jonathan, Glad to be of assistance. I think reporting some percentage of success might be the most accurate way to report the CI results. Not necessarily flagging it good or bed but leave it for the consumers to see and compare. Also combine that with Anita's idea of when the CI last successfully reported and I think it could give a good barometer for consumers. Our systems all have their rough times so we need to avoid a 'snapshot in time' view and provide more of a 'activity over time' view. Third party CI is a good barometer of community activity and attention, but not always 100% accurate. Obviously there will need to be some information included with the results explaining what they are and helping guide interpretations. Jay > > __ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] Improving Vendor Driver Discoverability
On 01/16/2017 12:19 PM, Jonathan Bryce wrote: On Jan 16, 2017, at 11:58 AM, Jay S. Bryantwrote: On 01/13/2017 10:29 PM, Mike Perez wrote: The way validation works is completely up to the project team. In my research as shown in the Summit etherpad [5] there's a clear trend in projects doing continuous integration for validation. If we wanted to we could also have the marketplace give the current CI results, which was also requested in the feedback from driver maintainers. Having the CI results reported would be an interesting experiment. I wonder if having the results even more publicly reported would result in more stable CI's. It is a dual edged sword however. Given the instability of many CI's it could make OpenStack look bad to customers who don't understand what they are looking at. Just my thoughts on that idea. That’s very useful feedback. Having that kind of background upfront is really helpful. As we make updates on the display side, we can take into account if certain attributes are potentially unreliable or at a higher risk of showing instability and have the interface better support that without it looking like everything is failing and a river of red X’s. Are there other things that might be similar? Jonathan Jonathan, Glad to be of assistance. I think reporting some percentage of success might be the most accurate way to report the CI results. Not necessarily flagging it good or bed but leave it for the consumers to see and compare. Also combine that with Anita's idea of when the CI last successfully reported and I think it could give a good barometer for consumers. Our systems all have their rough times so we need to avoid a 'snapshot in time' view and provide more of a 'activity over time' view. Third party CI is a good barometer of community activity and attention, but not always 100% accurate. Obviously there will need to be some information included with the results explaining what they are and helping guide interpretations. Jay __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] Improving Vendor Driver Discoverability
On 2017-01-16 01:19 PM, Jonathan Bryce wrote: On Jan 16, 2017, at 11:58 AM, Jay S. Bryantwrote: On 01/13/2017 10:29 PM, Mike Perez wrote: The way validation works is completely up to the project team. In my research as shown in the Summit etherpad [5] there's a clear trend in projects doing continuous integration for validation. If we wanted to we could also have the marketplace give the current CI results, which was also requested in the feedback from driver maintainers. Having the CI results reported would be an interesting experiment. I wonder if having the results even more publicly reported would result in more stable CI's. It is a dual edged sword however. Given the instability of many CI's it could make OpenStack look bad to customers who don't understand what they are looking at. Just my thoughts on that idea. That’s very useful feedback. Having that kind of background upfront is really helpful. As we make updates on the display side, we can take into account if certain attributes are potentially unreliable or at a higher risk of showing instability and have the interface better support that without it looking like everything is failing and a river of red X’s. You could show the timestamp since the last passing test, rather than pass or fail as well as how long the driver has been tested. If a driver has been tested for 2 years or longer and has gone a week since the last passing test chances are the team is working on a bug, either with the driver code or the ci system (this can be explained on the page in a legend of some sort). This gives the reader more context with which to evaluate comparable drivers regarding their elapsed time since last successful completion of their ci as well as how long their ci has been active. This might be a more useful and consumable approach for the audience which might have little understanding of continuous integration, its meaning and its artifacts. Thanks, Anita. Are there other things that might be similar? Jonathan __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] Improving Vendor Driver Discoverability
> On Jan 16, 2017, at 11:58 AM, Jay S. Bryant> wrote: > > On 01/13/2017 10:29 PM, Mike Perez wrote: >> The way validation works is completely up to the project team. In my research >> as shown in the Summit etherpad [5] there's a clear trend in projects doing >> continuous integration for validation. If we wanted to we could also have the >> marketplace give the current CI results, which was also requested in the >> feedback from driver maintainers. > Having the CI results reported would be an interesting experiment. I wonder > if having the results even more publicly reported would result in more stable > CI's. It is a dual edged sword however. Given the instability of many CI's > it could make OpenStack look bad to customers who don't understand what they > are looking at. Just my thoughts on that idea. That’s very useful feedback. Having that kind of background upfront is really helpful. As we make updates on the display side, we can take into account if certain attributes are potentially unreliable or at a higher risk of showing instability and have the interface better support that without it looking like everything is failing and a river of red X’s. Are there other things that might be similar? Jonathan __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] Improving Vendor Driver Discoverability
On 01/13/2017 10:29 PM, Mike Perez wrote: Hello all, In the spirit of recent Technical Committee discussions I would like to bring focus on how we're doing vendor driver discoverability. Today we're doing this with the OpenStack Foundation marketplace [1] which is powered by the driverlog project. In a nutshell, it is a big JSON file [2] that has information on which vendor solutions are supported by which projects in which releases. This information is then parsed to generate the marketplace so that users can discover them. As discussed in previous TC meetings [3] we need to recognize vendors that are trying to make great products work in OpenStack so that they can be successful, which allows our community to be successful and healthy. In the feedback I have received from various people in the community, some didn’t know how it worked, and were unhappy that the projects themselves weren’t owning this. I totally agree that project teams should own this and should be encouraged to be involved in the reviews. Today that’s not happening. I’d like to propose we come up with a way for the marketplace to be more community-driven by the projects that are validating these solutions. At the Barcelona Summit [4] we discussed ways to improve driverlog. Projects like Nova have a support matrix of hypervisors in their in-tree documentation. Various members of the Cinder project also expressed interest in using this solution. It was suggested in the session that the marketplace should just link to the projects' appropriate documentation. The problem with this solution is the information is not presented in a consistent way across projects, as driverlog does it today. We could accomplish this instead by using a parsable format that is stored in each appropriate project's git repository. I'm thinking of pretty much how driverlog works today, but broken up into individual projects. The marketplace can parse this information and present it in one place consistently. Projects may also continue to parse this information in their own documentation, and we can even write a common tool to do this. The way a vendor is listed here is based on being validated by the project team itself. Keeping things in the marketplace would also address the suggestions that came out of the recent feedback we received from various driver maintainers [4]. The way validation works is completely up to the project team. In my research as shown in the Summit etherpad [5] there's a clear trend in projects doing continuous integration for validation. If we wanted to we could also have the marketplace give the current CI results, which was also requested in the feedback from driver maintainers. Having the CI results reported would be an interesting experiment. I wonder if having the results even more publicly reported would result in more stable CI's. It is a dual edged sword however. Given the instability of many CI's it could make OpenStack look bad to customers who don't understand what they are looking at. Just my thoughts on that idea. I would like to volunteer in creating the initial files for each project with what the marketplace says today. [1] - https://www.openstack.org/marketplace/drivers/ [2] - http://git.openstack.org/cgit/openstack/driverlog/tree/etc/default_data.json [3] - http://eavesdrop.openstack.org/meetings/tc/2017/tc.2017-01-10-20.01.log.html#l-106 [4] - http://lists.openstack.org/pipermail/openstack-dev/2017-January/109855.html [5] - https://etherpad.openstack.org/p/driverlog-validation __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [all] Improving Vendor Driver Discoverability
Hello all, In the spirit of recent Technical Committee discussions I would like to bring focus on how we're doing vendor driver discoverability. Today we're doing this with the OpenStack Foundation marketplace [1] which is powered by the driverlog project. In a nutshell, it is a big JSON file [2] that has information on which vendor solutions are supported by which projects in which releases. This information is then parsed to generate the marketplace so that users can discover them. As discussed in previous TC meetings [3] we need to recognize vendors that are trying to make great products work in OpenStack so that they can be successful, which allows our community to be successful and healthy. In the feedback I have received from various people in the community, some didn’t know how it worked, and were unhappy that the projects themselves weren’t owning this. I totally agree that project teams should own this and should be encouraged to be involved in the reviews. Today that’s not happening. I’d like to propose we come up with a way for the marketplace to be more community-driven by the projects that are validating these solutions. At the Barcelona Summit [4] we discussed ways to improve driverlog. Projects like Nova have a support matrix of hypervisors in their in-tree documentation. Various members of the Cinder project also expressed interest in using this solution. It was suggested in the session that the marketplace should just link to the projects' appropriate documentation. The problem with this solution is the information is not presented in a consistent way across projects, as driverlog does it today. We could accomplish this instead by using a parsable format that is stored in each appropriate project's git repository. I'm thinking of pretty much how driverlog works today, but broken up into individual projects. The marketplace can parse this information and present it in one place consistently. Projects may also continue to parse this information in their own documentation, and we can even write a common tool to do this. The way a vendor is listed here is based on being validated by the project team itself. Keeping things in the marketplace would also address the suggestions that came out of the recent feedback we received from various driver maintainers [4]. The way validation works is completely up to the project team. In my research as shown in the Summit etherpad [5] there's a clear trend in projects doing continuous integration for validation. If we wanted to we could also have the marketplace give the current CI results, which was also requested in the feedback from driver maintainers. I would like to volunteer in creating the initial files for each project with what the marketplace says today. [1] - https://www.openstack.org/marketplace/drivers/ [2] - http://git.openstack.org/cgit/openstack/driverlog/tree/etc/default_data.json [3] - http://eavesdrop.openstack.org/meetings/tc/2017/tc.2017-01-10-20.01.log.html#l-106 [4] - http://lists.openstack.org/pipermail/openstack-dev/2017-January/109855.html [5] - https://etherpad.openstack.org/p/driverlog-validation -- Mike Perez __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev