Re: [openstack-dev] [infra] [neutron] [tc] Neutron Incubator workflow
On Thu, Aug 28, 2014 at 12:22 PM, Richard Woo richardwoo2...@gmail.com wrote: I have another question about incubator proposal, for CLI and GUI. Do we imply that the incubator feature will need to branch python-neutron client, Horizon, and or Nova ( if changes are needed)? This is as good a place for the docs perspective as any on this thread. I think it's great to increase your dev velocity, but in the proposal I need to see a well-thought-out plan for when documentation should be on docs.openstack.org for deployers and users and when documentation should be in the API reference page at http://developer.openstack.org/api-ref.html. A section addressing the timing and requirements would be sufficient. The docs affected are: CLI Reference http://docs.openstack.org/cli-reference/content/ Config Reference http://docs.openstack.org/icehouse/config-reference/content/ User Guide http://docs.openstack.org/user-guide/content/ Admin User Guide http://docs.openstack.org/user-guide-admin/content/ Cloud Admin User Guide http://docs.openstack.org/admin-guide-cloud/content/ API Reference http://developer.openstack.org/api-ref.html I won't argue that the Install Guide should be included as these items aren't exactly happy path quite yet. Also in the wiki page, I see a line saying link to commits in private trees is acceptable -- is it really? How would readers get to it? Thanks, Anne On Tue, Aug 26, 2014 at 7:09 PM, James E. Blair cor...@inaugust.com wrote: Hi, After reading https://wiki.openstack.org/wiki/Network/Incubator I have some thoughts about the proposed workflow. We have quite a bit of experience and some good tools around splitting code out of projects and into new projects. But we don't generally do a lot of importing code into projects. We've done this once, to my recollection, in a way that preserved history, and that was with the switch to keystone-lite. It wasn't easy; it's major git surgery and would require significant infra-team involvement any time we wanted to do it. However, reading the proposal, it occurred to me that it's pretty clear that we expect these tools to be able to operate outside of the Neutron project itself, to even be releasable on their own. Why not just stick with that? In other words, the goal of this process should be to create separate projects with their own development lifecycle that will continue indefinitely, rather than expecting the code itself to merge into the neutron repo. This has advantages in simplifying workflow and making it more consistent. Plus it builds on known integration mechanisms like APIs and python project versions. But more importantly, it helps scale the neutron project itself. I think that a focused neutron core upon which projects like these can build on in a reliable fashion would be ideal. -Jim ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [infra] [neutron] [tc] Neutron Incubator workflow
On 08/27/2014 04:28 PM, Kevin Benton wrote: What are you talking about? The only reply was from me clarifying that one of the purposes of the incubator was for components of neutron that are experimental but are intended to be merged. Right. The special unicorns. In that case it might not make sense to have a life cycle of their own in another repo indefinitely. The main reasons these experimental components don't make sense to live in their own repo indefinitely are: a) Neutron's design doesn't make it easy or straightforward to build/layer other things on top of it, or: b) The experimental piece of code intends to replace whole-hog a large chunk of Neutron's existing codebase, or: c) The experimental piece of code relies so heavily on inconsistent, unversioned internal interface and plugin calls that it cannot be designed externally due to the fragility of those interfaces Fixing a) is the solution to these problems. An incubator area where experimental components can live will just continue to mask the true problem domain, which is that Neutron's design is cumbersome to build on top of, and its cross-component interfaces need to be versioned, made consistent, and cleaned up to use versioned data structures instead of passing random nested dicts of randomly-prefixed string key/values. Frankly, we're going through a similar problem in Nova right now. There is a group of folks who believe that separating the nova-scheduler code into the Gantt project will magically make placement decision code and solver components *easier* to work on (because the pace of coding can be increased if there wasn't that pesky nova-core review process). But this is not correct, IMO. Separating out the scheduler into its own project before internal interfaces and data structures are cleaned up and versioned will just lead to greater technical debt and an increase in frustration on the part of Nova developers and scheduler developers alike. -jay On Wed, Aug 27, 2014 at 11:52 AM, Jay Pipes jaypi...@gmail.com mailto:jaypi...@gmail.com wrote: On 08/26/2014 07:09 PM, James E. Blair wrote: Hi, After reading https://wiki.openstack.org/__wiki/Network/Incubator https://wiki.openstack.org/wiki/Network/Incubator I have some thoughts about the proposed workflow. We have quite a bit of experience and some good tools around splitting code out of projects and into new projects. But we don't generally do a lot of importing code into projects. We've done this once, to my recollection, in a way that preserved history, and that was with the switch to keystone-lite. It wasn't easy; it's major git surgery and would require significant infra-team involvement any time we wanted to do it. However, reading the proposal, it occurred to me that it's pretty clear that we expect these tools to be able to operate outside of the Neutron project itself, to even be releasable on their own. Why not just stick with that? In other words, the goal of this process should be to create separate projects with their own development lifecycle that will continue indefinitely, rather than expecting the code itself to merge into the neutron repo. This has advantages in simplifying workflow and making it more consistent. Plus it builds on known integration mechanisms like APIs and python project versions. But more importantly, it helps scale the neutron project itself. I think that a focused neutron core upon which projects like these can build on in a reliable fashion would be ideal. Despite replies to you saying that certain branches of Neutron development work are special unicorns, I wanted to say I *fully* support your above statement. Best, -jay _ OpenStack-dev mailing list OpenStack-dev@lists.openstack.__org mailto:OpenStack-dev@lists.openstack.org http://lists.openstack.org/__cgi-bin/mailman/listinfo/__openstack-dev http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- Kevin Benton ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [infra] [neutron] [tc] Neutron Incubator workflow
Right. The special unicorns. Repeating this without defining it isn't helping anything. b) The experimental piece of code intends to replace whole-hog a large chunk of Neutron's existing codebase, or: In the DVR example I gave this is is the only relevant reason. Regardless of how well the internal Neutron APIs are defined, this same problem would have arisen. DVR completely changed the reference L3 service plugin, which lives in the main tree. A well-defined, versioned internal API would not have helped any of the issues I brought up. The 'experimental' part of the DVR work isn't referring to its internal API modifications, its referring to how the L3 service plugin will integrate with the data plane. On Thu, Aug 28, 2014 at 7:45 AM, Jay Pipes jaypi...@gmail.com wrote: On 08/27/2014 04:28 PM, Kevin Benton wrote: What are you talking about? The only reply was from me clarifying that one of the purposes of the incubator was for components of neutron that are experimental but are intended to be merged. Right. The special unicorns. In that case it might not make sense to have a life cycle of their own in another repo indefinitely. The main reasons these experimental components don't make sense to live in their own repo indefinitely are: a) Neutron's design doesn't make it easy or straightforward to build/layer other things on top of it, or: b) The experimental piece of code intends to replace whole-hog a large chunk of Neutron's existing codebase, or: c) The experimental piece of code relies so heavily on inconsistent, unversioned internal interface and plugin calls that it cannot be designed externally due to the fragility of those interfaces Fixing a) is the solution to these problems. An incubator area where experimental components can live will just continue to mask the true problem domain, which is that Neutron's design is cumbersome to build on top of, and its cross-component interfaces need to be versioned, made consistent, and cleaned up to use versioned data structures instead of passing random nested dicts of randomly-prefixed string key/values. Frankly, we're going through a similar problem in Nova right now. There is a group of folks who believe that separating the nova-scheduler code into the Gantt project will magically make placement decision code and solver components *easier* to work on (because the pace of coding can be increased if there wasn't that pesky nova-core review process). But this is not correct, IMO. Separating out the scheduler into its own project before internal interfaces and data structures are cleaned up and versioned will just lead to greater technical debt and an increase in frustration on the part of Nova developers and scheduler developers alike. -jay On Wed, Aug 27, 2014 at 11:52 AM, Jay Pipes jaypi...@gmail.com mailto:jaypi...@gmail.com wrote: On 08/26/2014 07:09 PM, James E. Blair wrote: Hi, After reading https://wiki.openstack.org/__wiki/Network/Incubator https://wiki.openstack.org/wiki/Network/Incubator I have some thoughts about the proposed workflow. We have quite a bit of experience and some good tools around splitting code out of projects and into new projects. But we don't generally do a lot of importing code into projects. We've done this once, to my recollection, in a way that preserved history, and that was with the switch to keystone-lite. It wasn't easy; it's major git surgery and would require significant infra-team involvement any time we wanted to do it. However, reading the proposal, it occurred to me that it's pretty clear that we expect these tools to be able to operate outside of the Neutron project itself, to even be releasable on their own. Why not just stick with that? In other words, the goal of this process should be to create separate projects with their own development lifecycle that will continue indefinitely, rather than expecting the code itself to merge into the neutron repo. This has advantages in simplifying workflow and making it more consistent. Plus it builds on known integration mechanisms like APIs and python project versions. But more importantly, it helps scale the neutron project itself. I think that a focused neutron core upon which projects like these can build on in a reliable fashion would be ideal. Despite replies to you saying that certain branches of Neutron development work are special unicorns, I wanted to say I *fully* support your above statement. Best, -jay _ OpenStack-dev mailing list OpenStack-dev@lists.openstack.__org
Re: [openstack-dev] [infra] [neutron] [tc] Neutron Incubator workflow
On Aug 28, 2014, at 10:45 AM, Jay Pipes jaypi...@gmail.com wrote: On 08/27/2014 04:28 PM, Kevin Benton wrote: What are you talking about? The only reply was from me clarifying that one of the purposes of the incubator was for components of neutron that are experimental but are intended to be merged. Right. The special unicorns. In that case it might not make sense to have a life cycle of their own in another repo indefinitely. The main reasons these experimental components don't make sense to live in their own repo indefinitely are: a) Neutron's design doesn't make it easy or straightforward to build/layer other things on top of it, or: Correct and this something I want the team to address in Kilo. Many of the L7 services would be easier to build if we invest some time early in the cycle to establishing a well defined interface for a few items. I’m sure the LBaaS team has good feedback to share with everyone. b) The experimental piece of code intends to replace whole-hog a large chunk of Neutron's existing codebase, or: c) The experimental piece of code relies so heavily on inconsistent, unversioned internal interface and plugin calls that it cannot be designed externally due to the fragility of those interfaces I’m glad Jim reminded us of the pain of merging histories and the availability of feature branches. I think for items where we’re replacing large chunks of code feature branches make lots of sense. Fixing a) is the solution to these problems. An incubator area where experimental components can live will just continue to mask the true problem domain, which is that Neutron's design is cumbersome to build on top of, and its cross-component interfaces need to be versioned, made consistent, and cleaned up to use versioned data structures instead of passing random nested dicts of randomly-prefixed string key/values. Frankly, we're going through a similar problem in Nova right now. There is a group of folks who believe that separating the nova-scheduler code into the Gantt project will magically make placement decision code and solver components *easier* to work on (because the pace of coding can be increased if there wasn't that pesky nova-core review process). But this is not correct, IMO. Separating out the scheduler into its own project before internal interfaces and data structures are cleaned up and versioned will just lead to greater technical debt and an increase in frustration on the part of Nova developers and scheduler developers alike. Right hopefully the incubator will allow us to develop components that should be independent from the start without incurring too much debt. -jay On Wed, Aug 27, 2014 at 11:52 AM, Jay Pipes jaypi...@gmail.com mailto:jaypi...@gmail.com wrote: On 08/26/2014 07:09 PM, James E. Blair wrote: Hi, After reading https://wiki.openstack.org/__wiki/Network/Incubator https://wiki.openstack.org/wiki/Network/Incubator I have some thoughts about the proposed workflow. We have quite a bit of experience and some good tools around splitting code out of projects and into new projects. But we don't generally do a lot of importing code into projects. We've done this once, to my recollection, in a way that preserved history, and that was with the switch to keystone-lite. It wasn't easy; it's major git surgery and would require significant infra-team involvement any time we wanted to do it. However, reading the proposal, it occurred to me that it's pretty clear that we expect these tools to be able to operate outside of the Neutron project itself, to even be releasable on their own. Why not just stick with that? In other words, the goal of this process should be to create separate projects with their own development lifecycle that will continue indefinitely, rather than expecting the code itself to merge into the neutron repo. This has advantages in simplifying workflow and making it more consistent. Plus it builds on known integration mechanisms like APIs and python project versions. But more importantly, it helps scale the neutron project itself. I think that a focused neutron core upon which projects like these can build on in a reliable fashion would be ideal. Despite replies to you saying that certain branches of Neutron development work are special unicorns, I wanted to say I *fully* support your above statement. Best, -jay _ OpenStack-dev mailing list OpenStack-dev@lists.openstack.__org mailto:OpenStack-dev@lists.openstack.org http://lists.openstack.org/__cgi-bin/mailman/listinfo/__openstack-dev
Re: [openstack-dev] [infra] [neutron] [tc] Neutron Incubator workflow
I have another question about incubator proposal, for CLI and GUI. Do we imply that the incubator feature will need to branch python-neutron client, Horizon, and or Nova ( if changes are needed)? On Tue, Aug 26, 2014 at 7:09 PM, James E. Blair cor...@inaugust.com wrote: Hi, After reading https://wiki.openstack.org/wiki/Network/Incubator I have some thoughts about the proposed workflow. We have quite a bit of experience and some good tools around splitting code out of projects and into new projects. But we don't generally do a lot of importing code into projects. We've done this once, to my recollection, in a way that preserved history, and that was with the switch to keystone-lite. It wasn't easy; it's major git surgery and would require significant infra-team involvement any time we wanted to do it. However, reading the proposal, it occurred to me that it's pretty clear that we expect these tools to be able to operate outside of the Neutron project itself, to even be releasable on their own. Why not just stick with that? In other words, the goal of this process should be to create separate projects with their own development lifecycle that will continue indefinitely, rather than expecting the code itself to merge into the neutron repo. This has advantages in simplifying workflow and making it more consistent. Plus it builds on known integration mechanisms like APIs and python project versions. But more importantly, it helps scale the neutron project itself. I think that a focused neutron core upon which projects like these can build on in a reliable fashion would be ideal. -Jim ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [infra] [neutron] [tc] Neutron Incubator workflow
On 2014-08-28 08:31:26 -0700 (-0700), Kevin Benton wrote: [...] DVR completely changed the reference L3 service plugin, which lives in the main tree. A well-defined, versioned internal API would not have helped any of the issues I brought up. [...] Except, perhaps, insofar as that (in some ideal world) it might allow the reference L3 service plugin to be extracted from the main tree and developed within a separate source code repository with its own life cycle. -- Jeremy Stanley ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [infra] [neutron] [tc] Neutron Incubator workflow
Yes, in theory all of the plugins should be removable from the core neutron repo. So then it would only need to be responsible for the APIs, db models, etc. However, IIRC there are no plans to move any reference plugins from the tree. On Thu, Aug 28, 2014 at 11:20 AM, Jeremy Stanley fu...@yuggoth.org wrote: On 2014-08-28 08:31:26 -0700 (-0700), Kevin Benton wrote: [...] DVR completely changed the reference L3 service plugin, which lives in the main tree. A well-defined, versioned internal API would not have helped any of the issues I brought up. [...] Except, perhaps, insofar as that (in some ideal world) it might allow the reference L3 service plugin to be extracted from the main tree and developed within a separate source code repository with its own life cycle. -- Jeremy Stanley ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- Kevin Benton ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [infra] [neutron] [tc] Neutron Incubator workflow
On Wed, Aug 27, 2014 at 2:44 PM, Kevin Benton blak...@gmail.com wrote: Incubator doesn't mean being kicked out of tree, it just mean that the API and resource model needs to be baked for fast iteration, and can't be put in tree temporarily. That was exactly my point about developing a major feature like DVR. Even with a limited API change (the new distributed flag), it has an impact on the the router/agent DB resource model and currently isn't compatible with VLAN based ML2 deployments. It's not exactly a hidden optimization like an improvement to some RPC handling code. Flag is only for admin use, tenant can't see it, and the default policy for router is setup by config file. It COULD be compatible for DVR work on vlan, but there were some bugs several months before, that DVR mac are not written successfully on the egress packet, I'm not sure if it is fixed in the merged code. A huge piece of the DVR development had to happen in Neutron forks and 40+ revision chains of Gerrit patches. It was very difficult to follow without being heavily involved with the L3 team. This would have been a great candidate to develop in the incubator if it existed at the time. It would have been easier to try as it was developed and to explore the entire codebase. Also, more people could have been contributing bug fixes and improvements since an entire section of code wouldn't be 'owned' by one person like it is with the author of a Gerrit review. For DVR, as it has no influence on tenant facing API resource model, it works as the built-in backend, and this feature has accepted wide common interests, As was pointed out before, common interest has nothing to do with incubation. Incubation is to rapidly iterate on a new feature for Neutron. It shouldn't be restricted to API changes, it should be used for any major new features that are possible to develop outside of the Neutron core. If we are going to have this new incubator tool, we should use it to the fullest extent possible. On Tue, Aug 26, 2014 at 6:19 PM, loy wolfe loywo...@gmail.com wrote: Incubator doesn't mean being kicked out of tree, it just mean that the API and resource model needs to be baked for fast iteration, and can't be put in tree temporarily. As kyle has said, incubator is not talking about moving 3rd drivers out of tree, which is in another thread. For DVR, as it has no influence on tenant facing API resource model, it works as the built-in backend, and this feature has accepted wide common interests, it's just the internal performance optimization tightly coupled with existing code, so it should be developed in tree. On Wed, Aug 27, 2014 at 8:08 AM, Kevin Benton blak...@gmail.com wrote: From what I understand, the intended projects for the incubator can't operate without neutron because they are just extensions/plugins/drivers. For example, if the DVR modifications to the reference reference L3 plugin weren't already being developed in the tree, DVR could have been developed in the incubator and then merged into Neutron once the bugs were ironed out so a huge string of Gerrit patches didn't need to be tracked. If that had happened, would it make sense to keep the L3 plugin as a completely separate project or merge it? I understand this is the approach the load balancer folks took by making Octavia a separate project, but I think it can still operate on its own, where the reference L3 plugin (and many of the other incubator projects) are just classes that expect to be able to make core Neutron calls. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- Kevin Benton ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [infra] [neutron] [tc] Neutron Incubator workflow
Flag is only for admin use, tenant can't see it, and the default policy for router is setup by config file. It's still a public API that will have to follow a deprecation cycle. If a new API was going to be introduced for admins to control the distributed nature of routers, it would have been nice to introduce a little more control than distributed=True/False (e.g. how many SNAT IPs are consumed, etc). At this point we are stuck because any new API entries would require a blueprint and the feature proposal freeze deadline is long gone. It COULD be compatible for DVR work on vlan, but there were some bugs several months before, that DVR mac are not written successfully on the egress packet, I'm not sure if it is fixed in the merged code. The current DVR wiki[1] still shows that the tenant_network_type has to be VXLAN. I didn't know about this issue until just recently and I'm not sure if there are plans yet to fix it for Juno. If it were in an incubation tree somewhere and not on Gerrit for the majority of the cycle, the bug could have been worked on in parallel by other volunteers interested in VLAN support. Now that DVR is solidifying, VLAN support may even require a blueprint unless it's blessed by the right cores, which would mean people with VLAN deployments would not be able to use DVR until Kilo is released next next May. The whole point is that there is nowhere to work on big features like this in a fast, iterative, and open manner. The incubator could be a perfect place for this type of work. 1. https://wiki.openstack.org/wiki/Neutron/DVR/HowTo On Wed, Aug 27, 2014 at 1:22 AM, loy wolfe loywo...@gmail.com wrote: On Wed, Aug 27, 2014 at 2:44 PM, Kevin Benton blak...@gmail.com wrote: Incubator doesn't mean being kicked out of tree, it just mean that the API and resource model needs to be baked for fast iteration, and can't be put in tree temporarily. That was exactly my point about developing a major feature like DVR. Even with a limited API change (the new distributed flag), it has an impact on the the router/agent DB resource model and currently isn't compatible with VLAN based ML2 deployments. It's not exactly a hidden optimization like an improvement to some RPC handling code. Flag is only for admin use, tenant can't see it, and the default policy for router is setup by config file. It COULD be compatible for DVR work on vlan, but there were some bugs several months before, that DVR mac are not written successfully on the egress packet, I'm not sure if it is fixed in the merged code. A huge piece of the DVR development had to happen in Neutron forks and 40+ revision chains of Gerrit patches. It was very difficult to follow without being heavily involved with the L3 team. This would have been a great candidate to develop in the incubator if it existed at the time. It would have been easier to try as it was developed and to explore the entire codebase. Also, more people could have been contributing bug fixes and improvements since an entire section of code wouldn't be 'owned' by one person like it is with the author of a Gerrit review. For DVR, as it has no influence on tenant facing API resource model, it works as the built-in backend, and this feature has accepted wide common interests, As was pointed out before, common interest has nothing to do with incubation. Incubation is to rapidly iterate on a new feature for Neutron. It shouldn't be restricted to API changes, it should be used for any major new features that are possible to develop outside of the Neutron core. If we are going to have this new incubator tool, we should use it to the fullest extent possible. On Tue, Aug 26, 2014 at 6:19 PM, loy wolfe loywo...@gmail.com wrote: Incubator doesn't mean being kicked out of tree, it just mean that the API and resource model needs to be baked for fast iteration, and can't be put in tree temporarily. As kyle has said, incubator is not talking about moving 3rd drivers out of tree, which is in another thread. For DVR, as it has no influence on tenant facing API resource model, it works as the built-in backend, and this feature has accepted wide common interests, it's just the internal performance optimization tightly coupled with existing code, so it should be developed in tree. On Wed, Aug 27, 2014 at 8:08 AM, Kevin Benton blak...@gmail.com wrote: From what I understand, the intended projects for the incubator can't operate without neutron because they are just extensions/plugins/drivers. For example, if the DVR modifications to the reference reference L3 plugin weren't already being developed in the tree, DVR could have been developed in the incubator and then merged into Neutron once the bugs were ironed out so a huge string of Gerrit patches didn't need to be tracked. If that had happened, would it make sense to keep the L3 plugin as a completely separate project or merge it? I understand this is the approach the
Re: [openstack-dev] [infra] [neutron] [tc] Neutron Incubator workflow
On 08/26/2014 07:09 PM, James E. Blair wrote: Hi, After reading https://wiki.openstack.org/wiki/Network/Incubator I have some thoughts about the proposed workflow. We have quite a bit of experience and some good tools around splitting code out of projects and into new projects. But we don't generally do a lot of importing code into projects. We've done this once, to my recollection, in a way that preserved history, and that was with the switch to keystone-lite. It wasn't easy; it's major git surgery and would require significant infra-team involvement any time we wanted to do it. However, reading the proposal, it occurred to me that it's pretty clear that we expect these tools to be able to operate outside of the Neutron project itself, to even be releasable on their own. Why not just stick with that? In other words, the goal of this process should be to create separate projects with their own development lifecycle that will continue indefinitely, rather than expecting the code itself to merge into the neutron repo. This has advantages in simplifying workflow and making it more consistent. Plus it builds on known integration mechanisms like APIs and python project versions. But more importantly, it helps scale the neutron project itself. I think that a focused neutron core upon which projects like these can build on in a reliable fashion would be ideal. Despite replies to you saying that certain branches of Neutron development work are special unicorns, I wanted to say I *fully* support your above statement. Best, -jay ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [infra] [neutron] [tc] Neutron Incubator workflow
What are you talking about? The only reply was from me clarifying that one of the purposes of the incubator was for components of neutron that are experimental but are intended to be merged. In that case it might not make sense to have a life cycle of their own in another repo indefinitely. On Wed, Aug 27, 2014 at 11:52 AM, Jay Pipes jaypi...@gmail.com wrote: On 08/26/2014 07:09 PM, James E. Blair wrote: Hi, After reading https://wiki.openstack.org/wiki/Network/Incubator I have some thoughts about the proposed workflow. We have quite a bit of experience and some good tools around splitting code out of projects and into new projects. But we don't generally do a lot of importing code into projects. We've done this once, to my recollection, in a way that preserved history, and that was with the switch to keystone-lite. It wasn't easy; it's major git surgery and would require significant infra-team involvement any time we wanted to do it. However, reading the proposal, it occurred to me that it's pretty clear that we expect these tools to be able to operate outside of the Neutron project itself, to even be releasable on their own. Why not just stick with that? In other words, the goal of this process should be to create separate projects with their own development lifecycle that will continue indefinitely, rather than expecting the code itself to merge into the neutron repo. This has advantages in simplifying workflow and making it more consistent. Plus it builds on known integration mechanisms like APIs and python project versions. But more importantly, it helps scale the neutron project itself. I think that a focused neutron core upon which projects like these can build on in a reliable fashion would be ideal. Despite replies to you saying that certain branches of Neutron development work are special unicorns, I wanted to say I *fully* support your above statement. Best, -jay ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- Kevin Benton ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [infra] [neutron] [tc] Neutron Incubator workflow
I agree that components isolated to one service or something along those lines where there is a clear plugin point in Neutron, it might make sense to separate them permanently. However, at that point why even bother with the Neutron incubator when a new project can be started? The feature branch idea sounds interesting for the one-time big experimental changes. Are there any other projects that do this right now? On Wed, Aug 27, 2014 at 12:30 PM, James E. Blair cor...@inaugust.com wrote: Kevin Benton blak...@gmail.com writes: From what I understand, the intended projects for the incubator can't operate without neutron because they are just extensions/plugins/drivers. I could have phrased that better. What I meant was that they could operate without being actually in the Neutron repo, not that they could not operate without Neutron itself. The proposal for the incubator is that extensions be developed outside of the Neutron repo. My proposed refinement is that they stay outside of the Neutron repo. They live their entire lives as extension modules in separate projects. For example, if the DVR modifications to the reference reference L3 plugin weren't already being developed in the tree, DVR could have been developed in the incubator and then merged into Neutron once the bugs were ironed out so a huge string of Gerrit patches didn't need to be tracked. If that had happened, would it make sense to keep the L3 plugin as a completely separate project or merge it? I understand this is the approach the load balancer folks took by making Octavia a separate project, but I think it can still operate on its own, where the reference L3 plugin (and many of the other incubator projects) are just classes that expect to be able to make core Neutron calls. The list of Juno/Kilo candidates doesn't seem to have projects that are quite so low-level. If a feature is going to become part of the neutron core, then it should be developed in the neutron repository. If we need a place to land code that isn't master, it's actually far easier to just use a feature branch on the neutron repo. Commits can land there as needed, master can be periodically merged into it, and when the feature is ready, the feature branch can be merged into master. I think between those two options: incubate/spin-out components that are high-level enough not to have deep integration in the neutron core, and using feature branches for large experimental changes to the core itself, we can handle the problems the incubator repo is intended to address. -Jim ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- Kevin Benton ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [infra] [neutron] [tc] Neutron Incubator workflow
Quoting Stefano from a different thread [0]: The rationale for the separate repository is that Neutron's code needs a lot of love for the *existing* codebase, before new features can be added (and before core team can accept more responsibilities for it). But progress is unstoppable: new features are being proposed every cycle and reviewers bandwidth is not infinite. The long term purpose of the incubator should be to gather all the big new features that may require fast iteration before becoming stable and being eventually merged into the main tree. This is especially important for core new features, that may have a higher impact on Neutron's stability. A great example was made by Kevin with DVR. [0] http://lists.openstack.org/pipermail/openstack-dev/2014-August/044224.html On Wed, Aug 27, 2014 at 1:32 PM, Kevin Benton blak...@gmail.com wrote: I agree that components isolated to one service or something along those lines where there is a clear plugin point in Neutron, it might make sense to separate them permanently. However, at that point why even bother with the Neutron incubator when a new project can be started? The feature branch idea sounds interesting for the one-time big experimental changes. Are there any other projects that do this right now? On Wed, Aug 27, 2014 at 12:30 PM, James E. Blair cor...@inaugust.com wrote: Kevin Benton blak...@gmail.com writes: From what I understand, the intended projects for the incubator can't operate without neutron because they are just extensions/plugins/drivers. I could have phrased that better. What I meant was that they could operate without being actually in the Neutron repo, not that they could not operate without Neutron itself. The proposal for the incubator is that extensions be developed outside of the Neutron repo. My proposed refinement is that they stay outside of the Neutron repo. They live their entire lives as extension modules in separate projects. For example, if the DVR modifications to the reference reference L3 plugin weren't already being developed in the tree, DVR could have been developed in the incubator and then merged into Neutron once the bugs were ironed out so a huge string of Gerrit patches didn't need to be tracked. If that had happened, would it make sense to keep the L3 plugin as a completely separate project or merge it? I understand this is the approach the load balancer folks took by making Octavia a separate project, but I think it can still operate on its own, where the reference L3 plugin (and many of the other incubator projects) are just classes that expect to be able to make core Neutron calls. The list of Juno/Kilo candidates doesn't seem to have projects that are quite so low-level. If a feature is going to become part of the neutron core, then it should be developed in the neutron repository. If we need a place to land code that isn't master, it's actually far easier to just use a feature branch on the neutron repo. Commits can land there as needed, master can be periodically merged into it, and when the feature is ready, the feature branch can be merged into master. I think between those two options: incubate/spin-out components that are high-level enough not to have deep integration in the neutron core, and using feature branches for large experimental changes to the core itself, we can handle the problems the incubator repo is intended to address. -Jim ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- Kevin Benton ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [infra] [neutron] [tc] Neutron Incubator workflow
On 2014-08-27 16:05:36 -0700 (-0700), Joe Gordon wrote: We have feature branches? How do we use them? I am not even sure if feature branches were evaluated as an option when the incubator was proposed. Keystone's new API used a feature branch, the EC work in Swift did, so did the Gearman integration for Zuul... however it's also an option not to be taken lightly and comes with its own set of unique challenges. -- Jeremy Stanley ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [infra] [neutron] [tc] Neutron Incubator workflow
Also note that one of the features of the incubator is that it can be packaged/released independently (like python-neutronclient) and a feature branch is not designed for that workflow. Regards, Mandeep On Wed, Aug 27, 2014 at 4:55 PM, Jeremy Stanley fu...@yuggoth.org wrote: On 2014-08-27 16:05:36 -0700 (-0700), Joe Gordon wrote: We have feature branches? How do we use them? I am not even sure if feature branches were evaluated as an option when the incubator was proposed. Keystone's new API used a feature branch, the EC work in Swift did, so did the Gearman integration for Zuul... however it's also an option not to be taken lightly and comes with its own set of unique challenges. -- Jeremy Stanley ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [infra] [neutron] [tc] Neutron Incubator workflow
On 2014-08-27 16:53:39 -0700 (-0700), Clark Boylan wrote: [...] I thought there was a wiki article on how they work but I can't find it. Maybe someone else can link it here. [...] https://wiki.openstack.org/wiki/GerritJenkinsGit#Merge_Commits -- Jeremy Stanley ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [infra] [neutron] [tc] Neutron Incubator workflow
On 2014-08-27 16:59:54 -0700 (-0700), Mandeep Dhami wrote: Also note that one of the features of the incubator is that it can be packaged/released independently (like python-neutronclient) and a feature branch is not designed for that workflow. Good point, and that definitely is a reason to just keep those pieces in their own separate Git repositories outside of the core Neutron repository in perpetuity (even after they graduate from incubation). One package per repository. That should be chiseled in stone somewhere. -- Jeremy Stanley ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [infra] [neutron] [tc] Neutron Incubator workflow
On Wed, Aug 27, 2014 at 6:59 PM, Mandeep Dhami dh...@noironetworks.com wrote: Also note that one of the features of the incubator is that it can be packaged/released independently (like python-neutronclient) and a feature branch is not designed for that workflow. Which is a good reason to not use the word 'incubator'. It also has the connotations from Oslo of being 'copy-pasta' code. dt -- Dean Troyer dtro...@gmail.com ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [infra] [neutron] [tc] Neutron Incubator workflow
On Wed, Aug 27, 2014 at 5:06 PM, Jeremy Stanley fu...@yuggoth.org wrote: Good point, and that definitely is a reason to just keep those pieces in their own separate Git repositories outside of the core Neutron repository in perpetuity (even after they graduate from incubation). One package per repository. That should be chiseled in stone somewhere. Just asking, wasn't there a proposal about doing something similar for all the neutron services? (l3/fw/lb/vnp ...) Ivar. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [infra] [neutron] [tc] Neutron Incubator workflow
Hi, After reading https://wiki.openstack.org/wiki/Network/Incubator I have some thoughts about the proposed workflow. We have quite a bit of experience and some good tools around splitting code out of projects and into new projects. But we don't generally do a lot of importing code into projects. We've done this once, to my recollection, in a way that preserved history, and that was with the switch to keystone-lite. It wasn't easy; it's major git surgery and would require significant infra-team involvement any time we wanted to do it. However, reading the proposal, it occurred to me that it's pretty clear that we expect these tools to be able to operate outside of the Neutron project itself, to even be releasable on their own. Why not just stick with that? In other words, the goal of this process should be to create separate projects with their own development lifecycle that will continue indefinitely, rather than expecting the code itself to merge into the neutron repo. This has advantages in simplifying workflow and making it more consistent. Plus it builds on known integration mechanisms like APIs and python project versions. But more importantly, it helps scale the neutron project itself. I think that a focused neutron core upon which projects like these can build on in a reliable fashion would be ideal. -Jim ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [infra] [neutron] [tc] Neutron Incubator workflow
From what I understand, the intended projects for the incubator can't operate without neutron because they are just extensions/plugins/drivers. For example, if the DVR modifications to the reference reference L3 plugin weren't already being developed in the tree, DVR could have been developed in the incubator and then merged into Neutron once the bugs were ironed out so a huge string of Gerrit patches didn't need to be tracked. If that had happened, would it make sense to keep the L3 plugin as a completely separate project or merge it? I understand this is the approach the load balancer folks took by making Octavia a separate project, but I think it can still operate on its own, where the reference L3 plugin (and many of the other incubator projects) are just classes that expect to be able to make core Neutron calls. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [infra] [neutron] [tc] Neutron Incubator workflow
Incubator doesn't mean being kicked out of tree, it just mean that the API and resource model needs to be baked for fast iteration, and can't be put in tree temporarily. As kyle has said, incubator is not talking about moving 3rd drivers out of tree, which is in another thread. For DVR, as it has no influence on tenant facing API resource model, it works as the built-in backend, and this feature has accepted wide common interests, it's just the internal performance optimization tightly coupled with existing code, so it should be developed in tree. On Wed, Aug 27, 2014 at 8:08 AM, Kevin Benton blak...@gmail.com wrote: From what I understand, the intended projects for the incubator can't operate without neutron because they are just extensions/plugins/drivers. For example, if the DVR modifications to the reference reference L3 plugin weren't already being developed in the tree, DVR could have been developed in the incubator and then merged into Neutron once the bugs were ironed out so a huge string of Gerrit patches didn't need to be tracked. If that had happened, would it make sense to keep the L3 plugin as a completely separate project or merge it? I understand this is the approach the load balancer folks took by making Octavia a separate project, but I think it can still operate on its own, where the reference L3 plugin (and many of the other incubator projects) are just classes that expect to be able to make core Neutron calls. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
