Re: [openstack-dev] [fuel] [QA] running Fuel tests using nodepool

2016-05-10 Thread Spencer Krum 
As a frequent tinc user I'd be interested to see the code you are using
to manage tinc into doing this. Is that code available somewhere?

On Tue, May 10, 2016, at 09:02 AM, Monty Taylor wrote:
> On 05/10/2016 08:54 AM, Vladimir Eremin wrote:
> > Hi,
> > 
> > I've investigated status of nodepool multi node testing and fuel-qa
> > approaches, and I wanna share my opinion on moving Fuel testing on
> > OpenStack and nodepool.
> 
> Awesome! This is a great writeup - and hopefully will be useful as we
> validate our theory that zuul v3 should provide a richer environment for
> doing complex things like fuel testing than the current multi-node work.
> 
> > Our CI pipeline consists of next stages:
> > 
> > 1. Artifact building and publishing
> > 2. QA jobs:
> > 2.1. Master node installation from ISO
> > 2.2. Slave nodes provisioning
> > 2.3. Software deployment
> > 2.4. Workload verification
> > 
> > Current upstream nodepool limitations are pre-spwaned nodes, small
> > flavors and, only l3 connectivity. Also, we have no PXE booting and VLAN
> > trunking in OpenStack itself. So, the main problem with moving this
> > pipeline to nodepool is to emulate IT tasks: installation from ISO and
> > nodes provisioning.
> > 
> > Actually the point is: to test Fuel and test rest of OpenStack
> > components against Fuel we mostly need to test stage artifact building,
> > deployment and verification. So we need to make Fuel installable from
> > packages and create overlay L2 networking. I've found no unsolvable
> > problems right now to check most of scenarios with this approach.
> 
> 
> 
> > Besides artifact building step, there are next actions items to do to
> > run Fuel QA test:
> > 
> > 1. Automate overlay networking setup. I've
> > used https://www.tinc-vpn.org/ as a L2 switching overlay, but OpenVPN
> > could be tool of choice. Action items:
> >  - overlay networking setup should be integrated in fuel-devops
> 
> There is overlay work in the multi-node stuff for devstack. I believe
> clarkb has a todo-list item to make that networking setup more general
> and more generally available. (it's currently done in devstack-gate
> script) I'm not sure if you saw that or if it's suitable for what you
> need? If not, it would be good to understand deficiencies.
> 
> > 2. Automate Fuel master node codebase installation from packages,
> > including repo adding and deployment. Action items:
> > - installation should be integrated in fuel-devops or nodepool infra
> > - make bootstrap scripts working with more than one network on master
> > node ("Bringing down ALL network interfaces except...")
> > - fix iptables and ssh for underlay networking
> 
> We've talked a few times about handling packages and repos of packages
> for patches that have not yet landed, but have done exactly zero work on
> it. Since you're a concrete use case, perhaps we can design things with
> you in mind.
> 
> > 3. Automate Fuel slave node codebase installation and node enrollment.
> > Action items:
> > - nailgun-agent installation should be integrated in fuel-devops or
> > nodepool infra
> > - mcollective and ssh keys setup should be automated
> > - nailgun ang/or astute should be extended to allow pre-provisioned
> > nodes enrollment (I'm doing this part now)
> > - nailgun-agent and l23network should support overlay network interfaces
> 
> Exciting. I look forward to working on this with you - there are fun
> problems in here. :)
> 
> 
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe:
> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


-- 
  Spencer Krum
  n...@spencerkrum.com

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [fuel] [QA] running Fuel tests using nodepool

2016-05-10 Thread Clark Boylan 
On Tue, May 10, 2016, at 07:50 AM, Vladimir Eremin wrote:
> Hi Jeremy,
> 
> Yep, I saw it. Unfortunately, because Fuel deployment scenarios is about
> setting up OVS too, it could be kinda freaky to provide overlay
> networking for OVS on OVS. That's why I was looking on other L2 overlays
> (kernel space mcast vxlan was in scope too).

Neutron uses OVS + vxlan too and we very explicitly do not nest the
tunnels. Neutron is thus able to happily make its internal VM network
overlays as it would normally over the actual VM networks. Basically
this is a non issue. You only make overlays for the non managed
networking, things like floating IP networks and so on.

Clark

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [fuel] [QA] running Fuel tests using nodepool

2016-05-10 Thread Monty Taylor 
On 05/10/2016 08:54 AM, Vladimir Eremin wrote:
> Hi,
> 
> I've investigated status of nodepool multi node testing and fuel-qa
> approaches, and I wanna share my opinion on moving Fuel testing on
> OpenStack and nodepool.

Awesome! This is a great writeup - and hopefully will be useful as we
validate our theory that zuul v3 should provide a richer environment for
doing complex things like fuel testing than the current multi-node work.

> Our CI pipeline consists of next stages:
> 
> 1. Artifact building and publishing
> 2. QA jobs:
> 2.1. Master node installation from ISO
> 2.2. Slave nodes provisioning
> 2.3. Software deployment
> 2.4. Workload verification
> 
> Current upstream nodepool limitations are pre-spwaned nodes, small
> flavors and, only l3 connectivity. Also, we have no PXE booting and VLAN
> trunking in OpenStack itself. So, the main problem with moving this
> pipeline to nodepool is to emulate IT tasks: installation from ISO and
> nodes provisioning.
> 
> Actually the point is: to test Fuel and test rest of OpenStack
> components against Fuel we mostly need to test stage artifact building,
> deployment and verification. So we need to make Fuel installable from
> packages and create overlay L2 networking. I've found no unsolvable
> problems right now to check most of scenarios with this approach.



> Besides artifact building step, there are next actions items to do to
> run Fuel QA test:
> 
> 1. Automate overlay networking setup. I've
> used https://www.tinc-vpn.org/ as a L2 switching overlay, but OpenVPN
> could be tool of choice. Action items:
>  - overlay networking setup should be integrated in fuel-devops

There is overlay work in the multi-node stuff for devstack. I believe
clarkb has a todo-list item to make that networking setup more general
and more generally available. (it's currently done in devstack-gate
script) I'm not sure if you saw that or if it's suitable for what you
need? If not, it would be good to understand deficiencies.

> 2. Automate Fuel master node codebase installation from packages,
> including repo adding and deployment. Action items:
> - installation should be integrated in fuel-devops or nodepool infra
> - make bootstrap scripts working with more than one network on master
> node ("Bringing down ALL network interfaces except...")
> - fix iptables and ssh for underlay networking

We've talked a few times about handling packages and repos of packages
for patches that have not yet landed, but have done exactly zero work on
it. Since you're a concrete use case, perhaps we can design things with
you in mind.

> 3. Automate Fuel slave node codebase installation and node enrollment.
> Action items:
> - nailgun-agent installation should be integrated in fuel-devops or
> nodepool infra
> - mcollective and ssh keys setup should be automated
> - nailgun ang/or astute should be extended to allow pre-provisioned
> nodes enrollment (I'm doing this part now)
> - nailgun-agent and l23network should support overlay network interfaces

Exciting. I look forward to working on this with you - there are fun
problems in here. :)


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [fuel] [QA] running Fuel tests using nodepool

2016-05-10 Thread Vladimir Eremin 
Hi Jeremy,

Yep, I saw it. Unfortunately, because Fuel deployment scenarios is about 
setting up OVS too, it could be kinda freaky to provide overlay networking for 
OVS on OVS. That's why I was looking on other L2 overlays (kernel space mcast 
vxlan was in scope too).

But yes, we still can use this method (with l23network or multinode scripts for 
some cases). 

-- 
With best regards,
Vladimir Eremin,
Fuel Deployment Engineer,
Mirantis, Inc.



> On May 10, 2016, at 5:39 PM, Jeremy Stanley  wrote:
> 
> On 2016-05-10 15:54:34 +0300 (+0300), Vladimir Eremin wrote:
> [...]
>> 1. Automate overlay networking setup. I've used
>> https://www.tinc-vpn.org/  as a L2
>> switching overlay, but OpenVPN could be tool of choice. Action
>> items:
>>- overlay networking setup should be integrated in fuel-devops
> [...]
> 
> Just to be sure, you've seen the ovs_vxlan_bridge() implementation
> in devstack-gate where we set up an overlay L2 network using
> OVS/VXLAN? The same design also works fine with GRE (we used it for
> a while but ran into some service providers blocking IP protocol 47
> on their LANs).
> 
> http://git.openstack.org/cgit/openstack-infra/devstack-gate/tree/multinode_setup_info.txt
> http://git.openstack.org/cgit/openstack-infra/devstack-gate/tree/functions.sh#n1050
> -- 
> Jeremy Stanley
> 
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [fuel] [QA] running Fuel tests using nodepool

2016-05-10 Thread Jeremy Stanley 
On 2016-05-10 15:54:34 +0300 (+0300), Vladimir Eremin wrote:
[...]
> 1. Automate overlay networking setup. I've used
> https://www.tinc-vpn.org/  as a L2
> switching overlay, but OpenVPN could be tool of choice. Action
> items:
> - overlay networking setup should be integrated in fuel-devops
[...]

Just to be sure, you've seen the ovs_vxlan_bridge() implementation
in devstack-gate where we set up an overlay L2 network using
OVS/VXLAN? The same design also works fine with GRE (we used it for
a while but ran into some service providers blocking IP protocol 47
on their LANs).

http://git.openstack.org/cgit/openstack-infra/devstack-gate/tree/multinode_setup_info.txt
http://git.openstack.org/cgit/openstack-infra/devstack-gate/tree/functions.sh#n1050
-- 
Jeremy Stanley

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [fuel] [QA] running Fuel tests using nodepool

2016-05-10 Thread Vladimir Eremin 
Hi,

I've investigated status of nodepool multi node testing and fuel-qa approaches, 
and I wanna share my opinion on moving Fuel testing on OpenStack and nodepool.

Our CI pipeline consists of next stages:

1. Artifact building and publishing
2. QA jobs:
2.1. Master node installation from ISO
2.2. Slave nodes provisioning
2.3. Software deployment
2.4. Workload verification

Current upstream nodepool limitations are pre-spwaned nodes, small flavors and, 
only l3 connectivity. Also, we have no PXE booting and VLAN trunking in 
OpenStack itself. So, the main problem with moving this pipeline to nodepool is 
to emulate IT tasks: installation from ISO and nodes provisioning.

Actually the point is: to test Fuel and test rest of OpenStack components 
against Fuel we mostly need to test stage artifact building, deployment and 
verification. So we need to make Fuel installable from packages and create 
overlay L2 networking. I've found no unsolvable problems right now to check 
most of scenarios with this approach.

Besides artifact building step, there are next actions items to do to run Fuel 
QA test:

1. Automate overlay networking setup. I've used https://www.tinc-vpn.org/ 
 as a L2 switching overlay, but OpenVPN could be 
tool of choice. Action items:
 - overlay networking setup should be integrated in fuel-devops

2. Automate Fuel master node codebase installation from packages, including 
repo adding and deployment. Action items:
- installation should be integrated in fuel-devops or nodepool infra
- make bootstrap scripts working with more than one network on master 
node ("Bringing down ALL network interfaces except...")
- fix iptables and ssh for underlay networking

3. Automate Fuel slave node codebase installation and node enrollment. Action 
items:
- nailgun-agent installation should be integrated in fuel-devops or 
nodepool infra
- mcollective and ssh keys setup should be automated
- nailgun ang/or astute should be extended to allow pre-provisioned 
nodes enrollment (I'm doing this part now)
- nailgun-agent and l23network should support overlay network interfaces


-- 
With best regards,
Vladimir Eremin,
Fuel Deployment Engineer,
Mirantis, Inc.



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev