Re: [sfc-dev] [integration-dev] sfc csit failure

[Thanh Ha]

On Wed, Jan 24, 2018 at 12:16 PM, Jaime Caamaño Ruiz 
wrote:

>
> Thanh, issue confirmed fixed!
>
> > I can confirm that at least in my test environment I had to set this
> > parameter.
>
> Confirmed on my side too.
>
> BR
> Jaime


Awesome, that's great to hear!

Cheers,
Thanh
___
sfc-dev mailing list
sfc-dev@lists.opendaylight.org
https://lists.opendaylight.org/mailman/listinfo/sfc-dev

Re: [sfc-dev] [integration-dev] sfc csit failure

[Thanh Ha]

Ah that one should be fine then. As long as it has ZZCI prefix.

Thanh


On Wed, Jan 24, 2018 at 8:00 AM, Jaime Caamaño Ruiz 
wrote:

> Hello
>
> Thats great, thanks for the support! I will check it out and let you
> know.
>
> > For the record we used "ZZCI - CentOS 7 - docker - 20180110-1659" to
> > test. I'm not sure which docker image you're using but you might want
> > to switch to using one of the ZZCI ones moving forward.
>
> We are using the releng defaults:
>
> jjb/releng-defaults.yaml:docker_system_image: ZZCI - CentOS 7 -
> docker - 20180109-0346
>
> I don't know what's the change between the two but maybe we should step
> this one.
>
> BR
> Jaime.
>
> -Original Message-
> From: Thanh Ha 
> To: jcaam...@suse.de
> Cc: integration-...@lists.opendaylight.org  aylight.org>, sfc-dev@lists.opendaylight.org  t.org>
> Subject: Re: [integration-dev] [sfc-dev] sfc csit failure
> Date: Mon, 22 Jan 2018 15:04:59 -0500
>
> On Mon, Jan 22, 2018 at 5:14 AM, Jaime Caamaño Ruiz 
> wrote:
> > > - clean ip tables in Bsudo iptables -Fsudo iptables
> > > -t nat -FIn my testing cleaning the iptables in this way breaks the
> > > docker container's ability to route to the outside world so I would
> > > not recommend cleaning the iptables as this will make the container
> > > not be able to reach anything at all.
> >
> > A requirement of our test setup is that the docker container can
> > reach
> > the builder with no nat'ing in between. By default docker network
> > does
> > masquerading to the outside world and cleaning ip tables gets rid of
> > it. Is a rather rudimentary way of achieving it but worked for us.
> > The
> > proper way to do it is by creating a separate docker network with the
> > option com.docker.network.bridge.enable_ip_masquerade disabled.
> >
> > Disable masquerading also requires adding a route in the builder back
> > to the docker network. Otherwise, as you say, there wont be
> > connectivity:
> >
> > route add -net  netmask  gw  > host ip>
> >
> > Worth mentioning is that this whole setup worked for us pre cloud
> > provider change.
> >
>
>
> After working with our cloud provider this morning we were able to
> resolve this with this patch https://git.opendaylight.org/gerrit/67436
> to allow the docker IPs to be route-able rather than dropped by the
> network.
>
> This should allow your csit tests to communicate between hosts and
> containers now but let us know if there continues to be issues. For the
> record we used "ZZCI - CentOS 7 - docker - 20180110-1659" to test. I'm
> not sure which docker image you're using but you might want to switch
> to using one of the ZZCI ones moving forward.
>
> I can also confirm that even calling the 2 iptables -F commands won't
> break the routing with the patch above so you should be able to
> continue with your existing tests that do this.
>
>
> > > We should be setting iptables -P FORWARD ACCEPT to allow another
> > host
> > > to reach inside the docker container on the docker host however
> > even
> > > after setting this things are still unrouteable.
> >
> > This is one more thing that docker did by default but no longer does.
> > I
> > will crosscheck if it affects us but I dont think so. Our specific
> > problem is packets going out of the container towards the builder,
> > being forwarded in the docker host, but getting lost somewhere
> > between
> > the docker host and the builder.
> >
>
> I can confirm that at least in my test environment I had to set this
> parameter.
>
> Regards,
> Thanh
>
>
>
___
sfc-dev mailing list
sfc-dev@lists.opendaylight.org
https://lists.opendaylight.org/mailman/listinfo/sfc-dev

Re: [sfc-dev] [integration-dev] sfc csit failure

[Jaime Caamaño Ruiz]

Hello

Thats great, thanks for the support! I will check it out and let you
know.

> For the record we used "ZZCI - CentOS 7 - docker - 20180110-1659" to
> test. I'm not sure which docker image you're using but you might want
> to switch to using one of the ZZCI ones moving forward.

We are using the releng defaults:

jjb/releng-defaults.yaml:docker_system_image: ZZCI - CentOS 7 - docker - 
20180109-0346

I don't know what's the change between the two but maybe we should step
this one.

BR
Jaime.

-Original Message-
From: Thanh Ha 
To: jcaam...@suse.de
Cc: integration-...@lists.opendaylight.org , sfc-dev@lists.opendaylight.org 
Subject: Re: [integration-dev] [sfc-dev] sfc csit failure
Date: Mon, 22 Jan 2018 15:04:59 -0500

On Mon, Jan 22, 2018 at 5:14 AM, Jaime Caamaño Ruiz 
wrote:
> > - clean ip tables in Bsudo iptables -Fsudo iptables
> > -t nat -FIn my testing cleaning the iptables in this way breaks the
> > docker container's ability to route to the outside world so I would
> > not recommend cleaning the iptables as this will make the container
> > not be able to reach anything at all.
> 
> A requirement of our test setup is that the docker container can
> reach
> the builder with no nat'ing in between. By default docker network
> does
> masquerading to the outside world and cleaning ip tables gets rid of
> it. Is a rather rudimentary way of achieving it but worked for us.
> The
> proper way to do it is by creating a separate docker network with the
> option com.docker.network.bridge.enable_ip_masquerade disabled.
> 
> Disable masquerading also requires adding a route in the builder back
> to the docker network. Otherwise, as you say, there wont be
> connectivity:
> 
> route add -net  netmask  gw  host ip>
> 
> Worth mentioning is that this whole setup worked for us pre cloud
> provider change.
> 


After working with our cloud provider this morning we were able to
resolve this with this patch https://git.opendaylight.org/gerrit/67436
to allow the docker IPs to be route-able rather than dropped by the
network.

This should allow your csit tests to communicate between hosts and
containers now but let us know if there continues to be issues. For the
record we used "ZZCI - CentOS 7 - docker - 20180110-1659" to test. I'm
not sure which docker image you're using but you might want to switch
to using one of the ZZCI ones moving forward.

I can also confirm that even calling the 2 iptables -F commands won't
break the routing with the patch above so you should be able to
continue with your existing tests that do this.


> > We should be setting iptables -P FORWARD ACCEPT to allow another
> host
> > to reach inside the docker container on the docker host however
> even
> > after setting this things are still unrouteable.
> 
> This is one more thing that docker did by default but no longer does.
> I
> will crosscheck if it affects us but I dont think so. Our specific
> problem is packets going out of the container towards the builder,
> being forwarded in the docker host, but getting lost somewhere
> between
> the docker host and the builder.
> 

I can confirm that at least in my test environment I had to set this
parameter.

Regards,
Thanh

 
___
sfc-dev mailing list
sfc-dev@lists.opendaylight.org
https://lists.opendaylight.org/mailman/listinfo/sfc-dev

Re: [sfc-dev] [integration-dev] sfc csit failure

[Thanh Ha]

On Mon, Jan 22, 2018 at 5:14 AM, Jaime Caamaño Ruiz 
wrote:

> > - clean ip tables in Bsudo iptables -Fsudo iptables
> > -t nat -FIn my testing cleaning the iptables in this way breaks the
> > docker container's ability to route to the outside world so I would
> > not recommend cleaning the iptables as this will make the container
> > not be able to reach anything at all.
>
> A requirement of our test setup is that the docker container can reach
> the builder with no nat'ing in between. By default docker network does
> masquerading to the outside world and cleaning ip tables gets rid of
> it. Is a rather rudimentary way of achieving it but worked for us. The
> proper way to do it is by creating a separate docker network with the
> option com.docker.network.bridge.enable_ip_masquerade disabled.
>
> Disable masquerading also requires adding a route in the builder back
> to the docker network. Otherwise, as you say, there wont be
> connectivity:
>
> route add -net  netmask  gw  ip>
>
> Worth mentioning is that this whole setup worked for us pre cloud
> provider change.
>


After working with our cloud provider this morning we were able to resolve
this with this patch https://git.opendaylight.org/gerrit/67436 to allow the
docker IPs to be route-able rather than dropped by the network.

This should allow your csit tests to communicate between hosts and
containers now but let us know if there continues to be issues. For the
record we used "ZZCI - CentOS 7 - docker - 20180110-1659" to test. I'm not
sure which docker image you're using but you might want to switch to using
one of the ZZCI ones moving forward.

I can also confirm that even calling the 2 iptables -F commands won't break
the routing with the patch above so you should be able to continue with
your existing tests that do this.


> We should be setting iptables -P FORWARD ACCEPT to allow another host
> > to reach inside the docker container on the docker host however even
> > after setting this things are still unrouteable.
>
> This is one more thing that docker did by default but no longer does. I
> will crosscheck if it affects us but I dont think so. Our specific
> problem is packets going out of the container towards the builder,
> being forwarded in the docker host, but getting lost somewhere between
> the docker host and the builder.
>

I can confirm that at least in my test environment I had to set this
parameter.

Regards,
Thanh
___
sfc-dev mailing list
sfc-dev@lists.opendaylight.org
https://lists.opendaylight.org/mailman/listinfo/sfc-dev

Re: [sfc-dev] [integration-dev] sfc csit failure

[Jaime Caamaño Ruiz]

I took another look into this in oxygen.

The test that fails uses docker to setup multiple OVS instances. All
these instances run on a docker bridge network and must have
connectivity to ODL each with distinct IP from that network. To achieve
this, masquerading is disabled for the docker network, ip forwarding is
enabled on the docker host and a route for this network via the docker
host is added to the odl host.

This setup worked fine before the cloud provider transition. But now
packets are dropped between the docker host and the odl host when the
source IP is that of the docker network. I have verified forwarding is
done correctly on the docker host. I have also verified that the docker
network is reachable from the odl host, so the problem is on the other
direction. 

I am out of ideas now but this might be out of our control and the test
invalid as is.

BR
Jaime.



On Fri, 2017-12-22 at 12:17 -0800, Jamo Luhrsen wrote:
> On 12/22/2017 07:45 AM, Sam Hague wrote:
> > 
> > 
> > On Fri, Dec 22, 2017 at 7:37 AM, Jaime Caamaño Ruiz  > de > wrote:
> > 
> > Hello Jamo
> > 
> > Took a quick look.
> > 
> > The problem seems to be that the odl-sfc-openflow-renderer
> > feature is
> > installed but the dependant bundles are not activated. This
> > feature is
> > in charge of writing the flows, and the test fails because
> > there are no
> > flows on the  switches.
> 
> yeah, I saw a bunch of ugly karaf.log messages as well. There is an
> NPE showing
> up from sfc code that may or may not be important. Check this log:
> 
> https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/sfc-csit-3
> node-docker-full-deploy-all-carbon/390/odl1_karaf.log.gz
> 
> > This is not the only feature with this problem, seems to me
> > than more
> > than one bundle is just not properly started. So something is
> > wrong in
> > the feature or blueprint handling. Unfortunately, older logs
> > are not
> > available to compare but there were no changes in Carbon since
> > the last
> > success job.
> 
> I tried a job with Carbon SR2 since that should probably have been
> from
> before when it was passing. It failed as well. The logs didn't save,
> but
> you can see in the console log that it's the same failure:
> 
> https://jenkins.opendaylight.org/releng/view/sfc/job/sfc-csit-3node-d
> ocker-full-deploy-all-carbon/392/console
> 
> Thanks,
> JamO
> 
> 
> > Oxygen job is also failing from Dec 9. On this case, seems that
> > there
> > is no connectivity between the docker network running in the
> > tools
> > system and ODL. Something that we fixed few days ago caused by
> > a new
> > tools docker image that did not have ip forward enabled. But
> > now I dont
> > know what is the problem. When did we effectively switch to new
> > cloud
> > provider?
> > 
> > the switch was last week so around the 15th. 
> > 
> > 
> > BR
> > Jaime.
> > 
> > 
> > On Thu, 2017-12-21 at 16:57 -0800, Jamo Luhrsen wrote:
> > > Hi SFC,
> > >
> > > I can't remember if there was ever any discussion on this one
> > yet,
> > > but there is a
> > > consistent failure it would be nice to see fixed. Or if there
> > is a
> > > low sev bug
> > > we can point to, I'll edit the test to list that bug in the
> > report.
> > >
> > > looks like there is something not showing up in an ovs docker
> > > containers flow
> > > table. Specifically, it's not finding "*actions=pop_nsh*"
> > >
> > > I'm vaguely remembering something regarding this, but can't
> > recall it
> > > exactly.
> > >
> > > here's an example:
> > > https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/sf
> > c-csit-3
> >  > -csit-3>
> > > node-docker-full-deploy-all-carbon/390/log.html.gz#s1-s1-s1
> > >
> > >
> > > Thanks,
> > > JamO
> > > ___
> > > sfc-dev mailing list
> > > sfc-dev@lists.opendaylight.org  > ight.org>
> > > https://lists.opendaylight.org/mailman/listinfo/sfc-dev  > s://lists.opendaylight.org/mailman/listinfo/sfc-dev>
> > >
> > ___
> > integration-dev mailing list
> > integration-...@lists.opendaylight.org  > lists.opendaylight.org>
> > https://lists.opendaylight.org/mailman/listinfo/integration-dev
> >  > v>
> > 
> > 
> 
> 
___
sfc-dev mailing list
sfc-dev@lists.opendaylight.org
https://lists.opendaylight.org/mailman/listinfo/sfc-dev

Re: [sfc-dev] [integration-dev] sfc csit failure

[Sam Hague]

On Fri, Dec 22, 2017 at 7:37 AM, Jaime Caamaño Ruiz 
wrote:

> Hello Jamo
>
> Took a quick look.
>
> The problem seems to be that the odl-sfc-openflow-renderer feature is
> installed but the dependant bundles are not activated. This feature is
> in charge of writing the flows, and the test fails because there are no
> flows on the  switches.
>
> This is not the only feature with this problem, seems to me than more
> than one bundle is just not properly started. So something is wrong in
> the feature or blueprint handling. Unfortunately, older logs are not
> available to compare but there were no changes in Carbon since the last
> success job.
>
> Oxygen job is also failing from Dec 9. On this case, seems that there
> is no connectivity between the docker network running in the tools
> system and ODL. Something that we fixed few days ago caused by a new
> tools docker image that did not have ip forward enabled. But now I dont
> know what is the problem. When did we effectively switch to new cloud
> provider?
>
the switch was last week so around the 15th.

>
> BR
> Jaime.
>
>
> On Thu, 2017-12-21 at 16:57 -0800, Jamo Luhrsen wrote:
> > Hi SFC,
> >
> > I can't remember if there was ever any discussion on this one yet,
> > but there is a
> > consistent failure it would be nice to see fixed. Or if there is a
> > low sev bug
> > we can point to, I'll edit the test to list that bug in the report.
> >
> > looks like there is something not showing up in an ovs docker
> > containers flow
> > table. Specifically, it's not finding "*actions=pop_nsh*"
> >
> > I'm vaguely remembering something regarding this, but can't recall it
> > exactly.
> >
> > here's an example:
> > https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/sfc-csit-3
> > node-docker-full-deploy-all-carbon/390/log.html.gz#s1-s1-s1
> >
> >
> > Thanks,
> > JamO
> > ___
> > sfc-dev mailing list
> > sfc-dev@lists.opendaylight.org
> > https://lists.opendaylight.org/mailman/listinfo/sfc-dev
> >
> ___
> integration-dev mailing list
> integration-...@lists.opendaylight.org
> https://lists.opendaylight.org/mailman/listinfo/integration-dev
>
___
sfc-dev mailing list
sfc-dev@lists.opendaylight.org
https://lists.opendaylight.org/mailman/listinfo/sfc-dev