Re: Docker/Libvirt networking issue / bug?

2016-10-17 Thread Neil Horman 
On Fri, Oct 14, 2016 at 03:00:58PM -0600, Nathanael D. Noblet wrote:
> On Thu, 2016-10-06 at 14:02 -0500, Dan Williams wrote:
> > 
> > Try running 'iptables-save' before you start docker, and then running
> > 'iptables-save' after.  Diff the results.  Did docker remove
> > anything?
> 
> Hello,
> 
>   So this seems to be the source of the problem but I'm a little out of
> my depth to all its doing.
> 
> So I've attached three files
> 
> [1] iptables.onBoot (which is iptables after a clean boot)
> [2] iptables.afterDockerService (which is iptables after systemctl
> start docker)
> [3] iptables.diff ( the difference between the two files where I've
> removed differences that don't matter like packet counts etc).
> 
> So this seems like docker doesn't play well with libvirtd? Should I be
> filing a bug on docker? Or is this just a mis-configuration on my part?
> I don't think I've changed either libvirtd/qemu or docker's default
> configuration. Other than my VMs all attach to bridge0 instead of using
> NAT.
> 
> I'll start looking up what the -m addrtype --dst-type LOCAL does and
> all the docker related rules that are added but I'm really not sure
> what's going on. Particularly since VMs that are running and network
> connected when before I run a docker container continue to be. Only VMs
> brought up after that aren't. Also at a minimum if I stop the docker
> service I would expect these rules to go away which they don't. For
> example after systemctl stop docker I still have docker0 bridge
> interface up and 
> 

Thats alot to look at, but I would wager you're correct, this rule:
-A PREROUTING -m addrtype --dst-type LOCAL -j DOCKER

Probably mucks up a good deal of other services.  This basically says that
regardless of egress interface, or source interface, if a packet is destined for
an ip address that is local to the system (e.g. a local bridge interface to
either a docker container of a libvirt guest), it should traverse the docker
rule chain.  Likely that rule chain is built with docker interfaces specified
and all others ignored, and the default rule is drop or some such there.  The
rule above should almost certainly be further constrained to only apply to
interfaces used by docker.

Neil

> [gnat@iridium ~]$ sudo iptables -L -n | grep DOCKER
> DOCKER-ISOLATION  all  --  0.0.0.0/00.0.0.0/0   
> DOCKER all  --  0.0.0.0/00.0.0.0/0   
> Chain DOCKER (1 references)
> Chain DOCKER-ISOLATION (1 references)
> 
> still shows the chains are in place...

> # Generated by iptables-save v1.6.0 on Fri Oct 14 14:49:48 2016
> *nat
> :PREROUTING ACCEPT [25:1604]
> :INPUT ACCEPT [0:0]
> :OUTPUT ACCEPT [126:9336]
> :POSTROUTING ACCEPT [126:9336]
> :DOCKER - [0:0]
> :OUTPUT_direct - [0:0]
> :POSTROUTING_ZONES - [0:0]
> :POSTROUTING_ZONES_SOURCE - [0:0]
> :POSTROUTING_direct - [0:0]
> :POST_FedoraWorkstation - [0:0]
> :POST_FedoraWorkstation_allow - [0:0]
> :POST_FedoraWorkstation_deny - [0:0]
> :POST_FedoraWorkstation_log - [0:0]
> :POST_dmz - [0:0]
> :POST_dmz_allow - [0:0]
> :POST_dmz_deny - [0:0]
> :POST_dmz_log - [0:0]
> :POST_trusted - [0:0]
> :POST_trusted_allow - [0:0]
> :POST_trusted_deny - [0:0]
> :POST_trusted_log - [0:0]
> :PREROUTING_ZONES - [0:0]
> :PREROUTING_ZONES_SOURCE - [0:0]
> :PREROUTING_direct - [0:0]
> :PRE_FedoraWorkstation - [0:0]
> :PRE_FedoraWorkstation_allow - [0:0]
> :PRE_FedoraWorkstation_deny - [0:0]
> :PRE_FedoraWorkstation_log - [0:0]
> :PRE_dmz - [0:0]
> :PRE_dmz_allow - [0:0]
> :PRE_dmz_deny - [0:0]
> :PRE_dmz_log - [0:0]
> :PRE_trusted - [0:0]
> :PRE_trusted_allow - [0:0]
> :PRE_trusted_deny - [0:0]
> :PRE_trusted_log - [0:0]
> -A PREROUTING -j PREROUTING_direct
> -A PREROUTING -j PREROUTING_ZONES_SOURCE
> -A PREROUTING -j PREROUTING_ZONES
> -A PREROUTING -m addrtype --dst-type LOCAL -j DOCKER
> -A OUTPUT -j OUTPUT_direct
> -A OUTPUT ! -d 127.0.0.0/8 -m addrtype --dst-type LOCAL -j DOCKER
> -A POSTROUTING -s 172.17.0.0/16 ! -o docker0 -j MASQUERADE
> -A POSTROUTING -s 192.168.121.0/24 -d 224.0.0.0/24 -j RETURN
> -A POSTROUTING -s 192.168.121.0/24 -d 255.255.255.255/32 -j RETURN
> -A POSTROUTING -s 192.168.121.0/24 ! -d 192.168.121.0/24 -p tcp -j MASQUERADE 
> --to-ports 1024-65535
> -A POSTROUTING -s 192.168.121.0/24 ! -d 192.168.121.0/24 -p udp -j MASQUERADE 
> --to-ports 1024-65535
> -A POSTROUTING -s 192.168.121.0/24 ! -d 192.168.121.0/24 -j MASQUERADE
> -A POSTROUTING -j POSTROUTING_direct
> -A POSTROUTING -j POSTROUTING_ZONES_SOURCE
> -A POSTROUTING -j POSTROUTING_ZONES
> -A DOCKER -i docker0 -j RETURN
> -A POSTROUTING_ZONES -o em1 -g POST_dmz
> -A POSTROUTING_ZONES -o bridge0 -g POST_dmz
> -A POSTROUTING_ZONES -o virbr0 -j POST_trusted
> -A POSTROUTING_ZONES -o virbr0-nic -j POST_trusted
> -A POSTROUTING_ZONES -g POST_FedoraWorkstation
> -A POSTROUTING_ZONES_SOURCE -d 192.168.121.0/24 -g POST_dmz
> -A POSTROUTING_ZONES_SOURCE -d 192.168.4.0/24 -g POST_dmz
> -A POST_FedoraWorkstation -j POST_FedoraWorkstation_log
> -A

Re: Docker/Libvirt networking issue / bug?

2016-10-14 Thread Nathanael D. Noblet 
On Thu, 2016-10-06 at 14:02 -0500, Dan Williams wrote:
> 
> Try running 'iptables-save' before you start docker, and then running
> 'iptables-save' after.  Diff the results.  Did docker remove
> anything?

Hello,

  So this seems to be the source of the problem but I'm a little out of
my depth to all its doing.

So I've attached three files

[1] iptables.onBoot (which is iptables after a clean boot)
[2] iptables.afterDockerService (which is iptables after systemctl
start docker)
[3] iptables.diff ( the difference between the two files where I've
removed differences that don't matter like packet counts etc).

So this seems like docker doesn't play well with libvirtd? Should I be
filing a bug on docker? Or is this just a mis-configuration on my part?
I don't think I've changed either libvirtd/qemu or docker's default
configuration. Other than my VMs all attach to bridge0 instead of using
NAT.

I'll start looking up what the -m addrtype --dst-type LOCAL does and
all the docker related rules that are added but I'm really not sure
what's going on. Particularly since VMs that are running and network
connected when before I run a docker container continue to be. Only VMs
brought up after that aren't. Also at a minimum if I stop the docker
service I would expect these rules to go away which they don't. For
example after systemctl stop docker I still have docker0 bridge
interface up and 

[gnat@iridium ~]$ sudo iptables -L -n | grep DOCKER
DOCKER-ISOLATION  all  --  0.0.0.0/00.0.0.0/0   
DOCKER all  --  0.0.0.0/00.0.0.0/0   
Chain DOCKER (1 references)
Chain DOCKER-ISOLATION (1 references)

still shows the chains are in place...# Generated by iptables-save v1.6.0 on Fri Oct 14 14:49:48 2016
*nat
:PREROUTING ACCEPT [25:1604]
:INPUT ACCEPT [0:0]
:OUTPUT ACCEPT [126:9336]
:POSTROUTING ACCEPT [126:9336]
:DOCKER - [0:0]
:OUTPUT_direct - [0:0]
:POSTROUTING_ZONES - [0:0]
:POSTROUTING_ZONES_SOURCE - [0:0]
:POSTROUTING_direct - [0:0]
:POST_FedoraWorkstation - [0:0]
:POST_FedoraWorkstation_allow - [0:0]
:POST_FedoraWorkstation_deny - [0:0]
:POST_FedoraWorkstation_log - [0:0]
:POST_dmz - [0:0]
:POST_dmz_allow - [0:0]
:POST_dmz_deny - [0:0]
:POST_dmz_log - [0:0]
:POST_trusted - [0:0]
:POST_trusted_allow - [0:0]
:POST_trusted_deny - [0:0]
:POST_trusted_log - [0:0]
:PREROUTING_ZONES - [0:0]
:PREROUTING_ZONES_SOURCE - [0:0]
:PREROUTING_direct - [0:0]
:PRE_FedoraWorkstation - [0:0]
:PRE_FedoraWorkstation_allow - [0:0]
:PRE_FedoraWorkstation_deny - [0:0]
:PRE_FedoraWorkstation_log - [0:0]
:PRE_dmz - [0:0]
:PRE_dmz_allow - [0:0]
:PRE_dmz_deny - [0:0]
:PRE_dmz_log - [0:0]
:PRE_trusted - [0:0]
:PRE_trusted_allow - [0:0]
:PRE_trusted_deny - [0:0]
:PRE_trusted_log - [0:0]
-A PREROUTING -j PREROUTING_direct
-A PREROUTING -j PREROUTING_ZONES_SOURCE
-A PREROUTING -j PREROUTING_ZONES
-A PREROUTING -m addrtype --dst-type LOCAL -j DOCKER
-A OUTPUT -j OUTPUT_direct
-A OUTPUT ! -d 127.0.0.0/8 -m addrtype --dst-type LOCAL -j DOCKER
-A POSTROUTING -s 172.17.0.0/16 ! -o docker0 -j MASQUERADE
-A POSTROUTING -s 192.168.121.0/24 -d 224.0.0.0/24 -j RETURN
-A POSTROUTING -s 192.168.121.0/24 -d 255.255.255.255/32 -j RETURN
-A POSTROUTING -s 192.168.121.0/24 ! -d 192.168.121.0/24 -p tcp -j MASQUERADE 
--to-ports 1024-65535
-A POSTROUTING -s 192.168.121.0/24 ! -d 192.168.121.0/24 -p udp -j MASQUERADE 
--to-ports 1024-65535
-A POSTROUTING -s 192.168.121.0/24 ! -d 192.168.121.0/24 -j MASQUERADE
-A POSTROUTING -j POSTROUTING_direct
-A POSTROUTING -j POSTROUTING_ZONES_SOURCE
-A POSTROUTING -j POSTROUTING_ZONES
-A DOCKER -i docker0 -j RETURN
-A POSTROUTING_ZONES -o em1 -g POST_dmz
-A POSTROUTING_ZONES -o bridge0 -g POST_dmz
-A POSTROUTING_ZONES -o virbr0 -j POST_trusted
-A POSTROUTING_ZONES -o virbr0-nic -j POST_trusted
-A POSTROUTING_ZONES -g POST_FedoraWorkstation
-A POSTROUTING_ZONES_SOURCE -d 192.168.121.0/24 -g POST_dmz
-A POSTROUTING_ZONES_SOURCE -d 192.168.4.0/24 -g POST_dmz
-A POST_FedoraWorkstation -j POST_FedoraWorkstation_log
-A POST_FedoraWorkstation -j POST_FedoraWorkstation_deny
-A POST_FedoraWorkstation -j POST_FedoraWorkstation_allow
-A POST_dmz -j POST_dmz_log
-A POST_dmz -j POST_dmz_deny
-A POST_dmz -j POST_dmz_allow
-A POST_trusted -j POST_trusted_log
-A POST_trusted -j POST_trusted_deny
-A POST_trusted -j POST_trusted_allow
-A PREROUTING_ZONES -i em1 -g PRE_dmz
-A PREROUTING_ZONES -i bridge0 -g PRE_dmz
-A PREROUTING_ZONES -i virbr0 -j PRE_trusted
-A PREROUTING_ZONES -i virbr0-nic -j PRE_trusted
-A PREROUTING_ZONES -g PRE_FedoraWorkstation
-A PREROUTING_ZONES_SOURCE -s 192.168.121.0/24 -g PRE_dmz
-A PREROUTING_ZONES_SOURCE -s 192.168.4.0/24 -g PRE_dmz
-A PRE_FedoraWorkstation -j PRE_FedoraWorkstation_log
-A PRE_FedoraWorkstation -j PRE_FedoraWorkstation_deny
-A PRE_FedoraWorkstation -j PRE_FedoraWorkstation_allow
-A PRE_dmz -j PRE_dmz_log
-A PRE_dmz -j PRE_dmz_deny
-A PRE_dmz -j PRE_dmz_allow
-A PRE_trusted -j PRE_trusted_log
-A PRE_trusted -j PRE_trusted_deny
-A PRE_trusted -j PRE_trusted_allow

Re: Docker/Libvirt networking issue / bug?

2016-10-07 Thread Neil Horman 
On Fri, Oct 07, 2016 at 12:51:59AM +0300, Catalin wrote:
> maybe I'm come with not right answer ,
> I think that depend by network configuration over internet and routing process
> over applications. so will not be any conflict. I'm right?
> > Both docker and libvirt use iptables to direct traffic in various ways.  
> > They may well be in conflict
> 
I think you're comment doesn't justify your conclusion.  You are correct in your
indication that iptables rules may be different based on network and routing
configuration, but just because they may be different based on configuration in
no way precludes the two systems (docker and libvirt) from being in conflict.

Neil

> 
> 2016-10-06 22:18 GMT+03:00 Neil Horman :
> > On Thu, Oct 06, 2016 at 02:02:09PM -0500, Dan Williams wrote:
> >> On Thu, 2016-10-06 at 12:39 -0600, Nathanael D. Noblet wrote:
> >> > On Thu, 2016-10-06 at 14:28 -0400, Neil Horman wrote:
> >> > >
> >> > > I rarely mess with docker, but I expect that the docker0 bridge has
> >> > > an ip
> >> > > address on it which may conflict with the one on libvirt
> >> > > bridge.  That is to
> >> > > say, if they are on the same subnet, and the route for the docker0
> >> > > bridge takes
> >> > > precidence, you will loose dhcp.  Check the docker bridge ip and
> >> > > remove it if
> >> > > you see one, I expect that will restore your functionality
> >> > >
> >> >
> >> > Unfortunately that isn't the issue as the docker bridge is 172... and
> >> > bridge0 is 192.168... so they don't conflict. Also docker0 doesn't
> >> > seem
> >> > to have any devices (meaning it brctl show has no interfaces attached
> >> > to it). Finally shutting down docker and removing docker0 doesn't
> >> > restore connectivity, only a reboot does. Not even restarting NM
> >> > fixes
> >> > the issue.
> >>
> >> Try running 'iptables-save' before you start docker, and then running
> >> 'iptables-save' after.  Diff the results.  Did docker remove anything?
> >>
> > +1, thats the next thing to look at.  Both docker and libvirt use iptables 
> > to
> > direct traffic in various ways.  They may well be in conflict
> > Neil
> >
> >> Dan
> >> ___
> >> devel mailing list -- devel@lists.fedoraproject.org
> >> To unsubscribe send an email to devel-le...@lists.fedoraproject.org
> > ___
> > devel mailing list -- devel@lists.fedoraproject.org
> > To unsubscribe send an email to devel-le...@lists.fedoraproject.org
> ___
> devel mailing list -- devel@lists.fedoraproject.org
> To unsubscribe send an email to devel-le...@lists.fedoraproject.org
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org

Re: Docker/Libvirt networking issue / bug?

2016-10-06 Thread Catalin 
maybe I'm come with not right answer ,
I think that depend by network configuration over internet and routing process
over applications. so will not be any conflict. I'm right?
> Both docker and libvirt use iptables to direct traffic in various ways.  They 
> may well be in conflict


2016-10-06 22:18 GMT+03:00 Neil Horman :
> On Thu, Oct 06, 2016 at 02:02:09PM -0500, Dan Williams wrote:
>> On Thu, 2016-10-06 at 12:39 -0600, Nathanael D. Noblet wrote:
>> > On Thu, 2016-10-06 at 14:28 -0400, Neil Horman wrote:
>> > >
>> > > I rarely mess with docker, but I expect that the docker0 bridge has
>> > > an ip
>> > > address on it which may conflict with the one on libvirt
>> > > bridge.  That is to
>> > > say, if they are on the same subnet, and the route for the docker0
>> > > bridge takes
>> > > precidence, you will loose dhcp.  Check the docker bridge ip and
>> > > remove it if
>> > > you see one, I expect that will restore your functionality
>> > >
>> >
>> > Unfortunately that isn't the issue as the docker bridge is 172... and
>> > bridge0 is 192.168... so they don't conflict. Also docker0 doesn't
>> > seem
>> > to have any devices (meaning it brctl show has no interfaces attached
>> > to it). Finally shutting down docker and removing docker0 doesn't
>> > restore connectivity, only a reboot does. Not even restarting NM
>> > fixes
>> > the issue.
>>
>> Try running 'iptables-save' before you start docker, and then running
>> 'iptables-save' after.  Diff the results.  Did docker remove anything?
>>
> +1, thats the next thing to look at.  Both docker and libvirt use iptables to
> direct traffic in various ways.  They may well be in conflict
> Neil
>
>> Dan
>> ___
>> devel mailing list -- devel@lists.fedoraproject.org
>> To unsubscribe send an email to devel-le...@lists.fedoraproject.org
> ___
> devel mailing list -- devel@lists.fedoraproject.org
> To unsubscribe send an email to devel-le...@lists.fedoraproject.org
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org

Re: Docker/Libvirt networking issue / bug?

2016-10-06 Thread Neil Horman 
On Thu, Oct 06, 2016 at 02:02:09PM -0500, Dan Williams wrote:
> On Thu, 2016-10-06 at 12:39 -0600, Nathanael D. Noblet wrote:
> > On Thu, 2016-10-06 at 14:28 -0400, Neil Horman wrote:
> > > 
> > > I rarely mess with docker, but I expect that the docker0 bridge has
> > > an ip
> > > address on it which may conflict with the one on libvirt
> > > bridge.  That is to
> > > say, if they are on the same subnet, and the route for the docker0
> > > bridge takes
> > > precidence, you will loose dhcp.  Check the docker bridge ip and
> > > remove it if
> > > you see one, I expect that will restore your functionality
> > > 
> > 
> > Unfortunately that isn't the issue as the docker bridge is 172... and
> > bridge0 is 192.168... so they don't conflict. Also docker0 doesn't
> > seem
> > to have any devices (meaning it brctl show has no interfaces attached
> > to it). Finally shutting down docker and removing docker0 doesn't
> > restore connectivity, only a reboot does. Not even restarting NM
> > fixes
> > the issue.
> 
> Try running 'iptables-save' before you start docker, and then running
> 'iptables-save' after.  Diff the results.  Did docker remove anything?
> 
+1, thats the next thing to look at.  Both docker and libvirt use iptables to
direct traffic in various ways.  They may well be in conflict
Neil

> Dan
> ___
> devel mailing list -- devel@lists.fedoraproject.org
> To unsubscribe send an email to devel-le...@lists.fedoraproject.org
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org

Re: Docker/Libvirt networking issue / bug?

2016-10-06 Thread Dan Williams 
On Thu, 2016-10-06 at 12:39 -0600, Nathanael D. Noblet wrote:
> On Thu, 2016-10-06 at 14:28 -0400, Neil Horman wrote:
> > 
> > I rarely mess with docker, but I expect that the docker0 bridge has
> > an ip
> > address on it which may conflict with the one on libvirt
> > bridge.  That is to
> > say, if they are on the same subnet, and the route for the docker0
> > bridge takes
> > precidence, you will loose dhcp.  Check the docker bridge ip and
> > remove it if
> > you see one, I expect that will restore your functionality
> > 
> 
> Unfortunately that isn't the issue as the docker bridge is 172... and
> bridge0 is 192.168... so they don't conflict. Also docker0 doesn't
> seem
> to have any devices (meaning it brctl show has no interfaces attached
> to it). Finally shutting down docker and removing docker0 doesn't
> restore connectivity, only a reboot does. Not even restarting NM
> fixes
> the issue.

Try running 'iptables-save' before you start docker, and then running
'iptables-save' after.  Diff the results.  Did docker remove anything?

Dan
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org

Re: Docker/Libvirt networking issue / bug?

2016-10-06 Thread Nathanael D. Noblet 
On Thu, 2016-10-06 at 14:28 -0400, Neil Horman wrote:
> I rarely mess with docker, but I expect that the docker0 bridge has
> an ip
> address on it which may conflict with the one on libvirt
> bridge.  That is to
> say, if they are on the same subnet, and the route for the docker0
> bridge takes
> precidence, you will loose dhcp.  Check the docker bridge ip and
> remove it if
> you see one, I expect that will restore your functionality
> 

Unfortunately that isn't the issue as the docker bridge is 172... and
bridge0 is 192.168... so they don't conflict. Also docker0 doesn't seem
to have any devices (meaning it brctl show has no interfaces attached
to it). Finally shutting down docker and removing docker0 doesn't
restore connectivity, only a reboot does. Not even restarting NM fixes
the issue.

-- 
Nathanael
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org

Re: Docker/Libvirt networking issue / bug?

2016-10-06 Thread Neil Horman 
On Thu, Oct 06, 2016 at 11:08:21AM -0600, Nathanael D. Noblet wrote:
> Hello,
> 
>   I have an issue that likely needs a bug or something to fix but I'm
> not sure which component is at fault. So I'm writing here in the hopes
> that someone will know where to point me.
> 
>   I have a F24 Workstation where I've been using libvirt/qemu VMs for
> development for awhile now. It works great. I have a bridge0 that
> bridges all VMs and my main NIC so that the VMs are accessible on the
> network. That works well too.
> 
>   Recently I started playing with docker. Both times after running a
> few docker containers (with no defined networking / portmapping etc)
> just a bash shell in the container. My VMs are no longer able to get IP
> addresses from the router. They just time out. It has happened twice
> this week and both times it happens sometime after I've done something
> with docker. 
> 
>   I haven't done any custom configuration with docker so from what I
> understood the docker daemon listens on a unix socket, however there is
> a docker0 bridge that comes up (and doesn't go away when I stop the
> docker service).
> 
>   Any VM that was up and running and connected to the network continues
> to be. Any VM that I bring up after docker (if it is actually docker's
> fault) can no longer get an IP. Stopping the docker service, bringing
> down the docker0 bridge in no way fixes the issue. The only way I've
> been able to fix it is to reboot the workstation. Once I do that then
> the VMs are able to get IPs again.
> 
>   Where do I start looking/debugging this?
> 
I rarely mess with docker, but I expect that the docker0 bridge has an ip
address on it which may conflict with the one on libvirt bridge.  That is to
say, if they are on the same subnet, and the route for the docker0 bridge takes
precidence, you will loose dhcp.  Check the docker bridge ip and remove it if
you see one, I expect that will restore your functionality

Neil

> Thanks,
> -- 
> Nathanael
> ___
> devel mailing list -- devel@lists.fedoraproject.org
> To unsubscribe send an email to devel-le...@lists.fedoraproject.org
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org