You have been subscribed to a public bug:
port binding issues with docker-1.10.2 on Ubuntu 16.04
---Steps to Reproduce---
1. Install Ubuntu 16.04 guest on PowerKVM 3.1.1 #5.1
2. Install docker
apt-get update
apt-get install docker.io
3. create base image
debootstrap xenial xenial
root@u1604base:~# docker images
REPOSITORY TAG IMAGE ID CREATED
SIZE
xenial-base latest d1d4fe4bb11e About an hour ago
329.9 MB
4. create a container with with various port binding ranges
case 1 port ranges 600-700 -----------> Worked
root@u1604base:~# docker run --rm -it -p 600-700:600-700 xenial-base sh
#
case 2 port ranges 600-710 -------------> Not working
root@u1604base:~# docker run --rm -it -p 600-710:600-710 xenial-base sh
docker: Error response from daemon: failed to create endpoint nauseous_ride on
network bridge: iptables failed: iptables --wait -t filter -A DOCKER ! -i
docker0 -o docker0 -p tcp -d 172.17.0.2 --dport 602 -j ACCEPT: (fork/exec
/sbin/iptables: resource temporarily unavailable).
The reported port binding issue has been initially found from Watson
research team while creating more number of containers. We later
investigated and seen its an issue with port binding.
I'm still trying to investigate more on this by starting and stopping
daemon and checking, does that make any difference.
== Comment: #6 - Gowrishankar Muthukrishnan <gowrishanka...@in.ibm.com>
- 2016-03-15 08:21:56 ==
Ok. I spent sometime playing with docker daemon today and at last found real
cause for this. You could solve this by running:
echo `pidof docker` > /sys/fs/cgroup/pids/cgroup.procs
Details as below. First, the error as seen in docker run would mislead unless
the details are not read in journalctl. I turned on debug mode for daemon and
checked verbose logs through journalctl -p docker.service. In otherway, you can
stop systemctl service and run directly docker daemon -D, which would also
print
detail info on console.
In the verbose logs, interesting observation is that, I could see iptables
insert option (-I) being called until to some port mapping successfully done
(beginning from higher port number and decrement by one after updating nat and
filter tables). For ex, port map ranging between 45700 and 45600:
time="2016-03-15T07:22:39.929406000-04:00" level=debug msg="/sbin/iptables,
[--wait -t nat -A DOCKER -p tcp -d 0/0 --dport 45659 -j DNAT --to-destination
172.17.42.2:45659 ! -i docker0]"
time="2016-03-15T07:22:39.932421000-04:00" level=debug msg="/sbin/iptables,
[--wait -t filter -A DOCKER ! -i docker0 -o docker0 -p tcp -d 172.17.42.2
--dport 45659 -j ACCEPT]"
time="2016-03-15T07:22:39.934363000-04:00" level=debug msg="/sbin/iptables,
[--wait -t nat -A POSTROUTING -p tcp -s 172.17.42.2 -d 172.17.42.2 --dport
45659 -j MASQUERADE]"
# Confirms that, port mapping done until 45659 (from 45700).
time="2016-03-15T07:22:39.943160000-04:00" level=debug msg="/sbin/iptables,
[--wait -t nat -D DOCKER -p tcp -d 0/0 --dport 45659 -j DNAT --to-destination
172.17.42.2:45659 ! -i docker0]"
time="2016-03-15T07:22:39.943760000-04:00" level=warning msg="Failed to
allocate and map port 45659-45659: Error starting userland proxy: "
# Interestingly, userland proxies could not come up meanwhile. It could be
due to capping on forked processes (as iptables are run as commands along with
other docker threads). Also, below sample of error is actually thrown on
docker run command finally.
time="2016-03-15T07:22:39.944949000-04:00" level=debug msg="/sbin/iptables,
[--wait -t nat -D DOCKER -p tcp -d 0/0 --dport 45700 -j DNAT --to-destination
172.17.42.2:45700 ! -i docker0]"
time="2016-03-15T07:22:39.945419000-04:00" level=error msg="Error on iptables
delete: iptables failed: iptables --wait -t nat -D DOCKER -p tcp -d 0/0 --dport
45700 -j DNAT --to-destination 172.17.42.2:45700 ! -i docker0: (fork/exec
/sbin/iptables: resource temporarily unavailable)"
...
..
Then I compared 16.04 and 15.10 ubuntu kernels and found that latter did not
have this problem and also found cgroup PID controller enabled in 16.04 kernel
(CONFIG_CGROUP_PIDS). You can refer its doc for how we can exploit it for
containers.
https://www.kernel.org/doc/Documentation/cgroup-v1/pids.txt
Interestingly, docker daemon PID is not added in its parent group. Adding it
solves this problem.
root@ubuntu1604:/home/test# docker run --rm -it -p 600-700:600-700
ppc64le/ubuntu /bin/bash
root@2359fabc9d9d:/# exit
exit
Over to build/docker team to fix bringing up docker daemon correctly.
== Comment: #8 - Kalpana Shetty <kalsh...@in.ibm.com> - 2016-03-15 08:41:33 ==
Thanks Gowri; this helps.
root@u1604base:~# docker run --rm -it -p 200-500:200-500 xenial-base sh
#
root@u1604base:~# docker run --rm -it -p 200-1000:200-1000 xenial-base
sh
# #
It works... :)
== Comment: #9 - Mel Bakhshi <m...@ca.ibm.com> - 2016-03-15 11:30:32 ==
I also tested this on Ubuntu16.04 with docker 1.10.2 on ppc64le :
"docker daemon PID is not added in its parent group. Adding it
solves this problem."
When should we expect this fix to be GA?
== Comment: #10 - Kalpana Shetty <kalsh...@in.ibm.com> - 2016-03-15 11:52:02 ==
JFYI....
I have tried creating 1K containers with port option, without any issues. This
works fine after I followed suggested work around of docker pid (see comment
#6).
docker run cmd: docker run --rm -it -p 100-200:100-200 xenial-base ls
** Affects: docker.io (Ubuntu)
Importance: Undecided
Assignee: Taco Screen team (taco-screen-team)
Status: Triaged
** Tags: architecture-ppc64le bot-comment bugnameltc-138945 severity-high
targetmilestone-inin1604
--
port binding issues with docker-1.10.2 on Ubuntu 16.04
https://bugs.launchpad.net/bugs/1557669
You received this bug notification because you are a member of Ubuntu Server
Team, which is subscribed to docker.io in Ubuntu.
--
Ubuntu-server-bugs mailing list
Ubuntu-server-bugs@lists.ubuntu.com
Modify settings or unsubscribe at:
https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
