Hey guys, Is this an open issue?
On Mon, Aug 22, 2016 at 9:27 AM, Cory Keane <[email protected]> wrote: > Hi guys, > > I was wondering if you have any updates on this. Thanks for your help. > > On Sun, Aug 14, 2016 at 7:22 AM, Cory Keane <[email protected]> wrote: > >> The docker command I try running takes the format: `docker run --rm >> --name=blahblahblah -e APP_ENV=production quay.io/path/tocontainer:stable >> php aphpfile.php` >> >> Below you will find the following: >> 1. CoreOS Info >> 2. Output from running `mount` >> >> >> 1. CoreOS Info: >> >> NAME=CoreOS >> >> ID=coreos >> >> VERSION=1081.5.0 >> >> VERSION_ID=1081.5.0 >> >> BUILD_ID=2016-07-16-2255 >> >> PRETTY_NAME="CoreOS 1081.5.0 (MoreOS)" >> >> ANSI_COLOR="1;32" >> >> HOME_URL="https://coreos.com/"; >> >> BUG_REPORT_URL="https://github.com/coreos/bugs/issues"; >> >> >> Output from running `mount`: >> >> sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime) >> >> proc on /proc type proc (rw,nosuid,nodev,noexec,relatime) >> >> devtmpfs on /dev type devtmpfs (rw,nosuid,size=2008876k,nr_in >> odes=502219,mode=755) >> >> securityfs on /sys/kernel/security type securityfs >> (rw,nosuid,nodev,noexec,relatime) >> >> tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev) >> >> devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid >> =5,mode=620,ptmxmode=000) >> >> tmpfs on /run type tmpfs (rw,nosuid,nodev,mode=755) >> >> tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,mode=755) >> >> cgroup on /sys/fs/cgroup/systemd type cgroup >> (rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/ >> lib/systemd/systemd-cgroups-agent,name=systemd) >> >> pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime) >> >> cgroup on /sys/fs/cgroup/hugetlb type cgroup >> (rw,nosuid,nodev,noexec,relatime,hugetlb) >> >> cgroup on /sys/fs/cgroup/freezer type cgroup >> (rw,nosuid,nodev,noexec,relatime,freezer) >> >> cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup >> (rw,nosuid,nodev,noexec,relatime,cpu,cpuacct) >> >> cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relati >> me,blkio) >> >> cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relati >> me,memory) >> >> cgroup on /sys/fs/cgroup/devices type cgroup >> (rw,nosuid,nodev,noexec,relatime,devices) >> >> cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup >> (rw,nosuid,nodev,noexec,relatime,net_cls,net_prio) >> >> cgroup on /sys/fs/cgroup/perf_event type cgroup >> (rw,nosuid,nodev,noexec,relatime,perf_event) >> >> cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relati >> me,cpuset) >> >> cgroup on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relati >> me,pids) >> >> /dev/xvda9 on / type ext4 (rw,relatime,data=ordered) >> >> /dev/xvda3 on /usr type ext4 (ro,relatime,block_validity,de >> lalloc,barrier,user_xattr,acl) >> >> selinuxfs on /sys/fs/selinux type selinuxfs (rw,relatime) >> >> systemd-1 on /proc/sys/fs/binfmt_misc type autofs >> (rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5, >> direct,pipe_ino=11137) >> >> tmpfs on /tmp type tmpfs (rw) >> >> tmpfs on /media type tmpfs (rw,nosuid,nodev,noexec,relatime) >> >> hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime) >> >> debugfs on /sys/kernel/debug type debugfs (rw,relatime) >> >> xenfs on /proc/xen type xenfs (rw,relatime) >> >> systemd-1 on /boot type autofs (rw,relatime,fd=35,pgrp=1,time >> out=0,minproto=5,maxproto=5,direct,pipe_ino=11461) >> >> mqueue on /dev/mqueue type mqueue (rw,relatime) >> >> /dev/xvda6 on /usr/share/oem type ext4 (rw,nodev,relatime,commit=600, >> data=ordered) >> >> /dev/xvda1 on /boot type vfat (rw,relatime,fmask=0022,dmask= >> 0022,codepage=437,iocharset=ascii,shortname=mixed,errors=remount-ro) >> >> tmpfs on /run/user/500 type tmpfs (rw,nosuid,nodev,relatime,size >> =404996k,mode=700,uid=500,gid=500) >> >> >> Thank you, >> >> Cory >> >> On Sat, Aug 13, 2016 at 9:29 PM, Brandon Philips < >> [email protected]> wrote: >> >>> Hello Cory- >>> >>> Also, what versions of CoreOS is this happening under? `cat >>> /etc/os-release`. >>> >>> Brandon >>> >>> >>> On Sat, Aug 13, 2016 at 7:26 PM Brandon Philips < >>> [email protected]> wrote: >>> >>>> Hello Cory- >>>> >>>> There are a number of errors about the mounts which should be >>>> non-fatal: >>>> >>>> Aug 09 15:40:08 ip-172-31-16-118.ec2.internal dockerd[1107]: >>>> time="2016-08-09T15:40:08.520711349Z" level=error msg="Clean up Error! >>>> Cannot destroy container 26537dcd69eeb25dcd3a4867f81db9 >>>> 09e5ca992efa1e6861d9e804770ec11239: rmdriverfs: Driver overlay failed >>>> to remove root filesystem 26537dcd69eeb25dcd3a4867f81db9 >>>> 09e5ca992efa1e6861d9e804770ec11239: mount still active" >>>> Aug 09 15:40:08 ip-172-31-16-118.ec2.internal dockerd[1107]: >>>> time="2016-08-09T15:40:08.520749851Z" level=error msg="Handler for >>>> POST /v1.22/containers/create returned error: Error relabeling upper >>>> directory: operation not supported" >>>> >>>> Could you grab the output of `mount` from these machines? >>>> >>>> And it doesn't look like the daemon dies for several minutes after: >>>> >>>> Aug 09 15:43:41 ip-172-31-16-118.ec2.internal systemd[1]: Stopping >>>> Docker Application Container Engine... >>>> >>>> I really don't know what is getting messed up to cause this. What are >>>> the arguments to docker run? >>>> >>>> It looks sort of a like a repeat of https://github.com/coreos/b >>>> ugs/issues/1301 which was about SELinux and OverlayFS so I am going to >>>> cc in Nick, Matthew, and Crawford. >>>> >>>> Cheers, >>>> >>>> Brandon >>>> >>>> On Tue, Aug 9, 2016 at 10:41 AM Cory Keane <[email protected]> wrote: >>>> >>>>> Hey Brandon, >>>>> >>>>> I decided to just email the logs directly to you. >>>>> >>>>> Thanks! >>>>> >>>>> On Tue, Aug 9, 2016 at 12:37 PM, Brandon Philips < >>>>> [email protected]> wrote: >>>>> >>>>>> journalctl --no-pager -u docker.service > foo.log >>>>>> >>>>>> I think it turns on no-pager by default if it knows stdout isn't a tty >>>>>> >>>>>> On Tue, Aug 9, 2016 at 10:35 AM Cory Keane <[email protected]> >>>>>> wrote: >>>>>> >>>>>>> No worries Brandon. I will upload them. But what's the easiest way >>>>>>> to export them? The command you told me to run makes it so I have to >>>>>>> press >>>>>>> the space bar to cycle through all the lines of the file. >>>>>>> >>>>>>> Thanks so much. >>>>>>> >>>>>>> On Tue, Aug 9, 2016 at 12:34 PM, Brandon Philips < >>>>>>> [email protected]> wrote: >>>>>>> >>>>>>>> Hey Cory- >>>>>>>> >>>>>>>> It is really hard to know what to look for in the logs. If you >>>>>>>> don't want to share the logs publicly you can email them to me >>>>>>>> personally >>>>>>>> and I will take a look. >>>>>>>> >>>>>>>> Cheers, >>>>>>>> >>>>>>>> Brandon >>>>>>>> >>>>>>>> On Tue, Aug 9, 2016 at 8:42 AM Cory Keane <[email protected]> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> Brandon, I apologize. It appears that this worked for one of the >>>>>>>>> machines experiencing this issue, but not the other machine. Since I >>>>>>>>> posted >>>>>>>>> this, another one of our fleet machines auto updated to what appears >>>>>>>>> to be >>>>>>>>> this faulty version. >>>>>>>>> >>>>>>>>> So please let me know what I am looking for specifically when >>>>>>>>> running the `journalctl -u docker` command. >>>>>>>>> >>>>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> *Cory Keane* >>>>>>> Co-Founder & CTO >>>>>>> ThreadMeUp.com >>>>>>> >>>>>> >>>>> >>>>> >>>>> -- >>>>> *Cory Keane* >>>>> Co-Founder & CTO >>>>> ThreadMeUp.com >>>>> >>>> >> >> >> -- >> *Cory Keane* >> Co-Founder & CTO >> ThreadMeUp.com >> > > > > -- > *Cory Keane* > Co-Founder & CTO > ThreadMeUp.com > -- *Cory Keane* Co-Founder & CTO ThreadMeUp.com
