Thanks! There have been a few successful runs now. On Tue, Oct 23, 2018 at 8:52 AM Yifan Zou <yifan...@google.com> wrote:
> FYI, the docker was restarted on beam15. > > On Tue, Oct 23, 2018 at 7:08 AM Thomas Weise <t...@apache.org> wrote: > >> For the latter (createProcessWorker): >> https://github.com/apache/beam/pull/6793 >> >> >> On Tue, Oct 23, 2018 at 6:47 AM Thomas Weise <t...@apache.org> wrote: >> >>> Thanks for taking a look Yifan. Yes, it appears this was an intermittent >>> issue. >>> >>> For beam_PostCommit_Python_VR_Flink we are left with: >>> >>> * beam15 docker errors >>> * segmentation faults >>> * "Execution failed for task ':beam-sdks-python:createProcessWorker'" - >>> which should not even execute since we are using Docker >>> >>> >>> On Mon, Oct 22, 2018 at 10:50 PM Yifan Zou <yifan...@google.com> wrote: >>> >>>> I'm not able to reproduce that error in Beam6 (#459 >>>> <https://builds.apache.org/job/beam_PostCommit_Python_VR_Flink/459/>, >>>> #460 >>>> <https://builds.apache.org/job/beam_PostCommit_Python_VR_Flink/460/>), >>>> it probably due to some outage of Debian [1]. The image was successfully >>>> built, but the test failed in other reasons. >>>> And indeed, the beam_PostCommit_Python_VR_Flink is very flaky. >>>> >>>> Yifan >>>> >>>> [1] https://github.com/docker-library/python/issues/241 >>>> >>>> On Mon, Oct 22, 2018 at 5:39 PM Thomas Weise <t...@apache.org> wrote: >>>> >>>>> Looks like we have more container build related errors. >>>>> >>>>> This is from beam6 - >>>>> https://builds.apache.org/job/beam_PostCommit_Python_VR_Flink_PR/44/ >>>>> >>>>> Reading package lists... >>>>> [91mW: The repository 'http://deb.debian.org/debian stretch Release' >>>>> does not have a Release file. >>>>> >>>>> W: The repository 'http://deb.debian.org/debian stretch-updates Release' >>>>> does not have a Release file. >>>>> E: Failed to fetch >>>>> http://deb.debian.org/debian/dists/stretch/main/binary-amd64/Packages >>>>> 404 Not Found >>>>> E: Failed to fetch >>>>> http://deb.debian.org/debian/dists/stretch-updates/main/binary-amd64/Packages >>>>> 404 Not Found >>>>> E: Some index files failed to download. They have been ignored, or old >>>>> ones used instead. >>>>> >>>>> >>>>> On Mon, Oct 22, 2018 at 2:54 PM Ankur Goenka <goe...@google.com> >>>>> wrote: >>>>> >>>>>> Thanks Yifan! >>>>>> >>>>>> On Mon, Oct 22, 2018 at 2:53 PM Yifan Zou <yifan...@google.com> >>>>>> wrote: >>>>>> >>>>>>> So, looks like none of us have the permissions. I filed INFRA-17167 >>>>>>> <https://issues.apache.org/jira/browse/INFRA-17167> to the Infra >>>>>>> team to restart the docker on the beam15. >>>>>>> >>>>>>> Thanks. >>>>>>> Yifan >>>>>>> >>>>>>> On Mon, Oct 22, 2018 at 9:20 AM Scott Wegner <sc...@apache.org> >>>>>>> wrote: >>>>>>> >>>>>>>> I've seen the docker issue pop-up on website pre-commits as well: >>>>>>>> https://issues.apache.org/jira/browse/BEAM-5783. There were also >>>>>>>> on beam15. >>>>>>>> >>>>>>>> When I searched around the internet I found lots of instances of >>>>>>>> the same error; it seems to be some unreliability in the guts of Docker >>>>>>>> [1]. Perhaps restarting the VM or docker daemon could help. Does >>>>>>>> anybody >>>>>>>> have permissions to log on and try it? >>>>>>>> >>>>>>>> [1] >>>>>>>> https://github.com/moby/moby/issues/31849#issuecomment-320236354 >>>>>>>> >>>>>>>> On Sun, Oct 21, 2018 at 7:13 PM Thomas Weise <t...@apache.org> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> There are two issues with >>>>>>>>> https://builds.apache.org/job/beam_PostCommit_Python_VR_Flink/ >>>>>>>>> currently: >>>>>>>>> >>>>>>>>> 1) The mentioned issue with docker on beam15 - Jason, can you >>>>>>>>> possibly advise how to deal with it? >>>>>>>>> >>>>>>>>> 2) Frequent failure due to "Segmentation fault (core dumped)", as >>>>>>>>> exhibited by >>>>>>>>> https://builds.apache.org/job/beam_PostCommit_Python_VR_Flink/449/consoleText >>>>>>>>> >>>>>>>>> The Gradle scan is here: >>>>>>>>> >>>>>>>>> >>>>>>>>> https://scans.gradle.com/s/ebhxs4l65cow4/failure?openFailures=WzBd&openStackTraces=WzEse31d#top=0 >>>>>>>>> >>>>>>>>> There are multiple of those in sequence on beam13 >>>>>>>>> >>>>>>>>> Some more comments: >>>>>>>>> https://issues.apache.org/jira/browse/BEAM-5467 >>>>>>>>> >>>>>>>>> Any help to further investigate or fix would be appreciated! >>>>>>>>> >>>>>>>>> Thanks, >>>>>>>>> Thomas >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> On Fri, Oct 19, 2018 at 4:51 PM Yifan Zou <yifan...@google.com> >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> I got "Failed to restart docker.service: Interactive >>>>>>>>>> authentication required" while trying to restart the docker on >>>>>>>>>> beam15. >>>>>>>>>> Does anyone have the permission to do that? Or, we need to ask >>>>>>>>>> Apache Infra for help. >>>>>>>>>> >>>>>>>>>> Thanks. >>>>>>>>>> Yifan >>>>>>>>>> >>>>>>>>>> On Fri, Oct 19, 2018 at 2:51 PM Ankur Goenka <goe...@google.com> >>>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>>> Hi, >>>>>>>>>>> >>>>>>>>>>> Can we restart docker as it seems to have fixed the issue for >>>>>>>>>>> others https://github.com/moby/moby/issues/31849 ? >>>>>>>>>>> >>>>>>>>>>> Thanks, >>>>>>>>>>> Ankur >>>>>>>>>>> >>>>>>>>>>> On Fri, Oct 19, 2018 at 1:11 PM Yifan Zou <yifan...@google.com> >>>>>>>>>>> wrote: >>>>>>>>>>> >>>>>>>>>>>> Hi, >>>>>>>>>>>> >>>>>>>>>>>> The docker has been installed on all Jenkins VMs. The image >>>>>>>>>>>> build process was interrupted by a grpc connection issue. >>>>>>>>>>>> >>>>>>>>>>>> *11:02:12* Starting process 'command 'docker''. Working directory: >>>>>>>>>>>> /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_VR_Flink/src/sdks/python/container/build/docker >>>>>>>>>>>> Command: docker build --no-cache -t >>>>>>>>>>>> jenkins-docker-apache.bintray.io/beam/python:latest .*11:02:12* >>>>>>>>>>>> Successfully started process 'command 'docker''*11:02:12* Sending >>>>>>>>>>>> build context to Docker daemon 17.65MB >>>>>>>>>>>> *11:02:12* Step 1/9 : FROM python:2-stretch*11:02:12* ---> >>>>>>>>>>>> 3c43a5d4034a*11:02:12* Step 2/9 : MAINTAINER "Apache Beam >>>>>>>>>>>> <dev@beam.apache.org>"*11:02:12* ---> Running in >>>>>>>>>>>> f86bad9aef9c*11:02:12* ---> 610a5dec907e*11:02:12* Removing >>>>>>>>>>>> intermediate container f86bad9aef9c*11:02:12* Step 3/9 : RUN >>>>>>>>>>>> apt-get update && apt-get install -y libsnappy-dev >>>>>>>>>>>> libyaml-dev && rm -rf /var/lib/apt/lists/**11:02:12* >>>>>>>>>>>> ---> Running in 5e9b67be03f9*11:02:12* grpc: the connection is >>>>>>>>>>>> unavailable >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> - Yifan >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> On Fri, Oct 19, 2018 at 12:45 PM Ankur Goenka < >>>>>>>>>>>> goe...@google.com> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> Hi, >>>>>>>>>>>>> >>>>>>>>>>>>> Flink Validates Runner test cases are failing on Beam 15 >>>>>>>>>>>>> because docker is not installed. >>>>>>>>>>>>> Failing tasks >>>>>>>>>>>>> https://builds.apache.org/job/beam_PostCommit_Python_VR_Flink/buildTimeTrend >>>>>>>>>>>>> Can we install docker on all the machines as the Portable >>>>>>>>>>>>> Validates Runner tests need it. >>>>>>>>>>>>> >>>>>>>>>>>>> Thanks, >>>>>>>>>>>>> Ankur >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> Got feedback? tinyurl.com/swegner-feedback >>>>>>>> >>>>>>>
