FYI Fernando recently contributed a setup script for configuring a local development environment [1] that is continuously verified on mac and ubuntu [2].
[1] https://github.com/apache/beam/pull/14584 [2] https://github.com/apache/beam/actions/workflows/local_env_tests.yml On Wed, May 26, 2021 at 2:17 PM Alexey Romanenko <aromanenko....@gmail.com> wrote: > Thanks for the link! > > Well, going back to initial problem - seems that it’s MacOS-related issue > about permissions to “/var/run/docker.sock” and quick workaround, like “sudo > chmod 666 /var/run/docker.sock” fixes the problem. > Though, I’m not sure we need to do this when building a container (since > it’s not good in terms of security). Perhaps, there is another proper way > how to fix it. > > — > Alexey > > On 25 May 2021, at 19:27, Brian Hulette <bhule...@google.com> wrote: > > Yes this came up before, see BEAM-12149 [1]. I'm not sure what causes it, > I wasn't able to repro the issue. > > [1] https://issues.apache.org/jira/browse/BEAM-12149 > > On Tue, May 25, 2021 at 10:21 AM Alexey Romanenko < > aromanenko....@gmail.com> wrote: > >> I checked it again and I have failed Docker-dependent tests: >> >> > Task :sdks:java:io:clickhouse:test >> >> org.apache.beam.sdk.io.clickhouse.AtomicInsertTest > classMethod FAILED >> java.lang.IllegalStateException at >> DockerClientProviderStrategy.java:215 >> >> org.apache.beam.sdk.io.clickhouse.AtomicInsertTest > classMethod FAILED >> java.lang.NullPointerException at BaseClickHouseTest.java:134 >> >> org.apache.beam.sdk.io.clickhouse.ClickHouseIOTest > classMethod FAILED >> java.lang.IllegalStateException at >> DockerClientProviderStrategy.java:109 >> >> org.apache.beam.sdk.io.clickhouse.ClickHouseIOTest > classMethod FAILED >> java.lang.NullPointerException at BaseClickHouseTest.java:134 >> >> 29 tests completed, 4 failed >> >> > Task :sdks:java:io:clickhouse:test FAILED >> >> >> >> Btw, while running it for the first time once a container was started, I >> had an issue with file permissions that I foxed manually. Is it a known >> issue? >> >> $ ./gradlew -p sdks/java/io/clickhouse/ check >> Starting a Gradle Daemon, 1 incompatible and 2 stopped Daemons could not >> be reused, use --status for details >> Configuration on demand is an incubating feature. >> > Task :model:pipeline:generateProto FAILED >> >> FAILURE: Build failed with an exception. >> >> * What went wrong: >> Execution failed for task ':model:pipeline:generateProto'. >> > protoc: stdout: . stderr: >> /home/aromanenko/.gradle/caches/modules-2/files-2.1/io.grpc/protoc-gen-grpc-java/1.26.0/4f8bb54a74ab655cf2691e6eaa513fccc4c605d5/protoc-gen-grpc-java-1.26.0-linux-x86_64.exe: >> program not found or is not executable >> Please specify a program using absolute path or make sure the program >> is available in your PATH system variable >> --grpc_out: protoc-gen-grpc: Plugin failed with status code 1. >> >> $ ll >> /home/aromanenko/.gradle/caches/modules-2/files-2.1/io.grpc/protoc-gen-grpc-java/1.26.0/4f8bb54a74ab655cf2691e6eaa513fccc4c605d5/protoc-gen-grpc-java-1.26.0-linux-x86_64.exe >> -rw-r--r-- 1 aromanenko users 3009112 May 21 17:35 >> /home/aromanenko/.gradle/caches/modules-2/files-2.1/io.grpc/protoc-gen-grpc-java/1.26.0/4f8bb54a74ab655cf2691e6eaa513fccc4c605d5/protoc-gen-grpc-java-1.26.0-linux-x86_64.exe >> >> >> — >> Alexey >> >> On 21 May 2021, at 18:06, Brian Hulette <bhule...@google.com> wrote: >> >> I think the build environment was set up with that configured: >> https://github.com/apache/beam/blob/40326dd0a2a1c9b5dcbbcd6486a43e3875a64a43/start-build-env.sh#L110 >> Could there be something about your environment preventing that from >> working? >> >> Brian >> >> On Fri, May 21, 2021 at 3:34 AM Gleb Kanterov <g...@spotify.com> wrote: >> >>> Is it possible to mount the Docker socket inside the build-env Docker >>> container? We run a lot of similar tests in CI, and it always worked: >>> >>> --mount type=bind,source=/var/run/docker.sock,target=/var/run/docker.sock >>> >>> On Fri, May 21, 2021 at 12:26 PM Alexey Romanenko < >>> aromanenko....@gmail.com> wrote: >>> >>>> Hello, >>>> >>>> Beam provides a very cool feature to run a local development >>>> environment via Docker [1]. In the same time, some unit tests require to >>>> run Docker containers to test against “real” instances (for example, >>>> ClickHouseIOTest). So, it will end up with “docker-in-docker” issue and >>>> such tests will fail. >>>> >>>> What would be a proper solution for that? Annotate these tests with a >>>> specific “DockerImageRequired” annotation and skip them when running from >>>> inside container or something else? Any ideas on this? >>>> >>>> Thanks, >>>> Alexey >>>> >>>> >>>> [1] https://github.com/apache/beam/blob/master/start-build-env.sh >>> >>> >> >
