[
https://issues.apache.org/jira/browse/FLINK-40653?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18115147#comment-18115147
]
Martijn Visser edited comment on FLINK-40653 at 9/14/26 12:46 PM:
------------------------------------------------------------------
Root cause is an Azure Pipelines agent change, not anything on our side.
Agent v5.279.0 changed the default of mapDockerSocket to false on Linux
([microsoft/azure-pipelines-agent#5627|https://github.com/microsoft/azure-pipelines-agent/pull/5627]).
The Microsoft docs describe it under "Docker socket mapping":
{quote}Starting with Agent version 5.279.0, container jobs on Linux agents
don't mount the host Docker socket (/var/run/docker.sock) into the job
container by default. This default follows the principle of least privilege, so
jobs that don't need Docker-in-container behavior aren't granted access to the
host Docker daemon.
{quote}
[https://learn.microsoft.com/en-us/azure/devops/pipelines/process/container-phases#docker-socket-mapping]
The docker create lines confirm it. Build 78606 (2026-09-03, green) ran on
agent 5.278.0:
{code:java}
-v "/var/run/docker.sock":"/var/run/docker.sock"
-v "/home/vsts/work/1":"/__w/1"
...
{code}
Build 79017 (2026-09-14, red) ran on agent 5.279.0 with the same six other
mounts and no docker.sock.
The fix is the documented opt-in on the container resource:
{code:java}
resources:
containers:
- container: flink-build-container
image: apache/flink-ci-docker:java_8_11_17_21_25_maven_386_noble
mapDockerSocket: true
{code}
It should go on the shared resource rather than only on the one used by
cron_azure. The ASF self-hosted agents are still on the older agent version and
their container jobs do still get the socket mounted, which is why only
cron_azure is failing today. When those agents are upgraded, every container
based test on Azure breaks the same way.
Note the documented caveat that mapping the socket lets code in the container
run as root on the Docker host. This restores the behaviour we had until
5.279.0 rather than adding new exposure, but it is worth deciding knowingly.
This cannot be verified through a PR build, since azure-pipelines.yml only
contains the ci stage. It needs a nightly or a manual run.
was (Author: martijnvisser):
Root cause is an Azure Pipelines agent change, not anything on our side.
Agent v5.279.0 changed the default of mapDockerSocket to false on Linux
(microsoft/azure-pipelines-agent#5627). The Microsoft docs describe it under
"Docker socket mapping":
{quote}
Starting with Agent version 5.279.0, container jobs on Linux agents don't
mount the host Docker socket (/var/run/docker.sock) into the job container by
default. This default follows the principle of least privilege, so jobs that
don't need Docker-in-container behavior aren't granted access to the host
Docker daemon.
{quote}
https://learn.microsoft.com/en-us/azure/devops/pipelines/process/container-phases#docker-socket-mapping
The docker create lines confirm it. Build 78606 (2026-09-03, green) ran on
agent 5.278.0:
{code}
-v "/var/run/docker.sock":"/var/run/docker.sock"
-v "/home/vsts/work/1":"/__w/1"
...
{code}
Build 79017 (2026-09-14, red) ran on agent 5.279.0 with the same six other
mounts and no docker.sock.
The fix is the documented opt-in on the container resource:
{code}
resources:
containers:
- container: flink-build-container
image: apache/flink-ci-docker:java_8_11_17_21_25_maven_386_noble
mapDockerSocket: true
{code}
It should go on the shared resource rather than only on the one used by
cron_azure. The ASF self-hosted agents are still on the older agent version and
their container jobs do still get the socket mounted, which is why only
cron_azure is failing today. When those agents are upgraded, every container
based test on Azure breaks the same way.
Note the documented caveat that mapping the socket lets code in the
container run as root on the Docker host. This restores the behaviour we had
until 5.279.0 rather than adding new exposure, but it is worth deciding
knowingly.
This cannot be verified through a PR build, since azure-pipelines.yml only
contains the ci stage. It needs a nightly or a manual run.
> Testcontainers tests fail with "Could not find a valid Docker environment" on
> the cron_azure profile
> ----------------------------------------------------------------------------------------------------
>
> Key: FLINK-40653
> URL: https://issues.apache.org/jira/browse/FLINK-40653
> Project: Flink
> Issue Type: Bug
> Components: Build System
> Reporter: Martijn Visser
> Priority: Critical
>
> Since the 2026-09-04 nightly every testcontainers based test in the
> {{test_cron_azure}} legs fails:
> {code}
> java.lang.IllegalStateException: Could not find a valid Docker environment.
> Please see logs and check configuration
> at
> org.testcontainers.dockerclient.DockerClientProviderStrategy.getFirstValidStrategy(DockerClientProviderStrategy.java:265)
> at
> org.apache.flink.core.testutils.TestContainerExtension.instantiateTestContainer(TestContainerExtension.java:59)
> {code}
> On master that is SeaweedFsTestContainerTest and
> SeaweedFsNativeS3TestContainerTest, on the release branches
> MinioTestContainerTest.
> The socket is not present at all. The {{Restore docker images}} task in the
> same job says:
> {code}
> failed to connect to the docker API at unix:///var/run/docker.sock; check if
> the path is correct and if the daemon is running: dial unix
> /var/run/docker.sock: connect: no such file or directory
> {code}
> Docker itself works on those agents. All four {{e2e_*_cron_azure}} jobs
> passed in the same build and they use Docker heavily. e2e-template.yml
> declares no container so they run on the host, while the test jobs run inside
> flink-build-container.
> {{cron_azure}} is the only stage in build-apache-repo.yml whose
> test_pool_definition is {{vmImage: 'ubuntu-24.04'}} instead of {{name:
> Default}}. The other five stages run the same tests in the same container on
> the ASF pool and pass.
> Last green nightly 78606 (2026-09-03), first red 78686 (2026-09-04), and all
> eleven master nightlies since fail the same way. FLINK-40548 is not the
> cause, it was committed after 78686 had already run.
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=79017
> template.flink-ci.yml works around this on GitHub Actions by chmod'ing the
> socket and setting DOCKER_HOST, but that only helps where the socket exists.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
