[
https://issues.apache.org/jira/browse/FLINK-39871?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergey Nuyanzin closed FLINK-39871.
-----------------------------------
Fix Version/s: 1.20.6
Resolution: Fixed
> MinioTestContainerTest fails on JDK 8 CI agents in a non-UTC timezone because
> testcontainers cannot parse the +08:00 Network.created timestamp
> ----------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: FLINK-39871
> URL: https://issues.apache.org/jira/browse/FLINK-39871
> Project: Flink
> Issue Type: Bug
> Components: Build System / Azure Pipelines, Build System / CI, Test
> Infrastructure
> Affects Versions: 1.20.6
> Reporter: Martijn Visser
> Priority: Blocker
> Fix For: 1.20.6
>
>
> MinioTestContainerTest in flink-s3-fs-base fails deterministically on the
> Azure CI "connect" stage for release-1.20 (6 tests, 6 errors), while the same
> code passes on GitHub Actions and on release-2.0 and later. The container
> fails to start during the Docker network inspect with:
> {code}
> org.testcontainers.shaded.com.fasterxml.jackson.databind.exc.InvalidFormatException:
> Can not deserialize value of type java.util.Date from String
> "2025-06-11T15:22:00.047060135+08:00": ... while it seems to fit format
> 'yyyy-MM-dd'T'HH:mm:ss.SSSZ', parsing fails (leniency? null)
> {code}
> The exact value varies per agent (another run showed
> 2022-09-25T13:46:35.209299016+08:00); the constant is a +08:00 offset with
> sub-millisecond precision.
> Root cause: MinioTestContainer calls withNetworkAliases(...), which makes
> Testcontainers create and inspect a Docker network. docker-java deserializes
> Network.created, a java.util.Date. The Docker daemon serializes that
> timestamp in the host's local timezone. The Azure "Default" agent pool is
> self-hosted machines in the +08 (China) timezone, so Network.created is
> returned as ...+08:00 with nanosecond precision. The Jackson bundled in
> Testcontainers 1.21.4 cannot parse a colon timezone offset combined with
> sub-millisecond precision under JDK 8; it parses the same value fine under
> JDK 17 and 21.
> This is why the failure is environment-specific: it requires both a non-UTC
> Docker daemon host and JDK 8.
> {code}
> Environment JDK Network.created Result
> Azure Default pool (self-hosted, +08) 8 +08:00 FAIL
> Azure Default pool (same agents) 17 +08:00 PASS (JDK 17
> parses it)
> GitHub Actions (UTC runners) 8 Z PASS
> {code}
> release-2.0 and later are green because that stage runs on JDK 17; GitHub
> Actions is green because its runners are UTC. release-1.20 on Azure is the
> only combination with both conditions.
> I've been able to reproduce this locally with the Jackson bundled in
> Testcontainers 1.21.4: a "Z" timestamp with nanoseconds parses on JDK 8, 17
> and 21, while a "+08:00" timestamp with nanoseconds fails only on JDK 8 (the
> jackson-annotations version made no difference, only the JDK did). In
> https://github.com/apache/flink/pull/28313 I've tried both bumping the Minio
> image, and explicitly tagging a version, but that only confirmed that the
> value is generated by the daemon and does not come from any image.
> Testcontainers 1.21.4 and docker-java 3.4.2 are identical across
> release-1.20, release-2.0 and master. The only differing knob in the failing
> stage's pipeline config is jdk: 8 versus jdk: 17.
> I don't know why this is now occurring on the self-hosted machines and not
> before. A couple of options to potentially fix this are:
> 1. Run dockerd on the donated Default-pool agents in UTC (Environment=TZ=UTC
> in the docker service, or set the host timezone) and prune pre-existing
> networks.
> 2. Replace the MinIO Testcontainer with an in-JVM S3 mock, removing the
> Docker-inspect/date-parse path entirely.
> Failing build examples:
> *
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=75608&view=logs&j=f6116f8b-8a37-5717-446a-683b889a4082&t=c7d084eb-e1f0-5a0b-d4c7-bbe0f84d72b4&l=13962
> * bump of MinIO container
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=75648&view=logs&j=2e8cb2f7-b2d3-5c62-9c05-cd756d33a819&t=2dd510a3-5041-5201-6dc3-54d310f68906&l=13635
> * Pinned MinIO
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=75676&view=logs&j=2e8cb2f7-b2d3-5c62-9c05-cd756d33a819&t=2dd510a3-5041-5201-6dc3-54d310f68906&l=13962
> Until this is addressed, nothing can be completely validated in release-1.20
> builds anymore
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
