[
https://issues.apache.org/jira/browse/FLINK-39871?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18086701#comment-18086701
]
Sergey Nuyanzin commented on FLINK-39871:
-----------------------------------------
fixed as part of FLINK-38809
it was a bug in java8 (SimpleDateFormat which was fixed in java9)
also jackson since 2.9.x mitigated this in their own codebase
the problem was that testcontainers 1.x uses jackson 2.8.8 as their own shaded
dependency where the issue is not mitigated yet
> MinioTestContainerTest fails on JDK 8 CI agents in a non-UTC timezone because
> testcontainers cannot parse the +08:00 Network.created timestamp
> ----------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: FLINK-39871
> URL: https://issues.apache.org/jira/browse/FLINK-39871
> Project: Flink
> Issue Type: Bug
> Components: Build System / Azure Pipelines, Build System / CI, Test
> Infrastructure
> Affects Versions: 1.20.6
> Reporter: Martijn Visser
> Priority: Blocker
>
> MinioTestContainerTest in flink-s3-fs-base fails deterministically on the
> Azure CI "connect" stage for release-1.20 (6 tests, 6 errors), while the same
> code passes on GitHub Actions and on release-2.0 and later. The container
> fails to start during the Docker network inspect with:
> {code}
> org.testcontainers.shaded.com.fasterxml.jackson.databind.exc.InvalidFormatException:
> Can not deserialize value of type java.util.Date from String
> "2025-06-11T15:22:00.047060135+08:00": ... while it seems to fit format
> 'yyyy-MM-dd'T'HH:mm:ss.SSSZ', parsing fails (leniency? null)
> {code}
> The exact value varies per agent (another run showed
> 2022-09-25T13:46:35.209299016+08:00); the constant is a +08:00 offset with
> sub-millisecond precision.
> Root cause: MinioTestContainer calls withNetworkAliases(...), which makes
> Testcontainers create and inspect a Docker network. docker-java deserializes
> Network.created, a java.util.Date. The Docker daemon serializes that
> timestamp in the host's local timezone. The Azure "Default" agent pool is
> self-hosted machines in the +08 (China) timezone, so Network.created is
> returned as ...+08:00 with nanosecond precision. The Jackson bundled in
> Testcontainers 1.21.4 cannot parse a colon timezone offset combined with
> sub-millisecond precision under JDK 8; it parses the same value fine under
> JDK 17 and 21.
> This is why the failure is environment-specific: it requires both a non-UTC
> Docker daemon host and JDK 8.
> {code}
> Environment JDK Network.created Result
> Azure Default pool (self-hosted, +08) 8 +08:00 FAIL
> Azure Default pool (same agents) 17 +08:00 PASS (JDK 17
> parses it)
> GitHub Actions (UTC runners) 8 Z PASS
> {code}
> release-2.0 and later are green because that stage runs on JDK 17; GitHub
> Actions is green because its runners are UTC. release-1.20 on Azure is the
> only combination with both conditions.
> I've been able to reproduce this locally with the Jackson bundled in
> Testcontainers 1.21.4: a "Z" timestamp with nanoseconds parses on JDK 8, 17
> and 21, while a "+08:00" timestamp with nanoseconds fails only on JDK 8 (the
> jackson-annotations version made no difference, only the JDK did). In
> https://github.com/apache/flink/pull/28313 I've tried both bumping the Minio
> image, and explicitly tagging a version, but that only confirmed that the
> value is generated by the daemon and does not come from any image.
> Testcontainers 1.21.4 and docker-java 3.4.2 are identical across
> release-1.20, release-2.0 and master. The only differing knob in the failing
> stage's pipeline config is jdk: 8 versus jdk: 17.
> I don't know why this is now occurring on the self-hosted machines and not
> before. A couple of options to potentially fix this are:
> 1. Run dockerd on the donated Default-pool agents in UTC (Environment=TZ=UTC
> in the docker service, or set the host timezone) and prune pre-existing
> networks.
> 2. Replace the MinIO Testcontainer with an in-JVM S3 mock, removing the
> Docker-inspect/date-parse path entirely.
> Failing build examples:
> *
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=75608&view=logs&j=f6116f8b-8a37-5717-446a-683b889a4082&t=c7d084eb-e1f0-5a0b-d4c7-bbe0f84d72b4&l=13962
> * bump of MinIO container
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=75648&view=logs&j=2e8cb2f7-b2d3-5c62-9c05-cd756d33a819&t=2dd510a3-5041-5201-6dc3-54d310f68906&l=13635
> * Pinned MinIO
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=75676&view=logs&j=2e8cb2f7-b2d3-5c62-9c05-cd756d33a819&t=2dd510a3-5041-5201-6dc3-54d310f68906&l=13962
> Until this is addressed, nothing can be completely validated in release-1.20
> builds anymore
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
