1fanwang opened a new pull request, #20660: URL: https://github.com/apache/hudi/pull/20660
### Describe the issue this Pull Request addresses Closes https://github.com/apache/hudi/issues/20576 Flink streaming reads report a delay for the latest issued commit and for the commit being read. The delay reads the commit time in the reader JVM's default timezone, but a table with a UTC timeline writes commit times in UTC. A reader in Asia/Shanghai reports 29400 seconds for a commit made ten minutes ago, instead of 600. ### Summary and Changelog Commit times are now read in the timezone saved in the table config. A UTC table is read as UTC wherever the reader runs, and a table on the local timezone is read in the reader's zone, as before. A streaming read can start before its table exists, so the legacy reader looks the timezone up when it first parses a commit time, not when it sets up metrics. The Source V2 source is created after the table exists and reads it up front. https://github.com/apache/hudi/pull/19081 adds the same timezone-aware parsing helper. ### Impact Only metric values change, for UTC tables read from another timezone. The clustering, compaction and write metrics read commit times the same way and are not changed here. ### Risk Level low. Local-timezone tables behave exactly as before, and nothing fails while the table does not exist yet. ### Documentation Update none ### Contributor's checklist - [x] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute) - [x] Enough context is provided in the sections above - [x] Adequate tests were added if applicable ### Testing Setup: JDK 17.0.21, Maven 3.9.16, the Flink 2.2 profile, base master 88b20661a16ab228b312adf94faad909debbe9ce, JVM default zone America/New_York. The upstream modules were installed once from master: ``` $ mvn -T 2 -B -Dflink2.2 -Davro.version=1.11.4 -Dparquet.version=1.15.2 -pl hudi-flink-datasource/hudi-flink -am -DskipTests -Dcheckstyle.skip=true -Drat.skip=true -Djacoco.skip=true -Dmaven.javadoc.skip=true install ``` The new test starts the legacy monitor with no table, then creates the table with a UTC timeline in its config, adds a commit made ten minutes ago, and reads the issued instant delay while the JVM default zone is Asia/Shanghai, the zone of the issue. It logs the delay before asserting on it. Before, on unmodified master with only the new test class added: ``` $ mvn -B -Dflink2.2 -Davro.version=1.11.4 -Dparquet.version=1.15.2 -Drat.skip=true -Djacoco.skip=true -Dsurefire.failIfNoSpecifiedTests=false -Dsurefire.timeout=600 -pl hudi-common,hudi-flink-datasource/hudi-flink -Dtest=TestStreamReadMonitoringFunctionMetrics test 825 [main] INFO org.apache.hudi.source.TestStreamReadMonitoringFunctionMetrics [] - The issued instant delay is 29400 seconds for a commit made ten minutes ago org.opentest4j.AssertionFailedError: The commit was made ten minutes ago, but the delay is 29400 seconds ==> expected: <true> but was: <false> [INFO] BUILD FAILURE ``` After, the same command on this branch: ``` $ mvn -B -Dflink2.2 -Davro.version=1.11.4 -Dparquet.version=1.15.2 -Drat.skip=true -Djacoco.skip=true -Dsurefire.failIfNoSpecifiedTests=false -Dsurefire.timeout=600 -pl hudi-common,hudi-flink-datasource/hudi-flink -Dtest=TestStreamReadMonitoringFunctionMetrics test 809 [main] INFO org.apache.hudi.source.TestStreamReadMonitoringFunctionMetrics [] - The issued instant delay is 600 seconds for a commit made ten minutes ago [INFO] BUILD SUCCESS ``` Unit tests for the parser, the metrics class and the legacy operator's lookup are in the diff. Not run: the MiniCluster integration tests, the other Flink profiles, and a test of the Source V2 metrics path, which takes the timezone at construction and is covered only by the existing constructor tests. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
