1fanwang opened a new pull request, #20660:
URL: https://github.com/apache/hudi/pull/20660

   ### Describe the issue this Pull Request addresses
   
   Closes https://github.com/apache/hudi/issues/20576
   
   Flink streaming reads report a delay for the latest issued commit and for 
the commit being read. The delay reads the commit time in the reader JVM's 
default timezone, but a table with a UTC timeline writes commit times in UTC. A 
reader in Asia/Shanghai reports 29400 seconds for a commit made ten minutes 
ago, instead of 600.
   
   ### Summary and Changelog
   
   Commit times are now read in the timezone saved in the table config. A UTC 
table is read as UTC wherever the reader runs, and a table on the local 
timezone is read in the reader's zone, as before. A streaming read can start 
before its table exists, so the legacy reader looks the timezone up when it 
first parses a commit time, not when it sets up metrics. The Source V2 source 
is created after the table exists and reads it up front. 
https://github.com/apache/hudi/pull/19081 adds the same timezone-aware parsing 
helper.
   
   ### Impact
   
   Only metric values change, for UTC tables read from another timezone. The 
clustering, compaction and write metrics read commit times the same way and are 
not changed here.
   
   ### Risk Level
   
   low. Local-timezone tables behave exactly as before, and nothing fails while 
the table does not exist yet.
   
   ### Documentation Update
   
   none
   
   ### Contributor's checklist
   
   - [x] Read through [contributor's 
guide](https://hudi.apache.org/contribute/how-to-contribute)
   - [x] Enough context is provided in the sections above
   - [x] Adequate tests were added if applicable
   
   ### Testing
   
   Setup: JDK 17.0.21, Maven 3.9.16, the Flink 2.2 profile, base master 
88b20661a16ab228b312adf94faad909debbe9ce, JVM default zone America/New_York. 
The upstream modules were installed once from master:
   
   ```
   $ mvn -T 2 -B -Dflink2.2 -Davro.version=1.11.4 -Dparquet.version=1.15.2 -pl 
hudi-flink-datasource/hudi-flink -am -DskipTests -Dcheckstyle.skip=true 
-Drat.skip=true -Djacoco.skip=true -Dmaven.javadoc.skip=true install
   ```
   
   The new test starts the legacy monitor with no table, then creates the table 
with a UTC timeline in its config, adds a commit made ten minutes ago, and 
reads the issued instant delay while the JVM default zone is Asia/Shanghai, the 
zone of the issue. It logs the delay before asserting on it.
   
   Before, on unmodified master with only the new test class added:
   
   ```
   $ mvn -B -Dflink2.2 -Davro.version=1.11.4 -Dparquet.version=1.15.2 
-Drat.skip=true -Djacoco.skip=true -Dsurefire.failIfNoSpecifiedTests=false 
-Dsurefire.timeout=600 -pl hudi-common,hudi-flink-datasource/hudi-flink 
-Dtest=TestStreamReadMonitoringFunctionMetrics test
   825  [main] INFO  
org.apache.hudi.source.TestStreamReadMonitoringFunctionMetrics [] - The issued 
instant delay is 29400 seconds for a commit made ten minutes ago
   org.opentest4j.AssertionFailedError: The commit was made ten minutes ago, 
but the delay is 29400 seconds ==> expected: <true> but was: <false>
   [INFO] BUILD FAILURE
   ```
   
   After, the same command on this branch:
   
   ```
   $ mvn -B -Dflink2.2 -Davro.version=1.11.4 -Dparquet.version=1.15.2 
-Drat.skip=true -Djacoco.skip=true -Dsurefire.failIfNoSpecifiedTests=false 
-Dsurefire.timeout=600 -pl hudi-common,hudi-flink-datasource/hudi-flink 
-Dtest=TestStreamReadMonitoringFunctionMetrics test
   809  [main] INFO  
org.apache.hudi.source.TestStreamReadMonitoringFunctionMetrics [] - The issued 
instant delay is 600 seconds for a commit made ten minutes ago
   [INFO] BUILD SUCCESS
   ```
   
   Unit tests for the parser, the metrics class and the legacy operator's 
lookup are in the diff. Not run: the MiniCluster integration tests, the other 
Flink profiles, and a test of the Source V2 metrics path, which takes the 
timezone at construction and is covered only by the existing constructor tests.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to