yihua commented on PR #19898: URL: https://github.com/apache/hudi/pull/19898#issuecomment-6024342396
Thanks for the description and scaladoc updates, both look right. One more ask: a guard test for the listing itself. `TestIncrementalReadWithPartitionFilter` asserts row counts only, so the full listing could come back without any test failing. I ran a probe on master with and without this diff (4 partitions, one commit each, COW and MOR). With a start before the first commit and the metadata table disabled for the read, master lists the table root twice and every partition directory once. With the diff there is no `listStatus` outside `.hoodie`, and the row counts match master in every case. That gives a simple guard, which could go in the same test class: - After the writes, `RecordingLocalFileSystem.register(spark.sparkContext.hadoopConfiguration)` and `reset()`, with `unregister` in teardown. The class is in the `hudi-hadoop-common` tests since #20080, so this needs a rebase. - Run the incremental read with `hoodie.metadata.enable=false` and a start of `000` (as the test already uses) or `earliest` plus an end commit. That is the case where master lists the whole table. - Assert that an `open` of a data file was recorded, so the recorder is known to be live, then assert there is no `listStatus` call outside `.hoodie`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
