nsivabalan commented on issue #14463: URL: https://github.com/apache/hudi/issues/14463#issuecomment-4795165106
This appears to be fixed on current `master`. The executor-side listing in the original stack trace is no longer reachable from the MOR append path: - `HoodieLogFile.rollOver(String)` ([HoodieLogFile.java:181-188](https://github.com/apache/hudi/blob/master/hudi-common/src/main/java/org/apache/hudi/common/model/HoodieLogFile.java#L181-L188)) no longer calls `FSUtils.computeNextLogVersion`; it just increments `logVersion + 1` in memory. - `HoodieWriteHandle.createLogWriter` ([HoodieWriteHandle.java:288-321](https://github.com/apache/hudi/blob/master/hudi-client/hudi-client-common/src/main/java/org/apache/hudi/io/HoodieWriteHandle.java#L288-L321)) avoids listing: - For table version >= 8, it passes `writeToken` and lets `HoodieLogFormat$WriterBuilder.build()` ([HoodieLogFormat.java:245-265](https://github.com/apache/hudi/blob/master/hudi-common/src/main/java/org/apache/hudi/common/table/log/HoodieLogFormat.java#L245-L265)) take the `useBaseVersion` shortcut (uses `LOGFILE_BASE_VERSION`; uniqueness comes from the writeToken in the filename; existence is verified via a single `storage.exists(...)` in `HoodieLogFormatWriter.getOutputStream`). Added in b31c858 (HUDI-8210, #12206). - For pre-v8 tables, the log version is read from the driver-supplied `FileSlice.getLatestLogFile()`. The `FSUtils.getLatestLogVersion` → `getAllLogFiles` path remains only as a defensive fallback in the builder when neither path applies. If anyone is still seeing executor-side `listStatus` in MOR append on master, a fresh stack trace would be helpful — otherwise this can likely be closed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
