hudi-agent commented on code in PR #20220:
URL: https://github.com/apache/hudi/pull/20220#discussion_r4238417454
##########
hudi-cli/src/main/java/org/apache/hudi/cli/commands/ExportCommand.java:
##########
@@ -176,17 +289,34 @@ private int copyArchivedInstants(List<StoragePathInfo>
pathInfoList,
log.error("Could not load metadata for action {} at instant
time {}", action, instantTime);
continue;
}
- final String outPath = localFolder + StoragePath.SEPARATOR +
instantTime + "." + action;
- writeToFile(outPath, HoodieAvroUtils.avroToJson(metadata, true));
+ byte[] json = HoodieAvroUtils.avroToJson(metadata, true);
+ if (descending) {
+ fileEntries.add(new String[] {instantTime, action});
+ fileJsons.add(json);
Review Comment:
🤖 non-blocking: With `--desc` this now reads the whole archive file and
keeps the JSON of every completed entry in it, even for `--limit 1`. A v6
archive file written by 0.x on HDFS grew by appends until the log size
threshold (hundreds of MB), so that could be a lot of CLI heap. Would it make
sense to keep only the last `limit - copyCount` entries in a bounded deque?
When no limit is set, the entries could be written straight away, since the
order they're written in doesn't change which files get exported.
<sub><i>⚠️ AI-generated; verify before applying. React 👍/👎 to flag
quality.</i></sub>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
