GitHub user jayspomodoro edited a comment on the discussion: [C++] Footer dumps for IPC Feather/ Parquet
But I wanted to expand more on the reader side of things as well. Since before completion the files only get appended by new recordbatches, giving access to footer at any moment will give us power to read the file till a certain point (works even if the footer file is stale and also at the same moment when the file is being written on). So I was thinking in terms of giving reader two files to read - the main_file and the footer_file - through which it will be able to decipher the contents upto the mentioned recordbatches in the footer. Synchronous update of footer can simply be done by atomic rename operation. GitHub link: https://github.com/apache/arrow/discussions/46505#discussioncomment-13209115 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
