otterc opened a new pull request #30433: URL: https://github.com/apache/spark/pull/30433
### What changes were proposed in this pull request? This change contains fixs for bugs in `RemoteBlockPushResolver` where the number of chunks in meta file and index file are inconsistent due to exceptions while writing to either index file or meta file. This java class was introduced in https://github.com/apache/spark/pull/30062. 1. If the writing to index file fails, the position of meta file is not reset. This means that the number of chunks in meta file is inconsistent with index file. 2. We changed to using `RandomAccessFile` for meta/index files. However, during the exception handling while writing to these file, we just set the pointer to the start position. If the files are closed just after this then it doesn't get rid of any the extra bytes written to it. ### Why are the changes needed? This fix is needed for the bugs mentioned above. 1. Moved writing to meta file after index file. This fixes the issue because if there is an exception writing to meta file, then the index file position is already reset. With this change, if there is an exception writing to index file, then none of the files are effectively updated and same is true vice-versa. 2. Setting the lengths of meta/index files when the files are closed. ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? I have added unit tests for both of these bugs. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
