steveloughran commented on issue #3356: URL: https://github.com/apache/parquet-java/issues/3356#issuecomment-4089172411
problem with the vector IO buffer merging and the checksum blocks of local fs. See HADOOP-18296: setting `fs.file.checksum.verify` to false will stop this by handing down to the raw local code. (bonus benefit: vector reads handoff to java nio scatter/gather and you get maximum perfomance). 1. parquet needs a PR for this 2. someone who can understand the relevant bits of `ChecksumFileSystem` could see if it could be clever here and return the original buffers This doesn't actually surface in production because the ranges of parquet/ORC files is too big, though I think we've also been considering removing range merging from the s3a code. The issue there is correctly handling partial failures. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
