adamreeve commented on issue #46930: URL: https://github.com/apache/arrow/issues/46930#issuecomment-3017651383
Hi, I just wanted to add that I'm also interested in this and a while ago I made a branch that implemented this in order to do some benchmarking, but haven't made any further progress: https://github.com/apache/arrow/compare/apache-arrow-19.0.0...adamreeve:arrow:s3-crt Unfortunately it's not super straightforward to add this feature because it uses the separate "CRT" S3 library, which has nearly the exact same API as the standard S3 library but has different configuration options, and doesn't share any base classes to allow easily abstracting over the different libraries. It might be possible for Arrow to completely switch to the CRT library, but from what I understand this has very different performance characteristics and can cause performance regressions in some cases, so we might want to make use of the CRT library optional. In my branch I just copy-pasted most of the S3 file system implementation to get something working, but if this is something we want to add to Arrow without switching completely to the CRT library, we'd want to come up with a better approach. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@arrow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
