dosubot[bot] commented on issue #6373: URL: https://github.com/apache/opendal/issues/6373#issuecomment-3046731716
<!-- Greeting --> Hi @ronvohra! I'm [Dosu](https://go.dosu.dev/dosubot) and I’m helping the opendal team. <!-- Answer --> This is a known limitation: OpenDAL's Python bindings provide a file-like interface, but for S3, the underlying implementation does not support random-access or seekable writes, which libraries like pyarrow.ParquetWriter require. As a result, attempts to write directly to S3 using opendal.File will hang or fail silently because S3 (and OpenDAL's abstraction over it) can't support the necessary file operations for ParquetWriter to function correctly in write mode [reference](https://github.com/apache/opendal/blob/27d5677d4526519313cbce00cd28f8fbcc099642/bindings/python/src/file.rs). The recommended workaround is to write your pyarrow table to a local temporary file first, then upload that file to S3 using `operator.write`. This pattern avoids the need for random-access writes on S3 and is commonly used when working with object storage backends that don't support seekable writes [reference](https://github.com/apache/opendal/issues/5943). There is currently no direct support or workaround in OpenDAL for using opendal.File as a sink for pyarrow.ParquetWriter with S3. If you need true streaming or random-access write support for S3, it would require changes both in OpenDAL and in how pyarrow interacts with file-like objects. <!-- Dosu Comment Footer --> *To reply, just mention [@dosu](https://go.dosu.dev/dosubot).* --- <sup>How did I do? [Good](https://app.dosu.dev/response-feedback/c49ff2ac-ef1f-41bf-acb3-4f25abd3b105?feedback_type=great_response) | [Irrelevant](https://app.dosu.dev/response-feedback/c49ff2ac-ef1f-41bf-acb3-4f25abd3b105?feedback_type=irrelevant_answer) | [Incorrect](https://app.dosu.dev/response-feedback/c49ff2ac-ef1f-41bf-acb3-4f25abd3b105?feedback_type=incorrect_sources) | [Verbose](https://app.dosu.dev/response-feedback/c49ff2ac-ef1f-41bf-acb3-4f25abd3b105?feedback_type=too_verbose) | [Hallucination](https://app.dosu.dev/response-feedback/c49ff2ac-ef1f-41bf-acb3-4f25abd3b105?feedback_type=hallucination) | [Report 🐛](https://app.dosu.dev/response-feedback/c49ff2ac-ef1f-41bf-acb3-4f25abd3b105?feedback_type=bug_report) | [Other](https://app.dosu.dev/response-feedback/c49ff2ac-ef1f-41bf-acb3-4f25abd3b105?feedback_type=other)</sup> [](https://go.dosu.dev/discord-bot) [! [Share on X](https://img.shields.io/badge/X-share-black)](https://twitter.com/intent/tweet?text=%40dosu_ai%20helped%20me%20solve%20this%20issue!&url=https%3A//github.com/apache/opendal/issues/6373) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
