juliendurand opened a new issue, #10853:
URL: https://github.com/apache/arrow-rs/issues/10853
### Is your feature request related to a problem or challenge?
Arrow-rs assumes `std::fs::File` for I/O, creating hard barriers in:
1. **WASM**: `wasm32-unknown-unknown` has no filesystem; CSV readers must
accept `&[u8]` or delegate to JavaScript
2. **Embedded systems**: No `std`; can't use Arrow
3. **Custom backends**: S3, memory-mapped, streaming protocols require
duplicated parsing logic
4. **In-memory buffers**: Even CPU-based workflows often hold data in
memory, not files
Currently, workarounds require:
- Arrow-WASM (JavaScript bindings; doesn't help Rust in WASM)
- Custom CSV parsing (duplicating Arrow's logic)
- External fork (maintenance burden)
### Describe the solution you'd like
**Introduce optional I/O abstraction:**
- Add `ArrowRead`, `ArrowWrite`, `ArrowSeek` traits
- Implement for `std::fs::File`
- CSV reader and IPC modules adaptations to use trait
- Feature gate: pluggable-io (opt-in, zero cost when disabled)
- Default implementation: wraps std::fs::File (backward compatible)
**Core traits:**
```rust
pub trait ArrowRead: Send {
fn read(&mut self, buf: &mut [u8]) -> io::Result<usize>;
}
pub trait ArrowWrite: Send {
fn write(&mut self, buf: &[u8]) -> io::Result<usize>;
fn flush(&mut self) -> io::Result<()>;
}
```
**Benefits:**
- Backward compatible (default behavior unchanged)
- No performance overhead
- Aligns with modularity goals
- Enables extensions to support WASM , Embedded & Custom I/O without fork
Willing to develop a patch. Seeking early feedback before effort.
### Describe alternatives you've considered
_No response_
### Additional context
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
