juliendurand commented on issue #10853: URL: https://github.com/apache/arrow-rs/issues/10853#issuecomment-5435390580
Thanks for clarifying that arrow-rs is already using std::io traits. Let me try to clarify our specific use case and refine our proposal. **Use Case:** We're building a browser-based (serverless) data processing application using WASM (wasm32-unknown-unknown). The typical workflow is: 1. User uploads a CSV file in a browser environment ; 2. Parse CSV → encode to Arrow internal representation ; 3. Perform domain specific data operations (calculations, transformations, analysis) ; 4. Export result in Arrow format (Parquet or IPC) ; 5. Load into standard data science tools for further analysis (Python/R/Polars). **Current blocker:** wasm32-unknown-unknown is `no_std`; even though Arrow uses `std::io::Read` trait, the trait itself requires `std`. **Current workaround:** We maintain a custom CSV reader as a workaround, duplicating about 2k lines of Arrow's logic. **Proposal:** We'd like to propose a feature gate (`csv-no_std`, default-disabled) that: - Accepts pre-loaded `&[u8]` CSV buffers (instead of File handles) - Works in wasm32-unknown-unknown (no_std compatible) - Has zero impact on existing code paths (opt-in feature) - Reuses Arrow's schema inference and type detection logic I have analyzed the Arrow-rs source. Arrow indeed already respects the trait boundary (`std::io::Read`). Our refined proposal is to add a no_std code path that works with bytes directly, without refactoring existing code.The changes would be limited: 1. Add `std` feature flag, enabled by default ; 2. Gate 4 locations where `std` is direct ( ~40 lines, conditional compilation) ; 3. Add `BytesReader` and `BytesWriter` structs (~200 new lines, isolated to no_std path) ; 4. New API `ReaderBuilder::build_bytes()` (No existing API changes) This proposal should not introduce any breaking change for existing users. These show one way to maintain default functionality while extending into constrained environments. We're willing to: 1. Develop the patch 2. Iterate based on feedback 3. Maintain the no_std path (we have direct interest in it working) 4. Add integration tests for WASM compilation Would it make sense for Arrow-rs to explore: 1. A feature gate for `std` (default-enabled, zero-cost for existing users) 2. A `BytesReader` implementation for no_std CSV reading 3. Tests and documentation for the no_std path We're happy to develop a proof-of-concept PR to explore feasibility. Would that be useful? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
