Narendran-KT opened a new issue, #10986:
URL: https://github.com/apache/arrow-rs/issues/10986
### Describe the bug
## Description
`arrow-ipc` compression produces an invalid compression length prefix when
compiled for `wasm32`.
In `CompressionCodec::compress_to_vec`, the uncompressed length is
serialized directly from `usize`:
```rust
let uncompressed_data_len = input.len();
output.extend_from_slice(&uncompressed_data_len.to_le_bytes());
```
Since `usize` is 32-bit on `wasm32`, only 4 bytes are written. However, the
IPC compression format uses an 8-byte `i64` prefix, as indicated by
`LENGTH_OF_PREFIX_DATA = 8` and `read_uncompressed_size()`.
For example, a 132-byte buffer is written as:
```text
84 00 00 00 04 22 4d 18 ...
```
instead of:
```text
84 00 00 00 00 00 00 00 04 22 4d 18 ...
```
This causes the Arrow JS reader to fail while decompressing the RecordBatch.
In `_decompressBuffers`, the reader expects the first 8 bytes to contain the
uncompressed length:
```js
const byteBuf = new flatbuffers.ByteBuffer(
body.subarray(offset, offset + length)
);
const uncompressedLenth = bigIntToNumber(
byteBuf.readInt64(0)
);
```
Because the WASM payload contains only a 4-byte length prefix,
`readInt64(0)` reads the 4-byte length together with the first 4 bytes of the
LZ4 frame (`04 22 4d 18`) as a single `int64`. This results in an
invalid/unsafe integer conversion before `codec.decode()` is reached.
The serialized length should be explicitly represented as a 64-bit value,
for example:
```rust
let uncompressed_data_len = i64::try_from(input.len())?;
output.extend_from_slice(&uncompressed_data_len.to_le_bytes());
```
### Environment
* `arrow-ipc`: 55.2.0
* Target: `wasm32`
* Compression: LZ4 Frame
* Consumer: Arrow JavaScript IPC reader
### To Reproduce
### Reproduction Steps
1. Build `arrow-ipc 55.2.0` with LZ4 compression for `wasm32`.
2. Serialize a `RecordBatch` with LZ4 Frame compression.
3. Pass the generated IPC stream to the Arrow JS `RecordBatchReader`.
4. During `_decompressBuffers()`, `readInt64(0)` reads the compression
prefix.
5. The 4-byte `wasm32` length prefix is combined with the first 4 bytes of
the LZ4 frame, causing an unsafe integer conversion error.
Example:
```text
84 00 00 00 04 22 4d 18 ...
```
Expected:
```text
84 00 00 00 00 00 00 00 04 22 4d 18 ...
```
### Expected behavior
The compression length prefix should always be 8 bytes regardless of the
target architecture.
### Additional context
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
