mbutrovich opened a new issue, #27277: URL: https://github.com/apache/datafusion/issues/27277
**Describe the bug** [`ScalarValue::to_array_of_size`](https://github.com/apache/datafusion/blob/14b3edd9a350b983fae6d2f194df1a3ae343ba15/datafusion/common/src/scalar/mod.rs#L3677-L3686) returns a `Result`, but for a `Utf8` or `Binary` scalar it panics when the value's length times `size` is more than `i32::MAX` bytes. The [`Utf8`](https://github.com/apache/datafusion/blob/14b3edd9a350b983fae6d2f194df1a3ae343ba15/datafusion/common/src/scalar/mod.rs#L3788-L3791) and [`Binary`](https://github.com/apache/datafusion/blob/14b3edd9a350b983fae6d2f194df1a3ae343ba15/datafusion/common/src/scalar/mod.rs#L3805-L3810) arms call `GenericByteArray::new_repeated`, which builds its offsets with [`OffsetBuffer::from_repeated_length`](https://github.com/apache/arrow-rs/blob/16ac7c8878c470acd509a12f5db0524a92ee71e0/arrow-buffer/src/buffer/offset.rs#L177-L179), and that panics on offset overflow. The [`Utf8View`](https://github.com/apache/datafusion/blob/14b3edd9a350b983fae6d2f194df1a3ae343ba15/datafusion/common/src/scala r/mod.rs#L3792-L3800) and `BinaryView` arms use the fallible `try_append_value_n` instead. A query reaches it: a projection turns a scalar result into an array of the batch's rows through `ColumnarValue::into_array_of_size`, which calls `to_array_of_size`. The panic happens before the value bytes are allocated, so it needs no memory beyond the scalar. **To Reproduce** ```rust use datafusion_common::ScalarValue; fn main() { // 2^20 bytes in each of 2^11 rows is 2^31 bytes, one past i32::MAX let value = "a".repeat(1 << 20); let view = ScalarValue::Utf8View(Some(value.clone())).to_array_of_size(1 << 11); println!("Utf8View: {:?}", view.map(|a| a.len())); // Utf8View: Ok(2048) let utf8 = ScalarValue::Utf8(Some(value)).to_array_of_size(1 << 11); // panics: offset overflow: 2147483648 does not fit in i32 println!("Utf8: {:?}", utf8.map(|a| a.len())); } ``` The same panic from SQL, through `ProjectionExec`: ```rust use datafusion::prelude::SessionContext; #[tokio::main] async fn main() { let ctx = SessionContext::new(); let r = ctx .sql("SELECT repeat('a', 1048576) AS s FROM generate_series(1, 2048)") .await .unwrap() .collect() .await; // panics: offset overflow: 2147483648 does not fit in i32 println!("{:?}", r.map(|b| b.iter().map(|b| b.num_rows()).sum::<usize>())); } ``` The backtrace of the second runs through `Projector::project_batch`, `ColumnarValue::into_array_of_size`, `ScalarValue::to_array_of_size` and `StringArray::new_repeated`. **Expected behavior** `to_array_of_size` returns an error, as it does for `Utf8View`. The query fails with that error instead of panicking. `GenericByteArray::new_repeated` has no fallible form, so the `Utf8` and `Binary` arms could check `value.len().checked_mul(size)` against `i32::MAX` before calling it. **Additional context** Property-based tests found this on `main` at 14b3edd9a3, when a test broadcast a scalar result of `repeat` to the call's rows. Both reproductions panic on `main` at 14b3edd9a3, which builds on arrow 60.1.0. The `LargeUtf8` and `LargeBinary` arms have `i64` offsets, so for them the size limit is the allocation, not the offsets. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
