This is an automated email from the ASF dual-hosted git repository.
alamb pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/arrow-rs.git
The following commit(s) were added to refs/heads/main by this push:
new 2a77e106fd feat(ipc): add with_skip_validation to StreamDecoder
(#9749)
2a77e106fd is described below
commit 2a77e106fd762692c30540a577a08a9825e22245
Author: pantShrey <[email protected]>
AuthorDate: Fri Apr 24 01:33:43 2026 +0530
feat(ipc): add with_skip_validation to StreamDecoder (#9749)
# Which issue does this PR close?
N/A - this is a small API parity fix.
# Rationale for this change
`StreamReader` and `FileDecoder` both expose `with_skip_validation` but
`StreamDecoder` does not, despite having the same internal
`skip_validation` field.
Motivation: DataFusion's spill infrastructure is being refactored to use
`StreamDecoder` for reading spill files via a pluggable backend trait
([apache/datafusion#21215](https://github.com/apache/datafusion/issues/21215)).
Since DataFusion controls what it writes it can trust its own IPC
output, and needs this method to maintain the same performance
characteristic as the current StreamReader-based implementation
# What changes are included in this PR?
Adds `with_skip_validation` to `StreamDecoder` mirroring the existing
implementation on `FileDecoder`
# Are these changes tested?
The existing StreamDecoder tests cover the decoding path. No new tests
added as this method sets an internal flag already tested via
`FileDecoder` and `StreamReader`.
# Are there any user-facing changes?
Yes, `with_skip_validation` is now available on `StreamDecoder`. It is
marked unsafe with the same safety requirements as
`FileDecoder::with_skip_validation`.
---
arrow-ipc/src/reader/stream.rs | 17 +++++++++++++++--
1 file changed, 15 insertions(+), 2 deletions(-)
diff --git a/arrow-ipc/src/reader/stream.rs b/arrow-ipc/src/reader/stream.rs
index d0d833b471..bfecf7b6ff 100644
--- a/arrow-ipc/src/reader/stream.rs
+++ b/arrow-ipc/src/reader/stream.rs
@@ -45,9 +45,8 @@ pub struct StreamDecoder {
require_alignment: bool,
/// Should validation be skipped when reading data? Defaults to false.
///
- /// See [`FileDecoder::with_skip_validation`] for details.
+ /// See [`StreamDecoder::with_skip_validation`] for details.
///
- /// [`FileDecoder::with_skip_validation`]:
crate::reader::FileDecoder::with_skip_validation
skip_validation: UnsafeFlag,
}
@@ -114,6 +113,20 @@ impl StreamDecoder {
self.schema.as_ref().map(|schema| schema.clone())
}
+ /// Specifies if validation should be skipped when reading data (defaults
to `false`)
+ ///
+ /// # Safety
+ ///
+ /// This flag must only be set to `true` when you trust the input data and
are
+ /// sure the data you are reading is valid Arrow IPC stream data, otherwise
+ /// undefined behavior may result.
+ ///
+ /// For example, DataFusion uses this when reading spill files it wrote
itself.
+ pub unsafe fn with_skip_validation(mut self, skip_validation: bool) ->
Self {
+ unsafe { self.skip_validation.set(skip_validation) };
+ self
+ }
+
/// Try to read the next [`RecordBatch`] from the provided [`Buffer`]
///
/// [`Buffer::advance`] will be called on `buffer` for any consumed bytes.
