This is an automated email from the ASF dual-hosted git repository.
alamb pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/arrow-rs.git
The following commit(s) were added to refs/heads/main by this push:
new d03571bd23 feat(ipc): Remove per-message flush in IPC writer hot path
(#9763)
d03571bd23 is described below
commit d03571bd23fa0a02faa1cad7f52c9e708976ef05
Author: pchintar <[email protected]>
AuthorDate: Tue Apr 21 14:44:02 2026 -0400
feat(ipc): Remove per-message flush in IPC writer hot path (#9763)
# Which issue does this PR close?
- Closes #9762 .
# Rationale for this change
Currently, `flush()` is called unconditionally in `write_body_buffers`
and `write_continuation` - both executed per batch. This forces
per-batch syscalls, breaks write coalescing, and adds unnecessary
overhead. These flushes are not required for correctness (IPC boundaries
are length-prefixed, no durability guarantees).
# What changes are included in this PR?
- Remove `flush()` from `write_body_buffers` and `write_continuation`
- Add `self.writer.flush()?` to `StreamWriter::finish()` (missing;
`FileWriter::finish()` already has it)
# Are these changes tested?
Yes the changes were tested successfully by running:
```bash
cargo test -p arrow-ipc
cargo fmt --all
cargo clippy --all-targets --all-features -- -D warnings
```
# Are there any user-facing changes?
No, There are no changes made to any Public APIs
---
arrow-ipc/src/writer.rs | 4 +---
1 file changed, 1 insertion(+), 3 deletions(-)
diff --git a/arrow-ipc/src/writer.rs b/arrow-ipc/src/writer.rs
index 9d3fca2938..5fc4027a8f 100644
--- a/arrow-ipc/src/writer.rs
+++ b/arrow-ipc/src/writer.rs
@@ -1467,6 +1467,7 @@ impl<W: Write> StreamWriter<W> {
}
write_continuation(&mut self.writer, &self.write_options, 0)?;
+ self.writer.flush()?;
self.finished = true;
@@ -1618,7 +1619,6 @@ fn write_body_buffers<W: Write>(
writer.write_all(&PADDING[..pad_len])?;
}
- writer.flush()?;
Ok(total_len)
}
@@ -1652,8 +1652,6 @@ fn write_continuation<W: Write>(
z => panic!("Unsupported crate::MetadataVersion {z:?}"),
};
- writer.flush()?;
-
Ok(written)
}
