This is an automated email from the ASF dual-hosted git repository.
zeroshade pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/arrow-go.git
The following commit(s) were added to refs/heads/main by this push:
new 7c0238a9 fix(parquet): free excluded stats bytes in
ApplyStatSizeLimits (#885)
7c0238a9 is described below
commit 7c0238a913753ec23700760e06ecc3377c49b863
Author: Mahdi Dibaiee <[email protected]>
AuthorDate: Thu Jul 2 20:08:17 2026 +0100
fix(parquet): free excluded stats bytes in ApplyStatSizeLimits (#885)
### Rationale for this change
The current implementation has a memory leak where even if limits are
set for MinMax statistics, despite not using MinMax after they bypass
the limit, the code still retains a copy of the bytes, not allowing GC
to clean them up and accumulating memory.
In our use case we have noticed this causes memory accumulation over
time and leads to OOMs. After this fix we do not experience the OOMs
anymore.
### What changes are included in this PR?
Discard unused Min/Max stats bytes
### Are these changes tested?
Yes I have tested them in our own codebase
https://github.com/estuary/connectors
### Are there any user-facing changes?
No
---
parquet/metadata/statistics.go | 9 +++++++++
1 file changed, 9 insertions(+)
diff --git a/parquet/metadata/statistics.go b/parquet/metadata/statistics.go
index e7d8cfcd..9e568403 100644
--- a/parquet/metadata/statistics.go
+++ b/parquet/metadata/statistics.go
@@ -71,12 +71,21 @@ type EncodedStatistics struct {
// the rationale is that some engines may use the minimum value in the page
// as the true minimum for aggregations and there is no way to mark that
// a value has been truncated and is a lower bound and not in the page
+//
+// Excluded values are also cleared here (not just flagged via Has*), since
this
+// EncodedStatistics is stored as-is in the column chunk's persistent metadata
by
+// ColumnChunkMetaDataBuilder.SetStats and retained until the file is closed.
Leaving Max/Min
+// populated after excluding them means the writer keeps a live reference to a
full copy of every
+// oversized value, once per row group, for the file's entire lifetime --
Has*=false alone only
+// affects what gets serialized, not what's retained in memory in the meantime.
func (e *EncodedStatistics) ApplyStatSizeLimits(length int) {
if len(e.Max) > length {
e.HasMax = false
+ e.Max = nil
}
if len(e.Min) > length {
e.HasMin = false
+ e.Min = nil
}
}
