David Mollitor created SPARK-59056:
--------------------------------------
Summary: Pre-size HashSets in ParquetFilters IN/InSet predicate
pushdown
Key: SPARK-59056
URL: https://issues.apache.org/jira/browse/SPARK-59056
Project: Spark
Issue Type: Improvement
Components: SQL
Affects Versions: 4.1.0
Reporter: David Mollitor
h2. Summary
{{ParquetFilters.makeInPredicate}} builds a {{java.util.HashSet}} for every
{{IN}} predicate pushed down to Parquet, without an initial capacity. The final
element count is exactly {{values.length}} and is known before the set is
populated, so the set can be pre-sized to avoid internal rehashing and the
intermediate bucket-array allocations.
h2. Details
{{makeInPredicate}} has 12 near-identical sites of the form:
{code:scala}
val set = new HashSet[T]()
for (value <- values) {
set.add(...) // one element per entry in `values`
}
FilterApi.in(column(n), set)
{code}
Each {{HashSet}} starts at the default capacity (16) and rehashes as it grows
toward
{{values.length}} elements. This set is only built when the {{IN}} list size
exceeds
{{spark.sql.parquet.pushdown.inFilterThreshold}} (default 10); smaller lists
take a per-element equality OR-chain and never build a {{{}HashSet{}}}. So by
construction the set is always populated with more than the threshold's worth
of values, and for larger lists (dozens to thousands – generated {{IN}} lists,
dynamic filters) the default-capacity set rehashes several times per
pushed-down predicate, on every Parquet file split scanned.
h2. Proposed change
Pre-size each set from the known element count using Guava's
{{Sets.newHashSetWithExpectedSize(values.length)}} (already a Spark dependency;
consistent with the existing {{Maps.newHashMapWithExpectedSize}} usage in
{{{}catalyst{}}}). The Guava helper is used rather than {{new
HashSet(values.length)}} because {{{}java.util.HashSet{}}}'s int constructor
treats the argument as bucket capacity, not expected size, so the naive form
would still rehash at ~75% load.
The change is capacity-only and behavior-preserving – the resulting predicate
semantics are identical.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
