[
https://issues.apache.org/jira/browse/SPARK-61251?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-61251:
-----------------------------------
Labels: pull-request-available (was: )
> Sort-merge join and window frames leave spill file readers open when they
> stop reading a spilled buffer early
> -------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-61251
> URL: https://issues.apache.org/jira/browse/SPARK-61251
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 4.4.0
> Reporter: Luka Zdravic
> Priority: Minor
> Labels: pull-request-available
>
> SPARK-61248 fixed this for the sort-merge ASOF join. Other code that reads
> an ExternalAppendOnlyUnsafeRowArray has the same problem: it makes a new
> iterator over the buffer for each row and stops before the end. Once the
> buffer has spilled, that iterator keeps a reader open on every spill file it
> has not read, until the task ends. With L rows and S spill files, a task
> keeps up to L * S files open and can fail with "Too many open files".
> Places found by reading the code:
> - Sort-merge LeftSemi, LeftAnti and ExistenceJoin with a residual condition
> stop at the first passing buffered row, in SortMergeJoinEvaluatorFactory and
> in the generated codegenSemi, codegenAnti and codegenExistence loops.
> - With spark.sql.join.splitStreamedSideJoinCondition on, the generated
> sort-merge join makes the iterator before the streamed-side guard, and the
> guard skips it unread.
> - UnboundedFollowingWindowFunctionFrame.write reads one row and drops the
> iterator when the frame's lower bound does not move, for example a RANGE
> frame over duplicate order keys.
> - WindowSegmentTree.buildBlockLevels (spark.sql.window.segmentTree.enabled)
> reads one block and drops the iterator.
> Example: SELECT * FROM l WHERE EXISTS (SELECT 1 FROM r WHERE r.k = l.k AND
> r.v > l.v), where the buffered rows of a key spill.
> Fix: close the iterator when the code stops reading it. In generated Java,
> check instanceof
> java.io.Closeable. In the generated sort-merge join, make the iterator
> after the streamed-side guard.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]