viirya commented on code in PR #58097:
URL: https://github.com/apache/spark/pull/58097#discussion_r3980713436
##########
sql/core/src/main/scala/org/apache/spark/sql/classic/Dataset.scala:
##########
@@ -1570,7 +1570,18 @@ class Dataset[T] private[sql](
/** @inheritdoc */
def toLocalIterator(): java.util.Iterator[T] = {
- withAction("toLocalIterator", queryExecution) { plan =>
+ // This action submits one job per output partition. A channel cannot
retain its output
+ // between those jobs, so use a separate regular plan even if
queryExecution already ran.
+ // Clone the session to keep the setting off during lazy planning and AQE
replanning without
+ // changing the caller's session or its other actions.
+ val iteratorSession = SparkSession.getOrCloneSessionWithConfigsOff(
Review Comment:
You are right—the previous guards missed the Python entry points, and
`DeserializeToObjectExec` is not a reliable API boundary because optimization
can eliminate it.
Fixed in `022f26af749` with a shared `QueryExecution.withRegularShuffle`
helper applied at the actual exits:
- JVM Dataset RDD and `toLocalIterator`
- `javaToPython` and `toPythonIterator`
- checkpoint / localCheckpoint
- `OpenCursorExec`
When the feature is enabled, these paths use a separate regular-shuffle
plan, including after the original Dataset has already executed. The cloned
session keeps pipelining disabled during lazy planning and AQE without changing
the caller's setting or original plan. Python iterator execution tracking also
uses the fallback QueryExecution.
I did not change generic `SparkPlan.execute()` or `QueryExecution.toRdd`;
supported SQL execution also uses those paths. The existing plan-node guard
remains conservative, but these API exits no longer depend on it.
Added AQE-on/off regressions for typed RDD exports, RDD-backed DataFrames,
lazy checkpoints, and cursors. The PySpark tests exercise coalesce,
reduceByKey, and iterator execution with and without prefetch, including after
`collect`. The iterator tests confirm that four partition jobs evaluate 1,000
producer rows once per action. All pass.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
