sunchao commented on code in PR #7190:
URL: https://github.com/apache/datafusion-comet/pull/7190#discussion_r4392974666
##########
spark/src/main/scala/org/apache/comet/CometExecIterator.scala:
##########
@@ -449,6 +449,17 @@ object CometExecIterator extends Logging {
}
}
+ /** Called by the native planner before reducing keys for an eligible semi
join. */
+ def hasOtherExecutionMemory(context: TaskContext): Boolean = {
+ if (!CometSparkSessionExtensions.isOffHeapEnabled(SparkEnv.get.conf)) {
Review Comment:
Agreed, the local fair-limit refusal is specific to `fair_unified`. I
narrowed the guard to that pool in 39e82b337c. Under `greedy_unified`, a
resident JVM consumer no longer disables right-key reduction. The callback
comment and user guide now state the scope.
I parameterized the existing admission test over both pools, with and
without resident JVM memory, using the real JNI callback. The expanded test
fails on the previous guard for the greedy/resident-memory case and passes with
this fix. All four focused reduction tests, strict Spark/Delta compilation,
formatting and syntactic lint passed.
Native code is unchanged; these runs used the existing native library with
the temporary dependency lock documented in the PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
