dwsmith1983 commented on code in PR #5615:
URL: https://github.com/apache/datafusion-comet/pull/5615#discussion_r3921091580
##########
spark/src/main/scala/org/apache/spark/sql/comet/operators.scala:
##########
@@ -336,7 +412,10 @@ private[comet] object NativeScanPlanDataInjector extends
PlanDataInjector {
op.getNativeScan.hasCommon &&
!op.getNativeScan.hasFilePartition
- override def getKey(op: Operator): Option[String] =
Some(sourceKey(op.getNativeScan.getCommon))
+ override def getKey(op: Operator): Option[String] = {
+ val common = op.getNativeScan.getCommon
+ Some(PlanDataInjector.cachedOrCompute(keyCache, common)(sourceKey(common)))
Review Comment:
This was the right call, thanks. source_key now rides in the NativeScan
proto, derived once on the driver, and the keyCache is deleted. Reading the
transported key measures 0.07us against 0.4 to 0.55ms per derivation under
churn. It reaches the shuffle writer too via childNativeOp, and the executor
keeps a derivation fallback only for plans built without the field. The scalar
subquery caveat is handled by pinning the finalized bytes on each prepared
entry and honoring hits only on byte equality.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
