HeartSaVioR commented on a change in pull request #28975:
URL: https://github.com/apache/spark/pull/28975#discussion_r449729609
##########
File path:
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/SymmetricHashJoinStateManager.scala
##########
@@ -451,10 +451,25 @@ class SymmetricHashJoinStateManager(
}
private trait KeyWithIndexToValueRowConverter {
+ /** Defines the schema of the value row (the value side of K-V in state
store). */
def valueAttributes: Seq[Attribute]
+ /**
+ * Convert the value row to (actual value, match) pair.
+ *
+ * NOTE: implementations should ensure the result row is NOT reused during
execution, as
+ * caller may use the value to store without copy().
Review comment:
The disadvantage for the approach is, given the contract is applied on
the interface level, row from format V1 now would get unnecessary copy whereas
it doesn't need to. It only applies to the evicted rows and ideally end users
would need to go with format V2 (to address correctness on left/right outer
join), hope it is acceptable.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
