HeartSaVioR commented on a change in pull request #28975: URL: https://github.com/apache/spark/pull/28975#discussion_r449685443
########## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/SymmetricHashJoinStateManager.scala ########## @@ -451,10 +451,25 @@ class SymmetricHashJoinStateManager( } private trait KeyWithIndexToValueRowConverter { + /** Defines the schema of the value row (the value side of K-V in state store). */ def valueAttributes: Seq[Attribute] + /** + * Convert the value row to (actual value, match) pair. + * + * NOTE: implementations should ensure the result row is NOT reused during execution, as + * caller may use the value to store without copy(). Review comment: Yes. KeyWithIndexToValueRowConverterFormatV1 simply uses the row state store will provide, which is expected to be unique instance (as we copy it when putting to state store). I've had time to go through callers, and I'm still hesitated to let callers deal with copy, because the row to take care is inside of the ValueAndMatchPair, and some of current callers are even re-packaging it to other class, e.g. KeyWithIndexAndValue for getAll(), so not only direct callers should take care of it but caller of the caller may have to take care of it. It would bring too much complication. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org