HeartSaVioR commented on code in PR #44961:
URL: https://github.com/apache/spark/pull/44961#discussion_r1490354597
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDBStateStoreProvider.scala:
##########
@@ -65,6 +65,43 @@ private[sql] class RocksDBStateStoreProvider
value
}
+ override def valuesIterator(key: UnsafeRow, colFamilyName: String):
Iterator[UnsafeRow] = {
+ verify(key != null, "Key cannot be null")
+ verify(encoder.supportsMultipleValuesPerKey, "valuesIterator requires a
encoder " +
+ "that supports multiple values for a single key.")
+ val valueIterator =
encoder.decodeValues(rocksDB.get(encoder.encodeKey(key), colFamilyName))
+
+ if (!isValidated && valueIterator.nonEmpty) {
Review Comment:
Discussed offline. There had been a couple of cases which inspection of
state row in underneath were helpful on debugging. Maybe we could remove this
once we improve state data source reader to produce state key-value as binary,
but before that let's keep it as it is for existing codebase.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
