LuciferYang commented on code in PR #58467:
URL: https://github.com/apache/spark/pull/58467#discussion_r3915401181
##########
core/src/main/scala/org/apache/spark/status/protobuf/KVStoreProtobufSerializer.scala:
##########
@@ -51,6 +54,16 @@ private[spark] object KVStoreProtobufSerializer {
}.toMap
}
- def getSerializer(klass: Class[_]): Option[ProtobufSerDe[Any]] =
- serializerMap.get(klass)
+ private[this] val missedClasses = ConcurrentHashMap.newKeySet[Class[_]]()
+
+ private[protobuf] def resetMissedClassesForTesting(): Unit =
missedClasses.clear()
+
+ def getSerializer(klass: Class[_]): Option[ProtobufSerDe[Any]] = {
+ val serializer = serializerMap.get(klass)
+ if (serializer.isEmpty && missedClasses.add(klass)) {
Review Comment:
This warning fires wider than "missing registration": internal store values
go through the same serializer, not just app types. The metadata classes are
JSON by design -- every driver with `spark.ui.store.path` set warns for them on
start -- and the backend's `TypeAliases` warns both when a type is first
registered (serialize, so a brand-new store hits it too) and when a store with
written entities is reopened (deserialize). Healthy deployments log warnings
the operator cannot act on.
I'd skip the KVStore bookkeeping values: a Set of by-design-JSON classes in
the object (the two metadata classes plus
`RocksDB.TypeAliases`/`LevelDB.TypeAliases`), no log on hit. Separately,
`LogInfo` in the SHS listing store also has no serde; that one is a real gap,
so either add a serde for it or accept the warning, to be decided in a separate
issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
