laglangyue commented on code in PR #43526:
URL: https://github.com/apache/spark/pull/43526#discussion_r1401545560
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/HDFSMetadataLog.scala:
##########
@@ -47,16 +46,16 @@ import org.apache.spark.util.ArrayImplicits._
* Note: [[HDFSMetadataLog]] doesn't support S3-like file systems as they
don't guarantee listing
* files in a directory always shows the latest files.
*/
-class HDFSMetadataLog[T <: AnyRef : ClassTag](sparkSession: SparkSession,
path: String)
- extends MetadataLog[T] with Logging {
-
- private implicit val formats = Serialization.formats(NoTypeHints)
+class HDFSMetadataLog[T <: AnyRef: ClassTag](sparkSession: SparkSession, path:
String)(
+ private final implicit val manifest: Manifest[T])
Review Comment:
for class `CompactibleFileStreamLog` and `HDFSMetadataLog`, they has `T`,
and implement method `serialize` ,`deserialize`. And `deserialize` use
`org.json4s.jackson.JacksonSerialization#read`
`def read[A](in: Reader)(implicit formats: Formats, mf: Manifest[A]): A =
...`
if we use `private implicit val manifest =
Manifest.classType[T](implicitly[ClassTag[T]].runtimeClass)`,
the manifest can get T from ClassTag by type inference.
if we use `private implicit val manifest : Manifest[T] =
Manifest.classType[T](implicitly[ClassTag[T]].runtimeClass)`
the T is will be Object, then manifest can't get `T`.
We should declare Maintest [T] in the constructor signature, or use a
variable to save Classtag and use Classtag at runtime.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
