Hi, Why does ContinuousExecution.commit serialize an offset to JSON format just before deserializing it back (from JSON to an offset)? [1]
val offset = sources(0).deserializeOffset(offsetLog.get(epoch).get.offsets(0).get.json) [1] https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/continuous/ContinuousExecution.scala#L341 Pozdrawiam, Jacek Laskowski ---- https://about.me/JacekLaskowski The Internals of Spark SQL https://bit.ly/spark-sql-internals The Internals of Spark Structured Streaming https://bit.ly/spark-structured-streaming The Internals of Apache Kafka https://bit.ly/apache-kafka-internals Follow me at https://twitter.com/jaceklaskowski
