[GitHub] spark pull request #20946: [SPARK-23565] [SQL] New error message for structu...
Github user xuanyuanking commented on a diff in the pull request: https://github.com/apache/spark/pull/20946#discussion_r183447816 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/OffsetSeq.scala --- @@ -39,7 +39,9 @@ case class OffsetSeq(offsets: Seq[Option[Offset]], metadata: Option[OffsetSeqMet * cannot be serialized). */ def toStreamProgress(sources: Seq[BaseStreamingSource]): StreamProgress = { -assert(sources.size == offsets.size) +assert(sources.size == offsets.size, s"There are [${offsets.size}] sources in the " + + s"checkpoint offsets and now there are [${sources.size}] sources requested by the query. " + --- End diff -- nit: extra blank in the end --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #20946: [SPARK-23565] [SQL] New error message for structu...
Github user xuanyuanking commented on a diff in the pull request: https://github.com/apache/spark/pull/20946#discussion_r183447988 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/streaming/OffsetSeqLogSuite.scala --- @@ -125,6 +125,19 @@ class OffsetSeqLogSuite extends SparkFunSuite with SharedSQLContext { assert(offsetSeq.metadata === Some(OffsetSeqMetadata(0L, 1480981499528L))) } + test("assertion that number of checkpoint offsets match number of sources") { --- End diff -- maybe no need add UT for the log change. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #20946: [SPARK-23565] [SQL] New error message for structu...
GitHub user patrickmcgloin opened a pull request: https://github.com/apache/spark/pull/20946 [SPARK-23565] [SQL] New error message for structured streaming sources assertion ## What changes were proposed in this pull request? A more informative message to tell you why a structured streaming query cannot continue if you have added more sources, than there are in the existing checkpoint offsets. ## How was this patch tested? I added a Unit Test. You can merge this pull request into a Git repository by running: $ git pull https://github.com/patrickmcgloin/spark master Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/20946.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #20946 commit 58576fabf7b3ba93c177c78356bb17492e8b3d80 Author: Patrick McGloinDate: 2018-03-30T14:09:25Z SPARK-23565: Improved error message for when the number of sources for a query changes --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org