Repository: spark Updated Branches: refs/heads/master 73c8ea84a -> c3002c4a6
[SPARK-4294][Streaming] UnionDStream stream should express the requirements in the same way as TransformedDStream In class TransformedDStream: ```scala require(parents.length > 0, "List of DStreams to transform is empty") require(parents.map(.ssc).distinct.size == 1, "Some of the DStreams have different contexts") require(parents.map(.slideDuration).distinct.size == 1, "Some of the DStreams have different slide durations") ``` In class UnionDStream: ```scala if (parents.length == 0) { throw new IllegalArgumentException("Empty array of parents") } if (parents.map(.ssc).distinct.size > 1) { throw new IllegalArgumentException("Array of parents have different StreamingContexts") } if (parents.map(.slideDuration).distinct.size > 1) { throw new IllegalArgumentException("Array of parents have different slide times") } ``` The function is the same, but the realization is not. I think they shoule be the same. Author: Yadong Qi <qiyadong2...@gmail.com> Closes #3152 from watermen/bug-fix1 and squashes the following commits: ed66db6 [Yadong Qi] Change transform to union b6b3b8b [Yadong Qi] The same function should have the same realization. Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/c3002c4a Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/c3002c4a Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/c3002c4a Branch: refs/heads/master Commit: c3002c4a61c4fc5b966aa384c41c3cba33de0aa6 Parents: 73c8ea8 Author: Yadong Qi <qiyadong2...@gmail.com> Authored: Wed Nov 19 15:53:06 2014 -0800 Committer: Tathagata Das <tathagata.das1...@gmail.com> Committed: Wed Nov 19 15:53:06 2014 -0800 ---------------------------------------------------------------------- .../spark/streaming/dstream/UnionDStream.scala | 15 ++++----------- 1 file changed, 4 insertions(+), 11 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/spark/blob/c3002c4a/streaming/src/main/scala/org/apache/spark/streaming/dstream/UnionDStream.scala ---------------------------------------------------------------------- diff --git a/streaming/src/main/scala/org/apache/spark/streaming/dstream/UnionDStream.scala b/streaming/src/main/scala/org/apache/spark/streaming/dstream/UnionDStream.scala index 57429a1..abbc40b 100644 --- a/streaming/src/main/scala/org/apache/spark/streaming/dstream/UnionDStream.scala +++ b/streaming/src/main/scala/org/apache/spark/streaming/dstream/UnionDStream.scala @@ -28,17 +28,10 @@ private[streaming] class UnionDStream[T: ClassTag](parents: Array[DStream[T]]) extends DStream[T](parents.head.ssc) { - if (parents.length == 0) { - throw new IllegalArgumentException("Empty array of parents") - } - - if (parents.map(_.ssc).distinct.size > 1) { - throw new IllegalArgumentException("Array of parents have different StreamingContexts") - } - - if (parents.map(_.slideDuration).distinct.size > 1) { - throw new IllegalArgumentException("Array of parents have different slide times") - } + require(parents.length > 0, "List of DStreams to union is empty") + require(parents.map(_.ssc).distinct.size == 1, "Some of the DStreams have different contexts") + require(parents.map(_.slideDuration).distinct.size == 1, + "Some of the DStreams have different slide durations") override def dependencies = parents.toList --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org