Fei Shao created SPARK-21206: -------------------------------- Summary: the window slice of Dstream is wrong Key: SPARK-21206 URL: https://issues.apache.org/jira/browse/SPARK-21206 Project: Spark Issue Type: Bug Components: DStreams Affects Versions: 2.1.0 Reporter: Fei Shao
the code is : val conf = new SparkConf().setAppName("testDstream").setMaster("local[4]") val ssc = new StreamingContext(conf, Seconds(1)) ssc.checkpoint( "path") val lines = ssc.socketTextStream("IP", PORT) lines.countByValueAndWindow( Seconds(2), Seconds(8)).foreachRDD( s => { println( "RDD ID IS : " + s.id) s.foreach( e => println("data is " + e._1 + " :" + e._2)) println() }) The result is wrong. I checked the log, it showed: 17/06/25 17:31:26 DEBUG ReducedWindowedDStream: Time 1498383086000 ms is valid 17/06/25 17:31:26 DEBUG ReducedWindowedDStream: Window time = 2000 ms 17/06/25 17:31:26 DEBUG ReducedWindowedDStream: Slide time = 8000 ms 17/06/25 17:31:26 DEBUG ReducedWindowedDStream: Zero time = 1498383078000 ms 17/06/25 17:31:26 DEBUG ReducedWindowedDStream: Current window = [1498383085000 ms, 1498383086000 ms] 17/06/25 17:31:26 DEBUG ReducedWindowedDStream: Previous window = [1498383077000 ms, 1498383078000 ms] 17/06/25 17:31:26 INFO ShuffledDStream: Slicing from 1498383077000 ms to 1498383084000 ms (aligned to 1498383077000 ms and 1498383084000 ms) 17/06/25 17:31:26 INFO ShuffledDStream: Time 1498383078000 ms is invalid as zeroTime is 1498383078000 ms , slideDuration is 1000 ms and difference is 0 ms 17/06/25 17:31:26 DEBUG ShuffledDStream: Time 1498383079000 ms is valid 17/06/25 17:31:26 DEBUG MappedDStream: Time 1498383079000 ms is valid the slice time is wrong. [BTW]: Team members, If it was a bug, please don't fix it.I try to fix it myself.Thanks:) -- This message was sent by Atlassian JIRA (v6.4.14#64029) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org