[ https://issues.apache.org/jira/browse/SPARK-21206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16071110#comment-16071110 ]
Fei Shao commented on SPARK-21206: ---------------------------------- [~srowen] Here is the case I want to express: !screenshot-2.png! I sent an email named "the function of countByValueAndWindow and foreachRDD in DStream, would you like help me understand it please?". It is related to this issues. > the window slice of Dstream is wrong > ------------------------------------ > > Key: SPARK-21206 > URL: https://issues.apache.org/jira/browse/SPARK-21206 > Project: Spark > Issue Type: Bug > Components: DStreams > Affects Versions: 2.1.0 > Reporter: Fei Shao > Attachments: screenshot-1.png, screenshot-2.png > > > the code is : > val conf = new SparkConf().setAppName("testDstream").setMaster("local[4]") > val ssc = new StreamingContext(conf, Seconds(1)) > ssc.checkpoint( "path") > val lines = ssc.socketTextStream("IP", PORT) > lines.countByValueAndWindow( Seconds(2), Seconds(8)).foreachRDD( s => { > println( "RDD ID IS : " + s.id) > s.foreach( e => println("data is " + e._1 + " :" + e._2)) > println() > }) > The result is wrong. > I checked the log, it showed: > 17/06/25 17:31:26 DEBUG ReducedWindowedDStream: Time 1498383086000 ms is valid > 17/06/25 17:31:26 DEBUG ReducedWindowedDStream: Window time = 2000 ms > 17/06/25 17:31:26 DEBUG ReducedWindowedDStream: Slide time = 8000 ms > 17/06/25 17:31:26 DEBUG ReducedWindowedDStream: Zero time = 1498383078000 ms > 17/06/25 17:31:26 DEBUG ReducedWindowedDStream: Current window = > [1498383085000 ms, 1498383086000 ms] > 17/06/25 17:31:26 DEBUG ReducedWindowedDStream: Previous window = > [1498383077000 ms, 1498383078000 ms] > 17/06/25 17:31:26 INFO ShuffledDStream: Slicing from 1498383077000 ms to > 1498383084000 ms (aligned to 1498383077000 ms and 1498383084000 ms) > 17/06/25 17:31:26 INFO ShuffledDStream: Time 1498383078000 ms is invalid as > zeroTime is 1498383078000 ms , slideDuration is 1000 ms and difference is 0 ms > 17/06/25 17:31:26 DEBUG ShuffledDStream: Time 1498383079000 ms is valid > 17/06/25 17:31:26 DEBUG MappedDStream: Time 1498383079000 ms is valid > the slice time is wrong. > [BTW]: Team members, > If it was a bug, please don't fix it.I try to fix it myself.Thanks:) -- This message was sent by Atlassian JIRA (v6.4.14#64029) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org