hi estimated Sparkes, I have some doubt about Streaming Context batchDuration parameter. I've already read excellent explication by Tathagata Das about difference between batch & window duration. But the issue seem a few confusion as for exam.- without any window SparkStreaming plays as implicit window by batchDuration actions. So, 1. What is simple strategy to get the correct value batchDuration? My UCase-is Flume spooling directory source for new text file => SparkStreaming analytic app. And one question more- 2. What is the reason for batching streaming? I used BigInsights Streams and there is not this "batch" strategy as "the stream is the continuous flow".You run Streams with one from many source connectors to catch data input flow(stream).If you need- you can work with data on window frame.But anything case - your streams app gets(listens) continuously stream/ flow. Perhaps - I don't understand some important and powerful characteristic of Spark Streaming architecture. Thanks in advance.
-- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Spark-Streaming-batchDuration-for-streaming-tp14469.html Sent from the Apache Spark User List mailing list archive at Nabble.com. --------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
