Yes. On Mon, Feb 23, 2015 at 11:16 PM, Avi Levi <avile...@gmail.com> wrote:
> @Tathagata Das so basically you are saying it is supported out of the > box, but we should expect a significant performance hit - is that right? > > > > On Tue, Feb 24, 2015 at 5:37 AM, Tathagata Das <t...@databricks.com> > wrote: > >> The default persistence level is MEMORY_AND_DISK, so the LRU policy would >> discard the blocks to disk, so the streaming app will not fail. However, >> since things will get constantly read in and out of disk as windows are >> processed, the performance wont be great. So it is best to have sufficient >> memory to keep all the window data in memory. >> >> TD >> >> On Mon, Feb 23, 2015 at 8:26 AM, Shao, Saisai <saisai.s...@intel.com> >> wrote: >> >>> I don't think current Spark Streaming supports window operations which >>> beyond its available memory, internally Spark Streaming puts all the data >>> in the memory belongs to the effective window, if the memory is not enough, >>> BlockManager will discard the blocks at LRU policy, so something unexpected >>> will be occurred. >>> >>> Thanks >>> Jerry >>> >>> -----Original Message----- >>> From: avilevi3 [mailto:avile...@gmail.com] >>> Sent: Monday, February 23, 2015 12:57 AM >>> To: user@spark.apache.org >>> Subject: spark streaming window operations on a large window size >>> >>> Hi guys, >>> >>> does spark streaming supports window operations on a sliding window that >>> is data is larger than the available memory? >>> we would like to >>> currently we are using kafka as input, but we could change that if >>> needed. >>> >>> thanks >>> Avi >>> >>> >>> >>> -- >>> View this message in context: >>> http://apache-spark-user-list.1001560.n3.nabble.com/spark-streaming-window-operations-on-a-large-window-size-tp21764.html >>> Sent from the Apache Spark User List mailing list archive at Nabble.com. >>> >>> --------------------------------------------------------------------- >>> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For >>> additional commands, e-mail: user-h...@spark.apache.org >>> >>> >>> --------------------------------------------------------------------- >>> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org >>> For additional commands, e-mail: user-h...@spark.apache.org >>> >>> >> >
