Re: Why do we ever run out of memory in Spark Structured Streaming?

2017-04-05 Thread kant kodali
-late-data-and-watermarking >>> Monitoring - http://spark.apache.org/docs/latest/structured-streaming-p >>> rogramming-guide.html#monitoring-streaming-queries >>> >>> In case you were referring to something else, please give us more >>> context details

Re: Why do we ever run out of memory in Spark Structured Streaming?

2017-04-05 Thread kant kodali
; programming-guide.html#handling-late-data-and-watermarking >> Monitoring - http://spark.apache.org/docs/latest/structured-streaming- >> programming-guide.html#monitoring-streaming-queries >> >> In case you were referring to something else, please give us more context >>

Re: Why do we ever run out of memory in Spark Structured Streaming?

2017-04-05 Thread kant kodali
symptoms you are observing. > > On Tue, Apr 4, 2017 at 5:17 PM, kant kodali <kanth...@gmail.com> wrote: > >> Why do we ever run out of memory in Spark Structured Streaming especially >> when Memory can always spill to disk ? until the disk is full we shouldn't >> be out o

Re: Why do we ever run out of memory in Spark Structured Streaming?

2017-04-04 Thread Tathagata Das
kodali <kanth...@gmail.com> wrote: > Why do we ever run out of memory in Spark Structured Streaming especially > when Memory can always spill to disk ? until the disk is full we shouldn't > be out of memory.isn't it? sure thrashing will happen more frequently and > degrades performance

Why do we ever run out of memory in Spark Structured Streaming?

2017-04-04 Thread kant kodali
Why do we ever run out of memory in Spark Structured Streaming especially when Memory can always spill to disk ? until the disk is full we shouldn't be out of memory.isn't it? sure thrashing will happen more frequently and degrades performance but we do we ever run out Memory even in case