Re: Explanation regarding Spark Streaming

2016-08-06 Thread Mich Talebzadeh
park > <http://www.amazon.com/Big-Data-Analytics-Spark-Practitioners/dp/1484209656/> > > > > *From:* Mich Talebzadeh [mailto:mich.talebza...@gmail.com] > *Sent:* Saturday, August 6, 2016 12:25 PM > *To:* Mohammed Guller > *Cc:* Jacek Laskowski; Saurav Sinha; user

RE: Explanation regarding Spark Streaming

2016-08-06 Thread Mohammed Guller
@japila.pl>] Sent: Saturday, August 6, 2016 1:54 AM To: Mohammed Guller Cc: Saurav Sinha; user Subject: RE: Explanation regarding Spark Streaming Hi, Thanks for explanation, but it does not prove Spark will OOM at some point. You assume enough data to store but there could be none. Jacek On 6

Re: Explanation regarding Spark Streaming

2016-08-06 Thread Mich Talebzadeh
mmed > > > > *From:* Jacek Laskowski [mailto:ja...@japila.pl] > *Sent:* Saturday, August 6, 2016 1:54 AM > *To:* Mohammed Guller > *Cc:* Saurav Sinha; user > *Subject:* RE: Explanation regarding Spark Streaming > > > > Hi, > > Thanks for explanation, but it

RE: Explanation regarding Spark Streaming

2016-08-06 Thread Mohammed Guller
performance even worse. Mohammed From: Jacek Laskowski [mailto:ja...@japila.pl] Sent: Saturday, August 6, 2016 1:54 AM To: Mohammed Guller Cc: Saurav Sinha; user Subject: RE: Explanation regarding Spark Streaming Hi, Thanks for explanation, but it does not prove Spark will OOM at some point. You

Re: Explanation regarding Spark Streaming

2016-08-06 Thread Mich Talebzadeh
ntly at the same rate. >> >> Also keep in mind that windowing operations on a DStream implicitly >> persist every RDD in a DStream in memory. >> >> Mohammed >> >> -Original Message- >> From: Jacek Laskowski [mailto:ja...@japila.pl]

RE: Explanation regarding Spark Streaming

2016-08-06 Thread Jacek Laskowski
:ja...@japila.pl] > Sent: Thursday, August 4, 2016 4:25 PM > To: Mohammed Guller > Cc: Saurav Sinha; user > Subject: Re: Explanation regarding Spark Streaming > > On Fri, Aug 5, 2016 at 12:48 AM, Mohammed Guller <moham...@glassbeam.com> > wrote: > > and eventually you will run out of memory. > > Why? Mind elaborating? > > Jacek >

RE: Explanation regarding Spark Streaming

2016-08-05 Thread Mohammed Guller
[mailto:ja...@japila.pl] Sent: Thursday, August 4, 2016 4:25 PM To: Mohammed Guller Cc: Saurav Sinha; user Subject: Re: Explanation regarding Spark Streaming On Fri, Aug 5, 2016 at 12:48 AM, Mohammed Guller <moham...@glassbeam.com> wrote: > and eventually you will run out of memory.

Re: Explanation regarding Spark Streaming

2016-08-04 Thread Jacek Laskowski
On Fri, Aug 5, 2016 at 12:48 AM, Mohammed Guller wrote: > and eventually you will run out of memory. Why? Mind elaborating? Jacek - To unsubscribe e-mail: user-unsubscr...@spark.apache.org

Re: Explanation regarding Spark Streaming

2016-08-04 Thread Mich Talebzadeh
Also check spark UI streaming section for various helpful stats. by default it runs on 4040 but can change it by setting--conf "spark.ui.port=" HTH Dr Mich Talebzadeh LinkedIn * https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw

RE: Explanation regarding Spark Streaming

2016-08-04 Thread Mohammed Guller
The backlog will increase as time passes and eventually you will run out of memory. Mohammed Author: Big Data Analytics with Spark From: Saurav Sinha [mailto:sauravsinh...@gmail.com] Sent: Wednesday, August 3, 2016