Re: A scene with unstable Spark performance

This is a case where resources are fixed in the same SparkContext, but sqls have different priorities. Some SQLs are only allowed to be executed if there are spare resources, once the high priority sql comes in, those sqls taskset either are killed or stalled. If we set a high priority pool's

Re: Is RDD thread safe?

gt; > > > > > On Tue, Nov 12, 2019 at 7:18 AM Chang Chen wrote: > >> >> Hi all >> >> I meet a case where I need cache a source RDD, and then create different >> DataFrame from it in different threads to accelerate query. >> >> I know that

Is RDD thread safe?

Hi all I meet a case where I need cache a source RDD, and then create different DataFrame from it in different threads to accelerate query. I know that SparkSession is thread safe( https://issues.apache.org/jira/browse/SPARK-15135), but i am not sure whether RDD si thread safe or not Thanks

Re: The Future Of DStream

> Co-Founder & CTO | Equalum > > Mobile: +972-54-7801286 | Email: ofir.ma...@equalum.io > > On Wed, Jul 27, 2016 at 11:24 AM, Chang Chen <baibaic...@gmail.com> wrote: > >> >> I don't understand what kind of low level control that DStream can do >> whi

Re: The Future Of DStream

CTO | Equalum > > Mobile: +972-54-7801286 | Email: ofir.ma...@equalum.io > <javascript:_e(%7B%7D,'cvml','ofir.ma...@equalum.io');> > > On Wed, Jul 27, 2016 at 8:02 AM, Chang Chen <baibaic...@gmail.com > <javascript:_e(%7B%7D,'cvml','baibaic...@gmail.com');>>

The Future Of DStream

Hi guys Structure Stream is coming with spark 2.0, but I noticed that DStream is still here What's the future of the DStream, will it be deprecated and removed eventually? Or co-existed with Structure Stream forever? Thanks Chang