heap overflow within seconds : pyspark kinesis stream with Spark 2.1.0

Hi, I hope i am missing very simple point to stuck this kind of error. http://stackoverflow.com/questions/43560807/pyspark-streaming-from-kinesis-kills-heap Regards, Serkan

Off heap memory settings and Tungsten

Hi, I wonder when to enable spark's off heap settings. Shouldn't tungsten enable these automatically in 2.1? http://stackoverflow.com/questions/43330902/spark-off-heap-memory-config-and-tungsten Regards, Georg -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com

Re: how to add new column using regular expression within pyspark dataframe

Thanks a lot! Just another question, how can I extract the minutes as a number? I can use: .withColumn('duration_m',split(flight.duration,'h').getItem(1) to get strings like '10m' but how do I drop the charater "m" at the end? I can use substr(), but what's the function to get the length of the

Re: Spark SQL - Global Temporary View is not behaving as expected

Look at alluxio for sharing across drivers or spark jobserver Le 22 avr. 2017 10:24 AM, "Hemanth Gudela" a écrit : > Thanks for your reply. > > > > Creating a table is an option, but such approach slows down reads & writes > for a real-time analytics streaming use case that I’m currently working

Re: Spark SQL - Global Temporary View is not behaving as expected

Thanks for your reply. Creating a table is an option, but such approach slows down reads & writes for a real-time analytics streaming use case that I’m currently working on. If at all global temporary view could have been accessible across sessions/spark contexts, that would have simplified my u

Re: Spark SQL - Global Temporary View is not behaving as expected

Cross session is this context is multiple spark sessions from the same spark context. Since you are running two shells, you are having different spark context. Do you have to you a temp view? Could you create a table? _ From: Hemanth Gudela mailto:hemanth.gud...@qva

Spark SQL - Global Temporary View is not behaving as expected

Hi, According to documentation, global temporary views are cross-session accessible. But when I try to query a global temporary view from another spark shell like this --> Instance 1 of spark-shell --

Re: Spark structured streaming: Is it possible to periodically refresh static data frame?

Thank you Georg, Gene for your ideas. For now, I am using ”Futures” to asynchronously run a background thread that periodically creates a new dataframe fetching latest data from underlying table, and re-registers temp view with the same name as used by main thread’s static dataframe. This looks