how to trace sparkDriver context creation for pyspark

Regards,   Mihai IacobDSX Local - Security, IBM Analytics - To unsubscribe e-mail: user-unsubscr...@spark.apache.org

Re: spark 2.2.1

.   Regards,  Mihai IacobDSX Local - Security, IBM Analytics     - Original message -From: Bill Schwanitz <bil...@bilsch.org>To: Mihai Iaco

spark 2.2.1

(PersistenceEngine.scala:86)​​​     Regards,  Mihai IacobDSX Local - Security, IBM Analytics

spark 2.0 and spark 2.2

Does spark 2.2 have good backwards compatibility? Is there something that won't work that works in spark 2.0?   Regards,  Mihai IacobDSX Local - Security

Re: /tmp fills up to 100GB when using a window function

When does spark remove them?   Regards,  Mihai IacobDSX Local - Security, IBM Analytics

/tmp fills up to 100GB when using a window function

(sqlDf['***']) sqlDf = sqlDf.withColumn("***",rank().over(window)) df_w_least = sqlDf.filter("***=1")       Regards,  Mihai IacobDSX L

Re: Dynamic resource allocation to Spark on Mesos

What about Spark on Kubernetes, is there a way to manage dynamic resource allocation?   Regards,  Mihai Iacob

Re: Spark to Kafka communication encrypted ?

Does Spark support encryption for inter node communication ?   Regards,  Mihai IacobNext Generation Platform - Security IBM