Urgent : Changes required in the archive

2019-09-25 Thread Vishal Verma
Hi All, The help request sent by me on 20th Dec 2017 to apache spark developers need to be edited as it contains our clients sensitive information about the infrastructure and user Id. I understand the policies of ASF and regret the mistake made earlier. The mistake has been highlighted to the cli

Standalone Spark, How to find (driver's ) final status for an application

2019-09-25 Thread Nilkanth Patel
I am setting up *Spark 2.2.0 in standalone mode* ( https://spark.apache.org/docs/latest/spark-standalone.html) and submitting spark jobs programatically using SparkLauncher sparkAppLauncher = new SparkLauncher(userNameMap).setMaster(sparkMaster).setAppName(appName).; SparkAppHandle sparkAppH

Re: intermittent Kryo serialization failures in Spark

2019-09-25 Thread Jerry Vinokurov
Hi Julien, Thanks for the suggestion. If we don't do a broadcast, that would presumably affect the performance of the job, as the model that is failing to be broadcast is something that we need to be shared across the cluster. But it may be worth it if the trade-off is not having things run proper