RE: Classloading issues when using connectors with Uber jars with improper Shading in single Spark job

2019-09-15 Thread Sharma, Praneet
Bumping. All inputs appreciated Regards Praneet _ From: Sharma, Praneet Sent: Friday, August 23, 2019 11:24 AM To: 'user@spark.apache.org' Subject: Classloading issues when using connectors with Uber jars with improper Shading in single Spark job Hi

Conflicting PySpark Storage Level Defaults?

2019-09-15 Thread grp
Hi There Spark Users, Curious what is going on here. Not sure if possible bug or missing something. Extra eyes are much appreciated. Spark UI (Python API 2.4.3) by default is reporting persisted data-frames to be de-serialized MEMORY_AND_DISK however I always thought they were serialized for

Re: Monitor Spark Applications

2019-09-15 Thread Alex Landa
Hi Raman, The banzaicloud jar can also cover the JMX exports. Thanks, Alex On Fri, Sep 13, 2019 at 8:46 AM raman gugnani wrote: > Hi Alex, > > Thanks will check this out. > > Can it be done directly as spark also exposes the metrics or JVM. In > this my one doubt is how to assign fixed JMX p