Possible to limit number of IPC retries on spark-submit?

Greetings, Is it possible to limit the number of times the IPC client retries upon a spark-submit invocation? For context, see this StackOverflow post . In essence, I am

Re: Is there a way to get the final web URL from an active Spark context

To answer my own question, it turns out what I was after is the YARN ResourceManager URL for the Spark application. As alluded to in SPARK-20458 , it's possible to use the YARN API client to get this value. Here is a gist that shows how it can

Re: Problems during upgrade 2.2.2 -> 2.4.4

After digging deeper, we found that apps/workers inside zookeeper are not deserializable but drivers can. Due to this driver comes up (mysteriously). The deserialization is failing due to "RpcEndpointRef". I think somebody should be able to point me to a solution now, i guess. -- Sent from:

Problems during upgrade 2.2.2 -> 2.4.4

A few details about clusters - Current Version 2.2 - Resource manager: Spark standalone - Modes: cluster + supervise - HA setup: Zookeeper - Expected version after upgrade: 2.4.4 Note: Before and after the upgrade, everything works fine. During the upgrade, I see number of issues. - Spark