Hi, Pralabh.
Could you elaborate on your situation more? I'm interested in your needs.
Currently, the default value of spark.network.timeout, 120s, is quite
bigger than the default value of
spark.kubernetes.driver.connectionTimeout, 10s. It would be a breaking
change if we increase `spark.kubernetes.driver.connectionTimeout` to
`120s` blindly in the next release.
In addition, I don't think it's a good idea to adjust
`spark.network.timeout` for K8s control plane timeout issues.
`spark.network.timeout` has already many other side-effects in Spark
operation itself. I'd recommend having more directional error messages
to guide those novice users in that situation instead.
Lastly, the most expensive API call is polling the executor status. To
reduce the overhead of K8s server side and mitigate the root cause,
Apache Spark 3.3.0 allows K8s API server-side caching via SPARK-36334.
You may want to try the following configuration if you have very
limited control plan resources.
spark.kubernetes.executor.enablePollingWithResourceVersion=true
Dongjoon.
On Mon, Aug 1, 2022 at 7:52 AM Pralabh Kumar <pralabhku...@gmail.com> wrote:
>
> Hi Dev team
>
>
>
> Since spark.network.timeout is default for all the network transactions .
> Shouldn’t spark.kubernetes.driver.connectionTimeout,
> spark.kubernetes.submission.connectionTimeout by default to be set
> spark.network.timeout .
>
> Users migrating from Yarn to K8s are familiar with spark.network.timeout and
> if time out occurs on K8s , they need to explicitly set the above two
> properties. If the above properties are default set to spark.network.timeout
> then user don’t need to explicitly set above properties and it can work with
> spark.network.timeout.
>
>
>
> Please let me know if my understanding is correct
>
>
>
> Regards
>
> Pralabh Kumar
---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
