[
https://issues.apache.org/jira/browse/SPARK-24434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16599697#comment-16599697
]
Matt Cheah commented on SPARK-24434:
------------------------------------
Everyone, thank you for your contribution to this discussion. It is important
for us to agree upon constructive next steps to avoid this kind of
miscommunication in the future.
I have a few notes to make from having collaborated with Yifei and Onur on this
patch and on behalf of Palantir.
Firstly, we apologize that we did not communicate clearly enough on Apache
communication channels that we were working on this, and the urgency of which
we needed this work done. We agree with [~felixcheung]'s assessment that notes
from the weekly meetings that have bearing on Spark development should be sent
back to the wider community. We are specifically sorry for not having said
something to the effect of "I am taking a stab at implementing this at
https://github.com/... . Stavros, are you cool with that?" Palantir and the
Kubernetes Big Data group must improve our communication next time.
Secondly, we would suggest that a work in progress patch proposed early on in
the feature's development would have been helpful for users to prepare to be
able to use this feature in their internal tools. It's helpful for everyone to
see the API and expected behavior of a new feature so that they can plan to
take advantage of that feature ahead of time.
Thirdly, a small clarifying comment on timelines and urgency. While we don't
see the need for this to be in Spark 2.4, we will be taking the patch ahead of
time on our fork of Spark which follows the Apache master branch (see
[https://github.com/palantir/spark).] We were hoping to cherry-pick this patch
soon but could have been clearer in our communication of this need.
Fourthly, we are sorry for the wording in "On 15 Aug it was discussed that as
Stavros Kontopoulos was out, and was not actively working on this PR at that
moment, Yifei Huang and I can take over and start working on this.”: Instead of
“take over”, we should have said “contribute to this feature” in this comment
specifically.
Finally, moving forward we are happy to collaborate on what the community
believes to be the best implementation of this feature. We are happy to use
Onur's, but we can also use Stavros's. Regardless of the chosen implementation,
credit should be given to all parties. For example if Onur's implementation is
chosen, Stavros's design work should be called out in the pull request
description. Either way, we would like to see this feature merged by Friday,
September 07, though this will have to be delayed if the Spark 2.4 release
branch is not cut before that time (since we don't want this going into master
and ending up in Spark 2.4 as a result).
We are open to feedback on any of the above points and suggestions on how we
can improve the way we contribute to Spark in the future.
> Support user-specified driver and executor pod templates
> --------------------------------------------------------
>
> Key: SPARK-24434
> URL: https://issues.apache.org/jira/browse/SPARK-24434
> Project: Spark
> Issue Type: New Feature
> Components: Kubernetes
> Affects Versions: 2.4.0
> Reporter: Yinan Li
> Priority: Major
>
> With more requests for customizing the driver and executor pods coming, the
> current approach of adding new Spark configuration options has some serious
> drawbacks: 1) it means more Kubernetes specific configuration options to
> maintain, and 2) it widens the gap between the declarative model used by
> Kubernetes and the configuration model used by Spark. We should start
> designing a solution that allows users to specify pod templates as central
> places for all customization needs for the driver and executor pods.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
