[jira] [Comment Edited] (SPARK-24434) Support user-specified driver and executor pod templates
[ https://issues.apache.org/jira/browse/SPARK-24434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17218056#comment-17218056 ] Liu Runzhong edited comment on SPARK-24434 at 10/21/20, 2:34 AM: - I think yes, you need to provide a pod template file when spark-submit as follows [~prakki79] ```yaml apiversion: v1 kind: Pod spec: containers: - name: sidecar-container image: nginx:1.7.9 ports: - containerPort: 80 ``` was (Author: runzhliu): I think yes, you need to provide a pod template file when spark-submit as follows [~prakki79] ```yaml apiversion: v1 kind: Pod spec: containers: - name: sidecar-container # Simple sidecar: display log files using nginx. # In reality, this sidecar would be a custom image # that uploads logs to a third-party or storage service. image: nginx:1.7.9 ports: - containerPort: 80 ``` > Support user-specified driver and executor pod templates > > > Key: SPARK-24434 > URL: https://issues.apache.org/jira/browse/SPARK-24434 > Project: Spark > Issue Type: New Feature > Components: Kubernetes, Spark Core >Affects Versions: 2.4.0 >Reporter: Yinan Li >Assignee: Onur Satici >Priority: Major > Fix For: 3.0.0 > > > With more requests for customizing the driver and executor pods coming, the > current approach of adding new Spark configuration options has some serious > drawbacks: 1) it means more Kubernetes specific configuration options to > maintain, and 2) it widens the gap between the declarative model used by > Kubernetes and the configuration model used by Spark. We should start > designing a solution that allows users to specify pod templates as central > places for all customization needs for the driver and executor pods. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Comment Edited] (SPARK-24434) Support user-specified driver and executor pod templates
[ https://issues.apache.org/jira/browse/SPARK-24434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16599728#comment-16599728 ] Stavros Kontopoulos edited comment on SPARK-24434 at 9/1/18 8:38 PM: - Spark belongs to the community (no?) and should not serve any company's priorities like Palantirs, [~mcheah] we don't need the meeting then if we are going to overlap with each other, fine. I have a question, why serve Palantir's priorities and not mine? This is not healthy. You didnt collaborate with me why? .Why nobody respected my will to make it go in 2.4, It was my priority back then. We are implementing the same design, the whole discussion makes no sense, it is not about mine or your implementation... I am not going to implement the same thing again its not reasonable. But also I wouldnt create a PR that just works, you can check my comments, that was easy, just call load and load the template (it is not rocket science this work). Sorry I dont see any real arguments in the discussion, and as I said I dont want to reply but the politically correct replies leave me no choice. We have talked on slack several times and privately as well. You could always have pinged me but you decided to collaborate on this, without anyone knowing. Probably people don't understand the meaning of fairness, I am not going to explain it here. We can always create any RP we like and then we will see what work is merged, cool. For good or bad though the meeting has power because k8s committers have the final saying on merging no? So I dont agree. The whole discussion for me is pointless, the message communicated, culture and attitude is clear. That is for committers or others in the Spark project not to violate the rules fine. No rule was violated no worries. This is the first time I see this kind of competing "collaboration" among the people of a group that works for the same cause. It is disappointing. On the other hand, lesson learned, let's move on, no hard feelings. was (Author: skonto): Spark belongs to the community (no?) and should not serve any company's priorities like Palantirs, [~mcheah] we don't need the meeting then if we are going to overlap with each other, fine. I have a question, why serve Palantir's priorities and not mine? This is not healthy. You didnt collaborate with me why? .Why nobody respected my will to make it go in 2.4, It was my priority back then. We are implementing the same design, the whole discussion makes no sense, it is not about mine or your implementation... I am not going to implement the same thing again its not reasonable. But also I wouldnt create a PR that just works, you can check my comments, that was easy, just call load and load the template (it is not rocket science this work). Sorry I dont see any real arguments in the discussion, and as I said I dont want to reply but the politically correct replies leave me no choice. We have talked on slack several times and privately as well. You could always have pinged me but you decided to collaborate on this, without anyone knowing. Probably people don't understand the meaning of fairness, I am not going to explain it here. We can always create any RP we like and then we will see what work is merged, cool. For good or bad though the meeting has power because k8s committers have the final saying on merging no? So I dont agree. The whole discussion for me is pointless, the message communicated, culture and attitude is clear. That is for committers or others in the Spark project not to violate the rules fine. No rule was violated no worries. I am also ok. This is the first time I see this kind of competing "collaboration" among the people of a group that works for the same cause. It is disappointing. On the other hand, lesson learned, let's move on, no hard feelings. > Support user-specified driver and executor pod templates > > > Key: SPARK-24434 > URL: https://issues.apache.org/jira/browse/SPARK-24434 > Project: Spark > Issue Type: New Feature > Components: Kubernetes >Affects Versions: 2.4.0 >Reporter: Yinan Li >Priority: Major > > With more requests for customizing the driver and executor pods coming, the > current approach of adding new Spark configuration options has some serious > drawbacks: 1) it means more Kubernetes specific configuration options to > maintain, and 2) it widens the gap between the declarative model used by > Kubernetes and the configuration model used by Spark. We should start > designing a solution that allows users to specify pod templates as central > places for all customization needs for the driver and executor pods. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (SPARK-24434) Support user-specified driver and executor pod templates
[ https://issues.apache.org/jira/browse/SPARK-24434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16599728#comment-16599728 ] Stavros Kontopoulos edited comment on SPARK-24434 at 9/1/18 8:37 PM: - Spark belongs to the community (no?) and should not serve any company's priorities like Palantirs, [~mcheah] we don't need the meeting then if we are going to overlap with each other, fine. I have a question, why serve Palantir's priorities and not mine? This is not healthy. You didnt collaborate with me why? .Why nobody respected my will to make it go in 2.4, It was my priority back then. We are implementing the same design, the whole discussion makes no sense, it is not about mine or your implementation... I am not going to implement the same thing again its not reasonable. But also I wouldnt create a PR that just works, you can check my comments, that was easy, just call load and load the template (it is not rocket science this work). Sorry I dont see any real arguments in the discussion, and as I said I dont want to reply but the politically correct replies leave me no choice. We have talked on slack several times and privately as well. You could always have pinged me but you decided to collaborate on this, without anyone knowing. Probably people don't understand the meaning of fairness, I am not going to explain it here. We can always create any RP we like and then we will see what work is merged, cool. For good or bad though the meeting has power because k8s committers have the final saying on merging no? So I dont agree. The whole discussion for me is pointless, the message communicated, culture and attitude is clear. That is for committers or others in the Spark project not to violate the rules fine. No rule was violated no worries. I am also ok. This is the first time I see this kind of competing "collaboration" among the people of a group that works for the same cause. It is disappointing. On the other hand, lesson learned, let's move on, no hard feelings. was (Author: skonto): Spark belongs to the community (no?) and should not serve any company's priorities like Palantirs, [~mcheah] we don't need the meeting then if we are going to overlap with each other, fine. I have a question, why serve Palantir's priorities and not mine? This is not healthy. You didnt collaborate with me why? ) .Why nobody respected my will to make it go in 2.4, It was my priority back then too. We are implementing the same design, the whole discussion makes no sense, it is not about mine or your implementation... I am not going to implement the same thing again its not reasonable. But also I wouldnt create a PR that just works, you can check my comments, that was easy, just call load and load the template (it is not rocket science this work). Sorry I dont see any real arguments in the discussion, and as I said I dont want to reply but the politically correct replies leave me no choice. We have talked on slack several times and privately as well. You could always have pinged me but you decided to collaborate on this, without anyone knowing. Probably people don't understand the meaning of fairness, I am not going to explain it here. We can always create any RP we like and then we will see what work is merged, cool. For good or bad though the meeting has power because k8s committers have the final saying on merging no? So I dont agree. The whole discussion for me is pointless, the message communicated, culture and attitude is clear. That is for committers or others in the Spark project not to violate the rules fine. No rule was violated no worries. I am also ok. This is the first time I see this kind of competing "collaboration" among the people of a group that works for the same cause. It is disappointing. On the other hand, lesson learned, let's move on, no hard feelings. > Support user-specified driver and executor pod templates > > > Key: SPARK-24434 > URL: https://issues.apache.org/jira/browse/SPARK-24434 > Project: Spark > Issue Type: New Feature > Components: Kubernetes >Affects Versions: 2.4.0 >Reporter: Yinan Li >Priority: Major > > With more requests for customizing the driver and executor pods coming, the > current approach of adding new Spark configuration options has some serious > drawbacks: 1) it means more Kubernetes specific configuration options to > maintain, and 2) it widens the gap between the declarative model used by > Kubernetes and the configuration model used by Spark. We should start > designing a solution that allows users to specify pod templates as central > places for all customization needs for the driver and executor pods. -- This message was sent by Atlassian JIRA
[jira] [Comment Edited] (SPARK-24434) Support user-specified driver and executor pod templates
[ https://issues.apache.org/jira/browse/SPARK-24434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16599728#comment-16599728 ] Stavros Kontopoulos edited comment on SPARK-24434 at 9/1/18 8:36 PM: - Spark belongs to the community (no?) and should not serve any company's priorities like Palantirs, [~mcheah] we don't need the meeting then if we are going to overlap with each other, fine. I have a question, why serve Palantir's priorities and not mine? This is not healthy :). You didnt collaborate with me why? !/jira/images/icons/emoticons/smile.png|width=16,height=16! Why nobody respected my will to make it go in 2.4, It was my priority back then too. We are implementing the same design, the whole discussion makes no sense, it is not about mine or your implementation... I am not going to implement the same thing again its not reasonable. But also I wouldnt create a PR that just works, you can check my comments, that was easy, just call load and load the template (it is not rocket science this work). Sorry I dont see any real arguments in the discussion, and as I said I dont want to reply but the politically correct replies leave me no choice. We have talked on slack several times and privately as well. You could always have pinged me but you decided to collaborate on this, without anyone knowing. Probably people don't understand the meaning of fairness, I am not going to explain it here. We can always create any RP we like and then we will see what work is merged, cool. For good or bad though the meeting has power because k8s committers have the final saying on merging no? So I dont agree. The whole discussion for me is pointless, the message communicated, culture and attitude is clear. That is for committers or others in the Spark project not to violate the rules fine. No rule was violated no worries. I am also ok. This is the first time I see this kind of competing "collaboration" among the people of a group that works for the same cause. It is disappointing. On the other hand, lesson learned, let's move on, no hard feelings. was (Author: skonto): Spark belongs to the community (no?) and should not serve any company's priorities like Palantirs, [~mcheah] we don't need the meeting then if we are going to overlap with each other, fine. I have a question, why serve Palantir's priorities and not mine? This is not healthy :). You didnt collaborate with me why? :) Why nobody respected my will to make it go in 2.4, It was my priority back then too. We are implementing the same design, the whole discussion makes no sense, it is not about mine or your implementation... I am not going to implement the same thing again its not reasonable. But also I wouldnt create a PR that just works, you can check my comments, that was easy, just call load and load the template (it is not rocket science this work). Sorry I dont see any real arguments in the discussion, and as I said I dont want to reply but the politically correct replies leave me no choice. We have talked on slack several times and privately as well. You could always have pinged me but you decided to collaborate on this, without anyone knowing. Probably people don't understand the meaning of fairness, I am not going to explain it here. We can always create any RP we like and then we will see what work is merged, cool. For good or bad though the meeting has power because k8s committers have the final saying on merging no? So I dont agree. The whole discussion for me is pointless, the message communicated, culture and attitude is clear. That is for committers or others in the Spark project not to violate the rules fine. No rule was violated no worries. I am also ok. This is the first time I see this kind of competing collaboration among the people of a group that works for the same cause. It is disappointing. On the other hand, lesson learned, let's move on, no hard feelings. > Support user-specified driver and executor pod templates > > > Key: SPARK-24434 > URL: https://issues.apache.org/jira/browse/SPARK-24434 > Project: Spark > Issue Type: New Feature > Components: Kubernetes >Affects Versions: 2.4.0 >Reporter: Yinan Li >Priority: Major > > With more requests for customizing the driver and executor pods coming, the > current approach of adding new Spark configuration options has some serious > drawbacks: 1) it means more Kubernetes specific configuration options to > maintain, and 2) it widens the gap between the declarative model used by > Kubernetes and the configuration model used by Spark. We should start > designing a solution that allows users to specify pod templates as central > places for all customization needs for the
[jira] [Comment Edited] (SPARK-24434) Support user-specified driver and executor pod templates
[ https://issues.apache.org/jira/browse/SPARK-24434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16599728#comment-16599728 ] Stavros Kontopoulos edited comment on SPARK-24434 at 9/1/18 8:36 PM: - Spark belongs to the community (no?) and should not serve any company's priorities like Palantirs, [~mcheah] we don't need the meeting then if we are going to overlap with each other, fine. I have a question, why serve Palantir's priorities and not mine? This is not healthy. You didnt collaborate with me why? ) .Why nobody respected my will to make it go in 2.4, It was my priority back then too. We are implementing the same design, the whole discussion makes no sense, it is not about mine or your implementation... I am not going to implement the same thing again its not reasonable. But also I wouldnt create a PR that just works, you can check my comments, that was easy, just call load and load the template (it is not rocket science this work). Sorry I dont see any real arguments in the discussion, and as I said I dont want to reply but the politically correct replies leave me no choice. We have talked on slack several times and privately as well. You could always have pinged me but you decided to collaborate on this, without anyone knowing. Probably people don't understand the meaning of fairness, I am not going to explain it here. We can always create any RP we like and then we will see what work is merged, cool. For good or bad though the meeting has power because k8s committers have the final saying on merging no? So I dont agree. The whole discussion for me is pointless, the message communicated, culture and attitude is clear. That is for committers or others in the Spark project not to violate the rules fine. No rule was violated no worries. I am also ok. This is the first time I see this kind of competing "collaboration" among the people of a group that works for the same cause. It is disappointing. On the other hand, lesson learned, let's move on, no hard feelings. was (Author: skonto): Spark belongs to the community (no?) and should not serve any company's priorities like Palantirs, [~mcheah] we don't need the meeting then if we are going to overlap with each other, fine. I have a question, why serve Palantir's priorities and not mine? This is not healthy :). You didnt collaborate with me why? !/jira/images/icons/emoticons/smile.png|width=16,height=16! Why nobody respected my will to make it go in 2.4, It was my priority back then too. We are implementing the same design, the whole discussion makes no sense, it is not about mine or your implementation... I am not going to implement the same thing again its not reasonable. But also I wouldnt create a PR that just works, you can check my comments, that was easy, just call load and load the template (it is not rocket science this work). Sorry I dont see any real arguments in the discussion, and as I said I dont want to reply but the politically correct replies leave me no choice. We have talked on slack several times and privately as well. You could always have pinged me but you decided to collaborate on this, without anyone knowing. Probably people don't understand the meaning of fairness, I am not going to explain it here. We can always create any RP we like and then we will see what work is merged, cool. For good or bad though the meeting has power because k8s committers have the final saying on merging no? So I dont agree. The whole discussion for me is pointless, the message communicated, culture and attitude is clear. That is for committers or others in the Spark project not to violate the rules fine. No rule was violated no worries. I am also ok. This is the first time I see this kind of competing "collaboration" among the people of a group that works for the same cause. It is disappointing. On the other hand, lesson learned, let's move on, no hard feelings. > Support user-specified driver and executor pod templates > > > Key: SPARK-24434 > URL: https://issues.apache.org/jira/browse/SPARK-24434 > Project: Spark > Issue Type: New Feature > Components: Kubernetes >Affects Versions: 2.4.0 >Reporter: Yinan Li >Priority: Major > > With more requests for customizing the driver and executor pods coming, the > current approach of adding new Spark configuration options has some serious > drawbacks: 1) it means more Kubernetes specific configuration options to > maintain, and 2) it widens the gap between the declarative model used by > Kubernetes and the configuration model used by Spark. We should start > designing a solution that allows users to specify pod templates as central > places for all customization needs for the driver
[jira] [Comment Edited] (SPARK-24434) Support user-specified driver and executor pod templates
[ https://issues.apache.org/jira/browse/SPARK-24434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16599728#comment-16599728 ] Stavros Kontopoulos edited comment on SPARK-24434 at 9/1/18 8:35 PM: - Spark belongs to the community (no?) and should not serve any company's priorities like Palantirs, [~mcheah] we don't need the meeting then if we are going to overlap with each other, fine. I have a question, why serve Palantir's priorities and not mine? This is not healthy :). You didnt collaborate with me why? :) Why nobody respected my will to make it go in 2.4, It was my priority back then too. We are implementing the same design, the whole discussion makes no sense, it is not about mine or your implementation... I am not going to implement the same thing again its not reasonable. But also I wouldnt create a PR that just works, you can check my comments, that was easy, just call load and load the template (it is not rocket science this work). Sorry I dont see any real arguments in the discussion, and as I said I dont want to reply but the politically correct replies leave me no choice. We have talked on slack several times and privately as well. You could always have pinged me but you decided to collaborate on this, without anyone knowing. Probably people don't understand the meaning of fairness, I am not going to explain it here. We can always create any RP we like and then we will see what work is merged, cool. For good or bad though the meeting has power because k8s committers have the final saying on merging no? So I dont agree. The whole discussion for me is pointless, the message communicated, culture and attitude is clear. That is for committers or others in the Spark project not to violate the rules fine. No rule was violated no worries. I am also ok. This is the first time I see this kind of competing collaboration among the people of a group that works for the same cause. It is disappointing. On the other hand, lesson learned, let's move on, no hard feelings. was (Author: skonto): Spark belongs to the community (no?) and should not serve any company's priorities like Palantirs, [~mcheah] we don't need the meeting then if we are going to overlap with each other, fine. I have a question, why serve Palantir's priorities and not mine? This is not healthy :). You didnt collaborate with me why? :) Why nobody respected my will to make it go in 2.4, It was my priority back then too. We are implementing the same design, the whole discussion makes no sense, it is not about mine or your implementation... I am not going to implement the same thing again its not reasonable. But also I wouldnt create a PR that just works, you can check my comments, that was easy, just call load and load the template (it is not rocket science this work). Sorry I dont see any real arguments in the discussion, and as I said I dont want to reply but the politically correct replies leave me no choice. We have talked on slack several times and privately as well. You could always have pinged me but you decided to collaborate on this, without anyone knowing. Probably people don't understand the meaning of fairness, I am not going to explain it here. We can always create any RP we like and then we will see what work is merged, cool. For good or bad though the meeting has power because k8s committers have the final saying on merging no? So I dont agree. The whole discussion for me is pointless, the message communicated, culture and attitude is clear. That is for committers or others in the Spark project not to violate the rules fine. No rule was violated no worries. I am also ok. Lesson learned, let's move on, no hard feelings. > Support user-specified driver and executor pod templates > > > Key: SPARK-24434 > URL: https://issues.apache.org/jira/browse/SPARK-24434 > Project: Spark > Issue Type: New Feature > Components: Kubernetes >Affects Versions: 2.4.0 >Reporter: Yinan Li >Priority: Major > > With more requests for customizing the driver and executor pods coming, the > current approach of adding new Spark configuration options has some serious > drawbacks: 1) it means more Kubernetes specific configuration options to > maintain, and 2) it widens the gap between the declarative model used by > Kubernetes and the configuration model used by Spark. We should start > designing a solution that allows users to specify pod templates as central > places for all customization needs for the driver and executor pods. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional
[jira] [Comment Edited] (SPARK-24434) Support user-specified driver and executor pod templates
[ https://issues.apache.org/jira/browse/SPARK-24434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16599728#comment-16599728 ] Stavros Kontopoulos edited comment on SPARK-24434 at 9/1/18 8:28 PM: - Spark belongs to the community (no?) and should not serve any company's priorities like Palantirs, [~mcheah] we don't need the meeting then if we are going to overlap with each other, fine. I have a question, why serve Palantir's priorities and not mine? This is not healthy :). You didnt collaborate with me why? :) Why nobody respected my will to make it go in 2.4, It was my priority back then too. We are implementing the same design, the whole discussion makes no sense, it is not about mine or your implementation... I am not going to implement the same thing again its not reasonable. But also I wouldnt create a PR that just works, you can check my comments, that was easy, just call load and load the template (it is not rocket science this work). Sorry I dont see any real arguments in the discussion, and as I said I dont want to reply but the politically correct replies leave me no choice. We have talked on slack several times and privately as well. You could always have pinged me but you decided to collaborate on this, without anyone knowing. Probably people don't understand the meaning of fairness, I am not going to explain it here. We can always create any RP we like and then we will see what work is merged, cool. For good or bad though the meeting has power because k8s committers have the final saying on merging no? So I dont agree. The whole discussion for me is pointless, the message communicated, culture and attitude is clear. That is for committers or others in the Spark project not to violate the rules fine. No rule was violated no worries. I am also ok. Lesson learned, let's move on, no hard feelings. was (Author: skonto): Spark belongs to the community (no?) and should not serve any company's priorities like Palantirs, [~mcheah] we don't need the meeting then if we are going to overlap with each other, fine. I have a question, why serve Palantir's priorities and not mine? This is not healthy :). You didnt collaborate with me why? :) Why nobody respected my will to make it go in 2.4, It was my priority back then too. We are implementing the same design, the whole discussion makes no sense, it is not about mine or your implementation... I am not going to implement the same thing again its not reasonable. But also I wouldnt create a PR that just works, you can check my comments, that was easy, just call load and load the template (it is not rocket science this work). Sorry I dont see any real arguments in the discussion, and as I said I dont want to reply but the politically correct replies leave me no choice. We have talked on slack several times and privately as well. You could always have pinged me but you decided to collaborate on this, without anyone knowing. Probably people don't understand the meaning of fairness, I am not going to explain it here. We can always create any RP we like and then we will see what work is merged, cool. For good or bad though the meeting has power because k8s committers have the final saying on merging no? So I dont agree. The whole discussion for me is pointless, the message communicated, culture and attitude is clear. That is for committers or others in the Spark project not to violate the rules fine. No rule was violated no worries. I am also ok. Lesson learned, let's move on. > Support user-specified driver and executor pod templates > > > Key: SPARK-24434 > URL: https://issues.apache.org/jira/browse/SPARK-24434 > Project: Spark > Issue Type: New Feature > Components: Kubernetes >Affects Versions: 2.4.0 >Reporter: Yinan Li >Priority: Major > > With more requests for customizing the driver and executor pods coming, the > current approach of adding new Spark configuration options has some serious > drawbacks: 1) it means more Kubernetes specific configuration options to > maintain, and 2) it widens the gap between the declarative model used by > Kubernetes and the configuration model used by Spark. We should start > designing a solution that allows users to specify pod templates as central > places for all customization needs for the driver and executor pods. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Comment Edited] (SPARK-24434) Support user-specified driver and executor pod templates
[ https://issues.apache.org/jira/browse/SPARK-24434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16599728#comment-16599728 ] Stavros Kontopoulos edited comment on SPARK-24434 at 9/1/18 8:27 PM: - Spark belongs to the community (no?) and should not serve any company's priorities like Palantirs, [~mcheah] we don't need the meeting then if we are going to overlap with each other, fine. I have a question, why serve Palantir's priorities and not mine? This is not healthy :). You didnt collaborate with me why? :) Why nobody respected my will to make it go in 2.4, It was my priority back then too. We are implementing the same design, the whole discussion makes no sense, it is not about mine or your implementation... I am not going to implement the same thing again its not reasonable. But also I wouldnt create a PR that just works, you can check my comments, that was easy, just call load and load the template (it is not rocket science this work). Sorry I dont see any real arguments in the discussion, and as I said I dont want to reply but the politically correct replies leave me no choice. We have talked on slack several times and privately as well. You could always have pinged me but you decided to collaborate on this, without anyone knowing. Probably people don't understand the meaning of fairness, I am not going to explain it here. We can always create any RP we like and then we will see what work is merged, cool. For good or bad though the meeting has power because k8s committers have the final saying on merging no? So I dont agree. The whole discussion for me is pointless, the message communicated, culture and attitude is clear. That is for committers or others in the Spark project not to violate the rules fine. No rule was violated no worries. I am also ok. Lesson learned, let's move on. was (Author: skonto): Spark belongs to the community (no?) and should not serve any company's priorities like Palantirs, [~mcheah] we don't need the meeting then if we are going to overlap with each other, fine. I have a question, why serve Palantir's priorities and not mine? This is not healthy :). You didnt collaborate with me why? :) Why nobody respected my will to make it go in 2.4, It was my priority back then too. We are implementing the same design, the whole discussion makes no sense, it is not about mine or your implementation... I am not going to implement the same thing again its not reasonable. But also I wouldnt create a PR that just works, you can check my comments, that was easy, just call load and load the template (it is not rocket science this work). Sorry I dont see any real arguments in the discussion, and as I said I dont want to reply but replies leave me no choice. We have talked on slack several times and privately as well. You could always have pinged me but you decided to collaborate on this, without anyone knowing. Probably people don't understand the meaning of fairness, I am not going to explain it here. We can always create any RP we like and then we will see what work is merged, cool. For good or bad though the meeting has power because k8s committers have the final saying on merging no? So I dont agree. The whole discussion for me is pointless, the message communicated, culture and attitude is clear. That is for committers or others in the Spark project not to violate the rules fine. No rule was violated no worries. I am also ok. Lesson learned, let's move on. > Support user-specified driver and executor pod templates > > > Key: SPARK-24434 > URL: https://issues.apache.org/jira/browse/SPARK-24434 > Project: Spark > Issue Type: New Feature > Components: Kubernetes >Affects Versions: 2.4.0 >Reporter: Yinan Li >Priority: Major > > With more requests for customizing the driver and executor pods coming, the > current approach of adding new Spark configuration options has some serious > drawbacks: 1) it means more Kubernetes specific configuration options to > maintain, and 2) it widens the gap between the declarative model used by > Kubernetes and the configuration model used by Spark. We should start > designing a solution that allows users to specify pod templates as central > places for all customization needs for the driver and executor pods. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Comment Edited] (SPARK-24434) Support user-specified driver and executor pod templates
[ https://issues.apache.org/jira/browse/SPARK-24434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16599728#comment-16599728 ] Stavros Kontopoulos edited comment on SPARK-24434 at 9/1/18 8:26 PM: - Spark belongs to the community (no?) and should not serve any company's priorities like Palantirs, [~mcheah] we don't need the meeting then if we are going to overlap with each other, fine. I have a question, why serve Palantir's priorities and not mine? This is not healthy :). You didnt collaborate with me why? :) Why nobody respected my will to make it go in 2.4, It was my priority back then too. We are implementing the same design, the whole discussion makes no sense, it is not about mine or your implementation... I am not going to implement the same thing again its not reasonable. But also I wouldnt create a PR that just works, you can check my comments, that was easy, just call load and load the template (it is not rocket science this work). Sorry I dont see any real arguments in the discussion, and as I said I dont want to reply but replies leave me no choice. We have talked on slack several times and privately as well. You could always have pinged me but you decided to collaborate on this, without anyone knowing. Probably people don't understand the meaning of fairness, I am not going to explain it here. We can always create any RP we like and then we will see what work is merged, cool. For good or bad though the meeting has power because k8s committers have the final saying on merging no? So I dont agree. The whole discussion for me is pointless, the message communicated, culture and attitude is clear. That is for committers or others in the Spark project not to violate the rules fine. No rule was violated no worries. I am also ok. Lesson learned, let's move on. was (Author: skonto): Spark belongs to the community (no?) and should not serve any company's priorities like Palantirs, [~mcheah] we don't need the meeting then if we are going to overlap with each other, fine. I have a question, why serve Palantir's priorities and not mine? This is not healthy :). You didnt collaborate with me why? :) We are implementing the same design, the whole discussion makes no sense, it is not about mine or your implementation... I am not going to implement the same thing again its not reasonable. But also I wouldnt create a PR that just works, you can check my comments, that was easy, just call load and load the template (it is not rocket science this work). Sorry I dont see any real arguments in the discussion, and as I said I dont want to reply but replies leave me no choice. We have talked on slack several times and privately as well. You could always have pinged me but you decided to collaborate on this, without anyone knowing. Probably people don't understand the meaning of fairness, I am not going to explain it here. We can always create any RP we like and then we will see what work is merged, cool. For good or bad though the meeting has power because k8s committers have the final saying on merging no? So I dont agree. The whole discussion for me is pointless, the message communicated, culture and attitude is clear. That is for committers or others in the Spark project not to violate the rules fine. No rule was violated no worries. I am also ok. Lesson learned, let's move on. > Support user-specified driver and executor pod templates > > > Key: SPARK-24434 > URL: https://issues.apache.org/jira/browse/SPARK-24434 > Project: Spark > Issue Type: New Feature > Components: Kubernetes >Affects Versions: 2.4.0 >Reporter: Yinan Li >Priority: Major > > With more requests for customizing the driver and executor pods coming, the > current approach of adding new Spark configuration options has some serious > drawbacks: 1) it means more Kubernetes specific configuration options to > maintain, and 2) it widens the gap between the declarative model used by > Kubernetes and the configuration model used by Spark. We should start > designing a solution that allows users to specify pod templates as central > places for all customization needs for the driver and executor pods. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Comment Edited] (SPARK-24434) Support user-specified driver and executor pod templates
[ https://issues.apache.org/jira/browse/SPARK-24434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16599728#comment-16599728 ] Stavros Kontopoulos edited comment on SPARK-24434 at 9/1/18 8:25 PM: - Spark belongs to the community (no?) and should not serve any company's priorities like Palantirs, [~mcheah] we don't need the meeting then if we are going to overlap with each other, fine. I have a question, why serve Palantir's priorities and not mine? This is not healthy :). You didnt collaborate with me why? :) We are implementing the same design, the whole discussion makes no sense, it is not about mine or your implementation... I am not going to implement the same thing again its not reasonable. But also I wouldnt create a PR that just works, you can check my comments, that was easy, just call load and load the template (it is not rocket science this work). Sorry I dont see any real arguments in the discussion, and as I said I dont want to reply but replies leave me no choice. We have talked on slack several times and privately as well. You could always have pinged me but you decided to collaborate on this, without anyone knowing. Probably people don't understand the meaning of fairness, I am not going to explain it here. We can always create any RP we like and then we will see what work is merged, cool. For good or bad though the meeting has power because k8s committers have the final saying on merging no? So I dont agree. The whole discussion for me is pointless, the message communicated, culture and attitude is clear. That is for committers or others in the Spark project not to violate the rules fine. No rule was violated no worries. I am also ok. Lesson learned, let's move on. was (Author: skonto): Spark belongs to the community (no?) and should not serve any company's priorities like Palantirs, [~mcheah] we don't need the meeting then if we are going to overlap with each other, fine. I have a question, why serve Palantir's priorities and not mine? This is not healthy :). You didnt collaborate with me why? :) We are implementing the same design, the whole discussion makes no sense, it is not about mine or your implementation... I am not going to implement the same thing again its not reasonable. But also I wouldnt create a PR that just works, you can check my comments, that was easy, just call load and load the template (it is not rocket science this work). Sorry I dont see any real arguments in the discussion, and as I said I dont want to reply but replies leave me no choice. We have talked on slack several times and privately as well. You could always have pinged me but you decided to collaborate on this, without anyone knowing. Probably people don't understand the meaning of fairness, I am not going to explain it here. We can always create any RP we like and then we will see what work is merged, cool. For good or bad though the meeting has power because k8s committers have the final saying on merging no? So I dont agree. The whole discussion for me is pointless, the message communicated, culture and attitude is clear. The only point is for committers or others in the Spark project not to violate the rules fine. No rule was violated no worries. I am also ok. Lesson learned, let's move on. > Support user-specified driver and executor pod templates > > > Key: SPARK-24434 > URL: https://issues.apache.org/jira/browse/SPARK-24434 > Project: Spark > Issue Type: New Feature > Components: Kubernetes >Affects Versions: 2.4.0 >Reporter: Yinan Li >Priority: Major > > With more requests for customizing the driver and executor pods coming, the > current approach of adding new Spark configuration options has some serious > drawbacks: 1) it means more Kubernetes specific configuration options to > maintain, and 2) it widens the gap between the declarative model used by > Kubernetes and the configuration model used by Spark. We should start > designing a solution that allows users to specify pod templates as central > places for all customization needs for the driver and executor pods. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Comment Edited] (SPARK-24434) Support user-specified driver and executor pod templates
[ https://issues.apache.org/jira/browse/SPARK-24434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16599728#comment-16599728 ] Stavros Kontopoulos edited comment on SPARK-24434 at 9/1/18 8:25 PM: - Spark belongs to the community (no?) and should not serve any company's priorities like Palantirs, [~mcheah] we don't need the meeting then if we are going to overlap with each other, fine. I have a question, why serve Palantir's priorities and not mine? This is not healthy :). You didnt collaborate with me why? :) We are implementing the same design, the whole discussion makes no sense, it is not about mine or your implementation... I am not going to implement the same thing again its not reasonable. But also I wouldnt create a PR that just works, you can check my comments, that was easy, just call load and load the template (it is not rocket science this work). Sorry I dont see any real arguments in the discussion, and as I said I dont want to reply but replies leave me no choice. We have talked on slack several times and privately as well. You could always have pinged me but you decided to collaborate on this, without anyone knowing. Probably people don't understand the meaning of fairness, I am not going to explain it here. We can always create any RP we like and then we will see what work is merged, cool. For good or bad though the meeting has power because k8s committers have the final saying on merging no? So I dont agree. The whole discussion for me is pointless, the message communicated, culture and attitude is clear. That is for committers or others in the Spark project not to violate the rules fine. No rule was violated no worries. I am also ok. Lesson learned, let's move on. was (Author: skonto): Spark belongs to the community (no?) and should not serve any company's priorities like Palantirs, [~mcheah] we don't need the meeting then if we are going to overlap with each other, fine. I have a question, why serve Palantir's priorities and not mine? This is not healthy :). You didnt collaborate with me why? :) We are implementing the same design, the whole discussion makes no sense, it is not about mine or your implementation... I am not going to implement the same thing again its not reasonable. But also I wouldnt create a PR that just works, you can check my comments, that was easy, just call load and load the template (it is not rocket science this work). Sorry I dont see any real arguments in the discussion, and as I said I dont want to reply but replies leave me no choice. We have talked on slack several times and privately as well. You could always have pinged me but you decided to collaborate on this, without anyone knowing. Probably people don't understand the meaning of fairness, I am not going to explain it here. We can always create any RP we like and then we will see what work is merged, cool. For good or bad though the meeting has power because k8s committers have the final saying on merging no? So I dont agree. The whole discussion for me is pointless, the message communicated, culture and attitude is clear. That is for committers or others in the Spark project not to violate the rules fine. No rule was violated no worries. I am also ok. Lesson learned, let's move on. > Support user-specified driver and executor pod templates > > > Key: SPARK-24434 > URL: https://issues.apache.org/jira/browse/SPARK-24434 > Project: Spark > Issue Type: New Feature > Components: Kubernetes >Affects Versions: 2.4.0 >Reporter: Yinan Li >Priority: Major > > With more requests for customizing the driver and executor pods coming, the > current approach of adding new Spark configuration options has some serious > drawbacks: 1) it means more Kubernetes specific configuration options to > maintain, and 2) it widens the gap between the declarative model used by > Kubernetes and the configuration model used by Spark. We should start > designing a solution that allows users to specify pod templates as central > places for all customization needs for the driver and executor pods. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Comment Edited] (SPARK-24434) Support user-specified driver and executor pod templates
[ https://issues.apache.org/jira/browse/SPARK-24434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16599728#comment-16599728 ] Stavros Kontopoulos edited comment on SPARK-24434 at 9/1/18 8:24 PM: - Spark belongs to the community (no?) and should not serve any company's priorities like Palantirs, [~mcheah] we don't need the meeting then if we are going to overlap with each other, fine. I have a question, why serve Palantir's priorities and not mine? This is not healthy :). You didnt collaborate with me why? :) We are implementing the same design, the whole discussion makes no sense, it is not about mine or your implementation... I am not going to implement the same thing again its not reasonable. But also I wouldnt create a PR that just works, you can check my comments, that was easy, just call load and load the template (it is not rocket science this work). Sorry I dont see any real arguments in the discussion, and as I said I dont want to reply but replies leave me no choice. We have talked on slack several times and privately as well. You could always have pinged me but you decided to collaborate on this, without anyone knowing. Probably people don't understand the meaning of fairness, I am not going to explain it here. We can always create any RP we like and then we will see what work is merged, cool. For good or bad though the meeting has power because k8s committers have the final saying on merging no? So I dont agree. The whole discussion for me is pointless, the message communicated, culture and attitude is clear. The only point is for committers or others in the Spark project not to violate the rules fine. No rule was violated no worries. I am also ok. Lesson learned, let's move on. was (Author: skonto): Spark belongs to the community (no?) and should not serve any company's priorities like Palantirs, [~mcheah] we don't need the meeting then if we are going to overlap with each other, fine. I have question why serve Palantir's priorities and not mine? This is not healthy :). You didnt collaborate with me why? :) We are implementing the same design, the whole discussion makes no sense, it is not about mine or your implementation... I am not going to implement the same thing again its not reasonable. But also I wouldnt create a PR that just works, you can check my comments, that was easy, just call load and load the template (it is not rocket science this work). Sorry I dont see any real arguments in the discussion, and as I said I dont want to reply but replies leave me no choice. We have talked on slack several times and privately as well. You could always have pinged me but you decided to collaborate on this, without anyone knowing. Probably people don't understand the meaning of fairness, I am not going to explain it here. We can always create any RP we like and then we will see what work is merged, cool. For good or bad though the meeting has power because k8s committers have the final saying on merging no? So I dont agree. The whole discussion for me is pointless, the message communicated, culture and attitude is clear. The only point is for committers or others in the Spark project not to violate the rules fine. No rule was violated no worries. I am also ok. Lesson learned, let's move on. > Support user-specified driver and executor pod templates > > > Key: SPARK-24434 > URL: https://issues.apache.org/jira/browse/SPARK-24434 > Project: Spark > Issue Type: New Feature > Components: Kubernetes >Affects Versions: 2.4.0 >Reporter: Yinan Li >Priority: Major > > With more requests for customizing the driver and executor pods coming, the > current approach of adding new Spark configuration options has some serious > drawbacks: 1) it means more Kubernetes specific configuration options to > maintain, and 2) it widens the gap between the declarative model used by > Kubernetes and the configuration model used by Spark. We should start > designing a solution that allows users to specify pod templates as central > places for all customization needs for the driver and executor pods. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Comment Edited] (SPARK-24434) Support user-specified driver and executor pod templates
[ https://issues.apache.org/jira/browse/SPARK-24434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16599728#comment-16599728 ] Stavros Kontopoulos edited comment on SPARK-24434 at 9/1/18 8:23 PM: - Spark belongs to the community (no?) and should not serve any company's priorities like Palantirs, [~mcheah] we don't need the meeting then if we are going to overlap with each other, fine. I have question why serve Palantir's priorities and not mine? This is not healthy :). You didnt collaborate with me why? :) We are implementing the same design, the whole discussion makes no sense, it is not about mine or your implementation... I am not going to implement the same thing again its not reasonable. But also I wouldnt create a PR that just works, you can check my comments, that was easy, just call load and load the template (it is not rocket science this work). Sorry I dont see any real arguments in the discussion, and as I said I dont want to reply but replies leave me no choice. We have talked on slack several times and privately as well. You could always have pinged me but you decided to collaborate on this, without anyone knowing. Probably people don't understand the meaning of fairness, I am not going to explain it here. We can always create any RP we like and then we will see what work is merged, cool. For good or bad though the meeting has power because k8s committers have the final saying on merging no? So I dont agree. The whole discussion for me is pointless, the message communicated, culture and attitude is clear. The only point is for committers or others in the Spark project not to violate the rules fine. No rule was violated no worries. I am also ok. Lesson learned, let's move on. was (Author: skonto): Spark belongs to the community (no?) and should not serve any company's priorities like Palantirs, [~mcheah] we don't need the meeting then if we are going to overlap with each other, fine. I have question why serve Palantir's priorities and not mine? This is not healthy :). You didnt collaborate with me why? :) We are implementing the same design, the whole discussion makes no sense, it is not about mine or your implementation... I am not going to implement the same thing again its not reasonable. But also I wouldnt create a PR that just works, you can check my comments, that was easy, just call load and load the template (it is not rocket science this work). Sorry I dont see any real arguments in the discussion, and as I said I dont want to reply but replies leave me no choice. We have talked on slack several times and privately. You could always have pinged me but you decided to collaborate on this, without anyone knowing. Probably people don't understand the meaning of fairness, I am not going to explain it here. We can always create any RP we like and then we will see what work is merged, cool. For good or bad though the meeting has power because k8s committers have the final saying on merging no? So I dont agree. The whole discussion for me is pointless, the message communicated, culture and attitude is clear. The only point is for committers or others in the Spark project not to violate the rules fine. No rule was violated no worries. I am also ok. Lesson learned, let's move on. > Support user-specified driver and executor pod templates > > > Key: SPARK-24434 > URL: https://issues.apache.org/jira/browse/SPARK-24434 > Project: Spark > Issue Type: New Feature > Components: Kubernetes >Affects Versions: 2.4.0 >Reporter: Yinan Li >Priority: Major > > With more requests for customizing the driver and executor pods coming, the > current approach of adding new Spark configuration options has some serious > drawbacks: 1) it means more Kubernetes specific configuration options to > maintain, and 2) it widens the gap between the declarative model used by > Kubernetes and the configuration model used by Spark. We should start > designing a solution that allows users to specify pod templates as central > places for all customization needs for the driver and executor pods. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Comment Edited] (SPARK-24434) Support user-specified driver and executor pod templates
[ https://issues.apache.org/jira/browse/SPARK-24434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16599728#comment-16599728 ] Stavros Kontopoulos edited comment on SPARK-24434 at 9/1/18 8:21 PM: - Spark belongs to the community (no?) and should not serve any company's priorities like Palantirs, [~mcheah] we don't need the meeting then if we are going to overlap with each other, fine. I have question why serve Palantir's priorities and not mine? This is not healthy :). You didnt collaborate with me why? :) We are implementing the same design, the whole discussion makes no sense, it is not about mine or your implementation... I am not going to implement the same thing again its not reasonable. But also I wouldnt create a PR that just works, you can check my comments, that was easy, just call load and load the template (it is not rocket science this work). Sorry I dont see any real arguments in the discussion, and as I said I dont want to reply but replies leave me no choice. We have talked on slack several times and privately. You could always have pinged me but you decided to collaborate on this, without anyone knowing. Probably people don't understand the meaning of fairness, I am not going to explain it here. We can always create any RP we like and then we will see what work is merged, cool. For good or bad though the meeting has power because k8s committers have the final saying on merging no? So I dont agree. The whole discussion for me is pointless, the message communicated, culture and attitude is clear. The only point is for committers or others in the Spark project not to violate the rules fine. No rule was violated no worries. I am also ok. Lesson learned, let's move on. was (Author: skonto): Spark belongs to the community (no?) and should not serve any company's priorities like Palantirs, [~mcheah] we don't need the meeting then if we are going to overlap with each other, fine. I have question why serve Palantir's priorities and not mine? This is not healthy :). You didnt collaborate with me why? :) We are implementing the same design, the whole discussion makes no sense, it is not about mine or your implementation... Sorry I dont see any real arguments in the discussion, and as I said I dont want to reply but replies leave me no choice. We have talked on slack several times and privately. You could always have pinged me but you decided to collaborate on this, without anyone knowing. Probably people don't understand the meaning of fairness, I am not going to explain it here. We can always create any RP we like and then we will see what work is merged, cool. For good or bad though the meeting has power because k8s committers have the final saying on merging no? So I dont agree. The whole discussion for me is pointless, the message communicated, culture and attitude is clear. The only point is for committers or others in the Spark project not to violate the rules fine. No rule was violated no worries. I am also ok. Lesson learned, let's move on. > Support user-specified driver and executor pod templates > > > Key: SPARK-24434 > URL: https://issues.apache.org/jira/browse/SPARK-24434 > Project: Spark > Issue Type: New Feature > Components: Kubernetes >Affects Versions: 2.4.0 >Reporter: Yinan Li >Priority: Major > > With more requests for customizing the driver and executor pods coming, the > current approach of adding new Spark configuration options has some serious > drawbacks: 1) it means more Kubernetes specific configuration options to > maintain, and 2) it widens the gap between the declarative model used by > Kubernetes and the configuration model used by Spark. We should start > designing a solution that allows users to specify pod templates as central > places for all customization needs for the driver and executor pods. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Comment Edited] (SPARK-24434) Support user-specified driver and executor pod templates
[ https://issues.apache.org/jira/browse/SPARK-24434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16599728#comment-16599728 ] Stavros Kontopoulos edited comment on SPARK-24434 at 9/1/18 8:17 PM: - Spark belongs to the community (no?) and should not serve any company's priorities like Palantirs, [~mcheah] we don't need the meeting then if we are going to overlap with each other, fine. I have question why serve Palantir's priorities and not mine? This is not healthy :). You didnt collaborate with me why? :) We are implementing the same design, the whole discussion makes no sense, it is not about mine or your implementation... Sorry I dont see any real arguments in the discussion, and as I said I dont want to reply but replies leave me no choice. We have talked on slack several times and privately. You could always have pinged me but you decided to collaborate on this, without anyone knowing. Probably people don't understand the meaning of fairness, I am not going to explain it here. We can always create any RP we like and then we will see what work is merged, cool. For good or bad though the meeting has power because k8s committers have the final saying on merging no? So I dont agree. The whole discussion for me is pointless, the message communicated, culture and attitude is clear. The only point is for committers or others in the Spark project not to violate the rules fine. No rule was violated no worries. I am also ok. Lesson learned, let's move on. was (Author: skonto): Spark belongs to the community (no?) and should not serve any company's priorities like Palantirs, [~mcheah] we don't need the meeting then if we are going to overlap with each other, fine. I have question why serve Palantir's priorities and not mine? This is not healthy :). You didnt collaborate with me why? :) We are implementing the same design, the whole discussion makes no sense, it is not about mine or your implementation... Sorry I dont see any real arguments in the discussion, and as I said I dont want to reply but replies leave me no choice. We have talked on slack several times and privately. You could always have pinged me but you decided to collaborate on this, without anyone knowing. Probably people don't understand the meaning of fairness, I am not going to explain it here. We can always create any RP we like and then we will see what work is merged, cool. For good or bad though the meeting has power because k8s committers have the final saying on merging no? So I dont agree. The whole discussion for me is pointless, the message communicated, culture and attitude is clear. The only point is for committers or others in the Spark project not to violate the rules fine. No rule was violated no worries. I am also ok. Lesson learned, let's move on. > Support user-specified driver and executor pod templates > > > Key: SPARK-24434 > URL: https://issues.apache.org/jira/browse/SPARK-24434 > Project: Spark > Issue Type: New Feature > Components: Kubernetes >Affects Versions: 2.4.0 >Reporter: Yinan Li >Priority: Major > > With more requests for customizing the driver and executor pods coming, the > current approach of adding new Spark configuration options has some serious > drawbacks: 1) it means more Kubernetes specific configuration options to > maintain, and 2) it widens the gap between the declarative model used by > Kubernetes and the configuration model used by Spark. We should start > designing a solution that allows users to specify pod templates as central > places for all customization needs for the driver and executor pods. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Comment Edited] (SPARK-24434) Support user-specified driver and executor pod templates
[ https://issues.apache.org/jira/browse/SPARK-24434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16599728#comment-16599728 ] Stavros Kontopoulos edited comment on SPARK-24434 at 9/1/18 8:14 PM: - Spark belongs to the community (no?) and should not serve any company's priorities like Palantirs, [~mcheah] we don't need the meeting then if we are going to overlap with each other, fine. I have question why serve Palantir's priorities and not mine? This is not healthy :). You didnt collaborate with me why? :) We are implementing the same design, the whole discussion makes no sense, it is not about mine or your implementation... Sorry I dont see any real arguments in the discussion, and as I said I dont want to reply but replies leave me no choice. We have talked on slack several times and privately. You could always have pinged me but you decided to collaborate on this, without anyone knowing. Probably people don't understand the meaning of fairness, I am not going to explain it here. We can always create any RP we like and then we will see what work is merged, cool. For good or bad though the meeting has power because k8s committers have the final saying on merging no? So I dont agree. The whole discussion for me is pointless, the message communicated, culture and attitude is clear. The only point is for committers or others in the Spark project not to violate the rules fine. No rule was violated no worries. I am also ok. Lesson learned, let's move on. was (Author: skonto): Spark belongs to the community (no?) and should not serve any company's priorities like Palantirs, [~mcheah] we don't need the meeting then if we are going to overlap with each other, fine. I have question why serve Palantir's priorities and not mine? This is not healthy :). You didnt collaborate with me why? :) We are implementing the same design, the whole discussion makes no sense, it is not about mine or your implementation... Sorry I dont see any real arguments in the discussion, and as I said I dont want to reply but replies leave me no choice. We have talked on slack several times and privately. You could always have pinged me but you decided to collaborate on this, without anyone knowing. Probably people don't understand the meaning of fairness, I am not going to explain it here. We can always create any RP we like and then we will see what work is merged, cool. For good or bad though the meeting has power because k8s committers have the final saying on merging no? So I dont agree. The whole discussion for me is pointless, the message culture and attitude is clear. The only point is for committers, the Spark project not to violate the rules fine. > Support user-specified driver and executor pod templates > > > Key: SPARK-24434 > URL: https://issues.apache.org/jira/browse/SPARK-24434 > Project: Spark > Issue Type: New Feature > Components: Kubernetes >Affects Versions: 2.4.0 >Reporter: Yinan Li >Priority: Major > > With more requests for customizing the driver and executor pods coming, the > current approach of adding new Spark configuration options has some serious > drawbacks: 1) it means more Kubernetes specific configuration options to > maintain, and 2) it widens the gap between the declarative model used by > Kubernetes and the configuration model used by Spark. We should start > designing a solution that allows users to specify pod templates as central > places for all customization needs for the driver and executor pods. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Comment Edited] (SPARK-24434) Support user-specified driver and executor pod templates
[ https://issues.apache.org/jira/browse/SPARK-24434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16599728#comment-16599728 ] Stavros Kontopoulos edited comment on SPARK-24434 at 9/1/18 8:13 PM: - Spark belongs to the community (no?) and should not serve any company's priorities like Palantirs, [~mcheah] we don't need the meeting then if we are going to overlap with each other, fine. I have question why serve Palantir's priorities and not mine? This is not healthy :). You didnt collaborate with me why? :) We are implementing the same design, the whole discussion makes no sense, it is not about mine or your implementation... Sorry I dont see any real arguments in the discussion, and as I said I dont want to reply but replies leave me no choice. We have talked on slack several times and privately. You could always have pinged me but you decided to collaborate on this, without anyone knowing. Probably people don't understand the meaning of fairness, I am not going to explain it here. We can always create any RP we like and then we will see what work is merged, cool. For good or bad though the meeting has power because k8s committers have the final saying on merging no? So I dont agree. The whole discussion for me is pointless, the message culture and attitude is clear. The only point is for committers, the Spark project not to violate the rules fine. was (Author: skonto): Spark belongs to the community (no?) and should not serve any company's priorities like Palantirs, [~mcheah] we don't need the meeting then if we are going to overlap with each other, fine. I have question why serve Palantir's priorities and not mine? This is not healthy :). You didnt collaborate with me why? :) We are implementing the same design, the whole discussion makes no sense, it is not about mine or your implementation... Sorry I dont see any rela arguments in the discussion, and as I said I dont want to reply but you leave me no choice. We have talked on slack several times and privately. You could always have pinged me but you decided to collaborate on this, without anyone knowing. Probably people don't understand the meaning of fairness, I am not going to explain it here. We can always create any RP we like and then we will see what work is merged, cool. For good or bad though the meeting has power because k8s committers have the final saying on merging no? So I dont agree. The whole discussion for me is pointless, the message culture and attitude is clear. The only point is for committers, the Spark project not to violate the rules fine. > Support user-specified driver and executor pod templates > > > Key: SPARK-24434 > URL: https://issues.apache.org/jira/browse/SPARK-24434 > Project: Spark > Issue Type: New Feature > Components: Kubernetes >Affects Versions: 2.4.0 >Reporter: Yinan Li >Priority: Major > > With more requests for customizing the driver and executor pods coming, the > current approach of adding new Spark configuration options has some serious > drawbacks: 1) it means more Kubernetes specific configuration options to > maintain, and 2) it widens the gap between the declarative model used by > Kubernetes and the configuration model used by Spark. We should start > designing a solution that allows users to specify pod templates as central > places for all customization needs for the driver and executor pods. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Comment Edited] (SPARK-24434) Support user-specified driver and executor pod templates
[ https://issues.apache.org/jira/browse/SPARK-24434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16599728#comment-16599728 ] Stavros Kontopoulos edited comment on SPARK-24434 at 9/1/18 8:12 PM: - Spark belongs to the community (no?) and should not serve any company's priorities like Palantirs, [~mcheah] we don't need the meeting then if we are going to overlap with each other, fine. I have question why serve Palantir's priorities and not mine? This is not healthy :). You didnt collaborate with me why? :) We are implementing the same design, the whole discussion makes no sense, it is not about mine or your implementation... Sorry I dont see any rela arguments in the discussion, and as I said I dont want to reply but you leave me no choice. We have talked on slack several times and privately. You could always have pinged me but you decided to collaborate on this, without anyone knowing. Probably people don't understand the meaning of fairness, I am not going to explain it here. We can always create any RP we like and then we will see what work is merged, cool. For good or bad though the meeting has power because k8s committers have the final saying on merging no? So I dont agree. The whole discussion for me is pointless, the message culture and attitude is clear. The only point is for committers, the Spark project not to violate the rules fine. was (Author: skonto): Spark belongs to the community (no?) and should not serve any company's priorities like Palantirs, [~mcheah] we don't need the meeting then if we are going to overlap with each other, fine. I have question why serve Palantir's priorities and not mine? This is not healthy :). You didnt collaborate with me why? :) We are implementing the same design, the whole discussion makes no sense, it is not about my or your implementation... We have talked on slack several times and privately. You could always have pinged me but you decided to collaborate on this, without anyone knowing. Probably people don't understand the meaning of fairness, I am not going to explain it here. We can always create any RP we like and then we will see what work is merged, cool. For good or bad though the meeting has power because k8s committers have the final saying on merging no? So I dont agree. The whole discussion for me is pointless, the message culture and attitude is clear. The only point is for committers, the Spark project not to violate the rules fine. > Support user-specified driver and executor pod templates > > > Key: SPARK-24434 > URL: https://issues.apache.org/jira/browse/SPARK-24434 > Project: Spark > Issue Type: New Feature > Components: Kubernetes >Affects Versions: 2.4.0 >Reporter: Yinan Li >Priority: Major > > With more requests for customizing the driver and executor pods coming, the > current approach of adding new Spark configuration options has some serious > drawbacks: 1) it means more Kubernetes specific configuration options to > maintain, and 2) it widens the gap between the declarative model used by > Kubernetes and the configuration model used by Spark. We should start > designing a solution that allows users to specify pod templates as central > places for all customization needs for the driver and executor pods. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Comment Edited] (SPARK-24434) Support user-specified driver and executor pod templates
[ https://issues.apache.org/jira/browse/SPARK-24434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16599728#comment-16599728 ] Stavros Kontopoulos edited comment on SPARK-24434 at 9/1/18 8:11 PM: - Spark belongs to the community (no?) and should not serve any company's priorities like Palantirs, [~mcheah] we don't need the meeting then if we are going to overlap with each other, fine. I have question why serve Palantir's priorities and not mine? This is not healthy :). You didnt collaborate with me why? :) We are implementing the same design, the whole discussion makes no sense, it is not about my or your implementation... We have talked on slack several times and privately. You could always have pinged me but you decided to collaborate on this, without anyone knowing. Probably people don't understand the meaning of fairness, I am not going to explain it here. We can always create any RP we like and then we will see what work is merged, cool. For good or bad though the meeting has power because k8s committers have the final saying on merging no? So I dont agree. The whole discussion for me is pointless, the message culture and attitude is clear. The only point is for committers, the Spark project not to violate the rules fine. was (Author: skonto): Spark belongs to the community (no?) and should not serve any company's priorities like Palantirs, [~mcheah] we don't need the meeting then if we are going to overlap with each other, fine. I have question why serve Palantir's priorities and not mine? This is not healthy :). You didnt collaborate with me why? :) We have talked on slack several times and privately. You could always have pinged me but you decided to collaborate on this, without anyone knowing. Probably people don't understand the meaning of fairness, I am not going to explain it here. We can always create any RP we like and then we will see what work is merged, cool. For good or bad though the meeting has power because k8s committers have the final saying on merging no? So I dont agree. The whole discussion for me is pointless, the message culture and attitude is clear. The only point is for committers, the Spark project not to violate the rules fine. > Support user-specified driver and executor pod templates > > > Key: SPARK-24434 > URL: https://issues.apache.org/jira/browse/SPARK-24434 > Project: Spark > Issue Type: New Feature > Components: Kubernetes >Affects Versions: 2.4.0 >Reporter: Yinan Li >Priority: Major > > With more requests for customizing the driver and executor pods coming, the > current approach of adding new Spark configuration options has some serious > drawbacks: 1) it means more Kubernetes specific configuration options to > maintain, and 2) it widens the gap between the declarative model used by > Kubernetes and the configuration model used by Spark. We should start > designing a solution that allows users to specify pod templates as central > places for all customization needs for the driver and executor pods. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Comment Edited] (SPARK-24434) Support user-specified driver and executor pod templates
[ https://issues.apache.org/jira/browse/SPARK-24434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16599728#comment-16599728 ] Stavros Kontopoulos edited comment on SPARK-24434 at 9/1/18 8:10 PM: - Spark belongs to the community (no?) and should not serve any company's priorities like Palantirs, [~mcheah] we don't need the meeting then if we are going to overlap with each other, fine. I have question why serve Palantir's priorities and not mine? This is not healthy :). You didnt collaborate with me why? :) We have talked on slack several times and privately. You could always have pinged me but you decided to collaborate on this, without anyone knowing. Probably people don't understand the meaning of fairness, I am not going to explain it here. We can always create any RP we like and then we will see what work is merged, cool. For good or bad though the meeting has power because k8s committers have the final saying on merging no? So I dont agree. The whole discussion for me is pointless, the message culture and attitude is clear. The only point is for committers, the Spark project not to violate the rules fine. was (Author: skonto): Spark belongs to the community (no?) and should not serve any company's priorities like Palantirs, [~mcheah] we don't need the meeting then if we are going to overlap with each other, fine. I have question why serve Palantir's priorities and not mine? This is not healthy :). We have talked on slack several times and privately. You could always have pinged me but you decided to collaborate on this, without anyone knowing. Probably people don't understand the meaning of fairness, I am not going to explain it here. We can always create any RP we like and then we will see what work is merged, cool. For good or bad though the meeting has power because k8s committers have the final saying on merging no? So I dont agree. The whole discussion for me is pointless, the message culture and attitude is clear. The only point is for committers, the Spark project not to violate the rules fine. > Support user-specified driver and executor pod templates > > > Key: SPARK-24434 > URL: https://issues.apache.org/jira/browse/SPARK-24434 > Project: Spark > Issue Type: New Feature > Components: Kubernetes >Affects Versions: 2.4.0 >Reporter: Yinan Li >Priority: Major > > With more requests for customizing the driver and executor pods coming, the > current approach of adding new Spark configuration options has some serious > drawbacks: 1) it means more Kubernetes specific configuration options to > maintain, and 2) it widens the gap between the declarative model used by > Kubernetes and the configuration model used by Spark. We should start > designing a solution that allows users to specify pod templates as central > places for all customization needs for the driver and executor pods. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Comment Edited] (SPARK-24434) Support user-specified driver and executor pod templates
[ https://issues.apache.org/jira/browse/SPARK-24434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16599728#comment-16599728 ] Stavros Kontopoulos edited comment on SPARK-24434 at 9/1/18 8:10 PM: - Spark belongs to the community (no?) and should not serve any company's priorities like Palantirs, [~mcheah] we don't need the meeting then if we are going to overlap with each other, fine. I have question why serve Palantir's priorities and not mine? This is not healthy :). We have talked on slack several times and privately. You could always have pinged me but you decided to collaborate on this, without anyone knowing. Probably people don't understand the meaning of fairness, I am not going to explain it here. We can always create any RP we like and then we will see what work is merged, cool. For good or bad though the meeting has power because k8s committers have the final saying on merging no? So I dont agree. The whole discussion for me is pointless, the message culture and attitude is clear. The only point is for committers, the Spark project not to violate the rules fine. was (Author: skonto): Spark belongs to the community (no?) and should not serve any company's priorities like Palantirs, [~mcheah] we don't need the meeting then if we are going to overlap with each other, fine. I have question why sever Palantir's priorities and not mine? This is not healthy :). We have talked on slack several times and privately. You could always have pinged me but you decided to collaborate on this, without anyone knowing. Probably people don't understand the meaning of fairness, I am not going to explain it here. We can always create any RP we like and then we will see what work is merged, cool. For good or bad though the meeting has power because k8s committers have the final saying on merging no? So I dont agree. The whole discussion for me is pointless, the message culture and attitude is clear. The only point is for committers, the Spark project not to violate the rules fine. > Support user-specified driver and executor pod templates > > > Key: SPARK-24434 > URL: https://issues.apache.org/jira/browse/SPARK-24434 > Project: Spark > Issue Type: New Feature > Components: Kubernetes >Affects Versions: 2.4.0 >Reporter: Yinan Li >Priority: Major > > With more requests for customizing the driver and executor pods coming, the > current approach of adding new Spark configuration options has some serious > drawbacks: 1) it means more Kubernetes specific configuration options to > maintain, and 2) it widens the gap between the declarative model used by > Kubernetes and the configuration model used by Spark. We should start > designing a solution that allows users to specify pod templates as central > places for all customization needs for the driver and executor pods. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Comment Edited] (SPARK-24434) Support user-specified driver and executor pod templates
[ https://issues.apache.org/jira/browse/SPARK-24434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16599728#comment-16599728 ] Stavros Kontopoulos edited comment on SPARK-24434 at 9/1/18 8:09 PM: - Spark belongs to the community (no?) and should not serve any company's priorities like Palantirs, [~mcheah] we don't need the meeting then if we are going to overlap with each other, fine. I have question why sever Palantir's priorities and not mine? This is not healthy :). We have talked on slack several times and privately. You could always have pinged me but you decided to collaborate on this, without anyone knowing. Probably people don't understand the meaning of fairness, I am not going to explain it here. We can always create any RP we like and then we will see what work is merged, cool. For good or bad though the meeting has power because k8s committers have the final saying on merging no? So I dont agree. The whole discussion for me is pointless, the message culture and attitude is clear. The only point is for committers, the Spark project not to violate the rules fine. was (Author: skonto): Spark belongs to the community (no?) and should not serve any company's priorities like Palantirs, [~mcheah] we don't need the meeting then if we are going to overlap with each other, fine. I have question why sever Palantir's priorities and not mine? This is not healthy :). We have talked on slack several times and privately. YOu could always have pinged but you decided to collaborate on this without anyone knowing. Probably people don't understand the meaning of fairness, I am not going to explain it here. We can always create any RP we like and then we will see what work is merged, cool. For good or bad though the meeting has power because k8s committers have the final saying on merging no? So I dont agree. The whole discussion for me is pointless, the message culture and attitude is clear. The only point is for committers, the Spark project not to violate the rules fine. > Support user-specified driver and executor pod templates > > > Key: SPARK-24434 > URL: https://issues.apache.org/jira/browse/SPARK-24434 > Project: Spark > Issue Type: New Feature > Components: Kubernetes >Affects Versions: 2.4.0 >Reporter: Yinan Li >Priority: Major > > With more requests for customizing the driver and executor pods coming, the > current approach of adding new Spark configuration options has some serious > drawbacks: 1) it means more Kubernetes specific configuration options to > maintain, and 2) it widens the gap between the declarative model used by > Kubernetes and the configuration model used by Spark. We should start > designing a solution that allows users to specify pod templates as central > places for all customization needs for the driver and executor pods. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Comment Edited] (SPARK-24434) Support user-specified driver and executor pod templates
[ https://issues.apache.org/jira/browse/SPARK-24434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16599728#comment-16599728 ] Stavros Kontopoulos edited comment on SPARK-24434 at 9/1/18 8:08 PM: - Spark belongs to the community (no?) and should not serve any company's priorities like Palantirs, [~mcheah] we don't need the meeting then if we are going to overlap with each other, fine. I have question why sever Palantir's priorities and not mine? This is not healthy :). We have talked on slack several times and privately. YOu could always have pinged but you decided to collaborate on this without anyone knowing. Probably people don't understand the meaning of fairness, I am not going to explain it here. We can always create any RP we like and then we will see what work is merged, cool. For good or bad though the meeting has power because k8s committers have the final saying on merging no? So I dont agree. The whole discussion for me is pointless, the message culture and attitude is clear. The only point is for committers, the Spark project not to violate the rules fine. was (Author: skonto): Spark belongs to the community (no?) and should not serve any company's priorities, [~mcheah] we don't need the meeting then if we are going to overlap with each other, fine. We have talked on slack several times and privately. YOu could always have pinged but you decided to collaborate on this without anyone knowing. Probably people don't understand the meaning of fairness, I am not going to explain it here. We can always create any RP we like and then we will see what work is merged, cool. For good or bad though the meeting has power because k8s committers have the final saying on merging no? So I dont agree. The whole discussion for me is pointless, the message culture and attitude is clear. The only point is for committers, the Spark project not to violate the rules fine. > Support user-specified driver and executor pod templates > > > Key: SPARK-24434 > URL: https://issues.apache.org/jira/browse/SPARK-24434 > Project: Spark > Issue Type: New Feature > Components: Kubernetes >Affects Versions: 2.4.0 >Reporter: Yinan Li >Priority: Major > > With more requests for customizing the driver and executor pods coming, the > current approach of adding new Spark configuration options has some serious > drawbacks: 1) it means more Kubernetes specific configuration options to > maintain, and 2) it widens the gap between the declarative model used by > Kubernetes and the configuration model used by Spark. We should start > designing a solution that allows users to specify pod templates as central > places for all customization needs for the driver and executor pods. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Comment Edited] (SPARK-24434) Support user-specified driver and executor pod templates
[ https://issues.apache.org/jira/browse/SPARK-24434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16599284#comment-16599284 ] Stavros Kontopoulos edited comment on SPARK-24434 at 8/31/18 9:15 PM: -- [~eje] if not mistaken all of us are on the same meeting including Palantir guys no? Do you see value when we are on the same call doing double work? If so fine. Not to mention that this case has nothing to do with multiple PRs, we had a design doc we agreed upon it. Anyway we disagree, its ok. I understand why its important not to violate the FOSS stuff and communicate that all here was fine, but honestly that this is not the point I am trying to make. Anyway I will refrain from adding more comments it does not make any sense. was (Author: skonto): [~eje] if not mistaken all of us are on the same meeting including Palantir guys no? Do you see value when we are on the same call doing double work? If so fine. Not to mention that this case has nothing to do with multiple PRs, we had a design doc we agreed upon it. Anyway we disagree, its ok. I understand why its important not to violate the FOSS stuff, but honestly that this is not the point I am trying to make. I am disappointed, anyway I will refrain from adding more comments it does not make any sense. > Support user-specified driver and executor pod templates > > > Key: SPARK-24434 > URL: https://issues.apache.org/jira/browse/SPARK-24434 > Project: Spark > Issue Type: New Feature > Components: Kubernetes >Affects Versions: 2.4.0 >Reporter: Yinan Li >Priority: Major > > With more requests for customizing the driver and executor pods coming, the > current approach of adding new Spark configuration options has some serious > drawbacks: 1) it means more Kubernetes specific configuration options to > maintain, and 2) it widens the gap between the declarative model used by > Kubernetes and the configuration model used by Spark. We should start > designing a solution that allows users to specify pod templates as central > places for all customization needs for the driver and executor pods. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Comment Edited] (SPARK-24434) Support user-specified driver and executor pod templates
[ https://issues.apache.org/jira/browse/SPARK-24434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16599284#comment-16599284 ] Stavros Kontopoulos edited comment on SPARK-24434 at 8/31/18 9:14 PM: -- [~eje] if not mistaken all of us are on the same meeting including Palantir guys no? Do you see value when we are on the same call doing double work? If so fine. Not to mention that this case has nothing to do with multiple PRs, we had a design doc we agreed upon it. Anyway we disagree, its ok. I understand why its important not to violate the FOSS stuff, but honestly that this is not the point I am trying to make. I am disappointed, anyway I will refrain from adding more comments it does not make any sense. was (Author: skonto): [~eje] if not mistaken all of us are on the same meeting including Palantir guys no? Do you see value when we are on the same call doing double work? If so fine. Not to mention that this case has nothing to do with multiple PRs, we had a design doc we agreed upon it. Anyway we disagree, its ok. I understand why its important not to violate the FOSS stuff, but honestly that this is not the point. > Support user-specified driver and executor pod templates > > > Key: SPARK-24434 > URL: https://issues.apache.org/jira/browse/SPARK-24434 > Project: Spark > Issue Type: New Feature > Components: Kubernetes >Affects Versions: 2.4.0 >Reporter: Yinan Li >Priority: Major > > With more requests for customizing the driver and executor pods coming, the > current approach of adding new Spark configuration options has some serious > drawbacks: 1) it means more Kubernetes specific configuration options to > maintain, and 2) it widens the gap between the declarative model used by > Kubernetes and the configuration model used by Spark. We should start > designing a solution that allows users to specify pod templates as central > places for all customization needs for the driver and executor pods. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Comment Edited] (SPARK-24434) Support user-specified driver and executor pod templates
[ https://issues.apache.org/jira/browse/SPARK-24434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16599284#comment-16599284 ] Stavros Kontopoulos edited comment on SPARK-24434 at 8/31/18 9:13 PM: -- [~eje] if not mistaken all of us are on the same meeting including Palantir guys no? Do you see value when we are on the same call doing double work? If so fine. Not to mention that this case has nothing to do with multiple PRs, we had a design doc we agreed upon it. Anyway we disagree, its ok. I understand why its important not to violate the FOSS stuff, but honestly that this is not the point. was (Author: skonto): [~eje] if not mistaken all of us are on the same meeting including Palantir guys no? Do you see value when we are on the same call doing double work? If so fine. Not to mention that this case has nothing to do with multiple PRs, we had a design doc we agreed upon it. Anyway we disagree. > Support user-specified driver and executor pod templates > > > Key: SPARK-24434 > URL: https://issues.apache.org/jira/browse/SPARK-24434 > Project: Spark > Issue Type: New Feature > Components: Kubernetes >Affects Versions: 2.4.0 >Reporter: Yinan Li >Priority: Major > > With more requests for customizing the driver and executor pods coming, the > current approach of adding new Spark configuration options has some serious > drawbacks: 1) it means more Kubernetes specific configuration options to > maintain, and 2) it widens the gap between the declarative model used by > Kubernetes and the configuration model used by Spark. We should start > designing a solution that allows users to specify pod templates as central > places for all customization needs for the driver and executor pods. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Comment Edited] (SPARK-24434) Support user-specified driver and executor pod templates
[ https://issues.apache.org/jira/browse/SPARK-24434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16599284#comment-16599284 ] Stavros Kontopoulos edited comment on SPARK-24434 at 8/31/18 9:12 PM: -- [~eje] if not mistaken all of us are on the same meeting including Palantir guys no? Do you see value when we are on the same call doing double work? If so fine. Not to mention that this case has nothing to do with multiple PRs, we had a design doc we agreed upon it. Anyway we disagree. was (Author: skonto): [~eje] if not mistaken all of us are on the same meeting including Palantir guys no? Do you see value when we are on the same call doing double work? If so fine. > Support user-specified driver and executor pod templates > > > Key: SPARK-24434 > URL: https://issues.apache.org/jira/browse/SPARK-24434 > Project: Spark > Issue Type: New Feature > Components: Kubernetes >Affects Versions: 2.4.0 >Reporter: Yinan Li >Priority: Major > > With more requests for customizing the driver and executor pods coming, the > current approach of adding new Spark configuration options has some serious > drawbacks: 1) it means more Kubernetes specific configuration options to > maintain, and 2) it widens the gap between the declarative model used by > Kubernetes and the configuration model used by Spark. We should start > designing a solution that allows users to specify pod templates as central > places for all customization needs for the driver and executor pods. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Comment Edited] (SPARK-24434) Support user-specified driver and executor pod templates
[ https://issues.apache.org/jira/browse/SPARK-24434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16599258#comment-16599258 ] Stavros Kontopoulos edited comment on SPARK-24434 at 8/31/18 9:09 PM: -- [~eje] Personally I will just stick to the facts (the ones I am aware of): 1) Several weeks ago I asked if this feature (in one of the meetings) should go in 2.4 and you responded that this cannot be the case, as it needs testing etc. I had no objection. It seems now that Palantir is pushing this for their own reasons and will be marked as experimental. The PR created though is not that big (not complete yet if you ask me). Given that I suspect we had time back then even with the old dates of the 2.4 cut to make a similar PR. Next time will push harder. 2) Before I leave on vacations I left a comment on our slack channel, not to mention the explicit comment in this Jira above: Stavros [3:27 PM] @liyinan926 @eje I am working on the pod template PR but i will be off for a couple of weeks, work more on that after. eje [5:04 PM] @Stavros thanks! liyinan926 [7:29 PM] @Stavros Thanks for working on that! As you see you agreed that im working on it no? In any other Jira I have seen before, people just need to state that they are working on something. They dont need to create a WIP PR AFAIK (Next time I will just commit a few lines of code to declare assignment ). 3) Copying again from the meeting notes: [https://docs.google.com/document/d/1pnF38NF6N5eM8DlK088XUW85Vms4V2uTsGZvSp8MNIA/edit] 15th of August * Custom YAML * Stavros on vacation * Complete it without him? * Palantir will look at completing I missed that part sorry. I was expecting some update on the Jira ticket, because I only checked emails on my vacations for good or bad. You could have pinged me though not just do it without me. I created the design doc dont you think I want to finish the work? 4) I almost always join the meetings and im active on the project. But nobody me pinged AFAIK. Fine. 5) Palantir guys didnt update the Jira so all people (outside meetings) know the status of things, also in the minutes doc I dont see any decision about who is going to do the PR. I think the reasonable thing to do is ask what I have done, so people dont do double effort. If the whole thing looks ok in terms of collaboration on this project, then fine, my misunderstanding then, will adapt. It is not awkward at all, people on the call decided to assign it to Palantir guys without me knowing anything, that's all. Nobody is obliged to inform me about anything, im just a contributor here, but I took it for granted that this would be the case when collaborating in a healthy community, my mistake. was (Author: skonto): [~eje] Personally I will just stick to the facts (the ones I am aware of): 1) Several weeks ago I asked if this feature (in one of the meetings) should go in 2.4 and you responded that this cannot be the case, as it needs testing etc. I had no objection. It seems now that Palantir is pushing this for their own reasons and will be marked as experimental. The PR created though is not that big (not complete yet if you ask me). Given that I suspect we had time back then even with the old dates of the 2.4 cut to make a similar PR. Next time will push harder. 2) Before I leave on vacations I left a comment on our slack channel, not to mention the explicit comment in this Jira above: Stavros [3:27 PM] @liyinan926 @eje I am working on the pod template PR but i will be off for a couple of weeks, work more on that after. eje [5:04 PM] @Stavros thanks! liyinan926 [7:29 PM] @Stavros Thanks for working on that! As you see you agreed that im working on it no? In any other Jira I have seen before, people just need to state that they are working on something. They dont need to create a WIP PR AFAIK (Next time I will just commit a few lines of code to declare assignment ). 3) Copying again from the meeting notes: [https://docs.google.com/document/d/1pnF38NF6N5eM8DlK088XUW85Vms4V2uTsGZvSp8MNIA/edit] 15th of August * Custom YAML * Stavros on vacation * Complete it without him? * Palantir will look at completing I missed that part sorry. I was expecting some update on the Jira ticket, because I only checked emails on my vacations for good or bad. You could have pinged me though not just do it without me. I created the design doc dont you think I want to finish the work? 4) I almost always join the meetings and im active on the project. But nobody me pinged AFAIK. Fine. 5) Palantir guys didnt update the Jira so all people (outside meetings) know the status of things, also in the minutes doc I dont see any decision about who is going to do the PR. I think the reasonable thing to do is ask what I have done, so people dont do double effort. If the whole thing looks ok in terms of
[jira] [Comment Edited] (SPARK-24434) Support user-specified driver and executor pod templates
[ https://issues.apache.org/jira/browse/SPARK-24434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16599258#comment-16599258 ] Stavros Kontopoulos edited comment on SPARK-24434 at 8/31/18 8:51 PM: -- [~eje] Personally I will just stick to the facts (the ones I am aware of): 1) Several weeks ago I asked if this feature (in one of the meetings) should go in 2.4 and you responded that this cannot be the case, as it needs testing etc. I had no objection. It seems now that Palantir is pushing this for their own reasons and will be marked as experimental. The PR created though is not that big (not complete yet if you ask me). Given that I suspect we had time back then even with the old dates of the 2.4 cut to make a similar PR. Next time will push harder. 2) Before I leave on vacations I left a comment on our slack channel, not to mention the explicit comment in this Jira above: Stavros [3:27 PM] @liyinan926 @eje I am working on the pod template PR but i will be off for a couple of weeks, work more on that after. eje [5:04 PM] @Stavros thanks! liyinan926 [7:29 PM] @Stavros Thanks for working on that! As you see you agreed that im working on it no? In any other Jira I have seen before, people just need to state that they are working on something. They dont need to create a WIP PR AFAIK (Next time I will just commit a few lines of code to declare assignment ). 3) Copying again from the meeting notes: [https://docs.google.com/document/d/1pnF38NF6N5eM8DlK088XUW85Vms4V2uTsGZvSp8MNIA/edit] 15th of August * Custom YAML * Stavros on vacation * Complete it without him? * Palantir will look at completing I missed that part sorry. I was expecting some update on the Jira ticket, because I only checked emails on my vacations for good or bad. You could have pinged me though not just do it without me. I created the design doc dont you think I want to finish the work? 4) I almost always join the meetings and im active on the project. But nobody me pinged AFAIK. Fine. 5) Palantir guys didnt update the Jira so all people (outside meetings) know the status of things, also in the minutes doc I dont see any decision about who is going to do the PR. I think the reasonable thing to do is ask what I have done, so people dont do double effort. If the whole thing looks ok in terms of collaboration on this project, then fine, my misunderstanding then, will adapt. It is not awkward at all, people on the call decided to assign it to Palantir guys without me knowing anything, that's all. Nobody is obliged to inform me about anything, im just a contributor here, but I took it for granted that this would be the case when collaborating in a healthy community. was (Author: skonto): [~eje] Personally I will just stick to the facts (the ones I am aware of): 1) Several weeks ago I asked if this feature (in one of the meetings) should go in 2.4 and you responded that this cannot be the case, as it needs testing etc. I had no objection. It seems now that Palantir is pushing this for their own reasons and will be marked as experimental. The PR created though is not that big (not complete yet if you ask me). Given that I suspect we had time back then even with the old dates of the 2.4 cut to make a similar PR. Next time will push harder. 2) Before I leave on vacations I left a comment on our slack channel, not to mention the explicit comment in this Jira above: Stavros [3:27 PM] @liyinan926 @eje I am working on the pod template PR but i will be off for a couple of weeks, work more on that after. eje [5:04 PM] @Stavros thanks! liyinan926 [7:29 PM] @Stavros Thanks for working on that! As you see you agreed that im working on it no? In any other Jira I have seen before, people just need to state that they are working on something. They dont need to create a WIP PR AFAIK (Next time I will just commit a few lines of code to declare assignment ). 3) Copying again from the meeting notes: [https://docs.google.com/document/d/1pnF38NF6N5eM8DlK088XUW85Vms4V2uTsGZvSp8MNIA/edit] 15th of August * Custom YAML * Stavros on vacation * Complete it without him? * Palantir will look at completing I missed that part sorry. I was expecting some update on the Jira ticket, because I only checked emails on my vacations for good or bad. You could have pinged me though not just do it without me. I created the design doc dont you think I want to finish the work? 4) I almost always join the meetings and im active on the project. But nobody me pinged AFAIK. Fine. 5) Palantir guys didnt update the Jira so all people (outside meetings) know the status of things, also in the minutes doc I dont see any decision about who is going to do the PR. I think the reasonable thing to do is ask what I have done, so people dont do double effort. If the whole thing looks ok in terms of collaboration,
[jira] [Comment Edited] (SPARK-24434) Support user-specified driver and executor pod templates
[ https://issues.apache.org/jira/browse/SPARK-24434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16599258#comment-16599258 ] Stavros Kontopoulos edited comment on SPARK-24434 at 8/31/18 8:48 PM: -- [~eje] Personally I will just stick to the facts (the ones I am aware of): 1) Several weeks ago I asked if this feature (in one of the meetings) should go in 2.4 and you responded that this cannot be the case, as it needs testing etc. I had no objection. It seems now that Palantir is pushing this for their own reasons and will be marked as experimental. The PR created though is not that big (not complete yet if you ask me). Given that I suspect we had time back then even with the old dates of the 2.4 cut to make a similar PR. Next time will push harder. 2) Before I leave on vacations I left a comment on our slack channel, not to mention the explicit comment in this Jira above: Stavros [3:27 PM] @liyinan926 @eje I am working on the pod template PR but i will be off for a couple of weeks, work more on that after. eje [5:04 PM] @Stavros thanks! liyinan926 [7:29 PM] @Stavros Thanks for working on that! As you see you agreed that im working on it no? In any other Jira I have seen before, people just need to state that they are working on something. They dont need to create a WIP PR AFAIK (Next time I will just commit a few lines of code to declare assignment ). 3) Copying again from the meeting notes: [https://docs.google.com/document/d/1pnF38NF6N5eM8DlK088XUW85Vms4V2uTsGZvSp8MNIA/edit] 15th of August * Custom YAML * Stavros on vacation * Complete it without him? * Palantir will look at completing I missed that part sorry. I was expecting some update on the Jira ticket, because I only checked emails on my vacations for good or bad. You could have pinged me though not just do it without me. I created the design doc dont you think I want to finish the work? 4) I almost always join the meetings and im active on the project. But nobody me pinged AFAIK. Fine. 5) Palantir guys didnt update the Jira so all people (outside meetings) know the status of things, also in the minutes doc I dont see any decision about who is going to do the PR. I think the reasonable thing to do is ask what I have done, so people dont do double effort. If the whole thing looks ok in terms of collaboration, then fine, my misunderstanding then, will adapt. It is not awkward at all, people on the call decided to assign it to Palantir guys without me knowing anything, that's all. Nobody is obliged to inform me about anything, im just a contributor here, but I took it for granted that this would be the case when collaborating in a healthy community. was (Author: skonto): [~eje] Personally I will just stick to the facts (the ones I am aware of): 1) Several weeks ago I asked if this feature (in one of the meetings) should go in 2.4 and you responded that this cannot be the case, as it needs testing etc. I had no objection. It seems now that Palantir is pushing this for their own reasons and will be marked as experimental. The PR created though is not that big (not complete yet if you ask me). Given that I suspect we had time back then even with the old dates of the 2.4 cut to make a similar PR. Next time will push harder. 2) Before I leave on vacations I left a comment on our slack channel, not to mention the explicit comment in this Jira above: Stavros [3:27 PM] @liyinan926 @eje I am working on the pod template PR but i will be off for a couple of weeks, work more on that after. eje [5:04 PM] @Stavros thanks! liyinan926 [7:29 PM] @Stavros Thanks for working on that! As you see you agreed that im working on it no? In any other Jira I have seen before, people just need to state that they are working on something. They dont need to create a WIP PR AFAIK (Next time I will just commit a few lines of code to declare assignment ). 3) Copying again from the meeting notes: [https://docs.google.com/document/d/1pnF38NF6N5eM8DlK088XUW85Vms4V2uTsGZvSp8MNIA/edit] 15th of August * Custom YAML * Stavros on vacation * Complete it without him? * Palantir will look at completing I missed that part sorry. I was expecting some update on the Jira ticket, because I only checked emails on my vacations for good or bad. You could have pinged me though not just do it without me. I created the design doc dont you think I want to finish the work? 4) I almost always join the meetings and im active on the project. But nobody me pinged AFAIK. Fine. 5) Palantir guys didnt update the Jira so all people (outside meetings) know the status of things, also in the minutes doc I dont see any decision about who is going to do the PR. I think the reasonable thing to do is ask what I have done, so people dont do double effort. If the whole thing looks ok in terms of collaboration, then fine, my
[jira] [Comment Edited] (SPARK-24434) Support user-specified driver and executor pod templates
[ https://issues.apache.org/jira/browse/SPARK-24434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16599258#comment-16599258 ] Stavros Kontopoulos edited comment on SPARK-24434 at 8/31/18 8:47 PM: -- [~eje] Personally I will just stick to the facts (the ones I am aware of): 1) Several weeks ago I asked if this feature (in one of the meetings) should go in 2.4 and you responded that this cannot be the case, as it needs testing etc. I had no objection. It seems now that Palantir is pushing this for their own reasons and will be marked as experimental. The PR created though is not that big (not complete yet if you ask me). Given that I suspect we had time back then even with the old dates of the 2.4 cut to make a similar PR. Next time will push harder. 2) Before I leave on vacations I left a comment on our slack channel, not to mention the explicit comment in this Jira above: Stavros [3:27 PM] @liyinan926 @eje I am working on the pod template PR but i will be off for a couple of weeks, work more on that after. eje [5:04 PM] @Stavros thanks! liyinan926 [7:29 PM] @Stavros Thanks for working on that! As you see you agreed that im working on it no? In any other Jira I have seen before, people just need to state that they are working on something. They dont need to create a WIP PR AFAIK (Next time I will just commit a few lines of code to declare assignment ). 3) Copying again from the meeting notes: [https://docs.google.com/document/d/1pnF38NF6N5eM8DlK088XUW85Vms4V2uTsGZvSp8MNIA/edit] 15th of August * Custom YAML * Stavros on vacation * Complete it without him? * Palantir will look at completing I missed that part sorry. I was expecting some update on the Jira ticket, because I only checked emails on my vacations for good or bad. You could have pinged me though not just do it without me. I created the design doc dont you think I want to finish the work? 4) I almost always join the meetings and im active on the project. But nobody me pinged AFAIK. Fine. 5) Palantir guys didnt update the Jira so all people (outside meetings) know the status of things, also in the minutes doc I dont see any decision about who is going to do the PR. I think the reasonable thing to do is ask what I have done, so people dont do double effort. If the whole thing looks ok in terms of collaboration, then fine, my misunderstanding then, will adapt. It is not awkward at all, people on the call decided to assign it to Palantir guys without me knowing anything, that's all. Nobody is obliged to inform me about anything, but I took it for granted that this is the case when collaborating in a healthy community. was (Author: skonto): [~eje] Personally I will just stick to the facts (the ones I am aware of): 1) Several weeks ago I asked if this feature (in one of the meetings) should go in 2.4 and you responded that this cannot be the case, as it needs testing etc. I had no objection. It seems now that Palantir is pushing this for their own reasons and will be marked as experimental. The PR created though is not that big (not complete yet if you ask me). Given that I suspect we had time back then even with the old dates of the 2.4 cut to make a similar PR. Next time will push harder. 2) Before I leave on vacations I left a comment on our slack channel, not to mention the explicit comment in this Jira above: Stavros [3:27 PM] @liyinan926 @eje I am working on the pod template PR but i will be off for a couple of weeks, work more on that after. eje [5:04 PM] @Stavros thanks! liyinan926 [7:29 PM] @Stavros Thanks for working on that! As you see you agreed that im working on it no? In any other Jira I have seen before, people just need to state that they are working on something. They dont need to create a WIP PR AFAIK (Next time I will just commit a few lines of code to declare assignment ). 3) Copying again from the meeting notes: [https://docs.google.com/document/d/1pnF38NF6N5eM8DlK088XUW85Vms4V2uTsGZvSp8MNIA/edit] 15th of August * Custom YAML * Stavros on vacation * Complete it without him? * Palantir will look at completing I missed that part sorry. I was expecting some update on the Jira ticket, because I only checked emails on my vacations for good or bad. You could have pinged me though not just do it without me. I created the design doc dont you think I want to finish the work? 4) I almost always join the meetings and im active on the project. But nobody me pinged AFAIK. Fine. 5) Palantir guys didnt update the Jira so all people (outside meetings) know the status of things, also in the minutes doc I dont see any decision about who is going to do the PR. I think the reasonable thing to do is ask what I have done, so people dont do double effort. If the whole thing looks ok in terms of collaboration, then fine, my misunderstanding then, will adapt.
[jira] [Comment Edited] (SPARK-24434) Support user-specified driver and executor pod templates
[ https://issues.apache.org/jira/browse/SPARK-24434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16599258#comment-16599258 ] Stavros Kontopoulos edited comment on SPARK-24434 at 8/31/18 8:46 PM: -- [~eje] Personally I will just stick to the facts (the ones I am aware of): 1) Several weeks ago I asked if this feature (in one of the meetings) should go in 2.4 and you responded that this cannot be the case, as it needs testing etc. I had no objection. It seems now that Palantir is pushing this for their own reasons and will be marked as experimental. The PR created though is not that big (not complete yet if you ask me). Given that I suspect we had time back then even with the old dates of the 2.4 cut to make a similar PR. Next time will push harder. 2) Before I leave on vacations I left a comment on our slack channel, not to mention the explicit comment in this Jira above: Stavros [3:27 PM] @liyinan926 @eje I am working on the pod template PR but i will be off for a couple of weeks, work more on that after. eje [5:04 PM] @Stavros thanks! liyinan926 [7:29 PM] @Stavros Thanks for working on that! As you see you agreed that im working on it no? In any other Jira I have seen before, people just need to state that they are working on something. They dont need to create a WIP PR AFAIK (Next time I will just commit a few lines of code to declare assignment ). 3) Copying again from the meeting notes: [https://docs.google.com/document/d/1pnF38NF6N5eM8DlK088XUW85Vms4V2uTsGZvSp8MNIA/edit] 15th of August * Custom YAML * Stavros on vacation * Complete it without him? * Palantir will look at completing I missed that part sorry. I was expecting some update on the Jira ticket, because I only checked emails on my vacations for good or bad. You could have pinged me though not just do it without me. I created the design doc dont you think I want to finish the work? 4) I almost always join the meetings and im active on the project. But nobody me pinged AFAIK. Fine. 5) Palantir guys didnt update the Jira so all people (outside meetings) know the status of things, also in the minutes doc I dont see any decision about who is going to do the PR. I think the reasonable thing to do is ask what I have done, so people dont do double effort. If the whole thing looks ok in terms of collaboration, then fine, my misunderstanding then, will adapt. It is not awkward at all, people on the call decided to assign it to Palantir guys without me knowing anything, that's all. Nobody is obliged to inform me about anything, but I thought that is normal when collaborate in a healthy community. was (Author: skonto): [~eje] Personally I will just stick to the facts (the ones I am aware of): 1) Several weeks ago I asked if this feature (in one of the meetings) should go in 2.4 and you responded that this cannot be the case, as it needs testing etc. I had no objection. It seems now that Palantir is pushing this for their own reasons and will be marked as experimental. The PR created though is not that big (not complete yet if you ask me). Given that I suspect we had time back then even with the old dates of the 2.4 cut to make a similar PR. Next time will push harder. 2) Before I leave on vacations I left a comment on our slack channel, not to mention the explicit comment in this Jira above: Stavros [3:27 PM] @liyinan926 @eje I am working on the pod template PR but i will be off for a couple of weeks, work more on that after. eje [5:04 PM] @Stavros thanks! liyinan926 [7:29 PM] @Stavros Thanks for working on that! As you see you agreed that im working on it no? In any other Jira I have seen before, people just need to state that they are working on something. They dont need to create a WIP PR AFAIK (Next time I will just commit a few lines of code to declare assignment ). 3) Copying again from the meeting notes: [https://docs.google.com/document/d/1pnF38NF6N5eM8DlK088XUW85Vms4V2uTsGZvSp8MNIA/edit] 15th of August * Custom YAML * Stavros on vacation * Complete it without him? * Palantir will look at completing I missed that part sorry. I was expecting some update on the Jira ticket, because I only checked emails on my vacations for good or bad. You could have pinged me though not just do it without me. I created the design doc dont you think I want to finish the work? 4) I almost always join the meetings and im active on the project. But nobody me pinged AFAIK. Fine. 5) Palantir guys didnt update the Jira so all people (outside meetings) know the status of things, also in the minutes doc I dont see any decision about who is going to do the PR. I think the reasonable thing to do is ask what I have done, so people dont do double effort. If the whole thing looks ok in terms of collaboration, then fine, my misunderstanding then, will adapt. It is not awkward at
[jira] [Comment Edited] (SPARK-24434) Support user-specified driver and executor pod templates
[ https://issues.apache.org/jira/browse/SPARK-24434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16599258#comment-16599258 ] Stavros Kontopoulos edited comment on SPARK-24434 at 8/31/18 8:45 PM: -- [~eje] Personally I will just stick to the facts (the ones I am aware of): 1) Several weeks ago I asked if this feature (in one of the meetings) should go in 2.4 and you responded that this cannot be the case, as it needs testing etc. I had no objection. It seems now that Palantir is pushing this for their own reasons and will be marked as experimental. The PR created though is not that big (not complete yet if you ask me). Given that I suspect we had time back then even with the old dates of the 2.4 cut to make a similar PR. Next time will push harder. 2) Before I leave on vacations I left a comment on our slack channel, not to mention the explicit comment in this Jira above: Stavros [3:27 PM] @liyinan926 @eje I am working on the pod template PR but i will be off for a couple of weeks, work more on that after. eje [5:04 PM] @Stavros thanks! liyinan926 [7:29 PM] @Stavros Thanks for working on that! As you see you agreed that im working on it no? In any other Jira I have seen before, people just need to state that they are working on something. They dont need to create a WIP PR AFAIK (Next time I will just commit a few lines of code to declare assignment ). 3) Copying again from the meeting notes: [https://docs.google.com/document/d/1pnF38NF6N5eM8DlK088XUW85Vms4V2uTsGZvSp8MNIA/edit] 15th of August * Custom YAML * Stavros on vacation * Complete it without him? * Palantir will look at completing I missed that part sorry. I was expecting some update on the Jira ticket, because I only checked emails on my vacations for good or bad. You could have pinged me though not just do it without me. I created the design doc dont you think I want to finish the work? 4) I almost always join the meetings and im active on the project. But nobody me pinged AFAIK. Fine. 5) Palantir guys didnt update the Jira so all people (outside meetings) know the status of things, also in the minutes doc I dont see any decision about who is going to do the PR. I think the reasonable thing to do is ask what I have done, so people dont do double effort. If the whole thing looks ok in terms of collaboration, then fine, my misunderstanding then, will adapt. It is not awkward at all, people on the call decided to assign it to Palantir guys without me knowing anything, that's all. was (Author: skonto): [~eje] Personally I will just stick to the facts (the ones I am aware of): 1) Several weeks ago I asked if this feature (in one of the meetings) should go in 2.4 and you responded that this cannot be the case, as it needs testing etc. I had no objection. It seems now that Palantir is pushing this for their own reasons and will be marked as experimental. The PR created though is not that big (not complete yet if you ask me). Given that I suspect we had time back then even with the old dates of the 2.4 cut to make a similar PR. Next time will push harder. 2) Before I leave on vacations I left a comment on our slack channel, not to mention the explicit comment in this Jira above: Stavros [3:27 PM] @liyinan926 @eje I am working on the pod template PR but i will be off for a couple of weeks, work more on that after. eje [5:04 PM] @Stavros thanks! liyinan926 [7:29 PM] @Stavros Thanks for working on that! As you see you agreed that im working on it no? In any other Jira I have seen before, people just need to state that they are working on something. They dont need to create a WIP PR AFAIK (Next time I will just commit a few lines of code to declare assignment ). 3) Copying again from the meeting notes: [https://docs.google.com/document/d/1pnF38NF6N5eM8DlK088XUW85Vms4V2uTsGZvSp8MNIA/edit] 15th of August * Custom YAML * Stavros on vacation * Complete it without him? * Palantir will look at completing I missed that part sorry. I was expecting some update on the Jira ticket, because I only checked emails on my vacations for good or bad. You could have pinged me though not just do it without me. I created the design doc dont you think I want to finish the work? 4) I almost always join the meetings and im active on the project. But nobody me pinged AFAIK. Fine. 5) Palantir guys didnt update the Jira so all people (outside meetings) know the status of things, also in the minutes doc I dont see any decision about who is going to do the PR. I think the reasonable thing to do is ask what I have done, so people dont do double effort. If the whole thing looks ok in terms of collaboration, then fine, my misunderstanding then, will adapt. It is not awkward people on the call decided to assign it to Palantir without knowing anything that's all. > Support user-specified
[jira] [Comment Edited] (SPARK-24434) Support user-specified driver and executor pod templates
[ https://issues.apache.org/jira/browse/SPARK-24434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16599258#comment-16599258 ] Stavros Kontopoulos edited comment on SPARK-24434 at 8/31/18 8:44 PM: -- [~eje] Personally I will just stick to the facts (the ones I am aware of): 1) Several weeks ago I asked if this feature (in one of the meetings) should go in 2.4 and you responded that this cannot be the case, as it needs testing etc. I had no objection. It seems now that Palantir is pushing this for their own reasons and will be marked as experimental. The PR created though is not that big (not complete yet if you ask me). Given that I suspect we had time back then even with the old dates of the 2.4 cut to make a similar PR. Next time will push harder. 2) Before I leave on vacations I left a comment on our slack channel, not to mention the explicit comment in this Jira above: Stavros [3:27 PM] @liyinan926 @eje I am working on the pod template PR but i will be off for a couple of weeks, work more on that after. eje [5:04 PM] @Stavros thanks! liyinan926 [7:29 PM] @Stavros Thanks for working on that! As you see you agreed that im working on it no? In any other Jira I have seen before, people just need to state that they are working on something. They dont need to create a WIP PR AFAIK (Next time I will just commit a few lines of code to declare assignment ). 3) Copying again from the meeting notes: [https://docs.google.com/document/d/1pnF38NF6N5eM8DlK088XUW85Vms4V2uTsGZvSp8MNIA/edit] 15th of August * Custom YAML * Stavros on vacation * Complete it without him? * Palantir will look at completing I missed that part sorry. I was expecting some update on the Jira ticket, because I only checked emails on my vacations for good or bad. You could have pinged me though not just do it without me. I created the design doc dont you think I want to finish the work? 4) I almost always join the meetings and im active on the project. But nobody me pinged AFAIK. Fine. 5) Palantir guys didnt update the Jira so all people (outside meetings) know the status of things, also in the minutes doc I dont see any decision about who is going to do the PR. I think the reasonable thing to do is ask what I have done, so people dont do double effort. If the whole thing looks ok in terms of collaboration, then fine, my misunderstanding then, will adapt. It is not awkward people on the call decided to assign it to Palantir without knowing anything that's all. was (Author: skonto): [~eje] Personally I will just stick to the facts (the ones I am aware of): 1) Several weeks ago I asked if this feature (in one of the meetings) should go in 2.4 and you responded that this cannot be the case, as it needs testing etc. I had no objection. It seems now that Palantir is pushing this for their own reasons and will be marked as experimental. The PR created though is not that big (not complete yet if you ask me). Given that I suspect we had time back then even with the old dates of the 2.4 cut to make a similar PR. Next time will push harder. 2) Before I leave on vacations I left a comment on our slack channel, not to mention the explicit comment in this Jira above: Stavros [3:27 PM] @liyinan926 @eje I am working on the pod template PR but i will be off for a couple of weeks, work more on that after. eje [5:04 PM] @Stavros thanks! liyinan926 [7:29 PM] @Stavros Thanks for working on that! As you see you agreed that im working on it no? In any other Jira I have seen before, people just need to state that they are working on something. They dont need to create a WIP PR AFAIK (Next time I will just commit a few lines of code to declare assignment ). 3) Copying again from the meeting notes: [https://docs.google.com/document/d/1pnF38NF6N5eM8DlK088XUW85Vms4V2uTsGZvSp8MNIA/edit] 15th of August * Custom YAML * Stavros on vacation * Complete it without him? * Palantir will look at completing I missed that part sorry. I was expecting some update on the Jira ticket, because I only checked emails on my vacations for good or bad. You could have pinged me though not just do it without me. I created the design doc dont you think I want to finish the work? 4) I almost always join the meetings and im active on the project. But nobody me pinged AFAIK. Fine. 5) Palantir guys didnt update the Jira so all people (outside meetings) know the status of things, also in the minutes doc I dont see any decision about who is going to do the PR. I think the reasonable thing to do is ask what I have done, so people dont do double effort. If the whole thing looks ok in terms of collaboration, then fine, my misunderstanding then, will adapt. > Support user-specified driver and executor pod templates > > > Key: SPARK-24434 >
[jira] [Comment Edited] (SPARK-24434) Support user-specified driver and executor pod templates
[ https://issues.apache.org/jira/browse/SPARK-24434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16599258#comment-16599258 ] Stavros Kontopoulos edited comment on SPARK-24434 at 8/31/18 8:38 PM: -- [~eje] Personally I will just stick to the facts (the ones I am aware of): 1) Several weeks ago I asked if this feature (in one of the meetings) should go in 2.4 and you responded that this cannot be the case, as it needs testing etc. I had no objection. It seems now that Palantir is pushing this for their own reasons and will be marked as experimental. The PR created though is not that big (not complete yet if you ask me). Given that I suspect we had time back then even with the old dates of the 2.4 cut to make a similar PR. Next time will push harder. 2) Before I leave on vacations I left a comment on our slack channel, not to mention the explicit comment in this Jira above: Stavros [3:27 PM] @liyinan926 @eje I am working on the pod template PR but i will be off for a couple of weeks, work more on that after. eje [5:04 PM] @Stavros thanks! liyinan926 [7:29 PM] @Stavros Thanks for working on that! As you see you agreed that im working on it no? In any other Jira I have seen before, people just need to state that they are working on something. They dont need to create a WIP PR AFAIK (Next time I will just commit a few lines of code to declare assignment ). 3) Copying again from the meeting notes: [https://docs.google.com/document/d/1pnF38NF6N5eM8DlK088XUW85Vms4V2uTsGZvSp8MNIA/edit] 15th of August * Custom YAML * Stavros on vacation * Complete it without him? * Palantir will look at completing I missed that part sorry. I was expecting some update on the Jira ticket, because I only checked emails on my vacations for good or bad. You could have pinged me though not just do it without me. I created the design doc dont you think I want to finish the work? 4) I almost always join the meetings and im active on the project. But nobody me pinged AFAIK. Fine. 5) Palantir guys didnt update the Jira so all people (outside meetings) know the status of things, also in the minutes doc I dont see any decision about who is going to do the PR. I think the reasonable thing to do is ask what I have done, so people dont do double effort. If the whole thing looks ok in terms of collaboration, then fine, my misunderstanding then, will adapt. was (Author: skonto): [~eje] Personally I will just stick to the facts (the ones I am aware of): 1) Several weeks ago I asked if this feature (in one of the meetings) should go in 2.4 and you responded that this cannot be the case, as it needs testing etc. I had no objection. It seems now that Palantir is pushing this for their own reasons and will be marked as experimental. The PR created though is not that big (not complete yet if you ask me). Given that I suspect we had time back then even with the old dates of the 2.4 cut to make a similar PR. Next time will push harder. 2) Before I leave on vacations I left a comment on our slack channel, not to mention the explicit comment in this Jira above: Stavros [3:27 PM] @liyinan926 @eje I am working on the pod template PR but i will be off for a couple of weeks, work more on that after. eje [5:04 PM] @Stavros thanks! liyinan926 [7:29 PM] @Stavros Thanks for working on that! As you see you agreed that im working on it no? In any other Jira I have seen before, people just need to state that they are working on something. They dont need to create a WIP PR AFAIK (Next time I will just commit a few lines of code to declare assignment ). 3) Copying again from the meeting notes: [https://docs.google.com/document/d/1pnF38NF6N5eM8DlK088XUW85Vms4V2uTsGZvSp8MNIA/edit] 15th of August * Custom YAML * Stavros on vacation * Complete it without him? * Palantir will look at completing I missed that part sorry. I was expecting some update on the Jira ticket, because I only checked emails on my vacations for good or bad. You could have pinged me though not just do it without me. I created the design doc dont you think I want to finish the work? 4) I almost always join the meetings and im active on the project. But nobody me pinged AFAIK. Fine. 5) Palantir guys didnt update the Jira so all people (outside meetings) know the status of things, also in the minutes doc I dont see any decision about who is going to do the PR. I think the reasonable thing to do is ask what I have done, so people dont do double effort. If the whole things looks ok in terms of collaboration, then fine. > Support user-specified driver and executor pod templates > > > Key: SPARK-24434 > URL: https://issues.apache.org/jira/browse/SPARK-24434 > Project: Spark > Issue Type: New Feature >
[jira] [Comment Edited] (SPARK-24434) Support user-specified driver and executor pod templates
[ https://issues.apache.org/jira/browse/SPARK-24434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16599258#comment-16599258 ] Stavros Kontopoulos edited comment on SPARK-24434 at 8/31/18 8:37 PM: -- [~eje] Personally I will just stick to the facts (the ones I am aware of): 1) Several weeks ago I asked if this feature (in one of the meetings) should go in 2.4 and you responded that this cannot be the case, as it needs testing etc. I had no objection. It seems now that Palantir is pushing this for their own reasons and will be marked as experimental. The PR created though is not that big (not complete yet if you ask me). Given that I suspect we had time back then even with the old dates of the 2.4 cut to make a similar PR. Next time will push harder. 2) Before I leave on vacations I left a comment on our slack channel, not to mention the explicit comment in this Jira above: Stavros [3:27 PM] @liyinan926 @eje I am working on the pod template PR but i will be off for a couple of weeks, work more on that after. eje [5:04 PM] @Stavros thanks! liyinan926 [7:29 PM] @Stavros Thanks for working on that! As you see you agreed that im working on it no? In any other Jira I have seen before, people just need to state that they are working on something. They dont need to create a WIP PR AFAIK (Next time I will just commit a few lines of code to declare assignment ). 3) Copying again from the meeting notes: [https://docs.google.com/document/d/1pnF38NF6N5eM8DlK088XUW85Vms4V2uTsGZvSp8MNIA/edit] 15th of August * Custom YAML * Stavros on vacation * Complete it without him? * Palantir will look at completing I missed that part sorry. I was expecting some update on the Jira ticket, because I only checked emails on my vacations for good or bad. You could have pinged me though not just do it without me. I created the design doc dont you think I want to finish the work? 4) I almost always join the meetings and im active on the project. But nobody me pinged AFAIK. Fine. 5) Palantir guys didnt update the Jira so all people (outside meetings) know the status of things, also in the minutes doc I dont see any decision about who is going to do the PR. I think the reasonable thing to do is ask what I have done, so people dont do double effort. If the whole things looks ok in terms of collaboration, then fine. was (Author: skonto): [~eje] Personally I will just stick to the facts (the ones I am aware of): 1) Several weeks ago I asked if this feature (in one of the meetings) should go in 2.4 and you responded that this cannot be the case, as it needs testing etc. I had no objection. It seems now that Palantir is pushing this for their own reasons and will be marked as experimental. The PR created though is not that big (not complete yet if you ask me). Given that I suspect we had time back then even with the old dates of the 2.4 cut to make a similar PR. Next time will push harder. 2) Before I leave on vacations I left a comment on our slack channel, not to mention the explicit comment in this Jira above: Stavros [3:27 PM] @liyinan926 @eje I am working on the pod template PR but i will be off for a couple of weeks, work more on that after. eje [5:04 PM] @Stavros thanks! liyinan926 [7:29 PM] @Stavros Thanks for working on that! As you see you agreed that im working on it no? In any other Jira I have seen before, people just need to state that they are working on something. They dont need to create a WIP PR AFAIK (Next time I will just commit a few lines of code to declare assignment ). 3) Copying again from the meeting notes: [https://docs.google.com/document/d/1pnF38NF6N5eM8DlK088XUW85Vms4V2uTsGZvSp8MNIA/edit] 15th of August * Custom YAML * Stavros on vacation * Complete it without him? * Palantir will look at completing I missed that part sorry. I was expecting some update on the Jira ticket, because I only checked emails on my vacations for good or bad. 4) I almost always join the meetings and im active on the project. But nobody me pinged AFAIK. Fine. 5) Palantir guys didnt update the Jira so all people (outside meetings) know the status of things, also in the minutes doc I dont see any decision about who is going to do the PR. I think the reasonable thing to do is ask what I have done, so people dont do double effort. If the whole things looks ok in terms of collaboration, then fine. > Support user-specified driver and executor pod templates > > > Key: SPARK-24434 > URL: https://issues.apache.org/jira/browse/SPARK-24434 > Project: Spark > Issue Type: New Feature > Components: Kubernetes >Affects Versions: 2.4.0 >Reporter: Yinan Li >Priority: Major > > With more requests for customizing the driver and
[jira] [Comment Edited] (SPARK-24434) Support user-specified driver and executor pod templates
[ https://issues.apache.org/jira/browse/SPARK-24434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16599258#comment-16599258 ] Stavros Kontopoulos edited comment on SPARK-24434 at 8/31/18 8:36 PM: -- [~eje] Personally I will just stick to the facts (the ones I am aware of): 1) Several weeks ago I asked if this feature (in one of the meetings) should go in 2.4 and you responded that this cannot be the case, as it needs testing etc. I had no objection. It seems now that Palantir is pushing this for their own reasons and will be marked as experimental. The PR created though is not that big (not complete yet if you ask me). Given that I suspect we had time back then even with the old dates of the 2.4 cut to make a similar PR. Next time will push harder. 2) Before I leave on vacations I left a comment on our slack channel, not to mention the explicit comment in this Jira above: Stavros [3:27 PM] @liyinan926 @eje I am working on the pod template PR but i will be off for a couple of weeks, work more on that after. eje [5:04 PM] @Stavros thanks! liyinan926 [7:29 PM] @Stavros Thanks for working on that! As you see you agreed that im working on it no? In any other Jira I have seen before, people just need to state that they are working on something. They dont need to create a WIP PR AFAIK (Next time I will just commit a few lines of code to declare assignment ). 3) Copying again from the meeting notes: [https://docs.google.com/document/d/1pnF38NF6N5eM8DlK088XUW85Vms4V2uTsGZvSp8MNIA/edit] 15th of August * Custom YAML * Stavros on vacation * Complete it without him? * Palantir will look at completing I missed that part sorry. I was expecting some update on the Jira ticket, because I only checked emails on my vacations for good or bad. 4) I almost always join the meetings and im active on the project. But nobody me pinged AFAIK. Fine. 5) Palantir guys didnt update the Jira so all people (outside meetings) know the status of things, also in the minutes doc I dont see any decision about who is going to do the PR. I think the reasonable thing to do is ask what I have done, so people dont do double effort. If the whole things looks ok in terms of collaboration, then fine. was (Author: skonto): [~eje] Personally I will just stick to the facts (the ones I am aware of): 1) Several weeks ago I asked if this feature (in one of the meetings) should go in 2.4 and you responded that this cannot be the case, as it needs testing etc. I had no objection. It seems now that Palantir is pushing this for their own reasons and will be marked as experimental. The PR created though is not that big (not complete yet if you ask me). Given that I suspect we had time back then even with the old dates of the 2.4 cut to make a similar PR. Next time will push harder. 2) Before I leave on vacations I left a comment on our slack channel, not to mention the explicit comment in this Jira above: Stavros [3:27 PM] @liyinan926 @eje I am working on the pod template PR but i will be off for a couple of weeks, work more on that after. eje [5:04 PM] @Stavros thanks! liyinan926 [7:29 PM] @Stavros Thanks for working on that! As you see you agreed that im working on it no? In any other Jira I have seen before, people just need to state that they are working on something. They dont need to create a WIP PR AFAIK (Next time I will just commit a few lines of code to declare assignment ). 3) Copying again from the meeting notes: [https://docs.google.com/document/d/1pnF38NF6N5eM8DlK088XUW85Vms4V2uTsGZvSp8MNIA/edit] * Custom YAML * Stavros on vacation * Complete it without him? * Palantir will look at completing 4) I almost always join the meetings and im active on the project. But nobody me pinged AFAIK. Fine. 5) Palantir guys didnt update the Jira so all people (outside meetings) know the status of things, also in the minutes doc I dont see any decision about who is going to do the PR. I think the reasonable thing to do is ask what I have done, so people dont do double effort. If the whole things looks ok in terms of collaboration, then fine. > Support user-specified driver and executor pod templates > > > Key: SPARK-24434 > URL: https://issues.apache.org/jira/browse/SPARK-24434 > Project: Spark > Issue Type: New Feature > Components: Kubernetes >Affects Versions: 2.4.0 >Reporter: Yinan Li >Priority: Major > > With more requests for customizing the driver and executor pods coming, the > current approach of adding new Spark configuration options has some serious > drawbacks: 1) it means more Kubernetes specific configuration options to > maintain, and 2) it widens the gap between the declarative model used by > Kubernetes and the
[jira] [Comment Edited] (SPARK-24434) Support user-specified driver and executor pod templates
[ https://issues.apache.org/jira/browse/SPARK-24434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16599258#comment-16599258 ] Stavros Kontopoulos edited comment on SPARK-24434 at 8/31/18 8:34 PM: -- [~eje] Personally I will just stick to the facts (the ones I am aware of): 1) Several weeks ago I asked if this feature (in one of the meetings) should go in 2.4 and you responded that this cannot be the case, as it needs testing etc. I had no objection. It seems now that Palantir is pushing this for their own reasons and will be marked as experimental. The PR created though is not that big (not complete yet if you ask me). Given that I suspect we had time back then even with the old dates of the 2.4 cut to make a similar PR. Next time will push harder. 2) Before I leave on vacations I left a comment on our slack channel, not to mention the explicit comment in this Jira above: Stavros [3:27 PM] @liyinan926 @eje I am working on the pod template PR but i will be off for a couple of weeks, work more on that after. eje [5:04 PM] @Stavros thanks! liyinan926 [7:29 PM] @Stavros Thanks for working on that! As you see you agreed that im working on it no? In any other Jira I have seen before, people just need to state that they are working on something. They dont need to create a WIP PR AFAIK (Next time I will just commit a few lines of code to declare assignment ). 3) Copying again from the meeting notes: [https://docs.google.com/document/d/1pnF38NF6N5eM8DlK088XUW85Vms4V2uTsGZvSp8MNIA/edit] * Custom YAML * Stavros on vacation * Complete it without him? * Palantir will look at completing 4) I almost always join the meetings and im active on the project. But nobody me pinged AFAIK. Fine. 5) Palantir guys didnt update the Jira so all people (outside meetings) know the status of things, also in the minutes doc I dont see any decision about who is going to do the PR. I think the reasonable thing to do is ask what I have done, so people dont do double effort. If the whole things looks ok in terms of collaboration, then fine. was (Author: skonto): [~eje] Personally I will just stick to the facts (the ones I am aware of): 1) Several weeks ago I asked if this feature (in one of the meetings) should go in 2.4 and you responded that this cannot be the case, as it needs testing etc. I had no objection. It seems now that Palantir is pushing this for their own reasons and will be marked as experimental. The PR created though is not that big (not complete yet if you ask me). Given that I suspect we had time back then even with the old dates of the 2.4 cut to make a similar PR. Next time will push harder. 2) Before I leave on vacations I left a comment on our slack channel, not to mention the explicit comment in this Jira above: Stavros [3:27 PM] @liyinan926 @eje I am working on the pod template PR but i will be off for a couple of weeks, work more on that after. eje [5:04 PM] @Stavros thanks! liyinan926 [7:29 PM] @Stavros Thanks for working on that! As you see you agreed that im working on it no? In any other Jira I have seen before, people just need to state that they are working on something. They dont need to create a WIP PR AFAIK (Next time I will just commit a few lines of code to declare assignment ). 3) Copying again from the meeting notes: [https://docs.google.com/document/d/1pnF38NF6N5eM8DlK088XUW85Vms4V2uTsGZvSp8MNIA/edit] * Custom YAML * Stavros on vacation * Complete it without him? * Palantir will look at completing 4) I almost always join the meetings and im active on the project. But nobody me pinged AFAIK. Fine. 5) Palantir guys didnt update the Jira so all people (outside meetings) know the status of things, also in the minutes doc I dont see any decision about who is going to do the PR. I think the reasonable thing to do is ask what I have done, so people dont do double effort. If the whole things looks ok, then fine. > Support user-specified driver and executor pod templates > > > Key: SPARK-24434 > URL: https://issues.apache.org/jira/browse/SPARK-24434 > Project: Spark > Issue Type: New Feature > Components: Kubernetes >Affects Versions: 2.4.0 >Reporter: Yinan Li >Priority: Major > > With more requests for customizing the driver and executor pods coming, the > current approach of adding new Spark configuration options has some serious > drawbacks: 1) it means more Kubernetes specific configuration options to > maintain, and 2) it widens the gap between the declarative model used by > Kubernetes and the configuration model used by Spark. We should start > designing a solution that allows users to specify pod templates as central > places for all customization needs for the driver and
[jira] [Comment Edited] (SPARK-24434) Support user-specified driver and executor pod templates
[ https://issues.apache.org/jira/browse/SPARK-24434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16599258#comment-16599258 ] Stavros Kontopoulos edited comment on SPARK-24434 at 8/31/18 8:34 PM: -- [~eje] Personally I will just stick to the facts (the ones I am aware of): 1) Several weeks ago I asked if this feature (in one of the meetings) should go in 2.4 and you responded that this cannot be the case, as it needs testing etc. I had no objection. It seems now that Palantir is pushing this for their own reasons and will be marked as experimental. The PR created though is not that big (not complete yet if you ask me). Given that I suspect we had time back then even with the old dates of the 2.4 cut to make a similar PR. Next time will push harder. 2) Before I leave on vacations I left a comment on our slack channel, not to mention the explicit comment in this Jira above: Stavros [3:27 PM] @liyinan926 @eje I am working on the pod template PR but i will be off for a couple of weeks, work more on that after. eje [5:04 PM] @Stavros thanks! liyinan926 [7:29 PM] @Stavros Thanks for working on that! As you see you agreed that im working on it no? In any other Jira I have seen before, people just need to state that they are working on something. They dont need to create a WIP PR AFAIK (Next time I will just commit a few lines of code to declare assignment ). 3) Copying again from the meeting notes: [https://docs.google.com/document/d/1pnF38NF6N5eM8DlK088XUW85Vms4V2uTsGZvSp8MNIA/edit] * Custom YAML * Stavros on vacation * Complete it without him? * Palantir will look at completing 4) I almost always join the meetings and im active on the project. But nobody me pinged AFAIK. Fine. 5) Palantir guys didnt update the Jira so all people (outside meetings) know the status of things, also in the minutes doc I dont see any decision about who is going to do the PR. I think the reasonable thing to do is ask what I have done, so people dont do double effort. If the whole things looks ok, then fine. was (Author: skonto): [~eje] Personally I will just stick to the facts (the ones I am aware of): 1) Several weeks ago I asked if this feature (in one of the meetings) should go in 2.4 and you responded that this cannot be the case, as it needs testing etc. I had no objection. It seems now that Palantir is pushing this for their own reasons and will be marked as experimental. The PR created is not that big we had time back then even with the old dates of the 2.4 cut. 2) Before I leave on vacations I left a comment on our slack channel, not to mention the explicit comment in this Jira above: Stavros [3:27 PM] @liyinan926 @eje I am working on the pod template PR but i will be off for a couple of weeks, work more on that after. eje [5:04 PM] @Stavros thanks! liyinan926 [7:29 PM] @Stavros Thanks for working on that! As you see you agreed that im working on it no? In any other Jira I have seen before, people just need to state that they are working on something. They dont need to create a WIP PR AFAIK (Next time I will just commit a few lines of code to declare assignment ). 3) Copying again from the meeting notes: [https://docs.google.com/document/d/1pnF38NF6N5eM8DlK088XUW85Vms4V2uTsGZvSp8MNIA/edit] * Custom YAML * Stavros on vacation * Complete it without him? * Palantir will look at completing 4) I almost always join the meetings and im active on the project. But nobody me pinged AFAIK. Fine. 5) Palantir guys didnt update the Jira so all people (outside meetings) know the status of things, also in the minutes doc I dont see any decision about who is going to do the PR. I think the reasonable thing to do is ask what I have done, so people dont do double effort. If the whole things looks ok, then fine. > Support user-specified driver and executor pod templates > > > Key: SPARK-24434 > URL: https://issues.apache.org/jira/browse/SPARK-24434 > Project: Spark > Issue Type: New Feature > Components: Kubernetes >Affects Versions: 2.4.0 >Reporter: Yinan Li >Priority: Major > > With more requests for customizing the driver and executor pods coming, the > current approach of adding new Spark configuration options has some serious > drawbacks: 1) it means more Kubernetes specific configuration options to > maintain, and 2) it widens the gap between the declarative model used by > Kubernetes and the configuration model used by Spark. We should start > designing a solution that allows users to specify pod templates as central > places for all customization needs for the driver and executor pods. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (SPARK-24434) Support user-specified driver and executor pod templates
[ https://issues.apache.org/jira/browse/SPARK-24434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16599258#comment-16599258 ] Stavros Kontopoulos edited comment on SPARK-24434 at 8/31/18 8:32 PM: -- [~eje] Personally I will just stick to the facts (the ones I am aware of): 1) Several weeks ago I asked if this feature (in one of the meetings) should go in 2.4 and you responded that this cannot be the case, as it needs testing etc. I had no objection. It seems now that Palantir is pushing this for their own reasons and will be marked as experimental. The PR created is not that big we had time back then even with the old dates of the 2.4 cut. 2) Before I leave on vacations I left a comment on our slack channel, not to mention the explicit comment in this Jira above: Stavros [3:27 PM] @liyinan926 @eje I am working on the pod template PR but i will be off for a couple of weeks, work more on that after. eje [5:04 PM] @Stavros thanks! liyinan926 [7:29 PM] @Stavros Thanks for working on that! As you see you agreed that im working on it no? In any other Jira I have seen before, people just need to state that they are working on something. They dont need to create a WIP PR AFAIK (Next time I will just commit a few lines of code to declare assignment ). 3) Copying again from the meeting notes: [https://docs.google.com/document/d/1pnF38NF6N5eM8DlK088XUW85Vms4V2uTsGZvSp8MNIA/edit] * Custom YAML * Stavros on vacation * Complete it without him? * Palantir will look at completing 4) I almost always join the meetings and im active on the project. But nobody me pinged AFAIK. Fine. 5) Palantir guys didnt update the Jira so all people (outside meetings) know the status of things, also in the minutes doc I dont see any decision about who is going to do the PR. I think the reasonable thing to do is ask what I have done, so people dont do double effort. If the whole things looks ok, then fine. was (Author: skonto): [~eje] Personally I will just stick to the facts: 1) Several weeks ago I asked if this feature (in one of the meetings) should go in 2.4 and you responded that this cannot be the case, as it needs testing etc. I had no objection. It seems now that Palantir is pushing this for their own reasons and will be marked as experimental. The PR created is not that big we had time back then even with the old dates of the 2.4 cut. 2) Before I leave on vacations I left a comment on our slack channel, not to mention the explicit comment in this Jira above: Stavros [3:27 PM] @liyinan926 @eje I am working on the pod template PR but i will be off for a couple of weeks, work more on that after. eje [5:04 PM] @Stavros thanks! liyinan926 [7:29 PM] @Stavros Thanks for working on that! As you see you agreed that im working on it no? In any other Jira I have seen before, people just need to state that they are working on something. They dont need to create a WIP PR AFAIK (Next time I will just commit a few lines of code to declare assignment ). 3) Copying again from the meeting notes: [https://docs.google.com/document/d/1pnF38NF6N5eM8DlK088XUW85Vms4V2uTsGZvSp8MNIA/edit] * * Custom YAML * Stavros on vacation * Complete it without him? * Palantir will look at completing 4) I almost always join the meetings and im active on the project. But nobody me pinged AFAIK. Fine. 5) Palantir guys didnt update the Jira so all people (outside meetings) know the status of things, also in the minutes doc I dont see any decision about who is going to do the PR. I think the reasonable thing to do is ask what I have done, so people dont do double effort. If the whole things looks ok, then fine. > Support user-specified driver and executor pod templates > > > Key: SPARK-24434 > URL: https://issues.apache.org/jira/browse/SPARK-24434 > Project: Spark > Issue Type: New Feature > Components: Kubernetes >Affects Versions: 2.4.0 >Reporter: Yinan Li >Priority: Major > > With more requests for customizing the driver and executor pods coming, the > current approach of adding new Spark configuration options has some serious > drawbacks: 1) it means more Kubernetes specific configuration options to > maintain, and 2) it widens the gap between the declarative model used by > Kubernetes and the configuration model used by Spark. We should start > designing a solution that allows users to specify pod templates as central > places for all customization needs for the driver and executor pods. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Comment Edited] (SPARK-24434) Support user-specified driver and executor pod templates
[ https://issues.apache.org/jira/browse/SPARK-24434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16598406#comment-16598406 ] Stavros Kontopoulos edited comment on SPARK-24434 at 8/31/18 8:20 AM: -- [~onursatici] I was never informed about the decision and how urgent was that, otherwise I could have responded to that, had no chance. was (Author: skonto): [~onursatici] I was never informed about the decision and how urgent was that, otherwise I could respond to that, had no chance. > Support user-specified driver and executor pod templates > > > Key: SPARK-24434 > URL: https://issues.apache.org/jira/browse/SPARK-24434 > Project: Spark > Issue Type: New Feature > Components: Kubernetes >Affects Versions: 2.4.0 >Reporter: Yinan Li >Priority: Major > > With more requests for customizing the driver and executor pods coming, the > current approach of adding new Spark configuration options has some serious > drawbacks: 1) it means more Kubernetes specific configuration options to > maintain, and 2) it widens the gap between the declarative model used by > Kubernetes and the configuration model used by Spark. We should start > designing a solution that allows users to specify pod templates as central > places for all customization needs for the driver and executor pods. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Comment Edited] (SPARK-24434) Support user-specified driver and executor pod templates
[ https://issues.apache.org/jira/browse/SPARK-24434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16598406#comment-16598406 ] Stavros Kontopoulos edited comment on SPARK-24434 at 8/31/18 8:19 AM: -- [~onursatici] I was never informed about the decision and how urgent was that, otherwise I could respond to that, had no chance. was (Author: skonto): [~onursatici] I was never informed about the decision. Anyway. > Support user-specified driver and executor pod templates > > > Key: SPARK-24434 > URL: https://issues.apache.org/jira/browse/SPARK-24434 > Project: Spark > Issue Type: New Feature > Components: Kubernetes >Affects Versions: 2.4.0 >Reporter: Yinan Li >Priority: Major > > With more requests for customizing the driver and executor pods coming, the > current approach of adding new Spark configuration options has some serious > drawbacks: 1) it means more Kubernetes specific configuration options to > maintain, and 2) it widens the gap between the declarative model used by > Kubernetes and the configuration model used by Spark. We should start > designing a solution that allows users to specify pod templates as central > places for all customization needs for the driver and executor pods. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Comment Edited] (SPARK-24434) Support user-specified driver and executor pod templates
[ https://issues.apache.org/jira/browse/SPARK-24434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16598406#comment-16598406 ] Stavros Kontopoulos edited comment on SPARK-24434 at 8/31/18 8:19 AM: -- [~onursatici] I was never informed about the decision. Anyway. was (Author: skonto): [~onursatici] I was never informed about the decision. > Support user-specified driver and executor pod templates > > > Key: SPARK-24434 > URL: https://issues.apache.org/jira/browse/SPARK-24434 > Project: Spark > Issue Type: New Feature > Components: Kubernetes >Affects Versions: 2.4.0 >Reporter: Yinan Li >Priority: Major > > With more requests for customizing the driver and executor pods coming, the > current approach of adding new Spark configuration options has some serious > drawbacks: 1) it means more Kubernetes specific configuration options to > maintain, and 2) it widens the gap between the declarative model used by > Kubernetes and the configuration model used by Spark. We should start > designing a solution that allows users to specify pod templates as central > places for all customization needs for the driver and executor pods. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Comment Edited] (SPARK-24434) Support user-specified driver and executor pod templates
[ https://issues.apache.org/jira/browse/SPARK-24434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16598406#comment-16598406 ] Stavros Kontopoulos edited comment on SPARK-24434 at 8/31/18 8:18 AM: -- [~onursatici] I was never informed about the decision. was (Author: skonto): [~onursatici] I was never informed about the decision. Also I notified people on the slack channel that im working on it (8th of August): Stavros [3:27 PM] @liyinan926 @eje I am working on the pod template PR but i will be off for a couple of weeks, work more on that after. I never said im not actively working on it. I just said that it will be delayed since im off. Anyway. > Support user-specified driver and executor pod templates > > > Key: SPARK-24434 > URL: https://issues.apache.org/jira/browse/SPARK-24434 > Project: Spark > Issue Type: New Feature > Components: Kubernetes >Affects Versions: 2.4.0 >Reporter: Yinan Li >Priority: Major > > With more requests for customizing the driver and executor pods coming, the > current approach of adding new Spark configuration options has some serious > drawbacks: 1) it means more Kubernetes specific configuration options to > maintain, and 2) it widens the gap between the declarative model used by > Kubernetes and the configuration model used by Spark. We should start > designing a solution that allows users to specify pod templates as central > places for all customization needs for the driver and executor pods. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Comment Edited] (SPARK-24434) Support user-specified driver and executor pod templates
[ https://issues.apache.org/jira/browse/SPARK-24434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16598406#comment-16598406 ] Stavros Kontopoulos edited comment on SPARK-24434 at 8/31/18 8:17 AM: -- [~onursatici] I was never informed about the decision. Also I notified people on the slack channel that im working on it (8th of August): Stavros [3:27 PM] @liyinan926 @eje I am working on the pod template PR but i will be off for a couple of weeks, work more on that after. I never said im not actively working on it. I just said that it will be delayed since im off. Anyway. was (Author: skonto): [~onursatici] I was never informed about the decision. Also I notified people in the slack channel that im working on it (8th of August): Stavros [3:27 PM] @liyinan926 @eje I am working on the pod template PR but i will be off for a couple of weeks, work more on that after. I never said im not actively working on it. I just said that it will be delayed since im off. Anyway. > Support user-specified driver and executor pod templates > > > Key: SPARK-24434 > URL: https://issues.apache.org/jira/browse/SPARK-24434 > Project: Spark > Issue Type: New Feature > Components: Kubernetes >Affects Versions: 2.4.0 >Reporter: Yinan Li >Priority: Major > > With more requests for customizing the driver and executor pods coming, the > current approach of adding new Spark configuration options has some serious > drawbacks: 1) it means more Kubernetes specific configuration options to > maintain, and 2) it widens the gap between the declarative model used by > Kubernetes and the configuration model used by Spark. We should start > designing a solution that allows users to specify pod templates as central > places for all customization needs for the driver and executor pods. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Comment Edited] (SPARK-24434) Support user-specified driver and executor pod templates
[ https://issues.apache.org/jira/browse/SPARK-24434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16598393#comment-16598393 ] Onur Satici edited comment on SPARK-24434 at 8/31/18 8:00 AM: -- Hello [~felixcheung], sorry I think the miscommunication here was because of the discrepancy between this Jira and k8s-sig-big-data weekly meeting notes. On [15 Aug|https://docs.google.com/document/d/1pnF38NF6N5eM8DlK088XUW85Vms4V2uTsGZvSp8MNIA/edit#heading=h.d1p209nfiamv] it was discussed that as [~skonto] was out, and was not actively working on this PR at that moment, [~yifeih] and I can take over and start working on this. I think we should have reflected that decision in this Jira after the meeting to clear our intentions. We needed this change urgently as we had a couple of PR's adding new spark configuration options to customize Spark pods, and they were all blocked by this. was (Author: onursatici): Hello [~felixcheung], sorry I think the miscommunication here was because of the discrepancy between this Jira and k8s-sig-big-data weekly meeting notes. On [link 15 Aug|https://docs.google.com/document/d/1pnF38NF6N5eM8DlK088XUW85Vms4V2uTsGZvSp8MNIA/edit#heading=h.d1p209nfiamv] it was discussed that as [~skonto] was out, and was not actively working on this PR at that moment, [~yifeih] and I can take over and start working on this. I think we should have reflected that decision in this Jira after the meeting to clear our intentions. We needed this change urgently as we had a couple of PR's adding new spark configuration options to customize Spark pods, and they were all blocked by this. > Support user-specified driver and executor pod templates > > > Key: SPARK-24434 > URL: https://issues.apache.org/jira/browse/SPARK-24434 > Project: Spark > Issue Type: New Feature > Components: Kubernetes >Affects Versions: 2.4.0 >Reporter: Yinan Li >Priority: Major > > With more requests for customizing the driver and executor pods coming, the > current approach of adding new Spark configuration options has some serious > drawbacks: 1) it means more Kubernetes specific configuration options to > maintain, and 2) it widens the gap between the declarative model used by > Kubernetes and the configuration model used by Spark. We should start > designing a solution that allows users to specify pod templates as central > places for all customization needs for the driver and executor pods. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Comment Edited] (SPARK-24434) Support user-specified driver and executor pod templates
[ https://issues.apache.org/jira/browse/SPARK-24434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16597929#comment-16597929 ] Stavros Kontopoulos edited comment on SPARK-24434 at 8/30/18 9:11 PM: -- Btw [~liyinan926] at some if you want to keep a healthy community you need to address these kind of issues, here the wrong message is communicated. Being fair is more important than any PR if you ask me or any commit at the end of the day. Also the fact that none from the people of the PR explained their intentions is also not respectful. was (Author: skonto): Btw [~liyinan926] at some if you want to keep a healthy community you need to address these kind of issues, here the wrong message is communicated. Being fair is more important than any PR if you ask me or any commit at the end of the day. Also the fact that none from the people of the PR explained their intentions is also not respectful. > Support user-specified driver and executor pod templates > > > Key: SPARK-24434 > URL: https://issues.apache.org/jira/browse/SPARK-24434 > Project: Spark > Issue Type: New Feature > Components: Kubernetes >Affects Versions: 2.4.0 >Reporter: Yinan Li >Priority: Major > > With more requests for customizing the driver and executor pods coming, the > current approach of adding new Spark configuration options has some serious > drawbacks: 1) it means more Kubernetes specific configuration options to > maintain, and 2) it widens the gap between the declarative model used by > Kubernetes and the configuration model used by Spark. We should start > designing a solution that allows users to specify pod templates as central > places for all customization needs for the driver and executor pods. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Comment Edited] (SPARK-24434) Support user-specified driver and executor pod templates
[ https://issues.apache.org/jira/browse/SPARK-24434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16597929#comment-16597929 ] Stavros Kontopoulos edited comment on SPARK-24434 at 8/30/18 9:09 PM: -- Btw [~liyinan926] at some if you want to keep a healthy community you need to address these kind of issues, here the wrong message is communicated. Being fair is more important than any PR if you ask me or any commit at the end of the day. Also the fact that none from the people of the PR explained their intentions is also not respectful. was (Author: skonto): Btw [~liyinan926] at some if you want to keep a healthy community you need to address these kind of issues, here the wrong message is communicated. Being fair is more important than any PR if you ask me or any commit at the end of the day. Also the fact that none from Palantir explained their intentions is also not respectful. > Support user-specified driver and executor pod templates > > > Key: SPARK-24434 > URL: https://issues.apache.org/jira/browse/SPARK-24434 > Project: Spark > Issue Type: New Feature > Components: Kubernetes >Affects Versions: 2.4.0 >Reporter: Yinan Li >Priority: Major > > With more requests for customizing the driver and executor pods coming, the > current approach of adding new Spark configuration options has some serious > drawbacks: 1) it means more Kubernetes specific configuration options to > maintain, and 2) it widens the gap between the declarative model used by > Kubernetes and the configuration model used by Spark. We should start > designing a solution that allows users to specify pod templates as central > places for all customization needs for the driver and executor pods. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Comment Edited] (SPARK-24434) Support user-specified driver and executor pod templates
[ https://issues.apache.org/jira/browse/SPARK-24434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16585689#comment-16585689 ] Stavros Kontopoulos edited comment on SPARK-24434 at 8/20/18 10:17 AM: --- [~onursatici] I am working on this (check last comment above), but since I am on vacations I haven't created a PR. If you are going to do it, pls read the design doc and cover the listed cases, will review. At some point we need to coordinate, [~liyinan926] next time pls assign people, I spent quite some time on this. was (Author: skonto): [~onursatici] I am working on this, but since I am on vacations I haven't created a PR. If you are going to do it, pls read the design doc and cover the listed cases, will review. At some point we need to coordinate, [~liyinan926] next time pls assign people, I spent quite some time on this. > Support user-specified driver and executor pod templates > > > Key: SPARK-24434 > URL: https://issues.apache.org/jira/browse/SPARK-24434 > Project: Spark > Issue Type: New Feature > Components: Kubernetes >Affects Versions: 2.4.0 >Reporter: Yinan Li >Priority: Major > > With more requests for customizing the driver and executor pods coming, the > current approach of adding new Spark configuration options has some serious > drawbacks: 1) it means more Kubernetes specific configuration options to > maintain, and 2) it widens the gap between the declarative model used by > Kubernetes and the configuration model used by Spark. We should start > designing a solution that allows users to specify pod templates as central > places for all customization needs for the driver and executor pods. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Comment Edited] (SPARK-24434) Support user-specified driver and executor pod templates
[ https://issues.apache.org/jira/browse/SPARK-24434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16585689#comment-16585689 ] Stavros Kontopoulos edited comment on SPARK-24434 at 8/20/18 9:34 AM: -- [~onursatici] I am working on this, but since I am on vacations I haven't created a PR. If you are going to do it, pls read the design doc and cover the listed cases, will review. At some point we need to coordinate, [~liyinan926] next time pls assign people, I spent quite some time on this. was (Author: skonto): [~onursatici] I am working on this, but since I am on vacations I haven't created a PR. If you are going to do it, pls read the design doc and cover the listed cases. At some point we need to coordinate, [~liyinan926] next time pls assign people. > Support user-specified driver and executor pod templates > > > Key: SPARK-24434 > URL: https://issues.apache.org/jira/browse/SPARK-24434 > Project: Spark > Issue Type: New Feature > Components: Kubernetes >Affects Versions: 2.4.0 >Reporter: Yinan Li >Priority: Major > > With more requests for customizing the driver and executor pods coming, the > current approach of adding new Spark configuration options has some serious > drawbacks: 1) it means more Kubernetes specific configuration options to > maintain, and 2) it widens the gap between the declarative model used by > Kubernetes and the configuration model used by Spark. We should start > designing a solution that allows users to specify pod templates as central > places for all customization needs for the driver and executor pods. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Comment Edited] (SPARK-24434) Support user-specified driver and executor pod templates
[ https://issues.apache.org/jira/browse/SPARK-24434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16585689#comment-16585689 ] Stavros Kontopoulos edited comment on SPARK-24434 at 8/20/18 9:32 AM: -- [~onursatici] I am working on this, but since I am on vacations I haven't created a PR. If you are going to do it, pls read the design doc and cover the listed cases. At some point we need to coordinate, [~liyinan926] next time pls assign people. was (Author: skonto): [~onursatici] I am working on this, but since I am on vacations I haven't created a PR. If you are going to do it, pls read the design doc and cover the listed cases. > Support user-specified driver and executor pod templates > > > Key: SPARK-24434 > URL: https://issues.apache.org/jira/browse/SPARK-24434 > Project: Spark > Issue Type: New Feature > Components: Kubernetes >Affects Versions: 2.4.0 >Reporter: Yinan Li >Priority: Major > > With more requests for customizing the driver and executor pods coming, the > current approach of adding new Spark configuration options has some serious > drawbacks: 1) it means more Kubernetes specific configuration options to > maintain, and 2) it widens the gap between the declarative model used by > Kubernetes and the configuration model used by Spark. We should start > designing a solution that allows users to specify pod templates as central > places for all customization needs for the driver and executor pods. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Comment Edited] (SPARK-24434) Support user-specified driver and executor pod templates
[ https://issues.apache.org/jira/browse/SPARK-24434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16554099#comment-16554099 ] Stavros Kontopoulos edited comment on SPARK-24434 at 7/24/18 11:06 AM: --- [~liyinan926] [~eje] Should we move with the implementation? Any more questions? Could you do another review round pls? was (Author: skonto): [~liyinan926] Should we move with the implementation? Any more questions? Could you do another review round pls? > Support user-specified driver and executor pod templates > > > Key: SPARK-24434 > URL: https://issues.apache.org/jira/browse/SPARK-24434 > Project: Spark > Issue Type: New Feature > Components: Kubernetes >Affects Versions: 2.4.0 >Reporter: Yinan Li >Priority: Major > > With more requests for customizing the driver and executor pods coming, the > current approach of adding new Spark configuration options has some serious > drawbacks: 1) it means more Kubernetes specific configuration options to > maintain, and 2) it widens the gap between the declarative model used by > Kubernetes and the configuration model used by Spark. We should start > designing a solution that allows users to specify pod templates as central > places for all customization needs for the driver and executor pods. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Comment Edited] (SPARK-24434) Support user-specified driver and executor pod templates
[ https://issues.apache.org/jira/browse/SPARK-24434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16507975#comment-16507975 ] Stavros Kontopoulos edited comment on SPARK-24434 at 6/11/18 1:02 PM: -- [~liyinan926] [~eje] here is the design doc: [https://docs.google.com/document/d/1pcyH5f610X2jyJW9WbWHnj8jktQPLlbbmmUwdeK4fJk] Please have a look and let me know if I need to add anything else, especially for the section of alternative approaches. was (Author: skonto): [~liyinan926] [~eje] here is the design doc: [https://docs.google.com/document/d/1pcyH5f610X2jyJW9WbWHnj8jktQPLlbbmmUwdeK4fJk] Please have a look and let me know if I need to add anything else, especially for the section of alternative approaches. > Support user-specified driver and executor pod templates > > > Key: SPARK-24434 > URL: https://issues.apache.org/jira/browse/SPARK-24434 > Project: Spark > Issue Type: New Feature > Components: Kubernetes >Affects Versions: 2.4.0 >Reporter: Yinan Li >Priority: Major > > With more requests for customizing the driver and executor pods coming, the > current approach of adding new Spark configuration options has some serious > drawbacks: 1) it means more Kubernetes specific configuration options to > maintain, and 2) it widens the gap between the declarative model used by > Kubernetes and the configuration model used by Spark. We should start > designing a solution that allows users to specify pod templates as central > places for all customization needs for the driver and executor pods. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Comment Edited] (SPARK-24434) Support user-specified driver and executor pod templates
[ https://issues.apache.org/jira/browse/SPARK-24434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16507975#comment-16507975 ] Stavros Kontopoulos edited comment on SPARK-24434 at 6/11/18 12:50 PM: --- [~liyinan926] [~eje] here is the design doc: [https://docs.google.com/document/d/1pcyH5f610X2jyJW9WbWHnj8jktQPLlbbmmUwdeK4fJk] Please have a look and let me know if I need to add anything else, especially for the section of alternative approaches. was (Author: skonto): [~liyinan926] [~eje] here is the design doc: [https://docs.google.com/document/d/1pcyH5f610X2jyJW9WbWHnj8jktQPLlbbmmUwdeK4fJk] Please have a look and let me know if I need to anything else, especially for the section of alternative approaches. > Support user-specified driver and executor pod templates > > > Key: SPARK-24434 > URL: https://issues.apache.org/jira/browse/SPARK-24434 > Project: Spark > Issue Type: New Feature > Components: Kubernetes >Affects Versions: 2.4.0 >Reporter: Yinan Li >Priority: Major > > With more requests for customizing the driver and executor pods coming, the > current approach of adding new Spark configuration options has some serious > drawbacks: 1) it means more Kubernetes specific configuration options to > maintain, and 2) it widens the gap between the declarative model used by > Kubernetes and the configuration model used by Spark. We should start > designing a solution that allows users to specify pod templates as central > places for all customization needs for the driver and executor pods. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Comment Edited] (SPARK-24434) Support user-specified driver and executor pod templates
[ https://issues.apache.org/jira/browse/SPARK-24434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16507975#comment-16507975 ] Stavros Kontopoulos edited comment on SPARK-24434 at 6/11/18 12:49 PM: --- [~liyinan926] [~eje] here is the design doc: [https://docs.google.com/document/d/1pcyH5f610X2jyJW9WbWHnj8jktQPLlbbmmUwdeK4fJk] Please have a look and let me know if I need to anything else, especially for the section of alternative approaches. was (Author: skonto): [~liyinan926] here is the design doc: https://docs.google.com/document/d/1pcyH5f610X2jyJW9WbWHnj8jktQPLlbbmmUwdeK4fJk > Support user-specified driver and executor pod templates > > > Key: SPARK-24434 > URL: https://issues.apache.org/jira/browse/SPARK-24434 > Project: Spark > Issue Type: New Feature > Components: Kubernetes >Affects Versions: 2.4.0 >Reporter: Yinan Li >Priority: Major > > With more requests for customizing the driver and executor pods coming, the > current approach of adding new Spark configuration options has some serious > drawbacks: 1) it means more Kubernetes specific configuration options to > maintain, and 2) it widens the gap between the declarative model used by > Kubernetes and the configuration model used by Spark. We should start > designing a solution that allows users to specify pod templates as central > places for all customization needs for the driver and executor pods. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Comment Edited] (SPARK-24434) Support user-specified driver and executor pod templates
[ https://issues.apache.org/jira/browse/SPARK-24434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16499140#comment-16499140 ] Stavros Kontopoulos edited comment on SPARK-24434 at 6/2/18 6:22 PM: - I agree with [~felixcheung] and [~liyinan926]. From the design doc for affinity stuff it was quickly obvious that things will only get more complex if we try to map yaml to Spark conf and you also loose some power with that mapping. In the past I have used json in a production system for passing config options to Spark Jobs. It was proven to be fine for one good reason, which was json schema validation that would validate also business properties early enough, for example is a number in valid range? This was pretty cool to fail fast. The tedious thing was developing with json libs (I ended up using jakson btw). Here though I would start with the most simple solution at least from an architectural perspective. That is point to the yaml spec. Yaml is the way config is specified to K8s pods so I would start with that. Regarding precedence, semantically yaml options are just spark options and precedence is defined by the order Spark config sees in general spark options specified either in the properties file or as Java properties etc. The format should't violate the precedence. Yaml and java properties are not exactly equivalent though, as yaml is more expressive when it comes to complex structures, at least it makes your life easier. For example on mesos in order to specify multiple secrets you need to specify them with [commas|https://spark.apache.org/docs/latest/running-on-mesos.html] and order matters. Also commas cant be part of the name. Of course yaml is not Spark-like but K8s is a sophisticated deployment env anyway. So one question that comes here is whether all these infrastructure configuration properties belong to Spark anyway (another way to view the whole problem). There was a long discussion for resource managers to be out of the upstream project, but was blocked due to the required changes for a common API. Thus from that angle the problem is a bit easier to solve, properties dont need to be semantically the same with Spark config options and could be resource manager specific. For example secret management is specific to the env, Spark does not care. Even Spark Java Options passed to containers is something that depends on the env. So it would be good to have a separation here. But since this separation never happened I guess we can stick with yaml as another way for passing options for Spark on K8s. Thoughts? was (Author: skonto): I agree with [~felixcheung] and [~liyinan926]. From the design doc for affinity stuff it was quickly obvious that things will only get more complex if we try to map yaml to Spark conf and you also loose some power with that mapping. In the past I have used json in a production system for passing config options to Spark Jobs. It was proven to be fine for one good reason, which was json schema validation that would validate also business properties early enough, for example is a number in valid range? This was pretty cool to fail fast. The tedious thing was developing with json libs (I ended up using jakson btw). Here though I would start with the most simple solution at least from an architectural perspective. That is point to the yaml spec. Yaml is the way config is specified to K8s pods so I would start with that. Regarding precedence, semantically yaml options are just spark options and precedence is defined by the order Spark config sees in general spark options specified either in the properties file or as Java properties etc. The format should't violate the precedence. Yaml and java properties are not exactly equivalent though, as yaml is more expressive when it comes to complex structures, at least it makes your life easier. For example on mesos in order to specify multiple secrets you need to specify them with [commas|https://spark.apache.org/docs/latest/running-on-mesos.html] and order matters. Also commas cant be part of the name. Of course yaml is not Spark-like but K8s is a sophisticated deployment env anyway. So one question that comes here is whether all these infrastructure configuration properties belong to Spark anyway (another way to view the whole problem). There was a long discussion for resource managers to be out of the upstream project, but was blocked due to the required changes for a common API. Thus from that angle the problem is a bit easier to solve, properties dont need to be semantically the same with Spark config options and could be resource manager specific. But since this separation never happened I guess we stick with yaml as another way for passing options. > Support user-specified driver and executor pod templates >
[jira] [Comment Edited] (SPARK-24434) Support user-specified driver and executor pod templates
[ https://issues.apache.org/jira/browse/SPARK-24434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16499140#comment-16499140 ] Stavros Kontopoulos edited comment on SPARK-24434 at 6/2/18 6:20 PM: - I agree with [~felixcheung] and [~liyinan926]. From the design doc for affinity stuff it was quickly obvious that things will only get more complex if we try to map yaml to Spark conf and you also loose some power with that mapping. In the past I have used json in a production system for passing config options to Spark Jobs. It was proven to be fine for one good reason, which was json schema validation that would validate also business properties early enough, for example is a number in valid range? This was pretty cool to fail fast. The tedious thing was developing with json libs (I ended up using jakson btw). Here though I would start with the most simple solution at least from an architectural perspective. That is point to the yaml spec. Yaml is the way config is specified to K8s pods so I would start with that. Regarding precedence, semantically yaml options are just spark options and precedence is defined by the order Spark config sees in general spark options specified either in the properties file or as Java properties etc. The format should't violate the precedence. Yaml and java properties are not exactly equivalent though, as yaml is more expressive when it comes to complex structures, at least it makes your life easier. For example on mesos in order to specify multiple secrets you need to specify them with [commas|https://spark.apache.org/docs/latest/running-on-mesos.html] and order matters. Also commas cant be part of the name. Of course yaml is not Spark-like but K8s is a sophisticated deployment env anyway. So one question that comes here is whether all these infrastructure configuration properties belong to Spark anyway (another way to view the whole problem). There was a long discussion for resource managers to be out of the upstream project, but was blocked due to the required changes for a common API. Thus from that angle the problem is a bit easier to solve, properties dont need to be semantically the same with Spark config options and could be resource manager specific. But since this separation never happened I guess we stick with yaml as another way for passing options. was (Author: skonto): I agree with [~felixcheung] and [~liyinan926]. From the design doc for affinity stuff it was quickly obvious that things will only get more complex if we try to map yaml to Spark conf and you also loose some power with that mapping. In the past I have used json in a production system for passing config options to Spark Jobs. It was proven to be fine for one good reason, which was json schema validation that would validate also business properties early enough, for example is a number in valid range? This was pretty cool to fail fast. The tedious thing was developing with json libs (I ended up using jakson btw). Here though I would start with the most simple solution at least from an architectural perspective. That is point to the yaml spec. Yaml is the way config is specified to K8s pods so I would start with that. Regarding precedence, semantically yaml options are just spark options and precedence is defined by the order Spark config sees in general spark options specified either in the properties file or as Java properties etc. The format should't violate the precedence. Yaml and java properties are not exactly equivalent though, as yaml is more expressive when it comes to complex structures, at least it makes your life easier. For example on mesos in order to specify multiple secrets you need to specify them with [commas|https://spark.apache.org/docs/latest/running-on-mesos.html] and order matters. Also commas cant be part of the name. Of course yaml is not Spark-like but K8s is a sophisticated deployment env anyway. So one question that comes here is all this infrastructure configuration properties may not belong to Spark anyway (another way to view the whole problem). There was a long discussion for resource managers to be out of the upstream project, but was blocked due to the required changes for a common API. Thus from that angle the problem is a bit easier to solve, properties dont need to be semantically the same with Spark config options and could be resource manager specific. But since this separation never happened I guess we stick with yaml as another way for passing options. > Support user-specified driver and executor pod templates > > > Key: SPARK-24434 > URL: https://issues.apache.org/jira/browse/SPARK-24434 > Project: Spark > Issue Type: New Feature > Components: Kubernetes >Affects Versions:
[jira] [Comment Edited] (SPARK-24434) Support user-specified driver and executor pod templates
[ https://issues.apache.org/jira/browse/SPARK-24434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16499140#comment-16499140 ] Stavros Kontopoulos edited comment on SPARK-24434 at 6/2/18 6:19 PM: - I agree with [~felixcheung] and [~liyinan926]. From the design doc for affinity stuff it was quickly obvious that things will only get more complex if we try to map yaml to Spark conf and you also loose some power with that mapping. In the past I have used json in a production system for passing config options to Spark Jobs. It was proven to be fine for one good reason, which was json schema validation that would validate also business properties early enough, for example is a number in valid range? This was pretty cool to fail fast. The tedious thing was developing with json libs (I ended up using jakson btw). Here though I would start with the most simple solution at least from an architectural perspective. That is point to the yaml spec. Yaml is the way config is specified to K8s pods so I would start with that. Regarding precedence, semantically yaml options are just spark options and precedence is defined by the order Spark config sees in general spark options specified either in the properties file or as Java properties etc. The format should't violate the precedence. Yaml and java properties are not exactly equivalent though, as yaml is more expressive when it comes to complex structures, at least it makes your life easier. For example on mesos in order to specify multiple secrets you need to specify them with [commas|https://spark.apache.org/docs/latest/running-on-mesos.html] and order matters. Also commas cant be part of the name. Of course yaml is not Spark-like but K8s is a sophisticated deployment env anyway. So one question that comes here is all this infrastructure configuration properties may not belong to Spark anyway (another way to view the whole problem). There was a long discussion for resource managers to be out of the upstream project, but was blocked due to the required changes for a common API. Thus from that angle the problem is a bit easier to solve, properties dont need to be semantically the same with Spark config options and could be resource manager specific. But since this separation never happened I guess we stick with yaml as another way for passing options. was (Author: skonto): I agree with [~felixcheung] and [~liyinan926]. From the design doc for affinity stuff it was quickly obvious that things will only get more complex if we try to map yaml to Spark conf and you also loose some power with that mapping. In the past I have used json in a production system for passing config options to Spark Jobs. It was proven to be fine for one good reason, which was json schema validation that would validate also business properties early enough, for example is a number in valid range? This was pretty cool to fail fast. The tedious thing was developing with json libs (I ended up using jakson btw). Here though I would start with the most simple solution at least from an architectural perspective. That is point to the yaml spec. Yaml is the way config is specified to K8s pods so I would start with that. Regarding precedence, semantically yaml options are just spark options and precedence is defined by the order Spark config sees in general spark options specified either in the properties file or as Java properties etc. The format should't violate the precedence. Yaml and java properties are not exactly equivalent though, as yaml is more expressive when it comes to complex structures, at least it makes your life easier. For example on mesos in order to specify multiple secrets you need to specify them with [commas|https://spark.apache.org/docs/latest/running-on-mesos.html] and order matters. Also commas cant be part of the name. Of course yaml is not Spark-like but K8s is a sophisticated deployment env anyway. So one question that comes here is all this infrastructure configuration properties may not belong to Spark anyway (another way to view the whole problem). There was a long discussion for resource managers to be out of the upstream project, but was blocked due to the required changes for a common API. Thus from that angle the problem is a bit easier to solve, properties dont need to be semantically the same with Spark config options and could be manager specific. > Support user-specified driver and executor pod templates > > > Key: SPARK-24434 > URL: https://issues.apache.org/jira/browse/SPARK-24434 > Project: Spark > Issue Type: New Feature > Components: Kubernetes >Affects Versions: 2.4.0 >Reporter: Yinan Li >Priority: Major > > With more requests for customizing the
[jira] [Comment Edited] (SPARK-24434) Support user-specified driver and executor pod templates
[ https://issues.apache.org/jira/browse/SPARK-24434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16499140#comment-16499140 ] Stavros Kontopoulos edited comment on SPARK-24434 at 6/2/18 6:17 PM: - I agree with [~felixcheung] and [~liyinan926]. From the design doc for affinity stuff it was quickly obvious that things will only get more complex if we try to map yaml to Spark conf and you also loose some power with that mapping. In the past I have used json in a production system for passing config options to Spark Jobs. It was proven to be fine for one good reason, which was json schema validation that would validate also business properties early enough, for example is a number in valid range? This was pretty cool to fail fast. The tedious thing was developing with json libs (I ended up using jakson btw). Here though I would start with the most simple solution at least from an architectural perspective. That is point to the yaml spec. Yaml is the way config is specified to K8s pods so I would start with that. Regarding precedence, semantically yaml options are just spark options and precedence is defined by the order Spark config sees in general spark options specified either in the properties file or as Java properties etc. The format should't violate the precedence. Yaml and java properties are not exactly equivalent though, as yaml is more expressive when it comes to complex structures, at least it makes your life easier. For example on mesos in order to specify multiple secrets you need to specify them with [commas|https://spark.apache.org/docs/latest/running-on-mesos.html] and order matters. Also commas cant be part of the name. Of course yaml is not Spark-like but K8s is a sophisticated deployment env anyway. So one question that comes here is all this infrastructure configuration properties may not belong to Spark anyway (another way to view the whole problem). There was a long discussion for resource managers to be out of the upstream project, but was blocked due to the required changes for a common API. Thus from that angle the problem is a bit easier to solve, properties dont need to be semantically the same with Spark config options and could be manager specific. was (Author: skonto): I agree with [~felixcheung] and [~liyinan926]. From the design doc for affinity stuff it was quickly obvious that things will only get more complex if we try to map yaml to Spark conf and you also loose some power with that mapping. In the past I have used json in a production system for passing config options to Spark Jobs. It was proven to be fine for one good reason, which was json schema validation that would validate also business properties early enough, for example is a number in valid range? This was pretty cool to fail fast. The tedious thing was developing with json libs (I ended up using jakson btw). Here though I would start with the most simple solution at least from an architectural perspective. That is point to the yaml spec. Yaml is the way config is specified to K8s pods so I would start with that. Regarding precedence, semantically yaml options are just spark options and precedence is defined by the order Spark config sees in general spark options specified either in the properties file or as Java properties etc. The format should't violate the precedence. Yaml and java properties are not exactly equivalent though, as yaml is more expressive when it comes to complex structures, at least it makes your life easier. For example on mesos in order to specify multiple secrets you need to specify them with [commas|https://spark.apache.org/docs/latest/running-on-mesos.html] and order matters. Also commas cant be part of the name. Of course yaml is not Spark-like but K8s is a sophisticated deployment env anyway. One question would be why other envs dont have this requirements. > Support user-specified driver and executor pod templates > > > Key: SPARK-24434 > URL: https://issues.apache.org/jira/browse/SPARK-24434 > Project: Spark > Issue Type: New Feature > Components: Kubernetes >Affects Versions: 2.4.0 >Reporter: Yinan Li >Priority: Major > > With more requests for customizing the driver and executor pods coming, the > current approach of adding new Spark configuration options has some serious > drawbacks: 1) it means more Kubernetes specific configuration options to > maintain, and 2) it widens the gap between the declarative model used by > Kubernetes and the configuration model used by Spark. We should start > designing a solution that allows users to specify pod templates as central > places for all customization needs for the driver and executor pods. -- This message was sent by
[jira] [Comment Edited] (SPARK-24434) Support user-specified driver and executor pod templates
[ https://issues.apache.org/jira/browse/SPARK-24434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16499140#comment-16499140 ] Stavros Kontopoulos edited comment on SPARK-24434 at 6/2/18 6:12 PM: - I agree with [~felixcheung] and [~liyinan926]. From the design doc for affinity stuff it was quickly obvious that things will only get more complex if we try to map yaml to Spark conf and you also loose some power with that mapping. In the past I have used json in a production system for passing config options to Spark Jobs. It was proven to be fine for one good reason, which was json schema validation that would validate also business properties early enough, for example is a number in valid range? This was pretty cool to fail fast. The tedious thing was developing with json libs (I ended up using jakson btw). Here though I would start with the most simple solution at least from an architectural perspective. That is point to the yaml spec. Yaml is the way config is specified to K8s pods so I would start with that. Regarding precedence, semantically yaml options are just spark options and precedence is defined by the order Spark config sees in general spark options specified either in the properties file or as Java properties etc. The format should't violate the precedence. Yaml and java properties are not exactly equivalent though, as yaml is more expressive when it comes to complex structures, at least it makes your life easier. For example on mesos in order to specify multiple secrets you need to specify them with [commas|https://spark.apache.org/docs/latest/running-on-mesos.html] and order matters. Also commas cant be part of the name. Of course yaml is not Spark-like but K8s is a sophisticated deployment env anyway. One question would be why other envs dont have this requirements. was (Author: skonto): I agree with [~felixcheung] and [~liyinan926]. From the design doc for affinity stuff it was quickly obvious that things will only get more complex if we try to map yaml to Spark conf and you also loose some power with that mapping. In the past I have used json in a production system for passing config options to Spark Jobs. It was proven to be fine for one good reason, which was json schema validation that would validate also business properties early enough, for example is a number in valid range? This was pretty cool to fail fast. The tedious thing was developing with json libs (I ended up using jakson btw). Here though I would start with the most simple solution at least from an architectural perspective. That is point to the yaml spec. Yaml is the way config is specified to K8s pods so I would start with that. Regarding precedence, semantically yaml options are just spark options and precedence is defined by the order Spark config sees in general spark options specified either in the properties file or as Java properties etc. The format should't violate the precedence. Yaml and java properties are not exactly equivalent though, as yaml is more expressive when it comes to complex structures, at least it makes your life easier. For example on mesos in order to specify multiple secrets you need to specify them with [commas|https://spark.apache.org/docs/latest/running-on-mesos.html] and order matters. Also commas cant be part of the name. > Support user-specified driver and executor pod templates > > > Key: SPARK-24434 > URL: https://issues.apache.org/jira/browse/SPARK-24434 > Project: Spark > Issue Type: New Feature > Components: Kubernetes >Affects Versions: 2.4.0 >Reporter: Yinan Li >Priority: Major > > With more requests for customizing the driver and executor pods coming, the > current approach of adding new Spark configuration options has some serious > drawbacks: 1) it means more Kubernetes specific configuration options to > maintain, and 2) it widens the gap between the declarative model used by > Kubernetes and the configuration model used by Spark. We should start > designing a solution that allows users to specify pod templates as central > places for all customization needs for the driver and executor pods. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Comment Edited] (SPARK-24434) Support user-specified driver and executor pod templates
[ https://issues.apache.org/jira/browse/SPARK-24434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16498337#comment-16498337 ] Erik Erlandson edited comment on SPARK-24434 at 6/1/18 5:53 PM: My current take on UX around this feature is that there's not much precedent from the Spark world. Assuming I'm right about that it's more likely to be driven by what expectations Kubernetes users have. In my experience that is along the lines of "pointing at a yaml file," but maybe there's more variety of user workflows than I think. JSON definitely seems more amenable for including with command-line arguments. I have been assuming that if users were specifying pod configurations that they'd be somewhat larger pod sub-structures and not easy to supply embedded on a command line. Are "small" pod modifications also likely? was (Author: eje): My current take on UX around this feature is that there's not much precedent from the Spark world. Assuming I'm right about that it's more likely to be driven by what expectations Kubernetes users have. In my experience that is along the lines of "pointing at a yaml file," but maybe there's more variety of user workflows than I think. > Support user-specified driver and executor pod templates > > > Key: SPARK-24434 > URL: https://issues.apache.org/jira/browse/SPARK-24434 > Project: Spark > Issue Type: New Feature > Components: Kubernetes >Affects Versions: 2.4.0 >Reporter: Yinan Li >Priority: Major > > With more requests for customizing the driver and executor pods coming, the > current approach of adding new Spark configuration options has some serious > drawbacks: 1) it means more Kubernetes specific configuration options to > maintain, and 2) it widens the gap between the declarative model used by > Kubernetes and the configuration model used by Spark. We should start > designing a solution that allows users to specify pod templates as central > places for all customization needs for the driver and executor pods. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Comment Edited] (SPARK-24434) Support user-specified driver and executor pod templates
[
https://issues.apache.org/jira/browse/SPARK-24434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16498307#comment-16498307
]
Yinan Li edited comment on SPARK-24434 at 6/1/18 5:39 PM:
--
The pod template is basically a pod specification and can contain every
possible pieces of information about a pod. It should look similar to what the
core workload types (deployments and statefulsets for example) use, which
contains a
{{[PodSpec|https://github.com/kubernetes/kubernetes/blob/master/staging/src/k8s.io/api/core/v1/types.go#L2636]}}.
The problem is unique for the Kubernetes mode as there are many things to
customize for a pod. Currently we basically just introduce a new Spark config
property for each new customization aspect of a pod. Given the number of things
to customize, this will soon become hard to maintain if we keep introducing new
config properties.
was (Author: liyinan926):
The pod template is basically a pod specification and can contain every
possible pieces of information about a pod. It should look similar to what the
core workload types (deployments and statefulsets for example) use, which
contains a
{{[PodSpec|https://github.com/kubernetes/kubernetes/blob/master/staging/src/k8s.io/api/core/v1/types.go#L2636]}}.
The problem is unique for the Kubernetes mode as there are many things to
customize for a pod. Currently we basically just introduce a new Spark config
property for each new customization aspect of a pod. Given the number of things
to customize, this will soon become hard to maintain if we keep introducing new
config properties.
> Support user-specified driver and executor pod templates
>
>
> Key: SPARK-24434
> URL: https://issues.apache.org/jira/browse/SPARK-24434
> Project: Spark
> Issue Type: New Feature
> Components: Kubernetes
>Affects Versions: 2.4.0
>Reporter: Yinan Li
>Priority: Major
>
> With more requests for customizing the driver and executor pods coming, the
> current approach of adding new Spark configuration options has some serious
> drawbacks: 1) it means more Kubernetes specific configuration options to
> maintain, and 2) it widens the gap between the declarative model used by
> Kubernetes and the configuration model used by Spark. We should start
> designing a solution that allows users to specify pod templates as central
> places for all customization needs for the driver and executor pods.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Comment Edited] (SPARK-24434) Support user-specified driver and executor pod templates
[
https://issues.apache.org/jira/browse/SPARK-24434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16498307#comment-16498307
]
Yinan Li edited comment on SPARK-24434 at 6/1/18 5:38 PM:
--
The pod template is basically a pod specification and can contain every
possible pieces of information about a pod. It should look similar to what the
core workload types (deployments and statefulsets for example) use, which
contains a
{{[PodSpec|https://github.com/kubernetes/kubernetes/blob/master/staging/src/k8s.io/api/core/v1/types.go#L2636]}}.
The problem is unique for the Kubernetes mode as there are many things to
customize for a pod. Currently we basically just introduce a new Spark config
property for each new customization aspect of a pod. Given the number of things
to customize, this will soon become hard to maintain if we keep introducing new
config properties.
was (Author: liyinan926):
The pod template is basically a pod specification and can contain every
possible pieces of information about a pod. It should look similar to what the
core workload types (deployments and statefulsets for example) use, which
contains a
{{[PodSpec|https://github.com/kubernetes/kubernetes/blob/master/staging/src/k8s.io/api/core/v1/types.go#L2636]}}.
> Support user-specified driver and executor pod templates
>
>
> Key: SPARK-24434
> URL: https://issues.apache.org/jira/browse/SPARK-24434
> Project: Spark
> Issue Type: New Feature
> Components: Kubernetes
>Affects Versions: 2.4.0
>Reporter: Yinan Li
>Priority: Major
>
> With more requests for customizing the driver and executor pods coming, the
> current approach of adding new Spark configuration options has some serious
> drawbacks: 1) it means more Kubernetes specific configuration options to
> maintain, and 2) it widens the gap between the declarative model used by
> Kubernetes and the configuration model used by Spark. We should start
> designing a solution that allows users to specify pod templates as central
> places for all customization needs for the driver and executor pods.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Comment Edited] (SPARK-24434) Support user-specified driver and executor pod templates
[ https://issues.apache.org/jira/browse/SPARK-24434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16497009#comment-16497009 ] Anirudh Ramanathan edited comment on SPARK-24434 at 5/31/18 6:54 PM: - I was basing my suggestion of JSON on allowing specifying JSON strings inline as configuration, but I guess one could also specify a YAML file with the template and have spark configuration point to that file. [~skonto], you make a good point, it is another configuration mechanism that people may have to learn. This decision should be based more on UX and consistency with what Spark users expect in general. [~eje], to your point, I think we could support both if needed, but it might be prudent to find the one that's more intuitive to users in order to do first. Sidenote: There's also [jsonpath|https://kubernetes.io/docs/reference/kubectl/] that kubectl supports but that could be overkill here. was (Author: foxish): Good point. I was basing my suggestion of JSON on allowing specifying JSON strings inline as configuration, but I guess one could also specify a YAML file with the template and have spark configuration point to that file. [~skonto], you make a good point, it is another configuration mechanism that people may have to learn. This decision should be based more on UX and consistency with what Spark users expect in general. [~eje], to your point, I think we could support both if needed, but it might be prudent to find the one that's more intuitive to users in order to do first. Sidenote: There's also [jsonpath|https://kubernetes.io/docs/reference/kubectl/] that kubectl supports but that could be overkill here. > Support user-specified driver and executor pod templates > > > Key: SPARK-24434 > URL: https://issues.apache.org/jira/browse/SPARK-24434 > Project: Spark > Issue Type: New Feature > Components: Kubernetes >Affects Versions: 2.4.0 >Reporter: Yinan Li >Priority: Major > > With more requests for customizing the driver and executor pods coming, the > current approach of adding new Spark configuration options has some serious > drawbacks: 1) it means more Kubernetes specific configuration options to > maintain, and 2) it widens the gap between the declarative model used by > Kubernetes and the configuration model used by Spark. We should start > designing a solution that allows users to specify pod templates as central > places for all customization needs for the driver and executor pods. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Comment Edited] (SPARK-24434) Support user-specified driver and executor pod templates
[ https://issues.apache.org/jira/browse/SPARK-24434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16497000#comment-16497000 ] Stavros Kontopoulos edited comment on SPARK-24434 at 5/31/18 6:49 PM: -- [~foxish] JSON will be exposed to the user? IMHO If that is the case it seems a bit weird, people are used to yaml with k8s and java properties with Spark. Is there an option to keep it more simple? was (Author: skonto): [~foxish] JSON will be exposed to the user? IMHO If that is the case it seems a bit weird, people are used to yaml with k8s and java properties with Spark. > Support user-specified driver and executor pod templates > > > Key: SPARK-24434 > URL: https://issues.apache.org/jira/browse/SPARK-24434 > Project: Spark > Issue Type: New Feature > Components: Kubernetes >Affects Versions: 2.4.0 >Reporter: Yinan Li >Priority: Major > > With more requests for customizing the driver and executor pods coming, the > current approach of adding new Spark configuration options has some serious > drawbacks: 1) it means more Kubernetes specific configuration options to > maintain, and 2) it widens the gap between the declarative model used by > Kubernetes and the configuration model used by Spark. We should start > designing a solution that allows users to specify pod templates as central > places for all customization needs for the driver and executor pods. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Comment Edited] (SPARK-24434) Support user-specified driver and executor pod templates
[ https://issues.apache.org/jira/browse/SPARK-24434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16495779#comment-16495779 ] Stavros Kontopoulos edited comment on SPARK-24434 at 5/30/18 10:12 PM: --- [~eje] I agree will give it a shot and try exhaust options, will see if I can come up with pros and cons etc. This also blocks the affinity/anti-affinity work. was (Author: skonto): [~eje] I agree will give it a shot and try exhaust options, will see if I can come up with pros and cons etc. > Support user-specified driver and executor pod templates > > > Key: SPARK-24434 > URL: https://issues.apache.org/jira/browse/SPARK-24434 > Project: Spark > Issue Type: New Feature > Components: Kubernetes >Affects Versions: 2.4.0 >Reporter: Yinan Li >Priority: Major > > With more requests for customizing the driver and executor pods coming, the > current approach of adding new Spark configuration options has some serious > drawbacks: 1) it means more Kubernetes specific configuration options to > maintain, and 2) it widens the gap between the declarative model used by > Kubernetes and the configuration model used by Spark. We should start > designing a solution that allows users to specify pod templates as central > places for all customization needs for the driver and executor pods. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Comment Edited] (SPARK-24434) Support user-specified driver and executor pod templates
[ https://issues.apache.org/jira/browse/SPARK-24434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16495601#comment-16495601 ] Stavros Kontopoulos edited comment on SPARK-24434 at 5/30/18 7:55 PM: -- [~liyinan926] I will work on a design document. From a first glance K8s pod spec could be enough or a subset of it at least. was (Author: skonto): [~liyinan926] I will work on a design document. From a first glance K8s pod spec could be enough or a subset of it. > Support user-specified driver and executor pod templates > > > Key: SPARK-24434 > URL: https://issues.apache.org/jira/browse/SPARK-24434 > Project: Spark > Issue Type: New Feature > Components: Kubernetes >Affects Versions: 2.4.0 >Reporter: Yinan Li >Priority: Major > > With more requests for customizing the driver and executor pods coming, the > current approach of adding new Spark configuration options has some serious > drawbacks: 1) it means more Kubernetes specific configuration options to > maintain, and 2) it widens the gap between the declarative model used by > Kubernetes and the configuration model used by Spark. We should start > designing a solution that allows users to specify pod templates as central > places for all customization needs for the driver and executor pods. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
