Github user ssuchter commented on a diff in the pull request:
https://github.com/apache/spark/pull/21652#discussion_r199610045
--- Diff:
resource-managers/kubernetes/integration-tests/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/KubernetesSuite.scala
---
@@ -74,10
Github user ssuchter commented on a diff in the pull request:
https://github.com/apache/spark/pull/21652#discussion_r199609986
--- Diff:
resource-managers/kubernetes/integration-tests/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/KubernetesSuite.scala
---
@@ -74,10
Github user ssuchter commented on a diff in the pull request:
https://github.com/apache/spark/pull/21652#discussion_r199609501
--- Diff:
resource-managers/kubernetes/integration-tests/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/KubernetesSuite.scala
---
@@ -265,6
Github user ssuchter commented on the issue:
https://github.com/apache/spark/pull/21672
BTW, for committers - I think this patch is good to merge.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
Github user ssuchter commented on the issue:
https://github.com/apache/spark/pull/21672
I see why the old behavior was there. I made a minimal change to some
existing code to fix a bug:
https://github.com/ssuchter/spark/commit/1d8a265d13b65dcec8db11a5be09d4a029037d2c
Github user ssuchter commented on the issue:
https://github.com/apache/spark/pull/21672
So this changes behavior, I think. In the old behavior, if the args were
['a', 'b'] then you'd get a single arg of ['a b'] passed through, and with this
you'd get ['a', 'b'].
This new
Github user ssuchter commented on the issue:
https://github.com/apache/spark/pull/21517
retest this please
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews
Github user ssuchter commented on the issue:
https://github.com/apache/spark/pull/21517
retest this please
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews
Github user ssuchter commented on the issue:
https://github.com/apache/spark/pull/21517
retest this please
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews
Github user ssuchter commented on the issue:
https://github.com/apache/spark/pull/21517
retest this please
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews
Github user ssuchter commented on the issue:
https://github.com/apache/spark/pull/21517
retest this please
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews
Github user ssuchter commented on the issue:
https://github.com/apache/spark/pull/21517
retest this please
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews
Github user ssuchter commented on the issue:
https://github.com/apache/spark/pull/21517
retest this please
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews
Github user ssuchter commented on the issue:
https://github.com/apache/spark/pull/21517
retest this please
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews
Github user ssuchter commented on the issue:
https://github.com/apache/spark/pull/21517
retest this please
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews
Github user ssuchter commented on the issue:
https://github.com/apache/spark/pull/21517
retest this please
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews
Github user ssuchter commented on the issue:
https://github.com/apache/spark/pull/21517
retest this please
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews
Github user ssuchter commented on the issue:
https://github.com/apache/spark/pull/21517
retest this please
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews
Github user ssuchter commented on the issue:
https://github.com/apache/spark/pull/21517
retest this please
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews
Github user ssuchter commented on the issue:
https://github.com/apache/spark/pull/21517
comment
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h
Github user ssuchter commented on the issue:
https://github.com/apache/spark/pull/21517
retest this please
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews
Github user ssuchter commented on the issue:
https://github.com/apache/spark/pull/21517
retest this please
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews
Github user ssuchter commented on the issue:
https://github.com/apache/spark/pull/21517
comment
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h
Github user ssuchter commented on the issue:
https://github.com/apache/spark/pull/21517
comment
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h
Github user ssuchter commented on the issue:
https://github.com/apache/spark/pull/21517
retest this please
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews
Github user ssuchter commented on the issue:
https://github.com/apache/spark/pull/21517
comment
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h
Github user ssuchter commented on the issue:
https://github.com/apache/spark/pull/21517
retest this please
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews
Github user ssuchter commented on the issue:
https://github.com/apache/spark/pull/21517
retest this please
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews
Github user ssuchter commented on the issue:
https://github.com/apache/spark/pull/21517
comment
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h
Github user ssuchter commented on the issue:
https://github.com/apache/spark/pull/21517
retest this please
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews
Github user ssuchter commented on the issue:
https://github.com/apache/spark/pull/21517
another comment
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h
Github user ssuchter commented on the issue:
https://github.com/apache/spark/pull/21517
comment
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h
Github user ssuchter commented on the issue:
https://github.com/apache/spark/pull/21517
retest this please
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews
Github user ssuchter commented on the issue:
https://github.com/apache/spark/pull/21517
Retest this please
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews
Github user ssuchter commented on the issue:
https://github.com/apache/spark/pull/21517
Comment
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h
Github user ssuchter commented on the issue:
https://github.com/apache/spark/pull/21517
retest this please
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews
Github user ssuchter commented on the issue:
https://github.com/apache/spark/pull/21517
retest this please
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews
Github user ssuchter commented on the issue:
https://github.com/apache/spark/pull/21517
Another commnet
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h
Github user ssuchter commented on the issue:
https://github.com/apache/spark/pull/21517
retest this please
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews
Github user ssuchter commented on the issue:
https://github.com/apache/spark/pull/21517
retest this please
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews
Github user ssuchter commented on the issue:
https://github.com/apache/spark/pull/21517
retest this please
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews
GitHub user ssuchter opened a pull request:
https://github.com/apache/spark/pull/21517
Testing k8s change - please ignore (13)
Please ignore this change - testing k8s pull request building.
You can merge this pull request into a Git repository by running:
$ git pull https
Github user ssuchter commented on a diff in the pull request:
https://github.com/apache/spark/pull/20697#discussion_r194150705
--- Diff:
resource-managers/kubernetes/integration-tests/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/KubernetesSuite.scala
---
@@ -0,0
Github user ssuchter commented on a diff in the pull request:
https://github.com/apache/spark/pull/20697#discussion_r194150614
--- Diff:
resource-managers/kubernetes/integration-tests/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/Utils.scala
---
@@ -0,0 +1,88
Github user ssuchter commented on a diff in the pull request:
https://github.com/apache/spark/pull/20697#discussion_r194150624
--- Diff:
resource-managers/kubernetes/integration-tests/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/Utils.scala
---
@@ -0,0 +1,88
Github user ssuchter commented on a diff in the pull request:
https://github.com/apache/spark/pull/20697#discussion_r194133234
--- Diff:
resource-managers/kubernetes/integration-tests/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/KubernetesSuite.scala
---
@@ -0,0
Github user ssuchter commented on a diff in the pull request:
https://github.com/apache/spark/pull/20697#discussion_r193957378
--- Diff:
resource-managers/kubernetes/integration-tests/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/KubernetesSuite.scala
---
@@ -0,0
Github user ssuchter commented on the issue:
https://github.com/apache/spark/pull/20697
Hi folks - I don't think there is any more work to do on this PR. Is there
something you're waiting for before merging
Github user ssuchter commented on the issue:
https://github.com/apache/spark/pull/20697
@mccheah If you want to merge, and then I can fix the commented out test in
another PR, that's ok too.
---
-
To unsubscribe, e
Github user ssuchter commented on a diff in the pull request:
https://github.com/apache/spark/pull/20697#discussion_r192488928
--- Diff:
resource-managers/kubernetes/integration-tests/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/KubernetesSuite.scala
---
@@ -0,0
Github user ssuchter commented on the issue:
https://github.com/apache/spark/pull/20697
@skonto I'll test that and discuss with @shaneknapp. It wouldn't involve
directly changing code in this PR, since the minikube creation/destruction is
done by Jenkins job config
Github user ssuchter commented on a diff in the pull request:
https://github.com/apache/spark/pull/20697#discussion_r192466840
--- Diff:
resource-managers/kubernetes/integration-tests/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/KubernetesSuite.scala
---
@@ -0,0
Github user ssuchter commented on a diff in the pull request:
https://github.com/apache/spark/pull/20697#discussion_r192452101
--- Diff:
resource-managers/kubernetes/integration-tests/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/KubernetesSuite.scala
---
@@ -0,0
Github user ssuchter commented on a diff in the pull request:
https://github.com/apache/spark/pull/20697#discussion_r192447027
--- Diff:
resource-managers/kubernetes/integration-tests/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/KubernetesSuite.scala
---
@@ -0,0
Github user ssuchter commented on the issue:
https://github.com/apache/spark/pull/20697
Correct. I will change the K8s integration tests to do exactly what you
describe above, after this is merged. @vanzin do you want to comment? One of
the two of you will need to do the merge
Github user ssuchter commented on a diff in the pull request:
https://github.com/apache/spark/pull/20697#discussion_r191966223
--- Diff:
resource-managers/kubernetes/integration-tests/scripts/setup-integration-test-env.sh
---
@@ -0,0 +1,91 @@
+#!/usr/bin/env bash
Github user ssuchter commented on a diff in the pull request:
https://github.com/apache/spark/pull/20697#discussion_r191962980
--- Diff:
resource-managers/kubernetes/integration-tests/scripts/setup-integration-test-env.sh
---
@@ -0,0 +1,91 @@
+#!/usr/bin/env bash
Github user ssuchter commented on a diff in the pull request:
https://github.com/apache/spark/pull/20697#discussion_r191961749
--- Diff:
resource-managers/kubernetes/integration-tests/scripts/setup-integration-test-env.sh
---
@@ -0,0 +1,91 @@
+#!/usr/bin/env bash
Github user ssuchter commented on a diff in the pull request:
https://github.com/apache/spark/pull/20697#discussion_r191961317
--- Diff:
resource-managers/kubernetes/integration-tests/scripts/setup-integration-test-env.sh
---
@@ -0,0 +1,91 @@
+#!/usr/bin/env bash
Github user ssuchter commented on the issue:
https://github.com/apache/spark/pull/20697
Any more issues that should be addressed now? Is this ready for merge?
---
-
To unsubscribe, e-mail: reviews-unsubscr
Github user ssuchter commented on a diff in the pull request:
https://github.com/apache/spark/pull/20697#discussion_r191855713
--- Diff:
resource-managers/kubernetes/integration-tests/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/backend/IntegrationTestBackend.scala
Github user ssuchter commented on a diff in the pull request:
https://github.com/apache/spark/pull/20697#discussion_r191629581
--- Diff:
resource-managers/kubernetes/integration-tests/scripts/setup-integration-test-env.sh
---
@@ -0,0 +1,91 @@
+#!/usr/bin/env bash
Github user ssuchter commented on the issue:
https://github.com/apache/spark/pull/20697
Ok, I think all open issues have been resolved. The PRB failures are
because of github request failures, so they are spurious. @vanzin @mccheah I
think it's ready for another look
Github user ssuchter commented on the issue:
https://github.com/apache/spark/pull/20697
Ok, I think all open issues have been resolved. The PRB failures are
because of github request failures, so they are spurious. @vanzin @mccheah I
think it's ready for another look
Github user ssuchter commented on a diff in the pull request:
https://github.com/apache/spark/pull/20697#discussion_r191487129
--- Diff:
resource-managers/kubernetes/integration-tests/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/KubernetesTestComponents.scala
Github user ssuchter commented on a diff in the pull request:
https://github.com/apache/spark/pull/20697#discussion_r191486261
--- Diff:
resource-managers/kubernetes/integration-tests/src/test/resources/log4j.properties
---
@@ -0,0 +1,31 @@
+#
+# Licensed to the Apache
Github user ssuchter commented on a diff in the pull request:
https://github.com/apache/spark/pull/20697#discussion_r191485369
--- Diff: resource-managers/kubernetes/integration-tests/pom.xml ---
@@ -0,0 +1,230 @@
+
+
+http://maven.apache.org/POM/4.0.0;
xmlns:xsi="
Github user ssuchter commented on a diff in the pull request:
https://github.com/apache/spark/pull/20697#discussion_r191483976
--- Diff: resource-managers/kubernetes/integration-tests/pom.xml ---
@@ -0,0 +1,230 @@
+
+
+http://maven.apache.org/POM/4.0.0;
xmlns:xsi="
Github user ssuchter commented on a diff in the pull request:
https://github.com/apache/spark/pull/20697#discussion_r191483921
--- Diff: resource-managers/kubernetes/integration-tests/pom.xml ---
@@ -0,0 +1,230 @@
+
+
+http://maven.apache.org/POM/4.0.0;
xmlns:xsi="
Github user ssuchter commented on a diff in the pull request:
https://github.com/apache/spark/pull/20697#discussion_r191474516
--- Diff:
resource-managers/kubernetes/integration-tests/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/Logging.scala
---
@@ -0,0 +1,35
Github user ssuchter commented on the issue:
https://github.com/apache/spark/pull/20697
I'll work on Matt's comments from Friday next.
Here's the output (after the bugfix) from running against mainline:
```
MBP:~/src/ssuchter-spark% git remote get-url origin
Github user ssuchter commented on the issue:
https://github.com/apache/spark/pull/20697
Fixed the bug. @mccheah I'd appreciate your eyes on commit 1d8a265, for
both correctness and style. (I haven't used Scala before this project, so I'm
very not confidence in the best way to do
Github user ssuchter commented on the issue:
https://github.com/apache/spark/pull/20697
Hm, I'm glad I tried to paste output of running against master (well,
strictly my rebase of this PR, but functionally the same thing). It's throwing
some error that I need to debug
Github user ssuchter commented on the issue:
https://github.com/apache/spark/pull/20697
Ok, I didn't do anything wrong. There is a bug in the Github web UI that
the PR's web view doesn't get updated when I rebase it. I had to change what
Github thought the branch I was merging
Github user ssuchter commented on the issue:
https://github.com/apache/spark/pull/20697
Oops, I really did something wrong there. Please ignore while I fix.
---
-
To unsubscribe, e-mail: reviews-unsubscr
Github user ssuchter commented on the issue:
https://github.com/apache/spark/pull/20697
FYI - I just rebased to an updated master.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user ssuchter commented on the issue:
https://github.com/apache/spark/pull/20697
Thanks for the pointer about @ifilonenko's comment. I removed the
Kerberos/secrets related code. I'll post the output momentarily
Github user ssuchter commented on a diff in the pull request:
https://github.com/apache/spark/pull/20697#discussion_r191031379
--- Diff:
resource-managers/kubernetes/integration-tests/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/KubernetesSuite.scala
---
@@ -0,0
Github user ssuchter commented on the issue:
https://github.com/apache/spark/pull/20697
retest this please
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews
Github user ssuchter commented on the issue:
https://github.com/apache/spark/pull/20697
I saw that @mccheah put a comment in the e2e-prow.sh that we don't need it
here, so I took that as an answer to my previous question.
I think I got rid of the cloning and building Spark
Github user ssuchter commented on a diff in the pull request:
https://github.com/apache/spark/pull/20697#discussion_r191028169
--- Diff: resource-managers/kubernetes/integration-tests/e2e/e2e-prow.sh ---
@@ -0,0 +1,69 @@
+#!/bin/bash
+
+# Licensed to the Apache Software
Github user ssuchter commented on a diff in the pull request:
https://github.com/apache/spark/pull/20697#discussion_r191028204
--- Diff: resource-managers/kubernetes/integration-tests/include/util.sh ---
@@ -0,0 +1,43 @@
+#!/usr/bin/env bash
+
+# Licensed to the Apache
Github user ssuchter commented on the issue:
https://github.com/apache/spark/pull/20697
I will take a look at the docs too.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands
Github user ssuchter commented on the issue:
https://github.com/apache/spark/pull/20697
You're right, the prow-based codepath still does the cloning. Sorry! I did
remove the cloning and building option in the entry point
resource-managers/kubernetes/integration-tests/dev/dev-run
Github user ssuchter commented on the issue:
https://github.com/apache/spark/pull/20697
I fixed all the other issues (removal of cloud-based k8s backend,
repository clone logic).
@erikerlandson @mccheah @vanzin PTAL
Github user ssuchter commented on the issue:
https://github.com/apache/spark/pull/20697
> If not all hosts currently can run them, then it's ok to postpone this.
That's exactly what I'm proposing too, so it seems like we agree. I also
agree, btw, on the future state - w
Github user ssuchter commented on the issue:
https://github.com/apache/spark/pull/20697
Hm, that wasn't the answer I expected to hear. It leads me to another
question:
If we don't have developers using this script directly, what's the value of
trying to incorporate
Github user ssuchter commented on the issue:
https://github.com/apache/spark/pull/20697
@vanzin @felixcheung I'm trying to look into this again. My main question
is how people use this dev/run-tests.py suite. Every time I run it, even though
I've only changed things in the kubernetes
Github user ssuchter commented on the issue:
https://github.com/apache/spark/pull/21317
Hm, I started and stopped minikube, but I don't know why that would fix
this. I suspect that it was actually fixed by the state on the node changing in
the last few hours. Please highlight
Github user ssuchter commented on the issue:
https://github.com/apache/spark/pull/21317
retest this please
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews
Github user ssuchter commented on the issue:
https://github.com/apache/spark/pull/21317
Something seems odd on the minikube on AMPLab jenkins. Lookingâ¦
On May 17, 2018 at 10:57:32 AM, Anirudh Ramanathan
(notificati...@github.com)
wrote:
@skonto <ht
Github user ssuchter commented on the issue:
https://github.com/apache/spark/pull/20697
What about the issue of moving the invocation of the tests into
run-tests.py? Erik - do you not believe that is a requirement
Github user ssuchter commented on the issue:
https://github.com/apache/spark/pull/20697
Yeah, I just figured that out. (That it's because of the dist/ directory.)
I'll change dev/tox.ini so I don't have to keep cleaning. Thanks
Github user ssuchter commented on the issue:
https://github.com/apache/spark/pull/20697
@vanzin I'm working on the run-tests.py integration. It seems to be failing
on a bunch of comments on the file dist/python/docs/conf.py
./dist/python/docs/conf.py:258:1: E265 block comment
Github user ssuchter commented on a diff in the pull request:
https://github.com/apache/spark/pull/20697#discussion_r172707045
--- Diff: resource-managers/kubernetes/integration-tests/include/util.sh ---
@@ -0,0 +1,43 @@
+#!/usr/bin/env bash
+
+# Licensed to the Apache
Github user ssuchter commented on the issue:
https://github.com/apache/spark/pull/20697
I agree with your point about using a non-tgz version, but I'd prefer to do
that as a separate change. (However, I reserve my right to change this stance
if it turns out to be easier to make
Github user ssuchter commented on a diff in the pull request:
https://github.com/apache/spark/pull/20697#discussion_r172691439
--- Diff:
resource-managers/kubernetes/integration-tests/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/backend/cloud/CloudTestBackend.scala
Github user ssuchter commented on the issue:
https://github.com/apache/spark/pull/20697
It's hard to detect when there are interesting changes, because changes to
spark core code can meaningfully affect the K8s code paths. Many, many changes
might be considered "intere
Github user ssuchter commented on the issue:
https://github.com/apache/spark/pull/20697
@vanzin I fixed the title (thanks @liyinan926) with the bug.
I'll work on integrating its build with the main build.
In terms of integration with the dev/run-tests.py, we at most
Github user ssuchter commented on the issue:
https://github.com/apache/spark/pull/20697
I should note that the integration test success that came from SparkQA is
**not** using the copy of the integration test code mentioned in this PR. It is
using the original copy of the code
1 - 100 of 118 matches
Mail list logo
