Github user ssaavedra commented on the issue:
https://github.com/apache/spark/pull/20383
Yes, sorry for the misunderstanding I was also probably too eager with
this. However, if the changes I'm stating up there don't work, I am not sure
what I'm missing now. I'll take a further look
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/20383
I agree. sorry to merge it so quickly, let me revert it.
@ssaavedra would you please submit PR again when everything is done, thanks!
---
Github user felixcheung commented on the issue:
https://github.com/apache/spark/pull/20383
my take is not that it doesn't work but some names are out of date because
it was done for the k8s fork.
I think we should revert the commit and wait till it is tested out
complete. WDYT?
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/20383
@ssaavedra Do you mean the your current patch doesn't work even for master
branch? In case of that do we need to revert the current patch? CC @felixcheung
@foxish .
---
Github user ssaavedra commented on the issue:
https://github.com/apache/spark/pull/20383
Sorry, I hadn't answered yet because it seems my patch does not work
cleanly on 2.3. Many names were rewritten as part of the merge and some logic
on how the executor pods look up the configMap
Github user foxish commented on the issue:
https://github.com/apache/spark/pull/20383
That plan LGTM - we can merge into 2.3 after removing the non-existent
config, and getting a clean test run against the 2.3 branch.
Should be low risk.
---
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/20383
@ssaavedra why don't you submit a follow-up PR to remove the inexistent
configuration as mentioned above?
If we agreed to backport to 2.3, then you can create another backport PR
Github user ssaavedra commented on the issue:
https://github.com/apache/spark/pull/20383
I can probably take a look at testing this over 2.3.0-rc2 on Monday. I did
not test this on a clean 2.3.0-ish branch.
---
-
Github user felixcheung commented on the issue:
https://github.com/apache/spark/pull/20383
So you have tested this on latest Spark 2.3.0 bit?
Test aside, do people think it is useful to include this fix in the 2.3.0
release?
---
Github user ssaavedra commented on the issue:
https://github.com/apache/spark/pull/20383
spark-integration was created much later. I originally opened this as
https://github.com/apache-spark-on-k8s/spark/pull/516 last September. However,
the integration tests repo exists since
Github user felixcheung commented on the issue:
https://github.com/apache/spark/pull/20383
without this fix streaming app can't properly recovery from checkpoint,
correct?
that seems fairly important to me.
was apache-spark-on-k8s/spark-integration done before this PR was
Github user foxish commented on the issue:
https://github.com/apache/spark/pull/20383
I'm ok with backporting it once the non-existent config is removed and
we're confident we're covering all the requisite config. Also would make sense
to have a test under
Github user ssaavedra commented on the issue:
https://github.com/apache/spark/pull/20383
However, Spark Streaming should always be used with checkpoint enabled if
you are using at least `updateStateByKey` or `reduceByKeyAndWindow` and you
don't want to lose data, or miscalculate
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/20383
This is not a big issue unless when we run Spark Streaming with checkpoint
enabled. I'm not sure for now it is OK to add to 2.3.0 release (as this is not
a block issue).
@felixcheung up
Github user felixcheung commented on the issue:
https://github.com/apache/spark/pull/20383
We are going to cut another RC?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands,
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/20383
hmmm.. The target version in the JIRA is 2.4.0, and we about to release
2.3.0, are we sure we want to backport?
---
-
To
Github user felixcheung commented on the issue:
https://github.com/apache/spark/pull/20383
actually, can we backport this to branch-2.3? k8s is added in 2.3.0
---
-
To unsubscribe, e-mail:
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/20383
Merging to Master, thanks!
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail:
Github user zsxwing commented on the issue:
https://github.com/apache/spark/pull/20383
LGTM
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail:
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/20383
LGTM.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail:
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20383
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/86596/
Test PASSed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20383
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20383
**[Test build #86596 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86596/testReport)**
for PR 20383 at commit
Github user felixcheung commented on the issue:
https://github.com/apache/spark/pull/20383
@zsxwing @jerryshao
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail:
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20383
**[Test build #86596 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86596/testReport)**
for PR 20383 at commit
Github user felixcheung commented on the issue:
https://github.com/apache/spark/pull/20383
Jenkins test this please
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail:
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20383
Can one of the admins verify this patch?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20383
Can one of the admins verify this patch?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
28 matches
Mail list logo
