Github user jose-torres commented on the issue:
https://github.com/apache/spark/pull/20622
The difference in ContinuousExecution is that the thread isn't doing any
metadata work like looking for new batches - it's either running the Spark job
or cleaning up after finishing it.
Github user tdas commented on the issue:
https://github.com/apache/spark/pull/20622
Unfortunately, we havent figured out any good to avoid that for
MicroBatchExecution till now. The streaming thread is doing a whole lot of
different things, and interrupting is the only reliable way
Github user zsxwing commented on the issue:
https://github.com/apache/spark/pull/20622
> Using a thread interrupt to kill the stream appears to be a fundamentally
fragile solution, requiring us to maintain a whitelist of exceptions we think
Spark execution might surface in response
Github user jose-torres commented on the issue:
https://github.com/apache/spark/pull/20622
The alternate solution doesn't solve the problem SPARK-23441 sets out to
address. Using a thread interrupt to kill the spark job is a fundamentally
fragile solution, requiring us to maintain a
Github user jose-torres commented on the issue:
https://github.com/apache/spark/pull/20622
The issue that caused the test flakiness is as you say. But I had
understood that interrupting is itself problematic, requiring us to maintain an
awkward whitelist of possible exceptions in
Github user tdas commented on the issue:
https://github.com/apache/spark/pull/20622
@jose-torres I had a long offline chat with @zsxwing, kudos to him for
catching a corner case in the current solution. The following sequence of
events may occur.
- In the query thread, the
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20622
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20622
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/87517/
Test PASSed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20622
**[Test build #87517 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87517/testReport)**
for PR 20622 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20622
**[Test build #87517 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87517/testReport)**
for PR 20622 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20622
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/87499/
Test PASSed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20622
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20622
**[Test build #87499 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87499/testReport)**
for PR 20622 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20622
**[Test build #87499 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87499/testReport)**
for PR 20622 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20622
Merged build finished. Test FAILed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20622
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/87495/
Test FAILed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20622
**[Test build #87495 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87495/testReport)**
for PR 20622 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20622
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/87491/
Test FAILed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20622
Merged build finished. Test FAILed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20622
**[Test build #87491 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87491/testReport)**
for PR 20622 at commit
Github user jose-torres commented on the issue:
https://github.com/apache/spark/pull/20622
@zsxwing pointed out that the original behavior was more subtly wrong than
I expected.
What we want to do is cancel the Spark job, and then cleanly restart it
from the last checkpoint.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20622
**[Test build #87495 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87495/testReport)**
for PR 20622 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20622
**[Test build #87491 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87491/testReport)**
for PR 20622 at commit
Github user jose-torres commented on the issue:
https://github.com/apache/spark/pull/20622
retest this please
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail:
Github user jose-torres commented on the issue:
https://github.com/apache/spark/pull/20622
StreamingOuterJoinSuite failure is a known flakiness issue.
---
-
To unsubscribe, e-mail:
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20622
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/87486/
Test FAILed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20622
Merged build finished. Test FAILed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20622
**[Test build #87486 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87486/testReport)**
for PR 20622 at commit
Github user mgaido91 commented on the issue:
https://github.com/apache/spark/pull/20622
thanks for pinging me @jose-torres! Unfortunately I don't know yet
structured streaming codebase well enough to give a feedback. Thanks anyway for
looking at it!
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20622
**[Test build #87486 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87486/testReport)**
for PR 20622 at commit
Github user jose-torres commented on the issue:
https://github.com/apache/spark/pull/20622
cc @mgaido91 - this should completely resolve the other symptom you posted
in SPARK-23416
cc @zsxwing @tdas
---
-
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20622
Can one of the admins verify this patch?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
32 matches
Mail list logo
