Github user srowen commented on the issue:
https://github.com/apache/spark/pull/21488
Ack I missed something here: there's an override of kafka.version for Scala
2.12, from when it had to be bumped _up_ to work with 2.12. That no longer
works when compiling with 2.12. I'll submit a
Github user ijuma commented on the issue:
https://github.com/apache/spark/pull/21488
@wangyum, can you please file a Kafka JIRA with details of what the test is
doing (even if the failure is transient)? From the stacktrace, it looks like a
potential broker issue (assuming there are
Github user srowen commented on the issue:
https://github.com/apache/spark/pull/21488
Hm, looking at the dashboard, I don't see this failure consistently in the
master tests:
https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test%20(Dashboard)/
Let's keep a
Github user tedyu commented on the issue:
https://github.com/apache/spark/pull/21488
I used the following command and the test passed:
mvn test -Phadoop-2.6 -Pyarn -Phive -Dtest=KafkaMicroBatchSourceSuite -rf
external/kafka-0-10-sql
Please take a look at the 'Disk
Github user wangyum commented on the issue:
https://github.com/apache/spark/pull/21488
It seems this commit cause `KafkaSourceStressForDontFailOnDataLossSuite`
failed:
```java
...
[info] KafkaSourceStressSuite:
[info] - stress test with multiple topics and partitions
Github user zsxwing commented on the issue:
https://github.com/apache/spark/pull/21488
Thanks! Merging to master.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail:
Github user srowen commented on the issue:
https://github.com/apache/spark/pull/21488
Yes no objections. There's no concrete problem, and there is upside to
making the change. I think you're aware of the potential issues, and have
thought through them in the context of deeper
Github user zsxwing commented on the issue:
https://github.com/apache/spark/pull/21488
@srowen May I read your comment as "no objections"? The current PR looks
good to me. If you don't have objections, I will go ahead and merge it.
---
Github user srowen commented on the issue:
https://github.com/apache/spark/pull/21488
Yes, good argument. Having been burned in the past (actually, by Kafka and
ZK changes, though that's in the past), I'm aware that even compiling against
version B instead of A, with no library code
Github user ijuma commented on the issue:
https://github.com/apache/spark/pull/21488
Anyway, overall I think you should definitely make this change. Spark users
are currently penalised heavily when running on clusters with the message
format introduced in 0.11.0, which has important
Github user zsxwing commented on the issue:
https://github.com/apache/spark/pull/21488
@ijuma I see. I was looking at 0.10 jar. Thanks for correcting me.
---
-
To unsubscribe, e-mail:
Github user ijuma commented on the issue:
https://github.com/apache/spark/pull/21488
@zsxwing `org.apache.kafka.clients.admin.AdminClient` is in the clients
jar. It's not relevant for this PR, I am just mentioning it so if Spark decides
to migrate to that eventually. I see code using
Github user ijuma commented on the issue:
https://github.com/apache/spark/pull/21488
Yeah, the Java client libraries have been evolved in a compatible manner
for the most part since 0.10.0. The set of broker versions supported by 0.10.0
and 2.0.0 is exactly the same.
The
Github user zsxwing commented on the issue:
https://github.com/apache/spark/pull/21488
@srowen just to be clear, `AdminClient` is not in `kafka-clients` jar. The
user has to add `kafka` jar as a dependency to use `AdminClient`.
In addition, even if this upgrade has some
Github user srowen commented on the issue:
https://github.com/apache/spark/pull/21488
Yes, and that then impacts programs that depend on this module to use Kafka
0.10+ from Spark. They'd have to update `kafka-clients` too, or would be
automatically updated really. I don't see a
Github user ijuma commented on the issue:
https://github.com/apache/spark/pull/21488
@srowen I meant that the library is compatible, so if you just change the
version in the pom, it's fine. If you changed the code to use some methods in
AdminClient, then you'd have to be more
Github user srowen commented on the issue:
https://github.com/apache/spark/pull/21488
@ijuma this changes the version of `kakfa-clients` in compile scope, too --
right? looks like it in the POM. That's not just a test change. It's probably
OK if the client is backwards-compatible
Github user ijuma commented on the issue:
https://github.com/apache/spark/pull/21488
Yes, Kafka clients 0.10.2 and higher support brokers from 0.10.0 and higher
if the protocols being used are available in that version. Since we only
changed test code in this PR, upgrading to 2.0.0
Github user zsxwing commented on the issue:
https://github.com/apache/spark/pull/21488
> I might have missed this in the shuffle here -- is this fully compatible
with 0.10.x brokers too?
Yep. @ijuma could you confirm it?
---
Github user srowen commented on the issue:
https://github.com/apache/spark/pull/21488
I might have missed this in the shuffle here -- is this fully compatible
with 0.10.x brokers too?
---
-
To unsubscribe, e-mail:
Github user tedyu commented on the issue:
https://github.com/apache/spark/pull/21488
@zsxwing
Is there anything I should do for this PR ?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/21488
https://github.com/apache/kafka/releases KAFKA 2.0.0 release is publicly
available. We can finish it before the code freeze?
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21488
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21488
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/93745/
Test PASSed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21488
**[Test build #93745 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93745/testReport)**
for PR 21488 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21488
**[Test build #93745 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93745/testReport)**
for PR 21488 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21488
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21488
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user tedyu commented on the issue:
https://github.com/apache/spark/pull/21488
retest this please
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail:
Github user tedyu commented on the issue:
https://github.com/apache/spark/pull/21488
```
22:36:05.028 WARN org.apache.spark.scheduler.TaskSetManager: Lost task 0.0
in stage 16314.0 (TID 39181, localhost, executor driver):
java.io.FileNotFoundException: File
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21488
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/93736/
Test FAILed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21488
Merged build finished. Test FAILed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21488
**[Test build #93736 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93736/testReport)**
for PR 21488 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21488
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21488
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21488
**[Test build #93736 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93736/testReport)**
for PR 21488 at commit
Github user tedyu commented on the issue:
https://github.com/apache/spark/pull/21488
Ryan:
Thanks for the close follow-up.
Once Kafka 2.0.0 is released, I will incorporate the above.
---
-
To
Github user zsxwing commented on the issue:
https://github.com/apache/spark/pull/21488
@tedyu fyi, here is the last fix for 2.0.0:
https://github.com/zsxwing/spark/commit/7ccc520b540106ca4c2dac2e42834984332d6804
---
Github user zsxwing commented on the issue:
https://github.com/apache/spark/pull/21488
@tedyu I just realized topic deletion might be used by some users. This
Kafka version will introduce a regression for them. Since 2.0 fixes this issue
and is going out soon, how about just wait for
Github user tedyu commented on the issue:
https://github.com/apache/spark/pull/21488
@zsxwing
Is there anything that needs to be done from my side ?
---
-
To unsubscribe, e-mail:
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21488
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/93374/
Test PASSed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21488
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21488
**[Test build #93374 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93374/testReport)**
for PR 21488 at commit
Github user tedyu commented on the issue:
https://github.com/apache/spark/pull/21488
Test failure was in Hive test, not related to this PR.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21488
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/93372/
Test FAILed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21488
Merged build finished. Test FAILed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21488
**[Test build #93372 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93372/testReport)**
for PR 21488 at commit
Github user guozhangwang commented on the issue:
https://github.com/apache/spark/pull/21488
> 1.1.1 has been released, maybe we can upgrade to that.
+1
---
-
To unsubscribe, e-mail:
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21488
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21488
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21488
**[Test build #93374 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93374/testReport)**
for PR 21488 at commit
Github user tedyu commented on the issue:
https://github.com/apache/spark/pull/21488
Thanks for the reminder, @ijuma
Updated pom.xml and title accordingly.
---
-
To unsubscribe, e-mail:
Github user ijuma commented on the issue:
https://github.com/apache/spark/pull/21488
1.1.1 has been released, maybe we can upgrade to that.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21488
**[Test build #93372 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93372/testReport)**
for PR 21488 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21488
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21488
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user tedyu commented on the issue:
https://github.com/apache/spark/pull/21488
Ryan:
Thanks for the reminder.
I have disabled that test.
---
-
To unsubscribe, e-mail:
57 matches
Mail list logo
