GitHub user zsxwing opened a pull request:
https://github.com/apache/spark/pull/22293
[SPARK-25288][Tests]Fix flaky Kafka transaction tests
## What changes were proposed in this pull request?
Here are the failures:
http://spark-tests.appspot.com/test-details?suite_name=org.apache.spark.sql.kafka010.KafkaRelationSuite&test_name=read+Kafka+transactional+messages%3A+read_committed
http://spark-tests.appspot.com/test-details?suite_name=org.apache.spark.sql.kafka010.KafkaMicroBatchV1SourceSuite&test_name=read+Kafka+transactional+messages%3A+read_committed
http://spark-tests.appspot.com/test-details?suite_name=org.apache.spark.sql.kafka010.KafkaMicroBatchV2SourceSuite&test_name=read+Kafka+transactional+messages%3A+read_committed
I found the Kafka consumer may not see the committed messages for a short
time. This PR just adds a new method `waitUntilOffsetAppears` and uses it to
make sure the consumer can see a specified offset before checking the result.
## How was this patch tested?
Jenkins
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/zsxwing/spark SPARK-25288
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/22293.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #22293
----
commit 6439367e3dfd9612f30395ae445df67a87ede871
Author: Shixiong Zhu <zsxwing@...>
Date: 2018-08-30T21:44:42Z
Fix flaky Kafka transaction tests
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
