gaborgsomogyi commented on issue #25135: [SPARK-28367][SS] Use new KafkaConsumer.poll API in Kafka connector URL: https://github.com/apache/spark/pull/25135#issuecomment-512162339 Little bit explanation to the last change: * While I was debugging the issue I've realized `requirement failed` message is not really talkative so added a meaningful message * When `consumer.poll(Duration.ZERO)` with `Thread.sleep(100)` used then under some circumstances even 2 minutes was not enough to get the assignment. To test this I've put the standalone application into a 1k execution loop. The end result is that with non-zero timeout (including `Duration.ofMillis(1)`) everything works fine. So the end judgement is that the actual implementation works but as a side track I'm going to analyze the `consumer.poll(Duration.ZERO)` behavior with the Kafka guys.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
