[ https://issues.apache.org/jira/browse/SPARK-28367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17159244#comment-17159244 ]
Gabor Somogyi edited comment on SPARK-28367 at 7/16/20, 2:22 PM: ----------------------------------------------------------------- I'm going to have a chat w/ the Flink guys how they've overcome offset fetch issue w/ the new poll(Duration) API. Maybe we learn something. was (Author: gsomogyi): I'm going to have a chat w/ the Flink guys how they've overcome offset fetch issue. Maybe we learn something. > Kafka connector infinite wait because metadata never updated > ------------------------------------------------------------ > > Key: SPARK-28367 > URL: https://issues.apache.org/jira/browse/SPARK-28367 > Project: Spark > Issue Type: Bug > Components: Structured Streaming > Affects Versions: 2.1.3, 2.2.3, 2.3.3, 2.4.3, 3.0.0, 3.1.0 > Reporter: Gabor Somogyi > Priority: Critical > > Spark uses an old and deprecated API named poll(long) which never returns and > stays in live lock if metadata is not updated (for instance when broker > disappears at consumer creation). > I've created a small standalone application to test it and the alternatives: > https://github.com/gaborgsomogyi/kafka-get-assignment -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org