[ https://issues.apache.org/jira/browse/SPARK-28367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17140319#comment-17140319 ]
Gabor Somogyi commented on SPARK-28367: --------------------------------------- I think we can split the problem into 2 pieces. Driver and executor side. The executor side is not problematic and can be done w/o new API. The driver side requires further consideration and effort. Creating subtasks and PR for executor side. > Kafka connector infinite wait because metadata never updated > ------------------------------------------------------------ > > Key: SPARK-28367 > URL: https://issues.apache.org/jira/browse/SPARK-28367 > Project: Spark > Issue Type: Bug > Components: Structured Streaming > Affects Versions: 2.1.3, 2.2.3, 2.3.3, 2.4.3, 3.0.0 > Reporter: Gabor Somogyi > Priority: Critical > > Spark uses an old and deprecated API named poll(long) which never returns and > stays in live lock if metadata is not updated (for instance when broker > disappears at consumer creation). > I've created a small standalone application to test it and the alternatives: > https://github.com/gaborgsomogyi/kafka-get-assignment -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org