[ https://issues.apache.org/jira/browse/SPARK-32032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17215352#comment-17215352 ]
Hélder Hugo Ferreira commented on SPARK-32032: ---------------------------------------------- Hi, we are getting the following error when using the kafka StreamReader functionality "_option("startingOffsetsByTimestamp")_": {code:java} WARN KafkaOffsetReader: Error in attempt 1 getting Kafka offsets: java.lang.AssertionError: assertion failed: No offset matched from request of topic-partition MyTopic-6 and timestamp 1602673380000. at scala.Predef$.assert(Predef.scala:223) at org.apache.spark.sql.kafka010.KafkaOffsetReader.$anonfun$fetchSpecificTimestampBasedOffsets$6(KafkaOffsetReader.scala:238) at scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:238) at scala.collection.Iterator.foreach(Iterator.scala:941) at scala.collection.Iterator.foreach$(Iterator.scala:941) at scala.collection.AbstractIterator.foreach(Iterator.scala:1429) at scala.collection.IterableLike.foreach(IterableLike.scala:74) at scala.collection.IterableLike.foreach$(IterableLike.scala:73) at scala.collection.AbstractIterable.foreach(Iterable.scala:56) at scala.collection.TraversableLike.map(TraversableLike.scala:238) at scala.collection.TraversableLike.map$(TraversableLike.scala:231) at scala.collection.AbstractTraversable.map(Traversable.scala:108) at org.apache.spark.sql.kafka010.KafkaOffsetReader.$anonfun$fetchSpecificTimestampBasedOffsets$4(KafkaOffsetReader.scala:236) at org.apache.spark.sql.kafka010.KafkaOffsetReader.$anonfun$fetchSpecificOffsets0$1(KafkaOffsetReader.scala:265) at org.apache.spark.sql.kafka010.KafkaOffsetReader.$anonfun$partitionsAssignedToConsumer$2(KafkaOffsetReader.scala:550) at org.apache.spark.sql.kafka010.KafkaOffsetReader.$anonfun$withRetriesWithoutInterrupt$1(KafkaOffsetReader.scala:600) at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23) at org.apache.spark.util.UninterruptibleThread.runUninterruptibly(UninterruptibleThread.scala:77) at org.apache.spark.sql.kafka010.KafkaOffsetReader.withRetriesWithoutInterrupt(KafkaOffsetReader.scala:599) at org.apache.spark.sql.kafka010.KafkaOffsetReader.$anonfun$partitionsAssignedToConsumer$1(KafkaOffsetReader.scala:536) at org.apache.spark.sql.kafka010.KafkaOffsetReader.runUninterruptibly(KafkaOffsetReader.scala:567) at org.apache.spark.sql.kafka010.KafkaOffsetReader.partitionsAssignedToConsumer(KafkaOffsetReader.scala:536) at org.apache.spark.sql.kafka010.KafkaOffsetReader.fetchSpecificOffsets0(KafkaOffsetReader.scala:261) at org.apache.spark.sql.kafka010.KafkaOffsetReader.fetchSpecificTimestampBasedOffsets(KafkaOffsetReader.scala:254) at org.apache.spark.sql.kafka010.KafkaMicroBatchStream.$anonfun$getOrCreateInitialPartitionOffsets$1(KafkaMicroBatchStream.scala:157) at scala.Option.getOrElse(Option.scala:189) at org.apache.spark.sql.kafka010.KafkaMicroBatchStream.getOrCreateInitialPartitionOffsets(KafkaMicroBatchStream.scala:148) at org.apache.spark.sql.kafka010.KafkaMicroBatchStream.initialOffset(KafkaMicroBatchStream.scala:76) at ... {code} We figured out in this topic partition there was only 1 message older than the input timestamps set in the _startingOffsetsByTimestamp_. We are using kafka 2.5.1. Since we noticed in this PR changes are being performed in the _fetchSpecificTimestampBasedOffsets_ function, any chance that this will also resolve our issue? Thanks in advance. Best Regards, Hélder Hugo Ferreira > Eliminate deprecated poll(long) API calls to avoid infinite wait in driver > -------------------------------------------------------------------------- > > Key: SPARK-32032 > URL: https://issues.apache.org/jira/browse/SPARK-32032 > Project: Spark > Issue Type: Sub-task > Components: Structured Streaming > Affects Versions: 3.1.0 > Reporter: Gabor Somogyi > Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org