[ https://issues.apache.org/jira/browse/SPARK-26848?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sean Owen resolved SPARK-26848. ------------------------------- Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 23747 [https://github.com/apache/spark/pull/23747] > Introduce new option to Kafka source - specify timestamp to start and end > offset > -------------------------------------------------------------------------------- > > Key: SPARK-26848 > URL: https://issues.apache.org/jira/browse/SPARK-26848 > Project: Spark > Issue Type: Improvement > Components: SQL > Affects Versions: 3.0.0 > Reporter: Jungtaek Lim > Assignee: Jungtaek Lim > Priority: Major > Fix For: 3.0.0 > > > Kafka source (for SQL/SS) provides options to set specific offset per topic > partition to let source starts reading from start offsets and ends reading > until end offsets. ("startingOffsets" and "endingOffsets" in below document.) > http://spark.apache.org/docs/latest/structured-streaming-kafka-integration.html > I'd like to introduce new options "startingOffsetsByTimestamp" and > "endingOffsetsByTimestamp" to set specific timestamp per topic (since we're > unlikely to set the different value per partition) to let source starts > reading from offsets which have equal of greater timestamp, and ends reading > until offsets which have equal of greater timestamp. > The option would be optional of course, and have a preference to apply the > options 1) timestamp option applies first, and if it doesn't exist, offset > option applies. -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org