[ https://issues.apache.org/jira/browse/SPARK-15406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15486131#comment-15486131 ]
Cody Koeninger commented on SPARK-15406: ---------------------------------------- I've got a minimal working Source and SourceProvider, at least for topics that are String key and value only, at https://github.com/apache/spark/compare/master...koeninger:SPARK-15406 If you haven't already attempted an implementation, I'd suggest at least looking at that before writing up a design doc that may or may not address some of the pragmatic issues. The big thing I'm running into, and maybe I'm just not understanding the intention behind the SourceProvider interface, is that putting all configuration through a Map[String, String] makes it super awkward to configure types, or classes, or collections of offsets, or... anything really. Another significant issue is that I have no idea how rate limiting is supposed to work. > Structured streaming support for consuming from Kafka > ----------------------------------------------------- > > Key: SPARK-15406 > URL: https://issues.apache.org/jira/browse/SPARK-15406 > Project: Spark > Issue Type: New Feature > Reporter: Cody Koeninger > > Structured streaming doesn't have support for kafka yet. I personally feel > like time based indexing would make for a much better interface, but it's > been pushed back to kafka 0.10.1 > https://cwiki.apache.org/confluence/display/KAFKA/KIP-33+-+Add+a+time+based+log+index -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org