[
https://issues.apache.org/jira/browse/SPARK-15406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15488558#comment-15488558
]
Cody Koeninger commented on SPARK-15406:
----------------------------------------
So you're saying the type of K and V will always be bytearray, custom kafka
deserializers won't be used, and message schema will be handled by mapping
after the fact? That's potentially OK, but I'd feel a lot better with a working
example. I can play around with that a bit.
Regarding the statefulness of kafka consumers, I just mean you can't construct
a kafka consumer that is already subscribed and has correct positions; you need
to instantiate it and then call side-effecrting operations on it in order to
get it in the right state.
Regarding Json convenience constructors, that's not purely additive. If we know
that's the goal, we shouldn't use hacks like comma separated strings or
prefixed keys now and get people relying on them.
> Structured streaming support for consuming from Kafka
> -----------------------------------------------------
>
> Key: SPARK-15406
> URL: https://issues.apache.org/jira/browse/SPARK-15406
> Project: Spark
> Issue Type: New Feature
> Reporter: Cody Koeninger
>
> This is the parent JIRA to track all the work for the building a Kafka source
> for Structured Streaming. Here is the design doc for an initial version of
> the Kafka Source.
> https://docs.google.com/document/d/19t2rWe51x7tq2e5AOfrsM9qb8_m7BRuv9fel9i0PqR8/edit?usp=sharing
> ================== Old description =========================
> Structured streaming doesn't have support for kafka yet. I personally feel
> like time based indexing would make for a much better interface, but it's
> been pushed back to kafka 0.10.1
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-33+-+Add+a+time+based+log+index
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
