Ok... in general it seems to me like effort would be better spent trying to help upstream, as opposed to us making a 5th slightly different interface to kafka (currently have 0.8 receiver, 0.8 dstream, 0.10 dstream, 0.10 structured stream)
On Tue, Nov 8, 2016 at 10:05 PM, Mark Grover <m...@apache.org> wrote: > I think they are open to others helping, in fact, more than one person has > worked on the JIRA so far. And, it's been crawling really slowly and that's > preventing adoption of Spark's new connector in secure Kafka environments. > > On Tue, Nov 8, 2016 at 7:59 PM, Cody Koeninger <c...@koeninger.org> wrote: >> >> Have you asked the assignee on the Kafka jira whether they'd be >> willing to accept help on it? >> >> On Tue, Nov 8, 2016 at 5:26 PM, Mark Grover <m...@apache.org> wrote: >> > Hi all, >> > We currently have a new direct stream connector, thanks to work by Cody >> > and >> > others on SPARK-12177. >> > >> > However, that can't be used in secure clusters that require Kerberos >> > authentication. That's because Kafka currently doesn't support >> > delegation >> > tokens (KAFKA-1696). Unfortunately, very little work has been done on >> > that >> > JIRA, so, in my opinion, folks who want to use secure Kafka (using the >> > norm >> > - Kerberos) can't do so because Spark Streaming can't consume from it >> > today. >> > >> > The right way is, of course, to get delegation tokens in Kafka but >> > honestly >> > I don't know if that's happening in the near future. I am wondering if >> > we >> > should consider something to remedy this - for example, we could come up >> > with a receiver based connector based on the new Kafka consumer API >> > that'd >> > support kerberos authentication. It won't require delegation tokens >> > since >> > there's only a very small number of executors talking to Kafka. Of >> > course, >> > for anyone who cares about high throughput and other direct connector >> > benefits would have to use direct connector. Another thing we could do >> > is >> > ship the keytab to the executors in the direct connector, so delegation >> > tokens are not required but the latter would be a pretty comprising >> > solution, and I'd prefer not doing that. >> > >> > What do folks think? Would love to hear your thoughts, especially about >> > the >> > receiver. >> > >> > Thanks! >> > Mark > > --------------------------------------------------------------------- To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
