I think they are open to others helping, in fact, more than one person has worked on the JIRA so far. And, it's been crawling really slowly and that's preventing adoption of Spark's new connector in secure Kafka environments.
On Tue, Nov 8, 2016 at 7:59 PM, Cody Koeninger <c...@koeninger.org> wrote: > Have you asked the assignee on the Kafka jira whether they'd be > willing to accept help on it? > > On Tue, Nov 8, 2016 at 5:26 PM, Mark Grover <m...@apache.org> wrote: > > Hi all, > > We currently have a new direct stream connector, thanks to work by Cody > and > > others on SPARK-12177. > > > > However, that can't be used in secure clusters that require Kerberos > > authentication. That's because Kafka currently doesn't support delegation > > tokens (KAFKA-1696). Unfortunately, very little work has been done on > that > > JIRA, so, in my opinion, folks who want to use secure Kafka (using the > norm > > - Kerberos) can't do so because Spark Streaming can't consume from it > today. > > > > The right way is, of course, to get delegation tokens in Kafka but > honestly > > I don't know if that's happening in the near future. I am wondering if we > > should consider something to remedy this - for example, we could come up > > with a receiver based connector based on the new Kafka consumer API > that'd > > support kerberos authentication. It won't require delegation tokens since > > there's only a very small number of executors talking to Kafka. Of > course, > > for anyone who cares about high throughput and other direct connector > > benefits would have to use direct connector. Another thing we could do is > > ship the keytab to the executors in the direct connector, so delegation > > tokens are not required but the latter would be a pretty comprising > > solution, and I'd prefer not doing that. > > > > What do folks think? Would love to hear your thoughts, especially about > the > > receiver. > > > > Thanks! > > Mark >
