Re: Kafka encryption

2016-05-30 Thread eugene miretsky
If you are looking to do encryprion in the serializer, you could use this as a refernce - https://github.com/rogers/change-data-capture/tree/master/kafka FS encryprion is easier, but if you cannot do that, this is a good alternative. It also offloads the encryption work off the broker. On May 26,

Re: Kakfa Connect Converter per Connector

2016-02-02 Thread eugene miretsky
Hi Eric, We have a slightly different use case where we publish to Kafka using a (modified) Connect Source and are using Spark Streaming to read the data from Kafka and write to C* - it was really easy to write simple code to parse SchemaAndValue objects. Setting up Spark Streaming is extremely

Retrieving last message offset in high level consumer

2015-10-01 Thread eugene miretsky
Hi, We would like to log the offset of a Kafka message if we fail to process it (so we can try to re-process it later). Is it possible to get the offset using the high level consumer? I took a quick look at the code, and: - It seems like the offset it private in the current Scala consumer

What is the current status of security features in Kafka?

2015-07-02 Thread eugene miretsky
HI, There is some work being done on security in Kafka: Confluence: https://cwiki.apache.org/confluence/display/KAFKA/Security Jira: https://issues.apache.org/jira/browse/KAFKA-1682 It seems like the main blockers are KAFKA-1477 , KAFKA-1691