Re: validate identity of producer in each record

2017-03-20 Thread Matt Magoffin
OK, thank you for that. I looked at org.apache.kafka.connect.transforms.Transformation and org.apache.kafka.connect.source.SourceRecord, but am not discovering where the authenticated username (or Principal) might be available for the call to Transformation.apply()… or does Connect even support

Re: validate identity of producer in each record

2017-03-20 Thread Hans Jespersen
Nothing on the broker today but if you use Kafka Connect API in 0.10.2 and above there is a pluggable interface called Transformations. See org.apache.kafka.connect.transforms in https://kafka.apache.org/0102/javadoc/index.html?org/apache/kafka/connect Source Connector transformations happen b

Re: validate identity of producer in each record

2017-03-20 Thread Matt Magoffin
Thanks, Hans. Signing messages is a good idea. Other than that, is there possibly an extension point in Kafka itself on the receiving of records, before they are stored/distributed? I was thinking along the lines of org.apache.kafka.clients.producer.ProducerInterceptor but on the server side?

Re: validate identity of producer in each record

2017-03-20 Thread Matt Magoffin
Thanks, Hans. Signing messages is a good idea. Other than that, is there possibly an extension point in Kafka itself on the receiving of records, before they are stored/distributed? I was thinking along the lines of org.apache.kafka.clients.producer.ProducerInterceptor but on the server side?

Re: validate identity of producer in each record

2017-03-20 Thread Hans Jespersen
You can configure Kafka with ACLs that only allow certain users to produce/consume to certain topics but if multiple producers are allowed to produce to a shared topic then you cannot identify them without adding something to the messages. For example, you can have each producer digitally sign (or

validate identity of producer in each record

2017-03-20 Thread Matt Magoffin
Hello, I am new to Kafka and am looking for a way for consumers to be able to identify the producer of each message in a topic. There are a large number of producers (lets say on the order of millions), and each producer would be connecting via SSL and using a unique client certificate. Essenti