You can configure Kafka with ACLs that only allow certain users to produce/consume to certain topics but if multiple producers are allowed to produce to a shared topic then you cannot identify them without adding something to the messages.
For example, you can have each producer digitally sign (or encrypt) each message and include the signature as a separate field (ie. separate from the original message body). Then the consumers can independently check that the signature is valid and that he message comes from a known/valid publisher. This pattern is similar to how signed email messages work. -hans /** * Hans Jespersen, Principal Systems Engineer, Confluent Inc. * h...@confluent.io (650)924-2670 */ On Mon, Mar 20, 2017 at 3:54 PM, Matt Magoffin <apache....@msqr.us> wrote: > Hello, > > I am new to Kafka and am looking for a way for consumers to be able to > identify the producer of each message in a topic. There are a large number > of producers (lets say on the order of millions), and each producer would > be connecting via SSL and using a unique client certificate. Essentially > I'd like consumers to know the certificate of the producer of any given > message. > > The producer identify of the message must not be forgeable, i.e. producer > A must not be allowed to generate records that appear to consumers to be > from producer B. > > Is it possible for Kafka to be configured to perform this type of identity > validation? For example: > > * reject records that contain a certificate identity that differs from the > producer connection’s client certificate > * inject the producer connection’s certificate identity into each record > > Or would a proxy application need to sit in front of Kafka to perform one > of these functions? > > Thank you in advance for offering any advice, > Matt > >
