Thanks Ewen! This will help us. ________________________________________ From: Ewen Cheslack-Postava <e...@confluent.io> Sent: Thursday, December 10, 2015 1:50 AM To: users@kafka.apache.org Cc: Michael Pasacrita; Satish Sankaranarayanan Subject: Re: Match Producer and RecordMetadata with Consumer and ConsumerRecord
John, Your question was a bit confusing because CorrelationID has a specific meaning in the Kafka protocols, but those are an implementation detail that you, as a user of the API, should not need to worry about. CorrelationIDs as defined by the protocol are not exposed to the user (and do not correspond to individual Kafka messages -- they correspond to request/response pairs in the protocol). In Kafka you can uniquely identify a message by the (topic, partition, offset) tuple. To match up the record passed to a producer and the record received by a consumer, you would use, as you suggested in your first email: Producer: RecordMetadata.topic(), RecordMetadata.partition(), RecordMetadata.offset() Consumer: ConsumerRecord.topic(), ConsumerRecord.partition(), ConsumerRecord.offset() -Ewen On Wed, Dec 9, 2015 at 11:29 AM, John Menke <jme...@numerix.com> wrote: > Gwen, thanks for the reply: > > Just to confirm - are we talking about the combination of the IDs that I > layed out as being the "CorrelationID" ? > > I guess the alternative is to pass our own CorrelationID as part of the > data. Would you agree with this? > > I do see something at this address that talks about a "user defined" > CorrelationID > > > https://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+The+Kafka+Protocol#AGuideToTheKafkaProtocol-ProduceResponse > > CorrelationId: > > This is a user-supplied integer. It will be passed back in the response by > the server, unmodified. It is useful for matching request and response > between the client and server. > > > > -----Original Message----- > From: Gwen Shapira [mailto:g...@confluent.io] > Sent: Wednesday, December 09, 2015 12:50 PM > To: users@kafka.apache.org > Subject: Re: Match Producer and RecordMetadata with Consumer and > ConsumerRecord > > Correlation ID is for a request (i.e. separate ID for produce request and > a fetch request), not a record. So it can't be used in the way you are > trying to. > > On Wed, Dec 9, 2015 at 9:30 AM, John Menke <jme...@numerix.com> wrote: > > > Can a correlationID be created from a ConsumerRecord that will allow > > for identification of the corresponding RecordMetaData instance that > > was returned from the Producer.send() method? > > > > I am Looking at the JavaDocs and the Producer returns RecordMetadata > > which has the following signature: > > > > RecordMetadata(TopicPartition topicPartition, long baseOffset, long > > relativeOffset) > > > > I am not sure if this can match to > > > > ConsumerRecord(java.lang.String topic, int partition, long offset, K > > key, V value) > > > > Can we match producer requests to Consumer Records by matching these > > values? > > > > RecordMetaData.TopicPartion.topic = ConsumerRecord.topic > > RecordMetaData.TopicPartion.partition = ConsumerRecord.partition > > RecordMetaData(baseOffset + relativeOffset) = ConsumerRecord.offset > > > > In other words, can a CorrelationID be created from these values that > > will allow Consumers to link back to the Producer send() event. (In > > the client calling code) > > > > > > > > > > > > > > > -- Thanks, Ewen
