Hi Frederik - thanks for the KIP, this will be a fantastic and elegant addition to Kafka Streams.
I have a higher level question about this, which is that the `poll` interface returns multiple records and yet the DeserializationException will be thrown if any record in the batch cannot be processed. I suspect that makes your example of a DLQ challenging, since it would skip all records up until the record that could not be deserialized (even if they were valid). People would have to be careful of polling up until the offset of the "bad" record... I don't have a great suggestion for an API that could address this, here are a few suggestions that come to mind: 1. add an optional callback to `poll` that could specify behavior for records that fail to deserialize (that way the callback could specify a return value of "fail" or "ignore" individual failing records within the batch) 2. have a version of consumer#poll that returns a version of ConsumerRecords that has two lists: all successfully polled records and all failed records (the obvious downside is that devs might forget to check the failed records) 3. have the serialization exception contain all successful and all failed records (that's just not very elegant). Anyway, there are many people much smarter than I watching this thread -- they may have better suggestions! (Or I may have misunderstood anything, in which case please carry on...) Cheers, Almog On Fri, Apr 12, 2024 at 1:26 PM Sophie Blee-Goldman <sop...@responsive.dev> wrote: > I think the bigger question here is: why is checkstyle complaining about > this import? Does anyone know? > > On Thu, Apr 11, 2024 at 11:12 AM Frédérik Rouleau > <froul...@confluent.io.invalid> wrote: > > > Hi everyone, > > > > I have made some changes to take in account comments. I have replaced the > > ConsumerRecord by Record. As it was not allowed by checkstyle, I have > > modified its configuration. I hope that's ok. > > I find this new version better. Thanks for your help. > > > > Regards, > > Fred > > >
