On Wed, Mar 8, 2017, at 17:28, Maarek, Stephane wrote: > Hi, > > Two ideas that I would like to get feedback on before putting KIPs > together. > > 1) Ability to have the kafka client consumer “skip” data that can’t be > de-serialized. it would be a consumer config such as > “ignore.deserialization.errors” (got better naming?) that defaults to > false to it’s backwards compatible, but if set to true, would produce a > warning on the consumer client log but wouldn’t stop the processing - no > errors thrown. The message would just be discarded. The use case is for > example when reading an avro topic but someone pushes a message that’s > not avro, currently consumers would break.
Interesting idea. How about a Deserializer implementation that wraps other Deserializer implementations in a try...catch block? best, Colin > > 2) Ability to delete messages on topic. I believe log compaction already > has a mechanism to do that so we would leverage that code. The idea would > be to have an API to delete a message or a range of message based on > topic / partition / offset. It would come with a command line tool. This > would allow to delete messages from a topic so that if some bad data is > pushed, it doesn’t break downstream consumers. > > > Additionally, I may be able to write 1) by myself, but I believe I won’t > have the capability to write 2), so I’d look for someone to help out > there > > Looking forward to feedback. > > > Best regards, > Stephane > > This email, and any attachments, is confidential and may be covered by > legal professional privilege or other legal rules. If you are not the > intended recipient you must not disclose or use the information contained > in it. If you have received this email in error please notify us > immediately by return email or by calling our main switchboard on +613 > 9868 2100 and delete the email.
