Hi, Two ideas that I would like to get feedback on before putting KIPs together.
1) Ability to have the kafka client consumer “skip” data that can’t be de-serialized. it would be a consumer config such as “ignore.deserialization.errors” (got better naming?) that defaults to false to it’s backwards compatible, but if set to true, would produce a warning on the consumer client log but wouldn’t stop the processing - no errors thrown. The message would just be discarded. The use case is for example when reading an avro topic but someone pushes a message that’s not avro, currently consumers would break. 2) Ability to delete messages on topic. I believe log compaction already has a mechanism to do that so we would leverage that code. The idea would be to have an API to delete a message or a range of message based on topic / partition / offset. It would come with a command line tool. This would allow to delete messages from a topic so that if some bad data is pushed, it doesn’t break downstream consumers. Additionally, I may be able to write 1) by myself, but I believe I won’t have the capability to write 2), so I’d look for someone to help out there Looking forward to feedback. Best regards, Stephane This email, and any attachments, is confidential and may be covered by legal professional privilege or other legal rules. If you are not the intended recipient you must not disclose or use the information contained in it. If you have received this email in error please notify us immediately by return email or by calling our main switchboard on +613 9868 2100 and delete the email.
