Hi Colin, Thanks a lot for your feedback. Please note that I only propose to log when a message is lost this for a set of consumer groups, not as default behaviour for all consumer groups. But in fact, I agree with you that to log a line per message expired can be quite lot, and that is not the better way do it. I can propose to add a dedicated JMX metric of type counter "expired messages" per consumer group. What do you think ?
About monitoring the lag to ensure that messages are not lost, I know that is what clients do, to set up alerting when the lag is above a threshold. But even if the alert is triggered, we dont know if messages have been lost or not. Implementing this KIP clients would know if something has been missed or not. Thanks, Jose On Mon, Jul 22, 2019 at 5:51 PM Colin McCabe <cmcc...@apache.org> wrote: > Hi Jose, > > One issue that I see here is that the number of log messages could be > huge. I've seen people create tens of thousands of consumer groups. > People can also have settings that create pretty small log files. A > message per log file per group could be quite a lot of messages. > > A log message on the broker is also not that useful for detecting bad > client behavior. People generally only look at the server logs after they > become aware that something is wrong through some other means. > > Perhaps the clients should just monitor their lag? There is a JMX metric > for this, which means it can be hooked into traditional metrics / reporting > systems. > > best, > Colin > > > On Mon, Jul 22, 2019, at 03:12, Jose M wrote: > > Hello, > > > > I didn't get any feedback on this small KIP-490 > > < > https://cwiki.apache.org/confluence/display/KAFKA/KIP-490%3A+log+when+consumer+groups+lose+a+message+because+offset+has+been+deleted > >. > > In summary, I propose a way to be noticed when messages are being > > removed > > due to retention policy, without being consumed by a given consumer > > group. > > It will be useful to realize that some important messages have been > > lost. > > > > As Im new to the codebase, I have technical questions about how to > achieve > > this, but before going deeper, I would like your feedback on the feature. > > > > Thanks a lot, > > > > > > Jose Morales > > > > On Sun, Jul 14, 2019 at 12:51 AM Jose M <yoz...@gmail.com> wrote: > > > > > Hello, > > > > > > I would like to know what do you think on KIP-490: > > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-490%3A+log+when+consumer+groups+lose+a+message+because+offset+has+been+deleted > > > < > https://cwiki.apache.org/confluence/display/KAFKA/KIP-490%3A+log+when+consumer+groups+lose+a+message+because+offset+has+expired > > > > > > > > > > > Thanks a lot ! > > > -- > > > Jose M > > > > > > > > > -- > > J > > > -- J
