semistone commented on issue #22601: URL: https://github.com/apache/pulsar/issues/22601#issuecomment-2134203674
@lhotari Thanks to point out that config issue I tested and verified it worked. also talk about our release history. -------- actually we have notice that issue https://github.com/apache/pulsar/issues/21421 during our pre release load test and since that error logs have ledger id and entry id and it happens about 4 times during about 10M events. so we could still manually recover the data from bookkeeper manually or by auto fix tool. and we have about 10 consumers, only 1 consumer is too slow to consume data from bookkeeper so only one consumer been impacted and we could still wait bookkeeper side to fix that issue. and after upgrading bookkeeper version, it seems improved. but this error, the data in bookkeeper is correct. that why at beginning we didn't aware it's bookkeeper client issue and there is no ledger/entry id in error logs. so it's impossible for us to fix the data. and it mainly impact real time consumer so we don't have choice but rollback our release. ------- so I will use that settings in our broker setting and I will suggest to print ledger/entry id in case of that happen again (I could create PR if it's ok) then at least user could still re publish the data from bookkeeper. and I think it still need to report to bookkeeper team. seems those issues are already closed, I will create issue in bookkeeper tomorrow if there is no existed issue to trace this issue. I will try to understand bookkeeper's code if I have time :) I already traced this issue for one month. Thanks for helping us. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
