We are having issues with producers and consumers frequently fully disconnecting (from both the brokers and ZK) and reconnecting without any apparent cause. On our production systems it can happen anywhere from every 10-15 seconds to 15-20 minutes. On our less beefy test systems and developer laptops, it can happen almost constantly.
We see no errors in the logs (sample attached), just a message for each of our our consumers and producers disconnecting, then reconnecting. On the systems where it happens constantly, the consumers are not making any progress. The logs on the brokers are equally unhelpful, they show only frequent connects and reconnects, without any apparent cause. What could be causing this behavior?