I think at some point we should think about using Jepsen for safety-testing of Pinot. While the tool has been predominantly used in databases that have distributed transactions, extensive node communication, algorithms relying on clock-skew, it can still be used to introduced faults in the cluster and check for existence of problems.
I believe Helix and ZK are the perfect candidates (but that work is going to be orthogonal to Pinot). However, since we use these extensively, it might still be worthwhile to see if Jepsen framework can expose some problems in Pinot. https://jepsen.io/ Thanks, Sidd
