#troubleshooting
@diogo.baeder: Hi guys, I'm having an issue running Pinot 0.10 locally where, sometimes after I stop my local docker-compose services and start them again, my segments go missing and marked as in "bad" status. It doesn't happen all the time, like now I tested and after the 3rd restart of the services I managed to have them in a "bad" state. I compared directories before and after and couldn't find out what's happening that caused them to "spoil". Any clues why this might be happening? More details in the thread here.
@diogo.baeder: Log file for the docker-compose execution:
@diogo.baeder: My docker-compose file:
@diogo.baeder: The segments are being saved to `/tmp/data/segments`, which is mounted in the containers so that I don't lose the data between restarts.
@mayanks: Are you tearing down the entire cluster, including ZK? If so you will lose data
@diogo.baeder: Well, it's just a local experiment really, not a production cluster. I'm aware that in production I shouldn't have it running like that, of course.
@diogo.baeder: For development, however, it's annoying to keep losing my test data, I'd like to make sure it will always be there after I restart the containers.
@mayanks: If you’d like durablity then do a rolling restart or ensure ZK and deep store are not completely shutdown
@diogo.baeder: What's the order of when the components can go down? I cannot afford keeping ZK running forever (since it's a test cluster running in my personal computer), but "deep store" is easy to ensure since I just mount a directory as a volume there to keep it between restarts. Currently I have: • ZK starting first • Controller depending on ZK (so starting second) • Broker and Server depending on Controller (so starting third)
@diogo.baeder: This is the order in which docker-compose is stopping them now: ```^CGracefully stopping... (press Ctrl+C again to force) [+] Running 5/5 ⠿ Container pinot-broker-xavier Stopped 10.4s ⠿ Container trino-server-xavier Stopped 0.4s ⠿ Container pinot-server-xavier Stopped 0.6s ⠿ Container pinot-controller-xavier Stopped 1.0s ⠿ Container zookeeper-xavier Stopped 0.6s``` does it make sense?
@mayanks: Shutdown order server broker controller zk
@mayanks: You can snapshot ZK and bring it back up from snapshot
@diogo.baeder: Currently I'm mounting a volume for ZK, instead of having to manage snapshots or anything like that. Locally it's easier to handle.
@diogo.baeder: Thanks for the component order, I'll change that here!
@mayanks: If zk state isn’t snapshoted and everything is torn apart, it will lead to data loss. Not just for Pinot, any system out there.
@diogo.baeder: I was taking a look at the ZK image docs, and it seems that the snapshots are saved to `/data`, and I have both `/data` and `/datalog` mounted as volumes, so I believe I should be safe.
--------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@pinot.apache.org For additional commands, e-mail: dev-h...@pinot.apache.org
