#general
@gabrielweich.dev: @gabrielweich.dev has joined the channel
#random
@gabrielweich.dev: @gabrielweich.dev has joined the channel
#troubleshooting
@kangren.chia: when i downscale the number of instances in my k8s helm chart for pinot components, i have instances that are not removed?
@kangren.chia: ```# healthy component { "id": "Controller_pinot-controller-0.pinot-controller-headless.pinot.svc.cluster.local_9000", "simpleFields": { "HELIX_VERSION": "0.9.8", "LIVE_INSTANCE": "1@pinot-controller-0", "SESSION_ID": "100045e48f1002d" }, "mapFields": {}, "listFields": {} } # removed component { "code": 404, "error": "ZKPath /pinot/LIVEINSTANCES/Controller_pinot-controller-1.pinot-controller-headless.pinot.svc.cluster.local_9000 does not exist:" }```
@kangren.chia: am i supposed to manually use the REST api to drop instances? for what it’s worth, i update my helm chart deployment via terraform
@dlavoie: Yes
@dlavoie: Scaling down involves segment rebalance
@kangren.chia: i wasn’t able to find this info in the docs or the pinot github issues. would you know where i might find more information?
@dlavoie: There’s a bit of orchestration involved since Pinot doesn’t know if the servers will come back or not.
@kangren.chia: hm but its not just servers, controllers and brokers also have to be manually dropped?
@dlavoie: Yes, Pinot is a stateful system, everyone has an identity and a role. the pinot doc is generic and not precise about the specifics of helm based deployment.
@dlavoie: it can certainly be improved
@kangren.chia: thanks for the answers!
@kangren.chia: i was hoping to use autoscaling for my pinot deployment
@kangren.chia: are there any resources on production deployments of pinot where i can read about how they manage this?
@dlavoie: Usually, you start with resizing your statefulsets, then trigger rebalance.
@dlavoie: scaling or scaling down requires a rebalance trigger on pinot apis.
@dlavoie: Having dead reference is not the end of the world if you plan to scale back up later.
@kangren.chia: the rebalance operations put the segments back on my dead servers
@kangren.chia: and i can’t drop the dead servers because ```{"code":409,"error":"Failed to drop instance Server_pinot-server-1.pinot-server-headless.pinot.svc.cluster.local_8098 - Instance Server_pinot-server-1.pinot-server-headless.pinot.svc.cluster.local_8098 exists in ideal state for events-10-partitions_OFFLINE"}```
@kangren.chia: am i missing something here?
@dlavoie: Sounds like cleaning dead references from the ideal state is a prerequisite to rebalance.
@kangren.chia: its not obvious to me which api i should use to clean dead references
@kangren.chia: so i saw is that IdealState is a helix concept, and i can probably update the zookeeper znode that stores the idealstate to remove the dead servers. is this the recommended way?
@kangren.chia: alright so what it took was for me to 1. update (remove) the tags of the dead servers/brokers so that they are no longer tagged to a tenant 2. rebalance the segments thanks @dlavoie!
@deemish2: Hello everyone , after setting retention time 1 day ,there is no real time segment available while i see offline segment there after setting same retention period - 1 day.
@xiangfu0: this is an offline table, you need to create a realtime table. See here for hybrid table:
@gabrielweich.dev: @gabrielweich.dev has joined the channel
@mrpringle: Anyone else find that the pinot server port is not closed when the server process exits. So upon restart the port is in use and it cannot start. Will do some more digging today, rhel 7, java 11, p 0.71
@mayanks: Haven’t experienced this myself, please let us know what you find
@mrpringle: For kafka are people using group.id. Trying to work out how the replication works e.g. say 2 replicas of a segment, is this 2 servers ingesting the same kafka messages (in which case guess setting group.id won't work) or one server ingesting from kafka and then copying the segments to another server for replication.
@g.kishore: Two servers ingesting the same Kafka messages
@g.kishore:
@mrpringle: what is managing the committed offsets, pinot or kafka? guess I should try removing kafka group ID from the table spec
@g.kishore: Pinot
@mrpringle: any easy way to check consumer lags in the controller ui?
#pinot-perf-tuning
@kangren.chia: @kangren.chia has joined the channel
--------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
