somandal opened a new pull request, #14894: URL: https://github.com/apache/pinot/pull/14894
Today Pinot has various mechanisms to control the segment operations that can be occur in parallel via various code paths such as STATE_TRANSITIONs, segment upload APIs and reload APIs. Multiple steps occur while adding/refreshing/reloading segments such as download, untar, index rebuild, etc. Today some knobs exist to throttle certain segment operations such as download and untar. No such throttling exists for segment rebuild operations (reload) when triggered via state transitions (a throttle does exist for handling segment refresh / reload operations coming in as USER_DEFINE_MSGs). This PR adds a throttling mechanism specifically for throttling index rebuilds. The configuration will exist in CLUSTER configs and will be modifiable without Pinot server restart by adding a config change listener. This PR adds the following: - A ClusterConfigChangeHandler for Pinot Servers which can is registered with Helix to obtain CLUSTER config changes - Adds a CLUSTER configuration to control the max preprocess parallelism allowed - Adds a SegmentPreprocessThrottler class to control how many index rebuild operations can happen in parallel. This is plugged into the ImmutableSegmentLoader on code paths where index rebuild throttling is desirable cc @Jackie-Jiang @klsince @npawar @KKcorps @yashmayya -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
