jackjlli commented on a change in pull request #4979: Update quota manager to
reduce zk access
URL: https://github.com/apache/incubator-pinot/pull/4979#discussion_r368773524
##########
File path:
pinot-broker/src/main/java/org/apache/pinot/broker/queryquota/HelixExternalViewBasedQueryQuotaManager.java
##########
@@ -323,11 +321,43 @@ public void processQueryQuotaChange() {
}
int onlineBrokerCount = otherOnlineBrokerCount + 1;
- double overallRate =
Double.parseDouble(quotaConfig.getMaxQueriesPerSecond());
+ // Get stat from property store
+ String tableConfigPath = constructTableConfigPath(tableNameWithType);
+ Stat stat = _propertyStore.getStat(tableConfigPath,
AccessOption.PERSISTENT);
+ if (stat == null) {
+ LOGGER.info("Table {} gets deleted from property store. Removing its
rate limit.", tableNameWithType);
+ it.remove();
+ continue;
+ }
+
+ // If number of online brokers and table config don't change, there is
no need to re-calculate the dynamic rate.
+ if (onlineBrokerCount == queryQuotaEntity.getNumOnlineBrokers() &&
stat.getVersion() == queryQuotaEntity
+ .getTableConfigStatVersion()) {
+ continue;
+ }
+
+ double overallRate;
+ // Get latest quota config only if stat don't match.
+ if (stat.getVersion() != queryQuotaEntity.getTableConfigStatVersion()) {
+ QuotaConfig quotaConfig =
getQuotaConfigFromPropertyStore(tableNameWithType);
+ if (quotaConfig == null || quotaConfig.getMaxQueriesPerSecond() ==
null || !quotaConfig
+ .isMaxQueriesPerSecondValid()) {
+ LOGGER.info("No query quota config or the config is invalid for
Table {}. Removing its rate limit.",
+ tableNameWithType);
+ it.remove();
+ continue;
+ }
+ overallRate = Double.parseDouble(quotaConfig.getMaxQueriesPerSecond());
+ } else {
+ overallRate = queryQuotaEntity.getOverallRate();
+ }
double latestRate = overallRate / onlineBrokerCount;
- double previousRate = queryQuotaConfig.getRateLimiter().getRate();
+ double previousRate = queryQuotaEntity.getRateLimiter().getRate();
if (Math.abs(latestRate - previousRate) > 0.001) {
- queryQuotaConfig.getRateLimiter().setRate(latestRate);
+ queryQuotaEntity.getRateLimiter().setRate(latestRate);
+ queryQuotaEntity.setNumOnlineBrokers(onlineBrokerCount);
+ queryQuotaEntity.setOverallRate(overallRate);
+ queryQuotaEntity.setTableConfigStatVersion(stat.getVersion());
Review comment:
Table config change won't trigger qps limiter change. Only when there's a
**broker resource change** should this method be triggered, i.e. when broker
resource change happens triggered by either broker online/offline or new table
added, this method will check the stat and update the stat version. There is no
need to keep the stat version update all the time, because there's no listener
on table configs, nor it isn't a good idea to have a table config listener.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
