mcvsubbu commented on a change in pull request #4979: Update quota manager to
reduce zk access
URL: https://github.com/apache/incubator-pinot/pull/4979#discussion_r368330092
##########
File path:
pinot-broker/src/main/java/org/apache/pinot/broker/queryquota/HelixExternalViewBasedQueryQuotaManager.java
##########
@@ -323,11 +321,43 @@ public void processQueryQuotaChange() {
}
int onlineBrokerCount = otherOnlineBrokerCount + 1;
- double overallRate =
Double.parseDouble(quotaConfig.getMaxQueriesPerSecond());
+ // Get stat from property store
+ String tableConfigPath = constructTableConfigPath(tableNameWithType);
+ Stat stat = _propertyStore.getStat(tableConfigPath,
AccessOption.PERSISTENT);
+ if (stat == null) {
+ LOGGER.info("Table {} gets deleted from property store. Removing its
rate limit.", tableNameWithType);
+ it.remove();
+ continue;
+ }
+
+ // If number of online brokers and table config don't change, there is
no need to re-calculate the dynamic rate.
+ if (onlineBrokerCount == queryQuotaEntity.getNumOnlineBrokers() &&
stat.getVersion() == queryQuotaEntity
+ .getTableConfigStatVersion()) {
+ continue;
+ }
+
+ double overallRate;
+ // Get latest quota config only if stat don't match.
+ if (stat.getVersion() != queryQuotaEntity.getTableConfigStatVersion()) {
+ QuotaConfig quotaConfig =
getQuotaConfigFromPropertyStore(tableNameWithType);
+ if (quotaConfig == null || quotaConfig.getMaxQueriesPerSecond() ==
null || !quotaConfig
+ .isMaxQueriesPerSecondValid()) {
+ LOGGER.info("No query quota config or the config is invalid for
Table {}. Removing its rate limit.",
+ tableNameWithType);
+ it.remove();
+ continue;
+ }
+ overallRate = Double.parseDouble(quotaConfig.getMaxQueriesPerSecond());
+ } else {
+ overallRate = queryQuotaEntity.getOverallRate();
+ }
double latestRate = overallRate / onlineBrokerCount;
- double previousRate = queryQuotaConfig.getRateLimiter().getRate();
+ double previousRate = queryQuotaEntity.getRateLimiter().getRate();
if (Math.abs(latestRate - previousRate) > 0.001) {
- queryQuotaConfig.getRateLimiter().setRate(latestRate);
+ queryQuotaEntity.getRateLimiter().setRate(latestRate);
+ queryQuotaEntity.setNumOnlineBrokers(onlineBrokerCount);
+ queryQuotaEntity.setOverallRate(overallRate);
+ queryQuotaEntity.setTableConfigStatVersion(stat.getVersion());
Review comment:
Even if we don't update the rate due to small difference, should we not
update the tableConfigStatVersion?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
