shounakmk219 opened a new pull request, #13544:
URL: https://github.com/apache/pinot/pull/13544
# Goal
- Ability to impose query rate limits at database level. This will consider
all the queries that are made to the tables present under the database.
- Way to specify query quota at database level
# Introduce database rate limiter
Pinot decides if the query quota is reached or not with the help of
`QueryQuotaManager`.
`QueryQuotaManager` interface right now has an `acquire(tableName)` method
which decides if the quota for the provided table is reached or not by
returning a boolean.
To allow similar mechanism for database query quota a new interface method
`acquireDatabase(databaseName)` is introduced on `QueryQuotaManager` interface
# Query quota provisioning
Right now as the query and storage quotas on tables are maintained at
TableConfig itself, we follow the same approach for database quota configs as
well.
We don't have a place to provide database specific configs as it not being a
first class entity. But for this usecase and similar future usecases we need to
maintain a separate DatabaseConfig znode.
Assumptions on the new DatabaseConfig znode:
- This config does not represent the logical database entity.
- Absence of this config does not prevent users from creating tables under a
database.
- Deletion of thus config does not delete the tables under the database.
Provide default database query quota in `ClusterConfig` as below
```
"databaseMaxQueriesPerSecond" : "1000"
```
If we wish to provide an override for a particular database we can do so by
creating a znode under `PROPERTYSTORE/CONFIGS/DATABASE/<databaseName>`
```
{
"databaseName" : "db1",
"quota" : {
"maxQueriesPerSecond" : "200"
}
}
```
Exposed APIs on controller to manage this:
- `POST /databases/{databaseName}/quotas?maxQueriesPerSecond=<query quota>`
- `GET /databases/{databaseName}/quotas`
# Query quota update handling
There are 2 source based on which the database query quota is decided
1. Default query quota provided at cluster config
2. Override query quota provided at database config
On top of the overall quota provided by above configs, the per broker quota
is also affected by the number of live brokers in the cluster. Considering all
the above factors, rate limiter updates are handled as below.
- Introduced new user defined message `DatabaseConfigRefreshMessage` which
is sent to brokers upon the database config updates.
- Added `ClusterConfigChangeListener` to `ClusterChangeMediator` to handle
the changes made to database query quota in cluster configs.
- Update per broker rate quota upon broker resource EV change
- Create database rate limiter upon `OFFLINE` -> `ONLINE` transition of
tables in `BrokerResourceOnlineOfflineStateModel`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
