dosubot[bot] commented on issue #29099:
URL: https://github.com/apache/superset/issues/29099#issuecomment-2151440952
<!-- Answer -->
The errors you're encountering during load testing on Superset dashboards
are likely due to a combination of rate limiting and database connection pool
limits. Here are some steps to address these issues:
1. **Increase Database Connection Pool Size**:
Adjust the database connection pool settings to allow more concurrent
connections. You can do this by modifying the `SQLALCHEMY_DATABASE_URI` and
`SQLALCHEMY_ENGINE_OPTIONS` in your `superset_config.py` file:
```python
# superset_config.py
SQLALCHEMY_DATABASE_URI = (
'postgresql+psycopg2://user:password@localhost/mydatabase'
'?pool_size=20&max_overflow=10&pool_timeout=30&pool_recycle=3600'
)
SQLALCHEMY_ENGINE_OPTIONS = {
'pool_size': 20,
'max_overflow': 10,
'pool_timeout': 30,
'pool_recycle': 3600,
}
```
2. **Customize Rate Limiting Settings**:
Increase the rate limits to handle higher loads by modifying the rate
limiting settings in your `superset_config.py` file:
```python
RATELIMIT_APPLICATION = "100 per second"
AUTH_RATE_LIMIT = "10 per second"
RATELIMIT_STORAGE_URI = "redis://localhost:6379"
```
3. **Implement Caching**:
Use caching mechanisms to reduce the number of direct database queries.
Configure caching in your `superset_config.py` file:
```python
CACHE_CONFIG = {
'CACHE_TYPE': 'redis',
'CACHE_DEFAULT_TIMEOUT': 300,
'CACHE_KEY_PREFIX': 'superset_',
'CACHE_REDIS_HOST': 'localhost',
'CACHE_REDIS_PORT': 6379,
'CACHE_REDIS_DB': 1,
'CACHE_REDIS_URL': 'redis://localhost:6379/1'
}
DATA_CACHE_CONFIG = CACHE_CONFIG
```
4. **Rate Limiting in Load Testing Scripts**:
Implement rate limiting in your Selenium and pytest scripts to avoid
overwhelming the server with too many concurrent requests.
By following these steps, you should be able to mitigate the errors related
to rate limiting and database connection pool limits during your load testing
<sup>[[1]](https://github.com/apache/superset/blob/master/superset/errors.py)</sup><sup>[[2]](https://github.com/apache/superset/blob/master/superset/config.py)</sup><sup>[[3]](https://github.com/apache/superset/blob/master/docs/docs/configuration/configuring-superset.mdx)</sup>.
<!-- Dosu Comment Footer -->
*To continue talking to [Dosu](https://dosu.dev), mention @dosu.*
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscr...@superset.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscr...@superset.apache.org
For additional commands, e-mail: notifications-h...@superset.apache.org
