Tarun-kishore opened a new issue, #20117: URL: https://github.com/apache/pinot/issues/20117
## Problem Pinot currently has no per-table limit on the number of primary keys held in memory for upsert tables. As the number of keys grows, the upsert metadata consumes more JVM heap and can eventually cause the server to OOM, affecting every table hosted on that server. This problem has already been raised in upstream issues: - **#18785:** Highlights that upsert primary-key maps can grow without bounds, eventually causing server OOMs and impacting other tables on the same host. - **#11658:** Raised concerns about heap pressure from upsert primary-key metadata. Pinot already has two levels of protection for disk usage: a host-level resource utilization check and a per-table storage quota. For heap, **#18784 introduces a host-level safeguard, but there is still no per-table limit** to prevent an individual upsert table from consuming excessive memory. ## Proposal Introduce `quota.maxPrimaryKeysPerPartition` for upsert tables, using Pinot's existing quota enforcement mechanisms. - Configure a maximum number of primary keys per partition. - Reuse the existing storage quota enforcement flow to pause ingestion when the limit is exceeded. - Support `SHADOW` mode to monitor quota breaches without pausing ingestion. - Use the maximum primary-key count across replicas of a partition, rather than summing replica counts, to avoid counting replicated keys multiple times. ## Why a per-partition quota? **Host-level protection alone is insufficient.** The safeguard proposed in #18784 is reactive and applies to the entire host, potentially throttling unrelated tables. It also cannot recover heap already occupied by primary-key maps, since stopping ingestion does not release those keys. A per-partition quota provides a way to identify and limit the table responsible before it affects other workloads. Primary-key count is also a reasonable proxy for memory usage. Pinot's `/upsert/estimateHeapUsage` endpoint estimates heap consumption based on the number of keys and the estimated bytes per key. For example, at approximately 100 bytes per LONG key, **100 million keys in one partition could consume around 10 GB of heap per replica**. Enforcing the limit per partition also catches uneven key distribution that a table-wide quota might miss. ## Limitations - **Recovery:** Pausing ingestion does not release existing primary-key metadata. Ingestion remains paused until the quota is increased or keys are removed. - **Enforcement delay:** The check runs hourly by default, so the primary-key count may exceed the configured limit before ingestion is paused. - **Monitoring overhead:** The existing server endpoint being reused performs more work than is strictly necessary to retrieve the primary-key count. - **Rollout:** All controllers must be upgraded before enabling `ENFORCED` mode on any table. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
