Tarun-kishore opened a new pull request, #20118: URL: https://github.com/apache/pinot/pull/20118
Adds an optional per-partition primary key cap for upsert tables, maxPrimaryKeysPerPartition on QuotaConfig, with its own enforcement mode, primaryKeyQuotaEnforcementMode (DISABLED, SHADOW, ENFORCED). Unset means no cap, and both fields are omitted from the serialized form when null, so existing configs are unchanged. The cap is per partition because each partition's primary key map lives wholly on one server per replica, so the partition is the unit that drives upsert heap. A breach is when any partition's count, taken as the max across the replicas that host it (not the sum), exceeds the cap. Counts come from the existing server table metadata endpoint via ServerSegmentMetadataReader. PrimaryKeyQuotaChecker evaluates the cap and emits a new controller PRIMARY_KEY_QUOTA_EXCEEDED meter on breach in both SHADOW and ENFORCED. It returns UNDETERMINED when counts cannot be read, so a paused table is never resumed on a server outage. controller.enable.primary.key.quota.check=false is a controller-wide off switch. In ENFORCED mode RealtimeSegmentValidationManager pauses consumption with the new PauseState reason PRIMARY_KEY_QUOTA_EXCEEDED. Storage and primary key quota share the single IdealState pause flag, so the pause logic is now reason-aware: a table resumes only when every quota is within limits, never because the one reason it was paused for cleared. When both are exceeded the stored reason is storage. When the active reason changes the pause state is relabeled without a second force-commit. The cap is only valid on upsert-enabled realtime tables (validated in TableConfigUtils), and ensureStorageQuotaConstraints now preserves all quota fields. Primary key counts do not drop while paused, so a primary key pause lasts until keys are evicted or the cap is raised. Roll out with SHADOW first and upgrade all controllers before using ENFORCED, since an older controller cannot parse the new pause reason. Closes #20117 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
