shivam-startree opened a new pull request, #19723: URL: https://github.com/apache/pinot/pull/19723
## Problem `AuditRequestProcessor.captureRequestPayload()` gates only on `capture.request.payload.enabled` and `hasEntity()`. There is no media-type check, so a `multipart/form-data` segment upload has its gzipped tarball read and passed through `new String(bytes, UTF_8)`. That decode amplifies rather than truncates. Every byte that isn't valid UTF-8 becomes `U+FFFD`, and each `U+FFFD` costs three bytes again when the record is serialised: | captured | emitted record | | |---|---|---| | 8 KiB | ~19 KB | ×2.4 | | 16 KiB | ~39 KB | ×2.4 | | 64 KiB (`MAX_AUDIT_PAYLOAD_SIZE_BYTES`) | ~155 KB | ×2.4 | So `request.payload.size.max.bytes` isn't the bound an operator expects — it bounds what is **read**, not what is **emitted**. And none of those bytes are auditable, because the record can't be read back as text: the entry ends up both larger and less informative. On a busy controller this made segment uploads the single largest contributor to audit log volume, which is how we found it — oversized audit records were saturating the node's log agent. ## Changes 1. **Only capture text-shaped bodies** — `text/*`, `application/json`, `xml`, `x-www-form-urlencoded`, `+json`, `+xml`. Anything else records its media type in place of its bytes, so the audit trail still shows a body was present and what kind. A missing `Content-Type` is treated as textual, so behaviour is unchanged for clients that don't set one. 2. **Don't emit replacement-character soup for accepted types either.** If the decoded body is mostly `U+FFFD`, record the byte count instead. The 10% threshold in (2) is the part worth a close look: a body truncated mid-character loses only the few bytes of a single character, so ordinary truncated JSON must stay classified as text and keep being audited. There's a test pinning exactly that boundary. ## Tests Added to `AuditRequestProcessorTest`: - `multipart/form-data` and `application/octet-stream` skipped - `json` / `text` / `xml` / `x-www-form-urlencoded` / `application/merge-patch+json` still captured - null media type behaves as before - a 2 KiB random body emits its size and contains no `U+FFFD` - truncated UTF-8 text is still audited (the false-positive guard) - the ratio classifier directly, including the one-bad-character case ## Compatibility `capture.request.payload.enabled` defaults to `false`, so stock deployments are unaffected. This only changes behaviour for clusters that have turned payload capture on, and there the change is strictly a reduction in junk: no request that was audited before stops being audited, only the unreadable bytes are replaced by a description of them. Suggested label: `bugfix`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
