juan-pablo-guereca opened a new pull request, #77232: URL: https://github.com/apache/airflow/pull/77232
A secret that contains spaces or special characters was left unmasked when a formatter printed it differently from how it was registered, for example when the repr of a Kubernetes model is logged through `pprint`. `pprint` splits a string that does not fit the line width at its spaces into adjacent literals (`'abc '` / `'def'`), and `repr` and `pprint` print backslashes, mixed quote kinds and non-printable characters as escape sequences, so the registered text never appeared in the output. This extends the multi-line handling from #75651 to those cases. Two commits: 1. **Spaces:** `add_mask` registers one extra pattern per secret with spaces, built from its words joined by a separator that also matches the quote, line break and quote `pprint` puts between pieces, and the escaped forms of whitespace (`\t`, `\r`, `\xNN`, `\uNNNN`). Masking each word separately was rejected: it redacts every long word globally and leaks a short word left alone on its own literal. 2. **Escapes:** `_adaptations` also yields the secret as `repr` prints it, and the words of the wrapped pattern are matched as `repr` prints them. The replacer alternation is ordered by `_get_pattern_length`, which counts the separator as the single escaped space it stands for, so a wrapped pattern does not sort ahead of the pattern of a longer secret that contains it. ### Existing tests changed * `test_calling_mask_secret_not_set`: `"a secret"` now registers its wrapped pattern as well. * `test_secret_with_trailing_newline_registers_stripped_value`: the `repr` form of `token-value\n` masks the escaped newline too, so the output is `'***'` instead of `'***\n'`. The value is still fully masked. ### Known limits Each is pinned by a strict `xfail` test and listed in `mask-sensitive-values.rst`: * A secret with leading or trailing whitespace that `pprint` splits. * A secret with both quote kinds in different words when `pprint` splits it, because `pprint` escapes quotes per piece. * The lines of a multi-line secret that contain special characters or are long enough to be split further. * A longer secret that continues a shorter one only modulo the kind of whitespace can lose to the wrapped pattern of the shorter one. ### Performance `shared/secrets_masker` runs for every log line, so I measured the base commit and this change alternately, 5 runs each, same environment, medians. The control (`json.dumps`, untouched by the change) moved 0%. | Case | Base | This PR | |---|---|---| | Register 300 tokens without spaces | 2.5 ms | 4.5 ms | | Register 300 phrases of 5 words | 5.1 ms | 57.9 ms | | Redact 5,000 log lines, 100 token secrets | 113 ms | 115 ms | | Redact 5,000 log lines, 50 token and 50 phrase secrets | 112 ms | 161 ms | Secrets without spaces cost a few microseconds more to register and nothing more to redact. A phrase adds a second, longer pattern to the alternation, which is where the slower redaction comes from, and a registration recompiles the combined regex once per secret. I think that is acceptable for correctness on values that were leaking, but I am happy to look at a cheaper structure if reviewers prefer. ### Testing * New tests cover the whitespace separators (every character `str.isspace()` accepts, through real `pprint`), `repr` escapes, wrapped secrets with backslashes, non-printable characters and mixed quote kinds, and the ordering fix; each fails on the previous source. * Also checked manually in an Airflow 3 dev environment with a patched image. --- ##### Was generative AI tooling used to co-author this PR? - [X] Yes (Claude Code, Sonnet 5.5) Generated-by: Claude Code (Sonnet 5.5) following [the guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions) --- * Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#pull-request-guidelines)** for more information. Note: commit author/co-author name and email in commits become permanently public when merged. * For fundamental code changes, an Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals)) is needed. * When adding dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x). * For significant user-facing changes create newsfragment: `{pr_number}.significant.rst`, in [airflow-core/newsfragments](https://github.com/apache/airflow/tree/main/airflow-core/newsfragments). You can add this file in a follow-up commit after the PR is created so you know the PR number. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
