korex-f opened a new pull request, #71669:
URL: https://github.com/apache/airflow/pull/71669
<!-- SPDX-License-Identifier: Apache-2.0
https://www.apache.org/licenses/LICENSE-2.0 -->
<!--
-->
`make_buffered_socket_reader` re-scanned the entire accumulated buffer from
offset 0 on every socket read, making newline-scanning O(N²) in the size of a
single record. A 5 MiB single-line trigger log record (as reported in #66158)
could cost gigabytes of cumulative prefix scanning before the terminating
newline was found, delaying drainage of all supervised sockets under triggerer
load.
This PR:
* Adds a persistent scan cursor (`search_from`) so each byte in the buffer
is examined once per delivery, not once per chunk. A 20 MiB single-line record
that previously took ~2.7s to scan now completes in well under 1s.
* Sends each completed record to the generator as an explicit immutable
`bytes` copy rather than a `bytearray` slice, so the generator never holds a
view that could alias the buffer across a later resize.
* Changes EOF handling: a non-newline-terminated fragment left in the buffer
when the socket closes is no longer forwarded to the JSON decoder as if it were
a complete record. It is dropped and logged once with bounded metadata
(fragment byte length only, never the fragment content, which may contain
secrets or multi-megabyte data).
No wire-protocol change. The newline-delimited framing between supervisor
and subprocess is unchanged; this is scoped to the supervisor-side reader only.
`make_buffered_socket_reader`'s return shape, generator contract, `data=`
pre-seeding behavior, default buffer size, and selector semantics are all
preserved.
A separate GCS-handler symptom mentioned later in #66158 (missing files even
for ordinary-sized messages) is not addressed here and looks like a different
issue, flagging in case a maintainer wants to split it out.
Note: `test_exit_by_signal`, `test_process_log_messages_from_subprocess`,
`test_reinit_supervisor_comms` (missing `airflow_shared` module) and
`test_remote_logging_conn_caches_connection_not_client` (missing `amazon`
provider) fail identically for me on unmodified `main` in my local dev
environment, pre-existing environment gaps, unrelated to this change.
closes: #66158
---
-->
---
##### Was generative AI tooling used to co-author this PR?
<!--
If generative AI tooling has been used in the process of authoring this PR,
please
change below checkbox to `[X]` followed by the name of the tool, uncomment
the "Generated-by".
-->
- [ ] Yes (please specify the tool below)
<!--
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
