andygrove commented on PR #2657: URL: https://github.com/apache/datafusion-ballista/pull/2657#issuecomment-6446714298
**Follow-up: the `ConnectionReset`s are accept-queue overflows on the executors** I reran q2 at `5b57a9e8` (32 executors × 8 tasks, SF1000, pipelined, `h2`/`hyper`/`tonic` debug logging) and read the kernel TCP counters from every executor pod (`/proc/net/netstat`, `/proc/net/snmp`) right after the job failed. The pods were fresh, so the counters cover only this run's startup and the failed q2 job. | counter (summed over 32 executors) | value | |---|---| | `TcpExt.ListenOverflows` | 54,476 (nonzero on all 32) | | `TcpExt.ListenDrops` | 54,476 | | `TcpExt.TCPSynRetrans` | 49,793 | | `TcpExt.TCPAbortOnData` | 12,042 | | `Tcp.EstabResets` | 2,630 | | `Tcp.OutRsts` | 26,296 | | `Tcp.PassiveOpens` (accepted connections) | 72,887, up to 2,392 on one executor | So within about 1.6 s of the job starting, each executor was asked to accept thousands of connections and its listen queue overflowed. A dropped final ACK leaves the client believing the connection is open, and the server then answers its first data with an RST. That produces the `Kind(ConnectionReset)` in the task error, and `Connection reset by peer (os error 104)` in the executors' tonic logs. No process logs anything before the first error, because the drop happens in the kernel. The timing matches. The scheduler placed the job at 13:44:13.36, the first `h2` `IO error error=ConnectionReset` on a client connection is at 13:44:14.921, and the scheduler failed the job right after. The same happened in the earlier runs: 1.7 s and 1.6 s after the job started. (An even earlier run took 13.8 s; I haven't captured counters for that one.) I think the cause is the number of connections: each reader drains all producers concurrently, so at 256 partitions the job opens about 73k connections across 32 executors, compared with the 31 that one HTTP/2 connection per executor pair would need. The executor binds its gRPC listener with `TcpIncoming::bind`, which uses the default backlog. I haven't confirmed whether each stream gets its own connection. I'm guessing that from the `PassiveOpens` count. Things that might help, in the order I'd try them: - Share one `Channel` (one HTTP/2 connection, multiplexed) per executor pair for the Flight fetches instead of connecting per stream. - Retry a stream open when the connection is reset before any batch has been consumed. That is safe, and it would turn this failure into a retry. - Let the executor bind with a larger backlog, and document `net.core.somaxconn` for pipelined mode. This explains the 7 failing queries of 22 in my earlier comment. It would also explain why they are the ones with the most stages and the widest fan-in. I can run any of these changes against the same cluster if that helps. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
