andygrove commented on PR #2657: URL: https://github.com/apache/datafusion-ballista/pull/2657#issuecomment-6449615565
**Follow-up: `d0572fc7` still fails 8 of 22 TPC-H queries with `ConnectionReset`, and the accept-queue counters don't separate the passing runs from the failing ones** I rebuilt at `d0572fc7` (includes e573d270, one shared connection per producer executor) and ran the whole TPC-H SF1000 suite (32 executors × 8 tasks, pipelined, 2 iterations). The runner confirmed `commit=d0572fc7…`. - **Passed (14):** q1, q3, q4, q6, q9, q10, q12, q13, q14, q15, q17, q18, q19, q22. Their sum is 167.3 s against 262.3 s for the same queries in blocking mode. - **Failed (8):** q2, q5, q7, q8, q11, q16, q20, q21, all with `Kind(ConnectionReset)` while streaming a pipelined partition, as before. q7 passed on `34a1151a` and fails now; q2 failed in this run although it passed when I ran it alone on the same image (14.6 s). So the shared connection did not fix the failures, and q2's single passing run was luck. I also need to correct my previous comment, where I said accept-queue overflow explains the resets. The kernel counters I read from the 32 executors after a failing q2 on `5b57a9e8` and after the passing q2 on `d0572fc7` are about the same: | counter (summed over 32 executors) | failing q2 (`5b57a9e8`) | passing q2 (`d0572fc7`) | |---|---|---| | `ListenOverflows` / `ListenDrops` | 54,476 | 61,480 | | `TCPSynRetrans` | 49,793 | 52,613 | | `Tcp.PassiveOpens` | 72,887 | 72,764 | | `Tcp.EstabResets` | 2,630 | 2,529 | The queue overflowed in the run that passed too, and the number of accepted connections didn't fall, so I can't say overflow is the cause, or that most of the 73k connections are the reader's streams. These counters cover each pod's whole life, so they include startup and other traffic. I'm going to snapshot them before the job and sample them during the run, so I can line counter jumps up with the failures. What I do know is the first error in each failing run is a TCP reset seen by a reader (`os error 104`, `Kind(ConnectionReset)`), with nothing logged by either process before it. One thing that might be useful regardless: a stream open that gets reset before any batch is consumed could be retried. All 8 failures are of that kind, and pipelined jobs don't retry tasks. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
