andygrove commented on PR #2657: URL: https://github.com/apache/datafusion-ballista/pull/2657#issuecomment-6452341100
**`cb49f30f`: all 22 TPC-H queries pass at SF1000, 44% less total time than blocking, and the accept queues no longer overflow** I reran the full suite at `cb49f30f` (includes 5d4aa6c3, one connection per executor pair). Same setup as before: 32 executors × 8 tasks, SF1000 Parquet on S3, push-staged, `ballista.execution.mode = pipelined`, 2 iterations per query. The runner reported `commit=cb49f30f…`. All 22 queries completed, where `d0572fc7` failed 8 and the first build failed 7, all with `ConnectionReset`. ## Time Mean of 2 iterations, in seconds, against blocking mode on the same build and cluster shape from earlier today: | Query | blocking | pipelined | | Query | blocking | pipelined | |---|---|---|---|---|---|---| | q1 | 15.7 | 13.8 | | q12 | 10.8 | 4.4 | | q2 | 29.7 | 13.5 | | q13 | 10.9 | 5.8 | | q3 | 19.9 | 7.3 | | q14 | 6.1 | 1.4 | | q4 | 12.6 | 6.1 | | q15 | 11.2 | 6.9 | | q5 | 38.5 | 13.9 | | q16 | 13.4 | 5.1 | | q6 | 1.7 | 1.4 | | q17 | 23.6 | 15.4 | | q7 | 36.7 | 18.7 | | q18 | 57.1 | 35.4 | | q8 | 21.7 | 14.1 | | q19 | 10.6 | 7.4 | | q9 | 34.0 | 24.2 | | q20 | 15.3 | 6.3 | | q10 | 37.4 | 22.0 | | q21 | 66.1 | 45.9 | | q11 | 17.0 | 4.4 | | q22 | 10.8 | 5.4 | The total is 278.8 s against 500.8 s. Every query is faster, from 12% (q1) to 76% (q14). ## Connections I read each executor's kernel TCP counters (`/proc/net/netstat`, `/proc/net/snmp`) before the first query and again after the last, so these are deltas over the run, summed over the 32 executors: | counter | `d0572fc7` (q7 alone) | `cb49f30f` (full suite) | |---|---|---| | `ListenOverflows` / `ListenDrops` | 37,779 | 0 (0 on every executor) | | `TCPSynRetrans` | 36,833 | 0 | | `Tcp.PassiveOpens` | 56,230 | 3,590 | | `Tcp.OutRsts` | 21,562 | 278 | The earlier builds overflowed the executors' accept queues by tens of thousands in every run, passing ones included. With the reworked connection sharing there are about 16 times fewer accepted connections and no overflows. I can't tell which of the three fixes (backlog, retry, shared connections) matters most in this run, and I don't need to. ## Not checked - I haven't compared results with blocking mode. The row counts look right for most queries (q1 4, q3 10, q5 5, q9 175, q18 100, q22 7), but q11 returns 0 rows and q13 returns 30. I don't know whether blocking mode returns the same, since I only kept its timings. I'll compare and report. - This is one run of the suite with 2 iterations, and the blocking numbers come from a different run on the same day. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
