NoahKusaba commented on PR #2579: URL: https://github.com/apache/datafusion-ballista/pull/2579#issuecomment-6328948009
Note on the "Retrigger CI" commit (d273b17): the failure on 47fc2f9 was a flaky chaos test, not this change. [`test linux crates`](https://github.com/apache/datafusion-ballista/actions/runs/38662611398/job/116187325468) failed in `ballista-chaos --test ha`, on `baseline_matches_local_datafusion::case_2_aqe_on`, with `cluster must start: timed out waiting for 2 executors to register`. The other 15 `ha` tests passed, and so did the rest of the workflow. The executor logs show both executors got the same Flight port (`127.0.0.1:45565`). Executor 0 then failed with `AddrInUse` and shut down, so only one executor was left to register: ``` executor-0: Built-in arrow flight server listening on: 127.0.0.1:45565 executor-0: Could not start built-in arrow flight server. executor-0: executor services stopped with reason Err(TonicError(... AddrInUse ...)) executor-1: Built-in arrow flight server listening on: 127.0.0.1:45565 ``` This is a race in the test harness. `spawn_executor` calls `free_ports()` once per executor (`chaos-testing/src/cluster.rs`). That function binds to `:0` to find a free port, then releases it before the child process binds. Within one call the ports are distinct, but nothing stops a later call from getting the same number. Executor 0 starts its Flight server only after gRPC setup and registration, so executor 1's `free_ports()` got 45565 back from the kernel before executor 0 had bound it. This PR doesn't touch the chaos harness or executor startup, and the same suite passed on the rerun. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
