andygrove opened a new issue, #2147: URL: https://github.com/apache/datafusion-ballista/issues/2147
**Describe the bug** When a `TPC-H SF10` job fails, the `Upload cluster logs on failure` step in `.github/workflows/tpch.yml` is supposed to attach both `scheduler.log` and `executor.log`. Only `scheduler.log` ever makes it into the artifact. That is the more valuable of the two to lose. Most SF10 failures are task-level (spill, shuffle write, OOM, disk), and those only show up in the executor log. Without it you are left with the 200 line tail the cleanup trap echoes into the job output, which is usually not enough to see what led up to the failure. **To Reproduce** Two independent failed runs, different failure modes, same result: - Run [29923718090](https://github.com/apache/datafusion-ballista/actions/runs/29923718090) (`No space left on device` during query 21), artifact `tpch-sf10-cluster-logs-aqe-on-mpt` contains only `__w/_temp/scheduler.log`. - Run [29749959848](https://github.com/apache/datafusion-ballista/actions/runs/29749959848) (executor registration failure), artifact `tpch-sf10-cluster-logs` contains only `__w/_temp/scheduler.log`. ``` $ gh run download 29923718090 -n tpch-sf10-cluster-logs-aqe-on-mpt -D art $ find art -type f art/__w/_temp/scheduler.log ``` In both cases `executor.log` demonstrably existed at upload time, because the cleanup trap in the same step successfully ran `tail -n 200 "$EXECUTOR_LOG"` and its output is visible in the job log a fraction of a second earlier. **Expected behavior** The failure artifact contains both `scheduler.log` and `executor.log`. **Additional context** The step config is: ```yaml - name: Upload cluster logs on failure if: failure() uses: actions/upload-artifact@v7 with: name: tpch-sf10-cluster-logs-${{ matrix.slug }} path: | ${{ runner.temp }}/scheduler.log ${{ runner.temp }}/executor.log if-no-files-found: ignore ``` I have not pinned down the mechanism, so treat the following as clues rather than a diagnosis. From the upload step's own output: ``` path: /home/runner/work/_temp/scheduler.log /home/runner/work/_temp/executor.log Multiple search paths detected. Calculating the least common ancestor of all paths The least common ancestor is /. This will be the root directory of the artifact With the provided path, there will be 1 file uploaded ``` Two things stand out. The action only resolved one of the two search paths, and the least common ancestor came out as `/` rather than the shared `/home/runner/work/_temp` directory, which suggests the two paths did not resolve to the same root. Note also that this is a container job (`image: amd64/rust`), `${{ runner.temp }}` expands to the host path `/home/runner/work/_temp`, and the file lands in the artifact under the container path `/__w/_temp`. Some interaction between the container path mapping and multi-path glob resolution looks like the likely culprit, but that needs confirming. Since `if-no-files-found: ignore` is set, the miss is silent. Possible fixes, roughly in order of how much I like them: - Point `path` at the directory or a glob (`${{ runner.temp }}/*.log`) so only a single search path is involved and the LCA logic is not exercised. - Write both logs into a dedicated subdirectory (e.g. `${{ runner.temp }}/cluster-logs/`) and upload that one directory. - Switch `if-no-files-found` to `warn` so a future silent miss is at least visible in the job log. Found while investigating the SF10 disk exhaustion in #1996. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
