The GitHub Actions job "Required Checks" on texera.git/main has failed. Run started by GitHub user github-merge-queue[bot] (triggered by github-merge-queue[bot]).
Head commit for run: 7a6ee7cf9198de7b103bacf685b4237b804a6ef4 / Yicong Huang <[email protected]> feat(local-dev): pick a usable Python and offer to install a missing toolchain (#7078) > **Depends on the Linux-support PR** (`feat(local-dev): support Linux`), which this needs for the platform detection. That branch is this one's parent commit, so it shows up in this diff too — review it there, and read only the second commit here. I'll rebase once it lands. ### What changes were proposed in this PR? Two halves of the same fresh-machine problem. **The interpreter that runs Python UDFs was picked blindly.** `UDF_PYTHON_PATH` defaulted to `command -v python3` at source time — the system interpreter, which almost never has `amber/requirements.txt` installed. Python UDFs then failed at worker launch on a stack of import errors that pointed nowhere near the interpreter choice. The venv AGENTS.md tells contributors to create (`<workspace>/venv312`) was ignored, as was an already-activated `$VIRTUAL_ENV`. Resolution is now lazy, and takes the first candidate that is Python 3.12 **and** can import what a worker needs: ``` $UDF_PYTHON_PATH -> $VIRTUAL_ENV -> <workspace>/venv312 -> pyenv 3.12 -> python3.12 on PATH -> python3 on PATH ``` The choice is reported the way the JDK probe reports `JAVA_HOME`. An explicit `UDF_PYTHON_PATH` still wins — it is the documented override — but now says so when it can't import the deps. "No 3.12 anywhere" and "3.12 without amber's deps" are reported differently, because they need different fixes and the old default silently produced the second one. Lazy so `status` / `logs` / `--help` don't spawn an interpreter per candidate, and never fatal: the JVM services run fine without a Python toolchain, only UDFs don't. **A missing tool was described, never offered.** `_install_hint` printed a suggestion and gave up, so a fresh machine meant hand-installing five or six things, re-running `up` after each to discover the next gap. A missing JDK 17, Node or Python 3.12 is now offered for install, after showing the exact command: | Tool | Installer | | --- | --- | | JDK 17 | distro package manager (`apt-get`/`dnf`/`yum`/`pacman`/`zypper`), `brew` on macOS, SDKMAN when there is none | | Node 24 | `nvm` (bootstraps nvm first if absent) | | Python 3.12 | `pyenv` (bootstraps pyenv first if absent) | `jenv` is not used: it switches between JDKs, it cannot install one. The decision helpers (`_pkg_manager`, `_install_cmd_for`) only ever *print* what would run; a separate step executes it. That keeps the choice unit-testable without installing anything, and it means the user sees the command — `sudo` included — before being asked. Behaviour deliberately preserved for non-interactive callers: **without a TTY nothing is ever prompted** and the old print-a-hint-and-fail path stands, so scripted and CI runs cannot hang on a `read`. `--install-missing` answers yes without asking, `--no-install` only ever prints the hint, and the contradiction is refused rather than silently resolved. docker and sbt stay hint-only — docker needs a daemon, group membership and a re-login, which is not something to do behind a y/n prompt. ### Any related issues, documentation, discussions? Closes #7066 Depends on the Linux-support PR (#7065). ### How was this PR tested? New coverage in the existing `infra`-job suite. Everything goes through the pure decision helpers, so CI asserts on *what would be installed* without installing anything; the Python probes are driven by fake interpreters that echo a version or set an exit code: ``` $ bash bin/local-dev/tests/test_local_dev_sh.sh ... ✓ pkg manager: apt-get → apt-get ✓ pkg manager: dnf only → dnf ✓ pkg manager: pacman only → pacman ✓ pkg manager: apt-get preferred over dnf → apt-get ✓ pkg manager: nothing installed (refuses) ✓ _pkg_manager keeps the Darwin/brew branch ✓ install cmd: java via apt names openjdk-17-jdk ✓ install cmd: java via apt asks for sudo explicitly ✓ install cmd: java via dnf names java-17-openjdk-devel ✓ install cmd: java with no package manager falls back to SDKMAN ✓ install cmd: node uses nvm ✓ install cmd: python uses pyenv ✓ install cmd: python pins 3.12 ✓ install cmd: refuses 'docker' ✓ install cmd: refuses 'sbt' ✓ install cmd: refuses 'definitely-not-a-tool' ✓ consent: refuses without a TTY (never prompts) ✓ consent: TEXERA_INSTALL_MISSING=1 assumes yes ✓ consent: TEXERA_INSTALL_MISSING=0 refuses ✓ python probe: 3.12 accepted ✓ python probe: 3.11 rejected ✓ python probe: missing interpreter rejected ✓ python probe: importable deps accepted ✓ python probe: missing deps rejected ✓ python probe: empty path rejected ✓ python candidates: $VIRTUAL_ENV before sibling venv312 ✓ python candidates: venv312 before bare python3 on PATH ✓ _require_udf_python helper is defined ✓ cmd_up resolves the UDF interpreter before launching ✓ cmd_auto resolves the UDF interpreter before launching ✓ cmd_up_one resolves the UDF interpreter before launching ✓ _require_udf_python points at amber's requirements, not the TUI hint ✓ UDF_PYTHON_PATH no longer defaults to bare python3 at source time ✓ --help documents --install-missing / --no-install ✓ up rejects --install-missing together with --no-install (rc=2) 103 passed, 0 failed ``` ``` $ python -m pytest bin/local-dev/tests/ -q 1 failed, 42 passed ``` `test_is_dirty_after_seed_then_edit` is **pre-existing and unrelated** — it reproduces identically on a pristine `upstream/main` checkout on this machine, and this PR touches neither `tui.py` nor that test. Filed as #7075 and fixed there. Manually, on Ubuntu 24.04.4 with the amber deps installed into a sibling `venv312` and nothing in `UDF_PYTHON_PATH` — it finds the venv rather than the system interpreter the old default would have taken: ``` $ bin/local-dev.sh auto ✓ python: /home/…/Repos/venv312/bin/python (runs Python UDFs) $ command -v python3 # what the old default resolved to /usr/bin/python3 $ /usr/bin/python3 -c "import pyarrow" ModuleNotFoundError: No module named 'pyarrow' ``` The consent path, forced by pinning a version that cannot exist (`TEXERA_PYTHON_VERSION=3.99`). With a TTY it shows the command and asks; the answer here was `n`, and the run continued: ``` ⚠ python: no Python 3.99 found — Python UDFs will not run looked in: $VIRTUAL_ENV, <workspace>/venv312, pyenv, PATH or point at one yourself: export UDF_PYTHON_PATH=/path/to/python3.99 python is missing. I can run: curl -fsSL https://pyenv.run | bash && export PYENV_ROOT="$HOME/.pyenv" && … && pyenv install -s 3.99 Run it now? [y/N] ``` Without a TTY (`</dev/null`) the same run prints the warning, never prompts, does not hang, and exits 0. Both flags are accepted on their own (`auto --no-install`, `auto --install-missing` → rc 0) and rejected together (rc 2). Not exercised end-to-end: actually running the JDK / Node / pyenv installers, since this machine already has all three. The commands themselves are asserted by the tests above; the execution step is a `bash -lc` of exactly the string shown to the user. ### Was this PR authored or co-authored using generative AI tooling? Generated-by: Claude Code (Claude Opus 5) Report URL: https://github.com/apache/texera/actions/runs/31673351101 With regards, GitHub Actions via GitBox
