The GitHub Actions job "Benchmarks" on texera.git/main has failed. Run started by GitHub user github-merge-queue[bot] (triggered by github-merge-queue[bot]).
Head commit for run: 57d4dba20ae395ccd76795308cda8b190c3c1df9 / Xinyuan Lin <[email protected]> chore(amber): reduce engine log verbosity and pin the CI log-level backstop (#6797) ### What changes were proposed in this PR? **Root cause.** `AmberLogging` names each logger `"<actorId>] [<ClassName>"` so the logback pattern `[%logger]` renders both the actor id and the class. The side effect is that these logger names start with the **actor id** (`WF6-…`, `COORDINATOR`), not `org.apache…`, so the `<logger name="org.apache" level="WARN"/>` rule in `amber/src/main/resources/logback.xml` never matches the engine classes. They all fall through to the root logger and emit at `INFO`. In loop/integration tests, workers are recreated every iteration, so per-ECM / per-worker / per-region lines fire dozens of times and flood the CI console. This PR demotes those statements to `DEBUG` so they no longer print at the default `INFO` level (CI, local, and prod), and demotes three `WARN`s that fire on **normal** operation: | Message | Was | Source | |---|---|---| | `receive` / `process` / `send ECM` (2–3× per control message, full `ChannelIdentity` dump) | INFO | `DataProcessor.scala`, `main_loop.py` | | `register <id> -> Actor[…]` | INFO | `PekkoActorRefMappingService.scala` | | `…is not reachable anymore, it might have crashed` — fires on graceful teardown | WARN | `PekkoActorRefMappingService.scala` | | `unknown identifier` — registration race | WARN | `PekkoActorRefMappingService.scala` | | `worker replay log writing conf`, `DP thread started`/`exits`, `Starting the worker.` | INFO | `WorkflowActor`, `DPThread`, `StartHandler` | | `Region N successfully terminated.` | INFO | `RegionExecutionManager.scala` | | `…completed, # of input ports…` | INFO | `DataProcessor.scala` | | `client actor cannot handle …` — `//skip` fallthrough | WARN | `ClientActor.scala` | Genuine fault paths are **untouched**: `DP Thread exists unexpectedly` (ERROR), `Failed to fetch actorRef` (WARN), and `Error when terminating region` (WARN, the failure branch — distinct from the demoted `successfully terminated` success branch). As a CI backstop, the amber unit and integration test steps in `build.yml` pin the log level so any remaining `INFO` chatter is suppressed there even before the source demotions take effect. The two logging stacks spell the level differently, which is a trap worth calling out: logback takes `TEXERA_SERVICE_LOG_LEVEL=WARN`, but the spawned Python UDF worker uses **loguru**, whose only warning level is `WARNING`. Feeding loguru `WARN` raises `ValueError: Level 'WARN' does not exist` at worker startup — before the worker hands its Flight port back to the JVM — and the JVM then blocks on the proxy handshake with no internal timeout, hanging `amber-integration` until GitHub's 6 h runner cap. So the Python var is pinned to `UDF_PYTHON_LOG_STREAMHANDLER_LEVEL=WARNING`, and, symmetrically, logback keeps `WARN` (it falls back to `DEBUG` on an unknown level like `WARNING`, which would flood logs instead). `amber-integration` additionally gets `timeout-minutes: 40` so any future worker-startup deadlock fails fast instead of burning a full runner. Net effect: the bulk of the per-iteration output from loop/integration specs disappears from the default-level log, while `DEBUG` keeps it available on demand. ### Any related issues, documentation, discussions? Closes #6796 ### How was this PR tested? The source demotions are a log-level-only change — no behavior changes — so they rely on the existing test suites run in CI. Verified additionally by inspection and by the CI backstop's own behavior: - **No test asserts on a demoted message.** Grepped all amber Scala and Python test sources for each demoted string; none is asserted. (`ReplayLogGeneratorSpec` matches `"cannot handle"`, but against an unrelated `RuntimeException` from `ReplayLogGenerator`, not the `ClientActor` log.) - **The CI `WARN`/`WARNING` env does not break config specs.** `UdfConfigSpec`'s and `PekkoConfigSpec`'s default-value assertions are guarded (`ifUnset` / `!sys.env.contains(...)`), and `pekko.stdout-loglevel` is hardcoded `INFO` in `cluster.conf` (only `pekko.loglevel` is env-driven), so its unguarded assertion still passes. - **loguru level names verified.** On the pinned loguru 0.7.0, `logger.add(sys.stderr, level="WARN")` raises `ValueError: Level 'WARN' does not exist` while `level="WARNING"` succeeds (loguru's levels are `TRACE/DEBUG/INFO/SUCCESS/WARNING/ERROR/CRITICAL`); logback accepts `WARN`. With `WARNING`, `amber-integration` runs green again — the ubuntu leg finishes the integration step in ~8.5 min, versus the earlier `WARN` runs that sat in that step ~3 h until cancelled. - **Lint clean.** No imports added (so `scalafixAll --check` is unaffected); the longest changed Scala line is 96 columns (< the 100 `maxColumn`), so `scalafmtCheckAll` stays green. A follow-up sweep for other hot-path `INFO`/`WARN` logs (e.g. `AsyncRPCClient` null control-reply, the range-shuffle partitioner, `Tuple` cast warnings) is captured in #6796 rather than expanded here. ### Was this PR authored or co-authored using generative AI tooling? Generated-by: Claude Code (Opus 4.8 [1M context]) Report URL: https://github.com/apache/texera/actions/runs/30157840410 With regards, GitHub Actions via GitBox
