Script 'mail_helper' called by obssrc Hello community, here is the log from the commit of package os-autoinst for openSUSE:Factory checked in at 2026-04-08 17:18:03 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Comparing /work/SRC/openSUSE:Factory/os-autoinst (Old) and /work/SRC/openSUSE:Factory/.os-autoinst.new.21863 (New) ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Package is "os-autoinst" Wed Apr 8 17:18:03 2026 rev:582 rq:1345191 version:5.1775569433.3681116 Changes: -------- --- /work/SRC/openSUSE:Factory/os-autoinst/os-autoinst.changes 2026-04-04 19:07:54.310324947 +0200 +++ /work/SRC/openSUSE:Factory/.os-autoinst.new.21863/os-autoinst.changes 2026-04-08 17:18:31.098801514 +0200 @@ -2 +2 @@ -Wed Apr 01 14:29:33 UTC 2026 - [email protected] +Tue Apr 07 13:44:02 UTC 2026 - [email protected] @@ -4,7 +4,17 @@ -- Update to version 5.1775053765.0f55e29: - * fix(xt): exclude build artifacts from Perl::Critic checks - * style: Call builtin functions without parentheses - * style: Ignore files under external/ in perlcritic - * style: Sort files for perlcritic - * style: Add CodeLayout::ProhibitParensWithBuiltins - * style: Turn .perlcritic into real file instead of symlink +- Update to version 5.1775569433.3681116: + * fix(consoles): support newer x3270 versions + * refactor: unconditionally import LLM analysis to fix coverage + * feat(llm): integrate LLM analysis into isotovideo + * feat(llm): add core LLM failure analysis module and tests + * fix(t/34-git): make test resilient to default branch name changes + * chore(AGENTS.md): extend with better proactive style following + +------------------------------------------------------------------- +Mon Mar 30 08:21:41 UTC 2026 - [email protected] + +- Update to version 5.1774620706.b22e21b: + * feat: Add option to add signatures + * style: Add tool to add Mojo::Base include automatically + * test: Add CoverageWorkaround.pm + * test: Use Syntax::Keyword::Try::Deparse to avoid warnings + * chore: Reduce permissions of remaining workflow @@ -12 +21,0 @@ - * test: replace Test::Strict with Test::Compile for faster syntax checks @@ -15,0 +25,384 @@ +Fri Mar 27 01:51:47 UTC 2026 - [email protected] + +- Update to version 5.1774551362.dd2a78c: + * feat(svirt): add "stop_vm" in consoles::sshVirtsh + * feat: correct isotovideo handle_shutdown log calls + * refactor(consoles/sshVirtsh): properly concatenate remote_vmm + * feat(svirt): retry disk create also in case of copy-img + * chore: Reduce permissions of workflows + * feat(VNC): Add AltGr key + +------------------------------------------------------------------- +Wed Mar 25 10:46:33 UTC 2026 - [email protected] + +- Update to version 5.1774435114.41aff24: + * style: Remove superfluous use of strict + * git subrepo pull (merge) --force external/os-autoinst-common + * Revert "style: Remove local Perl::Critic::Policy::HashKeyQuotes" + * style: Remove local Perl::Critic::Policy::HashKeyQuotes + * style: Add xt/02-perlcritic.t + * style: Fix Community::WhileDiamondDefaultAssignment + * style: Fix various no strict / no critic violations + * style: Fix OpenQA::HashKeyQuotes + * style: Fix OpenQA::SpaceAfterSubroutineName + * style: Fix ProhibitConditionalDeclarations + * style: Disable OpenQA::RedundantStrictWarning temporarily + +------------------------------------------------------------------- +Mon Mar 23 17:02:21 UTC 2026 - [email protected] + +- Update to version 5.1774283485.5af53fe: + * Revert "style: Remove local Perl::Critic::Policy::HashKeyQuotes" + * style: Remove local Perl::Critic::Policy::HashKeyQuotes + * style: Add xt/02-perlcritic.t + * style: Fix Community::WhileDiamondDefaultAssignment + * fix(ci): correct import of commit-message-checker + * style: Fix various no strict / no critic violations + * style: Fix OpenQA::HashKeyQuotes + +------------------------------------------------------------------- +Mon Mar 23 02:21:40 UTC 2026 - [email protected] + +- Update to version 5.1774101470.e82b4cb: + * feat: implement 'always_run' test flag + * refactor: use gitlint from os-autoinst-common + * git subrepo pull (merge) --force external/os-autoinst-common + * feat(snd2png): restore erroneously deleted test + * style: fix copyright in crop.py + * chore: remove unused pyproject line + * chore(deps): Add PPI to development dependencies + * chore(snd2png): update test.png.md5.original based on current snd2png + * test(full-stack): optimize execution time by reducing timeouts + * feat(vnc): make connection retry sleep configurable + * feat: add configurable secret key hiding support + +------------------------------------------------------------------- +Fri Mar 13 22:24:45 UTC 2026 - [email protected] + +- Update to version 5.1773429030.ba0de6e: + * fix: Correct number of internal test_count + * chore(AGENTS.md): add customized file + * chore(deps): add perl-Test-Perl-Critic dependency for parallel execution + * fix: Remove logger message from else condition + * style: Use single quotes for strings without interpolation + * docs: convert doc/backend_vars.asciidoc to Markdown + * docs: convert README.asciidoc to Markdown + * docs: convert doc/memorydumps.asciidoc to Markdown + * feat: add gitlint pre-commit setup + +------------------------------------------------------------------- +Fri Mar 13 05:30:17 UTC 2026 - [email protected] + +- Update to version 5.1773327169.ae7c574: + * chore(AGENTS.md): add customized file + * chore(deps): add perl-Test-Perl-Critic dependency for parallel execution + * chore(deps): Update perltidy + * fix: Remove logger message from else condition + * docs: convert doc/backend_vars.asciidoc to Markdown + * docs: convert README.asciidoc to Markdown + * docs: convert doc/memorydumps.asciidoc to Markdown + * feat: add gitlint pre-commit setup + +------------------------------------------------------------------- +Wed Mar 11 18:30:15 UTC 2026 - [email protected] + +- Update to version 5.1773245056.43fc8f0: + * chore(deps): Update perltidy + * fix: Remove logger message from else condition + * style: Use single quotes for strings without interpolation + * fix: restore author tests in CI and optimize git message check + * docs: convert doc/backend_vars.asciidoc to Markdown + +------------------------------------------------------------------- +Mon Mar 09 13:35:51 UTC 2026 - [email protected] + +- Update to version 5.1773054031.9ab699d: + * chore(deps): Update perltidy + * fix: Remove logger message from else condition + * style: Use single quotes for strings without interpolation + * fix: restore author tests in CI and optimize git message check + * refactor: move scheduling rules out of basetest::is_applicable + +------------------------------------------------------------------- +Thu Mar 05 17:55:20 UTC 2026 - [email protected] + +- Update to version 5.1772729929.93a4b15: + * feat: normalize gre tunnel script for NetworkManager and wicked + * refactor: use more Mojo::File operations in commands.pm + * refactor: use more Mojo::File operations in bmwqemu.pm + * refactor: use more Mojo::File operations in t/ + * refactor: move scheduling rules out of basetest::is_applicable + * feat: add EXIT_AFTER_MODULE to stop after a specified module + +------------------------------------------------------------------- +Thu Mar 05 00:12:06 UTC 2026 - [email protected] + +- Update to version 5.1772663930.9a9bd7d: + * feat: add EXIT_AFTER_MODULE to stop after a specified module + * fix: Update gre_tunnel_preup script to support NetworkManager + * feat: Handle timeout when typing command in `background_script_run` + * feat: Allow opting-out of check when typing command in `script_run` + * feat: Handle timeout when typing command in `script_run` + * test: implement conventional commits check with gitlint + +------------------------------------------------------------------- +Thu Feb 26 17:45:58 UTC 2026 - [email protected] + +- Update to version 5.1772097392.f4e2912: + * fix: Update gre_tunnel_preup script to support NetworkManager + * build(Makefile): add top-level help target + * test: implement conventional commits check with gitlint + * fix: Fix wrong uses of "checkout" that should be "check out" + * git subrepo pull (merge) --force external/os-autoinst-common + +------------------------------------------------------------------- +Wed Feb 25 00:22:47 UTC 2026 - [email protected] + +- Update to version 5.1771958644.63a1790: + * build(Makefile): add top-level help target + * test: implement conventional commits check with gitlint + * fix: Fix wrong uses of "checkout" that should be "check out" + * git subrepo pull (merge) --force external/os-autoinst-common + * style: Fix crop.py style issues + * parse_extra_log: Allow passing additional args to upload_logs + +------------------------------------------------------------------- +Mon Feb 23 16:33:17 UTC 2026 - [email protected] + +- Update to version 5.1771858186.01b8328: + * test: implement conventional commits check with gitlint + * fix: Fix wrong uses of "checkout" that should be "check out" + * git subrepo pull (merge) --force external/os-autoinst-common + * style: Fix crop.py style issues + * workaround: Remove "get_mempolicy" warning from qemu-img output + +------------------------------------------------------------------- +Thu Feb 19 19:58:34 UTC 2026 - [email protected] + +- Update to version 5.1771520411.2601197: + * fix: Fix wrong uses of "checkout" that should be "check out" + * git subrepo pull (merge) --force external/os-autoinst-common + * style: Fix crop.py style issues + * workaround: Remove "get_mempolicy" warning from qemu-img output + * parse_extra_log: Allow passing additional args to upload_logs + +------------------------------------------------------------------- +Wed Feb 18 16:16:06 UTC 2026 - [email protected] + +- Update to version 5.1771353921.c8005c9: + * git subrepo pull (merge) --force external/os-autoinst-common + * style: Fix crop.py style issues + * workaround: Remove "get_mempolicy" warning from qemu-img output + * parse_extra_log: Allow passing additional args to upload_logs + * refactor: Distinguish tests by the script path in `loadtest` + * refactor: Simplify approach for avoiding redefine warnings + +------------------------------------------------------------------- +Tue Feb 10 15:20:22 UTC 2026 - [email protected] + +- Update to version 5.1770715824.6a80a85: + * style: Fix crop.py style issues + * workaround: Remove "get_mempolicy" warning from qemu-img output + * parse_extra_log: Allow passing additional args to upload_logs + * refactor: Distinguish tests by the script path in `loadtest` + * refactor: Simplify approach for avoiding redefine warnings + * test: Allow running tests with `Test::Warnings<0.033` + * test: Format test of `loadtestdir` in a more compact way + +------------------------------------------------------------------- +Thu Feb 05 14:22:35 UTC 2026 - [email protected] + +- Update to version 5.1770127521.c249fe9: + * refactor: Distinguish tests by the script path in `loadtest` + * refactor: Simplify approach for avoiding redefine warnings + * test: Allow running tests with `Test::Warnings<0.033` + * test: Format test of `loadtestdir` in a more compact way + * test: Use `ENABLE_MODERN_PERL_FEATURES=1` in test suite + * feat: Allow enabling strict/warnings/signatures globally + * fix: Improve wrong comment about enablement of modern Perl features + +------------------------------------------------------------------- +Wed Jan 28 12:34:32 UTC 2026 - [email protected] + +- Update to version 5.1769602729.9728790: + * fix: Improve wrong comment about enablement of modern Perl features + * Replace remaining functions with subroutine signatures in 18-qemu.t + * Fix snapshot overlay mechanism to avoid duplication + * fix(dist): provide proper copyright headers in all spec-files + * fix(dist): try to fix os-autoinst-obs-auto-submit reverting content + * Remove deprecated BIOS and UEFI_PFLASH variables + +------------------------------------------------------------------- +Fri Jan 23 21:33:15 UTC 2026 - [email protected] + +- Update to version 5.1769153586.72cabd0: + * Replace remaining functions with subroutine signatures in 18-qemu.t + * Fix snapshot overlay mechanism to avoid duplication + * fix(dist): provide proper copyright headers in all spec-files + * fix(dist): try to fix os-autoinst-obs-auto-submit reverting content + * fix(dist): exclude unstable t/28-signalblocker.t in OBS checks + * Add documentation of APPEND variable + * Add undocumented KERNEL/INITRD to the supported variables + * os-autoinst-generate-needle-preview: Embed PNG + +------------------------------------------------------------------- +Fri Jan 16 20:43:12 UTC 2026 - [email protected] + +- Update to version 5.1768577300.b85e486: + * fix(dist): provide proper copyright headers in all spec-files + * fix(dist): try to fix os-autoinst-obs-auto-submit reverting content + * fix(dist): exclude unstable t/28-signalblocker.t in OBS checks + * Remove deprecated BIOS and UEFI_PFLASH variables + * Add documentation of APPEND variable + * os-autoinst-generate-needle-preview: Embed PNG + +------------------------------------------------------------------- +Thu Jan 08 17:47:42 UTC 2026 - [email protected] + +- Update to version 5.1767893100.fd5003c: + * Add documentation of APPEND variable + * Add undocumented KERNEL/INITRD to the supported variables + * os-autoinst-generate-needle-preview: Embed PNG + * Tweak curl call not to hang + * Fix opencv dependency due to upstream changes + +------------------------------------------------------------------- +Mon Jan 05 16:08:57 UTC 2026 - [email protected] + +- Update to version 5.1767623406.688dd0e: + * os-autoinst-generate-needle-preview: Embed PNG + * Tweak curl call not to hang + * Fix opencv dependency due to upstream changes + * Restore package builds on older openSUSE versions + * Remove `ShellCheck` from devel dependencies on s390x + +------------------------------------------------------------------- +Thu Dec 18 21:51:02 UTC 2025 - [email protected] + +- Update to version 5.1766037062.44c7d2a: + * Tweak curl call not to hang + * Fix opencv dependency due to upstream changes + * Restore package builds on older openSUSE versions + * Remove `ShellCheck` from devel dependencies on s390x + * Remove obsolete 'bin/' folder + +------------------------------------------------------------------- +Wed Dec 17 17:11:55 UTC 2025 - [email protected] + ++++ 119 more lines (skipped) ++++ between /work/SRC/openSUSE:Factory/os-autoinst/os-autoinst.changes ++++ and /work/SRC/openSUSE:Factory/.os-autoinst.new.21863/os-autoinst.changes Old: ---- os-autoinst-5.1775053765.0f55e29.obscpio New: ---- os-autoinst-5.1775569433.3681116.obscpio ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Other differences: ------------------ ++++++ os-autoinst-devel-test.spec ++++++ --- /var/tmp/diff_new_pack.kIoPoH/_old 2026-04-08 17:18:32.198846809 +0200 +++ /var/tmp/diff_new_pack.kIoPoH/_new 2026-04-08 17:18:32.202846974 +0200 @@ -18,7 +18,7 @@ %define short_name os-autoinst-devel Name: %{short_name}-test -Version: 5.1775053765.0f55e29 +Version: 5.1775569433.3681116 Release: 0 Summary: Test package for %{short_name} License: GPL-2.0-or-later ++++++ os-autoinst-openvswitch-test.spec ++++++ --- /var/tmp/diff_new_pack.kIoPoH/_old 2026-04-08 17:18:32.234848291 +0200 +++ /var/tmp/diff_new_pack.kIoPoH/_new 2026-04-08 17:18:32.238848457 +0200 @@ -19,7 +19,7 @@ %define name_ext -test %define short_name os-autoinst-openvswitch Name: %{short_name}%{?name_ext} -Version: 5.1775053765.0f55e29 +Version: 5.1775569433.3681116 Release: 0 Summary: test package for %{short_name} License: GPL-2.0-or-later ++++++ os-autoinst-test.spec ++++++ --- /var/tmp/diff_new_pack.kIoPoH/_old 2026-04-08 17:18:32.286850433 +0200 +++ /var/tmp/diff_new_pack.kIoPoH/_new 2026-04-08 17:18:32.290850597 +0200 @@ -19,7 +19,7 @@ %define name_ext -test %define short_name os-autoinst Name: %{short_name}%{?name_ext} -Version: 5.1775053765.0f55e29 +Version: 5.1775569433.3681116 Release: 0 Summary: test package for os-autoinst License: GPL-2.0-or-later ++++++ os-autoinst.spec ++++++ --- /var/tmp/diff_new_pack.kIoPoH/_old 2026-04-08 17:18:32.330852244 +0200 +++ /var/tmp/diff_new_pack.kIoPoH/_new 2026-04-08 17:18:32.330852244 +0200 @@ -17,7 +17,7 @@ Name: os-autoinst -Version: 5.1775053765.0f55e29 +Version: 5.1775569433.3681116 Release: 0 Summary: OS-level test automation License: GPL-2.0-or-later ++++++ os-autoinst-5.1775053765.0f55e29.obscpio -> os-autoinst-5.1775569433.3681116.obscpio ++++++ diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/os-autoinst-5.1775053765.0f55e29/AGENTS.md new/os-autoinst-5.1775569433.3681116/AGENTS.md --- old/os-autoinst-5.1775053765.0f55e29/AGENTS.md 2026-04-01 16:29:25.000000000 +0200 +++ new/os-autoinst-5.1775569433.3681116/AGENTS.md 2026-04-07 15:43:53.000000000 +0200 @@ -14,7 +14,11 @@ - Code style: Run `tools/tidyall --all` (or `tools/tidyall --git` for changed files only). -- Testing: Always add tests for new features or bug fixes in `t/`. +- Linter: Always run `make test-perl-testsuite TESTS="xt/01-style.t xt/02-perlcritic.t"` + for Perl changes before claiming completion. +- Testing: Always add tests for new features or bug fixes in `t/`. Prefer + reusing existing failing test modules (e.g. from `t/data/tests`) for + integration tests. - Dependencies: Update `dependencies.yaml` and run `make update-deps`. ## Constraints diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/os-autoinst-5.1775053765.0f55e29/CMakeLists.txt new/os-autoinst-5.1775569433.3681116/CMakeLists.txt --- old/os-autoinst-5.1775053765.0f55e29/CMakeLists.txt 2026-04-01 16:29:25.000000000 +0200 +++ new/os-autoinst-5.1775569433.3681116/CMakeLists.txt 2026-04-07 15:43:53.000000000 +0200 @@ -62,6 +62,7 @@ OpenQA/Isotovideo/CommandHandler.pm OpenQA/Isotovideo/Dewebsockify.pm OpenQA/Isotovideo/Interface.pm + OpenQA/Isotovideo/LLMAnalysis.pm OpenQA/Isotovideo/NeedleDownloader.pm OpenQA/Isotovideo/Runner.pm OpenQA/Isotovideo/Utils.pm diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/os-autoinst-5.1775053765.0f55e29/OpenQA/Isotovideo/LLMAnalysis.pm new/os-autoinst-5.1775569433.3681116/OpenQA/Isotovideo/LLMAnalysis.pm --- old/os-autoinst-5.1775053765.0f55e29/OpenQA/Isotovideo/LLMAnalysis.pm 1970-01-01 01:00:00.000000000 +0100 +++ new/os-autoinst-5.1775569433.3681116/OpenQA/Isotovideo/LLMAnalysis.pm 2026-04-07 15:43:53.000000000 +0200 @@ -0,0 +1,130 @@ +# Copyright SUSE LLC +# SPDX-License-Identifier: GPL-2.0-or-later + +package OpenQA::Isotovideo::LLMAnalysis; +use Mojo::Base -signatures; +use bmwqemu; +use Feature::Compat::Try; +use IPC::Run (); +use Mojo::UserAgent; +use Mojo::JSON qw(decode_json); +use Mojo::File qw(path); +use Text::ParseWords; + +use constant MAX_PAYLOAD_SIZE => 8000; + +sub _get_file_tail ($filename, $result_dir, $max_lines) { + my $file = $filename; + $file = "$result_dir/$filename" unless -e $file; + return '' unless -e $file; + my $tail = qx(tail -n $max_lines \Q$file\E 2>/dev/null) || ''; + chomp $tail; + return $tail; +} + +sub gather_context ($result_dir) { + my @failed_tests; + for my $res_file (glob "$result_dir/result-*.json") { + my $json = eval { decode_json(path($res_file)->slurp) }; + next unless $json && ($json->{result} // '') eq 'fail'; + my $name = $json->{name}; + ($name) = $res_file =~ /result-(.*)\.json$/ unless $name; + push @failed_tests, $name if $name; + } + return undef unless @failed_tests; + my $failed_str = join ', ', @failed_tests; + my $log_tail = _get_file_tail('autoinst-log.txt', $result_dir, 200); + my $serial_tail = _get_file_tail('serial0', $result_dir, 100); + my $context = { + failed_tests => $failed_str, + log_tail => $log_tail, + serial_tail => $serial_tail, + distri => $bmwqemu::vars{DISTRI} || 'unknown', + version => $bmwqemu::vars{VERSION} || 'unknown', + arch => $bmwqemu::vars{ARCH} || 'unknown', + }; + return $context; +} + +sub build_prompt ($context) { + my $distri = $context->{distri}; + my $version = $context->{version}; + my $arch = $context->{arch}; + my $failed_tests = $context->{failed_tests}; + my $log_tail = $context->{log_tail}; + my $serial_tail = $context->{serial_tail}; + + # Truncate inputs to preserve instructions at the end of the prompt + $log_tail = substr $log_tail, -MAX_PAYLOAD_SIZE if length($log_tail) > MAX_PAYLOAD_SIZE; + $serial_tail = substr $serial_tail, -MAX_PAYLOAD_SIZE if length($serial_tail) > MAX_PAYLOAD_SIZE; + + my $prompt = <<"EOF"; +You are analyzing an automated test run of $distri $version $arch. +The following tests failed: $failed_tests. + +Relevant log tail: +$log_tail + +Serial output tail: +$serial_tail + +Provide exactly 2-3 sentences answering: +1. Why did the tests fail? +2. What should be done to prevent these failures? +3. Is this likely a product regression or a test infrastructure problem + (false positive)? +EOF + + return $prompt; +} + +sub query_llm_api ($prompt, $url, $model) { + my $ua = Mojo::UserAgent->new; + $ua->connect_timeout(300); + $ua->inactivity_timeout(300); + my $req = { + model => $model, + messages => [{role => 'user', content => $prompt}], + temperature => 0.3 + }; + my $tx = $ua->post($url => json => $req); + if (my $err = $tx->error) { + return 'Error: HTTP API failed with status ' . $err->{code} if $err->{code}; + return 'Error: ' . ($err->{message} || 'Connection failed'); + } + my $res = $tx->result; + my $json = eval { decode_json($res->body) }; + return $json->{choices}[0]{message}{content} if $json && $json->{choices} && $json->{choices}[0]{message}{content}; + return 'Error: Unexpected response format from LLM API.'; +} + +sub query_llm_cmd ($prompt, $cmd) { + my @cmd_array = Text::ParseWords::shellwords($cmd); + my $out; + my $err; + try { + my $success = IPC::Run::run(\@cmd_array, \$prompt, \$out, \$err, IPC::Run::timeout(300)); + return "Error: Command exited with $? - " . ($err || $out || '') unless $success; + } catch ($e) { return "Error: Command failed - $e" } + return $out || $err || 'Error: Command produced no output.'; +} + +sub run ($result_dir) { + my $context = gather_context($result_dir); + return unless $context; + bmwqemu::diag('Starting LLM Analysis…'); + my $prompt = build_prompt($context); + my $output; + if (my $cmd = $bmwqemu::vars{LLM_FAILURE_ANALYSIS_CMD}) { + $output = query_llm_cmd($prompt, $cmd); + } + else { + my $url = $bmwqemu::vars{LLM_FAILURE_ANALYSIS_URL} || 'http://localhost:8080/v1/chat/completions'; + my $model = $bmwqemu::vars{LLM_FAILURE_ANALYSIS_MODEL} || 'gemma-4-26B-A4B-it'; + $output = query_llm_api($prompt, $url, $model); + } + path("$result_dir/llm-failure-analysis.txt")->spurt($output); + bmwqemu::diag("LLM Analysis complete. Saved to $result_dir/llm-failure-analysis.txt"); +} + +1; diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/os-autoinst-5.1775053765.0f55e29/consoles/s3270.pm new/os-autoinst-5.1775569433.3681116/consoles/s3270.pm --- old/os-autoinst-5.1775053765.0f55e29/consoles/s3270.pm 2026-04-01 16:29:25.000000000 +0200 +++ new/os-autoinst-5.1775569433.3681116/consoles/s3270.pm 2026-04-07 15:43:53.000000000 +0200 @@ -235,8 +235,7 @@ sub wait_output ($self, $timeout = 0) { my $r = $self->send_3270("Wait($timeout,Output)", command_status => 'any'); return 1 if $r->{command_status} eq 'ok'; - return 0 if $r->{command_output}[0] eq 'Wait: Timed out'; - return 0 if $r->{command_output}[0] eq 'Wait(): Timed out'; + return 0 if $r->{command_output}[0] =~ /^Wait[^:]*: Timed out$/; confess "has the s3270 wait timeout failure response changed?\n" . Dumper $r; } diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/os-autoinst-5.1775053765.0f55e29/distribution.pm new/os-autoinst-5.1775569433.3681116/distribution.pm --- old/os-autoinst-5.1775053765.0f55e29/distribution.pm 2026-04-01 16:29:25.000000000 +0200 +++ new/os-autoinst-5.1775569433.3681116/distribution.pm 2026-04-07 15:43:53.000000000 +0200 @@ -14,6 +14,7 @@ $self->{consoles} = {}; $self->{serial_failures} = []; $self->{autoinst_failures} = []; + $self->{_serial_marker_level} = {}; =head2 serial_term_prompt @@ -144,26 +145,45 @@ if (testapi::is_serial_terminal) { testapi::wait_serial($self->{serial_term_prompt}, no_regex => 1, quiet => $args{quiet}); } - testapi::type_string "$cmd", max_interval => $args{max_interval}; + if ($args{timeout} > 0) { die "Terminator '&' found in script_run call. script_run can not check script success. Use 'background_script_run' instead." if $cmd =~ qr/(?<!\\)&$/; - my $str = testapi::hashed_string('SR' . $cmd . $args{timeout}); - my $marker = "; echo $str-\$?-" . ($args{output} ? "Comment: $args{output}" : ''); - if (testapi::is_serial_terminal) { - testapi::type_string($marker, max_interval => $args{max_interval}); - testapi::wait_serial($cmd . $marker, no_regex => 1, quiet => $args{quiet}, buffer_size => length($cmd) + 128) - or _handle_cmd_typing_error($cmd, \%args); - testapi::type_string("\n", max_interval => $args{max_interval}); + + my $level = $self->_detect_serial_marker_capability(); + my ($str, $wait_pattern); + if ($level == 3) { + testapi::query_isotovideo('backend_clear_serial_buffer', {}); + testapi::type_string "$cmd\n", max_interval => $args{max_interval}; + my $res = testapi::wait_serial(qr/OA:DONE-(\d+)-/, timeout => $args{timeout}, quiet => $args{quiet}); + return unless $res; + return ($res =~ /OA:DONE-(\d+)-/)[0]; + } + $str = testapi::hashed_string('SR' . $cmd . $args{timeout}); + $wait_pattern = qr/$str-(\d+)-/; + if ($level == 2) { + testapi::type_string "export __OA_MARK=$str; $cmd\n", max_interval => $args{max_interval}; } else { - testapi::type_string "$marker > /dev/$testapi::serialdev\n", max_interval => $args{max_interval}; + my $marker = "; echo $str-\$?-" . ($args{output} ? "Comment: $args{output}" : ''); + if (testapi::is_serial_terminal) { + testapi::type_string "$cmd", max_interval => $args{max_interval}; + testapi::type_string $marker, max_interval => $args{max_interval}; + testapi::wait_serial($cmd . $marker, no_regex => 1, quiet => $args{quiet}, buffer_size => (length $cmd) + 128) + or _handle_cmd_typing_error($cmd, \%args); + testapi::type_string "\n", max_interval => $args{max_interval}; + } + else { + testapi::type_string "$cmd", max_interval => $args{max_interval}; + testapi::type_string "$marker > /dev/$testapi::serialdev\n", max_interval => $args{max_interval}; + } } - my $res = testapi::wait_serial(qr/$str-\d+-/, timeout => $args{timeout}, quiet => $args{quiet}); + my $res = testapi::wait_serial($wait_pattern, timeout => $args{timeout}, quiet => $args{quiet}); return unless $res; - return ($res =~ /$str-(\d+)-/)[0]; + return ($res =~ $wait_pattern)[0]; } else { + testapi::type_string "$cmd", max_interval => $args{max_interval}; testapi::send_key 'ret'; return; } @@ -194,9 +214,9 @@ my $str = testapi::hashed_string('SR' . $cmd); my $marker = "& echo $str-\$!-" . ($args{output} ? "Comment: $args{output}" : ''); if (testapi::is_serial_terminal) { - testapi::type_string($marker); + testapi::type_string $marker; testapi::wait_serial($cmd . $marker, no_regex => 1, quiet => $args{quiet}) or _handle_cmd_typing_error($cmd, \%args); - testapi::type_string("\n"); + testapi::type_string "\n"; } else { testapi::type_string "$marker > /dev/$testapi::serialdev\n"; @@ -262,7 +282,7 @@ quiet => undef, # 80 is approximate quantity of chars typed during 'curl' approach # if script length is lower there is no point to proceed with more complex solution - type_command => length($script) < 80, + type_command => length $script < 80, }, ['timeout'], @args); my $marker = testapi::hashed_string("SO$script"); @@ -279,21 +299,21 @@ my $cat = "cat > $script_path << '$heretag'; echo $marker-\$?-"; testapi::wait_serial($self->{serial_term_prompt}, no_regex => 1, quiet => $args{quiet}); bmwqemu::log_call("Content of $script_path :\n \"$cat\" \n"); - testapi::type_string($cat . "\n"); + testapi::type_string $cat . "\n"; testapi::wait_serial("$cat", no_regex => 1, quiet => $args{quiet}); # Wait for input prompt of here tag before typing $script. This avoids # messy output, like duplicate output of $script. We do this in a second # wait_serial() call, to avoid issues during new line detection. testapi::wait_serial('> ', no_regex => 1, quiet => $args{quiet}); - testapi::type_string("$script\n$heretag\n"); + testapi::type_string "$script\n$heretag\n"; testapi::wait_serial("> $heretag", no_regex => 1, quiet => $args{quiet}); testapi::wait_serial("$marker-0-", quiet => $args{quiet}); } elsif ($args{type_command}) { my $cat = "cat - > $script_path;"; - testapi::type_string($cat); - testapi::type_string("\n", wait_still_screen => testapi::backend_get_wait_still_screen_on_here_doc_input()); - testapi::type_string($script . "\n", timeout => $args{timeout}); + testapi::type_string $cat; + testapi::type_string "\n", wait_still_screen => testapi::backend_get_wait_still_screen_on_here_doc_input(); + testapi::type_string $script . "\n", timeout => $args{timeout}; testapi::send_key('ctrl-d'); } else { @@ -313,11 +333,11 @@ my $run_script = "echo $marker; $shell_cmd $script_path ; echo SCRIPT_FINISHED$marker-\$?-"; if (testapi::is_serial_terminal) { testapi::wait_serial($self->{serial_term_prompt}, no_regex => 1, quiet => $args{quiet}); - testapi::type_string("$run_script\n"); + testapi::type_string "$run_script\n"; testapi::wait_serial($run_script, no_regex => 1, quiet => $args{quiet}); } else { - testapi::type_string("($run_script) | tee /dev/$testapi::serialdev\n"); + testapi::type_string "($run_script) | tee /dev/$testapi::serialdev\n"; } my $output = testapi::wait_serial("SCRIPT_FINISHED$marker-\\d+-", timeout => $args{timeout}, record_output => 1, quiet => $args{quiet}) || croak "script timeout: $script"; @@ -387,4 +407,85 @@ # override sub console_selected ($self, $console) { } +=head2 sut_marker + + sut_marker($cmd) + +Generate a unique marker string for a command to be used for synchronization +with the SUT. Used primarily for internal testing. + +=cut + +sub sut_marker ($self, $cmd) { + my $c = $cmd; + $c =~ s/^\s+|\s+$//g; + my $l = length $c; + my $head = substr $c, 0, 4; + my $tail = $l >= 4 ? substr $c, -4 : $c; + return "OA:${head}${l}${tail}"; +} + +=head2 install_serial_marker_hook + + install_serial_marker_hook($level) + +Install shell hooks (like PROMPT_COMMAND) into the SUT to emit synchronization +markers to serial. + +=cut + +sub install_serial_marker_hook ($self, $level) { + return if $level < 2; + my $pc; + my $dev = "/dev/$testapi::serialdev"; + if ($level == 3) { + $pc = "PROMPT_COMMAND='printf \"OA:DONE-%d-\\n\" \$? > $dev'"; + } + else { + $pc = "PROMPT_COMMAND='if [ -n \"\$__OA_MARK\" ]; then echo \"\${__OA_MARK}-\$?-\" > $dev; unset __OA_MARK; fi'"; + } + testapi::type_string "$pc\n"; +} + +=head2 _detect_serial_marker_capability + + _detect_serial_marker_capability() + +Detect the SUT's shell capabilities for pretty serial markers. +Returns: +- 1: Fallback (classic markers) +- 2: Basic bash (PROMPT_COMMAND support) +- 3: Advanced bash (PROMPT_COMMAND + history/fc support) + +=cut + +sub _detect_serial_marker_capability ($self) { + my $console = testapi::current_console() // 'sut'; + return $self->{_serial_marker_level}->{$console} if $self->{_serial_marker_level}->{$console}; + + my $level = 1; + my $pretty = testapi::get_var('PRETTY_SERIAL_MARKER'); + my $serial_term = testapi::is_serial_terminal(); + if ($pretty && !$serial_term) { + testapi::type_string "echo \"BASH:\$BASH_VERSION:\" > /dev/$testapi::serialdev\n"; + my $out = testapi::wait_serial(qr/BASH:([^:]*):/, 10); + if ($out && $out =~ /BASH:([3-9]|\d{2,})/) { + $level = 2; + # Check if bash and history features are available to use pretty serial markers + testapi::type_string "type fc && set -o | grep -q 'history.*on' && echo \"FC:OK:\" > /dev/$testapi::serialdev\n"; + if (testapi::wait_serial(qr/FC:OK:/, 10)) { + $level = 3; + } + $self->install_serial_marker_hook($level); + bmwqemu::log_call("serial_marker: console '$console' Level $level detected"); + } + else { + bmwqemu::log_call("serial_marker: console '$console' Level 1 detected (fallback)"); + return 1; + } + } + $self->{_serial_marker_level}->{$console} = $level; + return $level; +} + 1; diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/os-autoinst-5.1775053765.0f55e29/doc/backend_vars.md new/os-autoinst-5.1775569433.3681116/doc/backend_vars.md --- old/os-autoinst-5.1775053765.0f55e29/doc/backend_vars.md 2026-04-01 16:29:25.000000000 +0200 +++ new/os-autoinst-5.1775569433.3681116/doc/backend_vars.md 2026-04-07 15:43:53.000000000 +0200 @@ -41,12 +41,17 @@ | PAUSE_ON_NEXT_COMMAND | boolean | 0 | Pause test execution on the next test API command. Same notes as for `PAUSE_AT` apply. | | PAUSE_ON_FAILURE | boolean | 0 | Pause test execution on a test failure (instead of invoking the post-fail hook and terminating). Same notes as for `PAUSE_AT` apply. | | _QUIET_SCRIPT_CALLS | boolean | 0 | Add quiet flag to all the calls to script_run, script_output and validate_script_output. It will omit all the squares "wait_serial expected" on the Details view of the test case. This option might be useful for serial terminal tests. | +| PRETTY_SERIAL_MARKER | boolean | 0 | Enable "pretty" serial markers. When enabled, os-autoinst attempts to automatically detect SUT shell capabilities (like bash PROMPT_COMMAND and fc history) to forward command exit codes to serial without typing them visibly over VNC. Falls back to regular serial markers if no advanced shell features are detected. | | _WAIT_STILL_SCREEN_ON_HERE_DOC_INPUT | float | 0 | If this value is greater then 0, it is used by `wait_still_screen` before starting to write the script into the here document used in `testapi::script_output()` function (see: poo#60566). By default this depends on the backend. | | AUTOINST_URL_HOSTNAME | string | | hostname or IP address of host running the autoinst webserver endpoint, defaults to the local IP address within the qemu network for the qemu backend or the `WORKER_HOSTNAME` otherwise. | | UPLOAD_METER | boolean | 0 | Display curl progress meter in `upload_logs()` and `upload_assets()` test API functions. | | UPLOAD_MAX_MESSAGE_SIZE_GB | integer | 0 | Specifies the max. upload size in GiB for the test API functions `upload_logs()` and `upload_assets()` and the underlying command server API. Zero denotes infinity. | | UPLOAD_INACTIVITY_TIMEOUT | integer | 300 | Specifies the inactivity timeout in seconds for the test API functions `upload_logs()` and `upload_assets()` and underlying the command server API. | | NO_DEPRECATE_BACKEND_$backend | boolean | 0 | Only warn about deprecated backends instead of aborting | +| LLM_FAILURE_ANALYSIS | boolean | 0 | Enable LLM failure analysis | +| LLM_FAILURE_ANALYSIS_URL | string | http://localhost:8080/v1/chat/completions | OpenAI-compatible API endpoint | +| LLM_FAILURE_ANALYSIS_MODEL | string | gemma-4-26B-A4B-it | Model name sent in the API request | +| LLM_FAILURE_ANALYSIS_CMD | string | | If set, run this CLI command instead of the HTTP API (prompt piped via stdin). For demo/one-off use. | | XRES | integer | 1024 | Resolution of display on x axis. Sets the resolution of the video encoder, and in qemu, the initial console resolution when OFW is set (Power and SPARC), and the EDID resolution for devices that support EDID | | YRES | integer | 768 | Resolution of display on y axis. Sets the resolution of the video encoder, and in qemu, the initial console resolution when OFW is set (Power and SPARC), and the EDID resolution for devices that support EDID | | VIDEO_ENCODER_BLOCKING_PIPE | boolean | 0 | Whether the pipe for writing data to the video encoder should be blocking or not. Making it blocking might allow following the live view in realtime despite large screenshot file sizes but it is not a well tested configuration | diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/os-autoinst-5.1775053765.0f55e29/script/isotovideo new/os-autoinst-5.1775569433.3681116/script/isotovideo --- old/os-autoinst-5.1775053765.0f55e29/script/isotovideo 2026-04-01 16:29:25.000000000 +0200 +++ new/os-autoinst-5.1775569433.3681116/script/isotovideo 2026-04-07 15:43:53.000000000 +0200 @@ -113,6 +113,7 @@ use OpenQA::Isotovideo::Interface; use OpenQA::Isotovideo::Runner; use OpenQA::Isotovideo::Utils qw(git_rev_parse spawn_debuggers handle_generated_assets); +use OpenQA::Isotovideo::LLMAnalysis; my %options; @@ -152,6 +153,7 @@ my $clean_shutdown = $runner->handle_shutdown(\$RETURN_CODE); bmwqemu::load_vars(); # read calculated variables from backend and tests $RETURN_CODE = handle_generated_assets($runner->command_handler, $clean_shutdown) unless $RETURN_CODE; + OpenQA::Isotovideo::LLMAnalysis::run(bmwqemu::result_dir()) if $bmwqemu::vars{LLM_FAILURE_ANALYSIS}; diag 'isotovideo completed handle_shutdown: ' . ($RETURN_CODE ? 'failed' : 'done'); } diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/os-autoinst-5.1775053765.0f55e29/t/03-testapi.t new/os-autoinst-5.1775569433.3681116/t/03-testapi.t --- old/os-autoinst-5.1775053765.0f55e29/t/03-testapi.t 2026-04-01 16:29:25.000000000 +0200 +++ new/os-autoinst-5.1775569433.3681116/t/03-testapi.t 2026-04-07 15:43:53.000000000 +0200 @@ -54,6 +54,9 @@ my $cmd = $lcmd->{cmd}; if ($cmd eq 'backend_wait_serial') { my $str = $lcmd->{regexp}; + $str =~ s/\(\?\^.*?://; + $str =~ s/\)$//; + $str =~ s/\((.*?)\)/$1/g; $str =~ s,\\d\+(\\s\+\\S\+)?,$fake_exit,; return {ret => {matched => $fake_matched, string => $str}}; } diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/os-autoinst-5.1775053765.0f55e29/t/05-distribution.t new/os-autoinst-5.1775569433.3681116/t/05-distribution.t --- old/os-autoinst-5.1775053765.0f55e29/t/05-distribution.t 2026-04-01 16:29:25.000000000 +0200 +++ new/os-autoinst-5.1775569433.3681116/t/05-distribution.t 2026-04-07 15:43:53.000000000 +0200 @@ -70,6 +70,80 @@ qr/typing command 'foo' timed out/, 'timeout while typing command just logged when opted-out'; }; +subtest 'pretty_serial_marker' => sub { + my $d = distribution->new; + my $mock_testapi = Test::MockModule->new('testapi'); + my $mock_bmwqemu = Test::MockModule->new('bmwqemu'); + $mock_bmwqemu->noop('log_call'); + my $typed_string = ''; + $mock_testapi->redefine(query_isotovideo => sub { }); + $mock_testapi->redefine(type_string => sub { $typed_string .= $_[0] }); + $mock_testapi->redefine(hashed_string => sub { return 'SR' . substr $_[0], 0, 8 }); + $mock_testapi->redefine(is_serial_terminal => sub { 0 }); + $mock_testapi->redefine(current_console => sub { 'test-console' }); + $mock_testapi->redefine(get_var => sub { $_[0] eq 'PRETTY_SERIAL_MARKER' ? 1 : undef }); + $testapi::serialdev = 'ttyS0'; + + $mock_testapi->redefine(wait_serial => sub { + my ($regexp) = @_; + return 'BASH:4.4:' if ref($regexp) eq 'Regexp' && 'BASH:4.4:' =~ $regexp; + return undef if ref($regexp) eq 'Regexp' && $regexp =~ /FC/; + return 'SRfoo-0-'; + }); + + $typed_string = ''; + $d->script_run('foo'); + like $typed_string, qr/export __OA_MARK=.*; foo\n/, 'Level 2 uses export marker'; + + $mock_testapi->redefine(wait_serial => sub { + my ($regexp) = @_; + return 'BASH:4.4:' if ref($regexp) eq 'Regexp' && 'BASH:4.4:' =~ $regexp; + return 'FC:OK:' if ref($regexp) eq 'Regexp' && 'FC:OK:' =~ $regexp; + return 'OA:foo3foo-0-'; + }); + + $d->{_serial_marker_level} = {}; + $typed_string = ''; + $d->script_run('foo'); + like $typed_string, qr/foo\n$/, 'Level 3 ends with command + newline'; + is substr($typed_string, -4), "foo\n", 'Level 3 uses clean command line'; + + $mock_testapi->redefine(wait_serial => sub { undef }); + $d->{_serial_marker_level} = {}; + $typed_string = ''; + is $d->_detect_serial_marker_capability(), 1, 'Fallback to Level 1 if BASH detection fails'; + + $d->{_serial_marker_level}->{'test-console'} = 3; + $mock_testapi->redefine(wait_serial => sub { undef }); + is $d->script_run('foo'), undef, 'script_run returns undef if wait_serial fails (Level 2)'; + + $d->{_serial_marker_level}->{'test-console'} = 1; + $mock_testapi->redefine(wait_serial => sub { 'SRfoo-0-' }); + + $mock_testapi->redefine(is_serial_terminal => sub { 0 }); + $typed_string = ''; + $d->script_run('foo'); + like $typed_string, qr/foo; echo SR.*-.*- > \/dev\/ttyS0\n/, 'Level 1 uses classic marker with redirection'; + + $mock_testapi->redefine(is_serial_terminal => sub { 1 }); + $typed_string = ''; + $d->script_run('foo'); + like $typed_string, qr/foo; echo SR.*-.*-\n/, 'Level 1 uses classic marker on serial terminal'; + + $mock_testapi->redefine(wait_serial => sub ($pat, %args) { + return 0 if $pat =~ /foo; echo SR.*-\$\?-/; + return 'SRfoo-0-'; + }); + throws_ok { $d->script_run('foo') } qr/typing command 'foo' timed out/, 'typing error handled in Level 1'; +}; + +subtest 'sut_marker' => sub { + my $d = distribution->new; + is $d->sut_marker('ls -la /tmp'), 'OA:ls -11/tmp', 'sut_marker for normal command'; + is $d->sut_marker(' ls '), 'OA:ls2ls', 'sut_marker trims and handles short command'; + is $d->sut_marker('a'), 'OA:a1a', 'sut_marker for very short command'; +}; + subtest 'set expected serial and autoinst failures' => sub { my $d = distribution->new; # Define the expected failures data diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/os-autoinst-5.1775053765.0f55e29/t/14-isotovideo.t new/os-autoinst-5.1775569433.3681116/t/14-isotovideo.t --- old/os-autoinst-5.1775053765.0f55e29/t/14-isotovideo.t 2026-04-01 16:29:25.000000000 +0200 +++ new/os-autoinst-5.1775569433.3681116/t/14-isotovideo.t 2026-04-07 15:43:53.000000000 +0200 @@ -336,6 +336,27 @@ delete $bmwqemu::vars{FORCE_PUBLISH_HDD_1}; }; + subtest 'LLM failure analysis' => sub { + chdir $pool_dir; + path('vars.json')->remove if -e 'vars.json'; + path('testresults/')->remove_tree; + path('testresults/')->make_path; + # Create a dummy failed test result to trigger gathering context + path('testresults/result-failing_module.json')->spurt('{"result": "fail", "name": "failing_module"}'); + path('autoinst-log.txt')->spurt("Something went wrong in the log\n"); + path('serial0')->spurt("Kernel panic in serial output\n"); + my $log = combined_from { + isotovideo( + opts => "casedir=$data_dir/tests schedule=module1 LLM_FAILURE_ANALYSIS=1 LLM_FAILURE_ANALYSIS_CMD=cat", + exit_code => 0) + }; + like $log, qr/Starting LLM Analysis/, 'LLM analysis started'; + like $log, qr/LLM Analysis complete/, 'LLM analysis finished'; + my $analysis_file = path($pool_dir, 'testresults', 'llm-failure-analysis.txt'); + ok -e $analysis_file, 'LLM analysis output file exists'; + like $analysis_file->slurp, qr/analyzing an automated test run/, 'LLM analysis output contains expected content'; + }; + subtest 'unclean shutdown' => sub { $bmwqemu::vars{PUBLISH_HDD_1} = 'publish_test.qcow2'; $command_handler->test_completed(1); diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/os-autoinst-5.1775053765.0f55e29/t/27-consoles-s3270.t new/os-autoinst-5.1775569433.3681116/t/27-consoles-s3270.t --- old/os-autoinst-5.1775053765.0f55e29/t/27-consoles-s3270.t 2026-04-01 16:29:25.000000000 +0200 +++ new/os-autoinst-5.1775569433.3681116/t/27-consoles-s3270.t 2026-04-07 15:43:53.000000000 +0200 @@ -148,6 +148,14 @@ subtest 'wait_output test' => sub { my $s3270_console_mock = Test::MockModule->new('consoles::s3270'); + $s3270_console_mock->redefine(send_3270 => {'command_status' => 'ok'}); + is $s3270_console->wait_output(), 1; + + for my $errormsg (('Wait: Timed out', 'Wait(): Timed out', 'Wait(Output): Timed out')) { + $s3270_console_mock->redefine(send_3270 => {'command_output' => [$errormsg], 'command_status' => 'any'}); + is $s3270_console->wait_output(), 0; + } + $s3270_console_mock->redefine(send_3270 => {'command_output' => ['None'], 'command_status' => 'any'}); warnings { throws_ok { $s3270_console->wait_output() } qr/has the s3270 wait timeout.*\n.*/, 'wait timeout failure expected' }; }; diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/os-autoinst-5.1775053765.0f55e29/t/34-git.t new/os-autoinst-5.1775569433.3681116/t/34-git.t --- old/os-autoinst-5.1775053765.0f55e29/t/34-git.t 2026-04-01 16:29:25.000000000 +0200 +++ new/os-autoinst-5.1775569433.3681116/t/34-git.t 2026-04-07 15:43:53.000000000 +0200 @@ -155,8 +155,9 @@ subtest 'clone default branch' => sub { $working_tree_dir->remove_tree; # ensure we actually clone the repo again my @clone_args = ($repo, $url, 1, '', $repo, '?', 1); + chomp(my $branch = qx{git -C $git_dir symbolic-ref --short HEAD}); combined_like { ok OpenQA::Isotovideo::Utils::clone_git(@clone_args), 'cloned repo with default branch' } - qr/master/, 'detected master branch'; + qr/$branch/, "detected $branch branch"; }; subtest 'index creation' => sub { @@ -205,7 +206,6 @@ git init >/dev/null 2>&1 && \ git config user.email "you\@example.com" >/dev/null && \ git config user.name "Your Name" >/dev/null && \ -git config init.defaultBranch main >/dev/null && \ git config commit.gpgsign false >/dev/null && \ touch README.md && \ git add README.md && \ diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/os-autoinst-5.1775053765.0f55e29/t/44-llm-failure-analysis.t new/os-autoinst-5.1775569433.3681116/t/44-llm-failure-analysis.t --- old/os-autoinst-5.1775053765.0f55e29/t/44-llm-failure-analysis.t 1970-01-01 01:00:00.000000000 +0100 +++ new/os-autoinst-5.1775569433.3681116/t/44-llm-failure-analysis.t 2026-04-07 15:43:53.000000000 +0200 @@ -0,0 +1,140 @@ +#!/usr/bin/env perl +# Copyright SUSE LLC +# SPDX-License-Identifier: GPL-2.0-or-later + +use Test::Most; +use Test::Warnings ':report_warnings'; +use Test::MockModule; +use Mojo::Base -signatures; +use Mojo::File qw(path tempdir); +use FindBin '$Bin'; +use lib "$FindBin::Bin/lib", "$Bin/..", "$Bin/../external/os-autoinst-common/lib"; +use OpenQA::Test::TimeLimit '10'; +use bmwqemu; +use OpenQA::Isotovideo::LLMAnalysis; + +my $tmpdir = tempdir; +chdir $tmpdir or die "Cannot chdir to $tmpdir: $!"; +$bmwqemu::result_dir = 'testresults'; + +sub setup_results (@results) { + my $testresults = path($bmwqemu::result_dir); + $testresults->make_path; + $testresults->list->each(sub ($f, $) { $f->remove }); + + my $i = 1; + for my $res (@results) { + $testresults->child("result-$i.json")->spurt(qq({"name":"test$i", "result":"$res"})); + $i++; + } +} + +subtest 'Context gathering and truncation' => sub { + setup_results('ok', 'fail'); + ok OpenQA::Isotovideo::LLMAnalysis::gather_context($bmwqemu::result_dir), 'Returns context when failures exist'; + + setup_results('ok'); + ok !OpenQA::Isotovideo::LLMAnalysis::gather_context($bmwqemu::result_dir), 'Skips when no failures'; + + # Missing names + setup_results('fail'); + path($bmwqemu::result_dir)->child('result-1.json')->spurt('{"result":"fail"}'); + my $ctx = OpenQA::Isotovideo::LLMAnalysis::gather_context($bmwqemu::result_dir); + is $ctx->{failed_tests}, '1', 'Fallback to filename for test name'; + + # Truncation logic + path($bmwqemu::result_dir)->child('autoinst-log.txt')->spurt(join "\n", map { "L$_" } 1 .. 300); + path($bmwqemu::result_dir)->child('serial0')->spurt(join "\n", map { "S$_" } 1 .. 150); + $ctx = OpenQA::Isotovideo::LLMAnalysis::gather_context($bmwqemu::result_dir); + is scalar(split "\n", $ctx->{log_tail}), 200, 'Log tail truncated'; + is scalar(split "\n", $ctx->{serial_tail}), 100, 'Serial tail truncated'; + + # Empty files + path($bmwqemu::result_dir)->child('autoinst-log.txt')->spurt(''); + path($bmwqemu::result_dir)->child('serial0')->spurt(''); + $ctx = OpenQA::Isotovideo::LLMAnalysis::gather_context($bmwqemu::result_dir); + is $ctx->{log_tail}, '', 'Handles empty log'; +}; + +subtest 'Prompt generation' => sub { + my $ctx = {distri => 'D', version => 'V', arch => 'A', failed_tests => 'F', log_tail => 'L', serial_tail => 'S'}; + my $prompt = OpenQA::Isotovideo::LLMAnalysis::build_prompt($ctx); + like $prompt, qr/D V A.*F.*L.*S/s, 'Prompt contains all context'; + + my $long = 'A' x 10000; + $ctx->{log_tail} = $long; + $ctx->{serial_tail} = $long; + $prompt = OpenQA::Isotovideo::LLMAnalysis::build_prompt($ctx); + ok length($prompt) < 17000, 'Prompt truncated'; + like $prompt, qr/sentences answering/s, 'Instructions preserved at end'; +}; + +subtest 'HTTP API mode' => sub { + my $mock_ua = Test::MockModule->new('Mojo::UserAgent'); + require Mojo::Transaction::HTTP; + require Mojo::Message::Response; + + my $test_api = sub ($res_body, $res_error = undef) { + $mock_ua->redefine(post => sub { + my $tx = Mojo::Transaction::HTTP->new; + $tx->res->code(200); + $tx->res->body($res_body) if defined $res_body; + $tx->res->error($res_error) if $res_error; + return $tx; + }); + return OpenQA::Isotovideo::LLMAnalysis::query_llm_api('p', 'u', 'm'); + }; + + is $test_api->('{"choices":[{"message":{"content":"OK"}}]}'), 'OK', 'Success path'; + like $test_api->(undef, {message => 'Fail', code => 500}), qr/status 500/, 'HTTP error'; + is $test_api->(undef, {message => 'Timeout'}), 'Error: Timeout', 'Connection error'; + is $test_api->('{"malformed":1}'), 'Error: Unexpected response format from LLM API.', 'Malformed JSON'; + is $test_api->('Not JSON'), 'Error: Unexpected response format from LLM API.', 'Non-JSON response'; + is $test_api->('{"choices":[]}'), 'Error: Unexpected response format from LLM API.', 'Empty choices'; +}; + +subtest 'CLI command mode' => sub { + my $mock_ipc = Test::MockModule->new('IPC::Run'); + my $test_cmd = sub ($out, $err, $exit_code, $die_msg = undef) { + $mock_ipc->redefine(run => sub ($cmd, $in, $out_ref, $err_ref, @rest) { + die $die_msg if $die_msg; + $$out_ref = $out; + $$err_ref = $err; + $? = $exit_code << 8; + return $exit_code == 0; + }); + return OpenQA::Isotovideo::LLMAnalysis::query_llm_cmd('p', 'c'); + }; + + is $test_cmd->('Out', '', 0), 'Out', 'Success path'; + like $test_cmd->('', '', 0, 'dead'), qr/Command failed - dead/, 'Command death'; + like $test_cmd->('Out', 'Err', 1), qr/exited with 256 - Err/, 'Non-zero exit with stderr'; + like $test_cmd->('Out', '', 1), qr/exited with 256 - Out/, 'Non-zero exit with stdout only'; + is $test_cmd->('', '', 0), 'Error: Command produced no output.', 'No output'; +}; + +subtest 'Execution routing' => sub { + my $mock_llm = Test::MockModule->new('OpenQA::Isotovideo::LLMAnalysis'); + my $mock_bmwqemu = Test::MockModule->new('bmwqemu'); + $mock_bmwqemu->noop('diag'); + + $mock_llm->redefine(gather_context => sub { return {distri => 'D'} }); + $mock_llm->redefine(build_prompt => sub { return 'P' }); + $mock_llm->redefine(query_llm_api => sub { return 'api' }); + $mock_llm->redefine(query_llm_cmd => sub { return 'cmd' }); + + delete $bmwqemu::vars{LLM_FAILURE_ANALYSIS_CMD}; + OpenQA::Isotovideo::LLMAnalysis::run($bmwqemu::result_dir); + is path($bmwqemu::result_dir)->child('llm-failure-analysis.txt')->slurp, 'api', 'Default to API'; + + $bmwqemu::vars{LLM_FAILURE_ANALYSIS_CMD} = 'c'; + OpenQA::Isotovideo::LLMAnalysis::run($bmwqemu::result_dir); + is path($bmwqemu::result_dir)->child('llm-failure-analysis.txt')->slurp, 'cmd', 'Route to CMD'; + + $mock_llm->redefine(gather_context => sub { return undef }); + $mock_bmwqemu->redefine(diag => sub { die 'No context should return' }); + ok !OpenQA::Isotovideo::LLMAnalysis::run($bmwqemu::result_dir), 'Early return if no context'; +}; + +done_testing; +chdir '/'; diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/os-autoinst-5.1775053765.0f55e29/t/99-full-stack.t new/os-autoinst-5.1775569433.3681116/t/99-full-stack.t --- old/os-autoinst-5.1775053765.0f55e29/t/99-full-stack.t 2026-04-01 16:29:25.000000000 +0200 +++ new/os-autoinst-5.1775569433.3681116/t/99-full-stack.t 2026-04-07 15:43:53.000000000 +0200 @@ -47,7 +47,7 @@ "VNC_CONNECT_TIMEOUT_LOCAL" : "0.001", "VNC_CONNECT_TIMEOUT_REMOTE" : "0.001", "NAME" : "00001-1-i386@32bit", - "TEST_NON_STRICT_MODULE": "1", + "TEST_NON_STRICT_MODULE": "1" } EOV # create screenshots ++++++ os-autoinst.obsinfo ++++++ --- /var/tmp/diff_new_pack.kIoPoH/_old 2026-04-08 17:18:34.714950413 +0200 +++ /var/tmp/diff_new_pack.kIoPoH/_new 2026-04-08 17:18:34.726950907 +0200 @@ -1,5 +1,5 @@ name: os-autoinst -version: 5.1775053765.0f55e29 -mtime: 1775053765 -commit: 0f55e2989cd92c694cd556092cbe999000fe15be +version: 5.1775569433.3681116 +mtime: 1775569433 +commit: 36811160762e2dc6a3d3564b0e6d9870094bffcf
