This is an automated email from the ASF dual-hosted git repository.
rombert pushed a commit to branch feat/llmao
in repository https://gitbox.apache.org/repos/asf/sling-tooling-jenkins.git
The following commit(s) were added to refs/heads/feat/llmao by this push:
new 986951e feat: validate result deterministically
986951e is described below
commit 986951e6e46d64d30f775573a82cb9cd8e932717
Author: Robert Munteanu <[email protected]>
AuthorDate: Wed Oct 14 09:57:42 2026 +0100
feat: validate result deterministically
---
README.md | 38 ++++----
.../org/apache/sling/jenkins/llmao/parent-pom.py | 101 +++++++++++++++++++++
.../org/apache/sling/jenkins/llmao/pi-events.mjs | 57 ++----------
.../org/apache/sling/jenkins/llmao/run-docker.sh | 19 +---
resources/org/apache/sling/jenkins/llmao/run-pi.sh | 8 +-
.../apache/sling/jenkins/llmao/run-verification.sh | 48 ++++++++++
.../org/apache/sling/jenkins/llmao/verify-build.sh | 28 ++++++
vars/slingOsgiBundleBuild.groovy | 69 +++++++++++---
8 files changed, 275 insertions(+), 93 deletions(-)
diff --git a/README.md b/README.md
index c7071bf..3995dc9 100644
--- a/README.md
+++ b/README.md
@@ -44,7 +44,7 @@ When the branch job related to the modified `Jenkinsfile` is
built, it will use
## Calling Apache LLMAO from Jenkins
-`slingOsgiBundleBuild(llmaoOnly: true)` runs a separate Pi automation job, not
the normal Maven matrix, SonarCloud analysis, deployment, or normal build
notifications. Pi can run Maven as part of verifying its requested changes.
Projects using `slingOsgiBundleBuild()` without this argument continue to build
as before; they do not call LLMAO.
+`slingOsgiBundleBuild(llmaoOnly: true)` runs a separate Pi parent-POM upgrade
job, not the normal Maven matrix, SonarCloud analysis, deployment, or normal
build notifications. Pi can run Maven as part of verifying its requested
changes; Jenkins independently checks the resulting parent version and runs
Maven afterward. Projects using `slingOsgiBundleBuild()` without this argument
continue to build as before; they do not call LLMAO.
The dedicated job's stages reflect the local agent workflow:
@@ -52,9 +52,12 @@ The dedicated job's stages reflect the local agent workflow:
2. **Configure Job** — apply scheduling, retention, and concurrency settings,
then read and validate the task input.
3. **Build Pi Image** — build or reuse the cached Docker image, without
binding credentials.
4. **Prepare Pi Task** — write the prompts, model settings, runtime script,
and bundled extensions outside the checkout.
-5. **Run Pi Task** — run the agent, including skill loading, LLMAO requests,
project edits, verification, and diff output. An unsuccessful task fails this
stage.
-6. **Publish Token Usage** — stop any remaining container and summarize and
archive observed usage, also after build or task failures.
-7. **Clean Up** — remove generated runtime inputs and temporary reports,
leaving project edits untouched.
+5. **Determine Parent Target** — parse the original parent and independently
select its latest numeric release from Maven Central. Preserve this expectation
in a read-only agent input.
+6. **Run Pi Task** — run the agent, including skill loading, LLMAO requests,
project edits, verification, and diff output. Process errors, incomplete
responses, and cancellation fail this stage; assistant wording is diagnostic
only. Skip Pi when the parent is already current.
+7. **Verify Parent Update** — require unchanged parent coordinates and the
independently selected target version.
+8. **Verify Updated Project** — run `mvn -B -ntp clean verify` in a fresh
container using the same pinned Java 21/Maven image, without Pi or the LLMAO
credential. Maven's actual exit status determines this stage's result.
+9. **Publish Token Usage** — stop any remaining containers and summarize and
archive observed usage, also after build, task, or verification failures.
+10. **Clean Up** — remove generated runtime inputs and temporary reports,
leaving project edits untouched.
### A separate scheduled job
@@ -73,7 +76,7 @@ slingOsgiBundleBuild(
llmaoOnly: true,
llmaoRebuildFrequency: '', // Manual only; switch to '@weekly' for
scheduling.
llmaoModel: 'gemma4-26b',
- llmaoInput: 'Update this project to the most recent released parent POM
version and respond only with SUCCESS or FAILURE'
+ llmaoInput: 'Update this project to the most recent released parent POM
version'
)
```
@@ -89,19 +92,19 @@ These settings can be provided as library arguments or
under `jenkins` in `.slin
| `llmaoInput` | Empty | Task prompt to send to the Pi agent. |
| `llmaoInputFile` | Empty | Workspace-relative UTF-8 prompt file to read
instead of `llmaoInput`. |
-Provide either an input string or an input file; the library has no default
task. The library allocates its own node and checks out the project;
`llmaoInputFile` must exist in that checked-out workspace. Input files are not
deleted. The checkout must be clean before Pi starts, so existing changes are
never silently discarded. Avoid exposing secrets to the agent, and check the
Models page for who can see prompts and source files.
+Provide either an input string or an input file describing the parent upgrade;
the library has no default task. Deterministic verification supports
`org.apache.sling:sling-parent` and `org.apache.sling:sling-bundle-parent` with
explicit numeric release versions. Missing/unsupported parents, unresolved
version properties, unavailable metadata, or metadata older than the current
parent fail before Pi starts. The library allocates its own node and checks out
the project; `llmaoInputFile` mu [...]
`llmaoMaxTokens` replaces Pi's default 16,384-token output allowance for this
custom model. A smaller allowance may help a request fit within the remaining
PAT quota, but does not reset or increase that quota or cap the total tokens
used by a multi-step job.
The library runs [Pi](https://pi.dev) in Docker, configured for LLMAO's
OpenAI-compatible endpoint at `https://llm.apache.org/v1`, with the selected
model and temperature `0.0`. Pi can read, search, edit, and write project files
and run shell commands. Automatic extensions, MCP, automatic project
instructions, stored sessions, and compaction remain disabled.
-A bundled output logger is explicitly loaded. It writes Pi's assistant text,
model-response stop reasons/errors, agent/request progress, tool starts and
completion status, and tool-result tails to stderr in the Jenkins console.
Tool-call summaries include shell commands, paths, and search patterns (up to
2,000 characters), but not file contents supplied to edit/write tools.
Assistant text and tool results are limited to their last 6,000 characters;
successful file/skill-read contents, pr [...]
+A bundled output logger is explicitly loaded. It writes Pi's assistant text,
model-response stop reasons/errors, agent/request progress, tool starts and
completion status, and tool-result tails to stderr in the Jenkins console.
Tool-call summaries include shell commands, paths, and search patterns (up to
2,000 characters), but not file contents supplied to edit/write tools.
Assistant text and tool results are limited to their last 6,000 characters;
successful file/skill-read contents, pr [...]
Pi retries transient provider/network errors, including HTTP 429 rate limits,
up to 10 times per failed model turn at one-minute intervals. Retries stay in
the same Pi session, preserving tool history and edits; provider-level retries
remain disabled. This polls for quota recovery rather than parsing LLMAO's
custom reset timestamp, and the existing job/process timeouts still apply.
Each run fetches `skills/` from the `master` branch of [Apache Sling
Whiteboard](https://github.com/apache/sling-whiteboard/tree/master/skills),
including references and scripts. All its skills are explicitly available to
Pi, including `update-sling-parent-pom`. The fetched Git revision is logged.
Skills live outside the project checkout and do not appear in the diff. The
parent-POM skill currently targets the latest supported release, not simply the
current version plus one.
-Pi is instructed to respond only with `SUCCESS` after completing and verifying
its update, or `FAILURE` otherwise, without staging, committing, pushing,
switching branches, or altering Git metadata. The wrapper maps an exact
`SUCCESS` and a successful Pi process to exit code `0`; `FAILURE`,
empty/malformed answers, or execution errors return a nonzero exit code and
fail the Jenkins job. The final response is kept separate from diff output, so
strings in the diff cannot be mistaken for an [...]
+Pi is instructed to summarize its changes and verification, without staging,
committing, pushing, switching branches, or altering Git metadata. It must use
the independently selected parent target, which it can read but cannot
overwrite. The job requires a normally completed Pi process, unchanged parent
coordinates with the target version, and a successful independent Maven build.
Neither `SUCCESS` nor `FAILURE` in assistant text or a diff decides the result.
If the parent is already cur [...]
The resulting Git diff is printed to stdout in the Jenkins build log,
including partial changes on failure and new, non-ignored files without staging
them. Project edits remain in the job workspace for inspection; no commit,
push, or automatic recovery is performed. Generated input/configuration files
are cleaned up, and sessions are not archived.
@@ -177,9 +180,9 @@ run_pi() {
The wrapper forwards `LLMAO_PAT` for the LLMAO provider. For another provider,
add only its required environment variable(s) as `--env NAME` options, or put
only that provider's credential in the mounted `auth.json`. The container's Pi
configuration and Maven home are temporary and are discarded when it exits.
-#### Exact non-interactive reproducer
+#### Agent-only non-interactive reproducer
-This follows the Jenkins prompt and flags: it selects the LLMAO model, uses
the same read/search/edit/write/shell tools, disables automatic
extensions/MCP/skills/prompts/context files, and explicitly loads the supplied
skills. Change the task text if reproducing a different update.
+This selects the LLMAO model, uses the same read/search/edit/write/shell
tools, disables automatic extensions/MCP/skills/prompts/context files, and
explicitly loads the supplied skills. Unlike the Jenkins job, it uses plain
print output and does not supply an independently selected parent target.
Change the task text if reproducing a different update.
```sh
run_pi \
@@ -187,11 +190,11 @@ run_pi \
--print --tools read,bash,edit,write,grep,find,ls --no-session \
--no-extensions --no-mcp --no-skills --no-prompt-templates \
--skill /skills --no-context-files --no-approve \
- --system-prompt 'You are a coding assistant working on the checked-out
project. Follow the relevant supplied Apache Sling skills; use
update-sling-parent-pom for upgrades to the most recent released parent POM
version. Complete the requested update and its required verification. Java 21
is the only available JDK; mvn and mvn21 both run Maven with Java 21. Do not
stage, unstage, commit, push, switch branches, or change Git metadata. Respond
with exactly SUCCESS if the update and verific [...]
- 'Update this project to the most recent released parent POM version and
respond only with SUCCESS or FAILURE'
+ --system-prompt 'You are a coding assistant working on the checked-out
project. Follow the relevant supplied Apache Sling skills; use
update-sling-parent-pom for upgrades to the most recent released parent POM
version. Complete the requested update and its required verification. Java 21
is the only available JDK; mvn and mvn21 both run Maven with Java 21. Do not
stage, unstage, commit, push, switch branches, or change Git metadata. Give a
concise summary of the changes and verification [...]
+ 'Update this project to the most recent released parent POM version'
```
-Pi prints its final response to stdout; inspect the resulting project changes
on the host with `git -C "$PROJECT_DIR" diff` and `git -C "$PROJECT_DIR" status
--short`. Git metadata is mounted read-only, so Pi cannot stage or commit
changes.
+This local print-mode command does not include Jenkins' independent target
selection and verification stages. Pi prints its final response to stdout;
inspect the resulting project changes on the host with `git -C "$PROJECT_DIR"
diff` and `git -C "$PROJECT_DIR" status --short`. Git metadata is mounted
read-only, so Pi cannot stage or commit changes.
#### Interactive Pi use
@@ -214,7 +217,7 @@ LLMAO builds that reach the image-build stage publish a
console summary, add the
A dedicated Pi extension collects gateway-reported usage across all model
responses, including tool-calling turns and any usage reported on failed or
retried turns. The report contains:
- `provider`, `model`, and Jenkins `build` metadata (`job`, `number`, and
`url`).
-- `result` (`SUCCESS`, `FAILURE`, or `INCOMPLETE` when no process exit code
was returned), `exitCode`, Pi `state`, and `aborted`.
+- `result` (`SUCCESS` only after independent verification, `FAILURE` for a
failed Pi/parent/build check, or `INCOMPLETE` when the checks did not finish),
`exitCode` (the failing check's exit code, or zero after all checks pass), Pi
`state`, and `aborted`.
- `requests` (provider request attempts), `responsesWithUsage`, and
`responsesWithoutUsage`. Counters are `null` and Pi state is `unknown` if the
reporter could not produce a file.
- `tokens`: `input` (uncached input), `output`, `cacheRead`, `cacheWrite`, and
`total`. Cached input is counted separately, not added to input twice;
reasoning tokens are already included in output.
- `usageStatus`: `complete` if Pi settled without interruption and every
response reported usage; `partial` if only some usage is available or the run
was interrupted; `unavailable` if no valid usage was reported. `tokens` is
`null` for unavailable usage, rather than a misleading zero. A gateway-reported
zero remains zero.
@@ -231,7 +234,7 @@ The library builds `sling-llmao-pi:1.1.0-jdk21-v3` from its
[Dockerfile](resourc
Java 21 is the only available JDK. Both `mvn` and the `mvn21` wrapper use it,
and Pi is told to use these commands. No `mvn8`, `mvn11`, or `mvn17` wrappers
are installed. `MAVEN_ARGS='-B -ntp'` applies batch mode and suppresses
download-progress output for both Maven commands, even when a skill invokes
`mvn clean verify` without those flags.
-Image building happens before credential binding. The container receives
`LLMAO_PAT` from the Jenkins environment, not as a token value in its command
line or configuration files. It runs as the Jenkins user's UID/GID with
writable project and usage-report mounts and read-only `.git` and input mounts.
Its private configuration, skills checkout, and Maven home/cache live in
container-local `/tmp`; host credentials and the Docker socket are not mounted.
The Pi process is bounded to 25 minu [...]
+Image building and target selection happen before credential binding. The Pi
container receives `LLMAO_PAT` from the Jenkins environment, not as a token
value in its command line or configuration files. It runs as the Jenkins user's
UID/GID with writable project and usage-report mounts and read-only `.git` and
input mounts. Its private configuration, skills checkout, and Maven home/cache
live in container-local `/tmp`; host credentials and the Docker socket are not
mounted. Verification [...]
### Testing
@@ -241,6 +244,8 @@ Run offline checks from the repository root:
groovy tests/llmao.groovy
node tests/tool-output-logger.mjs
node tests/usage-reporter.mjs
+node tests/pi-events.mjs
+python3 tests/verification.py
python3 tests/pi-docker.py
```
@@ -249,6 +254,7 @@ These stub Jenkins steps and Docker/Pi output; they require
no credentials or LL
```sh
docker build --tag sling-llmao-pi:1.1.0-jdk21-v3
resources/org/apache/sling/jenkins/llmao
python3 tests/pi-docker.py --integration
+python3 tests/verification.py --integration
```
-The integration test exercises skill discovery/loading, real
read/edit/write/find/shell tool calls, assistant/progress/tool console output
with private read contents, Java 21 with both Maven commands and the absence of
older JDKs/wrappers, Maven batch-mode settings, environment-token
authentication, model selection, tracked/new-file diffs, SUCCESS/FAILURE
mapping, malformed/empty answers, API errors, multi-turn token/cache totals,
retries, missing/zero usage, cancellation, and container [...]
+The offline tests cover latest-release selection, XML namespaces,
unchanged/wrong parent versions and coordinates, already-current parents,
credential-free verification mounts and exit codes, lifecycle failures,
independent-stage failures, usage publishing, and cleanup. The Pi integration
test exercises skill discovery/loading, real read/edit/write/find/shell tool
calls, assistant/progress/tool console output with private read contents, Java
21 with both Maven commands and the absence of [...]
diff --git a/resources/org/apache/sling/jenkins/llmao/parent-pom.py
b/resources/org/apache/sling/jenkins/llmao/parent-pom.py
new file mode 100644
index 0000000..99360a1
--- /dev/null
+++ b/resources/org/apache/sling/jenkins/llmao/parent-pom.py
@@ -0,0 +1,101 @@
+#!/usr/bin/env python3
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""Select and check a released Sling parent without executing project code."""
+
+import json
+from pathlib import Path
+import re
+import sys
+from urllib.request import urlopen
+import xml.etree.ElementTree as ET
+
+
+def read_parent(path):
+ root = ET.parse(path).getroot()
+ if root.tag == "{http://maven.apache.org/POM/4.0.0}project":
+ namespace = "{http://maven.apache.org/POM/4.0.0}";
+ elif root.tag == "project":
+ namespace = ""
+ else:
+ raise ValueError("Expected a Maven project document")
+ parent = root.find(f"{namespace}parent")
+ if parent is None:
+ raise ValueError("pom.xml has no direct parent")
+ result = {key: (parent.findtext(f"{namespace}{key}") or "").strip()
+ for key in ("groupId", "artifactId", "version")}
+ if result["groupId"] != "org.apache.sling" or result["artifactId"] not in (
+ "sling-parent", "sling-bundle-parent"):
+ raise ValueError("Expected org.apache.sling:sling-parent or
sling-bundle-parent")
+ if not re.fullmatch(r"[0-9]+", result["version"]):
+ raise ValueError("The Sling parent must have an explicit numeric
release version")
+ return result
+
+
+def select_target(parent, metadata):
+ # Sling parent release versions are integers. Ignore snapshots and other
+ # qualifiers rather than relying on lexical order or metadata's <latest>.
+ versions = [int(version.text.strip())
+ for version in
ET.fromstring(metadata).findall("./versioning/versions/version")
+ if version.text and re.fullmatch(r"[0-9]+",
version.text.strip())]
+ if not versions:
+ raise ValueError("Maven Central metadata contains no released Sling
parent versions")
+ target = str(max(versions))
+ if int(target) < int(parent["version"]):
+ raise ValueError("Maven Central's latest release is older than the
current parent")
+ return dict(parent, targetVersion=target, updateRequired=target !=
parent["version"])
+
+
+def check_parent(parent, expected):
+ for key in ("groupId", "artifactId"):
+ if parent[key] != expected[key]:
+ raise ValueError(f"Parent {key} changed: expected {expected[key]},
found {parent[key]}")
+ if parent["version"] != expected["targetVersion"]:
+ raise ValueError(f"Expected parent version
{expected['targetVersion']}, found {parent['version']} "
+ f"(original version {expected['version']})")
+ if expected["updateRequired"] and parent["version"] == expected["version"]:
+ raise ValueError("The parent version was not updated")
+
+
+def main():
+ action = sys.argv[1]
+ parent = read_parent("/workspace/pom.xml")
+ if action == "prepare-parent":
+ url = ("https://repo.maven.apache.org/maven2/org/apache/sling/";
+ f"{parent['artifactId']}/maven-metadata.xml")
+ with urlopen(url, timeout=60) as response:
+ expected = select_target(parent, response.read())
+ print(json.dumps(expected))
+ elif action == "check-parent":
+ expected =
json.loads(Path("/inputs/expected-parent.json").read_text(encoding="utf-8"))
+ check_parent(parent, expected)
+ if expected["updateRequired"]:
+ print(f"Parent update verified:
{parent['groupId']}:{parent['artifactId']} "
+ f"{expected['version']} -> {parent['version']}")
+ else:
+ print(f"No parent update needed: already at released version
{parent['version']}")
+ else:
+ raise ValueError(f"Unknown parent check: {action}")
+
+
+if __name__ == "__main__":
+ try:
+ main()
+ except Exception as error:
+ print(f"Parent POM check failed: {error}", file=sys.stderr)
+ sys.exit(1)
diff --git a/resources/org/apache/sling/jenkins/llmao/pi-events.mjs
b/resources/org/apache/sling/jenkins/llmao/pi-events.mjs
index b463718..dd057c2 100644
--- a/resources/org/apache/sling/jenkins/llmao/pi-events.mjs
+++ b/resources/org/apache/sling/jenkins/llmao/pi-events.mjs
@@ -20,8 +20,8 @@
import { spawn } from 'node:child_process';
import fs from 'node:fs';
-// Consume the JSONL protocol continuously. Progress belongs on stderr; only
-// the final SUCCESS/FAILURE decision belongs on stdout for run-docker.sh.
+// Consume JSONL lifecycle events without interpreting assistant prose.
+// The bundled output logger handles progress and tool/assistant diagnostics.
const secret = process.env.LLMAO_PAT;
function sanitize(value, limit = 6000) {
let text = String(value ?? '');
@@ -32,22 +32,16 @@ function sanitize(value, limit = 6000) {
function log(value, limit) {
process.stderr.write(`[Pi] ${sanitize(value, limit)}\n`);
}
-function textBlocks(content) {
- if (typeof content === 'string') return content;
- return (Array.isArray(content) ? content : []).filter(block => block?.type
=== 'text').map(block => block.text).join('\n');
-}
-
let lastAssistant;
let settled = false;
let aborted = false;
let protocolError = false;
let quotaReported = false;
-const tools = new Map();
function diagnostic(text) {
if (!quotaReported && /\b429\b|rate limit exceeded/i.test(text)) {
quotaReported = true;
- log('LLMAO quota/rate limit rejected the request. Wait for the
reported reset or request a higher limit; retries are disabled.');
+ log('LLMAO quota/rate limit rejected the request. Pi retry settings
and the job timeout still apply.');
}
log(text);
}
@@ -56,52 +50,25 @@ function event(record) {
switch (record.type) {
case 'agent_start':
settled = false;
- log('Agent started');
- break;
- case 'turn_start':
- log('Requesting the next model response');
- break;
- case 'tool_execution_start': {
- tools.set(record.toolCallId, record.toolName);
- const detail = record.toolName === 'bash' ? record.args?.command :
record.args?.path ?? record.args?.pattern ?? '';
- log(`Tool ${record.toolName}: ${detail}`, 2000);
+ aborted = false;
+ lastAssistant = undefined;
break;
- }
- case 'tool_execution_end': {
- const name = record.toolName ?? tools.get(record.toolCallId) ??
'unknown';
- log(`Tool ${name}: ${record.isError ? 'ERROR' :
'OK'}${record.durationMs === undefined ? '' : ` (${record.durationMs} ms)`}`);
- // Do not dump source/skills/prompts. Shell output and tool errors
- // carry the build diagnostics; keep their tail bounded and
redacted.
- if (name === 'bash' || record.isError) {
- const output = textBlocks(record.result?.content);
- if (output) diagnostic(output);
- }
- tools.delete(record.toolCallId);
- break;
- }
case 'message_end': {
const message = record.message;
if (message?.role !== 'assistant') break;
lastAssistant = message;
- if (message.usage) {
- const usage = message.usage;
- log(`Tokens: input=${usage.input ?? 0}, output=${usage.output
?? 0}, cacheRead=${usage.cacheRead ?? 0}, cacheWrite=${usage.cacheWrite ?? 0}`);
- }
- log(`Model response ended: ${message.stopReason ?? 'unknown'}`);
if (message.errorMessage) diagnostic(message.errorMessage);
- if (message.stopReason === 'length') log('Model output was
truncated. Increase llmaoMaxTokens only if the gateway quota allows it.');
break;
}
case 'agent_settled':
settled = true;
aborted = Boolean(record.aborted);
- log(aborted ? 'Agent aborted' : 'Agent settled');
break;
}
}
// Split only on LF, not Unicode line separators (which can occur in JSON).
-function lines(stream, consume) {
+function lines(stream, consume, keepEmpty = false) {
let pending = '';
stream.setEncoding('utf8');
stream.on('data', chunk => {
@@ -110,7 +77,7 @@ function lines(stream, consume) {
while ((newline = pending.indexOf('\n')) !== -1) {
const line = pending.slice(0, newline).replace(/\r$/, '');
pending = pending.slice(newline + 1);
- if (line) consume(line);
+ if (line || keepEmpty) consume(line);
}
});
stream.on('end', () => { if (pending) consume(pending); });
@@ -134,20 +101,16 @@ lines(child.stdout, line => {
log('Invalid Pi JSON event; the run will fail.');
}
});
-lines(child.stderr, diagnostic);
+lines(child.stderr, text => process.stderr.write(`${sanitize(text)}\n`), true);
child.on('error', error => {
diagnostic(`Could not start Pi: ${error.message}`);
protocolError = true;
});
child.on('close', (code, signal) => {
if (signal) log(`Pi terminated by ${signal}`);
- const answer = textBlocks(lastAssistant?.content).trim();
const hasToolCalls = Array.isArray(lastAssistant?.content) &&
lastAssistant.content.some(block => block?.type === 'toolCall');
const complete = code === 0 && !signal && !protocolError && settled &&
!aborted &&
- lastAssistant?.stopReason === 'stop' && !hasToolCalls;
- const success = complete && answer === 'SUCCESS';
- if (complete && answer !== 'SUCCESS' && answer !== 'FAILURE') log('Final
model response was not exactly SUCCESS or FAILURE.');
+ lastAssistant?.stopReason === 'stop' && !lastAssistant?.errorMessage
&& !hasToolCalls;
if (!complete) log('Pi did not complete a successful, settled model
response.');
- process.stdout.write(`${success ? 'SUCCESS' : 'FAILURE'}\n`);
- process.exitCode = success ? 0 : (code && code > 0 ? code : 1);
+ process.exitCode = complete ? 0 : (code && code > 0 ? code : 1);
});
diff --git a/resources/org/apache/sling/jenkins/llmao/run-docker.sh
b/resources/org/apache/sling/jenkins/llmao/run-docker.sh
index 391dbf9..a848301 100644
--- a/resources/org/apache/sling/jenkins/llmao/run-docker.sh
+++ b/resources/org/apache/sling/jenkins/llmao/run-docker.sh
@@ -32,7 +32,7 @@ fi
status=0
mkdir -p "$LLMAO_REPORT_DIR"
-result=$(docker run --rm --init \
+docker run --rm --init \
--name "$LLMAO_CONTAINER" \
--user "$(id -u):$(id -g)" \
--read-only --tmpfs /tmp:rw,nosuid,nodev,mode=1777 \
@@ -42,21 +42,10 @@ result=$(docker run --rm --init \
--mount "type=bind,source=$LLMAO_WORKSPACE,target=/workspace" \
--mount
"type=bind,source=$LLMAO_WORKSPACE/.git,target=/workspace/.git,readonly" \
--workdir /workspace \
- "$LLMAO_IMAGE" sh /inputs/run-pi.sh) || status=$?
+ "$LLMAO_IMAGE" sh /inputs/run-pi.sh || status=$?
-result=$(printf '%s' "$result" | sed 's/^[[:space:]]*//;s/[[:space:]]*$//')
-if [ "$status" -eq 0 ] && [ "$result" = SUCCESS ]; then
- echo SUCCESS
-else
- echo FAILURE
- if [ "$status" -eq 0 ]; then
- if [ "$result" != FAILURE ]; then
- echo 'Pi did not return exactly SUCCESS or FAILURE.' >&2
- printf 'Unexpected Pi response: <%s>\n' "$result" >&2
- fi
- status=1
- fi
-fi
+# Assistant wording is diagnostic only. Parent and Maven checks in later
+# stages decide whether the requested update actually succeeded.
# Report partial edits on failure as well. Include new files without staging
# them, and disable external diff drivers/text converters from the checkout.
diff --git a/resources/org/apache/sling/jenkins/llmao/run-pi.sh
b/resources/org/apache/sling/jenkins/llmao/run-pi.sh
index db8a1f4..a4dd3fa 100644
--- a/resources/org/apache/sling/jenkins/llmao/run-pi.sh
+++ b/resources/org/apache/sling/jenkins/llmao/run-pi.sh
@@ -40,11 +40,11 @@ git -C "$HOME/sling-whiteboard" sparse-checkout set skills
>&2
printf 'Sling skills revision: ' >&2
git -C "$HOME/sling-whiteboard" rev-parse HEAD >&2
-# Print mode keeps the final assistant text on stdout for the result check.
-# The output logger sends assistant/progress/tool diagnostics to stderr.
-exec timeout --kill-after=5s 1500s pi \
+# Consume structured lifecycle events to detect incomplete/API-error runs,
+# independently of the final assistant's wording. The logger uses stderr.
+exec timeout --kill-after=5s 1500s node /inputs/pi-events.mjs \
--provider llmao --model "$LLMAO_MODEL" \
- --print --tools read,bash,edit,write,grep,find,ls --no-session \
+ --print --mode json --tools read,bash,edit,write,grep,find,ls --no-session
\
--no-extensions --no-mcp --no-skills --no-prompt-templates \
--extension /inputs/tool-output-logger.mjs \
--extension /inputs/usage-reporter.mjs \
diff --git a/resources/org/apache/sling/jenkins/llmao/run-verification.sh
b/resources/org/apache/sling/jenkins/llmao/run-verification.sh
new file mode 100644
index 0000000..d80b8ac
--- /dev/null
+++ b/resources/org/apache/sling/jenkins/llmao/run-verification.sh
@@ -0,0 +1,48 @@
+#!/bin/bash
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+set +x
+set -eu
+export GIT_OPTIONAL_LOCKS=0
+
+if [ "$LLMAO_CHECK" = prepare-parent ]; then
+ checkout_status=$(git -C "$LLMAO_WORKSPACE" status --porcelain)
+ if [ -n "$checkout_status" ]; then
+ echo 'The project checkout must be clean before selecting the parent
target.' >&2
+ exit 1
+ fi
+fi
+
+case "$LLMAO_CHECK" in
+ prepare-parent|check-parent) command=(python3 /inputs/parent-pom.py
"$LLMAO_CHECK") ;;
+ verify-build) command=(sh /inputs/verify-build.sh) ;;
+ *) echo 'Unknown verification action.' >&2; exit 1 ;;
+esac
+
+# Use the same pinned image, but a new process/home with no agent or PAT.
+container="${LLMAO_CONTAINER}-verify"
+trap 'docker rm --force "$container" >/dev/null 2>&1 || true' EXIT HUP INT TERM
+docker run --rm --init \
+ --name "$container" \
+ --user "$(id -u):$(id -g)" \
+ --read-only --tmpfs /tmp:rw,nosuid,nodev,mode=1777 \
+ --mount "type=bind,source=$(pwd)/inputs,target=/inputs,readonly" \
+ --mount "type=bind,source=$LLMAO_WORKSPACE,target=/workspace" \
+ --mount
"type=bind,source=$LLMAO_WORKSPACE/.git,target=/workspace/.git,readonly" \
+ --workdir /workspace \
+ "$LLMAO_IMAGE" "${command[@]}"
diff --git a/resources/org/apache/sling/jenkins/llmao/verify-build.sh
b/resources/org/apache/sling/jenkins/llmao/verify-build.sh
new file mode 100644
index 0000000..8140245
--- /dev/null
+++ b/resources/org/apache/sling/jenkins/llmao/verify-build.sh
@@ -0,0 +1,28 @@
+#!/bin/sh
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+set -eu
+export HOME=/tmp/verify-home
+export MAVEN_CONFIG="$HOME/.m2"
+export MAVEN_OPTS="-Duser.home=$HOME"
+export MAVEN_ARGS=''
+export GIT_OPTIONAL_LOCKS=0
+mkdir -p "$MAVEN_CONFIG"
+git config --global --add safe.directory /workspace
+# No pipes: Jenkins receives Maven's actual exit status and full diagnostics.
+exec mvn -B -ntp clean verify
diff --git a/vars/slingOsgiBundleBuild.groovy b/vars/slingOsgiBundleBuild.groovy
index 3bd83f4..9c3e2a3 100644
--- a/vars/slingOsgiBundleBuild.groovy
+++ b/vars/slingOsgiBundleBuild.groovy
@@ -154,7 +154,7 @@ def runLlmaoOnlyBuild(def globalConfig, Map params) {
]
node(globalConfig.mainNodeLabel) {
- timeout(time: 30, unit: 'MINUTES') {
+ timeout(time: 45, unit: 'MINUTES') {
stage('Checkout and Load Configuration') {
checkout scm
if ( fileExists('.sling-module.json') ) {
@@ -223,7 +223,10 @@ def runLlmaoOnlyBuild(def globalConfig, Map params) {
writeFile file: 'run-pi.sh', text:
libraryResource('org/apache/sling/jenkins/llmao/run-pi.sh')
writeFile file: 'tool-output-logger.mjs',
text: libraryResource('org/apache/sling/jenkins/llmao/tool-output-logger.mjs')
writeFile file: 'usage-reporter.mjs', text:
libraryResource('org/apache/sling/jenkins/llmao/usage-reporter.mjs')
- writeFile file: 'system-prompt.txt', text:
'You are a coding assistant working on the checked-out project. Follow the
relevant supplied Apache Sling skills; use update-sling-parent-pom for upgrades
to the most recent released parent POM version. Complete the requested update
and its required verification. Java 21 is the only available JDK; mvn and mvn21
both run Maven with Java 21. Do not stage, unstage, commit, push, switch
branches, or change Git metadat [...]
+ writeFile file: 'pi-events.mjs', text:
libraryResource('org/apache/sling/jenkins/llmao/pi-events.mjs')
+ writeFile file: 'parent-pom.py', text:
libraryResource('org/apache/sling/jenkins/llmao/parent-pom.py')
+ writeFile file: 'verify-build.sh', text:
libraryResource('org/apache/sling/jenkins/llmao/verify-build.sh')
+ writeFile file: 'system-prompt.txt', text:
'You are a coding assistant working on the checked-out project. Follow the
relevant supplied Apache Sling skills; use update-sling-parent-pom for upgrades
to the most recent released parent POM version. Read
/inputs/expected-parent.json for the independently selected parent coordinates
and targetVersion; keep the parent coordinates unchanged and upgrade to that
target version. Complete the requested update and its [...]
writeFile file: 'prompt_input.txt', text:
contextData, encoding: 'UTF-8'
writeJSON file: 'models.json', json: [
providers: [llmao: [
@@ -252,18 +255,60 @@ def runLlmaoOnlyBuild(def globalConfig, Map params) {
]
}
}
+ def expectedParent
+ stage('Determine Parent Target') {
+ withEnv(['LLMAO_CHECK=prepare-parent']) {
+ expectedParent = readJSON(text: sh(
+ script:
libraryResource('org/apache/sling/jenkins/llmao/run-verification.sh'),
+ returnStdout: true
+ ).trim(), returnPojo: true)
+ }
+ // Inputs are mounted read-only for Pi; its
writable reports
+ // cannot change the original parent or selected
target.
+ writeJSON file: 'inputs/expected-parent.json',
json: expectedParent
+ echo "Parent target:
${expectedParent.groupId}:${expectedParent.artifactId}
${expectedParent.version} -> ${expectedParent.targetVersion}"
+ }
stage('Run Pi Task') {
- withCredentials([string(credentialsId:
'llmao-pat', variable: 'LLMAO_PAT')]) {
- updateStatus = sh(
- script:
libraryResource('org/apache/sling/jenkins/llmao/run-docker.sh'),
+ if ( expectedParent.updateRequired ) {
+ def piStatus
+ withCredentials([string(credentialsId:
'llmao-pat', variable: 'LLMAO_PAT')]) {
+ piStatus = sh(
+ script:
libraryResource('org/apache/sling/jenkins/llmao/run-docker.sh'),
+ returnStatus: true
+ )
+ }
+ if ( piStatus != 0 ) {
+ updateStatus = piStatus
+ error("Pi did not complete normally (exit
code ${piStatus}).")
+ }
+ echo 'Pi completed normally; independently
verifying the update.'
+ } else {
+ echo 'No parent update needed; skipping Pi and
verifying the existing project.'
+ }
+ }
+ stage('Verify Parent Update') {
+ def parentStatus
+ withEnv(['LLMAO_CHECK=check-parent']) {
+ parentStatus = sh(
+ script:
libraryResource('org/apache/sling/jenkins/llmao/run-verification.sh'),
returnStatus: true
)
}
- // Attribute update failures to the task, not a
later response stage.
- if ( updateStatus == 0 ) {
- echo 'Pi completed the project update
successfully.'
- } else {
- error("Pi could not complete the project
update (exit code ${updateStatus}).")
+ if ( parentStatus != 0 ) {
+ updateStatus = parentStatus
+ error("Parent POM update verification failed
(exit code ${parentStatus}).")
+ }
+ }
+ stage('Verify Updated Project') {
+ timeout(time: 15, unit: 'MINUTES') {
+ withEnv(['LLMAO_CHECK=verify-build']) {
+ updateStatus = sh(
+ script:
libraryResource('org/apache/sling/jenkins/llmao/run-verification.sh'),
+ returnStatus: true
+ )
+ }
+ if ( updateStatus != 0 )
+ error("Independent mvn clean verify failed
(exit code ${updateStatus}).")
}
}
} finally {
@@ -273,6 +318,7 @@ def runLlmaoOnlyBuild(def globalConfig, Map params) {
sh '''
set +x
docker rm --force "$LLMAO_CONTAINER"
>/dev/null 2>&1 || true
+ docker rm --force
"${LLMAO_CONTAINER}-verify" >/dev/null 2>&1 || true
'''
dir(reportDirectory) {
publishLlmaoUsage(llmaoConfig.llmaoModel,
updateStatus)
@@ -284,7 +330,8 @@ def runLlmaoOnlyBuild(def globalConfig, Map params) {
set +x
rm -f image/Dockerfile inputs/models.json
inputs/settings.json \\
inputs/system-prompt.txt
inputs/prompt_input.txt inputs/run-pi.sh \\
- inputs/tool-output-logger.mjs
inputs/usage-reporter.mjs
+ inputs/tool-output-logger.mjs
inputs/usage-reporter.mjs inputs/pi-events.mjs \\
+ inputs/parent-pom.py
inputs/expected-parent.json inputs/verify-build.sh
rm -f "$LLMAO_REPORT_DIR/llmao-usage.json"
"$LLMAO_REPORT_DIR/llmao-usage.json.tmp"
rmdir "$LLMAO_REPORT_DIR" >/dev/null 2>&1
|| true
'''
