This is an automated email from the ASF dual-hosted git repository.

rombert pushed a commit to branch feat/llmao
in repository https://gitbox.apache.org/repos/asf/sling-tooling-jenkins.git

commit 194020caa20779ebb10a34b9958b5ef2177a6693
Author: Robert Munteanu <[email protected]>
AuthorDate: Mon Oct 12 19:33:11 2026 +0100

    feat: token usage reporting
---
 README.md                                          |  22 ++-
 .../org/apache/sling/jenkins/llmao/pi-events.mjs   | 153 +++++++++++++++++++++
 .../org/apache/sling/jenkins/llmao/run-docker.sh   |   2 +
 resources/org/apache/sling/jenkins/llmao/run-pi.sh |   2 +
 .../apache/sling/jenkins/llmao/usage-reporter.mjs  | 121 ++++++++++++++++
 vars/slingOsgiBundleBuild.groovy                   |  57 ++++++--
 6 files changed, 345 insertions(+), 12 deletions(-)

diff --git a/README.md b/README.md
index d08a842..70bdcfd 100644
--- a/README.md
+++ b/README.md
@@ -95,6 +95,22 @@ Pi is instructed to respond only with `SUCCESS` after 
completing and verifying i
 
 The resulting Git diff is printed to stdout in the Jenkins build log, 
including partial changes on failure and new, non-ignored files without staging 
them. Project edits remain in the job workspace for inspection; no commit, 
push, or automatic recovery is performed. Generated input/configuration files 
are cleaned up, and sessions are not archived.
 
+### Per-build token usage
+
+LLMAO builds that reach the submission stage publish a console summary, add 
the observed total to their build description (for example, `LLMAO: SUCCESS | 
18420 tokens`), and archive `llmao-usage.json`. Normal non-LLMAO builds are 
unaffected. No model prices or monetary cost estimates are configured.
+
+A dedicated Pi extension collects gateway-reported usage across all model 
responses, including tool-calling turns and any usage reported on failed or 
retried turns. The report contains:
+
+- `provider`, `model`, and Jenkins `build` metadata (`job`, `number`, and 
`url`).
+- `result` (`SUCCESS`, `FAILURE`, or `INCOMPLETE` when no process exit code 
was returned), `exitCode`, Pi `state`, and `aborted`.
+- `requests` (provider request attempts), `responsesWithUsage`, and 
`responsesWithoutUsage`. Counters are `null` and Pi state is `unknown` if the 
reporter could not produce a file.
+- `tokens`: `input` (uncached input), `output`, `cacheRead`, `cacheWrite`, and 
`total`. Cached input is counted separately, not added to input twice; 
reasoning tokens are already included in output.
+- `usageStatus`: `complete` if Pi settled without interruption and every 
response reported usage; `partial` if only some usage is available or the run 
was interrupted; `unavailable` if no valid usage was reported. `tokens` is 
`null` for unavailable usage, rather than a misleading zero. A gateway-reported 
zero remains zero.
+
+Usage is checkpointed atomically in a unique output directory outside the 
project checkout after each response. Jenkins stops the container before 
publishing and archiving the report, then cleans up the temporary report files. 
Reports are published on success and failure, and on cancellation where the 
Jenkins agent remains available. They contain counts and metadata only, not 
prompts, source content, reasoning, credentials, or session history.
+
+These are **observed token totals**, not billing records: interrupted requests 
and gateway errors may not return usage, so partial totals can undercount 
actual consumption. Failed requests without usage are not guessed to be free. 
Artifacts follow the job's existing ten-build retention policy; download the 
JSON reports before expiry if you need longer-term cost tracking.
+
 ### Docker runtime
 
 The `ubuntu` agent needs Bash, Git, and access to a **local Docker daemon**. 
The daemon must see the project and temporary workspaces at the same absolute 
paths for bind mounts; containerized executors must share those paths with the 
Docker host. The job also requires the Jenkins Pipeline Utility Steps plugin 
(`writeJSON`/`readJSON`). No host Node.js, Java, Maven, Pi installation, or 
Jenkins Docker Pipeline plugin is required.
@@ -103,7 +119,7 @@ The library builds `sling-llmao-pi:1.1.0-jdks8-21-v2` from 
its [Dockerfile](reso
 
 Java 21 remains the default for `java` and `mvn`. The `mvn8`, `mvn11`, 
`mvn17`, and `mvn21` wrappers select the corresponding JDK through `JAVA_HOME` 
and `PATH` for that invocation only. Pi is told which wrappers are available 
and decides which JDK to use. `MAVEN_ARGS='-B -ntp'` applies batch mode and 
suppresses download-progress output for Maven and all wrappers, even when a 
skill invokes `mvn clean verify` without those flags.
 
-Image building happens before credential binding. The container receives 
`LLMAO_PAT` from the Jenkins environment, not as a token value in its command 
line or configuration files. It runs as the Jenkins user's UID/GID with a 
writable project mount and read-only `.git` and input mounts. Its private 
configuration, skills checkout, and Maven home/cache live in container-local 
`/tmp`; host credentials and the Docker socket are not mounted. The Pi process 
is bounded to 25 minutes within a 30- [...]
+Image building happens before credential binding. The container receives 
`LLMAO_PAT` from the Jenkins environment, not as a token value in its command 
line or configuration files. It runs as the Jenkins user's UID/GID with 
writable project and usage-report mounts and read-only `.git` and input mounts. 
Its private configuration, skills checkout, and Maven home/cache live in 
container-local `/tmp`; host credentials and the Docker socket are not mounted. 
The Pi process is bounded to 25 minu [...]
 
 ### Testing
 
@@ -111,6 +127,8 @@ Run offline checks from the repository root:
 
 ```sh
 groovy tests/llmao.groovy
+node tests/tool-output-logger.mjs
+node tests/usage-reporter.mjs
 python3 tests/pi-docker.py
 ```
 
@@ -121,4 +139,4 @@ docker build --tag sling-llmao-pi:1.1.0-jdks8-21-v2 
resources/org/apache/sling/j
 python3 tests/pi-docker.py --integration
 ```
 
-The integration test exercises skill discovery/loading, real 
read/edit/write/find/shell tool calls, all four Maven/JDK wrappers, Maven 
batch-mode settings, environment-token authentication, model selection, 
tracked/new-file diffs, SUCCESS/FAILURE mapping, malformed/empty answers, API 
errors, cancellation, and container removal. It uses only disposable fixture 
projects and does not call LLMAO or update the real project. A real Jenkins run 
is still required to validate gateway access and t [...]
+The integration test exercises skill discovery/loading, real 
read/edit/write/find/shell tool calls, all four Maven/JDK wrappers, Maven 
batch-mode settings, environment-token authentication, model selection, 
tracked/new-file diffs, SUCCESS/FAILURE mapping, malformed/empty answers, API 
errors, multi-turn token/cache totals, retries, missing/zero usage, 
cancellation, and container removal. It uses only disposable fixture projects 
and does not call LLMAO or update the real project. A real Je [...]
diff --git a/resources/org/apache/sling/jenkins/llmao/pi-events.mjs 
b/resources/org/apache/sling/jenkins/llmao/pi-events.mjs
new file mode 100644
index 0000000..b463718
--- /dev/null
+++ b/resources/org/apache/sling/jenkins/llmao/pi-events.mjs
@@ -0,0 +1,153 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import { spawn } from 'node:child_process';
+import fs from 'node:fs';
+
+// Consume the JSONL protocol continuously. Progress belongs on stderr; only
+// the final SUCCESS/FAILURE decision belongs on stdout for run-docker.sh.
+const secret = process.env.LLMAO_PAT;
+function sanitize(value, limit = 6000) {
+    let text = String(value ?? '');
+    if (secret) text = text.split(secret).join('[REDACTED]');
+    text = text.replace(/\x1b\[[0-9;]*[a-zA-Z]/g, '');
+    return text.length > limit ? `[...truncated; last ${limit} 
characters...]\n${text.slice(-limit)}` : text;
+}
+function log(value, limit) {
+    process.stderr.write(`[Pi] ${sanitize(value, limit)}\n`);
+}
+function textBlocks(content) {
+    if (typeof content === 'string') return content;
+    return (Array.isArray(content) ? content : []).filter(block => block?.type 
=== 'text').map(block => block.text).join('\n');
+}
+
+let lastAssistant;
+let settled = false;
+let aborted = false;
+let protocolError = false;
+let quotaReported = false;
+const tools = new Map();
+
+function diagnostic(text) {
+    if (!quotaReported && /\b429\b|rate limit exceeded/i.test(text)) {
+        quotaReported = true;
+        log('LLMAO quota/rate limit rejected the request. Wait for the 
reported reset or request a higher limit; retries are disabled.');
+    }
+    log(text);
+}
+
+function event(record) {
+    switch (record.type) {
+        case 'agent_start':
+            settled = false;
+            log('Agent started');
+            break;
+        case 'turn_start':
+            log('Requesting the next model response');
+            break;
+        case 'tool_execution_start': {
+            tools.set(record.toolCallId, record.toolName);
+            const detail = record.toolName === 'bash' ? record.args?.command : 
record.args?.path ?? record.args?.pattern ?? '';
+            log(`Tool ${record.toolName}: ${detail}`, 2000);
+            break;
+        }
+        case 'tool_execution_end': {
+            const name = record.toolName ?? tools.get(record.toolCallId) ?? 
'unknown';
+            log(`Tool ${name}: ${record.isError ? 'ERROR' : 
'OK'}${record.durationMs === undefined ? '' : ` (${record.durationMs} ms)`}`);
+            // Do not dump source/skills/prompts. Shell output and tool errors
+            // carry the build diagnostics; keep their tail bounded and 
redacted.
+            if (name === 'bash' || record.isError) {
+                const output = textBlocks(record.result?.content);
+                if (output) diagnostic(output);
+            }
+            tools.delete(record.toolCallId);
+            break;
+        }
+        case 'message_end': {
+            const message = record.message;
+            if (message?.role !== 'assistant') break;
+            lastAssistant = message;
+            if (message.usage) {
+                const usage = message.usage;
+                log(`Tokens: input=${usage.input ?? 0}, output=${usage.output 
?? 0}, cacheRead=${usage.cacheRead ?? 0}, cacheWrite=${usage.cacheWrite ?? 0}`);
+            }
+            log(`Model response ended: ${message.stopReason ?? 'unknown'}`);
+            if (message.errorMessage) diagnostic(message.errorMessage);
+            if (message.stopReason === 'length') log('Model output was 
truncated. Increase llmaoMaxTokens only if the gateway quota allows it.');
+            break;
+        }
+        case 'agent_settled':
+            settled = true;
+            aborted = Boolean(record.aborted);
+            log(aborted ? 'Agent aborted' : 'Agent settled');
+            break;
+    }
+}
+
+// Split only on LF, not Unicode line separators (which can occur in JSON).
+function lines(stream, consume) {
+    let pending = '';
+    stream.setEncoding('utf8');
+    stream.on('data', chunk => {
+        pending += chunk;
+        let newline;
+        while ((newline = pending.indexOf('\n')) !== -1) {
+            const line = pending.slice(0, newline).replace(/\r$/, '');
+            pending = pending.slice(newline + 1);
+            if (line) consume(line);
+        }
+    });
+    stream.on('end', () => { if (pending) consume(pending); });
+}
+
+try {
+    const config = 
JSON.parse(fs.readFileSync(`${process.env.PI_CODING_AGENT_DIR}/models.json`, 
'utf8'));
+    const model = config.providers.llmao.models.find(item => item.id === 
process.env.LLMAO_MODEL);
+    log(`Model llmao/${process.env.LLMAO_MODEL}, max output 
tokens=${model?.maxTokens ?? 'provider default'}`);
+} catch {
+    log('Could not read model budget; Pi will report configuration errors.');
+}
+
+const child = spawn('pi', process.argv.slice(2), { stdio: ['inherit', 'pipe', 
'pipe'] });
+lines(child.stdout, line => {
+    try {
+        event(JSON.parse(line));
+    } catch {
+        protocolError = true;
+        // Never print an invalid protocol record: it could contain a prompt.
+        log('Invalid Pi JSON event; the run will fail.');
+    }
+});
+lines(child.stderr, diagnostic);
+child.on('error', error => {
+    diagnostic(`Could not start Pi: ${error.message}`);
+    protocolError = true;
+});
+child.on('close', (code, signal) => {
+    if (signal) log(`Pi terminated by ${signal}`);
+    const answer = textBlocks(lastAssistant?.content).trim();
+    const hasToolCalls = Array.isArray(lastAssistant?.content) && 
lastAssistant.content.some(block => block?.type === 'toolCall');
+    const complete = code === 0 && !signal && !protocolError && settled && 
!aborted &&
+        lastAssistant?.stopReason === 'stop' && !hasToolCalls;
+    const success = complete && answer === 'SUCCESS';
+    if (complete && answer !== 'SUCCESS' && answer !== 'FAILURE') log('Final 
model response was not exactly SUCCESS or FAILURE.');
+    if (!complete) log('Pi did not complete a successful, settled model 
response.');
+    process.stdout.write(`${success ? 'SUCCESS' : 'FAILURE'}\n`);
+    process.exitCode = success ? 0 : (code && code > 0 ? code : 1);
+});
diff --git a/resources/org/apache/sling/jenkins/llmao/run-docker.sh 
b/resources/org/apache/sling/jenkins/llmao/run-docker.sh
index d1101b7..26ed2c7 100644
--- a/resources/org/apache/sling/jenkins/llmao/run-docker.sh
+++ b/resources/org/apache/sling/jenkins/llmao/run-docker.sh
@@ -31,12 +31,14 @@ if [ -n "$checkout_status" ]; then
 fi
 
 status=0
+mkdir -p "$LLMAO_REPORT_DIR"
 result=$(docker run --rm --init \
     --name "$LLMAO_CONTAINER" \
     --user "$(id -u):$(id -g)" \
     --read-only --tmpfs /tmp:rw,nosuid,nodev,mode=1777 \
     --env LLMAO_PAT --env LLMAO_MODEL \
     --mount "type=bind,source=$(pwd)/inputs,target=/inputs,readonly" \
+    --mount "type=bind,source=$LLMAO_REPORT_DIR,target=/reports" \
     --mount "type=bind,source=$LLMAO_WORKSPACE,target=/workspace" \
     --mount 
"type=bind,source=$LLMAO_WORKSPACE/.git,target=/workspace/.git,readonly" \
     --workdir /workspace \
diff --git a/resources/org/apache/sling/jenkins/llmao/run-pi.sh 
b/resources/org/apache/sling/jenkins/llmao/run-pi.sh
index bb867ff..ee94cf6 100644
--- a/resources/org/apache/sling/jenkins/llmao/run-pi.sh
+++ b/resources/org/apache/sling/jenkins/llmao/run-pi.sh
@@ -23,6 +23,7 @@ set -eu
 export HOME=/tmp/pi-home
 export PI_CODING_AGENT_DIR=/tmp/pi-agent
 export PI_OFFLINE=1 PI_SKIP_VERSION_CHECK=1 PI_TELEMETRY=0
+export LLMAO_USAGE_FILE=/reports/llmao-usage.json
 export GIT_OPTIONAL_LOCKS=0
 export MAVEN_CONFIG="$HOME/.m2"
 export MAVEN_OPTS="-Duser.home=$HOME"
@@ -45,6 +46,7 @@ exec timeout --kill-after=5s 1500s pi \
     --print --tools read,bash,edit,write,grep,find,ls --no-session \
     --no-extensions --no-mcp --no-skills --no-prompt-templates \
     --extension /inputs/tool-output-logger.mjs \
+    --extension /inputs/usage-reporter.mjs \
     --skill "$HOME/sling-whiteboard/skills" \
     --no-context-files --no-approve \
     --system-prompt /inputs/system-prompt.txt \
diff --git a/resources/org/apache/sling/jenkins/llmao/usage-reporter.mjs 
b/resources/org/apache/sling/jenkins/llmao/usage-reporter.mjs
new file mode 100644
index 0000000..52a7232
--- /dev/null
+++ b/resources/org/apache/sling/jenkins/llmao/usage-reporter.mjs
@@ -0,0 +1,121 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import fs from 'node:fs';
+
+export default function usageReporter(pi) {
+    const reportFile = process.env.LLMAO_USAGE_FILE;
+    const totals = { input: 0, output: 0, cacheRead: 0, cacheWrite: 0, total: 
0 };
+    const seenMessages = new WeakSet();
+    let model = process.env.LLMAO_MODEL;
+    let requests = 0;
+    let responsesWithUsage = 0;
+    let responsesWithoutUsage = 0;
+    let requestInFlight = false;
+    let reportedUsage = false;
+    let settled = false;
+    let aborted = false;
+    let writeWarning = false;
+
+    function summary() {
+        const complete = settled && !aborted && !requestInFlight && 
responsesWithoutUsage === 0;
+        return {
+            schemaVersion: 1,
+            provider: 'llmao',
+            model,
+            state: settled ? 'settled' : 'running',
+            aborted,
+            usageStatus: responsesWithUsage === 0 ? 'unavailable' : complete ? 
'complete' : 'partial',
+            requests,
+            responsesWithUsage,
+            responsesWithoutUsage,
+            tokens: responsesWithUsage === 0 ? null : { ...totals }
+        };
+    }
+
+    function checkpoint() {
+        try {
+            if (!reportFile) throw new Error('No report file configured');
+            // Persist outside the checkout after each response, not only at 
exit.
+            // Rename ensures Jenkins never reads a half-written report on 
abort.
+            fs.writeFileSync(`${reportFile}.tmp`, `${JSON.stringify(summary(), 
null, 2)}\n`, { mode: 0o600 });
+            fs.renameSync(`${reportFile}.tmp`, reportFile);
+        } catch {
+            if (!writeWarning) {
+                process.stderr.write('[Pi usage] Could not persist the token 
usage report.\n');
+                writeWarning = true;
+            }
+        }
+    }
+
+    pi.on('session_start', (_event, ctx) => {
+        model = ctx.model?.id ?? model;
+        checkpoint();
+    });
+
+    pi.on('before_provider_request', () => {
+        requests++;
+        requestInFlight = true;
+        reportedUsage = false;
+        settled = false;
+        checkpoint();
+    });
+
+    pi.on('provider_stream_event', event => {
+        // Pi initializes message.usage to zero even when the gateway sends no
+        // usage. Observe the raw OpenAI chunk to distinguish missing from 
zero.
+        const usage = event.data?.usage ?? event.data?.choices?.[0]?.usage;
+        if (Number.isSafeInteger(usage?.prompt_tokens) && usage.prompt_tokens 
>= 0 &&
+            Number.isSafeInteger(usage?.completion_tokens) && 
usage.completion_tokens >= 0) {
+            reportedUsage = true;
+        }
+    });
+
+    pi.on('message_end', event => {
+        const message = event.message;
+        if (message?.role !== 'assistant' || seenMessages.has(message)) return;
+        seenMessages.add(message);
+        requestInFlight = false;
+        const usage = message.usage;
+        const tokens = {
+            input: usage?.input,
+            output: usage?.output,
+            cacheRead: usage?.cacheRead ?? 0,
+            cacheWrite: usage?.cacheWrite ?? 0,
+            total: usage?.totalTokens ?? (usage?.input + usage?.output + 
(usage?.cacheRead ?? 0) + (usage?.cacheWrite ?? 0))
+        };
+        if (reportedUsage && Object.values(tokens).every(value => 
Number.isSafeInteger(value) && value >= 0)) {
+            responsesWithUsage++;
+            for (const key of Object.keys(totals)) totals[key] += tokens[key];
+        } else {
+            responsesWithoutUsage++;
+        }
+        reportedUsage = false;
+        checkpoint();
+        // Observe only: do not replace the assistant message or its usage.
+    });
+
+    pi.on('agent_settled', event => {
+        settled = true;
+        aborted = Boolean(event.aborted);
+        checkpoint();
+        const report = summary();
+        process.stderr.write(`[Pi usage] ${report.tokens ? `Observed total: 
${totals.total} tokens (${report.usageStatus})` : 'Token usage 
unavailable'}\n`);
+    });
+}
diff --git a/vars/slingOsgiBundleBuild.groovy b/vars/slingOsgiBundleBuild.groovy
index eba66ca..44bdc58 100644
--- a/vars/slingOsgiBundleBuild.groovy
+++ b/vars/slingOsgiBundleBuild.groovy
@@ -203,11 +203,14 @@ def runLlmaoOnlyBuild(def globalConfig, Map params) {
                     script: 'printf "sling-llmao-%s" "$(cat 
/proc/sys/kernel/random/uuid)"',
                     returnStdout: true
                 ).trim()
+                // A unique directory prevents old usage from leaking into a 
new build.
+                def reportDirectory = "outputs/${containerName}"
                 withEnv([
                     "LLMAO_CONTAINER=${containerName}",
                     'LLMAO_IMAGE=sling-llmao-pi:1.1.0-jdks8-21-v2',
                     "LLMAO_MODEL=${llmaoConfig.llmaoModel}",
-                    "LLMAO_WORKSPACE=${projectWorkspace}"
+                    "LLMAO_WORKSPACE=${projectWorkspace}",
+                    "LLMAO_REPORT_DIR=${pwd()}/${reportDirectory}"
                 ]) {
                     try {
                         stage('Submit to LLMAO') {
@@ -220,6 +223,7 @@ def runLlmaoOnlyBuild(def globalConfig, Map params) {
                             dir('inputs') {
                                 writeFile file: 'run-pi.sh', text: 
libraryResource('org/apache/sling/jenkins/llmao/run-pi.sh')
                                 writeFile file: 'tool-output-logger.mjs', 
text: libraryResource('org/apache/sling/jenkins/llmao/tool-output-logger.mjs')
+                                writeFile file: 'usage-reporter.mjs', text: 
libraryResource('org/apache/sling/jenkins/llmao/usage-reporter.mjs')
                                 writeFile file: 'system-prompt.txt', text: 
'You are a coding assistant working on the checked-out project. Follow the 
relevant supplied Apache Sling skills; use update-sling-parent-pom for parent 
POM upgrades. Complete the requested update and its required verification. Java 
21 is the default; mvn8, mvn11, mvn17, and mvn21 run Maven with the 
corresponding JDK. Do not stage, unstage, commit, push, switch branches, or 
change Git metadata. Respond with exactl [...]
                                 writeFile file: 'prompt_input.txt', text: 
contextData, encoding: 'UTF-8'
                                 writeJSON file: 'models.json', json: [
@@ -256,13 +260,25 @@ def runLlmaoOnlyBuild(def globalConfig, Map params) {
                             }
                         }
                     } finally {
-                        // Also clean up on Jenkins cancellation, not only 
normal Docker exit.
-                        sh '''
-                            set +x
-                            docker rm --force "$LLMAO_CONTAINER" >/dev/null 
2>&1 || true
-                            rm -f image/Dockerfile inputs/models.json 
inputs/settings.json \\
-                              inputs/system-prompt.txt inputs/prompt_input.txt 
inputs/run-pi.sh inputs/tool-output-logger.mjs
-                        '''
+                        try {
+                            // Stop writes before archiving, including on 
Jenkins cancellation.
+                            sh '''
+                                set +x
+                                docker rm --force "$LLMAO_CONTAINER" 
>/dev/null 2>&1 || true
+                            '''
+                            dir(reportDirectory) {
+                                publishLlmaoUsage(llmaoConfig.llmaoModel, 
updateStatus)
+                            }
+                        } finally {
+                            sh '''
+                                set +x
+                                rm -f image/Dockerfile inputs/models.json 
inputs/settings.json \\
+                                  inputs/system-prompt.txt 
inputs/prompt_input.txt inputs/run-pi.sh \\
+                                  inputs/tool-output-logger.mjs 
inputs/usage-reporter.mjs
+                                rm -f "$LLMAO_REPORT_DIR/llmao-usage.json" 
"$LLMAO_REPORT_DIR/llmao-usage.json.tmp"
+                                rmdir "$LLMAO_REPORT_DIR" >/dev/null 2>&1 || 
true
+                            '''
+                        }
                     }
                 }
             }
@@ -270,9 +286,7 @@ def runLlmaoOnlyBuild(def globalConfig, Map params) {
             stage('Process LLMAO Response') {
                 if ( updateStatus == 0 ) {
                     echo 'Pi completed the project update successfully.'
-                    currentBuild.description = 'LLMAO: SUCCESS'
                 } else {
-                    currentBuild.description = 'LLMAO: FAILURE'
                     error("Pi could not complete the project update (exit code 
${updateStatus}).")
                 }
             }
@@ -280,6 +294,29 @@ def runLlmaoOnlyBuild(def globalConfig, Map params) {
     }
 }
 
+def publishLlmaoUsage(def model, def updateStatus) {
+    def report = fileExists('llmao-usage.json') ? readJSON(file: 
'llmao-usage.json', returnPojo: true) : [
+        schemaVersion: 1, provider: 'llmao', model: model,
+        state: 'unknown', aborted: null, usageStatus: 'unavailable',
+        requests: null, responsesWithUsage: null, responsesWithoutUsage: null, 
tokens: null
+    ]
+    report.build = [job: env.JOB_NAME, number: env.BUILD_NUMBER, url: 
env.BUILD_URL]
+    report.result = updateStatus == null ? 'INCOMPLETE' : updateStatus == 0 ? 
'SUCCESS' : 'FAILURE'
+    report.exitCode = updateStatus
+    def description = "LLMAO: ${report.result}"
+    if ( report.tokens != null ) {
+        def tokens = report.tokens
+        echo "LLMAO observed tokens: input=${tokens.input}, 
output=${tokens.output}, cacheRead=${tokens.cacheRead}, 
cacheWrite=${tokens.cacheWrite}, total=${tokens.total} (${report.usageStatus})."
+        description += " | ${tokens.total} tokens${report.usageStatus == 
'partial' ? ' (partial)' : ''}"
+    } else {
+        echo 'LLMAO token usage unavailable; no gateway-reported usage was 
collected.'
+        description += ' | tokens unavailable'
+    }
+    currentBuild.description = description
+    writeJSON file: 'llmao-usage.json', json: report
+    archiveArtifacts artifacts: 'llmao-usage.json', allowEmptyArchive: false
+}
+
 def jenkinsJdkLabel(int jdkVersion, def globalConfig) {
     def label = globalConfig.availableJDKs[jdkVersion]
     if ( !label )

Reply via email to