goutamadwant opened a new issue, #12191:
URL: https://github.com/apache/seatunnel/issues/12191
### Search before asking
- [x] Searched existing issues and PRs and found no equivalent opt-in
conversion check. #7617/#7620 address capitalized JSON boolean parsing, not
malformed values becoming false; that existing support is preserved.
### Description
Add optional strict BOOLEAN result validation to the LLM transform. Current
conversion uses `Boolean.parseBoolean` on the first returned element. Values
such as `"unknown"`, `""`, or a null element become `false`, and extra elements
are ignored.
Once this conversion has happened, downstream validation cannot distinguish
a genuine `false` from an invalid model result. Changing the default could
break existing jobs, so this proposes an explicit opt-in mode instead.
### Usage Scenario
A pipeline uses an LLM to produce a BOOLEAN classification or routing field.
An uncertain or malformed answer should be distinguishable from an actual
negative answer when the operator requests strict validation.
### Reproduction and before behavior
Baseline: `dev` at `8bea8c681cacccbd99fa642ca0853718980b9234`.
The reproduction uses the actual `LLMTransform` and an in-process HTTP
fixture. Configure `output_data_type = BOOLEAN`, supply one input row, and
return a chat-completion envelope whose message content is:
```json
["unknown"]
```
The resulting BOOLEAN field is `false`. The same occurs for `[""]`,
`[null]`, `[" true "]`, and `["yes"]`; `[false, true]` uses the first element
and ignores the extra result. This behavior was confirmed on Java 8 and Java
11. Existing valid/capitalized boolean controls pass.
The compare includes
`seatunnel-transforms-v2/src/test/java/org/apache/seatunnel/transform/llm/LLMBooleanOutputTest.java`,
with both unchanged-default controls and new strict expectations:
```shell
./mvnw -f seatunnel-transforms-v2/pom.xml \
-Dskip.spotless=true -Dtest=LLMBooleanOutputTest test
```
Run after the usual repository dependency setup. Before production changes,
the original 38-case fixture had 13 expected failures on both Java versions
because strict validation was unavailable. Those failures demonstrate the
proposed contract gap, not that the existing documented default must change.
### Proposed contract and after behavior
```hocon
output_data_type = BOOLEAN
strict_boolean_output = true
```
- New option `strict_boolean_output` defaults to `false`.
- With BOOLEAN output and strict mode enabled, accept exactly one non-null
result equal to `true` or `false`, case-insensitively and without whitespace
trimming.
- Reject malformed, empty, null, or multiple results with a validation error
that includes neither the input row nor the model response.
- With the option omitted or disabled, preserve the current conversion
behavior. Other output types, valid capitalized literals, and custom-provider
response parsing are unchanged.
For the same `"unknown"` response, the default still returns false; strict
mode fails validation. This does not add retries, row skipping, or error-table
routing.
Please confirm this opt-in contract during review; this issue does not
assume approval to tighten the default.
### Compatibility and validation
- Actual Java 8 (Corretto 1.8.0_504) and Java 11 (Temurin 11.0.32.1): 1,150
transform tests passed on each, zero failures/errors/skips.
- 41 regression cases cover defaults, valid/malformed/null/cardinality
cases, sanitized error stack traces, non-BOOLEAN types, row metadata, custom
response formats, and serialization.
- The transform retains its baseline Java serialization identifier. New
default/strict factory wrappers round-trip with their behavior intact. A
separate manual probe reads actual archived baseline wrapper bytes under the
corrected classes on both Java versions; malformed output still maps to false.
This is not a full cluster-upgrade test.
- Repository Spotless and full-repository `./mvnw -q -DskipTests verify`
passed, with the latter on Java 11. That build skips tests and is separate from
the Java 8/11 suites.
- No external model service or paid API was used. No public API is removed
and no dependency is added. EN/ZH documentation describes the option and its
limits.
### Related issues and implementation
Related historical behavior: #7617/#7620. The proposal works after provider
parsing and leaves those fixes intact.
[Reviewable compare
branch](https://github.com/apache/seatunnel/compare/dev...goutamadwant:feature/model-boolean-validation).
A PR will be linked after review.
### Are you willing to submit a PR?
- [x] Yes, I am willing to submit a PR.
### Code of Conduct
- [x] I agree to follow this project's [Code of
Conduct](https://www.apache.org/foundation/policies/conduct).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
