mattcasters commented on issue #8711:
URL: https://github.com/apache/hop/issues/8711#issuecomment-5918849385
## JSON implementation plan — part 1 of 3
The full saved plan exceeds one GitHub comment, so it is published in 3
parts. This is a proposed implementation plan; code snippets and expected test
results have not been executed. The following comments contain the remaining
parts.
---
<!--
SPDX-License-Identifier: Apache-2.0
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
https://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
-->
# Apache Hop #8711: phased JSON implementation plan
## Goal
Resolve or explicitly disposition every child of
[apache/hop#8711](https://github.com/apache/hop/issues/8711), improving JSON
correctness and bounded-memory file output without silently changing existing
pipelines or duplicating work already in review.
## Current context / assumptions
### Evidence and scope
- Inspected workspace: `/home/matt/git/mattcasters/hop`, branch
`issue-8711`, clean tracked worktree at
`2fbe81f7caeb92745be63396f0d6b17b89dd6137` before writing this plan.
- Issue bodies and **all returned comments** for the umbrella and its 15
children were read on 2026-09-30. Collection was checked programmatically: 15
entries, 15 unique issue numbers.
- This is a **planning-only** deliverable. No Maven builds, tests, checkout,
commits, issue edits, or merges were executed. Maven writes `target/` and
invokes Spotless, so even a baseline test run would violate this turn's
restrictions. Every RED/GREEN result below is a required future outcome, **not
an observed result**. Do not quote prospective test counts as measurements.
- Live JDK is OpenJDK 21.0.12.1. Root `pom.xml` targets Java 21. The JSON
module is `plugins/transforms/json/pom.xml`, artifact `hop-transform-json`,
version `2.20.0-SNAPSHOT`.
- Jackson databind/core are pinned to `2.21.5` in `lib/pom.xml`; Jayway
JsonPath is `2.10.0` in root `pom.xml`. Local jars were checked with `javap`:
`ObjectMapper.readTree/readValue/writeValueAsString/convertValue/valueToTree`,
`JsonFactory.createGenerator(Writer)`,
`JsonGenerator.writeTree/writeStartArray/writeEndArray`, and
`JacksonJsonNodeJsonProvider(ObjectMapper)` exist.
- **json-simple and json-smart are different libraries.** #7995 concerns
`org.json.simple` / `com.googlecode.json-simple:json-simple:1.1.1`. Jayway's
current string-input provider uses `net.minidev:json-smart:2.6.0`; do not
remove or replace that provider as a side effect of #7995.
- Repository rules: read `AGENTS.md` and `CONTRIBUTING.md`; retain persisted
XML keys and transform IDs; use Hop VFS for file I/O; use Lombok; use
annotated, grouped widgets for new settings. Preserve existing custom tables
and existing forms rather than rebuilding every dialog.
### A real source prerequisite, not an assumed green baseline
`plugins/transforms/json/src/main/java/org/apache/hop/pipeline/transforms/jsonoutputenhanced/JsonEOutput.java:319`
literally contains the invalid Java identifier `getKeyJson...Name`. The same
token occurs at its call sites. Both a filesystem read and `git show
HEAD:<path>` returned it, despite the clean worktree. PR #8713's diff also
retains that token. This cannot be treated as a valid accessor or silently
normalized in the plan. Task 0.2 explicitly diagnoses and, if it remains
present on the implementation baseline, repairs this syntax prerequisite before
feature tests. GitHub reports passing checks on the PR snapshots; that does
**not** establish that this local source compiles.
### Child issue disposition and effort
All rows remain tracker obligations until verification or an explicit
maintainer decision. “Already implemented” means source evidence, not that the
ticket has been closed.
| Child | Requirement | Evidence / proposed disposition | Effort | Phase |
|---|---|---|---|---|
| [#2569](https://github.com/apache/hop/issues/2569) | Enhanced output
differs from old wiki example | Reproduce the attached pipeline; document
grouping versus field serialization, do not change output solely to match an
obsolete screenshot | Low | 1 / 4 |
| [#2570](https://github.com/apache/hop/issues/2570) | Missing/renamed group
key produces unexpected exception | Validate key lookup **before** adding its
metadata, in both metadata and runtime paths; include the missing name | Low |
2 |
| [#2572](https://github.com/apache/hop/issues/2572) | Compatibility tooltip
claims Hop 4.3.0 | English tooltip is already behavior-based; stale Italian
legacy key remains. Correct that translation; no new compatibility mode | Low |
1 |
| [#2929](https://github.com/apache/hop/issues/2929) | Streaming enhanced
output | **Reuse/review open PR #8713**, not a second implementation. Its
row-at-a-time file mode does not prove constant memory for one enormous group |
High; existing implementation | 4 |
| [#4344](https://github.com/apache/hop/issues/4344) | NDJSON append | Add
an opt-in, GUI-backed format after streaming has landed. Preserve the default
concatenated-document append behavior | Medium | 5 |
| [#4373](https://github.com/apache/hop/issues/4373) | Distinguish JSON null
from empty string with system setting | Fix JSON converter's null input
handoff; do not alter global CSV/text null conventions | Low | 2 |
| [#4395](https://github.com/apache/hop/issues/4395) | Object key name/value
by ordinal | Recommend declining custom `$.[0].name()` syntax: objects do not
have portable positional meaning. Document quoted-key lookup, existing
`keys()`, and an explicit ordered entries array | Low decision/docs | 1 |
| [#4644](https://github.com/apache/hop/issues/4644) | JSON-typed source
fields | Source already routes `TYPE_JSON` through `jsonFieldIterator()` and
Jackson provider; existing `0007-json-input-json-value` integration test
exists. Verify and document rather than adding generic Object/Map coercion |
Low verification/docs | 1 |
| [#4907](https://github.com/apache/hop/issues/4907) | Timestamp conversion,
NPE, UI freeze | Current parser has both null guards; source/tests support >3
`S` fraction digits. Reporter mask requires an offset absent from the string.
Verify corrected mask and invalid-mask failure; treat freeze separately if it
reproduces | Low verification, possible separate bug | 1 |
| [#5167](https://github.com/apache/hop/issues/5167) | Group Key Get Fields
| Add button that inserts upstream fields not already mapped on Fields or Group
Key, preserving mappings and order | Medium | 3 |
| [#5313](https://github.com/apache/hop/issues/5313) | Grouping
documentation / element names | Shares #5167; element name **is** used in file
group envelopes, but does not rename forwarded row keys. Clarify instead of
deleting it | Low, dependent on 3 / 4 | 3 / 4 |
| [#5953](https://github.com/apache/hop/issues/5953) | Configured source
field disappears on upstream lookup failure | **Reuse/review open PR #8588**,
including its common UI helpers and tests; do not duplicate its broad dialog
sweep | Medium review; existing implementation | 0 / 3 |
| [#7995](https://github.com/apache/hop/issues/7995) | Remove json-simple |
Cross-repository migration, not just JSON transforms; 68 Java source/test
files. Public signatures expose JSONObject. Approve compatibility policy, then
migrate callers before removing dependency | High | 6 |
| [#8444](https://github.com/apache/hop/issues/8444) | Literal root-array
JSONPath collides with `$[0]` variable escape | Add `resolveJsonPaths` default
**true** for old pipelines; allow false in a grouped GUI settings tab, apply
only to extraction paths | Medium | 3 |
| [#8593](https://github.com/apache/hop/issues/8593) | Negative integers
fail under scheduler locale | PR #8639 merged 2026-09-27; commit `18d16444b3`
is present locally. Verify existing locale tests and release workaround; no
JSON-local minus normalization | Low | 1 |
### Existing work that must not be duplicated
- [PR #8713](https://github.com/apache/hop/pull/8713), OPEN, reviewed
snapshot `2b6d3b1eac7f96d71b3ba9445dfb4af07e3d3b6f`, branch `2929`: enhanced
file streaming; modifies `JsonEOutput.java`, `JsonEOutputData.java`, enhanced
output manual, and JSON integration cases 0018/0019. GitHub returned successful
build/UI/license/Antora checks for this snapshot. Still review behavior and
memory guarantees locally.
- [PR #8588](https://github.com/apache/hop/pull/8588), OPEN, reviewed
snapshot `ed7e15e24d2603d1ca1e5c4f078f2cf578132292`, branch `5953`: common
`ComboItems` / `PreviousFields` helpers and many dialog tests. GitHub checks
returned success. Its comments request manual testing; a pending review flags
successful-empty Get Fields behavior. Treat that as an explicit acceptance gate.
- [PR #8639](https://github.com/apache/hop/pull/8639), MERGED:
negative-number locale handling in shared `ValueMetaBase`; existing
`ValueMetaBaseLocaleMinusTest` covers nb-NO, sv-SE, fi-FI, ar-SA, ASCII/Unicode
minus and invalid text.
- Timestamp guard fix is already in history: `4c889cfb01`, “fix NPE in
Select Values when converting String to Timestamp”, #6315 / #6306. Do not add
the same null guard again.
## Architecture / proposed approach
Keep Jayway as the JSONPath evaluator and Jackson as Hop's JSON value/file
representation; fix shared conversion and metadata-validation boundaries rather
than applying string hacks inside individual transforms. Land independent
correctness/documentation changes first, integrate the two existing PRs, and
then extend the resulting file writer with an explicit NDJSON mode. Treat
json-simple removal as its own API-sensitive migration with characterization
tests, staged caller conversion, and packaging verification; it must not be
bundled into the streaming or JsonPath change.
## Execution conventions (read before any task)
**Timeboxes:** each numbered edit/review/test setup item is 2–5 minutes of
focused work. A Maven build or stress run may take longer unattended. Finish
one RED → minimal GREEN slice before adding the next test. A phase is not a
five-minute task. Never combine all phase tests into one speculative batch
before implementation.
**Frequent commits:** commit only green slices, using explicit paths, not
`git add .`. Commands below are for the future implementer; no commits have
been made by the planner. Do not push or merge without authorization.
**New Java files:** prepend exactly the Apache header from lines 1–16 of the
neighboring existing Java file. The snippets below are complete class
bodies/import sets or complete replacement members; a header is mandatory, not
optional. Keep the package shown. Existing files keep their headers.
**Fast feedback after the baseline install:** define these shell functions
in the repository root. They are not files to add to the project.
```bash
json_test() {
./mvnw -pl plugins/transforms/json -Pskip-uitest -DskipTests=false \
-Dspotless.skip=true -Drat.skip=true -Dtest="$1" test
}
core_test() {
./mvnw -pl core -Pskip-uitest -DskipTests=false \
-Dspotless.skip=true -Drat.skip=true -Dtest="$1" test
}
json_ui_test() {
./tools/with-isolated-display.sh ./mvnw -pl plugins/transforms/json \
-Puitest -DskipTests=false -Dspotless.skip=true -Drat.skip=true \
-Dtest="$1" test
}
```
GREEN means exit 0, `BUILD SUCCESS`, the specified class/method actually ran
with failures/errors zero, and was **not skipped**. RED means the specified
assertion or explicitly introduced missing API fails; missing dependencies, a
skipped suite, or the unrelated syntax prerequisite are not the desired RED.
Inspect
`plugins/transforms/json/target/surefire-reports/TEST-<fully-qualified-class>.xml`
or the corresponding `core/target/surefire-reports/` file to verify execution.
Do not accept zero tests.
### Shared finalizer for each green slice
For each task use its listed production/test/docs paths in the `git add`
command. Run `./mvnw -pl <touched-module> spotless:apply`, rerun the focused
tests, then `git diff --check` (expected exit 0, no output), `git diff --stat`,
and `git commit -m '<listed message>'`. If formatting touches unrelated files,
restore **only** formatter changes proven unrelated; do not discard user work.
Documentation-only and already-fixed characterization tasks do not manufacture
a failing test by breaking production code: verify the behavior, make the
documentation/test addition, then commit.
## Phase 0 — baseline and coordinate existing work
### 0.1 Recheck checkout and prerequisites (2 minutes)
Read `AGENTS.md`, `CONTRIBUTING.md`, root `pom.xml`,
`plugins/transforms/json/pom.xml`, and `tools/with-isolated-display.sh`.
```bash
pwd
git status --short
git branch --show-current
git rev-parse HEAD
java -version
./mvnw --version
gh issue view 8711 --repo apache/hop --json body,state
gh pr view 8588 --repo apache/hop --json state,headRefOid,mergedAt
gh pr view 8713 --repo apache/hop --json state,headRefOid,mergedAt
```
Expected: correct repository and Java 21; record current SHA and uncommitted
work. PR heads/states may have changed; the hashes in this plan are evidence
snapshots, not instructions to reset the branch. Stop rather than overwrite an
implementer's changes. No commit for read-only preflight.
### 0.2 Establish a compilable baseline (one diagnosis slice + one repair
slice if necessary)
1. Verify the literal syntax anomaly with `git show
HEAD:plugins/transforms/json/src/main/java/org/apache/hop/pipeline/transforms/jsonoutputenhanced/JsonEOutput.java`
and search the exact token `getKeyJson...Name`. If the new baseline no longer
contains it, do not make an unnecessary rename.
2. Run the following compiler baseline. Expected if the token remains:
nonzero exit, syntax error in `JsonEOutput.java` around the invalid name;
record the actual message instead of inventing one.
```bash
./tools/with-isolated-display.sh ./mvnw \
-pl plugins/transforms/json -am -Pskip-uitest \
-DskipTests -Dspotless.skip=true -Drat.skip=true compile
```
3. **Only if confirmed**, explicitly replace the declaration and every
invocation of the literal token with `getKeyJsonAttributeName`. This is a
source repair, not an assumed existing symbol. Its complete body is:
```java
private String getKeyJsonAttributeName(JsonEOutputKeyField field) {
String elementName = variables.resolve(field.getElementName());
return Const.NVL(elementName, field.getFieldName());
}
```
Rerun the compile command; require `BUILD SUCCESS` before starting feature
TDD. Commit the confirmed prerequisite separately:
```bash
git add
plugins/transforms/json/src/main/java/org/apache/hop/pipeline/transforms/jsonoutputenhanced/JsonEOutput.java
git commit -m 'Restore the enhanced JSON key attribute method identifier'
```
This is compiler RED/GREEN for a pre-existing syntax failure, not a claimed
behavioral regression test.
### 0.3 Seed dependencies and record a real test baseline (3 minutes setup;
build runs unattended)
```bash
./tools/with-isolated-display.sh ./mvnw \
-pl plugins/transforms/json -am -Pskip-uitest -DskipTests=false \
-Dspotless.skip=true -Drat.skip=true install
json_test '*Test,*Tests'
core_test
'SimpleTimestampFormatTest,ValueMetaTimestampTest,ValueMetaBaseLocaleMinusTest'
```
Expected: exit 0 and `BUILD SUCCESS`; save the **actual** executed counts in
implementation/PR notes. Installation is necessary because plugin tests use
core/engine/UI test-jars. After a later core or UI edit, reinstall that module
and its dependencies before testing the JSON module against the new code. If
offline artifacts are absent, resolve them normally; do not silently switch to
`-DskipTests` for GREEN.
### 0.4 Coordinate the existing PRs (two independent 3-minute review slices)
Read-only commands:
```bash
gh pr diff 8588 --repo apache/hop --color never
gh pr checks 8588 --repo apache/hop
gh pr diff 8713 --repo apache/hop --color never
gh pr checks 8713 --repo apache/hop
```
Do not cherry-pick a whole foreign PR onto this branch as a planning
shortcut. Once authorized and merged, update the implementation branch
normally, then read back the target files and rebase the remaining plan. For
#8588 inspect `ui/src/main/java/org/apache/hop/ui/core/widget/ComboItems.java`,
`ui/src/main/java/org/apache/hop/ui/pipeline/transform/PreviousFields.java`,
and
`ui/src/main/java/org/apache/hop/ui/pipeline/transform/BaseTransformDialog.java`
at the merged revision; these are **PR-introduced paths**, not files assumed
present at this plan's baseline. Verify exact paths against `gh pr diff` before
importing them.
Acceptance: a failed lookup preserves text, raises at most one error per
dialog; a successful lookup returning zero fields is distinguishable from
failure and must not silently retain a stale mapping when the user explicitly
requests replacement. Preserve JSON Input and JSON Normalize settings. No
second sweep of unrelated dialogs under #8711.
## Phase 1 — resolve already-supported behavior and documentation
### 1.1 Verify JSON-typed source support; update instructions (#4644, 3
minutes)
Existing implementation paths:
-
`plugins/transforms/json/src/main/java/org/apache/hop/pipeline/transforms/jsoninput/JsonInput.java:245`:
metadata-driven String versus JSON routing.
-
`plugins/transforms/json/src/main/java/org/apache/hop/pipeline/transforms/jsoninput/reader/InputsReader.java:86`:
`jsonFieldIterator()`.
-
`plugins/transforms/json/src/main/java/org/apache/hop/pipeline/transforms/jsoninput/reader/FastJsonReader.java:200`:
native Jackson-node parsing.
- `integration-tests/json/0007-json-input-json-value.hpl` and
`integration-tests/json/main-0007-json-input-json-value.hwf`: existing
pipeline-level coverage.
Add the following characterization test to
`plugins/transforms/json/src/test/java/org/apache/hop/pipeline/transforms/jsoninput/JsonInputTest.java`
(existing imports/helpers suffice):
```java
@Test
void jsonTypedSourceFieldExtractsAValue() throws Exception {
JsonInputField field = new JsonInputField("value");
field.setPath("$.value");
field.setType(IValueMeta.TYPE_STRING);
JsonInputMeta meta = createSimpleMeta("json", field);
meta.setRemoveSourceField(true);
JsonNode input = HopJson.newMapper().readTree("{\"value\":\"ok\"}");
JsonInput transform = createJsonInputWithJsonNode("json", meta, new
Object[] {input});
List<Object[]> rows = new ArrayList<>();
transform.addRowListener(new RowAdapter() {
@Override
public void rowWrittenEvent(IRowMeta rowMeta, Object[] row) {
rows.add(row.clone());
}
});
try {
processRows(transform, 3);
assertEquals(0, transform.getErrors());
assertEquals(1, rows.size());
assertEquals("ok", rows.getFirst()[0]);
} finally {
transform.dispose();
}
}
```
Run `json_test 'JsonInputTest#jsonTypedSourceFieldExtractsAValue'`; expected
GREEN **already**, because this is a stale-ticket characterization, not new
functionality. If it fails, classify the actual failure before touching
`StringFieldIterator`; do not serialize arbitrary Java objects with
`toString()`.
In
`docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/jsoninput.adoc`,
replace the String-only chained-input example with:
```adoc
* To read a nested JSON value, use either a String field containing
serialized JSON or a JSON-typed field containing a Jackson JSON value. In the
next JSON Input transform, enable *Source is from a previous transform* and
select that field. Leave *Use field as file names* and *Read source as URL*
cleared for direct JSON content. A filename or URL source remains a String
field; an arbitrary Java Object is not a supported JSON source.
```
Commit the exact test and manual paths, message `Document and verify
JSON-typed JSON Input sources`.
### 1.2 Verify timestamp precision and failure propagation (#4907, three
3-minute slices)
Do not repeat the issue thread's inaccurate “nanoseconds unsupported” claim.
Current `SimpleTimestampFormat` replaces **four or more `S` characters** with
its fraction parser; `ValueMetaTimestampTest` already checks 6- and 9-digit
fractions. There is no proposed `n` format letter.
**Slice A:** add to
`core/src/test/java/org/apache/hop/core/row/value/ValueMetaTimestampTest.java`:
```java
@Test
void parsesTheIssue4907TimestampWithAMatchingMask() throws Exception {
ValueMetaTimestamp meta = new ValueMetaTimestamp("created");
meta.setConversionMask("yyyy-MM-dd'T'HH:mm:ss.SSSSSS");
assertEquals(
Timestamp.valueOf("2025-02-13 02:10:14.746582"),
meta.convertStringToTimestamp("2025-02-13T02:10:14.746582"));
}
```
Run `core_test
'ValueMetaTimestampTest#parsesTheIssue4907TimestampWithAMatchingMask'`.
Prospective GREEN on current source. Do not reduce precision in the assertion
to make it pass.
**Slice B:** add to
`core/src/test/java/org/apache/hop/core/row/value/timestamp/SimpleTimestampFormatTest.java`,
plus `import static org.junit.jupiter.api.Assertions.assertThrows;`:
```java
@Test
void missingRequiredOffsetIsAParseFailureNotANullPointer() {
SimpleTimestampFormat format =
new SimpleTimestampFormat("yyyy-MM-dd'T'HH:mm:ss.SSSXXX", Locale.US);
assertThrows(
ParseException.class,
() -> format.parse("2025-02-13T02:10:14.746582"));
}
```
Run `core_test
'SimpleTimestampFormatTest#missingRequiredOffsetIsAParseFailureNotANullPointer'`;
expected GREEN on the existing guards. If the high-precision test fails,
reproduce it with this exact mask first; do not replace the entire date/time
formatter under this ticket.
**Slice C:** update
`docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/jsoninput.adoc`
timestamp examples with this complete replacement text:
```adoc
For Timestamp values, match the fraction width in the Format column. For
example, `2025-02-13T02:10:14.746582` uses `yyyy-MM-dd'T'HH:mm:ss.SSSSSS`.
Timestamp preserves the six fractional digits; Date has millisecond precision.
`XXX` requires an offset such as `Z` or `+02:00` in the input. Do not use
`XXX` for a value that contains no offset. A mismatched format must report a
conversion error, not silently assume an offset or discard precision.
```
Then manually run the JSON Input and Select Values reproductions with the
invalid mask under an isolated GUI: both must terminate with a conversion error
and permit the preview/dialog to close. If a freeze remains, capture the actual
thread stacks and open a narrowly scoped follow-up; **do not close #4907 based
on parser tests alone**. Commit each test slice separately and the
documentation after confirmation; messages `Characterize microsecond timestamp
conversion for issue 4907`, `Verify missing timestamp offset fails cleanly`,
`Correct JSON Input timestamp format examples`.
### 1.3 Verify the merged locale fix (#8593, 3 minutes)
```bash
git merge-base --is-ancestor 18d16444b3 HEAD
core_test ValueMetaBaseLocaleMinusTest
```
Expected ancestor command exit 0 and locale tests GREEN. No production edit.
Verify a scheduler-style headless run with ASCII `-2` under nb-NO as part of
Phase 7, not only in Hop GUI, and include Calculator/Filter Rows because the
cause is shared. Existing release workaround is `HOP_OPTIONS=-Duser.language=en
-Duser.country=US` for versions lacking the fix; do not set the user's global
locale in code. Close only with merged SHA/release evidence and test output.
### 1.4 Correct the stale compatibility tooltip (#2572, 2 minutes)
`plugins/transforms/json/src/main/resources/org/apache/hop/pipeline/transforms/jsonoutputenhanced/messages/messages_en_US.properties`
already says the option forces arrays for single entries. The stale version
reference is in the Italian bundle:
`plugins/transforms/json/src/main/resources/org/apache/hop/pipeline/transforms/jsonoutputenhanced/messages/messages_it_IT.properties`
Replace the exact old `JsonOutputDialog.CompatibilityMode.Tooltip` value
with:
```properties
JsonOutputDialog.CompatibilityMode.Tooltip=Selezionare quest''opzione per
generare array JSON anche quando la lista contiene un solo elemento
JsonEOutputDialog.CompatibilityMode.Tooltip=Selezionare quest''opzione per
generare array JSON anche quando la lista contiene un solo elemento
JsonEOutputDialog.UseArrayWihSingleInstanceMode.Tooltip=Selezionare
quest''opzione per generare array JSON anche quando la lista contiene un solo
elemento
```
Retain the old key because other translations/legacy references may still
use it; add the currently used `JsonEOutputDialog` key. Search the bundles for
`4.3.0`; expected **no obsolete Hop-version claim** in these tooltips. Run
`json_test JsonEOutputMetaTest` plus the final RAT/docs gates. Commit this
bundle alone, message `Describe enhanced JSON array behavior without a
fictitious version`.
### 1.5 Decide object ordinal access; document supported lookup (#4395, 3
minutes)
Recommend explicit maintainer approval to decline `$.[0].name()` / `$.[0]`
on objects rather than inventing a private JSONPath dialect. The comment's
quoted-key suggestion retrieves a known key's value; it does **not** implement
ordinal key enumeration. Do not mark the request completed without that
distinction.
Add to
`docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/jsoninput.adoc`:
```adoc
*Object keys and array indexes*
A key containing a dot is one key, not a nested path. Use `$['doc.tipo']` to
retrieve its value. The existing `$.keys()` function returns the object's keys
as one JSON value; it does not define portable object-member ordering or return
a key/value pair by ordinal.
Array indexes address array elements. To select the first key/value pair by
position, make the order explicit in the source, for example
`[{"name":"doc.tipo","value":"XG"}]`, and use `$[0].name` and `$[0].value` with
JsonPath variable resolution disabled. The expressions `$.[0].name()` and
`$.[0]` do not define positional access to object members.
```
Verification: `json_test FastJsonReaderTest` must retain function/key tests
GREEN; inspect the rendered text. Commit only the manual, message `Clarify
JSONPath object keys versus array indexes`. If maintainers instead require an
ordinal feature, leave #4395 open and require a separate contract proposal
(ordering, duplicate-key handling, out-of-range behavior, GUI), not an
improvised parser extension.
## Phase 2 — localized correctness fixes
### 2.1 Preserve JSON null before generic String conversion (#4373)
**RED (3 minutes):** create
`plugins/transforms/json/src/test/java/org/apache/hop/pipeline/transforms/jsoninput/reader/RowOutputConverterNullTest.java`,
with the mandatory header and this complete class:
```java
package org.apache.hop.pipeline.transforms.jsoninput.reader;
import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.junit.jupiter.api.Assertions.assertNull;
import static org.junit.jupiter.api.Assertions.assertTrue;
import static org.mockito.Mockito.mock;
import com.fasterxml.jackson.databind.node.NullNode;
import com.fasterxml.jackson.databind.node.TextNode;
import java.util.BitSet;
import org.apache.hop.core.Const;
import org.apache.hop.core.logging.ILogChannel;
import org.apache.hop.core.row.RowMeta;
import org.apache.hop.core.row.value.ValueMetaString;
import org.apache.hop.pipeline.transforms.jsoninput.JsonInputData;
import org.junit.jupiter.params.ParameterizedTest;
import org.junit.jupiter.params.provider.ValueSource;
class RowOutputConverterNullTest {
@ParameterizedTest
@ValueSource(strings = {"Y", "N"})
void jsonNullIsNotConvertedIntoAnEmptyString(String distinguish) throws
Exception {
String before =
System.getProperty(Const.HOP_EMPTY_STRING_DIFFERS_FROM_NULL);
try {
System.setProperty(Const.HOP_EMPTY_STRING_DIFFERS_FROM_NULL,
distinguish);
JsonInputData data = new JsonInputData();
data.outputRowMeta = new RowMeta();
data.outputRowMeta.addValueMeta(new ValueMetaString("nullValue"));
data.outputRowMeta.addValueMeta(new ValueMetaString("emptyValue"));
data.convertRowMeta =
data.outputRowMeta.cloneToType(ValueMetaString.TYPE_STRING);
data.totalpreviousfields = 0;
data.repeatedFields = new BitSet();
RowOutputConverter converter = new
RowOutputConverter(mock(ILogChannel.class));
Object[] row = converter.getRow(
new Object[0],
new Object[] {NullNode.getInstance(), TextNode.valueOf("")},
data);
assertNull(row[0]);
assertEquals("", row[1]);
if ("Y".equals(distinguish)) {
assertTrue(!data.outputRowMeta.getValueMeta(1).isNull(row[1]));
}
} finally {
if (before == null) {
System.clearProperty(Const.HOP_EMPTY_STRING_DIFFERS_FROM_NULL);
} else {
System.setProperty(Const.HOP_EMPTY_STRING_DIFFERS_FROM_NULL, before);
}
}
}
}
```
`JsonInputData.repeatedFields` was inspected at
`plugins/transforms/json/src/main/java/org/apache/hop/pipeline/transforms/jsoninput/JsonInputData.java:58`:
its declared type is `BitSet`, so the initialization above matches the real
API.
Run `json_test RowOutputConverterNullTest`. Expected RED with `Y`: actual
null column is `""`, because `ValueMetaBase.convertDataFromString` rewrites
null to `Const.NULL_STRING` under that setting. `N` is the compatibility case
and should remain green.
**GREEN (2 minutes):** in
`plugins/transforms/json/src/main/java/org/apache/hop/pipeline/transforms/jsoninput/reader/RowOutputConverter.java`,
immediately after `String strValue = getStringValue(value);`, insert:
```java
if (strValue == null) {
return null;
}
```
This preserves the JSON parser's null/NullNode/MissingNode result before
text-source conventions change it. Keep numeric conversion, trim processing,
and repeated-field handling unchanged. Run `json_test
'RowOutputConverterNullTest,RowOutputConverterTest,JsonInputTest'`; expected
GREEN. Add separate raw-null/MissingNode/repeat cases only after this first
slice is green; do not change `ValueMetaBase` globally. Commit the converter
and new test, message `Preserve JSON null when empty strings differ from null`.
### 2.2 Validate absent group keys in metadata (#2570, one 3-minute RED +
one 3-minute GREEN slice)
In
`plugins/transforms/json/src/test/java/org/apache/hop/pipeline/transforms/jsonoutputenhanced/JsonEOutputMetaTest.java`,
add imports `assertThrows`,
`org.apache.hop.core.exception.HopTransformException`,
`org.apache.hop.core.row.RowMeta`,
`org.apache.hop.core.row.value.ValueMetaString`,
`org.apache.hop.core.variables.Variables`, and `java.util.List`. Add:
```java
@Test
void missingGroupKeyNamesTheMissingField() {
JsonEOutputMeta meta = new JsonEOutputMeta();
meta.setOperationType(JsonEOutputMeta.OperationType.OUTPUT_VALUE);
meta.setKeyFields(List.of(new JsonEOutputKeyField("Field1")));
RowMeta row = new RowMeta();
row.addValueMeta(new ValueMetaString("Field2"));
HopTransformException error = assertThrows(
HopTransformException.class,
() -> meta.getFields(row, "json", null, null, new Variables(), new
MemoryMetadataProvider()));
assertTrue(error.getMessage().contains("Field1"));
}
```
Run `json_test 'JsonEOutputMetaTest#missingGroupKeyNamesTheMissingField'`;
prospective RED is the wrong exception or no useful field-named exception.
In
`plugins/transforms/json/src/main/java/org/apache/hop/pipeline/transforms/jsonoutputenhanced/JsonEOutputMeta.java`,
replace the lookup/add pair inside its `getFields()` key loop with:
```java
int index = rowMeta.indexOfValue(keyField.getFieldName());
if (index < 0) {
throw new HopTransformException(
BaseMessages.getString(PKG, "JsonEOutput.Error.GroupFieldNotFound",
keyField.getFieldName()));
}
IValueMeta vmi = rowMeta.getValueMeta(index).clone();
row.addValueMeta(vmi);
```
Add to
`plugins/transforms/json/src/main/resources/org/apache/hop/pipeline/transforms/jsonoutputenhanced/messages/messages_en_US.properties`:
```properties
JsonEOutput.Error.GroupFieldNotFound=Group key field ''{0}'' was not found
in the incoming row.
```
Run `json_test JsonEOutputMetaTest`; expected GREEN. Commit meta, test and
bundle, message `Validate enhanced JSON group keys during metadata discovery`.
### 2.3 Validate the same key before runtime metadata access (#2570, next
vertical slice)
The runtime
`plugins/transforms/json/src/main/java/org/apache/hop/pipeline/transforms/jsonoutputenhanced/JsonEOutput.java:onFirstRecord`
performs the same unsafe lookup **before** `initKeyFieldsPositionArray`. Move
validation before that loop; do not catch an `IndexOutOfBoundsException` after
the fact. The test harness and full runtime regression code are supplied in
Phase 4's test appendix, which is usable before streaming integration.
RED: `json_test
'JsonEOutputBehaviorTest#missingRuntimeGroupKeyIsAnActionableError'`; expected
the original index failure rather than a `HopException` naming the test's
configured key `gone`.
Minimal replacement inside the runtime key loop:
```java
int index = data.inputRowMeta.indexOfValue(keyFields.get(i).getFieldName());
if (index < 0) {
throw new HopException(
BaseMessages.getString(PKG, "JsonEOutput.Error.GroupFieldNotFound",
keyFields.get(i).getFieldName()));
}
data.outputRowMeta.addValueMeta(data.inputRowMeta.getValueMeta(index).clone());
```
Run the runtime test and `json_test
'JsonEOutputMetaTest,JsonEOutputBehaviorTest'`; require GREEN. Commit
runtime/test paths, message `Report missing enhanced JSON group keys before
runtime lookup`.
### 2.4 Preserve metadata on failure and share group-key validation (next
3-minute RED/GREEN slice)
After 2.2/2.3, add Appendix A's `missingKeyIsNamedDuringSchemaDiscovery`.
Run `json_test
'JsonEOutputBehaviorTest#missingKeyIsNamedDuringSchemaDiscovery'`; expected RED
because the existing `getFields()` clears the caller's row before discovering
that a key is missing. This is a separate defect from exception naming.
In
`plugins/transforms/json/src/main/java/org/apache/hop/pipeline/transforms/jsonoutputenhanced/JsonEOutputMeta.java`,
add the complete shared resolver:
```java
public int[] resolveKeyFieldIndexes(IRowMeta row) throws
HopTransformException {
int[] indexes = new int[keyFields.size()];
for (int i = 0; i < keyFields.size(); i++) {
String name = keyFields.get(i).getFieldName();
indexes[i] = row.indexOfValue(name);
if (indexes[i] < 0) {
throw new HopTransformException(
BaseMessages.getString(PKG,
"JsonEOutput.Error.GroupFieldNotFound", name));
}
}
return indexes;
}
```
At the start of `getFields()`'s non-file-only branch, replace its
`rowMeta`/clear/key-loop portion with:
```java
IRowMeta rowMeta = row.clone();
int[] keyIndexes = resolveKeyFieldIndexes(rowMeta);
row.clear();
for (int keyIndex : keyIndexes) {
row.addValueMeta(rowMeta.getValueMeta(keyIndex).clone());
}
```
Retain the following generated-JSON/size metadata. In
`plugins/transforms/json/src/main/java/org/apache/hop/pipeline/transforms/jsonoutputenhanced/JsonEOutput.java:onFirstRecord()`,
immediately after `data.inputRowMetaSize` is set, assign
`data.keysGroupIndexes = meta.resolveKeyFieldIndexes(data.inputRowMeta);`.
Replace the key-metadata loop with:
```java
for (int i = 0; i < meta.getKeyFields().size(); i++) {
data.outputRowMeta.addValueMeta(data.inputRowMeta.getValueMeta(data.keysGroupIndexes[i]).clone());
}
```
Replace the final `if (initKeyFieldsPositionArray(r)) return true; return
false;` with `return false;` and remove the now-unused
`initKeyFieldsPositionArray` method. Do not retain two competing key-index
resolvers. Run the same metadata-preservation test plus the runtime test;
require GREEN.
For pipeline Validate, add the same resolver call inside
`JsonEOutputMeta.check()`'s existing `prev != null && !prev.isEmpty()` branch:
```java
try {
resolveKeyFieldIndexes(prev);
} catch (HopTransformException e) {
remarks.add(new CheckResult(ICheckResult.TYPE_RESULT_ERROR,
e.getMessage(), transformMeta));
}
```
Add to
`plugins/transforms/json/src/test/java/org/apache/hop/pipeline/transforms/jsonoutputenhanced/JsonEOutputMetaTest.java`
(existing imports plus `java.util.ArrayList`,
`org.apache.hop.core.ICheckResult`):
```java
@Test
void validationReportsAMissingGroupKey() {
JsonEOutputMeta meta = new JsonEOutputMeta();
meta.setKeyFields(List.of(new JsonEOutputKeyField("gone")));
RowMeta previous = new RowMeta();
previous.addValueMeta(new ValueMetaString("payload"));
List<ICheckResult> remarks = new ArrayList<>();
meta.check(remarks, null, new TransformMeta("EnhancedJsonOutput", "json",
meta),
previous, new String[] {"upstream"}, new String[0], null,
new Variables(), new MemoryMetadataProvider());
assertTrue(remarks.stream().anyMatch(result ->
result.getType() == ICheckResult.TYPE_RESULT_ERROR &&
result.getText().contains("gone")));
}
```
Run `json_test 'JsonEOutputMetaTest#validationReportsAMissingGroupKey'`
before adding the check (RED, no named key remark), then after (GREEN). Commit
this shared validation/preservation slice separately, using both production
files, both test files and the existing message bundle; message `Share enhanced
JSON key validation without clearing metadata on failure`.
## Phase 3 — small GUI-backed features, after the UI helper PR
### 3.1 Literal root-array paths (#8444): failing transform test (3 minutes)
Add this complete method to
`plugins/transforms/json/src/test/java/org/apache/hop/pipeline/transforms/jsoninput/JsonInputTest.java`.
Use the existing imports and helpers. Add the configuration flag in 3.2, then
run the test **before** changing `createReader()`, so the behavioral RED is
visible rather than only a missing-getter compiler error.
```java
@Test
void literalRootArrayPathExtractsTheFirstValue() throws Exception {
String path = "$[0].similarity";
String input = "[{\"Id\":3902113,\"similarity\":0.95}]";
JsonInputField field = new JsonInputField("similarity");
field.setPath(path);
field.setType(IValueMeta.TYPE_NUMBER);
JsonInputMeta meta = createSimpleMeta("json", field);
meta.setResolveJsonPaths(false);
meta.setRemoveSourceField(true);
JsonInput transform = createJsonInput("json", meta, new Object[] {input});
List<Object[]> rows = new ArrayList<>();
transform.addRowListener(new RowAdapter() {
@Override
public void rowWrittenEvent(IRowMeta rowMeta, Object[] row) {
rows.add(row.clone());
}
});
try {
assertEquals(0, transform.getErrors(), "literal JSONPath must compile");
processRows(transform, 3);
assertEquals(0, transform.getErrors());
assertEquals(1, rows.size());
assertEquals(0.95d, rows.getFirst()[0]);
assertEquals(path, meta.getInputFields().getFirst().getPath());
} finally {
transform.dispose();
}
}
```
Command: `json_test
'JsonInputTest#literalRootArrayPathExtractsTheFirstValue'`. Expected behavioral
RED: the old `resolve()` transforms `$[0]` into a control character, so
initialization/extraction fails.
### 3.2 Persist the backwards-compatible flag and gate only JsonPath
resolution (two 3-minute slices)
In
`plugins/transforms/json/src/main/java/org/apache/hop/pipeline/transforms/jsoninput/JsonInputMeta.java`,
add the following imports, `@GuiPlugin` to the class, and members. Do **not**
change the transform ID `JsonInput`.
```java
import org.apache.hop.core.gui.plugin.GuiElementType;
import org.apache.hop.core.gui.plugin.GuiPlugin;
import org.apache.hop.core.gui.plugin.GuiWidgetElement;
import org.apache.hop.core.gui.plugin.GuiWidgetGroupType;
```
```java
public static final String GUI_PLUGIN_ELEMENT_PARENT_ID =
"JsonInput.JsonPathOptions";
@HopMetadataProperty(
key = "resolveJsonPaths",
defaultBoolean = true,
injectionKey = "RESOLVE_JSON_PATHS",
injectionKeyDescription = "JsonInput.Injection.RESOLVE_JSON_PATHS")
@GuiWidgetElement(
id = "resolveJsonPaths",
order = "0100",
type = GuiElementType.CHECKBOX,
label = "i18n::JsonInputMeta.ResolveJsonPaths.Label",
toolTip = "i18n::JsonInputMeta.ResolveJsonPaths.Tooltip",
parentId = GUI_PLUGIN_ELEMENT_PARENT_ID,
groupType = GuiWidgetGroupType.BOXES,
group = "JSONPath",
groupOrder = "0100")
private boolean resolveJsonPaths = true;
```
Lombok already exists on this class. `defaultBoolean=true` is essential:
metadata deserialization applies annotation defaults, not just constructor
defaults. Missing legacy XML/JSON property must retain **true**.
Add to
`plugins/transforms/json/src/main/resources/org/apache/hop/pipeline/transforms/jsoninput/messages/messages_en_US.properties`:
```properties
JsonInputMeta.ResolveJsonPaths.Label=Resolve variables in JSONPath
JsonInputMeta.ResolveJsonPaths.Tooltip=Clear this option to use extraction
paths literally, including root array indexes such as '$[0]'. When selected,
existing variable expressions such as '${JSON_PATH}' are resolved.
JsonInput.Injection.RESOLVE_JSON_PATHS=Resolve variables in JSONPath
expressions? (Y/N)
JsonInputDialog.PathOptions.TabTitle=JSONPath
```
Then, in
`plugins/transforms/json/src/main/java/org/apache/hop/pipeline/transforms/jsoninput/JsonInput.java:createReader()`,
replace only the unconditional `field.setPath(resolve(field.getPath()));` with:
```java
if (meta.isResolveJsonPaths()) {
field.setPath(resolve(field.getPath()));
}
```
Keep its defensive `new JsonInputField(...)` copy. Do not change file/URL
resolution or `Variables.resolve()` globally. Run:
```bash
json_test 'JsonInputTest#literalRootArrayPathExtractsTheFirstValue'
json_test 'JsonInputTest#testJsonInputPathResolutionSuccess'
json_test 'JsonInputTest#testJsonInputMetaInputFieldsNotOverwritten'
```
Expected GREEN for all three, including unchanged `${PARAM_PATH}` behavior
with default true. Stage the runtime, meta, test and bundle paths; commit
message `Allow literal JSONPath expressions without changing legacy resolution`
**only after the GUI slice below is green**. The test-first flag/runtime/GUI
work is one vertical feature; do not ship a file-only setting.
### 3.3 Put the new flag in one grouped tab, not a whole-dialog redesign (3
minutes)
In
`plugins/transforms/json/src/main/java/org/apache/hop/pipeline/transforms/jsoninput/JsonInputDialog.java`,
add imports:
```java
import org.apache.hop.ui.core.gui.GuiCompositeWidgets;
import org.apache.hop.ui.core.gui.GuiCompositeWidgetsAdapter;
```
Add the field and complete method:
```java
private GuiCompositeWidgets pathWidgets;
private void addPathOptionsTab() {
CTabItem tab = new CTabItem(wTabFolder, SWT.NONE);
tab.setText(BaseMessages.getString(PKG,
"JsonInputDialog.PathOptions.TabTitle"));
Composite parent = new Composite(wTabFolder, SWT.NONE);
PropsUi.setLook(parent);
parent.setLayout(new FormLayout());
pathWidgets = GuiCompositeWidgets.addScrolledComposite(
parent, variables, null, null,
JsonInputMeta.GUI_PLUGIN_ELEMENT_PARENT_ID, input);
pathWidgets.setWidgetsListener(new GuiCompositeWidgetsAdapter() {
@Override
public void widgetModified(GuiCompositeWidgets widgets, Control control,
String widgetId) {
input.setChanged();
}
});
tab.setControl(parent);
}
```
Call `addPathOptionsTab();` immediately after `addContentTab();` in
`open()`. At the start of `getInfo(JsonInputMeta in)` add:
```java
pathWidgets.getWidgetsContents(in,
JsonInputMeta.GUI_PLUGIN_ELEMENT_PARENT_ID);
```
The existing OK/Cancel/Preview bar stays pinned below `wTabFolder`; do not
add shell-level `FormAttachment` rows. Preview uses `getInfo(oneMeta)`, so it
receives the flag too.
**GUI RED/GREEN:** create
`plugins/transforms/json/src/test/java/org/apache/hop/pipeline/transforms/jsoninput/JsonInputPathOptionsDialogTest.java`,
mandatory header plus:
```java
package org.apache.hop.pipeline.transforms.jsoninput;
import static org.junit.jupiter.api.Assertions.assertFalse;
import static org.junit.jupiter.api.Assertions.assertTrue;
import java.lang.reflect.Field;
import org.apache.hop.core.gui.plugin.GuiRegistry;
import org.apache.hop.core.gui.plugin.GuiWidgetElement;
import org.apache.hop.core.variables.Variables;
import org.apache.hop.i18n.BaseMessages;
import org.apache.hop.metadata.serializer.memory.MemoryMetadataProvider;
import org.apache.hop.pipeline.PipelineMeta;
import org.apache.hop.pipeline.transform.TransformMeta;
import org.apache.hop.ui.testing.SwtBotTestBase;
import org.junit.jupiter.api.BeforeAll;
import org.junit.jupiter.api.Tag;
import org.junit.jupiter.api.Test;
@Tag("uitest")
class JsonInputPathOptionsDialogTest extends SwtBotTestBase {
@BeforeAll
static void registerWidgets() {
GuiRegistry registry = GuiRegistry.getInstance();
for (Field field : JsonInputMeta.class.getDeclaredFields()) {
GuiWidgetElement element = field.getAnnotation(GuiWidgetElement.class);
if (element != null) {
registry.addGuiWidgetElement(JsonInputMeta.class.getName(), element,
field);
}
}
}
private PipelineMeta pipeline(JsonInputMeta meta) {
PipelineMeta pipeline = new PipelineMeta();
pipeline.setMetadataProvider(new MemoryMetadataProvider());
pipeline.addTransform(new TransformMeta("JsonInput", "json", meta));
return pipeline;
}
@Test
void okPersistsLiteralPathMode() {
JsonInputMeta meta = new JsonInputMeta();
PipelineMeta pipeline = pipeline(meta);
withDialog(
parent -> new JsonInputDialog(parent, new Variables(), meta,
pipeline).open(),
bot -> {
var dialog = bot.shell(BaseMessages.getString(JsonInputMeta.class,
"JsonInputDialog.DialogTitle")).activate().bot();
dialog.cTabItem("JSONPath").activate();
dialog.checkBox("Resolve variables in JSONPath").deselect();
dialog.button(buttonLabel("System.Button.OK")).click();
});
assertFalse(meta.isResolveJsonPaths());
}
@Test
void cancelDoesNotPersistLiteralPathMode() {
JsonInputMeta meta = new JsonInputMeta();
PipelineMeta pipeline = pipeline(meta);
withDialog(
parent -> new JsonInputDialog(parent, new Variables(), meta,
pipeline).open(),
bot -> {
var dialog = bot.shell(BaseMessages.getString(JsonInputMeta.class,
"JsonInputDialog.DialogTitle")).activate().bot();
dialog.cTabItem("JSONPath").activate();
dialog.checkBox("Resolve variables in JSONPath").deselect();
dialog.button(buttonLabel("System.Button.Cancel")).click();
});
assertTrue(meta.isResolveJsonPaths());
}
}
```
Run `json_ui_test JsonInputPathOptionsDialogTest` before adding the tab
(expected missing-tab/checkbox failure), and again after (expected GREEN). Run
in the normal en-US test environment used by existing UI tests; translate
lookup labels with `BaseMessages` if this environment is customized. Require
both tests to execute, not be headless-skipped.
### 3.4 Verify old and new serialization behavior (3 minutes)
In
`plugins/transforms/json/src/test/java/org/apache/hop/pipeline/transforms/jsoninput/JsonInputMetaTest.java`,
add XML round-trip cases following its existing `XmlMetadataUtil` pattern,
with this complete reusable method in `JsonInputTest.java` to test the
**behavior** of a deserialized setting (existing imports plus `XmlHandler`,
`XmlMetadataUtil`, `MemoryMetadataProvider`, `TransformMeta`):
```java
@Test
void serializedLiteralPathModeStillExtractsRootArray() throws Exception {
JsonInputMeta original = new JsonInputMeta();
original.setResolveJsonPaths(false);
String xml = "<transform>" +
org.apache.hop.metadata.serializer.xml.XmlMetadataUtil.serializeObjectToXml(original)
+
"</transform>";
JsonInputMeta loaded = new JsonInputMeta();
org.apache.hop.metadata.serializer.xml.XmlMetadataUtil.deSerializeFromXml(
org.apache.hop.core.xml.XmlHandler.loadXmlString(xml,
org.apache.hop.pipeline.transform.TransformMeta.XML_TAG),
JsonInputMeta.class, loaded,
new
org.apache.hop.metadata.serializer.memory.MemoryMetadataProvider());
assertFalse(loaded.isResolveJsonPaths());
JsonInputField field = new JsonInputField("value");
field.setPath("$[0].value");
field.setType(IValueMeta.TYPE_STRING);
loaded.setInputFields(new ArrayList<>(List.of(field)));
loaded.setInFields(true);
loaded.setFieldValue("json");
loaded.setRemoveSourceField(true);
JsonInput transform = createJsonInput("json", loaded,
new Object[] {"[{\"value\":\"ok\"}]"});
RowComparatorListener rows = new RowComparatorListener(new Object[]
{"ok"});
transform.addRowListener(rows);
try {
processRows(transform, 3);
assertEquals(0, transform.getErrors());
assertEquals(1, rows.rowNbr);
} finally {
transform.dispose();
}
}
```
For absent-property default, deserialize `<transform><fields/></transform>`
into a fresh meta and assert `isResolveJsonPaths()` is true, then run the
already-supplied `${PARAM_PATH}` behavior test. Command `json_test
'JsonInputMetaTest,JsonInputTest'`; expected GREEN. Commit the extra
persistence test separately, message `Verify JSONPath mode survives pipeline
serialization`.
Update
`docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/jsoninput.adoc`
with:
```adoc
=== JSONPath tab
*Resolve variables in JSONPath* is enabled by default, including for
existing pipelines. Clear it to use every extraction Path literally. This is
required for root-array paths such as `$[0].similarity`, whose `$[0]` otherwise
has a meaning in Hop variable resolution. Clearing it also disables
`${VARIABLE}` substitution inside extraction paths. File and URL settings
continue to resolve variables normally.
```
### 3.5 Verify #5953 on the existing PR rather than implementing it again (3
minutes)
After #8588 has been approved and integrated, run:
```bash
json_ui_test
'JsonInputDialogKeepsValuesTest,JsonNormalizeInputDialogKeepsValuesTest'
./tools/with-isolated-display.sh ./mvnw -pl rcp -Puitest \
-DskipTests=false '-Dtest=ComboItemsTest,PreviousFieldsTest' test
```
Exact introduced test paths:
-
`plugins/transforms/json/src/test/java/org/apache/hop/pipeline/transforms/jsoninput/JsonInputDialogKeepsValuesTest.java`
-
`plugins/transforms/json/src/test/java/org/apache/hop/pipeline/transforms/jsonnormalize/JsonNormalizeInputDialogKeepsValuesTest.java`
- `rcp/src/test/java/org/apache/hop/ui/core/widget/ComboItemsTest.java`
-
`rcp/src/test/java/org/apache/hop/ui/pipeline/transform/PreviousFieldsTest.java`
Expected GREEN after integration; error shown once and `id_json` still saved
on OK. Test a successful **empty** upstream separately from a failing upstream;
`PreviousFields.isAvailable()` distinguishes them. The source field combo must
retain user text in both cases, but an explicit grid-replacement command must
follow the agreed empty-result policy. Do not paper over failed UI tests by
clearing fields.
## Phase 4 — enhanced-output grouping, Get Fields, and streaming PR
acceptance
The next section supplies complete proposed code/test members for
#5167/#5313/#4344 and a shared behavioral harness. Source APIs for grouped
widgets and button listeners were independently read; all snippets remain
uncompiled in this planning turn. Copy **one test method plus its harness** for
the current vertical slice, not all future tests at once.
### 4.1 Adopt the streaming PR with explicit acceptance gates (five 3-minute
setup/review slices)
**4.1a — file contract review:** read
`plugins/transforms/json/src/main/java/org/apache/hop/pipeline/transforms/jsonoutputenhanced/JsonEOutput.java`
and
`plugins/transforms/json/src/main/java/org/apache/hop/pipeline/transforms/jsonoutputenhanced/JsonEOutputData.java`
from integrated #8713. Confirm `streamFileRows`, `collectGroupItems`,
`pendingFileItem`, `fileItemCount`, `fileGenerator`, `finishGroup`,
`writeFileItem`, `finishFile` match the reviewed snapshot; re-plan this section
if they do not. Without keys or forced merging, file-only output should retain
at most one pending item plus the row being written. Grouped output still
retains the current group; BOTH and forced merging are not constant-memory
promises.
**4.1b — correctness RED/GREEN:** install only Appendix A's harness plus
`bothKeepsRowShapeAndWritesEveryGroup`. Run `json_test
'JsonEOutputBehaviorTest#bothKeepsRowShapeAndWritesEveryGroup'`. On the
reviewed PR this is expected to fail because `outputRow()` always appends two
cells even when its metadata has no JSON-size field. Replace its
allocation/population block with:
```java
Object[] additionalRowFields =
Utils.isEmpty(meta.getJsonSizeFieldName())
? new Object[] {data.jsonSerialized}
: new Object[] {data.jsonSerialized, data.jsonLength};
```
Keep `RowDataUtil.addRowData`, counters and `putRow` unchanged. Rerun the
same test; require GREEN and all groups present in the file. Commit runtime and
Appendix A's test path, message `Match enhanced JSON output row width to its
metadata`.
**4.1c — PR fixture validation:** use the exact PR-introduced files, do not
create a parallel streaming implementation:
- `integration-tests/json/0018-enhanced-json-output-streaming.hpl`
- `integration-tests/json/main-0018-enhanced-json-output-streaming.hwf`
- `integration-tests/json/0019-enhanced-json-output-grouped-file.hpl`
- `integration-tests/json/main-0019-enhanced-json-output-grouped-file.hwf`
Run the integration command in Phase 7 with
`TEST_FILTER='0018-enhanced-json-output-streaming,0019-enhanced-json-output-grouped-file'`;
expected both workflows recorded as passed. #8713 deliberately changes split
counting to complete file items: rows without keys, groups with keys; never cut
one group across files. Review and disclose this compatibility change.
**4.1d — memory evidence:** the reviewed 0018 file is 126,888,906 bytes
while `-Xmx128m` is 134,217,728 bytes. It catches retained-tree overhead, but
is not a file larger than the entire maximum heap. Correct the inaccurate
“several times larger” claim in
`integration-tests/json/0018-enhanced-json-output-streaming.hpl`, or strengthen
the existing workflow with the exact larger-row calculation in Phase 7. Do not
claim bounded memory for an unlimited single group. Cancellation/write-error
cleanup must close the generator before the writer; Appendix C supplies a test
and minimal cleanup.
**4.1e — #2569 attachment:** attempted read-only retrieval of
`https://github.com/apache/hop/files/10979443/json.enhanced.test.zip` was
blocked by an unanswered tool approval, so its contents are **not verified**.
Obtain the attachment from the user/maintainer with approval during
implementation; do not bypass the approval or guess its settings. Open its
`.hpl` in the isolated GUI, run unchanged before/after the integrated PR,
compare parsed JSON (not whitespace), and preserve it as a regression fixture
only if it exposes behavior not covered by 0019. Leave #2569 open until the
sample-specific result is recorded.
**Commit discipline for #5167:** first add Appendix A's
`keySuggestionsExcludeLiveOutputAndExistingKeys` and run `json_test
'JsonEOutputBehaviorTest#keySuggestionsExcludeLiveOutputAndExistingKeys'`
(initial missing-helper compiler RED); add only `suggestKeyFieldNames`, rerun
(GREEN). Then add the real GUI test in Appendix C and observe
missing-button/geometry RED; wire the grouped button/table below and rerun GUI
+ selection test. Commit the meta, dialog, GUI/behavior tests, and message
bundle, message `Offer unmapped incoming fields as enhanced JSON group keys`.
Do not use stale `input.getOutputFields()` to exclude unsaved live table
selections. Existing key aliases must survive repeat clicks and Cancel must not
mutate the metadata list.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
