mattcasters commented on issue #8711: URL: https://github.com/apache/hop/issues/8711#issuecomment-5918849976
## JSON implementation plan — part 2 of 3 [Plan index / part 1](https://github.com/apache/hop/issues/8711#issuecomment-5918849385) --- ### 4.2 Scoped annotated Group Key controls (#5167) and the file-format GUI island Add imports in `plugins/transforms/json/src/main/java/org/apache/hop/pipeline/transforms/jsonoutputenhanced/JsonEOutputMeta.java`: ```java import org.apache.hop.core.gui.plugin.GuiElementType; import org.apache.hop.core.gui.plugin.GuiPlugin; import org.apache.hop.core.gui.plugin.GuiWidgetElement; import org.apache.hop.core.gui.plugin.GuiWidgetGroupType; ``` Add `@GuiPlugin` to the Meta class and these members. Existing Lombok getters/setters cover the new boolean: ```java public static final String KEY_GUI_PARENT = "JsonEOutput.GroupKeys"; public static final String FORMAT_GUI_PARENT = "JsonEOutput.FileFormat"; public static final String WIDGET_GET_KEYS = "getKeyFields"; @GuiWidgetElement( id = WIDGET_GET_KEYS, order = "0100", type = GuiElementType.BUTTON, label = "i18n::JsonEOutputDialog.Get.Button", toolTip = "i18n::JsonEOutputDialog.GetKeys.Tooltip", parentId = KEY_GUI_PARENT, groupType = GuiWidgetGroupType.BOXES, group = "i18n::JsonEOutputDialog.KeyConfigTab.TabTitle", groupOrder = "0100") public void getKeyFieldsFromPrevious() { // The dialog's composite-button listener operates on the live tables. } @HopMetadataProperty( key = "newline_delimited", injectionKey = "NEWLINE_DELIMITED", injectionKeyDescription = "JsonEOutput.Injection.NEWLINE_DELIMITED") @GuiWidgetElement( id = "newlineDelimited", order = "0100", type = GuiElementType.CHECKBOX, label = "i18n::JsonEOutputDialog.NdJson.Label", toolTip = "i18n::JsonEOutputDialog.NdJson.Tooltip", parentId = FORMAT_GUI_PARENT, groupType = GuiWidgetGroupType.BOXES, group = "i18n::JsonEOutputDialog.FileFormat.TabTitle", groupOrder = "0100") private boolean newlineDelimited; ``` In `plugins/transforms/json/src/main/java/org/apache/hop/pipeline/transforms/jsonoutputenhanced/JsonEOutputDialog.java`, import `GuiCompositeWidgets` and `GuiCompositeWidgetsAdapter` from `org.apache.hop.ui.core.gui`, and add: ```java public static final String GUI_PLUGIN_ELEMENT_PARENT_ID = JsonEOutputMeta.FORMAT_GUI_PARENT; private GuiCompositeWidgets keyWidgets; private GuiCompositeWidgets formatWidgets; ``` Keep the existing Group Key tab and its two `ColumnInfo` definitions. Replace only its `wKeyFields` construction/layout block with: ```java wKeyConfigComp.setLayout(keyConfigLayout); keyWidgets = GuiCompositeWidgets.addScrolledComposite( wKeyConfigComp, variables, null, null, JsonEOutputMeta.KEY_GUI_PARENT, input, widgets -> { keyWidgets = widgets; widgets.registerExtraGroup( BaseMessages.getString(PKG, "JsonEOutputDialog.KeyConfigTab.TabTitle"), "0100", null, this::createKeyFieldsTable); }); keyWidgets.setCompositeButtonsListener(source -> getKeyFieldsFromPrevious()); ``` Add these complete methods. The only manual attachments are for the custom TableView inside its registered group: ```java private void createKeyFieldsTable(Composite parent) { wKeyFields = new TableView( variables, parent, SWT.BORDER | SWT.FULL_SELECTION | SWT.MULTI, keyColInf, input.getKeyFields().size(), event -> input.setChanged(), props); Control button = keyWidgets.getWidgetsMap().get(JsonEOutputMeta.WIDGET_GET_KEYS); FormData fd = new FormData(); fd.left = new FormAttachment(0, 0); fd.right = new FormAttachment(100, 0); fd.top = new FormAttachment(button, margin); fd.bottom = new FormAttachment(100, 0); wKeyFields.setLayoutData(fd); } static List<String> suggestKeyFieldNames( IRowMeta previous, List<String> outputNames, List<String> existingKeys) { java.util.Set<String> excluded = new java.util.HashSet<>(outputNames); excluded.addAll(existingKeys); List<String> names = new ArrayList<>(); for (int i = 0; i < previous.size(); i++) { String name = previous.getValueMeta(i).getName(); if (excluded.add(name)) { names.add(name); } } return names; } private void getKeyFieldsFromPrevious() { try { IRowMeta previous = pipelineMeta.getPrevTransformFields(variables, transformName); if (previous == null) { return; } List<String> outputNames = new ArrayList<>(); for (TableItem item : wFields.getNonEmptyItems()) { outputNames.add(item.getText(1)); } List<String> existingKeys = new ArrayList<>(); for (TableItem item : wKeyFields.getNonEmptyItems()) { existingKeys.add(item.getText(1)); } List<String> additions = suggestKeyFieldNames(previous, outputNames, existingKeys); for (String name : additions) { TableItem item = new TableItem(wKeyFields.table, SWT.NONE); item.setText(1, name); item.setText(2, name); } if (!additions.isEmpty()) { wKeyFields.removeEmptyRows(); wKeyFields.setRowNums(); wKeyFields.optWidth(true); input.setChanged(); } } catch (HopException e) { new ErrorDialog( shell, BaseMessages.getString(PKG, "System.Dialog.GetFieldsFailed.Title"), BaseMessages.getString(PKG, "System.Dialog.GetFieldsFailed.Message"), e); } } ``` For NDJSON, add one tab to the existing `wTabFolder`, without touching the existing General/Fields/Additional tabs: ```java CTabItem formatTab = new CTabItem(wTabFolder, SWT.NONE); formatTab.setText(BaseMessages.getString(PKG, "JsonEOutputDialog.FileFormat.TabTitle")); Composite formatComp = new Composite(wTabFolder, SWT.NONE); PropsUi.setLook(formatComp); formatComp.setLayout(new FormLayout()); formatWidgets = GuiCompositeWidgets.addScrolledComposite( formatComp, variables, null, null, GUI_PLUGIN_ELEMENT_PARENT_ID, input); formatWidgets.setWidgetsListener( new GuiCompositeWidgetsAdapter() { @Override public void widgetModified( GuiCompositeWidgets widgets, Control control, String widgetId) { input.setChanged(); } }); formatTab.setControl(formatComp); ``` In `getInfo(JsonEOutputMeta jsometa)`, add: ```java formatWidgets.getWidgetsContents(jsometa, GUI_PLUGIN_ELEMENT_PARENT_ID); ``` At the end of `updateOperation()`, add: ```java if (formatWidgets != null) { for (Control control : formatWidgets.getWidgetsMap().values()) { control.setEnabled(activeFile); } for (Control control : formatWidgets.getLabelsMap().values()) { control.setEnabled(activeFile); } } ``` No new hand-laid-out shell rows and no whole-dialog migration are needed. GUI tests must register both annotated fields **and the annotated method**, since normal test initialization does not populate the GUI registry. ## Phase 5 — opt-in NDJSON after streaming integration ### 5.1 Lock the NDJSON contract before adding its flag (3 minutes) Default is regular JSON; absence of persisted `newline_delimited` is false. Append in regular JSON keeps raw document concatenation. NDJSON is file-only/BOTH, UTF-8, compact, one LF-terminated item per physical line, with grouping handled as one completed group per line. JSON block names wrap each line; there is no outer file array. Forced arrays affect the nested group value only, not a per-line outer array. Reject Pretty Print or non-UTF-8 before opening a file; never override a user's settings silently. Existing append targets must already be valid UTF-8 NDJSON and end in LF; the boundary check does not validate historical contents. Parallel writers to one target are unsupported. These are proposed explicit semantics; obtain maintainer agreement for the new mode before changing existing pipelines. **Task order (2–5 minutes each):** (1) add config + GUI test scaffold, (2) add `ndjsonAppendEscapesEmbeddedNewlines`, run behavioral RED, implement writer branch, rerun GREEN and commit; (3) add `ndjsonSplitsRecordsWithoutOuterArrays`, RED/GREEN and commit; (4) add `ndjsonWrapsEachCompleteGroup`, RED/GREEN and commit; (5) add pretty/encoding rejection test, RED/GREEN and commit; (6) add append-boundary rejection, RED/GREEN and commit. All runtime edits are in `plugins/transforms/json/src/main/java/org/apache/hop/pipeline/transforms/jsonoutputenhanced/JsonEOutput.java`; the flag is in `plugins/transforms/json/src/main/java/org/apache/hop/pipeline/transforms/jsonoutputenhanced/JsonEOutputMeta.java`; test methods are in Appendix A's exact path; new labels are in Appendix B's exact bundle path. For each method run its literal `json_test` selector from Phase 7 both before and after its minimal runtime change. Expected RED is missing delimiter, wrong split/group/block framing, accepted invalid config, or mutation of an invalid append target respectively; GREEN is zero failures/errors for the selected test, exit 0. The exact commands are listed in Phase 7. Stage these exact runtime/meta/test/resource paths and commit the green slice with `Add compact NDJSON output`, `Split NDJSON on complete records`, `Wrap each NDJSON group independently`, `Reject incompatible NDJSON settings`, or `Validate NDJSON append boundaries` respectively. ### 5.2 Complete minimal NDJSON runtime additions In `plugins/transforms/json/src/main/java/org/apache/hop/pipeline/transforms/jsonoutputenhanced/JsonEOutput.java:init()`, inside `if (super.init())`, **before opening any file**, add: ```java if (meta.isNewlineDelimited() && meta.getOperationType() != JsonEOutputMeta.OperationType.OUTPUT_VALUE && (meta.isJsonPrettified() || !Const.UTF_8.equalsIgnoreCase(resolve(meta.getEncoding())))) { logError(BaseMessages.getString(PKG, "JsonEOutput.Error.NdJsonSettings")); setErrors(1); stopAll(); return false; } ``` At the beginning of #8713's `writeFileItem(JsonNode item)`: ```java if (meta.isNewlineDelimited()) { writeNdJsonItem(item); return; } ``` Add: ```java private void writeNdJsonItem(JsonNode item) throws HopException { try { if (!openNewFile()) { throw new HopTransformException( BaseMessages.getString(PKG, "JsonEOutput.Error.OpenNewFile", buildFilename())); } if (data.fileGenerator == null) { data.fileGenerator = fileMapper.getFactory().createGenerator(data.writer); data.fileGenerator.disable(JsonGenerator.Feature.AUTO_CLOSE_TARGET); data.fileGenerator.setPrettyPrinter(null); data.fileGenerator.setRootValueSeparator(null); } String block = resolve(meta.getJsonBloc()); if (!Utils.isEmpty(block)) { data.fileGenerator.writeStartObject(); data.fileGenerator.writeFieldName(block); } data.fileGenerator.writeTree(item); if (!Utils.isEmpty(block)) { data.fileGenerator.writeEndObject(); } data.fileGenerator.writeRaw('\n'); } catch (IOException e) { throw new HopTransformException( BaseMessages.getString(PKG, "JsonEOutput.Error.Writing"), e); } data.fileItemCount++; if (!data.isOutputValue) { incrementLinesOutput(); } int split = meta.getFileSettings().getSplitOutputAfter(); if (split > 0 && data.fileItemCount >= split) { finishFile(); } } private void validateNdJsonAppend(String filename) throws IOException { if (!meta.isNewlineDelimited() || !meta.getFileSettings().isFileAppended()) { return; } try (FileObject file = HopVfs.getFileObject(filename, variables)) { if (!file.exists()) { return; } long size = file.getContent().getSize(); if (size == 0) { return; } try (var stream = HopVfs.getInputStream(file)) { stream.skipNBytes(size - 1); if (stream.read() != '\n') { throw new IOException( BaseMessages.getString(PKG, "JsonEOutput.Error.NdJsonAppendBoundary", filename)); } } } } ``` In `openNewFile()`, immediately after `createParentFolder(filename)`, and before adding the result file/opening the output stream: ```java validateNdJsonAppend(filename); ``` In #8713's `finishFile()`, immediately after the existing zero-item early return: ```java if (meta.isNewlineDelimited()) { try { data.fileGenerator.close(); } catch (IOException e) { throw new HopTransformException( BaseMessages.getString(PKG, "JsonEOutput.Error.Writing"), e); } data.fileGenerator = null; data.pendingFileItem = null; data.fileItemCount = 0; if (!closeFile()) { throw new HopTransformException( BaseMessages.getString(PKG, "JsonEOutput.Error.ClosingFile", data.openedFilename)); } return; } ``` These additions reuse #8713's grouping, VFS writer, split counter, mapper, and generator state. They do not introduce another streaming implementation. Generator/root-separator/write-tree APIs were checked with `javap` against pinned Jackson **2.21.5**, managed in `lib/pom.xml`; no Jackson dependency addition is needed. ## Appendix A — complete enhanced-output behavioral harness and vertical tests Create `plugins/transforms/json/src/test/java/org/apache/hop/pipeline/transforms/jsonoutputenhanced/JsonEOutputBehaviorTest.java` using the complete class below. Apply its methods **one vertical RED→GREEN slice at a time**, not all future tests before all implementation. The harness substitutes only pipeline transport/logging; JSON construction, grouping, serialization, initialization, and RAM-VFS file I/O remain real. ```java /* * Licensed to the Apache Software Foundation (ASF) under one or more * contributor license agreements. See the NOTICE file distributed with * this work for additional information regarding copyright ownership. * The ASF licenses this file to You under the Apache License, Version 2.0 * (the "License"); you may not use this file except in compliance with * the License. You may obtain a copy of the License at * * http://www.apache.org/licenses/LICENSE-2.0 * * Unless required by applicable law or agreed to in writing, software * distributed under the License is distributed on an "AS IS" BASIS, * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. * See the License for the specific language governing permissions and * limitations under the License. */ package org.apache.hop.pipeline.transforms.jsonoutputenhanced; import static org.junit.jupiter.api.Assertions.assertEquals; import static org.junit.jupiter.api.Assertions.assertFalse; import static org.junit.jupiter.api.Assertions.assertThrows; import static org.junit.jupiter.api.Assertions.assertTrue; import static org.mockito.ArgumentMatchers.any; import static org.mockito.Mockito.when; import com.fasterxml.jackson.databind.DeserializationFeature; import com.fasterxml.jackson.databind.JsonNode; import java.nio.charset.StandardCharsets; import java.util.ArrayList; import java.util.Arrays; import java.util.List; import java.util.UUID; import org.apache.commons.vfs2.FileObject; import org.apache.hop.core.HopClientEnvironment; import org.apache.hop.core.RowMetaAndData; import org.apache.hop.core.exception.HopException; import org.apache.hop.core.exception.HopTransformException; import org.apache.hop.core.json.HopJson; import org.apache.hop.core.logging.ILoggingObject; import org.apache.hop.core.row.IRowMeta; import org.apache.hop.core.row.RowMeta; import org.apache.hop.core.row.value.ValueMetaString; import org.apache.hop.core.variables.Variables; import org.apache.hop.core.vfs.HopVfs; import org.apache.hop.pipeline.transforms.mock.TransformMockHelper; import org.junit.jupiter.api.AfterEach; import org.junit.jupiter.api.BeforeAll; import org.junit.jupiter.api.Test; class JsonEOutputBehaviorTest { private final String base = "ram:///enhanced-json-" + UUID.randomUUID(); @BeforeAll static void initialize() throws HopException { HopClientEnvironment.init(); } @AfterEach void removeFiles() throws Exception { try (FileObject file = HopVfs.getFileObject(base)) { if (file.exists()) { file.deleteAll(); } } } private JsonEOutputMeta meta(JsonEOutputMeta.OperationType operation) { JsonEOutputMeta meta = new JsonEOutputMeta(); meta.setOperationType(operation); meta.setOutputValue("rows"); meta.setJsonBloc(""); meta.setEncoding("UTF-8"); JsonEOutputField field = new JsonEOutputField(); field.setFieldName("payload"); field.setElementName("payload"); meta.getOutputFields().add(field); meta.getFileSettings().setFileName(base + "/out"); meta.getFileSettings().setExtension("json"); meta.getFileSettings().setCreateParentFolder(true); meta.getFileSettings().setDoNotOpenNewFileInit(true); return meta; } private static IRowMeta rowMeta(String... names) { IRowMeta row = new RowMeta(); for (String name : names) { row.addValueMeta(new ValueMetaString(name)); } return row; } private static String read(String path) throws Exception { try (FileObject file = HopVfs.getFileObject(path); var input = HopVfs.getInputStream(file)) { return new String(input.readAllBytes(), StandardCharsets.UTF_8); } } private static JsonNode parse(String json) throws Exception { return HopJson.newMapper() .enable(DeserializationFeature.FAIL_ON_TRAILING_TOKENS) .readTree(json); } private static String payload(String value) { return HopJson.newMapper().createObjectNode().put("payload", value).toString(); } private static final class Harness implements AutoCloseable { final TransformMockHelper<JsonEOutputMeta, JsonEOutputData> helper; final List<RowMetaAndData> written = new ArrayList<>(); final JsonEOutput transform; Harness(JsonEOutputMeta meta, IRowMeta inputMeta, Object[]... rows) { helper = new TransformMockHelper<>("enhanced JSON", JsonEOutputMeta.class, JsonEOutputData.class); when(helper.logChannelFactory.create(any(), any(ILoggingObject.class))) .thenReturn(helper.iLogChannel); when(helper.pipeline.isRunning()).thenReturn(true); when(helper.transformMeta.getTransform()).thenReturn(meta); var iterator = Arrays.asList(rows).iterator(); transform = new JsonEOutput( helper.transformMeta, meta, new JsonEOutputData(), 0, helper.pipelineMeta, helper.pipeline) { @Override public Object[] getRow() { return iterator.hasNext() ? iterator.next() : null; } @Override public void putRow(IRowMeta rowMeta, Object[] row) { written.add(new RowMetaAndData(rowMeta.clone(), row.clone())); } }; transform.setInputRowMeta(inputMeta); } void run() throws HopException { assertTrue(transform.init()); while (transform.processRow()) { // Drain the real transform through EOF. } } @Override public void close() { try { transform.dispose(); } finally { helper.cleanUp(); } } } @Test void missingRuntimeGroupKeyIsAnActionableError() throws Exception { JsonEOutputMeta meta = meta(JsonEOutputMeta.OperationType.OUTPUT_VALUE); meta.getKeyFields().add(new JsonEOutputKeyField("gone")); try (Harness h = new Harness(meta, rowMeta("payload"), new Object[] {"x"})) { assertTrue(h.transform.init()); HopException error = assertThrows(HopException.class, h.transform::processRow); assertTrue(error.getMessage().contains("gone"), error.getMessage()); } } @Test void missingKeyIsNamedDuringSchemaDiscovery() { JsonEOutputMeta meta = meta(JsonEOutputMeta.OperationType.OUTPUT_VALUE); meta.getKeyFields().add(new JsonEOutputKeyField("gone")); IRowMeta input = rowMeta("payload"); HopTransformException error = assertThrows( HopTransformException.class, () -> meta.getFields(input, "json", null, null, new Variables(), null)); assertTrue(error.getMessage().contains("gone"), error.getMessage()); assertEquals("payload", input.getValueMeta(0).getName()); } @Test void bothKeepsRowShapeAndWritesEveryGroup() throws Exception { JsonEOutputMeta meta = meta(JsonEOutputMeta.OperationType.BOTH); JsonEOutputKeyField key = new JsonEOutputKeyField("grp"); key.setElementName("group"); meta.getKeyFields().add(key); try (Harness h = new Harness(meta, rowMeta("grp", "payload"), new Object[] {"a", "x"}, new Object[] {"a", "y"}, new Object[] {"b", "z"})) { h.run(); assertEquals(2, h.written.size()); for (RowMetaAndData row : h.written) { assertEquals(row.getRowMeta().size(), row.getData().length); assertTrue(row.getRowMeta().indexOfValue("grp") >= 0); assertEquals(-1, row.getRowMeta().indexOfValue("group")); } } JsonNode file = parse(read(base + "/out.json")); assertEquals(2, file.size()); assertEquals("a", file.get(0).get("group").asText()); assertEquals(2, file.get(0).get("rows").size()); assertEquals("z", file.get(1).get("rows").get("payload").asText()); } @Test void keySuggestionsExcludeLiveOutputAndExistingKeys() { assertEquals( List.of("grp"), JsonEOutputDialog.suggestKeyFieldNames( rowMeta("payload", "grp", "already"), List.of("payload"), List.of("already"))); assertEquals( List.of(), JsonEOutputDialog.suggestKeyFieldNames( rowMeta("payload", "grp"), List.of("payload"), List.of("grp"))); } @Test void legacyAppendHasNoNewSeparators() throws Exception { JsonEOutputMeta meta = meta(JsonEOutputMeta.OperationType.WRITE_TO_FILE); meta.getFileSettings().setFileAppended(true); try (Harness h = new Harness(meta, rowMeta("payload"), new Object[] {"x"})) { h.run(); } try (Harness h = new Harness(meta, rowMeta("payload"), new Object[] {"y"})) { h.run(); } assertEquals(payload("x") + payload("y"), read(base + "/out.json")); } @Test void ndjsonAppendEscapesEmbeddedNewlines() throws Exception { JsonEOutputMeta meta = meta(JsonEOutputMeta.OperationType.WRITE_TO_FILE); meta.setNewlineDelimited(true); meta.getFileSettings().setFileAppended(true); try (Harness h = new Harness(meta, rowMeta("payload"), new Object[] {"a\nb"})) { h.run(); } try (Harness h = new Harness(meta, rowMeta("payload"), new Object[] {"c\rd"})) { h.run(); } String content = read(base + "/out.json"); assertEquals(payload("a\nb") + "\n" + payload("c\rd") + "\n", content); List<String> lines = content.lines().toList(); assertEquals(2, lines.size()); assertEquals("a\nb", parse(lines.get(0)).get("payload").asText()); assertEquals("c\rd", parse(lines.get(1)).get("payload").asText()); } @Test void ndjsonSplitsRecordsWithoutOuterArrays() throws Exception { JsonEOutputMeta meta = meta(JsonEOutputMeta.OperationType.WRITE_TO_FILE); meta.setNewlineDelimited(true); meta.setUseArrayWithSingleInstance(true); meta.getFileSettings().setSplitOutputAfter(2); try (Harness h = new Harness(meta, rowMeta("payload"), new Object[] {"x"}, new Object[] {"y"}, new Object[] {"z"})) { h.run(); assertEquals(3, h.written.size()); } assertEquals(payload("x") + "\n" + payload("y") + "\n", read(base + "/out_0.json")); assertEquals(payload("z") + "\n", read(base + "/out_1.json")); try (FileObject extra = HopVfs.getFileObject(base + "/out_2.json")) { assertFalse(extra.exists()); } } @Test void ndjsonWrapsEachCompleteGroup() throws Exception { JsonEOutputMeta meta = meta(JsonEOutputMeta.OperationType.BOTH); meta.setNewlineDelimited(true); meta.setJsonBloc("data"); JsonEOutputKeyField key = new JsonEOutputKeyField("grp"); key.setElementName("group"); meta.getKeyFields().add(key); try (Harness h = new Harness(meta, rowMeta("grp", "payload"), new Object[] {"a", "x"}, new Object[] {"a", "y"}, new Object[] {"b", "z"})) { h.run(); assertEquals(2, h.written.size()); } List<String> lines = read(base + "/out.json").lines().toList(); assertEquals(2, lines.size()); JsonNode first = parse(lines.get(0)).get("data"); JsonNode last = parse(lines.get(1)).get("data"); assertEquals("a", first.get("group").asText()); assertEquals(2, first.get("rows").size()); assertEquals("b", last.get("group").asText()); assertEquals("z", last.get("rows").get("payload").asText()); } @Test void ndjsonRejectsPrettyPrintingBeforeOpeningFile() throws Exception { JsonEOutputMeta meta = meta(JsonEOutputMeta.OperationType.WRITE_TO_FILE); meta.setNewlineDelimited(true); meta.setJsonPrettified(true); meta.getFileSettings().setDoNotOpenNewFileInit(false); try (Harness h = new Harness(meta, rowMeta("payload"), new Object[] {"x"})) { assertFalse(h.transform.init()); } try (FileObject file = HopVfs.getFileObject(base + "/out.json")) { assertFalse(file.exists()); } } @Test void ndjsonRejectsAppendWithoutFinalLf() throws Exception { String existing = payload("old"); try (var out = HopVfs.getOutputStream(base + "/out.json", false)) { out.write(existing.getBytes(StandardCharsets.UTF_8)); } JsonEOutputMeta meta = meta(JsonEOutputMeta.OperationType.WRITE_TO_FILE); meta.setNewlineDelimited(true); meta.getFileSettings().setFileAppended(true); try (Harness h = new Harness(meta, rowMeta("payload"), new Object[] {"new"})) { assertTrue(h.transform.init()); assertThrows(HopException.class, h.transform::processRow); } assertEquals(existing, read(base + "/out.json")); } } ``` Add subsequent vertical tests for UTF-16 rejection, deferred/eager zero-row file behavior, forced single grouped-item field arrays, JSON-size populated/unpopulated row shape, grouped split boundaries, and generator error/cancellation cleanup. The class above is the minimal executable behavior suite, not a claim that every final acceptance case has already been supplied or run. For metadata compatibility, extend the existing `JsonEOutputMetaTest.testLoadSave()` pattern: deserialize legacy XML with no `newline_delimited`, run the resulting meta through the legacy-append behavior, then serialize an enabled meta and deserialize/run it through the NDJSON behavior. This tests persisted configuration **and its effect**, rather than only its accessor. ## Appendix B — exact enhanced-output resources and manual text Add to `plugins/transforms/json/src/main/resources/org/apache/hop/pipeline/transforms/jsonoutputenhanced/messages/messages_en_US.properties`: ```properties JsonEOutputDialog.GetKeys.Tooltip=Add input fields not already listed in Fields or Group Key. Existing keys and element names are retained. JsonEOutputDialog.FileFormat.TabTitle=File format JsonEOutputDialog.NdJson.Label=Newline-delimited JSON (NDJSON) JsonEOutputDialog.NdJson.Tooltip=Write one compact JSON item per line. Requires UTF-8 and Pretty Print JSON disabled. Applies only to file output. JsonEOutput.Injection.NEWLINE_DELIMITED=Write newline-delimited JSON to the output file? (Y/N) JsonEOutput.Error.NdJsonSettings=NDJSON file output requires UTF-8 encoding and Pretty Print JSON disabled. JsonEOutput.Error.NdJsonAppendBoundary=Cannot append NDJSON to [{0}]: the nonempty existing file must end with LF. ``` Replace the incorrect legacy Italian tooltip in the same package's `messages_it_IT.properties`, and supply the current active key: ```properties JsonOutputDialog.CompatibilityMode.Tooltip=Genera array JSON anche quando un elenco contiene una sola voce. JsonEOutputDialog.UseArrayWihSingleInstanceMode.Tooltip=Genera array JSON anche quando un elenco contiene una sola voce. ``` In `docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/enhancedjsonoutput.adoc`, replace the Group Key introductory text/Element name description with: ```adoc This tab defines the input fields used to group consecutive rows into JSON fragments. The input stream must be sorted by all group key fields. The key fields are forwarded under their original field names when generated JSON is output to the next transform. In Write to file mode, the original input rows are forwarded unchanged. The Get Fields button adds available input fields that are not already listed in Fields or Group Key. Existing key rows and their element names are retained. Fields can still be added manually to both tabs when a group key must also appear in the generated fragment. ``` ```adoc |Element name|The property name for this group key in each grouped file-output item. It does not rename the downstream Hop field and does not insert the key into the generated output-value fragment. To include a key in that fragment, also select it on the Fields tab. ``` Add the NDJSON option and explicitly retain the legacy-append warning: ```adoc |Newline-delimited JSON (NDJSON)|Disabled by default for compatibility. When enabled for file output, each input-row item or completed group is written as one compact JSON value followed by LF. A JSON block name wraps each line separately. There is no outer document array; Force arrays applies to the value inside a grouped item. UTF-8 is required and Pretty Print JSON must be disabled. Multiline NDJSON records are not supported; newline characters inside string values are escaped. Appending requires an empty file or an existing valid UTF-8 NDJSON file ending in LF. NOTE: Without NDJSON, Append concatenates complete JSON documents without adding separators. It does not merge arrays or produce one valid JSON document. ``` Do not claim #2569's exact attachment is fixed until it has been inspected and executed against the chosen PR baseline. ## Appendix C — GUI acceptance and generator cleanup (required, not optional follow-ups) ### C.1 Complete Group Key UI regression (two 3–5 minute slices) Create `plugins/transforms/json/src/test/java/org/apache/hop/pipeline/transforms/jsonoutputenhanced/JsonEOutputDialogTest.java`, mandatory header plus the class below. This depends on #8588's **verified PR-introduced** `ui/src/test/java/org/apache/hop/ui/testing/UpstreamFixture.java` and improved harness, not an assumed baseline API. RED before adding the annotated Get Fields button is a missing button; GREEN requires real live-table contents, idempotence, preserved aliases, and non-overlapping control geometry. ```java package org.apache.hop.pipeline.transforms.jsonoutputenhanced; import static org.junit.jupiter.api.Assertions.assertEquals; import static org.junit.jupiter.api.Assertions.assertFalse; import static org.junit.jupiter.api.Assertions.assertTrue; import java.lang.reflect.Field; import java.lang.reflect.Method; import java.util.Arrays; import java.util.List; import java.util.concurrent.atomic.AtomicReference; import org.apache.hop.core.gui.plugin.GuiRegistry; import org.apache.hop.core.gui.plugin.GuiWidgetElement; import org.apache.hop.core.variables.Variables; import org.apache.hop.i18n.BaseMessages; import org.apache.hop.ui.core.widget.TableView; import org.apache.hop.ui.testing.SwtBotTestBase; import org.apache.hop.ui.testing.UpstreamFixture; import org.eclipse.swt.SWT; import org.eclipse.swt.graphics.Point; import org.eclipse.swt.graphics.Rectangle; import org.eclipse.swt.widgets.Control; import org.eclipse.swt.widgets.TableItem; import org.junit.jupiter.api.BeforeAll; import org.junit.jupiter.api.Tag; import org.junit.jupiter.api.Test; @Tag("uitest") class JsonEOutputDialogTest extends SwtBotTestBase { @BeforeAll static void registerWidgets() { GuiRegistry registry = GuiRegistry.getInstance(); Class<JsonEOutputMeta> type = JsonEOutputMeta.class; for (Field field : type.getDeclaredFields()) { GuiWidgetElement element = field.getAnnotation(GuiWidgetElement.class); if (element != null) { registry.addGuiWidgetElement(type.getName(), element, field); } } for (Method method : type.getDeclaredMethods()) { GuiWidgetElement element = method.getAnnotation(GuiWidgetElement.class); if (element != null) { registry.addGuiWidgetElement(element, method, type.getName(), type.getClassLoader()); } } } private static TableView table(JsonEOutputDialog dialog, String name) { try { Field field = JsonEOutputDialog.class.getDeclaredField(name); field.setAccessible(true); return (TableView) field.get(dialog); } catch (ReflectiveOperationException e) { throw new AssertionError(e); } } private static Rectangle onDisplay(Control control) { Point point = control.toDisplay(0, 0); Rectangle bounds = control.getBounds(); return new Rectangle(point.x, point.y, bounds.width, bounds.height); } @Test void getKeysUsesLiveSelectionsAndCancelKeepsMetadata() { JsonEOutputMeta meta = new JsonEOutputMeta(); meta.setOperationType(JsonEOutputMeta.OperationType.OUTPUT_VALUE); JsonEOutputField payload = new JsonEOutputField(); payload.setFieldName("payload"); payload.setElementName("payload"); meta.getOutputFields().add(payload); JsonEOutputKeyField existing = new JsonEOutputKeyField("already"); existing.setElementName("custom_alias"); meta.getKeyFields().add(existing); var pipeline = UpstreamFixture.upstreamWithFields( "json", meta, "payload", "grp", "already", "unsaved"); AtomicReference<JsonEOutputDialog> opened = new AtomicReference<>(); AtomicReference<List<String>> names = new AtomicReference<>(); AtomicReference<List<String>> aliases = new AtomicReference<>(); AtomicReference<Rectangle> tableBounds = new AtomicReference<>(); AtomicReference<Rectangle> buttonBounds = new AtomicReference<>(); String title = BaseMessages.getString(JsonEOutputMeta.class, "JsonEOutputDialog.DialogTitle"); String keyTab = BaseMessages.getString(JsonEOutputMeta.class, "JsonEOutputDialog.KeyConfigTab.TabTitle"); String getButton = BaseMessages.getString(JsonEOutputMeta.class, "JsonEOutputDialog.Get.Button"); withDialog( parent -> { opened.set(new JsonEOutputDialog(parent, new Variables(), meta, pipeline)); opened.get().open(); }, bot -> { var shell = bot.shell(title).activate(); var dialog = shell.bot(); display.syncExec(() -> { TableItem item = new TableItem(table(opened.get(), "wFields").table, SWT.NONE); item.setText(1, "unsaved"); item.setText(2, "unsaved_alias"); }); dialog.cTabItem(keyTab).activate(); dialog.button(getButton).click(); dialog.button(getButton).click(); var getControl = dialog.button(getButton).widget; display.syncExec(() -> { TableView keys = table(opened.get(), "wKeyFields"); names.set(keys.getNonEmptyItems().stream().map(item -> item.getText(1)).toList()); aliases.set(keys.getNonEmptyItems().stream().map(item -> item.getText(2)).toList()); shell.widget.setSize(600, 400); shell.widget.layout(true, true); tableBounds.set(onDisplay(keys)); buttonBounds.set(onDisplay(getControl)); }); dialog.button(buttonLabel("System.Button.Cancel")).click(); }); assertEquals(List.of("already", "grp"), names.get()); assertEquals(List.of("custom_alias", "grp"), aliases.get()); assertEquals(1, meta.getKeyFields().size(), "Cancel does not persist table additions"); assertTrue(tableBounds.get().width > 0 && tableBounds.get().height > 0); assertFalse(tableBounds.get().intersects(buttonBounds.get()), tableBounds.get() + " overlaps " + buttonBounds.get()); } } ``` `TableView.getNonEmptyItems()` was verified at `ui/src/main/java/org/apache/hop/ui/core/widget/TableView.java:4097` to return `List<TableItem>`. All widget reads/resizes happen on the display thread; the SWTBot button widget is looked up on the worker before `display.syncExec` to avoid a cross-thread lookup deadlock. Commands, both before and after wiring: ```bash json_ui_test 'JsonEOutputDialogTest#getKeysUsesLiveSelectionsAndCancelKeepsMetadata' json_test 'JsonEOutputBehaviorTest#keySuggestionsExcludeLiveOutputAndExistingKeys' ``` Commit the GUI-backed feature only once both are green, using the exact paths in 4.2 and the two test files. ### C.2 NDJSON GUI persistence test (3 minutes) Add to the same `plugins/transforms/json/src/test/java/org/apache/hop/pipeline/transforms/jsonoutputenhanced/JsonEOutputDialogTest.java`: ```java @Test void ndjsonOptionIsSavedFromTheGroupedTab() { JsonEOutputMeta meta = new JsonEOutputMeta(); meta.setOperationType(JsonEOutputMeta.OperationType.WRITE_TO_FILE); var pipeline = UpstreamFixture.upstreamWithFields("json", meta, "payload"); String title = BaseMessages.getString(JsonEOutputMeta.class, "JsonEOutputDialog.DialogTitle"); withDialog( parent -> new JsonEOutputDialog(parent, new Variables(), meta, pipeline).open(), bot -> { var dialog = bot.shell(title).activate().bot(); dialog.cTabItem("File format").activate(); dialog.checkBox("Newline-delimited JSON (NDJSON)").select(); dialog.button(buttonLabel("System.Button.OK")).click(); }); assertTrue(meta.isNewlineDelimited()); } ``` Run `json_ui_test 'JsonEOutputDialogTest#ndjsonOptionIsSavedFromTheGroupedTab'`. RED before format-tab/persistence wiring; GREEN after. Run it with the runtime NDJSON test so the checkbox cannot be an accessor-only feature. Also verify Cancel on this tab using C.1's pattern and output-only disabling via the checkbox's enabled state; use the enabled/read-back API from the installed SWTBot jar, not an invented lookup. ### C.3 Close the real generator before the writer on dispose (3-minute test + 2-minute fix) In Appendix A's harness add `final JsonEOutputData data = new JsonEOutputData();` and pass `data`, rather than `new JsonEOutputData()`, into the transform constructor. This exposes the real writer state for behavioral cleanup assertions, not a mocked serialization result. Add this complete test method to `plugins/transforms/json/src/test/java/org/apache/hop/pipeline/transforms/jsonoutputenhanced/JsonEOutputBehaviorTest.java`: ```java @Test void disposalClosesAnActiveFileGenerator() throws Exception { JsonEOutputMeta meta = meta(JsonEOutputMeta.OperationType.WRITE_TO_FILE); try (Harness h = new Harness(meta, rowMeta("payload"), new Object[] {"a"}, new Object[] {"b"}, new Object[] {"c"})) { assertTrue(h.transform.init()); assertTrue(h.transform.processRow()); assertTrue(h.transform.processRow()); var generator = h.data.fileGenerator; org.junit.jupiter.api.Assertions.assertNotNull(generator); h.transform.dispose(); assertTrue(generator.isClosed()); org.junit.jupiter.api.Assertions.assertNull(h.data.fileGenerator); org.junit.jupiter.api.Assertions.assertNull(h.data.writer); } } ``` RED on the reviewed PR: the writer is closed but the generator remains open/referenced. In `plugins/transforms/json/src/main/java/org/apache/hop/pipeline/transforms/jsonoutputenhanced/JsonEOutput.java:dispose()`, **before** `closeFile()`, add: ```java if (data.fileGenerator != null) { try { data.fileGenerator.close(); } catch (IOException e) { logError(BaseMessages.getString(PKG, "JsonEOutput.Error.ClosingFile", e.toString())); setErrors(1); } finally { data.fileGenerator = null; data.pendingFileItem = null; } } ``` Do not call `finishGroup()` on cancellation or claim all rows were written; a partially written file is not an atomic output transaction. `AUTO_CLOSE_TARGET` remains disabled so the VFS writer/lineage path closes exactly once. Run `json_test 'JsonEOutputBehaviorTest#disposalClosesAnActiveFileGenerator'` and the complete enhanced suite. Commit runtime/test paths, message `Close enhanced JSON generators when disposal interrupts output`. ## Phase 6 — remove json-simple deliberately, across the repository (#7995) ### 6.1 Public-API gate (3 minutes; blocking decision) Read `core/src/main/java/org/apache/hop/core/row/IValueMeta.java:1409`, `core/src/main/java/org/apache/hop/core/row/value/ValueMetaFactory.java:212`, `core/src/main/java/org/apache/hop/metadata/serializer/json/JsonMetadataParser.java:258`, and the graph/Databricks public methods in the inventory below. They expose `JSONObject` in signatures. Replacing it with a JDK interface is source-compatible with many old callers but **not binary-compatible** with already compiled external plugins. **Proposed policy:** JSON object boundaries use `Map<String,Object>`; arrays use `List<Object>`; parsing/serialization uses the pinned Jackson version through small additions to existing `HopJson`. Existing native `TYPE_JSON` stays Jackson `JsonNode`. Do not implement a new JSON library, relocate json-simple classes, fake `org.json.simple` on the classpath, or add a generic Jackson POJO mapper to the annotated metadata serializer. Obtain a maintainer decision that the public signature change is allowed in the targeted release. If not, stop this phase: keep #7995 open or deliberately decline/defer it with approval, and do not remove its jar. A staged deprecation window or compatibility plugin is a separate release/API design; this plan does not silently choose one. All other phases are independently shippable. The umbrella cannot be declared complete merely because this phase is expensive. ### 6.2 Add focused JSON-value helpers to the existing core utility (two 3-minute vertical slices) Exact paths: `core/src/main/java/org/apache/hop/core/json/HopJson.java` and `core/src/test/java/org/apache/hop/core/json/HopJsonValueTest.java` (new). First create the new test file, Apache header plus this complete class. Add one test at a time as the helper becomes available. ```java package org.apache.hop.core.json; import static org.junit.jupiter.api.Assertions.assertEquals; import static org.junit.jupiter.api.Assertions.assertNull; import static org.junit.jupiter.api.Assertions.assertThrows; import static org.junit.jupiter.api.Assertions.assertTrue; import java.math.BigDecimal; import java.util.ArrayList; import java.util.LinkedHashMap; import java.util.List; import java.util.Map; import org.apache.hop.core.exception.HopException; import org.junit.jupiter.api.Test; class HopJsonValueTest { @Test void parsedObjectsAndArraysUseStandardJavaContainers() throws Exception { Object value = HopJson.parseValue("{\"id\":-2,\"values\":[true,null,\"\"]}"); assertTrue(value instanceof Map<?, ?>); Map<?, ?> object = (Map<?, ?>) value; assertEquals(-2L, object.get("id")); assertTrue(object.get("values") instanceof List<?>); List<?> values = (List<?>) object.get("values"); assertEquals(Boolean.TRUE, values.get(0)); assertNull(values.get(1)); assertEquals("", values.get(2)); } @Test void valuesRoundTripWithoutLosingDecimalDigitsOrEscapes() throws Exception { Map<String, Object> value = new LinkedHashMap<>(); value.put("decimal", new BigDecimal("12345678901234567890.123456789")); value.put("text", "a\nb\"c\\d"); List<Object> values = new ArrayList<>(); values.add(null); values.add(Boolean.FALSE); value.put("values", values); var tree = HopJson.newMapper().readTree(HopJson.toJson(value)); assertEquals(value.get("text"), tree.get("text").asText()); assertTrue(tree.get("values").get(0).isNull()); assertEquals(false, tree.get("values").get(1).asBoolean()); assertEquals("12345678901234567890.123456789", HopJson.toJson(value).split("[,:}]")[1]); } @Test void invalidJsonFailsRatherThanReturningAnUnexplainedNull() { assertThrows(HopException.class, () -> HopJson.parseValue("{")); assertThrows(HopException.class, () -> HopJson.parseValue("{} {}")); } } ``` Commands: `core_test 'HopJsonValueTest#parsedObjectsAndArraysUseStandardJavaContainers'` then `core_test HopJsonValueTest`. Initial RED: the intentionally new `parseValue` / `toJson` methods are absent. This is planned API RED, not a misspelled symbol. Do not weaken the decimal test to double precision. Add imports `java.io.IOException`, `org.apache.hop.core.exception.HopException`, and `org.apache.hop.core.exception.HopRuntimeException` to `HopJson.java`; its existing Jackson imports cover the feature enums. Add these complete members without changing `newMapper()`: ```java private static final ObjectMapper VALUE_MAPPER = newMapper() .enable(DeserializationFeature.USE_LONG_FOR_INTS) .enable(DeserializationFeature.FAIL_ON_TRAILING_TOKENS); public static Object parseValue(String json) throws HopException { try { return VALUE_MAPPER.readValue(json, Object.class); } catch (IOException e) { throw new HopException("Unable to parse JSON value", e); } } public static String toJson(Object value) { try { return VALUE_MAPPER.writeValueAsString(value); } catch (IOException e) { throw new HopRuntimeException("Unable to serialize JSON value", e); } } ``` `USE_LONG_FOR_INTS` preserves the previous untyped integer convention; otherwise Jackson changes small integers to Integer and legacy `(long)` casts fail. Decimal serialization uses BigDecimal's digits; untyped floating parsing remains the legacy Double convention. The mapper is private, configured once, and not reconfigured by callers. Errors retain causes without logging payloads/secrets. Require `core_test 'HopJsonValueTest,HopJsonTest'` GREEN; commit only these two core paths, message `Provide Jackson JSON-value parsing and serialization for migration`. **Explicit serializer edge-case gate:** json-simple writes non-finite Double/Float as JSON null, while default Jackson quotes them. Before migrating a call path that can receive NaN/infinity, add a failing behavior assertion for that path and configure a dedicated Jackson serializer to preserve null, or reject such values with an agreed error. Do not silently turn invalid numbers into strings. This is a release-contract decision, not a reason to alter every mapper globally. ### 6.3 Convert the core metadata API as one small coherent dependency cluster Use separate 2–5 minute edit slices, but commit only after the full cluster compiles and its behavior tests pass. The cluster cannot be halfway installed because `ValueMetaAvroRecord` overrides the changed interface. Do not expose a non-green intermediate install to plugin tests. 1. **RED:** create `core/src/test/java/org/apache/hop/core/row/JsonRowMetaMapTest.java` with the mandatory header and complete test below. It deliberately calls `storeMetaInJson(Map)` so old JSONObject-only signatures fail compilation until the boundary has been migrated. 2. Edit `core/src/main/java/org/apache/hop/core/row/IValueMeta.java`: remove the org.json.simple import; ensure `java.util.Map` is imported; replace only these signatures: ```java void storeMetaInJson(Map<String, Object> jValue) throws HopException; void loadMetaFromJson(Map<String, Object> jValue); ``` 3. In `core/src/main/java/org/apache/hop/core/row/value/ValueMetaBase.java`, change the two override parameter types to `Map<String,Object>`. The complete loading body is: ```java @Override public void loadMetaFromJson(Map<String, Object> jValue) { setLength(((Number) jValue.get(CONST_LENGTH)).intValue()); setPrecision(((Number) jValue.get(CONST_PRECISION)).intValue()); setConversionMask((String) jValue.get("conversionMask")); } ``` The existing store body is unchanged except its parameter type. Remove only the json-simple import, **not** unrelated `java.text.ParseException` numeric conversion code. 4. In `core/src/main/java/org/apache/hop/core/row/value/ValueMetaFactory.java` replace the complete method with: ```java public static IValueMeta loadValueMetaFromJson(Map<String, Object> jValue) throws HopPluginException { String name = (String) jValue.get("name"); int type = ((Number) jValue.get("type")).intValue(); IValueMeta valueMeta = ValueMetaFactory.createValueMeta(name, type); valueMeta.loadMetaFromJson(jValue); return valueMeta; } ``` 5. In `core/src/main/java/org/apache/hop/core/row/value/ValueMetaAvroRecord.java`, change both overridden metadata parameter types to `Map<String,Object>`, add its `java.util.Map` / `HopJson` imports, and replace `((JSONObject) jSchema).toJSONString()` with `HopJson.toJson(jSchema)`. Leave its datum/schema behavior intact; its existing source-guard cleanup and full parser migration are a subsequent slice. The existing JSONParser-produced object is also a Map during this short transition. 6. Replace the entire body (preserve package/header) of `core/src/main/java/org/apache/hop/core/row/JsonRowMeta.java` with the following complete class: ```java package org.apache.hop.core.row; import java.util.ArrayList; import java.util.LinkedHashMap; import java.util.List; import java.util.Map; import org.apache.hop.core.exception.HopRuntimeException; import org.apache.hop.core.json.HopJson; import org.apache.hop.core.row.value.ValueMetaFactory; public class JsonRowMeta { public static String toJson(IRowMeta rowMeta) { try { Map<String, Object> json = new LinkedHashMap<>(); List<Object> values = new ArrayList<>(); json.put("values", values); for (int i = 0; i < rowMeta.size(); i++) { Map<String, Object> value = new LinkedHashMap<>(); rowMeta.getValueMeta(i).storeMetaInJson(value); values.add(value); } return HopJson.toJson(json); } catch (Exception e) { throw new HopRuntimeException("Error converting row metadata to JSON", e); } } @SuppressWarnings("unchecked") public static IRowMeta fromJson(String rowMetaJson) { try { Map<String, Object> json = (Map<String, Object>) HopJson.parseValue(rowMetaJson); List<Object> values = (List<Object>) json.get("values"); IRowMeta result = new RowMeta(); for (Object value : values) { result.addValueMeta(ValueMetaFactory.loadValueMetaFromJson((Map<String, Object>) value)); } return result; } catch (Exception e) { throw new HopRuntimeException("Error converting row metadata JSON to row metadata", e); } } } ``` Test file body: ```java package org.apache.hop.core.row; import static org.junit.jupiter.api.Assertions.assertEquals; import java.util.LinkedHashMap; import java.util.Map; import org.apache.hop.core.HopClientEnvironment; import org.apache.hop.core.row.value.ValueMetaString; import org.junit.jupiter.api.BeforeAll; import org.junit.jupiter.api.Test; class JsonRowMetaMapTest { @BeforeAll static void initialize() throws Exception { HopClientEnvironment.init(); } @Test void standardMapMetadataRoundTripsItsConversionProperties() throws Exception { ValueMetaString value = new ValueMetaString("name"); value.setLength(17); value.setPrecision(-1); value.setConversionMask("mask"); Map<String, Object> jsonValue = new LinkedHashMap<>(); value.storeMetaInJson(jsonValue); assertEquals("name", jsonValue.get("name")); RowMeta row = new RowMeta(); row.addValueMeta(value); IRowMeta loaded = JsonRowMeta.fromJson(JsonRowMeta.toJson(row)); assertEquals(1, loaded.size()); assertEquals("name", loaded.getValueMeta(0).getName()); assertEquals(17, loaded.getValueMeta(0).getLength()); assertEquals(-1, loaded.getValueMeta(0).getPrecision()); assertEquals("mask", loaded.getValueMeta(0).getConversionMask()); } } ``` Run `core_test 'JsonRowMetaMapTest,ValueMetaAvroRecordTest'` for RED/GREEN; then install the core/engine dependency cluster with the inventory's module command. Commit the six production paths plus the new test, message `Use standard JSON containers at the value-metadata boundary`. Do not remove the dependency yet: the remaining core consumers and plugins still use it. ### 6.4 Per-method migration recipe (repeat one 2–5 minute slice, then one module gate) The inventory below lists **all 68 observed Java files**, grouped and sorted programmatically by module, including unused exception imports and tests. For each production file, read the whole method containing each legacy reference; add or retain its behavior fixture, then perform the following exact conversion rules. A complete class rewrite is not required for a constructor/import change, but do not leave `add conversion here` placeholders. | Legacy operation | Replacement code / semantics | |---|---| | `JSONObject` variable/return/parameter | `Map<String,Object>`; preserve method name and existing error wrapper | | `new JSONObject()` | `new LinkedHashMap<>()` | | `new JSONObject(existingMap)` | `new LinkedHashMap<>(existingMap)` | | `JSONArray` variable/return/parameter | `List<Object>` | | `new JSONArray()` | `new ArrayList<>()` | | `new JSONParser().parse(text)` / reused parser's `parse(text)` | `HopJson.parseValue(text)`; remove now-unused parser field/local | | `(JSONObject) parsed` | `(Map<String,Object>) parsed` (one localized unchecked cast) | | `(JSONArray) parsed` | `(List<Object>) parsed` | | `obj.toJSONString()` | `HopJson.toJson(obj)` | | `JSONValue.toJSONString(value)` | `HopJson.toJson(value)` | | `list.writeJSONString(writer)` | `writer.write(HopJson.toJson(list))` | | `(long) object.get(key)` | `((Number) object.get(key)).longValue()`; preserve required/missing checks | | `instanceof JSONObject object` | `instanceof Map<?,?> object`; pass through a localized cast only if the callee needs string keys | | `instanceof JSONArray array` | `instanceof List<?> array` | | `case JSONObject object ->` | `case Map<?,?> object ->`; preserve the switch's other branches | | `catch (org.json.simple.parser.ParseException e)` | `catch (HopException e)` around `HopJson.parseValue`; retain the original provider/context error message | | Unused legacy `throws ParseException` | Remove the **json-simple** exception declaration/import; do not replace `java.text.ParseException` or `DateTimeParseException` | Required imports for ordinary converted files are `java.util.Map`, `java.util.List`, `java.util.LinkedHashMap`, `java.util.ArrayList`, `org.apache.hop.core.json.HopJson`; add only the ones the file actually uses. Read callers whenever a signature changes; change that tightly coupled caller/test cluster together, then compile/test before moving on. Do not blanket-replace generic tokens in comments, schemas, sample JSON, or `net.minidev` classes. **Important exceptions:** - `core/src/main/java/org/apache/hop/metadata/serializer/json/JsonMetadataParser.java`: replace its *save-side containers* using these rules; keep its streaming Jackson *load-side* and `@HopMetadataProperty` traversal, passwords, enum codes, store-with-name references and object factories. Do not substitute `ObjectMapper.convertValue()` for Hop metadata serialization. `getJsonObject` changes to `Map<String,Object>`; coordinate its known callers in core/engine/metadata/AI as shown below. - `plugins/tech/salesforce/src/main/java/org/apache/hop/pipeline/transforms/salesforce/SalesforceConnection.java`: raw objects use QName keys before serialization. Use `Map<javax.xml.namespace.QName,Object>` for that map, not a cast to String keys; keep the QName/local-part behavior pinned by `SalesforceConnectionRecordExtractionTest`. Jackson's key serialization must be compared to the existing golden text; a namespace change is not an acceptable migration side effect. - `plugins/tech/avro/src/main/java/org/apache/hop/avro/transforms/avrodecode/AvroDecodeFieldFinder.java` and `core/src/main/java/org/apache/hop/core/row/value/ValueMetaAvroRecord.java`: retain schema wrapper structure, complex unions, large values and binary/datum behavior; assert decoded datum, not merely parse success. - `plugins/tech/neo4j/src/main/java/org/apache/hop/neo4j/core/data/GraphPropertyData.java`: preserve native value types and the graph schema; run graph round-trip/importer tests. Do not Jackson-map Neo4j driver objects directly. - `plugins/misc/git/src/main/java/org/apache/hop/git/provider/GitApiHttp.java`: convert shared parse/get helpers first, then the provider caller cluster; preserve pagination, null-safe missing nested objects and all stored raw JSON. Never test against a real repository write. - JSON Output and HTTP/REST header producers: preserve nested arrays, booleans, integers, nulls and duplicate-header lists. Compare parsed JSON for semantic invariance; object-member order and slash escaping are not portable guarantees, but byte consumers require explicit compatibility tests. **TDD for a behavior-preserving migration:** characterization tests may already be GREEN; do not fake a failure by breaking code. Add the module-local failing source-usage guard below to prove the remaining legacy dependency is present, migrate one method at a time with existing behavior tests kept green, and finish that module with the guard green. Every behavior change (invalid input, precision, null handling) gets its own real failing test before code. Do not migrate all 68 files in one blind search-and-replace commit. ### 6.5 A complete, module-local dependency-usage RED/GREEN guard (3 minutes per module) For each module named in the inventory, create `<module>/src/test/java/org/apache/hop/jsonmigration/JsonSimpleUsageTest.java` with the mandatory header and this complete body. The inventory's literal module prefix supplies the exact filename. Maven's default test working directory is that module's basedir; assert that precondition so a run cannot accidentally scan the wrong tree. ```java package org.apache.hop.jsonmigration; import static org.junit.jupiter.api.Assertions.assertFalse; import static org.junit.jupiter.api.Assertions.assertTrue; import java.nio.charset.StandardCharsets; import org.apache.commons.vfs2.FileObject; import org.apache.commons.vfs2.FileType; import org.apache.hop.core.vfs.HopVfs; import org.junit.jupiter.api.Test; class JsonSimpleUsageTest { @Test void productionSourcesDoNotUseJsonSimple() throws Exception { try (FileObject root = HopVfs.getFileObject("src/main/java")) { assertTrue(root.exists(), "Run this test with Maven from its owning module"); inspect(root); } } private static void inspect(FileObject file) throws Exception { if (file.getType() == FileType.FOLDER) { for (FileObject child : file.getChildren()) { try (child) { inspect(child); } } } else if ("java".equals(file.getName().getExtension())) { try (var stream = HopVfs.getInputStream(file)) { String source = new String(stream.readAllBytes(), StandardCharsets.UTF_8); assertFalse(source.contains("org.json.simple"), file.getName().getURI()); } } } } ``` Run the module's command with `-Dtest=JsonSimpleUsageTest -Dsurefire.failIfNoSpecifiedTests=false` before converting its remaining consumers: expected RED names a real production file. After the batch, remove legacy imports from its test fixtures too and run the **whole module** with no `-Dtest`: expected behavior tests and guard all GREEN. This guard alone does not prove functional compatibility; the module's existing JSON fixtures and new exact precision/null/error assertions are also mandatory. ### 6.6 Complete observed per-module source inventory and commands The paths below were collected from tracked Java sources and verified to total 68 unique entries. Source references should only shrink, not be reintroduced. Add each module's guard path from 6.5 and any changed neighboring tests explicitly when staging. Build/install of a module takes longer than a 2–5-minute active slice; the estimate applies to one method's test/edit/review. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
