hansva opened a new issue, #9557: URL: https://github.com/apache/hop/issues/9557
### Apache Hop version? 2.20.0 ### Java version? 21 ### Operating system Other ### What happened? `IValueMeta.storeMetaInJson` / `loadMetaFromJson` (used by `JsonRowMeta.toJson` / `fromJson`) keep only a field's name, type, length, precision and conversion mask. Everything else a field carries is lost on the way through JSON and comes back as the default: | Lost | Effect after the round trip | |---|---| | origin | The transform a field comes from is empty, e.g. *Metadata structure* shows an empty Origin | | decimal, grouping and currency symbol | Numbers are formatted and parsed with the default symbols instead of the field's, e.g. `1.234,56` becomes `1,234.56` or fails to parse | | rounding type | Rounding falls back to the default | | trim type | Strings are no longer trimmed as configured | | date format locale, time zone and leniency | Dates are formatted and parsed in the JVM's locale and time zone, and strictness changes | | lenient string to number | Number parsing becomes strict or lenient depending on the default | | string encoding | Binary/string conversion uses the default encoding | | case insensitive, sorted descending | Comparisons and sorts ignore the field's case and order settings | | comments, collator settings, output padding | Lost as well | The binary format (`ValueMetaBase.writeMeta` / `readMeta`) keeps all of these, so the same row metadata behaves differently depending on which of the two formats it went through. ### Where this shows `JsonRowMeta` is how row metadata is handed over when it can't stay in memory: - **Beam engine:** every Hop transform that runs inside Beam gets its input and output row metadata as JSON (`TransformFn`, `TransformBatchTransform`, the Beam IO and group-by functions). A transform there formats and parses numbers and dates with the default symbols, locale and time zone instead of the field's, and compares without the field's case and sort settings. The same pipeline gives different results on the local engine and on Beam. - **Neo4j execution information location:** stores the row metadata of execution data samples as JSON. Samples read back in the execution viewer lose the field's formatting settings. - **Execution information transform:** writes row metadata as JSON into its output. ### Expected A field read back from JSON is the same as the field that was written, as with the binary format. ### Proposed fix In `ValueMetaBase.storeMetaInJson` / `loadMetaFromJson`, also write and read origin, comments, the decimal, grouping and currency symbols, rounding type, trim type, date format locale (language tag), time zone (ID) and leniency, lenient string to number, string encoding, case insensitive, the collator settings, sorted descending and output padding. Each key is only applied when present, so JSON written by older versions loads exactly as before, and older readers ignore the new keys. ### Tests `JsonRowMetaTest`: a row metadata with every one of these set to a non-default value survives `toJson` / `fromJson` unchanged. JSON without the new keys still loads, with the defaults. ### Issue Priority Priority: 2 ### Issue Component Component: API -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
