hansva opened a new issue, #9448: URL: https://github.com/apache/hop/issues/9448
### What needs to happen? Three problems in the JSON plugin (`plugins/transforms/json`): two are performance problems, one is a bug. 1. **JSON output keeps every row in memory.** Each row becomes a json-simple object in one array, and the array is turned into text when the block or the input ends. With "Nr rows in a bloc" at 0, which is the only way to get one valid JSON document, 2 million rows need about 8 GB, and garbage collection uses more CPU than the transform itself. 2. **JSON input and JSON normalize throw an exception for most numbers.** `RowOutputConverter.getValue` first tries `getNativeDataType`, which fails with a cast error for a JSON whole number in an Integer field and for every Jackson node. The error is wrapped in a `HopValueException`, then dropped, and the value is converted a second time from its text. In a profile this was about 22% of JSON input and 43% of JSON normalize. 3. **JSON normalize reports a false error.** With "Do not fail if no file" off, the transform reads all its rows and still ends with the "no files" error, because the check runs before the file list is loaded. The integration tests switch the option on, which hides it. ### Proposed changes **JSON output: write rows as they come** - Build the text of each row when it arrives. When the transform only writes to a file, write it to the file at once; when it puts the JSON in an output field, collect the text of the block, not the objects. - Keep the text exactly as it is today. json-simple writes the element names in the order of its hash map, keeps the last field when a name is used twice, and escapes `/` as `\/`. Work out the name order once, on the first row, and write the values with `JSONValue.toJSONString` and the names with `JSONValue.escape`. Do not switch to another JSON library here: it would change the order of the names and the escaping. - The blocks, the file per block, the output field on the last row of a block, an empty input (no file content) and the behavior in a Beam bundle stay as they are. **JSON input and JSON normalize: convert numbers directly** Convert a JSON number without the failing cast when the result is certain to be the same as converting its text: | JSON value | Target field | Direct conversion | |---|---|---| | Whole number that fits a long | Integer | yes | | Whole number that fits a long | Number, BigNumber | yes | | Decimal | Number | yes, when the decimal symbol is `.` or not set | | Decimal | Integer, BigNumber | no | | Any number, with a conversion mask that has anything other than `#`, `0`, `,` and `.` | any | no | Everything in a "no" row goes through today's code, unchanged. Decimals that Java writes in scientific notation, such as 1.0E10, also stay on today's path for now; check separately whether that path parses them correctly. **JSON normalize: the false error** Run the "no files" check after the file list is loaded. Add an integration test with "Do not fail if no file" off. ### Measured gain Prototypes of the first two changes against `apache/main`, 3 runs each: | | Before | After | |---|---|---| | JSON output, 2 million rows to one file | 93k-245k rows/s | 301k-313k rows/s | | JSON output, peak memory | 7.4-8.0 GB | 2.3-3.0 GB | | JSON input, 500,000 rows | 67k-69k rows/s | 97k-102k rows/s | | JSON normalize, 500,000 rows | 107k-112k rows/s | 403k-441k rows/s | That is about 1.5x for JSON output with a third of the memory, about 1.45x for JSON input and about 4x for JSON normalize. ### Issue Priority Priority: 2 ### Issue Component Component: Transforms -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
