hansva opened a new issue, #9448:
URL: https://github.com/apache/hop/issues/9448

   ### What needs to happen?
   
   Three problems in the JSON plugin (`plugins/transforms/json`): two are 
performance problems, one is a bug.
   
   1. **JSON output keeps every row in memory.** Each row becomes a json-simple 
object in one array, and the array is turned into text when the block or the 
input ends. With "Nr rows in a bloc" at 0, which is the only way to get one 
valid JSON document, 2 million rows need about 8 GB, and garbage collection 
uses more CPU than the transform itself.
   2. **JSON input and JSON normalize throw an exception for most numbers.** 
`RowOutputConverter.getValue` first tries `getNativeDataType`, which fails with 
a cast error for a JSON whole number in an Integer field and for every Jackson 
node. The error is wrapped in a `HopValueException`, then dropped, and the 
value is converted a second time from its text. In a profile this was about 22% 
of JSON input and 43% of JSON normalize.
   3. **JSON normalize reports a false error.** With "Do not fail if no file" 
off, the transform reads all its rows and still ends with the "no files" error, 
because the check runs before the file list is loaded. The integration tests 
switch the option on, which hides it.
   
   ### Proposed changes
   
   **JSON output: write rows as they come**
   
   - Build the text of each row when it arrives. When the transform only writes 
to a file, write it to the file at once; when it puts the JSON in an output 
field, collect the text of the block, not the objects.
   - Keep the text exactly as it is today. json-simple writes the element names 
in the order of its hash map, keeps the last field when a name is used twice, 
and escapes `/` as `\/`. Work out the name order once, on the first row, and 
write the values with `JSONValue.toJSONString` and the names with 
`JSONValue.escape`. Do not switch to another JSON library here: it would change 
the order of the names and the escaping.
   - The blocks, the file per block, the output field on the last row of a 
block, an empty input (no file content) and the behavior in a Beam bundle stay 
as they are.
   
   **JSON input and JSON normalize: convert numbers directly**
   
   Convert a JSON number without the failing cast when the result is certain to 
be the same as converting its text:
   
   | JSON value | Target field | Direct conversion |
   |---|---|---|
   | Whole number that fits a long | Integer | yes |
   | Whole number that fits a long | Number, BigNumber | yes |
   | Decimal | Number | yes, when the decimal symbol is `.` or not set |
   | Decimal | Integer, BigNumber | no |
   | Any number, with a conversion mask that has anything other than `#`, `0`, 
`,` and `.` | any | no |
   
   Everything in a "no" row goes through today's code, unchanged. Decimals that 
Java writes in scientific notation, such as 1.0E10, also stay on today's path 
for now; check separately whether that path parses them correctly.
   
   **JSON normalize: the false error**
   
   Run the "no files" check after the file list is loaded. Add an integration 
test with "Do not fail if no file" off.
   
   ### Measured gain
   
   Prototypes of the first two changes against `apache/main`, 3 runs each:
   
   | | Before | After |
   |---|---|---|
   | JSON output, 2 million rows to one file | 93k-245k rows/s | 301k-313k 
rows/s |
   | JSON output, peak memory | 7.4-8.0 GB | 2.3-3.0 GB |
   | JSON input, 500,000 rows | 67k-69k rows/s | 97k-102k rows/s |
   | JSON normalize, 500,000 rows | 107k-112k rows/s | 403k-441k rows/s |
   
   That is about 1.5x for JSON output with a third of the memory, about 1.45x 
for JSON input and about 4x for JSON normalize.
   
   ### Issue Priority
   
   Priority: 2
   
   ### Issue Component
   
   Component: Transforms


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to