hansva opened a new issue, #9435:
URL: https://github.com/apache/hop/issues/9435

   ### What needs to happen?
   
   The Data set input transform is slow and memory hungry on large data sets, 
for two reasons.
   
   1. `DataSetCsvUtil.getAllRows(IVariables, DataSet)` clones the value 
metadata of every field for every row. A fresh clone has no cached number or 
date format, so every Integer, Number, BigNumber and Date value builds a new 
`DecimalFormat` or `SimpleDateFormat`. In a profile, 77% of the transform's CPU 
was spent building formats.
   2. `DataSetInput.processRow()` calls `getAllRows`, which parses the complete 
file into a list, and only then writes the rows. A data set of 2 million rows 
with 11 fields needs about 2.8 GB before the first row leaves the transform.
   
   ### Proposed changes
   
   1. **Value metadata once per field.** Before the read loop, prepare one 
value meta and one String conversion meta per field, and use them for every 
row. The other `getAllRows` variant, which unit tests use for input and golden 
data sets, already converts without a clone per row, so the result does not 
change.
   2. **Stream the rows.** Add a reader to `DataSetCsvUtil` that hands over one 
row at a time, and let Data set input write each row as it is parsed. The 
transform should stop reading when the pipeline is stopped. `getAllRows` stays 
for the callers that need the full list (the data set editor, the location 
dialog, unit test injection and golden data comparison) and is built on the 
same reader.
   
   Behavior to keep:
   
   - A data set whose file does not exist yet gives no rows and no error in 
Data set input.
   - Field order, types, the storage masks from `setValueFormats`, hex-encoded 
Binary fields and the trim of padded BigNumber values stay as they are.
   - A record with fewer values than the data set has fields leaves the missing 
fields null.
   
   ### Measured gain
   
   Change 1 alone, as a prototype with only `DataSetCsvUtil.class` swapped, on 
a data set of 2 million rows with 11 fields, 3 runs each:
   
   | | Before | After |
   |---|---|---|
   | Rows/s | 91k-114k | 442k-482k |
   | Engine time | 17.6-22.1 s | 4.2-4.6 s |
   | Peak memory | 2.3-3.0 GB | 1.6-2.2 GB |
   
   
   ### Issue Priority
   
   Priority: 2
   
   ### Issue Component
   
   Component: Transforms


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to