hansva opened a new issue, #9478:
URL: https://github.com/apache/hop/issues/9478

   ### What needs to happen?
   
   Join rows (`JoinRows`, `plugins/transforms/joinrows`) keeps the rows of 
every stream except the main one in a cache. For each output row it takes a row 
from each cache and clones it:
   
   ```java
   rowData = data.cache[filenr].get(data.position[data.filenr]);
   
   // Don't forget to clone the data to protect it against data alteration 
downstream.
   //
   rowData = data.fileRowMeta[filenr].cloneRow(rowData);
   ```
   
   `outputRow()` then copies the values of all streams into a new output array. 
So the cloned array is thrown away right after it is made. The only part of the 
clone that protects the cache is the deep copy of values that can be changed in 
place, and those are Binary and Serializable values only 
(`IValueMeta.requiresRealClone()`).
   
   In a profile of a cartesian join this clone is about 10% of the CPU time of 
the transform and 39% of its allocations.
   
   ### Proposed change
   
   - Read the cached row without cloning it.
   - While the output row is put together, deep-copy only the values of cached 
streams whose value metadata requires a real clone.
   
   The output is the same: every output row still gets its own array, and the 
same values get a deep copy as before. Rows that are read back from the 
temporary file (when a stream has more rows than the cache size) are new 
objects and were never cloned; that path does not change.
   
   ### Expected gain
   
   Measured with a prototype on 2 million rows joined with 10 cached rows, 20 
million output rows (medians of 5 runs, `apache/hop:Development`, JDK 21):
   
   | Run configuration | Before | After | Gain |
   |---|---|---|---|
   | Local, lock-free row sets | 8.72M rows/s | 13.9M rows/s | 1.60x |
   | Local, default | 3.07M rows/s | 3.79M rows/s | 1.23x |
   
   ### Tests
   
   Add before the change, and check they pass on the current code:
   
   - A cached stream with Binary and Serializable values: change a value 
downstream and check that the next output row still has the original value.
   - The same with the main stream in another position, and with three inputs.
   - A cached stream with null values in the fields that need a real clone.
   
   ### Issue Priority
   
   Priority: 2
   
   ### Issue Component
   
   Component: Transforms


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to