hansva opened a new issue, #9478: URL: https://github.com/apache/hop/issues/9478
### What needs to happen? Join rows (`JoinRows`, `plugins/transforms/joinrows`) keeps the rows of every stream except the main one in a cache. For each output row it takes a row from each cache and clones it: ```java rowData = data.cache[filenr].get(data.position[data.filenr]); // Don't forget to clone the data to protect it against data alteration downstream. // rowData = data.fileRowMeta[filenr].cloneRow(rowData); ``` `outputRow()` then copies the values of all streams into a new output array. So the cloned array is thrown away right after it is made. The only part of the clone that protects the cache is the deep copy of values that can be changed in place, and those are Binary and Serializable values only (`IValueMeta.requiresRealClone()`). In a profile of a cartesian join this clone is about 10% of the CPU time of the transform and 39% of its allocations. ### Proposed change - Read the cached row without cloning it. - While the output row is put together, deep-copy only the values of cached streams whose value metadata requires a real clone. The output is the same: every output row still gets its own array, and the same values get a deep copy as before. Rows that are read back from the temporary file (when a stream has more rows than the cache size) are new objects and were never cloned; that path does not change. ### Expected gain Measured with a prototype on 2 million rows joined with 10 cached rows, 20 million output rows (medians of 5 runs, `apache/hop:Development`, JDK 21): | Run configuration | Before | After | Gain | |---|---|---|---| | Local, lock-free row sets | 8.72M rows/s | 13.9M rows/s | 1.60x | | Local, default | 3.07M rows/s | 3.79M rows/s | 1.23x | ### Tests Add before the change, and check they pass on the current code: - A cached stream with Binary and Serializable values: change a value downstream and check that the next output row still has the original value. - The same with the main stream in another position, and with three inputs. - A cached stream with null values in the fields that need a real clone. ### Issue Priority Priority: 2 ### Issue Component Component: Transforms -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
