hansva opened a new issue, #9332:
URL: https://github.com/apache/hop/issues/9332
### Apache Hop version?
2.20.0
### Java version?
21
### Operating system
Other
### What happened?
The workflow engine clones the previous result before every action
(`Workflow.java:789`). `Result.clone()` clones each result row with
`RowMetaAndData.clone()`, which clones the row metadata for every row:
```java
for (RowMetaAndData row : rows) {
clonedRows.add(row.clone()); // c.rowMeta = rowMeta.clone() for every row
}
```
Cloning row metadata rebuilds every number and date format in it. 1,000,000
result rows passed through 10 actions means 10,000,000 metadata clones,
although all rows written by one Copy rows to result share a single metadata
object. Profiling a Workflow executor whose child workflow carries result rows
shows 94% of the CPU in `Result.clone()`. `ActionResult.clone()` and
`WorkflowExecutionConfiguration` go through the same method.
### Proposed fix
In `Result.clone()`, clone each distinct row metadata once, and give that
clone to every row that shared it:
```java
Map<IRowMeta, IRowMeta> clonedRowMetas = new IdentityHashMap<>();
for (RowMetaAndData row : rows) {
IRowMeta rowMeta = row.getRowMeta();
if (rowMeta == null) {
clonedRows.add(row.clone());
continue;
}
IRowMeta clonedRowMeta = clonedRowMetas.computeIfAbsent(rowMeta,
IRowMeta::clone);
clonedRows.add(new RowMetaAndData(clonedRowMeta,
rowMeta.cloneRow(row.getData())));
}
```
What an action receives does not change:
- The clone is still a complete copy. Every row gets a new value array from
the same `cloneRow()` call, and every metadata object in the clone is new, so
changing the original never affects the clone and the other way round.
- The map compares by identity, not equality. Rows that shared a metadata
object in the original share one clone; rows with their own metadata, such as
mixed layouts, keep their own clone.
- The only difference is inside one clone: rows that shared a metadata
object share its clone instead of each getting a copy. Code would notice only
if it changed one result row's metadata in place. No code in the repository
does that: of the 54 classes that read result rows, only the HTTP action and
Cassandra Exec CQL change metadata, and both build their own. A third-party
plugin that changes a result row's metadata in place would now change it for
every row of that clone.
- Uncloned result rows already share one metadata object, because Copy rows
to result stores the same one for every row. The Pipeline executor, Simple
mapping and hop-run read those rows today.
- A row without metadata, the exception thrown when cloning fails,
`lightClone()`, the XML form and the result files are unchanged.
### Issue Priority
Priority: 2
### Issue Component
Component: Workflows
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
