xumanbu opened a new issue, #11394:
URL: https://github.com/apache/incubator-gluten/issues/11394
### Backend
VL (Velox)
### Bug description
### Problem
The `UnsupportedOperators.tsv` report shows inflated operator counts that
don't match the actual number of operator instances in the execution plan.
### Example
For this execution plan:
```
Execute InsertIntoHadoopFsRelationCommand (node 0, unsupported #1 - ROOT)
└── Project (node 1)
└── SortMergeJoin(skew=true) (node 2, unsupported #2)
├── Scan table1 (node 3)
└── Scan table2 (node 4)
```
**Expected**: 2 operators (1 InsertIntoHadoopFsRelationCommand + 1
SortMergeJoin)
**Actual**: 4 in the report
In production event logs, the issue is more severe:
- `Execute InsertIntoHadoopFsRelationCommand`: reported 62, should be 4
- `SortMergeJoin(skew=true)`: reported 62, should be 4
### Root Cause
In `ResultVisitor.java`, the `increment()` method is called for every parent
node during post-order traversal:
```java
// Current (wrong) behavior:
currentNodeUnsupportedOperators.forEach(id -> {
currentCost.addCpuDuration(durationMetric.getDuration());
currentCost.increment(); // ❌ Called for each parent node
});
```
When traversing node 1 (Project), it inherits unsupported operator from node
2 (SortMergeJoin) and increments the count again. Same for node 0, causing the
count to be 3 instead of 1 for the SortMergeJoin.
### Solution
Only count when the operator is first encountered:
```java
// Fixed behavior:
if (isOperatorNotSupported(nodeId)) {
currentNodeUnsupportedOperators.add(nodeId);
unsupportedOperatorImpactCostMap.put(nodeId, new UnsupportedImpact());
unsupportedOperatorImpactCostMap.get(nodeId).increment(); // ✅ Count
once
}
// When visiting parent nodes, only accumulate CPU time:
currentNodeUnsupportedOperators.forEach(id -> {
currentCost.addCpuDuration(durationMetric.getDuration());
// No increment() here
});
```
### Impact
- Count values are 3-15x higher than actual operator instances
- Makes the report misleading for users
- CPU time metrics are NOT affected (still correct)
### Gluten version
_No response_
### Spark version
None
### Spark configurations
_No response_
### System information
_No response_
### Relevant logs
```bash
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
