NEUpanning opened a new issue, #9789:
URL: https://github.com/apache/incubator-gluten/issues/9789
### Backend
VL (Velox)
### Bug description
WriteFiles operator requires a specific order that is guaranteed by Sort
operator added by V1Writes rule. If the ordering of child operator like SMJ
satisfies the order, Sort will not be needed. However, Gluten could transform
SMJ to SHJ, which can disrupt the order. And EnsureLocalSortRequirements rule
can't ensure the order because WriteFiles operator lacks the
requiredChildOrdering property. As a result, there is a risk of data loss.
Here is an example :
vanilla spark plan
```
+- == Final Plan ==
Execute InsertIntoHiveTable
+- WriteFiles
+- Project
+- SortMergeJoin [dynamic_partition_column#186]
```
gluten plan
```
+- == Final Plan ==
Execute InsertIntoHiveTable
+- WriteFiles
+- VeloxColumnarToRow
+- ProjectExecTransformer
+- ShuffledHashJoinExecTransformer [dynamic_partition_column#186]
```
Potential solutions:
1. Disable `spark.sql.optimizer.plannedWrite.enabled` to the logic of
ensuring the order requirement after Gluten plan transformation.
2. Perhaps, WriteFiles should have the requiredChildOrdering property.
3. EnsureLocalSortRequirements rule supports ensuring ordering for
WriteFiles.
### Gluten version
_No response_
### Spark version
None
### Spark configurations
_No response_
### System information
_No response_
### Relevant logs
```bash
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
