github-actions[bot] commented on code in PR #68837:
URL: https://github.com/apache/doris/pull/68837#discussion_r4246784933
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/glue/translator/PhysicalPlanTranslator.java:
##########
@@ -3719,6 +3720,24 @@ private void addPlanRoot(PlanFragment fragment, PlanNode
planNode, AbstractPlan
updateLegacyPlanIdToPhysicalPlan(planNode, physicalPlan);
}
+ @VisibleForTesting
+ static DataPartition createHiveTableSinkDataPartition(
+ List<Expr> partitionExprs, int beExecVersion) {
+ if (partitionExprs.isEmpty()) {
+ TPartitionType type = beExecVersion
+ <
Config.ALIGNED_EXTERNAL_TABLE_SINK_THRIFT_MIN_BE_EXEC_VERSION
+ ? TPartitionType.EXTERNAL_TABLE_SINK_UNPARTITIONED
+ : TPartitionType.HIVE_TABLE_SINK_UNPARTITIONED;
+ return new DataPartition(type);
+ }
+ if (beExecVersion <
Config.ALIGNED_EXTERNAL_TABLE_SINK_THRIFT_MIN_BE_EXEC_VERSION) {
+ return new
DataPartition(TPartitionType.EXTERNAL_TABLE_SINK_HASH_PARTITIONED,
Review Comment:
[P2] Keep partitioned Hive inserts usable below execution version 13
For a parallel partitioned Hive INSERT ... SELECT with be_exec_version=12
and at least two destination writer instances, this reduced plan reaches the
new helper:
```text
ConnectorTableSink(partitioned Hive)
Distribute(Hive hash by partition column)
Scan
```
PhysicalPlanTranslator now emits EXTERNAL_TABLE_SINK_HASH_PARTITIONED, but
ExchangeSinkLocalState::init rejects that type below
SUPPORT_EXTERNAL_TABLE_SINK_HASH_VERSION (13). This also fails with every FE
and BE on this head; before the change, the Hive partition type used
ScaleWriterPartitioner without that restriction. Config still supports versions
8-12, while the new tests only check 15/16. Preserve an executable Hive routing
path for those lower versions. This differs from the existing
version-13/older-BE enum thread: the version is fixed and both processes
support the new enum.
##########
fe/fe-core/src/main/java/org/apache/doris/qe/CoordinatorContext.java:
##########
@@ -276,7 +276,8 @@ public QueryProcessor asQueryProcessor() {
public static CoordinatorContext buildForSql(NereidsPlanner planner,
NereidsCoordinator coordinator) {
ConnectContext connectContext =
planner.getCascadesContext().getConnectContext();
- TQueryOptions queryOptions = initQueryOptions(connectContext);
+ TQueryOptions queryOptions = initQueryOptions(
+ connectContext, planner.getQueryOptions().getBeExecVersion());
Review Comment:
[P1] Share the pinned version with scan feature admission
This pins dispatch to the early planning version, but scan feature guards
still read the live Config.be_exec_version. Consider a distributed OpenCSVSerde
scan with a remote gather exchange:
```text
ResultSink
Exchange(GATHER, remote)
FileScan(Hive OpenCSVSerde, released BE)
```
Capture version 13, then raise the setting to 15 before scan finalization:
PluginDrivenScanNode.getFileAttributes admits hive_open_csv=true, but this
coordinator sends 13. A 4.1/4.2 BE ignores the unknown optional flag and uses
its legacy CSV parser. With comma separator, quote disabled (NUL), escape=e and
record eeabc,tail,last, it forwards eabc instead of Hive's abc (the existing
AbsentSemanticFlagRetainsLegacyDecoding test pins this difference). Before this
change the request carried 15 and that BE's Block::serialize rejected it before
remote transfer; the pinned 13 now passes. Make the feature guards consume the
same statement version, or reject a feature/version mismatch before dispatch.
This is a separate scan path from the existing Hive sink/legacy-coordinator
race.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
