Yohahaha commented on issue #9311: URL: https://github.com/apache/incubator-gluten/issues/9311#issuecomment-2817570603
commit fdd37fc83
```
25/04/21 11:40:44 WARN [main] PlanChangeLogger:
=== Applying Rule
org.apache.gluten.extension.columnar.transition.RemoveTransitions ===
Execute InsertIntoHadoopFsRelationCommand
hdfs:///user/hive/warehouse/test_hive_bucket, false, [dt#25], 8 buckets, bucket
columns: [bid], Parquet, [serialization.format=1, mergeSchema=false,
__hive_compatible_bucketed_table_insertion__=true,
partitionOverwriteMode=DYNAMIC], Overwrite,
`spark_catalog`.`default`.`test_hive_bucket`,
org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe,
org.apache.spark.sql.execution.datasources.CatalogFileIndex(hdfs:///user/hive/warehouse/test_hive_bucket),
[id, bid, dt] Execute InsertIntoHadoopFsRelationCommand
hdfs:///user/hive/warehouse/test_hive_bucket, false, [dt#25], 8 buckets, bucket
columns: [bid], Parquet, [serialization.format=1, mergeSchema=false,
__hive_compatible_bucketed_table_insertion__=true,
partitionOverwriteMode=DYNAMIC], Overwrite,
`spark_catalog`.`default`.`test_hive_bucket`,
org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe,
org.apache.spark.sql.execution.datasources.CatalogFileIndex(hdfs:///user/hive/warehou
se/test_hive_bucket), [id, bid, dt]
+- WriteFiles
+- WriteFiles
+- Sort [dt#25 ASC NULLS FIRST, pmod((hive-hash(bid#17) & 2147483647),
8) ASC NULLS FIRST], false, 0
+- Sort [dt#25 ASC NULLS FIRST, pmod((hive-hash(bid#17) &
2147483647), 8) ASC NULLS FIRST], false, 0
+- Project [id#15, bid#17, empty2null(dt#16) AS dt#25]
+- Project [id#15, bid#17, empty2null(dt#16) AS dt#25]
! +- ColumnarToRow
+- FileScan parquet
spark_catalog.default.test[id#15,dt#16,bid#17] Batched: true, DataFilters: [],
Format: Parquet, Location: InMemoryFileIndex(1 paths)[hdfs://...,
PartitionFilters: [], PushedFilters: [], ReadSchema:
struct<id:int,dt:string,bid:int>
! +- FileScan parquet
spark_catalog.default.test[id#15,dt#16,bid#17] Batched: true, DataFilters: [],
Format: Parquet, Location: InMemoryFileIndex(1 paths)[hdfs://...,
PartitionFilters: [], PushedFilters: [], ReadSchema:
struct<id:int,dt:string,bid:int>
25/04/21 11:40:45 WARN [main] PlanChangeLogger:
=== Applying Rule
org.apache.gluten.extension.columnar.rewrite.RewriteSparkPlanRulesManager ===
Execute InsertIntoHadoopFsRelationCommand
hdfs:///user/hive/warehouse/test_hive_bucket, false, [dt#25], 8 buckets, bucket
columns: [bid], Parquet, [serialization.format=1, mergeSchema=false,
__hive_compatible_bucketed_table_insertion__=true,
partitionOverwriteMode=DYNAMIC], Overwrite,
`spark_catalog`.`default`.`test_hive_bucket`,
org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe,
org.apache.spark.sql.execution.datasources.CatalogFileIndex(hdfs:///user/hive/warehouse/test_hive_bucket),
[id, bid, dt] Execute InsertIntoHadoopFsRelationCommand
hdfs:///user/hive/warehouse/test_hive_bucket, false, [dt#25], 8 buckets, bucket
columns: [bid], Parquet, [serialization.format=1, mergeSchema=false,
__hive_compatible_bucketed_table_insertion__=true,
partitionOverwriteMode=DYNAMIC], Overwrite,
`spark_catalog`.`default`.`test_hive_bucket`,
org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe,
org.apache.spark.sql.execution.datasources.CatalogFileIndex(hdfs:///user/hive/warehou
se/test_hive_bucket), [id, bid, dt]
+- WriteFiles
+- WriteFiles
! +- Sort [dt#25 ASC NULLS FIRST, pmod((hive-hash(bid#17) & 2147483647),
8) ASC NULLS FIRST], false, 0
+- Project [id#15, bid#17, dt#25]
! +- Project [id#15, bid#17, empty2null(dt#16) AS dt#25]
+- Sort [dt#25 ASC NULLS FIRST, _pre_0#26 ASC NULLS FIRST], false, 0
! +- FileScan parquet spark_catalog.default.test[id#15,dt#16,bid#17]
Batched: true, DataFilters: [], Format: Parquet, Location: InMemoryFileIndex(1
paths)[hdfs://..., PartitionFilters: [], PushedFilters: [], ReadSchema:
struct<id:int,dt:string,bid:int>
+- Project [id#15, bid#17, dt#25,
pmod((hive-hash(bid#17) & 2147483647), 8) AS _pre_0#26]
!
+- Project [id#15, bid#17, empty2null(dt#16) AS dt#25]
!
+- FileScan parquet
spark_catalog.default.test[id#15,dt#16,bid#17] Batched: true, DataFilters: [],
Format: Parquet, Location: InMemoryFileIndex(1 paths)[hdfs://...,
PartitionFilters: [], PushedFilters: [], ReadSchema:
struct<id:int,dt:string,bid:int>
25/04/21 11:40:45 WARN [main] PlanChangeLogger:
=== Applying Rule
org.apache.gluten.extension.columnar.MiscColumnarRules$TransformPreOverrides ===
Execute InsertIntoHadoopFsRelationCommand
hdfs:///user/hive/warehouse/test_hive_bucket, false, [dt#25], 8 buckets, bucket
columns: [bid], Parquet, [serialization.format=1, mergeSchema=false,
__hive_compatible_bucketed_table_insertion__=true,
partitionOverwriteMode=DYNAMIC], Overwrite,
`spark_catalog`.`default`.`test_hive_bucket`,
org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe,
org.apache.spark.sql.execution.datasources.CatalogFileIndex(hdfs:///user/hive/warehouse/test_hive_bucket),
[id, bid, dt] Execute InsertIntoHadoopFsRelationCommand
hdfs:///user/hive/warehouse/test_hive_bucket, false, [dt#25], 8 buckets, bucket
columns: [bid], Parquet, [serialization.format=1, mergeSchema=false,
__hive_compatible_bucketed_table_insertion__=true,
partitionOverwriteMode=DYNAMIC], Overwrite,
`spark_catalog`.`default`.`test_hive_bucket`,
org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe,
org.apache.spark.sql.execution.datasources.CatalogFileIndex(hdfs:///user/hive/warehou
se/test_hive_bucket), [id, bid, dt]
+- WriteFiles
+- WriteFiles
! +- Project [id#15, bid#17, dt#25]
+- ProjectExecTransformer [id#15, bid#17, dt#25]
! +- Sort [dt#25 ASC NULLS FIRST, _pre_0#26 ASC NULLS FIRST], false, 0
+- SortExecTransformer [dt#25 ASC NULLS FIRST, _pre_0#26 ASC NULLS
FIRST], false, 0
+- Project [id#15, bid#17, dt#25, pmod((hive-hash(bid#17) &
2147483647), 8) AS _pre_0#26]
+- Project [id#15, bid#17, dt#25, pmod((hive-hash(bid#17)
& 2147483647), 8) AS _pre_0#26]
+- Project [id#15, bid#17, empty2null(dt#16) AS dt#25]
+- Project [id#15, bid#17, empty2null(dt#16) AS dt#25]
! +- FileScan parquet
spark_catalog.default.test[id#15,dt#16,bid#17] Batched: true, DataFilters: [],
Format: Parquet, Location: InMemoryFileIndex(1 paths)[hdfs://...,
PartitionFilters: [], PushedFilters: [], ReadSchema:
struct<id:int,dt:string,bid:int>
+- NativeFileScan parquet
spark_catalog.default.test[id#15,dt#16,bid#17] Batched: true, DataFilters: [],
Format: Parquet, Location: InMemoryFileIndex(1 paths)[hdfs://...,
PartitionFilters: [], PushedFilters: [], ReadSchema:
struct<id:int,dt:string,bid:int>
25/04/21 11:40:45 WARN [main] PlanChangeLogger:
=== Applying Rule
org.apache.gluten.extension.columnar.MiscColumnarRules$TransformPreOverrides ===
Execute InsertIntoHadoopFsRelationCommand
hdfs:///user/hive/warehouse/test_hive_bucket, false, [dt#25], 8 buckets, bucket
columns: [bid], Parquet, [serialization.format=1, mergeSchema=false,
__hive_compatible_bucketed_table_insertion__=true,
partitionOverwriteMode=DYNAMIC], Overwrite,
`spark_catalog`.`default`.`test_hive_bucket`,
org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe,
org.apache.spark.sql.execution.datasources.CatalogFileIndex(hdfs:///user/hive/warehouse/test_hive_bucket),
[id, bid, dt] Execute InsertIntoHadoopFsRelationCommand
hdfs:///user/hive/warehouse/test_hive_bucket, false, [dt#25], 8 buckets, bucket
columns: [bid], Parquet, [serialization.format=1, mergeSchema=false,
__hive_compatible_bucketed_table_insertion__=true,
partitionOverwriteMode=DYNAMIC], Overwrite,
`spark_catalog`.`default`.`test_hive_bucket`,
org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe,
org.apache.spark.sql.execution.datasources.CatalogFileIndex(hdfs:///user/hive/warehou
se/test_hive_bucket), [id, bid, dt]
+- WriteFiles
+- WriteFiles
! +- Project [id#15, bid#17, dt#25]
+- ProjectExecTransformer [id#15, bid#17, dt#25]
! +- Sort [dt#25 ASC NULLS FIRST, _pre_0#26 ASC NULLS FIRST], false, 0
+- SortExecTransformer [dt#25 ASC NULLS FIRST, _pre_0#26 ASC NULLS
FIRST], false, 0
+- Project [id#15, bid#17, dt#25, pmod((hive-hash(bid#17) &
2147483647), 8) AS _pre_0#26]
+- Project [id#15, bid#17, dt#25, pmod((hive-hash(bid#17)
& 2147483647), 8) AS _pre_0#26]
+- Project [id#15, bid#17, empty2null(dt#16) AS dt#25]
+- Project [id#15, bid#17, empty2null(dt#16) AS dt#25]
! +- FileScan parquet
spark_catalog.default.test[id#15,dt#16,bid#17] Batched: true, DataFilters: [],
Format: Parquet, Location: InMemoryFileIndex(1 paths)[hdfs://...,
PartitionFilters: [], PushedFilters: [], ReadSchema:
struct<id:int,dt:string,bid:int>
+- NativeFileScan parquet
spark_catalog.default.test[id#15,dt#16,bid#17] Batched: true, DataFilters: [],
Format: Parquet, Location: InMemoryFileIndex(1 paths)[hdfs://...,
PartitionFilters: [], PushedFilters: [], ReadSchema:
struct<id:int,dt:string,bid:int>
25/04/21 11:40:45 WARN [main] PlanChangeLogger:
=== Applying Rule
org.apache.gluten.extension.columnar.transition.InsertTransitions ===
Execute InsertIntoHadoopFsRelationCommand
hdfs:///user/hive/warehouse/test_hive_bucket, false, [dt#25], 8 buckets, bucket
columns: [bid], Parquet, [serialization.format=1, mergeSchema=false,
__hive_compatible_bucketed_table_insertion__=true,
partitionOverwriteMode=DYNAMIC], Overwrite,
`spark_catalog`.`default`.`test_hive_bucket`,
org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe,
org.apache.spark.sql.execution.datasources.CatalogFileIndex(hdfs:///user/hive/warehouse/test_hive_bucket),
[id, bid, dt] Execute InsertIntoHadoopFsRelationCommand
hdfs:///user/hive/warehouse/test_hive_bucket, false, [dt#25], 8 buckets, bucket
columns: [bid], Parquet, [serialization.format=1, mergeSchema=false,
__hive_compatible_bucketed_table_insertion__=true,
partitionOverwriteMode=DYNAMIC], Overwrite,
`spark_catalog`.`default`.`test_hive_bucket`,
org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe,
org.apache.spark.sql.execution.datasources.CatalogFileIndex(hdfs:///user/hive/warehou
se/test_hive_bucket), [id, bid, dt]
+- WriteFiles
+- WriteFiles
! +- ProjectExecTransformer [id#15, bid#17, dt#25]
+- VeloxColumnarToRowExec
! +- SortExecTransformer [dt#25 ASC NULLS FIRST, _pre_0#26 ASC NULLS
FIRST], false, 0
+- ProjectExecTransformer [id#15, bid#17, dt#25]
! +- Project [id#15, bid#17, dt#25, pmod((hive-hash(bid#17) &
2147483647), 8) AS _pre_0#26]
+- SortExecTransformer [dt#25 ASC NULLS FIRST, _pre_0#26
ASC NULLS FIRST], false, 0
! +- Project [id#15, bid#17, empty2null(dt#16) AS dt#25]
+- RowToVeloxColumnar
! +- NativeFileScan parquet
spark_catalog.default.test[id#15,dt#16,bid#17] Batched: true, DataFilters: [],
Format: Parquet, Location: InMemoryFileIndex(1 paths)[hdfs://...,
PartitionFilters: [], PushedFilters: [], ReadSchema:
struct<id:int,dt:string,bid:int>
+- Project [id#15, bid#17, dt#25,
pmod((hive-hash(bid#17) & 2147483647), 8) AS _pre_0#26]
!
+- Project [id#15, bid#17, empty2null(dt#16) AS dt#25]
!
+- VeloxColumnarToRowExec
!
+- NativeFileScan parquet
spark_catalog.default.test[id#15,dt#16,bid#17] Batched: true, DataFilters: [],
Format: Parquet, Location: InMemoryFileIndex(1 paths)[hdfs://...,
PartitionFilters: [], PushedFilters: [], ReadSchema:
struct<id:int,dt:string,bid:int>
25/04/21 11:40:45 WARN [main] PlanChangeLogger:
=== Result of Batch Columnar (Phase [transform]) ===
Execute InsertIntoHadoopFsRelationCommand
hdfs:///user/hive/warehouse/test_hive_bucket, false, [dt#25], 8 buckets, bucket
columns: [bid], Parquet, [serialization.format=1, mergeSchema=false,
__hive_compatible_bucketed_table_insertion__=true,
partitionOverwriteMode=DYNAMIC], Overwrite,
`spark_catalog`.`default`.`test_hive_bucket`,
org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe,
org.apache.spark.sql.execution.datasources.CatalogFileIndex(hdfs:///user/hive/warehouse/test_hive_bucket),
[id, bid, dt] Execute InsertIntoHadoopFsRelationCommand
hdfs:///user/hive/warehouse/test_hive_bucket, false, [dt#25], 8 buckets, bucket
columns: [bid], Parquet, [serialization.format=1, mergeSchema=false,
__hive_compatible_bucketed_table_insertion__=true,
partitionOverwriteMode=DYNAMIC], Overwrite,
`spark_catalog`.`default`.`test_hive_bucket`,
org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe,
org.apache.spark.sql.execution.datasources.CatalogFileIndex(hdfs:///user/hive/warehou
se/test_hive_bucket), [id, bid, dt]
+- WriteFiles
+- WriteFiles
! +- Sort [dt#25 ASC NULLS FIRST, pmod((hive-hash(bid#17) & 2147483647),
8) ASC NULLS FIRST], false, 0
+- VeloxColumnarToRowExec
! +- Project [id#15, bid#17, empty2null(dt#16) AS dt#25]
+- ProjectExecTransformer [id#15, bid#17, dt#25]
! +- ColumnarToRow
+- SortExecTransformer [dt#25 ASC NULLS FIRST, _pre_0#26 ASC
NULLS FIRST], false, 0
! +- FileScan parquet
spark_catalog.default.test[id#15,dt#16,bid#17] Batched: true, DataFilters: [],
Format: Parquet, Location: InMemoryFileIndex(1 paths)[hdfs://...,
PartitionFilters: [], PushedFilters: [], ReadSchema:
struct<id:int,dt:string,bid:int>
+- RowToVeloxColumnar
!
+- Project [id#15, bid#17, dt#25, pmod((hive-hash(bid#17)
& 2147483647), 8) AS _pre_0#26]
!
+- Project [id#15, bid#17, empty2null(dt#16) AS dt#25]
!
+- VeloxColumnarToRowExec
!
+- NativeFileScan parquet
spark_catalog.default.test[id#15,dt#16,bid#17] Batched: true, DataFilters: [],
Format: Parquet, Location: InMemoryFileIndex(1 paths)[hdfs://...,
PartitionFilters: [], PushedFilters: [], ReadSchema:
struct<id:int,dt:string,bid:int>
```
commit cce8d24c
```
25/04/21 11:43:02 WARN [main] PlanChangeLogger:
=== Applying Rule
org.apache.gluten.extension.columnar.transition.RemoveTransitions ===
Execute InsertIntoHadoopFsRelationCommand
hdfs:///user/hive/warehouse/test_hive_bucket, false, [dt#25], 8 buckets, bucket
columns: [bid], Parquet, [serialization.format=1, mergeSchema=false,
__hive_compatible_bucketed_table_insertion__=true,
partitionOverwriteMode=DYNAMIC], Overwrite,
`spark_catalog`.`default`.`test_hive_bucket`,
org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe,
org.apache.spark.sql.execution.datasources.CatalogFileIndex(hdfs:///user/hive/warehouse/test_hive_bucket),
[id, bid, dt] Execute InsertIntoHadoopFsRelationCommand
hdfs:///user/hive/warehouse/test_hive_bucket, false, [dt#25], 8 buckets, bucket
columns: [bid], Parquet, [serialization.format=1, mergeSchema=false,
__hive_compatible_bucketed_table_insertion__=true,
partitionOverwriteMode=DYNAMIC], Overwrite,
`spark_catalog`.`default`.`test_hive_bucket`,
org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe,
org.apache.spark.sql.execution.datasources.CatalogFileIndex(hdfs:///user/hive/warehou
se/test_hive_bucket), [id, bid, dt]
+- WriteFiles
+- WriteFiles
+- Sort [dt#25 ASC NULLS FIRST, pmod((hive-hash(bid#17) & 2147483647),
8) ASC NULLS FIRST], false, 0
+- Sort [dt#25 ASC NULLS FIRST, pmod((hive-hash(bid#17) &
2147483647), 8) ASC NULLS FIRST], false, 0
+- Project [id#15, bid#17, empty2null(dt#16) AS dt#25]
+- Project [id#15, bid#17, empty2null(dt#16) AS dt#25]
! +- ColumnarToRow
+- FileScan parquet
spark_catalog.default.test[id#15,dt#16,bid#17] Batched: true, DataFilters: [],
Format: Parquet, Location: InMemoryFileIndex(1 paths)[hdfs://...,
PartitionFilters: [], PushedFilters: [], ReadSchema:
struct<id:int,dt:string,bid:int>
! +- FileScan parquet
spark_catalog.default.test[id#15,dt#16,bid#17] Batched: true, DataFilters: [],
Format: Parquet, Location: InMemoryFileIndex(1 paths)[hdfs://...,
PartitionFilters: [], PushedFilters: [], ReadSchema:
struct<id:int,dt:string,bid:int>
25/04/21 11:43:02 WARN [main] PlanChangeLogger:
=== Applying Rule
org.apache.gluten.extension.columnar.heuristic.HeuristicTransform ===
Execute InsertIntoHadoopFsRelationCommand
hdfs:///user/hive/warehouse/test_hive_bucket, false, [dt#25], 8 buckets, bucket
columns: [bid], Parquet, [serialization.format=1, mergeSchema=false,
__hive_compatible_bucketed_table_insertion__=true,
partitionOverwriteMode=DYNAMIC], Overwrite,
`spark_catalog`.`default`.`test_hive_bucket`,
org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe,
org.apache.spark.sql.execution.datasources.CatalogFileIndex(hdfs:///user/hive/warehouse/test_hive_bucket),
[id, bid, dt] Execute InsertIntoHadoopFsRelationCommand
hdfs:///user/hive/warehouse/test_hive_bucket, false, [dt#25], 8 buckets, bucket
columns: [bid], Parquet, [serialization.format=1, mergeSchema=false,
__hive_compatible_bucketed_table_insertion__=true,
partitionOverwriteMode=DYNAMIC], Overwrite,
`spark_catalog`.`default`.`test_hive_bucket`,
org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe,
org.apache.spark.sql.execution.datasources.CatalogFileIndex(hdfs:///user/hive/warehou
se/test_hive_bucket), [id, bid, dt]
!+- WriteFiles
+- VeloxColumnarWriteFiles WriteFilesExecTransformer Parquet, [dt#25], 8
buckets, bucket columns: [bid], [serialization.format=1, mergeSchema=false,
__hive_compatible_bucketed_table_insertion__=true,
partitionOverwriteMode=DYNAMIC], Parquet, [dt#25], 8 buckets, bucket columns:
[bid], [serialization.format=1, mergeSchema=false, __hive_compatible_bucke
ted_table_insertion__=true, partitionOverwriteMode=DYNAMIC]
! +- Sort [dt#25 ASC NULLS FIRST, pmod((hive-hash(bid#17) & 2147483647),
8) ASC NULLS FIRST], false, 0
:- WriteFilesExecTransformer Parquet, [dt#25], 8 buckets, bucket
columns: [bid], [serialization.format=1, mergeSchema=false,
__hive_compatible_bucketed_table_insertion__=true,
partitionOverwriteMode=DYNAMIC]
! +- Project [id#15, bid#17, empty2null(dt#16) AS dt#25]
: +- ProjectExecTransformer [id#15, bid#17, dt#25]
! +- FileScan parquet spark_catalog.default.test[id#15,dt#16,bid#17]
Batched: true, DataFilters: [], Format: Parquet, Location: InMemoryFileIndex(1
paths)[hdfs://..., PartitionFilters: [], PushedFilters: [], ReadSchema:
struct<id:int,dt:string,bid:int>
: +- SortExecTransformer [dt#25 ASC
NULLS FIRST, _pre_0#26 ASC NULLS FIRST], false, 0
!
: +- Project [id#15, bid#17, dt#25, pmod((hive-hash(bid#17) &
2147483647), 8) AS _pre_0#26]
!
: +- ProjectExecTransformer [id#15, bid#17,
empty2null(dt#16) AS dt#25]
!
: +- FileScanTransformer parquet
spark_catalog.default.test[id#15,dt#16,bid#17] Batched: true, DataFilters: [],
Format: Parquet, Location: InMemoryFileIndex(1 paths)[hdfs://...,
PartitionFilters: [], PushedFilters: [], ReadSchema:
struct<id:int,dt:string,bid:int> NativeFilters: []
!
+- !WriteFiles
!
+- NoopLeaf
25/04/21 11:43:02 WARN [main] PlanChangeLogger:
=== Applying Rule
org.apache.gluten.extension.columnar.transition.InsertTransitions ===
Execute InsertIntoHadoopFsRelationCommand
hdfs:///user/hive/warehouse/test_hive_bucket, false, [dt#25], 8 buckets, bucket
columns: [bid], Parquet, [serialization.format=1, mergeSchema=false,
__hive_compatible_bucketed_table_insertion__=true,
partitionOverwriteMode=DYNAMIC], Overwrite,
`spark_catalog`.`default`.`test_hive_bucket`,
org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe,
org.apache.spark.sql.execution.datasources.CatalogFileIndex(hdfs:///user/hive/warehouse/test_hive_bucket),
[id, bid, dt] Execute InsertIntoHadoopFsRelationCommand
hdfs:///user/hive/warehouse/test_hive_bucket, false, [dt#25], 8 buckets, bucket
columns: [bid], Parquet, [serialization.format=1, mergeSchema=false,
__hive_compatible_bucketed_table_insertion__=true,
partitionOverwriteMode=DYNAMIC], Overwrite,
`spark_catalog`.`default`.`test_hive_bucket`,
org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe,
org.apache.spark.sql.execution.datasources.CatalogFileIndex(hdfs:///user/hive/warehou
se/test_hive_bucket), [id, bid, dt]
+- VeloxColumnarWriteFiles WriteFilesExecTransformer Parquet, [dt#25], 8
buckets, bucket columns: [bid], [serialization.format=1, mergeSchema=false,
__hive_compatible_bucketed_table_insertion__=true,
partitionOverwriteMode=DYNAMIC], Parquet, [dt#25], 8 buckets, bucket columns:
[bid], [serialization.format=1, mergeSchema=false,
__hive_compatible_bucketed_table_insertion__=true,
partitionOverwriteMode=DYNAMIC]
+- VeloxColumnarWriteFiles WriteFilesExecTransformer
Parquet, [dt#25], 8 buckets, bucket columns: [bid], [serialization.format=1,
mergeSchema=false, __hive_compatible_bucketed_table_insertion__=true,
partitionOverwriteMode=DYNAMIC], Parquet, [dt#25], 8 buckets, bucket columns:
[bid], [serialization.format=1, mergeSchema=false, __hive_compatible_bucke
ted_table_insertion__=true, partitionOverwriteMode=DYNAMIC]
:- WriteFilesExecTransformer Parquet, [dt#25], 8 buckets, bucket
columns: [bid], [serialization.format=1, mergeSchema=false,
__hive_compatible_bucketed_table_insertion__=true,
partitionOverwriteMode=DYNAMIC]
:-
WriteFilesExecTransformer Parquet, [dt#25], 8 buckets, bucket columns: [bid],
[serialization.format=1, mergeSchema=false,
__hive_compatible_bucketed_table_insertion__=true,
partitionOverwriteMode=DYNAMIC]
: +- ProjectExecTransformer [id#15, bid#17, dt#25]
: +- ProjectExecTransformer [id#15, bid#17, dt#25]
: +- SortExecTransformer [dt#25 ASC NULLS FIRST, _pre_0#26 ASC NULLS
FIRST], false, 0
: +- SortExecTransformer [dt#25 ASC NULLS FIRST, _pre_0#26 ASC
NULLS FIRST], false, 0
! : +- Project [id#15, bid#17, dt#25, pmod((hive-hash(bid#17) &
2147483647), 8) AS _pre_0#26]
: +- RowToVeloxColumnar
! : +- ProjectExecTransformer [id#15, bid#17, empty2null(dt#16)
AS dt#25]
: +- Project [id#15, bid#17, dt#25, pmod((hive-hash(bid#17)
& 2147483647), 8) AS _pre_0#26]
! : +- FileScanTransformer parquet
spark_catalog.default.test[id#15,dt#16,bid#17] Batched: true, DataFilters: [],
Format: Parquet, Location: InMemoryFileIndex(1 paths)[hdfs://...,
PartitionFilters: [], PushedFilters: [], ReadSchema:
struct<id:int,dt:string,bid:int> NativeFilters: []
: +- VeloxColumnarToRow
! +- !WriteFiles
: +- ProjectExecTransformer [id#15, bid#17,
empty2null(dt#16) AS dt#25]
! +- NoopLeaf
: +- FileScanTransformer parquet
spark_catalog.default.test[id#15,dt#16,bid#17] Batched: true, DataFilters: [],
Format: Parquet, Location: InMemoryFileIndex(1 paths)[hdfs://...,
PartitionFilters: [], PushedFilters: [], ReadSchema:
struct<id:int,dt:string,bid:int> NativeFilters: []
!
+- RowToVeloxColumnar
!
+- !WriteFiles
!
+- NoopLeaf
25/04/21 11:43:02 WARN [main] PlanChangeLogger:
=== Result of Batch Columnar (Phase [transform]) ===
Execute InsertIntoHadoopFsRelationCommand
hdfs:///user/hive/warehouse/test_hive_bucket, false, [dt#25], 8 buckets, bucket
columns: [bid], Parquet, [serialization.format=1, mergeSchema=false,
__hive_compatible_bucketed_table_insertion__=true,
partitionOverwriteMode=DYNAMIC], Overwrite,
`spark_catalog`.`default`.`test_hive_bucket`,
org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe,
org.apache.spark.sql.execution.datasources.CatalogFileIndex(hdfs:///user/hive/warehouse/test_hive_bucket),
[id, bid, dt] Execute InsertIntoHadoopFsRelationCommand
hdfs:///user/hive/warehouse/test_hive_bucket, false, [dt#25], 8 buckets, bucket
columns: [bid], Parquet, [serialization.format=1, mergeSchema=false,
__hive_compatible_bucketed_table_insertion__=true,
partitionOverwriteMode=DYNAMIC], Overwrite,
`spark_catalog`.`default`.`test_hive_bucket`,
org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe,
org.apache.spark.sql.execution.datasources.CatalogFileIndex(hdfs:///user/hive/warehou
se/test_hive_bucket), [id, bid, dt]
!+- WriteFiles
+- VeloxColumnarWriteFiles WriteFilesExecTransformer Parquet, [dt#25], 8
buckets, bucket columns: [bid], [serialization.format=1, mergeSchema=false,
__hive_compatible_bucketed_table_insertion__=true,
partitionOverwriteMode=DYNAMIC], Parquet, [dt#25], 8 buckets, bucket columns:
[bid], [serialization.format=1, mergeSchema=false, __hive_compatible_bucke
ted_table_insertion__=true, partitionOverwriteMode=DYNAMIC]
! +- Sort [dt#25 ASC NULLS FIRST, pmod((hive-hash(bid#17) & 2147483647),
8) ASC NULLS FIRST], false, 0
:- WriteFilesExecTransformer Parquet, [dt#25], 8 buckets, bucket
columns: [bid], [serialization.format=1, mergeSchema=false,
__hive_compatible_bucketed_table_insertion__=true,
partitionOverwriteMode=DYNAMIC]
! +- Project [id#15, bid#17, empty2null(dt#16) AS dt#25]
: +- ProjectExecTransformer [id#15, bid#17, dt#25]
! +- ColumnarToRow
: +- SortExecTransformer [dt#25 ASC NULLS FIRST, _pre_0#26 ASC
NULLS FIRST], false, 0
! +- FileScan parquet
spark_catalog.default.test[id#15,dt#16,bid#17] Batched: true, DataFilters: [],
Format: Parquet, Location: InMemoryFileIndex(1 paths)[hdfs://...,
PartitionFilters: [], PushedFilters: [], ReadSchema:
struct<id:int,dt:string,bid:int>
: +- RowToVeloxColumnar
!
: +- Project [id#15, bid#17, dt#25, pmod((hive-hash(bid#17)
& 2147483647), 8) AS _pre_0#26]
!
: +- VeloxColumnarToRow
!
: +- ProjectExecTransformer [id#15, bid#17,
empty2null(dt#16) AS dt#25]
!
: +- FileScanTransformer parquet
spark_catalog.default.test[id#15,dt#16,bid#17] Batched: true, DataFilters: [],
Format: Parquet, Location: InMemoryFileIndex(1 paths)[hdfs://...,
PartitionFilters: [], PushedFilters: [], ReadSchema:
struct<id:int,dt:string,bid:int> NativeFilters: []
!
+- RowToVeloxColumnar
!
+- !WriteFiles
!
+- NoopLeaf
```
@zhztheplayer
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
