Joy-2000 opened a new issue, #20438: URL: https://github.com/apache/hudi/issues/20438
### Describe the problem you faced [20261026-184822.svg.zip](https://github.com/user-attachments/files/33965524/20261026-184822.svg.zip) While investigating the CPU cost of a Spark query over a Hudi Copy-on-Write (COW) Parquet table, we found two `UnsafeProjection` stages on the scan path: | Sampled stack | Inclusive samples | Share of 7,258 samples | | --- | ---: | ---: | | Hudi `HoodieFileGroupReaderBasedFileFormat.projectIter -> SpecificUnsafeProjection.apply` | 812 | 11.19% | | Spark `FileSourceScanExec.doExecute -> SpecificUnsafeProjection.apply` | 291 | 4.01% | Both have generated `writeFields_*` calls beneath them. The scan also contains `VectorizedParquetRecordReader` and `VectorizedColumnReader`, indicating vectorized Parquet decoding while this plan supplies rows upstream. We have not observed a wrong query result or an exception. Could maintainers help clarify what each projection is required to do in this COW row-output path? In particular, can a row already projected by Hudi pass through Spark without another full-row materialization when its schema, column order, representation, and lifetime satisfy Spark's requirements? The profile shows two stages, but does not establish that every row is copied twice or that either stage is redundant. ### To Reproduce 1. Set spark.sql.codegen.maxFields=30 2. Read all columns of a cow table (like TPCDS catalog_sales table) whose fields greater than 30; 3. Capture a task CPU flame graph Query SQL： ``` SELECT sum(length(cs_sold_time_sk)), sum(length(cs_ship_date_sk)), sum(length(cs_bill_customer_sk)), sum(length(cs_bill_cdemo_sk)), sum(length(cs_bill_hdemo_sk)), sum(length(cs_bill_addr_sk)), sum(length(cs_ship_customer_sk)), sum(length(cs_ship_cdemo_sk)), sum(length(cs_ship_hdemo_sk)), sum(length(cs_ship_addr_sk)), sum(length(cs_call_center_sk)), sum(length(cs_catalog_page_sk)), sum(length(cs_ship_mode_sk)), sum(length(cs_warehouse_sk)), sum(length(cs_item_sk)), sum(length(cs_promo_sk)), sum(length(cs_order_number)), sum(length(cs_quantity)), sum(length(cs_wholesale_cost)), sum(length(cs_list_price)), sum(length(cs_sales_price)), sum(length(cs_ext_discount_amt)), sum(length(cs_ext_sales_price)), sum(length(cs_ext_wholesale_cost)), sum(length(cs_ext_list_price)), sum(length(cs_ext_tax)), sum(length(cs_coupon_amt)), sum(length(cs_ext_ship_cost)), sum(length(cs_net_paid)), sum(length(cs_net_paid_inc_tax)), sum(length(cs_net_paid_inc_ship)), sum(length(cs_net_paid_inc_ship_tax)), sum(length(cs_net_profit)), sum(length(cs_sold_date_sk)) FROM tmp.tpcds_sf1000_catalog_sales; ``` ### Expected behavior If Hudi's and Spark's projections serve the same purpose and produce equivalent rows with the required schema, column order, values, and lifetime, we expect one of the two materializations to be avoidable. If both are necessary, we would like to understand their distinct responsibilities and whether there is another safe way to reduce the row-projection cost. ### Environment Description * Hudi version: 1.1.1 * Spark version: 3.4 * Flink version: * Hive version: * Hadoop version: * Storage (HDFS/S3/GCS..): * Running on Docker? (yes/no): ### Additional context _No response_ ### Stacktrace ```shell ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
