Joy-2000 opened a new issue, #20437:
URL: https://github.com/apache/hudi/issues/20437
### Bug Description
**What happened:**
In a CPU flame graph of a Spark query over a Hudi Copy-on-Write (COW)
Parquet table, the same scan pipeline shows two row projections:
| Stack | Inclusive samples | Share of 7,258 samples |
| --- | ---: | ---: |
| Hudi `HoodieFileGroupReaderBasedFileFormat.projectIter ->
SpecificUnsafeProjection.apply` | 812 | 11.19% |
| Spark `FileSourceScanExec.doExecute -> SpecificUnsafeProjection.apply` |
291 | 4.01% |
Both projections spend samples in generated `writeFields_*` methods. The
Hudi projection also spends samples in `UnsafeRowWriter.write` and decimal
reads. `VectorizedParquetRecordReader` and `VectorizedColumnReader` appear in
the same scan, so Parquet decoding is vectorized even though this plan supplies
rows upstream.
This suggests that a row may be materialized by Hudi and then materialized
again by Spark. The flame graph establishes two projection stages, but does not
establish that every row passes through both or that the second write is
redundant. No query-result error or exception was observed.
**What you expected:**
If the Hudi and Spark projections serve the same purpose and produce an
equivalent row with the required schema, column order, values, and lifetime, we
expect one of the two materializations to be eliminated. If they perform
different necessary work, we expect the reason for both projections to be
identified and the duplicate-copy hypothesis to be ruled out.
**Steps to reproduce:**
1. Set spark.sql.codegen.maxFields=30
2. Read all columns of a cow table (like TPCDS catalog_sales table) whose
fields greater than 30;
3. Capture a task CPU flame graph
Query SQL：
```
SELECT
sum(length(cs_sold_time_sk)),
sum(length(cs_ship_date_sk)),
sum(length(cs_bill_customer_sk)),
sum(length(cs_bill_cdemo_sk)),
sum(length(cs_bill_hdemo_sk)),
sum(length(cs_bill_addr_sk)),
sum(length(cs_ship_customer_sk)),
sum(length(cs_ship_cdemo_sk)),
sum(length(cs_ship_hdemo_sk)),
sum(length(cs_ship_addr_sk)),
sum(length(cs_call_center_sk)),
sum(length(cs_catalog_page_sk)),
sum(length(cs_ship_mode_sk)),
sum(length(cs_warehouse_sk)),
sum(length(cs_item_sk)),
sum(length(cs_promo_sk)),
sum(length(cs_order_number)),
sum(length(cs_quantity)),
sum(length(cs_wholesale_cost)),
sum(length(cs_list_price)),
sum(length(cs_sales_price)),
sum(length(cs_ext_discount_amt)),
sum(length(cs_ext_sales_price)),
sum(length(cs_ext_wholesale_cost)),
sum(length(cs_ext_list_price)),
sum(length(cs_ext_tax)),
sum(length(cs_coupon_amt)),
sum(length(cs_ext_ship_cost)),
sum(length(cs_net_paid)),
sum(length(cs_net_paid_inc_tax)),
sum(length(cs_net_paid_inc_ship)),
sum(length(cs_net_paid_inc_ship_tax)),
sum(length(cs_net_profit)),
sum(length(cs_sold_date_sk))
FROM tmp.tpcds_sf1000_catalog_sales;
```
### Environment
**Hudi version:** 1.1.1
**Query engine:** (Spark/Flink/Trino etc) Spark 3.4
**Relevant configs:** set spark.sql.codegen.maxFields=30;
<img width="1200" height="2102" alt="Image"
src="https://github.com/user-attachments/assets/4482903f-89ed-40f2-b5c0-6355454c663b";
/>
### Logs and Stack Trace
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
