mazhengxuan created HBASE-30562:
-----------------------------------

             Summary: Spark 4 connector reads the wrong Avro fields after 
nested schema pruning
                 Key: HBASE-30562
                 URL: https://issues.apache.org/jira/browse/HBASE-30562
             Project: HBase
          Issue Type: Sub-task
          Components: hbase-connectors
            Reporter: mazhengxuan


Selecting a nested field from an Avro record in the Spark 4 HBase connector can 
return the wrong value or fail with a `ClassCastException`.

`HBaseScan.readSchema()` reports the pruned struct, but `HBasePartitionReader` 
decodes and returns the full struct from the catalog. Spark reads the field by 
its position in the reported schema. If `second` is the only field in that 
schema, Spark reads position 0, which still contains `first` in the returned 
row.

### Minimal reproduction
 # With Spark 4.0.0 and the Spark 4 HBase connector, create a table named 
`nested_probe` in the `default` namespace with column family `cf`.
 # Write one row with key `row1`. Store a single binary Avro record in `cf:p` 
using the schema below, with `first = "first-value"`, `second = 
"second-value"`, and `quantity = 42`.
 # Read the table and run the two projections separately.

```scala
spark.conf.set("spark.sql.optimizer.nestedSchemaPruning.enabled", "true")

val avroSchema = """{
  "type": "record",
  "name": "Payload",
  "fields": [
    \{"name": "first", "type": "string"},
    \{"name": "second", "type": "string"},
    \{"name": "quantity", "type": "int"}
  ]
}"""

val catalog = """{
  "table": \{"namespace": "default", "name": "nested_probe"},
  "rowkey": "key",
  "columns": {
    "key": \{"cf": "rowkey", "col": "key", "type": "string"},
    "payload": \{"cf": "cf", "col": "p", "avro": "payloadSchema"}
  }
}"""

val df = spark.read
  .format("org.apache.hadoop.hbase.spark.datasources.HBaseTableProvider")
  .option("catalog", catalog)
  .option("payloadSchema", avroSchema)
  .option("hbase.spark.config.location", "/path/to/hbase-site.xml")
  .load()

df.select("payload.second").collect()
df.select("payload.quantity").collect()
```

| Projection | Expected | Actual |
| --- | --- | --- |
| `payload.second` | `second-value` | `first-value` |
| `payload.quantity` | `42` | `ClassCastException`: `UTF8String` cannot be cast 
to `Integer` |

Both queries return the expected values when 
`spark.sql.optimizer.nestedSchemaPruning.enabled` is set to `false`. Reproduced 
with Spark 4.0.0 and an HBase MiniCluster using connector commit `52e5231`.

### Proposed fix

Keep top-level column pruning in `HBaseScanBuilder.pruneColumns()`, but retain 
the full catalog type for each selected column. This makes the scan schema 
match what the reader returns and lets Spark select the nested fields. Add a 
regression test for both projections with nested schema pruning enabled.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

Reply via email to