Copilot commented on code in PR #12453:
URL: https://github.com/apache/gluten/pull/12453#discussion_r3550503195
##########
backends-velox/src/main/scala/org/apache/gluten/backendsapi/velox/VeloxBackend.scala:
##########
@@ -233,7 +233,7 @@ object VeloxBackendSettings extends BackendSettingsApi {
}
def validateDataSchema(): Option[String] = {
- if (VeloxConfig.get.parquetUseColumnNames &&
VeloxConfig.get.orcUseColumnNames) {
+ if (VeloxConfig.get.parquetUseColumnNames) {
return None
}
Review Comment:
`validateDataSchema()` now only skips validation based on
`parquetUseColumnNames`, but ORC/DWRF scans can also use (and now always
attach) the table schema for positional mapping (forced via
`orc.force.positional.evolution` or per-file `_col*` fallback). This can bypass
schema-type validation for ORC/DWRF and lead to native failures instead of an
early fallback when unsupported Spark types appear in the table schema.
##########
backends-velox/src/test/scala/org/apache/gluten/execution/FallbackSuite.scala:
##########
@@ -316,27 +316,23 @@ class FallbackSuite extends
VeloxWholeStageTransformerSuite with AdaptiveSparkPl
format =>
Seq("true", "false").foreach {
parquetUseColumnNames =>
- Seq("true", "false").foreach {
- orcUseColumnNames =>
- withSQLConf(
- VeloxConfig.PARQUET_USE_COLUMN_NAMES.key ->
parquetUseColumnNames,
- VeloxConfig.ORC_USE_COLUMN_NAMES.key -> orcUseColumnNames
- ) {
- withTable("test") {
- spark
- .range(100)
- .selectExpr("to_timestamp_ntz(from_unixtime(id % 3)) as
c1", "id as c2")
- .write
- .format(format)
- .saveAsTable("test")
-
- runQueryAndCompare(query) {
- df =>
- val plan = df.queryExecution.executedPlan
- assert(collect(plan) { case g: GlutenPlan => g
}.nonEmpty)
- }
- }
+ withSQLConf(
+ VeloxConfig.PARQUET_USE_COLUMN_NAMES.key -> parquetUseColumnNames
+ ) {
Review Comment:
The `parquetUseColumnNames` loop is still applied when `format` is `orc`,
which makes the ORC branch run twice with identical behavior (the parquet-only
config doesn’t affect ORC). This adds redundant test runtime after removing the
ORC dimension.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
