hudi-agent commented on code in PR #20193:
URL: https://github.com/apache/hudi/pull/20193#discussion_r4235264786
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/HoodieFileGroupReaderBasedFileFormat.scala:
##########
@@ -346,8 +346,8 @@ class HoodieFileGroupReaderBasedFileFormat(tablePath:
String,
val dataStructTypeWithMandatoryPartitionFields =
StructType(dataStructType.fields ++ partitionSchema.fields.filter(f =>
mandatoryFields.contains(f.name) && !isNestedPartitionField(f.name)))
val dataSchema = HoodieSchemaUtils.pruneDataSchema(schema,
HoodieSchemaConversionUtils.convertStructTypeToHoodieSchema(dataStructTypeWithMandatoryPartitionFields,
sanitizedTableName), exclusionFields)
-
spark.sessionState.conf.setConfString("spark.sql.parquet.enableVectorizedReader",
supportVectorizedRead.toString)
-
+ // Keep vectorization query-local; mutating the session configuration here
leaks this
Review Comment:
🤖 That failure fits how the incremental relations work: they hand
`incrementalSpanRecordFilters` to the base-file readers as required filters
(`MergeOnReadIncrementalRelationV2.scala:76`), so the commit-range cut seems to
depend on parquet record-level filtering being on, and a follow-up would
probably need to set those two flags on the per-read conf passed to the reader
rather than just drop them.
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/functional/TestIncrementalReadWithFileGroupReader.scala:
##########
@@ -177,6 +178,53 @@ class TestIncrementalReadWithFileGroupReader extends
SparkClientFunctionalTestHa
Set(("k1", 5), ("k2", 4), ("k3", 5), ("k4", 5)))
}
+ @Test
+ def testCowIncrementalReadDoesNotMutateVectorizedReaderSessionConf(): Unit =
{
+ val configs = Seq(
+ "spark.sql.parquet.enableVectorizedReader" -> "true"
+ )
+
+ val previousConfigs = configs.map { case (key, _) =>
Review Comment:
🤖 nit: could you drop the `configs`/`previousConfigs` Seq machinery and use
`withSQLConf` (or a single key constant) since only one key is set?
<sub><i>⚠️ AI-generated; verify before applying. React 👍/👎 to flag
quality.</i></sub>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
