LuciferYang commented on code in PR #13506:
URL: https://github.com/apache/gluten/pull/13506#discussion_r4549940219
##########
gluten-substrait/src/main/scala/org/apache/gluten/execution/WriteFilesExecTransformer.scala:
##########
@@ -68,7 +68,15 @@ case class WriteFilesExecTransformer(
override def output: Seq[Attribute] = Seq.empty
- val caseInsensitiveOptions: CaseInsensitiveMap[String] =
CaseInsensitiveMap(options)
+ /**
+ * Write options used for validation and to build the native writer
configuration. For a
+ * `HiveFileFormat` the Hive table properties are merged in: `options` then
only carries the
+ * bucket-write flag, while vanilla Spark resolves the Parquet compression
codec from the table
+ * properties (e.g. `TBLPROPERTIES ('parquet.compression'='zstd')`) via
+ * `HiveOptions.getHiveWriteCompression`. Explicit `options` take precedence.
+ */
+ val caseInsensitiveOptions: CaseInsensitiveMap[String] =
+ CaseInsensitiveMap(HiveFileFormatUtil.tableProperties(fileFormat) ++
options)
Review Comment:
A Hive Parquet table that sets both `compression` and `parquet.compression`
still gets a different codec from vanilla. `getCompressionCodec` takes
`compression` first, as `ParquetOptions` does. Vanilla also tries `compression`
first, but at task time Hive's
`ParquetRecordWriterWrapper.initializeSerProperties` overwrites
`parquet.compression` on the conf with the table's `parquet.compression`, so on
a Hive write that property wins. With `TBLPROPERTIES ('compression'='snappy',
'parquet.compression'='zstd')`, vanilla writes ZSTD and the native writer
writes SNAPPY. With `'compression'='brotli'` instead, validation rejects brotli
and the write falls back, although the codec vanilla ends up using is ZSTD,
which Velox can write.
The merge only happens for `HiveFileFormat`, so it can let the table's
`parquet.compression` win there, for example by dropping the table's
`compression` when `parquet.compression` is set and non-empty.
##########
backends-velox/src/main/scala/org/apache/gluten/backendsapi/velox/VeloxTransformerApi.scala:
##########
@@ -164,7 +164,8 @@ class VeloxTransformerApi extends TransformerApi with
Logging {
val shortName = "parquet"
val writeOptions = Option(write.session).map {
session =>
- val hadoopConf =
session.sessionState.newHadoopConfWithOptions(write.options)
+ val hadoopConf =
+
session.sessionState.newHadoopConfWithOptions(write.caseInsensitiveOptions)
Review Comment:
Explicit write options now reach the Hadoop conf with lower-cased keys,
because `newHadoopConfWithOptions` iterates the `CaseInsensitiveMap` it gets.
This also affects plain `ParquetFileFormat` writes. With `SET
parquet.gzip.windowSize=...` and `.option("parquet.gzip.windowSize", "4096")`,
the conf ends up with both `parquet.gzip.windowSize` and
`parquet.gzip.windowsize`. When the conf is folded back into a
`CaseInsensitiveMap`, which of the two is kept depends on hash order, so the
explicit option can lose.
Passing `write.caseInsensitiveOptions.originalMap` keeps the original keys
and still includes the table properties.
##########
gluten-substrait/src/main/scala/org/apache/gluten/utils/HiveFileFormatUtil.scala:
##########
@@ -0,0 +1,63 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.gluten.utils
+
+import org.apache.spark.sql.execution.datasources.FileFormat
+import org.apache.spark.sql.hive.execution.HiveFileFormat
+
+import java.util.Properties
+
+import scala.collection.JavaConverters._
+
+/**
+ * Accessors for the Hive `TableDesc` a [[HiveFileFormat]] writes to.
`HiveFileFormat` keeps its
+ * `FileSinkDesc` in a private constructor parameter and `TableDesc` lives in
hive-exec, so both are
+ * reached through reflection.
+ */
+object HiveFileFormatUtil {
+ val PARQUET_OUTPUT_FORMAT =
"org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat"
+
+ private def tableInfo(format: HiveFileFormat): AnyRef = {
+ val fileSinkConfField = format.getClass.getDeclaredField("fileSinkConf")
+ fileSinkConfField.setAccessible(true)
+ val fileSinkConf = fileSinkConfField.get(format)
+ val tableInfoField = fileSinkConf.getClass.getDeclaredField("tableInfo")
+ tableInfoField.setAccessible(true)
+ tableInfoField.get(fileSinkConf)
+ }
+
+ /** The Hive output format class name, e.g. [[PARQUET_OUTPUT_FORMAT]]. */
+ def outputFileFormatClassName(format: HiveFileFormat): String = {
+ val info = tableInfo(format)
+
info.getClass.getMethod("getOutputFileFormatClassName").invoke(info).asInstanceOf[String]
+ }
+
+ /**
+ * The Hive table properties, including TBLPROPERTIES such as
`parquet.compression`. Vanilla Spark
+ * copies these into the job configuration
(`Utilities.copyTableJobPropertiesToConf`) and resolves
+ * the Parquet compression codec from them in
`HiveOptions.getHiveWriteCompression`, so the native
+ * writer has to see them as well to produce the same files. Empty for every
other [[FileFormat]].
+ */
+ def tableProperties(format: FileFormat): Map[String, String] = format match {
+ case hive: HiveFileFormat =>
+ val info = tableInfo(hive)
+ val properties =
+
info.getClass.getMethod("getProperties").invoke(info).asInstanceOf[Properties]
Review Comment:
`TableDesc` itself doesn't need reflection: gluten-substrait already imports
`org.apache.hadoop.hive.ql.plan.TableDesc` in `HiveTableScanExecTransformer`.
Only two steps do: `fileSinkConf` is private in `HiveFileFormat` up to Spark
4.0, and the class that holds `tableInfo` differs between Spark 3.4 and 3.5+.
If `tableInfo(...)` returned a `TableDesc`, the reflective
`getOutputFileFormatClassName` and `getProperties` calls could become
`info.getOutputFileFormatClassName` and `info.getProperties`, and the class
comment could give the real reason for the reflection that's left.
##########
backends-velox/src/main/scala/org/apache/gluten/backendsapi/velox/VeloxTransformerApi.scala:
##########
@@ -164,7 +164,8 @@ class VeloxTransformerApi extends TransformerApi with
Logging {
val shortName = "parquet"
val writeOptions = Option(write.session).map {
session =>
- val hadoopConf =
session.sessionState.newHadoopConfWithOptions(write.options)
+ val hadoopConf =
+
session.sessionState.newHadoopConfWithOptions(write.caseInsensitiveOptions)
Review Comment:
A table property now beats a session value for every lower-case key
`nativeConf` reads, because the table properties are set on top of the session
Hadoop conf. Vanilla Hive only does that for `parquet.compression`,
`parquet.block.size` and `parquet.enable.dictionary`
(`ParquetRecordWriterWrapper.initializeSerProperties`). Other table properties
go through `Utilities.copyTableJobPropertiesToConf`, which copies a key only
when the job conf doesn't already have it. So with `SET
parquet.writer.version=v2` and a table that has
`'parquet.writer.version'='v1'`, vanilla writes v2 pages, while the native
writer now writes v1 (before this PR it wrote v2). `parquet.page.size` has the
same problem.
Matching Hive would mean overriding only those three keys and setting the
others only when the session conf doesn't have them. Since the data stays
readable, documenting it is also fine: the comment on
`HiveFileFormatUtil.tableProperties` could say that table properties override
session values.
##########
gluten-substrait/src/main/scala/org/apache/gluten/utils/HiveFileFormatUtil.scala:
##########
@@ -0,0 +1,63 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.gluten.utils
+
+import org.apache.spark.sql.execution.datasources.FileFormat
+import org.apache.spark.sql.hive.execution.HiveFileFormat
+
+import java.util.Properties
+
+import scala.collection.JavaConverters._
+
+/**
+ * Accessors for the Hive `TableDesc` a [[HiveFileFormat]] writes to.
`HiveFileFormat` keeps its
+ * `FileSinkDesc` in a private constructor parameter and `TableDesc` lives in
hive-exec, so both are
+ * reached through reflection.
+ */
+object HiveFileFormatUtil {
+ val PARQUET_OUTPUT_FORMAT =
"org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat"
+
+ private def tableInfo(format: HiveFileFormat): AnyRef = {
+ val fileSinkConfField = format.getClass.getDeclaredField("fileSinkConf")
+ fileSinkConfField.setAccessible(true)
+ val fileSinkConf = fileSinkConfField.get(format)
+ val tableInfoField = fileSinkConf.getClass.getDeclaredField("tableInfo")
+ tableInfoField.setAccessible(true)
+ tableInfoField.get(fileSinkConf)
+ }
+
+ /** The Hive output format class name, e.g. [[PARQUET_OUTPUT_FORMAT]]. */
+ def outputFileFormatClassName(format: HiveFileFormat): String = {
+ val info = tableInfo(format)
+
info.getClass.getMethod("getOutputFileFormatClassName").invoke(info).asInstanceOf[String]
+ }
+
+ /**
+ * The Hive table properties, including TBLPROPERTIES such as
`parquet.compression`. Vanilla Spark
+ * copies these into the job configuration
(`Utilities.copyTableJobPropertiesToConf`) and resolves
+ * the Parquet compression codec from them in
`HiveOptions.getHiveWriteCompression`, so the native
+ * writer has to see them as well to produce the same files. Empty for every
other [[FileFormat]].
+ */
+ def tableProperties(format: FileFormat): Map[String, String] = format match {
+ case hive: HiveFileFormat =>
+ val info = tableInfo(hive)
+ val properties =
+
info.getClass.getMethod("getProperties").invoke(info).asInstanceOf[Properties]
+ properties.stringPropertyNames().asScala.map(k => k ->
properties.getProperty(k)).toMap
Review Comment:
`caseInsensitiveOptions` now includes every table property, so validation
also reads write options like `maxRecordsPerFile` from the table, which vanilla
Hive writes never do. With `TBLPROPERTIES ('maxRecordsPerFile'='0')` and
`spark.sql.files.maxRecordsPerFile=1000`, the write used to fall back and
vanilla split files at 1000 rows. Now it goes native and the limit is ignored.
A non-numeric value makes the `.toLong` in validation throw instead of falling
back.
The native writer only needs the Parquet keys here, so `tableProperties`
could keep just `parquet.*` plus `compression`, filtered before the `++
options` merge so the bucket flag in `options` is untouched. `compression` has
to stay because vanilla's `HiveOptions.getHiveWriteCompression` reads it.
##########
backends-velox/src/test/scala/org/apache/spark/sql/execution/VeloxParquetWriteForHiveSuite.scala:
##########
@@ -267,11 +267,7 @@ class VeloxParquetWriteForHiveSuite extends
GlutenQueryTest with SQLTestUtils wi
}
}
- // Ignored: the native writer uses spark.sql.parquet.compression.codec
instead of the
- // parquet.compression table property, so this writes GZIP where zstd is
expected. The
- // comparison below used to be a discarded Boolean, which is why the case
passed before.
- // TODO: fix and re-enable, see #12987.
- ignore("native writer should respect table properties") {
+ test("native writer should respect table properties") {
Review Comment:
The PR also changes validation: `validateCompressionCodec` now sees the
table's `parquet.compression`, so a Hive table whose property names a codec
Velox rejects falls back instead of being written natively with the session
codec. The re-enabled test only covers `zstd`, which passes validation either
way, so no test checks the fallback.
Could you add a case with `TBLPROPERTIES ('parquet.compression'='lz4_raw')`
that asserts `expectNative = false`? It's the only codec in Velox's unsupported
set that the vanilla Hive writer can write without an extra codec jar (brotli
and lzo need jars Spark doesn't ship). Spark 3.4's `ParquetOptions` rejects
`lz4_raw`, so the case needs `testWithMinSparkVersion(..., "3.5")`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]