cloud-fan commented on code in PR #58317:
URL: https://github.com/apache/spark/pull/58317#discussion_r3928083280
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/ApplyCharTypePadding.scala:
##########
@@ -51,34 +51,71 @@ object ApplyCharTypePadding extends Rule[LogicalPlan] {
}
override def apply(plan: LogicalPlan): LogicalPlan = {
+ val standardSemantics = conf.charVarcharStandardSemantics
+
+ // Bind into case-class state, not a TreeNodeTag: tags are dropped by
canonicalization
Review Comment:
**Nit (P3):** `TreeNode.makeCopy` calls `copyTagsFrom`, so canonicalized
copies retain `TreeNodeTag` values. The reason a tag is unsuitable here is that
it does not participate in structural plan equality / `sameResult`, not that
canonicalization drops it. Could this comment be reworded accordingly?
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/DataSourceScanExec.scala:
##########
@@ -752,15 +757,30 @@ case class FileSourceScanExec(
lazy val inputRDD: RDD[InternalRow] = {
val options = relation.options +
(FileFormat.OPTION_RETURNING_BATCH -> supportsColumnar.toString)
- val readFile: (PartitionedFile) => Iterator[InternalRow] =
- relation.fileFormat.buildReaderWithPartitionValues(
- sparkSession = relation.sparkSession,
- dataSchema = relation.dataSchema,
- partitionSchema = relation.partitionSchema,
- requiredSchema = requiredSchema,
- filters = pushedDownFilters,
- options = options,
- hadoopConf = getHadoopConf(relation.sparkSession, relation.options))
+ val hadoopConf = getHadoopConf(relation.sparkSession, relation.options)
+ val readFile: (PartitionedFile) => Iterator[InternalRow] =
relation.fileFormat match {
+ case format: OrcFileFormat
Review Comment:
**Blocking (P1):** This exact-class branch restores the existing subclass
override, but it still loses the analyzed mode when that override delegates to
`super`: the public `OrcFileFormat.buildReaderWithPartitionValues` overload
always forwards `charVarcharStandardSemantics = false`. Under standard
semantics, a delegating subclass can therefore read physical `"abcdef"` as
`VARCHAR(4)`, receive native ORC truncation to `"abcd"`, and bypass Spark's
`EXCEED_LIMIT_LENGTH` check.
Please remove the ORC-specific dispatch here by adding a mode-aware overload
to `FileFormat.buildReaderWithPartitionValues`. Its default implementation
should clone the per-call Hadoop configuration, attach the bound mode as
engine-owned compatibility context, and invoke the existing seven-argument
method virtually. Existing subclasses will continue to receive their old
override; when an ORC override calls `super`, the public ORC implementation can
recover the mode from that cloned context and call its internal reader builder.
Direct legacy callers of the old method can retain the current preserve-native
default.
Please extend the subclass regression with an over-length value: bound
`true` must raise `EXCEED_LIMIT_LENGTH`, while bound `false` retains the
intended native truncation. This should cover both row and vectorized readers.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
