srielau commented on code in PR #58317:
URL: https://github.com/apache/spark/pull/58317#discussion_r3927534912
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/SaveIntoDataSourceCommand.scala:
##########
@@ -71,7 +71,7 @@ case class SaveIntoDataSourceCommand(
try {
val logicalRelation = LogicalRelation(relation,
toAttributes(relation.schema), None,
- false, None)
+ false, None, None)
Review Comment:
Addressed in 4b6e0e564ab. Added CacheManager.recacheByV1Relation, which
matches cached LogicalRelations by the written BaseRelation and ignores the
CHAR/VARCHAR result-semantics field. SaveIntoDataSourceCommand (and
InsertIntoDataSourceCommand) now invoke it instead of building an unbound
LogicalRelation probe, so a successful V1 save invalidates None, Some(false),
and Some(true) cache entries for that relation. Ordinary sameResult/cache-reuse
identity is unchanged, so preserve-only and standard read plans stay distinct.
Extended the SPARK-39952 regression to repeat the save/recache cycle under
preserve-only and standard semantics.
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/DataSourceScanExec.scala:
##########
@@ -752,15 +757,28 @@ case class FileSourceScanExec(
lazy val inputRDD: RDD[InternalRow] = {
val options = relation.options +
(FileFormat.OPTION_RETURNING_BATCH -> supportsColumnar.toString)
- val readFile: (PartitionedFile) => Iterator[InternalRow] =
- relation.fileFormat.buildReaderWithPartitionValues(
- sparkSession = relation.sparkSession,
- dataSchema = relation.dataSchema,
- partitionSchema = relation.partitionSchema,
- requiredSchema = requiredSchema,
- filters = pushedDownFilters,
- options = options,
- hadoopConf = getHadoopConf(relation.sparkSession, relation.options))
+ val hadoopConf = getHadoopConf(relation.sparkSession, relation.options)
+ val readFile: (PartitionedFile) => Iterator[InternalRow] =
relation.fileFormat match {
+ case format: OrcFileFormat if charVarcharStandardSemantics.isDefined =>
+ format.buildReaderWithPartitionValues(
Review Comment:
Addressed in 4b6e0e564ab. FileSourceScanExec now takes the mode-aware
eight-argument ORC reader only when format.getClass == classOf[OrcFileFormat],
so external OrcFileFormat subclasses fall through to their public
seven-argument buildReaderWithPartitionValues override. No private[sql] method
is required from subclasses, and the built-in ORC path keeps analyzed-mode
row/vector readers. Added a TrackingOrcFileFormat subclass regression whose
public reader override must run under both Some(false) and Some(true).
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/orc/OrcUtils.scala:
##########
@@ -535,11 +548,14 @@ object OrcUtils extends Logging {
dataSchema: StructType,
resultSchema: StructType,
partitionSchema: StructType,
- conf: Configuration): String = {
+ conf: Configuration,
+ charVarcharStandardSemantics: Boolean): String = {
Review Comment:
Addressed in 4b6e0e564ab. Added the @param charVarcharStandardSemantics
entry on orcResultSchemaString: true requests physical ORC STRING for
Spark-side CHAR/VARCHAR length checks, false keeps native constrained ORC types.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
