srielau commented on code in PR #58317:
URL: https://github.com/apache/spark/pull/58317#discussion_r4085800604
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/CacheManager.scala:
##########
@@ -422,7 +464,9 @@ class CacheManager extends Logging with
AdaptiveSparkPlanHelper {
case r @ ExtractV2CatalogAndIdentifier(catalog, ident) if
r.timeTravelSpec.isEmpty =>
val table = CatalogV2Util.getTable(catalog, ident, options =
r.options)
if (r.table.id == table.id) {
- Some(DataSourceV2Relation.create(table, Some(catalog),
Some(ident), r.options))
+ Some(DataSourceV2Relation
+ .create(table, Some(catalog), Some(ident), r.options)
+ .copy(charVarcharScanMode = r.charVarcharScanMode))
Review Comment:
Tracked in SPARK-59751: https://issues.apache.org/jira/browse/SPARK-59751.
Explicit Legacy identity and preserving it through recache/refresh/rename are
out of SPARK-58814. I reverted b8d27c0e so that work lands on 59751 with the
rest of the cache-lifecycle contract.
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/LogicalRelation.scala:
##########
@@ -41,7 +41,10 @@ case class LogicalRelation(
output: Seq[AttributeReference],
catalogTable: Option[CatalogTable],
override val isStreaming: Boolean,
- @transient stream: Option[SparkDataStream])
+ @transient stream: Option[SparkDataStream],
+ // Bound at analysis so sameResult / cache reuse distinguish preserve-only
vs standard
+ // CHAR/VARCHAR scans. None means the relation was not analyzed under
first-class types.
+ charVarcharScanMode: Option[CharVarcharScanMode])
Review Comment:
The SPARK-58814 refresh path keeps V1 catalog/BaseRelation and Hive
qualified-name dispatch. The remaining three-policy V1/V2/Hive lifecycle
matrix, including isolated restoration, is SPARK-59751:
https://issues.apache.org/jira/browse/SPARK-59751
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/SaveIntoDataSourceCommand.scala:
##########
@@ -71,7 +71,7 @@ case class SaveIntoDataSourceCommand(
try {
val logicalRelation = LogicalRelation(relation,
toAttributes(relation.schema), None,
- false, None)
+ false, None, None)
Review Comment:
The V1 save recache-by-BaseRelation path stays here. The FakeV1DataSource
LONG-only fixture cannot distinguish bound scan modes; a CHAR/VARCHAR-bound
save regression is SPARK-59751:
https://issues.apache.org/jira/browse/SPARK-59751
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/InsertIntoDataSourceCommand.scala:
##########
@@ -45,7 +45,8 @@ case class InsertIntoDataSourceCommand(
// Re-cache all cached plans(including this relation itself, if it's
cached) that refer to this
// data source relation.
- sparkSession.sharedState.cacheManager.recacheByPlan(sparkSession,
logicalRelation)
+ sparkSession.sharedState.cacheManager.recacheByV1Relation(
Review Comment:
Agreed that the InsertSuite fixture is still INT/STRING, so the branches
stay unbound. Moving a CHAR/VARCHAR-bound insert recache regression to
SPARK-59751: https://issues.apache.org/jira/browse/SPARK-59751
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
