pan3793 opened a new pull request, #58314:
URL: https://github.com/apache/spark/pull/58314
### What changes were proposed in this pull request?
Add an internal config `spark.sql.useSequentialCacheName` (default `false`).
When it is true and the cached table has no name, `CachedRDDBuilder` uses a
sequential number like `CachedRDD 1` as the cached name instead of the
abbreviated plan tree string:
```scala
val cachedName: String = tableName.map(n => s"In-memory table $n").getOrElse
{
if (cachedPlan.session.conf.get(SQLConf.USE_SEQUENTIAL_CACHE_NAME)) {
s"CachedRDD ${CachedRDDBuilder.nextCachedRDDId()}"
} else {
Utils.abbreviate(cachedPlan.toString, 1024)
}
}
```
### Why are the changes needed?
For anonymous cached tables, the cached name is built from the plan's tree
string (`cachedPlan.toString`, abbreviated to 1024 chars). Rendering the plan
tree string can be expensive for large plans, so caching unnamed large
DataFrames pays this cost even though the name is only used for display.
This is another spot, besides the SQL event plan description addressed in
SPARK-59023, that hurts the same customer job: it constructs a huge plan whose
`treeString` exceeds 280,000 lines, and rendering the plan tree string takes
minutes per iteration and contributes to driver OOM.
### Does this PR introduce _any_ user-facing change?
No. The new config is internal and defaults to `false`, keeping the existing
cached name behavior.
### How was this patch tested?
A new unit test in `InMemoryRelationSuite` (`sequential cached name for
anonymous cached tables`) verifies:
- anonymous cached tables get distinct `CachedRDD <n>` names when the config
is enabled
- named tables keep the usual `In-memory table <name>` name
- the default behavior (abbreviated plan tree string) is unchanged
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Qwen3.8 Max
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
