vicennial commented on code in PR #46012:
URL: https://github.com/apache/spark/pull/46012#discussion_r1565802827
##########
connector/connect/server/src/main/scala/org/apache/spark/sql/connect/service/SessionHolder.scala:
##########
@@ -381,6 +405,53 @@ case class SessionHolder(userId: String, sessionId:
String, session: SparkSessio
*/
private[connect] val pythonAccumulator: Option[PythonAccumulator] =
Try(session.sparkContext.collectionAccumulator[Array[Byte]]).toOption
+
+ /**
+ * Transform a relation into a logical plan, using the plan cache if enabled.
+ * The plan cache is enable only if
`spark.connect.session.planCache.maxSize` is greater than zero
+ * AND `spark.connect.session.planCache.enabled` is true.
+ * @param rel The relation to transform.
+ * @param cachePlan Whether to cache the result logical plan.
+ * @param transform Function to transform the relation into a logical plan.
+ * @return The logical plan.
+ */
+ private[connect] def usePlanCache(rel: proto.Relation, cachePlan: Boolean)(
Review Comment:
> 1, LOCAL_RELATION, its size might be large;
> 2, CACHED_REMOTE_RELATION, it is already cached in dataFrameCache in the
SessionHolder;
We use a low default value for `CONNECT_SESSION_PLAN_CACHE_SIZE`, `5`
entries to be specific, to prevent the cache from becoming too large. We
haven't been too surgical in what kind of plans we cache to keep it as a simple
wrapper in aiding common scenarios that may generate some repeated
analysis/execute calls.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
