Github user liancheng commented on a diff in the pull request:
https://github.com/apache/spark/pull/8023#discussion_r37427369
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/CacheManager.scala ---
@@ -156,10 +165,27 @@ private[sql] class CacheManager(sqlContext:
SQLContext) extends Logging {
* function will over invalidate.
*/
private[sql] def invalidateCache(plan: LogicalPlan): Unit = writeLock {
- cachedData.foreach {
- case data if data.plan.collect { case p if p.sameResult(plan) => p
}.nonEmpty =>
- data.cachedRepresentation.recache()
- case _ =>
+ var i = 0
+ var locatedIdx = -1
+ // find the index of the cached data, according to the specified
logical plan
+ while (i < cachedData.length && locatedIdx < 0) {
+ cachedData(i) match {
+ case data if data.plan.collect { case p if p.sameResult(plan) => p
}.nonEmpty =>
+ locatedIdx = i
+ case _ =>
+ }
+ i += 1
+ }
+
+ if (locatedIdx >= 0) {
+ // if the cached data exists, remove it from the cache data list, as
we need to
+ // re-generate the spark plan, and we don't want the this to be used
during the
+ // re-generation
+ val entry = cachedData.remove(locatedIdx) // TODO do we have to use
ArrayBuffer?
+ // rebuild the cache
+ entry.recache(sqlContext)
+ // add it back to the cache data list
+ cachedData += entry
--- End diff --
A problem of this change is that, `plan` can appear multiple times in
`cachedData`. For example (PySpark snippet):
```python
df0 = sqlContext.range(10)
df1 = df0.filter(df0.id > 5).cache()
df2 = df0.filter(df0.id > 1).cache()
df1.count()
df2.count()
```
In the above case, query plan of `df0` appears twice in `df1` and `df2`.
Since this method isn't performance critical, we can correct and simplify the
block within `writeLock` to:
```scala
cachedData.foreach { cached =>
if (cached.plan.find(_.sameResult(plan)).isDefined) {
cached.recache()
}
}
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
