[ https://issues.apache.org/jira/browse/SPARK-6354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14368649#comment-14368649 ]
Liang-Chi Hsieh commented on SPARK-6354: ---------------------------------------- h2. Introduction Currently we use the cached data in SparkSQL by looking for fully the same logical plan. The logic is implemented in {code}CacheManager.useCachedData{code}. If we find a cached version of the logical plan, we can replace it with the cached version. This ticker expands the approach and looks for the the logical plan that contains all output of the given logical plan. If we find a cached plan satisfying this condition, we can replace it with the cached version. h2. Current approach The comparison logic is in {code}LogicalPlan.sameResult{code}. To have two logical plans considered the same one, it should satisfy few conditions: # They are the same class # Their children sizes are the same # Their cleanArgs are the same # All their children satisfying the conditions above h2. Proposed approach This ticker wants to expand the current approach. The expanded approach uses the cached data by looking for the logical plan that is superset of current logical plan. In other words, the current logical plan will return part of the results of the cached plan. The comparison logic is in {code}LogicalPlan.partResult{code}. It has a parameter {code}plan: LogicalPlan{code}. To have the given {code}plan{code} considered the part of another logical plan (called {code}this plan{code} below), it should also satisfy few conditions: # They are the same class # Their children sizes are the same # The cleanArgs of given {code}plan{code] are contained in {code}this plan{code} # All their children satisfying the conditions above In order to test if the condition 3 is satisfied, we iterate through the elements in the cleanArgs of the given {code}plan{code}. For each element, we check if the cleanArgs of {code}this plan{code} contains it. If any element is not, the condition 3 is failed. Basically, the proposed approach just relaxes one of the previous conditions. Previously, the condition requires that two plans are having the same args. Now, it is modified to only require that the args in the given plan are all contained in another plan. If the condition is met, the given plan is the part of the another plan. > Replace the plan which is part of cached query > ---------------------------------------------- > > Key: SPARK-6354 > URL: https://issues.apache.org/jira/browse/SPARK-6354 > Project: Spark > Issue Type: Improvement > Components: SQL > Reporter: Liang-Chi Hsieh > Priority: Minor > > Currently we only replace the plan which equals to cached query. This > approach can be extended to replace the plan which is part of cached query. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org