[ https://issues.apache.org/jira/browse/SPARK-4849?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14594193#comment-14594193 ]
Eric Pederson commented on SPARK-4849: -------------------------------------- Does this also apply to in-memory tables created as the result of cached partitioned hive tables? For example, say {{hivetable}} is partitioned by {{(s string, c string)}}. {code} val sql = new HiveContext(sc) val t = sql.table("hivetable") val c1 = t.cached() val f1 = c1.filter("s = 'FNM30' and c = '3.0'") val s1 = f1.groupBy("g").sum("a", "b", "c") {code} Should it be able to prune parts {{c1}} because of the original paritioning? > Pass partitioning information (distribute by) to In-memory caching > ------------------------------------------------------------------ > > Key: SPARK-4849 > URL: https://issues.apache.org/jira/browse/SPARK-4849 > Project: Spark > Issue Type: Improvement > Components: SQL > Affects Versions: 1.2.0 > Reporter: Nitin Goyal > Priority: Minor > > HQL "distribute by <column_name>" partitions data based on specified column > values. We can pass this information to in-memory caching for further > performance improvements. e..g. in Joins, an extra partition step can be > saved based on this information. > Refer - > http://apache-spark-user-list.1001560.n3.nabble.com/SchemaRDD-partition-on-specific-column-values-td20350.html -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org