Hi, On 06/12/2014 05:47 PM, Toby Douglass wrote:
> In these future jobs, when I come to load the aggregted RDD, will Spark > load and only load the columns being accessed by the query? or will Spark > load everything, to convert it into an internal representation, and then > execute the query? The aforementioned native Parquet support in Spark 1.0 supports column projections which means only the columns that appear in the query will be loaded. The next release will also support record filters for simple pruning predicates ("int-column smaller value" and such). This is different from using a Hadoop Input/Output format and requires no additional setup (jars in classpath and such). For more details see: http://spark.apache.org/docs/latest/sql-programming-guide.html#using-parquet Andre