Re: How to load partial data from HDFS using Spark SQL

from table where id = ") > //filtered data frame > df.count > > On Sat, Jan 2, 2016 at 11:56 AM, SRK <swethakasire...@gmail.com> wrote: > >> Hi, >> >> How to load partial data from hdfs using Spark SQL? Suppose I want to load >> data based on a filter like &

How to load partial data from HDFS using Spark SQL

Hi, How to load partial data from hdfs using Spark SQL? Suppose I want to load data based on a filter like "Select * from table where id = " using Spark SQL with DataFrames, how can that be done? The idea here is that I do not want to load the whole data into memory when I use the

Re: How to load partial data from HDFS using Spark SQL

Ok, so whats wrong in using : var df=HiveContext.sql("Select * from table where id = ") //filtered data frame df.count On Sat, Jan 2, 2016 at 11:56 AM, SRK <swethakasire...@gmail.com> wrote: > Hi, > > How to load partial data from hdfs using Spark SQL? Suppose I