parquetFile accepts a comma separated list of files. Also, unionAll does not write to disk. However, unless you are running a recent version (compiled from master since this was added <https://github.com/apache/spark/commit/f858f466862541c3faad76a1fa2391f1c17ec9dd>) its missing an optimization and thus reads all the columns, instead of just the ones required, from the disk
On Thu, Oct 2, 2014 at 6:05 PM, Mohnish Kodnani <[email protected]> wrote: > Hi, > I am trying to play around with Spark and Spark SQL. > I have logs being stored in HDFS on a 10 minute window. Each 10 minute > window could have as many as 10 files with random names of 2GB each. > Now, I want to run some analysis on these files. These files are parquet > files. > I am trying to run Spark SQL queries on them. > I notice that the API only can take a single parquet File and not a > directory or a GLOB pattern where all the files can be loaded as a single > Schema RDD. > > I tried doing a unionAll, but from the output of the job it looked like it > was merging the files and writing to disk ( not confirmed but from the time > it took I am assuming). > > I tried insertInto, but that definitely wrote to disk and times were > comparable to unionAll operation. > Is there a way to run jobs on multiple files as if they were a single RDD. > I am not restricted to using Spark SQL, this is what I started to play > around with. > What has stopped us from creating an API that takes a GLOB pattern and > create a single RDD from all of the files inside. > > Thanks > mohnish > >
