Thanks Nick, DB - that was helpful.

On Mon, Oct 13, 2014 at 5:44 PM, DB Tsai <[email protected]> wrote:

> For now, with SparkSPARK-3462 parquet pushdown for unionAll PR, you
> can do the following for unionAll schemaRDD.
>
>   val files = Array("hdfs://file1.parquet", "hdfs://file2.parquet",
> "hdfs://file3.parquet")
>   val rdds = paths.map(hc.parquetFile(_))
>
>   val unionedRDD = {
>     var temp = rdds(0)
>     for (i <- 1 until rdds.length) {
>       temp = temp.unionAll(rdds(i))
>     }
>     temp
>   }
>
> Sincerely,
>
> DB Tsai
> -------------------------------------------------------
> My Blog: https://www.dbtsai.com
> LinkedIn: https://www.linkedin.com/in/dbtsai
>
>
> On Mon, Oct 13, 2014 at 7:22 PM, Nicholas Chammas
> <[email protected]> wrote:
> > Right now I believe the only supported option is to pass a
> comma-delimited
> > list of paths.
> >
> > I've opened SPARK-3928: Support wildcard matches on Parquet files to
> request
> > this feature.
> >
> > Nick
> >
> > On Mon, Oct 13, 2014 at 12:21 PM, Sadhan Sood <[email protected]>
> wrote:
> >>
> >> How can we read all parquet files in a directory in spark-sql. We are
> >> following this example which shows a way to read one file:
> >>
> >> // Read in the parquet file created above.  Parquet files are
> >> self-describing so the schema is preserved.
> >> // The result of loading a Parquet file is also a SchemaRDD.
> >> val parquetFile = sqlContext.parquetFile("people.parquet")
> >>
> >> //Parquet files can also be registered as tables and then used in SQL
> >> statements.
> >> parquetFile.registerTempTable("parquetFile")
> >
> >
>

Reply via email to