HI ,
You can try this
sqlContext.read.format("json").option("samplingRatio","0.1").load("path")
If it still takes time , feel free to experiment with the samplingRatio.
Thanks,
Vishnu
On Wed, Jan 6, 2016 at 12:43 PM, Gavin Yue <[email protected]> wrote:
> I am trying to read json files following the example:
>
> val path = "examples/src/main/resources/jsonfile"val people =
> sqlContext.read.json(path)
>
> I have 1 Tb size files in the path. It took 1.2 hours to finish the reading
> to infer the schema.
>
> But I already know the schema. Could I make this process short?
>
> Thanks a lot.
>
>
>
>
