Pyspark dataframe read

2015-10-06 Thread Blaž Šnuderl
Hello everyone. It seems pyspark dataframe read is broken for reading multiple files. sql.read.json( "file1,file2") fails with java.io.IOException: No input paths specified in job. This used to work in spark 1.4 and also still work with sc.textFile Blaž

Re: Pyspark dataframe read

2015-10-06 Thread Koert Kuipers
i ran into the same thing in scala api. we depend heavily on comma separated paths, and it no longer works. On Tue, Oct 6, 2015 at 3:02 AM, Blaž Šnuderl <snud...@gmail.com> wrote: > Hello everyone. > > It seems pyspark dataframe read is broken for reading multiple files. &g

Re: Pyspark dataframe read

2015-10-06 Thread Reynold Xin
t; On Tue, Oct 6, 2015 at 3:02 AM, Blaž Šnuderl <snud...@gmail.com >> <javascript:_e(%7B%7D,'cvml','snud...@gmail.com');>> wrote: >> >>> Hello everyone. >>> >>> It seems pyspark dataframe read is broken for reading multiple files. >>> >>>

Re: Pyspark dataframe read

2015-10-06 Thread Koert Kuipers
t;> separated paths, and it no longer works. >>> >>> >>> On Tue, Oct 6, 2015 at 3:02 AM, Blaž Šnuderl <snud...@gmail.com> wrote: >>> >>>> Hello everyone. >>>> >>>> It seems pyspark dataframe read is broken for reading

Re: Pyspark dataframe read

2015-10-06 Thread Josh Rosen
t; > On Tue, Oct 6, 2015 at 3:02 AM, Blaž Šnuderl <snud...@gmail.com> wrote: > >> Hello everyone. >> >> It seems pyspark dataframe read is broken for reading multiple files. >> >> sql.read.json( "file1,file2") fails with java.io.IOException: No