How to set the file size for parquet Part

2015-05-19 Thread Richard Grossman
Hi

I'm using spark 1.3.1 and now I can't set the size of the part generated
file for parquet.
The size is only 512Kb it's really to small I must made them bigger.
How can set this ?
Thanks


Write Parquet File with spark-streaming with Spark 1.3

2015-03-26 Thread Richard Grossman
Hi

I've succeed to write kafka stream to parquet file in Spark 1.2 but I can't
make it with spark 1.3
As in streaming I can't use saveAsParquetFile() because I can't add data to
an existing parquet File

I know that it's possible to stream data directly into parquet
could you help me by providing a little sample what API I need to use ??

Thanks