Re: How to use DStream reparation() ?

2016-01-29 Thread Andy Davidson
ntegration.com> Date: Friday, January 29, 2016 at 1:54 PM To: "user @spark" <user@spark.apache.org> Subject: How to use DStream reparation() ? > My Streaming app has a requirement that my output be saved in the smallest > number of file possible such that each file

How to use DStream reparation() ?

2016-01-29 Thread Andy Davidson
My Streaming app has a requirement that my output be saved in the smallest number of file possible such that each file does not exceed a max number of rows. Based on my experience it appears that each partition will be written to separate output file. This was really easy to do in my batch