Re: Structured Streaming partition logic with respect to storage and fileformat

) > val dsText = ds.as[String].map(x =>(x.split(" ")(0),x.split(" > ")(1))).toDF("name","age") > > val dsParquet = > sqlContext.readStream.format("parquet").parquet("/Users/sachin/testSpark/inputParquet") > > > > -- > > Thanks & Regards > > Sachin Aggarwal > 7760502772 > > -- Thanks & Regards Sachin Aggarwal 7760502772

Structured Streaming partition logic with respect to storage and fileformat

rquet = sqlContext.readStream.format("parquet").parquet("/Users/sachin/testSpark/inputParquet") -- Thanks & Regards Sachin Aggarwal 7760502772

Re: submissionTime vs batchTime, DirectKafka

ference > between the batchTime and SubmissionTime for that nth batch > > > thanks > Mario > > > > > > > On Thu, Mar 10, 2016 at 10:29 AM, Sachin Aggarwal < > *different.sac...@gmail.com* <different.sac...@gmail.com>> wrote: > >Hi cody

Re: submissionTime vs batchTime, DirectKafka

what you're asking. > > On Wed, Mar 9, 2016 at 12:43 PM, Sachin Aggarwal > <different.sac...@gmail.com> wrote: > > where are we capturing this delay? > > I am aware of scheduling delay which is defined as processing > > time-submission time not the batch creat

Re: submissionTime vs batchTime, DirectKafka

tch until the > current batch is finished. So if your processing time is larger than > your batch time, delays will build up. > > On Wed, Mar 9, 2016 at 11:09 AM, Sachin Aggarwal > <different.sac...@gmail.com> wrote: > > Hi All, > > > > we have batchTime and

add to user list

-- Thanks Regards Sachin Aggarwal 7760502772