Fwd: Time stamp in Kafka

Hi, > > We have a producer application that has written data to Kafka topic. > > We are reading the data from Kafka topic using spark streaming but the time > stamp on Kafka is 1969-12-31 format for all the data. > > Is there a way to fix this while reading ? > > Thanks, > Asmath >

Re: Appropriate checkpoint interval in a spark streaming application

Guys any inputs explaining the rationale on the below question will really help. Requesting some expert opinion. Regards, Sheel On Sat, 15 Aug, 2020, 1:47 PM Sheel Pancholi, wrote: > Hello, > > I am trying to figure an appropriate checkpoint interval for my spark > streaming application. Its

Re: Spark3 on k8S reading encrypted data from HDFS with KMS in HA

Hi guys, Does anyone have an idea on this issue? even some tips to troubleshoot it? I got the impression that after the creation of the delegation for the KMS, the token is not sent to the executor or maybe not saved? I'm sure I'm not the only one using Spark with HDFS encrypted with KMS :-)

Appropriate checkpoint interval in a spark streaming application

Hello, I am trying to figure an appropriate checkpoint interval for my spark streaming application. Its Spark Kafka integration based on Direct Streams. If my *micro batch interval is 2 mins*, and let's say *each microbatch takes only 15 secs to process* then shouldn't my checkpoint interval