Hi Users, I am trying to build fault tolerant spark solace consumer.
Issue :- we have to take restart of the job due to multiple issue load average is one of them. At that time whatever spark is processing or batches in the queue is lost. We can't replay it because we already had send ack while calling store(). Solution:- I have tried implementing WAL and checkpointing in the solution. Job is able to identify the lost batches, records are not being written in the log file but throwing NPE. We are creating sparkcontext using sc.getorcreate() Thanks, Nayan
