I'm going to check the value, but I didn't change it., normally, the process is always running but sometimes I have to restarted to apply some changes. Sometimes it starts from the beginning and others continue for the last offset.
El mié., 27 feb. 2019 a las 12:25, Akshay Bhardwaj (< akshay.bhardwaj1...@gmail.com>) escribió: > Hi Gabor, > > I am talking about offset.retention.minutes which is set default as 1440 > (or 24 hours) > > Akshay Bhardwaj > +91-97111-33849 > > > On Wed, Feb 27, 2019 at 4:47 PM Gabor Somogyi <gabor.g.somo...@gmail.com> > wrote: > >> Hi Akshay, >> >> The feature what you've mentioned has a default value of 7 days... >> >> BR, >> G >> >> >> On Wed, Feb 27, 2019 at 7:38 AM Akshay Bhardwaj < >> akshay.bhardwaj1...@gmail.com> wrote: >> >>> Hi Guillermo, >>> >>> What was the interval in between restarting the spark job? As a feature >>> in Kafka, a broker deleted offsets for a consumer group after inactivity of >>> 24 hours. >>> In such a case, the newly started spark streaming job will read offsets >>> from beginning for the same groupId. >>> >>> Akshay Bhardwaj >>> +91-97111-33849 >>> >>> >>> On Thu, Feb 21, 2019 at 9:08 PM Gabor Somogyi <gabor.g.somo...@gmail.com> >>> wrote: >>> >>>> From the info you've provided not much to say. >>>> Maybe you could collect sample app, logs etc, open a jira and we can >>>> take a deeper look at it... >>>> >>>> BR, >>>> G >>>> >>>> >>>> On Thu, Feb 21, 2019 at 4:14 PM Guillermo Ortiz <konstt2...@gmail.com> >>>> wrote: >>>> >>>>> I' working with Spark Streaming 2.0.2 and Kafka 1.0.0 using Direct >>>>> Stream as connector. I consume data from Kafka and autosave the offsets. >>>>> I can see Spark doing commits in the logs of the last offsets >>>>> processed, Sometimes I have restarted spark and it starts from the >>>>> beginning, when I'm using the same groupId. >>>>> >>>>> Why could it happen? it only happen rarely. >>>>> >>>>
