Hello Nayanjyoti, Did you find anything else from the streams log entries (is it enabled on DEBUG or TRACE?), and what version of Kafka are you using?
Guozhang On Sun, May 19, 2019 at 1:04 PM Nayanjyoti Deka <nayanjy...@ixigo.com> wrote: > Forgot to add that there is no transition to RUNNING state. > > On Mon, May 20, 2019 at 1:10 AM Nayanjyoti Deka <nayanjy...@ixigo.com> > wrote: > > > Hey guys, > > > > We are running a stream application in our production environment. On our > > latest restart, the application is consistently moving between these two > > states. > > > > From our logs: > > > > grep "State transition from " application.log | jq -r '.message' | sort | > > uniq -c | sort -n -r > > > > 40 stream-thread [yyyy-StreamThread-9] State transition from > > PARTITIONS_REVOKED to PARTITIONS_ASSIGNED > > > > 40 stream-thread [yyyy-StreamThread-8] State transition from > > PARTITIONS_REVOKED to PARTITIONS_ASSIGNED > > > > 40 stream-thread [yyyy-StreamThread-7] State transition from > > PARTITIONS_REVOKED to PARTITIONS_ASSIGNED > > > > 40 stream-thread [yyyy-StreamThread-6] State transition from > > PARTITIONS_REVOKED to PARTITIONS_ASSIGNED > > > > 40 stream-thread [yyyy-StreamThread-5] State transition from > > PARTITIONS_REVOKED to PARTITIONS_ASSIGNED > > > > 40 stream-thread [yyyy-StreamThread-4] State transition from > > PARTITIONS_REVOKED to PARTITIONS_ASSIGNED > > > > 40 stream-thread [yyyy-StreamThread-3] State transition from > > PARTITIONS_REVOKED to PARTITIONS_ASSIGNED > > > > 40 stream-thread [yyyy-StreamThread-2] State transition from > > PARTITIONS_REVOKED to PARTITIONS_ASSIGNED > > > > 40 stream-thread [yyyy-StreamThread-1] State transition from > > PARTITIONS_REVOKED to PARTITIONS_ASSIGNED > > > > 40 stream-thread [yyyy-StreamThread-12] State transition from > > PARTITIONS_REVOKED to PARTITIONS_ASSIGNED > > > > 40 stream-thread [yyyy-StreamThread-11] State transition from > > PARTITIONS_REVOKED to PARTITIONS_ASSIGNED > > > > 40 stream-thread [yyyy-StreamThread-10] State transition from > > PARTITIONS_REVOKED to PARTITIONS_ASSIGNED > > > > 39 stream-thread [yyyy-StreamThread-9] State transition from > > PARTITIONS_ASSIGNED to PARTITIONS_REVOKED > > > > 39 stream-thread [yyyy-StreamThread-8] State transition from > > PARTITIONS_ASSIGNED to PARTITIONS_REVOKED > > > > 39 stream-thread [yyyy-StreamThread-7] State transition from > > PARTITIONS_ASSIGNED to PARTITIONS_REVOKED > > > > 39 stream-thread [yyyy-StreamThread-6] State transition from > > PARTITIONS_ASSIGNED to PARTITIONS_REVOKED > > > > 39 stream-thread [yyyy-StreamThread-5] State transition from > > PARTITIONS_ASSIGNED to PARTITIONS_REVOKED > > > > 39 stream-thread [yyyy-StreamThread-4] State transition from > > PARTITIONS_ASSIGNED to PARTITIONS_REVOKED > > > > 39 stream-thread [yyyy-StreamThread-3] State transition from > > PARTITIONS_ASSIGNED to PARTITIONS_REVOKED > > > > 39 stream-thread [yyyy-StreamThread-2] State transition from > > PARTITIONS_ASSIGNED to PARTITIONS_REVOKED > > > > 39 stream-thread [yyyy-StreamThread-1] State transition from > > PARTITIONS_ASSIGNED to PARTITIONS_REVOKED > > > > 39 stream-thread [yyyy-StreamThread-12] State transition from > > PARTITIONS_ASSIGNED to PARTITIONS_REVOKED > > > > 39 stream-thread [yyyy-StreamThread-11] State transition from > > PARTITIONS_ASSIGNED to PARTITIONS_REVOKED > > > > 39 stream-thread [yyyy-StreamThread-10] State transition from > > PARTITIONS_ASSIGNED to PARTITIONS_REVOKED > > > > > > > > As we can see the stream threads are first revoked than again assigned. > > > > Also we can see the logs of resetting of offsets continuously as follows: > > > > Resetting offset for partition xxxx-2 to offset 9166288. > > > > > > We had actually deleted the consumer group on broker before the restart > as > > there was considerable lag in the topic and processing of the stale data > > was not intended. We had assumed that on deleting the group, the > > application will start processing from latest offset as mentioned in the > > config auto.offset.reset policy. > > > > On describing the consumer group on broker side, we receive output with > > the current offset and lag set as -- (Eg shown below) > > > > > > TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG > > CONSUMER-ID > > > > xxx 10 - 129822997 - > > yyyy-StreamThread-2-consumer-a-b-c-d > > > > > > > > Please help us understand why this can be happening and how to solve > this. > > > > > > > -- -- Guozhang
