In the particular case I'm referring to, I believe some partitions were replicated to new nodes. These new nodes had been added to the cluster as a result of human error, and were removed shortly thereafter. It's my hypothesis that this resulted in many partitions having their replicas on nodes that were no longer in the cluster and were never expected to rejoin.
Maybe this was a case in which the partition reassignment CLI tool would have been useful? On Thu, Sep 6, 2018 at 11:09 PM Brett Rann <br...@zendesk.com.invalid> wrote: > We have partitions that are in the 100s of GBs. > > It shouldn't have to shuffle around GB chunks of data unless you have done > partition moves, or had a broker offline for a while. Is that the case? > > If not your ISR problem is probably related to something else other than > retention size. > > On Fri, Sep 7, 2018 at 7:25 AM Emmett Butler <emm...@parsely.com> wrote: > > > Hi users, > > > > What's the biggest topic you've seen in production use? The factors I'm > > interested in are log.retention.* config parameters and number of > > partitions. > > > > We run several 200-partition topics that have log retention set to > roughly > > 1GB per partition. Is this big by common usage standards, or not so big? > > > > I ask because we've had some situations in which ISRs appeared to take an > > inordinately long time to synchronize, and I'm guessing this is partially > > related to the need to shuttle around 1GB hunks of data. > > > > -- > > Emmett Butler | Senior Software Engineer > > < > > > http://www.parsely.com/?utm_source=Signature&utm_medium=emmett-butler&utm_campaign=Signature > > < > http://www.parsely.com/?utm_source=Signature&utm_medium=emmett-butler&utm_campaign=Signature > >> > > > -- Emmett Butler | Senior Software Engineer <http://www.parsely.com/?utm_source=Signature&utm_medium=emmett-butler&utm_campaign=Signature>
