Re: Upgraded from 0.8.2 to 0.9.0 and FetchFollower request time has increased
Hmm over the weekend looks like this slowly crept back up as shown in this graph: http://i.imgur.com/DH9Qa09.png I performed the upgrade on 1/8 around 12am as shown by the spike in the graph. Here's a graph with a little more history going back ~1 month. This was pretty stable prior to the upgrade. http://i.imgur.com/95Gw0MS.png It looks like its entirely "RemoteTime" - http://i.imgur.com/Uamy42q.png Any ideas of what I should check? I have num.replica.fetchers set to 8 currently. Thanks! On Sat, Jan 9, 2016 at 12:19 AM, Ismael Juma wrote: > Great! > On 8 Jan 2016 13:53, "Stephen Powis" wrote: > > > Hey Ismael, > > > > It did, just took a few hours :) It seems like its just hit where it > was > > before the upgrade. > > > > > > > > > > Thanks! > > Stephen > > > > On Fri, Jan 8, 2016 at 8:03 PM, Ismael Juma wrote: > > > > > Hi Stephen, > > > > > > Have things settled down since? > > > > > > Thanks, > > > Ismael > > > > > > On Fri, Jan 8, 2016 at 7:39 AM, Stephen Powis > > > wrote: > > > > > > > Hey! > > > > > > > > So I upgraded our production kafka cluster from 0.8.2 to 0.9.0 this > > > > morning, and it seems like everything went smoothly. > > > > > > > > As the last step when I changed the inter.broker.protocol.version to > > > > 0.9.0.0 and did a rolling restart, I noticed that the FetchFollower > > > request > > > > times increased fairly significantly. Does anyone know if this is > > > expected > > > > with the new protocol version? > > > > > > > > Link to graphite graph: http://i.imgur.com/uhvjCBH.png > > > > > > > > Perhaps I'm just being paranoid and it will settle out? This is the > > same > > > > graph over a larger time period - http://i.imgur.com/FB5Ey2a.png > > > > > > > > Thanks! > > > > > > > > > >
Re: Upgraded from 0.8.2 to 0.9.0 and FetchFollower request time has increased
Great! On 8 Jan 2016 13:53, "Stephen Powis" wrote: > Hey Ismael, > > It did, just took a few hours :) It seems like its just hit where it was > before the upgrade. > > > > > Thanks! > Stephen > > On Fri, Jan 8, 2016 at 8:03 PM, Ismael Juma wrote: > > > Hi Stephen, > > > > Have things settled down since? > > > > Thanks, > > Ismael > > > > On Fri, Jan 8, 2016 at 7:39 AM, Stephen Powis > > wrote: > > > > > Hey! > > > > > > So I upgraded our production kafka cluster from 0.8.2 to 0.9.0 this > > > morning, and it seems like everything went smoothly. > > > > > > As the last step when I changed the inter.broker.protocol.version to > > > 0.9.0.0 and did a rolling restart, I noticed that the FetchFollower > > request > > > times increased fairly significantly. Does anyone know if this is > > expected > > > with the new protocol version? > > > > > > Link to graphite graph: http://i.imgur.com/uhvjCBH.png > > > > > > Perhaps I'm just being paranoid and it will settle out? This is the > same > > > graph over a larger time period - http://i.imgur.com/FB5Ey2a.png > > > > > > Thanks! > > > > > >
Re: Upgraded from 0.8.2 to 0.9.0 and FetchFollower request time has increased
Hey Ismael, It did, just took a few hours :) It seems like its just hit where it was before the upgrade. Thanks! Stephen On Fri, Jan 8, 2016 at 8:03 PM, Ismael Juma wrote: > Hi Stephen, > > Have things settled down since? > > Thanks, > Ismael > > On Fri, Jan 8, 2016 at 7:39 AM, Stephen Powis > wrote: > > > Hey! > > > > So I upgraded our production kafka cluster from 0.8.2 to 0.9.0 this > > morning, and it seems like everything went smoothly. > > > > As the last step when I changed the inter.broker.protocol.version to > > 0.9.0.0 and did a rolling restart, I noticed that the FetchFollower > request > > times increased fairly significantly. Does anyone know if this is > expected > > with the new protocol version? > > > > Link to graphite graph: http://i.imgur.com/uhvjCBH.png > > > > Perhaps I'm just being paranoid and it will settle out? This is the same > > graph over a larger time period - http://i.imgur.com/FB5Ey2a.png > > > > Thanks! > > >
Re: Upgraded from 0.8.2 to 0.9.0 and FetchFollower request time has increased
Hi Stephen, Have things settled down since? Thanks, Ismael On Fri, Jan 8, 2016 at 7:39 AM, Stephen Powis wrote: > Hey! > > So I upgraded our production kafka cluster from 0.8.2 to 0.9.0 this > morning, and it seems like everything went smoothly. > > As the last step when I changed the inter.broker.protocol.version to > 0.9.0.0 and did a rolling restart, I noticed that the FetchFollower request > times increased fairly significantly. Does anyone know if this is expected > with the new protocol version? > > Link to graphite graph: http://i.imgur.com/uhvjCBH.png > > Perhaps I'm just being paranoid and it will settle out? This is the same > graph over a larger time period - http://i.imgur.com/FB5Ey2a.png > > Thanks! >
Upgraded from 0.8.2 to 0.9.0 and FetchFollower request time has increased
Hey! So I upgraded our production kafka cluster from 0.8.2 to 0.9.0 this morning, and it seems like everything went smoothly. As the last step when I changed the inter.broker.protocol.version to 0.9.0.0 and did a rolling restart, I noticed that the FetchFollower request times increased fairly significantly. Does anyone know if this is expected with the new protocol version? Link to graphite graph: http://i.imgur.com/uhvjCBH.png Perhaps I'm just being paranoid and it will settle out? This is the same graph over a larger time period - http://i.imgur.com/FB5Ey2a.png Thanks!