Disadvantages of Upgrading Kafka server without upgrading client libraries?
Hi Everyone, I have an install of Kafka 0.8.2.1 which I'm upgrading to 0.10.1.0. I see that Kafka 0.10.1.0 should be backwards compatible with client libraries written for older versions but that newer client libraries are only compatible with their version and up. My question is what disadvantages would there be to never upgrading the clients? I'm mainly asking because it would be advantageous to save some time here with a little technical debt if the costs weren't too high. If there are major issues then I can take on the client upgrade as well. Thanks in advance! -- In Christ, Timmy V. http://blog.twonegatives.com/ http://five.sentenc.es/ -- Spend less time on mail
log.dirs balance?
Hello, My kafka deploy has 5 servers with 3 log disks each. Over the weekend I noticed that on 2 of the 5 servers the partitions appear to be imbalanced amongst the log.dirs. ``` kafka3 /var/lib/kafka/disk1 3 /var/lib/kafka/disk2 3 /var/lib/kafka/disk3 3 kafka5 /var/lib/kafka/disk1 3 /var/lib/kafka/disk2 4 /var/lib/kafka/disk3 2 kafka1 /var/lib/kafka/disk1 3 /var/lib/kafka/disk2 3 /var/lib/kafka/disk3 3 kafka4 /var/lib/kafka/disk1 4 /var/lib/kafka/disk2 2 /var/lib/kafka/disk3 3 kafka2 /var/lib/kafka/disk1 3 /var/lib/kafka/disk2 3 /var/lib/kafka/disk3 3 ``` You can see that 5 and 4 are both unbalanced. Is there a reason for that? The partitions themselves are pretty much perfectly balanced, but the directory chosen for them is not. Is this an anti-pattern to be using multiple log.dirs per server? Thanks in advance! -- In Christ, Timmy V. http://blog.twonegatives.com/ http://five.sentenc.es/ -- Spend less time on mail
Re: log.dirs balance?
Thanks for the report, Karolis.
I have a potential theory for how this happened and I'm wondering if it's
possibly valid:
I have 9 partitions on a machine with 3 disks and they get assigned exactly
as you'd expect:
d1: t{1,2,3}
d2: t{4,5,6}
d3: t{7,8,9}
Then, a disk fails or something somewhere else and kafka decides to assign
t10 to d1 on this machine.
d1: t{1,2,3,10}
d2: t{4,5,6}
d3: t{7,8,9}
Then something happens where kafka wants to move a partition off of d2:
d1: t{1,2,3,10}
d2: t{4,5}
d3: t{7,8,9}
Is this scenario something that can happen?
In our actual deployment we have 5 servers with 3 disks each, 1 topic, 15
partitions, and a replication factor of 3.
On Tue, Nov 29, 2016 at 4:04 PM, Karolis Pocius
wrote:
> It's difficult enough to balance kafka brokers with a single log
> directory, not to mention attempting to juggle multiple ones. While JBOD is
> great in terms of capacity, it's a pain in terms of management. After 6
> months of constant manual reassignments I ended up going with RAID1+0 which
> is what LinkedIn uses as well as Confluent recommends.
>
> Hats off to you if you manage to find a solution to this, just wanted to
> share my painful experience.
>
>
>
> On 2016.11.29 21:35, Tim Visher wrote:
>
>> Hello,
>>
>> My kafka deploy has 5 servers with 3 log disks each. Over the weekend I
>> noticed that on 2 of the 5 servers the partitions appear to be imbalanced
>> amongst the log.dirs.
>>
>> ```
>> kafka3
>> /var/lib/kafka/disk1
>> 3
>> /var/lib/kafka/disk2
>> 3
>> /var/lib/kafka/disk3
>> 3
>> kafka5
>> /var/lib/kafka/disk1
>> 3
>> /var/lib/kafka/disk2
>> 4
>> /var/lib/kafka/disk3
>> 2
>> kafka1
>> /var/lib/kafka/disk1
>> 3
>> /var/lib/kafka/disk2
>> 3
>> /var/lib/kafka/disk3
>> 3
>> kafka4
>> /var/lib/kafka/disk1
>> 4
>> /var/lib/kafka/disk2
>> 2
>> /var/lib/kafka/disk3
>> 3
>> kafka2
>> /var/lib/kafka/disk1
>> 3
>> /var/lib/kafka/disk2
>> 3
>> /var/lib/kafka/disk3
>> 3
>> ```
>>
>> You can see that 5 and 4 are both unbalanced.
>>
>> Is there a reason for that? The partitions themselves are pretty much
>> perfectly balanced, but the directory chosen for them is not.
>>
>> Is this an anti-pattern to be using multiple log.dirs per server?
>>
>> Thanks in advance!
>>
>> --
>>
>> In Christ,
>>
>> Timmy V.
>>
>> http://blog.twonegatives.com/
>> http://five.sentenc.es/ -- Spend less time on mail
>>
>>
>
> Best Regards
>
> Karolis Pocius
> IT System Engineer
>
> Email: karolis.poc...@adform.com
> Mobile: +370 620 22108
> Sporto g. 18, LT-09238 Vilnius, Lithuania
>
> Disclaimer: The information contained in this message and attachments is
> intended solely for the attention and use of the named addressee and may be
> confidential. If you are not the intended recipient, you are reminded that
> the information remains the property of the sender. You must not use,
> disclose, distribute, copy, print or rely on this e-mail. If you have
> received this message in error, please contact the sender immediately and
> irrevocably delete this message and any copies.
Deleting a topic without delete.topic.enable=true?
Hi Everyone, I'm really confused at the moment. We created a topic with brokers set to delete.topic.enable=false. We now need to delete that topic. To do that we shut down all the brokers, deleted everything under log.dirs and logs.dir on all the kafka brokers, `rmr`ed the entire chroot that kafka was storing things under in zookeeper, and then brought kafka back up. After doing all that, the topic comes back, every time. What can we do to delete that topic? -- In Christ, Timmy V. http://blog.twonegatives.com/ http://five.sentenc.es/ -- Spend less time on mail
Re: Deleting a topic without delete.topic.enable=true?
I did all of that because setting delete.topic.enable=true wasn't effective. We set that across every broker, restarted them, and then deleted the topic, and it was still stuck in existence. On Fri, Dec 9, 2016 at 11:11 AM, Ali Akhtar wrote: > You need to also delete / restart zookeeper, its probably storing the > topics there. (Or yeah, just enable it and then delete the topic) > > On Fri, Dec 9, 2016 at 9:09 PM, Rodrigo Sandoval < > rodrigo.madfe...@gmail.com > > wrote: > > > Why did you do all those things instead of just setting > > delete.topic.enable=true? > > > > On Dec 9, 2016 13:40, "Tim Visher" wrote: > > > > > Hi Everyone, > > > > > > I'm really confused at the moment. We created a topic with brokers set > to > > > delete.topic.enable=false. > > > > > > We now need to delete that topic. To do that we shut down all the > > brokers, > > > deleted everything under log.dirs and logs.dir on all the kafka > brokers, > > > `rmr`ed the entire chroot that kafka was storing things under in > > zookeeper, > > > and then brought kafka back up. > > > > > > After doing all that, the topic comes back, every time. > > > > > > What can we do to delete that topic? > > > > > > -- > > > > > > In Christ, > > > > > > Timmy V. > > > > > > http://blog.twonegatives.com/ > > > http://five.sentenc.es/ -- Spend less time on mail > > > > > >
Re: Deleting a topic without delete.topic.enable=true?
On Fri, Dec 9, 2016 at 11:34 AM, Todd Palino wrote: > Given that you said you removed the log directories, and provided that when > you did the rmr on Zookeeper it was to the “/brokers/topics/(topic name)” > path, you did the right things for a manual deletion. It sounds like you > may have a consumer (or other client) that is recreating the topic. Do you > have auto topic creation enabled? > That was the last epiphany we had. We had shut down the producer but not all the consumers and we do allow auto-topic creation. That said, we then proceeded to shut all of them down (the consumers) and the topic came back. I'm glad that we were doing the right steps though. > > -Todd > > > On Fri, Dec 9, 2016 at 8:25 AM, Tim Visher wrote: > > > I did all of that because setting delete.topic.enable=true wasn't > > effective. We set that across every broker, restarted them, and then > > deleted the topic, and it was still stuck in existence. > > > > On Fri, Dec 9, 2016 at 11:11 AM, Ali Akhtar > wrote: > > > > > You need to also delete / restart zookeeper, its probably storing the > > > topics there. (Or yeah, just enable it and then delete the topic) > > > > > > On Fri, Dec 9, 2016 at 9:09 PM, Rodrigo Sandoval < > > > rodrigo.madfe...@gmail.com > > > > wrote: > > > > > > > Why did you do all those things instead of just setting > > > > delete.topic.enable=true? > > > > > > > > On Dec 9, 2016 13:40, "Tim Visher" wrote: > > > > > > > > > Hi Everyone, > > > > > > > > > > I'm really confused at the moment. We created a topic with brokers > > set > > > to > > > > > delete.topic.enable=false. > > > > > > > > > > We now need to delete that topic. To do that we shut down all the > > > > brokers, > > > > > deleted everything under log.dirs and logs.dir on all the kafka > > > brokers, > > > > > `rmr`ed the entire chroot that kafka was storing things under in > > > > zookeeper, > > > > > and then brought kafka back up. > > > > > > > > > > After doing all that, the topic comes back, every time. > > > > > > > > > > What can we do to delete that topic? > > > > > > > > > > -- > > > > > > > > > > In Christ, > > > > > > > > > > Timmy V. > > > > > > > > > > http://blog.twonegatives.com/ > > > > > http://five.sentenc.es/ -- Spend less time on mail > > > > > > > > > > > > > > > > > > -- > *Todd Palino* > Staff Site Reliability Engineer > Data Infrastructure Streaming > > > > linkedin.com/in/toddpalino >
Re: Deleting a topic without delete.topic.enable=true?
I wonder if datadog monitoring triggers that behavior. That's the only other piece of our infrastructure that may have been talking to that topic. On Mon, Dec 12, 2016 at 12:40 AM, Surendra , Manchikanti < surendra.manchika...@gmail.com> wrote: > If "auto.create.topics.enable" is set to true in your configurations , any > producer/consumer or fetch request will create the topic again. Set it to > false and delete the topic. > > -- Surendra Manchikanti > > On Sat, Dec 10, 2016 at 10:59 AM, Todd Palino wrote: > > > Are you running something else besides the consumers that would maintain > a > > memory of the topics and potentially recreate them by issuing a metadata > > request? For example, Burrow (the consumer monitoring app I wrote) does > > this because it maintains a list of all topics in memory, and will end up > > recreating a topic that has been deleted as it issues a metadata request > to > > try and find out what happened after an offset request for the topic > fails. > > > > -Todd > > > > > > On Fri, Dec 9, 2016 at 8:37 AM, Tim Visher wrote: > > > > > On Fri, Dec 9, 2016 at 11:34 AM, Todd Palino > wrote: > > > > > > > Given that you said you removed the log directories, and provided > that > > > when > > > > you did the rmr on Zookeeper it was to the “/brokers/topics/(topic > > name)” > > > > path, you did the right things for a manual deletion. It sounds like > > you > > > > may have a consumer (or other client) that is recreating the topic. > Do > > > you > > > > have auto topic creation enabled? > > > > > > > > > > That was the last epiphany we had. We had shut down the producer but > not > > > all the consumers and we do allow auto-topic creation. > > > > > > That said, we then proceeded to shut all of them down (the consumers) > and > > > the topic came back. I'm glad that we were doing the right steps > though. > > > > > > > > > > > -Todd > > > > > > > > > > > > On Fri, Dec 9, 2016 at 8:25 AM, Tim Visher > > wrote: > > > > > > > > > I did all of that because setting delete.topic.enable=true wasn't > > > > > effective. We set that across every broker, restarted them, and > then > > > > > deleted the topic, and it was still stuck in existence. > > > > > > > > > > On Fri, Dec 9, 2016 at 11:11 AM, Ali Akhtar > > > > wrote: > > > > > > > > > > > You need to also delete / restart zookeeper, its probably storing > > the > > > > > > topics there. (Or yeah, just enable it and then delete the topic) > > > > > > > > > > > > On Fri, Dec 9, 2016 at 9:09 PM, Rodrigo Sandoval < > > > > > > rodrigo.madfe...@gmail.com > > > > > > > wrote: > > > > > > > > > > > > > Why did you do all those things instead of just setting > > > > > > > delete.topic.enable=true? > > > > > > > > > > > > > > On Dec 9, 2016 13:40, "Tim Visher" > wrote: > > > > > > > > > > > > > > > Hi Everyone, > > > > > > > > > > > > > > > > I'm really confused at the moment. We created a topic with > > > brokers > > > > > set > > > > > > to > > > > > > > > delete.topic.enable=false. > > > > > > > > > > > > > > > > We now need to delete that topic. To do that we shut down all > > the > > > > > > > brokers, > > > > > > > > deleted everything under log.dirs and logs.dir on all the > kafka > > > > > > brokers, > > > > > > > > `rmr`ed the entire chroot that kafka was storing things under > > in > > > > > > > zookeeper, > > > > > > > > and then brought kafka back up. > > > > > > > > > > > > > > > > After doing all that, the topic comes back, every time. > > > > > > > > > > > > > > > > What can we do to delete that topic? > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > > In Christ, > > > > > > > > > > > > > > > > Timmy V. > > > > > > > > > > > > > > > > http://blog.twonegatives.com/ > > > > > > > > http://five.sentenc.es/ -- Spend less time on mail > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > *Todd Palino* > > > > Staff Site Reliability Engineer > > > > Data Infrastructure Streaming > > > > > > > > > > > > > > > > linkedin.com/in/toddpalino > > > > > > > > > > > > > > > -- > > *Todd Palino* > > Staff Site Reliability Engineer > > Data Infrastructure Streaming > > > > > > > > linkedin.com/in/toddpalino > > >
How to enable `kafka.(producer|consumer)` JMX metrics?
Hello Everyone, I'm trying to get my datadog kafka integration working as expected and I noticed that the documented `kafka.(producer|consumer)` JMX MBeans do not appear to be available. I verified this by using JConsole to browse the MBeans. Sure enough, kafka.server, kafka.log, kafka.network etc. are all there but none are there for kafka.producer or kafka.consumer. Where are those metrics supposed to come from? Thanks in advance! -- In Christ, Timmy V. https://blog.twonegatives.com https://five.sentenc.es
Re: How to enable `kafka.(producer|consumer)` JMX metrics?
:facepalm: They are in the producer and consumer JVMs, as would be totally expected. I'm only collecting JMX metrics from the brokers. Duh. Sorry for the noise. On Wed, Jan 3, 2018 at 12:01 PM, Tim Visher wrote: > Hello Everyone, > > I'm trying to get my datadog kafka integration working as expected and I > noticed that the documented `kafka.(producer|consumer)` JMX MBeans do not > appear to be available. > > I verified this by using JConsole to browse the MBeans. Sure enough, > kafka.server, kafka.log, kafka.network etc. are all there but none are > there for kafka.producer or kafka.consumer. > > Where are those metrics supposed to come from? > > Thanks in advance! > > -- > > In Christ, > > Timmy V. > > https://blog.twonegatives.com > https://five.sentenc.es > >
Re: Any suggestion about Kafka/Zookeeper migration.
On Wed, Jan 3, 2018 at 11:39 PM, Tony Liu wrote: > This post here is aimed to ask experience about what did you do migration > `Kafka/zookeeper` ? :) > > I think the easiest way to migrate kafka is to do a blue/green deploy. Stand up an entire new cluster, cut over producers to it, wait for the old cluster to drain, cut over consumers, then shut down the old cluster. Especially in AWS or any other virtualized environment this should be very easy assuming that you don't need to deploy many things to do the cutover. DNS or some other load balancing solution could help but you may be in too deep to put that in place now. As far as migrating zookeeper, you could stand up a new zookeeper cluster when spinning up your new kafka cluster and do the same thing. My team is currently doing exactly that right now. -- In Christ, Timmy V. https://blog.twonegatives.com https://five.sentenc.es
Re: Any suggestion about Kafka/Zookeeper migration.
On Thu, Jan 4, 2018 at 11:55 AM, Tony Liu wrote: > Thanks Timmy. > > I appreciate your suggestion, but for us, it's hard to do that , the > problem are : > >- 6+ Kafka cluster. >- more than 200 micro services or applications are talking with Kafka >now. > > Any other suggestion for our case ? > I've never done it but the standard migration guide should get you an upgrade in place. It's just a terribly tedious sounding process. https://kafka.apache.org/documentation/#upgrade With that kind of topology I'd probably recommend getting something in front of kafka that can do a cut over for you like HAProxy or DNS. Obviously that's a future consideration though. For now, definitely go with the in place migration. > > > On Thu, Jan 4, 2018 at 6:06 AM, Tim Visher wrote: > > > On Wed, Jan 3, 2018 at 11:39 PM, Tony Liu > wrote: > > > > > This post here is aimed to ask experience about what did you do > migration > > > `Kafka/zookeeper` ? :) > > > > > > > > I think the easiest way to migrate kafka is to do a blue/green deploy. > > Stand up an entire new cluster, cut over producers to it, wait for the > old > > cluster to drain, cut over consumers, then shut down the old cluster. > > > > Especially in AWS or any other virtualized environment this should be > very > > easy assuming that you don't need to deploy many things to do the > cutover. > > DNS or some other load balancing solution could help but you may be in > too > > deep to put that in place now. > > > > As far as migrating zookeeper, you could stand up a new zookeeper cluster > > when spinning up your new kafka cluster and do the same thing. > > > > My team is currently doing exactly that right now. > > > > -- > > > > In Christ, > > > > Timmy V. > > > > https://blog.twonegatives.com > > https://five.sentenc.es > > >
Re: Upgrading Kafka from Version 0.10.2 to 1.0.0
On Tue, Jan 9, 2018 at 4:50 PM, ZigSphere Tech wrote: > Is it easy to upgrade from Kafka version 0.10.2 to 1.0.0 or do I need to > upgrade to version 0.11.0 first? Anything to expect? > We just did (almost) exactly this upgrade. 2.11-0.10.1.0 to 2.11-1.0.0. The main issue we faced was the broker memory leak that was discussed several times on this list. https://lists.apache.org/thread.html/a1d6ea46d29d8a4e4e7aaee57b09a3c7de44e911efd2ddbe3ab11cf5@%3Cusers.kafka.apache.org%3E We ended up having to upgrade all the things to mitigate. I believe the official recommendation is to wait till 1.0.1. -- In Christ, Timmy V. https://blog.twonegatives.com https://five.sentenc.es
