Re: Topics marked for deletion stuck as ineligible for deletion

[Vincent Rischmann]

The broker id wasn't reused no, it's a new id.

Unfortunately we can't afford bringing down the cluster, I'll have to do this 
with the cluster online.

On Mon, Dec 16, 2019, at 17:27, Peter Bukowinski wrote:
> If it was replaced, and a new broker was brought online with the same 
> id, wherever topic partitions had been previously assigned to it should 
> have been recreated.
> 
> At this point, however, I would shut down the cluster, delete the 
> znodes, delete the topic directories from the brokers, then bring the 
> cluster back up.
> 
> -- Peter
> 
> > On Dec 16, 2019, at 3:00 AM, Vincent Rischmann  wrote:
> > 
> > It doesn't exist anymore, we replaced it after a hardware failure.
> > 
> > Thinking about it I don't think I reassigned the partitions for broker 5 to 
> > the new broker before deleting these topics, I didn't realize that it was 
> > necessary for all brokers to be online.
> > 
> > Since broker 5 is never coming back again I'm guessing my only choice is to 
> > manually modify the znodes ? 
> > 
> >> On Fri, Dec 13, 2019, at 19:07, Peter Bukowinski wrote:
> >> If any brokers are offline, kafka can’t successfully delete a topic. 
> >> What’s the state of broker 5?
> >> 
> >> -- Peter (from phone)
> >> 
>  On Dec 13, 2019, at 8:55 AM, Vincent Rischmann  
>  wrote:
> >>> 
> >>> Hi,
> >>> 
> >>> I've deleted a bunch of topics yesterday on our cluster but some are now 
> >>> stuck in "marked for deletion".
> >>> 
> >>> * i've looked in the data directory of every broker and there's no data 
> >>> left for the topics, the directory doesn't exist anymore.
> >>> * in zookeeper the znode `brokers/topics/mytopic` still exists
> >>> * the znode `admin/delete_topics/mytopic` still exists
> >>> 
> >>> I've tried the following to no avail:
> >>> 
> >>> * restarting all brokers
> >>> * removing the `admin/delete_topics/mytopic` node and re-running 
> >>> `kafka-topics.sh --delete --topic mytopic`
> >>> 
> >>> In the kafka-controller.log of some brokers I see this which seems 
> >>> relevant:
> >>> 
> >>>   [2019-12-13 10:15:07,244] WARN [Channel manager on controller 6]: Not 
> >>> sending request (type=StopReplicaRequest, controllerId=6, 
> >>> controllerEpoch=78, deletePartitions=false, partitions=mytopic-17) to 
> >>> broker 5, since it is offline. (kafka.controller.ControllerChannelManager)
> >>>   [2019-12-13 10:15:07,244] WARN [Channel manager on controller 6]: Not 
> >>> sending request (type=StopReplicaRequest, controllerId=6, 
> >>> controllerEpoch=78, deletePartitions=false, partitions=mytopic-24) to 
> >>> broker 5, since it is offline. (kafka.controller.ControllerChannelManager)
> >>> 
> >>> and
> >>> 
> >>>   12061:[2019-12-12 10:35:55,290] INFO [Topic Deletion Manager 1], 
> >>> Handling deletion for topics mytopic 
> >>> (kafka.controller.TopicDeletionManager)
> >>>   12062:[2019-12-12 10:35:55,292] INFO [Topic Deletion Manager 1], Not 
> >>> retrying deletion of topic mytopic at this time since it is marked 
> >>> ineligible for deletion (kafka.controller.TopicDeletionManager)
> >>> 
> >>> Since the data directory is already deleted I'm thinking of simply 
> >>> removing the znode `brokers/topics/mytopic` from zookeeper manually.
> >>> 
> >>> Does anyone has another suggestion ? Is it safe to remove the znode 
> >>> manually ?
> >>> 
> >>> Thanks.
> >> 
>

Re: Health Check

[M. Manna]

Jai,

On Tue, 17 Dec 2019 at 17:33, Jai Nanda Kumar 
wrote:

> Hi,
>
> How to perform a health check on a running Kafka server in AWS EC2
> server.
>

 SHouldn't this be part of your liveness probe? Or, are you trying to do
this adhoc (not how folks do it anyway)?

>
> Thanks and Regards,
> A. Jai Nanda Kumar,
> Senior Software Engineer,
> 9066260799.
> Confidentiality Notice: This e-mail message, including any attachments, is
> for the sole use of the intended recipient(s) and may contain confidential
> and privileged information. Any unauthorized review, use, disclosure or
> distribution is prohibited. If you are not the intended recipient, please
> contact the sender by reply e-mail and destroy all copies of the original
> message.
>

Health Check

[Jai Nanda Kumar]

Hi,

How to perform a health check on a running Kafka server in AWS EC2 server.

Thanks and Regards,
A. Jai Nanda Kumar,
Senior Software Engineer,
9066260799.
Confidentiality Notice: This e-mail message, including any attachments, is for 
the sole use of the intended recipient(s) and may contain confidential and 
privileged information. Any unauthorized review, use, disclosure or 
distribution is prohibited. If you are not the intended recipient, please 
contact the sender by reply e-mail and destroy all copies of the original 
message.

Kafka quotas metrics

[Aleksandar Stankovic]

Greetings!

For some time I've been exploring on how to get information for
producer/consumer bytes per second and throttling per client on specific
Kafka broker. I would like to do so using Kafka MBeans.

Is there any way to actually get a value for example "bytes per second" for
all producers on one Kafka broker using MBeans only? I saw that by enabling
Quotas I can read a value for max permitted bytes for all users (or for
specific one) but I'm not sure if I can do it so for current byte rate for
specific user.

Best regards,

*Aleksandar Stanković*
*Professional Bachelor in Electrical and Computer Engineering*
*Republic of Serbia*
*Skype: alxbxbx*

Re: Which container should you use when deploying on docker ?

[M. Manna]

Robin,

On Tue, 17 Dec 2019 at 01:58, Yu Watanabe  wrote:

> Robin.
>
> Thank you for the reply.
>
> I am about to run kafka on docker in development environment for first time
> and also in production.
> To get started, I searched images in docker hub that has "Official Images"
> tag to find if there is a docker image maintained by the community.
> But there wasn't one so I asked the question.
>
> Thanks,
> Yu Watanabe
>
> On Tue, Dec 17, 2019 at 12:26 AM Robin Moffatt  wrote:
>
> > There are various Kafka images available, including:
> >
> > https://hub.docker.com/r/confluentinc/cp-kafka/
> > 
> > https://hub.docker.com/r/wurstmeister/kafka/
> >
> > I'm not 100% clear what your doubt is? If these are legitimate Kafka
> > images, or something else?
> > 
> >
>

 I guess the confusion is due to fact is Kafka is not as official as, for
example, “Tomcat” on public dockerhub.  When referring to official ASF
image, I would always first go to Apache maintained dockerhub site for
images (not elsewhere). It’s perfectly valid to have CP image (given how
major CP contribution is for Kafka on cloud). But it's still not ASF image
(but licensed under Confluent licence).

If you don’t agree, let’s google “Official Apache Kafka Docker Image” vs
“Official Apache Tomcat docker image" - let's see what the top search
(minus Ads) says.

It certainly wouldn't cause that much of an issue if it's advertised both
on Kafka site and Github main site that CP kafka image is the official go
to. I use CP image for our test K8s cluster, and I would love this to be
clarified in official capacity on Kafka/Github site.

Hope this makes sense.

> >
> > --
> >
> > Robin Moffatt | Senior Developer Advocate | ro...@confluent.io | @rmoff
> >
> >
> > On Sat, 14 Dec 2019 at 04:18, Yu Watanabe  wrote:
> >
> > > Hello.
> > >
> > > I would like to ask question related to kafka on docker engine.
> > > Which container should you use for kafka when deploying on docker in
> > > production ?
> > >
> > > When I look in docker hub , I do not see neither of below tagged for
> > kafka
> > > container .
> > >
> > > Docker certified
> > > Verified publisher
> > > Official Images
> > >
> > > Repository "confluent" seems be the closest one since its the creator
> of
> > > kafka but it does not have above tag .
> > >
> > > Thanks,
> > > Yu Watanabe
> > >
> > > --
> > > Yu Watanabe
> > > Weekend Freelancer who loves to challenge building data platform
> > > yu.w.ten...@gmail.com
> > > [image: LinkedIn icon] 
> > [image:
> > > Twitter icon] 
> > >
> >
>
>
> --
> Yu Watanabe
> Weekend Freelancer who loves to challenge building data platform
> yu.w.ten...@gmail.com
> [image: LinkedIn icon]   [image:
> Twitter icon] 
>