Re: Message batch & compression doesn't work as expected

[Frank Zhou]

Hi,

I think the root cause of this is transaction. The producer we are using
has enabled transaction, and batching then is not working as expected. Once
we turn it off, the situation is back to normal, I can see batching is
performed, and everything looks fine. Is transaction not working well with
batching?

On Fri, Dec 13, 2019 at 10:09 AM Frank Zhou  wrote:

> Hi,
>
> Definitely will check this out, thanks. We just started tuning recently,
> and we are quite new to kafka world. The problem we are facing is with
> batch.size and linger.ms both set, the patch sent out doesn't seem to
> meet either one condition. We tried with more "reasonable" values as well,
> it just doesn't seem to work, that's why we tested with more extreme case.
> And we are using 2.2.2 kafka server with 2.3.1 kafka client, not sure if
> there will be any compatibility issue in between. Also we are using
> transactions as same message will be pushed to multiple topics in our case.
>
> On Thu, Dec 12, 2019 at 9:10 PM M. Manna  wrote:
>
>> Frank,
>>
>> On Thu, 12 Dec 2019 at 11:28, Frank Zhou  wrote:
>>
>>> Hi,
>>>
>>> I am testing kafka client on message batch and compression. I have
>>> enabled message batching along with compression, with batch.size set to 3M,
>>> linger.ms set to 5000ms and compression.type set to gzip(Attached whole
>>> producer config). Then testing with wireshark to check the details.
>>> First issue our team notice is that compression codec seems to have some
>>> issue. Since we set it as gzip, but we notice in wireshark, it will display
>>> as other compression codec, like Snappy in attached screenshot(Not sure if
>>> this is wireshark's issue, or a real issue on Kafka, but the whole packet
>>> details display seems fine in wireshark).
>>> Second issue is that we have set the latency and batch number so high,
>>> but it still send the Produce request to server much more frequently than
>>> we expected. Size per message that is sending before batch & compression
>>> should be around 200 bytes, and during testing, all the message generated
>>> by us should be around 200KB, so we are expecting much less packets
>>> transferred than this(screenshot only shows small amount of them, total
>>> number is 1472).
>>> [image: 2019-12-12_19h00_15.png]
>>>  Is it we miss some config or the config is not correct leading to this?
>>>
>>>
>>>
>> I recently tuned our GCP based test cluster using batch of 800K, no
>> compression, and no linger.ms. We got the desired consistency and
>> desired throughput. But we used 2.3.0 version, and I don't suppose it
>> matters much at that point.
>>
>> https://www.youtube.com/watch?v=oQe7PpDDdzA
>>
>> The above shows a very good and detailed analysis done by Becket
>> regarding throughput calculation and estimation. Have you checked this to
>> see what matches your scenario?
>> The sending is impacted by linger.ms too, not just batch.size. So tuning
>> them together is a bit tricky. Perhaps you want to see which one you need
>> more.
>>
>>
>>
>>
>>> --
>>> *Frank Zhou*
>>> R&D, Itiviti
>>> Java Developer
>>> D +852 2521 7480
>>> frank.z...@itiviti.com
>>>
>>> __
>>>
>>> itiviti.com 
>>>
>>> *The information contained in or attached to this email is strictly
>>> confidential. If you are not the intended recipient, please notify us
>>> immediately by telephone and return the message to us.*
>>>
>>> *Email communications by definition contain personal information. The
>>> Itiviti group of companies is subject to European data protection
>>> regulations. Itiviti’s Privacy Notice is available at www.itiviti.com
>>> . Itiviti expects the recipient of this email to
>>> be compliant with Itiviti’s Privacy Notice and applicable regulations.
>>> Please advise us immediately at dataprotectiont...@itiviti.com if you are
>>> not compliant with these.*
>>>
>>> __
>>>
>>> itiviti.com 
>>>  Follow Itiviti on Linkedin
>>> 
>>>
>>> The information contained in or attached to this email is strictly
>>> confidential. If you are not the intended recipient, please notify us
>>> immediately by telephone and return the message to us. Email communications
>>> by definition contain personal information. The Itiviti group of companies
>>> is subject to European data protection regulations.
>>>
>>> Itiviti’s Privacy Notice is available at www.itiviti.com. Itiviti
>>> expects the recipient of this email to be compliant with Itiviti’s Privacy
>>> Notice and applicable regulations. Please advise us immediately at
>>> dataprotectiont...@itiviti.com if you are not compliant with these.
>>>
>>
>
> --
> *Frank Zhou*
> R&D, Itiviti
> Java Developer
> D +852 2521 7480
> frank.z...@itiviti.com
>
> __
>
> itiviti.com 
>
> *The information contained in or attac

Re: Which container should you use when deploying on docker ?

[Yu Watanabe]

Robin.

Thank you for the reply.

I am about to run kafka on docker in development environment for first time
and also in production.
To get started, I searched images in docker hub that has "Official Images"
tag to find if there is a docker image maintained by the community.
But there wasn't one so I asked the question.

Thanks,
Yu Watanabe

On Tue, Dec 17, 2019 at 12:26 AM Robin Moffatt  wrote:

> There are various Kafka images available, including:
>
> https://hub.docker.com/r/confluentinc/cp-kafka/
> 
> https://hub.docker.com/r/wurstmeister/kafka/
>
> I'm not 100% clear what your doubt is? If these are legitimate Kafka
> images, or something else?
> 
>
>
> --
>
> Robin Moffatt | Senior Developer Advocate | ro...@confluent.io | @rmoff
>
>
> On Sat, 14 Dec 2019 at 04:18, Yu Watanabe  wrote:
>
> > Hello.
> >
> > I would like to ask question related to kafka on docker engine.
> > Which container should you use for kafka when deploying on docker in
> > production ?
> >
> > When I look in docker hub , I do not see neither of below tagged for
> kafka
> > container .
> >
> > Docker certified
> > Verified publisher
> > Official Images
> >
> > Repository "confluent" seems be the closest one since its the creator of
> > kafka but it does not have above tag .
> >
> > Thanks,
> > Yu Watanabe
> >
> > --
> > Yu Watanabe
> > Weekend Freelancer who loves to challenge building data platform
> > yu.w.ten...@gmail.com
> > [image: LinkedIn icon] 
> [image:
> > Twitter icon] 
> >
>


-- 
Yu Watanabe
Weekend Freelancer who loves to challenge building data platform
yu.w.ten...@gmail.com
[image: LinkedIn icon]   [image:
Twitter icon] 

Re: [ANNOUNCE] Apache Kafka 2.4.0

[Manikumar]

Thanks to all who contributed to the Apache blog post about 2.4.  It's live
now at https://blogs.apache.org/kafka/date/20191216

On Mon, Dec 16, 2019 at 10:30 PM Manikumar  wrote:

> The Apache Kafka community is pleased to announce the release for Apache
> Kafka 2.4.0
>
> This release includes many new features, including:
> - Allow consumers to fetch from closest replica
> - Support for incremental cooperative rebalancing to the consumer
> rebalance protocol
> - MirrorMaker 2.0 (MM2), a new multi-cluster, cross-datacenter replication
> engine
> - New Java authorizer Interface
> - Support for non-key joining in KTable
> - Administrative API for replica reassignment
> - Securing Internal connect REST endpoints
> - API to delete consumer offsets and expose it via the AdminClient.
>
> All of the changes in this release can be found in the release notes:
> https://www.apache.org/dist/kafka/2.4.0/RELEASE_NOTES.html
>
> You can download the source and binary release (Scala 2.11, 2.12 and 2.13)
> from:
> https://kafka.apache.org/downloads#2.4.0
>
>
> ---
>
> Apache Kafka is a distributed streaming platform with four core APIs:
>
> ** The Producer API allows an application to publish a stream records to
> one or more Kafka topics.
>
> ** The Consumer API allows an application to subscribe to one or more
> topics and process the stream of records produced to them.
>
> ** The Streams API allows an application to act as a stream processor,
> consuming an input stream from one or more topics and producing an
> output stream to one or more output topics, effectively transforming the
> input streams to output streams.
>
> ** The Connector API allows building and running reusable producers or
> consumers that connect Kafka topics to existing applications or data
> systems. For example, a connector to a relational database might
> capture every change to a table.
>
> With these APIs, Kafka can be used for two broad classes of application:
>
> ** Building real-time streaming data pipelines that reliably get data
> between systems or applications.
>
> ** Building real-time streaming applications that transform or react
> to the streams of data.
>
> Apache Kafka is in use at large and small companies worldwide, including
> Capital One, Goldman Sachs, ING, LinkedIn, Netflix, Pinterest, Rabobank,
> Target, The New York Times, Uber, Yelp, and Zalando, among others.
>
> A big thank you for the following 121 contributors to this release!
>
> A. Sophie Blee-Goldman, Adam Bellemare, Alex Diachenko, Alex Leung, Alex
> Mironov, Alice, Almog Gavra, Anastasia Vela, anatasiavela, Andy Coates,
> Antony Stubbs, Arjun Satish, Arlo Louis O'Keeffe, Arvind Thirunarayanan,
> asutosh936, Bill Bejeck, Bob Barrett, Boyang Chen, Brian Bushree, Bruno
> Cadonna, cadonna, Carlos Manuel Duclos Vergara, Cheng Pan, Chia-Ping Tsai,
> Chris Egerton, Chris Pettitt, Chris Stromberger, Colin Hicks, Colin P.
> Mccabe, Colin Patrick McCabe, cpettitt-confluent, cwildman, Cyrus Vafadari,
> David Arthur, David Jacot, Dejan Stojadinović, Dhruvil Shah, Florian
> Hussonnois, Gardner Vickers, Gemma Singleton, Grant Henke, Greg Harris,
> Gunnar Morling, Guozhang Wang, Gwen Shapira, Hai-Dang Dam, highluck, huxi,
> Igor Soarez, Ismael Juma, James Cheng, Jason Gustafson, Jeff Huang, Joel
> Hamill, John Roesler, jolshan, José Armando García Sancio, Jukka Karvanen,
> Justine Olshan, Kamal Chandraprakash, Karan Kumar, Kengo Seki, Kevin Lu,
> khairy, Konstantine Karantasis, Lee Dongjin, Lifei Chen, Lucas Bradstreet,
> LuyingLiu, Magesh Nandakumar, Manikumar Reddy, Matthias J. Sax, Michał
> Borowiecki, Michał Siatkowski, Mickael Maison, mjarvie, mmanna-sapfgl,
> Nacho Muñoz Gómez, Nathan Murthy, Nigel Liang, NIkhil Bhatia, Nikolay, Omar
> Al-Safi, Omkar Mestry, Paul, pkleindl, Rajan Chauhan, Rajini Sivaram,
> Randall Hauch, Richard Yu, Robert Yokota, Rohan, Ron Dagostino, Ryanne
> Dolan, saisandeep, Scott Hendricks, sdreynolds, Sean Glover, Sergey
> Prokofiev, slim, soondenana, Stanislav Kozlovski, Stanislav Vodetskyi,
> SuryaTeja Duggi, tadsul, teebee, Tirtha Chatterjee, Tu Tran, Tu V. Tran,
> Vahid Hashemian, Victoria Bialas, Vikas Singh, Viktor Somogyi, Viktor
> Somogyi-Vass, vinoth chandar, wenhoujx, Wennn, Will James, wineandcheeze,
> Yaroslav Tkachenko, 康智冬
>
> We welcome your help and feedback. For more information on how to
> report problems, and to get involved, visit the project website at
> https://kafka.apache.org/
>
> Thank you!
>
> Regards,
> Manikumar
>

Re: Missing link in online document

[Guozhang Wang]

Hello Yu,

Thanks for the reference, and I can confirm it is indeed broken. Will try
to fix the link.


Guozhang


On Sun, Dec 15, 2019 at 3:45 PM Yu Watanabe  wrote:

> Hello Guozhang.
>
> Thank you for the reply.
> The link was in this page.
>
>
> https://kafka.apache.org/23/documentation/streams/developer-guide/running-app.html
>
> Below ** section has the corresponding link.
>
> >For more information about how you can package your application in this
> way, see the ** Streams code examples **.
> >
> >When you start your application you are launching a Kafka Streams instance
> of your application. You can run multiple instances of your application.
> >A common scenario is that there are multiple instances of your application
> running in parallel. For more information, see Parallelism Model.
>
> On Mon, Dec 16, 2019 at 3:16 AM Guozhang Wang  wrote:
>
> > Hello Yu,
> >
> > Could you point to me which page has the reference to this link?
> >
> >
> > Guozhang
> >
> > On Sun, Dec 15, 2019 at 2:24 AM Yu Watanabe 
> wrote:
> >
> > > Hello.
> > >
> > > I was walking through kafka streams document  and below link seems to
> be
> > > invalid. It returns below page.
> > >
> > > > Not Found
> > > > The requested URL was not found on this server.
> > >
> > >
> > >
> >
> https://kafka.apache.org/23/documentation/streams/code-examples.html#streams-code-examples
> > >
> > > May I ask for correct link for kafka stream example ?
> > >
> > > Thanks,
> > > Yu Watanabe
> > >
> > > --
> > > Yu Watanabe
> > > Weekend Freelancer who loves to challenge building data platform
> > > yu.w.ten...@gmail.com
> > > [image: LinkedIn icon] 
> > [image:
> > > Twitter icon] 
> > >
> >
> >
> > --
> > -- Guozhang
> >
>
>
> --
> Yu Watanabe
> Weekend Freelancer who loves to challenge building data platform
> yu.w.ten...@gmail.com
> [image: LinkedIn icon]   [image:
> Twitter icon] 
>


-- 
-- Guozhang

[ANNOUNCE] Apache Kafka 2.4.0

[Manikumar]

The Apache Kafka community is pleased to announce the release for Apache
Kafka 2.4.0

This release includes many new features, including:
- Allow consumers to fetch from closest replica
- Support for incremental cooperative rebalancing to the consumer rebalance
protocol
- MirrorMaker 2.0 (MM2), a new multi-cluster, cross-datacenter replication
engine
- New Java authorizer Interface
- Support for non-key joining in KTable
- Administrative API for replica reassignment
- Securing Internal connect REST endpoints
- API to delete consumer offsets and expose it via the AdminClient.

All of the changes in this release can be found in the release notes:
https://www.apache.org/dist/kafka/2.4.0/RELEASE_NOTES.html

You can download the source and binary release (Scala 2.11, 2.12 and 2.13)
from:
https://kafka.apache.org/downloads#2.4.0

---

Apache Kafka is a distributed streaming platform with four core APIs:

** The Producer API allows an application to publish a stream records to
one or more Kafka topics.

** The Consumer API allows an application to subscribe to one or more
topics and process the stream of records produced to them.

** The Streams API allows an application to act as a stream processor,
consuming an input stream from one or more topics and producing an
output stream to one or more output topics, effectively transforming the
input streams to output streams.

** The Connector API allows building and running reusable producers or
consumers that connect Kafka topics to existing applications or data
systems. For example, a connector to a relational database might
capture every change to a table.

With these APIs, Kafka can be used for two broad classes of application:

** Building real-time streaming data pipelines that reliably get data
between systems or applications.

** Building real-time streaming applications that transform or react
to the streams of data.

Apache Kafka is in use at large and small companies worldwide, including
Capital One, Goldman Sachs, ING, LinkedIn, Netflix, Pinterest, Rabobank,
Target, The New York Times, Uber, Yelp, and Zalando, among others.

A big thank you for the following 121 contributors to this release!

A. Sophie Blee-Goldman, Adam Bellemare, Alex Diachenko, Alex Leung, Alex
Mironov, Alice, Almog Gavra, Anastasia Vela, anatasiavela, Andy Coates,
Antony Stubbs, Arjun Satish, Arlo Louis O'Keeffe, Arvind Thirunarayanan,
asutosh936, Bill Bejeck, Bob Barrett, Boyang Chen, Brian Bushree, Bruno
Cadonna, cadonna, Carlos Manuel Duclos Vergara, Cheng Pan, Chia-Ping Tsai,
Chris Egerton, Chris Pettitt, Chris Stromberger, Colin Hicks, Colin P.
Mccabe, Colin Patrick McCabe, cpettitt-confluent, cwildman, Cyrus Vafadari,
David Arthur, David Jacot, Dejan Stojadinović, Dhruvil Shah, Florian
Hussonnois, Gardner Vickers, Gemma Singleton, Grant Henke, Greg Harris,
Gunnar Morling, Guozhang Wang, Gwen Shapira, Hai-Dang Dam, highluck, huxi,
Igor Soarez, Ismael Juma, James Cheng, Jason Gustafson, Jeff Huang, Joel
Hamill, John Roesler, jolshan, José Armando García Sancio, Jukka Karvanen,
Justine Olshan, Kamal Chandraprakash, Karan Kumar, Kengo Seki, Kevin Lu,
khairy, Konstantine Karantasis, Lee Dongjin, Lifei Chen, Lucas Bradstreet,
LuyingLiu, Magesh Nandakumar, Manikumar Reddy, Matthias J. Sax, Michał
Borowiecki, Michał Siatkowski, Mickael Maison, mjarvie, mmanna-sapfgl,
Nacho Muñoz Gómez, Nathan Murthy, Nigel Liang, NIkhil Bhatia, Nikolay, Omar
Al-Safi, Omkar Mestry, Paul, pkleindl, Rajan Chauhan, Rajini Sivaram,
Randall Hauch, Richard Yu, Robert Yokota, Rohan, Ron Dagostino, Ryanne
Dolan, saisandeep, Scott Hendricks, sdreynolds, Sean Glover, Sergey
Prokofiev, slim, soondenana, Stanislav Kozlovski, Stanislav Vodetskyi,
SuryaTeja Duggi, tadsul, teebee, Tirtha Chatterjee, Tu Tran, Tu V. Tran,
Vahid Hashemian, Victoria Bialas, Vikas Singh, Viktor Somogyi, Viktor
Somogyi-Vass, vinoth chandar, wenhoujx, Wennn, Will James, wineandcheeze,
Yaroslav Tkachenko, 康智冬

We welcome your help and feedback. For more information on how to
report problems, and to get involved, visit the project website at
https://kafka.apache.org/

Thank you!

Regards,
Manikumar

Re: Topics marked for deletion stuck as ineligible for deletion

[Peter Bukowinski]

If it was replaced, and a new broker was brought online with the same id, 
wherever topic partitions had been previously assigned to it should have been 
recreated.

At this point, however, I would shut down the cluster, delete the znodes, 
delete the topic directories from the brokers, then bring the cluster back up.

-- Peter

> On Dec 16, 2019, at 3:00 AM, Vincent Rischmann  wrote:
> 
> It doesn't exist anymore, we replaced it after a hardware failure.
> 
> Thinking about it I don't think I reassigned the partitions for broker 5 to 
> the new broker before deleting these topics, I didn't realize that it was 
> necessary for all brokers to be online.
> 
> Since broker 5 is never coming back again I'm guessing my only choice is to 
> manually modify the znodes ? 
> 
>> On Fri, Dec 13, 2019, at 19:07, Peter Bukowinski wrote:
>> If any brokers are offline, kafka can’t successfully delete a topic. 
>> What’s the state of broker 5?
>> 
>> -- Peter (from phone)
>> 
 On Dec 13, 2019, at 8:55 AM, Vincent Rischmann  
 wrote:
>>> 
>>> Hi,
>>> 
>>> I've deleted a bunch of topics yesterday on our cluster but some are now 
>>> stuck in "marked for deletion".
>>> 
>>> * i've looked in the data directory of every broker and there's no data 
>>> left for the topics, the directory doesn't exist anymore.
>>> * in zookeeper the znode `brokers/topics/mytopic` still exists
>>> * the znode `admin/delete_topics/mytopic` still exists
>>> 
>>> I've tried the following to no avail:
>>> 
>>> * restarting all brokers
>>> * removing the `admin/delete_topics/mytopic` node and re-running 
>>> `kafka-topics.sh --delete --topic mytopic`
>>> 
>>> In the kafka-controller.log of some brokers I see this which seems relevant:
>>> 
>>>   [2019-12-13 10:15:07,244] WARN [Channel manager on controller 6]: Not 
>>> sending request (type=StopReplicaRequest, controllerId=6, 
>>> controllerEpoch=78, deletePartitions=false, partitions=mytopic-17) to 
>>> broker 5, since it is offline. (kafka.controller.ControllerChannelManager)
>>>   [2019-12-13 10:15:07,244] WARN [Channel manager on controller 6]: Not 
>>> sending request (type=StopReplicaRequest, controllerId=6, 
>>> controllerEpoch=78, deletePartitions=false, partitions=mytopic-24) to 
>>> broker 5, since it is offline. (kafka.controller.ControllerChannelManager)
>>> 
>>> and
>>> 
>>>   12061:[2019-12-12 10:35:55,290] INFO [Topic Deletion Manager 1], Handling 
>>> deletion for topics mytopic (kafka.controller.TopicDeletionManager)
>>>   12062:[2019-12-12 10:35:55,292] INFO [Topic Deletion Manager 1], Not 
>>> retrying deletion of topic mytopic at this time since it is marked 
>>> ineligible for deletion (kafka.controller.TopicDeletionManager)
>>> 
>>> Since the data directory is already deleted I'm thinking of simply removing 
>>> the znode `brokers/topics/mytopic` from zookeeper manually.
>>> 
>>> Does anyone has another suggestion ? Is it safe to remove the znode 
>>> manually ?
>>> 
>>> Thanks.
>> 

Re: Kafka 2.4.0 & Mirror Maker 2.0 Error

[Ryanne Dolan]

Hey Jamie, are you running the MM2 connectors on an existing Connect
cluster, or with the connet-mirror-maker.sh driver? Given your question
about plugin.path I'm guessing the former. Is the Connect cluster running
2.4.0 as well? The jars should land in the Connect runtime without any need
to modify the plugin.path or copy jars around.

Ryanne

On Mon, Dec 16, 2019, 6:23 AM Jamie  wrote:

> Hi All,
> I'm trying to set up mirror maker 2.0 with Kafka 2.4.0 however, I'm
> receiving the following errors on startup:
> ERROR Plugin class loader for connector
> 'org.apache.kafka.connect.mirror.MirrorSourceConnector' was not found.
> Returning:
> org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader@187eb9a8
> (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
> ERROR Plugin class loader for connector
> 'org.apache.kafka.connect.mirror.MirrorHeartbeatConnector' was not
> found. Returning:
> org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader@187eb9a8
> (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
> ERROR Plugin class loader for connector
> 'org.apache.kafka.connect.mirror.MirrorCheckpointConnector' was not
> found. Returning:
> org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader@187eb9a8
> (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
>
> I've checked the jar file containing these class file is in the class
> path.
> Is there anything I need to add to plugin.path for the connect properties
> when running mirror maker?
> Many Thanks,
> Jamie

Re: Which container should you use when deploying on docker ?

[Robin Moffatt]

There are various Kafka images available, including:

https://hub.docker.com/r/confluentinc/cp-kafka/

https://hub.docker.com/r/wurstmeister/kafka/

I'm not 100% clear what your doubt is? If these are legitimate Kafka
images, or something else?



-- 

Robin Moffatt | Senior Developer Advocate | ro...@confluent.io | @rmoff


On Sat, 14 Dec 2019 at 04:18, Yu Watanabe  wrote:

> Hello.
>
> I would like to ask question related to kafka on docker engine.
> Which container should you use for kafka when deploying on docker in
> production ?
>
> When I look in docker hub , I do not see neither of below tagged for kafka
> container .
>
> Docker certified
> Verified publisher
> Official Images
>
> Repository "confluent" seems be the closest one since its the creator of
> kafka but it does not have above tag .
>
> Thanks,
> Yu Watanabe
>
> --
> Yu Watanabe
> Weekend Freelancer who loves to challenge building data platform
> yu.w.ten...@gmail.com
> [image: LinkedIn icon]   [image:
> Twitter icon] 
>

Kafka 2.4.0 & Mirror Maker 2.0 Error

[Jamie]

Hi All, 
I'm trying to set up mirror maker 2.0 with Kafka 2.4.0 however, I'm receiving 
the following errors on startup: 
ERROR Plugin class loader for connector 
'org.apache.kafka.connect.mirror.MirrorSourceConnector' was not found. 
Returning: 
org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader@187eb9a8 
(org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
ERROR Plugin class loader for connector 
'org.apache.kafka.connect.mirror.MirrorHeartbeatConnector' was not found. 
Returning: 
org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader@187eb9a8 
(org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
ERROR Plugin class loader for connector 
'org.apache.kafka.connect.mirror.MirrorCheckpointConnector' was not found. 
Returning: 
org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader@187eb9a8 
(org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)

I've checked the jar file containing these class file is in the class path. 
Is there anything I need to add to plugin.path for the connect properties when 
running mirror maker?
Many Thanks, 
Jamie

Re: Topics marked for deletion stuck as ineligible for deletion

[Vincent Rischmann]

It doesn't exist anymore, we replaced it after a hardware failure.

Thinking about it I don't think I reassigned the partitions for broker 5 to the 
new broker before deleting these topics, I didn't realize that it was necessary 
for all brokers to be online.

Since broker 5 is never coming back again I'm guessing my only choice is to 
manually modify the znodes ? 

On Fri, Dec 13, 2019, at 19:07, Peter Bukowinski wrote:
> If any brokers are offline, kafka can’t successfully delete a topic. 
> What’s the state of broker 5?
> 
> -- Peter (from phone)
> 
> > On Dec 13, 2019, at 8:55 AM, Vincent Rischmann  wrote:
> > 
> > Hi,
> > 
> > I've deleted a bunch of topics yesterday on our cluster but some are now 
> > stuck in "marked for deletion".
> > 
> > * i've looked in the data directory of every broker and there's no data 
> > left for the topics, the directory doesn't exist anymore.
> > * in zookeeper the znode `brokers/topics/mytopic` still exists
> > * the znode `admin/delete_topics/mytopic` still exists
> > 
> > I've tried the following to no avail:
> > 
> > * restarting all brokers
> > * removing the `admin/delete_topics/mytopic` node and re-running 
> > `kafka-topics.sh --delete --topic mytopic`
> > 
> > In the kafka-controller.log of some brokers I see this which seems relevant:
> > 
> >[2019-12-13 10:15:07,244] WARN [Channel manager on controller 6]: Not 
> > sending request (type=StopReplicaRequest, controllerId=6, 
> > controllerEpoch=78, deletePartitions=false, partitions=mytopic-17) to 
> > broker 5, since it is offline. (kafka.controller.ControllerChannelManager)
> >[2019-12-13 10:15:07,244] WARN [Channel manager on controller 6]: Not 
> > sending request (type=StopReplicaRequest, controllerId=6, 
> > controllerEpoch=78, deletePartitions=false, partitions=mytopic-24) to 
> > broker 5, since it is offline. (kafka.controller.ControllerChannelManager)
> > 
> > and
> > 
> >12061:[2019-12-12 10:35:55,290] INFO [Topic Deletion Manager 1], 
> > Handling deletion for topics mytopic (kafka.controller.TopicDeletionManager)
> >12062:[2019-12-12 10:35:55,292] INFO [Topic Deletion Manager 1], Not 
> > retrying deletion of topic mytopic at this time since it is marked 
> > ineligible for deletion (kafka.controller.TopicDeletionManager)
> > 
> > Since the data directory is already deleted I'm thinking of simply removing 
> > the znode `brokers/topics/mytopic` from zookeeper manually.
> > 
> > Does anyone has another suggestion ? Is it safe to remove the znode 
> > manually ?
> > 
> > Thanks.
>