Re: Inquiry on Scale Down Process for KRaft Kafka Cluster

[sunil chaudhari]

So if I understand this discussion correctly, there is no clear path to
scale down cluster is it?



On Tue, 29 Oct 2024 at 7:40 AM, Luke Chen  wrote:

> Hi hayoung,
>
> This is the problem that KIP-1066
> <
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1066%3A+Mechanism+to+cordon+brokers+and+log+directories
> >
> is trying to address.
> It is under discussion, and should be included in the upcoming releases.
>
> Thanks.
> Luke
>
>
>
> On Tue, Oct 29, 2024 at 12:45 AM hayoung lee  wrote:
>
> > Hi Team,
> >
> > I am currently operating a Kafka cluster in KRaft mode and would like to
> > raise a few questions and suggestions regarding the cluster scale-down
> > process.
> >
> > Recently, I scaled down the cluster by removing one broker; however, I
> > encountered a situation where the removed broker still appeared in the
> > cluster, causing new partitions to be assigned to it. To address this, I
> > was able to remove the broker from the controller metadata using the
> > kafka-cluster.sh command. However, I could not find any documentation on
> > this method, so I had to refer to the Kafka source code to discover this
> > solution. The issue is now resolved, but I would like to ask if there is
> a
> > more streamlined approach for scaling down.
> >
> > Additional Questions and Suggestions:
> >
> >1.
> >
> >*Automatic Cluster Exclusion*
> >I initially expected that a removed broker would be automatically
> >excluded from the cluster after a certain period. However, the broker
> >remained part of the cluster for quite some time. I am curious whether
> >Kafka has a feature that automatically detects and excludes removed
> > brokers
> >from the cluster.
> >2.
> >
> >*Atomic Scale Down*
> >If the above feature is not available, I would like to inquire if
> there
> >is a way to atomically exclude a broker and simultaneously remove it
> > from
> >the cluster metadata. Since we manage a Kafka cluster with continuous
> > topic
> >creation, a non-atomic process might lead to new topics being assigned
> > to
> >the broker between its exclusion and metadata removal, necessitating
> > manual
> >partition reassignments.
> >3.
> >
> >*Documentation Enhancement (Suggestion)*
> >While addressing the scale-down issue, I could not find any relevant
> >information in the Kafka User Guide. I had to refer to the source
> code,
> >specifically the unregister functionality, to learn that this could be
> >done using kafka-cluster.sh. Including documentation on this process
> >would likely make it easier for other users to manage their clusters
> >effectively.
> >
> > Thank you, and I look forward to any guidance or recommendations.
> >
> > Best regards,
> >
> > hayoung
> >
>

Re: Kakfa Topic gets auto-created post deletion

[sunil chaudhari]

Hi Abhishek,
as I told you before, topic is getting created only because of consumer
polling.
The only way is to stop consumer, remove topic name from the consumer,
delete topic.

Here I assume your consumer knows which topic to read from( by static topic
name)
If not, I dont know how your consumer knows excat topic name where to read
from?

In my case we had static topic names given in consumer config. So removed
those before topic deletion.

Alternatively you can set auto create false by rolling restart all your
brokers if number of brokers and not much.
Tbat will be quick fix



On Sun, 18 Feb 2024 at 6:07 PM, Abhishek Singla 
wrote:

> Hi Megh,
>
> Thanks for the detailed explanation. I tested it out and below are the
> observations.
>
> When the delete topic command is issued, the topic is marked for deletion
> and it is deleted within the next 1 minute. In this 1 minute interval, the
> consumer offsets are there in __consumer_offsets topic but the describe
> group command fails with below error.
> *Error: Executing consumer group command failed due to
> org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This
> server does not host this topic-partition.*
>
> Now, within that one minute if consumer is running then the topic gets
> created again and offset is reset to 0 and if consumer is not running then
> offsets are cleared automatically from __consumer_offsets topic and topic
> is not created upon resuming consumers.
>
> kafka-consumer-groups.sh --delete-offsets will expedite the offset
> clearance bit but it also requires consumers to be stopped. One more thing
> I noticed is --consumer-property allow.auto.create.topics=false does not
> restrict topic creation in this particular scenario.
>
> Is there any other way I can delete the topics without stopping consumers?
>
> Regards,
> Abhishek Singla
>
>
>
> On Sun, Feb 18, 2024 at 3:18 PM megh vidani 
> wrote:
>
> > Hi Abhishek,
> >
> > The consumer offsets get stored in the __consumer_offsets topic by the
> > kafka brokers and don't get automatically deleted when you delete a
> topic.
> > If you have auto.create.topics.enable set to true in your broker configs,
> > th consumer automatically creates the topic again.
> >
> > You will have to delete the offsets for that topic using the
> > kafka-consumer-groups.sh for your consumer group in order to remove the
> > topic offsets for that consumer group.
> >
> > Also, you can disable automatic creation of topic in the kafka brokers
> > config using auto.create.topics.enable property.
> >
> > Thanks,
> > Megh
> >
> >
> > On Sun, Feb 18, 2024, 15:09 Abhishek Singla  >
> > wrote:
> >
> > > We only delete topics which does not have any active producers.
> > >
> > > Nowhere in our consumer config we explicitly specify topic name, these
> > are
> > > subscribed based on pattern. How do I remove topic from consumer
> config?
> > >
> > > On Sun, 18 Feb 2024 at 2:51 PM, sunil chaudhari <
> > > sunilmchaudhar...@gmail.com>
> > > wrote:
> > >
> > > > correcte missig thing.
> > > > You have to remove that topic from producer as well as consumer
> config
> > > > both.
> > > >
> > > > On Sun, 18 Feb 2024 at 2:49 PM, sunil chaudhari <
> > > > sunilmchaudhar...@gmail.com>
> > > > wrote:
> > > >
> > > > > You have to remove that topic from consumer config.
> > > > > restart consumer.
> > > > > the wait for some time.
> > > > > Then delete topic.
> > > > > this time it wont create again.
> > > > >
> > > > > On Sun, 18 Feb 2024 at 1:07 PM, Abhishek Singla <
> > > > > abhisheksingla...@gmail.com> wrote:
> > > > >
> > > > >> Hi Team,
> > > > >>
> > > > >> Kafka version: 2_2.12-2.6.0
> > > > >> Zookeeper version: 3.4.14
> > > > >> Java version: 1.8.0_301
> > > > >>
> > > > >> Kafka Subscribe Pattern API is used to Subscribe to all topics
> > > matching
> > > > >> specified pattern to get dynamically assigned partitions. New
> topics
> > > of
> > > > >> specified pattern get created with time.
> > > > >>
> > > > >> > KafkaConsumer subscribe(Pattern pattern,
> > > > >> Handler>
> > > > >> completionHandler);
> > > > >>
> > > > >> The issue arises when we try to delete the topic. After the delete
> > > topic
> > > > >> command is issued, the topic is deleted successfully. However, it
> > gets
> > > > >> auto-created again within 5 mins. Broker offsets are reset to
> zero,
> > > new
> > > > >> topic partitions could be on same broker nodes or different.
> > > > >>
> > > > >> Below are some of the configs used:
> > > > >>
> > > > >>
> > > > >> *Server Configs:*
> > > > >> delete.topic.enable=true
> > > > >> auto.create.topics.enable: true
> > > > >>
> > > > >> *Consumer Configs*
> > > > >> allow.auto.create.topics=false
> > > > >>
> > > > >> Could someone help us out in understanding what might be the root
> > > cause
> > > > of
> > > > >> this issue?
> > > > >>
> > > > >> Regards,
> > > > >> Abhishek Singla
> > > > >>
> > > > >
> > > >
> > >
> >
>

Re: Kakfa Topic gets auto-created post deletion

[sunil chaudhari]

correcte missig thing.
You have to remove that topic from producer as well as consumer config
both.

On Sun, 18 Feb 2024 at 2:49 PM, sunil chaudhari 
wrote:

> You have to remove that topic from consumer config.
> restart consumer.
> the wait for some time.
> Then delete topic.
> this time it wont create again.
>
> On Sun, 18 Feb 2024 at 1:07 PM, Abhishek Singla <
> abhisheksingla...@gmail.com> wrote:
>
>> Hi Team,
>>
>> Kafka version: 2_2.12-2.6.0
>> Zookeeper version: 3.4.14
>> Java version: 1.8.0_301
>>
>> Kafka Subscribe Pattern API is used to Subscribe to all topics matching
>> specified pattern to get dynamically assigned partitions. New topics of
>> specified pattern get created with time.
>>
>> > KafkaConsumer subscribe(Pattern pattern,
>> Handler>
>> completionHandler);
>>
>> The issue arises when we try to delete the topic. After the delete topic
>> command is issued, the topic is deleted successfully. However, it gets
>> auto-created again within 5 mins. Broker offsets are reset to zero, new
>> topic partitions could be on same broker nodes or different.
>>
>> Below are some of the configs used:
>>
>>
>> *Server Configs:*
>> delete.topic.enable=true
>> auto.create.topics.enable: true
>>
>> *Consumer Configs*
>> allow.auto.create.topics=false
>>
>> Could someone help us out in understanding what might be the root cause of
>> this issue?
>>
>> Regards,
>> Abhishek Singla
>>
>

Re: Kakfa Topic gets auto-created post deletion

[sunil chaudhari]

You have to remove that topic from consumer config.
restart consumer.
the wait for some time.
Then delete topic.
this time it wont create again.

On Sun, 18 Feb 2024 at 1:07 PM, Abhishek Singla 
wrote:

> Hi Team,
>
> Kafka version: 2_2.12-2.6.0
> Zookeeper version: 3.4.14
> Java version: 1.8.0_301
>
> Kafka Subscribe Pattern API is used to Subscribe to all topics matching
> specified pattern to get dynamically assigned partitions. New topics of
> specified pattern get created with time.
>
> > KafkaConsumer subscribe(Pattern pattern, Handler>
> completionHandler);
>
> The issue arises when we try to delete the topic. After the delete topic
> command is issued, the topic is deleted successfully. However, it gets
> auto-created again within 5 mins. Broker offsets are reset to zero, new
> topic partitions could be on same broker nodes or different.
>
> Below are some of the configs used:
>
>
> *Server Configs:*
> delete.topic.enable=true
> auto.create.topics.enable: true
>
> *Consumer Configs*
> allow.auto.create.topics=false
>
> Could someone help us out in understanding what might be the root cause of
> this issue?
>
> Regards,
> Abhishek Singla
>

Script to delete list of topics

[sunil chaudhari]

Hi,
anybody there who has a shell script to delete topics from cluster.
I have list of elegible topics to be deleted.
Script should accept list of files as an input.
Delete topics one by one.
please share.
Thanks in advance.

regards,
Sunil.

Re: Kafka architecture

[sunil chaudhari]

Hi,

point 1…. If you want to mutate the messsage you have this  option.

1. Start a Ksql server
2. read from parent_topic, mutate and create child_topic where mutated
 message will be published
3. your consumer will read  child_topic to consume and processe the
message.

regards,
Sunil.


child_topic
On Sun, 7 Jan 2024 at 8:14 PM, Andrew Grant 
wrote:

> Hey Winstein,
>
> 1. You can’t modify events after they are written. They are immutable.
>
> 2. Consumers always see messages in order. Note this guarantee is per
> partition. That is, the consumer is guaranteed to read messages in order
> for a given partition. There isn’t a guarantee across partitions.
>
>
> Andrew
>
> Sent from my iPhone
>
> > On Jan 7, 2024, at 9:36 AM, Winstein Martins 
> wrote:
> >
> > Hello everyone, I have two questions about Kafka's operation.
> >
> > 1. Can I modify events after they are written to Kafka, or are they
> > immutable?
> > 2. Do consumers always receive messages in the order they were sent by
> > Kafka?
> >
> > Thank you in advance!
>

Re: License Cost for Apache Kafka with Support

[sunil chaudhari]

Hi Vinod,
you may want to contact confluent in this case.
You can contact separately, I can guide more on this.

thanks and regards,
Sunil.

On Thu, 9 Nov 2023 at 3:07 PM, Vinothkumar S
 wrote:

> Hi Team,
>
> We would like to use licensed version of Kafka along with Support.
> Could you please share the contact details to discuss further.
>
>
> Best Regards,
>
> Vinothkumar S
> Cloud Infra Specialist
> TATA ELXSI
> ITPB Road Whitefield Bangalore 560 048 India
> 
> Cell +91 9686707172
> www.tataelxsi.com
> "Always Smile:), because your smile itself is a reason for many others to
> smile!"
>
> 
> Disclaimer: This email and any files transmitted with it are confidential
> and intended solely for the use of the individual or entity to whom they
> are addressed. If you are not the intended recipient of this message , or
> if this message has been addressed to you in error, please immediately
> alert the sender by reply email and then delete this message and any
> attachments. If you are not the intended recipient, you are hereby notified
> that any use, dissemination, copying, or storage of this message or its
> attachments is strictly prohibited. Email transmission cannot be guaranteed
> to be secure or error-free, as information could be intercepted, corrupted,
> lost, destroyed, arrive late or incomplete, or contain viruses. The sender,
> therefore, does not accept liability for any errors, omissions or
> contaminations in the contents of this message which might have occurred as
> a result of email transmission. If verification is required, please request
> for a hard-copy version.
> 
>

Re: Apache Kafka Installation and Env Set up

[sunil chaudhari]

Hi Venkat,
are you planning to use Open source Apache Kafka Or Confluent?
what is your use case apart from streaming?

regards,
Sunil.

On Fri, 29 Sep 2023 at 12:27 PM, ANANTHA VENKATA GANJAM
 wrote:

> TCS Confidential
>
> Hi Team,
>
> We are planning to set up Lab environment for Kafka in TCS. Please guide
> us on the next steps.
>
>
> Thanks & Regards,
> Vijaya Sri
>
>
>
> TCS Confidential
> =-=-=
> Notice: The information contained in this e-mail
> message and/or attachments to it may contain
> confidential or privileged information. If you are
> not the intended recipient, any dissemination, use,
> review, distribution, printing or copying of the
> information contained in this e-mail message
> and/or attachments to it are strictly prohibited. If
> you have received this communication in error,
> please notify us by reply e-mail or telephone and
> immediately and permanently delete the message
> and any attachments. Thank you
>
>
>

Re: Regarding Distributed Kafka-connect cluster

[sunil chaudhari]

Kudos to Nikhil.
your explanation adds to my knowledge.

🙏

On Mon, 25 Sep 2023 at 7:16 PM, Nikhil Srivastava <
nikhilsrivastava4...@gmail.com> wrote:

> Hi Yeikel,
>
> Sharing my two cents. Would let others chime in to add to this.
>
> Based on my understanding, if connect workers (which are all part of the
> same cluster) can communicate with the kafka brokers (which happens to be
> the Group Coordinator and facilitates Connect Leader Election via Group
> Membership Protocol), then only 1 connect worker will be elected as leader
> amongst all others in the cluster. Outside of that, I believe a bunch of
> REST calls to connect workers are forwarded to the connect leader (if the
> REST request lands on a connect worker which isn't a leader). In case of a
> non-retriable network partition between the non-leader worker and leader
> worker, those REST requests will fail. I'm referring to REST requests like
> CREATE / UPDATE / DELETE.
>
> Hope this helps a little.
>
> Thanks,
> -Nikhil
>
> On Sun, 24 Sept 2023 at 06:36, Yeikel Santana  wrote:
>
> > Hello everyone,I'm currently designing a new Kafka Connect cluster, and
> > I'm trying to understand how connectivity functions among workers.In my
> > setup, I have a single Kafka Connect cluster connected to the same Kafka
> > topics and Kafka cluster. However, the workers are deployed in
> > geographically separated data centers, each of which is fully isolated at
> > the networkI suspect that this setup might not work with Kafka Connect
> > because my current understanding is that ALL workers need to communicate
> > with the leader for task coordination and heartbeats.In terms of leader
> > election, can this result in multiple leaders and other potential
> > issues?Any input and suggestions would be appreciated
>

Re: Kafka connect Graceful stop of task failed

[sunil chaudhari]

I think when you delete connector it removes the task and workers continues
to run.
When you stop it actually stops the worker.
Both different things.
Point to be noted is Worker has connector.
So connector should be removed before stopping the worker.

Though I am not expert in this.

On Mon, 21 Aug 2023 at 7:10 PM, Robson Hermes 
wrote:

> Hello Sunil
>
> I'm not calling a stop, I'm straight deleting the connectors with the
> DELETE. Stopping the connector is done internally during deletion.
>
> Regards
> Robson
>
> On Mon, 21 Aug 2023 at 15:36, sunil chaudhari  >
> wrote:
>
> > You have to remove connectors first using delete api
> > and then stop the connector
> >
> > On Thu, 17 Aug 2023 at 2:51 AM, Robson Hermes 
> > wrote:
> >
> > > Hello
> > >
> > > I'm using kafka connect 7.4.0 to read data from Postgres views and
> write
> > to
> > > another Postgres tables. So using JDBC source and sink connectors.
> > > All works good, but whenever I stop the source connectors via the rest
> > api:
> > >
> > > DEL http://kafka-connect:8083/connectors/connector_name_here
> > >
> > > The connector stops fine, but not the task:
> > >
> > >
> > > Graceful stop of connector (connector-name-here) succeeded.
> > >
> > > Graceful stop of task (task-name-here) failed.
> > >
> > >
> > > It only happens with the *source* connector tasks. The sink connector
> > > and tasks shutdown gracefully and fine.
> > >
> > > The timeout for task shutdown has been increased, but didn't help:
> > >
> > > task.shutdown.graceful.timeout.ms=6
> > >
> > >
> > >
> > > The connectors are running once per day (during the night) to load a
> > > lot of data, and the error happens when I try to delete the connectors
> > > in the middle of the day. That is, they are not actually
> > > executing/loading any data, it has finished already.
> > >
> > > offset.flush.interval.ms=1 in development and integration
> > > environments.
> > >
> > >  offset.flush.interval.ms=6 in production and uat.
> > >
> > >
> > > The rest of the config is pretty much the default.
> > >
> > > What could be the issue?
> > >
> > > The errors of the graceful stop of the tasks are triggering our alert
> > > system, so trying to get rid of those.
> > >
> > >
> > > Thanks a lot
> > >
> > > Robson
> > >
> >
>

Re: Kafka connect Graceful stop of task failed

[sunil chaudhari]

You have to remove connectors first using delete api
and then stop the connector

On Thu, 17 Aug 2023 at 2:51 AM, Robson Hermes 
wrote:

> Hello
>
> I'm using kafka connect 7.4.0 to read data from Postgres views and write to
> another Postgres tables. So using JDBC source and sink connectors.
> All works good, but whenever I stop the source connectors via the rest api:
>
> DEL http://kafka-connect:8083/connectors/connector_name_here
>
> The connector stops fine, but not the task:
>
>
> Graceful stop of connector (connector-name-here) succeeded.
>
> Graceful stop of task (task-name-here) failed.
>
>
> It only happens with the *source* connector tasks. The sink connector
> and tasks shutdown gracefully and fine.
>
> The timeout for task shutdown has been increased, but didn't help:
>
> task.shutdown.graceful.timeout.ms=6
>
>
>
> The connectors are running once per day (during the night) to load a
> lot of data, and the error happens when I try to delete the connectors
> in the middle of the day. That is, they are not actually
> executing/loading any data, it has finished already.
>
> offset.flush.interval.ms=1 in development and integration
> environments.
>
>  offset.flush.interval.ms=6 in production and uat.
>
>
> The rest of the config is pretty much the default.
>
> What could be the issue?
>
> The errors of the graceful stop of the tasks are triggering our alert
> system, so trying to get rid of those.
>
>
> Thanks a lot
>
> Robson
>

Re: Single Broker Constantly Reaches 100% CPU

[sunil chaudhari]

Point 2 may impact if the size of partitions is too big.
too many log segments will cause those many iops
I am not expert though

On Wed, 9 Aug 2023 at 6:43 PM, Tiansu Yu 
wrote:

> 1. We use cruise-control to actively balance the partitions across all
> brokers. So point 1 could be ruled out.
> 2. I am not sure how much this would impact the broker, as we do have some
> exceptionally large partitions around. I have to check to know if they live
> on the aforementioned broker. So far I don't see there is strong
> correlation between total producer / consumer byte rates with CPU spikes on
> this broker.
>
> Tiansu Yu
> Engineer
> Data Ingestion & Streaming
>
> Klarna Bank AB German Branch
> Chausseestraße 117
> <https://www.google.com/maps/search/Chausseestra%C3%9Fe+117+10115+Berlin?entry=gmail&source=g>
> 10115 Berlin
> <https://www.google.com/maps/search/Chausseestra%C3%9Fe+117+10115+Berlin?entry=gmail&source=g>
> Tel: +49 221 669 501 00
> klarna.de
>
> Klarna Bank AB, German Branch
> Sitz: Berlin, Amtsgericht Charlottenburg HRB 217291 B
> USt-Nr.: DE 815 867 324
> Zweigstelle der Klarna Bank AB (publ), AG schwedischen Rechts mit
> Hauptsitz in Stockholm,
> Schw. Gesellschaftsregister 556737-0431
> Verwaltungsratsvorsitzender: Michael Moritz
> Geschäftsführender Direktor: Sebastian Siemiatkowski
> Leiter Zweigniederlassung: Yaron Shaer, Björn Petersen
>
> On 9. Aug 2023, at 12:05, sunil chaudhari 
> wrote:
>
> Hi I can guess two problems here.
> 1. Either too many partition’s concentrated on this broker compared to
> other broker
> 2. The partitions on this broker might have larger size as compared to the
> parition on other brokers
>
> please chech if all brokers are evenly balanced in terms of number of
> partitions and the total topic size on each broker.
>
> On Wed, 9 Aug 2023 at 1:29 PM, Tiansu Yu 
> wrote:
>
> Hi Kafka community,
>
> We have an issue with our Kafka cluster from time to time, that a single
> (one and only one) broker (leader) in the cluster reaches 100% CPU
> utilisation. We could not see any apparent issue from the metrics. There is
> no heap memory usage increase, no excessive connections made on the broker,
> no misbehaving producers and consumers trying to dump or load excessively
> during these periods. The only difference we could see is that thread usage
> decreases during these period. Despite the problem, the service is still
> available (understandable from Kafka's perspective.)
>
> We are trying to understand what else might be the cause of the issue and
> how we can mitigate them.
>
> Tiansu Yu
> Engineer
> Data Ingestion & Streaming
>
> Klarna Bank AB German Branch
> Chausseestraße 117
> <https://www.google.com/maps/search/Chausseestra%C3%9Fe+117?entry=gmail&source=g>
>
> <
> https://www.google.com/maps/search/Chausseestra%C3%9Fe+117+10115+Berlin?entry=gmail&source=g
> >
> 10115 Berlin
> <
> https://www.google.com/maps/search/Chausseestra%C3%9Fe+117+10115+Berlin?entry=gmail&source=g
> >
>
>
> Tel: +49 221 669 501 00
> klarna.de
>
> Klarna Bank AB, German Branch
> Sitz: Berlin, Amtsgericht Charlottenburg HRB 217291 B
> USt-Nr.: DE 815 867 324
> Zweigstelle der Klarna Bank AB (publ), AG schwedischen Rechts mit
> Hauptsitz in Stockholm,
> Schw. Gesellschaftsregister 556737-0431
> Verwaltungsratsvorsitzender: Michael Moritz
> Geschäftsführender Direktor: Sebastian Siemiatkowski
> Leiter Zweigniederlassung: Yaron Shaer, Björn Petersen
>
>
>

Re: Single Broker Constantly Reaches 100% CPU

[sunil chaudhari]

Hi I can guess two problems here.
1. Either too many partition’s concentrated on this broker compared to
other broker
2. The partitions on this broker might have larger size as compared to the
parition on other brokers

please chech if all brokers are evenly balanced in terms of number of
partitions and the total topic size on each broker.

On Wed, 9 Aug 2023 at 1:29 PM, Tiansu Yu 
wrote:

> Hi Kafka community,
>
> We have an issue with our Kafka cluster from time to time, that a single
> (one and only one) broker (leader) in the cluster reaches 100% CPU
> utilisation. We could not see any apparent issue from the metrics. There is
> no heap memory usage increase, no excessive connections made on the broker,
> no misbehaving producers and consumers trying to dump or load excessively
> during these periods. The only difference we could see is that thread usage
> decreases during these period. Despite the problem, the service is still
> available (understandable from Kafka's perspective.)
>
> We are trying to understand what else might be the cause of the issue and
> how we can mitigate them.
>
> Tiansu Yu
> Engineer
> Data Ingestion & Streaming
>
> Klarna Bank AB German Branch
> Chausseestraße 117
> 
> 10115 Berlin
> 
> Tel: +49 221 669 501 00
> klarna.de
>
> Klarna Bank AB, German Branch
> Sitz: Berlin, Amtsgericht Charlottenburg HRB 217291 B
> USt-Nr.: DE 815 867 324
> Zweigstelle der Klarna Bank AB (publ), AG schwedischen Rechts mit
> Hauptsitz in Stockholm,
> Schw. Gesellschaftsregister 556737-0431
> Verwaltungsratsvorsitzender: Michael Moritz
> Geschäftsführender Direktor: Sebastian Siemiatkowski
> Leiter Zweigniederlassung: Yaron Shaer, Björn Petersen
>
>

Re: Kafka server.log deletion config

[sunil chaudhari]

Hi Gaurav,
you can make use of log4j.properties for managing log files.
Define housekeeping policy and get rid of too many files.


On Sat, 22 Jul 2023 at 10:58 AM, Gaurav Pande  wrote:

> Hi All,
>
> I am running Apache kafka 2.7.0 and I see that presently there are many
> server.logv files but how is the deletion of these file governed or can be
> governed? Instead of manually deleting them.
>
> Is there some config that I can set to delete them regularly.
>
> Thanks,
> GP
>

Re: Kafka data directory Move without data loss

[sunil chaudhari]

I suggest you practice this thing in some dev or test environment before
doing it to prod environment.
And copy the data,  Not move.



On Tue, 18 Jul 2023 at 8:22 PM, sunil chaudhari 
wrote:

>
> Better you have a rollback plan. In case the new mountpount has some issue
> then you should be able to point to the old fs and restart the broker.
>
>
> On Tue, 18 Jul 2023 at 5:23 PM, Gaurav Pande  wrote:
>
>> Okay and this process doesn't corrupt the files/file system right?
>>
>> And to perform this copy activity do we need to stop Kafka broker as well?
>>
>> Regards,
>> GP
>>
>> On Tue, 18 Jul, 2023, 16:24 sunil chaudhari, > >
>> wrote:
>>
>> > Actually the broker refers to the base directory.
>> > Example:
>> > log.paths: /var/log/mykafka/data/
>> >
>> > then copy. /data/. to the new path completely and refer it in the
>> > server.properties
>> >
>> > On Tue, 18 Jul 2023 at 1:09 PM, Gaurav Pande 
>> wrote:
>> >
>> > > Okay so the data directory has some files like checkpoints,
>> > meta.properties
>> > > in existing file system all of these should also be copied to new
>> > disk/file
>> > > system?
>> > >
>> > > Regards,
>> > > GP
>> > >
>> > > On Tue, 18 Jul, 2023, 12:32 sunil chaudhari, <
>> > sunilmchaudhar...@gmail.com>
>> > > wrote:
>> > >
>> > > > I think you can copy whole data directory to new one. I dont think
>> > there
>> > > > will be loss or corruption
>> > > >
>> > > > On Tue, 18 Jul 2023 at 12:20 PM, Gaurav Pande > >
>> > > > wrote:
>> > > >
>> > > > > Hi Guys,
>> > > > >
>> > > > > Any help on this query please.
>> > > > >
>> > > > > Regards,
>> > > > > GP
>> > > > >
>> > > > > On Mon, 17 Jul, 2023, 21:00 Gaurav Pande, 
>> > > wrote:
>> > > > >
>> > > > > > Hi Guys,
>> > > > > >
>> > > > > > I have 3 node Kafka cluster running on version 2.7.0 . On one of
>> > the
>> > > > > > broker out of three iam noticing data directory getting filled
>> and
>> > > > broker
>> > > > > > going down.
>> > > > > >
>> > > > > > Presently the data is being written to / mount point(which is
>> > wrong)
>> > > > so I
>> > > > > > want to add a disk  on my VM.
>> > > > > >
>> > > > > > For this  my query is how do I migrate/move my existing data
>> from
>> > > > root(/)
>> > > > > > mount point to this new one without data loss?
>> > > > > >
>> > > > > > Is there any documentation on this.
>> > > > > >
>> > > > > > Thanks,
>> > > > > > GP
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>>
>

Re: Kafka data directory Move without data loss

[sunil chaudhari]

Better you have a rollback plan. In case the new mountpount has some issue
then you should be able to point to the old fs and restart the broker.


On Tue, 18 Jul 2023 at 5:23 PM, Gaurav Pande  wrote:

> Okay and this process doesn't corrupt the files/file system right?
>
> And to perform this copy activity do we need to stop Kafka broker as well?
>
> Regards,
> GP
>
> On Tue, 18 Jul, 2023, 16:24 sunil chaudhari, 
> wrote:
>
> > Actually the broker refers to the base directory.
> > Example:
> > log.paths: /var/log/mykafka/data/
> >
> > then copy. /data/. to the new path completely and refer it in the
> > server.properties
> >
> > On Tue, 18 Jul 2023 at 1:09 PM, Gaurav Pande 
> wrote:
> >
> > > Okay so the data directory has some files like checkpoints,
> > meta.properties
> > > in existing file system all of these should also be copied to new
> > disk/file
> > > system?
> > >
> > > Regards,
> > > GP
> > >
> > > On Tue, 18 Jul, 2023, 12:32 sunil chaudhari, <
> > sunilmchaudhar...@gmail.com>
> > > wrote:
> > >
> > > > I think you can copy whole data directory to new one. I dont think
> > there
> > > > will be loss or corruption
> > > >
> > > > On Tue, 18 Jul 2023 at 12:20 PM, Gaurav Pande 
> > > > wrote:
> > > >
> > > > > Hi Guys,
> > > > >
> > > > > Any help on this query please.
> > > > >
> > > > > Regards,
> > > > > GP
> > > > >
> > > > > On Mon, 17 Jul, 2023, 21:00 Gaurav Pande, 
> > > wrote:
> > > > >
> > > > > > Hi Guys,
> > > > > >
> > > > > > I have 3 node Kafka cluster running on version 2.7.0 . On one of
> > the
> > > > > > broker out of three iam noticing data directory getting filled
> and
> > > > broker
> > > > > > going down.
> > > > > >
> > > > > > Presently the data is being written to / mount point(which is
> > wrong)
> > > > so I
> > > > > > want to add a disk  on my VM.
> > > > > >
> > > > > > For this  my query is how do I migrate/move my existing data from
> > > > root(/)
> > > > > > mount point to this new one without data loss?
> > > > > >
> > > > > > Is there any documentation on this.
> > > > > >
> > > > > > Thanks,
> > > > > > GP
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: Kafka data directory Move without data loss

[sunil chaudhari]

Actually the broker refers to the base directory.
Example:
log.paths: /var/log/mykafka/data/

then copy. /data/. to the new path completely and refer it in the
server.properties

On Tue, 18 Jul 2023 at 1:09 PM, Gaurav Pande  wrote:

> Okay so the data directory has some files like checkpoints, meta.properties
> in existing file system all of these should also be copied to new disk/file
> system?
>
> Regards,
> GP
>
> On Tue, 18 Jul, 2023, 12:32 sunil chaudhari, 
> wrote:
>
> > I think you can copy whole data directory to new one. I dont think there
> > will be loss or corruption
> >
> > On Tue, 18 Jul 2023 at 12:20 PM, Gaurav Pande 
> > wrote:
> >
> > > Hi Guys,
> > >
> > > Any help on this query please.
> > >
> > > Regards,
> > > GP
> > >
> > > On Mon, 17 Jul, 2023, 21:00 Gaurav Pande, 
> wrote:
> > >
> > > > Hi Guys,
> > > >
> > > > I have 3 node Kafka cluster running on version 2.7.0 . On one of the
> > > > broker out of three iam noticing data directory getting filled and
> > broker
> > > > going down.
> > > >
> > > > Presently the data is being written to / mount point(which is wrong)
> > so I
> > > > want to add a disk  on my VM.
> > > >
> > > > For this  my query is how do I migrate/move my existing data from
> > root(/)
> > > > mount point to this new one without data loss?
> > > >
> > > > Is there any documentation on this.
> > > >
> > > > Thanks,
> > > > GP
> > > >
> > >
> >
>

Re: Kafka data directory Move without data loss

[sunil chaudhari]

I think you can copy whole data directory to new one. I dont think there
will be loss or corruption

On Tue, 18 Jul 2023 at 12:20 PM, Gaurav Pande  wrote:

> Hi Guys,
>
> Any help on this query please.
>
> Regards,
> GP
>
> On Mon, 17 Jul, 2023, 21:00 Gaurav Pande,  wrote:
>
> > Hi Guys,
> >
> > I have 3 node Kafka cluster running on version 2.7.0 . On one of the
> > broker out of three iam noticing data directory getting filled and broker
> > going down.
> >
> > Presently the data is being written to / mount point(which is wrong) so I
> > want to add a disk  on my VM.
> >
> > For this  my query is how do I migrate/move my existing data from root(/)
> > mount point to this new one without data loss?
> >
> > Is there any documentation on this.
> >
> > Thanks,
> > GP
> >
>

Re: Partition stickiness on consumer group rebalance?

[sunil chaudhari]

I will try to answer.
rebalancing triggers when one or two consuemrs(client) leaves the group
because of any reason.
The thumb rule is Number of partitions should be equal to number of
consumer threads.
If there are 300 partitions assigned one thread each it wont rebalance
untill some consumer marked as dead.
How it marks as dead: if the kafka doesnt receive heartbeat from consumer
in 5 mins(or defined at client side).

If 20 consumers are dead from kafka persepctive at time T1 then it will
trigger rebalance.
It will trigger rebalance at time T2 when there is new consumer added to
the group and there is poll request from new consumer.

If there is no issue with number of partitions and number of consumers then
it wont trigger rebalance.

Terms I used may not be accurate😊

Regards,
Sunil.

On Thu, 8 Jun 2023 at 3:59 PM, Edvard Fagerholm 
wrote:

> Hello,
>
> I couldn't find an answer in the documentation to the following. If a new
> machine joins a consumer group and Kafka triggers a rebalance, will it
> randomly reassign partitions or will it hand over partitions from existing
> consumers to the newly joined one? In other words, will it attempt to move
> as few partitions as possible between consumers?
>
> The main implications of this is local in-memory caches and scaling up the
> number of machines in a consumer group, since the scaling up operation will
> require nuking any local caches for the partitions that were moved. This
> would cause a spike on any DBs that are being cached on the consumers.
>
> Best,
> Edvard
>

Re: Question about KRaft

[sunil chaudhari]

Hi Luke,
This docu is good.
Does it apply for confluent as well?



On Fri, 10 Mar 2023 at 8:47 AM, Luke Chen  wrote:

> Hi Zhenyu,
>
> Answering your question:
>
> > Should I simply
> 1. download 3.4 binary
> 2. stop ZK & Kafka service
> 3. upgrade Kafka to 3.4
> 4. start only Kafka service with KRaft server.properties
>
> That is not migrating, actually. That is just creating another kafka
> cluster in KRaft mode.
> The point for migration is to move metadata in ZK into KRaft controllers.
> You can follow the guide here to do migration:
> https://kafka.apache.org/documentation/#kraft_zk_migration
>
> Thank you.
> Luke
>
> On Tue, Mar 7, 2023 at 11:07 PM Zhenyu Wang 
> wrote:
>
> > Hi Sunil,
> >
> > As mentioned earlier in my question, I have only one "combined" node as
> > both controller and broker, and I totally accept downtime (stop service)
> >
> > So just want to ask for my case, single node, if I want to upgrade to 3.4
> > then start service under KRaft (get rid of ZK), what would be the steps?
> >
> > Thanks~
> >
> > On Mon, Mar 6, 2023 at 11:49 PM sunil chaudhari <
> > sunilmchaudhar...@gmail.com>
> > wrote:
> >
> > > How will you achieve zero downtime of you stop zookeeper and kafka?
> > > There must be some standard steps so that stop zookeeper one by one and
> > > start kraft same time so that it will be migrated gradually.
> > >
> > >
> > >
> > > On Tue, 7 Mar 2023 at 9:26 AM, Zhenyu Wang 
> > wrote:
> > >
> > > > Hi team,
> > > >
> > > > Here is a question about KRaft from normal user, who starts to use
> and
> > > > learn Kafka since 3.2
> > > >
> > > > Last month Kafka 3.4, the first bridge release was available, and I
> am
> > > > considering to have a plan to use KRaft (get rid of ZK) since this
> > > version
> > > >
> > > > Currently I am using 3.3.2 (upgrade from 3.2) with only one node,
> which
> > > is
> > > > both controller & broker, even ZK is installed on this node too
> (sorry
> > I
> > > > know it is not distributed and I will try to improve it with more
> > > knowledge
> > > > learned in future)
> > > >
> > > > When I read KIP-866, ZK to KRaft migration, from section Migration
> > > > Overview, seems like the document is for multi-nodes with no or
> almost
> > no
> > > > downtime, enable KRaft node by node; however my case accepts downtime
> > > (one
> > > > node -_-!!), just want to have Kafka upgrade to 3.4 then start
> service
> > > > under KRaft mode, make sure everything works well and no log lost
> > > >
> > > > Should I simply
> > > > 1. download 3.4 binary
> > > > 2. stop ZK & Kafka service
> > > > 3. upgrade Kafka to 3.4
> > > > 4. start only Kafka service with KRaft server.properties
> > > >
> > > > Or any other thing I need to pay attention to?
> > > >
> > > > If there is a documentation as guide that would be quite helpful
> > > >
> > > > Really appreciate
> > > >
> > >
> >
>

Re: Question about KRaft

[sunil chaudhari]

How will you achieve zero downtime of you stop zookeeper and kafka?
There must be some standard steps so that stop zookeeper one by one and
start kraft same time so that it will be migrated gradually.



On Tue, 7 Mar 2023 at 9:26 AM, Zhenyu Wang  wrote:

> Hi team,
>
> Here is a question about KRaft from normal user, who starts to use and
> learn Kafka since 3.2
>
> Last month Kafka 3.4, the first bridge release was available, and I am
> considering to have a plan to use KRaft (get rid of ZK) since this version
>
> Currently I am using 3.3.2 (upgrade from 3.2) with only one node, which is
> both controller & broker, even ZK is installed on this node too (sorry I
> know it is not distributed and I will try to improve it with more knowledge
> learned in future)
>
> When I read KIP-866, ZK to KRaft migration, from section Migration
> Overview, seems like the document is for multi-nodes with no or almost no
> downtime, enable KRaft node by node; however my case accepts downtime (one
> node -_-!!), just want to have Kafka upgrade to 3.4 then start service
> under KRaft mode, make sure everything works well and no log lost
>
> Should I simply
> 1. download 3.4 binary
> 2. stop ZK & Kafka service
> 3. upgrade Kafka to 3.4
> 4. start only Kafka service with KRaft server.properties
>
> Or any other thing I need to pay attention to?
>
> If there is a documentation as guide that would be quite helpful
>
> Really appreciate
>

Re: Kafka process killed during first attempt of startup

[sunil chaudhari]

Ok sorry to misinterpret your message.
However if you  are learning and its your first time, you can clear the
data from the logs directory defined in server.properties.
And start fresh instance and then see what happens.
I hope you have followed all documents properly.

On Thu, 16 Feb 2023 at 7:07 AM, Angel Motta  wrote:

> Hello Sunil
> Thanks for your message. Only to clarify that I never could successfully
> start Kafka. This my first attempt to start Kafka and I get the "killed"
> message as shown in the log.
> Knowing this fact is necessary to remove all data and logs? If this helps,
> could you tell me how to do this?
>
>
>
> On Wed, Feb 15, 2023 at 8:29 PM sunil chaudhari <
> sunilmchaudhar...@gmail.com>
> wrote:
>
> > Remove all data and logs.
> > And start it.
> > Next time when you want to stop then dont kill the process with kill
> > command.
> > Stop it gracefully using kafka-server-stop under /bin
> > Kafka needs stop signal to do some cleanup operations before it stops. So
> > kill is not the option.
> >
> > On Thu, 16 Feb 2023 at 6:49 AM, Angel Motta 
> wrote:
> >
> > > Hello everyone!
> > > New Kafka user here. I am learning Kafka and trying to run it for the
> > > first time (part of my CS undergrad thesis)
> > > I have the minimum server in AWS EC2 with 500MB of RAM and whenever I
> try
> > > to start Kafka (kraft mode) I get "killed" as a last message.
> > >
> > > First I had to change the heap size from default (1GB) because I
> received
> > > an error about it so now It is 256M and now I see the logs of startup
> > > process but the process is killed.
> > >
> > > How can I know the cause of this error?
> > > I paste the last part of the log here. I just followed the quickstart
> > > https://kafka.apache.org/quickstart
> > > I also quickly saw the log directory (server, controller) but any hint
> to
> > > me.
> > >
> > > Thanks in advance for your assistance.
> > >
> > > [2023-02-16 01:00:54,082] INFO [LogLoader
> partition=__cluster_metadata-0,
> > > dir=/tmp/kraft-combined-logs] Producer state recovery took 3ms for
> > snapshot
> > > load and 0ms for segment recovery from offset 28
> (kafka.log.UnifiedLog$)
> > >
> > > [2023-02-16 01:00:54,131] INFO Initialized snapshots with IDs
> SortedSet()
> > > from /tmp/kraft-combined-logs/__cluster_metadata-0
> > > (kafka.raft.KafkaMetadataLog$)
> > >
> > > [2023-02-16 01:00:54,183] INFO [raft-expiration-reaper]: Starting
> > > (kafka.raft.TimingWheelExpirationService$ExpiredOperationReaper)
> > >
> > > [2023-02-16 01:00:54,523] INFO [RaftManager nodeId=1] Completed
> > transition
> > > to ResignedState(localId=1, epoch=9, voters=[1],
> electionTimeoutMs=1695,
> > > unackedVoters=[], preferredSuccessors=[])
> > > (org.apache.kafka.raft.QuorumState)
> > >
> > > [2023-02-16 01:00:54,533] INFO [RaftManager nodeId=1] Completed
> > transition
> > > to CandidateState(localId=1, epoch=10, retries=1,
> electionTimeoutMs=1539)
> > > (org.apache.kafka.raft.QuorumState)
> > >
> > > [2023-02-16 01:00:54,549] INFO [RaftManager nodeId=1] Completed
> > transition
> > > to Leader(localId=1, epoch=10, epochStartOffset=28,
> > > highWatermark=Optional.empty, voterStates={1=ReplicaState(nodeId=1,
> > > endOffset=Optional.empty, lastFetchTimestamp=-1,
> > lastCaughtUpTimestamp=-1,
> > > hasAcknowledgedLeader=true)}) (org.apache.kafka.raft.QuorumState)
> > >
> > > [2023-02-16 01:00:54,615] INFO [kafka-raft-outbound-request-thread]:
> > > Starting (kafka.raft.RaftSendThread)
> > >
> > > [2023-02-16 01:00:54,620] INFO [kafka-raft-io-thread]: Starting
> > > (kafka.raft.KafkaRaftManager$RaftIoThread)
> > >
> > > [2023-02-16 01:00:54,694] INFO [RaftManager nodeId=1] High watermark
> set
> > to
> > > LogOffsetMetadata(offset=29,
> > >
> metadata=Optional[(segmentBaseOffset=0,relativePositionInSegment=3044)])
> > > for the first time for epoch 10 based on indexOfHw 0 and voters
> > > [ReplicaState(nodeId=1, endOffset=Optional[LogOffsetMetadata(offset=29,
> > >
> >
> metadata=Optional[(segmentBaseOffset=0,relativePositionInSegment=3044)])],
> > > lastFetchTimestamp=-1, lastCaughtUpTimestamp=-1,
> > > hasAcknowledgedLeader=true)] (org.apache.kafka.raft.LeaderState)
> > >
> > > [2023-02-16 01:00:54,723] INFO [RaftManager nodeId=1

Re: Kafka process killed during first attempt of startup

[sunil chaudhari]

Remove all data and logs.
And start it.
Next time when you want to stop then dont kill the process with kill
command.
Stop it gracefully using kafka-server-stop under /bin
Kafka needs stop signal to do some cleanup operations before it stops. So
kill is not the option.

On Thu, 16 Feb 2023 at 6:49 AM, Angel Motta  wrote:

> Hello everyone!
> New Kafka user here. I am learning Kafka and trying to run it for the
> first time (part of my CS undergrad thesis)
> I have the minimum server in AWS EC2 with 500MB of RAM and whenever I try
> to start Kafka (kraft mode) I get "killed" as a last message.
>
> First I had to change the heap size from default (1GB) because I received
> an error about it so now It is 256M and now I see the logs of startup
> process but the process is killed.
>
> How can I know the cause of this error?
> I paste the last part of the log here. I just followed the quickstart
> https://kafka.apache.org/quickstart
> I also quickly saw the log directory (server, controller) but any hint to
> me.
>
> Thanks in advance for your assistance.
>
> [2023-02-16 01:00:54,082] INFO [LogLoader partition=__cluster_metadata-0,
> dir=/tmp/kraft-combined-logs] Producer state recovery took 3ms for snapshot
> load and 0ms for segment recovery from offset 28 (kafka.log.UnifiedLog$)
>
> [2023-02-16 01:00:54,131] INFO Initialized snapshots with IDs SortedSet()
> from /tmp/kraft-combined-logs/__cluster_metadata-0
> (kafka.raft.KafkaMetadataLog$)
>
> [2023-02-16 01:00:54,183] INFO [raft-expiration-reaper]: Starting
> (kafka.raft.TimingWheelExpirationService$ExpiredOperationReaper)
>
> [2023-02-16 01:00:54,523] INFO [RaftManager nodeId=1] Completed transition
> to ResignedState(localId=1, epoch=9, voters=[1], electionTimeoutMs=1695,
> unackedVoters=[], preferredSuccessors=[])
> (org.apache.kafka.raft.QuorumState)
>
> [2023-02-16 01:00:54,533] INFO [RaftManager nodeId=1] Completed transition
> to CandidateState(localId=1, epoch=10, retries=1, electionTimeoutMs=1539)
> (org.apache.kafka.raft.QuorumState)
>
> [2023-02-16 01:00:54,549] INFO [RaftManager nodeId=1] Completed transition
> to Leader(localId=1, epoch=10, epochStartOffset=28,
> highWatermark=Optional.empty, voterStates={1=ReplicaState(nodeId=1,
> endOffset=Optional.empty, lastFetchTimestamp=-1, lastCaughtUpTimestamp=-1,
> hasAcknowledgedLeader=true)}) (org.apache.kafka.raft.QuorumState)
>
> [2023-02-16 01:00:54,615] INFO [kafka-raft-outbound-request-thread]:
> Starting (kafka.raft.RaftSendThread)
>
> [2023-02-16 01:00:54,620] INFO [kafka-raft-io-thread]: Starting
> (kafka.raft.KafkaRaftManager$RaftIoThread)
>
> [2023-02-16 01:00:54,694] INFO [RaftManager nodeId=1] High watermark set to
> LogOffsetMetadata(offset=29,
> metadata=Optional[(segmentBaseOffset=0,relativePositionInSegment=3044)])
> for the first time for epoch 10 based on indexOfHw 0 and voters
> [ReplicaState(nodeId=1, endOffset=Optional[LogOffsetMetadata(offset=29,
> metadata=Optional[(segmentBaseOffset=0,relativePositionInSegment=3044)])],
> lastFetchTimestamp=-1, lastCaughtUpTimestamp=-1,
> hasAcknowledgedLeader=true)] (org.apache.kafka.raft.LeaderState)
>
> [2023-02-16 01:00:54,723] INFO [RaftManager nodeId=1] Registered the
> listener org.apache.kafka.image.loader.MetadataLoader@1633422834
> (org.apache.kafka.raft.KafkaRaftClient)
>
> [2023-02-16 01:00:54,761] INFO [RaftManager nodeId=1] Registered the
> listener
>
> org.apache.kafka.controller.QuorumController$QuorumMetaLogListener@629125780
> (org.apache.kafka.raft.KafkaRaftClient)
>
> [2023-02-16 01:00:54,780] INFO [MetadataLoader 1] Publishing initial
> snapshot at offset 27 to SnapshotGenerator
> (org.apache.kafka.image.loader.MetadataLoader)
>
> [2023-02-16 01:00:54,798] INFO [ThrottledChannelReaper-Fetch]: Starting
> (kafka.server.ClientQuotaManager$ThrottledChannelReaper)
>
> [2023-02-16 01:00:54,799] INFO [ThrottledChannelReaper-Produce]: Starting
> (kafka.server.ClientQuotaManager$ThrottledChannelReaper)
>
> [2023-02-16 01:00:54,808] INFO [ThrottledChannelReaper-Request]: Starting
> (kafka.server.ClientQuotaManager$ThrottledChannelReaper)
>
> [2023-02-16 01:00:54,813] INFO [ThrottledChannelReaper-ControllerMutation]:
> Starting (kafka.server.ClientQuotaManager$ThrottledChannelReaper)
>
> [2023-02-16 01:00:54,849] INFO [ExpirationReaper-1-AlterAcls]: Starting
> (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
>
> [2023-02-16 01:00:54,865] INFO [SocketServer listenerType=CONTROLLER,
> nodeId=1] Enabling request processing. (kafka.network.SocketServer)
>
> [2023-02-16 01:00:54,874] INFO [BrokerServer id=1] Transition from SHUTDOWN
> to STARTING (kafka.server.BrokerServer)
>
> [2023-02-16 01:00:54,877] INFO [BrokerServer id=1] Starting broker
> (kafka.server.BrokerServer)
>
> [2023-02-16 01:00:54,918] INFO [ThrottledChannelReaper-Fetch]: Starting
> (kafka.server.ClientQuotaManager$ThrottledChannelReaper)
>
> [2023-02-16 01:00:54,918] INFO [ThrottledChannelReaper-Produce]: Starting
> (kafka.server.ClientQuotaManager$Throttle

Re: Some question about the kafka website language

[sunil chaudhari]

I  am just wondering even if you read document in your language, but
ultimately server.config and server.log will be in english only.. right?



On Tue, 31 Jan 2023 at 9:00 PM, Nengda Ouyang 
wrote:

> I see the kafka website only have language version. Why doesn't provide
> other language.
>

Re: Trouble running kafka-server-start.sh from quickstart example

[sunil chaudhari]

So did you provided server.properties properly?
Check if property file exists at location.
JAVA_HOME os availble?

Regards,
Sunil.

On Mon, 30 Jan 2023 at 4:47 AM, xiao cheng  wrote:

> Hi Sunil,
> bin/kafka-server-start.sh --help give me output of
>
> bin/kafka-server-start.sh --help
>
> [2023-01-30 00:14:32,122] INFO Registered kafka:type=kafka.Log4jController
> MBean (kafka.utils.Log4jControllerRegistration$)
>
> USAGE: java [options] kafka.Kafka server.properties [--override
> property=value]*
>
> Option   Description
>
> --   ---
>
> --override   Optional property that should override values set in
>
>server.properties file
>
> --versionPrint version information and exit.
>
>
>
>
> On Sat, Jan 28, 2023 at 12:58 AM sunil chaudhari <
> sunilmchaudhar...@gmail.com> wrote:
>
> > Please try executing..
> >
> > bin/kafka-server-start.sh —help
> >
> >
> >
> > On Fri, 27 Jan 2023 at 8:47 PM, xiao cheng 
> > wrote:
> >
> > > Hey all,
> > >
> > > I recently tried to run kafka locally on my macos again after a while
> (it
> > > used to work). I followed the quickstart guide from
> > > https://kafka.apache.org/quickstart.
> > >
> > > However, when I try to run it (bin/kafka-server-start.sh
> > > config/server.properties) again today I am seeing error message of
> > > [2023-01-27 15:39:11,900] ERROR Encountered an error while configuring
> > the
> > > connection, closing it. (kafka.network.DataPlaneAcceptor)
> > > java.net.SocketException: Invalid argument
> > > at java.base/sun.nio.ch.Net.setIntOption0(Native Method)
> > > at java.base/sun.nio.ch.Net.setSocketOption(Net.java:455)
> > > at java.base/sun.nio.ch.Net.setSocketOption(Net.java:393)
> > > at
> > > java.base/sun.nio.ch
> > > .SocketChannelImpl.setOption(SocketChannelImpl.java:280)
> > > at
> > > java.base/sun.nio.ch
> > > .SocketAdaptor.setBooleanOption(SocketAdaptor.java:237)
> > > at java.base/sun.nio.ch
> > > .SocketAdaptor.setTcpNoDelay(SocketAdaptor.java:273)
> > > at
> > >
> > >
> >
> kafka.network.Acceptor.configureAcceptedSocketChannel(SocketServer.scala:759)
> > > at kafka.network.Acceptor.accept(SocketServer.scala:737)
> > > at kafka.network.Acceptor.acceptNewConnections(SocketServer.scala:703)
> > > at kafka.network.Acceptor.run(SocketServer.scala:645)
> > > at java.base/java.lang.Thread.run(Thread.java:833)
> > >
> > > I tried:
> > > 1. clean up kafka logs from tmp folder
> > > 2. downloaded the latest kafka tar ball kafka_2.13-3.3.1.tgz
> > > <https://archive.apache.org/dist/kafka/3.3.1/kafka_2.13-3.3.1.tgz> 3.
> > > change the client port from 2181 to some other value
> > > 4. Double check to make sure no other process is using the port
> > >
> > > zookeeper comes up fine and with below env:
> > > [2023-01-27 15:37:06,512] INFO Server environment:host.name
> =10.1.30.184
> > (
> > > org.apache.zookeeper.server.ZooKeeperServer)
> > > [2023-01-27 15:37:06,512] INFO Server environment:java.version=17.0.5 (
> > > org.apache.zookeeper.server.ZooKeeperServer)
> > > [2023-01-27 15:37:06,512] INFO Server environment:java.vendor=Homebrew
> (
> > > org.apache.zookeeper.server.ZooKeeperServer)
> > > [2023-01-27 15:37:06,512] INFO Server environment:java.home
> > > =/opt/homebrew/Cellar/openjdk@17
> > /17.0.5/libexec/openjdk.jdk/Contents/Home
> > > (
> > > org.apache.zookeeper.server.ZooKeeperServer)
> > > Please advise on things I need to do to get this fixed. I am almost
> > certain
> > > it is my machine issue but I cannot figure out why...
> > >
> >
>

Re: Trouble running kafka-server-start.sh from quickstart example

[sunil chaudhari]

Please try executing..

bin/kafka-server-start.sh —help



On Fri, 27 Jan 2023 at 8:47 PM, xiao cheng  wrote:

> Hey all,
>
> I recently tried to run kafka locally on my macos again after a while (it
> used to work). I followed the quickstart guide from
> https://kafka.apache.org/quickstart.
>
> However, when I try to run it (bin/kafka-server-start.sh
> config/server.properties) again today I am seeing error message of
> [2023-01-27 15:39:11,900] ERROR Encountered an error while configuring the
> connection, closing it. (kafka.network.DataPlaneAcceptor)
> java.net.SocketException: Invalid argument
> at java.base/sun.nio.ch.Net.setIntOption0(Native Method)
> at java.base/sun.nio.ch.Net.setSocketOption(Net.java:455)
> at java.base/sun.nio.ch.Net.setSocketOption(Net.java:393)
> at
> java.base/sun.nio.ch
> .SocketChannelImpl.setOption(SocketChannelImpl.java:280)
> at
> java.base/sun.nio.ch
> .SocketAdaptor.setBooleanOption(SocketAdaptor.java:237)
> at java.base/sun.nio.ch
> .SocketAdaptor.setTcpNoDelay(SocketAdaptor.java:273)
> at
>
> kafka.network.Acceptor.configureAcceptedSocketChannel(SocketServer.scala:759)
> at kafka.network.Acceptor.accept(SocketServer.scala:737)
> at kafka.network.Acceptor.acceptNewConnections(SocketServer.scala:703)
> at kafka.network.Acceptor.run(SocketServer.scala:645)
> at java.base/java.lang.Thread.run(Thread.java:833)
>
> I tried:
> 1. clean up kafka logs from tmp folder
> 2. downloaded the latest kafka tar ball kafka_2.13-3.3.1.tgz
>  3.
> change the client port from 2181 to some other value
> 4. Double check to make sure no other process is using the port
>
> zookeeper comes up fine and with below env:
> [2023-01-27 15:37:06,512] INFO Server environment:host.name=10.1.30.184 (
> org.apache.zookeeper.server.ZooKeeperServer)
> [2023-01-27 15:37:06,512] INFO Server environment:java.version=17.0.5 (
> org.apache.zookeeper.server.ZooKeeperServer)
> [2023-01-27 15:37:06,512] INFO Server environment:java.vendor=Homebrew (
> org.apache.zookeeper.server.ZooKeeperServer)
> [2023-01-27 15:37:06,512] INFO Server environment:java.home
> =/opt/homebrew/Cellar/openjdk@17/17.0.5/libexec/openjdk.jdk/Contents/Home
> (
> org.apache.zookeeper.server.ZooKeeperServer)
> Please advise on things I need to do to get this fixed. I am almost certain
> it is my machine issue but I cannot figure out why...
>

Re: Supported Kafka/Zookeeper Version with ELK 8.4.3

[sunil chaudhari]

Hi All,
Well this question can be asked on discuss.elastic.co
However I would like to answer this.
I am sure you need compatibility with logstash.
You can go through this document for logstash-kafka integration
https://www.elastic.co/guide/en/logstash/current/plugins-integrations-kafka.html

https://cwiki.apache.org/confluence/plugins/servlet/mobile?contentId=73638194#content/view/73638194


Please feel free to contact me if you have any questions or concerns on
this matter.

Cheers,
Sunil Chaudhari.


On Mon, 5 Dec 2022 at 11:33 PM, Colin McCabe  wrote:

> Hi,
>
> Sorry, we do not develop ELK. In fact, I'm not sure what that acronym
> refers to. I would suggest checking in with support for that product /
> project, since it is not part of Apache Kafka.
>
> best,
> Colin
>
>
> On Fri, Oct 28, 2022, at 06:23, Kumar, Sudip wrote:
> > Hi Team,
> >
> > We are still waiting for the reply. Please update we must know what
> version of Kafka is compatible with ELK 8.4 version.
> >
> > Still, I can see no one replied on user and Dev community portal
> >
> >
> >
> >
> > Thanks
> > Sudip
> >
> >
> > *From:* Kumar, Sudip
> > *Sent:* Monday, October 17, 2022 5:23 PM
> > *To:* users@kafka.apache.org; d...@kafka.apache.org
> > *Cc:* Rajendra Bangal, Nikhil ;
> Verma, Harshit ; Verma, Deepak Kumar <
> deepak-kumar.ve...@capgemini.com>; Arkal, Dinesh Balaji <
> dinesh-balaji.ar...@capgemini.com>; Saurabh, Shobhit <
> shobhit.saur...@capgemini.com>
> > *Subject:* Supported Kafka/Zookeeper Version with ELK 8.4.3
> > *Importance:* High
> >
> >
> > Hi Kafka Team,
> >
> > Currently we are planning to upgrade ELK 7.16 to 8.4.3 version. In our
> ecosystem we are using Kafka as middleware which is ingesting data coming
> from different sources where publisher (Logstash shipper) publishing data
> in different Kafka Topics and subscriber (Logstash indexer) consuming the
> data.
> >
> > We have an integration of ELK 7.16 with Kafka V2.5.1 and zookeeper
> 3.5.8. Please suggest if we upgrade on ELK 8.4.3 version which Kafka and
> Zookeeper version will be supported? Provide us handful documents.
> >
> > Let me know if you any further questions.
> >
> > *Thanks*
> > *Sudip Kumar*
> > *Capgemini-India *
> >
> >
> > This message contains information that may be privileged or confidential
> and is the property of the Capgemini Group. It is intended only for the
> person to whom it is addressed. If you are not the intended recipient, you
> are not authorized to read, print, retain, copy, disseminate, distribute,
> or use this message or any part thereof. If you receive this message in
> error, please notify the sender immediately and delete all copies of this
> message.
>

Re: Hi! An observation about your site and software!

[sunil chaudhari]

I like the way you written it without fullstop…. Lol

On Mon, 28 Nov 2022 at 8:48 PM, Schehrzade 
wrote:

> I like the author Kafka and I was so impressed someone had written code or
> whatever I don’t know because I’m not from this country and I don’t know
> stuff about science and all but it was really cool and even the logo and
> font and then I started to watch your video and the idea was so
> creative and interesting and sounded so useful and I thought aw how cute
> some like little start up thing and then I went to see who uses it and I
> saw Salesforce and I’m in customer service and I love Salesforce s much
> and I’m sooo happy with it it makes things so much more convenient and then
> for me to realize I was like one degree of separation away from and not
> even that because I’m literally using Kafka atm at work in a way that’s
> insane but I felt so honoured and happy and then I saw 80% of the biggest
> whatever’s we’re using it and man I was so emotional that someone was
> keeping kafkas name alive and had kept him remembered and he was just this
> cute sad little bug person and you guys just made him rule everywhere in
> the most random industry and you made everyone remember that he mattered
> that’s so sweet and you did it so beautifully and meticulously and it’s
> s satisfying even just to see the website and fonts and colours and
> everything let alone the ideas and it’s all so perfect to Kafka too right
> like I feel like ve would have liked the aesthetic were he alive today so
> amazing honestly THANK YOU  👏🏽
>

Re: Kafka 3.3.1, unable to start services in cygwin

[sunil chaudhari]

Hi,
Are you running on windows?
If Yes, please check documentation once.
There are different executables for windows under /bin
Also make sure you are using correct version of jdk for windows.

Regards,
Sunil.

On Mon, 21 Nov 2022 at 2:26 AM, ravi r  wrote:

> I downloaded
>
> kafka_2.13-3.3.1.tgz
>
> and am unable to start services from a cygwin bash shell on my desktop
> using zookeeper. Relevant error
>
> $ zookeeper-server-start.sh $KAFKA/config/zookeeper.properties
> [0.003s][error][logging] Invalid decorator
> '/cygwin64/usr/local/kafka/kafka_2.13-3.3.1/logs/zookeeper-gc.log'.
> Invalid -Xlog option
>
> '-Xlog:gc*:file=C:/cygwin64/usr/local/kafka/kafka_2.13-3.3.1/logs/zookeeper-gc.log:time,tags:filecount=10,filesize=100M',
> see error log for
>  details.
> Error: Could not create the Java Virtual Machine.
> Error: A fatal exception has occurred. Program will exit.
>

Re: Leader election strategy

[sunil chaudhari]

Hi,
Use confluent. It has auto balancing feature.
You dont need to do these manual things.


On Tue, 15 Nov 2022 at 7:22 PM, Pierre Coquentin 
wrote:

> Hello Luke, and thank you for your answer.
> What I would have hoped for is something more automatic, something that
> will spread the load when a Kafka broker goes down without any human
> intervention. The reassign script is a bit complicated, you need to
> generate the topics and partitions list, then get the current assignment
> and rework it to force a new leader.
>
> On Tue, Nov 15, 2022 at 5:18 AM Luke Chen  wrote:
>
> > Hi Pierre,
> >
> > Try using kafka-reassign-partitions.sh to reassign partitions to
> different
> > replicas you like.
> > ref:  https://kafka.apache.org/documentation/#basic_ops_automigrate
> >
> > Luke
> >
> > On Mon, Nov 14, 2022 at 3:55 PM Pierre Coquentin <
> > pierre.coquen...@gmail.com>
> > wrote:
> >
> > > Hello,
> > > We have a Kafka cluster (2.4.1) with a replication factor of 3. I
> notice
> > > when we stop a broker that only one broker takes all the load from the
> > > missing broker and becomes the leader to all partitions.
> > > I would have thought that Kafka would split the load evenly among the
> > > remaining brokers.
> > >
> > > So if I have this kind of configuration
> > > Topic: test
> > > Partition 0 - Leader: 1 - Replicas: 1,2,3 - Isr: 1,2,3
> > > Partition 1 - Leader: 2 - Replicas: 2,3,1 - Isr: 1,2,3
> > > Partition 2 - Leader: 3 - Replicas: 3,1,2 - Isr: 1,2,3
> > > Partition 3 - Leader: 1 - Replicas: 1,2,3 - Isr: 1,2,3
> > > Partition 4 - Leader: 2 - Replicas: 2,3,1 - Isr: 1,2,3
> > > Partition 5 - Leader: 3 - Replicas: 3,1,2 - Isr: 1,2,3
> > >
> > > If I stop broker 1, I want something like this (load is split evenly
> > among
> > > broker 2 and 3):
> > > Topic: test
> > > Partition 0 - Leader: 2 - Replicas: 1,2,3 - Isr: 2,3
> > > Partition 1 - Leader: 2 - Replicas: 2,3,1 - Isr: 2,3
> > > Partition 2 - Leader: 3 - Replicas: 3,1,2 - Isr: 2,3
> > > Partition 3 - Leader: 3 - Replicas: 1,2,3 - Isr: 2,3
> > > Partition 4 - Leader: 2 - Replicas: 2,3,1 - Isr: 2,3
> > > Partition 5 - Leader: 3 - Replicas: 3,1,2 - Isr: 2,3
> > >
> > > What I observe is currently this (broker 2 takes all the load from
> broker
> > > 1):
> > > Partition 0 - Leader: 2 - Replicas: 1,2,3 - Isr: 2,3
> > > Partition 1 - Leader: 2 - Replicas: 2,3,1 - Isr: 2,3
> > > Partition 2 - Leader: 3 - Replicas: 3,1,2 - Isr: 2,3
> > > Partition 3 - Leader: 2 - Replicas: 1,2,3 - Isr: 2,3
> > > Partition 4 - Leader: 2 - Replicas: 2,3,1 - Isr: 2,3
> > > Partition 5 - Leader: 3 - Replicas: 3,1,2 - Isr: 2,3
> > >
> > > My concern here is that at all times, a broker should not exceed 50% of
> > its
> > > network bandwidth which could be a problem in my case.
> > > Is there a way to change this behavior (manually by forcing a leader,
> > > programmatically, or by configuration)?
> > > From my understanding, the script kafka-leader-election.sh allows only
> to
> > > set the preferred (the first in the list of replicas) or uncleaned
> > > (replicas not in sync can become a leader).
> > > Regards,
> > >
> > > Pierre
> > >
> >
>

Re: Balancing traffic between multiple directories

[sunil chaudhari]

Hi Lehar,
You are right. There is no better way in open source Kafka.
However confluent has something called as Auto Rebalancing feature.
Can you check if there is free version with this feature?

It start balancing of  brokers automatically when it see there is uneven
distribution of partitions.

Regards,
Sunil.
On Wed, 26 Oct 2022 at 12:03 PM, Lehar Jain 
wrote:

> Hey Andrew,
>
> Thanks for the reply. Currently, we are using the same method as you
> described. Wanted to make sure if there is a better way.
>
> It seems there isn't currently. So we will keep using this only.
>
> On Tue, Oct 25, 2022 at 7:23 PM Andrew Grant 
> wrote:
>
> > Hey Lehar,
> >
> >
> > I don’t think there’s a way to control this during topic creation. I just
> > took a look through
> >
> >
> https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/admin/AdminUtils.scala
> > and it does appear partition assignment does not account for each
> broker’s
> > different log directories. I also took a look at the kafka-topics.sh
> script
> > and it has a --replica-assignment argument but that looks to only allow
> > specifying brokers. During topic creation, once a replica has been
> chosen I
> > think we then choose the directory with the fewest number of partitions -
> > see
> >
> >
> https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/log/LogManager.scala#L1192
> >
> >
> > What I think you can do is move existing partitions around with the
> > kafka-reassign-partitions.sh script. From running the command locally:
> >
> >
> > --reassignment-json-file  >
> >   manual assignment json file path>   reassignment configurationThe
> > format
> >
> >   to use is -
> >
> > {"partitions":
> >
> > [{"topic": "foo",
> >
> >   "partition": 1,
> >
> >   "replicas": [1,2,3],
> >
> >   "log_dirs":
> > ["dir1","dir2","dir3"]
> >
> >   }],
> >
> > "version":1
> >
> > }
> >
> > Note that "log_dirs" is optional.
> > When
> >
> >   it is specified, its length
> must
> >
> >   equal the length of the
> replicas
> >
> >   list. The value in this list
> can
> > be
> >
> >   either "any" or the absolution
> > path
> >
> >   of the log directory on the
> > broker.
> >
> >   If absolute log directory path
> is
> >
> >   specified, the replica will be
> > moved
> >
> >   to the specified log directory
> on
> >
> >   the broker.
> >
> >
> > There’s the log_dirs field you can use in the JSON file to move
> partitions
> > between directories.
> >
> >
> > Hope that helps a bit.
> >
> >
> > Andrew
> >
> > On Tue, Oct 25, 2022 at 6:56 AM Lehar Jain 
> > wrote:
> >
> > > Hey,
> > >
> > > We run Kafka brokers with multiple log directories. I wanted to know
> how
> > > Kafka balances traffic between various directories. Can we have our own
> > > strategy to distribute different partitions to different directories.
> As
> > > currently, we are facing an imbalance in sizes of the aforementioned
> > > directories, some directories have a lot of empty space whereas others
> > are
> > > getting filled quickly.
> > >
> > >
> > > Regards
> > >
> >
>

Re: SSL configuration in Apache Kafka

[sunil chaudhari]

You can try two things.
Instead of localhost, can you publish the kafka service on Hostname?

Since ur client.auth is none, can you try removing keystore from the
producer?

Regards,
Sunil.

On Fri, 7 Oct 2022 at 2:56 PM, Namita Jaokar 
wrote:

> Hi All,
>
> I am trying to enable SSL in my kafka broker and have generated required
> certificates following the steps provided in the official documentation.
> I have also made changes in Kafka's server.properties to enable SSL.  On
> starting zookeeper followed by kafka server startup, There are no error in
> the logs in either of them.
> However, When I configured my spring-boot producer application and tried
> publishing the message, I am getting below error in the producer
> application
>
> *Producer application error log:*
>
> [Producer clientId=producer-1] Bootstrap broker localhost:9093 (id: -1
> rack: null) disconnected
>
> [Producer clientId=producer-1] Connection to node -1 (localhost/
> 127.0.0.1:9093) could not be established. Broker may not be available.
>
>
>
> Simultaneously, The kafka server as well gives below error
>
>
> *Kafka server log:*
>
>  INFO [SocketServer listenerType=ZK_BROKER, nodeId=0] Failed authentication
> with /127.0.0.1 (SSL handshake failed)
> (org.apache.kafka.common.network.Selector)
>
>
> I have added below properties in my producer application for SSL
> Configuration:
>
> spring.kafka.bootstrap-servers: localhost:9092,localhost:9093
> spring.kafka.properties.ssl.endpoint.identification.algorithm:
> spring.kafka.properties.ssl.protocol: SSL
> spring.kafka.properties.ssl.trust-store-location:
> pathtocert/kafka.server.truststore.jks
> spring.kafka.properties.ssl.trust-store-password: mypassword
> spring.kafka.properties.ssl.key-store-location:
> pathtocert/kafka.server.keystore.jks
> spring.kafka.properties.ssl.key-store-password: mypassword
> spring.kafka.properties.ssl.key-password: mypassword
>
> Below are my server.properties configurations of kafka server to enable
> SSL.
>
> listeners=PLAINTEXT://:9092,SSL://localhost:9093
> advertised.listeners=PLAINTEXT://localhost:9092,SSL://localhost:9093
> ssl.endpoint.identification.algorithm=
> ssl.client.auth=none
> ssl.enabled.protocols=TLSv1.2,TLSv1.1,TLSv1.3,TLSv1
> ssl.keystore.type=jks
> ssl.truststore.type=jks
> ssl.keystore.location=pathtocert/kafka.server.keystore.jks
> ssl.keystore.password=mypassword
> ssl.key.password=password
> ssl.truststore.location=pathtocert/kafka.server.truststore.jks
> ssl.truststore.password=mypassword
> security.inter.broker.protocol = SSL
> listener.security.protocol.map=PLAINTEXT:PLAINTEXT,SSL:SSL
>
>
>
> Can someone please help me resolve the issue.
>
>
> Thanks & Regards,
>
> Namita
>

Re: Apache Kafka Connect

[sunil chaudhari]

Hi Namita and Pasi,
Logstash as middlemen is good if and only if:
1. You dont need buffer in between and you are Ok with tight coupling
between source and destination.
2. There are sufficient number of logstash servers available in case of
large data.
In case of logstash, source and destinations should be always available.
If MongoDB is not available, there are chances of mossing data. In case of
kafka as broker, you have chances of recovery of failed ingestions. And
there many other benefits of using kafka.

Regards,
Sunil.

On Sat, 1 Oct 2022 at 4:27 PM, Pasi Hiltunen  wrote:

> Hi Namita,
>
> You can use Logstash as middleman:
>
> https://www.elastic.co/guide/en/logstash/current/plugins-inputs-elasticsearch.html
> https://www.elastic.co/guide/en/logstash/current/plugins-outputs-kafka.html
>
> -pasi
>
>
> From: Namita Jaokar 
> Date: Friday, 30. September 2022 at 19.56
> To: users@kafka.apache.org 
> Subject: Apache Kafka Connect
> Hi All,
>
> I have a scenario where I want to send data from elasticsearch to Mongodb
> through kafka and while researching I came across Kafka connect.
>
> Through Kafka connect is it possible to have the elasticsearch as a source
> connector that will send data/messages from elasticsearch to kafka? I came
> across Kafka sink connector which can receive messages/Data from the Kafka
> server through topics.
>
> In case of Mongodb sink connector,  what would be the behaviour if I have
> larger data that is greater than the maximum size of document in mongodb
> which is 16MB. Is there a way to handle this?
>
> Also, Is there a prerequisite to have docker setup before installing the
> connectors
>
> Thanks & Regards,
> Namita
>

Re: Apache Kafka Connect

[sunil chaudhari]

Hi Namita,
For Moving data from Elasticsearch to Kafka you need Elasticsearch Source
connector. I guess this is not supported connector. You may have to rely on
some community developed connector where you may not get instant support.

https://github.com/DarioBalinzo/kafka-connect-elasticsearch-source/tree/1.4.2

Check the supported version of ES before you start.
If that doesn’t suit your case then you can explore logstash with input as
ES and output as  Kafka.
>From Kafka to MongoDB you need Sink connector.


Regards,
Sunil.

KSQL is lagging

[sunil chaudhari]

Hi,
I have one topic with 300 partitions.
I have 4 KSQL instances with 8 threads each on 8 core machine.
Topic has lag of around million records.
Can I increase number of threads equal to number of partitions so hat lag
will be reduced?
Or I have to reduce partitions to match total number of threads?

please advise how do I reduce lag on source topic.
Thanks

Re: Consumer Lag-Apache_kafka_JMX metrics

[sunil chaudhari]

You can try this, if you know what prometheus and how its installed
configured.

https://www.confluent.io/blog/monitor-kafka-clusters-with-prometheus-grafana-and-confluent/


On Wed, 17 Aug 2022 at 2:25 AM, Peter Bukowinski  wrote:

> Richard recently answered your query. A kafka cluster does not keep track
> of lag on behalf of external consumers and it therefore is not available in
> JMX. This is why tools like Burrow were written. The java kafka consumer
> published consumer lag metrics, and perhaps some other third-party clients
> do, as well.
>
> > On Aug 16, 2022, at 12:05 PM, Kafka Life  wrote:
> >
> > Hello Experts, Any info or pointers on my query please.
> >
> >
> >
> > On Mon, Aug 15, 2022 at 11:36 PM Kafka Life 
> wrote:
> >
> >> Dear Kafka Experts
> >> we need to monitor the consumer lag in kafka clusters 2.5.1 and 2.8.0
> >> versions of kafka in Grafana.
> >>
> >> 1/ What is the correct path for JMX metrics to evaluate Consumer Lag in
> >> kafka cluster.
> >>
> >> 2/ I had thought it is FetcherLag  but it looks like it is not as per
> the
> >> link below.
> >>
> >>
> https://www.instaclustr.com/support/documentation/kafka/monitoring-information/fetcher-lag-metrics/#:~:text=Aggregated%20Fetcher%20Consumer%20Lag%20This%20metric%20aggregates%20lag,in%20sync%20with%20partitions%20that%20it%20is%20replicating
> >> .
> >>
> >> Could one of you experts please guide on which JMX i should use for
> >> consumer lag apart from kafka burrow or such intermediate tools
> >>
> >> Thanking you in advance
> >>
> >>
>
>

Re: Need help on upgrade kafka from 2.7.0 to 3.2.1

[sunil chaudhari]

Hi Sowjanya,
I am not technical support from Kafka, but I can help you in this.
Recently I upgraded one of the confluent version, so I will try to help you.
Please let me know what exactly you need.

On Tue, 16 Aug 2022 at 7:43 PM, sowjanya reddy <
sowjanyabairapured...@gmail.com> wrote:

> I Team,
>
> I need contact details of technical support on kafka upgrade.
>
>
>
> Thanks and regards,
> Sowjanya.
>

What happens when kafka gracefully stopped

[sunil chaudhari]

Hi all,
Can anyone tell me in detail what exactly happens when I stop kafka using
kafka-stop script in /bin?
Doest it releases session or files or what cleanup operations?


Regards,
Sunil.

Re: What does Apache Kafka do?

[sunil chaudhari]

Hi
Good evening from India😃
I am using Kafka(confluent Platform) since long time.
Its actually a message broker in simple term which can persist data on disk
for short time in days or weeks.
It acts like a buffer in between message producer and message consumer.
Kafka reduces loading of consumers by acting as a buffer in real time data
processing apps.
In short.. you can achieve lose coupling between producers and consumers
using kafka. (Asynchronous communication)
And there are lot of other features like Rest Proxy, connectors, Schema
Registry etc. you need to invest time in reading for good returns in long
term 😀

I hope it helps.

Cheers,
Sunil.



On Wed, 18 May 2022 at 8:01 PM, Turritopsis Dohrnii Teo En Ming <
ceo.teo.en.m...@gmail.com> wrote:

> Subject: What does Apache Kafka do?
>
> Good day from Singapore,
>
> I notice my company/organization is using Apache Kafka. What does it do?
>
> Just being curious.
>
> Regards,
>
> Mr. Turritopsis Dohrnii Teo En Ming
> Targeted Individual in Singapore
> 18 May 2022 Wed
>

Re: Unexpected Rebalances ，Any tips on APIs or debug techniques to figure out rebalance causes?

[sunil chaudhari]

Rebalancing happens mainly because of these reasons:

You restart consumer
Consumer host is not reachable
You stop consumer

All above situations are fine when you have sufficient number of
consumers(threads) to read from the available partitions and all consumers
are logically distributed across multiple consumer groups.

In case you have grouped many consumers into same group reading from
multiple topics then it may cause frequent rebalancing.
Even if one consumer has availability issue whole group will go for
rebalancing.

Recently we gone thru same problem and what we did is, whole group is
devided in many groups logically.
Each group is associated with specific topic and consumers are distributed
across those groups.

Now in case of consumer fails, only that small group is rebalanced not
others.

I hope i have answered your question with my limited knowledge😊

Corrections are welcome!


Regards,
Sunil.


On Thu, 28 Apr 2022 at 8:42 AM, Luke Chen  wrote:

> 1. is it  correctly the idea about  B ?   i think it maybe the major factor
> for rebalance in my case  ,because  consuming data is slowly.
> => Looks like so, but we cannot confirm that because we don't have other
> information. You should check the consumer log to see why the consumer
> leave group.
>
> 2. i have do an experiment  for  assert  B. but not reproduce
>  A：consumer fast or slow  can't trigger it
> You should also adjust the heartbeat interval to allow the heartbeat detect
> the poll expiration.
> You can refer to this test:
>
> https://github.com/apache/kafka/blob/trunk/core/src/test/scala/integration/kafka/api/PlaintextConsumerTest.scala#L167
>
> 3. Any tips on APIs or debug techniques to figure out rebalance causes
> On server side, you can check log like this:
> "Preparing to rebalance group xxx ... (reason: y)
>
>
>
>  4. How can I trigger it manually ?
> => same as question 2
>
>  5. Is it a bad idea to have the same Consumer Group (Same ID) consuming
> from multiple topics ?
> => Depends on your use case, no good or bad.
>
>
> Thank you.
> Luke
>
>
> On Wed, Apr 27, 2022 at 11:58 PM 杨宝栓  wrote:
>
> >
> >
> > HI：
> >  We are seeing unexpected rebalances in golang consumers, described
> below.
> > 1. We have a topic with 36 partitions，and one consumer (lets name
> > it consumer1) consuming it.
> > 2. Run kafka  in Docker and configuration：We use defaults
> > 3. Consumer  consuming data is slowly about  cost 1s for  one
> > piece of data
> > 4. All the consumers for topic A are in the same group
> >5. The rebalances are intermittent and hard to reproduce.  We see
> > no obvious errors in the logs.
> > 6.  No matter how to change  the configuration that affects
> > rebalance ，it always rebalance
> > The configuration that affects rebalance below:
> >  max.poll.interval.ms
> > max.poll.records
> > request.timeout.ms
> > session.timeout.ms
> >As far as I am concerned,  conditions of  the rebalance contains：
> >   a consumer is considered DEAD by the group coordinator.
> >   A.  when the consumer is busy, which means that no
> > heartbeats has been sent in the meanwhile by the consumer to the group
> > coordinator within the configured session interval
> >   B.  when the consumer is slowly with a long-running
> > processing, which means that  interval of  poll() is too long  within the
> > configured max.poll.interval.ms。
> >   question：
> >1. is it  correctly the idea about  B ?   i think it maybe the
> > major factor for rebalance in my case  ,because  consuming data is
> slowly.
> >2. i have do an experiment  for  assert  B. but not reproduce
> >  A：consumer fast or slow  can't trigger it
> >3. Any tips on APIs or debug techniques to figure out rebalance
> > causes
> >   4. How can I trigger it manually ?
> >5. Is it a bad idea to have the same Consumer Group (Same ID)
> > consuming from multiple topics ?
> >
> >
> >
> >
> >
> >
>

Re: About the order of messages when a single consumer is consuming messages from multiple partitions

[sunil chaudhari]

hi Roger,
What consumer u r using?
Is there a chance to mention consumer threads?

Example: logstash kafka consumer has configurable number of threads under
each consumer instance. That may help up to some extent.


Regards,
Sunil.
On Thu, 6 Jan 2022 at 7:27 PM, Roger Kasinsky 
wrote:

> Hi Luke,
>
> > The solution I can think of is to create only one partition for the
> topic.
>
> That would work, but then I lose the benefits of the partitions.
>
> > Or you can create 4 consumers in one group, to consume from 4 partitions.
> That works, too.
>
> That does not work, because I need only one consumer receiving all the
> messages in the same order on every run.
>
> Hi Suni,
>
> > Why dont you provide new name to consumer group each time you restart
> your consumer? This new consumer group will not conflict with the earlier
> one and it will be treated as new consumer thread next time to get all
> messages again.
>
> That does not solve my problem, which is to have the same consumer getting
> all the topic messages in the same order. I'm not worried about conflicts.
> In fact, I want the exact same consumer to run twice in a row. Renaming the
> consumer group does not help with anything related to message order.
>
> Thanks!
>
> -R
>
>
>
> On Thu, Jan 6, 2022 at 12:21 AM sunil chaudhari <
> sunilmchaudhar...@gmail.com>
> wrote:
>
> > hi,
> > Why dont you provide new name to consumer group each time you restart
> your
> > consumer?
> > This new consumer group will not conflict with the earlier one and it
> will
> > be treated as new consumer thread next time to get all messages again.
> >
> >
> > Regards,
> > Sunil.
> >
> > On Wed, 5 Jan 2022 at 10:45 PM, Roger Kasinsky  >
> > wrote:
> >
> > > Hi,
> > >
> > > I have a topic divided into 4 partitions. I have a consumer that needs
> to
> > > consume all messages from the topic (all messages from all 4
> partitions).
> > > So to do that I have this consumer sitting by itself in its own
> consumer
> > > group. I'm not committing any offsets, because I want to read all
> > messages
> > > again on every restart of the consumer.
> > >
> > > *Question:* *Can anything be said about the order of the messages
> > consumed
> > > by my consumer? Is there a way to enforce the same order of messages
> for
> > > every restart of my consumer?*
> > >
> > > Thanks!
> > >
> > > -R
> > >
> >
>

Re: About the order of messages when a single consumer is consuming messages from multiple partitions

[sunil chaudhari]

hi,
Why dont you provide new name to consumer group each time you restart your
consumer?
This new consumer group will not conflict with the earlier one and it will
be treated as new consumer thread next time to get all messages again.


Regards,
Sunil.

On Wed, 5 Jan 2022 at 10:45 PM, Roger Kasinsky 
wrote:

> Hi,
>
> I have a topic divided into 4 partitions. I have a consumer that needs to
> consume all messages from the topic (all messages from all 4 partitions).
> So to do that I have this consumer sitting by itself in its own consumer
> group. I'm not committing any offsets, because I want to read all messages
> again on every restart of the consumer.
>
> *Question:* *Can anything be said about the order of the messages consumed
> by my consumer? Is there a way to enforce the same order of messages for
> every restart of my consumer?*
>
> Thanks!
>
> -R
>

Re: Kafka : High avaialability settings

[sunil chaudhari]

Hi,
You can try reducing min.insynch replicas to 1



On Tue, 9 Nov 2021 at 1:56 AM, Kafka Life  wrote:

> Dear Kafka experts
>
> i have a 10 broker kafka cluster with all topics having replication factor
> as 3 and partition 50
>
> min.in.synch replicas is 2.
>
>
> One broker went down for a hardware failure, but many applications
> complained they are not able to produce /consume messages.
>
> I request you to please suggest, how do i overcome this problem and make
> kafka high available even during broker being down or during rolling
> restarts.
>
> IS there a configuration at a topic level i can set it up to have new
> partition created in active and running brokers when a node is down ?
> i read through ack -0/1/all to be set at application /producer end .But
> applications are not ready to change ack all .
> Can you please suggest .
>
> many thanks in advance
>

Re: Ensuring that the message is persisted after acknowledgement

[sunil chaudhari]

Hi Kunal,
This article may help you.

https://betterprogramming.pub/kafka-acks-explained-c0515b3b707e


Cheers,
Sunil.

On Fri, 20 Aug 2021 at 8:11 PM, Kunal Goyal 
wrote:

> Hello,
>
> We are exploring using Kafka for our application. Our requirement is that
> once we write some messages to Kafka, it should be guaranteed that the
> messages are persisted to disk.
> We found this
> <
> https://www.quora.com/Does-Kafka-sync-data-to-disk-asynchronously-like-Redis-does
> >
> article which says that a Kafka broker acknowledges a record after it has
> written the record to the buffer of the I/O device; it does not issue an
> explicit fsync operation nor does it wait for the OS to confirm that the
> data has been written. Is this statement true for the current
> implementation? If so, is there any way in which we can ensure fsync is
> called before acknowledgement of messages?
> Any help would be appreciated.
>
> --
>
> Thanks & Regards
>
> Kunal Goyal
>

Re: Advertised.listeners to be used behind VIP Load Balancer

[sunil chaudhari]

Honestly I didnt get this question.
Please elaborate.


On Sun, 4 Jul 2021 at 5:34 PM, M. Manna  wrote:

> Hello,
>
> Is it currently possible to use a single endpoint for advertised.listeners,
> which is in front of all my brokers? the flow for example
>
> broker1-->| V
> broker2-->| I
> broker3-->| P
>
> I was under the impression that I can reach any broker I wanted (assuming
> that broker is 100% guaranteed to be healthy at all times, but it can never
> be), and the broker coordination service will pass it onto the leader for
> that topicpartition. May be I am wrong.
>
> Has anyone done this successfully in production/staging/development?
>
> Regards,
>

Re: consumer group exit :explanation

[sunil chaudhari]

Hi,
There is something called as heartbet consumer thread.
This threads running on consumer keeps sending heartbeats at regular
interval as per the setting heartbeat.interval.ms. It keeps on telling
broker that I am very much alive.

https://docs.confluent.io/platform/current/installation/configuration/consumer-configs.html#consumerconfigs_heartbeat.interval.ms

There is another setting session.timeout.ms. If the time duration between 2
heartbeats exceeds the time limit set in this setting then broker removes
the client(consumer) from the group.

Please read above settings in documentation.

I hope you can co-relate this with your problem.

Regards,
Sunil.




On Sun, 4 Jul 2021 at 1:36 PM, Kafka Life  wrote:

> Dear kafka Experts
>
> Could one of you please help to explain what this below log in broker
> instance mean..and what scenarios it would occur when there is no change
> done .
>
>  INFO [GroupCoordinator 9610]: Member
>
> webhooks-retry-app-840d3107-833f-4908-90bc-ea8c394c07c3-StreamThread-2-consumer-f87c3b85-5aa1-40f5-a42f-58927421b89e
> in group webhooks-retry-app has failed, removing it from the group
> (kafka.coordinator.group.GroupCoordinator)
>
>
>  INFO [GroupCoordinator 9611]: Member
> cm.consumer.9-d65d39d3-703f-408b-bf4b-fbf087321d8c in group
> cm_group_apac_sy_cu_01 has failed, removing it from the group
> (kafka.coordinator.group.GroupCoordinator)
>
>
> Please help to explain .
>
> thanks
>

Re: kafka partition_assignment_strategy

[sunil chaudhari]

Hi Marcus,
Your first understanding is correct, provided each “consumer” means a
“consumer thread”

IMO, Second understanding about message distribution is incorrect because
there is something called as max poll records for each consumer. Its 500 by
default.
And the time between 2 polls is also very small in few milliseconds.
Thats why this is happening.

You may need to try this on a big number of messages so that other
partitions get assigned.

I tried my best to participate in discussion I am not expert though😊




On Thu, 1 Jul 2021 at 2:53 AM, Marcus Schäfer 
wrote:

> Hi,
>
> I was reading through the docs and would like to make use
> of kafka's dynamic partition assignment feature in a way
> that partitions gets distributed evenly across all consumers.
> To my understanding the RoundRobinAssignor would be sufficient
> and I left the problem of rebalance if a consumer dies aside
> for the moment.
>
> If I understood it correctly consumers are associated to at
> least one partition. This means if you have e.g 10 consumers
> you need at least 10 partitions such that the dynamic partition
> assignment is possible. Is this correct ?
>
> Given my understanding is correct and there are 1000 messages
> written to topic "A" and 10 consumers using the RoundRobinAssignor
> I would expect 100 messages to be consumed by each consumer.
> Is this still correct ?
>
> Based on this I setup kafka in Amazon using Amazon MSK and
> went through the manual setup procedure such that there is a
> chance to influence the default settings. I setup the cluster
> with a max of 100partitions.
>
> Next I wrote python code using the kafka-python module (v2.0.2)
> to proof if I understood the concept before starting the
> real implementation.
>
> My producer does:
>
> write.py
> 
> from kafka import KafkaProducer
>
> producer = KafkaProducer(
> bootstrap_servers='...9092'
> )
> for count in range(1,100):
> producer.send('cb-request', 'Message {0}'.format(count).encode())
>
> producer.flush()
> 
>
>
> My consumer does:
>
> read.py
> 
> from kafka import KafkaConsumer
> from kafka.coordinator.assignors.roundrobin import
> RoundRobinPartitionAssignor
>
> consumer = KafkaConsumer(
> 'cb-request',
> partition_assignment_strategy=[RoundRobinPartitionAssignor],
> auto_offset_reset='earliest',
> enable_auto_commit=False,
> bootstrap_servers='...9092',
> group_id='cb-request-group'
> )
>
> try:
> while(True):
> raw_messages = consumer.poll(timeout_ms=1)
> for topic_partition, message_list in raw_messages.items():
> print(topic_partition)
> print(message_list)
> consumer.commit()
> finally:
> consumer.close()
> 
>
>
> Now if I run this it works in the way that all produced messages
> are delivered to a consumer but it never scales. Meaning if I run
> multiple read.py's there is always only one that gets all the
> messages.
>
> The messages looks like this:
>
> ConsumerRecord(topic='cb-request', partition=0, offset=3964,
> timestamp=1625085873665, timestamp_type=0, key=None, value=b'Message 99',
> headers=[], checksum=None, serialized_key_size=-1,
> serialized_value_size=10, serialized_header_size=-1)
>
> and the partition assignment is always "partition=0"
>
> I think I'm doing it wrong or I misunderstood the concepts.
>
> Thus I'm writing here and kindly ask for help or to put me right
>
> Thanks much
>
> Regards,
> Marcus
> --
>  Public Key available via: https://keybase.io/marcus_schaefer/key.asc
>  keybase search marcus_schaefer
>  ---
>  Marcus Schäfer Am Unterösch 9
> 
>  Tel: +49 7562 905437   D-88316 Isny / Rohrdorf
>  Germany
>  ---
>

Re: ssl.client.auth=required (MTLS issue)

[sunil chaudhari]

I think similar issue is being discussed in other email thread.

On Fri, 25 Jun 2021 at 6:09 PM, meghna murthy  wrote:

> Hi Team ,
>
> When ssl.client.auth=required is set , Srver is sending Certificate
> request with DN with junk certificates to client . Server has to send what
> certificates we have deployed since its sending junk certificate handshake
> is failing. Attaching the screenshot of junk certificates sent in
> Certificate request.
>
> Are these default certificate sent by kafka ??
>
> I have attached pcap also for the same Please help in resolving issue.
> [image: image.png]
> Regards,
> Meghna
>

Re: please unsubscribe me

[sunil chaudhari]

Hehehe,
Banging on wrong door🤣🤣🤣 . I mean wrong email.

On Sat, 5 Jun 2021 at 8:29 PM, Jayashree Sanyal
 wrote:

> Please unsubscribe me . I have tried sending several mails with no success
> .
> =
> Please refer to https://northamerica.altran.com/email-disclaimer
> for important disclosures regarding this electronic communication.
> =
> This message contains information that may be privileged or confidential
> and is the property of the Capgemini Group. It is intended only for the
> person to whom it is addressed. If you are not the intended recipient, you
> are not authorized to read, print, retain, copy, disseminate, distribute,
> or use this message or any part thereof. If you receive this message in
> error, please notify the sender immediately and delete all copies of this
> message.
>

Re: In case of Max topic size is reached

[sunil chaudhari]

And when is that message segment closed? I mean what is the criteria to
close the message segment?
Can I change that criteria with configuration?


On Tue, 1 Jun 2021 at 11:36 PM, Ran Lupovich  wrote:

> They work simultaneously,  topic with cleanup policy  of DELETE , will
> clean old message older than the retention period and also deletes the
> oldest messages when retention bytes limit is breached,  notice this limit
> is for each partition in a topic and not total size of the topic, notice as
> well that deletion kicks off only on "closed" message segments
>
> בתאריך יום ג׳, 1 ביוני 2021, 20:57, מאת sunil chaudhari ‏<
> sunilmchaudhar...@gmail.com>:
>
> > Hi,
> > Suppose:
> > Maximum Topic size is set to 1 GB
> > Retention hours: 168
> >  What happens in case  topic size reaches the maximum size before 168
> > hours.
> > Will it delete few messages before its expiry though they are eligible to
> > stay for 168 hrs?
> >
> >
> > Regards,
> > Sunil.
> >
>

In case of Max topic size is reached

[sunil chaudhari]

Hi,
Suppose:
Maximum Topic size is set to 1 GB
Retention hours: 168
 What happens in case  topic size reaches the maximum size before 168 hours.
Will it delete few messages before its expiry though they are eligible to
stay for 168 hrs?


Regards,
Sunil.

Re: Issue using Https with elasticsearch source connector

[sunil chaudhari]

Yeah.
I am trying to add truststore in java keystore
Lets see

On Fri, 28 May 2021 at 5:40 PM, Ran Lupovich  wrote:

> Anyways you need to remmber it is a java application and you can pass many
> variables that not formally supported by the application as jvm input
> setting or in the connector OPTS, does not have experience with this
> specfic source connector did something similar as work arounf for the
> mongodb sink connector before they fixed the support for ssl, so I do
> beleive its possible , its matter of guess try and see , 😉 but i do
> believe its possible
>
> בתאריך יום ו׳, 28 במאי 2021, 15:05, מאת sunil chaudhari ‏<
> sunilmchaudhar...@gmail.com>:
>
> > Hello Ran,
> > Whatever link you have provided is the supported SINK connector.
> > It has all settings for SSL.
> >
> > The connector I am talking about is the Souce connector and its not
> > supported by Confluent.
> > If you see the documentation you will find that there is no setting for
> SSL
> > certs.
> >
> > https://github.com/DarioBalinzo/kafka-connect-elasticsearch-source
> >
> >
> > Thats where I am stuck.
> >
> >
> > Regards,
> > Sunil.
> >
> > On Fri, 28 May 2021 at 9:34 AM, Ran Lupovich 
> > wrote:
> >
> > >
> > >
> >
> name=elasticsearch-sinkconnector.class=io.confluent.connect.elasticsearch.ElasticsearchSinkConnectortasks.max=1topics=test-elasticsearch-sinkkey.ignore=trueconnection.url=
> > > https://localhost:9200type.name=kafka-connect
> > >
> > >
> >
> elastic.security.protocol=SSLelastic.https.ssl.keystore.location=/home/directory/elasticsearch-6.6.0/config/certs/keystore.jkselastic.https.ssl.keystore.password=asdfasdfelastic.https.ssl.key.password=asdfasdfelastic.https.ssl.keystore.type=JKSelastic.https.ssl.truststore.location=/home/directory/elasticsearch-6.6.0/config/certs/truststore.jkselastic.https.ssl.truststore.password=asdfasdfelastic.https.ssl.truststore.type=JKSelastic.https.ssl.protocol=TLS
> > >
> > >
> > > בתאריך יום ו׳, 28 במאי 2021, 07:03, מאת Ran Lupovich ‏<
> > > ranlupov...@gmail.com
> > > >:
> > >
> > > >
> > >
> >
> https://docs.confluent.io/kafka-connect-elasticsearch/current/security.html
> > > >
> > > > בתאריך יום ו׳, 28 במאי 2021, 07:00, מאת sunil chaudhari ‏<
> > > > sunilmchaudhar...@gmail.com>:
> > > >
> > > >> The configurations doesnt have provision for the truststore. Thats
> my
> > > >> concern.
> > > >>
> > > >>
> > > >> On Thu, 27 May 2021 at 10:47 PM, Ran Lupovich <
> ranlupov...@gmail.com>
> > > >> wrote:
> > > >>
> > > >> > For https connections you need to set truststore configuration
> > > >> parameters ,
> > > >> > giving it jks with password , the jks needs the contain the
> > certficate
> > > >> of
> > > >> > CA that is signing your certifcates
> > > >> >
> > > >> > בתאריך יום ה׳, 27 במאי 2021, 19:55, מאת sunil chaudhari ‏<
> > > >> > sunilmchaudhar...@gmail.com>:
> > > >> >
> > > >> > > Hi Ran,
> > > >> > > That problem is solved already.
> > > >> > > If you read complete thread and see that last problem is about
> > https
> > > >> > > connection.
> > > >> > >
> > > >> > >
> > > >> > > On Thu, 27 May 2021 at 8:01 PM, Ran Lupovich <
> > ranlupov...@gmail.com
> > > >
> > > >> > > wrote:
> > > >> > >
> > > >> > > > Try setting  es.port = "9200" without quotes?
> > > >> > > >
> > > >> > > > בתאריך יום ה׳, 27 במאי 2021, 04:21, מאת sunil chaudhari ‏<
> > > >> > > > sunilmchaudhar...@gmail.com>:
> > > >> > > >
> > > >> > > > > Hello team,
> > > >> > > > > Can anyone help me with this issue?
> > > >> > > > >
> > > >> > > > >
> > > >> > > > >
> > > >> > > >
> > > >> > >
> > > >> >
> > > >>
> > >
> >
> https://github.com/DarioBalinzo/kafka-connect-elasticsearch-source/issues/44
> > > >> > > > >
> > > >> > > > >
> > > >> > > > > Regards,
> > > >> > > > > Sunil.
> > > >> > > > >
> > > >> > > >
> > > >> > >
> > > >> >
> > > >>
> > > >
> > >
> >
>

Re: Issue using Https with elasticsearch source connector

[sunil chaudhari]

Hello Ran,
Whatever link you have provided is the supported SINK connector.
It has all settings for SSL.

The connector I am talking about is the Souce connector and its not
supported by Confluent.
If you see the documentation you will find that there is no setting for SSL
certs.

https://github.com/DarioBalinzo/kafka-connect-elasticsearch-source


Thats where I am stuck.


Regards,
Sunil.

On Fri, 28 May 2021 at 9:34 AM, Ran Lupovich  wrote:

>
> name=elasticsearch-sinkconnector.class=io.confluent.connect.elasticsearch.ElasticsearchSinkConnectortasks.max=1topics=test-elasticsearch-sinkkey.ignore=trueconnection.url=
> https://localhost:9200type.name=kafka-connect
>
> elastic.security.protocol=SSLelastic.https.ssl.keystore.location=/home/directory/elasticsearch-6.6.0/config/certs/keystore.jkselastic.https.ssl.keystore.password=asdfasdfelastic.https.ssl.key.password=asdfasdfelastic.https.ssl.keystore.type=JKSelastic.https.ssl.truststore.location=/home/directory/elasticsearch-6.6.0/config/certs/truststore.jkselastic.https.ssl.truststore.password=asdfasdfelastic.https.ssl.truststore.type=JKSelastic.https.ssl.protocol=TLS
>
>
> בתאריך יום ו׳, 28 במאי 2021, 07:03, מאת Ran Lupovich ‏<
> ranlupov...@gmail.com
> >:
>
> >
> https://docs.confluent.io/kafka-connect-elasticsearch/current/security.html
> >
> > בתאריך יום ו׳, 28 במאי 2021, 07:00, מאת sunil chaudhari ‏<
> > sunilmchaudhar...@gmail.com>:
> >
> >> The configurations doesnt have provision for the truststore. Thats my
> >> concern.
> >>
> >>
> >> On Thu, 27 May 2021 at 10:47 PM, Ran Lupovich 
> >> wrote:
> >>
> >> > For https connections you need to set truststore configuration
> >> parameters ,
> >> > giving it jks with password , the jks needs the contain the certficate
> >> of
> >> > CA that is signing your certifcates
> >> >
> >> > בתאריך יום ה׳, 27 במאי 2021, 19:55, מאת sunil chaudhari ‏<
> >> > sunilmchaudhar...@gmail.com>:
> >> >
> >> > > Hi Ran,
> >> > > That problem is solved already.
> >> > > If you read complete thread and see that last problem is about https
> >> > > connection.
> >> > >
> >> > >
> >> > > On Thu, 27 May 2021 at 8:01 PM, Ran Lupovich  >
> >> > > wrote:
> >> > >
> >> > > > Try setting  es.port = "9200" without quotes?
> >> > > >
> >> > > > בתאריך יום ה׳, 27 במאי 2021, 04:21, מאת sunil chaudhari ‏<
> >> > > > sunilmchaudhar...@gmail.com>:
> >> > > >
> >> > > > > Hello team,
> >> > > > > Can anyone help me with this issue?
> >> > > > >
> >> > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
> https://github.com/DarioBalinzo/kafka-connect-elasticsearch-source/issues/44
> >> > > > >
> >> > > > >
> >> > > > > Regards,
> >> > > > > Sunil.
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
> >
>

Re: Issue using Https with elasticsearch source connector

[sunil chaudhari]

The configurations doesnt have provision for the truststore. Thats my
concern.


On Thu, 27 May 2021 at 10:47 PM, Ran Lupovich  wrote:

> For https connections you need to set truststore configuration parameters ,
> giving it jks with password , the jks needs the contain the certficate of
> CA that is signing your certifcates
>
> בתאריך יום ה׳, 27 במאי 2021, 19:55, מאת sunil chaudhari ‏<
> sunilmchaudhar...@gmail.com>:
>
> > Hi Ran,
> > That problem is solved already.
> > If you read complete thread and see that last problem is about https
> > connection.
> >
> >
> > On Thu, 27 May 2021 at 8:01 PM, Ran Lupovich 
> > wrote:
> >
> > > Try setting  es.port = "9200" without quotes?
> > >
> > > בתאריך יום ה׳, 27 במאי 2021, 04:21, מאת sunil chaudhari ‏<
> > > sunilmchaudhar...@gmail.com>:
> > >
> > > > Hello team,
> > > > Can anyone help me with this issue?
> > > >
> > > >
> > > >
> > >
> >
> https://github.com/DarioBalinzo/kafka-connect-elasticsearch-source/issues/44
> > > >
> > > >
> > > > Regards,
> > > > Sunil.
> > > >
> > >
> >
>

Re: Issue using Https with elasticsearch source connector

[sunil chaudhari]

Hi Ran,
That problem is solved already.
If you read complete thread and see that last problem is about https
connection.


On Thu, 27 May 2021 at 8:01 PM, Ran Lupovich  wrote:

> Try setting  es.port = "9200" without quotes?
>
> בתאריך יום ה׳, 27 במאי 2021, 04:21, מאת sunil chaudhari ‏<
> sunilmchaudhar...@gmail.com>:
>
> > Hello team,
> > Can anyone help me with this issue?
> >
> >
> >
> https://github.com/DarioBalinzo/kafka-connect-elasticsearch-source/issues/44
> >
> >
> > Regards,
> > Sunil.
> >
>

Issue using Https with elasticsearch source connector

[sunil chaudhari]

Hello team,
Can anyone help me with this issue?

https://github.com/DarioBalinzo/kafka-connect-elasticsearch-source/issues/44


Regards,
Sunil.

Re: Multiple producers using same message key

[sunil chaudhari]

Hi Neeraj,
I dont think there is relation of key and the partition in that sense..



On Sat, 8 May 2021 at 3:16 AM, Neeraj Vaidya
 wrote:

> Hi all,
> I think I kind of know the answer but wanted to confirm.
> If I have multiple producers sending messages with the same key, will they
> end up in the same partition (assuming I am using the default partitioner) ?
>
> Regards,
> Neeraj
>
> Sent from my iPhone
>

Re: Confluence Control Center On Windows 10

[sunil chaudhari]

Hi,
By the way why do you want it to setup on windows?
You can setup and run it on linux and access the user interface from
windows if you have firewall opened for running port. Default is 9021.


On Fri, 26 Mar 2021 at 10:39 AM, Satendra Negi 
wrote:

> Hello Guys,
>
> Is there any way to run the confluent kafka control center on windows ? the
> binaries provided by confluent seems not to support the windows 10 by
> default.
>
> Thanks in advance.
>
> --
>
>
>
> *Thanks & Regards*
>

Re: Unable to list Kafka topics

[sunil chaudhari]

Hi,
You can refer to the documentation for logging levels.


https://zookeeper.apache.org/doc/r3.4.11/zookeeperAdmin.html#Debug+Log+Cleanup+%28log4j%29

By clearing current logs will help you to read and understand new logs.

Old logs wont get mixed up with the new one and you can read and debug
events from the beginning of startup.

Regards,
Sunil.

On Sun, 22 Nov 2020 at 5:52 PM, prat 007  wrote:

> Hi Sunil,
>
> Thanks for your reply.
> You mean to say to empty kafka.log file? (I don't see zookeeper.log file
> in server), but not sure how clearing the log file itself will help.
>
> I will search for debug switch for zookeeper start command but I was
> talking about in which zookeeper config file I can change the value of a
> parameter from INFO to DEBUG or TRACE level to enable the whole log file to
> show the logs in more depth.
>
> thanks,
>
> On 2020/11/22 12:00:26, sunil chaudhari 
> wrote:
> > Hi I am talking about the Error :
> >
> > 18:16:08,027] ERROR Invalid configuration, only one server specified
> > > > (ignoring) (org.apache.zookeeper.server.quorum.QuorumPeerConfig)
> >
> >
> > This seems to be the problem of whole cluster.
> >
> > Clear all logs means, clear existing server logs. Not data.
> >
> > You can see help of the zookeeper start command for debug switch.
> >
> > Regards,
> > Sunil.
> >
> >
> > On Sun, 22 Nov 2020 at 5:00 PM, prat 007  wrote:
> >
> > > Hi Sunil,
> > >
> > > Thanks for reply. Q. Are you pointing to below line from point 6 to be
> > > similar with url you have given (in Troubleshooting, under section:
> Server
> > > not coming up because of file corruption)?
> > >
> > > ov 19 18:16:08  zookeeper-server-start.sh:
> java.io.EOFException
> > >
> > > Q. `Clear all logs` - you mean to say delete all files from
> > > `datadir/version-2 and datalogdir/version-2/`?
> > > If yes, is it safe to delete them. Do we require them later ?
> > >
> > > the url says,
> > > `make sure all the other servers in your ensemble are up and working.
> Use
> > > "stat" command on the command port to see if they are in good health`.
> > >
> > > Q. Is below command correct to verify good health ?
> > > #  echo stat | nc localhost 2181
> > >
> > > Q. Can you please let me know where can I set zookeeper logs for
> verbose
> > > mode for debugging the issue?
> > >
> > > Thanks,
> > >
> > > On 2020/11/22 01:26:52, sunil chaudhari 
> > > wrote:
> > > > Hi,
> > > > Please check if it helps:
> > > >
> > >
> http://zookeeper.apache.org/doc/r3.4.5/zookeeperAdmin.html#sc_supportedPlatforms
> > > >
> > > > You have similar symptoms in point 6 in your original email.
> > > >
> > > > Try running it manually with below steps:
> > > > Clear all logs
> > > > Start zookeeper manually, watch logs for any error. Rectify it
> referring
> > > > above link.
> > > > Start kafka manually. Watch info/error logs.
> > > > Then try to list topics.
> > > >
> > > > Regards,
> > > > Sunil.
> > > >
> > > >
> > > > On Sat, 21 Nov 2020 at 10:16 PM, prat 007 
> wrote:
> > > >
> > > > > Hi Team,
> > > > >
> > > > > I am running ansible to install elk on one server. (on Dev env,
> > > standalone
> > > > > setup) but my playbook is getting failed at kafka role for task
> > > listing of
> > > > > Kafka topics. Can you please help.
> > > > > below are the logs.
> > > > >
> > > > > 1.
> > > > > playbook getting failed with error: -
> > > > > ```
> > > > > TASK [kafka : Kafka Topics | List topics]
> > > > >
> > >
> **
> > > > > fatal: [kafka_1]: FAILED! => {
> > > > > "changed": false,
> > > > > "cmd": [
> > > > > "/opt/kafka/kafka_2.12-2.2.2/bin/kafka-topics.sh",
> > > > > "--zookeeper",
> > > > > ":5181",
> > > > > "--list"
> > > > > ],
> > > > > "delta": "0:00:34.084478",
> > > > 

Re: Unable to list Kafka topics

[sunil chaudhari]

Hi I am talking about the Error :

18:16:08,027] ERROR Invalid configuration, only one server specified
> > (ignoring) (org.apache.zookeeper.server.quorum.QuorumPeerConfig)


This seems to be the problem of whole cluster.

Clear all logs means, clear existing server logs. Not data.

You can see help of the zookeeper start command for debug switch.

Regards,
Sunil.


On Sun, 22 Nov 2020 at 5:00 PM, prat 007  wrote:

> Hi Sunil,
>
> Thanks for reply. Q. Are you pointing to below line from point 6 to be
> similar with url you have given (in Troubleshooting, under section: Server
> not coming up because of file corruption)?
>
> ov 19 18:16:08  zookeeper-server-start.sh: java.io.EOFException
>
> Q. `Clear all logs` - you mean to say delete all files from
> `datadir/version-2 and datalogdir/version-2/`?
> If yes, is it safe to delete them. Do we require them later ?
>
> the url says,
> `make sure all the other servers in your ensemble are up and working. Use
> "stat" command on the command port to see if they are in good health`.
>
> Q. Is below command correct to verify good health ?
> #  echo stat | nc localhost 2181
>
> Q. Can you please let me know where can I set zookeeper logs for verbose
> mode for debugging the issue?
>
> Thanks,
>
> On 2020/11/22 01:26:52, sunil chaudhari 
> wrote:
> > Hi,
> > Please check if it helps:
> >
> http://zookeeper.apache.org/doc/r3.4.5/zookeeperAdmin.html#sc_supportedPlatforms
> >
> > You have similar symptoms in point 6 in your original email.
> >
> > Try running it manually with below steps:
> > Clear all logs
> > Start zookeeper manually, watch logs for any error. Rectify it referring
> > above link.
> > Start kafka manually. Watch info/error logs.
> > Then try to list topics.
> >
> > Regards,
> > Sunil.
> >
> >
> > On Sat, 21 Nov 2020 at 10:16 PM, prat 007  wrote:
> >
> > > Hi Team,
> > >
> > > I am running ansible to install elk on one server. (on Dev env,
> standalone
> > > setup) but my playbook is getting failed at kafka role for task
> listing of
> > > Kafka topics. Can you please help.
> > > below are the logs.
> > >
> > > 1.
> > > playbook getting failed with error: -
> > > ```
> > > TASK [kafka : Kafka Topics | List topics]
> > >
> **
> > > fatal: [kafka_1]: FAILED! => {
> > > "changed": false,
> > > "cmd": [
> > > "/opt/kafka/kafka_2.12-2.2.2/bin/kafka-topics.sh",
> > > "--zookeeper",
> > > ":5181",
> > > "--list"
> > > ],
> > > "delta": "0:00:34.084478",
> > > "end": "2020-11-19 22:33:42.624353",
> > > "rc": 1,
> > > "start": "2020-11-19 22:33:08.539875"
> > > }
> > >
> > > STDERR:
> > >
> > > Exception in thread "main"
> > > kafka.zookeeper.ZooKeeperClientTimeoutException: Timed out waiting for
> > > connection while in state: CONNECTING
> > > at
> > >
> kafka.zookeeper.ZooKeeperClient.$anonfun$waitUntilConnected$3(ZooKeeperClient.scala:242)
> > > at
> > > scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
> > > at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:251)
> > > at
> > >
> kafka.zookeeper.ZooKeeperClient.waitUntilConnected(ZooKeeperClient.scala:238)
> > > at
> kafka.zookeeper.ZooKeeperClient.(ZooKeeperClient.scala:96)
> > > at kafka.zk.KafkaZkClient$.apply(KafkaZkClient.scala:1825)
> > > ```
> > >
> > > 2. Getting same error when run command manually which playbook is
> trying
> > > to run.
> > >
> > > ```
> > > /opt/kafka/kafka_2.12-2.2.2/bin/kafka-topics.sh --zookeeper  > > localhost>:5181 --list
> > >
> > > Exception in thread "main"
> > > kafka.zookeeper.ZooKeeperClientTimeoutException: Timed out waiting for
> > > connection while in state: CONNECTING
> > > at
> > >
> kafka.zookeeper.ZooKeeperClient.$anonfun$waitUntilConnected$3(ZooKeeperClient.scala:242)
> > > at
> > > scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
> > > at kafka.utils.CoreUtils$.inLock(CoreUtils.sc

Re: Unable to list Kafka topics

[sunil chaudhari]

Hi,
Please check if it helps:
http://zookeeper.apache.org/doc/r3.4.5/zookeeperAdmin.html#sc_supportedPlatforms

You have similar symptoms in point 6 in your original email.

Try running it manually with below steps:
Clear all logs
Start zookeeper manually, watch logs for any error. Rectify it referring
above link.
Start kafka manually. Watch info/error logs.
Then try to list topics.

Regards,
Sunil.


On Sat, 21 Nov 2020 at 10:16 PM, prat 007  wrote:

> Hi Team,
>
> I am running ansible to install elk on one server. (on Dev env, standalone
> setup) but my playbook is getting failed at kafka role for task listing of
> Kafka topics. Can you please help.
> below are the logs.
>
> 1.
> playbook getting failed with error: -
> ```
> TASK [kafka : Kafka Topics | List topics]
> **
> fatal: [kafka_1]: FAILED! => {
> "changed": false,
> "cmd": [
> "/opt/kafka/kafka_2.12-2.2.2/bin/kafka-topics.sh",
> "--zookeeper",
> ":5181",
> "--list"
> ],
> "delta": "0:00:34.084478",
> "end": "2020-11-19 22:33:42.624353",
> "rc": 1,
> "start": "2020-11-19 22:33:08.539875"
> }
>
> STDERR:
>
> Exception in thread "main"
> kafka.zookeeper.ZooKeeperClientTimeoutException: Timed out waiting for
> connection while in state: CONNECTING
> at
> kafka.zookeeper.ZooKeeperClient.$anonfun$waitUntilConnected$3(ZooKeeperClient.scala:242)
> at
> scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
> at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:251)
> at
> kafka.zookeeper.ZooKeeperClient.waitUntilConnected(ZooKeeperClient.scala:238)
> at kafka.zookeeper.ZooKeeperClient.(ZooKeeperClient.scala:96)
> at kafka.zk.KafkaZkClient$.apply(KafkaZkClient.scala:1825)
> ```
>
> 2. Getting same error when run command manually which playbook is trying
> to run.
>
> ```
> /opt/kafka/kafka_2.12-2.2.2/bin/kafka-topics.sh --zookeeper  localhost>:5181 --list
>
> Exception in thread "main"
> kafka.zookeeper.ZooKeeperClientTimeoutException: Timed out waiting for
> connection while in state: CONNECTING
> at
> kafka.zookeeper.ZooKeeperClient.$anonfun$waitUntilConnected$3(ZooKeeperClient.scala:242)
> at
> scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
> at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:251)
> at
> kafka.zookeeper.ZooKeeperClient.waitUntilConnected(ZooKeeperClient.scala:238)
> at kafka.zookeeper.ZooKeeperClient.(ZooKeeperClient.scala:96)
> at kafka.zk.KafkaZkClient$.apply(KafkaZkClient.scala:1825)
> at
> kafka.admin.TopicCommand$ZookeeperTopicService$.apply(TopicCommand.scala:262)
> at kafka.admin.TopicCommand$.main(TopicCommand.scala:53)
> at kafka.admin.TopicCommand.main(TopicCommand.scala)
> ```
>
> 3. both Kafka and zookeeper are failed and not getting started.
>
> ```
> systemctl status  kafka.service -l
> ● kafka.service
>Loaded: loaded (/etc/systemd/system/kafka.service; enabled; vendor
> preset: disabled)
>Active: failed (Result: exit-code) since Thu 2020-11-19 22:33:17 IST;
> 5min ago
>   Process: 5448
> ExecStop=/opt/kafka/kafka_2.12-2.2.2/bin/kafka-server-stop.sh (code=exited,
> status=1/FAILURE)
>   Process: 4742 ExecStart=/bin/sh -c
> /opt/kafka/kafka_2.12-2.2.2/bin/kafka-server-start.sh
> /opt/kafka/kafka_2.12-2.2.2/config/server.properties >
> /opt/kafka/kafka_2.12-2.2.2/kafka.log 2>&1 (code=exited, status=1/FAILURE)
>  Main PID: 4742 (code=exited, status=1/FAILURE)
>
> Nov 19 22:33:08  systemd[1]: Started kafka.service.
> Nov 19 22:33:17  systemd[1]: kafka.service: main process exited,
> code=exited, status=1/FAILURE
> Nov 19 22:33:17  systemd[1]: kafka.service: control process
> exited, code=exited status=1
> Nov 19 22:33:17  systemd[1]: Unit kafka.service entered failed
> state.
> Nov 19 22:33:17  systemd[1]: kafka.service failed.
>
>
>  systemctl status  zookeeper.service  -l
> ● zookeeper.service
>Loaded: loaded (/etc/systemd/system/zookeeper.service; enabled; vendor
> preset: disabled)
>Active: failed (Result: exit-code) since Thu 2020-11-19 22:33:09 IST;
> 5min ago
>   Process: 5281
> ExecStop=/opt/kafka/kafka_2.12-2.2.2/bin/zookeeper-server-stop.sh
> (code=exited, status=1/FAILURE)
>   Process: 4448
> ExecStart=/opt/kafka/kafka_2.12-2.2.2/bin/zookeeper-server-start.sh
> /opt/kafka/kafka_2.12-2.2.2/config/zookeeper.properties (code=exited,
> status=1/FAILURE)
>  Main PID: 4448 (code=exited, status=1/FAILURE)
>
> Nov 19 22:33:08 zookeeper-server-start.sh[4448]: at
> org.apache.zookeeper.server.NIOServerCnxnFactory.startup(NIOServerCnxnFactory.java:118)
> Nov 19 22:33:08 zookeeper-server-start.sh[4448]: at
> org.apache.zookeeper.server.ZooKeeperServerMain.runFromConfig(ZooKeeperServerMain.java:122)
> Nov 19 22:33:08 zookeeper-server-start.sh[4448]: at
> org.apache.zookeeper.serv

Re: Unable to list Kafka topics

[sunil chaudhari]

Hi,
Whats your real problem?
Kafka zookeeper failing OR cant list topics.
Unless you start kafka, you wont be able to list your topics.


On Sun, 22 Nov 2020 at 12:23 AM, prat 007  wrote:

> Hi SasiKumar,
>
> Thanks for your reply. I have pasted telnet output.
>
> Thanks,
>
> On 2020/11/21 17:08:51, SasiKumar Senthilnathan 
> wrote:
> > Hai,
> >
> > Looks the host is not reachable.
> >
> > Can you try tcptraceroute or telnet?
> >
> >
> > On Sat, Nov 21, 2020, 22:16 prat 007  wrote:
> >
> > > Hi Team,
> > >
> > > I am running ansible to install elk on one server. (on Dev env,
> standalone
> > > setup) but my playbook is getting failed at kafka role for task
> listing of
> > > Kafka topics. Can you please help.
> > > below are the logs.
> > >
> > > 1.
> > > playbook getting failed with error: -
> > > ```
> > > TASK [kafka : Kafka Topics | List topics]
> > >
> **
> > > fatal: [kafka_1]: FAILED! => {
> > > "changed": false,
> > > "cmd": [
> > > "/opt/kafka/kafka_2.12-2.2.2/bin/kafka-topics.sh",
> > > "--zookeeper",
> > > ":5181",
> > > "--list"
> > > ],
> > > "delta": "0:00:34.084478",
> > > "end": "2020-11-19 22:33:42.624353",
> > > "rc": 1,
> > > "start": "2020-11-19 22:33:08.539875"
> > > }
> > >
> > > STDERR:
> > >
> > > Exception in thread "main"
> > > kafka.zookeeper.ZooKeeperClientTimeoutException: Timed out waiting for
> > > connection while in state: CONNECTING
> > > at
> > >
> kafka.zookeeper.ZooKeeperClient.$anonfun$waitUntilConnected$3(ZooKeeperClient.scala:242)
> > > at
> > > scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
> > > at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:251)
> > > at
> > >
> kafka.zookeeper.ZooKeeperClient.waitUntilConnected(ZooKeeperClient.scala:238)
> > > at
> kafka.zookeeper.ZooKeeperClient.(ZooKeeperClient.scala:96)
> > > at kafka.zk.KafkaZkClient$.apply(KafkaZkClient.scala:1825)
> > > ```
> > >
> > > 2. Getting same error when run command manually which playbook is
> trying
> > > to run.
> > >
> > > ```
> > > /opt/kafka/kafka_2.12-2.2.2/bin/kafka-topics.sh --zookeeper  > > localhost>:5181 --list
> > >
> > > Exception in thread "main"
> > > kafka.zookeeper.ZooKeeperClientTimeoutException: Timed out waiting for
> > > connection while in state: CONNECTING
> > > at
> > >
> kafka.zookeeper.ZooKeeperClient.$anonfun$waitUntilConnected$3(ZooKeeperClient.scala:242)
> > > at
> > > scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
> > > at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:251)
> > > at
> > >
> kafka.zookeeper.ZooKeeperClient.waitUntilConnected(ZooKeeperClient.scala:238)
> > > at
> kafka.zookeeper.ZooKeeperClient.(ZooKeeperClient.scala:96)
> > > at kafka.zk.KafkaZkClient$.apply(KafkaZkClient.scala:1825)
> > > at
> > >
> kafka.admin.TopicCommand$ZookeeperTopicService$.apply(TopicCommand.scala:262)
> > > at kafka.admin.TopicCommand$.main(TopicCommand.scala:53)
> > > at kafka.admin.TopicCommand.main(TopicCommand.scala)
> > > ```
> > >
> > > 3. both Kafka and zookeeper are failed and not getting started.
> > >
> > > ```
> > > systemctl status  kafka.service -l
> > > ● kafka.service
> > >Loaded: loaded (/etc/systemd/system/kafka.service; enabled; vendor
> > > preset: disabled)
> > >Active: failed (Result: exit-code) since Thu 2020-11-19 22:33:17
> IST;
> > > 5min ago
> > >   Process: 5448
> > > ExecStop=/opt/kafka/kafka_2.12-2.2.2/bin/kafka-server-stop.sh
> (code=exited,
> > > status=1/FAILURE)
> > >   Process: 4742 ExecStart=/bin/sh -c
> > > /opt/kafka/kafka_2.12-2.2.2/bin/kafka-server-start.sh
> > > /opt/kafka/kafka_2.12-2.2.2/config/server.properties >
> > > /opt/kafka/kafka_2.12-2.2.2/kafka.log 2>&1 (code=exited,
> status=1/FAILURE)
> > >  Main PID: 4742 (code=exited, status=1/FAILURE)
> > >
> > > Nov 19 22:33:08  systemd[1]: Started kafka.service.
> > > Nov 19 22:33:17  systemd[1]: kafka.service: main process
> exited,
> > > code=exited, status=1/FAILURE
> > > Nov 19 22:33:17  systemd[1]: kafka.service: control process
> > > exited, code=exited status=1
> > > Nov 19 22:33:17  systemd[1]: Unit kafka.service entered
> failed
> > > state.
> > > Nov 19 22:33:17  systemd[1]: kafka.service failed.
> > >
> > >
> > >  systemctl status  zookeeper.service  -l
> > > ● zookeeper.service
> > >Loaded: loaded (/etc/systemd/system/zookeeper.service; enabled;
> vendor
> > > preset: disabled)
> > >Active: failed (Result: exit-code) since Thu 2020-11-19 22:33:09
> IST;
> > > 5min ago
> > >   Process: 5281
> > > ExecStop=/opt/kafka/kafka_2.12-2.2.2/bin/zookeeper-server-stop.sh
> > > (code=exited, status=1/FAILURE)
> > >   Process: 4448
> > > ExecStart=/opt/kafka/kafka_2.12-2.2.2/bin/zookeeper-server-start.sh
> > > /opt/kafka/kafka

Re: Isolation.level default behavior

[sunil chaudhari]

I am not very sure about the isolation.level setting.
However duplicates may cause due to the commit failed on the consumer side.
Please do read about max.poll.interval.ms and max.poll.records settings.
You may get some valuable inputs.

Recently i solved duplicates issue in my consumer by tuning  above values.

Hope it helps.

Regards,
Sunil.

On Thu, 29 Oct 2020 at 3:38 PM, usha rani  wrote:

> Hi Team,
>
> Recently we ended up having huge duplicates sending out of the connector
> due to the segment roll out ended in creating out of range issue. To avoid
> the occurance of the above issue we made couple of changes as part of it we
> are planning to change 'isolation.level' to 'read_committed'.
>
> So before making that change I wanted to understand why the default of
> 'isolation.level' is 'read_uncommitted' and not 'read_committed'. When I
> see read_uncommitted end up in dirty read and producing duplicates.
>
> Thanks,
> Usha
>

Re: backup

[sunil chaudhari]

Hi Don,
Kafka is not meant to be a general purpose database. Its a streaming
platform. So think about retention of kafka messages rather than taking
backup.
Kafka itself has retention capability. So you can tune it as per your need.

Regards,
Sunil.

On Sun, 9 Aug 2020 at 5:13 PM, Dor Ben Dov  wrote:

> Hi All,
> What is the best recommended way, and tool to backup kafka in production?
> Regards,
> Dor
>
> This email and the information contained herein is proprietary and
> confidential and subject to the Amdocs Email Terms of Service, which you
> may review at https://www.amdocs.com/about/email-terms-of-service <
> https://www.amdocs.com/about/email-terms-of-service>
>
>

Re: How to Change number of partitions without Rolling restart?

[sunil chaudhari]

Thanks all for this.
Las question around this.

If I write scripts for topics, I am afraid it will change partitions for
some default/internal system created topics which I don't want,
In that Case, I guess, I have to filter topic list first and then provide
it to the script...

 is that correct?


On Mon, 22 Jun 2020 at 9:59 AM, Liam Clarke-Hutchinson <
liam.cla...@adscale.co.nz> wrote:

> Hi Sunil,
>
> You'd want to do a bit of shell scripting for this.
>
> for topic in $(./kafka-topics.sh --bootstrap-server your-kafka-here:9092
> --list); do
>   ./kafka-topics.sh --bootstrap-server your-kafka-here:9092 --topic $topic
> --partitions X --alter
> done
>
> Etc. etc. And yep, to make sure further auto created topics have your
> desired partitions, you will need to change that property on the brokers
> and rolling restart.
>
> Cheers,
>
> Liam Clarke-Hutchinson
>
> On Mon, Jun 22, 2020 at 4:16 PM sunil chaudhari <
> sunilmchaudhar...@gmail.com>
> wrote:
>
> > Manoj,
> > You mean I have execute this command manually for all 350 Topics which I
> > already have?
> > Is there any possibility I can use any wild cards?
> >
> >
> > On Mon, 22 Jun 2020 at 9:28 AM,  wrote:
> >
> > > You can use below command to alter to partition
> > >
> > > ./bin/kafka-topics.sh --alter --zookeeper localhost:2181 --topic
> my-topic
> > > --partitions 6
> > >
> > >  Thanks
> > > Manoj
> > >
> > >
> > >
> > > On 6/21/20, 7:38 PM, "sunil chaudhari" 
> > > wrote:
> > >
> > > [External]
> > >
> > >
> > > Hi,
> > > I already have 350 topics created. Please guide me how can I do
> that
> > > for
> > > these many topics?
> > > Also I want each new topic to be created with more number
> partitions
> > > automatically than previous number 3, which I had set in
> properties.
> > >
> > > Regards,
> > > Sunil.
> > >
> > > On Mon, 22 Jun 2020 at 6:31 AM, Liam Clarke-Hutchinson <
> > > liam.cla...@adscale.co.nz> wrote:
> > >
> > >     > Hi Sunil,
> > > >
> > > > The broker setting num.partitions only applies to automatically
> > > created
> > > > topics (if that is enabled) at the time of creation. To change
> > > partitions
> > > > for a topic you need to use kafka-topics.sh to do so for each
> > topic.
> > > >
> > > > Kind regards,
> > > >
> > > > Liam Clarke-Hutchinson
> > > >
> > > > On Mon, Jun 22, 2020 at 3:16 AM sunil chaudhari <
> > > > sunilmchaudhar...@gmail.com>
> > > > wrote:
> > > >
> > > > > Hi,
> > > > > I want to change number of partitions for all topics.
> > > > > How can I change that? Is it server.properties which I need to
> > > change?
> > > > > Then, in that case I have to restart broker right?
> > > > >
> > > > > I checked from confluent control center, there is no option to
> > > change
> > > > > partitions.
> > > > >
> > > > > Please advise.
> > > > >
> > > > > Regards,
> > > > > Sunil
> > > > >
> > > >
> > >
> > >
> > > This e-mail and any files transmitted with it are for the sole use of
> the
> > > intended recipient(s) and may contain confidential and privileged
> > > information. If you are not the intended recipient(s), please reply to
> > the
> > > sender and destroy all copies of the original message. Any unauthorized
> > > review, use, disclosure, dissemination, forwarding, printing or copying
> > of
> > > this email, and/or any action taken in reliance on the contents of this
> > > e-mail is strictly prohibited and may be unlawful. Where permitted by
> > > applicable law, this e-mail and other e-mail communications sent to and
> > > from Cognizant e-mail addresses may be monitored.
> > > This e-mail and any files transmitted with it are for the sole use of
> the
> > > intended recipient(s) and may contain confidential and privileged
> > > information. If you are not the intended recipient(s), please reply to
> > the
> > > sender and destroy all copies of the original message. Any unauthorized
> > > review, use, disclosure, dissemination, forwarding, printing or copying
> > of
> > > this email, and/or any action taken in reliance on the contents of this
> > > e-mail is strictly prohibited and may be unlawful. Where permitted by
> > > applicable law, this e-mail and other e-mail communications sent to and
> > > from Cognizant e-mail addresses may be monitored.
> > >
> >
>

Re: How to Change number of partitions without Rolling restart?

[sunil chaudhari]

Manoj,
You mean I have execute this command manually for all 350 Topics which I
already have?
Is there any possibility I can use any wild cards?


On Mon, 22 Jun 2020 at 9:28 AM,  wrote:

> You can use below command to alter to partition
>
> ./bin/kafka-topics.sh --alter --zookeeper localhost:2181 --topic my-topic
> --partitions 6
>
>  Thanks
> Manoj
>
>
>
> On 6/21/20, 7:38 PM, "sunil chaudhari" 
> wrote:
>
> [External]
>
>
> Hi,
> I already have 350 topics created. Please guide me how can I do that
> for
> these many topics?
> Also I want each new topic to be created with more number partitions
> automatically than previous number 3, which I had set in properties.
>
> Regards,
> Sunil.
>
> On Mon, 22 Jun 2020 at 6:31 AM, Liam Clarke-Hutchinson <
> liam.cla...@adscale.co.nz> wrote:
>
> > Hi Sunil,
> >
> > The broker setting num.partitions only applies to automatically
> created
> > topics (if that is enabled) at the time of creation. To change
> partitions
> > for a topic you need to use kafka-topics.sh to do so for each topic.
> >
> > Kind regards,
> >
> > Liam Clarke-Hutchinson
> >
> > On Mon, Jun 22, 2020 at 3:16 AM sunil chaudhari <
> > sunilmchaudhar...@gmail.com>
> > wrote:
> >
> > > Hi,
> > > I want to change number of partitions for all topics.
> > > How can I change that? Is it server.properties which I need to
> change?
> > > Then, in that case I have to restart broker right?
> > >
> > > I checked from confluent control center, there is no option to
> change
> > > partitions.
> > >
> > > Please advise.
> > >
> > > Regards,
> > > Sunil
> > >
> >
>
>
> This e-mail and any files transmitted with it are for the sole use of the
> intended recipient(s) and may contain confidential and privileged
> information. If you are not the intended recipient(s), please reply to the
> sender and destroy all copies of the original message. Any unauthorized
> review, use, disclosure, dissemination, forwarding, printing or copying of
> this email, and/or any action taken in reliance on the contents of this
> e-mail is strictly prohibited and may be unlawful. Where permitted by
> applicable law, this e-mail and other e-mail communications sent to and
> from Cognizant e-mail addresses may be monitored.
> This e-mail and any files transmitted with it are for the sole use of the
> intended recipient(s) and may contain confidential and privileged
> information. If you are not the intended recipient(s), please reply to the
> sender and destroy all copies of the original message. Any unauthorized
> review, use, disclosure, dissemination, forwarding, printing or copying of
> this email, and/or any action taken in reliance on the contents of this
> e-mail is strictly prohibited and may be unlawful. Where permitted by
> applicable law, this e-mail and other e-mail communications sent to and
> from Cognizant e-mail addresses may be monitored.
>

Re: How to Change number of partitions without Rolling restart?

[sunil chaudhari]

Hi,
I already have 350 topics created. Please guide me how can I do that for
these many topics?
Also I want each new topic to be created with more number partitions
automatically than previous number 3, which I had set in properties.

Regards,
Sunil.

On Mon, 22 Jun 2020 at 6:31 AM, Liam Clarke-Hutchinson <
liam.cla...@adscale.co.nz> wrote:

> Hi Sunil,
>
> The broker setting num.partitions only applies to automatically created
> topics (if that is enabled) at the time of creation. To change partitions
> for a topic you need to use kafka-topics.sh to do so for each topic.
>
> Kind regards,
>
> Liam Clarke-Hutchinson
>
> On Mon, Jun 22, 2020 at 3:16 AM sunil chaudhari <
> sunilmchaudhar...@gmail.com>
> wrote:
>
> > Hi,
> > I want to change number of partitions for all topics.
> > How can I change that? Is it server.properties which I need to change?
> > Then, in that case I have to restart broker right?
> >
> > I checked from confluent control center, there is no option to change
> > partitions.
> >
> > Please advise.
> >
> > Regards,
> > Sunil
> >
>

How to Change number of partitions without Rolling restart?

[sunil chaudhari]

Hi,
I want to change number of partitions for all topics.
How can I change that? Is it server.properties which I need to change?
Then, in that case I have to restart broker right?

I checked from confluent control center, there is no option to change
partitions.

Please advise.

Regards,
Sunil

Memory for a broker

[sunil chaudhari]

Hi,
I was going through this document.
https://docs.confluent.io/current/kafka/deployment.html
“ does not require setting heap sizes more than 6 GB. This will result in a
file system cache of up to 28-30 GB on a 32 GB machine.”

Can someone please put focus on above statement? Its bit unclear to me as
why file system cache will reach to 28-30 GB ?
I have 64 GB machine for each broker. Should I stick to 6 GB still? Or I
can assign some more?

Regards,
Sunil.

Re: Broker thread pool sizing

[sunil chaudhari]

This is awesome. Thanks Ricardo.

On Fri, 19 Jun 2020 at 9:31 PM, Ricardo Ferreira 
wrote:

> Gérald,
>
> Typically you should set the `num.io.threads` to something greater than
> the # of disks since data hits the page cache and the disk. Using the
> default of 8 when you have a JBOD of 12 attached volumes would cause an
> increase of CPU context switching, for example.
>
> `num.network.threads` is usually fine since most machines have 1 or 2
> NIC's attached at tops and then the rule of *N + 1* suits the case.
> However, you should double that if TLS is enabled in the broker.
>
> Thanks,
>
> -- Ricardo
>
> On 6/19/20 8:56 AM, Gérald Quintana wrote:
> > Hello,
> >
> > How do you size Kafka broker thread pools, in particular num.io.threads
> (8
> > by default) and num.network.threads (3 by default) depending on the
> number
> > of CPU cores available on the host?
> >
> > Regards,
> > Gérald
> >
>

Re: Duplicate records on consumer side.

[sunil chaudhari]

Hi,
Thanks for the clarification.
This means, for “A“ consumer group, Running one Consumer instance with 3
threads on one server is equal to running 3 different instances with one
thread each on 3 different servers.

So Now if I already started 3 instances on 3 servers with 3 threads each,
then To better utilise it, i have to increase partitions. Right?

What is impact on Existing topics, if i increase number of partitions for
all topics and reatart cluster?

Or I can do that from CLI or Confluent control Center without restarting
cluster?


About duplicate records, it seems problem of max.poll.records and polling
interval. I am working on that.
Offset commit is failing before next poll for a consumer group. Thats the
problem.
Now I dont know what is default value in cluster for above 2 parameters and
what value should I set in logstash kafka input?

Sorry to mixup so many things in one mail😃


Regards,
Sunil.


On Fri, 19 Jun 2020 at 7:59 PM, Ricardo Ferreira 
wrote:

> Sunil,
>
> Kafka ensures that each partition is read by one given thread only from a
> consumer group. Since your topic has three partitions, the rationale is
> that at least three threads from the consumer group will be properly served.
>
> However, though your calculation is correct (3 instances, each one of 3
> threads will total 9 threads) the design and usage is incorrect. As stated
> above only three threads will be served and the remaining six other threads
> will be kept waiting -- likely to starve if all of them belong to the
> consumer group that the other three threads belong.
>
> Please note that the `client-id` property has nothing to do with this
> thread group management. This property is used internally by Kafka to
> correlate events sent from the same machine in order to better adjust quota
> management. So the only property taking place where is the `group-id` in
> the matter of partition assignment.
>
> Regarding duplicated data, this is another problem that would require a
> better investigation of your topology, how Logstash connect to Kafka, and
> how the code is implemented.
>
> Thanks,
>
> -- Ricardo
> On 6/19/20 7:13 AM, sunil chaudhari wrote:
>
> Hi,
> I am using kafka as a broker in my event data pipeline.
> Filebeat as producer
> Logstash as consumer.
>
>
> Filebeat simply pushes to Kafka.
> Logstash has 3 instances.
> Each instance has a consumer group say consumer_mytopic which reads from
> mytopic.
>
> mytopic has 3 partitions and 2 replica.
>
> As per my understanding, each consumer group can have threads equal to
> number of partitions so i kept 3 threads for each consumer.
>
> Here I am considering one logstash instance as a one consumer which is part
> of consumer_mytopic.
> Similar consumer running on some other server which has group_id same as
> above. Note that 3 servers has client Id different so that they wont read
> duplicate data.
> So 3 instances of logstash running with group_id as consumer_mytopic with 3
> threads each, and diff client id. Means 9 threads total.
>
> My understanding is each consumer(instance) can read with 3 threads from 3
> partitions. And another consumer with 3 threads.
>
> Is this good design?
> Can it create duplicate?
> This thread and partitions trade-off is related to client_id or Consumer
> group Id?
> I hope because of diff client_id 3 instances wont read duplicate data even
> if group_id is same.
> I am getting duplicate data in my consumer side.
> Please help in this.
>
> Regards,
> Sunil.
>
>
>

Duplicate records on consumer side.

[sunil chaudhari]

Hi,
I am using kafka as a broker in my event data pipeline.
Filebeat as producer
Logstash as consumer.


Filebeat simply pushes to Kafka.
Logstash has 3 instances.
Each instance has a consumer group say consumer_mytopic which reads from
mytopic.

mytopic has 3 partitions and 2 replica.

As per my understanding, each consumer group can have threads equal to
number of partitions so i kept 3 threads for each consumer.

Here I am considering one logstash instance as a one consumer which is part
of consumer_mytopic.
Similar consumer running on some other server which has group_id same as
above. Note that 3 servers has client Id different so that they wont read
duplicate data.
So 3 instances of logstash running with group_id as consumer_mytopic with 3
threads each, and diff client id. Means 9 threads total.

My understanding is each consumer(instance) can read with 3 threads from 3
partitions. And another consumer with 3 threads.

Is this good design?
Can it create duplicate?
This thread and partitions trade-off is related to client_id or Consumer
group Id?
I hope because of diff client_id 3 instances wont read duplicate data even
if group_id is same.
I am getting duplicate data in my consumer side.
Please help in this.

Regards,
Sunil.

Re: Is that a bug?

[sunil chaudhari]

Simple..
Topic auto creation ON.
As soon as it encounters producer and consumers request for some topic then
it creates that topic automatically.

This is very common question and problem people come across, if they are
new. Same happened with me.😄


On Fri, 22 May 2020 at 3:28 PM, Jiamei Xie  wrote:

> Hi Manna, Luke
>  I used the default configure files without any modification.  The default
> value for allow.auto.create.topics is true according to
> https://kafka.apache.org/documentation/#brokerconfigs. Thanks for your
> reply.
>
> -Original Message-
> From: M. Manna 
> Sent: 2020年5月22日 16:50
> To: users@kafka.apache.org
> Subject: Re: Is that a bug?
>
> Hey Xie,
>
> On Fri, 22 May 2020 at 08:31, Jiamei Xie  wrote:
>
> > Hi
> >
> > Kill all zookeeper and kafka process. Clear zookeeper and kafka data dir.
> > Restart zookeeper and kafka. If there are any active client.  Topic
> > used by client will be auto-created.
> >
> > How to reproduce?
> >
> >
> >   1.  Start zookeeper and kafka zookeeper and kafka config file.
> >
> > nohup bin/zookeeper-server-start.sh config/zookeeper.properties &
> >
> > nohup bin/kafka-server-start.sh config/server.properties &
> >
> >
> >
> >   1.  Create topic test with 2 partitions
> >
> > bin/kafka-topics.sh --create --zookeeper localhost:2181 --topic test
> > --partitions 2 --replication-factor 1
> >
> >
> >
> >   1.  Produce some data to topic test
> >
> > bin/kafka-producer-perf-test.sh --topic test --num-records 5000
> > --record-size 100 --throughput=-1 --producer-props
> > bootstrap.servers=localhost:9092
> >
> >
> >
> >   1.  Kill zookeeper and kafka. ProducerPerformance is still running.
> >
> > jps
> >
> > 21072 QuorumPeerMain
> >
> > 21704 ProducerPerformance
> >
> > 21230 Kafka
> >
> > 21854 Jps
> >
> > kill -9 21072 21230
> >
> >
> >
> >   1.  Remove Zookeeper and Kafka data
> >
> > rm -rf /tmp/zookeeper/
> >
> > rm -rf /tmp/kafka-logs/
> >
> >
> >
> >   1.  Start zookeeper and kafka
> >
> > nohup bin/zookeeper-server-start.sh config/zookeeper.properties &
> >
> > nohup bin/kafka-server-start.sh config/server.properties &
> >
> >
> >
> >   1.  Check topic and you'll see there is topic named test with
> > partition 1.  And the ProducerPerformance process continues to run
> normally.
> >
> > bin/kafka-topics.sh --describe --zookeeper localhost:2181
> >
> > Topic: test PartitionCount: 1   ReplicationFactor: 1Configs:
> >
> > Topic: test Partition: 0Leader: 0   Replicas: 0
> >  Isr: 0
> >
> >
> > Some output of ProducerPerformance process.
> > 1995632 records sent, 399126.4 records/sec (38.06 MB/sec), 378.6 ms
> > avg latency, 435.0 ms max latency.
> > org.apache.kafka.common.errors.TimeoutException: Expiring 148
> > record(s) for test-1:12 ms has passed since batch creation
> > .
> > org.apache.kafka.common.errors.TimeoutException: Expiring 148
> > record(s) for test-1:121774 ms has passed since batch creation
> > 1711254 records sent, 342250.8 records/sec (32.64 MB/sec), 2324.5 ms
> > avg latency, 123473.0 ms max latency.
> >
> >
> >
> > Is that a bug?
>
>
>   How have you configured your server.properties and consumer.peoperties?
> Have you set auto creation ON?
>
> Regards,
>
> >
> >
> > Best Wishes,
> > Jiamei
> >
> > IMPORTANT NOTICE: The contents of this email and any attachments are
> > confidential and may also be privileged. If you are not the intended
> > recipient, please notify the sender immediately and do not disclose
> > the contents to any other person, use it for any purpose, or store or
> > copy the information in any medium. Thank you.
> >
> IMPORTANT NOTICE: The contents of this email and any attachments are
> confidential and may also be privileged. If you are not the intended
> recipient, please notify the sender immediately and do not disclose the
> contents to any other person, use it for any purpose, or store or copy the
> information in any medium. Thank you.
>

How to disable OPTIONS method in confluent-control-center

[Sunil CHAUDHARI]

Hi,
I don't know whether this question is relevant to this group?
Sorry If I posted in wrong group.
I want to disable OPTIONS method in Confluent-control center running on port 
9091.
Can someone guide me for required configurations?

Regards,
Sunil.

Re: Scaling Apache Kafka Producers & Consumers

[sunil chaudhari]

Again
A consumer can have one or more consumer thread.
The analogy of 12 partitions and 4 consumer is true when each consumer has
3 consumer threads.
Please don’t skip the important factor “consumer thread” in this matter.

If you run each consumer with threads then you may need max 3 consumers for
that consumer group.

If you have 12 partitions and you run 4 consumers with 4 consumer threads
then 4 threads will be idle at any time T1.

I hope this is clear.

Thanks,
Sunil.

On Thu, 26 Mar 2020 at 7:52 PM, Hans Jespersen  wrote:

> > As per my understanding, in Apache Kafka a single consumer from a
> consumer
> > group can consume messages from one partition only.
>
> Not correct. A single consumer from a consumer group can consume from many
> partitions. For example if you had a topic with 12 partitions and 4
> consumers in a consumer group, each consumer in the group would consume
> from 3 partitions.
>
> -hans

Re: Scaling Apache Kafka Producers & Consumers

[sunil chaudhari]

Hi Prasad,
Want to correct a bit. Ots not one consumer per partitions.
Its one consumer thread per partitions.


On Thu, 26 Mar 2020 at 4:49 PM, Prasad Suhas Shembekar <
ps00516...@techmahindra.com> wrote:

> Hi,
>
> I am using Apache Kafka as a Message Broker in our application. The
> producers and consumers are running as Docker containers in Kubernetes.
> Right now, the producer publishes messages to a topic in single partition.
> While the consumer consumes it from the topic.
> As per my understanding, in Apache Kafka a single consumer from a consumer
> group can consume messages from one partition only. Meaning, if there is
> only a single partition and multiple consumers in a consumer group, only
> one consumer will consume the message and the rest will remain idle, till
> Apache Kafka does the partition rebalancing.
> As mentioned earlier, we have a single topic and single partition and
> multiple consumers in a single group. Thus we won't be able to achieve the
> horizontal scaling for message consumption.
>
> Please let me know if the above understanding is correct.
>
> I am looking out on how to create partitions dynamically in the topic, as
> and when a new consumer is added to consumer group (K8S auto scaling of
> PODS).
> Also, how to make the producer write to these different partitions created
> dynamically, without overloading few partitions.
>
> Request you to provide some inputs / suggestions on how to achieve this.
>
> Thanks & Regards,
> Prasad Shembekar
> Blue Marble
> WST-020, D Non-ODC, Mihan SEZ,
> Nagpur
> Extension: 6272148
> Direct: 0712-6672148
>
> 
> Disclaimer: This message and the information contained herein is
> proprietary and confidential and subject to the Tech Mahindra policy
> statement, you may review the policy at
> http://www.techmahindra.com/Disclaimer.html externally
> http://tim.techmahindra.com/tim/disclaimer.html internally within
> TechMahindra.
> 
>

Re: Kafka JMX monitoring

[sunil chaudhari]

If you are not aware of Grafana and Prometheus then...
You can enable JMx_PORT on kafka broker.
Install and run kafka manager.
https://github.com/lemastero/kafka-manager

There you can see metrics on kafka manager UI.

But yes, grafana-prometheus is Better one.


On Thu, 19 Mar 2020 at 12:25 PM, 张祥  wrote:

> Hi,
>
> I want to know what the best practice to collect Kafka JMX metrics is. I
> haven't found a decent way to collect and parse JMX in Java (because it is
> too much) and I learn that there are tools like tools like jmxtrans to do
> this. I wonder if there is more. Thanks. Regards.
>

RE: [External] Re: Very Urgent: Error while creating ephemeral at /brokers/ids/0, node already exists and owner

[Sunil CHAUDHARI]

Hi Jonathan,

"are you gracefully shutting down the broker?"



above question solved my prob gracefully 😊.  Hahaha

Thanks a lot!





Regards,

Sunil.



-Original Message-
From: Jonathan Santilli 
Sent: Thursday, March 12, 2020 5:17 PM
To: Kafka Users 
Subject: [External] Re: Very Urgent: Error while creating ephemeral at 
/brokers/ids/0, node already exists and owner



This mail originated from an external party outside DBS - 
users-return-39785-sunilchaudhari=dbs@kafka.apache.org<mailto:users-return-39785-sunilchaudhari=dbs@kafka.apache.org>.
 Do not click on links or open attachments unless you recognize the sender and 
know the content is safe.



Hello Sunil,



are you gracefully shutting down the broker?



can you verify there is no other broker within the cluster with the same id 0?



can you check in the logs that the connection to ZooKeeper is getting 
close/disconnected when shutting down the Broker?





Cheers!







On Thu, Mar 12, 2020 at 10:36 AM Sunil CHAUDHARI 
mailto:sunilchaudh...@dbs.com.invalid>> wrote:



> Hi,

> Whenever I try to do rolling restart and start one of my broker, I get

> this error.

> Can Anyone help me to get rid of this?

>

> [2020-03-12 18:33:34,661] INFO Logs loading complete in 1553 ms.

> (kafka.log.LogManager)

> [2020-03-12 18:33:34,677] INFO Starting log cleanup with a period of

> 30 ms. (kafka.log.LogManager)

> [2020-03-12 18:33:34,678] INFO Starting log flusher with a default

> period of 9223372036854775807 ms. (kafka.log.LogManager)

> [2020-03-12 18:33:35,005] INFO Awaiting socket connections on 0.0.0.0:9092.

> (kafka.network.Acceptor)

> [2020-03-12 18:33:35,049] INFO [SocketServer brokerId=0] Created

> data-plane acceptor and processors for endpoint :

> EndPoint(null,9092,ListenerName(PLAINTEXT),PLAINTEXT)

> (kafka.network.SocketServer)

> [2020-03-12 18:33:35,050] INFO [SocketServer brokerId=0] Started 1

> acceptor threads for data-plane (kafka.network.SocketServer)

> [2020-03-12 18:33:35,109] INFO [ExpirationReaper-0-Produce]: Starting

> (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)

> [2020-03-12 18:33:35,116] INFO [ExpirationReaper-0-ElectPreferredLeader]:

> Starting

> (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)

> [2020-03-12 18:33:35,116] INFO [ExpirationReaper-0-DeleteRecords]:

> Starting

> (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)

> [2020-03-12 18:33:35,116] INFO [ExpirationReaper-0-Fetch]: Starting

> (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)

> [2020-03-12 18:33:35,121] INFO [ExpirationReaper-0-ListOffsets]:

> Starting

> (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)

> [2020-03-12 18:33:35,139] INFO [LogDirFailureHandler]: Starting

> (kafka.server.ReplicaManager$LogDirFailureHandler)

> [2020-03-12 18:33:35,225] INFO Creating /brokers/ids/0 (is it secure?

> false) (kafka.zk.KafkaZkClient)

> [2020-03-12 18:33:35,245] ERROR Error while creating ephemeral at

> /brokers/ids/0, node already exists and owner '216411990614016005'

> does not match current session '144234981936660481'

> (kafka.zk.KafkaZkClient$CheckedEphemeral)

> [2020-03-12 18:33:35,252] ERROR [KafkaServer id=0] Fatal error during

> KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer)

> org.apache.zookeeper.KeeperException$NodeExistsException:

> KeeperErrorCode = NodeExists

> at

> org.apache.zookeeper.KeeperException.create(KeeperException.java:122)

> at

> kafka.zk.KafkaZkClient$CheckedEphemeral.getAfterNodeExists(KafkaZkClient.scala:1784)

> at

> kafka.zk.KafkaZkClient$CheckedEphemeral.create(KafkaZkClient.scala:1722)

> at

> kafka.zk.KafkaZkClient.checkedEphemeralCreate(KafkaZkClient.scala:1689)

> at kafka.zk.KafkaZkClient.registerBroker(KafkaZkClient.scala:97)

> at kafka.server.KafkaServer.startup(KafkaServer.scala:300)

> at

> io.confluent.support.metrics.SupportedServerStartable.startup(SupportedServerStartable.java:114)

> at

> io.confluent.support.metrics.SupportedKafka.main(SupportedKafka.java:6

> 6)

> CONFIDENTIAL NOTE:

> The information contained in this email is intended only for the use

> of the individual or entity named above and may contain information

> that is privileged, confidential and exempt from disclosure under applicable 
> law.

> If the reader of this message is not the intended recipient, you are

> hereby notified that any dissemination, distribution or copying of

> this communication is strictly prohibited. If you have received this

> message in error, please immediately notify the sender and delete the mail. 
> Thank you.

>





--

Santilli Jonathan



CO

Very Urgent: Error while creating ephemeral at /brokers/ids/0, node already exists and owner

[Sunil CHAUDHARI]

Hi,
Whenever I try to do rolling restart and start one of my broker, I get this 
error.
Can Anyone help me to get rid of this?

[2020-03-12 18:33:34,661] INFO Logs loading complete in 1553 ms. 
(kafka.log.LogManager)
[2020-03-12 18:33:34,677] INFO Starting log cleanup with a period of 30 ms. 
(kafka.log.LogManager)
[2020-03-12 18:33:34,678] INFO Starting log flusher with a default period of 
9223372036854775807 ms. (kafka.log.LogManager)
[2020-03-12 18:33:35,005] INFO Awaiting socket connections on 0.0.0.0:9092. 
(kafka.network.Acceptor)
[2020-03-12 18:33:35,049] INFO [SocketServer brokerId=0] Created data-plane 
acceptor and processors for endpoint : 
EndPoint(null,9092,ListenerName(PLAINTEXT),PLAINTEXT) 
(kafka.network.SocketServer)
[2020-03-12 18:33:35,050] INFO [SocketServer brokerId=0] Started 1 acceptor 
threads for data-plane (kafka.network.SocketServer)
[2020-03-12 18:33:35,109] INFO [ExpirationReaper-0-Produce]: Starting 
(kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2020-03-12 18:33:35,116] INFO [ExpirationReaper-0-ElectPreferredLeader]: 
Starting (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2020-03-12 18:33:35,116] INFO [ExpirationReaper-0-DeleteRecords]: Starting 
(kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2020-03-12 18:33:35,116] INFO [ExpirationReaper-0-Fetch]: Starting 
(kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2020-03-12 18:33:35,121] INFO [ExpirationReaper-0-ListOffsets]: Starting 
(kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2020-03-12 18:33:35,139] INFO [LogDirFailureHandler]: Starting 
(kafka.server.ReplicaManager$LogDirFailureHandler)
[2020-03-12 18:33:35,225] INFO Creating /brokers/ids/0 (is it secure? false) 
(kafka.zk.KafkaZkClient)
[2020-03-12 18:33:35,245] ERROR Error while creating ephemeral at 
/brokers/ids/0, node already exists and owner '216411990614016005' does not 
match current session '144234981936660481' 
(kafka.zk.KafkaZkClient$CheckedEphemeral)
[2020-03-12 18:33:35,252] ERROR [KafkaServer id=0] Fatal error during 
KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer)
org.apache.zookeeper.KeeperException$NodeExistsException: KeeperErrorCode = 
NodeExists
at org.apache.zookeeper.KeeperException.create(KeeperException.java:122)
at 
kafka.zk.KafkaZkClient$CheckedEphemeral.getAfterNodeExists(KafkaZkClient.scala:1784)
at 
kafka.zk.KafkaZkClient$CheckedEphemeral.create(KafkaZkClient.scala:1722)
at 
kafka.zk.KafkaZkClient.checkedEphemeralCreate(KafkaZkClient.scala:1689)
at kafka.zk.KafkaZkClient.registerBroker(KafkaZkClient.scala:97)
at kafka.server.KafkaServer.startup(KafkaServer.scala:300)
at 
io.confluent.support.metrics.SupportedServerStartable.startup(SupportedServerStartable.java:114)
at 
io.confluent.support.metrics.SupportedKafka.main(SupportedKafka.java:66)
CONFIDENTIAL NOTE:
The information contained in this email is intended only for the use of the 
individual or entity named above and may contain information that is 
privileged, confidential and exempt from disclosure under applicable law. If 
the reader of this message is not the intended recipient, you are hereby 
notified that any dissemination, distribution or copying of this communication 
is strictly prohibited. If you have received this message in error, please 
immediately notify the sender and delete the mail. Thank you.

Re: Adding additional nodes to Existing ZK cluster

[sunil chaudhari]

Hi Peter,
That was great explanation.
However I have question about the last stage where you mentioned to update
the zookeeper server in the services where single zookeeper is used.
Why do I need to do that?
Is it because only single zookeeper is used and you want to make sure high
availability of zookeeper?

What if tomorrow I add 2 more instances of zookeeper, total 5. Is it
required to update 2 new zK instances to my kafka brokers?


Regards,
Sunil.

On Sat, 7 Mar 2020 at 11:08 PM, Peter Bukowinski  wrote:

> This change will require brief interruption for services depending on the
> current zookeeper — but only for the amount of time it takes the service on
> the original zookeeper to restart. Here’s the basic process:
>
> 1. Provision two new zookeepers hosts, but don’t start the service on the
> new hosts.
> 2. Edit the zoo.cfg file on all hosts to contain the following lines
> (assuming default ports):
>
> server.1=ORIGINAL_ZK_IP:2888:3888
> server.2=SECOND_ZK_IP:2888:3888
> server.3=THIRD_ZK_IP:2888:3888
>
> 3. Ensure the myid file on the second node contains ‘2’ and on the third
> node contains ‘3'
> 4. Start the second and third zookeeper services and ensure they have
> become followers:
>
> echo stat | nc ZK2_IP 2181 | grep state
> echo stat | nc ZK3_IP 2181 | grep state
>
> 5. Restart the original zookeeper service and then check the state of all
> three zookeepers
>
> echo stat | nc ZK1_IP 2181 | grep state
> echo stat | nc ZK2_IP 2181 | grep state
> echo stat | nc ZK3_IP 2181 | grep state
>
> You should see that one of the new zookeepers has become the leader.
>
> Now all that’s left to do is update your zookeeper server strings in the
> services that were previously using the single zookeeper.
>
> Hope this helped!
>
> —
> Peter
>
> > On Mar 6, 2020, at 12:50 PM, JOHN, BIBIN  wrote:
> >
> > Team,
> > I currently have a 1 node ZK cluster and which is working fine. Now I
> want to add additional 2 more nodes to ZK cluster. Could you please provide
> best practice so I don't loose existing data?
> >
> >
> > Thanks
> > Bibin John
>
>

RE: Please help: How to print --reporting-interval in the perf metrics?

[Sunil CHAUDHARI]

Hello Experts,
Any thought on this?

From: Sunil CHAUDHARI
Sent: Tuesday, March 3, 2020 5:46 PM
To: users@kafka.apache.org
Subject: Please help: How to print --reporting-interval in the perf metrics?

Hi,
I want to test consumer perf using kafka-consumer-perf-test.sh
I am running below command:

./kafka-consumer-perf-test.sh --broker-list localhost:9092 --topic MB3P3R 
--messages 65495 --num-fetch-threads  9 --print-metrics --reporting-interval 
5000 --show-detailed-stats  > MB3P3R-consumer-Perf2.log

I am getting metrics output as below:  Its not printing anything under the 
highlighted columns.
Another question: Whatevere number of partitions and replicas my topic has, it 
always giving  approx value "2159" for records-consumed-rate

time, threadId, data.consumed.in.MB, MB.sec, data.consumed.in.nMsg, nMsg.sec, 
rebalance.time.ms, fetch.time.ms, fetch.MB.sec, fetch.nMsg.sec

Metric Name 
Value
consumer-coordinator-metrics:assigned-partitions:{client-id=consumer-1} 
  : 2.000
consumer-coordinator-metrics:commit-latency-avg:{client-id=consumer-1}  
  : 6.000
consumer-coordinator-metrics:commit-latency-max:{client-id=consumer-1}  
  : 6.000
consumer-coordinator-metrics:commit-rate:{client-id=consumer-1} 
  : 0.033
consumer-coordinator-metrics:commit-total:{client-id=consumer-1}
  : 1.000
consumer-coordinator-metrics:heartbeat-rate:{client-id=consumer-1}  
  : 0.000
consumer-coordinator-metrics:heartbeat-response-time-max:{client-id=consumer-1} 
  : NaN
consumer-coordinator-metrics:heartbeat-total:{client-id=consumer-1} 
  : 0.000
consumer-coordinator-metrics:join-rate:{client-id=consumer-1}   
  : 0.033

And so on...


CONFIDENTIAL NOTE:
The information contained in this email is intended only for the use of the 
individual or entity named above and may contain information that is 
privileged, confidential and exempt from disclosure under applicable law. If 
the reader of this message is not the intended recipient, you are hereby 
notified that any dissemination, distribution or copying of this communication 
is strictly prohibited. If you have received this message in error, please 
immediately notify the sender and delete the mail. Thank you.

RE: Producer-Perf-test

[Sunil CHAUDHARI]

Hi Himanshu,
Pasting screenshot for better view.
My Observations:

  *   When I increased partitions, and keep replica same, then Throughput 
increases. Compare Row  1 and 2
  *   Increase number of replica  and  partitions same: Throughput decreased 
than before. Row 2 and 3
  *   Increase number of partitions and keep replica same as before: TP 
increased.  Row 3 and 4
  *   Increase number of Replica and keep partitions same: TP decreased. Row 4 
and 5.

Question: Does it make any impact on throughput when number of partitions and 
replicas are equal?


[cid:image001.png@01D5F202.89FE2BD0]
From: Sunil CHAUDHARI
Sent: Tuesday, March 3, 2020 2:43 PM
To: users@kafka.apache.org
Subject: Producer-Perf-test

Hi,
I have done performance testing of Kafka cluster using 
kafka-producer-perf-test.sh
I created diff type of topics and did perf testing. Example: MB1P1R= MB is my 
topic name with 1 Partition and 1 replica.
I have 3 nodes cluster.

My Observations:

  *   When I increased partitions, and keep replica same, then Throughput 
increases. Compare Row  1 and 2
  *   Increase number of replica  and  partitions same: Throughput decreased 
than before. Row 2 and 3
  *   Increase number of partitions and keep replica same as before: TP 
increased.  Row 3 and 4
  *   Increase number of Replica and keep partitions same: TP decreased. Row 4 
and 5.

Question: Does it make any impact on throughput when number of partitions and 
replicas are equal?

Sr NO

Topic

Partitions

Replicas

Network.threads

IO-Threads

Records Sent

Batch-size-avg

Throughput

Latency (ms)

Records/sec

Size/sec

Avg

Max

1

MB1P1R

1

1

3

8

65495

16220

68653.03983

13.78 MB/sec

163.67

245

2

MB2P1R

2

1

3

8

65495

15865.719

58269.57295

11.70 MB/sec

173.81

470

3

MB2P2R

2

2

3

8

65495

16202.813

46286.21908

9.29 MB/sec

417.33

704

4

MB3P2R

3

2

3

8

65495

16184

56412.57537

11.32 MB/sec

229.82

550

5

MB3P3R

3

3

3

8

65495



46417.43444

9.32 MB/sec

411.44

705



Regards,
Sunil.
CONFIDENTIAL NOTE:
The information contained in this email is intended only for the use of the 
individual or entity named above and may contain information that is 
privileged, confidential and exempt from disclosure under applicable law. If 
the reader of this message is not the intended recipient, you are hereby 
notified that any dissemination, distribution or copying of this communication 
is strictly prohibited. If you have received this message in error, please 
immediately notify the sender and delete the mail. Thank you.

Re: Producer-Perf-test

[sunil chaudhari]

Hi Himanshu,
Sorry but I pasted from excel. Dont know how it got messed up?

Will resend it.

On Tue, 3 Mar 2020 at 6:48 PM, Himanshu Shukla 
wrote:

> could you please share the result in some proper way? Each field is line by
> line.
>
> On Tue, Mar 3, 2020 at 2:43 PM Sunil CHAUDHARI
>  wrote:
>
> > Hi,
> > I have done performance testing of Kafka cluster using
> > kafka-producer-perf-test.sh
> > I created diff type of topics and did perf testing. Example: MB1P1R= MB
> is
> > my topic name with 1 Partition and 1 replica.
> > I have 3 nodes cluster.
> >
> > My Observations:
> >
> >   *   When I increased partitions, and keep replica same, then Throughput
> > increases. Compare Row  1 and 2
> >   *   Increase number of replica  and  partitions same: Throughput
> > decreased than before. Row 2 and 3
> >   *   Increase number of partitions and keep replica same as before: TP
> > increased.  Row 3 and 4
> >   *   Increase number of Replica and keep partitions same: TP decreased.
> > Row 4 and 5.
> >
> > Question: Does it make any impact on throughput when number of partitions
> > and replicas are equal?
> >
> > Sr NO
> >
> > Topic
> >
> > Partitions
> >
> > Replicas
> >
> > Network.threads
> >
> > IO-Threads
> >
> > Records Sent
> >
> > Batch-size-avg
> >
> > Throughput
> >
> > Latency (ms)
> >
> > Records/sec
> >
> > Size/sec
> >
> > Avg
> >
> > Max
> >
> > 1
> >
> > MB1P1R
> >
> > 1
> >
> > 1
> >
> > 3
> >
> > 8
> >
> > 65495
> >
> > 16220
> >
> > 68653.03983
> >
> > 13.78 MB/sec
> >
> > 163.67
> >
> > 245
> >
> > 2
> >
> > MB2P1R
> >
> > 2
> >
> > 1
> >
> > 3
> >
> > 8
> >
> > 65495
> >
> > 15865.719
> >
> > 58269.57295
> >
> > 11.70 MB/sec
> >
> > 173.81
> >
> > 470
> >
> > 3
> >
> > MB2P2R
> >
> > 2
> >
> > 2
> >
> > 3
> >
> > 8
> >
> > 65495
> >
> > 16202.813
> >
> > 46286.21908
> >
> > 9.29 MB/sec
> >
> > 417.33
> >
> > 704
> >
> > 4
> >
> > MB3P2R
> >
> > 3
> >
> > 2
> >
> > 3
> >
> > 8
> >
> > 65495
> >
> > 16184
> >
> > 56412.57537
> >
> > 11.32 MB/sec
> >
> > 229.82
> >
> > 550
> >
> > 5
> >
> > MB3P3R
> >
> > 3
> >
> > 3
> >
> > 3
> >
> > 8
> >
> > 65495
> >
> >
> >
> > 46417.43444
> >
> > 9.32 MB/sec
> >
> > 411.44
> >
> > 705
> >
> >
> >
> > Regards,
> > Sunil.
> > CONFIDENTIAL NOTE:
> > The information contained in this email is intended only for the use of
> > the individual or entity named above and may contain information that is
> > privileged, confidential and exempt from disclosure under applicable law.
> > If the reader of this message is not the intended recipient, you are
> hereby
> > notified that any dissemination, distribution or copying of this
> > communication is strictly prohibited. If you have received this message
> in
> > error, please immediately notify the sender and delete the mail. Thank
> you.
> >
>
>
> --
> Regards,
> Himanshu Shukla
>

Please help: How to print --reporting-interval in the perf metrics?

[Sunil CHAUDHARI]

Hi,
I want to test consumer perf using kafka-consumer-perf-test.sh
I am running below command:

./kafka-consumer-perf-test.sh --broker-list localhost:9092 --topic MB3P3R 
--messages 65495 --num-fetch-threads  9 --print-metrics --reporting-interval 
5000 --show-detailed-stats  > MB3P3R-consumer-Perf2.log

I am getting metrics output as below:  Its not printing anything under the 
highlighted columns.
Another question: Whatevere number of partitions and replicas my topic has, it 
always giving  approx value "2159" for records-consumed-rate

time, threadId, data.consumed.in.MB, MB.sec, data.consumed.in.nMsg, nMsg.sec, 
rebalance.time.ms, fetch.time.ms, fetch.MB.sec, fetch.nMsg.sec

Metric Name 
Value
consumer-coordinator-metrics:assigned-partitions:{client-id=consumer-1} 
  : 2.000
consumer-coordinator-metrics:commit-latency-avg:{client-id=consumer-1}  
  : 6.000
consumer-coordinator-metrics:commit-latency-max:{client-id=consumer-1}  
  : 6.000
consumer-coordinator-metrics:commit-rate:{client-id=consumer-1} 
  : 0.033
consumer-coordinator-metrics:commit-total:{client-id=consumer-1}
  : 1.000
consumer-coordinator-metrics:heartbeat-rate:{client-id=consumer-1}  
  : 0.000
consumer-coordinator-metrics:heartbeat-response-time-max:{client-id=consumer-1} 
  : NaN
consumer-coordinator-metrics:heartbeat-total:{client-id=consumer-1} 
  : 0.000
consumer-coordinator-metrics:join-rate:{client-id=consumer-1}   
  : 0.033

And so on...


CONFIDENTIAL NOTE:
The information contained in this email is intended only for the use of the 
individual or entity named above and may contain information that is 
privileged, confidential and exempt from disclosure under applicable law. If 
the reader of this message is not the intended recipient, you are hereby 
notified that any dissemination, distribution or copying of this communication 
is strictly prohibited. If you have received this message in error, please 
immediately notify the sender and delete the mail. Thank you.

Producer-Perf-test

[Sunil CHAUDHARI]

Hi,
I have done performance testing of Kafka cluster using 
kafka-producer-perf-test.sh
I created diff type of topics and did perf testing. Example: MB1P1R= MB is my 
topic name with 1 Partition and 1 replica.
I have 3 nodes cluster.

My Observations:

  *   When I increased partitions, and keep replica same, then Throughput 
increases. Compare Row  1 and 2
  *   Increase number of replica  and  partitions same: Throughput decreased 
than before. Row 2 and 3
  *   Increase number of partitions and keep replica same as before: TP 
increased.  Row 3 and 4
  *   Increase number of Replica and keep partitions same: TP decreased. Row 4 
and 5.

Question: Does it make any impact on throughput when number of partitions and 
replicas are equal?

Sr NO

Topic

Partitions

Replicas

Network.threads

IO-Threads

Records Sent

Batch-size-avg

Throughput

Latency (ms)

Records/sec

Size/sec

Avg

Max

1

MB1P1R

1

1

3

8

65495

16220

68653.03983

13.78 MB/sec

163.67

245

2

MB2P1R

2

1

3

8

65495

15865.719

58269.57295

11.70 MB/sec

173.81

470

3

MB2P2R

2

2

3

8

65495

16202.813

46286.21908

9.29 MB/sec

417.33

704

4

MB3P2R

3

2

3

8

65495

16184

56412.57537

11.32 MB/sec

229.82

550

5

MB3P3R

3

3

3

8

65495



46417.43444

9.32 MB/sec

411.44

705



Regards,
Sunil.
CONFIDENTIAL NOTE:
The information contained in this email is intended only for the use of the 
individual or entity named above and may contain information that is 
privileged, confidential and exempt from disclosure under applicable law. If 
the reader of this message is not the intended recipient, you are hereby 
notified that any dissemination, distribution or copying of this communication 
is strictly prohibited. If you have received this message in error, please 
immediately notify the sender and delete the mail. Thank you.

Re: when to expand cluster

[sunil chaudhari]

Hi,
I am now in the process of deciding partitions and replicas for my cluster.
I am making use of perf test utilities and it really helps a lot.
Just measure perf by creating multiple topics with same number of recodrs
with diff partitions and replicas.
Then compare the througput and also look at latency.
If possible change network and io threads at broker level and then again do
perf testing for above scenarios.

Thanks,
Sunil

On Fri, 28 Feb 2020 at 11:40 AM, 张祥  wrote:

> Thanks, it helps a lot.
>
> Peter Bukowinski  于2020年2月28日周五 上午5:18写道：
>
> > No, it’s not bad. Kafka is designed to serve data to many consumers at
> the
> > same time, whether they are independent of each other or in the same
> > consumer group.
> >
> > I would encourage you to play with different partition counts and use
> > kafka’s performance testing tools (kafka-producer-perf-test.sh and
> > kafka-consumer-perf-test.sh) to test throughput in different scenarios
> and
> > see the results for yourself.
> >
> > —
> > Peter
> >
> > > On Feb 27, 2020, at 1:28 AM, 张祥  wrote:
> > >
> > > I believe no matter the partition count exceeds the broker count, we
> can
> > > always have the same number of consumer instances as the partition
> count.
> > >
> > > So what I want to know is when two partition exists on the same broker,
> > two
> > > consumer instances will be talking to same broker, is that bad ?
> > >
> > > 张祥  于2020年2月27日周四 下午2:20写道：
> > >
> > >> Thanks. What influence does it have for consumers and producers when
> > >> partition number is more than broker number, which means at least one
> > >> broker serves two partitions for one topic ? performance wise.
> > >>
> > >> Peter Bukowinski  于2020年2月26日周三 下午11:02写道：
> > >>
> > >>> Disk usage is one reason to expand. Another reason is if you need
> more
> > >>> ingest or output throughout for your topic data. If your producers
> > aren’t
> > >>> able to send data to kafka fast enough or your consumers are lagging,
> > you
> > >>> might benefit from more brokers and more partitions.
> > >>>
> > >>> -- Peter
> > >>>
> >  On Feb 26, 2020, at 12:56 AM, 张祥  wrote:
> > 
> >  In documentation, it is described how to expand cluster:
> > 
> > >>>
> >
> https://kafka.apache.org/20/documentation.html#basic_ops_cluster_expansion
> > >>> .
> >  But I am wondering what the criteria for expand is. I can only think
> > of
> >  disk usage threshold. For example, suppose several disk usage exceed
> > >>> 80%.
> >  Is this correct and is there more ?
> > >>>
> > >>
> >
> >
>

RE: HELP in Usage of JMX port in Kafka

[Sunil CHAUDHARI]

Hi All,
Sorry to bother you all.
It was simple. 😊
Just put one line in required .sh file
export JMX_PORT=

and it will run 😊

thanks,
Sunil.

From: Sunil CHAUDHARI
Sent: Friday, February 28, 2020 10:06 AM
To: users@kafka.apache.org
Subject: HELP in Usage of JMX port in Kafka

Hi all,
I have used JMX_PORT 9099 in environment variable and started Kafka.
There is not problem till now. I can see metrics on kafka-manager console. This 
is fine.

However when I run kafka-consumer-perf-test.sh and kafka-producer-perf-test.sh  
and similar utility under /bin then I get error given at the end:
I understand the root cause. But I don’t know solution.
Cant I make use of those test utilities given under /bin, when I enabled JMX?
I want to measure performance of running kafka cluster and at the same time 
want to monitor metrics on kafka-manager.
How can I skip JMX_port while running manual utilities? Or is there a way I can 
give different port for  kafka-producer-perf-test.sh ?
Please help me in this regards.

Error: Exception thrown by the agent : java.rmi.server.ExportException: Port 
already in use: 9099; nested exception is:
java.net.BindException: Address already in use (Bind failed)
jdk.internal.agent.AgentConfigurationError: java.rmi.server.ExportException: 
Port already in use: 9099; nested exception is:
java.net.BindException: Address already in use (Bind failed)
at 
jdk.management.agent/sun.management.jmxremote.ConnectorBootstrap.startRemoteConnectorServer(ConnectorBootstrap.java:491)
at 
jdk.management.agent/jdk.internal.agent.Agent.startAgent(Agent.java:447)
at 
jdk.management.agent/jdk.internal.agent.Agent.startAgent(Agent.java:599)
Caused by: java.rmi.server.ExportException: Port already in use: 9099; nested 
exception is:
java.net.BindException: Address already in use (Bind failed)
at 
java.rmi/sun.rmi.transport.tcp.TCPTransport.listen(TCPTransport.java:335)
at 
java.rmi/sun.rmi.transport.tcp.TCPTransport.exportObject(TCPTransport.java:243)
at 
java.rmi/sun.rmi.transport.tcp.TCPEndpoint.exportObject(TCPEndpoint.java:411)
at java.rmi/sun.rmi.transport.LiveRef.exportObject(LiveRef.java:147)
at 
java.rmi/sun.rmi.server.UnicastServerRef.exportObject(UnicastServerRef.java:233)
at java.rmi/sun.rmi.registry.RegistryImpl.setup(RegistryImpl.java:220)
at java.rmi/sun.rmi.registry.RegistryImpl.(RegistryImpl.java:180)
at 
jdk.management.agent/sun.management.jmxremote.SingleEntryRegistry.(SingleEntryRegistry.java:49)
at 
jdk.management.agent/sun.management.jmxremote.ConnectorBootstrap.exportMBeanServer(ConnectorBootstrap.java:836)
at 
jdk.management.agent/sun.management.jmxremote.ConnectorBootstrap.startRemoteConnectorServer(ConnectorBootstrap.java:479)
... 2 more
Caused by: java.net.BindException: Address already in use (Bind failed)
at java.base/java.net.PlainSocketImpl.socketBind(Native Method)
at 
java.base/java.net.AbstractPlainSocketImpl.bind(AbstractPlainSocketImpl.java:436)
at java.base/java.net.ServerSocket.bind(ServerSocket.java:381)
at java.base/java.net.ServerSocket.(ServerSocket.java:243)
at java.base/java.net.ServerSocket.(ServerSocket.java:135)
at 
java.rmi/sun.rmi.transport.tcp.TCPDirectSocketFactory.createServerSocket(TCPDirectSocketFactory.java:45)
at 
java.rmi/sun.rmi.transport.tcp.TCPEndpoint.newServerSocket(TCPEndpoint.java:666)
at 
java.rmi/sun.rmi.transport.tcp.TCPTransport.listen(TCPTransport.java:324)
... 11 more
CONFIDENTIAL NOTE:
The information contained in this email is intended only for the use of the 
individual or entity named above and may contain information that is 
privileged, confidential and exempt from disclosure under applicable law. If 
the reader of this message is not the intended recipient, you are hereby 
notified that any dissemination, distribution or copying of this communication 
is strictly prohibited. If you have received this message in error, please 
immediately notify the sender and delete the mail. Thank you.

HELP in Usage of JMX port in Kafka

[Sunil CHAUDHARI]

Hi all,
I have used JMX_PORT 9099 in environment variable and started Kafka.
There is not problem till now. I can see metrics on kafka-manager console. This 
is fine.

However when I run kafka-consumer-perf-test.sh and kafka-producer-perf-test.sh  
and similar utility under /bin then I get error given at the end:
I understand the root cause. But I don't know solution.
Cant I make use of those test utilities given under /bin, when I enabled JMX?
I want to measure performance of running kafka cluster and at the same time 
want to monitor metrics on kafka-manager.
How can I skip JMX_port while running manual utilities? Or is there a way I can 
give different port for  kafka-producer-perf-test.sh ?
Please help me in this regards.

Error: Exception thrown by the agent : java.rmi.server.ExportException: Port 
already in use: 9099; nested exception is:
java.net.BindException: Address already in use (Bind failed)
jdk.internal.agent.AgentConfigurationError: java.rmi.server.ExportException: 
Port already in use: 9099; nested exception is:
java.net.BindException: Address already in use (Bind failed)
at 
jdk.management.agent/sun.management.jmxremote.ConnectorBootstrap.startRemoteConnectorServer(ConnectorBootstrap.java:491)
at 
jdk.management.agent/jdk.internal.agent.Agent.startAgent(Agent.java:447)
at 
jdk.management.agent/jdk.internal.agent.Agent.startAgent(Agent.java:599)
Caused by: java.rmi.server.ExportException: Port already in use: 9099; nested 
exception is:
java.net.BindException: Address already in use (Bind failed)
at 
java.rmi/sun.rmi.transport.tcp.TCPTransport.listen(TCPTransport.java:335)
at 
java.rmi/sun.rmi.transport.tcp.TCPTransport.exportObject(TCPTransport.java:243)
at 
java.rmi/sun.rmi.transport.tcp.TCPEndpoint.exportObject(TCPEndpoint.java:411)
at java.rmi/sun.rmi.transport.LiveRef.exportObject(LiveRef.java:147)
at 
java.rmi/sun.rmi.server.UnicastServerRef.exportObject(UnicastServerRef.java:233)
at java.rmi/sun.rmi.registry.RegistryImpl.setup(RegistryImpl.java:220)
at java.rmi/sun.rmi.registry.RegistryImpl.(RegistryImpl.java:180)
at 
jdk.management.agent/sun.management.jmxremote.SingleEntryRegistry.(SingleEntryRegistry.java:49)
at 
jdk.management.agent/sun.management.jmxremote.ConnectorBootstrap.exportMBeanServer(ConnectorBootstrap.java:836)
at 
jdk.management.agent/sun.management.jmxremote.ConnectorBootstrap.startRemoteConnectorServer(ConnectorBootstrap.java:479)
... 2 more
Caused by: java.net.BindException: Address already in use (Bind failed)
at java.base/java.net.PlainSocketImpl.socketBind(Native Method)
at 
java.base/java.net.AbstractPlainSocketImpl.bind(AbstractPlainSocketImpl.java:436)
at java.base/java.net.ServerSocket.bind(ServerSocket.java:381)
at java.base/java.net.ServerSocket.(ServerSocket.java:243)
at java.base/java.net.ServerSocket.(ServerSocket.java:135)
at 
java.rmi/sun.rmi.transport.tcp.TCPDirectSocketFactory.createServerSocket(TCPDirectSocketFactory.java:45)
at 
java.rmi/sun.rmi.transport.tcp.TCPEndpoint.newServerSocket(TCPEndpoint.java:666)
at 
java.rmi/sun.rmi.transport.tcp.TCPTransport.listen(TCPTransport.java:324)
... 11 more
CONFIDENTIAL NOTE:
The information contained in this email is intended only for the use of the 
individual or entity named above and may contain information that is 
privileged, confidential and exempt from disclosure under applicable law. If 
the reader of this message is not the intended recipient, you are hereby 
notified that any dissemination, distribution or copying of this communication 
is strictly prohibited. If you have received this message in error, please 
immediately notify the sender and delete the mail. Thank you.

PCF to kafka message sending

[Sunil CHAUDHARI]

Hi,
We have one case where we want to send messages from PCF to Kafka endpoints.
Is it possible?  How?

Regards,
Sunil.
CONFIDENTIAL NOTE:
The information contained in this email is intended only for the use of the 
individual or entity named above and may contain information that is 
privileged, confidential and exempt from disclosure under applicable law. If 
the reader of this message is not the intended recipient, you are hereby 
notified that any dissemination, distribution or copying of this communication 
is strictly prohibited. If you have received this message in error, please 
immediately notify the sender and delete the mail. Thank you.

Unsupported protocol[TLSv1.2

[Sunil CHAUDHARI]

Hi,
I just configured SSL on 3 brokers
Here is my configuration:
I just replaced hostnames with dummy hostname.

inter.broker.listener.name=CLIENT
listeners=CLIENT://dummyhost.mycom.com:9092,SSL://dummyhost.mycom.com:9093
advertised.listeners=CLIENT://dummyhost.mycom.com:9092
#security.inter.broker.protocol = SSL
ssl.keystore.location = /kafka/certs/myproj-kafka.jks
ssl.keystore.password = xxx
ssl.key.password = xxx
ssl.truststore.location = /kafka/certs/trustchain.jks
ssl.truststore.password = xxx
ssl.enabled.protocols = [TLSv1.2, TLSv1.3]
listener.security.protocol.map=CLIENT:PLAINTEXT,PLAINTEXT:PLAINTEXT,SSL:SSL,SASL_PLAINTEXT:SASL_PLAINTEXT,SASL_SSL:SASL_SSL

After this configuration is done, I started 3 brokers. I am getting below 
errors in logs. Please help to understand this.

[2020-02-25 18:00:59,446] INFO [GroupCoordinator 1]: Member 
consumer-186-a4e444ba-db55-45db-9d09-f3e698f14bf1 in group KMOffsetCache- 
dummyhost1.mycom.com has failed, removing it from the group 
(kafka.coordinator.group.GroupCoordinator)
Feb 25, 2020 6:02:48 PM sun.rmi.transport.tcp.TCPTransport$AcceptLoop run
WARNING: RMI TCP Accept-0: accept loop for 
ServerSocket[addr=0.0.0.0/0.0.0.0,localport=34895] throws
java.io.IOException: The server sockets created using the 
LocalRMIServerSocketFactory only accept connections from clients running on the 
host where the RMI remote objects have been exported.
at 
jdk.management.agent/sun.management.jmxremote.LocalRMIServerSocketFactory$1.accept(LocalRMIServerSocketFactory.java:114)
at 
java.rmi/sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:394)
at 
java.rmi/sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:366)
at java.base/java.lang.Thread.run(Thread.java:834)
[2020-02-25 18:02:56,277] INFO Failed to create channel due to  
(org.apache.kafka.common.network.SslChannelBuilder)
java.lang.IllegalArgumentException: Unsupported protocol[TLSv1.2
at 
java.base/sun.security.ssl.ProtocolVersion.namesOf(ProtocolVersion.java:292)
at 
java.base/sun.security.ssl.SSLEngineImpl.setEnabledProtocols(SSLEngineImpl.java:789)
at 
org.apache.kafka.common.security.ssl.SslEngineBuilder.createSslEngine(SslEngineBuilder.java:209)
at 
org.apache.kafka.common.security.ssl.SslFactory.createSslEngine(SslFactory.java:169)
at 
org.apache.kafka.common.network.SslChannelBuilder.buildTransportLayer(SslChannelBuilder.java:115)
at 
org.apache.kafka.common.network.SslChannelBuilder.buildChannel(SslChannelBuilder.java:100)
at 
org.apache.kafka.common.network.Selector.buildAndAttachKafkaChannel(Selector.java:336)
at 
org.apache.kafka.common.network.Selector.registerChannel(Selector.java:327)
at org.apache.kafka.common.network.Selector.register(Selector.java:314)
at 
kafka.network.Processor.configureNewConnections(SocketServer.scala:1014)
at kafka.network.Processor.run(SocketServer.scala:759)
at java.base/java.lang.Thread.run(Thread.java:834)
[2020-02-25 18:02:56,281] ERROR Processor 3 closed connection from null 
(kafka.network.Processor)
java.io.IOException: Channel could not be created for socket 
java.nio.channels.SocketChannel[closed]
at 
org.apache.kafka.common.network.Selector.buildAndAttachKafkaChannel(Selector.java:345)
at 
org.apache.kafka.common.network.Selector.registerChannel(Selector.java:327)
at org.apache.kafka.common.network.Selector.register(Selector.java:314)
at 
kafka.network.Processor.configureNewConnections(SocketServer.scala:1014)
at kafka.network.Processor.run(SocketServer.scala:759)
at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: org.apache.kafka.common.KafkaException: 
java.lang.IllegalArgumentException: Unsupported protocol[TLSv1.2
at 
org.apache.kafka.common.network.SslChannelBuilder.buildChannel(SslChannelBuilder.java:106)
at 
org.apache.kafka.common.network.Selector.buildAndAttachKafkaChannel(Selector.java:336)
... 5 more
Caused by: java.lang.IllegalArgumentException: Unsupported protocol[TLSv1.2
at 
java.base/sun.security.ssl.ProtocolVersion.namesOf(ProtocolVersion.java:292)
at 
java.base/sun.security.ssl.SSLEngineImpl.setEnabledProtocols(SSLEngineImpl.java:789)
at 
org.apache.kafka.common.security.ssl.SslEngineBuilder.createSslEngine(SslEngineBuilder.java:209)
at 
org.apache.kafka.common.security.ssl.SslFactory.createSslEngine(SslFactory.java:169)
at 
org.apache.kafka.common.network.SslChannelBuilder.buildTransportLayer(SslChannelBuilder.java:115)
at 
org.apache.kafka.common.network.SslChannelBuilder.buildChannel(SslChannelBuilder.java:100)
CONFIDENTIAL NOTE:
The information contained in this email is intended only for the use of the 
individual or entity named above and may contain information that is 
privileged, confidential and exempt from disclosure under applicable law. 

Sample configuration for inter.broker.listener.name

[Sunil CHAUDHARI]

Hello experts,
I am facing errors while starting broker with advertised.listeners.
Can someone send me the sample configuration where all below settings are 
mentioned with dummy hosts names alongwith SSL?
Listeners
advertised.listeners
inter.broker.listener.name


thanks
Sunil.
CONFIDENTIAL NOTE:
The information contained in this email is intended only for the use of the 
individual or entity named above and may contain information that is 
privileged, confidential and exempt from disclosure under applicable law. If 
the reader of this message is not the intended recipient, you are hereby 
notified that any dissemination, distribution or copying of this communication 
is strictly prohibited. If you have received this message in error, please 
immediately notify the sender and delete the mail. Thank you.

consumer-perf -test metrics

[Sunil CHAUDHARI]

Hi,
I have run consumer performance test on my kafka cluster.
Can you please help me to understand below parameters?  Basically I don't know 
whats unit of those parameters? I cant assume it blindly
Its only given in 2 columnts,  the "Metric Name" and its "Value"

Metric Name 

ValueUnit I assume?
consumer-coordinator-metrics:commit-latency-avg:{client-id=consumer-1}  
 : 4.000
 ms
consumer-coordinator-metrics:commit-latency-max:{client-id=consumer-1}  
 : 4.000
ms
consumer-coordinator-metrics:commit-rate:{client-id=consumer-1} 
  : 0.033   
 ?
consumer-coordinator-metrics:commit-total:{client-id=consumer-1}
  : 1.000   
 ?
consumer-fetch-manager-metrics:fetch-rate:{client-id=consumer-1}
: 0.493 
  ?
consumer-fetch-manager-metrics:fetch-size-avg:{client-id=consumer-1, 
topic=mbtest1}: 958250.400   ?
consumer-fetch-manager-metrics:fetch-size-avg:{client-id=consumer-1}
 : 958250.400  ?
consumer-fetch-manager-metrics:fetch-size-max:{client-id=consumer-1, 
topic=mbtest1}: 71.000   ?
consumer-fetch-manager-metrics:fetch-size-max:{client-id=consumer-1}
 : 71.000  ?
consumer-fetch-manager-metrics:fetch-total:{client-id=consumer-1}   
: 15.000
  ?
consumer-fetch-manager-metrics:records-per-request-avg:{client-id=consumer-1, 
topic=mbtest1}   : 4366.333
consumer-fetch-manager-metrics:records-per-request-avg:{client-id=consumer-1}   
: 4366.333


consumer-fetch-manager-metrics:records-lead-avg:{client-id=consumer-1, 
topic=mbtest1, partition=0} : 32679.907
consumer-fetch-manager-metrics:records-lead-min:{client-id=consumer-1, 
topic=mbtest1, partition=0} : 500.000
consumer-fetch-manager-metrics:records-lead-min:{client-id=consumer-1}  
   : 500.000
consumer-fetch-manager-metrics:records-lead:{client-id=consumer-1, 
topic=mbtest1, partition=0} : 65495.000

Whats mean by recods-lead?

Can I consider below metrics as throughput of my consumer?
consumer-fetch-manager-metrics:records-consumed-rate:{client-id=consumer-1, 
topic=mbtest1} : 2159.769
consumer-fetch-manager-metrics:records-consumed-rate:{client-id=consumer-1} 
   : 2159.698

What is diff between in "records-per-request-avg" and "records-consumed-rate" ?

Regards,
Sunil.
CONFIDENTIAL NOTE:
The information contained in this email is intended only for the use of the 
individual or entity named above and may contain information that is 
privileged, confidential and exempt from disclosure under applicable law. If 
the reader of this message is not the intended recipient, you are hereby 
notified that any dissemination, distribution or copying of this communication 
is strictly prohibited. If you have received this message in error, please 
immediately notify the sender and delete the mail. Thank you.

Consumer-perf-test shows only headers, NO values

[sunil chaudhari]

Hi,
When I run Consumer-perf-test  from kafka bin directory to measure
performance, thne in the output file, it shows only headers.
Ot doesnt prints their values.
Any thought on this?

When I run producer-perf-test then it prints detailed information like,
Number of records read:
Number of records sent
Records per seconds, avg latency etc.

But in case of consumer perf, its not showing the throughput. How many
records read per seconds?


Regards,
Sunil Chaudhari

Re: Port already in use

[sunil chaudhari]

I have used this port to emable jmx while starting kafka.
Kafka started properly. I can see metrics on kafka manager console. Uptill
now its fine.

However when I started using utilities under /bin for some testing, those
are not running.

This has happened because i enable jmx port 9099.

When you say change port, then what does it
Mean?
Port of kafka jmx-port?



On Fri, 21 Feb 2020 at 5:18 PM, Karolis Pocius
 wrote:

> Choose a different port or check what's already listening on 9099 using
> something like: `ss -tunapl | grep 9099`
>
> On Fri, Feb 21, 2020 at 1:08 PM sunil chaudhari <
> sunilmchaudhar...@gmail.com>
> wrote:
>
> > Hi,
> > I just enlabled the Jmx port in kafka broker.
> > Since then I am not able to run utilities under /bin
> > Example when I run
> >  ./kafka-topics.sh —create
> > Then it throws bindException port already in use 9099
> >
> > Before it was running.
> > Same thing happening for perf test utilities under /bin.
> >
> > Please help. How can i run those utilities with JMx port ON.
> >
> > Regards,
> > Sunil.
> >
>

Port already in use

[sunil chaudhari]

Hi,
I just enlabled the Jmx port in kafka broker.
Since then I am not able to run utilities under /bin
Example when I run
 ./kafka-topics.sh —create
Then it throws bindException port already in use 9099

Before it was running.
Same thing happening for perf test utilities under /bin.

Please help. How can i run those utilities with JMx port ON.

Regards,
Sunil.

RE: [External] Re: Urgent helep please! How to measure producer and consumer throughput for single partition?

[Sunil CHAUDHARI]

Hi Liam and George,

Thanks for support.

Just now I got the same Jolokio from elastic documentation.

Checking its configurations.  There are no metrics mentioned by you .  
kafka.producer:type=producer-metrics,client-id=(.+),topic=(.+)record-send-rate

It seems I have to add few metrics in the jolokia.yml.   is that correct?



[cid:image001.png@01D5E8A9.3858D300]





-Original Message-
From: Liam Clarke 
Sent: Friday, February 21, 2020 10:52 AM
To: users@kafka.apache.org
Subject: Re: [External] Re: Urgent helep please! How to measure producer and 
consumer throughput for single partition?



This mail originated from an external party outside DBS - 
users-return-39580-sunilchaudhari=dbs@kafka.apache.org<mailto:users-return-39580-sunilchaudhari=dbs@kafka.apache.org>.
 Do not click on links or open attachments unless you recognize the sender and 
know the content is safe.



Hi Sunil,



Looks like Metricbeats has a Jolokia module that will capture JMX exposed 
metrics for you:

https://www.elastic.co/blog/brewing-in-beats-add-support-for-jolokia-lmx



Kind regards,



Liam Clarke



On Fri, Feb 21, 2020 at 6:16 PM Sunil CHAUDHARI 
mailto:sunilchaudh...@dbs.com.invalid>> wrote:



> Hi Liam Clarke,

> Thanks for this elaboration.

> Surely I will google.

> One more question about Mbeans. If I am capturing system matrix on

> kafka broker using metricbeat, then is it possible that I will get those 
> Mbeans?

> I know this is out of the topic, but in general, if I capture JVM

> metric with any of the monitoring tools, then will I get it?

>

>

>

> -Original Message-

> From: Liam Clarke 
> mailto:liam.cla...@adscale.co.nz>>

> Sent: Friday, February 21, 2020 1:03 AM

> To: users@kafka.apache.org<mailto:users@kafka.apache.org>

> Subject: Re: [External] Re: Urgent helep please! How to measure

> producer and consumer throughput for single partition?

>

> This mail originated from an external party outside DBS -

> users-return-39570-sunilchaudhari=dbs@kafka.apache.org<mailto:users-return-39570-sunilchaudhari=dbs@kafka.apache.org>.
>  Do not

> click on links or open attachments unless you recognize the sender and

> know the content is safe.

>

> The metrics are exposed in the JVM the producer is running within as

> Mbeans. The long string I gave you is the relevant MBean object name.

> You can connect to the JVM using JConsole to view the MBeans. There

> are also multiple libraries that will scrape a JVM via JMX to extract

> values from MBeans.

>

> If you're not familiar with JMX or JConsole, there's plenty of great

> documentation on the Internet, have a Google :)

>

> On Thu, 20 Feb. 2020, 11:51 pm Sunil CHAUDHARI,

> mailto:sunilchaudh...@dbs.com.invalid>>

> wrote:

>

> > Hi Liam Clarke,

> > Sorry but this is bit unclear for me.

> > Can you please elaborate your answer? I am beginner to Kafka.

> > " Producers emit metrics via JMX ":

> > - How to enable this? I have kafka-Manager. Can I make use

> > of kafka-manager? How?

> >

> “kafka.producer:type=producer-metrics,client-id=(.+),topic=(.+)record-send-rate”

> > please help to explain this.

> >

> > Regards,

> > Sunil.

> >

> > -Original Message-

> > From: Liam Clarke 
> > mailto:liam.cla...@adscale.co.nz>>

> > Sent: Thursday, February 20, 2020 11:16 AM

> > To: users@kafka.apache.org<mailto:users@kafka.apache.org>

> > Subject: [External] Re: Urgent helep please! How to measure producer

> > and consumer throughput for single partition?

> >

> > This mail originated from an external party outside DBS - mailto:

> > users-return-39554-sunilchaudhari=dbs@kafka.apache.org<mailto:users-return-39554-sunilchaudhari=dbs@kafka.apache.org>.
> >  Do not

> > click on links or open attachments unless you recognize the sender

> > and know the content is safe.

> >

> > Hi Sunil,

> >

> > Producers emit metrics via JMX that will help you, assuming that

> > your producers are using a round robin partition assignment

> > strategy, you could divide this metric by your number of partitions,

> >

> >

> > kafka.producer:type=producer-metrics,client-id=(.+),topic=(.+)record

> > -s

> > end-rate

> >

> > Kind regards,

> >

> > Liam Clarke

> >

> > On Thu, 20 Feb. 2020, 5:57 pm Sunil CHAUDHARI,  > sunilchaudh...@dbs.com.invalid<mailto:sunilchaudh...@dbs.com.invalid>> 
> > wrote:

> >

> > > Hi

> > > I was referring to the article by Mr. June Rao about partitions in

> > > kafka cluster.

&

RE: [External] Re: Urgent helep please! How to measure producer and consumer throughput for single partition?

[Sunil CHAUDHARI]

Hi Liam Clarke,
Thanks for this elaboration.
Surely I will google.
One more question about Mbeans. If I am capturing system matrix on kafka broker 
using metricbeat, then is it possible that I will get those Mbeans?
I know this is out of the topic, but in general, if I capture JVM metric with 
any of the monitoring tools, then will I get it?



-Original Message-
From: Liam Clarke 
Sent: Friday, February 21, 2020 1:03 AM
To: users@kafka.apache.org
Subject: Re: [External] Re: Urgent helep please! How to measure producer and 
consumer throughput for single partition?

This mail originated from an external party outside DBS - 
users-return-39570-sunilchaudhari=dbs@kafka.apache.org. Do not click on 
links or open attachments unless you recognize the sender and know the content 
is safe.

The metrics are exposed in the JVM the producer is running within as Mbeans. 
The long string I gave you is the relevant MBean object name. You can connect 
to the JVM using JConsole to view the MBeans. There are also multiple libraries 
that will scrape a JVM via JMX to extract values from MBeans.

If you're not familiar with JMX or JConsole, there's plenty of great 
documentation on the Internet, have a Google :)

On Thu, 20 Feb. 2020, 11:51 pm Sunil CHAUDHARI, 
 wrote:

> Hi Liam Clarke,
> Sorry but this is bit unclear for me.
> Can you please elaborate your answer? I am beginner to Kafka.
> " Producers emit metrics via JMX ":
> - How to enable this? I have kafka-Manager. Can I make use of
> kafka-manager? How?
> “kafka.producer:type=producer-metrics,client-id=(.+),topic=(.+)record-send-rate”
> please help to explain this.
>
> Regards,
> Sunil.
>
> -Original Message-
> From: Liam Clarke 
> Sent: Thursday, February 20, 2020 11:16 AM
> To: users@kafka.apache.org
> Subject: [External] Re: Urgent helep please! How to measure producer
> and consumer throughput for single partition?
>
> This mail originated from an external party outside DBS - mailto:
> users-return-39554-sunilchaudhari=dbs@kafka.apache.org. Do not
> click on links or open attachments unless you recognize the sender and
> know the content is safe.
>
> Hi Sunil,
>
> Producers emit metrics via JMX that will help you, assuming that your
> producers are using a round robin partition assignment strategy, you
> could divide this metric by your number of partitions,
>
>
> kafka.producer:type=producer-metrics,client-id=(.+),topic=(.+)record-s
> end-rate
>
> Kind regards,
>
> Liam Clarke
>
> On Thu, 20 Feb. 2020, 5:57 pm Sunil CHAUDHARI,  sunilchaudh...@dbs.com.invalid> wrote:
>
> > Hi
> > I was referring to the article by Mr. June Rao about partitions in
> > kafka cluster.
> > https://www.confluent.io/blog/how-choose-number-topics-partitions-ka
> > fk
> > a-cluster/
> >
> > "A rough formula for picking the number of partitions is based on
> > throughput. You measure the throughout that you can achieve on a
> > single partition for production (call it p) and consumption (call it
> > c). Let's say your target throughput is t. Then you need to have at
> > least max(t/p, t/c) partitions."
> >
> > I have the data pipeline as below.
> >
> > Filebeat-->Kafka-->Logstash-->Elasticsearch
> > There are many filebeat agents sending data to kafka. I want to
> > understand , how can I measure the events per seconds getting
> > written to Kafka? This will help me to know 'p'  in above formula.
> > I can measure the consumer throughput by monitoring logsatsh
> > pipelines on Kibana. So it will give me 'c' in above formula.
> >
> > I know target throughput in my cluster, that is 't'. 30k events/s.
> >
> > Please let me know if I am going wrong?
> >
> > Regards,
> > Sunil.
> > CONFIDENTIAL NOTE:
> > The information contained in this email is intended only for the use
> > of the individual or entity named above and may contain information
> > that is privileged, confidential and exempt from disclosure under
> applicable law.
> > If the reader of this message is not the intended recipient, you are
> > hereby notified that any dissemination, distribution or copying of
> > this communication is strictly prohibited. If you have received this
> > message in error, please immediately notify the sender and delete
> > the
> mail. Thank you.
> >
>
>
> CONFIDENTIAL NOTE:
> The information contained in this email is intended only for the use
> of the individual or entity named above and may contain information
> that is privileged, confidential and exempt from disclosure under applicable 
> law.
> If the reader

RE: [External] Re: Urgent helep please! How to measure producer and consumer throughput for single partition?

[Sunil CHAUDHARI]

Hi Liam Clarke,
Sorry but this is bit unclear for me.
Can you please elaborate your answer? I am beginner to Kafka.
" Producers emit metrics via JMX ":
- How to enable this? I have kafka-Manager. Can I make use of 
kafka-manager? How?
“kafka.producer:type=producer-metrics,client-id=(.+),topic=(.+)record-send-rate”
 please help to explain this.

Regards,
Sunil.

-Original Message-
From: Liam Clarke 
Sent: Thursday, February 20, 2020 11:16 AM
To: users@kafka.apache.org
Subject: [External] Re: Urgent helep please! How to measure producer and 
consumer throughput for single partition?

This mail originated from an external party outside DBS - 
mailto:users-return-39554-sunilchaudhari=dbs@kafka.apache.org. Do not click 
on links or open attachments unless you recognize the sender and know the 
content is safe.

Hi Sunil,

Producers emit metrics via JMX that will help you, assuming that your producers 
are using a round robin partition assignment strategy, you could divide this 
metric by your number of partitions,

kafka.producer:type=producer-metrics,client-id=(.+),topic=(.+)record-send-rate

Kind regards,

Liam Clarke

On Thu, 20 Feb. 2020, 5:57 pm Sunil CHAUDHARI, 
<mailto:sunilchaudh...@dbs.com.invalid> wrote:

> Hi
> I was referring to the article by Mr. June Rao about partitions in
> kafka cluster.
> https://www.confluent.io/blog/how-choose-number-topics-partitions-kafk
> a-cluster/
>
> "A rough formula for picking the number of partitions is based on
> throughput. You measure the throughout that you can achieve on a
> single partition for production (call it p) and consumption (call it
> c). Let's say your target throughput is t. Then you need to have at
> least max(t/p, t/c) partitions."
>
> I have the data pipeline as below.
>
> Filebeat-->Kafka-->Logstash-->Elasticsearch
> There are many filebeat agents sending data to kafka. I want to
> understand , how can I measure the events per seconds getting written
> to Kafka? This will help me to know 'p'  in above formula.
> I can measure the consumer throughput by monitoring logsatsh pipelines
> on Kibana. So it will give me 'c' in above formula.
>
> I know target throughput in my cluster, that is 't'. 30k events/s.
>
> Please let me know if I am going wrong?
>
> Regards,
> Sunil.
> CONFIDENTIAL NOTE:
> The information contained in this email is intended only for the use
> of the individual or entity named above and may contain information
> that is privileged, confidential and exempt from disclosure under applicable 
> law.
> If the reader of this message is not the intended recipient, you are
> hereby notified that any dissemination, distribution or copying of
> this communication is strictly prohibited. If you have received this
> message in error, please immediately notify the sender and delete the mail. 
> Thank you.
>


CONFIDENTIAL NOTE:
The information contained in this email is intended only for the use of the 
individual or entity named above and may contain information that is 
privileged, confidential and exempt from disclosure under applicable law. If 
the reader of this message is not the intended recipient, you are hereby 
notified that any dissemination, distribution or copying of this communication 
is strictly prohibited. If you have received this message in error, please 
immediately notify the sender and delete the mail. Thank you.

Urgent helep please! How to measure producer and consumer throughput for single partition?

[Sunil CHAUDHARI]

Hi
I was referring to the article by Mr. June Rao about partitions in kafka 
cluster.  
https://www.confluent.io/blog/how-choose-number-topics-partitions-kafka-cluster/

"A rough formula for picking the number of partitions is based on throughput. 
You measure the throughout that you can achieve on a single partition for 
production (call it p) and consumption (call it c). Let's say your target 
throughput is t. Then you need to have at least max(t/p, t/c) partitions."

I have the data pipeline as below.

Filebeat-->Kafka-->Logstash-->Elasticsearch
There are many filebeat agents sending data to kafka. I want to understand , 
how can I measure the events per seconds getting written to Kafka? This will 
help me to know 'p'  in above formula.
I can measure the consumer throughput by monitoring logsatsh pipelines on 
Kibana. So it will give me 'c' in above formula.

I know target throughput in my cluster, that is 't'. 30k events/s.

Please let me know if I am going wrong?

Regards,
Sunil.
CONFIDENTIAL NOTE:
The information contained in this email is intended only for the use of the 
individual or entity named above and may contain information that is 
privileged, confidential and exempt from disclosure under applicable law. If 
the reader of this message is not the intended recipient, you are hereby 
notified that any dissemination, distribution or copying of this communication 
is strictly prohibited. If you have received this message in error, please 
immediately notify the sender and delete the mail. Thank you.

SSL from Kafka to Zookeeper

[Sunil CHAUDHARI]

Hi,
I have enabled SSL on zookeeper nodes using reference: 
https://zookeeper.apache.org/doc/r3.5.6/zookeeperAdmin.html#Quorum+TLS
After doing this I started getting logs in zookeeper logs.  10.xx.yy.ss  this 
is my kafka broker
Now  I want to enable SSL communication between Kafka and the zookeeper.
Please share the steps or  documentation.

ERROR in zookeeper nodes:
2020-02-06 18:06:46,863 [myid:0] - WARN  
[epollEventLoopGroup-4-7:NettyServerCnxn@540] - Closing connection to 
/10.xx.yy.ss:58656
java.io.IOException: ZK down
at 
org.apache.zookeeper.server.NettyServerCnxn.receiveMessage(NettyServerCnxn.java:474)
   at 
org.apache.zookeeper.server.NettyServerCnxn.processMessage(NettyServerCnxn.java:360)
at 
org.apache.zookeeper.server.NettyServerCnxnFactory$CnxnChannelHandler.channelRead(NettyServerCnxnFactory.java:184)
at 
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
at 
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
at 
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
at 
io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1434)
at 
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
at 
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
at 
io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:965)
at 
io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:808)
at 
io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:410)
at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:310)
at 
io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884)
at 
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Thread.java:834)
CONFIDENTIAL NOTE:
The information contained in this email is intended only for the use of the 
individual or entity named above and may contain information that is 
privileged, confidential and exempt from disclosure under applicable law. If 
the reader of this message is not the intended recipient, you are hereby 
notified that any dissemination, distribution or copying of this communication 
is strictly prohibited. If you have received this message in error, please 
immediately notify the sender and delete the mail. Thank you.