Kafka Authentication + Authorization
Dear Users, Few questions regarding authentication 1- in case of implementation of SASL/SCRAM without SSL with below param listeners=SASL_PLAINTEXT://apache-kafka.abc.com:9090 security.inter.broker.protocol=SASL_PLAINTEXT sasl.mechanism.inter.broker.protocol=SCRAM-SHA-256 sasl.enabled.mechanisms=SCRAM-SHA-256 2- zookeeper is still not secured 3- New broker gets added without any jaas file in existing broker config. Am i missing something? New borker should fail with failed Authentication message ? 4- client is working properly .Client does not work unless client property is passed Regards, Rohit Garg
Re: Not able to connect to bootstrap server when one broker down
Please check your ISR using describe command and see if there is leader available when one of the broker is down . Thanks and Regards, Rohit On Tue, Aug 25, 2020, 20:20 Prateek Rajput wrote: > Hi, please if anyone can help, will be a huge favor. > > *Regards,* > *Prateek Rajput* > > > On Tue, Aug 25, 2020 at 12:06 AM Prateek Rajput < > prateek.raj...@flipkart.com> > wrote: > > > Hi everyone, > > I am new to Kafka, and recently started working on kafka in my company. > We > > recently migrated our client and cluster from the *0.10.x* version to > > *2.3.0*. I am facing this issue quite often. > > I have provided all brokers in *bootstrap.servers* config to instantiate > > the producer client but while using this client for batch publishing, > > sometimes some of my mappers get stuck. > > I debugged and found that one broker was down (for some maintenance > > activity). Now it was getting stuck because the mapper's client was > trying > > to connect to that node only for the very first time. And it was failing > > with NoRouteToHost Exception. > > I have read that the very first time the client will select a random > > broker and will try to connect with that broker to get the meta-data of > the > > whole cluster. Is there any way so that on such exceptions it can switch > to > > another node dynamically and should not try to connect to the same box > > again and again. > > > > *Regards,* > > *Prateek Rajput* > > > > -- > > > > > *-* > > > *This email and any files transmitted with it are confidential and > intended solely for the use of the individual or entity to whom they are > addressed. If you have received this email in error, please notify the > system manager. This message contains confidential information and is > intended only for the individual named. If you are not the named > addressee, > you should not disseminate, distribute or copy this email. Please notify > the sender immediately by email if you have received this email by mistake > and delete this email from your system. If you are not the intended > recipient, you are notified that disclosing, copying, distributing or > taking any action in reliance on the contents of this information is > strictly prohibited.* > > > > *Any views or opinions presented in this > email are solely those of the author and do not necessarily represent > those > of the organization. Any information on shares, debentures or similar > instruments, recommended product pricing, valuations and the like are for > information purposes only. It is not meant to be an instruction or > recommendation, as the case may be, to buy or to sell securities, > products, > services nor an offer to buy or sell securities, products or services > unless specifically stated to be so on behalf of the Flipkart group. > Employees of the Flipkart group of companies are expressly required not to > make defamatory statements and not to infringe or authorise any > infringement of copyright or any other legal right by email > communications. > Any such communication is contrary to organizational policy and outside > the > scope of the employment of the individual concerned. The organization will > not accept any liability in respect of such communication, and the > employee > responsible will be personally liable for any damages or other liability > arising.* > > > > *Our organization accepts no liability for the > content of this email, or for the consequences of any actions taken on the > basis of the information *provided,* unless that information is > subsequently confirmed in writing. If you are not the intended recipient, > you are notified that disclosing, copying, distributing or taking any > action in reliance on the contents of this information is strictly > prohibited.* > > > > _-_ > >
Re: Leader count is skewed on kafka brokers
Hello, If it is getting skewed means brokers are restarting . You can prepare manual assignment once and rerun same assignment again for preferred election. On Fri, Aug 21, 2020, 19:57 aishwary thakur wrote: > Hi, > I have deployed Kafka on K8S cluster over 3 nodes. > Leaders are not getting equally distributed over the brokers. > I have tried kafka-preferred-replica-election.sh script also. > Sometime it works and sometime it doesn't. > Even when it works, over time it again gets skewed. > Any help would be appreciated. > > Please ignore if this is not the right group and help me to reach out the > right group. >
Re: kafka per topic metrics
You can try using kafka manager and check it will fullfill most of requirement of yours. Thanks and Regards, Rohit On Wed, Jul 15, 2020, 10:33 Pushkar Deole wrote: > Hi All, > > Any inputs as to how the kafka consumer and producer metrics can be hooked > up to a monitoring system such as prometheus ? > > On Tue, Jul 14, 2020 at 4:22 PM Pushkar Deole > wrote: > > > i did find these metrics from confluent docs: however how can i get this > > metric ? is it available at kafka broker? > > > > Per-Topic Metrics > > > > MBean: > > > kafka.producer:type=producer-topic-metrics,client-id=([-.w]+),topic=([-.w]+) > > record-send-rateThe average number of records sent per second for a > topic. > > > > On Tue, Jul 14, 2020 at 3:27 PM Pushkar Deole > > wrote: > > > >> Hi All, > >> > >> Need some help on kafka metrics, i am interested in certain metrics e.g. > >> i need to know the number of records published on a particular topic and > >> number of records consumed from that topic by a specific consumer > group, i > >> would need a total of these 2 and also average per second for them. > >> > >> Are those metrics available on kafka brokers and if yes then which are > >> those metrics that would give me above counts? > >> > > >
Re: kafka-console-consumer.sh: Port already in use Exception after enable JMX
There are two ways to solve this issue, Exit and relogin again , Or again after broker is started , unset jmx port and you are good to go . Thanks and Regards, Rohit Garg On Mon, May 11, 2020, 10:10 wrote: > Please share the consumer.sh . > Are you using Apache kafka and what version ? > > From: "wangl...@geekplus.com.cn" > Date: Sunday, May 10, 2020 at 9:38 PM > To: users > Cc: "Agrawal, Manoj (Cognizant)" > Subject: Re: Re: kafka-console-consumer.sh: Port already in use Exception > after enable JMX > > [External] > Hi Manoj， > > I have tried, not work. > > When i changed to 9998, the consumer.sh tells me Port already in use: 9998 > > Thanks, > Lei > > > > wangl...@geekplus.com.cn<mailto:wangl...@geekplus.com.cn> > > > From: manoj.agraw...@cognizant.com<mailto:manoj.agraw...@cognizant.com> > Date: 2020-05-11 12:04 > To: users@kafka.apache.org<mailto:users@kafka.apache.org> > Subject: Re: kafka-console-consumer.sh: Port already in use Exception > after enable JMX > You can change jmx-port to any available port - 9992 > > On 5/10/20, 7:49 PM, "wangl...@geekplus.com.cn" > wrote: > > [External] > > > Add JMX_PORT=9988 to kafka-run-class.sh to enable JMX > > After execute bin/kafka-console-consumer.sh there‘s exception： > > Error: Exception thrown by the agent : > java.rmi.server.ExportException: Port already in use: 9988; nested > exception is: > java.net.BindException: Address already in use (Bind failed) > > > > > wangl...@geekplus.com.cn > > > > This e-mail and any files transmitted with it are for the sole use of the > intended recipient(s) and may contain confidential and privileged > information. If you are not the intended recipient(s), please reply to the > sender and destroy all copies of the original message. Any unauthorized > review, use, disclosure, dissemination, forwarding, printing or copying of > this email, and/or any action taken in reliance on the contents of this > e-mail is strictly prohibited and may be unlawful. Where permitted by > applicable law, this e-mail and other e-mail communications sent to and > from Cognizant e-mail addresses may be monitored. > This e-mail and any files transmitted with it are for the sole use of the > intended recipient(s) and may contain confidential and privileged > information. If you are not the intended recipient(s), please reply to the > sender and destroy all copies of the original message. Any unauthorized > review, use, disclosure, dissemination, forwarding, printing or copying of > this email, and/or any action taken in reliance on the contents of this > e-mail is strictly prohibited and may be unlawful. Where permitted by > applicable law, this e-mail and other e-mail communications sent to and > from Cognizant e-mail addresses may be monitored. > This e-mail and any files transmitted with it are for the sole use of the > intended recipient(s) and may contain confidential and privileged > information. If you are not the intended recipient(s), please reply to the > sender and destroy all copies of the original message. Any unauthorized > review, use, disclosure, dissemination, forwarding, printing or copying of > this email, and/or any action taken in reliance on the contents of this > e-mail is strictly prohibited and may be unlawful. Where permitted by > applicable law, this e-mail and other e-mail communications sent to and > from Cognizant e-mail addresses may be monitored. This e-mail and any files > transmitted with it are for the sole use of the intended recipient(s) and > may contain confidential and privileged information. If you are not the > intended recipient(s), please reply to the sender and destroy all copies of > the original message. Any unauthorized review, use, disclosure, > dissemination, forwarding, printing or copying of this email, and/or any > action taken in reliance on the contents of this e-mail is strictly > prohibited and may be unlawful. Where permitted by applicable law, this > e-mail and other e-mail communications sent to and from Cognizant e-mail > addresses may be monitored. >
Caused by: java.lang.IllegalArgumentException: Out of order offsets found in List
Hello Team , Recently i am facing this issue . kafka.common.KafkaException: Error processing data for partition event-35 offset 22428196775 at kafka.server.AbstractFetcherThread.$anonfun$processFetchRequest$7(AbstractFetcherThread.scala:205) at scala.Option.foreach(Option.scala:257) at kafka.server.AbstractFetcherThread.$anonfun$processFetchRequest$6(AbstractFetcherThread.scala:172) at kafka.server.AbstractFetcherThread.$anonfun$processFetchRequest$6$adapted(AbstractFetcherThread.scala:169) at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:59) at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:52) at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48) at kafka.server.AbstractFetcherThread.$anonfun$processFetchRequest$5(AbstractFetcherThread.scala:169) at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:12) at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:217) at kafka.server.AbstractFetcherThread.processFetchRequest(AbstractFetcherThread.scala:169) at kafka.server.AbstractFetcherThread.doWork(AbstractFetcherThread.scala:113) at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:64) Caused by: java.lang.IllegalArgumentException: Out of order offsets found in List(22428196775, 22428196776, 22428196777, 22428196778, 22428196779, 22428196780, 22428 196781, 22428196782, 22428196783, 22428196784, 22428196785, 22428196786, 22428196787, 22428196788,22428197974) at kafka.log.Log.$anonfun$append$2(Log.scala:683) at kafka.log.Log.maybeHandleIOException(Log.scala:1669) at kafka.log.Log.append(Log.scala:624) at kafka.log.Log.appendAsFollower(Log.scala:607) at kafka.server.ReplicaFetcherThread.processPartitionData(ReplicaFetcherThread.scala:102) at kafka.server.ReplicaFetcherThread.processPartitionData(ReplicaFetcherThread.scala:41) at kafka.server.AbstractFetcherThread.$anonfun$processFetchRequest$7(AbstractFetcherThread.scala:184) ... 12 more [2019-02-12 14:46:29,214] INFO [ReplicaFetcher replicaId=33, leaderId=114, fetcherId=5] Stopped (kafka.server.ReplicaFetcherThread) Thanks and Regards, Rohit Garg
