Re: [ANNOUNCE] New committer: Boyang Chen

Congratulations, Boyang -Harsha On Mon, Jun 22, 2020 at 4:32 PM AJ Chen wrote: > Congrats, Boyang! > > -aj > > > > > On Mon, Jun 22, 2020 at 4:26 PM Guozhang Wang wrote: > > > The PMC for Apache Kafka has invited Boyang Chen as a committer and we > are

Re: TLS Communication in With Zookeeper Cluster

Here is the guide https://cwiki.apache.org/confluence/display/ZOOKEEPER/ZooKeeper+SSL+User+Guide you need zookeeper 3.5 or higher for TLS. On Mon, Jul 29, 2019, at 1:21 AM, Nayak, Soumya R. wrote: > Hi Team, > > Is there any way mutual TLS communication set up can be done with > zookeeper. If

Re: [ANNOUNCE] Apache Kafka 2.2.1

Thanks Vahid. -Harsha On Mon, Jun 3, 2019, at 9:21 AM, Jonathan Santilli wrote: > That's fantastic! thanks a lot Vahid for managing the release. > > -- > Jonathan > > > > > On Mon, Jun 3, 2019 at 5:18 PM Mickael Maison > wrote: > > > Thank you Vahid

Re: [VOTE] 2.2.1 RC1

+1 (binding) 1. Ran unit tests 2. System tests 3. 3 node cluster with few manual tests. Thanks, Harsha On Wed, May 22, 2019, at 8:09 PM, Vahid Hashemian wrote: > Bumping this thread to get some more votes, especially from committers, so > we can hopefully make a decision on this RC by t

Re: Question about Kafka TLS

a certificate to enable TLS. JKS stores are for doing it manually. You can check out https://github.com/spiffe/java-spiffe which talks spiffee agent to get a certificate and pass it to Kafka's SSL context. Thanks, Harsha On Thu, May 16, 2019, at 3:57 PM, Zhou, Thomas wrote: > Hi, > &g

Question related to Kafka Connect

. Thanks, Harsha Mobile: +919440849355

Re: [ANNOUNCE] New Kafka PMC member: Sriharsh Chintalapan

Thanks, Everyone. -Harsha On Fri, Apr 19, 2019, at 2:39 AM, Satish Duggana wrote: > Congrats Harsha! > > On Fri, Apr 19, 2019 at 2:58 PM Mickael Maison > wrote: > > > Congratulations Harsha! > > > > > > On Fri, Apr 19, 2019 at 5:49 AM Manikumar

Re: plaintext connection attempts to SSL secured broker

Hi, Yes, this needs to be handled more elegantly. Can you please file a JIRA here https://issues.apache.org/jira/projects/KAFKA/issues Thanks, Harsha On Mon, Apr 1, 2019, at 1:52 AM, jorg.heym...@gmail.com wrote: > Hi, > > We have our brokers secured with these standard p

Re: Broker continuously expand and shrinks to itself

option to try is to lower the socket timeout to be lower and increase the lag.time.max.ms . Thanks, Harsha On Jan 27, 2019, 8:48 AM -0800, Ashish Karalkar , wrote: > Hi Harsha, > Thanks for the reply. > Issue is resolved as of now and the root cause was a runaway application > s

Re: Kafka consumer configuration to minimize rebalance time

-Harsha On Jan 24, 2019, 9:51 AM -0800, Marcos Juarez , wrote: > One of our internal customers is working on a service that spans around 120 > kubernetes pods. Due to design constraints, every one of these pods has a > single kafka consumer, and they're all using the same consumer group id

Re: Broker continuously expand and shrinks to itself

Hi Ashish,            Whats your replica.lag.time.max.ms set to and do you see any network issues between brokers. -Harsha On Jan 22, 2019, 10:09 PM -0800, Ashish Karalkar , wrote: > Hi All, > We just upgraded from 0.10.x to 1.1 and enabled rack awareness on an existing > clusters

Re: kafka.consumers Java Package

Chris,         You are upgrading from 0.10.2.2 to 2.0.0 . There will be quite few changes and it looks like you might be using classes other than KafkaConsumer which are not public API? What classes specifically are not available. -Harsha On Nov 9, 2018, 7:47 AM -0800, Chris Barlock , wrote

Re: [VOTE] 2.0.1 RC0

+1. Ran a 3 node cluster with few simple tests. Thanks, Harsha On Nov 1, 2018, 9:50 AM -0700, Eno Thereska , wrote: > Anything else holding this up? > > Thanks > Eno > > On Thu, Nov 1, 2018 at 10:27 AM Jakub Scholz wrote: > > > +1 (non-binding) ... I used the st

Re: [DISCUSS] KIP-369 Alternative Partitioner to Support "Always Round-Robin" Selection

. Use your custom partitioner to be configured in your producer clients. Thanks, Harsha On Thu, Aug 30, 2018, at 1:45 AM, M. Manna wrote: > Hello, > > I opened a very simple KIP and there exists a JIRA for it. > > I would be grateful if any comments are available for action. > > Regards,

Re: Performance Impact with Apache Kafka Security

the impact in my case? Thank you, Harsha Sent from Outlook<http://aka.ms/weboutlook> From: Eric Azama Sent: Friday, August 24, 2018 2:04 PM To: users@kafka.apache.org Cc: ka...@harsha.io Subject: Re: Performance Impact with Apache Kafka Security I saw a similar

Re: Performance Impact with Apache Kafka Security

Hi Harsha, Given below are all the details. We are using Kafka On CDH. Do you have any suggestion based on the below statistics. CDK - 2.2.0 - 0.10.2.0+kafka2.2.0+110 Apache Kafka - 0.10.2. java version "1.8.0_151" Tried Using Java 9 with not much difference. We need to make a sm

Re: Performance Impact with Apache Kafka Security

Hi, Which Kafka version and Java version are you using? Did you try this with Java 9 which has 2.5x perf improvements over Java 8 for SSL? Can you try using a slightly weaker cipher suite to improve the performance? -Harsha On Wed, Aug 22, 2018, at 1:11 PM, Sri Harsha Chavali wrote: >

Performance Impact with Apache Kafka Security

ms 95th, 2914 ms 99th, 3211 ms 99.9th Thank you, Harsha

Re: Restrict access on kafka with multiple listener

read/writes on the secure topics and it will reject any request on PLAINTEXT port for these topics as AuthorizationException and rest of the topics you can continue access through both the ports. -Harsha On Tue, Jul 17, 2018, at 5:09 PM, Matt L wrote: > Hi, > > I have an existing Kafk

Re: [VOTE] 1.1.1 RC3

+1. * Ran unit tests * Installed in a cluster and ran simple tests Thanks, Harsha On Mon, Jul 9th, 2018 at 6:38 AM, Ted Yu wrote: > > > > +1 > > Ran test suite. > > Checked signatures. > > > > On Sun, Jul 8, 2018 at 3:36 PM Dong Lin < lindon..

Re: [VOTE] 2.0.0 RC1

+1.  1) Ran unit tests  2) 3 node cluster , tested basic operations.  Thanks, Harsha On Mon, Jul 2nd, 2018 at 11:13 AM, "Vahid S Hashemian" wrote: > > > > +1 (non-binding) > > Built from source and ran quickstart successfully on Ubuntu (with Java 8). >

Re: [VOTE] 2.0.0 RC1

+1 . * Ran unit tests * Verified signatures * Ran 3 node cluster with basic operations Thanks, Harsha On Mon, Jul 2nd, 2018 at 11:13 AM, "Vahid S Hashemian" wrote: > > > > +1 (non-binding) > > Built from source and ran quickstart successfully on Ubuntu

Re: [kafka-clients] [VOTE] 1.0.2 RC1

+1.      1) Ran unit tests 2) 3 node cluster , tested basic operations. Thanks, Harsha On Mon, Jul 2nd, 2018 at 11:57 AM, Jun Rao wrote: > > > > Hi, Matthias, > > Thanks for the running the release. Verified quickstart on scala 2.12 > binary. +1 > > Jun > &

Re: [VOTE] 0.11.0.3 RC0

+1 * Ran Unit tests * 3 node cluster . Ran simple tests. Thanks, Harsha On Sat, Jun 23rd, 2018 at 9:7 AM, Ted Yu wrote: > > > > +1 > > Checked signatures. > > Ran unit test suite. > > On Fri, Jun 22, 2018 at 4:56 PM, Vahid S Hashemian < > vahidhas

Re: [VOTE] 1.1.1 RC1

+1 .   1. Ran unit tests.   2. Ran few tests on 3-node cluster Thanks, Harsha On Fri, Jun 22nd, 2018 at 2:41 PM Jakob Homan wrote: > > > > +1 (binding) > > * verified sigs, NOTICE, LICENSE > * ran unit tests > * spot checked headers > > -Jakob > > &

Re: Recording - Storm & Kafka Meetup on April 20th 2017

Hi Aditya, Thanks for your interest. We entatively planning one in June 1st week. If you haven't already please register here https://www.meetup.com/Apache-Storm-Apache-Kafka/ . I'll keep the Storm lists updated once we finalize the date & location. Thanks, Harsha On Mon, Ap

Re: Securing Multi-Node single broker kafka instance

factor and see if you can produce & consume messages Before stepping into security make sure your non-secure Kafka cluster works ok. Once you’ve a stable & working cluster follow instructions in the doc to enable SSL. -Harsha On Mar 1, 2017, 1:08 PM -0800, IT Consultant <0binarybudd..

Re: Securing Multi-Node single broker kafka instance

For inter broker communication over SSL all you need is to add security.inter.broker.protocol to SSL. "How do i make zookeeper talk to each other and brokers?" Not sure I understand the question. You need to make sure zookeeper hosts and port are reachable from your broker nodes. -Hars

Re: Kafka SASL and custom LoginModule and Authorizer

-48+Delegation+token+support+for+Kafka . Thanks, Harsha On Sat, Feb 25, 2017 at 6:48 PM Christian <engr...@gmail.com> wrote: > We have implemented our own LoginModule and Authorizer. The LoginModule > does an authentication on the client side, obtains a token and passes that > to

Re: [ANNOUNCE] New committer: Jiangjie (Becket) Qin

Congrats Becket! -Harsha On Mon, Oct 31, 2016 at 2:13 PM Rajini Sivaram <rajinisiva...@googlemail.com> wrote: > Congratulations, Becket! > > On Mon, Oct 31, 2016 at 8:38 PM, Matthias J. Sax <matth...@confluent.io> > wrote: > > > -BEGIN PGP SIG

Re: [VOTE] Add REST Server to Apache Kafka

This Vote now closed. There is the majority who are not in favor of including the REST server. Thanks everyone for participating. Vote tally: +1 Votes: Harsha Chintalapani Parth Brahbhatt Ken Jackson Suresh Srinivas Jungtaek Lim Ali Akthar Haohui Mai Shekhar

Re: [VOTE] Add REST Server to Apache Kafka

Jeff, Thanks for participating. We already have the discussion thread going on. Please add your thoughts there . I'll keep this as interest check vote thread. Thanks, Harsha On Tue, Oct 25, 2016 at 3:12 PM Stevo Slavić <ssla...@gmail.com> wrote: > -1 > > I fully agree wi

[VOTE] Add REST Server to Apache Kafka

/%3ccamvt_aymqeudm39znsxgktpdde46sowmqhsxop-+jmbcuv7...@mail.gmail.com%3E This is a VOTE thread to check interest in the community for adding REST Server implementation in Apache Kafka. Thanks, Harsha

Re: SSL Kafka 0.9.0.1

Shri, SSL in 0.9.0.1 is not beta and can be used in production. If you want to put authorizer on top of SSL to enable ACLs for clients and topics that's possible too. Thanks, Harsha On Mon, Oct 3, 2016 at 8:30 AM Shrikant Patel <spa...@pdxinc.com> wrote: > We are are 0.9.0.1

Re: a broker is already registered on path /brokers/ids/1

how many brokers you've in this cluster. Do you try using a stable zookeeper release like 3.4.8? -Harhsa On Mon, Aug 29, 2016 at 5:21 AM Nomar Morado wrote: > we are using kafka 0.9.0.1 and zk 3.5.0-alpha > > On Mon, Aug 29, 2016 at 8:12 AM, Nomar Morado

Re: [kafka-clients] [VOTE] 0.10.0.1 RC2

+1 (binding) 1. Ran 3 node cluser 2. Ran few tests in creating, producing , consuming from secure & non-secure clients. Thanks, Harsha On Fri, Aug 5, 2016 at 8:50 PM Manikumar Reddy <manikumar.re...@gmail.com> wrote: > +1 (non-binding). > verified quick start and artifacts. &

Re: [kafka-clients] [VOTE] 0.10.0.1 RC0

Thanks Ismael. On Sat, Jul 30, 2016 at 7:43 PM Ismael Juma wrote: > Hi Dana, > > Thanks for testing releases so promptly. Very much appreciated! > > It's funny, Ewen had suggested something similar with regards to the > release notes a couple of days ago. We now have a Python

Re: [VOTE] 0.10.0.1 RC0

Hi Ismael, I would like to this JIRA included in the minor release https://issues.apache.org/jira/browse/KAFKA-3950. Thanks, Harsha On Fri, Jul 29, 2016 at 7:46 AM Ismael Juma <ism...@juma.me.uk> wrote: > Hello Kafka users, developers and client-developers, > > Th

Re: TLS based ACL: Does Kafka support multiple CA Certs on broker

Did you make sure both those CA's are imported into Broker's truststore? -Harsha On Fri, Jul 15, 2016 at 5:12 PM Raghavan, Gopal <gopal.ragha...@here.com> wrote: > Hi, > > Can Kakfa support multiple CA certs on broker. > If yes, can you please point me to an example. &

Re: Deleting a topic on the 0.8x brokers

One way to delete is to delete the topic partition directories from disks and delete /broker/topics. If you just shutdown those brokers controller might try to replicate the topic onto brokers and since you don't have any leaders you might replica fetcher errors in the logs. Thanks, Harsha On Thu

Re: Error in znode creation after adding SASL digest on server and client

it expects all the nodes to have an ACL on it. This could be an issue with ZkClient tha we use or we need to navigate this case differently. Can you file a JIRA for this. Thanks, Harsha On Thu, Jul 7, 2016 at 10:48 PM Vipul Sharma <vipulsharma2...@gmail.com> wrote: > I am running

Re: Error in znode creation after adding SASL digest on server and client

it expects all the nodes to have an ACL on it. This could be an issue with ZkClient tha we use or we need to navigate this case differently. Can you file a JIRA for this. Thanks, Harsha On Fri, Jul 8, 2016 at 3:24 PM Harsha Ch <harsha...@gmail.com> wrote: > Hi, > So we speci

Re: ZkUtils creating parent path with different ACL

If i remember correctly it was because we wanted to allow non-secure client still get into child consumers node and create their zookeeper nodes to keep track of offset. If we add the acl at the parent path they won't be able to write to the child nodes. Thanks, Harsha On Fri, Jul 1, 2016, at 06

Re: SSL support for command line tools

Radu, Please follow the instructions here http://kafka.apache.org/documentation.html#security_ssl . At the end of the SSL section we've an example for produce and consumer command line tools to pass in ssl configs. Thanks, Harsha On Wed, Jun 22, 2016, at 07:40

Re: [DISCUSS] Java 8 as a minimum requirement

lot of features . If we can make this as part of 0.11 and cutting 0.10.1 features moving to 0.11 and giving rough timeline when that would be released would be ideal. Thanks, Harsha On Fri, Jun 17, 2016, at 11:13 AM, Ismael Juma wrote: > Hi Har

Re: [DISCUSS] Java 8 as a minimum requirement

va 7 support when the release is minor in general not a good idea to users. -Harsha "Also note that Kafka bug fixes go to 0.10.0.1, not 0.10.1 and > 0.10.0.x would still be available for users using Java 7." On Fri, Jun 17, 2016, at 12:18 AM, Ismael Juma wrote: > Hi Harsha, > >

Re: [DISCUSS] Java 8 as a minimum requirement

on the dates . Thanks, Harsha On Thu, Jun 16, 2016, at 03:08 PM, Ismael Juma wrote: > Hi Jan, > > That's interesting. Do you have some references you can share on this? It > would be good to know which Java 8 versions have been tested and whether > it > is something th

Re: Questions on Kafka Security

ported yet. Also dropping the encryption in SSL channel is not possible yet. Any reason for not use kerberos for this since we support non-encrypted channel for kerberos. Thanks, harsha On Wed, Jun 8, 2016, at 02:06 PM, Samir Shah wrote: > Hello, > > Few questions on Kafka Security. &

Re: Not able to monitor consumer group lag with new consumer and kerberos

Hi Pierre, Do you see any errors in the server.log when this command ran. Can you please open a thread here https://community.hortonworks.com/answers/index.html . Thanks, Harsha On Tue, Jun 7, 2016, at 09:22 AM, Pierre Labiausse wrote: > Hi, >

Re: Kafka Newbie question

Pradeep, How about https://kafka.apache.org/090/javadoc/org/apache/kafka/clients/consumer/KafkaConsumer.html#seekToBeginning%28org.apache.kafka.common.TopicPartition...%29 -Harsha On Sat, Apr 9, 2016, at 09:48 PM, Pradeep Bhattiprolu wrote: > Liquan , tha

Re: Kafka Error while doing SSL authentication on 0.9.0

/documentation.html#security_ssl -Harsha On Thu, Mar 10, 2016, at 12:41 AM, Ranjan Baral wrote: > i getting below warning while doing produce from client side which is > connecting to server side which contains SSL based authentication. > > *[2016-03-10 07:09:13,018] WARN The c

Re: Kafka 0.9 Spout

, Harsha On Tue, Mar 1, 2016, at 12:56 PM, Ankit Jain wrote: > Hi All, > > We would need to use the SSL feature of Kafka and that would require the > Kafka Spout upgrade from version 0.8.x to 0.9.x as the SSL is only > supported by new consumer API. > > Please share

Re: Kerberized Kafka setup issues

principal="kafka/ubuntu.oleg@oleg.com"; }; -Harsha On Tue, Feb 23, 2016, at 11:37 AM, Harsha wrote: > can you try adding "serviceName=zookeeper" to KafkaServer section like > KafkaServer { > com.sun.security.auth.module.Krb5LoginModule required >

Re: Kerberized Kafka setup issues

ssuming Zookeeper is started as ‘zookeeper’ > and Kafka as ‘kafka’ > > Oleg > > > On Feb 23, 2016, at 2:22 PM, Oleg Zhurakousky > > <ozhurakou...@hortonworks.com> wrote: > > > > Harsha > > > > Thanks for following up. Here

Re: Kerberized Kafka setup issues

Oleg, Can you post your jaas configs. Its important that serviceName must match the principal name with which zookeeper is running. Whats the principal name zookeeper service is running with. -Harsha On Tue, Feb 23, 2016, at 11:01 AM, Oleg Zhurakousky wrote: > Hey g

Re: Java Client connection errors with Kafka 0.9.0.0 when SSL is enabled

Did you try what Adam is suggesting in the earlier email. Also to quickly check you can try remove keystore and key.password configs from client side. -Harsha On Thu, Feb 18, 2016, at 02:49 PM, Srikrishna Alla wrote: > Hi, > > We are getting the below error when trying to use a Java new

Re: 答复: NoAuth for /controller

Kafka doesn't have security enabled for 0.8.2.2 so make sure zookeeper root that you're using doesn't have any acls set -Harsha On Sun, Feb 14, 2016, at 06:51 PM, 赵景波 wrote: > Can you help me? > > ___ > JingBo Zhao<h

Re: behavior of two live brokers with the same id?

You won't be able to start and register to brokers with same id in zookeeper. On Thu, Feb 4, 2016, at 06:26 AM, Carlo Cabanilla wrote: > Hello, > > How does a kafka cluster behave if there are two live brokers with the > same > broker id, particularly if that broker is a leader? Is it

Re: Consumer - Failed to find leader

/keytabs/kafka1.keytab" principal="kafka/kafka1.witzend@witzend.com"; }; and KafkaClient KafkaClient { com.sun.security.auth.module.Krb5LoginModule required useTicketCache=true serviceName="kafka"; }; On Wed, Dec 30, 2015, at 03:10 AM, prabhu v wrote: > Hi Harsha,

Re: Consumer - Failed to find leader

nnection to Zookeeper server without SASL authentication, if Zookeeper" did you try kinit with that keytab at the command line. -Harsha On Mon, Dec 28, 2015, at 04:07 AM, prabhu v wrote: > Thanks for the input Ismael. > > I will try and let you know. > > Also need your v

Re: The JIRA Awakens [KAFKA-1841]

kafka 0.9.0 + additional patches . We are making sure on our side not to miss any compatibility patches like these with 3rd party developers and have tests to ensure that. Thanks, Harsha On Wed, Dec 23, 2015, at 04:11 PM, Dana Powers wrote: > Hi all, > > I've been helping debug an is

Re: kafka 0.8.2 build problem

Can you try using the 0.8.2.2 source here https://www.apache.org/dyn/closer.cgi?path=/kafka/0.8.2.2/kafka-0.8.2.2-src.tgz https://kafka.apache.org/downloads.html On Sun, Nov 15, 2015, at 05:35 AM, jinxing wrote: > Hi all, I am newbie for kafka, and I try to build kafka 0.8.2 from source > code

Re: Kafka Broker process: Connection refused

Mark,       Can you try running the /usr/hdp/current/kafka- broker/bin/kafka start manually as user Kafka and also do you see jars under kafka-broker/libs/kafka_*.jar . Can you try asking the question here as well http://hortonworks.com/community/forums/forum/kafka/ Thanks, Harsha On Sun, Oct 4

EOL JDK 1.6 for Kafka

potential MITM. This api doesn't exist in Java 1.6. Are there any users still want 1.6 support or can we stop supporting 1.6 from next release on wards. Thanks, Harsha

Re: Failure in Leader Election on broker shutdown

Sandeep,         You need to have multiple replicas. Having single replica means you've one copy of the data and if that machine goes down there isn't another replica who can take over and be the leader for that partition.-Harsha  _ From: Sandeep Bishnoi

Re: Increasing replication factor of existing topics

Hi Navneet,           Any reason that you are looking to modify the zk nodes directly to increase the topic partition. If you are looking for an api to do this there is AdminUtils.addPartitions .  --  Harsha On April 7, 2015 at 6:45:40 AM, Navneet Gupta (Tech - BLR) (navneet.gu

Re: Per-topic retention.bytes uses kilobytes not bytes?

log size to 200 bytes. --  Harsha On April 2, 2015 at 2:29:45 PM, Willy Hoang (w...@knewton.com) wrote: Hello, I’ve been having trouble using the retention.bytes per-topic configuration (using Kafka version 0.8.2.1). I had the same issue that users described in these two threads where logs

Re: Kafka Consumer

instance within each subscribing consumer group. Consumer instances can be in separate processes or on separate machines.” More info on this page http://kafka.apache.org/documentation.html look for consumer group”.  --  Harsha On March 31, 2015 at 6:10:59 AM, James King (jakwebin

Re: lost messages -?

Victor,         Its under kaka.tools.DumpLogSegments you can use kafka-run-class to execute it. --  Harsha On March 26, 2015 at 5:29:32 AM, Victor L (vlyamt...@gmail.com) wrote: Where's this tool (DumpLogSegments) in Kafka distro? Is it Java class in kafka jar, or is it third party binary

Re: Check topic exists after deleting it.

node gets deleted as well. Also make sure you don’t have any producers or consumers are running while the topic deleting is going on. --  Harsha On March 23, 2015 at 1:29:50 AM, anthony musyoki (anthony.musy...@gmail.com) wrote: On deleting a topic via TopicCommand.deleteTopic() I get Topic

Re: Check topic exists after deleting it.

. --  Harsha On March 23, 2015 at 9:57:53 AM, Grant Henke (ghe...@cloudera.com) wrote: What happens when producers or consumers are running while the topic deleting is going on? On Mon, Mar 23, 2015 at 10:02 AM, Harsha ka...@harsha.io wrote: DeleteTopic makes a node in zookeeper to let

Re: Check topic exists after deleting it.

Just to be clear, one needs to stop producers and consumers that writing/reading from a topic “test” if they are trying to delete that specific topic “test”. Not all producers and clients. --  Harsha On March 23, 2015 at 10:13:47 AM, Harsha (harsh...@fastmail.fm) wrote: Currently we have

Re: KafkaSpout forceFromStart Issue

Hi Francois,            Looks like this belong storm mailing lists. Can you please send this question on storm mailing lists. Thanks, Harsha On March 23, 2015 at 11:17:47 AM, François Méthot (fmetho...@gmail.com) wrote: Hi, We have a storm topology that uses Kafka to read a topic with 6

Re: Kafka-Storm: troubleshooting low R/W throughput

Hi Emmanuel,        Can you post your kafka server.properties and in your producer are your distributing your messages into all kafka topic partitions. --  Harsha On March 20, 2015 at 12:33:02 PM, Emmanuel (ele...@msn.com) wrote: Kafka on test cluster:  2 Kafka nodes, 2GB, 2CPUs 3 Zookeeper

Re: kafka topic information

Yuheng,           kafka keeps cluster metadata in zookeeper along with topic metadata as well. You can use zookeeper-shell.sh or zkCli.sh to check zk nodes, /brokers/topics will give you the list of topics . --  Harsha On March 9, 2015 at 8:20:59 AM, Yuheng Du (yuheng.du.h...@gmail.com) wrote

Re: kafka topic information

machine failure and incase of 5 there should be at least 3 nodes to be up and running.   For more info on zookeeper you can look under here  http://zookeeper.apache.org/doc/r3.4.6/ http://zookeeper.apache.org/doc/r3.4.6/zookeeperAdmin.html --  Harsha On March 9, 2015 at 8:39:00 AM, Yuheng Du

Re: Problem deleting topics in 0.8.2?

stopping any of your consumers or producers run the delete topic command again. -Harsha On Wed, Mar 4, 2015, at 10:28 AM, Jeff Schroeder wrote: So I've got 3 kafka brokers that were started with delete.topic.enable set to true. When they start, I can see in the logs that the property

Re: 0.7 design doc?

These docs might help https://kafka.apache.org/08/design.html http://research.microsoft.com/en-us/um/people/srikanth/netdb11/netdb11papers/netdb11-final12.pdf -Harsha On Sun, Mar 1, 2015, at 09:42 PM, Philip O'Toole wrote: Thanks Guozhang -- no this isn't quite it. The doc I read before

Re: Delete Topic in 0.8.2

Hi Hema, Can you attach controller.log and state-change.log. Image is not showing up at least for me. Can you also give us details on how big the cluster is and topic's partitions and replication-factor and any steps on reproducing this. Thanks, Harsha On Sun, Mar 1, 2015, at 12:40 PM, Hema

Re: How replicas catch up the leader

you can increase num.replica.fetchers by default its 1 and also try increasing replica.fetch.max.bytes -Harsha On Fri, Feb 27, 2015, at 11:15 PM, tao xiao wrote: Hi team, I had a replica node that was shutdown improperly due to no disk space left. I managed to clean up the disk and restarted

Re: Batching with new Producer API

Akshat, Produce.batch_size is in bytes and if your messages avg size is 310 bytes and your current number of messages per batch is 46 you are getting close to the max batch size 16384. Did you try increasing the producer batch_size bytes? -Harsha On Thu, Feb 26, 2015

Re: Broker Server Crash with HW failure. Broker throwing java.lang.NumberFormatException and will not restart without removing all partitions

Hi Gene, Looks like you might be running into this https://issues.apache.org/jira/browse/KAFKA-1758 . -Harsha On Tue, Feb 24, 2015, at 07:17 AM, Gene Robichaux wrote: About a week ago one of our brokers crashed with a hardware failure. When the server restarted the Kafka broker

Re: Kakfa question about starting kafka with same broker id

on the path /brokers/ids/0. This probably indicates that you either have configured a brokerid that is already in use, or else you have shutdown this broker and restarted it faster than the zookeeper timeout so it appears to be re-registering. -Harsha On Wed, Feb 18, 2015, at 03:16 PM, Deepak Dhakal wrote

Re: MetadataRequest vs Zookeeper

Paul, There is ongoing work to move to Kafka API instead of making calls to zookeeper. Here is the JIRA https://issues.apache.org/jira/browse/STORM-650 . -Harsha On Fri, Feb 13, 2015, at 01:02 PM, Paul Mackles wrote: I noticed that the standard Kafka storm spout gets topic

[DISCUSS] KIP-12 - Kafka Sasl/Kerberos implementation

the work is in progress. I'll update the thread with a initial prototype patch. Thanks, Harsha

Fwd: [DISCUSS] KIP-12 - Kafka Sasl/Kerberos implementation

the work is in progress. I'll update the thread with a initial prototype patch. Thanks, Harsha

Re: [DISCUSS] KIP-12 - Kafka Sasl/Kerberos implementation

Thanks Joe. It will be part of KafkaServer and will run on its own thread. Since each kafka server will run with a keytab we should make sure they are all getting renewed. On Wed, Feb 11, 2015, at 10:00 AM, Joe Stein wrote: Thanks Harsha, looks good so far. How were you thinking of running

Re: Issue with topic deletion

Sumit, lets say you are deleting a older topic test1 do you have any consumers running simultaneously for the topic test1 while deletion of topic going on. -Harsha On Tue, Feb 3, 2015, at 06:17 PM, Joel Koshy wrote: Thanks for the logs - will take a look tomorrow unless

Re: Issue with topic deletion

you try stopping the consumer first and issue the topic delete. -Harsha On Tue, Feb 3, 2015, at 08:37 PM, Sumit Rangwala wrote: On Tue, Feb 3, 2015 at 6:48 PM, Harsha ka...@harsha.io wrote: Sumit, lets say you are deleting a older topic test1 do you have any consumers running

Re: Issue with topic deletion

-142388317 which looks to be getting deleted properly. Do you see same issues with the above topic i.e /admin/delete_topics/LAX1-GRIFFIN-r45-142388317 still exists. If you can post the logs from 23:00 onwards that will be helpful. -Harsha On Tue, Feb 3, 2015, at 09:19 PM, Harsha wrote: you

Re: create topic does not really executed successfully

. -Harsha On Mon, Feb 2, 2015, at 07:03 PM, Xinyi Su wrote: Hi, -bash-4.1$ bin/kafka-topics.sh --zookeeper zkhosst:2181 --create --topic zerg.hydra --partitions 3 --replication-factor 2 Created topic zerg.hydra. -bash-4.1$ ls -lrt /tmp/kafka-logs/zerg.hydra-2 total 0 -rw-r--r-- 1 users

Re: create topic does not really executed successfully

. -Harsha On Mon, Feb 2, 2015, at 07:03 PM, Xinyi Su wrote: Hi, -bash-4.1$ bin/kafka-topics.sh --zookeeper zkhosst:2181 --create --topic zerg.hydra --partitions 3 --replication-factor 2 Created topic zerg.hydra. -bash-4.1$ ls -lrt /tmp/kafka-logs/zerg.hydra-2 total 0 -rw-r--r-- 1 users

Re: unable to delete topic with 0.8.2 rc2

the topic will be recreated. -Harsha On Sun, Jan 25, 2015, at 11:26 PM, Jason Rosenberg wrote: cversion did change (incremented by 2) when I issue the delete command. From the logs on the conroller broker (also the leader for the topic), it looks like the delete proceeds, and then the topic gets

Re: unable to delete topic with 0.8.2 rc2

Jun, I made an attempt at fixing that issue as part of this JIRA https://issues.apache.org/jira/browse/KAFKA-1507 . As Jay pointed out there should be admin api if there is more info on this api I am interested in adding/fixing this issue. Thanks, Harsha On Mon, Jan 26, 2015, at 07

Re: Delete topic API in 0.8.2

Sumit, You can use AdminUtils.deleteTopic(zkClient, topicName) . This will initiate the deleteTopic process by zookeeper notification to KafkaController.deleteTopicManager. It deletes log files along with zookeeper topic path as Timothy mentioned. -Harsha On Fri, Jan

Re: kafka topic not exist after creation with api

Manu, Are you passing ZkStringSerializer to zkClient . ZkClient zkClient = new ZkClient(ZK_CONN_STRING,3, 3, new ZkStringSerializer()); AdminUtils.createTopic(zkClient, topic, 1, 1, props); -Harsha On Thu, Jan 22, 2015, at 09:44 PM, Manu Zhang wrote: Hi all , My application

Re: kafka brokers going down within 24 hrs

Tousif, I meant to say if kafka broker is going down often its better to analyze whats the root of cause of the crash. Using supervisord to monitor kafka broker is fine, sorry about the confusion. -Harsha On Fri, Jan 16, 2015, at 11:25 AM, Gwen Shapira wrote: Those errors

Re: kafka brokers going down within 24 hrs

to see why its going down for the first time. -Harsha On Wed, Jan 14, 2015, at 10:50 PM, Tousif wrote: Hello Chia-Chun Shih, There are multiple issues, First thing is i don't see out of memory error every time and OOM happens after supervisord keep retrying to start kafka. It goes down when

Re: kafka brokers going down within 24 hrs

Tousif, Do you see any other errors in server.log -Harsha On Wed, Jan 14, 2015, at 01:51 AM, Tousif wrote: Hello, I have configured kafka nodes to run via supervisord and see following exceptions and eventually brokers going out of memory. i have given enough memory and process 1

Re: Create topic, then fetch offset

It looks like you are not adding a ZkStringSerializer to your zkClient. ZkClient zkClient = new ZkClient(ZK_CONN_STRING); in Kafka TopicCommand uses new ZkClient(opts.options.valueOf(opts.zkConnectOpt), 3, 3, ZKStringSerializer) because of this although your topic is getting created and

  1   2   >