Trying to understand - timeindex and index

hi We use kafka_2.11-2.0.0. With our partitions i see following 4 files in each partition directory. I have done basic research and understand what each file is used for. 0 Jul 11 19:17 leader-epoch-checkpoint 0 Jul 11 19:17 .log 10485756 Jul 11 19:26 00

Kafka consumer lag does not work for just one topic.

kafka-consumer-groups.sh fails for just one consumer group - group-vendor-cust. It work for every other consumer group. Command errors out complaining about the timeout, but adding --timeout does not help either. I don't know whats wrong or how go about debug this. Any help or suggestion?? Thank

RE: Re: Using ACLs without Kerberos

Yes, as per my understanding for now the only way to secure ZK is SASL. https://issues.apache.org/jira/browse/ZOOKEEPER-2125 once released ZK could also be secured using SSL. Also remember not use any OS user, anyone user on n\w who can connect to ZK host:port will be able to modify the ACLs.

Kafka consumer questoin.

Hi As I understand the producer in Kafka connect to broker list first to fetch the metadata. Procedure uses that data to directly connect to leader for partition that it's trying to publish to. >From my understanding of Kafka protocol and following other threads, the >consumer so will be doing

RE: Re: Re: ZK and Kafka failover testing

Just to add, I see below behavior repeat with even command line console producer and consumer that come with Kafka. Thanks, Shri __ Shrikant Patel | 817.367.4302 Enterprise Architecture Team PDX-NHIN -Original Message- From: Shrikant

RE: Re: Re: ZK and Kafka failover testing

t; loss >> > scenario that was a result of this[1]. Maybe this is an attempt at >> leaning >> > towards availability over consistency. Personally I think that >> > brokers should stop accepting requests when it disconnects from zookeeper. >> > >> > [1] The

RE: Re: ZK and Kafka failover testing

...@confluent.io (650)924-2670 */ On Tue, Apr 18, 2017 at 4:10 PM, Shrikant Patel wrote: > Hi All, > > I am seeing strange behavior between ZK and Kafka. We have 5 node in > ZK and Kafka cluster each. Kafka version - 2.11-0.10.1.1 > > The min.insync.replicas is 3, replication.factor i

ZK and Kafka failover testing

Hi All, I am seeing strange behavior between ZK and Kafka. We have 5 node in ZK and Kafka cluster each. Kafka version - 2.11-0.10.1.1 The min.insync.replicas is 3, replication.factor is 5 for all topics, unclean.leader.election.enable is false. We have 15 partitions for each topic. The step we

Kafka ACL \ SASL issue.

Hi All, We using SASL for Authentication between Kafka and ZK. Followed - https://www.confluent.io/blog/apache-kafka-security-authorization-authentication-encryption/ We have 3 Kafka node, on each node, we have principal="kafka/server_no.xxx@xxx.com. So On first node in kafka_server_jaas.c

Help with SASL configuration for Zookeeper on the Microsoft AD.

Hi Has anyone experience with securing Kafka to Zookeeper configuration and setting up SASL on Microsoft AD account. We create keytab and principal for Kafka and ZK using https://www.confluent.io/blog/apache-kafka-security-authorization-authentication-encryption/ We see these principal in our

RE: Clarification on min.insync.replicas​

exists no topic override for that configuration for that config. -Todd On Mon, Mar 6, 2017 at 4:38 PM, Shrikant Patel wrote: > Hi All, > > Need details about min.insync.replicas​ in the server.properties. > > I thought once I add this to server.properties, all subsequent topic

Clarification on min.insync.replicas​

Hi All, Need details about min.insync.replicas​ in the server.properties. I thought once I add this to server.properties, all subsequent topic create should have this as default value. C:\JAVA_INSTALLATION\kafka\kafka_2.11-0.10.1.1>bin\windows\kafka-topics.bat --zookeeper localhost:2181/chroot

RE: Kafka partition no migrating to another broker.

(see http://kafka.apache.org/documentation/#replication ) Some extra auto balancing features are available in Confluent Enterprise platform (e.g see http://docs.confluent.io/3.2.0/cp-docker-images/docs/tutorials/automatic-data-balancing.html ) Kind regards, Stevo Slavic. On Thu, Mar 2, 2017 at 10:20 PM, Shri

Kafka partition no migrating to another broker.

I have 5 broker kafka cluster. Replication factor = 3, Number of partition = 12, Min Insync repica (ISR) = 3 First output is when all server are up and running. Second output is when I bring down server id = 4. Another server from ISR take server 4's place as leader of partition, so that's good

Kafka Lag monitor + SASL

Our ZK server is secured using SASL. We have looked at Linkedin Burrow\ KafkaOffsetMonitor, it seems that they does not support connecting to ZK using SASL. Does any know of any other Kafka consumer lag \ offset monitoring tool that support SASL?? Thanks, Shri This e-mail and its contents (to

SASL for ZK\Kafka

QuorumPeerMain.java:111) at org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:78) Thanks, Shri __ Shrikant Patel | PDX-NHIN Enterprise Architecture Team Asserting the Role of Pharmacy in Healthcare www.pdxinc.com<http://www.pdxinc.

RE: Kafka ACL's with SSL Protocol is not working

You need to execute kafka-acls.sh with --consumer to enable consumption from kafka. _ Shrikant Patel | 817.367.4302 Enterprise Architecture Team PDX-NHIN -Original Message- From: Raghu B [mailto:raghu98...@gmail.com] Sent: Wednesday

Restrict who can change ACLs

operations: Create from hosts: * Am I missing anything??? Thanks in advance, Shri __ Shrikant Patel | PDX-NHIN Enterprise Architecture Team Asserting the Role of Pharmacy in Healthcare www.pdxinc.com<http://www.pdxinc.com/>

SSL Kafka 0.9.0.1

Shri __ Shrikant Patel | PDX-NHIN Enterprise Architecture Team Asserting the Role of Pharmacy in Healthcare www.pdxinc.com<http://www.pdxinc.com/> main 817.246.6760 | ext 4302 101 Jim Wright Freeway South, Suite 200, Fort Worth, Texas 76108-2202<http://maps.google.

RE: SSL - kafka producer cannot publish to topic

parameter pointing to just ssl configuration. It does not pick it up from producer.properties. -Original Message- From: Shrikant Patel [mailto:spa...@pdxinc.com] Sent: Thursday, December 10, 2015 2:09 PM To: users@kafka.apache.org Subject: SSL - kafka producer cannot publish to topic I am

SSL - kafka producer cannot publish to topic

I am trying to configure ssl communication between broker and producer. I followed the instruction on the https://cwiki.apache.org/confluence/display/KAFKA/Deploying+SSL+for+Kafka to create the key and trust store. My broker comes up without issue, I can run this command - openssl s_client -de

RE: Kafka on AWS - issue publishing from remote producer

ugins-not-working-on-windows/25253/3?u=joe_lawson> and here: https://discuss.elastic.co/t/logstash-1-5-3-is-not-able-to-connect-to-kafka/27611/5?u=joe_lawson On Sep 26, 2015 1:38 AM, "Shrikant Patel" wrote: > I have Kafka 2.10-0.8.2.1 and zookeeper installed on the AWS EC2. The >

Kafka on AWS - issue publishing from remote producer

I have Kafka 2.10-0.8.2.1 and zookeeper installed on the AWS EC2. The instance is working fine. [kafka@ip-xx-xx-xx-xx bin]$ ./kafka-topics.sh --topic topic1 --zookeeper localhost:2181 --describe Topic:topic1PartitionCount:1ReplicationFactor:1 Configs: Topic: topic1 Part

RE: 0.9.0 release

I think the answer is yes, I was confused because on Jira it say 0.9.0. __ Shrikant Patel | 817.246.6760 | ext. 4302 Enterprise Architecture Team PDX-NHIN-Rx.com -Original Message- From: Shrikant Patel [mailto:spa...@pdxinc.com] Sent

RE: 0.9.0 release

ypted. __ Shrikant Patel | 817.246.6760 | ext. 4302 Enterprise Architecture Team PDX-NHIN-Rx.com -Original Message- From: Gwen Shapira [mailto:g...@confluent.io] Sent: Monday, August 03, 2015 3:23 PM To: users@kafka.apache.org Subject: Re: 0.9.0 release According t

0.9.0 release

https://cwiki.apache.org/confluence/display/KAFKA/Future+release+plan Any idea when 0.9.0 will be released? Thanks, Shri This message and its contents (to include attachments) are the property of National Health Systems, Inc. and may contain confidential and pr

2 kafka cluster sharing same ZK Ensemble.

Can 2 separate kafka cluster share same ZK Ensemble?? If yes, how does ZK deal with that 2 clusters having brokers with same id. Thanks, Shri This message and its contents (to include attachments) are the property of National Health Systems, Inc. and may contain

Kafka deployment across DC

When I sent the email last time the message formatting was messed. Hopefully this does not have happen this time. We have very unique problem. We have an application deployed on WebLogic cluster that is spread across 2 datacenter (active-active) DC1 and DC2 (different LAN but same WAN). This pr

Kafka deployment across DC.

Wehave very unique problem.  Wehave a application deployed on weblogic cluster that is spread across 2 datacenter(active-active) DC1 and DC2 (different LAN but same WAN). This producer app generatesdifferent user events, which other apps (consumer apps) are interested in.Right now we use JMS a