Hi Arunkumar, I'm glad you were able to fix the issue. Also glad that the article was helpful.
Regarding Kafka SSL configuration, I'm sending some links: - Kafka documentation: http://kafka.apache.org/documentation.html#security_ssl - Apache Kafka Security 101: https://www.confluent.io/blog/apache-kafka-security-authorization-authentication-encryption/ - Configuring Kafka Security: https://www.cloudera.com/documentation/kafka/latest/topics/kafka_security.html I hope they help you get started with SSL configuration. --Vahid From: Arunkumar <pm_arunku...@yahoo.com.INVALID> To: <users@kafka.apache.org> Date: 06/16/2017 03:47 PM Subject: Re: UNKNOWN_TOPIC_OR_PARTITION with SASL_PLAINTEXT ACL Hi Vahid I deleted the dataDir and dataLogDir and restarted zookeeper,brokers, producers and consumer. Now it works!!!! All the messages produced are consumed from the producer. Thanks for all the help. The link you shared helped a lot. I am planning to setup SASL_SSL, I appreciate you advice for the same. Thanks Arunkumar Pichaimuthu, PMP -------------------------------------------- On Fri, 6/16/17, Arunkumar <pm_arunku...@yahoo.com.INVALID> wrote: Subject: Re: UNKNOWN_TOPIC_OR_PARTITION with SASL_PLAINTEXT ACL To: users@kafka.apache.org Date: Friday, June 16, 2017, 4:15 PM Hi Vahid I am working on the same use case ): . As per the document I was trying to set ACL's for topic which worked and now I am able to start my producer without error. Then I set ACL for the consumer and when I start my consumer it starts without issue. and also able to set second ACL for committing offsets to group arun-group which is also good. Current ACLs for resource `Group:arun-group`: User:admin has Allow permission for operations: All from hosts: * User:arun has Allow permission for operations: All from hosts: * User:arun has Allow permission for operations: Read from hosts: * User:admin has Allow permission for operations: Read from hosts: * Then when I try to get the proper listing of offsets in the group I get the following error bin/kafka-consumer-groups --bootstrap-server producerhost:9097 --group arun-group --describe --command-config etc/kafka/producer.properties Note: This will only show information about consumers that use the Java consumer API (non-ZooKeeper-based consumers). [2017-06-16 16:05:42,535] INFO Successfully logged in. (org.apache.kafka.common.security.authenticator.AbstractLogin) Error: Executing consumer group command failed due to The group coordinator is not available. Also I don't see the message producer posted on consumer. I am using the confluent opensource kafka which is bundled with Zookeeper - 3.4.6 Kafka - 10.2.0 Please let me know if you need any more info. I appreciate your time. Thanks Arunkumar Pichaimuthu, PMP -------------------------------------------- On Fri, 6/16/17, Vahid S Hashemian <vahidhashem...@us.ibm.com> wrote: Subject: Re: UNKNOWN_TOPIC_OR_PARTITION with SASL_PLAINTEXT ACL To: users@kafka.apache.org Date: Friday, June 16, 2017, 1:56 PM Hi Arunkumar, Were you trying the same steps in the document when you got this error? Or you are working on a different use case? Also, I might have missed it in previous emails. What version of Kafka are you using? Thanks. --Vahid From: Arunkumar <pm_arunku...@yahoo.com.INVALID> To: <users@kafka.apache.org> Date: 06/16/2017 10:22 AM Subject: Re: UNKNOWN_TOPIC_OR_PARTITION with SASL_PLAINTEXT ACL Hi Vahid Thank you for sharing link to set it up. It is really a very useful document. When I ran describe command for group I see this error bin/kafka-consumer-groups --bootstrap-server host:9097 --describe --group arun-group --command-config etc/kafka/producer.properties Note: This will only show information about consumers that use the Java consumer API (non-ZooKeeper-based consumers). [2017-06-16 11:32:23,790] INFO Successfully logged in. (org.apache.kafka.common.security.authenticator.AbstractLogin) Error: Executing consumer group command failed due to The group coordinator is not available. I googled to figure out the issue and many say that it may be because of the port which I am not convinced. Any help is highly appreciated. Thanks Arunkumar Pichaimuthu, PMP -------------------------------------------- On Thu, 6/15/17, Vahid S Hashemian <vahidhashem...@us.ibm.com> wrote: Subject: Re: UNKNOWN_TOPIC_OR_PARTITION with SASL_PLAINTEXT ACL To: users@kafka.apache.org Date: Thursday, June 15, 2017, 6:49 PM Hi Arunkumar, Could you please take a look at this article: https://developer.ibm.com/opentech/2017/05/31/kafka-acls-in-practice/ The error message you posted earlier suggests there is some missing ACL (as indicated in the article). Let me know if that doesn't resolve the issue. Thanks. --Vahid From: Arunkumar <pm_arunku...@yahoo.com.INVALID> To: <users@kafka.apache.org> Date: 06/15/2017 04:37 PM Subject: Re: UNKNOWN_TOPIC_OR_PARTITION with SASL_PLAINTEXT ACL Hi Vahid Thank you for quick response. I set the ACL for topic and also created jaas permission as per the document for both producer and consumer. I have set what I have posted below. Do I need to set ACL like we set for Topics -- bin/kafka-acls --topic * --add -allow-host host:9097 --allow-principal User:arun --operation Write --authorizer-properties zookeeper.connect=host:2182 ? Please let me know. If you need all configuration for zookeeper, Broker, producer and consumer. I can share it as well. Thanks in advance KafkaServer { org.apache.kafka.common.security.plain.PlainLoginModule required username="admin" password="admin-secret" user_admin="admin-secret" user_arun="Arun123"; }; Client { org.apache.kafka.common.security.plain.PlainLoginModule required username="arun" password="Arun123"; }; KafkaClient { org.apache.kafka.common.security.plain.PlainLoginModule required username="arun" password="Arun123"; }; Thanks Arunkumar Pichaimuthu, PMP -------------------------------------------- On Thu, 6/15/17, Vahid S Hashemian <vahidhashem...@us.ibm.com> wrote: Subject: Re: UNKNOWN_TOPIC_OR_PARTITION with SASL_PLAINTEXT ACL To: users@kafka.apache.org Date: Thursday, June 15, 2017, 6:16 PM Hi Arunkumar, Have you given your Kafka consumer/producer necessary permissions to consume/produce messages? --Vahid From: Arunkumar <pm_arunku...@yahoo.com.INVALID> To: <users@kafka.apache.org> Date: 06/15/2017 04:07 PM Subject: UNKNOWN_TOPIC_OR_PARTITION with SASL_PLAINTEXT ACL Hi I am setting up ACL with SALS_PLAINTEXT. My zookeeper and broker starts without error. But when I try to start my consumer or if I send message through a producer it throws an exception (Both producer and consumer are kafka CLI) Stack trace for my consumer below. Any insight is highly appreciated. Thanks in advance bin/kafka-console-consumer --topic sample1 --from-beginning --consumer.config=etc/kafka/consumer.properties --bootstrap-server hostname:9097 [2017-06-15 17:21:45,286] INFO ConsumerConfig values: auto.commit.interval.ms = 5000 auto.offset.reset = earliest bootstrap.servers = [hostname:9097] check.crcs = true client.id = connections.max.idle.ms = 540000 enable.auto.commit = true exclude.internal.topics = true fetch.max.bytes = 52428800 fetch.max.wait.ms = 500 fetch.min.bytes = 1 group.id = test-consumer-group heartbeat.interval.ms = 1000 interceptor.classes = null key.deserializer = class org.apache.kafka.common.serialization.ByteArrayDeserializer max.partition.fetch.bytes = 1048576 max.poll.interval.ms = 300000 max.poll.records = 500 metadata.max.age.ms = 300000 metric.reporters = [] metrics.num.samples = 2 metrics.recording.level = INFO metrics.sample.window.ms = 30000 partition.assignment.strategy = [class org.apache.kafka.clients.consumer.RangeAssignor] receive.buffer.bytes = 65536 reconnect.backoff.ms = 50 request.timeout.ms = 305000 retry.backoff.ms = 100 sasl.jaas.config = null sasl.kerberos.kinit.cmd = /usr/bin/kinit sasl.kerberos.min.time.before.relogin = 60000 sasl.kerberos.service.name = null sasl.kerberos.ticket.renew.jitter = 0.05 sasl.kerberos.ticket.renew.window.factor = 0.8 sasl.mechanism = PLAIN security.protocol = SASL_PLAINTEXT send.buffer.bytes = 131072 session.timeout.ms = 10000 ssl.cipher.suites = null ssl.enabled.protocols = [TLSv1.2, TLSv1.1, TLSv1] ssl.endpoint.identification.algorithm = null ssl.key.password = null ssl.keymanager.algorithm = SunX509 ssl.keystore.location = null ssl.keystore.password = null ssl.keystore.type = JKS ssl.protocol = TLS ssl.provider = null ssl.secure.random.implementation = null ssl.trustmanager.algorithm = PKIX ssl.truststore.location = path.truststore ssl.truststore.password = [hidden] ssl.truststore.type = JKS value.deserializer = class org.apache.kafka.common.serialization.ByteArrayDeserializer (org.apache.kafka.clients.consumer.ConsumerConfig) [2017-06-15 17:21:45,438] INFO Successfully logged in. (org.apache.kafka.common.security.authenticator.AbstractLogin) [2017-06-15 17:21:45,522] INFO Kafka version : 0.10.2.1-cp1 (org.apache.kafka.common.utils.AppInfoParser) [2017-06-15 17:21:45,523] INFO Kafka commitId : 078e7dc02a100018 (org.apache.kafka.common.utils.AppInfoParser) [2017-06-15 17:21:45,781] WARN Error while fetching metadata with correlation id 2 : {sample1=UNKNOWN_TOPIC_OR_PARTITION} (org.apache.kafka.clients.NetworkClient) [2017-06-15 17:21:45,878] WARN Error while fetching metadata with correlation id 3 : {sample1=UNKNOWN_TOPIC_OR_PARTITION} (org.apache.kafka.clients.NetworkClient) [2017-06-15 17:21:45,980] WARN Error while fetching metadata with correlation id 4 : {sample1=UNKNOWN_TOPIC_OR_PARTITION} (org.apache.kafka.clients.NetworkClient) [2017-06-15 17:21:46,084] WARN Error while fetching metadata with correlation id 5 : {sample1=UNKNOWN_TOPIC_OR_PARTITION} (org.apache.kafka.clients.NetworkClient) [2017-06-15 17:21:46,185] WARN Error while fetching metadata with correlation id 6 : {sample1=UNKNOWN_TOPIC_OR_PARTITION} (org.apache.kafka.clients.NetworkClient) [2017-06-15 17:21:46,289] WARN Error while fetching metadata with correlation id 7 : {sample1=UNKNOWN_TOPIC_OR_PARTITION} (org.apache.kafka.clients.NetworkClient) [2017-06-15 17:21:46,392] WARN Error while fetching metadata with correlation id 8 : {sample1=UNKNOWN_TOPIC_OR_PARTITION} (org.apache.kafka.clients.NetworkClient) [2017-06-15 17:21:46,495] WARN Error while fetching metadata with correlation id 9 : {sample1=UNKNOWN_TOPIC_OR_PARTITION} (org.apache.kafka.clients.NetworkClient) [2017-06-15 17:21:46,598] WARN Error while fetching metadata with correlation id 10 : {sample1=UNKNOWN_TOPIC_OR_PARTITION} (org.apache.kafka.clients.NetworkClient) [2017-06-15 17:21:46,702] WARN Error while fetching metadata with correlation id 11 : {sample1=UNKNOWN_TOPIC_OR_PARTITION} (org.apache.kafka.clients.NetworkClient) [2017-06-15 17:21:46,804] WARN Error while fetching metadata with correlation id 12 : {sample1=UNKNOWN_TOPIC_OR_PARTITION} (org.apache.kafka.clients.NetworkClient) [2017-06-15 17:21:46,908] WARN Error while fetching metadata with correlation id 13 : {sample1=UNKNOWN_TOPIC_OR_PARTITION} (org.apache.kafka.clients.NetworkClient) [2017-06-15 17:21:47,013] WARN Error while fetching metadata with correlation id 14 : {sample1=UNKNOWN_TOPIC_OR_PARTITION} (org.apache.kafka.clients.NetworkClient) [2017-06-15 17:21:47,117] WARN Error while fetching metadata with correlation id 15 : {sample1=UNKNOWN_TOPIC_OR_PARTITION} (org.apache.kafka.clients.NetworkClient) [2017-06-15 17:21:47,220] WARN Error while fetching metadata with correlation id 16 : {sample1=UNKNOWN_TOPIC_OR_PARTITION} (org.apache.kafka.clients.NetworkClient) [2017-06-15 17:21:47,323] WARN Error while fetching metadata with correlation id 17 : {sample1=UNKNOWN_TOPIC_OR_PARTITION} (org.apache.kafka.clients.NetworkClient) [2017-06-15 17:21:47,426] WARN Error while fetching metadata with correlation id 18 : {sample1=UNKNOWN_TOPIC_OR_PARTITION} (org.apache.kafka.clients.NetworkClient) [2017-06-15 17:21:47,530] WARN Error while fetching metadata with correlation id 19 : {sample1=UNKNOWN_TOPIC_OR_PARTITION} (org.apache.kafka.clients.NetworkClient) [2017-06-15 17:21:47,634] WARN Error while fetching metadata with correlation id 20 : {sample1=UNKNOWN_TOPIC_OR_PARTITION} (org.apache.kafka.clients.NetworkClient) Thanks Arunkumar Pichaimuthu, PMP
