Hi Vahid
I am working on the same use case ): . As per the document I was trying to set
ACL's for topic which worked and now I am able to start my producer without
error.
Then I set ACL for the consumer and when I start my consumer it starts without
issue. and also able to set second ACL for committing offsets to group
arun-group which is also good.
Current ACLs for resource `Group:arun-group`:
User:admin has Allow permission for operations: All from hosts: *
User:arun has Allow permission for operations: All from hosts: *
User:arun has Allow permission for operations: Read from hosts: *
User:admin has Allow permission for operations: Read from hosts: *
Then when I try to get the proper listing of offsets in the group I get the
following error
bin/kafka-consumer-groups --bootstrap-server producerhost:9097 --group
arun-group --describe --command-config etc/kafka/producer.properties
Note: This will only show information about consumers that use the Java
consumer API (non-ZooKeeper-based consumers).
[2017-06-16 16:05:42,535] INFO Successfully logged in.
(org.apache.kafka.common.security.authenticator.AbstractLogin)
Error: Executing consumer group command failed due to The group coordinator is
not available.
Also I don't see the message producer posted on consumer.
I am using the confluent opensource kafka which is bundled with
Zookeeper - 3.4.6
Kafka - 10.2.0
Please let me know if you need any more info. I appreciate your time.
Thanks
Arunkumar Pichaimuthu, PMP
--------------------------------------------
On Fri, 6/16/17, Vahid S Hashemian <[email protected]> wrote:
Subject: Re: UNKNOWN_TOPIC_OR_PARTITION with SASL_PLAINTEXT ACL
To: [email protected]
Date: Friday, June 16, 2017, 1:56 PM
Hi Arunkumar,
Were you trying the same steps in the
document when you got this error? Or
you are working on a different use
case?
Also, I might have missed it in
previous emails. What version of Kafka are
you using?
Thanks.
--Vahid
From: Arunkumar <[email protected]>
To: <[email protected]>
Date: 06/16/2017 10:22 AM
Subject: Re:
UNKNOWN_TOPIC_OR_PARTITION with SASL_PLAINTEXT ACL
Hi Vahid
Thank you for sharing link to set it
up. It is really a very useful
document. When I ran describe command
for group I see this error
bin/kafka-consumer-groups
--bootstrap-server host:9097 --describe --group
arun-group --command-config
etc/kafka/producer.properties
Note: This will only show information
about consumers that use the Java
consumer API (non-ZooKeeper-based
consumers).
[2017-06-16 11:32:23,790] INFO
Successfully logged in.
(org.apache.kafka.common.security.authenticator.AbstractLogin)
Error: Executing consumer group command
failed due to The group
coordinator is not available.
I googled to figure out the issue and
many say that it may be because of
the port which I am not convinced. Any
help is highly appreciated.
Thanks
Arunkumar Pichaimuthu, PMP
--------------------------------------------
On Thu, 6/15/17, Vahid S Hashemian
<[email protected]>
wrote:
Subject: Re:
UNKNOWN_TOPIC_OR_PARTITION with SASL_PLAINTEXT ACL
To: [email protected]
Date: Thursday, June 15, 2017, 6:49
PM
Hi Arunkumar,
Could you please take a look at this
article:
https://developer.ibm.com/opentech/2017/05/31/kafka-acls-in-practice/
The error message you posted earlier
suggests
there is some missing ACL
(as indicated in
the article).
Let me know
if that doesn't resolve the issue.
Thanks.
--Vahid
From:
Arunkumar <[email protected]>
To: <[email protected]>
Date: 06/15/2017 04:37 PM
Subject:
Re:
UNKNOWN_TOPIC_OR_PARTITION with
SASL_PLAINTEXT ACL
Hi Vahid
Thank you for quick
response.
I set the ACL for
topic and also created jaas permission
as per the
document for both producer and
consumer. I have
set what I have posted
below. Do I need to
set ACL like we set for Topics
-- bin/kafka-acls
--topic * --add -allow-host host:9097
--allow-principal User:arun
--operation
Write --authorizer-properties
zookeeper.connect=host:2182 ?
Please let me know. If you need all
configuration for zookeeper, Broker,
producer and consumer. I can share it
as well.
Thanks in advance
KafkaServer {
org.apache.kafka.common.security.plain.PlainLoginModule
required
username="admin"
password="admin-secret"
user_admin="admin-secret"
user_arun="Arun123";
};
Client {
org.apache.kafka.common.security.plain.PlainLoginModule
required
username="arun"
password="Arun123";
};
KafkaClient {
org.apache.kafka.common.security.plain.PlainLoginModule
required
username="arun"
password="Arun123";
};
Thanks
Arunkumar Pichaimuthu, PMP
--------------------------------------------
On Thu, 6/15/17, Vahid S Hashemian
<[email protected]>
wrote:
Subject: Re:
UNKNOWN_TOPIC_OR_PARTITION with
SASL_PLAINTEXT ACL
To: [email protected]
Date: Thursday, June 15, 2017,
6:16 PM
Hi Arunkumar,
Have you given your Kafka
consumer/producer
necessary permissions to
consume/produce
messages?
--Vahid
From: Arunkumar <[email protected]>
To: <[email protected]>
Date: 06/15/2017 04:07
PM
Subject:
UNKNOWN_TOPIC_OR_PARTITION
with SASL_PLAINTEXT ACL
Hi
I am setting up ACL with
SALS_PLAINTEXT. My
zookeeper and broker
starts
without error.
But
when I try to start my consumer or if
I send message
through a producer it throws an
exception
(Both
producer and consumer are
kafka CLI)
Stack trace for my
consumer below. Any insight
is highly
appreciated.
Thanks in advance
bin/kafka-console-consumer
--topic sample1
--from-beginning
--consumer.config=etc/kafka/consumer.properties
--bootstrap-server
hostname:9097
[2017-06-15 17:21:45,286]
INFO ConsumerConfig
values:
auto.commit.interval.ms
= 5000
auto.offset.reset =
earliest
bootstrap.servers =
[hostname:9097]
check.crcs =
true
client.id =
connections.max.idle.ms = 540000
enable.auto.commit = true
exclude.internal.topics = true
fetch.max.bytes = 52428800
fetch.max.wait.ms = 500
fetch.min.bytes = 1
group.id =
test-consumer-group
heartbeat.interval.ms = 1000
interceptor.classes = null
key.deserializer = class
org.apache.kafka.common.serialization.ByteArrayDeserializer
max.partition.fetch.bytes =
1048576
max.poll.interval.ms =
300000
max.poll.records = 500
metadata.max.age.ms = 300000
metric.reporters = []
metrics.num.samples = 2
metrics.recording.level = INFO
metrics.sample.window.ms =
30000
partition.assignment.strategy =
[class
org.apache.kafka.clients.consumer.RangeAssignor]
receive.buffer.bytes = 65536
reconnect.backoff.ms = 50
request.timeout.ms = 305000
retry.backoff.ms = 100
sasl.jaas.config = null
sasl.kerberos.kinit.cmd =
/usr/bin/kinit
sasl.kerberos.min.time.before.relogin =
60000
sasl.kerberos.service.name =
null
sasl.kerberos.ticket.renew.jitter =
0.05
sasl.kerberos.ticket.renew.window.factor = 0.8
sasl.mechanism = PLAIN
security.protocol =
SASL_PLAINTEXT
send.buffer.bytes
= 131072
session.timeout.ms =
10000
ssl.cipher.suites =
null
ssl.enabled.protocols =
[TLSv1.2,
TLSv1.1, TLSv1]
ssl.endpoint.identification.algorithm =
null
ssl.key.password = null
ssl.keymanager.algorithm =
SunX509
ssl.keystore.location =
null
ssl.keystore.password =
null
ssl.keystore.type = JKS
ssl.protocol = TLS
ssl.provider
= null
ssl.secure.random.implementation =
null
ssl.trustmanager.algorithm
= PKIX
ssl.truststore.location =
path.truststore
ssl.truststore.password =
[hidden]
ssl.truststore.type = JKS
value.deserializer = class
org.apache.kafka.common.serialization.ByteArrayDeserializer
(org.apache.kafka.clients.consumer.ConsumerConfig)
[2017-06-15 17:21:45,438] INFO
Successfully
logged in.
(org.apache.kafka.common.security.authenticator.AbstractLogin)
[2017-06-15 17:21:45,522] INFO
Kafka version
:
0.10.2.1-cp1
(org.apache.kafka.common.utils.AppInfoParser)
[2017-06-15 17:21:45,523] INFO
Kafka commitId
:
078e7dc02a100018
(org.apache.kafka.common.utils.AppInfoParser)
[2017-06-15 17:21:45,781] WARN
Error while
fetching metadata with
correlation id 2 :
{sample1=UNKNOWN_TOPIC_OR_PARTITION}
(org.apache.kafka.clients.NetworkClient)
[2017-06-15 17:21:45,878] WARN Error
while
fetching metadata with
correlation id 3
:
{sample1=UNKNOWN_TOPIC_OR_PARTITION}
(org.apache.kafka.clients.NetworkClient)
[2017-06-15 17:21:45,980] WARN
Error while
fetching metadata with
correlation id 4 :
{sample1=UNKNOWN_TOPIC_OR_PARTITION}
(org.apache.kafka.clients.NetworkClient)
[2017-06-15 17:21:46,084] WARN Error
while
fetching metadata with
correlation id 5
:
{sample1=UNKNOWN_TOPIC_OR_PARTITION}
(org.apache.kafka.clients.NetworkClient)
[2017-06-15 17:21:46,185] WARN
Error while
fetching metadata with
correlation id 6 :
{sample1=UNKNOWN_TOPIC_OR_PARTITION}
(org.apache.kafka.clients.NetworkClient)
[2017-06-15 17:21:46,289] WARN Error
while
fetching metadata with
correlation id 7
:
{sample1=UNKNOWN_TOPIC_OR_PARTITION}
(org.apache.kafka.clients.NetworkClient)
[2017-06-15 17:21:46,392] WARN
Error while
fetching metadata with
correlation id 8 :
{sample1=UNKNOWN_TOPIC_OR_PARTITION}
(org.apache.kafka.clients.NetworkClient)
[2017-06-15 17:21:46,495] WARN Error
while
fetching metadata with
correlation id 9
:
{sample1=UNKNOWN_TOPIC_OR_PARTITION}
(org.apache.kafka.clients.NetworkClient)
[2017-06-15 17:21:46,598] WARN
Error while
fetching metadata with
correlation id 10 :
{sample1=UNKNOWN_TOPIC_OR_PARTITION}
(org.apache.kafka.clients.NetworkClient)
[2017-06-15 17:21:46,702] WARN Error
while
fetching metadata with
correlation id 11
:
{sample1=UNKNOWN_TOPIC_OR_PARTITION}
(org.apache.kafka.clients.NetworkClient)
[2017-06-15 17:21:46,804] WARN
Error while
fetching metadata with
correlation id 12 :
{sample1=UNKNOWN_TOPIC_OR_PARTITION}
(org.apache.kafka.clients.NetworkClient)
[2017-06-15 17:21:46,908] WARN Error
while
fetching metadata with
correlation id 13
:
{sample1=UNKNOWN_TOPIC_OR_PARTITION}
(org.apache.kafka.clients.NetworkClient)
[2017-06-15 17:21:47,013] WARN
Error while
fetching metadata with
correlation id 14 :
{sample1=UNKNOWN_TOPIC_OR_PARTITION}
(org.apache.kafka.clients.NetworkClient)
[2017-06-15 17:21:47,117] WARN Error
while
fetching metadata with
correlation id 15
:
{sample1=UNKNOWN_TOPIC_OR_PARTITION}
(org.apache.kafka.clients.NetworkClient)
[2017-06-15 17:21:47,220] WARN
Error while
fetching metadata with
correlation id 16 :
{sample1=UNKNOWN_TOPIC_OR_PARTITION}
(org.apache.kafka.clients.NetworkClient)
[2017-06-15 17:21:47,323] WARN Error
while
fetching metadata with
correlation id 17
:
{sample1=UNKNOWN_TOPIC_OR_PARTITION}
(org.apache.kafka.clients.NetworkClient)
[2017-06-15 17:21:47,426] WARN
Error while
fetching metadata with
correlation id 18 :
{sample1=UNKNOWN_TOPIC_OR_PARTITION}
(org.apache.kafka.clients.NetworkClient)
[2017-06-15 17:21:47,530] WARN Error
while
fetching metadata with
correlation id 19
:
{sample1=UNKNOWN_TOPIC_OR_PARTITION}
(org.apache.kafka.clients.NetworkClient)
[2017-06-15 17:21:47,634] WARN
Error while
fetching metadata with
correlation id 20 :
{sample1=UNKNOWN_TOPIC_OR_PARTITION}
(org.apache.kafka.clients.NetworkClient)
Thanks
Arunkumar Pichaimuthu, PMP
