Re: Slow Consumer Group Startup

[J Pai]

When are the topics on which these consumer groups consume, created?

-Jaikiran
On 13-Jun-2017, at 3:18 AM, Bryan Baugher  wrote:

Hi everyone,

We are currently experiencing slow startup times for our consumer groups
(16-32 processes for a hundred or more partitions) in the range of minutes
(3-15 minutes), where little to no messages are consumed before suddenly
everything just starts working at full speed.

I'm currently using Kafka 0.9.0.1 but we are in the middle of upgrading to
Kafka 0.10.2.1. We also using the newer kafka consumer API and group
management on a simple Apache Storm topology. We don't make use of Storm's
kafka spout but instead wrote a simple one ourselves.

Using the kafka AdminClient I was able to poll for consumer group summary
information. What I've found is that the group seems to sit
in PreparingRebalance state for minutes before finally becoming Stable
which then everything starts processing quickly. I've also enabled debug
logging around the consumer's coordinator classes but didn't see anything
to indicate the issue.

I'm hoping that just upgrading to 0.10 or tweaking how we use our consumer
in Apache Storm is the problem but are there any pointers on things I
should look at or try?

Bryan

Re: Kafka 0.10.1 cluster using 100% disk usage (reads)

[Ismael Juma]

For what is worth, 0.9.0.0 and above don't accept messages with no key for
compacted topics.

Ismael

On Mon, Jun 12, 2017 at 10:17 PM, ext-gfenol...@eramet-sln.nc <
ext-gfenol...@eramet-sln.nc> wrote:

> Hello Milind, and thank you for your answer.
>
> We just found the solution to our problem.
> Apparently, some developers were sending bad messages with no key, so the
> log cleaner couldn’t wipe them properly and crashed at startup. The topics
> were set in compact mode.
> We have set the compact.policy to delete instead and after the cleaning,
> the problem disappeared, and we were back to a normal IO operation.
>
> I consider this as a serious bug, as we’re running on SSDs and several
> weeks or less of this 100% usage can easily break this kind of hardware.
>
> Guillaume
>
>
> __
> FENOLLAR Guillaume (SLN-EXT)
> Prestataire pour DSI/ESI
> Société Le Nickel - SLN
>
>
>
> Courriel : ext-gfenol...@eramet-sln.nc
> Site internet : www.sln.nc
> Page Facebook : LeNickel.SLN
> __
>
> -Message d'origine-
> De : Milind Vaidya [mailto:kava...@gmail.com]
> Envoyé : mardi 13 juin 2017 05:17
> À : users@kafka.apache.org
> Objet : Re: Kafka 0.10.1 cluster using 100% disk usage (reads)
>
> This sound exactly similar to what I experienced in the similar scenario.
>
> Can you please take a look at the File System time stamp of the actual log
> files on one of the broker hosts ?
>
> For me when I restarted the new brokers with version 0.10.0 it changes to
> current ts. Meaning if I have set 48 hrs of retention window, the file
> which is about to get deleted is stamped with latest ts so it will be
> scheduled to be deleted 48 hrs later. More times you restart the broker it
> will push deletion operation further into future. So this will lead to
> additional data retained on the disk.
>
> I posted the same question to which I was told it should be fixed in
> 0.10.1 but I observed same behaviour.
> Apparently the change from 0.8 to 0.10 is that, the file ts of the log
> chunk is not more taken into consideration but the time stamp with in the
> message as set by new protocol is used and it is backward compatible,
> meaning 0.8 logs files will be treated for their actual file TS. But does
> not look like in practical.
>
> Here are solutions I used.
>
> 1. Reduce the retention period if you can so the disk will still hold the
> logs till they expire in near future after which things become smooth 2.
> Use log.retention.bytes option for brokers to limit size of the logs
> retained.
>
> My request is that this should be clearly added to documentation of
> upgrade.
>
> Hope this helps.
>
>
> On Sun, Jun 11, 2017 at 7:46 PM, ext-gfenol...@eramet-sln.nc <
> ext-gfenol...@eramet-sln.nc> wrote:
>
> > Hello,
> >
> > Since we upgraded from Kafka 0.8 to Kafka0.10.1, our brokers are using
> > 100% of our disk capacity (100% util in iostat), with from 100MB/s to
> > 1000MB/s constant read stats.
> >
> > It’s been half a week now, and usage is not decreasing. Note that we
> > didn’t experience that before the upgrade.
> >
> >
> >
> > There’s nothing particularly helpful in the logs (except the fact that
> > we have corrupted index files at startup that kafka recreated
> > corrupted again, but it was already present in Kafka 0.8).
> >
> > It leads to a fairly high amount of CPU iowait, in all our environments.
> >
> > Even weirder thing is that the usage is not exactly the same on all
> > brokers. In first one, we have 100MB/s, 200MB/s for the 2nd one, and
> > 1000MB/s for the 3rd one, all of them having the same datastore, same
> > disk speed.
> >
> >
> >
> > With htop, I can see that only one Java thread is doing that, but I’m
> > not sure how to gather much info about it.
> >
> >
> >
> > Can you help ?
> >
> > Thanks,
> >
> >
> >
> > Guillaume
> >
> >
> >
> >
> >
> > *Guillaume FENOLLAR*
> >
> > Prestataire pour DSI/ESI
> >
> > Société Le Nickel - SLN
> >
> > Courriel : ext-gfenol...@eramet-sln.nc
> >
> > Site internet : www.sln.nc
> >
> > Page Facebook : LeNickel.SLN
> >
> >
> >
> >
> >
> >
> > --
> > CONFIDENTIALITE
> > L'information contenue dans ce courrier électronique et ses pièces
> > jointes est confidentielle, et est établie à l'intention exclusive de
> > ses destinataires. Dans le cas où ce message ne vous serait pas
> > destiné, nous vous remercions de bien vouloir en aviser immédiatement
> > l'émetteur et de procéder à sa suppression. Toutes copies, diffusions
> > ou accès non autorisés à ce message sont interdits à toutes personnes,
> > autre que le(s) destinataire(s). Un courrier électronique est
> > susceptible d’altération ou de falsification et peut entrainer des
> > pertes et/ou la destruction de données. Le Groupe ERAMET et/ou ses
> > filiales déclinent toute responsabilité en la matière. En conséquence
> > ce courrier électronique ainsi que ses pièces jointes sont utilisés à
> votre propre 

Slow Consumer Group Startup

[Bryan Baugher]

Hi everyone,

We are currently experiencing slow startup times for our consumer groups
(16-32 processes for a hundred or more partitions) in the range of minutes
(3-15 minutes), where little to no messages are consumed before suddenly
everything just starts working at full speed.

I'm currently using Kafka 0.9.0.1 but we are in the middle of upgrading to
Kafka 0.10.2.1. We also using the newer kafka consumer API and group
management on a simple Apache Storm topology. We don't make use of Storm's
kafka spout but instead wrote a simple one ourselves.

Using the kafka AdminClient I was able to poll for consumer group summary
information. What I've found is that the group seems to sit
in PreparingRebalance state for minutes before finally becoming Stable
which then everything starts processing quickly. I've also enabled debug
logging around the consumer's coordinator classes but didn't see anything
to indicate the issue.

I'm hoping that just upgrading to 0.10 or tweaking how we use our consumer
in Apache Storm is the problem but are there any pointers on things I
should look at or try?

Bryan

RE: Kafka 0.10.1 cluster using 100% disk usage (reads)

[ext-gfenol...@eramet-sln.nc]

Hello Milind, and thank you for your answer.

We just found the solution to our problem.
Apparently, some developers were sending bad messages with no key, so the log 
cleaner couldn’t wipe them properly and crashed at startup. The topics were set 
in compact mode.
We have set the compact.policy to delete instead and after the cleaning, the 
problem disappeared, and we were back to a normal IO operation.

I consider this as a serious bug, as we’re running on SSDs and several weeks or 
less of this 100% usage can easily break this kind of hardware.

Guillaume


__
FENOLLAR Guillaume (SLN-EXT)
Prestataire pour DSI/ESI
Société Le Nickel - SLN



Courriel : ext-gfenol...@eramet-sln.nc
Site internet : www.sln.nc
Page Facebook : LeNickel.SLN
__

-Message d'origine-
De : Milind Vaidya [mailto:kava...@gmail.com]
Envoyé : mardi 13 juin 2017 05:17
À : users@kafka.apache.org
Objet : Re: Kafka 0.10.1 cluster using 100% disk usage (reads)

This sound exactly similar to what I experienced in the similar scenario.

Can you please take a look at the File System time stamp of the actual log 
files on one of the broker hosts ?

For me when I restarted the new brokers with version 0.10.0 it changes to 
current ts. Meaning if I have set 48 hrs of retention window, the file which is 
about to get deleted is stamped with latest ts so it will be scheduled to be 
deleted 48 hrs later. More times you restart the broker it will push deletion 
operation further into future. So this will lead to additional data retained on 
the disk.

I posted the same question to which I was told it should be fixed in 0.10.1 but 
I observed same behaviour.
Apparently the change from 0.8 to 0.10 is that, the file ts of the log chunk is 
not more taken into consideration but the time stamp with in the message as set 
by new protocol is used and it is backward compatible, meaning 0.8 logs files 
will be treated for their actual file TS. But does not look like in practical.

Here are solutions I used.

1. Reduce the retention period if you can so the disk will still hold the logs 
till they expire in near future after which things become smooth 2. Use 
log.retention.bytes option for brokers to limit size of the logs retained.

My request is that this should be clearly added to documentation of upgrade.

Hope this helps.


On Sun, Jun 11, 2017 at 7:46 PM, ext-gfenol...@eramet-sln.nc < 
ext-gfenol...@eramet-sln.nc> wrote:

> Hello,
>
> Since we upgraded from Kafka 0.8 to Kafka0.10.1, our brokers are using
> 100% of our disk capacity (100% util in iostat), with from 100MB/s to
> 1000MB/s constant read stats.
>
> It’s been half a week now, and usage is not decreasing. Note that we
> didn’t experience that before the upgrade.
>
>
>
> There’s nothing particularly helpful in the logs (except the fact that
> we have corrupted index files at startup that kafka recreated
> corrupted again, but it was already present in Kafka 0.8).
>
> It leads to a fairly high amount of CPU iowait, in all our environments.
>
> Even weirder thing is that the usage is not exactly the same on all
> brokers. In first one, we have 100MB/s, 200MB/s for the 2nd one, and
> 1000MB/s for the 3rd one, all of them having the same datastore, same
> disk speed.
>
>
>
> With htop, I can see that only one Java thread is doing that, but I’m
> not sure how to gather much info about it.
>
>
>
> Can you help ?
>
> Thanks,
>
>
>
> Guillaume
>
>
>
>
>
> *Guillaume FENOLLAR*
>
> Prestataire pour DSI/ESI
>
> Société Le Nickel - SLN
>
> Courriel : ext-gfenol...@eramet-sln.nc
>
> Site internet : www.sln.nc
>
> Page Facebook : LeNickel.SLN
>
>
>
>
>
>
> --
> CONFIDENTIALITE
> L'information contenue dans ce courrier électronique et ses pièces
> jointes est confidentielle, et est établie à l'intention exclusive de
> ses destinataires. Dans le cas où ce message ne vous serait pas
> destiné, nous vous remercions de bien vouloir en aviser immédiatement
> l'émetteur et de procéder à sa suppression. Toutes copies, diffusions
> ou accès non autorisés à ce message sont interdits à toutes personnes,
> autre que le(s) destinataire(s). Un courrier électronique est
> susceptible d’altération ou de falsification et peut entrainer des
> pertes et/ou la destruction de données. Le Groupe ERAMET et/ou ses
> filiales déclinent toute responsabilité en la matière. En conséquence
> ce courrier électronique ainsi que ses pièces jointes sont utilisés à votre 
> propre risque.
>
> CONFIDENTIALITY
> The information contained in this e-mail and any accompanying
> documents is confidential or otherwise protected from disclosure. If
> you are not the intended recipient, please immediately alert the
> sender by reply e-mail and delete this message and any attachments.
> Any copy, dissemination or unauthorized access of the contents of this
> message by anyone other than the intended recipient is strictly
> 

mirror-maker-thread-failure

[Fernando Vega]

Does anyone has experience or had similar issue like

https://community.hortonworks.com/questions/106969/mirror-maker-thread-failure-due-to-kafkatoolsmirro.html

If you can point me to something that I can try that will be highly
appreciated

Thank you

Re: Kafka 0.10.1 cluster using 100% disk usage (reads)

[Milind Vaidya]

This sound exactly similar to what I experienced in the similar scenario.

Can you please take a look at the File System time stamp of the actual log
files on one of the broker hosts ?

For me when I restarted the new brokers with version 0.10.0 it changes to
current ts. Meaning if I have set 48 hrs of retention window, the file
which is about to get deleted is stamped with latest ts so it will be
scheduled to be deleted 48 hrs later. More times you restart the broker it
will push deletion operation further into future. So this will lead to
additional data retained on the disk.

I posted the same question to which I was told it should be fixed in 0.10.1
but I observed same behaviour.
Apparently the change from 0.8 to 0.10 is that, the file ts of the log
chunk is not more taken into consideration but the time stamp with in the
message as set by new protocol is used and it is backward compatible,
meaning 0.8 logs files will be treated for their actual file TS. But does
not look like in practical.

Here are solutions I used.

1. Reduce the retention period if you can so the disk will still hold the
logs till they expire in near future after which things become smooth
2. Use log.retention.bytes option for brokers to limit size of the logs
retained.

My request is that this should be clearly added to documentation of
upgrade.

Hope this helps.


On Sun, Jun 11, 2017 at 7:46 PM, ext-gfenol...@eramet-sln.nc <
ext-gfenol...@eramet-sln.nc> wrote:

> Hello,
>
> Since we upgraded from Kafka 0.8 to Kafka0.10.1, our brokers are using
> 100% of our disk capacity (100% util in iostat), with from 100MB/s to
> 1000MB/s constant read stats.
>
> It’s been half a week now, and usage is not decreasing. Note that we
> didn’t experience that before the upgrade.
>
>
>
> There’s nothing particularly helpful in the logs (except the fact that we
> have corrupted index files at startup that kafka recreated corrupted again,
> but it was already present in Kafka 0.8).
>
> It leads to a fairly high amount of CPU iowait, in all our environments.
>
> Even weirder thing is that the usage is not exactly the same on all
> brokers. In first one, we have 100MB/s, 200MB/s for the 2nd one, and
> 1000MB/s for the 3rd one, all of them having the same datastore, same
> disk speed.
>
>
>
> With htop, I can see that only one Java thread is doing that, but I’m not
> sure how to gather much info about it.
>
>
>
> Can you help ?
>
> Thanks,
>
>
>
> Guillaume
>
>
>
>
>
> *Guillaume FENOLLAR*
>
> Prestataire pour DSI/ESI
>
> Société Le Nickel - SLN
>
> Courriel : ext-gfenol...@eramet-sln.nc
>
> Site internet : www.sln.nc
>
> Page Facebook : LeNickel.SLN
>
>
>
>
>
>
> --
> CONFIDENTIALITE
> L'information contenue dans ce courrier électronique et ses pièces jointes
> est confidentielle, et est établie à l'intention exclusive de ses
> destinataires. Dans le cas où ce message ne vous serait pas destiné, nous
> vous remercions de bien vouloir en aviser immédiatement l'émetteur et de
> procéder à sa suppression. Toutes copies, diffusions ou accès non autorisés
> à ce message sont interdits à toutes personnes, autre que le(s)
> destinataire(s). Un courrier électronique est susceptible d’altération ou
> de falsification et peut entrainer des pertes et/ou la destruction de
> données. Le Groupe ERAMET et/ou ses filiales déclinent toute responsabilité
> en la matière. En conséquence ce courrier électronique ainsi que ses pièces
> jointes sont utilisés à votre propre risque.
>
> CONFIDENTIALITY
> The information contained in this e-mail and any accompanying documents is
> confidential or otherwise protected from disclosure. If you are not the
> intended recipient, please immediately alert the sender by reply e-mail and
> delete this message and any attachments. Any copy, dissemination or
> unauthorized access of the contents of this message by anyone other than
> the intended recipient is strictly prohibited. E-mails may be susceptible
> to falsification or alteration and cause data corruption and/or loss of
> data. ERAMET and/or any of its subsidiaries decline any liability resulting
> from the consequences thereof. Therefore, this e-mail and any attachments
> are used at your own risk.
>

Re: [DISCUSS] KIP-163: Lower the Minimum Required ACL Permission of OffsetFetch

[Vahid S Hashemian]

Hi Colin,

Thanks for the feedback.

To be honest, I'm not sure either why Read was selected instead of Write 
for mutating APIs in the initial design (I asked Ewen on the corresponding 
JIRA and he seemed unsure too).
Perhaps someone who was involved in the design can clarify.

Thanks.
--Vahid




From:   Colin McCabe 
To: users@kafka.apache.org
Date:   06/12/2017 10:11 AM
Subject:Re: [DISCUSS] KIP-163: Lower the Minimum Required ACL 
Permission of OffsetFetch



Hi Vahid,

I think you make a valid point that the ACLs controlling group
operations are not very intuitive.

This is probably a dumb question, but why are we using Read for mutating
APIs?  Shouldn't that be Write?

The distinction between Describe and Read makes a lot of sense for
Topics.  A group isn't really something that you "read" from in the same
way as a topic, so it always felt kind of weird there.

best,
Colin


On Thu, Jun 8, 2017, at 11:29, Vahid S Hashemian wrote:
> Hi all,
> 
> I'm resending my earlier note hoping it would spark some conversation
> this 
> time around :)
> 
> Thanks.
> --Vahid
> 
> 
> 
> 
> From:   "Vahid S Hashemian" 
> To: dev , "Kafka User" 

> Date:   05/30/2017 08:33 AM
> Subject:KIP-163: Lower the Minimum Required ACL Permission of 
> OffsetFetch
> 
> 
> 
> Hi,
> 
> I started a new KIP to improve the minimum required ACL permissions of 
> some of the APIs: 
> 
https://cwiki.apache.org/confluence/display/KAFKA/KIP-163%3A+Lower+the+Minimum+Required+ACL+Permission+of+OffsetFetch

> 
> The KIP is to address KAFKA-4585.
> 
> Feedback and suggestions are welcome!
> 
> Thanks.
> --Vahid
> 
> 
> 
> 
> 

Re: [DISCUSS] KIP-163: Lower the Minimum Required ACL Permission of OffsetFetch

[Colin McCabe]

Hi Vahid,

I think you make a valid point that the ACLs controlling group
operations are not very intuitive.

This is probably a dumb question, but why are we using Read for mutating
APIs?  Shouldn't that be Write?

The distinction between Describe and Read makes a lot of sense for
Topics.  A group isn't really something that you "read" from in the same
way as a topic, so it always felt kind of weird there.

best,
Colin


On Thu, Jun 8, 2017, at 11:29, Vahid S Hashemian wrote:
> Hi all,
> 
> I'm resending my earlier note hoping it would spark some conversation
> this 
> time around :)
> 
> Thanks.
> --Vahid
> 
> 
> 
> 
> From:   "Vahid S Hashemian" 
> To: dev , "Kafka User" 
> Date:   05/30/2017 08:33 AM
> Subject:KIP-163: Lower the Minimum Required ACL Permission of 
> OffsetFetch
> 
> 
> 
> Hi,
> 
> I started a new KIP to improve the minimum required ACL permissions of 
> some of the APIs: 
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-163%3A+Lower+the+Minimum+Required+ACL+Permission+of+OffsetFetch
> 
> The KIP is to address KAFKA-4585.
> 
> Feedback and suggestions are welcome!
> 
> Thanks.
> --Vahid
> 
> 
> 
> 
> 

kafka streams first time did not print output

[john cheng]

Hi there, I'm testing Kafka Streams's print() method, here is the code:

props.put(StreamsConfig.APPLICATION_ID_CONFIG, "dsl-wc1");
props.put(StreamsConfig.CACHE_MAX_BYTES_BUFFERING_CONFIG, 0);
KStream source = builder.stream("dsl-input1");
KTable countTable = source
.flatMapValues(value -> Arrays.asList(value.split(" ")))
.map((key,value) -> KeyValue.pair(value, value))
.groupByKey().count("Counts1");
countTable.print();  // [KSTREAM-AGGREGATE-03]: hello ,
(1<-null)
countTable.to(Serdes.String(), Serdes.Long(), "ktable-output1");
countTable.print();  // print after to, output is the same as print
before to

This App output as I expected, no matter print called before to() or after
to().

[KSTREAM-AGGREGATE-03]: hello , (1<-null)
[KSTREAM-AGGREGATE-03]: kafka , (1<-null)
[KSTREAM-AGGREGATE-03]: hello , (2<-null)
[KSTREAM-AGGREGATE-03]: kafka , (2<-null)
[KSTREAM-AGGREGATE-03]: streams , (1<-null)

Then I replace countTable.print() with below code:

countTable.to(Serdes.String(), Serdes.Long(), "ktable-output1");
KTable table1 = builder.table(Serdes.String(),
Serdes.Long(), "ktable-output1", "Counts_1");
table1.print();

But things happen strange. When first time startup the app, there're no
output.
After stop this app and restarted, The second time running app, there're
output now.

[KTABLE-SOURCE-10]: hello , (1<-null)
[KTABLE-SOURCE-10]: kafka , (1<-null)
[KTABLE-SOURCE-10]: hello , (2<-null)
[KTABLE-SOURCE-10]: kafka , (2<-null)
[KTABLE-SOURCE-10]: streams , (1<-null)

And when I append two message to input-topic, there're outpout too.

[KTABLE-SOURCE-10]: kafka , (3<-null)
[KTABLE-SOURCE-10]: streams , (2<-null)

MyQuestion is why no output present at first time? Did I miss something?

PS: If I have two app, the first app is just:
countTable.to("ktable-output1")
and the second app is: builder.table("ktable-output1").print()
The result is also like what I observed before.

Re: Kafka connector throughput reduction upon avro schema change

[Dave Hamilton]

Hi, does anyone have advice on how to deal with this issue? Is it possible that 
changing a schema compatibility setting could correct it?

Thanks,
Dave


On 5/26/17, 1:44 PM, "Dave Hamilton"  wrote:

We are currently using the Kafka S3 connector to ship Avro data to S3. We 
made a change to one of our Avro schemas and have noticed consumer throughput 
on the Kafka connector drop considerably. I am wondering if there is anything 
we can do to avoid such issues when we update schemas in the future?

This is what I believe is happening:


· The avro producer application is running on 12 instances. They 
are restarted in a rolling fashion, switching from producing schema version 1 
before the restart to schema version 2 afterward.

· While the rolling restart is occurring, data on schema version 1 
and schema version 2 is simultaneously being written to the topic.

· The Kafka connector has to close the current avro file for a 
partition and ship it whenever it detects a schema change, which is happening 
several times due to the rolling nature of the schema update deployment and the 
mixture of message versions being written during this time. This process causes 
the overall consumer throughput to plummet.

Am I reasoning correctly about what we’re observing here? Is there any way 
to avoid this when we change schemas (short of stopping all instances of the 
service and bringing them up together on the new schema version)?

Thanks,
Dave

Re: Why authorization deny Read ACL doesn't work

[linbo liao]

Finally I figure it out, I miss add super user in config/server.properties.
Now everything works fine.

2017-06-12 19:19 GMT+08:00 linbo liao :

> Thanks Tom, I miss it.
>
> I added authorizer.class.name=kafka.security.auth.SimpleAclAuthorizer,
> and grant user:alice to operation permission to cluster, but still failed
> when publish message. The error message told LEADER_NOT_AVAILABLE.
>
> $ bin/kafka-acls.sh --authorizer-properties zookeeper.connect=localhost:2181
>> --list
>> Current ACLs for resource `Topic:test`:
>> User:alice has Allow permission for operations: Write from hosts:
>> 127.0.0.1
>> User:alice has Deny permission for operations: Read from hosts: *
>>
>> Current ACLs for resource `Cluster:kafka-cluster`:
>> User:alice has Allow permission for operations: All from hosts: *
>>
>> $ bin/kafka-console-producer.sh --broker-list localhost:9092
>> --producer.config config/producer.properties --topic test
>> job
>> [2017-06-12 11:14:41,318] WARN Error while fetching metadata with
>> correlation id 1 : {test=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.
>> NetworkClient)
>>
>
>> $ bin/kafka-topics.sh --zookeeper localhost:2181 --topic test --describe
>> Topic:testPartitionCount:1ReplicationFactor:1Configs:
>> Topic: testPartition: 0Leader: 0Replicas: 0Isr: 0
>>
>
>
>
> 2017-06-12 16:43 GMT+08:00 Tom Bentley :
>
>> Hi,
>>
>> Did you set
>>
>> authorizer.class.name=kafka.security.auth.SimpleAclAuthorizer
>>
>> as described here
>>  at
>> http://docs.confluent.io/current/kafka/authorization.html#
>> further-configuration
>>
>> HTH,
>>
>> Tom
>>
>> On 11 June 2017 at 04:40, linbo liao  wrote:
>>
>> > Hi,
>> >
>> > I try to set Kafka ACL for topic access permission followed by kafka
>> > security document > entation/#security_authz>,
>> > but looks deny acl doesn't work.
>> >
>> > *My Environment:*
>> >
>> > VM: Ubuntu 12.04 LTS x86_64
>> > JAVA:  openjdk version "1.8.0_111"
>> > Kafka:  kafka_2.12-0.10.2.1
>> >
>> > I setup one broker, and use kafka-console-consumer.sh and
>> > kafka-console-producer.sh to test.
>> >
>> > *Broker setup:*
>> >
>> > broker startup script already add jaas parameter
>> >
>> > $ cat kafka_server_jaas.conf
>> >
>> > > KafkaServer {
>> > > org.apache.kafka.common.security.plain.PlainLoginModule
>> > > required
>> > > username="admin"
>> > > password="admin"
>> > > user_admin="admin"
>> > > user_alice="alice";
>> > > };
>> > >
>> >
>> > config/server.properties
>> >
>> > listeners=SASL_PLAINTEXT://0.0.0.0:9092
>> > > security.inter.broker.protocol=SASL_PLAINTEXT
>> > > sasl.mechanism.inter.broker.protocol=PLAIN
>> > > sasl.enabled.mechanisms=PLAIN
>> > >
>> >
>> >
>> > *Client setup:*
>> > producer/consumer startup script already add jaas parameter
>> >
>> > $ cat client_jaas.conf
>> >
>> > > KafkaClient {
>> > > org.apache.kafka.common.security.plain.PlainLoginModule required
>> > > username="alice"
>> > > password="alice";
>> > > };
>> >
>> >
>> > config/consumer.properties & config/producer.properties
>> >
>> > > security.protocol=SASL_PLAINTEXT
>> > > sasl.mechanism=PLAIN
>> > >
>> >
>> >
>> > 1. create topic
>> >
>> > $ bin/kafka-topics.sh --create --zookeeper localhost:2181
>> > > --replication-factor 1 --partitions 1 --topic test
>> > >
>> >
>> > 2. setup topic acl
>> >
>> > $ bin/kafka-acls.sh --authorizer-properties
>> > > zookeeper.connect=localhost:2181 --list --topic test
>> > > Current ACLs for resource `Topic:test`:
>> > > User:alice has Allow permission for operations: Write from
>> hosts:
>> > > 127.0.0.1
>> > > User:alice has Deny permission for operations: Read from
>> hosts: *
>> > >
>> >
>> > Although I deny Read permission for user alice from all host, I start
>> > consumer still can receive message.
>> >
>> > produce a message "test"
>> >
>> > > $ bin/kafka-console-producer.sh --broker-list localhost:9092
>> > > --producer.config config/producer.properties --topic test
>> > > test
>> > >
>> >
>> > consumer receive this message
>> >
>> > $ bin/kafka-console-consumer.sh --bootstrap-server localhost:9092
>> --topic
>> > > test --consumer.config config/consumer.properties --from-beginning
>> > > [2017-06-11 03:37:55,998] WARN The configuration 'zookeeper.connect'
>> was
>> > > supplied but isn't a known config.
>> > > (org.apache.kafka.clients.consumer.ConsumerConfig)
>> > > [2017-06-11 03:37:55,999] WARN The configuration '
>> > > zookeeper.connection.timeout.ms' was supplied but isn't a known
>> config.
>> > > (org.apache.kafka.clients.consumer.ConsumerConfig)
>> > > test
>> > >
>> >
>> > Why deny read operation doesn't work, do I miss something?
>> >
>> > Thanks,
>> > Linbo
>> >
>>
>
>

Login using LAP for Kafka

[Arunkumar]

Hi 

I have a requirement to configure LDAP Login for Kafka. I see there is a 
discussion going on to integrate it with future release of Kafka version. Is 
there any one who have done it ( I am sure some one would have done it ):- ) 
can help with sample configuration with code. I see we can extend the 
PlainLoginModule class and write our own implementation any insight is highly 
appreciated. Thanks in advance.

Thanks
Arunkumar Pichaimuthu, PMP

Re: Why authorization deny Read ACL doesn't work

[linbo liao]

Thanks Tom, I miss it.

I added authorizer.class.name=kafka.security.auth.SimpleAclAuthorizer, and
grant user:alice to operation permission to cluster, but still failed when
publish message. The error message told LEADER_NOT_AVAILABLE.

$ bin/kafka-acls.sh --authorizer-properties
> zookeeper.connect=localhost:2181 --list
> Current ACLs for resource `Topic:test`:
> User:alice has Allow permission for operations: Write from hosts:
> 127.0.0.1
> User:alice has Deny permission for operations: Read from hosts: *
>
> Current ACLs for resource `Cluster:kafka-cluster`:
> User:alice has Allow permission for operations: All from hosts: *
>
> $ bin/kafka-console-producer.sh --broker-list localhost:9092
> --producer.config config/producer.properties --topic test
> job
> [2017-06-12 11:14:41,318] WARN Error while fetching metadata with
> correlation id 1 : {test=LEADER_NOT_AVAILABLE}
> (org.apache.kafka.clients.NetworkClient)
>

> $ bin/kafka-topics.sh --zookeeper localhost:2181 --topic test --describe
> Topic:testPartitionCount:1ReplicationFactor:1Configs:
> Topic: testPartition: 0Leader: 0Replicas: 0Isr: 0
>



2017-06-12 16:43 GMT+08:00 Tom Bentley :

> Hi,
>
> Did you set
>
> authorizer.class.name=kafka.security.auth.SimpleAclAuthorizer
>
> as described here
>  at
> http://docs.confluent.io/current/kafka/authorization.
> html#further-configuration
>
> HTH,
>
> Tom
>
> On 11 June 2017 at 04:40, linbo liao  wrote:
>
> > Hi,
> >
> > I try to set Kafka ACL for topic access permission followed by kafka
> > security document  >,
> > but looks deny acl doesn't work.
> >
> > *My Environment:*
> >
> > VM: Ubuntu 12.04 LTS x86_64
> > JAVA:  openjdk version "1.8.0_111"
> > Kafka:  kafka_2.12-0.10.2.1
> >
> > I setup one broker, and use kafka-console-consumer.sh and
> > kafka-console-producer.sh to test.
> >
> > *Broker setup:*
> >
> > broker startup script already add jaas parameter
> >
> > $ cat kafka_server_jaas.conf
> >
> > > KafkaServer {
> > > org.apache.kafka.common.security.plain.PlainLoginModule
> > > required
> > > username="admin"
> > > password="admin"
> > > user_admin="admin"
> > > user_alice="alice";
> > > };
> > >
> >
> > config/server.properties
> >
> > listeners=SASL_PLAINTEXT://0.0.0.0:9092
> > > security.inter.broker.protocol=SASL_PLAINTEXT
> > > sasl.mechanism.inter.broker.protocol=PLAIN
> > > sasl.enabled.mechanisms=PLAIN
> > >
> >
> >
> > *Client setup:*
> > producer/consumer startup script already add jaas parameter
> >
> > $ cat client_jaas.conf
> >
> > > KafkaClient {
> > > org.apache.kafka.common.security.plain.PlainLoginModule required
> > > username="alice"
> > > password="alice";
> > > };
> >
> >
> > config/consumer.properties & config/producer.properties
> >
> > > security.protocol=SASL_PLAINTEXT
> > > sasl.mechanism=PLAIN
> > >
> >
> >
> > 1. create topic
> >
> > $ bin/kafka-topics.sh --create --zookeeper localhost:2181
> > > --replication-factor 1 --partitions 1 --topic test
> > >
> >
> > 2. setup topic acl
> >
> > $ bin/kafka-acls.sh --authorizer-properties
> > > zookeeper.connect=localhost:2181 --list --topic test
> > > Current ACLs for resource `Topic:test`:
> > > User:alice has Allow permission for operations: Write from
> hosts:
> > > 127.0.0.1
> > > User:alice has Deny permission for operations: Read from
> hosts: *
> > >
> >
> > Although I deny Read permission for user alice from all host, I start
> > consumer still can receive message.
> >
> > produce a message "test"
> >
> > > $ bin/kafka-console-producer.sh --broker-list localhost:9092
> > > --producer.config config/producer.properties --topic test
> > > test
> > >
> >
> > consumer receive this message
> >
> > $ bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic
> > > test --consumer.config config/consumer.properties --from-beginning
> > > [2017-06-11 03:37:55,998] WARN The configuration 'zookeeper.connect'
> was
> > > supplied but isn't a known config.
> > > (org.apache.kafka.clients.consumer.ConsumerConfig)
> > > [2017-06-11 03:37:55,999] WARN The configuration '
> > > zookeeper.connection.timeout.ms' was supplied but isn't a known
> config.
> > > (org.apache.kafka.clients.consumer.ConsumerConfig)
> > > test
> > >
> >
> > Why deny read operation doesn't work, do I miss something?
> >
> > Thanks,
> > Linbo
> >
>

Re: Why authorization deny Read ACL doesn't work

[Tom Bentley]

Hi,

Did you set

authorizer.class.name=kafka.security.auth.SimpleAclAuthorizer

as described here
 at
http://docs.confluent.io/current/kafka/authorization.html#further-configuration

HTH,

Tom

On 11 June 2017 at 04:40, linbo liao  wrote:

> Hi,
>
> I try to set Kafka ACL for topic access permission followed by kafka
> security document ,
> but looks deny acl doesn't work.
>
> *My Environment:*
>
> VM: Ubuntu 12.04 LTS x86_64
> JAVA:  openjdk version "1.8.0_111"
> Kafka:  kafka_2.12-0.10.2.1
>
> I setup one broker, and use kafka-console-consumer.sh and
> kafka-console-producer.sh to test.
>
> *Broker setup:*
>
> broker startup script already add jaas parameter
>
> $ cat kafka_server_jaas.conf
>
> > KafkaServer {
> > org.apache.kafka.common.security.plain.PlainLoginModule
> > required
> > username="admin"
> > password="admin"
> > user_admin="admin"
> > user_alice="alice";
> > };
> >
>
> config/server.properties
>
> listeners=SASL_PLAINTEXT://0.0.0.0:9092
> > security.inter.broker.protocol=SASL_PLAINTEXT
> > sasl.mechanism.inter.broker.protocol=PLAIN
> > sasl.enabled.mechanisms=PLAIN
> >
>
>
> *Client setup:*
> producer/consumer startup script already add jaas parameter
>
> $ cat client_jaas.conf
>
> > KafkaClient {
> > org.apache.kafka.common.security.plain.PlainLoginModule required
> > username="alice"
> > password="alice";
> > };
>
>
> config/consumer.properties & config/producer.properties
>
> > security.protocol=SASL_PLAINTEXT
> > sasl.mechanism=PLAIN
> >
>
>
> 1. create topic
>
> $ bin/kafka-topics.sh --create --zookeeper localhost:2181
> > --replication-factor 1 --partitions 1 --topic test
> >
>
> 2. setup topic acl
>
> $ bin/kafka-acls.sh --authorizer-properties
> > zookeeper.connect=localhost:2181 --list --topic test
> > Current ACLs for resource `Topic:test`:
> > User:alice has Allow permission for operations: Write from hosts:
> > 127.0.0.1
> > User:alice has Deny permission for operations: Read from hosts: *
> >
>
> Although I deny Read permission for user alice from all host, I start
> consumer still can receive message.
>
> produce a message "test"
>
> > $ bin/kafka-console-producer.sh --broker-list localhost:9092
> > --producer.config config/producer.properties --topic test
> > test
> >
>
> consumer receive this message
>
> $ bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic
> > test --consumer.config config/consumer.properties --from-beginning
> > [2017-06-11 03:37:55,998] WARN The configuration 'zookeeper.connect' was
> > supplied but isn't a known config.
> > (org.apache.kafka.clients.consumer.ConsumerConfig)
> > [2017-06-11 03:37:55,999] WARN The configuration '
> > zookeeper.connection.timeout.ms' was supplied but isn't a known config.
> > (org.apache.kafka.clients.consumer.ConsumerConfig)
> > test
> >
>
> Why deny read operation doesn't work, do I miss something?
>
> Thanks,
> Linbo
>