Re: Consumer group ACL limited to new consumer API?
I'd be happy to do that, but in this case it looks like the next release has it covered: https://www.elastic.co/blog/logstash-5-0-0-alpha1-released (See the Kafka 0.9 section) On 19 May 2016 at 10:50, Tom Crayford wrote: > You could always contribute back to logstash - I'm sure they'd appreciate > it. > > On Thu, May 19, 2016 at 3:47 PM, David Hawes wrote: > >> Thanks for the confirmation. >> >> I like the idea about only allowing authenticated customers >> (definitely what I want). Unfortunately, I'm running Kafka with an ELK >> installation and was hoping for some kind of stopgap while the >> logstash input plugins catch up and support TLS. When the logstash >> kafka plugin supports TLS, this sounds like a viable option. >> >> On 19 May 2016 at 08:55, Tom Crayford wrote: >> > Hi there, >> > >> > One way to disable the old consumer is to only allow authenticated >> > consumers (via SSL or another authentication system) - the old consumers >> > don't support authentication at all. If you care about ACLs anyway, you >> > probably don't want unauthenticated consumers or producers in the system >> at >> > all. >> > >> > The ACL for sure only works on the new consumer API, because the old one >> > talks directly to zookeeper so there's no good way to apply the same ACLs >> > there. >> > >> > Thanks >> > >> > Tom Crayford >> > Heroku Kafka >> > >> > On Thu, May 19, 2016 at 1:28 AM, David Hawes wrote: >> > >> >> I have been playing around with ACLs and was hoping to limit access to >> >> a topic and consumer group by IP, but was unable to get it working. >> >> Basically, I was able to Read from a topic as a consumer group that >> >> was not allowed. >> >> >> >> KIP-11 has the following line about consumer groups: >> >> >> >> In order to consume from a topic using the new consumer API, the >> >> principal will need: READ on TOPIC and READ on CONSUMER-GROUP. >> >> >> >> This tipped me off that the ACL may only work with the new consumer >> >> API, which I was not using. Sure enough, using the new consumer API >> >> denied my access by consumer group until I added an appropriate ACL. >> >> >> >> Is there some way to disable the old consumer API in Kafka? I see the >> >> inter.broker.protocol.version directive, but nothing about clients. >> >> Will there ever be support for group ACLs with the old consumer API? >> >> >> >> Without some way to disable the old consumer from being used, the >> >> consumer group ACLs are effectively useless as of version 0.9.0.1. >> >> >>
Re: Consumer group ACL limited to new consumer API?
You could always contribute back to logstash - I'm sure they'd appreciate it. On Thu, May 19, 2016 at 3:47 PM, David Hawes wrote: > Thanks for the confirmation. > > I like the idea about only allowing authenticated customers > (definitely what I want). Unfortunately, I'm running Kafka with an ELK > installation and was hoping for some kind of stopgap while the > logstash input plugins catch up and support TLS. When the logstash > kafka plugin supports TLS, this sounds like a viable option. > > On 19 May 2016 at 08:55, Tom Crayford wrote: > > Hi there, > > > > One way to disable the old consumer is to only allow authenticated > > consumers (via SSL or another authentication system) - the old consumers > > don't support authentication at all. If you care about ACLs anyway, you > > probably don't want unauthenticated consumers or producers in the system > at > > all. > > > > The ACL for sure only works on the new consumer API, because the old one > > talks directly to zookeeper so there's no good way to apply the same ACLs > > there. > > > > Thanks > > > > Tom Crayford > > Heroku Kafka > > > > On Thu, May 19, 2016 at 1:28 AM, David Hawes wrote: > > > >> I have been playing around with ACLs and was hoping to limit access to > >> a topic and consumer group by IP, but was unable to get it working. > >> Basically, I was able to Read from a topic as a consumer group that > >> was not allowed. > >> > >> KIP-11 has the following line about consumer groups: > >> > >> In order to consume from a topic using the new consumer API, the > >> principal will need: READ on TOPIC and READ on CONSUMER-GROUP. > >> > >> This tipped me off that the ACL may only work with the new consumer > >> API, which I was not using. Sure enough, using the new consumer API > >> denied my access by consumer group until I added an appropriate ACL. > >> > >> Is there some way to disable the old consumer API in Kafka? I see the > >> inter.broker.protocol.version directive, but nothing about clients. > >> Will there ever be support for group ACLs with the old consumer API? > >> > >> Without some way to disable the old consumer from being used, the > >> consumer group ACLs are effectively useless as of version 0.9.0.1. > >> >
Re: Consumer group ACL limited to new consumer API?
Thanks for the confirmation. I like the idea about only allowing authenticated customers (definitely what I want). Unfortunately, I'm running Kafka with an ELK installation and was hoping for some kind of stopgap while the logstash input plugins catch up and support TLS. When the logstash kafka plugin supports TLS, this sounds like a viable option. On 19 May 2016 at 08:55, Tom Crayford wrote: > Hi there, > > One way to disable the old consumer is to only allow authenticated > consumers (via SSL or another authentication system) - the old consumers > don't support authentication at all. If you care about ACLs anyway, you > probably don't want unauthenticated consumers or producers in the system at > all. > > The ACL for sure only works on the new consumer API, because the old one > talks directly to zookeeper so there's no good way to apply the same ACLs > there. > > Thanks > > Tom Crayford > Heroku Kafka > > On Thu, May 19, 2016 at 1:28 AM, David Hawes wrote: > >> I have been playing around with ACLs and was hoping to limit access to >> a topic and consumer group by IP, but was unable to get it working. >> Basically, I was able to Read from a topic as a consumer group that >> was not allowed. >> >> KIP-11 has the following line about consumer groups: >> >> In order to consume from a topic using the new consumer API, the >> principal will need: READ on TOPIC and READ on CONSUMER-GROUP. >> >> This tipped me off that the ACL may only work with the new consumer >> API, which I was not using. Sure enough, using the new consumer API >> denied my access by consumer group until I added an appropriate ACL. >> >> Is there some way to disable the old consumer API in Kafka? I see the >> inter.broker.protocol.version directive, but nothing about clients. >> Will there ever be support for group ACLs with the old consumer API? >> >> Without some way to disable the old consumer from being used, the >> consumer group ACLs are effectively useless as of version 0.9.0.1. >>
Re: Consumer group ACL limited to new consumer API?
Hi there, One way to disable the old consumer is to only allow authenticated consumers (via SSL or another authentication system) - the old consumers don't support authentication at all. If you care about ACLs anyway, you probably don't want unauthenticated consumers or producers in the system at all. The ACL for sure only works on the new consumer API, because the old one talks directly to zookeeper so there's no good way to apply the same ACLs there. Thanks Tom Crayford Heroku Kafka On Thu, May 19, 2016 at 1:28 AM, David Hawes wrote: > I have been playing around with ACLs and was hoping to limit access to > a topic and consumer group by IP, but was unable to get it working. > Basically, I was able to Read from a topic as a consumer group that > was not allowed. > > KIP-11 has the following line about consumer groups: > > In order to consume from a topic using the new consumer API, the > principal will need: READ on TOPIC and READ on CONSUMER-GROUP. > > This tipped me off that the ACL may only work with the new consumer > API, which I was not using. Sure enough, using the new consumer API > denied my access by consumer group until I added an appropriate ACL. > > Is there some way to disable the old consumer API in Kafka? I see the > inter.broker.protocol.version directive, but nothing about clients. > Will there ever be support for group ACLs with the old consumer API? > > Without some way to disable the old consumer from being used, the > consumer group ACLs are effectively useless as of version 0.9.0.1. >
Consumer group ACL limited to new consumer API?
I have been playing around with ACLs and was hoping to limit access to a topic and consumer group by IP, but was unable to get it working. Basically, I was able to Read from a topic as a consumer group that was not allowed. KIP-11 has the following line about consumer groups: In order to consume from a topic using the new consumer API, the principal will need: READ on TOPIC and READ on CONSUMER-GROUP. This tipped me off that the ACL may only work with the new consumer API, which I was not using. Sure enough, using the new consumer API denied my access by consumer group until I added an appropriate ACL. Is there some way to disable the old consumer API in Kafka? I see the inter.broker.protocol.version directive, but nothing about clients. Will there ever be support for group ACLs with the old consumer API? Without some way to disable the old consumer from being used, the consumer group ACLs are effectively useless as of version 0.9.0.1.
