(KIP-368) SASL mechanism GSSAPI authentication failure
Hello, I have an issue using using SASL with GSSAPI and kerberos. When I lose a node and a new one is built by our automation all of the kafka nodes continue trying to connect to the replacement node with the old nodes kerberos details which fails. I was expecting KIP-368: "Allow SASL Connections to Periodically Re-Authenticate" to resolve this issue by allowing nodes to clear out their saved credentials and reconnect with new credentials which would fix the problem. After testing kafka 2.2.0 with connections.max.reauth.ms=30 set on the nodes I am still having the same problem. It doesn't look like kafka is clearing out saved credentials and fully going through the reconnect process to get new credentials for the node. Does anyone know if I am missing something or if this is still an issue with kafka? Tyler Monahan
Clearing saved kerberos credentials on login failure
Hello,
I am having an issue with kerberos auth when one of my brokers is lost and
is replaced by a new instance the running brokers/consumers/producers are
still trying to use the authentication information for the original broker
to login to the new broker. This leaves me in a state of constantly failing
to authenticate until all brokers/consumers/producers are restarted and
then the load in the new broker information and everything is fine.
I am wondering if there is a way to force kafka brokers/consumers/producers
to expire the credentials they save for brokers after a period of time so
they can connect to the new instance. If there is another better way to fix
that this that would be great as well.
I have tried setting expiration times in /etc/krb5.conf however it doesn't
seem like kafka is honoring them. I have also tried different jaas.conf but
they don't seem to work. I haven't gotten one without storeKey=true to work
which would in theory solve my problem. This is my current jaas config.
KafkaServer {
com.sun.security.auth.module.Krb5LoginModule required
useKeyTab=true
storeKey=true
keyTab="/etc/krb5.keytab"
principal="kafka2$";
};
KafkaClient {
com.sun.security.auth.module.Krb5LoginModule required
useKeyTab=true
storeKey=true
keyTab="/etc/krb5.keytab"
serviceName="kafka"
principal="kafka2$";
};
Tyler Monahan
Re: 90 open file handles for one segment
Matt, I am seeing similar behavior with kafka 1.1.0 with 80 copies of the same file being open. I am getting up to ~500,000 open files on each broker with it dying around 600,000. I have yet to find any documentation saying how many open files kafka should have per topic/partition. The only documentation I have seen is having the limit set to at least 100,000 which we are both well past. [2018-09-07 14:35:33,872] ERROR Error while accepting connection (kafka.network.Acceptor) java.io.IOException: Too many open files Tyler Monahan On 8/27/18, 12:29 PM, "Matt Kocubinski" wrote: A 5 node cluster is running Kafka 1.0.0 with 126 topics, most of them with 24 partitions. Brokers are keeping around 800k file handles open each, and regularly crashing with "Too many open file handles". Using one broker as a study I discovered There are ~13k log file segments/indices in the Kafka data dir discovered like: admin@kafka-1:/kafka/logs$ find . -type f | wc -l 12368 I've also noticed that a single segment file is open in 90 different threads: admin@kafka-1:/kafka/logs$ sudo lsof | grep "/site.time.tracking-6/0297.log" | wc -l 90 This is about 1 per broker thread running! Extrapolating this nearly adds up to 1.3mm, or the 800k I'm seeing. Something must be grossly mis-configured here. Why are the same segments being opened so many times?
Re: Query related to Kafka Consumer Limit
Satarupa, In my experience kafka has a 10k partition limit per topic. I don't think you are going to be able to get 1 million partitions to work on a single topic. A consumer would need to subscribe to listen to a kafka topic. You will probably need to have multiple consumer groups or multiple topics to handle that many consumers. I would recommend doing some research on other very large kafka setups. Consumers also talk directly to kafka in newer versions and shouldn't be talking to zookeeper unless you are using a very old version of kafka. Tyler Monahan
Re: Configuring Kerberos behind an ELB
Martin, I think I tried that already. I setup a user in ad and assigned the shared SPN record for the ELB to that user. I then added the user to the keytab file for the kafka servers and had the kafka servers use the SPN for the user with the dns record. That worked fine for authing through the ELB but it broke inter broker communication. Since kafka talks to the other nodes directly and doesn't go through the elb it would fail on the direct connection as it doesn't have valid credentials as mentioned. I don't know a solution to the interbroker communication unless there is some way I can have it use different information for inter broker then it does for incoming consumer/producer connections. Tyler Monahan On Fri, Jun 22, 2018 at 10:27 AM, Martin Gainty wrote: > it appears you want: > common-principal name with common-key distributed to all subdomains > > > Use only one common Service Principal Name: > > One of the solutions is to create a new Service Principal in the > KDC for the name HTTP/all.ipa@ipa.dom > then generate a keytab and > distribute it (keytab) to all servers. > The servers will use no other key, and they will identify > themselves with the common name, > so if a client tries to contact them using their individual > name, then authentication will fail, > as the KDC will not have a principal for the other names > and the services themselves are not configure to use their hostname only > the common name. > > assuming you generated common SPN in the KDC > assuming you generated keytab and distributed to all subdomains > does this not work for you? > M- > > > > > -- > *From:* Tyler Monahan > *Sent:* Friday, June 22, 2018 1:09 PM > *To:* Ben Wood > *Cc:* users@kafka.apache.org; Martin Gainty > *Subject:* Re: Configuring Kerberos behind an ELB > > Ben, > > Yes. I want to be able to provide consumers/producers with a single > address they can use to connect to the cluster. Having it behind an elb > lets us scale up and replace nodes with out needing to mess with > consumer/producer configurations. I have considered setting up individual > dns records for each broker and feeding in a list of instances to connect > to but this is not as flexible as using an elb and does not match our > general strategy for infrastructure. If at all possible I would like to get > kafka working behind and elb with kerberos. > > Tyler Monahan > > On Fri, Jun 22, 2018 at 9:44 AM, Ben Wood wrote: > > Hey Tyler, > > What is your end goal? To have a single publicly / internally available > address to be able to provide to consumers / producers to connect to the > Kerberized Kafka? > > On Fri, Jun 22, 2018 at 9:20 AM, Tyler Monahan > wrote: > > Martin, > > I have read that stack overflow post but it doesn't help with my specific > problem. An ELB will work if I am not using kerberos just fine. The issue > started happening when I added kerberos auth to the cluster. The auth has > to happen before the meta data request so it never gets to the point where > it is by passing the load balancer. Because I am connecting with the load > balancers dns record I don't have a valid spn on the brokers for the load > balancers dns record. This blog post has some work arounds for kerberos > with a load balancer and details the problem but I haven't been able to get > any of them to work with kafka because it gets traffic through and ELB but > also talks to the other brokers directly in my setup. > https://ssimo.org/blog/id_019.html > > Tyler Monahan > > On Fri, Jun 22, 2018 at 5:36 AM, Martin Gainty > wrote: > > > MG>quoting stackoverflow below > > > > "You can use an ELB as the bootstrap.servers, > > *The ELB will be used for the initial metadata request the client makes > to > > figure out which topic partitions are on which brokers, * > > but after (the initial metadata request) > > the > > *brokers still need to be directly accessible to the client. *that it'll > > use the hostname of the server (or advertised.listeners setting if you > > need to customize it, > > which, e.g. might be necessary on EC2 instances to get the public IP of a > > server). > > If you were trying to use an ELB to make a Kafka cluster publicly > > available, > > you'd need to make sure the advertised.listeners for each broker also > > makes it publicly accessible. " > > > > MG> initial metadata request you will see elb > > MG>after metadata request and topic partition locations are determined > > MG>elb drops out and client will talk to directly to broker > > MG>use healthcheck algorithm to determine assigned server/port assigned >
Re: Configuring Kerberos behind an ELB
Ben, Yes. I want to be able to provide consumers/producers with a single address they can use to connect to the cluster. Having it behind an elb lets us scale up and replace nodes with out needing to mess with consumer/producer configurations. I have considered setting up individual dns records for each broker and feeding in a list of instances to connect to but this is not as flexible as using an elb and does not match our general strategy for infrastructure. If at all possible I would like to get kafka working behind and elb with kerberos. Tyler Monahan On Fri, Jun 22, 2018 at 9:44 AM, Ben Wood wrote: > Hey Tyler, > > What is your end goal? To have a single publicly / internally available > address to be able to provide to consumers / producers to connect to the > Kerberized Kafka? > > On Fri, Jun 22, 2018 at 9:20 AM, Tyler Monahan > wrote: > >> Martin, >> >> I have read that stack overflow post but it doesn't help with my specific >> problem. An ELB will work if I am not using kerberos just fine. The issue >> started happening when I added kerberos auth to the cluster. The auth has >> to happen before the meta data request so it never gets to the point where >> it is by passing the load balancer. Because I am connecting with the load >> balancers dns record I don't have a valid spn on the brokers for the load >> balancers dns record. This blog post has some work arounds for kerberos >> with a load balancer and details the problem but I haven't been able to >> get >> any of them to work with kafka because it gets traffic through and ELB but >> also talks to the other brokers directly in my setup. >> https://ssimo.org/blog/id_019.html >> >> Tyler Monahan >> >> On Fri, Jun 22, 2018 at 5:36 AM, Martin Gainty >> wrote: >> >> > MG>quoting stackoverflow below >> > >> > "You can use an ELB as the bootstrap.servers, >> > *The ELB will be used for the initial metadata request the client makes >> to >> > figure out which topic partitions are on which brokers, * >> > but after (the initial metadata request) >> > the >> > *brokers still need to be directly accessible to the client. *that it'll >> > use the hostname of the server (or advertised.listeners setting if you >> > need to customize it, >> > which, e.g. might be necessary on EC2 instances to get the public IP of >> a >> > server). >> > If you were trying to use an ELB to make a Kafka cluster publicly >> > available, >> > you'd need to make sure the advertised.listeners for each broker also >> > makes it publicly accessible. " >> > >> > MG> initial metadata request you will see elb >> > MG>after metadata request and topic partition locations are determined >> > MG>elb drops out and client will talk to directly to broker >> > MG>use healthcheck algorithm to determine assigned server/port assigned >> to >> > broker from /brokers/id/$id >> > MG>echo dump | nc localhost 2181 | grep brokers >> > >> > >> > https://stackoverflow.com/questions/38666795/does-kafka- >> > support-elb-in-front-of-broker-cluster >> > >> > <https://stackoverflow.com/questions/38666795/does-kafka-sup >> port-elb-in-front-of-broker-cluster> >> > Does Kafka support ELB in front of broker cluster? - Stack ... >> > <https://stackoverflow.com/questions/38666795/does-kafka-sup >> port-elb-in-front-of-broker-cluster> >> > stackoverflow.com >> > I had a question regarding Kafka broker clusters on AWS. Right now there >> > is an AWS ELB sitting in front of the cluster, but when I set the >> > "bootstrap.servers" property of my producer or consumer to... >> > >> > does this help? >> > >> > Martin >> > __ >> > >> > >> > >> > -- >> > *From:* Tyler Monahan >> > *Sent:* Thursday, June 21, 2018 6:17 PM >> > *To:* users@kafka.apache.org >> > *Subject:* Configuring Kerberos behind an ELB >> > >> > Hello, >> > >> > I have setup kafka using kerberos successfully however if I try and >> reach >> > kafka through an elb the kerberos authentication fails. The kafka >> brokers >> > are each using their unique hostname for kerberos and when going >> through an >> > elb the consumer/producer only sees the elb's dns record which doesn't >> have >> > kerberos setup for it causing auth to fail. If
Re: Configuring Kerberos behind an ELB
Martin, I have read that stack overflow post but it doesn't help with my specific problem. An ELB will work if I am not using kerberos just fine. The issue started happening when I added kerberos auth to the cluster. The auth has to happen before the meta data request so it never gets to the point where it is by passing the load balancer. Because I am connecting with the load balancers dns record I don't have a valid spn on the brokers for the load balancers dns record. This blog post has some work arounds for kerberos with a load balancer and details the problem but I haven't been able to get any of them to work with kafka because it gets traffic through and ELB but also talks to the other brokers directly in my setup. https://ssimo.org/blog/id_019.html Tyler Monahan On Fri, Jun 22, 2018 at 5:36 AM, Martin Gainty wrote: > MG>quoting stackoverflow below > > "You can use an ELB as the bootstrap.servers, > *The ELB will be used for the initial metadata request the client makes to > figure out which topic partitions are on which brokers, * > but after (the initial metadata request) > the > *brokers still need to be directly accessible to the client. *that it'll > use the hostname of the server (or advertised.listeners setting if you > need to customize it, > which, e.g. might be necessary on EC2 instances to get the public IP of a > server). > If you were trying to use an ELB to make a Kafka cluster publicly > available, > you'd need to make sure the advertised.listeners for each broker also > makes it publicly accessible. " > > MG> initial metadata request you will see elb > MG>after metadata request and topic partition locations are determined > MG>elb drops out and client will talk to directly to broker > MG>use healthcheck algorithm to determine assigned server/port assigned to > broker from /brokers/id/$id > MG>echo dump | nc localhost 2181 | grep brokers > > > https://stackoverflow.com/questions/38666795/does-kafka- > support-elb-in-front-of-broker-cluster > > <https://stackoverflow.com/questions/38666795/does-kafka-support-elb-in-front-of-broker-cluster> > Does Kafka support ELB in front of broker cluster? - Stack ... > <https://stackoverflow.com/questions/38666795/does-kafka-support-elb-in-front-of-broker-cluster> > stackoverflow.com > I had a question regarding Kafka broker clusters on AWS. Right now there > is an AWS ELB sitting in front of the cluster, but when I set the > "bootstrap.servers" property of my producer or consumer to... > > does this help? > > Martin > __ > > > > -- > *From:* Tyler Monahan > *Sent:* Thursday, June 21, 2018 6:17 PM > *To:* users@kafka.apache.org > *Subject:* Configuring Kerberos behind an ELB > > Hello, > > I have setup kafka using kerberos successfully however if I try and reach > kafka through an elb the kerberos authentication fails. The kafka brokers > are each using their unique hostname for kerberos and when going through an > elb the consumer/producer only sees the elb's dns record which doesn't have > kerberos setup for it causing auth to fail. If I try to setup a service > principle name for that dns record I can only associate it with one of the > brokers behind the elb so the other ones fail. > > I have tried setting up a service account and having the kafka brokers use > that which works when a consumer/producer is talking to the instances > through the elb however inter broker communication which is also over > kerberos fails at that point because it is going directly to the other > nodes instead of through the elb. I am not sure where to go from here as > there doesn't appear to be a way to configure the inter broker > communication to work differently then the incoming consumer communication > short of getting rid of kerberos. > > Any advice would be greatly appreciated. > > Tyler Monahan >
Configuring Kerberos behind an ELB
Hello, I have setup kafka using kerberos successfully however if I try and reach kafka through an elb the kerberos authentication fails. The kafka brokers are each using their unique hostname for kerberos and when going through an elb the consumer/producer only sees the elb's dns record which doesn't have kerberos setup for it causing auth to fail. If I try to setup a service principle name for that dns record I can only associate it with one of the brokers behind the elb so the other ones fail. I have tried setting up a service account and having the kafka brokers use that which works when a consumer/producer is talking to the instances through the elb however inter broker communication which is also over kerberos fails at that point because it is going directly to the other nodes instead of through the elb. I am not sure where to go from here as there doesn't appear to be a way to configure the inter broker communication to work differently then the incoming consumer communication short of getting rid of kerberos. Any advice would be greatly appreciated. Tyler Monahan
