Hi Alessandro, Thanks for the help. It looks like the issue is on our side: KDC hasn’t been properly setup for Zookeeper: required principals don’t exist.
I just wonder why the error message cannot be more descriptive and if we could improve it by properly logging the original exception. Andor > On 2019. Oct 29., at 14:35, Alessandro Luccaroni - Diennea > <[email protected]> wrote: > > Hi Andor, > Enrico's collegue here. > > If I remember correctly the issue in our case was related to the > ticket_lifetime and renew_lifetime options. > These two krb.conf options didn't matter before Java 9 (see > https://bugs.openjdk.java.net/browse/JDK-8044500 and > https://bugs.openjdk.java.net/browse/JDK-8131051) and, as soon as we updated > the JDK version, we started to see weird issue related to the ticket > expiration. We simply decided to remove the option from the krb.conf and use > the Kerberos default. > > With JDK8/Unlimited Strength the problem was related with the enctype: I see > that you fixed it on the krb.conf by adding the option to the client, we > instead changed the option at the krb level so to ensure that the keytab > generated were compatible (supported_enctypes option). I guess this is less > of a problem with modern JDKs. > > Regards, > Alessandro Luccaroni > Platform Manager @ Diennea - MagNews > Tel.: (+39) 0546 066100 Int. 924 > Viale G.Marconi 30/14 - 48018 Faenza (RA) - Italy > >> -----Messaggio originale----- >> Da: Enrico Olivelli <[email protected]> >> Inviato: martedì 29 ottobre 2019 14:23 >> A: UserZooKeeper <[email protected]> >> Oggetto: Re: Kerberos login error: Message stream modified (41) >> >> Andor >> did you try with a smaller file ? >> >> Enrico >> >> Il giorno mar 29 ott 2019 alle ore 11:09 Enrico Olivelli - Diennea < >> [email protected]> ha scritto: >> >>> I would try to shrink the file to the minimum and add one line at a time. >>> >>> With JDK8 we also had problems with Unlimited Strength policy stuff >>> >>> Hope that helps >>> >>> Enrico Olivelli >>> MagNews Platform Development Manager @ Diennea – MagNews >>> Tel.: (+39) 0546 066100 - Int. 125 >>> Viale G.Marconi 30/14 - 48018 Faenza (RA) >>> >>> >>> www.diennea.com/en < >>> >> https://www.diennea.com/en?utm_source=Firma&utm_medium=Web&ut >> m_campaig >>> n=Firma_Outlook> >>> | www.magnews.com < >>> >> https://www.magnews.com/?utm_source=Firma&utm_medium=Web&utm >> _campaign= >>> Firma_Outlook >>>> >>> < >>> https://www.linkedin.com/company/diennea--- >> magnews/?utm_source=Firma&u >>> tm_medium=Web&utm_campaign=Firma_Outlook >>>> >>> < >>> >> https://twitter.com/DienneaMagNews?utm_source=Firma&utm_medium= >> Web&utm >>> _campaign=Firma_Outlook >>>> >>> < >>> >> https://www.facebook.com/DienneaMagNews/?utm_source=Firma&utm_ >> medium=W >>> eb&utm_campaign=Firma_Outlook >>>> >>> >>> >>> >>> Il giorno 29/10/19, 10:55 "Andor Molnar" <[email protected]> ha scritto: >>> >>> Thanks Enrico for the quick help. >>> >>> Here’s my krb5.conf: >>> >>> [libdefaults] >>> default_realm = STREAMANALYTICS >>> dns_lookup_kdc = false >>> dns_lookup_realm = false >>> ticket_lifetime = 86400 >>> renew_lifetime = 604800 >>> forwardable = true >>> default_tgs_enctypes = aes256-cts aes128-cts des3-hmac-sha1 >>> arcfour-hmac des3-hmac-sha1 des-cbc-md5 >>> default_tkt_enctypes = aes256-cts aes128-cts des3-hmac-sha1 >>> arcfour-hmac des3-hmac-sha1 des-cbc-md5 >>> permitted_enctypes = aes256-cts aes128-cts des3-hmac-sha1 >>> arcfour-hmac >>> des3-hmac-sha1 des-cbc-md5 >>> udp_preference_limit = 1 >>> kdc_timeout = 3000 >>> [realms] >>> STREAMANALYTICS = { >>> kdc = ldap0.mydomain.com >>> admin_server = ldap0.mydomain.com >>> } >>> [domain_realm] >>> >>> ;———— >>> >>> I wonder if the default encryption type settings could be the problem. >>> I need to verify if it works with Java 8, because it might be a Java >>> 11 or ZK 3.5 thing. Or both. >>> >>> Andor >>> >>> >>> >>> >>> >>>> On 2019. Oct 29., at 8:42, Enrico Olivelli - Diennea < >>> [email protected]> wrote: >>>> >>>> Andor, >>>> this is a minimal krb5.conf file that is working from jdk8 to >>> jdk13 and ZooKeeper >>>> >>>> maybe you can compare to your one and start dropping >>> configuration lines that are not needed. >>>> >>>> Java is adding more and more capabilities to GSSAPI support and >>> this sometimes leads to behavior changes >>>> >>>> >>>> [libdefaults] >>>> default_realm = MYDOMAIN >>>> >>>> [realms] >>>> MYDOMAIN = { >>>> kdc = kerberos1.mydomain.com >>>> kdc = kerberos2. mydomain.com >>>> kdc = kerberos3. mydomain.com >>>> } >>>> >>>> >>>> >>>> Enrico Olivelli >>>> MagNews Platform Development Manager @ Diennea – MagNews >>>> Tel.: (+39) 0546 066100 - Int. 125 >>>> Viale G.Marconi 30/14 - 48018 Faenza (RA) >>>> >>>> >>>> >>>> Il giorno 28/10/19, 17:56 "Enrico Olivelli" >>> <[email protected]> ha scritto: >>>> >>>> Andor >>>> >>>> Il lun 28 ott 2019, 17:44 Andor Molnar <[email protected]> ha >>> scritto: >>>> >>>>> Hi, >>>>> >>>>> I’m facing the following error message when trying to run >>> ZooKeeper >>> 3.5.5 >>>>> on Java 11 with Kerberos authentication: >>>>> >>>>> 2019-10-28 16:30:04,811 INFO >>>>> org.apache.zookeeper.server.ServerCnxnFactory: Using >>>>> org.apache.zookeeper.server.NIOServerCnxnFactory as server >>> connection >>>>> factory >>>>> 2019-10-28 16:30:04,823 INFO >> org.apache.zookeeper.common.X509Util: >>> Setting >>>>> -D jdk.tls.rejectClientInitiatedRenegotiation=true to disable >>>>> client-initiated TLS renegotiation >>>>> 2019-10-28 16:30:05,012 ERROR >>>>> org.apache.zookeeper.server.quorum.QuorumPeerMain: >> Unexpected >>> exception, >>>>> exiting abnormally >>>>> java.io.IOException: Could not configure server because SASL >>> configuration >>>>> did not allow the ZooKeeper server to authenticate itself properly: >>>>> javax.security.auth.login.LoginException: Message stream >>> modified >>> (41) >>>>> at >>>>> >>> >> org.apache.zookeeper.server.ServerCnxnFactory.configureSaslLogin(Server >> CnxnFactory.java:243) >>>>> at >>>>> >>> >> org.apache.zookeeper.server.NIOServerCnxnFactory.configure(NIOServerC >> nxnFactory.java:646) >>>>> at >>>>> >>> >> org.apache.zookeeper.server.quorum.QuorumPeerMain.runFromConfig(Qu >> orumPeerMain.java:148) >>>>> at >>>>> >>> >> org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(Q >> uorumPeerMain.java:123) >>>>> at >>>>> >>> >> org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeer >> Main.java:82) >>>>> … >>>>> >>>>> zoo.cfg: >>>>> ———— >>>>> tickTime=2000 >>>>> initLimit=10 >>>>> syncLimit=5 >>>>> >>>>> >>> >> 4lw.commands.whitelist=conf,cons,crst,dirs,dump,envi,gtmk,ruok,stmk,srst, >> srvr,stat,wchs,mntr,isro >>>>> dataDir=/var/lib/zookeeper >>>>> dataLogDir=/var/lib/zookeeper >>>>> clientPort=2181 >>>>> maxClientCnxns=60 >>>>> minSessionTimeout=4000 >>>>> maxSessionTimeout=60000 >>>>> autopurge.purgeInterval=24 >>>>> autopurge.snapRetainCount=5 >>>>> quorum.auth.enableSasl=true >>>>> quorum.cnxn.threads.size=20 >>>>> admin.enableServer=false >>>>> admin.serverPort=5181 >>>>> server.1=cdf1-dc1.mydomain.com:3181:4181 >>>>> server.2=cdf1-dc2.mydomain.com:3181:4181 >>>>> server.3=cdf1-dc3.mydomain.com:3181:4181 >>>>> leaderServes=yes >>>>> >>> >> authProvider.1=org.apache.zookeeper.server.auth.SASLAuthenticationProvi >> der >>>>> kerberos.removeHostFromPrincipal=true >>>>> kerberos.removeRealmFromPrincipal=true >>>>> quorum.auth.kerberos.servicePrincipal=zookeeper/_HOST >>>>> quorum.auth.learnerRequireSasl=true >>>>> quorum.auth.serverRequireSasl=true >>>>> >>>>> java -version: >>>>> —————— >>>>> openjdk version "11.0.4" 2019-07-16 >>>>> OpenJDK Runtime Environment AdoptOpenJDK (build 11.0.4+11) >>>>> OpenJDK 64-Bit Server VM AdoptOpenJDK (build 11.0.4+11, mixed >> mode) >>>>> >>>>> >>>>> Has anyone seen this problem before? >>>>> What does the error message mean? >>>>> >>>>> Unfortunately we swallow the original exception in >>> ServerCnxnFactory and >>>>> only log the message without stacktrace. >>>>> >>>> >>>> Did you enable debug? >>>> >>> https://stackoverflow.com/questions/15382056/enable-detailed-logging- >> for-kerberos-in-java >>>> >>>> I remember we had some issue while switching from jdk8 to jdk9 >>>> >>>> There were something in krb.conf that was not compatible due to >>> some >>>> stricter condig check but we didn't need that line and we dropped >>> it. >>>> I can check only tomorrow at work. >>>> Unfortunately java Kerberos client is not so verbose. >>>> >>>> Can you share your krb config files? Without hostnames >>>> >>>> Enrico >>>> >>>> >>>>> Thanks, >>>>> Andor >>>>> >>>>> >>>>> >>>> >>>> >>>> >>>> ________________________________ >>>> >>>> CONFIDENTIALITY & PRIVACY NOTICE >>>> This e-mail (including any attachments) is strictly confidential >>> and may also contain privileged information. If you are not the >>> intended recipient you are not authorised to read, print, save, >>> process or disclose this message. If you have received this message by >>> mistake, please inform the sender immediately and destroy this e-mail, >>> its attachments and any copies. Any use, distribution, reproduction or >>> disclosure by any person other than the intended recipient is strictly >>> prohibited and the person responsible may incur in penalties. >>>> The use of this e-mail is only for professional purposes; there >>> is no guarantee that the correspondence towards this e-mail will be >>> read only by the recipient, because, under certain circumstances, >>> there may be a need to access this email by third subjects belonging to the >> Company. >>> >>> >>> >>> >>> ________________________________ >>> >>> CONFIDENTIALITY & PRIVACY NOTICE >>> This e-mail (including any attachments) is strictly confidential and >>> may also contain privileged information. If you are not the intended >>> recipient you are not authorised to read, print, save, process or >>> disclose this message. If you have received this message by mistake, >>> please inform the sender immediately and destroy this e-mail, its >> attachments and any copies. >>> Any use, distribution, reproduction or disclosure by any person other >>> than the intended recipient is strictly prohibited and the person >>> responsible may incur in penalties. >>> The use of this e-mail is only for professional purposes; there is no >>> guarantee that the correspondence towards this e-mail will be read >>> only by the recipient, because, under certain circumstances, there may >>> be a need to access this email by third subjects belonging to the Company. >>> > > ________________________________ > > CONFIDENTIALITY & PRIVACY NOTICE > This e-mail (including any attachments) is strictly confidential and may also > contain privileged information. If you are not the intended recipient you are > not authorised to read, print, save, process or disclose this message. If you > have received this message by mistake, please inform the sender immediately > and destroy this e-mail, its attachments and any copies. Any use, > distribution, reproduction or disclosure by any person other than the > intended recipient is strictly prohibited and the person responsible may > incur in penalties. > The use of this e-mail is only for professional purposes; there is no > guarantee that the correspondence towards this e-mail will be read only by > the recipient, because, under certain circumstances, there may be a need to > access this email by third subjects belonging to the Company.
