Excellent, thanks for spotting that Damien! Chris Nauroth
On Tue, Apr 22, 2025 at 2:07 AM Damien Diederen <[email protected]> wrote: > > Hi Pearson, Chris, all, > > Here is a snippet extracted from Pearson's original report: > > > # Reference: https://issues.apache.org/jira/browse/ZOOKEEPER-3743 > > Server { > > org.apache.zookeeper.server.auth.DigestLoginModule required > > user_zookeeper="test"; # Or use a different user/password as needed > > }; > > It turns out that the first three characters of the indentation as well > as the one following the semicolon are *not* (normal ASCII) SPACEs, but > NO-BREAK SPACEs: > > >> character: (displayed as ) (codepoint 160, #o240, #xa0) > >> name: NO-BREAK SPACE > > I suspect these were present as-is in the JAAS configuration file, as > they can trigger the "No JAAS configuration section" error: > > >> java.io.IOException: No JAAS configuration section named 'Server' was > found in '[...]'. > > This is the actual exception, which is now not swallowed thanks to > Chris' fix: > > >> java.lang.SecurityException: java.io.IOException: Konfigurationsfehler: > >> Zeile 3: [=] erwartet, [user_super] gefunden > >> at > java.base/sun.security.provider.ConfigFile$Spi.<init>(ConfigFile.java:137) > > HTH, -D > > > > -------- > > Chris Nauroth <[email protected]> writes: > > I just realized that I hadn't included the user@ list in my latest > reply. > > Including it now. > > > > The proposed patch now has one +1 from a committer, and I'll wait for > > another to chime in. > > > > Chris Nauroth > > > > > > On Thu, Mar 13, 2025 at 9:01 PM Chris Nauroth <[email protected]> > wrote: > > > >> Hi again, > >> > >> Sorry that we haven't been able to find a solution yet. I think we're > >> being slowed down by a limitation in the ZooKeeper code: it doesn't log > the > >> full details of the error when there is a problem loading the JAAS > >> configuration file. I filed a small patch to improve this: > >> > >> https://issues.apache.org/jira/browse/ZOOKEEPER-4906 > >> > >> https://github.com/apache/zookeeper/pull/2234 > >> > >> My only other thought is that if you have the option to apply this patch > >> and make a custom build, you'll probably get more details from the > >> additional logging. > >> > >> Chris Nauroth > >> > >> > >> On Wed, Mar 12, 2025 at 10:20 PM Pearson <[email protected]> > >> wrote: > >> > >>> Thank you for the email Chris. Yes, I can cat the file as the > >>> "zookeeper" user. Unfortunately, I tried without the comments and it > >>> still doesn't work. I'm wondering what I can do next. > >>> > >>> Thanks > >>> > >>> On 10/03/2025 16:01, Chris Nauroth wrote: > >>> > Hello, > >>> > > >>> > At a high level, the setup seems right to me. I believe this error > can > >>> > happen if the JAAS configuration file is not found or otherwise > >>> > inaccessible, so it's good that you took care of chown'ing to the > >>> zookeeper > >>> > user. Just to make sure, does "sudo -u zookeeper cat > >>> > /opt/zookeeper/conf/jaas_server.conf" succeed? I'm wondering if there > >>> is a > >>> > permission problem on an intermediate directory. > >>> > > >>> > My only other thought is a potential syntax violation in the JAAS > >>> > configuration file itself. For example, I don't know the syntax for > >>> > comments in these files, so I don't know if the '#' in-line comment > is > >>> > problematic. > >>> > > >>> > Chris Nauroth > >>> > > >>> > > >>> > On Sun, Mar 9, 2025 at 5:02 PM Pearson <[email protected] > > > >>> wrote: > >>> > > >>> >> I should mention that I'm using zookeeper version 3.9.3 > >>> >> > >>> >> On 09/03/2025 23:57, Pearson wrote: > >>> >>> Hello, > >>> >>> > >>> >>> I'm very new to Zookeeper, and even after reading a bunch of > threads > >>> >>> and forum posts I am unable to solve this problem. Would really > >>> >>> appreciate your help. > >>> >>> > >>> >>> Let me begin with my zoo.cfg: > >>> >>> > >>> >>> |cat /opt/zookeeper/conf/zoo.cgf| > >>> >>> > >>> >>> |dataDir=/var/lib/zookeeper > >>> >>> dataLogDir=/var/log/zookeeper > >>> >>> > >>> >>> clientPort=2181 > >>> >>> > >>> >>> initLimit=5 > >>> >>> syncLimit=2 > >>> >>> > >>> >>> # Configure SASL authentication from Clients > >>> >>> > >>> >> > >>> > authProvider.1=org.apache.zookeeper.server.auth.SASLAuthenticationProvider > >>> >>> | > >>> >>> > >>> >>> |clientCnxnSocket=org.apache.zookeeper.ClientCnxnSocketNetty > >>> >>> > serverCnxnFactory=org.apache.zookeeper.server.NettyServerCnxnFactory| > >>> >>> > >>> >>> |# Configure SASL for Quorun > >>> >>> # Reference: > >>> >>> > >>> >> > >>> > https://cwiki.apache.org/confluence/display/ZOOKEEPER/Server-Server+mutual+authentication > >>> >>> quorum.auth.enableSasl=true > >>> >>> quorum.auth.learnerRequireSasl=true > >>> >>> quorum.auth.serverRequireSasl=true > >>> >>> quorum.auth.learner.saslLoginContext=QuorumLearner > >>> >>> quorum.auth.server.saslLoginContext=QuorumServer > >>> >>> # quorum.auth.kerberos.servicePrincipal=servicename/_HOST > >>> >>> quorum.cnxn.threads.size=20 > >>> >>> > >>> >>> autopurge.snapRetainCount=3 > >>> >>> autopurge.purgeInterval=1 > >>> >>> > >>> >>> # Clients > >>> >>> server.1=192.168.122.121:2888:3888 > >>> >>> server.2=192.168.122.122:2888:3888 > >>> >>> server.3=192.168.122.123:2888:3888| > >>> >>> > >>> >>> > >>> >>> This is my jaas_server.conf: > >>> >>> > >>> >>> |cat /opt/zookeeper/conf/jaas_server.conf| > >>> >>> > >>> >>> |# Reference: > >>> >>> > >>> >> > >>> > https://github.com/symat/zookeeper-docker-test/blob/master/conf/digest_jaas.conf > >>> >>> # Reference: https://issues.apache.org/jira/browse/ZOOKEEPER-3743 > >>> >>> Server { > >>> >>> org.apache.zookeeper.server.auth.DigestLoginModule required > >>> >>> user_zookeeper="test"; # Or use a different user/password as > >>> needed > >>> >>> }; > >>> >>> # This section is for ZooKeeper servers (QuorumServers) to > >>> authenticate > >>> >>> QuorumServer { > >>> >>> org.apache.zookeeper.server.auth.DigestLoginModule required > >>> >>> user_zookeeper="test"; > >>> >>> }; > >>> >>> # This section is for Quorum Learners to authenticate > >>> >>> QuorumLearner { > >>> >>> org.apache.zookeeper.server.auth.DigestLoginModule required > >>> >>> username="zookeeper" > >>> >>> password="test"; > >>> >>> };| > >>> >>> > >>> >>> > >>> >>> And this is my jaas_client.conf: > >>> >>> > >>> >>> |cat /opt/zookeeper/conf/jaas_client.conf > >>> >>> # Reference: https://issues.apache.org/jira/browse/ZOOKEEPER-3743 > >>> >>> > >>> >>> # This section is for ZooKeeper clients to authenticate > >>> >>> Client { > >>> >>> org.apache.zookeeper.server.auth.DigestLoginModule required > >>> >>> username="zookeeper" > >>> >>> password="test"; # Use the correct client credentials here > >>> >>> };| > >>> >>> > >>> >>> > >>> >>> Next, my java.env: > >>> >>> > >>> >>> |cat /opt/zookeeper/conf/java.env| > >>> >>> > >>> >>> |# Reference: > >>> >>> > >>> >> > >>> > https://cwiki.apache.org/confluence/display/ZOOKEEPER/Client-Server+mutual+authentication > >>> >>> # Reference: https://issues.apache.org/jira/browse/ZOOKEEPER-3743 > >>> >>> > >>> >>> # Defining your server side JAAS config file path: > >>> >>> > >>> >> > >>> > SERVER_JVMFLAGS="-Djava.security.auth.login.config=/opt/zookeeper/conf/jaas_server.conf" > >>> >>> # REQUIRED SASL RELATED CONFIGS: > >>> >>> # ==== java.security.auth.login.config: > >>> >>> # Defining your client side JAAS config file path: > >>> >>> CLIENT_JVMFLAGS="${CLIENT_JVMFLAGS} > >>> >>> > >>> -Djava.security.auth.login.config=/opt/zookeeper/conf/jaas_client.conf" > >>> >>> > >>> >>> # OPTIONAL SASL RELATED CONFIGS: > >>> >>> > >>> >>> # ==== zookeeper.sasl.client: > >>> >>> # You can disable SASL authentication on the client side (it is > true > >>> >>> by default): > >>> >>> CLIENT_JVMFLAGS="${CLIENT_JVMFLAGS} -Dzookeeper.sasl.client=true"| > >>> >>> > >>> >>> > >>> >>> Finally, my zookeeper.service: > >>> >>> > >>> >>> |cat /etc/systemd/system/zookeeper.service| > >>> >>> > >>> >>> |[Unit] > >>> >>> Description=Zookeeper Server > >>> >>> After=network.target > >>> >>> > >>> >>> [Service] > >>> >>> User=zookeeper > >>> >>> Group=zookeeper > >>> >>> Type=forking > >>> >>> EnvironmentFile=/opt/zookeeper/conf/java.env > >>> >>> ExecStart=/opt/zookeeper/bin/zkServer.sh start > >>> >>> ExecStop=/opt/zookeeper/bin/zkServer.sh stop > >>> >>> ExecReload=/opt/zookeeper/bin/zkServer.sh restart > >>> >>> Restart=on-failure > >>> >>> > >>> >>> [Install] > >>> >>> WantedBy=multi-user.target| > >>> >>> > >>> >>> > >>> >>> Coming to the error I see: > >>> >>> > >>> >>> |2025-03-09 19:22:12,539 [myid:1] - ERROR > >>> >>> [main:o.a.z.s.ServerCnxnFactory@267] - No JAAS configuration > section > >>> >>> named 'Server' was found in '/opt/zookeeper/conf/jaas_server.conf'. > >>> >>> 2025-03-09 19:22:12,541 [myid:1] - ERROR > >>> >>> [main:o.a.z.s.q.QuorumPeerMain@114] - Unexpected exception, > exiting > >>> >>> abnormally > >>> >>> java.io.IOException: No JAAS configuration section named 'Server' > was > >>> >>> found in '/opt/zookeeper/conf/jaas_server.conf'. > >>> >>> at > >>> >>> > >>> >> > >>> > org.apache.zookeeper.server.ServerCnxnFactory.configureSaslLogin(ServerCnxnFactory.java:268) > >>> >>> at > >>> >>> > >>> >> > >>> > org.apache.zookeeper.server.NIOServerCnxnFactory.configure(NIOServerCnxnFactory.java:623) > >>> >>> at > >>> >>> > >>> >> > >>> > org.apache.zookeeper.server.quorum.QuorumPeerMain.runFromConfig(QuorumPeerMain.java:169) > >>> >>> at > >>> >>> > >>> >> > >>> > org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:137) > >>> >>> at > >>> >>> > >>> >> > >>> > org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:91) > >>> >>> 2025-03-09 19:22:12,542 [myid:1] - INFO > >>> >>> [main:o.a.z.a.ZKAuditProvider@42] - ZooKeeper audit is disabled. > >>> >>> 2025-03-09 19:22:12,543 [myid:1] - ERROR > >>> >>> [main:o.a.z.u.ServiceUtils@48] - Exiting JVM with code 1| > >>> >>> > >>> >>> I have looked at many forum posts - the config I posted above is > >>> >>> directly from Mate Szalay-Beko's Github repo: > >>> >>> > >>> >> > >>> > https://github.com/symat/zookeeper-docker-test/blob/master/conf/digest_jaas.conf > >>> >>> Please tell me what I need to do to fix this. I have checked the > >>> >>> permissions to the files, necessary ports are open and everything > has > >>> >>> been |chown|ed to the zookeeper user and group. > >>> >>> > >>> >>> Thank you for your help. > >>> >>> > >>> >>> > >>> > >> >
