Bug#1025012: zookeeper: starts but is completely unusable
Hello Chris, Le 06/12/2022 à 23:33, Christoph Anton Mitterer a écrit : Hey Pierre. On Tue, 2022-12-06 at 23:08 +0100, Pierre Gruet wrote: Thanks for the bug report (and the follow-up precisions you sent)! Yet I fail to reproduce it on testing. I installed zookeeper and zookeeperd on testing, then ran $ /usr/share/zookeeper/bin/zkCli.sh specifying nothing about the classpath (it is already handled in the MANIFEST.MF file of the zookeeper jar), and I could enter commands (though with no prompt, as you say). When you've started your zookeeper (I assume you also use the sysvinit script, respectively the virtual systemd unit generated from that?), what does your process' command line show, e.g.: # ps ax | grep org.apache.zookeeper.server.quorum.QuorumPeerMain 3156 ?Sl 167:08 /usr/bin/java -cp /etc/zookeeper/conf:/usr/share/java/zookeeper.jar:/usr/share/java/slf4j-log4j12.jar:/usr/share/java/log4j-1.2.jar -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.local.only=true -Dzookeeper.log.dir=/var/log/zookeeper -Dzookeeper.root.logger=INFO,ROLLINGFILE org.apache.zookeeper.server.quorum.QuorumPeerMain /etc/zookeeper/conf/zoo.cfg Especially, which CLASSPATH (seems to be the -cp)? Yes this is the -cp argument. I got 4558 ?Sl 0:02 /usr/bin/java -cp /etc/zookeeper/conf:/usr/share/java/zookeeper.jar -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.local.only=true -Dzookeeper.log.dir=/var/log/zookeeper -Dzookeeper.root.logger=INFO,ROLLINGFILE org.apache.zookeeper.server.quorum.QuorumPeerMain /etc/zookeeper/conf/zoo.cfg Could you please send the contents of your /etc/zookeeper/conf/environment? I think this is where the classpath is set, and we might have caught a wrong bullseye-to-bookworm upgrade case here. Is your test system clustered (i.e. more than one ZK node?)? I humbly admit I cannot answer as I am not a zookeeper user... but if you tell me where to look I might tell. (However, I also get the hanging issue if I do not install zookeeperd.) Well I guess that's clear, cause then nothing would have started zookeeper?! Sure :) I was stating this because here is the only way I could get the same kind of error as yours. I guess there is some difference in the versions of the dependencies between stable and testing, as you underline that the dependencies come from stable. That would be quite surprising, IMO,.. cause the issue seems to be quite clearly the missing CLASSPATH stuff. I mean if there would be some other package, that is used to start the java process... and that other package would have a different version and thus be incompatible, I could imagine this. But it seems as if zookeeper's sysvinit script does all on it's own and simply execute java in the end. So it's hard to imagine, that something interferes, and somehow breaks the CLASSPATH there?! As you suggest (see my request above), we don't have the same classpath and this might be because you already had zookeeper installed from old bullseye package and then upgraded. I think /etc/zookeeper/conf contents is set up by update-alternatives and maybe something is not working well in our post-install treatments in the case of an upgrade from an already-installed zookeeper. [...] Because of the last paragraph in the relevant section therein, I was unsure I should choose a SLF4J binding as this would impose my choice to the end user. What do you think? Well since that two infamous security holes, log4j has a somewhat damaged reputation ;-) ... but apart from that I think this would make the most sense here. As I've said in https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1025012#20 : /usr/share/java/slf4j-log4j12.jar wasn't enough for me and I needed to add /usr/share/java/log4j-1.2.jar to the CLASSPATH instead, in order to get output to /var/log/zookeeper You're right, this makes sense then. Finally, as: - I think there is a mismatch somewhere in the versions of the dependencies between stable and testing; - I cannot reproduce on a system with dependencies fetched from testing only, I would tend to discuss only the SLF4J issue and ignore the rest. Please tell me what you think, I think we should look into both, and at best split things apart. I mean I can try to reproduce the whole thing on a sid or testing only systemd an see whether it still fails to start, but right now I'd be quite surprised why it wouldn't start because of that... and even *if* that would be the reason, then bullseye to bookwork upgrades need to be supported, and therefore the package (zookeeper) would need to have correct versioned dependencies (*if* it's because of a incompatible version of some dependency). That's right, seeing your classpath contents in the top part of your last mail makes me think there is something wrong in the upgrade process and this should be addressed -- let's see this. I would be interested in your /etc/zookeeper/conf/en
Bug#1025012: zookeeper: starts but is completely unusable
Hey Pierre. On Tue, 2022-12-06 at 23:08 +0100, Pierre Gruet wrote: > Thanks for the bug report (and the follow-up precisions you sent)! > > Yet I fail to reproduce it on testing. I installed zookeeper and > zookeeperd on testing, then ran > > $ /usr/share/zookeeper/bin/zkCli.sh > specifying nothing about the classpath (it is already handled in the > MANIFEST.MF file of the zookeeper jar), and I could enter commands > (though with no prompt, as you say). When you've started your zookeeper (I assume you also use the sysvinit script, respectively the virtual systemd unit generated from that?), what does your process' command line show, e.g.: # ps ax | grep org.apache.zookeeper.server.quorum.QuorumPeerMain 3156 ?Sl 167:08 /usr/bin/java -cp /etc/zookeeper/conf:/usr/share/java/zookeeper.jar:/usr/share/java/slf4j-log4j12.jar:/usr/share/java/log4j-1.2.jar -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.local.only=true -Dzookeeper.log.dir=/var/log/zookeeper -Dzookeeper.root.logger=INFO,ROLLINGFILE org.apache.zookeeper.server.quorum.QuorumPeerMain /etc/zookeeper/conf/zoo.cfg Especially, which CLASSPATH (seems to be the -cp)? Is your test system clustered (i.e. more than one ZK node?)? > (However, I also get the hanging issue if I do not install > zookeeperd.) Well I guess that's clear, cause then nothing would have started zookeeper?! > I guess there is some difference in the versions of the dependencies > between stable and testing, as you underline that the dependencies > come > from stable. That would be quite surprising, IMO,.. cause the issue seems to be quite clearly the missing CLASSPATH stuff. I mean if there would be some other package, that is used to start the java process... and that other package would have a different version and thus be incompatible, I could imagine this. But it seems as if zookeeper's sysvinit script does all on it's own and simply execute java in the end. So it's hard to imagine, that something interferes, and somehow breaks the CLASSPATH there?! > Concerning the SLF4J warning: in the past I already stumbled upon > this > and visited the URI > https://www.slf4j.org/codes.html#StaticLoggerBinder > which is in the warning message. Yeah, also stumbled over that via the error message. > Because of the last paragraph in the > relevant section therein, I was unsure I should choose a SLF4J > binding > as this would impose my choice to the end user. What do you think? Well since that two infamous security holes, log4j has a somewhat damaged reputation ;-) ... but apart from that I think this would make the most sense here. As I've said in https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1025012#20 : /usr/share/java/slf4j-log4j12.jar wasn't enough for me and I needed to add /usr/share/java/log4j-1.2.jar to the CLASSPATH instead, in order to get output to /var/log/zookeeper > Finally, as: > - I think there is a mismatch somewhere in the versions of the > dependencies between stable and testing; > - I cannot reproduce on a system with dependencies fetched from > testing > only, > I would tend to discuss only the SLF4J issue and ignore the rest. > > Please tell me what you think, I think we should look into both, and at best split things apart. I mean I can try to reproduce the whole thing on a sid or testing only systemd an see whether it still fails to start, but right now I'd be quite surprised why it wouldn't start because of that... and even *if* that would be the reason, then bullseye to bookwork upgrades need to be supported, and therefore the package (zookeeper) would need to have correct versioned dependencies (*if* it's because of a incompatible version of some dependency). Thanks, Chris.
Bug#1025012: zookeeper: starts but is completely unusable
Control: tags -1 moreinfo unreproducible Hello Chris, On Mon, 28 Nov 2022 20:49:22 +0100 Christoph Anton Mitterer wrote: > Package: zookeeper > Version: 3.8.0-10 > Severity: grave > Justification: renders package unusable > > > Hey. > > I've tried the new packagin, but while all my config and data files are in place, > and while the server "runs", there is no logging (neither to stdout/err for systemd > nor /var/log/zookeeper .. not even any files get created in there). > > Connecting via zkCli gives: > # /usr/share/zookeeper/bin/zkCli.sh > SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder". > SLF4J: Defaulting to no-operation (NOP) logger implementation > SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details. > Connecting to localhost:2181 > Welcome to ZooKeeper! > JLine support is disabled > SLF4J: Failed to load class "org.slf4j.impl.StaticMDCBinder". > SLF4J: Defaulting to no-operation MDCAdapter implementation. > SLF4J: See http://www.slf4j.org/codes.html#no_static_mdc_binder for further details. > > > > Perhaps some missing dependency? Thanks for the bug report (and the follow-up precisions you sent)! Yet I fail to reproduce it on testing. I installed zookeeper and zookeeperd on testing, then ran $ /usr/share/zookeeper/bin/zkCli.sh specifying nothing about the classpath (it is already handled in the MANIFEST.MF file of the zookeeper jar), and I could enter commands (though with no prompt, as you say). (However, I also get the hanging issue if I do not install zookeeperd.) I guess there is some difference in the versions of the dependencies between stable and testing, as you underline that the dependencies come from stable. Concerning the SLF4J warning: in the past I already stumbled upon this and visited the URI https://www.slf4j.org/codes.html#StaticLoggerBinder which is in the warning message. Because of the last paragraph in the relevant section therein, I was unsure I should choose a SLF4J binding as this would impose my choice to the end user. What do you think? Finally, as: - I think there is a mismatch somewhere in the versions of the dependencies between stable and testing; - I cannot reproduce on a system with dependencies fetched from testing only, I would tend to discuss only the SLF4J issue and ignore the rest. Please tell me what you think, > > > Thanks, > Chris. > > Thanks to you, -- Pierre OpenPGP_signature Description: OpenPGP digital signature
Bug#1025012: zookeeper: starts but is completely unusable
And here we go: CLASSPATH="/etc/zookeeper/conf:/usr/share/java/zookeeper.jar:/usr/share/java/slf4j-log4j12.jar:/usr/share/java/log4j-1.2.jar" Seems to do the trick to get logging to /var/log/zookeeper/foobar . The zkCli shows still no prompt, though. It also needs the /usr/share/java/log4j-1.2.jar, /usr/share/java/slf4j-log4j12.jar alone ain't enough. I wonder a bit whether the classpath is just completely mess up? When installing, I needed the following dependencies: [INSTALL, DEPENDENCIES] junit4:amd64 4.13.1-2 [INSTALL, DEPENDENCIES] libactivation-java:amd64 1.2.0-2 [INSTALL, DEPENDENCIES] libapache-pom-java:amd64 18-1 [INSTALL, DEPENDENCIES] libasm-java:amd64 9.1-1 [INSTALL, DEPENDENCIES] libatinject-jsr330-api-java:amd64 1.0+ds1-5 [INSTALL, DEPENDENCIES] libcommons-cli-java:amd64 1.4-2 [INSTALL, DEPENDENCIES] libcommons-io-java:amd64 2.8.0-1 [INSTALL, DEPENDENCIES] libcommons-logging-java:amd64 1.2-2 [INSTALL, DEPENDENCIES] libcommons-parent-java:amd64 43-1 [INSTALL, DEPENDENCIES] libdropwizard-metrics-java:amd64 3.2.6-1 [INSTALL, DEPENDENCIES] libeclipse-jdt-core-java:amd64 3.24.0+eclipse4.18-1 [INSTALL, DEPENDENCIES] libel-api-java:amd64 3.0.0-3 [INSTALL, DEPENDENCIES] libfindbugs-annotations-java:amd64 3.1.0~preview2-3 [INSTALL, DEPENDENCIES] libguava-java:amd64 29.0-6 [INSTALL, DEPENDENCIES] libhamcrest-java:amd64 1.3-9 [INSTALL, DEPENDENCIES] libjackson2-annotations-java:amd64 2.12.1-1 [INSTALL, DEPENDENCIES] libjackson2-core-java:amd64 2.12.1-1 [INSTALL, DEPENDENCIES] libjackson2-databind-java:amd64 2.12.1-1+deb11u1 [INSTALL, DEPENDENCIES] libjaxb-api-java:amd64 2.3.1-1 [INSTALL, DEPENDENCIES] libjctools-java:amd64 2.0.2-1 [INSTALL, DEPENDENCIES] libjetty9-extra-java:amd64 9.4.39-3+deb11u1 [INSTALL, DEPENDENCIES] libjetty9-java:amd64 9.4.39-3+deb11u1 [INSTALL, DEPENDENCIES] libjffi-java:amd64 1.2.7-11 [INSTALL, DEPENDENCIES] libjffi-jni:amd64 1.2.7-11+b1 [INSTALL, DEPENDENCIES] libjnr-constants-java:amd64 0.10.1-1 [INSTALL, DEPENDENCIES] libjnr-enxio-java:amd64 0.32.3-2 [INSTALL, DEPENDENCIES] libjnr-ffi-java:amd64 2.1.7-1 [INSTALL, DEPENDENCIES] libjnr-posix-java:amd64 3.0.45-2 [INSTALL, DEPENDENCIES] libjnr-unixsocket-java:amd64 0.18-4 [INSTALL, DEPENDENCIES] libjnr-x86asm-java:amd64 1.0.2-5.1 [INSTALL, DEPENDENCIES] libjsp-api-java:amd64 2.3.4-3 [INSTALL, DEPENDENCIES] libjsr305-java:amd64 0.1~+svn49-11 [INSTALL, DEPENDENCIES] liblog4j1.2-java:amd64 1.2.17-10+deb11u1 [INSTALL, DEPENDENCIES] libmail-java:amd64 1.6.5-1 [INSTALL, DEPENDENCIES] libnetty-java:amd64 1:4.1.48-4 [INSTALL, DEPENDENCIES] libnetty-tcnative-java:amd64 2.0.28-1 [INSTALL, DEPENDENCIES] libnetty-tcnative-jni:amd64 2.0.28-1 [INSTALL, DEPENDENCIES] libservlet-api-java:amd64 4.0.1-2 [INSTALL, DEPENDENCIES] libservlet3.1-java:amd64 1:4.0.1-2 [INSTALL, DEPENDENCIES] libslf4j-java:amd64 1.7.30-1 [INSTALL, DEPENDENCIES] libsnappy-java:amd64 1.1.8.3-1 [INSTALL, DEPENDENCIES] libsnappy-jni:amd64 1.1.8.3-1 [INSTALL, DEPENDENCIES] libsnappy1v5:amd64 1.1.8-1 [INSTALL, DEPENDENCIES] libspring-beans-java:amd64 4.3.30-1 [INSTALL, DEPENDENCIES] libspring-core-java:amd64 4.3.30-1 [INSTALL, DEPENDENCIES] libtaglibs-standard-impl-java:amd64 1.2.5-2.1 [INSTALL, DEPENDENCIES] libtaglibs-standard-spec-java:amd64 1.2.5-2.1 [INSTALL, DEPENDENCIES] libtomcat9-java:amd64 9.0.43-2~deb11u4 [INSTALL, DEPENDENCIES] libwebsocket-api-java:amd64 1.1-2 [INSTALL, DEPENDENCIES] libzookeeper-java:amd64 3.8.0-10 [INSTALL, DEPENDENCIES] zookeeperd:amd64 3.8.0-10 [INSTALL] zookeeper:amd64 3.8.0-10 Wouldn't there need to be some CLASSPATH settings for more or less all of them? I guess there's at least a problem with that: 2022-11-28 21:41:34,760 - WARN [main:AdminServerFactory@58] - Unable to load jetty, not starting JettyAdminServer java.lang.NoClassDefFoundError: javax/servlet/Servlet at java.base/java.lang.Class.forName0(Native Method) at java.base/java.lang.Class.forName(Class.java:315) at org.apache.zookeeper.server.admin.AdminServerFactory.createAdminServer(AdminServerFactory.java:43) at org.apache.zookeeper.server.quorum.QuorumPeer.(QuorumPeer.java:1070) at org.apache.zookeeper.server.quorum.QuorumPeerMain.getQuorumPeer(QuorumPeerMain.java:246) at org.apache.zookeeper.server.quorum.QuorumPeerMain.runFromConfig(QuorumPeerMain.java:177) at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:137) at org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:91) Caused by: java.lang.ClassNotFoundException: javax.servlet.Servlet at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:581) at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178) at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:522) ... 8 more Thanks, Chris.
Bug#1025012: zookeeper: starts but is completely unusable
I got a bit further: Setting: CLASSPATH="/etc/zookeeper/conf:/usr/share/java/zookeeper.jar:/usr/share/java/slf4j-simple.jar" i.e. adding the ":/usr/share/java/slf4j-simple.jar" helps a bit... The server seems to start now, and via zkCli, I can `ls` my paths and `get` values. But there's still no logging (neither to file nor to systemd)... ...and zkCli used to show a "prompt" like: [zk: localhost:2181(CONNECTED) 3] commands go here but now, there is nothing, though I can enter commands. May be unrelated though
Bug#1025012: zookeeper: starts but is completely unusable
I should perhaps add, that I have installed the zookeeper packages (zookeeper zookeeperd libzookeeper-java) from testing into stable (bullseye), all other dependencies were already met with bullseye versions. Also, according to https://www.slf4j.org/codes.html#StaticLoggerBinder and there the question "Failed to load class org.slf4j.impl.StaticLoggerBinder"... the files "slf4j-nop.jar slf4j- simple.jar, slf4j-log4j12.jar, slf4j-jdk14.jar or logback-classic.jar" are needed,... but any except for the latter is in place. All are in /usr/share/java. Though manually adding this to the CLASSPATH in /etc/zookeeper/conf/environment doesn't seem to help.
Bug#1025012: zookeeper: starts but is completely unusable
Package: zookeeper Version: 3.8.0-10 Severity: grave Justification: renders package unusable Hey. I've tried the new packagin, but while all my config and data files are in place, and while the server "runs", there is no logging (neither to stdout/err for systemd nor /var/log/zookeeper .. not even any files get created in there). Connecting via zkCli gives: # /usr/share/zookeeper/bin/zkCli.sh SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder". SLF4J: Defaulting to no-operation (NOP) logger implementation SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details. Connecting to localhost:2181 Welcome to ZooKeeper! JLine support is disabled SLF4J: Failed to load class "org.slf4j.impl.StaticMDCBinder". SLF4J: Defaulting to no-operation MDCAdapter implementation. SLF4J: See http://www.slf4j.org/codes.html#no_static_mdc_binder for further details. Perhaps some missing dependency? Thanks, Chris.
