Bug#884344: ejabberd: After updating to latest erlang packages, ejabberd refuses to start
Package: ejabberd Version: 16.09-4 Severity: grave Justification: renders package unusable Just updated my system to the latest security updates for erlang packages, and after doing so, ejabberd refuses to start. Continually gives a "timeout" from systemd: Dec 14 02:02:53 myhost systemd[1]: Failed to start A distributed, fault-tolerant Jabber/XMPP server. Dec 14 02:02:53 myhost systemd[1]: ejabberd.service: Unit entered failed state. Dec 14 02:02:53 myhost systemd[1]: ejabberd.service: Failed with result 'timeout'. There doesn't seem to be anything useful to troubleshoot the problem in the systemd logs. Erlang packages that were updated that may be related to this breakage are: erlang-asn1 erlang-base erlang-corba erlang-crypto erlang-dev erlang-diameter erlang-doc erlang-edoc erlang-eldap erlang-erl-docgen erlang-eunit erlang-ic erlang-ic-java erlang-inets erlang-jinterface erlang-manpages erlang-mnesia erlang-nox erlang-odbc erlang-os-mon erlang-parsetools erlang-percept erlang-public-key erlang-runtime-tools erlang-snmp erlang-ssh erlang-ssl erlang-syntax-tools erlang-tools erlang-xmerl -- System Information: Debian Release: 9.3 APT prefers stable APT policy: (500, 'stable') Architecture: amd64 (x86_64) Kernel: Linux 3.16.0-4-amd64 (SMP w/4 CPU cores) Locale: LANG=C, LC_CTYPE=C (charmap=ANSI_X3.4-1968), LANGUAGE= (charmap=ANSI_X3.4-1968) Shell: /bin/sh linked to /bin/dash Init: systemd (via /run/systemd/system) Versions of packages ejabberd depends on: ii adduser3.115 ii debconf [debconf-2.0] 1.5.61 ii erlang-asn11:19.2.1+dfsg-2+deb9u1 ii erlang-base [erlang-abi-17.0] 1:19.2.1+dfsg-2+deb9u1 ii erlang-crypto 1:19.2.1+dfsg-2+deb9u1 ii erlang-inets 1:19.2.1+dfsg-2+deb9u1 ii erlang-jiffy 0.14.8+dfsg-1 ii erlang-lager 3.2.4-1 ii erlang-mnesia 1:19.2.1+dfsg-2+deb9u1 ii erlang-odbc1:19.2.1+dfsg-2+deb9u1 ii erlang-p1-cache-tab1.0.4-2 ii erlang-p1-iconv1.0.2-2 ii erlang-p1-stringprep 1.0.6-2 ii erlang-p1-tls 1.0.7-2+deb9u1 ii erlang-p1-utils1.0.5-3 ii erlang-p1-xml 1.1.15-2 ii erlang-p1-yaml 1.0.6-2 ii erlang-p1-zlib 1.0.1-4 ii erlang-public-key 1:19.2.1+dfsg-2+deb9u1 ii erlang-ssl 1:19.2.1+dfsg-2+deb9u1 ii erlang-syntax-tools1:19.2.1+dfsg-2+deb9u1 ii erlang-xmerl 1:19.2.1+dfsg-2+deb9u1 ii init-system-helpers1.48 ii lsb-base 9.20161125 ii openssl1.1.0f-3+deb9u1 ii ucf3.0036 ejabberd recommends no packages. Versions of packages ejabberd suggests: pn apparmor pn apparmor-utils ii ejabberd-contrib 0.2016.09.12~dfsg0-2 pn erlang-luerl pn erlang-p1-mysql pn erlang-p1-oauth2 pn erlang-p1-pam pn erlang-p1-pgsql pn erlang-p1-sip pn erlang-p1-sqlite3 pn erlang-p1-stun pn erlang-redis-client ii imagemagick 8:6.9.7.4+dfsg-11+deb9u3 ii imagemagick-6.q16 [imagemagick] 8:6.9.7.4+dfsg-11+deb9u3 ii libunix-syslog-perl 1.1-2+b6 pn yamllint -- Configuration Files: /etc/ejabberd/inetrc [Errno 13] Permission denied: '/etc/ejabberd/inetrc' /etc/ejabberd/modules.d/README.modules [Errno 13] Permission denied: '/etc/ejabberd/modules.d/README.modules' -- debconf information excluded
Bug#884344: ejabberd: After updating to latest erlang packages, ejabberd refuses to start
Hi, since the security patch in Erlang should not have any impact for ejabberd and because I don't have this problem on my Stretch ejabberd servers I doubt it is a generic issue and thus I will lower the severity. Please make sure that epmd and all other erlang processes are stopped before trying to start ejabberd again. If necessary, kill them manually. Please also check whether AppArmor is enabled (dmesg | grep -i apparmor). Have you tried rebooting the server yet? Without access to your ejabberd logs + dmesg I cannot really help you. Regards, -- .''`. Philipp Huebner : :' : pgp fp: 6719 25C5 B8CD E74A 5225 3DF9 E5CA 8C49 25E4 205F `. `'` `- signature.asc Description: OpenPGP digital signature
Bug#884344: ejabberd: After updating to latest erlang packages, ejabberd refuses to start
I am not using AppArmor, but I reconfirmed that by no results being found from "dmesg | grep -i apparmor". I killed all the epmd processes (there were two) and as far as I know ejabberd is my only erlang process. Restarting after killing the epmd processes results in two new epmd processes, but ejabberd still times out, and the server doesn't answer. The related processes that are running: 3700069 ?Ss 0:00 /bin/sh -c /usr/sbin/ejabberdctl start && /usr/sbin/ejabberdctl started 3700070 ?S 0:00 /bin/bash /usr/sbin/ejabberdctl start 3699902 ?Ss 0:00 /usr/bin/epmd -systemd 3700502 ?S 0:00 /usr/bin/epmd -names Here is all that is in the ejabberd.log from a start attempt: 2017-12-14 18:03:49.040 [info] <0.31.0> Application lager started on node ejabberd@kinison 2017-12-14 18:03:49.188 [info] <0.31.0> Application crypto started on node ejabberd@kinison 2017-12-14 18:03:49.256 [info] <0.31.0> Application sasl started on node ejabberd@kinison 2017-12-14 18:03:49.314 [info] <0.31.0> Application asn1 started on node ejabberd@kinison 2017-12-14 18:03:49.315 [info] <0.31.0> Application public_key started on node ejabberd@kinison 2017-12-14 18:03:49.357 [info] <0.31.0> Application ssl started on node ejabberd@kinison 2017-12-14 18:03:49.436 [info] <0.31.0> Application fast_yaml started on node ejabberd@kinison 2017-12-14 18:03:49.539 [info] <0.31.0> Application fast_tls started on node ejabberd@kinison 2017-12-14 18:03:49.614 [info] <0.31.0> Application fast_xml started on node ejabberd@kinison 2017-12-14 18:03:49.650 [info] <0.31.0> Application stringprep started on node ejabberd@kinison 2017-12-14 18:03:49.669 [info] <0.31.0> Application cache_tab started on node ejabberd@kinison 2017-12-14 18:03:50.469 [info] <0.31.0> Application mnesia started on node ejabberd@kinison There is nothing indicating any info about failure in the ejabberd.log. On Thu, 14 Dec 2017, Philipp Huebner wrote: Hi, since the security patch in Erlang should not have any impact for ejabberd and because I don't have this problem on my Stretch ejabberd servers I doubt it is a generic issue and thus I will lower the severity. Please make sure that epmd and all other erlang processes are stopped before trying to start ejabberd again. If necessary, kill them manually. Please also check whether AppArmor is enabled (dmesg | grep -i apparmor). Have you tried rebooting the server yet? Without access to your ejabberd logs + dmesg I cannot really help you. Regards, -- .''`. Philipp Huebner : :' : pgp fp: 6719 25C5 B8CD E74A 5225 3DF9 E5CA 8C49 25E4 205F `. `'` `-
Bug#884344: ejabberd: After updating to latest erlang packages, ejabberd refuses to start
Hi, please set the log level to debug (5) as described at https://docs.ejabberd.im/admin/guide/troubleshooting/#log-files and try again. It is very unusual that you seem not to get any error messages. Also check out error.log and crash.log. Do you do any kind of firewalling on the system in question? The module that should be started next after the log excerpt you sent is "inets" and then the tcp listeners. I assume all your erlang packages including erlang-inets have status "ii" in "dpkg -l | grep erlang" and the system is fully updated? I see that you have ejabberd-contrib installed, are you using any extra modules? Regards, -- .''`. Philipp Huebner : :' : pgp fp: 6719 25C5 B8CD E74A 5225 3DF9 E5CA 8C49 25E4 205F `. `'` `- signature.asc Description: OpenPGP digital signature
Bug#884344: ejabberd: After updating to latest erlang packages, ejabberd refuses to start
uI was already running loglevel 4, and changing it to 5 gave no additional info. There was nothing in the error.log or crash.log either, in fact those files hadn't even been opened, as the timestamps were still of times pre-upgrade. However, your hint to look for firewall issues was a good one. Looks like this was a problem with ejabberdctl using ipv6 by default. We don't use ipv6 here, so had no rules loaded for ipv6, and the default iptables policy was set to drop. Since Debian configures ipv6 addresses automatically I guess it was trying to use ipv6 anyway and never fell back to using ipv4. Apologies for the error. But this failure mode should probably give something more useful that just a timeout with no useful messages. Perhaps it should emit a message to say that ejabberdctl couldn't contact the daemon? Thanks.