Bug#884344: ejabberd: After updating to latest erlang packages, ejabberd refuses to start

[E Harris]

uI was already running loglevel 4, and changing it to 5 gave no additional 
info.  There was nothing in the error.log or crash.log either, in fact those 
files hadn't even been opened, as the timestamps were still of times 
pre-upgrade.


However, your hint to look for firewall issues was a good one.

Looks like this was a problem with ejabberdctl using ipv6 by default.  We 
don't use ipv6 here, so had no rules loaded for ipv6, and the default 
iptables policy was set to drop.  Since Debian configures ipv6 addresses 
automatically I guess it was trying to use ipv6 anyway and never fell 
back to using ipv4.


Apologies for the error.  But this failure mode should probably give 
something more useful that just a timeout with no useful messages.  Perhaps 
it should emit a message to say that ejabberdctl couldn't contact the 
daemon?


Thanks.

Bug#884344: ejabberd: After updating to latest erlang packages, ejabberd refuses to start

[Philipp Huebner]

Hi,

please set the log level to debug (5) as described at
https://docs.ejabberd.im/admin/guide/troubleshooting/#log-files
and try again. It is very unusual that you seem not to get any error
messages.

Also check out error.log and crash.log.

Do you do any kind of firewalling on the system in question?
The module that should be started next after the log excerpt you sent is
"inets" and then the tcp listeners. I assume all your erlang packages
including erlang-inets have status "ii" in "dpkg -l | grep erlang" and
the system is fully updated?

I see that you have ejabberd-contrib installed, are you using any extra
modules?

Regards,
-- 
 .''`.   Philipp Huebner 
: :'  :  pgp fp: 6719 25C5 B8CD E74A 5225  3DF9 E5CA 8C49 25E4 205F
`. `'`
  `-



signature.asc
Description: OpenPGP digital signature

Bug#884344: ejabberd: After updating to latest erlang packages, ejabberd refuses to start

[E Harris]

I am not using AppArmor, but I reconfirmed that by no results being found 
from "dmesg | grep -i apparmor".


I killed all the epmd processes (there were two) and as far as I know 
ejabberd is my only erlang process.  Restarting after killing the epmd 
processes results in two new epmd processes, but ejabberd still times out, 
and the server doesn't answer.


The related processes that are running:

3700069 ?Ss 0:00 /bin/sh -c /usr/sbin/ejabberdctl start && 
/usr/sbin/ejabberdctl started
3700070 ?S  0:00 /bin/bash /usr/sbin/ejabberdctl start
3699902 ?Ss 0:00 /usr/bin/epmd -systemd
3700502 ?S  0:00 /usr/bin/epmd -names


Here is all that is in the ejabberd.log from a start attempt:

2017-12-14 18:03:49.040 [info] <0.31.0> Application lager started on node 
ejabberd@kinison
2017-12-14 18:03:49.188 [info] <0.31.0> Application crypto started on node 
ejabberd@kinison
2017-12-14 18:03:49.256 [info] <0.31.0> Application sasl started on node 
ejabberd@kinison
2017-12-14 18:03:49.314 [info] <0.31.0> Application asn1 started on node 
ejabberd@kinison
2017-12-14 18:03:49.315 [info] <0.31.0> Application public_key started on node 
ejabberd@kinison
2017-12-14 18:03:49.357 [info] <0.31.0> Application ssl started on node 
ejabberd@kinison
2017-12-14 18:03:49.436 [info] <0.31.0> Application fast_yaml started on node 
ejabberd@kinison
2017-12-14 18:03:49.539 [info] <0.31.0> Application fast_tls started on node 
ejabberd@kinison
2017-12-14 18:03:49.614 [info] <0.31.0> Application fast_xml started on node 
ejabberd@kinison
2017-12-14 18:03:49.650 [info] <0.31.0> Application stringprep started on node 
ejabberd@kinison
2017-12-14 18:03:49.669 [info] <0.31.0> Application cache_tab started on node 
ejabberd@kinison
2017-12-14 18:03:50.469 [info] <0.31.0> Application mnesia started on node 
ejabberd@kinison

There is nothing indicating any info about failure in the ejabberd.log.


On Thu, 14 Dec 2017, Philipp Huebner wrote:


Hi,

since the security patch in Erlang should not have any impact for
ejabberd and because I don't have this problem on my Stretch ejabberd
servers I doubt it is a generic issue and thus I will lower the severity.

Please make sure that epmd and all other erlang processes are stopped
before trying to start ejabberd again. If necessary, kill them manually.

Please also check whether AppArmor is enabled (dmesg | grep -i
apparmor). Have you tried rebooting the server yet?

Without access to your ejabberd logs + dmesg I cannot really help you.

Regards,
--
.''`.   Philipp Huebner 
: :'  :  pgp fp: 6719 25C5 B8CD E74A 5225  3DF9 E5CA 8C49 25E4 205F
`. `'`
 `-

Bug#884344: ejabberd: After updating to latest erlang packages, ejabberd refuses to start

[Philipp Huebner]

Hi,

since the security patch in Erlang should not have any impact for
ejabberd and because I don't have this problem on my Stretch ejabberd
servers I doubt it is a generic issue and thus I will lower the severity.

Please make sure that epmd and all other erlang processes are stopped
before trying to start ejabberd again. If necessary, kill them manually.

Please also check whether AppArmor is enabled (dmesg | grep -i
apparmor). Have you tried rebooting the server yet?

Without access to your ejabberd logs + dmesg I cannot really help you.

Regards,
-- 
 .''`.   Philipp Huebner 
: :'  :  pgp fp: 6719 25C5 B8CD E74A 5225  3DF9 E5CA 8C49 25E4 205F
`. `'`
  `-



signature.asc
Description: OpenPGP digital signature

Bug#884344: ejabberd: After updating to latest erlang packages, ejabberd refuses to start

[E Harris]

Package: ejabberd
Version: 16.09-4
Severity: grave
Justification: renders package unusable

Just updated my system to the latest security updates for erlang packages, 
and after doing so, ejabberd refuses to start.  Continually gives a 
"timeout" from systemd:

Dec 14 02:02:53 myhost systemd[1]: Failed to start A distributed, 
fault-tolerant Jabber/XMPP server.
Dec 14 02:02:53 myhost systemd[1]: ejabberd.service: Unit entered failed state.
Dec 14 02:02:53 myhost systemd[1]: ejabberd.service: Failed with result 
'timeout'.

There doesn't seem to be anything useful to troubleshoot the problem in the
systemd logs.

Erlang packages that were updated that may be related to this breakage are:

erlang-asn1 erlang-base erlang-corba erlang-crypto erlang-dev erlang-diameter 
erlang-doc erlang-edoc erlang-eldap erlang-erl-docgen erlang-eunit erlang-ic 
erlang-ic-java erlang-inets erlang-jinterface erlang-manpages erlang-mnesia 
erlang-nox erlang-odbc erlang-os-mon erlang-parsetools erlang-percept
erlang-public-key erlang-runtime-tools erlang-snmp erlang-ssh erlang-ssl 
erlang-syntax-tools erlang-tools erlang-xmerl

-- System Information:
Debian Release: 9.3
  APT prefers stable
  APT policy: (500, 'stable')
Architecture: amd64 (x86_64)

Kernel: Linux 3.16.0-4-amd64 (SMP w/4 CPU cores)
Locale: LANG=C, LC_CTYPE=C (charmap=ANSI_X3.4-1968), LANGUAGE= 
(charmap=ANSI_X3.4-1968)
Shell: /bin/sh linked to /bin/dash
Init: systemd (via /run/systemd/system)

Versions of packages ejabberd depends on:
ii  adduser3.115
ii  debconf [debconf-2.0]  1.5.61
ii  erlang-asn11:19.2.1+dfsg-2+deb9u1
ii  erlang-base [erlang-abi-17.0]  1:19.2.1+dfsg-2+deb9u1
ii  erlang-crypto  1:19.2.1+dfsg-2+deb9u1
ii  erlang-inets   1:19.2.1+dfsg-2+deb9u1
ii  erlang-jiffy   0.14.8+dfsg-1
ii  erlang-lager   3.2.4-1
ii  erlang-mnesia  1:19.2.1+dfsg-2+deb9u1
ii  erlang-odbc1:19.2.1+dfsg-2+deb9u1
ii  erlang-p1-cache-tab1.0.4-2
ii  erlang-p1-iconv1.0.2-2
ii  erlang-p1-stringprep   1.0.6-2
ii  erlang-p1-tls  1.0.7-2+deb9u1
ii  erlang-p1-utils1.0.5-3
ii  erlang-p1-xml  1.1.15-2
ii  erlang-p1-yaml 1.0.6-2
ii  erlang-p1-zlib 1.0.1-4
ii  erlang-public-key  1:19.2.1+dfsg-2+deb9u1
ii  erlang-ssl 1:19.2.1+dfsg-2+deb9u1
ii  erlang-syntax-tools1:19.2.1+dfsg-2+deb9u1
ii  erlang-xmerl   1:19.2.1+dfsg-2+deb9u1
ii  init-system-helpers1.48
ii  lsb-base   9.20161125
ii  openssl1.1.0f-3+deb9u1
ii  ucf3.0036

ejabberd recommends no packages.

Versions of packages ejabberd suggests:
pn  apparmor 
pn  apparmor-utils   
ii  ejabberd-contrib 0.2016.09.12~dfsg0-2
pn  erlang-luerl 
pn  erlang-p1-mysql  
pn  erlang-p1-oauth2 
pn  erlang-p1-pam
pn  erlang-p1-pgsql  
pn  erlang-p1-sip
pn  erlang-p1-sqlite3
pn  erlang-p1-stun   
pn  erlang-redis-client  
ii  imagemagick  8:6.9.7.4+dfsg-11+deb9u3
ii  imagemagick-6.q16 [imagemagick]  8:6.9.7.4+dfsg-11+deb9u3
ii  libunix-syslog-perl  1.1-2+b6
pn  yamllint 

-- Configuration Files:
/etc/ejabberd/inetrc [Errno 13] Permission denied: '/etc/ejabberd/inetrc'
/etc/ejabberd/modules.d/README.modules [Errno 13] Permission denied: 
'/etc/ejabberd/modules.d/README.modules'

-- debconf information excluded