Re: [ClusterLabs] Upgrade corosync problem
On 29/06/18 19:13 +0200, Salvatore D'angelo wrote: > Good to know. I'll try it. I'll try to work on VM too. If that won't work, you can also try: docker run ... --ulimit memlock=33554432 ... where 32768 (kiB) may still be not enough (assuming the default of 16384), hard to say, since proper root user may normally be bypassing any such limitations. Good luck. > Il Ven 29 Giu 2018, 5:46 PM Jan Pokorný ha scritto: > >> On 26/06/18 11:03 +0200, Salvatore D'angelo wrote: >>> Yes, sorry you’re right I could find it by myself. >>> However, I did the following: >>> >>> 1. Added the line you suggested to /etc/fstab >>> 2. mount -o remount /dev/shm >>> 3. Now I correctly see /dev/shm of 512M with df -h >>> Filesystem Size Used Avail Use% Mounted on >>> overlay 63G 11G 49G 19% / >>> tmpfs64M 4.0K 64M 1% /dev >>> tmpfs 1000M 0 1000M 0% /sys/fs/cgroup >>> osxfs 466G 158G 305G 35% /Users >>> /dev/sda163G 11G 49G 19% /etc/hosts >>> shm 512M 15M 498M 3% /dev/shm >>> tmpfs 1000M 0 1000M 0% /sys/firmware >>> tmpfs 128M 0 128M 0% /tmp >>> >>> The errors in log went away. Consider that I remove the log file >>> before start corosync so it does not contains lines of previous >>> executions. >>> >>> >>> But the command: >>> corosync-quorumtool -ps >>> >>> still give: >>> Cannot initialize QUORUM service >>> >>> Consider that few minutes before it gave me the message: >>> Cannot initialize CFG service >>> >>> I do not know the differences between CFG and QUORUM in this case. >>> >>> If I try to start pacemaker the service is OK but I see only >>> pacemaker and the Transport does not work if I try to run a cam >>> command. >>> Any suggestion? >> >> Frankly, best generic suggestion I can serve with is to learn >> sufficient portions of the details about the tool you are relying on. >> >> I had a second look and it seems that what drives the actual >> size of the container's /dev/shm mountpoint with docker >> (per other response, you don't seem to be using --ipc switch) is >> it's --shm-size option for "run" subcommand (hence it's rather >> a property of the run-time, as the default of "64m" may be >> silently overriding your believed-to-be-persistent static changes >> within the container). >> >> Try using that option and you'll see. Definitely keep you mind open >> regarding "container != magic-less system" inequality. -- Jan (Poki) pgpqx78u6Noj5.pgp Description: PGP signature ___ Users mailing list: Users@clusterlabs.org https://lists.clusterlabs.org/mailman/listinfo/users Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
Re: [ClusterLabs] corosync-qdevice doesn't daemonize (or stay running)
On Thu, Jun 21, 2018 at 10:47 AM Jason Gauthier wrote: > > On Thu, Jun 21, 2018 at 9:49 AM Jan Pokorný wrote: > > > > On 21/06/18 07:05 -0400, Jason Gauthier wrote: > > > On Thu, Jun 21, 2018 at 5:11 AM Christine Caulfield > > > wrote: > > >> On 19/06/18 18:47, Jason Gauthier wrote: > > >>> Attached! > > >> > > >> That's very odd. I can see communication with the server and corosync in > > >> there (do it's doing something) but no logging at all. When I start > > >> qdevice on my systems it logs loads of messages even if it doesn't > > >> manage to contact the server. Do you have any logging entries in > > >> corosync.conf that might be stopping it? > > > > > > I haven't checked the corosync logs for any entries before, but I just > > > did. There isn't anything logged. > > > > What about syslog entries (may boil down to /var/log/messages, > > journald log, or whatever sink is configured)? > > I took a look, since both you and Chrissie mentioned that. > > There aren't any new entries added to any of the /var/log files. > > # corosync-qdevice -f -d > # date > Thu Jun 21 10:36:06 EDT 2018 > > # ls -lt|head > total 152072 > -rw-r- 1 rootadm 68018 Jun 21 10:34 auth.log > -rw-rw-r-- 1 rootutmp 18704352 Jun 21 10:34 lastlog > -rw-rw-r-- 1 rootutmp107136 Jun 21 10:34 wtmp > -rw-r- 1 rootadm 248444 Jun 21 10:34 daemon.log > -rw-r- 1 rootadm 160899 Jun 21 10:34 syslog > -rw-r- 1 rootadm1119856 Jun 21 09:46 kern.log > > I did look through daemon, messages, and syslog just to be sure. > > > >> Where did the binary come from? did you build it yourself or is it from > > >> a package? I wonder if it's got corrupted or is a bad version. Possibly > > >> linked against a 'dodgy' libqb - there have been some things going on > > >> there that could cause logging to go missing in some circumstances. > > >> > > >> Honza (the qdevice expert) is away at the moment, so I'm guessing a bit > > >> here anyway! > > > > > > Hmm. Interesting. I installed the debian package. When it didn't > > > work, I grabbed the source from github. They both act the same way, > > > but if there is an underlying library issue then that will continue to > > > be a problem. > > > > > > It doesn't say much: > > > /usr/lib/x86_64-linux-gnu/libqb.so.0.18.1 > > > > You are likely using libqb v1.0.1. > > Correct. I didn't even think to look at the output of dpkg -l for the > package version. > Debian 9 also packages binutils-2.28 > > > Ability to figure out the proper package version is one of the most > > basic skills to provide useful diagnostics about the issues with > > distro-provided packages. > > > > With Debian, the proper incantation seems to be > > > > dpkg -s libqb-dev | grep -i version > > > > or > > > > apt list libqb-dev > > > > (or substitute libqb0 for libqb-dev). > > > > As Chrissie mentioned, there is some fishiness possible if you happen > > to use ld linker from binutils 2.29+ for the building with this old > > libqb in the mix, so if the issues persist and logging seems to be > > missing, try recompiling with the downgraded binutils package below > > said breakage point. > > Since the system already has a lower numbered binutils (2.28) I wonder > if I should attempt to build a newer version of the libqb library. > > As Chrissie mentioned, I will open a bug with Debian in the Interim. > But I don 't believe I will see resolution to that any time soon. :) I was finally able to look at this problem again, and found that qnetd is giving me some messaging, but I don't know what to do with it. Jun 29 16:34:35 debug New client connected Jun 29 16:34:35 debug cluster name = zeta Jun 29 16:34:35 debug tls started = 1 Jun 29 16:34:35 debug tls peer certificate verified = 1 Jun 29 16:34:35 debug node_id = 1084772368 Jun 29 16:34:35 debug pointer = 0x563afd609d70 Jun 29 16:34:35 debug addr_str = :::192.168.80.16:38010 Jun 29 16:34:35 debug ring id = (40a85010.89ec) Jun 29 16:34:35 debug cluster dump: Jun 29 16:34:35 debug client = :::192.168.80.16:38010, node_id = 1084772368 Jun 29 16:34:35 debug Client :::192.168.80.16:38010 (cluster zeta, node_id 1084772368) sent initial node list. Jun 29 16:34:35 debug msg seq num 4 Jun 29 16:34:35 debug node list: Jun 29 16:34:35 error ffsplit: Received empty config node list for client :::192.168.80.16:38010 Jun 29 16:34:35 error Algorithm returned error code. Sending error reply. Jun 29 16:34:35 debug Client :::192.168.80.16:38010 (cluster zeta, node_id 1084772368) sent membership node list. Jun 29 16:34:35 debug msg seq num 5 Jun 29 16:34:35 debug ring id = (40a85010.89ec) Jun 29 16:34:35 debug node list: Jun 29 16:34:35 debug node_id = 1084772368, data_center_id = 0, node_state = not set Jun 29 16:34:35 debug node_id = 1084772369, data_center_id = 0, node_state = not set Jun 29 16:34:35
Re: [ClusterLabs] Upgrade corosync problem
Good to know. I'll try it. I'll try to work on VM too. Il Ven 29 Giu 2018, 5:46 PM Jan Pokorný ha scritto: > On 26/06/18 11:03 +0200, Salvatore D'angelo wrote: > > Yes, sorry you’re right I could find it by myself. > > However, I did the following: > > > > 1. Added the line you suggested to /etc/fstab > > 2. mount -o remount /dev/shm > > 3. Now I correctly see /dev/shm of 512M with df -h > > Filesystem Size Used Avail Use% Mounted on > > overlay 63G 11G 49G 19% / > > tmpfs64M 4.0K 64M 1% /dev > > tmpfs 1000M 0 1000M 0% /sys/fs/cgroup > > osxfs 466G 158G 305G 35% /Users > > /dev/sda163G 11G 49G 19% /etc/hosts > > shm 512M 15M 498M 3% /dev/shm > > tmpfs 1000M 0 1000M 0% /sys/firmware > > tmpfs 128M 0 128M 0% /tmp > > > > The errors in log went away. Consider that I remove the log file > > before start corosync so it does not contains lines of previous > > executions. > > > > > > But the command: > > corosync-quorumtool -ps > > > > still give: > > Cannot initialize QUORUM service > > > > Consider that few minutes before it gave me the message: > > Cannot initialize CFG service > > > > I do not know the differences between CFG and QUORUM in this case. > > > > If I try to start pacemaker the service is OK but I see only > > pacemaker and the Transport does not work if I try to run a cam > > command. > > Any suggestion? > > Frankly, best generic suggestion I can serve with is to learn > sufficient portions of the details about the tool you are relying on. > > I had a second look and it seems that what drives the actual > size of the container's /dev/shm mountpoint with docker > (per other response, you don't seem to be using --ipc switch) is > it's --shm-size option for "run" subcommand (hence it's rather > a property of the run-time, as the default of "64m" may be > silently overriding your believed-to-be-persistent static changes > within the container). > > Try using that option and you'll see. Definitely keep you mind open > regarding "container != magic-less system" inequality. > > -- > Jan (Poki) > ___ > Users mailing list: Users@clusterlabs.org > https://lists.clusterlabs.org/mailman/listinfo/users > > Project Home: http://www.clusterlabs.org > Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf > Bugs: http://bugs.clusterlabs.org > ___ Users mailing list: Users@clusterlabs.org https://lists.clusterlabs.org/mailman/listinfo/users Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
Re: [ClusterLabs] Upgrade corosync problem
On 29/06/18 10:00 +0100, Christine Caulfield wrote: > On 27/06/18 08:35, Salvatore D'angelo wrote: >> One thing that I do not understand is that I tried to compare corosync >> 2.3.5 (the old version that worked fine) and 2.4.4 to understand >> differences but I haven’t found anything related to the piece of code >> that affects the issue. The quorum tool.c and cfg.c are almost the same. >> Probably the issue is somewhere else. >> > > This might be asking a bit much, but would it be possible to try this > using Virtual Machines rather than Docker images? That would at least > eliminate a lot of complex variables. Salvatore, you can ignore the part below, try following the "--shm" advice in other part of this thread. Also the previous suggestion to compile corosync with --small-memory-footprint may be of help, but comes with other costs (expect lower throughput). Chrissie, I have a plausible explanation and if it's true, then the same will be reproduced wherever /dev/shm is small enough. If I am right, then the offending commit is https://github.com/corosync/corosync/commit/238e2e62d8b960e7c10bfa0a8281d78ec99f3a26 (present since 2.4.3), and while it arranges things for the better in the context of prioritized, low jitter process, it all of a sudden prevents as-you-need memory acquisition from the system, meaning that the memory consumption constraints are checked immediately when the memory is claimed (as it must fit into dedicated physical memory in full). Hence this impact we likely never realized may be perceived as a sort of a regression. Since we can calculate the approximate requirements statically, might be worthy to add something like README.requirements, detailing how much space will be occupied for typical configurations at minimum, e.g.: - standard + --small-memory-footprint configuration - 2 + 3 + X nodes (5?) - without any service on top + teamed with qnetd + teamed with pacemaker atop (including just IPC channels between pacemaker daemons and corosync's CPG service, indeed) -- Jan (Poki) pgpbz8BsKG8wW.pgp Description: PGP signature ___ Users mailing list: Users@clusterlabs.org https://lists.clusterlabs.org/mailman/listinfo/users Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
Re: [ClusterLabs] Upgrade corosync problem
On 26/06/18 11:03 +0200, Salvatore D'angelo wrote: > Yes, sorry you’re right I could find it by myself. > However, I did the following: > > 1. Added the line you suggested to /etc/fstab > 2. mount -o remount /dev/shm > 3. Now I correctly see /dev/shm of 512M with df -h > Filesystem Size Used Avail Use% Mounted on > overlay 63G 11G 49G 19% / > tmpfs64M 4.0K 64M 1% /dev > tmpfs 1000M 0 1000M 0% /sys/fs/cgroup > osxfs 466G 158G 305G 35% /Users > /dev/sda163G 11G 49G 19% /etc/hosts > shm 512M 15M 498M 3% /dev/shm > tmpfs 1000M 0 1000M 0% /sys/firmware > tmpfs 128M 0 128M 0% /tmp > > The errors in log went away. Consider that I remove the log file > before start corosync so it does not contains lines of previous > executions. > > > But the command: > corosync-quorumtool -ps > > still give: > Cannot initialize QUORUM service > > Consider that few minutes before it gave me the message: > Cannot initialize CFG service > > I do not know the differences between CFG and QUORUM in this case. > > If I try to start pacemaker the service is OK but I see only > pacemaker and the Transport does not work if I try to run a cam > command. > Any suggestion? Frankly, best generic suggestion I can serve with is to learn sufficient portions of the details about the tool you are relying on. I had a second look and it seems that what drives the actual size of the container's /dev/shm mountpoint with docker (per other response, you don't seem to be using --ipc switch) is it's --shm-size option for "run" subcommand (hence it's rather a property of the run-time, as the default of "64m" may be silently overriding your believed-to-be-persistent static changes within the container). Try using that option and you'll see. Definitely keep you mind open regarding "container != magic-less system" inequality. -- Jan (Poki) pgpwkcAAfuhlc.pgp Description: PGP signature ___ Users mailing list: Users@clusterlabs.org https://lists.clusterlabs.org/mailman/listinfo/users Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
Re: [ClusterLabs] Upgrade corosync problem
On 27/06/18 08:35, Salvatore D'angelo wrote:
> Hi,
>
> Thanks for reply and detailed explaination. I am not using the
> —network=host option.
> I have a docker image based on Ubuntu 14.04 where I only deploy this
> additional software:
>
> *RUN apt-get update && apt-get install -y wget git xz-utils
> openssh-server \*
> *systemd-services make gcc pkg-config psmisc fuse libpython2.7
> libopenipmi0 \*
> *libdbus-glib-1-2 libsnmp30 libtimedate-perl libpcap0.8*
>
> configure ssh with key pairs to communicate easily. The containers are
> created with these simple commands:
>
> *docker create -it --cap-add=MKNOD --cap-add SYS_ADMIN --device
> /dev/loop0 --device /dev/fuse --net ${PUBLIC_NETWORK_NAME} --publish
> ${PG1_SSH_PORT}:22 --ip ${PG1_PUBLIC_IP} --name ${PG1_PRIVATE_NAME}
> --hostname ${PG1_PRIVATE_NAME} -v ${MOUNT_FOLDER}:/Users ngha /bin/bash*
>
> *docker create -it --cap-add=MKNOD --cap-add SYS_ADMIN --device
> /dev/loop1 --device /dev/fuse --net ${PUBLIC_NETWORK_NAME} --publish
> ${PG2_SSH_PORT}:22 --ip ${PG2_PUBLIC_IP} --name ${PG2_PRIVATE_NAME}
> --hostname ${PG2_PRIVATE_NAME} -v ${MOUNT_FOLDER}:/Users ngha /bin/bash*
>
> *docker create -it --cap-add=MKNOD --cap-add SYS_ADMIN --device
> /dev/loop2 --device /dev/fuse --net ${PUBLIC_NETWORK_NAME} --publish
> ${PG3_SSH_PORT}:22 --ip ${PG3_PUBLIC_IP} --name ${PG3_PRIVATE_NAME}
> --hostname ${PG3_PRIVATE_NAME} -v ${MOUNT_FOLDER}:/Users ngha /bin/bash*
>
> /dev/fuse is used to configure glusterfs on two others nodes and
> /dev/loopX just to simulate better my bare metal env.
>
> One thing that I do not understand is that I tried to compare corosync
> 2.3.5 (the old version that worked fine) and 2.4.4 to understand
> differences but I haven’t found anything related to the piece of code
> that affects the issue. The quorum tool.c and cfg.c are almost the same.
> Probably the issue is somewhere else.
>
This might be asking a bit much, but would it be possible to try this
using Virtual Machines rather than Docker images? That would at least
eliminate a lot of complex variables.
Chrissie
>
>> On 27 Jun 2018, at 08:34, Jan Pokorný > > wrote:
>>
>> On 26/06/18 17:56 +0200, Salvatore D'angelo wrote:
>>> I did another test. I modified docker container in order to be able
>>> to run strace.
>>> Running strace corosync-quorumtool -ps I got the following:
>>
>>> [snipped]
>>> connect(5, {sa_family=AF_LOCAL, sun_path=@"cfg"}, 110) = 0
>>> setsockopt(5, SOL_SOCKET, SO_PASSCRED, [1], 4) = 0
>>> sendto(5,
>>> "\377\377\377\377\0\0\0\0\30\0\0\0\0\0\0\0\0\0\20\0\0\0\0\0", 24,
>>> MSG_NOSIGNAL, NULL, 0) = 24
>>> setsockopt(5, SOL_SOCKET, SO_PASSCRED, [0], 4) = 0
>>> recvfrom(5, 0x7ffd73bd7ac0, 12328, 16640, 0, 0) = -1 EAGAIN (Resource
>>> temporarily unavailable)
>>> poll([{fd=5, events=POLLIN}], 1, 4294967295) = 1 ([{fd=5,
>>> revents=POLLIN}])
>>> recvfrom(5,
>>> "\377\377\377\377\0\0\0\0(0\0\0\0\0\0\0\365\377\377\377\0\0\0\0\0\0\0\0\0\0\0\0"...,
>>> 12328, MSG_WAITALL|MSG_NOSIGNAL, NULL, NULL) = 12328
>>> shutdown(5, SHUT_RDWR) = 0
>>> close(5) = 0
>>> write(2, "Cannot initialise CFG service\n", 30Cannot initialise CFG
>>> service) = 30
>>> [snipped]
>>
>> This just demonstrated the effect of already detailed server-side
>> error in the client, which communicates with the server just fine,
>> but as soon as the server hits the mmap-based problem, it bails
>> out the observed way, leaving client unsatisfied.
>>
>> Note one thing, abstract Unix sockets are being used for the
>> communication like this (observe the first line in the strace
>> output excerpt above), and if you happen to run container via
>> a docker command with --network=host, you may also be affected with
>> issues arising from abstract sockets not being isolated but rather
>> sharing the same namespace. At least that was the case some years
>> back and what asked for a switch in underlying libqb library to
>> use strictly the file-backed sockets, where the isolation
>> semantics matches the intuition:
>>
>> https://lists.clusterlabs.org/pipermail/users/2017-May/013003.html
>>
>> + way to enable (presumably only for container environments, note
>> that there's no per process straightforward granularity):
>>
>> https://clusterlabs.github.io/libqb/1.0.2/doxygen/qb_ipc_overview.html
>> (scroll down to "IPC sockets (Linux only)")
>>
>> You may test that if you are using said --network=host switch.
>>
>>> I tried to understand what happen behind the scene but it is not easy
>>> for me.
>>> Hoping someone on this list can help.
>>
>> Containers are tricky, just as Ansible (as shown earlier on the list)
>> can be, when encumbered with false believes and/or misunderstandings.
>> Virtual machines may serve better wrt. insights for the later bare
>> metal deployments.
>>
>> --
>> Jan (Poki)
>> ___
>> Users mailing list: Users@clusterlabs.org
>>
