Re: [ClusterLabs] Pacemaker detail log directory permissions
On 30/04/19 07:55 +0200, Ulrich Windl wrote: Jan Pokorný schrieb am 29.04.2019 um 17:22 in Nachricht <20190429152200.ga19...@redhat.com>: >> On 29/04/19 14:58 +0200, Jan Pokorný wrote: >>> On 29/04/19 08:20 +0200, Ulrich Windl wrote: >>> Jan Pokorný schrieb am 25.04.2019 um 18:49 >>> in Nachricht <20190425164946.gf23...@redhat.com>: > I think the prime and foremost use case is that half of the actual > pacemaker daemons run as hacluster:haclient themselves, and it's > preferred for them to be not completely muted about what they do, > correct? :‑) I interesting aspect: So the commands write directly to one file concurrently? How's synchronization done. It's especially interesting with multiple threads. >>> >>> You don't believe your libc (fprintf in particular) to abide the >>> POSIX constraints, do you? :-) >>> All functions that reference (FILE *) objects, except those with names ending in _unlocked, shall behave as if they use flockfile() and funlockfile() internally to obtain ownership of these (FILE *) objects. >>> >>> > [https://pubs.opengroup.org/onlinepubs/9699919799/functions/flockfile.html] >>> Wouldn't it be much easier to use some socket-bases syslog like service that serializes all the messages to one file. Then the clients just need write access to the socket, not to the files. > > I agree that multiple threads in one thread have no problem using > printf(), but (at least in the buffered case) if multiple processes > write to the same file, that type of locking doesn't help much IMHO. Oops, you are right, I made a logical shortcut connecting flockfile(3) and flock(1), which was entirely unbacked. You are correct it would matter only amongst the threads, not otherwise unsynchronized processes. Sorry about the noise :-/ Shamefully, this rather important (nobody wants garbled log messages) aspect is in no way documented in libqb's context (it does not do any explicit locking on its own), especially since the length of the logged messages can go above the default of 512 B (in the upcoming libqb 2, IIUIC) ... and luckily, I was steering the direction to still stay modest and cap that on 4 kiB, even if for other reasons: https://github.com/ClusterLabs/libqb/pull/292#issuecomment-361745575 which still might be within Linux + typical FSs (ext4) boundaries to guarantee atomicity of an append (or maybe not even that, it all seems a gray area of any guarantees provided by the underlying system, inputs from the experts welcome). Anyway, ISTM that we should at the very least increase the buffer size for block buffering to the configured one (up to 4 kiB as mentioned) if BUFSIZ would be less, to prevent tainting this expected atomicity from the get-go. Anyway, correctness of the message durability is still dependent on the syslog implementation to behave correctly if syslog logging sink is enabled (holds for both corosync and pacemaker daemons), so file-based logging can still be considered optional, and e.g. for pacemaker, it should be possible to turn it off altogether in the future as mentioned (already possible with corosync, IIRC). Lowest level I/O a very complex and difficult area to reason about, sadly, we didn't do our homework at the very beginning, so problems are likely to snowball (incl. brain farts like the one I did above ... it might have been avoided if the contraints of libqb logging usage were documented properly). > (In clasical syslog there's _one_ process that receives the messages > and writes them to files. I can only remember HP-UX kernel messages > that were synchronized byte-wise, so if multiple cores emitted the > same message concurrently you go some funny messages like > "OpOpeeniingng"... ;_) Truly nasty to not have even somewhat atomic blocks (see block based buffering, which is a default for fprintf et al. unless the output stream equals stdout) that gets interleaved. >> FWIW, have just realized that by virtue of this, we have actually >> silently alleviated a hidden lock contention (against the realtime >> corosync process! but allegedly, it is not too verbose if nothing >> interesting happens unless set to be more verbose) since >> Pacemaker-2.0.0:l >> >> [...] >> Already disproved, thanks to Ulrich, sorry. >> But we can do better for those who would prefer to serialize >> multiple (per daemon) file outputs by hand (if ever at all, complete >> serialization can be found in the syslog sink, perhaps) as briefly >> touched on earlier in this thread[1]. >> >> [1] https://lists.clusterlabs.org/pipermail/users/2019-April/025709.html That would be a low-effort assuring the atomicity (except for spurious duplicate instances of the same daemon, but that shall be reduced to some extent with the fix of the recent important-ranked CVE). -- Jan (Poki) pgpLFZuT7bPe3.pgp Description: PGP signature ___ Manage your subs
Re: [ClusterLabs] Pacemaker detail log directory permissions
On 29/04/19 14:58 +0200, Jan Pokorný wrote: > On 29/04/19 08:20 +0200, Ulrich Windl wrote: > Jan Pokorný schrieb am 25.04.2019 um 18:49 > in Nachricht <20190425164946.gf23...@redhat.com>: >>> I think the prime and foremost use case is that half of the actual >>> pacemaker daemons run as hacluster:haclient themselves, and it's >>> preferred for them to be not completely muted about what they do, >>> correct? :‑) >> >> I interesting aspect: So the commands write directly to one file >> concurrently? How's synchronization done. It's especially >> interesting with multiple threads. > > You don't believe your libc (fprintf in particular) to abide the > POSIX constraints, do you? :-) > >> All functions that reference (FILE *) objects, except those with >> names ending in _unlocked, shall behave as if they use flockfile() >> and funlockfile() internally to obtain ownership of these (FILE *) >> objects. > > [https://pubs.opengroup.org/onlinepubs/9699919799/functions/flockfile.html] > >> Wouldn't it be much easier to use some socket-bases syslog like >> service that serializes all the messages to one file. Then the >> clients just need write access to the socket, not to the files. FWIW, have just realized that by virtue of this, we have actually silently alleviated a hidden lock contention (against the realtime corosync process! but allegedly, it is not too verbose if nothing interesting happens unless set to be more verbose) since Pacemaker-2.0.0: https://github.com/ClusterLabs/pacemaker/commit/b8075c86d35f3d37b0cbac86a8c90f1ac1091c33 Great! But we can do better for those who would prefer to serialize multiple (per daemon) file outputs by hand (if ever at all, complete serialization can be found in the syslog sink, perhaps) as briefly touched on earlier in this thread[1]. [1] https://lists.clusterlabs.org/pipermail/users/2019-April/025709.html -- Jan (Poki) pgpz1GguYMzqk.pgp Description: PGP signature ___ Manage your subscription: https://lists.clusterlabs.org/mailman/listinfo/users ClusterLabs home: https://www.clusterlabs.org/
Re: [ClusterLabs] Pacemaker detail log directory permissions
On 29/04/19 08:20 +0200, Ulrich Windl wrote: Jan Pokorný schrieb am 25.04.2019 um 18:49 in Nachricht <20190425164946.gf23...@redhat.com>: >> On 24/04/19 09:32 ‑0500, Ken Gaillot wrote: >>> On Wed, 2019‑04‑24 at 16:08 +0200, wf...@niif.hu wrote: Make install creates /var/log/pacemaker with mode 0770, owned by hacluster:haclient. However, if I create the directory as root:root instead, pacemaker.log appears as hacluster:haclient all the same. What breaks in this setup besides log rotation (which can be fixed by removing the su directive)? Why is it a good idea to let the haclient group write the logs? >>> >>> Cluster administrators are added to the haclient group. It's a >>> minor use case, but the group write permission allows such users >>> to run commands that log to the detail log. An example would be >>> running "crm_resource ‑‑force‑start" for a resource agent that >>> writes debug information to the log. >> >> I think the prime and foremost use case is that half of the actual >> pacemaker daemons run as hacluster:haclient themselves, and it's >> preferred for them to be not completely muted about what they do, >> correct? :‑) > > I interesting aspect: So the commands write directly to one file > concurrently? How's synchronization done. It's especially > interesting with multiple threads. You don't believe your libc (fprintf in particular) to abide the POSIX constraints, do you? :-) > All functions that reference (FILE *) objects, except those with > names ending in _unlocked, shall behave as if they use flockfile() > and funlockfile() internally to obtain ownership of these (FILE *) > objects. [https://pubs.opengroup.org/onlinepubs/9699919799/functions/flockfile.html] > Wouldn't it be much easier to use some socket-bases syslog like > service that serializes all the messages to one file. Then the > clients just need write access to the socket, not to the files. Two things: - pacemaker _already_ uses the syslog interface (in daemon context, anyway; it can disabled, but I think there should at the very least be a toggle to optionally disable file-based logging altogether) - it doesn't make the serialization problem any better (yes, it could perform some sort of buffered time-stamp based reordering on-the-fly, but that wouldn't make the situation any better than an explicit post-processing on the files) >> Indeed, users can configure whatever log routing they desire >> (I was actually toying with an idea to make it a lot more flexible, >> log‑per‑type‑of‑daemon and perhaps even distinguished by PID, >> configurable log formats since currently it's arguably a heavy >> overkill to keep the hostname stated repeatedly over and over >> without actually bothering to recheck it from time to time, etc.). > > Reminds me of a project I had started more than 10 years ago: I > wanted to start with a log mechanism, but the idea was rejected. > At some later time out local mail system was brought to a halt > by >30 mail messages being created within a few minutes. Most > GUI clients behave very badly once you have that many messages > But that's a different story in the subject... Agree in a sense that explicit message spooling is not a substitute for proper logging (as opposed to true alerts etc., but you can usually configure mechanisms so as to forward very _select_ subset of log messages like that in the syslog daemon of your choice). >> Also note, relying on almighty root privileges (like with the >> pristine deployment) is a silent misconception that cannot be taken >> for fully granted, so again arguably, even the root daemons should >> take a haclient group's coat on top of their own just in case [*]. > > See above. > >> >>> If ACLs are not in use, such users already have full read/write >>> access to the CIB, so being able to read and write the log is not an >>> additional concern. >>> >>> With ACLs, I could see wanting to change the permissions, and that idea >>> has come up already. One approach might be to add a PCMK_log_mode >>> option that would default to 0660, and users could make it more strict >>> if desired. > > With 0660 you actually mean "ug+w" (why care about read > permissions)? Octal absolute modes are obsolet for at least > 20 years... Note sure about this opinion :-) Luckily, chmod(1) still documents both. >> It looks reasonable to prevent read‑backs by anyone but root, that >> could be applied without any further toggles, assuming the pacemaker >> code won't flip once purposefully allowed read bits for group back >> automatically and unconditionally. > > See above. "gu+w" could actually be "a-x,ug+w,o-rw" (which ist still > different from 0660). Generally, I read it you seem to agree with the idea of mode updates as opposed to hard-coded modes, at least when it comes to run-time enforcement (there are more subtopics in parallel). >> >> >> [*] for instance when SELinux hits hard (which is curre
Re: [ClusterLabs] Pacemaker detail log directory permissions
On 26/04/19 09:33 -0500, Ken Gaillot wrote: > On Thu, 2019-04-25 at 18:49 +0200, Jan Pokorný wrote: >> On 24/04/19 09:32 -0500, Ken Gaillot wrote: >>> On Wed, 2019-04-24 at 16:08 +0200, wf...@niif.hu wrote: Make install creates /var/log/pacemaker with mode 0770, owned by hacluster:haclient. However, if I create the directory as root:root instead, pacemaker.log appears as hacluster:haclient all the same. What breaks in this setup besides log rotation (which can be fixed by removing the su directive)? Why is it a good idea to let the haclient group write the logs? >>> >>> Cluster administrators are added to the haclient group. It's a >>> minor use case, but the group write permission allows such users >>> to run commands that log to the detail log. An example would be >>> running "crm_resource --force-start" for a resource agent that >>> writes debug information to the log. >> >> I think the prime and foremost use case is that half of the actual >> pacemaker daemons run as hacluster:haclient themselves, and it's >> preferred for them to be not completely muted about what they do, >> correct? :-) > > The logs are owned by hacluster user, so the daemons don't have an > issue. Well, the it's not the particular file that's in question, but rather the directory in the traversal path to them. It can indeed create problems with 0770 mode for it while not owned with those particular non-root credentials (for non-root daemons or similarly, with the current ownership and root not having the DAC override permissions for root daemons as mentioned), so potentially there's a catch with the arrangement OP asks about, at least IIUIC. >> Indeed, users can configure whatever log routing they desire >> (I was actually toying with an idea to make it a lot more flexible, >> log-per-type-of-daemon and perhaps even distinguished by PID, >> configurable log formats since currently it's arguably a heavy >> overkill to keep the hostname stated repeatedly over and over >> without actually bothering to recheck it from time to time, etc.). >> >> Also note, relying on almighty root privileges (like with the >> pristine >> deployment) is a silent misconception that cannot be taken for fully >> granted, so again arguably, even the root daemons should take >> a haclient group's coat on top of their own just in case [*]. >> >>> If ACLs are not in use, such users already have full read/write >>> access to the CIB, so being able to read and write the log is not >>> an additional concern. >>> >>> With ACLs, I could see wanting to change the permissions, and that >>> idea has come up already. One approach might be to add a >>> PCMK_log_mode option that would default to 0660, and users could >>> make it more strict if desired. >> >> It looks reasonable to prevent read-backs by anyone but root, that >> could be applied without any further toggles, assuming the pacemaker >> code won't flip once purposefully allowed read bits for group back >> automatically and unconditionally. > > Pacemaker does indeed ensure the detail log has specific ownerships and > permissions -- see crm_add_logfile(). Yes, and that'd intefere with my idea; it would explicitly need _not_ to zero read bit for group once the possibly predestined default of not allowing that was overridden by the admin. Still more elegant than yet another toggle -- file system level "configuration" is the definite one without any need of intermediate levels just increasing complexity. -- Jan (Poki) pgp_Z6tLz9aGY.pgp Description: PGP signature ___ Manage your subscription: https://lists.clusterlabs.org/mailman/listinfo/users ClusterLabs home: https://www.clusterlabs.org/
Re: [ClusterLabs] Pacemaker detail log directory permissions
On Thu, 2019-04-25 at 18:49 +0200, Jan Pokorný wrote: > On 24/04/19 09:32 -0500, Ken Gaillot wrote: > > On Wed, 2019-04-24 at 16:08 +0200, wf...@niif.hu wrote: > > > Make install creates /var/log/pacemaker with mode 0770, owned by > > > hacluster:haclient. However, if I create the directory as > > > root:root > > > instead, pacemaker.log appears as hacluster:haclient all the > > > same. What breaks in this setup besides log rotation (which can > > > be > > > fixed by removing the su directive)? Why is it a good idea to > > > let > > > the haclient group write the logs? > > > > Cluster administrators are added to the haclient group. It's a > > minor > > use case, but the group write permission allows such users to run > > commands that log to the detail log. An example would be running > > "crm_resource --force-start" for a resource agent that writes debug > > information to the log. > > I think the prime and foremost use case is that half of the actual > pacemaker daemons run as hacluster:haclient themselves, and it's > preferred for them to be not completely muted about what they do, > correct? :-) The logs are owned by hacluster user, so the daemons don't have an issue. > > Indeed, users can configure whatever log routing they desire > (I was actually toying with an idea to make it a lot more flexible, > log-per-type-of-daemon and perhaps even distinguished by PID, > configurable log formats since currently it's arguably a heavy > overkill to keep the hostname stated repeatedly over and over > without actually bothering to recheck it from time to time, etc.). > > Also note, relying on almighty root privileges (like with the > pristine > deployment) is a silent misconception that cannot be taken for fully > granted, so again arguably, even the root daemons should take > a haclient group's coat on top of their own just in case [*]. > > > If ACLs are not in use, such users already have full read/write > > access to the CIB, so being able to read and write the log is not > > an > > additional concern. > > > > With ACLs, I could see wanting to change the permissions, and that > > idea > > has come up already. One approach might be to add a PCMK_log_mode > > option that would default to 0660, and users could make it more > > strict > > if desired. > > It looks reasonable to prevent read-backs by anyone but root, that > could be applied without any further toggles, assuming the pacemaker > code won't flip once purposefully allowed read bits for group back > automatically and unconditionally. Pacemaker does indeed ensure the detail log has specific ownerships and permissions -- see crm_add_logfile(). > > [*] for instance when SELinux hits hard (which is currently not the > case for Fedora/EL family), even though the executor(s) would > need > to be exempted if process inheritance taints the tree once > forever: > https://danwalsh.livejournal.com/69478.html -- Ken Gaillot ___ Manage your subscription: https://lists.clusterlabs.org/mailman/listinfo/users ClusterLabs home: https://www.clusterlabs.org/
Re: [ClusterLabs] Pacemaker detail log directory permissions
On 24/04/19 09:32 -0500, Ken Gaillot wrote: > On Wed, 2019-04-24 at 16:08 +0200, wf...@niif.hu wrote: >> Make install creates /var/log/pacemaker with mode 0770, owned by >> hacluster:haclient. However, if I create the directory as root:root >> instead, pacemaker.log appears as hacluster:haclient all the >> same. What breaks in this setup besides log rotation (which can be >> fixed by removing the su directive)? Why is it a good idea to let >> the haclient group write the logs? > > Cluster administrators are added to the haclient group. It's a minor > use case, but the group write permission allows such users to run > commands that log to the detail log. An example would be running > "crm_resource --force-start" for a resource agent that writes debug > information to the log. I think the prime and foremost use case is that half of the actual pacemaker daemons run as hacluster:haclient themselves, and it's preferred for them to be not completely muted about what they do, correct? :-) Indeed, users can configure whatever log routing they desire (I was actually toying with an idea to make it a lot more flexible, log-per-type-of-daemon and perhaps even distinguished by PID, configurable log formats since currently it's arguably a heavy overkill to keep the hostname stated repeatedly over and over without actually bothering to recheck it from time to time, etc.). Also note, relying on almighty root privileges (like with the pristine deployment) is a silent misconception that cannot be taken for fully granted, so again arguably, even the root daemons should take a haclient group's coat on top of their own just in case [*]. > If ACLs are not in use, such users already have full read/write > access to the CIB, so being able to read and write the log is not an > additional concern. > > With ACLs, I could see wanting to change the permissions, and that idea > has come up already. One approach might be to add a PCMK_log_mode > option that would default to 0660, and users could make it more strict > if desired. It looks reasonable to prevent read-backs by anyone but root, that could be applied without any further toggles, assuming the pacemaker code won't flip once purposefully allowed read bits for group back automatically and unconditionally. [*] for instance when SELinux hits hard (which is currently not the case for Fedora/EL family), even though the executor(s) would need to be exempted if process inheritance taints the tree once forever: https://danwalsh.livejournal.com/69478.html -- Jan (Poki) pgp3dAgFvEUfh.pgp Description: PGP signature ___ Manage your subscription: https://lists.clusterlabs.org/mailman/listinfo/users ClusterLabs home: https://www.clusterlabs.org/
Re: [ClusterLabs] Pacemaker detail log directory permissions
On Wed, 2019-04-24 at 16:08 +0200, wf...@niif.hu wrote: > Hi, > > Make install creates /var/log/pacemaker with mode 0770, owned by > hacluster:haclient. However, if I create the directory as root:root > instead, pacemaker.log appears as hacluster:haclient all the > same. What > breaks in this setup besides log rotation (which can be fixed by > removing the su directive)? Why is it a good idea to let the > haclient > group write the logs? Cluster administrators are added to the haclient group. It's a minor use case, but the group write permission allows such users to run commands that log to the detail log. An example would be running "crm_resource --force-start" for a resource agent that writes debug information to the log. If ACLs are not in use, such users already have full read/write access to the CIB, so being able to read and write the log is not an additional concern. With ACLs, I could see wanting to change the permissions, and that idea has come up already. One approach might be to add a PCMK_log_mode option that would default to 0660, and users could make it more strict if desired. -- Ken Gaillot ___ Manage your subscription: https://lists.clusterlabs.org/mailman/listinfo/users ClusterLabs home: https://www.clusterlabs.org/
[ClusterLabs] Pacemaker detail log directory permissions
Hi, Make install creates /var/log/pacemaker with mode 0770, owned by hacluster:haclient. However, if I create the directory as root:root instead, pacemaker.log appears as hacluster:haclient all the same. What breaks in this setup besides log rotation (which can be fixed by removing the su directive)? Why is it a good idea to let the haclient group write the logs? -- Thanks, Feri ___ Manage your subscription: https://lists.clusterlabs.org/mailman/listinfo/users ClusterLabs home: https://www.clusterlabs.org/
