Hello Heiner, just a heads up for you and the other storage admins, regularly cleaning up /tmp, regarding one aspect to keep in mind:
- If you are using Spectrum Scale software call home (mmcallhome), it would be using the directory ${dataStructureDump}/callhome to save the copies of the uploaded data. This would be /tmp/mmfs/callhome/ in your case, which you would be automatically regularly removing. - These copies are used by one of the features of call home: "mmcallhome status diff" - This feature allows to see an overview of the Spectrum Scale configuration changes, that occurred between 2 different points in time. - This effectively allows to quickly find out if any config changes occurred prior to an outage, thereby helping to find the root cause of self-caused problems in the Scale cluster. - It was added in Scale 5.0.5.0 See IBM KC for more details: https://www.ibm.com/docs/en/spectrum-scale/5.1.0?topic=cch-use-cases-detecting-system-changes-by-using-mmcallhome-command - As a source of the "config snapshots", mmcallhome status diff is using the DC packages inside of ${dataStructureDump}/callhome, which you would be regularly deleting, thereby hugely reducing the usability of this particular feature. - Of course, software call home automatically makes sure, it will not use too much space in dataStructureDump and it automatically removes the oldest entries, keeping at most 2GB or 300 files inside (default values, configurable). Mit freundlichen Grüßen / Kind regards Pavel Safre Software Engineer IBM Systems Group, IBM Spectrum Scale Development Dept. M925 Phone: IBM Deutschland Research & Development GmbH Email: psa...@de.ibm.com Wilhelm-Fay-Straße 32 65936 Frankfurt am Main IBM Data Privacy Statement IBM Deutschland Research & Development GmbH / Vorsitzender des Aufsichtsrats: Gregor Pillen Geschäftsführung: Dirk Wittkopp Sitz der Gesellschaft: Böblingen / Registergericht: Amtsgericht Stuttgart, HRB 243294 From: "Billich Heinrich Rainer (ID SD)" <heinrich.bill...@id.ethz.ch> To: "gpfsug main discussion list" <gpfsug-discuss@spectrumscale.org> Date: 16.11.2021 17:44 Subject: [EXTERNAL] Re: [gpfsug-discuss] /tmp/mmfs vanishes randomly? Sent by: gpfsug-discuss-boun...@spectrumscale.org Hello Olaf, Thank you, you are right. I was ignorant about the systemd-tmpfiles* services and timers. The cleanup in /tmp wasn’t present in RHEL7, at least not on our nodes. I consider to modify the configuration a bit to keep the directory /tmp/mmfs - or even create it – but to clean it’s content . Best regards, Heiner From: <gpfsug-discuss-boun...@spectrumscale.org> on behalf of Olaf Weiser <olaf.wei...@de.ibm.com> Reply to: gpfsug main discussion list <gpfsug-discuss@spectrumscale.org> Date: Monday, 8 November 2021 at 10:53 To: "gpfsug-discuss@spectrumscale.org" <gpfsug-discuss@spectrumscale.org> Cc: "gpfsug-discuss@spectrumscale.org" <gpfsug-discuss@spectrumscale.org> Subject: Re: [gpfsug-discuss] /tmp/mmfs vanishes randomly? Hallo Heiner, multiple levels of answers.. (1st) ... it the directory is not there, the gpfs trace would create it automatically - just like this: [root@ess5-ems1 ~]# ls -l /tmp/mmfs ls: cannot access '/tmp/mmfs': No such file or directory [root@ess5-ems1 ~]# mmtracectl --start -N ems5k.mmfsd.net mmchconfig: Command successfully completed mmchconfig: Propagating the cluster configuration data to all affected nodes. This is an asynchronous process. [root@ess5-ems1 ~]# [root@ess5-ems1 ~]# [root@ess5-ems1 ~]# ls -l /tmp/mmfs total 0 -rw-r--r-- 1 root root 0 Nov 8 10:47 lxtrace.trcerr.ems5k [root@ess5-ems1 ~]# (2nd) I think - the cleaning of /tmp is something done by the OS - please check - systemctl status systemd-tmpfiles-setup.service or look at this config file [root@ess5-ems1 ~]# cat /usr/lib/tmpfiles.d/tmp.conf # This file is part of systemd. # # systemd is free software; you can redistribute it and/or modify it # under the terms of the GNU Lesser General Public License as published by # the Free Software Foundation; either version 2.1 of the License, or # (at your option) any later version. # See tmpfiles.d(5) for details # Clear tmp directories separately, to make them easier to override q /tmp 1777 root root 10d q /var/tmp 1777 root root 30d # Exclude namespace mountpoints created with PrivateTmp=yes x /tmp/systemd-private-%b-* X /tmp/systemd-private-%b-*/tmp x /var/tmp/systemd-private-%b-* X /var/tmp/systemd-private-%b-*/tmp # Remove top-level private temporary directories on each boot R! /tmp/systemd-private-* R! /var/tmp/systemd-private-* [root@ess5-ems1 ~]# hope this helps - cheers Mit freundlichen Grüßen / Kind regards Olaf Weiser IBM Systems, SpectrumScale Client Adoption ------------------------------------------------------------------------------------------------------------------------------------------- IBM Deutschland IBM Allee 1 71139 Ehningen Phone: +49-170-579-44-66 E-Mail: olaf.wei...@de.ibm.com ------------------------------------------------------------------------------------------------------------------------------------------- IBM Deutschland GmbH / Vorsitzender des Aufsichtsrats: Martin Jetter Geschäftsführung: Gregor Pillen (Vorsitzender), Agnes Heftberger, Norbert Janzen, Markus Koerner, Christian Noll, Nicole Reimer Sitz der Gesellschaft: Ehningen / Registergericht: Amtsgericht Stuttgart, HRB 14562 / WEEE-Reg.-Nr. DE 99369940 ----- Ursprüngliche Nachricht ----- Von: "Billich Heinrich Rainer (ID SD)" <heinrich.bill...@id.ethz.ch> Gesendet von: gpfsug-discuss-boun...@spectrumscale.org An: "gpfsug main discussion list" <gpfsug-discuss@spectrumscale.org> CC: Betreff: [EXTERNAL] [gpfsug-discuss] /tmp/mmfs vanishes randomly? Datum: Mo, 8. Nov 2021 10:35 Hello, We use /tmp/mmfs as dataStructureDump directory. Since a while I notice that this directory randomly vanishes. Mmhealth does not complain but just notes that it will no longer monitor the directory. Still I doubt that trace collection and similar will create the directory when needed? Do you know of any spectrum scale internal mechanism that could cause /tmp/mmfs to get deleted? It happens on ESS nodes, with a plain IBM installation, too. It happens just on one or two nodes at a time, it's no cluster-wide cleanup or similar. We run scale 5.0.5 and ESS 6.0.2.2 and 6.0.2.2. Thank you, Mmhealth message: local_fs_path_not_found INFO The configured dataStructureDump path /tmp/mmfs does not exists. Skipping monitoring. Kind regards, Heiner --- ======================= Heinrich Billich ETH Zürich Informatikdienste Tel.: +41 44 632 72 56 heinrich.bill...@id.ethz.ch ======================== _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss
_______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss