Re: Howto do "a detailed cleanup with the aid of the sysclean package"?
Look, can we all just agree it is just a question of adding appropriate verbiage to the port/package, sticking a huge CAVEAT on the documentation leading to it, and go back to writing actual code ?
Re: Howto do "a detailed cleanup with the aid of the sysclean package"?
On 2022-05-04, Theo de Raadt wrote: > I have also pointed out a couple of times now that sysclean ignores the > lessons of "find -print0" and "xargs -0", and I worry it could find a > file called > > "/somewhere/matchingpattern/\n/etc/spwd.db" Thus is easily fixed by adding a "delete" mode which removes the files internally in sysclean.
Re: Howto do "a detailed cleanup with the aid of the sysclean package"?
To Nobody in particular: Confucius is attributed with stating: "The beginning of wisdom is to call things by their proper name" I suggest that "sysclean" is not the name of the code inside this utility. The sysupgrade utility actually upgrades my system and fw_update updates my firmware. sysclean does not clean anything - it makes suggestions for a specific type of user (not really all users). It may have aspirations of sysclean(ing), but it is not there today and maybe should have a different name. Based on this conversation thread, any suggestion I might make would sound humorous, so I will defer possible names to others more invested in the code The sysupgrade utility has different modes of operation with the -s option. Maybe the final sysclean will have a similar option? I do not use sysclean. After reading this thread it appears I am correct to not use it because I'm not running from snapshots (and don't have as much to clean). When hearing discussion about sysclean I felt like an outsider for not using it, along with sysupgrade. Now, I better understand sysclean's intended purpose. Clearly sysclean is a difficult task and if anyone could produce a version I would use (some day) it would be this group! 0.02 - Thanks for reading On 5/4/22 07:36, Theo de Raadt wrote: > Sebastien Marie wrote: > >> a package could use old libraries, and such libraries will not be listed by >> sysclean. > the sysclean manual page claims that it correctly identifies "obsolete > filenames". > > Obsolete, adj. > > 1.no longer produced or used; out of date. > > But this is innaccurate. By your own admission, the test it performs to > decide on whether a file is "not used" is flawed. > > Yet, people continue to use rm. > >> yes it will. but as sysclean only inspects files under directories >> controlled by >> the admin, it means that the administrator created such files and so they >> know >> what it is doing. > The "controlled by admin" file does not exist by default, so normally this > will look in a lot of system locations, and falsely identify unused files. > > Let me be clear: the program is lying to the user. It is documented vaguely > to hide that what the program does is not truthful. It says "obsolete" all > over the place, but no actual test for that condition is performed. > >>> And then someone will rm -f `sysclean`. >> sysclean isn't designed for such usage. > Yet, that is precisely what numerous people have done. > >> I could saying the same about 'ls'. Someone will rm -f `ls` and a file named >> "/somewhere/matchingpattern/\n/etc/spwd.db" will do bad thing. > Yet, noone is doing that. > >> Should we add -0 to ls ? or remove it because of possible stupid usage ? >> >>> I think sysclean is below the normal standard for our group. >> Yes. ls too. it could hurt users which might call rm -f `ls`. > > Clear you don't care that people are getting hurt by this code you wrote. >
Re: Howto do "a detailed cleanup with the aid of the sysclean package"?
How about someone simply (better than I) updating the manual page and pkg_info output with warnings and clarifications about the intended use case and risks? Maybe the man page could say (just as an idea): "Warning: this is intended to suggest files for removal, that it guesses are obsolete; if you build software locally or modify other system files, it may mistakenly suggest removal of libraries or other files that you want to keep. It should only be used by those who are familiar with their system and accept the risks involved [and maybe?: or who know for certain they have not changed anything ... in the directories it checks.]" Followed possibly by something like: "It has not (yet) applied the lessons of "find -print0" and "xargs -0", and could recommend removing something you don't expect: review carefully its output before proceeding with file deletions." And in the man page and pkg_info output, wherever "obsolete" is found, something like: < [...] It only reports obsolete > [...] It only reports *possibly* obsolete [...]; see the manual page for > more information. And similarly in the upgrade guide, if it were mentioned there in the future. Another possible step could be making it emit a concise message about the risk and intent, when it runs, if there is a practical way in a comment near the beginning of the output without breaking its function; or comment the output with that included so users have to think before they can use it (passing a switch or something). With some of those, maybe it can remain in the learnable flow, for new or maturing users. Just thoughts; if not helpful, forgive the noise.
Re: Howto do "a detailed cleanup with the aid of the sysclean package"?
On Wed, May 04, 2022 at 09:54:26AM -0600, Theo de Raadt wrote: > Marc Espie wrote: > > > All the horrors stories I've seen in this discussion are related > > to people trusting it blindly/automatically. > > But why wouldn't people trust it? > > All the documentation claims it produces a list of files that is obsolete. > > It says those files are obsolete & unused -- with such authority! > > Nowhere does it say "Oh, but these obsolete files may still be used" > > Why is it blind for people to run rm? Are they stupidly making assumptions > based upon the documentation? diff are accepted. -- Sebastien Marie
Re: Howto do "a detailed cleanup with the aid of the sysclean package"?
Marc Espie wrote: > All the horrors stories I've seen in this discussion are related > to people trusting it blindly/automatically. But why wouldn't people trust it? All the documentation claims it produces a list of files that is obsolete. It says those files are obsolete & unused -- with such authority! Nowhere does it say "Oh, but these obsolete files may still be used" Why is it blind for people to run rm? Are they stupidly making assumptions based upon the documentation?
Re: Howto do "a detailed cleanup with the aid of the sysclean package"?
On Wed, May 04, 2022 at 07:45:32AM -0400, Raul Miller wrote: > On Wed, May 4, 2022 at 4:15 AM Sebastien Marie wrote: > > The main problem I am seeing would be maintaining such lists, and it > > necessary > > means manual addition to add only "safe" files to remove (no libraries at > > least). > > Conceptually speaking, it's possible to track library dependencies, > and it's possible to build tools for doing so, and it's possible to > automate this. ldd already does a lot of the work here, and a > conceptual sysclean replacement might provide toolchain and/or install > time and/or backup time support for tracking executable locations. > (Conceptually, this kind of tracking would use some directory > structure to support independent updates, and would itself be subject > to [careful] cleanup. Some small redundancy here would probably be > fine.) It is not that simple. Ports has such a tool for library tracking and automatic generation of WANTLIB. There are lots of caveats - it's slow, - taking care of library paths is complicated, - taking care of library paths is ambiguous, - some modern software prefer dlopen. (We have a handful of ports that have WANTLIB that are not statically detected. Some modern stuff in qt and gnome land will tend to dlopen some specific components like database drivers and crypto libraries, in order to be able to use it if it's installed or not if it's not. I *think*, from what Antoine told me, that there's even a generic framework somewhere in gnome that tries to use this all the time instead of classic dynamic linking). That said, I value the information sysclean gives me... but then I clean my system manually regularly, and most often I know exactly what I'm getting rid of. All the horrors stories I've seen in this discussion are related to people trusting it blindly/automatically.
Re: Howto do "a detailed cleanup with the aid of the sysclean package"?
Sebastien Marie wrote: > a package could use old libraries, and such libraries will not be listed by > sysclean. the sysclean manual page claims that it correctly identifies "obsolete filenames". Obsolete, adj. 1.no longer produced or used; out of date. But this is innaccurate. By your own admission, the test it performs to decide on whether a file is "not used" is flawed. Yet, people continue to use rm. > yes it will. but as sysclean only inspects files under directories controlled > by > the admin, it means that the administrator created such files and so they > know > what it is doing. The "controlled by admin" file does not exist by default, so normally this will look in a lot of system locations, and falsely identify unused files. Let me be clear: the program is lying to the user. It is documented vaguely to hide that what the program does is not truthful. It says "obsolete" all over the place, but no actual test for that condition is performed. > > And then someone will rm -f `sysclean`. > > sysclean isn't designed for such usage. Yet, that is precisely what numerous people have done. > I could saying the same about 'ls'. Someone will rm -f `ls` and a file named > "/somewhere/matchingpattern/\n/etc/spwd.db" will do bad thing. Yet, noone is doing that. > Should we add -0 to ls ? or remove it because of possible stupid usage ? > > > I think sysclean is below the normal standard for our group. > > Yes. ls too. it could hurt users which might call rm -f `ls`. Clear you don't care that people are getting hurt by this code you wrote.
Re: Howto do "a detailed cleanup with the aid of the sysclean package"?
On Wed, May 04, 2022 at 08:03:14AM -0600, Theo de Raadt wrote: > Sebastien Marie wrote: > > > semarie@ spoke about integrating some elements inside the installer when he > > was > > about "clean _other things_". It isn't about "stepping back". Even if the > > installer would clean all it is possible to remove safely, I would still > > use a > > program to list libraries without registered packages using them. > > There is nothing you can do in the installer to solve this problem of > deleting old libraries. Old libraries MUST STAY, because sysclean does > not traverse and inspect all files in the entire filesystem to conclude > there are no use of them. I am differenciate what the installer could do, and what I could do. Old libraries ON MY CONTROLLED SYSTEMS will be removed, because I KNOW that they could be. I would not expect the installer to do that (because general case isn't my use case). > > I wrote sysclean because it solves *my* problems regarding maintaining a > > system, > > and I shared it because it could help *some* others people. It wasn't > > created > > with intent to solve the general use case for all possibles users. > > The pkg_info blob says: > > sysclean is a script designed to help remove obsolete files between > OpenBSD > upgrades. > > But sysclean can help people remove non-obsolete files. > > sysclean does not remove any files on the system. It only reports obsolete > filenames or packages using out-of-date libraries. > > The sysclean report includes files without doing a comprehensive search to > validate that they are NOT IN USE. They are not obsolute, yet they show up > in the list. it does, but only in some extent: it only considere base system and packages. a package could use old libraries, and such libraries will not be listed by sysclean. > > You don't like it, fine. Just don't use it. > > This conversation is happening because sysclean makes it too easy for > people who don't understand the complete system sufficiently, to then > use rm, and break their system. > > You can make that claim of "Don't use it" against me all day long, but > people will keep discovering sysclean and potentially using it to break > their systems. > > I have also pointed out a couple of times now that sysclean ignores the > lessons of "find -print0" and "xargs -0", and I worry it could find a > file called > > "/somewhere/matchingpattern/\n/etc/spwd.db" > > Pehraps it won't match such patterns? I suspec it will. yes it will. but as sysclean only inspects files under directories controlled by the admin, it means that the administrator created such files and so they know what it is doing. > And then someone will rm -f `sysclean`. sysclean isn't designed for such usage. I could saying the same about 'ls'. Someone will rm -f `ls` and a file named "/somewhere/matchingpattern/\n/etc/spwd.db" will do bad thing. Should we add -0 to ls ? or remove it because of possible stupid usage ? > I think sysclean is below the normal standard for our group. Yes. ls too. it could hurt users which might call rm -f `ls`. -- Sebastien Marie
Re: Howto do "a detailed cleanup with the aid of the sysclean package"?
Sebastien Marie wrote: > semarie@ spoke about integrating some elements inside the installer when he > was > about "clean _other things_". It isn't about "stepping back". Even if the > installer would clean all it is possible to remove safely, I would still use > a > program to list libraries without registered packages using them. There is nothing you can do in the installer to solve this problem of deleting old libraries. Old libraries MUST STAY, because sysclean does not traverse and inspect all files in the entire filesystem to conclude there are no use of them. > I wrote sysclean because it solves *my* problems regarding maintaining a > system, > and I shared it because it could help *some* others people. It wasn't created > with intent to solve the general use case for all possibles users. The pkg_info blob says: sysclean is a script designed to help remove obsolete files between OpenBSD upgrades. But sysclean can help people remove non-obsolete files. sysclean does not remove any files on the system. It only reports obsolete filenames or packages using out-of-date libraries. The sysclean report includes files without doing a comprehensive search to validate that they are NOT IN USE. They are not obsolute, yet they show up in the list. > You don't like it, fine. Just don't use it. This conversation is happening because sysclean makes it too easy for people who don't understand the complete system sufficiently, to then use rm, and break their system. You can make that claim of "Don't use it" against me all day long, but people will keep discovering sysclean and potentially using it to break their systems. I have also pointed out a couple of times now that sysclean ignores the lessons of "find -print0" and "xargs -0", and I worry it could find a file called "/somewhere/matchingpattern/\n/etc/spwd.db" Pehraps it won't match such patterns? I suspec it will. And then someone will rm -f `sysclean`. I think sysclean is below the normal standard for our group.
Re: Howto do "a detailed cleanup with the aid of the sysclean package"?
On Wed, May 04, 2022 at 06:20:59AM -0600, Theo de Raadt wrote: > > Users are capable of creating linking against older lib*.so.* files. > Such binaries could be anywhere on-disk, and we should not walk the entire > disk to find them. Therefore such old libraries should NEVER be deleted. > > It isn't that track as been lost. The linker isn't going to create a log > file about what lib* files have been used at compile time. > > New OpenBSD snapshots come with new libraries, because we crank > major.minor for incompatible changes, and there are some who think they > can delete the old libraries. > > Such old libraries should NEVER be deleted. yes. it is why it isn't a simple problem in the general case. some users could rely on old files. > sysclean wasn't designed to delete these libraries, but these libraries > are the biggest win as far as diskspace, so the selection algorithm > started hoovering them up, and it seems semarie doesn't want to walk > back "OK, I can clean _other things_ but I must leave the libaries alone". > > Because then I guess then sysclean doesn't feel like such a winning > proposition at all... semarie@ spoke about integrating some elements inside the installer when he was about "clean _other things_". It isn't about "stepping back". Even if the installer would clean all it is possible to remove safely, I would still use a program to list libraries without registered packages using them. I wrote sysclean because it solves *my* problems regarding maintaining a system, and I shared it because it could help *some* others people. It wasn't created with intent to solve the general use case for all possibles users. My use case is I don't build local program without packages (and I created some "local" packages for personal stuff). So I know I could remove old libraries. I know the systems I am maintaing, so I know what I could remove or not. You don't like it, fine. Just don't use it. And I recall it comes with a ISC license: # Permission to use, copy, modify, and distribute this software for any # purpose with or without fee is hereby granted, provided that the above # copyright notice and this permission notice appear in all copies. # # THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES # WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF # MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR # ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES # WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN # ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF # OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE. Thanks. -- Sebastien Marie
Re: Howto do "a detailed cleanup with the aid of the sysclean package"?
Harald Dunkel wrote: > Hi folks, > > I think the main problem is pretty easy to describe: OpenBSD loses track > about what it had installed and cannot clean up its own files on a system > upgrade. No, that is incorrect. Users are capable of creating linking against older lib*.so.* files. Such binaries could be anywhere on-disk, and we should not walk the entire disk to find them. Therefore such old libraries should NEVER be deleted. It isn't that track as been lost. The linker isn't going to create a log file about what lib* files have been used at compile time. New OpenBSD snapshots come with new libraries, because we crank major.minor for incompatible changes, and there are some who think they can delete the old libraries. Such old libraries should NEVER be deleted. sysclean wasn't designed to delete these libraries, but these libraries are the biggest win as far as diskspace, so the selection algorithm started hoovering them up, and it seems semarie doesn't want to walk back "OK, I can clean _other things_ but I must leave the libaries alone". Because then I guess then sysclean doesn't feel like such a winning proposition at all...
Re: Howto do "a detailed cleanup with the aid of the sysclean package"?
On Wed, May 4, 2022 at 4:15 AM Sebastien Marie wrote: > The main problem I am seeing would be maintaining such lists, and it necessary > means manual addition to add only "safe" files to remove (no libraries at > least). Conceptually speaking, it's possible to track library dependencies, and it's possible to build tools for doing so, and it's possible to automate this. ldd already does a lot of the work here, and a conceptual sysclean replacement might provide toolchain and/or install time and/or backup time support for tracking executable locations. (Conceptually, this kind of tracking would use some directory structure to support independent updates, and would itself be subject to [careful] cleanup. Some small redundancy here would probably be fine.) That said, I don't personally have a use for this approach -- I lean towards the "build a new image, and shut down the old one (ideally hanging onto it for a month, in case there's problems)" mechanism for cleanup. Thanks, -- Raul
Re: Howto do "a detailed cleanup with the aid of the sysclean package"?
On Wed, May 04, 2022 at 07:42:54AM +0200, Harald Dunkel wrote: > Hi folks, > > I think the main problem is pretty easy to describe: OpenBSD loses track > about what it had installed and cannot clean up its own files on a system > upgrade. technically we are already tracking which files were installed or removed (see src/distrib/sets/lists files which are under cvs). but we can't just decide to delete any old files at upgrade time: user might still use them. the simpler example is libraries: at upgrade time, your packages are still "old" packages which might be still using "old" libraries. if the "still used" library is removed before you update your packages, the program will not run... if the program is your window-manager, a package providing a login_* method, or some server used to auth your users, you start having big problems. For now, I am thinking that *some* files might be removable at upgrade time. A naive approch would be to remove whole directories (assuming the content is fully owned by the system (like /usr/include or /usr/share/man), which might be hazardous assumption in the general case). but if the upgrades doesn't work/finish for some reason, the system might not be as usable as expected (which is bad). Removing only (some) "old" files means having a list of all files previously installed-but-no-more in the installer. and it might be a different list by arch (but it shouldn't hurt to try to remove /usr/include/g++/alpha-unknown-openbsd7.0/bits/atomic_word.h on amd64). Some numbers: since 2021-10-31 (last 6 months, about 1 release), I counted 369 files removed (only from src, not accounting xenocara). But some files were installed, next removed, and next reinstalled, so the list should be revisited (I only counted removed files). The main problem I am seeing would be maintaining such lists, and it necessary means manual addition to add only "safe" files to remove (no libraries at least). Currently, sets files are mostly maintained by deraadt@ (>70% of the commits). And I don't want to add more work on his side for that. -- Sebastien Marie
Re: Howto do "a detailed cleanup with the aid of the sysclean package"?
On 2022-05-04 07:42 +02, Harald Dunkel wrote: > Hi folks, > > I think the main problem is pretty easy to describe: OpenBSD loses track > about what it had installed and cannot clean up its own files on a system > upgrade. The general case is a hard problem and people who deemed this important have not stepped up to implement the perfect solution that works for everybody. > > > Regards > Harri > -- I'm not entirely sure you are real.
Re: Howto do "a detailed cleanup with the aid of the sysclean package"?
Hi folks, I think the main problem is pretty easy to describe: OpenBSD loses track about what it had installed and cannot clean up its own files on a system upgrade. Regards Harri
Re: Howto do "a detailed cleanup with the aid of the sysclean package"?
On 2022-04-20 21:25:49, Ryan Kavanagh wrote: On Wed, Apr 20, 2022 at 08:39:09PM +0200, Harald Dunkel wrote: sysclean lists 4180 files and directories on my home server Could you please elaborate how sysclean is going to help me to keep my openbsd hosts clean? How is the usage model of this tool? Here's what I do: 1) List all of the directories or files I want sysclean to ignore in /etc/sysclean.ignore (format is documented in sysclean(8)). Got that. 2) Run "sysclean" to list all files that are obsolete. Check. 3) Manually review the output. If it contains files that are not obsolete, goto 1. Too many files to be a practical approach. If I would know each and every file to keep or to throw away, then I don't need sysclean. Not to mention that an important file or directory for the current release might become obsolete in a future release. Maintaining sysclean.ignore is unsustainable. You have to start from scratch with each release for each host running OpenBSD. Thats a lot of error-prone work. 4) Delete the files / directories listed in sysclean's output. Won't do. Regards Harri
Re: Howto do "a detailed cleanup with the aid of the sysclean package"?
Stuart Henderson wrote: > > Btw. there is another school of thought that says old cruft doesn't need > > to be removed, it's not causing any harm. If you need a clean system > > just reinstall and restore config and data from backups. It's a good > > excercise to check that your backups are working. > > I disagree, it can cause harm sometimes. Not least when you run out of > space in /usr halfway through untarring sets during an update... That is a false history. Many people got burned because we used to make /usr a certain size, then we switched to clang. But there were 2-3 other cases of this happening, where something quite big was added to /usr Most users upgrade release-to-release, and will only accumulate a few large lib*.so.*.* files. The upgrade process cleans out old clang/gcc leftovers. So there isn't much growth. If those users upgrade via snapshots, then the library collections can become a problem. I think it is dangerous to push our users to use sysclean. Many of them newbies. There are people who review the data, then rm -rf `sysclean' or sysclean | xargs rm -rf About about paths containing \n ? The lessons of find -print0 forgotten so easily? What about TOCTOU concerns, there is no mention of the multiuser possibility that someone can create a path that plays with the \n concern above. I think the program is misdesigned. It attacks files which are not in the current upgrade set, rather than attacking files which were in previous upgrades. It attacks non-OpenBSD files. It traverses directories which have nothing to do with the OpenBSD system itself. It attacks files which OpenBSD never supplied. The manual page does not document the heuristic it will use, so that people can decide "whoa, that will screw me because of how I use my directories". It is quite simply unable to make correct decisions perhaps because there isn't correct information to based this upon, and then there is this magic leap that people won't use one of the two approaches above and after doing so they will have a non-reversable problem. Actually probably the biggest problems are the (1) vagueness of the manual page, assuming people will do the right thing, and (2) people pushing (for newbies) to use this on social media. The manual page doesn't tell people to remove. But oh, here is a list, of what you should remove. But we didn't tell you to remove anything. Isn't that a bit deceitful? I've heard of experts misusing sysclean, so I very much suspect there are many users who have misused it, destroyed their system, and been to shy to complain.
Re: Howto do "a detailed cleanup with the aid of the sysclean package"?
On 2022-04-21, Florian Obser wrote: > On 2022-04-20 21:42 UTC, Stuart Henderson wrote: >> On 2022-04-20, Florian Obser wrote: >>> You will need a carefully curated /etc/sysclean.ignore file. >>> >>> You decided to put maildirs somewhere on the system, sysclean is not >>> omniscient, you need to tell it to leave them alone. Same with .git >>> directories. >>> I don't recall needing to tell it about package config files though, that's >>> a bit weird. >> >> e.g. files which are added to /etc that aren't distributed in the package but >> you create yourself > > Ah, yes. But it does understand directories, e.g. I have a lot of > changes in /etc/icinga2 but I don't need to ignore it. I guess that's > why I only need to ignore very little in /etc. > >> >>> It's a bit daunting on first run if a lot of cruft has accumulated >>> over the years, but it gets better. I'm using it for years, and I >>> can't recall the last time I had to add anything to the ignore file. >>> >>> I run it from daily and also by hand after every upgrade to a snapshot. >>> >>> If it outputs a really long list I cleanup incrementally, for example: >>> sysclean | fgrep /usr >> >> For a first run I would review "| fgrep /usr/local" as that's the most likely >> place where files might exist that should not be cleaned, and it's >> easier to > > tzk tzk, someone has been naughty and installed things without packages? > ;) I don't, but if someone has a machine with layer of accumulated cruft and they do local build/installs then that's a good way to find them. > I don't do that and I imagine if one installs compiled, dynamically > linked programs by hand sysclean's returns deminish really fast because > it won't understand that old libs are still needed. > > It's an awesome tool that takles a hard problem and for me succeedes > with a bit of hand holding. Me too. > Btw. there is another school of thought that says old cruft doesn't need > to be removed, it's not causing any harm. If you need a clean system > just reinstall and restore config and data from backups. It's a good > excercise to check that your backups are working. I disagree, it can cause harm sometimes. Not least when you run out of space in /usr halfway through untarring sets during an update...
Re: Howto do "a detailed cleanup with the aid of the sysclean package"?
On Thu, Apr 21, 2022 at 08:24:31AM +0200, Florian Obser wrote: > On 2022-04-20 21:42 UTC, Stuart Henderson wrote: > > On 2022-04-20, Florian Obser wrote: > >> You will need a carefully curated /etc/sysclean.ignore file. > >> > >> You decided to put maildirs somewhere on the system, sysclean is not > >> omniscient, you need to tell it to leave them alone. Same with .git > >> directories. > >> I don't recall needing to tell it about package config files though, > >> that's a bit weird. > > > > e.g. files which are added to /etc that aren't distributed in the package > > but > > you create yourself > > Ah, yes. But it does understand directories, e.g. I have a lot of > changes in /etc/icinga2 but I don't need to ignore it. I guess that's > why I only need to ignore very little in /etc. sysclean is using PLIST information from packages for knowing what is expected to be present or not. so if you had to add a file/directory in /etc/sysclean.ignore whereas it should belong to a port, it usually means a @sample or @extra entry is missing in PLIST :-) > > > >> It's a bit daunting on first run if a lot of cruft has accumulated > >> over the years, but it gets better. I'm using it for years, and I > >> can't recall the last time I had to add anything to the ignore file. > >> > >> I run it from daily and also by hand after every upgrade to a snapshot. > >> > >> If it outputs a really long list I cleanup incrementally, for example: > >> sysclean | fgrep /usr > > > > For a first run I would review "| fgrep /usr/local" as that's the most > > likely > > place where files might exist that should not be cleaned, and it's > > easier to /usr/local is globally ignored by sysclean. if you want to compare it with filesets from packages, the base system carries pkg_check (with pkg_check -qF). reviewing "| fgrep /usr" should be fine for a first run. > > tzk tzk, someone has been naughty and installed things without packages? > ;) > > I don't do that and I imagine if one installs compiled, dynamically > linked programs by hand sysclean's returns deminish really fast because > it won't understand that old libs are still needed. yes, it is a know drawback: if you compile locally a binary, sysclean will not know that you still need some libraries... I have few binaries in my $HOME for example, and I considere that sysclean helps to me rebuild them (because it breaks them when I remove old libc.so). Maybe one day I will create a (local) package for properly track them. -- Sebastien Marie
Re: Howto do "a detailed cleanup with the aid of the sysclean package"?
On 2022-04-20 21:42 UTC, Stuart Henderson wrote: > On 2022-04-20, Florian Obser wrote: >> You will need a carefully curated /etc/sysclean.ignore file. >> >> You decided to put maildirs somewhere on the system, sysclean is not >> omniscient, you need to tell it to leave them alone. Same with .git >> directories. >> I don't recall needing to tell it about package config files though, that's >> a bit weird. > > e.g. files which are added to /etc that aren't distributed in the package but > you create yourself Ah, yes. But it does understand directories, e.g. I have a lot of changes in /etc/icinga2 but I don't need to ignore it. I guess that's why I only need to ignore very little in /etc. > >> It's a bit daunting on first run if a lot of cruft has accumulated >> over the years, but it gets better. I'm using it for years, and I >> can't recall the last time I had to add anything to the ignore file. >> >> I run it from daily and also by hand after every upgrade to a snapshot. >> >> If it outputs a really long list I cleanup incrementally, for example: >> sysclean | fgrep /usr > > For a first run I would review "| fgrep /usr/local" as that's the most likely > place where files might exist that should not be cleaned, and it's > easier to tzk tzk, someone has been naughty and installed things without packages? ;) I don't do that and I imagine if one installs compiled, dynamically linked programs by hand sysclean's returns deminish really fast because it won't understand that old libs are still needed. It's an awesome tool that takles a hard problem and for me succeedes with a bit of hand holding. Btw. there is another school of thought that says old cruft doesn't need to be removed, it's not causing any harm. If you need a clean system just reinstall and restore config and data from backups. It's a good excercise to check that your backups are working. -- I'm not entirely sure you are real.
Re: Howto do "a detailed cleanup with the aid of the sysclean package"?
On 2022-04-20, Florian Obser wrote: > You will need a carefully curated /etc/sysclean.ignore file. > > You decided to put maildirs somewhere on the system, sysclean is not > omniscient, you need to tell it to leave them alone. Same with .git > directories. > I don't recall needing to tell it about package config files though, that's a > bit weird. e.g. files which are added to /etc that aren't distributed in the package but you create yourself > It's a bit daunting on first run if a lot of cruft has accumulated over the > years, but it gets better. I'm using it for years, and I can't recall the > last time I had to add anything to the ignore file. > > I run it from daily and also by hand after every upgrade to a snapshot. > > If it outputs a really long list I cleanup incrementally, for example: > sysclean | fgrep /usr For a first run I would review "| fgrep /usr/local" as that's the most likely place where files might exist that should not be cleaned, and it's easier to check for those if you don't have to wade through maybe thousands of lines of old headers, fonts, manpages, obsolete perl components and timezone files. If that is clear then I'm usually pretty happy to just remove anything else under /usr. If you want to be on the safe side then tar up the files before rm'ing. I don't do that though. > There really shouldn't be a false positive there, so after review I run > sysclean | fgrep /usr | xargs rm -r > next up is /etc. > If there is more output afterwards something is either very weird or an > intentional decision by me to store something in that location so it goes > into the ignore file.
Re: Howto do "a detailed cleanup with the aid of the sysclean package"?
You will need a carefully curated /etc/sysclean.ignore file. You decided to put maildirs somewhere on the system, sysclean is not omniscient, you need to tell it to leave them alone. Same with .git directories. I don't recall needing to tell it about package config files though, that's a bit weird. It's a bit daunting on first run if a lot of cruft has accumulated over the years, but it gets better. I'm using it for years, and I can't recall the last time I had to add anything to the ignore file. I run it from daily and also by hand after every upgrade to a snapshot. If it outputs a really long list I cleanup incrementally, for example: sysclean | fgrep /usr There really shouldn't be a false positive there, so after review I run sysclean | fgrep /usr | xargs rm -r next up is /etc. If there is more output afterwards something is either very weird or an intentional decision by me to store something in that location so it goes into the ignore file. On 20 April 2022 20:39:09 CEST, Harald Dunkel wrote: >Hi folks, > >the upgrade guide claims > > A detailed cleanup can be done with the aid of the sysclean package. > >sysclean lists 4180 files and directories on my home server, including mail >directories, config files of various external packages, generated files, .git >directories, etc. A lot of stuff I wouldn't like to lose. Apparently it also >lists a lot of old crap, but since it lists *so many* important files I don't >trust it at all. > >Could you please elaborate how sysclean is going to help me to keep my openbsd >hosts clean? How is the usage model of this tool? > > >Thank you very much in advance >Harri > -- Sent from a mobile device. Please excuse poor formatting.
Re: Howto do "a detailed cleanup with the aid of the sysclean package"?
On Wed, Apr 20, 2022 at 08:39:09PM +0200, Harald Dunkel wrote: > Hi folks, > > the upgrade guide claims > > A detailed cleanup can be done with the aid of the sysclean package. > > sysclean lists 4180 files and directories on my home server, including mail > directories, config files of various external packages, generated files, .git > directories, etc. A lot of stuff I wouldn't like to lose. Apparently it also > lists a lot of old crap, but since it lists *so many* important files I don't > trust it at all. > > Could you please elaborate how sysclean is going to help me to keep my openbsd > hosts clean? How is the usage model of this tool? Like any base tool, start with its man page: man sysclean Add any directories you want to keep into /etc/sysclean.ignore (start with the sample provided to ensure you keep the include at the end).
