Re: Why does OpenBSD still include Perl in its base installation?
Another thing to consider: why is perl in the base system. Assume you need a script language, because writing everything in C is cumbersome. What are the choices ? - you need something under and acceptable licence, so python is out. (Artistic Licence is "close enough"); - you need something that builds everywhere, so python is out (hard to build without dynamic libraries, that was vax...); - you want a modicum of security, so shell and tcl and php are out. - awk would kind of work, except it's not that readable, and it wouldn't scale up to some of the things we use perl for. Perl constituted a great compromise back in the day I chose it to replace pkg_add. It still IS a great compromise, and it's not THAT large compared to other pieces. Over the years, things have moved back from language to language. makewhatis used to be C+shell, then it became perl, and now it's pure C because it's integrated in mandoc... I shudder to think how much time was spent in there. Note that Ingo also moved /etc/security from shell to perl, so he's not adverse to perl. As far as I'm concerned, having perl in the base system is a strength of OpenBSD... it does minimize the number of script languages used for ports infrastructure as compared to other languages. perl also has impressive support tools, be they in the base system or in ports. NYTProf is still the best profiler I've ever seen on any language.
Re: Why does OpenBSD still include Perl in its base installation?
Good morning Dawid, Have you looked at different installations methods and post install options ? Customizing the Install Process https://www.openbsd.org/faq/faq4.html#site This will certainly be worth looking at if you intend to optimize and customize installation. Regards, Jean-François Le 23/05/2020 à 17:38, Dawid Czeluśniak a écrit : An important note: If you do any of that and subsequently encounter a problem, you must 1. Assume you created that problem for yourself 2. Not file a bug report 3. Not complain to others on OpenBSD mailing lists. Fair enough. Thank you all for a detailed explanation.
Re: Why does OpenBSD still include Perl in its base installation?
> An important note: > > If you do any of that and subsequently encounter a problem, you must > > 1. Assume you created that problem for yourself > 2. Not file a bug report > 3. Not complain to others on OpenBSD mailing lists. Fair enough. Thank you all for a detailed explanation.
Re: Why does OpenBSD still include Perl in its base installation?
An important note: If you do any of that and subsequently encounter a problem, you must 1. Assume you created that problem for yourself 2. Not file a bug report 3. Not complain to others on OpenBSD mailing lists. If that is an acceptable tradeoff, go ahead, make a mess. Philippe wrote: > On 19/05/2020 22:19, Dawid Czeluśniak wrote: > > Hi OpenBSD community, > > > > First of all, thank you for 6.7 release. > > > > I am a huge fan of minimal and custom installations > > as I mostly use OpenBSD to host simple HTTP servers. > > I basically use vi to edit httpd.conf file, > > obtain a certificate from Let's Encrypt using built-in acme-client, > > then start the server and things just work. > > > As the others already told you, you're on your own for this. > > > That being said, there are a few leads to get what you're asking. > Remind you that a good part of the actual size of the base install comes > from the kernel objects (almost 350 MB on my install), since they are > required for KARL, meaning you won't get a 10 MB install unless you drop > this functionality (and I guess nobody will recommend that). > Applications from the base install mostly do not take too much place. > > Let's get a taste of what an actual OpenBSD looks like, size-wise: > > # du ... > 3.7M/usr/local/bin/ > 6.5M/bin > 21.2M /sbin/ > 22.7M /usr/sbin/ > 41.7M /usr/share/man/ # almost 42 MB just for manuals > 149M/usr/bin/ # mostly unused programs for my use > 253M/usr/lib # thanks to C++, for a good part > 349M/usr/share/relink/kernel/ > > So yes, there a lot we can remove for a minimal install, not just Perl. > (and one may argue that it's required for a few important things, we > may don't really want to remove it with the current OpenBSD code base) > > For people running several instances of OpenBSD there is no point having > 10 times the same files. Yes, snapshots could be a good thing to solve > the problem, but let's try other approaches. > > > For the first solution, you can get a simpler base system by compiling a > release of you own. To achieve that, read at least mk.conf(5) and the > FAQ about making a release [1]. I didn't play a lot with this, but I > successfully removed compilers and a few programs. Here is what I did. > > # In file /etc/mk.conf > # Remove games, compilers (and a few other things with gnu/), manuals > # and a few applications. > # You can remove other applications, just add directories to the list. > SKIPDIR= games \ > gnu \ > lib/cxx \ > share/man \ > share/snmp \ > usr.sbin/ospfd usr.sbin/ospf6d usr.sbin/ospfctl usr.sbin/ospf6ctl \ > usr.sbin/ripd usr.sbin/ripctl > > (This was just a test, there a lot more to remove to get a minimal > install) > > I would like to drop the single core kernel since I don't need it, but > I'm not entirely sure it could be achieved by adding "GENERIC/" in the > above list. That's not super relevant either, since it won't be > installed on the system, so there is not much to care here. > Besides, I guess we could add an option in /etc/mk.conf for the size: > `CFLAGS=-Os`. > The good part of this solution is that it _completely_ works as the > upstream OpenBSD. You end-up with installation medias and sets, you have > the installer… that's kinda great. > > > Another solution could be resflash [2]. Resflash is a tool to create > OpenBSD images designed for flash storage. I don't like this solution > much, because it changes a bit the way the system is used. However, you > can probably learn from it. > > > Finally, I think crunchgen(8) could be a good lead if someone truly > wants to get basic image with the size of Busybox (or toybox) coreutils. > The whole point is to get several applications into a single one. It > sounds a bit retarded, but I'm sure there is a legit use to it, > somewhere in the wild. That's the last step to get _the_ smallest > install, I guess. Of course, this is not really _just_ to get a minimal > install, but more to get openbsd working on ISP routers with just a few > megabytes of RAM and disk. That's a bit experimental. > > > Food for thoughts: on FreeBSD, there is NanoBSD [3] which can be > configured to remove up to everything except the kernel and init. This > tool is used to create appliances, and I guess it could be nice to have > something like that on OpenBSD. I would use it for sure for minimal > installs on RPi-like boards. > > > I hope it will help! > Have a nice day everyone. > > [1]: https://www.openbsd.org/faq/faq5.html > [2]: https://stable.rcesoftware.com/resflash/ > [3]: https://www.freebsd.org/doc/en_US.ISO8859-1/articles/nanobsd/article.html >
Re: Why does OpenBSD still include Perl in its base installation?
On 19/05/2020 22:19, Dawid Czeluśniak wrote: > Hi OpenBSD community, > > First of all, thank you for 6.7 release. > > I am a huge fan of minimal and custom installations > as I mostly use OpenBSD to host simple HTTP servers. > I basically use vi to edit httpd.conf file, > obtain a certificate from Let's Encrypt using built-in acme-client, > then start the server and things just work. As the others already told you, you're on your own for this. That being said, there are a few leads to get what you're asking. Remind you that a good part of the actual size of the base install comes from the kernel objects (almost 350 MB on my install), since they are required for KARL, meaning you won't get a 10 MB install unless you drop this functionality (and I guess nobody will recommend that). Applications from the base install mostly do not take too much place. Let's get a taste of what an actual OpenBSD looks like, size-wise: # du ... 3.7M/usr/local/bin/ 6.5M/bin 21.2M /sbin/ 22.7M /usr/sbin/ 41.7M /usr/share/man/ # almost 42 MB just for manuals 149M/usr/bin/ # mostly unused programs for my use 253M/usr/lib # thanks to C++, for a good part 349M/usr/share/relink/kernel/ So yes, there a lot we can remove for a minimal install, not just Perl. (and one may argue that it's required for a few important things, we may don't really want to remove it with the current OpenBSD code base) For people running several instances of OpenBSD there is no point having 10 times the same files. Yes, snapshots could be a good thing to solve the problem, but let's try other approaches. For the first solution, you can get a simpler base system by compiling a release of you own. To achieve that, read at least mk.conf(5) and the FAQ about making a release [1]. I didn't play a lot with this, but I successfully removed compilers and a few programs. Here is what I did. # In file /etc/mk.conf # Remove games, compilers (and a few other things with gnu/), manuals # and a few applications. # You can remove other applications, just add directories to the list. SKIPDIR= games \ gnu \ lib/cxx \ share/man \ share/snmp \ usr.sbin/ospfd usr.sbin/ospf6d usr.sbin/ospfctl usr.sbin/ospf6ctl \ usr.sbin/ripd usr.sbin/ripctl (This was just a test, there a lot more to remove to get a minimal install) I would like to drop the single core kernel since I don't need it, but I'm not entirely sure it could be achieved by adding "GENERIC/" in the above list. That's not super relevant either, since it won't be installed on the system, so there is not much to care here. Besides, I guess we could add an option in /etc/mk.conf for the size: `CFLAGS=-Os`. The good part of this solution is that it _completely_ works as the upstream OpenBSD. You end-up with installation medias and sets, you have the installer… that's kinda great. Another solution could be resflash [2]. Resflash is a tool to create OpenBSD images designed for flash storage. I don't like this solution much, because it changes a bit the way the system is used. However, you can probably learn from it. Finally, I think crunchgen(8) could be a good lead if someone truly wants to get basic image with the size of Busybox (or toybox) coreutils. The whole point is to get several applications into a single one. It sounds a bit retarded, but I'm sure there is a legit use to it, somewhere in the wild. That's the last step to get _the_ smallest install, I guess. Of course, this is not really _just_ to get a minimal install, but more to get openbsd working on ISP routers with just a few megabytes of RAM and disk. That's a bit experimental. Food for thoughts: on FreeBSD, there is NanoBSD [3] which can be configured to remove up to everything except the kernel and init. This tool is used to create appliances, and I guess it could be nice to have something like that on OpenBSD. I would use it for sure for minimal installs on RPi-like boards. I hope it will help! Have a nice day everyone. [1]: https://www.openbsd.org/faq/faq5.html [2]: https://stable.rcesoftware.com/resflash/ [3]: https://www.freebsd.org/doc/en_US.ISO8859-1/articles/nanobsd/article.html
Re: Why does OpenBSD still include Perl in its base installation?
On Tue, May 19, 2020 at 10:19:23PM +0200, Dawid Czeluśniak wrote: > 3. Are there people among this community who think that > the base installation of OpenBSD should not include Perl > and Perl-dependent programs and utilities? I assume there are people in the community who think that OpenBSD should not include a lot of things, fortunately it is not the community who decides what OpenBSD becomes, but instead the developers. > Looking at FreeBSD for a moment it seems like Perl left the > base system in May 2002 (18 years ago)[0]. > In a nutshell the rationale was more or less the following[1]: > > > It appears that having Perl in the base distro has > > the following problems: > > > 1. It increases the distro image size. FreeBSD's minimal iso download is 370M while OpenBSD's is largest install media is 450M, however if you are looking for a more minimal install of OpenBSD, you can ~10M cd67.iso and avoid a little bit by only installing the absolutely necessary sets. https://download.freebsd.org/ftp/releases/amd64/amd64/ISO-IMAGES/12.1/ http://ftp.openbsd.org/pub/OpenBSD/6.7/amd64/ In my opinion, the things that OpenBSD includes in the base system provide an excellent trade-off between size and utility. 18 years ago, an extra 60M of installed files may have made a difference. These days, it's insignificant. > > 2. It forces everyone to use the same version of Perl. It forces everyone to use the same default version of perl, and in exchange promises working packages for programs written in perl. There are several ways to manage having additional versions of perl installed if you need them. Both perlbrew and plenv come to mind. https://perlbrew.pl/ https://github.com/tokuhirom/plenv > > 3. If someone tries to install over that version or just even patch it, > > it can break stuff in BSD which needed the old version. I patch the in-base perl regularly, both breaking stuff and not. Not knowing what you're doing is a not only a great way to learn, but also a great way to break stuff. > > 4. Installing multiple copies imposes weird symlink tricks > > or else breaks other stuff. See above for some ways to do this that work really well, these days that's mostly a solved problem, but it does require both reading and understanding. > All comments and opinions are appreciated. Thanks! Oh good. Perl fits a really important niche between shell scripts and C, its combination of "whipupitude" and "manipulexity" make it great for a large range of tasks. While there are now several languages that also fit into that space, choices between them boil down to personal preference and momentum. The fact that there are so many lines of perl doing the amazingly complex work of managing the package system means that it is here to stay. Unix is full of text streams and having a tool available that excels at processing text is a significant benefit to the OpenBSD base system. Having perl in base was one of the things that brought me joy when I started using OpenBSD and it continues to bring me joy. > [0] https://lists.freebsd.org/pipermail/freebsd-announce/2002-May/000823.html Being the OpenBSD "Perl5 guy", I consider keeping OpenBSD's "base perl" up to date well worth the effort and don't think it is a significant amount of work. There has been some headway by the perl5-porters in removing things from the perl core. Go see the last few upgrade guides where there are instructions on deleting the modules that have been removed. FreeBSD's incompatibility with the Perl 5 release cycle, where Perl gets security updates for the current and previous release, would have meant that the maintainer there would need to do significant work to backport fixes to several old versions of perl. That can be "hellish", but fortunately OpenBSD does not have that particular problem. Fortunately, I can count the number of backwards incompatible changes to Perl since I've been using it on one hand, so importing those new versions into OpenBSD is generally the work of only a few days, most of which turns out to be uneventful testing that things still work. > [1] https://bsd.slashdot.org/comments.pl?sid=32564=3513689 See above. l8rZ, -- andrew - http://afresh1.com I wish life had an UNDO function.
Re: Why does OpenBSD still include Perl in its base installation?
Hi David, Dawid Czelu?niak wrote on Tue, May 19, 2020 at 10:19:23PM +0200: > I am a huge fan of minimal and custom installations > as I mostly use OpenBSD to host simple HTTP servers. That's perfectly fine. These days, we consider the "minimal installation" the base file sets, including xbase. If you want, you can leave out the game file set and the other x* file sets, but it makes maintenance significantly harder for no benefit. Your choice, really. > I basically use vi to edit httpd.conf file, Sure. How else? I think almost everybody used vi(1) or mg(1) for that. > obtain a certificate from Let's Encrypt using built-in acme-client, > then start the server and things just work. Sure. That's exactly the same way as i'm maintaining machines like man.openbsd.org and mandoc.bsd.lv, too. > I was wondering if I need the package manager in the minimal > installation of the system as I only use built-in deamons (httpd, sshd) > and UNIX utilities (vi, sed)? By package manager I mean > pkg_* executables as well as its dependencies (most notably Perl). > (The size of /usr/libdata/perl5 is about 50MB on my machine). > I just want the minimal installation without any unnecessary > scripting language. One way to achieve that could be > to manually remove pkg_* executables and Perl from a machine, > but it can easily end up with the broken system especially when > done by unexperienced users. I can confirm that. Removing files that come with base in order to make base smaller is a bad idea. It tried that 20 years ago when i already had several years of experience with various flavours of Unix, and yet i consistently ended up with systems broken in one way or another. I did have a moderately good reason back then: i needed to run OpenBSD (around 2.6 to 2.8 times) on a i386 machine that only had 200 MB of disk space. Even then, that reason was only moderately good; buying an additional disk of 4 GB would not have been very expensive back then, so that would probably have been the most reasonable choice. But i did not want to spend money on that machine, so i chose to, instead of installing any file sets, to hand-pick the files i wanted, one by one. That way, i did actually get a working system, and i used it in production for a number of years, doing several upgrades in the same way. I learnt quite a bit about the purposes of various files in the process. I never asked anyone for help with it. Repeating such a stunt today would be quite pointless. A 20GB disk is more than sufficient for doing a base install, and you will have problems to find a 20GB disk even in the trash. The disks you will readily find in the trash are typically at least ten times larger nowadays. > But then I thought about > the second possible way: why not to extract pkg_* executables > and Perl to the separate file set (pkgXX.tgz)? As others said, absolutely not going to happen. I think having fewer sets might provide some benefits, but splitting sets would be a waste of time and asking for trouble. Yours, Ingo
Re: Why does OpenBSD still include Perl in its base installation?
You are on your own. You knew that would be the answer. Be happy about it. Dawid Czeluśniak wrote: > Hi OpenBSD community, > > First of all, thank you for 6.7 release. > > I am a huge fan of minimal and custom installations > as I mostly use OpenBSD to host simple HTTP servers. > I basically use vi to edit httpd.conf file, > obtain a certificate from Let's Encrypt using built-in acme-client, > then start the server and things just work. > > I was wondering if I need the package manager in the minimal > installation of the system as I only use built-in deamons (httpd, sshd) > and UNIX utilities (vi, sed)? By package manager I mean > pkg_* executables as well as its dependencies (most notably Perl). > (The size of /usr/libdata/perl5 is about 50MB on my machine). > I just want the minimal installation without any unnecessary > scripting language. One way to achieve that could be > to manually remove pkg_* executables and Perl from a machine, > but it can easily end up with the broken system especially when > done by unexperienced users. But then I thought about > the second possible way: why not to extract pkg_* executables > and Perl to the separate file set (pkgXX.tgz)? > In this way, people who are not going to use the package manager > could just simply not install it. Moreover, it could break > the base installation free from the heavy dependence on Perl. > > I would like to get your opinion on that. > > So the questions to the OpenBSD community are the following: > > 1. Is it possible, from the technical point of view, to extract > pkg_* utilities, perlutil(1), any other Perl-dependent code > and Perl itself from baseXX.tgz to the separate file set > so that users that just want the *minimal* installation of OpenBSD > don't have to remove it manually? > > 2. Are there any other programs in the baseXX.tgz file set > that depend on Perl other than: > - pkg_* utilities > - perlutil(1) (rather obvious) > - fw_update(1) > - adduser(8) (adduser and rmuser utilities) > - vi.recover script > - cvs contrib scripts > > 3. Are there people among this community who think that > the base installation of OpenBSD should not include Perl > and Perl-dependent programs and utilities? > Looking at FreeBSD for a moment it seems like Perl left the > base system in May 2002 (18 years ago)[0]. > In a nutshell the rationale was more or less the following[1]: > > > It appears that having Perl in the base distro has > > the following problems: > > > 1. It increases the distro image size. > > 2. It forces everyone to use the same version of Perl. > > 3. If someone tries to install over that version or just even patch it, > > it can break stuff in BSD which needed the old version. > > 4. Installing multiple copies imposes weird symlink tricks > > or else breaks other stuff. > > All comments and opinions are appreciated. Thanks! > > [0] https://lists.freebsd.org/pipermail/freebsd-announce/2002-May/000823.html > [1] https://bsd.slashdot.org/comments.pl?sid=32564=3513689 >
Re: Why does OpenBSD still include Perl in its base installation?
On 2020-05-21 09:55, Anders Andersson wrote: >> I am a huge fan of minimal and custom installations >> as I mostly use OpenBSD to host simple HTTP servers. > ... >> I would like to get your opinion on that. > From what I've seen, those goals are not compatible with OpenBSD, as > in: You're just making it harder for you and anyone trying to help > debugging something if you change the default installation. I've seen > some wishes about even getting rid of the whole "sets" thing and just > install everything. I agree and disagree. I agree with the unsupported compatible side, with a big question of why do you want to? I like having a well considered useful base, rather than a million linux kernel/package options. I disagree with OpenBSD being incompatible. Take bsd.rd or single user and a binary and it could be a very small executer atleast. Anyone that has hosed the root fs and and had to fix the shared cache with ldconfig, probably realises that. No idea, why you would want to go that small and not use something like app engine and go though. I guess you could use pledge/unveil if you have all the networking side sorted. I can't see involatile storage as an issue there or anywhere OpenBSD runs though and it's focus on security means updates are fairly infrequent?? p.s. Even app engine uses a fairly big and useful install, before the deploy stage.
Re: Why does OpenBSD still include Perl in its base installation?
On 2020-05-19, Dawid Czeluśniak wrote: > But then I thought about > the second possible way: why not to extract pkg_* executables > and Perl to the separate file set (pkgXX.tgz)? I think it is safe to say this is not going to happen. > 2. Are there any other programs in the baseXX.tgz file set > that depend on Perl other than: > - pkg_* utilities > - perlutil(1) (rather obvious) > - fw_update(1) > - adduser(8) (adduser and rmuser utilities) > - vi.recover script > - cvs contrib scripts libtool /usr/libexec/security
Re: Why does OpenBSD still include Perl in its base installation?
On Thu, May 21, 2020 at 11:19 AM Dawid Czeluśniak wrote: > > Hi OpenBSD community, > > First of all, thank you for 6.7 release. > > I am a huge fan of minimal and custom installations > as I mostly use OpenBSD to host simple HTTP servers. ... > I would like to get your opinion on that. >From what I've seen, those goals are not compatible with OpenBSD, as in: You're just making it harder for you and anyone trying to help debugging something if you change the default installation. I've seen some wishes about even getting rid of the whole "sets" thing and just install everything. I tend to agree and would welcome such a move, because these days we're talking about such a tiny amount of space in comparison. Even if you're in a situation where you want to host thousands of virtual OpenBSD machines and then maybe get some sort of gain from removing those 50 MB, well, just use a CoW filesystem and clone the same base install. What I love with OpenBSD is that everything is just there to be used, there aren't 20 different filesystems, 20 different scripting languages, 20 different web servers. I don't have to fiddle with everything, it just works. There's the file system, perl, httpd, etc, and they are well designed. Why would I want multiple different perls when it is already so mature?
Why does OpenBSD still include Perl in its base installation?
Hi OpenBSD community, First of all, thank you for 6.7 release. I am a huge fan of minimal and custom installations as I mostly use OpenBSD to host simple HTTP servers. I basically use vi to edit httpd.conf file, obtain a certificate from Let's Encrypt using built-in acme-client, then start the server and things just work. I was wondering if I need the package manager in the minimal installation of the system as I only use built-in deamons (httpd, sshd) and UNIX utilities (vi, sed)? By package manager I mean pkg_* executables as well as its dependencies (most notably Perl). (The size of /usr/libdata/perl5 is about 50MB on my machine). I just want the minimal installation without any unnecessary scripting language. One way to achieve that could be to manually remove pkg_* executables and Perl from a machine, but it can easily end up with the broken system especially when done by unexperienced users. But then I thought about the second possible way: why not to extract pkg_* executables and Perl to the separate file set (pkgXX.tgz)? In this way, people who are not going to use the package manager could just simply not install it. Moreover, it could break the base installation free from the heavy dependence on Perl. I would like to get your opinion on that. So the questions to the OpenBSD community are the following: 1. Is it possible, from the technical point of view, to extract pkg_* utilities, perlutil(1), any other Perl-dependent code and Perl itself from baseXX.tgz to the separate file set so that users that just want the *minimal* installation of OpenBSD don't have to remove it manually? 2. Are there any other programs in the baseXX.tgz file set that depend on Perl other than: - pkg_* utilities - perlutil(1) (rather obvious) - fw_update(1) - adduser(8) (adduser and rmuser utilities) - vi.recover script - cvs contrib scripts 3. Are there people among this community who think that the base installation of OpenBSD should not include Perl and Perl-dependent programs and utilities? Looking at FreeBSD for a moment it seems like Perl left the base system in May 2002 (18 years ago)[0]. In a nutshell the rationale was more or less the following[1]: > It appears that having Perl in the base distro has > the following problems: > 1. It increases the distro image size. > 2. It forces everyone to use the same version of Perl. > 3. If someone tries to install over that version or just even patch it, > it can break stuff in BSD which needed the old version. > 4. Installing multiple copies imposes weird symlink tricks > or else breaks other stuff. All comments and opinions are appreciated. Thanks! [0] https://lists.freebsd.org/pipermail/freebsd-announce/2002-May/000823.html [1] https://bsd.slashdot.org/comments.pl?sid=32564=3513689
