Re: [arch-general] Bashification of initscripts for moderate speedup
On Sun, 2010-06-27 at 22:10 -0500, Victor Lowther wrote: I have spent some time over the last month or so rewriting the Arch initscripts in idiomatic bash -- since they rely on bash-specific features, trying to keep them fairly POSIX is rather a waste of time and a net performance loss -- using bash-style conditionals is about 30% faster when nothing needs to be touched on the filesystem and about 200% easier to read, and you can use bash regexps in place of most trivial uses of sed, grep, and awk when mangling parameters or variables -- the fewer short-lived processes we fork the better. You can browse changes I have made in the git repo @ http://git.fnordovax.org/arch-initscripts/log/?h=bashification and you can checkout my changes to play around with from git://fnordovax.org/~victor/arch-initscripts OK, the bashification branch has been rebased and updated to fix issues pointed out in the patch-by-patch feedback. Thomas, if you want I will repost the branch to the mailing list. -- Victor Lowther LPIC2 UCP RHCE
Re: [arch-general] Bashification of initscripts for moderate speedup
Am 03.07.2010 21:07, schrieb Victor Lowther: On Sun, 2010-06-27 at 22:10 -0500, Victor Lowther wrote: I have spent some time over the last month or so rewriting the Arch initscripts in idiomatic bash -- since they rely on bash-specific features, trying to keep them fairly POSIX is rather a waste of time and a net performance loss -- using bash-style conditionals is about 30% faster when nothing needs to be touched on the filesystem and about 200% easier to read, and you can use bash regexps in place of most trivial uses of sed, grep, and awk when mangling parameters or variables -- the fewer short-lived processes we fork the better. You can browse changes I have made in the git repo @ http://git.fnordovax.org/arch-initscripts/log/?h=bashification and you can checkout my changes to play around with from git://fnordovax.org/~victor/arch-initscripts OK, the bashification branch has been rebased and updated to fix issues pointed out in the patch-by-patch feedback. Thomas, if you want I will repost the branch to the mailing list. I will finish reviewing the rest of the patches first, when I get the time again. Sorry for being so slow on this. signature.asc Description: OpenPGP digital signature
Re: [arch-general] Bashification of initscripts for moderate speedup
2010-06-28 05:10, Victor Lowther: You can browse changes I have made in the git repo @ http://git.fnordovax.org/arch-initscripts/log/?h=bashification The feed link appears broken to me. link rel='alternate' title='Atom feed' href='http://git.fnordovax.orghttp://git.fnordovax.org/arch-initscripts/atom/?h=bashification' type='application/atom+xml'/ -- kujub (whining)
Re: [arch-general] Bashification of initscripts for moderate speedup
On Jul 2, 2010, at 10:35 AM, Kurt J. Bosch kjb-temp-2...@alpenjodel.de wrote: 2010-06-28 05:10, Victor Lowther: You can browse changes I have made in the git repo @ http://git.fnordovax.org/arch-initscripts/log/?h=bashification The feed link appears broken to me. link rel='alternate' title='Atom feed' href='http://git.fnordovax.orghttp://git.fnordovax.org/arch-initscripts/atom/?h=bashification' type='application/atom+xml'/ Thanks. I might eventually fix it. -- kujub (whining)
Re: [arch-general] Bashification of initscripts for moderate speedup
On 06/29/2010 11:44 AM, Victor Lowther wrote: We might want to steal some content from http://mywiki.wooledge.org/BashGuide/Practices to update the Arch bash coding page. Regardless which way the bashification issue goes, I agree with Victor, the above contains a very good rule-set for bash coding and style. -- David C. Rankin, J.D.,P.E. Rankin Law Firm, PLLC 510 Ochiltree Street Nacogdoches, Texas 75961 Telephone: (936) 715-9333 Facsimile: (936) 715-9339 www.rankinlawfirm.com
Re: [arch-general] Bashification of initscripts for moderate speedup
On Tue, Jun 29, 2010 at 1:57 PM, Aaron Griffin aaronmgrif...@gmail.com wrote: I have no idea where the number came from, but both 80 and 132 are standard column widths used for normal and wide terminals. should be 78 columns I do believe that's the standard for kernel/git it allows for email comments more easily apparently. -- Caleb Cushing http://xenoterracide.blogspot.com
Re: [arch-general] Bashification of initscripts for moderate speedup
Excerpts from Aaron Griffin's message of 2010-06-30 17:55:40 +0200: On Wed, Jun 30, 2010 at 10:00 AM, Caleb Cushing xenoterrac...@gmail.com wrote: On Tue, Jun 29, 2010 at 1:57 PM, Aaron Griffin aaronmgrif...@gmail.com wrote: I have no idea where the number came from, but both 80 and 132 are standard column widths used for normal and wide terminals. should be 78 columns I do believe that's the standard for kernel/git it allows for email comments more easily apparently. No, physical terminals went to 80 columns. The 78 number was likely related to editor chrome, such as line numbering... I'm not sure of the details. I believe email typically has 78 width due to the fact that is expected for replies. I also read about 72, and it's my current setting for vim when editing mails, but I have no idea where that comes from... -- Regards, Philipp -- Wir stehen selbst enttäuscht und sehn betroffen / Den Vorhang zu und alle Fragen offen. Bertolt Brecht, Der gute Mensch von Sezuan
Re: [arch-general] Bashification of initscripts for moderate speedup
Am 28.06.2010 05:10, schrieb Victor Lowther: I have spent some time over the last month or so rewriting the Arch initscripts in idiomatic bash -- since they rely on bash-specific features, trying to keep them fairly POSIX is rather a waste of time and a net performance loss -- using bash-style conditionals is about 30% faster when nothing needs to be touched on the filesystem and about 200% easier to read, and you can use bash regexps in place of most trivial uses of sed, grep, and awk when mangling parameters or variables -- the fewer short-lived processes we fork the better. You can browse changes I have made in the git repo @ http://git.fnordovax.org/arch-initscripts/log/?h=bashification and you can checkout my changes to play around with from git://fnordovax.org/~victor/arch-initscripts Okay, those are LOTS of patches. I agree with most of them, but not all. However, reviewing and commenting them like this is a bit tedious - can you use git send-email to send them to this list as a patch series, so I can comment properly? signature.asc Description: OpenPGP digital signature
Re: [arch-general] Bashification of initscripts for moderate speedup
On Jun 30, 2010, at 12:40 PM, Thomas Bächler tho...@archlinux.org wrote: Am 28.06.2010 05:10, schrieb Victor Lowther: I have spent some time over the last month or so rewriting the Arch initscripts in idiomatic bash -- since they rely on bash-specific features, trying to keep them fairly POSIX is rather a waste of time and a net performance loss -- using bash-style conditionals is about 30% faster when nothing needs to be touched on the filesystem and about 200% easier to read, and you can use bash regexps in place of most trivial uses of sed, grep, and awk when mangling parameters or variables -- the fewer short-lived processes we fork the better. You can browse changes I have made in the git repo @ http://git.fnordovax.org/arch-initscripts/log/?h=bashification and you can checkout my changes to play around with from git://fnordovax.org/~victor/arch-initscripts Okay, those are LOTS of patches. I agree with most of them, but not all. However, reviewing and commenting them like this is a bit tedious - can you use git send-email to send them to this list as a patch series, so I can comment properly? Sure. I will do so tonight.
Re: [arch-general] Bashification of initscripts for moderate speedup
On Wed 30 Jun 2010 18:48 +0200, Philipp Überbacher wrote: Excerpts from Aaron Griffin's message of 2010-06-30 17:55:40 +0200: On Wed, Jun 30, 2010 at 10:00 AM, Caleb Cushing xenoterrac...@gmail.com wrote: On Tue, Jun 29, 2010 at 1:57 PM, Aaron Griffin aaronmgrif...@gmail.com wrote: I have no idea where the number came from, but both 80 and 132 are standard column widths used for normal and wide terminals. should be 78 columns I do believe that's the standard for kernel/git it allows for email comments more easily apparently. No, physical terminals went to 80 columns. The 78 number was likely related to editor chrome, such as line numbering... I'm not sure of the details. I believe email typically has 78 width due to the fact that is expected for replies. I also read about 72, and it's my current setting for vim when editing mails, but I have no idea where that comes from... Yeah I generally use 72 for emails.
Re: [arch-general] Bashification of initscripts for moderate speedup
Le lundi 28 juin 2010 à 17:55 +0200, Lukáš Jirkovský a écrit : Actually I see the point of doing this. Arch is a modern distribution with the newest software around so why stuck with shell constructs which are probably dozens of years old? Lukas Yes. I definitely agree. We have to pray that a dev is interested in the patch now ;-) @Victor: You last commit that is supposed to fix whitespace mess in fact create a mess with whitespace. Also it changes all the modelines for vim: for example from # vim: set ft=sh sw=2 ts=2 et: to # vim: set ft=sh sw=4 ts=4 et: in fact, I would suggest that you revert your commit 59c7059e78491988c5d8d495c731fb66fe8f7224. It does not fix anything. check for yourself with git checkout bashification git diff master but git diff master HEAD~1 is clearly better
Re: [arch-general] Bashification of initscripts for moderate speedup
On Jun 29, 2010, at 5:55 AM, solsTiCe d'Hiver solstice.dhi...@gmail.com wrote: Le lundi 28 juin 2010 à 17:55 +0200, Lukáš Jirkovský a écrit : Actually I see the point of doing this. Arch is a modern distribution with the newest software around so why stuck with shell constructs which are probably dozens of years old? Lukas Yes. I definitely agree. We have to pray that a dev is interested in the patch now ;-) Thomas seemed moderatly interested. @Victor: You last commit that is supposed to fix whitespace mess in fact create a mess with whitespace. Also it changes all the modelines for vim: for example from # vim: set ft=sh sw=2 ts=2 et: to # vim: set ft=sh sw=4 ts=4 et: When it comes to whitespace, I do whatever indent-region and whitespace-cleanup tell me to. in fact, I would suggest that you revert your commit 59c7059e78491988c5d8d495c731fb66fe8f7224. It does not fix anything. check for yourself with git checkout bashification git diff master but git diff master HEAD~1 is clearly better
Re: [arch-general] Bashification of initscripts for moderate speedup
I have uploaded some shiny bootchart runs from booting with and without bashification on my laptop to http://bugs.archlinux.org/20006
Re: [arch-general] Bashification of initscripts for moderate speedup
On Tue, Jun 29, 2010 at 8:20 AM, Victor Lowther victor.lowt...@gmail.com wrote: On Jun 29, 2010, at 5:55 AM, solsTiCe d'Hiver solstice.dhi...@gmail.com wrote: Le lundi 28 juin 2010 à 17:55 +0200, Lukáš Jirkovský a écrit : Actually I see the point of doing this. Arch is a modern distribution with the newest software around so why stuck with shell constructs which are probably dozens of years old? Lukas Yes. I definitely agree. We have to pray that a dev is interested in the patch now ;-) Thomas seemed moderatly interested. @Victor: You last commit that is supposed to fix whitespace mess in fact create a mess with whitespace. Also it changes all the modelines for vim: for example from # vim: set ft=sh sw=2 ts=2 et: to # vim: set ft=sh sw=4 ts=4 et: When it comes to whitespace, I do whatever indent-region and whitespace-cleanup tell me to. Which is change the modelines? No thanks. http://wiki.archlinux.org/index.php/DeveloperWiki:Bash_Coding_Style -Dan
Re: [arch-general] Bashification of initscripts for moderate speedup
On Jun 29, 2010, at 9:27 AM, Dan McGee dpmc...@gmail.com wrote: On Tue, Jun 29, 2010 at 8:20 AM, Victor Lowther victor.lowt...@gmail.com wrote: On Jun 29, 2010, at 5:55 AM, solsTiCe d'Hiver solstice.dhi...@gmail.com wrote: Le lundi 28 juin 2010 à 17:55 +0200, Lukáš Jirkovský a écrit : Actually I see the point of doing this. Arch is a modern distribution with the newest software around so why stuck with shell constructs which are probably dozens of years old? Lukas Yes. I definitely agree. We have to pray that a dev is interested in the patch now ;-) Thomas seemed moderatly interested. @Victor: You last commit that is supposed to fix whitespace mess in fact create a mess with whitespace. Also it changes all the modelines for vim: for example from # vim: set ft=sh sw=2 ts=2 et: to # vim: set ft=sh sw=4 ts=4 et: When it comes to whitespace, I do whatever indent-region and whitespace-cleanup tell me to. Which is change the modelines? No thanks. http://wiki.archlinux.org/index.php/DeveloperWiki:Bash_Coding_Style I know we are getting into holy war territory here, but: Overall, they are sane, except for using tabs vs. spaces (at least with spaces it is never ambiguous how far you wanted a given indent to be in a world where tab does not always map to a set number of spaces), and 8 character indentation (leads to excessive horizontal scrolling in any moderatly complex nested flow control situation. I prefer 4 space indentation, it tends to align nicely with most flow control statements), and using the source keyword instead of . to source files (personal preference from also maintaining a package written in posix sh). We might want to steal some content from http://mywiki.wooledge.org/BashGuide/Practices to update the Arch bash coding page. -Dan
Re: [arch-general] Bashification of initscripts for moderate speedup
On Jun 28, 2010, at 11:36 AM, Isaac Dupree m...@isaac.cedarswampstudios.org wrote: On 06/28/10 09:35, Victor Lowther wrote: On Jun 28, 2010, at 7:42 AM, Caleb Cushing xenoterrac...@gmail.com wrote: On Sun, Jun 27, 2010 at 11:10 PM, Victor Lowther victor.lowt...@gmail.com wrote: Questions, comments, flames, etc. welcome. why go this way instead of the other? (clarification why go deeper into bash instead of trying to posix-ify the scripts) ...and the main source of slowdowns in the boot sequence is I/O based -- mounting filesystems and launching programs. IIRC, Busybox shell can get notable speed boost by incorporating versions of tools like sed into the same busybox executable, such that it often doesn't have to fork and load other short-lived programs. You can eliminate a fair number of trivial uses of grep, awk, and sed by using all of the parameter expansion modifiers that bash offers, and by using the native regex support from bash 3 and above. I made these changes wherever it seemed appropriate in my changes. (I believe it doesn't do bash-array-syntax.) -Isaac
Re: [arch-general] Bashification of initscripts for moderate speedup
On Tue 29 Jun 2010 09:27 -0500, Dan McGee wrote: Which is change the modelines? No thanks. http://wiki.archlinux.org/index.php/DeveloperWiki:Bash_Coding_Style Neat. Where does the 132 columns come from?
Re: [arch-general] Bashification of initscripts for moderate speedup
On Tue, Jun 29, 2010 at 12:47 PM, Loui Chang louipc@gmail.com wrote: On Tue 29 Jun 2010 09:27 -0500, Dan McGee wrote: Which is change the modelines? No thanks. http://wiki.archlinux.org/index.php/DeveloperWiki:Bash_Coding_Style Neat. Where does the 132 columns come from? Back in the day of real physical terminals, they were typically made to display 80 columns on the screen. At some point, wide terminals came out, supporting 132 columns. I have no idea where the number came from, but both 80 and 132 are standard column widths used for normal and wide terminals.
Re: [arch-general] Bashification of initscripts for moderate speedup
On Sun, Jun 27, 2010 at 11:10 PM, Victor Lowther victor.lowt...@gmail.com wrote: Questions, comments, flames, etc. welcome. why go this way instead of the other? (clarification why go deeper into bash instead of trying to posix-ify the scripts) -- Caleb Cushing http://xenoterracide.blogspot.com
Re: [arch-general] Bashification of initscripts for moderate speedup
On Mon, Jun 28, 2010 at 7:42 AM, Caleb Cushing xenoterrac...@gmail.com wrote: On Sun, Jun 27, 2010 at 11:10 PM, Victor Lowther victor.lowt...@gmail.com wrote: Questions, comments, flames, etc. welcome. why go this way instead of the other? (clarification why go deeper into bash instead of trying to posix-ify the scripts) Because we are never going to eliminate arrays from rc.conf. -Dan
Re: [arch-general] Bashification of initscripts for moderate speedup
On Jun 28, 2010, at 7:42 AM, Caleb Cushing xenoterrac...@gmail.com wrote: On Sun, Jun 27, 2010 at 11:10 PM, Victor Lowther victor.lowt...@gmail.com wrote: Questions, comments, flames, etc. welcome. why go this way instead of the other? (clarification why go deeper into bash instead of trying to posix-ify the scripts) Because arrays and associative arrays are very useful, using native bash costructs instead of posix gives you a decent speed boost, and the main source of slowdowns in the boot sequence is I/O based -- mounting filesystems and launching programs. Serious boot speed optimization is a matter of optimizing I/O. -- Caleb Cushing http://xenoterracide.blogspot.com
Re: [arch-general] Bashification of initscripts for moderate speedup
On Mon 28 Jun 2010 08:04 -0500, Dan McGee wrote: On Mon, Jun 28, 2010 at 7:42 AM, Caleb Cushing xenoterrac...@gmail.com wrote: On Sun, Jun 27, 2010 at 11:10 PM, Victor Lowther victor.lowt...@gmail.com wrote: Questions, comments, flames, etc. welcome. why go this way instead of the other? (clarification why go deeper into bash instead of trying to posix-ify the scripts) Because we are never going to eliminate arrays from rc.conf. Well, it may still be beneficial to some. The scripts could be used with a different style rc.conf in other environments.
Re: [arch-general] Bashification of initscripts for moderate speedup
On Jun 28, 2010, at 8:59 AM, Loui Chang louipc@gmail.com wrote: On Mon 28 Jun 2010 08:04 -0500, Dan McGee wrote: On Mon, Jun 28, 2010 at 7:42 AM, Caleb Cushing xenoterrac...@gmail.com wrote: On Sun, Jun 27, 2010 at 11:10 PM, Victor Lowther victor.lowt...@gmail.com wrote: Questions, comments, flames, etc. welcome. why go this way instead of the other? (clarification why go deeper into bash instead of trying to posix-ify the scripts) Because we are never going to eliminate arrays from rc.conf. Well, it may still be beneficial to some. The scripts could be used with a different style rc.conf in other environments. Then it will not be Arch.
Re: [arch-general] Bashification of initscripts for moderate speedup
On Mon 28 Jun 2010 09:13 -0500, Victor Lowther wrote: On Jun 28, 2010, at 8:59 AM, Loui Chang louipc@gmail.com wrote: On Mon 28 Jun 2010 08:04 -0500, Dan McGee wrote: On Mon, Jun 28, 2010 at 7:42 AM, Caleb Cushing xenoterrac...@gmail.com wrote: On Sun, Jun 27, 2010 at 11:10 PM, Victor Lowther victor.lowt...@gmail.com wrote: Questions, comments, flames, etc. welcome. why go this way instead of the other? (clarification why go deeper into bash instead of trying to posix-ify the scripts) Because we are never going to eliminate arrays from rc.conf. Well, it may still be beneficial to some. The scripts could be used with a different style rc.conf in other environments. Then it will not be Arch. Depends on what you define as Arch. I've heard that Arch is what you make of it. Hehe. I don't know if you could disqualify it from a difference of one file.
Re: [arch-general] Bashification of initscripts for moderate speedup
On Jun 28, 2010, at 9:49 AM, Loui Chang louipc@gmail.com wrote: On Mon 28 Jun 2010 09:13 -0500, Victor Lowther wrote: On Jun 28, 2010, at 8:59 AM, Loui Chang louipc@gmail.com wrote: On Mon 28 Jun 2010 08:04 -0500, Dan McGee wrote: On Mon, Jun 28, 2010 at 7:42 AM, Caleb Cushing xenoterrac...@gmail.com wrote: On Sun, Jun 27, 2010 at 11:10 PM, Victor Lowther victor.lowt...@gmail.com wrote: Questions, comments, flames, etc. welcome. why go this way instead of the other? (clarification why go deeper into bash instead of trying to posix-ify the scripts) Because we are never going to eliminate arrays from rc.conf. Well, it may still be beneficial to some. The scripts could be used with a different style rc.conf in other environments. Then it will not be Arch. Depends on what you define as Arch. I've heard that Arch is what you make of it. Hehe. I don't know if you could disqualify it from a difference of one file. For me, part of it is that bash is used pretty ubiquitously as the configuration and scripting language of choice. Changing that to posix sh in one of the main config files would be a big shift.
Re: [arch-general] Bashification of initscripts for moderate speedup
Actually I see the point of doing this. Arch is a modern distribution with the newest software around so why stuck with shell constructs which are probably dozens of years old? Lukas
Re: [arch-general] Bashification of initscripts for moderate speedup
On Mon, Jun 28, 2010 at 11:03 AM, Victor Lowther victor.lowt...@gmail.com wrote: For me, part of it is that bash is used pretty ubiquitously as the configuration and scripting language of choice. Changing that to posix sh in one of the main config files would be a big shift. there seems to be a shift to dash as the main system shell in some distro's these days... and as openrc show's it's possible to do things similar to how arch does it (and gentoo used to) without bash. -- Caleb Cushing http://xenoterracide.blogspot.com
Re: [arch-general] Bashification of initscripts for moderate speedup
On 06/28/10 09:35, Victor Lowther wrote: On Jun 28, 2010, at 7:42 AM, Caleb Cushing xenoterrac...@gmail.com wrote: On Sun, Jun 27, 2010 at 11:10 PM, Victor Lowther victor.lowt...@gmail.com wrote: Questions, comments, flames, etc. welcome. why go this way instead of the other? (clarification why go deeper into bash instead of trying to posix-ify the scripts) ...and the main source of slowdowns in the boot sequence is I/O based -- mounting filesystems and launching programs. IIRC, Busybox shell can get notable speed boost by incorporating versions of tools like sed into the same busybox executable, such that it often doesn't have to fork and load other short-lived programs. (I believe it doesn't do bash-array-syntax.) -Isaac
Re: [arch-general] Bashification of initscripts for moderate speedup
On Mon, Jun 28, 2010 at 12:36 PM, Isaac Dupree m...@isaac.cedarswampstudios.org wrote: IIRC, Busybox shell can get notable speed boost by incorporating versions of tools like sed into the same busybox executable, such that it often doesn't have to fork and load other short-lived programs. I don't think it loads them into it... it provides them... for example running busybox vi will load busybox's version of vi. Doesn't mean this isn't a good thing... symlinking vi to busybox and then running that vi is the same as typing busybox vi. -- Caleb Cushing http://xenoterracide.blogspot.com
Re: [arch-general] Bashification of initscripts for moderate speedup
On Mon 28 Jun 2010 10:03 -0500, Victor Lowther wrote: On Jun 28, 2010, at 9:49 AM, Loui Chang louipc@gmail.com wrote: Depends on what you define as Arch. I've heard that Arch is what you make of it. Hehe. I don't know if you could disqualify it from a difference of one file. For me, part of it is that bash is used pretty ubiquitously as the configuration and scripting language of choice. Changing that to posix sh in one of the main config files would be a big shift. Ah. For me Arch is much more than bash configs or scripts. They could all become POSIX compatible and I'd still see it as Arch. If Arch started to require python, or more I might start to see it differently - it would be harder to maintain a slimmer system.
[arch-general] Bashification of initscripts for moderate speedup
I have spent some time over the last month or so rewriting the Arch initscripts in idiomatic bash -- since they rely on bash-specific features, trying to keep them fairly POSIX is rather a waste of time and a net performance loss -- using bash-style conditionals is about 30% faster when nothing needs to be touched on the filesystem and about 200% easier to read, and you can use bash regexps in place of most trivial uses of sed, grep, and awk when mangling parameters or variables -- the fewer short-lived processes we fork the better. You can browse changes I have made in the git repo @ http://git.fnordovax.org/arch-initscripts/log/?h=bashification and you can checkout my changes to play around with from git://fnordovax.org/~victor/arch-initscripts There is a gigantic whitespace fixup patch at the end, all it does is fix all the whitespace errors I introduced while hacking and reformat everything to a consistent indentation style (emacs shell-script mode, if you are interested). I have tested these changes on a VM and on my local laptop. I inlcuded a PKGBUILD that will build an initscripts-git package, but be sure to backup and restore your rc.*local, inittab, and rc.conf files because they will get nuked the first time you install it. Questions, comments, flames, etc. welcome. I can also spam the list with patches if y'all really want. -- Victor Lowther LPIC2 UCP RHCE
