Re: [gentoo-user] Howto speed up compilations
[EMAIL PROTECTED] wrote: http://gentoo-wiki.com/TIP_Speeding_up_portage That's a portage optimization rather than a compilation optimization. If you're doing portage optimization then you might also want to put /var/cache/edb/dep on a special filesystem. In fact, portage supports depcache plugins for further optimization. Zac -- gentoo-user@gentoo.org mailing list
Re: [gentoo-user] Howto speed up compilations
On Monday 18 July 2005 19:07, John J. Foster wrote: > Good afternoon all, > > A few weeks ago I read in one of the newgroups a way to greatly decrease > compilation times. The author noted that this was particularly noticable > when working with something like OO. The general jist of it was to > create temporary file system in memory and mount your portage tmpdir > there. For the life of me, I can't find that thread anymore. Does anyone > do something similar to this? Are there noticable gains to be had. I > have an Athlon 2800XP and 1 GB ram. > > Thanks, > John Yes Sure :) Here John read This Sektion -> http://gentoo-wiki.com/TIP_Speeding_up_portage -- gentoo-user@gentoo.org mailing list
Re: [gentoo-user] Howto speed up compilations
On 7/18/05, John J. Foster <[EMAIL PROTECTED]> wrote: > A few weeks ago I read in one of the newgroups a way to greatly decrease > compilation times. The author noted that this was particularly noticable > when working with something like OO. The general jist of it was to > create temporary file system in memory and mount your portage tmpdir > there. For the life of me, I can't find that thread anymore. Does anyone > do something similar to this? Are there noticable gains to be had. I > have an Athlon 2800XP and 1 GB ram. I am not sure if this will give a tremendous speedup. Granted, the source files won't need to be read from disk, which is an advantage, however, the file reading time should be very small compared to the time it takes for the compiler to translate the source code into machine code. Also, there's the ammount of memory you will lose, memory that could be used by the compiler. In some cases, gcc can eat very big chunks of memory. And if you use -pipe in your cflags, the gcc output isn't really written to disk during the various stages of compilation. Instead, it's piped through the processes. Well... maybe someone will give you some accurate results. Just my 2c. -- Bruno Lustosa, aka Lofofora | Email: [EMAIL PROTECTED] Network Administrator/Web Programmer | ICQ: 1406477 Rio de Janeiro - Brazil | -- gentoo-user@gentoo.org mailing list
Re: [gentoo-user] Howto speed up compilations
On Mon, Jul 18, 2005 at 01:07:38PM -0400, John J. Foster wrote: > Good afternoon all, > > A few weeks ago I read in one of the newgroups a way to greatly decrease > compilation times. The author noted that this was particularly noticable > when working with something like OO. The general jist of it was to > create temporary file system in memory and mount your portage tmpdir > there. For the life of me, I can't find that thread anymore. Does anyone > do something similar to this? Are there noticable gains to be had. I > have an Athlon 2800XP and 1 GB ram. > > Thanks, > John OK, this happens all the time. I search, can't find what I want, post a question, search again, and there it is. This is not the "thread" I was talking about, but it was right in front of me on the Gentoo Wiki. http://gentoo-wiki.com/TIP_Speeding_up_portage_with_tmpfs Does anyone have any "tips" on these "tips"? John > > -- > Contrary to the lie machine, the world is not safer. -- Contrary to the lie machine, the world is not safer. pgpOCzlVjdruu.pgp Description: PGP signature
Re: [gentoo-user] Howto speed up compilations
That sounded nice, if you can come with a "cluster" for compilation you may see good results, I've never used distcc but a friend got three athlon xps running and did it, he was able to install the three gentoos within a day (wich is more than I could ever expect after my 2 days stage1 install with no x or openoffice). On 7/18/05, Colin <[EMAIL PROTECTED]> wrote: > Neil Bothwick wrote: > > >On Mon, 18 Jul 2005 14:14:41 -0300, Bruno Lustosa wrote: > > > > > > > >>>A few weeks ago I read in one of the newgroups a way to greatly > >>>decrease compilation times. The author noted that this was > >>>particularly noticable when working with something like OO. The > >>>general jist of it was to create temporary file system in memory and > >>>mount your portage tmpdir there. For the life of me, I can't find > >>>that thread anymore. Does anyone do something similar to this? Are > >>>there noticable gains to be had. I have an Athlon 2800XP and 1 GB ram. > >>> > >>> > >>I am not sure if this will give a tremendous speedup. Granted, the > >>source files won't need to be read from disk, which is an advantage, > >>however, the file reading time should be very small compared to the > >>time it takes for the compiler to translate the source code into > >>machine code. > >>Also, there's the ammount of memory you will lose, memory that could > >>be used by the compiler. In some cases, gcc can eat very big chunks of > >>memory. > >> > >> > > > >Not to mention the OOo ebuild needing around 3GB of space in TMPDIR, so > >this approach would only result in the emerge failing quicker. > > > > > Not if you've got a machine with more than 3 GB of memory. A dual-proc > Power Mac G5 can handle up to 8 GB of physical RAM. If you did this > trick on one of those, you might see some serious improvement! But with > most PC's being limited (by the x86 and motherboard designs) to 2 GB of > physical RAM, it wouldn't work with large apps. > > A good suggestion would be to grab some old computers, Gentoo-ize them, > network them over 100BaseTX or Gigabit and make a little distcc farm. > Plus, you can charge people if they want to come over and rent your > computing power. (Virginia Tech does that with their "System X," 1,100 > dual-2.3GHz-processor XServe G5's.) :-) > -- > Colin > -- > gentoo-user@gentoo.org mailing list > > -- Daniel da Veiga Computer Operator - RS - Brazil -- gentoo-user@gentoo.org mailing list
Re: [gentoo-user] Howto speed up compilations
On Mon, 18 Jul 2005 14:14:41 -0300, Bruno Lustosa wrote: > > A few weeks ago I read in one of the newgroups a way to greatly > > decrease compilation times. The author noted that this was > > particularly noticable when working with something like OO. The > > general jist of it was to create temporary file system in memory and > > mount your portage tmpdir there. For the life of me, I can't find > > that thread anymore. Does anyone do something similar to this? Are > > there noticable gains to be had. I have an Athlon 2800XP and 1 GB ram. > > I am not sure if this will give a tremendous speedup. Granted, the > source files won't need to be read from disk, which is an advantage, > however, the file reading time should be very small compared to the > time it takes for the compiler to translate the source code into > machine code. > Also, there's the ammount of memory you will lose, memory that could > be used by the compiler. In some cases, gcc can eat very big chunks of > memory. Not to mention the OOo ebuild needing around 3GB of space in TMPDIR, so this approach would only result in the emerge failing quicker. -- Neil Bothwick Capt'n! The spellchecker kinna take this abuse! pgpQFIWR2sIPC.pgp Description: PGP signature
Re: [gentoo-user] Howto speed up compilations
Neil Bothwick wrote: On Mon, 18 Jul 2005 14:14:41 -0300, Bruno Lustosa wrote: A few weeks ago I read in one of the newgroups a way to greatly decrease compilation times. The author noted that this was particularly noticable when working with something like OO. The general jist of it was to create temporary file system in memory and mount your portage tmpdir there. For the life of me, I can't find that thread anymore. Does anyone do something similar to this? Are there noticable gains to be had. I have an Athlon 2800XP and 1 GB ram. I am not sure if this will give a tremendous speedup. Granted, the source files won't need to be read from disk, which is an advantage, however, the file reading time should be very small compared to the time it takes for the compiler to translate the source code into machine code. Also, there's the ammount of memory you will lose, memory that could be used by the compiler. In some cases, gcc can eat very big chunks of memory. Not to mention the OOo ebuild needing around 3GB of space in TMPDIR, so this approach would only result in the emerge failing quicker. Not if you've got a machine with more than 3 GB of memory. A dual-proc Power Mac G5 can handle up to 8 GB of physical RAM. If you did this trick on one of those, you might see some serious improvement! But with most PC's being limited (by the x86 and motherboard designs) to 2 GB of physical RAM, it wouldn't work with large apps. A good suggestion would be to grab some old computers, Gentoo-ize them, network them over 100BaseTX or Gigabit and make a little distcc farm. Plus, you can charge people if they want to come over and rent your computing power. (Virginia Tech does that with their "System X," 1,100 dual-2.3GHz-processor XServe G5's.) :-) -- Colin -- gentoo-user@gentoo.org mailing list
Re: [gentoo-user] Howto speed up compilations
On Mon, Jul 18, 2005 at 02:40:18PM -0400, Colin wrote: > Plus, you can charge people if they want to come over and rent your > computing power. (Virginia Tech does that with their "System X," 1,100 > dual-2.3GHz-processor XServe G5's.) :-) As I'm a little short of this total right now, I think I'll stick with the -pipe option for now ;-) John -- Contrary to the lie machine, the world is not safer. pgpplUmibjrZX.pgp Description: PGP signature
Re: [gentoo-user] Howto speed up compilations
On 13:07 Mon 18 Jul , John J. Foster wrote: > Good afternoon all, > > A few weeks ago I read in one of the newgroups a way to greatly decrease > compilation times. The author noted that this was particularly noticable > when working with something like OO. The general jist of it was to > create temporary file system in memory and mount your portage tmpdir > there. For the life of me, I can't find that thread anymore. Does anyone > do something similar to this? Are there noticable gains to be had. I > have an Athlon 2800XP and 1 GB ram. > > Thanks, > John > There was a thread about ways of speeding up compilation on gentoo-dev (including what you mentioned) a while ago. Good look compiling OOo with $PORTAGE_TMPDIR mounted in ram though ;) Dave -- gentoo-user@gentoo.org mailing list
Re: [gentoo-user] Howto speed up compilations
Zac Medico wrote: John J. Foster wrote: OK, this happens all the time. I search, can't find what I want, post a question, search again, and there it is. This is not the "thread" I was talking about, but it was right in front of me on the Gentoo Wiki. http://gentoo-wiki.com/TIP_Speeding_up_portage_with_tmpfs Does anyone have any "tips" on these "tips"? I sincerely doubt that this technique is worth the trouble because the Linux virtual memory system will automatically cache frequently accessed files in ram. The tmpfs will eliminate filesystem overhead and will certainly result in a reduction of total build time. However, I'd go with Bruno's recommendation and just put -pipe in the cflags. For builds that work well in parallel, which most large builds do, distcc is a great way to speed things up. Zac I'm a bit dubious about this well...unless you have a huge amount of memory (as in, more than 2G). But even then, if you have sufficient RAM, most of the source files should still be in memory as a result of the extraction of the source archive before compilation. And if buffers get flushed/recycled as a result of the compilation, that generally means that the compiler needed a large amount of memory, and giving the compiler your RAM is a much better choice, as otherwise swap has to be used, and that will *kill* your compilation time! That isn't to say that I don't think some smart and simple things beyond -pipe can speed up builds. I believe /var should be it's own filesystem, about 5G in size, and positioned just before /usr if you have one, or just after / if not. The filesystem should be either XFS (caches files in memory very agressively, and in fact doesn't even update the disk for short-lived temporary files) or reiserfs v3 (creates and deletes files very very quickly). However, I can pretty much guarantee that nothing short of a 16-processor, 16GB system (or a distcc farm, as Colin suggested) will allow you to "emerge -Dv --emptytree kde-meta" in 15 minutes!!! There is simply no getting around the CPU and memory bandwidth required for compiling! -Richard -- gentoo-user@gentoo.org mailing list
Re: [gentoo-user] Howto speed up compilations
John J. Foster wrote: OK, this happens all the time. I search, can't find what I want, post a question, search again, and there it is. This is not the "thread" I was talking about, but it was right in front of me on the Gentoo Wiki. http://gentoo-wiki.com/TIP_Speeding_up_portage_with_tmpfs Does anyone have any "tips" on these "tips"? I sincerely doubt that this technique is worth the trouble because the Linux virtual memory system will automatically cache frequently accessed files in ram. The tmpfs will eliminate filesystem overhead and will certainly result in a reduction of total build time. However, I'd go with Bruno's recommendation and just put -pipe in the cflags. For builds that work well in parallel, which most large builds do, distcc is a great way to speed things up. Zac -- gentoo-user@gentoo.org mailing list
