Re: speed up port compiling using RAM (tmpfs) ???
Peter Jeremy <[EMAIL PROTECTED]> writes: > On Sat, 2006-Jan-21 14:30:57 -0600, Matthew D. Fuller wrote: > > Yes, but portupgrade and friends already do most of that, so they can > > upgrade stuff "in order". > Actually, they rely on there being an up-to-date INDEX file and build > their own dependency database from that. porteasy doesn't; it parses the Makefile to attempt to detect master / slave relationships, then uses make -V to obtain fetch / build / lib / run dependencies. Replacing make with a specialized tool with a more expressive syntax would make this much easier, and much faster. DES -- Dag-Erling Smørgrav - [EMAIL PROTECTED] ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: speed up port compiling using RAM (tmpfs) ???
On Sun, Jan 22, 2006 at 08:09:56AM +1100 I heard the voice of Peter Jeremy, and lo! it spake thus: > > Given that a port's dependency tree can depend on the options it is > invoked with, it would be nicer if the dependency tree was generated > dynamically, rather than pulled out of the latest INDEX file. I'm pretty sure it _is_, since portupgrade finds things related to OPTIONS and such for me, and I don't blow multiple hours on INDEX builds. I'm pretty sure it uses all-depends-list (or one of its siblings). -- Matthew Fuller (MF4839) | [EMAIL PROTECTED] Systems/Network Administrator | http://www.over-yonder.net/~fullermd/ On the Internet, nobody can hear you scream. ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: speed up port compiling using RAM (tmpfs) ???
On Sat, 2006-Jan-21 14:30:57 -0600, Matthew D. Fuller wrote: >On Sat, Jan 21, 2006 at 03:23:21PM -0500 I heard the voice of >Kris Kennaway, and lo! it spake thus: >> On Sat, Jan 21, 2006 at 10:07:39AM -0600, Matthew D. Fuller wrote: >> > >> > This is something that may be easier to: >> > >> > 3) Implement in portupgrade or portmanager or some such higher-level >> >tool in a language that gives a little more flexibility than make, >> >and which is already apparently pulling in most of the information >> >it may need to do the job. >> >> You still have the same issue as 1). > > [ 1 == building dependancy tree to know what depends on what ] > >Yes, but portupgrade and friends already do most of that, so they can >upgrade stuff "in order". Actually, they rely on there being an up-to-date INDEX file and build their own dependency database from that. Actually building the INDEX file is non-trivial (it takes roughly an hour for me). Tools like p5-FreeBSD-Portindex-1.4 cache intermediate output from "make index" but still have the up-front "make index" cost (and the documentation recommends a full "make index" regularly). You can save time by fetching the INDEX, but then you can't be certain that it matches your ports tree or your port options. Given that a port's dependency tree can depend on the options it is invoked with, it would be nicer if the dependency tree was generated dynamically, rather than pulled out of the latest INDEX file. If the INDEX dependencies are used to generate a parallel build tree then it's still important that the actual build process has interlocks to prevent unforeseen dependencies causing clashes. -- Peter Jeremy ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: speed up port compiling using RAM (tmpfs) ???
On Sat, Jan 21, 2006 at 03:23:21PM -0500 I heard the voice of Kris Kennaway, and lo! it spake thus: > On Sat, Jan 21, 2006 at 10:07:39AM -0600, Matthew D. Fuller wrote: > > > > This is something that may be easier to: > > > > 3) Implement in portupgrade or portmanager or some such higher-level > >tool in a language that gives a little more flexibility than make, > >and which is already apparently pulling in most of the information > >it may need to do the job. > > You still have the same issue as 1). [ 1 == building dependancy tree to know what depends on what ] Yes, but portupgrade and friends already do most of that, so they can upgrade stuff "in order". The biggest thing it seems like portupgrade (which is the only one I'm personally familiar with) lacks is that it doesn't of itself find out "which of these dependancies are already installed", and lets the ports tree itself recurse down. It sounds, from reading the emails, like the script dougb has been putting together does this, though. Given that capability, and the information portupgrade builds (from all-depends-list, I think?) to determine which order to upgrade things in, it seems like it would have right there most of what it needs. There are still issues like "after you start building something and it does the make config" and the like to handle (as well as terminal arbitration issues with multiple possibly interactive compiles going at once), of course. Not an easy or trivial thing to do even with all that, certainly, but probably easier in perl/ruby/C/etc than in make... -- Matthew Fuller (MF4839) | [EMAIL PROTECTED] Systems/Network Administrator | http://www.over-yonder.net/~fullermd/ On the Internet, nobody can hear you scream. ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: speed up port compiling using RAM (tmpfs) ???
On Sat, Jan 21, 2006 at 10:07:39AM -0600, Matthew D. Fuller wrote: > [ Cc trim a bit ] > > On Fri, Jan 20, 2006 at 08:53:11PM -0500 I heard the voice of > Kris Kennaway, and lo! it spake thus: > > > > In order to do better you either have to: > > This is something that may be easier to: > > 3) Implement in portupgrade or portmanager or some such higher-level >tool in a language that gives a little more flexibility than make, >and which is already apparently pulling in most of the information >it may need to do the job. You still have the same issue as 1). Kris pgpzJKQnkm0kC.pgp Description: PGP signature
Re: speed up port compiling using RAM (tmpfs) ???
[ Cc trim a bit ] On Fri, Jan 20, 2006 at 08:53:11PM -0500 I heard the voice of Kris Kennaway, and lo! it spake thus: > > In order to do better you either have to: This is something that may be easier to: 3) Implement in portupgrade or portmanager or some such higher-level tool in a language that gives a little more flexibility than make, and which is already apparently pulling in most of the information it may need to do the job. -- Matthew Fuller (MF4839) | [EMAIL PROTECTED] Systems/Network Administrator | http://www.over-yonder.net/~fullermd/ On the Internet, nobody can hear you scream. ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: speed up port compiling using RAM (tmpfs) ???
On Fri, Jan 20, 2006 at 08:36:17PM -0500, Sergey Babkin wrote: > > If (as I said) you impose the correct dependency information. > > Currently there is no such information provided. > > Ah, so we don't have any reliable information about dependencies > between the ports either (not just between files inside each > particular port)? Hm, I think it would present a problem even > when building them sequentially. The port dependency checking of one port upon other ports doesn't really use the make(1) dependency system. Instead there is a shell script in bsd.port.mk that looks for the file listed in the *_DEPENDS variable and goes and explicitly spawns a new make to build the port if it is not found. This is fine for building ports serially, but it's not trivially parallelizable since it only has local knowledge: In order to do better you either have to: 1a) construct an enormous make(1) dependency web of the entire ports collection (in order to have global knowledge and be able to direct makes to avoid collisions), and 1b) somehow keep it up-to-date each time the ports tree is modified. or 2) Introduce mutual exclusion to prevent conflicts from two makes trying to operate on the same port at once. You also need to serialize in other places, e.g. modifying the database state in /var/db. This won't be as efficient as 1) since the second make will block instead of going off to build some other dependency in the meantime, which you could do with 1) if you could manage to implement it. Kris pgp6LclSDGMdd.pgp Description: PGP signature
Re: speed up port compiling using RAM (tmpfs) ???
Kris Kennaway wrote: > > On Fri, Jan 20, 2006 at 04:54:33PM -0500, Sergey Babkin wrote: > > Kris Kennaway wrote: > > > > > > On Fri, Jan 20, 2006 at 11:25:33AM -0600, Sergey Babkin wrote: > > > > >From: =?ISO646-US?Q?Dag-Erling_Sm=3Frgrav?= <[EMAIL PROTECTED]> > > > > > > > > >Gary Thorpe <[EMAIL PROTECTED]> writes: > > > > >> This effectively means that you cannot take advantage of SMP to > > > > >> compile FreeBSD's ports collection. That sounds like a big > > > > >> limitation...especially for people trying to speed up bulk builds. > > > > > > > > > >We cannot be held responsible for race conditions in the Makefiles of > > > > >third-party software. > > > > > > > > Well, maybe we can then build multiple ports in parallel. > > > > I guess the way to do it would be to run the top-level make with > > > > -j but then disable it when calling the makefiles of the > > > > individual ports. Not that I have any idea how to actually > > > > do that. > > > > > > It's harder than that, because you need to impose dependency > > > information and mutual exclusion between different makes. e.g. they > > > can't both be compiling the same port at the same time, which will > > > happen if you just do the naive thing. > > > > That's the part that "make -j" is supposed to take care of, > > since it should build in parallel only the targets independent > > of each other. > > If (as I said) you impose the correct dependency information. > Currently there is no such information provided. Ah, so we don't have any reliable information about dependencies between the ports either (not just between files inside each particular port)? Hm, I think it would present a problem even when building them sequentially. -SB ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: speed up port compiling using RAM (tmpfs) ???
On Fri, 2006-Jan-20 16:54:33 -0500, Sergey Babkin wrote: >Kris Kennaway wrote: >> It's harder than that, because you need to impose dependency >> information and mutual exclusion between different makes. e.g. they >> can't both be compiling the same port at the same time, which will >> happen if you just do the naive thing. > >That's the part that "make -j" is supposed to take care of, >since it should build in parallel only the targets independent >of each other. It doesn't quite work like that. A single make execution will correctly parallelize independent targets (as long as the makefile dependency tree is correct - which is not true for all ports). The problem occurs when the targets are sub-makes. In this case, you have multiple sub-makes running in parallel with no knowledge of the dependency trees of the other sub-makes. Unless the top-level makefile has full knowledge of all the dependencies (which is not practical), it is quite likely that the sub-makes will collide. -- Peter Jeremy ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: speed up port compiling using RAM (tmpfs) ???
On Fri, Jan 20, 2006 at 03:28:08PM -0800, Gayn Winters wrote: > > [mailto:[EMAIL PROTECTED] On Behalf Of Kris Kennaway > > Sent: Friday, January 20, 2006 11:47 AM > > To: Gary Thorpe > > Cc: Wesley Shields; Ashok Shrestha; Brandon Flowers; Kris > > Kennaway; Mike Meyer; freebsd-hackers@freebsd.org; Dag-Erling Sm?rgrav > > Subject: Re: speed up port compiling using RAM (tmpfs) ??? > > > > > > On Fri, Jan 20, 2006 at 11:49:29AM -0500, Gary Thorpe wrote: > > > > > >-j is not safe to use with port builds since many ported software > > > >contain race conditions in the build. > > > > > > > >Kris > > > > > > This effectively means that you cannot take advantage of SMP to > compile > > > FreeBSD's ports collection. That sounds like a big > > > limitation...especially for people trying to speed up bulk builds. > > > > Yeah, but what do you propose to do about it? We have 14000 ports > > that need to be inspected for build race conditions and fixed before > > you can turn on -j by default. > > > > Kris > > > With a dual processor (even dual core?) is there a way to assign the big > make to one processor and to assign the foreground activity (email, > edits, etc) to another? Is SMP somehow smart enough to do this > automagically? Yes, this is what will happen, more or less. Kris pgpZMaai7Amfh.pgp Description: PGP signature
RE: speed up port compiling using RAM (tmpfs) ???
> [mailto:[EMAIL PROTECTED] On Behalf Of Kris Kennaway > Sent: Friday, January 20, 2006 11:47 AM > To: Gary Thorpe > Cc: Wesley Shields; Ashok Shrestha; Brandon Flowers; Kris > Kennaway; Mike Meyer; freebsd-hackers@freebsd.org; Dag-Erling Sm?rgrav > Subject: Re: speed up port compiling using RAM (tmpfs) ??? > > > On Fri, Jan 20, 2006 at 11:49:29AM -0500, Gary Thorpe wrote: > > > >-j is not safe to use with port builds since many ported software > > >contain race conditions in the build. > > > > > >Kris > > > > This effectively means that you cannot take advantage of SMP to compile > > FreeBSD's ports collection. That sounds like a big > > limitation...especially for people trying to speed up bulk builds. > > Yeah, but what do you propose to do about it? We have 14000 ports > that need to be inspected for build race conditions and fixed before > you can turn on -j by default. > > Kris With a dual processor (even dual core?) is there a way to assign the big make to one processor and to assign the foreground activity (email, edits, etc) to another? Is SMP somehow smart enough to do this automagically? -gayn ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: speed up port compiling using RAM (tmpfs) ???
On Fri, Jan 20, 2006 at 04:54:33PM -0500, Sergey Babkin wrote: > Kris Kennaway wrote: > > > > On Fri, Jan 20, 2006 at 11:25:33AM -0600, Sergey Babkin wrote: > > > >From: =?ISO646-US?Q?Dag-Erling_Sm=3Frgrav?= <[EMAIL PROTECTED]> > > > > > > >Gary Thorpe <[EMAIL PROTECTED]> writes: > > > >> This effectively means that you cannot take advantage of SMP to > > > >> compile FreeBSD's ports collection. That sounds like a big > > > >> limitation...especially for people trying to speed up bulk builds. > > > > > > > >We cannot be held responsible for race conditions in the Makefiles of > > > >third-party software. > > > > > > Well, maybe we can then build multiple ports in parallel. > > > I guess the way to do it would be to run the top-level make with > > > -j but then disable it when calling the makefiles of the > > > individual ports. Not that I have any idea how to actually > > > do that. > > > > It's harder than that, because you need to impose dependency > > information and mutual exclusion between different makes. e.g. they > > can't both be compiling the same port at the same time, which will > > happen if you just do the naive thing. > > That's the part that "make -j" is supposed to take care of, > since it should build in parallel only the targets independent > of each other. If (as I said) you impose the correct dependency information. Currently there is no such information provided. kris pgpof8ZmKKyl2.pgp Description: PGP signature
Re: speed up port compiling using RAM (tmpfs) ???
Kris Kennaway wrote: > > On Fri, Jan 20, 2006 at 11:25:33AM -0600, Sergey Babkin wrote: > > >From: =?ISO646-US?Q?Dag-Erling_Sm=3Frgrav?= <[EMAIL PROTECTED]> > > > > >Gary Thorpe <[EMAIL PROTECTED]> writes: > > >> This effectively means that you cannot take advantage of SMP to > > >> compile FreeBSD's ports collection. That sounds like a big > > >> limitation...especially for people trying to speed up bulk builds. > > > > > >We cannot be held responsible for race conditions in the Makefiles of > > >third-party software. > > > > Well, maybe we can then build multiple ports in parallel. > > I guess the way to do it would be to run the top-level make with > > -j but then disable it when calling the makefiles of the > > individual ports. Not that I have any idea how to actually > > do that. > > It's harder than that, because you need to impose dependency > information and mutual exclusion between different makes. e.g. they > can't both be compiling the same port at the same time, which will > happen if you just do the naive thing. That's the part that "make -j" is supposed to take care of, since it should build in parallel only the targets independent of each other. -SB ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: speed up port compiling using RAM (tmpfs) ???
On Sat, Jan 21, 2006 at 07:52:20AM +1100, Peter Jeremy wrote:5C > IMHO, the biggest problem (as des pointed out) is that there's nothing > to prevent two makes attempting to build the same port (this can > easily happen when both ports A and B depend on port C). One possible > solution would be to create another status file in the work directory > and hold a file lock (flock/lockf) on it whilst a make is in progress. > Any parallel attempt to make that port would block. > > If the above was implemented, an enhancement would then be to process > the port's dependency list in parallel, rather than serially. This > would allow a make that was blocked on one dependency to continue with > a different dependency. There have been a couple of half-hearted attempts to implement this, but no complete ones (see an old portmgr PR for details + discussion). I'd welcome a complete version :) Kris pgptDcR8KK7mK.pgp Description: PGP signature
Re: speed up port compiling using RAM (tmpfs) ???
On Fri, 2006-Jan-20 14:47:00 -0500, Kris Kennaway wrote: >On Fri, Jan 20, 2006 at 11:49:29AM -0500, Gary Thorpe wrote: > >> >-j is not safe to use with port builds since many ported software >> >contain race conditions in the build. >> > >> >Kris >> >> This effectively means that you cannot take advantage of SMP to compile >> FreeBSD's ports collection. That sounds like a big >> limitation...especially for people trying to speed up bulk builds. > >Yeah, but what do you propose to do about it? We have 14000 ports >that need to be inspected for build race conditions and fixed before >you can turn on -j by default. There are some areas of the FreeBSD port building infrastructure that could be enhanced to make it possible to run multiple top-level makes in parallel, even if individual ports can't be build with '-j'. IMHO, the biggest problem (as des pointed out) is that there's nothing to prevent two makes attempting to build the same port (this can easily happen when both ports A and B depend on port C). One possible solution would be to create another status file in the work directory and hold a file lock (flock/lockf) on it whilst a make is in progress. Any parallel attempt to make that port would block. If the above was implemented, an enhancement would then be to process the port's dependency list in parallel, rather than serially. This would allow a make that was blocked on one dependency to continue with a different dependency. -- Peter Jeremy ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: speed up port compiling using RAM (tmpfs) ???
On Fri, Jan 20, 2006 at 11:49:29AM -0500, Gary Thorpe wrote: > >-j is not safe to use with port builds since many ported software > >contain race conditions in the build. > > > >Kris > > This effectively means that you cannot take advantage of SMP to compile > FreeBSD's ports collection. That sounds like a big > limitation...especially for people trying to speed up bulk builds. Yeah, but what do you propose to do about it? We have 14000 ports that need to be inspected for build race conditions and fixed before you can turn on -j by default. Kris pgpbkaEytid09.pgp Description: PGP signature
Re: Re: speed up port compiling using RAM (tmpfs) ???
On Fri, Jan 20, 2006 at 11:25:33AM -0600, Sergey Babkin wrote: > >From: =?ISO646-US?Q?Dag-Erling_Sm=3Frgrav?= <[EMAIL PROTECTED]> > > >Gary Thorpe <[EMAIL PROTECTED]> writes: > >> This effectively means that you cannot take advantage of SMP to > >> compile FreeBSD's ports collection. That sounds like a big > >> limitation...especially for people trying to speed up bulk builds. > > > >We cannot be held responsible for race conditions in the Makefiles of > >third-party software. > > Well, maybe we can then build multiple ports in parallel. > I guess the way to do it would be to run the top-level make with > -j but then disable it when calling the makefiles of the > individual ports. Not that I have any idea how to actually > do that. It's harder than that, because you need to impose dependency information and mutual exclusion between different makes. e.g. they can't both be compiling the same port at the same time, which will happen if you just do the naive thing. Kris pgp6U8a9JihqQ.pgp Description: PGP signature
Re: Re: speed up port compiling using RAM (tmpfs) ???
>From: =?ISO646-US?Q?Dag-Erling_Sm=3Frgrav?= <[EMAIL PROTECTED]> >Gary Thorpe <[EMAIL PROTECTED]> writes: >> This effectively means that you cannot take advantage of SMP to >> compile FreeBSD's ports collection. That sounds like a big >> limitation...especially for people trying to speed up bulk builds. > >We cannot be held responsible for race conditions in the Makefiles of >third-party software. Well, maybe we can then build multiple ports in parallel. I guess the way to do it would be to run the top-level make with -j but then disable it when calling the makefiles of the individual ports. Not that I have any idea how to actually do that. -SB ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: speed up port compiling using RAM (tmpfs) ???
Gary Thorpe <[EMAIL PROTECTED]> writes: > This effectively means that you cannot take advantage of SMP to > compile FreeBSD's ports collection. That sounds like a big > limitation...especially for people trying to speed up bulk builds. We cannot be held responsible for race conditions in the Makefiles of third-party software. > Is "-pipe" still a good idea? It should provide a significant > performance boost, if the process is spending a lot of time doing disk > I/O, without using up a lot of memory. -pipe is fine. DES -- Dag-Erling Smørgrav - [EMAIL PROTECTED] ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: speed up port compiling using RAM (tmpfs) ???
Kris Kennaway wrote: On Thu, Jan 19, 2006 at 05:32:58PM -0500, Gary Thorpe wrote: Ashok Shrestha wrote: I mounted part of RAM as such: mdmfs -s 500m md /mnt Then put WRKDIRPREFIX=/path/to/md in /etc/make.conf. It substantially reduces compile time by about 5-10 times. Thanx to all ur replies. -Ashok Shrestha An alternative is to try using the "-pipe" flag with GCC: this eliminates the need to use some temporary files by using a unix pipe for IPC. Setting another flag "-j 2" will allow 2 jobs to be done at the same time and should eliminate (by hiding) the I/O delays (this slows down the CPU slightly because of more task switches, but even with one job there are still lots of generated tasks). -j is not safe to use with port builds since many ported software contain race conditions in the build. Kris This effectively means that you cannot take advantage of SMP to compile FreeBSD's ports collection. That sounds like a big limitation...especially for people trying to speed up bulk builds. Is "-pipe" still a good idea? It should provide a significant performance boost, if the process is spending a lot of time doing disk I/O, without using up a lot of memory. ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: speed up port compiling using RAM (tmpfs) ???
Wesley Shields <[EMAIL PROTECTED]> writes: > I think he is trying to get at a scenario where WRKDIR is on a seperate > disk from the one /usr/ports is on. There is no performance advantage in doing that. I can only see two possible reasons for pointing WRKDIRPREFIX to another disk: - insufficient space on /usr (but then again, you could just move the ports tree to that other disk and mount it on /usr/ports) - /usr/ports on NFS (I do that myself to facilitate testing changes on different platforms) DES -- Dag-Erling Smørgrav - [EMAIL PROTECTED] ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: speed up port compiling using RAM (tmpfs) ???
On Thu, Jan 19, 2006 at 05:32:58PM -0500, Gary Thorpe wrote: > Ashok Shrestha wrote: > >I mounted part of RAM as such: > > > >mdmfs -s 500m md /mnt > > > >Then put WRKDIRPREFIX=/path/to/md in /etc/make.conf. > > > >It substantially reduces compile time by about 5-10 times. > > > > > >Thanx to all ur replies. > > > >-Ashok Shrestha > > An alternative is to try using the "-pipe" flag with GCC: this > eliminates the need to use some temporary files by using a unix pipe for > IPC. Setting another flag "-j 2" will allow 2 jobs to be done at the > same time and should eliminate (by hiding) the I/O delays (this slows > down the CPU slightly because of more task switches, but even with one > job there are still lots of generated tasks). -j is not safe to use with port builds since many ported software contain race conditions in the build. Kris pgpviV0QJrOwr.pgp Description: PGP signature
Re: speed up port compiling using RAM (tmpfs) ???
Ashok Shrestha wrote: I mounted part of RAM as such: mdmfs -s 500m md /mnt Then put WRKDIRPREFIX=/path/to/md in /etc/make.conf. It substantially reduces compile time by about 5-10 times. Thanx to all ur replies. -Ashok Shrestha An alternative is to try using the "-pipe" flag with GCC: this eliminates the need to use some temporary files by using a unix pipe for IPC. Setting another flag "-j 2" will allow 2 jobs to be done at the same time and should eliminate (by hiding) the I/O delays (this slows down the CPU slightly because of more task switches, but even with one job there are still lots of generated tasks). If you combine these, you should see your compilations becoming CPU-bound. ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: speed up port compiling using RAM (tmpfs) ???
I mounted part of RAM as such: mdmfs -s 500m md /mnt Then put WRKDIRPREFIX=/path/to/md in /etc/make.conf. It substantially reduces compile time by about 5-10 times. Thanx to all ur replies. -Ashok Shrestha On 1/19/06, Wesley Shields <[EMAIL PROTECTED]> wrote: > On Thu, Jan 19, 2006 at 05:54:02PM +0100, Dag-Erling Sm?rgrav wrote: > > Mike Meyer <[EMAIL PROTECTED]> writes: > > > Will using a swap-backed disk change anything? > > > > Not really. > > > > > How about the best way to configure things to use two disks for the > > > compile? > > > > I'm not sure what you are trying to achieve. Unlike the base system, > > the ports tree does not use separate source and object directories. > > I think he is trying to get at a scenario where WRKDIR is on a seperate > disk from the one /usr/ports is on. > > To answer his question, assuming this is what he is going, for why not > just add a new physical disk to the system per the handbook, and set > WRKDIR to be where ever that disk is mounted. > > -- WXS > -- Ashok Shrestha ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: speed up port compiling using RAM (tmpfs) ???
On Thu, Jan 19, 2006 at 05:54:02PM +0100, Dag-Erling Sm?rgrav wrote: > Mike Meyer <[EMAIL PROTECTED]> writes: > > Will using a swap-backed disk change anything? > > Not really. > > > How about the best way to configure things to use two disks for the > > compile? > > I'm not sure what you are trying to achieve. Unlike the base system, > the ports tree does not use separate source and object directories. I think he is trying to get at a scenario where WRKDIR is on a seperate disk from the one /usr/ports is on. To answer his question, assuming this is what he is going, for why not just add a new physical disk to the system per the handbook, and set WRKDIR to be where ever that disk is mounted. -- WXS ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: speed up port compiling using RAM (tmpfs) ???
Mike Meyer <[EMAIL PROTECTED]> writes: > Will using a swap-backed disk change anything? Not really. > How about the best way to configure things to use two disks for the > compile? I'm not sure what you are trying to achieve. Unlike the base system, the ports tree does not use separate source and object directories. DES -- Dag-Erling Smørgrav - [EMAIL PROTECTED] ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: speed up port compiling using RAM (tmpfs) ???
Ashok Shrestha <[EMAIL PROTECTED]> writes: > I am curious to know if there is a way to compile a port such as X11 > or KDE faster. > > I know in Gentoo, you can mount a part of RAM and compile in that. > This substantially decreases the compile time. Reference: > http://gentoo-wiki.com/TIP_Speeding_up_portage_with_tmpfs > > Does anyone know how to do this in Freebsd? I doubt it would help. You get faster disk accesses, but significantly reduce the amount of memory available for the buffer cache. DES -- Dag-Erling Smørgrav - [EMAIL PROTECTED] ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: speed up port compiling using RAM (tmpfs) ???
On Sunday 15 January 2006 18:15, Ashok Shrestha wrote: > I am curious to know if there is a way to compile a port such as X11 > or KDE faster. > > I know in Gentoo, you can mount a part of RAM and compile in that. > This substantially decreases the compile time. Reference: > http://gentoo-wiki.com/TIP_Speeding_up_portage_with_tmpfs > > Does anyone know how to do this in Freebsd? Make a RAM drive using mdconfig and the mount it somewhere. Then put WRKDIRPREFIX=/path/to/md in /etc/make.conf -- Daniel O'Connor software and network engineer for Genesis Software - http://www.gsoft.com.au "The nice thing about standards is that there are so many of them to choose from." -- Andrew Tanenbaum GPG Fingerprint - 5596 B766 97C0 0E94 4347 295E E593 DC20 7B3F CE8C pgp2RLZaVYRSf.pgp Description: PGP signature
Re: speed up port compiling using RAM (tmpfs) ???
you can mount a small memory filesystem think it's called mbfs or something and change the work dir to that then you should be able to compile KDE using ram instead of the HD > -BEGIN PGP SIGNED MESSAGE- > Hash: SHA1 > > Ashok Shrestha wrote: >> Hi, >> >> I am curious to know if there is a way to compile a port such as X11 >> or KDE faster. >> >> I know in Gentoo, you can mount a part of RAM and compile in that. >> This substantially decreases the compile time. Reference: >> http://gentoo-wiki.com/TIP_Speeding_up_portage_with_tmpfs >> >> Does anyone know how to do this in Freebsd? >> >> -- >> Ashok Shrestha > > You can also take a look at devel/ccache and devel/distcc from ports. > > - --niki > > > -BEGIN PGP SIGNATURE- > Version: GnuPG v1.4.2 (MingW32) > Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org > > iD8DBQFDym6JHNAJ/fLbfrkRAuV5AKCw01ZCh5/wmc5cBxXsY2NaOGCR6ACfc1VN > 7Tx/hA8eUmS65P0Nf0tvF3Y= > =uOVv > -END PGP SIGNATURE- > ___ > freebsd-hackers@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-hackers > To unsubscribe, send any mail to "[EMAIL PROTECTED]" > ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: speed up port compiling using RAM (tmpfs) ???
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Ashok Shrestha wrote: > Hi, > > I am curious to know if there is a way to compile a port such as X11 > or KDE faster. > > I know in Gentoo, you can mount a part of RAM and compile in that. > This substantially decreases the compile time. Reference: > http://gentoo-wiki.com/TIP_Speeding_up_portage_with_tmpfs > > Does anyone know how to do this in Freebsd? > > -- > Ashok Shrestha You can also take a look at devel/ccache and devel/distcc from ports. - --niki -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.2 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFDym6JHNAJ/fLbfrkRAuV5AKCw01ZCh5/wmc5cBxXsY2NaOGCR6ACfc1VN 7Tx/hA8eUmS65P0Nf0tvF3Y= =uOVv -END PGP SIGNATURE- ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: speed up port compiling using RAM (tmpfs) ???
日曜日 15 1月 2006 16:45、Ashok Shrestha さんは書きました: > Hi, > > I am curious to know if there is a way to compile a port such as X11 > or KDE faster. > > I know in Gentoo, you can mount a part of RAM and compile in that. > This substantially decreases the compile time. Reference: > http://gentoo-wiki.com/TIP_Speeding_up_portage_with_tmpfs > > Does anyone know how to do this in Freebsd? Sure. Read the ports(7) man page paying special attention to the WRKDIRPREFIX variable. Then man mount_mfs and mdconfig. Those should do the trick. Eric -- The signature is a location used to give a personalised feel to each E-mail without having to personalise each E-mail. pgpEJMReFz5sW.pgp Description: PGP signature
Re: speed up port compiling using RAM (tmpfs) ???
On Sun, Jan 15, 2006 at 02:45:30AM -0500, Ashok Shrestha wrote: > Hi, > > I am curious to know if there is a way to compile a port such as X11 > or KDE faster. > > I know in Gentoo, you can mount a part of RAM and compile in that. > This substantially decreases the compile time. Reference: > http://gentoo-wiki.com/TIP_Speeding_up_portage_with_tmpfs > > Does anyone know how to do this in Freebsd? You should take a look at mdconfig(8) and ports(7). With mdconfig you create the ram-based disk and with WRKDIRPREFIX you tell the ports to use that disk instead of the default workdir. -- La prueba mas fehaciente de que existe vida inteligente en otros planetas, es que no han intentado contactar con nosotros. ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "[EMAIL PROTECTED]"
