Double buffered cp(1)
Has anyone attempted to create a double buffered version of cp(1)? When copying from one disk to another, disk activity seems to ping-pong between the two, rather than keeping both active at the same time. If I were to fork and do something similar to afio, or maybe even doing something weird like using sendfile(it's faster than it sounds, and zero-copy), does anyone think I'd see any kind of speed boost? I'm effectively getting a little less than half the performance of just writing files filled with zero's, so I'm guessing this is where the bottleneck is, correct? -- Kevin To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: Double buffered cp(1)
* Kevin Day <[EMAIL PROTECTED]> [000421 20:24] wrote: > > Has anyone attempted to create a double buffered version of cp(1)? When > copying from one disk to another, disk activity seems to ping-pong between > the two, rather than keeping both active at the same time. > > If I were to fork and do something similar to afio, or maybe even doing > something weird like using sendfile(it's faster than it sounds, and > zero-copy), does anyone think I'd see any kind of speed boost? > > I'm effectively getting a little less than half the performance of just > writing files filled with zero's, so I'm guessing this is where the > bottleneck is, correct? extend (using truncate) and then mmap() the destination file, then read() directly into the mmap()'d portion. I'd like to see what numbers you get. :) -- -Alfred Perlstein - [[EMAIL PROTECTED]|[EMAIL PROTECTED]] "I have the heart of a child; I keep it in a jar on my desk." To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: Double buffered cp(1)
:extend (using truncate) and then mmap() the destination file, then :read() directly into the mmap()'d portion. : :I'd like to see what numbers you get. :) : :-- :-Alfred Perlstein - [[EMAIL PROTECTED]|[EMAIL PROTECTED]] Probably not so good considering the number of faults that will be taken. Also, if the destination filesystem runs out of room cp will take a random seg-fault trying to access the map. Plus the dirtying of that many VM pages will seriously effect performance. read + write is a better way to do it. It is still possible to double buffer. In this case simply create a small anonymous shared mmap that fits in the L2 cache (like 128K), setup a pipe, fork, and have one process read() from the source while the other write()s to the destination. The added overhead is actually less then 'one buffer copy' worth if the added buffering fits in the L1 or L2 cache. -Matt Matthew Dillon <[EMAIL PROTECTED]> To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: Double buffered cp(1)
> :extend (using truncate) and then mmap() the destination file, then > :read() directly into the mmap()'d portion. > : > :I'd like to see what numbers you get. :) > read + write is a better way to do it. It is still possible to > double buffer. In this case simply create a small anonymous shared > mmap that fits in the L2 cache (like 128K), setup a pipe, fork, and > have one process read() from the source while the other write()s to the > destination. The added overhead is actually less then 'one buffer copy' > worth if the added buffering fits in the L1 or L2 cache. It seems silly to implement something as trivial and straightforward as copying a file in userland. The process designated to copy a file just sits in a tight loop invoking the read()/write() syscalls repeatedly. Since this operation is already system bound and very simple, what's the arguement against absorbing it into the kernel? -MB To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: Double buffered cp(1)
: : :> :extend (using truncate) and then mmap() the destination file, then :> :read() directly into the mmap()'d portion. :> : :> :I'd like to see what numbers you get. :) : :> read + write is a better way to do it. It is still possible to :> double buffer. In this case simply create a small anonymous shared :> mmap that fits in the L2 cache (like 128K), setup a pipe, fork, and :> have one process read() from the source while the other write()s to the :> destination. The added overhead is actually less then 'one buffer copy' :> worth if the added buffering fits in the L1 or L2 cache. : :It seems silly to implement something as trivial and straightforward as :copying a file in userland. The process designated to copy a file just :sits in a tight loop invoking the read()/write() syscalls :repeatedly. Since this operation is already system bound and very simple, :what's the arguement against absorbing it into the kernel? : :-MB I don't think anyone has suggested that it be absorbed into the kernel. We are talking about userland code here. The argument for double-buffering is a simple one - it allows the process read()ing from the source file to block without stalling the process write()ing to the destination file. I think the reality, though, is that at least insofar as copying a single large file the source is going to be relatively contiguous on the disk and thus will tend not to block. More specifically, the disk itself is probably the bottleneck. Disk writes tend to be somewhat slower then disk reads and the seeking alone (between source file and destination file), even when using a large block size, will reduce performance drastically verses simply reading or writing a single file linearly. Double buffering may help a disk-to-disk file copy, but I doubt it will help a disk-to-same-disk file copy. -Matt Matthew Dillon <[EMAIL PROTECTED]> To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: Double buffered cp(1)
Matthew Dillon wrote: > > : > : > :> :extend (using truncate) and then mmap() the destination file, then > :> :read() directly into the mmap()'d portion. > :> : > :> :I'd like to see what numbers you get. :) > : > :> read + write is a better way to do it. It is still possible to > :> double buffer. In this case simply create a small anonymous shared > :> mmap that fits in the L2 cache (like 128K), setup a pipe, fork, and > :> have one process read() from the source while the other write()s to the > :> destination. The added overhead is actually less then 'one buffer copy' > :> worth if the added buffering fits in the L1 or L2 cache. > : > :It seems silly to implement something as trivial and straightforward as > :copying a file in userland. The process designated to copy a file just > :sits in a tight loop invoking the read()/write() syscalls > :repeatedly. Since this operation is already system bound and very simple, > :what's the arguement against absorbing it into the kernel? > : > :-MB > > I don't think anyone has suggested that it be absorbed into the kernel. > We are talking about userland code here. > > The argument for double-buffering is a simple one - it allows the > process read()ing from the source file to block without stalling the > process write()ing to the destination file. > > I think the reality, though, is that at least insofar as copying a > single large file the source is going to be relatively contiguous on > the disk and thus will tend not to block. More specifically, the > disk itself is probably the bottleneck. Disk writes tend to be > somewhat slower then disk reads and the seeking alone (between source > file and destination file), even when using a large block size, > will reduce performance drastically verses simply reading or writing > a single file linearly. Double buffering may help a disk-to-disk > file copy, but I doubt it will help a disk-to-same-disk file copy. I made some tests on my FreeBSD machine. In the past, double buffering only helps if you have concurrent I/O capability. You only have that if you have dual access to each I/O device (HD) via different data channels. We don't have that capability on PC's. The typical drives that we purchase have only one data path, i.e., the ribbon cable. I tested build worlds where I created my /usr/obj on one controller and left the /usr/src on a different controller. The buildworlds are pretty much I/O bound. They ran faster but not that much faster when different controllers were used. I have an IBM uw scsi on that system and I haven't tried to do a build world using it for one of the file systems. The test I like is iozone because it uses everything I normally use. I tell it to test using 160MB, which is 20+ times the available cache from top. Rawio doesn't mean anything when everything is cached. The tests showed the scsi system was slower than the UDMA-33 drives in everything except for randon I/O and then it was much faster. It could mean that the buffering logic on the scsi HD and scsi controller were smart enough that everything was in cache. It is too easy to split your obj and src filesystem on to different controllers and test if cached I/O between controllers helps on your system. I think that even cp is cached on a normal system and that means you already have "n" buffers available for reading and "N" buffers available for writing. Trying to make cp more complicated won't help because the copy of a file will still be twice the time to just write the file + the number of times that you had to wait for full revolutions of the disk before you could do your next scheduled read or write I/O operation. Kent > > -Matt > Matthew Dillon > <[EMAIL PROTECTED]> > > To Unsubscribe: send mail to [EMAIL PROTECTED] > with "unsubscribe freebsd-hackers" in the body of the message -- Kent Stewart Richland, WA mailto:[EMAIL PROTECTED] http://www.3-cities.com/~kstewart/index.html FreeBSD News http://daily.daemonnews.org/ SETI(Search for Extraterrestrial Intelligence) @ HOME http://setiathome.ssl.berkeley.edu/ To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: Double buffered cp(1)
On Sat, 22 Apr 2000, Matthew Dillon wrote: > > :extend (using truncate) and then mmap() the destination file, then > :read() directly into the mmap()'d portion. > : > :I'd like to see what numbers you get. :) > : > :-- > :-Alfred Perlstein - [[EMAIL PROTECTED]|[EMAIL PROTECTED]] > > Probably not so good considering the number of faults that will > be taken. Also, if the destination filesystem runs out of room > cp will take a random seg-fault trying to access the map. Plus > the dirtying of that many VM pages will seriously effect performance. Err... I think what Alfred (should have) meant was that you should mmap the source file and madvise it MADV_SEQUENTIAL, then write() to the new file directly from that. How bad do you foresee performance being then? -- Brian Fundakowski Feldman \ FreeBSD: The Power to Serve! / [EMAIL PROTECTED]`--' To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: Double buffered cp(1)
On Sat, 22 Apr 2000, Matthew Dillon wrote: > :> :extend (using truncate) and then mmap() the destination file, then > :> :read() directly into the mmap()'d portion. > :> : > :> :I'd like to see what numbers you get. :) > : > :> read + write is a better way to do it. It is still possible to > :> double buffer. In this case simply create a small anonymous shared > :> mmap that fits in the L2 cache (like 128K), setup a pipe, fork, and > :> have one process read() from the source while the other write()s to the > :> destination. The added overhead is actually less then 'one buffer copy' > :> worth if the added buffering fits in the L1 or L2 cache. > : > :It seems silly to implement something as trivial and straightforward as > :copying a file in userland. The process designated to copy a file just > :sits in a tight loop invoking the read()/write() syscalls > :repeatedly. Since this operation is already system bound and very simple, > :what's the arguement against absorbing it into the kernel? > : > :-MB > > I don't think anyone has suggested that it be absorbed into the kernel. > We are talking about userland code here. > > The argument for double-buffering is a simple one - it allows the > process read()ing from the source file to block without stalling the > process write()ing to the destination file. > > I think the reality, though, is that at least insofar as copying a > single large file the source is going to be relatively contiguous on > the disk and thus will tend not to block. More specifically, the > disk itself is probably the bottleneck. Disk writes tend to be > somewhat slower then disk reads and the seeking alone (between source > file and destination file), even when using a large block size, > will reduce performance drastically verses simply reading or writing > a single file linearly. Double buffering may help a disk-to-disk > file copy, but I doubt it will help a disk-to-same-disk file copy. Speaking about requential file read, the asynchronous read-ahead mechanism in the kernel already has the same effect as a double-buffering. In addition, real disks do prefetch data based on physical position and this also help when the file is not too fragmented. However, some bottleneck may exist when reads and writes transverse the same controller or involve a single device. This problem _is_ addressed by SCSI. The disconnection feature allows the BUS bandwidth not to be wasted and tagged command queuing allows to provide devices with several IO requests simultaneously. It is also addressed by ATA using the same mechanisms, but I doubt disconnections and tagged commands will ever be reliable enough to be actually usable on this interface given that it targets personnal computers that donnot require fast multi-streamed disk IOs. This let me think that: - User-space double-bufferred cp will not help at all given a decent IO sub-system and decent devices. - It will also not help when the controller and/or the device (as legacy IDE) just act as an IO bottleneck for cp (double bottleneck in case of reading and writing to the same disk ;-) ). By experience, connecting a real hard disk like a Cheetah to a real SCSI controller (LVD preferred) and using a real O/S help a lot better. ;-) Gérard. To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: Double buffered cp(1)
Gérard Roudier wrote: > > On Sat, 22 Apr 2000, Matthew Dillon wrote: > > > :> :extend (using truncate) and then mmap() the destination file, then > > :> :read() directly into the mmap()'d portion. > > :> : > > :> :I'd like to see what numbers you get. :) > > : > > :> read + write is a better way to do it. It is still possible to > > :> double buffer. In this case simply create a small anonymous shared > > :> mmap that fits in the L2 cache (like 128K), setup a pipe, fork, and > > :> have one process read() from the source while the other write()s to the > > :> destination. The added overhead is actually less then 'one buffer copy' > > :> worth if the added buffering fits in the L1 or L2 cache. > > : > > :It seems silly to implement something as trivial and straightforward as > > :copying a file in userland. The process designated to copy a file just > > :sits in a tight loop invoking the read()/write() syscalls > > :repeatedly. Since this operation is already system bound and very simple, > > :what's the arguement against absorbing it into the kernel? > > : > > :-MB > > > > I don't think anyone has suggested that it be absorbed into the kernel. > > We are talking about userland code here. > > > > The argument for double-buffering is a simple one - it allows the > > process read()ing from the source file to block without stalling the > > process write()ing to the destination file. > > > > I think the reality, though, is that at least insofar as copying a > > single large file the source is going to be relatively contiguous on > > the disk and thus will tend not to block. More specifically, the > > disk itself is probably the bottleneck. Disk writes tend to be > > somewhat slower then disk reads and the seeking alone (between source > > file and destination file), even when using a large block size, > > will reduce performance drastically verses simply reading or writing > > a single file linearly. Double buffering may help a disk-to-disk > > file copy, but I doubt it will help a disk-to-same-disk file copy. > > Speaking about requential file read, the asynchronous read-ahead mechanism > in the kernel already has the same effect as a double-buffering. In > addition, real disks do prefetch data based on physical position and this > also help when the file is not too fragmented. When I did my buildworld's on different IDE controllers, my flags were 0xa0ffa0ff on both controllers, which pretty much used everything the IDE drive could provide because of 32-bit transfers, UDMA-33, and 16 sector read aheads. Kent > > However, some bottleneck may exist when reads and writes transverse the > same controller or involve a single device. This problem _is_ addressed by > SCSI. The disconnection feature allows the BUS bandwidth not to be wasted > and tagged command queuing allows to provide devices with several IO > requests simultaneously. > It is also addressed by ATA using the same mechanisms, but I doubt > disconnections and tagged commands will ever be reliable enough to be > actually usable on this interface given that it targets personnal > computers that donnot require fast multi-streamed disk IOs. > > This let me think that: > - User-space double-bufferred cp will not help at all given a decent > IO sub-system and decent devices. > - It will also not help when the controller and/or the device (as legacy > IDE) just act as an IO bottleneck for cp (double bottleneck in case of > reading and writing to the same disk ;-) ). > > By experience, connecting a real hard disk like a Cheetah to a real SCSI > controller (LVD preferred) and using a real O/S help a lot better. ;-) > > Gérard. > > To Unsubscribe: send mail to [EMAIL PROTECTED] > with "unsubscribe freebsd-hackers" in the body of the message -- Kent Stewart Richland, WA mailto:[EMAIL PROTECTED] http://www.3-cities.com/~kstewart/index.html FreeBSD News http://daily.daemonnews.org/ SETI(Search for Extraterrestrial Intelligence) @ HOME http://setiathome.ssl.berkeley.edu/ Hunting Archibald Stewart, b 1802 in Ballymena, Antrim Co., NIR http://www.3-cities.com/~kstewart/genealogy/archibald_stewart.html To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: Double buffered cp(1)
:Err... I think what Alfred (should have) meant was that you should mmap :the source file and madvise it MADV_SEQUENTIAL, then write() to the new :file directly from that. How bad do you foresee performance being then? : :-- : Brian Fundakowski Feldman \ FreeBSD: The Power to Serve! / : [EMAIL PROTECTED]`--' Between 3.x and 4.x I doubled the VM fault readahead size, but we are still talking relatively small I/O's (around 64KB to 128KB), so you are still going to see a lot of faults. (The VM system in 4.x is also able to optimize reverse-indexed scans, is better able to discern random from sequential performance, and the heuristic is now partitioned and doesn't get confused by multiple accessors operating on the same file, but these features will have no impact on a simple file copy). Also,since the heuristic is already detecting sequential operation, setting MADV_SEQUENTIAL in this case isn't going to change anything. MADV_SEQUENTIAL is useful when you are jumping around a mapping enough to defeat the sequential detection, but still want the VM system to perform large-block I/O when it takes a fault. -Matt Matthew Dillon <[EMAIL PROTECTED]> To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: Double buffered cp(1)
: :I made some tests on my FreeBSD machine. In the past, double buffering :only helps if you have concurrent I/O capability. You only have that :if you have dual access to each I/O device (HD) via different data :channels. We don't have that capability on PC's. The typical drives :that we purchase have only one data path, i.e., the ribbon cable. This is true for IDE. This is not true for SCSI. While it is true that the ops have to run over the same physical cable, SCSI supports (and our device drivers use) disconnection and multiple-command queueing. This means that both reads and writes can be queued to the drive during the period where the drive is seeking or searching. Since command and data processing is asynchronous for all writes and read lookaheads, you generally get the benefit of concurrent I/O over a SCSI bus. This also means that adding multiple drives to a single SCSI bus tends to multiply the performance where as doing the same thing on an IDE bus results in little improvement. :I tested build worlds where I created my /usr/obj on one controller :and left the /usr/src on a different controller. The buildworlds are :pretty much I/O bound. They ran faster but not that much faster when Buildworlds are NOT I/O bound. They are *CPU* bound. This becomes glaringly obvious when you mount /usr/src and /usr/obj over NFS and look at the network traffic. :is cached. The tests showed the scsi system was slower than the :UDMA-33 drives in everything except for randon I/O and then it was :much faster. It could mean that the buffering logic on the scsi HD and :scsi controller were smart enough that everything was in cache. This is a degenerate test case. SCSI systems have slightly higher command processing overhead. Also, most IDE drives lie about write-completion (they return a success status before actually writing the data to the disk). This means that in tests where you are not stressing the disk subsystem (and a buildworld does NOT stress the disk subsystem!), IDE may appear to win out. -Matt :Kent : :-- :Kent Stewart :Richland, WA To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: Double buffered cp(1)
On Sat, 22 Apr 2000, Michael Bacarella wrote: > It seems silly to implement something as trivial and straightforward as > copying a file in userland. The process designated to copy a file just > sits in a tight loop invoking the read()/write() syscalls > repeatedly. Since this operation is already system bound and very simple, > what's the arguement against absorbing it into the kernel? > VMS To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: Double buffered cp(1)
Matthew Dillon wrote: > > :I tested build worlds where I created my /usr/obj on one controller > :and left the /usr/src on a different controller. The buildworlds are > :pretty much I/O bound. They ran faster but not that much faster when > > Buildworlds are NOT I/O bound. They are *CPU* bound. This becomes > glaringly obvious when you mount /usr/src and /usr/obj over NFS > and look at the network traffic. > > You are right but that is because I haven't started keeping record on 4.0-Stable and we were comparing apples and oranges. A buildworld of 3.4-Stable required around 2000u seconds using gcc-2.8.2 on my system. Setiathome, which is running at a nice of 19, still consumed 90% of the cpu. A buildworld on 4.0-Stable required 3500u seconds using gcc-2.95.2 and setiathome didn't accrue any appreciable cpu time during the build. There were definitely some changes there :). Kent -- Kent Stewart Richland, WA mailto:[EMAIL PROTECTED] http://www.3-cities.com/~kstewart/index.html FreeBSD News http://daily.daemonnews.org/ SETI(Search for Extraterrestrial Intelligence) @ HOME http://setiathome.ssl.berkeley.edu/ Hunting Archibald Stewart, b 1802 in Ballymena, Antrim Co., NIR http://www.3-cities.com/~kstewart/genealogy/archibald_stewart.html To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: Double buffered cp(1)
:You are right but that is because I haven't started keeping record on :4.0-Stable and we were comparing apples and oranges. A buildworld of :3.4-Stable required around 2000u seconds using gcc-2.8.2 on my system. :Setiathome, which is running at a nice of 19, still consumed 90% of :the cpu. A buildworld on 4.0-Stable required 3500u seconds using :gcc-2.95.2 and setiathome didn't accrue any appreciable cpu time :during the build. There were definitely some changes there :). : :Kent : :-- :Kent Stewart :Richland, WA Both 3.4 and 4.0 buildworlds are cpu-bound. If you are trying to test buildworlds, then don't run setiathome (or anything else) while doing the test... it will skew the results of your tests due to differences between the 3.4 and 4.x schedulers (specifically, various scheduler bugs were fixed in 4.x that effect niced cpu-bound background programs such as setiathome, giving them way, way too much cpu). It is simply impossible to fairly measure I/O performance in the presence of unrelated background-running programs, especially under 3.x. And even though 4.x does a better job of it, it will still skew the results. -Matt To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: Double buffered cp(1)
Matthew Dillon wrote: > > :You are right but that is because I haven't started keeping record on > :4.0-Stable and we were comparing apples and oranges. A buildworld of > :3.4-Stable required around 2000u seconds using gcc-2.8.2 on my system. > :Setiathome, which is running at a nice of 19, still consumed 90% of > :the cpu. A buildworld on 4.0-Stable required 3500u seconds using > :gcc-2.95.2 and setiathome didn't accrue any appreciable cpu time > :during the build. There were definitely some changes there :). > : > :Kent > : > :-- > :Kent Stewart > :Richland, WA > > Both 3.4 and 4.0 buildworlds are cpu-bound. If you are trying to test > buildworlds, then don't run setiathome (or anything else) while doing > the test... it will skew the results of your tests due to differences > between the 3.4 and 4.x schedulers (specifically, various scheduler > bugs were fixed in 4.x that effect niced cpu-bound background programs > such as setiathome, giving them way, way too much cpu). The bugs were fixed in 4.0? What I saw was far too much cpu going to setiathome on 3.4. Seti hardly ran on 4.0. I don't have quantities because I have only noticed the really large increase in cpu time required to build 4.0. The wall clock time on a buildworld hardly changed (55-60 minutes) whether I ran seti or not on 3.4. I can watch the wall clock time on some of my future builds and see how they are skewed by stopping seti before I being the buildworld. I just haven't got 4.0 to the capability I had with with 3.4 before I tried to upgrade to 4.0 and it died in the middle of creating the installkernel. The rest of the system was pretty much broken at that point and I used the opportunity to restructure everything. It has been a good check on some of the ports because I found a few that assumed you have things like Bison, automake, and autoconf installed and I didn't. > > It is simply impossible to fairly measure I/O performance in the > presence of unrelated background-running programs, especially under 3.x. > And even though 4.x does a better job of it, it will still skew the > results. I was looking at this as more of a real world setup simulation. Seti is almost pure cpu and the buildworld used everything else. I ran the build world from an x-term and from the command line. That didn't seem to matter much. The system also provided my dialup and nat'ing for my internal network. Seti was run from a script that was started from an login before I did my startx. It would chug along from one network problem at Berkeley until the next one with out any intervention on my part. The system was on a UPS and didn't go down on the occasional 1 or 2 second long power outages. Between the two codes, they mimic most of the types of calculations I've been associated with for many years. I have people that are using Windows 9x machines and I think they would be better off with something like FreeBSD. The programs were developed in unix environment. A lot of users are using Linux. Some are using PVM to combine systems into a single parallel virtual calculation. Lehey Fortran-90(77) running on Win9x with their protected mode interface setup has to be a terrible choice. The problem is proving it and providing an alternative :). A couple would run better on a scsi because of the queued read/write I/O that you identified. You can say anything is a POS but people won't listen unless you can show them a better way. I'm retired and no longer have a contract manager to answer to and can experiment. Cheers, Kent > > -Matt -- Kent Stewart Richland, WA mailto:[EMAIL PROTECTED] http://www.3-cities.com/~kstewart/index.html FreeBSD News http://daily.daemonnews.org/ SETI(Search for Extraterrestrial Intelligence) @ HOME http://setiathome.ssl.berkeley.edu/ Hunting Archibald Stewart, b 1802 in Ballymena, Antrim Co., NIR http://www.3-cities.com/~kstewart/genealogy/archibald_stewart.html To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: Double buffered cp(1)
:The bugs were fixed in 4.0? What I saw was far too much cpu going to :setiathome on 3.4. Seti hardly ran on 4.0. I don't have quantities :because I have only noticed the really large increase in cpu time :required to build 4.0. The wall clock time on a buildworld hardly :changed (55-60 minutes) whether I ran seti or not on 3.4. I can watch :the wall clock time on some of my future builds and see how they are I don't know what your setup is, Kent, but when I do a buildworld my system is 95% cpu bound, with virtual no idle time at all. It's all going to the buildworld. On both 3.x and 4.x. Try mounting /usr/obj with softupdates turned on, and if your /tmp is not a softupdates partition then make sure you are building with -pipe in /etc/make.conf: (For 3.x) CFLAGS= -O2 -pipe (For 4.x) CFLAGS= -Os -pipe :I was looking at this as more of a real world setup simulation. Seti :is almost pure cpu and the buildworld used everything else. I ran the :build world from an x-term and from the command line. That didn't seem :to matter much. The system also provided my dialup and nat'ing for my :internal network. Seti was run from a script that was started from an : login before I did my startx. It would chug along from one :network problem at Berkeley until the next one with out any Seti does almost no I/O, which means that in general it will have a much lower scheduling priority then even a cpu-bound buildworld. Most people run seti niced to +20, which means that it does not run at all if there are other 'normal' processes using the cpu. Under 3.x this was broken - seti got cpu when it shouldn't have. Under 4.x this is fixed. If you run a cpu-bound process niced to +20 it will get very little cpu in the face of other unniced cpu-bound processing. -Matt To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: Double buffered cp(1)
Matthew Dillon wrote: > > :The bugs were fixed in 4.0? What I saw was far too much cpu going to > :setiathome on 3.4. Seti hardly ran on 4.0. I don't have quantities > :because I have only noticed the really large increase in cpu time > :required to build 4.0. The wall clock time on a buildworld hardly > :changed (55-60 minutes) whether I ran seti or not on 3.4. I can watch > :the wall clock time on some of my future builds and see how they are > > I don't know what your setup is, Kent, but when I do a buildworld > my system is 95% cpu bound, with virtual no idle time at all. It's > all going to the buildworld. On both 3.x and 4.x. It is on 4.x but it wasn't on 3.4. At least, seti was taking time that it shouldn't have been getting. > > Try mounting /usr/obj with softupdates turned on, and if your /tmp > is not a softupdates partition then make sure you are building with > -pipe in /etc/make.conf: > > (For 3.x) > > CFLAGS= -O2 -pipe > > (For 4.x) > > CFLAGS= -Os -pipe I am trying this right now. I thought that optimizing this way was dangerous for the kernel. The problem for the new people is what works and when. You have to understand that Murphy sits on my shoulder. If there are 2+ bugs in a product, I will eventually end up seeing one of them :). I have been going to turn on softupdates but haven't got there yet. I want to link /usr/obj to the scsi drive first. Then, I think I will try softupdates. Try the worst combo's first and then add features but write the times down when you try the different arrangements. If 50% of my Cray throughput was write behind caching, then softupdates could be an equal share on FreeBSD. A benchmark that ran in four hours versus eight hours is a big change. I don't have a good batch system but the speed of my FreeBSD systems have a lot going for them clock wise. The Cray had 16MW (128MB) of memory and that is pretty much a starter system now days. Kent -- Kent Stewart Richland, WA mailto:[EMAIL PROTECTED] http://www.3-cities.com/~kstewart/index.html FreeBSD News http://daily.daemonnews.org/ SETI(Search for Extraterrestrial Intelligence) @ HOME http://setiathome.ssl.berkeley.edu/ Hunting Archibald Stewart, b 1802 in Ballymena, Antrim Co., NIR http://www.3-cities.com/~kstewart/genealogy/archibald_stewart.html To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: Double buffered cp(1)
:>
:> I don't know what your setup is, Kent, but when I do a buildworld
:> my system is 95% cpu bound, with virtual no idle time at all. It's
:> all going to the buildworld. On both 3.x and 4.x.
:
:It is on 4.x but it wasn't on 3.4. At least, seti was taking time that
:it shouldn't have been getting.
Right. I think we're in agreement, it's just getting lost somewhere :-)
:> (For 3.x)
:>
:> CFLAGS= -O2 -pipe
:>
:> (For 4.x)
:>
:> CFLAGS= -Os -pipe
:
:I am trying this right now. I thought that optimizing this way was
:dangerous for the kernel. The problem for the new people is what works
:and when. You have to understand that Murphy sits on my shoulder. If
:there are 2+ bugs in a product, I will eventually end up seeing one of
:them :).
-O2 and -Os are 'safe'. Other optimization levels (-O3 for example)
are *NOT* safe. Most machine-specific optimizations (-m options) should
generally be considered NOT safe.
:I have been going to turn on softupdates but haven't got there yet. I
:want to link /usr/obj to the scsi drive first. Then, I think I will
:try softupdates. Try the worst combo's first and then add features but
:write the times down when you try the different arrangements.
:...
:Kent
I also turn off atime updates ('noatime' mount option in /etc/fstab)
on /usr/src, since buildworlds do a lot of file scanning and there
is simply no reason to update every single file's inode with a new
atime. I doubt it makes much of a difference but it does leave
more clean buffer cache buffers available for other things.
-pipe makes a significant difference since without it every source
file being compiled creates several files in /tmp. Softupdates makes
a significant difference in its ability to asynchronize file creation
and meta-data updates (which helps /tmp as well as /usr/obj, though
/tmp is helped even more by using -pipe).
-Matt
Matthew Dillon
<[EMAIL PROTECTED]>
To Unsubscribe: send mail to [EMAIL PROTECTED]
with "unsubscribe freebsd-hackers" in the body of the message
Re: Double buffered cp(1)
Matthew Dillon <[EMAIL PROTECTED]> wrote: > -pipe makes a significant difference since without it every source > file being compiled creates several files in /tmp. Hasn't O'Brien recently said that in fact "-pipe" is already the default for our cc, so explicitly specifying the option doesn't do anything? -- Christian "naddy" Weisgerber [EMAIL PROTECTED] To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: Double buffered cp(1)
:Matthew Dillon <[EMAIL PROTECTED]> wrote: : :> -pipe makes a significant difference since without it every source :> file being compiled creates several files in /tmp. : :Hasn't O'Brien recently said that in fact "-pipe" is already the :default for our cc, so explicitly specifying the option doesn't do :anything? : :-- :Christian "naddy" Weisgerber [EMAIL PROTECTED] I don't recall him saying anything like that. There was an argument over making the -pipe the default a few months ago, but I was under the impression that it didn't happen because some people were concerned about the memory requirements nixing compiles on machines with small amounts of memory (since cpp, cc1, AND as wind up all running simultaniously). The documentation doesn't indicate that -pipe is turned on by default. If I ktrace a cc command it sure looks like it uses /tmp files when -pipe is not specified! So I would say that -pipe is NOT the default. -Matt Matthew Dillon <[EMAIL PROTECTED]> To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: Double buffered cp(1)
On 24/04 00:08, Christian Weisgerber wrote: > Hasn't O'Brien recently said that in fact "-pipe" is already the default > for our cc, so explicitly specifying the option doesn't do anything? Try compiling a 'hello world' program with and without the '-pipe' option but with '-v' switched on in both cases and compare the difference in the messages. best; gjvc -- George Cox <[EMAIL PROTECTED]> Tel +44 1235 544 127 To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: Double buffered cp(1)
George Cox wrote: > > On 24/04 00:08, Christian Weisgerber wrote: > > > Hasn't O'Brien recently said that in fact "-pipe" is already the default > > for our cc, so explicitly specifying the option doesn't do anything? > > Try compiling a 'hello world' program with and without the '-pipe' option > but with '-v' switched on in both cases and compare the difference in the > messages. This is what I see on a buildworld with 4.0-Stable Modified /etc/make.conf and commented out CFLAGS= -Os -pipe 3707.4u 799.6s 1:35:52.46 78.3% 1374+1477k 56974+173232io 2337pf+0w 3693.9u 800.5s 1:29:45.73 83.4% 1375+1477k 55201+173224io 2160pf+0w Modified /etc/make.conf and added CFLAGS= -pipe 3559.2u 807.2s 1:28:00.05 82.6% 1608+1286k 56499+174033io 2516pf+0w Kent > > best; > > gjvc > > -- > George Cox <[EMAIL PROTECTED]> > Tel +44 1235 544 127 > > To Unsubscribe: send mail to [EMAIL PROTECTED] > with "unsubscribe freebsd-hackers" in the body of the message -- Kent Stewart Richland, WA mailto:[EMAIL PROTECTED] http://www.3-cities.com/~kstewart/index.html FreeBSD News http://daily.daemonnews.org/ SETI(Search for Extraterrestrial Intelligence) @ HOME http://setiathome.ssl.berkeley.edu/ Hunting Archibald Stewart, b 1802 in Ballymena, Antrim Co., NIR http://www.3-cities.com/~kstewart/genealogy/archibald_stewart.html To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: Double buffered cp(1)
On Sat, 22 Apr 2000, Matthew Dillon wrote: [snip] > disk itself is probably the bottleneck. Disk writes tend to be > somewhat slower then disk reads and the seeking alone (between source > file and destination file), even when using a large block size, > will reduce performance drastically verses simply reading or writing > a single file linearly. Double buffering may help a disk-to-disk > file copy, but I doubt it will help a disk-to-same-disk file copy. > Wouldn't coping the file to another disk and then back to the original one than just a simple copy in some cases be faster then? After all, you are saving a lot of head seeks. > -Matt > Matthew Dillon > <[EMAIL PROTECTED]> > To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: Double buffered cp(1)
Narvi wrote: > > On Sat, 22 Apr 2000, Matthew Dillon wrote: > > [snip] > > > disk itself is probably the bottleneck. Disk writes tend to be > > somewhat slower then disk reads and the seeking alone (between source > > file and destination file), even when using a large block size, > > will reduce performance drastically verses simply reading or writing > > a single file linearly. Double buffering may help a disk-to-disk > > file copy, but I doubt it will help a disk-to-same-disk file copy. > > > > Wouldn't coping the file to another disk and then back to the original one > than just a simple copy in some cases be faster then? > > After all, you are saving a lot of head seeks. The problem is that if they are on the same data channel then you have conflicts for access to the data bus. Some products like SCSI are better suited for that environment but data still only flows one direction at a time. On the old ISA bus, you could just about watch data flow with an analyzer plugged on to a new interface card. We had one interface card that the data ready went true before the data was stored into the on_card memory. The cpu was fast enough that I could watch the bits in the memory change. I couldn't get good data until memory had stablized. Cray's were really good at handling data in 1988 because they had 100MB/s + 1000MB/s data channels. I'm not sure how wide the data channels were. It seems like the 100MB/s was 16 bytes wide and the 1000 MB/s was 32 bytes wide. Their high speed disks were striped to provide 20MB/s continuous transfers. Getting data off of the disk was far more important to throughput than writing the data. The queued read/write ability of the scsi drives made life better but the disk was still the bottle neck to high speed data. The PCI bus probably shares that honor on the modern PC. It is only 4 bytes wide and relatively slow. The new 64bit PCI bus will help but that is a long way from the 256 bit or 512 bit wide Cray data channels that Cray was using in 1988. A while back I was given some parameters that could change the time required for a 4.0-Stable buildworld. The only option that improved my buildworld time was "CFLAGS= -pipe". It saved about 140u seconds. The option "-Os" made a buildworld take 1/3 longer. I also did an installworld (and kernel) of a -Os system and that really didn't change the times. The jump for version 4.0 on a Celeron 433a was 3560u + 800s to 4950u + 800s. There are still some options to try but "softupdates" didn't help. The buildworld using an UW scsi for /usr/obj was actually a little bit longer than using a UDMA66 ATA HD. The scsi was flat out at 8MB/s where as the UDMA HD was capable of 12-14MB/s. The Maxtor UDMA drive had a 2MB cache and the IBM uw was much smaller. Kent -- Kent Stewart Richland, WA mailto:[EMAIL PROTECTED] http://www.3-cities.com/~kstewart/index.html FreeBSD News http://daily.daemonnews.org/ SETI(Search for Extraterrestrial Intelligence) @ HOME http://setiathome.ssl.berkeley.edu/ To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: Double buffered cp(1)
The standard PCI bus can do 130 MBytes/sec. Even with overhead issues (setup for a DMA burst) it can still do 100 MBytes/sec. A standard SCSI controller can do 40, 80, and now even 160 MBytes/sec over the wire - standard copper cabling w/ LVD connectors (example below). A modern hard disk can do 10-30 MBytes/sec to/from the platter, assuming no seeks. But the moment it needs to seek the performance drops drastically ... generally down to 1-5 MBytes/sec. So in the case of a file copy over a SCSI bus, the physical disk is almost always going to be the limiting factor. -Matt da0 at ahc0 bus 0 target 0 lun 0 da0: Fixed Direct Access SCSI-2 device da0: 80.000MB/s transfers (40.000MHz, offset 15, 16bit), Tagged Queueing Enabled da0: 4340MB (924 512 byte sectors: 255H 63S/T 553C) da1 at ahc2 bus 0 target 0 lun 0 da1: Fixed Direct Access SCSI-2 device da1: 40.000MB/s transfers (20.000MHz, offset 15, 16bit), Tagged Queueing Enabled da1: 17366MB (35566480 512 byte sectors: 255H 63S/T 2213C) da2 at ahc2 bus 0 target 1 lun 0 da2: Fixed Direct Access SCSI-2 device da2: 40.000MB/s transfers (20.000MHz, offset 15, 16bit), Tagged Queueing Enabled da2: 17366MB (35566480 512 byte sectors: 255H 63S/T 2213C) Mounting root from ufs:/dev/da0s1a To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: Double buffered cp(1)
Matthew Dillon wrote: > > The standard PCI bus can do 130 MBytes/sec. Even with overhead issues > (setup for a DMA burst) it can still do 100 MBytes/sec. But that depends on what is also going on at the same time. There are three other cards in my PCI bus. You can eliminate one because I wasn't using the sound card :). > > A standard SCSI controller can do 40, 80, and now even 160 MBytes/sec > over the wire - standard copper cabling w/ LVD connectors (example > below). This is where the cache size on the HD becomes important. It also acts like a hardware double buffering. > > A modern hard disk can do 10-30 MBytes/sec to/from the platter, assuming > no seeks. But the moment it needs to seek the performance drops > drastically ... generally down to 1-5 MBytes/sec. I haven't seen any 30MB/s. The 10K LVD IBM's were just about the fastest at 20MB/s continuous. The drop for UDMA drives is to even lower rates. > > So in the case of a file copy over a SCSI bus, the physical disk is > almost always going to be the limiting factor. I just noticed that mine isn't showing "Tagged Queueing Enabled" is that something I can set? The adapter is an Adaptec 2940uw. da0 at ahc0 bus 0 target 4 lun 0 da0: Fixed Direct Access SCSI-2 device da0: 40.000MB/s transfers (20.000MHz, offset 8, 16bit) da0: 4134MB (8467200 512 byte sectors: 255H 63S/T 527C) Kent -- Kent Stewart Richland, WA mailto:[EMAIL PROTECTED] http://www.3-cities.com/~kstewart/index.html FreeBSD News http://daily.daemonnews.org/ SETI(Search for Extraterrestrial Intelligence) @ HOME http://setiathome.ssl.berkeley.edu/ To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: Double buffered cp(1)
In the last episode (Apr 26), Kent Stewart said: > I just noticed that mine isn't showing "Tagged Queueing Enabled" is > that something I can set? The adapter is an Adaptec 2940uw. > > da0 at ahc0 bus 0 target 4 lun 0 > da0: Fixed Direct Access SCSI-2 device > da0: 40.000MB/s transfers (20.000MHz, offset 8, 16bit) > da0: 4134MB (8467200 512 byte sectors: 255H 63S/T 527C) The CAM code has a quirk entry for IBM DCAS drives that turns it off; check out /sys/cam/cam_xpt.c, line 320. I suggest benchmarking with and without the quirk entry, and if it's faster with tagged queueing, update PR kern/10398. -- Dan Nelson [EMAIL PROTECTED] To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: Double buffered cp(1)
:In the last episode (Apr 26), Kent Stewart said: :> I just noticed that mine isn't showing "Tagged Queueing Enabled" is :> that something I can set? The adapter is an Adaptec 2940uw. :> :> da0 at ahc0 bus 0 target 4 lun 0 :> da0: Fixed Direct Access SCSI-2 device :> da0: 40.000MB/s transfers (20.000MHz, offset 8, 16bit) :> da0: 4134MB (8467200 512 byte sectors: 255H 63S/T 527C) : :The CAM code has a quirk entry for IBM DCAS drives that turns it off; :check out /sys/cam/cam_xpt.c, line 320. I suggest benchmarking with :and without the quirk entry, and if it's faster with tagged queueing, :update PR kern/10398. : :-- : Dan Nelson : [EMAIL PROTECTED] I seem to remember a discussion last year about tagged commands being a lot slower on IBM drives being the reason why tagged queueing was turned off. I wonder if IBM has fixed the problem in their later drives? Note that tagging is different from disconnection. Tagging allows you to queue multiple commands (e.g. multiple reads & writes) to the drive. Disconnection is a bus-level protocol which allows the SCSI bus to be freed up after sending a command (e.g. like a read) so other SCSI devices can use the bus while the drive is seeking. Disconnection is more important. Tagging is nice in that it can absorb command queueing overheads and allows the drive to prioritize the transactions based on the characteristics of the media, but tagging is not going to make much of a difference when you only have one or two processes banging on the drive. Tagging shows its suds on more heavily loaded systems, when dozens of processes are banging on the disks. -Matt Matthew Dillon <[EMAIL PROTECTED]> To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: Double buffered cp(1)
> > A modern hard disk can do 10-30 MBytes/sec to/from the platter, assuming > > no seeks. But the moment it needs to seek the performance drops > > drastically ... generally down to 1-5 MBytes/sec. > > I haven't seen any 30MB/s. The 10K LVD IBM's were just about the > fastest at 20MB/s continuous. The drop for UDMA drives is to even > lower rates. The IBM DPTA-372730 will do 23 MB/s continuous - this is a 7200 RPM IDE drive with very high bit density. There are IDE drives that do more, and I certainly expect the new 36 GB 1" 10K drives to do more. > I just noticed that mine isn't showing "Tagged Queueing Enabled" is > that something I can set? The adapter is an Adaptec 2940uw. > > da0 at ahc0 bus 0 target 4 lun 0 > da0: Fixed Direct Access SCSI-2 device > da0: 40.000MB/s transfers (20.000MHz, offset 8, 16bit) > da0: 4134MB (8467200 512 byte sectors: 255H 63S/T 527C) The DCAS-34330 is quirked out - which may not always be sensible. See /sys/cam/cam_xpt.c. Steinar Haug, Nethelp consulting, [EMAIL PROTECTED] To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: Double buffered cp(1)
Dan Nelson wrote: > > In the last episode (Apr 26), Kent Stewart said: > > I just noticed that mine isn't showing "Tagged Queueing Enabled" is > > that something I can set? The adapter is an Adaptec 2940uw. > > > > da0 at ahc0 bus 0 target 4 lun 0 > > da0: Fixed Direct Access SCSI-2 device > > da0: 40.000MB/s transfers (20.000MHz, offset 8, 16bit) > > da0: 4134MB (8467200 512 byte sectors: 255H 63S/T 527C) > > The CAM code has a quirk entry for IBM DCAS drives that turns it off; > check out /sys/cam/cam_xpt.c, line 320. I suggest benchmarking with > and without the quirk entry, and if it's faster with tagged queueing, > update PR kern/10398. What would you suggest. Do you remove the entry or just change the tags field? It has mintags=0 and maxtags=0. The other entries have mintags=24 and maxtags=32. I 've have about 6 hours of buildworlds on it right now. Plus some other benchmarking. Kent > > -- > Dan Nelson > [EMAIL PROTECTED] > > To Unsubscribe: send mail to [EMAIL PROTECTED] > with "unsubscribe freebsd-hackers" in the body of the message -- Kent Stewart Richland, WA mailto:[EMAIL PROTECTED] http://www.3-cities.com/~kstewart/index.html FreeBSD News http://daily.daemonnews.org/ SETI(Search for Extraterrestrial Intelligence) @ HOME http://setiathome.ssl.berkeley.edu/ Hunting Archibald Stewart, b 1802 in Ballymena, Antrim Co., NIR http://www.3-cities.com/~kstewart/genealogy/archibald_stewart.html To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: Double buffered cp(1)
On Mon, 24 Apr 2000, Kent Stewart wrote: > This is what I see on a buildworld with 4.0-Stable > > Modified /etc/make.conf and commented out CFLAGS= -Os -pipe > 3707.4u 799.6s 1:35:52.46 78.3% 1374+1477k 56974+173232io 2337pf+0w > 3693.9u 800.5s 1:29:45.73 83.4% 1375+1477k 55201+173224io 2160pf+0w > Modified /etc/make.conf and added CFLAGS= -pipe > 3559.2u 807.2s 1:28:00.05 82.6% 1608+1286k 56499+174033io 2516pf+0w This is an old message, but what you're seeing here is that if CFLAGS is not overridden, it is set by sys.mk to "-O -pipe" Setting CFLAGS explicitly to "-pipe" is faster because it does no optimization, "-Os -pipe" would be slower because it does more. Leaving out -pipe would be slower still, because the compiler does data passing using temporary files in /tmp instead of via a pipe. Kris In God we Trust -- all others must submit an X.509 certificate. -- Charles Forsythe <[EMAIL PROTECTED]> To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
RE: Double buffered cp(1)
> > Leaving out -pipe would be slower still, because the > compiler does data passing using temporary files in /tmp > instead of via a pipe. > Unless this has been changed from 3.4 to 4.0, gcc defaults to /var/tmp. I never understood why, and the gcc manual page claims that it's /tmp (I think). MFS users, synchronize your TMPDIR variables ... now. :-) Kees Jan == You are only young once, but you can stay immature all your life To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
RE: Double buffered cp(1)
On Fri, 12 May 2000, Koster, K.J. wrote: > Unless this has been changed from 3.4 to 4.0, gcc defaults to /var/tmp. I > never understood why, and the gcc manual page claims that it's /tmp (I > think). MFS users, synchronize your TMPDIR variables ... now. :-) It did. Compiling a simple test program just now shows: + -rw--- 1 root wheel0 May 12 00:16 /tmp/ccl22910.i + -rw--- 1 root wheel0 May 12 00:16 /tmp/ccc22910.s + -rw--- 1 root wheel0 May 12 00:16 /tmp/ccP22910.o - -rw--- 1 root wheel0 May 12 00:16 /tmp/ccl22910.i - -rw--- 1 root wheel0 May 12 00:16 /tmp/ccc22910.s - -rw--- 1 root wheel0 May 12 00:16 /tmp/ccP22910.o (incidentally, another reason to use -pipe is that the above filenames are predictable and probably handled insecurely so that another user can cause any of your files to be overwritten when you compile something. This is on my list of things to fix). Kris In God we Trust -- all others must submit an X.509 certificate. -- Charles Forsythe <[EMAIL PROTECTED]> To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: Double buffered cp(1)
Kris Kennaway wrote: > > On Mon, 24 Apr 2000, Kent Stewart wrote: > > > This is what I see on a buildworld with 4.0-Stable > > > > Modified /etc/make.conf and commented out CFLAGS= -Os -pipe > > 3707.4u 799.6s 1:35:52.46 78.3% 1374+1477k 56974+173232io 2337pf+0w > > 3693.9u 800.5s 1:29:45.73 83.4% 1375+1477k 55201+173224io 2160pf+0w > > Modified /etc/make.conf and added CFLAGS= -pipe > > 3559.2u 807.2s 1:28:00.05 82.6% 1608+1286k 56499+174033io 2516pf+0w > > This is an old message, but what you're seeing here is that if CFLAGS is > not overridden, it is set by sys.mk to "-O -pipe" > > Setting CFLAGS explicitly to "-pipe" is faster because it does no > optimization, "-Os -pipe" would be slower because it does more. Leaving > out -pipe would be slower still, because the compiler does data passing > using temporary files in /tmp instead of via a pipe. Part of this you had to go back about 15-20 messages. There were some comments about options that would speed the system up. I then ran both styles of buildworlds on kernels built with the -Os to see if my buildworld times changed. It wasn't significant. A long about this same time I ran some tests with using this for IBM DCAS drive current setup. da0: Fixed Direct Access SCSI-2 device da0: 40.000MB/s transfers (20.000MHz, offset 8, 16bit), Tagged Queueing Enabled da0: 4134MB (8467200 512 byte sectors: 255H 63S/T 527C) Previously, the tagged queueing was turned off. I have run a number of tests with 4.0-Stable and enabling tagged queueing on this drive didn't slow the disk down. It really didn't speed it up to speak of either. Kent > > Kris > > > In God we Trust -- all others must submit an X.509 certificate. > -- Charles Forsythe <[EMAIL PROTECTED]> > > To Unsubscribe: send mail to [EMAIL PROTECTED] > with "unsubscribe freebsd-hackers" in the body of the message -- Kent Stewart Richland, WA mailto:[EMAIL PROTECTED] http://www.3-cities.com/~kstewart/index.html FreeBSD News http://daily.daemonnews.org/ SETI(Search for Extraterrestrial Intelligence) @ HOME http://setiathome.ssl.berkeley.edu/ Hunting Archibald Stewart, b 1802 in Ballymena, Antrim Co., NIR http://www.3-cities.com/~kstewart/genealogy/archibald_stewart.html To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: Double buffered cp(1)
Kris Kennaway writes: > On Fri, 12 May 2000, Koster, K.J. wrote: > > > Unless this has been changed from 3.4 to 4.0, gcc defaults to /var/tmp. I > > never understood why, and the gcc manual page claims that it's /tmp (I > > think). MFS users, synchronize your TMPDIR variables ... now. :-) > > It did. > > Compiling a simple test program just now shows: > > + -rw--- 1 root wheel0 May 12 00:16 /tmp/ccl22910.i > + -rw--- 1 root wheel0 May 12 00:16 /tmp/ccc22910.s > + -rw--- 1 root wheel0 May 12 00:16 /tmp/ccP22910.o > - -rw--- 1 root wheel0 May 12 00:16 /tmp/ccl22910.i > - -rw--- 1 root wheel0 May 12 00:16 /tmp/ccc22910.s > - -rw--- 1 root wheel0 May 12 00:16 /tmp/ccP22910.o > > (incidentally, another reason to use -pipe is that the above filenames are > predictable and probably handled insecurely so that another user can cause > any of your files to be overwritten when you compile something. This is > on my list of things to fix). Just use own subdirectory in /tmp with some anticrackers manoevres. see PR bin/18275 and http://www.links.ru/FreeBSD/mkinittmpdir/ which do this work. -- @BABOLO http://links.ru/ To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: Double buffered cp(1)
In message <[EMAIL PROTECTED]> Kris Kennaway writes: : (incidentally, another reason to use -pipe is that the above filenames are : predictable and probably handled insecurely so that another user can cause : any of your files to be overwritten when you compile something. This is : on my list of things to fix). This is one reason I have TMPDIR set to "." when I'm running as me. Warner To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: Double buffered cp(1)
Warner Losh writes: > In message <[EMAIL PROTECTED]> Kris >Kennaway writes: > : (incidentally, another reason to use -pipe is that the above filenames are > : predictable and probably handled insecurely so that another user can cause > : any of your files to be overwritten when you compile something. This is > : on my list of things to fix). > > This is one reason I have TMPDIR set to "." when I'm running as me. ...and lose a lot of files to delete... I propose create (or use old) subdirectory in /tmp on startup and use this subdirectory for set TMPDIR See PR bin/18275 -- @BABOLO http://links.ru/ To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: Double buffered cp(1)
In message <[EMAIL PROTECTED]> "Aleksandr A.Babaylov" writes: : > This is one reason I have TMPDIR set to "." when I'm running as me. : ...and lose a lot of files to delete... My sources are cvs controlled, and a cvs update -n tells me what to delete when gcc doesn't do it for me. And when I'm not running -pipe :-). : I propose create (or use old) subdirectory in /tmp on startup : and use this subdirectory for set TMPDIR : See PR bin/18275 I'll have to take a look a tthis, but my plate is very full these days. Warner To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
