Re: Replace/rewrite reverse.c for tail(1)
Charles Randall said: Out of 128M of ram, it's swapped nearly everything else out to keep 85M of this 400M file in ram, even though it will never touch it again. :) I see two possible fixes for this. One could be madvise'ing periodically with MADV_DONTNEED. If I understand correctly, this would help a bit, right? Or, mmap smaller regions of the file, and keep moving the buffer. This would also help with files exceeding mmap's limits. Any thoughts? Unless the ability has been removed recently, setting the physical memory soft limit can help a very little bit also: bash ulimit -m 16384 can make a program a little more friendly to it's neighbors. The soft (in both senses) limit is mushy, but provides some hints when paging ensues. -- John | Never try to teach a pig to sing, [EMAIL PROTECTED] | it makes one look stupid [EMAIL PROTECTED] | and it irritates the pig. To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: Replace/rewrite reverse.c for tail(1)
Charles Randall said: Out of 128M of ram, it's swapped nearly everything else out to keep 85M of this 400M file in ram, even though it will never touch it again. :) I see two possible fixes for this. One could be madvise'ing periodically with MADV_DONTNEED. If I understand correctly, this would help a bit, right? Or, mmap smaller regions of the file, and keep moving the buffer. This would also help with files exceeding mmap's limits. Any thoughts? Unless the ability has been removed recently, setting the physical memory soft limit can help a very little bit also: bash ulimit -m 16384 can make a program a little more friendly to it's neighbors. The soft (in both senses) limit is mushy, but provides some hints when paging ensues. -- John | Never try to teach a pig to sing, dy...@iquest.net | it makes one look stupid jdy...@nc.com | and it irritates the pig. To Unsubscribe: send mail to majord...@freebsd.org with unsubscribe freebsd-hackers in the body of the message
Re: Replace/rewrite reverse.c for tail(1)
Er, the original TAC was a BSD utility which was rewritten by Jay Lepreau at Utah (who also happens to be my boss)... The source for it that I have sitting around (1986) doesn't actually list a copyright, but I'm fairly sure that we're in control of the copyright for that version. The authors are listed as Unknown off the net long ago, and Jay Lepreau. :) It's likely to not be as fast as the newer gnu version, of course, but if you'd like, I can check on the copyright issue and plop you the source if it's OK. -Dave Lo and Behold, Kevin Day said: Because of licensing restrictions in our product, we are unable to ship with any GNU/GPL'ed tools, so I'm forced to fix 'tail' rather than use tac. (I saw tac, and agree that it is faster for this specific use) Any VM people wanna pipe up and make a suggestion so that I may make up a patch? Kevin [Charset iso-8859-1 unsupported, filtering to ASCII...] I'd suggest that you use tac from GNU textutils. Charles -Original Message- From: Kevin Day [mailto:toa...@dragondata.com] Sent: Wednesday, July 28, 1999 3:09 AM To: hack...@freebsd.org Subject: Replace/rewrite reverse.c for tail(1) An application I use quite often requires me to reverse the lines in the file to get the desired output. 'tail -r' appears to be very inefficient in it's use of mmap(). It mmap's the entire file in, which encourages the kernel to swap out the rest of the system to keep pages of the input file in memory. 58350 root 54 0 412M 85244K RUN 0:14 19.78% 19.19% tail Out of 128M of ram, it's swapped nearly everything else out to keep 85M of this 400M file in ram, even though it will never touch it again. :) I see two possible fixes for this. One could be madvise'ing periodically with MADV_DONTNEED. If I understand correctly, this would help a bit, right? Or, mmap smaller regions of the file, and keep moving the buffer. This would also help with files exceeding mmap's limits. Any thoughts? Kevin To Unsubscribe: send mail to majord...@freebsd.org with unsubscribe freebsd-hackers in the body of the message To Unsubscribe: send mail to majord...@freebsd.org with unsubscribe freebsd-hackers in the body of the message To Unsubscribe: send mail to majord...@freebsd.org with unsubscribe freebsd-hackers in the body of the message -- work: dande...@cs.utah.edu me: an...@pobox.com University of Utah CS Department http://www.angio.net/ If you haul a geek up a crack, you will bloody their fingers for a day... If you teach a geek to climb, you will bloody their fingers for life. To Unsubscribe: send mail to majord...@freebsd.org with unsubscribe freebsd-hackers in the body of the message
RE: Replace/rewrite reverse.c for tail(1)
I'd suggest that you use "tac" from GNU textutils. Charles -Original Message- From: Kevin Day [mailto:[EMAIL PROTECTED]] Sent: Wednesday, July 28, 1999 3:09 AM To: [EMAIL PROTECTED] Subject: Replace/rewrite reverse.c for tail(1) An application I use quite often requires me to reverse the lines in the file to get the desired output. 'tail -r' appears to be very inefficient in it's use of mmap(). It mmap's the entire file in, which encourages the kernel to swap out the rest of the system to keep pages of the input file in memory. 58350 root 54 0 412M 85244K RUN 0:14 19.78% 19.19% tail Out of 128M of ram, it's swapped nearly everything else out to keep 85M of this 400M file in ram, even though it will never touch it again. :) I see two possible fixes for this. One could be madvise'ing periodically with MADV_DONTNEED. If I understand correctly, this would help a bit, right? Or, mmap smaller regions of the file, and keep moving the buffer. This would also help with files exceeding mmap's limits. Any thoughts? Kevin To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: Replace/rewrite reverse.c for tail(1)
Because of licensing restrictions in our product, we are unable to ship with any GNU/GPL'ed tools, so I'm forced to fix 'tail' rather than use tac. (I saw tac, and agree that it is faster for this specific use) Any VM people wanna pipe up and make a suggestion so that I may make up a patch? Kevin [Charset iso-8859-1 unsupported, filtering to ASCII...] I'd suggest that you use "tac" from GNU textutils. Charles -Original Message- From: Kevin Day [mailto:[EMAIL PROTECTED]] Sent: Wednesday, July 28, 1999 3:09 AM To: [EMAIL PROTECTED] Subject: Replace/rewrite reverse.c for tail(1) An application I use quite often requires me to reverse the lines in the file to get the desired output. 'tail -r' appears to be very inefficient in it's use of mmap(). It mmap's the entire file in, which encourages the kernel to swap out the rest of the system to keep pages of the input file in memory. 58350 root 54 0 412M 85244K RUN 0:14 19.78% 19.19% tail Out of 128M of ram, it's swapped nearly everything else out to keep 85M of this 400M file in ram, even though it will never touch it again. :) I see two possible fixes for this. One could be madvise'ing periodically with MADV_DONTNEED. If I understand correctly, this would help a bit, right? Or, mmap smaller regions of the file, and keep moving the buffer. This would also help with files exceeding mmap's limits. Any thoughts? Kevin To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: Replace/rewrite reverse.c for tail(1)
:Because of licensing restrictions in our product, we are unable to ship with :any GNU/GPL'ed tools, so I'm forced to fix 'tail' rather than use tac. (I :saw tac, and agree that it is faster for this specific use) : :Any VM people wanna pipe up and make a suggestion so that I may make up a :patch? : :Kevin I don't think madvise()ing it MADV_DONTNEED will work right now. The madvise() calls only work with OBJT_DEFAULT and OBJT_SWAP objects -- i.e. swap-backed memory. They will not do anything to pages owned by file mmaps. We could fix vm/vm_object.c/vm_object_madvise() to handle MADV_DONTNEED for other objects. It would not be too difficult to do, actually, since we would be doing nothing more then moving the (must be clean) page to VM cache to get it to be reused more quickly. This is a fix I could make to CURRENT without too much trouble. -Matt To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: Replace/rewrite reverse.c for tail(1)
:Because of licensing restrictions in our product, we are unable to ship with :any GNU/GPL'ed tools, so I'm forced to fix 'tail' rather than use tac. (I :saw tac, and agree that it is faster for this specific use) : :Any VM people wanna pipe up and make a suggestion so that I may make up a :patch? : :Kevin I don't think madvise()ing it MADV_DONTNEED will work right now. The madvise() calls only work with OBJT_DEFAULT and OBJT_SWAP objects -- i.e. swap-backed memory. They will not do anything to pages owned by file mmaps. We could fix vm/vm_object.c/vm_object_madvise() to handle MADV_DONTNEED for other objects. It would not be too difficult to do, actually, since we would be doing nothing more then moving the (must be clean) page to VM cache to get it to be reused more quickly. This is a fix I could make to CURRENT without too much trouble. -Matt Wow, that's interesting. While I never looked at the code, I *thought* I was able to measure a speed boost in a certain large embedded application I'm working on, with adding some madvise()'s in to mmap'ed files. (Streaming video madvise'ed with MADV_SEQUENTIAL seemed to be slightly faster, but it may have been insignificant) This is a problem I've run into a few times before, actually. One-time use data that will shove the entire system out of ram, and no convenient way to tell the kernel that it shouldn't bother holding on to it. I believe dg ran into this too, and had some fix he was using on ftp.cdrom.com. In my application, when I play a movie, it'd be nice to be able to specify a priority. "Try to keep this, even if it means possibly swapping" (current behavior) or "Don't swap out anything for this object, it's far too large to bother with trying to hold on to it and/or it will only be used once." Granted 'large' is one of those fuzzy terms, but when you do know that the 900M file you're reading is being done sequentially from start to finish, swapping things out to try to keep more of it in swap is probably bad. :) Kevin To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: Replace/rewrite reverse.c for tail(1)
:Wow, that's interesting. While I never looked at the code, I *thought* I was :able to measure a speed boost in a certain large embedded application I'm :working on, with adding some madvise()'s in to mmap'ed files. (Streaming :video madvise'ed with MADV_SEQUENTIAL seemed to be slightly faster, but it :may have been insignificant) MADV_SEQUENTIAL *WILL* work. It changes the page fault read-behind/read-ahead. Normally the VM system reads behind a bit as well as reads ahead. If you set MADV_SEQUENTIAL it shifts to just doing read-ahead, and it does more of it. The madvise() types that just flag the VM object work on any type of object. The madvise() types that actually mess with VM pages currently only work on swap-backed pages. MADV_WILLNEED scans for VM pages in object MADV_DONTNEED scans for VM pages in object MADV_FREE scans for VM pages in object MADV_SEQUENTIAL just flags the object MADV_RANDOM just flags the object MADV_NORMAL just flags the object :Granted 'large' is one of those fuzzy terms, but when you do know that the :900M file you're reading is being done sequentially from start to finish, :swapping things out to try to keep more of it in swap is probably bad. :) : :Kevin I agree completely. When you run through the filesystem these pages will be reused more quickly. When you run through the VM system these pages are considered to be more 'active'. I think it makes sense to have madvise() MADV_WILLNEED and MADV_DONTNEED operate on file mmaps. There is a slight security concern with MADV_DONTNEED (i.e. if you loop running it on a system file that is used a lot, like /etc/group), but after thinking about it a bit I do not think it is a big deal. -Matt Matthew Dillon [EMAIL PROTECTED] To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
RE: Replace/rewrite reverse.c for tail(1)
I'd suggest that you use tac from GNU textutils. Charles -Original Message- From: Kevin Day [mailto:toa...@dragondata.com] Sent: Wednesday, July 28, 1999 3:09 AM To: hack...@freebsd.org Subject: Replace/rewrite reverse.c for tail(1) An application I use quite often requires me to reverse the lines in the file to get the desired output. 'tail -r' appears to be very inefficient in it's use of mmap(). It mmap's the entire file in, which encourages the kernel to swap out the rest of the system to keep pages of the input file in memory. 58350 root 54 0 412M 85244K RUN 0:14 19.78% 19.19% tail Out of 128M of ram, it's swapped nearly everything else out to keep 85M of this 400M file in ram, even though it will never touch it again. :) I see two possible fixes for this. One could be madvise'ing periodically with MADV_DONTNEED. If I understand correctly, this would help a bit, right? Or, mmap smaller regions of the file, and keep moving the buffer. This would also help with files exceeding mmap's limits. Any thoughts? Kevin To Unsubscribe: send mail to majord...@freebsd.org with unsubscribe freebsd-hackers in the body of the message To Unsubscribe: send mail to majord...@freebsd.org with unsubscribe freebsd-hackers in the body of the message
Re: Replace/rewrite reverse.c for tail(1)
Because of licensing restrictions in our product, we are unable to ship with any GNU/GPL'ed tools, so I'm forced to fix 'tail' rather than use tac. (I saw tac, and agree that it is faster for this specific use) Any VM people wanna pipe up and make a suggestion so that I may make up a patch? Kevin [Charset iso-8859-1 unsupported, filtering to ASCII...] I'd suggest that you use tac from GNU textutils. Charles -Original Message- From: Kevin Day [mailto:toa...@dragondata.com] Sent: Wednesday, July 28, 1999 3:09 AM To: hack...@freebsd.org Subject: Replace/rewrite reverse.c for tail(1) An application I use quite often requires me to reverse the lines in the file to get the desired output. 'tail -r' appears to be very inefficient in it's use of mmap(). It mmap's the entire file in, which encourages the kernel to swap out the rest of the system to keep pages of the input file in memory. 58350 root 54 0 412M 85244K RUN 0:14 19.78% 19.19% tail Out of 128M of ram, it's swapped nearly everything else out to keep 85M of this 400M file in ram, even though it will never touch it again. :) I see two possible fixes for this. One could be madvise'ing periodically with MADV_DONTNEED. If I understand correctly, this would help a bit, right? Or, mmap smaller regions of the file, and keep moving the buffer. This would also help with files exceeding mmap's limits. Any thoughts? Kevin To Unsubscribe: send mail to majord...@freebsd.org with unsubscribe freebsd-hackers in the body of the message To Unsubscribe: send mail to majord...@freebsd.org with unsubscribe freebsd-hackers in the body of the message To Unsubscribe: send mail to majord...@freebsd.org with unsubscribe freebsd-hackers in the body of the message
Re: Replace/rewrite reverse.c for tail(1)
:Because of licensing restrictions in our product, we are unable to ship with :any GNU/GPL'ed tools, so I'm forced to fix 'tail' rather than use tac. (I :saw tac, and agree that it is faster for this specific use) : :Any VM people wanna pipe up and make a suggestion so that I may make up a :patch? : :Kevin I don't think madvise()ing it MADV_DONTNEED will work right now. The madvise() calls only work with OBJT_DEFAULT and OBJT_SWAP objects -- i.e. swap-backed memory. They will not do anything to pages owned by file mmaps. We could fix vm/vm_object.c/vm_object_madvise() to handle MADV_DONTNEED for other objects. It would not be too difficult to do, actually, since we would be doing nothing more then moving the (must be clean) page to VM cache to get it to be reused more quickly. This is a fix I could make to CURRENT without too much trouble. -Matt To Unsubscribe: send mail to majord...@freebsd.org with unsubscribe freebsd-hackers in the body of the message
Re: Replace/rewrite reverse.c for tail(1)
:Because of licensing restrictions in our product, we are unable to ship with :any GNU/GPL'ed tools, so I'm forced to fix 'tail' rather than use tac. (I :saw tac, and agree that it is faster for this specific use) : :Any VM people wanna pipe up and make a suggestion so that I may make up a :patch? : :Kevin I don't think madvise()ing it MADV_DONTNEED will work right now. The madvise() calls only work with OBJT_DEFAULT and OBJT_SWAP objects -- i.e. swap-backed memory. They will not do anything to pages owned by file mmaps. We could fix vm/vm_object.c/vm_object_madvise() to handle MADV_DONTNEED for other objects. It would not be too difficult to do, actually, since we would be doing nothing more then moving the (must be clean) page to VM cache to get it to be reused more quickly. This is a fix I could make to CURRENT without too much trouble. -Matt Wow, that's interesting. While I never looked at the code, I *thought* I was able to measure a speed boost in a certain large embedded application I'm working on, with adding some madvise()'s in to mmap'ed files. (Streaming video madvise'ed with MADV_SEQUENTIAL seemed to be slightly faster, but it may have been insignificant) This is a problem I've run into a few times before, actually. One-time use data that will shove the entire system out of ram, and no convenient way to tell the kernel that it shouldn't bother holding on to it. I believe dg ran into this too, and had some fix he was using on ftp.cdrom.com. In my application, when I play a movie, it'd be nice to be able to specify a priority. Try to keep this, even if it means possibly swapping (current behavior) or Don't swap out anything for this object, it's far too large to bother with trying to hold on to it and/or it will only be used once. Granted 'large' is one of those fuzzy terms, but when you do know that the 900M file you're reading is being done sequentially from start to finish, swapping things out to try to keep more of it in swap is probably bad. :) Kevin To Unsubscribe: send mail to majord...@freebsd.org with unsubscribe freebsd-hackers in the body of the message
Re: Replace/rewrite reverse.c for tail(1)
:Wow, that's interesting. While I never looked at the code, I *thought* I was :able to measure a speed boost in a certain large embedded application I'm :working on, with adding some madvise()'s in to mmap'ed files. (Streaming :video madvise'ed with MADV_SEQUENTIAL seemed to be slightly faster, but it :may have been insignificant) MADV_SEQUENTIAL *WILL* work. It changes the page fault read-behind/read-ahead. Normally the VM system reads behind a bit as well as reads ahead. If you set MADV_SEQUENTIAL it shifts to just doing read-ahead, and it does more of it. The madvise() types that just flag the VM object work on any type of object. The madvise() types that actually mess with VM pages currently only work on swap-backed pages. MADV_WILLNEED scans for VM pages in object MADV_DONTNEED scans for VM pages in object MADV_FREE scans for VM pages in object MADV_SEQUENTIAL just flags the object MADV_RANDOM just flags the object MADV_NORMAL just flags the object :Granted 'large' is one of those fuzzy terms, but when you do know that the :900M file you're reading is being done sequentially from start to finish, :swapping things out to try to keep more of it in swap is probably bad. :) : :Kevin I agree completely. When you run through the filesystem these pages will be reused more quickly. When you run through the VM system these pages are considered to be more 'active'. I think it makes sense to have madvise() MADV_WILLNEED and MADV_DONTNEED operate on file mmaps. There is a slight security concern with MADV_DONTNEED (i.e. if you loop running it on a system file that is used a lot, like /etc/group), but after thinking about it a bit I do not think it is a big deal. -Matt Matthew Dillon dil...@backplane.com To Unsubscribe: send mail to majord...@freebsd.org with unsubscribe freebsd-hackers in the body of the message
