Bug#505718: grep: Ridiculous memory usage
reassign 505718 chkrootkit
severity 505718 normal
retitle 505718 Please limit grep output, can cause automatic DOS.
thanks
Hello,
While checking e.g. for php based rootkit, the invocation of grep does
not limit the amount of output returned, for example in:
fileshead="`${find} ${ROOTDIR}tmp ${ROOTDIR}var/tmp ${findargs} -type f
-exec head -1 {} \; | grep php 2> /dev/null`"
I had a 2Gb swap file in /tmp and grep kept the current "line" in
memory waiting to see if it would match 'php'. For some reason this
swap file was composed of very long "lines", and my machine was
unreasonably slow because of this grep. Generally speaking, grep's
behaviour here is correct because it needs to store all of the current
line to be able to display it if it matches, but some sort of output
limitation should be used by chkrootkit to avoid a local DOS by the
cron job.
Regards,
Laurent.
--
To UNSUBSCRIBE, email to [EMAIL PROTECTED]
with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
Bug#505718: grep: Ridiculous memory usage
Hello, * Nicolas François [Fri, Nov 14, 2008 at 08:45:57PM +0100]: > grep is line-based, so I'm not really surprised it eats all the memory > when it receives only zeroes. > > What other "big file" did you tested this on? A 2Gb swap file. It could be that the first newline in this file is far from the beginning, I've deleted it since. > For chkrootkit, I guess checking for fixed string (at least when we can > expect long greps) is sufficient and would fasten it a lot. It was in fact checking for fixed string when it nearly stopped my machine, the invocation was `grep php'. I see that since grep usually prints out matching lines, it needs to allocate enough memory for the current line. Thanks for pointing that out. I guess this bug becomes a wishlist bug against chkrootkit. Regards, Laurent. -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
Bug#505718: grep: Ridiculous memory usage
On Fri, Nov 14, 2008 at 05:03:30PM +0100, Laurent Fousse wrote: > Hello, > > * Aioanei Rares [Fri, Nov 14, 2008 at 05:49:35PM +0200]: > > Maybe the kernel is at fault? > > The particular argument `/dev/zero/' I supplied to grep in my example > is not necessary, if this is what you mean. I can reproduce it with > any big file. Since /dev/zero is infinite in size, it will eventually > lead to an allocation error. grep is line-based, so I'm not really surprised it eats all the memory when it receives only zeroes. What other "big file" did you tested this on? Can you test if using "LC_ALL=C grep" instead helps or if "grep -F" works? If you are greping in a binary file, I would recommend using strings + grep instead. For chkrootkit, I guess checking for fixed string (at least when we can expect long greps) is sufficient and would fasten it a lot. Best Regards, -- Nekral -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
Bug#505718: grep: Ridiculous memory usage
Hello, * Aioanei Rares [Fri, Nov 14, 2008 at 05:49:35PM +0200]: > Maybe the kernel is at fault? The particular argument `/dev/zero/' I supplied to grep in my example is not necessary, if this is what you mean. I can reproduce it with any big file. Since /dev/zero is infinite in size, it will eventually lead to an allocation error. Laurent. -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
Bug#505718: grep: Ridiculous memory usage
On Fri, Nov 14, 2008 at 5:41 PM, Laurent Fousse <[EMAIL PROTECTED]> wrote: > Package: grep > Version: 2.5.1.ds2-6 > Severity: important > > Hello, > > A grep for a simple pattern on a big file results in a similarly big > memory consumption, rendering the machine unusable as it swaps as hell > and/or a memory allocation failure occurs. E.g.: > >$ grep foo /dev/zero >grep: /dev/zero: Cannot allocate memory > > I noticed this problem because of a simple `grep php' test in > chkrootkit (launched from tiger) making my machine unusable. > > I can reproduce this problem with 2.5.3~dfsg-6 on amd64. > > Regards, > > Laurent. > > -- System Information: > Debian Release: 4.0 > APT prefers stable > APT policy: (500, 'stable') > Architecture: amd64 (x86_64) > Shell: /bin/sh linked to /bin/bash > Kernel: Linux 2.6.18-6-amd64 > Locale: LANG=en_US, LC_CTYPE=fr_FR.UTF-8 (charmap=UTF-8) (ignored: LC_ALL > set to en_US.UTF-8) > > Versions of packages grep depends on: > ii libc6 2.3.6.ds1-13etch7 GNU C Library: Shared > libraries > > grep recommends no packages. > > -- no debconf information > > > > -- > To UNSUBSCRIBE, email to [EMAIL PROTECTED] > with a subject of "unsubscribe". Trouble? Contact > [EMAIL PROTECTED] > > Maybe the kernel is at fault?
Bug#505718: grep: Ridiculous memory usage
Package: grep Version: 2.5.1.ds2-6 Severity: important Hello, A grep for a simple pattern on a big file results in a similarly big memory consumption, rendering the machine unusable as it swaps as hell and/or a memory allocation failure occurs. E.g.: $ grep foo /dev/zero grep: /dev/zero: Cannot allocate memory I noticed this problem because of a simple `grep php' test in chkrootkit (launched from tiger) making my machine unusable. I can reproduce this problem with 2.5.3~dfsg-6 on amd64. Regards, Laurent. -- System Information: Debian Release: 4.0 APT prefers stable APT policy: (500, 'stable') Architecture: amd64 (x86_64) Shell: /bin/sh linked to /bin/bash Kernel: Linux 2.6.18-6-amd64 Locale: LANG=en_US, LC_CTYPE=fr_FR.UTF-8 (charmap=UTF-8) (ignored: LC_ALL set to en_US.UTF-8) Versions of packages grep depends on: ii libc6 2.3.6.ds1-13etch7 GNU C Library: Shared libraries grep recommends no packages. -- no debconf information -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
