Re: [PATCH v2] vmcore: prevent PT_NOTE p_memsz overflow during header update
On Mon, Feb 03, 2014 at 10:57:58PM +, Pearson, Greg wrote:
> On 02/03/2014 02:38 PM, Vivek Goyal wrote:
> > On Mon, Feb 03, 2014 at 01:18:38PM -0700, Greg Pearson wrote:
> >
> > [..]
> >> diff --git a/fs/proc/vmcore.c b/fs/proc/vmcore.c
> >> index 2ca7ba0..051c803 100644
> >> --- a/fs/proc/vmcore.c
> >> +++ b/fs/proc/vmcore.c
> >> @@ -468,12 +468,14 @@ static int __init
> >> update_note_header_size_elf64(const Elf64_Ehdr *ehdr_ptr)
> >>return rc;
> >>}
> >>nhdr_ptr = notes_section;
> >> - while (real_sz < max_sz) {
> >> - if (nhdr_ptr->n_namesz == 0)
> >> - break;
> >> + while (nhdr_ptr->n_namesz != 0) {
> >>sz = sizeof(Elf64_Nhdr) +
> >>((nhdr_ptr->n_namesz + 3) & ~3) +
> >>((nhdr_ptr->n_descsz + 3) & ~3);
> >> + if ((real_sz + sz) > max_sz) {
> >> + pr_warn("Warning: dropping PT_NOTE entry\n");
> >> + break;
> >> + }
> > Hi Greg,
> >
> > Couple of minor nits.
> >
> > I think it is a good idea to give more data in warning which tells why
> > are we dropping a note entry. May be something like.
> >
> > "Warning: Total note entry size exceeded PT_NOTE memsz. Dropping PT_NOTE
> > entry, n_namesz=<> n_descsz=<>".
>
> Sounds good I'll add more information to the pr_warn().
>
> >
> > Secondly, if there is only on note entry in a PT_NOTE header and we drop
> > it, then that PT_NOTE header is empty and needs to be cleaned up.
> >
> > I think you will have to modify get_note_number_and_size_elf64() and
> > other relevant functions which are not expecting ->p_memsz=0.
>
> What about treating this as an error condition and adding a check to the
> update_note_header_size_elf32()/update_note_header_size_elf64() routines
> that would return a failure, something like the following at the end of
> the routine:
>
> if (real_sz == 0)
> return -EINVAL
>
> I could also add a pr_warn() with a message indicating no PT_NOTE
> entries were found.
>
> This seems like a lower risk change to handle the case I'm currently
> seeing as opposed to changing the code to handle a p_memsz==0.
>
> Thoughts?
If you retrun with -EINVAL, that means /proc/vmcore will not be
created. This probably is too harsh an action. If a PT_NOTE is corrupt,
we can just ignore it and continue with construction of /proc/vmcore. We
will just loose cpu registers of one cpu. May be that cpu was idling
and it did not matter for the purpose of crash analysis.
IMO, we can just ignore empty PT_NOTE (real_sz), give a warning and move
on.
Thanks
Vivek
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH v2] vmcore: prevent PT_NOTE p_memsz overflow during header update
On Mon, Feb 03, 2014 at 10:57:58PM +, Pearson, Greg wrote:
On 02/03/2014 02:38 PM, Vivek Goyal wrote:
On Mon, Feb 03, 2014 at 01:18:38PM -0700, Greg Pearson wrote:
[..]
diff --git a/fs/proc/vmcore.c b/fs/proc/vmcore.c
index 2ca7ba0..051c803 100644
--- a/fs/proc/vmcore.c
+++ b/fs/proc/vmcore.c
@@ -468,12 +468,14 @@ static int __init
update_note_header_size_elf64(const Elf64_Ehdr *ehdr_ptr)
return rc;
}
nhdr_ptr = notes_section;
- while (real_sz max_sz) {
- if (nhdr_ptr-n_namesz == 0)
- break;
+ while (nhdr_ptr-n_namesz != 0) {
sz = sizeof(Elf64_Nhdr) +
((nhdr_ptr-n_namesz + 3) ~3) +
((nhdr_ptr-n_descsz + 3) ~3);
+ if ((real_sz + sz) max_sz) {
+ pr_warn(Warning: dropping PT_NOTE entry\n);
+ break;
+ }
Hi Greg,
Couple of minor nits.
I think it is a good idea to give more data in warning which tells why
are we dropping a note entry. May be something like.
Warning: Total note entry size exceeded PT_NOTE memsz. Dropping PT_NOTE
entry, n_namesz= n_descsz=.
Sounds good I'll add more information to the pr_warn().
Secondly, if there is only on note entry in a PT_NOTE header and we drop
it, then that PT_NOTE header is empty and needs to be cleaned up.
I think you will have to modify get_note_number_and_size_elf64() and
other relevant functions which are not expecting -p_memsz=0.
What about treating this as an error condition and adding a check to the
update_note_header_size_elf32()/update_note_header_size_elf64() routines
that would return a failure, something like the following at the end of
the routine:
if (real_sz == 0)
return -EINVAL
I could also add a pr_warn() with a message indicating no PT_NOTE
entries were found.
This seems like a lower risk change to handle the case I'm currently
seeing as opposed to changing the code to handle a p_memsz==0.
Thoughts?
If you retrun with -EINVAL, that means /proc/vmcore will not be
created. This probably is too harsh an action. If a PT_NOTE is corrupt,
we can just ignore it and continue with construction of /proc/vmcore. We
will just loose cpu registers of one cpu. May be that cpu was idling
and it did not matter for the purpose of crash analysis.
IMO, we can just ignore empty PT_NOTE (real_sz), give a warning and move
on.
Thanks
Vivek
--
To unsubscribe from this list: send the line unsubscribe linux-kernel in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH v2] vmcore: prevent PT_NOTE p_memsz overflow during header update
On 02/03/2014 02:38 PM, Vivek Goyal wrote:
> On Mon, Feb 03, 2014 at 01:18:38PM -0700, Greg Pearson wrote:
>
> [..]
>> diff --git a/fs/proc/vmcore.c b/fs/proc/vmcore.c
>> index 2ca7ba0..051c803 100644
>> --- a/fs/proc/vmcore.c
>> +++ b/fs/proc/vmcore.c
>> @@ -468,12 +468,14 @@ static int __init update_note_header_size_elf64(const
>> Elf64_Ehdr *ehdr_ptr)
>> return rc;
>> }
>> nhdr_ptr = notes_section;
>> -while (real_sz < max_sz) {
>> -if (nhdr_ptr->n_namesz == 0)
>> -break;
>> +while (nhdr_ptr->n_namesz != 0) {
>> sz = sizeof(Elf64_Nhdr) +
>> ((nhdr_ptr->n_namesz + 3) & ~3) +
>> ((nhdr_ptr->n_descsz + 3) & ~3);
>> +if ((real_sz + sz) > max_sz) {
>> +pr_warn("Warning: dropping PT_NOTE entry\n");
>> +break;
>> +}
> Hi Greg,
>
> Couple of minor nits.
>
> I think it is a good idea to give more data in warning which tells why
> are we dropping a note entry. May be something like.
>
> "Warning: Total note entry size exceeded PT_NOTE memsz. Dropping PT_NOTE
> entry, n_namesz=<> n_descsz=<>".
Sounds good I'll add more information to the pr_warn().
>
> Secondly, if there is only on note entry in a PT_NOTE header and we drop
> it, then that PT_NOTE header is empty and needs to be cleaned up.
>
> I think you will have to modify get_note_number_and_size_elf64() and
> other relevant functions which are not expecting ->p_memsz=0.
What about treating this as an error condition and adding a check to the
update_note_header_size_elf32()/update_note_header_size_elf64() routines
that would return a failure, something like the following at the end of
the routine:
if (real_sz == 0)
return -EINVAL
I could also add a pr_warn() with a message indicating no PT_NOTE
entries were found.
This seems like a lower risk change to handle the case I'm currently
seeing as opposed to changing the code to handle a p_memsz==0.
Thoughts?
--
Greg
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH v2] vmcore: prevent PT_NOTE p_memsz overflow during header update
On Mon, Feb 03, 2014 at 01:18:38PM -0700, Greg Pearson wrote:
[..]
> diff --git a/fs/proc/vmcore.c b/fs/proc/vmcore.c
> index 2ca7ba0..051c803 100644
> --- a/fs/proc/vmcore.c
> +++ b/fs/proc/vmcore.c
> @@ -468,12 +468,14 @@ static int __init update_note_header_size_elf64(const
> Elf64_Ehdr *ehdr_ptr)
> return rc;
> }
> nhdr_ptr = notes_section;
> - while (real_sz < max_sz) {
> - if (nhdr_ptr->n_namesz == 0)
> - break;
> + while (nhdr_ptr->n_namesz != 0) {
> sz = sizeof(Elf64_Nhdr) +
> ((nhdr_ptr->n_namesz + 3) & ~3) +
> ((nhdr_ptr->n_descsz + 3) & ~3);
> + if ((real_sz + sz) > max_sz) {
> + pr_warn("Warning: dropping PT_NOTE entry\n");
> + break;
> + }
Hi Greg,
Couple of minor nits.
I think it is a good idea to give more data in warning which tells why
are we dropping a note entry. May be something like.
"Warning: Total note entry size exceeded PT_NOTE memsz. Dropping PT_NOTE entry,
n_namesz=<> n_descsz=<>".
Secondly, if there is only on note entry in a PT_NOTE header and we drop
it, then that PT_NOTE header is empty and needs to be cleaned up.
I think you will have to modify get_note_number_and_size_elf64() and
other relevant functions which are not expecting ->p_memsz=0.
Thanks
Vivek
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
[PATCH v2] vmcore: prevent PT_NOTE p_memsz overflow during header update
Currently, update_note_header_size_elf64() and
update_note_header_size_elf32() will add the size
of a PT_NOTE entry to real_sz even if that causes real_sz
to exceeds max_sz. This patch corrects the while loop logic
in those routines to ensure that does not happen and prints
a warning if a PT_NOTE entry is dropped.
One possible negative side effect of exceeding the max_sz
limit is an allocation failure in merge_note_headers_elf64()
or merge_note_headers_elf32() which would produce console
output such as the following while booting the crash
kernel.
vmalloc: allocation failure: 14076997632 bytes
swapper/0: page allocation failure: order:0, mode:0x80d2
CPU: 0 PID: 1 Comm: swapper/0 Not tainted 3.10.0-gbp1 #7
817dcc30 88003025fc28 815bdb0b 88003025fcb0
8113b3d0 817dcc30 88003025fc48 c918
88003025fcc0 88003025fc60 88003025fc80 88002b5df980
Call Trace:
[] dump_stack+0x19/0x1b
[] warn_alloc_failed+0xf0/0x160
[] ? merge_note_headers_elf64.constprop.9+0x116/0x24a
[] __vmalloc_node_range+0x19e/0x250
[] ? read_from_oldmem.part.0+0xa4/0xe0
[] vmalloc_user+0x4c/0x70
[] ? merge_note_headers_elf64.constprop.9+0x116/0x24a
[] merge_note_headers_elf64.constprop.9+0x116/0x24a
[] vmcore_init+0x2d4/0x76c
[] ? kcore_update_ram+0x1f0/0x1f0
[] ? walk_system_raange+0x112/0x130
[] ? merge_note_headers_elf32.constprop.8+0x249/0x249
[] do_one_initcall+0xe2/0x190
[] kernel_init_freeable+0x17c/0x207
[] ? do_early_param+0x88/0x88
[] ? rest_init+0x80/0x80
[] kernel_init+0xe/0x180
[] ret_from_fork+0x7c/0xb0
[] ? rest_init+0x80/0x80
Kdump: vmcore not initialized
kdump: dump target is /dev/sda4
kdump: saving to /sysroot//var/crash/127.0.0.1-2014.01.28-13:58:52/
kdump: saving vmcore-dmesg.txt
Cannot open /proc/vmcore: No such file or directory
kdump: saving vmcore-dmesg.txt failed
kdump: saving vmcore
kdump: saving vmcore failed
This type of failure has been seen on a four socket prototype
system with certain memory configurations. Most PT_NOTE sections
have a single entry similar to:
n_namesz = 0x5
n_descsz = 0x150
n_type = 0x1
Occasionally, a second entry is encountered with very
large n_namesz and n_descsz sizes:
n_namesz = 0x8008
n_descsz = 0x510ae163
n_type = 0x8008
Not yet sure of the source of these extra entries, they
seem bogus, but they shouldn't cause crash dump to fail.
Signed-off-by: Greg Pearson
---
fs/proc/vmcore.c | 16 ++--
1 file changed, 10 insertions(+), 6 deletions(-)
diff --git a/fs/proc/vmcore.c b/fs/proc/vmcore.c
index 2ca7ba0..051c803 100644
--- a/fs/proc/vmcore.c
+++ b/fs/proc/vmcore.c
@@ -468,12 +468,14 @@ static int __init update_note_header_size_elf64(const
Elf64_Ehdr *ehdr_ptr)
return rc;
}
nhdr_ptr = notes_section;
- while (real_sz < max_sz) {
- if (nhdr_ptr->n_namesz == 0)
- break;
+ while (nhdr_ptr->n_namesz != 0) {
sz = sizeof(Elf64_Nhdr) +
((nhdr_ptr->n_namesz + 3) & ~3) +
((nhdr_ptr->n_descsz + 3) & ~3);
+ if ((real_sz + sz) > max_sz) {
+ pr_warn("Warning: dropping PT_NOTE entry\n");
+ break;
+ }
real_sz += sz;
nhdr_ptr = (Elf64_Nhdr*)((char*)nhdr_ptr + sz);
}
@@ -648,12 +650,14 @@ static int __init update_note_header_size_elf32(const
Elf32_Ehdr *ehdr_ptr)
return rc;
}
nhdr_ptr = notes_section;
- while (real_sz < max_sz) {
- if (nhdr_ptr->n_namesz == 0)
- break;
+ while (nhdr_ptr->n_namesz != 0) {
sz = sizeof(Elf32_Nhdr) +
((nhdr_ptr->n_namesz + 3) & ~3) +
((nhdr_ptr->n_descsz + 3) & ~3);
+ if ((real_sz + sz) > max_sz) {
+ pr_warn("Warning: dropping PT_NOTE entry\n");
+ break;
+ }
real_sz += sz;
nhdr_ptr = (Elf32_Nhdr*)((char*)nhdr_ptr + sz);
}
--
1.8.3.2
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
[PATCH v2] vmcore: prevent PT_NOTE p_memsz overflow during header update
Currently, update_note_header_size_elf64() and
update_note_header_size_elf32() will add the size
of a PT_NOTE entry to real_sz even if that causes real_sz
to exceeds max_sz. This patch corrects the while loop logic
in those routines to ensure that does not happen and prints
a warning if a PT_NOTE entry is dropped.
One possible negative side effect of exceeding the max_sz
limit is an allocation failure in merge_note_headers_elf64()
or merge_note_headers_elf32() which would produce console
output such as the following while booting the crash
kernel.
vmalloc: allocation failure: 14076997632 bytes
swapper/0: page allocation failure: order:0, mode:0x80d2
CPU: 0 PID: 1 Comm: swapper/0 Not tainted 3.10.0-gbp1 #7
817dcc30 88003025fc28 815bdb0b 88003025fcb0
8113b3d0 817dcc30 88003025fc48 c918
88003025fcc0 88003025fc60 88003025fc80 88002b5df980
Call Trace:
[815bdb0b] dump_stack+0x19/0x1b
[8113b3d0] warn_alloc_failed+0xf0/0x160
[81a1267d] ? merge_note_headers_elf64.constprop.9+0x116/0x24a
[8116d34e] __vmalloc_node_range+0x19e/0x250
[81210454] ? read_from_oldmem.part.0+0xa4/0xe0
[8116d4ec] vmalloc_user+0x4c/0x70
[81a1267d] ? merge_note_headers_elf64.constprop.9+0x116/0x24a
[81a1267d] merge_note_headers_elf64.constprop.9+0x116/0x24a
[81a12cce] vmcore_init+0x2d4/0x76c
[8120f9f0] ? kcore_update_ram+0x1f0/0x1f0
[81063b92] ? walk_system_raange+0x112/0x130
[81a129fa] ? merge_note_headers_elf32.constprop.8+0x249/0x249
[810020e2] do_one_initcall+0xe2/0x190
[819e20c4] kernel_init_freeable+0x17c/0x207
[819e18d0] ? do_early_param+0x88/0x88
[815a0d20] ? rest_init+0x80/0x80
[815a0d2e] kernel_init+0xe/0x180
[815cd8ac] ret_from_fork+0x7c/0xb0
[815a0d20] ? rest_init+0x80/0x80
Kdump: vmcore not initialized
kdump: dump target is /dev/sda4
kdump: saving to /sysroot//var/crash/127.0.0.1-2014.01.28-13:58:52/
kdump: saving vmcore-dmesg.txt
Cannot open /proc/vmcore: No such file or directory
kdump: saving vmcore-dmesg.txt failed
kdump: saving vmcore
kdump: saving vmcore failed
This type of failure has been seen on a four socket prototype
system with certain memory configurations. Most PT_NOTE sections
have a single entry similar to:
n_namesz = 0x5
n_descsz = 0x150
n_type = 0x1
Occasionally, a second entry is encountered with very
large n_namesz and n_descsz sizes:
n_namesz = 0x8008
n_descsz = 0x510ae163
n_type = 0x8008
Not yet sure of the source of these extra entries, they
seem bogus, but they shouldn't cause crash dump to fail.
Signed-off-by: Greg Pearson greg.pear...@hp.com
---
fs/proc/vmcore.c | 16 ++--
1 file changed, 10 insertions(+), 6 deletions(-)
diff --git a/fs/proc/vmcore.c b/fs/proc/vmcore.c
index 2ca7ba0..051c803 100644
--- a/fs/proc/vmcore.c
+++ b/fs/proc/vmcore.c
@@ -468,12 +468,14 @@ static int __init update_note_header_size_elf64(const
Elf64_Ehdr *ehdr_ptr)
return rc;
}
nhdr_ptr = notes_section;
- while (real_sz max_sz) {
- if (nhdr_ptr-n_namesz == 0)
- break;
+ while (nhdr_ptr-n_namesz != 0) {
sz = sizeof(Elf64_Nhdr) +
((nhdr_ptr-n_namesz + 3) ~3) +
((nhdr_ptr-n_descsz + 3) ~3);
+ if ((real_sz + sz) max_sz) {
+ pr_warn(Warning: dropping PT_NOTE entry\n);
+ break;
+ }
real_sz += sz;
nhdr_ptr = (Elf64_Nhdr*)((char*)nhdr_ptr + sz);
}
@@ -648,12 +650,14 @@ static int __init update_note_header_size_elf32(const
Elf32_Ehdr *ehdr_ptr)
return rc;
}
nhdr_ptr = notes_section;
- while (real_sz max_sz) {
- if (nhdr_ptr-n_namesz == 0)
- break;
+ while (nhdr_ptr-n_namesz != 0) {
sz = sizeof(Elf32_Nhdr) +
((nhdr_ptr-n_namesz + 3) ~3) +
((nhdr_ptr-n_descsz + 3) ~3);
+ if ((real_sz + sz) max_sz) {
+ pr_warn(Warning: dropping PT_NOTE entry\n);
+ break;
+ }
real_sz += sz;
nhdr_ptr = (Elf32_Nhdr*)((char*)nhdr_ptr + sz);
}
--
1.8.3.2
--
To unsubscribe from this list: send the line unsubscribe linux-kernel in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH v2] vmcore: prevent PT_NOTE p_memsz overflow during header update
On Mon, Feb 03, 2014 at 01:18:38PM -0700, Greg Pearson wrote:
[..]
diff --git a/fs/proc/vmcore.c b/fs/proc/vmcore.c
index 2ca7ba0..051c803 100644
--- a/fs/proc/vmcore.c
+++ b/fs/proc/vmcore.c
@@ -468,12 +468,14 @@ static int __init update_note_header_size_elf64(const
Elf64_Ehdr *ehdr_ptr)
return rc;
}
nhdr_ptr = notes_section;
- while (real_sz max_sz) {
- if (nhdr_ptr-n_namesz == 0)
- break;
+ while (nhdr_ptr-n_namesz != 0) {
sz = sizeof(Elf64_Nhdr) +
((nhdr_ptr-n_namesz + 3) ~3) +
((nhdr_ptr-n_descsz + 3) ~3);
+ if ((real_sz + sz) max_sz) {
+ pr_warn(Warning: dropping PT_NOTE entry\n);
+ break;
+ }
Hi Greg,
Couple of minor nits.
I think it is a good idea to give more data in warning which tells why
are we dropping a note entry. May be something like.
Warning: Total note entry size exceeded PT_NOTE memsz. Dropping PT_NOTE entry,
n_namesz= n_descsz=.
Secondly, if there is only on note entry in a PT_NOTE header and we drop
it, then that PT_NOTE header is empty and needs to be cleaned up.
I think you will have to modify get_note_number_and_size_elf64() and
other relevant functions which are not expecting -p_memsz=0.
Thanks
Vivek
--
To unsubscribe from this list: send the line unsubscribe linux-kernel in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH v2] vmcore: prevent PT_NOTE p_memsz overflow during header update
On 02/03/2014 02:38 PM, Vivek Goyal wrote:
On Mon, Feb 03, 2014 at 01:18:38PM -0700, Greg Pearson wrote:
[..]
diff --git a/fs/proc/vmcore.c b/fs/proc/vmcore.c
index 2ca7ba0..051c803 100644
--- a/fs/proc/vmcore.c
+++ b/fs/proc/vmcore.c
@@ -468,12 +468,14 @@ static int __init update_note_header_size_elf64(const
Elf64_Ehdr *ehdr_ptr)
return rc;
}
nhdr_ptr = notes_section;
-while (real_sz max_sz) {
-if (nhdr_ptr-n_namesz == 0)
-break;
+while (nhdr_ptr-n_namesz != 0) {
sz = sizeof(Elf64_Nhdr) +
((nhdr_ptr-n_namesz + 3) ~3) +
((nhdr_ptr-n_descsz + 3) ~3);
+if ((real_sz + sz) max_sz) {
+pr_warn(Warning: dropping PT_NOTE entry\n);
+break;
+}
Hi Greg,
Couple of minor nits.
I think it is a good idea to give more data in warning which tells why
are we dropping a note entry. May be something like.
Warning: Total note entry size exceeded PT_NOTE memsz. Dropping PT_NOTE
entry, n_namesz= n_descsz=.
Sounds good I'll add more information to the pr_warn().
Secondly, if there is only on note entry in a PT_NOTE header and we drop
it, then that PT_NOTE header is empty and needs to be cleaned up.
I think you will have to modify get_note_number_and_size_elf64() and
other relevant functions which are not expecting -p_memsz=0.
What about treating this as an error condition and adding a check to the
update_note_header_size_elf32()/update_note_header_size_elf64() routines
that would return a failure, something like the following at the end of
the routine:
if (real_sz == 0)
return -EINVAL
I could also add a pr_warn() with a message indicating no PT_NOTE
entries were found.
This seems like a lower risk change to handle the case I'm currently
seeing as opposed to changing the code to handle a p_memsz==0.
Thoughts?
--
Greg
--
To unsubscribe from this list: send the line unsubscribe linux-kernel in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
