[PATCH] mm/page-writeback - highmem_is_dirtyable option (replaces dirty_highmem patch)
On Tue, Nov 27, 2007 at 11:10:28PM +1100, Bron Gondwana wrote:
> On Mon, Nov 26, 2007 at 09:53:15PM -0800, Andrew Morton wrote:
> > umm, really you want
> > /proc/sys/vm/dont-account-highmem-in-dirty-memory-calculations, only
> > shorter.
> >
> > Do you agree?
>
> I still read dirty_highmem as:
>
> /proc/sys/vm/do-account-highmem-in-dirty-memory-calculations
>
> ... so we're still talking one negative apart!
>
> > It would be simpler to have
> > /proc/sys/vm/do-account-highmem-in-dirty-memory-calculations,
> > defaulting to "true" - this has no negations.
>
> No, that's not true. The whole point is that between 2.6.16 and
> 2.6.20 the kernel behaviour changed. It currently doesn't count
> highmem in dirty memory calculations, which is why the memory pressure
> appears to be so great when actually there's still 4Gb of unused
> memory in the box.
>
> > OK, I give up. Please see if you can think of something less confusing
> > which involves no negations?
>
> I think this might be slightly clearer:
>
> /proc/sys/vm/highmem_is_dirtyable - defaults to false
>
> [ ... ]
>
> Would you like me to re-submit the patch based on this? I'm
> certainly not happy with dirty_highmem as it is now in mm
> because it looks backwards and unclear to me.
Well, it was quick enough to just do - here's the patch. I've
also updated the documentation a bit to clarify the intention and
the reasons why you might want to use it (based in part on the
comments to the original change that made highmem uncountable for
dirtyness purposes)
Tested and applied against 2.6.23.9 (our build script makes Debian
packages from a clean unpack of kernel major plus patch minor plus
svn checkout of out quilt series and apply regardless, so it was
just as easy to bump the version number while I was at it).
Builds, boots, passes a quick run of the test program I used last
time around.
Bron.
Add vm.highmem_is_dirtyable toggle
A 32 bit machine with HIGHMEM64 enabled running DCC has an MMAPed file
of approximately 2Gb size which contains a hash format that is written
randomly by the dbclean process. On 2.6.16 this process took a few
minutes. With lowmem only accounting of dirty ratios, this takes about
12 hours of 100% disk IO, all random writes.
This patch includes some code cleanup from Linus and a toggle in
/proc/sys/vm/highmem_is_dirtyable which can be set to 1 to add the
highmem back to the total available memory count.
Signed-off-by: Bron Gondwana <[EMAIL PROTECTED]>
Index: linux-2.6.23.8-reiserfix-fai-vmdirty/mm/page-writeback.c
===
--- linux-2.6.23.8-reiserfix-fai-vmdirty.orig/mm/page-writeback.c
2007-11-21 21:58:20.0 -0500
+++ linux-2.6.23.8-reiserfix-fai-vmdirty/mm/page-writeback.c2007-11-27
07:27:51.0 -0500
@@ -70,6 +70,12 @@
int dirty_background_ratio = 5;
/*
+ * free highmem will not be subtracted from the total free memory
+ * for calculating free ratios if vm_highmem_is_dirtyable is true
+ */
+int vm_highmem_is_dirtyable;
+
+/*
* The generator of dirty data starts writeback at this percentage
*/
int vm_dirty_ratio = 10;
@@ -153,7 +159,10 @@
x = global_page_state(NR_FREE_PAGES)
+ global_page_state(NR_INACTIVE)
+ global_page_state(NR_ACTIVE);
- x -= highmem_dirtyable_memory(x);
+
+ if (!vm_highmem_is_dirtyable)
+ x -= highmem_dirtyable_memory(x);
+
return x + 1; /* Ensure that we never return 0 */
}
@@ -163,20 +172,12 @@
{
int background_ratio; /* Percentages */
int dirty_ratio;
- int unmapped_ratio;
long background;
long dirty;
unsigned long available_memory = determine_dirtyable_memory();
struct task_struct *tsk;
- unmapped_ratio = 100 - ((global_page_state(NR_FILE_MAPPED) +
- global_page_state(NR_ANON_PAGES)) * 100) /
- available_memory;
-
dirty_ratio = vm_dirty_ratio;
- if (dirty_ratio > unmapped_ratio / 2)
- dirty_ratio = unmapped_ratio / 2;
-
if (dirty_ratio < 5)
dirty_ratio = 5;
Index: linux-2.6.23.8-reiserfix-fai-vmdirty/include/linux/writeback.h
===
--- linux-2.6.23.8-reiserfix-fai-vmdirty.orig/include/linux/writeback.h
2007-10-09 16:31:38.0 -0400
+++ linux-2.6.23.8-reiserfix-fai-vmdirty/include/linux/writeback.h
2007-11-27 07:22:17.0 -0500
@@ -95,6 +95,7 @@
extern int vm_dirty_ratio;
extern int dirty_writeback_interval;
extern int dirty_expire_interval;
+extern int vm_highmem_is_dirtyable;
extern int block_dump;
extern int laptop_mode;
Index: linux-2.6.23.8-reiserfix-fai-vmdirty/kernel/s
dirty highmem calculation sysctl name (Was: [PATCH 1/1] mm: add dirty_highmem option)
On Mon, Nov 26, 2007 at 09:53:15PM -0800, Andrew Morton wrote:
> On Tue, 27 Nov 2007 16:24:24 +1100 "Bron Gondwana" <[EMAIL PROTECTED]> wrote:
>
> > On Mon, 26 Nov 2007 20:54:28 -0800, "Andrew Morton" <[EMAIL PROTECTED]>
> > said:
> > > On Thu, 22 Nov 2007 14:42:04 +1100 Bron Gondwana <[EMAIL PROTECTED]>
> > > wrote:
> > >
> > > > /*
> > > > + * free highmem will not be subtracted from the total free memory
> > > > + * for calculating free ratios if vm_dirty_highmem is true
> > > > + */
> > > > +int vm_dirty_highmem;
> > >
> > > One would expect that setting dirty_highmem to true would cause highmem
> > > to
> > > be accounted in dirty-memory calculations. However with this change
> > > reality is in fact the inverse of that.
> > >
> > > So how about this?
> >
> > Actually, I'm confused now. Maybe I chose a bad name to begin with.
> > Does it mean "I am allowed to dirty high memory" or "my high memory
> > will be dirty if this is on"?
>
> But we're always allowed to dirty highmem - there'd be no point in having
> it otherwise. Hence the term dirty_highmem is confusing.
>
> umm, really you want
> /proc/sys/vm/dont-account-highmem-in-dirty-memory-calculations, only
> shorter.
>
> Do you agree?
I still read dirty_highmem as:
/proc/sys/vm/do-account-highmem-in-dirty-memory-calculations
... so we're still talking one negative apart!
> If so, then it's still not a very pleasing interface - setting something to
> "true" to disable a particular piece of kernel behaviour implies a single
> negation which we don't really need.
Well, the particular piece of kernel behaviour is already a negative:
"decrease the amount of memory allowed to get dirty so we never dirty
more than a percentage of available lowmem"
So what this flag is saying is:
"DON'T decrease the amount of memory allowed to get dirty down to just
the lowmem - dirty a percentage of total available including highmem"
As Linus said - the alternative of allowing more than 100% of lowmem
to be dirty is just plain too wierd, hence this approach of allowing
an option to turn off the new restriction that appeared since 2.6.16.
> It would be simpler to have
> /proc/sys/vm/do-account-highmem-in-dirty-memory-calculations,
> defaulting to "true" - this has no negations.
No, that's not true. The whole point is that between 2.6.16 and
2.6.20 the kernel behaviour changed. It currently doesn't count
highmem in dirty memory calculations, which is why the memory pressure
appears to be so great when actually there's still 4Gb of unused
memory in the box.
/proc/sys/vm/do-account-highmem-in-dirty-memory-calculations would
default to "false" to get the current behaviour post-2.6.16 kernels.
Setting a flag to make it true would stop the kernel subtracting
highmem from the available count, giving the "old" behaviour of
allowing a percentage of all the memory in the system to be dirty
before forcing writeback.
> So... how about /proc/sys/vm/, umm.
>
>
>
> OK, I give up. Please see if you can think of something less confusing
> which involves no negations?
I've spent a while thinking about this, and looking at the code.
I think this might be slightly clearer:
/proc/sys/vm/highmem_is_dirtyable - defaults to false
Here's how it would look in the code:
static unsigned long determine_dirtyable_memory(void)
{
unsigned long x;
x = global_page_state(NR_FREE_PAGES)
+ global_page_state(NR_INACTIVE)
+ global_page_state(NR_ACTIVE);
if (!vm_highmem_is_dirtyable)
x -= highmem_dirtyable_memory(x);
return x + 1; /* Ensure that we never return 0 */
}
I think that's very clear.
"Unless highmem is dirtyable, subtract the otherwise dirtyable
pages from the total dirtyable memory count if they are in
highmem"
Or without negatives:
"If highmem is dirtyable then include highmem pages in the
total dirtyable memory count"
Unfortunately this isn't expressed without negatives in the
code because it's easier (I presume without looking at all the
different zones on a cruddy piece of old i386) to add up the
total and remove the pages in the HIGHMEM zone than to add up
all the non-HIGHMEM zones separately.
Does that suit you? Does my explaination make sense?
Would you like me to re-submit the patch based on this? I'm
certainly not happy with dirty_highmem as it is now in mm
because it looks backwards and unclear to me.
Bron.
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
dirty highmem calculation sysctl name (Was: [PATCH 1/1] mm: add dirty_highmem option)
On Mon, Nov 26, 2007 at 09:53:15PM -0800, Andrew Morton wrote:
On Tue, 27 Nov 2007 16:24:24 +1100 Bron Gondwana [EMAIL PROTECTED] wrote:
On Mon, 26 Nov 2007 20:54:28 -0800, Andrew Morton [EMAIL PROTECTED]
said:
On Thu, 22 Nov 2007 14:42:04 +1100 Bron Gondwana [EMAIL PROTECTED]
wrote:
/*
+ * free highmem will not be subtracted from the total free memory
+ * for calculating free ratios if vm_dirty_highmem is true
+ */
+int vm_dirty_highmem;
One would expect that setting dirty_highmem to true would cause highmem
to
be accounted in dirty-memory calculations. However with this change
reality is in fact the inverse of that.
So how about this?
Actually, I'm confused now. Maybe I chose a bad name to begin with.
Does it mean I am allowed to dirty high memory or my high memory
will be dirty if this is on?
But we're always allowed to dirty highmem - there'd be no point in having
it otherwise. Hence the term dirty_highmem is confusing.
umm, really you want
/proc/sys/vm/dont-account-highmem-in-dirty-memory-calculations, only
shorter.
Do you agree?
I still read dirty_highmem as:
/proc/sys/vm/do-account-highmem-in-dirty-memory-calculations
... so we're still talking one negative apart!
If so, then it's still not a very pleasing interface - setting something to
true to disable a particular piece of kernel behaviour implies a single
negation which we don't really need.
Well, the particular piece of kernel behaviour is already a negative:
decrease the amount of memory allowed to get dirty so we never dirty
more than a percentage of available lowmem
So what this flag is saying is:
DON'T decrease the amount of memory allowed to get dirty down to just
the lowmem - dirty a percentage of total available including highmem
As Linus said - the alternative of allowing more than 100% of lowmem
to be dirty is just plain too wierd, hence this approach of allowing
an option to turn off the new restriction that appeared since 2.6.16.
It would be simpler to have
/proc/sys/vm/do-account-highmem-in-dirty-memory-calculations,
defaulting to true - this has no negations.
No, that's not true. The whole point is that between 2.6.16 and
2.6.20 the kernel behaviour changed. It currently doesn't count
highmem in dirty memory calculations, which is why the memory pressure
appears to be so great when actually there's still 4Gb of unused
memory in the box.
/proc/sys/vm/do-account-highmem-in-dirty-memory-calculations would
default to false to get the current behaviour post-2.6.16 kernels.
Setting a flag to make it true would stop the kernel subtracting
highmem from the available count, giving the old behaviour of
allowing a percentage of all the memory in the system to be dirty
before forcing writeback.
So... how about /proc/sys/vm/, umm.
looks at inbox, brain explodes
OK, I give up. Please see if you can think of something less confusing
which involves no negations?
I've spent a while thinking about this, and looking at the code.
I think this might be slightly clearer:
/proc/sys/vm/highmem_is_dirtyable - defaults to false
Here's how it would look in the code:
static unsigned long determine_dirtyable_memory(void)
{
unsigned long x;
x = global_page_state(NR_FREE_PAGES)
+ global_page_state(NR_INACTIVE)
+ global_page_state(NR_ACTIVE);
if (!vm_highmem_is_dirtyable)
x -= highmem_dirtyable_memory(x);
return x + 1; /* Ensure that we never return 0 */
}
I think that's very clear.
Unless highmem is dirtyable, subtract the otherwise dirtyable
pages from the total dirtyable memory count if they are in
highmem
Or without negatives:
If highmem is dirtyable then include highmem pages in the
total dirtyable memory count
Unfortunately this isn't expressed without negatives in the
code because it's easier (I presume without looking at all the
different zones on a cruddy piece of old i386) to add up the
total and remove the pages in the HIGHMEM zone than to add up
all the non-HIGHMEM zones separately.
Does that suit you? Does my explaination make sense?
Would you like me to re-submit the patch based on this? I'm
certainly not happy with dirty_highmem as it is now in mm
because it looks backwards and unclear to me.
Bron.
-
To unsubscribe from this list: send the line unsubscribe linux-kernel in
the body of a message to [EMAIL PROTECTED]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
[PATCH] mm/page-writeback - highmem_is_dirtyable option (replaces dirty_highmem patch)
On Tue, Nov 27, 2007 at 11:10:28PM +1100, Bron Gondwana wrote:
On Mon, Nov 26, 2007 at 09:53:15PM -0800, Andrew Morton wrote:
umm, really you want
/proc/sys/vm/dont-account-highmem-in-dirty-memory-calculations, only
shorter.
Do you agree?
I still read dirty_highmem as:
/proc/sys/vm/do-account-highmem-in-dirty-memory-calculations
... so we're still talking one negative apart!
It would be simpler to have
/proc/sys/vm/do-account-highmem-in-dirty-memory-calculations,
defaulting to true - this has no negations.
No, that's not true. The whole point is that between 2.6.16 and
2.6.20 the kernel behaviour changed. It currently doesn't count
highmem in dirty memory calculations, which is why the memory pressure
appears to be so great when actually there's still 4Gb of unused
memory in the box.
OK, I give up. Please see if you can think of something less confusing
which involves no negations?
I think this might be slightly clearer:
/proc/sys/vm/highmem_is_dirtyable - defaults to false
[ ... ]
Would you like me to re-submit the patch based on this? I'm
certainly not happy with dirty_highmem as it is now in mm
because it looks backwards and unclear to me.
Well, it was quick enough to just do - here's the patch. I've
also updated the documentation a bit to clarify the intention and
the reasons why you might want to use it (based in part on the
comments to the original change that made highmem uncountable for
dirtyness purposes)
Tested and applied against 2.6.23.9 (our build script makes Debian
packages from a clean unpack of kernel major plus patch minor plus
svn checkout of out quilt series and apply regardless, so it was
just as easy to bump the version number while I was at it).
Builds, boots, passes a quick run of the test program I used last
time around.
Bron.
Add vm.highmem_is_dirtyable toggle
A 32 bit machine with HIGHMEM64 enabled running DCC has an MMAPed file
of approximately 2Gb size which contains a hash format that is written
randomly by the dbclean process. On 2.6.16 this process took a few
minutes. With lowmem only accounting of dirty ratios, this takes about
12 hours of 100% disk IO, all random writes.
This patch includes some code cleanup from Linus and a toggle in
/proc/sys/vm/highmem_is_dirtyable which can be set to 1 to add the
highmem back to the total available memory count.
Signed-off-by: Bron Gondwana [EMAIL PROTECTED]
Index: linux-2.6.23.8-reiserfix-fai-vmdirty/mm/page-writeback.c
===
--- linux-2.6.23.8-reiserfix-fai-vmdirty.orig/mm/page-writeback.c
2007-11-21 21:58:20.0 -0500
+++ linux-2.6.23.8-reiserfix-fai-vmdirty/mm/page-writeback.c2007-11-27
07:27:51.0 -0500
@@ -70,6 +70,12 @@
int dirty_background_ratio = 5;
/*
+ * free highmem will not be subtracted from the total free memory
+ * for calculating free ratios if vm_highmem_is_dirtyable is true
+ */
+int vm_highmem_is_dirtyable;
+
+/*
* The generator of dirty data starts writeback at this percentage
*/
int vm_dirty_ratio = 10;
@@ -153,7 +159,10 @@
x = global_page_state(NR_FREE_PAGES)
+ global_page_state(NR_INACTIVE)
+ global_page_state(NR_ACTIVE);
- x -= highmem_dirtyable_memory(x);
+
+ if (!vm_highmem_is_dirtyable)
+ x -= highmem_dirtyable_memory(x);
+
return x + 1; /* Ensure that we never return 0 */
}
@@ -163,20 +172,12 @@
{
int background_ratio; /* Percentages */
int dirty_ratio;
- int unmapped_ratio;
long background;
long dirty;
unsigned long available_memory = determine_dirtyable_memory();
struct task_struct *tsk;
- unmapped_ratio = 100 - ((global_page_state(NR_FILE_MAPPED) +
- global_page_state(NR_ANON_PAGES)) * 100) /
- available_memory;
-
dirty_ratio = vm_dirty_ratio;
- if (dirty_ratio unmapped_ratio / 2)
- dirty_ratio = unmapped_ratio / 2;
-
if (dirty_ratio 5)
dirty_ratio = 5;
Index: linux-2.6.23.8-reiserfix-fai-vmdirty/include/linux/writeback.h
===
--- linux-2.6.23.8-reiserfix-fai-vmdirty.orig/include/linux/writeback.h
2007-10-09 16:31:38.0 -0400
+++ linux-2.6.23.8-reiserfix-fai-vmdirty/include/linux/writeback.h
2007-11-27 07:22:17.0 -0500
@@ -95,6 +95,7 @@
extern int vm_dirty_ratio;
extern int dirty_writeback_interval;
extern int dirty_expire_interval;
+extern int vm_highmem_is_dirtyable;
extern int block_dump;
extern int laptop_mode;
Index: linux-2.6.23.8-reiserfix-fai-vmdirty/kernel/sysctl.c
===
--- linux-2.6.23.8-reiserfix-fai-vmdirty.orig/kernel/sysctl.c 2007-10-09
16:31:38.0 -0400
+++ linux-2.6.23.8-reiserfix-fai
Re: [PATCH 1/1] mm: add dirty_highmem option
On Mon, 26 Nov 2007 20:54:28 -0800, "Andrew Morton" <[EMAIL PROTECTED]> said: > On Thu, 22 Nov 2007 14:42:04 +1100 Bron Gondwana <[EMAIL PROTECTED]> > wrote: > > > /* > > + * free highmem will not be subtracted from the total free memory > > + * for calculating free ratios if vm_dirty_highmem is true > > + */ > > +int vm_dirty_highmem; > > One would expect that setting dirty_highmem to true would cause highmem > to > be accounted in dirty-memory calculations. However with this change > reality is in fact the inverse of that. > > So how about this? Actually, I'm confused now. Maybe I chose a bad name to begin with. Does it mean "I am allowed to dirty high memory" or "my high memory will be dirty if this is on"? Hmm... I'm even having trouble articulating what's odd about it. I guess my internal model was: "if this flag is set then you are allowed to make high memory dirty without needing to flush it immediately", which is why I made it that way around. No - you're wrong. My patch _did_ include high memory in the dirty memory calculations when dirty_highmem was true. > x = global_page_state(NR_FREE_PAGES) > + global_page_state(NR_INACTIVE) > + global_page_state(NR_ACTIVE); This is the total memory, _including_ high memory. > x -= highmem_dirtyable_memory(x); This removes the high memory from the total count. I think I got it right. If dirty_highmem is set to true, then don't subtract highmem from the total memory count before calculating the percentages. That's what I meant, and that's what the toggle did. Removed the subtraction. Bron. > Documentation/filesystems/proc.txt |4 ++-- > mm/page-writeback.c|8 > 2 files changed, 6 insertions(+), 6 deletions(-) > > diff -puN > Documentation/filesystems/proc.txt~mm-add-dirty_highmem-option-fix > Documentation/filesystems/proc.txt > --- a/Documentation/filesystems/proc.txt~mm-add-dirty_highmem-option-fix > +++ a/Documentation/filesystems/proc.txt > @@ -1265,8 +1265,8 @@ Contains, as a boolean, a switch to allo > part of the "available" memory against which the dirty ratios will be > applied. > > -Setting this to 1 can be useful on 32 bit machines where you want to > make > -random changes within an MMAPed file that is larger than your available > +Setting this to 0 (false) can be useful on 32 bit machines where you > wish to > +make random changes within an MMAPed file that is larger than your > available > lowmem, however it is potentially dangerous and has serious > bounce-buffer > issues. > > diff -puN mm/page-writeback.c~mm-add-dirty_highmem-option-fix > mm/page-writeback.c > --- a/mm/page-writeback.c~mm-add-dirty_highmem-option-fix > +++ a/mm/page-writeback.c > @@ -69,10 +69,10 @@ static inline long sync_writeback_pages( > int dirty_background_ratio = 5; > > /* > - * free highmem will not be subtracted from the total free memory > - * for calculating free ratios if vm_dirty_highmem is true > + * free highmem will be subtracted from the total free memory for > calculating > + * free ratios if vm_dirty_highmem is true > */ > -int vm_dirty_highmem; > +int vm_dirty_highmem = 1; > > /* > * The generator of dirty data starts writeback at this percentage > @@ -293,7 +293,7 @@ static unsigned long determine_dirtyable > x = global_page_state(NR_FREE_PAGES) > + global_page_state(NR_INACTIVE) > + global_page_state(NR_ACTIVE); > - if (!vm_dirty_highmem) > + if (vm_dirty_highmem) > x -= highmem_dirtyable_memory(x); > return x + 1; /* Ensure that we never return 0 */ > } > _ > > > > > (I dropped the already-merged part of your patch) > > (I fixed a build error in kernel/sysctl.c: "one" was defined twice when > suitable config options were set). > > (It's an unpleasing patch, btw. But it's an unpleasant problem and at > least > this way people can tell us "hey, I did and it started to work") -- Bron Gondwana [EMAIL PROTECTED] - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH 1/1] mm: add dirty_highmem option
On Mon, Nov 26, 2007 at 09:53:17AM -0800, Linus Torvalds wrote: > > > On Thu, 22 Nov 2007, Bron Gondwana wrote: > > > > This patch includes some code cleanup from Linus and a toggle in > > /proc/sys/vm/dirty_highmem which can be set to 1 to add the highmem > > back to the total available memory count. > > Just to verify - can you confirm that this "just fixes it" for you? > > I think this is the right approach to take, and seems very safe (ie people > who know that their loads are ok can just set the flag), but I do want to > verify that there was nothing else going on, and that you now see the same > performance as you did in 2.6.16? > > The other alternative, of course, would be to simply allow the dirty > percentages to be > 100%, but that's just *odd* ;) Yes, toggling dirty_highmem "just fixes it" in all our tests. I hadn't tested it on the production machine yet - but I'm just installing it there now since it's been running fine on a less important machine for a few days now. I did wonder about allowing the dirty percentage to go way up, but that would have cause "this one goes up to 110%" comments in the sysctl limits code and people would have thought I was childish. Can't have that. Much better to have "int one = 1" instead. Bron. - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH 1/1] mm: add dirty_highmem option
On Mon, Nov 26, 2007 at 09:53:17AM -0800, Linus Torvalds wrote: On Thu, 22 Nov 2007, Bron Gondwana wrote: This patch includes some code cleanup from Linus and a toggle in /proc/sys/vm/dirty_highmem which can be set to 1 to add the highmem back to the total available memory count. Just to verify - can you confirm that this just fixes it for you? I think this is the right approach to take, and seems very safe (ie people who know that their loads are ok can just set the flag), but I do want to verify that there was nothing else going on, and that you now see the same performance as you did in 2.6.16? The other alternative, of course, would be to simply allow the dirty percentages to be 100%, but that's just *odd* ;) Yes, toggling dirty_highmem just fixes it in all our tests. I hadn't tested it on the production machine yet - but I'm just installing it there now since it's been running fine on a less important machine for a few days now. I did wonder about allowing the dirty percentage to go way up, but that would have cause this one goes up to 110% comments in the sysctl limits code and people would have thought I was childish. Can't have that. Much better to have int one = 1 instead. Bron. - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH 1/1] mm: add dirty_highmem option
On Mon, 26 Nov 2007 20:54:28 -0800, Andrew Morton [EMAIL PROTECTED] said: On Thu, 22 Nov 2007 14:42:04 +1100 Bron Gondwana [EMAIL PROTECTED] wrote: /* + * free highmem will not be subtracted from the total free memory + * for calculating free ratios if vm_dirty_highmem is true + */ +int vm_dirty_highmem; One would expect that setting dirty_highmem to true would cause highmem to be accounted in dirty-memory calculations. However with this change reality is in fact the inverse of that. So how about this? Actually, I'm confused now. Maybe I chose a bad name to begin with. Does it mean I am allowed to dirty high memory or my high memory will be dirty if this is on? Hmm... I'm even having trouble articulating what's odd about it. I guess my internal model was: if this flag is set then you are allowed to make high memory dirty without needing to flush it immediately, which is why I made it that way around. No - you're wrong. My patch _did_ include high memory in the dirty memory calculations when dirty_highmem was true. x = global_page_state(NR_FREE_PAGES) + global_page_state(NR_INACTIVE) + global_page_state(NR_ACTIVE); This is the total memory, _including_ high memory. x -= highmem_dirtyable_memory(x); This removes the high memory from the total count. I think I got it right. If dirty_highmem is set to true, then don't subtract highmem from the total memory count before calculating the percentages. That's what I meant, and that's what the toggle did. Removed the subtraction. Bron. Documentation/filesystems/proc.txt |4 ++-- mm/page-writeback.c|8 2 files changed, 6 insertions(+), 6 deletions(-) diff -puN Documentation/filesystems/proc.txt~mm-add-dirty_highmem-option-fix Documentation/filesystems/proc.txt --- a/Documentation/filesystems/proc.txt~mm-add-dirty_highmem-option-fix +++ a/Documentation/filesystems/proc.txt @@ -1265,8 +1265,8 @@ Contains, as a boolean, a switch to allo part of the available memory against which the dirty ratios will be applied. -Setting this to 1 can be useful on 32 bit machines where you want to make -random changes within an MMAPed file that is larger than your available +Setting this to 0 (false) can be useful on 32 bit machines where you wish to +make random changes within an MMAPed file that is larger than your available lowmem, however it is potentially dangerous and has serious bounce-buffer issues. diff -puN mm/page-writeback.c~mm-add-dirty_highmem-option-fix mm/page-writeback.c --- a/mm/page-writeback.c~mm-add-dirty_highmem-option-fix +++ a/mm/page-writeback.c @@ -69,10 +69,10 @@ static inline long sync_writeback_pages( int dirty_background_ratio = 5; /* - * free highmem will not be subtracted from the total free memory - * for calculating free ratios if vm_dirty_highmem is true + * free highmem will be subtracted from the total free memory for calculating + * free ratios if vm_dirty_highmem is true */ -int vm_dirty_highmem; +int vm_dirty_highmem = 1; /* * The generator of dirty data starts writeback at this percentage @@ -293,7 +293,7 @@ static unsigned long determine_dirtyable x = global_page_state(NR_FREE_PAGES) + global_page_state(NR_INACTIVE) + global_page_state(NR_ACTIVE); - if (!vm_dirty_highmem) + if (vm_dirty_highmem) x -= highmem_dirtyable_memory(x); return x + 1; /* Ensure that we never return 0 */ } _ (I dropped the already-merged part of your patch) (I fixed a build error in kernel/sysctl.c: one was defined twice when suitable config options were set). (It's an unpleasing patch, btw. But it's an unpleasant problem and at least this way people can tell us hey, I did this and it started to work) -- Bron Gondwana [EMAIL PROTECTED] - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
[PATCH 1/1] mm: add dirty_highmem option
On Thu, Nov 15, 2007 at 01:14:32PM -0800, Linus Torvalds wrote:
> Examples of non-broken solutions:
> (a) always use lowmem sizes (what we do now)
> (b) always use total mem sizes (sane but potentially dangerous: but the
> VM pressure should work! It has serious bounce-buffer issues, though,
> which is why I think it's crazy even if it's otherwise consistent)
>
> Btw, I actually suspect that while (a) is what we do now, for the specific
> case that Bron has, we could have a /proc/sys/vm option to just enable
> (b). So we don't have to have just one consistent model, we can allow odd
> users (and Bron sounds like one - sorry Bron ;) to just force other, odd,
> but consistent models.
A 32 bit machine with HIGHMEM64 enabled running DCC has an MMAPed file
of approximately 2Gb size which contains a hash format that is written
"randomly" by the dbclean process. On 2.6.16 this process took a few
minutes. With lowmem only accounting of dirty ratios, this takes about
12 hours of 100% disk IO, all random writes.
This patch includes some code cleanup from Linus and a toggle in
/proc/sys/vm/dirty_highmem which can be set to 1 to add the highmem
back to the total available memory count.
Signed-off-by: Bron Gondwana <[EMAIL PROTECTED]>
Index: linux-2.6.23.8-reiserfix-fai-vmdirty/mm/page-writeback.c
===
--- linux-2.6.23.8-reiserfix-fai-vmdirty.orig/mm/page-writeback.c
2007-11-22 01:48:20.0 +
+++ linux-2.6.23.8-reiserfix-fai-vmdirty/mm/page-writeback.c2007-11-22
02:42:04.0 +
@@ -70,6 +70,12 @@ static inline long sync_writeback_pages(
int dirty_background_ratio = 5;
/*
+ * free highmem will not be subtracted from the total free memory
+ * for calculating free ratios if vm_dirty_highmem is true
+ */
+int vm_dirty_highmem;
+
+/*
* The generator of dirty data starts writeback at this percentage
*/
int vm_dirty_ratio = 10;
@@ -153,7 +159,8 @@ static unsigned long determine_dirtyable
x = global_page_state(NR_FREE_PAGES)
+ global_page_state(NR_INACTIVE)
+ global_page_state(NR_ACTIVE);
- x -= highmem_dirtyable_memory(x);
+ if (!vm_dirty_highmem)
+ x -= highmem_dirtyable_memory(x);
return x + 1; /* Ensure that we never return 0 */
}
@@ -163,20 +170,12 @@ get_dirty_limits(long *pbackground, long
{
int background_ratio; /* Percentages */
int dirty_ratio;
- int unmapped_ratio;
long background;
long dirty;
unsigned long available_memory = determine_dirtyable_memory();
struct task_struct *tsk;
- unmapped_ratio = 100 - ((global_page_state(NR_FILE_MAPPED) +
- global_page_state(NR_ANON_PAGES)) * 100) /
- available_memory;
-
dirty_ratio = vm_dirty_ratio;
- if (dirty_ratio > unmapped_ratio / 2)
- dirty_ratio = unmapped_ratio / 2;
-
if (dirty_ratio < 5)
dirty_ratio = 5;
Index: linux-2.6.23.8-reiserfix-fai-vmdirty/include/linux/writeback.h
===
--- linux-2.6.23.8-reiserfix-fai-vmdirty.orig/include/linux/writeback.h
2007-10-09 20:31:38.0 +
+++ linux-2.6.23.8-reiserfix-fai-vmdirty/include/linux/writeback.h
2007-11-22 01:48:21.0 +
@@ -92,6 +92,7 @@ void throttle_vm_writeout(gfp_t gfp_mask
/* These are exported to sysctl. */
extern int dirty_background_ratio;
+extern int vm_dirty_highmem;
extern int vm_dirty_ratio;
extern int dirty_writeback_interval;
extern int dirty_expire_interval;
Index: linux-2.6.23.8-reiserfix-fai-vmdirty/kernel/sysctl.c
===
--- linux-2.6.23.8-reiserfix-fai-vmdirty.orig/kernel/sysctl.c 2007-10-09
20:31:38.0 +
+++ linux-2.6.23.8-reiserfix-fai-vmdirty/kernel/sysctl.c2007-11-22
01:48:21.0 +
@@ -776,6 +776,7 @@ static ctl_table kern_table[] = {
/* Constants for minimum and maximum testing in vm_table.
We use these as one-element integer vectors. */
static int zero;
+static int one = 1;
static int two = 2;
static int one_hundred = 100;
@@ -1066,6 +1067,19 @@ static ctl_table vm_table[] = {
.extra1 = ,
},
#endif
+#ifdef CONFIG_HIGHMEM
+ {
+ .ctl_name = CTL_UNNUMBERED,
+ .procname = "dirty_highmem",
+ .data = _dirty_highmem,
+ .maxlen = sizeof(vm_dirty_highmem),
+ .mode = 0644,
+ .proc_handler = _dointvec_minmax,
+ .strategy = _intvec,
+ .extra1 = ,
+ .extra2 = ,
+ },
+#endif
/*
* NOTE: do not add new entries to this table
Re: mmap dirty limits on 32 bit kernels (Was: [BUG] New Kernel Bugs)
On Thu, Nov 22, 2007 at 10:51:15AM +1100, Bron Gondwana wrote: > On Thu, Nov 15, 2007 at 08:32:22AM -0800, Linus Torvalds wrote: > > If this patch makes a difference, please holler. I think it's the correct > > thing to do, but I'm not going to actually commit it without somebody > > saying that it makes a difference (and preferably Peter Zijlstra and > > Andrew acking it too). > > mmap: mmap call failed: errno: 12 errmsg: Cannot allocate memory > > Yep, that's "fixed" the problem alright! No way this puppy is > dirtying 2Gb of memory any more. > > http://linux.brong.fastmail.fm/2007-11-22/bmtest.pl Alternatively perhaps I'm just a moron who used a config file with: CONFIG_PAGE_OFFSET=0x8000 set to build the new kernel (I hadn't committed it because it turned out not to solve the issue it was there for). That would explain a few things. [EMAIL PROTECTED] perl]$ free total used free sharedbuffers cached Mem: 415062022722841878336 0 112122066536 -/+ buffers/cache: 1945363956084 Swap: 2096472 02096472 That's more the usage I would expect to see. Now for the downside. It works again, but it still runs slow. Seems to hit (and this is totally unscientific, I'm just watching the numbers scroll by) at about 12 writes rather than 7 writes, but that's still not fitting the while file dirty. I notice that PF_LESS_THROTTLE gets set by nfsd to get an extra 25% bonus free space allocated. Potentially dcc could use similar tricks to claim extra space if that knob is available up in userspace. I'm happy to patch dcc as well if I have to, I'm already backporting it, so adding another little quilt directory and applying it is pretty trivial (must try guilt/stgit one of these days) Bron. - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: mmap dirty limits on 32 bit kernels (Was: [BUG] New Kernel Bugs)
On Thu, Nov 15, 2007 at 08:32:22AM -0800, Linus Torvalds wrote:
> On Thu, 15 Nov 2007, Bron Gondwana wrote:
> >
> > I guess we'll be doing the one-liner kernel mod and testing
> > that then.
>
> The thing to look at is "get_dirty_limits()" in mm/page-writeback.c, and
> in this particular case it's the
>
> unsigned long available_memory = determine_dirtyable_memory();
>
> that's going to bite you. In particular, note the
>
> x -= highmem_dirtyable_memory(x);
>
> that we do in determine_dirtyable_memory().
>
> So in this case, if you basically remove that line, it will allow all of
> memory to be dirtied (including highmem), and then the background_ratio
> will work on the whole 6GB.
>
> HOWEVER! It's worth noting that we also have some other old legacy cruft
> there that may interfere with your code. In particular, if you look at the
> top of "get_dirty_limits()", it *also* does a
>
> unmapped_ratio = 100 - ((global_page_state(NR_FILE_MAPPED) +
> global_page_state(NR_ANON_PAGES)) * 100) /
> available_memory;
>
> dirty_ratio = vm_dirty_ratio;
> if (dirty_ratio > unmapped_ratio / 2)
> dirty_ratio = unmapped_ratio / 2;
>
> and that whole "unmapped_ratio" comparison is probably bogus these days,
> since we now take the mapped dirty pages into account. That code harks
> back to the days before we did that, and dirty ratios only affected
> non-mapped pages.
>
> And in particular, now that I look at it, I wonder if it can even go
> negative (because "available_memory" may be *smaller* than the
> NR_FILE_MAPPED|ANON_PAGES sum!).
>
> We'll fix up a negative value anyway (because of the clamping of
> dirty_ratio to no less than 5), but the point is that the whole
> "unmapped_ratio" thing probably doesn't make sense any more, and may well
> make the dirty_ratio not work for you, because you may have a very small
> unmapped_ratio that effectively makes all dirty limits always clamp to a
> very small value.
>
> So regardless, I think you may want to try the appended patch *first*.
>
> If this patch makes a difference, please holler. I think it's the correct
> thing to do, but I'm not going to actually commit it without somebody
> saying that it makes a difference (and preferably Peter Zijlstra and
> Andrew acking it too).
mmap: mmap call failed: errno: 12 errmsg: Cannot allocate memory
Yep, that's "fixed" the problem alright! No way this puppy is
dirtying 2Gb of memory any more.
http://linux.brong.fastmail.fm/2007-11-22/bmtest.pl
That said, pushing the size down to 1700 rather than 2000 in that
file makes it run, and the behaviour matches the 2000 Mb case on
2.6.16.55 rather than 2.6.20.20 or 2.6.23.1 (my other test case
kernels that happened to be pre-built on that machine)
[EMAIL PROTECTED] ~]$ free
total used free sharedbuffers cached
Mem: 414983620730562076780 0 220361846096
-/+ buffers/cache: 2049243944912
Swap: 2096472 02096472
That's after running the 1700Mb version. You can see this machine is our
one remaining 4Gb machine (it's not running any production services unlike
the 6Gb machine, so it's better for testing)
Anyway - looks like this may be a "good enough" solution for out1 if
it can manage an ~2Gb file with 6Gb of memory available. I'll test
that later today - but I should drag myself into the office now...
Bron.
(patch left attached below for reference)
> Only *after* testing this change is it probably a good idea to test the
> real hack of then removing the highmem_dirtyable_memory() thing.
>
> Peter? Andrew?
>
> Linus
>
> ---
> mm/page-writeback.c |8
> 1 files changed, 0 insertions(+), 8 deletions(-)
>
> diff --git a/mm/page-writeback.c b/mm/page-writeback.c
> index 81a91e6..d55cfca 100644
> --- a/mm/page-writeback.c
> +++ b/mm/page-writeback.c
> @@ -297,20 +297,12 @@ get_dirty_limits(long *pbackground, long *pdirty, long
> *pbdi_dirty,
> {
> int background_ratio; /* Percentages */
> int dirty_ratio;
> - int unmapped_ratio;
> long background;
> long dirty;
> unsigned long available_memory = determine_dirtyable_memory();
> struct task_struct *tsk;
>
> - unmapped_ratio = 100 - ((global_page_state(NR_FILE_MAPPED) +
> - global_page_state(NR_ANON_PAGES)) * 100) /
> - available_memory;
> -
> dirty_ratio = vm_dirty_ratio;
> - if (dirty_ratio > unm
Re: mmap dirty limits on 32 bit kernels (Was: [BUG] New Kernel Bugs)
On Thu, Nov 15, 2007 at 08:32:22AM -0800, Linus Torvalds wrote:
On Thu, 15 Nov 2007, Bron Gondwana wrote:
I guess we'll be doing the one-liner kernel mod and testing
that then.
The thing to look at is get_dirty_limits() in mm/page-writeback.c, and
in this particular case it's the
unsigned long available_memory = determine_dirtyable_memory();
that's going to bite you. In particular, note the
x -= highmem_dirtyable_memory(x);
that we do in determine_dirtyable_memory().
So in this case, if you basically remove that line, it will allow all of
memory to be dirtied (including highmem), and then the background_ratio
will work on the whole 6GB.
HOWEVER! It's worth noting that we also have some other old legacy cruft
there that may interfere with your code. In particular, if you look at the
top of get_dirty_limits(), it *also* does a
unmapped_ratio = 100 - ((global_page_state(NR_FILE_MAPPED) +
global_page_state(NR_ANON_PAGES)) * 100) /
available_memory;
dirty_ratio = vm_dirty_ratio;
if (dirty_ratio unmapped_ratio / 2)
dirty_ratio = unmapped_ratio / 2;
and that whole unmapped_ratio comparison is probably bogus these days,
since we now take the mapped dirty pages into account. That code harks
back to the days before we did that, and dirty ratios only affected
non-mapped pages.
And in particular, now that I look at it, I wonder if it can even go
negative (because available_memory may be *smaller* than the
NR_FILE_MAPPED|ANON_PAGES sum!).
We'll fix up a negative value anyway (because of the clamping of
dirty_ratio to no less than 5), but the point is that the whole
unmapped_ratio thing probably doesn't make sense any more, and may well
make the dirty_ratio not work for you, because you may have a very small
unmapped_ratio that effectively makes all dirty limits always clamp to a
very small value.
So regardless, I think you may want to try the appended patch *first*.
If this patch makes a difference, please holler. I think it's the correct
thing to do, but I'm not going to actually commit it without somebody
saying that it makes a difference (and preferably Peter Zijlstra and
Andrew acking it too).
mmap: mmap call failed: errno: 12 errmsg: Cannot allocate memory
Yep, that's fixed the problem alright! No way this puppy is
dirtying 2Gb of memory any more.
http://linux.brong.fastmail.fm/2007-11-22/bmtest.pl
That said, pushing the size down to 1700 rather than 2000 in that
file makes it run, and the behaviour matches the 2000 Mb case on
2.6.16.55 rather than 2.6.20.20 or 2.6.23.1 (my other test case
kernels that happened to be pre-built on that machine)
[EMAIL PROTECTED] ~]$ free
total used free sharedbuffers cached
Mem: 414983620730562076780 0 220361846096
-/+ buffers/cache: 2049243944912
Swap: 2096472 02096472
That's after running the 1700Mb version. You can see this machine is our
one remaining 4Gb machine (it's not running any production services unlike
the 6Gb machine, so it's better for testing)
Anyway - looks like this may be a good enough solution for out1 if
it can manage an ~2Gb file with 6Gb of memory available. I'll test
that later today - but I should drag myself into the office now...
Bron.
(patch left attached below for reference)
Only *after* testing this change is it probably a good idea to test the
real hack of then removing the highmem_dirtyable_memory() thing.
Peter? Andrew?
Linus
---
mm/page-writeback.c |8
1 files changed, 0 insertions(+), 8 deletions(-)
diff --git a/mm/page-writeback.c b/mm/page-writeback.c
index 81a91e6..d55cfca 100644
--- a/mm/page-writeback.c
+++ b/mm/page-writeback.c
@@ -297,20 +297,12 @@ get_dirty_limits(long *pbackground, long *pdirty, long
*pbdi_dirty,
{
int background_ratio; /* Percentages */
int dirty_ratio;
- int unmapped_ratio;
long background;
long dirty;
unsigned long available_memory = determine_dirtyable_memory();
struct task_struct *tsk;
- unmapped_ratio = 100 - ((global_page_state(NR_FILE_MAPPED) +
- global_page_state(NR_ANON_PAGES)) * 100) /
- available_memory;
-
dirty_ratio = vm_dirty_ratio;
- if (dirty_ratio unmapped_ratio / 2)
- dirty_ratio = unmapped_ratio / 2;
-
if (dirty_ratio 5)
dirty_ratio = 5;
-
To unsubscribe from this list: send the line unsubscribe linux-kernel in
the body of a message to [EMAIL PROTECTED]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Re: mmap dirty limits on 32 bit kernels (Was: [BUG] New Kernel Bugs)
On Thu, Nov 22, 2007 at 10:51:15AM +1100, Bron Gondwana wrote: On Thu, Nov 15, 2007 at 08:32:22AM -0800, Linus Torvalds wrote: If this patch makes a difference, please holler. I think it's the correct thing to do, but I'm not going to actually commit it without somebody saying that it makes a difference (and preferably Peter Zijlstra and Andrew acking it too). mmap: mmap call failed: errno: 12 errmsg: Cannot allocate memory Yep, that's fixed the problem alright! No way this puppy is dirtying 2Gb of memory any more. http://linux.brong.fastmail.fm/2007-11-22/bmtest.pl Alternatively perhaps I'm just a moron who used a config file with: CONFIG_PAGE_OFFSET=0x8000 set to build the new kernel (I hadn't committed it because it turned out not to solve the issue it was there for). That would explain a few things. [EMAIL PROTECTED] perl]$ free total used free sharedbuffers cached Mem: 415062022722841878336 0 112122066536 -/+ buffers/cache: 1945363956084 Swap: 2096472 02096472 That's more the usage I would expect to see. Now for the downside. It works again, but it still runs slow. Seems to hit (and this is totally unscientific, I'm just watching the numbers scroll by) at about 12 writes rather than 7 writes, but that's still not fitting the while file dirty. I notice that PF_LESS_THROTTLE gets set by nfsd to get an extra 25% bonus free space allocated. Potentially dcc could use similar tricks to claim extra space if that knob is available up in userspace. I'm happy to patch dcc as well if I have to, I'm already backporting it, so adding another little quilt directory and applying it is pretty trivial (must try guilt/stgit one of these days) Bron. - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
[PATCH 1/1] mm: add dirty_highmem option
On Thu, Nov 15, 2007 at 01:14:32PM -0800, Linus Torvalds wrote:
Examples of non-broken solutions:
(a) always use lowmem sizes (what we do now)
(b) always use total mem sizes (sane but potentially dangerous: but the
VM pressure should work! It has serious bounce-buffer issues, though,
which is why I think it's crazy even if it's otherwise consistent)
Btw, I actually suspect that while (a) is what we do now, for the specific
case that Bron has, we could have a /proc/sys/vm option to just enable
(b). So we don't have to have just one consistent model, we can allow odd
users (and Bron sounds like one - sorry Bron ;) to just force other, odd,
but consistent models.
A 32 bit machine with HIGHMEM64 enabled running DCC has an MMAPed file
of approximately 2Gb size which contains a hash format that is written
randomly by the dbclean process. On 2.6.16 this process took a few
minutes. With lowmem only accounting of dirty ratios, this takes about
12 hours of 100% disk IO, all random writes.
This patch includes some code cleanup from Linus and a toggle in
/proc/sys/vm/dirty_highmem which can be set to 1 to add the highmem
back to the total available memory count.
Signed-off-by: Bron Gondwana [EMAIL PROTECTED]
Index: linux-2.6.23.8-reiserfix-fai-vmdirty/mm/page-writeback.c
===
--- linux-2.6.23.8-reiserfix-fai-vmdirty.orig/mm/page-writeback.c
2007-11-22 01:48:20.0 +
+++ linux-2.6.23.8-reiserfix-fai-vmdirty/mm/page-writeback.c2007-11-22
02:42:04.0 +
@@ -70,6 +70,12 @@ static inline long sync_writeback_pages(
int dirty_background_ratio = 5;
/*
+ * free highmem will not be subtracted from the total free memory
+ * for calculating free ratios if vm_dirty_highmem is true
+ */
+int vm_dirty_highmem;
+
+/*
* The generator of dirty data starts writeback at this percentage
*/
int vm_dirty_ratio = 10;
@@ -153,7 +159,8 @@ static unsigned long determine_dirtyable
x = global_page_state(NR_FREE_PAGES)
+ global_page_state(NR_INACTIVE)
+ global_page_state(NR_ACTIVE);
- x -= highmem_dirtyable_memory(x);
+ if (!vm_dirty_highmem)
+ x -= highmem_dirtyable_memory(x);
return x + 1; /* Ensure that we never return 0 */
}
@@ -163,20 +170,12 @@ get_dirty_limits(long *pbackground, long
{
int background_ratio; /* Percentages */
int dirty_ratio;
- int unmapped_ratio;
long background;
long dirty;
unsigned long available_memory = determine_dirtyable_memory();
struct task_struct *tsk;
- unmapped_ratio = 100 - ((global_page_state(NR_FILE_MAPPED) +
- global_page_state(NR_ANON_PAGES)) * 100) /
- available_memory;
-
dirty_ratio = vm_dirty_ratio;
- if (dirty_ratio unmapped_ratio / 2)
- dirty_ratio = unmapped_ratio / 2;
-
if (dirty_ratio 5)
dirty_ratio = 5;
Index: linux-2.6.23.8-reiserfix-fai-vmdirty/include/linux/writeback.h
===
--- linux-2.6.23.8-reiserfix-fai-vmdirty.orig/include/linux/writeback.h
2007-10-09 20:31:38.0 +
+++ linux-2.6.23.8-reiserfix-fai-vmdirty/include/linux/writeback.h
2007-11-22 01:48:21.0 +
@@ -92,6 +92,7 @@ void throttle_vm_writeout(gfp_t gfp_mask
/* These are exported to sysctl. */
extern int dirty_background_ratio;
+extern int vm_dirty_highmem;
extern int vm_dirty_ratio;
extern int dirty_writeback_interval;
extern int dirty_expire_interval;
Index: linux-2.6.23.8-reiserfix-fai-vmdirty/kernel/sysctl.c
===
--- linux-2.6.23.8-reiserfix-fai-vmdirty.orig/kernel/sysctl.c 2007-10-09
20:31:38.0 +
+++ linux-2.6.23.8-reiserfix-fai-vmdirty/kernel/sysctl.c2007-11-22
01:48:21.0 +
@@ -776,6 +776,7 @@ static ctl_table kern_table[] = {
/* Constants for minimum and maximum testing in vm_table.
We use these as one-element integer vectors. */
static int zero;
+static int one = 1;
static int two = 2;
static int one_hundred = 100;
@@ -1066,6 +1067,19 @@ static ctl_table vm_table[] = {
.extra1 = zero,
},
#endif
+#ifdef CONFIG_HIGHMEM
+ {
+ .ctl_name = CTL_UNNUMBERED,
+ .procname = dirty_highmem,
+ .data = vm_dirty_highmem,
+ .maxlen = sizeof(vm_dirty_highmem),
+ .mode = 0644,
+ .proc_handler = proc_dointvec_minmax,
+ .strategy = sysctl_intvec,
+ .extra1 = zero,
+ .extra2 = one,
+ },
+#endif
/*
* NOTE: do not add new entries to this table unless you have read
* Documentation/sysctl
Re: mmap dirty limits on 32 bit kernels (Was: [BUG] New Kernel Bugs)
On Thu, Nov 15, 2007 at 01:14:32PM -0800, Linus Torvalds wrote: Sorry about not replying to this earlier. I actually got a weekend away from the computer pretty much last weekend - took the kids swimming, helped a friend clear dead wood from around her house before the fire season. Shocking I know. > On Thu, 15 Nov 2007, Linus Torvalds wrote: > > > > Unacceptable. We used to do exactly what your patch does, and it got fixed > > once. We're not introducing that fundamentally broken concept again. > > Examples of non-broken solutions: > (a) always use lowmem sizes (what we do now) > (b) always use total mem sizes (sane but potentially dangerous: but the > VM pressure should work! It has serious bounce-buffer issues, though, > which is why I think it's crazy even if it's otherwise consistent) > (c) make all dirty counting be *purely* per-bdi, so that everybody can > disagree on what the limits are, but at least they also then use > different counters > > So it's just the "different writers look at the same dirty counts but then > interpret it to mean totally different things" that I think is so > fundamentally bogus. I'm not claiming that what we do now is the only way > to do things, I just don't think your approach is tenable. > > Btw, I actually suspect that while (a) is what we do now, for the specific > case that Bron has, we could have a /proc/sys/vm option to just enable > (b). So we don't have to have just one consistent model, we can allow odd > users (and Bron sounds like one - sorry Bron ;) to just force other, odd, > but consistent models. Hey, if Andrew Morton can tell us we find all the interesting bugs, you can call me odd. I've been called worse! We also run ReiserFS (3 of course, I tried 4 and it et my laptop disk) on all our production IMAP servers. Tried ext3 and the performance was so horrible that our users hated us (and I hated being woken in the night by things timing out and paging me). And I'm spending far too long still writing C thanks to Cyrus having enough bugs to keep me busy for the rest of my natural life if I don't break and go write my own IMAP server at some point. *clears throat* > I'd also like to point out that while the "bounce buffer" issue is not so > much a HIGHMEM issue on its own (it's really about the device DMA limits, > which are _independent_ of HIGHMEM, of course), the reason HIGHMEM is > special is that without HIGHMEM the bounce buffers generally work > perfectly fine. > > The problem with HIGHMEM is that it causes various metadata (dentries, > inodes, page struct tables etc) to eat up memory "prime real estate" under > the same kind of conditions that also dirty a lot of memory. So the reason > we disallow HIGHMEM from dirty limits is only *partly* the per-device or > mapping DMA limits, and to a large degree the fact that non-highmem memory > is special in general, and it is usually the non-highmem areas that are > constrained - and need to be protected. I'm going to finish off writing a decent test case so I can reliably reproduce the problem first, and then go compile a small set of kernels with the various patches that have been thrown around here and see if they solve the problems for me. Thankfully I don't have the same problem you do Linus - I don't care if any particular patch isn't consistent - isn't fair in the general sense - even "doesn't work for anyone else". So long as it's stable and it works on this machine I'm happy to support it through the next couple of years until we either get a world facing 64 bit machine with the spare capacity to run DCC or we drop DCC. The only reason to upgrade the kernel there at all is keeping up-to-date with security patches, and the relative tradeoffs of backporting (or expecting Adrian Bunk to keep doing it for us) rather than maintaing a small patch to keep the behaviour of one thing we like. And to all of you in this thread (especially Linus and Peter) - thanks heaps for grabbing on to a throw away line in an unrelated discussion and putting the work in to: a) explain the problem and the cause to me before I put in heaps of work tracking it down; and b) putting together some patches for me to test. I saw a couple of days ago someone posted an "Ask Slashdot" whether 6 weeks was an appropriate time for a software vendor to get a fix out to a customer and implying that the customer was an unrealistic whiner to expect anyone to be able to do better. I'll be able to point to this thread if anyone suggests that you can't get decent support on Linux! Bron. - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: mmap dirty limits on 32 bit kernels (Was: [BUG] New Kernel Bugs)
On Sun, Nov 18, 2007 at 04:13:18PM -0700, Daniel Phillips wrote: > On Thursday 15 November 2007 14:24, Rob Mueller wrote: > > > That's my personal opinion, and I realize that some of the > > > commercial vendors may care about their insane customers' > > > satisfaction, but I'm simply not interested in insane users. If > > > they have that much RAM (and bought it a few years ago when a > > > 64-bit CPU wasn't an option), they can't be poor. > > > > From our perspective, the main issue is that some of these machines > > we spent quite a bit of money on the big RAM (for it's day) + lots of > > 15k RPM SCSI drives + multi-year support contracts. They're highly IO > > bound, and barely use 10-20% of their old 2.4Ghz Prestonia Xeon CPUs. > > It's hard to justify junking those machines < 5 years. > > > > We have a couple of 6G machines and some 8G machines using PAE. On > > the whole, they actually have been working really well (hmmm, apart > > from the recent dirty pages issue + reiserfs data=journal leaks + > > inodes in lowmem limits) > > Junk everything except the 15K drives, you will be glad you did. Too > bad about those multi-year support contracts, hopefully you got a deal > on them. Actually, the 15K drives are the bit we're getting the least use out of now, since we're moving everything to external SATA units that are more easily swapable. > Prediction: after these dirty pages issues are gone, there will be more > dirty page issues because the notion of dirty page limit is > fundamentally broken. Your smartest recourse is to re-motherboard to a > place where the dirty page limit borkage does not hurt as much, and in > the process you will get a cheap hardware upgrade. Everybody will be > happy, the sun will come out, the birds will sing. Or just keep running 2.6.16 where it's all been working quite fine thanks very much, or maintain a simple patch that rips all that out since we don't care too much about "fairness" - we only run a couple of things on that machine and they run fine. Bron ( going to settle down and really test this stuff to make sure we have an acceptable "fix" for us then do it! ) - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: mmap dirty limits on 32 bit kernels (Was: [BUG] New Kernel Bugs)
On Sun, Nov 18, 2007 at 04:13:18PM -0700, Daniel Phillips wrote: On Thursday 15 November 2007 14:24, Rob Mueller wrote: That's my personal opinion, and I realize that some of the commercial vendors may care about their insane customers' satisfaction, but I'm simply not interested in insane users. If they have that much RAM (and bought it a few years ago when a 64-bit CPU wasn't an option), they can't be poor. From our perspective, the main issue is that some of these machines we spent quite a bit of money on the big RAM (for it's day) + lots of 15k RPM SCSI drives + multi-year support contracts. They're highly IO bound, and barely use 10-20% of their old 2.4Ghz Prestonia Xeon CPUs. It's hard to justify junking those machines 5 years. We have a couple of 6G machines and some 8G machines using PAE. On the whole, they actually have been working really well (hmmm, apart from the recent dirty pages issue + reiserfs data=journal leaks + inodes in lowmem limits) Junk everything except the 15K drives, you will be glad you did. Too bad about those multi-year support contracts, hopefully you got a deal on them. Actually, the 15K drives are the bit we're getting the least use out of now, since we're moving everything to external SATA units that are more easily swapable. Prediction: after these dirty pages issues are gone, there will be more dirty page issues because the notion of dirty page limit is fundamentally broken. Your smartest recourse is to re-motherboard to a place where the dirty page limit borkage does not hurt as much, and in the process you will get a cheap hardware upgrade. Everybody will be happy, the sun will come out, the birds will sing. Or just keep running 2.6.16 where it's all been working quite fine thanks very much, or maintain a simple patch that rips all that out since we don't care too much about fairness - we only run a couple of things on that machine and they run fine. Bron ( going to settle down and really test this stuff to make sure we have an acceptable fix for us then do it! ) - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: mmap dirty limits on 32 bit kernels (Was: [BUG] New Kernel Bugs)
On Thu, Nov 15, 2007 at 01:14:32PM -0800, Linus Torvalds wrote: Sorry about not replying to this earlier. I actually got a weekend away from the computer pretty much last weekend - took the kids swimming, helped a friend clear dead wood from around her house before the fire season. Shocking I know. On Thu, 15 Nov 2007, Linus Torvalds wrote: Unacceptable. We used to do exactly what your patch does, and it got fixed once. We're not introducing that fundamentally broken concept again. Examples of non-broken solutions: (a) always use lowmem sizes (what we do now) (b) always use total mem sizes (sane but potentially dangerous: but the VM pressure should work! It has serious bounce-buffer issues, though, which is why I think it's crazy even if it's otherwise consistent) (c) make all dirty counting be *purely* per-bdi, so that everybody can disagree on what the limits are, but at least they also then use different counters So it's just the different writers look at the same dirty counts but then interpret it to mean totally different things that I think is so fundamentally bogus. I'm not claiming that what we do now is the only way to do things, I just don't think your approach is tenable. Btw, I actually suspect that while (a) is what we do now, for the specific case that Bron has, we could have a /proc/sys/vm option to just enable (b). So we don't have to have just one consistent model, we can allow odd users (and Bron sounds like one - sorry Bron ;) to just force other, odd, but consistent models. Hey, if Andrew Morton can tell us we find all the interesting bugs, you can call me odd. I've been called worse! We also run ReiserFS (3 of course, I tried 4 and it et my laptop disk) on all our production IMAP servers. Tried ext3 and the performance was so horrible that our users hated us (and I hated being woken in the night by things timing out and paging me). And I'm spending far too long still writing C thanks to Cyrus having enough bugs to keep me busy for the rest of my natural life if I don't break and go write my own IMAP server at some point. *clears throat* I'd also like to point out that while the bounce buffer issue is not so much a HIGHMEM issue on its own (it's really about the device DMA limits, which are _independent_ of HIGHMEM, of course), the reason HIGHMEM is special is that without HIGHMEM the bounce buffers generally work perfectly fine. The problem with HIGHMEM is that it causes various metadata (dentries, inodes, page struct tables etc) to eat up memory prime real estate under the same kind of conditions that also dirty a lot of memory. So the reason we disallow HIGHMEM from dirty limits is only *partly* the per-device or mapping DMA limits, and to a large degree the fact that non-highmem memory is special in general, and it is usually the non-highmem areas that are constrained - and need to be protected. I'm going to finish off writing a decent test case so I can reliably reproduce the problem first, and then go compile a small set of kernels with the various patches that have been thrown around here and see if they solve the problems for me. Thankfully I don't have the same problem you do Linus - I don't care if any particular patch isn't consistent - isn't fair in the general sense - even doesn't work for anyone else. So long as it's stable and it works on this machine I'm happy to support it through the next couple of years until we either get a world facing 64 bit machine with the spare capacity to run DCC or we drop DCC. The only reason to upgrade the kernel there at all is keeping up-to-date with security patches, and the relative tradeoffs of backporting (or expecting Adrian Bunk to keep doing it for us) rather than maintaing a small patch to keep the behaviour of one thing we like. And to all of you in this thread (especially Linus and Peter) - thanks heaps for grabbing on to a throw away line in an unrelated discussion and putting the work in to: a) explain the problem and the cause to me before I put in heaps of work tracking it down; and b) putting together some patches for me to test. I saw a couple of days ago someone posted an Ask Slashdot whether 6 weeks was an appropriate time for a software vendor to get a fix out to a customer and implying that the customer was an unrealistic whiner to expect anyone to be able to do better. I'll be able to point to this thread if anyone suggests that you can't get decent support on Linux! Bron. - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [alsa-devel] [BUG] New Kernel Bugs
On Thu, Nov 15, 2007 at 06:59:34AM +0100, Rene Herman wrote: > On 15-11-07 05:16, Bron Gondwana wrote: > >> Totally unrelated - I sent something to the kolab mailing list a couple > > [ ... ] > >> I'm sure if I had something that I considered worth informing the ALSA >> project of, I'd be wary of spending the same effort writing a good post >> knowing it may be dropped in between the by a list moderator just selecing >> all and bouncing them. > > Totally unrelated indeed so why are spouting crap? If the kohab list has a > problem take it up with them but keep ALSA out of it. alsa-devel has only > ever moderated out spam -- nothing else. As an outsider to the list, how do I know what your policy will be other than "I've been rejected out of hand by someone else's list, so my experience is that member only lists aren't willing to listen to something I have to say unless I make the effort to sign up and have yet another folder accumulating unread messages". I don't. Well, ok - maybe I do here since I've let myself be dragged in to the debate. Oops. I get the same information from both project websites: "moderated for non-members, public archives" - no way of knowing that ALSA will accept me informing them of something they would be interested without committing to reading or bit-bucketing their list. The alternative is to subscribe just long enough to send something and then unsubscribe again or cold-email a member and ask them to pass a message along. Or post and hope it doesn't get rejected, not even knowing for a day or so. Bron. - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
mmap dirty limits on 32 bit kernels (Was: [BUG] New Kernel Bugs)
On Wed, Nov 14, 2007 at 09:53:38PM -0800, Linus Torvalds wrote: > > > On Wed, 14 Nov 2007, Linus Torvalds wrote: > > > > So even at 100% dirty limits, it won't let you dirty more than 1GB on the > > default 32-bit setup. > > Side note: all of these are obviously still just heuristics. If you really > *do* run on a 32-bit kernel, and you want to have the pain, I'm sure you > can just disable the dirty limits with a one-liner kernel mod. And if it's > useful enough, we can certainly expose flags like that.. Not that I expect > that much anybody else will ever care, but it's not like it's wrong to > expose the silly heuristics the kernel has to users that have very > specific loads. > > That said, I still do hope you aren't actually using HIGHMEM64G. I was > really hoping that the people who had enough moolah to buy >4GB of RAM had > long since also upgraded to a 64-bit machine ;) I'm afraid we are, which probably explains it. We have a bunch of 64 bit machines, but this particular machine is one of our somewhat more ancient IBM x235 machines. It's got stacks of fast SCSI drives and a couple of hyperthreading Xeons in it. Very nice machine in its day, and very reliable which is why we have kept them, even though at 6RU it chews through disk space. Unfortunately none of the 64 bit machines are world facing, and we're running HIGHMEM64G on a bunch of machines both for consistency value and because we only have one machine left with only 2Gb. I guess we'll be doing the one-liner kernel mod and testing that then. I'd certainly like to build a test case anyway so I'm not spending too much time rebooting that machine, it's also our outbound SMTP gateway. And I'll keep in mind finding a 64 bit capable machine for the role when I can. Thanks for the feedback on this - I'll come back with more details once we've done some testing, but this sounds likely, and I don't think DCC is going to change how it works, so we're stuck supporting it. Bron. - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
mmap dirty limits on 32 bit kernels (Was: [BUG] New Kernel Bugs)
On Wed, Nov 14, 2007 at 09:53:38PM -0800, Linus Torvalds wrote: On Wed, 14 Nov 2007, Linus Torvalds wrote: So even at 100% dirty limits, it won't let you dirty more than 1GB on the default 32-bit setup. Side note: all of these are obviously still just heuristics. If you really *do* run on a 32-bit kernel, and you want to have the pain, I'm sure you can just disable the dirty limits with a one-liner kernel mod. And if it's useful enough, we can certainly expose flags like that.. Not that I expect that much anybody else will ever care, but it's not like it's wrong to expose the silly heuristics the kernel has to users that have very specific loads. That said, I still do hope you aren't actually using HIGHMEM64G. I was really hoping that the people who had enough moolah to buy 4GB of RAM had long since also upgraded to a 64-bit machine ;) I'm afraid we are, which probably explains it. We have a bunch of 64 bit machines, but this particular machine is one of our somewhat more ancient IBM x235 machines. It's got stacks of fast SCSI drives and a couple of hyperthreading Xeons in it. Very nice machine in its day, and very reliable which is why we have kept them, even though at 6RU it chews through disk space. Unfortunately none of the 64 bit machines are world facing, and we're running HIGHMEM64G on a bunch of machines both for consistency value and because we only have one machine left with only 2Gb. I guess we'll be doing the one-liner kernel mod and testing that then. I'd certainly like to build a test case anyway so I'm not spending too much time rebooting that machine, it's also our outbound SMTP gateway. And I'll keep in mind finding a 64 bit capable machine for the role when I can. Thanks for the feedback on this - I'll come back with more details once we've done some testing, but this sounds likely, and I don't think DCC is going to change how it works, so we're stuck supporting it. Bron. - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [alsa-devel] [BUG] New Kernel Bugs
On Thu, Nov 15, 2007 at 06:59:34AM +0100, Rene Herman wrote: On 15-11-07 05:16, Bron Gondwana wrote: Totally unrelated - I sent something to the kolab mailing list a couple [ ... ] I'm sure if I had something that I considered worth informing the ALSA project of, I'd be wary of spending the same effort writing a good post knowing it may be dropped in between the by a list moderator just selecing all and bouncing them. Totally unrelated indeed so why are spouting crap? If the kohab list has a problem take it up with them but keep ALSA out of it. alsa-devel has only ever moderated out spam -- nothing else. As an outsider to the list, how do I know what your policy will be other than I've been rejected out of hand by someone else's list, so my experience is that member only lists aren't willing to listen to something I have to say unless I make the effort to sign up and have yet another folder accumulating unread messages. I don't. Well, ok - maybe I do here since I've let myself be dragged in to the debate. Oops. I get the same information from both project websites: moderated for non-members, public archives - no way of knowing that ALSA will accept me informing them of something they would be interested without committing to reading or bit-bucketing their list. The alternative is to subscribe just long enough to send something and then unsubscribe again or cold-email a member and ask them to pass a message along. Or post and hope it doesn't get rejected, not even knowing for a day or so. Bron. - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [BUG] New Kernel Bugs
On Wed, Nov 14, 2007 at 08:24:53PM -0800, Linus Torvalds wrote: > > > On Thu, 15 Nov 2007, Bron Gondwana wrote: > > > > And congratulations to him for that. We almost entirely dropped 2.6.16, > > but there's a regression some time since then that makes large MMAPed > > files a major pain (specifically the dcc database clean takes about 5 > > minutes on 2.6.16 and about 12 hours on 2.6.20 or 2.6.23 series kernels) > > > > But we keep putting off writing a small testcase that can repeat the > > issue so we can bisect it - because it's working fine with 2.6.16 on > > that machine. > > Heh. I suspect you don't even need to bisect it. > > The big difference with large mmap'ed files is that later kernels will > actually track dirty ratios for dirty mmap'ed pages. Earlier kernels never > did. > > So in older kernels, you can dirty as much memory as you want, and the > kernel will never try to write it back (well - "never" here means one of > either (a) you ask it to with msync or (b) you run out of memory, when the > kernel then totally falls down and the machine is essentially unusuable). > > So *if* the symptom seems to be that the later kernels do a lot more IO, > then try to change > > /proc/sys/vm/dirty_[background_]ratio > > which is just a percentage of memory (defaults to 5% for background and > 10% for foreground dirtying). Turn them both up a lot (say to 50 and 80 > percent respectively) and see if that makes a difference. >From our sysctl.conf: # This should help reduce flushing on Cache::FastMmap files vm.dirty_background_ratio = 50 vm.dirty_expire_centisecs = 9000 vm.dirty_ratio = 80 vm.dirty_writeback_centisecs = 3000 So we've already been running those settings for a while. They didn't help. We also gave this thing its very own dedicated ServeRAID card and associated RAID1 set of high speed SCSI drives (mainly because they were just sitting there already attached to the machine and unused, we don't love DCC that much) and it didn't help. Helped the rest of the machine now that the system drive wasn't being pegged 100% for 12 hours a day, but it didn't speed things up any. It was making some pretty random little scattered changes all through that file. Hmm.. here's what the developers said about it: First dbclean creates a new dcc_db file by copying from the old file. As it copies, it decides whether each record is worth keeping. That involves looking up the checksums in the old hash table. This is as almost afast a simple /bin/cp if the old dcc_db and dcc_db.hash files fit in RAM. The dbclean creates a new dcc_db.hash file. This starts with creating an empty new dcc_db.hash file. Then the new dcc_db and dcc_db.hash files are mapped into memory, and dbclean creates pointers to each checksum in the dcc_db file in the dcc_db.hash file. While dbclean is running, dccd unmaps everything and tries to stay out of the way. > If so, you'll be the first one to officially even notice this change, I > think. Yay for us. Thankfully it doesn't affect Cyrus's MMAP usage (read only with direct seek and write calls to change anything, then remap) or we would have suffered pretty badly! Guess we'd better get on to figuring building a simple test app. The mmap file that DCC uses is about 2Gb if that makes any difference: -rw-r--r-- 1 dcc dcc 2035138560 Nov 15 00:15 dcc_db -rw-r--r-- 1 dcc dcc 516612096 Nov 14 06:27 dcc_db.hash The machine has 6Gb of memory and should be able to fit these files fine: [EMAIL PROTECTED] hm]$ free total used free sharedbuffers cached Mem: 62323645758112 474252 0 417563002528 -/+ buffers/cache:27138283518536 Swap: 2048248 749441973304 And here's what top says about the process: 15 0 1914m 57m 41m D5 1.0 346:07.79 dccd This is on: 2.6.16.55-reiserfix-fai (one small patch to reiserfs, and built with netboot support for FAI) So yeah - we'll try to get a clearer idea of what it's doing, but the knob twiddle didn't work for us. Bron. - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [BUG] New Kernel Bugs
On Tue, Nov 13, 2007 at 10:56:01PM +0100, Christian Kujau wrote: > On Tue, 13 Nov 2007, Andrew Morton wrote: >> There are a number of process things we _could_ do. Like >> - have bugfix-only kernel releases > > Adrian Bunk does (did?) this with 2.6.16.x, although it always seemed to me > like an unrewarded one man show. AFAIK not even the big distros are begging > for bugfix-only versions, as they too want to have (sell) new features. > Mission critical systems might want to require such versions, but I guess > they're using heavily customized trees anyway. And congratulations to him for that. We almost entirely dropped 2.6.16, but there's a regression some time since then that makes large MMAPed files a major pain (specifically the dcc database clean takes about 5 minutes on 2.6.16 and about 12 hours on 2.6.20 or 2.6.23 series kernels) But we keep putting off writing a small testcase that can repeat the issue so we can bisect it - because it's working fine with 2.6.16 on that machine. Bron. - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [alsa-devel] [BUG] New Kernel Bugs
On Wed, Nov 14, 2007 at 12:46:24PM +0100, Rene Herman wrote: > On 14-11-07 11:07, David Miller wrote: > > Added Jaroslav and Takashi to the already extensive CC > >> From: Russell King <[EMAIL PROTECTED]> > >>> So, when are you creating a replacement alsa-devel mailing list on >>> vger? That's also subscribers-only. >> The operative term is "alternative" rather than "replacement". >> Perhaps this misunderstanding is what you're so upset about. >> And yes, that alsa list bugs the crap out of me too. I'm more than >> happy to provide an alternative for that one as well. > > [EMAIL PROTECTED] is not subscriber-only. Same as that arm list, > it's _moderated_ for non-subscribers and given that I and other moderators > have been doing our best to moderate quickly (I tend to stay logged in to > the moderation interface all day for example) what specifically bugged the > crap out of you? It's not something a poster needs to concern himself with. Totally unrelated - I sent something to the kolab mailing list a couple of days ago (it's moderated for non subscribers) informing them that I had found the cause of some Cyrus bugs that they had problems with in the past and providing a link to my post to the cyrus list with the patches attached. It sat in the moderation queue and then was rejected with "non subscriber post to subscription only list". Not only was the reponse a day later when I had moved on to other things, but it got me really pissed off that I had put some effort into providing a good quality post that outlined the specific issues and how they applied to their project, and had been summarily dismissed, probably without the effort being put in. There's no way for a non-subscriber to know in advance if the list they are trying to post to will do that to them, completely negating the effort put in to writing something worthwhile to inform that community. It's insular, and it sucks. So yeah, my attitude now is that the Kolab folks can go screw themselves and track down the fix on their own or wait until I've convinced upstream to accept the fixes (likely) and they have moved to the new version (unlikely for a long time, and meanwhile they're missing out on the performance increases that having a more stable skiplist library would give them) I'm sure if I had something that I considered worth informing the ALSA project of, I'd be wary of spending the same effort writing a good post knowing it may be dropped in between the by a list moderator just selecing all and bouncing them. Bron. - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [alsa-devel] [BUG] New Kernel Bugs
On Wed, Nov 14, 2007 at 12:46:24PM +0100, Rene Herman wrote: On 14-11-07 11:07, David Miller wrote: Added Jaroslav and Takashi to the already extensive CC From: Russell King [EMAIL PROTECTED] So, when are you creating a replacement alsa-devel mailing list on vger? That's also subscribers-only. The operative term is alternative rather than replacement. Perhaps this misunderstanding is what you're so upset about. And yes, that alsa list bugs the crap out of me too. I'm more than happy to provide an alternative for that one as well. [EMAIL PROTECTED] is not subscriber-only. Same as that arm list, it's _moderated_ for non-subscribers and given that I and other moderators have been doing our best to moderate quickly (I tend to stay logged in to the moderation interface all day for example) what specifically bugged the crap out of you? It's not something a poster needs to concern himself with. Totally unrelated - I sent something to the kolab mailing list a couple of days ago (it's moderated for non subscribers) informing them that I had found the cause of some Cyrus bugs that they had problems with in the past and providing a link to my post to the cyrus list with the patches attached. It sat in the moderation queue and then was rejected with non subscriber post to subscription only list. Not only was the reponse a day later when I had moved on to other things, but it got me really pissed off that I had put some effort into providing a good quality post that outlined the specific issues and how they applied to their project, and had been summarily dismissed, probably without the effort being put in. There's no way for a non-subscriber to know in advance if the list they are trying to post to will do that to them, completely negating the effort put in to writing something worthwhile to inform that community. It's insular, and it sucks. So yeah, my attitude now is that the Kolab folks can go screw themselves and track down the fix on their own or wait until I've convinced upstream to accept the fixes (likely) and they have moved to the new version (unlikely for a long time, and meanwhile they're missing out on the performance increases that having a more stable skiplist library would give them) I'm sure if I had something that I considered worth informing the ALSA project of, I'd be wary of spending the same effort writing a good post knowing it may be dropped in between the by a list moderator just selecing all and bouncing them. Bron. - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [BUG] New Kernel Bugs
On Tue, Nov 13, 2007 at 10:56:01PM +0100, Christian Kujau wrote: On Tue, 13 Nov 2007, Andrew Morton wrote: There are a number of process things we _could_ do. Like - have bugfix-only kernel releases Adrian Bunk does (did?) this with 2.6.16.x, although it always seemed to me like an unrewarded one man show. AFAIK not even the big distros are begging for bugfix-only versions, as they too want to have (sell) new features. Mission critical systems might want to require such versions, but I guess they're using heavily customized trees anyway. And congratulations to him for that. We almost entirely dropped 2.6.16, but there's a regression some time since then that makes large MMAPed files a major pain (specifically the dcc database clean takes about 5 minutes on 2.6.16 and about 12 hours on 2.6.20 or 2.6.23 series kernels) But we keep putting off writing a small testcase that can repeat the issue so we can bisect it - because it's working fine with 2.6.16 on that machine. Bron. - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [BUG] New Kernel Bugs
On Wed, Nov 14, 2007 at 08:24:53PM -0800, Linus Torvalds wrote: On Thu, 15 Nov 2007, Bron Gondwana wrote: And congratulations to him for that. We almost entirely dropped 2.6.16, but there's a regression some time since then that makes large MMAPed files a major pain (specifically the dcc database clean takes about 5 minutes on 2.6.16 and about 12 hours on 2.6.20 or 2.6.23 series kernels) But we keep putting off writing a small testcase that can repeat the issue so we can bisect it - because it's working fine with 2.6.16 on that machine. Heh. I suspect you don't even need to bisect it. The big difference with large mmap'ed files is that later kernels will actually track dirty ratios for dirty mmap'ed pages. Earlier kernels never did. So in older kernels, you can dirty as much memory as you want, and the kernel will never try to write it back (well - never here means one of either (a) you ask it to with msync or (b) you run out of memory, when the kernel then totally falls down and the machine is essentially unusuable). So *if* the symptom seems to be that the later kernels do a lot more IO, then try to change /proc/sys/vm/dirty_[background_]ratio which is just a percentage of memory (defaults to 5% for background and 10% for foreground dirtying). Turn them both up a lot (say to 50 and 80 percent respectively) and see if that makes a difference. From our sysctl.conf: # This should help reduce flushing on Cache::FastMmap files vm.dirty_background_ratio = 50 vm.dirty_expire_centisecs = 9000 vm.dirty_ratio = 80 vm.dirty_writeback_centisecs = 3000 So we've already been running those settings for a while. They didn't help. We also gave this thing its very own dedicated ServeRAID card and associated RAID1 set of high speed SCSI drives (mainly because they were just sitting there already attached to the machine and unused, we don't love DCC that much) and it didn't help. Helped the rest of the machine now that the system drive wasn't being pegged 100% for 12 hours a day, but it didn't speed things up any. It was making some pretty random little scattered changes all through that file. Hmm.. here's what the developers said about it: First dbclean creates a new dcc_db file by copying from the old file. As it copies, it decides whether each record is worth keeping. That involves looking up the checksums in the old hash table. This is as almost afast a simple /bin/cp if the old dcc_db and dcc_db.hash files fit in RAM. The dbclean creates a new dcc_db.hash file. This starts with creating an empty new dcc_db.hash file. Then the new dcc_db and dcc_db.hash files are mapped into memory, and dbclean creates pointers to each checksum in the dcc_db file in the dcc_db.hash file. While dbclean is running, dccd unmaps everything and tries to stay out of the way. If so, you'll be the first one to officially even notice this change, I think. Yay for us. Thankfully it doesn't affect Cyrus's MMAP usage (read only with direct seek and write calls to change anything, then remap) or we would have suffered pretty badly! Guess we'd better get on to figuring building a simple test app. The mmap file that DCC uses is about 2Gb if that makes any difference: -rw-r--r-- 1 dcc dcc 2035138560 Nov 15 00:15 dcc_db -rw-r--r-- 1 dcc dcc 516612096 Nov 14 06:27 dcc_db.hash The machine has 6Gb of memory and should be able to fit these files fine: [EMAIL PROTECTED] hm]$ free total used free sharedbuffers cached Mem: 62323645758112 474252 0 417563002528 -/+ buffers/cache:27138283518536 Swap: 2048248 749441973304 And here's what top says about the process: 15 0 1914m 57m 41m D5 1.0 346:07.79 dccd This is on: 2.6.16.55-reiserfix-fai (one small patch to reiserfs, and built with netboot support for FAI) So yeah - we'll try to get a clearer idea of what it's doing, but the knob twiddle didn't work for us. Bron. - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: Ideas on column length in kernel "problem"?
On Wed, Aug 22, 2007 at 11:54:41PM -0400, Scott Thompson wrote: > Many free (and not-free) mail clients wordwrap. Hushmail wraps at > 68 (verified), Yahoo has options to wrap at a max of 99, and Gmail > was somewhere around 85-90 as I recall. Not sure on other free / > inexpensive clients. I work for them FastMail (http://fastmail.fm/) have a tickbox in the web interface so you can turn off line-wrapping if you need to. Otherwise, as other people have said, use direct SMTP (we don't allow it for non-paying accounts, but do for all levels of paying user) and use a sane local client (in my case mutt+offlineimap) Bron. - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: Ideas on column length in kernel problem?
On Wed, Aug 22, 2007 at 11:54:41PM -0400, Scott Thompson wrote: Many free (and not-free) mail clients wordwrap. Hushmail wraps at 68 (verified), Yahoo has options to wrap at a max of 99, and Gmail was somewhere around 85-90 as I recall. Not sure on other free / inexpensive clients. disclaimerI work for them/disclaimer FastMail (http://fastmail.fm/) have a tickbox in the web interface so you can turn off line-wrapping if you need to. Otherwise, as other people have said, use direct SMTP (we don't allow it for non-paying accounts, but do for all levels of paying user) and use a sane local client (in my case mutt+offlineimap) Bron. - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: how about mutual compatibility between Linux's GPLv2 and GPLv3?
On Thu, Jun 21, 2007 at 06:39:07AM -0300, Alexandre Oliva wrote: > If GPLv3 were to have a clause that permitted combination/linking with > code under GPLv2, this wouldn't be enough for GPLv3 projects to use > Linux code, and it wouldn't be enough for Linux code to use GPLv3 > projects. That's because GPLv2 would still demand all code to be > licensed under GPLv2, and GPLv3 wouldn't permit this. > > However, if GPLv3 had a permission to combine/link with code under > GPLv2, *and* Linux (and any other projects interested in mutual > compatibility) introduced an additional permission to combine/link > with code under GPLv3 (or even GPLv3+, constrained by some condition > if you will), then: My god, you really have come totally unhinged in your attempt to reconcile two incompatible ideas. Ouch. The reason the GPLv2 ecosystem is so strong is that you can take any code under GPLv2 and combine it with any other code under GPLv2 and the result is GPLv2. All you have to check is that the original code is either GPLv2 or a licence that allows conversion to GPLv2, that's it. None of this "Projects" nonsense. Who says what code is a "project" and if it has any special relationships with other "projects" that allow code sharing above and beyond their standard licence terms. Suddenly using other GPLv2 code becomes fraught with "which path did I obtain this licence down" games and either a big fat pile of paperwork or plain not being able to be clear about the licencing of of the code. It's not about projects, it's about the code. Gah. You're not going to make a happy, happy merging code sharing world by fragmenting the licence landscape even more. Bron. - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: how about mutual compatibility between Linux's GPLv2 and GPLv3?
On Thu, Jun 21, 2007 at 08:23:57PM -0300, Alexandre Oliva wrote: > On Jun 21, 2007, "David Schwartz" <[EMAIL PROTECTED]> wrote: > > >> > Wouldn't that defeat the entire purpose of the GPLv3? Couldn't > >> > I take any > >> > GPLv3 program, combine it with a few lines of Linux code, and > >> > Tivoize the > >> > result? > > >> No. This is not permission to relicense. This is permission to > >> combine. Each author still gets to enforce the terms of her own code. > > > This makes no sense. We are not talking mere aggregation here, we are > > talking developmental convergence. If I wrote some code that was in the > > Linux kernel, how can I enforce the terms of my code (guaranteed write to > > Tivoize) in the derivative work that it becomes mixed with? > > In just the same way you'd enforce it today: with help from a lawyer > who understands these issues that you clearly don't understand. Great, so for ever and ever afterwards the code would have to keep a clear separation between the bits that are under different licences and make sure that no re-factor ever blurred the lines between them enough that you had trouble telling which was which. And don't even get me started about some poor innocent end-user who just wants to use some code from your mutant frankenstein "project" in her pong game (or was it tetris, I lose track of what you're supposed to be playing on your hacked TiVo while it's not allowed to connect to their network any more) and understand what licence the final work is under. Ouch. Bron. - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: TUX2 filesystem
On Thu, Jun 21, 2007 at 03:26:15PM -0700, Zach Brown wrote: > > Second, Oracle is now working on Btrfs (if ever a FS needed a better > > name... is that pronounced ButterFS?). > > (In our silliest moments, yes. Absolutely.) I'm sure when the PHBen are around it's "Better FS". It's all a Free(software)Mason conspiracy, I tell you. Bron. - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: TUX2 filesystem
On Thu, Jun 21, 2007 at 03:26:15PM -0700, Zach Brown wrote: Second, Oracle is now working on Btrfs (if ever a FS needed a better name... is that pronounced ButterFS?). (In our silliest moments, yes. Absolutely.) I'm sure when the PHBen are around it's Better FS. It's all a Free(software)Mason conspiracy, I tell you. Bron. - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: how about mutual compatibility between Linux's GPLv2 and GPLv3?
On Thu, Jun 21, 2007 at 06:39:07AM -0300, Alexandre Oliva wrote: If GPLv3 were to have a clause that permitted combination/linking with code under GPLv2, this wouldn't be enough for GPLv3 projects to use Linux code, and it wouldn't be enough for Linux code to use GPLv3 projects. That's because GPLv2 would still demand all code to be licensed under GPLv2, and GPLv3 wouldn't permit this. However, if GPLv3 had a permission to combine/link with code under GPLv2, *and* Linux (and any other projects interested in mutual compatibility) introduced an additional permission to combine/link with code under GPLv3 (or even GPLv3+, constrained by some condition if you will), then: My god, you really have come totally unhinged in your attempt to reconcile two incompatible ideas. Ouch. The reason the GPLv2 ecosystem is so strong is that you can take any code under GPLv2 and combine it with any other code under GPLv2 and the result is GPLv2. All you have to check is that the original code is either GPLv2 or a licence that allows conversion to GPLv2, that's it. None of this Projects nonsense. Who says what code is a project and if it has any special relationships with other projects that allow code sharing above and beyond their standard licence terms. Suddenly using other GPLv2 code becomes fraught with which path did I obtain this licence down games and either a big fat pile of paperwork or plain not being able to be clear about the licencing of of the code. It's not about projects, it's about the code. Gah. You're not going to make a happy, happy merging code sharing world by fragmenting the licence landscape even more. Bron. - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: how about mutual compatibility between Linux's GPLv2 and GPLv3?
On Thu, Jun 21, 2007 at 08:23:57PM -0300, Alexandre Oliva wrote: On Jun 21, 2007, David Schwartz [EMAIL PROTECTED] wrote: Wouldn't that defeat the entire purpose of the GPLv3? Couldn't I take any GPLv3 program, combine it with a few lines of Linux code, and Tivoize the result? No. This is not permission to relicense. This is permission to combine. Each author still gets to enforce the terms of her own code. This makes no sense. We are not talking mere aggregation here, we are talking developmental convergence. If I wrote some code that was in the Linux kernel, how can I enforce the terms of my code (guaranteed write to Tivoize) in the derivative work that it becomes mixed with? In just the same way you'd enforce it today: with help from a lawyer who understands these issues that you clearly don't understand. Great, so for ever and ever afterwards the code would have to keep a clear separation between the bits that are under different licences and make sure that no re-factor ever blurred the lines between them enough that you had trouble telling which was which. And don't even get me started about some poor innocent end-user who just wants to use some code from your mutant frankenstein project in her pong game (or was it tetris, I lose track of what you're supposed to be playing on your hacked TiVo while it's not allowed to connect to their network any more) and understand what licence the final work is under. Ouch. Bron. - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: Versioning file system
On Mon, Jun 18, 2007 at 11:10:42PM -0400, Kyle Moffett wrote: > On Jun 18, 2007, at 13:56:05, Bryan Henderson wrote: >>> The question remains is where to implement versioning: directly in >>> individual filesystems or in the vfs code so all filesystems can use it? >> >> Or not in the kernel at all. I've been doing versioning of the types I >> described for years with user space code and I don't remember feeling that >> I compromised in order not to involve the kernel. > > What I think would be particularly interesting in this domain is something > similar in concept to GIT, except in a file-system: I've written a couple of user-space things very much like this - one being a purely database (blobs in database, yeah I know) system for managing medical data, where signatures and auditability were the most important part of the system. Performance really wasn't a consideration. The other one is my current job, FastMail - we have a virtual filesystem which uses files stored by sha1 on ordainary filesystems for data storage and a database for metadata (filename to sha1 mappings, mtime, mimetype, directory structure, etc). Multiple machine distribution is handled by a daemon on each machine which can be asked to make sure the file gets sent out to every machine that matches the prefix and will only return success once it's written to at least one other machine. Database replication is a different beast. It can work, but there's one big pain at the file level: no mmap. If you don't want to support mmap it can work reasonably happily, though you may want to keep your sha1 (or other digest) state as well as the final digest so you can cheaply calculate the digest for a small append without walking the entire file. You may also want to keep state checkpoints every so often along a big file so that truncates don't cost too much to recalculate. Luckily in a userspace VFS that's only accessed via FTP and DAV we can support a limited set of operations (basically create, append, read, delete) You don't get that luxury for a general purpose filesystem, and that's the problem. There will always be particular usage patterns (especially something that mmaps or seeks and touches all over the place like a loopback mounted filesystem or a database file) that just dodn't work for file-level sha1s. It does have some lovely properties though. I'd enjoy working in an envionment that didn't look much like POSIX but had the strong guarantees and auditability that addressing by sha1 buys you. Bron. - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: Versioning file system
On Mon, Jun 18, 2007 at 11:10:42PM -0400, Kyle Moffett wrote: On Jun 18, 2007, at 13:56:05, Bryan Henderson wrote: The question remains is where to implement versioning: directly in individual filesystems or in the vfs code so all filesystems can use it? Or not in the kernel at all. I've been doing versioning of the types I described for years with user space code and I don't remember feeling that I compromised in order not to involve the kernel. What I think would be particularly interesting in this domain is something similar in concept to GIT, except in a file-system: I've written a couple of user-space things very much like this - one being a purely database (blobs in database, yeah I know) system for managing medical data, where signatures and auditability were the most important part of the system. Performance really wasn't a consideration. The other one is my current job, FastMail - we have a virtual filesystem which uses files stored by sha1 on ordainary filesystems for data storage and a database for metadata (filename to sha1 mappings, mtime, mimetype, directory structure, etc). Multiple machine distribution is handled by a daemon on each machine which can be asked to make sure the file gets sent out to every machine that matches the prefix and will only return success once it's written to at least one other machine. Database replication is a different beast. It can work, but there's one big pain at the file level: no mmap. If you don't want to support mmap it can work reasonably happily, though you may want to keep your sha1 (or other digest) state as well as the final digest so you can cheaply calculate the digest for a small append without walking the entire file. You may also want to keep state checkpoints every so often along a big file so that truncates don't cost too much to recalculate. Luckily in a userspace VFS that's only accessed via FTP and DAV we can support a limited set of operations (basically create, append, read, delete) You don't get that luxury for a general purpose filesystem, and that's the problem. There will always be particular usage patterns (especially something that mmaps or seeks and touches all over the place like a loopback mounted filesystem or a database file) that just dodn't work for file-level sha1s. It does have some lovely properties though. I'd enjoy working in an envionment that didn't look much like POSIX but had the strong guarantees and auditability that addressing by sha1 buys you. Bron. - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: Dual-Licensing Linux Kernel with GPL V2 and GPL V3
On Mon, Jun 18, 2007 at 01:03:40PM -0700, David Schwartz wrote: > So you're arguing two sides of no argument at all. Yeah, pretty much. I take back my arguments in the previous couple of my posts up this thread. They don't actually hold together! Sorry for wasting your time correct me. Bron. - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: Versioning file system
On Mon, Jun 18, 2007 at 11:32:38AM -0400, Chris Mason wrote: > On Mon, Jun 18, 2007 at 03:45:24AM -0600, Andreas Dilger wrote: > > Too bad everyone is spending time on 10 similar-but-slightly-different > > filesystems. This will likely end up with a bunch of filesystems that > > implement some easy subset of features, but will not get polished for > > users or have a full set of features implemented (e.g. ACL, quota, fsck, > > etc). While I don't think there is a single answer to every question, > > it does seem that the number of filesystem projects has climbed lately. > > > > Maybe there should be a BOF at OLS to merge these filesystem projects > > (btrfs, chunkfs, tilefs, logfs, etc) into a single project with multiple > > people working on getting it solid, scalable (parallel readers/writers on > > lots of CPUs), robust (checksums, failure localization), recoverable, etc. > > I thought Val's FS summits were designed to get developers to collaborate, > > but it seems everyone has gone back to their corners to work on their own > > filesystem? > > Unfortunately, I can't do OLS this year, but anyone who wants to talk on > these things can drop me a line and we can setup phone calls or whatever > for planning. Adding polish to any FS is not a one man show, and so I know > I'll need to get more people on board to really finish btrfs off. > > One of my long term goals for btrfs is to figure out the features and > layout people are most interested in for filesystems that don't have to > be ext* backwards compatible. I've got a pretty good start, but I'm > sure parts of it will change if I can get a big enough developer base. I have no filesystem programming experience, but I am certainly interested, and I'm spending some time reading through the code that you've written so far. Oh, and running it - though I'm probably going to want to fiddle with some smaller filesystems than my entire Maildir set if I want to make any sense of the structure dumps! That and of course if I get involved in development I can be sure that my favourite workload (big Cyrus installs) is well optimized for! Actually, my biggest interest is decent unlink performance, in particular when you are unlinking multiple items in a directory or even the entire directory plus everything in it. I find that to be an incredibly slow and IO hurting operation. We run cyr_expire (the process in Cyrus that actually deletes expunged messages) once per week, and only one process at a time on a machine which might have 20 otherwise busy instances of Cyrus running - because the IO hit on those data partitions is massive. Load average more than doubles and the log entries for commands which took longer than a second to return increase massively. And this is on a Sunday when there's barely any use compared to a weekday. So yeah, my main interest is making unlink (especially multiple unlinks from the same directory) into a less extreme experience. Bron. - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: Dual-Licensing Linux Kernel with GPL V2 and GPL V3
On Sun, Jun 17, 2007 at 07:45:38PM -0500, Chris Adams wrote: > Once upon a time, Bron Gondwana <[EMAIL PROTECTED]> said: > > To be fair here, this could also be accomplished by having to flip a > > physical switch on the router, especially if you did something funky > > like: > > > > [---] push this button for a 5 minute access pass to upload new > > software through physical cable port 1. > > > > More complex, but not unreasonable. > > Well, there is no restriction on putting files on the routing engine's > storage devices (flash and hard drive); it is running OpenSSH, so > scp/sftp work fine, and you can drop to a shell easily. The restriction > is that the kernel won't run unsigned binaries. > > Also, flipping physical switches is pretty much an unreasonable > expectation for core router operation. These are often in other > locations, sometimes other telcos' central offices (where you have to > pay to have "remote hands" do something and then hope they don't screw > it up). You can easily go the entire life of a device where the primary > operators never physically see the device. Every server I run is like that, but if something is important enough I can remote control a robot over to push the button for me (actually, I think they implement this under the hood by having a human read the ticket I submit and go push the button for me manually, but that could be my imagination. So long as the button gets pushed the black box is functioning) Bron. - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: Dual-Licensing Linux Kernel with GPL V2 and GPL V3
On Sun, Jun 17, 2007 at 07:45:38PM -0500, Chris Adams wrote: Once upon a time, Bron Gondwana [EMAIL PROTECTED] said: To be fair here, this could also be accomplished by having to flip a physical switch on the router, especially if you did something funky like: [---] push this button for a 5 minute access pass to upload new software through physical cable port 1. More complex, but not unreasonable. Well, there is no restriction on putting files on the routing engine's storage devices (flash and hard drive); it is running OpenSSH, so scp/sftp work fine, and you can drop to a shell easily. The restriction is that the kernel won't run unsigned binaries. Also, flipping physical switches is pretty much an unreasonable expectation for core router operation. These are often in other locations, sometimes other telcos' central offices (where you have to pay to have remote hands do something and then hope they don't screw it up). You can easily go the entire life of a device where the primary operators never physically see the device. Every server I run is like that, but if something is important enough I can remote control a robot over to push the button for me (actually, I think they implement this under the hood by having a human read the ticket I submit and go push the button for me manually, but that could be my imagination. So long as the button gets pushed the black box is functioning) Bron. - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: Versioning file system
On Mon, Jun 18, 2007 at 11:32:38AM -0400, Chris Mason wrote: On Mon, Jun 18, 2007 at 03:45:24AM -0600, Andreas Dilger wrote: Too bad everyone is spending time on 10 similar-but-slightly-different filesystems. This will likely end up with a bunch of filesystems that implement some easy subset of features, but will not get polished for users or have a full set of features implemented (e.g. ACL, quota, fsck, etc). While I don't think there is a single answer to every question, it does seem that the number of filesystem projects has climbed lately. Maybe there should be a BOF at OLS to merge these filesystem projects (btrfs, chunkfs, tilefs, logfs, etc) into a single project with multiple people working on getting it solid, scalable (parallel readers/writers on lots of CPUs), robust (checksums, failure localization), recoverable, etc. I thought Val's FS summits were designed to get developers to collaborate, but it seems everyone has gone back to their corners to work on their own filesystem? Unfortunately, I can't do OLS this year, but anyone who wants to talk on these things can drop me a line and we can setup phone calls or whatever for planning. Adding polish to any FS is not a one man show, and so I know I'll need to get more people on board to really finish btrfs off. One of my long term goals for btrfs is to figure out the features and layout people are most interested in for filesystems that don't have to be ext* backwards compatible. I've got a pretty good start, but I'm sure parts of it will change if I can get a big enough developer base. I have no filesystem programming experience, but I am certainly interested, and I'm spending some time reading through the code that you've written so far. Oh, and running it - though I'm probably going to want to fiddle with some smaller filesystems than my entire Maildir set if I want to make any sense of the structure dumps! That and of course if I get involved in development I can be sure that my favourite workload (big Cyrus installs) is well optimized for! Actually, my biggest interest is decent unlink performance, in particular when you are unlinking multiple items in a directory or even the entire directory plus everything in it. I find that to be an incredibly slow and IO hurting operation. We run cyr_expire (the process in Cyrus that actually deletes expunged messages) once per week, and only one process at a time on a machine which might have 20 otherwise busy instances of Cyrus running - because the IO hit on those data partitions is massive. Load average more than doubles and the log entries for commands which took longer than a second to return increase massively. And this is on a Sunday when there's barely any use compared to a weekday. So yeah, my main interest is making unlink (especially multiple unlinks from the same directory) into a less extreme experience. Bron. - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: Dual-Licensing Linux Kernel with GPL V2 and GPL V3
On Mon, Jun 18, 2007 at 01:03:40PM -0700, David Schwartz wrote: So you're arguing two sides of no argument at all. Yeah, pretty much. I take back my arguments in the previous couple of my posts up this thread. They don't actually hold together! Sorry for wasting your time correct me. Bron. - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: Dual-Licensing Linux Kernel with GPL V2 and GPL V3
On Sun, Jun 17, 2007 at 04:58:40PM -0500, Chris Adams wrote: > Once upon a time, Jesper Juhl <[EMAIL PROTECTED]> said: > >Let's say I'm the owner of a company selling some device that uses a > >GPLv2 OS and some GPLv2 applications to do the job. Let's say that for > >some reason I don't want the end users of my device to tinker with the > >software inside my device. Obviously I release the source for any > >modifications I may have made, but I use the hardware to prevent users > >from installing modified versions on the device (basically I TiVO'ize > >the device). > > BTW: Another reason a vendor might lock down the device is for security. > For example, Juniper routers (which now run a significant portion of the > "core" of the Internet) run FreeBSD on the routing engine. They include > several GNU software utilities (for example gawk, diff, and gdb). > Starting with JUNOS 7.6 (IIRC), end-users can no longer build and run > their own binaries on the routing engine. This means that the GPLv2 > code cannot be modified in-place (similar to TiVo altough done using > different means). > > The reason is that if there ever is a security hole in the routing > engine software (FreeBSD kernel, OpenSSH, etc.), it would be a really > bad thing if crackers could load arbitrary software (rootkits, spam > software, etc.) directly on Internet core routers. If you think spam > zombies on cable modems or DSL are bad, imagine them on 100 megabit > links! To be fair here, this could also be accomplished by having to flip a physical switch on the router, especially if you did something funky like: [---] push this button for a 5 minute access pass to upload new software through physical cable port 1. More complex, but not unreasonable. Bron. - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: Dual-Licensing Linux Kernel with GPL V2 and GPL V3
On Sat, Jun 16, 2007 at 05:58:11PM -0700, Linus Torvalds wrote: > > > On Sun, 17 Jun 2007, Bron Gondwana wrote: > > > > No, I'm arguing that it's not "mere aggregation" - the kernel is useless > > on that machine unless the BIOS is present or replaced with something > > else with equivalent functionality. > > That's *not* a valid argument! > > I know, I know, it's a common one, but it is *nonsense*. Further to my other response on this (yeah, I know, I should think first). Where the BIOS author and 'work aggregator' are different organisation with no shady backroom links (other than the usual industry cabal(TINC) of course) then it's clear. When they are the same organisation then the derivedness state is a lot less clear and more "discoverable", leading to a higher risk of ambush by litigation. This isn't specific to any particular licence, but it's something that the "intent" theory of hardware limitations being a GPL3 violation makes extra dangerous, because that clause can be used as a hook to drag a claim through summary judgement (IMHO, IANAL, etc) Bron ( mostly arguing the same things that you are Linus, I think, but I didn't clarify that I was writing from a devil's advocate position in an alternative reality where Tivo was illegal purely due to "intent" ) - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: Dual-Licensing Linux Kernel with GPL V2 and GPL V3
On Sun, Jun 17, 2007 at 02:38:43AM -0300, Alexandre Oliva wrote: > On Jun 17, 2007, Daniel Hazelton <[EMAIL PROTECTED]> wrote: > > > Ah, but giving the user half the key doesn't mean they still don't have > > access > > to the entire key. QED: Giving people half the key won't cut it under the > > GPLv3 (dd4) > > I meant really giving, rather than giving a copy, or giving the > original and keeping a copy. > > You could make it require a pair of signatures, one from the vendor, > that the vendor keeps, one from the user, that the vendor never sees, > too. Like some bank PINs, it gets generated, used to generate some > hash (the signature for the initial installation), printed in an > envelope for you and stored in the package along with the machine. Or > something like that. Wow, and I thought losing a microsoft "certificate of authenticity" and associated key was a pain. Ouch. Talk about your paper "dongle". - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: Dual-Licensing Linux Kernel with GPL V2 and GPL V3
On Sat, Jun 16, 2007 at 05:58:11PM -0700, Linus Torvalds wrote: > On Sun, 17 Jun 2007, Bron Gondwana wrote: > > > > No, I'm arguing that it's not "mere aggregation" - the kernel is useless > > on that machine unless the BIOS is present or replaced with something > > else with equivalent functionality. > > That's *not* a valid argument! > > But it's all still "mere aggregation", because they are not related to > each other in the sense of being derived works! > > So "mere aggregation" is not about intimacy. OF COURSE high-tech products > depend intimately on each other. The Linux kernel cannot boot on a PC > without a BIOS or something equivalent. You cannot run your graphical > environment without a kernel, an X server, the CPU, the memory, the > display, the BIOS, the power company (or an equivalent hand-crank) etc etc > etc, and these things are all very much dependent on each other to make a > "usable system", that has absolutely _zero_ relevance to whether they are > "mere aggregation" or not. Ok, can I please rewrite my argument to: "The hardware manufacturer has built a custom BIOS and also written Linux kernel support for said BIOS. They have released the kernel drivers under GPL as required, but have not released the code to the BIOS, instead just releasing the interface documentation. The BIOS didn't exist before, and as they only intend to run Linux on the device, the BIOS design was heavily influenced around working well with Linux." Actually, we don't know that last bit, maybe they created the BIOS in a total vacuum and then wrote the Linux kernel driver later. Maybe not. Anyway, I think I've wound up arguing two sides of the same argument, oops. Bron. - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: Dual-Licensing Linux Kernel with GPL V2 and GPL V3
On Sat, Jun 16, 2007 at 05:58:11PM -0700, Linus Torvalds wrote: On Sun, 17 Jun 2007, Bron Gondwana wrote: No, I'm arguing that it's not mere aggregation - the kernel is useless on that machine unless the BIOS is present or replaced with something else with equivalent functionality. That's *not* a valid argument! But it's all still mere aggregation, because they are not related to each other in the sense of being derived works! So mere aggregation is not about intimacy. OF COURSE high-tech products depend intimately on each other. The Linux kernel cannot boot on a PC without a BIOS or something equivalent. You cannot run your graphical environment without a kernel, an X server, the CPU, the memory, the display, the BIOS, the power company (or an equivalent hand-crank) etc etc etc, and these things are all very much dependent on each other to make a usable system, that has absolutely _zero_ relevance to whether they are mere aggregation or not. Ok, can I please rewrite my argument to: The hardware manufacturer has built a custom BIOS and also written Linux kernel support for said BIOS. They have released the kernel drivers under GPL as required, but have not released the code to the BIOS, instead just releasing the interface documentation. The BIOS didn't exist before, and as they only intend to run Linux on the device, the BIOS design was heavily influenced around working well with Linux. Actually, we don't know that last bit, maybe they created the BIOS in a total vacuum and then wrote the Linux kernel driver later. Maybe not. Anyway, I think I've wound up arguing two sides of the same argument, oops. Bron. - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: Dual-Licensing Linux Kernel with GPL V2 and GPL V3
On Sun, Jun 17, 2007 at 02:38:43AM -0300, Alexandre Oliva wrote: On Jun 17, 2007, Daniel Hazelton [EMAIL PROTECTED] wrote: Ah, but giving the user half the key doesn't mean they still don't have access to the entire key. QED: Giving people half the key won't cut it under the GPLv3 (dd4) I meant really giving, rather than giving a copy, or giving the original and keeping a copy. You could make it require a pair of signatures, one from the vendor, that the vendor keeps, one from the user, that the vendor never sees, too. Like some bank PINs, it gets generated, used to generate some hash (the signature for the initial installation), printed in an envelope for you and stored in the package along with the machine. Or something like that. Wow, and I thought losing a microsoft certificate of authenticity and associated key was a pain. Ouch. Talk about your paper dongle. - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: Dual-Licensing Linux Kernel with GPL V2 and GPL V3
On Sat, Jun 16, 2007 at 05:58:11PM -0700, Linus Torvalds wrote: On Sun, 17 Jun 2007, Bron Gondwana wrote: No, I'm arguing that it's not mere aggregation - the kernel is useless on that machine unless the BIOS is present or replaced with something else with equivalent functionality. That's *not* a valid argument! I know, I know, it's a common one, but it is *nonsense*. Further to my other response on this (yeah, I know, I should think first). Where the BIOS author and 'work aggregator' are different organisation with no shady backroom links (other than the usual industry cabal(TINC) of course) then it's clear. When they are the same organisation then the derivedness state is a lot less clear and more discoverable, leading to a higher risk of ambush by litigation. This isn't specific to any particular licence, but it's something that the intent theory of hardware limitations being a GPL3 violation makes extra dangerous, because that clause can be used as a hook to drag a claim through summary judgement (IMHO, IANAL, etc) Bron ( mostly arguing the same things that you are Linus, I think, but I didn't clarify that I was writing from a devil's advocate position in an alternative reality where Tivo was illegal purely due to intent ) - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: Dual-Licensing Linux Kernel with GPL V2 and GPL V3
On Sun, Jun 17, 2007 at 04:58:40PM -0500, Chris Adams wrote: Once upon a time, Jesper Juhl [EMAIL PROTECTED] said: Let's say I'm the owner of a company selling some device that uses a GPLv2 OS and some GPLv2 applications to do the job. Let's say that for some reason I don't want the end users of my device to tinker with the software inside my device. Obviously I release the source for any modifications I may have made, but I use the hardware to prevent users from installing modified versions on the device (basically I TiVO'ize the device). BTW: Another reason a vendor might lock down the device is for security. For example, Juniper routers (which now run a significant portion of the core of the Internet) run FreeBSD on the routing engine. They include several GNU software utilities (for example gawk, diff, and gdb). Starting with JUNOS 7.6 (IIRC), end-users can no longer build and run their own binaries on the routing engine. This means that the GPLv2 code cannot be modified in-place (similar to TiVo altough done using different means). The reason is that if there ever is a security hole in the routing engine software (FreeBSD kernel, OpenSSH, etc.), it would be a really bad thing if crackers could load arbitrary software (rootkits, spam software, etc.) directly on Internet core routers. If you think spam zombies on cable modems or DSL are bad, imagine them on 100 megabit links! To be fair here, this could also be accomplished by having to flip a physical switch on the router, especially if you did something funky like: [---] push this button for a 5 minute access pass to upload new software through physical cable port 1. More complex, but not unreasonable. Bron. - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: Dual-Licensing Linux Kernel with GPL V2 and GPL V3
[note: I'm writting this while offline and likely to remain so for the next 8 hours or so, so I'll probably miss a bunch of other replies] On Sat, Jun 16, 2007 at 02:14:29PM -0300, Alexandre Oliva wrote: > On Jun 16, 2007, Bron Gondwana <[EMAIL PROTECTED]> wrote: > > > On Sat, Jun 16, 2007 at 05:22:21AM -0300, Alexandre Oliva wrote: > >> On Jun 15, 2007, Bron Gondwana <[EMAIL PROTECTED]> wrote: > >> > >> > because it could easily be argued that they linked the BIOS with the > >> > Linux kernel > >> > >> How so? > > > Er, they installed it in the same piece of equipment, and the kernel > > couldn't function without it in that work. > > I see what you're getting at. You're thinking of a license that > doesn't respect the idea of "mere aggregation", right? No, I'm arguing that it's not "mere aggregation" - the kernel is useless on that machine unless the BIOS is present or replaced with something else with equivalent functionality. I suspect any decent lawyer could make the theory that this made the kernel as compiled on to that machine with specific chipset support selected for that hardware into a "derived work" of the BIOS - especially if the vendor had contributed GPLed code for drivers which interact with their hardware into said kernel. In fact, particularly if the hardware vendor has also contributed GPL code that interacts on one side of the software/(firmware, hardware) boundard which worked around bugs in said firmware/hardware which they also had the ability to change. The two really are a combined work of which only one part is GPLed. Ringing any binary kernel module video card driver bells yet? It's really the same thing from the opposite direction - the only criteria is where you fit in the pecking order - hardware manufacturers work around Windows bugs, Linux kernel drivers work around hardware bugs - it's all about who has more to lose if they aren't compatible. > For starters, this wouldn't evidently not qualify as an Open Source > license, and I'm pretty sure it wouldn't qualify as a Free Software > license either. Strawman licence? > > By using GPLix as part of their boot process along with their > > non-GPL BIOS, they're subverting the freedoms that the user should > > have in being able to control the entire boot process. > > You're pushing the "freedom to change" too far. Sure, I'd like to be > able to do that, and I prefer hardware that lets me do it, but it's > not like this BIOS in the scenario you described is being used as a > means to stop me from modifying the GPLed software. Well, yeah - except this is the direction GPL3 takes us, and it's a theory that GPL3 makes more likely to fly in court than GPL2 does - meaning that hardware vendor lawyers lie awake at night worrying about stuff (I'd hate to be a good lawyer - I'd never get any sleep!) > I have never said that including a GPLed piece of software should > grant users the right to modify anything whatsoever in the system, or > grant them control over the entire system. Others have, but it's not > true, it just shows how much mis-information is floating around. No, but your interactions with Linus (lazy bums 'r' us) have shown that the logical result of what you do want includes this. It's a lot harder to objectively judge one of these than the other: a) have they provided the source code to this binary to anyone who asks. b) have any of the limitations of this piece of hardware been created with the intent of making it more difficult for J. Random Enduser to build modified binaries from said source and have them function correctly. (b) has much more scope for shenanigans by bad apples on the copyright owner side - and don't pretend that only the hardware vendors are bad guys - it takes all sorts and the idea of a licence is to protect both parties. > All the GPL stands for is to defend the freedom of the users over the > particular program it applies to. You can't impose further > restrictions on the user's ability to modify what *that* software > does. Except where they run into limitations of the platform itself, or just plain bugs. Oops. The lawyers will have a field day discovering intent every time J. Random's kernel doesn't do what he wants after he fiddles the code a bit. > If you wanted to change something else, but this something else is not > covered by the license, and is not being used to contradict the terms > of the license, well, too bad, you lose. > > > b) deny themselves the ability to every offer a patch to said BIOS if > >bugs are found > > > Point (b) is also exactly on topic for the discussion of enforcing > > legal safety obligations in hardware on a peripheral rather than the > > software dr
Re: Dual-Licensing Linux Kernel with GPL V2 and GPL V3
On Sat, Jun 16, 2007 at 05:22:21AM -0300, Alexandre Oliva wrote: > On Jun 15, 2007, Bron Gondwana <[EMAIL PROTECTED]> wrote: > > > because it could easily be argued that they linked the BIOS with the > > Linux kernel > > How so? (I'm going to refer to Linux as GPLix from here on since this argument is more general than a specific GPLed operating system) Er, they installed it in the same piece of equipment, and the kernel couldn't function without it in that work. What's more 'linked' than that. It's a vital part of the boot process on that piece of hardware in exactly the same way that the public-key check is a vital part of the boot process. If your printer^wPC isn't doing what you want and you know how to change it to do what you want but it needs a BIOS patch. Guess what, you can't do it - your vendor can. By using GPLix as part of their boot process along with their non-GPL BIOS, they're subverting the freedoms that the user should have in being able to control the entire boot process. Right? Or are you unclear about the fact that there's a big grey area cutting through this part of usage, and Linus sat down pretty clearly on one side of it while you're arguing that the goalposts should be "where I feel that my rights to make changes are being infringed". While the vendor reserves the ability to change components of the system (post sale, i.e. a BIOS flash update) and doesn't hand those same rights on to you, they have partially Tivoised (hoover, kleenex, you've got nothing on these guys for having your name associated with a concept) the hardware. By logical extention of your arguments over the past few days, this denies them the ability to use any GPLed software in 'the spirit of the licence' anyware on the machine because they are denying you rights regarding the instance of the product they shipped to you that they are retaining for themselves. The very freedoms you so vocally claim. Now, the position I'm seeing here is that the above behaviour (every single hardware manufacturer that has ever shipped a machine with pre-installed Linux) violates the spirit of the GPL by the "retaining exclusive freedoms to modify shipped product" rule, and hence their BIOS is in the doghouse unless they either: a) offer full source code access and rights per the holy spirit ghost of the GPL; or b) deny themselves the ability to every offer a patch to said BIOS if bugs are found Point (b) is also exactly on topic for the discussion of enforcing legal safety obligations in hardware on a peripheral rather than the software drivers. It's requiring that these limitations be placed in a technically inferior location to hack around a legal "bug". (A bug is in the eye of the beholder, please wear glasses while cycling, it's your own responsibility to protect your eyes) Er, I think I'm done. Yes. Executive summary: a) by not providing the BIOS source code but retaining the right to change the BIOS the vendor is linking the GPLix kernel and the BIOS (you can't run the kernel without it) b) legislating intent is fraught. c) by your arguments, (a) is violating the spirit and (b) is necessary to get around that. Bron. - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: Dual-Licensing Linux Kernel with GPL V2 and GPL V3
On Fri, Jun 15, 2007 at 11:24:08AM -0300, Tomas Neme wrote: >> 1) What is "tat"? >> >> 2) How can I get some? >> >> 3) Where do I go to trade it in? > > 4) is it legal to consume it in my country? > > 5) should I have a designed driver when I do? 6) Is that allowed to be a binary-only driver or does it have to come with source code? - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: Dual-Licensing Linux Kernel with GPL V2 and GPL V3
On Fri, Jun 15, 2007 at 11:24:08AM -0300, Tomas Neme wrote: 1) What is tat? 2) How can I get some? 3) Where do I go to trade it in? 4) is it legal to consume it in my country? 5) should I have a designed driver when I do? 6) Is that allowed to be a binary-only driver or does it have to come with source code? - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: Dual-Licensing Linux Kernel with GPL V2 and GPL V3
On Sat, Jun 16, 2007 at 05:22:21AM -0300, Alexandre Oliva wrote: On Jun 15, 2007, Bron Gondwana [EMAIL PROTECTED] wrote: because it could easily be argued that they linked the BIOS with the Linux kernel How so? (I'm going to refer to Linux as GPLix from here on since this argument is more general than a specific GPLed operating system) Er, they installed it in the same piece of equipment, and the kernel couldn't function without it in that work. What's more 'linked' than that. It's a vital part of the boot process on that piece of hardware in exactly the same way that the public-key check is a vital part of the boot process. If your printer^wPC isn't doing what you want and you know how to change it to do what you want but it needs a BIOS patch. Guess what, you can't do it - your vendor can. By using GPLix as part of their boot process along with their non-GPL BIOS, they're subverting the freedoms that the user should have in being able to control the entire boot process. Right? Or are you unclear about the fact that there's a big grey area cutting through this part of usage, and Linus sat down pretty clearly on one side of it while you're arguing that the goalposts should be where I feel that my rights to make changes are being infringed. While the vendor reserves the ability to change components of the system (post sale, i.e. a BIOS flash update) and doesn't hand those same rights on to you, they have partially Tivoised (hoover, kleenex, you've got nothing on these guys for having your name associated with a concept) the hardware. By logical extention of your arguments over the past few days, this denies them the ability to use any GPLed software in 'the spirit of the licence' anyware on the machine because they are denying you rights regarding the instance of the product they shipped to you that they are retaining for themselves. The very freedoms you so vocally claim. Now, the position I'm seeing here is that the above behaviour (every single hardware manufacturer that has ever shipped a machine with pre-installed Linux) violates the spirit of the GPL by the retaining exclusive freedoms to modify shipped product rule, and hence their BIOS is in the doghouse unless they either: a) offer full source code access and rights per the holy spirit ghost of the GPL; or b) deny themselves the ability to every offer a patch to said BIOS if bugs are found Point (b) is also exactly on topic for the discussion of enforcing legal safety obligations in hardware on a peripheral rather than the software drivers. It's requiring that these limitations be placed in a technically inferior location to hack around a legal bug. (A bug is in the eye of the beholder, please wear glasses while cycling, it's your own responsibility to protect your eyes) Er, I think I'm done. Yes. Executive summary: a) by not providing the BIOS source code but retaining the right to change the BIOS the vendor is linking the GPLix kernel and the BIOS (you can't run the kernel without it) b) legislating intent is fraught. c) by your arguments, (a) is violating the spirit and (b) is necessary to get around that. Bron. - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: Dual-Licensing Linux Kernel with GPL V2 and GPL V3
[note: I'm writting this while offline and likely to remain so for the next 8 hours or so, so I'll probably miss a bunch of other replies] On Sat, Jun 16, 2007 at 02:14:29PM -0300, Alexandre Oliva wrote: On Jun 16, 2007, Bron Gondwana [EMAIL PROTECTED] wrote: On Sat, Jun 16, 2007 at 05:22:21AM -0300, Alexandre Oliva wrote: On Jun 15, 2007, Bron Gondwana [EMAIL PROTECTED] wrote: because it could easily be argued that they linked the BIOS with the Linux kernel How so? Er, they installed it in the same piece of equipment, and the kernel couldn't function without it in that work. I see what you're getting at. You're thinking of a license that doesn't respect the idea of mere aggregation, right? No, I'm arguing that it's not mere aggregation - the kernel is useless on that machine unless the BIOS is present or replaced with something else with equivalent functionality. I suspect any decent lawyer could make the theory that this made the kernel as compiled on to that machine with specific chipset support selected for that hardware into a derived work of the BIOS - especially if the vendor had contributed GPLed code for drivers which interact with their hardware into said kernel. In fact, particularly if the hardware vendor has also contributed GPL code that interacts on one side of the software/(firmware, hardware) boundard which worked around bugs in said firmware/hardware which they also had the ability to change. The two really are a combined work of which only one part is GPLed. Ringing any binary kernel module video card driver bells yet? It's really the same thing from the opposite direction - the only criteria is where you fit in the pecking order - hardware manufacturers work around Windows bugs, Linux kernel drivers work around hardware bugs - it's all about who has more to lose if they aren't compatible. For starters, this wouldn't evidently not qualify as an Open Source license, and I'm pretty sure it wouldn't qualify as a Free Software license either. Strawman licence? By using GPLix as part of their boot process along with their non-GPL BIOS, they're subverting the freedoms that the user should have in being able to control the entire boot process. You're pushing the freedom to change too far. Sure, I'd like to be able to do that, and I prefer hardware that lets me do it, but it's not like this BIOS in the scenario you described is being used as a means to stop me from modifying the GPLed software. Well, yeah - except this is the direction GPL3 takes us, and it's a theory that GPL3 makes more likely to fly in court than GPL2 does - meaning that hardware vendor lawyers lie awake at night worrying about stuff (I'd hate to be a good lawyer - I'd never get any sleep!) I have never said that including a GPLed piece of software should grant users the right to modify anything whatsoever in the system, or grant them control over the entire system. Others have, but it's not true, it just shows how much mis-information is floating around. No, but your interactions with Linus (lazy bums 'r' us) have shown that the logical result of what you do want includes this. It's a lot harder to objectively judge one of these than the other: a) have they provided the source code to this binary to anyone who asks. b) have any of the limitations of this piece of hardware been created with the intent of making it more difficult for J. Random Enduser to build modified binaries from said source and have them function correctly. (b) has much more scope for shenanigans by bad apples on the copyright owner side - and don't pretend that only the hardware vendors are bad guys - it takes all sorts and the idea of a licence is to protect both parties. All the GPL stands for is to defend the freedom of the users over the particular program it applies to. You can't impose further restrictions on the user's ability to modify what *that* software does. Except where they run into limitations of the platform itself, or just plain bugs. Oops. The lawyers will have a field day discovering intent every time J. Random's kernel doesn't do what he wants after he fiddles the code a bit. If you wanted to change something else, but this something else is not covered by the license, and is not being used to contradict the terms of the license, well, too bad, you lose. b) deny themselves the ability to every offer a patch to said BIOS if bugs are found Point (b) is also exactly on topic for the discussion of enforcing legal safety obligations in hardware on a peripheral rather than the software drivers. It's requiring that these limitations be placed in a technically inferior location to hack around a legal bug. I don't think this last sentence is true. If you implement hardware locks that prevent modification of the software even by yourself, then you're in compliance with the terms of the GPLv3dd4. But IANAL. I obviously
Re: Dual-Licensing Linux Kernel with GPL V2 and GPL V3
On Fri, Jun 15, 2007 at 04:26:34PM -0300, Alexandre Oliva wrote: > On Jun 15, 2007, Bron Gondwana <[EMAIL PROTECTED]> wrote: > > > What happens if you're debugging something you think is a bug in the > > Linux kernel and then you run bang into some interactions that make you > > think the bug might be in the BIOS instead. > > > have denied your freedom to modify and debug the system they sold you > > If the bug is in the non-GPLed BIOS, not in the GPLed code, too bad. > One more reason to dislike non-Free Software. If the bug is in the non-GPLed binary module, not in the GPLed code, too bad. One more reason to dislike non-Free Software. It's the same argument from the other direction. The BIOS is linked (inside the machine, sure) to the kernel for all intents and purposes through a defined interface. This doesn't affect the BIOS developers who ship me a machine on to which I then install Linux, but it _does_ affect a hardware vendor who ships me a system with Linux pre-installed, because it could easily be argued that they linked the BIOS with the Linux kernel and hence produced a combined work (remember, they reserve the right to modify the BIOS, but don't give that that right to me) and the BIOS should now come under the GPL. Talk about your chilling effects. It's a strong reason for vendors not to ship GPL3 or GPL2[your interpretation] code pre-installed while the legal boundaries of work combination are in any way grey. Regards, Bron. - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: Dual-Licensing Linux Kernel with GPL V2 and GPL V3
On Fri, Jun 15, 2007 at 02:38:41AM -0300, Alexandre Oliva wrote: > On Jun 15, 2007, Bron Gondwana <[EMAIL PROTECTED]> wrote: > > > #define Dell CFG_FAVOURITE_VENDOR > > > A Dell desktop machine is a piece of hardware. The manufacturer has the > > source code (hypothetically) to the BIOS. The BIOS is required for the > > machine to boot and run Linux. > > > Riddle me this (especially Alexandre, I'm just latching on to Ingo's > > post because it has the right hook to grab) - are Dell required to give > > out the source to the bios to enable people to have the same rights Dell > > engineers do to modify the behaviour of the system? > > What is the license for the bios? Does it say anything about 'no > further restrictions on the freedoms to modify and share the > software'? It's a necessary part of the boot process, without which Linux could not be started. Indeed, the Linux kernel interacts with it through a (loosely, incompletely and frequently buggy) documented interface, much like how binary modules interact with the linux kernel (even if they do get loaded into the sacred ring0 execution space, ooh err) What happens if you're debugging something you think is a bug in the Linux kernel and then you run bang into some interactions that make you think the bug might be in the BIOS instead. Oh unhappy day, you don't have access to the source code to the BIOS so you can't check. Those cretins at Dell (does a #define still work when it's 2 levels quoted?) have denied your freedom to modify and debug the system they sold you which is based _in_a_large_part_ on the GPL$mumble Linux kernel and hence needs to be interoperable. Regardless of your sophistry, it's a slipery slope by which Dell could be forced to exert their corporate might back up the tree to the BIOS vendor and get the right to release that BIOS source code to you or stop distributing Linux on their machines. > Does it include any mechanisms to stop people from booting modified > versions of the Linux that ships with the machine? Maybe, and either way, a future update could, and you couldn't undo it unless the BIOS flash system lets you "downgrade" again. Bron. - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: Dual-Licensing Linux Kernel with GPL V2 and GPL V3
On Fri, Jun 15, 2007 at 02:38:41AM -0300, Alexandre Oliva wrote: On Jun 15, 2007, Bron Gondwana [EMAIL PROTECTED] wrote: #define Dell CFG_FAVOURITE_VENDOR A Dell desktop machine is a piece of hardware. The manufacturer has the source code (hypothetically) to the BIOS. The BIOS is required for the machine to boot and run Linux. Riddle me this (especially Alexandre, I'm just latching on to Ingo's post because it has the right hook to grab) - are Dell required to give out the source to the bios to enable people to have the same rights Dell engineers do to modify the behaviour of the system? What is the license for the bios? Does it say anything about 'no further restrictions on the freedoms to modify and share the software'? It's a necessary part of the boot process, without which Linux could not be started. Indeed, the Linux kernel interacts with it through a (loosely, incompletely and frequently buggy) documented interface, much like how binary modules interact with the linux kernel (even if they do get loaded into the sacred ring0 execution space, ooh err) What happens if you're debugging something you think is a bug in the Linux kernel and then you run bang into some interactions that make you think the bug might be in the BIOS instead. Oh unhappy day, you don't have access to the source code to the BIOS so you can't check. Those cretins at Dell (does a #define still work when it's 2 levels quoted?) have denied your freedom to modify and debug the system they sold you which is based _in_a_large_part_ on the GPL$mumble Linux kernel and hence needs to be interoperable. Regardless of your sophistry, it's a slipery slope by which Dell could be forced to exert their corporate might back up the tree to the BIOS vendor and get the right to release that BIOS source code to you or stop distributing Linux on their machines. Does it include any mechanisms to stop people from booting modified versions of the Linux that ships with the machine? Maybe, and either way, a future update could, and you couldn't undo it unless the BIOS flash system lets you downgrade again. Bron. - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: Dual-Licensing Linux Kernel with GPL V2 and GPL V3
On Fri, Jun 15, 2007 at 04:26:34PM -0300, Alexandre Oliva wrote: On Jun 15, 2007, Bron Gondwana [EMAIL PROTECTED] wrote: What happens if you're debugging something you think is a bug in the Linux kernel and then you run bang into some interactions that make you think the bug might be in the BIOS instead. have denied your freedom to modify and debug the system they sold you If the bug is in the non-GPLed BIOS, not in the GPLed code, too bad. One more reason to dislike non-Free Software. If the bug is in the non-GPLed binary module, not in the GPLed code, too bad. One more reason to dislike non-Free Software. It's the same argument from the other direction. The BIOS is linked (inside the machine, sure) to the kernel for all intents and purposes through a defined interface. This doesn't affect the BIOS developers who ship me a machine on to which I then install Linux, but it _does_ affect a hardware vendor who ships me a system with Linux pre-installed, because it could easily be argued that they linked the BIOS with the Linux kernel and hence produced a combined work (remember, they reserve the right to modify the BIOS, but don't give that that right to me) and the BIOS should now come under the GPL. Talk about your chilling effects. It's a strong reason for vendors not to ship GPL3 or GPL2[your interpretation] code pre-installed while the legal boundaries of work combination are in any way grey. Regards, Bron. - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: Dual-Licensing Linux Kernel with GPL V2 and GPL V3
On Fri, Jun 15, 2007 at 01:50:04AM +0200, Ingo Molnar wrote: > the GPL applies to software. It is a software license. > > the Tivo box is a piece of hardware. > > a disk is put into it with software copied to it already: a bootloader, > a Linux kernel plus a handful of applications. The free software bits > are available for download. #define Dell CFG_FAVOURITE_VENDOR A Dell desktop machine is a piece of hardware. The manufacturer has the source code (hypothetically) to the BIOS. The BIOS is required for the machine to boot and run Linux. Riddle me this (especially Alexandre, I'm just latching on to Ingo's post because it has the right hook to grab) - are Dell required to give out the source to the bios to enable people to have the same rights Dell engineers do to modify the behaviour of the system? Bron. - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: Dual-Licensing Linux Kernel with GPL V2 and GPL V3
On Thu, Jun 14, 2007 at 05:25:19PM -0400, Dmitry Torokhov wrote: > On 6/14/07, Dave Neuer <[EMAIL PROTECTED]> wrote: >> On 6/14/07, Lennart Sorensen <[EMAIL PROTECTED]> wrote: >> > Nothing prevents you from taking tivos kernel >> > changes and building your own hardware to run that code on, and as such >> > the spirit of the GPL v2 seems fulfilled. >> >> Oh, come on: you're not serious, right? Something indeed prevents me >> -- the fact that I'm not a hardware manufacturer, I don't have fabs, >> outsource vendors to provide me w/ designs, ASICs, etc. Nor to I have >> the money to pay one-off prices for various components if they're even >> available in batches that small. >> > > So your objection here is that one needs additional resources to do > excersise their rights. Well, what about spending time and money to > get education to be able to do programming work? Being able to > understand C and hardware, etc is also an additional restriction > imposed on an average person. Do you advocate that every copy of GPL > program should be accompanied with an engineer who would explain how > it all works? Yes please. Can she be spunky as well? ta. - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: Dual-Licensing Linux Kernel with GPL V2 and GPL V3
On Thu, Jun 14, 2007 at 10:14:21AM -0400, Robin Getz wrote: > - gambling devices - which must have their software certified by various > government agencies - to make sure that the odds are known, and there are no > backdoors, and consumers don't get screwed - the manufacture can not allow > non-certified software to be loaded on it. If these are in a hotel - where > various people live - is that considered "incorporation into a dwelling"? > > Not wanting to start a debate about the morality of being involved in the > gambling industry - (if the statically challenged are giving the government > money to keep my taxes down, I am mostly OK with it) - but I'm not "happy" > thinking that someone can ledgistate restrictions on embedded OS choice, just > because it must be verified by a third party. Why not go really controversial and dive straight in with "voting machines". There's a whole 'nother can of worms. Bron. - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: Dual-Licensing Linux Kernel with GPL V2 and GPL V3
On Thu, Jun 14, 2007 at 01:58:26AM -0300, Alexandre Oliva wrote: > On Jun 14, 2007, Bron Gondwana <[EMAIL PROTECTED]> wrote: > > > Tivo gets sick of the endless flamewars on lkml, signs a copy > > of QNX, pushes it out to the hardware. No more Linux on Tivo. > > What do we lose? > > Do we actually get any benefit whatsoever from TiVO's choice of Linux > as the kernel for its device? Sure, if they make any changes or fixes to Linux. Other than that, only the same benefit that Microsoft get from Windows piracy - TiVo employees become familiar with Linux and are more likely to use it and maybe contribute more in another job later. What we don't get is TiVo having a better kernel than everyone else because they've put some work into extending it without giving that work back. I see stuff in arch/powerpc/kernel/ which is Copyright "TiVo, Inc" and more recent stuff in usb/net/asix.c and usb/net/mcs7830.c which is more than I've ever contributed to the kernel, despite making extensive use and even selling services where I ran servers with Linux on them but didn't allow my customers to change the kernel on the servers if there was some feature they wanted to play with. > Do TiVO customers lose anything from the change from one non-Free > software to another? (the Linux binary, as shipped in the TiVO, has > become non-Free) Not particularly, no. Other than maybe some nice features that TiVo gains from being able to use Linux. Bron. - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: Dual-Licensing Linux Kernel with GPL V2 and GPL V3
On Thu, Jun 14, 2007 at 01:58:26AM -0300, Alexandre Oliva wrote: On Jun 14, 2007, Bron Gondwana [EMAIL PROTECTED] wrote: Tivo gets sick of the endless flamewars on lkml, signs a copy of QNX, pushes it out to the hardware. No more Linux on Tivo. What do we lose? Do we actually get any benefit whatsoever from TiVO's choice of Linux as the kernel for its device? Sure, if they make any changes or fixes to Linux. Other than that, only the same benefit that Microsoft get from Windows piracy - TiVo employees become familiar with Linux and are more likely to use it and maybe contribute more in another job later. What we don't get is TiVo having a better kernel than everyone else because they've put some work into extending it without giving that work back. I see stuff in arch/powerpc/kernel/ which is Copyright TiVo, Inc and more recent stuff in usb/net/asix.c and usb/net/mcs7830.c which is more than I've ever contributed to the kernel, despite making extensive use and even selling services where I ran servers with Linux on them but didn't allow my customers to change the kernel on the servers if there was some feature they wanted to play with. Do TiVO customers lose anything from the change from one non-Free software to another? (the Linux binary, as shipped in the TiVO, has become non-Free) Not particularly, no. Other than maybe some nice features that TiVo gains from being able to use Linux. Bron. - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: Dual-Licensing Linux Kernel with GPL V2 and GPL V3
On Thu, Jun 14, 2007 at 10:14:21AM -0400, Robin Getz wrote: - gambling devices - which must have their software certified by various government agencies - to make sure that the odds are known, and there are no backdoors, and consumers don't get screwed - the manufacture can not allow non-certified software to be loaded on it. If these are in a hotel - where various people live - is that considered incorporation into a dwelling? Not wanting to start a debate about the morality of being involved in the gambling industry - (if the statically challenged are giving the government money to keep my taxes down, I am mostly OK with it) - but I'm not happy thinking that someone can ledgistate restrictions on embedded OS choice, just because it must be verified by a third party. Why not go really controversial and dive straight in with voting machines. There's a whole 'nother can of worms. Bron. - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: Dual-Licensing Linux Kernel with GPL V2 and GPL V3
On Thu, Jun 14, 2007 at 05:25:19PM -0400, Dmitry Torokhov wrote: On 6/14/07, Dave Neuer [EMAIL PROTECTED] wrote: On 6/14/07, Lennart Sorensen [EMAIL PROTECTED] wrote: Nothing prevents you from taking tivos kernel changes and building your own hardware to run that code on, and as such the spirit of the GPL v2 seems fulfilled. Oh, come on: you're not serious, right? Something indeed prevents me -- the fact that I'm not a hardware manufacturer, I don't have fabs, outsource vendors to provide me w/ designs, ASICs, etc. Nor to I have the money to pay one-off prices for various components if they're even available in batches that small. So your objection here is that one needs additional resources to do excersise their rights. Well, what about spending time and money to get education to be able to do programming work? Being able to understand C and hardware, etc is also an additional restriction imposed on an average person. Do you advocate that every copy of GPL program should be accompanied with an engineer who would explain how it all works? Yes please. Can she be spunky as well? ta. - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: Dual-Licensing Linux Kernel with GPL V2 and GPL V3
On Fri, Jun 15, 2007 at 01:50:04AM +0200, Ingo Molnar wrote: the GPL applies to software. It is a software license. the Tivo box is a piece of hardware. a disk is put into it with software copied to it already: a bootloader, a Linux kernel plus a handful of applications. The free software bits are available for download. #define Dell CFG_FAVOURITE_VENDOR A Dell desktop machine is a piece of hardware. The manufacturer has the source code (hypothetically) to the BIOS. The BIOS is required for the machine to boot and run Linux. Riddle me this (especially Alexandre, I'm just latching on to Ingo's post because it has the right hook to grab) - are Dell required to give out the source to the bios to enable people to have the same rights Dell engineers do to modify the behaviour of the system? Bron. - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: Dual-Licensing Linux Kernel with GPL V2 and GPL V3
On Thu, Jun 14, 2007 at 02:52:48AM +0100, Alan Cox wrote: > > What if TiVo had put the kernel in a burned-in ROM (not flash, or on a > > flash ROM with no provision for reprogramming it)? Would that also > > violate the "spirit" of the GPL? Must any device that wishes to include > > GPL code include additional hardware to support replacing that code > > (even if that hardware is otherwise superfluous)? > > As a PS to the GPL3 comment here is the basic difference > > ROM - I can't modify the code on the device > The creator can't modify the code further on the device > > Tivo - I can't modify the code on the device > The owner can modify the code Tivo gets sick of the endless flamewars on lkml, signs a copy of QNX, pushes it out to the hardware. No more Linux on Tivo. You also can't replace that but Tivo can. As I see it the two are completely orthagonal: a) Can anyone but the manufacturer upload new software into a a device without taking extreme measures (soldering a new public-key-containing-chip onto the board) b) Is the software currently installed on a device licenced under a rule which requires the distributor to also distribute source code upon request. Now I think it would reasonable to ask that the source code be able to be built by [same compiler, same flags, same ...] to produce an identical binary to the one running on the device so you can confirm that it's exactly the same code. That's separate from being able to upload a changed binary. Bron. - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: Dual-Licensing Linux Kernel with GPL V2 and GPL V3
On Thu, Jun 14, 2007 at 02:52:48AM +0100, Alan Cox wrote: What if TiVo had put the kernel in a burned-in ROM (not flash, or on a flash ROM with no provision for reprogramming it)? Would that also violate the spirit of the GPL? Must any device that wishes to include GPL code include additional hardware to support replacing that code (even if that hardware is otherwise superfluous)? As a PS to the GPL3 comment here is the basic difference ROM - I can't modify the code on the device The creator can't modify the code further on the device Tivo - I can't modify the code on the device The owner can modify the code Tivo gets sick of the endless flamewars on lkml, signs a copy of QNX, pushes it out to the hardware. No more Linux on Tivo. You also can't replace that but Tivo can. As I see it the two are completely orthagonal: a) Can anyone but the manufacturer upload new software into a a device without taking extreme measures (soldering a new public-key-containing-chip onto the board) b) Is the software currently installed on a device licenced under a rule which requires the distributor to also distribute source code upon request. Now I think it would reasonable to ask that the source code be able to be built by [same compiler, same flags, same ...] to produce an identical binary to the one running on the device so you can confirm that it's exactly the same code. That's separate from being able to upload a changed binary. Bron. - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: Dual-Licensing Linux Kernel with GPL V2 and GPL V3
On Mon, Jun 11, 2007 at 11:03:48AM +0200, Ingo Molnar wrote: > > * Tarkan Erimer <[EMAIL PROTECTED]> wrote: > > > > (*) And I've been pushing for that since before they even released > > > it - I walked out on Bill Joy at a private event where they > > > discussed their horrible previous Java license. > > > > Thanks for making things more clear :-) Some really strong indications > > that Sun is very willing to,at least, "Dual-License" the OpenSolaris > > with GPLv3. I think; in a very short time; we will see when the GPLv3 > > finalized and released. > > that would certainly be a good and productive move from them. Note the > issue that others have pointed out to you: OpenSolaris is probably more > interested in picking up code from Linux than the other way around! :-) > You mentioned "dtrace" and "ZFS". Firstly, Linux already has a "dtrace" > equivalent. Secondly, ZFS might be interesting in theory ^^ Having just got my first Sun box (one of those 48 disk Thumper machines) I can say it's very interesting in practice too. The admin tools for zfs are a dream to use (unlike just about everything else on Solaris which is a bugwards compatible nightmare to use) and the attitude of checksumming everything on to disk and checking that those checksums match on the way back out presses my "do your own safety checking and don't trust the hardware" buttons very much the right way - especially with that much hardware in there. I would love to see Sun GPL3 OpenSolaris, not so much for the code itself (maybe not portable into Linux) but for the clarity it would give to the patent position. The patent peace would be portable back to GPL2. Bron. - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: Dual-Licensing Linux Kernel with GPL V2 and GPL V3
On Mon, Jun 11, 2007 at 11:03:48AM +0200, Ingo Molnar wrote: * Tarkan Erimer [EMAIL PROTECTED] wrote: (*) And I've been pushing for that since before they even released it - I walked out on Bill Joy at a private event where they discussed their horrible previous Java license. Thanks for making things more clear :-) Some really strong indications that Sun is very willing to,at least, Dual-License the OpenSolaris with GPLv3. I think; in a very short time; we will see when the GPLv3 finalized and released. that would certainly be a good and productive move from them. Note the issue that others have pointed out to you: OpenSolaris is probably more interested in picking up code from Linux than the other way around! :-) You mentioned dtrace and ZFS. Firstly, Linux already has a dtrace equivalent. Secondly, ZFS might be interesting in theory ^^ Having just got my first Sun box (one of those 48 disk Thumper machines) I can say it's very interesting in practice too. The admin tools for zfs are a dream to use (unlike just about everything else on Solaris which is a bugwards compatible nightmare to use) and the attitude of checksumming everything on to disk and checking that those checksums match on the way back out presses my do your own safety checking and don't trust the hardware buttons very much the right way - especially with that much hardware in there. I would love to see Sun GPL3 OpenSolaris, not so much for the code itself (maybe not portable into Linux) but for the clarity it would give to the patent position. The patent peace would be portable back to GPL2. Bron. - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: Linux 2.6.16.40
On Mon, Feb 12, 2007 at 05:13:47PM +1100, Bron Gondwana wrote: > On Sat, Feb 10, 2007 at 05:41:13PM +0100, Adrian Bunk wrote: > > New drivers since 2.6.16.39: > > - Areca ARC11X0/ARC12X0 SATA-RAID support > > - AMD Athlon64/FX and Opteron temperature sensor > > Sorry - I think I just sent a blank reply to this! Oops. > > I was going to say - thanks. We'll definitely be using > this kernel since we've found that 2.6.19.2 has a > significantly worse IO profile and either the 2.6.16.40-rc1 > kernel or 2.6.18 on otherwise identical machines with pretty > stable loads (so we can compare back to previous weeks). > > Now to figure out what's causing the extra load in the > 2.6.19 branch! By the way, we've just tried a 2.6.20.11 kernel on another machine and so far (fingers crossed) we're not seeing the load issues we had with the 2.6.19 kernel we were using. Given that the Areca driver is the same in both, the only other funky thing we did was a single patch to ReiserFS, but we're applying that same patch on 2.6.20. (my sneaking suspicion was that the delayed bitmap loading was to blame, but it's hard to test these things short of multiple reboots, and we don't have a testing Cyrus install that we can load that heavily) Anyway - if this goes well for the next couple of days we'll probably stop tracking the 2.6.16 series. Thanks again, very much, for your work with 2.6.16. It was fantastic to have this resource to keep us from having to chase more recent kernels just to get the patches we needed! Bron. - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: Linux 2.6.16.40
On Mon, Feb 12, 2007 at 05:13:47PM +1100, Bron Gondwana wrote: On Sat, Feb 10, 2007 at 05:41:13PM +0100, Adrian Bunk wrote: New drivers since 2.6.16.39: - Areca ARC11X0/ARC12X0 SATA-RAID support - AMD Athlon64/FX and Opteron temperature sensor Sorry - I think I just sent a blank reply to this! Oops. I was going to say - thanks. We'll definitely be using this kernel since we've found that 2.6.19.2 has a significantly worse IO profile and either the 2.6.16.40-rc1 kernel or 2.6.18 on otherwise identical machines with pretty stable loads (so we can compare back to previous weeks). Now to figure out what's causing the extra load in the 2.6.19 branch! By the way, we've just tried a 2.6.20.11 kernel on another machine and so far (fingers crossed) we're not seeing the load issues we had with the 2.6.19 kernel we were using. Given that the Areca driver is the same in both, the only other funky thing we did was a single patch to ReiserFS, but we're applying that same patch on 2.6.20. (my sneaking suspicion was that the delayed bitmap loading was to blame, but it's hard to test these things short of multiple reboots, and we don't have a testing Cyrus install that we can load that heavily) Anyway - if this goes well for the next couple of days we'll probably stop tracking the 2.6.16 series. Thanks again, very much, for your work with 2.6.16. It was fantastic to have this resource to keep us from having to chase more recent kernels just to get the patches we needed! Bron. - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: Recent wireless breakage (ipw2200, iwconfig, NetworkManager)
On Mon, Mar 05, 2007 at 03:14:25PM -0600, Matt Mackall wrote: > On Mon, Mar 05, 2007 at 04:46:09PM +0100, Tomasz Torcz wrote: > > > That's not the point. The point is that Debian/unstable as of _this > > > morning_ doesn't work. For reference, I'm running both the latest > > > releases of both hal (0.5.8.1-6.1) and network-manager (0.6.4-6). And > > > there are people telling me I need a copy of HAL out of git that > > > hasn't even been released for Debian to package. Debian isn't the > > > problem here. > > > > hal 0.5.9-rc1 (released, not from git) should work. It will be > > problably released soon and picked by sane distributions. Debian is very > > irritating corner case. > > Presumably the -rc1 stands for "release candidate". Which means "not > yet released". And when did it show up? 04-Mar-2007 at 18:31. That's > right, YESTERDAY. Almost a full month after Greg's commit. > > For the last time, DEBIAN IS NOT THE PROBLEM. Can I please second this (having been burned by hell that was udev of the 0.5ish era) - Greg, please try to make changes in a cross-compatible way so that versions of userspace and kernel are not so closely dependant on tracking each other. The whole 2.6.8 -> 2.6.12 series of kernels and associated udevs are fraught with race conditions where upgrading one but not the other will leave your machine unbootable. I read the "manifesto" for udev showing how crap devfs was, it was broken, it could never be fixed etc - yet my experience was that devfs systems "just worked"[tm] and udev was very dangerous. My thinking is going to be tarnished by that for a while and my mental image of udev is "unreliable POS". I'm hoping enough good experiences with udev might make me feel less scared whenever I have to deal with it. Similarly, I'm hoping I don't have to think "oh shit, will this break boot" every time I upgrade either a kernel or hal version for the next year, because it would really suck to do that all over again. It contributes to the meme that linux is unreliable and perpetually unstable. Regards, Bron. - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: Recent wireless breakage (ipw2200, iwconfig, NetworkManager)
On Mon, Mar 05, 2007 at 03:14:25PM -0600, Matt Mackall wrote: On Mon, Mar 05, 2007 at 04:46:09PM +0100, Tomasz Torcz wrote: That's not the point. The point is that Debian/unstable as of _this morning_ doesn't work. For reference, I'm running both the latest releases of both hal (0.5.8.1-6.1) and network-manager (0.6.4-6). And there are people telling me I need a copy of HAL out of git that hasn't even been released for Debian to package. Debian isn't the problem here. hal 0.5.9-rc1 (released, not from git) should work. It will be problably released soon and picked by sane distributions. Debian is very irritating corner case. Presumably the -rc1 stands for release candidate. Which means not yet released. And when did it show up? 04-Mar-2007 at 18:31. That's right, YESTERDAY. Almost a full month after Greg's commit. For the last time, DEBIAN IS NOT THE PROBLEM. Can I please second this (having been burned by hell that was udev of the 0.5ish era) - Greg, please try to make changes in a cross-compatible way so that versions of userspace and kernel are not so closely dependant on tracking each other. The whole 2.6.8 - 2.6.12 series of kernels and associated udevs are fraught with race conditions where upgrading one but not the other will leave your machine unbootable. I read the manifesto for udev showing how crap devfs was, it was broken, it could never be fixed etc - yet my experience was that devfs systems just worked[tm] and udev was very dangerous. My thinking is going to be tarnished by that for a while and my mental image of udev is unreliable POS. I'm hoping enough good experiences with udev might make me feel less scared whenever I have to deal with it. Similarly, I'm hoping I don't have to think oh shit, will this break boot every time I upgrade either a kernel or hal version for the next year, because it would really suck to do that all over again. It contributes to the meme that linux is unreliable and perpetually unstable. Regards, Bron. - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [ANNOUNCE] DualFS: File System with Meta-data and Data Separation
On Tue, Feb 20, 2007 at 12:57:50AM +0100, Juan Piernas Canovas wrote: > Now, let us assume that the data device takes 90% of the disk space, and > the meta-data device the other 10%. When the data device gets full, the > meta-data blocks will be using the half of the meta-data device, and the > other half (5% of the entire disk) will be free. Frankly, 5% is not too > much. I'd like to very strongly second this. We have a target maximum of 80% usage on all partitions, with emailed warnings at 85% and paged warnings at 90%. I consider any partition over 90% to be dangerously over-full. Given that disks space is approximately US$1/Gb in RAID6 SATA these days, it doesn't cost much to make sure there's some free. What matters a lot more is the speed at which you can get data on and off these huge devices, because drive speed isn't keeping up with capacity. It takes about a week to fill one of the external storage monsters we're buying these days, and that's streaming writes. Makes restoring from backup a scary proposition given the downtime it will take. But back on topic - I'd be very willing to pay a 5% disk space penalty for the sort of performance that DualFS is promising, but I'd need a 2.6 series kernel with the O(1) scheduler or performance would go down the toilet in the other direction. Bron. P.S. do you have any figures for high MMAP load? We run big Cyrus, and it has a serious MMAP fetish. - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [ANNOUNCE] DualFS: File System with Meta-data and Data Separation
On Tue, Feb 20, 2007 at 12:57:50AM +0100, Juan Piernas Canovas wrote: Now, let us assume that the data device takes 90% of the disk space, and the meta-data device the other 10%. When the data device gets full, the meta-data blocks will be using the half of the meta-data device, and the other half (5% of the entire disk) will be free. Frankly, 5% is not too much. I'd like to very strongly second this. We have a target maximum of 80% usage on all partitions, with emailed warnings at 85% and paged warnings at 90%. I consider any partition over 90% to be dangerously over-full. Given that disks space is approximately US$1/Gb in RAID6 SATA these days, it doesn't cost much to make sure there's some free. What matters a lot more is the speed at which you can get data on and off these huge devices, because drive speed isn't keeping up with capacity. It takes about a week to fill one of the external storage monsters we're buying these days, and that's streaming writes. Makes restoring from backup a scary proposition given the downtime it will take. But back on topic - I'd be very willing to pay a 5% disk space penalty for the sort of performance that DualFS is promising, but I'd need a 2.6 series kernel with the O(1) scheduler or performance would go down the toilet in the other direction. Bron. P.S. do you have any figures for high MMAP load? We run big Cyrus, and it has a serious MMAP fetish. - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: Linux 2.6.16.40
On Sat, Feb 10, 2007 at 05:41:13PM +0100, Adrian Bunk wrote: > New drivers since 2.6.16.39: > - Areca ARC11X0/ARC12X0 SATA-RAID support > - AMD Athlon64/FX and Opteron temperature sensor Sorry - I think I just sent a blank reply to this! Oops. I was going to say - thanks. We'll definitely be using this kernel since we've found that 2.6.19.2 has a significantly worse IO profile and either the 2.6.16.40-rc1 kernel or 2.6.18 on otherwise identical machines with pretty stable loads (so we can compare back to previous weeks). Now to figure out what's causing the extra load in the 2.6.19 branch! Bron. - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: Linux 2.6.16.40
On Sat, Feb 10, 2007 at 05:41:13PM +0100, Adrian Bunk wrote: > New drivers since 2.6.16.39: > - Areca ARC11X0/ARC12X0 SATA-RAID support > - AMD Athlon64/FX and Opteron temperature sensor > > > Location: > ftp://ftp.kernel.org/pub/linux/kernel/v2.6/ > > git tree: > git://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-2.6.16.y.git > > > Changes since 2.6.16.39: > > Adrian Bunk (2): > Linux 2.6.16.40-rc1 > Linux 2.6.16.40 > > Andrew Morton (1): > [SCSI] areca sysfs fix > > Bartlomiej Zolnierkiewicz (1): > ia64: add pci_get_legacy_ide_irq() > > Erich Chen (1): > [SCSI] arcmsr: initial driver, version 1.20.00.13 > > James Bottomley (1): > [SCSI] arcmsr: fix up sysfs values > > Jeff Garzik (1): > [libata] use kmap_atomic(KM_IRQ0) in SCSI simulator > > Neil Brown (1): > Make 'repair' actually work for raid1. > > Rudolf Marek (4): > hwmon: New driver k8temp > k8temp: Add documentation > k8temp: Documentation update > hwmon: Update Rudolf Marek's e-mail address > > Vladimir Saveliev (1): > reiserfs: avoid tail packing if an inode was ever mmapped > > > Documentation/hwmon/k8temp | 55 + > Documentation/scsi/ChangeLog.arcmsr | 56 + > Documentation/scsi/arcmsr_spec.txt | 574 ++ > MAINTAINERS |6 > Makefile|2 > drivers/hwmon/Kconfig | 12 > drivers/hwmon/Makefile |1 > drivers/hwmon/hwmon-vid.c |4 > drivers/hwmon/k8temp.c | 292 + > drivers/hwmon/w83792d.c |2 > drivers/i2c/busses/i2c-ali1563.c|2 > drivers/md/md.c |2 > drivers/md/raid1.c |5 > drivers/scsi/Kconfig| 14 > drivers/scsi/Makefile |1 > drivers/scsi/arcmsr/Makefile|6 > drivers/scsi/arcmsr/arcmsr.h| 472 > drivers/scsi/arcmsr/arcmsr_attr.c | 381 +++ > drivers/scsi/arcmsr/arcmsr_hba.c| 1496 > drivers/scsi/libata-scsi.c |4 > fs/reiserfs/file.c | 20 > fs/reiserfs/inode.c |2 > include/asm-ia64/pci.h |6 > include/linux/pci_ids.h | 18 > include/linux/reiserfs_fs_i.h |2 > 25 files changed, 3426 insertions(+), 9 deletions(-) > > > - > To unsubscribe from this list: send the line "unsubscribe linux-kernel" in > the body of a message to [EMAIL PROTECTED] > More majordomo info at http://vger.kernel.org/majordomo-info.html > Please read the FAQ at http://www.tux.org/lkml/ - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: Linux 2.6.16.40
On Sat, Feb 10, 2007 at 05:41:13PM +0100, Adrian Bunk wrote: New drivers since 2.6.16.39: - Areca ARC11X0/ARC12X0 SATA-RAID support - AMD Athlon64/FX and Opteron temperature sensor Location: ftp://ftp.kernel.org/pub/linux/kernel/v2.6/ git tree: git://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-2.6.16.y.git Changes since 2.6.16.39: Adrian Bunk (2): Linux 2.6.16.40-rc1 Linux 2.6.16.40 Andrew Morton (1): [SCSI] areca sysfs fix Bartlomiej Zolnierkiewicz (1): ia64: add pci_get_legacy_ide_irq() Erich Chen (1): [SCSI] arcmsr: initial driver, version 1.20.00.13 James Bottomley (1): [SCSI] arcmsr: fix up sysfs values Jeff Garzik (1): [libata] use kmap_atomic(KM_IRQ0) in SCSI simulator Neil Brown (1): Make 'repair' actually work for raid1. Rudolf Marek (4): hwmon: New driver k8temp k8temp: Add documentation k8temp: Documentation update hwmon: Update Rudolf Marek's e-mail address Vladimir Saveliev (1): reiserfs: avoid tail packing if an inode was ever mmapped Documentation/hwmon/k8temp | 55 + Documentation/scsi/ChangeLog.arcmsr | 56 + Documentation/scsi/arcmsr_spec.txt | 574 ++ MAINTAINERS |6 Makefile|2 drivers/hwmon/Kconfig | 12 drivers/hwmon/Makefile |1 drivers/hwmon/hwmon-vid.c |4 drivers/hwmon/k8temp.c | 292 + drivers/hwmon/w83792d.c |2 drivers/i2c/busses/i2c-ali1563.c|2 drivers/md/md.c |2 drivers/md/raid1.c |5 drivers/scsi/Kconfig| 14 drivers/scsi/Makefile |1 drivers/scsi/arcmsr/Makefile|6 drivers/scsi/arcmsr/arcmsr.h| 472 drivers/scsi/arcmsr/arcmsr_attr.c | 381 +++ drivers/scsi/arcmsr/arcmsr_hba.c| 1496 drivers/scsi/libata-scsi.c |4 fs/reiserfs/file.c | 20 fs/reiserfs/inode.c |2 include/asm-ia64/pci.h |6 include/linux/pci_ids.h | 18 include/linux/reiserfs_fs_i.h |2 25 files changed, 3426 insertions(+), 9 deletions(-) - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/ - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: Linux 2.6.16.40
On Sat, Feb 10, 2007 at 05:41:13PM +0100, Adrian Bunk wrote: New drivers since 2.6.16.39: - Areca ARC11X0/ARC12X0 SATA-RAID support - AMD Athlon64/FX and Opteron temperature sensor Sorry - I think I just sent a blank reply to this! Oops. I was going to say - thanks. We'll definitely be using this kernel since we've found that 2.6.19.2 has a significantly worse IO profile and either the 2.6.16.40-rc1 kernel or 2.6.18 on otherwise identical machines with pretty stable loads (so we can compare back to previous weeks). Now to figure out what's causing the extra load in the 2.6.19 branch! Bron. - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Reiserfs and MMAP (was: How many people are using 2.6.16?)
On Wed, Jan 31, 2007 at 08:02:37AM +0100, Adrian Bunk wrote: > reiserfs: > commit de14569f94513279e3d44d9571a421e9da1759ae > [PATCH] resierfs: avoid tail packing if an inode was ever mmapped > backport to 2.6.16 required Which would explain the "notail" I've been careful to cargo-cult into every mount string since I started at this job, even though we're storing mainly very small files. Referring back to: <[EMAIL PROTECTED]> (which went to reiserfs-dev and a couple of the ever-growing CC list above) we're still not 100% sure if it's safe to remove the patch that I attached there: --- file.c~ 2004-10-02 12:29:33.223660850 +0400 +++ file.c 2004-10-08 10:03:03.001561661 +0400 @@ -1137,6 +1137,8 @@ return result; } +return generic_file_write(file, buf, count, ppos); + if ( unlikely((ssize_t) count < 0 )) return -EINVAL; which Hans asserted was about 5% slower than the resierfs custom write implementation, but we countered at least meant that we didn't crash in a steaming pile of processes stuck in D state with no way out every few days. It doesn't apply against 2.6.19 any more, which may be a good sign. I haven't seen anything in the changelogs that jumped out at me as the fix though. Regards, Bron. - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Reiserfs and MMAP (was: How many people are using 2.6.16?)
On Wed, Jan 31, 2007 at 08:02:37AM +0100, Adrian Bunk wrote: reiserfs: commit de14569f94513279e3d44d9571a421e9da1759ae [PATCH] resierfs: avoid tail packing if an inode was ever mmapped backport to 2.6.16 required Which would explain the notail I've been careful to cargo-cult into every mount string since I started at this job, even though we're storing mainly very small files. Referring back to: [EMAIL PROTECTED] (which went to reiserfs-dev and a couple of the ever-growing CC list above) we're still not 100% sure if it's safe to remove the patch that I attached there: --- file.c~ 2004-10-02 12:29:33.223660850 +0400 +++ file.c 2004-10-08 10:03:03.001561661 +0400 @@ -1137,6 +1137,8 @@ return result; } +return generic_file_write(file, buf, count, ppos); + if ( unlikely((ssize_t) count 0 )) return -EINVAL; which Hans asserted was about 5% slower than the resierfs custom write implementation, but we countered at least meant that we didn't crash in a steaming pile of processes stuck in D state with no way out every few days. It doesn't apply against 2.6.19 any more, which may be a good sign. I haven't seen anything in the changelogs that jumped out at me as the fix though. Regards, Bron. - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: How many people are using 2.6.16?
On Tue, Jan 30, 2007 at 06:36:48PM -0800, Linus Torvalds wrote: > > > On Tue, 30 Jan 2007, Mark Lord wrote: > > > > I believe our featherless leader said he though it was an ancient bug, > > exasperated by something that went into 2.6.19. > > > > If Linus's opinion is correct (still?), then the bug exists in all > > kernels since somewhere back in the 2.4.xx days. > > The issue was somewhat confused by people certainly *reporting* it for > older kernels. Also, as part of the dirty bit cleanups and sanity > checkingwe did actually seem to fix a long-standing CIFS corruption (and > apparently reisertfs/XFS problems too). > > But the *common* case was actually introduced with 2.6.19, and 2.6.16 > wouldn't be affected. We run on reiserfs. I did try ext3 for a little while on a couple of servers but performance was really awful compared to reiser, and we heaved a sigh of relief when we finally migrated all the users off those filesystems. There were many complaints about the speed of our service for a while. I'm really hoping this is the cause, because do still see occasional corruption of MMAPed files under heavy load, though less often now that we've balanced our servers to the point where load spikes are much less common. The servers are using either internal Areca cards or LSI SCSI adaptors connected to external SATA raid boxes. Either way, there's a few terabytes of SATA attached to each box, with 10kRPM drives in RAID1 for Cyrus's metadata and 7,2k bigger drives in RAID5 for the actual emails. According to iostat these drives are being utilised at over 50% of available bandwidth even now during the "quiet time" - there are many tens of thousands of users per machine - so we tend to stress the IO subsystem quite a lot. Cyrus is also very liberal in its use of MMAP, so we get to push all sorts of exciting edge cases. We were still applying patches to reiserfs until recently, and I'm not sure what the status of that is (Hans Reiser said to keep harassing him about it - but he's hardly in a position to be dealing with our issues right now) Thankfully, now that we're using 300Gb maximum rather than 2Tb partitions (running multiple instances of Cyrus instead) with the associated smaller mailboxes.db (the biggest MMAPed and frequently updated file) things seem less edgy. I don't like edgy (queue Ubuntu jokes). Anyway, I'm hoping to update one of our boxes to 2.6.19.2 soon. We do have one box running a 2.6.18 series kernel which has been fine as well. I'll give feedback if we see any issues with MMAP on there. Bron. - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: How many people are using 2.6.16?
On Tue, Jan 30, 2007 at 06:36:48PM -0800, Linus Torvalds wrote: On Tue, 30 Jan 2007, Mark Lord wrote: I believe our featherless leader said he though it was an ancient bug, exasperated by something that went into 2.6.19. If Linus's opinion is correct (still?), then the bug exists in all kernels since somewhere back in the 2.4.xx days. The issue was somewhat confused by people certainly *reporting* it for older kernels. Also, as part of the dirty bit cleanups and sanity checkingwe did actually seem to fix a long-standing CIFS corruption (and apparently reisertfs/XFS problems too). But the *common* case was actually introduced with 2.6.19, and 2.6.16 wouldn't be affected. We run on reiserfs. I did try ext3 for a little while on a couple of servers but performance was really awful compared to reiser, and we heaved a sigh of relief when we finally migrated all the users off those filesystems. There were many complaints about the speed of our service for a while. I'm really hoping this is the cause, because do still see occasional corruption of MMAPed files under heavy load, though less often now that we've balanced our servers to the point where load spikes are much less common. The servers are using either internal Areca cards or LSI SCSI adaptors connected to external SATA raid boxes. Either way, there's a few terabytes of SATA attached to each box, with 10kRPM drives in RAID1 for Cyrus's metadata and 7,2k bigger drives in RAID5 for the actual emails. According to iostat these drives are being utilised at over 50% of available bandwidth even now during the quiet time - there are many tens of thousands of users per machine - so we tend to stress the IO subsystem quite a lot. Cyrus is also very liberal in its use of MMAP, so we get to push all sorts of exciting edge cases. We were still applying patches to reiserfs until recently, and I'm not sure what the status of that is (Hans Reiser said to keep harassing him about it - but he's hardly in a position to be dealing with our issues right now) Thankfully, now that we're using 300Gb maximum rather than 2Tb partitions (running multiple instances of Cyrus instead) with the associated smaller mailboxes.db (the biggest MMAPed and frequently updated file) things seem less edgy. I don't like edgy (queue Ubuntu jokes). Anyway, I'm hoping to update one of our boxes to 2.6.19.2 soon. We do have one box running a 2.6.18 series kernel which has been fine as well. I'll give feedback if we see any issues with MMAP on there. Bron. - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: How many people are using 2.6.16?
On Mon, Jan 29, 2007 at 04:04:48PM -0500, Mike Houston wrote: > On Mon, 29 Jan 2007 15:30:00 -0500 > Chuck Ebbert <[EMAIL PROTECTED]> wrote: > > > Is there any way to estimate the size of the user base for 2.6.16? > > > > e.g. how many downloads does it get? > > I've often wondered that myself, as I'm concerned for it to continue > to be maintained. I'm very appreciative of what Adrian is doing with > it. (Thanks!) We're still running 2.6.16 kernels on a bunch of machines, though 2.6.19 has been looking pretty nice on the couple of machines that are testing it. 2.6.17 and 2.6.18 felt less stable. We do a lot of Cyrus which does a lot of MMAP - and we also use the Areca driver - which are both strong reasons to move to 2.6.19.2, but if the MMAP fix was ported back to 2.6.16 we might consider staying there instead. Bron. - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: How many people are using 2.6.16?
On Mon, Jan 29, 2007 at 04:04:48PM -0500, Mike Houston wrote: On Mon, 29 Jan 2007 15:30:00 -0500 Chuck Ebbert [EMAIL PROTECTED] wrote: Is there any way to estimate the size of the user base for 2.6.16? e.g. how many downloads does it get? I've often wondered that myself, as I'm concerned for it to continue to be maintained. I'm very appreciative of what Adrian is doing with it. (Thanks!) We're still running 2.6.16 kernels on a bunch of machines, though 2.6.19 has been looking pretty nice on the couple of machines that are testing it. 2.6.17 and 2.6.18 felt less stable. We do a lot of Cyrus which does a lot of MMAP - and we also use the Areca driver - which are both strong reasons to move to 2.6.19.2, but if the MMAP fix was ported back to 2.6.16 we might consider staying there instead. Bron. - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: Areca driver 2.6.19 on x86_64
On Wed, Dec 06, 2006 at 05:51:32PM +0100, [EMAIL PROTECTED] wrote: > OS distro used: > CentOS 4.4 x86_64 > Kernel 2.6.19 with hand-crafted config, that we are > able to use successfully with kernel 2.6.16.20. What patches were you applying to 2.6.16.20, since that didn't have an Areca driver in it I presume you're at least using that. > Have you any suggestions to resolve this issue ? 32 bit kernel? I'm somewhat serious here, depending what applications you're running on the machine. Otherwise, no clue sorry - we're running 32 bit kernels everywhere even on the couple of new Core machines we have. Bron. - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: Areca driver 2.6.19 on x86_64
On Wed, Dec 06, 2006 at 05:51:32PM +0100, [EMAIL PROTECTED] wrote: OS distro used: CentOS 4.4 x86_64 Kernel 2.6.19 with hand-crafted config, that we are able to use successfully with kernel 2.6.16.20. What patches were you applying to 2.6.16.20, since that didn't have an Areca driver in it I presume you're at least using that. Have you any suggestions to resolve this issue ? 32 bit kernel? I'm somewhat serious here, depending what applications you're running on the machine. Otherwise, no clue sorry - we're running 32 bit kernels everywhere even on the couple of new Core machines we have. Bron. - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: Pathetic write performance from Areca PCIe cards
On Mon, Nov 27, 2006 at 11:34:23AM +0800, erich wrote: > Dear Maurice Volaski, > > Please update Areca Firmware version into 1.42. > Areca's firmware team found some problems on high capacity transfer. > Hope the weird phenomenon should disappear. Erich, is there anyone at Areca that you can pass on the message to ++ | Please update your ftp server/website when | | there is a new firmware or driver release! | ++ that would be great. I followed the links from www.areca.us to the firmware at: ftp://ftp.areca.com.tw/RaidCards/BIOS_Firmware/ARC1130/ for our cards, but the 1210 and 1220 that Maurice was speaking about suffer from the same problem - there is no mention of a 1.42 firmware anywhere, just the 1.41 that's been out for ages. ... And speaking of 1.41, there appear to have been two releases on two different dates both called 1.41, as well as two different versions of the driver that both call themselves version 1.41 despite the second one fixing a major bug we suffered from. Please also avoid that behaviour and label each new version of the driver with a new number if you're using version numbers. Numbers are cheap, but identifying if a machine is running the patches it needs to not crash every few weeks under the loads we run them at is not (well, not until it crashes anyway!) Thanks for listening, and hopefully thanks in advance for making your drivers and firmware easier to find and identify in future. Regards, Bron. - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: Pathetic write performance from Areca PCIe cards
On Mon, Nov 27, 2006 at 11:34:23AM +0800, erich wrote: Dear Maurice Volaski, Please update Areca Firmware version into 1.42. Areca's firmware team found some problems on high capacity transfer. Hope the weird phenomenon should disappear. Erich, is there anyone at Areca that you can pass on the message to ++ | Please update your ftp server/website when | | there is a new firmware or driver release! | ++ that would be great. I followed the links from www.areca.us to the firmware at: ftp://ftp.areca.com.tw/RaidCards/BIOS_Firmware/ARC1130/ for our cards, but the 1210 and 1220 that Maurice was speaking about suffer from the same problem - there is no mention of a 1.42 firmware anywhere, just the 1.41 that's been out for ages. ... And speaking of 1.41, there appear to have been two releases on two different dates both called 1.41, as well as two different versions of the driver that both call themselves version 1.41 despite the second one fixing a major bug we suffered from. Please also avoid that behaviour and label each new version of the driver with a new number if you're using version numbers. Numbers are cheap, but identifying if a machine is running the patches it needs to not crash every few weeks under the loads we run them at is not (well, not until it crashes anyway!) Thanks for listening, and hopefully thanks in advance for making your drivers and firmware easier to find and identify in future. Regards, Bron. - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: 2.6.12.2 dies after 24 hours
On Tue, 12 Jul 2005 14:13:01 +0200, "Lars Roland" <[EMAIL PROTECTED]> said: > You have irq balancing, the line > > CONFIG_IRQBALANCE=y > > in your config file confirms it - I am not completely sure that it is > the root of the problem but when I experienced the problem I changed > two things: my acpi code and irq balancing and one of then made the > difference, I am just to lazy to check which one it is (also it is > production servers so I cannot do whatever I want). Our ACPI looked very similar to yours, so I've disabled IRQBALANCE. We'll be rebooting the server during a less busy time to try the new kernel, so not for about 12 hours or so. > > append="elevator=deadline" > > I use the same io scheduler so that should not be a problem. I have > uploaded my config file - it works on ibm 335/336 servers, and a quick > look at your boot msg seams to indicate that your server have some of > the same hardware - note however that I load ide/scsi/filesystem stuff > as modules so you will need to build a initrd to use my config. > > the config is here > > http://randompage.org/static/kernel.conf Great, thanks for that. I've had a look through and I think the IRQBALANCE issue is the most likely cause. We're also applying the attached patch. There's a bug in reiserfs that gets tickled by our huge MMAP usage (it's amazing what really busy Cyrus daemons can do to a server, ouch). It's fixed in generic_write, so we take the few percent performance hit for something that doesn't break! Bron. -- Bron Gondwana [EMAIL PROTECTED] patch-2.6.12.2-reiserfix.bz2 Description: Binary data
Re: 2.6.12.2 dies after 24 hours
On Tue, 12 Jul 2005 14:13:01 +0200, Lars Roland [EMAIL PROTECTED] said: You have irq balancing, the line CONFIG_IRQBALANCE=y in your config file confirms it - I am not completely sure that it is the root of the problem but when I experienced the problem I changed two things: my acpi code and irq balancing and one of then made the difference, I am just to lazy to check which one it is (also it is production servers so I cannot do whatever I want). Our ACPI looked very similar to yours, so I've disabled IRQBALANCE. We'll be rebooting the server during a less busy time to try the new kernel, so not for about 12 hours or so. append=elevator=deadline I use the same io scheduler so that should not be a problem. I have uploaded my config file - it works on ibm 335/336 servers, and a quick look at your boot msg seams to indicate that your server have some of the same hardware - note however that I load ide/scsi/filesystem stuff as modules so you will need to build a initrd to use my config. the config is here http://randompage.org/static/kernel.conf Great, thanks for that. I've had a look through and I think the IRQBALANCE issue is the most likely cause. We're also applying the attached patch. There's a bug in reiserfs that gets tickled by our huge MMAP usage (it's amazing what really busy Cyrus daemons can do to a server, ouch). It's fixed in generic_write, so we take the few percent performance hit for something that doesn't break! Bron. -- Bron Gondwana [EMAIL PROTECTED] patch-2.6.12.2-reiserfix.bz2 Description: Binary data
