Re: data corruption with nvidia chipsets and IDE/SATA drives // memory hole mapping related bug?!

[Dan Halbert]

I'd like to mention what might be a new twist on this problem. We are 
seeing the same kind of 4k-block data corruption on multiple Tyan 
dual-Opteron boards (S3870) with a ServerWorks chipset, not Nvidia. I 
wonder if it really an Nvidia-specific issue. The Nvidia boards are a 
lot more popular, so maybe it's just more likely to be seen with Nvidia. 
I added a more detailed comment about this to the kernel bug report: 
http://bugzilla.kernel.org/show_bug.cgi?id=7768#c15.


-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: data corruption with nvidia chipsets and IDE/SATA drives // memory hole mapping related bug?!

[Christoph Anton Mitterer]

Hi folks.

1) Are there any new developments in this issue? Does someone know if
AMD and Nvidia is still investigating?

2) Steve Langasek from Debian sent me a patch that disables the hw-iommu
per default on Nvidia boards.
I've attached it in the kernel bugzilla and asked for inclusion in the
kernel (until we find a real solution).
I'd be pleased if all of you (who experienced the data corruption) could
test this patch.
Note: This patch is NOT a real solution for the issue, it just applies
our workaround (iommu=soft) per default

Bugreport at kernel.org: http://bugzilla.kernel.org/show_bug.cgi?id=7768
Bugreport at Debian: http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=404148

Best wishes,
Chris.
begin:vcard
fn:Mitterer, Christoph Anton
n:Mitterer;Christoph Anton
email;internet:[EMAIL PROTECTED]
x-mozilla-html:TRUE
version:2.1
end:vcard

Re: data corruption with nvidia chipsets and IDE/SATA drives // memory hole mapping related bug?!

[Christoph Anton Mitterer]

Hi folks.

1) Are there any new developments in this issue? Does someone know if
AMD and Nvidia is still investigating?

2) Steve Langasek from Debian sent me a patch that disables the hw-iommu
per default on Nvidia boards.
I've attached it in the kernel bugzilla and asked for inclusion in the
kernel (until we find a real solution).
I'd be pleased if all of you (who experienced the data corruption) could
test this patch.
Note: This patch is NOT a real solution for the issue, it just applies
our workaround (iommu=soft) per default

Bugreport at kernel.org: http://bugzilla.kernel.org/show_bug.cgi?id=7768
Bugreport at Debian: http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=404148

Best wishes,
Chris.
begin:vcard
fn:Mitterer, Christoph Anton
n:Mitterer;Christoph Anton
email;internet:[EMAIL PROTECTED]
x-mozilla-html:TRUE
version:2.1
end:vcard

Re: data corruption with nvidia chipsets and IDE/SATA drives // memory hole mapping related bug?!

[Dan Halbert]

I'd like to mention what might be a new twist on this problem. We are 
seeing the same kind of 4k-block data corruption on multiple Tyan 
dual-Opteron boards (S3870) with a ServerWorks chipset, not Nvidia. I 
wonder if it really an Nvidia-specific issue. The Nvidia boards are a 
lot more popular, so maybe it's just more likely to be seen with Nvidia. 
I added a more detailed comment about this to the kernel bug report: 
http://bugzilla.kernel.org/show_bug.cgi?id=7768#c15.


-
To unsubscribe from this list: send the line unsubscribe linux-kernel in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: data corruption with nvidia chipsets and IDE/SATA drives // memory hole mapping related bug?!

[Robert Hancock]

Christoph Anton Mitterer wrote:

Ok,.. that sounds reasonable,.. so the whole thing might (!) actually be
a hardware design error,... but we just don't use that hardware any
longer when accessing devices via sata_nv.

So this doesn't solve our problem with PATA drives or other devices
(although we had until now no reports of errors with other devices) and
we have to stick with iommu=soft.

If one use iommu=soft the sata_nv will continue to use the new code for
the ADMA, right?


Right, that shouldn't affect it.
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: data corruption with nvidia chipsets and IDE/SATA drives // memory hole mapping related bug?!

[Christoph Anton Mitterer]

Robert Hancock wrote:
>> What is that GART thing exactly? Is this the hardware IOMMU? I've always
>> thought GART was something graphics card related,.. but if so,.. how
>> could this solve our problem (that seems to occur mainly on harddisks)?
>> 
> The GART built into the Athlon 64/Opteron CPUs is normally used for 
> remapping graphics memory so that an AGP graphics card can see 
> physically non-contiguous memory as one contiguous region. However, 
> Linux can also use it as an IOMMU which allows devices which normally 
> can't access memory above 4GB to see a mapping of that memory that 
> resides below 4GB. In pre-2.6.20 kernels both the SATA and PATA 
> controllers on the nForce 4 chipsets can only access memory below 4GB so 
> transfers to memory above this mark have to go through the IOMMU. In 
> 2.6.20 this limitation is lifted on the nForce4 SATA controllers.
>   
Ah, I see. Thanks for that introduction :-)


>> Does this mean that PATA is no related? The corruption appears on PATA
>> disks to, so why should it only solve the issue at SATA disks? Sounds a
>> bit strange to me?
>> 
> The PATA controller will still be using 32-bit DMA and so may also use 
> the IOMMU, so this problem would not be avoided.
>   
>   
>> Can you explain this a little bit more please? Is this a drawback (like
>> a performance decrease)? Like under Windows where they never use the
>> hardware iommu but always do it via software?
>> 
>
> No, it shouldn't cause any performance loss. In previous kernels the 
> nForce4 SATA controller was controlled using an interface quite similar 
> to a PATA controller. In 2.6.20 kernels they use a more efficient 
> interface that NVidia calls ADMA, which in addition to supporting NCQ 
> also supports DMA without any 4GB limitations, so it can access all 
> memory directly without requiring IOMMU assistance.
>
> Note that if this corruption problem is, as has been suggested, related 
> to memory hole remapping and the IOMMU, then this change only prevents 
> the SATA controller transfers from experiencing this problem. Transfers 
> on the PATA controller as well as any other devices with 32-bit DMA 
> limitations might still have problems. As such this really just avoids 
> the problem, not fixes it.
>   
Ok,.. that sounds reasonable,.. so the whole thing might (!) actually be
a hardware design error,... but we just don't use that hardware any
longer when accessing devices via sata_nv.

So this doesn't solve our problem with PATA drives or other devices
(although we had until now no reports of errors with other devices) and
we have to stick with iommu=soft.

If one use iommu=soft the sata_nv will continue to use the new code for
the ADMA, right?


Best wishes,
Chris.
begin:vcard
fn:Mitterer, Christoph Anton
n:Mitterer;Christoph Anton
email;internet:[EMAIL PROTECTED]
x-mozilla-html:TRUE
version:2.1
end:vcard

Re: data corruption with nvidia chipsets and IDE/SATA drives // memory hole mapping related bug?!

[Christoph Anton Mitterer]

Robert Hancock wrote:
 What is that GART thing exactly? Is this the hardware IOMMU? I've always
 thought GART was something graphics card related,.. but if so,.. how
 could this solve our problem (that seems to occur mainly on harddisks)?
 
 The GART built into the Athlon 64/Opteron CPUs is normally used for 
 remapping graphics memory so that an AGP graphics card can see 
 physically non-contiguous memory as one contiguous region. However, 
 Linux can also use it as an IOMMU which allows devices which normally 
 can't access memory above 4GB to see a mapping of that memory that 
 resides below 4GB. In pre-2.6.20 kernels both the SATA and PATA 
 controllers on the nForce 4 chipsets can only access memory below 4GB so 
 transfers to memory above this mark have to go through the IOMMU. In 
 2.6.20 this limitation is lifted on the nForce4 SATA controllers.
   
Ah, I see. Thanks for that introduction :-)


 Does this mean that PATA is no related? The corruption appears on PATA
 disks to, so why should it only solve the issue at SATA disks? Sounds a
 bit strange to me?
 
 The PATA controller will still be using 32-bit DMA and so may also use 
 the IOMMU, so this problem would not be avoided.
   
   
 Can you explain this a little bit more please? Is this a drawback (like
 a performance decrease)? Like under Windows where they never use the
 hardware iommu but always do it via software?
 

 No, it shouldn't cause any performance loss. In previous kernels the 
 nForce4 SATA controller was controlled using an interface quite similar 
 to a PATA controller. In 2.6.20 kernels they use a more efficient 
 interface that NVidia calls ADMA, which in addition to supporting NCQ 
 also supports DMA without any 4GB limitations, so it can access all 
 memory directly without requiring IOMMU assistance.

 Note that if this corruption problem is, as has been suggested, related 
 to memory hole remapping and the IOMMU, then this change only prevents 
 the SATA controller transfers from experiencing this problem. Transfers 
 on the PATA controller as well as any other devices with 32-bit DMA 
 limitations might still have problems. As such this really just avoids 
 the problem, not fixes it.
   
Ok,.. that sounds reasonable,.. so the whole thing might (!) actually be
a hardware design error,... but we just don't use that hardware any
longer when accessing devices via sata_nv.

So this doesn't solve our problem with PATA drives or other devices
(although we had until now no reports of errors with other devices) and
we have to stick with iommu=soft.

If one use iommu=soft the sata_nv will continue to use the new code for
the ADMA, right?


Best wishes,
Chris.
begin:vcard
fn:Mitterer, Christoph Anton
n:Mitterer;Christoph Anton
email;internet:[EMAIL PROTECTED]
x-mozilla-html:TRUE
version:2.1
end:vcard

Re: data corruption with nvidia chipsets and IDE/SATA drives // memory hole mapping related bug?!

[Robert Hancock]

Christoph Anton Mitterer wrote:

Ok,.. that sounds reasonable,.. so the whole thing might (!) actually be
a hardware design error,... but we just don't use that hardware any
longer when accessing devices via sata_nv.

So this doesn't solve our problem with PATA drives or other devices
(although we had until now no reports of errors with other devices) and
we have to stick with iommu=soft.

If one use iommu=soft the sata_nv will continue to use the new code for
the ADMA, right?


Right, that shouldn't affect it.
-
To unsubscribe from this list: send the line unsubscribe linux-kernel in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: data corruption with nvidia chipsets and IDE/SATA drives // memory hole mapping related bug?!

[Robert Hancock]

Christoph Anton Mitterer wrote:

Sorry, as always I've forgot some things... *g*


Robert Hancock wrote:

If this is related to some problem with using the GART IOMMU with memory 
hole remapping enabled

What is that GART thing exactly? Is this the hardware IOMMU? I've always
thought GART was something graphics card related,.. but if so,.. how
could this solve our problem (that seems to occur mainly on harddisks)?


The GART built into the Athlon 64/Opteron CPUs is normally used for 
remapping graphics memory so that an AGP graphics card can see 
physically non-contiguous memory as one contiguous region. However, 
Linux can also use it as an IOMMU which allows devices which normally 
can't access memory above 4GB to see a mapping of that memory that 
resides below 4GB. In pre-2.6.20 kernels both the SATA and PATA 
controllers on the nForce 4 chipsets can only access memory below 4GB so 
transfers to memory above this mark have to go through the IOMMU. In 
2.6.20 this limitation is lifted on the nForce4 SATA controllers.




then 2.6.20-rc kernels may avoid this problem on 
nForce4 CK804/MCP04 chipsets as far as transfers to/from the SATA 
controller are concerned

Does this mean that PATA is no related? The corruption appears on PATA
disks to, so why should it only solve the issue at SATA disks? Sounds a
bit strange to me?


The PATA controller will still be using 32-bit DMA and so may also use 
the IOMMU, so this problem would not be avoided.




as the sata_nv driver now supports 64-bit DMA 
on these chipsets and so no longer requires the IOMMU.
  

Can you explain this a little bit more please? Is this a drawback (like
a performance decrease)? Like under Windows where they never use the
hardware iommu but always do it via software?


No, it shouldn't cause any performance loss. In previous kernels the 
nForce4 SATA controller was controlled using an interface quite similar 
to a PATA controller. In 2.6.20 kernels they use a more efficient 
interface that NVidia calls ADMA, which in addition to supporting NCQ 
also supports DMA without any 4GB limitations, so it can access all 
memory directly without requiring IOMMU assistance.


Note that if this corruption problem is, as has been suggested, related 
to memory hole remapping and the IOMMU, then this change only prevents 
the SATA controller transfers from experiencing this problem. Transfers 
on the PATA controller as well as any other devices with 32-bit DMA 
limitations might still have problems. As such this really just avoids 
the problem, not fixes it.


--
Robert Hancock  Saskatoon, SK, Canada
To email, remove "nospam" from [EMAIL PROTECTED]
Home Page: http://www.roberthancock.com/


-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: data corruption with nvidia chipsets and IDE/SATA drives // memory hole mapping related bug?!

[Christoph Anton Mitterer]

Sorry, as always I've forgot some things... *g*


Robert Hancock wrote:

> If this is related to some problem with using the GART IOMMU with memory 
> hole remapping enabled
What is that GART thing exactly? Is this the hardware IOMMU? I've always
thought GART was something graphics card related,.. but if so,.. how
could this solve our problem (that seems to occur mainly on harddisks)?

> then 2.6.20-rc kernels may avoid this problem on 
> nForce4 CK804/MCP04 chipsets as far as transfers to/from the SATA 
> controller are concerned
Does this mean that PATA is no related? The corruption appears on PATA
disks to, so why should it only solve the issue at SATA disks? Sounds a
bit strange to me?

> as the sata_nv driver now supports 64-bit DMA 
> on these chipsets and so no longer requires the IOMMU.
>   
Can you explain this a little bit more please? Is this a drawback (like
a performance decrease)? Like under Windows where they never use the
hardware iommu but always do it via software?


Best wishes,
Chris.
begin:vcard
fn:Mitterer, Christoph Anton
n:Mitterer;Christoph Anton
email;internet:[EMAIL PROTECTED]
x-mozilla-html:TRUE
version:2.1
end:vcard

Re: data corruption with nvidia chipsets and IDE/SATA drives // memory hole mapping related bug?!

[Christoph Anton Mitterer]

Hi everybody.

Sorry again for my late reply...

Robert gave us the following interesting information some days ago:

Robert Hancock wrote:
> If this is related to some problem with using the GART IOMMU with memory 
> hole remapping enabled, then 2.6.20-rc kernels may avoid this problem on 
> nForce4 CK804/MCP04 chipsets as far as transfers to/from the SATA 
> controller are concerned as the sata_nv driver now supports 64-bit DMA 
> on these chipsets and so no longer requires the IOMMU.
>   


I've just tested it with my "normal" BIOS settings, that is memhole
mapping = hardware, IOMMU = enabled and 64MB and _without_ (!)
iommu=soft as kernel parameters.
I only had the time for a small test (that is 3 passes with each 10
complete sha512sums cyles over about 30GB data)... but sofar, no
corruption occured.

It is surely far to eraly to tell that our issue was solved by
2.6.20-rc-something but I ask all of you that had systems that
suffered from the corruption to make _intensive_ tests with the most
recent rc of 2.6.20 (I've used 2.6.20-rc5) and report your results.
I'll do a extensive test tomorrow.

And of course (!!): Test without using iommu=soft and with enabled
memhole mapping (in the BIOS). (It won't make any sense to look if the
new kernel solves our problem while still applying one of our two
workarounds).


Please also note that there might be two completely data corruption
problems. The onle "solved" by iommu=soft and another reported by Kurtis
D. Rader.
I've asked him to clarify this in a post. :-)



Ok,... now if this (the new kernel) would really solve the issue... we
should try to find out what exactly was changed in the code, and if it
sounds logical that this solved the problem or not.
The new kernel could just make the corruption even more rare.


Best wishes,
Chris.


begin:vcard
fn:Mitterer, Christoph Anton
n:Mitterer;Christoph Anton
email;internet:[EMAIL PROTECTED]
x-mozilla-html:TRUE
version:2.1
end:vcard

Re: data corruption with nvidia chipsets and IDE/SATA drives // memory hole mapping related bug?!

[Christoph Anton Mitterer]

Hi.

Some days ago I received the following message from "Sunny Days". I
think he did not send it lkml so I forward it now:

Sunny Days wrote:
> hello,
>
> i have done some extensive testing on this.
>
> various opterons, always single socket
> various dimms 1 and 2gb modules
> and hitachi+seagate disks with various firmwares and sizes
> but i am getting a diferent pattern in the corruption.
> My test file was 10gb.
>
> I have mapped the earliest corruption as low as 10mb in the written data.
> i have also monitor the adress range used from the cp /md5sum proccess
> under /proc//$PID/maps to see if i could find a pattern but i was
> unable to.
>
> i also tested ext2 and lvm with similar results aka corruption.
> later on the week i should get a pci promise controller and test on that one.
>
> Things i have not tested is the patch that linus released 10 days ago
> and reiserfs3/4
>
> my nvidia chipset was ck804 (a3)
>
> Hope somehow we get to the bottom of this.
>
> Hope this helps
>
>
> btw amd erratas that could possible influence this are
>
> 115, 123, 156 with the latter been fascinating as it the workaround
> suggested is 0x0 page entry.
>
>   

Does anyone has any opinions about this? Could you please read the
mentioned erratas and tell me what you think?

Best wishes,
Chris.

@ Sunny Days: Thanks for you mail.
begin:vcard
fn:Mitterer, Christoph Anton
n:Mitterer;Christoph Anton
email;internet:[EMAIL PROTECTED]
x-mozilla-html:TRUE
version:2.1
end:vcard

Re: data corruption with nvidia chipsets and IDE/SATA drives // memory hole mapping related bug?!

[Christoph Anton Mitterer]

Hi.

Some days ago I received the following message from Sunny Days. I
think he did not send it lkml so I forward it now:

Sunny Days wrote:
 hello,

 i have done some extensive testing on this.

 various opterons, always single socket
 various dimms 1 and 2gb modules
 and hitachi+seagate disks with various firmwares and sizes
 but i am getting a diferent pattern in the corruption.
 My test file was 10gb.

 I have mapped the earliest corruption as low as 10mb in the written data.
 i have also monitor the adress range used from the cp /md5sum proccess
 under /proc//$PID/maps to see if i could find a pattern but i was
 unable to.

 i also tested ext2 and lvm with similar results aka corruption.
 later on the week i should get a pci promise controller and test on that one.

 Things i have not tested is the patch that linus released 10 days ago
 and reiserfs3/4

 my nvidia chipset was ck804 (a3)

 Hope somehow we get to the bottom of this.

 Hope this helps


 btw amd erratas that could possible influence this are

 115, 123, 156 with the latter been fascinating as it the workaround
 suggested is 0x0 page entry.

   

Does anyone has any opinions about this? Could you please read the
mentioned erratas and tell me what you think?

Best wishes,
Chris.

@ Sunny Days: Thanks for you mail.
begin:vcard
fn:Mitterer, Christoph Anton
n:Mitterer;Christoph Anton
email;internet:[EMAIL PROTECTED]
x-mozilla-html:TRUE
version:2.1
end:vcard

Re: data corruption with nvidia chipsets and IDE/SATA drives // memory hole mapping related bug?!

[Christoph Anton Mitterer]

Hi everybody.

Sorry again for my late reply...

Robert gave us the following interesting information some days ago:

Robert Hancock wrote:
 If this is related to some problem with using the GART IOMMU with memory 
 hole remapping enabled, then 2.6.20-rc kernels may avoid this problem on 
 nForce4 CK804/MCP04 chipsets as far as transfers to/from the SATA 
 controller are concerned as the sata_nv driver now supports 64-bit DMA 
 on these chipsets and so no longer requires the IOMMU.
   


I've just tested it with my normal BIOS settings, that is memhole
mapping = hardware, IOMMU = enabled and 64MB and _without_ (!)
iommu=soft as kernel parameters.
I only had the time for a small test (that is 3 passes with each 10
complete sha512sums cyles over about 30GB data)... but sofar, no
corruption occured.

It is surely far to eraly to tell that our issue was solved by
2.6.20-rc-something but I ask all of you that had systems that
suffered from the corruption to make _intensive_ tests with the most
recent rc of 2.6.20 (I've used 2.6.20-rc5) and report your results.
I'll do a extensive test tomorrow.

And of course (!!): Test without using iommu=soft and with enabled
memhole mapping (in the BIOS). (It won't make any sense to look if the
new kernel solves our problem while still applying one of our two
workarounds).


Please also note that there might be two completely data corruption
problems. The onle solved by iommu=soft and another reported by Kurtis
D. Rader.
I've asked him to clarify this in a post. :-)



Ok,... now if this (the new kernel) would really solve the issue... we
should try to find out what exactly was changed in the code, and if it
sounds logical that this solved the problem or not.
The new kernel could just make the corruption even more rare.


Best wishes,
Chris.


begin:vcard
fn:Mitterer, Christoph Anton
n:Mitterer;Christoph Anton
email;internet:[EMAIL PROTECTED]
x-mozilla-html:TRUE
version:2.1
end:vcard

Re: data corruption with nvidia chipsets and IDE/SATA drives // memory hole mapping related bug?!

[Christoph Anton Mitterer]

Sorry, as always I've forgot some things... *g*


Robert Hancock wrote:

 If this is related to some problem with using the GART IOMMU with memory 
 hole remapping enabled
What is that GART thing exactly? Is this the hardware IOMMU? I've always
thought GART was something graphics card related,.. but if so,.. how
could this solve our problem (that seems to occur mainly on harddisks)?

 then 2.6.20-rc kernels may avoid this problem on 
 nForce4 CK804/MCP04 chipsets as far as transfers to/from the SATA 
 controller are concerned
Does this mean that PATA is no related? The corruption appears on PATA
disks to, so why should it only solve the issue at SATA disks? Sounds a
bit strange to me?

 as the sata_nv driver now supports 64-bit DMA 
 on these chipsets and so no longer requires the IOMMU.
   
Can you explain this a little bit more please? Is this a drawback (like
a performance decrease)? Like under Windows where they never use the
hardware iommu but always do it via software?


Best wishes,
Chris.
begin:vcard
fn:Mitterer, Christoph Anton
n:Mitterer;Christoph Anton
email;internet:[EMAIL PROTECTED]
x-mozilla-html:TRUE
version:2.1
end:vcard

Re: data corruption with nvidia chipsets and IDE/SATA drives // memory hole mapping related bug?!

[Robert Hancock]

Christoph Anton Mitterer wrote:

Sorry, as always I've forgot some things... *g*


Robert Hancock wrote:

If this is related to some problem with using the GART IOMMU with memory 
hole remapping enabled

What is that GART thing exactly? Is this the hardware IOMMU? I've always
thought GART was something graphics card related,.. but if so,.. how
could this solve our problem (that seems to occur mainly on harddisks)?


The GART built into the Athlon 64/Opteron CPUs is normally used for 
remapping graphics memory so that an AGP graphics card can see 
physically non-contiguous memory as one contiguous region. However, 
Linux can also use it as an IOMMU which allows devices which normally 
can't access memory above 4GB to see a mapping of that memory that 
resides below 4GB. In pre-2.6.20 kernels both the SATA and PATA 
controllers on the nForce 4 chipsets can only access memory below 4GB so 
transfers to memory above this mark have to go through the IOMMU. In 
2.6.20 this limitation is lifted on the nForce4 SATA controllers.




then 2.6.20-rc kernels may avoid this problem on 
nForce4 CK804/MCP04 chipsets as far as transfers to/from the SATA 
controller are concerned

Does this mean that PATA is no related? The corruption appears on PATA
disks to, so why should it only solve the issue at SATA disks? Sounds a
bit strange to me?


The PATA controller will still be using 32-bit DMA and so may also use 
the IOMMU, so this problem would not be avoided.




as the sata_nv driver now supports 64-bit DMA 
on these chipsets and so no longer requires the IOMMU.
  

Can you explain this a little bit more please? Is this a drawback (like
a performance decrease)? Like under Windows where they never use the
hardware iommu but always do it via software?


No, it shouldn't cause any performance loss. In previous kernels the 
nForce4 SATA controller was controlled using an interface quite similar 
to a PATA controller. In 2.6.20 kernels they use a more efficient 
interface that NVidia calls ADMA, which in addition to supporting NCQ 
also supports DMA without any 4GB limitations, so it can access all 
memory directly without requiring IOMMU assistance.


Note that if this corruption problem is, as has been suggested, related 
to memory hole remapping and the IOMMU, then this change only prevents 
the SATA controller transfers from experiencing this problem. Transfers 
on the PATA controller as well as any other devices with 32-bit DMA 
limitations might still have problems. As such this really just avoids 
the problem, not fixes it.


--
Robert Hancock  Saskatoon, SK, Canada
To email, remove nospam from [EMAIL PROTECTED]
Home Page: http://www.roberthancock.com/


-
To unsubscribe from this list: send the line unsubscribe linux-kernel in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: data corruption with nvidia chipsets and IDE/SATA drives // memory hole mapping related bug?!

[Christoph Anton Mitterer]

Hi.

Just for you information: I've put the issue into the kernel.org bugzilla.
http://bugzilla.kernel.org/show_bug.cgi?id=7768

Chris.
begin:vcard
fn:Mitterer, Christoph Anton
n:Mitterer;Christoph Anton
email;internet:[EMAIL PROTECTED]
x-mozilla-html:TRUE
version:2.1
end:vcard

Re: data corruption with nvidia chipsets and IDE/SATA drives // memory hole mapping related bug?!

[Christoph Anton Mitterer]

Hi.

Just for you information: I've put the issue into the kernel.org bugzilla.
http://bugzilla.kernel.org/show_bug.cgi?id=7768

Chris.
begin:vcard
fn:Mitterer, Christoph Anton
n:Mitterer;Christoph Anton
email;internet:[EMAIL PROTECTED]
x-mozilla-html:TRUE
version:2.1
end:vcard

Re: data corruption with nvidia chipsets and IDE/SATA drives // memory hole mapping related bug?!

[Robert Hancock]

Christoph Anton Mitterer wrote:

Hi.

Perhaps some of you have read my older two threads:
http://marc.theaimsgroup.com/?t=11631244001=1=2 and the even
older http://marc.theaimsgroup.com/?t=11629131451=1=2

The issue was basically the following:
I found a severe bug mainly by fortune because it occurs very rarely.
My test looks like the following: I have about 30GB of testing data on
my harddisk,... I repeat verifying sha512 sums on these files and check
if errors occur.
One test pass verifies the 30GB 50 times,... about one to four
differences are found in each pass.

The corrupted data is not one single completely wrong block of data or
so,.. but if you look at the area of the file where differences are
found,.. than some bytes are ok,.. some are wrong,.. and so on (seems to
be randomly).

Also, there seems to be no event that triggers the corruption,.. it
seems to be randomly, too.

It is really definitely not a harware issue (see my old threads my
emails to Tyan/Hitachi and my "workaround" below. My system isn't
overclocked.



My System:
Mainboard: Tyan S2895
Chipsets: Nvidia nforce professional 2200 and 2050 and AMD 8131
CPU: 2x DualCore Opterons model 275
RAM: 4GB Kingston Registered/ECC
Diskdrives: IBM/Hitachi: 1 PATA, 2 SATA


The data corruption error occurs on all drives.


You might have a look at the emails between me and Tyan and Hitachi,..
they contain probalby lots of valuable information (especially my
different tests).



Some days ago,.. an engineer of Tyan suggested me to boot the kernel
with mem=3072M.
When doing this,.. the issue did not occur (I don't want to say it was
solved. Why? See my last emails to Tyan!)
Then he suggested me to disable the memory hole mapping in the BIOS,...
When doing so,.. the error doesn't occur, too.
But I loose about 2GB RAM,.. and,.. more important,.. I cant believe
that this is responsible for the whole issue. I don't consider it a
solution but more a poor workaround which perhaps only by fortune solves
the issue (Why? See my last eMails to Tyan ;) )



So I'd like to ask you if you perhaps could read the current information
in this and previous mails,.. and tell me your opinions.
It is very likely that a large number of users suffer from this error
(namely all Nvidia chipset users) but only few (there are some,.. I
found most of them in the Nvidia forums,.. and they have exactly the
same issue) identify this as an error because it's so rare.

Perhaps someone have an idea why disabling the memhole mapping solves
it. I've always thought that memhole mapping just moves some address
space to higher addreses to avoid the conflict between address space for
PCI devices and address space for pyhsical memory.
But this should be just a simple addition and not solve this obviously
complex error.


If this is related to some problem with using the GART IOMMU with memory 
hole remapping enabled, then 2.6.20-rc kernels may avoid this problem on 
nForce4 CK804/MCP04 chipsets as far as transfers to/from the SATA 
controller are concerned as the sata_nv driver now supports 64-bit DMA 
on these chipsets and so no longer requires the IOMMU.


--
Robert Hancock  Saskatoon, SK, Canada
To email, remove "nospam" from [EMAIL PROTECTED]
Home Page: http://www.roberthancock.com/

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: data corruption with nvidia chipsets and IDE/SATA drives // memory hole mapping related bug?!

[Christoph Anton Mitterer]

Hi everybody.

After my last mails to this issue (btw: anything new in the meantime? I
received no replys..) I wrote again to nvidia and AMD...
This time with some more success.

Below is the answer from Mr. Friedman to my mail. He says that he wasn't
able to reproduce the problem and asks for a testing system.
Unfortunately I cannot ship my system as this is my only home PC and I
need it for daily work. But perhaps someone else here might has a system
(with the error) that he can send to Nvidia...

I cc'ed Mr. Friedman so he'll read your replies.

To Mr. Friedman: What system did you exactly use for your testing?
(Hardware configuration, BIOS settings and so on). As we've seen before
it might be possible that some BIOSes correct the problem.

Best wishes,
Chris.




Lonni J Friedman wrote:
> Christoph,
> Thanks for your email.  I'm aware of the LKML threads, and have spent 
> considerable time attempting to reproduce this problem on one of our 
> reference motherboards without success.  If you could ship a system 
> which reliably reproduces the problem, I'd be happy to investigate further.
>
> Thanks,
> Lonni J Friedman
> NVIDIA Corporation
>
> Christoph Anton Mitterer wrote:
>   
>> Hi.
>>
>> First of all: This is only a copy from a thread to nvnews.net
>> (http://www.nvnews.net/vbulletin/showthread.php?t=82909). You probably
>> should read the description there.
>>
>> Please note that his is also a very important issue. It is most likely
>> not only Linux related but a general nforce chipset design flaw, so
>> perhaps you should forwad this mail to your engineers too. (Please CC me
>> in all mails).
>>
>> Also note: I'm not one of the normal "end users" with simple problems or
>> damaged hardware. I study computer science and work in one of Europes
>> largest supercomputing centres (Leibniz supercomputing centre).
>> Believe me: I know what I'm talking about and I'm investigating in
>> this issue (with many others) for some weeks now.
>>
>> Please answer either to the specific lkml thread, to the nvnews.net post
>> or directly to me (via email).
>> And I'd be grateful if you could give me email-addresses from your
>> developers or enginers, or even better, forward this email to them and
>> CC me. Of course I'll keep their emails-addresses absolutely confident
>> if you wish.
>>
>> Best wishes,
>> Christoph Anton Mitterer.
>> Munich University of Applied Sciences / Department of Mathematics and
>> Computer Science
>> Leibniz Supercomputing Centre / Department for High Performance
>> Computing and Compute Servers
>>
>>
>>
>>
>> Here is the copy:
>> Hi.
>>
>> I've already tried to "resolve" this via the nvidia knowledgebase but
>> either they don't want to know about that issue or there is noone who is
>> competent enought to give information/solutions about it.
>> They finally pointed me to this fourm and told me that Linux
>>  support would
>> be handled here (they did not realise that this is probably a hardware
>>  flaw and not
>> OS related).
>>
>> I must admit that I'm a little bit bored with Nvidia's policy in such
>> matters and thus I only describe the problem in brief.
>> If here is any competent chipset engineer who reads this, than he might
>> read the main discussion-thread (and some spin-off threads) of the issue
>> which takes place at the linux-kernel mailing list (again this is
>> probably not Linux related).
>> You can find the archive here:
>> http://marc.theaimsgroup.com/?t=11650212181=1=2
>> 
>>
>>
>> Now a short description:
>> -I (and many others) found a data corruption issue that happens on AMD
>> Opteron / Nvidia chipset systems
>> .
>>
>> -What happens: If one reads/writes large amounts of data there are errors.
>> We test this the following way: Create some test data (huge amounts
>> of),.. make md5sums of it (or with other hash algorithms), then verify
>> them over and over.
>> The test shoes differences (refer the lkml thread for more information
>> about this). Always at differnt files (). It may happen at read AND
>> write access .
>> Note that even for affected users the error occurs rarely (but this is
>> of course still far to often): My personal tests shows about the following:
>> Test data: 30GB (of random data), I verify sha512sum 50 times (that is
>> what I call one complete test). So I verify 30*50GB. In one complete
>> test there are about 1-3 files with differences. With about 100
>> corrupted bytes (at leas very low data sizes, far below an MB)
>>
>> -It probably happens with all the nforce chipsets (see the lkml thread
>> where everybody tells his hardware)
>>
>> -The reasons are not single hardware defects (dozens of hight quality
>> memory 

Re: data corruption with nvidia chipsets and IDE/SATA drives // memory hole mapping related bug?!

[Christoph Anton Mitterer]

Hi everybody.

After my last mails to this issue (btw: anything new in the meantime? I
received no replys..) I wrote again to nvidia and AMD...
This time with some more success.

Below is the answer from Mr. Friedman to my mail. He says that he wasn't
able to reproduce the problem and asks for a testing system.
Unfortunately I cannot ship my system as this is my only home PC and I
need it for daily work. But perhaps someone else here might has a system
(with the error) that he can send to Nvidia...

I cc'ed Mr. Friedman so he'll read your replies.

To Mr. Friedman: What system did you exactly use for your testing?
(Hardware configuration, BIOS settings and so on). As we've seen before
it might be possible that some BIOSes correct the problem.

Best wishes,
Chris.




Lonni J Friedman wrote:
 Christoph,
 Thanks for your email.  I'm aware of the LKML threads, and have spent 
 considerable time attempting to reproduce this problem on one of our 
 reference motherboards without success.  If you could ship a system 
 which reliably reproduces the problem, I'd be happy to investigate further.

 Thanks,
 Lonni J Friedman
 NVIDIA Corporation

 Christoph Anton Mitterer wrote:
   
 Hi.

 First of all: This is only a copy from a thread to nvnews.net
 (http://www.nvnews.net/vbulletin/showthread.php?t=82909). You probably
 should read the description there.

 Please note that his is also a very important issue. It is most likely
 not only Linux related but a general nforce chipset design flaw, so
 perhaps you should forwad this mail to your engineers too. (Please CC me
 in all mails).

 Also note: I'm not one of the normal end users with simple problems or
 damaged hardware. I study computer science and work in one of Europes
 largest supercomputing centres (Leibniz supercomputing centre).
 Believe me: I know what I'm talking about and I'm investigating in
 this issue (with many others) for some weeks now.

 Please answer either to the specific lkml thread, to the nvnews.net post
 or directly to me (via email).
 And I'd be grateful if you could give me email-addresses from your
 developers or enginers, or even better, forward this email to them and
 CC me. Of course I'll keep their emails-addresses absolutely confident
 if you wish.

 Best wishes,
 Christoph Anton Mitterer.
 Munich University of Applied Sciences / Department of Mathematics and
 Computer Science
 Leibniz Supercomputing Centre / Department for High Performance
 Computing and Compute Servers




 Here is the copy:
 Hi.

 I've already tried to resolve this via the nvidia knowledgebase but
 either they don't want to know about that issue or there is noone who is
 competent enought to give information/solutions about it.
 They finally pointed me to this fourm and told me that Linux
 http://www.nvnews.net/vbulletin/showthread.php?t=82909# support would
 be handled here (they did not realise that this is probably a hardware
 http://www.nvnews.net/vbulletin/showthread.php?t=82909# flaw and not
 OS related).

 I must admit that I'm a little bit bored with Nvidia's policy in such
 matters and thus I only describe the problem in brief.
 If here is any competent chipset engineer who reads this, than he might
 read the main discussion-thread (and some spin-off threads) of the issue
 which takes place at the linux-kernel mailing list (again this is
 probably not Linux related).
 You can find the archive here:
 http://marc.theaimsgroup.com/?t=11650212181r=1w=2
 http://marc.theaimsgroup.com/?t=11650212181r=1w=2


 Now a short description:
 -I (and many others) found a data corruption issue that happens on AMD
 Opteron / Nvidia chipset systems
 http://www.nvnews.net/vbulletin/showthread.php?t=82909#.

 -What happens: If one reads/writes large amounts of data there are errors.
 We test this the following way: Create some test data (huge amounts
 of),.. make md5sums of it (or with other hash algorithms), then verify
 them over and over.
 The test shoes differences (refer the lkml thread for more information
 about this). Always at differnt files (). It may happen at read AND
 write access http://www.nvnews.net/vbulletin/showthread.php?t=82909#.
 Note that even for affected users the error occurs rarely (but this is
 of course still far to often): My personal tests shows about the following:
 Test data: 30GB (of random data), I verify sha512sum 50 times (that is
 what I call one complete test). So I verify 30*50GB. In one complete
 test there are about 1-3 files with differences. With about 100
 corrupted bytes (at leas very low data sizes, far below an MB)

 -It probably happens with all the nforce chipsets (see the lkml thread
 where everybody tells his hardware)

 -The reasons are not single hardware defects (dozens of hight quality
 memory http://www.nvnews.net/vbulletin/showthread.php?t=82909#, CPU,
 PCI bus, HDD bad block scans, PCI parity, ECC, etc. tests showed this,
 and even with different hardware compontents the issue remained)

 -It is 

Re: data corruption with nvidia chipsets and IDE/SATA drives // memory hole mapping related bug?!

[Robert Hancock]

Christoph Anton Mitterer wrote:

Hi.

Perhaps some of you have read my older two threads:
http://marc.theaimsgroup.com/?t=11631244001r=1w=2 and the even
older http://marc.theaimsgroup.com/?t=11629131451r=1w=2

The issue was basically the following:
I found a severe bug mainly by fortune because it occurs very rarely.
My test looks like the following: I have about 30GB of testing data on
my harddisk,... I repeat verifying sha512 sums on these files and check
if errors occur.
One test pass verifies the 30GB 50 times,... about one to four
differences are found in each pass.

The corrupted data is not one single completely wrong block of data or
so,.. but if you look at the area of the file where differences are
found,.. than some bytes are ok,.. some are wrong,.. and so on (seems to
be randomly).

Also, there seems to be no event that triggers the corruption,.. it
seems to be randomly, too.

It is really definitely not a harware issue (see my old threads my
emails to Tyan/Hitachi and my workaround below. My system isn't
overclocked.



My System:
Mainboard: Tyan S2895
Chipsets: Nvidia nforce professional 2200 and 2050 and AMD 8131
CPU: 2x DualCore Opterons model 275
RAM: 4GB Kingston Registered/ECC
Diskdrives: IBM/Hitachi: 1 PATA, 2 SATA


The data corruption error occurs on all drives.


You might have a look at the emails between me and Tyan and Hitachi,..
they contain probalby lots of valuable information (especially my
different tests).



Some days ago,.. an engineer of Tyan suggested me to boot the kernel
with mem=3072M.
When doing this,.. the issue did not occur (I don't want to say it was
solved. Why? See my last emails to Tyan!)
Then he suggested me to disable the memory hole mapping in the BIOS,...
When doing so,.. the error doesn't occur, too.
But I loose about 2GB RAM,.. and,.. more important,.. I cant believe
that this is responsible for the whole issue. I don't consider it a
solution but more a poor workaround which perhaps only by fortune solves
the issue (Why? See my last eMails to Tyan ;) )



So I'd like to ask you if you perhaps could read the current information
in this and previous mails,.. and tell me your opinions.
It is very likely that a large number of users suffer from this error
(namely all Nvidia chipset users) but only few (there are some,.. I
found most of them in the Nvidia forums,.. and they have exactly the
same issue) identify this as an error because it's so rare.

Perhaps someone have an idea why disabling the memhole mapping solves
it. I've always thought that memhole mapping just moves some address
space to higher addreses to avoid the conflict between address space for
PCI devices and address space for pyhsical memory.
But this should be just a simple addition and not solve this obviously
complex error.


If this is related to some problem with using the GART IOMMU with memory 
hole remapping enabled, then 2.6.20-rc kernels may avoid this problem on 
nForce4 CK804/MCP04 chipsets as far as transfers to/from the SATA 
controller are concerned as the sata_nv driver now supports 64-bit DMA 
on these chipsets and so no longer requires the IOMMU.


--
Robert Hancock  Saskatoon, SK, Canada
To email, remove nospam from [EMAIL PROTECTED]
Home Page: http://www.roberthancock.com/

-
To unsubscribe from this list: send the line unsubscribe linux-kernel in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: data corruption with nvidia chipsets and IDE/SATA drives // memory hole mapping related bug?!

[Christoph Anton Mitterer]

John A Chaves wrote:
> I didn't need to run a specific test for this.  The normal workload of the
> machine approximates a continuous selftest for almost the last year.
>
> Large files (4-12GB is typical) are being continuously packed and unpacked
> with gzip and bzip2.  Statistical analysis of the datasets is followed by
> verification of the data, sometimes using diff, or md5sum, or python
> scripts using numarray to mmap 2GB chunks at a time.  The machine
> often goes for days with a load level of 20+ and 32GB RAM + another 32GB
> swap in use.  It would be very unlikely for data corruption to go unnoticed.
>
> When I first got the machine I did have some problems with disks being
> dropped from the RAID and occasional log messages implicating the IOMMU.
> But that was with kernel 2.6.16.?, Kernels since 2.6.17 haven't had any
> problem.
>   
Ah thanks for that info,.. as far as I can tell,.. this "testing
environment" should have found any corruptions I there had been any.

So I think we could take this as our first working system where the
issue don't occur although we would expect it...

Chris.
begin:vcard
fn:Mitterer, Christoph Anton
n:Mitterer;Christoph Anton
email;internet:[EMAIL PROTECTED]
x-mozilla-html:TRUE
version:2.1
end:vcard

Re: data corruption with nvidia chipsets and IDE/SATA drives // memory hole mapping related bug?!

[John A Chaves]

On Friday 22 December 2006 20:04, Christoph Anton Mitterer wrote:
> This brings me to:
> Chris Wedgwood wrote:
> > Does anyone have an amd64 with an nforce4 chipset and >4GB that does
> > NOT have this problem? If so it might be worth chasing the BIOS
> > vendors to see what errata they are dealing with.
> John Chaves replied and claimed that he wouldn't suffer from that
> problem (I've CC'ed him to this post).
> You can read his message at the bottom of this post.
> @ John: Could you please tell us in detail how you've tested your system?

I didn't need to run a specific test for this.  The normal workload of the
machine approximates a continuous selftest for almost the last year.

Large files (4-12GB is typical) are being continuously packed and unpacked
with gzip and bzip2.  Statistical analysis of the datasets is followed by
verification of the data, sometimes using diff, or md5sum, or python
scripts using numarray to mmap 2GB chunks at a time.  The machine
often goes for days with a load level of 20+ and 32GB RAM + another 32GB
swap in use.  It would be very unlikely for data corruption to go unnoticed.

When I first got the machine I did have some problems with disks being
dropped from the RAID and occasional log messages implicating the IOMMU.
But that was with kernel 2.6.16.?, Kernels since 2.6.17 haven't had any
problem.

John
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: data corruption with nvidia chipsets and IDE/SATA drives // memory hole mapping related bug?!

[Christoph Anton Mitterer]

Hi my friends

It became a little bit silent about this issue... any new ideas or results?



Karsten Weiss wrote:
> BTW: Did someone already open an official bug at
> http://bugzilla.kernel.org ?
Karsten, did you already file a bug?



I told the whole issue to the Debian people which are about to release
etch and suggested them to use iommu=soft by default.
This brings me to:
Chris Wedgwood wrote:
> Does anyone have an amd64 with an nforce4 chipset and >4GB that does
> NOT have this problem? If so it might be worth chasing the BIOS
> vendors to see what errata they are dealing with.
John Chaves replied and claimed that he wouldn't suffer from that
problem (I've CC'ed him to this post).
You can read his message at the bottom of this post.
@ John: Could you please tell us in detail how you've tested your system?



Muli told us some information about the iommu options (when he
discuessed Karstens patch) has anybody made tests with the other iommu
options?



Ok and what does it all come down to? We still don't know the exact
reason...
Perhaps a kernel bug, a Opteron and/or Chipset bug,.. and perhaps there
are even some BIOSes that solve the issue...

For the kernel-bug reason,... who is the responsible developer for the
relevant code? Can we contact him to read our threads and perhaps review
the code?

Is anyone able (or wants to try) to inform AMD and/or Nvidia about the
issue (perhaps with pointing to that thread).

Someone might even try to contact some board vendors (some of us seem to
have Tyan boards). Although I'm in contact with the German support Team
of Tyan, I wasn't very successful with the US team... perhaps they have
other ideas.

Last but not least if we don't find a solution what should we do?
In my opinion at least the following:
1) Inform other OS communities (*BSD) and point the to our thread. Some
of you claimed that Windows wouldn't use the hwiommu at all so I think
we don't have to contact big evil.
2) Contact the major Linux Distributions (I've already did it for
Debian) and inform them about the potential issue and pointing them to
this thread (where one can find all the relevant information, I think)
3) Workaround for the kernel:
I have to less knowledge to know exactly what to do but I remember there
are other fixes for mainboard flaws and buggy chipsets in the kernel
(e.g. the RZ1000 or something like this in the "old" IDE driver)...
Perhaps someone (who knows what to do ;-) ) could write some code that
automatically uses iommu=soft,... but then we have the question: In
which case :-( . I imagine that the AMD users who don't suffer from this
issue would like to continue using their hwiommus..


What I'm currently plan to do:
1) If know one else is willing to try contacting AMD/Nvidia,.. I'd try
again.
2) I told you that I'm going to test the whole issue in the Leibniz
Supercomputing Centre where I work as student...
This is a little bit delayed (organisational problems :-) )
Anyway,... I'm not only going to test it on our Linux Cluster but also
some Sun Fire's (whe have mny of them ;-) ). According to my
boss they have nvidia chipsets... (He is probably contacting Sun for the
issue).



So much for now.

Best wishes,
Chris.


John Chaves message:
Here's another data point in case it helps.
The following system does *not* have the data corruption issue.

Motherboard: Iwill DK88 
Chipset: NVIDIA nForce4 Professional 2200
CPUs: Two Dual Core AMD Opteron(tm) Processor 280
Memory: 32GB
Disks: Four 500GB SATA in linux RAID1 over RAID0 setup
Kernel: 2.6.18

This system is a workhorse with extreme disk I/O of huge files,
and the nature of the work done would have revealed data
corruption pretty quickly.

FWIW,
John Chaves

His lspic:
:00:00.0 Memory controller: nVidia Corporation CK804 Memory
Controller (rev a3)
Flags: bus master, 66MHz, fast devsel, latency 0
Capabilities: [44] #08 [01e0]
Capabilities: [e0] #08 [a801]

:00:01.0 ISA bridge: nVidia Corporation CK804 ISA Bridge (rev a3)
Subsystem: nVidia Corporation: Unknown device cb84
Flags: bus master, 66MHz, fast devsel, latency 0

:00:01.1 SMBus: nVidia Corporation CK804 SMBus (rev a2)
Subsystem: nVidia Corporation: Unknown device cb84
Flags: 66MHz, fast devsel, IRQ 9
I/O ports at d400 [size=32]
I/O ports at 4c00 [size=64]
I/O ports at 4c40 [size=64]
Capabilities: [44] Power Management version 2

:00:02.0 USB Controller: nVidia Corporation CK804 USB Controller
(rev a2) (prog-if 10 [OHCI])
Subsystem: nVidia Corporation: Unknown device cb84
Flags: bus master, 66MHz, fast devsel, latency 0, IRQ 209
Memory at feafc000 (32-bit, non-prefetchable) [size=4K]
Capabilities: [44] Power Management version 2

:00:02.1 USB Controller: nVidia Corporation CK804 USB Controller
(rev a3) (prog-if 20 [EHCI])
Subsystem: nVidia Corporation: Unknown device cb84
Flags: bus master, 66MHz, fast devsel, latency 0, IRQ 193
Memory at feafdc00 (32-bit, non-prefetchable) [size=256]

Re: data corruption with nvidia chipsets and IDE/SATA drives // memory hole mapping related bug?!

[Christoph Anton Mitterer]

Hi my friends

It became a little bit silent about this issue... any new ideas or results?



Karsten Weiss wrote:
 BTW: Did someone already open an official bug at
 http://bugzilla.kernel.org ?
Karsten, did you already file a bug?



I told the whole issue to the Debian people which are about to release
etch and suggested them to use iommu=soft by default.
This brings me to:
Chris Wedgwood wrote:
 Does anyone have an amd64 with an nforce4 chipset and 4GB that does
 NOT have this problem? If so it might be worth chasing the BIOS
 vendors to see what errata they are dealing with.
John Chaves replied and claimed that he wouldn't suffer from that
problem (I've CC'ed him to this post).
You can read his message at the bottom of this post.
@ John: Could you please tell us in detail how you've tested your system?



Muli told us some information about the iommu options (when he
discuessed Karstens patch) has anybody made tests with the other iommu
options?



Ok and what does it all come down to? We still don't know the exact
reason...
Perhaps a kernel bug, a Opteron and/or Chipset bug,.. and perhaps there
are even some BIOSes that solve the issue...

For the kernel-bug reason,... who is the responsible developer for the
relevant code? Can we contact him to read our threads and perhaps review
the code?

Is anyone able (or wants to try) to inform AMD and/or Nvidia about the
issue (perhaps with pointing to that thread).

Someone might even try to contact some board vendors (some of us seem to
have Tyan boards). Although I'm in contact with the German support Team
of Tyan, I wasn't very successful with the US team... perhaps they have
other ideas.

Last but not least if we don't find a solution what should we do?
In my opinion at least the following:
1) Inform other OS communities (*BSD) and point the to our thread. Some
of you claimed that Windows wouldn't use the hwiommu at all so I think
we don't have to contact big evil.
2) Contact the major Linux Distributions (I've already did it for
Debian) and inform them about the potential issue and pointing them to
this thread (where one can find all the relevant information, I think)
3) Workaround for the kernel:
I have to less knowledge to know exactly what to do but I remember there
are other fixes for mainboard flaws and buggy chipsets in the kernel
(e.g. the RZ1000 or something like this in the old IDE driver)...
Perhaps someone (who knows what to do ;-) ) could write some code that
automatically uses iommu=soft,... but then we have the question: In
which case :-( . I imagine that the AMD users who don't suffer from this
issue would like to continue using their hwiommus..


What I'm currently plan to do:
1) If know one else is willing to try contacting AMD/Nvidia,.. I'd try
again.
2) I told you that I'm going to test the whole issue in the Leibniz
Supercomputing Centre where I work as student...
This is a little bit delayed (organisational problems :-) )
Anyway,... I'm not only going to test it on our Linux Cluster but also
some Sun Fire's (whe have mny of them ;-) ). According to my
boss they have nvidia chipsets... (He is probably contacting Sun for the
issue).



So much for now.

Best wishes,
Chris.


John Chaves message:
Here's another data point in case it helps.
The following system does *not* have the data corruption issue.

Motherboard: Iwill DK88 http://www.iwill.net/product_2.asp?p_id=102
Chipset: NVIDIA nForce4 Professional 2200
CPUs: Two Dual Core AMD Opteron(tm) Processor 280
Memory: 32GB
Disks: Four 500GB SATA in linux RAID1 over RAID0 setup
Kernel: 2.6.18

This system is a workhorse with extreme disk I/O of huge files,
and the nature of the work done would have revealed data
corruption pretty quickly.

FWIW,
John Chaves

His lspic:
:00:00.0 Memory controller: nVidia Corporation CK804 Memory
Controller (rev a3)
Flags: bus master, 66MHz, fast devsel, latency 0
Capabilities: [44] #08 [01e0]
Capabilities: [e0] #08 [a801]

:00:01.0 ISA bridge: nVidia Corporation CK804 ISA Bridge (rev a3)
Subsystem: nVidia Corporation: Unknown device cb84
Flags: bus master, 66MHz, fast devsel, latency 0

:00:01.1 SMBus: nVidia Corporation CK804 SMBus (rev a2)
Subsystem: nVidia Corporation: Unknown device cb84
Flags: 66MHz, fast devsel, IRQ 9
I/O ports at d400 [size=32]
I/O ports at 4c00 [size=64]
I/O ports at 4c40 [size=64]
Capabilities: [44] Power Management version 2

:00:02.0 USB Controller: nVidia Corporation CK804 USB Controller
(rev a2) (prog-if 10 [OHCI])
Subsystem: nVidia Corporation: Unknown device cb84
Flags: bus master, 66MHz, fast devsel, latency 0, IRQ 209
Memory at feafc000 (32-bit, non-prefetchable) [size=4K]
Capabilities: [44] Power Management version 2

:00:02.1 USB Controller: nVidia Corporation CK804 USB Controller
(rev a3) (prog-if 20 [EHCI])
Subsystem: nVidia Corporation: Unknown device cb84
Flags: bus master, 66MHz, fast devsel, latency 0, IRQ 193
Memory at feafdc00 (32-bit, non-prefetchable) [size=256]

Re: data corruption with nvidia chipsets and IDE/SATA drives // memory hole mapping related bug?!

[John A Chaves]

On Friday 22 December 2006 20:04, Christoph Anton Mitterer wrote:
 This brings me to:
 Chris Wedgwood wrote:
  Does anyone have an amd64 with an nforce4 chipset and 4GB that does
  NOT have this problem? If so it might be worth chasing the BIOS
  vendors to see what errata they are dealing with.
 John Chaves replied and claimed that he wouldn't suffer from that
 problem (I've CC'ed him to this post).
 You can read his message at the bottom of this post.
 @ John: Could you please tell us in detail how you've tested your system?

I didn't need to run a specific test for this.  The normal workload of the
machine approximates a continuous selftest for almost the last year.

Large files (4-12GB is typical) are being continuously packed and unpacked
with gzip and bzip2.  Statistical analysis of the datasets is followed by
verification of the data, sometimes using diff, or md5sum, or python
scripts using numarray to mmap 2GB chunks at a time.  The machine
often goes for days with a load level of 20+ and 32GB RAM + another 32GB
swap in use.  It would be very unlikely for data corruption to go unnoticed.

When I first got the machine I did have some problems with disks being
dropped from the RAID and occasional log messages implicating the IOMMU.
But that was with kernel 2.6.16.?, Kernels since 2.6.17 haven't had any
problem.

John
-
To unsubscribe from this list: send the line unsubscribe linux-kernel in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: data corruption with nvidia chipsets and IDE/SATA drives // memory hole mapping related bug?!

[Christoph Anton Mitterer]

John A Chaves wrote:
 I didn't need to run a specific test for this.  The normal workload of the
 machine approximates a continuous selftest for almost the last year.

 Large files (4-12GB is typical) are being continuously packed and unpacked
 with gzip and bzip2.  Statistical analysis of the datasets is followed by
 verification of the data, sometimes using diff, or md5sum, or python
 scripts using numarray to mmap 2GB chunks at a time.  The machine
 often goes for days with a load level of 20+ and 32GB RAM + another 32GB
 swap in use.  It would be very unlikely for data corruption to go unnoticed.

 When I first got the machine I did have some problems with disks being
 dropped from the RAID and occasional log messages implicating the IOMMU.
 But that was with kernel 2.6.16.?, Kernels since 2.6.17 haven't had any
 problem.
   
Ah thanks for that info,.. as far as I can tell,.. this testing
environment should have found any corruptions I there had been any.

So I think we could take this as our first working system where the
issue don't occur although we would expect it...

Chris.
begin:vcard
fn:Mitterer, Christoph Anton
n:Mitterer;Christoph Anton
email;internet:[EMAIL PROTECTED]
x-mozilla-html:TRUE
version:2.1
end:vcard

Re: data corruption with nvidia chipsets and IDE/SATA drives // memory hole mapping related bug?!

[Paul Slootman]

[EMAIL PROTECTED] wrote:
>On Wed, Dec 13, 2006 at 09:11:29PM +0100, Christoph Anton Mitterer wrote:
>
>> - error in the Opteron (memory controller)
>> - error in the Nvidia chipsets
>> - error in the kernel
>
>My guess without further information would be that some, but not all
>BIOSes are doing some work to avoid this.
>
>Does anyone have an amd64 with an nforce4 chipset and >4GB that does
>NOT have this problem?  If so it might be worth chasing the BIOS
>vendors to see what errata they are dealing with.

We have a number of Tyan S2891 systems at work, most with 8GB but all at
least 4GB (data corruption still occurs whether 4 or 8GB is installed;
didn't try less than 4GB...). All have 2 of the following CPUs:
vendor_id   : AuthenticAMD
cpu family  : 15
model   : 37
model name  : AMD Opteron(tm) Processor 248
stepping: 1
cpu MHz : 2210.208
cache size  : 1024 KB


- the older models have no problem with data corruption,
  but fail to boot 2.6.18 and up (exactly like
  http://bugzilla.kernel.org/show_bug.cgi?id=7505 )

- the newer models had problems with data corruption (running md5sum
  over a large number of files would show differences from run to run).
  Sometimes the system would hang (no messages on the serial console,
  no magic sysrq, nothing).
  These have no problem booting 2.6.18 and up, however.
  These were delivered with a 2.02 BIOS version.
  On a whim I tried booting with "nosmp noapic", and running on one CPU
  the systems seemed stable, no data corruption and no crashes.

- The older models flashed to the latest 2.02 BIOS from the Tyan website
  still have no data corruption but still won't boot 2.6.18 and up.

- The newer models flashed (downgraded!) to the 2.01 BIOS available from the 
Tyan
  website seem to work fine, no data corruption while running on both
  CPUs and no crashes (although perhaps time is too short to tell for
  sure, first one I did was 10 days ago).

- I have an idea that perhaps the 2.02 BIOS the newer systems were
  delivered with is a subtely different version than the one on the
  website. I may try flashing 2.02 again once the current 2.01 on these
  systems has proven to be stable.

- Apparently there's something different on the motherboards from the
  first batch and the second batch, otherwise I couldn't explain the
  difference in ability to boot 2.6.18 and up. However, I haven't had an
  opportunity to open two systems up to compare them visually.



Paul Slootman
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: data corruption with nvidia chipsets and IDE/SATA drives // memory hole mapping related bug?!

[Paul Slootman]

[EMAIL PROTECTED] wrote:
On Wed, Dec 13, 2006 at 09:11:29PM +0100, Christoph Anton Mitterer wrote:

 - error in the Opteron (memory controller)
 - error in the Nvidia chipsets
 - error in the kernel

My guess without further information would be that some, but not all
BIOSes are doing some work to avoid this.

Does anyone have an amd64 with an nforce4 chipset and 4GB that does
NOT have this problem?  If so it might be worth chasing the BIOS
vendors to see what errata they are dealing with.

We have a number of Tyan S2891 systems at work, most with 8GB but all at
least 4GB (data corruption still occurs whether 4 or 8GB is installed;
didn't try less than 4GB...). All have 2 of the following CPUs:
vendor_id   : AuthenticAMD
cpu family  : 15
model   : 37
model name  : AMD Opteron(tm) Processor 248
stepping: 1
cpu MHz : 2210.208
cache size  : 1024 KB


- the older models have no problem with data corruption,
  but fail to boot 2.6.18 and up (exactly like
  http://bugzilla.kernel.org/show_bug.cgi?id=7505 )

- the newer models had problems with data corruption (running md5sum
  over a large number of files would show differences from run to run).
  Sometimes the system would hang (no messages on the serial console,
  no magic sysrq, nothing).
  These have no problem booting 2.6.18 and up, however.
  These were delivered with a 2.02 BIOS version.
  On a whim I tried booting with nosmp noapic, and running on one CPU
  the systems seemed stable, no data corruption and no crashes.

- The older models flashed to the latest 2.02 BIOS from the Tyan website
  still have no data corruption but still won't boot 2.6.18 and up.

- The newer models flashed (downgraded!) to the 2.01 BIOS available from the 
Tyan
  website seem to work fine, no data corruption while running on both
  CPUs and no crashes (although perhaps time is too short to tell for
  sure, first one I did was 10 days ago).

- I have an idea that perhaps the 2.02 BIOS the newer systems were
  delivered with is a subtely different version than the one on the
  website. I may try flashing 2.02 again once the current 2.01 on these
  systems has proven to be stable.

- Apparently there's something different on the motherboards from the
  first batch and the second batch, otherwise I couldn't explain the
  difference in ability to boot 2.6.18 and up. However, I haven't had an
  opportunity to open two systems up to compare them visually.



Paul Slootman
-
To unsubscribe from this list: send the line unsubscribe linux-kernel in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: data corruption with nvidia chipsets and IDE/SATA drives // memory hole mapping related bug?!

[Dax Kelson]

On Sat, 2006-12-02 at 01:56 +0100, Christoph Anton Mitterer wrote:
> Hi.
> 
> Perhaps some of you have read my older two threads:
> http://marc.theaimsgroup.com/?t=11631244001=1=2 and the even
> older http://marc.theaimsgroup.com/?t=11629131451=1=2
> 
> The issue was basically the following:
> I found a severe bug mainly by fortune because it occurs very rarely.
> My test looks like the following: I have about 30GB of testing data on
> my harddisk,... I repeat verifying sha512 sums on these files and check
> if errors occur.
> One test pass verifies the 30GB 50 times,... about one to four
> differences are found in each pass.

This sounds very similar to a corruption issue I was experiencing on my
nforce4 based system. After replacing most of my hardware to no avail, I
discovered that if increased the voltage for my RAM chips the corruption
went away. Note that I was not overclocking at all.

Worth a try.

Dax Kelson


-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: data corruption with nvidia chipsets and IDE/SATA drives // memory hole mapping related bug?!

[Christoph Anton Mitterer]

Muli Ben-Yehuda wrote:
>> 4)
>> And does someone know if the nforce/opteron iommu requires IBM Calgary
>> IOMMU support?
>> 
> It doesn't, Calgary isn't found in machine with Opteron CPUs or NForce
> chipsets (AFAIK). However, compiling Calgary in should make no
> difference, as we detect in run-time which IOMMU is found and the
> machine.
Yes,.. I've read the relevant section shortly after sending that email ;-)

btw & for everybody:
I'm working (as student) at the LRZ (Leibniz Computing Centre) in Munich
where we have very large Linux Cluster and lots of different other
machines,...
I'm going to test for that error on most of the different types of
systems we have,.. and will inform you about my results (if they're
interesting).

Chris.
begin:vcard
fn:Mitterer, Christoph Anton
n:Mitterer;Christoph Anton
email;internet:[EMAIL PROTECTED]
x-mozilla-html:TRUE
version:2.1
end:vcard

Re: [PATCH 2nd try] Re: data corruption with nvidia chipsets and IDE/SATA drives // memory hole mapping related bug?!

[Muli Ben-Yehuda]

On Thu, Dec 14, 2006 at 02:16:31PM +0100, Karsten Weiss wrote:
> On Thu, 14 Dec 2006, Muli Ben-Yehuda wrote:
> 
> > The rest looks good. Please resend and I'll add my Acked-by.
> 
> Thanks a lot for your comments and suggestions. Here's my 2nd try:
> 
> ===
> 
> From: Karsten Weiss <[EMAIL PROTECTED]>
> 
> $ diffstat ~/iommu-patch_v2.patch
>  Documentation/kernel-parameters.txt   |3
>  Documentation/x86_64/boot-options.txt |  104 
> +++---
>  arch/x86_64/Kconfig   |   10 ++-
>  arch/x86_64/kernel/pci-dma.c  |   28 +
>  4 files changed, 87 insertions(+), 58 deletions(-)
> 
> Patch description:
> 
> - add SWIOTLB config help text
> - mention Documentation/x86_64/boot-options.txt in
>   Documentation/kernel-parameters.txt
> - remove the duplication of the iommu kernel parameter documentation.
> - Better explanation of some of the iommu kernel parameter options.
> - "32MB< - Mention the default "order" value.
> - list the four existing PCI-DMA mapping implementations of arch x86_64
> - group the iommu= option keywords by PCI-DMA mapping implementation.
> - Distinguish iommu= option keywords from number arguments.
> - Explain the meaning of DAC and SAC.
> 
> Signed-off-by: Karsten Weiss <[EMAIL PROTECTED]>

Acked-by: Muli Ben-Yehuda <[EMAIL PROTECTED]>

Cheers,
Muli
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[PATCH 2nd try] Re: data corruption with nvidia chipsets and IDE/SATA drives // memory hole mapping related bug?!

[Karsten Weiss]

On Thu, 14 Dec 2006, Muli Ben-Yehuda wrote:

> The rest looks good. Please resend and I'll add my Acked-by.

Thanks a lot for your comments and suggestions. Here's my 2nd try:

===

From: Karsten Weiss <[EMAIL PROTECTED]>

$ diffstat ~/iommu-patch_v2.patch
 Documentation/kernel-parameters.txt   |3
 Documentation/x86_64/boot-options.txt |  104 +++---
 arch/x86_64/Kconfig   |   10 ++-
 arch/x86_64/kernel/pci-dma.c  |   28 +
 4 files changed, 87 insertions(+), 58 deletions(-)

Patch description:

- add SWIOTLB config help text
- mention Documentation/x86_64/boot-options.txt in
  Documentation/kernel-parameters.txt
- remove the duplication of the iommu kernel parameter documentation.
- Better explanation of some of the iommu kernel parameter options.
- "32MB<

---

--- linux-2.6.19/arch/x86_64/kernel/pci-dma.c.original  2006-12-14 
11:15:38.348598021 +0100
+++ linux-2.6.19/arch/x86_64/kernel/pci-dma.c   2006-12-14 12:14:48.176967312 
+0100
@@ -223,30 +223,10 @@
 }
 EXPORT_SYMBOL(dma_set_mask);
 
-/* 
iommu=[size][,noagp][,off][,force][,noforce][,leak][,memaper[=order]][,merge]
- [,forcesac][,fullflush][,nomerge][,biomerge]
-   size  set size of iommu (in bytes)
-   noagp don't initialize the AGP driver and use full aperture.
-   off   don't use the IOMMU
-   leak  turn on simple iommu leak tracing (only when CONFIG_IOMMU_LEAK is on)
-   memaper[=order] allocate an own aperture over RAM with size 32MB^order.
-   noforce don't force IOMMU usage. Default.
-   force  Force IOMMU.
-   merge  Do lazy merging. This may improve performance on some block devices.
-  Implies force (experimental)
-   biomerge Do merging at the BIO layer. This is more efficient than merge,
-but should be only done with very big IOMMUs. Implies merge,force.
-   nomerge Don't do SG merging.
-   forcesac For SAC mode for masks <40bits  (experimental)
-   fullflush Flush IOMMU on each allocation (default)
-   nofullflush Don't use IOMMU fullflush
-   allowed  overwrite iommu off workarounds for specific chipsets.
-   soft Use software bounce buffering (default for Intel machines)
-   noaperture Don't touch the aperture for AGP.
-   allowdac Allow DMA >4GB
-   nodacForbid DMA >4GB
-   panicForce panic when IOMMU overflows
-*/
+/*
+ * See  for the iommu kernel parameter
+ * documentation.
+ */
 __init int iommu_setup(char *p)
 {
iommu_merge = 1;
--- linux-2.6.19/arch/x86_64/Kconfig.original   2006-12-14 11:37:35.832142506 
+0100
+++ linux-2.6.19/arch/x86_64/Kconfig2006-12-14 14:01:24.009193996 +0100
@@ -431,8 +431,8 @@
  on systems with more than 3GB. This is usually needed for USB,
  sound, many IDE/SATA chipsets and some other devices.
  Provides a driver for the AMD Athlon64/Opteron/Turion/Sempron GART
- based IOMMU and a software bounce buffer based IOMMU used on Intel
- systems and as fallback.
+ based hardware IOMMU and a software bounce buffer based IOMMU used
+ on Intel systems and as fallback.
  The code is only active when needed (enough memory and limited
  device) unless CONFIG_IOMMU_DEBUG or iommu=force is specified
  too.
@@ -458,6 +458,12 @@
 # need this always selected by IOMMU for the VIA workaround
 config SWIOTLB
bool
+   help
+ Support for software bounce buffers used on x86-64 systems
+ which don't have a hardware IOMMU (e.g. the current generation
+ of Intel's x86-64 CPUs). Using this PCI devices which can only
+ access 32-bits of memory can be used on systems with more than
+ 3 GB of memory. If unsure, say Y.
 
 config X86_MCE
bool "Machine check support" if EMBEDDED
--- linux-2.6.19/Documentation/x86_64/boot-options.txt.original 2006-12-14 
11:11:32.099300994 +0100
+++ linux-2.6.19/Documentation/x86_64/boot-options.txt  2006-12-14 
14:14:55.869560532 +0100
@@ -180,39 +180,79 @@
   pci=lastbus=NUMBER  Scan upto NUMBER busses, no matter what the 
mptable says.
   pci=noacpi   Don't use ACPI to set up PCI interrupt routing.
 
-IOMMU
+IOMMU (input/output memory management unit)
 
- iommu=[size][,noagp][,off][,force][,noforce][,leak][,memaper[=order]][,merge]
- [,forcesac][,fullflush][,nomerge][,noaperture]
-   size  set size of iommu (in bytes)
-   noagp don't initialize the AGP driver and use full aperture.
-   off   don't use the IOMMU
-   leak  turn on simple iommu leak tracing (only when CONFIG_IOMMU_LEAK is on)
-   memaper[=order] allocate an own aperture over RAM with size 32MB^order.
-   noforce don't force IOMMU usage. Default.
-   force  Force IOMMU.
-   merge  Do SG merging. Implies force (experimental)
-   nomerge Don't do SG merging.
-   forcesac For SAC mode for masks <40bits  (experimental)
-   fullflush Flush IOMMU on each allocation (default)
-   nofullflush Don't use IOMMU fullflush
-   allowed  overwrite iommu off workarounds 

Re: [PATCH] Re: data corruption with nvidia chipsets and IDE/SATA drives // memory hole mapping related bug?!

[Muli Ben-Yehuda]

On Thu, Dec 14, 2006 at 12:38:08PM +0100, Karsten Weiss wrote:
> On Thu, 14 Dec 2006, Muli Ben-Yehuda wrote:
> 
> > On Wed, Dec 13, 2006 at 09:34:16PM +0100, Karsten Weiss wrote:
> > 
> > > BTW: It would be really great if this area of the kernel would get some 
> > > more and better documentation. The information at 
> > > linux-2.6/Documentation/x86_64/boot_options.txt is very terse. I had to 
> > > read the code to get a *rough* idea what all the "iommu=" options 
> > > actually do and how they interact.
> > 
> > Patches happily accepted :-)
> 
> Well, you asked for it. :-) So here's my little contribution. Please 
> *double* *check*!

Looks good, some nits below.

> (BTW: I would like to know what "DAC" and "SAC" means in this
> context)

Single / Double Address Cycle. DAC is used with 32-bit PCI to push a
64-bit address in two cycles.

> @@ -458,6 +458,11 @@
>  # need this always selected by IOMMU for the VIA workaround
>  config SWIOTLB
>   bool
> + help
> +   Support for a software bounce buffer based IOMMU used on Intel
> +   systems which don't have a hardware IOMMU. Using this code
> +   PCI devices with 32bit memory access only are able to be
> +   used on systems with more than 3 GB.

I would rephrase as follows: "Support for software bounce buffers used
on x86-64 systems which don't have a hardware IOMMU. Using this PCI
devices which can only access 32-bits of memory can be used on systems
with more than 3 GB of memory".

> +   size set size of IOMMU (in bytes)

Due to historical precedence, some of these options are only valid for
GART. Perhaps mention for each option which IOMMUs it is valid for or
split them on a per IOMMU basis?

This one (size) is gart only.

> +   noagpdon't initialize the AGP driver and use full
> aperture.

gart only.

> +   off  don't initialize and use any kind of IOMMU.

all.

> +   leak turn on simple iommu leak tracing (only when
> +CONFIG_IOMMU_LEAK is on)

gart only.

> +   memaper[=order]  allocate an own aperture over RAM with size 32MB< +(default: order=1, i.e. 64MB)

gart only.

> +   noforce  don't force hardware IOMMU usage when it is not needed.
> +(default).

all.

> +   forceForce the use of the hardware IOMMU even when it is
> +not actually needed (e.g. because < 3 GB
>  memory).

all.

> +   mergeDo scather-gather (SG) merging. Implies force
> (experimental)

gart only.

> +   nomerge  Don't do scather-gather (SG) merging.

gart only.

> +   forcesac For SAC mode for masks <40bits  (experimental)

gart only.

> +   fullflushFlush AMD GART based hardware IOMMU on each allocation
> +(default)

gart only.

> +   nofullflush  Don't use IOMMU fullflush

gart only.

> +   allowed  overwrite iommu off workarounds for specific
> chipsets.

gart only.

> +   soft Use software bounce buffering (SWIOTLB) (default for 
> Intel
> +machines). This can be used to prevent the usage
> +of a available hardware IOMMU.

all.

> +   noaperture   Ask the AMD GART based hardware IOMMU driver not to 
> +touch the aperture for AGP.

gart only.

> +   allowdac Allow DMA >4GB
> +When off all DMA over >4GB is forced through an IOMMU or
> +bounce buffering.

gart only.

> +   nodacForbid DMA >4GB

gart only.

> +   panicAlways panic when IOMMU overflows

gart and Calgary.

The rest looks good. Please resend and I'll add my Acked-by.

Cheers,
Muli
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[PATCH] Re: data corruption with nvidia chipsets and IDE/SATA drives // memory hole mapping related bug?!

[Karsten Weiss]

On Thu, 14 Dec 2006, Muli Ben-Yehuda wrote:

> On Wed, Dec 13, 2006 at 09:34:16PM +0100, Karsten Weiss wrote:
> 
> > BTW: It would be really great if this area of the kernel would get some 
> > more and better documentation. The information at 
> > linux-2.6/Documentation/x86_64/boot_options.txt is very terse. I had to 
> > read the code to get a *rough* idea what all the "iommu=" options 
> > actually do and how they interact.
> 
> Patches happily accepted :-)

Well, you asked for it. :-) So here's my little contribution. Please 
*double* *check*!

(BTW: I would like to know what "DAC" and "SAC" means in this context)

===

From: Karsten Weiss <[EMAIL PROTECTED]>

Patch summary:

- Better explanation of some of the iommu kernel parameter options.
- "32MB<

---

--- linux-2.6.19/arch/x86_64/kernel/pci-dma.c.original  2006-12-14 
11:15:38.348598021 +0100
+++ linux-2.6.19/arch/x86_64/kernel/pci-dma.c   2006-12-14 12:14:48.176967312 
+0100
@@ -223,30 +223,10 @@
 }
 EXPORT_SYMBOL(dma_set_mask);
 
-/* 
iommu=[size][,noagp][,off][,force][,noforce][,leak][,memaper[=order]][,merge]
- [,forcesac][,fullflush][,nomerge][,biomerge]
-   size  set size of iommu (in bytes)
-   noagp don't initialize the AGP driver and use full aperture.
-   off   don't use the IOMMU
-   leak  turn on simple iommu leak tracing (only when CONFIG_IOMMU_LEAK is on)
-   memaper[=order] allocate an own aperture over RAM with size 32MB^order.
-   noforce don't force IOMMU usage. Default.
-   force  Force IOMMU.
-   merge  Do lazy merging. This may improve performance on some block devices.
-  Implies force (experimental)
-   biomerge Do merging at the BIO layer. This is more efficient than merge,
-but should be only done with very big IOMMUs. Implies merge,force.
-   nomerge Don't do SG merging.
-   forcesac For SAC mode for masks <40bits  (experimental)
-   fullflush Flush IOMMU on each allocation (default)
-   nofullflush Don't use IOMMU fullflush
-   allowed  overwrite iommu off workarounds for specific chipsets.
-   soft Use software bounce buffering (default for Intel machines)
-   noaperture Don't touch the aperture for AGP.
-   allowdac Allow DMA >4GB
-   nodacForbid DMA >4GB
-   panicForce panic when IOMMU overflows
-*/
+/*
+ * See  for the iommu kernel parameter
+ * documentation.
+ */
 __init int iommu_setup(char *p)
 {
iommu_merge = 1;
--- linux-2.6.19/arch/x86_64/Kconfig.original   2006-12-14 11:37:35.832142506 
+0100
+++ linux-2.6.19/arch/x86_64/Kconfig2006-12-14 11:47:24.346056710 +0100
@@ -431,8 +431,8 @@
  on systems with more than 3GB. This is usually needed for USB,
  sound, many IDE/SATA chipsets and some other devices.
  Provides a driver for the AMD Athlon64/Opteron/Turion/Sempron GART
- based IOMMU and a software bounce buffer based IOMMU used on Intel
- systems and as fallback.
+ based hardware IOMMU and a software bounce buffer based IOMMU used
+ on Intel systems and as fallback.
  The code is only active when needed (enough memory and limited
  device) unless CONFIG_IOMMU_DEBUG or iommu=force is specified
  too.
@@ -458,6 +458,11 @@
 # need this always selected by IOMMU for the VIA workaround
 config SWIOTLB
bool
+   help
+ Support for a software bounce buffer based IOMMU used on Intel
+ systems which don't have a hardware IOMMU. Using this code
+ PCI devices with 32bit memory access only are able to be
+ used on systems with more than 3 GB.
 
 config X86_MCE
bool "Machine check support" if EMBEDDED
--- linux-2.6.19/Documentation/x86_64/boot-options.txt.original 2006-12-14 
11:11:32.099300994 +0100
+++ linux-2.6.19/Documentation/x86_64/boot-options.txt  2006-12-14 
12:10:24.028009890 +0100
@@ -180,35 +180,66 @@
   pci=lastbus=NUMBER  Scan upto NUMBER busses, no matter what the 
mptable says.
   pci=noacpi   Don't use ACPI to set up PCI interrupt routing.
 
-IOMMU
+IOMMU (input/output memory management unit)
+
+ Currently four x86_64 PCI DMA mapping implementations exist:
+
+   1. : use no hardware/software IOMMU at all
+  (e.g. because you have < 3 GB memory).
+  Kernel boot message: "PCI-DMA: Disabling IOMMU"
+
+   2. : AMD GART based hardware IOMMU.
+  Kernel boot message: "PCI-DMA: using GART IOMMU"
+
+   3.  : Software IOMMU implementation. Used
+  e.g. if there is no hardware IOMMU in the system and it is need because
+  you have >3GB memory or told the kernel to us it (iommu=soft))
+  Kernel boot message: "PCI-DMA: Using software bounce buffering
+  for IO (SWIOTLB)"
+
+   4.  : IBM Calgary hardware IOMMU. Used in IBM
+  pSeries and xSeries servers. This hardware IOMMU supports DMA address
+  mapping with memory protection, etc.
+  Kernel boot message: "PCI-DMA: Using Calgary IOMMU" 
 
  iommu=[size][,noagp][,off][,force][,noforce][,leak][,memaper[=order]][,merge]
  

Re: data corruption with nvidia chipsets and IDE/SATA drives // memory hole mapping related bug?!

[Muli Ben-Yehuda]

On Thu, Dec 14, 2006 at 02:52:35AM -0700, Erik Andersen wrote:
> On Thu Dec 14, 2006 at 11:23:11AM +0200, Muli Ben-Yehuda wrote:
> > > I just realized that booting with "iommu=soft" makes my pcHDTV
> > > HD5500 DVB cards not work.  Time to go back to disabling the
> > > memhole and losing 1 GB.  :-(
> > 
> > That points to a bug in the driver (likely) or swiotlb (unlikely), as
> > the IOMMU in use should be transparent to the driver. Which driver is
> > it?
> 
> presumably one of cx88xx, cx88_blackbird, cx8800, cx88_dvb,
> cx8802, cx88_alsa, lgdt330x, tuner, cx2341x, btcx_risc,
> video_buf, video_buf_dvb, tveeprom, or dvb_pll.  It seems
> to take an amazing number of drivers to make these devices
> actually work...

Yikes! do you know which one actually handles the DMA mappings? I
suspect a missnig unmap or sync, which swiotlb requires to sync back
the bounce buffer with the driver's buffer.

Cheers,
Muli
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: data corruption with nvidia chipsets and IDE/SATA drives // memory hole mapping related bug?!

[Erik Andersen]

On Thu Dec 14, 2006 at 11:23:11AM +0200, Muli Ben-Yehuda wrote:
> > I just realized that booting with "iommu=soft" makes my pcHDTV
> > HD5500 DVB cards not work.  Time to go back to disabling the
> > memhole and losing 1 GB.  :-(
> 
> That points to a bug in the driver (likely) or swiotlb (unlikely), as
> the IOMMU in use should be transparent to the driver. Which driver is
> it?

presumably one of cx88xx, cx88_blackbird, cx8800, cx88_dvb,
cx8802, cx88_alsa, lgdt330x, tuner, cx2341x, btcx_risc,
video_buf, video_buf_dvb, tveeprom, or dvb_pll.  It seems
to take an amazing number of drivers to make these devices
actually work...

 -Erik

--
Erik B. Andersen http://codepoet-consulting.com/
--This message was written using 73% post-consumer electrons--
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: data corruption with nvidia chipsets and IDE/SATA drives // memory hole mapping related bug?!

[Chris Wedgwood]

On Wed, Dec 13, 2006 at 09:11:29PM +0100, Christoph Anton Mitterer wrote:

> - error in the Opteron (memory controller)
> - error in the Nvidia chipsets
> - error in the kernel

My guess without further information would be that some, but not all
BIOSes are doing some work to avoid this.

Does anyone have an amd64 with an nforce4 chipset and >4GB that does
NOT have this problem?  If so it might be worth chasing the BIOS
vendors to see what errata they are dealing with.
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: data corruption with nvidia chipsets and IDE/SATA drives // memory hole mapping related bug?!

[Muli Ben-Yehuda]

On Thu, Dec 14, 2006 at 12:33:23AM +0100, Christoph Anton Mitterer wrote:

> 4)
> And does someone know if the nforce/opteron iommu requires IBM Calgary
> IOMMU support?

It doesn't, Calgary isn't found in machine with Opteron CPUs or NForce
chipsets (AFAIK). However, compiling Calgary in should make no
difference, as we detect in run-time which IOMMU is found and the
machine.

Cheers,
Muli
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: data corruption with nvidia chipsets and IDE/SATA drives // memory hole mapping related bug?!

[Muli Ben-Yehuda]

On Wed, Dec 13, 2006 at 01:29:25PM -0700, Erik Andersen wrote:
> On Mon Dec 11, 2006 at 10:24:02AM +0100, Karsten Weiss wrote:
> > We could not reproduce the data corruption anymore if we boot
> > the machines with the kernel parameter "iommu=soft" i.e. if we
> > use software bounce buffering instead of the hw-iommu.
> 
> I just realized that booting with "iommu=soft" makes my pcHDTV
> HD5500 DVB cards not work.  Time to go back to disabling the
> memhole and losing 1 GB.  :-(

That points to a bug in the driver (likely) or swiotlb (unlikely), as
the IOMMU in use should be transparent to the driver. Which driver is
it?

Cheers,
Muli
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: data corruption with nvidia chipsets and IDE/SATA drives // memory hole mapping related bug?!

[Muli Ben-Yehuda]

On Wed, Dec 13, 2006 at 09:34:16PM +0100, Karsten Weiss wrote:

> FWIW: As far as I understand the linux kernel code (I am no kernel 
> developer so please correct me if I am wrong) the PCI dma mapping code is 
> abstracted by struct dma_mapping_ops. I.e. there are currently four 
> possible implementations for x86_64 (see
> linux-2.6/arch/x86_64/kernel/)
> 
> 1. pci-nommu.c : no IOMMU at all (e.g. because you have < 4 GB memory)
>Kernel boot message: "PCI-DMA: Disabling IOMMU."
> 
> 2. pci-gart.c : (AMD) Hardware-IOMMU.
>Kernel boot message: "PCI-DMA: using GART IOMMU" (this message
>first appeared in 2.6.16)
> 
> 3. pci-swiotlb.c : Software-IOMMU (used e.g. if there is no hw iommu)
>Kernel boot message: "PCI-DMA: Using software bounce buffering 
>for IO (SWIOTLB)"

Used if there's no HW IOMMU *and* it's needed (because you have >4GB
memory) or you told the kernel to use it (iommu=soft).

> 4. pci-calgary.c : Calgary HW-IOMMU from IBM; used in pSeries servers. 
>This HW-IOMMU supports dma address mapping with memory proctection,
>etc.
>Kernel boot message: "PCI-DMA: Using Calgary IOMMU" (since
>2.6.18!)

Calgary is found in pSeries servers, but also in high-end xSeries
(Intel based) servers. It would be a little awkward if pSeries servers
(which are based on PowerPC processors) used code under arch/x86-64
:-)

> BTW: It would be really great if this area of the kernel would get some 
> more and better documentation. The information at 
> linux-2.6/Documentation/x86_64/boot_options.txt is very terse. I had to 
> read the code to get a *rough* idea what all the "iommu=" options 
> actually do and how they interact.

Patches happily accepted :-)

Cheers,
Muli
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: data corruption with nvidia chipsets and IDE/SATA drives // memory hole mapping related bug?!

[Muli Ben-Yehuda]

On Wed, Dec 13, 2006 at 09:34:16PM +0100, Karsten Weiss wrote:

 FWIW: As far as I understand the linux kernel code (I am no kernel 
 developer so please correct me if I am wrong) the PCI dma mapping code is 
 abstracted by struct dma_mapping_ops. I.e. there are currently four 
 possible implementations for x86_64 (see
 linux-2.6/arch/x86_64/kernel/)
 
 1. pci-nommu.c : no IOMMU at all (e.g. because you have  4 GB memory)
Kernel boot message: PCI-DMA: Disabling IOMMU.
 
 2. pci-gart.c : (AMD) Hardware-IOMMU.
Kernel boot message: PCI-DMA: using GART IOMMU (this message
first appeared in 2.6.16)
 
 3. pci-swiotlb.c : Software-IOMMU (used e.g. if there is no hw iommu)
Kernel boot message: PCI-DMA: Using software bounce buffering 
for IO (SWIOTLB)

Used if there's no HW IOMMU *and* it's needed (because you have 4GB
memory) or you told the kernel to use it (iommu=soft).

 4. pci-calgary.c : Calgary HW-IOMMU from IBM; used in pSeries servers. 
This HW-IOMMU supports dma address mapping with memory proctection,
etc.
Kernel boot message: PCI-DMA: Using Calgary IOMMU (since
2.6.18!)

Calgary is found in pSeries servers, but also in high-end xSeries
(Intel based) servers. It would be a little awkward if pSeries servers
(which are based on PowerPC processors) used code under arch/x86-64
:-)

 BTW: It would be really great if this area of the kernel would get some 
 more and better documentation. The information at 
 linux-2.6/Documentation/x86_64/boot_options.txt is very terse. I had to 
 read the code to get a *rough* idea what all the iommu= options 
 actually do and how they interact.

Patches happily accepted :-)

Cheers,
Muli
-
To unsubscribe from this list: send the line unsubscribe linux-kernel in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: data corruption with nvidia chipsets and IDE/SATA drives // memory hole mapping related bug?!

[Muli Ben-Yehuda]

On Wed, Dec 13, 2006 at 01:29:25PM -0700, Erik Andersen wrote:
 On Mon Dec 11, 2006 at 10:24:02AM +0100, Karsten Weiss wrote:
  We could not reproduce the data corruption anymore if we boot
  the machines with the kernel parameter iommu=soft i.e. if we
  use software bounce buffering instead of the hw-iommu.
 
 I just realized that booting with iommu=soft makes my pcHDTV
 HD5500 DVB cards not work.  Time to go back to disabling the
 memhole and losing 1 GB.  :-(

That points to a bug in the driver (likely) or swiotlb (unlikely), as
the IOMMU in use should be transparent to the driver. Which driver is
it?

Cheers,
Muli
-
To unsubscribe from this list: send the line unsubscribe linux-kernel in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: data corruption with nvidia chipsets and IDE/SATA drives // memory hole mapping related bug?!

[Muli Ben-Yehuda]

On Thu, Dec 14, 2006 at 12:33:23AM +0100, Christoph Anton Mitterer wrote:

 4)
 And does someone know if the nforce/opteron iommu requires IBM Calgary
 IOMMU support?

It doesn't, Calgary isn't found in machine with Opteron CPUs or NForce
chipsets (AFAIK). However, compiling Calgary in should make no
difference, as we detect in run-time which IOMMU is found and the
machine.

Cheers,
Muli
-
To unsubscribe from this list: send the line unsubscribe linux-kernel in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: data corruption with nvidia chipsets and IDE/SATA drives // memory hole mapping related bug?!

[Chris Wedgwood]

On Wed, Dec 13, 2006 at 09:11:29PM +0100, Christoph Anton Mitterer wrote:

 - error in the Opteron (memory controller)
 - error in the Nvidia chipsets
 - error in the kernel

My guess without further information would be that some, but not all
BIOSes are doing some work to avoid this.

Does anyone have an amd64 with an nforce4 chipset and 4GB that does
NOT have this problem?  If so it might be worth chasing the BIOS
vendors to see what errata they are dealing with.
-
To unsubscribe from this list: send the line unsubscribe linux-kernel in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: data corruption with nvidia chipsets and IDE/SATA drives // memory hole mapping related bug?!

[Erik Andersen]

On Thu Dec 14, 2006 at 11:23:11AM +0200, Muli Ben-Yehuda wrote:
  I just realized that booting with iommu=soft makes my pcHDTV
  HD5500 DVB cards not work.  Time to go back to disabling the
  memhole and losing 1 GB.  :-(
 
 That points to a bug in the driver (likely) or swiotlb (unlikely), as
 the IOMMU in use should be transparent to the driver. Which driver is
 it?

presumably one of cx88xx, cx88_blackbird, cx8800, cx88_dvb,
cx8802, cx88_alsa, lgdt330x, tuner, cx2341x, btcx_risc,
video_buf, video_buf_dvb, tveeprom, or dvb_pll.  It seems
to take an amazing number of drivers to make these devices
actually work...

 -Erik

--
Erik B. Andersen http://codepoet-consulting.com/
--This message was written using 73% post-consumer electrons--
-
To unsubscribe from this list: send the line unsubscribe linux-kernel in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: data corruption with nvidia chipsets and IDE/SATA drives // memory hole mapping related bug?!

[Muli Ben-Yehuda]

On Thu, Dec 14, 2006 at 02:52:35AM -0700, Erik Andersen wrote:
 On Thu Dec 14, 2006 at 11:23:11AM +0200, Muli Ben-Yehuda wrote:
   I just realized that booting with iommu=soft makes my pcHDTV
   HD5500 DVB cards not work.  Time to go back to disabling the
   memhole and losing 1 GB.  :-(
  
  That points to a bug in the driver (likely) or swiotlb (unlikely), as
  the IOMMU in use should be transparent to the driver. Which driver is
  it?
 
 presumably one of cx88xx, cx88_blackbird, cx8800, cx88_dvb,
 cx8802, cx88_alsa, lgdt330x, tuner, cx2341x, btcx_risc,
 video_buf, video_buf_dvb, tveeprom, or dvb_pll.  It seems
 to take an amazing number of drivers to make these devices
 actually work...

Yikes! do you know which one actually handles the DMA mappings? I
suspect a missnig unmap or sync, which swiotlb requires to sync back
the bounce buffer with the driver's buffer.

Cheers,
Muli
-
To unsubscribe from this list: send the line unsubscribe linux-kernel in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[PATCH] Re: data corruption with nvidia chipsets and IDE/SATA drives // memory hole mapping related bug?!

[Karsten Weiss]

On Thu, 14 Dec 2006, Muli Ben-Yehuda wrote:

 On Wed, Dec 13, 2006 at 09:34:16PM +0100, Karsten Weiss wrote:
 
  BTW: It would be really great if this area of the kernel would get some 
  more and better documentation. The information at 
  linux-2.6/Documentation/x86_64/boot_options.txt is very terse. I had to 
  read the code to get a *rough* idea what all the iommu= options 
  actually do and how they interact.
 
 Patches happily accepted :-)

Well, you asked for it. :-) So here's my little contribution. Please 
*double* *check*!

(BTW: I would like to know what DAC and SAC means in this context)

===

From: Karsten Weiss [EMAIL PROTECTED]

Patch summary:

- Better explanation of some of the iommu kernel parameter options.
- 32MBorder instead of 32MB^order.
- Mention the default order.
- SWIOTLB config help text
- removed the duplication of the iommu kernel parameter documentation.
- mention Documentation/x86_64/boot-options.txt in 
  Documentation/kernel-parameters.txt
- list the four existing PCI DMA mapping implementations of arch x86_64

Signed-off-by: Karsten Weiss [EMAIL PROTECTED]

---

--- linux-2.6.19/arch/x86_64/kernel/pci-dma.c.original  2006-12-14 
11:15:38.348598021 +0100
+++ linux-2.6.19/arch/x86_64/kernel/pci-dma.c   2006-12-14 12:14:48.176967312 
+0100
@@ -223,30 +223,10 @@
 }
 EXPORT_SYMBOL(dma_set_mask);
 
-/* 
iommu=[size][,noagp][,off][,force][,noforce][,leak][,memaper[=order]][,merge]
- [,forcesac][,fullflush][,nomerge][,biomerge]
-   size  set size of iommu (in bytes)
-   noagp don't initialize the AGP driver and use full aperture.
-   off   don't use the IOMMU
-   leak  turn on simple iommu leak tracing (only when CONFIG_IOMMU_LEAK is on)
-   memaper[=order] allocate an own aperture over RAM with size 32MB^order.
-   noforce don't force IOMMU usage. Default.
-   force  Force IOMMU.
-   merge  Do lazy merging. This may improve performance on some block devices.
-  Implies force (experimental)
-   biomerge Do merging at the BIO layer. This is more efficient than merge,
-but should be only done with very big IOMMUs. Implies merge,force.
-   nomerge Don't do SG merging.
-   forcesac For SAC mode for masks 40bits  (experimental)
-   fullflush Flush IOMMU on each allocation (default)
-   nofullflush Don't use IOMMU fullflush
-   allowed  overwrite iommu off workarounds for specific chipsets.
-   soft Use software bounce buffering (default for Intel machines)
-   noaperture Don't touch the aperture for AGP.
-   allowdac Allow DMA 4GB
-   nodacForbid DMA 4GB
-   panicForce panic when IOMMU overflows
-*/
+/*
+ * See Documentation/x86_64/boot-options.txt for the iommu kernel parameter
+ * documentation.
+ */
 __init int iommu_setup(char *p)
 {
iommu_merge = 1;
--- linux-2.6.19/arch/x86_64/Kconfig.original   2006-12-14 11:37:35.832142506 
+0100
+++ linux-2.6.19/arch/x86_64/Kconfig2006-12-14 11:47:24.346056710 +0100
@@ -431,8 +431,8 @@
  on systems with more than 3GB. This is usually needed for USB,
  sound, many IDE/SATA chipsets and some other devices.
  Provides a driver for the AMD Athlon64/Opteron/Turion/Sempron GART
- based IOMMU and a software bounce buffer based IOMMU used on Intel
- systems and as fallback.
+ based hardware IOMMU and a software bounce buffer based IOMMU used
+ on Intel systems and as fallback.
  The code is only active when needed (enough memory and limited
  device) unless CONFIG_IOMMU_DEBUG or iommu=force is specified
  too.
@@ -458,6 +458,11 @@
 # need this always selected by IOMMU for the VIA workaround
 config SWIOTLB
bool
+   help
+ Support for a software bounce buffer based IOMMU used on Intel
+ systems which don't have a hardware IOMMU. Using this code
+ PCI devices with 32bit memory access only are able to be
+ used on systems with more than 3 GB.
 
 config X86_MCE
bool Machine check support if EMBEDDED
--- linux-2.6.19/Documentation/x86_64/boot-options.txt.original 2006-12-14 
11:11:32.099300994 +0100
+++ linux-2.6.19/Documentation/x86_64/boot-options.txt  2006-12-14 
12:10:24.028009890 +0100
@@ -180,35 +180,66 @@
   pci=lastbus=NUMBER  Scan upto NUMBER busses, no matter what the 
mptable says.
   pci=noacpi   Don't use ACPI to set up PCI interrupt routing.
 
-IOMMU
+IOMMU (input/output memory management unit)
+
+ Currently four x86_64 PCI DMA mapping implementations exist:
+
+   1. arch/x86_64/kernel/pci-nommu.c: use no hardware/software IOMMU at all
+  (e.g. because you have  3 GB memory).
+  Kernel boot message: PCI-DMA: Disabling IOMMU
+
+   2. arch/x86_64/kernel/pci-gart.c: AMD GART based hardware IOMMU.
+  Kernel boot message: PCI-DMA: using GART IOMMU
+
+   3. arch/x86_64/kernel/pci-swiotlb.c : Software IOMMU implementation. Used
+  e.g. if there is no hardware IOMMU in the system and it is need because
+  you have 3GB memory or 

Re: [PATCH] Re: data corruption with nvidia chipsets and IDE/SATA drives // memory hole mapping related bug?!

[Muli Ben-Yehuda]

On Thu, Dec 14, 2006 at 12:38:08PM +0100, Karsten Weiss wrote:
 On Thu, 14 Dec 2006, Muli Ben-Yehuda wrote:
 
  On Wed, Dec 13, 2006 at 09:34:16PM +0100, Karsten Weiss wrote:
  
   BTW: It would be really great if this area of the kernel would get some 
   more and better documentation. The information at 
   linux-2.6/Documentation/x86_64/boot_options.txt is very terse. I had to 
   read the code to get a *rough* idea what all the iommu= options 
   actually do and how they interact.
  
  Patches happily accepted :-)
 
 Well, you asked for it. :-) So here's my little contribution. Please 
 *double* *check*!

Looks good, some nits below.

 (BTW: I would like to know what DAC and SAC means in this
 context)

Single / Double Address Cycle. DAC is used with 32-bit PCI to push a
64-bit address in two cycles.

 @@ -458,6 +458,11 @@
  # need this always selected by IOMMU for the VIA workaround
  config SWIOTLB
   bool
 + help
 +   Support for a software bounce buffer based IOMMU used on Intel
 +   systems which don't have a hardware IOMMU. Using this code
 +   PCI devices with 32bit memory access only are able to be
 +   used on systems with more than 3 GB.

I would rephrase as follows: Support for software bounce buffers used
on x86-64 systems which don't have a hardware IOMMU. Using this PCI
devices which can only access 32-bits of memory can be used on systems
with more than 3 GB of memory.

 +   size set size of IOMMU (in bytes)

Due to historical precedence, some of these options are only valid for
GART. Perhaps mention for each option which IOMMUs it is valid for or
split them on a per IOMMU basis?

This one (size) is gart only.

 +   noagpdon't initialize the AGP driver and use full
 aperture.

gart only.

 +   off  don't initialize and use any kind of IOMMU.

all.

 +   leak turn on simple iommu leak tracing (only when
 +CONFIG_IOMMU_LEAK is on)

gart only.

 +   memaper[=order]  allocate an own aperture over RAM with size 32MBorder.
 +(default: order=1, i.e. 64MB)

gart only.

 +   noforce  don't force hardware IOMMU usage when it is not needed.
 +(default).

all.

 +   forceForce the use of the hardware IOMMU even when it is
 +not actually needed (e.g. because  3 GB
  memory).

all.

 +   mergeDo scather-gather (SG) merging. Implies force
 (experimental)

gart only.

 +   nomerge  Don't do scather-gather (SG) merging.

gart only.

 +   forcesac For SAC mode for masks 40bits  (experimental)

gart only.

 +   fullflushFlush AMD GART based hardware IOMMU on each allocation
 +(default)

gart only.

 +   nofullflush  Don't use IOMMU fullflush

gart only.

 +   allowed  overwrite iommu off workarounds for specific
 chipsets.

gart only.

 +   soft Use software bounce buffering (SWIOTLB) (default for 
 Intel
 +machines). This can be used to prevent the usage
 +of a available hardware IOMMU.

all.

 +   noaperture   Ask the AMD GART based hardware IOMMU driver not to 
 +touch the aperture for AGP.

gart only.

 +   allowdac Allow DMA 4GB
 +When off all DMA over 4GB is forced through an IOMMU or
 +bounce buffering.

gart only.

 +   nodacForbid DMA 4GB

gart only.

 +   panicAlways panic when IOMMU overflows

gart and Calgary.

The rest looks good. Please resend and I'll add my Acked-by.

Cheers,
Muli
-
To unsubscribe from this list: send the line unsubscribe linux-kernel in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[PATCH 2nd try] Re: data corruption with nvidia chipsets and IDE/SATA drives // memory hole mapping related bug?!

[Karsten Weiss]

On Thu, 14 Dec 2006, Muli Ben-Yehuda wrote:

 The rest looks good. Please resend and I'll add my Acked-by.

Thanks a lot for your comments and suggestions. Here's my 2nd try:

===

From: Karsten Weiss [EMAIL PROTECTED]

$ diffstat ~/iommu-patch_v2.patch
 Documentation/kernel-parameters.txt   |3
 Documentation/x86_64/boot-options.txt |  104 +++---
 arch/x86_64/Kconfig   |   10 ++-
 arch/x86_64/kernel/pci-dma.c  |   28 +
 4 files changed, 87 insertions(+), 58 deletions(-)

Patch description:

- add SWIOTLB config help text
- mention Documentation/x86_64/boot-options.txt in
  Documentation/kernel-parameters.txt
- remove the duplication of the iommu kernel parameter documentation.
- Better explanation of some of the iommu kernel parameter options.
- 32MBorder instead of 32MB^order.
- Mention the default order value.
- list the four existing PCI-DMA mapping implementations of arch x86_64
- group the iommu= option keywords by PCI-DMA mapping implementation.
- Distinguish iommu= option keywords from number arguments.
- Explain the meaning of DAC and SAC.

Signed-off-by: Karsten Weiss [EMAIL PROTECTED]

---

--- linux-2.6.19/arch/x86_64/kernel/pci-dma.c.original  2006-12-14 
11:15:38.348598021 +0100
+++ linux-2.6.19/arch/x86_64/kernel/pci-dma.c   2006-12-14 12:14:48.176967312 
+0100
@@ -223,30 +223,10 @@
 }
 EXPORT_SYMBOL(dma_set_mask);
 
-/* 
iommu=[size][,noagp][,off][,force][,noforce][,leak][,memaper[=order]][,merge]
- [,forcesac][,fullflush][,nomerge][,biomerge]
-   size  set size of iommu (in bytes)
-   noagp don't initialize the AGP driver and use full aperture.
-   off   don't use the IOMMU
-   leak  turn on simple iommu leak tracing (only when CONFIG_IOMMU_LEAK is on)
-   memaper[=order] allocate an own aperture over RAM with size 32MB^order.
-   noforce don't force IOMMU usage. Default.
-   force  Force IOMMU.
-   merge  Do lazy merging. This may improve performance on some block devices.
-  Implies force (experimental)
-   biomerge Do merging at the BIO layer. This is more efficient than merge,
-but should be only done with very big IOMMUs. Implies merge,force.
-   nomerge Don't do SG merging.
-   forcesac For SAC mode for masks 40bits  (experimental)
-   fullflush Flush IOMMU on each allocation (default)
-   nofullflush Don't use IOMMU fullflush
-   allowed  overwrite iommu off workarounds for specific chipsets.
-   soft Use software bounce buffering (default for Intel machines)
-   noaperture Don't touch the aperture for AGP.
-   allowdac Allow DMA 4GB
-   nodacForbid DMA 4GB
-   panicForce panic when IOMMU overflows
-*/
+/*
+ * See Documentation/x86_64/boot-options.txt for the iommu kernel parameter
+ * documentation.
+ */
 __init int iommu_setup(char *p)
 {
iommu_merge = 1;
--- linux-2.6.19/arch/x86_64/Kconfig.original   2006-12-14 11:37:35.832142506 
+0100
+++ linux-2.6.19/arch/x86_64/Kconfig2006-12-14 14:01:24.009193996 +0100
@@ -431,8 +431,8 @@
  on systems with more than 3GB. This is usually needed for USB,
  sound, many IDE/SATA chipsets and some other devices.
  Provides a driver for the AMD Athlon64/Opteron/Turion/Sempron GART
- based IOMMU and a software bounce buffer based IOMMU used on Intel
- systems and as fallback.
+ based hardware IOMMU and a software bounce buffer based IOMMU used
+ on Intel systems and as fallback.
  The code is only active when needed (enough memory and limited
  device) unless CONFIG_IOMMU_DEBUG or iommu=force is specified
  too.
@@ -458,6 +458,12 @@
 # need this always selected by IOMMU for the VIA workaround
 config SWIOTLB
bool
+   help
+ Support for software bounce buffers used on x86-64 systems
+ which don't have a hardware IOMMU (e.g. the current generation
+ of Intel's x86-64 CPUs). Using this PCI devices which can only
+ access 32-bits of memory can be used on systems with more than
+ 3 GB of memory. If unsure, say Y.
 
 config X86_MCE
bool Machine check support if EMBEDDED
--- linux-2.6.19/Documentation/x86_64/boot-options.txt.original 2006-12-14 
11:11:32.099300994 +0100
+++ linux-2.6.19/Documentation/x86_64/boot-options.txt  2006-12-14 
14:14:55.869560532 +0100
@@ -180,39 +180,79 @@
   pci=lastbus=NUMBER  Scan upto NUMBER busses, no matter what the 
mptable says.
   pci=noacpi   Don't use ACPI to set up PCI interrupt routing.
 
-IOMMU
+IOMMU (input/output memory management unit)
 
- iommu=[size][,noagp][,off][,force][,noforce][,leak][,memaper[=order]][,merge]
- [,forcesac][,fullflush][,nomerge][,noaperture]
-   size  set size of iommu (in bytes)
-   noagp don't initialize the AGP driver and use full aperture.
-   off   don't use the IOMMU
-   leak  turn on simple iommu leak tracing (only when CONFIG_IOMMU_LEAK is on)
-   memaper[=order] allocate an own aperture over RAM with 

Re: [PATCH 2nd try] Re: data corruption with nvidia chipsets and IDE/SATA drives // memory hole mapping related bug?!

[Muli Ben-Yehuda]

On Thu, Dec 14, 2006 at 02:16:31PM +0100, Karsten Weiss wrote:
 On Thu, 14 Dec 2006, Muli Ben-Yehuda wrote:
 
  The rest looks good. Please resend and I'll add my Acked-by.
 
 Thanks a lot for your comments and suggestions. Here's my 2nd try:
 
 ===
 
 From: Karsten Weiss [EMAIL PROTECTED]
 
 $ diffstat ~/iommu-patch_v2.patch
  Documentation/kernel-parameters.txt   |3
  Documentation/x86_64/boot-options.txt |  104 
 +++---
  arch/x86_64/Kconfig   |   10 ++-
  arch/x86_64/kernel/pci-dma.c  |   28 +
  4 files changed, 87 insertions(+), 58 deletions(-)
 
 Patch description:
 
 - add SWIOTLB config help text
 - mention Documentation/x86_64/boot-options.txt in
   Documentation/kernel-parameters.txt
 - remove the duplication of the iommu kernel parameter documentation.
 - Better explanation of some of the iommu kernel parameter options.
 - 32MBorder instead of 32MB^order.
 - Mention the default order value.
 - list the four existing PCI-DMA mapping implementations of arch x86_64
 - group the iommu= option keywords by PCI-DMA mapping implementation.
 - Distinguish iommu= option keywords from number arguments.
 - Explain the meaning of DAC and SAC.
 
 Signed-off-by: Karsten Weiss [EMAIL PROTECTED]

Acked-by: Muli Ben-Yehuda [EMAIL PROTECTED]

Cheers,
Muli
-
To unsubscribe from this list: send the line unsubscribe linux-kernel in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: data corruption with nvidia chipsets and IDE/SATA drives // memory hole mapping related bug?!

[Christoph Anton Mitterer]

Muli Ben-Yehuda wrote:
 4)
 And does someone know if the nforce/opteron iommu requires IBM Calgary
 IOMMU support?
 
 It doesn't, Calgary isn't found in machine with Opteron CPUs or NForce
 chipsets (AFAIK). However, compiling Calgary in should make no
 difference, as we detect in run-time which IOMMU is found and the
 machine.
Yes,.. I've read the relevant section shortly after sending that email ;-)

btw  for everybody:
I'm working (as student) at the LRZ (Leibniz Computing Centre) in Munich
where we have very large Linux Cluster and lots of different other
machines,...
I'm going to test for that error on most of the different types of
systems we have,.. and will inform you about my results (if they're
interesting).

Chris.
begin:vcard
fn:Mitterer, Christoph Anton
n:Mitterer;Christoph Anton
email;internet:[EMAIL PROTECTED]
x-mozilla-html:TRUE
version:2.1
end:vcard

Re: data corruption with nvidia chipsets and IDE/SATA drives // memory hole mapping related bug?!

[Dax Kelson]

On Sat, 2006-12-02 at 01:56 +0100, Christoph Anton Mitterer wrote:
 Hi.
 
 Perhaps some of you have read my older two threads:
 http://marc.theaimsgroup.com/?t=11631244001r=1w=2 and the even
 older http://marc.theaimsgroup.com/?t=11629131451r=1w=2
 
 The issue was basically the following:
 I found a severe bug mainly by fortune because it occurs very rarely.
 My test looks like the following: I have about 30GB of testing data on
 my harddisk,... I repeat verifying sha512 sums on these files and check
 if errors occur.
 One test pass verifies the 30GB 50 times,... about one to four
 differences are found in each pass.

This sounds very similar to a corruption issue I was experiencing on my
nforce4 based system. After replacing most of my hardware to no avail, I
discovered that if increased the voltage for my RAM chips the corruption
went away. Note that I was not overclocking at all.

Worth a try.

Dax Kelson


-
To unsubscribe from this list: send the line unsubscribe linux-kernel in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: data corruption with nvidia chipsets and IDE/SATA drives // memory hole mapping related bug?!

[Christoph Anton Mitterer]

Hi.

I've just looked for some kernel config options that might relate to our
issue:


1)
Old style AMD Opteron NUMA detection (CONFIG_K8_NUMA)
Enable K8 NUMA node topology detection.  You should say Y here if you
have a multi processor AMD K8 system. This uses an old method to read
the NUMA configuration directly from the builtin Northbridge of Opteron.
It is recommended to use X86_64_ACPI_NUMA instead, which also takes
priority if both are compiled in.

ACPI NUMA detection (CONFIG_X86_64_ACPI_NUMA)
Enable ACPI SRAT based node topology detection.

What should one select for the Opterons? And is it possible that this
has something to do with our datacorruption error?


2)
The same two questions for the memory model (Discontiguous or Sparse)



3)
The same two questions for CONFIG_MIGRATION ()


4)
And does someone know if the nforce/opteron iommu requires IBM Calgary
IOMMU support?



This is unrelated to our issue,.. but it would be nice if some of your
could send me their .config,.. I'd like to compare them with my own and
see if I could something tweak or so.
(Of course only people with 2x DualCore Systems ;) )



Chris.
begin:vcard
fn:Mitterer, Christoph Anton
n:Mitterer;Christoph Anton
email;internet:[EMAIL PROTECTED]
x-mozilla-html:TRUE
version:2.1
end:vcard

Re: data corruption with nvidia chipsets and IDE/SATA drives // memory hole mapping related bug?!

[Lennart Sorensen]

On Wed, Dec 13, 2006 at 08:57:23PM +0100, Christoph Anton Mitterer wrote:
> Don't understand me wrong,.. I don't use Windows (expect for upgrading
> my Plextor firmware and EAC ;) )... but I ask because the more
> information we get (even if it's not Linux specific) the more steps we
> can take ;)

I upgrade my plextor firmware using linux.  pxupdate for most devices,
and pxfw for new drivers (like the PX760).  Works perfectly for me.  It
is one of the reasons I buy plextors.

--
Len Sorensen
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: data corruption with nvidia chipsets and IDE/SATA drives // memory hole mapping related bug?!

[Christoph Anton Mitterer]

Lennart Sorensen wrote:
> I upgrade my plextor firmware using linux.  pxupdate for most devices,
> and pxfw for new drivers (like the PX760).  Works perfectly for me.  It
> is one of the reasons I buy plextors.
Yes I know about it,.. although never tested it,... anyway the main
reason for Windows is Exact Audio Copy (but Andre Wiehthoff is working
on a C port :-D )

Unfortunately my PX760 seems to be defect,.. posted about the issue to
lkml but no success :-(

Best wishes,
Chris.

begin:vcard
fn:Mitterer, Christoph Anton
n:Mitterer;Christoph Anton
email;internet:[EMAIL PROTECTED]
x-mozilla-html:TRUE
version:2.1
end:vcard

Re: data corruption with nvidia chipsets and IDE/SATA drives // memory hole mapping related bug?!

[Karsten Weiss]

On Wed, 13 Dec 2006, Chris Wedgwood wrote:

> > Any ideas why iommu=disabled in the bios does not solve the issue?
> 
> The kernel will still use the IOMMU if the BIOS doesn't set it up if
> it can, check your dmesg for IOMMU strings, there might be something
> printed to this effect.

FWIW: As far as I understand the linux kernel code (I am no kernel 
developer so please correct me if I am wrong) the PCI dma mapping code is 
abstracted by struct dma_mapping_ops. I.e. there are currently four 
possible implementations for x86_64 (see linux-2.6/arch/x86_64/kernel/)

1. pci-nommu.c : no IOMMU at all (e.g. because you have < 4 GB memory)
   Kernel boot message: "PCI-DMA: Disabling IOMMU."

2. pci-gart.c : (AMD) Hardware-IOMMU.
   Kernel boot message: "PCI-DMA: using GART IOMMU" (this message
   first appeared in 2.6.16)

3. pci-swiotlb.c : Software-IOMMU (used e.g. if there is no hw iommu)
   Kernel boot message: "PCI-DMA: Using software bounce buffering 
   for IO (SWIOTLB)"

4. pci-calgary.c : Calgary HW-IOMMU from IBM; used in pSeries servers. 
   This HW-IOMMU supports dma address mapping with memory proctection,
   etc.
   Kernel boot message: "PCI-DMA: Using Calgary IOMMU" (since 2.6.18!)

What all this means is that you can use "dmesg|grep ^PCI-DMA:" to see 
which implementation your kernel is currently using.

As far as our problem machines are concerned the "PCI-DMA: using GART 
IOMMU" case is broken (data corruption). But both "PCI-DMA: Disabling 
IOMMU" (trigged with mem=2g) and "PCI-DMA: Using software bounce buffering 
for IO (SWIOTLB)" (triggered with iommu=soft) are stable.

BTW: It would be really great if this area of the kernel would get some 
more and better documentation. The information at 
linux-2.6/Documentation/x86_64/boot_options.txt is very terse. I had to 
read the code to get a *rough* idea what all the "iommu=" options 
actually do and how they interact.
 
> > 1) And does this now mean that there's an error in the hardware
> > (chipset or CPU/memcontroller)?
> 
> My guess is it's a kernel bug, I don't know for certain.  Perhaps we
> shaould start making a more comprehensive list of affected kernels &
> CPUs?

BTW: Did someone already open an official bug at 
http://bugzilla.kernel.org ?

Best regards,
Karsten

-- 
__creating IT solutions
Dipl.-Inf. Karsten Weiss   science + computing ag
phone:+49 7071 9457 452Hagellocher Weg 73
teamline: +49 7071 9457 68172070 Tuebingen, Germany
email:[EMAIL PROTECTED] www.science-computing.de

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: data corruption with nvidia chipsets and IDE/SATA drives // memory hole mapping related bug?!

[Christoph Anton Mitterer]

Erik Andersen wrote:
> I just realized that booting with "iommu=soft" makes my pcHDTV
> HD5500 DVB cards not work.  Time to go back to disabling the
> memhole and losing 1 GB.  :-(
Crazy,...
I have a Hauppauge Nova-T 500 DualDVB-T card,... I'll check it later if
I have the same problem and will inform you (please remember me if I
forget ;) )


Chris.
begin:vcard
fn:Mitterer, Christoph Anton
n:Mitterer;Christoph Anton
email;internet:[EMAIL PROTECTED]
x-mozilla-html:TRUE
version:2.1
end:vcard

Re: data corruption with nvidia chipsets and IDE/SATA drives // memory hole mapping related bug?!

[Erik Andersen]

On Mon Dec 11, 2006 at 10:24:02AM +0100, Karsten Weiss wrote:
> We could not reproduce the data corruption anymore if we boot
> the machines with the kernel parameter "iommu=soft" i.e. if we
> use software bounce buffering instead of the hw-iommu.

I just realized that booting with "iommu=soft" makes my pcHDTV
HD5500 DVB cards not work.  Time to go back to disabling the
memhole and losing 1 GB.  :-(

 -Erik

--
Erik B. Andersen http://codepoet-consulting.com/
--This message was written using 73% post-consumer electrons--
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: data corruption with nvidia chipsets and IDE/SATA drives // memory hole mapping related bug?!

[Karsten Weiss]

On Wed, 13 Dec 2006, Erik Andersen wrote:

> On Mon Dec 11, 2006 at 10:24:02AM +0100, Karsten Weiss wrote:
> > Last week we did some more testing with the following result:
> > 
> > We could not reproduce the data corruption anymore if we boot the machines 
> > with the kernel parameter "iommu=soft" i.e. if we use software bounce 
> > buffering instead of the hw-iommu. (As mentioned before, booting with 
> > mem=2g works fine, too, because this disables the iommu altogether.)
> > 
> > I.e. on these systems the data corruption only happens if the hw-iommu 
> > (PCI-GART) of the Opteron CPUs is in use.
> > 
> > Christoph, Erik, Chris: I would appreciate if you would test and hopefully 
> > confirm this workaround, too.
> 
> What did you set the BIOS to when testing this setting?
> Memory Hole enabled?  IOMMU enabled?

"Memory hole mapping" was set to "hardware". With "disabled" we only
see 3 of our 4 GB memory.

Best regards,
Karsten

-- 
__creating IT solutions
Dipl.-Inf. Karsten Weiss   science + computing ag
phone:+49 7071 9457 452Hagellocher Weg 73
teamline: +49 7071 9457 68172070 Tuebingen, Germany
email:[EMAIL PROTECTED] www.science-computing.de

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: data corruption with nvidia chipsets and IDE/SATA drives // memory hole mapping related bug?!

[Erik Andersen]

On Mon Dec 11, 2006 at 10:24:02AM +0100, Karsten Weiss wrote:
> Last week we did some more testing with the following result:
> 
> We could not reproduce the data corruption anymore if we boot the machines 
> with the kernel parameter "iommu=soft" i.e. if we use software bounce 
> buffering instead of the hw-iommu. (As mentioned before, booting with 
> mem=2g works fine, too, because this disables the iommu altogether.)
> 
> I.e. on these systems the data corruption only happens if the hw-iommu 
> (PCI-GART) of the Opteron CPUs is in use.
> 
> Christoph, Erik, Chris: I would appreciate if you would test and hopefully 
> confirm this workaround, too.

What did you set the BIOS to when testing this setting?
Memory Hole enabled?  IOMMU enabled?

 -Erik

--
Erik B. Andersen http://codepoet-consulting.com/
--This message was written using 73% post-consumer electrons--
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: data corruption with nvidia chipsets and IDE/SATA drives // memory hole mapping related bug?!

[Christoph Anton Mitterer]

Chris Wedgwood wrote:
>> Did anyone made any test under Windows? I cannot set there
>> iommu=soft, can I?
>> 
> Windows never uses the hardware iommu, so it's always doing the
> equivalent on iommu=soft
>   
That would mean that I'm not able to reproduce the issue unter windows,
right?
Does that apply for all versions (up to and including Vista).

Don't understand me wrong,.. I don't use Windows (expect for upgrading
my Plextor firmware and EAC ;) )... but I ask because the more
information we get (even if it's not Linux specific) the more steps we
can take ;)

Chris.

begin:vcard
fn:Mitterer, Christoph Anton
n:Mitterer;Christoph Anton
email;internet:[EMAIL PROTECTED]
x-mozilla-html:TRUE
version:2.1
end:vcard

Re: data corruption with nvidia chipsets and IDE/SATA drives // memory hole mapping related bug?!

[Christoph Anton Mitterer]

Karsten Weiss wrote:
> "Memory hole mapping" was set to "hardware". With "disabled" we only
> see 3 of our 4 GB memory.
>   
That sounds reasonable,... I even only see 2,5 GB,.. as my memhole takes
1536 MB (don't ask me which PCI device needs that much address space ;) )
begin:vcard
fn:Mitterer, Christoph Anton
n:Mitterer;Christoph Anton
email;internet:[EMAIL PROTECTED]
x-mozilla-html:TRUE
version:2.1
end:vcard

Re: data corruption with nvidia chipsets and IDE/SATA drives // memory hole mapping related bug?!

[Christoph Anton Mitterer]

Karsten Weiss wrote:

> Of course, the big question "Why does the hardware iommu *not*
> work on those machines?" still remains.
>   
I'm going to check AMDs errata docs these days,.. perhaps I find
something that relates. But I'd ask you to do the same as I  don't
consider myself as an expert in these issues ;-)

Chris Wedgwood said that iommu isn't used unter windows at all,.. so I
think the following three solutions would be possible:
- error in the Opteron (memory controller)
- error in the Nvidia chipsets
- error in the kernel


> I have also tried setting "memory hole mapping" to "disabled"
> instead of "hardware" on some of the machines and this *seems*
> to work stable, too. However, I did only test it on about a
> dozen machines because this bios setting costs us 1 GB memory
> (and iommu=soft does not).
>   
Yes... loosing so much memory is a big drawback,.. anyway it would be
great if you can make some more extensive tests that we'd be able to say
if memholemapping=disabled in the BIOS really solves that issue, too, or
not.

Does anyone know how memhole mapping in the BIOS relates to the iommu stuff?
Is it likely or explainable that both would sovle the issue?


> BTW: Maybe I should also mention that other machines types
> (e.g. the HP xw9300 dual opteron workstations) which also use a
> NVIDIA chipset and Opterons never had this problem as far as I
> know.
>   
Uhm,.. that's really strange,... I would have thought that this would
affect all systems that uses either the (mayby) buggy nforce chipset,..
or the (mayby) buggy Opteron.

Did those systems have exactly the same Nvidia-Type? Same question for
the CPU (perhaps the issue only occurs for a speciffic stepping)
Again I have:
nforce professional 2200
nforce professional 2050
Opteron model 275 (stepping E6)


btw: I think that is already clear but again:
Both "solutions" solve the problem for me:
Either
- memhole mapping=disabled in the BIOS (but you loose some memory)
- without any iommu= option for the kernel
or
- memhole mapping=hardware in the BIOS (I suppuse it will work with
software too)
- with iommu=soft for the kernel



Best wishes,
Chris.
begin:vcard
fn:Mitterer, Christoph Anton
n:Mitterer;Christoph Anton
email;internet:[EMAIL PROTECTED]
x-mozilla-html:TRUE
version:2.1
end:vcard

Re: data corruption with nvidia chipsets and IDE/SATA drives // memory hole mapping related bug?!

[Karsten Weiss]

On Wed, 13 Dec 2006, Christoph Anton Mitterer wrote:


Christoph, I will carefully re-read your entire posting and the
included links on Monday and will also try the memory hole
setting.


And did you get out anything new?


As I already mentioned the kernel parameter "iommu=soft" fixes
the data corruption for me. We saw no more data corruption
during a test on 48 machines over the last week-end. Chris
Wedgewood already confirmed that this setting fixed the data
corruption for him, too.

Of course, the big question "Why does the hardware iommu *not*
work on those machines?" still remains.

I have also tried setting "memory hole mapping" to "disabled"
instead of "hardware" on some of the machines and this *seems*
to work stable, too. However, I did only test it on about a
dozen machines because this bios setting costs us 1 GB memory
(and iommu=soft does not).

BTW: Maybe I should also mention that other machines types
(e.g. the HP xw9300 dual opteron workstations) which also use a
NVIDIA chipset and Opterons never had this problem as far as I
know.

Best regards,
Karsten

--
Dipl.-Inf. Karsten Weiss - http://www.machineroom.de/knweiss
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: data corruption with nvidia chipsets and IDE/SATA drives // memory hole mapping related bug?!

[Chris Wedgwood]

On Wed, Dec 13, 2006 at 08:18:21PM +0100, Christoph Anton Mitterer wrote:

> booting with iommu=soft => works fine
> booting with iommu=noagp => DOESN'T solve the error
> booting with iommu=off => the system doesn't even boot and panics

> When I set IOMMU to disabled in the BIOS the error is not solved-
> I tried to set bigger space for the IOMMU in the BIOS (256MB instead of
> 64MB),.. but it does not solve the problem.

> Any ideas why iommu=disabled in the bios does not solve the issue?

The kernel will still use the IOMMU if the BIOS doesn't set it up if
it can, check your dmesg for IOMMU strings, there might be something
printed to this effect.

> 1) And does this now mean that there's an error in the hardware
> (chipset or CPU/memcontroller)?

My guess is it's a kernel bug, I don't know for certain.  Perhaps we
shaould start making a more comprehensive list of affected kernels &
CPUs?
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: data corruption with nvidia chipsets and IDE/SATA drives // memory hole mapping related bug?!

[Chris Wedgwood]

On Wed, Dec 13, 2006 at 08:20:59PM +0100, Christoph Anton Mitterer wrote:

> Did anyone made any test under Windows? I cannot set there
> iommu=soft, can I?

Windows never uses the hardware iommu, so it's always doing the
equivalent on iommu=soft
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: data corruption with nvidia chipsets and IDE/SATA drives // memory hole mapping related bug?!

[Christoph Anton Mitterer]

Ah and I forgot,...

Did anyone made any test under Windows? I cannot set there iommu=soft,
can I?

Chris.
begin:vcard
fn:Mitterer, Christoph Anton
n:Mitterer;Christoph Anton
email;internet:[EMAIL PROTECTED]
x-mozilla-html:TRUE
version:2.1
end:vcard

Re: data corruption with nvidia chipsets and IDE/SATA drives // memory hole mapping related bug?!

[Christoph Anton Mitterer]

Karsten Weiss wrote:
> Last week we did some more testing with the following result:
>
> We could not reproduce the data corruption anymore if we boot the machines 
> with the kernel parameter "iommu=soft" i.e. if we use software bounce 
> buffering instead of the hw-iommu. (As mentioned before, booting with 
> mem=2g works fine, too, because this disables the iommu altogether.)
>   
I can confirm this,...
booting with mem=2G => works fine,...

(all of the following tests were made with memory hole mapping=hardware
in the BIOS,.. so I could access my full ram):
booting with iommu=soft => works fine
booting with iommu=noagp => DOESN'T solve the error
booting with iommu=off => the system doesn't even boot and panics

When I set IOMMU to disabled in the BIOS the error is not solved-
I tried to set bigger space for the IOMMU in the BIOS (256MB instead of
64MB),.. but it does not solve the problem.

Any ideas why iommu=disabled in the bios does not solve the issue?

> I.e. on these systems the data corruption only happens if the hw-iommu 
> (PCI-GART) of the Opteron CPUs is in use.
>   
1) And does this now mean that there's an error in the hardware (chipset
or CPU/memcontroller)?


> Christoph, Erik, Chris: I would appreciate if you would test and hopefully 
> confirm this workaround, too.
>   
Yes I can absolutely confirm this...
Do my additional tests help you?



Do you have any ideas why the issue doesn't occur (even with memhole
mapping=hardware in the bios and no iommu=soft at kernel command line)
when dma is disabled for the disks (or a slower dma mode is used)?


Chris.
begin:vcard
fn:Mitterer, Christoph Anton
n:Mitterer;Christoph Anton
email;internet:[EMAIL PROTECTED]
x-mozilla-html:TRUE
version:2.1
end:vcard

Re: data corruption with nvidia chipsets and IDE/SATA drives // memory hole mapping related bug?!

[Christoph Anton Mitterer]

Karsten Weiss wrote:
> Here's a diff of a corrupted and a good file written during our
> testcase:
>
> ("-" == corrupted file, "+" == good file)
> ...
>   009f2ff0  67 2a 4c c4 6d 9d 34 44  ad e6 3c 45 05 9a 4d c4  
> |g*L.m.4D.. -009f3000  39 60 e6 44 20 ab 46 44  56 aa 46 44 c2 35 e6 44  |9.D .FDV.FD.5.D|
> 
> +009f3ff0  f3 55 92 44 c1 10 6c 45  5e 12 a0 c3 60 31 93 44  |.U.D..lE^...1.D|
>   009f4000  88 cd 6b 45 c1 6d cd c3  00 a5 8b 44 f2 ac 6b 45  
> |..kE.m.D..kE|
>   
Well as I told in my mails to the list I made the experience that not
all bytes of the corrupted area are invalid,.. but only some,.. while it
seems that in you diff ALL the bytes are wrong, right?


> Please notice:
>
> a) the corruption begins at a page boundary
> b) the corrupted byte range is a single memory page and
> c) almost every fourth byte is set to 0x44 in the corrupted case
> (but the other bytes changed, too)
>
> To me this looks as if a wrong memory page got written into the
> file.
>   
Hmm and do you have any ideas what's the reason for all this? Defect in
the nforce chipset? Or even in the CPU (the Opterons do have integrated
memory controllers).


> >From our testing I can also tell that the data corruption does
> *not* appear at all when we are booting the nodes with mem=2G.
> However, when we are using all the 4GB the data corruption
> shows up - but not everytime and thus not on all nodes.
> Sometimes a node runs for ours without any problem. That's why
> we are testing on 32 nodes in parallel most of the time. I have
> the impression that it has something to do with physical memory
> layout of the running processes.
>   
Hmm maybe,.. but I have absolutely no idea ;)


> Please also notice that this is a silent data corruption. I.e.
> there are no error or warning messages in the kernel log or the
> mce log at all.
>   
Yes I can confirm that.


> Christoph, I will carefully re-read your entire posting and the
> included links on Monday and will also try the memory hole
> setting.
>   
And did you get out anything new?
begin:vcard
fn:Mitterer, Christoph Anton
n:Mitterer;Christoph Anton
email;internet:[EMAIL PROTECTED]
x-mozilla-html:TRUE
version:2.1
end:vcard

Re: data corruption with nvidia chipsets and IDE/SATA drives // memory hole mapping related bug?!

[Christoph Anton Mitterer]

Karsten Weiss wrote:
 Here's a diff of a corrupted and a good file written during our
 testcase:

 (- == corrupted file, + == good file)
 ...
   009f2ff0  67 2a 4c c4 6d 9d 34 44  ad e6 3c 45 05 9a 4d c4  
 |g*L.m.4D..E..M.|
 -009f3000  39 60 e6 44 20 ab 46 44  56 aa 46 44 c2 35 e6 44  |9.D .FDV.FD.5.D|
 
 +009f3ff0  f3 55 92 44 c1 10 6c 45  5e 12 a0 c3 60 31 93 44  |.U.D..lE^...1.D|
   009f4000  88 cd 6b 45 c1 6d cd c3  00 a5 8b 44 f2 ac 6b 45  
 |..kE.m.D..kE|
   
Well as I told in my mails to the list I made the experience that not
all bytes of the corrupted area are invalid,.. but only some,.. while it
seems that in you diff ALL the bytes are wrong, right?


 Please notice:

 a) the corruption begins at a page boundary
 b) the corrupted byte range is a single memory page and
 c) almost every fourth byte is set to 0x44 in the corrupted case
 (but the other bytes changed, too)

 To me this looks as if a wrong memory page got written into the
 file.
   
Hmm and do you have any ideas what's the reason for all this? Defect in
the nforce chipset? Or even in the CPU (the Opterons do have integrated
memory controllers).


 From our testing I can also tell that the data corruption does
 *not* appear at all when we are booting the nodes with mem=2G.
 However, when we are using all the 4GB the data corruption
 shows up - but not everytime and thus not on all nodes.
 Sometimes a node runs for ours without any problem. That's why
 we are testing on 32 nodes in parallel most of the time. I have
 the impression that it has something to do with physical memory
 layout of the running processes.
   
Hmm maybe,.. but I have absolutely no idea ;)


 Please also notice that this is a silent data corruption. I.e.
 there are no error or warning messages in the kernel log or the
 mce log at all.
   
Yes I can confirm that.


 Christoph, I will carefully re-read your entire posting and the
 included links on Monday and will also try the memory hole
 setting.
   
And did you get out anything new?
begin:vcard
fn:Mitterer, Christoph Anton
n:Mitterer;Christoph Anton
email;internet:[EMAIL PROTECTED]
x-mozilla-html:TRUE
version:2.1
end:vcard

Re: data corruption with nvidia chipsets and IDE/SATA drives // memory hole mapping related bug?!

[Christoph Anton Mitterer]

Karsten Weiss wrote:
 Last week we did some more testing with the following result:

 We could not reproduce the data corruption anymore if we boot the machines 
 with the kernel parameter iommu=soft i.e. if we use software bounce 
 buffering instead of the hw-iommu. (As mentioned before, booting with 
 mem=2g works fine, too, because this disables the iommu altogether.)
   
I can confirm this,...
booting with mem=2G = works fine,...

(all of the following tests were made with memory hole mapping=hardware
in the BIOS,.. so I could access my full ram):
booting with iommu=soft = works fine
booting with iommu=noagp = DOESN'T solve the error
booting with iommu=off = the system doesn't even boot and panics

When I set IOMMU to disabled in the BIOS the error is not solved-
I tried to set bigger space for the IOMMU in the BIOS (256MB instead of
64MB),.. but it does not solve the problem.

Any ideas why iommu=disabled in the bios does not solve the issue?

 I.e. on these systems the data corruption only happens if the hw-iommu 
 (PCI-GART) of the Opteron CPUs is in use.
   
1) And does this now mean that there's an error in the hardware (chipset
or CPU/memcontroller)?


 Christoph, Erik, Chris: I would appreciate if you would test and hopefully 
 confirm this workaround, too.
   
Yes I can absolutely confirm this...
Do my additional tests help you?



Do you have any ideas why the issue doesn't occur (even with memhole
mapping=hardware in the bios and no iommu=soft at kernel command line)
when dma is disabled for the disks (or a slower dma mode is used)?


Chris.
begin:vcard
fn:Mitterer, Christoph Anton
n:Mitterer;Christoph Anton
email;internet:[EMAIL PROTECTED]
x-mozilla-html:TRUE
version:2.1
end:vcard

Re: data corruption with nvidia chipsets and IDE/SATA drives // memory hole mapping related bug?!

[Christoph Anton Mitterer]

Ah and I forgot,...

Did anyone made any test under Windows? I cannot set there iommu=soft,
can I?

Chris.
begin:vcard
fn:Mitterer, Christoph Anton
n:Mitterer;Christoph Anton
email;internet:[EMAIL PROTECTED]
x-mozilla-html:TRUE
version:2.1
end:vcard

Re: data corruption with nvidia chipsets and IDE/SATA drives // memory hole mapping related bug?!

[Chris Wedgwood]

On Wed, Dec 13, 2006 at 08:20:59PM +0100, Christoph Anton Mitterer wrote:

 Did anyone made any test under Windows? I cannot set there
 iommu=soft, can I?

Windows never uses the hardware iommu, so it's always doing the
equivalent on iommu=soft
-
To unsubscribe from this list: send the line unsubscribe linux-kernel in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: data corruption with nvidia chipsets and IDE/SATA drives // memory hole mapping related bug?!

[Chris Wedgwood]

On Wed, Dec 13, 2006 at 08:18:21PM +0100, Christoph Anton Mitterer wrote:

 booting with iommu=soft = works fine
 booting with iommu=noagp = DOESN'T solve the error
 booting with iommu=off = the system doesn't even boot and panics

 When I set IOMMU to disabled in the BIOS the error is not solved-
 I tried to set bigger space for the IOMMU in the BIOS (256MB instead of
 64MB),.. but it does not solve the problem.

 Any ideas why iommu=disabled in the bios does not solve the issue?

The kernel will still use the IOMMU if the BIOS doesn't set it up if
it can, check your dmesg for IOMMU strings, there might be something
printed to this effect.

 1) And does this now mean that there's an error in the hardware
 (chipset or CPU/memcontroller)?

My guess is it's a kernel bug, I don't know for certain.  Perhaps we
shaould start making a more comprehensive list of affected kernels 
CPUs?
-
To unsubscribe from this list: send the line unsubscribe linux-kernel in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: data corruption with nvidia chipsets and IDE/SATA drives // memory hole mapping related bug?!

[Christoph Anton Mitterer]

Karsten Weiss wrote:

 Of course, the big question Why does the hardware iommu *not*
 work on those machines? still remains.
   
I'm going to check AMDs errata docs these days,.. perhaps I find
something that relates. But I'd ask you to do the same as I  don't
consider myself as an expert in these issues ;-)

Chris Wedgwood said that iommu isn't used unter windows at all,.. so I
think the following three solutions would be possible:
- error in the Opteron (memory controller)
- error in the Nvidia chipsets
- error in the kernel


 I have also tried setting memory hole mapping to disabled
 instead of hardware on some of the machines and this *seems*
 to work stable, too. However, I did only test it on about a
 dozen machines because this bios setting costs us 1 GB memory
 (and iommu=soft does not).
   
Yes... loosing so much memory is a big drawback,.. anyway it would be
great if you can make some more extensive tests that we'd be able to say
if memholemapping=disabled in the BIOS really solves that issue, too, or
not.

Does anyone know how memhole mapping in the BIOS relates to the iommu stuff?
Is it likely or explainable that both would sovle the issue?


 BTW: Maybe I should also mention that other machines types
 (e.g. the HP xw9300 dual opteron workstations) which also use a
 NVIDIA chipset and Opterons never had this problem as far as I
 know.
   
Uhm,.. that's really strange,... I would have thought that this would
affect all systems that uses either the (mayby) buggy nforce chipset,..
or the (mayby) buggy Opteron.

Did those systems have exactly the same Nvidia-Type? Same question for
the CPU (perhaps the issue only occurs for a speciffic stepping)
Again I have:
nforce professional 2200
nforce professional 2050
Opteron model 275 (stepping E6)


btw: I think that is already clear but again:
Both solutions solve the problem for me:
Either
- memhole mapping=disabled in the BIOS (but you loose some memory)
- without any iommu= option for the kernel
or
- memhole mapping=hardware in the BIOS (I suppuse it will work with
software too)
- with iommu=soft for the kernel



Best wishes,
Chris.
begin:vcard
fn:Mitterer, Christoph Anton
n:Mitterer;Christoph Anton
email;internet:[EMAIL PROTECTED]
x-mozilla-html:TRUE
version:2.1
end:vcard

Re: data corruption with nvidia chipsets and IDE/SATA drives // memory hole mapping related bug?!

[Karsten Weiss]

On Wed, 13 Dec 2006, Christoph Anton Mitterer wrote:


Christoph, I will carefully re-read your entire posting and the
included links on Monday and will also try the memory hole
setting.


And did you get out anything new?


As I already mentioned the kernel parameter iommu=soft fixes
the data corruption for me. We saw no more data corruption
during a test on 48 machines over the last week-end. Chris
Wedgewood already confirmed that this setting fixed the data
corruption for him, too.

Of course, the big question Why does the hardware iommu *not*
work on those machines? still remains.

I have also tried setting memory hole mapping to disabled
instead of hardware on some of the machines and this *seems*
to work stable, too. However, I did only test it on about a
dozen machines because this bios setting costs us 1 GB memory
(and iommu=soft does not).

BTW: Maybe I should also mention that other machines types
(e.g. the HP xw9300 dual opteron workstations) which also use a
NVIDIA chipset and Opterons never had this problem as far as I
know.

Best regards,
Karsten

--
Dipl.-Inf. Karsten Weiss - http://www.machineroom.de/knweiss
-
To unsubscribe from this list: send the line unsubscribe linux-kernel in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: data corruption with nvidia chipsets and IDE/SATA drives // memory hole mapping related bug?!

[Christoph Anton Mitterer]

Chris Wedgwood wrote:
 Did anyone made any test under Windows? I cannot set there
 iommu=soft, can I?
 
 Windows never uses the hardware iommu, so it's always doing the
 equivalent on iommu=soft
   
That would mean that I'm not able to reproduce the issue unter windows,
right?
Does that apply for all versions (up to and including Vista).

Don't understand me wrong,.. I don't use Windows (expect for upgrading
my Plextor firmware and EAC ;) )... but I ask because the more
information we get (even if it's not Linux specific) the more steps we
can take ;)

Chris.

begin:vcard
fn:Mitterer, Christoph Anton
n:Mitterer;Christoph Anton
email;internet:[EMAIL PROTECTED]
x-mozilla-html:TRUE
version:2.1
end:vcard

Re: data corruption with nvidia chipsets and IDE/SATA drives // memory hole mapping related bug?!

[Erik Andersen]

On Mon Dec 11, 2006 at 10:24:02AM +0100, Karsten Weiss wrote:
 Last week we did some more testing with the following result:
 
 We could not reproduce the data corruption anymore if we boot the machines 
 with the kernel parameter iommu=soft i.e. if we use software bounce 
 buffering instead of the hw-iommu. (As mentioned before, booting with 
 mem=2g works fine, too, because this disables the iommu altogether.)
 
 I.e. on these systems the data corruption only happens if the hw-iommu 
 (PCI-GART) of the Opteron CPUs is in use.
 
 Christoph, Erik, Chris: I would appreciate if you would test and hopefully 
 confirm this workaround, too.

What did you set the BIOS to when testing this setting?
Memory Hole enabled?  IOMMU enabled?

 -Erik

--
Erik B. Andersen http://codepoet-consulting.com/
--This message was written using 73% post-consumer electrons--
-
To unsubscribe from this list: send the line unsubscribe linux-kernel in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: data corruption with nvidia chipsets and IDE/SATA drives // memory hole mapping related bug?!

[Karsten Weiss]

On Wed, 13 Dec 2006, Erik Andersen wrote:

 On Mon Dec 11, 2006 at 10:24:02AM +0100, Karsten Weiss wrote:
  Last week we did some more testing with the following result:
  
  We could not reproduce the data corruption anymore if we boot the machines 
  with the kernel parameter iommu=soft i.e. if we use software bounce 
  buffering instead of the hw-iommu. (As mentioned before, booting with 
  mem=2g works fine, too, because this disables the iommu altogether.)
  
  I.e. on these systems the data corruption only happens if the hw-iommu 
  (PCI-GART) of the Opteron CPUs is in use.
  
  Christoph, Erik, Chris: I would appreciate if you would test and hopefully 
  confirm this workaround, too.
 
 What did you set the BIOS to when testing this setting?
 Memory Hole enabled?  IOMMU enabled?

Memory hole mapping was set to hardware. With disabled we only
see 3 of our 4 GB memory.

Best regards,
Karsten

-- 
__creating IT solutions
Dipl.-Inf. Karsten Weiss   science + computing ag
phone:+49 7071 9457 452Hagellocher Weg 73
teamline: +49 7071 9457 68172070 Tuebingen, Germany
email:[EMAIL PROTECTED] www.science-computing.de

-
To unsubscribe from this list: send the line unsubscribe linux-kernel in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: data corruption with nvidia chipsets and IDE/SATA drives // memory hole mapping related bug?!

[Erik Andersen]

On Mon Dec 11, 2006 at 10:24:02AM +0100, Karsten Weiss wrote:
 We could not reproduce the data corruption anymore if we boot
 the machines with the kernel parameter iommu=soft i.e. if we
 use software bounce buffering instead of the hw-iommu.

I just realized that booting with iommu=soft makes my pcHDTV
HD5500 DVB cards not work.  Time to go back to disabling the
memhole and losing 1 GB.  :-(

 -Erik

--
Erik B. Andersen http://codepoet-consulting.com/
--This message was written using 73% post-consumer electrons--
-
To unsubscribe from this list: send the line unsubscribe linux-kernel in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: data corruption with nvidia chipsets and IDE/SATA drives // memory hole mapping related bug?!

[Christoph Anton Mitterer]

Erik Andersen wrote:
 I just realized that booting with iommu=soft makes my pcHDTV
 HD5500 DVB cards not work.  Time to go back to disabling the
 memhole and losing 1 GB.  :-(
Crazy,...
I have a Hauppauge Nova-T 500 DualDVB-T card,... I'll check it later if
I have the same problem and will inform you (please remember me if I
forget ;) )


Chris.
begin:vcard
fn:Mitterer, Christoph Anton
n:Mitterer;Christoph Anton
email;internet:[EMAIL PROTECTED]
x-mozilla-html:TRUE
version:2.1
end:vcard

Re: data corruption with nvidia chipsets and IDE/SATA drives // memory hole mapping related bug?!

[Karsten Weiss]

On Wed, 13 Dec 2006, Chris Wedgwood wrote:

  Any ideas why iommu=disabled in the bios does not solve the issue?
 
 The kernel will still use the IOMMU if the BIOS doesn't set it up if
 it can, check your dmesg for IOMMU strings, there might be something
 printed to this effect.

FWIW: As far as I understand the linux kernel code (I am no kernel 
developer so please correct me if I am wrong) the PCI dma mapping code is 
abstracted by struct dma_mapping_ops. I.e. there are currently four 
possible implementations for x86_64 (see linux-2.6/arch/x86_64/kernel/)

1. pci-nommu.c : no IOMMU at all (e.g. because you have  4 GB memory)
   Kernel boot message: PCI-DMA: Disabling IOMMU.

2. pci-gart.c : (AMD) Hardware-IOMMU.
   Kernel boot message: PCI-DMA: using GART IOMMU (this message
   first appeared in 2.6.16)

3. pci-swiotlb.c : Software-IOMMU (used e.g. if there is no hw iommu)
   Kernel boot message: PCI-DMA: Using software bounce buffering 
   for IO (SWIOTLB)

4. pci-calgary.c : Calgary HW-IOMMU from IBM; used in pSeries servers. 
   This HW-IOMMU supports dma address mapping with memory proctection,
   etc.
   Kernel boot message: PCI-DMA: Using Calgary IOMMU (since 2.6.18!)

What all this means is that you can use dmesg|grep ^PCI-DMA: to see 
which implementation your kernel is currently using.

As far as our problem machines are concerned the PCI-DMA: using GART 
IOMMU case is broken (data corruption). But both PCI-DMA: Disabling 
IOMMU (trigged with mem=2g) and PCI-DMA: Using software bounce buffering 
for IO (SWIOTLB) (triggered with iommu=soft) are stable.

BTW: It would be really great if this area of the kernel would get some 
more and better documentation. The information at 
linux-2.6/Documentation/x86_64/boot_options.txt is very terse. I had to 
read the code to get a *rough* idea what all the iommu= options 
actually do and how they interact.
 
  1) And does this now mean that there's an error in the hardware
  (chipset or CPU/memcontroller)?
 
 My guess is it's a kernel bug, I don't know for certain.  Perhaps we
 shaould start making a more comprehensive list of affected kernels 
 CPUs?

BTW: Did someone already open an official bug at 
http://bugzilla.kernel.org ?

Best regards,
Karsten

-- 
__creating IT solutions
Dipl.-Inf. Karsten Weiss   science + computing ag
phone:+49 7071 9457 452Hagellocher Weg 73
teamline: +49 7071 9457 68172070 Tuebingen, Germany
email:[EMAIL PROTECTED] www.science-computing.de

-
To unsubscribe from this list: send the line unsubscribe linux-kernel in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: data corruption with nvidia chipsets and IDE/SATA drives // memory hole mapping related bug?!

[Christoph Anton Mitterer]

Lennart Sorensen wrote:
 I upgrade my plextor firmware using linux.  pxupdate for most devices,
 and pxfw for new drivers (like the PX760).  Works perfectly for me.  It
 is one of the reasons I buy plextors.
Yes I know about it,.. although never tested it,... anyway the main
reason for Windows is Exact Audio Copy (but Andre Wiehthoff is working
on a C port :-D )

Unfortunately my PX760 seems to be defect,.. posted about the issue to
lkml but no success :-(

Best wishes,
Chris.

begin:vcard
fn:Mitterer, Christoph Anton
n:Mitterer;Christoph Anton
email;internet:[EMAIL PROTECTED]
x-mozilla-html:TRUE
version:2.1
end:vcard

Re: data corruption with nvidia chipsets and IDE/SATA drives // memory hole mapping related bug?!

[Lennart Sorensen]

On Wed, Dec 13, 2006 at 08:57:23PM +0100, Christoph Anton Mitterer wrote:
 Don't understand me wrong,.. I don't use Windows (expect for upgrading
 my Plextor firmware and EAC ;) )... but I ask because the more
 information we get (even if it's not Linux specific) the more steps we
 can take ;)

I upgrade my plextor firmware using linux.  pxupdate for most devices,
and pxfw for new drivers (like the PX760).  Works perfectly for me.  It
is one of the reasons I buy plextors.

--
Len Sorensen
-
To unsubscribe from this list: send the line unsubscribe linux-kernel in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: data corruption with nvidia chipsets and IDE/SATA drives // memory hole mapping related bug?!

[Christoph Anton Mitterer]

Hi.

I've just looked for some kernel config options that might relate to our
issue:


1)
Old style AMD Opteron NUMA detection (CONFIG_K8_NUMA)
Enable K8 NUMA node topology detection.  You should say Y here if you
have a multi processor AMD K8 system. This uses an old method to read
the NUMA configuration directly from the builtin Northbridge of Opteron.
It is recommended to use X86_64_ACPI_NUMA instead, which also takes
priority if both are compiled in.

ACPI NUMA detection (CONFIG_X86_64_ACPI_NUMA)
Enable ACPI SRAT based node topology detection.

What should one select for the Opterons? And is it possible that this
has something to do with our datacorruption error?


2)
The same two questions for the memory model (Discontiguous or Sparse)



3)
The same two questions for CONFIG_MIGRATION ()


4)
And does someone know if the nforce/opteron iommu requires IBM Calgary
IOMMU support?



This is unrelated to our issue,.. but it would be nice if some of your
could send me their .config,.. I'd like to compare them with my own and
see if I could something tweak or so.
(Of course only people with 2x DualCore Systems ;) )



Chris.
begin:vcard
fn:Mitterer, Christoph Anton
n:Mitterer;Christoph Anton
email;internet:[EMAIL PROTECTED]
x-mozilla-html:TRUE
version:2.1
end:vcard

amd64 iommu causing corruption? (was Re: data corruption with nvidia chipsets and IDE/SATA drives // memory hole mapping related bug?!)

[Chris Wedgwood]

On Mon, Dec 11, 2006 at 10:24:02AM +0100, Karsten Weiss wrote:

> We could not reproduce the data corruption anymore if we boot the
> machines with the kernel parameter "iommu=soft" i.e. if we use
> software bounce buffering instead of the hw-iommu. (As mentioned
> before, booting with mem=2g works fine, too, because this disables
> the iommu altogether.)

I can confirm this also seems to be the case for me, I'm still doing
more testing to confirm this.  But it would seem:

nforce4, transfer of a large mount of data with 4GB+ of RAM I get some
corruption.  This is present on both the nv SATA and also Sil 3112
connected drives.

Using iommu=soft so far seems to be working without any corruption.



I still need to do more testing on other machines which have less
memory (so the IOMMU won't be in use there either) and see if there
are problems there.
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: data corruption with nvidia chipsets and IDE/SATA drives // memory hole mapping related bug?!

[Karsten Weiss]

On Sat, 2 Dec 2006, Karsten Weiss wrote:

> On Sat, 2 Dec 2006, Christoph Anton Mitterer wrote:
> 
> > I found a severe bug mainly by fortune because it occurs very rarely.
> > My test looks like the following: I have about 30GB of testing data on
> 
> This sounds very familiar! One of the Linux compute clusters I
> administer at work is a 336 node system consisting of the
> following components:
> 
> * 2x Dual-Core AMD Opteron 275
> * Tyan S2891 mainboard
> * Hitachi HDS728080PLA380 harddisk
> * 4 GB RAM (some nodes have 8 GB) - intensively tested with
>   memtest86+
> * SUSE 9.3 x86_64 (kernel 2.6.11.4-21.14-smp) - But I've also
>   e.g. tried the latest openSUSE 10.2 RC1+ kernel 2.6.18.2-33 which
>   makes no difference.
> 
> We are running LS-Dyna on these machines and discovered a
> testcase which shows a similar data corruption. So I can
> confirm that the problem is for real an not a hardware defect
> of a single machine!

Last week we did some more testing with the following result:

We could not reproduce the data corruption anymore if we boot the machines 
with the kernel parameter "iommu=soft" i.e. if we use software bounce 
buffering instead of the hw-iommu. (As mentioned before, booting with 
mem=2g works fine, too, because this disables the iommu altogether.)

I.e. on these systems the data corruption only happens if the hw-iommu 
(PCI-GART) of the Opteron CPUs is in use.

Christoph, Erik, Chris: I would appreciate if you would test and hopefully 
confirm this workaround, too.

Best regards,
Karsten

-- 
__creating IT solutions
Dipl.-Inf. Karsten Weiss   science + computing ag
phone:+49 7071 9457 452Hagellocher Weg 73
teamline: +49 7071 9457 68172070 Tuebingen, Germany
email:[EMAIL PROTECTED] www.science-computing.de

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: data corruption with nvidia chipsets and IDE/SATA drives // memory hole mapping related bug?!

[Karsten Weiss]

On Sat, 2 Dec 2006, Karsten Weiss wrote:

 On Sat, 2 Dec 2006, Christoph Anton Mitterer wrote:
 
  I found a severe bug mainly by fortune because it occurs very rarely.
  My test looks like the following: I have about 30GB of testing data on
 
 This sounds very familiar! One of the Linux compute clusters I
 administer at work is a 336 node system consisting of the
 following components:
 
 * 2x Dual-Core AMD Opteron 275
 * Tyan S2891 mainboard
 * Hitachi HDS728080PLA380 harddisk
 * 4 GB RAM (some nodes have 8 GB) - intensively tested with
   memtest86+
 * SUSE 9.3 x86_64 (kernel 2.6.11.4-21.14-smp) - But I've also
   e.g. tried the latest openSUSE 10.2 RC1+ kernel 2.6.18.2-33 which
   makes no difference.
 
 We are running LS-Dyna on these machines and discovered a
 testcase which shows a similar data corruption. So I can
 confirm that the problem is for real an not a hardware defect
 of a single machine!

Last week we did some more testing with the following result:

We could not reproduce the data corruption anymore if we boot the machines 
with the kernel parameter iommu=soft i.e. if we use software bounce 
buffering instead of the hw-iommu. (As mentioned before, booting with 
mem=2g works fine, too, because this disables the iommu altogether.)

I.e. on these systems the data corruption only happens if the hw-iommu 
(PCI-GART) of the Opteron CPUs is in use.

Christoph, Erik, Chris: I would appreciate if you would test and hopefully 
confirm this workaround, too.

Best regards,
Karsten

-- 
__creating IT solutions
Dipl.-Inf. Karsten Weiss   science + computing ag
phone:+49 7071 9457 452Hagellocher Weg 73
teamline: +49 7071 9457 68172070 Tuebingen, Germany
email:[EMAIL PROTECTED] www.science-computing.de

-
To unsubscribe from this list: send the line unsubscribe linux-kernel in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

amd64 iommu causing corruption? (was Re: data corruption with nvidia chipsets and IDE/SATA drives // memory hole mapping related bug?!)

[Chris Wedgwood]

On Mon, Dec 11, 2006 at 10:24:02AM +0100, Karsten Weiss wrote:

 We could not reproduce the data corruption anymore if we boot the
 machines with the kernel parameter iommu=soft i.e. if we use
 software bounce buffering instead of the hw-iommu. (As mentioned
 before, booting with mem=2g works fine, too, because this disables
 the iommu altogether.)

I can confirm this also seems to be the case for me, I'm still doing
more testing to confirm this.  But it would seem:

nforce4, transfer of a large mount of data with 4GB+ of RAM I get some
corruption.  This is present on both the nv SATA and also Sil 3112
connected drives.

Using iommu=soft so far seems to be working without any corruption.



I still need to do more testing on other machines which have less
memory (so the IOMMU won't be in use there either) and see if there
are problems there.
-
To unsubscribe from this list: send the line unsubscribe linux-kernel in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: data corruption with nvidia chipsets and IDE/SATA drives // memory hole mapping related bug?!

[Christoph Anton Mitterer]

Ville Herva wrote:
> I saw something very similar with Via KT133 years ago. Then the culprit was
> botched PCI implementation that sometimes corrupted PCI transfers when there
> was heavy PCI I/O going on. Usually than meant running two disk transfers at
> the same time. Doing heavy network I/O at the time made it more likely
> happen.
Hm I do only on concurrent test,... and network is not used very much
during the tests.


> I used this crude hack:
> http://v.iki.fi/~vherva/tmp/wrchk.c
>   
I'll have a look at it :)


> If the problem in your case is that the PCI transfer gets corrupted when it
> happens to a certain memory area, I guess you could try to binary search for
> the bad spot with the kernel BadRam patches
> http://www.linuxjournal.com/article/4489 (I seem to recall it was possible
> to turn off memory areas with vanilla kernel boot params without a patch,
> but I can't find a reference.)
>   

I know badram,.. but the thing is,.. that it's highly unlikely that my
RAMs are damaged. Many hours of memtest86+ runs did not show any error
(not even ECC errors),...

And why should memhol mapping disabled solve the issue if memory was
damaged? That could only be if the badblocks would be in the address
space used by the memhole

Chris.
begin:vcard
fn:Mitterer, Christoph Anton
n:Mitterer;Christoph Anton
email;internet:[EMAIL PROTECTED]
x-mozilla-html:TRUE
version:2.1
end:vcard

Re: data corruption with nvidia chipsets and IDE/SATA drives // memory hole mapping related bug?!

[Christoph Anton Mitterer]

Ville Herva wrote:
 I saw something very similar with Via KT133 years ago. Then the culprit was
 botched PCI implementation that sometimes corrupted PCI transfers when there
 was heavy PCI I/O going on. Usually than meant running two disk transfers at
 the same time. Doing heavy network I/O at the time made it more likely
 happen.
Hm I do only on concurrent test,... and network is not used very much
during the tests.


 I used this crude hack:
 http://v.iki.fi/~vherva/tmp/wrchk.c
   
I'll have a look at it :)


 If the problem in your case is that the PCI transfer gets corrupted when it
 happens to a certain memory area, I guess you could try to binary search for
 the bad spot with the kernel BadRam patches
 http://www.linuxjournal.com/article/4489 (I seem to recall it was possible
 to turn off memory areas with vanilla kernel boot params without a patch,
 but I can't find a reference.)
   

I know badram,.. but the thing is,.. that it's highly unlikely that my
RAMs are damaged. Many hours of memtest86+ runs did not show any error
(not even ECC errors),...

And why should memhol mapping disabled solve the issue if memory was
damaged? That could only be if the badblocks would be in the address
space used by the memhole

Chris.
begin:vcard
fn:Mitterer, Christoph Anton
n:Mitterer;Christoph Anton
email;internet:[EMAIL PROTECTED]
x-mozilla-html:TRUE
version:2.1
end:vcard

Re: data corruption with nvidia chipsets and IDE/SATA drives // memory hole mapping related bug?!

[Christoph Anton Mitterer]

Chris Wedgwood wrote:
> Heh, I see this also with an Tyan S2866 (nforce4 chipset).  I've been
> aware something is a miss for a while because if I transfer about 40GB
> of data from one machine to another there are checksum mismatches and
> some files have to be transfered again.
>   
It seems that this may be occur on _all_ nvidia chipsets (of course I'm
talking about mainboard chips ;) )
Which harddisk types to you use (vendor an interface type)

> I've kept quite about it so far because it's not been clear what the
> cause is and because i can mostly ignore it now that I checksum all my
> data and check after xfer that it's sane (so I have 2+ copies of all
> this stuff everywhere).
>   
I assume that a large number of users actually experience this error,..
but as it's so rare only few correctly identify it.
Most of them might think that its filesystem related or so.



>> The corrupted data is not one single completely wrong block of data
>> or so,.. but if you look at the area of the file where differences
>> are found,.. than some bytes are ok,.. some are wrong,.. and so on
>> (seems to be randomly).
>> 
>
> For me it seems that a single block in the file would be bad and the
> rest OK --- we I'm talking about 2 random blocks in 30BG or so.
>   
Did you check this with an hex editor? I did it an while the errors were
restricted to one "region" of a file it was not so that that region
was completely corrupted but only some single bytes.
Actually it was that mostly one bit was wrong,..


Chris.
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: data corruption with nvidia chipsets and IDE/SATA drives // memory hole mapping related bug?!

[Christoph Anton Mitterer]

Alan wrote:
> See the thread http://lkml.org/lkml/2006/8/16/305
>   

Hi Alan.

Thanks for your reply. I've read this thread already some weeks ago
but from my limited knowledge I understood, that this was an issue
related to a SCSI adapter or so. Or did I understand this wrong. And as
soon as he removed the card everything was fine.

I don't have any PCI SCSI cards,... (but I have an onboard LSI53C1030
controller).
The only cards I have are:
PCIe bus (two slots):
Asus Nividia 7800GTX based card

PCI bus: no card (one slot):
no card

PCI-X bus A (100MHz) (two slots):
Hauppauge Nova T 500 Dual DVB-T card (which is actually a "normal" PCI
card,.. but should be compatible with PCI-X)
TerraTec Aureon 7.1 Universe Soundcard (which is actually a "normal" PCI
card,.. but should be compatible with PCI-X)

PCI-X bus B (133MHz) (one slots):
no card



Chris.
begin:vcard
fn:Mitterer, Christoph Anton
n:Mitterer;Christoph Anton
email;internet:[EMAIL PROTECTED]
x-mozilla-html:TRUE
version:2.1
end:vcard

Re: data corruption with nvidia chipsets and IDE/SATA drives // memory hole mapping related bug?!

[Alan]

On Sat, 2 Dec 2006 12:00:36 +0100 (CET)
Karsten Weiss <[EMAIL PROTECTED]> wrote:

> Hello Christoph!
> 
> On Sat, 2 Dec 2006, Christoph Anton Mitterer wrote:
> 
> > I found a severe bug mainly by fortune because it occurs very rarely.
> > My test looks like the following: I have about 30GB of testing data on
> 
> This sounds very familiar! One of the Linux compute clusters I
> administer at work is a 336 node system consisting of the
> following components:

See the thread http://lkml.org/lkml/2006/8/16/305

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: data corruption with nvidia chipsets and IDE/SATA drives // memory hole mapping related bug?!

[Karsten Weiss]

Hello Christoph!

On Sat, 2 Dec 2006, Christoph Anton Mitterer wrote:


I found a severe bug mainly by fortune because it occurs very rarely.
My test looks like the following: I have about 30GB of testing data on


This sounds very familiar! One of the Linux compute clusters I
administer at work is a 336 node system consisting of the
following components:

* 2x Dual-Core AMD Opteron 275
* Tyan S2891 mainboard
* Hitachi HDS728080PLA380 harddisk
* 4 GB RAM (some nodes have 8 GB) - intensively tested with
  memtest86+
* SUSE 9.3 x86_64 (kernel 2.6.11.4-21.14-smp) - But I've also
  e.g. tried the latest openSUSE 10.2 RC1+ kernel 2.6.18.2-33 which
  makes no difference.

We are running LS-Dyna on these machines and discovered a
testcase which shows a similar data corruption. So I can
confirm that the problem is for real an not a hardware defect
of a single machine!

Here's a diff of a corrupted and a good file written during our
testcase:

("-" == corrupted file, "+" == good file)
...
 009f2ff0  67 2a 4c c4 6d 9d 34 44  ad e6 3c 45 05 9a 4d c4  |g*L.m.4D..
From our testing I can also tell that the data corruption does

*not* appear at all when we are booting the nodes with mem=2G.
However, when we are using all the 4GB the data corruption
shows up - but not everytime and thus not on all nodes.
Sometimes a node runs for ours without any problem. That's why
we are testing on 32 nodes in parallel most of the time. I have
the impression that it has something to do with physical memory
layout of the running processes.

Please also notice that this is a silent data corruption. I.e.
there are no error or warning messages in the kernel log or the
mce log at all.

Christoph, I will carefully re-read your entire posting and the
included links on Monday and will also try the memory hole
setting.

If somebody has an explanation for this problem I can offer
some of our compute nodes+time for testing because we really
want to get this fixed as soon as possible.

Best regards,
Karsten

--
Dipl.-Inf. Karsten Weiss - http://www.machineroom.de/knweiss
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: data corruption with nvidia chipsets and IDE/SATA drives // memory hole mapping related bug?!

[Karsten Weiss]

Hello Christoph!

On Sat, 2 Dec 2006, Christoph Anton Mitterer wrote:


I found a severe bug mainly by fortune because it occurs very rarely.
My test looks like the following: I have about 30GB of testing data on


This sounds very familiar! One of the Linux compute clusters I
administer at work is a 336 node system consisting of the
following components:

* 2x Dual-Core AMD Opteron 275
* Tyan S2891 mainboard
* Hitachi HDS728080PLA380 harddisk
* 4 GB RAM (some nodes have 8 GB) - intensively tested with
  memtest86+
* SUSE 9.3 x86_64 (kernel 2.6.11.4-21.14-smp) - But I've also
  e.g. tried the latest openSUSE 10.2 RC1+ kernel 2.6.18.2-33 which
  makes no difference.

We are running LS-Dyna on these machines and discovered a
testcase which shows a similar data corruption. So I can
confirm that the problem is for real an not a hardware defect
of a single machine!

Here's a diff of a corrupted and a good file written during our
testcase:

(- == corrupted file, + == good file)
...
 009f2ff0  67 2a 4c c4 6d 9d 34 44  ad e6 3c 45 05 9a 4d c4  |g*L.m.4D..E..M.|
-009f3000  39 60 e6 44 20 ab 46 44  56 aa 46 44 c2 35 e6 44  |9.D .FDV.FD.5.D|
-009f3010  45 e1 48 44 88 3d 47 44  f3 81 e6 44 93 0b 46 44  |E.HD.=GD...D..FD|
-009f3020  d4 eb 48 44 22 57 e6 44  3d 3d 48 44 ac 89 49 44  |..HDW.D==HD..ID|
-009f3030  00 8c e9 44 39 af 2d 44  e7 1b 8d 44 a8 6e e9 44  |...D9.-D...D.n.D|
-009f3040  16 d4 2e 44 5e 12 8c 44  78 51 e9 44 c0 f5 2f 44  |...D^..DxQ.D../D|
...
-009f3fd0  22 ae 4e 44 81 b5 ee 43  0c 8a df 44 8d e2 6b 44  |.ND...C...D..kD|
-009f3fe0  6c a0 e8 43 b6 8f e9 44  22 ae 4e 44 55 e9 ed 43  |l..C...D.NDU..C|
-009f3ff0  a8 b2 e0 44 78 7c 69 44  56 6f e8 43 5e b2 e0 44  |...Dx|iDVo.C^..D|
+009f3000  1b 32 30 44 50 59 3d 45  a2 79 4e c4 66 6e 2f 44  |.20DPY=E.yN.fn/D|
+009f3010  40 91 3d 45 d1 b6 4e c4  a1 6c 31 44 1b cb 3d 45  |@.=E..N..l1D..=E|
+009f3020  0d f6 4e c4 57 7c 33 44  bf cb 3c 45 88 9a 4d c4  |..N.W|3D..E..M.|
+009f3030  79 e9 29 44 3a 10 3d 45  d3 e1 4d c4 17 28 2c 44  |y.)D:.=E..M..(,D|
+009f3040  f6 50 3d 45 dc 25 4e c4  b6 50 2e 44 b3 4f 3c 45  |.P=E.%N..P.D.OE|
...
+009f3fd0  9c 70 6c 45 04 be 9f c3  fe fc 8f 44 ce 65 6c 45  |.plE...D.elE|
+009f3fe0  fc 56 9c c3 32 f7 90 44  e5 3c 6c 45 cd 79 9c c3  |.V..2..D.lE.y..|
+009f3ff0  f3 55 92 44 c1 10 6c 45  5e 12 a0 c3 60 31 93 44  |.U.D..lE^...1.D|
 009f4000  88 cd 6b 45 c1 6d cd c3  00 a5 8b 44 f2 ac 6b 45  |..kE.m.D..kE|
...

Please notice:

a) the corruption begins at a page boundary
b) the corrupted byte range is a single memory page and
c) almost every fourth byte is set to 0x44 in the corrupted case
   (but the other bytes changed, too)

To me this looks as if a wrong memory page got written into the
file.


From our testing I can also tell that the data corruption does

*not* appear at all when we are booting the nodes with mem=2G.
However, when we are using all the 4GB the data corruption
shows up - but not everytime and thus not on all nodes.
Sometimes a node runs for ours without any problem. That's why
we are testing on 32 nodes in parallel most of the time. I have
the impression that it has something to do with physical memory
layout of the running processes.

Please also notice that this is a silent data corruption. I.e.
there are no error or warning messages in the kernel log or the
mce log at all.

Christoph, I will carefully re-read your entire posting and the
included links on Monday and will also try the memory hole
setting.

If somebody has an explanation for this problem I can offer
some of our compute nodes+time for testing because we really
want to get this fixed as soon as possible.

Best regards,
Karsten

--
Dipl.-Inf. Karsten Weiss - http://www.machineroom.de/knweiss
-
To unsubscribe from this list: send the line unsubscribe linux-kernel in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: data corruption with nvidia chipsets and IDE/SATA drives // memory hole mapping related bug?!

[Alan]

On Sat, 2 Dec 2006 12:00:36 +0100 (CET)
Karsten Weiss [EMAIL PROTECTED] wrote:

 Hello Christoph!
 
 On Sat, 2 Dec 2006, Christoph Anton Mitterer wrote:
 
  I found a severe bug mainly by fortune because it occurs very rarely.
  My test looks like the following: I have about 30GB of testing data on
 
 This sounds very familiar! One of the Linux compute clusters I
 administer at work is a 336 node system consisting of the
 following components:

See the thread http://lkml.org/lkml/2006/8/16/305

-
To unsubscribe from this list: send the line unsubscribe linux-kernel in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: data corruption with nvidia chipsets and IDE/SATA drives // memory hole mapping related bug?!

[Christoph Anton Mitterer]

Alan wrote:
 See the thread http://lkml.org/lkml/2006/8/16/305
   

Hi Alan.

Thanks for your reply. I've read this thread already some weeks ago
but from my limited knowledge I understood, that this was an issue
related to a SCSI adapter or so. Or did I understand this wrong. And as
soon as he removed the card everything was fine.

I don't have any PCI SCSI cards,... (but I have an onboard LSI53C1030
controller).
The only cards I have are:
PCIe bus (two slots):
Asus Nividia 7800GTX based card

PCI bus: no card (one slot):
no card

PCI-X bus A (100MHz) (two slots):
Hauppauge Nova T 500 Dual DVB-T card (which is actually a normal PCI
card,.. but should be compatible with PCI-X)
TerraTec Aureon 7.1 Universe Soundcard (which is actually a normal PCI
card,.. but should be compatible with PCI-X)

PCI-X bus B (133MHz) (one slots):
no card



Chris.
begin:vcard
fn:Mitterer, Christoph Anton
n:Mitterer;Christoph Anton
email;internet:[EMAIL PROTECTED]
x-mozilla-html:TRUE
version:2.1
end:vcard