Re: Should cam_imask be part of bio_imask ?
Matthew Dillon wrote: >Oh my, did I really say "Thor" ? Sorry about that Tor! Actually, no, I managed to insert that typo into Matt's quote. Sorry to both Tor and Matt. Peter To Unsubscribe: send mail to majord...@freebsd.org with "unsubscribe freebsd-hackers" in the body of the message
Re: Should cam_imask be part of bio_imask ?
:[General discussion of VM buffer corruption deleted] : :Matthew Dillon wrote: :> Thor suggested adding the CACHETHEN bit back in the adaptec controller. : :To save anyone else the effort, this change only affects Adaptecs :that identify as "aic7890/91" or "aic7896/97". : :Peter Oh my, did I really say "Thor" ? Sorry about that Tor! -Matt Matthew Dillon To Unsubscribe: send mail to majord...@freebsd.org with "unsubscribe freebsd-hackers" in the body of the message
Re: Should cam_imask be part of bio_imask ?
Matthew Dillon <[EMAIL PROTECTED]> wrote: >Oh my, did I really say "Thor" ? Sorry about that Tor! Actually, no, I managed to insert that typo into Matt's quote. Sorry to both Tor and Matt. Peter To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: Should cam_imask be part of bio_imask ?
:[General discussion of VM buffer corruption deleted] : :Matthew Dillon <[EMAIL PROTECTED]> wrote: :> Thor suggested adding the CACHETHEN bit back in the adaptec controller. : :To save anyone else the effort, this change only affects Adaptecs :that identify as "aic7890/91" or "aic7896/97". : :Peter Oh my, did I really say "Thor" ? Sorry about that Tor! -Matt Matthew Dillon <[EMAIL PROTECTED]> To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: Should cam_imask be part of bio_imask ?
[General discussion of VM buffer corruption deleted] Matthew Dillon wrote: > Thor suggested adding the CACHETHEN bit back in the adaptec controller. To save anyone else the effort, this change only affects Adaptecs that identify as "aic7890/91" or "aic7896/97". Peter To Unsubscribe: send mail to majord...@freebsd.org with "unsubscribe freebsd-hackers" in the body of the message
Re: Should cam_imask be part of bio_imask ?
[General discussion of VM buffer corruption deleted] Matthew Dillon <[EMAIL PROTECTED]> wrote: > Thor suggested adding the CACHETHEN bit back in the adaptec controller. To save anyone else the effort, this change only affects Adaptecs that identify as "aic7890/91" or "aic7896/97". Peter To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: Should cam_imask be part of bio_imask ?
Oh, Tor didn't Cc hackers. Tor suggested adding the CACHETHEN bit back in the adaptec controller, undoing part of Justin's commit on the 15th (see the commitlogs for sys and search for 'CACHETHEN'). This fixed the problem! Ah, sunshine here I come! -Matt To Unsubscribe: send mail to majord...@freebsd.org with "unsubscribe freebsd-hackers" in the body of the message
Re: Should cam_imask be part of bio_imask ?
Oh, Tor didn't Cc hackers. Tor suggested adding the CACHETHEN bit back in the adaptec controller, undoing part of Justin's commit on the 15th (see the commitlogs for sys and search for 'CACHETHEN'). This fixed the problem! Ah, sunshine here I come! -Matt To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: Should cam_imask be part of bio_imask ?
>
> I strongly doubt that this is a CAM isr problem- the error pattern isn't
> entirely clear from what you said, but it looks more like a FIFO or CACHE
> LINE sized type of problem- it looks to be < 16 bytes, but not a short
> count. Because this isn't one of the wacky systems I spent most of my
> career on at Sun where the first and usual suspect was a system memory
> cache line because IO wasn't cache coherent on Suns between the Sun
> 3/{50,60,75,150} and the advent SuperSparc Viking Chipset, I'd guess a
> FIFO somewhere in the I/O movement path.
>
> Justin- any changes lately where flushing a FIFO in the Adaptec at the end
> of tranfer might have been spoodged?
If you happen to be using an Adaptec SCSI controller on the server,
check your PCI bus latency setting. We've seen several reports now of
this sort of stuff getting corrupted as a side-effect of the Adaptec
parts being arbitrated off the bus too quickly. If you're doing lots
of network traffic, the chances are a lot better that you'll hit this.
32 is a value that seems to cause problems; try 80 or so.
--
\\ The mind's the standard \\ Mike Smith
\\ of the man. \\ msm...@freebsd.org
\\-- Joseph Merrick \\ msm...@cdrom.com
To Unsubscribe: send mail to majord...@freebsd.org
with "unsubscribe freebsd-hackers" in the body of the message
Re: Should cam_imask be part of bio_imask ?
> :I strongly doubt that this is a CAM isr problem- the error pattern isn't
> :entirely clear from what you said, but it looks more like a FIFO or CACHE
> :LINE sized type of problem- it looks to be < 16 bytes, but not a short
> :count. Because this isn't one of the wacky systems I spent most of my
> :career on at Sun where the first and usual suspect was a system memory
> :cache line because IO wasn't cache coherent on Suns between the Sun
> :3/{50,60,75,150} and the advent SuperSparc Viking Chipset, I'd guess a
> :FIFO somewhere in the I/O movement path.
> :
> :Justin- any changes lately where flushing a FIFO in the Adaptec at the end
> :of tranfer might have been spoodged?
> :
> :-matt
>
> The problem is definitely aligned in some way. Here's a diff of
> a hexdump of one error. Sometimes I lose a whole page, sometimes two
> pages, sometimes 16 bytes, but the error is always page aligned.
>
> 1536c1536
> < 0005ff0 2033 3434 3434 7c20 207c 3030 3030
> ---
> > 0005ff0 7365 3d20 3120 093b 2309 6720 6f6c 6162
>
> A cache-line problem would fit the symptoms. I know it isn't the
> hardware... this 1xCPU PPro/200 system has been with me for several
> years and this test didn't fail like this a month ago. When I updated
> the machine last (unfortunately w/ about a month's worth of changes),
> my buildworlds started failing with odd errors.
>
> I then switched away from the failing buildworlds (which take an hour)
> and started doing cp -r's and then diff -r's (takes only 20 min), and as
> you can see I'm still seeing the problem.
>
> Maybe this is DMA related. Perhaps the cache is not getting cleared?
> Maybe an MMU optimization someone threw in recently?
That's possible too- I'll admit I'm a bit hazy on i386 specifics- it's
always been a "just works wrt I/O" so for all I know there's a required
i/o flush command when you switch mappings. Gawd I hate these kind of
problems.
To Unsubscribe: send mail to majord...@freebsd.org
with "unsubscribe freebsd-hackers" in the body of the message
Re: Should cam_imask be part of bio_imask ?
:I strongly doubt that this is a CAM isr problem- the error pattern isn't
:entirely clear from what you said, but it looks more like a FIFO or CACHE
:LINE sized type of problem- it looks to be < 16 bytes, but not a short
:count. Because this isn't one of the wacky systems I spent most of my
:career on at Sun where the first and usual suspect was a system memory
:cache line because IO wasn't cache coherent on Suns between the Sun
:3/{50,60,75,150} and the advent SuperSparc Viking Chipset, I'd guess a
:FIFO somewhere in the I/O movement path.
:
:Justin- any changes lately where flushing a FIFO in the Adaptec at the end
:of tranfer might have been spoodged?
:
:-matt
The problem is definitely aligned in some way. Here's a diff of
a hexdump of one error. Sometimes I lose a whole page, sometimes two
pages, sometimes 16 bytes, but the error is always page aligned.
1536c1536
< 0005ff0 2033 3434 3434 7c20 207c 3030 3030
---
> 0005ff0 7365 3d20 3120 093b 2309 6720 6f6c 6162
A cache-line problem would fit the symptoms. I know it isn't the
hardware... this 1xCPU PPro/200 system has been with me for several
years and this test didn't fail like this a month ago. When I updated
the machine last (unfortunately w/ about a month's worth of changes),
my buildworlds started failing with odd errors.
I then switched away from the failing buildworlds (which take an hour)
and started doing cp -r's and then diff -r's (takes only 20 min), and as
you can see I'm still seeing the problem.
Maybe this is DMA related. Perhaps the cache is not getting cleared?
Maybe an MMU optimization someone threw in recently?
-Matt
Matthew Dillon
To Unsubscribe: send mail to majord...@freebsd.org
with "unsubscribe freebsd-hackers" in the body of the message
Re: Should cam_imask be part of bio_imask ?
I strongly doubt that this is a CAM isr problem- the error pattern isn't
entirely clear from what you said, but it looks more like a FIFO or CACHE
LINE sized type of problem- it looks to be < 16 bytes, but not a short
count. Because this isn't one of the wacky systems I spent most of my
career on at Sun where the first and usual suspect was a system memory
cache line because IO wasn't cache coherent on Suns between the Sun
3/{50,60,75,150} and the advent SuperSparc Viking Chipset, I'd guess a
FIFO somewhere in the I/O movement path.
Justin- any changes lately where flushing a FIFO in the Adaptec at the end
of tranfer might have been spoodged?
-matt
To Unsubscribe: send mail to majord...@freebsd.org
with "unsubscribe freebsd-hackers" in the body of the message
Re: Should cam_imask be part of bio_imask ?
:> Isn't all the work in CAM done at splsoftcam? : :Not all. There's completions done at splcam. I've had some worries and :concerns about this, but (wince) I really have to admit that the ins and :outs of the cam code often escape me, and they're documented only in the :DNA of certain human subspecies that reside in Colorado. Hmm, ok. Shoot. This is what I'm getting. If I have an NFS server and client running CURRENT, and (from the client) I copy the CVS repository from the server to the client, and then diff them, I get little discrepancies. An example is included below. *** /FreeBSD/FreeBSD-CVS/src/sys/pci/ncr.c,vFri Aug 27 17:51:03 1999 --- /home/ncvs/src/sys/pci/ncr.c,v Fri Aug 27 17:51:03 1999 *** *** 12211,12217 d55 1 d1345 1 a1345 1 ! "\nhis product 1.112 1997/11/07 09:20:56 phk Exp $\n"; @ --- 12211,12217 d55 1 d1345 1 a1345 1 ! "\n$Id: ncr.c,v 1.112 1997/11/07 09:20:56 phk Exp $\n"; @ I'm still experimenting trying to focus in on where the problem is. It is a very weird problem. Sometimes the errors are small, sometimes who pages are wrong. The scary part is that they are wrong in the server's cache! If I catch the error quickly enough and cat the file on the server, the error shows up on the server! If I then, on the server, eat up memory to flush the caches and then cat the file again, the error is gone again. I'm going to try changing it over from an NFSv3 to an NFSv2 mount to see if that does anything. -Matt To Unsubscribe: send mail to majord...@freebsd.org with "unsubscribe freebsd-hackers" in the body of the message
Re: Should cam_imask be part of bio_imask ?
> Isn't all the work in CAM done at splsoftcam? Not all. There's completions done at splcam. I've had some worries and concerns about this, but (wince) I really have to admit that the ins and outs of the cam code often escape me, and they're documented only in the DNA of certain human subspecies that reside in Colorado. > (or am I thinking of old code) > It starts off an ISR similar to the netisr doesn't it? > > On Sat, 28 Aug 1999, Matthew Dillon wrote: > > > I'm trying to track down some VFS/BIO corruption on an NFS server > > being tested under heavy loads and noticed that my SCSI interrupts > > do not appear to be blocked by splbio(). > > > > (The adaptec's are on irq 5 and irq 12) > > > > (kgdb) print bio_imask > > $1 = 0x40080040 > > (kgdb) print cam_imask > > $2 = 0x400c1020 > > (kgdb) > > > > This doesn't sound right to me. > > > > -Matt > > Matthew Dillon > > > > > > > > To Unsubscribe: send mail to majord...@freebsd.org > > with "unsubscribe freebsd-hackers" in the body of the message > > > > > > To Unsubscribe: send mail to majord...@freebsd.org > with "unsubscribe freebsd-hackers" in the body of the message > To Unsubscribe: send mail to majord...@freebsd.org with "unsubscribe freebsd-hackers" in the body of the message
Re: Should cam_imask be part of bio_imask ?
>
> I strongly doubt that this is a CAM isr problem- the error pattern isn't
> entirely clear from what you said, but it looks more like a FIFO or CACHE
> LINE sized type of problem- it looks to be < 16 bytes, but not a short
> count. Because this isn't one of the wacky systems I spent most of my
> career on at Sun where the first and usual suspect was a system memory
> cache line because IO wasn't cache coherent on Suns between the Sun
> 3/{50,60,75,150} and the advent SuperSparc Viking Chipset, I'd guess a
> FIFO somewhere in the I/O movement path.
>
> Justin- any changes lately where flushing a FIFO in the Adaptec at the end
> of tranfer might have been spoodged?
If you happen to be using an Adaptec SCSI controller on the server,
check your PCI bus latency setting. We've seen several reports now of
this sort of stuff getting corrupted as a side-effect of the Adaptec
parts being arbitrated off the bus too quickly. If you're doing lots
of network traffic, the chances are a lot better that you'll hit this.
32 is a value that seems to cause problems; try 80 or so.
--
\\ The mind's the standard \\ Mike Smith
\\ of the man. \\ [EMAIL PROTECTED]
\\-- Joseph Merrick \\ [EMAIL PROTECTED]
To Unsubscribe: send mail to [EMAIL PROTECTED]
with "unsubscribe freebsd-hackers" in the body of the message
Re: Should cam_imask be part of bio_imask ?
Isn't all the work in CAM done at splsoftcam? (or am I thinking of old code) It starts off an ISR similar to the netisr doesn't it? On Sat, 28 Aug 1999, Matthew Dillon wrote: > I'm trying to track down some VFS/BIO corruption on an NFS server > being tested under heavy loads and noticed that my SCSI interrupts > do not appear to be blocked by splbio(). > > (The adaptec's are on irq 5 and irq 12) > > (kgdb) print bio_imask > $1 = 0x40080040 > (kgdb) print cam_imask > $2 = 0x400c1020 > (kgdb) > > This doesn't sound right to me. > > -Matt > Matthew Dillon > > > > To Unsubscribe: send mail to majord...@freebsd.org > with "unsubscribe freebsd-hackers" in the body of the message > To Unsubscribe: send mail to majord...@freebsd.org with "unsubscribe freebsd-hackers" in the body of the message
Re: Should cam_imask be part of bio_imask ?
> :I strongly doubt that this is a CAM isr problem- the error pattern isn't
> :entirely clear from what you said, but it looks more like a FIFO or CACHE
> :LINE sized type of problem- it looks to be < 16 bytes, but not a short
> :count. Because this isn't one of the wacky systems I spent most of my
> :career on at Sun where the first and usual suspect was a system memory
> :cache line because IO wasn't cache coherent on Suns between the Sun
> :3/{50,60,75,150} and the advent SuperSparc Viking Chipset, I'd guess a
> :FIFO somewhere in the I/O movement path.
> :
> :Justin- any changes lately where flushing a FIFO in the Adaptec at the end
> :of tranfer might have been spoodged?
> :
> :-matt
>
> The problem is definitely aligned in some way. Here's a diff of
> a hexdump of one error. Sometimes I lose a whole page, sometimes two
> pages, sometimes 16 bytes, but the error is always page aligned.
>
> 1536c1536
> < 0005ff0 2033 3434 3434 7c20 207c 3030 3030
> ---
> > 0005ff0 7365 3d20 3120 093b 2309 6720 6f6c 6162
>
> A cache-line problem would fit the symptoms. I know it isn't the
> hardware... this 1xCPU PPro/200 system has been with me for several
> years and this test didn't fail like this a month ago. When I updated
> the machine last (unfortunately w/ about a month's worth of changes),
> my buildworlds started failing with odd errors.
>
> I then switched away from the failing buildworlds (which take an hour)
> and started doing cp -r's and then diff -r's (takes only 20 min), and as
> you can see I'm still seeing the problem.
>
> Maybe this is DMA related. Perhaps the cache is not getting cleared?
> Maybe an MMU optimization someone threw in recently?
That's possible too- I'll admit I'm a bit hazy on i386 specifics- it's
always been a "just works wrt I/O" so for all I know there's a required
i/o flush command when you switch mappings. Gawd I hate these kind of
problems.
To Unsubscribe: send mail to [EMAIL PROTECTED]
with "unsubscribe freebsd-hackers" in the body of the message
Re: Should cam_imask be part of bio_imask ?
:I strongly doubt that this is a CAM isr problem- the error pattern isn't
:entirely clear from what you said, but it looks more like a FIFO or CACHE
:LINE sized type of problem- it looks to be < 16 bytes, but not a short
:count. Because this isn't one of the wacky systems I spent most of my
:career on at Sun where the first and usual suspect was a system memory
:cache line because IO wasn't cache coherent on Suns between the Sun
:3/{50,60,75,150} and the advent SuperSparc Viking Chipset, I'd guess a
:FIFO somewhere in the I/O movement path.
:
:Justin- any changes lately where flushing a FIFO in the Adaptec at the end
:of tranfer might have been spoodged?
:
:-matt
The problem is definitely aligned in some way. Here's a diff of
a hexdump of one error. Sometimes I lose a whole page, sometimes two
pages, sometimes 16 bytes, but the error is always page aligned.
1536c1536
< 0005ff0 2033 3434 3434 7c20 207c 3030 3030
---
> 0005ff0 7365 3d20 3120 093b 2309 6720 6f6c 6162
A cache-line problem would fit the symptoms. I know it isn't the
hardware... this 1xCPU PPro/200 system has been with me for several
years and this test didn't fail like this a month ago. When I updated
the machine last (unfortunately w/ about a month's worth of changes),
my buildworlds started failing with odd errors.
I then switched away from the failing buildworlds (which take an hour)
and started doing cp -r's and then diff -r's (takes only 20 min), and as
you can see I'm still seeing the problem.
Maybe this is DMA related. Perhaps the cache is not getting cleared?
Maybe an MMU optimization someone threw in recently?
-Matt
Matthew Dillon
<[EMAIL PROTECTED]>
To Unsubscribe: send mail to [EMAIL PROTECTED]
with "unsubscribe freebsd-hackers" in the body of the message
Re: Should cam_imask be part of bio_imask ?
I strongly doubt that this is a CAM isr problem- the error pattern isn't
entirely clear from what you said, but it looks more like a FIFO or CACHE
LINE sized type of problem- it looks to be < 16 bytes, but not a short
count. Because this isn't one of the wacky systems I spent most of my
career on at Sun where the first and usual suspect was a system memory
cache line because IO wasn't cache coherent on Suns between the Sun
3/{50,60,75,150} and the advent SuperSparc Viking Chipset, I'd guess a
FIFO somewhere in the I/O movement path.
Justin- any changes lately where flushing a FIFO in the Adaptec at the end
of tranfer might have been spoodged?
-matt
To Unsubscribe: send mail to [EMAIL PROTECTED]
with "unsubscribe freebsd-hackers" in the body of the message
Re: Should cam_imask be part of bio_imask ?
:> Isn't all the work in CAM done at splsoftcam? : :Not all. There's completions done at splcam. I've had some worries and :concerns about this, but (wince) I really have to admit that the ins and :outs of the cam code often escape me, and they're documented only in the :DNA of certain human subspecies that reside in Colorado. Hmm, ok. Shoot. This is what I'm getting. If I have an NFS server and client running CURRENT, and (from the client) I copy the CVS repository from the server to the client, and then diff them, I get little discrepancies. An example is included below. *** /FreeBSD/FreeBSD-CVS/src/sys/pci/ncr.c,vFri Aug 27 17:51:03 1999 --- /home/ncvs/src/sys/pci/ncr.c,v Fri Aug 27 17:51:03 1999 *** *** 12211,12217 d55 1 d1345 1 a1345 1 ! "\nhis product 1.112 1997/11/07 09:20:56 phk Exp $\n"; @ --- 12211,12217 d55 1 d1345 1 a1345 1 ! "\n$Id: ncr.c,v 1.112 1997/11/07 09:20:56 phk Exp $\n"; @ I'm still experimenting trying to focus in on where the problem is. It is a very weird problem. Sometimes the errors are small, sometimes who pages are wrong. The scary part is that they are wrong in the server's cache! If I catch the error quickly enough and cat the file on the server, the error shows up on the server! If I then, on the server, eat up memory to flush the caches and then cat the file again, the error is gone again. I'm going to try changing it over from an NFSv3 to an NFSv2 mount to see if that does anything. -Matt To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: Should cam_imask be part of bio_imask ?
> Isn't all the work in CAM done at splsoftcam? Not all. There's completions done at splcam. I've had some worries and concerns about this, but (wince) I really have to admit that the ins and outs of the cam code often escape me, and they're documented only in the DNA of certain human subspecies that reside in Colorado. > (or am I thinking of old code) > It starts off an ISR similar to the netisr doesn't it? > > On Sat, 28 Aug 1999, Matthew Dillon wrote: > > > I'm trying to track down some VFS/BIO corruption on an NFS server > > being tested under heavy loads and noticed that my SCSI interrupts > > do not appear to be blocked by splbio(). > > > > (The adaptec's are on irq 5 and irq 12) > > > > (kgdb) print bio_imask > > $1 = 0x40080040 > > (kgdb) print cam_imask > > $2 = 0x400c1020 > > (kgdb) > > > > This doesn't sound right to me. > > > > -Matt > > Matthew Dillon > > <[EMAIL PROTECTED]> > > > > > > To Unsubscribe: send mail to [EMAIL PROTECTED] > > with "unsubscribe freebsd-hackers" in the body of the message > > > > > > To Unsubscribe: send mail to [EMAIL PROTECTED] > with "unsubscribe freebsd-hackers" in the body of the message > To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: Should cam_imask be part of bio_imask ?
Isn't all the work in CAM done at splsoftcam? (or am I thinking of old code) It starts off an ISR similar to the netisr doesn't it? On Sat, 28 Aug 1999, Matthew Dillon wrote: > I'm trying to track down some VFS/BIO corruption on an NFS server > being tested under heavy loads and noticed that my SCSI interrupts > do not appear to be blocked by splbio(). > > (The adaptec's are on irq 5 and irq 12) > > (kgdb) print bio_imask > $1 = 0x40080040 > (kgdb) print cam_imask > $2 = 0x400c1020 > (kgdb) > > This doesn't sound right to me. > > -Matt > Matthew Dillon > <[EMAIL PROTECTED]> > > > To Unsubscribe: send mail to [EMAIL PROTECTED] > with "unsubscribe freebsd-hackers" in the body of the message > To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
