Re: top io mode

[Andriy Gapon]

on 26/11/2010 02:10 Adam Vande More said the following:
> On Thu, Nov 25, 2010 at 5:21 PM, Kostik Belousov wrote:
> 
>> What is /tmp/delete.me ? A file ? On what kind of filesystem is it
>> located ?
>>
>> Summoning some psychic power, I can predict that delete.me is
>> located on ZFS or tmpfs filesystem. Is this right ? If yes, then
>> the result is expected and nothing is broken there (except ZFS,
>> but I already described it).
>>
> 
> I thought I made it clear earlier in thread this was a ZFS only system, sry
> for any confusion.

So have you tried already the patch for ZFS as have been suggested by couple of
people already?

-- 
Andriy Gapon
___
freebsd-stable@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: top io mode

[Adam Vande More]

On Thu, Nov 25, 2010 at 5:21 PM, Kostik Belousov wrote:

> What is /tmp/delete.me ? A file ? On what kind of filesystem is it
> located ?
>
> Summoning some psychic power, I can predict that delete.me is
> located on ZFS or tmpfs filesystem. Is this right ? If yes, then
> the result is expected and nothing is broken there (except ZFS,
> but I already described it).
>

I thought I made it clear earlier in thread this was a ZFS only system, sry
for any confusion.

FWIW, I do make an effort to keep track of ZFS issues including FreeBSD ZFS
Wiki and various mailing lists and this is the first I've heard of this
particular ZFS shortcoming.  Is there a better place to see what's well
known about ZFS on FreeBSD?

-- 
Adam Vande More
___
freebsd-stable@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: top io mode

[Kostik Belousov]

On Thu, Nov 25, 2010 at 05:18:13PM -0600, Adam Vande More wrote:
> On Thu, Nov 25, 2010 at 4:44 PM, Kostik Belousov wrote:
> 
> > Can you show exact command and describe some details about setup for
> > the case where you still do not observe the counter in top ?
> > (With my patch applied).
> >
> -
> Still broken with patch applied;
> dd if=/dev/zero of=/tmp/delete.me bs=64k
What is /tmp/delete.me ? A file ? On what kind of filesystem is it
located ?

Summoning some psychic power, I can predict that delete.me is
located on ZFS or tmpfs filesystem. Is this right ? If yes, then
the result is expected and nothing is broken there (except ZFS,
but I already described it).

> 
> during this "top -m io" displays for dd:
>  2235 adam   14 24  0  0  0  0   0.00% dd
> 
> -
> Fixed with patch applied;
> 
> dd if=/dev/ada0 of=/dev/null bs=64k
> 
> during this "top -m io" displays for dd:
>  2248 adam 3262  0   3262  0  0   3262 100.00% dd
> 
> 
> 
> -- 
> Adam Vande More


pgpFbyWSWKOjh.pgp
Description: PGP signature

Re: top io mode

[Adam Vande More]

On Thu, Nov 25, 2010 at 4:44 PM, Kostik Belousov wrote:

> Can you show exact command and describe some details about setup for
> the case where you still do not observe the counter in top ?
> (With my patch applied).
>
-
Still broken with patch applied;
dd if=/dev/zero of=/tmp/delete.me bs=64k

during this "top -m io" displays for dd:
 2235 adam   14 24  0  0  0  0   0.00% dd

-
Fixed with patch applied;

dd if=/dev/ada0 of=/dev/null bs=64k

during this "top -m io" displays for dd:
 2248 adam 3262  0   3262  0  0   3262 100.00% dd



-- 
Adam Vande More
___
freebsd-stable@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: top io mode

[Kostik Belousov]

On Thu, Nov 25, 2010 at 04:35:53PM -0600, Adam Vande More wrote:
> On Thu, Nov 25, 2010 at 3:04 PM, Jeremy Chadwick
> wrote:
> 
> > Bad form to follow up to my own Email of course, but some discussion
> > material:
> >
> 
> I'm a frequent offender myself so I won't be pointing any fingers.
> 
> "top -m io" doesn't show any I/O writes, while gstat(8) does, and to
> > numerous devices all which make up some form of ZFS pool.
> >
> 
> Yes, it's a generic ZFS mirror.
> 
> If you do something like "dd if=/dev/urandom of=/pool/file bs=64k", does
> > "top -m io" show write I/O for the "dd" process?
> >
> 
> It does not on my ZFS STABLE system with Kostick's path.  So the patch fixes
> reads, but not writes. cc'ing to notify in case he has more ideas.
Can you show exact command and describe some details about setup for
the case where you still do not observe the counter in top ?
(With my patch applied).

There are two mixed things in the thread:
First, there is (was) a missed accounting for i/o to physical devices,
and the patch I posted should fix it.

Second, it is relatively well-known that ZFS does not properly accounts
i/o. Might be, the patch by Andrew fixes it, I do not know.


pgpnsJ18JpLPt.pgp
Description: PGP signature

Re: top io mode

[Adam Vande More]

On Thu, Nov 25, 2010 at 3:04 PM, Jeremy Chadwick
wrote:

> Bad form to follow up to my own Email of course, but some discussion
> material:
>

I'm a frequent offender myself so I won't be pointing any fingers.

"top -m io" doesn't show any I/O writes, while gstat(8) does, and to
> numerous devices all which make up some form of ZFS pool.
>

Yes, it's a generic ZFS mirror.

If you do something like "dd if=/dev/urandom of=/pool/file bs=64k", does
> "top -m io" show write I/O for the "dd" process?
>

It does not on my ZFS STABLE system with Kostick's path.  So the patch fixes
reads, but not writes. cc'ing to notify in case he has more ideas.

It does on my UFS CURRENT VM w/ out the patch.


> If so, then possibly ZFS "internally" is doing something quietly/behind
> the scenes (such as a disk flush or some sort of internal transaction
> "thing"), thus the kernel would be responsible and not a process.
>

That was kind of my thought, and if that's the case there are two problems:

1.  Can't identify what's responsible for IO.
2.  Any money your company saves due to ZFS features is lost when you get
the electric bill. /sarcasm

So that said, how about "top -S -m io -U root" instead, which includes
> system processes, since we can (probably) assume a non-root process
> isn't responsible?
>

I should have mentioned I had already tried that (-HS) along with -o
read/write to see what was happening, but no luck.

Hopefully I don't get too confusing here because I'm going to contradict
something I said earlier.  The stats I posted from gstat are perhaps not
directly related to the root issue.  Similar stats to what I linked to from
gstats can be seen in approx 10 - 15 intervals and is 0.0 %busy in between
those updates.  However the HD indicator light blinks in 1 second intervals
constantly and this is what I want to track down.

One more datapoint is that in single user mode, the blinking stops.

sa(8) may suffer from the same affliction as a "dd if=/dev/zero of=/tmp/
delete.me bs=64k" had been running along with my constant blinking:

galacticdominator# sa -m
root30 0.00cpu0tio1972k*sec
adam 6 0.27cpu0tio2254k*sec

Maybe it's some weird hardware issue, but there's at least one other FreeBSD
user who is seeing similar behavior.

-- 
Adam Vande More
___
freebsd-stable@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: top io mode

[Jeremy Chadwick]

On Thu, Nov 25, 2010 at 12:57:33PM -0800, Jeremy Chadwick wrote:
> On Thu, Nov 25, 2010 at 02:28:35PM -0600, Adam Vande More wrote:
> > On Thu, Nov 25, 2010 at 2:08 PM, Jeremy Chadwick
> > wrote:
> > 
> > > >  Well the top bug didn't seem to resolve my actual issue and yes it it a
> > > > complete ZFS system.  Problem is that the HD activity indicator light is
> > > > constantly flickering even though should be minimal activity.  top still
> > > > shows no activity around the blinks, and there's no swapping happening
> > > > although gstat does seem to roughly match the blinks.  I can't tell what
> > > > your patch does, would it enable me to see what's touching the disk?
> > >
> > > Please try using gstat(8) instead.
> > >
> > 
> > Like I said gstat shows activity, but I have no way find what process is
> > causing the activity unless there is some hidden gstat option I'm unaware
> > of.
> 
> Sorry, I missed that part of your explanation, my apologies.
> 
> > to be more clear, if I have top -m io in one terminal, I see no movement in
> > the READ/WRITE columns.
> > 
> > while polling gstat output I captured an IO "spike"
> > 
> > http://pastebin.com/f84nuzxt
> > 
> > the percent busy is at 0.00 for every entry with the exception of these
> > spikes.
> 
> I don't have an answer for you, other than "maybe top shows a different
> 'kind' of I/O than what gstat or iostat does".
> 
> Only ideas I have:
> 
> 1) Possibly acct(2) with sa(8) could provide some insight, but I have no
> idea how well it works.
> 
> 2) I see some ps(8) -O parameters labelled "inblk" and "oublk", but I'm
> not sure what "block" means given the man page context.  If these are
> "disk blocks", then possibly a shell script that repetitively calls ps
> with these arguments could narrow it down.
> 
> It's too bad, since sar(1M) on Solaris can be used to achieve exactly
> what you're looking for -- and the closest thing to that on FreeBSD that
> I know of is acct(2) and sa(8).

Bad form to follow up to my own Email of course, but some discussion
material:

"top -m io" doesn't show any I/O writes, while gstat(8) does, and to
numerous devices all which make up some form of ZFS pool.

If you do something like "dd if=/dev/urandom of=/pool/file bs=64k", does
"top -m io" show write I/O for the "dd" process?

If so, then possibly ZFS "internally" is doing something quietly/behind
the scenes (such as a disk flush or some sort of internal transaction
"thing"), thus the kernel would be responsible and not a process.

So that said, how about "top -S -m io -U root" instead, which includes
system processes, since we can (probably) assume a non-root process
isn't responsible?

HTH.

-- 
| Jeremy Chadwick   j...@parodius.com |
| Parodius Networking   http://www.parodius.com/ |
| UNIX Systems Administrator  Mountain View, CA, USA |
| Making life hard for others since 1977.  PGP: 4BD6C0CB |
___
freebsd-stable@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: top io mode

[Jeremy Chadwick]

On Thu, Nov 25, 2010 at 02:28:35PM -0600, Adam Vande More wrote:
> On Thu, Nov 25, 2010 at 2:08 PM, Jeremy Chadwick
> wrote:
> 
> > >  Well the top bug didn't seem to resolve my actual issue and yes it it a
> > > complete ZFS system.  Problem is that the HD activity indicator light is
> > > constantly flickering even though should be minimal activity.  top still
> > > shows no activity around the blinks, and there's no swapping happening
> > > although gstat does seem to roughly match the blinks.  I can't tell what
> > > your patch does, would it enable me to see what's touching the disk?
> >
> > Please try using gstat(8) instead.
> >
> 
> Like I said gstat shows activity, but I have no way find what process is
> causing the activity unless there is some hidden gstat option I'm unaware
> of.

Sorry, I missed that part of your explanation, my apologies.

> to be more clear, if I have top -m io in one terminal, I see no movement in
> the READ/WRITE columns.
> 
> while polling gstat output I captured an IO "spike"
> 
> http://pastebin.com/f84nuzxt
> 
> the percent busy is at 0.00 for every entry with the exception of these
> spikes.

I don't have an answer for you, other than "maybe top shows a different
'kind' of I/O than what gstat or iostat does".

Only ideas I have:

1) Possibly acct(2) with sa(8) could provide some insight, but I have no
idea how well it works.

2) I see some ps(8) -O parameters labelled "inblk" and "oublk", but I'm
not sure what "block" means given the man page context.  If these are
"disk blocks", then possibly a shell script that repetitively calls ps
with these arguments could narrow it down.

It's too bad, since sar(1M) on Solaris can be used to achieve exactly
what you're looking for -- and the closest thing to that on FreeBSD that
I know of is acct(2) and sa(8).

-- 
| Jeremy Chadwick   j...@parodius.com |
| Parodius Networking   http://www.parodius.com/ |
| UNIX Systems Administrator  Mountain View, CA, USA |
| Making life hard for others since 1977.  PGP: 4BD6C0CB |

___
freebsd-stable@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: top io mode

[Adam Vande More]

On Thu, Nov 25, 2010 at 2:08 PM, Jeremy Chadwick
wrote:

> >  Well the top bug didn't seem to resolve my actual issue and yes it it a
> > complete ZFS system.  Problem is that the HD activity indicator light is
> > constantly flickering even though should be minimal activity.  top still
> > shows no activity around the blinks, and there's no swapping happening
> > although gstat does seem to roughly match the blinks.  I can't tell what
> > your patch does, would it enable me to see what's touching the disk?
>
> Please try using gstat(8) instead.
>

Like I said gstat shows activity, but I have no way find what process is
causing the activity unless there is some hidden gstat option I'm unaware
of.

to be more clear, if I have top -m io in one terminal, I see no movement in
the READ/WRITE columns.

while polling gstat output I captured an IO "spike"

http://pastebin.com/f84nuzxt

the percent busy is at 0.00 for every entry with the exception of these
spikes.

-- 
Adam Vande More
___
freebsd-stable@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: top io mode

[Jeremy Chadwick]

On Thu, Nov 25, 2010 at 01:12:17PM -0600, Adam Vande More wrote:
> On Thu, Nov 25, 2010 at 10:03 AM, jhell  wrote:
> 
> > Is this on ZFS ?
> >
> > If that is the case it is a known problem that needs the following patch
> > to fix the issue.
> >
> > http://people.freebsd.org/~avg/zfs-ru.diff
> >
> > I believe it is planned to be MFC'd but am not quite sure so I have CC'd
> > Andriy and he should be able to advise further.
> >
> 
>  Well the top bug didn't seem to resolve my actual issue and yes it it a
> complete ZFS system.  Problem is that the HD activity indicator light is
> constantly flickering even though should be minimal activity.  top still
> shows no activity around the blinks, and there's no swapping happening
> although gstat does seem to roughly match the blinks.  I can't tell what
> your patch does, would it enable me to see what's touching the disk?

Please try using gstat(8) instead.

-- 
| Jeremy Chadwick   j...@parodius.com |
| Parodius Networking   http://www.parodius.com/ |
| UNIX Systems Administrator  Mountain View, CA, USA |
| Making life hard for others since 1977.  PGP: 4BD6C0CB |

___
freebsd-stable@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: top io mode

[Adam Vande More]

On Thu, Nov 25, 2010 at 10:03 AM, jhell  wrote:

> Is this on ZFS ?
>
> If that is the case it is a known problem that needs the following patch
> to fix the issue.
>
> http://people.freebsd.org/~avg/zfs-ru.diff
>
> I believe it is planned to be MFC'd but am not quite sure so I have CC'd
> Andriy and he should be able to advise further.
>

 Well the top bug didn't seem to resolve my actual issue and yes it it a
complete ZFS system.  Problem is that the HD activity indicator light is
constantly flickering even though should be minimal activity.  top still
shows no activity around the blinks, and there's no swapping happening
although gstat does seem to roughly match the blinks.  I can't tell what
your patch does, would it enable me to see what's touching the disk?

-- 
Adam Vande More
___
freebsd-stable@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: top io mode

[Adam Vande More]

On Thu, Nov 25, 2010 at 6:08 AM, Kostik Belousov wrote:

> On Thu, Nov 25, 2010 at 05:28:30AM -0600, Adam Vande More wrote:
> > top io doesn't seem to display stats when dealing direct with a block
> device
> > like so:
> >
> > dd if=/dev/ada0 of=/dev/null
> >
> > However if dd runs on a regular file eg
> >
> > dd if=test.file of=/dev/null
> >
> > then stats are reported in top.
> >
> > Is this the expected behavior?
>
> I do not think so, and the patch at the end of the message worked for me.
>

Yes that works for me as well.

Thanks,

-- 
Adam Vande More
___
freebsd-stable@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: top io mode

[Andriy Gapon]

on 25/11/2010 18:03 jhell said the following:
> On 11/25/2010 06:28, Adam Vande More wrote:
>> top io doesn't seem to display stats when dealing direct with a block device
>> like so:
>>
>> dd if=/dev/ada0 of=/dev/null
>>
>> However if dd runs on a regular file eg
>>
>> dd if=test.file of=/dev/null
>>
>> then stats are reported in top.
>>
>> Is this the expected behavior?
>>
> 
> Is this on ZFS ?
> 
> If that is the case it is a known problem that needs the following patch
> to fix the issue.
> 
> http://people.freebsd.org/~avg/zfs-ru.diff
> 
> I believe it is planned to be MFC'd but am not quite sure so I have CC'd
> Andriy and he should be able to advise further.

I think that you got confused :-)
Read Adam's report again :-)

-- 
Andriy Gapon
___
freebsd-stable@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: top io mode

[jhell]

On 11/25/2010 06:28, Adam Vande More wrote:
> top io doesn't seem to display stats when dealing direct with a block device
> like so:
> 
> dd if=/dev/ada0 of=/dev/null
> 
> However if dd runs on a regular file eg
> 
> dd if=test.file of=/dev/null
> 
> then stats are reported in top.
> 
> Is this the expected behavior?
> 

Is this on ZFS ?

If that is the case it is a known problem that needs the following patch
to fix the issue.

http://people.freebsd.org/~avg/zfs-ru.diff

I believe it is planned to be MFC'd but am not quite sure so I have CC'd
Andriy and he should be able to advise further.


Regards,

-- 

 jhell,v
___
freebsd-stable@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: top io mode

[Kostik Belousov]

On Thu, Nov 25, 2010 at 05:28:30AM -0600, Adam Vande More wrote:
> top io doesn't seem to display stats when dealing direct with a block device
> like so:
> 
> dd if=/dev/ada0 of=/dev/null
> 
> However if dd runs on a regular file eg
> 
> dd if=test.file of=/dev/null
> 
> then stats are reported in top.
> 
> Is this the expected behavior?

I do not think so, and the patch at the end of the message worked for me.

I cannot explain the
if (!TD_IS_IDLETHREAD(curthread))
curthread->td_ru.ru_inblock++;
checks that are done in vfs_bio.c.

diff --git a/sys/kern/kern_physio.c b/sys/kern/kern_physio.c
index d6be6e7..34072f3 100644
--- a/sys/kern/kern_physio.c
+++ b/sys/kern/kern_physio.c
@@ -57,10 +57,13 @@ physio(struct cdev *dev, struct uio *uio, int ioflag)
for (i = 0; i < uio->uio_iovcnt; i++) {
while (uio->uio_iov[i].iov_len) {
bp->b_flags = 0;
-   if (uio->uio_rw == UIO_READ)
+   if (uio->uio_rw == UIO_READ) {
bp->b_iocmd = BIO_READ;
-   else 
+   curthread->td_ru.ru_inblock++;
+   } else {
bp->b_iocmd = BIO_WRITE;
+   curthread->td_ru.ru_oublock++;
+   }
bp->b_iodone = bdone;
bp->b_data = uio->uio_iov[i].iov_base;
bp->b_bcount = uio->uio_iov[i].iov_len;



pgpfkjhFTcdCl.pgp
Description: PGP signature