Re: How to debug high system load?

[Suvayu Ali]

On Tue, Jul 31, 2012 at 08:21:57AM -0700, Jack Craig wrote:
> On Mon, Jul 30, 2012 at 4:28 PM, Suvayu Ali 
> wrote:
> > 
> > I did mention it. I also tried many other things all of which I reported
> > back to the list. You can find the gory details in the archive.
> > 
> >  > 421944>
> 
> Interesting! So the audio chip is the culprit or just the most recent
> candidate?
>  

Are you refering to the Intel HD audio chipsets issue?  I don't think
so, because I haven't had any problems with audio in quite a while.
Moreover I think my hardware was not one of the affected ones:

  description: Audio device
  product: 5 Series/3400 Series Chipset High Definition Audio
  vendor: Intel Corporation

As I mentioned in my earlier posts; it seemed like a file system issue,
but then I'm no expert.  Would interesting to know what was causing the
problem though.  Maybe I'll reboot to the older kernel when I have some
time and see if I can replicate it.

-- 
Suvayu

Open source is the future. It sets us free.
-- 
users mailing list
users@lists.fedoraproject.org
To unsubscribe or change subscription options:
https://admin.fedoraproject.org/mailman/listinfo/users
Guidelines: http://fedoraproject.org/wiki/Mailing_list_guidelines
Have a question? Ask away: http://ask.fedoraproject.org

Re: How to debug high system load?

[Jack Craig]

On Mon, Jul 30, 2012 at 4:28 PM, Suvayu Ali wrote:

>
>
> I did mention it. I also tried many other things all of which I reported
> back to the list. You can find the gory details in the archive.
>
> <
> http://lists.fedoraproject.org/pipermail/users/2012-July/thread.html#421944
> >
>

Interesting! So the audio chip is the culprit or just the most recent
candidate?


>
> --
> Suvayu
>
> Open source is the future. It sets us free.
> --
> users mailing list
> users@lists.fedoraproject.org
> To unsubscribe or change subscription options:
> https://admin.fedoraproject.org/mailman/listinfo/users
> Guidelines: http://fedoraproject.org/wiki/Mailing_list_guidelines
> Have a question? Ask away: http://ask.fedoraproject.org
>
-- 
users mailing list
users@lists.fedoraproject.org
To unsubscribe or change subscription options:
https://admin.fedoraproject.org/mailman/listinfo/users
Guidelines: http://fedoraproject.org/wiki/Mailing_list_guidelines
Have a question? Ask away: http://ask.fedoraproject.org

Re: How to debug high system load?

[Suvayu Ali]

On Mon, Jul 30, 2012 at 03:04:27PM -0700, Jack Craig wrote:
> 
> 
> On Mon, Jul 30, 2012 at 1:59 PM, Suvayu Ali 
> wrote:
> 
> Hello everyone,
> 
> On Thu, Jul 12, 2012 at 01:00:34PM +0200, Suvayu Ali wrote:

[...]

> > 3. top only says currently Xorg is the most active process with 1% CPU
> >activity. From the activity summary of atop since boot, I see it's
> >Xorg with 3% CPU usage. And while idling all my 4 logical cores are
> >throttled down and are at about 1%.
> >

[...]

> 
>  Just curious, I dont see your comments on what top tells you?
> you did try running top ?
> 

I did mention it. I also tried many other things all of which I reported
back to the list. You can find the gory details in the archive.



-- 
Suvayu

Open source is the future. It sets us free.
-- 
users mailing list
users@lists.fedoraproject.org
To unsubscribe or change subscription options:
https://admin.fedoraproject.org/mailman/listinfo/users
Guidelines: http://fedoraproject.org/wiki/Mailing_list_guidelines
Have a question? Ask away: http://ask.fedoraproject.org

Re: How to debug high system load?

[Jack Craig]

On Mon, Jul 30, 2012 at 1:59 PM, Suvayu Ali wrote:

> Hello everyone,
>
> On Thu, Jul 12, 2012 at 01:00:34PM +0200, Suvayu Ali wrote:
> > Hi,
> >
> > I have a ThinkPad (Core i7, 2.67GHz) with F17 and I'm seeing high system
> > loads persistently. Right from boot up I see loads of 1, and often it
> > goes up to 2 or more.
> >
> > Things I have looked at:
> > 1. I'm using Xfce, so the desktop environment is also light.
> > 2. I looked at the output of `systemctl list-units' and don't see any
> >unnecessary services enabled.
> > 3. top only says currently Xorg is the most active process with 1% CPU
> >activity. From the activity summary of atop since boot, I see it's
> >Xorg with 3% CPU usage. And while idling all my 4 logical cores are
> >throttled down and are at about 1%.
> >
> > So my question is since CPU usage is already so low, what could be
> > driving up the system load? This is bugging me since I also see a
> > general sluggishness compared to a much faster system early this year.
> >
>
> To answer myself; after I upgraded to kernel 3.4.6-2.fc17.x86_64
> yesterday, I haven't seen any issues with load. My average load when
> browsing, reading email, or other regular desktop activities is ~ 0.22.
>
> Cheers,
>

 Just curious, I dont see your comments on what *top* tells you?
you did try running top ?

>
> --
> Suvayu
>
> Open source is the future. It sets us free.
> --
> users mailing list
> users@lists.fedoraproject.org
> To unsubscribe or change subscription options:
> https://admin.fedoraproject.org/mailman/listinfo/users
> Guidelines: http://fedoraproject.org/wiki/Mailing_list_guidelines
> Have a question? Ask away: http://ask.fedoraproject.org
>
-- 
users mailing list
users@lists.fedoraproject.org
To unsubscribe or change subscription options:
https://admin.fedoraproject.org/mailman/listinfo/users
Guidelines: http://fedoraproject.org/wiki/Mailing_list_guidelines
Have a question? Ask away: http://ask.fedoraproject.org

Re: How to debug high system load?

[Suvayu Ali]

Hello everyone,

On Thu, Jul 12, 2012 at 01:00:34PM +0200, Suvayu Ali wrote:
> Hi,
> 
> I have a ThinkPad (Core i7, 2.67GHz) with F17 and I'm seeing high system
> loads persistently. Right from boot up I see loads of 1, and often it
> goes up to 2 or more.
> 
> Things I have looked at:
> 1. I'm using Xfce, so the desktop environment is also light.
> 2. I looked at the output of `systemctl list-units' and don't see any
>unnecessary services enabled.
> 3. top only says currently Xorg is the most active process with 1% CPU
>activity. From the activity summary of atop since boot, I see it's
>Xorg with 3% CPU usage. And while idling all my 4 logical cores are
>throttled down and are at about 1%.
> 
> So my question is since CPU usage is already so low, what could be
> driving up the system load? This is bugging me since I also see a
> general sluggishness compared to a much faster system early this year.
> 

To answer myself; after I upgraded to kernel 3.4.6-2.fc17.x86_64
yesterday, I haven't seen any issues with load. My average load when
browsing, reading email, or other regular desktop activities is ~ 0.22.

Cheers,

-- 
Suvayu

Open source is the future. It sets us free.
-- 
users mailing list
users@lists.fedoraproject.org
To unsubscribe or change subscription options:
https://admin.fedoraproject.org/mailman/listinfo/users
Guidelines: http://fedoraproject.org/wiki/Mailing_list_guidelines
Have a question? Ask away: http://ask.fedoraproject.org

Re: How to debug high system load?

[Andrew Haley]

On 07/16/2012 12:00 AM, Jared K. Smith wrote:
> A co-worker has a similar problem with a new Lenovo ThinkPad -- it
> turns out something about the audio chip was taking a lot more cpu
> cycles than it should be -- I'll try to dig out some more details from
> him.

Fascinating.  If so, how come those cycles weren't being accounted for?
H

Andrew.
-- 
users mailing list
users@lists.fedoraproject.org
To unsubscribe or change subscription options:
https://admin.fedoraproject.org/mailman/listinfo/users
Guidelines: http://fedoraproject.org/wiki/Mailing_list_guidelines
Have a question? Ask away: http://ask.fedoraproject.org

Re: How to debug high system load?

[Suvayu Ali]

On Sun, Jul 15, 2012 at 07:00:23PM -0400, Jared K. Smith wrote:
> On Sun, Jul 15, 2012 at 4:27 PM, Suvayu Ali  
> wrote:
> > Thanks for the nice explanation. But sadly this doesn't help either.
> > Earlier the load average on login would be ~ 1-1.5. With these changes,
> > it starts at ~ 2 and after about 5 minutes settles to ~ 1-1.2. For
> > example now my laptop has been running for 50 minutes and load is
> > hovering over 1. :(
> 
> A co-worker has a similar problem with a new Lenovo ThinkPad -- it
> turns out something about the audio chip was taking a lot more cpu
> cycles than it should be -- I'll try to dig out some more details from
> him.
> 

Thanks a lot Jared. :)

-- 
Suvayu

Open source is the future. It sets us free.
-- 
users mailing list
users@lists.fedoraproject.org
To unsubscribe or change subscription options:
https://admin.fedoraproject.org/mailman/listinfo/users
Guidelines: http://fedoraproject.org/wiki/Mailing_list_guidelines
Have a question? Ask away: http://ask.fedoraproject.org

Re: How to debug high system load?

[Jared K. Smith]

On Sun, Jul 15, 2012 at 4:27 PM, Suvayu Ali  wrote:
> Thanks for the nice explanation. But sadly this doesn't help either.
> Earlier the load average on login would be ~ 1-1.5. With these changes,
> it starts at ~ 2 and after about 5 minutes settles to ~ 1-1.2. For
> example now my laptop has been running for 50 minutes and load is
> hovering over 1. :(

A co-worker has a similar problem with a new Lenovo ThinkPad -- it
turns out something about the audio chip was taking a lot more cpu
cycles than it should be -- I'll try to dig out some more details from
him.

--
Jared Smith
-- 
users mailing list
users@lists.fedoraproject.org
To unsubscribe or change subscription options:
https://admin.fedoraproject.org/mailman/listinfo/users
Guidelines: http://fedoraproject.org/wiki/Mailing_list_guidelines
Have a question? Ask away: http://ask.fedoraproject.org

Re: How to debug high system load?

[Joe Zeff]

On 07/15/2012 01:27 PM, Suvayu Ali wrote:

Thanks for the nice explanation. But sadly this doesn't help either.
Earlier the load average on login would be ~ 1-1.5. With these changes,
it starts at ~ 2 and after about 5 minutes settles to ~ 1-1.2. For
example now my laptop has been running for 50 minutes and load is
hovering over 1. :(


OK, you've eliminated disk IO as the cause of your high load.
--
users mailing list
users@lists.fedoraproject.org
To unsubscribe or change subscription options:
https://admin.fedoraproject.org/mailman/listinfo/users
Guidelines: http://fedoraproject.org/wiki/Mailing_list_guidelines
Have a question? Ask away: http://ask.fedoraproject.org

Re: How to debug high system load?

[Suvayu Ali]

Hi Paweł and Heinz,

On Sun, Jul 15, 2012 at 03:55:45PM +0200, Paweł Brodacki wrote:
> 2012/7/15 suvayu ali :
> > Hi Heinz,
> >
> > Sorry for the late response.
> >
> > On Thu, Jul 12, 2012 at 7:31 PM, Heinz Diehl  wrote:
> >> On 12.07.2012, Suvayu Ali wrote:
> (...)
> >> If you're using cfq as your scheduler, try this in rc.local:
> >>
> >> echo "32" > /sys/block/sda/queue/iosched/quantum
> >> echo "0" > /sys/block/sda/queue/iosched/slice_idle
> >> echo "1" > /sys/block/sda/queue/iosched/low_latency
> >> echo "51200" > /sys/block/sda/queue/nr_requests
> >>
> >> Together with this in /etc/sysctl.conf:
> >>
> >> vm.dirty_ratio = 10
> >> vm.dirty_background_ratio = 5
> >>
> >
> > I tend not to try things that I don't understand. Could please outline
> > briefly what the above suggestions do? I would like to understand before
> > I try them out.
> >
> > Thanks,
> >
> 
> When vm.dirty_ratio percent of total system memory is taken up by
> dirty pages (data waiting to be saved to disk), the process which is
> generating disk writes will itself start writing out dirty data. It
> means, that program will be made to stop using system buffers to hide
> cost of writes, and to write the data to the disk.
> 
> When vm.dirty_background_ratio of total system memory is taken up by
> dirty pages, the pdflush background writeback daemon will start
> writing out dirty data.
> 
> If you keep these lower, the system will try to prevent accumulation
> of large amounts of data to write. Therefore, when sync comes, you
> will not have to wait for the accumulated data to be written to disk.
> 
> vm tunables are explained here:
> https://www.kernel.org/doc/Documentation/sysctl/vm.txt.

Thanks for the nice explanation. But sadly this doesn't help either.
Earlier the load average on login would be ~ 1-1.5. With these changes,
it starts at ~ 2 and after about 5 minutes settles to ~ 1-1.2. For
example now my laptop has been running for 50 minutes and load is
hovering over 1. :(

-- 
Suvayu

Open source is the future. It sets us free.
-- 
users mailing list
users@lists.fedoraproject.org
To unsubscribe or change subscription options:
https://admin.fedoraproject.org/mailman/listinfo/users
Guidelines: http://fedoraproject.org/wiki/Mailing_list_guidelines
Have a question? Ask away: http://ask.fedoraproject.org

Re: How to debug high system load?

[Paweł Brodacki]

2012/7/15 suvayu ali :
> Hi Heinz,
>
> Sorry for the late response.
>
> On Thu, Jul 12, 2012 at 7:31 PM, Heinz Diehl  wrote:
>> On 12.07.2012, Suvayu Ali wrote:
(...)
>> If you're using cfq as your scheduler, try this in rc.local:
>>
>> echo "32" > /sys/block/sda/queue/iosched/quantum
>> echo "0" > /sys/block/sda/queue/iosched/slice_idle
>> echo "1" > /sys/block/sda/queue/iosched/low_latency
>> echo "51200" > /sys/block/sda/queue/nr_requests
>>
>> Together with this in /etc/sysctl.conf:
>>
>> vm.dirty_ratio = 10
>> vm.dirty_background_ratio = 5
>>
>
> I tend not to try things that I don't understand. Could please outline
> briefly what the above suggestions do? I would like to understand before
> I try them out.
>
> Thanks,
>

When vm.dirty_ratio percent of total system memory is taken up by
dirty pages (data waiting to be saved to disk), the process which is
generating disk writes will itself start writing out dirty data. It
means, that program will be made to stop using system buffers to hide
cost of writes, and to write the data to the disk.

When vm.dirty_background_ratio of total system memory is taken up by
dirty pages, the pdflush background writeback daemon will start
writing out dirty data.

If you keep these lower, the system will try to prevent accumulation
of large amounts of data to write. Therefore, when sync comes, you
will not have to wait for the accumulated data to be written to disk.

vm tunables are explained here:
https://www.kernel.org/doc/Documentation/sysctl/vm.txt.
-- 
users mailing list
users@lists.fedoraproject.org
To unsubscribe or change subscription options:
https://admin.fedoraproject.org/mailman/listinfo/users
Guidelines: http://fedoraproject.org/wiki/Mailing_list_guidelines
Have a question? Ask away: http://ask.fedoraproject.org

Re: How to debug high system load?

[suvayu ali]

Hi Heinz,

Sorry for the late response.

On Thu, Jul 12, 2012 at 7:31 PM, Heinz Diehl  wrote:
> On 12.07.2012, Suvayu Ali wrote:
>
>> 1. `fsync() on a file': Firefox, Google Chrome, pidgin, emacs
>> 2. `Waiting for buffer IO to complete': jbd2/dm-*
> []
>
> Fsync is expensive, but that's the way it is.
>
> If you're using cfq as your scheduler, try this in rc.local:
>
> echo "32" > /sys/block/sda/queue/iosched/quantum
> echo "0" > /sys/block/sda/queue/iosched/slice_idle
> echo "1" > /sys/block/sda/queue/iosched/low_latency
> echo "51200" > /sys/block/sda/queue/nr_requests
>
> Together with this in /etc/sysctl.conf:
>
> vm.dirty_ratio = 10
> vm.dirty_background_ratio = 5
>

I tend not to try things that I don't understand. Could please outline
briefly what the above suggestions do? I would like to understand before
I try them out.

Thanks,

-- 
Suvayu

Open source is the future. It sets us free.
-- 
users mailing list
users@lists.fedoraproject.org
To unsubscribe or change subscription options:
https://admin.fedoraproject.org/mailman/listinfo/users
Guidelines: http://fedoraproject.org/wiki/Mailing_list_guidelines
Have a question? Ask away: http://ask.fedoraproject.org

Re: How to debug high system load?

[Heinz Diehl]

On 12.07.2012, Suvayu Ali wrote: 

> 1. `fsync() on a file': Firefox, Google Chrome, pidgin, emacs
> 2. `Waiting for buffer IO to complete': jbd2/dm-*
[]

Fsync is expensive, but that's the way it is.

If you're using cfq as your scheduler, try this in rc.local:

echo "32" > /sys/block/sda/queue/iosched/quantum
echo "0" > /sys/block/sda/queue/iosched/slice_idle
echo "1" > /sys/block/sda/queue/iosched/low_latency
echo "51200" > /sys/block/sda/queue/nr_requests

Together with this in /etc/sysctl.conf:

vm.dirty_ratio = 10
vm.dirty_background_ratio = 5

-- 
users mailing list
users@lists.fedoraproject.org
To unsubscribe or change subscription options:
https://admin.fedoraproject.org/mailman/listinfo/users
Guidelines: http://fedoraproject.org/wiki/Mailing_list_guidelines
Have a question? Ask away: http://ask.fedoraproject.org

Re: How to debug high system load?

[Suvayu Ali]

Hi Bryn,

On Thu, Jul 12, 2012 at 03:34:56PM +0100, Bryn M. Reeves wrote:
> 
> It's possible that you have processes spending brief periods in
> D-state - short enough that they are hard to spot in top with the
> default sample period but long enough to count toward the load average
> (it's possible there's some other explanation though but that's what
> I'd try to rule out first).
> 
> You could use tools like blktrace, iotop and latencytop to try to
> investigate further or if you're willing to install systemtap you
> could use the "sleepingBeauties.stp" script to look for processes
> spending >10ms in this state (and dump their stacks when they do).
> 


With latencytop I see that the large latency processes are usually
caused by

1. `fsync() on a file': Firefox, Google Chrome, pidgin, emacs
2. `Waiting for buffer IO to complete': jbd2/dm-*
3. `SCSI disk ioctl': hddtemp

The wait times are of the order of 60ms - 200ms.

`btrace /dev/sda' gives me the following:

CPU0 (8,0):
 Reads Queued:1,572, 22,536KiB  Writes Queued: 5,211, 32,112KiB
 Read Dispatches: 1,612, 22,896KiB  Write Dispatches:  3,956, 39,424KiB
 Reads Requeued:  2 Writes Requeued:  11
 Reads Completed: 2,156, 27,844KiB  Writes Completed:  7,028, 69,784KiB
 Read Merges: 0,  0KiB  Write Merges:  1,568,  6,280KiB
 Read depth: 98 Write depth:  33
 PC Reads Queued: 0,  0KiB  PC Writes Queued:  0,  0KiB
 PC Read Disp.: 136,  0KiB  PC Write Disp.:0,  0KiB
 PC Reads Req.:  22 PC Writes Req.:0
 PC Reads Compl.:90 PC Writes Compl.:  0
 IO unplugs:  1,224 Timer unplugs:35
CPU1 (8,0):
 Reads Queued:  107,  1,272KiB  Writes Queued: 1,162,  6,768KiB
 Read Dispatches:95,  1,140KiB  Write Dispatches:244,  4,708KiB
 Reads Requeued:  0 Writes Requeued:   2
 Reads Completed: 0,  0KiB  Writes Completed:  0,  0KiB
 Read Merges: 0,  0KiB  Write Merges:612,  2,448KiB
 Read depth: 98 Write depth:  33
 PC Reads Queued: 0,  0KiB  PC Writes Queued:  0,  0KiB
 PC Read Disp.:  28,  0KiB  PC Write Disp.:0,
0KiB
 PC Reads Req.:   0 PC Writes Req.:0
 PC Reads Compl.: 0 PC Writes Compl.:  0
 IO unplugs:264 Timer unplugs: 0
CPU2 (8,0):
 Reads Queued:  249,  2,284KiB  Writes Queued: 4,206, 26,020KiB
 Read Dispatches:   238,  2,220KiB  Write Dispatches:  1,463, 22,116KiB
 Reads Requeued:  0 Writes Requeued:   1
 Reads Completed: 0,  0KiB  Writes Completed:  0,
0KiB
 Read Merges: 0,  0KiB  Write Merges:  2,214,  8,952KiB
 Read depth: 98 Write depth:  33
 PC Reads Queued: 0,  0KiB  PC Writes Queued:  0,  0KiB
 PC Read Disp.:  21,  0KiB  PC Write Disp.:0,
0KiB
 PC Reads Req.:   0 PC Writes Req.:0
 PC Reads Compl.: 0 PC Writes Compl.:  0
 IO unplugs:736 Timer unplugs: 9
CPU3 (8,0):
 Reads Queued:  247,  1,752KiB  Writes Queued:   448,  4,884KiB
 Read Dispatches:   214,  1,588KiB  Write Dispatches:210,  3,536KiB
 Reads Requeued:  1 Writes Requeued:   0
 Reads Completed: 0,  0KiB  Writes Completed:  0,  0KiB
 Read Merges:18, 72KiB  Write Merges:184,736KiB
 Read depth: 98 Write depth:  33
 PC Reads Queued: 0,  0KiB  PC Writes Queued:  0,  0KiB
 PC Read Disp.:  23,  0KiB  PC Write Disp.:0,  0KiB
 PC Reads Req.:   0 PC Writes Req.:0
 PC Reads Compl.: 0 PC Writes Compl.:  0
 IO unplugs:171 Timer unplugs: 1

Total (8,0):
 Reads Queued:2,175, 27,844KiB  Writes Queued:11,027, 69,784KiB
 Read Dispatches: 2,159, 27,844KiB  Write Dispatches:  5,873, 69,784KiB
 Reads Requeued:  3 Writes Requeued:  14
 Reads Completed: 2,156, 27,844KiB  Writes Completed:  7,028, 69,784KiB
 Read Merges:18, 72KiB  Write Merges:  4,578, 18,416KiB
 PC Reads Queued: 0,  0KiB  PC Writes Queued:  0,  0KiB
 PC Read Disp.: 208,  0KiB  PC Write Disp.:0,  0KiB
 PC Reads Req.:  22 PC Writes Req.:0
 PC Reads Compl.:90 PC Writes Compl.:  0
 IO unplugs:  2,395 Timer unplugs:45

Throughput (R/W): 6KiB/s / 16KiB/s
Events (8,0): 147,334 entries
Skips: 0 forward (0 -   0.0%)

iotop doesn't show anything in particular other than once or twice one
of the desktop apps like Firefox showing up and going away almost
immediately.

I don't quiet know how to interpret any of the above output though.

PS: I will run the 

Re: How to debug high system load?

[Bryn M. Reeves]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

On 07/12/2012 03:29 PM, Suvayu Ali wrote:
> Hi Bryn,
> 
> On Thu, Jul 12, 2012 at 03:17:01PM +0100, Bryn M. Reeves wrote:
>> 
>> That would normally also be reflected to some extent in CPU usage
>> and process activity as shown in top (unless the problem was
>> buffering them from a really slow network file system or
>> something).
>> 
> 
> I do have AFS (authenticated with Kerberos) on my laptop, although
> I'm not using it at the moment. There are no background jobs that
> should access it either. Only time AFS is used is when I explicitly
> run some jobs (through python scripts) for my research.
> 

Hi Suvayu,

It's possible that you have processes spending brief periods in
D-state - short enough that they are hard to spot in top with the
default sample period but long enough to count toward the load average
(it's possible there's some other explanation though but that's what
I'd try to rule out first).

You could use tools like blktrace, iotop and latencytop to try to
investigate further or if you're willing to install systemtap you
could use the "sleepingBeauties.stp" script to look for processes
spending >10ms in this state (and dump their stacks when they do).

Regards,
Bryn.

-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.12 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAk/+4JAACgkQ6YSQoMYUY95XTACfWX2MwOa2GkWGujMC37NUVaOP
GDoAnjrWVT1pafqf7Qv/Yf1xs4tvta32
=FzSI
-END PGP SIGNATURE-
-- 
users mailing list
users@lists.fedoraproject.org
To unsubscribe or change subscription options:
https://admin.fedoraproject.org/mailman/listinfo/users
Guidelines: http://fedoraproject.org/wiki/Mailing_list_guidelines
Have a question? Ask away: http://ask.fedoraproject.org

Re: How to debug high system load?

[Suvayu Ali]

Hi Bryn,

On Thu, Jul 12, 2012 at 03:17:01PM +0100, Bryn M. Reeves wrote:
> 
> That would normally also be reflected to some extent in CPU usage and
> process activity as shown in top (unless the problem was buffering
> them from a really slow network file system or something).
> 

I do have AFS (authenticated with Kerberos) on my laptop, although I'm
not using it at the moment. There are no background jobs that should
access it either. Only time AFS is used is when I explicitly run some
jobs (through python scripts) for my research.

-- 
Suvayu

Open source is the future. It sets us free.
-- 
users mailing list
users@lists.fedoraproject.org
To unsubscribe or change subscription options:
https://admin.fedoraproject.org/mailman/listinfo/users
Guidelines: http://fedoraproject.org/wiki/Mailing_list_guidelines
Have a question? Ask away: http://ask.fedoraproject.org

Re: How to debug high system load?

[Suvayu Ali]

On Thu, Jul 12, 2012 at 03:12:00PM +0100, Andrew Haley wrote:
> On 07/12/2012 03:02 PM, Suvayu Ali wrote:
> > On Thu, Jul 12, 2012 at 02:29:54PM +0100, Andrew Haley wrote:
> >> On 07/12/2012 12:00 PM, Suvayu Ali wrote:
> >>> So my question is since CPU usage is already so low, what could be
> >>> driving up the system load? This is bugging me since I also see a
> >>> general sluggishness compared to a much faster system early this year.
> >>
> >> top should show you.  How much time in idle and wait states?
> >>
> > 
> > They are as I would expect, large idle times, and almost no wait time:
> > 
> > top - 15:59:42 up  3:24,  6 users,  load average: 0.74, 0.87, 0.95
> > Tasks: 257 total,   1 running, 254 sleeping,   1 stopped,   1 zombie
> > Cpu0: 2.6%us, 4.0%sy, 0.0%ni, 92.7%id, 0.0%wa, 0.7%hi, 0.0%si, 0.0%st
> > Cpu1: 1.0%us, 0.3%sy, 0.0%ni, 98.0%id, 0.0%wa, 0.3%hi, 0.3%si, 0.0%st
> > Cpu2: 4.3%us, 4.3%sy, 0.0%ni, 91.4%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
> > Cpu3: 0.3%us, 0.0%sy, 0.0%ni, 99.7%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
> > Mem:   3843356k total,  3471700k used,   371656k free,   206600k buffers
> > Swap:  6029308k total,0k used,  6029308k free,  1209176k cached
> 
> That's very odd.  Processes in 'D' = uninterruptible sleep ?
> 

I looked through the entire process list all but 3 were "sleeping", 2
were stopped (I did that in a terminal, a less instance and an
emacsclient instance) and one was zombie (again I know this one, it's
the src-hilite attached to the stopped less instance).

> Andrew.
> 

-- 
Suvayu

Open source is the future. It sets us free.
-- 
users mailing list
users@lists.fedoraproject.org
To unsubscribe or change subscription options:
https://admin.fedoraproject.org/mailman/listinfo/users
Guidelines: http://fedoraproject.org/wiki/Mailing_list_guidelines
Have a question? Ask away: http://ask.fedoraproject.org

Re: How to debug high system load?

[Bryn M. Reeves]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

On 07/12/2012 02:36 PM, Steven Stern wrote:
> On 07/12/2012 06:00 AM, Suvayu Ali wrote: rottled down and are at
> about 1%.
>> 
>> So my question is since CPU usage is already so low, what could
>> be driving up the system load? This is bugging me since I also
>> see a general sluggishness compared to a much faster system early
>> this year.
>> 
>> Thanks for any ideas.
>> 
> 
> Are you listening to music?  Doing background uploads or
> downloads?
> 

That would normally also be reflected to some extent in CPU usage and
process activity as shown in top (unless the problem was buffering
them from a really slow network file system or something).

I would think Andrew is correct - the load is most likely being driven
up by processes that are waiting on I/O (D-state, aka uninterruptible
sleep).

On Linux the load average represents the 1m, 5m and 15m decaying
average of the number of processes that are either runnable (waiting
on a runqueue or actually executing on the CPU) /and/ the number of
processes blocked on I/O.

There's a fair bit of inaccurate information about the definition of
the load average on Linux floating around - this comes from the fact
that it's calculated slightly differently on the Sys V UNIXes and BSD
vs. Linux. Traditionally only the runqueue length (tasks running or
waiting to be scheduled) are counted as "active".

On Linux the "number of active processes" includes both tasks on the
runqueue and tasks blocked on I/O; this means it's hard to compare the
numbers meaningfully between BSD/Solaris and Linux even when running
on the same hardware.

Even well-known Linux magazines have managed to print entire articles
that got this completely wrong in the past..

Regards,
Bryn.
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.12 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAk/+3FwACgkQ6YSQoMYUY96EwgCgquBBvyp8Y5gc6NPirj/STDmw
fGcAoLwQzPw1gcVAUgVZuxdZas/08Wp4
=Mznf
-END PGP SIGNATURE-
-- 
users mailing list
users@lists.fedoraproject.org
To unsubscribe or change subscription options:
https://admin.fedoraproject.org/mailman/listinfo/users
Guidelines: http://fedoraproject.org/wiki/Mailing_list_guidelines
Have a question? Ask away: http://ask.fedoraproject.org

Re: How to debug high system load?

[Andrew Haley]

On 07/12/2012 03:02 PM, Suvayu Ali wrote:
> On Thu, Jul 12, 2012 at 02:29:54PM +0100, Andrew Haley wrote:
>> On 07/12/2012 12:00 PM, Suvayu Ali wrote:
>>> So my question is since CPU usage is already so low, what could be
>>> driving up the system load? This is bugging me since I also see a
>>> general sluggishness compared to a much faster system early this year.
>>
>> top should show you.  How much time in idle and wait states?
>>
> 
> They are as I would expect, large idle times, and almost no wait time:
> 
> top - 15:59:42 up  3:24,  6 users,  load average: 0.74, 0.87, 0.95
> Tasks: 257 total,   1 running, 254 sleeping,   1 stopped,   1 zombie
> Cpu0: 2.6%us, 4.0%sy, 0.0%ni, 92.7%id, 0.0%wa, 0.7%hi, 0.0%si, 0.0%st
> Cpu1: 1.0%us, 0.3%sy, 0.0%ni, 98.0%id, 0.0%wa, 0.3%hi, 0.3%si, 0.0%st
> Cpu2: 4.3%us, 4.3%sy, 0.0%ni, 91.4%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
> Cpu3: 0.3%us, 0.0%sy, 0.0%ni, 99.7%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
> Mem:   3843356k total,  3471700k used,   371656k free,   206600k buffers
> Swap:  6029308k total,0k used,  6029308k free,  1209176k cached

That's very odd.  Processes in 'D' = uninterruptible sleep ?

Andrew.

-- 
users mailing list
users@lists.fedoraproject.org
To unsubscribe or change subscription options:
https://admin.fedoraproject.org/mailman/listinfo/users
Guidelines: http://fedoraproject.org/wiki/Mailing_list_guidelines
Have a question? Ask away: http://ask.fedoraproject.org

Re: How to debug high system load?

[Suvayu Ali]

On Thu, Jul 12, 2012 at 08:36:31AM -0500, Steven Stern wrote:
> On 07/12/2012 06:00 AM, Suvayu Ali wrote:
> rottled down and are at about 1%.
> > 
> > So my question is since CPU usage is already so low, what could be
> > driving up the system load? This is bugging me since I also see a
> > general sluggishness compared to a much faster system early this year.
> > 
> > Thanks for any ideas.
> > 
> 
> Are you listening to music?  Doing background uploads or downloads?
> 

No nothing like that. On login, I start two processes `conky -D' and
`emacs --daemon'. Apart from this I only have a few terminals, Firefox
and a few emacs frames open. The Xfce session amanger starts pidgin and
dropbox. That's about all the processes.

I also use OfflineIMAP to sync my email, but that is setup to run every
5 minutes, and it has been setup as a cron job for ages; I never had
problems with it before.

Right now, my network usage is less than a KiB/s, both upload and
download, and my lod average is 0.98!

-- 
Suvayu

Open source is the future. It sets us free.
-- 
users mailing list
users@lists.fedoraproject.org
To unsubscribe or change subscription options:
https://admin.fedoraproject.org/mailman/listinfo/users
Guidelines: http://fedoraproject.org/wiki/Mailing_list_guidelines
Have a question? Ask away: http://ask.fedoraproject.org

Re: How to debug high system load?

[Suvayu Ali]

On Thu, Jul 12, 2012 at 02:29:54PM +0100, Andrew Haley wrote:
> On 07/12/2012 12:00 PM, Suvayu Ali wrote:
> > So my question is since CPU usage is already so low, what could be
> > driving up the system load? This is bugging me since I also see a
> > general sluggishness compared to a much faster system early this year.
> 
> top should show you.  How much time in idle and wait states?
> 

They are as I would expect, large idle times, and almost no wait time:

top - 15:59:42 up  3:24,  6 users,  load average: 0.74, 0.87, 0.95
Tasks: 257 total,   1 running, 254 sleeping,   1 stopped,   1 zombie
Cpu0: 2.6%us, 4.0%sy, 0.0%ni, 92.7%id, 0.0%wa, 0.7%hi, 0.0%si, 0.0%st
Cpu1: 1.0%us, 0.3%sy, 0.0%ni, 98.0%id, 0.0%wa, 0.3%hi, 0.3%si, 0.0%st
Cpu2: 4.3%us, 4.3%sy, 0.0%ni, 91.4%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu3: 0.3%us, 0.0%sy, 0.0%ni, 99.7%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Mem:   3843356k total,  3471700k used,   371656k free,   206600k buffers
Swap:  6029308k total,0k used,  6029308k free,  1209176k cached

> Andrew.

Thanks,

-- 
Suvayu

Open source is the future. It sets us free.
-- 
users mailing list
users@lists.fedoraproject.org
To unsubscribe or change subscription options:
https://admin.fedoraproject.org/mailman/listinfo/users
Guidelines: http://fedoraproject.org/wiki/Mailing_list_guidelines
Have a question? Ask away: http://ask.fedoraproject.org

Re: How to debug high system load?

[Steven Stern]

On 07/12/2012 06:00 AM, Suvayu Ali wrote:
rottled down and are at about 1%.
> 
> So my question is since CPU usage is already so low, what could be
> driving up the system load? This is bugging me since I also see a
> general sluggishness compared to a much faster system early this year.
> 
> Thanks for any ideas.
> 

Are you listening to music?  Doing background uploads or downloads?

-- 
-- Steve


-- 
users mailing list
users@lists.fedoraproject.org
To unsubscribe or change subscription options:
https://admin.fedoraproject.org/mailman/listinfo/users
Guidelines: http://fedoraproject.org/wiki/Mailing_list_guidelines
Have a question? Ask away: http://ask.fedoraproject.org

Re: How to debug high system load?

[Andrew Haley]

On 07/12/2012 12:00 PM, Suvayu Ali wrote:
> So my question is since CPU usage is already so low, what could be
> driving up the system load? This is bugging me since I also see a
> general sluggishness compared to a much faster system early this year.

top should show you.  How much time in idle and wait states?

Andrew.
-- 
users mailing list
users@lists.fedoraproject.org
To unsubscribe or change subscription options:
https://admin.fedoraproject.org/mailman/listinfo/users
Guidelines: http://fedoraproject.org/wiki/Mailing_list_guidelines
Have a question? Ask away: http://ask.fedoraproject.org

How to debug high system load?

[Suvayu Ali]

Hi,

I have a ThinkPad (Core i7, 2.67GHz) with F17 and I'm seeing high system
loads persistently. Right from boot up I see loads of 1, and often it
goes up to 2 or more.

Things I have looked at:
1. I'm using Xfce, so the desktop environment is also light.
2. I looked at the output of `systemctl list-units' and don't see any
   unnecessary services enabled.
3. top only says currently Xorg is the most active process with 1% CPU
   activity. From the activity summary of atop since boot, I see it's
   Xorg with 3% CPU usage. And while idling all my 4 logical cores are
   throttled down and are at about 1%.

So my question is since CPU usage is already so low, what could be
driving up the system load? This is bugging me since I also see a
general sluggishness compared to a much faster system early this year.

Thanks for any ideas.

-- 
Suvayu

Open source is the future. It sets us free.
-- 
users mailing list
users@lists.fedoraproject.org
To unsubscribe or change subscription options:
https://admin.fedoraproject.org/mailman/listinfo/users
Guidelines: http://fedoraproject.org/wiki/Mailing_list_guidelines
Have a question? Ask away: http://ask.fedoraproject.org