Re: CPU time accounting broken on 8-STABLE machine after a few hours of uptime

2010-09-28 Thread Don Lewis
On 28 Sep, Don Lewis wrote: > Looking at the timestamps of things and comparing to my logs, I > discovered that the last instance of ntp instability happened when I was > running "make index" in /usr/ports. I tried it again with entertaining > results. After a while, the machine became unrespons

Re: cpu timer issues

2010-09-28 Thread Jurgen Weber
Interesting, using systat everything looks fine. The interrupts hang around 2000. Thanks Jurgen On 28/09/10 8:33 PM, borislav nikolov wrote: Hello, vmsat -i calculates interrupt rate based on interrupt count/uptime, and the interrupt count is 32 bit integer. With high values of kern.hz it wi

Re: fetch: Non-recoverable resolver failure

2010-09-28 Thread Brian Reichert
On Tue, Sep 28, 2010 at 08:12:00PM +0200, Miroslav Lachman wrote: > The exact lines from crontab are: > > */5 * * * * fetch -qo /dev/null > "https://hiden.example.com/cron/fiveminutes"; > > */5 * * * * fetch -qo /dev/null > "http://another.example.com/wd.php?hash=cslhakjs87LJ3rysalj79"; In a

Re: cpu timer issues

2010-09-28 Thread Jurgen Weber
Andriy You can find everything you are after here: http://pastebin.com/WH4V2W0F Thanks Jurgen On 28/09/10 8:07 PM, Andriy Gapon wrote: on 28/09/2010 10:54 Jurgen Weber said the following: # dmesg | grep Timecounter Timecounter "i8254" frequency 1193182 Hz quality 0 Timecounters tick every 1

Re: cpu timer issues

2010-09-28 Thread Jurgen Weber
Jeremy Thanks for having a look. Nothing in loader.conf # cat /etc/sysctl.conf # Do not send RSTs for packets to closed ports net.inet.tcp.blackhole=2 # Do not send ICMP port unreach messages for closed ports net.inet.udp.blackhole=1 # Generate random IP_ID's net.inet.ip.random_id=1 # Breaks RF

Re: Still getting kmem exhausted panic

2010-09-28 Thread Artem Belevich
On Tue, Sep 28, 2010 at 3:22 PM, Andriy Gapon wrote: > BTW, have you seen my posts about UMA and ZFS on hackers@ ? > I found it advantageous to use UMA for ZFS I/O buffers, but only after > reducing > size of per-CPU caches for the zones with large-sized items. > I further modified the code in my

Re: CPU time accounting broken on 8-STABLE machine after a few hours of uptime

2010-09-28 Thread Don Lewis
On 28 Sep, Jeremy Chadwick wrote: > Still speaking purely about ntpd: > > The above doesn't indicate a single problem. The deltas shown in both > delay, offset, and jitter are all 100% legitimate. A dd (to induce more > interrupt use) isn't going to exacerbate the problem (depending on your > s

[releng_8 tinderbox] failure on mips/mips

2010-09-28 Thread FreeBSD Tinderbox
TB --- 2010-09-28 19:44:25 - tinderbox 2.6 running on freebsd-current.sentex.ca TB --- 2010-09-28 19:44:25 - starting RELENG_8 tinderbox run for mips/mips TB --- 2010-09-28 19:44:25 - cleaning the object tree TB --- 2010-09-28 19:45:51 - cvsupping the source tree TB --- 2010-09-28 19:45:51 - /usr/b

Still getting kmem exhausted panic

2010-09-28 Thread Leroy van Logchem
>> Thanks for the clarification. I just wish I knew how vm.kmem_size_scale >> fit into the picture (meaning what it does, etc.). The sysctl >> description isn't very helpful. Again, my lack of VM knowledge... >> >Roughly, vm.kmem_size would get set to divided by >vm.kmem_size_scale. http://lis

Re: Still getting kmem exhausted panic

2010-09-28 Thread Andriy Gapon
on 29/09/2010 01:01 Ben Kelly said the following: > Thanks. Yea, there is a lot of aggressive tuning there. In particular, the > slow growth algorithm is somewhat dubious. What I found, though, was that > the fragmentation jumped whenever the arc was reduced in size, so it was an > attempt to ma

Re: Still getting kmem exhausted panic

2010-09-28 Thread Ben Kelly
On Sep 28, 2010, at 5:30 PM, Andriy Gapon wrote: << snipped lots of good info here... probably won't have time to look at it in detail until the weekend >> >> there seems to be a layering violation in that the buffer cache signals >> directly to the upper page daemon layer to trigger page recla

Re: Still getting kmem exhausted panic

2010-09-28 Thread Andriy Gapon
on 28/09/2010 21:40 Ben Kelly said the following: > > On Sep 28, 2010, at 1:17 PM, Andriy Gapon wrote: > >> on 28/09/2010 19:46 Ben Kelly said the following: >>> Hmm. My server is currently idle with no I/O happening: >>> >>> kstat.zfs.misc.arcstats.c: 25165824 kstat.zfs.misc.arcstats.c_max: >>

Re: CPU time accounting broken on 8-STABLE machine after a few hours of uptime

2010-09-28 Thread Don Lewis
On 28 Sep, Don Lewis wrote: > % vmstat -i > interrupt total rate > irq0: clk 60683442 1000 > irq1: atkbd0 6 0 > irq8: rtc7765537127 > irq9: acpi0

Re: CPU time accounting broken on 8-STABLE machine after a few hours of uptime

2010-09-28 Thread Don Lewis
On 28 Sep, Jeremy Chadwick wrote: > On Tue, Sep 28, 2010 at 10:15:34AM -0700, Don Lewis wrote: >> My time source is another FreeBSD box with a GPS receiver on my LAN. My >> other client machine isn't seeing these time jumps. The only messages >> from ntp in its log from this period are these: >>

Re: fetch: Non-recoverable resolver failure

2010-09-28 Thread Miroslav Lachman
Jeremy Chadwick wrote: On Tue, Sep 28, 2010 at 08:12:00PM +0200, Miroslav Lachman wrote: Hi, we are using fetch command from cron to run PHP scripts periodically and sometimes cron sends error e-mails like this: fetch: https://hiden.example.com/cron/fiveminutes: Non-recoverable resolver failur

Re: resume slow on Thinkpad T42 FreeBSD 8-STABLE

2010-09-28 Thread Vitaly Magerya
Chuck Swiger wrote: >> MCA: Bank 1, Status 0xe20001f5 >> MCA: Global Cap 0x0005, Status 0x >> MCA: Vendor "GenuineIntel", ID 0x695, APIC ID 0 >> MCA: CPU 0 UNCOR PCC OVER DCACHE L1 ??? error > > That is very likely to be a matter of luck. If I translate thi

Re: resume slow on Thinkpad T42 FreeBSD 8-STABLE

2010-09-28 Thread Chuck Swiger
On Sep 28, 2010, at 12:57 PM, Vitaly Magerya wrote: > I also have this kernel message once in a few hours (seemingly random) > if I used sleep/resume before: > > MCA: Bank 1, Status 0xe20001f5 > MCA: Global Cap 0x0005, Status 0x > MCA: Vendor "GenuineIntel",

Re: resume slow on Thinkpad T42 FreeBSD 8-STABLE

2010-09-28 Thread Vitaly Magerya
Jung-uk Kim wrote: >> - the mouse doesn't work until I restart moused manually > > I always use hint.psm.0.flags="0x6000" in /boot/loader.conf, i.e., > turn on both HOOKRESUME and INITAFTERSUSPEND, to work around similar > problem on different laptop. Yes, that helps (after the stall period).

[releng_8 tinderbox] failure on i386/pc98

2010-09-28 Thread FreeBSD Tinderbox
TB --- 2010-09-28 18:55:35 - tinderbox 2.6 running on freebsd-current.sentex.ca TB --- 2010-09-28 18:55:35 - starting RELENG_8 tinderbox run for i386/pc98 TB --- 2010-09-28 18:55:35 - cleaning the object tree TB --- 2010-09-28 18:58:07 - cvsupping the source tree TB --- 2010-09-28 18:58:07 - /usr/b

Re: fetch: Non-recoverable resolver failure

2010-09-28 Thread Jeremy Chadwick
On Tue, Sep 28, 2010 at 08:12:00PM +0200, Miroslav Lachman wrote: > Hi, > > we are using fetch command from cron to run PHP scripts periodically > and sometimes cron sends error e-mails like this: > > fetch: https://hiden.example.com/cron/fiveminutes: Non-recoverable > resolver failure > > The e

Re: Still getting kmem exhausted panic

2010-09-28 Thread Ben Kelly
On Sep 28, 2010, at 1:17 PM, Andriy Gapon wrote: > on 28/09/2010 19:46 Ben Kelly said the following: >> Hmm. My server is currently idle with no I/O happening: >> >> kstat.zfs.misc.arcstats.c: 25165824 >> kstat.zfs.misc.arcstats.c_max: 46137344 >> kstat.zfs.misc.arcstats.size: 91863156 >> >

fetch: Non-recoverable resolver failure

2010-09-28 Thread Miroslav Lachman
Hi, we are using fetch command from cron to run PHP scripts periodically and sometimes cron sends error e-mails like this: fetch: https://hiden.example.com/cron/fiveminutes: Non-recoverable resolver failure The exact lines from crontab are: */5 * * * * fetch -qo /dev/null "https://hiden.

Re: CPU time accounting broken on 8-STABLE machine after a few hours of uptime

2010-09-28 Thread Jeremy Chadwick
On Tue, Sep 28, 2010 at 10:15:34AM -0700, Don Lewis wrote: > My time source is another FreeBSD box with a GPS receiver on my LAN. My > other client machine isn't seeing these time jumps. The only messages > from ntp in its log from this period are these: > > Sep 23 04:12:23 mousie ntpd[]: ke

Re: Still getting kmem exhausted panic

2010-09-28 Thread Andriy Gapon
on 28/09/2010 20:17 Andriy Gapon said the following: > on 28/09/2010 19:46 Ben Kelly said the following: >> If what you say is true, this shouldn't happen, should it? This system is >> an i386 machine with kmem max at 800M and arc set to 40M. This is running >> head from April 6, 2010, so it is

Re: Still getting kmem exhausted panic

2010-09-28 Thread Andriy Gapon
on 28/09/2010 19:46 Ben Kelly said the following: > Hmm. My server is currently idle with no I/O happening: > > kstat.zfs.misc.arcstats.c: 25165824 > kstat.zfs.misc.arcstats.c_max: 46137344 > kstat.zfs.misc.arcstats.size: 91863156 > > If what you say is true, this shouldn't happen, should

Re: CPU time accounting broken on 8-STABLE machine after a few hours of uptime

2010-09-28 Thread Don Lewis
On 28 Sep, Chip Camden wrote: > Quoth Don Lewis on Monday, 27 September 2010: >> CPU time accounting is broken on one of my machines running 8-STABLE. I >> ran a test with a simple program that just loops and consumes CPU time: >> >> % time ./a.out >> 94.544u 0.000s 19:14.10 8.1% 62+2054k 0+0io 0

Re: Still getting kmem exhausted panic

2010-09-28 Thread Ben Kelly
On Sep 28, 2010, at 12:30 PM, Andriy Gapon wrote: > on 28/09/2010 18:50 Ben Kelly said the following: >> >> On Sep 28, 2010, at 9:36 AM, Andriy Gapon wrote: >>> Well, no time for me to dig through all that history. arc_max should be a >>> hard limit and it is now. If it ever wasn't then it was a

Re: Still getting kmem exhausted panic

2010-09-28 Thread Andriy Gapon
on 28/09/2010 18:50 Ben Kelly said the following: > > On Sep 28, 2010, at 9:36 AM, Andriy Gapon wrote: >> Well, no time for me to dig through all that history. arc_max should be a >> hard limit and it is now. If it ever wasn't then it was a bug. > > I believe the size of the arc could exceed the

Re: Still getting kmem exhausted panic

2010-09-28 Thread Ben Kelly
On Sep 28, 2010, at 9:36 AM, Andriy Gapon wrote: > Well, no time for me to dig through all that history. > arc_max should be a hard limit and it is now. If it ever wasn't then it was a > bug. I believe the size of the arc could exceed the limit if your working set was larger than arc_max. The

Re: CPU time accounting broken on 8-STABLE machine after a few hours of uptime

2010-09-28 Thread Chip Camden
Quoth Don Lewis on Monday, 27 September 2010: > CPU time accounting is broken on one of my machines running 8-STABLE. I > ran a test with a simple program that just loops and consumes CPU time: > > % time ./a.out > 94.544u 0.000s 19:14.10 8.1% 62+2054k 0+0io 0pf+0w > > The display in top shows

Re: Still getting kmem exhausted panic

2010-09-28 Thread Andriy Gapon
on 28/09/2010 17:30 Willem Jan Withagen said the following: > So in my case (no other serious apps) with 12G phys mem: > > vm.kmem_size=17G > vfs.zfs.arc_max=11G > Should be good. -- Andriy Gapon ___ freebsd-stable@freebsd.org mailing list http://lis

Re: Still getting kmem exhausted panic

2010-09-28 Thread Willem Jan Withagen
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 28-9-2010 16:25, Andriy Gapon wrote: > on 28/09/2010 17:09 Willem Jan Withagen said the following: >> On 28-9-2010 16:07, Andriy Gapon wrote: >>> on 28/09/2010 17:02 Willem Jan Withagen said the following: I do have (read) this document, but st

Re: Still getting kmem exhausted panic

2010-09-28 Thread Andriy Gapon
on 28/09/2010 17:09 Willem Jan Withagen said the following: > On 28-9-2010 16:07, Andriy Gapon wrote: >> on 28/09/2010 17:02 Willem Jan Withagen said the following: >>> I do have (read) this document, but still that doesn't really give you >>> guidelines for tuning on FreeBSD. It is a fileserver wi

Re: Still getting kmem exhausted panic

2010-09-28 Thread Willem Jan Withagen
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 28-9-2010 16:07, Andriy Gapon wrote: > on 28/09/2010 17:02 Willem Jan Withagen said the following: >> I do have (read) this document, but still that doesn't really give you >> guidelines for tuning on FreeBSD. It is a fileserver without any serious

Re: Still getting kmem exhausted panic

2010-09-28 Thread Andriy Gapon
on 28/09/2010 17:02 Willem Jan Withagen said the following: > I do have (read) this document, but still that doesn't really give you > guidelines for tuning on FreeBSD. It is a fileserver without any serious > other apps. > I was using "auto-tuned", and that crashed my box. That is what started > t

Re: Still getting kmem exhausted panic

2010-09-28 Thread Willem Jan Withagen
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 28-9-2010 15:46, Andriy Gapon wrote: > on 28/09/2010 16:25 Willem Jan Withagen said the following: >> Well advises seem to vary, and the latest I understood was that >> 8.1-stable did not need any tuning. (The other system with a much >> older kern

Re: Still getting kmem exhausted panic

2010-09-28 Thread Andriy Gapon
on 28/09/2010 16:25 Willem Jan Withagen said the following: > Well advises seem to vary, and the latest I understood was that > 8.1-stable did not need any tuning. (The other system with a much older > kernel is tuned as to what most here are suggesting) > And I was shure led to believe that even s

Re: Still getting kmem exhausted panic

2010-09-28 Thread Andriy Gapon
on 28/09/2010 16:23 Jeremy Chadwick said the following: > On Tue, Sep 28, 2010 at 03:22:01PM +0300, Andriy Gapon wrote: >> I believe that "the trick" is to set vm.kmem_size high enough, eitehr using >> this >> tunable or vm.kmem_size_scale. > > Thanks for the clarification. I just wish I knew ho

Re: Still getting kmem exhausted panic

2010-09-28 Thread Andriy Gapon
on 28/09/2010 16:23 Jeremy Chadwick said the following: > On Tue, Sep 28, 2010 at 03:22:01PM +0300, Andriy Gapon wrote: >> on 28/09/2010 14:50 Jeremy Chadwick said the following: >>> I believe the trick -- Andriy, please correct me if I'm wrong -- is the >> >> Wouldn't hurt to CC me, so that I coul

Re: Still getting kmem exhausted panic

2010-09-28 Thread Willem Jan Withagen
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 28-9-2010 13:50, Jeremy Chadwick wrote: > On Tue, Sep 28, 2010 at 01:24:28PM +0200, Willem Jan Withagen wrote: >> This is with stable as of yesterday,but with an un-tunned ZFS box I >> was still able to generate a kmem exhausted panic. >> Hard panic

Re: Still getting kmem exhausted panic

2010-09-28 Thread Jeremy Chadwick
On Tue, Sep 28, 2010 at 03:22:01PM +0300, Andriy Gapon wrote: > on 28/09/2010 14:50 Jeremy Chadwick said the following: > > I believe the trick -- Andriy, please correct me if I'm wrong -- is the > > Wouldn't hurt to CC me, so that I could do it :-) > > > tuning of vfs.zfs.arc_max, which is now a

Re: Still getting kmem exhausted panic

2010-09-28 Thread Andriy Gapon
on 28/09/2010 14:50 Jeremy Chadwick said the following: > I believe the trick -- Andriy, please correct me if I'm wrong -- is the Wouldn't hurt to CC me, so that I could do it :-) > tuning of vfs.zfs.arc_max, which is now a hard limit rather than a "high > watermark". Not sure what you mean here

[releng_8_1 tinderbox] failure on powerpc/powerpc

2010-09-28 Thread FreeBSD Tinderbox
TB --- 2010-09-28 12:06:50 - tinderbox 2.6 running on freebsd-current.sentex.ca TB --- 2010-09-28 12:06:50 - starting RELENG_8_1 tinderbox run for powerpc/powerpc TB --- 2010-09-28 12:06:50 - cleaning the object tree TB --- 2010-09-28 12:09:09 - cvsupping the source tree TB --- 2010-09-28 12:09:10

Re: Still getting kmem exhausted panic

2010-09-28 Thread Jeremy Chadwick
On Tue, Sep 28, 2010 at 01:24:28PM +0200, Willem Jan Withagen wrote: > This is with stable as of yesterday,but with an un-tunned ZFS box I > was still able to generate a kmem exhausted panic. > Hard panic, just 3 lines. > > The box contains 12Gb memory, runs on a 6 core (with HT) xeon. > 6* 2T WD

Still getting kmem exhausted panic

2010-09-28 Thread Willem Jan Withagen
Hi, This is with stable as of yesterday,but with an un-tunned ZFS box I was still able to generate a kmem exhausted panic. Hard panic, just 3 lines. The box contains 12Gb memory, runs on a 6 core (with HT) xeon. 6* 2T WD black caviar in raidz2 with 2*512Mb mirrored log. The box died while rsy

Re: cpu timer issues

2010-09-28 Thread borislav nikolov
On 28.09.2010, at 10:54, Jurgen Weber wrote: > Hello List > > We have been having issues with some firewall machines of ours using pfSense. > > FreeBSD smash01.ish.com.au 7.2-RELEASE-p5 FreeBSD 7.2-RELEASE-p5 #0: Sun Dec > 6 23:20:31 EST 2009 > sullr...@freebsd_7.2_pfsense_1.2.3_snaps.pfsen

Re: cpu timer issues

2010-09-28 Thread Andriy Gapon
on 28/09/2010 10:54 Jurgen Weber said the following: > # dmesg | grep Timecounter > Timecounter "i8254" frequency 1193182 Hz quality 0 > Timecounters tick every 1.000 msec > # sysctl kern.timecounter.hardware > kern.timecounter.hardware: i8254 > > Only have one timer to choose from. Can you provi

Re: cpu timer issues

2010-09-28 Thread Jeremy Chadwick
On Tue, Sep 28, 2010 at 05:54:15PM +1000, Jurgen Weber wrote: > Hello List > > We have been having issues with some firewall machines of ours using > pfSense. > > FreeBSD smash01.ish.com.au 7.2-RELEASE-p5 FreeBSD 7.2-RELEASE-p5 #0: > Sun Dec 6 23:20:31 EST 2009 > sullr...@freebsd_7.2_pfsense_1

cpu timer issues

2010-09-28 Thread Jurgen Weber
Hello List We have been having issues with some firewall machines of ours using pfSense. FreeBSD smash01.ish.com.au 7.2-RELEASE-p5 FreeBSD 7.2-RELEASE-p5 #0: Sun Dec 6 23:20:31 EST 2009 sullr...@freebsd_7.2_pfsense_1.2.3_snaps.pfsense.org:/usr/obj.pfSense/usr/pfSensesrc/src/sys/pfSense_SMP