MFC: Distributed audit daemon committed (was: svn commit: r243752 - in head: etc etc/defaults etc/mail etc/mtree etc/rc.d share/man/man4 usr.sbin usr.sbin/auditdistd (fwd)) (fwd)
Dear all: Just an FYI that the new distributed audit daemon has been MFC'd to 9-STABLE. As noted in UPDATING, you will need to run "mergemaster -p" before using installkernel or installworld targets in order to add the new "auditdistd" system user. This should be part of the regular update cycle anyway, but after the experience of adding auditdistd in 10-CURRENT, we've discovered that many people are skipping that step in the update cycle, so I figured it best to point out here. (Technically, only installworld requires the user, but the user-check guards in the system Makefiles are enforced for both targets.) More details on the daemon below. Robert N M Watson Computer Laboratory University of Cambridge -- Forwarded message -- Date: Sat, 1 Dec 2012 15:15:11 + (GMT) From: Robert Watson To: curr...@freebsd.org Cc: secur...@freebsd.org Subject: Distributed audit daemon committed (was: svn commit: r243752 - in head: etc etc/defaults etc/mail etc/mtree etc/rc.d share/man/man4 usr.sbin usr.sbin/auditdistd (fwd)) Dear all: I've now committed the build glue required to install the recently merged Audit Distribution Daemon (auditdistd) contributed by the Pawel Dawidek, and sponsored by the FreeBSD Foundation. This allows individual hosts generating audit trails to submit trails to a central audit server for review and safe keeping. Part of the goal is to ensure that a host submitting trail data can't later modify the trails. Pawel uses a variety of useful security- and resilience-related features such as TLS, Capsicum, etc, in auditdistd. As the recent security incident in the FreeBSD.org cluster illustrated, having reliable and detailed audit trails makes a big difference in forensic work, and hopefully this will allow the FreeBSD Project (and our users) to do that better in the future. Robert N M Watson Computer Laboratory University of Cambridge -- Forwarded message -- Date: Sat, 1 Dec 2012 15:11:46 + (UTC) From: Robert Watson To: src-committ...@freebsd.org, svn-src-...@freebsd.org, svn-src-h...@freebsd.org Subject: svn commit: r243752 - in head: etc etc/defaults etc/mail etc/mtree etc/rc.d share/man/man4 usr.sbin usr.sbin/auditdistd Author: rwatson Date: Sat Dec 1 15:11:46 2012 New Revision: 243752 URL: http://svnweb.freebsd.org/changeset/base/243752 Log: Merge a number of changes required to hook up OpenBSM 1.2-alpha2's auditdistd (distributed audit daemon) to the build: - Manual cross references - Makefile for auditdistd - rc.d script, rc.conf entrie - New group and user for auditdistd; associated aliases, etc. The audit trail distribution daemon provides reliable, cryptographically protected (and sandboxed) delivery of audit tails from live clients to audit server hosts in order to both allow centralised analysis, and improve resilience in the event of client compromises: clients are not permitted to change trail contents after submission. Submitted by: pjd Sponsored by: The FreeBSD Foundation (auditdistd) Added: head/etc/rc.d/auditdistd (contents, props changed) head/usr.sbin/auditdistd/ head/usr.sbin/auditdistd/Makefile (contents, props changed) Modified: head/etc/defaults/rc.conf head/etc/ftpusers head/etc/mail/aliases head/etc/master.passwd head/etc/mtree/BSD.var.dist head/etc/rc.d/Makefile head/share/man/man4/audit.4 head/usr.sbin/Makefile Modified: head/etc/defaults/rc.conf == --- head/etc/defaults/rc.conf Sat Dec 1 13:46:37 2012(r243751) +++ head/etc/defaults/rc.conf Sat Dec 1 15:11:46 2012(r243752) @@ -590,6 +590,9 @@ sendmail_rebuild_aliases="NO" # Run newa auditd_enable="NO" # Run the audit daemon. auditd_program="/usr/sbin/auditd"# Path to the audit daemon. auditd_flags="" # Which options to pass to the audit daemon. +auditdistd_enable="NO" # Run the audit daemon. +auditdistd_program="/usr/sbin/auditdistd" # Path to the auditdistd daemon. +auditdistd_flags="" # Which options to pass to the auditdistd daemon. cron_enable="YES"# Run the periodic job daemon. cron_program="/usr/sbin/cron"# Which cron executable to run (if enabled). cron_dst="YES" # Handle DST transitions intelligently (YES/NO) Modified: head/etc/ftpusers == --- head/etc/ftpusers Sat Dec 1 13:46:37 2012(r243751) +++ head/etc/ftpusers Sat Dec 1 15:11:46 2012(r243752) @@ -19,6 +19,7 @@ _pflogd _dhcp uucp pop +auditdistd www hast nobody Modified: head/etc/mail/aliases == --- head/etc/mail/aliases
FYI: Userspace DTrace MFC to stable/8
Dear all:
Just an FYI that I've gone ahead and merged userspace DTrace support to
FreeBSD 8.x from 9.x. While it appeared to pass build tests locally, boot and
run, etc, this is a non-trivial merge, and it's possible I've messed up. If
so, apologies in advance, and I'll try to resolve any problems as quickly as I
can!
And of course, many thanks go to Rui Paulo, who did the port of userspace
DTrace to FreeBSD 9.x with support from the FreeBSD Foundation!
Thanks,
Robert N M Watson
Computer Laboratory
University of Cambridge
-- Forwarded message --
Date: Mon, 28 Feb 2011 23:28:35 +0000 (UTC)
From: Robert Watson
To: src-committ...@freebsd.org, svn-src-...@freebsd.org,
svn-src-sta...@freebsd.org, svn-src-stabl...@freebsd.org
Subject: svn commit: r219107 - in stable/8/sys: amd64/amd64 amd64/include
boot/common cddl/compat/opensolaris/kern cddl/compat/opensolaris/sys
cddl/contrib/opensolaris/uts/common/dtrace cddl/contrib/opensol...
Author: rwatson
Date: Mon Feb 28 23:28:35 2011
New Revision: 219107
URL: http://svn.freebsd.org/changeset/base/219107
Log:
Merge userspace DTrace support from head to stable/8:
r209721:
Merge from vendor-sys/opensolaris:
* add fasttrap files
r209731:
Introduce USD_{SET,GET}{BASE,LIMIT}. These help setting up the user
segment descriptor hi and lo values. Idea from Solaris.
Reviewed by: kib
r209763:
Fix style issues with the previous commit, namely
use-tab-instead-of-space and don't use underscores in macro variables.
Pointed out by: bde
r210292:
Fix typo in comment.
r210357:
MFamd64:
Add USD_GETBASE(), USD_SETBASE(), USD_GETLIMIT() and USD_SETLIMIT().
r210611:
Bump the witness pendlist to 768 to accomodate the increased number of
spinlocks.
r211553:
Add sysname to struct opensolaris_utsname. This is needed by one DTrace
test.
r211566:
Add a sysname char * to struct opensolaris_utsname.
r211606:
Add the FreeBSD definition for the fasttrap ioctls.
r211607:
Add a function compatibility function dtrace_instr_size_isa() that on
FreeBSD does the same as dtrace_dis_isize().
r211608:
Kernel DTrace support for:
o uregs (sson@)
o ustack (sson@)
o /dev/dtrace/helper device (needed for USDT probes)
r211610:
Add more compatibility structure members needed by the upcoming fasttrap
DTrace device.
r211611:
Destroy the helper device when unloading.
r211613:
Fix style issues.
r211614:
Bump KDTRACE_THREAD_ZERO and use M_ZERO as a malloc flag instead of
calling bzero.
r211615:
Remove an elif and add an or-clause.
r211616:
Add an extra comment to the SDT probes definition. This allows us to get
use '-' in probe names, matching the probe names in Solaris.
Add userland SDT probes definitions to sys/sdt.h.
r211617:
Call the systrace_probe_func() when the error value.
r211618:
Port this to FreeBSD. We miss some suword functions, so we use copyout.
r211738:
Port the fasttrap provider to FreeBSD. This provider is responsible for
injecting debugging probes in the userland programs and is the basis for
the pid provider and the usdt provider.
r211744:
MD fasttrap implementation.
r211745:
Replace a pksignal() call with tdksignal().
Pointed out by: kib
r211746:
Update for the recent location of the fasttrap code.
r211747:
Replace structure assignments with explicity memcpy calls. This allows
Clang to compile this file: it was using the builtin memcpy and we want
to use the memcpy defined in gptboot.c. (Clang can't compile boot2 yet).
Submitted by: Dimitry Andric
Reviewed by: jhb
r211751:
Add a trap code for DTrace induced traps.
r211752:
Add two DTrace trap type values. Used by fasttrap.
r211753:
Enable fasttrap and make dtraceall depend on fasttrap when building i386
or amd64.
r211804:
Call the necessary DTrace function pointers when we have different kinds
of traps.
r211813:
Add the necessary DTrace function pointers.
r211839:
Sync DTrace bits with amd64 and fix the build.
r211924:
Register an interrupt vector for DTrace return probes. There is some
code missing in lapic to make sure that we don't overwrite this entry,
but this will be done on a sequent commit.
r211925:
Replace a memory barrier with a mutex barrier.
r211926:
Add the path necessary to find fasttrap_isa.h to CFLAGS.
r211929:
Remove debugging.
r212004:
When DTrace is enabled, make sure we don't overwrite the IDT_DTRACE_RET
entry with an IRQ for some hardware component.
Reviewed by: jhb
r212093:
Make the /dev/dtrace/helper node have the mode 0660. This allows
programs that refuse to run as root (pgsql) to install probes when the
Re: HEADS UP: FreeBSD 6.4 and 8.0 EoLs coming soon
On Wed, 8 Sep 2010, Vadim Goncharov wrote: Which part of "support for the Giant lock *over the network stack* was removed" [emphasis mine] do you not understand? No, component removed was (1), I've underlined. The reason is performance for overall network stack, not ideology. For a practical reasons, "it works but slow" is better than "doesn't work at all (due to absence of code in the src tree)". "Make it work. Make it right. Make it fast. In that order", know this? Sacrificing "work" for "fast"?.. Hmm, if it is not ideology, then what is it?.. Doug has already clarified, but just to follow up with some detail: Moving to a parallel network stack required that all portions of the stack code be updated to operate without the Giant lock present -- the Giant lock was a fundamental assumption in all kernel code in FreeBSD 4.x and earlier. This decade-long project was highly successful, and relied on members of the community stepping forward to adapt a very large code base by adding fine-grained locking to each component. The results have been extremely impressive, allowing our network stack to scale to 8+ CPUs (I'm actually testing with 32-thread systems as part of some network stack work I'm doing right now). Towards the end of the project, it was clear that a few components in the stack had attracted no interest from the community, and as such, were not going to get updated. As such, we went through a public deprecation and removal process, in which we appealed repeatedly for community members to update the code. This included i4b, one of our three ATM implementations, and one of our two IPSEC implementations. I've attached the i4b schedule below (a three-year process), but you can find information on the full process here: http://wiki.freebsd.org/NONMPSAFE_DEORBIT This was not an issue of i4b operating more slowly than the rest of the stack: it was that the code required fundamental architectural changes without which it couldn't compile, let alone run. We're all happy to have ISDN support come back in the tree if there's an owner for doing it! In the end, any significant code base in the kernel requires ownership -- it can continue through many minor changes without an owner, but major retrofits, such as moving to fine-grained locking, need the attention of someone who understands the code, is able to test the code, and has the time to invest in the code. We do a pretty good job at arranging this with a multi-million line code base, all told. Robert DateDoneEvent 18 July 2005yes Post MPSAFE network stack plan to a...@. 04 July 2007yes Disconnect parts of I4B from the build. HEADS-UP to i...@. 17 July 2007yes Post NET_NEEDS_GIANT() reminder to a...@. 27 July 2007yes Remove NET_NEEDS_GIANT(). 22 March 2008 yes Last call to seek for help rewriting I4B to keep it alive. 15 May 2008 yes Final announcement on isdn@ that I4B will be removed from 8/7. 26 May 2008 yes Remove i4b from HEAD. ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: HEADS UP: FreeBSD 6.4 and 8.0 EoLs coming soon
On Wed, 1 Sep 2010, Hans Petter Selasky wrote: - Or whatever other method to get ISDN back in kernel ? It seems code exists :-) http://old.nabble.com/ISDN4BSD-on-8-current-td23919925.html ISDN4BSD package has been updated to compile on FreeBSD 8-current http://www.selasky.org/hans_petter/isdn4bsd/ Apparently needs massaging into main FreeBSD tree. I agree that my I4B code should be re-written somewhat before committed. Possibly we should update the API's present too, to support IP-telephony aswell. Just to clarify things a little for those following it: the original I4B code was removed for entirely practical reasons: it couldn't run without the Giant lock, and support for the Giant lock over the network stack was removed. I'm happy to see ISDN support reintroduced as long as it will see ongoing maintenance/etc. I'm not familiar with Hans's most recent code, but the integration of his USB stack and his recent receipt of a FreeBSD commit bit suggest a promising future. I would suggest trying to rope in a reveiwer and collaborator (perhaps someone like Bjoern Zeeb?) to work through it before considering a merge, however. This is especially important with projects like VIMAGE, network stack parallelism projects, etc, on-going to make sure that the new ISDN code will be able to support these new features rather than become a potential obstacle (as the old code did for the MPSAFEty work). Robert ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: 8.1 speed issues
On Fri, 18 Jun 2010, William D. Colburn (Schlake) wrote: So I've just upgraded from whatever was stable in 2004 to 8.1 (it's a private file server in my house, I pay no attention to it until it crashes), and uh, the speed difference is very noticeable. In short, it's like I bought a brand new machine that eleven times as fast when all I did was upgrade the OS. How on earth did you guys make such a dramatic improvement? Well, I guess the good news is that we did a lot in six years. The bad news is that figuring out which changes helped your specific configuration and workload would be quite tricky. Certainly, our work on SMP optimization for scheduling, network stack, threading, storage, etc, has been a big deal. Thre are also countless protocol improvements in TCP/IP, significant changes in file system code, ... that could well be involved. Needless to say, it's satisfying to hear that it worked out -- I'd like to think we have even further good stuff coming in 9! Robert ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: Results of BIND RFC
On Fri, 2 Apr 2010, Poul-Henning Kamp wrote:
The result of the RFC was that bind is not a mandatory component to make "a
usable system", so you argument suffers from bad logic.
With an eye on the date of Doug's suggestive e-mail, I actually am concerned
that we maintain support for DNSSEC validation in the base system. If this
can be accomplished by keeping DNS debugging tools and the lightweight
resolver in the base, then I'm fine with that world view. However, if we
can't do DNSSEC record validation without installing the BIND package, then
that worries me.
As we go forward, DNSSEC is going to become increasingly important, and being
unable to bootstrap a system will be a problem, and it will become an
increasingly critical part of the security bootstrap process for networked
systems. While some DNSSEC folk consider it anathema ("DNS is not a directory
service!"), the ability to securely distribute keying material via an existing
network service has enourmous value: for example, early DNSSEC prototypes in
the late 1990's/early 2000's included SSH key distribution via cert records in
DNSSEC. Similarly, as proposals to tie DHCP security and mobility security to
DNSSEC expand, any decision to require a package to do DNSSEC would mean any
component depending on that also has to be outside our base.
If all requirements along these lines are met by the lightweight resolver,
then this is less of a concern.
Robert
___
freebsd-stable@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: Survey results very helpful, thanks! (was: Re: net.inet.tcp.timer_race: does anyone have a non-zero value?)
On Mon, 8 Mar 2010, Doug Hardie wrote: I run a number of 4 core systems with em interfaces. These are production systems that are unmanned and located a long way from me. Under unusual conditions it can take up to 6 hours to get there. I have been waiting to switch to 8.0 because of the discussions on the em device and now it sounds like I had better just skip 8.x and wait for 9. 7.2 is working just fine. Not sure that any information in this survey thread should be relevant to that decision. This race has existed since before FreeBSD, having appeared in the original BSD network stack, and is just as present in FreeBSD 7.x as 8.x or 9.x. When I learned about the race during the early 7.x development cycle, I added a counter/statistic to measure how much it happened in practice, but was not able to exercise it in my testing, and so left the counter in to appear in 7.0 and later so that we could perform this survey as core counts/etc increase. The two likely outcomes were "it is never exercised" and "it is exercised but only very infrequently", neither really justifying the quite complex change to correct it given requirements at the time. On-going development work on the virtual network stack is what justifies correcting the bug at this point, moving from detecting and handling the race to preventing it from occuring as an invariant. The motivation here, BTW, is that we'd like to eliminate the type-stable storage requirement for connection state (which ensures that memory once used for a connection block is only ever used for connection blocks in the future), allowing memory to be fully freed when a virtual network stack is destroyed. Using type-stable storage helped address this bug, but was primarily present to reduce the overhead of monitoring using netstat(1). We'll now need to use a slightly more expensive solution (true reference counts) in that context, although in practice it will almost certainly be an unmeasurable cost. Which is to say that while there might be something in the em/altq/... thread to reasonably lead you to avoid 8.0, nothing in the TCP timer race thread should do so, since it affects 7.2 just as much as 8.0. Even if you do see a non-zero counter, that's not a matter for operational concern, just useful from the perspective of a network stack developer to understanding timing and behaviors in the stack. :-) Robert ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Survey results very helpful, thanks! (was: Re: net.inet.tcp.timer_race: does anyone have a non-zero value?)
On Sun, 7 Mar 2010, Robert Watson wrote: If your system shows a non-zero value, please send me a *private e-mail* with the output of that command, plus also the output of "sysctl kern.smp", "uptime", and a brief description of the workload and network interface configuration. For example: it's a busy 8-core web server with roughly X connections/second, and that has three em network interfaces used to load balance from an upstream source. IPSEC is used for management purposes (but not bulk traffic), and there's a local MySQL database. I've now received a number of reports that confirm our suspicion that the race does occur, albeit very rarely, and particularly on systems with many cores or multiple network interfaces. Fixing it is definitely on the TODO for 9.0, both to improve our ability to do multiple virtual network stacks, but with an appropriately scalable fix in mind given our improved TCP scalability for 9.0 as well. Thanks for all the responses, Robert N M Watson Computer Laboratory University of Cambridge ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
net.inet.tcp.timer_race: does anyone have a non-zero value?
Dear all: I'm embarking on some new network stack locking work, which requires me to address a number of loose ends in the current model. A few years ago, my attention was drawn to a largly theoretical race, which had existed in the BSD code since inception. It is detected and handled in practice, but relies on type stability of TCP connection data structures, which will need to change in the future due to on-going virtualization work. I didn't fix it at the time, but did add a counter so that we could see if it was happening in the field -- that counter, net.inet.tcp.timer_race, indicates whether or not the stack has detected it happening (and then handled it). This e-mail is to collect the results of that in-the-field survey. Please check the results of the following command: % sysctl net.inet.tcp.timer_race net.inet.tcp.timer_race: 0 If your system shows a non-zero value, please send me a *private e-mail* with the output of that command, plus also the output of "sysctl kern.smp", "uptime", and a brief description of the workload and network interface configuration. For example: it's a busy 8-core web server with roughly X connections/second, and that has three em network interfaces used to load balance from an upstream source. IPSEC is used for management purposes (but not bulk traffic), and there's a local MySQL database. I've already seen one non-zero report, but would be interested in knowing a bit more about the kinds of situations where it's happening so that I can prioritize fixing it appropriately, but also reason about the frequency at which it happens so we can select a fix that avoids adding significant overhead in the common case. Thanks, Robert N M Watson Computer Laboratory University of Cambridge ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: is dtrace usable?
On Sat, 6 Mar 2010, Alexander Leidinger wrote: Take a look at the DTrace configuration information here: http://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/dtrace.html I've just reread it (despite the fact that I already used it). Some comments: Last time I tried, I didn't see any problems by adding makeoptions WITH_CTF=yes to the kernel config instead of doing make WITH_CTF=1 kernel Did I miss something, and if not, shouldn't we tell about the makeoptions part instead (a kernel rebuild later will not cause trouble when someone forgets to do the WITH_CTF part as it is already in the kernel makefile)? I'll leave John to answer this one, CC line broadended. What is DDB_CTF doing? Some additional dtrace access in ddb? I didn't had it before but the dtrace probes I had in KLDs worked without problems when running dtrace probes from userland. This is a poorly named kernel option, and should probably be changed. This has to do with the kernel loading CTF information for kernel modules. My recollection is hazy, but it may well be required in order for fbt to be used with modules. Robert N M Watson Computer Laboratory University of Cambridge ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: is dtrace usable?
On Sat, 6 Mar 2010, Daniel Braniss wrote: link_elf_obj: symbol lapic_cyclic_clock_func undefined when trying kldload dtraceall this is with a fearly resent 8-stable I'm trying to help Rick Maclem debug the NSF/UDP problem, and I thought it would be a good chance to learn dtrace, but :-( Take a look at the DTrace configuration information here: http://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/dtrace.html And here: http://wiki.freebsd.org/DTrace It looks like options KDTRACE_HOOKS may not be defined in your kernel configuration, but there are some other details, such as WITH_CTF=1, that you'll also need to make sure are appropriately set. Robert ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: FreeBSD-8.0 802.11n support with ath
On Sat, 27 Feb 2010, Spil Oss wrote: Thanks for the confirmation! Is anything known re. a timeline for implementation of wireless-N? (8.1? 9.0?) I know that Rui Paulo is working on this actively; I've added him to the CC line as I'm not sure if he follows freebsd-stable. Robert N M Watson Computer Laboratory University of Cambridge Kind regards, Spil On Sat, Feb 27, 2010 at 1:43 PM, Bernhard Schmidt wrote: On Sat, Feb 27, 2010 at 01:27:09PM +0100, Spil Oss wrote: Hi All, Got myself an Atheros AR5416 card to upgrade my HostAP to Wireless-N speed. Somehow I can't find a way to convince the driver to go into 11n mode # ifconfig wlan0 mode 11b # ifconfig wlan0 mode 11g # ifconfig wlan0 mode 11n ifconfig: SIOCSIFMEDIA (media): Device not configured It's either mode 11na or mode 11ng. The man-page for ifconfig specifies only 11a, 11b and 11g as modes. The part is functioning fine in 11g mode. Am I missing something obvious or is Wireless-N support not fully implemented yet? There is no rate control algo fuer 11n, afaik, you will only be able to use legacy rates. -- Bernhard ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org" ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
The press release (was: Re: 8.0-RELEASE completed...)
On Thu, 26 Nov 2009, Ken Smith wrote: Just a quick note in case there are people here who aren't subscribed to the freebsd-announce@ mailing list. We have completed the 8.0-RELEASE cycle. Details about the release are available from the main web site, in particular the announcement itself is available here: http://www.freebsd.org/releases/8.0R/announce.html Thanks for all the help with testing during the release process, as well as your continued support of FreeBSD. For those wanting to do advocacy work for the release, in addition to the release announcement and highly detail release notes, there's also a press release: http://www.freebsd.org/releases/8.0R/pressrelease.html It is a bit more verbose and salesy than the announcement, but a lot shorter than the release notes. Robert N M Watson Computer Laboratory University of Cambridge ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
[libdispatch-dev] FreeBSD 8-STABLE now supports GCD, libdispatch port updated (fwd)
FYI to FreeBSD 8-STABLE followers who may be interested in using Apple's GCD technology on FreeBSD. GCD, for those who may have missed it, is a concurrent programming framework introdued in Mac OS X Snow Leopard, now also supported on FreeBSD. There are a number of useful links on the wiki page, but this provides the best high-level introduction: http://images.apple.com/macosx/technology/docs/GrandCentral_TB_brief_20090903.pdf My announcement text (and wiki link) below. Robert N M Watson Computer Laboratory University of Cambridge -- Forwarded message -- Date: Fri, 13 Nov 2009 12:21:40 + (GMT) From: Robert Watson To: libdispatch-...@lists.macosforge.org Subject: [libdispatch-dev] FreeBSD 8-STABLE now supports GCD, libdispatch port updated Dear all: Just an FYI that all the parts are now in place to use GCD (libdispatch) on FreeBSD 8-STABLE. You will need the following: - FreeBSD 8-STABLE snapshot from at least 2009-11-01 (r198732) - FreeBSD libdispatch port from at least 2009-11-11 Do the upgrade to the 8-STABLE snapshot first so that the port can find the required kernel features. As a reminder, you can find FreeBSD/GCD port status, quick start guide, and general information here: http://wiki.freebsd.org/GCD I have updated it to reflect 8.x support. Thanks are due to Stacey Son and Stanislav Sedov for their work to make this happen! Robert N M Watson Computer Laboratory University of Cambridge ___ libdispatch-dev mailing list libdispatch-...@lists.macosforge.org http://lists.macosforge.org/mailman/listinfo.cgi/libdispatch-dev ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: Extreme console latency during disk IO (8.0-RC1, previous releases also affected according to others)
On Tue, 13 Oct 2009, Ivan Voras wrote: Thomas Backman wrote: I'm copying this over from the freebsd-performance list, as I'm looking for a few more opinions - not on the problems *I* am having, but rather to check whether the problem is universal or not, and if not, find a possible common factor. In other words: I want to hear about your experiences, *good or bad*! Here's the original thread (not from the beginning, though): http://lists.freebsd.org/pipermail/freebsd-performance/2009-October/003843.html Long story short, my version: when the disk is stressed hard enough, console IO becomes COMPLETELY unbearable. 10+ seconds to switch between windows in screen(1), running (or even typing) simple commands, etc. This happens both via SSH and the serial console. Hmm, this looks familiar - I've noticed it before on the physical (VGA) console and I notice it all the time under VMWare. It sort of looks like disk IO really blocks network IO in this case - I use the VMs over ssh. Real hardware and virtual hardware have vastly different performance properties, so I'd be careful not to assume that the issue described by the original reporter and the issue you're experiencing are the same. In our kernel, low level network protocols will essentially always take precedence over disk I/O activity. So on face value "disk IO really blocks network IO" is highly unlikely. There are two much more likely possibilities: (1) poor VM implementation causes the virtual CPU to be suspended behind synchronous host OS I/O or (2) the network stack is running fine but the interactive user application is getting I/O or locks scheduled behind a bulk process. A useful diagnostic here is to compare the behavior of three kinds of network latency tests: (1) ping from the host OS to the guest OS (2) netperf TCP_RR from the host OS to the guest OS (3) ssh interactive latency If (1) is highly variable during I/O, it's almost certainly a property of the VM technology you're using, and there's nought to be done about it in the guest OS. If (2) but not (1) is highly variable, it may well be a scheduling issue, although under high memory pressure you couldn't rule out paging out of netserver pages/etc causing latency. If (3) but not (1) or (2) is highly variable, it's most likely an I/O scheduling issue, perhaps caused by priority inversion on lockmgr locks on a vnode, disk I/O scheduling leading to starvation, etc. Robert ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: openssh concerns
On Thu, 8 Oct 2009, Oliver Fromme wrote: Are you sure? The majority of BSD machines in my vicinity have multiple accounts. And even if there's only one account, there is no reason to be careless with potential port-takeover risks. Therefore I advise against running critical daemons on unprivileged ports, especially on machines with shell accounts. And if you need to bind to a port >= 1024, use mac_portacl(4) to protect it. It's easy to use. Alternatively you can increase the value of the sysctl net.inet.ip.portrange.reservedhigh, but this is less flexible and might have unwanted side effects. And, for those that haven't already noticed, "options MAC" is compiled into GENERIC on 8.0, so working with MAC policies no longer requires a recompile (or in many cases, even a reboot). Robert N M Watson Computer Laboratory University of Cambridge ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: samba - SIGABRT
On Thu, 8 Oct 2009, Daniel Eischen wrote: While it's probably a bug that the Samba port compiles --pie, it's also a bug that our linking bits aren't handling PIE properly either. The goal is to fix PIE with the non-NULL mapping feature in the immediate future, so with any luck the abort message won't matter too much longer. How about reverting this change or defaulting security.bsd.map_at_zero=1 until either ports can handle this properly or our -pie is fixed, and we've had at least a release with pre-built packages that don't have the problem? Sorry, I should have been more clear: the problem is with run-time linking, not compile-time linking. Kostik has just posted patches for the run-time linker to current@, which should allow the existing binaries to work with map_at_zero=0. If we aren't able to get the run-time linker fixes into 8.0, we will definitely revert the default change for map_at_zero so that it is enabled. However, since there is a significant security benefit to shipping with map_at_zero disabled, I think we should try hard to ship 8.0 with a fixed rtld. Robert ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: samba - SIGABRT
On Thu, 8 Oct 2009, Oliver Lehmann wrote: This was caused by your setting of the following: security.bsd.map_at_zero=0 You can reset that value to 1 and you should be alright to operate like normal otherwise you will have to compile samba over again with the above mentioned configure options. Yeah this caused the problem. I wonder how I could have find this by myself (google is not counting ;)) I mean, the Abort message is not very verbose what the problem is here Hi Oliver-- While it's probably a bug that the Samba port compiles --pie, it's also a bug that our linking bits aren't handling PIE properly either. The goal is to fix PIE with the non-NULL mapping feature in the immediate future, so with any luck the abort message won't matter too much longer. Robert ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: 8.0-RC1: kernel page fault in NLM master thread (VIMAGE or ZFS related?)
On Fri, 25 Sep 2009, Jamie Gritton wrote: It seems to be NFS related. I think the null pointer in question is from the export's anonymous credential. Try the patch below and see if it helps (which I guess means run it overnight and see if it crashes again). I've also patched a similar missing cred prison in GSS_SVC, since I'm not versed enough in NFS/RPC stuff to know if it might be the problem. This is one of the reasons I really dislike "magic" credentials and special handling of NULL credentials -- they always get into code the author doesn't expect, and either there are bad pointer dereferences, or incorrect security decisions. It's almost always the case that a correct credential should have been cached or generated at some earlier point to represent the security context... Robert - Jamie Index: kern/vfs_export.c === --- kern/vfs_export.c (revision 197506) +++ kern/vfs_export.c (working copy) @@ -122,6 +122,8 @@ np->netc_anon->cr_uid = argp->ex_anon.cr_uid; crsetgroups(np->netc_anon, argp->ex_anon.cr_ngroups, argp->ex_anon.cr_groups); + np->netc_anon->cr_prison = &prison0; + prison_hold(np->netc_anon->cr_prison); np->netc_numsecflavors = argp->ex_numsecflavors; bcopy(argp->ex_secflavors, np->netc_secflavors, sizeof(np->netc_secflavors)); @@ -206,6 +208,8 @@ np->netc_anon->cr_uid = argp->ex_anon.cr_uid; crsetgroups(np->netc_anon, argp->ex_anon.cr_ngroups, np->netc_anon->cr_groups); + np->netc_anon->cr_prison = &prison0; + prison_hold(np->netc_anon->cr_prison); np->netc_numsecflavors = argp->ex_numsecflavors; bcopy(argp->ex_secflavors, np->netc_secflavors, sizeof(np->netc_secflavors)); Index: rpc/rpcsec_gss/svc_rpcsec_gss.c === --- rpc/rpcsec_gss/svc_rpcsec_gss.c (revision 197506) +++ rpc/rpcsec_gss/svc_rpcsec_gss.c (working copy) @@ -449,6 +449,8 @@ cr->cr_uid = cr->cr_ruid = cr->cr_svuid = uc->uid; cr->cr_rgid = cr->cr_svgid = uc->gid; crsetgroups(cr, uc->gidlen, uc->gidlist); + cr->cr_prison = &prison0; + prison_hold(cr->cr_prison); *crp = crhold(cr); return (TRUE); ___ freebsd-curr...@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org" ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: Blocked process
On Sat, 22 Aug 2009, Daniel O'Connor wrote: On Fri, 21 Aug 2009, CmdLnKid wrote: came back or the machine was rebooted. I continued for a while using /var/mail over NFS while setting or unset mail variables for the shell. You may also want to check into whether something is trying to acquire a lock on a file over that NFS mount which could accrue some extra time making it seem like a process is hung. We don't have any NFS mounts so I don't think that's it :( Hi Daniel-- A number of issues were corrected over the course of the 6.x life span involving scheduing, including some relating to "lost wakeups". Many bug fixes relating to threading were also introduced (not sure if that's relevant to your workload). While it's never a particularly fun recommendation, I think I'd suggest sliding forward to the most recent 6.x kernel (but otherwise identical configuration), perhaps sticking with your current userspace, and seeing if that resolves the issue. Robert N M Watson Computer Laboratory University of Cambridge ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: Quick 8.0 update: BETA3 builds in progress
On Sat, 22 Aug 2009, Pete French wrote: I don't have a specific ETA on BETA3 going out the door, except to say that so far several architectures have reported back on successful builds, so probably quite soon. Is there any point in making bug reports for BETA2 at this point ? I only got to try it a coupleof days ago, and had a big issue with installing (had to use 7.2 to partition the drive) but I am thinkingthat maybe just waiting and re-doing the test with BETA3 is better ? I think it's probably more useful to wait a day and retry with BETA3. I don't know that the issue you're experiencing has been fixed, but given that there were some partitioning-related fixes (see the wiki page for a complete list) it's possible that something like it was. Robert N M Watson Computer Laboratory University of Cambridge ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: Upgrade FreeBSD 7.1 to 7.2
On Fri, 21 Aug 2009, Miroslav Lachman wrote: I would like to do a binary upgrade from 7.1 to 7.2. I've seen the instructions here: http://www.freebsd.org/releases/7.2R/announce.html I've heard that it's safest to start the machine in single user mode when doing upgrades, but I see no notice about it in the announcement. So my question is: Is it ok to do this binary upgrade without start single user mode first? If no, must I reboot my machine to enter single user mode? I always did upgrade in multiuser for minimalising the downtime of servers. I always do it this way, but accept the risk involved. I actually did my 7.x -> 8.x upgrade on my web/shell server this way, with users logged in, but I think that's probably only for the brave of heart. FWIW, the main problem I ran into with my 8.0 upgrade is that 8.0 uses the uart(4) driver for serial ports, and sio(4), 7.2's default, has been removed. They require mutually exclusive lines in device.hints and different /etc/ttys lines. If you rely on a serial console, I would recommend first switching 7.2 to using uart, pausing for a bit, and then switching forward to 8 so that you separate the risks associated with changing console drivers from those associated with sliding a major kernel version. Robert ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Quick 8.0 update: BETA3 builds in progress
For those tracking the 8.0 release process, BETA3 builds have now started. They were held up for a few days while a few critical issues were resolved: - Changes to make newbus MPSAFE were reverted as they lead to reports of a number of WITNESS warnings and panics during device driver attach/detach. - Another nit in the Subversion->CVS updater was resolved. - The flowtable crash at boot reported by several users has been resolved. - A boot-time hang for users of the htprr driver has been resolved. - ZFS zpool import was fixed. - freebsd-update now backs up the old kernel before installing a new one. You can find a regularly updated release status, the above information, and much more, on the 8.0 release engineering wiki page: http://wiki.freebsd.org/8.0TODO I don't have a specific ETA on BETA3 going out the door, except to say that so far several architectures have reported back on successful builds, so probably quite soon. Robert N M Watson Computer Laboratory University of Cambridge ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Update - RELENG_8 open for business (but you probably noticed)
Just a quick status update from the release engineering team: As was discussed on this mailing list, problems with the Subversion->CVS exporter arose during the RELENG_8 branching process. These have now been resolved, and for the last day or so, pending bug fixes have been rushing into the branch. We'll wait a couple more days while things catch up, and then cut 8.0 BETA3. You can read more about the status of 8.0, including tracking pending and approved fixes, on the 8.0 release engineering wiki: http://wiki.freebsd.org/8.0TODO Your help in testing, diagnosing, and fixing FreeBSD 8.x problems *before* the release is much appreciated! Robert N M Watson Computer Laboratory University of Cambridge ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: FW: 8.0-BETA2 sysinstall ignoring setting of nonInteractive
On Mon, 3 Aug 2009, Rink Springer wrote: On Mon, Aug 03, 2009 at 11:04:31AM -0400, David Boyd wrote: Can someone "PLEASE" commit this fix. This fix looks OK to me; I'll ask re@ for permission. Just a status update: this is in the re@ queue but approval is pending completion of the stable/7 branch. The plan is that this should appear in beta3. Robert N M Watson Computer Laboratory University of Cambridge ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: Loading ng_socket at runtime?
On Wed, 29 Jul 2009, Matthew Fleming wrote: I'm doing a migration from releng/6.1 to stable/7, and one of the many new things is that I get a warning when doing things with ng_socket that didn't used to happen. WARNING: attempt to net_add_domain(netgraph) after domainfinalize() I've wondered about these warnings also, and am not sure they're justified -- any protocol loading after domainfinalize() should expect that timers are already started, etc, and loadable protocols are clearly desirable. - ignore the warning (usually a bad idea...) Probably remove the warning. Some more synchronization is likely required in domain registration than is currently there -- on the other hand, it's probably not a big issue that it's missing as write operations on the domain list are conservative and extremely rare. I've CC'd Max Laier, who added the warnings -- perhaps he could lend some insight into the types of problems he anticipated. The main one I'm aware of is that mutating the domain list on a live system is risky because it's not well-synchronized -- however, adding domains should be pretty safe in practice. Robert ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: smbfs panic when lost connection or unmount --force
On Fri, 10 Jul 2009, Oliver Pinter wrote: It is a kernel panic, when force unmount the smbfs volume or lost the connection with the samba server. This is a NULL pointer dereference in the kernel. Per Attilio's e-mail, a stack trace should help us track it down. Thanks! Robert N M Watson Computer Laboratory University of Cambridge -- Thes OS is: kern.ostype: FreeBSD kern.osrelease: 7.2-STABLE kern.osrevision: 199506 kern.version: FreeBSD 7.2-STABLE #4: Sat Jun 27 21:44:32 CEST 2009 r...@oliverp:/usr/obj/usr/src/sys/stable kern.osreldate: 702103 -- make.conf: CPUTYPE?=core2 CFLAGS= -O2 -fno-strict-aliasing -pipe MODULES_OVERRIDE=smbfs libiconv libmchain zfs opensolaris drm cd9660 cd9660_iconv -- panic message: Jul 10 01:58:39 oliverp syslogd: kernel boot file is /boot/kernel/kernel Jul 10 01:58:39 oliverp kernel: kernel trap 12 with interrupts disabled Jul 10 01:58:39 oliverp kernel: Jul 10 01:58:39 oliverp kernel: Jul 10 01:58:39 oliverp kernel: Fatal trap 12: page fault while in kernel mode Jul 10 01:58:39 oliverp kernel: cpuid = 2; apic id = 02 Jul 10 01:58:39 oliverp kernel: fault virtual address = 0x30 Jul 10 01:58:39 oliverp kernel: fault code = supervisor read data, page not present Jul 10 01:58:39 oliverp kernel: instruction pointer = 0x8:0x80327fd0 Jul 10 01:58:39 oliverp kernel: stack pointer = 0x10:0xff8078360940 Jul 10 01:58:39 oliverp kernel: frame pointer = 0x10:0xff0004c31390 Jul 10 01:58:39 oliverp kernel: code segment= base 0x0, limit 0xf, type 0x1b Jul 10 01:58:39 oliverp kernel: = DPL 0, pres 1, long 1, def32 0, gran 1 Jul 10 01:58:39 oliverp kernel: processor eflags= resume, IOPL = 0 Jul 10 01:58:39 oliverp kernel: current process = 60406 (smbiod0) Jul 10 01:58:39 oliverp kernel: trap number = 12 Jul 10 01:58:39 oliverp kernel: panic: page fault Jul 10 01:58:39 oliverp kernel: cpuid = 2 Jul 10 01:58:39 oliverp kernel: Uptime: 6h51m16s Jul 10 01:58:39 oliverp kernel: Physical memory: 4087 MB Jul 10 01:58:39 oliverp kernel: Dumping 2448 MB:Copyright (c) 1992-2009 The FreeBSD Project. ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org" ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: trap 12
On Fri, 3 Jul 2009, Ian J Hart wrote: Is this likely to be hardware? Details will follow if not. This looks like a kernel NULL pointer deference (faulting address 0x0), which means it is most likely a kernel bug, although it could be triggered by a hardare problem. If this early in the boot or a diskless box, hence no dump device? Robert N M Watson Computer Laboratory University of Cambridge [copied from a screen dump] Fatal trap 12: page fault while in kernel mode cpuid = 1; apic id = 01 fault virtual address = 0x0 fault code = supervisor write data, page not present instruction pointer = 0x8:0x807c6c12 stack pointer = 0x10:0x510e7890 frame pointer = 0x10:0xff00054a6c90 code segment = base 0x0, limit 0xf, type 0x1b = DPL 0, pres 1, long 1 def32 0, gran 1 processor eflags = interrupt enabled, resume, IOPL = 0 current process = 75372 (printf) trap number = 12 panic: page fault cpuid = 1 uptime: 8m2s Cannot dump. No dump device defined. -- ian j hart This message was sent using IMP, the Internet Messaging Program. ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org" ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: RELENG_7 crash
On Tue, 21 Apr 2009, Ruslan Ermilov wrote: Is it possible I am running into some of the interface lock fixes rwatson has been working on ? This box has a lot of ng interfaces which come and go. Perhaps snmp asking about an interface that just went away caused the panic ? I disabled bsnmp since the reboot and the box has been up for 10hrs so far. It's a documented bug: Just to follow up on this thread: I've committed a work-around to 7-STABLE as well as the 7.2 release engineering branch. The on-going ifnet reference-counting/address list locking/etc work in 8-CURRENT is the long-term solution, and I will MFC it after 7.2 is out the door and it's settled some, but with any luck the work-around will prevent instant panics in the described scenario. Robert N M Watson Computer Laboratory University of Cambridge : revision 1.281 : date: 2008/06/26 23:05:28; author: rwatson; state: Exp; lines: +69 -12 : SVN rev 180042 on 2008-06-26 23:05:28Z by rwatson : : Introduce locking around use of ifindex_table, whose use was previously : unsynchronized. While races were extremely rare, we've now had a : couple of reports of panics in environments involving large numbers of : IPSEC tunnels being added very quickly on an active system. : : - Add accessor functions ifnet_byindex(), ifaddr_byindex(), : ifdev_byindex() to replace existing accessor macros. These functions : now acquire the ifnet lock before derefencing the table. : - Add IFNET_WLOCK_ASSERT(). : - Add static accessor functions ifnet_setbyindex(), ifdev_setbyindex(), : which set values in the table either asserting of acquiring the ifnet : lock. : - Use accessor functions throughout if.c to modify and read : ifindex_table. : - Rework ifnet attach/detach to lock around ifindex_table modification. : : Note that these changes simply close races around use of ifindex_table, : and make no attempt to solve the probem of disappearing ifnets. Further ^^ : refinement of this work, including with respect to ifindex_table : resizing, is still required. : : In a future change, the ifnet lock should be converted from a mutex to an : rwlock in order to reduce contention. : : Reviewed and tested by: brooks Cheers, -- Ruslan Ermilov r...@freebsd.org FreeBSD committer ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: RELENG_7 crash
On Tue, 21 Apr 2009, Mike Tancsa wrote:
At 04:53 PM 4/21/2009, Mikolaj Golub wrote:
Just FYI, the same problem has already been registered in pr database as
kern/132734.
Thanks,
http://www.freebsd.org/cgi/query-pr.cgi?pr=132734 does look familiar
:) If you disable the snmpwalk, is the box stable ?
It would be interesting to know if this tweak at least eliminates the instant
panic:
Index: if_mib.c
===
--- if_mib.c(revision 191424)
+++ if_mib.c(working copy)
@@ -82,11 +82,9 @@
return EINVAL;
if (name[0] <= 0 || name[0] > if_index ||
- ifnet_byindex(name[0]) == NULL)
+ (ifp = ifnet_byindex(name[0])) == NULL)
return ENOENT;
- ifp = ifnet_byindex(name[0]);
-
switch(name[1]) {
default:
return ENOENT;
Robert N M Watson
Computer Laboratory
University of Cambridge
___
freebsd-stable@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: RELENG_7 crash
On Wed, 22 Apr 2009, Mikolaj Golub wrote: RW> There are several bugs here, one difficult to fix (lack of RW> refcounting), but also stuff like ifp being derived from an interface RW> number twice, but checked against NULL only the first time (line 85 RW> checked for NULL, re-queried but no check line 88). Fixing the top RW> bit of the function to only query the ifp once and check it for NULL RW> then would be a good idea. More fundamentally, we do need to refcount RW> ifnets when used from the management path, which is not all that hard RW> a change, but preferably to try the easy way first given where we are RW> in the release cycle. I was thinking a bit on this problem (the same issue was reported in kern/132734) and eliminating double call of ifnet_byindex() was the first thing I did. But I decided then that the proper fix would be to wrap all critical code in sysctl_ifdata in IFNET_RLOCK/IFNET_RUNLOCK (the patch is attached). It looks like I am wrong and my idea about how the kernel works is oversimplified? :-) Unfortunately, I didn't manage to reproduce the panic in my environment so I was not able to do some experiments and tests. The problem is that you can't hold IFNET_RLOCK() over the sysctl copyouts. I'm preparing a patch for 8-CURRENT that will add a refcount to struct ifnet to handle this case, but that isn't an MFC candidate for 7.2. If fixing the return value checks for ifnet_byindex() avoids the insta-panic Mike's running into, it's a better 7.2 change at this point. We can target the ifnet refcount changes for 7.3. Robert N M Watson Computer Laboratory University of Cambridge ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: RELENG_7 crash
On Tue, 21 Apr 2009, Mike Tancsa wrote: At 11:31 AM 4/21/2009, Ruslan Ermilov wrote: : : Note that these changes simply close races around use of ifindex_table, : and make no attempt to solve the probem of disappearing ifnets. Further ^^ : refinement of this work, including with respect to ifindex_table : resizing, is still required. : : In a future change, the ifnet lock should be converted from a mutex to an : rwlock in order to reduce contention. Thanks for the info! In the mean, time, apart from disabling snmpwalking, is there anything I can do to mitigate triggering this bug ? The box runs ospf/zebra for routing daemons and mpd53 for l2tp LNS termination. There are several bugs here, one difficult to fix (lack of refcounting), but also stuff like ifp being derived from an interface number twice, but checked against NULL only the first time (line 85 checked for NULL, re-queried but no check line 88). Fixing the top bit of the function to only query the ifp once and check it for NULL then would be a good idea. More fundamentally, we do need to refcount ifnets when used from the management path, which is not all that hard a change, but preferably to try the easy way first given where we are in the release cycle. However, I wonder if your debugger is being totally honest with you. Line 127 is after several other dereferences of ifp, and there are calls to functions with locking, so the compiler really shouldn't have reordered the post-sysctl calls to be before the pre-sysctl calls that also dereference it. Could you try using addr2line and see if it gives you a different line number, and/or check source and object file dates? Robert N M Watson Computer Laboratory University of Cambridge ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: FreeBSD 7.2 Release process starting...
On Wed, 18 Mar 2009, kama wrote: What I meant was the todo page on www.freebsd.org. Like: http://www.freebsd.org/releases/7.2R/TODO.html Where problems and showstoppers where brought up. I found that information very valueble. Especially when the release went overdue I could easily see what caused the delay. The last release I did not really get information about why the release was delayed. At least not as easily as reloading a webpage. We do plan to create such a page, but are currently finding things to populate it with since we're early in the release process yet. We'll send out an e-mail once it's up. Robert N M Watson Computer Laboratory University of Cambridge ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: FreeBSD 7.2 Release process starting...
On Thu, 19 Mar 2009, Jack Raats wrote: One of the most important things for us to keep an eye on in this release is that the boot loader now works on a number of pieces of hardware on which it reressed for 6.4/7.1. If it proves successful, we'll likely also do errata notes and roll new ISOs for 6.4. About FreeBSD 6.4. If you consider to make new 6.4 iso's, perhaps it would be better to think about an 6.5 release It's a question of bandwidth for the release engineering team, ports team, security officer team, etc -- I think a 6.5 is pretty much out of the question right now with 8.0 preparing to ramp up and 7.2 now in flight. That gives us a short menu of options: - Errata patches - Errata patches + ISO reroll - Point release Because of the boot loader issues, errata patches don't really cut it alone, as if you can't install, you definitely can't apply errata patches :-). This suggests a reroll or a point release. For me the distinction remains fuzzy, but I think a key to either approach would be avoiding having to fully re-QA, do BETAs, build new packages, etc. This suggests taking RELENG_6_4 on some date, perhaps rebranching if it's a point release, or not if it's an ISO reroll, and bundling it with exactly the same packages we shipped in 6.4 (etc) and bumping a few documentation parts. We'd cut a release candidate just to make sure we had the bits right, ask people to test install it, etc, but as there would be no new features, we'd expect relatively little change. I think I wouldn't even change the proposed EoL date of the branch -- 7.x is doing very well, and we need developers to focus on getting 8.0 ready to ship. Robert N M Watson Computer Laboratory University of Cambridge ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: FreeBSD 7.2 Release process starting...
On Wed, 18 Mar 2009, Ken Smith wrote:
On Wed, 2009-03-18 at 10:23 +0100, kama wrote:
Is it possible to get back the todo page during this release phase?
During the last couple of releases I simply didn't have time to do
everything and this was one of the things that fell by the wayside.
After the dust from 7.1 settled we started to make arrangements for someone
else to set this up and watch over it for us during the release. It might
take him a little time to get rolling with it (as in it might not appear
immediately upon the release cycle starting) but hopefully we will have
something along these lines come back at some point during this release.
While such lists do have downsides ("you shipped even though problem X on your
showstopper list wasn't fixed!") they also have some nice advantages -- one is
focusing user community testing time and developer community fixing time. At
least in the case of 7.2, though, I'm not aware of any major outstanding
problems, so focuses for testing are presumably just in the areas of major
change from 7.1.
Robert N M Watson
Computer Laboratory
University of Cambridge
___
freebsd-stable@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: NICs locking up, "*tcp_sc_h"
On Sun, 15 Mar 2009, Nick Withers wrote: I'll need to think a bit about a proper fix for this, but you'll find the problem likely goes away if you eliminate all uid/gid/jail rules from your firewall. You could also tweak the syncache logic not to use a retransmit timer, which might slightly extend the time it takes for systems to connect to your host in the presence of packet loss, but would eliminate this transmission path entirely. We'll need a real and more general fix, however, to commit, and I'll look and see what I can come up with. Brilliant, thanks very much. I'll work without uid rules for the time being, then. Could I ask you to file a PR on this problem, btw, with the two traces I singled out as interesting included, then forward me the PR receipt? That will make the problem easier to keep track of. We're currently pondering ways to fix the problem that don't disturb the stability of the ABI, and may have a workaround patch available shortly that's appropriate for MFC. Robert N M Watson Computer Laboratory University of Cambridge ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: FreeBSD 7.2 Release process starting...
On Wed, 18 Mar 2009, kama wrote: Since it's often the case that developers process quite a few outstanding MFCs during the last couple days before a code freeze starts I have changed RELENG_7 to say it is 7.2-PRERELEASE now as a bit of a heads-up that the release cycle is imminent. You might need to be a tiny bit more careful using RELENG_7 right now because the odds of you getting a snapshot of the tree taken part way through someone doing something that required multiple commits goes up during this phase of a release. Is it possible to get back the todo page during this release phase? One of the most important things for us to keep an eye on in this release is that the boot loader now works on a number of pieces of hardware on which it reressed for 6.4/7.1. If it proves successful, we'll likely also do errata notes and roll new ISOs for 6.4. Since superpage support was MFC'd, keeping an eye out for new VM problems is important. A number of audit-related changes have been merged improving support for audit pipes, so extra testing of that functionality would be welcome. It would probably be worth skimming svn logs for stable/7 to see what other testing focuses would be particularly useful. Robert N M Watson Computer Laboratory University of Cambridge ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Appeal for active bug reports relating to TCP, UDP, routing locking in 7-STABLE
Dear all: With 7.2 approaching, I wanted to review the set of known network bug reports (especially panics, hangs, lock order reversals) relating to TCP, UDP, sockets, and routing in 7-STABLE. If you are aware of problems along these that you can confirm definitely occur with 7-STABLE checked out no earlier than 17 March, 2009, please drop me a private e-mail with a pointer to the thread, PR, or a reminder that you've sent me the details already. If you don't have a PR open on the problem, opening one and forwarding me the receipt so I can grab ownership would be most welcome. I don't promise I can get them fixed by the release, but doing a review and prioritizing the bugs that are known is a useful step in that direction. I am specifically not interested in device driver-related problems, not because they shouldn't be fixed, but because there's only so much time in the day and it appears folks like Pyun have it well in hand :-). Robert N M Watson Computer Laboratory University of Cambridge ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: NICs locking up, "*tcp_sc_h"
On Sat, 14 Mar 2009, Nick Withers wrote: Right, here we go! ... Turns out that the problem is a lock cycle triggered by the syncache calling, indirectly, the firewall during output, and the firewall trying to look up the connection for the packet. Thread one: Tracing PID 31 tid 100030 td 0xff00012016e0 sched_switch() at sched_switch+0xdf mi_switch() at mi_switch+0x18b turnstile_wait() at turnstile_wait+0x1c4 _mtx_lock_sleep() at _mtx_lock_sleep+0x76 _mtx_lock_flags() at _mtx_lock_flags+0x95 syncache_lookup() at syncache_lookup+0xee syncache_expand() at syncache_expand+0x38 tcp_input() at tcp_input+0x99b ip_input() at ip_input+0xaf ether_demux() at ether_demux+0x1b9 ether_input() at ether_input+0x1bb fxp_intr() at fxp_intr+0x224 ithread_loop() at ithread_loop+0xe9 fork_exit() at fork_exit+0x112 fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xfffe80174d30, rbp = 0 --- This thread holds TCP locks and is trying to acquire the syncache lock. Thread two: sched_switch() at sched_switch+0xdf mi_switch() at mi_switch+0x18b turnstile_wait() at turnstile_wait+0x1c4 _rw_rlock() at _rw_rlock+0x9c ipfw_chk() at ipfw_chk+0x3ac1 ipfw_check_out() at ipfw_check_out+0xb1 pfil_run_hooks() at pfil_run_hooks+0xac ip_output() at ip_output+0x357 syncache_respond() at syncache_respond+0x2fd syncache_timer() at syncache_timer+0x15a softclock() at softclock+0x270 ithread_loop() at ithread_loop+0xe9 fork_exit() at fork_exit+0x112 fork_trampoline() at fork_trampoline+0xe This is the syncache timer holding syncache locks, calling IP output, and IPFW trying to acquire TCP locks. Am I right in thinking that you are using uid/gid/jail firewall rules? They suffer from a fundamental architectural problem in that they require reaching "up" to a higher level of the stack at times when it's not always a good idea to do so. In general we solve the problem by passing "down" the inpcb for a connection in the output path so that TCP doesn't have to look it up -- however, in the case of the syncache we actually don't have the inpcb easily in hand (or at least, we have it, but we can't just lock it because syncache locks are after TCP locks in the lock order...). It transpires that what the firewall really wants is not the inpcb, but the credential, but those are interfaces we can't change right now. I'll need to think a bit about a proper fix for this, but you'll find the problem likely goes away if you eliminate all uid/gid/jail rules from your firewall. You could also tweak the syncache logic not to use a retransmit timer, which might slightly extend the time it takes for systems to connect to your host in the presence of packet loss, but would eliminate this transmission path entirely. We'll need a real and more general fix, however, to commit, and I'll look and see what I can come up with. Robert N M Watson Computer Laboratory University of Cambridge ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: NICs locking up, "*tcp_sc_h"
On Fri, 13 Mar 2009, Nick Withers wrote: Sorry for the original double-post, by the way, not quite sure how that happened... I can reproduce this problem relatively easily, by the way (every 3 days, on average). I meant to say this before, too, but it seems to happen a lot more often on the fxp than on rl. I'm sorry to ask what is probably a very simple question, but is there somewhere I should look to get clues on debugging from a manually generated dump? I tried "panic" after manually envoking the kernel debugger but proved highly inept at getting from the dump the same information "ps" / "where" gave me within the debugger live. If this is, in fact, a TCP input lock leak of some sort, then most likely some particular property of a host your system talks to, or a network it runs over, triggers this (presumably) unusual edge case -- perhaps a firewall that mucks with TCP in a funny way, etc. Of course, it might be something completely different -- the fact that everything is blocked on *tcp_sc_h and *tcp, simply means that something holding TCP locks hasn't released them, and this could happen for a number of reasons. Once you've acquired a crashdump, you can run crashinfo(8), which will produce a summary of useful debugging information. There are some things that are a bit easier to do in the run-time debugger, such as lock analysis, as the run-time debugger is more up-close and personal with in-kernel data structures; other things are easier in kgdb, which has complete source code and C type access. I find kgdb works pretty well for everything but "show much what locks are held". Many of our system monitoring tools, including ps and portions of netstat, can actually be run on crashdumps to report the state of the system at the time it crashed -- take a look at the -M and -N command line arguments, which respectively allow you to point those tools at the crashdump and at a kernel with debugging symbols (typically kernel.debug or kernel.symbols) matching the kernel that was booted at the time of the crash. Robert N M Watson Computer Laboratory University of Cambridge Ta for your help! Robert N M Watson Computer Laboratory University of Cambridge Tracing PID 31 tid 100030 td 0xff00012016e0 sched_switch() at sched_switch+0xf1 mi_switch() at mi_switch+0x18f turnstile_wait() at turnstile_wait+0x1cf _mtx_lock_sleep() at _mtx_lock_sleep+0x76 syncache_lookup() at syncache_lookup+0x176 syncache_expand() at syncache_expand+0x38 tcp_input() at tcp_input+0xa7d ip_input() at ip_input+0xa8 ether_demux() at ether_demux+0x1b9 ether_input() at ether_input+0x1bb fxp_intr() at fxp_intr+0x233 ithread_loop() at ithread_loop+0x17f fork_exit() at fork_exit+0x11f fork_trampoline() at fork_trampoline+0xe A "where" on a process stuck in "*tcp", in this case "[swi4: clock]", gave the somewhat similar: sched_switch() at sched_switch+0xf1 mi_switch() at mi_switch+0x18f turnstile_wait() at turnstile_wait+0x1cf _rw_rlock() at _rw_rlock+0x8c ipfw_chk() at ipfw_chk+0x3ab2 ipfw_check_out() at ipfw_check_out+0xb1 pfil_run_hooks() at pfil_run_hooks+0x9c ip_output() at ip_output+0x367 syncache_respond() at syncache_respond+0x2fd syncache_timer() at syncache_timer+0x15a (...) In this particular case, the fxp0 card is in a lagg with rl0, but this problem can be triggered with either card on their own... The scheduler is SCHED_ULE. I'm not too sure how to give more useful information that this, I'm afraid. It's a custom kernel, too... Do I need to supply information on what code actually exists at the relevant addresses (I'm not at all clued in on how to do this... Sorry!)? Should I chuck WITNESS, INVARIANTS et al. in? I *think* every time this has been triggered there's been a "python2.5" process in the "*tcp" state. This machine runs net-p2p/deluge and generally has at least 100 TCP connections on the go at any given time. Can anyone give me a clue as to what I might do to track this down? Appreciate any pointers. -- Nick Withers email: n...@nickwithers.com Web: http://www.nickwithers.com Mobile: +61 414 397 446 -- Nick Withers email: n...@nickwithers.com Web: http://www.nickwithers.com Mobile: +61 414 397 446 ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: NICs locking up, "*tcp_sc_h"
On Fri, 13 Mar 2009, Robert Watson wrote: Sounds like a lock leak -- if you're running INVARIANTS, then "show allocks" should read WITNESS and "show allchains" would be useful. I've had a report of a TCP lock leak possibly in tcp_input(), but haven't managed to track it down yet -- this could well be it as well. Robert N M Watson Computer Laboratory University of Cambridge ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: NICs locking up, "*tcp_sc_h"
On Fri, 13 Mar 2009, Nick Withers wrote: I recently installed my first amd64 system (currently running RELENG_7 from 2009-03-11) to replace an aged ppc box and have been having dramas with the network locking up. Breaking into the debugger manually and ps-ing shows the network card (e.g., "[irq20: fxp0+]") in state "LL" in "*tcp_sc_h". It seems the process(es) trying to access the card at the time is / are in state "L" in "*tcp". I thought this may have been something-or-other in the fxp driver, so installed an rl card and sadly ran into the issue again. The console appears unresponsive, but I can get into the debugger (and as soon as I have, input I'd sent seems to "go through", e.g., if I hit "Enter" a couple o' times, nothing happens; when I ++ into the debugger a few login prompts pop up before the debugger output). A "where" on the fxp / rl process (thread?) gives (transcribed from the console): Sounds like a lock leak -- if you're running INVARIANTS, then "show allocks" and "show allchains" would be useful. I've had a report of a TCP lock leak possibly in tcp_input(), but haven't managed to track it down yet -- this could well be it as well. Robert N M Watson Computer Laboratory University of Cambridge Tracing PID 31 tid 100030 td 0xff00012016e0 sched_switch() at sched_switch+0xf1 mi_switch() at mi_switch+0x18f turnstile_wait() at turnstile_wait+0x1cf _mtx_lock_sleep() at _mtx_lock_sleep+0x76 syncache_lookup() at syncache_lookup+0x176 syncache_expand() at syncache_expand+0x38 tcp_input() at tcp_input+0xa7d ip_input() at ip_input+0xa8 ether_demux() at ether_demux+0x1b9 ether_input() at ether_input+0x1bb fxp_intr() at fxp_intr+0x233 ithread_loop() at ithread_loop+0x17f fork_exit() at fork_exit+0x11f fork_trampoline() at fork_trampoline+0xe A "where" on a process stuck in "*tcp", in this case "[swi4: clock]", gave the somewhat similar: sched_switch() at sched_switch+0xf1 mi_switch() at mi_switch+0x18f turnstile_wait() at turnstile_wait+0x1cf _rw_rlock() at _rw_rlock+0x8c ipfw_chk() at ipfw_chk+0x3ab2 ipfw_check_out() at ipfw_check_out+0xb1 pfil_run_hooks() at pfil_run_hooks+0x9c ip_output() at ip_output+0x367 syncache_respond() at syncache_respond+0x2fd syncache_timer() at syncache_timer+0x15a (...) In this particular case, the fxp0 card is in a lagg with rl0, but this problem can be triggered with either card on their own... The scheduler is SCHED_ULE. I'm not too sure how to give more useful information that this, I'm afraid. It's a custom kernel, too... Do I need to supply information on what code actually exists at the relevant addresses (I'm not at all clued in on how to do this... Sorry!)? Should I chuck WITNESS, INVARIANTS et al. in? I *think* every time this has been triggered there's been a "python2.5" process in the "*tcp" state. This machine runs net-p2p/deluge and generally has at least 100 TCP connections on the go at any given time. Can anyone give me a clue as to what I might do to track this down? Appreciate any pointers. -- Nick Withers email: n...@nickwithers.com Web: http://www.nickwithers.com Mobile: +61 414 397 446 ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: Where is nfsiod now?
On Sun, 8 Mar 2009, Yoshihiro Ota wrote: I thought rc used to start nfsiod if you set nfs_cilent_enable back years ago. Now, on my 7.1-RELEASE machine, it sets up a couple of sysctls in /etc/rc.d/nfsclient script but not nfsiod. Is nfsiod obsolete by now? It is still on the system; does it still improve nfs performance? In the historic BSD kernel, kernel threads for services such as asynchronous NFS write-behind worker (which is what nfsiod is) were created via user processes that "donated" their thread context to the kernel by blocking in a syscall. In modern FreeBSD kernels, kernel threads can be created and destroyed arbitrarily within the kernel as required, so the size of the worker thread pool is controlled by sysctls -- by default there will be none unless required for an NFS mount, and then a pool will be maintained based on use with a limit. For example, on my box without NFS active, I see: rob...@fledge:/usr/src/sbin/nfsiod> sysctl vfs.nfs | grep iod vfs.nfs.iodmax: 20 vfs.nfs.iodmin: 0 vfs.nfs.iodmaxidle: 120 We have a minimum of 0, a maximum of 20 workers, and idle workers are killed off after a timeout of 120 seconds. When using an NFS client with write-behind, you should see kernel threads named "nfsiod X" where X is the worker thread number. Take a look at src/sys/nfsclient/nfs_nfsiod.c for details. Robert N M Watson Computer Laboratory University of Cambridge ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Panics involving ppp following routing fixes
On Sun, 8 Mar 2009, Ruben van Staveren wrote: Just a minor heads up: I've merged both Kip Macy's lock order fixes to the kernel routing code, and the route locking and reference counting fixes from kern/130652 to stable/7. These fixes should correct a number of reported network-related hangs. We might want to release a subset of these as an errata patch to 7.1 if they shake out well in 7-stable. Unfortunately these changes let my system panic during early boot, around the time when ppp/routing is started. Backing out these changes prevents the panic. I've filed this with some textdumps as: kern/132404: panic sleeping thread after 25th Feb src/sys/net commits http://www.freebsd.org/cgi/query-pr.cgi?pr=132404 Hi Ruben-- I've had a number of similar reports, but this one contained enough information for me to (hopefully) track it down. I've merged a fix from head as r189531 that corrects lock leak in error-handling in the routing socket code. Could you let me know if that helps with the problem? Robert N M Watson Computer Laboratory University of Cambridge ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Various route locking fixes merged to stable/7 (was: Re: Big problems with 7.1 locking up :-()
Just a minor heads up: I've merged both Kip Macy's lock order fixes to the kernel routing code, and the route locking and reference counting fixes from kern/130652 to stable/7. These fixes should correct a number of reported network-related hangs. We might want to release a subset of these as an errata patch to 7.1 if they shake out well in 7-stable. Thanks again, especially, to Pete for his evaluation of bugs and patches, Kip for his fixes in head, and to Dmitrij Tejblum for his submission of the fixes in the above-mentioned PR. Robert N M Watson Computer Laboratory University of Cambridge On Wed, 25 Feb 2009, Robert Watson wrote: On Wed, 25 Feb 2009, Pete French wrote: FYI, I'm currently awaiting testing results from Pete on the MFC of a number of routing table locking fixes, and once that's merged (hopefully tomorrow?) I'll start on the patches in the above PR. I've taken a crash-course in routing table locking in the last few days... :-) Just to let you know that I have had zero crashes since I out the patch live on sunday. Of course thats only three days, but it does look very much like it has fixed it. I am also running with the other routing table patch too.. At this point no news is good news, as it is just sitting there ticking away nicely to itself. I will roll it out to a few more machines over the next few days. But looking good so far, I would encourage other people to try the ptches if they are having problems... Thanks -- I've gone ahead and merged the patch to 7.x (r189026) so that I can look at the PR and get that in-progress. Since the code affected by the PR is no longer in 8.x, I'll merge directly to 7.x, and probably fairly quickly since you've had it in production for a while. Robert N M Watson Computer Laboratory University of Cambridge ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org" ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: Big problems with 7.1 locking up :-(
On Wed, 25 Feb 2009, Pete French wrote: FYI, I'm currently awaiting testing results from Pete on the MFC of a number of routing table locking fixes, and once that's merged (hopefully tomorrow?) I'll start on the patches in the above PR. I've taken a crash-course in routing table locking in the last few days... :-) Just to let you know that I have had zero crashes since I out the patch live on sunday. Of course thats only three days, but it does look very much like it has fixed it. I am also running with the other routing table patch too.. At this point no news is good news, as it is just sitting there ticking away nicely to itself. I will roll it out to a few more machines over the next few days. But looking good so far, I would encourage other people to try the ptches if they are having problems... Thanks -- I've gone ahead and merged the patch to 7.x (r189026) so that I can look at the PR and get that in-progress. Since the code affected by the PR is no longer in 8.x, I'll merge directly to 7.x, and probably fairly quickly since you've had it in production for a while. Robert N M Watson Computer Laboratory University of Cambridge ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: Big problems with 7.1 locking up :-(
On Mon, 23 Feb 2009, aneeth wrote: http://www.freebsd.org/cgi/query-pr.cgi?pr=130652&cat= OK, will give this a try, unless anyone else wants any traces from this locked machine ? Is there a known way to tickle this bug when I've rebooted, to make sure it's fixed ? We'v been having similar issues with a couple of our servers as well (7.0 and 7.1). However the problem shows up only on quad core machines. The dual core machines r running fine. FYI, I'm currently awaiting testing results from Pete on the MFC of a number of routing table locking fixes, and once that's merged (hopefully tomorrow?) I'll start on the patches in the above PR. I've taken a crash-course in routing table locking in the last few days... :-) The patches I sent him are at: http://www.watson.org/~robert/freebsd/20090221-route-locking.diff They do not include the patch from the above PR which I want to handle separately as it's a significantly different issue. Robert N M Watson Computer Laboratory University of Cambridge ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: The machdep.hyperthreading_allowed & ULE weirdness in 7.1
On Mon, 23 Feb 2009, Maxim Sobolev wrote: Unfortunately access to BIOS is not always an option and also some BIOSes don't even provide a feature to turn HTT off. It's not quite that simple -- in a world of device drivers pinning threads to CPUs for workload distribution, callout threads and sched_bind()/sched_pin() for crypto load distribution, etc, you need a whole infrastructure for software-disabled CPUs. Disabling it using the BIOS or device.hints is the only reliable way to do this right now. Changing the architecture of the kernel to disable CPU cores after boot is a significant investment of work, and as I mentioned elsewhere, it is disable to do this so that we can support dynamic reconfiguration in the presence of a hypervisor, but it's highly non-trivial. There may be some shortcuts that can be taken for policy reasons in the probing of CPUs when the topology is detected that avoid the full dynamic solution having to be done in the short-term, that in effect are a short-hand for device.hints entries, but I don't know to what extent the CPU topology from ACPI is available at the point where we'd need to know that. So, are you suggesting that we should disable machdep.hyperthreading_allowed with ULE in 7.x and current to avoid confusion? Possibly even without ULE. Robert N M Watson Computer Laboratory University of Cambridge ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: The machdep.hyperthreading_allowed & ULE weirdness in 7.1
On Mon, 23 Feb 2009, Robert Watson wrote: It's not quite that simple -- in a world of device drivers pinning threads to CPUs for workload distribution, callout threads and sched_bind()/sched_pin() for crypto load distribution, etc, you need a whole infrastructure for software-disabled CPUs. Disabling it using the BIOS or device.hints is the only reliable way to do this right now. Changing the architecture of the kernel to disable CPU cores after boot is a significant investment of work, s/disable/desirable/ Robert N M Watson Computer Laboratory University of Cambridge ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: The machdep.hyperthreading_allowed & ULE weirdness in 7.1
On Mon, 23 Feb 2009, Maxim Sobolev wrote: Robert Watson wrote: In the mean time, it sounds like the sysctl does need to be reimplemented or removed, but one question is how far to take it -- caches are shared to varying degrees at varying levels of the topology. However, I believe the recommendation has generally moved to disabling hyperthreading using the BIOS, as that uses the vendor's notion of hyperthreading. The idea of changing the setting at run-time is currently untenable because we don't have the OS infrastructure to take CPUs out of service, although growing it would be useful in order to support virtual machine dynamic CPU reconfiguration. Well, as far as I know, what SCHED_4BSD does is simply stopping scheduling threads to the logical core(s). One doesn't need infrastructure to take CPU off-line for doing the same in SCHED_ULE. Unfortunately access to BIOS is not always an option and also some BIOSes don't even provide a feature to turn HTT off. It's not quite that simple -- in a world of device drivers pinning threads to CPUs for workload distribution, callout threads and sched_bind()/sched_pin() for crypto load distribution, etc, you need a whole infrastructure for software-disabled CPUs. Disabling it using the BIOS or device.hints is the only reliable way to do this right now. Changing the architecture of the kernel to disable CPU cores after boot is a significant investment of work, and as I mentioned elsewhere, it is disable to do this so that we can support dynamic reconfiguration in the presence of a hypervisor, but it's highly non-trivial. There may be some shortcuts that can be taken for policy reasons in the probing of CPUs when the topology is detected that avoid the full dynamic solution having to be done in the short-term, that in effect are a short-hand for device.hints entries, but I don't know to what extent the CPU topology from ACPI is available at the point where we'd need to know that. Robert N M Watson Computer Laboratory University of Cambridge ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: The machdep.hyperthreading_allowed & ULE weirdness in 7.1
On Sun, 22 Feb 2009, Maxim Sobolev wrote: Hi Jeff, I have a single-CPU system with P4 HTT-enabled processor (7.1-RELEASE-p3), kernel compiled with SCHED_ULE. This is because machdep.hlt_logical_cpus doesn't do what you think it does. It causes HTT cores to invoke the hlt instruction in their idle loop, causing them to sleep until they receive an interrupt. For 4BSD, which uses a "pull" model (on the whole) to bring work from work queues, this means that CPUs will go to sleep and remain that way unless they're actively receiving interrupts. For ULE, which uses "push" as well as "pull", threads will constantly be being shed from too-busy CPUs to apparently idle ones under load. The only reliable way to disable hyperthreading is to do so using your BIOS setting, or use loader.conf to disable probing of the pics on unwanted cores, which will cause the CPUs not to be enumerated and hence not used. We don't support taking CPUs fully offline with any scheduler, although you can use the cpuset facility to prevent threads from running on specific cores. You could imagine teaching ULE (and presumably 4BSD) about policies such as "Don't use HTT cores", or perhaps just cpuset about those policies. Robert N M Watson Computer Laboratory University of Cambridge Copyright (c) 1992-2009 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 7.1-RELEASE-p3 #0: Fri Feb 20 09:53:32 UTC 2009 r...@foo.com:/usr/obj/usr/src/sys/SSP-PRODUCTION7 Timecounter "i8254" frequency 1193182 Hz quality 0 CPU: Intel(R) Pentium(R) 4 CPU 3.00GHz (2992.51-MHz 686-class CPU) Origin = "GenuineIntel" Id = 0xf34 Stepping = 4 Features=0xbfebfbff Features2=0x441d Logical CPUs per core: 2 real memory = 3758030848 (3583 MB) avail memory = 3674083328 (3503 MB) ACPI APIC Table: FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs cpu0 (BSP): APIC ID: 0 cpu1 (AP): APIC ID: 1 ioapic0: Changing APIC ID to 2 ioapic0 irqs 0-23 on motherboard ioapic1 irqs 24-47 on motherboard machdep.cpu_idle_hlt: 1 machdep.hlt_cpus: 0 machdep.hlt_logical_cpus: 0 machdep.logical_cpus_mask: 2 If I flip machdep.hyperthreading_allowed sysctl, I still can see processes scheduled to both logical CPUs, yet, if I check IRQ status in the systat -vm, the CPU1 is not getting any timer interrupts, while CPU0 gets twice the normal amount. I wonder if something is broken - I would expect no processes to be scheduled to the CPU1. machdep.hyperthreading_allowed=1: top: http://sobomax.sippysoft.com/~sobomax/ScreenShot458.png systat -vm: http://sobomax.sippysoft.com/~sobomax/ScreenShot459.png machdep.hyperthreading_allowed=0: top: http://sobomax.sippysoft.com/~sobomax/ScreenShot460.png systat -vm: http://sobomax.sippysoft.com/~sobomax/ScreenShot461.png Please let me know if any other debug information is needed. -Maxim ___ freebsd-curr...@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org" ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: Big problems with 7.1 locking up :-(
On Tue, 17 Feb 2009, Mike Tancsa wrote:
Do you have any other details about these issues ? Were the fixes
ever MFC'd
Earlier today I handed off some patches for Pete to test (attached below),
which he's running alongside the patches in kern/130652. When I run with the
patches, basically an MFC of a subset of Kip's routing improvements in 8.x, I
can no longer reproduce the lock reversal, which will hopefully mean Pete can
no longer reproduce the hang. I plan to merge these in a couple of days once
(with any luck) he's confirmed that is the case. We may want to get a subset
of this patch on the errata note path, if we can get the ICMP redirect fix
down to a very short patch.
Robert N M Watson
Computer Laboratory
University of Cambridge
Merge r185747, r185774, r185807, r185849, r185964, r185965, r186051,
r186052 from head to stable/7; note that only the locking fixes and
invariants checking are added from r185747, but not the move to an
rwlock which would modify the kernel binary interface, nor the move
to a non-recursible lock, which is still seeing problem reports in
head. This corrects, among other things, a deadlock that may occur
when processing incoming ICMP redirects.
r185747:
- convert radix node head lock from mutex to rwlock
- make radix node head lock not recursive
- fix LOR in rtexpunge
- fix LOR in rtredirect
Reviewed by: sam
r185774:
- avoid recursively locking the radix node head lock
- assert that it is held if RTF_RNH_LOCKED is not passed
r185807:
Fix a bug introduced in r185747: rather than dereferencing an
uninitialized *rt to something undefined, use the fibnum that came in as
function argument.
Found with: Coverity Prevent(tm)
CID: 4168
r185849:
fix a reported panic when adding a route and one hit here when deleting a
route
- pass RTF_RNH_LOCKED to rtalloc1_fib in 2 cases where the lock is held
- make sure the rnh lock is held across rt_setgate and rt_getifa_fib
r185964:
Pass RTF_RNH_LOCKED to rtalloc1 sunce the node head is locked, this avoids
a recursive lock panic on inet6 detach.
Reviewed by: kmacy
r185965:
RTF_RNH_LOCKED needs to be passed in the flags arg not report,
apologies to thompsa
r186051:
in6_addroute is called through rnh_addadr which is always called with the
radix node head lock held exclusively. Pass RTF_RNH_LOCKED to rtalloc so
that rtalloc1_fib will not try to re-acquire the lock.
r186052:
don't acquire lock recursively
All original commits to head were by Kip Macy , except r185964 by
.
Reviewed by:bz
Tested by: Pete French
Property changes on: sys
___
Modified: svn:mergeinfo
Merged /head/sys:r185747,185774,185807,185849,185964-185965,186051-186052
Index: sys/netinet/in_rmx.c
===
--- sys/netinet/in_rmx.c(revision 188767)
+++ sys/netinet/in_rmx.c(working copy)
@@ -111,7 +111,7 @@
* ARP entry and delete it if so.
*/
rt2 = in_rtalloc1((struct sockaddr *)sin, 0,
- RTF_CLONING, rt->rt_fibnum);
+ RTF_CLONING|RTF_RNH_LOCKED, rt->rt_fibnum);
if (rt2) {
if (rt2->rt_flags & RTF_LLINFO &&
rt2->rt_flags & RTF_HOST &&
Property changes on: sys/dev/cxgb
___
Modified: svn:mergeinfo
Merged
/head/sys/dev/cxgb:r185747,185774,185807,185849,185964-185965,186051-186052
Property changes on: sys/dev/ath/ath_hal
___
Modified: svn:mergeinfo
Merged
/head/sys/dev/ath/ath_hal:r185747,185774,185807,185849,185964-185965,186051-186052
Index: sys/net/route.c
===
--- sys/net/route.c (revision 188767)
+++ sys/net/route.c (working copy)
@@ -277,6 +277,7 @@
struct rt_addrinfo info;
u_long nflags;
int err = 0, msgtype = RTM_MISS;
+ int needlock;
KASSERT((fibnum < rt_numfibs), ("rtalloc1_fib: bad fibnum"));
if (dst->sa_family != AF_INET) /* Only INET supports > 1 fib now
*/
@@ -290,7 +291,13 @@
rtstat.rts_unreach++;
goto miss2;
}
- RADIX_NODE_HEAD_LOCK(rnh);
+ needlock = !(ignflags & RTF_RNH_LOCKED);
+ if (needlock)
+ RADIX_NODE_HEAD_LOCK(rnh);
+#ifdef INVARIANTS
+ else
+ RADIX_NODE_HEAD_LOCK_ASSERT(rnh);
+#endif
if ((rn = rnh->rnh_matchaddr(dst, rnh)) &&
(rn->rn_flags & RNF_ROOT) == 0) {
/*
@@ -343,7 +350,8 @@
RT_LOCK(newrt);
RT_ADDREF(newrt);
}
- RADIX_NODE_HEAD_UNLOCK(rnh);
+ if (needlock)
+
Re: Big problems with 7.1 locking up :-(
On Tue, 17 Feb 2009, Mike Tancsa wrote: At 05:38 PM 1/29/2009, Robert Watson wrote: On Fri, 9 Jan 2009, Pete French wrote: I have a number of HP 1U servers, all of which were running 7.0 perfectly happily. I have been testing 7.1 in it's various incarnations for the last couple of months on our test server and it has performed perfectly. So the last two days I have been round upgrading all our servers, knowing that I had run the system stably on identical hardware for some time. For those following this other than Pete, who I've been in private correspondence with: it seems that he is running into two different deadlocks in the routing code. One of them (at least) is triggered by a lock order problem relating to the processing of ICMP redirects -- uncommon in most configurations, but quite a few on his network, which triggers quickly under load. Kip Macy has corrected at least one (both?) problems in head, and plans to MFC the fixes in the near future. We'll follow up further once the fixes are merged, and if any further problems transpire. Do you have any other details about these issues ? Were the fixes ever MFC'd Hi Mike, et al, I gave Kip a ping about MFCing the fixes and he said he would do that, but has apparently been preoccupied. I'm working on an MFC patch currently, but as I'm not all that familiar with the routing code, and the bug fixes were mixed with feature enhancements in his original commits, it will probably take me a bit longer to produce a candidate patch. Robert N M Watson Computer Laboratory University of Cambridge ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: FreeBSD 7.1-Stable only support 16 CPUs !?
On Thu, 12 Feb 2009, James Chang wrote:
Does any ever try FreeBSD 7.1-stable on box that has more than 16 CPU?
I got a HP ProLiant DL 785 G5 with 32 core (Quad-Core AMD Opteron(tm)
Processor 8356 (2300.10-MHz K8-class CPU) and 256G memory. When I boot this
machine, it could detect 32 core, but only 16 core could be uesd!?
What should I do to get FreeBSD 7.1-stable work with 32 core? Or, FreeBSD
7.1 only support 16 Core Max ?
I can't speak to the specific hardware you have, but one known limitation in
several versions of FreeBSD is that the MAXCPU constant for i386 and amd64 is
16. You can redefine it in src/sys/${arch}/include/param.h and recompile your
kernel to see the full 32 cores. You may need to recompile some userspace
tools in order for them to properly report statistics for all CPUs, so a full
rebuild of world wouldn't hurt.
Robert N M Watson
Computer Laboratory
University of Cambridge
___
freebsd-stable@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: impossible packet length ...
On Sun, 8 Feb 2009, Danny Braniss wrote:
On Sun, 8 Feb 2009, Danny Braniss wrote:
looking at the bce source, it's not clear (to me :-). If errors are
detected in bce_rx_intr(), the packet gets dropped, which I would expect
to be the treatment of an offloded chekcum error, but it seems that is not
the case.
I think we're thinking of different checksums -- devices/device drivers
drop frames with bad ethernet checksums, but not IP and above layer
checksums.
I know I'm stepping on thin ice hear - haven't touched Stevens for a while,
(and I doubt it mentions offloading), but if the offload checksum is bad,
why not just drop the packet?
The way I read the driver, if the offload checksum is on, and if no errors
where detected, then it's marked as ok.
There are a few good reasons I can think of, but this is hardly a
comprehensive list:
(1) If there are bad higher level checksums on the wire, you want to see them
in tcpdump, so allow them to get up to a higher layer if network layer
checksums aren't good.
(2) It's a matter of local policy as to whether UDP checksums (for example)
are observed or not.
(3) If you're forwarding or bridging packets, it should be up to the end nodes
how they deal with bad UDP checksums on packets to them, not the routers.
Looking at if_bce.c, the following seems to be reasonable logic; first,
ethernet-layer checksums:
5902 /* Check the received frame for errors. */
5903 if (status & (L2_FHDR_ERRORS_BAD_CRC |
5904 L2_FHDR_ERRORS_PHY_DECODE |
L2_FHDR_ERRORS_ALIGNMENT |
5905 L2_FHDR_ERRORS_TOO_SHORT |
L2_FHDR_ERRORS_GIANT_FRAME)) {
5906
5907 /* Log the error and release the mbuf. */
5908 ifp->if_ierrors++;
5909 DBRUN(sc->l2fhdr_status_errors++);
5910
5911 m_freem(m0);
5912 m0 = NULL;
5913 goto bce_rx_int_next_rx;
5914 }
I.e., if there are ethernet-level CRC failures, drop the packet.
5922 /* Validate the checksum if offload enabled. */
5923 if (ifp->if_capenable & IFCAP_RXCSUM) {
5924
5925 /* Check for an IP datagram. */
5926 if (!(status & L2_FHDR_STATUS_SPLIT) &&
5927 (status & L2_FHDR_STATUS_IP_DATAGRAM)) {
5928 m0->m_pkthdr.csum_flags |=
CSUM_IP_CHECKED;
5929
5930 /* Check if the IP checksum is valid. */
5931 if ((l2fhdr->l2_fhdr_ip_xsum ^ 0x) ==
0)
5932 m0->m_pkthdr.csum_flags |=
CSUM_IP_VALID;
5933 }
5934
5935 /* Check for a valid TCP/UDP frame. */
5936 if (status & (L2_FHDR_STATUS_TCP_SEGMENT |
5937 L2_FHDR_STATUS_UDP_DATAGRAM)) {
5938
5939 /* Check for a good TCP/UDP checksum. */
5940 if ((status & (L2_FHDR_ERRORS_TCP_XSUM |
5941 L2_FHDR_ERRORS_UDP_XSUM))
== 0) {
5942 m0->m_pkthdr.csum_data =
5943 l2fhdr->l2_fhdr_tcp_udp_xsum;
5944 m0->m_pkthdr.csum_flags |=
(CSUM_DATA_VALID
5945 | CSUM_PSEUDO_HDR);
5946 }
5947 }
5948 }
Only look at higher level checksums if policy enables it on the interface;
then, only if the hardware has a view on the IP-layer checksums, propagte that
information to the mbuf flags from the descriptor ring entry flags, both
whether or not the checksum was verified, and whether or not it was good. If
policy disables it, or the hardware expresses no view, we don't set flags,
which simply defers checksumming to a higher layer (if required -- for
forwarded packets, we won't test UDP-layer checksums at all).
Robert N M Watson
Computer Laboratory
University of Cambridge
___
freebsd-stable@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: impossible packet length ...
On Sun, 8 Feb 2009, Danny Braniss wrote: looking at the bce source, it's not clear (to me :-). If errors are detected in bce_rx_intr(), the packet gets dropped, which I would expect to be the treatment of an offloded chekcum error, but it seems that is not the case. I think we're thinking of different checksums -- devices/device drivers drop frames with bad ethernet checksums, but not IP and above layer checksums. Robert N M Watson Computer Laboratory University of Cambridge ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: impossible packet length ...
On Sun, 8 Feb 2009, Peter Jeremy wrote: On 2009-Feb-08 11:31:45 +0200, Danny Braniss wrote: Q: with rxcsum on, and a bad checksum packet is received, is it dropped by the NIC? if not, then it somewhat explains the behaviour If checksum offloading is working correctly then a bad packet should be dropped by the NIC. If checksum offloading isn't working correctly then you can wind up in the situation where both the NIC and the driver think the other party has verified the checksum. It's also possible that you may be running into corruption during DMA transfer from the NIC to RAM. ISTR there have been some issues reported recently with checksum offloading on some NICs - though I don't have details to hand - you might like to search the lists. changing the nic is tough, but if needed will be done. If disabling checksum offloading fixes the problem and the additional CPU load is acceptable (at least until you find a real fix) then there's no need to change NICs. Actually, my understanding was that packets with bad checksums are delivered to software, and flag the descriptor ring header for each packet tells us whether the checksum was (a) checked and (b) validated by the hardware. We then propagate these to mbuf flags so that higher stack layers know whether or not to calculate the checksum themselves. Regardless of the specifics, though, packets with checked but bad checksums shouldn't make it to the socket layer where they would be visible to NFS. If the NIC is marking apparently bad packets as good, there are a number of possible sources -- be it bad checksum handling in the card, corruption between the card and higher levels of the stack (a DMA problem, as you point out, would have this symptom). Robert N M Watson Computer Laboratory University of Cambridge ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: jail: external and localhost distinction
On Sat, 7 Feb 2009, Dmitry Morozovsky wrote: On Fri, 6 Feb 2009, Robert Watson wrote: RW> > Thank you for clarification, now I see this is actually expected behaviour RW> > :) RW> > RW> > Would then starting second jail with the same root and, say, 127.10.0.1 as RW> > an address be a workaround? RW> RW> There's no technical reason you can't have more than one jail using the same RW> file system root, and even IP -- you'll find that ps(1) in one jail can't RW> see processes in the other (and can't signal, etc) but otherwise works as RW> expected. Of course, any given process has to be a member of at most one of RW> the two. But, in the case of IP sharing, I suppose, the second process tries to bind to the same port will got "socket already in use", won't it? In general, if two processes independently bind the same port but using two specific IPs, then there won't be a conflict and both will be allowed to succeed. Conflicts arise if there are two bindings of the same address and port, so if both jails use the same IP and one binds it, then the other will get a socket already in use error, yes. FYI, I see that Bjoern has now committed the multi-IP patch for Jail to 7-STABLE, which should make Jails a lot more flexible. Robert N M Watson Computer Laboratory University of Cambridge ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: jail: external and localhost distinction
On Thu, 29 Jan 2009, Dmitry Morozovsky wrote: Thank you for clarification, now I see this is actually expected behaviour :) Would then starting second jail with the same root and, say, 127.10.0.1 as an address be a workaround? There's no technical reason you can't have more than one jail using the same file system root, and even IP -- you'll find that ps(1) in one jail can't see processes in the other (and can't signal, etc) but otherwise works as expected. Of course, any given process has to be a member of at most one of the two. Robert N M Watson Computer Laboratory University of Cambridge ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: To John Birrell: weird behaviors of DTrace on amd64
On Thu, 5 Feb 2009, Klapper Zhu wrote:
I am exploring DTrace on "7.1-STABLE FreeBSD amd64" and I found several
weird behaviors:
1) Not all kernel functions show up in fbt provider. Take isp(4) as example:
"dtrace -l" shows
static void isp_freeze_loopdown(ispsoftc_t *, int, char *);
___but not___
static void isp_handle_platform_atio2(ispsoftc_t *, at2_entry_t *);
Both are static functions. But one shows up in fbt, another not.
What's the rational behind it ? Any way to fix it ?
Possibly gcc decided to inline one but not the other; you could try disabling
inlining and see if the other function appears. fbt is sensitive to a number
of compiler choices, so generally if there's a long-term desire to trace at
that point, we should add explicit trace points. (Solaris makes similar
recommendations -- that fbt is a useful but unstable interface).
2) The symptom described below only shows in 64-bit platform (amd64).
Here is the D Code:
fbt::isp_handle_platform_atio2:entry
{
self->cdb =args[1]->at_cmnd.cdb_dl.sf.fcp_cmnd_cdb[0];
printf("%s(%x, %x, cdb_cmd %x)\n", probefunc, arg0, arg1, self->cdb);
}
It will never fire.
I have to add another 2 probes on top of it, then it
(fbt::isp_handle_platform_atio2:entry) will trace.
Even the 2 probes on top of it never fire.
I've seen a number of cases where entry fbt points fire but return fbt points
don't; for example, in 8.x I've noticed that fbt::softclock:enter fires each
time the softclock loop runs, even though the function is only entered once
when the kernel thread starts, and that it never fires the return. This
suggests fbt may be putting the probe in the wrong spot, perhaps the beginning
of the block where local variables are used rather than the beginning of the
function itself. I've reported this problem to John also.
Robert N M Watson
Computer Laboratory
University of Cambridge
___
freebsd-stable@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: Puzzling change in performance
On Fri, 30 Jan 2009, Borja Marcos wrote: The attached graphs are from a server running FreeBSD 7.1-i386 (now) with the typical Apache2+MySQL with forums, Joomla... I just cannot explain this. Disk I/O bandwidth was suffering a lot, and after the update the disks are almost idle. Any ideas? I cannot imagine a change between versions to justify this. The latest update to 7-STABLE had been done in August. Maybe it was just degradation due to not being rebooted since August? But it's strange anyway, and I had restarted MySQL and Apache some times during this period. There are basically three ways to go about exploring this, none particularly good: (1) Do a more formal before and after analysis of performance on the box, perhaps using tools like kernel profiling, hwpmc, dtrace, etc. (2) Do a binary search to narrow down the date of the change that improved things until it becomes clear which mattered. (3) Hope someone annecdotally remembers something that might or might not be it and assume they're right. Of these, I'd guess (2) is actually the most effective way to go about it, but is potentially time-consuming. As you point out, the most interesting question is whether, when you go back to 7.0, things suddenly get slower again, or not. Typically long uptimes don't lead to performance problems on FreeBSD (in my experience) so I think that's unlikely to be the source. There are a lot of improvements in 7.1 relating to performance, but none particularly stands out for me as having the effect you describe. If you're really curious, I would try to pin it down with a binary search. Robert N M Watson Computer Laboratory University of Cambridge ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: Big problems with 7.1 locking up :-(
On Fri, 9 Jan 2009, Pete French wrote: I have a number of HP 1U servers, all of which were running 7.0 perfectly happily. I have been testing 7.1 in it's various incarnations for the last couple of months on our test server and it has performed perfectly. So the last two days I have been round upgrading all our servers, knowing that I had run the system stably on identical hardware for some time. For those following this other than Pete, who I've been in private correspondence with: it seems that he is running into two different deadlocks in the routing code. One of them (at least) is triggered by a lock order problem relating to the processing of ICMP redirects -- uncommon in most configurations, but quite a few on his network, which triggers quickly under load. Kip Macy has corrected at least one (both?) problems in head, and plans to MFC the fixes in the near future. We'll follow up further once the fixes are merged, and if any further problems transpire. Robert N M Watson Computer Laboratory University of Cambridge Since then I have starte seeing machines lock up. This always happens under heavy disc load. When I bring the machine back up then sometimes it fails to fsck due to a partialy truncated inode. The locksup appear to be disc related - on my mysql msater machine it will come back up with files somewhat shorted than those which ahve aready been transmitted to the slave (i.e. some data was in memory, and claimed to have been written to the drive, but never made it onto the disc). The only time I have seen anything useful on the screen was during one lockup where I got a message about a spin lock being held too long and some comment in parentheses about it being a turnstile lock. Help! :-( I am now downgrading all the machine to 7.0 as fast as I can - though the machine I am trying to compile it on has locked up once during the compile so I havent got anywhere so far. The machines are HP Proliant DL360 G5s - they have an embedded P400i RAID controller with a pair of mirrored drives connected. Each one has both ethernets connected, bundled using lagg and LACP. Advice ? -pete. ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org" ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: jail: external and localhost distinction
On Thu, 29 Jan 2009, Dmitry Morozovsky wrote: am I right concluding that under FreeBSD jail there is no way to attach two processes to the same port of external interface address and localhost? I tried to move rather standard two-tier nginx(ip:80)+apache(127.1:80) scheme into a jail and on apache start got [Thu Jan 29 00:09:32 2009] [crit] (48)Address already in use: make_sock: could not bind to address 127.0.0.1 port 80 (this is under RELENG_7 if it's relevant) Any thoughts? Thanks in advance. The way Jail is implemented is that the jail IP is silently substituted for the loopback IP is used. This has some downsides, and this is one of them. The virtual network stack (VIMAGE) project for FreeBSD 8.0 is intended to address this, among many other things, by providing full virtualization of all network stack data structures for jails. Robert N M Watson Computer Laboratory University of Cambridge ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: unkillable proceess
On Wed, 21 Jan 2009, dikshie wrote: Hi, how to kill unkillable process: # ps axuf |grep http www 66005 73.4 1.3 87656 13164 ?? R 4:58PM 62:24.41 /usr/local/sbin/httpd -DSSL -DNOHTTPACCEPT www 4277 71.6 1.4 88680 13964 ?? R 4:12PM 48:23.40 /usr/local/sbin/httpd -DSSL -DNOHTTPACCEPT www 2708 65.5 1.4 88680 14112 ?? R 4:12PM 54:39.50 /usr/local/sbin/httpd -DSSL -DNOHTTPACCEPT above processes unkillable Could you show us the output of "procstat -kk" on one or all of the above processes? Robert N M Watson Computer Laboratory University of Cambridge ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: Big problems with 7.1 locking up :-(
On Fri, 16 Jan 2009, Pete French wrote: I rather feared as much. Let's run down the path of "perhaps there's a problem with the new UDP locking code" for a bit and see where it takes us. Is it possible to run those boxes with WITNESS -- I believe that the fact that "show alllocks" is failing is because WITNESS isn't present. Yes, I can do that. The only reason I wasn't running with WITNESS is that it didn't lock up when I added the BREAK_TO_DEBUGGER so I was seeing if a simple GENERIC kernel would lock up when I added that. I will go back and add WITNESS when you tell me theres nothing more we can get out of this lock up (recompiling will involve restarting the machine so I loose the 'boekn to debugger' state). Should I add anything else ? Skip spinlocks ? Invariants ? The other thing we can do is revert UDP to using purely write locks -- the risk there is that it might change the timing but not actually resolve the bug, so if we can analyze it a bit using WITNESS first that would be useful. Yes, I will run with WITNESS and anything else you might want. Is there anything else you, or anyone else, wants from this kernel ? It may take another day to lock up when I've restarted it unfortunately. If you do INVARIANTS + WITNESS + WITNESS_SKIPSPIN, that should be good. WITNESS does a number of things, including tracking (and being judgemental about) lock order. One nice side effect of that tracking is that we keep track of a lot more lock state explicitly, so DDB's "show allocks", "show locks", etc, commands can build on that. "show lockedvnods" works without WITNESS, though, so your results so far suggest this is likely not related to vnode locking. Robert N M Watson Computer Laboratory University of Cambridge ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: Big problems with 7.1 locking up :-(
On Fri, 16 Jan 2009, Pete French wrote: hi, please type: show lock 0xff0001254d20 and then show thread 0xXXX where X is 'owner' of previous output. http://toybox.twisted.org.uk/~pete/71_pdns_lock.png That's in Power DNS - which is interesting because the one difference between the boxes that lock and those which dont is that the locking ones are serving DNS. I rather feared as much. Let's run down the path of "perhaps there's a problem with the new UDP locking code" for a bit and see where it takes us. Is it possible to run those boxes with WITNESS -- I believe that the fact that "show alllocks" is failing is because WITNESS isn't present. The other thing we can do is revert UDP to using purely write locks -- the risk there is that it might change the timing but not actually resolve the bug, so if we can analyze it a bit using WITNESS first that would be useful. Robert N M Watson Computer Laboratory University of Cambridge ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: Big problems with 7.1 locking up :-(
On Thu, 15 Jan 2009, Pete French wrote: desirable. You might want to give the NMI a test run just to make sure it behaves as you think it should, though -- be aware that if DDB/KDB aren't compiled into the kernel, then an NMI will panic the box. Unfortunately it does this... http://toybox.twisted.org.uk/~pete/71_nmi1.png That is locked up too - hitting return does nothing. I was hoping it was just garbled output but had actually gone to the debugger. Apparently not. Thats with a config file containing KDB, DDB and BREAK_TO_DEBUGGER, which does work as I have tested it with CTRL_ALT_ESC. Er, that's rather upsetting. John, do you have any ideas about this? Robert N M Watson Computer Laboratory University of Cambridge ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: Big problems with 7.1 locking up :-(
On Thu, 15 Jan 2009, Pete French wrote: In any case, if it starts to reproduceably recur, send out mail and we can see if we can track it down some more. BTW, did you establish if the version of iLo you have has a remote NMI? I seem to recall that some do, and being able to deliver an NMI is really quite valuable. OK, thanks. My iiLO2 appears to have the ability to generate an NMI oon demand, so that could be used if/whhen the fault crops up again. thanks, will let this lie for now and resurrect the thread when I can get some more useful data. Excellent WRT NMI. As long as you have DDB, KDB, and BREAK_TO_DEBUGGER compiled into the kernel, generating that should reliably get you into the debugger. If it's possible to keep running with INVARIANTS and WITNESS, or just INVARIANTS if WITNESS slows things down too much, that would be desirable. You might want to give the NMI a test run just to make sure it behaves as you think it should, though -- be aware that if DDB/KDB aren't compiled into the kernel, then an NMI will panic the box. Robert N M Watson Computer Laboratory University of Cambridge ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: Big problems with 7.1 locking up :-(
On Thu, 15 Jan 2009, Pete French wrote: Just an update on this - I tried the various kernels, but now the machine is not locking up at all. As I havent actually chnaged anything then this does not make me as happy as you might expect. I don;t know what to do now - I daare not upgrade the machines to an OS that I know locks, but if I cant make it lock then it is impossible to get any useful debugging info out of. maybe waiting for 7.2 is the best move... Well, one slightly pessimistic (or realistic) view says that all software contains bugs, it's just a question of whether or not your workload and environment trigger those bugs in a noticeable way. Given the inconsistency of the symptoms, I wouldn't preclude something environmental: could it be that it was the bottom, or more likely, top box in a rack and that your air conditioning isn't quite as effective there when the outside temperature is above/below some threshold? Alternatively, could it be that the workload changed very slightly -- you're doing less DNS queries, or the network latency to the DNS server changed? Certainly, whoever gave the advise on checking BIOS revisions is right: you can spend a lot of time tracking down a bug to realize that one box has a slightly different BIOS rev and therefore does/doesn't suffer from an obscure SMI bug. In any case, if it starts to reproduceably recur, send out mail and we can see if we can track it down some more. BTW, did you establish if the version of iLo you have has a remote NMI? I seem to recall that some do, and being able to deliver an NMI is really quite valuable. Robert N M Watson Computer Laboratory University of Cambridge ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: Big problems with 7.1 locking up :-(
On Wed, 14 Jan 2009, Pete French wrote: If you have BREAK_TO_DEBUGGER compiled into the kernel, then try pressing ctrl-alt-break on the console to see if you can drop into the debugger, or issue a serial break on a serial console. Well, I added BREAK_TO_DEBUGGER to the kernel config I had which contained all the other stuff (WITNESS etc...). The end result... ...it no longer crashes :-( I am not sure what to make of that! Wat could adding this to the kernel possibly do which would make my problems go away ? Should I try just adding this option to my GENERIC kernel and seeing if that also gives me something stable ? Yeah, that is unexpected -- the BREAK_TO_DEBUGGER path should have almost know effect on control flow, unlike, say, WITNESS, which significantly distorts timing. Is there any chance you picked up any of the recent fixes that went into RELENG_7 without noticing, and that perhaps one of those did it? With regard to what to do: if you didn't pick up a fix without noticing, yeah, I think it's worth testing the hypothesis that BREAK_TO_DEBUGGER fixed (or at least, masked) the problem. Generally with this sort of testing one has to be pretty rigorous in testing assumptions, because it's easy for changes to sneak in. Particularly annoying are seemingly innocuous code changes that do things like slightly rearrange kernel memory. FWIW, I suspect the various reports we are seeing reflect more than one problem, and that they must be relatively edge-case individually but reports of a few problems have lead to more "coming out of the woodwork". Obviously, the problems are not edge-case to the people experiencing them... Robert N M Watson Computer Laboratory University of Cambridge ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: Big problems with 7.1 locking up :-(
On Tue, 13 Jan 2009, Pete French wrote: I can't (fortunately) make it lock up. I have a DL360 G5 which is unused atm. and can test on it if needed. Would it be possible to install that under amd64 and hammer it with DNS requests ? I have been trying to think what the difference might be between my webservers and the machines which are freezing, and the opnly one I an come up with is UDP traffic as the locking machines are serving DNS and also NFS. There are significant changes in UDP locking between 7.0 and 7.1, so it could be that we're looking at a regression there. If you're able to reproduce this reliably, it might well be worth doing a little search-and-replace in udp_usrreq.c along the following lines: INP_RLOCK_ASSERT -> INP_WLOCK_ASSERT INP_RLOCK -> INP_WLOCK INP_RUNLOCK -> INP_WUNLOCK However, before making these changes for debugging purposes, make sure it's 100% reproduceable without them in the configuration so that we don't find ourselves barking up the wrong tree. Normally deadlocks along these lines *do* allow breaking into the debugger from a serial console, but since there are significant changes here in 7.1 it is worth trying to see if this might be related. Robert N M Watson Computer Laboratory University of Cambridge ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: Big problems with 7.1 locking up :-(
On Tue, 13 Jan 2009, Pete French wrote: Features like WITNESS and INVARIANTS may change the timing of the kernel making certain race conditions less likely; I'd run with them for a bit and see if you can reproduce the hang with them present, as they will make debugging the problem a lot easier, if it's possible. Uh, the above *was* me reproducing the hang with them present ;-)) It quite happily hangs with thoise things in the kernel - indeed the next hang was immediately after I rebooted the machine. But even with WITNESS and INVARIANTS and all the rest it does not drop to a debugger, it simply locks up. That machine is currently turned off, but still has 7.1 installed. What would you like me to try now ? I have a lockup I can reproduce pretty reliably now (just wait and it will always lock up). I also found that my other 7.1 box locks up fairly reliably when doing a buildworld. The only similarily between these two machines and the ones which dont lock up is that these are serving DNS. The others don't. Note that all the hardware is identical, as is the installed software and the configuration. If you have BREAK_TO_DEBUGGER compiled into the kernel, then try pressing ctrl-alt-break on the console to see if you can drop into the debugger, or issue a serial break on a serial console. For somewhat complicated reasons to explain, serial breaks are more effective at getting into the debugger, so are preferable -- also because you can more easily log output from the debugger. If you are able to get into the debugger, the normal commands would be most helpful, especially if you can log the results: ps show lockedvnods show alllocks Robert N M Watson Computer Laboratory University of Cambridge ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: Big problems with 7.1 locking up :-(
On Mon, 12 Jan 2009, Garance A Drosihn wrote: He is not eager to do a whole lot of experiments to track down the problem, since this is happening on busy production machines and he can't afford to have a lot of downtime on them (especially now that the semester at RPI has started up). The systems have some large (2 TB) filesystems on them, and the lockups occur in high disk-I/O situations. He's seeing the problem on one system which is a dual CPU quad-core xeon, and another which is a 64 bit P4 with hyperthreading. The one thing in common between the two setups is that the boot drives + a 3ware controller (with its array of RAID disks) is moved from one machine to the other one: I think playing the combinatorics game on compile-time flags, kernel features, etc, is probably not the best way to go about debugging this. Instead, I'd debug this as a kernel hang by breaking into the debugger once it occurs, if possible, and ideally on a serial console. Often times hangs can be debugged looking solely at DDB output, or if possible, a crash dump. Robert N M Watson Computer Laboratory University of Cambridge ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: Big problems with 7.1 locking up :-(
On Mon, 12 Jan 2009, Pete French wrote: I'm not sure if you've done this already, but the normal suggestions apply: have you compiled with INVARIANTS/WITNESS/DDB/KDB/BREAK_TO_DEBUGGER, and do any results / panics / etc result? Sometimes these debugging tools are able to convert hangs into panics, which gives us much more ability to debug them. OK, I have now had a machine hand again, with the correct debug options in the kernel. The screen looked like this when I went to restart it: http://toybox.twisted.org.uk/~pete/71_lor2.png It had not, however, dropped into any kind of debugger. Also there appear to me console messages after the lock order reversal - is that normal ? Lock order reversals are warnings of potential deadlock due to a lock cycle, but deadlocks may not actually result, either because it's a false positive (some locking construct that is deadlock free but involves lock cycles), or because a cycle didn't actually form. The message is suggestive, but if you have significant system activity after the message, then it may be unrelated. The machine did stay up for a signifanct amount of time before doing this. I notice that it is more or less identical to the one I posted whenI had WITNESS_KDB in the kernel too, so maybe those results arent entirely suprious after all ? Given it hasnt dropped to a debugger, is there anything else I can try ? Features like WITNESS and INVARIANTS may change the timing of the kernel making certain race conditions less likely; I'd run with them for a bit and see if you can reproduce the hang with them present, as they will make debugging the problem a lot easier, if it's possible. Robert N M Watson Computer Laboratory University of Cambridge ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: Big problems with 7.1 locking up :-(
On Mon, 12 Jan 2009, Tomas Randa wrote: I have similar problems. The last "good" kernel I have from stable brach, october the 8. Then in next upgrade, I saw big problems with performance. I tried ULE, 4BSD etc, but nothing helps, only downgrading system back. Now I am trying 7.1-p1 and problems are here again. Mysql is waiting a lot of time with status "waiting for opening table" or "waiting for close tables" I have 32bit FreeBSD with PAE, 1x xeon 5420, supermicro motherboard, areca SATA controller. Could not be problem in "da" device for example? So far, this sounds like a different problem than the one others have been posting about, which involves full system freezes rather than specific processes wedging or responding poorly. I'd suggest starting by using "procstat -k" on the process ID to look at where specific threads are waiting in the kernel. Is it simply that MySQL is being unreasonably slow in certain situations, or does it actually entirely stop operating? If you're able to narrow down the date on the 7.x branch where the problem you're experiencing "begins", that would be most helpful. I'd suggest leaving your userspace on the 8th october, and sliding the kernel forward in a binary search until you've narrowed it down a bit. Obviously, this takes a bit of patience, but narrowing it down could be quite informative. Robert N M Watson Computer Laboratory University of Cambridge Thanks Tomas Randa Garance A Drosihn wrote: At 2:55 PM + 1/12/09, Robert Watson wrote: On Fri, 9 Jan 2009, Garance A Drosihn wrote: At 2:39 PM -0500 1/9/09, Robert Blayzor wrote: On Jan 8, 2009, at 8:58 PM, Pete French wrote: I have a number of HP 1U servers, all of which were running 7.0 perfectly happily. I have been testing 7.1 in it's various incarnations for the last couple of months on our test server and it has performed perfectly. I noticed a problem with 7.0 on a couple of Dell servers. [...] We've since then compiled the kernel under the BSD scheduler to rule that out, and so far so good. Since ULE is now default in 7.1 and not in 7.0, perhaps you can try that? FWIW, the other guy I know who is having this problem had already switched to using ULE under 7.0-release, and did not have any problems with it. So *his* problem was probably not related to SCHED_ULE, unless something has recently changed there. Turns out he hasn't reverted back to 7.0-release just yet, so he's going to try SCHED_4BSD and see if that helps his situation. Scheduler changes always come with some risk of exposing bugs that have existed in the code for a long time but never really manifested themselves. ULE is well shaken-out, having been under development for at least five years, but it is possible that some problems will become visible as a result of the switch. I would encourage people to stick with ULE, but if you're having a stability problem then experimenting with scheduler as a variable that could be triggering the problem may well be useful to help track down the bug. Just to followup on this: My friend did switch back to a 7.1 kernel with SCHED_4BSD, and he still ran into problems. The error messages weren't the same, but errors did happen in the same high disk-I/O situations as the lockup happened with SCHED_ULE. At this point he's fallen back to the 7.0-kernel that he had been running (which also has SCHED_ULE), and all the problems have gone away. So at the moment he's running with a 7.0-ish kernel and the 7.1-release userland, without the hanging problems. So the problem is something in the kernel, but it is *NOT* the scheduler (at least, not in his case). He is not eager to do a whole lot of experiments to track down the problem, since this is happening on busy production machines and he can't afford to have a lot of downtime on them (especially now that the semester at RPI has started up). The systems have some large (2 TB) filesystems on them, and the lockups occur in high disk-I/O situations. He's seeing the problem on one system which is a dual CPU quad-core xeon, and another which is a 64 bit P4 with hyperthreading. The one thing in common between the two setups is that the boot drives + a 3ware controller (with its array of RAID disks) is moved from one machine to the other one: "its a 3ware 9500 12 port model, the boot drive is connected to an ICH6 in IDE mode, and yes, I've run it in single, single with hyper threading, and 8 way mode. All 64 bit." We still have no idea where the problem really is. For all we know, someone spilled a Pepsi on it when he wasn't looking... ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: Big problems with 7.1 locking up :-(
On Sat, 10 Jan 2009, Pete French wrote: FWIW, the other guy I know who is having this problem had already switched to using ULE under 7.0-release, and did not have any problems with it. So *his* problem was probably not related to SCHED_ULE, unless something has recently changed there. Well, one of my machines just locked up again, even with SCHED_4BSD on it, so I am now thinking it is unrelated. The machine has completely locked - no response to pings, no response to keypresses, nor to the power button. There is nothing printed on the console - it is just sitting there with a login prompt :-( This is really not good - these are extremely common servers after all, and I am just running bog standard 7.1 with apache and mysql. This is happening across several different servers, all of which are slight variants on the DL360, so I dont think it is something perculiar to me. I'm not sure if you've done this already, but the normal suggestions apply: have you compiled with INVARIANTS/WITNESS/DDB/KDB/BREAK_TO_DEBUGGER, and do any results / panics / etc result? Sometimes these debugging tools are able to convert hangs into panics, which gives us much more ability to debug them. If it still hangs rather than panicking, are you able to break into the debugger on the console? If you're using a video console and not able to get to the debugger, would it be possible to configure a serial console and use that -- serial breaks are often more successful at getting to the debugger than keyboard breaks. Likewise, I'm not sure if this hardware has an NMI button -- some HP servers have one on the motherboard that you can press -- but that is also potentially a way to get into the debugger the analyze the crash. Robert N M Watson Computer Laboratory University of Cambridge ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: Big problems with 7.1 locking up :-(
On Fri, 9 Jan 2009, Garance A Drosihn wrote: At 2:39 PM -0500 1/9/09, Robert Blayzor wrote: On Jan 8, 2009, at 8:58 PM, Pete French wrote: I have a number of HP 1U servers, all of which were running 7.0 perfectly happily. I have been testing 7.1 in it's various incarnations for the last couple of months on our test server and it has performed perfectly. I noticed a problem with 7.0 on a couple of Dell servers. [...] We've since then compiled the kernel under the BSD scheduler to rule that out, and so far so good. Since ULE is now default in 7.1 and not in 7.0, perhaps you can try that? FWIW, the other guy I know who is having this problem had already switched to using ULE under 7.0-release, and did not have any problems with it. So *his* problem was probably not related to SCHED_ULE, unless something has recently changed there. Turns out he hasn't reverted back to 7.0-release just yet, so he's going to try SCHED_4BSD and see if that helps his situation. Scheduler changes always come with some risk of exposing bugs that have existed in the code for a long time but never really manifested themselves. ULE is well shaken-out, having been under development for at least five years, but it is possible that some problems will become visible as a result of the switch. I would encourage people to stick with ULE, but if you're having a stability problem then experimenting with scheduler as a variable that could be triggering the problem may well be useful to help track down the bug. Most of the time the bugs will not be in ULE itself, rather, triggered because ULE will change the ordering or balancing of work in the system, so we should try to avoid situations where people switch to 4BSD from ULE and stick with it rather than getting the underlying problem fixed! Robert N M Watson Computer Laboratory University of Cambridge ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: Panic in RELENG_7_1 with fxp(4)
On Tue, 6 Jan 2009, Brandon Weisz wrote: http://people.freebsd.org/~yongari/fxp/if_fxp.c http://people.freebsd.org/~yongari/fxp/if_fxpreg.h http://people.freebsd.org/~yongari/fxp/if_fxpvar.h With this version, the system still panics as before. After the system panic with this patch, I went into the bios and disabled all unnecessary hardware such as parallel port, usb controller and on-board audio. The resulting panic below appears different. Without contributing anything too constructive to this conversation, this crash looks a bit like it might be a device driver bug in which a packet is passed to if_input() from the device driver, but then modified by the device driver after it is handed off. Robert N M Watson Computer Laboratory University of Cambridge Fatal trap 12: page fault while in kernel mode cpuid = 0; apic id = 00 fault virtual address = 0x400 fault code = supervisor read, page not present instruction pointer = 0x20:0xc07eefec stack pointer = 0x28:0xe4339ac0 frame pointer = 0x28:0xe4339ae4 code segment= base 0x0, limit 0xf, type 0x1b = DPL 0, pres 1, def32 1, gran 1 processor eflags= interrupt enabled, resume, IOPL = 0 current process = 28 (irq23: vr0) trap number = 12 panic: page fault cpuid = 0 Uptime: 50s Physical memory: 995 MB Dumping 162 MB: 147 131 115 99 83 67 51 35 19 3 (kgdb) list *0xc07eefec 0xc07eefec is in sbappendaddr_locked (/usr/src.local/sys/kern/uipc_sockbuf.c:652). 647 if (n) 648 n->m_next = m0; /* concatenate data to control */ 649 else 650 control = m0; 651 m->m_next = control; 652 for (n = m; n->m_next != NULL; n = n->m_next) 653 sballoc(sb, n); 654 sballoc(sb, n); 655 nlast = n; 656 SBLINKRECORD(sb, m); (kgdb) bt #0 doadump () at pcpu.h:196 #1 0xc079cf07 in boot (howto=260) at /usr/src.local/sys/kern/kern_shutdown.c:418 #2 0xc079d1d9 in panic (fmt=Variable "fmt" is not available. ) at /usr/src.local/sys/kern/kern_shutdown.c:574 #3 0xc0ab110c in trap_fatal (frame=0xe4339a80, eva=1024) at /usr/src.local/sys/i386/i386/trap.c:939 #4 0xc0ab1390 in trap_pfault (frame=0xe4339a80, usermode=0, eva=1024) at /usr/src.local/sys/i386/i386/trap.c:852 #5 0xc0ab1d4c in trap (frame=0xe4339a80) at /usr/src.local/sys/i386/i386/trap.c:530 #6 0xc0a97bbb in calltrap () at /usr/src.local/sys/i386/i386/exception.s:159 #7 0xc07eefec in sbappendaddr_locked (sb=0xc5bacd50, asa=0xe4339b90, m0=0xc4167000, control=0xc4167000) at /usr/src.local/sys/kern/uipc_sockbuf.c:652 #8 0xc08ecb71 in udp_append (inp=Variable "inp" is not available. ) at /usr/src.local/sys/netinet/udp_usrreq.c:254 #9 0xc08edf7a in udp_input (m=0xc4167000, off=20) at /usr/src.local/sys/netinet/udp_usrreq.c:567 #10 0xc087d320 in ip_input (m=0xc4167000) at /usr/src.local/sys/netinet/ip_input.c:665 #11 0xc0842a85 in netisr_dispatch (num=2, m=0xc4167000) at /usr/src.local/sys/net/netisr.c:185 #12 0xc08389f1 in ether_demux (ifp=0xc41b6800, m=0xc4167000) at /usr/src.local/sys/net/if_ethersubr.c:834 #13 0xc0838de3 in ether_input (ifp=0xc41b6800, m=0xc4167000) at /usr/src.local/sys/net/if_ethersubr.c:692 #14 0xc071750b in vr_intr (arg=0xc41c8000) at /usr/src.local/sys/dev/vr/if_vr.c:1415 #15 0xc077bf0b in ithread_loop (arg=0xc41c5550) at /usr/src.local/sys/kern/kern_intr.c:1088 #16 0xc0778a79 in fork_exit (callout=0xc077bd50 , arg=0xc41c5550, frame=0xe4339d38) at /usr/src.local/sys/kern/kern_fork.c:804 #17 0xc0a97c30 in fork_trampoline () at /usr/src.local/sys/i386/i386/exception.s:264 (kgdb) ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org" ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: TCP packet out-of-order problem
On Tue, 6 Jan 2009, Lin Jui-Nan Eric wrote: I thought that the system auto-tune improperly in this case. Hmm. Do you have a custom setting for kern.ipc.nmbclusters in loader.conf or sysctl.conf? What does kern.ipc.nmbclusters configure itself to on your system? Also, could you send me the output of uname -a on the system? Thanks, Robert N M Watson Computer Laboratory University of Cambridge On Mon, Jan 5, 2009 at 9:13 PM, Robert Watson wrote: On Fri, 2 Jan 2009, Lin Jui-Nan Eric wrote: After running "netstat -s -p tcp", we found that lots of packets are discarded due to memory problems. We googled for it, and found that sysctl oid "net.inet.tcp.reass.maxsegments" became 0, therefore packets never reassembled. Then we checked our /boot/loader.conf and /etc/sysctl.conf, and found that setting kern.ipc.nmbclusters="0" makes net.inet.tcp.reass.maxsegments=0. After setting net.inet.tcp.reass.maxsegments="1600" in /boot/loader.conf, the network works perfectly now. Was it set to 0 through a configuration error, or did the system auto-tune improperly? Robert N M Watson Computer Laboratory University of Cambridge Thank you all for the help! ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org" ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: rdump stuck in sbwait state (RELENG_7)
On Mon, 5 Jan 2009, Terry Kennedy wrote: I may have missed this earlier in the thread, but I don't see a kernel stack trace of the stuck thread/process. Could you grab one using procstat -k, DDB, or KGDB? I'd like to confirm that the 'sbwait' really reflects waiting to send, rather than waiting to receive, which (for better or worse) uses the same wmesg. procstat -k may be the simplest of the above to do if your system is reasonable recent. I didn't post that earlier as no-one had asked for it 8-) Indeed :-). The system is current as of December 29th. Here's the relevant info: Could I ask you to also send me procstat -f output? More below the quote. (0:10) test4:/sysprog/terry# uname -a FreeBSD test4.tmk.com 7.1-PRERELEASE FreeBSD 7.1-PRERELEASE #0: Mon Dec 29 11:48:04 EST 2008 te...@test4.tmk.com:/usr/obj/usr/src/sys/PE1550 i386 (0:11) test4:/sysprog/terry# ps -axwww | grep dump UID PID PPID CPU PRI NI VSZ RSS MWCHAN STAT TT TIME COMMAND 0 4436 4411 0 8 0 35896 34552 wait I+p10:00.70 /sbin/rdump 0uLa -b 64 -C 32 -f server /usr (rdump) 0 4439 4436 0 4 0 35896 34784 sbwait I+p10:03.05 rdump: /dev/amrd0s1f: pass 4: 18.48% done, finished in 0:17 at Sat Jan 3 21:02:05 2009 (rdump) 0 4440 4439 0 20 0 35896 34624 pause I+p10:05.26 /sbin/rdump 0uLa -b 64 -C 32 -f server /usr (rdump) 0 4441 4439 0 20 0 35896 34624 pause I+p10:05.26 /sbin/rdump 0uLa -b 64 -C 32 -f server /usr (rdump) 0 4442 4439 0 4 0 35896 34624 sbwait I+p10:05.26 /sbin/rdump 0uLa -b 64 -C 32 -f server /usr (rdump) (0:12) test4:/sysprog/terry# procstat -k 4436 PIDTID COMM TDNAME KSTACK 4436 100115 rdump-mi_switch sleepq_switch sleepq_catch_signals sleepq_wait_sig _sleep kern_wait wait4 syscall Xint0x80_syscall (0:13) test4:/sysprog/terry# procstat -k 4439 PIDTID COMM TDNAME KSTACK 4439 100127 rdump-mi_switch sleepq_switch sleepq_catch_signals sleepq_wait_sig _sleep sbwait soreceive_generic soreceive soo_read dofileread kern_readv read syscall Xint0x80_syscall (0:14) test4:/sysprog/terry# procstat -k 4440 PIDTID COMM TDNAME KSTACK 4440 100131 rdump-mi_switch sleepq_switch sleepq_catch_signals sleepq_wait_sig _sleep kern_sigsuspend sigsuspend syscall Xint0x80_syscall (0:15) test4:/sysprog/terry# procstat -k 4441 PIDTID COMM TDNAME KSTACK 4441 100105 rdump-mi_switch sleepq_switch sleepq_catch_signals sleepq_wait_sig _sleep kern_sigsuspend sigsuspend syscall Xint0x80_syscall (0:16) test4:/sysprog/terry# procstat -k 4442 PIDTID COMM TDNAME KSTACK 4442 100135 rdump-mi_switch sleepq_switch sleepq_catch_signals sleepq_wait_sig _sleep sbwait soreceive_generic soreceive soo_read dofileread kern_readv read syscall Xint0x80_syscall As I understand it, the processes in sbwait state are waiting to receive. That would seem to indicate that they don't see the ACKs from the other end, despite the tcpdump showing that they were received. In general, being blocked in soreceive() means that the application at the other end hasn't sent data, or the other end hasn't received or correctly processed ACKs from the local end, so isn't sending more data that it has queued up. The condition you describe sounds more like what would happen in a sender: that it has data to send, but the remote side hasn't ACK'd sufficiently to send it all. If you have kgdb handy, it would be useful to look at *so and *so->so_domain in the soreceive_generic frame of proc 4439. If it's an inet socket, we'd like to see *(struct inpcb *)so->so_pcb, and if it's a TCP socket, *(struct tcpcb *)((struct inpcb *)so->so_pcb)->inp_ppcb. Robert N M Watson Computer Laboratory University of Cambridge ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: rdump stuck in sbwait state (RELENG_7)
On Sat, 3 Jan 2009, Terry Kennedy wrote:
Sorry, I can't think of any - by the time you see it hung, whatever went
wrong has already happened. You might glean some insight from the TCP
socket state (on the FreeBSD side, use 'netstat -A' to print the PCB
address and gdb to dump the contents but I'm not sure how to get this data
out of OpenVMS). The '-C' and '-W' options to tcpdump will help.
Ok, I found some time to reproduce this while capturing a trace with
tcpdump.
Here's the relevant output from netstat / kgdb:
I may have missed this earlier in the thread, but I don't see a kernel stack
trace of the stuck thread/process. Could you grab one using procstat -k, DDB,
or KGDB? I'd like to confirm that the 'sbwait' really reflects waiting to
send, rather than waiting to receive, which (for better or worse) uses the
same wmesg. procstat -k may be the simplest of the above to do if your system
is reasonable recent.
Robert N M Watson
Computer Laboratory
University of Cambridge
(0:31) test4:~terry# netstat -A
Active Internet connections
TcpcbProto Recv-Q Send-Q Local Address Foreign Address(state)
c73eeae0 tcp4 0 0 test4.892 server.shell ESTABLISHED
[snip]
(0:32) test4:~terry# kgdb
GNU gdb 6.1.1 [FreeBSD]
[snip]
#0 0x in ?? ()
(kgdb) print * (struct tcpcb *) 0xc73eeae0
$1 = {t_segq = {lh_first = 0x0}, t_segqlen = 0, t_dupacks = 0,
t_timers = 0xc73eec24, t_inpcb = 0xc7387708, t_state = 4, t_flags = 484,
snd_una = 292841209, snd_max = 292841209, snd_nxt = 292841209,
snd_up = 292780017, snd_wl1 = 3606352422, snd_wl2 = 292841209,
iss = 3955646224, irs = 3606284909, rcv_nxt = 3606352422,
rcv_adv = 3606415910, rcv_wnd = 63488, rcv_up = 3606352422, snd_wnd = 65535,
snd_cwnd = 65535, snd_bwnd = 1073725440, snd_ssthresh = 1073725440,
snd_bandwidth = 0, snd_recover = 3955646224, t_maxopd = 1460,
t_rcvtime = 11273919, t_starttime = 11024967, t_rtttime = 0,
t_rtseq = 292839154, t_bw_rtttime = 11024966, t_bw_rtseq = 3955646224,
t_rxtcur = 230, t_maxseg = 1448, t_srtt = 145, t_rttvar = 34,
t_rxtshift = 0, t_rttmin = 30, t_rttbest = 67, t_rttupdated = 232101,
max_sndwnd = 65535, t_softerror = 0, t_oobflags = 0 '\0', t_iobc = 0 '\0',
snd_scale = 0 '\0', rcv_scale = 3 '\003', request_r_scale = 3 '\003',
ts_recent = 1207233, ts_recent_age = 11273919, ts_offset = 0,
last_ack_sent = 3606352422, snd_cwnd_prev = 0, snd_ssthresh_prev = 0,
snd_recover_prev = 0, t_badrxtwin = 0, snd_limited = 0 '\0',
snd_numholes = 0, snd_holes = {tqh_first = 0x0, tqh_last = 0xc73eebb8},
snd_fack = 0, rcv_numsacks = 0, sackblks = {{start = 0, end = 0}, {
start = 0, end = 0}, {start = 0, end = 0}, {start = 0, end = 0}, {
start = 0, end = 0}, {start = 0, end = 0}}, sack_newdata = 0,
sackhint = {nexthole = 0x0, sack_bytes_rexmit = 0}, t_rttlow = 1,
rfbuf_ts = 0, rfbuf_cnt = 0, t_pspare = {0x0, 0x0, 0x0}, t_tu = 0x0,
t_toe = 0x0}
(kgdb) q
Rather than pasting the decoded tcpdump output here, the raw capture
file is at http://www.tmk.com/transient/rdump30.gz (it is only 76KB
compressed, 270KB uncompressed). It looks to me like the remote host
(the VMS box) has correctly ack'd all outstanding data from the FreeBSD
host, but that the FreeBSD host is just sitting there for some reason.
As before, I have this sitting in the wedged state so if anyone needs
more data, I can either collect it or give you access to the system.
Terry Kennedy http://www.tmk.com
te...@tmk.com New York, NY USA
___
freebsd-stable@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
___
freebsd-stable@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: TCP packet out-of-order problem
On Fri, 2 Jan 2009, Lin Jui-Nan Eric wrote: After running "netstat -s -p tcp", we found that lots of packets are discarded due to memory problems. We googled for it, and found that sysctl oid "net.inet.tcp.reass.maxsegments" became 0, therefore packets never reassembled. Then we checked our /boot/loader.conf and /etc/sysctl.conf, and found that setting kern.ipc.nmbclusters="0" makes net.inet.tcp.reass.maxsegments=0. After setting net.inet.tcp.reass.maxsegments="1600" in /boot/loader.conf, the network works perfectly now. Was it set to 0 through a configuration error, or did the system auto-tune improperly? Robert N M Watson Computer Laboratory University of Cambridge Thank you all for the help! ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org" ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: MFC ZFS: when?
On Fri, 21 Nov 2008, Zaphod Beeblebrox wrote: In several of the recent ZFS posts, multiple people have asked when this will be MFC'd to 7.x. This query has been studiously ignored as other chatter about whatever ZFS issue is discussed. Presumably the MFC schedule is largely up to Pawel, who did the work. However, Pawel was on travel last weekend and week attending MeetBSD and the FreeBSD developer summit in the bay area, and hasn't been seen on stable@ since the 17th. I think it's likely not so much that anyone is being studiously ignored, it's that the person who can best answer he question hasn't been keeping up with the list for a bit. Robert N M Watson Computer Laboratory University of Cambridge So in a post with no other bug report or discussion content to distract us, when is it intended that ZFS be MFC'd to 7.x? ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "[EMAIL PROTECTED]" ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: Install issues with 7.x
On Wed, 29 Oct 2008, Ryan wrote: Hello, I purchased a new Clevo M860TU on the account that it ran linux very well and was hoping it would fair the same on FreeBSD. Not so much, little help? I posted this in mobile originally but though stable would be a better choice. Don't know if it is more appropriate here or ACPI. I'm giving you as much information as I know how to get. as I cannot get sysinstall to load I am having to type all these dmesg. The boot process is hanging. This is all with 7.x, I can give 6.x if needed. xpt_config is the CAM configuration wait, so basically the system is waiting for a storage device to report back on whether it could be used as a root file system. I recently saw a similar report of problems involving a firewire controller on an nvidia motherboard following an upgrade to 7.x, and I wonder if you might try the following: see if 6.4 will install, and if so, install it. Then cvsup 7.x, and do a buildworld but not an installworld. This will let you build and experiment with 7.x kernels from a known-working environment. Make sure to keep a working 6.x kernel around -- I suggest something like "cp -r /boot/kernel /boot/kernel.good" before starting so you can always fall back to a good kernel. Now try building a 7.x kernel without USB or firewire support, and booting that? Also, it's worth checking there are no BIOS upgrades available for the motherboard... Robert N M Watson Computer Laboratory University of Cambridge Hardware: Intel P9500 4gb DDR3-1066 Nvidia 9800M GT Atheros AR5006e FreeBSD 7.1-BETA2 These snippets of dmesg happen around the end where it hangs. 1. Default ... cpu0: on acpi0 ACPI Error (dsopcode-0350): No pointer back to NS node in buffer obj 0xc6a02d40 [20070320] ACPI Exception (dswexec-0556): AE_AML_INTERNAL, While resolving operands for [OpcodeName unavailable] [20070320] ACPI Error (psparse-0626): Method parse/execution failed [\_PR_.CPU0._OSC] (Node 0xc68556e0), AE_AML_INTERNAL est0: on cpu0 p4tcc0: on cpu0 cpu1: on acpi0 ACPI Error (dsopcode-0350): No pointer back to NS node in buffer obj 0xc6a0e300 [20070320] ACPI Exception (dswexec-0556): AE_AML_INTERNAL, While resolving operands for [OpcodeName unavailable] [20070320] ACPI Error (psparse-0626): Method parse/execution failed [\_PR_.CPU1._OSC] (Node 0xc685560), AE_AML_INTERNAL est1: on cpu1 p4tcc1: on cpu1 ... cpu0: Cx states changed cpu1: Cx states changed unknown: timeout waiting for read DRQ unknown: timeout waiting for read DRQ acd0: DVDR at ata3-master UDMA33 GEOM_LABEL: Label for provider acd0 is iso9660/FreeBSD_Install run_interrupt_driven_hooks: still waiting after 60 seconds for xpt_config run_interrupt_driven_hooks: still waiting after 120 seconds for xpt_config run_interrupt_driven_hooks: still waiting after 180 seconds for xpt_config run_interrupt_driven_hooks: still waiting after 240 seconds for xpt_config run_interrupt_driven_hooks: still waiting after 300 seconds for xpt_config Then just stalls 2. No ACPI ... unknown: timeout waiting for read DRQ unknown: timeout waiting for read DRQ acd0: DVDR at ata3-master UDMA33 GEOM_LABEL: Label for provider acd0 is iso9660/FreeBSD_Install run_interrupt_driven_hooks: still waiting after 60 seconds for xpt_config run_interrupt_driven_hooks: still waiting after 120 seconds for xpt_config run_interrupt_driven_hooks: still waiting after 180 seconds for xpt_config run_interrupt_driven_hooks: still waiting after 240 seconds for xpt_config run_interrupt_driven_hooks: still waiting after 300 seconds for xpt_config Then just stalls 3. Safe Mode I can only tell you a little because console is spammed. It is the same as no ACPI, but with an interrupt storm. ... unknown: timeout waiting for read DRQ unknown: timeout waiting for read DRQ acd0: DVDR at ata3-master UDMA33 GEOM_LABEL: Label for provider acd0 is iso9660/FreeBSD_Install run_interrupt_driven_hooks: still waiting after 60 seconds for xpt_config run_interrupt_driven_hooks: still waiting after 120 seconds for xpt_config run_interrupt_driven_hooks: still waiting after 180 seconds for xpt_config run_interrupt_driven_hooks: still waiting after 240 seconds for xpt_config run_interrupt_driven_hooks: still waiting after 300 seconds for xpt_config When it gets to the unknowns, this is spammed. interrupt storm detected on "irq10:"; throttling interrupt source Other than the interrupt storm spam, it is halted like the others. 4. Single User Mode Same as 1, Default 5. Verbose All I can tell you is what is spammed at the end. acpi: bad write to port 0x080 (32), val hex Where hex is ever increasing and loops when it hits 0xff01. I can also see run_interrupt_driven_hooks message in all the spam. Using some googling if you add the sysctl before boot debug.acpi.block_bad_io=1 it might be of some help. This just leads to a never ending loop of acpi errors - the scroll very fast and difficult to record might I add! ... acpi: bad write to port 0x080 (32),
Re: 7.x and multiple IPs in jails
On Tue, 28 Oct 2008, Chris St Denis wrote: Serious question here (not trolling). These patches have been around for years, why have they never been committed to trunk/stable? Network stacks are incredibly complicated pieces of software, and some of the short-cuts jail took to accomplish its goals without implementing true virtualization break down when generalized. The patches to date have made the code more complicated and added more edge cases, and we'd like to avoid adding more edge cases to the stack if we can avoid it, as every edge case offers the opportunity for bugs and maintenance issues. The current patches Bjoern is preparing address most of these concerns, and they've been undergoing review and testing for a few months now. I'd like to think they will be in 8.x relatively shortly (next week or two), and in 7.x before 7.2. Robert N M Watson Computer Laboratory University of Cambridge ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: UDP LOR with the latest RELENG_7
On Fri, 10 Oct 2008, Robert Watson wrote: On Fri, 10 Oct 2008, Jeremy Chadwick wrote: I'll see whether the system still locks up or not though.. Okay, I'm bringing rwatson@ into the thread since this is specific to UDP. I've now fixed the bug leading to the lock order reversal; I'd be interested in knowing if it also corrects the stability issue. This was r183753 in svn; I'm not sure it's hit CVS/cvsup yet but should do in a few minutes. Dear Vlad: Could you confirm that with udp_usrreq.c:1.218.2.7 (or newer), the problem has gone away? CVS log excerpt below. Thanks, Robert N M Watson Computer Laboratory University of Cambridge revision 1.218.2.7 date: 2008/10/10 21:18:12; author: rwatson; state: Exp; lines: +8 -8 SVN rev 183753 on 2008-10-10 21:18:12Z by rwatson Test the soreceive_dgram tunable and set up the pr_usrreqs structure during UDP initialization, not during UDP inpcb initialization. Reported by:Vlad GALU Approved by:re (kib) ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: UDP LOR with the latest RELENG_7
On Fri, 10 Oct 2008, Jeremy Chadwick wrote: I'll see whether the system still locks up or not though.. Okay, I'm bringing rwatson@ into the thread since this is specific to UDP. I've now fixed the bug leading to the lock order reversal; I'd be interested in knowing if it also corrects the stability issue. This was r183753 in svn; I'm not sure it's hit CVS/cvsup yet but should do in a few minutes. Thanks, Robert N M Watson Computer Laboratory University of Cambridge ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: UDP LOR with the latest RELENG_7
On Fri, 10 Oct 2008, Jeremy Chadwick wrote: I'll see whether the system still locks up or not though.. Okay, I'm bringing rwatson@ into the thread since this is specific to UDP. Crumbs. It looks like the tunable fetch got dropped into the wrong function of udp_inpcb_init() and udp_init(), and this LOR is a side effect of that bug. I'll see about creating a patch this evening. Robert N M Watson Computer Laboratory University of Cambridge ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: stable 7.0 and nslookup help command
On Tue, 7 Oct 2008, Jeremy Chadwick wrote: Not to dissuade you from what you're trying to accomplish, but "nslookup" has been deprecated (this has been stated a few times by the BIND folks), and "host" is probably on its way out as well (though I remember somewhere, sometime, nslookup used to state "being deprecated, use 'host' or 'dig' instead" -- or something like that). Please learn to use the "dig" command. I realise it has a somewhat high learning curve at first (syntax-wise it can be somewhat messy), but ultimately it's an immensely powerful -- or simple! -- DNS tool. Ditto here -- dig requires a bit more understanding of DNS, but is actually a much more informative tool when it comes to querying DNS and debugging its quirks. Robert N M Watson Computer Laboratory University of Cambridge ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: Is FreeBSD a suitable choice for a MacBook? --- WHY?
On Mon, 6 Oct 2008, Dr. Aharon Friedman wrote: Sorry, I meant BSD. Here is the link: http://www.freebsd.org/news/press-rel-3.html Aharon Friedman I don't see the origina message you replied to on the list, so am replying to it via your post... I'm just a lurker, but even I know that only some of the userland apps in OS X are BSD-based. The kernel is mach microkernel based and not even slightly similar. This claim regarding the kernel is highly inaccurate. There are significant quantities of FreeBSD, Mach, and Apple-originated code in the Mac OS X kernel, both because Apple pulled in a lot of FreeBSD code early on, but also because code moves between the two kernels fairly easily and fairly frequently, and in both directions. You'll find a FreeBSD-derived VFS, network stack, and countless other kernel parts in Mac OS X from their first open source drop forward. More recently, though, you'll find that the Audit implementation present in FreeBSD 6.x and later is based on the Mac OS X kernel audit code, and the TrustedBSD MAC Framework that appeared in Mac OS X Leopard is straight from FreeBSD. It's certainly true that there's a lot of non-FreeBSD code -- XNU uses the Mach scheduler and Mach IPC, and a quite different driver framework, for example. There's also some convergent evolution: FreeBSD contains a Mach-derived VM that also comes from the original Mach project. Finally, just to be clear: XNU is not a micro-kernel, even though it contains significant amounts of Mach code. The "microkernel" and remainder of the kernel run in a single address space, and although certain separation is (often) maintained in the source code / abstractions, the Mach, FreeBSD, and device driver parts run in a unified and tightly integrated way. Robert N M Watson Computer Laboratory University of Cambridge ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: bad NFS/UDP performance
On Sat, 4 Oct 2008, Danny Braniss wrote: at the moment, the best I can do is run it on a different hardware that has if_em, the results are in ftp://ftp.cs.huji.ac.il/users/danny/lock.prof/7.1-1000.em the benchmark ran better with the Intel NIC, averaged UDP 54MB/s, TCP 53MB/s (I get the same numbers with an older kernel). Dear Danny: Unfortunately, I was left slightly unclear on the comparison you are making above. Could you confirm whether or not, with if_em, you see a performance regression using UDP NFS between 7.0-RELEASE and the most recent 7.1-STABLE, and if you do, whether or not the RLOCK->WLOCK change has any effect on performance? It would be nice to know on the same hardware but at least with different hardware we get a sense of whether or not this might affect other systems or whether it's limited to a narrower set of configurations. Thanks, Robert N M Watson Computer Laboratory University of Cambridge ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: bad NFS/UDP performance
On Fri, 3 Oct 2008, Danny Braniss wrote:
On Fri, 3 Oct 2008, Danny Braniss wrote:
gladly, but have no idea how to do LOCK_PROFILING, so some pointers would
be helpfull.
The LOCK_PROFILING(9) man page isn't a bad starting point -- I find that
the defaults work fine most of the time, so just use them. Turn the enable
syscl on just before you begin a run, and turn it off immediately
afterwards. Make sure to reset between reruns (rebooting to a new kernel
is fine too!).
in ftp://ftp.cs.huji.ac.il/users/danny/lock.prof
there 3 files:
7.1-100 host connected at 100 running -prerelease
7.1-1000same but connected at 1000
7.0-1000-stable with your 'patch'
at 100 my benchmark didn't suffer from the profiling, average was about 9.
at 1000 the benchmark got realy hit, average was around 12 for the patched,
and 4 for the unpatched (less than at 100).
Interesting. A bit of post-processing:
[EMAIL PROTECTED]:/tmp> cat 7.1-1000 | awk -F' ' '{print $3" "$9}' | sort -n |
tail -10
2413283 /r+d/7/sys/kern/kern_mutex.c:141
2470096 /r+d/7/sys/nfsclient/nfs_socket.c:1218
2676282 /r+d/7/sys/net/route.c:293
2754866 /r+d/7/sys/kern/vfs_bio.c:1468
3196298 /r+d/7/sys/nfsclient/nfs_bio.c:1664
3318742 /r+d/7/sys/net/route.c:1584
3711139 /r+d/7/sys/dev/bge/if_bge.c:3287
3753518 /r+d/7/sys/net/if_ethersubr.c:405
3961312 /r+d/7/sys/nfsclient/nfs_subs.c:1066
10688531 /r+d/7/sys/dev/bge/if_bge.c:3726
[EMAIL PROTECTED]:/tmp> cat 7.0-1000 | awk -F' ' '{print $3" "$9}' | sort -n |
tail -10
468631 /r+d/hunt/src/sys/nfsclient/nfs_nfsiod.c:286
501989 /r+d/hunt/src/sys/nfsclient/nfs_vnops.c:1148
631587 /r+d/hunt/src/sys/nfsclient/nfs_socket.c:1198
701155 /r+d/hunt/src/sys/nfsclient/nfs_socket.c:1258
718211 /r+d/hunt/src/sys/kern/kern_mutex.c:141
1118711 /r+d/hunt/src/sys/nfsclient/nfs_bio.c:1664
1169125 /r+d/hunt/src/sys/nfsclient/nfs_subs.c:1066
1222867 /r+d/hunt/src/sys/kern/vfs_bio.c:1468
3876072 /r+d/hunt/src/sys/netinet/udp_usrreq.c:545
5198927 /r+d/hunt/src/sys/netinet/udp_usrreq.c:864
The first set above is with the unmodified 7-STABLE tree, the second with a
reversion of read locking on the UDP inpcb. The big blinking sign of interest
is that the bge interface lock is massively contended in the first set of
output, and basically doesn't appear in the second. There are various reasons
bge could stand out quite so much -- one possibly is that previously, the udp
lock serialized all access to the interface from the send code, preventing the
send and receive paths from contending.
A few things to try:
- Let's look compare the context switch rates on the two benchmarks. Could
you run vmstat and look at the cpu cs line during the benchmarks and see how
similar the two are as the benchmarks run? You'll want to run it with
vmstat -w 1 and collect several samples per benchmark, since we're really
interested in the distribution rather than an individual sample.
- Is there any chance you could drop an if_em card into the same box and run
the identical benchmarks with and without LOCK_PROFILING to see whether it
behaves differently than bge when the patch is applied? if_em's interrupt
handling is quite different, and may significantly affect lock use, and
hence contention.
Thanks,
Robert N M Watson
Computer Laboratory
University of Cambridge
___
freebsd-stable@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: bad NFS/UDP performance
On Fri, 3 Oct 2008, Danny Braniss wrote: gladly, but have no idea how to do LOCK_PROFILING, so some pointers would be helpfull. The LOCK_PROFILING(9) man page isn't a bad starting point -- I find that the defaults work fine most of the time, so just use them. Turn the enable syscl on just before you begin a run, and turn it off immediately afterwards. Make sure to reset between reruns (rebooting to a new kernel is fine too!). as a side note, many years ago I checked out NFS/TCP and it was really bad, I even remember NetApp telling us to drop TCP, but now, things look rather better. Wonder what caused it. Well, the virtues of TCP become more apparent with higher network speeds, as the logic to fill pipes using TCP, manage flow control, etc, is a lot more sophisticated than what's in the RPC code for using UDP. The downsides to UDP are also becoming more apparent: as network speeds go up, fragmented UDP risks IP ID collisions which could lead to data corruption, or at the very least, dropped packets. We have changed the default for NFSv3 mounts to TCP in 8.x, and talked about doing it for 7.1; unfortunately the timing wasn't quite right, so it most likely will appear in 7.2. Robert N M Watson Computer Laboratory University of Cambridge ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: bad NFS/UDP performance
On Fri, 3 Oct 2008, Danny Braniss wrote:
OK, so it looks like this was almost certainly the rwlock change. What
happens if you pretty much universally substitute the following in
udp_usrreq.c:
Currently Change to
- -
INP_RLOCK INP_WLOCK
INP_RUNLOCK INP_WUNLOCK
INP_RLOCK_ASSERTINP_WLOCK_ASSERT
I guess you were almost certainly correct :-) I did the global subst. on the
udp_usrreq.c from 19/08, __FBSDID("$FreeBSD: src/sys/netinet/udp_usrreq.c,v
1.218.2.3 2008/08/18 23:00:41 bz Exp $"); and now udp is fine again!
OK. This is a change I'd rather not back out since it significantly improves
performance for many other UDP workloads, so we need to figure out why it's
hurting us so much here so that we know if there are reasonable alternatives.
Would it be possible for you to do a run of the workload with both kernels
using LOCK_PROFILING around the benchmark, and then we can compare lock
contention in the two cases? What we often find is that relieving contention
at one point causes new contention at another point, and if the primitive used
at that point handles contention less well for whatever reason, performance
can be reduced rather than improved. So maybe we're looking at an issue in
the dispatched UDP code from so_upcall? Another less satisfying (and
fundamentally more difficult) answer might be "something to do with the
scheduler", but a bit more analysis may shed some light.
Robert N M Watson
Computer Laboratory
University of Cambridge
danny
Robert N M Watson
Computer Laboratory
University of Cambridge
server is a NetApp:
kernel from 18/08/08 00:00:0 :
/- UDP // TCP ---/
1*512 38528 0.19s 83.50MB 0.20s 80.82MB/s
2*512 19264 0.21s 76.83MB 0.21s 77.57MB/s
4*512 9632 0.19s 85.51MB 0.22s 73.13MB/s
8*512 4816 0.19s 83.76MB 0.21s 75.84MB/s
16*512 2408 0.19s 83.99MB 0.21s 77.18MB/s
32*512 1204 0.19s 84.45MB 0.22s 71.79MB/s
64*512602 0.20s 79.98MB 0.20s 78.44MB/s
128*512301 0.18s 86.51MB 0.22s 71.53MB/s
256*512150 0.19s 82.83MB 0.20s 78.86MB/s
512*512 75 0.19s 82.77MB 0.21s 76.39MB/s
1024*512 37 0.19s 85.62MB 0.21s 76.64MB/s
2048*512 18 0.21s 77.72MB 0.20s 80.30MB/s
4096*512 9 0.26s 61.06MB 0.30s 53.79MB/s
8192*512 4 0.83s 19.20MB 0.41s 39.12MB/s
16384*512 2 0.84s 19.01MB 0.41s 39.03MB/s
32768*512 1 0.82s 19.59MB 0.39s 40.89MB/s
kernel from 19/08/08 00:00:00:
1*512 38528 0.45s 35.59MB 0.20s 81.43MB/s
2*512 19264 0.45s 35.56MB 0.20s 79.24MB/s
4*512 9632 0.49s 32.66MB 0.22s 73.72MB/s
8*512 4816 0.47s 34.06MB 0.21s 75.52MB/s
16*512 2408 0.53s 30.16MB 0.22s 72.58MB/s
32*512 1204 0.31s 51.68MB 0.40s 40.14MB/s
64*512602 0.43s 37.23MB 0.25s 63.57MB/s
128*512301 0.51s 31.39MB 0.26s 62.70MB/s
256*512150 0.47s 34.02MB 0.23s 69.06MB/s
512*512 75 0.47s 34.01MB 0.23s 70.52MB/s
1024*512 37 0.53s 30.12MB 0.22s 73.01MB/s
2048*512 18 0.55s 29.07MB 0.23s 70.64MB/s
4096*512 9 0.46s 34.69MB 0.21s 75.92MB/s
8192*512 4 0.81s 19.66MB 0.43s 36.89MB/s
16384*512 2 0.80s 19.99MB 0.40s 40.29MB/s
32768*512 1 1.11s 14.41MB 0.38s 42.56MB/s
___
freebsd-stable@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: bad NFS/UDP performance
On Fri, 3 Oct 2008, Danny Braniss wrote: it more difficult than I expected. for one, the kernel date was missleading, the actual source update is the key, so the window of changes is now 28/July to 19/August. I have the diffs, but nothing yet seems relevant. on the other hand, I tried NFS/TCP, and there things seem ok, ie the 'good' and the 'bad' give the same throughput, which seem to point to UDP changes ... Can you post the network-numbers? so I ran some more test, these are for writes IO: OK, so it looks like this was almost certainly the rwlock change. What happens if you pretty much universally substitute the following in udp_usrreq.c: Currently Change to - - INP_RLOCK INP_WLOCK INP_RUNLOCK INP_WUNLOCK INP_RLOCK_ASSERTINP_WLOCK_ASSERT Robert N M Watson Computer Laboratory University of Cambridge server is a NetApp: kernel from 18/08/08 00:00:0 : /- UDP // TCP ---/ 1*512 38528 0.19s 83.50MB 0.20s 80.82MB/s 2*512 19264 0.21s 76.83MB 0.21s 77.57MB/s 4*512 9632 0.19s 85.51MB 0.22s 73.13MB/s 8*512 4816 0.19s 83.76MB 0.21s 75.84MB/s 16*512 2408 0.19s 83.99MB 0.21s 77.18MB/s 32*512 1204 0.19s 84.45MB 0.22s 71.79MB/s 64*512602 0.20s 79.98MB 0.20s 78.44MB/s 128*512301 0.18s 86.51MB 0.22s 71.53MB/s 256*512150 0.19s 82.83MB 0.20s 78.86MB/s 512*512 75 0.19s 82.77MB 0.21s 76.39MB/s 1024*512 37 0.19s 85.62MB 0.21s 76.64MB/s 2048*512 18 0.21s 77.72MB 0.20s 80.30MB/s 4096*512 9 0.26s 61.06MB 0.30s 53.79MB/s 8192*512 4 0.83s 19.20MB 0.41s 39.12MB/s 16384*512 2 0.84s 19.01MB 0.41s 39.03MB/s 32768*512 1 0.82s 19.59MB 0.39s 40.89MB/s kernel from 19/08/08 00:00:00: 1*512 38528 0.45s 35.59MB 0.20s 81.43MB/s 2*512 19264 0.45s 35.56MB 0.20s 79.24MB/s 4*512 9632 0.49s 32.66MB 0.22s 73.72MB/s 8*512 4816 0.47s 34.06MB 0.21s 75.52MB/s 16*512 2408 0.53s 30.16MB 0.22s 72.58MB/s 32*512 1204 0.31s 51.68MB 0.40s 40.14MB/s 64*512602 0.43s 37.23MB 0.25s 63.57MB/s 128*512301 0.51s 31.39MB 0.26s 62.70MB/s 256*512150 0.47s 34.02MB 0.23s 69.06MB/s 512*512 75 0.47s 34.01MB 0.23s 70.52MB/s 1024*512 37 0.53s 30.12MB 0.22s 73.01MB/s 2048*512 18 0.55s 29.07MB 0.23s 70.64MB/s 4096*512 9 0.46s 34.69MB 0.21s 75.92MB/s 8192*512 4 0.81s 19.66MB 0.43s 36.89MB/s 16384*512 2 0.80s 19.99MB 0.40s 40.29MB/s 32768*512 1 1.11s 14.41MB 0.38s 42.56MB/s ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: resource leak
On Wed, 1 Oct 2008, Stephen Clark wrote: A big part of problem is this seems to take about 100 days of uptime to occur. We have some inhouse test boxes but have never seen the problem, probably because non of them have been up more than about 45 days. The units in the field, of which there is about 300, are headless and none are physically close. When the boxes are rebooted there are no error messages in any of the log files, only the absence of information that would normally be logged by new processes that would be spawned. We are getting ready to install a patch that will try to gather more information. I thought about writing an app the would try to fork a child periodically and record in a log file if there was an error. But EAGAIN is nonspecific as to the real reason the fork failed. I was looking for some way to periodically log the resources that would cause the fork failure. The narrowness of the UNIX errno space is, at times, fairly unhelpful. As far as I'm aware, the two main causes of EAGAIN out of fork() are an exhaustion of maxprocs or an exhaustion of per-user process limits. This suggests one or more run-away applications or services, or a gradual leak of processes from a service (perhaps a failure to GC dead children, or a gradual increase but never decrease of worker processes?). Robert N M Watson Computer Laboratory University of Cambridge procstat -k looks like it would have been a good candidate but unfortunately we are running 6.1. Thanks for the response. Steve -- "They that give up essential liberty to obtain temporary safety, deserve neither liberty nor safety." (Ben Franklin) "The course of history shows that as a government grows, liberty decreases." (Thomas Jefferson) ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: resource leak
On Wed, 1 Oct 2008, Gary Palmer wrote: Periodically logging "ps -auxw" output to a file would be useful, as ideally you'd gradually see the list get longer and longer over time; it's possible you have many zombie processes as a result of a parent which is not reaping its children (calling waitpid(2) or its friends). "ps alxw" may be of interest in addition to "ps auxw" as it displays what the processes are waiting on. It could conceivably be a problem of some kind at the filesystem level. I've seen situations before where a problem escalates to the point where "ls /" hangs, and at that point you're stuck with an unresponsive box. If you want an even greater level of detail than ps -l, you can use procstat -k to generate kernel stack traces for all user/kernel threads. Wait channels are very useful, but they only tell you what the code that invoked the wait thinks it is for, not how that code was reached. A classic example is waiting on an exhausted UMA zone -- you get a uma wait channel, but no indication of what subsystem performed the memory allocation... This required FreeBSD 7.1 and higher, however. (Obviously, the same can be done easily using DDB, but that's hard on a box without a serial console, and requires interrupting the flow of the operating system, compiling with DDB, etc). Robert N M Watson Computer Laboratory University of Cambridge ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: jails and mac_seeotheruids problems in 6-STABLE
On Tue, 30 Sep 2008, George Mamalakis wrote: It works like a charm! Thank you very much for your time and help, No problem -- I've gone ahead and committed that change to stable/6. If you're able to test 6.4RC1 when it comes out to confirm that the fix works there as desired, that would be most helpful. Thanks, Robert N M Watson Computer Laboratory University of Cambridge regards, Robert Watson wrote: On Tue, 30 Sep 2008, George Mamalakis wrote: I have 3 servers in my lab. 2 of them are running 6-STABLE and one of them is running 7-STABLE. All three have services running in jails. I noticed a very peculiar behavior in 6-STABLE when I set the sysctl security.mac.seeotheruids.enabled=1. The root user in my jails was not able to see processes and sockets owned by other users of the same jail, whereas the root user of the host system could see every process (thank the Almighty). The same behavior does not apply on the server running 7-STABLE. In one sense it is more secure, since the root user in a jail is not as "strong" as the root user should be in a UNIX system. On the other hand, the root user looses its purpose of existence, which I suppose is a bug. Below are the security.mac sysctl settings of both 6 and 7-STABLE: Could you try modifying src/sys/security/mac_seeotheruids/mac_seeotheruids.c in a 6.x tree so that the call to suser_cred() in mac_seeotheruids_check() passes the SUSER_ALLOWJAIL flag rather than 0? This may correct the problem you're experiencing. Let me know and I can merge that change to 6.x. Robert N M Watson Computer Laboratory University of Cambridge 6-STABLE: security.mac.max_slots: 4 security.mac.enforce_network: 1 security.mac.enforce_pipe: 1 security.mac.enforce_posix_sem: 1 security.mac.enforce_suid: 1 security.mac.mmap_revocation_via_cow: 0 security.mac.mmap_revocation: 1 security.mac.enforce_vm: 1 security.mac.enforce_process: 1 security.mac.enforce_socket: 1 security.mac.enforce_system: 1 security.mac.enforce_kld: 1 security.mac.enforce_sysv_msg: 1 security.mac.enforce_sysv_sem: 1 security.mac.enforce_sysv_shm: 1 security.mac.enforce_fs: 1 security.mac.seeotheruids.specificgid: 0 security.mac.seeotheruids.specificgid_enabled: 0 security.mac.seeotheruids.primarygroup_enabled: 0 security.mac.seeotheruids.enabled: 1 security.mac.portacl.rules: uid:80:tcp:80,uid:80:tcp:443 security.mac.portacl.port_high: 1023 security.mac.portacl.autoport_exempt: 1 security.mac.portacl.suser_exempt: 1 security.mac.portacl.enabled: 1 7-STABLE: security.mac.max_slots: 4 security.mac.version: 3 security.mac.mmap_revocation_via_cow: 0 security.mac.mmap_revocation: 1 security.mac.seeotheruids.specificgid: 0 security.mac.seeotheruids.specificgid_enabled: 0 security.mac.seeotheruids.suser_privileged: 1 security.mac.seeotheruids.primarygroup_enabled: 0 security.mac.seeotheruids.enabled: 1 I would be very glad if someone could inform me whether I am doing something wrong; if not I think I should inform FreeBSD about this bug. Thank you guys in advance, -- George Mamalakis IT Officer Electrical and Computer Engineer (Aristotle Un. of Thessaloniki), MSc (Imperial College of London) Department of Electrical and Computer Engineering Faculty of Engineering Aristotle University of Thessaloniki phone number : +30 (2310) 994379 ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "[EMAIL PROTECTED]" -- George Mamalakis IT Officer Electrical and Computer Engineer (Aristotle Un. of Thessaloniki), MSc (Imperial College of London) Department of Electrical and Computer Engineering Faculty of Engineering Aristotle University of Thessaloniki phone number : +30 (2310) 994379 ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: 7.1-PRERELEASE : bad network performance (nfe0)
On Tue, 30 Sep 2008, Arno J. Klaassen wrote: However, the "request/respones" tests are awfull for my notebook (test repeated on the notebook for the sake of conviction) : Is it possible to rerun these tests with a 7.0 kernel of the same general configuration? That would help us determine if it's a regression between 7.0 and 7.1, 7.0-RELEASE-p4 kernel (and 7.1 world) as well as 7.0-RELEASE life-cd give same results : great streaming, very poor request/response Thanks for testing this -- it rules out a host of potential issues that could have been from changes in flight between 7.0 and 7.1, which is very helpful. At least for me, since I made many of those changes :-). or perhaps a more general issue between 6.x and 7.x. nve(4) does not recognise this chip. If someone does have a bootable 6-stable .iso with a backported nfe(4) ... or email if_nfe.ko to me and I will tes under 6-stable For now I will test the patches Pyun and Luigi sent me and let you know. OK. I'll drop out of the loop on this one unless it's determined that the network stack itself is implicated rather than the drivers. Robert N M Watson Computer Laboratory University of Cambridge ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: system hangup - I'm lost
On Tue, 30 Sep 2008, Gavin Atkinson wrote: On Mon, 2008-09-29 at 22:14 +0200, Oliver Lehmann wrote: Any idea what I could do to shed some more light on this behaviour? Why it is happening and what really is causing it? Would enabling the kernel debugger really help here? I mean the system is really hanging up - except ping response it is not responding to anything except the reset switch ;) If it's responding to ping, you should be able to get into the debugger. Compile it in, along with "options WITNESS" and "options WITNESS_SKIPSPIN", and press ctrl-alt-escape when the machine next hangs. From there, it should hopefully be possible to get more info. It's been a long time since I've used the debugger under 6.x so some of the more useful commands may not exist, but the output of at least "sh locks", "sh alllocks" and "bt" on any processes that seem to be holding locks. Also "sh pcpu" and "ps" will help to determine exactly what was running at the time. "show lockedvnods" is also quite useful if the problem originates in the file system, as it lists vnodes that have been locked, and by which threads. Robert N M Watson Computer Laboratory University of Cambridge ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "[EMAIL PROTECTED]"
