Re: Preserving early console output (pre-Copyright stuff)
Matthew Mondor wrote: > Sometimes I've been thinking about this as more and more hardware don't > ship with RS232 anymore. Is there a relatively common BIOS interface > which would allow, even if non-efficiently, to use a USB port as a > serial device without too much code? I doubt it. All uses of serial-via-USB that I've seen have required electronics (often buggy electronics) to do the conversion, and the one or two USB video interfaces I've also seen have also required electronics _and_ custom kernel drivers, and (you guessed it) there were bugs galore there too in the one I tried (hardware, software, or both I don't know). On the "big iron" I used to work with that had consoles with network support, the network connection was a pretty dumb piece of hardware which spoke ssh or telnet on the network side ... and RS-232 on the console side. :-( (OK, it also offered remote access to the reset and power switches, but the console was still -- in the 21st century -- based off RS-232. Maybe that changed in later models, but I wouldn't bet much on the possibility.) A USB console is a nice idea, but the industry doesn't seem to have picked it up(*). Which with all the EFI, IPMI, and other overly complex industry "standards" necessary to build a modern machine plus the prevalence of notebook PCs all with USB ports this is amusing, if you have a warped enough sense of humour ... (*) Or if there _is_ a USB console standard I've missed the memo and would welcome someone setting me straight, please. Regards, Giles
Re: MP locking?
>> [...roll forward pre-MP code to become MP-ready...] >> lock(9) outlines locking facilities which I believe I can use to do >> [the locking] I want - but there are other issues [...] > You need memory barriers, modern CPUs can do speculative reads, and > some can reorder writes. AFAIK, lock operations act as memory > barriers. atomic ops do not. But are lock calls enough? As I understand a memory barrier (which understanding is mostly based on CPU documentation of barrier instructions), it is not enough, since it does not imply any cache synchronization with other CPUs. If CPU A writes a shared data structure and then issues a memory barrier, either (i) this has no particular effect on CPU B's cache, leading B to possibly use stale data, or (ii) this forces CPU B to discard its whole read cache, incurring a mostly (possibly entirely) unnecessary performance hit. (ii) can be avoided if memory barriers apply to only certain addresses, but, since the locking calls do not take any addresses except those of the lock, the implicit memory barrier you refer to cannot have that kind of information associated with it. (This actually leads me to wonder: are those implicit memory barriers good for anything besides device access?) I find it hard to believe that releasing a lock causes all other CPUs to discard their entire read caches This leaves me with (i). I know some cache hardware does snooping of writes by other CPUs (or, semi-equivalently, DMA engines) to avoid the issue entirely, but I also know some hardware doesn't. How does NetBSD address the issue? The only thing "man -k barrier" turned up, besides a handful of pthread calls, is bus_space(9), which AIUI is not relevant here because these are data structures in ordinary kernel memory, not bus space - am I missing something? Does this just mean that 4.0.1 (which is what I was checking the manpages on) does not support MP on hardware which doesn't do cache snooping, or is there some facility I've missed for arranging for other-CPU cache flushes, or what? /~\ The ASCII Mouse \ / Ribbon Campaign X Against HTMLmo...@rodents-montreal.org / \ Email! 7D C8 61 52 5D E7 2D 39 4E F1 31 3E E8 B3 27 4B
Re: MP locking?
> Having just recently spent quite a bit of time porting a kernel > module from NetBSD-3 to NetBSD-5, and working out a bunch of > synchronization issues, I suggest skipping over NetBSD-4 and going > straight for 5. Noted, though I doubt a major version jump is in the cards for that machine in the foreseeable future. And, really, because it still needs to run in the original pre-MP kernel with minimal changes, what I'm going to be doing is pulling all the locking out so those differences can be hidden from everything but a relatively small and well-defined piece of the code, making further ports to the locking scheme du jour relatively simple. /~\ The ASCII Mouse \ / Ribbon Campaign X Against HTMLmo...@rodents-montreal.org / \ Email! 7D C8 61 52 5D E7 2D 39 4E F1 31 3E E8 B3 27 4B
Re: Preserving early console output (pre-Copyright stuff)
On Thu, 1 Jul 2010, Allen Briggs wrote: Heh. You do need a serial cable and another computer, but it's not too hard to set up. You don't need to find any options in the BIOS if you install the right boot blocks. Yeah, OK, I got a cable, plugged it in, run cu(1) on one end, and added a new entry to the /boot.cfg on the other end: menu=Serial/Single/Verbose/Debug:consdev com0; boot -svx Then just boot, and things magically work! Thanks for the clue-by-four - | Paul Goyette | PGP Key fingerprint: | E-mail addresses: | | Customer Service | FA29 0E3B 35AF E8AE 6651 | paul at whooppee.com| | Network Engineer | 0786 F758 55DE 53BA 7731 | pgoyette at juniper.net | | Kernel Developer | | pgoyette at netbsd.org | -
re: updating COMPAT_LINUX for linux 2.6.x support (take 2)
> On Thu, Jul 01, 2010 at 01:32:47AM +0200, Rhialto wrote: > > On Wed 30 Jun 2010 at 08:57:44 -0700, Chuck Silvers wrote: > > >- the assembler apparently now uses a different opcode for reloading > > > %gs > > > than it used to, so the check for this instruction in INTRFASTEXIT > > > being the cause of a kernel trap wasn't working. I fixed the check > > > to match the current generated code. > > > > Would it not be necessary for compatibility with older binaries (how > > much older??) to check for both cases? > > there's no compatibility issue here, the kernel is looking at its own > instructions, not an application's. we require that a kernel be built > with the matching toolchain, so there's only one version of the assembler > that would matter for a particular kernel binary. we shouldn't depend upon GCC/gas versions so heavily IMO. testing for both should be failry easy, right? sometimes newer / older toolchain components are useful for various reasons.. .mrg.
Re: updating COMPAT_LINUX for linux 2.6.x support (take 2)
On Thu, Jul 01, 2010 at 01:32:47AM +0200, Rhialto wrote: > On Wed 30 Jun 2010 at 08:57:44 -0700, Chuck Silvers wrote: > >- the assembler apparently now uses a different opcode for reloading %gs > > than it used to, so the check for this instruction in INTRFASTEXIT > > being the cause of a kernel trap wasn't working. I fixed the check > > to match the current generated code. > > Would it not be necessary for compatibility with older binaries (how > much older??) to check for both cases? there's no compatibility issue here, the kernel is looking at its own instructions, not an application's. we require that a kernel be built with the matching toolchain, so there's only one version of the assembler that would matter for a particular kernel binary. -Chuck
Re: MP locking?
Hello. Having just recently spent quite a bit of time porting a kernel module from NetBSD-3 to NetBSD-5, and working out a bunch of synchronization issues, I suggest skipping over NetBSD-4 and going straight for 5. You'll get better multi-cpu capability, and it looks like the locking APIs in 5.x will be longer lived going forward than the now depracated and no longer functioning APIs in 4.x and earlier. -Brian On Jul 1, 11:13am, Manuel Bouyer wrote: } Subject: Re: MP locking? } On Wed, Jun 30, 2010 at 05:15:53PM -0400, der Mouse wrote: } > I have some kernel code which was written for a pre-MP kernel; it uses } > spl*() for locking. } > I'd like to roll this forward to something at } > least slightly more modern - specifically, a dual-CPU 4.0.1 machine. } > } > lock(9) outlines locking facilities which I believe I can use to do } > what I want - but there are other issues, such as cache coherency; do I } > need to do anything special with shared data structures to ensure } > coherency between processors? Is it enough to declare them volatile, } > or do I need memory barriers as well, or what? } } You need memory barriers, modern CPUs can do speculative reads, and some } can reorder writes. AFAIK, lock operations act as memory barriers. } atomic ops do not. } } -- } Manuel Bouyer } NetBSD: 26 ans d'experience feront toujours la difference } -- >-- End of excerpt from Manuel Bouyer
Re: Move the vnode lock into file systems
On Tue, Jun 29, 2010 at 04:23:53PM +0100, Mindaugas Rasiukevicius wrote: > David Holland wrote: > > Vaguely related to which, does anyone object to wrapping VOP_UNLOCK in > > a vn_unlock() function (doing nothing extra), so as to be symmetric > > with vn_lock()? I think I've mentioned this before, but I'm not sure, > > and if so it was a while back... > > I think it would be useful to have symmetric vn_unlock(), yes. Can be a > static inline in vnode.h header. vn_unlock() would be dressing, which is fair enough. Consider that vn_lock() and LK_RETRY appear to have no legitimate purpose now that VFS has been multithreaded, with the result that the clowning about with v_interlock -> v_lock handoff is dead. VOP_LOCK() and VOP_UNLOCK() are all that's needed.
Re: Move the vnode lock into file systems
On Tue, Jun 29, 2010 at 04:54:46PM +0200, Juergen Hannken-Illjes wrote:
> On Sat, Jun 26, 2010 at 10:39:27AM +0200, Juergen Hannken-Illjes wrote:
> > The vnode lock operations currently work on a rw lock located inside the
> > vnode. I propose to move this lock into the file system node.
> >
> > This place is more logical as we lock a file system node and not a vnode.
> > This becomes clear if we think of a file system where one file system node
> > is attached to more than one vnode. Ptyfs allowing multiple mounts is such
> > a candidate.
> >
> > A diff implemeting this for ufs, lfs and ext2fs is attached.
> >
> > Comments or objections anyone?
>
> Looks like the minimal consensus for now is to remove the vlockmgr().
> The attached diff does it and rearranges vnode locking (in the genfs case)
> into the functions genfs_lock(), genfs_unlock() and genfs_islocked().
> The lock itself goes into the vnode and the remaining LK_* flags move
> from sys/lock.h to sys/vnode.h. All in-file system calls to vlockmgr()
> get replaced with VOP_LOCK() or VOP_UNLOCK().
>
> Comments or objections anyone?
I think, if I was doing this, I'd look at putting the lock into genfs_node,
and directly into the in-core inode for file systems that don't make use
of genfs.
Not sure what advantage there is to converting in-FS vlockmgr() calls over
to VOP_LOCK/VOP_UNLOCK.
(Then kill the in-vnode lock completely and take a critical look
at the fields in the vnode that are protected by the vnode lock. I know
of at least a couple of external places that have implicit dependencies
on the vnode lock, example the namecache.)
>
> --
> Juergen Hannken-Illjes - hann...@eis.cs.tu-bs.de - TU Braunschweig (Germany)
>
> Index: sys/coda/coda_vnops.c
> ===
> RCS file: /cvsroot/src/sys/coda/coda_vnops.c,v
> retrieving revision 1.74
> diff -p -u -4 -r1.74 coda_vnops.c
> --- sys/coda/coda_vnops.c 24 Jun 2010 13:03:06 - 1.74
> +++ sys/coda/coda_vnops.c 29 Jun 2010 10:27:29 -
> @@ -1774,9 +1774,8 @@ coda_lock(void *v)
> /* true args */
> struct vop_lock_args *ap = v;
> struct vnode *vp = ap->a_vp;
> struct cnode *cp = VTOC(vp);
> -int flags = ap->a_flags;
> /* upcall decl */
> /* locals */
>
> ENTRY;
> @@ -1785,9 +1784,9 @@ coda_lock(void *v)
> myprintf(("Attempting lock on %s\n",
> coda_f2s(&cp->c_fid)));
> }
>
> -return (vlockmgr(&vp->v_lock, flags));
> +return genfs_lock(v);
> }
>
> int
> coda_unlock(void *v)
> @@ -1804,19 +1803,18 @@ coda_unlock(void *v)
> myprintf(("Attempting unlock on %s\n",
> coda_f2s(&cp->c_fid)));
> }
>
> -return (vlockmgr(&vp->v_lock, LK_RELEASE));
> +return genfs_unlock(v);
> }
>
> int
> coda_islocked(void *v)
> {
> /* true args */
> -struct vop_islocked_args *ap = v;
> ENTRY;
>
> -return (vlockstatus(&ap->a_vp->v_lock));
> +return genfs_islocked(v);
> }
>
> /*
> * Given a device and inode, obtain a locked vnode. One reference is
> Index: sys/fs/adosfs/adutil.c
> ===
> RCS file: /cvsroot/src/sys/fs/adosfs/adutil.c,v
> retrieving revision 1.11
> diff -p -u -4 -r1.11 adutil.c
> --- sys/fs/adosfs/adutil.c14 Mar 2009 21:04:23 - 1.11
> +++ sys/fs/adosfs/adutil.c29 Jun 2010 10:27:31 -
> @@ -86,9 +86,9 @@ start_over:
> */
> void
> adosfs_ainshash(struct adosfsmount *amp, struct anode *ap)
> {
> - vlockmgr(&ap->vp->v_lock, LK_EXCLUSIVE);
> + VOP_LOCK(ATOV(ap), LK_EXCLUSIVE);
>
> simple_lock(&adosfs_hashlock);
> LIST_INSERT_HEAD(&->anodetab[AHASH(ap->block)], ap, link);
> simple_unlock(&adosfs_hashlock);
> Index: sys/fs/cd9660/cd9660_node.c
> ===
> RCS file: /cvsroot/src/sys/fs/cd9660/cd9660_node.c,v
> retrieving revision 1.25
> diff -p -u -4 -r1.25 cd9660_node.c
> --- sys/fs/cd9660/cd9660_node.c 24 Jun 2010 13:03:09 - 1.25
> +++ sys/fs/cd9660/cd9660_node.c 29 Jun 2010 10:27:31 -
> @@ -177,9 +177,9 @@ cd9660_ihashins(struct iso_node *ip)
> ipp = &isohashtbl[INOHASH(ip->i_dev, ip->i_number)];
> LIST_INSERT_HEAD(ipp, ip, i_hash);
> mutex_exit(&cd9660_ihash_lock);
>
> - vlockmgr(&ip->i_vnode->v_lock, LK_EXCLUSIVE);
> + VOP_LOCK(ITOV(ip), LK_EXCLUSIVE);
> }
>
> /*
> * Remove the inode from the hash table.
> Index: sys/fs/efs/efs_ihash.c
> ===
> RCS file: /cvsroot/src/sys/fs/efs/efs_ihash.c,v
> retrieving revision 1.4
> diff -p -u -4 -r1.4 efs_ihash.c
> --- sys/fs/efs/efs_ihash.c5 May 2008 17:11:16 - 1.4
> +++ sys/fs/efs/efs_ihash.c29 Jun 2010 10:27:31 -
> @@ -166,9 +166,9 @@ efs_ihashins(struct efs_inode *eip)
>
> KASSERT(mutex_owned(&efs_hashlock));
>
> /* lock the
Re: lockdebug messages in dmesg (was: Preserving early console output (pre-Copyright stuff))
On Thu, Jul 01, 2010 at 04:13:36PM +0200, Adam Hoka wrote: > btw, is this intentional that lockdebug panic messages > dont end up in dmesg? Yes, just because that requires taking additional locks. > -- > NetBSD - Simplicity is prerequisite for reliability
Re: Preserving early console output (pre-Copyright stuff)
On Thu, 1 Jul 2010 06:00:41 -0700 (PDT) Paul Goyette wrote: > That's what I thought I'd get for an answer! :) > > There is a serial port, but I haven't figured out yet how to make it > work in the BIOS. And while I do have other machines with serial ports > I've never used those ports and don't even have serial cables! (The > last time I used a serial cable was way back in the days of modems and > dial-up 'net access!) Sometimes I've been thinking about this as more and more hardware don't ship with RS232 anymore. Is there a relatively common BIOS interface which would allow, even if non-efficiently, to use a USB port as a serial device without too much code? If so, possibly that a special usb-serial bootblock could be using that sometime in the future? If there is no BIOS common interface, I can see it's a problem because of all the driver code that'd be needed at boot time... Thanks, -- Matt
Re: Preserving early console output (pre-Copyright stuff)
On Thu, Jul 01, 2010 at 06:00:41AM -0700, Paul Goyette wrote: > There is a serial port, but I haven't figured out yet how to make it > work in the BIOS. And while I do have other machines with serial ports > I've never used those ports and don't even have serial cables! (The > last time I used a serial cable was way back in the days of modems and > dial-up 'net access!) Heh. You do need a serial cable and another computer, but it's not too hard to set up. You don't need to find any options in the BIOS if you install the right boot blocks. > The "pause every N calls" is probably over-kill - I'd be happy to pause > on every call, if there is an appropriate pause() function that can be > called at this early part of the boot process. You could always do a simple spin-loop. Assuming this is a temporary thing you're doing for a single (or limited number of) system(s). -allen -- Allen Briggs - bri...@ninthwonder.com
lockdebug messages in dmesg (was: Preserving early console output (pre-Copyright stuff))
btw, is this intentional that lockdebug panic messages dont end up in dmesg? -- NetBSD - Simplicity is prerequisite for reliability
Re: Preserving early console output (pre-Copyright stuff)
On Thu, 1 Jul 2010 05:18:36 -0700 (PDT) Paul Goyette wrote: > b) a way to pause long enough to manually transcribe the output? (A > simple timed delay would work, although a "Press any key to continue" > would be easier!) Get a semi-decent digital camera, set high ISO speed in a well lit room and take a snapshot of the screen. Might take you a few tries to capture all data, but it may be quicker than figuring out how to set up serial console. If the image is too blurry, increase the ISO + add more light.
Re: Preserving early console output (pre-Copyright stuff)
On Thu, 1 Jul 2010, Allen Briggs wrote: On Thu, Jul 01, 2010 at 05:18:36AM -0700, Paul Goyette wrote: a) a way to capture these messages in the normal message buffer for later collection by syslog, or b) a way to pause long enough to manually transcribe the output? (A simple timed delay would work, although a "Press any key to continue" would be easier!) I don't know of an easy way to do either (although I could see adding something to pause every N calls if it's not logging yet), but this is the kind of thing that a serial console can be really handy for. Do you have serial ports on the system in question? That's what I thought I'd get for an answer! :) There is a serial port, but I haven't figured out yet how to make it work in the BIOS. And while I do have other machines with serial ports I've never used those ports and don't even have serial cables! (The last time I used a serial cable was way back in the days of modems and dial-up 'net access!) The "pause every N calls" is probably over-kill - I'd be happy to pause on every call, if there is an appropriate pause() function that can be called at this early part of the boot process. - | Paul Goyette | PGP Key fingerprint: | E-mail addresses: | | Customer Service | FA29 0E3B 35AF E8AE 6651 | paul at whooppee.com| | Network Engineer | 0786 F758 55DE 53BA 7731 | pgoyette at juniper.net | | Kernel Developer | | pgoyette at netbsd.org | -
Re: Preserving early console output (pre-Copyright stuff)
On Thu, Jul 01, 2010 at 05:18:36AM -0700, Paul Goyette wrote: > a) a way to capture these messages in the normal message buffer for > later collection by syslog, or > > b) a way to pause long enough to manually transcribe the output? (A > simple timed delay would work, although a "Press any key to continue" > would be easier!) I don't know of an easy way to do either (although I could see adding something to pause every N calls if it's not logging yet), but this is the kind of thing that a serial console can be really handy for. Do you have serial ports on the system in question? -allen -- Allen Briggs - bri...@ninthwonder.com
Preserving early console output (pre-Copyright stuff)
I'm trying to debug some stuff on a new server, and there are some "messages of interest" that get printf'd on the console very early in the system's life, before the "Copyright (c)..." lines. There is a large amount of info being printed, way more than a screenful and way more than can be captured with an eyeball snapshot. Unfortunately, it seems that this information is NOT retained in the kernel's message buffer, so it never gets collected by syslog. (For those who want to know details, some of the messages in question are generated by add_mem_cluster() in src/sys/arch/x86/x86/x86_machdep.c) So, is there either a) a way to capture these messages in the normal message buffer for later collection by syslog, or b) a way to pause long enough to manually transcribe the output? (A simple timed delay would work, although a "Press any key to continue" would be easier!) Thanks in advance! - | Paul Goyette | PGP Key fingerprint: | E-mail addresses: | | Customer Service | FA29 0E3B 35AF E8AE 6651 | paul at whooppee.com| | Network Engineer | 0786 F758 55DE 53BA 7731 | pgoyette at juniper.net | | Kernel Developer | | pgoyette at netbsd.org | -
RE: MP locking?
> > ... > > You need memory barriers, modern CPUs can do speculative reads, and > some > > can reorder writes. AFAIK, lock operations act as memory barriers. > > atomic ops do not. > > With my PAT patch you can map a page uncached. This disables > speculative accesses. If other mechanisms are available to get the right synchronization or ordering of visibility, that's generally better. Depending on the machine, uncached can have other side effects you might not want. And it certainly is likely to have a significant performance impact. paul
Re: MP locking?
On Thursday 01 July 2010 11:13:12 Manuel Bouyer wrote: > On Wed, Jun 30, 2010 at 05:15:53PM -0400, der Mouse wrote: > > I have some kernel code which was written for a pre-MP kernel; it uses > > spl*() for locking. > > I'd like to roll this forward to something at > > least slightly more modern - specifically, a dual-CPU 4.0.1 machine. > > > > lock(9) outlines locking facilities which I believe I can use to do > > what I want - but there are other issues, such as cache coherency; do I > > need to do anything special with shared data structures to ensure > > coherency between processors? Is it enough to declare them volatile, > > or do I need memory barriers as well, or what? > > You need memory barriers, modern CPUs can do speculative reads, and some > can reorder writes. AFAIK, lock operations act as memory barriers. > atomic ops do not. With my PAT patch you can map a page uncached. This disables speculative accesses. Christoph
Re: MP locking?
On Wed, Jun 30, 2010 at 05:15:53PM -0400, der Mouse wrote: > I have some kernel code which was written for a pre-MP kernel; it uses > spl*() for locking. > I'd like to roll this forward to something at > least slightly more modern - specifically, a dual-CPU 4.0.1 machine. > > lock(9) outlines locking facilities which I believe I can use to do > what I want - but there are other issues, such as cache coherency; do I > need to do anything special with shared data structures to ensure > coherency between processors? Is it enough to declare them volatile, > or do I need memory barriers as well, or what? You need memory barriers, modern CPUs can do speculative reads, and some can reorder writes. AFAIK, lock operations act as memory barriers. atomic ops do not. -- Manuel Bouyer NetBSD: 26 ans d'experience feront toujours la difference --
