Apache2 in per-child mode (was Re: libthr and 1:1 threading)
Hi all, Just a note. Apache2 in per-child mode now works with freebsd, while it deadlocked in the old libc_r on STABLE and CURRENT. Thank you very much ! Martin ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: libthr and 1:1 threading.
Matthew Dillon wrote:
> :How does this break the read() API? The read() API, when called
> :on a NBIO fd is *supposed* to return EAGAIN, if the request cannot
> :be immediately satisfied, but could be satisfied later. Right now,
> :it blocks. This looks like breakage of disk I/O introducing a
> :stall, when socket I/O doesn't.
> :
> :If this breaks read() semantics, then socket I/O needs fixing to
> :unbreak them, right?
>
> Oh please. You know very well that every single UNIX out there
> operates on disk files as if their data was immediately available
> regardless of whether the process blocks in an uninterruptable
> disk wait or not.
False. SVR4.0.2 and SVR4.2 do not. They act as I describe. The
code was written by a guy named Steve Baumel.
> What you are suggesting is that we make our file interface
> incompatible with every other unix out there... ours will
> return EAGAIN in situations where programs wouldn't expect it.
According to the FreeBSD 5.x man page for read(2):
[EAGAIN] The file was marked for non-blocking I/O, and no data
were ready to be read.
...in other words, they mark it for non-blocking I/O, they
*better* expect it!
And at least /usr/src/lib/libc_r/uthread/uthread_read[v].c expects
it from the kernel.
/* Perform a non-blocking read syscall: */
while ((ret = __sys_read(fd, buf, nbytes)) < 0) {
if ((_thread_fd_getflags(fd) & O_NONBLOCK) == 0 &&
(errno == EWOULDBLOCK || errno == EAGAIN)) {
curthread->data.fd.fd = fd;
_thread_kern_set_timeout(NULL);
The kernel also certainly expects, if not EAGAIN, EWOULDBLOCK:
if ((error = fo_read(fp, &auio, td->td_ucred, flags, td))) {
if (auio.uio_resid != cnt && (error == ERESTART ||
error == EINTR || error == EWOULDBLOCK))
error = 0;
}
cnt -= auio.uio_resid;
> Additionally, the EAGAIN operation would be highly non-deterministic
> and it would be fairly difficult for a program to rely on it because
> there would be no easy way (short of experiementation or a sysctl) for
> it to determine whether the 'feature' is present or not.
??? It's in the man page! You *must* handle it, if it's in the
man page!
You know that there are a number of VOP_CLOSE routines that can
return EAGAIN, right? Including ufs_close().
> Also, the idea that the resulting block I/O operation is then queued
> and one returns immediately from way down deep in the filesystem device
> driver code, and that this whole mess is then tied into select()/kqueue()/
> poll(), is just asking for more non-determinism... now it would
> depend on the filesystem AND the OS supporting the feature, and other
> UNIX implementations (if they were to adopt the mechanism) would likely
> wind up with slightly different semantics, just like O_NONBLOCK on
> listen() sockets has wound up being broken on things like HPUX.
No. It creates no obligations on the part of applications or other
UNIX implementations which are not already there. It doesn't break
POSIX semantics.
> For example, how would one deal with, say, issuing a million of these
> special non-blocking reads() all of which fail. Do we queue a million
> I/Os? Do we queue just the last requested I/O? You see the problem?
> The API would be unstable and almost certainly implemented differently
> on each OS platform.
They aren't "special". You handle them by issuing an EAGAIN, if
they can't be immediately satisfied. Just like the man page says.
I don't think you are understanding. This is not a replacement
for AIO. It's a way of touching pages to force them into the
buffer cache, rather than blocking. It's permitted by POSIX
for read(2) to return EAGAIN to do this.
There's no requirement on the queuing of the I/O. I'd suggest
that you don't attempt more than one simultaneously on a
descriptor though, since it's not going to do anything for you,
since each one that fails is going to also fail to use the "resid"
value to update the file pointer.
So if you issue a million of these for one page, well... you've
just asked for the same page to be loaded into memory a million
times, because the read(2) system call doesn't advance the file
pointer except by the amount of its non-negative return value.
8-) 8-).
> A better solution would be to implement a new system call, similar to
> pread(), which simply checks the buffer cache and returns a short read
> or an error if the data is not present. If the call fails you would
> then know that reading that data would block in the disk subsystem and
> you could back-off to a more expensive mechanism like AIO. If want
> to select() on it you would then simply use kqueue with EVFIL
Re: libthr and 1:1 threading.
> I have commited libthr. To try this out you'll need to do the following > > 1. cvsup > 2. rebuild world and kernel > 3. install world and kernel > 4. build libthr from src/lib/libthr > 5. Either replace /usr/lib/libc_r.so.5 with /usr/lib/libthr.so.1 or > relink your applications against libthr.so.1 > > This works with mozilla and open office. Point of reference, it works well with KDE, but there's something non-kosher someplace. I've been getting this periodically and haven't tracked down where or why I'm getting it. On a different note, SCHED_4BSD causes the kernel to hang hard on KDE's startup during the loading peripherals phase. I'm not home and don't have any way of debugging this from a remote machine. Here's the backtrace for konsole. -sc 0x29063b63 in wait4 () from /usr/lib/libc.so.5 #0 0x29063b63 in wait4 () from /usr/lib/libc.so.5 #1 0x29055035 in waitpid () from /usr/lib/libc.so.5 #2 0x290105e5 in _waitpid (wpid=7, status=0x7, options=7) at /usr/src/lib/libthr/thread/thr_syscalls.c:386 #3 0x286d7b8a in KCrash::defaultCrashHandler(int) (sig=6) at kcrash.cpp:235 #4 #5 0x29063843 in kill () from /usr/lib/libc.so.5 #6 0x29345603 in TEPty::makePty(bool) (this=0xbfbfead8) at TEPty.cpp:534 #7 0x2934568a in TEPty::startPgm(char const*, QValueList&, char const*) (this=0x81c7a00, pgm=0x7 , [EMAIL PROTECTED], term=0x8124920 "xterm-color") at TEPty.cpp:550 #8 0x29345eca in TEPty::commSetupDoneC() (this=0x81c7a00) at qcstring.h:295 #9 0x28681189 in KProcess::start(KProcess::RunMode, KProcess::Communication) ( this=0x81c7a00, runmode=7, comm=NoCommunication) at kprocess.cpp:320 #10 0x29344ee6 in TEPty::run(char const*, QStrList&, char const*, bool, char const*, char const*) (this=0x81c7a00, _pgm=0x7 , [EMAIL PROTECTED], _term=0x7 , _addutmp=true, _konsole_dcop=0x7 , _konsole_dcop_session=0x7 ) at TEPty.cpp:321 #11 0x29369979 in TESession::run() (this=0x81cc300) at qcstring.h:295 #12 0x2936b7b9 in TESession::qt_invoke(int, QUObject*) (this=0x81cc300, _id=2, _o=0xbfbfef30) at session.moc:201 #13 0x28a25308 in QObject::activate_signal(QConnectionList*, QUObject*) () from /usr/X11R6/lib/libqt-mt.so.3 #14 0x28cf17ad in QSignal::signal(QVariant const&) () from /usr/X11R6/lib/libqt-mt.so.3 #15 0x28a3eca8 in QSignal::activate() () from /usr/X11R6/lib/libqt-mt.so.3 #16 0x28a45a53 in QSingleShotTimer::event(QEvent*) () from /usr/X11R6/lib/libqt-mt.so.3 #17 0x289c8535 in QApplication::internalNotify(QObject*, QEvent*) () from /usr/X11R6/lib/libqt-mt.so.3 #18 0x289c82fb in QApplication::notify(QObject*, QEvent*) () from /usr/X11R6/lib/libqt-mt.so.3 #19 0x2864c1a9 in KApplication::notify(QObject*, QEvent*) (this=0x7, receiver=0x80ba080, event=0xbfbff1d0) at kapplication.cpp:453 #20 0x289a48d7 in QEventLoop::activateTimers() () from /usr/X11R6/lib/libqt-mt.so.3 #21 0x28983cb1 in QEventLoop::processEvents(unsigned) () from /usr/X11R6/lib/libqt-mt.so.3 #22 0x289dbf20 in QEventLoop::enterLoop() () from /usr/X11R6/lib/libqt-mt.so.3 #23 0x289dbe58 in QEventLoop::exec() () from /usr/X11R6/lib/libqt-mt.so.3 #24 0x289c86c1 in QApplication::exec() () from /usr/X11R6/lib/libqt-mt.so.3 #25 0x2934a83d in main (argc=7, argv=0x7) at main.cpp:435 #26 0x0804cb16 in launch (argc=1, _name=0x805bc04 "konsole", args=0x805bc0c "\001", cwd=0x0, envc=1, envs=0x805bc1d "", reset_env=false, tty=0x0, avoid_loops=false, startup_id_str=0x7 ) at kinit.cpp:547 #27 0x0804d906 in handle_launcher_request (sock=7) at kinit.cpp:1021 #28 0x0804de57 in handle_requests (waitForPid=0) at kinit.cpp:1189 #29 0x0804ef43 in main (argc=3, argv=0xbfbffbac, envp=0x7) at kinit.cpp:1540 #30 0x0804b115 in _start () -- Sean Chittenden ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: libthr and 1:1 threading.
can we have a subject ID? the KSE list prefixes with [KSE] and I've grown used to not ignoring those :-) On Wed, 2 Apr 2003, Peter Wemm wrote: > Julian Elischer wrote: > > Yes I think so.. > > I think 'threads is a better name thatn 'kse' though kse > > is good in that it's real quick to type :-) > > OK, done. It seems to me we've needed one for a while now. > > Subscribe by either: > http://lists.freebsd.org/mailman/listinfo/freebsd-threads > or > echo "subscribe" | mail [EMAIL PROTECTED] > or > echo "anything at all" | mail [EMAIL PROTECTED] > > > On Wed, 2 Apr 2003, Peter Wemm wrote: > > > > > Terry Lambert wrote: > > > > > > > KSE mailing list, starting Monday or so: > > > > ] We still haven't heard from jeff with regard to the process > > > > ] signal mask removal. > > > > > > We can add new mailing lists really easily now - it takes about 20-30 secon > ds. > > > Would it be worth adding a freebsd-threads and/or freebsd-kse type list > > > where it is a bit higher profile? > > > > > > Cheers, > > > -Peter > > > -- > > > Peter Wemm - [EMAIL PROTECTED]; [EMAIL PROTECTED]; [EMAIL PROTECTED] > > > "All of this is for nothing if we don't go to the stars" - JMS/B5 > > > > > > ___ > > > [EMAIL PROTECTED] mailing list > > > http://lists.freebsd.org/mailman/listinfo/freebsd-current > > > To unsubscribe, send any mail to "[EMAIL PROTECTED]" > > > > > > > Cheers, > -Peter > -- > Peter Wemm - [EMAIL PROTECTED]; [EMAIL PROTECTED]; [EMAIL PROTECTED] > "All of this is for nothing if we don't go to the stars" - JMS/B5 > > ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: libthr and 1:1 threading.
Julian Elischer wrote: > Yes I think so.. > I think 'threads is a better name thatn 'kse' though kse > is good in that it's real quick to type :-) OK, done. It seems to me we've needed one for a while now. Subscribe by either: http://lists.freebsd.org/mailman/listinfo/freebsd-threads or echo "subscribe" | mail [EMAIL PROTECTED] or echo "anything at all" | mail [EMAIL PROTECTED] > On Wed, 2 Apr 2003, Peter Wemm wrote: > > > Terry Lambert wrote: > > > > > KSE mailing list, starting Monday or so: > > > ] We still haven't heard from jeff with regard to the process > > > ] signal mask removal. > > > > We can add new mailing lists really easily now - it takes about 20-30 secon ds. > > Would it be worth adding a freebsd-threads and/or freebsd-kse type list > > where it is a bit higher profile? > > > > Cheers, > > -Peter > > -- > > Peter Wemm - [EMAIL PROTECTED]; [EMAIL PROTECTED]; [EMAIL PROTECTED] > > "All of this is for nothing if we don't go to the stars" - JMS/B5 > > > > ___ > > [EMAIL PROTECTED] mailing list > > http://lists.freebsd.org/mailman/listinfo/freebsd-current > > To unsubscribe, send any mail to "[EMAIL PROTECTED]" > > > Cheers, -Peter -- Peter Wemm - [EMAIL PROTECTED]; [EMAIL PROTECTED]; [EMAIL PROTECTED] "All of this is for nothing if we don't go to the stars" - JMS/B5 ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: libthr and 1:1 threading.
A thought on 'fixing AIO..' On Wed, 2 Apr 2003, Matthew Dillon wrote: > > A better solution would be to implement a new system call, similar to > pread(), which simply checks the buffer cache and returns a short read > or an error if the data is not present. If the call fails you would > then know that reading that data would block in the disk subsystem and > you could back-off to a more expensive mechanism like AIO. If want > to select() on it you would then simply use kqueue with EVFILT_AIO and > AIO. A system call pread_cache(), or perhaps we could even use > recvmsg() with a flag. Such an interface would not have to touch the > filesystem code, only the buffer cache and the VM page cache, and > could be implemented in less then a day. > Just as a point of interest, we now have the ability for a non-threaded program to have several threads in the kernel.. By this I mean, it would be theoretically possible to re-implement aioread() in terms of some background threads (doing synchronous IO) in the kernel, that the program is not aware of.. We don't have this hapen at teh moment.. (hmm actually we do but...only in KSE programs) but we have the infrastructure that would allow it to be done by someone who has a spare day or so.. Basically teh aioread would return, but the process would have left a worker thread in the kernel, completing the work, and since the thread is attached to the process, when it is reactivated on data arrival, the correct address space would be there automatically.. All 'exit' cases would be handled automatically.. etc. etc. ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: libthr and 1:1 threading.
:How does this break the read() API? The read() API, when called :on a NBIO fd is *supposed* to return EAGAIN, if the request cannot :be immediately satisfied, but could be satisfied later. Right now, :it blocks. This looks like breakage of disk I/O introducing a :stall, when socket I/O doesn't. : :If this breaks read() semantics, then socket I/O needs fixing to :unbreak them, right? Oh please. You know very well that every single UNIX out there operates on disk files as if their data was immediately available regardless of whether the process blocks in an uninterruptable disk wait or not. What you are suggesting is that we make our file interface incompatible with every other unix out there... ours will return EAGAIN in situations where programs wouldn't expect it. Additionally, the EAGAIN operation would be highly non-deterministic and it would be fairly difficult for a program to rely on it because there would be no easy way (short of experiementation or a sysctl) for it to determine whether the 'feature' is present or not. Also, the idea that the resulting block I/O operation is then queued and one returns immediately from way down deep in the filesystem device driver code, and that this whole mess is then tied into select()/kqueue()/ poll(), is just asking for more non-determinism... now it would depend on the filesystem AND the OS supporting the feature, and other UNIX implementations (if they were to adopt the mechanism) would likely wind up with slightly different semantics, just like O_NONBLOCK on listen() sockets has wound up being broken on things like HPUX. For example, how would one deal with, say, issuing a million of these special non-blocking reads() all of which fail. Do we queue a million I/Os? Do we queue just the last requested I/O? You see the problem? The API would be unstable and almost certainly implemented differently on each OS platform. A better solution would be to implement a new system call, similar to pread(), which simply checks the buffer cache and returns a short read or an error if the data is not present. If the call fails you would then know that reading that data would block in the disk subsystem and you could back-off to a more expensive mechanism like AIO. If want to select() on it you would then simply use kqueue with EVFILT_AIO and AIO. A system call pread_cache(), or perhaps we could even use recvmsg() with a flag. Such an interface would not have to touch the filesystem code, only the buffer cache and the VM page cache, and could be implemented in less then a day. -Matt ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: libthr and 1:1 threading.
Yes I think so.. I think 'threads is a better name thatn 'kse' though kse is good in that it's real quick to type :-) On Wed, 2 Apr 2003, Peter Wemm wrote: > Terry Lambert wrote: > > > KSE mailing list, starting Monday or so: > > ] We still haven't heard from jeff with regard to the process > > ] signal mask removal. > > We can add new mailing lists really easily now - it takes about 20-30 seconds. > Would it be worth adding a freebsd-threads and/or freebsd-kse type list > where it is a bit higher profile? > > Cheers, > -Peter > -- > Peter Wemm - [EMAIL PROTECTED]; [EMAIL PROTECTED]; [EMAIL PROTECTED] > "All of this is for nothing if we don't go to the stars" - JMS/B5 > > ___ > [EMAIL PROTECTED] mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-current > To unsubscribe, send any mail to "[EMAIL PROTECTED]" > ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: libthr and 1:1 threading.
Terry Lambert wrote: Stijn Hoop wrote: On Mon, Mar 31, 2003 at 10:54:45PM -0500, Jeff Roberson wrote: I have commited libthr. To try this out you'll need to do the following I know very very little about threads, but I'm interested as to what the purpose is of this library. Is there a document available somewhere that describes the relationships between this, KSE, libc_r, pthreads, the Giant-unwinding-make-SMP-work-better project and some of the other threads and SMP related libraries and terminology? Here's a thumbnail sketch, though (forgive me, KSE folks, if I mung it too badly): Good explanations Terry! Even I could understand it :) If this is all reasonably accurate enough for everyone, could it be marked up for a FAQ or Handbook entry? As we move towards a 5-STABLE I think it would be useful to have a document to point to that describes these things. I'm happy to do the markup if you're happy for your words (or a close approximation) to be used. Thanks. Andrew. ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: libthr and 1:1 threading.
Matthew Dillon wrote: >Peter Wemm wrote: > :Terry Lambert wrote: > :> > No. It gives the ability for a thread to block on a syscall without > :> > stalling the entire system. Just try using mysqld on a system using libc_r > :> > and heavy disk IO. You can't select() on a read() from disk. Thats the > :> > ultimate reason to do it. The SMP parallelism is a bonus. > :> > :> Bug in FreeBSD's NBIO implementation. A read() that would result > :> in page-in needs to queue the request, but return EAGAIN to user > :> space to indicate the request cannot be satisfied. Making select() > :> come true for disk I/O after the fault is satisfied is a seperate > :> issue. Probably need to pass the fd all the way down. > : > :Umm Terry.. we have zero infrastructure to support this. > > It would be a very bad idea anyway. If someone is that dependant > on detecting page-fault or page-in behavior during I/O then they > ought to be using AIO (which does queue the request), not, read(), > or they should wire the memory in question. ??? I don't understand this statement. Specifically, we're not talking about a "dependency", we are talking about optimizing the number of stalls in that case that a page-in becomes necessary in uioread(), and it is a result of a read on a non-blocking file descriptor. The only thing really missing is knowing the original request was a read() of a non-blocking descriptor at the time you initiated the uioread() operation. That's one parameter to two functions, and a struct fileops flag, and a struct vnops flag additional to get the information to where it's needed. And then a flag to the underlying uioread "VM pig tricks" code to tell it to come back after triggering the fault. What am I missing? > I think I know what Terry wants... the best of both worlds when > faced with the classic performance tradeoff between a cached > synchronous operation and an asynchronous operation. Giving > read() + NBIO certain asynchronous characteristics solves the > performance problem but breaks the read() API (with or without > NBIO) in a major way. How does this break the read() API? The read() API, when called on a NBIO fd is *supposed* to return EAGAIN, if the request cannot be immediately satisfied, but could be satisfied later. Right now, it blocks. This looks like breakage of disk I/O introducing a stall, when socket I/O doesn't. If this breaks read() semantics, then socket I/O needs fixing to unbreak them, right? > A better solution would be to give AIO the capability to > operate synchronously if the operation would occur in a > non-blocking fashion (inclusive of blockages on page faults), > and asynchronously otherwise. This is a useful optimization, but it's different from what I want. What I want is disk I/O "in the pipeline" to not stall a read() request on an NBIO fd, the *same* way that network I/O "in the pipeline" doesn't stall a read() request on an NBIO fd. Does this make more sense? I actually think Peter definitely "gets" what I want; I just think he thinks it's harder to implement than it actually is: there's not a lot of "infrastructure" required, I think. -- Terry ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: libthr and 1:1 threading.
Peter Wemm wrote: > Terry Lambert wrote: > > Peter Wemm wrote: > > > No. It gives the ability for a thread to block on a syscall without > > > stalling the entire system. Just try using mysqld on a system using libc_r > > > and heavy disk IO. You can't select() on a read() from disk. Thats the > > > ultimate reason to do it. The SMP parallelism is a bonus. > > > > Bug in FreeBSD's NBIO implementation. A read() that would result > > in page-in needs to queue the request, but return EAGAIN to user > > space to indicate the request cannot be satisfied. Making select() > > come true for disk I/O after the fault is satisfied is a seperate > > issue. Probably need to pass the fd all the way down. > > Umm Terry.. we have zero infrastructure to support this. There are a couple of PROC_LOCK()/PROC_UNLOCK() pairs in trap_pfault(), and there's the translation of the fault for emulators, which isn't protected at all in trap(), but there's not really any proc references which are held for a long time in the fault handing path, at least for T_PAGEFLT. Hmm. The problem comes down to the vnops version of the struct fileops, which comes down to VOP_READ which comes down to ffs_read, which then falls down to "try get the data from the object using vm tricks" -- uioread(). Is ENABLE_VFS_IOOPT permanently disabled? Nope -- enabled; I see a prototype unconditionalized in uio.h. This doesn't look too hard to implement on a per struct fileops, per-VFS basis; it's not like there's sleeping on a process, rather than a vnode lock or anything; mostly everything is marked "GIANT_REQUIRED" after a certain point. Worst case, you could create a kernel-only thread pool in whose context you operated, after validating credentials (obviously). -- Terry ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: libthr and 1:1 threading.
That's a cute trick. The ultimate solution is to implement a semi-synchronous message passing API to replace the myrid system calls we have now. Roughly speaking, what the Amiga did for messages, ports, and I/O, is far superior then what is done in Linux and *BSD. You get the benefit of being able to operate syncnronously when possible, and having a convenient cup-holder for the operation state if you decide you have to 'block' the operation (instead of the state being strewn all over the call stack in a syscall implementation). Userland can decide whether to block or not block on an operation entirely independant of the OS deciding whether to block or not block on an operation. -Matt Matthew Dillon <[EMAIL PROTECTED]> :Without wanting to get too far off into the weeds, squid does something :interesting. They need to be able to nonblock for everything including :open(), read(), unlink(), readdir() etc. So what they do is implement a :fairly significant superset of the traditional AIO stuff using pthreads. It :seems to work pretty well for them, even with linuxthreads style threads. :Granted, squid's needs are not exactly typical. But I did want to point :out that a good part of the delays come not only from data IO but operations :like opening a file (pathname traversal), creating or removing a file, :reading a directory etc. This is a particular problem when the disk :is really busy. : :Cheers, :-Peter :-- :Peter Wemm - [EMAIL PROTECTED]; [EMAIL PROTECTED]; [EMAIL PROTECTED] :"All of this is for nothing if we don't go to the stars" - JMS/B5 ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: libthr and 1:1 threading.
Jeff Roberson wrote: > On Wed, 2 Apr 2003, Terry Lambert wrote: > > Peter Wemm wrote: > > > No. It gives the ability for a thread to block on a syscall without > > > stalling the entire system. Just try using mysqld on a system using libc_r > > > and heavy disk IO. You can't select() on a read() from disk. Thats the > > > ultimate reason to do it. The SMP parallelism is a bonus. > > > > Bug in FreeBSD's NBIO implementation. A read() that would result > > in page-in needs to queue the request, but return EAGAIN to user > > space to indicate the request cannot be satisfied. Making select() > > come true for disk I/O after the fault is satisfied is a seperate > > issue. Probably need to pass the fd all the way down. > > Please quote the standard that defines this behavior. This is entirely > counter to everything that I have ever read on the subject. There's no "standard" that specifies this behaviour, but there's code that defines it. I believe this is the way BSD 4.2 and 4.3 worked. You could maybe as Kirk about it, if you don't want to believe me on it. The thing that broke it was the switch to the vnode abstraction seperation from the fd, going to a vnode-based pager, and the switch to a unified VM and buffer cache. At that point, the non-blocking I/O flags weren't available to the pager to enable it to make a decision on whether or not to block the process pending completion of the paging operation. SVR4 and Solaris both support doing this; I used it to great effect in SVR4.0.2 in 1994 for NetWare for UNIX, at Novell. It's pretty clear from just thinking about the idea that it makes sense, if you are using non-blocking I/O, to start the operation and return early. It's the same effect that's exploited by having a seperate context to use for each blocking operation. -- Terry ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: libthr and 1:1 threading.
Matthew Dillon wrote: > A better solution would be to give AIO the capability to > operate synchronously if the operation would occur in a > non-blocking fashion (inclusive of blockages on page faults), > and asynchronously otherwise. Without wanting to get too far off into the weeds, squid does something interesting. They need to be able to nonblock for everything including open(), read(), unlink(), readdir() etc. So what they do is implement a fairly significant superset of the traditional AIO stuff using pthreads. It seems to work pretty well for them, even with linuxthreads style threads. Granted, squid's needs are not exactly typical. But I did want to point out that a good part of the delays come not only from data IO but operations like opening a file (pathname traversal), creating or removing a file, reading a directory etc. This is a particular problem when the disk is really busy. Cheers, -Peter -- Peter Wemm - [EMAIL PROTECTED]; [EMAIL PROTECTED]; [EMAIL PROTECTED] "All of this is for nothing if we don't go to the stars" - JMS/B5 ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: libthr and 1:1 threading.
:Terry Lambert wrote: :> Peter Wemm wrote: :> > No. It gives the ability for a thread to block on a syscall without :> > stalling the entire system. Just try using mysqld on a system using libc_r :> > and heavy disk IO. You can't select() on a read() from disk. Thats the :> > ultimate reason to do it. The SMP parallelism is a bonus. :> :> Bug in FreeBSD's NBIO implementation. A read() that would result :> in page-in needs to queue the request, but return EAGAIN to user :> space to indicate the request cannot be satisfied. Making select() :> come true for disk I/O after the fault is satisfied is a seperate :> issue. Probably need to pass the fd all the way down. : :Umm Terry.. we have zero infrastructure to support this. : :Cheers, :-Peter :-- :Peter Wemm - [EMAIL PROTECTED]; [EMAIL PROTECTED]; [EMAIL PROTECTED] :"All of this is for nothing if we don't go to the stars" - JMS/B5 It would be a very bad idea anyway. If someone is that dependant on detecting page-fault or page-in behavior during I/O then they ought to be using AIO (which does queue the request), not, read(), or they should wire the memory in question. I think I know what Terry wants... the best of both worlds when faced with the classic performance tradeoff between a cached synchronous operation and an asynchronous operation. Giving read() + NBIO certain asynchronous characteristics solves the performance problem but breaks the read() API (with or without NBIO) in a major way. A better solution would be to give AIO the capability to operate synchronously if the operation would occur in a non-blocking fashion (inclusive of blockages on page faults), and asynchronously otherwise. -Matt Matthew Dillon <[EMAIL PROTECTED]> ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: libthr and 1:1 threading.
On Wed, Apr 02, 2003 at 06:37:21PM -0500, Daniel Eischen wrote: > On Wed, 2 Apr 2003, Peter Wemm wrote: > > > Terry Lambert wrote: > > > > > KSE mailing list, starting Monday or so: > > > ] We still haven't heard from jeff with regard to the process > > > ] signal mask removal. > > > > We can add new mailing lists really easily now - it takes about 20-30 seconds. > > Would it be worth adding a freebsd-threads and/or freebsd-kse type list > > where it is a bit higher profile? > > That's up to the folks here (on the KSE list) I guess. > Is it possible to make it non-public? The nice thing > about the current kse list is it's relatively low > volume and lack of spam. That kind of flies in the face of the way we do things. I would imagine if nothing else, being able to read the archives would be a good thing. -gordon pgp0.pgp Description: PGP signature
Re: libthr and 1:1 threading.
Terry Lambert wrote: > Peter Wemm wrote: > > No. It gives the ability for a thread to block on a syscall without > > stalling the entire system. Just try using mysqld on a system using libc_r > > and heavy disk IO. You can't select() on a read() from disk. Thats the > > ultimate reason to do it. The SMP parallelism is a bonus. > > Bug in FreeBSD's NBIO implementation. A read() that would result > in page-in needs to queue the request, but return EAGAIN to user > space to indicate the request cannot be satisfied. Making select() > come true for disk I/O after the fault is satisfied is a seperate > issue. Probably need to pass the fd all the way down. Umm Terry.. we have zero infrastructure to support this. Cheers, -Peter -- Peter Wemm - [EMAIL PROTECTED]; [EMAIL PROTECTED]; [EMAIL PROTECTED] "All of this is for nothing if we don't go to the stars" - JMS/B5 ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: libthr and 1:1 threading.
On Wed, 2 Apr 2003, Terry Lambert wrote: > Peter Wemm wrote: > > No. It gives the ability for a thread to block on a syscall without > > stalling the entire system. Just try using mysqld on a system using libc_r > > and heavy disk IO. You can't select() on a read() from disk. Thats the > > ultimate reason to do it. The SMP parallelism is a bonus. > > Bug in FreeBSD's NBIO implementation. A read() that would result > in page-in needs to queue the request, but return EAGAIN to user > space to indicate the request cannot be satisfied. Making select() > come true for disk I/O after the fault is satisfied is a seperate > issue. Probably need to pass the fd all the way down. > Please quote the standard that defines this behavior. This is entirely counter to everything that I have ever read on the subject. ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: libthr and 1:1 threading.
On Wed, 2 Apr 2003, Peter Wemm wrote: > Terry Lambert wrote: > > > KSE mailing list, starting Monday or so: > > ] We still haven't heard from jeff with regard to the process > > ] signal mask removal. > > We can add new mailing lists really easily now - it takes about 20-30 seconds. > Would it be worth adding a freebsd-threads and/or freebsd-kse type list > where it is a bit higher profile? That's up to the folks here (on the KSE list) I guess. Is it possible to make it non-public? The nice thing about the current kse list is it's relatively low volume and lack of spam. -- Dan Eischen ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: libthr and 1:1 threading.
Peter Wemm wrote: > No. It gives the ability for a thread to block on a syscall without > stalling the entire system. Just try using mysqld on a system using libc_r > and heavy disk IO. You can't select() on a read() from disk. Thats the > ultimate reason to do it. The SMP parallelism is a bonus. Bug in FreeBSD's NBIO implementation. A read() that would result in page-in needs to queue the request, but return EAGAIN to user space to indicate the request cannot be satisfied. Making select() come true for disk I/O after the fault is satisfied is a seperate issue. Probably need to pass the fd all the way down. -- Terry ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: libthr and 1:1 threading.
Jeff Roberson wrote: > Perhaps I should start quoting posix. I wonder what my legal rights > are given the copyright. hm.. Educational use. FWIW, my reading of POSIX.1 says "Per process mask, per threads masks". The real question is "What happens when I kill -9/-15 a libthr process with a lot of threads?". Yeah, the KSE people are sorting out signals, too, but they were quite upset with the change, from their perspective, to the KSE kernel API semantics. Maybe they just felt rushed? Dunno... which is why I asked for an ETA: I think you should be talking to them. -- Terry ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: libthr and 1:1 threading.
On Wed, 2 Apr 2003, Peter Wemm wrote: :Terry Lambert wrote: : :> KSE mailing list, starting Monday or so: :> ] We still haven't heard from jeff with regard to the process :> ] signal mask removal. : :We can add new mailing lists really easily now - it takes about 20-30 seconds. :Would it be worth adding a freebsd-threads and/or freebsd-kse type list :where it is a bit higher profile? : :Cheers, :-Peter :-- I would like this. I seem to remember a mail awhile back stating that there was a 'secret' list going on for thread discussion. Since this is a big part of -CURRENT (and has been, duh), it's sorta scary to have it be hidden. Did I miss a message regarding how to sign up for this? -- Andrew R. Reiter [EMAIL PROTECTED] [EMAIL PROTECTED] ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: libthr and 1:1 threading.
Terry Lambert wrote: > KSE mailing list, starting Monday or so: > ] We still haven't heard from jeff with regard to the process > ] signal mask removal. We can add new mailing lists really easily now - it takes about 20-30 seconds. Would it be worth adding a freebsd-threads and/or freebsd-kse type list where it is a bit higher profile? Cheers, -Peter -- Peter Wemm - [EMAIL PROTECTED]; [EMAIL PROTECTED]; [EMAIL PROTECTED] "All of this is for nothing if we don't go to the stars" - JMS/B5 ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: libthr and 1:1 threading.
Juli Mallett wrote: > * De: Jeff Roberson <[EMAIL PROTECTED]> [ Data: 2003-04-02 ] > > On Wed, 2 Apr 2003, Terry Lambert wrote: > > > Also, any ETA on the per process signal mask handing bug in > > > libthr? Might not be safe to convert everything up front, in > > > a rush of eager enthusiasm... > > > > Which bug is that? I'm not aware of it. > > I think Terry is referring to the Uncertainty & Doubt as if it were > a bug over the lack of a process sigmask (moved into the threads), > as raised by the M:N group. Yes. This is the issue in question. -- Terry ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: libthr and 1:1 threading.
Jeff Roberson wrote: > On Wed, 2 Apr 2003, Terry Lambert wrote: > > Is the disk I/O really that big of an issue? All writes will > > be on underlying non-blocking descriptors; I guess you are > > saying that the interleaved I/O is more important, further > > down the system call interface than the top, and this becomes > > an issue? > > UNIX distinguishes between short term and long term blocking calls. Non > blocking io only avoids blocking for indeterminate periods of time. The > 20ms or so you're paying per IO on an IDE system is 20ms that no other > threads can run. This makes a difference. See other message. The fault should be tripped, but the process should not wait around for the result, if it was tripped as the result of a read on an async fd. Instead an EAGAIN should go to user space, and that 20ms should be used for something else in user space, while the drive gets busy with the DMA. This is arguably a bug in FreeBSD's handling of demand faults on non-blocking fd's, compared to SVR4 and Solaris' handling. > > It seems to me that maybe the correct fix for this is to use > > AIO instead of non-blocking I/O, then? > > Our AIO implementation needs some rethinking. I'm not sure how > effecient it actually is. I will guess it suffers from the same "faulting == halting" problem... > > Also, any ETA on the per process signal mask handing bug in > > libthr? Might not be safe to convert everything up front, in > > a rush of eager enthusiasm... > > Which bug is that? I'm not aware of it. KSE mailing list, starting Monday or so: ] We still haven't heard from jeff with regard to the process ] signal mask removal. -- Terry ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: libthr and 1:1 threading.
On Wed, 2 Apr 2003, Daniel Eischen wrote: > [ CC list trimmed somewhat ] > > On Wed, 2 Apr 2003, Jeff Roberson wrote: > > On Wed, 2 Apr 2003, Daniel Eischen wrote: > > > On Wed, 2 Apr 2003, Jeff Roberson wrote: > > > > > > > > Then set the mask to be the same on all threads in the process. The mask > > > > is set in swapcontext though so it seems reasonable to me that it is > > > > atomically updated when you schedule a new user thread on a kse. > > > > > > Jeff, are you _listening_ to us? We've said multiple times > > > that the UTS does not enter the kernel when performing thread > > > switches. The UTS does NOT use setcontext(), getcontext(), > > > or swapcontext(). > > > > I had not seen anyone mention this. > > This is only the 3rd or 4th time that I've sent or CC'd you > with email on the topic, most probably on the KSE list. I > know that Julian has sent several also. You must be filtering > us :-) It depends on where things have been CC'd. I should probably adjust my filter so that things sent directly to me hit my inbox only. I have been quite overloaded with mail lately. Both from FreeBSD and other endeavours. I apologize, it must have been frustrating to tell me so many times. > > If this is the case then I suggest > > the masks and pending sets be kept in user space. You can install blank > > handlers for everything so that they are kept pending until the uts has a > > chance to pick them up in the upcall. > > Well, they are kept in user-space. The kse mailbox has the > currently running thread's signal mask. The thread structure > also has a pending signal set, and there is also a global > pending signal set for signals pending on the process. > It's just that the kernel doesn't know about it. > > > If you really want a process wide mask allow me to do it. The single code > > is quite tricky and it's already been butchered enough. I think we should > > discuss this a bit more first though. > > Sure, we're discussing signal stuff on the kse list now. > Are you still monitoring that list? Perhaps that's > partially why you haven't seen any of our email? I'll go look over it soon. Sorry for the misunderstanding. > -- > Dan Eischen > > ___ > [EMAIL PROTECTED] mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-current > To unsubscribe, send any mail to "[EMAIL PROTECTED]" > ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: libthr and 1:1 threading.
[ CC list trimmed somewhat ] On Wed, 2 Apr 2003, Jeff Roberson wrote: > On Wed, 2 Apr 2003, Daniel Eischen wrote: > > On Wed, 2 Apr 2003, Jeff Roberson wrote: > > > > > > Then set the mask to be the same on all threads in the process. The mask > > > is set in swapcontext though so it seems reasonable to me that it is > > > atomically updated when you schedule a new user thread on a kse. > > > > Jeff, are you _listening_ to us? We've said multiple times > > that the UTS does not enter the kernel when performing thread > > switches. The UTS does NOT use setcontext(), getcontext(), > > or swapcontext(). > > I had not seen anyone mention this. This is only the 3rd or 4th time that I've sent or CC'd you with email on the topic, most probably on the KSE list. I know that Julian has sent several also. You must be filtering us :-) > If this is the case then I suggest > the masks and pending sets be kept in user space. You can install blank > handlers for everything so that they are kept pending until the uts has a > chance to pick them up in the upcall. Well, they are kept in user-space. The kse mailbox has the currently running thread's signal mask. The thread structure also has a pending signal set, and there is also a global pending signal set for signals pending on the process. It's just that the kernel doesn't know about it. > If you really want a process wide mask allow me to do it. The single code > is quite tricky and it's already been butchered enough. I think we should > discuss this a bit more first though. Sure, we're discussing signal stuff on the kse list now. Are you still monitoring that list? Perhaps that's partially why you haven't seen any of our email? -- Dan Eischen ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: libthr and 1:1 threading.
John Baldwin wrote: > > On 02-Apr-2003 Terry Lambert wrote: > > The only way I see for disk I/O to be involved in Mozilla is in > > local cache? You can turn that off. > > Umm, the idea here is to actually make threaded programs > _useful_. Not to require that you trim their functionality > down before we handle them in a sane way. Less FUD more signal > please. Just a reminder, we can selectively moderate folks with the new mailing list software.. Cheers, -Peter -- Peter Wemm - [EMAIL PROTECTED]; [EMAIL PROTECTED]; [EMAIL PROTECTED] "All of this is for nothing if we don't go to the stars" - JMS/B5 ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: libthr and 1:1 threading.
On 02-Apr-2003 Terry Lambert wrote: > The only way I see for disk I/O to be involved in Mozilla is in > local cache? You can turn that off. Umm, the idea here is to actually make threaded programs _useful_. Not to require that you trim their functionality down before we handle them in a sane way. Less FUD more signal please. -- John Baldwin <[EMAIL PROTECTED]> <>< http://www.FreeBSD.org/~jhb/ "Power Users Use the Power to Serve!" - http://www.FreeBSD.org/ ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: libthr and 1:1 threading.
On Wed, 2 Apr 2003, Daniel Eischen wrote: > On Wed, 2 Apr 2003, Jeff Roberson wrote: > > On Wed, 2 Apr 2003, Julian Elischer wrote: > > > > > > > > > > > On Wed, 2 Apr 2003, Juli Mallett wrote: > > > > > > > * De: Jeff Roberson <[EMAIL PROTECTED]> [ Data: 2003-04-02 ] > > > > [ Subjecte: Re: libthr and 1:1 threading. ] > > > > > On Wed, 2 Apr 2003, Terry Lambert wrote: > > > > > > Also, any ETA on the per process signal mask handing bug in > > > > > > libthr? Might not be safe to convert everything up front, in > > > > > > a rush of eager enthusiasm... > > > > > > > > > > Which bug is that? I'm not aware of it. > > > > > > > > I think Terry is referring to the Uncertainty & Doubt as if it were > > > > a bug over the lack of a process sigmask (moved into the threads), > > > > as raised by the M:N group. > > > > > > I think this IS a problem. We need a per-process mask. > > > to block signals that no thread is interested in. > > > Since M:N threads do not have a kernel thread for each userland thread, > > > there is nowhere to store this info any more. > > > > > > > Then set the mask to be the same on all threads in the process. The mask > > is set in swapcontext though so it seems reasonable to me that it is > > atomically updated when you schedule a new user thread on a kse. > > Jeff, are you _listening_ to us? We've said multiple times > that the UTS does not enter the kernel when performing thread > switches. The UTS does NOT use setcontext(), getcontext(), > or swapcontext(). I had not seen anyone mention this. If this is the case then I suggest the masks and pending sets be kept in user space. You can install blank handlers for everything so that they are kept pending until the uts has a chance to pick them up in the upcall. If you really want a process wide mask allow me to do it. The single code is quite tricky and it's already been butchered enough. I think we should discuss this a bit more first though. Cheers, Jeff ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: libthr and 1:1 threading.
Robert Watson wrote: > On Wed, 2 Apr 2003, Terry Lambert wrote: > > Is the disk I/O really that big of an issue? All writes will be on > > underlying non-blocking descriptors; I guess you are saying that the > > interleaved I/O is more important, further down the system call > > interface than the top, and this becomes an issue? > > The I/O issue is a big deal for things like mysql, yes. I'm not interested in the MySQL code; I've been in that threads code, deep (made it work on AIX). There are a lot better ways to deal with MySQL's latency issues. For example, the two phase commit stall that it has is not necessary on a Soft Updates FS, but, like qmail, it does it anyway, introducing latency. But Mozilla... the only issue I could see is interleaved network I/O, but that should not be an issue; HTTP is request/response, with the lion's share of the data coming as a result of the response. In other words, the rendering engine's going to have to wait for data back from the remote server, and that's got to be the primary latency. The only way I see for disk I/O to be involved in Mozilla is in local cache? You can turn that off. > > It seems to me that maybe the correct fix for this is to use AIO > > instead of non-blocking I/O, then? > > Well, they're both fixes. Another issue for applications that are > threaded and may be bumping up against the system memory limits is whether > or not the whole process stalls on a page fault or memory mapping fault, > or whether it's just the thread. This is what I meant by "deeper in the system calll layer". IMO, if you are stalled on something like this on an async fd, then it should queue the fault anyway, and return to user space for the next request. This may just be a bug in the kernel processing of demand faults on vnodes associated with async fd's (FWIW, System V and Solaris both queue the fault for kernel processing, and then return to user space). > If you have an application that is accessing a large memory mapped > file, there may be some long kernel sleeps as you pull in the pages. Again, this stall, if the fd in question is async, should be taken as system time, not as an application stall. That's a little harder, in this case, because you want to fail the pagein with the moral equivalent of "EAGAIN" plus a forced threads context switch after queing the fault request. It's a lot easier if we are talking about an explicit read() request, and not a fault, (obviously 8-)), since it should be expecting the possibility of an EAGAIN in the read() case. I guess, in this case, without an N:M upcall to indicate the forced context switch, you would just stall. This, though, doesn't seem to be an issue for Mozilla, to me. > Certainly, you can argue that the application should be structured > to make all I/O explicit and asynchronous, but for various reasons, > that's not the case :-). The mmap'ed file case is obviously not something that can be handled without an explicit contract between user and kernel for notification of the pagein temporary failure (I would use a signal for that, probably, as a gross first approximation, but per-process signal handling is currently not happy...). > Our VM and VFS subsystems may have limited concurrency from > an SMPng perspective, but probably have enough that a marked > benefit should be seen there too (you might have to wait for > another thread to block in the subsystem, but that will be a > short period of time compared to how long it takes to service > the page from disk). I would argue that this was in error, at least for explicit handing of non-blocking I/O. SVR4 and Solaris don't suffer from a stall, in the face of a missing page in a demand page-in on an async fd. The process of servicing the page-in is really independent of the process of retrying the call to see if the page is there yet. I think this is probably classifiable as a serious deficiency in the FreeBSD VM page handling, and in the decoupling of the fd and the VM backing object. I'm not sure how I'd go about fixing it; you'd have to pass the fd struct down as well, to get the flags, I think. Actually, people have wanted this for a while, in order to properly support per open instance data on cloned devices (e.g. this is needed in order to support multiple instances of VMWare on a single FreeBSD host). > > The GUI thread issues are something I hadn't considered; I don't > > generally think of user space CPU intensive operations like that, > > but I guess it has to be rendered some time. 8-). > > One of the problems I've run into is where you lose interactivity during > file saves and other disk-intensive operations in OpenOffice. Other > windows could in theory still be processing UI events, such as menu > clicks, etc, but since you're dumping several megabytes of data to disk or > doing interactive file operations that require waiting on disk latency, > you end up with a fairly nasty user experience. One way to explore this
Re: libthr and 1:1 threading.
On Wed, 2 Apr 2003, Jeff Roberson wrote: > On Wed, 2 Apr 2003, Julian Elischer wrote: > > > > > > > On Wed, 2 Apr 2003, Juli Mallett wrote: > > > > > * De: Jeff Roberson <[EMAIL PROTECTED]> [ Data: 2003-04-02 ] > > > [ Subjecte: Re: libthr and 1:1 threading. ] > > > > On Wed, 2 Apr 2003, Terry Lambert wrote: > > > > > Also, any ETA on the per process signal mask handing bug in > > > > > libthr? Might not be safe to convert everything up front, in > > > > > a rush of eager enthusiasm... > > > > > > > > Which bug is that? I'm not aware of it. > > > > > > I think Terry is referring to the Uncertainty & Doubt as if it were > > > a bug over the lack of a process sigmask (moved into the threads), > > > as raised by the M:N group. > > > > I think this IS a problem. We need a per-process mask. > > to block signals that no thread is interested in. > > Since M:N threads do not have a kernel thread for each userland thread, > > there is nowhere to store this info any more. > > > > Then set the mask to be the same on all threads in the process. The mask > is set in swapcontext though so it seems reasonable to me that it is > atomically updated when you schedule a new user thread on a kse. Jeff, are you _listening_ to us? We've said multiple times that the UTS does not enter the kernel when performing thread switches. The UTS does NOT use setcontext(), getcontext(), or swapcontext(). -- Dan Eischen ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: libthr and 1:1 threading.
On Wed, 2 Apr 2003, Julian Elischer wrote: > > > On Wed, 2 Apr 2003, Juli Mallett wrote: > > > * De: Jeff Roberson <[EMAIL PROTECTED]> [ Data: 2003-04-02 ] > > [ Subjecte: Re: libthr and 1:1 threading. ] > > > On Wed, 2 Apr 2003, Terry Lambert wrote: > > > > Also, any ETA on the per process signal mask handing bug in > > > > libthr? Might not be safe to convert everything up front, in > > > > a rush of eager enthusiasm... > > > > > > Which bug is that? I'm not aware of it. > > > > I think Terry is referring to the Uncertainty & Doubt as if it were > > a bug over the lack of a process sigmask (moved into the threads), > > as raised by the M:N group. > > I think this IS a problem. We need a per-process mask. > to block signals that no thread is interested in. > Since M:N threads do not have a kernel thread for each userland thread, > there is nowhere to store this info any more. > Then set the mask to be the same on all threads in the process. The mask is set in swapcontext though so it seems reasonable to me that it is atomically updated when you schedule a new user thread on a kse. > I'd be happy to have it be a per-ksegrp mask actually.. > (to help deliver the signal to the right group to lower the interaction > between UTS's in different groups.) > ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: libthr and 1:1 threading.
Terry Lambert wrote: leafy wrote: On Wed, Apr 02, 2003 at 07:38:14AM -0800, Terry Lambert wrote: Has anyone tried compiling X11 to use libthr? Someone reported success with KDE, so it should serve as a sign of working X11. Not X11 clients. The X11 server. -- Terry ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "[EMAIL PROTECTED]" The X server is not threaded: [junior] ~> ldd /usr/X11R6/bin/X /usr/X11R6/bin/X: libc.so.5 => /usr/lib/libc.so.5 (0x2806a000) That was supposed to be one of the great goals of X11R6, but it was never completed, and I believe that most of the (incomplete) work has long since been removed from XFree86. Scott ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: libthr and 1:1 threading.
On 02-Apr-2003 Terry Lambert wrote: > leafy wrote: >> On Wed, Apr 02, 2003 at 07:38:14AM -0800, Terry Lambert wrote: >> > Has anyone tried compiling X11 to use libthr? >> >> Someone reported success with KDE, so it should serve as a sign of working X11. > > Not X11 clients. > > The X11 server. Gee, I wonder if they like ran it locally on a local server? -- John Baldwin <[EMAIL PROTECTED]> <>< http://www.FreeBSD.org/~jhb/ "Power Users Use the Power to Serve!" - http://www.FreeBSD.org/ ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: libthr and 1:1 threading.
* De: Terry Lambert <[EMAIL PROTECTED]> [ Data: 2003-04-02 ] [ Subjecte: Re: libthr and 1:1 threading. ] > leafy wrote: > > On Wed, Apr 02, 2003 at 07:38:14AM -0800, Terry Lambert wrote: > > > Has anyone tried compiling X11 to use libthr? > > > > Someone reported success with KDE, so it should serve as a sign of working X11. > > Not X11 clients. > > The X11 server. In any case, "compiling" there is bogus. Just replace (symlink) lc_r. -- juli mallett. email: [EMAIL PROTECTED]; aim: bsdflata; efnet: juli; ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: libthr and 1:1 threading.
leafy wrote: > On Wed, Apr 02, 2003 at 07:38:14AM -0800, Terry Lambert wrote: > > Has anyone tried compiling X11 to use libthr? > > Someone reported success with KDE, so it should serve as a sign of working X11. Not X11 clients. The X11 server. -- Terry ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: libthr and 1:1 threading.
On Wed, 2 Apr 2003, Jeff Roberson wrote: > On Wed, 2 Apr 2003, Juli Mallett wrote: > > > * De: Jeff Roberson <[EMAIL PROTECTED]> [ Data: 2003-04-02 ] > > [ Subjecte: Re: libthr and 1:1 threading. ] > > > On Wed, 2 Apr 2003, Terry Lambert wrote: > > > > Also, any ETA on the per process signal mask handing bug in > > > > libthr? Might not be safe to convert everything up front, in > > > > a rush of eager enthusiasm... > > > > > > Which bug is that? I'm not aware of it. > > > > I think Terry is referring to the Uncertainty & Doubt as if it were > > a bug over the lack of a process sigmask (moved into the threads), > > as raised by the M:N group. > > POSIX specifically says that the signal mask is per thread. I'd be very > surprised if the 1:1 sigmask/sigpending stuff was wrong. I don't think > signal handling in M:N has really been totally worked out. Their concerns > were more like 'how do we do this given the new signal restructuring' > > Perhaps I should start quoting posix. I wonder what my legal rights > are given the copyright. hm.. Posix defines the interface at the 'top' of the library. In M:N threads, the process gets signals that are not blocked by any thread. There may be 10,000 threads, only 3 of which the kernel has any knowledge. The signal needs to be checked againsta a per-process mask before being routed to the UTS for forwarding to the apropriate thread. To achieve that efficiently we will have a per-process mask, and use that. 1:1 threads (and non-threaded processes) will continue to use the per-thread mask. On switching to M:N mode, we'll just clone the active thread's mask into the process mask. No biggie.. just needs code :-) When M:N threads are involved psignal() will just check against a different mask, and delivery is already handled in a different manner. The UTS will update the per-process mask as needed as only it knows what all the threads are masking (since the kernel doesn't even know what threads exist). ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: libthr and 1:1 threading.
On Wed, 2 Apr 2003, Juli Mallett wrote: > * De: Jeff Roberson <[EMAIL PROTECTED]> [ Data: 2003-04-02 ] > [ Subjecte: Re: libthr and 1:1 threading. ] > > On Wed, 2 Apr 2003, Terry Lambert wrote: > > > Also, any ETA on the per process signal mask handing bug in > > > libthr? Might not be safe to convert everything up front, in > > > a rush of eager enthusiasm... > > > > Which bug is that? I'm not aware of it. > > I think Terry is referring to the Uncertainty & Doubt as if it were > a bug over the lack of a process sigmask (moved into the threads), > as raised by the M:N group. I think this IS a problem. We need a per-process mask. to block signals that no thread is interested in. Since M:N threads do not have a kernel thread for each userland thread, there is nowhere to store this info any more. I'd be happy to have it be a per-ksegrp mask actually.. (to help deliver the signal to the right group to lower the interaction between UTS's in different groups.) ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: libthr and 1:1 threading.
Sheldon Hearn wrote: > On (2003/04/02 07:38), Terry Lambert wrote: > > Is the disk I/O really that big of an issue? All writes will > > be on underlying non-blocking descriptors; I guess you are > > saying that the interleaved I/O is more important, further > > down the system call interface than the top, and this becomes > > an issue? > > Dude, you should really try this stuff for yourself before naysaying > performance improvements on principle. It's actually quite impressive > for desktop users (at least). I have. I can't tell if it's the scheduler quantums or the concurrency from the threads. I'm going to have to specifically write code to find out, and it may take me a while to do it; I have to figure out a way to put the user space stalls back for descriptor accesses, so the tests run on an equal footing. Right now, I have to decide whether it's worth the hassle of combining the libc_r and libthr code to do that, or if I should just drop it, and let you guys turn FreeBSD's threads into Linux. PS: My gut tells me it's not the concurrency; the resolver is the bottleneck for things like Mozilla (IMO), and it still has to stall concurrency. PPS: I'll get back to you after I size the job, and decide. -- Terry ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: libthr and 1:1 threading.
On Wed, 2 Apr 2003, Jeff Roberson wrote: > On Wed, 2 Apr 2003, Juli Mallett wrote: > > > * De: Jeff Roberson <[EMAIL PROTECTED]> [ Data: 2003-04-02 ] > > [ Subjecte: Re: libthr and 1:1 threading. ] > > > On Wed, 2 Apr 2003, Terry Lambert wrote: > > > > Also, any ETA on the per process signal mask handing bug in > > > > libthr? Might not be safe to convert everything up front, in > > > > a rush of eager enthusiasm... > > > > > > Which bug is that? I'm not aware of it. > > > > I think Terry is referring to the Uncertainty & Doubt as if it were > > a bug over the lack of a process sigmask (moved into the threads), > > as raised by the M:N group. > > POSIX specifically says that the signal mask is per thread. I'd be very Yes, but you've moved the mask into the kernel thread, whereas in libkse threads are in the UTS. Mulitple threads can be run in one kernel thread, and the UTS doesn't want to enter the kernel to set the signal mask. -- Dan Eischen ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: libthr and 1:1 threading.
Terry Lambert wrote: > Jun Su wrote: > > > [ ... 1:1 kernel threads implementation ... ] > > > > A benchmark would be interested. > > This request doesn't make sense. > > The primary performance reasoning behind a 1:1 kernel threading > implementation, relative to the user space single kernel entry > scheduler in the libc_r implementation is SMP scalability for > threaded applications. No. It gives the ability for a thread to block on a syscall without stalling the entire system. Just try using mysqld on a system using libc_r and heavy disk IO. You can't select() on a read() from disk. Thats the ultimate reason to do it. The SMP parallelism is a bonus. Cheers, -Peter -- Peter Wemm - [EMAIL PROTECTED]; [EMAIL PROTECTED]; [EMAIL PROTECTED] "All of this is for nothing if we don't go to the stars" - JMS/B5 ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: libthr and 1:1 threading.
On Wed, 2 Apr 2003, Juli Mallett wrote: > * De: Jeff Roberson <[EMAIL PROTECTED]> [ Data: 2003-04-02 ] > [ Subjecte: Re: libthr and 1:1 threading. ] > > On Wed, 2 Apr 2003, Terry Lambert wrote: > > > Also, any ETA on the per process signal mask handing bug in > > > libthr? Might not be safe to convert everything up front, in > > > a rush of eager enthusiasm... > > > > Which bug is that? I'm not aware of it. > > I think Terry is referring to the Uncertainty & Doubt as if it were > a bug over the lack of a process sigmask (moved into the threads), > as raised by the M:N group. POSIX specifically says that the signal mask is per thread. I'd be very surprised if the 1:1 sigmask/sigpending stuff was wrong. I don't think signal handling in M:N has really been totally worked out. Their concerns were more like 'how do we do this given the new signal restructuring' Perhaps I should start quoting posix. I wonder what my legal rights are given the copyright. hm.. Cheers, Jeff ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: libthr and 1:1 threading.
* De: Jeff Roberson <[EMAIL PROTECTED]> [ Data: 2003-04-02 ] [ Subjecte: Re: libthr and 1:1 threading. ] > On Wed, 2 Apr 2003, Terry Lambert wrote: > > Also, any ETA on the per process signal mask handing bug in > > libthr? Might not be safe to convert everything up front, in > > a rush of eager enthusiasm... > > Which bug is that? I'm not aware of it. I think Terry is referring to the Uncertainty & Doubt as if it were a bug over the lack of a process sigmask (moved into the threads), as raised by the M:N group. -- juli mallett. email: [EMAIL PROTECTED]; aim: bsdflata; efnet: juli; ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: libthr and 1:1 threading.
On Wed, 2 Apr 2003, Terry Lambert wrote: > Robert Watson wrote: > > You should notice marked interactivity and UI latency improvements with > > threaded GUI apps over libc_r because GUI threads will generally no longer > > be blocked when disk I/O and blocking I/O occurs. For example, > > applications like Open Office, Netscape, et al, really get a lot better > > with 1:1. Likewise, non-interactive applications that are disk > > I/O-intensive, such as mysql, will also perform substantially better > > because a thread that hits blocking using an interface that doesn't > > support non-blocking I/O (such as the file system) won't clog up the > > application. > > Is the disk I/O really that big of an issue? All writes will > be on underlying non-blocking descriptors; I guess you are > saying that the interleaved I/O is more important, further > down the system call interface than the top, and this becomes > an issue? UNIX distinguishes between short term and long term blocking calls. Non blocking io only avoids blocking for indeterminate periods of time. The 20ms or so you're paying per IO on an IDE system is 20ms that no other threads can run. This makes a difference. > It seems to me that maybe the correct fix for this is to use > AIO instead of non-blocking I/O, then? Our AIO implementation needs some rethinking. I'm not sure how effecient it actually is. > The GUI thread issues are something I hadn't considered; I > don't generally think of user space CPU intensive operations > like that, but I guess it has to be rendered some time. 8-). > > Has anyone tried compiling X11 to use libthr? > > Also, any ETA on the per process signal mask handing bug in > libthr? Might not be safe to convert everything up front, in > a rush of eager enthusiasm... Which bug is that? I'm not aware of it. ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: libthr and 1:1 threading.
> > You should notice marked interactivity and UI latency improvements with > > threaded GUI apps over libc_r because GUI threads will generally no longer > > be blocked when disk I/O and blocking I/O occurs. For example, > > applications like Open Office, Netscape, et al, really get a lot better > > with 1:1. Likewise, non-interactive applications that are disk > > I/O-intensive, such as mysql, will also perform substantially better > > because a thread that hits blocking using an interface that doesn't > > support non-blocking I/O (such as the file system) won't clog up the > > application. > > Is the disk I/O really that big of an issue? In my experience, yes. Disk I/O is blocking, since you can't reliably use non-blocking descriptors for disk I/O, since you don't know if it will block, and for how long. Again, in my experience, it was much easier to setup non-blocking networking descriptors, since unlike disk I/O, you needed to deal with things not get written at the application. With disk I/O it was much harder to determine what if anything got written if you did non-blocking writes, and if you needed things written both effeciently and reliably to disk, you *have* to know how much was written successfully, and byte-at-a-time I/O is simply un-acceptably slow. I'm sure others have run into this problem as well. > It seems to me that maybe the correct fix for this is to use > AIO instead of non-blocking I/O, then? AIO is non-portable. Nate ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: libthr and 1:1 threading.
On Wed, 2 Apr 2003, Terry Lambert wrote: > Is the disk I/O really that big of an issue? All writes will be on > underlying non-blocking descriptors; I guess you are saying that the > interleaved I/O is more important, further down the system call > interface than the top, and this becomes an issue? The I/O issue is a big deal for things like mysql, yes. > It seems to me that maybe the correct fix for this is to use AIO instead > of non-blocking I/O, then? Well, they're both fixes. Another issue for applications that are threaded and may be bumping up against the system memory limits is whether or not the whole process stalls on a page fault or memory mapping fault, or whether it's just the thread. If you have an application that is accessing a large memory mapped file, there may be some long kernel sleeps as you pull in the pages. Certainly, you can argue that the application should be structured to make all I/O explicit and asynchronous, but for various reasons, that's not the case :-). Our VM and VFS subsystems may have limited concurrency from an SMPng perspective, but probably have enough that a marked benefit should be seen there too (you might have to wait for another thread to block in the subsystem, but that will be a short period of time compared to how long it takes to service the page from disk). > The GUI thread issues are something I hadn't considered; I don't > generally think of user space CPU intensive operations like that, but I > guess it has to be rendered some time. 8-). One of the problems I've run into is where you lose interactivity during file saves and other disk-intensive operations in OpenOffice. Other windows could in theory still be processing UI events, such as menu clicks, etc, but since you're dumping several megabytes of data to disk or doing interactive file operations that require waiting on disk latency, you end up with a fairly nasty user experience. One way to explore this effect is to do a side-by-side comparison of the behavior of OpenOffice and Mozilla linked against libc_r and linuxthreads. I haven't actually instrumented the kernel, but it might be quite interesting to do so--attempt to estimate the total impact of disk stalls on libc_r. From a purely qualitivative perspective, there is quite a noticeable difference. > Has anyone tried compiling X11 to use libthr? Not sure. > Also, any ETA on the per process signal mask handing bug in libthr? > Might not be safe to convert everything up front, in a rush of eager > enthusiasm... Can't speculate on that, except one thing that is useful to note is that many serious threaded applications are already being linked against linuxthreads on FreeBSD, which arguably has poorer semantics when it comes to signals, credentials, etc, than libthr already :-). For example, most sites I've talked to that deploy mysql do it with linuxthreads rather than libc_r to avoid the I/O issues, as well as avoid deadlocks. There are enough bits of the kernel (for example, POSIX fifos) that don't handle non-blocking operation that libc_r can stall or get into I/O buffer deadlocks. I seem to recall someone mentioning (but can't confirm easily) that Netscape at one point relied on using pipes to handle some sorts of asynchronous events and wakeups within the same process. If that pipe filled, the process would block on a pipe write for a pipe that would never drain. I can think of a couple of other interesting excercises to explore the problem -- implementing AIO "better" using the KSE primitives mixed between userspace and kernel, reimplementing libc_r to attempt to use AIO rather than a select loop where possible, etc. It might be quite interesting to see whether (for a bounded number of threads, due to our AIO implementation), a libc_r that used AIO rather than select demonstrated some of the performance improvements we see with 1:1 via linuxthreads (and now libthr). I'm not sure if there are any open source tools available to easily track process and thread scheduling and blocking, but there have been several pretty useful visual analysis and tracing tools for realtime. Some basic tools for thread tracing and visualization exist for Mac OS X, and presumably other COTS platforms. ktrace on FreeBSD makes some attempt to track context switches, but without enough context (har har) to be useful for this kind of analysis. I've been thinking about tweaking the local scheduler to put a bit more information into ktr and alq about blocking circumstances as well as some way to constrain the tracing to a particular bit of the process hierarchy with an inheritance flag of some sort. It might be quite helpful for understanding some of the nasty threading blocking/timing issues that we already run into with libc_r, and will continue to run into as our threading evolves. Robert N M Watson FreeBSD Core Team, TrustedBSD Projects [EMAIL PROTECTED] Network Associates Laboratories __
Re: libthr and 1:1 threading.
On Wed, Apr 02, 2003 at 07:38:14AM -0800, Terry Lambert wrote: > Has anyone tried compiling X11 to use libthr? Someone reported success with KDE, so it should serve as a sign of working X11. Jiawei -- "Without the userland, the kernel is useless." --inspired by The Tao of Programming ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: libthr and 1:1 threading.
On (2003/04/02 07:38), Terry Lambert wrote: > Is the disk I/O really that big of an issue? All writes will > be on underlying non-blocking descriptors; I guess you are > saying that the interleaved I/O is more important, further > down the system call interface than the top, and this becomes > an issue? Dude, you should really try this stuff for yourself before naysaying performance improvements on principle. It's actually quite impressive for desktop users (at least). Ciao, Sheldon. ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: libthr and 1:1 threading.
Robert Watson wrote: > You should notice marked interactivity and UI latency improvements with > threaded GUI apps over libc_r because GUI threads will generally no longer > be blocked when disk I/O and blocking I/O occurs. For example, > applications like Open Office, Netscape, et al, really get a lot better > with 1:1. Likewise, non-interactive applications that are disk > I/O-intensive, such as mysql, will also perform substantially better > because a thread that hits blocking using an interface that doesn't > support non-blocking I/O (such as the file system) won't clog up the > application. Is the disk I/O really that big of an issue? All writes will be on underlying non-blocking descriptors; I guess you are saying that the interleaved I/O is more important, further down the system call interface than the top, and this becomes an issue? It seems to me that maybe the correct fix for this is to use AIO instead of non-blocking I/O, then? The GUI thread issues are something I hadn't considered; I don't generally think of user space CPU intensive operations like that, but I guess it has to be rendered some time. 8-). Has anyone tried compiling X11 to use libthr? Also, any ETA on the per process signal mask handing bug in libthr? Might not be safe to convert everything up front, in a rush of eager enthusiasm... -- Terry ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: libthr and 1:1 threading.
Sheldon Hearn wrote: > On (2003/04/02 06:05), Terry Lambert wrote: > > > Do I misremember this? If not, does it not apply to UP systems as well? > > > > FWIW: the libc_r reentrancy isn't fixed by a 1:1 model for > > anything but calls for which there are no non-blocking > > alternative kernel APIs. [...long ramble...] When someone asks you a question and you answer it, it's not a "ramble", it's an "answer". 8-). > For all the rambling, I'm happy to report that my SCHED_ULE + libthr > UP workstation feels noticibly more responsive when I have several > Mozilla tabs all loading pages simultaneously while I'm trying to make a > threaded Java IDE do something sensible. You need to read things. I already explained that you were competing unfairly for quantum with other processes, in violation of POSIX.1, by virtue of defaulting to PTHREAD_SCOPE_SYSTEM. If you want to use PTHREAD_SCOPE_SYSTEM, you are supposed to have priviledges. Also, as I suggested, try nice'ing up your old version of Mozilla, and see if that gets you the same priority boost. 8-). > It's possible that I'm actually seeing the impact of other changes that > have been committed in the last week, I suppose. Jeff's recent sceduler changes have improved performance, in general, for most people who have tried them. It's naieve to change 8 or 9 things, and then attribute something as subjective perceived performance to one of them in particular. 8-) 8-). -- Terry ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: libthr and 1:1 threading.
On Wed, 2 Apr 2003, Sheldon Hearn wrote: > On (2003/04/02 06:05), Terry Lambert wrote: > > > > I think Jeff (or someone else?) said, that some web browsers gain > > > "something" too (serialization issues with libc_r)? I had the impression > > > that this also applies to UP systems. > > > > > > Do I misremember this? If not, does it not apply to UP systems as well? > > > > FWIW: the libc_r reentrancy isn't fixed by a 1:1 model for > > anything but calls for which there are no non-blocking > > alternative kernel APIs. [...long ramble...] > > For all the rambling, I'm happy to report that my SCHED_ULE + libthr UP > workstation feels noticibly more responsive when I have several Mozilla > tabs all loading pages simultaneously while I'm trying to make a > threaded Java IDE do something sensible. > > It's possible that I'm actually seeing the impact of other changes that > have been committed in the last week, I suppose. You should notice marked interactivity and UI latency improvements with threaded GUI apps over libc_r because GUI threads will generally no longer be blocked when disk I/O and blocking I/O occurs. For example, applications like Open Office, Netscape, et al, really get a lot better with 1:1. Likewise, non-interactive applications that are disk I/O-intensive, such as mysql, will also perform substantially better because a thread that hits blocking using an interface that doesn't support non-blocking I/O (such as the file system) won't clog up the application. Robert N M Watson FreeBSD Core Team, TrustedBSD Projects [EMAIL PROTECTED] Network Associates Laboratories ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: libthr and 1:1 threading.
On (2003/04/02 06:05), Terry Lambert wrote: > > I think Jeff (or someone else?) said, that some web browsers gain > > "something" too (serialization issues with libc_r)? I had the impression > > that this also applies to UP systems. > > > > Do I misremember this? If not, does it not apply to UP systems as well? > > FWIW: the libc_r reentrancy isn't fixed by a 1:1 model for > anything but calls for which there are no non-blocking > alternative kernel APIs. [...long ramble...] For all the rambling, I'm happy to report that my SCHED_ULE + libthr UP workstation feels noticibly more responsive when I have several Mozilla tabs all loading pages simultaneously while I'm trying to make a threaded Java IDE do something sensible. It's possible that I'm actually seeing the impact of other changes that have been committed in the last week, I suppose. Ciao, Sheldon. ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: libthr and 1:1 threading.
Alexander Leidinger wrote: > On Tue, 01 Apr 2003 23:28:01 -0800 > Terry Lambert <[EMAIL PROTECTED]> wrote: > > The primary performance reasoning behind a 1:1 kernel threading > > implementation, relative to the user space single kernel entry > > scheduler in the libc_r implementation is SMP scalability for > > threaded applications. > > I think Jeff (or someone else?) said, that some web browsers gain > "something" too (serialization issues with libc_r)? I had the impression > that this also applies to UP systems. > > Do I misremember this? If not, does it not apply to UP systems as well? FWIW: the libc_r reentrancy isn't fixed by a 1:1 model for anything but calls for which there are no non-blocking alternative kernel APIs. That means you are saved on things like System V IPC interfaces (which you were saved on before, if you used my kqueue patch for System V IPC to turn it into events), but the file I/O, which is mostly what a browser is doing, is not significantly improved. And for calls for which entrancy must be serialized, like the resolver, you are stuck serializing in user space. There is potential for some additional interleaved I/O occurring, when the kernel is multiply entrant, rather than EWOULDBLOCK plus a call conversion using a non-blocking file descriptor, that's true. I'm not sure that anything is gained by this in the UP case, however. Oh, and your process gets to compete for quantum as if it were "number of threads" processes. This is actually wrong, if you are PTHREAD_SCOPE_PROCESS, and setting the other option, PTHREAD_SCOPE_SYSTEM, requires root priviledges (but is on by default in the libthr implementation, because a 1:1 model requires more sophisticated scheduler support). You can get this same "cheat for extra quantum" effect with libc_r by using the same root priviledge to "nice" your process to a higher priority relative to the default. The other issue with going from the user space model to a kernel 1:1 model is that you are trading additional context switch overhead, by way of not fully utilizing your quantum, when you go into the kernel to block, instead of doing the call conversion: when you go to sleep, you hand the remainder of your quantum back to the system in 1:1, whereas with the libc_r, you hand it to another of your own threads. This is also what the N:M addresses. This also makes you less efficient when it comes to context switch overhead, since without thread-in-group affinity, when you give up the quantum, you are likely to take a full context switch (%cr3 reload for a different process; for example, cron runs once a second, and there are other "standard" system processes which are far from idle). In other words, you only see a benefit for a lone threaded application on a relatively quiecent system, which is OK, I guess, for a microbenchmark, but pretty poor when it comes to real world systems, which tend to be more heavily loaded. The last time and of this difference in overhead between the models was measured in any meaningful way (AFAIK) was in an article in (November?) 2001 in: Parallel and Distributed Systems, IEEE Trans. on (T-PDS) Which is where most of the academic work on scheduling and load balancing for parallel and distributed systems takes place. Unfortunately, these papers are not available online, unless you are an IEEE member, who also subscribes to T-PDS. Maybe you can find them, if you work at IBM and have access to a good technical library (e.g. Almaden), or if you have access to the Berkeley or Stanford libraries, or some other good technical library. There could have been a more recent paper, starting about 8 months back, which I don't know about (I need to take a trip to the library myself ;^)). The bottom line is that the 1:1 model is primarily useful for improving SMP scalability at this time, and that the overhead tradeoffs that are there right now don't really favor it, IMO, in a UP system. FreeBSD's libc_r has been in developement for nearly a decade; it is a *very good* pthreads implementation, and the only places it falls down, really, are in SMP scalability, and some blocking system calls that aren't amenable to conversion. You'll note that the System V IPC interfaces are not supported in a threaded environment by the POSIX standard; most of these blocking cases aren't really a problem. User space serialization in things like the resolver, which only opens one socket to do lookups, and can only deal with a single response context are a cross everyone bears. Mostly this bothers no one, because all browsers cache responses, there is response locality, and it's not a problem unless you compile in support for multiple protocol families (IPv4 + IPv6 support in the same browser usually means waiting for an IPv6 lookup timeout, if the remote host is not an RFC compliant DNS server, and fails to return an immediate reject response, like it's supposed to do). If you "fix" that for libthr, you also "fix" it for
Re: libthr and 1:1 threading.
On Tue, 01 Apr 2003 23:28:01 -0800 Terry Lambert <[EMAIL PROTECTED]> wrote: > The primary performance reasoning behind a 1:1 kernel threading > implementation, relative to the user space single kernel entry > scheduler in the libc_r implementation is SMP scalability for > threaded applications. I think Jeff (or someone else?) said, that some web browsers gain "something" too (serialization issues with libc_r)? I had the impression that this also applies to UP systems. Do I misremember this? If not, does it not apply to UP systems as well? Bye, Alexander. -- Actually, Microsoft is sort of a mixture between the Borg and the Ferengi. http://www.Leidinger.net Alexander @ Leidinger.net GPG fingerprint = C518 BC70 E67F 143F BE91 3365 79E2 9C60 B006 3FE7 ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: libthr and 1:1 threading.
Stijn Hoop wrote: > On Mon, Mar 31, 2003 at 10:54:45PM -0500, Jeff Roberson wrote: > > I have commited libthr. To try this out you'll need to do the following > > I know very very little about threads, but I'm interested as to what the > purpose is of this library. Is there a document available somewhere that > describes the relationships between this, KSE, libc_r, pthreads, the > Giant-unwinding-make-SMP-work-better project and some of the other > threads and SMP related libraries and terminology? No, not really: the new libthr was pretty much a "Skunk Works" project, and was not managed through the KSE project; it's really orthogonal, but builds on some of the KSE work already done in the kernel so far... most of KSE lives there. Here's a thumbnail sketch, though (forgive me, KSE folks, if I mung it too badly): pthreads: POSIX Threads is a threads API, which is specified by the standard ISO/IEC 9945-1:1996 (``POSIX.1''). libc_r: A user space implementation of the pthreads API; this implementation uses a "call conversion" scheduler, in user space, to convert blocking calls into non-blocking calls, plus a threads context switch via the user space scheduler. Like all interactive timesharing schedulers, it gives the illusion of concurrency. However, the kernel is not thread-reentrant in this model, so it does not scale to more than one CPU on an SMP system, since there is only a single scheduler context. KSE:"Kernel Schedulable Entitites" is the name of the modified scheduler activations framework, as well as the user space components, and kernel modifications, for an N:M model threading system. It has the advantage over the "libc_r" in that it causes the kernel to be thread reentrant, and so it provides SMP scalability. Because it's N:M, it also has the advantage over the 1:1 approach of causing full quantum utilization, and providing for CPU affinity for individual threads, and CPU negaffinity for threads within the same process, thereby providing for theoretically optimal CPU and other resource utilization. It also includes a user-space library component, which is incomplete at present. libthr: This is the recently committed 1:1 model threading library. It provides a simpler user space library component, which provides the same SMP scalability, as far as kernel thread reentrancy is concerned, but fails to provide for full quantum utilization, and, at present, does not directly address the CPU affinity issues itself (no sophisticated use of KSEGRP). It builds on the kernel modifications for KSE, and adds a couple of system call API's in order to manage creation, etc., of threads. The major intent is to provide for SMP scalability; as a side effect, it provides a proof-of-concept for the KSE code already in the kernel, and as such, has been very welcome. SMPng: "The Giant-unwinding-make-SMP-work-better project", to quote you. 8-). SMPng has it's own project page and documentation. To give another thumbnail drawing, it's about improving the granularity of locking and the logical seperation between kernel subsystems, so that stall barriers are eliminated. By doing this, inter-CPU contention is reduced, and you get better SMP scaling. Traditionally, when you added another CPU, you maybe got a 20% performance improvement for a 100% increase in cost. The idea is to get that 20% up to as close to 100% as possible (impossible, but you can approach that). SVR4, for example, scales well up to 4 processors on Intel. It scales higher than that, but the incremental improvement is about 80%, and so at about 4 processors, you hit a point where the cost of additional processors is higher than the value of the additional compute cycles. You may also find these resources useful: http://people.freebsd.org/~julian/threads/ http://www.freebsd.org/smp/index.html http://www.freebsd.org/projects/busdma/index.html http://www.freebsd.org/projects/projects.html Most of the documentation lives in mailing list archives, and is not terribly formal (Software
Re: libthr and 1:1 threading.
On Mon, Mar 31, 2003 at 10:54:45PM -0500, Jeff Roberson wrote: > I have commited libthr. To try this out you'll need to do the following I know very very little about threads, but I'm interested as to what the purpose is of this library. Is there a document available somewhere that describes the relationships between this, KSE, libc_r, pthreads, the Giant-unwinding-make-SMP-work-better project and some of the other threads and SMP related libraries and terminology? --Stijn -- "...I like logs. They give me a warm fuzzy feeling. I've been known to keep logs for 30 months at a time (generally when I thought I was rotating them daily, but was actually rotating them once a month)." -- Michael Lucas, in Big Scary Daemons article 'Controlling Bandwidth' pgp0.pgp Description: PGP signature
Re: libthr and 1:1 threading.
Jun Su wrote: > [ ... 1:1 kernel threads implementation ... ] > > A benchmark would be interested. This request doesn't make sense. The primary performance reasoning behind a 1:1 kernel threading implementation, relative to the user space single kernel entry scheduler in the libc_r implementation is SMP scalability for threaded applications. Basically, the only reasonable benchmark, given this, for a comparison of the two would be a threaded CPU-bound program on a *non-SMP* system. That really makes no sense, because that wasn't the use case for the design goal of SMP scalability; it doesn't really matter *what* the relative performance is on UP systems, relative to the libc_r library, so long as it adds SMP scalability. Which it does. It's apples and oranges; there's really no reasonable way to compare the two implementations, since they solve different problem sets. Could you maybe ask a different question? -- Terry ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: libthr and 1:1 threading.
--- "Jacques A. Vidrine" <[EMAIL PROTECTED]> 的正文：> On Mon, Mar 31, 2003 at 10:54:45PM -0500, Jeff > Roberson wrote: > > > 5. Either replace /usr/lib/libc_r.so.5 with > /usr/lib/libthr.so.1 or > > relink your applications against libthr.so.1 > > Happily strlen(libc_r.so.5) == strlen(libthr.so.1), > so you can also > edit your binaries' SONEEDED fields in place :-) > > ed -s /path/to/binary <<-EOF > /libc_r.so.5/ s/libc_r.so.5/libthr.so.1/ > w > q > EOF > > or similar ... > > Cheers, > -- > Jacques A. Vidrine <[EMAIL PROTECTED]> > http://www.celabo.org/ > NTT/Verio SME . FreeBSD UNIX . > Heimdal Kerberos > [EMAIL PROTECTED] . [EMAIL PROTECTED] . > [EMAIL PROTECTED] > ___ > [EMAIL PROTECTED] mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-current > To unsubscribe, send any mail to "[EMAIL PROTECTED]" A benchmark would be interested. Jun Su _ Do You Yahoo!? 更多惊喜，同样精彩，NetVista A30 热卖 http://ad.cn.doubleclick.net/clk;5313999;7930402;p?http://www.ibm.com/cn/promotion/pc/netvista_a30/index.shtml ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: libthr and 1:1 threading.
Jeff Roberson wrote: > > I have commited libthr. To try this out you'll need to do the following > > 1. cvsup > 2. rebuild world and kernel > 3. install world and kernel > 4. build libthr from src/lib/libthr > 5. Either replace /usr/lib/libc_r.so.5 with /usr/lib/libthr.so.1 or > relink your applications against libthr.so.1 > > This works with mozilla and open office. FWIW, it also appears to work fine with KDE 3.1 and multithreaded qt (I opted for replacing rather than relinking). Geoff ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: libthr and 1:1 threading.
On Tue, 1 Apr 2003, Scott Long wrote: > Jeff Roberson wrote: > > I have commited libthr. To try this out you'll need to do the following > > > > Excellent job Jeff and Jon, thanks a lot! > > Is anyone working on getting full Apache2 support for this? Also, > linking the Java 1.3 and 1.4 ports to this might be interesting. > > Scott > I'd be surprised if apache2 didn't just work. I have yet to try it out though. I have enough bug fixing and little features to do that I'm not worried about more app compat for now. Hopefully I can get things to a state where there are no known bugs and then find a maintainer for this. There is plenty of exciting work to do on it after that but I'm not sure what kind of bandwidth I will have for it. Cheers, Jeff ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: libthr and 1:1 threading.
Jeff Roberson wrote: I have commited libthr. To try this out you'll need to do the following Excellent job Jeff and Jon, thanks a lot! Is anyone working on getting full Apache2 support for this? Also, linking the Java 1.3 and 1.4 ports to this might be interesting. Scott ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: libthr and 1:1 threading.
Jeff Roberson wrote: I have commited libthr. To try this out you'll need to do the following Excellent job Jeff and Jon, thanks a lot! Is anyone working on getting full Apache2 support for this? Also, linking the Java 1.3 and 1.4 ports to this might be interesting. Scott ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: libthr and 1:1 threading.
On Mon, Mar 31, 2003 at 10:54:45PM -0500, Jeff Roberson wrote: > 5. Either replace /usr/lib/libc_r.so.5 with /usr/lib/libthr.so.1 or > relink your applications against libthr.so.1 Happily strlen(libc_r.so.5) == strlen(libthr.so.1), so you can also edit your binaries' SONEEDED fields in place :-) ed -s /path/to/binary <<-EOF /libc_r.so.5/ s/libc_r.so.5/libthr.so.1/ w q EOF or similar ... Cheers, -- Jacques A. Vidrine <[EMAIL PROTECTED]> http://www.celabo.org/ NTT/Verio SME . FreeBSD UNIX . Heimdal Kerberos [EMAIL PROTECTED] . [EMAIL PROTECTED] . [EMAIL PROTECTED] ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "[EMAIL PROTECTED]"
