Re: What is Fast task queue? (Was: How to understand what `swi5' kernel thread does?)
On 8/28/12 3:44 PM, John Baldwin wrote: On Monday, August 27, 2012 4:47:13 pm Lev Serebryakov wrote: Hello, John. You wrote 27 августа 2012 г., 20:26:03: What fast tasks are performed via this queue? Under network load it is main consumer of CPU. JB Certain NIC drivers perform much of their interrupt handling in that thread. Yep, I've found, that my if_vr uses it. One more question: does ipfw rules works in same thread? I have ``net.isr.dispatch=direct'' set. Yes, with the default setting of direct dispatch, all of the receive side of the network stack runs in the driver's interrupt handler. In the case of vr(4) that would be in this thread. Lev, If you have more than one CPU, you can try this patch [1]. It adds flowid in mbufs and makes possible to use multiple netisr threads. Apply the patch and put in your loader.conf the following: net.isr.maxthreads=$ncpu (2, 4 or whatever you want) net.isr.dispatch=deferred PS: original patch was written by melifaro. [1] http://people.freebsd.org/~zont/netisr.patch -- Andrey Zonov signature.asc Description: OpenPGP digital signature
Re: [PATCH] Add a -h flag to mv
On Tue, 28 Aug 2012 10:58:09 -0400 John Baldwin j...@freebsd.org wrote: I have a use case at work where I need to be able to update a symlink that points to a directory atomically (so that it points to a new directory). To give a conrete example, suppose I have two directories 'foo' and 'bar', and a symlink 'a' that I wish to atomically flip from 'foo' to 'bar'. Using 'ln -shf bar a' is not atomic as it uses separate unlink() and symlink() system calls, so there is a race where another thread may encounter ENOENT while traversing 'a'. The approach we used was to create a new symbolic link 'a.new' (e.g. via 'ln -s bar a.new') and then use rename() to rename 'a.new' on top of 'a'. Normally to do an atomic rename from userland one would use 'mv', but 'mv a.new a' doesn't do that. Instead, it moves 'a.new' into the directory referenced by the 'a' symlink. At work we have resorted to invoking python's os.rename() in a one-liner to handle this. While rehashing this discussion today it occurred to me that a -h flag to mv would allow it to work in this case (and is very similar to how ln treats its -h flag). To that end, I have a patch to add a new -h flag to mv that allows one to atomically update a symlink that points to a directory. I could not find any other mv commands that have adopted a -h (or a different flag that accomplishes the same task). Given that it functions identically to the -h flag for ln, -h seemed the logical choice. Any objections? [snip patch] Nope, seems like a reasonable extension to me (gj@). -- Gary Jennejohn ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: What is Fast task queue? (Was: How to understand what `swi5' kernel thread does?)
Hello, Andrey. You wrote 29 августа 2012 г., 11:17:09: AZ If you have more than one CPU, you can try this patch [1]. It adds I have only one cput (Geode LX 500 Mhz) without any HT or other virtual CPUs at all. One core, one thread :) -- // Black Lion AKA Lev Serebryakov l...@freebsd.org ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: Can't build FreeBSD-head with CLANG
-- Eir Nym On 29 August 2012 01:19, Dimitry Andric d...@freebsd.org wrote: On 2012-08-28 23:05, Eir Nym wrote: ... This one for example: cc -c -O2 -pipe -fno-strict-aliasing -std=c99 -Wall -Wredundant-decls -Wnested-externs -Wstrict-prototypes -Wmissing-prototypes -Wpointer-arith -Winline -Wcast-qual -Wundef -Wno-pointer-sign -fformat-extensions -Wmissing-include-dirs -fdiagnostics-show-option -Wno-error-tautological-compare -Wno-error-empty-body -Wno-error-parentheses-equality -nostdinc -I. -I/usr/head/src/sys -I/usr/head/src/sys/contrib/altq -D_KERNEL -DHAVE_KERNEL_OPTION_HEADERS -include opt_global.h -fno-omit-frame-pointer -mno-aes -mno-avx -mcmodel=kernel -mno-red-zone -mno-mmx -mno-sse -msoft-float -fno-asynchronous-unwind-tables -ffreestanding -fstack-protector -Werror /usr/head/src/sys/gnu/fs/xfs/xfs_alloc.c -I/usr/head/src/sys/gnu/fs/xfs/FreeBSD -I/usr/head/src/sys/gnu/fs/xfs/FreeBSD/support -I/usr/head/src/sys/gnu/fs/xfs /usr/head/src/sys/gnu/fs/xfs/xfs_alloc.c:1449:11: error: variable 'fbno' is used uninitialized whenever 'if' condition is false [-Werror,-Wsometimes-uninitialized] else if (args-minlen == 1 args-alignment == 1 !args-isfl ^~~ Weird, how are you building the xfs module? It has WERROR= in its Makefile, so the '-Werror' option above should not be there. This is because the XFS code was imported more than 6 years ago, and is very unlikely to ever be fixed. :) head SVN revision 239793, http://eroese.org/_/_/pub/bsd/GENERIC_PF.amd64 — kernel config ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: Can't build FreeBSD-head with CLANG
-- Eir Nym On 29 August 2012 12:41, Eir Nym eir...@gmail.com wrote: -- Eir Nym On 29 August 2012 01:19, Dimitry Andric d...@freebsd.org wrote: On 2012-08-28 23:05, Eir Nym wrote: ... This one for example: cc -c -O2 -pipe -fno-strict-aliasing -std=c99 -Wall -Wredundant-decls -Wnested-externs -Wstrict-prototypes -Wmissing-prototypes -Wpointer-arith -Winline -Wcast-qual -Wundef -Wno-pointer-sign -fformat-extensions -Wmissing-include-dirs -fdiagnostics-show-option -Wno-error-tautological-compare -Wno-error-empty-body -Wno-error-parentheses-equality -nostdinc -I. -I/usr/head/src/sys -I/usr/head/src/sys/contrib/altq -D_KERNEL -DHAVE_KERNEL_OPTION_HEADERS -include opt_global.h -fno-omit-frame-pointer -mno-aes -mno-avx -mcmodel=kernel -mno-red-zone -mno-mmx -mno-sse -msoft-float -fno-asynchronous-unwind-tables -ffreestanding -fstack-protector -Werror /usr/head/src/sys/gnu/fs/xfs/xfs_alloc.c -I/usr/head/src/sys/gnu/fs/xfs/FreeBSD -I/usr/head/src/sys/gnu/fs/xfs/FreeBSD/support -I/usr/head/src/sys/gnu/fs/xfs /usr/head/src/sys/gnu/fs/xfs/xfs_alloc.c:1449:11: error: variable 'fbno' is used uninitialized whenever 'if' condition is false [-Werror,-Wsometimes-uninitialized] else if (args-minlen == 1 args-alignment == 1 !args-isfl ^~~ Weird, how are you building the xfs module? It has WERROR= in its Makefile, so the '-Werror' option above should not be there. This is because the XFS code was imported more than 6 years ago, and is very unlikely to ever be fixed. :) head SVN revision 239793, http://eroese.org/_/_/pub/bsd/GENERIC_PF.amd64 — kernel config http://eroese.org/_/_/pub/bsd/kernel.amd64.GENERIC_PF — full logs for it. ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: [PATCH] Add a -h flag to mv
On Tue, Aug 28, 2012 at 10:58:09AM -0400, John Baldwin wrote: I have a use case at work where I need to be able to update a symlink that points to a directory atomically (so that it points to a new directory). To give a conrete example, suppose I have two directories 'foo' and 'bar', and a symlink 'a' that I wish to atomically flip from 'foo' to 'bar'. Using 'ln -shf bar a' is not atomic as it uses separate unlink() and symlink() system calls, so there is a race where another thread may encounter ENOENT while traversing 'a'. The approach we used was to create a new symbolic link 'a.new' (e.g. via 'ln -s bar a.new') and then use rename() to rename 'a.new' on top of 'a'. Normally to do an atomic rename from userland one would use 'mv', but 'mv a.new a' doesn't do that. Instead, it moves 'a.new' into the directory referenced by the 'a' symlink. At work we have resorted to invoking python's os.rename() in a one-liner to handle this. While rehashing this discussion today it occurred to me that a -h flag to mv would allow it to work in this case (and is very similar to how ln treats its -h flag). To that end, I have a patch to add a new -h flag to mv that allows one to atomically update a symlink that points to a directory. I could not find any other mv commands that have adopted a -h (or a different flag that accomplishes the same task). Given that it functions identically to the -h flag for ln, -h seemed the logical choice. Any objections? GNU coreutils mv (and also cp/install/ln) appears to use -T/--no-target-directory for a similar purpose: -T prevents the target being treated as a directory (whether it is a symlink or not). -- Jilles Tjoelker ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: per file descriptor device callbacks ?
On Wed, Aug 29, 2012 at 06:55:26AM +0200, Luigi Rizzo wrote: On Wed, Aug 29, 2012 at 07:12:40AM +0300, Konstantin Belousov wrote: On Tue, Aug 28, 2012 at 08:42:26PM +0200, Luigi Rizzo wrote: On Tue, Aug 28, 2012 at 08:26:06PM +0300, Konstantin Belousov wrote: ... dev_clone() is rather gross and a lot harder to use than devfs_set_cdevpriv(). If you are fine with the inherent problems of the device pager (you can't ever make mappings go away), you can just assign each client a unique offset into your shared object's memory space. However, if you are exporting shared memory buffers, then a better model might be to let your clients use shm_open(SHM_ANON) to create buffers, then pass them into your driver via an ioctl() and use shm_map() to map them into the kernel. Well, there is a new OBJT_MGTDEVICE pager, which together with d_mmap_single() allows to have even per-mapping data. i915kms uses it. You do not need cdevpriv for the per-mapping data. Also, MGTDEVICE does track the mappings of the pages, so you can clean up on device destruction. interesting, thanks for the pointer, i'll look it up in i915kms. Does this exist also in stable/9 ? It would be a shame otherwise... Yes, it was merged. Normal callbacks of the device pager allows to execute ctr/dtr methods at the time of mapping creation and tear down. what would be a good way to install my own ctr/dtr methods ? I only found out a rather crude one, and it only works for the destructor: See below. static struct cdev_pager_ops saved_cdev_pager_ops; static struct cdev_pager_ops netmap_cdev_pager_ops; static void netmap_dev_pager_dtor(void *handle) { saved_cdev_pager_ops.cdev_pg_dtor(handle); // my code here D(ready to release memory for %p, handle); } static int netmap_mmap_single(struct cdev *cdev, vm_ooffset_t *foff, vm_size_t objsize, vm_object_t *objp, int prot) { vm_object_t obj; /* XXX check size etc. */ obj = vm_pager_allocate(OBJT_DEVICE, cdev, objsize, prot, *foff, curthread-td_ucred); Use cdev_pager_allocate(). if (obj == NULL) return EINVAL; if (saved_cdev_pager_ops.cdev_pg_fault == NULL) { I do not understand what are you trying to accomplish with the check and reinitialization, but I assume that cdev_pager_allocate() would take care of it. First and foremost, I am trying to do things without requiring kernel modifications. This is strange. I am trying to reuse the constructor and destructor of the standard device pager, and around those add my own calls. Those methods are declared static in sys/vm/device_pager.c so i cannot invoke directly cdev_pager_allocate(). I fail to see how can you use old_dev_pager_{ctor,dtor} without also using the d_mmap. Note that use of d_mmap _does not_ mean that access to the mapped area never faults. I suggest you to go fully 'mgmt device pager' route. I could indeed rewrite the body of those three methods (ctor, dtor, fault) in my own code. I will look at this today. Perhaps I could even try to install all mappings at mmap() time so I never need to fault. Thanks again for the suggestions cheers luigi D(initialize cdev_pager_ops); saved_cdev_pager_ops = *(obj-un_pager.devp.ops); netmap_cdev_pager_ops = *(obj-un_pager.devp.ops); netmap_cdev_pager_ops.cdev_pg_dtor = netmap_dev_pager_dtor; }; obj-un_pager.devp.ops = netmap_cdev_pager_ops; *objp = obj; /* XXX perhaps do something more here, such as install * mappings for the pages so we have no faults later. */ return 0; } static struct cdevsw netmap_cdevsw = { .d_version = D_VERSION, .d_name = netmap, .d_open = netmap_open, .d_mmap = netmap_mmap, .d_mmap_single = netmap_mmap_single, .d_ioctl = netmap_ioctl, .d_poll = netmap_poll, .d_close = netmap_close, }; cheers luigi pgpJpall0KhX7.pgp Description: PGP signature
Re: [PATCH] Add a -h flag to mv
On Wednesday, August 29, 2012 6:02:47 am Jilles Tjoelker wrote: On Tue, Aug 28, 2012 at 10:58:09AM -0400, John Baldwin wrote: I have a use case at work where I need to be able to update a symlink that points to a directory atomically (so that it points to a new directory). To give a conrete example, suppose I have two directories 'foo' and 'bar', and a symlink 'a' that I wish to atomically flip from 'foo' to 'bar'. Using 'ln -shf bar a' is not atomic as it uses separate unlink() and symlink() system calls, so there is a race where another thread may encounter ENOENT while traversing 'a'. The approach we used was to create a new symbolic link 'a.new' (e.g. via 'ln -s bar a.new') and then use rename() to rename 'a.new' on top of 'a'. Normally to do an atomic rename from userland one would use 'mv', but 'mv a.new a' doesn't do that. Instead, it moves 'a.new' into the directory referenced by the 'a' symlink. At work we have resorted to invoking python's os.rename() in a one-liner to handle this. While rehashing this discussion today it occurred to me that a -h flag to mv would allow it to work in this case (and is very similar to how ln treats its -h flag). To that end, I have a patch to add a new -h flag to mv that allows one to atomically update a symlink that points to a directory. I could not find any other mv commands that have adopted a -h (or a different flag that accomplishes the same task). Given that it functions identically to the -h flag for ln, -h seemed the logical choice. Any objections? GNU coreutils mv (and also cp/install/ln) appears to use -T/--no-target-directory for a similar purpose: -T prevents the target being treated as a directory (whether it is a symlink or not). Hmm, I could find no documentation for this online via Google searches or the Linux manpages we have on www.FreeBSD.org. Bah, Google just makes searching for these sorts of things painful it seems (you have to put explicit quotes around --no-target-directory for it to actually be used). Also, it seems I just chose all the wrong Linux manpage sets to look at. It seems that Linux's -T flag is similar to -h for ln as well. I don't think we can deprecate -h for ln, but perhaps we could add -T as a compat flag to ln and mv? I'd be inclined to still add -h to mv so that it mirrors ln. Hmm, it seems RedHat's ln uses -n for this (OpenBSD, NetBSD, and Darwin all include a -n as an alias to -h for ln to support compat with other operating systems). OSF/1 (and Tru64) and SunOS use -n to mean complain if the file already exists similar to 'mv -n'. Also, looking at the Suse manpage on www.FreeBSD.org, it seems their ln (which does have -T) has both -n and -T with different descriptions, but to achieve the same purpose: http://www.freebsd.org/cgi/man.cgi?query=lnapropos=0sektion=0manpath=SuSE+Linux%2Fi386+11.3arch=defaultformat=html -n, --no-dereference treat destination that is a symlink to a directory as if it were a normal file -T, --no-target-directory treat LINK_NAME as a normal file (To me it seems LINK_NAME and destination are the same thing.) My inclination would be to add -h to mv, but perhaps add -T as an alias for -h to both ln and mv, and -n as an alias for -h to ln (if we want aliases to match coreutils). -- John Baldwin ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: [PATCH] Add a -h flag to mv
On Wed, Aug 29, 2012 at 08:09:20AM -0400, John Baldwin wrote: On Wednesday, August 29, 2012 6:02:47 am Jilles Tjoelker wrote: GNU coreutils mv (and also cp/install/ln) appears to use -T/--no-target-directory for a similar purpose: -T prevents the target being treated as a directory (whether it is a symlink or not). Hmm, I could find no documentation for this online via Google searches or the Linux manpages we have on www.FreeBSD.org. Bah, Google just makes searching for these sorts of things painful it seems (you have to put explicit quotes around --no-target-directory for it to actually be used). Also, it seems I just chose all the wrong Linux manpage sets to look at. It seems that Linux's -T flag is similar to -h for ln as well. I don't think we can deprecate -h for ln, but perhaps we could add -T as a compat flag to ln and mv? I'd be inclined to still add -h to mv so that it mirrors ln. Hmm, it seems RedHat's ln uses -n for this (OpenBSD, NetBSD, and Darwin all include a -n as an alias to -h for ln to support compat with other operating systems). OSF/1 (and Tru64) and SunOS use -n to mean complain if the file already exists similar to 'mv -n'. Also, looking at the Suse manpage on www.FreeBSD.org, it seems their ln (which does have -T) has both -n and -T with different descriptions, but to achieve the same purpose: http://www.freebsd.org/cgi/man.cgi?query=lnapropos=0sektion=0manpath=SuSE+Linux%2Fi386+11.3arch=defaultformat=html -n, --no-dereference treat destination that is a symlink to a directory as if it were a normal file -T, --no-target-directory treat LINK_NAME as a normal file (To me it seems LINK_NAME and destination are the same thing.) My inclination would be to add -h to mv, but perhaps add -T as an alias for -h to both ln and mv, and -n as an alias for -h to ln (if we want aliases to match coreutils). Coreutils ln -n is the same as our ln -h, and we already have compatibility for it. The coreutils -T option is different, though. It forces the ln source_file target_file synopsis instead of the ln source_file ... target_dir synopsis, without checking the type of the final operand. If there are not exactly two operands, a syntax error occurs. If the final operand is a directory and cannot be overwritten, an error occurs. -- Jilles Tjoelker ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: [PATCH] Add a -h flag to mv
On Wednesday, August 29, 2012 3:32:13 pm Jilles Tjoelker wrote: On Wed, Aug 29, 2012 at 08:09:20AM -0400, John Baldwin wrote: On Wednesday, August 29, 2012 6:02:47 am Jilles Tjoelker wrote: GNU coreutils mv (and also cp/install/ln) appears to use -T/--no-target-directory for a similar purpose: -T prevents the target being treated as a directory (whether it is a symlink or not). Hmm, I could find no documentation for this online via Google searches or the Linux manpages we have on www.FreeBSD.org. Bah, Google just makes searching for these sorts of things painful it seems (you have to put explicit quotes around --no-target-directory for it to actually be used). Also, it seems I just chose all the wrong Linux manpage sets to look at. It seems that Linux's -T flag is similar to -h for ln as well. I don't think we can deprecate -h for ln, but perhaps we could add -T as a compat flag to ln and mv? I'd be inclined to still add -h to mv so that it mirrors ln. Hmm, it seems RedHat's ln uses -n for this (OpenBSD, NetBSD, and Darwin all include a -n as an alias to -h for ln to support compat with other operating systems). OSF/1 (and Tru64) and SunOS use -n to mean complain if the file already exists similar to 'mv -n'. Also, looking at the Suse manpage on www.FreeBSD.org, it seems their ln (which does have -T) has both -n and -T with different descriptions, but to achieve the same purpose: http://www.freebsd.org/cgi/man.cgi?query=lnapropos=0sektion=0manpath=SuSE+Linux%2Fi386+11.3arch=defaultformat=html -n, --no-dereference treat destination that is a symlink to a directory as if it were a normal file -T, --no-target-directory treat LINK_NAME as a normal file (To me it seems LINK_NAME and destination are the same thing.) My inclination would be to add -h to mv, but perhaps add -T as an alias for -h to both ln and mv, and -n as an alias for -h to ln (if we want aliases to match coreutils). Coreutils ln -n is the same as our ln -h, and we already have compatibility for it. Bah, not sure how I missed the -n previously. The coreutils -T option is different, though. It forces the ln source_file target_file synopsis instead of the ln source_file ... target_dir synopsis, without checking the type of the final operand. If there are not exactly two operands, a syntax error occurs. If the final operand is a directory and cannot be overwritten, an error occurs. Ah, this is not quite what my mv -h patch does. It does not error if the destination is a directory. It is much closer to ln -h and exactly matches the language for ln -n I quoted above. -- John Baldwin ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
[RFT] Some updates to libc/rpc
Hello; The Bull NFS4 for linux project took our RPC support from libc and did some enhancements on it for a while. The libraries have been diverging extensively and many of the changes are linux specific. The complete log of their changes is here: http://git.infradead.org/users/steved/libtirpc.git/log I don't think it would be a good idea to consider them upstream vendors, but I thought it would be good to attempt to keep partially in sync. I only took a rather small subset of their changes and got to this patch: http://people.freebsd.org/~pfg/patches/patch-rpc For those interested in the changelog, it's here: http://people.freebsd.org/~pfg/patches/libtirpc-log This is libc and rather critical stuff so I have no hurry and would like extensive testing before considering it for head. Please give it a try. Pedro. ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
[PATCH] Add locking to adv(4)
Continuing with the theme of locking older storage drivers, I have patches to add locking to adv(4) and mark it MPSAFE. The patches are from HEAD but should apply to 8 or 9. If you test it on 8 or 9 please enable INVARIANTS for at least the initial testing. Thanks. http://www.FreeBSD.org/~jhb/patches/adv_locking.patch -- John Baldwin ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
[CFT] Some updates to libc/rpc (second try)
(Second try with a more benign mailer) Hello; A while back the Bull NFS4 for linux project took our RPC support from libcand did some enhancements on it. The libraries have been diverging extensively and many of the changes are linux specific. The complete log of their changes is here: http://git.infradead.org/users/steved/libtirpc.git/log I don't think it would be a good idea to consider them upstream vendors, but I thought it would be good to attempt to keep partially in sync. I only took a rather small subset of their changes and got to this patch: http://people.freebsd.org/~pfg/patches/patch-rpc http://people.freebsd.org/%7Epfg/patches/patch-rpc For those interested in the changelog, it's here: http://people.freebsd.org/~pfg/patches/libtirpc-log http://people.freebsd.org/%7Epfg/patches/libtirpc-log This is rather critical stuff (libc) so I have no hurry and would like extensive testing before considering it for head. Please give it a try and report any issue. Pedro. ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
[PATCH] Locking for adw(4)
Continuing with the theme of locking older storage drivers, I have patches to add locking to adw(4) and mark it MPSAFE. The patches are from HEAD but should apply to 8 or 9. If you test it on 8 or 9 please enable INVARIANTS for at least the initial testing. Thanks. http://www.FreeBSD.org/~jhb/patches/adw_locking.patch -- John Baldwin ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org