Re: devfs panic w/INVARIANTS
On Thu, Feb 04, 2010 at 03:40:28PM -0500, Andrew Gallatin wrote: I've got a commercial driver that uses device cloning. At unload time, the driver calls clone_cleanup(). When I unload the driver when the kernel is built with INVARIANTS, I'll see a panic in devfs_populate_loop(). This happens in 6-stable, as well as 8-stable. From what I can see the clone has been freed, but it remains on the devfs cdevp_list. Then the next time devfs_populate_loop() is called, it trips over the bad entry (cdp-cdp_dirents points to 0xdeadc0dedeadc0de) See appended kgdb session. If I trace the code path, it looks like clone_cleanup() calls destroy_devl(). And destroy_devl() will eventually call devfs_free() if the si_refcnt is zero. But I don't see anything which will get the cdev removed from the cdevp_list prior to it being freed. The only code I see which will get the cdev removed from the cdevp_list() seems to be the GC any lingering devices block in devfs_populate_loop What am I missing? You did not mentioned it, but my guess is that you create clones from the dev_clone event handler. Please note that devfs_lookup() that fires dev_clone event, consumes a device reference. Thus clone handlers shall do dev_ref(). Due to races with cleanup, you should use MAKEDEV_REF flag for make_dev_credv(9) KPI instead of doing make_dev()/dev_ref() pair. That said, do you really need clones at all ? pgpJxMOAun7Uh.pgp Description: PGP signature
Re: [PATCH] Support for thresholds in du(1)
On (04/02/2010 23:24), jhell wrote:
On Thu, 4 Feb 2010 06:55, gleb.kurtsou@ wrote:
On (25/02/2009 17:24), Mel wrote:
Hi,
attached is a small patch to add threshold support to du(1). I've been
using
it on 7-STABLE machines for a while, cause I got tired of the noise I get
when sorting and then reformatting to human-readable. Especially since
sorting isn't part of the equasion I'd like to see all dirs exceeding a
given size.
I've not updated the manpage on -STABLE yet, should be the same as HEAD.
Example usage:
# du -xht 20m .
29M./contrib/binutils
52M./contrib/gcc
237M./contrib
35M./crypto
28M./lib
20M./share
55M./sys/dev
139M./sys
545M.
I'll file a PR for it, if there's no objections to this feature /
implementation, the style(9) or the usage of -t.
--
Mel
Hi,
I've cleaned up original patch:
* fixed style and some bugs
* as suggested changed it to use expand_number
* implemented support for negative threshold values
I find the patch very useful. Does it look ok to be commited, or should
I file a PR so it won't get lost once again.
Example usage:
src/sys/crypto % du -ht 100k
137K./des
482K.
src/sys/crypto % du -A -ht +100k
129K./des
446K.
src/sys/crypto % du -ht -100k
56K./camellia
11K./salsa20
38K./sha2
9.0K./rc4
68K./des/arch/i386
70K./des/arch
88K./rijndael
37K./via
8.0K./hmac
36K./blowfish/arch/i386
37K./blowfish/arch
85K./blowfish
Also not to be picky but I can see this being raised in another email at a
later time.
For Each
du -ht +1.5{M,G,T,P}
Would be something nice to compliment this so you do not have to revert
back to say 1440k when wanting to say 1.5m and so on for larger sizes.
Also I think to be fully supportive it should have the capability to do at
the least -N bits +-N bytes.
Let's keep it simple, these features are are not hard to implement, but
overcomplicate such simple utility as du(1). If one needs more
sophisticated disk usage reporting he's likely to have 3rd party utils
installed already.
Thanks for the work that you have done on this.
--
jhell
___
freebsd-hackers@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
Re: Weekend PR smashing
On Thu, Feb 04, 2010 at 04:09:48AM -0600, Mark Linimon wrote: Reports that are duplicates indicate that various users are being affected by one underlying problem. At one point I was trying to gather them all into a page. I was hoping more people would do the analysis and send me additions for it. However, it looks as though the script that generates that page has rotted. I'll re-add it to my list of things-to-do ... http://people.freebsd.org/~linimon/studies/prs/well_known_prs.html This report is now fixed. We have more kern/ PRs than any other category. This category is overloaded to mean both kernel, libraries, networking, and device drivers. http://people.freebsd.org/~linimon/studies/prs/pr_tag_index.html makes this much more tractable. I forgot to mention the 2-level hierarchy that I have set up, where you can look at PRs starting with e.g. disk/driver and then drill down to a page that references all the related PRs by manpage. It may have been just as well, since the report had also gotten stale. However, it is once again up-to-date: http://people.freebsd.org/~linimon/studies/prs/prs_for_all_groups.html There would also a slightly different way of looking at things, the ones with '[panic]' in the Synopsis. Hmm, I thought there was such a page, but it doesn't seem so. I'll put it on my list to create one. Now created: http://people.freebsd.org/~linimon/studies/prs/prs_for_tag_panic.html Finally, I have fixed other problems, such as broken links, in other various pages under http://people.freebsd.org/~linimon/studies/prs/ mcl ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
Re: devfs panic w/INVARIANTS
Kostik Belousov wrote: On Thu, Feb 04, 2010 at 03:40:28PM -0500, Andrew Gallatin wrote: I've got a commercial driver that uses device cloning. At unload time, the driver calls clone_cleanup(). When I unload the driver when the kernel is built with INVARIANTS, I'll see a panic in devfs_populate_loop(). This happens in 6-stable, as well as 8-stable. From what I can see the clone has been freed, but it remains on the devfs cdevp_list. Then the next time devfs_populate_loop() is called, it trips over the bad entry (cdp-cdp_dirents points to 0xdeadc0dedeadc0de) See appended kgdb session. If I trace the code path, it looks like clone_cleanup() calls destroy_devl(). And destroy_devl() will eventually call devfs_free() if the si_refcnt is zero. But I don't see anything which will get the cdev removed from the cdevp_list prior to it being freed. The only code I see which will get the cdev removed from the cdevp_list() seems to be the GC any lingering devices block in devfs_populate_loop What am I missing? You did not mentioned it, but my guess is that you create clones from the dev_clone event handler. Please note that devfs_lookup() that fires Yes, I do. dev_clone event, consumes a device reference. Thus clone handlers shall do dev_ref(). Due to races with cleanup, you should use MAKEDEV_REF flag for make_dev_credv(9) KPI instead of doing make_dev()/dev_ref() pair. I need to support FreeBSD going all the way back to 6, so that's not an option in some versions. But, I'm talking about device removal time. If I call clone_cleanup() where the clones have dev-si_refcount==1, then I get the use-after-free panic. If I hack things to elevate the reference count (such that dev-si_refcount==2 when clone_cleanup() is called), then I don't get the panic. Are you saying I should have been taking the extra reference via my dev_clone eventhandler? Won't having the extra reference lead to a memory leak? Or am I just mis-reading the code, and this will lead to things being freed normally? That said, do you really need clones at all ? I need to support FreeBSD back to 6.x, and I need to support the linux-like model of opening the same /dev/node multiple times and getting unique handles. So I think I need clones. Thanks for the help! Drew ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
Re: devfs panic w/INVARIANTS
On Fri, Feb 05, 2010 at 08:51:25AM -0500, Andrew Gallatin wrote: Kostik Belousov wrote: On Thu, Feb 04, 2010 at 03:40:28PM -0500, Andrew Gallatin wrote: I've got a commercial driver that uses device cloning. At unload time, the driver calls clone_cleanup(). When I unload the driver when the kernel is built with INVARIANTS, I'll see a panic in devfs_populate_loop(). This happens in 6-stable, as well as 8-stable. From what I can see the clone has been freed, but it remains on the devfs cdevp_list. Then the next time devfs_populate_loop() is called, it trips over the bad entry (cdp-cdp_dirents points to 0xdeadc0dedeadc0de) See appended kgdb session. If I trace the code path, it looks like clone_cleanup() calls destroy_devl(). And destroy_devl() will eventually call devfs_free() if the si_refcnt is zero. But I don't see anything which will get the cdev removed from the cdevp_list prior to it being freed. The only code I see which will get the cdev removed from the cdevp_list() seems to be the GC any lingering devices block in devfs_populate_loop What am I missing? You did not mentioned it, but my guess is that you create clones from the dev_clone event handler. Please note that devfs_lookup() that fires Yes, I do. dev_clone event, consumes a device reference. Thus clone handlers shall do dev_ref(). Due to races with cleanup, you should use MAKEDEV_REF flag for make_dev_credv(9) KPI instead of doing make_dev()/dev_ref() pair. I need to support FreeBSD going all the way back to 6, so that's not an option in some versions. But, I'm talking about device removal time. If I call clone_cleanup() where the clones have dev-si_refcount==1, then I get the use-after-free panic. If I hack things to elevate the reference count (such that dev-si_refcount==2 when clone_cleanup() is called), then I don't get the panic. Are you saying I should have been taking the extra reference via my dev_clone eventhandler? Won't having the extra reference lead to a memory leak? Or am I just mis-reading the code, and this will lead to things being freed normally? Yes, clone handler shall do dev_ref(). Either by doing race-free make_dev_credf(MAKEDEV_REF) call, or by using dev_ref() after make_dev(). That said, do you really need clones at all ? I need to support FreeBSD back to 6.x, and I need to support the linux-like model of opening the same /dev/node multiple times and getting unique handles. So I think I need clones. Wouldn't it be cleaner to use cdevpriv for the 7/8/HEAD where it is present ? And have special #ifdef-ed code for 6, that could be eventually dropped. pgp4s5tFWCcUK.pgp Description: PGP signature
Re: devfs panic w/INVARIANTS
Kostik Belousov wrote: On Fri, Feb 05, 2010 at 08:51:25AM -0500, Andrew Gallatin wrote: Kostik Belousov wrote: On Thu, Feb 04, 2010 at 03:40:28PM -0500, Andrew Gallatin wrote: I've got a commercial driver that uses device cloning. At unload time, the driver calls clone_cleanup(). When I unload the driver when the kernel is built with INVARIANTS, I'll see a panic in devfs_populate_loop(). This happens in 6-stable, as well as 8-stable. From what I can see the clone has been freed, but it remains on the devfs cdevp_list. Then the next time devfs_populate_loop() is called, it trips over the bad entry (cdp-cdp_dirents points to 0xdeadc0dedeadc0de) See appended kgdb session. If I trace the code path, it looks like clone_cleanup() calls destroy_devl(). And destroy_devl() will eventually call devfs_free() if the si_refcnt is zero. But I don't see anything which will get the cdev removed from the cdevp_list prior to it being freed. The only code I see which will get the cdev removed from the cdevp_list() seems to be the GC any lingering devices block in devfs_populate_loop What am I missing? You did not mentioned it, but my guess is that you create clones from the dev_clone event handler. Please note that devfs_lookup() that fires Yes, I do. dev_clone event, consumes a device reference. Thus clone handlers shall do dev_ref(). Due to races with cleanup, you should use MAKEDEV_REF flag for make_dev_credv(9) KPI instead of doing make_dev()/dev_ref() pair. I need to support FreeBSD going all the way back to 6, so that's not an option in some versions. But, I'm talking about device removal time. If I call clone_cleanup() where the clones have dev-si_refcount==1, then I get the use-after-free panic. If I hack things to elevate the reference count (such that dev-si_refcount==2 when clone_cleanup() is called), then I don't get the panic. Are you saying I should have been taking the extra reference via my dev_clone eventhandler? Won't having the extra reference lead to a memory leak? Or am I just mis-reading the code, and this will lead to things being freed normally? Yes, clone handler shall do dev_ref(). Either by doing race-free make_dev_credf(MAKEDEV_REF) call, or by using dev_ref() after make_dev(). OK, cool. The man pages are handy. When I started this back in the FreeBSD 5 days, the man pages didn't exist :) That said, do you really need clones at all ? I need to support FreeBSD back to 6.x, and I need to support the linux-like model of opening the same /dev/node multiple times and getting unique handles. So I think I need clones. Wouldn't it be cleaner to use cdevpriv for the 7/8/HEAD where it is present ? And have special #ifdef-ed code for 6, that could be eventually dropped. Yes, the cdevpriv() is a much cleaner interface. I'll probably add support for that soon. Thanks for the help, Drew ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
ps time field jumps backward
Hi all, For most of 7.2, on up to a 7.3-PRERELEASE built yesterday, I've noticed that the time field reported by ps and top jumps around for some processes. I've particularly noticed it with MySQL. Here are some repeated ps results (ps axo pid,time,wchan,comm) for the same process over a few minutes: 1647 0:08.06 ucond mysqld 1647 0:08.06 ucond mysqld 1647 0:08.06 ucond mysqld 1647 0:08.06 ucond mysqld 1647 0:08.07 ucond mysqld 1647 0:10.24 wdrain mysqld 1647 0:08.08 ucond mysqld 1647 0:09.25 - mysqld 1647 0:08.08 ucond mysqld 1647 0:08.91 - mysqld 1647 0:09.13 wdrain mysqld 1647 0:09.76 wdrain mysqld 1647 0:08.10 ucond mysqld 1647 0:08.11 ucond mysqld 1647 0:08.12 ucond mysqld 1647 0:08.12 ucond mysqld 1647 0:08.12 ucond mysqld 1647 0:08.12 ucond mysqld 1647 0:08.38 wdrain mysqld 1647 0:08.13 ucond mysqld 1647 0:09.47 - mysqld 1647 0:08.13 ucond mysqld 1647 0:08.13 ucond mysqld It's like when it starts working it builds up time, but as soon as it goes back to ucond, it resets back to the earlier value. This particular machine is an 8-core amd64, but I've seen it on quad-core i386 as well. Does anybody have an idea what's going on here? Thanks for any suggestions! ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
Re: our little daemon abused as symbol of the evil
Hi Kirk, Christoph, Hackers, Julian H. Stacey wrote: Kirk McKusick wrote: Thanks for the pointer. As you note, the damage (or benefit :-) is done. Still I have sent an email to the editor at Spiegel notifying them of my copyright in the hopes that they will at least ask in the future. Kirk McKusick Good idea. You might want to contact authors of that PDF paper too. In case, as my browser still is fails on URL I posted earlier: http://www.iseclab.org/papers/sonda-TR.pdf I'll send you an off list copy of what I downloaded earlier at 2nd Feb 18:14 TZ=GMT+01:00. National German TV (WDR was showing the BSD Daemon graphic in close up, as part of that PDF report, just now, about 19:10 GMT+02:00 Fri 5th Feb. They quoted http://www.aktuelle-stunde.de/ I clicked to http://www.wdr.de/tv/aks/sendungsbeitraege/2010/kw05/0205/angeklickt.jsp Big Browser is watching you! Freitag, 05. Februar 2010, 18.50 - 19.30 Uhr Kein Versprecher, sondern wie es scheint, zunehmend ein Problem. Irgendwie wussten wir es ja schon immer: Wer sich viel in sozialen Netzwerken wie Facebook, SchuelerVZ, Xing, Stayfriends und Co. aufhält, der gibt so oder so schon eine Menge von und über sich preis, bewusst in der Regel. Aber wer hätte gedacht, dass man beim Ansteuern anderer Netzseiten zum gläsernen Surfer werden könnte? Genau das kann aber passieren, sagen Forscher in einer Studie. Wer sich viel in sozialen Netzwerken aufhält, muss damit rechnen, künftig beim Surfen erkannt zu werden. Jörg Schieb erklärt uns nun, was das bedeutet, wie das funktioniert und worum es eigentlich geht. Haben Sie Fragen oder Anregungen? Dann schicken Sie uns eine E-Mail. angekli...@wdr.de The BSD symbol was used IMO totaly out of all context on the TV clip. Though the Firefox symbol has more relevance. Kirk, you may want to remind the chaps in Wien/Vienna Uni who wrote the report, that the copyright on that symbol is yours, as report was written in English, no problem for you to do that. If you want to write the TV company: I'm British, I merely read German, don't write it well, There's numerous native German speakers on this list can translate to German much better than I ever could. I suspect this may be front edge of a wave, so it's perhaps worth trying to ensure the BSD daemon is not riding that wave. The more so, as if others realise this is about data harvesting, there was another German court ruling about data harvesting recently http://www.berklix.com/~jhs/free/scam_fees.html BSD being seen with harvesting not so good. PS an undefended trade mark loses its right to further defence or some such, (I'm not a lawyer). =-=-=-= From:Julian H. Stacey j...@berklix.com Date:Tue, 02 Feb 2010 19:30:29 +0100 To: Christoph Kukulies k...@kukulies.org Subject: Re: our little daemon abused as symbol of the evil Cc: freebsd-hackers@freebsd.org, Kirk McKusick mckus...@mckusick.com Organization: http://www.berklix.com BSD Unix Linux Consultancy, Munich Germany Christoph Kukulies wrote: Look here: http://www.spiegel.de/fotostrecke/fotostrecke-51396-2.html ( Well spotted Christoph ! ) For those that don't read German, tracing back, Text article starts here http://www.spiegel.de/netzwelt/web/0,1518,675395,00.html That is in German, (some might like a translator web, eg http://babelfish.org ) I did read the german article (but skipped graphics). Key paragraph: Es ist ein Horrorszenario für Datenschützer, was Thorsten Holz, Gilbert Wondracek, Engin Kirda und Christopher Kruegel in ihrem 15-seitigen Aufsatz beschreiben ( PDF-Datei hier, 803 KB): Die Experten vom Isec-Forschungslabor für IT-Sicherheit, einer Kooperation der Technischen Universität Wien, dem Institute Eurcom und der University of California, dokumentieren einen technisch eher simplen Angriff, der eine seit zehn Jahren bekannte Sicherheitslücke ausnutzt. In key para there I could click download sonda-TR.pdf (though now I can't seem to redownload http://www.iseclab.org/papers/sonda-TR.pdf ) A 15 page article in Engish. Page 4 uses the Firefox BSD logos. I havent read that English [yet], but with it, any interested here can now read form own opinions if it seems fair to use the Daemon logo, especially cc'd copyright holder of BSD daemon holder: Kirk McKusick mckus...@mckusick.com IMO The German article by weekly magazine Spiegel.de didnt really seem to have anything to do with BSD, they just copied the graphics. Personaly my 2c: Initial reaction was I'd be a happier if a generic PC graphic had been used in the spiegel.de web, but maybe its the price of fame, I guess tests were done using BSD, Spiegel
Re: our little daemon abused as symbol of the evil
I have gotten word from the authors that they are aware of the problem and are correcting it (e.g., taking out the daemon). Kirk McKusick =-=-=-= From:Engin Kirda e...@iseclab.org Date:Wed, 3 Feb 2010 19:03:49 +0100 To: mckus...@mckusick.com Subject: BSD logo misuse Cc: Gilbert Wondracek gilb...@iseclab.org, Thorsten Holz t...@iseclab.org, Christopher Kruegel ch...@cs.ucsb.edu Kirk, I colleague from Symantec pointed out the discussion about the BSD logo that we have, apparently, misused in our paper without realizing that it was the BSD logo :-/ We'd like to apologize for this. It was not intentional. The PDF we put up is a technical report and we can easily correct this. We'll make sure that we do not use it in the camera-ready version of the published paper. Best regards, --Engin ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
Re: ps time field jumps backward
In the last episode (Feb 05), Linda Messerschmidt said: For most of 7.2, on up to a 7.3-PRERELEASE built yesterday, I've noticed that the time field reported by ps and top jumps around for some processes. I've particularly noticed it with MySQL. Here are some repeated ps results (ps axo pid,time,wchan,comm) for the same process over a few minutes: 1647 0:08.06 ucond mysqld 1647 0:08.06 ucond mysqld 1647 0:08.06 ucond mysqld 1647 0:08.06 ucond mysqld 1647 0:08.07 ucond mysqld 1647 0:10.24 wdrain mysqld 1647 0:08.08 ucond mysqld 1647 0:09.25 - mysqld It's like when it starts working it builds up time, but as soon as it goes back to ucond, it resets back to the earlier value. By default, ps only lists one thread when listing threaded processes, and if it's anything like top, the CPU column seems to be randomly picked from one thread. Try running ps axHo pid,lwp,time,wchan,comm instead (add H and lwp). That will print the individual threads, plus the thread id so you can track them across runs. You should see one thread with a CPU value of 8.08 (or so), and other values for the rest. Ideally, top and ps would total up all the per-thread CPU counts when displaying the per-process numbers, but it doesn't seem to. -- Dan Nelson dnel...@allantgroup.com ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
Re: ps time field jumps backward
On Fri, Feb 5, 2010 at 4:28 PM, Dan Nelson dnel...@allantgroup.com wrote: Ideally, top and ps would total up all the per-thread CPU counts when displaying the per-process numbers, but it doesn't seem to. It does seem to total them: $ ps axHo pid,lwp,time,wchan,comm | awk '$1 == 1647' 1647 100401 0:00.63 select mysqld 1647 100466 0:11.08 sigwai mysqld 1647 100521 0:00.00 ucond mysqld $ ps axo pid,lwp,time,wchan,comm | awk '$1 == 1647' 1647 100521 0:11.71 ucond mysqld But you put me on the right track. I ran both side by side for awhile, and found that ps/top only sums up those threads that haven't already exited. I.e., once a thread has exited, it's as if its usage never happened from the perspective of ps and top's total calculation. That seems like undesirable behavior, particularly if it conceals CPU-churning behavior by short-lived threads, but possibly very hard to change. :( I wonder if the system accounting records are more accurate? Thanks for pointing me in the right direction! ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
