Re: [dtrace-discuss] dtrace wired issue
Hi Aubrey, Aubrey Li wrote: > Sorry for the delay(time difference). Now I got more details. > # truss dtrace -l > snip > mmap(0x, 4096, PROT_READ|PROT_WRITE|PROT_EXEC, > MAP_PRIVATE|MAP_ANON, 4294967295, 0) = 0xFD7FFF18 > mmap(0x0001, 24576, PROT_READ|PROT_WRITE|PROT_EXEC, > MAP_PRIVATE|MAP_ANON|MAP_ALIGN, -1, 0) = 0xFD7FFF17 > munmap(0xFD7FFF38, 32768) = 0 > getcontext(0xFD7FFFDFE9D0) > mmap(0x0001, 65536, PROT_READ|PROT_WRITE|PROT_EXEC, > MAP_PRIVATE|MAP_ANON|MAP_ALIGN, 4294967295, 0) = 0xFD7FFF15 > getrlimit(RLIMIT_STACK, 0xFD7FFFDFED30) = 0 > getpid()= 923 [922] > lwp_private(0, 0, 0xFD7FFF150200) = 0x > setustack(0xFD7FFF1502A8) > sysconfig(_CONFIG_PAGESIZE) = 4096 > sigfillset(0xFD7FFF006880) = 0 > brk(0x0041B580) = 0 > brk(0x0041F580) = 0 > getrlimit(RLIMIT_NOFILE, 0xFD7FFFDFEE80)= 0 > setrlimit(RLIMIT_NOFILE, 0xFD7FFFDFEE80)= 0 > openat(AT_FDCWD, "/dev/dtrace/provider", O_RDONLY|O_NDELAY|O_LARGEFILE) = 3 > fcntl(3, F_SETFD, 0x0001) = 0 > fstat(3, 0xFD7FFFDFE8E0)= 0 > getdents(3, 0xFD7FFF174000, 8192) (sleeping...) > ---here, dtrace sleep almost 4 minutes > ...and continue... > > > So, it's probably something obvious that I'm not seeing... What happens if you try tracing all calls made via getdents for dtrace itself, and do this instead of dtrace -l for the first time you run dtrace? max > Any thoughts? > > Thanks, > -Aubrey > > ___ dtrace-discuss mailing list dtrace-discuss@opensolaris.org
Re: [dtrace-discuss] dtrace wired issue
Hi Aubrey, Aubrey Li wrote: > Sorry for the delay(time difference). Now I got more details. > # truss dtrace -l > snip > mmap(0x, 4096, PROT_READ|PROT_WRITE|PROT_EXEC, > MAP_PRIVATE|MAP_ANON, 4294967295, 0) = 0xFD7FFF18 > mmap(0x0001, 24576, PROT_READ|PROT_WRITE|PROT_EXEC, > MAP_PRIVATE|MAP_ANON|MAP_ALIGN, -1, 0) = 0xFD7FFF17 > munmap(0xFD7FFF38, 32768) = 0 > getcontext(0xFD7FFFDFE9D0) > mmap(0x0001, 65536, PROT_READ|PROT_WRITE|PROT_EXEC, > MAP_PRIVATE|MAP_ANON|MAP_ALIGN, 4294967295, 0) = 0xFD7FFF15 > getrlimit(RLIMIT_STACK, 0xFD7FFFDFED30) = 0 > getpid()= 923 [922] > lwp_private(0, 0, 0xFD7FFF150200) = 0x > setustack(0xFD7FFF1502A8) > sysconfig(_CONFIG_PAGESIZE) = 4096 > sigfillset(0xFD7FFF006880) = 0 > brk(0x0041B580) = 0 > brk(0x0041F580) = 0 > getrlimit(RLIMIT_NOFILE, 0xFD7FFFDFEE80)= 0 > setrlimit(RLIMIT_NOFILE, 0xFD7FFFDFEE80)= 0 > openat(AT_FDCWD, "/dev/dtrace/provider", O_RDONLY|O_NDELAY|O_LARGEFILE) = 3 > fcntl(3, F_SETFD, 0x0001) = 0 > fstat(3, 0xFD7FFFDFE8E0)= 0 > getdents(3, 0xFD7FFF174000, 8192) (sleeping...) > ---here, dtrace sleep almost 4 minutes > ...and continue... > > How long does an ls -l /dev/dtrace/provider take? I doubt you are swapping with 2G of memory. You can run vmstat -S 2 to see if there is any swapping. I'll look a bit more... By the way, are you running on Indiana (just curious...). max > Any thoughts? > > Thanks, > -Aubrey > > ___ dtrace-discuss mailing list dtrace-discuss@opensolaris.org
Re: [dtrace-discuss] dtrace wired issue
Sorry for the delay(time difference). Now I got more details. # truss dtrace -l snip mmap(0x, 4096, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE|MAP_ANON, 4294967295, 0) = 0xFD7FFF18 mmap(0x0001, 24576, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE|MAP_ANON|MAP_ALIGN, -1, 0) = 0xFD7FFF17 munmap(0xFD7FFF38, 32768) = 0 getcontext(0xFD7FFFDFE9D0) mmap(0x0001, 65536, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE|MAP_ANON|MAP_ALIGN, 4294967295, 0) = 0xFD7FFF15 getrlimit(RLIMIT_STACK, 0xFD7FFFDFED30) = 0 getpid()= 923 [922] lwp_private(0, 0, 0xFD7FFF150200) = 0x setustack(0xFD7FFF1502A8) sysconfig(_CONFIG_PAGESIZE) = 4096 sigfillset(0xFD7FFF006880) = 0 brk(0x0041B580) = 0 brk(0x0041F580) = 0 getrlimit(RLIMIT_NOFILE, 0xFD7FFFDFEE80)= 0 setrlimit(RLIMIT_NOFILE, 0xFD7FFFDFEE80)= 0 openat(AT_FDCWD, "/dev/dtrace/provider", O_RDONLY|O_NDELAY|O_LARGEFILE) = 3 fcntl(3, F_SETFD, 0x0001) = 0 fstat(3, 0xFD7FFFDFE8E0)= 0 getdents(3, 0xFD7FFF174000, 8192) (sleeping...) ---here, dtrace sleep almost 4 minutes ...and continue... Any thoughts? Thanks, -Aubrey ___ dtrace-discuss mailing list dtrace-discuss@opensolaris.org
Re: [dtrace-discuss] dtrace wired issue
On Jan 15, 2008 12:44 AM, Adam Leventhal <[EMAIL PROTECTED]> wrote: > On Mon, Jan 14, 2008 at 12:52:55PM +, Sean McGrath - Sun Microsystems > Ireland wrote: > > Aubrey Li stated: > > < Every first time to run dtrace command after the system boot up, > > < It takes a very long time to get response. > > < But the second time is OK, as follows: > > < > > < # time dtrace -l > /dev/null > > < > > < real4m8.011s > > < user0m0.116s > > < sys 0m2.420s > > > > This first time is probably when the kernel is loading the dtrace modules. > > Though still seems slow, 4 minutes. > > What kind of system (cpu speed etc) is the machine ? > > The first time DTrace is loaded it needs to uncompress the CTF (type) > information for all of the currently loaded kernel modules which can take > quite a while. Four minutes does seem a bit excessive. Are you seeing any > swapping? How to check the swapping? -Aubrey ___ dtrace-discuss mailing list dtrace-discuss@opensolaris.org
Re: [dtrace-discuss] dtrace wired issue
On Jan 14, 2008 10:46 PM, Bryan Cantrill <[EMAIL PROTECTED]> wrote: > > On Mon, Jan 14, 2008 at 09:16:34PM +0800, Aubrey Li wrote: > > On Jan 14, 2008 8:52 PM, Sean McGrath - Sun Microsystems Ireland > > <[EMAIL PROTECTED]> wrote: > > > Aubrey Li stated: > > > < Every first time to run dtrace command after the system boot up, > > > < It takes a very long time to get response. > > > < But the second time is OK, as follows: > > > < > > > < # time dtrace -l > /dev/null > > > < > > > < real4m8.011s > > > < user0m0.116s > > > < sys 0m2.420s > > > > > > This first time is probably when the kernel is loading the dtrace > > > modules. > > > Though still seems slow, 4 minutes. > > > What kind of system (cpu speed etc) is the machine ? > > > > # psrinfo -vp > > The physical processor has 2 virtual processors (0 1) > > x86 (GenuineIntel 10674 family 6 model 23 step 4 clock 2400 MHz) > > Intel(r) CPU @ 2.40GHz > > > > So, I failed to understand the modules loading needs 4 minutes. > > Yes, this is definitely fishy. Is this a highly memory constrained system? # prtconf -vp | grep Mem Memory size: 2023 Megabytes > If you "modunload -i 0" enough times to get dtrace(7D) unloaded (that > is, "dtrace" doesn't appear in modinfo), does it again take 4 minutes? > As you can imagine, it's a little tough to investigate this problem because > we can't use DTrace to do it! ;) This doesn't work on my side. modinfo always shows me "dtrace". > > > > < # time dtrace -l > /dev/null > > > < > > > < real0m0.632s > > > < user0m0.075s > > > < sys 0m0.553s > > And 600+ milliseconds is still a long time. How many probes are we talking > about here? # dtrace -l | wc -l 52604 -Aubrey ___ dtrace-discuss mailing list dtrace-discuss@opensolaris.org
Re: [dtrace-discuss] destructive actions
On Mon, Jan 14, 2008 at 05:24:38PM -0800, Adam Leventhal wrote: > Just add this line to /etc/system: > > dtrace:dtrace_destructive_disallow That should have been: set dtrace:dtrace_destructive_disallow = 1 - ahl -- Adam Leventhal, FishWorkshttp://blogs.sun.com/ahl ___ dtrace-discuss mailing list dtrace-discuss@opensolaris.org
Re: [dtrace-discuss] destructive actions
On Mon, Jan 14, 2008 at 02:34:53PM -0500, Brian Kolaci wrote: > A customer would like to run DTrace on some production boxes > however they need a way to prove that no destructive actions > can be taken, not even by accident. This "proof" is required > before they can do anything on the production boxes. > > Is there an executable or possibly an RBAC controllable > action that can be done to strip the "-w" functionality from > the DTrace executable? Or possibly strip some privileges that > won't allow DTrace to do anything destructive? You can set the tunable dtrace_destructive_disallow which will do exactly what you require. Just add this line to /etc/system: dtrace:dtrace_destructive_disallow This is described in the documentation: http://wikis.sun.com/display/DTrace/Actions+and+Subroutines See toward the middle of the section titled 'Kernel Destructive Actions'. - ahl -- Adam Leventhal, FishWorkshttp://blogs.sun.com/ahl ___ dtrace-discuss mailing list dtrace-discuss@opensolaris.org
Re: [dtrace-discuss] use dtrace as a security auditing tool
陶捷 TaoJie wrote: > I've read a silder, wrote by some sun engineer, talking about some key > features and key fbt/syscall/usdt ways to monitor the user's > behavious. > Has anyone finished the whole scripts now? Is it practical in the real world? Wouldn't dtrace's design of dropping data when there's too much to avoid impacting the system conflict with the goals of recording absolutely everything that most auditing systems have? -- -Alan Coopersmith- [EMAIL PROTECTED] Sun Microsystems, Inc. - X Window System Engineering ___ dtrace-discuss mailing list dtrace-discuss@opensolaris.org
[dtrace-discuss] use dtrace as a security auditing tool
hi all: I've read a silder, wrote by some sun engineer, talking about some key features and key fbt/syscall/usdt ways to monitor the user's behavious. Has anyone finished the whole scripts now? Is it practical in the real world? Are there any references on the web? thank you kind regards, TJ ___ dtrace-discuss mailing list dtrace-discuss@opensolaris.org
Re: [dtrace-discuss] dtrace wired issue
Hi James, James C. McPherson wrote: > Aubrey Li wrote: > >> On Jan 14, 2008 8:52 PM, Sean McGrath - Sun Microsystems Ireland >> <[EMAIL PROTECTED]> wrote: >> >>> Aubrey Li stated: >>> < Every first time to run dtrace command after the system boot up, >>> < It takes a very long time to get response. >>> < But the second time is OK, as follows: >>> < >>> < # time dtrace -l > /dev/null >>> < >>> < real4m8.011s >>> < user0m0.116s >>> < sys 0m2.420s >>> >>> This first time is probably when the kernel is loading the dtrace modules. >>> Though still seems slow, 4 minutes. >>> What kind of system (cpu speed etc) is the machine ? >>> >> # psrinfo -vp >> The physical processor has 2 virtual processors (0 1) >> x86 (GenuineIntel 10674 family 6 model 23 step 4 clock 2400 MHz) >> Intel(r) CPU @ 2.40GHz >> >> So, I failed to understand the modules loading needs 4 minutes. >> > > > If you run "dtrace -l" with no args, *every* single loadable > module on the system will be loaded, interrogated by dtrace > and then unloaded if possible. > You mean every kernel module currently loaded will be interrogated. Not every kernel module on the system. So _init/_info/_fini/attach/detach should not enter into this. max > All those attach()es and detach()es need time, as does the > probe collation. > > > > James C. McPherson > -- > Senior Kernel Software Engineer, Solaris > Sun Microsystems > http://blogs.sun.com/jmcp http://www.jmcp.homeunix.com/blog > ___ > dtrace-discuss mailing list > dtrace-discuss@opensolaris.org > > ___ dtrace-discuss mailing list dtrace-discuss@opensolaris.org
[dtrace-discuss] Kernel's microstat accounting doesn't work in this situation, doesn't it? (Using DTrace as the trigger)
Hi all:
I'm working on revealing system performance now.
My testing program is an infinite loop. Inside the loop, it will do some
mathematical opertaions and call function callee(), then go to the next
loop.
I install a alarm(30) in the program. It means it will stop after running 30
seconds.
Before calling alarm(30), at the beginning of the main body, it will use
libkstat to read kstat-cpu-sys record first.
After the alarm signal raised, in the signal handler, it will re-read the
kstat-cpu-sys record, and show the difference in this 30 seconds period.
For example, following is output printed on the screen,
*30.00 secs, 0.00 events per secs
user=99.7279%, kernel=0.2721%, idle=0.%
per second: intr=308.64, ithr=208.63, syscall=1.80, csw=78.14
per second: smtx=0.33, srw=0.00
per second: migr=0.00
*//Events are some syscalls or some user-defined functions.
In a word, this program looks very like the mpstat.
One of my testcases is running this program with dtrace's fasttrap provider,
such as:
dtrace -n 'pid$targer::callee:entry{ ... }' -c my_program
dtrace -n 'test$target:::callee{ ... }' -c my_program
The output is very strange.
*30.00 secs, 213099.33 per secs
user=99.6749%, kernel=0.3251%, idle=0.%
per second: intr=513.15, ithr=312.13, syscall=48.67, csw=87.27
per second: smtx=0.10, srw=0.00
per second: migr= 0.00*
In the infinite loop, callee would be called more than 200 times. Each
time it is called, the program would trap into kernel because it is
monitored by dtrace.
Compared with the program running with no dtrace,
*30.00 secs, 100628.74 per secs
user=99.7396%, kernel=0.2604%, idle=0.%
per second: intr=308.89, ithr=208.89, syscall=3.13, csw=73.24
per second: smtx=0.17, srw=0.00
per second: migr= 0.00
*
No big difference between these 2 kernel time percentages!
I think this is unbelievable! And maybe the answer is microstat accounting
system cannot log this kind of interrupt.
Since I try this program on Intel P4 platform. In i86pc, dtrace using "int
$3" to help the user program to trap into dtrace kernel module.
So, does it means that the mircostat accouting does not work in "int $3" and
some other cases?
Am I right?
P.S. I'm using ON build 74.
Thank you.
Kind Regards,
TJ
___
dtrace-discuss mailing list
dtrace-discuss@opensolaris.org
Re: [dtrace-discuss] destructive actions
Peter Shoults wrote: > Brian Kolaci wrote: >> Chip Bennett wrote: >> >>> Chip Bennett wrote: >>> According to the manual, if you give all of the dtrace privs to a normal user, that user can access everything except destructive actions, which is still reserved for the super user. As such a user a ran a script that did a "chip(10)", and got the following error: dtrace: error on enabled probe ID 3 (ID 13: syscall::read:return): invalid kernel access in action #2 >>> Pardon me, That should have been a "chill(10)". Sorry. >>> >>> Chip (not 10) ;-) >>> >> >> Thanks alot. >> Its been a long time since I've read the manual. Should have known >> that... >> >> ___ >> dtrace-discuss mailing list >> dtrace-discuss@opensolaris.org >> > > > Also, you would have to start the dtrace sccript with a -w option, if > you do not, the destructive actions do not take. Actually, I think the > script will not even run Yep, they knew that. But audit wants to make sure they can't even do it by mistake (for example, hidden in the first line of a D script). ___ dtrace-discuss mailing list dtrace-discuss@opensolaris.org
Re: [dtrace-discuss] destructive actions
Brian Kolaci wrote: > Chip Bennett wrote: > >> Chip Bennett wrote: >> >>> According to the manual, if you give all of the dtrace privs to a >>> normal user, that user can access everything except destructive >>> actions, which is still reserved for the super user. As such a user a >>> ran a script that did a "chip(10)", and got the following error: >>> >>> dtrace: error on enabled probe ID 3 (ID 13: syscall::read:return): >>> invalid kernel access in action #2 >>> >>> >>> >> Pardon me, That should have been a "chill(10)". Sorry. >> >> Chip (not 10) ;-) >> > > Thanks alot. > Its been a long time since I've read the manual. Should have known that... > > ___ > dtrace-discuss mailing list > dtrace-discuss@opensolaris.org > Also, you would have to start the dtrace sccript with a -w option, if you do not, the destructive actions do not take. Actually, I think the script will not even run ___ dtrace-discuss mailing list dtrace-discuss@opensolaris.org
Re: [dtrace-discuss] destructive actions
Chip Bennett wrote: > Chip Bennett wrote: >> >> According to the manual, if you give all of the dtrace privs to a >> normal user, that user can access everything except destructive >> actions, which is still reserved for the super user. As such a user a >> ran a script that did a "chip(10)", and got the following error: >> >> dtrace: error on enabled probe ID 3 (ID 13: syscall::read:return): >> invalid kernel access in action #2 >> >> > Pardon me, That should have been a "chill(10)". Sorry. > > Chip (not 10) ;-) Thanks alot. Its been a long time since I've read the manual. Should have known that... ___ dtrace-discuss mailing list dtrace-discuss@opensolaris.org
Re: [dtrace-discuss] destructive actions
Chip Bennett wrote: > > According to the manual, if you give all of the dtrace privs to a normal > user, that user can access everything except destructive actions, which > is still reserved for the super user. As such a user a ran a script > that did a "chip(10)", and got the following error: > > dtrace: error on enabled probe ID 3 (ID 13: syscall::read:return): > invalid kernel access in action #2 > > Pardon me, That should have been a "chill(10)". Sorry. Chip (not 10) ;-) ___ dtrace-discuss mailing list dtrace-discuss@opensolaris.org
Re: [dtrace-discuss] destructive actions
Brian Kolaci wrote: > Hi, > > A customer would like to run DTrace on some production boxes > however they need a way to prove that no destructive actions > can be taken, not even by accident. This "proof" is required > before they can do anything on the production boxes. > > Is there an executable or possibly an RBAC controllable > action that can be done to strip the "-w" functionality from > the DTrace executable? Or possibly strip some privileges that > won't allow DTrace to do anything destructive? > > Thanks, > > Brian > ___ > dtrace-discuss mailing list > dtrace-discuss@opensolaris.org > According to the manual, if you give all of the dtrace privs to a normal user, that user can access everything except destructive actions, which is still reserved for the super user. As such a user a ran a script that did a "chip(10)", and got the following error: dtrace: error on enabled probe ID 3 (ID 13: syscall::read:return): invalid kernel access in action #2 Chip ___ dtrace-discuss mailing list dtrace-discuss@opensolaris.org
[dtrace-discuss] destructive actions
Hi, A customer would like to run DTrace on some production boxes however they need a way to prove that no destructive actions can be taken, not even by accident. This "proof" is required before they can do anything on the production boxes. Is there an executable or possibly an RBAC controllable action that can be done to strip the "-w" functionality from the DTrace executable? Or possibly strip some privileges that won't allow DTrace to do anything destructive? Thanks, Brian ___ dtrace-discuss mailing list dtrace-discuss@opensolaris.org
Re: [dtrace-discuss] USDT: specify a module
On Mon, Jan 14, 2008 at 07:46:31PM +0100, Paul van den Bogaard wrote: > If I understand you correctly it is not possible to specify a module > name. If this is the case than indeed it seems a good thing to remove > redundancy in the listing of probes (meaning removing the module name). > On the other hand there is now an artificial (driven by > implementation) separation between "system" probes and USDTs. My > initial feeling for this is not too positive. But than I do not know > the motivations that restulted in this decision. How is there a difference between system providers and USDT providers? Probes in the kernel reflect their containing kernel module; probes in user-land reflect their containing loadable object. > The thing that I would like though is the ability to specify probes > that have the same probename (all in the same provider). The > probename is in some situation a local thing. If I enter a function > it is that function I enter. So provider::function1:enter is the > unique key for it. However this is not possible. Being forced to use > provider::function1:enter1, provider::function2:enter2 etc is too > artificial. Here internal implementation things force me in a > direction I do not want. Implementing (informational) DTrace static > probes is not something done lightly. Although the syntax for > implementing them is rather straight forward the selection of proper > names in proper places is hard. I do not like it that it becomes even > more tougher due to these implementation constraints. Is this because you want to have different arguments to the two different 'entry' probes? You could effect that by using the old DTRACE_PROBE*() syntax which provides no mechanism for type checking. Again, it would be helpful to understand what you're trying to do. Adam -- Adam Leventhal, FishWorkshttp://blogs.sun.com/ahl ___ dtrace-discuss mailing list dtrace-discuss@opensolaris.org
Re: [dtrace-discuss] dtrace wired issue
On Mon, Jan 14, 2008 at 06:46:49AM -0800, Bryan Cantrill wrote:
> > So, I failed to understand the modules loading needs 4 minutes.
>
> Yes, this is definitely fishy. Is this a highly memory constrained system?
> If you "modunload -i 0" enough times to get dtrace(7D) unloaded (that
> is, "dtrace" doesn't appear in modinfo), does it again take 4 minutes?
> As you can imagine, it's a little tough to investigate this problem because
> we can't use DTrace to do it! ;)
Kicking the dtrace kernel module out is a bit trickier than back in the day
due to this bug:
6282201 USDT probes can prevent dtrace(7D) from being unloaded
In order to unload dtrace, you'll need to make sure there are no processes
with USDT probes. This is done as part of one of the DTrace test suite
tests:
--8<-- usr/src/cmd/dtrace/test/tst/common/predicates/tst.predcache.ksh --8<--
28 unload()
29 {
30 #
31 # Get the list of services whose processes have USDT probes. Ideally
32 # it would be possible to unload the fasttrap provider while USDT
33 # probes exist -- once that fix is integrated, this hack can go away
34 # We create two lists -- one of regular SMF services and one of legacy
35 # services -- since each must be enabled and disabled using a specific
36 # mechanism.
37 #
38 pids=$(dtrace -l | \
39 perl -ne 'print "$1\n" if (/^\s*\S+\s+\S*\D(\d+)\s+/);' | \
40 sort | uniq | tr '\n' ',')
41
42 ctids=$(ps -p $pids -o ctid | tail +2 | sort | uniq)
43 svcs=
44 lrcs=
45
46 for ct in $ctids
47 do
48 line=$(svcs -o fmri,ctid | grep " $ct\$")
49 svc=$(echo $line | cut -d' ' -f1)
50
51 if [[ $(svcs -Ho STA $svc) == "LRC" ]]; then
52 lrc=$(svcs -Ho SVC $svc | tr _ '?')
53 lrcs="$lrcs $lrc"
54 else
55 svcs="$svcs $svc"
56 fi
57 done
58
59 for svc in $svcs
60 do
61 svcadm disable -ts $svc
62 done
63
64 for lrc in $lrcs
65 do
66 #
67 # Does it seem a little paternalistic that lsvcrun requires
68 # this environment variable to be set? I'd say so...
69 #
70 SMF_RESTARTER=svc:/system/svc/restarter:default \
71 /lib/svc/bin/lsvcrun $lrc stop
72 done
73
74 modunload -i 0
75 modunload -i 0
76 modunload -i 0
77 modinfo | grep dtrace
78 success=$?
79
80 for svc in $svcs
81 do
82 svcadm enable -ts $svc
83 done
84
85 for lrc in $lrcs
86 do
87 SMF_RESTARTER=svc:/system/svc/restarter:default \
88 /lib/svc/bin/lsvcrun $lrc start
89 done
90
91 if [ ! $success ]; then
92 echo $tst: could not unload dtrace
93 exit 1
94 fi
95 }
--8<-- usr/src/cmd/dtrace/test/tst/common/predicates/tst.predcache.ksh --8<--
Adam
--
Adam Leventhal, FishWorkshttp://blogs.sun.com/ahl
___
dtrace-discuss mailing list
dtrace-discuss@opensolaris.org
Re: [dtrace-discuss] USDT: specify a module
Hi Paul,
We made a decision early on to keep the module and function components
uncostomizable and tied to the actual shared object and function which
contain the USDT probes. Some people have found it annoying that dtrace -l
displays what are essentially implementation details, and to that end
we've discussed hiding by default the components of a provider according
to some stability settings. In other words, dtrace -l would only list the
elements of the probe tuple that should be specified by the user (e.g.
plockstat123:::mutex-acquire not plockstat123:libc.so.1::mutex-acquire).
If you give a more concrete description of what you're trying to do, we may
be able to suggest a solution that fits with DTrace conventions.
Adam
On Mon, Jan 14, 2008 at 07:13:41AM -0800, Paul van den Bogaard wrote:
> A .d file defining a provider has an element called provider. This is the
> first element of the four tuple that uniquely defines a probe. The others are
> module name, function name and probe name.
>
> Assuming the following content
>
> provider myprov {
> probe enterA;
> probe enterB;
> }
>
>
> Doing a listing of all probes matching myprov give me something like
>
> myprov1234:myprov:function1:enterA
> myprov1234:myprov:function2:enterB
>
> Here I see that provider and module have too much redundancy in their naming.
> I would love to specify a different module name, in order to create a
> hierarchy or separation of roles. However I see no way to define the name for
> my module. Can this be done? If so, how?
>
> Also feel the above provider specification is bad practice from at least a
> readability point of view. Both enterA and enterB should be called enter. The
> actual probes are still uniquely defined, since the function names are
> different. However running
>
> provider myprov {
> probe enter;
> probe enter;
> };
>
> through dtrace to get an object file gets me an error: already defined.
>
> Even if they have different paramets sets and/or types this "already defined"
> pops up.
> Can this be resolved?
>
> Thanks
> Paul
>
>
> --
> This message posted from opensolaris.org
> ___
> dtrace-discuss mailing list
> dtrace-discuss@opensolaris.org
--
Adam Leventhal, FishWorkshttp://blogs.sun.com/ahl
___
dtrace-discuss mailing list
dtrace-discuss@opensolaris.org
Re: [dtrace-discuss] dtrace wired issue
On Mon, Jan 14, 2008 at 12:52:55PM +, Sean McGrath - Sun Microsystems Ireland wrote: > Aubrey Li stated: > < Every first time to run dtrace command after the system boot up, > < It takes a very long time to get response. > < But the second time is OK, as follows: > < > < # time dtrace -l > /dev/null > < > < real4m8.011s > < user0m0.116s > < sys 0m2.420s > > This first time is probably when the kernel is loading the dtrace modules. > Though still seems slow, 4 minutes. > What kind of system (cpu speed etc) is the machine ? The first time DTrace is loaded it needs to uncompress the CTF (type) information for all of the currently loaded kernel modules which can take quite a while. Four minutes does seem a bit excessive. Are you seeing any swapping? Adam -- Adam Leventhal, FishWorkshttp://blogs.sun.com/ahl ___ dtrace-discuss mailing list dtrace-discuss@opensolaris.org
[dtrace-discuss] USDT: specify a module
A .d file defining a provider has an element called provider. This is the first
element of the four tuple that uniquely defines a probe. The others are module
name, function name and probe name.
Assuming the following content
provider myprov {
probe enterA;
probe enterB;
}
Doing a listing of all probes matching myprov give me something like
myprov1234:myprov:function1:enterA
myprov1234:myprov:function2:enterB
Here I see that provider and module have too much redundancy in their naming. I
would love to specify a different module name, in order to create a hierarchy
or separation of roles. However I see no way to define the name for my module.
Can this be done? If so, how?
Also feel the above provider specification is bad practice from at least a
readability point of view. Both enterA and enterB should be called enter. The
actual probes are still uniquely defined, since the function names are
different. However running
provider myprov {
probe enter;
probe enter;
};
through dtrace to get an object file gets me an error: already defined.
Even if they have different paramets sets and/or types this "already defined"
pops up.
Can this be resolved?
Thanks
Paul
--
This message posted from opensolaris.org
___
dtrace-discuss mailing list
dtrace-discuss@opensolaris.org
Re: [dtrace-discuss] dtrace wired issue
On Mon, Jan 14, 2008 at 09:16:34PM +0800, Aubrey Li wrote: > On Jan 14, 2008 8:52 PM, Sean McGrath - Sun Microsystems Ireland > <[EMAIL PROTECTED]> wrote: > > Aubrey Li stated: > > < Every first time to run dtrace command after the system boot up, > > < It takes a very long time to get response. > > < But the second time is OK, as follows: > > < > > < # time dtrace -l > /dev/null > > < > > < real4m8.011s > > < user0m0.116s > > < sys 0m2.420s > > > > This first time is probably when the kernel is loading the dtrace modules. > > Though still seems slow, 4 minutes. > > What kind of system (cpu speed etc) is the machine ? > > # psrinfo -vp > The physical processor has 2 virtual processors (0 1) > x86 (GenuineIntel 10674 family 6 model 23 step 4 clock 2400 MHz) > Intel(r) CPU @ 2.40GHz > > So, I failed to understand the modules loading needs 4 minutes. Yes, this is definitely fishy. Is this a highly memory constrained system? If you "modunload -i 0" enough times to get dtrace(7D) unloaded (that is, "dtrace" doesn't appear in modinfo), does it again take 4 minutes? As you can imagine, it's a little tough to investigate this problem because we can't use DTrace to do it! ;) > > < # time dtrace -l > /dev/null > > < > > < real0m0.632s > > < user0m0.075s > > < sys 0m0.553s And 600+ milliseconds is still a long time. How many probes are we talking about here? - Bryan -- Bryan Cantrill, Sun Microsystems FishWorks. http://blogs.sun.com/bmc ___ dtrace-discuss mailing list dtrace-discuss@opensolaris.org
Re: [dtrace-discuss] DTrace in Perl: What probes should we have?
> >> Again, for the tool we are building relying on signal trickery is > >> not ideal, but > >> we can survive. If, however, this approach of "transferring" control > >> back and forth > >> between userspace and kernel can be generalized -- I think everybody > >> would > >> benefit from such a mechanism. > > > > Again, this just isn't the way the world works -- when you are deep > > in the context of the kernel, you cannot simply revector control to > > user-land for arbitrary execution. > > I don't think you followed my example (the actual D code). If you > did you would have seen that I specifically picked a kernel level > probe (syscall::read) to illustrate how our crude but useful mechanism > works. Yes, but you already have -- and have long had -- a mechanism to do exactly this: /proc. If you would like to stop a process on every system call -- or more generally if you are more interested in stopping and controlling processes than in observing their dynamic interactions with the system at-large -- use /proc and be done with it. > Nobody is suggesting immediately revectoring the control > flow. As in my example a signal doesn't get delivered immediately. > However it is guaranteed to be delivered before the process in question > gets any chance to chug forward. I agree with the points you made later > in your email, that instrumenting delicate places deep inside the kernel > can not rely on such vectoring of the control flow. I agree with that. > The same way these places can not rely on, lets say system(..) > destructive action. Yet I can use system(...) in such a handler, > can't I? Yes, but system() doesn't the work the way you appear to think that it does: system() merely records which command you wish to be executed -- the actual execution of that command does not occur until that datum is processed (asynchronously) back in user-land. The target process is not stopped, and the command certainly does not execute synchronously with respect to probe context (and nor could it). > >> Ok, let me try to be extra clear: what I need is a mechanism for > >> passing the control flow back and forth between the kernel executing > >> the action statements and a particular userspace process in the > >> system. > > > > You're asking for the operating system equivalent of a perpetual > > motion > > machine. > > Analogies cut both ways. You of all people should know that. Trying > to invent a perpetual motion machine is, perhaps unwise. Talking about > it and carrying out though experiments proved to be quite beneficial to > physics. I'm not sure how much grant money you could get to investigate perpetual motion machines, but you'd like to spend your time and energy on the software equivalent, don't let me get in your way... > > It's not a "potentially good idea", but rather a traditional > > debugger-centric > > notion that reflects a fundamental misunderstanding of the constraints > > of DTrace. Which is not to say that one couldn't implement a > > system around > > these notions, but rather that such a system would be so different > > as to > > longer be DTrace. Indeed, it its features and limitations, it > > would look > > much more like a conventional, process-oriented debugger -- which > > should be > > no surprise given your background and bias... > >Sorry. For a minute there I forgot that DTrace was perfect. And it > is just > silly me trying to do stupid things (like zeroing out struct elements > or even > strings) and asking questions that don't even deserve an answer. Sorry. > It won't happen again. I was merely pointing out that we have different constraints on our problem than you find on more traditional debuggers. Those different constraints convey both strength and weakness: on the one hand, DTrace is able to instrument contexts and answer system-level questions for which traditional debuggers are useless, but on the other, DTrace has a much more difficult time ascertaining information that is immediately useful to the developer. And if I was curt, it was out of frustration that the misunderstanding of those constraints was coupled with an unyielding assertion of a notion that completely violates them... - Bryan -- Bryan Cantrill, Sun Microsystems FishWorks. http://blogs.sun.com/bmc ___ dtrace-discuss mailing list dtrace-discuss@opensolaris.org
Re: [dtrace-discuss] dtrace wired issue
On Jan 14, 2008 9:38 PM, James C. McPherson <[EMAIL PROTECTED]> wrote: > Aubrey Li wrote: > > On Jan 14, 2008 9:26 PM, Aubrey Li <[EMAIL PROTECTED]> wrote: > >> On Jan 14, 2008 9:22 PM, James C. McPherson <[EMAIL PROTECTED]> wrote: > > >>> If you run "dtrace -l" with no args, *every* single loadable > >>> module on the system will be loaded, interrogated by dtrace > >>> and then unloaded if possible. > >>> > >>> All those attach()es and detach()es need time, as does the > >>> probe collation. > >>> > >> So may I ask, how long "dtrace -l" get response on your system? > >> And how fast the cpu speed on your system? > >> > >> 4 minutes, it is absolutely acceptable for me. > >> > > err, sorry, I mean unacceptable, ;-) > > (figured that :) > > *** On a u20m2 > > $ uname -a > SunOS farnarkle 5.11 snv_77 i86pc i386 i86pc > $ su root -c 'time dtrace -l > /dev/null' > Password: > > real2.2 > user0.1 > sys 1.0 > $ psrinfo -v > Status of virtual processor 0 as of: 01/14/2008 23:32:17 >on-line since 01/08/2008 09:11:53. >The i386 processor operates at 2200 MHz, > and has an i387 compatible floating point processor. > Status of virtual processor 1 as of: 01/14/2008 23:32:17 >on-line since 01/08/2008 09:11:56. >The i386 processor operates at 2200 MHz, > and has an i387 compatible floating point processor. > > > > *** on a u20 > > # time dtrace -l > /dev/null > > real0m0.219s > user0m0.063s > sys 0m0.089s > # psrinfo -v > Status of virtual processor 0 as of: 01/14/2008 23:33:12 >on-line since 01/14/2008 22:43:46. >The i386 processor operates at 2412 MHz, > and has an i387 compatible floating point processor. > Status of virtual processor 1 as of: 01/14/2008 23:33:12 >on-line since 01/14/2008 22:43:50. >The i386 processor operates at 2412 MHz, > and has an i387 compatible floating point processor. > > # uname -a > SunOS crashburn 5.11 snv_80 i86pc i386 i86pc > > > > > > *** on a u60 > > # time dtrace -l > /dev/null > > real4.4 > user0.9 > sys 3.2 > # psrinfo -v > Status of virtual processor 0 as of: 01/14/2008 23:47:33 >on-line since 09/14/2007 21:44:39. >The sparcv9 processor operates at 296 MHz, > and has a sparcv9 floating point processor. > Status of virtual processor 2 as of: 01/14/2008 23:47:33 >on-line since 09/14/2007 21:44:41. >The sparcv9 processor operates at 296 MHz, > and has a sparcv9 floating point processor. > > # uname -a > SunOS synergise 5.11 snv_71 sun4u sparc SUNW,Ultra-60 > > > > > You seem to be convinced that cpu is the limiting > factor. Why? No, I want to say, if your cpu is not faster than mine, but dtrace response is faster than mine, then your explanation snip All those attach()es and detach()es need time, as does the probe collation. snip is not reasonable. So, there must be something wrong on my system, but I can't figure it out so far. Thanks, -Aubrey ___ dtrace-discuss mailing list dtrace-discuss@opensolaris.org
Re: [dtrace-discuss] dtrace wired issue
Aubrey Li wrote: > On Jan 14, 2008 9:26 PM, Aubrey Li <[EMAIL PROTECTED]> wrote: >> On Jan 14, 2008 9:22 PM, James C. McPherson <[EMAIL PROTECTED]> wrote: >>> If you run "dtrace -l" with no args, *every* single loadable >>> module on the system will be loaded, interrogated by dtrace >>> and then unloaded if possible. >>> >>> All those attach()es and detach()es need time, as does the >>> probe collation. >>> >> So may I ask, how long "dtrace -l" get response on your system? >> And how fast the cpu speed on your system? >> >> 4 minutes, it is absolutely acceptable for me. >> > err, sorry, I mean unacceptable, ;-) (figured that :) *** On a u20m2 $ uname -a SunOS farnarkle 5.11 snv_77 i86pc i386 i86pc $ su root -c 'time dtrace -l > /dev/null' Password: real2.2 user0.1 sys 1.0 $ psrinfo -v Status of virtual processor 0 as of: 01/14/2008 23:32:17 on-line since 01/08/2008 09:11:53. The i386 processor operates at 2200 MHz, and has an i387 compatible floating point processor. Status of virtual processor 1 as of: 01/14/2008 23:32:17 on-line since 01/08/2008 09:11:56. The i386 processor operates at 2200 MHz, and has an i387 compatible floating point processor. *** on a u20 # time dtrace -l > /dev/null real0m0.219s user0m0.063s sys 0m0.089s # psrinfo -v Status of virtual processor 0 as of: 01/14/2008 23:33:12 on-line since 01/14/2008 22:43:46. The i386 processor operates at 2412 MHz, and has an i387 compatible floating point processor. Status of virtual processor 1 as of: 01/14/2008 23:33:12 on-line since 01/14/2008 22:43:50. The i386 processor operates at 2412 MHz, and has an i387 compatible floating point processor. # uname -a SunOS crashburn 5.11 snv_80 i86pc i386 i86pc *** on a u60 # time dtrace -l > /dev/null real4.4 user0.9 sys 3.2 # psrinfo -v Status of virtual processor 0 as of: 01/14/2008 23:47:33 on-line since 09/14/2007 21:44:39. The sparcv9 processor operates at 296 MHz, and has a sparcv9 floating point processor. Status of virtual processor 2 as of: 01/14/2008 23:47:33 on-line since 09/14/2007 21:44:41. The sparcv9 processor operates at 296 MHz, and has a sparcv9 floating point processor. # uname -a SunOS synergise 5.11 snv_71 sun4u sparc SUNW,Ultra-60 You seem to be convinced that cpu is the limiting factor. Why? James C. McPherson -- Senior Kernel Software Engineer, Solaris Sun Microsystems http://blogs.sun.com/jmcp http://www.jmcp.homeunix.com/blog ___ dtrace-discuss mailing list dtrace-discuss@opensolaris.org
Re: [dtrace-discuss] dtrace wired issue
Hi James, James C. McPherson wrote: > Aubrey Li wrote: > >> On Jan 14, 2008 8:52 PM, Sean McGrath - Sun Microsystems Ireland >> <[EMAIL PROTECTED]> wrote: >> >>> Aubrey Li stated: >>> < Every first time to run dtrace command after the system boot up, >>> < It takes a very long time to get response. >>> < But the second time is OK, as follows: >>> < >>> < # time dtrace -l > /dev/null >>> < >>> < real4m8.011s >>> < user0m0.116s >>> < sys 0m2.420s >>> >>> This first time is probably when the kernel is loading the dtrace modules. >>> Though still seems slow, 4 minutes. >>> What kind of system (cpu speed etc) is the machine ? >>> >> # psrinfo -vp >> The physical processor has 2 virtual processors (0 1) >> x86 (GenuineIntel 10674 family 6 model 23 step 4 clock 2400 MHz) >> Intel(r) CPU @ 2.40GHz >> >> So, I failed to understand the modules loading needs 4 minutes. >> > > > If you run "dtrace -l" with no args, *every* single loadable > module on the system will be loaded, interrogated by dtrace > and then unloaded if possible. > Are you sure of this? Why should it load everything? (And then unload them if they're not being used). # dtrace -l | wc -l 88939 # modload dedump # dtrace -l | wc -l 88963 I get different counts. I think dtrace only looks at what is currently on the system. But 4 minutes??? max > All those attach()es and detach()es need time, as does the > probe collation. > > > > James C. McPherson > -- > Senior Kernel Software Engineer, Solaris > Sun Microsystems > http://blogs.sun.com/jmcp http://www.jmcp.homeunix.com/blog > ___ > dtrace-discuss mailing list > dtrace-discuss@opensolaris.org > > ___ dtrace-discuss mailing list dtrace-discuss@opensolaris.org
Re: [dtrace-discuss] dtrace wired issue
On Jan 14, 2008 9:26 PM, Aubrey Li <[EMAIL PROTECTED]> wrote: > > On Jan 14, 2008 9:22 PM, James C. McPherson <[EMAIL PROTECTED]> wrote: > > Aubrey Li wrote: > > > On Jan 14, 2008 8:52 PM, Sean McGrath - Sun Microsystems Ireland > > > <[EMAIL PROTECTED]> wrote: > > >> Aubrey Li stated: > > >> < Every first time to run dtrace command after the system boot up, > > >> < It takes a very long time to get response. > > >> < But the second time is OK, as follows: > > >> < > > >> < # time dtrace -l > /dev/null > > >> < > > >> < real4m8.011s > > >> < user0m0.116s > > >> < sys 0m2.420s > > >> > > >> This first time is probably when the kernel is loading the dtrace > > >> modules. > > >> Though still seems slow, 4 minutes. > > >> What kind of system (cpu speed etc) is the machine ? > > > > > > # psrinfo -vp > > > The physical processor has 2 virtual processors (0 1) > > > x86 (GenuineIntel 10674 family 6 model 23 step 4 clock 2400 MHz) > > > Intel(r) CPU @ 2.40GHz > > > > > > So, I failed to understand the modules loading needs 4 minutes. > > > > > > If you run "dtrace -l" with no args, *every* single loadable > > module on the system will be loaded, interrogated by dtrace > > and then unloaded if possible. > > > > All those attach()es and detach()es need time, as does the > > probe collation. > > > So may I ask, how long "dtrace -l" get response on your system? > And how fast the cpu speed on your system? > > 4 minutes, it is absolutely acceptable for me. > err, sorry, I mean unacceptable, ;-) -Aubrey ___ dtrace-discuss mailing list dtrace-discuss@opensolaris.org
Re: [dtrace-discuss] dtrace wired issue
On Jan 14, 2008 9:22 PM, James C. McPherson <[EMAIL PROTECTED]> wrote: > Aubrey Li wrote: > > On Jan 14, 2008 8:52 PM, Sean McGrath - Sun Microsystems Ireland > > <[EMAIL PROTECTED]> wrote: > >> Aubrey Li stated: > >> < Every first time to run dtrace command after the system boot up, > >> < It takes a very long time to get response. > >> < But the second time is OK, as follows: > >> < > >> < # time dtrace -l > /dev/null > >> < > >> < real4m8.011s > >> < user0m0.116s > >> < sys 0m2.420s > >> > >> This first time is probably when the kernel is loading the dtrace > >> modules. > >> Though still seems slow, 4 minutes. > >> What kind of system (cpu speed etc) is the machine ? > > > > # psrinfo -vp > > The physical processor has 2 virtual processors (0 1) > > x86 (GenuineIntel 10674 family 6 model 23 step 4 clock 2400 MHz) > > Intel(r) CPU @ 2.40GHz > > > > So, I failed to understand the modules loading needs 4 minutes. > > > If you run "dtrace -l" with no args, *every* single loadable > module on the system will be loaded, interrogated by dtrace > and then unloaded if possible. > > All those attach()es and detach()es need time, as does the > probe collation. > So may I ask, how long "dtrace -l" get response on your system? And how fast the cpu speed on your system? 4 minutes, it is absolutely acceptable for me. Thanks, -Aubrey ___ dtrace-discuss mailing list dtrace-discuss@opensolaris.org
Re: [dtrace-discuss] dtrace wired issue
Aubrey Li wrote: > On Jan 14, 2008 8:52 PM, Sean McGrath - Sun Microsystems Ireland > <[EMAIL PROTECTED]> wrote: >> Aubrey Li stated: >> < Every first time to run dtrace command after the system boot up, >> < It takes a very long time to get response. >> < But the second time is OK, as follows: >> < >> < # time dtrace -l > /dev/null >> < >> < real4m8.011s >> < user0m0.116s >> < sys 0m2.420s >> >> This first time is probably when the kernel is loading the dtrace modules. >> Though still seems slow, 4 minutes. >> What kind of system (cpu speed etc) is the machine ? > > # psrinfo -vp > The physical processor has 2 virtual processors (0 1) > x86 (GenuineIntel 10674 family 6 model 23 step 4 clock 2400 MHz) > Intel(r) CPU @ 2.40GHz > > So, I failed to understand the modules loading needs 4 minutes. If you run "dtrace -l" with no args, *every* single loadable module on the system will be loaded, interrogated by dtrace and then unloaded if possible. All those attach()es and detach()es need time, as does the probe collation. James C. McPherson -- Senior Kernel Software Engineer, Solaris Sun Microsystems http://blogs.sun.com/jmcp http://www.jmcp.homeunix.com/blog ___ dtrace-discuss mailing list dtrace-discuss@opensolaris.org
Re: [dtrace-discuss] dtrace wired issue
On Jan 14, 2008 8:52 PM, Sean McGrath - Sun Microsystems Ireland <[EMAIL PROTECTED]> wrote: > Aubrey Li stated: > < Every first time to run dtrace command after the system boot up, > < It takes a very long time to get response. > < But the second time is OK, as follows: > < > < # time dtrace -l > /dev/null > < > < real4m8.011s > < user0m0.116s > < sys 0m2.420s > > This first time is probably when the kernel is loading the dtrace modules. > Though still seems slow, 4 minutes. > What kind of system (cpu speed etc) is the machine ? # psrinfo -vp The physical processor has 2 virtual processors (0 1) x86 (GenuineIntel 10674 family 6 model 23 step 4 clock 2400 MHz) Intel(r) CPU @ 2.40GHz So, I failed to understand the modules loading needs 4 minutes. > > < # time dtrace -l > /dev/null > < > < real0m0.632s > < user0m0.075s > < sys 0m0.553s > < > > And the second time, the dtrace modules are already loaded so less time. > This makes sense to me. > < Any clue? > < > < Thanks, > < -Aubrey > > Regards, > -- > Sean. > . > Thanks, -Aubrey ___ dtrace-discuss mailing list dtrace-discuss@opensolaris.org
Re: [dtrace-discuss] dtrace wired issue
Aubrey Li stated: < Every first time to run dtrace command after the system boot up, < It takes a very long time to get response. < But the second time is OK, as follows: < < # time dtrace -l > /dev/null < < real4m8.011s < user0m0.116s < sys 0m2.420s This first time is probably when the kernel is loading the dtrace modules. Though still seems slow, 4 minutes. What kind of system (cpu speed etc) is the machine ? < # time dtrace -l > /dev/null < < real0m0.632s < user0m0.075s < sys 0m0.553s < And the second time, the dtrace modules are already loaded so less time. < Any clue? < < Thanks, < -Aubrey Regards, -- Sean. . ___ dtrace-discuss mailing list dtrace-discuss@opensolaris.org
[dtrace-discuss] dtrace wired issue
Every first time to run dtrace command after the system boot up, It takes a very long time to get response. But the second time is OK, as follows: # time dtrace -l > /dev/null real4m8.011s user0m0.116s sys 0m2.420s # time dtrace -l > /dev/null real0m0.632s user0m0.075s sys 0m0.553s Any clue? Thanks, -Aubrey ___ dtrace-discuss mailing list dtrace-discuss@opensolaris.org
