Re: [dtrace-discuss] Is the nfs dtrace script right (from nfsv3 provider wiki)?
Hi Marcelo, Marcelo Leal wrote: > Sorry, but i do not agree. > We are talking about a NFSv3 provider, and not about how many cpu's there > are on the system. I do not have the knowledge to discuss with you the > aspects about the implementation, but as a user point of view, i think that > numbers don't make sense. If the fact that the number of cpu's is important > for the start/done for the NFS provider, i think it will for all other dtrace > providers. > I have found that almost all "bugs" with dtrace are either in the scripting or in the interpretation of the output. The mechanism used to implement dtrace is quite simple, and for the script you are running, I don't believe you are hitting any bug in the implementation of dtrace. Also, using dtrace to examine the system is like using a microscope. You need to know the big picture first, before you can determine what you need to trace. Otherwise you can end up getting a lot of detail about something that has nothing to do with the problem you are experiencing. In this instance, as Jim says, iostat will give you a better view of the big picture. As for the 20 minutes total time spent for I/O in a script running 10 minutes, this could happen even on a single CPU. For example, you start the script and immediately 10 I/O requests are made. Now, let's say all 10 requests are queued (i.e., block, waiting for some resource). If they all finish just before the end of the 10 minute period, the total elapsed time would be about 100 minutes. So long as the total time divided by the number of operations does not exceed 10 minutes, the output is reasonable. I suspect the outliers, (the I/Os that are taking 2-4 seconds) are due to queuing at the disk driver, but they could be due to scheduling as well. (I would have to look at exactly when the start and done probes fire to determine this. But fortunately, source code is available to determine this.). As I mentioned in an earlier post, speculative tracing can be used to determine the code path taken for the longest I/O. I have written a script that might work, but have no way to test it at this time. If you are interested, I'll post it. Jim, as for taking this offline, that is ok for you, but my posts to Marcelo are bouncing... max > Thanks a lot for your answers Max! > > __ > Leal > [http://www.eall.com.br/blog] > ___ dtrace-discuss mailing list dtrace-discuss@opensolaris.org
Re: [dtrace-discuss] Is the nfs dtrace script right (from nfsv3 provider wiki)?
Hi Marcelo, Marcelo Leal wrote: > Ok, but that is a bug, or should work like that? > We can not use dtrace on multiple processors systems? > Sorry, but i don't get it... > I don't consider this a bug. I think it depends on what you are trying to measure. The script you are using measures latency for read/write operations across all cpus. There is nothing wrong with the sum of the times being longer than the total time you are tracing, if there are multiple cpus. If you wanted to measure latency per cpu, the script would need to be changed. So, what are you trying to measure? I think more interesting is to find out why some I/O operations take longer than others. For this, you can use speculative tracing. Trace all function entries and returns, but only commit if the time spent is longer than the longest time found thus far. I'm sure others have ideas about this... max > __ > Leal > [http://www.eall.com.br/blog] > ___ dtrace-discuss mailing list dtrace-discuss@opensolaris.org
Re: [dtrace-discuss] Is the nfs dtrace script right (from nfsv3 provider wiki)?
Hi Marcelo, Marcelo Leal wrote: > I think (us) is microseconds. There is one division by "1000" on the source > code... > > Oops. You're right. I did not see that. (That might explain the 4-8 nanosecond I/Os, which I did think seemed pretty fast. They are actually 4-8 microsecond). So, you want to know how you can spend 19 or 20 minutes in a 10 minute trace? You have multiple cpu's, so each cpu can be working in parallel on different I/O requests. If you have 8 cpu's, the average time spent by each cpu would be about 2.5 minutes. This does sound a little high to me, but not extreme. If you got 80 minutes, I would be concerned that all cpus are working on requests all the time. It might be difficult to correlate per cpu, as I suspect the start and done probe for a given I/O could fire on different cpus. max > Leal > [http://www.eall.com.br/blog] > ___ dtrace-discuss mailing list dtrace-discuss@opensolaris.org
Re: [dtrace-discuss] Is the nfs dtrace script right (from nfsv3 provider wiki)?
Marcelo Leal wrote: >> Marcelo Leal wrote: >> >>> Hello all... >>> Thanks a lot for the answers! I think the problem >>> >> is "almost" fixed. Every dtrace documentation says to >> use predicates to guarantee the relation between the >> start/done probes... Max was the only one paying >> attention reading the docs. ;-) >> >>> But i'm still getting weird numbers: >>> >>> >> Which numbers don't look right? 3 of your reads took >> between 2 and 4 >> milliseconds, most were between >> 8 and 16 nanoseconds. 21 writes took between 2 and 4 >> milliseconds, the >> most amount of time >> spent doing read/write by host is about 1.2 seconds, >> and teste/file10 >> took about 1.1 seconds. >> Looks pretty good to me.(?). I'm curious about what >> you were expecting >> to see. >> > > The problem is the total numbers... > 1267135728 and 1126991407, for example. > 21 and 19 minutes in a ten minutes trace. > Or am i missing something? > When I do the arithmetic, I get about 1.2 seconds for the first number, and 1.1 seconds for the second number. These numbers are in nanoseconds, no? So, 1267135728/(1000*1000*1000) = 1.267... seconds. max > Leal > [http://www.eall.com.br/blog] > > >>> Wed Dec 10 08:36:33 BRST 2008 >>> Wed Dec 10 08:46:55 BRST 2008 >>> >>> cut here - >>> Tracing... Hit Ctrl-C to end. >>> ^C >>> NFSv3 read/write distributions (us): >>> >>> read >>>value - Distribution >>> >> - count >> >>>2 | >>> >> 0 >> 631 >> @@@ 145603 >> >>> 16 |@ >>> >>155926 >> 15970 >>6111 >> 942 >>372 >> 883 >>1649 >> 1090 >>8278 >> 24605 >>8868 >> 1694 >>304 >> 63 >>27 >> 31 >>43 >> 3 >>0 >> value - Distribution - >> count >> 128 | >> 0 >> 1083 >> @@@ 32622 >> >>> 1024 |@ >>> >>70353 >> 70851 >>47906 >> 44898 >>20481 >> 5633 >>1605 >> 1339 >>957 >> 380 >>143 >> 21 >>0 >> otal us): >> >>> x.16.0.x >>> >> 647019 >> >>> x.16.0.x >>> >> 734488 >> >>> x.16.0.x >>> >> 0890034 >> >>> x.16.0.x >>> >> 8852624 >> >>> x.16.0.x >>> >> 0407241 >> >>> x.16.0.x >>> >> 9028592 >> >>> x.16.0.x >>> >> 3013688 >> >>> x.16.0.x >>> >> 04045281 >> >>> x.16.0.x >>> >> 05245138 >> >>> x.16.0.x >>> >> 24286383 >> >>> x.16.0.x >>> >> 54526695 >> >>> x.16.0.x >>> >> 94419023 >> >>> x.16.0.x >>> >> 21794650 >> >>> x.16.0.x >>> >> 59302970 >> >>> x.16.0.x >>> >> 89694542 >> >>> x.16.0.x >>> >> 90207418 >> >>> x.16.0.x >>> >> 87983050 >> >>> x.16.0.x >>> >> 267135728 >> >>> NFSv3 read/write top 10 files (total us): >>> >>> /teste/file1 95870303 >>> /teste/file2 104212948 >>> /teste/file3 104311607 >>> /teste/file4 121076447 >>> /teste/file5 137687236 >>> /teste/file6 160895273 >>> /teste/file7 180765880 >>> /teste/file8 198827114 >>> /teste/file9 372380414 >>> /teste/file10 1126991407 >>> -- cut here -- >>> >>> >>> Max, will be difficult disable processors on that >>> >> machine (production). >> >>> >>> >> Yes. I understand. >> Regards, >> max >> >> >>> Thanks again! >>> >>> Leal >>> [http://www.eall.com.br/blog] >>> >>> >> ___ >> dtrace-discuss mailing l
Re: [dtrace-discuss] Is the nfs dtrace script right (from nfsv3 provider wiki)?
Marcelo Leal wrote: > Hello all... > Thanks a lot for the answers! I think the problem is "almost" fixed. Every > dtrace documentation says to use predicates to guarantee the relation between > the start/done probes... Max was the only one paying attention reading the > docs. ;-) > Actually, this is not from reading the docs. It is from being burnt a few times by getting "impossible" time values. By the way, I am replying to you and to the mailing list, but messages to you are getting bounced. > But i'm still getting weird numbers: > Which numbers don't look right? 3 of your reads took between 2 and 4 milliseconds, most were between 8 and 16 nanoseconds. 21 writes took between 2 and 4 milliseconds, the most amount of time spent doing read/write by host is about 1.2 seconds, and teste/file10 took about 1.1 seconds. Looks pretty good to me.(?). I'm curious about what you were expecting to see. > Wed Dec 10 08:36:33 BRST 2008 > Wed Dec 10 08:46:55 BRST 2008 > > cut here - > Tracing... Hit Ctrl-C to end. > ^C > NFSv3 read/write distributions (us): > > read >value - Distribution - count >2 | 0 >4 | 631 >8 | 145603 > 16 |@155926 > 32 |@@ 15970 > 64 |@6111 > 128 | 942 > 256 | 372 > 512 | 883 > 1024 | 1649 > 2048 | 1090 > 4096 |@8278 > 8192 |@@@ 24605 >16384 |@8868 >32768 | 1694 >65536 | 304 > 131072 | 63 > 262144 | 27 > 524288 | 31 > 1048576 | 43 > 2097152 | 3 > 4194304 | 0 > > write >value - Distribution - count > 128 | 0 > 256 | 1083 > 512 | 32622 > 1024 |@70353 > 2048 |@@ 70851 > 4096 |@@ 47906 > 8192 |@@ 44898 >16384 |@@@ 20481 >32768 |@5633 >65536 | 1605 > 131072 | 1339 > 262144 | 957 > 524288 | 380 > 1048576 | 143 > 2097152 | 21 > 4194304 | 0 > > NFSv3 read/write by host (total us): > > x.16.0.x 3647019 > x.16.0.x 8734488 > x.16.0.x 50890034 > x.16.0.x 68852624 > x.16.0.x 70407241 > x.16.0.x 79028592 > x.16.0.x 83013688 > x.16.0.x 104045281 > x.16.0.x 105245138 > x.16.0.x 124286383 > x.16.0.x154526695 > x.16.0.x 194419023 > x.16.0.x 221794650 > x.16.0.x 259302970 > x.16.0.x 289694542 > x.16.0.x 290207418 > x.16.0.x
Re: [dtrace-discuss] Is the nfs dtrace script right (from nfsv3 provider wiki)?
Hi Brendan,
Brendan Gregg - Sun Microsystems wrote:
> On Tue, Dec 09, 2008 at 05:04:49PM +0100, [EMAIL PROTECTED] wrote:
>
>> Hi,
>> I have looked at the script, and there is no correspondence between
>> start and done.
>> So, I am not sure how this script is supposed to work.
>> I think there should be a predicate in the done probes...
>> The way the script is written, it assumes that for any start, the done
>> that fires after it is for the same noi_xid.
>>
>> Current script:
>>
>> nfsv3:::op-read-start,
>> nfsv3:::op-write-start
>> {
>> start[args[1]->noi_xid] = timestamp;
>> }
>>
>> nfsv3:::op-read-done,
>> nfsv3:::op-write-done
>> {
>> ...
>>
>> Changed script:
>>
>> nfsv3:::op-read-start,
>> nfsv3:::op-write-start
>> {
>> start[args[1]->noi_xid] = timestamp;
>> }
>>
>> nfsv3:::op-read-done,
>> nfsv3:::op-write-done
>> /start[args[1]->noi_xid] != 0/
>> {
>>
>> That way, you don't have the done probe clause executing
>> for id's where the start has not fired first.
>>
>
> Thanks, unusual for me to miss that. Anyway, now fixed on the wiki.
>
>
>> (This still does not
>> match start/done for a given xid).
>> But what do I know...
>>
>
> The xid is the key in the hash...
>
>
Yes, and I think the check for non-zero is sufficient...
> Brendan
>
>
___
dtrace-discuss mailing list
dtrace-discuss@opensolaris.org
Re: [dtrace-discuss] Is the nfs dtrace script right (from nfsv3 provider wiki)?
Marcelo Leal wrote: > Oops, that would be a nice test, but something i cannot do. ;-) > What is it that you cannot do? > [http://www.eall.com.br/blog] > > Leal. > ___ dtrace-discuss mailing list dtrace-discuss@opensolaris.org
Re: [dtrace-discuss] Is the nfs dtrace script right (from nfsv3 provider wiki)?
Hi Marcelo, Marcelo Leal wrote: > Some kind of both... ;-) > I was investigating a "possible" performance problem, that i'm not sure if > is the NFS server or not. > So, i was faced with that weird numbers. I think one thing is not related > with the other, but we need to fix whatever is the problem with the script or > the provider, to have confidence on the tool. Except that numbers, the other > latency values seems to be fine, don't you agree? > Yes. And I think it is a problem with the script. Did you try modifying the script as I suggested in my earlier email? max > Leal > [http://www.eall.com.br/blog] > ___ dtrace-discuss mailing list dtrace-discuss@opensolaris.org
Re: [dtrace-discuss] Is the nfs dtrace script right (from nfsv3 provider wiki)?
Hi,
I have looked at the script, and there is no correspondence between
start and done.
So, I am not sure how this script is supposed to work.
I think there should be a predicate in the done probes...
The way the script is written, it assumes that for any start, the done
that fires after it is for the same noi_xid.
Current script:
nfsv3:::op-read-start,
nfsv3:::op-write-start
{
start[args[1]->noi_xid] = timestamp;
}
nfsv3:::op-read-done,
nfsv3:::op-write-done
{
...
Changed script:
nfsv3:::op-read-start,
nfsv3:::op-write-start
{
start[args[1]->noi_xid] = timestamp;
}
nfsv3:::op-read-done,
nfsv3:::op-write-done
/start[args[1]->noi_xid] != 0/
{
That way, you don't have the done probe clause executing
for id's where the start has not fired first. (This still does not
match start/done for a given xid).
But what do I know...
max
Jim Mauro wrote:
> Also (I meant to ask) - are you having performance problems, or
> just monitoring with the NFS provider scripts?
>
> Thanks,
> /jim
>
>
> Marcelo Leal wrote:
>
>> Hello Jim, this is not a benchmark. The filenames i did change for privacy...
>> This is a NFS server, yes.
>>
>> # uname -a
>> SunOS test 5.11 snv_89 i86pc i386 i86pc
>>
>> # cat /etc/release
>>Solaris Express Community Edition snv_89 X86
>>Copyright 2008 Sun Microsystems, Inc. All Rights Reserved.
>> Use is subject to license terms.
>> Assembled 06 May 2008
>>
>>
> ___
> dtrace-discuss mailing list
> dtrace-discuss@opensolaris.org
>
>
___
dtrace-discuss mailing list
dtrace-discuss@opensolaris.org
Re: [dtrace-discuss] Is the nfs dtrace script right (from nfsv3 provider wiki)?
Hi Marcelo and Jim, Marcelo Leal wrote: > Hello Jim, this is not a benchmark. The filenames i did change for privacy... > This is a NFS server, yes. > > # uname -a > SunOS test 5.11 snv_89 i86pc i386 i86pc > > # cat /etc/release >Solaris Express Community Edition snv_89 X86 >Copyright 2008 Sun Microsystems, Inc. All Rights Reserved. > Use is subject to license terms. > Assembled 06 May 2008 > Out of curiosity, what happens if you turn off all the processors except 1 (via psradm(1)), and then run the script? max ___ dtrace-discuss mailing list dtrace-discuss@opensolaris.org
[dtrace-discuss] tracemem again
Hi, I am using tracemem() to dump some memory contents. When the contents of the memory contain a mix of binary and ascii, the output looks like: 0 1 2 3 4 5 6 7 8 9 a b c d e f0123456789abcdef 0: 2f 79 2e 20 2e 9e 2e fc fe 2e 2e 38 2e 0a 34 38/y. ...8..48 10: 34 32 35 2c 20 72 65 6d 6f 74 65 20 69 70 3a 20 425, remote ip: 20: 31 32 37 2e 30 2e 30 2e 31 2c 20 72 65 6d 6f 74 127.0.0.1, remot ... But when the contents are all ascii, the output looks like: 10th 1st 2nd 3rd 4th 5th 6th 7th 8th 9th a AAA AAAS Aarhus Aaron ... So, any chance of getting the format to be consistent? It would make scripting much easier if I always got the output with first hex, then ascii or ".". thanks, max ___ dtrace-discuss mailing list dtrace-discuss@opensolaris.org
Re: [dtrace-discuss] tracemem question
Hi Chip,
Chip Bennett wrote:
> First, the length on tracemem is a constant because the output from one
> clause must be the same length every time it executes. With ASCII
> strings, you can fake it, because you can set the maximum length of
> strings to whatever you want. When the used length of a string varies
> each time the same clause is executed, this isn't a problem because D
> always records the full length of the string in the switch buffer, not
> just the used length. The full length never varies.
>
> However, the problem with this for binary data, is that all of the
> string format items in printf stop when a null character ('\0') is
> reached: not a good thing for binary data.
>
> I have a possible solution, but I don't think you're going to like it.
> I've thought about this before, and considered how I might solve it. If
> you know the buffer you want to trace is between say 1 and 8000 bytes,
> include enough additional probe specs and clauses for the same function
> over and over to display the whole thing, but trace it 100 bytes at a
> time. So for a maximum of 8000 bytes, you'd need 80 clauses. Then use
> a counter in a predicate to limit the number of clauses executed for
> each pass. It's crude, I know.
>
Yes, I thought of that. The problem is that I can not assume
that a given dump has, say 100 bytes in it. I would have to have a separate
probe for each byte (ugh). I think I'll end up dumping the maximum size
using tracemem, and then let an application whittle it down based
on a length I can place into the output.
> I considered a profile probe that fires over and over with a predicate
> that stops the output when the end of the buffer is reached, but the
> buffer would likely be modified before you'd get a chance to get all of
> the data traced.
>
> We need a tracemem that has two parameters: buffer len, a variable, a
> max length, a constant. Tracemem would then always record the full
> length in the switch buffer, but only the actual data would be
> displayed, along with the length.
>
Yes! That would be nice...
Thanks much,
max
> Good luck!
>
> Chip
>
>
>
>> -Original Message-
>> From: [EMAIL PROTECTED] [mailto:dtrace-discuss-
>> [EMAIL PROTECTED] On Behalf Of [EMAIL PROTECTED]
>> Sent: Friday, November 21, 2008 7:10 AM
>> To: dtrace-discuss@opensolaris.org
>> Subject: [dtrace-discuss] tracemem question
>>
>> Hi All,
>> I am sure this has been asked before (in fact, I think I asked it over
>> 2
>> years ago).
>> I am snooping tcp traffic on the e1000g using dtrace. Almost
>> everything
>> works (I print mac header,
>> ip header, and tcp header). I would like to use something like
>> tracemem() to dump the payload.
>> However, tracemem() won't let me specify anything but a constant as
>>
> the
>
>> length. Has anyone
>> succeeded in dumping an arbitrary number of hex bytes? What I want is
>> something
>> like:
>> tracemem(mp->b_rptr+offset, mp->b_wptr-(mp->b_rptr+offset));
>>
>> Or maybe there is a way to do this with printf???
>>
>> thanks,
>> max
>>
>> ___
>> dtrace-discuss mailing list
>> dtrace-discuss@opensolaris.org
>>
>
>
>
>
___
dtrace-discuss mailing list
dtrace-discuss@opensolaris.org
Re: [dtrace-discuss] Getting error ld: fatal: symbol `__SUNW_dof' is multiply-defined:
Please Ignore I linked .a's instead of individual .os, somehow it worked.
CC -G -o libD.so -mt -norunpath -L. -la -lb
--- On Tue, 11/18/08, [EMAIL PROTECTED] <[EMAIL PROTECTED]> wrote:
> From: [EMAIL PROTECTED] <[EMAIL PROTECTED]>
> Subject: [dtrace-discuss] Getting error ld: fatal: symbol `__SUNW_dof' is
> multiply-defined:
> To: dtrace-discuss@opensolaris.org
> Date: Tuesday, November 18, 2008, 12:16 PM
> I have 2 .cpp files in two different directories in which I
> need to put DTrace probes. They are compiled into two .a
> libraries one after another. In the end they are combined to
> a single .so file. This sequence I can not change. I get an
> error "ld: fatal: symbol `__SUNW_dof' is
> multiply-defined:". What is the solution for this?
>
> Here is my simulation of real world problem in sample
> files.
>
> $gmake
> CC -mt -xtarget=ultra -g -xs -c -o a.o a.cpp
> dtrace -G -32 -s 1.d a.o -o 1.o
> CC -xar -o liba.a a.o 1.o
> CC -mt -xtarget=ultra -g -xs -c -o b.o b.cpp
> dtrace -G -32 -s 2.d b.o -o 2.o
> CC -xar -o libb.a b.o 2.o
> CC -G -o libc.so a.o b.o 1.o 2.o -mt -norunpath
> ld: fatal: symbol `__SUNW_dof' is multiply-defined:
> (file 1.o type=OBJT; file 2.o type=OBJT);
> ld: fatal: File processing errors. No output written to
> libc.so
> gmake: *** [all] Error 1
>
> $cat Makefile
> all::
> CC -mt -xtarget=ultra -g -xs -c -o a.o a.cpp
> dtrace -G -32 -s 1.d a.o -o 1.o
> CC -xar -o liba.a a.o 1.o
> CC -mt -xtarget=ultra -g -xs -c -o b.o b.cpp
> dtrace -G -32 -s 2.d b.o -o 2.o
> CC -xar -o libb.a b.o 2.o
> CC -G -o libc.so a.o b.o 1.o 2.o -mt -norunpath
> clean::
> rm *.o *.a
>
> $cat 1.d
> provider sjsws1 {
> probe reached1(const char *);
> };
>
> $cat 2.d
> provider sjsws2 {
> probe reached2(const char *);
> };
>
> $cat a.cpp
> #include
> #include "sjsws.h"
>
> main()
> {
> DTRACE_REACHED1("one");
> }
>
> $cat b.cpp
> #include
> #include "sjsws.h"
>
> void myfunc(void)
> {
> DTRACE_REACHED2("two");
> }
>
> $cat sjsws.h
> #ifndef SJSWS
> #define SJSWS 1
>
> #include "sys/sdt.h"
> #ifdef __cplusplus
> extern "C" {
> #endif
>
> #define DTRACE_REACHED1(arg0) \
> __dtrace_sjsws1___reached1(arg0)
> #define DTRACE_REACHED2(arg0) \
> __dtrace_sjsws2___reached2(arg0)
>
> extern void __dtrace_sjsws1___reached1(const char *);
> extern void __dtrace_sjsws2___reached2(const char *);
>
> #ifdef __cplusplus
> }
> #endif
>
> #endif
>
>
>
>
> ___
> dtrace-discuss mailing list
> dtrace-discuss@opensolaris.org
___
dtrace-discuss mailing list
dtrace-discuss@opensolaris.org
[dtrace-discuss] Getting error ld: fatal: symbol `__SUNW_dof' is multiply-defined:
I have 2 .cpp files in two different directories in which I need to put DTrace
probes. They are compiled into two .a libraries one after another. In the end
they are combined to a single .so file. This sequence I can not change. I get
an error "ld: fatal: symbol `__SUNW_dof' is multiply-defined:". What is the
solution for this?
Here is my simulation of real world problem in sample files.
$gmake
CC -mt -xtarget=ultra -g -xs -c -o a.o a.cpp
dtrace -G -32 -s 1.d a.o -o 1.o
CC -xar -o liba.a a.o 1.o
CC -mt -xtarget=ultra -g -xs -c -o b.o b.cpp
dtrace -G -32 -s 2.d b.o -o 2.o
CC -xar -o libb.a b.o 2.o
CC -G -o libc.so a.o b.o 1.o 2.o -mt -norunpath
ld: fatal: symbol `__SUNW_dof' is multiply-defined:
(file 1.o type=OBJT; file 2.o type=OBJT);
ld: fatal: File processing errors. No output written to libc.so
gmake: *** [all] Error 1
$cat Makefile
all::
CC -mt -xtarget=ultra -g -xs -c -o a.o a.cpp
dtrace -G -32 -s 1.d a.o -o 1.o
CC -xar -o liba.a a.o 1.o
CC -mt -xtarget=ultra -g -xs -c -o b.o b.cpp
dtrace -G -32 -s 2.d b.o -o 2.o
CC -xar -o libb.a b.o 2.o
CC -G -o libc.so a.o b.o 1.o 2.o -mt -norunpath
clean::
rm *.o *.a
$cat 1.d
provider sjsws1 {
probe reached1(const char *);
};
$cat 2.d
provider sjsws2 {
probe reached2(const char *);
};
$cat a.cpp
#include
#include "sjsws.h"
main()
{
DTRACE_REACHED1("one");
}
$cat b.cpp
#include
#include "sjsws.h"
void myfunc(void)
{
DTRACE_REACHED2("two");
}
$cat sjsws.h
#ifndef SJSWS
#define SJSWS 1
#include "sys/sdt.h"
#ifdef __cplusplus
extern "C" {
#endif
#define DTRACE_REACHED1(arg0) \
__dtrace_sjsws1___reached1(arg0)
#define DTRACE_REACHED2(arg0) \
__dtrace_sjsws2___reached2(arg0)
extern void __dtrace_sjsws1___reached1(const char *);
extern void __dtrace_sjsws2___reached2(const char *);
#ifdef __cplusplus
}
#endif
#endif
___
dtrace-discuss mailing list
dtrace-discuss@opensolaris.org
Re: [dtrace-discuss] truss "-fall" equivalent in DTrace
I tried this script (attached below) but it printed only 1 thread (shown
below). When I tried :::entry and my system was almost hung. I think pstack is
good enough for my purpose. :-)
BTW any script to find out which two threads are causing lock contention
/deadlock?
#!/usr/sbin/dtrace -s
#pragma D option quiet
#pragma D option defaultargs
pid$1:::entry
/ this->thread_is_already_printed != 1 /
{
this->thread_is_already_printed = 1;
printf("thread %d: \n", tid);
ustack(50);
}
$./pstack.d 16028
thread 11:
libc.so.1`_lwp_mutex_lock
libc.so.1`_lwp_cond_reltimedwait+0x78
libc.so.1`_lwp_cond_timedwait+0x1c
libjvm.so`__1cHMonitorEwait6Mil_i_+0x328
libjvm.so`__1cIVMThreadDrun6M_v_+0x1b4
libjvm.so`__1cG_start6Fpv_0_+0x208
libc.so.1`_lwp_start
--- On Fri, 11/14/08, Adam Leventhal <[EMAIL PROTECTED]> wrote:
> From: Adam Leventhal <[EMAIL PROTECTED]>
> Subject: Re: [dtrace-discuss] truss "-fall" equivalent in DTrace
> To: "[EMAIL PROTECTED]" <[EMAIL PROTECTED]>
> Cc: "Mark Plotnick" <[EMAIL PROTECTED]>, dtrace-discuss@opensolaris.org
> Date: Friday, November 14, 2008, 7:32 PM
> On Fri, Nov 14, 2008 at 12:40:55AM -0800,
> [EMAIL PROTECTED] wrote:
> > Can I get pstack equivalent script using DTrace?
>
> You can use ustack() at any probe.
>
> Adam
>
> --
> Adam Leventhal, Fishworks
> http://blogs.sun.com/ahl
___
dtrace-discuss mailing list
dtrace-discuss@opensolaris.org
Re: [dtrace-discuss] truss "-fall" equivalent in DTrace
Thanx a lot. Can I get pstack equivalent script using DTrace? One problem is if we call java from C (using JNI) such stacks are not shown in pstack. One more problem is running instance is hang thread lock can we have a script which shows which all threads are causing deadlock? --- On Fri, 11/7/08, Mark Plotnick <[EMAIL PROTECTED]> wrote: > Check out Brendan Gregg's dtruss shell script (available > from his web site or as part of MacOSX Leopard). It emulates truss using > dtrace, including truss's -f option. ___ dtrace-discuss mailing list dtrace-discuss@opensolaris.org
Re: [dtrace-discuss] truss "-fall" equivalent in DTrace
What I mean to ask is will PID provider trace child processes forked by the
parent process?
--- On Wed, 11/5/08, Angelo Rajadurai <[EMAIL PROTECTED]> wrote:
> From: Angelo Rajadurai <[EMAIL PROTECTED]>
> Subject: Re: [dtrace-discuss] truss "-fall" equivalent in DTrace
> To: [EMAIL PROTECTED]
> Cc: dtrace-discuss@opensolaris.org
> Date: Wednesday, November 5, 2008, 2:52 PM
> Change $1 to $target!
>
> (ie)
>
> ./sample.d -c "a.out"
> pid$target::somefunctionname:entry
> {
>printf("%s is called by %d",probefunc, tid);
> }
>
> In case you want to find the funcstions of a running
> process (say pid 1234)
> you have two options.
>
> ./sample.d --p 1234
> pid$target::somefunctionname:entry
> {
>printf("%s is called by %d",probefunc, tid);
> }
>
> or
> ./sample.d 1234
> pid$1::somefunctionname:entry
> {
>printf("%s is called by %d",probefunc, tid);
> }
>
>
> HTHs
>
> Angelo
>
> On Nov 5, 2008, at 5:21 AM, [EMAIL PROTECTED] wrote:
>
> > I have an executable. When I attach a truss, we have
> to give $truss -o truss.out -fall ./a.out. It shows all
> system calls made by this program and all the child
> processes it forks.
> >
> > Where as if I am using a DTrace script I am not able
> to do it.
> >
> > ./sample.d -c "a.out"
> > pid$1::somefunctionname:entry
> > {
> >printf("%s is called by %d",probefunc,
> tid);
> > }
> >
> >
> >
> >
> > ___
> > dtrace-discuss mailing list
> > dtrace-discuss@opensolaris.org
___
dtrace-discuss mailing list
dtrace-discuss@opensolaris.org
Re: [dtrace-discuss] Passing arguments to DTrace script
Thanx, I will make it work with -x defaultargs that way both $./sample.d and $./sample.d verbose will work. ___ dtrace-discuss mailing list dtrace-discuss@opensolaris.org
[dtrace-discuss] truss "-fall" equivalent in DTrace
I have an executable. When I attach a truss, we have to give $truss -o
truss.out -fall ./a.out. It shows all system calls made by this program and all
the child processes it forks.
Where as if I am using a DTrace script I am not able to do it.
./sample.d -c "a.out"
pid$1::somefunctionname:entry
{
printf("%s is called by %d",probefunc, tid);
}
___
dtrace-discuss mailing list
dtrace-discuss@opensolaris.org
Re: [dtrace-discuss] Printing a const char *
> If the kernel is 64-bit and the traced program is 32-bit,
> you'll want to change your structure definition so that
> it reflects the bitness of the traced program. This means
> that for pointers you should use a uint32_t rather than a
> char * for example.
>
> Adam
>
Ok Thanx.
If my program is 32 bit I include this in DTrace script :
struct xxx
{
int yyy;
int zzz;
uint32_t name;
};
If my program is 64 bit I include this in DTrace script :
struct xxx
{
int yyy;
int zzz;
const char *name;
};
___
dtrace-discuss mailing list
dtrace-discuss@opensolaris.org
Re: [dtrace-discuss] Printing a const char *
Casting it explicitly as "uintptr_t" works for 64 bit program and not for 32
bit program.
$CC -xarch=v9 sample.cpp
$dtrace -s sample.d -c ./a.out
dtrace: script 'sample.d' matched 1 probe
CProgram: 20 30 ABCD
dtrace: pid 2974 has exited
CPU IDFUNCTION:NAME
0 47856 __1cEsub16FpnDxxx__v_:entry DTrace: 20 30
DTrace: name=ABCD
$CC sample.cpp
$dtrace -s sample.d -c ./a.out
dtrace: script 'sample.d' matched 1 probe
CProgram: 20 30 ABCD
dtrace: error on enabled probe ID 1 (ID 47856:
pid2979:a.out:__1cEsub16FpnDxxx__v_:entry): invalid address (0x28728)
in action #4 at DIF offset 28
dtrace: pid 2979 has exited
$cat sample.d
struct xxx
{
int yyy;
int zzz;
const char *name;
};
pid$target:a.out:*sub1*:entry
{
sp = (struct xxx *) copyin (arg0, sizeof (struct xxx));
printf ("DTrace: %d %d \n", sp->yyy, sp->zzz);
printf ("DTrace: name=%s\n", copyinstr((uintptr_t)sp->name));
exit (0);
}
___
dtrace-discuss mailing list
dtrace-discuss@opensolaris.org
Re: [dtrace-discuss] Printing a const char *
$dtrace -s sample.d -c ./a.out
dtrace: failed to compile script sample.d: line 12: copyinstr( ) argument #1 is
incompatible with prototype:
prototype: uintptr_t
argument: char *
$cat sample.d
struct xxx
{
int yyy;
int zzz;
const char *name;
};
pid$target:a.out:*sub1*:entry
{
sp = (struct xxx *) copyin (arg0, sizeof (struct xxx));
printf ("DTrace: %d %d \n", sp->yyy, sp->zzz);
printf ("DTrace: name=%s\n", copyinstr(sp->name));
exit (0);
}
>
> You've correctly copied the structure into DTrace's
> address space, but you
> didn't copy in the const char * (string). Rather than
> doing stringof() on
> sp->name, use the copyinstr() subroutine.
>
___
dtrace-discuss mailing list
dtrace-discuss@opensolaris.org
[dtrace-discuss] Passing arguments to DTrace script
I am very now to DTrace land.
1) I want to write a DTrace script which takes -verbose or --verbose as an
option.
#!/usr/sbin/dtrace -s
#pragma D option quiet
pid$1::func1:entry
{
/* should be called only when verbose option is provided in command line */
printf("Entered %s ...\n", probefunc);
}
pid$1::func2:entry,
{
/* should be called always */
}
END
{
/* should be called always */
/* print out summary */
}
It should be normally run as
$./myscript.d 9149
(where 9149 is the pid of the process I want to trace)
when I want more detailed information, I should be able to run it as
$./myscript.d 9149 --verbose
Is something of this sort possible ?
2) Also I saw that END gets executed when I press Control C. But when I want to
send output to another file via tee as shown below :
$./myscript.d |tee dtrace.log
and when I press control C, it doesn't get called. Any ideas?
___
dtrace-discuss mailing list
dtrace-discuss@opensolaris.org
[dtrace-discuss] Printing a const char *
I have a C program,
$cat sample.cpp
#include
#include
#include
struct xxx
{
int yyy;
int zzz;
const char *name;
};
void sub1 (struct xxx *p)
{
printf ("CProgram: %d %d %s\n", p->yyy, p->zzz, p->name);
}
main()
{
char * key = (char *)malloc(5);
key[0] = 'A';
key[1] = 'B';
key[2] = 'C';
key[3] = 'D';
key[4] = '\0';
struct xxx *t1 = new struct xxx;
t1->yyy = 20;
t1->zzz = 30;
t1->name = key;
sub1 (t1);
}
and a DTrace script :
$cat sample.d
struct xxx
{
int yyy;
int zzz;
const char *name;
};
pid$target:a.out:*sub1*:entry
{
sp = (struct xxx *) copyin (arg0, sizeof (struct xxx));
printf ("DTrace: %d %d \n", sp->yyy, sp->zzz);
printf ("DTrace: name=%s\n", stringof(sp->name));
exit (0);
}
$CC sample.cpp
$dtrace: script 'sample.d' matched 1 probe
CProgram: 20 30 ABCD
dtrace: pid 2624 has exited
dtrace: error on enabled probe ID 1 (ID 47665:
pid2624:a.out:__1cEsub16FpnDxxx__v_:entry): invalid address (0x28728)
in action #4
What is this error ? I get this error when I compile the program in 64 bit as
well.
$CC -xarch=v9 sample.cpp
$dtrace -s sample.d -c ./a.out
dtrace: script 'sample.d' matched 1 probe
CProgram: 20 30 ABCD
dtrace: pid 2629 has exited
dtrace: error on enabled probe ID 1 (ID 47665:
pid2629:a.out:__1cEsub16FpnDxxx__v_:entry): invalid address (0x10010a000) in
action #4
___
dtrace-discuss mailing list
dtrace-discuss@opensolaris.org
Re: [dtrace-discuss] Why could I get function names from a stripped exec?
Hi, Yasufumi CHIDA wrote: > Thanks. > > To tell the truth, I'm an instructor, so I have to reply to my trainees. > Of course I'm curious about the internal working of DTrace. > > > > BTW, you said "dynamic info", what is this on earth? > The names are needed for dynamic linking. They are in the .dynsym section of the file. You might want to look at elfdump output, then http://blogs.sun.com/ali/entry/inside_elf_symbol_tables, where Ali Bahrami has a decent description. (Or google for .dynsym, which is how I found Ali's blog). max > Yasufumi. > -- > This message posted from opensolaris.org > ___ > dtrace-discuss mailing list > dtrace-discuss@opensolaris.org > > ___ dtrace-discuss mailing list dtrace-discuss@opensolaris.org
Re: [dtrace-discuss] Solaris Dynamic Tracing Guide gets Wikified!
Hi All, Now that the dtrace guide has been wikified, what happens with the guide on docs.sun.com? Should I look at the wikified guide, or the (dated) guide on docs.sun.com? I have the same question about the Modular Debugger guide. Also, any chance of changing the dashboard page to include MDB with the "Solaris Modular Debugger" line? I went to wikis.sun.com and searched the page for mdb and didn't find it. Then I searched for debug and there it was. thanks, max Adam Leventhal wrote: > Agreed. We were thinking of doing exactly this. The provider chapters would > be a great place to start. There should be a table in each describing the > support on various platforms as well as any usage notes which might apply. > > Adam > > On Thu, Dec 13, 2007 at 10:37:01AM -0500, Jim Mauro wrote: > >> Great question. As a dtrace user and documentation reader, I would not >> want to >> need to flip to another chapter, or another section, to read about >> platform differences >> for a particular provider, function, etc. I'm not saying you suggested >> that, I'm just >> thinking out loud... >> >> I think a reasonable way to do this, and maintain consistency, is to add >> a "Platform Notes" >> heading to each section - or perhaps at a Chapter granularity. >> >> Within Platform Notes, we have subsections for "Mac OS X", "FreeBSD", etc, >> and add the platform specific notes there. >> >> It may also be useful to add a chapter that discusses platform >> differences at a >> high level. >> >> Just some thoughts...if the platform differences are not vast enough to >> warrant >> adding format changes to the document, perhaps they can be handled in >> footnotes. >> (keeping in mind footnotes are considered poor writing style). >> >> Thanks, >> /jim >> >> >> Quinn wrote: >> >>> At 10:23 -0800 28/11/07, Adam Leventhal wrote: >>> >>> >>>> On Tue, Nov 27, 2007 at 07:46:37PM +, Jon Haslam wrote: >>>> >>>> >>>>> To gain access to the DTrace wiki space contact Adam >>>>> ([EMAIL PROTECTED]). If you are working on a bug >>>>> from the bug list make sure you update the owner column of >>>>> the bug to reflect the fact. When you've finished please >>>>> update the column in the bug list to reflect that fact. >>>>> >>>>> >>>> Actually, there's no need to contact me or anyone: the wiki is open for all >>>> registered users to modify. We'll tighted up security if we have any >>>> problems. >>>> >>>> >>> How do you want to handle platform-specific changes? As I stumble >>> across Mac OS X specific stuff, I'd like to add it to the book. To >>> me that makes a lot more sense that maintaining a separate "DTrace on >>> Mac OS X" document. >>> >>> For example, the discussion of the "stop" action says that you can >>> start a process using "prun", but that's not available on Mac OS X. >>> On Mac OS X you can use "kill -CONT ". I'd love to just go in >>> there and add a note to that effect, but I figured I'd ask about >>> formatting conventions first. >>> >>> S+E >>> >>> >> ___ >> dtrace-discuss mailing list >> dtrace-discuss@opensolaris.org >> > > ___ dtrace-discuss mailing list dtrace-discuss@opensolaris.org
Re: [dtrace-discuss] dtrace wired issue
Aubrey Li wrote: > On Jan 15, 2008 2:45 PM, [EMAIL PROTECTED] <[EMAIL PROTECTED]> wrote: > >> Hi Aubrey, >> >>> ---snip-- >>> setrlimit(RLIMIT_NOFILE, 0xFD7FFFDFEE80)= 0 >>> openat(AT_FDCWD, "/dev/dtrace/provider", O_RDONLY|O_NDELAY|O_LARGEFILE) = 3 >>> fcntl(3, F_SETFD, 0x0001) = 0 >>> fstat(3, 0xFD7FFFDFE8E0)= 0 >>> getdents(3, 0xFD7FFF174000, 8192) (sleeping...) >>> ---here, dtrace sleep almost 4 minutes >>> ...and continue... >>> >>> >>> >> How long does an ls -l /dev/dtrace/provider take? >> > > Yes, it costs almost 4 minutes. > > So, what does ls -l /dev/dtrace/provider show? max >> I doubt you are swapping with 2G of memory. >> You can run vmstat -S 2 to see if there is any swapping. I'll look a >> bit more... >> > > # vmstat -S 2 > kthr memorypagedisk faults cpu > r b w swap free si so pi po fr de sr cd -- -- -- in sy cs us sy id > 0 0 0 1601116 1352560 0 0 717 2 53 0 2502 71 0 0 0 718 7222 2813 4 3 93 > 0 0 0 1516012 1236776 0 0 0 0 0 0 0 0 0 0 0 388 269 347 0 0 99 > 0 0 0 1516012 1236808 0 0 0 0 0 0 0 0 0 0 0 394 262 354 0 0 > 100 > 0 0 0 1516012 1236808 0 0 0 0 0 0 0 0 0 0 0 392 255 356 0 0 > 100 > 0 0 0 1516016 1236812 0 0 0 0 0 0 0 23 0 0 0 425 380 396 1 0 99 > 0 0 0 1516016 1236812 0 0 0 0 0 0 0 0 0 0 0 389 294 332 0 0 > 100 > 0 0 0 1516016 1236812 0 0 0 0 0 0 0 0 0 0 0 382 466 444 0 0 99 > 0 0 0 1516016 1236812 0 0 0 0 0 0 0 0 0 0 0 394 273 346 0 0 > 100 > 0 0 0 1516016 1236812 0 0 0 0 0 0 0 0 0 0 0 387 250 328 0 0 99 > 0 0 0 1516016 1236812 0 0 0 0 0 0 0 0 0 0 0 384 264 339 0 0 > 100 > > >> By the way, are you running on Indiana (just curious...). >> >> > > No, I'm using nevada. > > [EMAIL PROTECTED] /etc/release >Solaris Express Community Edition snv_77 X86 >Copyright 2007 Sun Microsystems, Inc. All Rights Reserved. > Use is subject to license terms. >Assembled 06 November 2007 > > Thanks, > -Aubrey > > ___ dtrace-discuss mailing list dtrace-discuss@opensolaris.org
Re: [dtrace-discuss] dtrace wired issue
Hi Aubrey, Aubrey Li wrote: > Sorry for the delay(time difference). Now I got more details. > # truss dtrace -l > snip > mmap(0x, 4096, PROT_READ|PROT_WRITE|PROT_EXEC, > MAP_PRIVATE|MAP_ANON, 4294967295, 0) = 0xFD7FFF18 > mmap(0x0001, 24576, PROT_READ|PROT_WRITE|PROT_EXEC, > MAP_PRIVATE|MAP_ANON|MAP_ALIGN, -1, 0) = 0xFD7FFF17 > munmap(0xFD7FFF38, 32768) = 0 > getcontext(0xFD7FFFDFE9D0) > mmap(0x0001, 65536, PROT_READ|PROT_WRITE|PROT_EXEC, > MAP_PRIVATE|MAP_ANON|MAP_ALIGN, 4294967295, 0) = 0xFD7FFF15 > getrlimit(RLIMIT_STACK, 0xFD7FFFDFED30) = 0 > getpid()= 923 [922] > lwp_private(0, 0, 0xFD7FFF150200) = 0x > setustack(0xFD7FFF1502A8) > sysconfig(_CONFIG_PAGESIZE) = 4096 > sigfillset(0xFD7FFF006880) = 0 > brk(0x0041B580) = 0 > brk(0x0041F580) = 0 > getrlimit(RLIMIT_NOFILE, 0xFD7FFFDFEE80)= 0 > setrlimit(RLIMIT_NOFILE, 0xFD7FFFDFEE80)= 0 > openat(AT_FDCWD, "/dev/dtrace/provider", O_RDONLY|O_NDELAY|O_LARGEFILE) = 3 > fcntl(3, F_SETFD, 0x0001) = 0 > fstat(3, 0xFD7FFFDFE8E0)= 0 > getdents(3, 0xFD7FFF174000, 8192) (sleeping...) > ---here, dtrace sleep almost 4 minutes > ...and continue... > > > So, it's probably something obvious that I'm not seeing... What happens if you try tracing all calls made via getdents for dtrace itself, and do this instead of dtrace -l for the first time you run dtrace? max > Any thoughts? > > Thanks, > -Aubrey > > ___ dtrace-discuss mailing list dtrace-discuss@opensolaris.org
Re: [dtrace-discuss] dtrace wired issue
Hi Aubrey, Aubrey Li wrote: > Sorry for the delay(time difference). Now I got more details. > # truss dtrace -l > snip > mmap(0x, 4096, PROT_READ|PROT_WRITE|PROT_EXEC, > MAP_PRIVATE|MAP_ANON, 4294967295, 0) = 0xFD7FFF18 > mmap(0x0001, 24576, PROT_READ|PROT_WRITE|PROT_EXEC, > MAP_PRIVATE|MAP_ANON|MAP_ALIGN, -1, 0) = 0xFD7FFF17 > munmap(0xFD7FFF38, 32768) = 0 > getcontext(0xFD7FFFDFE9D0) > mmap(0x0001, 65536, PROT_READ|PROT_WRITE|PROT_EXEC, > MAP_PRIVATE|MAP_ANON|MAP_ALIGN, 4294967295, 0) = 0xFD7FFF15 > getrlimit(RLIMIT_STACK, 0xFD7FFFDFED30) = 0 > getpid()= 923 [922] > lwp_private(0, 0, 0xFD7FFF150200) = 0x > setustack(0xFD7FFF1502A8) > sysconfig(_CONFIG_PAGESIZE) = 4096 > sigfillset(0xFD7FFF006880) = 0 > brk(0x0041B580) = 0 > brk(0x0041F580) = 0 > getrlimit(RLIMIT_NOFILE, 0xFD7FFFDFEE80)= 0 > setrlimit(RLIMIT_NOFILE, 0xFD7FFFDFEE80)= 0 > openat(AT_FDCWD, "/dev/dtrace/provider", O_RDONLY|O_NDELAY|O_LARGEFILE) = 3 > fcntl(3, F_SETFD, 0x0001) = 0 > fstat(3, 0xFD7FFFDFE8E0)= 0 > getdents(3, 0xFD7FFF174000, 8192) (sleeping...) > ---here, dtrace sleep almost 4 minutes > ...and continue... > > How long does an ls -l /dev/dtrace/provider take? I doubt you are swapping with 2G of memory. You can run vmstat -S 2 to see if there is any swapping. I'll look a bit more... By the way, are you running on Indiana (just curious...). max > Any thoughts? > > Thanks, > -Aubrey > > ___ dtrace-discuss mailing list dtrace-discuss@opensolaris.org
Re: [dtrace-discuss] dtrace wired issue
Hi James, James C. McPherson wrote: > Aubrey Li wrote: > >> On Jan 14, 2008 8:52 PM, Sean McGrath - Sun Microsystems Ireland >> <[EMAIL PROTECTED]> wrote: >> >>> Aubrey Li stated: >>> < Every first time to run dtrace command after the system boot up, >>> < It takes a very long time to get response. >>> < But the second time is OK, as follows: >>> < >>> < # time dtrace -l > /dev/null >>> < >>> < real4m8.011s >>> < user0m0.116s >>> < sys 0m2.420s >>> >>> This first time is probably when the kernel is loading the dtrace modules. >>> Though still seems slow, 4 minutes. >>> What kind of system (cpu speed etc) is the machine ? >>> >> # psrinfo -vp >> The physical processor has 2 virtual processors (0 1) >> x86 (GenuineIntel 10674 family 6 model 23 step 4 clock 2400 MHz) >> Intel(r) CPU @ 2.40GHz >> >> So, I failed to understand the modules loading needs 4 minutes. >> > > > If you run "dtrace -l" with no args, *every* single loadable > module on the system will be loaded, interrogated by dtrace > and then unloaded if possible. > You mean every kernel module currently loaded will be interrogated. Not every kernel module on the system. So _init/_info/_fini/attach/detach should not enter into this. max > All those attach()es and detach()es need time, as does the > probe collation. > > > > James C. McPherson > -- > Senior Kernel Software Engineer, Solaris > Sun Microsystems > http://blogs.sun.com/jmcp http://www.jmcp.homeunix.com/blog > ___ > dtrace-discuss mailing list > dtrace-discuss@opensolaris.org > > ___ dtrace-discuss mailing list dtrace-discuss@opensolaris.org
Re: [dtrace-discuss] dtrace wired issue
Hi James, James C. McPherson wrote: > Aubrey Li wrote: > >> On Jan 14, 2008 8:52 PM, Sean McGrath - Sun Microsystems Ireland >> <[EMAIL PROTECTED]> wrote: >> >>> Aubrey Li stated: >>> < Every first time to run dtrace command after the system boot up, >>> < It takes a very long time to get response. >>> < But the second time is OK, as follows: >>> < >>> < # time dtrace -l > /dev/null >>> < >>> < real4m8.011s >>> < user0m0.116s >>> < sys 0m2.420s >>> >>> This first time is probably when the kernel is loading the dtrace modules. >>> Though still seems slow, 4 minutes. >>> What kind of system (cpu speed etc) is the machine ? >>> >> # psrinfo -vp >> The physical processor has 2 virtual processors (0 1) >> x86 (GenuineIntel 10674 family 6 model 23 step 4 clock 2400 MHz) >> Intel(r) CPU @ 2.40GHz >> >> So, I failed to understand the modules loading needs 4 minutes. >> > > > If you run "dtrace -l" with no args, *every* single loadable > module on the system will be loaded, interrogated by dtrace > and then unloaded if possible. > Are you sure of this? Why should it load everything? (And then unload them if they're not being used). # dtrace -l | wc -l 88939 # modload dedump # dtrace -l | wc -l 88963 I get different counts. I think dtrace only looks at what is currently on the system. But 4 minutes??? max > All those attach()es and detach()es need time, as does the > probe collation. > > > > James C. McPherson > -- > Senior Kernel Software Engineer, Solaris > Sun Microsystems > http://blogs.sun.com/jmcp http://www.jmcp.homeunix.com/blog > ___ > dtrace-discuss mailing list > dtrace-discuss@opensolaris.org > > ___ dtrace-discuss mailing list dtrace-discuss@opensolaris.org
Re: [dtrace-discuss] Dtrace probes for voluntary and involuntary context switches
Hi Neelam, Neelam wrote: > Hi, > > I am profiling some workloads for the voluntary and involuntary context > switches. I am interested in finding out the reasons causing these two types > of context switches. As far as I understand, involuntary context switch > happens on expiration of time slice or when a higher priority process comes > in. While the voluntary switch generally happens when a process is waiting > for I/O etc. > > So to find out what system calls are causing voluntary context switches in my > workloads, I printed whenever a system calls is invoked and whenever a > context switch happens. I am profiling the system calls and context switched > inside critical sections (while some lock is being held). > > But I see something unexpected. I see > > * Voluntary context switches occur almost every time due to doorfs() > system call. They do occur for a few times due to lwp_park() and very > few times due to yield(). > > * Involuntary happens anytime. (lwp_park(), read(), fstat(), putmsg(), > gtime() and sometime without any system call!!) > > Does anyone have any idea, what could be the reason for this unexpected > behavior? > This behavior is not unexpected. In general, threads should not be sleeping while holding locks. What do you think is unexpected? max > Thanks, > Neelam > > > -- > This message posted from opensolaris.org > ___ > dtrace-discuss mailing list > dtrace-discuss@opensolaris.org > > ___ dtrace-discuss mailing list dtrace-discuss@opensolaris.org
