Just returned from vacation ...
Tom Lane wrote:
I notice that we have in md.c
TRACE_POSTGRESQL_SMGR_MD_READ_DONE(forknum, blocknum, reln-smgr_rnode.spcNode,
reln-smgr_rnode.dbNode, reln-smgr_rnode.relNode, relpath(reln-smgr_rnode,
forknum), nbytes, BLCKSZ);
Tom Lane wrote:
Zdenek Kotala zdenek.kot...@sun.com writes:
Answer why it happens when probes are disabled is, that for user
application there are piece of code which prepare DTrace probes
arguments which will be passed into kernel DTrace modul. This code has
performance penalty which
Tom Lane wrote:
Greg Stark st...@enterprisedb.com writes:
i like the idea of just have a separate pair of probes for table
extension. I bet there are people who would actually like to see that
alone sometimes too.
After further thought I concluded that the best solution for this is to
Tom Lane wrote:
Whilst cleaning up the dtrace probe documentation, I realized that
there is nothing the aforementioned probes tell you that you can't find
out (with a lot more data besides) from the final argument of the
BUFFER_READ_DONE probe. Furthermore, tallying them as-is would be
Zdenek Kotala wrote:
I tested your patch and it looks good. however I have several
comments/questions:
1) probes contains pointer but in probe.h it is declared as int. Is it
correct?
The probes cast the pointer to uintptr_t, so the correct type that will
work for both ILP32 and LP64
ITAGAKI Takahiro wrote:
This will introduce SLRU and Executor probes.
We already have Lock, LWLock, Buffer, I/O and XLogs probes.
I'd like to have the following probes, too. In my experience,
they could be bottlenecks or consume much time in some situations.
- idle in transaction
-
The attached patch contains the probes (originally came from Theo
Schlossnagle) that were removed from the previous submitted patch.
Zdenek had some concerns about the way the probes were implemented
http://archives.postgresql.org/pgsql-hackers/2008-07/msg01168.php. If
there are specific
Attached is the doc patch for the recently added probes.
-Robert
Index: doc/src/sgml/monitoring.sgml
===
RCS file: /projects/cvsroot/pgsql/doc/src/sgml/monitoring.sgml,v
retrieving revision 1.63
diff -u -3 -p -r1.63 monitoring.sgml
Tom Lane wrote:
Robert Lor robert@sun.com writes:
Tom Lane wrote:
I agree. If the probe is meant to track only *some* WAL writes
then it needs to be named something less generic than
TRACE_POSTGRESQL_WAL_BUFFER_WRITE.
How about change
Tom Lane wrote:
Fujii Masao masao.fu...@gmail.com writes:
I understood your intention. But, I think that its function name is somewhat
confusing.
I agree. If the probe is meant to track only *some* WAL writes
then it needs to be named something less generic than
Alvaro Herrera wrote:
But there are 5 callers of XLogWrite ... why aren't the other ones being
tracked too?
This probe originally came from Simon, so it can't possibly be wrong :-)
My understanding is that we only want to track the XLogWrite when
advancing to the next buffer page, and if
Fujii Masao wrote:
Hi,
On Wed, Dec 17, 2008 at 4:53 AM, Robert Lor robert@sun.com wrote:
@@ -1313,12 +1318,14 @@ AdvanceXLInsertBuffer(bool new_segment)
* Have to write buffers while holding insert
lock. This is
* not good
Peter Eisentraut wrote:
Robert Lor wrote:
The attached patch contains a couple of fixes in the existing probes
and includes a few new ones.
- Fixed compilation errors on OS X for probes that use typedefs
Could you explain what these errors are about? I don't see any errors
on my machine
Bruce Momjian wrote:
Should I apply this or hold it for 8.5?
I think it should go into 8.4 as it also fixes existing problems.
-Robert
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
Bruce Momjian wrote:
I am seeing the following error when linking the backend on a BSD machine:
The updated patch attached should fix this problem. My bad for
overlooking this.
-Robert
Index: src/backend/access/transam/xlog.c
The attached patch contains a couple of fixes in the existing probes and
includes a few new ones.
- Fixed compilation errors on OS X for probes that use typedefs
- Fixed a number of probes to pass ForkNumber per the relation forks patch
- The new probes are those that were taken out from the
pulled in only for Solaris?
It does work on OS X. I dare everyone to try it :-)
The implementation of DTrace on OS X is a bit different than on Solaris,
so the rule your referring to is only needed for Solaris.
--
Robert Lor Sun Microsystems
Austin, USA http://sun.com
that much.
It was overlooked :-( . Good catch!
Also, it seems I cannot sort pg_trace and pgstat consistently :-(
Not sure what you're trying to say here!
--
Robert Lor Sun Microsystems
Austin, USA http://sun.com/postgresql
--
Sent via pgsql-hackers mailing list (pgsql
.
--
Robert Lor Sun Microsystems
Austin, USA http://sun.com/postgresql
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
be committed.
I'd appreciate if it can be committed today.
Alvaro, thanks a bunch for the feedback!
--
Robert Lor Sun Microsystems
Austin, USA http://sun.com/postgresql
Index: src/backend/access/transam/clog.c
with
GNU sed. I hope this is the final change for this patch. Thanks for
catching all the issues, and my bad for not testing with DTrace disabled.
--
Robert Lor Sun Microsystems
Austin, USA http
manner.
--
Robert Lor Sun Microsystems
Austin, USA http://sun.com/postgresql
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
prototype in probes.h, but the
probes work okay. I think we can safely ignore the warnings.
--
Robert Lor Sun Microsystems
Austin, USA http://sun.com/postgresql
Index: src/backend/access/transam/clog.c
===
RCS file
Zdenek Kotala wrote:
However what I suggested is commit probes without issue now and the
rest will be processed on the next commit fest after rework/discussion.
Agreed. I'll fix up the remaining issues with the patch you sent.
--
Robert Lor Sun Microsystems
Austin, USA
tomorrow!
--
Robert Lor Sun Microsystems
Austin, USA http://sun.com/postgresql
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
;
provider postgresql {
probe transaction__start(LocalTransactionId);
I like this solution.
--
Robert Lor Sun Microsystems
Austin, USA http://sun.com/postgresql
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your
. (And I'm perfectly fine with telling people
that they can't compile Postgres dtrace support with less than the most
recent tool set, especially since it'll be fairly old by the time 8.4
ships.)
I tested on both Xcode 3.0 3.1 and both worked.
--
Robert Lor Sun Microsystems
Austin, USA
cases
that the new data will enable.
--
Robert Lor Sun Microsystems
Austin, USA http://sun.com/postgresql
Index: src/backend/access/transam/clog.c
===
RCS file: /projects/cvsroot/pgsql/src/backend/access/transam
Robert Treat wrote:
certainly by the time 8.4 ships, these should work with freebsd I'd think.
ideally we would need to confirm this by release time, certainly getting a
bsd buildfarm member to compile with them would be a start (and very unlikely
to cause issues)
As soon as the DTrace
Robert Treat wrote:
On Wednesday 04 June 2008 22:04:54 Greg Smith wrote:
I was just talking to someone today about building a monitoring tool for
this. Not having a clear way to recommend people monitor use of work_mem
and its brother spilled to disk sorts is an issue right now, I'll whack
Robert Treat wrote:
While it would be nice to have a clean merge of the two, it's probably simple enough to just
re-implement the differences into your patch (since yours already compiles on 8.4).
Should be straightforward ... I can do the merge.
As far as naming scheme, I'm not
Steve Atkins wrote:
I'd be interested in putting together a framework+GUI client to do this
cleanly in a cross-platform (Windows, Linux, Solaris, OS X as a bare
minimum) sort of way, if no-one else already has such a thing.
This is a great idea, and I was thinking along the same line. The
Tom Lane wrote:
This is even assuming that the tool needs to edit the file itself,
rather than just give advice. The advice is the hard part, folks;
could we stop obsessing about trivia?
+1. IMHO, the tool doesn't need to worry about generating a prettied
version of postgresql.conf. It should
Tom Lane wrote:
If those aren't enough questions, what else must we ask? Or maybe they
aren't the right questions at all --- maybe we should ask is this a
dedicated machine or not and try to extrapolate everything else from
what we (hopefully) can find out about the hardware.
I think
Greg Smith wrote:
There's also a big DTrace probe set patch available from OmniTI:
https://labs.omniti.com/project-dtrace/trunk/postgresql/
http://labs.omniti.com/trac/project-dtrace/wiki/Applications#PostgreSQL
I don't know if you've looked at that before. There's some overlap
but many
(Resending since it didn't work the first time. Not sure if attaching a
tar file was the culprit.)
I'd like to propose adding the following probes (some of which came from
Simon) to 8.4.
I think these probe provide very useful data. Although some of the data
can be collected now, the main
Peter Eisentraut wrote:
Well, yes. I meant to say, a build system that can supply the functionality
of Gen_fmgrtab can surely implement this new thing. I see there is Perl
being used, so it should be simple.
I was thinking of using a Perl script to generate the dummy header file
but
Peter Eisentraut wrote:
Am Dienstag, 18. März 2008 schrieb Tom Lane:
Well, I've got Leopard here, I'd be happy to test it ... but the patch
has rendered
http://developer.postgresql.org/pgdocs/postgres/dynamic-trace.html
into a pack of lies quite independently of which OSes are supported,
so
Heikki Linnakangas wrote:
The main point of my proposal is: let's make the *authors* who want
their stuff to be reviewed as part of a commitfest do the extra work.
There would be no extra work required for patch reviewers.
I agree with Heikki that for the process to be successful, it should
Tom Lane wrote:
This is reasonable for the sort of medium-to-large patch that the author
has put a lot of time into. But we also get a lot of small one-off
patches where it's not so reasonable. Now of course many of those get
applied right away, but not all. One of the services that Bruce's
Robert Lor wrote:
Proposed changes:
* Switch from using DTRACE_PROBEn macros to the dynamically generated
macros (remove pg_trace.h)
* Use dtrace -h to create a header file that contains the
dynamically generated macros to be used in the source code instead of
the DTRACE_PROBEn macros
Tom Lane wrote:
I'm unconvinced that there will be any probes that are common to all
databases. I'd skip this part...
Any reason why we can't consider probes like transaction-start,
transaction-commit, or transaction-abort as common probes that can also
be used in other (maybe no all)
Tom Lane wrote:
I'm unimpressed; it's not at all clear that you'd be measuring quite the
same thing in, say, mysql as in postgres.
I think it depends on the probe, but for transaction rates like start or
commit, don't you think it's pretty black white as long as the probes
are placed in
Gregory Stark wrote:
I think both types of probes are useful to different people.
I think certain higher level probes can be really useful to DBAs.
Perhaps looking at the standard database SNMP MIB counters would give us a
place to start for upward facing events people want to trace for
Tom Lane wrote:
The concern I've got about this is basically that it would encourage
plastering the same label on subtly different counts, leading to
confusion and perhaps mistaken conclusions. I would prefer to see any
common probes be reverse-engineered *after the fact*, ie, after you've
Motivation:
To enable probes to work with Mac OS X Leopard and other OSes that will
support DTrace in the future. Bug filed on this issue
http://archives.postgresql.org/pgsql-bugs/2007-10/msg00201.php
The problem:
There are two different mechanisms to implementing application level
probes in
Peter,
Peter Eisentraut [EMAIL PROTECTED] wrote:
Could you please use diff -c or -u for the patch? It's quite hard to
read this way.
Attached is the patch with diff -c.
Thanks,
-Robert
---(end of broadcast)---
TIP 3: Have you checked our
Robert Lor [EMAIL PROTECTED] wrote:
Peter,
Peter Eisentraut [EMAIL PROTECTED] wrote:
Could you please use diff -c or -u for the patch? It's quite hard
to
read this way.
Attached is the patch with diff -c.
Oops, here's the real attachment!
Regards,
-Robert
dtrace.patch
Greg,
Gregory Stark wrote:
I don't think DTrace is overkill either. The programmatic interface is
undocumented (but I've gotten Sun people to admit it exists -- I just have to
reverse engineer it from the existing code samples) but should be more or less
exactly what we need.
You probably
Sorry for the delayed response.
Robert Treat wrote:
Looking through -patches I don't see the doc patch, and outside of
installation.sgml there doesn't seem to be anything either. Robert, are you
still on the hook for these?
Josh will help submit the doc patch. I have documented the usage
Peter Eisentraut wrote:
Robert Treat wrote:
Also should installation.sgml
mention the issueswith building 32 vs 64 bit binaries
I'm not convinced there is an issue. dtrace will build the right
binaries by default. If you're messing with mixed environments *and*
delve into dtrace,
Tom Lane wrote:
I think the actual wave of the future for analyzing behavior at the
LWLock level is going to be DTrace. It seems way more flexible than
an aggregate-statistics view can be.
CVS head now has the following LWLock probes, and more can easily be
added. These probes can be
Bruce Momjian wrote:
Do we need to add detection logic to catch buggy versions?
Instead of adding extra logic, I think it's sufficient with
documentation since the issue will soon be fixed in the next Solaris update.
Regards,
-Robert
---(end of
Peter Eisentraut wrote:
That rings a bell. Can we get a more precise designation on what
version of DTrace we support? And where can one get that required
update?
Peter,
The problem with static function was fixed recently and is now available
in Solaris Express (the development
Excellent! I'll submit a doc patch shortly.
Regards,
-Robert
Peter Eisentraut wrote:
I've committed the dtrace patch. Some documentation would be nice now ...
---(end of broadcast)---
TIP 1: if posting/reading through Usenet, please send
Peter Eisentraut wrote:
As I understand this, the probe file is compiled into some sort of
object file which is linked into the binary.
Correct.
So if we ever have probes
in other components, we'd probably want to have separate probe source
and object files for them. That would seem
Tom Lane wrote:
Yeah, those seem plausible, although the hold time for
CheckpointStartLock seems awfully high --- about 20 msec
per transaction. Are you using a nonzero commit_delay?
I didn't change commit_delay which defaults to zero.
Regards,
-Robert
---(end
Tom Lane wrote:
Hmmm ... AFAICS this must mean that flushing the WAL data to disk
at transaction commit time takes (most of) 20 msec on your hardware.
Which still seems high --- on most modern disks that'd be at least two
disk revolutions, maybe more. What's the disk hardware you're testing
Peter Eisentraut wrote:
Here is a consolidated patch that contains all the files. I made some
configure and makefile adjustments and put standard comment headers in
all the files. You can use DTRACEFLAGS to pass options to configure,
which should help sorting out the 32/64-bit issue. The
Peter Eisentraut wrote:
Perhaps the attached patch in -c format will work better.
Now I'm getting a different type of error. After running the patch
command, the configure file is partially patched but not the others.
Attached is configure.rej. I just checked out the source from CVS.
Peter Eisentraut wrote:
Robert Lor wrote:
Now I'm getting a different type of error. After running the patch
command, the configure file is partially patched but not the others.
Attached is configure.rej. I just checked out the source from CVS.
Sorry, there must be something wrong
Tom Lane wrote:
Tatsuo Ishii [EMAIL PROTECTED] writes:
18% in s_lock is definitely bad :-(. Were you able to determine which
LWLock(s) are accounting for the contention?
Sorry for the delay. Finally I got the oprofile data. It's
huge(34MB). If you are interested, I can put
Peter Eisentraut wrote:
I would prefer to drop the PG_ prefixes on PG_TRACE and pg_trace.h. We
know which software we're dealing with.
I also agree with Martin Tom to keep the PG_ prefixes.
We should probably move the probes file to a subdirectory. Anyone know
a good place?
Also,
korry wrote:
How about the obvious DTRACE( ) or some similar variant?
The idea is to keep the macro name generic since it can be mapped to
other tracing facility on other platforms.
Regards,
-Robert
---(end of broadcast)---
TIP 9: In
Tom Lane wrote:
Also, it'd be interesting to count time spent holding shared lock
separately from time spent holding exclusive.
Tom,
Here is the break down between exclusive shared LWLocks. Do the
numbers look reasonable to you?
Regards,
-Robert
bash-3.00# time ./Tom_lwlock_acquire.d
Peter Eisentraut wrote:
Robert Lor wrote:
The user needs to have the flexibility to build a 32 bit PG binary
even when he run the 64 bit kernel. If I understand you correctly,
your suggestion will not allow a 32 bit binary to be built on a 64
bit OS.
I'm not sure about the context
Peter, I'll test the patch on Solaris. Thanks!
Regards,
-Robert
Peter Eisentraut wrote:
Here is a consolidated patch that contains all the files. I made some
configure and makefile adjustments and put standard comment headers in
all the files. You can use DTRACEFLAGS to pass options to
Simon Riggs wrote:
This needs to work on Linux and Windows, minimum, also.
The proposed solution will work on Linux Windows if they similar
facility that the macros can map to. Otherwise, the macros stay as
no-ops and will not affect those platforms at all.
It's obviously impossible to
Greg Stark wrote:
It seems pointless to me to expose things like lwlock_acuire that map 1-1 to C
function calls like LWLockAcquire. They're useless except to people who
understand what's going on and if people know the low level implementation
details of Postgres they can already trace those
Motivation:
--
The main goal for this Generic Monitoring Framework is to provide a common
interface for adding instrumentation points or probes to
Postgres so its behavior can be easily observed by developers and
administrators even in production systems. This framework will allow
Tom Lane wrote:
Robert Lor [EMAIL PROTECTED] writes:
The main goal for this Generic Monitoring Framework is to provide a
common interface for adding instrumentation points or probes to
Postgres so its behavior can be easily observed by developers and
administrators even in production
Theo Schlossnagle wrote:
Heh. Syscall probes and FBT probes in Dtrace have zero overhead.
User-space probes do have overhead, but it is only a few instructions
(two I think). Besically, the probe points are replaced by illegal
instructions and the kernel infrastructure for Dtrace will
contact Josh Berkus.
Regards,
Robert Lor
Sun Microsystems, Inc.
01-510-574-7189
---(end of broadcast)---
TIP 4: Have you searched our list archives?
http://archives.postgresql.org
Arjen van der Meijden wrote:
I can already confirm very good scalability (with our workload) on
postgresql on that machine. We've been testing a 32thread/16G-version
and it shows near-linear scaling when enabling 1, 2, 4, 6 and 8 cores
(with all four threads enabled).
The threads are a
capabilities (e.g. DTrace) specifically for PostgreSQL. I'll be posting
a Solaris performance tuning guide in a few weeks.
Regards,
Robert Lor
---(end of broadcast)---
TIP 2: Don't 'kill -9' the postmaster
Bruce,
The binary was compiled in 32bit mode using Sun Studio compiler, and we
plan to do 64bit soon. Can you point me to the patch? We can certainly
test it!
Regards,
Robert
Bruce Momjian wrote:
Do they work on x86-64 processors using the Solaris compiler? We have a
patch for it in CVS
76 matches
Mail list logo
