ould allow you to pass two (or more) script files to pgbench and adjust
the transaction mix. What happens when you do that right now is that
inevitably all the clients get blocked at once on whatever the hardest to
execute transaction is, and the results are kind of deceptive as a result
d of the "just enough" computation based on a
weighted moving average I wrote before, but there's certainly room for
multiple implementations of that part of the code to evolve.
--
* Greg Smith [EMAIL PROTECTED] http://www.gregsmith.com Baltimore, MD
---(en
Here are some more recent papers that also give good insight into research
in this area:
http://www.cs.usask.ca/~wew036/comprehensive.pdf
http://www.cse.ohio-state.edu/hpcs/WWW/HTML/publications/papers/TR-05-3.pdf
--
* Greg Smith [EMAIL PROTECTED] http://www.gregsmith.com Baltimore, MD
o 30 will creep that up to
closer to a 1.2GB footprint.
--
* Greg Smith [EMAIL PROTECTED] http://www.gregsmith.com Baltimore, MD
---(end of broadcast)---
TIP 1: if posting/reading through Usenet, please send an appropriate
subscribe-nomail co
On Fri, 29 Jun 2007, Jim Nasby wrote:
On Jun 26, 2007, at 11:57 PM, Greg Smith wrote:
I have a complete set of working code that tracks buffer usage
statistics...
Even if it's not used by bgwriter for self-tuning, having that information
available would be very useful for anyone tryi
problem worse--now you have an
ever bigger pile of dirty mess to shovel at checkpoint time. The existing
background writers are particularly unsuited to helping out in this
situation, I think the new planned implementation will be much better.
--
* Greg Smith [EMAIL PROTECTED] http
mostly filled with dirty buffers with high
usage counts, you are in for a long wait when you need new buffers
allocated and your next checkpoint is going to be traumatic. That's all
I'm suggesting is a problem with that situation.
--
* Greg Smith [EMAIL PROTECTED] http://www.gregsmi
obust enough disk-wise to run
things like DBT2 on the scale I know you normally work on. I gots a disk
for the database, one for the WAL, 256MB of cache on the controller, and a
single dual-core procesor; can't fit too many warehouses here.
--
* Greg Smith [EMAIL PROTECTED] http://www.gregs
help here.
--
* Greg Smith [EMAIL PROTECTED] http://www.gregsmith.com Baltimore, MD
---(end of broadcast)---
TIP 3: Have you checked our extensive FAQ?
http://www.postgresql.org/docs/faq
d the clock
sweep point may be overkill, but I think it will help enormously on
smaller caches that are often very dirty.
--
* Greg Smith [EMAIL PROTECTED] http://www.gregsmith.com Baltimore, MD
---(end of broadcast)---
TIP 3: Have you checked our ext
entation myself in the next
couple of days. I still have the test environment leftover from the last
time I worked on this code, and I think everybody else who could handle
this job has more important higher-level things they could be working on
instead.
--
* Greg Smith [EMAIL PROTECTED] htt
--and therefore there's a possibility it
could be lost.
--
* Greg Smith [EMAIL PROTECTED] http://www.gregsmith.com Baltimore, MD
---(end of broadcast)---
TIP 5: don't forget to increase your free space map settings
thing that people
might want look out for during beta rather than a task Heikki should worry
about before applying the patch.
--
* Greg Smith [EMAIL PROTECTED] http://www.gregsmith.com Baltimore, MD
---(end of broadcast)---
TIP 6: explain analyze is
f useful settings in the documentation.
The only other configuration I'd be curious to see is pushing the number
of warehouses even more to see if the 90% numbers spread further from
current behavior.
--
* Greg Smith [EMAIL PROTECTED] http://www.gregsmith.com Baltimore
n the final form for the patch.
--
* Greg Smith [EMAIL PROTECTED] http://www.gregsmith.com Baltimore, MD
---(end of broadcast)---
TIP 2: Don't 'kill -9' the postmaster
to catch up.
I can't think of any good reason why the bgwriter_delay can't be reduced
to 1s if that simplifies things. You'd need a pretty old system for a
longer delay than that to be appropriate.
--
* Greg Smith [EMAIL PROTECTED] http://www.gregsmith.com Baltimore
possible (albeit
difficult) to get into a situation where if the checkpoint doesn't write
out the dirty buffers fast enough, the client backends will evacuate them
instead in a way that makes the whole process less efficient than the
current behavior.
--
* Greg Smith [EMAIL PROTE
omething useful out of that one. I found the
performance impact of the JDBC layer in the middle so lowered overall
throughput and distanced me from what was happening that it blurred what
was going on.
--
* Greg Smith [EMAIL PROTECTED] http://www.gregsmith.com Baltimore, MD
say whether your suggestions for adjusting warehouse size
would accomplish that (because the flow is so different I had to abandon
working with that a while ago as not being representative of what I was
doing), but I'm glad you're thinking about it.
--
* Greg Smith [EMAIL PROTECTED]
/sourceware.org/systemtap/wiki ) which is a bit immature but is
going the right direction.
I will now bow my head and wait for someone to suggest you move to an OS
that supports dtrace.
--
* Greg Smith [EMAIL PROTECTED] http://www.gregsmith.com Baltimore, MD
---(e
9/print and
http://toadstool.se/journal/2006/05/27/monitoring-filesystem-activity-under-linux-with-block_dump
)
--
* Greg Smith [EMAIL PROTECTED] http://www.gregsmith.com Baltimore, MD
---(end of broadcast)---
TIP 2: Don't 'kill -9' the postmaster
dissapear if their hosting
sponsor goes away.
--
* Greg Smith [EMAIL PROTECTED] http://www.gregsmith.com Baltimore, MD
---(end of broadcast)---
TIP 5: don't forget to increase your free space map settings
ovement on for others to try and replicate it.
--
* Greg Smith [EMAIL PROTECTED] http://www.gregsmith.com Baltimore, MD
---(end of broadcast)---
TIP 9: In versions below 8.0, the planner will ignore your desire to
choose an index scan if yo
kpoint
process as well.
--
* Greg Smith [EMAIL PROTECTED] http://www.gregsmith.com Baltimore, MD
---(end of broadcast)---
TIP 7: You can help support the PostgreSQL project by donating at
http://www.postgresql.org/about/donate
elp. They're in a better position than you to
straighten out the incompatiblity of their corporation with the
requirements of a public mailing list. You may very well be doing them a
favor to point out the concern rather than continuing a dialog with them.
--
* Greg Smith [EMAIL PROTE
to the relevant documentation, and instead are told the specifics of their
messages can't be addressed on the list while there's a disclaimer
attached.
Just ignoring them altogether will just give the community a bad
reputation for being unresponsive.
--
* Greg Smith [EMAIL PROTE
onable is to expect them to tune obscure parts of the kernel just
for your application.
--
* Greg Smith [EMAIL PROTECTED] http://www.gregsmith.com Baltimore, MD
---(end of broadcast)---
TIP 2: Don't 'kill -9' the postmaster
that merely changes instead of going away completely if you throw more
hardware at it. I'm perversely glad to hear this is torturing more people
than just me as it improves the odds the situation will improve.
--
* Greg Smith [EMAIL PROTECTED] http://www.gregsmit
w long writes are taking to gauge how fast you can dump data
onto the disks. When you're suffering from one of the congestion
mechanisms, the initial writes start blocking, even before the fsync.
That behavior is almost undocumented outside of the relevant kernel source
code.
--
* Greg Smi
the checkpoint write? That's your best bet for guessing how long the
fsync will take.
--
* Greg Smith [EMAIL PROTECTED] http://www.gregsmith.com Baltimore, MD
---(end of broadcast)---
TIP 3: Have you checked our extensive FAQ?
http://www.postgresql.org/docs/faq
release, along with fairly verbose logging of what is happening at
checkpoint time (that's why I've been nudging development in that area,
along with making logs easier to aggregate). Collect up enough of that
information, then you're in a position to talk about useful automatic
t
stream.
Some of that reading is both informative and periodically hilarious; my
favorite quote is from Linus Torvalds, who says "if you actually like
using CVS, you shouldn't be here. You should be in some mental
institution".
--
* Greg Smith [EMAIL PROT
work because either the network wire
speed or the speed of the underlying database I/O are the real
bottlenecks.
--
* Greg Smith [EMAIL PROTECTED] http://www.gregsmith.com Baltimore, MD
---(end of broadcast)---
TIP 2: Don't 'kill -9' the postmaster
where I'm mapping out
the options.
--
* Greg Smith [EMAIL PROTECTED] http://www.gregsmith.com Baltimore, MD
---(end of broadcast)---
TIP 6: explain analyze is your friend
o do it than either using just diffs
or tagging.
I agree the wiki is the right place to hash this out; when that's
converged to something useful, then take a snapshot and convert for the
official manual. I'll work on it.
--
* Greg Smith [EMAIL PROTECTED
will sort through
any suggestions I get and turn that into formal documentation. This has
left enough shin damage that I'd enjoy completely slaying the cause.
--
* Greg Smith [EMAIL PROTECTED] http://www.gregsmith.com Baltimore, MD
---(end of broadcast)---
that went
along with the "OS/X startup scripts" thread recently made me realize how
many other people struggle(d) with this as well.
--
* Greg Smith [EMAIL PROTECTED] http://www.gregsmith.com Baltimore, MD
---(end of broadcast)---
TIP 1
application I found confusing.
--
* Greg Smith [EMAIL PROTECTED] http://www.gregsmith.com Baltimore, MD
---(end of broadcast)---
TIP 2: Don't 'kill -9' the postmaster
ow it rattles around inside the engine by logging, and get
some statistics on the result, now you've got someone who can build their
own simple patches without being so frustrated.
--
* Greg Smith [EMAIL PROTECTED] http://www.gregsmith.com Baltimore, MD
---
16MB write. I wouldn't mind seeing that exposed under
pg_stats instead, just had more interesting things to statify first.
--
* Greg Smith [EMAIL PROTECTED] http://www.gregsmith.com Baltimore, MD
---(end of broadcast)---
TIP 1: if posting/readi
tead of this log-based
patch. This circles back to the previous discussion of whether this
particular information is strictly developer-level.
--
* Greg Smith [EMAIL PROTECTED] http://www.gregsmith.com Baltimore, MD
---(end of broadcast)-
On Tue, 1 May 2007, Tom Lane wrote:
* Re: [PATCHES] Synchronized Scan WIP patch
/Simon Riggs/
Heikki is reviewing this one. Also I believe Greg Smith is doing more
performance testing.
Actually it was the "Automatic adjustment of bgwriter_lru_maxpages" patch
from Itagaki Tak
esign. It's harder to buffer those pesky
O_DIRECT WAL writes when they blow right though at least one level of
cache.
--
* Greg Smith [EMAIL PROTECTED] http://www.gregsmith.com Baltimore, MD
---(end of broadcast)---
TIP 9: In versions below
orkload that becomes
bottlenecked by WAL volume on a good OLTP server, particularly because
that's often going to a single or RAID-1 disk. Whether those workloads
also have the appropriate properties such that their WAL could be shrunk
usefully in real-time is a good question.
--
* Greg Smi
t of tweaking away from a
solid solution. Tuning the all scan, which is what you're talking about
when you speak in terms of the statistics about the overall buffer pool,
is a much harder job.
--
* Greg Smith [EMAIL PROTECTED] http://www.gregsmith.com Baltimore, MD
---
ly finished implementation yet; I just wanted to get an idea if this
was even feasible, or if there was some larger issue that made the whole
idea moot.
--
* Greg Smith [EMAIL PROTECTED] http://www.gregsmith.com Baltimore, MD
---(end of broadcast)--
ng the pages on the free
list, where the clients can grab them without running their own scan over
the buffer cache?
--
* Greg Smith [EMAIL PROTECTED] http://www.gregsmith.com Baltimore, MD
---(end of broadcast)---
TIP 2: Don't 'kill -9' the postmaster
in that it
doesn't write out more than it thinks it will need; you shouldn't get into
a situation where many more pages are written than will be used in the
near future. Given that mindset, shouldn't pages the LRU scan writes just
get moved onto the free list?
--
* Gre
L, http://hsqldb.org/ That's missing some useful
features Mimer added though, and the HSQL version comes with a funky
license. Their version is by no means bug-free either.
--
* Greg Smith [EMAIL PROTECTED] http://www.gregsmith.com Baltimore, MD
---
commit is done when it
hasn't hit disk is of no use for the applications I work with, so I
haven't even been paying attention to no-commit-wait.
--
* Greg Smith [EMAIL PROTECTED] http://www.gregsmith.com Baltimore, MD
---(end of broadcast)
That's the direct cause
of the biggest problem in that area of code, so why not stare right at it
rather than measuring it indirectly.
--
* Greg Smith [EMAIL PROTECTED] http://www.gregsmith.com Baltimore, MD
---(end of broadcast)---
TIP 6: explain analyze is your friend
e
something interesting after I collect more data.
--
* Greg Smith [EMAIL PROTECTED] http://www.gregsmith.com Baltimore, MD
---(end of broadcast)---
TIP 6: explain analyze is your friend
ling this "Felix Unger on
crack mode" in my notes). I needed some way to prioritize which buffers
to concentrate on when that happens, and so far the above has been a good
first-cut way to help with that.
--
* Greg Smith [EMAIL PROTECTED] http://www.gregsmith.com Baltimore, MD
7;m reporting the checkpoint details now,
still some work left to do on the bgwriter activity.
--
* Greg Smith [EMAIL PROTECTED] http://www.gregsmith.com Baltimore, MD
---(end of broadcast)---
TIP 6: explain analyze is your friend
the LRU background writer specifically tends to be very
underutilized when using pgbench.
--
* Greg Smith [EMAIL PROTECTED] http://www.gregsmith.com Baltimore, MD
---(end of broadcast)---
TIP 7: You can help support the PostgreSQ
skip_pinned" parameter to
something like "skip_active", which is closer to the real behavior. I
know Oracle refers to these as "hot" and "cold" LRU entries.
--
* Greg Smith [EMAIL PROTECTED] http://www.gregsmith.com Baltimore, MD
--
d soon (where it will conflict
considerably with your patch). If you have an 8.2 configuration you can
test with my patch applied, set log_min_messages = debug2, try it out, and
see what you get when running pgbench for a while. I think you'll
discover some interesting and unexpected things.
out,
and b) how many segments have been written. The timeout one is easy to
work with that way (from what I read of your code, you've worked that
angle). The part I had trouble doing was getting the WAL writers to
communicate a progress report on how many segments they filled back to the
bgwri
ak pages written
at any one checkpoint. Lowering that value is really the end-game for
optimizing the background writer, and the peaks are what will nail you
with a nasty fsync pause at checkpoint time.
--
* Greg Smith [EMAIL PROTECTED] http://www.gregsmith.com Baltimore
background writer logging than the one I originally outlined, that I'll
have ready for feedback in a few more days. Now it dumps a nice summary
report every time it completes a scan, rather than logging lower-level
info.
--
* Greg Smith [EMAIL PROTECTED] http://w
checkpoint data
because if you're in a situation where are inserting fast enough that the
checkpoints are spaced closely together, you'd end up having to poll
pg_stats all the time for make sure you catch them all, which becomes even
less efficient than just logging the data.
--
* Greg Smi
27;t strong enough
to override the issues you bring up with the average case.
--
* Greg Smith [EMAIL PROTECTED] http://www.gregsmith.com Baltimore, MD
---(end of broadcast)---
TIP 4: Have you searched our list archives?
http://archives.postgresql.org
you
just brought it back up here.
The arguments for COPY are performance and that you don't need to specify
the table name. INSERT is slower and you need a name, but it's easier to
build a UNIX tool style pipeline to import it in real-time.
--
* Greg Smith [EMAIL PROTECTED] htt
I am aware that I am too deep into this to
have an unbiased opinion at this point though, which is why I ask for
feedback on how to proceed here.
--
* Greg Smith [EMAIL PROTECTED] http://www.gregsmith.com Baltimore, MD
---(end of broadcast)---
TIP 7
dn't get distracted by fixing that implementation when it's
functional enough for most who are satisfied with stderr output.
--
* Greg Smith [EMAIL PROTECTED] http://www.gregsmith.com Baltimore, MD
---(end of broadcast)---
TIP 5: don't forget to increase your free space map settings
The main source of confusion about current support for this feature is
that the desktop/workstation version of Windows don't have it. For
Windows XP, you need the XP Professional version to get "dynamic disk"
support; it's not in the home edition.
--
* Greg Smith [E
modifies the first match will probably be OK.
I consider using the same name as the default log directory helpful, but
would understand that others might consider it confusing to overload the
name like that.
--
* Greg Smith [EMAIL PROTECTED] http://www.gregsmith.com Baltimore, MD
x27;s looked
into this topic, but I didn't find much information on it out there.
3) I'm open to suggestions for where to put this information at in a form
more integrated into the community. That's part of why I didn't sweat the
formatting yet, I'm expecting at least o
g on a per-device basis (which, by the way, isn't
available unless you're running a more recent Linux kernel than most many
distributions have available). You can tune the schedulers all day and
not make a lick of difference to what I've been running into; I know, I
tr
eady in the OS
buffer, therefore avoiding an actual disk read; that substantially reduces
the typical penalty for the database engine making a bad choice on what to
evict. I fear a move to direct writes would put more pressure on the LRU
implementation to be very smart, and that's co
1501 - 1570 of 1570 matches
Mail list logo
