I ran into the same problem with VACUUM on my Linux box. If you are running
Linux, take a look at "elvtune" or read this post:
http://groups.google.com/groups?q=stephen+vacuum+linux&hl=en&lr=&ie=UTF-8&se
lm=gRdjb.7484%241o2.77%40nntp-post.primus.ca&rnum=3
Regards,
f 'union'. Just a
thought.
Stephen
signature.asc
Description: Digital signature
w/ a test case that others could reproduce where
explain isn't returning? I think that would be very useful towards
solving at least that issue...
Thanks,
Stephen
signature.asc
Description: Digital signature
een 32bit and
64bit as you'd see on a sparc platform.
Enjoy,
Stephen
signature.asc
Description: Digital signature
* Ksenia Marasanova ([EMAIL PROTECTED]) wrote:
> Any tips are greatly appreciated.
EXPLAIN ANALYZE of the same queries would be much more useful.
Thanks,
Stephen
signature.asc
Description: Digital signature
for some reason) from a regular join to a left join with a
filtering in the application which is probably a bad move... If you can
use at least some filtering in the database I expect that'd help..
Thanks,
Stephen
signature.asc
Description: Digital signature
ing
full commits but it might be something to consider.
> When I finally get all of this sorted out and working reasonably optimally,
> I'll be sure to come back and report what techniques/settings did and didn't
> work for this workload.
That'd be great, many thank
client side? Or
does binary mode not work with copy (that wouldn't suprise me, but
perhaps copy could be made to support it)?
The other thought, of course, is that you could use PITR for your
backups instead of pgdump...
Thanks,
Stephen
signature.asc
Description: Digital signature
* Luke Lonergan ([EMAIL PROTECTED]) wrote:
> On 12/2/05 12:18 PM, "Stephen Frost" <[EMAIL PROTECTED]> wrote:
> > Just a thought, but couldn't psql be made to use the binary mode of
> > libpq and do at least some of the conversion on the client side? Or
>
ansaction safe,
and has problems with data reliability (aiui, equivilant to doing 'fsync
= false' for Postgres). InnoDB, again iirc, is transaction safe and
whatnot, and more akin to the default PostgreSQL setup.
I expect some others will comment along these lines too, if my response
isn
planner will, for example, do a hashjoin which is much faster than
sorting and merge-joining, but takes alot of memory... They may say
"hey, I like it being fast" but not consider what happens when alot of
those queries run at once..
Thanks!
Stephen
signature.asc
Description: Digital signature
he occational 16MB copy
off the disk...
Thanks,
Stephen
> On 28-May-07, at 12:31 PM, Heikki Linnakangas wrote:
>
> >Dave Cramer wrote:
> >>Since PITR has to enable archiving does this not increase the
> >>amount of disk I/O required ?
> >
> >
> AFAIK you can't RAID1 more than two drives, so the above doesn't make sense
> to me.
It's just more copies of the same data if it's really a RAID1, for the
extra, extra paranoid. Basically, in the example above, I'd read it as
"D1, D2, D5 have identical
r it will be slower to keep in sync on a heavy write system.
I'm not sure, but I don't think most RAID1 systems do reads against all
drives and compare the results before returning it to the caller... I'd
be curious if I'm wrong.
Thanks,
Stephen
si
ster when doing a hash-join vs. a sort + merge-join.
Could likely be because it doesn't think there's enough work memory
available for the hash, which might change based on the values it gets
from the statistics on how frequently something shows up, etc.
Enjoy,
Stephen
signature.asc
Description: Digital signature
ly run on our data warehouse box, which is a
nice dual-proc/dual-core DL385 w/ 16GB of ram. :) The annoying thing is
that I can still run it out of memory sometimes, even w/ 16GB. :/
Thanks,
Stephen
signature.asc
Description: Digital signature
s without doing 'explain analyze', but I think you have
a good idea of the best plan for this query already...
Thanks,
Stephen
signature.asc
Description: Digital signature
1138,1139}'::integer[]))
-> Bitmap Index Scan on reading_sensor
(cost=0.00..528.37 rows=28650 width=0)
Index Cond: (sensor_id = ANY
('{1137,1138,1139}'::integer[]))
(8 rows)
TIA,
Stephen Davies
--
=
ted faster than =a or =b or =c. Am I wrong for PostgreSQL?
Stephen
On Wednesday 22 August 2007 22:55, Michael Glaesemann wrote:
> On Aug 22, 2007, at 5:58 , Russell Smith wrote:
> > Stephen Davies wrote:
> >> select count(rdate),rdate from reading where sensor_id in
> >>
l | numeric(7,3)|
Indexes:
"reading_pkey" PRIMARY KEY, btree (id)
"unique_sensor_date" UNIQUE, btree (sensor_id, rdate)
"date" btree (rdate)
"reading_sensor" btree (sensor_id)
Foreign-key constraints:
"$1" FOREIGN KEY (sen
Herrera wrote:
> I don't think you showed us the EXPLAIN ANALYZE results that Scott
> requested.
--
This email is for the person(s) identified above, and is confidential to
the sender and the person(s). No one else is authorised to use or
disseminate this email
ting the various plans up somewhere online (perhaps a
pastebin like http://pgsql.privatepaste.com) instead of or in addition
to sending it in the email.
Thanks!
Stephen
signature.asc
Description: Digital signature
are almost the
> same). What's the best tool to time queries in postgresql? Can this be
> done from pgadmin?
As was mentioned elsewhere, certainly the best tool to test with is your
actual application, if that's possible.. Or at least the language your
application is in.
Thanks,
Stephen
signature.asc
Description: Digital signature
y more complicated).
> As in my previous tests, this is only a testing environment: so all data
> is in memory, no disk activity involved at all, no swap etc.
Yea, system calls still aren't free. I would recommend, if you care
about this query, bumping up your work_mem setting for it. Right now,
PG is using an external sort (meaning- on-disk), but the data set
appears to only be like 50M (49560kB). If you increased work_mem to,
say, 128MB (for this query, maybe or maybe not for the entire system),
it'd be able to do an in-memory sort (or maybe a hash or something else,
if it makes sense), which would be faster.
I'd probably rewrite this as a left-join too, to be honest, but based on
what I'm saying, that'd probably get the same query plan as you had
first anyway (the merge anti-join), so it's probably not necessary. I'd
love to hear how PG performs with work_mem bumped up to something
decent...
Thanks,
Stephen
signature.asc
Description: Digital signature
sure that those constraints are
really the right ones and that they make sense? You're using a bunch of
extract()'s there too, why not just specify a CHECK constraint on the
date ranges which are allowed in the table..?
Maybe I've misunderstood the whole point here, but I don't think so.
Thanks,
Stephen
signature.asc
Description: Digital signature
day > user_stop_year ||
user_stop) THEN
m.taken BETWEEN (user_start_year || user_start_day)::date AND
((user_stop_year || user_stop)::date + '1
year'::interval)::date
-- I don't think you need/want this..?
-- GROUP BY
s in your partitioning setup. What is
constraint_exclusion set to? What version of PG is this? Do the
results og this query look at all correct to you?
Have you considered an index on elevation, btw? How many records in
that city table are there and how many are actually in that range?
Thanks,
ly
efficient way (not doing seq. scans through everything because you're
operating on every row for something). It needs to be a couple
hundred-thousand rows, but it doesn't need to be the full data set, imv.
Thanks,
Stephen
signature.asc
Description: Digital signature
these stored procedures from within a C-based function which
> does the logging, math, control logic, and builds the result sets and
> cursors?
Uhh, I'd guess 'no' to that one.
Thanks,
Stephen
signature.asc
Description: Digital signature
saction.
That's not really relevant.. Is it called alot from the same
backend/database connection? If so, and if you're using regular SELECT
statements and the like (not EXECUTE), then they're getting prepared the
first time they're used and that is kept across transactions.
to be an overall improvment of, say,
10%, or a 10-fold improvment. :)
Thanks,
Stephen
signature.asc
Description: Digital signature
built-ins now available in 8.4 (array_agg), that copying
doesn't happen any more.
Thanks,
Stephen
signature.asc
Description: Digital signature
tabase. If you're building alot of indexes then you probably
want to split up the statements into multiple connections and run them
in parallel.
Thanks,
Stephen
signature.asc
Description: Digital signature
r any other PG operations (PG would use at most
4GB-shared_buffers, or so).
Thanks,
Stephen
signature.asc
Description: Digital signature
ULL) OR
> (persons.modified > indexing_persons.indexed))
>-> Seq Scan on persons (cost=0.00..4438.29
> rows=143629 width=16) (actual time=0.000..56.000 rows=143629 loops=1)
>-> Hash (cost=2534.86..2534.86 rows=142286 width=16)
> (actual tim
included (which, really, shouldn't be that big a deal).
Did you look at what the other reply suggested? Do you have
constraint_exclusion = 'on' in your postgresql.conf?
Thanks,
Stephen
signature.asc
Description: Digital signature
x27;s entirely possible that the index is *not*
the fastest way to pull this data (it's nearly 10% of the table..), if
the stats were better it might use a seq scan instead, not sure how bad
the cost of the filter itself would be.
Thanks,
Stephen
signature.asc
Description: Digital signature
etting to use...)? Increasing work_mem is often a good
idea if your system can afford it based on the number/kind of queries
running concurrently. Note that you can also increase that setting for
just a single role, single session, or even single query.
Thanks,
Stephen
signature.as
which can often be due to
missing pieces from the JOIN clause or misunderstanding of the database
schema...).
Stephen
signature.asc
Description: Digital signature
ntly useful to
test with..
Stephen
signature.asc
Description: Digital signature
could try to replicate the lseek() contention?
I can give it a shot, but the impression I had from the paper is that
the lseek() contention wouldn't be seen without the changes to the lock
manager...? Or did I misunderstand?
Thanks,
Stephen
signature.asc
Description: Digital signature
es atomic changes instead of spinlocks for
certain locking situations..
If that's all the MIT folks did, they certainly made it sound like alot
more. :)
Stephen
signature.asc
Description: Digital signature
running out of memory.
> I think the first thing to do is look into using a connection pooler
> like pgpool to reduce your connection memory overhead.
Yeah.. Having the number of database connections be close to the number
of processors is usually recommended.
Stephen
signature.asc
Description: Digital signature
#x27;t great, but honestly is surprisingly good all things
> considered.
I'm kind of suprised at each connection taking 100MB, especially ones
which are just doing simple inserts.
Thanks,
Stephen
signature.asc
Description: Digital signature
all the same actual memory, not 100M
per process.
Thanks,
Stephen
signature.asc
Description: Digital signature
en checkpoints, so that the checkpoints
aren't as big and painful. That can be done by making the background
writer more aggressive.
Thanks,
Stephen
signature.asc
Description: Digital signature
When
you run top, is your PG process constantly in 'D' state, or is it in 'R'
state, or what? Might help figure some of that out. Note that
parallelizing the query will help regardless of if it's disk bound or
CPU bound, when you're running on the kind of hardware you're talking
about (lots of spindles, multiple CPUs, etc).
Thanks,
Stephen
signature.asc
Description: Digital signature
be even *faster* would be the run all 5 of
those queries against the child tables in parallel (given that you have
over 5 CPUs and enough memory that you don't start swapping).
If it's still too big on the per-child basis, you might be able to use
conditionals to do the first 100 strands, then the next hundred, etc.
>I appreciate the comments thus far.
Let's hope you'll always appreciate them. :)
Thanks,
Stephen
signature.asc
Description: Digital signature
ex or
> cluster.
Just to share my experiences- I've found that creating a new table and
inserting into it is actually faster than doing full-table updates, if
that's an option for you.
Thanks,
Stephen
signature.asc
Description: Digital signature
Does it have less work to do? That's a
bit harder to say but my guess is "not so much that you'd actually be
able to notice it."..
Thanks,
Stephen
signature.asc
Description: Digital signature
* Mark Stosberg (m...@summersault.com) wrote:
> Recommendations?
PostGIS, geometry columns, and UTM.. I'm not sure where they are wrt
adding KNN support, but it's something they've been anxious to have for
a while, so I expect support will come quickly.
Thanks,
from the same problem that
work_mem has, specifically that PG still won't allocate more than
1GB of memory for any single operation.
Thanks,
Stephen
signature.asc
Description: Digital signature
e shortcuts but the
times you're posting for Postgres seem quite far off based on the
hardware and commands you've described...
Thanks,
Stephen
signature.asc
Description: Digital signature
* Tom Lane ([EMAIL PROTECTED]) wrote:
> Stephen Frost <[EMAIL PROTECTED]> writes:
> > While it's true that Access almost certainly takes some shortcuts, 24
> > minutes for an update across 1.2 millon rows seems an awefully long time
> > for Postgres.
>
>
Could someone explain the results of the following? This is with postgres 8.1.2 on a database that was just vacuum-verbose-analyzed. I have packets_i4 index which I am expecting to be used with this query but as you can see, I have have to convince its usage by turning off other scans. The total
"Steinar H. Gunderson" <[EMAIL PROTECTED]> wrote: It sounds like PostgreSQL badly overestimates the cost of the index scan.Does the table perchance fit completely into memory, withouteffective_cache_size indicating that? Don't know the exact way to answer your question, but my initial instinct is
I repeated explain analyze on the query 5 times and it came up with the same plan. You asked about index order and physical table order. In general the index order is indeed close to the same order as the physical table order. However, this query is likely an exception. The data is actually fro
06 at 12:38:18PM -0700, Stephen Byers wrote:> I repeated explain analyze on the query 5 times and it came up with the same plan.Yes, but did it end up with the same runtime? That's the interesting part --the plan will almost always be identical between explain analyze runs giventhat you haven
You may be comparing the values to Tom's suggestion to bump up work_mem. Take a look at the original posting (Total runtime: 777208.041 ms for the bitmap scan) -Steve "Steinar H. Gunderson" <[EMAIL PROTECTED]> wrote: On Thu, May 18, 2006 at 12:53:16PM -0700, Stephen By
he
cacheing, so as long as your OS can see all the memory you have in the
box, that benefit of running 64bit isn't going to be seen on Postgres.
On many other database systems (notably the 800-pound gorillas...) the
database handle the cacheing and so wants to basically have control over
all
led 'multiarch' and is being discussed with LSB and
other distros too, though I think it did mostly originated with Debian
folks.
Just my 2c.
Thanks,
Stephen
signature.asc
Description: Digital signature
lly dual-controller load-balanced and it works quite decently.
Indeed, the EMC SANs are generally the high-priced ones too, so not
really sure what to tell you about the poor performance you're seeing
out of it. Your IT folks and/or your EMC rep. should be able to resolve
that, really...
En
; >> that, really...
> >
> >
> >The only exception I've heard to this is the Clarion AX150. We looked at
> >one and we were warned off of it by some EMC gearheads.
Yeah, the Clarion is the EMC "cheap" line, and I think the AX150 was the
extra-cheap one which Dell rebranded and sold.
Thanks,
Stephen
signature.asc
Description: Digital signature
o the vacuum's run for?
If it's 3 hours, then that might start to be an issue with disk I/O
contention...
> Hopefully I've supplied enough information to start diagnosing the
> problem. Any ideas, thoughts, suggestions are greatly appreciated ...
Just my 2c, hopefully you'll get some better answers too. :)
Thanks,
Stephen
signature.asc
Description: Digital signature
OLTP) per transaction.
Enjoy,
Stephen
signature.asc
Description: Digital signature
* Florian Weimer ([EMAIL PROTECTED]) wrote:
> * Stephen Frost:
> > Actually, can't you stick multiple inserts into a given 'statement'?
> > ie: insert into abc (123); insert into abc (234);
>
> IIRC, this breaks with PQexecParams, which is the recommended m
uSe (ES, Fedora, who knows) and pretty much any kernel you build
using sources off of kernel.org or for any other distribution unless you
know exactly what versions/patches they support.
Feel free to contact me off-list if you'd like to continue this
discussion since I don't really see
ry useful tools too and can help with hardware
decisions but you probably want to review your queries and make sure the
database is performing as best it can with the setup you have today
before throwing more hardware at it.
Thanks,
Stephen
signature.asc
Description: Digital signature
e a distro-specific issue
and what wouldn't), the support folks are willing to help debug it.
Thanks,
Stephen
signature.asc
Description: Digital signature
ead and that
might be inconsistant between the systems (mainly it may have to do with
the gettimeofday() calls being implemented differently between Windows
and Linux..).
Thanks,
Stephen
signature.asc
Description: Digital signature
;t
make sense to split them across different nodes if they're accessing the
same memory- every memory access would have to be checked) like the
PostgreSQL server.
> 2. It is advantageous to buy AMD 64 rather than the Pentium IV?
Personally, I certainly think so. More registers, more me
register-starved i386 platforms which increases the speed for most
applications where it usually wouldn't when recompiled for 64bit.
Stephen
signature.asc
Description: Digital signature
mory if that were happening.
Could this optimization be added to PostgreSQL? It sounds like a very
reasonable thing to do. Hopefully there wouldn't be too much complexity
needed to add it.
Stephen
signature.asc
Description: Digital signature
* Tom Lane ([EMAIL PROTECTED]) wrote:
> Stephen Frost <[EMAIL PROTECTED]> writes:
> > * Tom Lane ([EMAIL PROTECTED]) wrote:
> >> [... thinks for awhile ...] It seems possible that they may use sort
> >> code that knows it is performing a DISTINCT operation and
* Stephen Frost ([EMAIL PROTECTED]) wrote:
> systems does in 40 seconds. My only other concern is the Oracle system
> having to do the write I/O while the postgres one doesn't... I don't
> see an obvious way to get around that though, and I'm not sure if it'd
width=83)
Filter: ((date = '2004-09-07'::date) AND ((stock)::text = 'QQQ'::text))
(2 rows)
Any help would be appreciated
--Stephen
Table "public.island_history"
Column | Type | Modifiers
--+---
On Mon, 13 Sep 2004 21:11:07 -0400, Tom Lane <[EMAIL PROTECTED]> wrote:
> Stephen Crowley <[EMAIL PROTECTED]> writes:
> > Does postgres cache the entire result set before it begins returning
> > data to the client?
>
> The backend doesn't, but libpq doe
I get some extra time I'll look into libpq and see what is required
to fix the API. Most thirdparty programs and existing JDBC apps won't
work with the current paradigm when returning large result sets.
Thanks,
Stephen
On Mon, 13 Sep 2004 21:49:14 -0400, Tom Lane <[EMAIL PROT
a.keyA,
a.keyB,
a.colA,
a.colB
from Y a left join X b
using (keyA, keyB)
where b.keyA is NULL and
b.keyB is NULL;
With the appropriate indexes, this is pretty fast but I think a merge
would be much faster.
Thanks,
Ste
4-09-07'::date) AND ((stock)::text =
'MSFT'::text))"
"Total runtime: 346759.000 ms"
Nearly 8 minutes.. Why would it take this long? Is there anything else
I can do to debug this?
When I set enable_seqscan to OFF and force everything to use the index
every stock I query ret
uot;
"Total runtime: 201009.000 ms"
So now this in all in proportion and works as expected.. the question
is, why would the fact that it needs to be vaccumed cause such a huge
hit in performance? When i vacuumed it did free up nearly 25% of the
space.
--Stephen
On Fri, 17 Sep 2004 2
Thanks for the explanation. So what sort of changes need to be made to
the client/server protocol to fix this problem?
On Thu, 23 Sep 2004 18:22:15 -0500 (EST), Kris Jurka <[EMAIL PROTECTED]> wrote:
>
>
> On Tue, 14 Sep 2004, Stephen Crowley wrote:
>
> > Problem solv
ould
having them be 'UNION ALL's work?
Stephen
signature.asc
Description: Digital signature
n, if you have time before you
have to go into production with the new solution (sounds like you do-
changing databases takes time anyway).
> Thanks in advance for any help you may have, I'll do my best to keep
> pgsql-performance up to date
> on the results.
Hope that helps. Others on here will correct me if I misspoke. :)
Stephen
signature.asc
Description: Digital signature
me way such that it can be spread across multiple machines, then
if you need to combine the data have it be replicated using slony to a
big box that has a view which joins all the tables and do your big
queries against that.
Just some thoughts.
Stephen
signature.asc
Description: Digital signature
* Herv? Piedvache ([EMAIL PROTECTED]) wrote:
> Le Jeudi 20 Janvier 2005 15:30, Stephen Frost a écrit :
> > * Herv? Piedvache ([EMAIL PROTECTED]) wrote:
> > > Is there any solution with PostgreSQL matching these needs ... ?
> >
> > You might look into pg_pool. Anot
/ the guy from Cox Communications and I thought he was using
it :).
> So, your only option is Oracle or another very expensive commercial
> database.
Or partition the data at the application layer.
Stephen
signature.asc
Description: Digital signature
which machine to contact for what
data.
Stephen
signature.asc
Description: Digital signature
* [EMAIL PROTECTED] ([EMAIL PROTECTED]) wrote:
> I think maybe a SAN in conjunction with tablespaces might be the answer.
> Still need one honking server.
That's interesting- can a PostgreSQL partition be acress multiple
tablespaces?
Stephen
signature.asc
Description: Digital signature
#x27;ll help w/ whatever hardware you end up going with.
Enjoy,
Stephen
signature.asc
Description: Digital signature
I think either would work; both PostgreSQL and MS SQL Server have
success stories out there running VLDBs. It really depends on what you
know and what you have. If you have a lot of experience with Postgres
running on Linux, and not much with SQL Server on Windows, of course the
former would
_cost_query'
variable where someone working in a data warehouse situation would get a
notice if the query he's hand-crafting has a very high cost (in which
case he could ctrl-c it if he thinks something is wrong, rather than
waiting 5 hours before realizing he forgot a join clause), but the
(id = $0)
Stephen Denne
Disclaimer:
At the Datamail Group we value team commitment, respect, achievement, customer
focus, and courage. This email with any attachments is confidential and may be
subject to legal privilege. If it is not intended for you please advise by
reply immediately, destroy
> "Stephen Denne" <[EMAIL PROTECTED]> writes:
> > A simple update query, over roughly 17 million rows,
> > populating a newly added column in a table, resulted in an
> > out of memory error when the process memory usage reached
> > 2GB. Could this be
I don't have a PostgreSQL build environment.
It is now Friday night for me. I left the alternate query running, and will
find out on Monday what happened.
If I drop the fk constraint, and/or its index, would I still be affected by the
leak you found?
Regards,
Stephen
>>"Stephen Denne" <[EMAIL PROTECTED]> writes:
>>> I altered the update statement slightly, and reran the query.
>>> The altered query has been running over 3 hours now,
>>> without using lots of memory (38M private bytes).
>>> 2046 temp
Tom Lane wrote:
> "Stephen Denne" <[EMAIL PROTECTED]> writes:
> > So dropping the fk constraint and index results in
> successful query execution with constant memory usage. Does
> this confirm that the memory leak you found is the one I was
> suffering fro
2113544.412 rows=0 loops=1)
> Filter: (fklistingsourceid = 5525)
Would it help to have a combined index on fklistingsourceid, entrydate?
Regards,
Stephen Denne.
Disclaimer:
At the Datamail Group we value team commitment, respect, achievement, customer
focus, and courage
ght have a huge number of dead rows in your table, slowing down
the sequential scan.
(Likewise updating a third of the rows, changing an indexed field.)
What do you get from:
VACUUM VERBOSE u_counts;
Regards,
Stephen Denne.
Disclaimer:
At the Datamail Group we value team commitment, respect, achie
ansaction is still alive.
Alternatively, it may be a different 2 million dead row versions now than
earlier, and may simply be a side effect of your particular usage, and nothing
to worry about. (Though it is exactly the same number of rows, which strongly
hints at being exactly the same rows.)
R
1 - 100 of 216 matches
Mail list logo