Does it make sense to pre-sort COPY FROM input to produce long runs of
increasing values of an indexed column, or does PostgreSQL perform
this optimization on its own?
--
Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org)
To make changes to your subscription:
http://www.p
the mistake, by undeleting the database files
at the operating system level. This has been made more difficult
(perhaps even impossible) by your subsequent write activity.
--
Florian Weimer
BFK edv-consulting GmbH http://www.bfk.de/
Kriegsstraße 100 tel: +49
* Yeb Havinga:
> On 2011-07-19 12:47, Florian Weimer wrote:
>>
>>> It would be interesting to see if the drives also show total xyz
>>> written, and if that differs a lot too.
>> Do you know how to check that with smartctl?
> smartctl -a /dev/ should show all
* Yeb Havinga:
> On 2011-07-19 09:56, Florian Weimer wrote:
>> * Yeb Havinga:
>>
>>> The biggest drawback of 2 SSD's with supercap in hardware raid 1, is
>>> that if they are both new and of the same model/firmware, they'd
>>> probably reach the
but I've got two Intel 320s (I suppose, the report
device model is "SSDSA2CT040G3") in a RAID 1 configuration, and after
about a month of testing, one is down to 89 on the media wearout
indicator, and the other is still at 96. Both devices are
deteriorating, but one at a significa
are might be
> to old to not know about the SSD lifetime indicator or not even show
> it.
3ware controllers offer SMART pass-through, and smartctl supports it.
I'm sure there's something similar for Areca controllers.
--
Florian Weimer
BFK edv-consulting GmbH
* Thomas Hägi:
> how can i get postgres to use the indexes when querying the master
> table?
I believe that this is a new feature in PostgreSQL 9.1 ("Allow
inheritance table queries to return meaningfully-sorted results").
--
Florian Weimer
BFK edv-consulting
ly in all
cases. (The E5320-based system is likely non-NUMA.)
Speaking about NUMA, do you know if there are some non-invasive tools
which can be used to monitor page migration and off-node memory
accesses?
--
Florian Weimer
BFK edv-consulting GmbH http://www.bfk.de/
Kriegsst
* Kevin Grittner:
> Dave Crooke wrote:
>
>> create table data
>>(id_key int,
>> time_stamp timestamp without time zone,
>> value double precision);
>>
>> create unique index data_idx on data (id_key, time_stamp);
>
>> I need to find the most recent value for each distinct value o
* Mladen Gogala:
> Did anyone try using "shake" while the cluster is active?
As far as I can tell, it's totally unsafe.
--
Florian Weimer
BFK edv-consulting GmbH http://www.bfk.de/
Kriegsstraße 100 tel: +49-721-96201-1
D-76133 Karlsruhe
cle Berkeley DB files). This phenomenon is
less pronounced with PostgreSQL because it splits large relations into
one-gigabyte chunks, and it writes the files sequentally. But a small
effect is probably still there.
--
Florian Weimer
BFK edv-consulting GmbH http://www.bfk
them are
cache misses.
--
Florian Weimer
BFK edv-consulting GmbH http://www.bfk.de/
Kriegsstraße 100 tel: +49-721-96201-1
D-76133 Karlsruhe fax: +49-721-96201-99
--
Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org)
To make
* Andres Freund:
> On Friday, January 07, 2011 01:45:25 PM Florian Weimer wrote:
>> On 9.0, this configuration
>>
>> checkpoint_segments = 512 # in logfile segments, min 1, 16MB each
>>
>> results in 1034 segments, so the effective logfile segment size is
would also make sense to mention that increasing the
segment count decreases WAL traffic, and that changing this value does
not have an impact on transaction sizes?
--
Florian Weimer
BFK edv-consulting GmbH http://www.bfk.de/
Kriegsstraße 100 tel: +49-721-96201
hy is such a broad lock needed? If the table was created in the
current transaction and is empty, the contents of the foreign key
table should not matter.
--
Florian Weimer
BFK edv-consulting GmbH http://www.bfk.de/
Kriegsstraße 100 tel: +49-721-96201-1
D-7
ing" in pg_stat_activity? In this case, I'm also wondering why
this is inecessary.
--
Florian Weimer
BFK edv-consulting GmbH http://www.bfk.de/
Kriegsstraße 100 tel: +49-721-96201-1
D-76133 Karlsruhe fax: +49-721-96201-99
--
Sent
* Mladen Gogala:
> I have a logical problem with asynchronous commit. The "commit"
> command should instruct the database to make the outcome of the
> transaction permanent. The application should wait to see whether the
> commit was successful or not. Asynchronous behavior in the commit
> stateme
* Greg Smith:
> Given the size of your database, I'd advise you consider a migration
> to a new version ASAP. 8.4 is a nice stable release at this point,
> that's the one to consider moving to.
It also offers asynchronous commits, which might be a good tradeoff
here (especially if the data gathe
to the comparison
> operators?
Ah, I see, I probably need to provide a RESTRICT clause in the
operator definition. That should do the trick, and should be fairly
easy to implement in this case.
Sorry, I just missed this piece of information in the documentation, I
should have read it more ca
types, but explicit BETWEEN ... AND queries.
--
Florian Weimer
BFK edv-consulting GmbH http://www.bfk.de/
Kriegsstraße 100 tel: +49-721-96201-1
D-76133 Karlsruhe fax: +49-721-96201-99
--
Sent via pgsql-performance mailing list (pgsql-p
igh number of write operations
per second, so a software-only solution might not be available.
--
Florian Weimer
BFK edv-consulting GmbH http://www.bfk.de/
Kriegsstraße 100 tel: +49-721-96201-1
D-76133 Karlsruhe fax: +49-721-96201-99
--
Sent vi
another 400 to 800 MB.
--
Florian Weimer
BFK edv-consulting GmbH http://www.bfk.de/
Kriegsstraße 100 tel: +49-721-96201-1
D-76133 Karlsruhe fax: +49-721-96201-99
--
Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org)
To make
We don't
see it on systems we have switched to the deadline I/O scheduler. But
data on this is a bit sketchy.
--
Florian Weimer
BFK edv-consulting GmbH http://www.bfk.de/
Kriegsstraße 100 tel: +49-721-96201-1
D-76133 Karlsruhe fax: +49-721-96201-99
> Florian Weimer:
>
>> > Do you have any further idea why 16MB/s seems to be the limit here?
>>
>> BYTEA deserialization is very slow, and this could be a factor here.
>> Have you checked that you are in fact I/O bound?
>
> Could you elaborate that a bit? I
ams().
--
Florian Weimer
BFK edv-consulting GmbH http://www.bfk.de/
Kriegsstraße 100 tel: +49-721-96201-1
D-76133 Karlsruhe fax: +49-721-96201-99
--
Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org)
To make changes to y
orm range queries? Or something like "WHERE col LIKE '%string%')?
--
Florian Weimer
BFK edv-consulting GmbH http://www.bfk.de/
Kriegsstraße 100 tel: +49-721-96201-1
D-76133 Karlsruhe fax: +49-721-96201-99
--
Sent via pgsql-perform
g to write the RAM disk out before
> losing power...
The cache warm-up time can also be quite annoying. Of course, with
flash-backed DRAM, this is a concern as long as you use the cheaper,
slower variants for the backing storage.
--
Florian Weimer
BFK edv-consultin
r makes sense. Do these
> applications allocate a terrabyte of memory? I doubt it.
SBCL sizes its allocated memory region based on the total amount of
RAM and swap space. In this case, buying larger disks does not
help. 8-P
--
Florian Weimer<[EMAIL PROTECTED]>
BFK edv-consul
VM (at least some extent).
--
Florian Weimer<[EMAIL PROTECTED]>
BFK edv-consulting GmbH http://www.bfk.de/
Kriegsstraße 100 tel: +49-721-96201-1
D-76133 Karlsruhe fax: +49-721-96201-99
--
Sent via pgsql-performance mailing list (pgsql
else runs on
> the box.
Have you disabled the OOM killer?
--
Florian Weimer<[EMAIL PROTECTED]>
BFK edv-consulting GmbH http://www.bfk.de/
Kriegsstraße 100 tel: +49-721-96201-1
D-76133 Karlsruhe fax: +49-721-96201-99
--
Sent via pgsql-
d I've since switched to the deadline scheduler, too.
So far, this particular behavior hasn't occurred again.
--
Florian Weimer<[EMAIL PROTECTED]>
BFK edv-consulting GmbH http://www.bfk.de/
Kriegsstraße 100 tel: +49-721-96201-1
D-76133 Karlsruhe
ll also periodically run queries to determine the size of
> the intersection of two sets for all pairs of sets (in order to
> generate some nice graphs).
I think it's very difficult to compute that efficiently, but I haven't
thought much about it. This type of query might benefit from your
ke sense for me to
> choose?
See 2.
In general, hashing is bad because it destroy locality. But in some
cases, there is no other choice.
--
Florian Weimer<[EMAIL PROTECTED]>
BFK edv-consulting GmbH http://www.bfk.de/
Kriegsstraße 100 tel: +49-721-9
* Dimi Paun:
> * 4.9 million records in a table (IP address info)
You should use the ip4r type for that.
--
Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance
d. Good luck finding a SSD NAS with a
60 usec round-trip time. 8->
Something which directly speaks SATA or PCI might offer comparable
performance, but SSD alone isn't sufficient.
--
Florian Weimer<[EMAIL PROTECTED]>
BFK edv-consulting GmbH http://www.bfk.de/
Krieg
A dedicated RAID controller with battery-backed cache of ssuficient
size and two mirrored disks should not perform that bad, and has the
advantage of easy availability.
--
Florian Weimer<[EMAIL PROTECTED]>
BFK edv-consulting GmbH http://www.bfk.de/
Kriegsstraße 100
* Guillaume Smet:
> On Jan 22, 2008 9:32 AM, Florian Weimer <[EMAIL PROTECTED]> wrote:
>> > Maybe it's just my test box.. single SATA-II drive, XFS on top of LVM.
>>
>> Ours was ext3, no LVM or RAID.
>
> Also with SATA?
Yes, desktop-class SATA.
>
test box.. single SATA-II drive, XFS on top of LVM.
Ours was ext3, no LVM or RAID.
--
Florian Weimer<[EMAIL PROTECTED]>
BFK edv-consulting GmbH http://www.bfk.de/
Kriegsstraße 100 tel: +49-721-96201-1
D-76133 Karlsruhe fax: +49-721-96201-99
ument for a separate CIDR type, not against a host
type.
I agree that a host type would be helpful. I currently use check
constraints to ensure no one accidentally stores data of the wrong
type, which is not ideal (space is not a main concern at this point).
--
Florian Weimer<
that the CLI accepts it, it's
also the standard[*] output format (that is, "192.0.2.0" instead of
"192.0.2.0/24"; if no prefix length is given, the one based on the
class is used).
[*] I don't think you can switch it off. Obviously, for backwards
compatibil
> seek/read/calculate/seek/write since the drive moves on after the
> read), when you read you must read _all_ drives in the set to check
> the data integrity.
I don't know of any RAID implementation that performs consistency
checking on each read operation. 8-(
---(end of
y
controllers support that.
--
Florian Weimer<[EMAIL PROTECTED]>
BFK edv-consulting GmbH http://www.bfk.de/
Kriegsstraße 100 tel: +49-721-96201-1
D-76133 Karlsruhe fax: +49-721-96201-99
---(end of broadcast)-
in case
someone performs a huge SELECT locally, resulting in a a client
process sucking up all available memory. With vm.overcommit_memory=2,
memory allocation in the client process will fail. Without it,
typically the postgres process feeding it is killed by the kernel.
--
Florian Weimer
he file in shared/backup mode. The only
lock that is created by that guards deletion and renaming. It can
still lead to obscure failures, but it's not a wholly-eclusive lock.
--
Florian Weimer<[EMAIL PROTECTED]>
BFK edv-consulting GmbH http://www.bfk.de/
Kriegsstra
tables and indexes manually among the file systems. This
is a bit disappointing, especially because the system is able to read
at 800+ MB/s, as shown by the software-RAID-on-hardware-RAID
configuration.
I haven't seen 24-disk benchmarks with Areca controllers. A
comparison mig
* Willo van der Merwe:
> Florian Weimer wrote:
>> You need to run "vmstat 10" (for ten-second averages) and report a
>> couple of lines.
> 2 80 1
> 5 0 61732 37052 28180 34319560014 987 2320 2021 38
> sda3 3.30 0.00
avg-cpu: %user %nice%sys %iowait %idle
> 17.180.001.930.81 80.08
Same for iostat.
Your initial numbers suggest that your server isn't I/O-bound, though
(the percentage spent in iowait is much too small, and so are the tps
numbers).
--
Florian Weimer
plement fast, out-of-order B-tree scans anyway. 8-/
I still think that preallocating in reasonably sized chunks is
beneficial.
--
Florian Weimer<[EMAIL PROTECTED]>
BFK edv-consulting GmbH http://www.bfk.de/
Kriegsstraße 100 tel: +49-721-96201-1
D-76133
leveling nowadays. I wouldn't worry
about the issue, unless your write rates are pretty high.
--
Florian Weimer<[EMAIL PROTECTED]>
BFK edv-consulting GmbH http://www.bfk.de/
Kriegsstraße 100 tel: +49-721-96201-1
D-76133 Karlsruhe f
encoding, mostly for
performance reasons. (Proper UTF-8 can be enforced through
constraints if necessary.)
--
Florian Weimer<[EMAIL PROTECTED]>
BFK edv-consulting GmbH http://www.bfk.de/
Kriegsstraße 100 tel: +49-721-96201-1
D-76133 Karls
a socket
management POV, though.)
--
Florian Weimer<[EMAIL PROTECTED]>
BFK edv-consulting GmbH http://www.bfk.de/
Kriegsstraße 100 tel: +49-721-96201-1
D-76133 Karlsruhe fax: +49-721-96201-99
---(end of broadcast)--
index scan be faster
> with this data type compared to plain varchar ?
It will be faster because less I/O is involved.
For purposes like yours, there is a special ip4 type in a contributed
package which brings down the byte count to 4. I'm not sure if it's
been ported to PostgreSQL 8
ch for my data
set? Are the individual strings too long, maybe?
(This is with PostgreSQL 8.2.0, BTW.)
--
Florian Weimer<[EMAIL PROTECTED]>
BFK edv-consulting GmbH http://www.bfk.de/
Kriegsstraße 100 tel: +49-721-96201-1
D-76133 Karlsruhe
matically after
major updates).
--
Florian Weimer<[EMAIL PROTECTED]>
BFK edv-consulting GmbH http://www.bfk.de/
Kriegsstraße 100 tel: +49-721-96201-1
D-76133 Karlsruhe fax: +49-721-96201-99
---(end of broadcast)---
is is far more effective than
vacuuming, but obviously, this approach cannot be used in all cases
(e.g. if you need more dynamic expiry rules).
--
Florian Weimer<[EMAIL PROTECTED]>
BFK edv-consulting GmbH http://www.bfk.de/
Kriegsstraße 100 tel: +49
Linux releases (from
kernel.org) are acceptable, you should assume that the problem will
eventually fix itself. FWIW, I see the 9x overhead on something that
is close to 2.6.17 (on AMD64/Opteron), so this could be wishful
thinking. 8-(
--
Florian Weimer<[EMAIL PROTECTED]&g
For instance, some chips don't like the CMOV
instruction at all, but others can process it with decent speed.
--
Florian Weimer<[EMAIL PROTECTED]>
BFK edv-consulting GmbH http://www.bfk.de/
Kriegsstraße 100 tel: +49-721-96201-1
D-76133 Karlsruhe
for 64K transfer sizes.
Linux enables window scaling, so the actual window size can be more
than 64K. Windows should cope with it, but some PIX firewalls and
other historic boxes won't.
--
Florian Weimer<[EMAIL PROTECTED]>
BFK edv-consulting GmbH http://www.bfk.de/
ivejournal.com/2116715.html>
Enabling write cache leads to various degrees of data corruption in
case of a power outage (possibly including file system corruption
requiring manual recover).
--
Florian Weimer<[EMAIL PROTECTED]>
BFK edv-consulting GmbH http://www.b
* Jim C. Nasby:
> What's interesting is that apparently FreeBSD also has overcommit (and
> IIRC no way to disable it), yet I never hear people going off on OOM
> kills in FreeBSD. My theory is that FreeBSD admins are smart enough to
> dedicate a decent amount of swap space, so that by the time you
conds
doesn't matter. Only if you need to wait five minutes, it's a
different story.
It seems that the situation is under control now. Thanks.
--
Florian Weimer<[EMAIL PROTECTED]>
BFK edv-consulting GmbH http://www.bfk.de/
Durlacher Allee 47
* Tom Lane:
> Florian Weimer <[EMAIL PROTECTED]> writes:
>> -> Bitmap Index Scan on large_rel_1_field_1 (cost=0.00..2003.09
>> rows=193739 width=0) (actual time=0.148..0.148 rows=12 loops=1)
>>Index Cond: (n.field_1 = "outer".fi
ash join would be faster even if
there where 74 matching rows in smaller_rel instead of just one. The
estimate decreases when I increase the portion of smaller_rel which is
scanned by ANALYZE (to something like 10% of the table), but this
doesn't look like a solution.
Any suggestions?
(The queries hav
after a crash. Even if
you've got a trustworthy file system checker (there are surprisingly
few of them, especially for advanced file systems without fixed data
structure locations), running it after a crash usually leads to
unacceptably high downtime.
--
Florian Weimer<[EMA
to our hardware supplier, beyound
8 GB, the price per GB goes up sharply.) Unfortunately, it seems that
the Core 2 Duo mainboards do not change that much in this area.
--
Florian Weimer<[EMAIL PROTECTED]>
BFK edv-consulting GmbH http://www.bfk.de/
Durlacher Allee 47
> I could I discover who is sending so many data to the disks?
Documentation/laptop-mode.txt in the Linux kernel tree has some
instructions how to track down unwanted disk writes.
--
Florian Weimer<[EMAIL PROTECTED]>
BFK edv-consulting GmbH http://www.bfk.de/
* Stephen Frost:
> Actually, can't you stick multiple inserts into a given 'statement'?
> ie: insert into abc (123); insert into abc (234);
IIRC, this breaks with PQexecParams, which is the recommended method
for executing SQL statements nowadays.
--
Florian Weimer
* FROM tmp;
If you need some kind of SELECT/INSERT/UPDATE cycle, it's far more
complex, of course, and I'm not quite happy with what I'm using right
now.
--
Florian Weimer<[EMAIL PROTECTED]>
BFK edv-consulting GmbH http://www.bfk.de/
Durlacher Allee
ry table if
you need more complex calculations. If you do this, there won't be a
huge difference between local and remote access as long as the
bandwidth is sufficient.
--
Florian Weimer<[EMAIL PROTECTED]>
BFK edv-consulting GmbH http://www.bfk.de/
Durlacher A
* Greg Stark:
> Didn't byteas used to get printed as hex?
No, they didn't. It would be useful to support hexadecimal BYTEA
literals, though. Unfortunately, X'DEADBEEF' has already been taken
by bit strings.
--
Florian Weimer<[EMAIL PROTECTED
* Jignesh K. Shah:
> * llseek is high which means you can obviously gain a bit with the
> right file system/files tuning by caching them right.
It might also make sense to switch from lseek-read/write to
pread/pwrite. It shouldn't be too hard to hack this into the virtual
file descriptor module.
* Hannes Dorbath:
> + Hardware Raids might be a bit easier to manage, if you never spend a
> few hours to learn Software Raid Tools.
I disagree. RAID management is complicated, and once there is a disk
failure, all kinds of oddities can occur which can make it quite a
challenge to get back a non
* Sriram Dandapani:
> Does the inet data type offer comparison/search performance benefits
> over plain text for ip addresses..
Queries like "host << '192.168.17.192/28'" use an available index on
the host column. In theory, you could do this with LIKE and strings,
but this gets pretty messy and
* Neil Conway:
> On Wed, 2006-02-15 at 18:28 -0500, Tom Lane wrote:
>> It seems clear that our qsort.c is doing a pretty awful job of picking
>> qsort pivots, while glibc is mostly managing not to make that mistake.
>> I haven't looked at the glibc code yet to see what they are doing
>> differentl
Vivek Khera wrote:
> If you've got the time, could you try also doing the full bulk insert
> test with the checkpoint log files on another physical disk? See if
> that's any faster.
We have been doing that for a few weeks, but the performance
improvements are less than what we expected. There i
Florian Weimer wrote:
> After an upgrade to 7.4.1 (from 7.3) we see a severe performance
> regression in bulk INSERTs.
In turns out that we were running the default configuration, and not the
tuned one in /etc/postgresql. *blush*
After increasing the number of checkpoint segments and the
After an upgrade to 7.4.1 (from 7.3) we see a severe performance
regression in bulk INSERTs.
This is apparently caused by constant checkpointing (every 10 to 20
seconds). I've already increased the number of checkpoint segments to
32, but currently, there are just 10 or 11 files in the pg_xlog
di
77 matches
Mail list logo