On Thu, Oct 24, 2013 at 5:41 AM, Thom Brown wrote:
> On 5 September 2013 22:24, Bruce Momjian wrote:
>> On Mon, Aug 19, 2013 at 09:27:57PM -0400, Stephen Frost wrote:
>>> * Andres Freund (and...@2ndquadrant.com) wrote:
>>> > I vote for adapting the patch to additionally zero out the file via
>>>
On 5 September 2013 22:24, Bruce Momjian wrote:
> On Mon, Aug 19, 2013 at 09:27:57PM -0400, Stephen Frost wrote:
>> * Andres Freund (and...@2ndquadrant.com) wrote:
>> > I vote for adapting the patch to additionally zero out the file via
>> > write(). In your tests that seemed to perform at least a
On Mon, Aug 19, 2013 at 09:27:57PM -0400, Stephen Frost wrote:
> * Andres Freund (and...@2ndquadrant.com) wrote:
> > I vote for adapting the patch to additionally zero out the file via
> > write(). In your tests that seemed to perform at least as good as the
> > old method... It also has the advant
(2013/09/05 0:04), Andres Freund wrote:
> I'd vote for adding zeroing *after* the fallocate() first.
+1, with FALLOC_FL_KEEP_SIZE flag.
At least, fallocate with FALLOC_FL_KEEP_SIZE flag is faster than nothing in my
developing sorted checkpoint. I adopted it to relation file, so I don't know
abo
On Wed, Sep 4, 2013 at 11:26 AM, Tom Lane wrote:
> Stephen Frost writes:
>> * Andres Freund (and...@2ndquadrant.com) wrote:
>>> I'd vote for adding zeroing *after* the fallocate() first. That's what's
>>> suggested by kernel hackers and what several other large applications
>>> do. As it looks li
Stephen Frost writes:
> * Andres Freund (and...@2ndquadrant.com) wrote:
>> I'd vote for adding zeroing *after* the fallocate() first. That's what's
>> suggested by kernel hackers and what several other large applications
>> do. As it looks like it's what we would have to do if we ever get to use
>
Hi,
On 2013-09-04 11:15:37 -0400, Stephen Frost wrote:
> * Andres Freund (and...@2ndquadrant.com) wrote:
> > I'd vote for adding zeroing *after* the fallocate() first. That's what's
> > suggested by kernel hackers and what several other large applications
> > do. As it looks like it's what we woul
* Andres Freund (and...@2ndquadrant.com) wrote:
> I'd vote for adding zeroing *after* the fallocate() first. That's what's
> suggested by kernel hackers and what several other large applications
> do. As it looks like it's what we would have to do if we ever get to use
> fallocate for relation exte
On Thu, 2013-08-08 at 17:42 -0400, Tom Lane wrote:
> Jon Nelson writes:
> > At this point I'm convinced that the issue is a pathological case in
> > ext4. The performance impact disappears as soon as the unwritten
> > extent(s) are written to with real data. Thus, even though allocating
> > files
On 2013-09-04 08:01:30 -0700, Jeff Davis wrote:
> On Thu, 2013-08-08 at 17:42 -0400, Tom Lane wrote:
> > Jon Nelson writes:
> > > At this point I'm convinced that the issue is a pathological case in
> > > ext4. The performance impact disappears as soon as the unwritten
> > > extent(s) are written
On Mon, Aug 19, 2013 at 09:27:57PM -0400, Stephen Frost wrote:
> * Andres Freund (and...@2ndquadrant.com) wrote:
> > I vote for adapting the patch to additionally zero out the file via
> > write(). In your tests that seemed to perform at least as good as the
> > old method... It also has the advant
* Andres Freund (and...@2ndquadrant.com) wrote:
> I vote for adapting the patch to additionally zero out the file via
> write(). In your tests that seemed to perform at least as good as the
> old method... It also has the advantage that we can use it a littlebit
> more as a testbed for possibly usi
On 2013-08-19 14:40:07 -0500, Jon Nelson wrote:
> On Fri, Aug 16, 2013 at 3:57 PM, Bruce Momjian wrote:
> > On Thu, Aug 15, 2013 at 12:08:57PM -0500, Jon Nelson wrote:
> >> > Where are we on this issue?
> >>
> >> I've been able to replicate it pretty easily with PostgreSQL and
> >> continue to loo
On Fri, Aug 16, 2013 at 3:57 PM, Bruce Momjian wrote:
> On Thu, Aug 15, 2013 at 12:08:57PM -0500, Jon Nelson wrote:
>> > Where are we on this issue?
>>
>> I've been able to replicate it pretty easily with PostgreSQL and
>> continue to look into it. I've contacted Theodore Ts'o and have gotten
>> s
On Thu, Aug 15, 2013 at 12:08:57PM -0500, Jon Nelson wrote:
> > Where are we on this issue?
>
> I've been able to replicate it pretty easily with PostgreSQL and
> continue to look into it. I've contacted Theodore Ts'o and have gotten
> some useful information, however I'm unable to replicate the b
> Where are we on this issue?
I've been able to replicate it pretty easily with PostgreSQL and
continue to look into it. I've contacted Theodore Ts'o and have gotten
some useful information, however I'm unable to replicate the behavior
with the test program (even one that's been modified). What I'
On Wed, Aug 7, 2013 at 07:04:43PM +0100, Thom Brown wrote:
> On 7 August 2013 18:49, Jon Nelson wrote:
> > On Wed, Aug 7, 2013 at 12:42 PM, Thom Brown wrote:
> >> for i in 1 2 5 10 100; do ./test_fallocate foo $i 1; done
> >> method: classic. 1 open/close iterations, 1 rewrite in 0.6380s
> >> me
On 2013-08-08 22:58:42 -0500, Jon Nelson wrote:
> On Thu, Aug 8, 2013 at 9:27 PM, Andres Freund wrote:
> > On 2013-08-08 16:12:06 -0500, Jon Nelson wrote:
> ...
>
> >> At this point I'm convinced that the issue is a pathological case in
> >> ext4. The performance impact disappears as soon as the
On Thu, Aug 8, 2013 at 9:27 PM, Andres Freund wrote:
> On 2013-08-08 16:12:06 -0500, Jon Nelson wrote:
...
>> At this point I'm convinced that the issue is a pathological case in
>> ext4. The performance impact disappears as soon as the unwritten
>> extent(s) are written to with real data. Thus,
On 2013-08-08 16:12:06 -0500, Jon Nelson wrote:
> On Thu, Aug 8, 2013 at 2:50 PM, Hannu Krosing wrote:
> > On 08/08/2013 05:28 PM, Jon Nelson wrote:
> ...
>
> > Just an idea - can you check if using a fillfactor different form 100
> > changes anything
> >
> > pgbench -s 20 -p 54320 -d pgb -F 90 -
On Thu, Aug 8, 2013 at 5:25 PM, Jon Nelson wrote:
> On Thu, Aug 8, 2013 at 5:23 PM, Tom Lane wrote:
>> Jon Nelson writes:
>>> On Thu, Aug 8, 2013 at 4:42 PM, Tom Lane wrote:
Does your test program use all the same writing options that the real
WAL writes do (like O_DIRECT)?
>>
>>> I b
On Thu, Aug 8, 2013 at 5:23 PM, Tom Lane wrote:
> Jon Nelson writes:
>> On Thu, Aug 8, 2013 at 4:42 PM, Tom Lane wrote:
>>> Does your test program use all the same writing options that the real
>>> WAL writes do (like O_DIRECT)?
>
>> I believe so.
>
>>> From xlog.c:
>
>> /* do not use get_sy
Jon Nelson writes:
> On Thu, Aug 8, 2013 at 4:42 PM, Tom Lane wrote:
>> Does your test program use all the same writing options that the real
>> WAL writes do (like O_DIRECT)?
> I believe so.
>> From xlog.c:
> /* do not use get_sync_bit() here --- want to fsync only at end of fill */
>
On Thu, Aug 8, 2013 at 4:42 PM, Tom Lane wrote:
> Jon Nelson writes:
>> At this point I'm convinced that the issue is a pathological case in
>> ext4. The performance impact disappears as soon as the unwritten
>> extent(s) are written to with real data. Thus, even though allocating
>> files with p
On Thu, Aug 8, 2013 at 04:33:05PM -0500, Jon Nelson wrote:
> > How much slower would it be if we wrote it with zeros after
> > posix_fallocate() --- that would still give use single extents. Has
> > anyone tested to see if the write without test_fallocate() still gives
> > us one extent?
>
> Act
Jon Nelson writes:
> At this point I'm convinced that the issue is a pathological case in
> ext4. The performance impact disappears as soon as the unwritten
> extent(s) are written to with real data. Thus, even though allocating
> files with posix_fallocate is - frequently - orders of magnitude
>
On Thu, Aug 8, 2013 at 4:24 PM, Bruce Momjian wrote:
> On Thu, Aug 8, 2013 at 04:12:06PM -0500, Jon Nelson wrote:
>> On Thu, Aug 8, 2013 at 2:50 PM, Hannu Krosing wrote:
>> > On 08/08/2013 05:28 PM, Jon Nelson wrote:
>> ...
>>
>> > Just an idea - can you check if using a fillfactor different for
On Thu, Aug 8, 2013 at 04:12:06PM -0500, Jon Nelson wrote:
> On Thu, Aug 8, 2013 at 2:50 PM, Hannu Krosing wrote:
> > On 08/08/2013 05:28 PM, Jon Nelson wrote:
> ...
>
> > Just an idea - can you check if using a fillfactor different form 100
> > changes anything
> >
> > pgbench -s 20 -p 54320 -d
On Thu, Aug 8, 2013 at 2:50 PM, Hannu Krosing wrote:
> On 08/08/2013 05:28 PM, Jon Nelson wrote:
...
> Just an idea - can you check if using a fillfactor different form 100
> changes anything
>
> pgbench -s 20 -p 54320 -d pgb -F 90 -i
>
>
>> pgbench -j 80 -c 80 -T 120 -p 54320 pgb
>> pg_ctl -D tt
On 08/08/2013 05:28 PM, Jon Nelson wrote:
> A follow-up.
> I found this thread that seems to explain some things:
>
> http://comments.gmane.org/gmane.comp.file-systems.ext4/33024
>
> Short version: if we are writing into the "middle" of the
> newly-fallocated file on ext4 (with extents) the extent
On 8 August 2013 04:05, Andres Freund wrote:
> On 2013-08-07 20:23:55 +0100, Thom Brown wrote:
>> >>> 269e78 was the commit immediately after 8800d8, so it appears that
>> >>> introduced the regression.
>> >>>
>> >>> "Use posix_fallocate() for new WAL files, where available."
>> >>
>> >> This is c
A follow-up.
I found this thread that seems to explain some things:
http://comments.gmane.org/gmane.comp.file-systems.ext4/33024
Short version: if we are writing into the "middle" of the
newly-fallocated file on ext4 (with extents) the extent tree can
fragment badly, causing poor performance due
On 2013-08-07 23:55:22 -0500, Jon Nelson wrote:
> > - enable log_checkpoints, post the lines spat out together with the results
> > - run pgbench without reinitializing the cluster/pgbench tables
> > inbetween?
>
> When I do this (as I note below), the performance difference (for me)
> disappear
On Wed, Aug 7, 2013 at 10:05 PM, Andres Freund wrote:
> On 2013-08-07 20:23:55 +0100, Thom Brown wrote:
>> >>> 269e78 was the commit immediately after 8800d8, so it appears that
>> >>> introduced the regression.
>> >>>
>> >>> "Use posix_fallocate() for new WAL files, where available."
>> >>
>> >>
On 2013-08-07 20:23:55 +0100, Thom Brown wrote:
> >>> 269e78 was the commit immediately after 8800d8, so it appears that
> >>> introduced the regression.
> >>>
> >>> "Use posix_fallocate() for new WAL files, where available."
> >>
> >> This is curious. Could you either run a longer test before/afte
On 08/07/2013 05:40 PM, Thom Brown wrote:
> On 8 August 2013 00:04, Thom Brown wrote:
>> On 7 August 2013 23:40, Greg Stark wrote:
>>> Did you report information about the system affected? What filesystem is it
>>> on? If it's ext4 does it have extents enabled?
>>
>> Yes, ext4. It's using whatev
On 8 August 2013 00:04, Thom Brown wrote:
> On 7 August 2013 23:40, Greg Stark wrote:
>> Did you report information about the system affected? What filesystem is it
>> on? If it's ext4 does it have extents enabled?
>
> Yes, ext4. It's using whatever the default options are, but running
> lsattr
On 7 August 2013 23:40, Greg Stark wrote:
> Did you report information about the system affected? What filesystem is it
> on? If it's ext4 does it have extents enabled?
Yes, ext4. It's using whatever the default options are, but running
lsattr on my data dir shows that extents are being used.
--
On Wed, Aug 7, 2013 at 4:18 PM, Kevin Grittner wrote:
> Thom Brown wrote:
>
>> pgbench -j 80 -c 80 -T 3600
>>
>> 269e78: 606.268013
>> 8800d8: 779.583129
I have also been running some tests and - as yet - they are
inconclusive. What I can say about them so far is that - at times -
they agree wit
Did you report information about the system affected? What filesystem is it
on? If it's ext4 does it have extents enabled?
I think on ext3 or ext4 without extents it won't have any benefit but it
shouldn't really be any slower either since the libc implementation is very
similar to what we used to
Thom Brown wrote:
> pgbench -j 80 -c 80 -T 3600
>
> 269e78: 606.268013
> 8800d8: 779.583129
As another data point I duplicated Thom's original tests:
max_connections = 500
shared_buffers = 4GB
effective_cache_size = 12GB
random_page_cost = 2.0
cpu_tuple_cost = 0.03
wal_buffers = 32MB
work_mem =
On 7 August 2013 17:54, Thom Brown wrote:
> On 7 August 2013 17:49, Andres Freund wrote:
>> On 2013-08-07 17:21:01 +0100, Thom Brown wrote:
>>> Only build option used was --enable-depend. I did have
>>> --enable-cassert for the shorter 5 min benchmarks, but was removed for
>>> the 30 min tests.
On 7 August 2013 18:49, Jon Nelson wrote:
> On Wed, Aug 7, 2013 at 12:42 PM, Thom Brown wrote:
>> for i in 1 2 5 10 100; do ./test_fallocate foo $i 1; done
>> method: classic. 1 open/close iterations, 1 rewrite in 0.6380s
>> method: posix_fallocate. 1 open/close iterations, 1 rewrite in 0.3204s
>
On Wed, Aug 7, 2013 at 12:42 PM, Thom Brown wrote:
> On 7 August 2013 17:54, Jon Nelson wrote:
>> On Wed, Aug 7, 2013 at 11:21 AM, Thom Brown wrote:
>>> Hi all,
>>>
>>> I recently tried a simple benchmark to see how far 9.4 had come since
>>> 8.4, but I discovered that I couldn't get 9.4 to even
On 7 August 2013 17:54, Jon Nelson wrote:
> On Wed, Aug 7, 2013 at 11:21 AM, Thom Brown wrote:
>> Hi all,
>>
>> I recently tried a simple benchmark to see how far 9.4 had come since
>> 8.4, but I discovered that I couldn't get 9.4 to even touch 8.4 for
>> performance. After checking 9.2 and 9.3
On 7 August 2013 17:49, Andres Freund wrote:
> On 2013-08-07 17:21:01 +0100, Thom Brown wrote:
>> Only build option used was --enable-depend. I did have
>> --enable-cassert for the shorter 5 min benchmarks, but was removed for
>> the 30 min tests.
>
>> pgbench -j 80 -c 80 -T 300:
>>
>> 8.4 - 535.
On Wed, Aug 7, 2013 at 11:21 AM, Thom Brown wrote:
> Hi all,
>
> I recently tried a simple benchmark to see how far 9.4 had come since
> 8.4, but I discovered that I couldn't get 9.4 to even touch 8.4 for
> performance. After checking 9.2 and 9.3 (as per Kevin Grittner's
> suggestion), I found th
On 2013-08-07 17:21:01 +0100, Thom Brown wrote:
> Only build option used was --enable-depend. I did have
> --enable-cassert for the shorter 5 min benchmarks, but was removed for
> the 30 min tests.
> pgbench -j 80 -c 80 -T 300:
>
> 8.4 - 535.990042
> 9.2 - 820.798141
> 9.3 - 828.395498
> 9.4 - 1
Hi all,
I recently tried a simple benchmark to see how far 9.4 had come since
8.4, but I discovered that I couldn't get 9.4 to even touch 8.4 for
performance. After checking 9.2 and 9.3 (as per Kevin Grittner's
suggestion), I found that those were fine, so the issue must be in
9.4devel. I used i
49 matches
Mail list logo
