Just saw your email between all the others .. Pinterest, Instagram,
Netflix, Shazam, NASDAQ, Cycle Computing (
http://arstechnica.com/business/2011/09/3-core-cluster-built-on-amazon-ec2-cloud/)
.. that list could go on and on, see
http://aws.amazon.com/solutions/case-studies/ for some more.
Fo
Disks are doing 150 read + 90 write ops/s when they should be able to do a
total of 1000 iops each as currently configured (this is the max that can
be set). Total bandwidth is 1000mb/s each too. So clearly, either there is
something wrong with ZFS/FreeBSD on Amazon (either because of config or
som
On 09/13/12 2:08 PM, Sébastien Lorion wrote:
I started db creation over, this time with 16GB maintenance_work_mem
and fsync=off and it does not seem to have a great effect. After again
5 hours, during index creation, disk and cpu are barely used: 95% idle
and 2-3 MB/s writes (150 reads/s, 90 wr
I started db creation over, this time with 16GB maintenance_work_mem and
fsync=off and it does not seem to have a great effect. After again 5 hours,
during index creation, disk and cpu are barely used: 95% idle and 2-3 MB/s
writes (150 reads/s, 90 writes/s).
Sébastien
On Thu, Sep 13, 2012 at 1:29
maintenance_work_mem is already 4GB. How large should it be during load
then ?
Sébastien
On Thu, Sep 13, 2012 at 1:29 AM, John R Pierce wrote:
> On 09/12/12 10:01 PM, Sébastien Lorion wrote:
>
>> pgbench initialization has been going on for almost 5 hours now and still
>> stuck before vacuum st
pgbench initialization has been going on for almost 5 hours now and still
stuck before vacuum starts .. something is definitely wrong as I don't
remember it took so long first time I created the db. Here are the current
stats now:
*iostat (xbd13-14 are WAL zpool)*
device r/s w/skr/s
On 09/12/12 10:01 PM, Sébastien Lorion wrote:
pgbench initialization has been going on for almost 5 hours now and
still stuck before vacuum starts .. something is definitely wrong as I
don't remember it took so long first time I created the db
pgbench initialization with a high scale facto
On Tue, Aug 21, 2012 at 1:18 AM, Vincent Veyron wrote:
> Le mardi 21 août 2012 à 01:33 -0400, Sébastien Lorion a écrit :
>
> >
> >
> > Since Amazon has added new high I/O instance types and EBS volumes,
> > anyone has done some benchmark of PostgreSQL on them ?
> >
>
> I wonder : is there a reaso
Forgot to say that this is it with new values suggested (see included
postgresql.conf) and ARC cache size set to 32GB.
Sébastien
On Wed, Sep 12, 2012 at 9:16 PM, Sébastien Lorion
wrote:
> I recreated the DB and WAL pools, and launched pgbench -i -s 1. Here
> are the stats during the load (st
I recreated the DB and WAL pools, and launched pgbench -i -s 1. Here
are the stats during the load (still running):
*iostat (xbd13-14 are WAL zpool)*
device r/s w/skr/skw/s qlen svc_t %b
xbd8 0.0 471.5 0.0 14809.3 40 67.9 84
xbd7 0.0 448.1 0.0 14072.6 3
One more question .. I could not set wal_sync_method to anything else but
fsync .. is that expected or should other choices be also available ? I am
not sure how the EC2 SSD cache flushing is handled on EC2, but I hope it is
flushing the whole cache on every sync .. As a side note, I got corrupted
Is dedicating 2 drives for WAL too much ? Since my whole raid is comprised
of SSD drives, should I just put it in the main pool ?
Sébastien
On Wed, Sep 12, 2012 at 8:28 PM, Sébastien Lorion
wrote:
> Ok, make sense .. I will update that as well and report back. Thank you
> for your advice.
>
> Sé
Ok, make sense .. I will update that as well and report back. Thank you for
your advice.
Sébastien
On Wed, Sep 12, 2012 at 8:04 PM, John R Pierce wrote:
> On 09/12/12 4:49 PM, Sébastien Lorion wrote:
>
>> You set shared_buffers way below what is suggested in Greg Smith book
>> (25% or more of R
The DB back-end of my application has 2 use cases:
- a normalized master DB, sharded by userid (based on their activity, not a
formula such as modulo, because some users can be 1-2 order of magnitude
more active than others)
- many denormalized read-only slaves, with some different models dependi
On 09/12/12 4:49 PM, Sébastien Lorion wrote:
You set shared_buffers way below what is suggested in Greg Smith book
(25% or more of RAM) .. what is the rationale behind that rule of
thumb ? Other values are more or less what I set, though I could lower
the effective_cache_size and vfs.zfs.arc_ma
You set shared_buffers way below what is suggested in Greg Smith book (25%
or more of RAM) .. what is the rationale behind that rule of thumb ? Other
values are more or less what I set, though I could lower the
effective_cache_size and vfs.zfs.arc_max and see how it goes.
Sébastien
On Wed, Sep 12
On 09/12/12 4:03 PM, Sébastien Lorion wrote:
I agree 1GB is a lot, I played around with that value, but it hardly
makes a difference. Is there a plateau in how that value affects query
performance ? On a master DB, I would set it low and raise as
necessary, but what would be a good average valu
I agree 1GB is a lot, I played around with that value, but it hardly makes
a difference. Is there a plateau in how that value affects query
performance ? On a master DB, I would set it low and raise as necessary,
but what would be a good average value on a read-only DB with same spec and
max_connec
On 09/12/12 3:17 PM, François Beausoleil wrote:
What are the iostat / vmstat numbers during the test?
note you need to run iostat with -x and ignore the first
sample as its average since reboot. I usually use 5, 10, or 30 second
intervals when analyzing IO performance problems.
on a s
Le 2012-09-12 à 17:08, Sébastien Lorion a écrit :
> As you can see, I am nowhere near the results John mentioned for a 10,000
> scale (about 8000 tps) and I am not sure why. My instance setup and
> configuration should be ok, but I am far from an expert (a startup founder
> has to wear many ha
Finally I got time to setup an instance and do some tests.
Instance:
High-Mem 4x large (8 cores, 68 GB)
EBS-Optimized flag set (allow up to 1000 Mbits/s transfer)
10GB standard EBS for OS
8x100GB in RAID10 for data (max 1000 iops)
2x100GB in RAID0 for WAL (max 1000 iops)
FreeBSD 9.0
PostgreSQL
I will be setting up an instance in the coming days and post the results
here.
While reading on the subject, I found this interesting discussion on
YCombinator:
http://news.ycombinator.com/item?id=4264754
Sébastien
On Thu, Aug 23, 2012 at 2:41 PM, John R Pierce wrote:
> On 08/23/12 11:24 AM,
On 08/23/12 11:24 AM, Sébastien Lorion wrote:
I think both kind of tests (general and app specific) are
complementary and useful in their own way. At a minimum, if the
general ones fail, why go to the expenses of doing the specific ones ?
Setting up a meaningful application test can take a lot
I think both kind of tests (general and app specific) are complementary and
useful in their own way. At a minimum, if the general ones fail, why go to
the expenses of doing the specific ones ? Setting up a meaningful
application test can take a lot of time and it can be hard to pinpoint
exactly whe
On 08/23/12 6:49 AM, Craig Ringer wrote:
In this case, what he's doing is seeking generalized performance
measurements. I don't think details were particularly necessary until
it got pulled off-track.
"42"
performance measurements without a very narrow definition of
'performance' are usele
On Wed, Aug 22, 2012 at 4:12 PM, Alan Hodgson wrote:
> On Wednesday, August 22, 2012 04:10:01 PM Andrew Hannon wrote:
>> Just looking into High IO instances for a DB deployment. In order to get
>> past 1TB, we are looking at RAID-0. I have heard
>> (http://hackerne.ws/item?id=4266119) there might
On 08/23/2012 07:39 PM, Vincent Veyron wrote:
Le mercredi 22 août 2012 à 13:30 -0400, Sébastien Lorion a écrit :
Vincent, I would appreciate that you stop assuming things based on
zero information about what I am doing. I understand that you are
trying to be helpful, but I can assure you that go
Vincent,
The original question can be summed up by "how is general performance of
PostgreSQL on Amazon IOPS". I fail to understand why that would require me
to explain the specifics of my application and/or my market. The only one
asking for that information is you, while others have provided usef
On Thu, Aug 23, 2012 at 7:39 AM, Vincent Veyron wrote:
> Lest we ridicule ourselves publicly, I suggest we leave the discussion
> at that and wish you luck in your endeavor.
If anyone has an answer to his question, I'd appreciate hearing it,
despite any faux pas that the OP has committed.
--
S
Le mercredi 22 août 2012 à 13:30 -0400, Sébastien Lorion a écrit :
> Vincent, I would appreciate that you stop assuming things based on
> zero information about what I am doing. I understand that you are
> trying to be helpful, but I can assure you that going bare-metal only
> does not make any sen
On Wednesday, August 22, 2012 04:10:01 PM Andrew Hannon wrote:
> Just looking into High IO instances for a DB deployment. In order to get
> past 1TB, we are looking at RAID-0. I have heard
> (http://hackerne.ws/item?id=4266119) there might be a problem if TRIM isn't
> supported. Does anyone know if
Just looking into High IO instances for a DB deployment. In order to get past
1TB, we are looking at RAID-0. I have heard
(http://hackerne.ws/item?id=4266119) there might be a problem if TRIM isn't
supported. Does anyone know if it is and has anyone used RAID-0 on these
instances? (Linux of cou
Vincent, I would appreciate that you stop assuming things based on zero
information about what I am doing. I understand that you are trying to be
helpful, but I can assure you that going bare-metal only does not make any
sense in my context.
Sébastien
On Wed, Aug 22, 2012 at 12:44 PM, Vincent Vey
Le mercredi 22 août 2012 à 13:15 +0800, Craig Ringer a écrit :
> He appears to be suggesting that buying access to real hardware in a
> datacenter (if not buying the hardware yourself) is more cost effective
> and easier to manage than using "cloud" style services with more
> transient hosts li
On 08/21/2012 09:40 PM, David Boreham wrote:
On 8/21/2012 2:18 AM, Vincent Veyron wrote:
I wonder : is there a reason why you have to go through the complexity
of such a setup, rather than simply use bare metal and get good
performance with simplicity?
In general I agree -- it is much (much!) c
Oops sorry, I thought I did hit reply all.
I am not sure this mailing list is the right place to have this debate
(assuming it is needed, there are plenty of articles stating the benefits
of using the cloud), so I will simply answer that you pay the cost of the
added layer up front (mostly scripti
Le mardi 21 août 2012 à 09:36 -0500, Merlin Moncure a écrit :
> here's a datapoint, stock config:
> pgbench -i -s 500
> pgbench -c 16 -T 60
> number of transactions actually processed: 418012
> tps = 6962.607292 (including connections establishing)
> tps = 6973.154593 (excluding connections establ
On Tue, Aug 21, 2012 at 12:33 AM, Sébastien Lorion
wrote:
> Hello,
>
> Since Amazon has added new high I/O instance types and EBS volumes, anyone
> has done some benchmark of PostgreSQL on them ?
>
> http://perspectives.mvdirona.com/2012/07/20/IOPerformanceNoLongerSucksInTheCloud.aspx
> http://per
On 8/21/2012 2:18 AM, Vincent Veyron wrote:
I wonder : is there a reason why you have to go through the complexity
of such a setup, rather than simply use bare metal and get good
performance with simplicity?
In general I agree -- it is much (much!) cheaper to buy tin and deploy
yourself vs any o
On 8/21/2012 7:10 AM, Oliver Kohll - Mailing Lists wrote:
This is a general 'cloud or dedicated' question, I won't go into it
but I believe cloud proponents cite management ease, scalability etc.
I'm sure there's a place for every type of hosting. However I would be
interested in hearing some
On 21 Aug 2012, at 13:32, Vincent Veyron wrote:
>>
>> Since Amazon has added new high I/O instance types and EBS volumes,
>> anyone has done some benchmark of PostgreSQL on them ?
>>
>
> I wonder : is there a reason why you have to go through the complexity
> of such a setup, rather than simpl
Le mardi 21 août 2012 à 01:33 -0400, Sébastien Lorion a écrit :
>
>
> Since Amazon has added new high I/O instance types and EBS volumes,
> anyone has done some benchmark of PostgreSQL on them ?
>
I wonder : is there a reason why you have to go through the complexity
of such a setup, rather th
Le mardi 21 août 2012 à 01:33 -0400, Sébastien Lorion a écrit :
>
>
> Since Amazon has added new high I/O instance types and EBS volumes,
> anyone has done some benchmark of PostgreSQL on them ?
>
I wonder : is there a reason why you have to go through the complexity
of such a setup, rather th
Hello,
Since Amazon has added new high I/O instance types and EBS volumes, anyone
has done some benchmark of PostgreSQL on them ?
http://perspectives.mvdirona.com/2012/07/20/IOPerformanceNoLongerSucksInTheCloud.aspx
http://perspectives.mvdirona.com/2012/08/01/EBSProvisionedIOPSOptimizedInstanceTy
44 matches
Mail list logo