Just saw your email between all the others .. Pinterest, Instagram,
Netflix, Shazam, NASDAQ, Cycle Computing (
http://arstechnica.com/business/2011/09/3-core-cluster-built-on-amazon-ec2-cloud/)
.. that list could go on and on, see
http://aws.amazon.com/solutions/case-studies/ for some more.
Disks are doing 150 read + 90 write ops/s when they should be able to do a
total of 1000 iops each as currently configured (this is the max that can
be set). Total bandwidth is 1000mb/s each too. So clearly, either there is
something wrong with ZFS/FreeBSD on Amazon (either because of config or
pgbench initialization has been going on for almost 5 hours now and still
stuck before vacuum starts .. something is definitely wrong as I don't
remember it took so long first time I created the db. Here are the current
stats now:
*iostat (xbd13-14 are WAL zpool)*
device r/s w/skr/s
maintenance_work_mem is already 4GB. How large should it be during load
then ?
Sébastien
On Thu, Sep 13, 2012 at 1:29 AM, John R Pierce pie...@hogranch.com wrote:
On 09/12/12 10:01 PM, Sébastien Lorion wrote:
pgbench initialization has been going on for almost 5 hours now and still
stuck
I started db creation over, this time with 16GB maintenance_work_mem and
fsync=off and it does not seem to have a great effect. After again 5 hours,
during index creation, disk and cpu are barely used: 95% idle and 2-3 MB/s
writes (150 reads/s, 90 writes/s).
Sébastien
On Thu, Sep 13, 2012 at
On 09/13/12 2:08 PM, Sébastien Lorion wrote:
I started db creation over, this time with 16GB maintenance_work_mem
and fsync=off and it does not seem to have a great effect. After again
5 hours, during index creation, disk and cpu are barely used: 95% idle
and 2-3 MB/s writes (150 reads/s, 90
Finally I got time to setup an instance and do some tests.
Instance:
High-Mem 4x large (8 cores, 68 GB)
EBS-Optimized flag set (allow up to 1000 Mbits/s transfer)
10GB standard EBS for OS
8x100GB in RAID10 for data (max 1000 iops)
2x100GB in RAID0 for WAL (max 1000 iops)
FreeBSD 9.0
PostgreSQL
Le 2012-09-12 à 17:08, Sébastien Lorion a écrit :
As you can see, I am nowhere near the results John mentioned for a 10,000
scale (about 8000 tps) and I am not sure why. My instance setup and
configuration should be ok, but I am far from an expert (a startup founder
has to wear many
On 09/12/12 3:17 PM, François Beausoleil wrote:
What are the iostat / vmstat numbers during the test?
note you need to run iostat with -x intervaland ignore the first
sample as its average since reboot. I usually use 5, 10, or 30 second
intervals when analyzing IO performance problems.
I agree 1GB is a lot, I played around with that value, but it hardly makes
a difference. Is there a plateau in how that value affects query
performance ? On a master DB, I would set it low and raise as necessary,
but what would be a good average value on a read-only DB with same spec and
On 09/12/12 4:03 PM, Sébastien Lorion wrote:
I agree 1GB is a lot, I played around with that value, but it hardly
makes a difference. Is there a plateau in how that value affects query
performance ? On a master DB, I would set it low and raise as
necessary, but what would be a good average
You set shared_buffers way below what is suggested in Greg Smith book (25%
or more of RAM) .. what is the rationale behind that rule of thumb ? Other
values are more or less what I set, though I could lower the
effective_cache_size and vfs.zfs.arc_max and see how it goes.
Sébastien
On Wed, Sep
On 09/12/12 4:49 PM, Sébastien Lorion wrote:
You set shared_buffers way below what is suggested in Greg Smith book
(25% or more of RAM) .. what is the rationale behind that rule of
thumb ? Other values are more or less what I set, though I could lower
the effective_cache_size and
The DB back-end of my application has 2 use cases:
- a normalized master DB, sharded by userid (based on their activity, not a
formula such as modulo, because some users can be 1-2 order of magnitude
more active than others)
- many denormalized read-only slaves, with some different models
Ok, make sense .. I will update that as well and report back. Thank you for
your advice.
Sébastien
On Wed, Sep 12, 2012 at 8:04 PM, John R Pierce pie...@hogranch.com wrote:
On 09/12/12 4:49 PM, Sébastien Lorion wrote:
You set shared_buffers way below what is suggested in Greg Smith book
Is dedicating 2 drives for WAL too much ? Since my whole raid is comprised
of SSD drives, should I just put it in the main pool ?
Sébastien
On Wed, Sep 12, 2012 at 8:28 PM, Sébastien Lorion
s...@thestrangefactory.comwrote:
Ok, make sense .. I will update that as well and report back. Thank you
One more question .. I could not set wal_sync_method to anything else but
fsync .. is that expected or should other choices be also available ? I am
not sure how the EC2 SSD cache flushing is handled on EC2, but I hope it is
flushing the whole cache on every sync .. As a side note, I got corrupted
I recreated the DB and WAL pools, and launched pgbench -i -s 1. Here
are the stats during the load (still running):
*iostat (xbd13-14 are WAL zpool)*
device r/s w/skr/skw/s qlen svc_t %b
xbd8 0.0 471.5 0.0 14809.3 40 67.9 84
xbd7 0.0 448.1 0.0 14072.6
Forgot to say that this is it with new values suggested (see included
postgresql.conf) and ARC cache size set to 32GB.
Sébastien
On Wed, Sep 12, 2012 at 9:16 PM, Sébastien Lorion
s...@thestrangefactory.comwrote:
I recreated the DB and WAL pools, and launched pgbench -i -s 1. Here
are the
On Tue, Aug 21, 2012 at 1:18 AM, Vincent Veyron vv.li...@wanadoo.fr wrote:
Le mardi 21 août 2012 à 01:33 -0400, Sébastien Lorion a écrit :
Since Amazon has added new high I/O instance types and EBS volumes,
anyone has done some benchmark of PostgreSQL on them ?
I wonder : is there a
On 09/12/12 10:01 PM, Sébastien Lorion wrote:
pgbench initialization has been going on for almost 5 hours now and
still stuck before vacuum starts .. something is definitely wrong as I
don't remember it took so long first time I created the db
pgbench initialization with a high scale
Le mercredi 22 août 2012 à 13:30 -0400, Sébastien Lorion a écrit :
Vincent, I would appreciate that you stop assuming things based on
zero information about what I am doing. I understand that you are
trying to be helpful, but I can assure you that going bare-metal only
does not make any sense
On Thu, Aug 23, 2012 at 7:39 AM, Vincent Veyron vv.li...@wanadoo.fr wrote:
Lest we ridicule ourselves publicly, I suggest we leave the discussion
at that and wish you luck in your endeavor.
If anyone has an answer to his question, I'd appreciate hearing it,
despite any faux pas that the OP has
Vincent,
The original question can be summed up by how is general performance of
PostgreSQL on Amazon IOPS. I fail to understand why that would require me
to explain the specifics of my application and/or my market. The only one
asking for that information is you, while others have provided
On 08/23/2012 07:39 PM, Vincent Veyron wrote:
Le mercredi 22 août 2012 à 13:30 -0400, Sébastien Lorion a écrit :
Vincent, I would appreciate that you stop assuming things based on
zero information about what I am doing. I understand that you are
trying to be helpful, but I can assure you that
On Wed, Aug 22, 2012 at 4:12 PM, Alan Hodgson ahodg...@simkin.ca wrote:
On Wednesday, August 22, 2012 04:10:01 PM Andrew Hannon wrote:
Just looking into High IO instances for a DB deployment. In order to get
past 1TB, we are looking at RAID-0. I have heard
(http://hackerne.ws/item?id=4266119)
On 08/23/12 6:49 AM, Craig Ringer wrote:
In this case, what he's doing is seeking generalized performance
measurements. I don't think details were particularly necessary until
it got pulled off-track.
42
performance measurements without a very narrow definition of
'performance' are
I think both kind of tests (general and app specific) are complementary and
useful in their own way. At a minimum, if the general ones fail, why go to
the expenses of doing the specific ones ? Setting up a meaningful
application test can take a lot of time and it can be hard to pinpoint
exactly
On 08/23/12 11:24 AM, Sébastien Lorion wrote:
I think both kind of tests (general and app specific) are
complementary and useful in their own way. At a minimum, if the
general ones fail, why go to the expenses of doing the specific ones ?
Setting up a meaningful application test can take a lot
I will be setting up an instance in the coming days and post the results
here.
While reading on the subject, I found this interesting discussion on
YCombinator:
http://news.ycombinator.com/item?id=4264754
Sébastien
On Thu, Aug 23, 2012 at 2:41 PM, John R Pierce pie...@hogranch.com wrote:
On
Le mercredi 22 août 2012 à 13:15 +0800, Craig Ringer a écrit :
He appears to be suggesting that buying access to real hardware in a
datacenter (if not buying the hardware yourself) is more cost effective
and easier to manage than using cloud style services with more
transient hosts like
Vincent, I would appreciate that you stop assuming things based on zero
information about what I am doing. I understand that you are trying to be
helpful, but I can assure you that going bare-metal only does not make any
sense in my context.
Sébastien
On Wed, Aug 22, 2012 at 12:44 PM, Vincent
Just looking into High IO instances for a DB deployment. In order to get past
1TB, we are looking at RAID-0. I have heard
(http://hackerne.ws/item?id=4266119) there might be a problem if TRIM isn't
supported. Does anyone know if it is and has anyone used RAID-0 on these
instances? (Linux of
On Wednesday, August 22, 2012 04:10:01 PM Andrew Hannon wrote:
Just looking into High IO instances for a DB deployment. In order to get
past 1TB, we are looking at RAID-0. I have heard
(http://hackerne.ws/item?id=4266119) there might be a problem if TRIM isn't
supported. Does anyone know if it
Le mardi 21 août 2012 à 01:33 -0400, Sébastien Lorion a écrit :
Since Amazon has added new high I/O instance types and EBS volumes,
anyone has done some benchmark of PostgreSQL on them ?
I wonder : is there a reason why you have to go through the complexity
of such a setup, rather than
Le mardi 21 août 2012 à 01:33 -0400, Sébastien Lorion a écrit :
Since Amazon has added new high I/O instance types and EBS volumes,
anyone has done some benchmark of PostgreSQL on them ?
I wonder : is there a reason why you have to go through the complexity
of such a setup, rather than
On 21 Aug 2012, at 13:32, Vincent Veyron vv.li...@wanadoo.fr wrote:
Since Amazon has added new high I/O instance types and EBS volumes,
anyone has done some benchmark of PostgreSQL on them ?
I wonder : is there a reason why you have to go through the complexity
of such a setup, rather
On 8/21/2012 7:10 AM, Oliver Kohll - Mailing Lists wrote:
This is a general 'cloud or dedicated' question, I won't go into it
but I believe cloud proponents cite management ease, scalability etc.
I'm sure there's a place for every type of hosting. However I would be
interested in hearing
On 8/21/2012 2:18 AM, Vincent Veyron wrote:
I wonder : is there a reason why you have to go through the complexity
of such a setup, rather than simply use bare metal and get good
performance with simplicity?
In general I agree -- it is much (much!) cheaper to buy tin and deploy
yourself vs any
On Tue, Aug 21, 2012 at 12:33 AM, Sébastien Lorion
s...@thestrangefactory.com wrote:
Hello,
Since Amazon has added new high I/O instance types and EBS volumes, anyone
has done some benchmark of PostgreSQL on them ?
Le mardi 21 août 2012 à 09:36 -0500, Merlin Moncure a écrit :
here's a datapoint, stock config:
pgbench -i -s 500
pgbench -c 16 -T 60
number of transactions actually processed: 418012
tps = 6962.607292 (including connections establishing)
tps = 6973.154593 (excluding connections
Oops sorry, I thought I did hit reply all.
I am not sure this mailing list is the right place to have this debate
(assuming it is needed, there are plenty of articles stating the benefits
of using the cloud), so I will simply answer that you pay the cost of the
added layer up front (mostly
On 08/21/2012 09:40 PM, David Boreham wrote:
On 8/21/2012 2:18 AM, Vincent Veyron wrote:
I wonder : is there a reason why you have to go through the complexity
of such a setup, rather than simply use bare metal and get good
performance with simplicity?
In general I agree -- it is much (much!)
Hello,
Since Amazon has added new high I/O instance types and EBS volumes, anyone
has done some benchmark of PostgreSQL on them ?
http://perspectives.mvdirona.com/2012/07/20/IOPerformanceNoLongerSucksInTheCloud.aspx
44 matches
Mail list logo
