Re: [PERFORM] Load experimentation

On Tue, Dec 8, 2009 at 12:22 AM, Ben Brehmer wrote: > Thanks for all the responses. I have one more thought; > > Since my input data is split into about 200 files (3GB each), I could > potentially spawn one load command for each file. What would be the maximum > number of input connections Postgre

Re: [PERFORM] Load experimentation

Ben Brehmer wrote: Since my input data is split into about 200 files (3GB each), I could potentially spawn one load command for each file. What would be the maximum number of input connections Postgres can handle without bogging down? You can expect to easily get one loader process per real CP

Re: [PERFORM] Load experimentation

Thanks for all the responses. I have one more thought; Since my input data is split into about 200 files (3GB each), I could potentially spawn one load command for each file. What would be the maximum number of input connections Postgres can handle without bogging down? When I say 'input conne

Re: [PERFORM] Dynamlically updating the estimated cost of a transaction

Hasini Gunasinghe wrote: I am looking for a way to let the user know what the estimated time for the current transaction he requested and while the transaction is in progress, how much time is elapsed for the transaction as a fraction of the total estimated time at a particular instance, by

[PERFORM] error occured in dbt2 against with postgresql

*$ dbt2-run-workload -a pgsql -d 10 -w 1 -c 1 -o /home/store/tmp/testdbt2/out-1.o* postmaster starting DBT-2 test for pgsql started... DATABASE SYSTEM: localhost DATABASE NAME: dbt2 DATABASE CONNECTIONS: 1 TERMINAL THREADS: 10 TERMINALS PER WAREHOUSE: 10 WAREHOUSES PER THREAD/CLIENT PAIR: 500 SCAL

Re: [PERFORM] performance penalty between Postgresql 8.3.8 and 8.4.1

On Mon, Dec 7, 2009 at 5:19 PM, Kevin Grittner wrote: > "Schmitz, David" wrote: > >> It is carried out with poor performance on postgresql 8.4.1 >> However postgresql 8.3.8 performs just fine. >> If you take a closer look at the query with EXPLAIN, it becomes >> obvious, that postgresql 8.4 does

[PERFORM] Dynamlically updating the estimated cost of a transaction

Hi, I am looking for a way to let the user know what the estimated time for the current transaction he requested and while the transaction is in progress, how much time is elapsed for the transaction as a fraction of the total estimated time at a particular instance, by dynamically estimating the

Re: [PERFORM] performance penalty between Postgresql 8.3.8 and 8.4.1

Hi David, On Monday 07 December 2009 23:05:14 Schmitz, David wrote: > With our data it is a performance difference from 1h16min (8.3.8) to > 2h43min (8.4.1) Can you afford a explain analyze run overnight or so for both? Andres -- Sent via pgsql-performance mailing list (pgsql-performance@postg

Re: [PERFORM] performance penalty between Postgresql 8.3.8 and 8.4.1

-Ursprüngliche Nachricht- Von:Schmitz, David Gesendet: Di 08.12.2009 00:14 An: Kevin Grittner Cc: Betreff:AW: [PERFORM] performance penalty between Postgresql 8.3.8 and 8.4.1 -Ursprüngliche Nachricht- Von:Kevin Grittner [mailto:kevin.gritt...@wico

Re: [PERFORM] RAID card recommendation

Greg Smith wrote: > Craig James wrote: >> ... and do I hear you saying that no other vendor is worth >> considering? Just how far off are they? > I wasn't trying to summarize every possible possibility, just the > complicated ones there's some debate over. > > What else is OK besides Areca and 3wa

Re: [PERFORM] RAID card recommendation

Karl Denninger wrote: Most common SSDs will NOT come up on the 3ware cards at present. Not sure why as of yet - I've tried several. Right, and they're being rather weasly at http://www.3ware.com/kb/Article.aspx?id=15470 talking about it too. Not had the time to screw with them on the ARECA

Re: [PERFORM] RAID card recommendation

Craig James wrote: ... and do I hear you saying that no other vendor is worth considering? Just how far off are they? I wasn't trying to summarize every possible possibility, just the complicated ones there's some debate over. What else is OK besides Areca and 3ware? HP's P800 is good, albei

Re: [PERFORM] RAID card recommendation

Greg Smith wrote: > Scott Carey wrote: >> 9650 was made by 3Ware, essentially a PCIe version of the 9550. The >> 9690SA >> was from some sort of acquisition/merger. They are not the same >> product line >> at all. >> > 3ware became a division of AMCC, which was then bought by LSI. The > 9590SA

Re: [PERFORM] RAID card recommendation

Greg Smith wrote: Let me try to summarize where things are at a little more clearly, with the data accumulated during this long thread: -Areca: Usually the fastest around. Management tools are limited enough that you really want the version with the on-board management NIC. May require som

Re: [PERFORM] performance penalty between Postgresql 8.3.8 and 8.4.1

"Schmitz, David" wrote: > It is carried out with poor performance on postgresql 8.4.1 > However postgresql 8.3.8 performs just fine. > If you take a closer look at the query with EXPLAIN, it becomes > obvious, that postgresql 8.4 does not consider the primary key at > level 3 and instead generat

[PERFORM] performance penalty between Postgresql 8.3.8 and 8.4.1

Hello everybody, we have severe performance penalty between Postgresql 8.3.8 and 8.4.1 Consider the following tables: CREATE TABLE xdf.xdf_admin_hierarchy ( admin_place_id integer NOT NULL, admin_order smallint NOT NULL, iso_country_code character(3) NOT NULL, country_id integer NOT NULL

Re: [PERFORM] RAID card recommendation

Scott Carey wrote: 9650 was made by 3Ware, essentially a PCIe version of the 9550. The 9690SA was from some sort of acquisition/merger. They are not the same product line at all. 3ware became a division of AMCC, which was then bought by LSI. The 9590SA came out while they were a part of AMCC

Re: [PERFORM] Load experimentation

Ben Brehmer wrote: By "Loading data" I am implying: "psql -U postgres -d somedatabase -f sql_file.sql". The sql_file.sql contains table creates and insert statements. There are no indexes present nor created during the load. COPY command: Unfortunately I'm stuck with INSERTS due to the nature

Re: [PERFORM] RAID card recommendation

Scott Carey wrote: > On 12/1/09 6:49 PM, "Greg Smith" wrote: > > >> Scott Carey wrote: >> >>> 3ware 95xx and 96xx had performance somewhere between PERC 5 (horrid) and >>> PERC 6 (mediocre) when I tested them with large SATA drives with RAID 10. >>> Haven't tried raid 6 or 5. Haven't trie

Re: [PERFORM] Load experimentation

On Monday 07 December 2009, Ben Brehmer wrote: > Disk Setup: Using a single disk Amazon image for the destination > (database). Source is coming from an EBS volume. I didn't think there > were any disk options in Amazon? I don't think any Amazon cloud service is particularly well suited to a dat

Re: [PERFORM] Load experimentation

Ben Brehmer wrote: Thanks for the quick responses. I will respond to all questions in one email: By "Loading data" I am implying: "psql -U postgres -d somedatabase -f sql_file.sql". The sql_file.sql contains table creates and insert statements. There are no indexes present nor created during

Re: [PERFORM] Load experimentation

Thanks for the quick responses. I will respond to all questions in one email: By "Loading data" I am implying: "psql -U postgres -d somedatabase -f sql_file.sql". The sql_file.sql contains table creates and insert statements. There are no indexes present nor created during the load. OS: x86

Re: [PERFORM] RAID card recommendation

On 12/1/09 6:49 PM, "Greg Smith" wrote: > Scott Carey wrote: >> 3ware 95xx and 96xx had performance somewhere between PERC 5 (horrid) and >> PERC 6 (mediocre) when I tested them with large SATA drives with RAID 10. >> Haven't tried raid 6 or 5. Haven't tried the "SA" model that supports SAS >

Re: [PERFORM] RAID card recommendation

On 12/1/09 6:08 PM, "Karl Denninger" wrote: > Scott Carey wrote: >> >> On 11/24/09 11:13 AM, "Scott Marlowe" >> wrote: >> >> >> >> >>> >>> They get good reviews as well. Both manufacturers have their "star" >>> performers, and their "utility" or w

Re: [PERFORM] Load experimentation

Ben Brehmer wrote: Hello All, I'm in the process of loading a massive amount of data (500 GB). After some initial timings, I'm looking at 260 hours to load the entire 500GB. You don't say how you are loading the data, so there's not much to go on. But generally, there are two primary ways t

Re: [PERFORM] Load experimentation

Kevin, This is running on on x86_64-unknown-linux-gnu, compiled by GCC gcc (GCC) 4.1.2 20080704 (Red Hat 4.1.2-44) Ben On 07/12/2009 10:33 AM, Kevin Grittner wrote: Ben Brehmer wrote: -7.5 GB memory -4 EC2 Compute Units (2 virtual cores with 2 EC2 Compute Units each) -64-bit platf

Re: [PERFORM] Load experimentation

2009/12/7 Kevin Grittner > Ben Brehmer wrote: > > > -7.5 GB memory > > -4 EC2 Compute Units (2 virtual cores with 2 EC2 Compute Units > >each) > > -64-bit platform > > What OS? > > > (PostgreSQL 8.1.3) > > Why use such an antiquated, buggy version? Newer versions are > faster. > > -Kevin >

Re: [PERFORM] Load experimentation

On Mon, Dec 7, 2009 at 1:12 PM, Ben Brehmer wrote: > Hello All, > > I'm in the process of loading a massive amount of data (500 GB). After some > initial timings, I'm looking at 260 hours to load the entire 500GB. 10 days > seems like an awfully long time so I'm searching for ways to speed this

Re: [PERFORM] Load experimentation

Ben Brehmer wrote: > -7.5 GB memory > -4 EC2 Compute Units (2 virtual cores with 2 EC2 Compute Units >each) > -64-bit platform What OS? > (PostgreSQL 8.1.3) Why use such an antiquated, buggy version? Newer versions are faster. -Kevin -- Sent via pgsql-performance mailing list (pgs

[PERFORM] Load experimentation

Hello All, I'm in the process of loading a massive amount of data (500 GB). After some initial timings, I'm looking at 260 hours to load the entire 500GB. 10 days seems like an awfully long time so I'm searching for ways to speed this up. The load is happening in the Amazon cloud (EC2), on a