Re: Bulk Insert into PostgreSQL

2018-07-05 Thread Srinivas Karthik V
Thanks for the link! Alternatively, when I am trying to create an index on a column of a table which is of size 400 GB, it is taking roughly 7 hrs. The index is created only on one column which is not a primary key. The query I am using is, create index on table (colname). I request your valuable

Re: Bulk Insert into PostgreSQL

2018-07-04 Thread Peter Geoghegan
On Tue, Jul 3, 2018 at 4:34 PM, Srinivas Karthik V wrote: > @Peter: I was indexing the primary key of all the tables in tpc-ds. Some of > the fact tables has multiple columns as part of the primary key. Also, most > of them are numeric type. Please see my mail to -hackers on suffix truncation: ht

Re: Bulk Insert into PostgreSQL

2018-07-03 Thread Ashwin Agrawal
On Sat, Jun 30, 2018 at 6:27 AM Craig Ringer wrote: > > You can also gain a bit by running with wal_level = minimal. On newer > version you can use UNLOGGED tables then convert them to logged, but that > won't be an option for 9.4. > Curious to know more on this does with standby also its faster

Re: Bulk Insert into PostgreSQL

2018-07-03 Thread Srinivas Karthik V
@Peter: I was indexing the primary key of all the tables in tpc-ds. Some of the fact tables has multiple columns as part of the primary key. Also, most of them are numeric type. On Mon, Jul 2, 2018 at 7:09 AM, Peter Geoghegan wrote: > On Sun, Jul 1, 2018 at 5:19 PM, Tsunakawa, Takayuki > wrote:

RE: Bulk Insert into PostgreSQL

2018-07-01 Thread Tsunakawa, Takayuki
From: Peter Geoghegan [mailto:p...@bowt.ie] > What kind of data was indexed? Was it a bigserial primary key, or > something else? Sorry, I don't remember it. But the table was for storing some machine usage data, and I don't think any sequence was used in the index. According to my faint memory

Re: Bulk Insert into PostgreSQL

2018-07-01 Thread Peter Geoghegan
On Sun, Jul 1, 2018 at 5:19 PM, Tsunakawa, Takayuki wrote: > 400 GB / 15 hours = 7.6 MB/s > > That looks too slow. I experienced a similar slowness. While our user tried > to INSERT (not COPY) a billion record, they reported INSERTs slowed down by > 10 times or so after inserting about 500 mil

RE: Bulk Insert into PostgreSQL

2018-07-01 Thread Tsunakawa, Takayuki
From: Srinivas Karthik V [mailto:skarthikv.i...@gmail.com] > I was using copy command to load. Removing the primary key constraint on > the table and then loading it helps a lot. In fact, a 400GB table was loaded > and the primary constraint was added in around 15 hours. Thanks for the > wonderful

Re: Bulk Insert into PostgreSQL

2018-06-30 Thread Craig Ringer
On 30 June 2018 at 06:47, Srinivas Karthik V wrote: > I was using copy command to load. Removing the primary key constraint on > the table and then loading it helps a lot. In fact, a 400GB table was > loaded and the primary constraint was added in around 15 hours. Thanks for > the wonderful sugg

Re: Bulk Insert into PostgreSQL

2018-06-29 Thread Srinivas Karthik V
I was using copy command to load. Removing the primary key constraint on the table and then loading it helps a lot. In fact, a 400GB table was loaded and the primary constraint was added in around 15 hours. Thanks for the wonderful suggestions. Regards, Srinivas Karthik On 28 Jun 2018 2:07 a.m.,

Re: Bulk Insert into PostgreSQL

2018-06-27 Thread Don Seiler
On Wed, Jun 27, 2018 at 6:25 AM, Pavel Stehule wrote: > > >> Other parameters are set to default value. Moreover, I have specified the >> primary key constraint during table creation. This is the only possible >> index being created before data loading and I am sure there are no other >> indexes

RE: Bulk Insert into PostgreSQL

2018-06-27 Thread ROS Didier
Hi I suggest to split the data to insert into several text files ( the number of CPUs) , create extension pg_background, and create a main transaction which calls x (number of CPUs) autonomous transactions. Each one insert the data from a specific test file via the

Re: Bulk Insert into PostgreSQL

2018-06-27 Thread Pavel Stehule
2018-06-27 13:18 GMT+02:00 Srinivas Karthik V : > Hi, > I am performing a bulk insert of 1TB TPC-DS benchmark data into PostgreSQL > 9.4. It's taking around two days to insert 100 GB of data. Please let me > know your suggestions to improve the performance. Below are the > configuration parameters