[ADMIN] Data split -- Creating a copy of database without outage

2012-05-29 Thread Igor Shmain
Can you please help with advice? I need to design a solution for a database which will grow and will require a horizontal split at some moment. Here is how I am planning to do it: Every record in every table has a shard number. After the database becomes too large (gets too many requests), the

Re: [ADMIN] Data split -- Creating a copy of database without outage

2012-05-29 Thread Haifeng Liu
Why not use a partitioned table? You can write a trigger to control which partition the coming data should be inserted. Regards, Liu Haifeng Home: http://liuhaifeng.com On May 30, 2012, at 4:13 AM, Igor Shmain wrote: > Can you please help with advice? > > I need to design a solution fo

Re: [ADMIN] Data split -- Creating a copy of database without outage

2012-05-30 Thread Igor Shmain
[mailto:liuhaif...@live.com] Sent: May-29-12 9:13 PM To: Igor Shmain Cc: pgsql-admin@postgresql.org Subject: Re: [ADMIN] Data split -- Creating a copy of database without outage Why not use a partitioned table? You can write a trigger to control which partition the coming data should be insert

Re: [ADMIN] Data split -- Creating a copy of database without outage

2012-05-30 Thread Craig James
;t be plagued by bloated indexes and files. Craig > > > ** ** > > If I missed something it would be very kind of you to point this out. > > ** ** > > Thank you once again, > > -igorS > > ** ** > > ** ** > > *From:* Haifeng Liu [m

Re: [ADMIN] Data split -- Creating a copy of database without outage

2012-05-30 Thread Amador Alvarez
-12 9:13 PM *To:* Igor Shmain *Cc:* pgsql-admin@postgresql.org *Subject:* Re: [ADMIN] Data split -- Creating a copy of database without outage Why not use a partitioned table? You can write a trigger to control which partition the coming data should be inserted. Regards, Liu Haif

Re: [ADMIN] Data split -- Creating a copy of database without outage

2012-05-30 Thread Igor Shmain
Thank you Craig and thank you Amador for your great help! Craig's ideas are very useful - I definitely want to look deeper into Londiste and to use partitioning. Amador's approach makes perfect sense to me. Best wishes, -igor

Re: [ADMIN] Data split -- Creating a copy of database without outage

2012-06-02 Thread Kevin Grittner
"Igor Shmain" wrote: > I need to design a solution for a database which will grow and will > require horizontal split at some moment. Just one more bit of "food for thought" -- we have a database with 3TB processing approximately 50 million database transactions per day (some with a great many

Re: [ADMIN] Data split -- Creating a copy of database without outage

2012-06-02 Thread Igor Shmain
-Original Message- From: Kevin Grittner [mailto:kevin.gritt...@wicourts.gov] Sent: June-02-12 11:12 AM To: igor.shm...@gmail.com; pgsql-admin@postgresql.org Subject: Re: [ADMIN] Data split -- Creating a copy of database without outage "Igor Shmain" wrote: > I need to design a solutio

Re: [ADMIN] Data split -- Creating a copy of database without outage

2012-06-02 Thread Jan Nielsen
Hi Igor, On Tue, May 29, 2012 at 2:13 PM, Igor Shmain wrote: > I need to design a solution for a database which will grow and will > require a horizontal split at some moment. > Ok > ** > > Here is how I am planning to do it: Every record in every table has a > shard number. After the databas

Re: [ADMIN] Data split -- Creating a copy of database without outage

2012-06-05 Thread Igor Shmain
M To: Igor Shmain Cc: pgsql-admin@postgresql.org Subject: Re: [ADMIN] Data split -- Creating a copy of database without outage Hi Igor, On Tue, May 29, 2012 at 2:13 PM, Igor Shmain wrote: I need to design a solution for a database which will grow and will require a horizontal split at some mome

Re: [ADMIN] Data split -- Creating a copy of database without outage

2012-06-06 Thread Kevin Grittner
"Igor Shmain" wrote: > Would it be possible for you to mention what hardware (cpu, ram, > disks, etc.) and software your system uses to support this db size > and number of transactions? We have 4 Intel Xeon X7350 @ 2.93GHz for 16 cores with 128GB RAM. We've got a pair of drives in RAID 1 fo

Re: [ADMIN] Data split -- Creating a copy of database without outage

2012-06-08 Thread Igor Shmain
Thank you, Kevin, for this detailed info. It was very helpful. Best wishes, -igor -Original Message- From: Kevin Grittner [mailto:kevin.gritt...@wicourts.gov] Sent: June-06-12 12:39 PM To: Igor Shmain; pgsql-admin@postgresql.org Subject: RE: [ADMIN] Data split -- Creating a copy of