Re: [ADMIN] Handling large volumes of data

2008-04-09 Thread Andrew Sullivan
On Tue, Apr 08, 2008 at 12:13:36PM +0200, Johann Spies wrote: > I have got 8x720G disks in a hardware raid 5 setup. It is a Dell 2950 Thow away your RAID 5. It's a loser for this. Raid 1+0 is what you need. > server. I am using an XFS-filesystem. I am not certain about the On another note, I'

Re: [ADMIN] Handling large volumes of data

2008-04-08 Thread Michael Monnerie
On Dienstag, 8. April 2008 Johann Spies wrote: > I have got 8x720G disks in a hardware raid 5 setup. It is a Dell 2950 > server. I am using an XFS-filesystem.  I am not certain about the > speed of the hard disk, but we bought the fastest we could get. 720G looks like SATA disks, probably with "on

Re: [ADMIN] Handling large volumes of data

2008-04-08 Thread Tena Sakai
othing complicated about running it) and you might find out a thing or two that is insightful. Regards, Tena Sakai [EMAIL PROTECTED] -Original Message- From: [EMAIL PROTECTED] on behalf of Johann Spies Sent: Tue 4/8/2008 2:42 AM To: pgsql-admin@postgresql.org Subject: [ADMIN] Handling la

Re: [ADMIN] Handling large volumes of data

2008-04-08 Thread Johann Spies
On Tue, Apr 08, 2008 at 11:55:00AM +0200, Tino Schwarze wrote: > The drop table will wait for the autovacuum to finish. You might want to > kill the autovacuum process (this doesn't do any harm, just aborts the > operation so the drop table may proceed). Thanks. That is good to know. Regards Joh

Re: [ADMIN] Handling large volumes of data

2008-04-08 Thread Tino Schwarze
On Tue, Apr 08, 2008 at 12:13:36PM +0200, Johann Spies wrote: > > > This took about a week on a 2xCPU quadcore server with 8Gb RAM. > > > > This is not the most interesting thing here. What disk I/O subsystem do > > you use? At least a hardware RAID controller with RAID 0 or 10 should > > be us

Re: [ADMIN] Handling large volumes of data

2008-04-08 Thread Johann Spies
On Tue, Apr 08, 2008 at 12:01:14PM +0200, Michael Monnerie wrote: > On Dienstag, 8. April 2008 Johann Spies wrote: > > This took about a week on a 2xCPU quadcore server with 8Gb RAM. > > This is not the most interesting thing here. What disk I/O subsystem do > you use? At least a hardware RAID co

Re: [ADMIN] Handling large volumes of data

2008-04-08 Thread Michael Monnerie
On Dienstag, 8. April 2008 Johann Spies wrote: > This took about a week on a 2xCPU quadcore server with 8Gb RAM. This is not the most interesting thing here. What disk I/O subsystem do you use? At least a hardware RAID controller with RAID 0 or 10 should be used, with 10krpm or 15krpm drives. SA

Re: [ADMIN] Handling large volumes of data

2008-04-08 Thread Tino Schwarze
On Tue, Apr 08, 2008 at 11:42:34AM +0200, Johann Spies wrote: > 12501 ?S 0:00 /usr/lib/postgresql/8.1/bin/postmaster -D > /var/lib/postgresql/8.1/main -c > config_file=/etc/postgresql/8.1/main/postgresql.conf > 12504 ?D 0:54 \_ postgres: writer process > 12505 ?

Re: [ADMIN] Handling large volumes of data

2008-04-08 Thread Shoaib Mir
On Tue, Apr 8, 2008 at 7:42 PM, Johann Spies <[EMAIL PROTECTED]> wrote: > Apparently the best approach is not to have very large tables. I am > thinking of making (as far as the firewall is concerned) a different > table for each day and then drop the older tables as necessary. > > Any advice on

[ADMIN] Handling large volumes of data

2008-04-08 Thread Johann Spies
I want to set up a system where logs of all kinds can be put in tables so that queries and reports can be generated from there. Our Firewall logs alone generate about 600,000,000 lines per month and that will increase as we get more bandwidth. I am testing postgresql's ability to handle large dat