Re: [HACKERS] How to keep a table in memory?

2008-03-07 Thread Bruce Momjian
Added to TODO: * Consider allowing higher priority queries to have referenced buffer cache pages stay in memory longer http://archives.postgresql.org/pgsql-hackers/2007-11/msg00562.php --- Tom Lane wrote: Devrim

Re: [HACKERS] How to keep a table in memory?

2007-11-14 Thread Zeugswetter Andreas ADI SD
Kevin Grittner wrote: . . .the abuse of such hints in applications I have seen is so rampant as to make me doubt the utility of adding them anyway. It's true that by adding hints, you give a facility to a good, competent designer who has a really I have trouble not seeing the point of any

Re: [HACKERS] How to keep a table in memory?

2007-11-13 Thread Heikki Linnakangas
Luke Lonergan wrote: Vacuum is a better thing to run, much less CPU usage. Vacuum is actually not good for this purpose, because it's been special-cased to not bump the usage count. -- Heikki Linnakangas EnterpriseDB http://www.enterprisedb.com ---(end of

Re: [HACKERS] How to keep a table in memory?

2007-11-13 Thread Gregory Stark
Tom Lane [EMAIL PROTECTED] writes: I'd be inclined to think instead about a scheme that lets references made by higher-priority queries bump buffers' use-counts by more than 1, or some other way of making the priority considerations visible to an automatic cache management algorithm. I don't

Re: [HACKERS] How to keep a table in memory?

2007-11-13 Thread Andrew Sullivan
On Mon, Nov 12, 2007 at 06:55:09PM -0800, Joshua D. Drake wrote: Cost is always an issue, even if implicit. If the person is so hung up on the idea of pushing things into ram there is a pretty good possibility they have priced out the 50 and 100 spindle devices needed to get the same type

Re: [HACKERS] How to keep a table in memory?

2007-11-13 Thread Andrew Sullivan
On Mon, Nov 12, 2007 at 10:54:34PM -0500, Tom Lane wrote: class. But if that's your problem, pin these tables in memory is still an awfully crude solution to the problem. I'd be inclined to think instead about a scheme that lets references made by higher-priority queries bump buffers'

Re: [HACKERS] How to keep a table in memory?

2007-11-13 Thread Josh Berkus
All, I'm not sure what the solution is. This scenario is going to be a problem for any system which tries to judge future usage based on past usage. If the infrequent query with a strict response time requirement is infrequent enough any automatic algorithm will evict it. The way Greg puts

Re: [HACKERS] How to keep a table in memory?

2007-11-13 Thread Greg Smith
On Tue, 13 Nov 2007, Andrew Sullivan wrote: I have to agree with what Tom says, however, about people thinking they're smarter than the system. Much of the time, this sort of thumb on the scale optimisation just moves the cost to some other place Sure, but in this case the reasoning seems

Re: [HACKERS] How to keep a table in memory?

2007-11-13 Thread Kevin Grittner
On Tue, Nov 13, 2007 at 2:05 PM, in message [EMAIL PROTECTED], Andrew Sullivan [EMAIL PROTECTED] wrote: On Tue, Nov 13, 2007 at 02:36:14PM -0500, Greg Smith wrote: I see this as similar to the old optimizer hint argument, where there certainly exist some edge cases where people know

Re: [HACKERS] How to keep a table in memory?

2007-11-13 Thread Ron Mayer
Heikki Linnakangas wrote: Luke Lonergan wrote: Vacuum is a better thing to run, much less CPU usage. Vacuum is actually not good for this purpose, because it's been special-cased to not bump the usage count. Though the OS's page cache will still see it as accesses, no?

Re: [HACKERS] How to keep a table in memory?

2007-11-13 Thread Heikki Linnakangas
Ron Mayer wrote: Heikki Linnakangas wrote: Luke Lonergan wrote: Vacuum is a better thing to run, much less CPU usage. Vacuum is actually not good for this purpose, because it's been special-cased to not bump the usage count. Though the OS's page cache will still see it as accesses, no?

Re: [HACKERS] How to keep a table in memory?

2007-11-13 Thread Merlin Moncure
On Nov 13, 2007 12:30 AM, Christopher Browne [EMAIL PROTECTED] wrote: Something I found *really* interesting was that whenever we pushed any high traffic systems onto PostgreSQL 8.1, I kept seeing measurable performance improvements taking place every day for a week. Evidently, it took that

Re: [HACKERS] How to keep a table in memory?

2007-11-13 Thread Simon Riggs
On Tue, 2007-11-13 at 14:36 -0500, Greg Smith wrote: On Tue, 13 Nov 2007, Andrew Sullivan wrote: I have to agree with what Tom says, however, about people thinking they're smarter than the system. Much of the time, this sort of thumb on the scale optimisation just moves the cost to

[HACKERS] How to keep a table in memory?

2007-11-12 Thread adrobj
I have a pretty small table (~20MB) that is accessed very frequently and randomly, so I want to make sure it's 100% in memory all the time. There is a lot of other staff that's also gets accessed frequently, so I don't want to just hope that Linux file cache would do the right thing for me. Is

Re: [HACKERS] How to keep a table in memory?

2007-11-12 Thread Heikki Linnakangas
adrobj wrote: I have a pretty small table (~20MB) that is accessed very frequently and randomly, so I want to make sure it's 100% in memory all the time. There is a lot of other staff that's also gets accessed frequently, so I don't want to just hope that Linux file cache would do the right

Re: [HACKERS] How to keep a table in memory?

2007-11-12 Thread Alex Drobychev
Hi Heikki, Thanks for the response! I understand that relying on cache management would be the easiest solution. However, I had a similar issue with other RDBMS (MSSQL, to be specific) in the past and observed a lot of disk activity until the table was pinned in memory (fortunately

Re: [HACKERS] How to keep a table in memory?

2007-11-12 Thread Andrew Dunstan
Alex Drobychev wrote: Hi Heikki, Thanks for the response! I understand that relying on cache management would be the easiest solution. However, I had a similar issue with other RDBMS (MSSQL, to be specific) in the past and observed a lot of disk activity until the table was pinned in

Re: [HACKERS] How to keep a table in memory?

2007-11-12 Thread Robert Treat
On Monday 12 November 2007 18:31, Andrew Dunstan wrote: 1. when someone replies to your post at the bottom, please don't put your reply at the top. It makes everything totally unreadable. +1 2. you should investigate one or more of: pg_memcache, solid state disk. you might also consider

Re: [HACKERS] How to keep a table in memory?

2007-11-12 Thread Joshua D. Drake
Robert Treat wrote: On Monday 12 November 2007 18:31, Andrew Dunstan wrote: 1. when someone replies to your post at the bottom, please don't put your reply at the top. It makes everything totally unreadable. +1 2. you should investigate one or more of: pg_memcache, solid state disk. you

Re: [HACKERS] How to keep a table in memory?

2007-11-12 Thread Devrim GÜNDÜZ
Hi, On Mon, 2007-11-12 at 21:15 -0500, Robert Treat wrote: 2. you should investigate one or more of: pg_memcache, solid state disk. you might also consider creating a tablespace on tmpfs or ramfs or something like pramfs IIRC, ramfs are not that good for database use: If you want to

Re: [HACKERS] How to keep a table in memory?

2007-11-12 Thread Andrew Dunstan
Joshua D. Drake wrote: FYI, Postgres is know to be used successfully on some *extremely* heavy websites, without using tables pinned in memory. +1 I give this a +/- 1. Yes extremely heavy websites can do this *but* they require extremely expensive hardware to do so. I expect

Re: [HACKERS] How to keep a table in memory?

2007-11-12 Thread Joshua D. Drake
Devrim GÜNDÜZ wrote: Hi, On Mon, 2007-11-12 at 21:15 -0500, Robert Treat wrote: 2. you should investigate one or more of: pg_memcache, solid state disk. you might also consider creating a tablespace on tmpfs or ramfs or something like pramfs IIRC, ramfs are not that good for database use:

Re: [HACKERS] How to keep a table in memory?

2007-11-12 Thread Joshua D. Drake
Andrew Dunstan wrote: I give this a +/- 1. Yes extremely heavy websites can do this *but* they require extremely expensive hardware to do so. I expect extremely heavy websites to require extremely expensive equipment regardless of the software they use. Cost was not the issue raised by

Re: [HACKERS] How to keep a table in memory?

2007-11-12 Thread Devrim GÜNDÜZ
Hi, On Mon, 2007-11-12 at 09:12 +, Heikki Linnakangas wrote: Just leave it to the cache management algorithms in Postgres and Linux. If it really is frequently accessed, it should stay in Postgres shared buffers. How is frequently accessed determined by PostgreSQL? I mean... You know,

Re: [HACKERS] How to keep a table in memory?

2007-11-12 Thread Tom Lane
Devrim =?ISO-8859-1?Q?G=DCND=DCZ?= [EMAIL PROTECTED] writes: So, IMHO, saying trust your OS + PostgreSQL is not a 100% perfect approach for the people who are asking to keep their objects on RAM, even though I know that there is nothing we can say right now. Well, nothing is a 100% solution.

Re: [HACKERS] How to keep a table in memory?

2007-11-12 Thread Greg Smith
On Mon, 12 Nov 2007, Alex Drobychev wrote: Or any other ideas for pinning a table in memory? If the table you're worried about is only 20MB, have you considered just running something regularly that touches the whole thing? This may be the only time I've ever considered running select

Re: [HACKERS] How to keep a table in memory?

2007-11-12 Thread Luke Lonergan
Vacuum is a better thing to run, much less CPU usage. - Luke Msg is shrt cuz m on ma treo -Original Message- From: Greg Smith [mailto:[EMAIL PROTECTED] Sent: Monday, November 12, 2007 11:59 PM Eastern Standard Time To: Alex Drobychev Cc: pgsql-hackers@postgresql.org

Re: [HACKERS] How to keep a table in memory?

2007-11-12 Thread Christopher Browne
In an attempt to throw the authorities off his trail, [EMAIL PROTECTED] (Joshua D. Drake) transmitted: Andrew Dunstan wrote: I give this a +/- 1. Yes extremely heavy websites can do this *but* they require extremely expensive hardware to do so. I expect extremely heavy websites to require

Re: [HACKERS] How to keep a table in memory?

2007-11-12 Thread Christopher Browne
Quoth [EMAIL PROTECTED] (Tom Lane): Devrim =?ISO-8859-1?Q?G=DCND=DCZ?= [EMAIL PROTECTED] writes: So, IMHO, saying trust your OS + PostgreSQL is not a 100% perfect approach for the people who are asking to keep their objects on RAM, even though I know that there is nothing we can say right now.