Re: [GENERAL] High SYS CPU - need advise

2012-11-24 Thread Merlin Moncure
On Tue, Nov 20, 2012 at 4:08 PM, Jeff Janes wrote: >> It strikes me as cavalier to be resetting >> trycounter while sitting under the #1 known contention point for read >> only workloads. > > The only use for the trycounter is to know when to ERROR out with "no > unpinned buffers available", so no

Re: [GENERAL] High SYS CPU - need advise

2012-11-24 Thread Merlin Moncure
On Wed, Nov 21, 2012 at 11:05 AM, Vlad wrote: > it's session mode and the pool size is 1200 (cause I need to grantee that in > the worst case we have enough slots for all possible clients), however even > at the times preceding high-cpu-sys-stall, the number postmasters are like > 15-20. When sta

Re: [GENERAL] High SYS CPU - need advise

2012-11-24 Thread Merlin Moncure
On Wed, Nov 21, 2012 at 10:43 AM, Jeff Janes wrote: > On Wed, Nov 21, 2012 at 7:29 AM, Vlad Marchenko wrote: > >> update on my problem: despite pgbouncer, the problem still occures on my >> end. > > As Merlin asked, how big is the pool? Maybe you are using a large > enough pool so as to defeat t

Re: [GENERAL] High SYS CPU - need advise

2012-11-24 Thread Merlin Moncure
On Tue, Nov 20, 2012 at 12:16 PM, Jeff Janes wrote: > On Tue, Nov 20, 2012 at 9:05 AM, Merlin Moncure wrote: >> On Tue, Nov 20, 2012 at 10:50 AM, Jeff Janes wrote: >>> >>> I wouldn't expect so. Increasing shared_buffers should either fix >>> free li

Re: [GENERAL] High SYS CPU - need advise

2012-11-23 Thread Merlin Moncure
On Wed, Nov 21, 2012 at 11:56 AM, Vlad wrote: > ok, understood. > I need to give some more thoughts to if it's possible for us to switch to > transaction mode from app standpoint of view. > > if yes, then setting pool size to 20 (for 8 cores) sounds OK? If it was me, I would be starting with exa

Re: [GENERAL] High SYS CPU - need advise

2012-11-23 Thread Merlin Moncure
On Wed, Nov 21, 2012 at 12:17 PM, Vlad wrote: > It turned out we can't use transaction mode, cause there are prepared > statement used a lot within code, while processing a single http request. prepare statements can be fudged within some constraints. if prepared statements are explicitly named

Re: [GENERAL] High SYS CPU - need advise

2012-11-21 Thread Merlin Moncure
On Wed, Nov 21, 2012 at 9:29 AM, Vlad Marchenko wrote: > update on my problem: despite pgbouncer, the problem still occures on my > end. > > Also, interesting observation - I ran several tests with pgbench, using > queries that I think are prone to trigger high-sys-cpu-stall. What I noticed > is w

Re: [GENERAL] Simple queries slowdown, maybe related to 3-minute long " in transaction"?

2012-11-20 Thread Merlin Moncure
On Tue, Nov 20, 2012 at 1:25 PM, Dmitry Koterov wrote: > Hello. > > Sometimes I see a strange slowdown on my PG 9.1 server: it looks like the > simplest queries which typically take 1ms or less (e.g. selection of a row > by its primary key) take 300ms or even more. It is related to all queries > w

Re: [GENERAL] High SYS CPU - need advise

2012-11-20 Thread Merlin Moncure
On Tue, Nov 20, 2012 at 10:50 AM, Jeff Janes wrote: > On Tue, Nov 20, 2012 at 8:03 AM, Merlin Moncure wrote: >> On Tue, Nov 20, 2012 at 9:02 AM, Shaun Thomas >> wrote: >>> On 11/16/2012 02:31 PM, Merlin Moncure wrote: >>> >>>> no single thing re

Re: [GENERAL] High SYS CPU - need advise

2012-11-20 Thread Merlin Moncure
On Thu, Nov 15, 2012 at 4:29 PM, Alvaro Herrera wrote: > Merlin Moncure escribió: > >> ok, excellent. reviewing the log, this immediately caught my eye: >> >> recvfrom(8, "\27\3\1\0@", 5, 0, NULL, NULL) = 5 >> recvfrom(8, >> "\327\327\nl\23

Re: [GENERAL] High SYS CPU - need advise

2012-11-20 Thread Merlin Moncure
On Tue, Nov 20, 2012 at 10:12 AM, Shaun Thomas wrote: > On 11/20/2012 10:03 AM, Merlin Moncure wrote: > >> Shared buffer manipulation changing contention is suggesting you're >> running into free list lock issues. How many active backends/cores? > > > Oh, the rea

Re: [GENERAL] High SYS CPU - need advise

2012-11-20 Thread Merlin Moncure
On Tue, Nov 20, 2012 at 9:02 AM, Shaun Thomas wrote: > On 11/16/2012 02:31 PM, Merlin Moncure wrote: > >> no single thing really stands out -- contention is all over the place. >> lwlock, pinbuffer, dynahash (especially). I am again suspicious of >> bad scheduler interac

Re: [GENERAL] High SYS CPU - need advise

2012-11-20 Thread Merlin Moncure
On Fri, Nov 16, 2012 at 2:13 PM, Vlad wrote: > ok, I've applied that patch and ran. The stall started around 13:50:45...50 > and lasted until the end > > https://dl.dropbox.com/u/109778/postgresql-2012-11-16_134904-stripped.log > > the actual log has more data (including statement following each '

Re: [GENERAL] High SYS CPU - need advise

2012-11-20 Thread Merlin Moncure
On Mon, Nov 19, 2012 at 12:02 PM, Vlad wrote: > > Some additional observation and food for thoughts. Our app uses connection > caching (Apache::DBI). By disabling Apache::DBI and forcing client > re-connection for every (http) request processed I eliminated the stall. The > user cpu usage jumped (

Re: [GENERAL] High SYS CPU - need advise

2012-11-20 Thread Merlin Moncure
On Fri, Nov 16, 2012 at 3:21 PM, Vlad wrote: > what would pgbouncer do in my case? Number of connections will decrease, but > number of active clients won't be smaller. As I understand the latter ones > are that important. Well, one thing that struck me was how little spinlock contention there ac

Re: [GENERAL] High SYS CPU - need advise

2012-11-19 Thread Merlin Moncure
On Mon, Nov 19, 2012 at 10:50 AM, Vlad wrote: > I just did a little experiment: extracted top four queries that were > executed the longest during stall times and launched pgbench test with 240 > clients. Yet I wasn't able to put the server into a stall with that. Also > load average was hitting

Re: [GENERAL] High SYS CPU - need advise

2012-11-19 Thread Merlin Moncure
On Sun, Nov 18, 2012 at 4:24 PM, Jeff Janes wrote: > On Fri, Nov 16, 2012 at 12:13 PM, Vlad wrote: >> ok, I've applied that patch and ran. The stall started around 13:50:45...50 >> and lasted until the end >> >> https://dl.dropbox.com/u/109778/postgresql-2012-11-16_134904-stripped.log > > That is

Re: [GENERAL] High SYS CPU - need advise

2012-11-19 Thread Merlin Moncure
On Fri, Nov 16, 2012 at 12:26 PM, Jeff Janes wrote: > On Fri, Nov 16, 2012 at 8:21 AM, Merlin Moncure wrote: >> On Fri, Nov 16, 2012 at 9:52 AM, Vlad wrote: >>> >>>> *) failing that, LWLOCK_STATS macro can be compiled in to give us some >>>> info

Re: [GENERAL] High SYS CPU - need advise

2012-11-19 Thread Merlin Moncure
On Fri, Nov 16, 2012 at 11:19 AM, Vlad wrote: > >> We're looking for spikes in 'blk' which represents when lwlocks bump. >> If you're not seeing any then this is suggesting a buffer pin related >> issue -- this is also supported by the fact that raising shared >> buffers didn't help. If you're n

Re: [GENERAL] Full text search ranking: ordering using index and proximiti ranking with OR queries

2012-11-16 Thread Merlin Moncure
On Wed, Mar 7, 2012 at 11:59 AM, Andrey Chursin wrote: > Is there any way to sort by ranking, avoiding seq scan? > The only way i see now is to use pg_trgm instead of ts_rank, but we > did not check yet how applicable is it for our purposes. pg_tgrm works very well in terms of measuring similarit

Re: [GENERAL] High SYS CPU - need advise

2012-11-16 Thread Merlin Moncure
On Fri, Nov 16, 2012 at 9:52 AM, Vlad wrote: > Merlin, > > >> Yeah -- you're right, this is definitely spinlock issue. Next steps: >> >> *) in mostly read workloads, we have a couple of known frequent >> offenders. In particular the 'BufFreelistLock'. One way we can >> influence that guy is to

Re: [GENERAL] High SYS CPU - need advise

2012-11-16 Thread Merlin Moncure
On Thu, Nov 15, 2012 at 6:07 PM, Jeff Janes wrote: > On Thu, Nov 15, 2012 at 2:44 PM, Merlin Moncure wrote: > >>>> select(0, NULL, NULL, NULL, {0, 1000}) = 0 (Timeout) >>>> select(0, NULL, NULL, NULL, {0, 1000}) = 0 (Timeout) >>>> select(0,

Re: [GENERAL] High SYS CPU - need advise

2012-11-15 Thread Merlin Moncure
On Thu, Nov 15, 2012 at 4:29 PM, Alvaro Herrera wrote: > Merlin Moncure escribió: > >> ok, excellent. reviewing the log, this immediately caught my eye: >> >> recvfrom(8, "\27\3\1\0@", 5, 0, NULL, NULL) = 5 >> recvfrom(8, >> "\327\327\nl\23

Re: [GENERAL] High SYS CPU - need advise

2012-11-15 Thread Merlin Moncure
On Thu, Nov 15, 2012 at 3:49 PM, Merlin Moncure wrote: > On Thu, Nov 15, 2012 at 2:44 PM, Vlad wrote: >> >>> >>> yeah. ok, nest steps: >>> *) can you confirm that postgres process is using high cpu (according >>> to top) during stall time >> &

Re: [GENERAL] High SYS CPU - need advise

2012-11-15 Thread Merlin Moncure
On Thu, Nov 15, 2012 at 2:44 PM, Vlad wrote: > >> >> yeah. ok, nest steps: >> *) can you confirm that postgres process is using high cpu (according >> to top) during stall time > > > yes, CPU is spread across a lot of postmasters > > PID USER PR NI VIRT RES SHR S %CPU %MEMTIME+ CO

Re: [GENERAL] High SYS CPU - need advise

2012-11-15 Thread Merlin Moncure
On Thu, Nov 15, 2012 at 2:20 PM, Vlad wrote: > Merlin, > > this is not my report, probably from a thread that I've referenced as having > a common symptoms. Here is info about my db: > > > Postgresql 9.1.6. > Postgres usually has 400-500 connected clients, most of them are idle. > Database is over

Re: [GENERAL] High SYS CPU - need advise

2012-11-15 Thread Merlin Moncure
On Thu, Nov 15, 2012 at 11:50 AM, Vlad wrote: > there is no big spike of queries that cause that, queries come in relatively > stable pace. It's just when the higher rate of queries coming, the more > likely this to happen. yes, when stall happens , the active queries pile up > - but that's the r

Re: [GENERAL] High SYS CPU - need advise

2012-11-15 Thread Merlin Moncure
On Wed, Nov 14, 2012 at 4:08 PM, John R Pierce wrote: > On 11/14/12 1:34 PM, Vlad wrote: >> >> thanks for your feedback. While implementing connection pooling would make >> resources utilization more efficient, I don't think it's the root of my >> problem. Most of the connected clients are at IDLE

Re: [GENERAL] Out of Shared Memory: max_locks_per_transaction

2012-11-13 Thread Merlin Moncure
On Fri, Nov 9, 2012 at 2:50 PM, Eliot Gable wrote: >>> one thing that can cause this unfortunately is advisory locks eating >>> up exactly the amount of shared memory you have. that's another thing >>> to rule out. >> >> How would I rule this out? > > It really was filling the locks table. > > Us

Re: [GENERAL] Out of Shared Memory: max_locks_per_transaction

2012-11-09 Thread Merlin Moncure
On Fri, Nov 9, 2012 at 8:43 AM, Eliot Gable wrote: > I have a PGSQL 9.0.1 database which is on the back-end of an app I was > stress testing last night. This morning, when I try to run psql, I get: > > psql: FATAL: out of shared memory > HINT: You might need to increase max_locks_per_transaction

Re: [GENERAL] Difference between varchar and text?

2012-11-05 Thread Merlin Moncure
On Mon, Nov 5, 2012 at 2:46 PM, Moshe Jacobson wrote: > Is there any practical difference between defining a column as a varchar(n) > vs. a varchar vs. a text field? not much. varchar(n) only forces the length to be <= n. I dislike inventing an 'n' when one is not known, but a lot of people do

Re: [GENERAL] Revert TRUNCATE CASCADE?

2012-10-22 Thread Merlin Moncure
On Mon, Oct 22, 2012 at 7:52 AM, Albe Laurenz wrote: > Hannes Erven wrote: >> today I ran into an issue I believed to be a FAQ, but fortunately it >> doesn't seem so as I could find any resources related to this... :-/ >> >> A misguided click in PGADMIN executed a "TRUNCATE CASCADE" on a rather >>

Re: [GENERAL] problem with distinct not distincting...

2012-10-17 Thread Merlin Moncure
On Wed, Oct 17, 2012 at 5:48 AM, John Beynon wrote: > I have a pretty basic query; > > select distinct on (name) name, length(name) from > drugs > where customer_id IS NOT NULL > order by name; > > which I'd expect to only return me a single drug name if there are > duplicates, yet I get > > name

Re: [GENERAL] Strategies/Best Practises Handling Large Tables

2012-10-16 Thread Merlin Moncure
On Fri, Oct 12, 2012 at 10:00 AM, Lonni J Friedman wrote: > On Fri, Oct 12, 2012 at 7:44 AM, Chitra Creta wrote: >> Hi, >> >> I currently have a table that is growing very quickly - i.e 7 million >> records in 5 days. This table acts as a placeholder for statistics, and >> hence the records are m

Re: [GENERAL] NOTIFY/LISTEN in Postgresql

2012-10-15 Thread Merlin Moncure
On Mon, Oct 15, 2012 at 8:44 AM, P. Broennimann wrote: > Thx for the feedback I will take a look. > > Here some details. Basically what I'd like to achieve: > > Internet <-> AppliA <-> PostgreSQL <-> AppliB > > 1) AppliA receives a request from the internet and calls a Pg/SQL function. > 2) The

Re: [GENERAL] moving from MySQL to pgsql

2012-10-13 Thread Merlin Moncure
On Sat, Oct 13, 2012 at 3:22 AM, Jasen Betts wrote: > On 2012-10-11, Vineet Deodhar wrote: > >> To give an example, I have tables for storing master records (year master, >> security master, etc.) for which pkid TINYINT is just sufficient. >> These pkid's are used as fk constraints in tables for

Re: [GENERAL] moving from MySQL to pgsql

2012-10-11 Thread Merlin Moncure
On Thu, Oct 11, 2012 at 4:44 AM, Vineet Deodhar wrote: > Thanks all for your replies. > This is my first experience with postgres mailing list. > Hats Off to the active community of pgsql. > This has definitely raised my confidence level with postgres. thanks. we like emails that start off 'movi

[GENERAL] Re: Suggestions for the best strategy to emulate returning multiple sets of results

2012-10-10 Thread Merlin Moncure
On Wed, Oct 10, 2012 at 8:27 AM, Seref Arikan wrote: > Hi Merlin, > Thanks for the response. At the moment, the main function is creating two > temp tables that drops on commit, and python functions fills these. Not too > bad, but I'd like to push these temp tables to ram, which is a bit tricky >

Re: [GENERAL] moving from MySQL to pgsql

2012-10-10 Thread Merlin Moncure
On Wed, Oct 10, 2012 at 3:47 AM, Vineet Deodhar wrote: > Hi ! > At present, I am using MySQL as backend for my work. > Because of the licensing implications, I am considering to shift from MySQL > to pgsql. > Typically, my apps are multi-user, web based or LAN based. > > 1) Read over the internet

[GENERAL] Re: Suggestions for the best strategy to emulate returning multiple sets of results

2012-10-10 Thread Merlin Moncure
On Mon, Oct 8, 2012 at 4:59 PM, Seref Arikan wrote: > Thanks Merlin, > I've tried arrays but plpython does not support returning arrays of custom > db types (which is what I'd need to do) hm -- yeah. can your custom types be broken down into plain SQL types (that is, composite types?). maybe

Re: [GENERAL] Suggestions for the best strategy to emulate returning multiple sets of results

2012-10-08 Thread Merlin Moncure
On Mon, Oct 8, 2012 at 3:14 PM, Seref Arikan wrote: > Greetings, > I have a binary blog which is passed to a plpython function by a plpgsql > function. plpython is used to create 2 different transformations of this > binary blob to sets of postgresql type instances. > The flow is: blob -> plpytho

Re: [GENERAL] Re: What's faster? BEGIN ... EXCEPTION or CREATE TEMP TABLE IF NOT EXISTS?

2012-10-05 Thread Merlin Moncure
On Fri, Oct 5, 2012 at 3:09 AM, Ivan Voras wrote: > I think I can make a fairly educated guess that catching exceptions > while dealing with session variables should be much, much faster than > creating any kind of a table :) That is true, but it's not clear how using session variables keeps you

Re: [GENERAL] [PL/PGSQL] column name substitution in PG8.4

2012-10-03 Thread Merlin Moncure
On Wed, Oct 3, 2012 at 8:15 AM, Léon Melis wrote: > For some of my customers I wrote a PL/PGSQL function that stores the > difference between an OLD en NEW record when updating a record. This system > can be applied as a trigger on the table the customer likes to audit. > Because the function can

Re: [GENERAL] Postgres will not start due to corrupt index

2012-10-03 Thread Merlin Moncure
On Wed, Oct 3, 2012 at 9:33 AM, Robert Sosinski wrote: > We are running Postgres 9.1.3, and after stopping it by physically shutting > off the machine, we rebooted and now get this error whenever we try to start > it. > > 2012-10-02 13:54:30 PDT LOG: database system was interrupted; last known up

Re: [GENERAL] Question about ip4r contrib and PostgreSQL 9.2

2012-10-02 Thread Merlin Moncure
On Thu, Sep 27, 2012 at 7:49 PM, Maxim Boguk wrote: > Hi, > > One of my project extensively uses ip4r addon ( > http://pgfoundry.org/projects/ip4r/ ). > > However, after migration of test environment to 9.2 that addon doesn't > install anymore without manual fixes. > Trouble very simple: > addon u

Re: [GENERAL] What's faster? BEGIN ... EXCEPTION or CREATE TEMP TABLE IF NOT EXISTS?

2012-10-02 Thread Merlin Moncure
On Mon, Oct 1, 2012 at 3:58 PM, Moshe Jacobson wrote: > On Mon, Oct 1, 2012 at 12:22 PM, Merlin Moncure wrote: >> >> >> *) Functions without exception blocks are faster than those with. >> >> *) Therefore, CREATE/IF NOT EXISTS is probably faster (test to be sure)

Re: [GENERAL] What's faster? BEGIN ... EXCEPTION or CREATE TEMP TABLE IF NOT EXISTS?

2012-10-01 Thread Merlin Moncure
On Mon, Oct 1, 2012 at 11:22 AM, Merlin Moncure wrote: >> We currently do use permanent tables using pg_backend_pid(). It's because of >> the connection pooling specifically that we are having problems with stale >> data. I have been unable to find a way to automatical

Re: [GENERAL] What's faster? BEGIN ... EXCEPTION or CREATE TEMP TABLE IF NOT EXISTS?

2012-10-01 Thread Merlin Moncure
On Mon, Oct 1, 2012 at 10:21 AM, Moshe Jacobson wrote: > Merlin, > > On Mon, Oct 1, 2012 at 10:28 AM, Merlin Moncure wrote: >> >> >> Couple points: >> *) Functions without exception blocks are faster than those with. > > > Clearly. > >> >&

Re: [GENERAL] What's faster? BEGIN ... EXCEPTION or CREATE TEMP TABLE IF NOT EXISTS?

2012-10-01 Thread Merlin Moncure
On Mon, Oct 1, 2012 at 8:36 AM, Moshe Jacobson wrote: > I am working on an audit logging trigger that gets called for every row > inserted, updated or deleted on any table. > For this, I need to store a couple of temporary session variables such as > the ID of the user performing the change, which

Re: [GENERAL] 9.1 vs 8.4 performance

2012-09-24 Thread Merlin Moncure
On Mon, Sep 24, 2012 at 5:47 AM, salah jubeh wrote: > Hello Guys, > > Thanks for reply, and sorry for late response. Here is more details. > > 1. Both servers are installed on parallel on the same machine, so there is > no difference in Hardware. > 2. Both servers have the same configuration sett

Re: [GENERAL] Expression to construct a anonymous record with named columns?

2012-09-21 Thread Merlin Moncure
On Fri, Sep 21, 2012 at 2:23 PM, Merlin Moncure wrote: > On Fri, Sep 21, 2012 at 12:39 PM, Benedikt Grundmann > wrote: >> On 21 September 2012 14:04, Merlin Moncure wrote: >>> >>> On Fri, Sep 21, 2012 at 4:18 AM, Benedikt Grundmann >>> wrote: >&g

Re: [GENERAL] Expression to construct a anonymous record with named columns?

2012-09-21 Thread Merlin Moncure
On Fri, Sep 21, 2012 at 12:39 PM, Benedikt Grundmann wrote: > On 21 September 2012 14:04, Merlin Moncure wrote: >> >> On Fri, Sep 21, 2012 at 4:18 AM, Benedikt Grundmann >> wrote: >> > >> > On 21 September 2012 07:50, Alban Hertroys wrote: >>

Re: [GENERAL] Expression to construct a anonymous record with named columns?

2012-09-21 Thread Merlin Moncure
On Fri, Sep 21, 2012 at 4:18 AM, Benedikt Grundmann wrote: > > On 21 September 2012 07:50, Alban Hertroys wrote: >> >> On 20 Sep 2012, at 20:36, Benedikt Grundmann wrote: >> >> > So named anonymous records / row types seem to be strangely second >> > class. Can somebody clarify the restrictions

Re: [GENERAL] Passing row set into PL/pgSQL function.

2012-09-20 Thread Merlin Moncure
On Wed, Sep 19, 2012 at 4:37 PM, Lucas Clemente Vella wrote: > I am trying to write a generic "upsert" function in PL/pgSQL, in a way > that I can specify the table were I want to insert/update, the columns > whose values I want to specify, and the values to be inserted. > > So far I have come up

Re: [GENERAL] Difference between ON and WHERE in JOINs

2012-09-19 Thread Merlin Moncure
On Tue, Sep 18, 2012 at 7:47 PM, David Johnston wrote: > On Sep 18, 2012, at 20:21, Jean-Christophe Boggio > wrote: > >> I'm looking for an article that explains the difference between these >> constructs IN POSTGRESQL (the rules seem to differ from one DB to another) : >> >> SELECT A.* >> FROM

Re: [GENERAL] Official C++ API for postgresql?

2012-09-17 Thread Merlin Moncure
On Mon, Sep 17, 2012 at 1:58 PM, niXman wrote: > Hello, > > Tell me please, whether there is an official C++ API for postgresql? Technically speaking, the only official C++ api is libpq, which, while a C API, is C++ compatible. libpxx is a set of wrappers to libpq that will give you C++ classes

Re: [GENERAL] foreign key from array element

2012-09-17 Thread Merlin Moncure
On Mon, Sep 17, 2012 at 7:24 AM, Rafal Pietrak wrote: > On Mon, 2012-09-17 at 19:58 +0800, Craig Ringer wrote: >> On 09/17/2012 04:46 PM, Rafal Pietrak wrote: > [--] >> There was some quite recent discussion on ELEMENT foreign keys on the >> -hackers list. Try searching pgsql-hackers f

Re: [GENERAL] Performance of pl/pgsql functions?

2012-09-14 Thread Merlin Moncure
On Fri, Sep 14, 2012 at 1:47 AM, Pavel Stehule wrote: > 2012/9/14 John R Pierce : >> On 09/13/12 10:17 PM, Wells Oliver wrote: >>> >>> Do these tend to perform well? I have some simple formulas in functions >>> like so: >> >> >> if you code your function in SQL instead of plpgsql, and mark it immu

Re: [GENERAL] AIX and ipv6

2012-09-11 Thread Merlin Moncure
On Tue, Sep 11, 2012 at 11:09 AM, John R Pierce wrote: > we're still having issues with PG on AIX 6.1, our configurations have no > ipv6 as our WAN is purely ipv4, but if we use listen_addresses='*', we > consistently get an error... > > LOG: could not bind IPv6 socket: Address already in use

Re: [GENERAL] Moving several databases into one database with several schemas

2012-09-07 Thread Merlin Moncure
On Thu, Sep 6, 2012 at 10:12 PM, Edson Richter wrote: > Em 06/09/2012 15:40, John R Pierce escreveu: > >> On 09/06/12 5:30 AM, Edson Richter wrote: You could change the default setting for the user with ALTER ROLE someuser SET search_path=... >>> >>> That is perfect! I can have

Re: [GENERAL] Multiple indexes, huge table

2012-09-07 Thread Merlin Moncure
On Thu, Sep 6, 2012 at 4:22 PM, Aram Fingal wrote: > I have a table which currently has about 500 million rows. For the most > part, the situation is going to be that I will import a few hundred million > more rows from text files once every few months but otherwise there won't be > any insert

Re: [GENERAL] crosstab

2012-09-04 Thread Merlin Moncure
On Tue, Sep 4, 2012 at 10:39 AM, punnoose wrote: > hi all > How could i use crostab to display variable number of columns. in the output > There could be variable number of columns > Regards > Punnoose No. The workaround I use is to write a query generator in pl/pgsql (you can also do it in the

Re: [GENERAL] Databas has no Object Identifier Types & Functions

2012-09-04 Thread Merlin Moncure
On Tue, Sep 4, 2012 at 6:57 AM, Rebecca Clarke wrote: > Returns 0 rows. how in the world did you get yourself in that situation? merlin -- Sent via pgsql-general mailing list (pgsql-general@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-general

Re: [GENERAL] String comparision in PostgreSQL

2012-08-30 Thread Merlin Moncure
On Thu, Aug 30, 2012 at 9:34 AM, Scott Marlowe wrote: > On Thu, Aug 30, 2012 at 6:09 AM, F. BROUARD / SQLpro > wrote: >> Le 30/08/2012 12:45, Craig Ringer a écrit : >> >> >>> That's my understanding, but I don't know which other database systems >>> you're talking about because you've never speci

Re: [GENERAL] Would my postgresql 8.4.12 profit from doubling RAM?

2012-08-30 Thread Merlin Moncure
On Thu, Aug 30, 2012 at 6:42 AM, Alexander Farber wrote: > Hello, > > I run CentOS 6.3 server with 16 GB RAM and: > postgresql-8.4.12-1.el6_2.x86_64 > pgbouncer-1.3.4-1.rhel6.x86_64 > > The modified params in postgresql.conf are: > max_connections = 100 > shared_buffers = 4096MB >

Re: [GENERAL] Refreshing functional index

2012-08-29 Thread Merlin Moncure
On Wed, Aug 29, 2012 at 4:01 PM, Grzegorz Tańczyk wrote: > Hello, > > I have a problem with functional index feature in Postgres 8.3 > > There are two tables, lets call them: PARENTS and CHILDREN(with timestamp > column) > > I created functional index on parents with function, which selects max va

Re: [GENERAL] Views versus user-defined functions: formatting, comments, performance, etc.

2012-08-29 Thread Merlin Moncure
On Wed, Aug 29, 2012 at 8:52 AM, Chris Travers wrote: > ALTER TABLE fruit ADD apple_id int; > ALTER TABLE fruit ADD FOREIGN KEY (apple_id, type) >REFERENCES apple (fruit_id, type) >DEFERRABLE INITIALLY DEFERRED; > > And then do the same for orange etc. you can then: > > AL

Re: [GENERAL] C locale versus en_US.UTF8. (Was: String comparision in PostgreSQL)

2012-08-29 Thread Merlin Moncure
On Wed, Aug 29, 2012 at 12:43 PM, Bruce Momjian wrote: > On Wed, Aug 29, 2012 at 10:31:21AM -0700, Aleksey Tsalolikhin wrote: >> On Wed, Aug 29, 2012 at 9:45 AM, Merlin Moncure wrote: >> > citext unfortunately doesn't allow for index optimization of LIKE >> > q

Re: [GENERAL] String comparision in PostgreSQL

2012-08-29 Thread Merlin Moncure
On Wed, Aug 29, 2012 at 10:56 AM, Nicola Cisternino wrote: > Il 29/08/2012 17.08, Merlin Moncure ha scritto: > > On Tue, Aug 28, 2012 at 9:46 AM, Nicola Cisternino > wrote: > > Hi all, > I'm valutating a complex porting of our application based on Sybase > SqlAnywher

Re: [GENERAL] String comparision in PostgreSQL

2012-08-29 Thread Merlin Moncure
On Tue, Aug 28, 2012 at 9:46 AM, Nicola Cisternino wrote: > Hi all, > I'm valutating a complex porting of our application based on Sybase > SqlAnywhere on PostgreSQL (I've love it ...) and I'd like to have your > opinion about searching/ordering funcionality. > The problem is about string comparis

Re: [GENERAL] PQfformat question and retrieving bytea data in C

2012-08-29 Thread Merlin Moncure
On Wed, Aug 29, 2012 at 8:05 AM, Dmitriy Igrishin wrote: > Hey Jason, > > 2012/8/29 Jason Armstrong >> >> I have a question regarding the return value of PQfformat() >> >> I have a 'data' column in my table, type bytea (postgresql 9.1.5). >> >> In postgresql.conf: >> bytea_output = 'escape' >> >>

Re: [GENERAL] Views versus user-defined functions: formatting, comments, performance, etc.

2012-08-29 Thread Merlin Moncure
On Tue, Aug 28, 2012 at 3:25 PM, Dmitriy Igrishin wrote: > 2012/8/20 Merlin Moncure >> >> On Sun, Aug 19, 2012 at 8:14 AM, Dmitriy Igrishin >> wrote: >> >> For various reasons, this often goes the wrong way. Views are often >> >> the right way to g

Re: [GENERAL] "Need some information about postgresql products and community"

2012-08-28 Thread Merlin Moncure
On Tue, Aug 28, 2012 at 9:29 AM, mithun wrote: > Dear Sir, > > We are gathering information regarding PostgreSQL open source product > quality and its community. Can you kindly help me to find following data. Your questions are incredibly broad, so you're going to get some broad answers. > 1.

Re: [GENERAL] Overlapping time ranges constraints in 8.4

2012-08-28 Thread Merlin Moncure
On Mon, Aug 27, 2012 at 11:46 AM, EXT-Rothermel, Peter M wrote: > I thought that this was going to be tricky. > > Perhaps I could use rules to populate a shadow table that is like a INNER > JOIN of the two tables. This would consolidate the Boolean on the separate > table into the same table tha

Re: [GENERAL] Postgres DBA in Berlin, Germany

2012-08-28 Thread Merlin Moncure
On Tue, Aug 28, 2012 at 5:34 AM, wrote: > Hi all, > > I´m not sure whether this is the appropriate place to post this but I don´t > want to keep it back from you: We´re currently looking for a postgres DBA in > our office in Berlin, Germany. > > Here is the link to the job posting: > http://ww

Re: [GENERAL] Documentation of Implicit Function Call /w Composite Types

2012-08-27 Thread Merlin Moncure
On Mon, Aug 27, 2012 at 3:46 PM, David Johnston wrote: > With Chris Travers recently going O-R gung-ho on us I decided I wanted to > find out where in the documentation the following behavior is specified: > > The expression: > > (some_composite).calculated_field > > resolves to the function call:

Re: [GENERAL] Some thoughts on table inheritance (which is uniquely awesome on PostgreSQL)

2012-08-23 Thread Merlin Moncure
On Wed, Aug 22, 2012 at 10:22 PM, Chris Travers wrote: > I have now been working with table inheritance for a while and after > starting to grapple with many of the use cases it has have become > increasingly impressed with this feature. I also think that some of > the apparent limitations fundam

Re: [GENERAL] How hard would a "path" operator be to implement in PostgreSQL

2012-08-23 Thread Merlin Moncure
On Tue, Aug 21, 2012 at 2:56 AM, Craig Ringer wrote: > On 08/21/2012 03:01 PM, Martijn van Oosterhout wrote: >> >> Well, Postgres in principle supports arrays of records, so I've >> wondered if a relationship join could stuff all the objects in a single >> field of the response using an aggregate.

Re: [GENERAL] Amazon High I/O instances

2012-08-23 Thread Merlin Moncure
On Wed, Aug 22, 2012 at 4:12 PM, Alan Hodgson wrote: > On Wednesday, August 22, 2012 04:10:01 PM Andrew Hannon wrote: >> Just looking into High IO instances for a DB deployment. In order to get >> past 1TB, we are looking at RAID-0. I have heard >> (http://hackerne.ws/item?id=4266119) there might

Re: [GENERAL] Performance implications of numeric?

2012-08-23 Thread Merlin Moncure
On Tue, Aug 21, 2012 at 11:27 PM, Wells Oliver wrote: > We have a lot of tables which store numeric data. These tables all use the > numeric type, where the values are 95% integer values. We used numeric > because it eliminated the need for casting during division to yield a > floating point value

Re: [GENERAL] At what point does a big table start becoming too big?

2012-08-23 Thread Merlin Moncure
On Wed, Aug 22, 2012 at 6:06 PM, Nick wrote: > I have a table with 40 million rows and haven't had any performance issues > yet. > > Are there any rules of thumb as to when a table starts getting too big? > > For example, maybe if the index size is 6x the amount of ram, if the table is > 10% of

Re: [GENERAL] Database Bloat

2012-08-22 Thread Merlin Moncure
On Wed, Aug 22, 2012 at 12:25 PM, elliott wrote: > Yes, it is a tif file. Uncompressed it is around 85M. ok, 85 -> 548mb is reasonable considering you have very narrow rows and an index that covers 2/3 of your column data. if you want to see dramatic reduction in table size, you probably need

Re: [GENERAL] Amazon High I/O instances

2012-08-21 Thread Merlin Moncure
On Tue, Aug 21, 2012 at 12:33 AM, Sébastien Lorion wrote: > Hello, > > Since Amazon has added new high I/O instance types and EBS volumes, anyone > has done some benchmark of PostgreSQL on them ? > > http://perspectives.mvdirona.com/2012/07/20/IOPerformanceNoLongerSucksInTheCloud.aspx > http://per

Re: [GENERAL] Database Bloat

2012-08-20 Thread Merlin Moncure
On Mon, Aug 20, 2012 at 2:33 PM, John R Pierce wrote: > On 08/20/12 11:46 AM, elliott wrote: >> >> envdb=# \d astgtm2_n60e073; >> Table "public.astgtm2_n60e073" >> Column | Type | Modifiers >> +-+--- >> lat| real| >> lon| real| >> alt| integer | >

Re: [GENERAL] function depend on view

2012-08-20 Thread Merlin Moncure
On Mon, Aug 20, 2012 at 9:59 AM, Andreas Kretschmer wrote: > salah jubeh wrote: > >> Hello Andreas, >> >> Thanks for the reply, The example I have posted is very simple and you are >> right it is very similar to select max (id) from table_that_does_not_exist; >> But >> there are more here, for

Re: [GENERAL] function depend on view

2012-08-20 Thread Merlin Moncure
On Mon, Aug 20, 2012 at 9:37 AM, salah jubeh wrote: > Hello Andreas, > > Thanks for the reply, The example I have posted is very simple and you are > right it is very similar to select max (id) from table_that_does_not_exist; > But there are more here, for example imagine I have something like >

Re: [GENERAL] Views versus user-defined functions: formatting, comments, performance, etc.

2012-08-20 Thread Merlin Moncure
On Sun, Aug 19, 2012 at 8:14 AM, Dmitriy Igrishin wrote: >> For various reasons, this often goes the wrong way. Views are often >> the right way to go. +1 on your comment above -- the right way to do >> views (and SQL in general) is to organize scripts and to try and avoid >> managing everything

Re: [GENERAL] Views versus user-defined functions: formatting, comments, performance, etc.

2012-08-17 Thread Merlin Moncure
On Fri, Aug 17, 2012 at 5:44 PM, Tom Lane wrote: > Adam Mackler writes: >> I notice when I save a view, I lose all the formatting and comments. >> As I was writing a complicated view, wanting to retain the format and >> comments, I thought I could just save it as a function that returns a >> tabl

Re: [GENERAL] Alternatives to very large tables with many performance-killing indicies?

2012-08-16 Thread Merlin Moncure
On Thu, Aug 16, 2012 at 3:54 PM, Wells Oliver wrote: > Hey folks, a question. We have a table that's getting large (6 million rows > right now, but hey, no end in sight). It's wide-ish, too, 98 columns. > > The problem is that each of these columns needs to be searchable quickly at > an applicatio

Re: [GENERAL] Visualize database schema

2012-08-14 Thread Merlin Moncure
On Tue, Aug 14, 2012 at 12:14 PM, Wolfgang Keller wrote: >> Can anyone advice about a tool to visualize a database schema? > > SQLalchemy, a Python module, can produce dot (Graphviz) output which you > can load into your favourite diagramming application such as e.g. > Omnigraffle, yEd or Dia: > >

Re: [GENERAL] json support for composites in upcoming 9.2

2012-08-13 Thread Merlin Moncure
On Sat, Aug 11, 2012 at 7:05 AM, jan zimmek wrote: > hi, > > i am looking into json support of the upcoming 9.2 release and have a > question about the row_to_json function. is there a way to specify the column > aliases of a nested composite row without creating a custom type ? > > here is an e

Re: [GENERAL] JSON in 9.2: limitations

2012-08-09 Thread Merlin Moncure
On Tue, Aug 7, 2012 at 7:26 PM, Craig Ringer wrote: > On 08/08/2012 03:45 AM, Merlin Moncure wrote: > >> Given that you can do that, if you had the ability to emit json from >> an hstore the OP's problem would be trivially handled. > > > That's where my thin

Re: [GENERAL] processing large amount of rows with plpgsql

2012-08-08 Thread Merlin Moncure
On Wed, Aug 8, 2012 at 2:41 PM, Geert Mak wrote: > hello everybody, > > we are trying to move the data from table1 into table2 using a plpgsql stored > procedure which is performing simple a data conversion > > there are about 50 million rows > > the tables are relatively simple, less than a doze

Re: [GENERAL] JSON in 9.2: limitations

2012-08-07 Thread Merlin Moncure
On Tue, Aug 7, 2012 at 11:31 AM, Merlin Moncure wrote: > On Tue, Aug 7, 2012 at 3:20 AM, Craig Ringer wrote: >> (Reposted as the list manager appears to have eaten the first copy): >> >> Hey all >> >> It seems to be surprisingly hard to build JSON structu

Re: [GENERAL] JSON in 9.2: limitations

2012-08-07 Thread Merlin Moncure
On Tue, Aug 7, 2012 at 3:20 AM, Craig Ringer wrote: > (Reposted as the list manager appears to have eaten the first copy): > > Hey all > > It seems to be surprisingly hard to build JSON structures with PostgreSQL > 9.2's json features, because: > > - There's no aggregate, function or operator that

Re: [GENERAL] Threads With Libpq

2012-08-01 Thread Merlin Moncure
On Wed, Aug 1, 2012 at 8:09 AM, Alban Hertroys wrote: > On 1 Aug 2012, at 14:32, dinesh kumar wrote: > >> Respected All, >> >> This is my first request/post in PG-Generals. If it is not the place for >> these kind of queries, then please guide me where i need to be. >> >> I have a quick question

Re: [GENERAL] Switching from OSX to Linux, multi-line queries in \copy don't work anymore

2012-07-27 Thread Merlin Moncure
On Fri, Jul 27, 2012 at 12:06 PM, Tom Lane wrote: > Ryan Kelly writes: >> I recently switched from OSX to Linux and \copy in psql no longer >> accepts multi-line queries. For instance: > >> \copy ( >> select >> * >> from >> pg_settings >> ) to '/tmp/settings.csv' with csv

Re: [GENERAL] log select access

2012-07-26 Thread Merlin Moncure
On Thu, Jul 26, 2012 at 4:32 PM, Little, Douglas wrote: > > Hello everybody, > > > > For PCI compliance I need to log user access to my PCI columns in a table > and retain for 2 years. > > I know I can grep the log, but with 1m log rows/day and infrequent PCI > access, I’m thinking this isn’t th

Re: [GENERAL] Smaller data types use same disk space

2012-07-26 Thread Merlin Moncure
On Thu, Jul 26, 2012 at 11:19 AM, Tom Lane wrote: > Merlin Moncure writes: >> On Thu, Jul 26, 2012 at 11:02 AM, Mike Christensen >> wrote: >>> I don't really think you'd need to decouple the internal column order >>> from what the user sees. A REOR

Re: [GENERAL] Smaller data types use same disk space

2012-07-26 Thread Merlin Moncure
On Thu, Jul 26, 2012 at 11:02 AM, Mike Christensen wrote: > I don't really think you'd need to decouple the internal column order > from what the user sees. A REORDER COLUMNS command should re-build > the table with the columns in the specified order. Internally, it > should be no different from

<    1   2   3   4   5   6   7   8   9   10   >