[ADMIN] Big UPDATE breaking replication
Hello, We are using the 9.1 built-in streaming replication. Recently our slave nodes fell behind because of an UPDATE statement. It took about 3 minutes to execute, but it affected half a million records, hence the replication broke with the requested WAL segment ... has already been removed series of error messages. The WAL settings we have are: max_wal_senders = 6 wal_keep_segments = 60 max_standby_archive_delay = 300s I guess increasing the wal_keep_segments value would prevent it from happening in the future, but increase it with how much? What value would be high enough? Also we noticed some strange error message appearing shortly before and after this same statement: LOG: out of file descriptors: Too many open files; release and retry. Could it be related somehow and what does it mean exactly? Here's an excerpt from the master DB log: May 30 12:23:09 DB1 postgres[28201]: [13-1] user=www,db=xxx LOG: out of file descriptors: Too many open files; release and retry May 30 12:23:09 DB1 postgres[28201]: [13-2] user=www,db=xxx CONTEXT: writing block 0 of relation base/2819385/2820788 May 30 12:23:09 DB1 postgres[28201]: [13-3] user=www,db=xxx STATEMENT: UPDATE May 30 12:23:09 DB1 postgres[28201]: [13-4] ^I message May 30 12:23:09 DB1 postgres[28201]: [13-5] ^ISET May 30 12:23:09 DB1 postgres[28201]: [13-6] ^I sender_has_deleted=TRUE, May 30 12:23:09 DB1 postgres[28201]: [13-7] ^I receiver_has_deleted=TRUE May 30 12:23:09 DB1 postgres[28201]: [13-8] ^IWHERE from_profile_sid=870 ... May 30 12:39:47 DB1 postgres[9053]: [2-1] user=postgres,db=[unknown] FATAL: requested WAL segment 000102DE00BD has already been removed Regards, -- Kouber Saparev -- Sent via pgsql-admin mailing list (pgsql-admin@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-admin
Re: [ADMIN] Does pg_reset_stats() influence autovacuum?
On 08/24/2012 05:47 AM, Bruce Momjian wrote: On Thu, Aug 23, 2012 at 10:59:56AM +0300, Kouber Saparev wrote: Hello, Does resetting of stats through pg_reset_stats() influence the way autovacuum determines whether autovacuum_vacuum_scale_factor and autovacuum_analyze_scale_factor settings are reached? In other words, are the n_live_tup and n_dead_tup values kept somewhere internally, apart pg_stat_all_tables? No. pg_reset_stats() will affect autovacuum, usually in a bad way. So, I guess the only safe moment to reset stats is right after manual vacuum analyze. Thanks. -- Kouber Saparev -- Sent via pgsql-admin mailing list (pgsql-admin@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-admin
[ADMIN] Does pg_reset_stats() influence autovacuum?
Hello, Does resetting of stats through pg_reset_stats() influence the way autovacuum determines whether autovacuum_vacuum_scale_factor and autovacuum_analyze_scale_factor settings are reached? In other words, are the n_live_tup and n_dead_tup values kept somewhere internally, apart pg_stat_all_tables? -- Kouber Saparev -- Sent via pgsql-admin mailing list (pgsql-admin@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-admin
Re: [ADMIN] Updatable view and default sequence values
Jim C. Nasby wrote:
I think you could get away with doing a CASE or COALESCE statement, ie:
INSERT INTO subscription_purchase ... SELECT
COALESCE(currval('purchase_purchase_sid_seq'), NEW.purchase_sid)
BTW, it would be interesting to share whatever you finally come up with;
it's an interesting problem.
Yeah, as I already wrote in my first mail, that's the workaround I'm
currently using - COALESCE everywhere. However, it bothers me that I'm
repeating the same expression multiple times.
I was thinking also of writing some stored procedure in order to
determine whether NEXTVAL was already called and in case it was - to
call CURRVAL instead. Something like that:
CREATE FUNCTION nextcurrval(x_sequence regclass)
RETURNS int8 AS
$BODY$
BEGIN
RETURN CURRVAL(x_sequence);
EXCEPTION
WHEN others THEN
RETURN NEXTVAL(x_sequence);
END;
$BODY$
LANGUAGE 'plpgsql' VOLATILE;
The thing is that it works only per session and not per SQL statement,
i.e. RULE. So, in case I have two or more inserts in one session it will
not work correctly - it will always return CURRVAL.
BTW, I didn't manage to find out what's the exception error code for the
CURRVAL sequence not yet defined error - that's why I used 'others'.
Anyway, I'll write here when I find other interesting solutions.
Regards,
--
Kouber Saparev
http://kouber.saparev.com
---(end of broadcast)---
TIP 1: if posting/reading through Usenet, please send an appropriate
subscribe-nomail command to [EMAIL PROTECTED] so that your
message can get through to the mailing list cleanly
Re: [ADMIN] Updatable view and default sequence values
Jim C. Nasby wrote:
On Tue, May 16, 2006 at 04:16:55PM +0300, Kouber Saparev wrote:
The tricky part comes when I try to make my view insertable. Normally
I'd insert without specifying the sequence values, but in some cases I'd
want also to have full control of what's going into the underlying
tables. The thing is that when I try to do it the simple way by
specifying default values in the view itself:
ALTER TABLE s_purchase ALTER COLUMN purchase_sid SET DEFAULT
NEXTVAL('purchase_purchase_sid_seq');
ALTER TABLE s_purchase ALTER COLUMN subscription_purchase_sid SET
DEFAULT NEXTVAL('subscription_purchase_subscription_purchase_sid_seq');
You're doing ALTER TABLE on a view?
Exactly, AFAIK there's no ALTER VIEW command.
CREATE RULE s_purchase_insert AS
ON INSERT TO s_purchase DO INSTEAD (
INSERT INTO purchase
(purchase_sid, data)
VALUES
(NEW.purchase_sid, NEW.pdata);
INSERT INTO subscription_purchase
(subscription_purchase_sid, purchase_sid, data)
VALUES
(NEW.subscription_purchase_sid, NEW.purchase_sid, NEW.sdata);
);
Why not just use CURRVAL('purchase_purchase_sid_seq') in the rule?
Because I would like to be able to insert data both by specifying and
without specifying values for primary keys. For example:
INSERT INTO s_purchase (pdata, sdata) VALUES ('x', 'y');
INSERT INTO s_purchase (purchase_sid, pdata, subscription_purchase_sid,
sdata) VALUES (123, 'x', 456, 'y');
If I specify CURRVAL and not NEW.primary_key, as you're proposing, I
will lose the second way of adding data, cause in the latter case the
values have nothing to do with the sequences, hence CURRVAL will give me
completely useless, or even worse - wrong data. That's why I'm using
default values of a view - if there's a value provided, it will be
entered as is, if not - then the default value (nextval in this case)
will be taken.
However, this solution is not robust enough. That's why I'm looking for
other possible solutions. :)
--
Kouber Saparev
http://kouber.saparev.com
---(end of broadcast)---
TIP 4: Have you searched our list archives?
http://archives.postgresql.org
[ADMIN] Updatable view and default sequence values
Hi All,
I am currently using PostgreSQL 8.1.3 and am trying to create an
updatable view with two (or more) joined tables and I also would like to
have the ability to indicate implicitly the value of the serial primary
key fields.
I have the following two tables:
CREATE TABLE purchase (
purchase_sid SERIAL PRIMARY KEY,
data TEXT
);
CREATE TABLE subscription_purchase (
subscription_purchase_sid SERIAL PRIMARY KEY,
purchase_sid INT NOT NULL UNIQUE REFERENCES purchase ON UPDATE
CASCADE ON DELETE CASCADE,
data TEXT
);
I have also created the following view:
CREATE VIEW
s_purchase AS
SELECT
p.purchase_sid,
p.data AS pdata,
sp.subscription_purchase_sid,
sp.data AS sdata
FROM
purchase p INNER JOIN subscription_purchase sp
ON
sp.purchase_sid = p.purchase_sid;
Now, in order to make the view updatable I added this rule:
CREATE RULE s_purchase_update AS
ON UPDATE TO s_purchase DO INSTEAD (
UPDATE
purchase
SET
purchase_sid = NEW.purchase_sid,
data = NEW.pdata
WHERE
purchase_sid = OLD.purchase_sid;
UPDATE
subscription_purchase
SET
subscription_purchase_sid = NEW.subscription_purchase_sid,
purchase_sid = NEW.purchase_sid,
data = NEW.sdata
WHERE
subscription_purchase_sid = OLD.subscription_purchase_sid;
);
The tricky part comes when I try to make my view insertable. Normally
I'd insert without specifying the sequence values, but in some cases I'd
want also to have full control of what's going into the underlying
tables. The thing is that when I try to do it the simple way by
specifying default values in the view itself:
ALTER TABLE s_purchase ALTER COLUMN purchase_sid SET DEFAULT
NEXTVAL('purchase_purchase_sid_seq');
ALTER TABLE s_purchase ALTER COLUMN subscription_purchase_sid SET
DEFAULT NEXTVAL('subscription_purchase_subscription_purchase_sid_seq');
CREATE RULE s_purchase_insert AS
ON INSERT TO s_purchase DO INSTEAD (
INSERT INTO purchase
(purchase_sid, data)
VALUES
(NEW.purchase_sid, NEW.pdata);
INSERT INTO subscription_purchase
(subscription_purchase_sid, purchase_sid, data)
VALUES
(NEW.subscription_purchase_sid, NEW.purchase_sid, NEW.sdata);
);
I get foreign key constraint violation. That's happening because default
values are executed *before* the rule, so NEXTVAL for the sequence
'purchase_purchase_sid_seq' is executed twice - once for each table.
The work around is to remove the default value for this sequence and to
call it in the rule itself with coalesce:
ALTER TABLE s_purchase ALTER COLUMN purchase_sid DROP DEFAULT;
CREATE RULE s_purchase_insert AS
ON INSERT TO s_purchase DO INSTEAD (
INSERT INTO purchase
(purchase_sid, data)
VALUES
(COALESCE(NEW.purchase_sid,
NEXTVAL('purchase_purchase_sid_seq')), NEW.pdata);
INSERT INTO subscription_purchase
(subscription_purchase_sid, purchase_sid, data)
VALUES
(NEW.subscription_purchase_sid, COALESCE(NEW.purchase_sid,
CURRVAL('purchase_purchase_sid_seq')), NEW.sdata);
);
The thing is that in the real case I have multiple tables that have to
be joined so I really want to get rid of all this COALESCE stuff and to
put everything in the view definition.
Any ideas how to suppress multiple invocations of nextval() or how to do
it anyway? :)
--
Kouber Saparev
http://kouber.saparev.com
---(end of broadcast)---
TIP 3: Have you checked our extensive FAQ?
http://www.postgresql.org/docs/faq
