Re: [GENERAL] Delete Duplicates with Using

2017-10-16 Thread Igal @ Lucee.org

FYI,

On 10/16/2017 8:58 AM, Igal @ Lucee.org wrote:




test=*# with keep as (select max(ctid) as ctid from dubletten group 
by c1,c2,c3) delete from dubletten where ctid not in (select ctid 
from keep);




I like this solution, but would using a subquery be much slower than 
the implicit join of `using`?  My tables are not big in 
Postgres-standards, so it's probably not an issue, but I'm trying to 
learn as much as I can about Postgres now that I'm getting ready to 
move it to production.


I was able to compare the performance on a table with about 350k rows, 
with duplicates over 7 columns and no indices.


The GROUP BY solution with the subquery (though I was using a simpler 
version of it without a CTE), was taking over 30 seconds so I killed the 
request.


I then ran the USING version which completed in 16 seconds and deleted 
39 rows.


Best,

Igal Sapir
Lucee Core Developer
Lucee.org 



Re: [GENERAL] Delete Duplicates with Using

2017-10-16 Thread Igal @ Lucee.org

Andreas,

On 10/15/2017 11:53 PM, Andreas Kretschmer wrote:
other solution, using the CTID-column: (rows with (1,1,1) and (5,5,5) 
are identical)


test=*# select * from dubletten ;
 c1 | c2 | c3
++
  1 |  1 |  1
  1 |  1 |  1
  1 |  2 |  3
  2 |  3 |  4
  3 |  4 |  5
  4 |  5 |  5
  5 |  5 |  5
  5 |  5 |  5
(8 Zeilen)

test=*# with keep as (select max(ctid) as ctid from dubletten group by 
c1,c2,c3) delete from dubletten where ctid not in (select ctid from 
keep);;

DELETE 2
test=*# select * from dubletten ;
 c1 | c2 | c3
++
  1 |  1 |  1
  1 |  2 |  3
  2 |  3 |  4
  3 |  4 |  5
  4 |  5 |  5
  5 |  5 |  5
(6 Zeilen)

test=*#

Regards, Andreas



I like this solution, but would using a subquery be much slower than the 
implicit join of `using`?  My tables are not big in Postgres-standards, 
so it's probably not an issue, but I'm trying to learn as much as I can 
about Postgres now that I'm getting ready to move it to production.


Thanks,

Igal Sapir
Lucee Core Developer
Lucee.org 



Re: [GENERAL] Delete Duplicates with Using

2017-10-15 Thread Andreas Kretschmer



Am 14.10.2017 um 08:20 schrieb Igal @ Lucee.org:


Hello,

I run the SQL query below to delete duplicates from a table. The 
subquery is used to identify the duplicated rows (row_num is a 
BIGSERIAL column).




other solution, using the CTID-column: (rows with (1,1,1) and (5,5,5) 
are identical)


test=*# select * from dubletten ;
 c1 | c2 | c3
++
  1 |  1 |  1
  1 |  1 |  1
  1 |  2 |  3
  2 |  3 |  4
  3 |  4 |  5
  4 |  5 |  5
  5 |  5 |  5
  5 |  5 |  5
(8 Zeilen)

test=*# with keep as (select max(ctid) as ctid from dubletten group by 
c1,c2,c3) delete from dubletten where ctid not in (select ctid from keep);;

DELETE 2
test=*# select * from dubletten ;
 c1 | c2 | c3
++
  1 |  1 |  1
  1 |  2 |  3
  2 |  3 |  4
  3 |  4 |  5
  4 |  5 |  5
  5 |  5 |  5
(6 Zeilen)

test=*#

Regards, Andreas

--
2ndQuadrant - The PostgreSQL Support Company.
www.2ndQuadrant.com



--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general


Re: [GENERAL] Delete Duplicates with Using

2017-10-15 Thread Igal @ Lucee.org

On 10/14/2017 12:32 AM, legrand legrand wrote:

DELETE FROM table_with_duplicates AS T1 USING table_with_duplicates AS T2
WHERE
 T1.column_1 = T2.column_1
 AND T1.column_2 = T2.column_2
 AND T1.column_3 = T2.column_3
 AND T1.row_num < T2.row_num


Thank you, I actually thought about that at first but it seemed "too 
easy" and I was looking for some solution with JOIN.


Anyway, this seems to work great.

Thanks,


Igal


--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general


Re: [GENERAL] Delete Duplicates with Using

2017-10-14 Thread legrand legrand
DELETE FROM table_with_duplicates AS T1 USING table_with_duplicates AS T2
WHERE 
T1.column_1 = T2.column_1
AND T1.column_2 = T2.column_2
AND T1.column_3 = T2.column_3
AND T1.row_num < T2.row_num




--
Sent from: http://www.postgresql-archive.org/PostgreSQL-general-f1843780.html


-- 
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general


[GENERAL] Delete Duplicates with Using

2017-10-13 Thread Igal @ Lucee.org

Hello,

I run the SQL query below to delete duplicates from a table.  The 
subquery is used to identify the duplicated rows (row_num is a BIGSERIAL 
column).


/** delete older copies of duplicates */
DELETE FROM table_with_duplicatesAS T
WHERE row_num IN (
    SELECT     T1.row_num
    FROM    table_with_duplicates  AS T1
        JOIN table_with_duplicates AS T2
            ON         T1.column_1 = T2.column_1
                AND T1.column_2 = T2.column_2
                AND T1.column_3 = T2.column_3
                AND T1.row_num < T2.row_num
);

Can anyone tell me how to rewrite that query to use the USING clause and 
hopefully remove the subquery?


The documentation mentions USING but there is no example and the only 
examples I found online are very trivial.


Thanks,

Igal Sapir
Lucee Core Developer
Lucee.org 



Re: [GENERAL] delete a file everytime pg server starts/crashes

2017-10-08 Thread Melvin Davidson
On Sun, Oct 8, 2017 at 10:01 AM, athinivas  wrote:

> Yes, will take the epoch value from same and create a new file. Upon
> subsequent calls, will access with that filename...if it fails(incase if
> the
> postmaster is restarted), will create a new one.
>
> @Melvin Does this capture all failure scenarios?? Or you have any other
> better ways to do it. Your comments are much appreciated !!
>
>
>
> --
> Sent from: http://www.postgresql-archive.org/PostgreSQL-general-
> f1843780.html
>
>
> --
> Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
> To make changes to your subscription:
> http://www.postgresql.org/mailpref/pgsql-general
>

*athinivas,*

*This is covered in the documentation:*


*https://www.postgresql.org/docs/9.6/static/functions-info.html
*

*9.25. System Information Functions*
*pg_postmaster_start_time()*

*It does not matter what caused the failure, it is the time Postgres is
started again.*

-- 
*Melvin Davidson*
I reserve the right to fantasize.  Whether or not you
wish to share my fantasy is entirely up to you.


Re: [GENERAL] delete a file everytime pg server starts/crashes

2017-10-08 Thread athinivas
Yes, will take the epoch value from same and create a new file. Upon
subsequent calls, will access with that filename...if it fails(incase if the
postmaster is restarted), will create a new one. 

@Melvin Does this capture all failure scenarios?? Or you have any other
better ways to do it. Your comments are much appreciated !!  



--
Sent from: http://www.postgresql-archive.org/PostgreSQL-general-f1843780.html


-- 
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general


Re: [GENERAL] delete a file everytime pg server starts/crashes

2017-10-08 Thread Melvin Davidson
On Sun, Oct 8, 2017 at 8:33 AM, athinivas  wrote:

> Hi,
>
> Thank you...will try it :) ...As of now, I'm creating the filename as
> pg_start_time so that, every time the server is up, a new file will be
> created.
>
>
> Regards,
> Athi
>
>
>
> --
> Sent from: http://www.postgresql-archive.org/PostgreSQL-general-
> f1843780.html
>
>
> --
> Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
> To make changes to your subscription:
> http://www.postgresql.org/mailpref/pgsql-general
>

>... I'm creating the filename as pg_start_time...

Just an FYI, you do know that

SELECT pg_postmaster_start_time();

will return start time for postgres?

-- 
*Melvin Davidson*
I reserve the right to fantasize.  Whether or not you
wish to share my fantasy is entirely up to you.


Re: [GENERAL] delete a file everytime pg server starts/crashes

2017-10-08 Thread athinivas
Hi, 

Thank you...will try it :) ...As of now, I'm creating the filename as
pg_start_time so that, every time the server is up, a new file will be
created.


Regards,
Athi 



--
Sent from: http://www.postgresql-archive.org/PostgreSQL-general-f1843780.html


-- 
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general


Re: [GENERAL] delete a file everytime pg server starts/crashes

2017-10-06 Thread pinker
Actually if the name of the file doesn't matter you could put it into
$PGDATA/pg_stat and name it global.stat. When postgres stops (clean or
because of failure), replaces the file with his own. So your content will be
erased.
I'm not sure it's completely safe but works in simple test.



--
Sent from: http://www.postgresql-archive.org/PostgreSQL-general-f1843780.html


-- 
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general


Re: [GENERAL] delete a file everytime pg server starts/crashes

2017-10-06 Thread pinker
Look at inotify: https://github.com/rvoicilas/inotify-tools
You can check for instance if postmaster.pid exists.



--
Sent from: http://www.postgresql-archive.org/PostgreSQL-general-f1843780.html


-- 
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general


Re: [GENERAL] delete a file everytime pg server starts/crashes

2017-10-06 Thread vinny
That seems like an odd requirement and I don't think PostgreSQL can do 
it itself,
because if postgresql should crash properly then the process that should 
write/remove that file would also crash


The simplest way would be to write a cronjob that connects to the 
database and does a simple query to see if things are ok,
and act accordingly. But cronjobs cannot be executed more than once a 
miunute so there would be a considerable delay.


If you need faster responses you may need to write a custom deamon or 
use something like supervisord to manage a long polling script.


I get the feeling there must be a more elegant solution to whatever your 
problem is though...


On 2017-10-05 16:04, athinivas wrote:

Hi,

I'm having a requirement to delete a file in system whenever pg server 
is

started/crashed. Any idea?

Thanks,
Athi



--
Sent from: 
http://www.postgresql-archive.org/PostgreSQL-general-f1843780.html



--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general


Re: [GENERAL] delete a file everytime pg server starts/crashes

2017-10-05 Thread Ray Cote
>
> On Thu, Oct 5, 2017 at 10:04 AM, athinivas  wrote:
>
>> Hi,
>>
>> I'm having a requirement to delete a file in system whenever pg server is
>> started/crashed. Any idea?
>>
>> Thanks,
>> Athi
>>
>>
If you’re running on Linux you can modify the init.d (or service) file and
add a line to delete the file.
To remove on crash is likely to require some sort of ongoing monitoring
service to see that the process is no longer running.
If you’re already using a service (such as monit) which brings an
application back up after crash then adding a line to the init.d file may
be sufficient.
—Ray

--
Raymond Cote, President
voice: +1.603.924.6079 email: rgac...@appropriatesolutions.com skype:
ray.cote
Schedule a meeting: https://calendly.com/ray_cote/60min/


Re: [GENERAL] delete a file everytime pg server starts/crashes

2017-10-05 Thread Melvin Davidson
On Thu, Oct 5, 2017 at 10:04 AM, athinivas  wrote:

> Hi,
>
> I'm having a requirement to delete a file in system whenever pg server is
> started/crashed. Any idea?
>
> Thanks,
> Athi
>
>
>
> --
> Sent from: http://www.postgresql-archive.org/PostgreSQL-general-
> f1843780.html
>
>
> --
> Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
> To make changes to your subscription:
> http://www.postgresql.org/mailpref/pgsql-general
>

It would be nice to know your Operating System and PostgreSQL version().

That being said, have you looked at the documentation for
"shared_preload_libraries" ?

*https://www.postgresql.org/docs/9.6/static/runtime-config-client.html
*

-- 
*Melvin Davidson*
I reserve the right to fantasize.  Whether or not you
wish to share my fantasy is entirely up to you.


[GENERAL] delete a file everytime pg server starts/crashes

2017-10-05 Thread athinivas
Hi, 

I'm having a requirement to delete a file in system whenever pg server is
started/crashed. Any idea?

Thanks,
Athi 



--
Sent from: http://www.postgresql-archive.org/PostgreSQL-general-f1843780.html


-- 
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general


Re: [GENERAL] DELETE and JOIN

2017-03-14 Thread Alexander Farber
Good morning and thank you for the replies.

I've ended up with the following DELETE USING (in order to delete reviews
coming from different user id, but same IP address in the last 24 hours):

DELETE  FROM words_reviews r
USING   words_users u
WHERE   r.uid = u.uid
AND r.uid = in_uid
AND AGE(r.updated) < INTERVAL '1 day'
AND u.ip = (SELECT ip FROM words_users WHERE uid = in_author);

Regards
Alex

PS: Here is my custom function:

CREATE OR REPLACE FUNCTION words_review_user(
in_uid integer,  /* the player in_uid... */
in_author integer, /* ... is reviewed by player in_author */
in_nice integer,
in_review varchar
) RETURNS void AS
$func$
BEGIN
DELETE  FROM words_reviews r
USING   words_users u
WHERE   r.uid = u.uid
AND r.uid = in_uid
AND AGE(r.updated) < INTERVAL '1 day'
AND u.ip = (SELECT ip FROM words_users WHERE uid = in_author);

UPDATE words_reviews SET
author= in_author,
nice  = in_nice,
review= in_review,
updated   = CURRENT_TIMESTAMP
WHERE uid = in_uid AND author = in_author;

IF NOT FOUND THEN
INSERT INTO words_reviews (
uid,
author,
nice,
review,
updated
) VALUES (
in_uid,
in_author,
in_nice,
in_review,
CURRENT_TIMESTAMP
);
END IF;
END
$func$ LANGUAGE plpgsql;

And here are the tables in question:

CREATE TABLE words_reviews (
uid integer NOT NULL CHECK (uid <> author) REFERENCES words_users
ON DELETE CASCADE,
author integer NOT NULL REFERENCES words_users(uid) ON DELETE
CASCADE,
nice integer NOT NULL CHECK (nice = 0 OR nice = 1),
review varchar(255),
updated timestamptz NOT NULL,
PRIMARY KEY(uid, author)
);

CREATE TABLE words_users (
uid SERIAL PRIMARY KEY,
ip inet NOT NULL,
..
);


Re: [GENERAL] DELETE and JOIN

2017-03-13 Thread David G. Johnston
On Mon, Mar 13, 2017 at 9:39 AM, Alexander Farber <
alexander.far...@gmail.com> wrote:

> Good evening,
>
> In a 9.5 database I would like players to rate each other and save the
> reviews in the table:
>
> CREATE TABLE words_reviews (
> uid integer NOT NULL CHECK (uid <> author) REFERENCES words_users
> ON DELETE CASCADE,
> author integer NOT NULL REFERENCES words_users(uid) ON DELETE
> CASCADE,
> nice integer NOT NULL CHECK (nice = 0 OR nice = 1),
> review varchar(255),
> updated timestamptz NOT NULL,
> PRIMARY KEY(uid, author)
> );
>
> while user names and IP addresses are saved in the other database:
>
> CREATE TABLE words_users (
> uid SERIAL PRIMARY KEY,
> ip inet NOT NULL,
> ..
> );​
>


> ​[...]
>  all previous reviews coming from the same IP in the past 24 hours:
>

​SELECT (uid, author)  -- locate reviews
FROM word_reviews
JOIN words_users USING (u_id)
WHERE u_id IN ( -- from each of the following users...
SELECT wu.u_id
FROM words_users wu
WHERE wu.ip = (SELECT wui.ip FROM words_users wui WHERE wui,uid = in_uid)
-- find all users sharing the ip address of this supplied user
)​
AND updated >= [...]  -- but only within the specified time period

David J.


Re: [GENERAL] DELETE and JOIN

2017-03-13 Thread Adrian Klaver

On 03/13/2017 09:39 AM, Alexander Farber wrote:

Good evening,

In a 9.5 database I would like players to rate each other and save the
reviews in the table:

CREATE TABLE words_reviews (
uid integer NOT NULL CHECK (uid <> author) REFERENCES
words_users ON DELETE CASCADE,
author integer NOT NULL REFERENCES words_users(uid) ON DELETE
CASCADE,
nice integer NOT NULL CHECK (nice = 0 OR nice = 1),
review varchar(255),
updated timestamptz NOT NULL,
PRIMARY KEY(uid, author)
);

while user names and IP addresses are saved in the other database:

CREATE TABLE words_users (
uid SERIAL PRIMARY KEY,
ip inet NOT NULL,
..
);

However, before saving a review, I would like to delete all previous
reviews coming from the same IP in the past 24 hours:

CREATE OR REPLACE FUNCTION words_review_user(
in_uid integer,/* this user is being rated */
in_author integer,   /* by the in_author user */
in_nice integer,
in_review varchar
) RETURNS void AS
$func$
DECLARE
_author_rep integer;
_author_ip integer;
BEGIN

/* find the current IP address of the author */

SELECT  ip
INTO_author_ip
FROMwords_users
WHERE   uid = in_author;

/* try to prevent review fraud - how to improve this query
please? */

DELETE  FROM words_reviews
WHERE   uid = in_uid
AND AGE(updated) < INTERVAL '1 day'
AND EXISTS (
SELECT 1
FROM words_reviews r INNER JOIN words_users u USING(uid)
WHERE u.ip = u._author_ip
AND r.author = in_author
);

UPDATE words_reviews set
author= in_author,
nice  = in_nice,
review= in_review,
updated   = CURRENT_TIMESTAMP
WHERE uid = in_uid AND author = in_author;

IF NOT FOUND THEN
INSERT INTO words_reviews (
author,
nice,
review,
updated
) VALUES (
in_author,
in_nice,
in_review,
CURRENT_TIMESTAMP
);
END IF;

END
$func$ LANGUAGE plpgsql;

I have the feeling that the _author_ip variable is not really necessary
and I could use some kind of "DELETE JOIN" here, but can not figure it out.


The USING clause?:

https://www.postgresql.org/docs/9.5/static/sql-delete.html

"PostgreSQL lets you reference columns of other tables in the WHERE 
condition by specifying the other tables in the USING clause. For 
example, to delete all films produced by a given producer, one can do:


DELETE FROM films USING producers
  WHERE producer_id = producers.id AND producers.name = 'foo';
"



Please advise a better query if possible

Best regards
Alex



--
Adrian Klaver
adrian.kla...@aklaver.com


--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general


Re: [GENERAL] DELETE and JOIN

2017-03-13 Thread Tom Lane
Alexander Farber  writes:
> ...
> However, before saving a review, I would like to delete all previous
> reviews coming from the same IP in the past 24 hours:
> ...
> I have the feeling that the _author_ip variable is not really necessary and
> I could use some kind of "DELETE JOIN" here, but can not figure it out.

Sure, see the USING clause in DELETE.  Although your example seems a
bit confused, since you're not actually referring to _author_ip anywhere.
And if you meant "_author_ip" where you wrote "u._author_ip", that's in
a sub-SELECT, where you could just add a join to words_users without
needing any nonstandard DELETE syntax.

regards, tom lane


-- 
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general


[GENERAL] DELETE and JOIN

2017-03-13 Thread Alexander Farber
Good evening,

In a 9.5 database I would like players to rate each other and save the
reviews in the table:

CREATE TABLE words_reviews (
uid integer NOT NULL CHECK (uid <> author) REFERENCES words_users
ON DELETE CASCADE,
author integer NOT NULL REFERENCES words_users(uid) ON DELETE
CASCADE,
nice integer NOT NULL CHECK (nice = 0 OR nice = 1),
review varchar(255),
updated timestamptz NOT NULL,
PRIMARY KEY(uid, author)
);

while user names and IP addresses are saved in the other database:

CREATE TABLE words_users (
uid SERIAL PRIMARY KEY,
ip inet NOT NULL,
..
);

However, before saving a review, I would like to delete all previous
reviews coming from the same IP in the past 24 hours:

CREATE OR REPLACE FUNCTION words_review_user(
in_uid integer,/* this user is being rated */
in_author integer,   /* by the in_author user */
in_nice integer,
in_review varchar
) RETURNS void AS
$func$
DECLARE
_author_rep integer;
_author_ip integer;
BEGIN

/* find the current IP address of the author */

SELECT  ip
INTO_author_ip
FROMwords_users
WHERE   uid = in_author;

/* try to prevent review fraud - how to improve this query please?
*/

DELETE  FROM words_reviews
WHERE   uid = in_uid
AND AGE(updated) < INTERVAL '1 day'
AND EXISTS (
SELECT 1
FROM words_reviews r INNER JOIN words_users u USING(uid)
WHERE u.ip = u._author_ip
AND r.author = in_author
);

UPDATE words_reviews set
author= in_author,
nice  = in_nice,
review= in_review,
updated   = CURRENT_TIMESTAMP
WHERE uid = in_uid AND author = in_author;

IF NOT FOUND THEN
INSERT INTO words_reviews (
author,
nice,
review,
updated
) VALUES (
in_author,
in_nice,
in_review,
CURRENT_TIMESTAMP
);
END IF;

END
$func$ LANGUAGE plpgsql;

I have the feeling that the _author_ip variable is not really necessary and
I could use some kind of "DELETE JOIN" here, but can not figure it out.

Please advise a better query if possible

Best regards
Alex


Re: [GENERAL] Delete from table conditionally

2016-12-16 Thread Charles Clavadetscher
Hello

> -Original Message-
> From: pgsql-general-ow...@postgresql.org 
> [mailto:pgsql-general-ow...@postgresql.org] On Behalf Of Arup Rakshit
> Sent: Samstag, 17. Dezember 2016 08:39
> To: Pgsql-general 
> Subject: [GENERAL] Delete from table conditionally
> 
> Hi,
> 
> I am using Postgresql 9.5.
> 
> Below is the sample data taken from the mail table with only required columns:
> 
> id | question_id | answer_id | content
> +-+---+--
> 2 |  25 |   270 | Arup
> 3 |  26 |   276 | Kajal
> 4 |  26 |   276 | Arup
> 5 |  27 |   280 | Kajal
> 6 |  25 |   270 | Arup
> 7 |  25 |   270 | Lalu
> 8 |  25 |   270 | Arup
> 9 |  26 |   274 |  Dilip
> 10 |  25 |   270 | doli
> 11 |  25 |   270 | Arup
> (10 rows)
> 
> 
> Say, I know the id 6 where content is "Arup". Now from this row, I want to 
> delete all next rows where the content is
> "Arup". How should I achieve this?

Assuming that "next" means id > 6:

DELETE FROM 
WHERE id > 6
AND content = 'Arup';

Regards
Charles

> 
> 
> Regards,
> Arup Rakshit
> 
> 
> --
> Sent via pgsql-general mailing list (pgsql-general@postgresql.org) To make 
> changes to your subscription:
> http://www.postgresql.org/mailpref/pgsql-general



-- 
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general


[GENERAL] Delete from table conditionally

2016-12-16 Thread Arup Rakshit
Hi,

I am using Postgresql 9.5.

Below is the sample data taken from the mail table with only required columns:

id | question_id | answer_id | content
+-+---+--
2 |  25 |   270 | Arup
3 |  26 |   276 | Kajal
4 |  26 |   276 | Arup
5 |  27 |   280 | Kajal
6 |  25 |   270 | Arup
7 |  25 |   270 | Lalu
8 |  25 |   270 | Arup
9 |  26 |   274 |  Dilip
10 |  25 |   270 | doli
11 |  25 |   270 | Arup
(10 rows)


Say, I know the id 6 where content is "Arup". Now from this row, I want to 
delete all next rows where the content is "Arup". How should I achieve this?


Regards,
Arup Rakshit


-- 
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general


Re: [GENERAL] Delete trigger

2015-09-18 Thread Leif Jensen
   Hi Ioana and David.

   Thank you. Yes, I can see the problem. I will look into your suggestions.

 Leif


- Original Message -
> On Friday, September 18, 2015, Leif Jensen  wrote:
> 
> >Hello Laurenz,
> >
> >Thank you for you suggestion. I really want to aviod that someone
> > 'accidentally' deletes too much by typing (programming) a not full
> > qualified DELETE ... statement. In your case one would have to always use
> > the delete function, but no restrictions on using the DELETE statement.
> >
> >
> There is no way you can prevent a superuser from shooting themselves in the
> foot.  For anyone else you can enforce use of the function to perform the
> delete.
> 
> You could make a field called ok-to-delete and add a partial unique index
> on it so that only a single record can be marked ok to delete at a time and
> then have your trigger abort if it tries to delete a field without the ok
> to delete field set to true.
> 
> David J.
> 


-- 
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general


Re: [GENERAL] Delete trigger

2015-09-18 Thread David G. Johnston
On Friday, September 18, 2015, Leif Jensen  wrote:

>Hello Laurenz,
>
>Thank you for you suggestion. I really want to aviod that someone
> 'accidentally' deletes too much by typing (programming) a not full
> qualified DELETE ... statement. In your case one would have to always use
> the delete function, but no restrictions on using the DELETE statement.
>
>
There is no way you can prevent a superuser from shooting themselves in the
foot.  For anyone else you can enforce use of the function to perform the
delete.

You could make a field called ok-to-delete and add a partial unique index
on it so that only a single record can be marked ok to delete at a time and
then have your trigger abort if it tries to delete a field without the ok
to delete field set to true.

David J.


Re: [GENERAL] Delete trigger

2015-09-18 Thread Ioana Danes
It depends on the size of the table and the frequency of updates, deletes
but cold consider an audit table with triggers for update, delete and
truncate. At least you have a way to recover deleted records.

Ioana


On Fri, Sep 18, 2015 at 5:52 AM, Leif Jensen  wrote:

>Hello Laurenz,
>
>Thank you for you suggestion. I really want to aviod that someone
> 'accidentally' deletes too much by typing (programming) a not full
> qualified DELETE ... statement. In your case one would have to always use
> the delete function, but no restrictions on using the DELETE statement.
>
>  Leif
>
>
> - Original Message -
> > Leif Jensen wrote:
> > >If I do "DELETE FROM devicegroup WHERE group=1" I do not want to
> delete
> > >anything. I only want to
> > > delete if I do "DELETE FROM devicegroup WHERE groupid=x AND ctrlid=y
> AND
> > > userid=z". I don't wanna let
> > > anyone delete more than 1 row at a time.
> >
> > I can't think of a way to do that with a trigger.
> >
> > I'd write a
> >   FUNCTION delete_devicegroup(groupid integer, ctrlid integer, userid
> >   integer)
> > RETURNS void CALLED ON NULL INPUT VOLATILE SECURITY DEFINER
> > that enables the user to delete a row and checks that all arguments
> > are NOT NULL.  The user doesn't get privileges to DELETE from the table
> > directly.
> >
> > Yours,
> > Laurenz Albe
> >
> >
>
>
> --
> Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
> To make changes to your subscription:
> http://www.postgresql.org/mailpref/pgsql-general
>


Re: [GENERAL] Delete trigger

2015-09-18 Thread Leif Jensen
   Hello Laurenz,

   Thank you for you suggestion. I really want to aviod that someone 
'accidentally' deletes too much by typing (programming) a not full qualified 
DELETE ... statement. In your case one would have to always use the delete 
function, but no restrictions on using the DELETE statement.

 Leif


- Original Message -
> Leif Jensen wrote:
> >If I do "DELETE FROM devicegroup WHERE group=1" I do not want to delete
> >anything. I only want to
> > delete if I do "DELETE FROM devicegroup WHERE groupid=x AND ctrlid=y AND
> > userid=z". I don't wanna let
> > anyone delete more than 1 row at a time.
> 
> I can't think of a way to do that with a trigger.
> 
> I'd write a
>   FUNCTION delete_devicegroup(groupid integer, ctrlid integer, userid
>   integer)
> RETURNS void CALLED ON NULL INPUT VOLATILE SECURITY DEFINER
> that enables the user to delete a row and checks that all arguments
> are NOT NULL.  The user doesn't get privileges to DELETE from the table
> directly.
> 
> Yours,
> Laurenz Albe
> 
> 


-- 
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general


Re: [GENERAL] Delete trigger

2015-09-18 Thread Albe Laurenz
Leif Jensen wrote:
>If I do "DELETE FROM devicegroup WHERE group=1" I do not want to delete 
> anything. I only want to
> delete if I do "DELETE FROM devicegroup WHERE groupid=x AND ctrlid=y AND 
> userid=z". I don't wanna let
> anyone delete more than 1 row at a time.

I can't think of a way to do that with a trigger.

I'd write a
  FUNCTION delete_devicegroup(groupid integer, ctrlid integer, userid integer)
RETURNS void CALLED ON NULL INPUT VOLATILE SECURITY DEFINER
that enables the user to delete a row and checks that all arguments
are NOT NULL.  The user doesn't get privileges to DELETE from the table 
directly.

Yours,
Laurenz Albe


-- 
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general


Re: [GENERAL] Delete trigger

2015-09-18 Thread Leif Jensen
   Hi Charles,

   If I do "DELETE FROM devicegroup WHERE group=1" I do not want to delete 
anything. I only want to delete if I do "DELETE FROM devicegroup WHERE 
groupid=x AND ctrlid=y AND userid=z". I don't wanna let anyone delete more than 
1 row at a time.

 Leif


- Original Message -
> Hello
> 
> Not sure I get it right, but all three fields are not nullable. So they will
> always have a value, which is what I understand of "are specified".
> What do you need the trigger for in that case?
> 
> Bye
> Charles
> 
> > -Original Message-
> > From: pgsql-general-ow...@postgresql.org
> > [mailto:pgsql-general-ow...@postgresql.org] On Behalf Of Leif Jensen
> > Sent: Freitag, 18. September 2015 10:23
> > To: pgsql-general 
> > Subject: [GENERAL] Delete trigger
> > 
> >Hi,
> > 
> >I am running PostgreSQL 9.3.4. I have a simple table with 3 fields:
> > 
> > CREATE TABLE devicegroup (
> > groupid integer NOT NULL,
> > ctrlid integer NOT NULL,
> > userid integer NOT NULL
> > );
> > ALTER TABLE ONLY devicegroup
> > ADD CONSTRAINT pk_devicegroup PRIMARY KEY (groupid, ctrlid, userid);
> > 
> >I want to make sure that records are only deleted when all 3 fields are
> >specified, so I tried make a trigger:
> > 
> > CREATE TRIGGER userDev_trig INSTEAD OF DELETE ON admdevgrid FOR EACH
> > STATEMENT
> >   EXECUTE PROCEDURE deleteUserDev();
> > 
> > which could check for NOT NULL on the 3 fields before actual doing the
> > delete. Unfortunately this is not possible to
> > do FOR EACH STATEMENT, and FOR EACH ROW would not give me the chance to
> > check for NOT NULL.
> > 
> >Any ideas ?
> > 
> >  Leif
> > 
> > 
> > --
> > Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
> > To make changes to your subscription:
> > http://www.postgresql.org/mailpref/pgsql-general
> 
> 


-- 
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general


Re: [GENERAL] Delete trigger

2015-09-18 Thread Charles Clavadetscher
Hello

Not sure I get it right, but all three fields are not nullable. So they will 
always have a value, which is what I understand of "are specified".
What do you need the trigger for in that case?

Bye
Charles

> -Original Message-
> From: pgsql-general-ow...@postgresql.org 
> [mailto:pgsql-general-ow...@postgresql.org] On Behalf Of Leif Jensen
> Sent: Freitag, 18. September 2015 10:23
> To: pgsql-general 
> Subject: [GENERAL] Delete trigger
> 
>Hi,
> 
>I am running PostgreSQL 9.3.4. I have a simple table with 3 fields:
> 
> CREATE TABLE devicegroup (
> groupid integer NOT NULL,
> ctrlid integer NOT NULL,
> userid integer NOT NULL
> );
> ALTER TABLE ONLY devicegroup
> ADD CONSTRAINT pk_devicegroup PRIMARY KEY (groupid, ctrlid, userid);
> 
>I want to make sure that records are only deleted when all 3 fields are 
> specified, so I tried make a trigger:
> 
> CREATE TRIGGER userDev_trig INSTEAD OF DELETE ON admdevgrid FOR EACH STATEMENT
> EXECUTE PROCEDURE deleteUserDev();
> 
> which could check for NOT NULL on the 3 fields before actual doing the 
> delete. Unfortunately this is not possible to
> do FOR EACH STATEMENT, and FOR EACH ROW would not give me the chance to check 
> for NOT NULL.
> 
>Any ideas ?
> 
>  Leif
> 
> 
> --
> Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
> To make changes to your subscription:
> http://www.postgresql.org/mailpref/pgsql-general



-- 
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general


[GENERAL] Delete trigger

2015-09-18 Thread Leif Jensen
   Hi,

   I am running PostgreSQL 9.3.4. I have a simple table with 3 fields:

CREATE TABLE devicegroup (
groupid integer NOT NULL,
ctrlid integer NOT NULL,
userid integer NOT NULL
);
ALTER TABLE ONLY devicegroup
ADD CONSTRAINT pk_devicegroup PRIMARY KEY (groupid, ctrlid, userid);

   I want to make sure that records are only deleted when all 3 fields are 
specified, so I tried make a trigger:

CREATE TRIGGER userDev_trig INSTEAD OF DELETE ON admdevgrid FOR EACH STATEMENT
  EXECUTE PROCEDURE deleteUserDev();

which could check for NOT NULL on the 3 fields before actual doing the delete. 
Unfortunately this is not possible to do FOR EACH STATEMENT, and FOR EACH ROW 
would not give me the chance to check for NOT NULL.

   Any ideas ?

 Leif


-- 
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general


Re: [GENERAL] Delete rule does not prevent truncate

2015-07-27 Thread Melvin Davidson
For crying out loud GET OVER IT! You've been given a very reasonable and
quick solution to your problem.
You can either
1. Keep crying and moaning until someone changes the rules.
2. Give up and port to another database.
3. Write the triggers and solve your problem!

On Fri, Jul 24, 2015 at 5:27 AM, Tim Smith  wrote:

> On 23 July 2015 at 19:25, Scott Marlowe  wrote:
> > stick to triggers, they're faster
>
>
> Erm, not according to your beloved manual !!!
>
> 38.7. Rules Versus Triggers
>
> " a rule issuing one extra command is likely to be faster than a trigger"
> "The summary is, rules will only be significantly slower than triggers
> if their actions result in large and badly qualified joins, a
> situation where the planner fails."
>
>
> --
> Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
> To make changes to your subscription:
> http://www.postgresql.org/mailpref/pgsql-general
>



-- 
*Melvin Davidson*
I reserve the right to fantasize.  Whether or not you
wish to share my fantasy is entirely up to you.


Re: [GENERAL] Delete rule does not prevent truncate

2015-07-27 Thread Tim Smith
>
> Just in case it has not been made obvious yet, rules are silently
> deprecated. They still exist because views depend on them, but it is
> generally considered best practices to not use them outside that realm.


Well, the manual doesn't seem to reflect that fact.

If that's how the developers feel about rules, i.e they can't be
bothered to make any changes to the rules code any more, no matter how
minor (e.g. this TRUNCATE issue), then you should explicitly state in
the manual that they are depreciated, and stop making stupid
statements like "so if many rows are affected in one statement, a rule
issuing one extra command is likely to be faster than a trigger" that
encourage people to use rules !


-- 
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general


Re: [GENERAL] Delete rule does not prevent truncate

2015-07-27 Thread Tim Smith
On 23 July 2015 at 19:25, Scott Marlowe  wrote:
> stick to triggers, they're faster


Erm, not according to your beloved manual !!!

38.7. Rules Versus Triggers

" a rule issuing one extra command is likely to be faster than a trigger"
"The summary is, rules will only be significantly slower than triggers
if their actions result in large and badly qualified joins, a
situation where the planner fails."


-- 
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general


Re: [GENERAL] Delete rule does not prevent truncate

2015-07-24 Thread Joshua D. Drake


On 07/24/2015 02:32 AM, Andres Freund wrote:


On 2015-07-24 10:29:21 +0100, Tim Smith wrote:

That's not the point.  Backups are important, but so is the concept of
various layers of anti-fat-finger protection.   Restoring off backups
should be last resort, not first.


Oh, comeon. Install a TRUNCATE trigger and let this thread die.

Andres



Please god, +1.

JD


--
Command Prompt, Inc. - http://www.commandprompt.com/  503-667-4564
PostgreSQL Centered full stack support, consulting and development.
Announcing "I'm offended" is basically telling the world you can't
control your own emotions, so everyone else should do it for you.


--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general


Re: [GENERAL] Delete rule does not prevent truncate

2015-07-24 Thread Scott Marlowe
On Fri, Jul 24, 2015 at 3:27 AM, Tim Smith  wrote:
> On 23 July 2015 at 19:25, Scott Marlowe  wrote:
>> stick to triggers, they're faster
>
>
> Erm, not according to your beloved manual !!!
>
> 38.7. Rules Versus Triggers
>
> " a rule issuing one extra command is likely to be faster than a trigger"
> "The summary is, rules will only be significantly slower than triggers
> if their actions result in large and badly qualified joins, a
> situation where the planner fails."

So, that's the sum total of what you took away from my post?
Nevermind, I'll leave you alone.


-- 
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general


Re: [GENERAL] Delete rule does not prevent truncate

2015-07-24 Thread Adrian Klaver

On 07/24/2015 02:24 AM, Tim Smith wrote:


Just in case it has not been made obvious yet, rules are silently
deprecated. They still exist because views depend on them, but it is
generally considered best practices to not use them outside that realm.



Well, the manual doesn't seem to reflect that fact.

If that's how the developers feel about rules, i.e they can't be
bothered to make any changes to the rules code any more, no matter how
minor (e.g. this TRUNCATE issue), then you should explicitly state in
the manual that they are depreciated, and stop making stupid
statements like "so if many rows are affected in one statement, a rule
issuing one extra command is likely to be faster than a trigger" that
encourage people to use rules !



Seems that section needs editing to make it clearer that this depends on 
what type of trigger you use. Or as it states here:


http://www.postgresql.org/docs/9.4/interactive/sql-createtrigger.html
"
A trigger that is marked FOR EACH ROW is called once for every row that 
the operation modifies. For example, a DELETE that affects 10 rows will 
cause any ON DELETE triggers on the target relation to be called 10 
separate times, once for each deleted row. In contrast, a trigger that 
is marked FOR EACH STATEMENT only executes once for any given operation, 
regardless of how many rows it modifies (in particular, an operation 
that modifies zero rows will still result in the execution of any 
applicable FOR EACH STATEMENT triggers).

"

At any rate you know what the situation is. Getting increasingly 
belligerent is not going to help that situation and the energy would 
seem to me better spent on adapting to reality and moving on.


--
Adrian Klaver
adrian.kla...@aklaver.com


--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general


Re: [GENERAL] Delete rule does not prevent truncate

2015-07-24 Thread Andrew Sullivan
On Fri, Jul 24, 2015 at 10:24:55AM +0100, Tim Smith wrote:
> 
> If that's how the developers feel about rules, i.e they can't be
> bothered to make any changes to the rules code any more, no matter how
> minor (e.g. this TRUNCATE issue)

Who is this "they"?  As I think I suggested in another mail, if you
think this is trivial and easy then I think you should propose the
patch to solve it.  I understand what you're saying; I think the
solution is self-evident (add a statement trigger that captures
TRUNCATE and DO INSTEAD NOTHING), so I just wouldn't be bothered to
fix this.  But I suspect things are the way they are partly because
nobody proposed or implemented a patch for this behaviour before.

The manual is also quite clear in what statements you can
write rules about; by implication, other statements are not covered,
so I'm not actually sure why you think the manual is misleading.

Best regards,

A
-- 
Andrew Sullivan
a...@crankycanuck.ca


-- 
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general


Re: [GENERAL] Delete rule does not prevent truncate

2015-07-24 Thread Andres Freund
On 2015-07-24 10:29:21 +0100, Tim Smith wrote:
> That's not the point.  Backups are important, but so is the concept of
> various layers of anti-fat-finger protection.   Restoring off backups
> should be last resort, not first.

Oh, comeon. Install a TRUNCATE trigger and let this thread die.

Andres


-- 
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general


Re: [GENERAL] Delete rule does not prevent truncate

2015-07-24 Thread Tim Smith
On 24 July 2015 at 01:37, Rob Sargent  wrote:

> Fair enough but both blackhats and the authorized are just as likely to drop
> the database as truncate something (intentionally or not) and backups
> stashed everywhere is the first order of business.


That's not the point.  Backups are important, but so is the concept of
various layers of anti-fat-finger protection.   Restoring off backups
should be last resort, not first.


-- 
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general


Re: [GENERAL] Delete rule does not prevent truncate

2015-07-23 Thread Adrian Klaver

On 07/23/2015 05:37 PM, Rob Sargent wrote:

On 07/23/2015 06:27 PM, Adrian Klaver wrote:

On 07/23/2015 05:08 PM, Rob Sargent wrote:

On 07/23/2015 04:15 PM, Karsten Hilbert wrote:

On Thu, Jul 23, 2015 at 12:28:32PM -0600, Rob Sargent wrote:


I'm suggesting OP might find changing truncate statements to deletes
(without a where clause) a simpler solution. Something has to change.

Well, OP isn't looking for a solution to "delete all rows"
but rather to _prevent_ deletion.

Tim can't go forth and tell Blackhats to "please use DELETE
rather than TRUNCATE", right ?

AFAICT it'd be more useful to advise OP to revoke TRUNCATE
rights on tables.

Karsten

Not sure about Tim and the Blackhats (there's a band name in there
somewhere) but Wouldn't OP have exact same code to fix, one way or
another?



I think the point was, the OP(Tim) might not have access to the code
that is trying to TRUNCATE. This could be because it is coming from
authorized users who are writing their own code or unauthorized
users(Blackhats) who are trying to sneak code in.



Fair enough but both blackhats and the authorized are just as likely to
drop the database as truncate something (intentionally or not) and
backups stashed everywhere is the first order of business.


Well that is a different crisis and not covered by rules or triggers:)


--
Adrian Klaver
adrian.kla...@aklaver.com


--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general


Re: [GENERAL] Delete rule does not prevent truncate

2015-07-23 Thread Rob Sargent

On 07/23/2015 06:27 PM, Adrian Klaver wrote:

On 07/23/2015 05:08 PM, Rob Sargent wrote:

On 07/23/2015 04:15 PM, Karsten Hilbert wrote:

On Thu, Jul 23, 2015 at 12:28:32PM -0600, Rob Sargent wrote:


I'm suggesting OP might find changing truncate statements to deletes
(without a where clause) a simpler solution. Something has to change.

Well, OP isn't looking for a solution to "delete all rows"
but rather to _prevent_ deletion.

Tim can't go forth and tell Blackhats to "please use DELETE
rather than TRUNCATE", right ?

AFAICT it'd be more useful to advise OP to revoke TRUNCATE
rights on tables.

Karsten

Not sure about Tim and the Blackhats (there's a band name in there
somewhere) but Wouldn't OP have exact same code to fix, one way or 
another?




I think the point was, the OP(Tim) might not have access to the code 
that is trying to TRUNCATE. This could be because it is coming from 
authorized users who are writing their own code or unauthorized 
users(Blackhats) who are trying to sneak code in.



Fair enough but both blackhats and the authorized are just as likely to 
drop the database as truncate something (intentionally or not) and 
backups stashed everywhere is the first order of business.


Re: [GENERAL] Delete rule does not prevent truncate

2015-07-23 Thread Adrian Klaver

On 07/23/2015 05:08 PM, Rob Sargent wrote:

On 07/23/2015 04:15 PM, Karsten Hilbert wrote:

On Thu, Jul 23, 2015 at 12:28:32PM -0600, Rob Sargent wrote:


I'm suggesting OP might find changing truncate statements to deletes
(without a where clause) a simpler solution. Something has to change.

Well, OP isn't looking for a solution to "delete all rows"
but rather to _prevent_ deletion.

Tim can't go forth and tell Blackhats to "please use DELETE
rather than TRUNCATE", right ?

AFAICT it'd be more useful to advise OP to revoke TRUNCATE
rights on tables.

Karsten

Not sure about Tim and the Blackhats (there's a band name in there
somewhere) but Wouldn't OP have exact same code to fix, one way or another?



I think the point was, the OP(Tim) might not have access to the code 
that is trying to TRUNCATE. This could be because it is coming from 
authorized users who are writing their own code or unauthorized 
users(Blackhats) who are trying to sneak code in.



--
Adrian Klaver
adrian.kla...@aklaver.com


--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general


Re: [GENERAL] Delete rule does not prevent truncate

2015-07-23 Thread Rob Sargent

On 07/23/2015 04:15 PM, Karsten Hilbert wrote:

On Thu, Jul 23, 2015 at 12:28:32PM -0600, Rob Sargent wrote:


I'm suggesting OP might find changing truncate statements to deletes
(without a where clause) a simpler solution. Something has to change.

Well, OP isn't looking for a solution to "delete all rows"
but rather to _prevent_ deletion.

Tim can't go forth and tell Blackhats to "please use DELETE
rather than TRUNCATE", right ?

AFAICT it'd be more useful to advise OP to revoke TRUNCATE
rights on tables.

Karsten
Not sure about Tim and the Blackhats (there's a band name in there 
somewhere) but Wouldn't OP have exact same code to fix, one way or another?




Re: [GENERAL] Delete rule does not prevent truncate

2015-07-23 Thread Karsten Hilbert
On Thu, Jul 23, 2015 at 12:28:32PM -0600, Rob Sargent wrote:

> I'm suggesting OP might find changing truncate statements to deletes
> (without a where clause) a simpler solution. Something has to change.

Well, OP isn't looking for a solution to "delete all rows"
but rather to _prevent_ deletion.

Tim can't go forth and tell Blackhats to "please use DELETE
rather than TRUNCATE", right ?

AFAICT it'd be more useful to advise OP to revoke TRUNCATE
rights on tables.

Karsten
-- 
GPG key ID E4071346 @ eu.pool.sks-keyservers.net
E167 67FD A291 2BEA 73BD  4537 78B9 A9F9 E407 1346


-- 
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general


Re: [GENERAL] Delete rule does not prevent truncate

2015-07-23 Thread Rob Sargent

On 07/23/2015 12:25 PM, Adrian Klaver wrote:

On 07/23/2015 11:15 AM, Rob Sargent wrote:

On 07/23/2015 12:09 PM, Adrian Klaver wrote:

On 07/23/2015 04:57 AM, Tim Smith wrote:

Andrew,

 From the manual:

It is important to realize that a rule is really a command
transformation mechanism, or command macro. The transformation happens
before the execution of the command starts. If you actually want an
operation that fires independently for each physical row, you probably
want to use a trigger, not a rule


Thus, I should not have to use a trigger for TRUNCATE because the 
"each
row" concept does not apply. Plus it makes perfect sense to 
want to

transform the truncate command and transform into ignore



Just in case it has not been made obvious yet, rules are silently
deprecated. They still exist because views depend on them, but it is
generally considered best practices to not use them outside that
realm. So if you want the rule behavior to change for TRUNCATE(if that
is even possible) you are fighting an uphill battle. You may pursue
that fight of course, but I would think you will get a quicker return
on your time if you just forget about using a RULE and stick to a
TRIGGER instead.


Or change to using delete instead of truncate?



Well Tim has an ON DELETE rule:

http://www.postgresql.org/message-id/CA+HuS5G2bZYYOGTJrw+VosjUPO298swxuU=jorfav54ut7v...@mail.gmail.com 



His expectation was that would also catch a TRUNCATE based on this:

"... It has the same effect as an unqualified DELETE on each table, ..."

from here:

http://www.postgresql.org/docs/9.4/interactive/sql-truncate.html

It was then explained that while TRUNCATE had the same end result as 
'DELETE FROM some_table' it was actually a separate command and 
action. Tim wants to catch a TRUNCATE and turn it into an ignore.



I'm suggesting OP might find changing truncate statements to deletes 
(without a where clause) a simpler solution. Something has to change.





Re: [GENERAL] Delete rule does not prevent truncate

2015-07-23 Thread Scott Marlowe
On Thu, Jul 23, 2015 at 12:09 PM, Adrian Klaver
 wrote:
> On 07/23/2015 04:57 AM, Tim Smith wrote:
>>
>> Andrew,
>>
>>  From the manual:
>>
>> It is important to realize that a rule is really a command
>> transformation mechanism, or command macro. The transformation happens
>> before the execution of the command starts. If you actually want an
>> operation that fires independently for each physical row, you probably
>> want to use a trigger, not a rule
>>
>>
>> Thus, I should not have to use a trigger for TRUNCATE because the "each
>> row" concept does not apply. Plus it makes perfect sense to want to
>> transform the truncate command and transform into ignore
>>
>
> Just in case it has not been made obvious yet, rules are silently
> deprecated. They still exist because views depend on them, but it is
> generally considered best practices to not use them outside that realm. So
> if you want the rule behavior to change for TRUNCATE(if that is even
> possible) you are fighting an uphill battle. You may pursue that fight of
> course, but I would think you will get a quicker return on your time if you
> just forget about using a RULE and stick to a TRIGGER instead.
>


Also OP needs to know that COPY commands are ignored by rules as well.
I agree, stick to triggers, they're faster and less error prone.


-- 
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general


Re: [GENERAL] Delete rule does not prevent truncate

2015-07-23 Thread Adrian Klaver

On 07/23/2015 11:15 AM, Rob Sargent wrote:

On 07/23/2015 12:09 PM, Adrian Klaver wrote:

On 07/23/2015 04:57 AM, Tim Smith wrote:

Andrew,

 From the manual:

It is important to realize that a rule is really a command
transformation mechanism, or command macro. The transformation happens
before the execution of the command starts. If you actually want an
operation that fires independently for each physical row, you probably
want to use a trigger, not a rule


Thus, I should not have to use a trigger for TRUNCATE because the "each
row" concept does not apply. Plus it makes perfect sense to want to
transform the truncate command and transform into ignore



Just in case it has not been made obvious yet, rules are silently
deprecated. They still exist because views depend on them, but it is
generally considered best practices to not use them outside that
realm. So if you want the rule behavior to change for TRUNCATE(if that
is even possible) you are fighting an uphill battle. You may pursue
that fight of course, but I would think you will get a quicker return
on your time if you just forget about using a RULE and stick to a
TRIGGER instead.


Or change to using delete instead of truncate?



Well Tim has an ON DELETE rule:

http://www.postgresql.org/message-id/CA+HuS5G2bZYYOGTJrw+VosjUPO298swxuU=jorfav54ut7v...@mail.gmail.com

His expectation was that would also catch a TRUNCATE based on this:

"... It has the same effect as an unqualified DELETE on each table, ..."

from here:

http://www.postgresql.org/docs/9.4/interactive/sql-truncate.html

It was then explained that while TRUNCATE had the same end result as 
'DELETE FROM some_table' it was actually a separate command and action. 
Tim wants to catch a TRUNCATE and turn it into an ignore.



--
Adrian Klaver
adrian.kla...@aklaver.com


--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general


Re: [GENERAL] Delete rule does not prevent truncate

2015-07-23 Thread Rob Sargent

On 07/23/2015 12:09 PM, Adrian Klaver wrote:

On 07/23/2015 04:57 AM, Tim Smith wrote:

Andrew,

 From the manual:

It is important to realize that a rule is really a command
transformation mechanism, or command macro. The transformation happens
before the execution of the command starts. If you actually want an
operation that fires independently for each physical row, you probably
want to use a trigger, not a rule


Thus, I should not have to use a trigger for TRUNCATE because the "each
row" concept does not apply. Plus it makes perfect sense to want to
transform the truncate command and transform into ignore



Just in case it has not been made obvious yet, rules are silently 
deprecated. They still exist because views depend on them, but it is 
generally considered best practices to not use them outside that 
realm. So if you want the rule behavior to change for TRUNCATE(if that 
is even possible) you are fighting an uphill battle. You may pursue 
that fight of course, but I would think you will get a quicker return 
on your time if you just forget about using a RULE and stick to a 
TRIGGER instead.



Or change to using delete instead of truncate?



Re: [GENERAL] Delete rule does not prevent truncate

2015-07-23 Thread Adrian Klaver

On 07/23/2015 04:57 AM, Tim Smith wrote:

Andrew,

 From the manual:

It is important to realize that a rule is really a command
transformation mechanism, or command macro. The transformation happens
before the execution of the command starts. If you actually want an
operation that fires independently for each physical row, you probably
want to use a trigger, not a rule


Thus, I should not have to use a trigger for TRUNCATE because the "each
row" concept does not apply. Plus it makes perfect sense to want to
transform the truncate command and transform into ignore



Just in case it has not been made obvious yet, rules are silently 
deprecated. They still exist because views depend on them, but it is 
generally considered best practices to not use them outside that realm. 
So if you want the rule behavior to change for TRUNCATE(if that is even 
possible) you are fighting an uphill battle. You may pursue that fight 
of course, but I would think you will get a quicker return on your time 
if you just forget about using a RULE and stick to a TRIGGER instead.


--
Adrian Klaver
adrian.kla...@aklaver.com


--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general


Re: [GENERAL] Delete rule does not prevent truncate

2015-07-23 Thread Andres Freund
On 2015-07-23 12:57:20 +0100, Tim Smith wrote:
> Thus, I should not have to use a trigger for TRUNCATE because the "each
> row" concept does not apply. Plus it makes perfect sense to want to
> transform the truncate command and transform into ignore

That'd entirely defeat the point of TRUNCATE being fast.


Either way, this isn't going to change, so it seems a bit pointless to
continue arguing around it circles.


-- 
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general


Re: [GENERAL] Delete rule does not prevent truncate

2015-07-23 Thread Tim Smith
Andrew,

>From the manual:

It is important to realize that a rule is really a command transformation
mechanism, or command macro. The transformation happens before the
execution of the command starts. If you actually want an operation that
fires independently for each physical row, you probably want to use a
trigger, not a rule


Thus, I should not have to use a trigger for TRUNCATE because the "each
row" concept does not apply. Plus it makes perfect sense to want to
transform the truncate command and transform into ignore



On Thursday, 23 July 2015, Andrew Sullivan  wrote:

> On Thu, Jul 23, 2015 at 08:06:19AM +0100, Tim Smith wrote:
> > What exactly is was the design decision that lead to TRUNCATE being
> > supported by triggers but not by rules ?
>
> There are two things.  First, probably the design decision was, "I
> care about triggers."  TRUNCATE was added (I believe) in version 7.0,
> and even then there was some caution indicated about the use of rules.
> See for instance
> http://www.postgresql.org/docs/7.0/static/rules19784.htm.  So you
> might be partly right.
>
> But second, it isn't clear what it would mean for TRUNCATE to be
> supported by rules.  Rules do query parse tree rewriting.  That is,
> they rewrite the query on the way through the system before they can
> possibly have any effect, changing one SQL statement into
> (effectively) a different one by the time it executes.  There is only
> one possible effect from TRUNCATE, and that is to eliminate all the
> data in the table.  I don't know what rewriting such a query would
> mean.
>
> Best regards,
>
> A
>
> --
> Andrew Sullivan
> a...@crankycanuck.ca 
>
>
> --
> Sent via pgsql-general mailing list (pgsql-general@postgresql.org
> )
> To make changes to your subscription:
> http://www.postgresql.org/mailpref/pgsql-general
>


Re: [GENERAL] Delete rule does not prevent truncate

2015-07-23 Thread Andrew Sullivan
On Thu, Jul 23, 2015 at 12:57:20PM +0100, Tim Smith wrote:
> It is important to realize that a rule is really a command transformation
> mechanism, or command macro. The transformation happens before the
> execution of the command starts. If you actually want an operation that
> fires independently for each physical row, you probably want to use a
> trigger, not a rule

Well, yes, but the discussion of the rules system in earlier manuals
was actually, I thought, somewhat more detailed; and it outlined what
rules really did, which was alter the command at the parse tree.
That's what I think the above is saying also, but it may not be quite
as plain.  So it's rather more like a statement-level trigger.  

> Thus, I should not have to use a trigger for TRUNCATE because the "each
> row" concept does not apply. Plus it makes perfect sense to want to
> transform the truncate command and transform into ignore

Well, yes, but really in this case you want a per-statement trigger,
and there's not the same distinction in rules, either.

I can't believe that people would reject a patch (though you should
ask on -hackers, not here); but you asked what was behind the design
decision and I told you.  But in general, the experience seems to be
that triggers are easier to get right (novice or no, _pace_ section
38.7).

Best regards,

A

-- 
Andrew Sullivan
a...@crankycanuck.ca


-- 
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general


Re: [GENERAL] Delete rule does not prevent truncate

2015-07-23 Thread Andrew Sullivan
On Thu, Jul 23, 2015 at 08:06:19AM +0100, Tim Smith wrote:
> What exactly is was the design decision that lead to TRUNCATE being
> supported by triggers but not by rules ?

There are two things.  First, probably the design decision was, "I
care about triggers."  TRUNCATE was added (I believe) in version 7.0,
and even then there was some caution indicated about the use of rules.
See for instance
http://www.postgresql.org/docs/7.0/static/rules19784.htm.  So you
might be partly right.

But second, it isn't clear what it would mean for TRUNCATE to be
supported by rules.  Rules do query parse tree rewriting.  That is,
they rewrite the query on the way through the system before they can
possibly have any effect, changing one SQL statement into
(effectively) a different one by the time it executes.  There is only
one possible effect from TRUNCATE, and that is to eliminate all the
data in the table.  I don't know what rewriting such a query would
mean.

Best regards,

A

-- 
Andrew Sullivan
a...@crankycanuck.ca


-- 
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general


Re: [GENERAL] Delete rule does not prevent truncate

2015-07-23 Thread Guillaume Lelarge
2015-07-23 9:06 GMT+02:00 Tim Smith :

> So tell me guys, instead of bashing away at the fact I only quoted
> half a sentence or whatever, how about you answer the following :
>
> What exactly is was the design decision that lead to TRUNCATE being
> supported by triggers but not by rules ?
>
>
Someone had time to implement it for triggers, no-one had time for rules.


> I suspect that TRUNCATE was added to triggers because some dev thought
> it would be a neat idea, and it was never implemented in rules as a
> result of an accidental omission for  rather than a
> deliberate design constraint.
>
>
It is a neat idea for tiggers. Slony uses that to replicate TRUNCATE on
slaves of a Slony cluster.

It wouldn't be such a neat idea for rules as, IIRC, rules are only
supported because views are based on them. Without that, they would
probably be ripped out of the code.


-- 
Guillaume.
  http://blog.guillaume.lelarge.info
  http://www.dalibo.com


Re: [GENERAL] Delete rule does not prevent truncate

2015-07-23 Thread Tim Smith
So tell me guys, instead of bashing away at the fact I only quoted
half a sentence or whatever, how about you answer the following :

What exactly is was the design decision that lead to TRUNCATE being
supported by triggers but not by rules ?

I suspect that TRUNCATE was added to triggers because some dev thought
it would be a neat idea, and it was never implemented in rules as a
result of an accidental omission for  rather than a
deliberate design constraint.


-- 
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general


Re: [GENERAL] Delete rule does not prevent truncate

2015-07-22 Thread Geoff Winkless
On 22 July 2015 at 16:55, Joshua D. Drake  wrote:

>
> On 07/22/2015 08:42 AM, Geoff Winkless wrote:
>
>> On 22 July 2015 at 16:32, Joshua D. Drake > >wrote:
>>
>> This is actually wrong. The end result is the same but it does not
>> in any way have the same effect.
>>
>> ​
>> "in any way"? ​I'd say in the primary way it has the same effect: all
>> rows are removed
>> from the table.
>>
>
> Thus the end result is the same as I said but the in practice effect is
> quite different from a visibility, maintenance and programmability
> perspective.


But to say "it does not in any way have the same effect" explicitly
excludes that any effect of the two things might be the same. In actual
fact, in the simple case (no triggers) the effect *is *the same.

FWIW, the difference between "in practice effect" and "end result" is
pretty esoteric, frankly - I'm not really sure what you're driving at
there. The "end result" would surely include the behaviour of any triggers
that might be fired.

TRUNCATE is NOT DELETE.
>>
>>
>> ​I don't think anyone is suggesting that it is.​
>>
>
> Except Tim Smith who started this thread.
>
>
​I don't believe he suggested that at all: he made an assumption that his
DELETE triggers would fire on TRUNCATE, and when it was pointed out that he
was mistaken he posted *half* a sentence from the documentation that
supported his view, ignoring the second half of that sentence that makes it
clear that the behaviour of the two commands is different, and ignoring the
explicit statement *on the very same page *of the documentation viz:

"
TRUNCATE will not fire any ON DELETE triggers that might exist for the
tables
​"​.

​Geoff​


Re: [GENERAL] Delete rule does not prevent truncate

2015-07-22 Thread Joshua D. Drake


On 07/22/2015 08:42 AM, Geoff Winkless wrote:

On 22 July 2015 at 16:32, Joshua D. Drake mailto:j...@commandprompt.com>>wrote:

This is actually wrong. The end result is the same but it does not
in any way have the same effect.

​
"in any way"? ​
​I'd say in the primary way it has the same effect: all rows are removed
from the table.


Thus the end result is the same as I said but the in practice effect is 
quite different from a visibility, maintenance and programmability 
perspective.




TRUNCATE is NOT DELETE.


​I don't think anyone is suggesting that it is.​


Except Tim Smith who started this thread.

JD

--
Command Prompt, Inc. - http://www.commandprompt.com/  503-667-4564
PostgreSQL Centered full stack support, consulting and development.
Announcing "I'm offended" is basically telling the world you can't
control your own emotions, so everyone else should do it for you.


--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general


Re: [GENERAL] Delete rule does not prevent truncate

2015-07-22 Thread Geoff Winkless
On 22 July 2015 at 16:32, Joshua D. Drake  wrote:

> This is actually wrong. The end result is the same but it does not in any
> way have the same effect.

​
"in any way"? ​

​I'd say in the primary way it has the same effect: all rows are removed
from the table.
​

> And I will submit a patch.
>

​
As long as the patch is ", although your attention should be brought to the
caveats listed below, since you're obviously
​incapable of​
 realis
​ing​
that there's a page and a half of
​information
 beneath this sentence"?
​


> TRUNCATE is NOT DELETE.
>

​I don't think anyone is suggesting that it is.​

​Otherwise there wouldn't be much point having it.​

​Geoff​


Re: [GENERAL] Delete rule does not prevent truncate

2015-07-22 Thread Joshua D. Drake


On 07/22/2015 06:24 AM, Tim Smith wrote:


Adrian,

It still doesn't make much sense, especially as given the rather
obscure and questionable design decision of allowing triggers to refer
to truncate ops, but not allowing rules to refer to truncate ops !!!


Actually it makes perfect sense because rules are a feature for 
compatibility (at this point) more than anything else. They are slower 
than triggers, less flexible and widely considered something you only 
use in very rare circumstances.


That and of course, patches are accepted if you feel it is a feature 
worth having.


JD




--
Command Prompt, Inc. - http://www.commandprompt.com/  503-667-4564
PostgreSQL Centered full stack support, consulting and development.
Announcing "I'm offended" is basically telling the world you can't
control your own emotions, so everyone else should do it for you.


--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general


Re: [GENERAL] Delete rule does not prevent truncate

2015-07-22 Thread Joshua D. Drake


On 07/22/2015 06:13 AM, Tim Smith wrote:

Melvin,

May I point out that the manual states :
"TRUNCATE quickly removes all rows from a set of tables. It has the same
effect as an unqualified DELETE on each table"


This is actually wrong. The end result is the same but it does not in 
any way have the same effect. And I will submit a patch.


TRUNCATE is NOT DELETE.

Sincerely,

JD

--
Command Prompt, Inc. - http://www.commandprompt.com/  503-667-4564
PostgreSQL Centered full stack support, consulting and development.
Announcing "I'm offended" is basically telling the world you can't
control your own emotions, so everyone else should do it for you.


--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general


Re: [GENERAL] Delete rule does not prevent truncate

2015-07-22 Thread Adrian Klaver

On 07/22/2015 06:24 AM, Tim Smith wrote:

Adrian,

It still doesn't make much sense, especially as given the rather
obscure and questionable design decision of allowing triggers to refer
to truncate ops, but not allowing rules to refer to truncate ops !!!

Surely either you say "look, truncate is truncate, its there for one
purpose and one purpose only".Or otherwise, you should handle it
consistently across the database, i.e. if you're going to allow
triggers interact with truncates, then you should allow rules to
interact with truncates.It really doesn't make much sense to adopt
a pick and choose mentality !


All I know is that TRUNCATE is a shortcut and RULEs do not understand it 
and TRIGGERs do. My guess is the answer somewhere in here:


http://www.postgresql.org/docs/9.4/interactive/rules.html

Word of advice, take two aspirin before reading above.

At any rate,  I have personally found using triggers results in less 
surprises then using rules.




On 22 July 2015 at 14:19, Adrian Klaver  wrote:

On 07/22/2015 06:13 AM, Tim Smith wrote:


Melvin,

May I point out that the manual states :
"TRUNCATE quickly removes all rows from a set of tables. It has the same
effect as an unqualified DELETE on each table"

Thus, if you are telling me to effectively think of TRUNCATE as an alias
to DELETE, then I would think its not entirely unreasonable of me to
expect a rule preventing DELETE to also cover truncate, since the rule
would no doubt prevent an unqualified DELETE, would it not ?!?



If  you go further down into the Notes section you find:

"TRUNCATE will not fire any ON DELETE triggers that might exist for the
tables. But it will fire ON TRUNCATE triggers. If ON TRUNCATE triggers are
defined for any of the tables, then all BEFORE TRUNCATE triggers are fired
before any truncation happens, and all AFTER TRUNCATE triggers are fired
after the last truncation is performed and any sequences are reset. The
triggers will fire in the order that the tables are to be processed (first
those listed in the command, and then any that were added due to cascading).
Warning

TRUNCATE is not MVCC-safe (see Chapter 13 for general information about
MVCC). After truncation, the table will appear empty to all concurrent
transactions, even if they are using a snapshot taken before the truncation
occurred. This will only be an issue for a transaction that did not access
the truncated table before the truncation happened — any transaction that
has done so would hold at least an ACCESS SHARE lock, which would block
TRUNCATE until that transaction completes. So truncation will not cause any
apparent inconsistency in the table contents for successive queries on the
same table, but it could cause visible inconsistency between the contents of
the truncated table and other tables in the database.

"


TRUNCATE is when you want fast over safety.



On 22 July 2015 at 14:03, Melvin Davidson mailto:melvin6...@gmail.com>> wrote:

 Actually, if you use a TRIGGER instead of rule, you can handle this.
 The manual states event can be:

 INSERT
 UPDATE [ OFcolumn_name  [, ... ] ]
 DELETE
 *TRUNCATE <-*

 http://www.postgresql.org/docs/9.4/interactive/sql-createtrigger.html

 I suggest you review carefully.

 On Wed, Jul 22, 2015 at 8:53 AM, Tim Smith
 mailto:randomdev4+postg...@gmail.com>> wrote:

 Hi,

 I very much hope this is an accidental bug rather than a
 deliberate feature !

 PostgreSQL 9.4.4

 create rule no_auditupd as on update to app_security.app_audit do
 instead nothing;
 create rule no_auditdel as on delete to app_security.app_audit do
 instead nothing;

 \d+  app_security.app_audit
 
 Rules:
  no_auditdel AS
  ON DELETE TO app_security.app_audit DO INSTEAD NOTHING
  no_auditupd AS
  ON UPDATE TO app_security.app_audit DO INSTEAD NOTHING

 The truncate trashes the whole table  ;-(

 According to the FabulousManual(TM) :
 event : The event is one of SELECT, INSERT, UPDATE, or DELETE.

 Thus I can't create a rule to "do nothing" on truncates, thus I
 am stuck !


 --
 Sent via pgsql-general mailing list
 (pgsql-general@postgresql.org
)
 To make changes to your subscription:
 http://www.postgresql.org/mailpref/pgsql-general




 --
 *Melvin Davidson*
 I reserve the right to fantasize.  Whether or not you
 wish to share my fantasy is entirely up to you.





--
Adrian Klaver
adrian.kla...@aklaver.com






--
Adrian Klaver
adrian.kla...@aklaver.com


--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general


Re: [GENERAL] Delete rule does not prevent truncate

2015-07-22 Thread Melvin Davidson
No,

I am saying if you

CREATE PROCEDURE do_nothing()
RETURNS VOID
$BODY$
BEGIN
RETURN;
END
LANGUAGE plpgsql;

CREATE TRIGGER no_trunc INSTEAD OF TRUNCATE ON your_table
EXECUTE PROCEDURE do_nothing;


Then you can handle the problem.

You should also create a TRIGGER for DELETE to do the same.

On Wed, Jul 22, 2015 at 9:13 AM, Tim Smith 
wrote:

> Melvin,
>
> May I point out that the manual states :
> "TRUNCATE quickly removes all rows from a set of tables. It has the same
> effect as an unqualified DELETE on each table"
>
> Thus, if you are telling me to effectively think of TRUNCATE as an alias
> to DELETE, then I would think its not entirely unreasonable of me to expect
> a rule preventing DELETE to also cover truncate, since the rule would no
> doubt prevent an unqualified DELETE, would it not ?!?
>
> On 22 July 2015 at 14:03, Melvin Davidson  wrote:
>
>> Actually, if you use a TRIGGER instead of rule, you can handle this.
>> The manual states event can be:
>>
>> INSERT
>> UPDATE [ OF column_name [, ... ] ]
>> DELETE*TRUNCATE   <-*
>>
>> http://www.postgresql.org/docs/9.4/interactive/sql-createtrigger.html
>>
>> I suggest you review carefully.
>>
>> On Wed, Jul 22, 2015 at 8:53 AM, Tim Smith > > wrote:
>>
>>> Hi,
>>>
>>> I very much hope this is an accidental bug rather than a deliberate
>>> feature !
>>>
>>> PostgreSQL 9.4.4
>>>
>>> create rule no_auditupd as on update to app_security.app_audit do
>>> instead nothing;
>>> create rule no_auditdel as on delete to app_security.app_audit do
>>> instead nothing;
>>>
>>> \d+  app_security.app_audit
>>> 
>>> Rules:
>>> no_auditdel AS
>>> ON DELETE TO app_security.app_audit DO INSTEAD NOTHING
>>> no_auditupd AS
>>> ON UPDATE TO app_security.app_audit DO INSTEAD NOTHING
>>>
>>> The truncate trashes the whole table  ;-(
>>>
>>> According to the FabulousManual(TM) :
>>> event : The event is one of SELECT, INSERT, UPDATE, or DELETE.
>>>
>>> Thus I can't create a rule to "do nothing" on truncates, thus I am stuck
>>> !
>>>
>>>
>>> --
>>> Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
>>> To make changes to your subscription:
>>> http://www.postgresql.org/mailpref/pgsql-general
>>>
>>
>>
>>
>> --
>> *Melvin Davidson*
>> I reserve the right to fantasize.  Whether or not you
>> wish to share my fantasy is entirely up to you.
>>
>
>


-- 
*Melvin Davidson*
I reserve the right to fantasize.  Whether or not you
wish to share my fantasy is entirely up to you.


Re: [GENERAL] Delete rule does not prevent truncate

2015-07-22 Thread Tim Smith
Adrian,

It still doesn't make much sense, especially as given the rather
obscure and questionable design decision of allowing triggers to refer
to truncate ops, but not allowing rules to refer to truncate ops !!!

Surely either you say "look, truncate is truncate, its there for one
purpose and one purpose only".Or otherwise, you should handle it
consistently across the database, i.e. if you're going to allow
triggers interact with truncates, then you should allow rules to
interact with truncates.It really doesn't make much sense to adopt
a pick and choose mentality !

On 22 July 2015 at 14:19, Adrian Klaver  wrote:
> On 07/22/2015 06:13 AM, Tim Smith wrote:
>>
>> Melvin,
>>
>> May I point out that the manual states :
>> "TRUNCATE quickly removes all rows from a set of tables. It has the same
>> effect as an unqualified DELETE on each table"
>>
>> Thus, if you are telling me to effectively think of TRUNCATE as an alias
>> to DELETE, then I would think its not entirely unreasonable of me to
>> expect a rule preventing DELETE to also cover truncate, since the rule
>> would no doubt prevent an unqualified DELETE, would it not ?!?
>
>
> If  you go further down into the Notes section you find:
>
> "TRUNCATE will not fire any ON DELETE triggers that might exist for the
> tables. But it will fire ON TRUNCATE triggers. If ON TRUNCATE triggers are
> defined for any of the tables, then all BEFORE TRUNCATE triggers are fired
> before any truncation happens, and all AFTER TRUNCATE triggers are fired
> after the last truncation is performed and any sequences are reset. The
> triggers will fire in the order that the tables are to be processed (first
> those listed in the command, and then any that were added due to cascading).
> Warning
>
> TRUNCATE is not MVCC-safe (see Chapter 13 for general information about
> MVCC). After truncation, the table will appear empty to all concurrent
> transactions, even if they are using a snapshot taken before the truncation
> occurred. This will only be an issue for a transaction that did not access
> the truncated table before the truncation happened — any transaction that
> has done so would hold at least an ACCESS SHARE lock, which would block
> TRUNCATE until that transaction completes. So truncation will not cause any
> apparent inconsistency in the table contents for successive queries on the
> same table, but it could cause visible inconsistency between the contents of
> the truncated table and other tables in the database.
>
> "
>
>
> TRUNCATE is when you want fast over safety.
>
>>
>> On 22 July 2015 at 14:03, Melvin Davidson > > wrote:
>>
>> Actually, if you use a TRIGGER instead of rule, you can handle this.
>> The manual states event can be:
>>
>> INSERT
>> UPDATE [ OFcolumn_name  [, ... ] ]
>> DELETE
>> *TRUNCATE <-*
>>
>> http://www.postgresql.org/docs/9.4/interactive/sql-createtrigger.html
>>
>> I suggest you review carefully.
>>
>> On Wed, Jul 22, 2015 at 8:53 AM, Tim Smith
>> > > wrote:
>>
>> Hi,
>>
>> I very much hope this is an accidental bug rather than a
>> deliberate feature !
>>
>> PostgreSQL 9.4.4
>>
>> create rule no_auditupd as on update to app_security.app_audit do
>> instead nothing;
>> create rule no_auditdel as on delete to app_security.app_audit do
>> instead nothing;
>>
>> \d+  app_security.app_audit
>> 
>> Rules:
>>  no_auditdel AS
>>  ON DELETE TO app_security.app_audit DO INSTEAD NOTHING
>>  no_auditupd AS
>>  ON UPDATE TO app_security.app_audit DO INSTEAD NOTHING
>>
>> The truncate trashes the whole table  ;-(
>>
>> According to the FabulousManual(TM) :
>> event : The event is one of SELECT, INSERT, UPDATE, or DELETE.
>>
>> Thus I can't create a rule to "do nothing" on truncates, thus I
>> am stuck !
>>
>>
>> --
>> Sent via pgsql-general mailing list
>> (pgsql-general@postgresql.org
>> )
>> To make changes to your subscription:
>> http://www.postgresql.org/mailpref/pgsql-general
>>
>>
>>
>>
>> --
>> *Melvin Davidson*
>> I reserve the right to fantasize.  Whether or not you
>> wish to share my fantasy is entirely up to you.
>>
>>
>
>
> --
> Adrian Klaver
> adrian.kla...@aklaver.com


-- 
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general


Re: [GENERAL] Delete rule does not prevent truncate

2015-07-22 Thread Adrian Klaver

On 07/22/2015 06:13 AM, Tim Smith wrote:

Melvin,

May I point out that the manual states :
"TRUNCATE quickly removes all rows from a set of tables. It has the same
effect as an unqualified DELETE on each table"

Thus, if you are telling me to effectively think of TRUNCATE as an alias
to DELETE, then I would think its not entirely unreasonable of me to
expect a rule preventing DELETE to also cover truncate, since the rule
would no doubt prevent an unqualified DELETE, would it not ?!?


If  you go further down into the Notes section you find:

"TRUNCATE will not fire any ON DELETE triggers that might exist for the 
tables. But it will fire ON TRUNCATE triggers. If ON TRUNCATE triggers 
are defined for any of the tables, then all BEFORE TRUNCATE triggers are 
fired before any truncation happens, and all AFTER TRUNCATE triggers are 
fired after the last truncation is performed and any sequences are 
reset. The triggers will fire in the order that the tables are to be 
processed (first those listed in the command, and then any that were 
added due to cascading).

Warning

TRUNCATE is not MVCC-safe (see Chapter 13 for general information about 
MVCC). After truncation, the table will appear empty to all concurrent 
transactions, even if they are using a snapshot taken before the 
truncation occurred. This will only be an issue for a transaction that 
did not access the truncated table before the truncation happened — any 
transaction that has done so would hold at least an ACCESS SHARE lock, 
which would block TRUNCATE until that transaction completes. So 
truncation will not cause any apparent inconsistency in the table 
contents for successive queries on the same table, but it could cause 
visible inconsistency between the contents of the truncated table and 
other tables in the database.


"


TRUNCATE is when you want fast over safety.



On 22 July 2015 at 14:03, Melvin Davidson mailto:melvin6...@gmail.com>> wrote:

Actually, if you use a TRIGGER instead of rule, you can handle this.
The manual states event can be:

INSERT
UPDATE [ OFcolumn_name  [, ... ] ]
DELETE
*TRUNCATE <-*

http://www.postgresql.org/docs/9.4/interactive/sql-createtrigger.html

I suggest you review carefully.

On Wed, Jul 22, 2015 at 8:53 AM, Tim Smith
mailto:randomdev4+postg...@gmail.com>> wrote:

Hi,

I very much hope this is an accidental bug rather than a
deliberate feature !

PostgreSQL 9.4.4

create rule no_auditupd as on update to app_security.app_audit do
instead nothing;
create rule no_auditdel as on delete to app_security.app_audit do
instead nothing;

\d+  app_security.app_audit

Rules:
 no_auditdel AS
 ON DELETE TO app_security.app_audit DO INSTEAD NOTHING
 no_auditupd AS
 ON UPDATE TO app_security.app_audit DO INSTEAD NOTHING

The truncate trashes the whole table  ;-(

According to the FabulousManual(TM) :
event : The event is one of SELECT, INSERT, UPDATE, or DELETE.

Thus I can't create a rule to "do nothing" on truncates, thus I
am stuck !


--
Sent via pgsql-general mailing list
(pgsql-general@postgresql.org )
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general




--
*Melvin Davidson*
I reserve the right to fantasize.  Whether or not you
wish to share my fantasy is entirely up to you.





--
Adrian Klaver
adrian.kla...@aklaver.com


--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general


Re: [GENERAL] Delete rule does not prevent truncate

2015-07-22 Thread Tim Smith
Melvin,

May I point out that the manual states :
"TRUNCATE quickly removes all rows from a set of tables. It has the same
effect as an unqualified DELETE on each table"

Thus, if you are telling me to effectively think of TRUNCATE as an alias to
DELETE, then I would think its not entirely unreasonable of me to expect a
rule preventing DELETE to also cover truncate, since the rule would no
doubt prevent an unqualified DELETE, would it not ?!?

On 22 July 2015 at 14:03, Melvin Davidson  wrote:

> Actually, if you use a TRIGGER instead of rule, you can handle this.
> The manual states event can be:
>
> INSERT
> UPDATE [ OF column_name [, ... ] ]
> DELETE*TRUNCATE   <-*
>
> http://www.postgresql.org/docs/9.4/interactive/sql-createtrigger.html
>
> I suggest you review carefully.
>
> On Wed, Jul 22, 2015 at 8:53 AM, Tim Smith 
> wrote:
>
>> Hi,
>>
>> I very much hope this is an accidental bug rather than a deliberate
>> feature !
>>
>> PostgreSQL 9.4.4
>>
>> create rule no_auditupd as on update to app_security.app_audit do
>> instead nothing;
>> create rule no_auditdel as on delete to app_security.app_audit do
>> instead nothing;
>>
>> \d+  app_security.app_audit
>> 
>> Rules:
>> no_auditdel AS
>> ON DELETE TO app_security.app_audit DO INSTEAD NOTHING
>> no_auditupd AS
>> ON UPDATE TO app_security.app_audit DO INSTEAD NOTHING
>>
>> The truncate trashes the whole table  ;-(
>>
>> According to the FabulousManual(TM) :
>> event : The event is one of SELECT, INSERT, UPDATE, or DELETE.
>>
>> Thus I can't create a rule to "do nothing" on truncates, thus I am stuck !
>>
>>
>> --
>> Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
>> To make changes to your subscription:
>> http://www.postgresql.org/mailpref/pgsql-general
>>
>
>
>
> --
> *Melvin Davidson*
> I reserve the right to fantasize.  Whether or not you
> wish to share my fantasy is entirely up to you.
>


Re: [GENERAL] Delete rule does not prevent truncate

2015-07-22 Thread Melvin Davidson
Actually, if you use a TRIGGER instead of rule, you can handle this.
The manual states event can be:

INSERT
UPDATE [ OF column_name [, ... ] ]
DELETE*TRUNCATE   <-*

http://www.postgresql.org/docs/9.4/interactive/sql-createtrigger.html

I suggest you review carefully.

On Wed, Jul 22, 2015 at 8:53 AM, Tim Smith 
wrote:

> Hi,
>
> I very much hope this is an accidental bug rather than a deliberate
> feature !
>
> PostgreSQL 9.4.4
>
> create rule no_auditupd as on update to app_security.app_audit do
> instead nothing;
> create rule no_auditdel as on delete to app_security.app_audit do
> instead nothing;
>
> \d+  app_security.app_audit
> 
> Rules:
> no_auditdel AS
> ON DELETE TO app_security.app_audit DO INSTEAD NOTHING
> no_auditupd AS
> ON UPDATE TO app_security.app_audit DO INSTEAD NOTHING
>
> The truncate trashes the whole table  ;-(
>
> According to the FabulousManual(TM) :
> event : The event is one of SELECT, INSERT, UPDATE, or DELETE.
>
> Thus I can't create a rule to "do nothing" on truncates, thus I am stuck !
>
>
> --
> Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
> To make changes to your subscription:
> http://www.postgresql.org/mailpref/pgsql-general
>



-- 
*Melvin Davidson*
I reserve the right to fantasize.  Whether or not you
wish to share my fantasy is entirely up to you.


[GENERAL] Delete rule does not prevent truncate

2015-07-22 Thread Tim Smith
Hi,

I very much hope this is an accidental bug rather than a deliberate feature !

PostgreSQL 9.4.4

create rule no_auditupd as on update to app_security.app_audit do
instead nothing;
create rule no_auditdel as on delete to app_security.app_audit do
instead nothing;

\d+  app_security.app_audit

Rules:
no_auditdel AS
ON DELETE TO app_security.app_audit DO INSTEAD NOTHING
no_auditupd AS
ON UPDATE TO app_security.app_audit DO INSTEAD NOTHING

The truncate trashes the whole table  ;-(

According to the FabulousManual(TM) :
event : The event is one of SELECT, INSERT, UPDATE, or DELETE.

Thus I can't create a rule to "do nothing" on truncates, thus I am stuck !


-- 
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general


Re: [GENERAL] delete is getting hung when there is a huge data in table

2015-05-06 Thread Merlin Moncure
On Wed, May 6, 2015 at 1:56 AM, Mitu Verma  wrote:
> Thank you so much all of you.
>
> Table audittraillogentry have PRIMARY KEY and FOREIGN KEY defined, below is 
> the detail of existing table audittraillogentry.
>
> As you can see ., it is referenced by 2 tables , "cdrdetails" and 
> "cdrlogentry" . "cdrdetails" table do not have the index whereas 
> "cdrlogentry" has the index.
> Now  after creating the index on "cdrdetails", deletion has become fast, 12 
> lakh records are deleted in 16 minutes, which is a drastic improvement in 
> performance.
> Before indexing deletion of 500 records were taking  ~2 minutes.

Yeah, this (unindexed foreign key causing slow deletes) is probably
the #1 performance gotcha in SQL.

If you're often doing very large deletes, sometimes it can help to
attempt to work out a better strategy, perhaps one of:

*) using TRUNCATE...CASADE
*) table partitioning organized such that you can drop a partition to
delete rows
*) temporarily disabling RI during large deletes (can be dangerous and
but in certain limited cases can be useful).

merlin


-- 
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general


Re: [GENERAL] delete is getting hung when there is a huge data in table

2015-05-05 Thread Mitu Verma
Thank you so much all of you.

Table audittraillogentry have PRIMARY KEY and FOREIGN KEY defined, below is the 
detail of existing table audittraillogentry.

As you can see ., it is referenced by 2 tables , "cdrdetails" and "cdrlogentry" 
. "cdrdetails" table do not have the index whereas "cdrlogentry" has the index.
Now  after creating the index on "cdrdetails", deletion has become fast, 12 
lakh records are deleted in 16 minutes, which is a drastic improvement in 
performance.
Before indexing deletion of 500 records were taking  ~2 minutes.

fm_db_Server1-> \d+ audittraillogentry
 Table "mmsuper.audittraillogentry"
  Column  |Type | Modifiers | Storage  
| Description
--+-+---+--+-
 event| smallint|   | plain|
 innodeid | character varying(80)   |   | extended |
 innodename   | character varying(80)   |   | extended |
 sourceid | character varying(300)  |   | extended |
 intime   | timestamp without time zone |   | plain|
 outnodeid| character varying(80)   |   | extended |
 outnodename  | character varying(80)   |   | extended |
 destinationid| character varying(300)  |   | extended |
 outtime  | timestamp without time zone |   | plain|
 bytes| bigint  |   | plain|
 cdrs | bigint  |   | plain|
 tableindex   | bigint  | not null  | plain|
 noofsubfilesinfile   | bigint  |   | plain|
 recordsequencenumberlist | character varying(1000) |   | extended |
Indexes:
"audittraillogentry_pkey" PRIMARY KEY, btree (tableindex), tablespace 
"mmdata"
"audit_destid_index" btree (destinationid), tablespace "mmindex"
"audit_intime_index" btree (intime DESC), tablespace "mmindex"
"audit_outtime_index" btree (outtime DESC), tablespace "mmindex"
"audit_sourceid_index" btree (sourceid), tablespace "mmindex"
Referenced by:
TABLE "cdrdetails" CONSTRAINT "audittableindex_fkey" FOREIGN KEY 
(audittableindex) REFERENCES audittraillogentry(tableindex) ON DELETE CASCADE
TABLE "cdrlogentry" CONSTRAINT "cdrlogentry_audittableindex_fkey" FOREIGN 
KEY (audittableindex) REFERENCES audittraillogentry(tableindex)
Has OIDs: no
Tablespace: "mmdata"


-Original Message-
From: Tom Lane [mailto:t...@sss.pgh.pa.us] 
Sent: May 03, 2015 9:43 AM
To: Mitu Verma
Cc: 'pgsql-general@postgresql.org' (pgsql-general@postgresql.org)
Subject: Re: [GENERAL] delete is getting hung when there is a huge data in table

Mitu Verma  writes:
> 1. If postgreSQL has some limitations for deletion of large data?

Not as such, but you've not given us any details that would permit comment.

A reasonably likely bet is that this table is referenced by a foreign key in 
some other table, and that other table has no index on the referencing column.  
That would make the FK is-it-ok-to-delete checks very slow.

regards, tom lane


-- 
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general


Re: [GENERAL] delete is getting hung when there is a huge data in table

2015-05-03 Thread Andomar
> Now issue is that when this script for the deletion of data is 
launched , it is taking more than 7 days and doing nothing i.e not a 
single row has been deleted.


Deleting a large number of rows can take a long time.  Often it's 
quicker to delete smaller chunks.  The LIMIT clause is not supported by 
DELETE, so you need some kind of subquery.


We use something like:

do $_$declare
num_rows bigint;
begin
loop
delete from YourTable where id in
(select id from YourTable where id < 500 limit 100);
get diagnostics num_rows = row_count;
raise notice 'deleted % rows', num_rows;
exit when num_rows = 0;
end loop;
end;$_$;

This deletes rows with an id smaller than 500 in chunks of 100.

Kind regards,
Andomar


--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general


Re: [GENERAL] delete is getting hung when there is a huge data in table

2015-05-02 Thread Uwe Schroeder

This delete runs in a single transaction. That means the entire transaction 
has to complete before you will see anything deleted. Interrupting the 
transaction simply rolls it back, so nothing is deleted.
Tom already pointed out the potential foreign key slowdown, another slowdown 
may simply be drive speed.

My recommendation: cut the delete in chunks. For example delete the data one 
week at a time. That way the transaction is smaller, the dataset to delete is 
smaller and it will finish quicker.  

Uwe



On Sun, May 03, 2015 03:24:25 AM Mitu Verma wrote:
> Hi,
> 
> I am facing an issue with the deletion of huge data.
> We have a cronscript which is used to delete the data of last 3 months from
> one of the tables. Data in the table is large (8872597 as you can see the
> count below) since it is from last 3 months.
> 
> fm_db_Server3=# select count(*) from audittraillogentry ;
> 
> 
>   count
> -
> 8872597
> (1 row)
> 
> Now issue is that when this script for the deletion of data is launched , it
> is taking more than 7 days and doing nothing i.e not a single row has been
> deleted.
> 
> Then we stopped the script,terminated the database sessions by using SELECT
> pg_terminate_backend(proc pid) and run the following command
> 
> delete from audittraillogentry where intime <= to_timestamp('2015-01-30
> 23:59:59.999', '/MM/DD-HH24:MI:SS.FF3') OR outtime  <=
> to_timestamp('2015-01-30 23:59:59.999', '/MM/DD-HH24:MI:SS.FF3'); still
> this delete operation is not working and not a single row has been deleted
> from the table.
> 
> Now i have following questions -
> 
> 1. If postgreSQL has some limitations for deletion of large data?
> 2. If i should run the vacumm, after stopping the cron script ? because
> probably to get the "smaller" table? 3. if dropping the indexes can help
> here? now sure.
> 4.if i should think about partitioning , if there is any limitation while
> delaing with large data in postgreSQL?
> 
> regards
> Mitu
> 
> _
> ___



-- 
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general


Re: [GENERAL] delete is getting hung when there is a huge data in table

2015-05-02 Thread David G. Johnston
On Saturday, May 2, 2015, Mitu Verma  wrote:

>
> still this delete operation is not working and not a single row has been
> deleted from the table.
>
>
Because of MVCC other sessions are not able to see partial deletions...and
as you aluded to knowing the data itself is not actually removed by a
deletion though a vacuum will eventually allow the same space to be reused
thus deferring the increase in size from future insertions.

David J.


Re: [GENERAL] delete is getting hung when there is a huge data in table

2015-05-02 Thread Tom Lane
Mitu Verma  writes:
> 1. If postgreSQL has some limitations for deletion of large data?

Not as such, but you've not given us any details that would permit
comment.

A reasonably likely bet is that this table is referenced by a foreign key
in some other table, and that other table has no index on the referencing
column.  That would make the FK is-it-ok-to-delete checks very slow.

regards, tom lane


-- 
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general


[GENERAL] delete is getting hung when there is a huge data in table

2015-05-02 Thread Mitu Verma
Hi,

I am facing an issue with the deletion of huge data.
We have a cronscript which is used to delete the data of last 3 months from one 
of the tables.
Data in the table is large (8872597 as you can see the count below) since it is 
from last 3 months.

fm_db_Server3=# select count(*) from audittraillogentry ;
 
 
  count
-
8872597
(1 row)

Now issue is that when this script for the deletion of data is launched , it is 
taking more than 7 days and doing nothing i.e not a single row has been deleted.

Then we stopped the script,terminated the database sessions by using SELECT 
pg_terminate_backend(proc pid) and run the following command

delete from audittraillogentry where intime <= to_timestamp('2015-01-30 
23:59:59.999', '/MM/DD-HH24:MI:SS.FF3') OR outtime  <= 
to_timestamp('2015-01-30 23:59:59.999', '/MM/DD-HH24:MI:SS.FF3');
still this delete operation is not working and not a single row has been 
deleted from the table.

Now i have following questions -

1. If postgreSQL has some limitations for deletion of large data?
2. If i should run the vacumm, after stopping the cron script ? because 
probably to get the "smaller" table?
3. if dropping the indexes can help here? now sure.
4.if i should think about partitioning , if there is any limitation while 
delaing with large data in postgreSQL?

regards
Mitu

_
___


-- 
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general


Re: [GENERAL] Delete trigger and data integrity

2014-05-27 Thread Alban Hertroys
On 27 May 2014 12:25, Yvonne Zannoun
 wrote:
> CREATE OR REPLACE FUNCTION delete_records()
> RETURNS TRIGGER AS $$
> BEGIN
> delete from "TABLE";
> RETURN NEW;
> END;
> $$
> LANGUAGE plpgsql;

You can't return NEW in an ON DELETE trigger - there is no NEW record.
Since you're going with a STATEMENT trigger instead, that's not really
relevant anymore (no NEW _or_ OLD record, since statements aren't
necessarily involved with single records), but I thought I'd mention
that slight oversight ;)

-- 
If you can't see the forest for the trees,
Cut the trees and you'll see there is no forest.


-- 
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general


Re: [GENERAL] Delete trigger and data integrity

2014-05-27 Thread Yvonne Zannoun
Thank you very much, that answers my question.

And yes, I think you are right with the FOR EACH ROW/STATEMENT, I didn't
think that through for this example.
Thank you for your help!

Kind regards,

*Yvonne Zannoun*
Graduate Technical Consultant
Snowflake Software

*Tel: +44 (0) 23 80238 232*

Email: yvonne.zann...@snowflakesoftware.com
Website: www.snowflakesoftware.com
Twitter: @sflakesoftware 
Follow us on LinkedIn 

Registered in England & Wales. Registered Number: 4294244
-



On 27 May 2014 11:44, Albe Laurenz  wrote:

> Yvonne Zannoun wrote:
> > I have this question regarding delete triggers and how it affects data
> integrity.
> > So here goes: I have this trigger which deletes everything before I
> insert new rows.
> >
> > CREATE OR REPLACE FUNCTION delete_records()
> > RETURNS TRIGGER AS $$
> > BEGIN
> > delete from "TABLE";
> > RETURN NEW;
> > END;
> > $$
> > LANGUAGE plpgsql;
> >
> > CREATE TRIGGER delete_on_insert
> > BEFORE INSERT ON "TABLE"
> > FOR EACH ROW EXECUTE PROCEDURE delete_records();
> >
> > My question is what happens while this function is executed? Is there
> any chance the table can return
> > empty data between the delete and insert commands? Or does a trigger
> like this block this possibility
> > somehow?
>
> Since the trigger has to run in the same transaction as the INSERT, no
> concurrent transaction will be able to see the "dirty" state between
> the DELETE and the INSERT.
>
> Are you sure that you want the trigger FOR EACH ROW and not FOR EACH
> STATEMENT?
> If the INSERT statement inserts more than one row, the trigger will run
> multiple
> times and you will end up with only one row in the table.
>
> Yours,
> Laurenz Albe
>

-- 
*Geospatial Technology Company of the Year*
*Read more *


Re: [GENERAL] Delete trigger and data integrity

2014-05-27 Thread Albe Laurenz
Yvonne Zannoun wrote:
> I have this question regarding delete triggers and how it affects data 
> integrity.
> So here goes: I have this trigger which deletes everything before I insert 
> new rows.
> 
> CREATE OR REPLACE FUNCTION delete_records()
> RETURNS TRIGGER AS $$
> BEGIN
> delete from "TABLE";
> RETURN NEW;
> END;
> $$
> LANGUAGE plpgsql;
> 
> CREATE TRIGGER delete_on_insert
> BEFORE INSERT ON "TABLE"
> FOR EACH ROW EXECUTE PROCEDURE delete_records();
> 
> My question is what happens while this function is executed? Is there any 
> chance the table can return
> empty data between the delete and insert commands? Or does a trigger like 
> this block this possibility
> somehow?

Since the trigger has to run in the same transaction as the INSERT, no
concurrent transaction will be able to see the "dirty" state between
the DELETE and the INSERT.

Are you sure that you want the trigger FOR EACH ROW and not FOR EACH STATEMENT?
If the INSERT statement inserts more than one row, the trigger will run multiple
times and you will end up with only one row in the table.

Yours,
Laurenz Albe

-- 
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general


[GENERAL] Delete trigger and data integrity

2014-05-27 Thread Yvonne Zannoun
Hello everyone,



I have this question regarding delete triggers and how it affects data
integrity.

So here goes: I have this trigger which deletes everything before I insert
new rows.



CREATE OR REPLACE FUNCTION delete_records()

RETURNS TRIGGER AS $$

BEGIN

delete from "TABLE";

RETURN NEW;

END;

$$

LANGUAGE plpgsql;



CREATE TRIGGER delete_on_insert

BEFORE INSERT ON "TABLE"

FOR EACH ROW EXECUTE PROCEDURE delete_records();



My question is what happens while this function is executed? Is there any
chance the table can return empty data between the delete and insert
commands? Or does a trigger like this block this possibility somehow?


Thank you

Kind regards,
*Yvonne Zannoun*

-- 
*Geospatial Technology Company of the Year*
*Read more *


Re: [GENERAL] Delete duplicate records with same fields

2013-08-29 Thread Hariraman Jayaraj
Hi,

For finding duplicates you can use ctid.

select o.ctid, o.a, o.b from test o
 where exists ( select 'x'
  from test i
 where i.a = o.a
   and i.b = o.b
   and i.ctid < o.ctid
 );

for deleting,
delete from test
 where exists ( select 'x'
  from test i
 where i.a = test.a
   and i.b = test.b
   and i.ctid < test.ctid
 );


On Thu, Aug 29, 2013 at 5:09 PM, Arun P.L  wrote:

> Hi all,
>
> Is there any way to delete duplicate rows in psql with no unique fields?
>  I have a set of old records with their duplicates in my db and they are
> all having  the same fields. How can I delete duplicates?
>
> Thanks in Advance,
> Arun
>



-- 
Hari


[GENERAL] Delete duplicate records with same fields

2013-08-29 Thread Arun P . L
Hi all, 
Is there any way to delete duplicate rows in psql with no unique fields?  I 
have a set of old records with their duplicates in my db and they are all 
having  the same fields. How can I delete duplicates? Thanks in Advance,Arun


[GENERAL] DELETE with LIMIT - workaround?

2013-06-27 Thread Chris Angelico
I have something that I think is a fairly common code model, but with
an SQL query that feels like it's fighting the system.

The 'cron' table has a number of tasks (one row = one task), and the
primary loop of the program (massively simplified) fetches one row,
processes it, commits. One row/task is
performed in one transaction, atomically, including the removal of the
row (so if something crashes out, the row remains and will be
processed later). The top of the loop therefore
needs to fetch one row, and delete it. If I were deleting *all* the
appropriate rows, it would be:

DELETE FROM cron WHERE tshttp://stackoverflow.com/questions/5170546/how-do-i-delete-a-fixed-number-of-rows-with-sorting-in-postgresql

DELETE FROM cron WHERE ctid=(SELECT ctid FROM cron WHERE tshttp://www.postgresql.org/message-id/26819.1291133...@sss.pgh.pa.us
and
surrounding), which is a pity because it does make good sense to do
this in a single pass rather than fetching some kind of unique
identifier and then re-locating by that. But is
the ctid somehow magical in being actually fast/simple enough to not
care about the difference?

Chris Angelico


-- 
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general


Re: [GENERAL] DELETE or TRUNCATE?

2013-05-16 Thread chiru r
Hi,

Yes,DELETE would be better this case.

The TRUNCATE operation required AccessExclusiveLock on Table before perform
TRUNCATE operation.

So,if you the table size is bing,it is batter to do  ANALYZE  after
report and VACUUM  non-peak(less business) hours.

Regards,
Chiru

On Thu, May 16, 2013 at 7:52 PM, François Beausoleil
wrote:

> Hi!
>
> I have a process that replaces the contents of a table. The canonical data
> store is somewhere else. At the moment, the import looks like this:
>
> CREATE TEMPORARY TABLE markets_import( LIKE x INCLUDING ALL );
> COPY markets_import FROM STDIN;
> ...
> \.
> -- COPY a bunch of other tables
>
> BEGIN;
>
> TRUNCATE markets;
> INSERT INTO markets SELECT * FROM markets_import;
> -- do the other tables here as well
>
> COMMIT;
>
> VACUUM ANALYZE markets;
>
> Sometimes, the import process will block on the TRUNCATE because some
> other query already holds a ShareLock on the markets table, because it's
> doing a long report. I'm guessing TRUNCATE prevents the use of MVCC, and
> DELETE would be better in this case? Especially since I'm doing a VACUUM
> ANALYZE at the end anyway.
>
> Thanks!
> François


[GENERAL] DELETE or TRUNCATE?

2013-05-16 Thread François Beausoleil
Hi!

I have a process that replaces the contents of a table. The canonical data 
store is somewhere else. At the moment, the import looks like this:

CREATE TEMPORARY TABLE markets_import( LIKE x INCLUDING ALL );
COPY markets_import FROM STDIN;
...
\.
-- COPY a bunch of other tables

BEGIN;

TRUNCATE markets;
INSERT INTO markets SELECT * FROM markets_import;
-- do the other tables here as well

COMMIT;

VACUUM ANALYZE markets;

Sometimes, the import process will block on the TRUNCATE because some other 
query already holds a ShareLock on the markets table, because it's doing a long 
report. I'm guessing TRUNCATE prevents the use of MVCC, and DELETE would be 
better in this case? Especially since I'm doing a VACUUM ANALYZE at the end 
anyway.

Thanks!
François

smime.p7s
Description: S/MIME cryptographic signature


Re: [PERFORM] [GENERAL] DELETE taking too much memory

2011-07-08 Thread Jose Ildefonso Camargo Tolosa
On Fri, Jul 8, 2011 at 4:35 AM, Dean Rasheed wrote:

> > On Thu, 2011-07-07 at 15:34 +0200, vincent dephily wrote:
> >> Hi,
> >>
> >> I have a delete query taking 7.2G of ram (and counting) but I do not
> >> understant why so much memory is necessary. The server has 12G, and
> >> I'm afraid it'll go into swap. Using postgres 8.3.14.
> >>
> >> I'm purging some old data from table t1, which should cascade-delete
> >> referencing rows in t2. Here's an anonymized rundown :
> >>
> >> # explain delete from t1 where t1id in (select t1id from t2 where
> >> foo=0 and bar < '20101101');
>
> It looks as though you're hitting one of the known issues with
> PostgreSQL and FKs. The FK constraint checks and CASCADE actions are
> implemented using AFTER triggers, which are queued up during the query
> to be executed at the end. For very large queries, this queue of
> pending triggers can become very large, using up all available memory.
>
> There's a TODO item to try to fix this for a future version of
> PostgreSQL (maybe I'll have another go at it for 9.2), but at the
> moment all versions of PostgreSQL suffer from this problem.
>
> The simplest work-around for you might be to break your deletes up
> into smaller chunks, say 100k or 1M rows at a time, eg:
>
> delete from t1 where t1id in (select t1id from t2 where foo=0 and bar
> < '20101101' limit 10);
>

I'd like to comment here I had serious performance issues with a similar
query (planner did horrible things), not sure if planner will do the same
dumb thing it did for me, my query was against the same table (ie, t1=t2).
I had this query:

delete from t1 where ctid in (select ctid from t1 where
created_at<'20101231' limit 1);   <--- this was slow.  Changed to:

delete from t1 where ctid = any(array(select ctid from t1 where
created_at<'20101231' limit 1));   <--- a lot faster.

So... will the same principle work here?, doing this?:

delete from t1 where t1id = any(array(select t1id from t2 where foo=0 and
bar
< '20101101' limit 10));  <-- would this query be faster then original
one?



>
> Regards,
> Dean
>
> --
> Sent via pgsql-performance mailing list (pgsql-performa...@postgresql.org)
> To make changes to your subscription:
> http://www.postgresql.org/mailpref/pgsql-performance
>


Re: [PERFORM] [GENERAL] DELETE taking too much memory

2011-07-08 Thread Claudio Freire
On Fri, Jul 8, 2011 at 12:48 PM, Dean Rasheed  wrote:
> Yes, it's the same issue that affects deferrable PK and FK
> constraints, but even non-deferrable FKs use AFTER ROW triggers that
> suffer from this problem. These triggers don't show up in a "\d" from
> psql, but they are there (try select * from pg_trigger where
> tgconstrrelid = 't1'::regclass) and because they fire AFTER rather
> than BEFORE, queuing up large numbers of them is a problem.

I would imagine an "easy" solution would be to "compress" the queue by
inserting a single element representing all rows of row version id X.

Ie: a delete or update will need to check all the row versions it
creates with its txid, this txid could be used to represent the rows
that need checking afterwards right?

-- 
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general


Re: [GENERAL] DELETE taking too much memory

2011-07-08 Thread Dean Rasheed
On 8 July 2011 10:44, Vincent de Phily
 wrote:
> On Friday 08 July 2011 10:05:47 Dean Rasheed wrote:
>> > On Thu, 2011-07-07 at 15:34 +0200, vincent dephily wrote:
>> >> Hi,
>> >>
>> >> I have a delete query taking 7.2G of ram (and counting) but I do not
>> >> understant why so much memory is necessary. The server has 12G, and
>> >> I'm afraid it'll go into swap. Using postgres 8.3.14.
>> >>
>> >> I'm purging some old data from table t1, which should cascade-delete
>> >> referencing rows in t2. Here's an anonymized rundown :
>> >>
>> >> # explain delete from t1 where t1id in (select t1id from t2 where
>> >> foo=0 and bar < '20101101');
>>
>> It looks as though you're hitting one of the known issues with
>> PostgreSQL and FKs. The FK constraint checks and CASCADE actions are
>> implemented using AFTER triggers, which are queued up during the query
>> to be executed at the end. For very large queries, this queue of
>> pending triggers can become very large, using up all available memory.
>>
>> There's a TODO item to try to fix this for a future version of
>> PostgreSQL (maybe I'll have another go at it for 9.2), but at the
>> moment all versions of PostgreSQL suffer from this problem.
>
> That's very interesting, and a more plausible not-optimized-yet item than my
> guesses so far, thanks. Drop me a mail if you work on this, and I'll find some
> time to test your code.
>
> I'm wondering though : this sounds like the behaviour of a "deferrable" fkey,
> which AFAICS is not the default and not my case ? I haven't explored that area
> of constraints yet, so there's certainly some detail that I'm missing.
>

Yes, it's the same issue that affects deferrable PK and FK
constraints, but even non-deferrable FKs use AFTER ROW triggers that
suffer from this problem. These triggers don't show up in a "\d" from
psql, but they are there (try select * from pg_trigger where
tgconstrrelid = 't1'::regclass) and because they fire AFTER rather
than BEFORE, queuing up large numbers of them is a problem.

Regards,
Dean


>
>> The simplest work-around for you might be to break your deletes up
>> into smaller chunks, say 100k or 1M rows at a time, eg:
>>
>> delete from t1 where t1id in (select t1id from t2 where foo=0 and bar
>> < '20101101' limit 10);
>
> Yes, that's what we ended up doing. We canceled the query after 24h, shortly
> before the OOM killer would have, and started doing things in smaller batches.
>
>
> --
> Vincent de Phily
>

-- 
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general


Re: [GENERAL] DELETE taking too much memory

2011-07-08 Thread Vincent de Phily
On Friday 08 July 2011 10:05:47 Dean Rasheed wrote:
> > On Thu, 2011-07-07 at 15:34 +0200, vincent dephily wrote:
> >> Hi,
> >> 
> >> I have a delete query taking 7.2G of ram (and counting) but I do not
> >> understant why so much memory is necessary. The server has 12G, and
> >> I'm afraid it'll go into swap. Using postgres 8.3.14.
> >> 
> >> I'm purging some old data from table t1, which should cascade-delete
> >> referencing rows in t2. Here's an anonymized rundown :
> >> 
> >> # explain delete from t1 where t1id in (select t1id from t2 where
> >> foo=0 and bar < '20101101');
> 
> It looks as though you're hitting one of the known issues with
> PostgreSQL and FKs. The FK constraint checks and CASCADE actions are
> implemented using AFTER triggers, which are queued up during the query
> to be executed at the end. For very large queries, this queue of
> pending triggers can become very large, using up all available memory.
> 
> There's a TODO item to try to fix this for a future version of
> PostgreSQL (maybe I'll have another go at it for 9.2), but at the
> moment all versions of PostgreSQL suffer from this problem.

That's very interesting, and a more plausible not-optimized-yet item than my 
guesses so far, thanks. Drop me a mail if you work on this, and I'll find some 
time to test your code.

I'm wondering though : this sounds like the behaviour of a "deferrable" fkey, 
which AFAICS is not the default and not my case ? I haven't explored that area 
of constraints yet, so there's certainly some detail that I'm missing.


> The simplest work-around for you might be to break your deletes up
> into smaller chunks, say 100k or 1M rows at a time, eg:
> 
> delete from t1 where t1id in (select t1id from t2 where foo=0 and bar
> < '20101101' limit 10);

Yes, that's what we ended up doing. We canceled the query after 24h, shortly 
before the OOM killer would have, and started doing things in smaller batches.


-- 
Vincent de Phily

-- 
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general


Re: [GENERAL] DELETE taking too much memory

2011-07-08 Thread Vincent de Phily
On Thursday 07 July 2011 22:26:45 Guillaume Lelarge wrote:
> On Thu, 2011-07-07 at 15:34 +0200, vincent dephily wrote:
> > Hi,
> > 
> > I have a delete query taking 7.2G of ram (and counting) but I do not
> > understant why so much memory is necessary. The server has 12G, and
> > I'm afraid it'll go into swap. Using postgres 8.3.14.
> > 
> > I'm purging some old data from table t1, which should cascade-delete
> > referencing rows in t2. Here's an anonymized rundown :
> > 
> > 
> > # \d t1
> > 
> >  Table
> >  "public.t1"
> >   
> >   Column   |Type | Modifiers
> > 
> > ---+-+--
> > ---
> > 
> >  t1id  | integer | not null default
> > 
> > nextval('t1_t1id_seq'::regclass)
> > (...snip...)
> > 
> > Indexes:
> > "message_pkey" PRIMARY KEY, btree (id)
> > 
> > (...snip...)
> > 
> > # \d t2
> > 
> >Table
> >"public.t
> >2"
> >  
> >  Column  |Type |Modifiers
> > 
> > -+-+
> > -
> > 
> >  t2id| integer | not null default
> > 
> > nextval('t2_t2id_seq'::regclass)
> > 
> >  t1id| integer | not null
> >  foo | integer | not null
> >  bar | timestamp without time zone | not null default now()
> > 
> > Indexes:
> > "t2_pkey" PRIMARY KEY, btree (t2id)
> > "t2_bar_key" btree (bar)
> > "t2_t1id_key" btree (t1id)
> > 
> > Foreign-key constraints:
> > "t2_t1id_fkey" FOREIGN KEY (t1id) REFERENCES t1(t1id) ON UPDATE
> > 
> > RESTRICT ON DELETE CASCADE
> > 
> > # explain delete from t1 where t1id in (select t1id from t2 where
> > foo=0 and bar < '20101101');
> > 
> >QUERY PLAN
> > 
> > 
> > -
> > 
> >  Nested Loop  (cost=5088742.39..6705282.32 rows=30849 width=6)
> >  
> >->  HashAggregate  (cost=5088742.39..5089050.88 rows=30849
> >width=4)
> >
> >  ->  Index Scan using t2_bar_key on t2 
> >  (cost=0.00..5035501.50
> > 
> > rows=21296354 width=4)
> > 
> >Index Cond: (bar < '2010-11-01
> >00:00:00'::timestamp
> > 
> > without time zone)
> > 
> >Filter: (foo = 0)
> >
> >->  Index Scan using t1_pkey on t1  (cost=0.00..52.38 rows=1
> >width=10)
> >
> >  Index Cond: (t1.t1id = t2.t1id)
> > 
> > (7 rows)
> > 
> > 
> > Note that the estimate of 30849 rows is way off : there should be
> > around 55M rows deleted from t1, and 2-3 times as much from t2.
> > 
> > When looking at the plan, I can easily imagine that data gets
> > accumulated below the nestedloop (thus using all that memory), but why
> > isn't each entry freed once one row has been deleted from t1 ? That
> > entry isn't going to be found again in t1 or in t2, so why keep it
> > around ?
> > 
> > Is there a better way to write this query ? Would postgres 8.4/9.0
> > handle things better ?
> 
> Do you have any DELETE triggers in t1 and/or t2?

No, there are triggers on insert/update to t1 which both insert into t2, but 
no delete trigger. Deletions do cascade from t1 to t2 because of the foreign 
key.
-- 
Vincent de Phily
Mobile Devices
+33 (0) 142 119 325
+353 (0) 85 710 6320 

Warning
This message (and any associated files) is intended only for the use of its
intended recipient and may contain information that is confidential, subject
to copyright or constitutes a trade secret. If you are not the intended
recipient you are hereby notified that any dissemination, copying or
distribution of this message, or files associated with this message, is
strictly prohibited. If you have received this message in error, please
notify us immediately by replying to the message and deleting it from your
computer. Any views or opinions presented are solely those of the author
vincent.deph...@mobile-devices.fr and do not necessarily represent those of 
the
company. Although the company has taken reasonable precautions to ensure no
viruses are present in this email, the company cannot accept responsibility
for any loss or damage arising from the use of this email or attachments.

-- 
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general


Re: [GENERAL] DELETE taking too much memory

2011-07-08 Thread Dean Rasheed
> On Thu, 2011-07-07 at 15:34 +0200, vincent dephily wrote:
>> Hi,
>>
>> I have a delete query taking 7.2G of ram (and counting) but I do not
>> understant why so much memory is necessary. The server has 12G, and
>> I'm afraid it'll go into swap. Using postgres 8.3.14.
>>
>> I'm purging some old data from table t1, which should cascade-delete
>> referencing rows in t2. Here's an anonymized rundown :
>>
>> # explain delete from t1 where t1id in (select t1id from t2 where
>> foo=0 and bar < '20101101');

It looks as though you're hitting one of the known issues with
PostgreSQL and FKs. The FK constraint checks and CASCADE actions are
implemented using AFTER triggers, which are queued up during the query
to be executed at the end. For very large queries, this queue of
pending triggers can become very large, using up all available memory.

There's a TODO item to try to fix this for a future version of
PostgreSQL (maybe I'll have another go at it for 9.2), but at the
moment all versions of PostgreSQL suffer from this problem.

The simplest work-around for you might be to break your deletes up
into smaller chunks, say 100k or 1M rows at a time, eg:

delete from t1 where t1id in (select t1id from t2 where foo=0 and bar
< '20101101' limit 10);

Regards,
Dean

-- 
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general


Re: [GENERAL] DELETE taking too much memory

2011-07-07 Thread Guillaume Lelarge
On Thu, 2011-07-07 at 15:34 +0200, vincent dephily wrote:
> Hi,
> 
> I have a delete query taking 7.2G of ram (and counting) but I do not
> understant why so much memory is necessary. The server has 12G, and
> I'm afraid it'll go into swap. Using postgres 8.3.14.
> 
> I'm purging some old data from table t1, which should cascade-delete
> referencing rows in t2. Here's an anonymized rundown :
> 
> 
> # \d t1
>  Table "public.t1"
>   Column   |Type | Modifiers
> ---+-+-
>  t1id  | integer | not null default
> nextval('t1_t1id_seq'::regclass)
> (...snip...)
> Indexes:
> "message_pkey" PRIMARY KEY, btree (id)
> (...snip...)
> 
> # \d t2
>Table "public.t2"
>  Column  |Type |Modifiers
> -+-+-
>  t2id| integer | not null default
> nextval('t2_t2id_seq'::regclass)
>  t1id| integer | not null
>  foo | integer | not null
>  bar | timestamp without time zone | not null default now()
> Indexes:
> "t2_pkey" PRIMARY KEY, btree (t2id)
> "t2_bar_key" btree (bar)
> "t2_t1id_key" btree (t1id)
> Foreign-key constraints:
> "t2_t1id_fkey" FOREIGN KEY (t1id) REFERENCES t1(t1id) ON UPDATE
> RESTRICT ON DELETE CASCADE
> 
> # explain delete from t1 where t1id in (select t1id from t2 where
> foo=0 and bar < '20101101');
>QUERY PLAN
> -
>  Nested Loop  (cost=5088742.39..6705282.32 rows=30849 width=6)
>->  HashAggregate  (cost=5088742.39..5089050.88 rows=30849 width=4)
>  ->  Index Scan using t2_bar_key on t2  (cost=0.00..5035501.50
> rows=21296354 width=4)
>Index Cond: (bar < '2010-11-01 00:00:00'::timestamp
> without time zone)
>Filter: (foo = 0)
>->  Index Scan using t1_pkey on t1  (cost=0.00..52.38 rows=1 width=10)
>  Index Cond: (t1.t1id = t2.t1id)
> (7 rows)
> 
> 
> Note that the estimate of 30849 rows is way off : there should be
> around 55M rows deleted from t1, and 2-3 times as much from t2.
> 
> When looking at the plan, I can easily imagine that data gets
> accumulated below the nestedloop (thus using all that memory), but why
> isn't each entry freed once one row has been deleted from t1 ? That
> entry isn't going to be found again in t1 or in t2, so why keep it
> around ?
> 
> Is there a better way to write this query ? Would postgres 8.4/9.0
> handle things better ?
> 

Do you have any DELETE triggers in t1 and/or t2?


-- 
Guillaume
  http://blog.guillaume.lelarge.info
  http://www.dalibo.com


-- 
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general


[GENERAL] DELETE taking too much memory

2011-07-07 Thread vincent dephily
Hi,

I have a delete query taking 7.2G of ram (and counting) but I do not
understant why so much memory is necessary. The server has 12G, and
I'm afraid it'll go into swap. Using postgres 8.3.14.

I'm purging some old data from table t1, which should cascade-delete
referencing rows in t2. Here's an anonymized rundown :


# \d t1
 Table "public.t1"
  Column   |Type | Modifiers
---+-+-
 t1id  | integer | not null default
nextval('t1_t1id_seq'::regclass)
(...snip...)
Indexes:
"message_pkey" PRIMARY KEY, btree (id)
(...snip...)

# \d t2
   Table "public.t2"
 Column  |Type |Modifiers
-+-+-
 t2id| integer | not null default
nextval('t2_t2id_seq'::regclass)
 t1id| integer | not null
 foo | integer | not null
 bar | timestamp without time zone | not null default now()
Indexes:
"t2_pkey" PRIMARY KEY, btree (t2id)
"t2_bar_key" btree (bar)
"t2_t1id_key" btree (t1id)
Foreign-key constraints:
"t2_t1id_fkey" FOREIGN KEY (t1id) REFERENCES t1(t1id) ON UPDATE
RESTRICT ON DELETE CASCADE

# explain delete from t1 where t1id in (select t1id from t2 where
foo=0 and bar < '20101101');
   QUERY PLAN
-
 Nested Loop  (cost=5088742.39..6705282.32 rows=30849 width=6)
   ->  HashAggregate  (cost=5088742.39..5089050.88 rows=30849 width=4)
 ->  Index Scan using t2_bar_key on t2  (cost=0.00..5035501.50
rows=21296354 width=4)
   Index Cond: (bar < '2010-11-01 00:00:00'::timestamp
without time zone)
   Filter: (foo = 0)
   ->  Index Scan using t1_pkey on t1  (cost=0.00..52.38 rows=1 width=10)
 Index Cond: (t1.t1id = t2.t1id)
(7 rows)


Note that the estimate of 30849 rows is way off : there should be
around 55M rows deleted from t1, and 2-3 times as much from t2.

When looking at the plan, I can easily imagine that data gets
accumulated below the nestedloop (thus using all that memory), but why
isn't each entry freed once one row has been deleted from t1 ? That
entry isn't going to be found again in t1 or in t2, so why keep it
around ?

Is there a better way to write this query ? Would postgres 8.4/9.0
handle things better ?



Thanks in advance.


-- 
Vincent de Phily

-- 
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general


Re: [GENERAL] Delete Trigger

2010-08-14 Thread Tom Lane
Chris Berry  writes:
> When I update, insert or delete to one table, I need a trigger to delete and
> possibly insert into another table. The trigger I have works for insert and
> update, but returns an error when I do a delete.

There's no "NEW" row in a delete trigger (... and the error message
should have told you that pretty explicitly ...).  You can look at the
OLD row instead.

BTW, you should probably think carefully about what the UPDATE case
is doing and whether it needs to look at NEW or OLD or a combination.

regards, tom lane

-- 
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general


[GENERAL] Delete Trigger

2010-08-14 Thread Chris Berry
When I update, insert or delete to one table, I need a trigger to delete and
possibly insert into another table. The trigger I have works for insert and
update, but returns an error when I do a delete.

The is no explicit link between the tables, so I can't do 'DELETE CASCADE'
or anything like that.

Here is the trigger:

CREATE OR REPLACE FUNCTION event_tweet() RETURNS TRIGGER AS
$textetweet_textetweet$
DECLARE
new_id varchar;
new_date date;
BEGIN
DELETE FROM textetweet_textetweet ;
IF(TG_OP='INSERT') THEN
DELETE FROM textetweet_textetweet WHERE link='/ievent/' || NEW.id;
INSERT INTO textetweet_textetweet (date,title,content,link,creation) values
(NEW.date,'new event','new event details','/ievent/' || NEW.id, NOW());
END IF;
IF(TG_OP='UPDATE') THEN
DELETE FROM textetweet_textetweet WHERE link='/ievent/' || NEW.id;
INSERT INTO textetweet_textetweet (date,title,content,link,creation) values
(NEW.date,'new event','new event details','/ievent/' || NEW.id, NOW());
END IF;
IF(TG_OP='DELETE') THEN
DELETE FROM textetweet_textetweet WHERE link='/ievent/' || NEW.id;
END IF;

The problem seems to be the 'NEW.id'. How do I get the deleted id for the
trigger?

Thanks a lot


Re: [GENERAL] delete query taking way too long

2010-08-12 Thread Ivan Sergio Borgonovo
On Thu, 12 Aug 2010 12:50:49 +0100
Thom Brown  wrote:

> On 12 August 2010 12:14, Ivan Sergio Borgonovo
>  wrote:
> > I've
> > delete from catalog_items where ItemID in (select id from
> > import.Articoli_delete);
> >
> > id and ItemID have an index.
> >
> > catalog_items is ~1M rows
> > Articoli_delete is less than 2K rows.
> >
> > This query has been running for roughly 50min.
> > Right now it is the only query running.
> >
> > PostgreSQL 8.3.4 on x86_64-pc-linux-gnu, compiled by GCC cc (GCC)
> > 4.1.2 20061115 (prerelease) (Debian 4.1.1-21)
> >
> > --
> You can try to do deletes in batches of 10,000:

ItemID is a PK.
Even if there is no constraint on Articoli_delete.id they *should*
be unique.

There are some other tables with indexes... but I don't expect that
more than 4K row for each table will be deleted.
There are a couple of other large (1M rows) table where an on delete
cascade is defined.

This is the query plan
Nested Loop  (cost=30.07..10757.29 rows=1766 width=6)
 ->  HashAggregate  (cost=30.07..47.73 rows=1766 width=8)
   ->  Seq Scan on articoli_delete  (cost=0.00..25.66 rows=1766
   width=8)
 ->  Index Scan using catalog_items_pkey on catalog_items
  (cost=0.00..6.05 rows=1 width=14)
Index Cond: (catalog_items.itemid = articoli_delete.id)

BTW it is happening again... after I stopped pg, restarted the whole
server and re-run the query.

This query get generally unnoticed in a longer process but I doubt
it ever lasted more than a couple of minutes in the past.

-- 
Ivan Sergio Borgonovo
http://www.webthatworks.it


-- 
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general


Re: [GENERAL] delete query taking way too long

2010-08-12 Thread Ivan Sergio Borgonovo
On Thu, 12 Aug 2010 17:14:17 +0530
Jayadevan M  wrote:

> > I've
> > delete from catalog_items where ItemID in (select id from
> > import.Articoli_delete);
> Does catalog_items have child tables where the FK columns are not
> indexed? Regards,

Possibly, but very small ones.

What I missed to say is... that query always worked reasonably fast
in the past. The size of the DB didn't grow substantially recently.

I'd say the query shouldn't be the problem... the question should
have been:
what should I look when postgresql start to behave strangely?
eg. missing resources, locks, solved bug (it is a reasonably old
version)...

-- 
Ivan Sergio Borgonovo
http://www.webthatworks.it


-- 
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general


Re: [GENERAL] delete query taking way too long

2010-08-12 Thread tv
> I've
> delete from catalog_items where ItemID in (select id from
> import.Articoli_delete);
>
> id and ItemID have an index.
>
> catalog_items is ~1M rows
> Articoli_delete is less than 2K rows.
>
> This query has been running for roughly 50min.
> Right now it is the only query running.
>
> PostgreSQL 8.3.4 on x86_64-pc-linux-gnu, compiled by GCC cc (GCC)
> 4.1.2 20061115 (prerelease) (Debian 4.1.1-21)

1) Run the query with EXPLAIN, e.g. something like EXPLAIN DELETE FROM ...

This won't actually execute the query, it will just prepare an execution
plan and print it. That might show some problems with the query.

Post the output of explain to explain.depesz.com and then send link to the
forum (you could post the explain output here, but it's difficult to
read).

2) Run ANALYZE on the tables involved in the query and then the EXPLIAIN
again (this might show some problems with obsolete statistics).

3) Are there any tables depending on the "catalog_items" table? I mean are
there any foreign keys referencing it through a foreign key? How large are
those tables? Are the FK columns indexed?

regards
Tomas


-- 
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general


Re: [GENERAL] delete query taking way too long

2010-08-12 Thread Thom Brown
On 12 August 2010 12:14, Ivan Sergio Borgonovo  wrote:
> I've
> delete from catalog_items where ItemID in (select id from
> import.Articoli_delete);
>
> id and ItemID have an index.
>
> catalog_items is ~1M rows
> Articoli_delete is less than 2K rows.
>
> This query has been running for roughly 50min.
> Right now it is the only query running.
>
> PostgreSQL 8.3.4 on x86_64-pc-linux-gnu, compiled by GCC cc (GCC)
> 4.1.2 20061115 (prerelease) (Debian 4.1.1-21)
>
> --
You can try to do deletes in batches of 10,000:

DELETE FROM catalog_items WHERE ItemID IN (SELECT id FROM
import.Articoli_delete LIMIT 1);

But an EXPLAIN would tell us a lot more.

-- 
Thom Brown
Registered Linux user: #516935

-- 
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general


Re: [GENERAL] delete query taking way too long

2010-08-12 Thread Jayadevan M
> From: Ivan Sergio Borgonovo 
> To: pgsql-general@postgresql.org
> Date: 12/08/2010 16:43
> Subject: [GENERAL] delete query taking way too long
> Sent by: pgsql-general-ow...@postgresql.org
> 
> I've
> delete from catalog_items where ItemID in (select id from
> import.Articoli_delete);
Does catalog_items have child tables where the FK columns are not indexed?
Regards,
Jayadevan





DISCLAIMER: 

"The information in this e-mail and any attachment is intended only for 
the person to whom it is addressed and may contain confidential and/or 
privileged material. If you have received this e-mail in error, kindly 
contact the sender and destroy all copies of the original communication. 
IBS makes no warranty, express or implied, nor guarantees the accuracy, 
adequacy or completeness of the information contained in this email or any 
attachment and is not liable for any errors, defects, omissions, viruses 
or for resultant loss or damage, if any, direct or indirect."






-- 
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general


Re: [GENERAL] delete query taking way too long

2010-08-12 Thread Glyn Astill
What's the output of explain?

--- On Thu, 12/8/10, Ivan Sergio Borgonovo  wrote:

> From: Ivan Sergio Borgonovo 
> Subject: [GENERAL] delete query taking way too long
> To: pgsql-general@postgresql.org
> Date: Thursday, 12 August, 2010, 12:14
> I've
> delete from catalog_items where ItemID in (select id from
> import.Articoli_delete);
> 
> id and ItemID have an index.
> 
> catalog_items is ~1M rows
> Articoli_delete is less than 2K rows.
> 
> This query has been running for roughly 50min.
> Right now it is the only query running.
> 
> PostgreSQL 8.3.4 on x86_64-pc-linux-gnu, compiled by GCC cc
> (GCC)
> 4.1.2 20061115 (prerelease) (Debian 4.1.1-21)
> 
> -- 
> Ivan Sergio Borgonovo
> http://www.webthatworks.it
> 
> 
> -- 
> Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
> To make changes to your subscription:
> http://www.postgresql.org/mailpref/pgsql-general
> 




-- 
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general


  1   2   3   4   >