Re: Partition Check not updated when insert into a partition

2021-10-15 Thread Amit Langote
On Wed, Oct 6, 2021 at 10:40 PM Pavel Borisov wrote: > > Hi, hackers! > > We've reviewed patch v3 and found it right. Completely agree that in case we > attach/detach partition relcaches for all child relations (if they exist) > need invalidation. > Installcheck world succeeds after the patch. T

installcheck fails when compute_query_id=on or pg_stat_statsement is loaded

2021-10-15 Thread Мельников Антон Андреевич
Hello!   When extension  pg_stat_statsement is loaded into memory or compute_query_id=on in postgesql.conf many of installcheck tests give errors. The thing is that in *.out files appear lines "queryid = x" where x - some numeric value. So 24 of 209 installcheck tests will fail. It seems

Re: [PATCH] Proposal for HIDDEN/INVISIBLE column

2021-10-15 Thread Aleksander Alekseev
Hi hackers, > > Just to remind here, there was recently a proposal to handle this > > problem another way - provide a list of columns to skip for "star > > selection" aka "SELECT * EXCEPT col1...". > > > > https://postgrespro.com/list/id/d51371a2-f221-1cf3-4a7d-b2242d4da...@gmail.com > > [...] > >

Re: Partition Check not updated when insert into a partition

2021-10-15 Thread Pavel Borisov
> > The lock taken on the parent is either ShareUpdateExclusiveLock or > AccessExclusiveLock depending on whether CONCURRENTLY is specified or > not. Maybe that should be considered also when locking the children. > > I've updated the patch that way. (Also, reintroduced the slightly > longer comm

Re: [PATCH] Proposal for HIDDEN/INVISIBLE column

2021-10-15 Thread Gilles Darold
Le 15/10/2021 à 09:47, Aleksander Alekseev a écrit : Just to remind here, there was recently a proposal to handle this problem another way - provide a list of columns to skip for "star selection" aka "SELECT * EXCEPT col1...". https://postgrespro.com/list/id/d51371a2-f221-1cf3-4a7d-b2242d4da..

Re: installcheck fails when compute_query_id=on or pg_stat_statsement is loaded

2021-10-15 Thread Julien Rouhaud
On Fri, Oct 15, 2021 at 3:36 PM Мельников Антон Андреевич wrote: > > When extension pg_stat_statsement is loaded into memory or > compute_query_id=on in postgesql.conf > many of installcheck tests give errors. > [...] > I think this is a problem because these fake errors can mask a real errors i

Re: [PATCH] Proposal for HIDDEN/INVISIBLE column

2021-10-15 Thread Aleksander Alekseev
Hi Gilles, > I can turn the column hidden and I will not have to modify my old very good application. I see your point. At the same time, I believe the statement above shows the root reason why we have a different view on this feature. The application should have never use SELECT * in the first p

Re: Multi-Column List Partitioning

2021-10-15 Thread Amit Langote
Hi Rajkumar, On Mon, Oct 11, 2021 at 2:36 PM Rajkumar Raghuwanshi wrote: > > Thanks for the patch, it applied cleanly and fixed the reported issue. I > observed another case where > In case of multi-col list partition on the same column query is not picking > partition wise join. Is this expec

RE: Allow escape in application_name

2021-10-15 Thread kuroda.hay...@fujitsu.com
Dear Horiguchi-san, > I'm not sure. All of it is a if-story. z/OS's isdigit is defined as > "Test for a decimal digit, as defined in the digit locale source file > and in the digit class of the LC_CTYPE category of the current > locale.", which I read it as "it accepts multibyte strings" but of >

Re: [PATCH] Added TRANSFORM FOR for COMMENT tab completion

2021-10-15 Thread Ken Kato
2021-10-15 13:29 に Shinya Kato さんは書きました: On 2021-10-14 14:30, katouknl wrote: It is very good, but it seems to me that there are some tab-completion missing in COMMENT command. For example, - CONSTRAINT ... ON DOMAIN - OPERATOR CLASS - OPERATOR FAMILY - POLICY ... ON - [PROCEDURAL] - RULE ... O

Re: [PATCH] Proposal for HIDDEN/INVISIBLE column

2021-10-15 Thread Gilles Darold
Le 15/10/2021 à 10:37, Aleksander Alekseev a écrit : Hi Gilles, > I can turn the column hidden and I will not have to modify my old very good application. I see your point. At the same time, I believe the statement above shows the root reason why we have a different view on this feature. Th

Re: [PATCH] Proposal for HIDDEN/INVISIBLE column

2021-10-15 Thread Laurenz Albe
On Thu, 2021-10-14 at 13:16 +0200, Gilles Darold wrote: > Here is a proposal to implement HIDDEN columns feature in PostgreSQL. > > The user defined columns are always visible in the PostgreSQL. If user > wants to hide some column(s) from a SELECT * returned values then the > hidden columns featur

Re: RFC: Logging plan of the running query

2021-10-15 Thread torikoshia
On 2021-10-15 15:17, torikoshia wrote: I only took a quick look at pg_query_state, I have some questions. pg_query_state seems using shm_mq to expose the plan information, but there was a discussion that this kind of architecture would be tricky to do properly [1]. Does pg_query_state handle dif

Data is copied twice when specifying both child and parent table in publication

2021-10-15 Thread houzj.f...@fujitsu.com
Hi, In another logical replication related thread[1], my colleague Greg found that if publish_via_partition_root is true, then the child table's data will be copied twice when adding both child and parent table to the publication. Example: - Pub: create table tbl1 (a int) partition by range

Re: Unbounded %s in sscanf

2021-10-15 Thread Daniel Gustafsson
> On 30 Jul 2021, at 18:03, Tom Lane wrote: > > Daniel Gustafsson writes: >> I took another look at this today, and propose to push the attached. The >> pg_dump fix goes all the way back to 9.6 whereas the pg_basebackup fix is >> from >> 11 and onwards. The adjacent shadowed variable bug in p

Re: refactoring basebackup.c

2021-10-15 Thread Jeevan Ladhe
Hi Robert, > The loop is gone, but CHUNK_SIZE itself seems to have evaded the executioner. I am sorry, but I did not really get it. Or it is what you have pointed in the following paragraphs? > I think this is not the best way to accomplish the goal. Adding > LZ4F_compressBound(0) to next_buf_le

Re: [PATCH] test/ssl: rework the sslfiles Makefile target

2021-10-15 Thread Daniel Gustafsson
> On 1 Oct 2021, at 09:02, Daniel Gustafsson wrote: > >> On 1 Oct 2021, at 08:59, Michael Paquier wrote: >> >> On Wed, Sep 15, 2021 at 12:31:31AM +0200, Daniel Gustafsson wrote: >>> Correct. In my head, "rebuild" is when dealing with individually changed >>> files >>> and "recreate" means reb

Re: refactoring basebackup.c

2021-10-15 Thread Robert Haas
On Fri, Oct 15, 2021 at 7:54 AM Jeevan Ladhe wrote: > > The loop is gone, but CHUNK_SIZE itself seems to have evaded the > > executioner. > > I am sorry, but I did not really get it. Or it is what you have pointed > in the following paragraphs? I mean #define CHUNK_SIZE is still in the patch. >

Re: [PATCH] Proposal for HIDDEN/INVISIBLE column

2021-10-15 Thread Aleksander Alekseev
Hi Gilles, > But we have more and more application coming from others RDBMS with sometime > no real possibility to modify the code or which requires lot of work. Somehow I feel everyone here very well understood the real motivation behind this proposal from the beginning, considering the e-mail o

Re: [Proposal] Global temporary tables

2021-10-15 Thread Andrew Bille
On master with the v55 patches applied the following script leads to crash: initdb -D data pg_ctl -w -t 5 -D data -l server.log start psql -t -c "begin; create global temp table gtt_with_index(a int primary key, b text); commit; select pg_sleep(5);" >psql1.log & psql -t -c "select pg_sleep(1); cre

Re: should we allow users with a predefined role to access pg_backend_memory_contexts view and pg_log_backend_memory_contexts function?

2021-10-15 Thread Robert Haas
On Thu, Oct 14, 2021 at 3:02 PM Jeff Davis wrote: > How do you feel about at least allowing the functions to execute (and > if it's SECURITY INVOKER, possibly encountering a permissions failure > during execution)? I think we'd at least need to check that the view owner has execute permission on

Partial aggregates pushdown

2021-10-15 Thread Alexander Pyhalov
Hi. One of the issues when we try to use sharding in PostgreSQL is absence of partial aggregates pushdown. I see several opportunities to alleviate this issue. If we look at Citus, it implements aggregate, calculating internal state of an arbitrary agregate function and exporting it as text.

Re: [PATCH] Proposal for HIDDEN/INVISIBLE column

2021-10-15 Thread Gilles Darold
Le 15/10/2021 à 14:24, Aleksander Alekseev a écrit : Hi Gilles, If we really want SELECT * to be reserved to DBA then why not removing the star from PG unless you have the admin privilege? Respectfully, I perceive this as a trolling (presumably, non-intentional one) and not going to answer thi

Re: [PATCH] Proposal for HIDDEN/INVISIBLE column

2021-10-15 Thread Dave Cramer
On Fri, 15 Oct 2021 at 09:29, Gilles Darold wrote: > Le 15/10/2021 à 14:24, Aleksander Alekseev a écrit : > > Hi Gilles, > > > >> If we really want SELECT * to be reserved to DBA then why not removing > the > >> star from PG unless you have the admin privilege? > > Respectfully, I perceive this a

Re: installcheck fails when compute_query_id=on or pg_stat_statsement is loaded

2021-10-15 Thread Tom Lane
Julien Rouhaud writes: > ... Note that you don't really need to enable > pg_stat_statements, enabling compute_query_id is enough. The query > identifier is only displayed for EXPLAIN (VERBOSE), so it's already a > bit filtered. I don't see any simple way to entirely avoid the > problem though.

Re: Unbounded %s in sscanf

2021-10-15 Thread Tom Lane
Daniel Gustafsson writes: > I propose to apply the attached all the way down (with the basebackup hunk > from > 11), or down to 10 if we want to be conservative with the final 9.6 re ancient > bugs that haven't seen complaints. LGTM. No objection to applying this in 9.6.

Re: BUG #17212: pg_amcheck fails on checking temporary relations

2021-10-15 Thread Andrew Dunstan
On 10/14/21 5:52 PM, Tom Lane wrote: > Andrew Dunstan writes: >> Yes, that's been puzzling me too. I've just been staring at it again and >> nothing jumps out. But maybe we can investigate that offline if this >> test is deemed not worth keeping. > As Mark says, it'd be interesting to know wheth

Re: Partial aggregates pushdown

2021-10-15 Thread Tomas Vondra
Hi Alexander, On 10/15/21 15:15, Alexander Pyhalov wrote: Hi. One of the issues when we try to use sharding in PostgreSQL is absence of partial aggregates pushdown. I see several opportunities to alleviate this issue. If we look at Citus, it implements aggregate, calculating internal state

Re: Partial aggregates pushdown

2021-10-15 Thread Alexander Pyhalov
Tomas Vondra писал 2021-10-15 17:56: Hi Alexander, Hi. And then we should extend this for aggregates with more complex internal states (e.g. avg), by supporting a function that "exports" the aggregate state - similar to serial/deserial functions, but needs to be portable. I think the tricki

Re: Partial aggregates pushdown

2021-10-15 Thread Tomas Vondra
On 10/15/21 17:05, Alexander Pyhalov wrote: Tomas Vondra писал 2021-10-15 17:56: Hi Alexander, Hi. And then we should extend this for aggregates with more complex internal states (e.g. avg), by supporting a function that "exports" the aggregate state - similar to serial/deserial functions,

Re: [PATCH] Proposal for HIDDEN/INVISIBLE column

2021-10-15 Thread Aleksander Alekseev
Hi Gilles, > Yes, I don't wanted to offend you or to troll. This was just to point > that the position of "SELECT * is bad practice" is not a good argument > in my point of view, just because it is allowed for every one. I mean > that in an extension or a client which allow user query input we mus

Doc patch

2021-10-15 Thread rir
In pgsql-docs, this patch has been recommended to you. Lacking consensus and so not included is the the deletion of comments pointing between the ref/MOVE and FETCH files. These were of the form: Thanks for the software, Rob >From 711a3299851cde9ce00b5ff2962f20cdc1796e72 Mon Sep 17 00:00

Trivial doc patch

2021-10-15 Thread rir
This removes the outer square brackets in the create_database.sgml file's synopsis. In the command sysopses, this is the only case where an optional group contains only optional groups. Rob >From dc59127d1a739e6de0cff20086baf47d15837f0b Mon Sep 17 00:00:00 2001 From: rir Date: Fri, 15 Oct 2021

Re: should we allow users with a predefined role to access pg_backend_memory_contexts view and pg_log_backend_memory_contexts function?

2021-10-15 Thread Jeff Davis
On Fri, 2021-10-15 at 09:08 -0400, Robert Haas wrote: > I think we'd at least need to check that the view owner has execute > permission on the function. I'm not sure whether there are any other > gotchas. Right, like we do for tables in a view now. The alternative is not very appealing: that we

Re: [PATCH] Proposal for HIDDEN/INVISIBLE column

2021-10-15 Thread Julien Rouhaud
On Fri, Oct 15, 2021 at 9:40 PM Dave Cramer wrote: > > On Fri, 15 Oct 2021 at 09:29, Gilles Darold wrote: >> >> Yes, I don't wanted to offend you or to troll. This was just to point >> that the position of "SELECT * is bad practice" is not a good argument >> in my point of view, just because it i

Re: should we allow users with a predefined role to access pg_backend_memory_contexts view and pg_log_backend_memory_contexts function?

2021-10-15 Thread Stephen Frost
Greetings, * Jeff Davis (pg...@j-davis.com) wrote: > On Fri, 2021-10-15 at 09:08 -0400, Robert Haas wrote: > > I think we'd at least need to check that the view owner has execute > > permission on the function. I'm not sure whether there are any other > > gotchas. > > Right, like we do for tables

Re: should we allow users with a predefined role to access pg_backend_memory_contexts view and pg_log_backend_memory_contexts function?

2021-10-15 Thread Jeff Davis
On Fri, 2021-10-15 at 13:52 -0400, Stephen Frost wrote: > While I appreciate that > it > isn't really addressing the complaint you have that it'd be nice if > we > made things simpler for administrators by making it so they don't > have > to GRANT access to both the view and the function, and I can

Re: [RFC] building postgres with meson

2021-10-15 Thread Andres Freund
Hi, On 2021-10-14 19:23:58 -0400, Tom Lane wrote: > Andres Freund writes: > > If, and perhaps that's too big an if, relative rpaths actually work despite > > SIP, it might be worth setting a relative install_rpath, because afaict that > > should then work both for a "real" installation and our te

Re: [PATCH] Proposal for HIDDEN/INVISIBLE column

2021-10-15 Thread Bruce Momjian
On Fri, Oct 15, 2021 at 11:32:53AM +0200, Laurenz Albe wrote: > On Thu, 2021-10-14 at 13:16 +0200, Gilles Darold wrote: > > Here is a proposal to implement HIDDEN columns feature in PostgreSQL. > > > > The user defined columns are always visible in the PostgreSQL. If user > > wants to hide some co

Re: XTS cipher mode for cluster file encryption

2021-10-15 Thread Stephen Frost
Greetings, * Bruce Momjian (br...@momjian.us) wrote: > As you might have seen from my email in another thread, thanks to > Stephen and Cybertec staff, I am back working on cluster file > encryption/TDE. > > Stephen was going to research if XTS cipher mode would be a good fit for > this since it w

Re: Partial aggregates pushdown

2021-10-15 Thread Stephen Frost
Greetings, * Tomas Vondra (tomas.von...@enterprisedb.com) wrote: > On 10/15/21 17:05, Alexander Pyhalov wrote: > >Tomas Vondra писал 2021-10-15 17:56: > >>And then we should extend this for aggregates with more complex > >>internal states (e.g. avg), by supporting a function that "exports" > >>the

Re: [PATCH] Proposal for HIDDEN/INVISIBLE column

2021-10-15 Thread Andrew Dunstan
On 10/15/21 2:51 PM, Bruce Momjian wrote: > On Fri, Oct 15, 2021 at 11:32:53AM +0200, Laurenz Albe wrote: >> On Thu, 2021-10-14 at 13:16 +0200, Gilles Darold wrote: >>> Here is a proposal to implement HIDDEN columns feature in PostgreSQL. >>> >>> The user defined columns are always visible in the

Re: XTS cipher mode for cluster file encryption

2021-10-15 Thread Tomas Vondra
On 10/15/21 21:22, Stephen Frost wrote: Greetings, * Bruce Momjian (br...@momjian.us) wrote: As you might have seen from my email in another thread, thanks to Stephen and Cybertec staff, I am back working on cluster file encryption/TDE. Stephen was going to research if XTS cipher mode would

Re: XTS cipher mode for cluster file encryption

2021-10-15 Thread Robert Haas
On Fri, Oct 15, 2021 at 3:22 PM Stephen Frost wrote: > Specifically: The default cipher for LUKS is nowadays aes-xts-plain64 > > and then this: > > https://gitlab.com/cryptsetup/cryptsetup/-/wikis/DMCrypt > > where plain64 is defined as: > > plain64: the initial vector is the 64-bit little-endian

Re: XTS cipher mode for cluster file encryption

2021-10-15 Thread Andres Freund
Hi, On 2021-10-15 15:22:48 -0400, Stephen Frost wrote: > * Bruce Momjian (br...@momjian.us) wrote: > > Finally, there is an interesting web page about when not to use XTS: > > > > https://sockpuppet.org/blog/2014/04/30/you-dont-want-xts/ > > This particular article always struck me as more o

Re: XTS cipher mode for cluster file encryption

2021-10-15 Thread Tomas Vondra
On 10/15/21 23:02, Robert Haas wrote: On Fri, Oct 15, 2021 at 3:22 PM Stephen Frost wrote: Specifically: The default cipher for LUKS is nowadays aes-xts-plain64 and then this: https://gitlab.com/cryptsetup/cryptsetup/-/wikis/DMCrypt where plain64 is defined as: plain64: the initial vecto

Re: [PATCH] Proposal for HIDDEN/INVISIBLE column

2021-10-15 Thread Gilles Darold
Le 15/10/2021 à 21:52, Andrew Dunstan a écrit : On 10/15/21 2:51 PM, Bruce Momjian wrote: On Fri, Oct 15, 2021 at 11:32:53AM +0200, Laurenz Albe wrote: On Thu, 2021-10-14 at 13:16 +0200, Gilles Darold wrote: Here is a proposal to implement HIDDEN columns feature in PostgreSQL. The user define

Re: [PATCH] Proof of concept for GUC improvements

2021-10-15 Thread Cary Huang
The following review has been posted through the commitfest application: make installcheck-world: tested, passed Implements feature: tested, passed Spec compliant: tested, passed Documentation:not tested Hi I quite like the feature this patch provides, it makes specia

Re: Partial aggregates pushdown

2021-10-15 Thread Tomas Vondra
On 10/15/21 21:31, Stephen Frost wrote: Greetings, * Tomas Vondra (tomas.von...@enterprisedb.com) wrote: On 10/15/21 17:05, Alexander Pyhalov wrote: Tomas Vondra писал 2021-10-15 17:56: And then we should extend this for aggregates with more complex internal states (e.g. avg), by supporting a

Re: [RFC] building postgres with meson

2021-10-15 Thread Andres Freund
Hi, On 2021-10-15 11:50:30 -0700, Andres Freund wrote: > It seems to work quite well to change our own references to libpq in binaries > / shared libs to be relative, but to leave the install_name of the libraries > intact. In combination with adding an rpath of @loader_path/../lib/ to > binaries

Re: [RFC] building postgres with meson

2021-10-15 Thread Andres Freund
Hi, On 2021-10-15 15:36:16 -0700, Andres Freund wrote: > On 2021-10-15 11:50:30 -0700, Andres Freund wrote: > > It seems to work quite well to change our own references to libpq in > > binaries > > / shared libs to be relative, but to leave the install_name of the libraries > > intact. In combina

pg_dump fails to lock partitioned tables

2021-10-15 Thread Tom Lane
While fooling with something else, I happened to notice $SUBJECT. The reason turns out to be that it's checking the wrong element of the tblinfo[] array; see one-liner fix attached. I had a feeling of deja vu about this bug, and indeed a dig in the git history shows that we fixed it in passing in

Re: Trivial doc patch

2021-10-15 Thread Michael Paquier
On Fri, Oct 15, 2021 at 01:13:14PM -0400, rir wrote: > This removes the outer square brackets in the create_database.sgml > file's synopsis. In the command sysopses, this is the only case > where an optional group contains only optional groups. > > CREATE DATABASE name > -[ [ WITH ] [ OWNER [

Re: Reset snapshot export state on the transaction abort

2021-10-15 Thread Michael Paquier
On Thu, Oct 14, 2021 at 02:58:55PM +0530, Dilip Kumar wrote: > On Thu, Oct 14, 2021 at 12:24 PM Michael Paquier wrote: >> Yes, you are right here. I did not remember the semantics this relies >> on. I have played more with the patch, reviewed the whole, and the >> fields you are resetting as par

RE: Data is copied twice when specifying both child and parent table in publication

2021-10-15 Thread houzj.f...@fujitsu.com
On Friday, October 15, 2021 7:23 PM houzj.f...@fujitsu.com wrote: > Attach a patch to fix it. Attach a new version patch which refactor the fix code in a cleaner way. Best regards, Hou zj v2-0001-fix-double-publish.patch Description: v2-0001-fix-double-publish.patch

Re: [PATCH] Proposal for HIDDEN/INVISIBLE column

2021-10-15 Thread Gilles Darold
Le 15/10/2021 à 18:42, Aleksander Alekseev a écrit : > Hi Gilles, > >> Yes, I don't wanted to offend you or to troll. This was just to point >> that the position of "SELECT * is bad practice" is not a good argument >> in my point of view, just because it is allowed for every one. I mean >> that in