Re: Minimal logical decoding on standbys

2020-01-16 Thread Andreas Joseph Krogh
På torsdag 16. januar 2020 kl. 05:42:24, skrev Amit Khandekar < amitdkhan...@gmail.com >: On Fri, 10 Jan 2020 at 17:50, Rahila Syed wrote: > > Hi Amit, > > Can you please rebase the patches as they don't apply on latest master? Thanks for notifying. Attached

Re: Making psql error out on output failures

2020-01-16 Thread David Zhang
On 2020-01-16 5:20 a.m., Daniel Verite wrote: David Zhang wrote: If I change the error log message like below, where "%m" is used to pass the value of strerror(errno), "could not write to output file:" is copied from function "WRITE_ERROR_EXIT". -

Re: [HACKERS] Block level parallel vacuum

2020-01-16 Thread Dilip Kumar
On Fri, Jan 17, 2020 at 11:39 AM Dilip Kumar wrote: I have performed cost delay testing on the latest test(I have used same script as attahced in [1] and [2]. vacuum_cost_delay = 10 vacuum_cost_limit = 2000 Observation: As we have concluded earlier, the delay time is in sync with the I/O

Re: Add pg_file_sync() to adminpack

2020-01-16 Thread Fujii Masao
On 2020/01/13 22:46, Michael Paquier wrote: On Sat, Jan 11, 2020 at 02:12:15AM +0900, Fujii Masao wrote: I'm not sure if returning false with WARNING only in some error cases is really good idea or not. At least for me, it's more intuitive to return true on success and emit an ERROR

Re: Add pg_file_sync() to adminpack

2020-01-16 Thread Fujii Masao
On 2020/01/17 13:36, Michael Paquier wrote: On Thu, Jan 16, 2020 at 09:51:24AM -0500, Robert Haas wrote: On Tue, Jan 14, 2020 at 10:08 AM Atsushi Torikoshi wrote: fails we can get error messages. So it seems straightforward for me to 'return true on success and emit an ERROR otherwise'.

Re: Batch insert in CTAS/MatView code

2020-01-16 Thread Paul Guo
I took some time on digging the issue yesterday so the main concern of the patch is to get the tuple length from ExecFetchSlotHeapTuple(). + ExecCopySlot(batchslot, slot); + tuple = ExecFetchSlotHeapTuple(batchslot, true, NULL); + + myState->mi_slots_num++; + myState->mi_slots_size +=

Re: SQL/JSON: functions

2020-01-16 Thread Pavel Stehule
Hi čt 14. 11. 2019 v 17:46 odesílatel Nikita Glukhov napsal: > Attached 40th version of the patches. > > > I have added some documentation which has been simply copied from [1]. > > Also, I have split patches more correctly, and extracted patch #2 with common > SQL/JSON clauses, that are used

Re: FETCH FIRST clause PERCENT option

2020-01-16 Thread Kyotaro Horiguchi
Thank you for the notification, Tomas. At Thu, 16 Jan 2020 22:33:58 +0100, Tomas Vondra wrote in > This patch is marked as RFC since September. Since then there was no > discussion on this thread, but Andrew proposed an alternative approach > based on window functions in a separate thread [1]

Re: Errors when update a view with conditional-INSTEAD rules

2020-01-16 Thread Pengzhou Tang
Thanks a lot, Dean, to look into this and also sorry for the late reply. On Sun, Jan 5, 2020 at 12:08 AM Dean Rasheed wrote: > Tracing it through, this seems to be a result of > cab5dc5daf2f6f5da0ce79deb399633b4bb443b5 which added support for > updatable views with a mix of updatable and

Re: [HACKERS] Block level parallel vacuum

2020-01-16 Thread Dilip Kumar
On Fri, Jan 17, 2020 at 11:34 AM Amit Kapila wrote: > > On Fri, Jan 17, 2020 at 11:00 AM Dilip Kumar wrote: > > > > On Fri, Jan 17, 2020 at 10:44 AM Amit Kapila > > wrote: > > > > > > On Fri, Jan 17, 2020 at 9:36 AM Dilip Kumar wrote: > > > > > > > > I have few small comments. > > > > > > > >

Re: [HACKERS] Block level parallel vacuum

2020-01-16 Thread Amit Kapila
On Fri, Jan 17, 2020 at 11:00 AM Dilip Kumar wrote: > > On Fri, Jan 17, 2020 at 10:44 AM Amit Kapila wrote: > > > > On Fri, Jan 17, 2020 at 9:36 AM Dilip Kumar wrote: > > > > > > I have few small comments. > > > > > > 1. > > > logical streaming for large in-progress transactions+ > > > + /*

Re: Unnecessary delay in streaming replication due to replay lag

2020-01-16 Thread Michael Paquier
On Fri, Jan 17, 2020 at 09:34:05AM +0530, Asim R P wrote: > Standby does not start walreceiver process until startup process > finishes WAL replay. The more WAL there is to replay, longer is the > delay in starting streaming replication. If replication connection is > temporarily disconnected,

Re: [HACKERS] Block level parallel vacuum

2020-01-16 Thread Dilip Kumar
On Fri, Jan 17, 2020 at 10:44 AM Amit Kapila wrote: > > On Fri, Jan 17, 2020 at 9:36 AM Dilip Kumar wrote: > > > > I have few small comments. > > > > 1. > > logical streaming for large in-progress transactions+ > > + /* Can't perform vacuum in parallel */ > > + if (parallel_workers <= 0) > > + {

Re: [HACKERS] Block level parallel vacuum

2020-01-16 Thread Amit Kapila
On Fri, Jan 17, 2020 at 9:36 AM Dilip Kumar wrote: > > I have few small comments. > > 1. > logical streaming for large in-progress transactions+ > + /* Can't perform vacuum in parallel */ > + if (parallel_workers <= 0) > + { > + pfree(can_parallel_vacuum); > + return lps; > + } > > why are we

Re: PATCH: standby crashed when replay block which truncated in standby but failed to truncate in master node

2020-01-16 Thread Michael Paquier
On Thu, Jan 16, 2020 at 11:17:36PM +0900, Fujii Masao wrote: > OK, I updated the patch that way. > Attached is the updated version of the patch. Thanks. I have few tweaks to propose to the docs. +raise a PANIC-level error, aborting the recovery. Setting Instead of "PANIC-level error", I

Re: Add pg_file_sync() to adminpack

2020-01-16 Thread Michael Paquier
On Thu, Jan 16, 2020 at 09:51:24AM -0500, Robert Haas wrote: > On Tue, Jan 14, 2020 at 10:08 AM Atsushi Torikoshi wrote: >> fails we can get error messages. So it seems straightforward for me to >> 'return true on success and emit an ERROR otherwise'. > > I agree with reporting errors using

Re: [HACKERS] Block level parallel vacuum

2020-01-16 Thread Dilip Kumar
On Fri, Jan 17, 2020 at 9:36 AM Dilip Kumar wrote: > > On Thu, Jan 16, 2020 at 5:34 PM Amit Kapila wrote: > > > > On Thu, Jan 16, 2020 at 4:46 PM Masahiko Sawada > > wrote: > > > > > > Right. Most indexes (all?) of tables that are used in the regression > > > tests are smaller than

Re: [HACKERS] Block level parallel vacuum

2020-01-16 Thread Dilip Kumar
On Thu, Jan 16, 2020 at 5:34 PM Amit Kapila wrote: > > On Thu, Jan 16, 2020 at 4:46 PM Masahiko Sawada > wrote: > > > > Right. Most indexes (all?) of tables that are used in the regression > > tests are smaller than min_parallel_index_scan_size. And we set > > min_parallel_index_scan_size to 0

Unnecessary delay in streaming replication due to replay lag

2020-01-16 Thread Asim R P
Hi Standby does not start walreceiver process until startup process finishes WAL replay. The more WAL there is to replay, longer is the delay in starting streaming replication. If replication connection is temporarily disconnected, this delay becomes a major problem and we are proposing a

Re: Reorderbuffer crash during recovery

2020-01-16 Thread Dilip Kumar
On Fri, Jan 17, 2020 at 7:42 AM vignesh C wrote: > > On Thu, Jan 16, 2020 at 9:17 AM Dilip Kumar wrote: > > > > One minor comment. Otherwise, the patch looks fine to me. > > + /* > > + * We set final_lsn on a transaction when we decode its commit or abort > > + * record, but we never see those

Re: pgindent && weirdness

2020-01-16 Thread Tom Lane
Alvaro Herrera writes: > On 2020-Jan-16, Tom Lane wrote: >> ... But I was hoping to >> hear Piotr's opinion before moving forward. > FWIW I think this code predates Piotr's involvement, I think; at least, > it was already there in the FreeBSD code he imported: >

Re: pgindent && weirdness

2020-01-16 Thread Alvaro Herrera
On 2020-Jan-16, Tom Lane wrote: > Alvaro Herrera writes: > > On 2020-Jan-17, Michael Paquier wrote: > >> Nice find! Could you commit that? I can see many places improved as > >> well, among explain.c, tablecmds.c, typecmds.c, and much more. > > > I think Tom is the only one who can commit

Re: pgindent && weirdness

2020-01-16 Thread Tom Lane
Alvaro Herrera writes: > On 2020-Jan-17, Michael Paquier wrote: >> Nice find! Could you commit that? I can see many places improved as >> well, among explain.c, tablecmds.c, typecmds.c, and much more. > I think Tom is the only one who can commit that, >

Re: Run-time pruning for ModifyTable

2020-01-16 Thread Michael Paquier
On Thu, Jan 16, 2020 at 10:45:25PM +0100, Tomas Vondra wrote: > David, this patch is marked as "waiting on author" since 11/27, and > there have been no updates or responses since then. Do you plan to > submit a new patch version in this CF? We're already half-way through, > so there's not much

Re: Setting min/max TLS protocol in clientside libpq

2020-01-16 Thread Michael Paquier
On Fri, Jan 17, 2020 at 10:09:54AM +0900, Michael Paquier wrote: > Could you please rebase and fix the remaining pieces of the patch? And while I remember, you may want to add checks for incorrect bounds when validating the values in fe-connect.c... The same arguments as for the backend part

Re: pgindent && weirdness

2020-01-16 Thread Alvaro Herrera
On 2020-Jan-17, Michael Paquier wrote: > On Thu, Jan 16, 2020 at 06:13:36PM -0300, Alvaro Herrera wrote: > > This is indeed a very good fix! Several badly formatted sites in our > > code are improved with this change. > > Nice find! Could you commit that? I can see many places improved as >

Re: Reorderbuffer crash during recovery

2020-01-16 Thread vignesh C
On Thu, Jan 16, 2020 at 9:17 AM Dilip Kumar wrote: > > One minor comment. Otherwise, the patch looks fine to me. > + /* > + * We set final_lsn on a transaction when we decode its commit or abort > + * record, but we never see those records for crashed transactions. To > + * ensure cleanup of

Re: making the backend's json parser work in frontend code

2020-01-16 Thread Andrew Dunstan
On Thu, Jan 16, 2020 at 7:33 AM Robert Haas wrote: > > > 0002 does some basic header cleanup to make it possible to include the > existing header file jsonapi.h in frontend code. The state of the JSON > headers today looks generally poor. There seems not to have been much > attempt to get the

Re: pgindent && weirdness

2020-01-16 Thread Michael Paquier
On Thu, Jan 16, 2020 at 06:13:36PM -0300, Alvaro Herrera wrote: > This is indeed a very good fix! Several badly formatted sites in our > code are improved with this change. Nice find! Could you commit that? I can see many places improved as well, among explain.c, tablecmds.c, typecmds.c, and

Re: Amcheck: do rightlink verification with lock coupling

2020-01-16 Thread Peter Geoghegan
On Mon, Jan 13, 2020 at 8:47 PM Andrey Borodin wrote: > > 11 янв. 2020 г., в 7:49, Peter Geoghegan написал(а): > > I'm curious why Andrey's corruption problems were not detected by the > > cross-page amcheck test, though. We compare the first non-pivot tuple > > on the right sibling leaf page

Re: Setting min/max TLS protocol in clientside libpq

2020-01-16 Thread Michael Paquier
On Thu, Jan 16, 2020 at 09:56:01AM +0100, Daniel Gustafsson wrote: > The patch looks fine to me, I don't an issue with splitting it into a > refactoring patch and a TLS min/max version patch. Thanks, committed the refactoring part then. If the buildfarm breaks for a reason or another, the window

Re: Verify true root on replicas with amcheck

2020-01-16 Thread Peter Geoghegan
On Thu, Jan 9, 2020 at 12:55 AM godjan • wrote: > Hi, we have trouble to detect true root corruptions on replicas. I made a > patch for resolving it with the locking meta page and potential root page. What do you mean by true root corruption? What is the cause of the problem? What symptom does

Re: row filtering for logical replication

2020-01-16 Thread Euler Taveira
Em qui., 16 de jan. de 2020 às 18:57, Tomas Vondra escreveu: > > Euler, this patch is still in "waiting on author" since 11/25. Do you > plan to review changes made by Amit in the patches he submitted, or what > are your plans with this patch? > Yes, I'm working on Amit suggestions. I'll post a

Re: making the backend's json parser work in frontend code

2020-01-16 Thread Tom Lane
Robert Haas writes: > It's possible that we're chasing a real problem here, and if there's > something we can agree on and get done I'd rather do that than argue, > but I am still quite suspicious that there's no actually serious > technical problem here. It's entirely possible that you're

Re: Improve search for missing parent downlinks in amcheck

2020-01-16 Thread Tomas Vondra
On Fri, Nov 29, 2019 at 03:03:01PM +0900, Michael Paquier wrote: On Mon, Aug 19, 2019 at 01:15:19AM +0300, Alexander Korotkov wrote: The revised patch seems to fix all of above. The latest patch is failing to apply. Please provide a rebase. This still does not apply (per cputube). Can you

Re: [PATCH] ltree, lquery, and ltxtquery binary protocol support

2020-01-16 Thread Tomas Vondra
On Fri, Nov 29, 2019 at 11:29:03AM +0900, Michael Paquier wrote: On Mon, Nov 11, 2019 at 03:44:54PM +0100, Nino Floris wrote: Alright, as usual life got in the way. I've attached a new version of the patch with pgindent changes. > What do you think about writing patch for ALTER TYPE? I'd

Re: row filtering for logical replication

2020-01-16 Thread Tomas Vondra
On Thu, Nov 28, 2019 at 11:32:01AM +0900, Michael Paquier wrote: On Mon, Nov 25, 2019 at 11:48:29AM +0900, Amit Langote wrote: On Mon, Nov 25, 2019 at 11:38 AM Amit Langote wrote: Needed to be rebased, which I did, to be able to test them; patches attached. Oops, really attached this time.

Re: Run-time pruning for ModifyTable

2020-01-16 Thread Tomas Vondra
On Wed, Nov 27, 2019 at 05:17:06PM +0900, Michael Paquier wrote: On Tue, Nov 05, 2019 at 04:04:25PM +1300, Thomas Munro wrote: On Thu, Sep 12, 2019 at 10:10 AM Alvaro Herrera wrote: > Here's a rebased version of this patch (it had a trivial conflict). Hi, FYI partition_prune.sql currently

Re: remove separate postgres.(sh)description files

2020-01-16 Thread Tomas Vondra
On Wed, Jan 08, 2020 at 02:33:23PM +0200, Heikki Linnakangas wrote: On 31/12/2019 02:08, John Naylor wrote: I'm guessing the initial data for pg_(sh)description is output into separate files because it was too difficult for the traditional shell script to maintain enough state to do otherwise.

Re: Recognizing superuser in pg_hba.conf

2020-01-16 Thread Tomas Vondra
Hi, I see this patch is marked as RFC since 12/30, but there seems to be quite a lot of discussion about the syntax, keywords and how exactly to identify the superuser. So I'll switch it back to needs review, which I think is a better representation of the current state. regards -- Tomas

Re: psql - add SHOW_ALL_RESULTS option

2020-01-16 Thread Fabien COELHO
My own vote would be to use the initial patch (after applying any unrelated changes per later review), ie. add the feature with its disable button. I can do that, but not if there is a veto from Tom on the feature. I wish definite negative opinions by senior committers would be expressed

Re: FETCH FIRST clause PERCENT option

2020-01-16 Thread Tomas Vondra
Hi, This patch is marked as RFC since September. Since then there was no discussion on this thread, but Andrew proposed an alternative approach based on window functions in a separate thread [1] (both for this and for the WITH TIES case). I'll set this patch back to "needs review" - at this

Re: psql - add SHOW_ALL_RESULTS option

2020-01-16 Thread Fabien COELHO
Hello Tom, This is one of the patches already marked as RFC (since September by Alvaro). Anyone interested in actually pushing it, so that it does not fall through to yet another commitfest? TBH, I think we'd be better off to reject it. This makes a nontrivial change in a very

Re: making the backend's json parser work in frontend code

2020-01-16 Thread Tom Lane
Robert Haas writes: > On Thu, Jan 16, 2020 at 3:11 PM Tom Lane wrote: >> Here's a reviewed version of 0001. You missed fixing the MSVC build, >> and there were assorted comments and other things referencing wchar.c >> that needed to be cleaned up. > Wow, thanks. Pushed that. >> Another thing

Re: pgindent && weirdness

2020-01-16 Thread Alvaro Herrera
On 2020-Jan-17, Thomas Munro wrote: > On Thu, Jan 16, 2020 at 3:59 PM Thomas Munro wrote: > > On Wed, Jan 15, 2020 at 11:30 AM Tom Lane wrote: > > > Yeah, it's been doing that for decades. I think the triggering > > > factor is the typedef name (Var, here) preceding the &&. > > Here's a

Re: psql - add SHOW_ALL_RESULTS option

2020-01-16 Thread Alvaro Herrera
On 2020-Jan-16, Tom Lane wrote: > The compatibility issue could be resolved by putting in the option > that I suppose was there at the beginning. But then we'd have to > have a debate about which behavior would be default, and there would > still be the question of who would find this to be an

Re: pgindent && weirdness

2020-01-16 Thread Thomas Munro
On Thu, Jan 16, 2020 at 3:59 PM Thomas Munro wrote: > On Wed, Jan 15, 2020 at 11:30 AM Tom Lane wrote: > > Yeah, it's been doing that for decades. I think the triggering > > factor is the typedef name (Var, here) preceding the &&. Here's a better fix: diff --git a/indent.c b/indent.c index

Re: psql - add SHOW_ALL_RESULTS option

2020-01-16 Thread Tomas Vondra
On Thu, Jan 16, 2020 at 01:08:16PM -0500, Tom Lane wrote: Tomas Vondra writes: This is one of the patches already marked as RFC (since September by Alvaro). Anyone interested in actually pushing it, so that it does not fall through to yet another commitfest? TBH, I think we'd be better off

Re: making the backend's json parser work in frontend code

2020-01-16 Thread Robert Haas
On Thu, Jan 16, 2020 at 1:58 PM David Steele wrote: > To do page-level incrementals (which this feature is intended to enable) > the user will need to be able to associate full and incremental backups > and the only way I see to do that (currently) is to read the manifests, > since the prior

Re: our checks for read-only queries are not great

2020-01-16 Thread Robert Haas
On Thu, Jan 16, 2020 at 12:22 PM Stephen Frost wrote: > I think I agree with you regarding the original intent, though even > there, as discussed elsewhere, it seems like there's perhaps either a > bug or a disagreement about the specifics of what that means when it > relates to committing a

Re: making the backend's json parser work in frontend code

2020-01-16 Thread Robert Haas
On Thu, Jan 16, 2020 at 3:11 PM Tom Lane wrote: > Robert Haas writes: > > 0001 moves wchar.c from src/backend/utils/mb to src/common. Unless I'm > > missing something, this seems like an overdue cleanup. > > Here's a reviewed version of 0001. You missed fixing the MSVC build, > and there were

Re: SlabCheck leaks memory into TopMemoryContext

2020-01-16 Thread Tomas Vondra
On Thu, Jan 16, 2020 at 03:15:41PM -0500, Tom Lane wrote: Tomas Vondra writes: I think the one possible argument against this approach might be that it adds a field to the struct, so if you have an extension using a Slab context, it'll break if you don't rebuild it. But that only matters if we

Re: SlabCheck leaks memory into TopMemoryContext

2020-01-16 Thread Tom Lane
Tomas Vondra writes: > I think the one possible argument against this approach might be that it > adds a field to the struct, so if you have an extension using a Slab > context, it'll break if you don't rebuild it. But that only matters if > we want to backpatch it (which I think is not the plan)

Re: making the backend's json parser work in frontend code

2020-01-16 Thread Tom Lane
Robert Haas writes: > 0001 moves wchar.c from src/backend/utils/mb to src/common. Unless I'm > missing something, this seems like an overdue cleanup. Here's a reviewed version of 0001. You missed fixing the MSVC build, and there were assorted comments and other things referencing wchar.c that

Re: making the backend's json parser work in frontend code

2020-01-16 Thread David Steele
On 1/16/20 12:26 PM, Andres Freund wrote: Hi, On 2020-01-16 14:20:28 -0500, Tom Lane wrote: David Steele writes: The way we handle this in pgBackRest is to put a TRY ... CATCH block in main() to log and exit on any uncaught THROW. That seems like a reasonable way to start here. Without

Re: Enabling B-Tree deduplication by default

2020-01-16 Thread Peter Geoghegan
On Thu, Jan 16, 2020 at 10:55 AM Robert Haas wrote: > On Wed, Jan 15, 2020 at 6:38 PM Peter Geoghegan wrote: > > There are some outstanding questions about how B-Tree deduplication > > [1] should be configured, and whether or not it should be enabled by > > default. I'm starting this new thread

Re: SlabCheck leaks memory into TopMemoryContext

2020-01-16 Thread Tomas Vondra
On Thu, Jan 16, 2020 at 12:33:03PM -0500, Tom Lane wrote: Tomas Vondra writes: On Thu, Jan 16, 2020 at 08:48:49AM -0800, Andres Freund wrote: I don't get why it's advantageous to allocate this once for each slab, rather than having it as a global once for all slabs. But anyway, still clearly

Re: Patch to document base64 encoding

2020-01-16 Thread Alvaro Herrera
I just wanted to throw this in the archives; this doesn't need to affect your patch. Because of how the new tables look in the PDF docs, I thought it might be a good time to research how to make each function-entry occupy two rows: one for prototype, return type and description, and the other for

Re: making the backend's json parser work in frontend code

2020-01-16 Thread Andres Freund
Hi, On 2020-01-16 14:20:28 -0500, Tom Lane wrote: > David Steele writes: > > The way we handle this in pgBackRest is to put a TRY ... CATCH block in > > main() to log and exit on any uncaught THROW. That seems like a > > reasonable way to start here. Without memory contexts that almost > >

Re: making the backend's json parser work in frontend code

2020-01-16 Thread Tom Lane
David Steele writes: > On 1/15/20 7:39 PM, Robert Haas wrote: >>> I agree that it's far from obvious that the hacks in the patch are >>> best; to the contrary, they are hacks. That said, I feel that the >>> semantics of throwing an error are not very well-defined in a >>> front-end environment. I

Re: making the backend's json parser work in frontend code

2020-01-16 Thread David Steele
On 1/15/20 7:39 PM, Robert Haas wrote: > On Wed, Jan 15, 2020 at 6:40 PM Andres Freund wrote: >> It's not obvious why the better approach here wouldn't be to just have a >> very simple ereport replacement, that needs to be explicitly included >> from frontend code. It'd not be meaningfully

Re: making the backend's json parser work in frontend code

2020-01-16 Thread David Steele
On 1/15/20 4:40 PM, Andres Freund wrote: > > I'm not sure where I come down between using json and a simple ad-hoc > format, when the dependency for the former is making the existing json > parser work in the frontend. But if the alternative is to add a second > json parser, it very clearly

Re: Enabling B-Tree deduplication by default

2020-01-16 Thread Robert Haas
On Wed, Jan 15, 2020 at 6:38 PM Peter Geoghegan wrote: > There are some outstanding questions about how B-Tree deduplication > [1] should be configured, and whether or not it should be enabled by > default. I'm starting this new thread in the hopes of generating > discussion on these high level

Re: making the backend's json parser work in frontend code

2020-01-16 Thread Robert Haas
On Thu, Jan 16, 2020 at 1:37 PM David Steele wrote: > I was starting to wonder if it wouldn't be simpler to go back to the > Postgres JSON parser and see if we can adapt it. I'm not sure that it > *is* simpler, but it would almost certainly be more acceptable. That is my feeling also. > So the

Re: Thoughts on "killed tuples" index hint bits support on standby

2020-01-16 Thread Peter Geoghegan
On Thu, Jan 16, 2020 at 9:54 AM Andres Freund wrote: > I don't think we can rely on hot_standby_feedback at all. We can to > avoid unnecessary cancellations, etc, and even assume it's setup up > reasonably for some configurations, but there always needs to be an > independent correctness

Re: making the backend's json parser work in frontend code

2020-01-16 Thread David Steele
Hi Robert, On 1/16/20 11:37 AM, David Steele wrote: The next question in my mind is given the caveat that the error handing is questionable in the front end, can we at least render/parse valid JSON with the code? Hrm, this bit was from an earlier edit. I meant: The next question in my

Re: making the backend's json parser work in frontend code

2020-01-16 Thread David Steele
Hi Robert, On 1/15/20 2:02 PM, Robert Haas wrote: > The discussion on the backup manifest thread has gotten bogged down on > the issue of the format that should be used to store the backup > manifest file. I want something simple and ad-hoc; David Steele and > Stephen Frost prefer JSON. That is

Re: psql - add SHOW_ALL_RESULTS option

2020-01-16 Thread Tom Lane
Tomas Vondra writes: > This is one of the patches already marked as RFC (since September by > Alvaro). Anyone interested in actually pushing it, so that it does not > fall through to yet another commitfest? TBH, I think we'd be better off to reject it. This makes a nontrivial change in a very

Re: SlabCheck leaks memory into TopMemoryContext

2020-01-16 Thread Andres Freund
Hi, On 2020-01-16 18:01:53 +0100, Tomas Vondra wrote: > Plus we don't have all that many slabs (like 2) FWIW, I have a local patch that adds additional ones, for the relcache and catcache, that's how I noticed the leak. Because a test pgbench absolutely *tanked* in performance. Just for

Re: Thoughts on "killed tuples" index hint bits support on standby

2020-01-16 Thread Andres Freund
Hi, On 2020-01-16 14:30:12 +0300, Michail Nikolaev wrote: > First thing we need to consider - checksums and wal_log_hints are > widely used these days. So, at any moment master could send FPW page > with new "killed tuples" hints and overwrite hints set by standby. > Moreover it is not possible

Re: psql - add SHOW_ALL_RESULTS option

2020-01-16 Thread Tomas Vondra
Hi, This is one of the patches already marked as RFC (since September by Alvaro). Anyone interested in actually pushing it, so that it does not fall through to yet another commitfest? regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support,

Re: SlabCheck leaks memory into TopMemoryContext

2020-01-16 Thread Tom Lane
Tomas Vondra writes: > On Thu, Jan 16, 2020 at 08:48:49AM -0800, Andres Freund wrote: >> I don't get why it's advantageous to allocate this once for each slab, >> rather than having it as a global once for all slabs. But anyway, still >> clearly better than the current situation. > It's largely

Re: our checks for read-only queries are not great

2020-01-16 Thread Stephen Frost
Greetings, * Robert Haas (robertmh...@gmail.com) wrote: > On Tue, Jan 14, 2020 at 1:46 PM Stephen Frost wrote: > > * Tom Lane (t...@sss.pgh.pa.us) wrote: > > > Robert Haas writes: > > > > Speaking of sensible progress, I think we've drifted off on a tangent > > > > here about ALTER SYSTEM. > >

Re: our checks for read-only queries are not great

2020-01-16 Thread Robert Haas
On Tue, Jan 14, 2020 at 1:46 PM Stephen Frost wrote: > * Tom Lane (t...@sss.pgh.pa.us) wrote: > > Robert Haas writes: > > > Speaking of sensible progress, I think we've drifted off on a tangent > > > here about ALTER SYSTEM. > > > > Agreed, that's not terribly relevant for the proposed patch. >

Re: Physical replication slot advance is not persistent

2020-01-16 Thread Alexey Kondratov
On 09.01.2020 09:36, Kyotaro Horiguchi wrote: Hello. At Sun, 29 Dec 2019 15:12:16 +0300, Alexey Kondratov wrote in On 2019-12-26 16:35, Alexey Kondratov wrote: Another concern is that ReplicationSlotIsDirty is added with the only one user. It also cannot be used by SaveSlotToPath due to the

Re: SlabCheck leaks memory into TopMemoryContext

2020-01-16 Thread Tomas Vondra
On Thu, Jan 16, 2020 at 11:43:34AM -0500, Tom Lane wrote: Tomas Vondra writes: The attached fix should do the trick - it pre-allocates the space when creating the context. There is a bit of complexity because we want to allocate the space as part of the context header, but nothin too bad. We

Re: Code cleanup for build_regexp_split_result

2020-01-16 Thread Tom Lane
Li Japin writes: > I find the build_regexp_split_result() has redundant codes, we can move it to > before the condition check, can we? Hm, yeah, that looks a bit strange. It was less strange before c8ea87e4bd950572cba4575e9a62284cebf85ac5, I think. Pushed with some additional simplification

Re: SlabCheck leaks memory into TopMemoryContext

2020-01-16 Thread Tomas Vondra
On Thu, Jan 16, 2020 at 10:27:01AM -0500, Tom Lane wrote: Andres Freund writes: ... I thought you were asking whether any additional memory could just be avoided... Well, I was kind of wondering that, but if it's not practical then preallocating the space instead will do. I don't think

Re: SlabCheck leaks memory into TopMemoryContext

2020-01-16 Thread Tom Lane
Andres Freund writes: > ... I thought you were asking whether > any additional memory could just be avoided... Well, I was kind of wondering that, but if it's not practical then preallocating the space instead will do. regards, tom lane

Re: [HACKERS] WIP: Aggregation push-down

2020-01-16 Thread Antonin Houska
Tomas Vondra wrote: > Hi, > > I've been looking at the last version (v14) of this patch series, > submitted way back in July and unfortunately quiet since then. Antonin > is probably right one of the reasons for the lack of reviews is that it > requires interest/experience with planner. > >

Re: empty range

2020-01-16 Thread Tom Lane
Emre Hasegeli writes: >> It's only suggestion, i don't now if somebody wants store empty range >> without bounds. > I thought about the same while developing the BRIN inclusion operator > class. I am not sure how useful empty ranges are in practice, but > keeping their bound would only bring

Re: progress report for ANALYZE

2020-01-16 Thread Justin Pryzby
On Wed, Jan 15, 2020 at 02:11:10PM -0300, Alvaro Herrera wrote: > I just pushed this after some small extra tweaks. > > Thanks, Yamada-san, for seeing this to completion! Find attached minor fixes to docs - sorry I didn't look earlier. Possibly you'd also want to change the other existing

Code cleanup for build_regexp_split_result

2020-01-16 Thread Li Japin
Hi hackers, I find the build_regexp_split_result() has redundant codes, we can move it to before the condition check, can we? Best regards. Japin Li 0001-Code-cleanup-for-build_regexp_split_result.patch Description: 0001-Code-cleanup-for-build_regexp_split_result.patch

Re: Add pg_file_sync() to adminpack

2020-01-16 Thread Robert Haas
On Tue, Jan 14, 2020 at 10:08 AM Atsushi Torikoshi wrote: > fails we can get error messages. So it seems straightforward for me to > 'return true on success and emit an ERROR otherwise'. I agree with reporting errors using ERROR, but it seems to me that we ought to then make the function return

Re: [PATCH v1] pg_ls_tmpdir to show directories

2020-01-16 Thread Justin Pryzby
On Thu, Jan 16, 2020 at 09:34:32AM +0100, Fabien COELHO wrote: > Also, I'm not fully sure why ".*" files should be skipped, maybe it should > be an option? Or the user can filter it with SQL if it does not want them? I think if someone wants the full generality, they can do this: postgres=#

Re: PATCH: standby crashed when replay block which truncated in standby but failed to truncate in master node

2020-01-16 Thread Fujii Masao
On Tue, Dec 17, 2019 at 2:19 PM Michael Paquier wrote: > > On Mon, Dec 16, 2019 at 12:22:18PM +0900, Fujii Masao wrote: > > > +Detection of WAL records having references to invalid pages > > > during > > > +recovery causes PostgreSQL to report > > > +an error, aborting

Re: Duplicate Workers entries in some EXPLAIN plans

2020-01-16 Thread Georgios Kokolatos
> Sounds good, I'll try that format. Any idea how to test YAML? With the > JSON format, I was able to rely on Postgres' own JSON-manipulating > functions to strip or canonicalize fields that can vary across > executions--I can't really do that with YAML. Yes, this approach was clear in the patch

Re: empty range

2020-01-16 Thread Emre Hasegeli
> It's only suggestion, i don't now if somebody wants store empty range without > bounds. I thought about the same while developing the BRIN inclusion operator class. I am not sure how useful empty ranges are in practice, but keeping their bound would only bring more flexibility, and eliminate

Re: SlabCheck leaks memory into TopMemoryContext

2020-01-16 Thread Tomas Vondra
On Wed, Jan 15, 2020 at 10:41:43PM -0800, Andres Freund wrote: Hi, On 2020-01-16 01:25:00 -0500, Tom Lane wrote: Andres Freund writes: > On 2020-01-16 00:09:53 -0500, Tom Lane wrote: >> It's basically assuming that the memory management mechanism is sane, >> which makes the whole thing

Re: Patch to document base64 encoding

2020-01-16 Thread Fabien COELHO
Hello Karl, New patch attached: doc_base64_v13.patch This required surprisingly little re-wording. Added word "binary" into the descriptions of convert(), substring(), convert_from(), and convert_to(). I also added data types to the call syntax of set_bit() and set_byte(). And this patch

Re: Making psql error out on output failures

2020-01-16 Thread Daniel Verite
David Zhang wrote: > If I change the error log message like below, where "%m" is used to pass the > value of strerror(errno), "could not write to output file:" is copied from > function "WRITE_ERROR_EXIT". > - pg_log_error("Error printing tuples"); > +

Re: remove some STATUS_* symbols

2020-01-16 Thread Robert Haas
On Thu, Jan 16, 2020 at 12:50 AM Michael Paquier wrote: > Thanks, that looks fine. I am still not sure whether the second patch > adding an enum via ProcWaitStatus improves the code readability > though, so my take would be to discard it for now. Perhaps others > think differently, I don't

Re: [HACKERS] Block level parallel vacuum

2020-01-16 Thread Prabhat Sahu
Hi all, I would like to share my observation on this PG feature "Block-level parallel vacuum". I have tested the earlier patch (i.e v48) with below high-level test scenarios, and those are working as expected. - I have played around with these GUC parameters while testing

Re: [HACKERS] Block level parallel vacuum

2020-01-16 Thread Amit Kapila
On Thu, Jan 16, 2020 at 4:46 PM Masahiko Sawada wrote: > > Right. Most indexes (all?) of tables that are used in the regression > tests are smaller than min_parallel_index_scan_size. And we set > min_parallel_index_scan_size to 0 in vacuum.sql but VACUUM would not > be speeded-up much because of

Thoughts on "killed tuples" index hint bits support on standby

2020-01-16 Thread Michail Nikolaev
Hello, hackers. Currently hint bits in the index pages (dead tuples) are set and taken into account only at primary server. Standby just ignores it. It is done for reasons, of course (see RelationGetIndexScan and [1]): * We do this because the xmin on the primary node could easily be

Re: [HACKERS] Block level parallel vacuum

2020-01-16 Thread Masahiko Sawada
On Thu, 16 Jan 2020 at 14:11, Amit Kapila wrote: > > On Thu, Jan 16, 2020 at 10:11 AM Mahendra Singh Thalor > wrote: > > > > On Thu, 16 Jan 2020 at 08:22, Amit Kapila wrote: > > > > > > > 2. > > > > I checked time taken by vacuum.sql test. Execution time is almost same > > > > with and without

Re: remove some STATUS_* symbols

2020-01-16 Thread Kyotaro Horiguchi
At Thu, 16 Jan 2020 14:50:01 +0900, Michael Paquier wrote in > On Sat, Jan 11, 2020 at 08:14:17AM +0100, Peter Eisentraut wrote: > > OK, pushed as it was then. > > Thanks, that looks fine. I am still not sure whether the second patch > adding an enum via ProcWaitStatus improves the code

empty range

2020-01-16 Thread david . turon
Hello, default constructor for ranges use lower bound closed '[' and upper bound open ')'. This is correct behavior, but when upper bound is same like lower bound then range is empty. Mathematically is correct again - but in database is lost information about range bounds (lower/upper is

Re: Implementing Incremental View Maintenance

2020-01-16 Thread nuko yokohama
Error occurs when updating user-defined type columns. Create an INCREMENTAL MATERIALIZED VIEW by specifying a query that includes user-defined type columns. After the view is created, an error occurs when inserting into the view source table (including the user-defined type column). ``` ERROR:

Re: Amcheck: do rightlink verification with lock coupling

2020-01-16 Thread Andrey Borodin
> 14 янв. 2020 г., в 9:47, Andrey Borodin написал(а): > > Page updates may be lost due to bug in backup software with incremental > backups, bug in storage layer of Aurora-style system, bug in page cache, > incorrect > fsync error handling, bug in ssd firmware etc. And our data checksums do

  1   2   >