Re: 64-bit XIDs in deleted nbtree pages

2021-02-16 Thread Masahiko Sawada
btvacuumscan()). The new btm_last_cleanup_num_delpages field > (which replaces btm_oldest_btpo_xact) has a value that just comes from > the bulk stats, which is easy anyway. Agreed. As I mentioned above, we might need to consider how btbulkdelete() can tell btvacuumcleanup() btm_last_cleanup_num_delpages

Re: 64-bit XIDs in deleted nbtree pages

2021-02-18 Thread Masahiko Sawada
parate calls -- since there is > no easy way to "send" the oldestBtpoXact value determined during a > btbulkdelete() to a later corresponding btvacuumcleanup(). That's a > bit of a kludge, but I'm not worried about it.) Agreed. Regards, -- Masahiko Sawada EDB: https://www.enterprisedb.com/

Re: 64-bit XIDs in deleted nbtree pages

2021-02-18 Thread Masahiko Sawada
On Tue, Feb 16, 2021 at 12:26 PM Peter Geoghegan wrote: > > On Mon, Feb 15, 2021 at 3:15 AM Masahiko Sawada wrote: > > Yes. I think this would simplify the problem by resolving almost all > > problems related to indefinite deferring page recycle. > > > > We will

Re: 64-bit XIDs in deleted nbtree pages

2021-02-18 Thread Masahiko Sawada
On Fri, Feb 19, 2021 at 3:12 PM Peter Geoghegan wrote: > > On Thu, Feb 18, 2021 at 3:13 AM Masahiko Sawada wrote: > > Agreed. Thanks for your explanation. > > Attached is v5, which has some of the changes I talked about. Changes > from v4 include: > > * Now o

Re: a misbehavior of partition row movement (?)

2021-02-19 Thread Masahiko Sawada
On Mon, Feb 15, 2021 at 10:37 PM Amit Langote wrote: > > Thank you for looking at this. > > On Mon, Feb 15, 2021 at 4:12 PM Masahiko Sawada wrote: > > On Wed, Jan 20, 2021 at 7:04 PM Amit Langote > > wrote: > > > This shows that the way we've made thes

Re: New IndexAM API controlling index vacuum strategies

2021-02-21 Thread Masahiko Sawada
On Wed, Feb 10, 2021 at 4:12 PM Peter Geoghegan wrote: > > On Tue, Feb 9, 2021 at 6:14 PM Masahiko Sawada wrote: > > Thanks. I think that's very good if we resolve this recycling stuff > > first then try the new approach to skip index vacuum in more cases. > >

Re: 64-bit XIDs in deleted nbtree pages

2021-02-22 Thread Masahiko Sawada
On Fri, Feb 19, 2021 at 3:18 PM Masahiko Sawada wrote: > > On Fri, Feb 19, 2021 at 3:12 PM Peter Geoghegan wrote: > > > > On Thu, Feb 18, 2021 at 3:13 AM Masahiko Sawada > > wrote: > > > Agreed. Thanks for your explanation. > > > > Attached is v5,

Re: 64-bit XIDs in deleted nbtree pages

2021-02-22 Thread Masahiko Sawada
On Tue, Feb 23, 2021 at 7:55 AM Peter Geoghegan wrote: > > On Mon, Feb 22, 2021 at 4:21 AM Masahiko Sawada wrote: > > The 0001 patch looks good to me. In the documentation, I think we need > > to update the following paragraph in the description of > > vacuum_cleanup_inde

Re: {CREATE INDEX, REINDEX} CONCURRENTLY improvements

2021-02-23 Thread Masahiko Sawada
movable is the oldest transaction id among data_oldest_nonremovable, slot_xmin, and slot_catalog_xmin. But with the patch, catalog_oldest_nonremovable doesn't consider slot_xmin. It seems okay to me as catalog_oldest_nonremovable is interested in only slot_catalog_xmin. Regards, -- Masahiko Sawada EDB: https://www.enterprisedb.com/

Re: Single transaction in the tablesync worker?

2021-02-23 Thread Masahiko Sawada
ication test_pub for table test; 3. On subscriber: create subscription test_sub connection 'dbname=postgres' publication test_pub"; -- wait until table sync finished drop table test; create table test (a int primary key); >From this point, you will get the WARNING message whe

Re: [PoC] Non-volatile WAL buffer

2021-02-24 Thread Masahiko Sawada
On Sat, Feb 13, 2021 at 12:18 PM Masahiko Sawada wrote: > > On Thu, Jan 28, 2021 at 1:41 AM Tomas Vondra > wrote: > > > > On 1/25/21 3:56 AM, Masahiko Sawada wrote: > > >> > > >> ... > > >> > > >> On 1/21/21 3:17 AM, Masahi

Re: 64-bit XIDs in deleted nbtree pages

2021-02-25 Thread Masahiko Sawada
until my bug fix commit > 48e12913 (which was kind of a hack itself). This meant that the > statistics set by btvacuumcleanup (in the case where btbulkdelete > doesn't get called, the relevant case for > vacuum_cleanup_index_scale_factor). So it was 100% wrong for months > before anybody noticed (or at least until anybody complained). > Maybe we need more regression tests here. Regards, -- Masahiko Sawada EDB: https://www.enterprisedb.com/

Re: Single transaction in the tablesync worker?

2021-02-25 Thread Masahiko Sawada
On Thu, Feb 25, 2021 at 1:52 PM Amit Kapila wrote: > > On Wed, Feb 24, 2021 at 5:55 PM Amit Kapila wrote: > > > > On Wed, Feb 24, 2021 at 12:47 PM Masahiko Sawada > > wrote: > > > > > > On Fri, Feb 12, 2021 at 2:49 PM Amit Kapila > > > wrote

Re: a misbehavior of partition row movement (?)

2021-02-25 Thread Masahiko Sawada
On Mon, Feb 22, 2021 at 3:04 PM Amit Langote wrote: > > On Fri, Feb 19, 2021 at 5:04 PM Masahiko Sawada wrote: > > On Mon, Feb 15, 2021 at 10:37 PM Amit Langote > > wrote: > > > Regarding the patch, I would have liked if it only prevented the > > > updat

Re: 64-bit XIDs in deleted nbtree pages

2021-02-26 Thread Masahiko Sawada
On Fri, Feb 26, 2021 at 9:58 AM Peter Geoghegan wrote: > > On Thu, Feb 25, 2021 at 5:42 AM Masahiko Sawada wrote: > > btvacuumcleanup() has been playing two roles: recycling deleted pages > > and collecting index statistics. > > Right. > > I pushed the VACUUM VERB

Re: 64-bit XIDs in deleted nbtree pages

2021-02-28 Thread Masahiko Sawada
ion is with the > data). I probably should have made btvacuumcleanup()-only VACUUMs set > "stats->estimate_count = true", purely to make sure that the core code > doesn't trust the statistics too much (it's okay for VACUUM VERBOSE > output only). Right now we can get a pg_class.reltuples that is > "exactly wrong" -- it would actually be a big improvement if it was > "approximately correct". Understood. Thank you for your explanation. > > Another new concern for me (another concern unique to Postgres 13) is > autovacuum_vacuum_insert_scale_factor-driven autovacuums. IIUC the purpose of autovacuum_vacuum_insert_scale_factor is visibility map maintenance. And as per this discussion, it seems not necessary to do an index scan in btvacuumcleanup() triggered by autovacuum_vacuum_insert_scale_factor. Regards, -- Masahiko Sawada EDB: https://www.enterprisedb.com/

Re: [BUG] Autovacuum not dynamically decreasing cost_limit and cost_delay

2021-03-01 Thread Masahiko Sawada
elay during vacuuming a table, which seems wrong to me. autovac_balance_cost() can change workers' wi_cost_delay, eventually applying to VacuumCostDelay. What do you think? Regards, -- Masahiko Sawada EDB: https://www.enterprisedb.com/

Re: 64-bit XIDs in deleted nbtree pages

2021-03-01 Thread Masahiko Sawada
On Tue, Mar 2, 2021 at 6:40 AM Peter Geoghegan wrote: > > On Sun, Feb 28, 2021 at 8:08 PM Masahiko Sawada wrote: > > > Even though "the letter of the law" favors removing the > > > vacuum_cleanup_index_scale_factor GUC + param in the way I have > > >

Removing vacuum_cleanup_index_scale_factor

2021-03-01 Thread Masahiko Sawada
poses. What do you think about removing vacuum_cleanup_index_scale_factor parameter? and we'd like to ask the design principles of amvacuumcleanup() considered in 2006 by various hackers (mostly Tom). What do you think, Tom? Regards, [1] https://www.postgresql.org/message-id/CAH2-WznUWHOL%2BnYYT2PLsn%2B3OWcq8OBfA1sB3FX885rE%3DZQVvA%40mail.gmail.com -- Masahiko Sawada EDB: https://www.enterprisedb.com/

Re: 64-bit XIDs in deleted nbtree pages

2021-03-01 Thread Masahiko Sawada
On Tue, Mar 2, 2021 at 1:06 PM Masahiko Sawada wrote: > > On Tue, Mar 2, 2021 at 6:40 AM Peter Geoghegan wrote: > > > > On Sun, Feb 28, 2021 at 8:08 PM Masahiko Sawada > > wrote: > > > > Even though "the letter of the law" favors removing the

Re: New IndexAM API controlling index vacuum strategies

2021-03-02 Thread Masahiko Sawada
On Tue, Mar 2, 2021 at 12:00 PM Peter Geoghegan wrote: > > On Sun, Feb 21, 2021 at 10:28 PM Masahiko Sawada > wrote: > > Sorry for the late response. > > Me too! No problem, thank you for your comment! > > > 1. Whereas skipping index vacuum and heap vacuum is a v

Re: New IndexAM API controlling index vacuum strategies

2021-03-02 Thread Masahiko Sawada
large table case. It seems to contradict the current strategy that we want to skip vacuum if relatively few blocks are modified. No? Regards, -- Masahiko Sawada EDB: https://www.enterprisedb.com/

Re: Freeze the inserted tuples during CTAS?

2021-03-02 Thread Masahiko Sawada
e but If there is much performance degradation due to populating visibility map, it might be better to provide a way to disable it. Regards, -- Masahiko Sawada EDB: https://www.enterprisedb.com/

n_mod_since_analyze isn't reset at table truncation

2021-03-03 Thread Masahiko Sawada
nce_vacuum -+-+ t | 6 | 0 (1 row) I've attached a small patch to fix this. Please review it. Regards, -- Masahiko Sawada EDB: https://www.enterprisedb.com/ reset_n_mod_since_analyze.patch Description: Binary data

Re: n_mod_since_analyze isn't reset at table truncation

2021-03-04 Thread Masahiko Sawada
On Thu, Mar 4, 2021 at 11:23 AM Julien Rouhaud wrote: > > On Thu, Mar 04, 2021 at 10:35:19AM +0900, Masahiko Sawada wrote: > > > > While reviewing "autoanalyze on partitioned table" patch, I realized > > that pg_stat_xxx_tables.n_mod_since_analyze is not reset

Re: n_mod_since_analyze isn't reset at table truncation

2021-03-04 Thread Masahiko Sawada
On Thu, Mar 4, 2021 at 12:39 PM Julien Rouhaud wrote: > > On Thu, Mar 04, 2021 at 12:21:14PM +0900, Fujii Masao wrote: > > > > > > On 2021/03/04 11:24, Julien Rouhaud wrote: > > > On Thu, Mar 04, 2021 at 10:35:19AM +0900, Masahiko Sawada wrote: > > >

Re: n_mod_since_analyze isn't reset at table truncation

2021-03-05 Thread Masahiko Sawada
truncate is > > not so "different" from the status of the table after truncate. > > > > I'm not sure how much this idea is "reliable" and would be helpful in > > practice, though. > > It seems like a better approach as it it would have t

Re: n_mod_since_analyze isn't reset at table truncation

2021-03-07 Thread Masahiko Sawada
On Fri, Mar 5, 2021 at 10:43 PM Masahiko Sawada wrote: > > On Fri, Mar 5, 2021 at 6:51 PM Julien Rouhaud wrote: > > > > On Fri, Mar 05, 2021 at 06:07:05PM +0900, Fujii Masao wrote: > > > > > > On 2021/03/05 15:59, Julien Rouhaud wrote: > > > > >

Re: 64-bit XIDs in deleted nbtree pages

2021-03-07 Thread Masahiko Sawada
On Tue, Mar 2, 2021 at 1:42 PM Peter Geoghegan wrote: > > On Mon, Mar 1, 2021 at 8:06 PM Masahiko Sawada wrote: > > I think that removing vacuum_cleanup_index_scale_factor in the back > > branches would affect the existing installation much. It would be > > better to ha

Re: a verbose option for autovacuum

2021-03-07 Thread Masahiko Sawada
system usage: CPU: user: 0.03 s, system: 0.00 s, elapsed: 0.06 s WAL usage: 2162 records, 4 full page images, 159047 bytes It still lacks some of what VACUUM VERBOSE shows (e.g., each index vacuum execution time etc) but it would be enough information to know the index page statistics. Probably we can output those by default without adding a new parameter controlling that. Regards, -- Masahiko Sawada EDB: https://www.enterprisedb.com/

Re: a verbose option for autovacuum

2021-03-07 Thread Masahiko Sawada
On Tue, Feb 2, 2021 at 10:51 AM Euler Taveira wrote: > > On Fri, Jan 29, 2021, at 4:35 AM, Masahiko Sawada wrote: > > I agree to have autovacuum log more information, especially index > vacuums. Currently, the log related to index vacuum is only the number > of index scans. I

Re: Boundary value check in lazy_tid_reaped()

2021-03-08 Thread Masahiko Sawada
On Mon, Mar 8, 2021 at 7:16 PM Peter Eisentraut wrote: > > On 21.01.21 14:11, Masahiko Sawada wrote: > > Agreed. bsearch with bound check showed a reasonable improvement in my > > evaluation in terms of performance. Regarding memory efficiency, we > > can experiment

Re: Removing vacuum_cleanup_index_scale_factor

2021-03-08 Thread Masahiko Sawada
s a patch that removes the > vacuum_cleanup_index_scale_factor mechanism for triggering an index > scan during VACUUM -- that's what the second patch does (this depends > on the first patch, really). 0002 patch looks good to me. Regards, -- Masahiko Sawada EDB: https://www.enterprisedb.com/

Re: Boundary value check in lazy_tid_reaped()

2021-03-09 Thread Masahiko Sawada
On Tue, Mar 9, 2021 at 9:57 AM Masahiko Sawada wrote: > > On Mon, Mar 8, 2021 at 7:16 PM Peter Eisentraut > wrote: > > > > On 21.01.21 14:11, Masahiko Sawada wrote: > > > Agreed. bsearch with bound check showed a reasonable improvement in my > > > evalua

Re: a verbose option for autovacuum

2021-03-09 Thread Masahiko Sawada
On Tue, Mar 9, 2021 at 12:58 AM Euler Taveira wrote: > > On Mon, Mar 8, 2021, at 2:32 AM, Masahiko Sawada wrote: > > * Proposed idea > LOG: automatic vacuum of table "postgres.public.test": index scans: 1 > pages: 0 removed, 443 remain, 0 skipped due to pins, 0 s

Re: a misbehavior of partition row movement (?)

2021-03-09 Thread Masahiko Sawada
(&buf, "(NOT t.tgisinternal OR (t.tgisinternal AND t.tgenabled = 'D') \n" - "OR EXISTS (SELECT 1 FROM pg_catalog.pg_depend WHERE objid = t.oid \n" -"AND refclassid = 'pg_catalog.pg_trigger'::pg_catalog.regclass))"); - else if (pset.sversion >= 9) + if (pset.sversion >= 9) I think we cannot remove this code. For instance, in PG13 since the trisinternal of a user-defined trigger that has descended from its parent table is true, executing \d against PG13 by the patched psql won't show that trigger. I'll look at 0002 patch. Regards, -- Masahiko Sawada EDB: https://www.enterprisedb.com/

Re: Boundary value check in lazy_tid_reaped()

2021-03-10 Thread Masahiko Sawada
On Wed, Mar 10, 2021 at 11:53 PM Peter Eisentraut wrote: > > On 10.03.21 02:29, Masahiko Sawada wrote: > >> There is no noticeable effect of inlining lazy_tid_reaped(). So it > >> would be better to not do that. > > > > Attached the patch that doesn'

Re: Removing vacuum_cleanup_index_scale_factor

2021-03-10 Thread Masahiko Sawada
eat! Thank you! Regards, -- Masahiko Sawada EDB: https://www.enterprisedb.com/

Re: Freeze the inserted tuples during CTAS?

2021-03-10 Thread Masahiko Sawada
On Wed, Mar 10, 2021 at 3:57 PM Paul Guo wrote: > > > On Mar 3, 2021, at 1:35 PM, Masahiko Sawada wrote: > >> On Sun, Feb 21, 2021 at 4:46 PM Paul Guo wrote: > >> Attached is the v2 version that fixes a test failure due to plan change > >> (bitmap index sca

Performance degradation of REFRESH MATERIALIZED VIEW

2021-03-11 Thread Masahiko Sawada
t/FB1F5E2D-CBD1-4645-B74C-E0A1BFAE4AC8%40vmware.com -- Masahiko Sawada EDB: https://www.enterprisedb.com/

Re: New IndexAM API controlling index vacuum strategies

2021-03-11 Thread Masahiko Sawada
On Wed, Mar 3, 2021 at 12:40 PM Peter Geoghegan wrote: > > On Tue, Mar 2, 2021 at 6:44 PM Masahiko Sawada wrote: > > A scale type parameter seems good to me but I wonder if how users can > > tune that parameter. We already have tuple-based parameters such as > > autova

Re: Dependency isn't created between extension and schema

2020-12-31 Thread Masahiko Sawada
On Tue, Dec 22, 2020 at 1:03 AM Tom Lane wrote: > > James Coleman writes: > > On Mon, Dec 21, 2020 at 2:59 AM Michael Paquier wrote: > >> On Mon, Dec 21, 2020 at 04:02:29PM +0900, Masahiko Sawada wrote: > >>> Is it a bug? Since the created schema obviously depen

Commitfest 2021-01 Now in Progress

2021-01-01 Thread Masahiko Sawada
tches of a similar number and complexity. The CF cannot move forward without patch review. Also, check the state of your patch at http://cfbot.cputube.org/ Happy hacking! -- Masahiko Sawada EnterpriseDB: https://www.enterprisedb.com/

Re: pg_waldump/heapdesc.c and struct field names

2021-01-03 Thread Masahiko Sawada
xlrec->node.relNode, xlrec->latestRemovedXid); break; } Regards, -- Masahiko Sawada EnterpriseDB: https://www.enterprisedb.com/

Re: {CREATE INDEX, REINDEX} CONCURRENTLY improvements

2021-01-04 Thread Masahiko Sawada
email. For the 0001 patch, since ReindexIndexInfo is used only within ReindexRelationConcurrently() I think it’s a function-local structure type. So we can declare it within the function. What do you think? Regards, -- Masahiko Sawada EnterpriseDB: https://www.enterprisedb.com/

Re: [PATCH] Feature improvement for CLOSE, FETCH, MOVE tab completion

2021-01-04 Thread Masahiko Sawada
--- The patch improves tab completion for CLOSE, FETCH, and MOVE but is there any reason why you didn't do that for DECLARE? I think DECLARE also can be improved and it's a good timing for that. Regards, -- Masahiko Sawada EnterpriseDB: https://www.enterprisedb.com/

Re: WIP: System Versioned Temporal Table

2021-01-04 Thread Masahiko Sawada
annot be applied to the current HEAD. I'm switching the patch as Waiting on Author. Regards, -- Masahiko Sawada EnterpriseDB: https://www.enterprisedb.com/

Re: libpq debug log

2021-01-04 Thread Masahiko Sawada
case LOG_CONTENTS: > + /* Suppresses printing terminating zero byte > */ > > --> Suppress printing of terminating zero byte > The patch got some review comments a couple weeks ago but the patch was still marked as "needs review", which was incorrect. Also cfbot[1] is unhappy with this patch. So I'm switching the patch as "waiting on author". Regards, [1] http://commitfest.cputube.org/ -- Masahiko Sawada EnterpriseDB: https://www.enterprisedb.com/

Re: VACUUM (DISABLE_PAGE_SKIPPING on)

2021-01-04 Thread Masahiko Sawada
On Tue, Dec 1, 2020 at 10:45 AM Masahiko Sawada wrote: > > On Fri, Nov 20, 2020 at 8:47 PM Simon Riggs wrote: > > > > On Fri, 20 Nov 2020 at 10:15, Simon Riggs wrote: > > > > > > On Fri, 20 Nov 2020 at 01:40, Masahiko Sawada > > > wrote: > >

Re: New IndexAM API controlling index vacuum strategies

2021-01-04 Thread Masahiko Sawada
On Tue, Dec 29, 2020 at 3:25 PM Masahiko Sawada wrote: > > On Tue, Dec 29, 2020 at 7:06 AM Peter Geoghegan wrote: > > > > On Sun, Dec 27, 2020 at 11:41 PM Peter Geoghegan wrote: > > > I experimented with this today, and I think that it is a good way to >

Re: fix typo in ReorderBufferProcessTXN

2021-01-04 Thread Masahiko Sawada
repeated > > > > Is there an extra ‘normal’ in the comment ? Yeah, I think so. > in contrast to normal the normal catalog during decoding == >> in contrast to > the normal catalog during decoding Looks good change to me. Regards, -- Masahiko Sawada EnterpriseDB: https://www.enterprisedb.com/

Re: Adding new commitfest entry?

2021-01-04 Thread Masahiko Sawada
to switch Commitfest 2021-03 as "open”. I’ve switched it now. Please check if you can add the patch to the Commitfest 2021-03. Regards, -- Masahiko Sawada EnterpriseDB: https://www.enterprisedb.com/

Re: Disable WAL logging to speed up data loading

2021-01-05 Thread Masahiko Sawada
On Tue, Jan 5, 2021 at 10:54 AM osumi.takami...@fujitsu.com wrote: > > Hi, Sawada-San > > > On Monday, December 28, 2020 7:12 PM Masahiko Sawada > wrote: > > On Mon, Dec 28, 2020 at 4:29 PM osumi.takami...@fujitsu.com > > wrote: > > > On Monday, De

Re: [PATCH] Feature improvement for CLOSE, FETCH, MOVE tab completion

2021-01-05 Thread Masahiko Sawada
t; Please let me know if there are any codes that can be improved. I've attached the patch improving the tab completion for DECLARE as an example. What do you think? BTW according to the documentation, the options of DECLARE statement (BINARY, INSENSITIVE, SCROLL, and NO SCROLL) are order

Re: [PATCH] Feature improvement for CLOSE, FETCH, MOVE tab completion

2021-01-05 Thread Masahiko Sawada
placed just after "FETCH". According to the documentation, the direction can be empty. For instance, we can do like: postgres(1:7668)=# begin; BEGIN postgres(1:7668)=# declare test cursor for select relname from pg_class; DECLARE CURSOR postgres(1:7668)=# fetch from test; relname -- pg_statistic (1 row) postgres(1:7668)=# fetch in test; relname - pg_type (1 row) Regards, -- Masahiko Sawada EnterpriseDB: https://www.enterprisedb.com/

Re: logical replication worker accesses catalogs in error context callback

2021-01-05 Thread Masahiko Sawada
typname() is safe to run in an error context > callback. But get_atttype() definitely isn't. > > get_attype() may do catalog accesses. That definitely can't happen > inside an error context callback - the entire transaction might be > borked at this point! You're right. Pe

Re: Deadlock between backend and recovery may not be detected

2021-01-05 Thread Masahiko Sawada
kip the back-patch to v9.5. > Because the next minor version release is the final one for v9.5. So if we > unexpectedly introduce the bug to v9.5 by the back-patch, there is no > chance to fix that. OTOH, of course, if we don't do the back-patch, there is > no chance to fi

Re: logical replication worker accesses catalogs in error context callback

2021-01-06 Thread Masahiko Sawada
On Wed, Jan 6, 2021 at 11:42 AM Masahiko Sawada wrote: > > On Wed, Jan 6, 2021 at 11:02 AM Andres Freund wrote: > > > > Hi, > > > > Due to a debug ereport I just noticed that worker.c's > > slot_store_error_callback is doing somet

Re: [PATCH] Feature improvement for CLOSE, FETCH, MOVE tab completion

2021-01-06 Thread Masahiko Sawada
758)=# declare test binary binary binary cursor for select * from pg_class; DECLARE CURSOR So how about simplify the above code as follows? @@ -3005,8 +3014,23 @@ psql_completion(const char *text, int start, int end) else if (Matches("DECLARE", MatchAny)) COMPLETE_WITH("BINARY", "INSENSITIVE", "SCROLL", "NO SCROLL", "CURSOR"); + /* +* Complete DECLARE with one of BINARY, INSENSITIVE, SCROLL, +* NO SCROLL, and CURSOR. The tail doesn't match any keywords for +* DECLARE, assume we want options. +*/ + else if (HeadMatches("DECLARE", MatchAny, "*") && +TailMatches(MatchAnyExcept("CURSOR|WITH|WITHOUT|HOLD|FOR"))) + COMPLETE_WITH("BINARY", "INSENSITIVE", "SCROLL", "NO SCROLL", + "CURSOR"); + /* +* Complete DECLARE ... CURSOR with one of WITH HOLD, WITHOUT HOLD, +* and FOR. +*/ else if (HeadMatches("DECLARE") && TailMatches("CURSOR")) COMPLETE_WITH("WITH HOLD", "WITHOUT HOLD", "FOR"); + else if (HeadMatches("DECLARE") && TailMatches("HOLD")) + COMPLETE_WITH("FOR"); Regards, -- Masahiko Sawada EnterpriseDB: https://www.enterprisedb.com/

Re: vacuum_cost_page_miss default value and modern hardware

2021-01-06 Thread Masahiko Sawada
at the cheaper hardware that has a small RAM etc would be likely to be affected by this change. Since the database doesn’t fit in the server’s RAM, pages are unlikely to be on neither the shared buffers nor OS page cache. Since PostgreSQL's default values seem conservative to me (which is okay to me), I think there might be an argument that this change could lead to trouble in such a cheaper environment that PostgreSQL’s default values are taking care of. Regards, -- Masahiko Sawada EnterpriseDB: https://www.enterprisedb.com/

Re: [PATCH] Feature improvement for CLOSE, FETCH, MOVE tab completion

2021-01-06 Thread Masahiko Sawada
On Thu, Jan 7, 2021 at 10:59 AM Fujii Masao wrote: > > > > On 2021/01/07 10:01, Masahiko Sawada wrote: > > On Wed, Jan 6, 2021 at 3:37 PM wrote: > >> > >>> +#define Query_for_list_of_cursors \ > >>> +" SELECT name FROM pg_

Re: [PATCH] Feature improvement for CLOSE, FETCH, MOVE tab completion

2021-01-06 Thread Masahiko Sawada
On Thu, Jan 7, 2021 at 1:30 PM Fujii Masao wrote: > > > > On 2021/01/07 12:42, Masahiko Sawada wrote: > > On Thu, Jan 7, 2021 at 10:59 AM Fujii Masao > > wrote: > >> > >> > >> > >> On 2021/01/07 10:01, Masahiko Sawada wrote: > >

Re: pgbench stopped supporting large number of client connections on Windows

2021-01-07 Thread Masahiko Sawada
I will also read that thread... > > > Why not if there is no other solution, but this is quite depressing, > > and because it breaks the interface it would be broken if windows > > changed its internals for some reason:-( > > It looks like if the internals of the structure

Re: Removing unneeded self joins

2021-01-07 Thread Masahiko Sawada
t;> 1 | 2 > >> 2 | 1 > >> (2 rows) > > Thanks, it is my fault. I tried to extend this patch with foreign key > references and made a mistake. > Currently I rollback this new option (see patch in attachment), but will > be working for a while to simplify this patch. Are you working to simplify the patch? This patch has been "Waiting on Author" for 1 month and doesn't seem to pass cfbot tests. Please update the patch. Regards, -- Masahiko Sawada EnterpriseDB: https://www.enterprisedb.com/

Re: Error on failed COMMIT

2021-01-07 Thread Masahiko Sawada
ated for almost 2 months. According to cfbot test, the patch conflicts on only src/include/utils/elog.h. Could you submit the rebased patch? Regards, -- Masahiko Sawada EnterpriseDB: https://www.enterprisedb.com/

Re: Disable WAL logging to speed up data loading

2021-01-08 Thread Masahiko Sawada
table AM) control it instead of filtering in XLogInsert(). Because otherwise third-party access methods that use LSN like gist indexes cannot write WAL records at all when wal_level = none even if they want. Regards, -- Masahiko Sawada EnterpriseDB: https://www.enterprisedb.com/

Re: logical replication worker accesses catalogs in error context callback

2021-01-10 Thread Masahiko Sawada
On Sat, Jan 9, 2021 at 2:57 PM Bharath Rupireddy wrote: > > On Thu, Jan 7, 2021 at 5:47 AM Masahiko Sawada wrote: > > > > On Wed, Jan 6, 2021 at 11:42 AM Masahiko Sawada > > wrote: > > > > > > On Wed, Jan 6, 2021 at 11:02 AM Andres Freund wrote: &g

Re: [PATCH] Feature improvement for CLOSE, FETCH, MOVE tab completion

2021-01-10 Thread Masahiko Sawada
On Thu, Jan 7, 2021 at 9:32 PM Fujii Masao wrote: > > > > On 2021/01/07 17:28, shinya11.k...@nttdata.com wrote: > >> On Thu, Jan 7, 2021 at 1:30 PM Fujii Masao > >> wrote: > >>> > >>> On 2021/01/07 12:42, Masahiko Sawada wrote: > >&g

Re: Key management with tests

2021-01-11 Thread Masahiko Sawada
ncryption to make sure the kms patch would be workable with other components using the encryption key managed by kmgr. Overall it’s good. While the buffer encryption patch is still PoC quality and there are some problems regarding nonce generation we need to deal with, it easily can use the relation k

Re: Key management with tests

2021-01-11 Thread Masahiko Sawada
forward, but since this is required even when changing the page is not the first time since the last checkpoint we might end up logging too many dummy WAL records. Regards, -- Masahiko Sawada EnterpriseDB: https://www.enterprisedb.com/

Re: [PATCH] Feature improvement for CLOSE, FETCH, MOVE tab completion

2021-01-11 Thread Masahiko Sawada
On Mon, Jan 11, 2021 at 11:00 PM Peter Eisentraut wrote: > > On 2021-01-05 10:56, Masahiko Sawada wrote: > > BTW according to the documentation, the options of DECLARE statement > > (BINARY, INSENSITIVE, SCROLL, and NO SCROLL) are order-sensitive. > > > > DECLARE

Re: logical replication worker accesses catalogs in error context callback

2021-01-11 Thread Masahiko Sawada
On Mon, Jan 11, 2021 at 4:46 PM Bharath Rupireddy wrote: > > On Mon, Jan 11, 2021 at 10:56 AM Masahiko Sawada > wrote: > > Agreed. Attached the updated patch. > > Thanks for the updated patch. Looks like the comment crosses the 80 > char limit, I have corrected it. And a

Fix a typo in xlogreader.c

2021-01-12 Thread Masahiko Sawada
(XLogReaderState *record, uint8 block_id, char *page) Attached the patch that fixes it. Regards, -- Masahiko Sawada EnterpriseDB: https://www.enterprisedb.com/ fix_typo_xlogreader.patch Description: Binary data

Executing inet_server_addr/port() in parallel workers

2021-01-12 Thread Masahiko Sawada
ity and is not good in terms of back-patching. So I'd prefer the option #2. I've attached the patch doing the option #2. Regards, -- Masahiko Sawada EnterpriseDB: https://www.enterprisedb.com/ fix_inet_server_addr_port.patch Description: Binary data

Re: Key management with tests

2021-01-12 Thread Masahiko Sawada
On Tue, Jan 12, 2021 at 11:09 AM Bruce Momjian wrote: > > On Tue, Jan 12, 2021 at 09:32:54AM +0900, Masahiko Sawada wrote: > > On Tue, Jan 12, 2021 at 3:23 AM Stephen Frost wrote: > > > Right, or ensure that the actual IV used is distinct (such as by using > >

Re: Executing inet_server_addr/port() in parallel workers

2021-01-13 Thread Masahiko Sawada
h I thought that backpatching would be helpful for users who start to use the supported PostgreSQL, I agree not to backpatching since the likelihood that parallel workers execute these functions is low. Thank you for committing the patch! Regards, -- Masahiko Sawada EnterpriseDB: https://www.enterprisedb.com/

Re: [PATCH] Feature improvement for CLOSE, FETCH, MOVE tab completion

2021-01-13 Thread Masahiko Sawada
On Wed, Jan 13, 2021 at 1:55 PM Fujii Masao wrote: > > On Tue, Jan 12, 2021 at 11:09 AM Fujii Masao wrote: > > > > On Tue, Jan 12, 2021 at 10:00 AM Masahiko Sawada > > wrote: > > > > > > On Mon, Jan 11, 2021 at 11:00 PM Peter Eisentraut > &g

Re: vacuum_cost_page_miss default value and modern hardware

2021-01-14 Thread Masahiko Sawada
tirely sequential), then that will naturally > provide additional throttling. +1 for this change. Lowering to 2 also looks good to me. Regards, -- Masahiko Sawada EnterpriseDB: https://www.enterprisedb.com/

Re: Transactions involving multiple postgres foreign servers, take 2

2021-01-14 Thread Masahiko Sawada
, breaking a long error message string in arbitrary places just to keep the code within 80 columns is probably not a net gain in readability.) These changes have been made in the local branch. I'll post the updated patch set after incorporating all the comments. Regards,

Re: Commitfest 2021-01 Now in Progress

2021-01-15 Thread Masahiko Sawada
Hi, On Fri, Jan 1, 2021 at 9:22 PM Masahiko Sawada wrote: > > Hi, > > Happy new year to all! > > The January Commitfest for PG14 is now in progress! I'm happy to > volunteer to manage it. > Now that a half month has passed, the current state is: Needs review: 16

Re: Is Recovery actually paused?

2021-01-15 Thread Masahiko Sawada
g. --- - PG_RETURN_BOOL(RecoveryIsPaused()); + if (!RecoveryPauseRequested()) + PG_RETURN_BOOL(false); + + /* loop until the recovery is actually paused */ + while(!RecoveryIsPaused()) + { + pg_usleep(1L); /* wait for 10 msec */ + + /* meanwhile if recovery is re

Re: Is Recovery actually paused?

2021-01-16 Thread Masahiko Sawada
On Sat, Jan 16, 2021 at 12:28 PM Masahiko Sawada wrote: > > --- > + /* test for recovery pause if user has requested the pause */ > + if (((volatile XLogCtlData *) XLogCtl)->recoveryPause) > + recoveryPausesHere(false); > + > + now = GetCurrent

Re: New IndexAM API controlling index vacuum strategies

2021-01-17 Thread Masahiko Sawada
On Tue, Jan 5, 2021 at 10:35 AM Masahiko Sawada wrote: > > On Tue, Dec 29, 2020 at 3:25 PM Masahiko Sawada wrote: > > > > On Tue, Dec 29, 2020 at 7:06 AM Peter Geoghegan wrote: > > > > > > On Sun, Dec 27, 2020 at 11:41 PM Peter Geoghegan wrote: > >

Re: Transactions involving multiple postgres foreign servers, take 2

2021-01-17 Thread Masahiko Sawada
kend = InvalidBackendId; > + LWLockRelease(FdwXactLock); > > There is no call to method inside the for loop which may take time. I wonder > if the lock can be obtained prior to the for loop and released coming out of > the for loop. Agreed. > > +FXRslvLoop(void) > >

Re: New IndexAM API controlling index vacuum strategies

2021-01-17 Thread Masahiko Sawada
On Mon, Jan 18, 2021 at 2:18 PM Masahiko Sawada wrote: > > On Tue, Jan 5, 2021 at 10:35 AM Masahiko Sawada wrote: > > > > On Tue, Dec 29, 2020 at 3:25 PM Masahiko Sawada > > wrote: > > > > > > On Tue, Dec 29, 2020 at 7:06 AM Peter Geoghegan wrote: >

Re: New IndexAM API controlling index vacuum strategies

2021-01-19 Thread Masahiko Sawada
rPage - (((BLCKSZ - SizeOfPageHeaderData) * (fillfactor/100)) / (sizeof(ItemIdData) + tuple_size))); //how many line pointers are available for LP_DEAD? (2) ((BLCKSZ - SizeOfPageHeaderData) * ((1 - fillfactor)/100)) / sizeof(ItemIdData); //how much space is available for LP_DEAD? But I'd prefer to increase MaxHeapTuplesPerPage but not to affect the bitmap much rather than introducing a complex theory. Regards, -- Masahiko Sawada EDB: https://www.enterprisedb.com/

Re: pg_stat_statements oddity with track = all

2021-01-20 Thread Masahiko Sawada
ies. > > Ah, I didn't think of the reset function. I also fail to see a > reasonable use case to provide a switch for the reset function. > > > 4. others > > == > > > > These comments are not related to this patch. > > > > A. about the topic of commitfests > > > > Since I think this feature is for monitoring, > > it's better to change the topic from "System Administration" > > to "Monitoring & Control". > > I agree, thanks for the change! I've changed the topic accordingly. Regards, -- Masahiko Sawada EDB: https://www.enterprisedb.com/

Re: [PoC] Non-volatile WAL buffer

2021-01-20 Thread Masahiko Sawada
On Thu, Jan 7, 2021 at 2:16 AM Tomas Vondra wrote: > > Hi, > > I think I've managed to get the 0002 patch [1] rebased to master and > working (with help from Masahiko Sawada). It's not clear to me how it > could have worked as submitted - my theory is that an incompl

Re: Boundary value check in lazy_tid_reaped()

2021-01-21 Thread Masahiko Sawada
On Wed, Jan 20, 2021 at 3:50 PM Peter Eisentraut wrote: > > On 2020-10-30 02:43, Masahiko Sawada wrote: > > Using the integer set is very memory efficient (5MB vs. 114MB in the > > base case) and there is no 1GB limitation. Looking at the execution > > time, I had expecte

Re: New IndexAM API controlling index vacuum strategies

2021-01-21 Thread Masahiko Sawada
On Wed, Jan 20, 2021 at 7:58 AM Peter Geoghegan wrote: > > On Sun, Jan 17, 2021 at 9:18 PM Masahiko Sawada wrote: > > After more thought, I think that ambulkdelete needs to be able to > > refer the answer to amvacuumstrategy. That way, the index can skip > > bulk-d

Re: About to add WAL write/fsync statistics to pg_stat_wal view

2021-01-21 Thread Masahiko Sawada
d. That way, this function is called at only once. So do we really need this function? * As far as I read the code, issue_xlog_fsync() seems to do fsync even if enableFsync is false. Why does the function return false in that case? I might be missing something. * void is missing as argument? * s/mothod/method/ Regards, -- Masahiko Sawada EDB: https://www.enterprisedb.com/

Re: doc review for v14

2021-01-22 Thread Masahiko Sawada
Can we mark this patch entry as "Committed"? Or waiting for something on the author? Regards, -- Masahiko Sawada EDB: https://www.enterprisedb.com/

Re: [PATCH] Keeps tracking the uniqueness with UniqueKey

2021-01-22 Thread Masahiko Sawada
e > the 2 situations without looking into the real/Poc code, even for very > experienced people. I will submit a new patch after I get the partitioned > relation, subquery works. Hope I can make it in one week. Status update for a commitfest entry. Are you planning to submit a new pat

Re: LogwrtResult contended spinlock

2021-01-22 Thread Masahiko Sawada
vance* a value rather than a generic swap. > > (I'm not 100% clear that that's the exact API we need.) > > > This CF entry was inactive for a while. Alvaro, are you going to continue > > working on it? > > Yes, please move it forward. I'll post an update sometime before the > next CF. Anything update on this? Regards, -- Masahiko Sawada EDB: https://www.enterprisedb.com/

Re: Error on failed COMMIT

2021-01-22 Thread Masahiko Sawada
we still need if any so that new reviewers can easily catch up? I think people who want to start reviewing are likely to search the patch marked as "Needs Review". So I think continuous updating and rebasing the patch would help get the patch reviewed also in terms of that. Regards,

Re: LogwrtResult contended spinlock

2021-01-22 Thread Masahiko Sawada
On Fri, Jan 22, 2021 at 10:39 PM Alvaro Herrera wrote: > > On 2021-Jan-22, Masahiko Sawada wrote: > > > On Wed, Nov 25, 2020 at 12:02 AM Alvaro Herrera > > wrote: > > > > Yes, please move it forward. I'll post an update sometime before the > >

Re: Commitfest 2021-01 Now in Progress

2021-01-22 Thread Masahiko Sawada
Hi all, On Fri, Jan 15, 2021 at 9:53 PM Masahiko Sawada wrote: > > Today I have gone over all the "Waiting on Author" patches and marked a few entries as "Needs Review" in cases where new versions had been posted. The current stats (and difference from the last repo

Re: Is Recovery actually paused?

2021-01-24 Thread Masahiko Sawada
On Sun, Jan 17, 2021 at 5:19 PM Dilip Kumar wrote: > > On Sat, Jan 16, 2021 at 8:59 AM Masahiko Sawada wrote: > > > > On Wed, Jan 13, 2021 at 9:20 PM Dilip Kumar wrote: > > > > > > On Wed, Jan 13, 2021 at 3:35 PM Dilip Kumar wrote: > > > > >

Re: About to add WAL write/fsync statistics to pg_stat_wal view

2021-01-24 Thread Masahiko Sawada
On Fri, Jan 22, 2021 at 10:05 PM Masahiro Ikeda wrote: > > On 2021-01-22 14:50, Masahiko Sawada wrote: > > On Fri, Dec 25, 2020 at 6:46 PM Masahiro Ikeda > > wrote: > >> > >> Hi, > >> > >> I rebased the patch to the master branch. > > &

Re: [PoC] Non-volatile WAL buffer

2021-01-24 Thread Masahiko Sawada
On Fri, Jan 22, 2021 at 11:32 AM Tomas Vondra wrote: > > > > On 1/21/21 3:17 AM, Masahiko Sawada wrote: > > On Thu, Jan 7, 2021 at 2:16 AM Tomas Vondra > > wrote: > >> > >> Hi, > >> > >> I think I've managed to get the 0002 patc

<    1   2   3   4   5   6   7   8   9   10   >