Get memory contexts of an arbitrary backend process

2020-08-31 Thread torikoshia
Hi, After commit 3e98c0bafb28de, we can display the usage of the memory contexts using pg_backend_memory_contexts system view. However, its target is limited to the process attached to the current session. As discussed in the thread[1], it'll be useful to make it possible to get the memory

Re: Get memory contexts of an arbitrary backend process

2020-09-10 Thread torikoshia
On 2020-09-04 21:46, Tomas Vondra wrote: On Fri, Sep 04, 2020 at 11:47:30AM +0900, Kasahara Tatsuhito wrote: On Fri, Sep 4, 2020 at 2:40 AM Tom Lane wrote: Kasahara Tatsuhito writes: > Yes, but it's not only for future expansion, but also for the > usability and the stability of this

Re: Get memory contexts of an arbitrary backend process

2020-09-03 Thread torikoshia
Thanks for reviewing! I'm going to modify the patch according to your comments. On 2020-09-01 10:54, Andres Freund wrote: Hi, On 2020-08-31 20:22:18 +0900, torikoshia wrote: After commit 3e98c0bafb28de, we can display the usage of the memory contexts using pg_backend_memory_contexts system

Re: Get memory contexts of an arbitrary backend process

2020-09-03 Thread torikoshia
On 2020-09-01 03:29, Pavel Stehule wrote: Hi po 31. 8. 2020 v 17:03 odesílatel Kasahara Tatsuhito napsal: Hi, On Mon, Aug 31, 2020 at 8:22 PM torikoshia wrote: As discussed in the thread[1], it'll be useful to make it possible to get the memory contexts of an arbitrary backend process

Re: Creating a function for exposing memory usage of backend process

2020-08-18 Thread torikoshia
On 2020-08-17 21:19, Fujii Masao wrote: On 2020/08/17 21:14, Fujii Masao wrote: On 2020-08-07 16:38, Kasahara Tatsuhito wrote: The following review has been posted through the commitfest application: make installcheck-world:  tested, passed Implements feature:   tested, passed Spec

Re: Creating a function for exposing memory usage of backend process

2020-08-18 Thread torikoshia
On 2020-08-18 22:54, Fujii Masao wrote: On 2020/08/18 18:41, torikoshia wrote: On 2020-08-17 21:19, Fujii Masao wrote: On 2020/08/17 21:14, Fujii Masao wrote: On 2020-08-07 16:38, Kasahara Tatsuhito wrote: The following review has been posted through the commitfest application: make

Re: Creating a function for exposing memory usage of backend process

2020-08-19 Thread torikoshia
On 2020-08-19 15:48, Fujii Masao wrote: On 2020/08/19 9:43, torikoshia wrote: On 2020-08-18 22:54, Fujii Masao wrote: On 2020/08/18 18:41, torikoshia wrote: On 2020-08-17 21:19, Fujii Masao wrote: On 2020/08/17 21:14, Fujii Masao wrote: On 2020-08-07 16:38, Kasahara Tatsuhito wrote

Re: Creating a function for exposing memory usage of backend process

2020-08-21 Thread torikoshia
Thanks for all your comments! Thankfully it seems that this feature is regarded as not meaningless one, I'm going to do some improvements. On Wed, Aug 19, 2020 at 10:56 PM Michael Paquier wrote: On Wed, Aug 19, 2020 at 06:12:02PM +0900, Fujii Masao wrote: On 2020/08/19 17:40, torikoshia

Re: Creating a function for exposing memory usage of backend process

2020-08-23 Thread torikoshia
On 2020-08-22 21:18, Michael Paquier wrote: Thanks for reviewing! On Fri, Aug 21, 2020 at 11:27:06PM +0900, torikoshia wrote: OK. Added a regression test on sysviews.sql. (0001-Added-a-regression-test-for-pg_backend_memory_contex.patch) Fujii-san gave us an example, but I added more simple

Re: Creating a function for exposing memory usage of backend process

2020-08-24 Thread torikoshia
On 2020-08-24 13:13, Fujii Masao wrote: On 2020/08/24 13:01, torikoshia wrote: On 2020-08-22 21:18, Michael Paquier wrote: Thanks for reviewing! On Fri, Aug 21, 2020 at 11:27:06PM +0900, torikoshia wrote: OK. Added a regression test on sysviews.sql. (0001-Added-a-regression-test

Re: Is it useful to record whether plans are generic or custom?

2020-09-28 Thread torikoshia
On 2020-09-17 13:46, Michael Paquier wrote: On Fri, Jul 31, 2020 at 06:47:48PM +0900, torikoshia wrote: Oops, sorry about that. I just fixed it there for now. The regression tests of the patch look unstable, and the CF bot is reporting a failure here: https://travis-ci.org/github/postgresql

Re: Get memory contexts of an arbitrary backend process

2020-09-25 Thread torikoshia
Hi, Thanks for all your comments, I updated the patch. On Tue, Sep 1, 2020 at 12:03 AM Kasahara Tatsuhito wrote: - How about managing the status of signal send/receive and dump operations on a shared hash or others ? Sending and receiving signals, dumping memory information, and

Re: Get memory contexts of an arbitrary backend process

2020-10-22 Thread torikoshia
On Thu, Oct 1, 2020 at 4:06 PM Kasahara Tatsuhito wrote: Hi, On Fri, Sep 25, 2020 at 4:28 PM torikoshia wrote: > Thanks for all your comments, I updated the patch. Thanks for updating the patch. I did a brief test and code review. Thanks for your tests and review! > I added a

Re: Get memory contexts of an arbitrary backend process

2020-10-28 Thread torikoshia
do as the follows and it's the way of git. v1-0001-Enabled...collect.patch v2-0001-Enabled...collect.patch The vn is added by -v option for git-format-patch. Sorry for the confusion. I'll follow that way next time. At Thu, 22 Oct 2020 21:32:00 +0900, torikoshia wrote in > > I added a

Re: Is it useful to record whether plans are generic or custom?

2020-07-22 Thread torikoshia
On 2020-07-20 13:57, torikoshia wrote: As I proposed earlier in this thread, I'm now trying to add information about generic/cudstom plan to pg_stat_statements. I'll share the idea and the poc patch soon. Attached a poc patch. Main purpose is to decide (1) the user interface and (2) the way

Re: Is it useful to record whether plans are generic or custom?

2020-07-31 Thread torikoshia
On 2020-07-30 14:31, Fujii Masao wrote: On 2020/07/22 16:49, torikoshia wrote: On 2020-07-20 13:57, torikoshia wrote: As I proposed earlier in this thread, I'm now trying to add information about generic/cudstom plan to pg_stat_statements. I'll share the idea and the poc patch soon

Re: Creating a function for exposing memory usage of backend process

2020-08-11 Thread torikoshia
On 2020-08-08 10:44, Michael Paquier wrote: On Fri, Jul 31, 2020 at 03:23:52PM -0400, Robert Haas wrote: On Fri, Jul 31, 2020 at 4:25 AM torikoshia wrote: And as Fujii-san told me in person, exposing memory address seems not preferable considering there are security techniques like address

Re: Creating a function for exposing memory usage of backend process

2020-07-01 Thread torikoshia
On Wed, Jul 1, 2020 at 4:43 PM Fujii Masao wrote: Thanks for reviewing! You treat pg_stat_local_memory_contexts view as a dynamic statistics view. But isn't it better to treat it as just system view like pg_shmem_allocations or pg_prepared_statements  because it's not statistics

Re: Creating a function for exposing memory usage of backend process

2020-07-01 Thread torikoshia
On 2020-07-01 20:47, Daniel Gustafsson wrote: For the next version (if there will be one), please remove the catversion bump from the patch as it will otherwise just break patch application without constant rebasing (as it's done now). The committer will handle the catversion change if/when

Modifying data type of slot_keep_segs from XLogRecPtr to XLogSegNo

2020-07-07 Thread torikoshia
Hi, Currently, slot_keep_segs is defined as "XLogRecPtr" in KeepLogSeg(), but it seems that should be "XLogSegNo" because this variable is segment number. How do you think? Regards, -- Atsushi Torikoshi NTT DATA CORPORATIONdiff --git a/src/backend/access/transam/xlog.c

Re: Modifying data type of slot_keep_segs from XLogRecPtr to XLogSegNo

2020-07-07 Thread torikoshia
On 2020-07-08 11:15, Fujii Masao wrote: On 2020/07/08 11:02, torikoshia wrote: Hi, Currently, slot_keep_segs is defined as "XLogRecPtr" in KeepLogSeg(), but it seems that should be "XLogSegNo" because this variable is segment number. How do you think? I agree t

[doc] modifying unit from characters to bytes

2020-07-07 Thread torikoshia
Hi, The manual describes the size of pg_stat_activity.query as below: | By default the query text is truncated at 1024 characters; When considering multibyte characters, it seems more accurate to change the unit from "characters" to "bytes". I also searched other "[0-9] characters" in the

Re: Is it useful to record whether plans are generic or custom?

2020-07-07 Thread torikoshia
On 2020-07-06 22:16, Fujii Masao wrote: On 2020/06/11 14:59, torikoshia wrote: On 2020-06-10 18:00, Kyotaro Horiguchi wrote: +    TupleDescInitEntry(tupdesc, (AttrNumber) 8, "last_plan", This could be a problem if we showed the last plan in this view.  I think "last_p

Re: Creating a function for exposing memory usage of backend process

2020-07-05 Thread torikoshia
On Fri, Jul 3, 2020 at 7:33 PM Fujii Masao wrote: Thanks for your review! I like more specific name like pg_backend_memory_contexts. Agreed. When I was trying to add this function as statistics function, I thought that naming pg_stat_getbackend_memory_context() might make people regarded

Re: [doc] modifying unit from characters to bytes

2020-07-09 Thread torikoshia
On 2020-07-09 13:47, Fujii Masao wrote: On 2020/07/08 17:12, Daniel Gustafsson wrote: On 8 Jul 2020, at 10:05, Fujii Masao wrote: On 2020/07/08 16:17, Daniel Gustafsson wrote: On 8 Jul 2020, at 04:25, Fujii Masao wrote: On 2020/07/08 10:54, torikoshia wrote: Hi, The manual describes

Re: Creating a function for exposing memory usage of backend process

2020-07-07 Thread torikoshia
On 2020-07-06 15:16, Fujii Masao wrote: On 2020/07/06 12:12, torikoshia wrote: On Fri, Jul 3, 2020 at 7:33 PM Fujii Masao wrote: Thanks for your review! I like more specific name like pg_backend_memory_contexts. Agreed. When I was trying to add this function as statistics function, I

Re: Creating a function for exposing memory usage of backend process

2020-07-10 Thread torikoshia
On 2020-07-08 22:12, Fujii Masao wrote: Thanks for updating the patch! It basically looks good to me. + + backend memory contexts + Do we need this indexterm? Thanks! it's not necessary. I remove this indexterm. +{ oid => '2282', descr => 'statistics: information about all memory

Re: Creating a function for exposing memory usage of backend process

2020-07-10 Thread torikoshia
On 2020-07-09 02:03, Andres Freund wrote: Hi, I think this is an incredibly useful feature. Thanks for your kind comments and suggestion! On 2020-07-07 22:02:10 +0900, torikoshia wrote: > There can be multiple memory contexts with the same name. So I'm afraid > that it's dif

Re: Is it useful to record whether plans are generic or custom?

2020-07-14 Thread torikoshia
On 2020-07-10 10:49, torikoshia wrote: On 2020-07-08 16:41, Fujii Masao wrote: On 2020/07/08 10:14, torikoshia wrote: On 2020-07-06 22:16, Fujii Masao wrote: On 2020/06/11 14:59, torikoshia wrote: On 2020-06-10 18:00, Kyotaro Horiguchi wrote: +    TupleDescInitEntry(tupdesc, (AttrNumber

Re: Creating a function for exposing memory usage of backend process

2020-07-02 Thread torikoshia
On Wed, Jul 1, 2020 at 10:15 PM torikoshia wrote: I'm going to do some renaming and transportations. - view name: pg_memory_contexts - function name: pg_get_memory_contexts() - source file: mainly src/backend/utils/mmgr/mcxt.c Attached an updated patch. On Wed, Jul 1, 2020 at 10:58 PM Fujii

Re: Creating a function for exposing memory usage of backend process

2020-06-30 Thread torikoshia
On Mon, Jun 29, 2020 at 3:13 PM Fujii Masao wrote: Could you add this patch to Commitfest 2020-07? Thanks for notifying, I've added it. BTW, I registered you as an author because this patch used lots of pg_cheat_funcs' codes. https://commitfest.postgresql.org/28/2622/ This patch

Re: Is it useful to record whether plans are generic or custom?

2020-07-09 Thread torikoshia
On 2020-07-08 16:41, Fujii Masao wrote: On 2020/07/08 10:14, torikoshia wrote: On 2020-07-06 22:16, Fujii Masao wrote: On 2020/06/11 14:59, torikoshia wrote: On 2020-06-10 18:00, Kyotaro Horiguchi wrote: +    TupleDescInitEntry(tupdesc, (AttrNumber) 8, "last_plan", This could be

Creating a function for exposing memory usage of backend process

2020-06-17 Thread torikoshia
Hi, As you may know better than I do, backend processes sometimes use a lot of memory because of the various reasons like caches, prepared statements and cursors. When supporting PostgreSQL, I face situations for investigating the reason of memory bloat. AFAIK, the way to examine it is

Re: Creating a function for exposing memory usage of backend process

2020-06-28 Thread torikoshia
On 2020-06-20 03:11, Robert Haas wrote: On Wed, Jun 17, 2020 at 11:56 PM Fujii Masao wrote: > As a first step, to deal with (3) I'd like to add > pg_stat_get_backend_memory_context() which target is limited to the > local backend process. +1 +1 from me, too. Attached a patch that adds a

Re: Is it useful to record whether plans are generic or custom?

2020-06-09 Thread torikoshia
On 2020-06-08 20:45, Masahiro Ikeda wrote: BTW, I found that the dependency between function's comments and the modified code is broken at latest patch. Before this is committed, please fix it. ``` diff --git a/src/backend/commands/prepare.c b/src/backend/commands/prepare.c index

Re: Is it useful to record whether plans are generic or custom?

2020-06-11 Thread torikoshia
On 2020-06-10 18:00, Kyotaro Horiguchi wrote: + TupleDescInitEntry(tupdesc, (AttrNumber) 8, "last_plan", This could be a problem if we showed the last plan in this view. I think "last_plan_type" would be better. + if (prep_stmt->plansource->last_plan_type == PLAN_CACHE_TYPE_CUSTOM)

Re: Is it useful to record whether plans are generic or custom?

2020-07-15 Thread torikoshia
On 2020-07-15 11:44, Fujii Masao wrote: On 2020/07/14 21:24, torikoshia wrote: On 2020-07-10 10:49, torikoshia wrote: On 2020-07-08 16:41, Fujii Masao wrote: On 2020/07/08 10:14, torikoshia wrote: On 2020-07-06 22:16, Fujii Masao wrote: On 2020/06/11 14:59, torikoshia wrote: On 2020-06-10

Re: Feature improvement: can we add queryId for pg_catalog.pg_stat_activity view?

2020-07-28 Thread torikoshia
On 2020-07-14 20:24, Julien Rouhaud wrote: On Tue, Jul 14, 2020 at 07:11:02PM +0900, Atsushi Torikoshi wrote: Hi, v9 patch fails to apply to HEAD, could you check and rebase it? Thanks for the notice, v10 attached! And here are minor typos. 79 +* utility statements. Note that we

Re: Is it useful to record whether plans are generic or custom?

2020-07-19 Thread torikoshia
On 2020-07-20 11:57, Fujii Masao wrote: On 2020/07/17 16:25, Fujii Masao wrote: On 2020/07/16 11:50, torikoshia wrote: On 2020-07-15 11:44, Fujii Masao wrote: On 2020/07/14 21:24, torikoshia wrote: On 2020-07-10 10:49, torikoshia wrote: On 2020-07-08 16:41, Fujii Masao wrote: On 2020/07

Re: Creating a function for exposing memory usage of backend process

2020-07-31 Thread torikoshia
On 2020-07-30 15:13, Kasahara Tatsuhito wrote: Hi, On Fri, Jul 10, 2020 at 5:32 PM torikoshia wrote: - whether information for identifying parent-child relation is necessary or it's an overkill I think it's important to understand the parent-child relationship of the context. Personally, I

Re: Get memory contexts of an arbitrary backend process

2021-01-12 Thread torikoshia
v7 that fixes recent conflicts. It also changed the behavior of requestor when another requestor is already working for simplicity. In this case, v6 patch makes the requestor wait. v7 patch makes the requestor quit. Regards, -- Atsushi TorikoshiFrom f20e48d99f2770bfec275805185aa5ce08661fce

Re: Is it useful to record whether plans are generic or custom?

2021-01-12 Thread torikoshia
wrote in ISTM now that creating pg_stat_statements_xxx views both for generic andcustom plans is better than my PoC patch. On my second thought, it also makes pg_stat_statements too complicated compared to what it makes possible.. I'm also worrying that whether taking generic and custom

Re: [doc] plan invalidation when statistics are update

2020-11-25 Thread torikoshia
On 2020-11-25 14:13, Fujii Masao wrote: On 2020/11/24 23:14, Fujii Masao wrote: On 2020/11/19 14:33, torikoshia wrote: On 2020-11-18 11:35, Fujii Masao wrote: Thanks for your comment! On 2020/11/18 11:04, torikoshia wrote: Hi, AFAIU, when the planner statistics are updated, generic

Re: Is it useful to record whether plans are generic or custom?

2020-12-03 Thread torikoshia
On 2020-12-04 14:29, Fujii Masao wrote: On 2020/11/30 15:24, Tatsuro Yamada wrote: Hi Torikoshi-san, In this patch, exposing new columns is mandatory, but I think it's better to make it optional by adding a GUC something like 'pgss.track_general_custom_plans. I also feel it makes the number

Re: Get memory contexts of an arbitrary backend process

2020-12-09 Thread torikoshia
On 2020-12-04 19:16, torikoshia wrote: On 2020-12-03 10:36, Tom Lane wrote: Fujii Masao writes: I'm starting to study how this feature behaves. At first, when I executed the following query, the function never returned. ISTM that since the autovacuum launcher cannot respond to the request

Re: Get memory contexts of an arbitrary backend process

2020-12-04 Thread torikoshia
On 2020-12-03 10:36, Tom Lane wrote: Fujii Masao writes: I'm starting to study how this feature behaves. At first, when I executed the following query, the function never returned. ISTM that since the autovacuum launcher cannot respond to the request of memory contexts dump, the function

adding wait_start column to pg_locks

2020-12-14 Thread torikoshia
Hi, When examining the duration of locks, we often do join on pg_locks and pg_stat_activity and use columns such as query_start or state_change. However, since these columns are the moment when queries have started or their state has changed, we cannot get the exact lock duration in this way.

Re: Is it useful to record whether plans are generic or custom?

2020-11-11 Thread torikoshia
On 2020-09-29 02:39, legrand legrand wrote: Hi Atsushi, +1: Your proposal is a good answer for time based performance analysis (even if parsing durationor blks are not differentiated) . As it makes pgss number of columns wilder, may be an other solution would be to create a

Re: Get memory contexts of an arbitrary backend process

2020-11-16 Thread torikoshia
On 2020-10-28 15:32, torikoshia wrote: On 2020-10-23 13:46, Kyotaro Horiguchi wrote: I think we might need to step-back to basic design of this feature since this patch seems to have unhandled corner cases that are difficult to find. I've written out the basic design below and attached

Re: [doc] plan invalidation when statistics are update

2020-11-18 Thread torikoshia
On 2020-11-18 11:35, Fujii Masao wrote: Thanks for your comment! On 2020/11/18 11:04, torikoshia wrote: Hi, AFAIU, when the planner statistics are updated, generic plans are invalidated and PostgreSQL recreates. However, the manual doesn't seem to explain it explicitly.   https

Re: [doc] adding way to examine the plan type of prepared statements

2020-11-18 Thread torikoshia
On 2020-11-18 11:04, torikoshia wrote: Hi, Currently, EXPLAIN is the only way to know whether the plan is generic or custom according to the manual of PREPARE. https://www.postgresql.org/docs/devel/sql-prepare.html After commit d05b172, we can also use pg_prepared_statements view

Re: Is it useful to record whether plans are generic or custom?

2020-11-17 Thread torikoshia
On 2020-11-12 14:23, Pavel Stehule wrote: yes, the plan self is very interesting information - and information if plan was generic or not is interesting too. It is other dimension of query - maybe there can be rule - for any query store max 100 most slows plans with all attributes. The next

[doc] adding way to examine the plan type of prepared statements

2020-11-17 Thread torikoshia
Hi, Currently, EXPLAIN is the only way to know whether the plan is generic or custom according to the manual of PREPARE. https://www.postgresql.org/docs/devel/sql-prepare.html After commit d05b172, we can also use pg_prepared_statements view to examine the plan types. How about adding

[doc] plan invalidation when statistics are update

2020-11-17 Thread torikoshia
Hi, AFAIU, when the planner statistics are updated, generic plans are invalidated and PostgreSQL recreates. However, the manual doesn't seem to explain it explicitly. https://www.postgresql.org/docs/devel/sql-prepare.html I guess this case is included in 'whenever database objects used in

Re: Get memory contexts of an arbitrary backend process

2021-01-14 Thread torikoshia
Since pg_get_target_backend_memory_contexts() waits to dump memory and it could lead dead lock as below. - session1 BEGIN; TRUNCATE t; - session2 BEGIN; TRUNCATE t; -- wait - session1 SELECT * FROM pg_get_target_backend_memory_contexts(2>); --wait Thanks for notifying me,

Re: adding wait_start column to pg_locks

2021-01-17 Thread torikoshia
On 2021-01-15 15:23, torikoshia wrote: Thanks for your reviewing and comments! On 2021-01-14 12:39, Ian Lawrence Barwick wrote: Looking at the code, this happens as the wait start time is being recorded in the lock record itself, so always contains the value reported by the latest lock

Re: adding wait_start column to pg_locks

2021-01-21 Thread torikoshia
On 2021-01-21 12:48, Fujii Masao wrote: Thanks for updating the patch! I think that this is really useful feature!! Thanks for reviewing! I have two minor comments. + role="column_definition"> + wait_start timestamptz The column name "wait_start" should be "waitstart" for the

Re: TOAST condition for column size

2021-01-20 Thread torikoshia
On 2021-01-19 19:32, Amit Kapila wrote: On Mon, Jan 18, 2021 at 7:53 PM torikoshia Because no benefit is to be expected by compressing it. The size will be mostly the same. Also, even if we somehow try to fit this data via toast, I think reading speed will be slower because for all such columns

TOAST condition for column size

2021-01-18 Thread torikoshia
Hi, When I created a table consisting of 400 VARCHAR columns and tried to INSERT a record which rows were all the same size, there were cases where I got an error due to exceeding the size limit per row. =# -- create a table consisting of 400 VARCHAR columns =# CREATE TABLE t1 (c1

Re: adding wait_start column to pg_locks

2021-01-14 Thread torikoshia
Thanks for your reviewing and comments! On 2021-01-14 12:39, Ian Lawrence Barwick wrote: Looking at the code, this happens as the wait start time is being recorded in the lock record itself, so always contains the value reported by the latest lock acquisition attempt. I think you are right

Re: adding wait_start column to pg_locks

2021-01-03 Thread torikoshia
On 2021-01-02 06:49, Justin Pryzby wrote: On Tue, Dec 15, 2020 at 11:47:23AM +0900, torikoshia wrote: So I'm now thinking about adding a new column in pg_locks which keeps the time at which locks started waiting. Attached a patch. This is failing make check-world, would you send an updated

Re: Get memory contexts of an arbitrary backend process

2021-01-03 Thread torikoshia
On Fri, Dec 25, 2020 at 6:08 PM Kasahara Tatsuhito wrote: Thanks for reviewing and kind suggestion! Attached a rewritten patch. Thanks for updating patch. But when I had applyed the patch to the current HEAD and did make, I got an error due to duplicate OIDs. You need to rebase the patch.

Re: Is it useful to record whether plans are generic or custom?

2021-02-03 Thread torikoshia
Chengxi Sun, Yamada-san, Horiguchi-san, Thanks for all your comments. Adding only the number of generic plan execution seems acceptable. On Mon, Jan 25, 2021 at 2:10 PM Kyotaro Horiguchi wrote: Note that ActivePortal is the closest nested portal. So it gives the wrong result for nested

Re: adding wait_start column to pg_locks

2021-02-04 Thread torikoshia
On 2021-02-03 11:23, Fujii Masao wrote: 64-bit fetches are not atomic on some platforms. So spinlock is necessary when updating "waitStart" without holding the partition lock? Also GetLockStatusData() needs spinlock when reading "waitStart"? Also it might be worth thinking to use 64-bit

Re: adding wait_start column to pg_locks

2021-02-02 Thread torikoshia
On 2021-01-25 23:44, Fujii Masao wrote: Another comment is; Doesn't the change of MyProc->waitStart need the lock table's partition lock? If yes, we can do that by moving LWLockRelease(partitionLock) just after the change of MyProc->waitStart, but which causes the time that lwlock is being held

Re: RFC: Logging plan of the running query

2021-06-14 Thread torikoshia
On 2021-06-11 01:20, Bharath Rupireddy wrote: Thanks for your review! On Wed, Jun 9, 2021 at 1:14 PM torikoshia wrote: Updated the patch. Thanks for the patch. Here are some comments on v3 patch: 1) We could just say "Requests to log query plan of the presently running query of a

Re: Delegating superuser tasks to new security roles

2021-06-14 Thread torikoshia
On 2021-05-26 05:33, Mark Dilger wrote: On May 13, 2021, at 12:30 PM, Mark Dilger wrote: On May 13, 2021, at 12:18 PM, Jacob Champion wrote: On Thu, 2021-05-13 at 11:42 -0700, Mark Dilger wrote: The distinction that Theme+Security would make is that capabilities can be categorized by

Re: RFC: Logging plan of the running query

2021-06-16 Thread torikoshia
On 2021-06-15 13:27, Bharath Rupireddy wrote: On Mon, Jun 14, 2021 at 5:48 PM torikoshia wrote: > 1) We could just say "Requests to log query plan of the presently > running query of a given backend along with an untruncated query > string in the server lo

Re: Delegating superuser tasks to new security roles

2021-06-14 Thread torikoshia
On 2021-06-14 23:53, Mark Dilger wrote: On Jun 14, 2021, at 5:51 AM, torikoshia wrote: Thanks for working on this topic, I appreciate it! Thank you for taking a look! BTW, do these patches enable non-superusers to create user with bypassrls? No, I did not break out the ability to create

Re: RFC: Logging plan of the running query

2021-06-09 Thread torikoshia
On 2021-05-28 15:51, torikoshia wrote: On 2021-05-13 21:57, Dilip Kumar wrote: On Thu, May 13, 2021 at 5:18 PM Dilip Kumar wrote: On Thu, May 13, 2021 at 5:15 PM Bharath Rupireddy wrote: > > On Thu, May 13, 2021 at 5:14 PM Dilip Kumar wrote: > > > > On Thu, May 13, 2021

Re: RFC: Logging plan of the running query

2021-06-21 Thread torikoshia
On 2021-06-16 20:36, torikoshia wrote: other background or parallel worker. As far as I looked around, there seems no easy ways to do so. If we were to invent a new mechanism just for addressing the above comment, I would rather choose to not do that as it seems like an overkill. We can

Re: RFC: Logging plan of the running query

2021-06-09 Thread torikoshia
On 2021-06-09 23:04, Fujii Masao wrote: Thanks for your review! auto_explain can log the plan of even nested statement if auto_explain.log_nested_statements is enabled. But ISTM that pg_log_current_plan() cannot log that plan. Is this intentional? I think that it's better to make

Re: Misplaced superuser check in pg_log_backend_memory_contexts()

2021-06-08 Thread torikoshia
On 2021-06-08 11:49, Michael Paquier wrote: On Sun, Jun 06, 2021 at 11:13:40AM -0400, Tom Lane wrote: Julien Rouhaud writes: However +1 for the patch, as it seems more consistent to always get a permission failure if you're not a superuser. Yeah, it's just weird if such a check is not the

RFC: Logging plan of the running query

2021-05-12 Thread torikoshia
Hi, During the discussion about memory contexts dumping[1], there was a comment that exposing not only memory contexts but also query plans and untruncated query string would be useful. I also feel that it would be nice when thinking about situations such as troubleshooting a long-running query

Re: wal stats questions

2021-05-17 Thread torikoshia
On 2021-05-13 09:05, Masahiro Ikeda wrote: On 2021/05/12 19:19, Fujii Masao wrote: On 2021/05/11 18:46, Masahiro Ikeda wrote: On 2021/05/11 16:44, Fujii Masao wrote: On 2021/04/28 9:10, Masahiro Ikeda wrote: On 2021/04/27 21:56, Fujii Masao wrote: On 2021/04/26 10:11, Masahiro

Re: RFC: Logging plan of the running query

2021-05-13 Thread torikoshia
On 2021-05-13 18:36, Bharath Rupireddy wrote: On Thu, May 13, 2021 at 2:57 PM Bharath Rupireddy wrote: On Thu, May 13, 2021 at 2:44 PM Dilip Kumar wrote: > +1 for the idea. I did not read the complete patch but while reading > through the patch, I noticed that you using elevel as LOG for

Re: RFC: Logging plan of the running query

2021-05-13 Thread torikoshia
Thank you all for your positive comments. On 2021-05-12 21:55, Matthias van de Meent wrote: Great idea. One feature I'd suggest would be adding a 'format' option as well, if such feature would be feasable. Thanks for the comment! During the development of pg_log_backend_memory_contexts(), I

Re: RFC: Logging plan of the running query

2021-05-13 Thread torikoshia
On 2021-05-13 01:08, Laurenz Albe wrote: On Wed, 2021-05-12 at 18:03 +0530, Bharath Rupireddy wrote: Since it also shows up the full query text and the plan in the server log as plain text, there are chances that the sensitive information might be logged into the server log which is a risky

Re: RFC: Logging plan of the running query

2021-05-28 Thread torikoshia
On 2021-05-13 21:57, Dilip Kumar wrote: On Thu, May 13, 2021 at 5:18 PM Dilip Kumar wrote: On Thu, May 13, 2021 at 5:15 PM Bharath Rupireddy wrote: > > On Thu, May 13, 2021 at 5:14 PM Dilip Kumar wrote: > > > > On Thu, May 13, 2021 at 4:16 PM Bharath Rupireddy > > wrote: > > > > > > I'm

Re: RFC: Logging plan of the running query

2021-07-08 Thread torikoshia
On 2021-07-02 23:21, Bharath Rupireddy wrote: On Tue, Jun 22, 2021 at 8:00 AM torikoshia wrote: Updated the patch. Thanks for the patch. Here are some comments on the v4 patch: Thanks for your comments and suggestions! I agree with you and updated the patch. On Thu, Jul 1, 2021 at 3:34 PM

Re: Is it useful to record whether plans are generic or custom?

2021-02-07 Thread torikoshia
On 2021-02-04 11:19, Kyotaro Horiguchi wrote: At Thu, 04 Feb 2021 10:16:47 +0900, torikoshia wrote in Chengxi Sun, Yamada-san, Horiguchi-san, Thanks for all your comments. Adding only the number of generic plan execution seems acceptable. On Mon, Jan 25, 2021 at 2:10 PM Kyotaro Horiguchi

Re: adding wait_start column to pg_locks

2021-02-09 Thread torikoshia
On 2021-02-05 18:49, Fujii Masao wrote: On 2021/02/05 0:03, torikoshia wrote: On 2021-02-03 11:23, Fujii Masao wrote: 64-bit fetches are not atomic on some platforms. So spinlock is necessary when updating "waitStart" without holding the partition lock? Also GetLockStatusData() need

Re: adding wait_start column to pg_locks

2021-02-09 Thread torikoshia
On 2021-02-09 22:54, Fujii Masao wrote: On 2021/02/09 19:11, Fujii Masao wrote: On 2021/02/09 18:13, Fujii Masao wrote: On 2021/02/09 17:48, torikoshia wrote: On 2021-02-05 18:49, Fujii Masao wrote: On 2021/02/05 0:03, torikoshia wrote: On 2021-02-03 11:23, Fujii Masao wrote: 64-bit

Re: Get memory contexts of an arbitrary backend process

2021-03-24 Thread torikoshia
On 2021-03-23 17:24, Kyotaro Horiguchi wrote: Thanks for reviewing and suggestions! At Mon, 22 Mar 2021 15:09:58 +0900, torikoshia wrote in >> If MemoryContextStatsPrint(), i.e. showing 100 children at most is >> enough, this hard limit may be acceptable. > Can't this numbe

Re: Is it useful to record whether plans are generic or custom?

2021-03-23 Thread torikoshia
On 2021-03-05 17:47, Fujii Masao wrote: Thanks for your comments! I just tried this feature. When I set plan_cache_mode to force_generic_plan and executed the following queries, I found that pg_stat_statements.generic_calls and pg_prepared_statements.generic_plans were not the same. Is this

Re: Get memory contexts of an arbitrary backend process

2021-03-25 Thread torikoshia
On 2021-03-25 22:02, Fujii Masao wrote: On 2021/03/25 0:17, torikoshia wrote: On 2021-03-23 17:24, Kyotaro Horiguchi wrote: Thanks for reviewing and suggestions! The patched version failed to be compiled as follows. Could you fix this issue? Sorry, it included a header file that's

Re: Is it useful to record whether plans are generic or custom?

2021-03-25 Thread torikoshia
On 2021-03-25 22:14, Fujii Masao wrote: On 2021/03/23 16:32, torikoshia wrote: On 2021-03-05 17:47, Fujii Masao wrote: Thanks for your comments! Thanks for updating the patch! PostgreSQL Patch Tester reported that the patched version failed to be compiled at Windows. Could you fix

Re: Get memory contexts of an arbitrary backend process

2021-03-30 Thread torikoshia
On 2021-03-30 02:28, Fujii Masao wrote: Thanks for reviewing and kind suggestions! It adds pg_log_backend_memory_contexts(pid) which logs memory contexts of the specified backend process. The number of child contexts to be logged per parent is limited to 100 as with MemoryContextStats(). As

Re: Get memory contexts of an arbitrary backend process

2021-03-28 Thread torikoshia
On 2021-03-26 14:08, Kyotaro Horiguchi wrote: At Fri, 26 Mar 2021 14:02:49 +0900, Fujii Masao wrote in On 2021/03/26 13:28, Kyotaro Horiguchi wrote: >> "some contexts are omitted" >> "n child contexts: total_bytes = ..." > Sorry I missed that is already implemented. So my opnion is I agree

Re: Get memory contexts of an arbitrary backend process

2021-03-31 Thread torikoshia
On 2021-03-31 04:36, Fujii Masao wrote: On 2021/03/30 22:06, torikoshia wrote: Modified the patch according to the suggestions. Thanks for updating the patch! I applied the cosmetic changes to the patch and added the example of the function call into the document. Attached is the updated

Re: Get memory contexts of an arbitrary backend process

2021-04-05 Thread torikoshia
On 2021-04-06 00:08, Fujii Masao wrote: On 2021/04/05 21:03, torikoshia wrote: On 2021-04-05 12:59, Fujii Masao wrote: On 2021/04/05 12:20, Zhihong Yu wrote: Thanks for reviewing! + * On receipt of this signal, a backend sets the flag in the signal + * handler, and then which causes

Re: Get memory contexts of an arbitrary backend process

2021-04-04 Thread torikoshia
On 2021-04-01 19:13, Fujii Masao wrote: On 2021/03/31 15:16, Kyotaro Horiguchi wrote: + The memory contexts will be logged based on the log configuration set. For example: How do you think? How about "The memory contexts will be logged in the server log" ? I think "server log" doesn't

Re: Get memory contexts of an arbitrary backend process

2021-04-05 Thread torikoshia
On 2021-04-05 12:59, Fujii Masao wrote: On 2021/04/05 12:20, Zhihong Yu wrote: Thanks for reviewing! + * On receipt of this signal, a backend sets the flag in the signal + * handler, and then which causes the next CHECK_FOR_INTERRUPTS() I think the 'and then' is not needed: Although I

Re: Is it useful to record whether plans are generic or custom?

2021-04-05 Thread torikoshia
On 2021-03-26 17:46, Fujii Masao wrote: On 2021/03/26 0:33, torikoshia wrote: On 2021-03-25 22:14, Fujii Masao wrote: On 2021/03/23 16:32, torikoshia wrote: On 2021-03-05 17:47, Fujii Masao wrote: Thanks for your comments! Thanks for updating the patch! PostgreSQL Patch Tester reported

Re: Should we improve "PID XXXX is not a PostgreSQL server process" warning for pg_terminate_backend(<>)?

2021-03-14 Thread torikoshia
On 2021-03-07 19:16, Bharath Rupireddy wrote: On Fri, Feb 5, 2021 at 5:15 PM Bharath Rupireddy wrote: pg_terminate_backend and pg_cancel_backend with postmaster PID produce "PID is not a PostgresSQL server process" warning [1], which basically implies that the postmaster is not a

Re: Should we improve "PID XXXX is not a PostgreSQL server process" warning for pg_terminate_backend(<>)?

2021-03-16 Thread torikoshia
On 2021-03-16 20:51, Bharath Rupireddy wrote: On Mon, Mar 15, 2021 at 11:23 AM torikoshia wrote: On 2021-03-07 19:16, Bharath Rupireddy wrote: > On Fri, Feb 5, 2021 at 5:15 PM Bharath Rupireddy > wrote: >> >> pg_terminate_backend and pg_cancel_backend with postmaster PID pr

Re: Get memory contexts of an arbitrary backend process

2021-03-17 Thread torikoshia
On 2021-03-05 14:22, Fujii Masao wrote: On 2021/03/04 18:32, torikoshia wrote: On 2021-01-14 19:11, torikoshia wrote: Since pg_get_target_backend_memory_contexts() waits to dump memory and it could lead dead lock as below.   - session1   BEGIN; TRUNCATE t;   - session2   BEGIN; TRUNCATE t

Re: Get memory contexts of an arbitrary backend process

2021-03-22 Thread torikoshia
On 2021-03-18 15:09, Fujii Masao wrote: Thanks for your comments! On 2021/03/17 22:24, torikoshia wrote: I remade the patch and introduced a function pg_print_backend_memory_contexts(PID) which prints the memory contexts of the specified PID to elog. Thanks for the patch!   =# SELECT

Re: Get memory contexts of an arbitrary backend process

2021-03-04 Thread torikoshia
On 2021-01-14 19:11, torikoshia wrote: Since pg_get_target_backend_memory_contexts() waits to dump memory and it could lead dead lock as below. - session1 BEGIN; TRUNCATE t; - session2 BEGIN; TRUNCATE t; -- wait - session1 SELECT * FROM pg_get_target_backend_memory_contexts

Re: Printing backtrace of postgres processes

2021-02-28 Thread torikoshia
Hi, I also think this feature would be useful when supporting environments that lack debugger or debug symbols. I think such environments are not rare. + for more information. This +will help in identifying where exactly the backend process is currently +executing.

  1   2   3   >