Memory leak fix in psql
Hi I think there is a newly introduced memory leak in your patch d2d3547. Try to fix it in the attached patch. Kindly to have a check. Regards, Tang v1-0001-fix-the-memory-leak-in-validateSQLNamePattern.patch Description: v1-0001-fix-the-memory-leak-in-validateSQLNamePattern.patch
Re: Memory leak fix in psql
On Tue, 19 Jul 2022 at 17:02, tanghy.f...@fujitsu.com wrote: > Hi > > I think there is a newly introduced memory leak in your patch d2d3547. > Try to fix it in the attached patch. > Kindly to have a check. > Yeah, it leaks, and the patch can fix it. After looking around, I found psql/describe.c also has some memory leaks, attached a patch to fix these leaks. -- Regrads, Japin Li. ChengDu WenWu Information Technology Co.,Ltd. >From 3036a0986f8ff490c133930524e2d5f5104249ff Mon Sep 17 00:00:00 2001 From: Japin Li Date: Tue, 19 Jul 2022 18:27:25 +0800 Subject: [PATCH v1 1/1] Fix the memory leak in psql describe --- src/bin/psql/describe.c | 150 1 file changed, 150 insertions(+) diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 0ce38e4b4c..11a441f52f 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -112,7 +112,10 @@ describeAggregates(const char *pattern, bool verbose, bool showSystem) "n.nspname", "p.proname", NULL, "pg_catalog.pg_function_is_visible(p.oid)", NULL, 3)) + { + termPQExpBuffer(&buf); return false; + } appendPQExpBufferStr(&buf, "ORDER BY 1, 2, 4;"); @@ -182,7 +185,10 @@ describeAccessMethods(const char *pattern, bool verbose) NULL, "amname", NULL, NULL, NULL, 1)) + { + termPQExpBuffer(&buf); return false; + } appendPQExpBufferStr(&buf, "ORDER BY 1;"); @@ -244,7 +250,10 @@ describeTablespaces(const char *pattern, bool verbose) NULL, "spcname", NULL, NULL, NULL, 1)) + { + termPQExpBuffer(&buf); return false; + } appendPQExpBufferStr(&buf, "ORDER BY 1;"); @@ -534,7 +543,10 @@ describeFunctions(const char *functypes, const char *func_pattern, "n.nspname", "p.proname", NULL, "pg_catalog.pg_function_is_visible(p.oid)", NULL, 3)) + { + termPQExpBuffer(&buf); return false; + } for (int i = 0; i < num_arg_patterns; i++) { @@ -561,7 +573,10 @@ describeFunctions(const char *functypes, const char *func_pattern, true, false, nspname, typname, ft, tiv, NULL, 3)) + { +termPQExpBuffer(&buf); return false; + } } else { @@ -682,7 +697,10 @@ describeTypes(const char *pattern, bool verbose, bool showSystem) "pg_catalog.format_type(t.oid, NULL)", "pg_catalog.pg_type_is_visible(t.oid)", NULL, 3)) + { + termPQExpBuffer(&buf); return false; + } appendPQExpBufferStr(&buf, "ORDER BY 1, 2;"); @@ -836,7 +854,10 @@ describeOperators(const char *oper_pattern, "n.nspname", "o.oprname", NULL, "pg_catalog.pg_operator_is_visible(o.oid)", NULL, 3)) + { + termPQExpBuffer(&buf); return false; + } if (num_arg_patterns == 1) appendPQExpBufferStr(&buf, " AND o.oprleft = 0\n"); @@ -866,7 +887,10 @@ describeOperators(const char *oper_pattern, true, false, nspname, typname, ft, tiv, NULL, 3)) + { +termPQExpBuffer(&buf); return false; + } } else { @@ -953,7 +977,10 @@ listAllDbs(const char *pattern, bool verbose) if (!validateSQLNamePattern(&buf, pattern, false, false, NULL, "d.datname", NULL, NULL, NULL, 1)) + { + termPQExpBuffer(&buf); return false; + } appendPQExpBufferStr(&buf, "ORDER BY 1;"); res = PSQLexec(buf.data); @@ -1106,7 +1133,10 @@ permissionsList(const char *pattern) "n.nspname", "c.relname", NULL, "n.nspname !~ '^pg_' AND pg_catalog.pg_table_is_visible(c.oid)", NULL, 3)) + { + termPQExpBuffer(&buf); return false; + } appendPQExpBufferStr(&buf, "ORDER BY 1, 2;"); @@ -1177,7 +1207,10 @@ listDefaultACLs(const char *pattern) "pg_catalog.pg_get_userbyid(d.defaclrole)", NULL, NULL, 3)) + { + termPQExpBuffer(&buf); return false; + } appendPQExpBufferStr(&buf, "ORDER BY 1, 2, 3;"); @@ -1428,7 +1461,10 @@ describeTableDetails(const char *pattern, bool verbose, bool showSystem) "n.nspname", "c.relname", NULL, "pg_catalog.pg_table_is_visible(c.oid)", NULL, 3)) + { + termPQExpBuffer(&buf); return false; + } appendPQExpBufferStr(&buf, "ORDER BY 2, 3;"); @@ -3614,7 +3650,10 @@ describeRoles(const char *pattern, bool verbose, bool showSystem) if (!validateSQLNamePattern(&buf, pattern, false, false, NULL, "r.rolname", NULL, NULL, NULL, 1)) + { + termPQExpBuffer(&buf); return false; + } appendPQExpBufferStr(&buf, "ORDER BY 1;"); @@ -3739,11 +3778,17 @@ listDbRoleSettings(const char *pattern, const char *pattern2) gettext_noop("Settings")); if (!validateSQLNamePattern(&buf, pattern, false, false, NULL, "r.rolname", NULL, NULL, &havewhere, 1)) + { + termPQExpBuffer(&buf); return false; + } if (!validateSQLNamePattern(&buf, pattern2, havewhere, false, NULL, "d.datname", NULL, NULL, NULL, 1)) + { + termPQE
Re: Memory leak fix in psql
On Tue, Jul 19, 2022 at 06:41:13PM +0800, Japin Li wrote: > After looking around, I found psql/describe.c also has some memory leaks, > attached a patch to fix these leaks. Indeed. There are quite a bit of them, so let's fix all that. You have missed a couple of code paths in objectDescription(). -- Michael signature.asc Description: PGP signature
Re: Memory leak fix in psql
On Tue, 19 Jul 2022 at 20:32, Michael Paquier wrote: > On Tue, Jul 19, 2022 at 06:41:13PM +0800, Japin Li wrote: >> After looking around, I found psql/describe.c also has some memory leaks, >> attached a patch to fix these leaks. > > Indeed. There are quite a bit of them, so let's fix all that. You > have missed a couple of code paths in objectDescription(). Thanks for reviewing. Attached fix the memory leak in objectDescription(). Please consider v2 for further review. -- Regrads, Japin Li. ChengDu WenWu Information Technology Co.,Ltd. >From b2bcc3a1bac67b8b414f2025607f8dd35e096289 Mon Sep 17 00:00:00 2001 From: Japin Li Date: Tue, 19 Jul 2022 18:27:25 +0800 Subject: [PATCH v2 1/1] Fix the memory leak in psql describe --- src/bin/psql/describe.c | 168 1 file changed, 168 insertions(+) diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 0ce38e4b4c..7a070a6cd0 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -112,7 +112,10 @@ describeAggregates(const char *pattern, bool verbose, bool showSystem) "n.nspname", "p.proname", NULL, "pg_catalog.pg_function_is_visible(p.oid)", NULL, 3)) + { + termPQExpBuffer(&buf); return false; + } appendPQExpBufferStr(&buf, "ORDER BY 1, 2, 4;"); @@ -182,7 +185,10 @@ describeAccessMethods(const char *pattern, bool verbose) NULL, "amname", NULL, NULL, NULL, 1)) + { + termPQExpBuffer(&buf); return false; + } appendPQExpBufferStr(&buf, "ORDER BY 1;"); @@ -244,7 +250,10 @@ describeTablespaces(const char *pattern, bool verbose) NULL, "spcname", NULL, NULL, NULL, 1)) + { + termPQExpBuffer(&buf); return false; + } appendPQExpBufferStr(&buf, "ORDER BY 1;"); @@ -534,7 +543,10 @@ describeFunctions(const char *functypes, const char *func_pattern, "n.nspname", "p.proname", NULL, "pg_catalog.pg_function_is_visible(p.oid)", NULL, 3)) + { + termPQExpBuffer(&buf); return false; + } for (int i = 0; i < num_arg_patterns; i++) { @@ -561,7 +573,10 @@ describeFunctions(const char *functypes, const char *func_pattern, true, false, nspname, typname, ft, tiv, NULL, 3)) + { +termPQExpBuffer(&buf); return false; + } } else { @@ -682,7 +697,10 @@ describeTypes(const char *pattern, bool verbose, bool showSystem) "pg_catalog.format_type(t.oid, NULL)", "pg_catalog.pg_type_is_visible(t.oid)", NULL, 3)) + { + termPQExpBuffer(&buf); return false; + } appendPQExpBufferStr(&buf, "ORDER BY 1, 2;"); @@ -836,7 +854,10 @@ describeOperators(const char *oper_pattern, "n.nspname", "o.oprname", NULL, "pg_catalog.pg_operator_is_visible(o.oid)", NULL, 3)) + { + termPQExpBuffer(&buf); return false; + } if (num_arg_patterns == 1) appendPQExpBufferStr(&buf, " AND o.oprleft = 0\n"); @@ -866,7 +887,10 @@ describeOperators(const char *oper_pattern, true, false, nspname, typname, ft, tiv, NULL, 3)) + { +termPQExpBuffer(&buf); return false; + } } else { @@ -953,7 +977,10 @@ listAllDbs(const char *pattern, bool verbose) if (!validateSQLNamePattern(&buf, pattern, false, false, NULL, "d.datname", NULL, NULL, NULL, 1)) + { + termPQExpBuffer(&buf); return false; + } appendPQExpBufferStr(&buf, "ORDER BY 1;"); res = PSQLexec(buf.data); @@ -1106,7 +1133,10 @@ permissionsList(const char *pattern) "n.nspname", "c.relname", NULL, "n.nspname !~ '^pg_' AND pg_catalog.pg_table_is_visible(c.oid)", NULL, 3)) + { + termPQExpBuffer(&buf); return false; + } appendPQExpBufferStr(&buf, "ORDER BY 1, 2;"); @@ -1177,7 +1207,10 @@ listDefaultACLs(const char *pattern) "pg_catalog.pg_get_userbyid(d.defaclrole)", NULL, NULL, 3)) + { + termPQExpBuffer(&buf); return false; + } appendPQExpBufferStr(&buf, "ORDER BY 1, 2, 3;"); @@ -1253,7 +1286,10 @@ objectDescription(const char *pattern, bool showSystem) false, "n.nspname", "pgc.conname", NULL, "pg_catalog.pg_table_is_visible(c.oid)", NULL, 3)) + { + termPQExpBuffer(&buf); return false; + } /* Domain constraint descriptions */ appendPQExpBuffer(&buf, @@ -1277,7 +1313,10 @@ objectDescription(const char *pattern, bool showSystem) false, "n.nspname", "pgc.conname", NULL, "pg_catalog.pg_type_is_visible(t.oid)", NULL, 3)) + { + termPQExpBuffer(&buf); return false; + } /* Operator class descriptions */ appendPQExpBuffer(&buf, @@ -1301,7 +1340,10 @@ objectDescription(const char *pattern, bool showSystem) "n.nspname", "o.opcname", NULL, "pg_catalog.pg_opclass_is_visible(o.oid)", NULL, 3)) + { + termPQExpBuffer(&buf); return false; + } /* Operator family descriptions */ appendPQExpBuffe
Re: Memory leak fix in psql
Hi, On 2022-07-19 21:08:53 +0800, Japin Li wrote: > From b2bcc3a1bac67b8b414f2025607f8dd35e096289 Mon Sep 17 00:00:00 2001 > From: Japin Li > Date: Tue, 19 Jul 2022 18:27:25 +0800 > Subject: [PATCH v2 1/1] Fix the memory leak in psql describe > > --- > src/bin/psql/describe.c | 168 > 1 file changed, 168 insertions(+) > > diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c > index 0ce38e4b4c..7a070a6cd0 100644 > --- a/src/bin/psql/describe.c > +++ b/src/bin/psql/describe.c > @@ -112,7 +112,10 @@ describeAggregates(const char *pattern, bool verbose, > bool showSystem) > "n.nspname", > "p.proname", NULL, > > "pg_catalog.pg_function_is_visible(p.oid)", > NULL, 3)) > + { > + termPQExpBuffer(&buf); > return false; > + } > > appendPQExpBufferStr(&buf, "ORDER BY 1, 2, 4;"); Adding copy over copy of this same block doesn't seem great. Can we instead add a helper for it or such? Greetings, Andres Freund
Re: Memory leak fix in psql
Andres Freund writes: > On 2022-07-19 21:08:53 +0800, Japin Li wrote: >> +{ >> +termPQExpBuffer(&buf); >> return false; >> +} > Adding copy over copy of this same block doesn't seem great. Can we instead > add a helper for it or such? The usual style in these files is something like if (bad things happened) goto fail; ... fail: termPQExpBuffer(&buf); return false; Yeah, it's old school, but please let's not have a few functions that do it randomly differently from all their neighbors. regards, tom lane
Re: Memory leak fix in psql
> On Jul 19, 2022, at 2:02 AM, tanghy.f...@fujitsu.com wrote: > > I think there is a newly introduced memory leak in your patch d2d3547. I agree. Thanks for noticing, and for the patch! > Try to fix it in the attached patch. > Kindly to have a check. This looks ok, but comments down-thread seem reasonable, so I suspect a new patch will be needed. Would you like to author it, or would you prefer that I, as the guilty party, do so? — Mark Dilger EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company
Re: Memory leak fix in psql
On Tue, Jul 19, 2022 at 12:36:31PM -0400, Tom Lane wrote: > Yeah, it's old school, but please let's not have a few functions that > do it randomly differently from all their neighbors. True enough. And it is not like we should free the PQExpBuffer given by the caller in validateSQLNamePattern(). -- Michael signature.asc Description: PGP signature
Re: Memory leak fix in psql
On Tue, Jul 19, 2022 at 09:43:21AM -0700, Mark Dilger wrote: > This looks ok, but comments down-thread seem reasonable, so I > suspect a new patch will be needed. Would you like to author it, or > would you prefer that I, as the guilty party, do so? If any of you could update the patch, that would be great. Thanks! -- Michael signature.asc Description: PGP signature
RE: Memory leak fix in psql
On Tuesday, July 19, 2022 7:41 PM, Japin Li wrote: > After looking around, I found psql/describe.c also has some memory > leaks, > attached a patch to fix these leaks. Thanks for your check and improvement. Your fix LGTM so please allow me to merge it in the attached patch. Based on your rebased version, now this new patch version is V3. Regards, Tang v3-0001-fix-the-memory-leak-in-psql-describe.patch Description: v3-0001-fix-the-memory-leak-in-psql-describe.patch
Re: Memory leak fix in psql
On Wed, Jul 20, 2022 at 03:14:35AM +, tanghy.f...@fujitsu.com wrote: > Your fix LGTM so please allow me to merge it in the attached patch. > Based on your rebased version, now this new patch version is V3. What about the argument of upthread where we could use a goto in functions where there are multiple pattern validation checks? Per se v4 attached. -- Michael From ccc8bb83baf618ea1a481193403a9009cd4a24a5 Mon Sep 17 00:00:00 2001 From: Michael Paquier Date: Wed, 20 Jul 2022 12:47:52 +0900 Subject: [PATCH v4] fix the memory leak in psql describe --- src/bin/psql/describe.c | 239 +++- 1 file changed, 212 insertions(+), 27 deletions(-) diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 88d92a08ae..1c40cc6d59 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -112,7 +112,10 @@ describeAggregates(const char *pattern, bool verbose, bool showSystem) "n.nspname", "p.proname", NULL, "pg_catalog.pg_function_is_visible(p.oid)", NULL, 3)) + { + termPQExpBuffer(&buf); return false; + } appendPQExpBufferStr(&buf, "ORDER BY 1, 2, 4;"); @@ -182,7 +185,10 @@ describeAccessMethods(const char *pattern, bool verbose) NULL, "amname", NULL, NULL, NULL, 1)) + { + termPQExpBuffer(&buf); return false; + } appendPQExpBufferStr(&buf, "ORDER BY 1;"); @@ -244,7 +250,10 @@ describeTablespaces(const char *pattern, bool verbose) NULL, "spcname", NULL, NULL, NULL, 1)) + { + termPQExpBuffer(&buf); return false; + } appendPQExpBufferStr(&buf, "ORDER BY 1;"); @@ -534,7 +543,9 @@ describeFunctions(const char *functypes, const char *func_pattern, "n.nspname", "p.proname", NULL, "pg_catalog.pg_function_is_visible(p.oid)", NULL, 3)) - return false; + { + goto error_return; + } for (int i = 0; i < num_arg_patterns; i++) { @@ -561,7 +572,9 @@ describeFunctions(const char *functypes, const char *func_pattern, true, false, nspname, typname, ft, tiv, NULL, 3)) -return false; + { +goto error_return; + } } else { @@ -599,6 +612,10 @@ describeFunctions(const char *functypes, const char *func_pattern, PQclear(res); return true; + +error_return: + termPQExpBuffer(&buf); + return false; } @@ -682,7 +699,10 @@ describeTypes(const char *pattern, bool verbose, bool showSystem) "pg_catalog.format_type(t.oid, NULL)", "pg_catalog.pg_type_is_visible(t.oid)", NULL, 3)) + { + termPQExpBuffer(&buf); return false; + } appendPQExpBufferStr(&buf, "ORDER BY 1, 2;"); @@ -836,7 +856,9 @@ describeOperators(const char *oper_pattern, "n.nspname", "o.oprname", NULL, "pg_catalog.pg_operator_is_visible(o.oid)", NULL, 3)) - return false; + { + goto error_return; + } if (num_arg_patterns == 1) appendPQExpBufferStr(&buf, " AND o.oprleft = 0\n"); @@ -866,7 +888,9 @@ describeOperators(const char *oper_pattern, true, false, nspname, typname, ft, tiv, NULL, 3)) -return false; + { +goto error_return; + } } else { @@ -890,6 +914,10 @@ describeOperators(const char *oper_pattern, PQclear(res); return true; + +error_return: + termPQExpBuffer(&buf); + return false; } @@ -953,7 +981,10 @@ listAllDbs(const char *pattern, bool verbose) if (!validateSQLNamePattern(&buf, pattern, false, false, NULL, "d.datname", NULL, NULL, NULL, 1)) + { + termPQExpBuffer(&buf); return false; + } appendPQExpBufferStr(&buf, "ORDER BY 1;"); res = PSQLexec(buf.data); @@ -1106,7 +1137,10 @@ permissionsList(const char *pattern) "n.nspname", "c.relname", NULL, "n.nspname !~ '^pg_' AND pg_catalog.pg_table_is_visible(c.oid)", NULL, 3)) + { + termPQExpBuffer(&buf); return false; + } appendPQExpBufferStr(&buf, "ORDER BY 1, 2;"); @@ -1177,16 +1211,15 @@ listDefaultACLs(const char *pattern) "pg_catalog.pg_get_userbyid(d.defaclrole)", NULL, NULL, 3)) - return false; + { + goto error_return; + } appendPQExpBufferStr(&buf, "ORDER BY 1, 2, 3;"); res = PSQLexec(buf.data); if (!res) - { - termPQExpBuffer(&buf); - return false; - } + goto error_return; myopt.nullPrint = NULL; printfPQExpBuffer(&buf, _("Default access privileges")); @@ -1200,6 +1233,10 @@ listDefaultACLs(const char *pattern) termPQExpBuffer(&buf); PQclear(res); return true; + +error_return: + termPQExpBuffer(&buf); + return false; } @@ -1253,7 +1290,9 @@ objectDescription(const char *pattern, bool showSystem) false, "n.nspname", "pgc.conname", NULL, "pg_catalog.pg_table_is_visible(c.oid)", NULL, 3)) - return false; + { + goto error_return; + } /* Domain constraint descriptions */ appendPQExpBuffer(&buf, @@ -1277,7 +1316,9 @@ objectDescription(const char *pattern, bool show
Re: Memory leak fix in psql
On Wed, 20 Jul 2022 at 11:51, Michael Paquier wrote: > On Wed, Jul 20, 2022 at 03:14:35AM +, tanghy.f...@fujitsu.com wrote: >> Your fix LGTM so please allow me to merge it in the attached patch. >> Based on your rebased version, now this new patch version is V3. > > What about the argument of upthread where we could use a goto in > functions where there are multiple pattern validation checks? Per se > v4 attached. +1. LGTM. -- Regrads, Japin Li. ChengDu WenWu Information Technology Co.,Ltd.
Re: Memory leak fix in psql
-1 Though the patch looks good, I myself think the patch should be edited and submitted by Tang It's easy to attach a fixed patch based on the comments of the thread, but coins should be given to Tang since he is the first one to find the mem leak. No offense, but that's what I think how open source works ;) On Wed, Jul 20, 2022 at 11:51 AM Michael Paquier wrote: > > On Wed, Jul 20, 2022 at 03:14:35AM +, tanghy.f...@fujitsu.com wrote: > > Your fix LGTM so please allow me to merge it in the attached patch. > > Based on your rebased version, now this new patch version is V3. > > What about the argument of upthread where we could use a goto in > functions where there are multiple pattern validation checks? Per se > v4 attached. > -- > Michael -- Regards Junwang Zhao
Re: Memory leak fix in psql
On Wed, Jul 20, 2022 at 12:51:24PM +0800, Junwang Zhao wrote: > Though the patch looks good, I myself think the patch should be edited > and submitted by Tang > It's easy to attach a fixed patch based on the comments of the thread, > but coins should be > given to Tang since he is the first one to find the mem leak. Please note that I sometimes edit slightly patches that I finish to merge into the tree, where the author listed in the commit log is the same as the original while I usually don't list mine. Credit goes where it should, and Tang is the one here who authored this patch. -- Michael signature.asc Description: PGP signature
Re: Memory leak fix in psql
Got it, thanks! On Wed, Jul 20, 2022 at 1:14 PM Michael Paquier wrote: > > On Wed, Jul 20, 2022 at 12:51:24PM +0800, Junwang Zhao wrote: > > Though the patch looks good, I myself think the patch should be edited > > and submitted by Tang > > It's easy to attach a fixed patch based on the comments of the thread, > > but coins should be > > given to Tang since he is the first one to find the mem leak. > > Please note that I sometimes edit slightly patches that I finish to > merge into the tree, where the author listed in the commit log is the > same as the original while I usually don't list mine. Credit goes > where it should, and Tang is the one here who authored this patch. > -- > Michael -- Regards Junwang Zhao
RE: Memory leak fix in psql
On Wednesday, July 20, 2022 12:52 PM, Michael Paquier wrote: > What about the argument of upthread where we could use a goto in > functions where there are multiple pattern validation checks? Per se > v4 attached. Thanks for your kindly remind and modification. I checked v4 patch, it looks good but I think there can be some minor improvement. So I deleted some redundant braces around "goto error_return; ". Also added an error handle section in validateSQLNamePattern. Kindly to have a check at the attached v5 patch. Regards, Tang v5-0001-fix-the-memory-leak-in-psql-describe.patch Description: v5-0001-fix-the-memory-leak-in-psql-describe.patch
RE: Memory leak fix in psql
> On Wed, Jul 20, 2022 at 12:51:24PM +0800, Junwang Zhao wrote: > > Though the patch looks good, I myself think the patch should be edited > > and submitted by Tang > > It's easy to attach a fixed patch based on the comments of the thread, > > but coins should be > > given to Tang since he is the first one to find the mem leak. Hello, Zhao Thanks for your check at this patch. I appreciate your kindly comment but there may be a misunderstanding here. As Michael explained, committers in Postgres will review carefully and help to improve contributors' patches. When the patch is finally committed by one committer, from what I can see, he or she will try to make sure the credit goes with everyone who contributed to the committed patch(such as bug reporter, patch author, tester, reviewer etc.). Also, developers and reviewers will try to help improving our proposed patch by rebasing it or adding an on-top patch(like Japin Li did in v2). These will make the patch better and to be committed ASAP. Good to see you at Postgres community. Regards, Tang
Re: Memory leak fix in psql
Thanks for your explanation, this time I know how it works, thanks ;) On Wed, Jul 20, 2022 at 3:04 PM tanghy.f...@fujitsu.com wrote: > > > On Wed, Jul 20, 2022 at 12:51:24PM +0800, Junwang Zhao wrote: > > > Though the patch looks good, I myself think the patch should be edited > > > and submitted by Tang > > > It's easy to attach a fixed patch based on the comments of the thread, > > > but coins should be > > > given to Tang since he is the first one to find the mem leak. > > Hello, Zhao > > Thanks for your check at this patch. > > I appreciate your kindly comment but there may be a misunderstanding here. > As Michael explained, committers in Postgres will review carefully and > help to improve contributors' patches. When the patch is finally committed > by one committer, from what I can see, he or she will try to make sure the > credit goes with everyone who contributed to the committed patch(such as > bug reporter, patch author, tester, reviewer etc.). > > Also, developers and reviewers will try to help improving our proposed patch > by rebasing it or adding an on-top patch(like Japin Li did in v2). > These will make the patch better and to be committed ASAP. > > Good to see you at Postgres community. > > Regards, > Tang -- Regards Junwang Zhao
Re: Memory leak fix in psql
More on the same tune. -- Álvaro HerreraBreisgau, Deutschland — https://www.EnterpriseDB.com/ "This is what I like so much about PostgreSQL. Most of the surprises are of the "oh wow! That's cool" Not the "oh shit!" kind. :)" Scott Marlowe, http://archives.postgresql.org/pgsql-admin/2008-10/msg00152.php >From 5e031fadc778365491f9e58da83a4b64f21e3bcd Mon Sep 17 00:00:00 2001 From: Alvaro Herrera Date: Wed, 20 Jul 2022 10:04:27 +0200 Subject: [PATCH] More goto error_return in describe.c --- src/bin/psql/describe.c | 33 + 1 file changed, 13 insertions(+), 20 deletions(-) diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 77b0f87e39..46999f26f2 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -1129,19 +1129,13 @@ permissionsList(const char *pattern) "n.nspname", "c.relname", NULL, "n.nspname !~ '^pg_' AND pg_catalog.pg_table_is_visible(c.oid)", NULL, 3)) - { - termPQExpBuffer(&buf); - return false; - } + goto error_return; appendPQExpBufferStr(&buf, "ORDER BY 1, 2;"); res = PSQLexec(buf.data); if (!res) - { - termPQExpBuffer(&buf); - return false; - } + goto error_return; myopt.nullPrint = NULL; printfPQExpBuffer(&buf, _("Access privileges")); @@ -1155,6 +1149,10 @@ permissionsList(const char *pattern) termPQExpBuffer(&buf); PQclear(res); return true; + +error_return: + termPQExpBuffer(&buf); + return false; } @@ -4983,19 +4981,13 @@ listSchemas(const char *pattern, bool verbose, bool showSystem) NULL, "n.nspname", NULL, NULL, NULL, 2)) - { - termPQExpBuffer(&buf); - return false; - } + goto error_return; appendPQExpBufferStr(&buf, "ORDER BY 1;"); res = PSQLexec(buf.data); if (!res) - { - termPQExpBuffer(&buf); - return false; - } + goto error_return; myopt.nullPrint = NULL; myopt.title = _("List of schemas"); @@ -5016,10 +5008,7 @@ listSchemas(const char *pattern, bool verbose, bool showSystem) pattern); result = PSQLexec(buf.data); if (!result) - { - termPQExpBuffer(&buf); - return false; - } + goto error_return; else pub_schema_tuples = PQntuples(result); @@ -5066,6 +5055,10 @@ listSchemas(const char *pattern, bool verbose, bool showSystem) } return true; + +error_return: + termPQExpBuffer(&buf); + return false; } -- 2.30.2
Re: Memory leak fix in psql
On Wed, 20 Jul 2022 at 14:21, tanghy.f...@fujitsu.com wrote: > On Wednesday, July 20, 2022 12:52 PM, Michael Paquier > wrote: >> What about the argument of upthread where we could use a goto in >> functions where there are multiple pattern validation checks? Per se >> v4 attached. > > Thanks for your kindly remind and modification. > I checked v4 patch, it looks good but I think there can be some minor > improvement. > So I deleted some redundant braces around "goto error_return; ". > Also added an error handle section in validateSQLNamePattern. > > Kindly to have a check at the attached v5 patch. > > Regards, > Tang Thanks for updating the patch. It looks good. However, it cannot be applied on 14 stable. The attached patches are for 10-14. -- Regrads, Japin Li. ChengDu WenWu Information Technology Co.,Ltd. >From 1dd828a40d5a45c0ee021f42f8eee354082a72cf Mon Sep 17 00:00:00 2001 From: Michael Paquier Date: Wed, 20 Jul 2022 12:47:52 +0900 Subject: [PATCH v5] fix the memory leak in psql describe diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 88d92a08ae..77b0f87e39 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -112,7 +112,10 @@ describeAggregates(const char *pattern, bool verbose, bool showSystem) "n.nspname", "p.proname", NULL, "pg_catalog.pg_function_is_visible(p.oid)", NULL, 3)) + { + termPQExpBuffer(&buf); return false; + } appendPQExpBufferStr(&buf, "ORDER BY 1, 2, 4;"); @@ -182,7 +185,10 @@ describeAccessMethods(const char *pattern, bool verbose) NULL, "amname", NULL, NULL, NULL, 1)) + { + termPQExpBuffer(&buf); return false; + } appendPQExpBufferStr(&buf, "ORDER BY 1;"); @@ -244,7 +250,10 @@ describeTablespaces(const char *pattern, bool verbose) NULL, "spcname", NULL, NULL, NULL, 1)) + { + termPQExpBuffer(&buf); return false; + } appendPQExpBufferStr(&buf, "ORDER BY 1;"); @@ -534,7 +543,7 @@ describeFunctions(const char *functypes, const char *func_pattern, "n.nspname", "p.proname", NULL, "pg_catalog.pg_function_is_visible(p.oid)", NULL, 3)) - return false; + goto error_return; for (int i = 0; i < num_arg_patterns; i++) { @@ -561,7 +570,7 @@ describeFunctions(const char *functypes, const char *func_pattern, true, false, nspname, typname, ft, tiv, NULL, 3)) -return false; +goto error_return; } else { @@ -599,6 +608,10 @@ describeFunctions(const char *functypes, const char *func_pattern, PQclear(res); return true; + +error_return: + termPQExpBuffer(&buf); + return false; } @@ -682,7 +695,10 @@ describeTypes(const char *pattern, bool verbose, bool showSystem) "pg_catalog.format_type(t.oid, NULL)", "pg_catalog.pg_type_is_visible(t.oid)", NULL, 3)) + { + termPQExpBuffer(&buf); return false; + } appendPQExpBufferStr(&buf, "ORDER BY 1, 2;"); @@ -836,7 +852,7 @@ describeOperators(const char *oper_pattern, "n.nspname", "o.oprname", NULL, "pg_catalog.pg_operator_is_visible(o.oid)", NULL, 3)) - return false; + goto error_return; if (num_arg_patterns == 1) appendPQExpBufferStr(&buf, " AND o.oprleft = 0\n"); @@ -866,7 +882,7 @@ describeOperators(const char *oper_pattern, true, false, nspname, typname, ft, tiv, NULL, 3)) -return false; +goto error_return; } else { @@ -890,6 +906,10 @@ describeOperators(const char *oper_pattern, PQclear(res); return true; + +error_return: + termPQExpBuffer(&buf); + return false; } @@ -953,7 +973,10 @@ listAllDbs(const char *pattern, bool verbose) if (!validateSQLNamePattern(&buf, pattern, false, false, NULL, "d.datname", NULL, NULL, NULL, 1)) + { + termPQExpBuffer(&buf); return false; + } appendPQExpBufferStr(&buf, "ORDER BY 1;"); res = PSQLexec(buf.data); @@ -1106,7 +1129,10 @@ permissionsList(const char *pattern) "n.nspname", "c.relname", NULL, "n.nspname !~ '^pg_' AND pg_catalog.pg_table_is_visible(c.oid)", NULL, 3)) + { + termPQExpBuffer(&buf); return false; + } appendPQExpBufferStr(&buf, "ORDER BY 1, 2;"); @@ -1177,16 +1203,13 @@ listDefaultACLs(const char *pattern) "pg_catalog.pg_get_userbyid(d.defaclrole)", NULL, NULL, 3)) - return false; + goto error_return; appendPQExpBufferStr(&buf, "ORDER BY 1, 2, 3;"); res = PSQLexec(buf.data); if (!res) - { - termPQExpBuffer(&buf); - return false; - } + goto error_return; myopt.nullPrint = NULL; printfPQExpBuffer(&buf, _("Default access privileges")); @@ -1200,6 +1223,10 @@ listDefaultACLs(const char *pattern) termPQExpBuffer(&buf); PQclear(res); return true; + +error_return: + termPQExpBuffer(&buf); + return false; } @@ -1253,7 +1280,7 @@ objectDescription(const char *pattern, bool showSystem) false,
Re: Memory leak fix in psql
Japin Li writes: > Thanks for updating the patch. It looks good. However, it cannot be > applied on 14 stable. The attached patches are for 10-14. While I think this is good cleanup, I'm doubtful that any of these leaks are probable enough to be worth back-patching into stable branches. The risk of breaking something should not be neglected. regards, tom lane
Re: Memory leak fix in psql
On Wed, Jul 20, 2022 at 10:05:47AM +0200, Alvaro Herrera wrote: > More on the same tune. Thanks. I have noticed that as well. I'll include all that in the set. -- Michael signature.asc Description: PGP signature
Re: Memory leak fix in psql
On Wed, 20 Jul 2022 at 21:54, Tom Lane wrote: > Japin Li writes: >> Thanks for updating the patch. It looks good. However, it cannot be >> applied on 14 stable. The attached patches are for 10-14. > > While I think this is good cleanup, I'm doubtful that any of these > leaks are probable enough to be worth back-patching into stable > branches. The risk of breaking something should not be neglected. > Yeah, we should take care of the backpatch risk. However, I think it makes sense to backpatch. -- Regrads, Japin Li. ChengDu WenWu Information Technology Co.,Ltd.
Re: Memory leak fix in psql
On Thu, Jul 21, 2022 at 09:10:43AM +0800, Japin Li wrote: > Yeah, we should take care of the backpatch risk. However, I think > it makes sense to backpatch. We are talking about 256 bytes being leaked in each loop when a validation pattern or when a query fails, so I don't see a strong argument in manipulating 10~14 more than necessary for this amount of memory. The contents of describe.c are the same for v15 though, and we are still in beta on REL_15_STABLE, so I have applied the patch down to v15, adding what Alvaro has sent on top of the rest. -- Michael signature.asc Description: PGP signature
Re: Memory leak fix in psql
On Thu, 21 Jul 2022 at 09:48, Michael Paquier wrote: > On Thu, Jul 21, 2022 at 09:10:43AM +0800, Japin Li wrote: >> Yeah, we should take care of the backpatch risk. However, I think >> it makes sense to backpatch. > > We are talking about 256 bytes being leaked in each loop when a > validation pattern or when a query fails, so I don't see a strong > argument in manipulating 10~14 more than necessary for this amount of > memory. The contents of describe.c are the same for v15 though, and > we are still in beta on REL_15_STABLE, so I have applied the patch > down to v15, adding what Alvaro has sent on top of the rest. Thanks for the explanation! IMO, we could ignore v10-13 branches, however, we should backpatch to v14 which also uses the validateSQLNamePattern() function leading to a memory leak. -- Regrads, Japin Li. ChengDu WenWu Information Technology Co.,Ltd.
Re: Memory leak fix in psql
On 2022-Jul-21, Japin Li wrote: > On Thu, 21 Jul 2022 at 09:48, Michael Paquier wrote: > > We are talking about 256 bytes being leaked in each loop when a > > validation pattern or when a query fails, so I don't see a strong > > argument in manipulating 10~14 more than necessary for this amount of > > memory. The contents of describe.c are the same for v15 though, and > > we are still in beta on REL_15_STABLE, so I have applied the patch > > down to v15, adding what Alvaro has sent on top of the rest. > > Thanks for the explanation! IMO, we could ignore v10-13 branches, however, > we should backpatch to v14 which also uses the validateSQLNamePattern() > function leading to a memory leak. I'd agree in principle, but in practice the commit from 15 does not apply cleanly on 14 -- there's a ton of conflicts. -- Álvaro Herrera 48°01'N 7°57'E — https://www.EnterpriseDB.com/ "Here's a general engineering tip: if the non-fun part is too complex for you to figure out, that might indicate the fun part is too ambitious." (John Naylor) https://postgr.es/m/CAFBsxsG4OWHBbSDM%3DsSeXrQGOtkPiOEOuME4yD7Ce41NtaAD9g%40mail.gmail.com
Re: Memory leak fix in psql
On Thu, 21 Jul 2022 at 14:23, Alvaro Herrera wrote: > On 2022-Jul-21, Japin Li wrote: > >> On Thu, 21 Jul 2022 at 09:48, Michael Paquier wrote: > >> > We are talking about 256 bytes being leaked in each loop when a >> > validation pattern or when a query fails, so I don't see a strong >> > argument in manipulating 10~14 more than necessary for this amount of >> > memory. The contents of describe.c are the same for v15 though, and >> > we are still in beta on REL_15_STABLE, so I have applied the patch >> > down to v15, adding what Alvaro has sent on top of the rest. >> >> Thanks for the explanation! IMO, we could ignore v10-13 branches, however, >> we should backpatch to v14 which also uses the validateSQLNamePattern() >> function leading to a memory leak. > > I'd agree in principle, but in practice the commit from 15 does not apply > cleanly on 14 -- there's a ton of conflicts. I attached a patch for v14 [1] based on master, if you want to apply it, please consider reviewing it. [1] https://www.postgresql.org/message-id/MEYP282MB166974D3883A1A6E25B5FDAEB68E9%40MEYP282MB1669.AUSP282.PROD.OUTLOOK.COM -- Regrads, Japin Li. ChengDu WenWu Information Technology Co.,Ltd.
Re: Memory leak fix in psql
On Thu, Jul 21, 2022 at 02:43:03PM +0800, Japin Li wrote: > I attached a patch for v14 [1] based on master, if you want to apply it, > please consider reviewing it. We are talking about a few hundred bytes leaked each time, so this does not worry me much in the older branches, honestly. -- Michael signature.asc Description: PGP signature