Memory leak fix in psql

[tanghy.f...@fujitsu.com]

Hi

I think there is a newly introduced memory leak in your patch d2d3547.
Try to fix it in the attached patch. 
Kindly to have a check.

Regards,
Tang


v1-0001-fix-the-memory-leak-in-validateSQLNamePattern.patch
Description:  v1-0001-fix-the-memory-leak-in-validateSQLNamePattern.patch

Re: Memory leak fix in psql

[Japin Li]

On Tue, 19 Jul 2022 at 17:02, tanghy.f...@fujitsu.com  
wrote:
> Hi
>
> I think there is a newly introduced memory leak in your patch d2d3547.
> Try to fix it in the attached patch. 
> Kindly to have a check.
>

Yeah, it leaks, and the patch can fix it.

After looking around, I found psql/describe.c also has some memory leaks,
attached a patch to fix these leaks.

-- 
Regrads,
Japin Li.
ChengDu WenWu Information Technology Co.,Ltd.

>From 3036a0986f8ff490c133930524e2d5f5104249ff Mon Sep 17 00:00:00 2001
From: Japin Li 
Date: Tue, 19 Jul 2022 18:27:25 +0800
Subject: [PATCH v1 1/1] Fix the memory leak in psql describe

---
 src/bin/psql/describe.c | 150 
 1 file changed, 150 insertions(+)

diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c
index 0ce38e4b4c..11a441f52f 100644
--- a/src/bin/psql/describe.c
+++ b/src/bin/psql/describe.c
@@ -112,7 +112,10 @@ describeAggregates(const char *pattern, bool verbose, bool showSystem)
 "n.nspname", "p.proname", NULL,
 "pg_catalog.pg_function_is_visible(p.oid)",
 NULL, 3))
+	{
+		termPQExpBuffer(&buf);
 		return false;
+	}
 
 	appendPQExpBufferStr(&buf, "ORDER BY 1, 2, 4;");
 
@@ -182,7 +185,10 @@ describeAccessMethods(const char *pattern, bool verbose)
 NULL, "amname", NULL,
 NULL,
 NULL, 1))
+	{
+		termPQExpBuffer(&buf);
 		return false;
+	}
 
 	appendPQExpBufferStr(&buf, "ORDER BY 1;");
 
@@ -244,7 +250,10 @@ describeTablespaces(const char *pattern, bool verbose)
 NULL, "spcname", NULL,
 NULL,
 NULL, 1))
+	{
+		termPQExpBuffer(&buf);
 		return false;
+	}
 
 	appendPQExpBufferStr(&buf, "ORDER BY 1;");
 
@@ -534,7 +543,10 @@ describeFunctions(const char *functypes, const char *func_pattern,
 "n.nspname", "p.proname", NULL,
 "pg_catalog.pg_function_is_visible(p.oid)",
 NULL, 3))
+	{
+		termPQExpBuffer(&buf);
 		return false;
+	}
 
 	for (int i = 0; i < num_arg_patterns; i++)
 	{
@@ -561,7 +573,10 @@ describeFunctions(const char *functypes, const char *func_pattern,
 		true, false,
 		nspname, typname, ft, tiv,
 		NULL, 3))
+			{
+termPQExpBuffer(&buf);
 return false;
+			}
 		}
 		else
 		{
@@ -682,7 +697,10 @@ describeTypes(const char *pattern, bool verbose, bool showSystem)
 "pg_catalog.format_type(t.oid, NULL)",
 "pg_catalog.pg_type_is_visible(t.oid)",
 NULL, 3))
+	{
+		termPQExpBuffer(&buf);
 		return false;
+	}
 
 	appendPQExpBufferStr(&buf, "ORDER BY 1, 2;");
 
@@ -836,7 +854,10 @@ describeOperators(const char *oper_pattern,
 "n.nspname", "o.oprname", NULL,
 "pg_catalog.pg_operator_is_visible(o.oid)",
 NULL, 3))
+	{
+		termPQExpBuffer(&buf);
 		return false;
+	}
 
 	if (num_arg_patterns == 1)
 		appendPQExpBufferStr(&buf, "  AND o.oprleft = 0\n");
@@ -866,7 +887,10 @@ describeOperators(const char *oper_pattern,
 		true, false,
 		nspname, typname, ft, tiv,
 		NULL, 3))
+			{
+termPQExpBuffer(&buf);
 return false;
+			}
 		}
 		else
 		{
@@ -953,7 +977,10 @@ listAllDbs(const char *pattern, bool verbose)
 		if (!validateSQLNamePattern(&buf, pattern, false, false,
 	NULL, "d.datname", NULL, NULL,
 	NULL, 1))
+		{
+			termPQExpBuffer(&buf);
 			return false;
+		}
 
 	appendPQExpBufferStr(&buf, "ORDER BY 1;");
 	res = PSQLexec(buf.data);
@@ -1106,7 +1133,10 @@ permissionsList(const char *pattern)
 "n.nspname", "c.relname", NULL,
 "n.nspname !~ '^pg_' AND pg_catalog.pg_table_is_visible(c.oid)",
 NULL, 3))
+	{
+		termPQExpBuffer(&buf);
 		return false;
+	}
 
 	appendPQExpBufferStr(&buf, "ORDER BY 1, 2;");
 
@@ -1177,7 +1207,10 @@ listDefaultACLs(const char *pattern)
 "pg_catalog.pg_get_userbyid(d.defaclrole)",
 NULL,
 NULL, 3))
+	{
+		termPQExpBuffer(&buf);
 		return false;
+	}
 
 	appendPQExpBufferStr(&buf, "ORDER BY 1, 2, 3;");
 
@@ -1428,7 +1461,10 @@ describeTableDetails(const char *pattern, bool verbose, bool showSystem)
 "n.nspname", "c.relname", NULL,
 "pg_catalog.pg_table_is_visible(c.oid)",
 NULL, 3))
+	{
+		termPQExpBuffer(&buf);
 		return false;
+	}
 
 	appendPQExpBufferStr(&buf, "ORDER BY 2, 3;");
 
@@ -3614,7 +3650,10 @@ describeRoles(const char *pattern, bool verbose, bool showSystem)
 	if (!validateSQLNamePattern(&buf, pattern, false, false,
 NULL, "r.rolname", NULL, NULL,
 NULL, 1))
+	{
+		termPQExpBuffer(&buf);
 		return false;
+	}
 
 	appendPQExpBufferStr(&buf, "ORDER BY 1;");
 
@@ -3739,11 +3778,17 @@ listDbRoleSettings(const char *pattern, const char *pattern2)
 	  gettext_noop("Settings"));
 	if (!validateSQLNamePattern(&buf, pattern, false, false,
 NULL, "r.rolname", NULL, NULL, &havewhere, 1))
+	{
+		termPQExpBuffer(&buf);
 		return false;
+	}
 	if (!validateSQLNamePattern(&buf, pattern2, havewhere, false,
 NULL, "d.datname", NULL, NULL,
 NULL, 1))
+	{
+		termPQE

Re: Memory leak fix in psql

[Michael Paquier]

On Tue, Jul 19, 2022 at 06:41:13PM +0800, Japin Li wrote:
> After looking around, I found psql/describe.c also has some memory leaks,
> attached a patch to fix these leaks.

Indeed.  There are quite a bit of them, so let's fix all that.  You
have missed a couple of code paths in objectDescription().
--
Michael


signature.asc
Description: PGP signature

Re: Memory leak fix in psql

[Japin Li]

On Tue, 19 Jul 2022 at 20:32, Michael Paquier  wrote:
> On Tue, Jul 19, 2022 at 06:41:13PM +0800, Japin Li wrote:
>> After looking around, I found psql/describe.c also has some memory leaks,
>> attached a patch to fix these leaks.
>
> Indeed.  There are quite a bit of them, so let's fix all that.  You
> have missed a couple of code paths in objectDescription().

Thanks for reviewing. Attached fix the memory leak in objectDescription().
Please consider v2 for further review.

-- 
Regrads,
Japin Li.
ChengDu WenWu Information Technology Co.,Ltd.

>From b2bcc3a1bac67b8b414f2025607f8dd35e096289 Mon Sep 17 00:00:00 2001
From: Japin Li 
Date: Tue, 19 Jul 2022 18:27:25 +0800
Subject: [PATCH v2 1/1] Fix the memory leak in psql describe

---
 src/bin/psql/describe.c | 168 
 1 file changed, 168 insertions(+)

diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c
index 0ce38e4b4c..7a070a6cd0 100644
--- a/src/bin/psql/describe.c
+++ b/src/bin/psql/describe.c
@@ -112,7 +112,10 @@ describeAggregates(const char *pattern, bool verbose, bool showSystem)
 "n.nspname", "p.proname", NULL,
 "pg_catalog.pg_function_is_visible(p.oid)",
 NULL, 3))
+	{
+		termPQExpBuffer(&buf);
 		return false;
+	}
 
 	appendPQExpBufferStr(&buf, "ORDER BY 1, 2, 4;");
 
@@ -182,7 +185,10 @@ describeAccessMethods(const char *pattern, bool verbose)
 NULL, "amname", NULL,
 NULL,
 NULL, 1))
+	{
+		termPQExpBuffer(&buf);
 		return false;
+	}
 
 	appendPQExpBufferStr(&buf, "ORDER BY 1;");
 
@@ -244,7 +250,10 @@ describeTablespaces(const char *pattern, bool verbose)
 NULL, "spcname", NULL,
 NULL,
 NULL, 1))
+	{
+		termPQExpBuffer(&buf);
 		return false;
+	}
 
 	appendPQExpBufferStr(&buf, "ORDER BY 1;");
 
@@ -534,7 +543,10 @@ describeFunctions(const char *functypes, const char *func_pattern,
 "n.nspname", "p.proname", NULL,
 "pg_catalog.pg_function_is_visible(p.oid)",
 NULL, 3))
+	{
+		termPQExpBuffer(&buf);
 		return false;
+	}
 
 	for (int i = 0; i < num_arg_patterns; i++)
 	{
@@ -561,7 +573,10 @@ describeFunctions(const char *functypes, const char *func_pattern,
 		true, false,
 		nspname, typname, ft, tiv,
 		NULL, 3))
+			{
+termPQExpBuffer(&buf);
 return false;
+			}
 		}
 		else
 		{
@@ -682,7 +697,10 @@ describeTypes(const char *pattern, bool verbose, bool showSystem)
 "pg_catalog.format_type(t.oid, NULL)",
 "pg_catalog.pg_type_is_visible(t.oid)",
 NULL, 3))
+	{
+		termPQExpBuffer(&buf);
 		return false;
+	}
 
 	appendPQExpBufferStr(&buf, "ORDER BY 1, 2;");
 
@@ -836,7 +854,10 @@ describeOperators(const char *oper_pattern,
 "n.nspname", "o.oprname", NULL,
 "pg_catalog.pg_operator_is_visible(o.oid)",
 NULL, 3))
+	{
+		termPQExpBuffer(&buf);
 		return false;
+	}
 
 	if (num_arg_patterns == 1)
 		appendPQExpBufferStr(&buf, "  AND o.oprleft = 0\n");
@@ -866,7 +887,10 @@ describeOperators(const char *oper_pattern,
 		true, false,
 		nspname, typname, ft, tiv,
 		NULL, 3))
+			{
+termPQExpBuffer(&buf);
 return false;
+			}
 		}
 		else
 		{
@@ -953,7 +977,10 @@ listAllDbs(const char *pattern, bool verbose)
 		if (!validateSQLNamePattern(&buf, pattern, false, false,
 	NULL, "d.datname", NULL, NULL,
 	NULL, 1))
+		{
+			termPQExpBuffer(&buf);
 			return false;
+		}
 
 	appendPQExpBufferStr(&buf, "ORDER BY 1;");
 	res = PSQLexec(buf.data);
@@ -1106,7 +1133,10 @@ permissionsList(const char *pattern)
 "n.nspname", "c.relname", NULL,
 "n.nspname !~ '^pg_' AND pg_catalog.pg_table_is_visible(c.oid)",
 NULL, 3))
+	{
+		termPQExpBuffer(&buf);
 		return false;
+	}
 
 	appendPQExpBufferStr(&buf, "ORDER BY 1, 2;");
 
@@ -1177,7 +1207,10 @@ listDefaultACLs(const char *pattern)
 "pg_catalog.pg_get_userbyid(d.defaclrole)",
 NULL,
 NULL, 3))
+	{
+		termPQExpBuffer(&buf);
 		return false;
+	}
 
 	appendPQExpBufferStr(&buf, "ORDER BY 1, 2, 3;");
 
@@ -1253,7 +1286,10 @@ objectDescription(const char *pattern, bool showSystem)
 false, "n.nspname", "pgc.conname", NULL,
 "pg_catalog.pg_table_is_visible(c.oid)",
 NULL, 3))
+	{
+		termPQExpBuffer(&buf);
 		return false;
+	}
 
 	/* Domain constraint descriptions */
 	appendPQExpBuffer(&buf,
@@ -1277,7 +1313,10 @@ objectDescription(const char *pattern, bool showSystem)
 false, "n.nspname", "pgc.conname", NULL,
 "pg_catalog.pg_type_is_visible(t.oid)",
 NULL, 3))
+	{
+		termPQExpBuffer(&buf);
 		return false;
+	}
 
 	/* Operator class descriptions */
 	appendPQExpBuffer(&buf,
@@ -1301,7 +1340,10 @@ objectDescription(const char *pattern, bool showSystem)
 "n.nspname", "o.opcname", NULL,
 "pg_catalog.pg_opclass_is_visible(o.oid)",
 NULL, 3))
+	{
+		termPQExpBuffer(&buf);
 		return false;
+	}
 
 	/* Operator family descriptions */
 	appendPQExpBuffe

Re: Memory leak fix in psql

[Andres Freund]

Hi,

On 2022-07-19 21:08:53 +0800, Japin Li wrote:
> From b2bcc3a1bac67b8b414f2025607f8dd35e096289 Mon Sep 17 00:00:00 2001
> From: Japin Li 
> Date: Tue, 19 Jul 2022 18:27:25 +0800
> Subject: [PATCH v2 1/1] Fix the memory leak in psql describe
> 
> ---
>  src/bin/psql/describe.c | 168 
>  1 file changed, 168 insertions(+)
> 
> diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c
> index 0ce38e4b4c..7a070a6cd0 100644
> --- a/src/bin/psql/describe.c
> +++ b/src/bin/psql/describe.c
> @@ -112,7 +112,10 @@ describeAggregates(const char *pattern, bool verbose, 
> bool showSystem)
>   "n.nspname", 
> "p.proname", NULL,
>   
> "pg_catalog.pg_function_is_visible(p.oid)",
>   NULL, 3))
> + {
> + termPQExpBuffer(&buf);
>   return false;
> + }
>  
>   appendPQExpBufferStr(&buf, "ORDER BY 1, 2, 4;");

Adding copy over copy of this same block doesn't seem great. Can we instead
add a helper for it or such?

Greetings,

Andres Freund

Re: Memory leak fix in psql

[Tom Lane]

Andres Freund  writes:
> On 2022-07-19 21:08:53 +0800, Japin Li wrote:
>> +{
>> +termPQExpBuffer(&buf);
>>  return false;
>> +}

> Adding copy over copy of this same block doesn't seem great. Can we instead
> add a helper for it or such?

The usual style in these files is something like

if (bad things happened)
goto fail;

...

fail:
termPQExpBuffer(&buf);
return false;

Yeah, it's old school, but please let's not have a few functions that
do it randomly differently from all their neighbors.

regards, tom lane

Re: Memory leak fix in psql

[Mark Dilger]

> On Jul 19, 2022, at 2:02 AM, tanghy.f...@fujitsu.com wrote:
> 
> I think there is a newly introduced memory leak in your patch d2d3547.

I agree.  Thanks for noticing, and for the patch!

> Try to fix it in the attached patch. 
> Kindly to have a check.

This looks ok, but comments down-thread seem reasonable, so I suspect a new 
patch will be needed.  Would you like to author it, or would you prefer that I, 
as the guilty party, do so?

—
Mark Dilger
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

Re: Memory leak fix in psql

[Michael Paquier]

On Tue, Jul 19, 2022 at 12:36:31PM -0400, Tom Lane wrote:
> Yeah, it's old school, but please let's not have a few functions that
> do it randomly differently from all their neighbors.

True enough.  And it is not like we should free the PQExpBuffer given
by the caller in validateSQLNamePattern().
--
Michael


signature.asc
Description: PGP signature

Re: Memory leak fix in psql

[Michael Paquier]

On Tue, Jul 19, 2022 at 09:43:21AM -0700, Mark Dilger wrote:
> This looks ok, but comments down-thread seem reasonable, so I
> suspect a new patch will be needed.  Would you like to author it, or
> would you prefer that I, as the guilty party, do so? 

If any of you could update the patch, that would be great.  Thanks!
--
Michael


signature.asc
Description: PGP signature

RE: Memory leak fix in psql

[tanghy.f...@fujitsu.com]

On Tuesday, July 19, 2022 7:41 PM, Japin Li  wrote:
> After looking around, I found psql/describe.c also has some memory
> leaks,
> attached a patch to fix these leaks.

Thanks for your check and improvement.
Your fix LGTM so please allow me to merge it in the attached patch. 
Based on your rebased version, now this new patch version is V3.

Regards,
Tang



v3-0001-fix-the-memory-leak-in-psql-describe.patch
Description: v3-0001-fix-the-memory-leak-in-psql-describe.patch

Re: Memory leak fix in psql

[Michael Paquier]

On Wed, Jul 20, 2022 at 03:14:35AM +, tanghy.f...@fujitsu.com wrote:
> Your fix LGTM so please allow me to merge it in the attached patch. 
> Based on your rebased version, now this new patch version is V3.

What about the argument of upthread where we could use a goto in
functions where there are multiple pattern validation checks?  Per se
v4 attached.
--
Michael
From ccc8bb83baf618ea1a481193403a9009cd4a24a5 Mon Sep 17 00:00:00 2001
From: Michael Paquier 
Date: Wed, 20 Jul 2022 12:47:52 +0900
Subject: [PATCH v4] fix the memory leak in psql describe

---
 src/bin/psql/describe.c | 239 +++-
 1 file changed, 212 insertions(+), 27 deletions(-)

diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c
index 88d92a08ae..1c40cc6d59 100644
--- a/src/bin/psql/describe.c
+++ b/src/bin/psql/describe.c
@@ -112,7 +112,10 @@ describeAggregates(const char *pattern, bool verbose, bool showSystem)
 "n.nspname", "p.proname", NULL,
 "pg_catalog.pg_function_is_visible(p.oid)",
 NULL, 3))
+	{
+		termPQExpBuffer(&buf);
 		return false;
+	}
 
 	appendPQExpBufferStr(&buf, "ORDER BY 1, 2, 4;");
 
@@ -182,7 +185,10 @@ describeAccessMethods(const char *pattern, bool verbose)
 NULL, "amname", NULL,
 NULL,
 NULL, 1))
+	{
+		termPQExpBuffer(&buf);
 		return false;
+	}
 
 	appendPQExpBufferStr(&buf, "ORDER BY 1;");
 
@@ -244,7 +250,10 @@ describeTablespaces(const char *pattern, bool verbose)
 NULL, "spcname", NULL,
 NULL,
 NULL, 1))
+	{
+		termPQExpBuffer(&buf);
 		return false;
+	}
 
 	appendPQExpBufferStr(&buf, "ORDER BY 1;");
 
@@ -534,7 +543,9 @@ describeFunctions(const char *functypes, const char *func_pattern,
 "n.nspname", "p.proname", NULL,
 "pg_catalog.pg_function_is_visible(p.oid)",
 NULL, 3))
-		return false;
+	{
+		goto error_return;
+	}
 
 	for (int i = 0; i < num_arg_patterns; i++)
 	{
@@ -561,7 +572,9 @@ describeFunctions(const char *functypes, const char *func_pattern,
 		true, false,
 		nspname, typname, ft, tiv,
 		NULL, 3))
-return false;
+			{
+goto error_return;
+			}
 		}
 		else
 		{
@@ -599,6 +612,10 @@ describeFunctions(const char *functypes, const char *func_pattern,
 
 	PQclear(res);
 	return true;
+
+error_return:
+	termPQExpBuffer(&buf);
+	return false;
 }
 
 
@@ -682,7 +699,10 @@ describeTypes(const char *pattern, bool verbose, bool showSystem)
 "pg_catalog.format_type(t.oid, NULL)",
 "pg_catalog.pg_type_is_visible(t.oid)",
 NULL, 3))
+	{
+		termPQExpBuffer(&buf);
 		return false;
+	}
 
 	appendPQExpBufferStr(&buf, "ORDER BY 1, 2;");
 
@@ -836,7 +856,9 @@ describeOperators(const char *oper_pattern,
 "n.nspname", "o.oprname", NULL,
 "pg_catalog.pg_operator_is_visible(o.oid)",
 NULL, 3))
-		return false;
+	{
+		goto error_return;
+	}
 
 	if (num_arg_patterns == 1)
 		appendPQExpBufferStr(&buf, "  AND o.oprleft = 0\n");
@@ -866,7 +888,9 @@ describeOperators(const char *oper_pattern,
 		true, false,
 		nspname, typname, ft, tiv,
 		NULL, 3))
-return false;
+			{
+goto error_return;
+			}
 		}
 		else
 		{
@@ -890,6 +914,10 @@ describeOperators(const char *oper_pattern,
 
 	PQclear(res);
 	return true;
+
+error_return:
+	termPQExpBuffer(&buf);
+	return false;
 }
 
 
@@ -953,7 +981,10 @@ listAllDbs(const char *pattern, bool verbose)
 		if (!validateSQLNamePattern(&buf, pattern, false, false,
 	NULL, "d.datname", NULL, NULL,
 	NULL, 1))
+		{
+			termPQExpBuffer(&buf);
 			return false;
+		}
 
 	appendPQExpBufferStr(&buf, "ORDER BY 1;");
 	res = PSQLexec(buf.data);
@@ -1106,7 +1137,10 @@ permissionsList(const char *pattern)
 "n.nspname", "c.relname", NULL,
 "n.nspname !~ '^pg_' AND pg_catalog.pg_table_is_visible(c.oid)",
 NULL, 3))
+	{
+		termPQExpBuffer(&buf);
 		return false;
+	}
 
 	appendPQExpBufferStr(&buf, "ORDER BY 1, 2;");
 
@@ -1177,16 +1211,15 @@ listDefaultACLs(const char *pattern)
 "pg_catalog.pg_get_userbyid(d.defaclrole)",
 NULL,
 NULL, 3))
-		return false;
+	{
+		goto error_return;
+	}
 
 	appendPQExpBufferStr(&buf, "ORDER BY 1, 2, 3;");
 
 	res = PSQLexec(buf.data);
 	if (!res)
-	{
-		termPQExpBuffer(&buf);
-		return false;
-	}
+		goto error_return;
 
 	myopt.nullPrint = NULL;
 	printfPQExpBuffer(&buf, _("Default access privileges"));
@@ -1200,6 +1233,10 @@ listDefaultACLs(const char *pattern)
 	termPQExpBuffer(&buf);
 	PQclear(res);
 	return true;
+
+error_return:
+	termPQExpBuffer(&buf);
+	return false;
 }
 
 
@@ -1253,7 +1290,9 @@ objectDescription(const char *pattern, bool showSystem)
 false, "n.nspname", "pgc.conname", NULL,
 "pg_catalog.pg_table_is_visible(c.oid)",
 NULL, 3))
-		return false;
+	{
+		goto error_return;
+	}
 
 	/* Domain constraint descriptions */
 	appendPQExpBuffer(&buf,
@@ -1277,7 +1316,9 @@ objectDescription(const char *pattern, bool show

Re: Memory leak fix in psql

[Japin Li]

On Wed, 20 Jul 2022 at 11:51, Michael Paquier  wrote:
> On Wed, Jul 20, 2022 at 03:14:35AM +, tanghy.f...@fujitsu.com wrote:
>> Your fix LGTM so please allow me to merge it in the attached patch. 
>> Based on your rebased version, now this new patch version is V3.
>
> What about the argument of upthread where we could use a goto in
> functions where there are multiple pattern validation checks?  Per se
> v4 attached.

+1. LGTM.

-- 
Regrads,
Japin Li.
ChengDu WenWu Information Technology Co.,Ltd.

Re: Memory leak fix in psql

[Junwang Zhao]

-1

Though the patch looks good, I myself think the patch should be edited
and submitted by Tang
It's easy to attach a fixed patch based on the comments of the thread,
but coins should be
given to Tang since he is the first one to find the mem leak.

No offense, but that's what I think how open source works ;)

On Wed, Jul 20, 2022 at 11:51 AM Michael Paquier  wrote:
>
> On Wed, Jul 20, 2022 at 03:14:35AM +, tanghy.f...@fujitsu.com wrote:
> > Your fix LGTM so please allow me to merge it in the attached patch.
> > Based on your rebased version, now this new patch version is V3.
>
> What about the argument of upthread where we could use a goto in
> functions where there are multiple pattern validation checks?  Per se
> v4 attached.
> --
> Michael



-- 
Regards
Junwang Zhao

Re: Memory leak fix in psql

[Michael Paquier]

On Wed, Jul 20, 2022 at 12:51:24PM +0800, Junwang Zhao wrote:
> Though the patch looks good, I myself think the patch should be edited
> and submitted by Tang
> It's easy to attach a fixed patch based on the comments of the thread,
> but coins should be
> given to Tang since he is the first one to find the mem leak.

Please note that I sometimes edit slightly patches that I finish to
merge into the tree, where the author listed in the commit log is the
same as the original while I usually don't list mine.  Credit goes
where it should, and Tang is the one here who authored this patch.
--
Michael


signature.asc
Description: PGP signature

Re: Memory leak fix in psql

[Junwang Zhao]

Got it, thanks!

On Wed, Jul 20, 2022 at 1:14 PM Michael Paquier  wrote:
>
> On Wed, Jul 20, 2022 at 12:51:24PM +0800, Junwang Zhao wrote:
> > Though the patch looks good, I myself think the patch should be edited
> > and submitted by Tang
> > It's easy to attach a fixed patch based on the comments of the thread,
> > but coins should be
> > given to Tang since he is the first one to find the mem leak.
>
> Please note that I sometimes edit slightly patches that I finish to
> merge into the tree, where the author listed in the commit log is the
> same as the original while I usually don't list mine.  Credit goes
> where it should, and Tang is the one here who authored this patch.
> --
> Michael



-- 
Regards
Junwang Zhao

RE: Memory leak fix in psql

[tanghy.f...@fujitsu.com]

On Wednesday, July 20, 2022 12:52 PM, Michael Paquier  
wrote:
> What about the argument of upthread where we could use a goto in
> functions where there are multiple pattern validation checks?  Per se
> v4 attached.

Thanks for your kindly remind and modification.
I checked v4 patch, it looks good but I think there can be some minor 
improvement.
So I deleted some redundant braces around "goto error_return; ".
Also added an error handle section in validateSQLNamePattern.

Kindly to have a check at the attached v5 patch.

Regards,
Tang


v5-0001-fix-the-memory-leak-in-psql-describe.patch
Description: v5-0001-fix-the-memory-leak-in-psql-describe.patch

RE: Memory leak fix in psql

[tanghy.f...@fujitsu.com]

> On Wed, Jul 20, 2022 at 12:51:24PM +0800, Junwang Zhao wrote:
> > Though the patch looks good, I myself think the patch should be edited
> > and submitted by Tang
> > It's easy to attach a fixed patch based on the comments of the thread,
> > but coins should be
> > given to Tang since he is the first one to find the mem leak.

Hello, Zhao

Thanks for your check at this patch. 

I appreciate your kindly comment but there may be a misunderstanding here.
As Michael explained, committers in Postgres will review carefully and 
help to improve contributors' patches. When the patch is finally committed 
by one committer, from what I can see, he or she will try to make sure the 
credit goes with everyone who contributed to the committed patch(such as 
bug reporter, patch author, tester, reviewer etc.). 

Also, developers and reviewers will try to help improving our proposed patch
by rebasing it or adding an on-top patch(like Japin Li did in v2).
These will make the patch better and to be committed ASAP.

Good to see you at Postgres community.

Regards,
Tang

Re: Memory leak fix in psql

[Junwang Zhao]

Thanks for your explanation, this time I know how it works, thanks ;)

On Wed, Jul 20, 2022 at 3:04 PM tanghy.f...@fujitsu.com
 wrote:
>
> > On Wed, Jul 20, 2022 at 12:51:24PM +0800, Junwang Zhao wrote:
> > > Though the patch looks good, I myself think the patch should be edited
> > > and submitted by Tang
> > > It's easy to attach a fixed patch based on the comments of the thread,
> > > but coins should be
> > > given to Tang since he is the first one to find the mem leak.
>
> Hello, Zhao
>
> Thanks for your check at this patch.
>
> I appreciate your kindly comment but there may be a misunderstanding here.
> As Michael explained, committers in Postgres will review carefully and
> help to improve contributors' patches. When the patch is finally committed
> by one committer, from what I can see, he or she will try to make sure the
> credit goes with everyone who contributed to the committed patch(such as
> bug reporter, patch author, tester, reviewer etc.).
>
> Also, developers and reviewers will try to help improving our proposed patch
> by rebasing it or adding an on-top patch(like Japin Li did in v2).
> These will make the patch better and to be committed ASAP.
>
> Good to see you at Postgres community.
>
> Regards,
> Tang



-- 
Regards
Junwang Zhao

Re: Memory leak fix in psql

[Alvaro Herrera]

More on the same tune.

-- 
Álvaro HerreraBreisgau, Deutschland  —  https://www.EnterpriseDB.com/
"This is what I like so much about PostgreSQL.  Most of the surprises
are of the "oh wow!  That's cool" Not the "oh shit!" kind.  :)"
Scott Marlowe, http://archives.postgresql.org/pgsql-admin/2008-10/msg00152.php
>From 5e031fadc778365491f9e58da83a4b64f21e3bcd Mon Sep 17 00:00:00 2001
From: Alvaro Herrera 
Date: Wed, 20 Jul 2022 10:04:27 +0200
Subject: [PATCH] More goto error_return in describe.c

---
 src/bin/psql/describe.c | 33 +
 1 file changed, 13 insertions(+), 20 deletions(-)

diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c
index 77b0f87e39..46999f26f2 100644
--- a/src/bin/psql/describe.c
+++ b/src/bin/psql/describe.c
@@ -1129,19 +1129,13 @@ permissionsList(const char *pattern)
 "n.nspname", "c.relname", NULL,
 "n.nspname !~ '^pg_' AND pg_catalog.pg_table_is_visible(c.oid)",
 NULL, 3))
-	{
-		termPQExpBuffer(&buf);
-		return false;
-	}
+		goto error_return;
 
 	appendPQExpBufferStr(&buf, "ORDER BY 1, 2;");
 
 	res = PSQLexec(buf.data);
 	if (!res)
-	{
-		termPQExpBuffer(&buf);
-		return false;
-	}
+		goto error_return;
 
 	myopt.nullPrint = NULL;
 	printfPQExpBuffer(&buf, _("Access privileges"));
@@ -1155,6 +1149,10 @@ permissionsList(const char *pattern)
 	termPQExpBuffer(&buf);
 	PQclear(res);
 	return true;
+
+error_return:
+		termPQExpBuffer(&buf);
+		return false;
 }
 
 
@@ -4983,19 +4981,13 @@ listSchemas(const char *pattern, bool verbose, bool showSystem)
 NULL, "n.nspname", NULL,
 NULL,
 NULL, 2))
-	{
-		termPQExpBuffer(&buf);
-		return false;
-	}
+		goto error_return;
 
 	appendPQExpBufferStr(&buf, "ORDER BY 1;");
 
 	res = PSQLexec(buf.data);
 	if (!res)
-	{
-		termPQExpBuffer(&buf);
-		return false;
-	}
+		goto error_return;
 
 	myopt.nullPrint = NULL;
 	myopt.title = _("List of schemas");
@@ -5016,10 +5008,7 @@ listSchemas(const char *pattern, bool verbose, bool showSystem)
 		  pattern);
 		result = PSQLexec(buf.data);
 		if (!result)
-		{
-			termPQExpBuffer(&buf);
-			return false;
-		}
+			goto error_return;
 		else
 			pub_schema_tuples = PQntuples(result);
 
@@ -5066,6 +5055,10 @@ listSchemas(const char *pattern, bool verbose, bool showSystem)
 	}
 
 	return true;
+
+error_return:
+	termPQExpBuffer(&buf);
+	return false;
 }
 
 
-- 
2.30.2

Re: Memory leak fix in psql

[Japin Li]

On Wed, 20 Jul 2022 at 14:21, tanghy.f...@fujitsu.com  
wrote:
> On Wednesday, July 20, 2022 12:52 PM, Michael Paquier  
> wrote:
>> What about the argument of upthread where we could use a goto in
>> functions where there are multiple pattern validation checks?  Per se
>> v4 attached.
>
> Thanks for your kindly remind and modification.
> I checked v4 patch, it looks good but I think there can be some minor 
> improvement.
> So I deleted some redundant braces around "goto error_return; ".
> Also added an error handle section in validateSQLNamePattern.
>
> Kindly to have a check at the attached v5 patch.
>
> Regards,
> Tang

Thanks for updating the patch.  It looks good.  However, it cannot be
applied on 14 stable.  The attached patches are for 10-14.

-- 
Regrads,
Japin Li.
ChengDu WenWu Information Technology Co.,Ltd.

>From 1dd828a40d5a45c0ee021f42f8eee354082a72cf Mon Sep 17 00:00:00 2001
From: Michael Paquier 
Date: Wed, 20 Jul 2022 12:47:52 +0900
Subject: [PATCH v5] fix the memory leak in psql describe


diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c
index 88d92a08ae..77b0f87e39 100644
--- a/src/bin/psql/describe.c
+++ b/src/bin/psql/describe.c
@@ -112,7 +112,10 @@ describeAggregates(const char *pattern, bool verbose, bool showSystem)
 "n.nspname", "p.proname", NULL,
 "pg_catalog.pg_function_is_visible(p.oid)",
 NULL, 3))
+	{
+		termPQExpBuffer(&buf);
 		return false;
+	}
 
 	appendPQExpBufferStr(&buf, "ORDER BY 1, 2, 4;");
 
@@ -182,7 +185,10 @@ describeAccessMethods(const char *pattern, bool verbose)
 NULL, "amname", NULL,
 NULL,
 NULL, 1))
+	{
+		termPQExpBuffer(&buf);
 		return false;
+	}
 
 	appendPQExpBufferStr(&buf, "ORDER BY 1;");
 
@@ -244,7 +250,10 @@ describeTablespaces(const char *pattern, bool verbose)
 NULL, "spcname", NULL,
 NULL,
 NULL, 1))
+	{
+		termPQExpBuffer(&buf);
 		return false;
+	}
 
 	appendPQExpBufferStr(&buf, "ORDER BY 1;");
 
@@ -534,7 +543,7 @@ describeFunctions(const char *functypes, const char *func_pattern,
 "n.nspname", "p.proname", NULL,
 "pg_catalog.pg_function_is_visible(p.oid)",
 NULL, 3))
-		return false;
+		goto error_return;
 
 	for (int i = 0; i < num_arg_patterns; i++)
 	{
@@ -561,7 +570,7 @@ describeFunctions(const char *functypes, const char *func_pattern,
 		true, false,
 		nspname, typname, ft, tiv,
 		NULL, 3))
-return false;
+goto error_return;
 		}
 		else
 		{
@@ -599,6 +608,10 @@ describeFunctions(const char *functypes, const char *func_pattern,
 
 	PQclear(res);
 	return true;
+
+error_return:
+	termPQExpBuffer(&buf);
+	return false;
 }
 
 
@@ -682,7 +695,10 @@ describeTypes(const char *pattern, bool verbose, bool showSystem)
 "pg_catalog.format_type(t.oid, NULL)",
 "pg_catalog.pg_type_is_visible(t.oid)",
 NULL, 3))
+	{
+		termPQExpBuffer(&buf);
 		return false;
+	}
 
 	appendPQExpBufferStr(&buf, "ORDER BY 1, 2;");
 
@@ -836,7 +852,7 @@ describeOperators(const char *oper_pattern,
 "n.nspname", "o.oprname", NULL,
 "pg_catalog.pg_operator_is_visible(o.oid)",
 NULL, 3))
-		return false;
+		goto error_return;
 
 	if (num_arg_patterns == 1)
 		appendPQExpBufferStr(&buf, "  AND o.oprleft = 0\n");
@@ -866,7 +882,7 @@ describeOperators(const char *oper_pattern,
 		true, false,
 		nspname, typname, ft, tiv,
 		NULL, 3))
-return false;
+goto error_return;
 		}
 		else
 		{
@@ -890,6 +906,10 @@ describeOperators(const char *oper_pattern,
 
 	PQclear(res);
 	return true;
+
+error_return:
+	termPQExpBuffer(&buf);
+	return false;
 }
 
 
@@ -953,7 +973,10 @@ listAllDbs(const char *pattern, bool verbose)
 		if (!validateSQLNamePattern(&buf, pattern, false, false,
 	NULL, "d.datname", NULL, NULL,
 	NULL, 1))
+		{
+			termPQExpBuffer(&buf);
 			return false;
+		}
 
 	appendPQExpBufferStr(&buf, "ORDER BY 1;");
 	res = PSQLexec(buf.data);
@@ -1106,7 +1129,10 @@ permissionsList(const char *pattern)
 "n.nspname", "c.relname", NULL,
 "n.nspname !~ '^pg_' AND pg_catalog.pg_table_is_visible(c.oid)",
 NULL, 3))
+	{
+		termPQExpBuffer(&buf);
 		return false;
+	}
 
 	appendPQExpBufferStr(&buf, "ORDER BY 1, 2;");
 
@@ -1177,16 +1203,13 @@ listDefaultACLs(const char *pattern)
 "pg_catalog.pg_get_userbyid(d.defaclrole)",
 NULL,
 NULL, 3))
-		return false;
+		goto error_return;
 
 	appendPQExpBufferStr(&buf, "ORDER BY 1, 2, 3;");
 
 	res = PSQLexec(buf.data);
 	if (!res)
-	{
-		termPQExpBuffer(&buf);
-		return false;
-	}
+		goto error_return;
 
 	myopt.nullPrint = NULL;
 	printfPQExpBuffer(&buf, _("Default access privileges"));
@@ -1200,6 +1223,10 @@ listDefaultACLs(const char *pattern)
 	termPQExpBuffer(&buf);
 	PQclear(res);
 	return true;
+
+error_return:
+	termPQExpBuffer(&buf);
+	return false;
 }
 
 
@@ -1253,7 +1280,7 @@ objectDescription(const char *pattern, bool showSystem)
 false, 

Re: Memory leak fix in psql

[Tom Lane]

Japin Li  writes:
> Thanks for updating the patch.  It looks good.  However, it cannot be
> applied on 14 stable.  The attached patches are for 10-14.

While I think this is good cleanup, I'm doubtful that any of these
leaks are probable enough to be worth back-patching into stable
branches.  The risk of breaking something should not be neglected.

regards, tom lane

Re: Memory leak fix in psql

[Michael Paquier]

On Wed, Jul 20, 2022 at 10:05:47AM +0200, Alvaro Herrera wrote:
> More on the same tune.

Thanks.  I have noticed that as well.  I'll include all that in the
set.
--
Michael


signature.asc
Description: PGP signature

Re: Memory leak fix in psql

[Japin Li]

On Wed, 20 Jul 2022 at 21:54, Tom Lane  wrote:
> Japin Li  writes:
>> Thanks for updating the patch.  It looks good.  However, it cannot be
>> applied on 14 stable.  The attached patches are for 10-14.
>
> While I think this is good cleanup, I'm doubtful that any of these
> leaks are probable enough to be worth back-patching into stable
> branches.  The risk of breaking something should not be neglected.
>

Yeah, we should take care of the backpatch risk.  However, I think
it makes sense to backpatch.

-- 
Regrads,
Japin Li.
ChengDu WenWu Information Technology Co.,Ltd.

Re: Memory leak fix in psql

[Michael Paquier]

On Thu, Jul 21, 2022 at 09:10:43AM +0800, Japin Li wrote:
> Yeah, we should take care of the backpatch risk.  However, I think
> it makes sense to backpatch.

We are talking about 256 bytes being leaked in each loop when a
validation pattern or when a query fails, so I don't see a strong
argument in manipulating 10~14 more than necessary for this amount of
memory.  The contents of describe.c are the same for v15 though, and
we are still in beta on REL_15_STABLE, so I have applied the patch
down to v15, adding what Alvaro has sent on top of the rest.
--
Michael


signature.asc
Description: PGP signature

Re: Memory leak fix in psql

[Japin Li]

On Thu, 21 Jul 2022 at 09:48, Michael Paquier  wrote:
> On Thu, Jul 21, 2022 at 09:10:43AM +0800, Japin Li wrote:
>> Yeah, we should take care of the backpatch risk.  However, I think
>> it makes sense to backpatch.
>
> We are talking about 256 bytes being leaked in each loop when a
> validation pattern or when a query fails, so I don't see a strong
> argument in manipulating 10~14 more than necessary for this amount of
> memory.  The contents of describe.c are the same for v15 though, and
> we are still in beta on REL_15_STABLE, so I have applied the patch
> down to v15, adding what Alvaro has sent on top of the rest.

Thanks for the explanation!  IMO, we could ignore v10-13 branches, however,
we should backpatch to v14 which also uses the validateSQLNamePattern()
function leading to a memory leak.

-- 
Regrads,
Japin Li.
ChengDu WenWu Information Technology Co.,Ltd.

Re: Memory leak fix in psql

[Alvaro Herrera]

On 2022-Jul-21, Japin Li wrote:

> On Thu, 21 Jul 2022 at 09:48, Michael Paquier  wrote:

> > We are talking about 256 bytes being leaked in each loop when a
> > validation pattern or when a query fails, so I don't see a strong
> > argument in manipulating 10~14 more than necessary for this amount of
> > memory.  The contents of describe.c are the same for v15 though, and
> > we are still in beta on REL_15_STABLE, so I have applied the patch
> > down to v15, adding what Alvaro has sent on top of the rest.
> 
> Thanks for the explanation!  IMO, we could ignore v10-13 branches, however,
> we should backpatch to v14 which also uses the validateSQLNamePattern()
> function leading to a memory leak.

I'd agree in principle, but in practice the commit from 15 does not apply
cleanly on 14 -- there's a ton of conflicts.

-- 
Álvaro Herrera   48°01'N 7°57'E  —  https://www.EnterpriseDB.com/
"Here's a general engineering tip: if the non-fun part is too complex for you
to figure out, that might indicate the fun part is too ambitious." (John Naylor)
https://postgr.es/m/CAFBsxsG4OWHBbSDM%3DsSeXrQGOtkPiOEOuME4yD7Ce41NtaAD9g%40mail.gmail.com

Re: Memory leak fix in psql

[Japin Li]

On Thu, 21 Jul 2022 at 14:23, Alvaro Herrera  wrote:
> On 2022-Jul-21, Japin Li wrote:
>
>> On Thu, 21 Jul 2022 at 09:48, Michael Paquier  wrote:
>
>> > We are talking about 256 bytes being leaked in each loop when a
>> > validation pattern or when a query fails, so I don't see a strong
>> > argument in manipulating 10~14 more than necessary for this amount of
>> > memory.  The contents of describe.c are the same for v15 though, and
>> > we are still in beta on REL_15_STABLE, so I have applied the patch
>> > down to v15, adding what Alvaro has sent on top of the rest.
>> 
>> Thanks for the explanation!  IMO, we could ignore v10-13 branches, however,
>> we should backpatch to v14 which also uses the validateSQLNamePattern()
>> function leading to a memory leak.
>
> I'd agree in principle, but in practice the commit from 15 does not apply
> cleanly on 14 -- there's a ton of conflicts.

I attached a patch for v14 [1] based on master, if you want to apply it,
please consider reviewing it.

[1] 
https://www.postgresql.org/message-id/MEYP282MB166974D3883A1A6E25B5FDAEB68E9%40MEYP282MB1669.AUSP282.PROD.OUTLOOK.COM
-- 
Regrads,
Japin Li.
ChengDu WenWu Information Technology Co.,Ltd.

Re: Memory leak fix in psql

[Michael Paquier]

On Thu, Jul 21, 2022 at 02:43:03PM +0800, Japin Li wrote:
> I attached a patch for v14 [1] based on master, if you want to apply it,
> please consider reviewing it.

We are talking about a few hundred bytes leaked each time, so this
does not worry me much in the older branches, honestly.
--
Michael


signature.asc
Description: PGP signature