RE: BUG #16481: Stored Procedure Triggered by Logical Replication is Unable to use Notification Events
Just to help those who find themselves in the same situation, there is a simple
application-level workaround which consists in listening to the notifications
in the replica when they are issued by the master and vice versa.
We are programming in .NET and we use a code like this:
Code in the replica side:
var csb = new NpgsqlConnectionStringBuilder
{
Host = "master",
Database = "MasterDB",
Port = 5432,
Username = "postgres",
Password = "XXX",
};
var connection = new NpgsqlConnection(csb.ConnectionString);
connection.Open();
using (var command = new NpgsqlCommand("listen \"Table\"",
connection))
{
command.ExecuteNonQuery();
}
connection.Notification += PostgresNotification;
So you can listen from the replica every changed raised by a trigger on the
master from the replica side on the table "Table".
CREATE TRIGGER master_trigger
AFTER INSERT OR DELETE OR UPDATE
ON public."TABLE"
FOR EACH ROW
EXECUTE PROCEDURE public.master_notice();
ALTER TABLE public."Tabele"
ENABLE ALWAYS TRIGGER master_trigger;
CREATE FUNCTION public. master_notice ()
RETURNS trigger
LANGUAGE 'plpgsql'
COST 100
VOLATILE NOT LEAKPROOF
AS $BODY$
BEGIN
PERFORM pg_notify('Table', Cast(NEW."ID" as text));
RETURN NEW;
END;
$BODY$;
ALTER FUNCTION public.incupdate1_notice()
OWNER TO postgres;
I hope that help someone, because the bug last from "years". I tried in version
10 11 and 12, so it is present since 2017-10-05 and I can't see any solution on
13 beta.
Best Regards.
Fabio.
-Original Message-
From: Vianello Fabio
Sent: lunedì 8 giugno 2020 11:14
To: Kyotaro Horiguchi
Cc: pgsql-b...@lists.postgresql.org; pgsql-hackers@lists.postgresql.org
Subject: RE: BUG #16481: Stored Procedure Triggered by Logical Replication is
Unable to use Notification Events
Hi Kyotaro Horiguchi, thanks for you helps.
We have a question about the bug. Why there isn't any solution in the HEAD?
This bug last since 10.4 version and I can't understand why it is not fixed in
the HEAD yet.
BR.
Fabio Vianello.
-Original Message-
From: Kyotaro Horiguchi [mailto:horikyota....@gmail.com]
Sent: lunedì 8 giugno 2020 10:28
To: Vianello Fabio ;
pgsql-b...@lists.postgresql.org; pgsql-hackers@lists.postgresql.org
Subject: Re: BUG #16481: Stored Procedure Triggered by Logical Replication is
Unable to use Notification Events
Hello.
It seems to me a bug.
At Fri, 05 Jun 2020 11:05:14 +, PG Bug reporting form
wrote in
> The following bug has been logged on the website:
>
> Bug reference: 16481
> Logged by: Fabio Vianello
> Email address: fabio.viane...@salvagninigroup.com
> PostgreSQL version: 12.3
> Operating system: Windows 10
> Description:
>
> About the bug BUG #15293, on PostgreSQL version 10.4 and 11.2 as
> describe below, we found the same issue on the PostgreSQL version 12.3.
The HEAD behaves the same way.
> Is it a feature?
> Becasue in the documentation we didn't found any constraint that says
> that we can not use NOTIFY/LISTEN on logical replication tables.
>
> "When using logical replication a stored procedure executed on the
> replica is unable to use NOTIFY to send messages to other listeners.
> The stored procedure does execute as expected but the pg_notify()
> doesn't appear to have any effect. If an insert is run on the replica
> side the trigger executes the stored procedure as expected and the
> NOTIFY correctly notifies listeners.
The message is actually queued, but logical replication worker doesn't signal
that to listener backends. If any ordinary session sent a message to the same
listener after that, both messages would be shown at once.
That can be fixed by calling ProcessCompletedNotifies() in apply_handle_commit.
The function has a code to write out notifications to connected clients but it
doesn't nothing on logical replication workers.
regards.
--
Kyotaro Horiguchi
NTT Open Source Software Center
SALVAGNINI ITALIA S.p.A.
Via Guido Salvagnini, 51 - IT - 36040 Sarego (VI)
T. +39 0444 725111 | F. +39 0444 43 6404
Società a socio unico - Attività direz. e coord.: Salvagnini Holding S.p.A.
Clicca qui<https://www.salvagninigroup.com/company-information> per le
informazioni societarie
salvagninigroup.com<https://www.salvagninigroup.com> |
salvagnini.it<http://www.salvagnini.it>
Le informazioni trasmesse sono destinate esclusivamente alla persona o alla
società in indirizzo e sono da intendersi confidenziali e riservate. Ogni
trasmissione, inoltro, diffusione o altro uso di q
RE: BUG #16481: Stored Procedure Triggered by Logical Replication is Unable to use Notification Events
I think you did follow all the thread. I only ask if it is a bug or not. If it as bug and last for years I understand your point our view but I have my. If you think that signal a bug is not give a contribution I am astonished. Improve PosgreSQL is the target Best Regard. Fabio Vianello. From: David G. Johnston [mailto:david.g.johns...@gmail.com] Sent: martedì 9 giugno 2020 17:10 To: Vianello Fabio Cc: PostgreSQL Hackers ; pgsql-b...@lists.postgresql.org; Kyotaro Horiguchi ; Euler Taveira Subject: Re: BUG #16481: Stored Procedure Triggered by Logical Replication is Unable to use Notification Events On Tue, Jun 9, 2020 at 12:36 AM Vianello Fabio mailto:fabio.viane...@salvagninigroup.com>> wrote: Is PostgreSQL a serious product? For me the answer is "NO". A product with a bug that last for years and the community knows. It is not serious. If you are trying to be a troll just go away, we don't need that here. If you are just venting consider that this project is no more or less likely to have oversights, incomplete documentation, or a myriad of other human induced issues than any other. Reading the linked bug report the conclusion was "won't fix - at least not right now". Sure, it probably should have been documented but wasn't. It happens. And given the lack of complaints in the intervening years the decision, to not devote volunteer resources to a marginal feature, seems like the right one. Your active recourse at this point is to either convince a volunteer hacker to take up the cause - which your attitude doesn't help - or pay someone to do it. Or figure out a personal work-around to live with the current reality. Given that there is ongoing discussion as a result of your report (i.e., the report has merit regardless of how it was reported) means you should either meaningfully contribute to the discussion or shut up until they are done - at which point you are back to making a decision. Prompting politely for attention if the thread seems to languish without a clear resolution is, IMO, acceptable. David J. SALVAGNINI ITALIA S.p.A. Via Guido Salvagnini, 51 - IT - 36040 Sarego (VI) T. +39 0444 725111 | F. +39 0444 43 6404 Società a socio unico - Attività direz. e coord.: Salvagnini Holding S.p.A. Clicca qui<https://www.salvagninigroup.com/company-information> per le informazioni societarie salvagninigroup.com<https://www.salvagninigroup.com> | salvagnini.it<http://www.salvagnini.it> Le informazioni trasmesse sono destinate esclusivamente alla persona o alla società in indirizzo e sono da intendersi confidenziali e riservate. Ogni trasmissione, inoltro, diffusione o altro uso di queste informazioni a persone o società differenti dal destinatario è proibita. Se avete ricevuto questa comunicazione per errore, per favore contattate il mittente e cancellate le informazioni da ogni computer. Questa casella di posta elettronica è riservata esclusivamente all’invio ed alla ricezione di messaggi aziendali inerenti all’attività lavorativa e non è previsto né autorizzato l’utilizzo per fini personali. Pertanto i messaggi in uscita e quelli di risposta in entrata verranno trattati quali messaggi aziendali e soggetti alla ordinaria gestione disposta con proprio disciplinare dall’azienda e, di conseguenza, eventualmente anche alla lettura da parte di persone diverse dall’intestatario della casella. Any information herein transmitted only concerns the person or the company named in the address and is deemed to be confidential. It is strictly forbidden to transmit, post, forward or otherwise use said information to anyone other than the recipient. If you have received this message by mistake, please contact the sender and delete any relevant information from your computer. This mailbox is only meant for sending and receiving messages pertaining business matters and any other use for personal purposes is forbidden and unauthorized. Therefore, any email sent and received will be handled as ordinary business messages and subject to the company's own rules, and may thus be read also by people other than the user named in the mailbox address.
Re: BUG #16481: Stored Procedure Triggered by Logical Replication is Unable to use Notification Events
On Tue, Jun 9, 2020 at 12:36 AM Vianello Fabio < fabio.viane...@salvagninigroup.com> wrote: > Is PostgreSQL a serious product? For me the answer is "NO". A product with > a bug that last for years and the community knows. > > It is not serious. > If you are trying to be a troll just go away, we don't need that here. If you are just venting consider that this project is no more or less likely to have oversights, incomplete documentation, or a myriad of other human induced issues than any other. Reading the linked bug report the conclusion was "won't fix - at least not right now". Sure, it probably should have been documented but wasn't. It happens. And given the lack of complaints in the intervening years the decision, to not devote volunteer resources to a marginal feature, seems like the right one. Your active recourse at this point is to either convince a volunteer hacker to take up the cause - which your attitude doesn't help - or pay someone to do it. Or figure out a personal work-around to live with the current reality. Given that there is ongoing discussion as a result of your report (i.e., the report has merit regardless of how it was reported) means you should either meaningfully contribute to the discussion or shut up until they are done - at which point you are back to making a decision. Prompting politely for attention if the thread seems to languish without a clear resolution is, IMO, acceptable. David J.
RE: BUG #16481: Stored Procedure Triggered by Logical Replication is Unable to use Notification Events
Is PostgreSQL a serious product? For me the answer is "NO". A product with a bug that last for years and the community knows. It is not serious. BR, Fabio. From: Euler Taveira [mailto:euler.tave...@2ndquadrant.com] Sent: lunedì 8 giugno 2020 12:51 To: Kyotaro Horiguchi Cc: Vianello Fabio ; pgsql-b...@lists.postgresql.org; PostgreSQL Hackers Subject: Re: BUG #16481: Stored Procedure Triggered by Logical Replication is Unable to use Notification Events On Mon, 8 Jun 2020 at 05:27, Kyotaro Horiguchi mailto:horikyota@gmail.com>> wrote: That can be fixed by calling ProcessCompletedNotifies() in apply_handle_commit. The function has a code to write out notifications to connected clients but it doesn't nothing on logical replication workers. This bug was already reported some time ago (#15293) but it slipped through the cracks. I don't think you should simply call ProcessCompletedNotifies [1]. [1] https://www.postgresql.org/message-id/13844.1532468610%40sss.pgh.pa.us -- Euler Taveira http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services SALVAGNINI ITALIA S.p.A. Via Guido Salvagnini, 51 - IT - 36040 Sarego (VI) T. +39 0444 725111 | F. +39 0444 43 6404 Società a socio unico - Attività direz. e coord.: Salvagnini Holding S.p.A. Clicca qui<https://www.salvagninigroup.com/company-information> per le informazioni societarie salvagninigroup.com<https://www.salvagninigroup.com> | salvagnini.it<http://www.salvagnini.it> Le informazioni trasmesse sono destinate esclusivamente alla persona o alla società in indirizzo e sono da intendersi confidenziali e riservate. Ogni trasmissione, inoltro, diffusione o altro uso di queste informazioni a persone o società differenti dal destinatario è proibita. Se avete ricevuto questa comunicazione per errore, per favore contattate il mittente e cancellate le informazioni da ogni computer. Questa casella di posta elettronica è riservata esclusivamente all’invio ed alla ricezione di messaggi aziendali inerenti all’attività lavorativa e non è previsto né autorizzato l’utilizzo per fini personali. Pertanto i messaggi in uscita e quelli di risposta in entrata verranno trattati quali messaggi aziendali e soggetti alla ordinaria gestione disposta con proprio disciplinare dall’azienda e, di conseguenza, eventualmente anche alla lettura da parte di persone diverse dall’intestatario della casella. Any information herein transmitted only concerns the person or the company named in the address and is deemed to be confidential. It is strictly forbidden to transmit, post, forward or otherwise use said information to anyone other than the recipient. If you have received this message by mistake, please contact the sender and delete any relevant information from your computer. This mailbox is only meant for sending and receiving messages pertaining business matters and any other use for personal purposes is forbidden and unauthorized. Therefore, any email sent and received will be handled as ordinary business messages and subject to the company's own rules, and may thus be read also by people other than the user named in the mailbox address.
Re: BUG #16481: Stored Procedure Triggered by Logical Replication is Unable to use Notification Events
Hello, Euler.
At Mon, 8 Jun 2020 07:51:18 -0300, Euler Taveira
wrote in
> On Mon, 8 Jun 2020 at 05:27, Kyotaro Horiguchi
> wrote:
>
> >
> > That can be fixed by calling ProcessCompletedNotifies() in
> > apply_handle_commit. The function has a code to write out
> > notifications to connected clients but it doesn't nothing on logical
> > replication workers.
> >
> >
> This bug was already reported some time ago (#15293) but it slipped through
> the
> cracks. I don't think you should simply call ProcessCompletedNotifies [1].
Yeah, Thanks for pointing that. I faintly thought of a similar thing
to the discussion there. Just calling ProcessCompletedNotifies in
apply_handle_commit is actually wrong.
We can move only SignalBackends() to AtCommit_Notify since
asyncQueueAdvanceTail() is no longer dependent on the result of
SignalBackends, but anyway we need to call asyncQueueAdvanceTail in
AtCommit_Notify and AtAbort_Notify since otherwise the queue cannot be
shorten while running logical replication. This can slightly defers
tail-advancing but I think it wouldn't be a significant problem.
> [1] https://www.postgresql.org/message-id/13844.1532468610%40sss.pgh.pa.us
regards.
--
Kyotaro Horiguchi
NTT Open Source Software Center
>From c3aa3e584cf57632284dc9b282dd635c418f3084 Mon Sep 17 00:00:00 2001
From: Kyotaro Horiguchi
Date: Tue, 9 Jun 2020 14:01:34 +0900
Subject: [PATCH v2] Fix notification signaling
Notifications are signaled at command loop. That prevents logical
replication apply loop from signaling properly. To fix, send signal
in AtCommit_Notify instead of the top-level command loop.
Discussion: https://www.postgresql.org/message-id/13844.1532468610%40sss.pgh.pa.us
Discussion: https://www.postgresql.org/message-id/20200608.172730.68580977059033.horikyota.ntt%40gmail.com
---
src/backend/commands/async.c | 103 +--
1 file changed, 63 insertions(+), 40 deletions(-)
diff --git a/src/backend/commands/async.c b/src/backend/commands/async.c
index 71b7577afc..590ad7fcc8 100644
--- a/src/backend/commands/async.c
+++ b/src/backend/commands/async.c
@@ -449,7 +449,7 @@ static void asyncQueueNotificationToEntry(Notification *n, AsyncQueueEntry *qe);
static ListCell *asyncQueueAddEntries(ListCell *nextNotify);
static double asyncQueueUsage(void);
static void asyncQueueFillWarning(void);
-static void SignalBackends(void);
+static bool SignalBackends(void);
static void asyncQueueReadAllNotifications(void);
static bool asyncQueueProcessPageEntries(volatile QueuePosition *current,
QueuePosition stop,
@@ -976,7 +976,8 @@ PreCommit_Notify(void)
*
* This is called at transaction commit, after committing to clog.
*
- * Update listenChannels and clear transaction-local state.
+ * Update listenChannels and clear transaction-local state. Send signals
+ * for notifications to other backends to process them.
*/
void
AtCommit_Notify(void)
@@ -1021,6 +1022,29 @@ AtCommit_Notify(void)
/* And clean up */
ClearPendingActionsAndNotifies();
+
+ /* signal our notifications to other backends */
+ if (backendHasSentNotifications)
+ {
+ /*
+ * No use reading the queue at idle time later if this backend is not a
+ * listener.
+ */
+ if (!SignalBackends())
+ backendHasSentNotifications = false;
+
+ /*
+ * If it's time to try to advance the global tail pointer, do that. We
+ * need do this here in case where many transactions are committed
+ * without returning to the top-level loop, like logical replication
+ * apply loop.
+ */
+ if (backendTryAdvanceTail)
+ {
+ backendTryAdvanceTail = false;
+ asyncQueueAdvanceTail();
+ }
+ }
}
/*
@@ -1196,10 +1220,8 @@ Exec_UnlistenAllCommit(void)
*
* This is called from postgres.c just before going idle at the completion
* of a transaction. If we issued any notifications in the just-completed
- * transaction, send signals to other backends to process them, and also
- * process the queue ourselves to send messages to our own frontend.
- * Also, if we filled enough queue pages with new notifies, try to advance
- * the queue tail pointer.
+ * transaction, process the queue ourselves to send messages to our own
+ * frontend.
*
* The reason that this is not done in AtCommit_Notify is that there is
* a nonzero chance of errors here (for example, encoding conversion errors
@@ -1208,17 +1230,11 @@ Exec_UnlistenAllCommit(void)
* to ensure that a transaction's self-notifies are delivered to the frontend
* before it gets the terminating ReadyForQuery message.
*
- * Note that we send signals and process the queue even if the transaction
- * eventually aborted. This is because we need to clean out whatever got
- * added to the queue.
- *
* NOTE: we are outside of any transaction here.
*/
void
ProcessCompletedNotifies(void)
{
- MemoryContext caller_context;
-
/* Nothing to do if we didn't send any notifications */
if (!backendHasSentNotifications)
return;
@@ -1230,43 +1246,32
Re: BUG #16481: Stored Procedure Triggered by Logical Replication is Unable to use Notification Events
On Mon, 8 Jun 2020 at 05:27, Kyotaro Horiguchi wrote: > > That can be fixed by calling ProcessCompletedNotifies() in > apply_handle_commit. The function has a code to write out > notifications to connected clients but it doesn't nothing on logical > replication workers. > > This bug was already reported some time ago (#15293) but it slipped through the cracks. I don't think you should simply call ProcessCompletedNotifies [1]. [1] https://www.postgresql.org/message-id/13844.1532468610%40sss.pgh.pa.us -- Euler Taveira http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
RE: BUG #16481: Stored Procedure Triggered by Logical Replication is Unable to use Notification Events
Hi Kyotaro Horiguchi, thanks for you helps. We have a question about the bug. Why there isn't any solution in the HEAD? This bug last since 10.4 version and I can't understand why it is not fixed in the HEAD yet. BR. Fabio Vianello. -Original Message- From: Kyotaro Horiguchi [mailto:horikyota@gmail.com] Sent: lunedì 8 giugno 2020 10:28 To: Vianello Fabio ; pgsql-b...@lists.postgresql.org; pgsql-hackers@lists.postgresql.org Subject: Re: BUG #16481: Stored Procedure Triggered by Logical Replication is Unable to use Notification Events Hello. It seems to me a bug. At Fri, 05 Jun 2020 11:05:14 +, PG Bug reporting form wrote in > The following bug has been logged on the website: > > Bug reference: 16481 > Logged by: Fabio Vianello > Email address: fabio.viane...@salvagninigroup.com > PostgreSQL version: 12.3 > Operating system: Windows 10 > Description: > > About the bug BUG #15293, on PostgreSQL version 10.4 and 11.2 as > describe below, we found the same issue on the PostgreSQL version 12.3. The HEAD behaves the same way. > Is it a feature? > Becasue in the documentation we didn't found any constraint that says > that we can not use NOTIFY/LISTEN on logical replication tables. > > "When using logical replication a stored procedure executed on the > replica is unable to use NOTIFY to send messages to other listeners. > The stored procedure does execute as expected but the pg_notify() > doesn't appear to have any effect. If an insert is run on the replica > side the trigger executes the stored procedure as expected and the > NOTIFY correctly notifies listeners. The message is actually queued, but logical replication worker doesn't signal that to listener backends. If any ordinary session sent a message to the same listener after that, both messages would be shown at once. That can be fixed by calling ProcessCompletedNotifies() in apply_handle_commit. The function has a code to write out notifications to connected clients but it doesn't nothing on logical replication workers. regards. -- Kyotaro Horiguchi NTT Open Source Software Center SALVAGNINI ITALIA S.p.A. Via Guido Salvagnini, 51 - IT - 36040 Sarego (VI) T. +39 0444 725111 | F. +39 0444 43 6404 Società a socio unico - Attività direz. e coord.: Salvagnini Holding S.p.A. Clicca qui<https://www.salvagninigroup.com/company-information> per le informazioni societarie salvagninigroup.com<https://www.salvagninigroup.com> | salvagnini.it<http://www.salvagnini.it> Le informazioni trasmesse sono destinate esclusivamente alla persona o alla società in indirizzo e sono da intendersi confidenziali e riservate. Ogni trasmissione, inoltro, diffusione o altro uso di queste informazioni a persone o società differenti dal destinatario è proibita. Se avete ricevuto questa comunicazione per errore, per favore contattate il mittente e cancellate le informazioni da ogni computer. Questa casella di posta elettronica è riservata esclusivamente all’invio ed alla ricezione di messaggi aziendali inerenti all’attività lavorativa e non è previsto né autorizzato l’utilizzo per fini personali. Pertanto i messaggi in uscita e quelli di risposta in entrata verranno trattati quali messaggi aziendali e soggetti alla ordinaria gestione disposta con proprio disciplinare dall’azienda e, di conseguenza, eventualmente anche alla lettura da parte di persone diverse dall’intestatario della casella. Any information herein transmitted only concerns the person or the company named in the address and is deemed to be confidential. It is strictly forbidden to transmit, post, forward or otherwise use said information to anyone other than the recipient. If you have received this message by mistake, please contact the sender and delete any relevant information from your computer. This mailbox is only meant for sending and receiving messages pertaining business matters and any other use for personal purposes is forbidden and unauthorized. Therefore, any email sent and received will be handled as ordinary business messages and subject to the company's own rules, and may thus be read also by people other than the user named in the mailbox address.
Re: BUG #16481: Stored Procedure Triggered by Logical Replication is Unable to use Notification Events
Hello.
It seems to me a bug.
At Fri, 05 Jun 2020 11:05:14 +, PG Bug reporting form
wrote in
> The following bug has been logged on the website:
>
> Bug reference: 16481
> Logged by: Fabio Vianello
> Email address: fabio.viane...@salvagninigroup.com
> PostgreSQL version: 12.3
> Operating system: Windows 10
> Description:
>
> About the bug BUG #15293, on PostgreSQL version 10.4 and 11.2 as describe
> below, we found the same issue on the PostgreSQL version 12.3.
The HEAD behaves the same way.
> Is it a feature?
> Becasue in the documentation we didn't found any constraint that says that
> we can not use NOTIFY/LISTEN on logical replication tables.
>
> "When using logical replication a stored procedure executed on the replica
> is
> unable to use NOTIFY to send messages to other listeners. The stored
> procedure does execute as expected but the pg_notify() doesn't appear to
> have any effect. If an insert is run on the replica side the trigger
> executes the stored procedure as expected and the NOTIFY correctly
> notifies
> listeners.
The message is actually queued, but logical replication worker doesn't
signal that to listener backends. If any ordinary session sent a
message to the same listener after that, both messages would be shown
at once.
That can be fixed by calling ProcessCompletedNotifies() in
apply_handle_commit. The function has a code to write out
notifications to connected clients but it doesn't nothing on logical
replication workers.
regards.
--
Kyotaro Horiguchi
NTT Open Source Software Center
>From b8df8c0cf6ae6c2bcd78ffd7d9bd629f51ab3bee Mon Sep 17 00:00:00 2001
From: Kyotaro Horiguchi
Date: Mon, 8 Jun 2020 16:07:41 +0900
Subject: [PATCH] Signal notifications from logical replication workers
Notifications need to be signaled to listeners but logical replication
worker forgets to do that. Fix that by signaling notifications after
committing a transaction.
---
src/backend/replication/logical/worker.c | 4
1 file changed, 4 insertions(+)
diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c
index a752a1224d..28ae89c574 100644
--- a/src/backend/replication/logical/worker.c
+++ b/src/backend/replication/logical/worker.c
@@ -33,6 +33,7 @@
#include "catalog/pg_inherits.h"
#include "catalog/pg_subscription.h"
#include "catalog/pg_subscription_rel.h"
+#include "commands/async.h"
#include "commands/tablecmds.h"
#include "commands/trigger.h"
#include "executor/executor.h"
@@ -517,6 +518,9 @@ apply_handle_commit(StringInfo s)
pgstat_report_stat(false);
store_flush_position(commit_data.end_lsn);
+
+ /* Send out notify signals */
+ ProcessCompletedNotifies();
}
else
{
--
2.18.2
