date:20190109

Re: [devel] [PATCH 1/1] imm: allow empty-value attribute with default-tag persisted [#2985]

2019-01-09 Thread Vu Minh Nguyen

Hi,

Any comment from you? I will push this ticket B/next week if no comment.

Regards, Vu

> -Original Message-
> From: Vu Minh Nguyen 
> Sent: Wednesday, December 19, 2018 6:11 PM
> To: hans.nordeb...@ericsson.com; lennart.l...@ericsson.com;
> gary@dektech.com.au
> Cc: opensaf-devel@lists.sourceforge.net; Vu Minh Nguyen
> 
> Subject: [PATCH 1/1] imm: allow empty-value attribute with default-tag
> persisted [#2985]
> 
> During runtime, when replacing value of an attribute which has
default-value
> tag with NULL, this NULL value is not persistent after cluster is rebooted
-
> NULL value will be automatically replaced with its default value by IMM.
> 
> This behavior causes several unexpected results. Below is a use case.
> 
> User defines an attribute with name `maxAge`. It is used to shows how many
> days
> user passwords will be expired, default value is 30 days; if replacing
with
> a NULL/empty, it means the passwords will never get expired.
> 
> During runtime, user changes the existing value with NULL - expect the
> passwords
> never get expired. Later on, then cluster is rebooted, that value is
silently
> replaced with the default without notice of user.
> 
> This patch makes some changes in immdump/immloader/imm om
> library/immnd
> to make NULL value in such case persisted even after cluster is rebooted.
> ---
>  src/imm/README   | 10 +---
>  src/imm/agent/imma_om_api.cc |  7 ++-
>  src/imm/immloadd/imm_loader.cc   | 23 +---
>  src/imm/immloadd/imm_pbe_load.cc | 16 --
>  src/imm/immnd/ImmModel.cc|  9 ++-
>  src/imm/tools/imm_xmlw_dump.cc   | 96 
>  6 files changed, 100 insertions(+), 61 deletions(-)
> 
> diff --git a/src/imm/README b/src/imm/README
> index 7ace34741..132ee0ac0 100644
> --- a/src/imm/README
> +++ b/src/imm/README
> @@ -351,14 +351,8 @@ as part of an attribute definition.
>  (i) A default declaration is only allowed for single valued attributes
(no
>  concept of a multivalued default exists).
> 
> -(ii) Default values are assigned at object creation. Default values are
NOT
> -assigned if an attribute is set to the empty/null value by a
modification.
> -
> -(iii) Default values are assigned at cluster restart for any attributes
that
> -are null/empty and that have a default. This is a special case of (i)
because
> -imm loading actually uses the regular imm API to recreate the imm
contents.
> -In particular, saImmOmCcbObjectCreate is used to recreate all objects
from
> -the file-system image.
> +(ii) Default values are assigned at object creation, except the creation
> comes
> +from IMM loader.
> 
>  Common missunderstandings about "system attributes" of an imm object.
>  -
> diff --git a/src/imm/agent/imma_om_api.cc
> b/src/imm/agent/imma_om_api.cc
> index 7155799d9..0d24b2335 100644
> --- a/src/imm/agent/imma_om_api.cc
> +++ b/src/imm/agent/imma_om_api.cc
> @@ -2037,7 +2037,7 @@ static SaAisErrorT ccb_object_create_common(
>  TRACE_2("ERR_INVALID_PARAM: Not allowed to set attribute %s",
>  sysaImplName);
>  goto mds_send_fail;
> -  } else if (attr->attrValuesNumber == 0) {
> +  } else if (attr->attrValuesNumber == 0 && !immOmIsLoader) {
>  TRACE("CcbObjectCreate ignoring attribute %s with no values",
>attr->attrName);
>  continue;
> @@ -2065,7 +2065,9 @@ static SaAisErrorT ccb_object_create_common(
> 
>const SaImmAttrValueT *avarr = attr->attrValues;
>/*alloc-5 */
> -  imma_copyAttrValue(&(p->n.attrValue), attr->attrValueType,
avarr[0]);
> +  if (attr->attrValuesNumber > 0) {
> +imma_copyAttrValue(&(p->n.attrValue), attr->attrValueType,
avarr[0]);
> +  }
> 
>if (attr->attrValuesNumber > 1) {
>  unsigned int numAdded = attr->attrValuesNumber - 1;
> @@ -2087,6 +2089,7 @@ static SaAisErrorT ccb_object_create_common(
>  }
>}
> 
> +
>rc = imma_evt_fake_evs(cb, , _evt, cl_node->syncr_timeout,
>   cl_node->handle, , false);
>cl_node = NULL;
> diff --git a/src/imm/immloadd/imm_loader.cc
> b/src/imm/immloadd/imm_loader.cc
> index e9a985c22..3bd3e2b2e 100644
> --- a/src/imm/immloadd/imm_loader.cc
> +++ b/src/imm/immloadd/imm_loader.cc
> @@ -117,6 +117,7 @@ typedef struct ParserStateStruct {
>SaImmAttrFlagsT attrFlags;
>SaUint32T attrNtfId;
>char *attrDefaultValueBuffer;
> +  bool is_null_value;
> 
>int attrValueTypeSet;
>int attrNtfIdSet;
> @@ -925,6 +926,13 @@ static void startElementHandler(void *userData,
> const xmlChar *name,
>  state->state[state->depth] = VALUE;
>  state->valueContinue = 0;
>  state->isBase64Encoded = isBase64Encoded(attrs);
> +state->is_null_value = false;
> +if (attrs) {
> +  char* null_value = getAttributeValue("xsi:nil", attrs);
> +  if (null_value && std::string{null_value} == "true") {
> +

Re: [devel] [PATCH 1/2] ntf: Limit the logger buffer [#2961]

2019-01-09 Thread Minh Hon Chau


Hi Canh,

ack with comments, please see with [M]

Thanks

Minh

On 9/1/19 8:22 pm, Canh Van Truong wrote:

When writing the notificaion fail with TRY_AGAIN in callback, the notificaion 
is pushed
again to the list. If this happens for long time, the list is going to be very 
big.
This cause NTFD take time to process writing all the notification in the list 
and
the request from NTFA come this time may be timeout.

The patch does:
- Limit the logger buffer
- Provide the env variable that user can set the value of the limit
- Return TRY_AGAIN error in case the limit of buffer is reached and write 
all the
  notifications in the buffer to the log file. The current of notification 
isn't
  written to log file.
---
  src/ntf/README|  10 
  src/ntf/ntfd/NtfAdmin.cc  |  44 +-
  src/ntf/ntfd/NtfLogger.cc | 149 +++---
  src/ntf/ntfd/NtfLogger.h  |  11 +++-
  src/ntf/ntfd/ntfd.conf|  10 
  src/ntf/ntfd/ntfs.h   |   2 +
  6 files changed, 147 insertions(+), 79 deletions(-)

diff --git a/src/ntf/README b/src/ntf/README
index 6dd5173e1..5bf670647 100644
--- a/src/ntf/README
+++ b/src/ntf/README
@@ -233,6 +233,16 @@ NTFSV_ENV_CACHE_SIZE
  The size of the notification cache in the NTF server processes running on the 
Controller nodes.
  The default value is 1 notification.
  
+NTFSV_LOGGER_BUFFER_CAPACITY

+
+The logger buffer is used to store the notification if writing notification
+to log file fail. This variable is set for limit of logger buffer size in
+NTFD. If the logger buffer is full and NTFD receives new notification,
+the TRY_AGAIN  error is returned to user. The limit should be set with relevant
+value to avoid congestion in NTFD. Because if this value is set too big and
+writing notification is fail for long time, NTF has to write a big number of
+notifications whenever handling sending notification request and that will 
delay
+to handle other requests come to NTFD. The value of variable is from 10 to 
5000.
  
  for debug see DEBUG.
  
diff --git a/src/ntf/ntfd/NtfAdmin.cc b/src/ntf/ntfd/NtfAdmin.cc

index 2cb99457c..6c2d69b43 100644
--- a/src/ntf/ntfd/NtfAdmin.cc
+++ b/src/ntf/ntfd/NtfAdmin.cc
@@ -193,19 +193,32 @@ void NtfAdmin::processNotification(unsigned int clientId,
notificationId, notificationType,
(unsigned int)notificationMap.size());
  
-  // log the notification. Callback from SAF log will confirm later.

-  logger.log(notification, activeController());
-
-  /* send notification to standby */
-  sendNotificationUpdate(clientId, notification->getNotInfo());
+  if ((logger.isLoggerBufferFull() == true) &&
+  (logger.isAlarmNotification(notification) == true)) {
+NtfClient *client = getClient(clientId);
+MDS_DEST dest = client->getMdsDest();
+LOG_WA("The logger buffer is full. Check if there is issue in writing");
+if (activeController())
+  notfication_result_lib(SA_AIS_ERR_TRY_AGAIN, notificationId,
+ mdsCtxt, dest);
+  } else {
+/* send notification to standby */
+sendNotificationUpdate(clientId, notification->getNotInfo());
  
-  ClientMap::iterator pos;

-  for (pos = clientMap.begin(); pos != clientMap.end(); pos++) {
-NtfClient *client = pos->second;
-client->notificationReceived(clientId, notification, mdsCtxt);
+ClientMap::iterator pos;
+for (pos = clientMap.begin(); pos != clientMap.end(); pos++) {
+  NtfClient *client = pos->second;
+  client->notificationReceived(clientId, notification, mdsCtxt);
+}
}
  
-  /* remove notification if sent to all subscribers and logged */

+  // Log the notification. Callback from SAF log will confirm later.
+  if (activeController())
+logger.log(notification);
+  // Add the notification to Reader list
+  logger.addNotificationToReaderList(notification);
+
+  // Remove the notification if it is sent to all subscribers and logged
if (notification->isSubscriptionListEmpty() && notification->loggedOk()) {
  NotificationMap::iterator posNot;
  posNot = notificationMap.find(notificationId);


[M]: If ntfd decides to return TRY_AGAIN, then the notification should 
not be added for readers to read, and for subscription checking, etc 
I think it looks like this


if (the buffer is not empty) {

    // try to flush all pending log

}

if (the buffer is still full) {

    // return try again

} else {

    // add this to buffer, checkpoint, add to reader lists,  as normal

}



@@ -341,9 +354,9 @@ void NtfAdmin::notificationReceivedColdSync(
TRACE_LEAVE();
  }
  /**
- * A cached notification is received in Cold Sync.
- * This cached notification will be marked as logged, and stored
- * only in NtfLogger class to serve the reader.
+ * A cached notifications are received in Cold Sync.
+ * This cached notifications are stored in NtfLogger
+ * class to serve the reader.
   *
   * @param clientId Node-wide unique id for the

[devel] FW: [PATCH 1/2] ntf: Limit the logger buffer [#2961]

2019-01-09 Thread Canh Van Truong

Thanks Lennart for quick review

 

is_buffer_full  is needed because we need to check if the buffer is full or
not before we log all notification in the buffer and empty the buffer. Then
we check if the buffer was full as before, the new notification shouldnt be
logged

 

Regards

Canh

 

From: Lennart Lund  
Sent: Wednesday, January 9, 2019 6:38 PM
To: Canh Van Truong ; Minh Hon Chau

Cc: opensaf-devel@lists.sourceforge.net
Subject: SV: [PATCH 1/2] ntf: Limit the logger buffer [#2961]

 

Hi Canh

 

Ack for "ntf: Limit the logger buffer" commit with some comments. See
[Lennart] below

 

Thanks

Lennart

 

  _  

Från: Canh Van Truong mailto:canh.v.tru...@dektech.com.au> >
Skickat: den 9 januari 2019 10:22
Till: Lennart Lund; Minh Hon Chau
Kopia: opensaf-devel@lists.sourceforge.net
 ; Canh Van Truong
Ämne: [PATCH 1/2] ntf: Limit the logger buffer [#2961] 

 

When writing the notificaion fail with TRY_AGAIN in callback, the
notificaion is pushed
again to the list. If this happens for long time, the list is going to be
very big.
This cause NTFD take time to process writing all the notification in the
list and
the request from NTFA come this time may be timeout.

The patch does:
   - Limit the logger buffer
   - Provide the env variable that user can set the value of the limit
   - Return TRY_AGAIN error in case the limit of buffer is reached and write
all the
 notifications in the buffer to the log file. The current of
notification isn't
 written to log file.
---
 src/ntf/README|  10 
 src/ntf/ntfd/NtfAdmin.cc  |  44 +-
 src/ntf/ntfd/NtfLogger.cc | 149
+++---
 src/ntf/ntfd/NtfLogger.h  |  11 +++-
 src/ntf/ntfd/ntfd.conf|  10 
 src/ntf/ntfd/ntfs.h   |   2 +
 6 files changed, 147 insertions(+), 79 deletions(-)

diff --git a/src/ntf/README b/src/ntf/README
index 6dd5173e1..5bf670647 100644
--- a/src/ntf/README
+++ b/src/ntf/README
@@ -233,6 +233,16 @@ NTFSV_ENV_CACHE_SIZE
 The size of the notification cache in the NTF server processes running on
the Controller nodes.
 The default value is 1 notification.
 
+NTFSV_LOGGER_BUFFER_CAPACITY
+
+The logger buffer is used to store the notification if writing notification
+to log file fail. This variable is set for limit of logger buffer size in
+NTFD. If the logger buffer is full and NTFD receives new notification,
+the TRY_AGAIN  error is returned to user. The limit should be set with
relevant
+value to avoid congestion in NTFD. Because if this value is set too big and
+writing notification is fail for long time, NTF has to write a big number
of
+notifications whenever handling sending notification request and that will
delay
+to handle other requests come to NTFD. The value of variable is from 10 to
5000.
 
 for debug see DEBUG.
 
diff --git a/src/ntf/ntfd/NtfAdmin.cc b/src/ntf/ntfd/NtfAdmin.cc
index 2cb99457c..6c2d69b43 100644
--- a/src/ntf/ntfd/NtfAdmin.cc
+++ b/src/ntf/ntfd/NtfAdmin.cc
@@ -193,19 +193,32 @@ void NtfAdmin::processNotification(unsigned int
clientId,
   notificationId, notificationType,
   (unsigned int)notificationMap.size());
 
-  // log the notification. Callback from SAF log will confirm later.
-  logger.log(notification, activeController());
-
-  /* send notification to standby */
-  sendNotificationUpdate(clientId, notification->getNotInfo());
+  if ((logger.isLoggerBufferFull() == true) &&
+  (logger.isAlarmNotification(notification) == true)) {
+NtfClient *client = getClient(clientId);
+MDS_DEST dest = client->getMdsDest();
+LOG_WA("The logger buffer is full. Check if there is issue in
writing");
+if (activeController())
+  notfication_result_lib(SA_AIS_ERR_TRY_AGAIN, notificationId,
+ mdsCtxt, dest);
+  } else {
+/* send notification to standby */
+sendNotificationUpdate(clientId, notification->getNotInfo());
 
-  ClientMap::iterator pos;
-  for (pos = clientMap.begin(); pos != clientMap.end(); pos++) {
-NtfClient *client = pos->second;
-client->notificationReceived(clientId, notification, mdsCtxt);
+ClientMap::iterator pos;
+for (pos = clientMap.begin(); pos != clientMap.end(); pos++) {
+  NtfClient *client = pos->second;
+  client->notificationReceived(clientId, notification, mdsCtxt);
+}
   }
 
-  /* remove notification if sent to all subscribers and logged */
+  // Log the notification. Callback from SAF log will confirm later.
+  if (activeController())
+logger.log(notification);
+  // Add the notification to Reader list
+  logger.addNotificationToReaderList(notification);
+
+  // Remove the notification if it is sent to all subscribers and logged
   if (notification->isSubscriptionListEmpty() && notification->loggedOk())
{
 NotificationMap::iterator posNot;
 posNot = notificationMap.find(notificationId);
@@ -341,9 +354,9 @@ void NtfAdmin::notificationReceivedColdSync(

Re: [devel] [PATCH 2/2] ntf: Update TRY_AGAIN error in ntfsend tool [#2961]

2019-01-09 Thread Lennart Lund

Hi Canh,


Ack, see minor comment below [Lennart]


Thanks

Lennart



Från: Canh Van Truong 
Skickat: den 9 januari 2019 10:22
Till: Lennart Lund; Minh Hon Chau
Kopia: opensaf-devel@lists.sourceforge.net; Canh Van Truong
Ämne: [PATCH 2/2] ntf: Update TRY_AGAIN error in ntfsend tool [#2961]

ntfsend is blocked in case TRY_AGAIN error. It should do the retry in
limit time.
---
 src/ntf/tools/ntfclient.c |  75 ---
 src/ntf/tools/ntfsend.c   | 240 +++---
 2 files changed, 120 insertions(+), 195 deletions(-)

diff --git a/src/ntf/tools/ntfclient.c b/src/ntf/tools/ntfclient.c
index 472be4bc3..ce93bb164 100644
--- a/src/ntf/tools/ntfclient.c
+++ b/src/ntf/tools/ntfclient.c
@@ -21,10 +21,13 @@
 #include 
 #include 
 #include 
+#include "base/osaf_time.h"
 #include "ntf/tools/ntfclient.h"
 #include "ntf/tools/ntfconsumer.h"
 #include 

+const uint64_t kWaitTime = 10*1000;  // Wait for timeout is 10 seconds
+
 /* help functions for printouts */
 void ntfsvtools_exitIfFalse(const char *file, unsigned int line, int 
expression)
 {
@@ -1097,18 +1100,19 @@ SaAisErrorT ntftool_saNtfInitialize(SaNtfHandleT 
*ntfHandle,
 const SaNtfCallbacksT *ntfCallbacks,
 SaVersionT *version)
 {
-   int curRetryTime = 0;
 SaAisErrorT rc;
+   struct timespec timeout_time;
 SaVersionT ntf_version = *version;
+
+   osaf_set_millis_timeout(kWaitTime, _time);
 do {
 rc = saNtfInitialize(ntfHandle, ntfCallbacks, _version);
 if (rc == SA_AIS_ERR_TRY_AGAIN) {
-   sleep(gl_apiRetry);
-   curRetryTime += gl_apiRetry;
+   osaf_nanosleep();
 ntf_version = *version;
 } else
 break;
-   } while (curRetryTime < gl_apiTolerance);
+   } while (!osaf_is_timeout(_time));
[Lennart] Why not use a "normal" while loop here? Is easier to read. You can 
directly on the
first line see that this is a timed out loop. Applies to all the loops below

 return rc;
 }
@@ -1119,16 +1123,17 @@ SaAisErrorT ntftool_saNtfInitialize(SaNtfHandleT 
*ntfHandle,
 SaAisErrorT ntftool_saNtfDispatch(SaNtfHandleT ntfHandle,
   SaDispatchFlagsT dispatchFlags)
 {
-   int curRetryTime = 0;
 SaAisErrorT rc;
+   struct timespec timeout_time;
+
+   osaf_set_millis_timeout(kWaitTime, _time);
 do {
 rc = saNtfDispatch(ntfHandle, dispatchFlags);
 if (rc == SA_AIS_ERR_TRY_AGAIN) {
-   sleep(gl_apiRetry);
-   curRetryTime += gl_apiRetry;
+   osaf_nanosleep();
 } else
 break;
-   } while (curRetryTime < gl_apiTolerance);
+   } while (!osaf_is_timeout(_time));

 return rc;
 }
@@ -1139,16 +1144,17 @@ SaAisErrorT ntftool_saNtfDispatch(SaNtfHandleT 
ntfHandle,
 SaAisErrorT
 ntftool_saNtfNotificationSend(SaNtfNotificationHandleT notificationHandle)
 {
-   int curRetryTime = 0;
 SaAisErrorT rc;
+   struct timespec timeout_time;
+
+   osaf_set_millis_timeout(kWaitTime, _time);
 do {
 rc = saNtfNotificationSend(notificationHandle);
 if (rc == SA_AIS_ERR_TRY_AGAIN) {
-   sleep(gl_apiRetry);
-   curRetryTime += gl_apiRetry;
+   osaf_nanosleep();
 } else
 break;
-   } while (curRetryTime < gl_apiTolerance);
+   } while (!osaf_is_timeout(_time));

 return rc;
 }
@@ -1160,17 +1166,18 @@ SaAisErrorT ntftool_saNtfNotificationSubscribe(
 const SaNtfNotificationTypeFilterHandlesT *notificationFilterHandles,
 SaNtfSubscriptionIdT subscriptionId)
 {
-   int curRetryTime = 0;
 SaAisErrorT rc;
+   struct timespec timeout_time;
+
+   osaf_set_millis_timeout(kWaitTime, _time);
 do {
 rc = saNtfNotificationSubscribe(notificationFilterHandles,
 subscriptionId);
 if (rc == SA_AIS_ERR_TRY_AGAIN) {
-   sleep(gl_apiRetry);
-   curRetryTime += gl_apiRetry;
+   osaf_nanosleep();
 } else
 break;
-   } while (curRetryTime < gl_apiTolerance);
+   } while (!osaf_is_timeout(_time));

 return rc;
 }
@@ -1181,16 +1188,17 @@ SaAisErrorT ntftool_saNtfNotificationSubscribe(
 SaAisErrorT
 ntftool_saNtfNotificationUnsubscribe(SaNtfSubscriptionIdT subscriptionId)
 {
-   int curRetryTime = 0;
 SaAisErrorT rc;
+   struct timespec timeout_time;
+
+   osaf_set_millis_timeout(kWaitTime, _time);
 do {
 rc =

Re: [devel] [PATCH 1/2] ntf: Limit the logger buffer [#2961]

2019-01-09 Thread Lennart Lund

Hi Canh


Ack for "ntf: Limit the logger buffer" commit with some comments. See [Lennart] 
below


Thanks

Lennart



Från: Canh Van Truong 
Skickat: den 9 januari 2019 10:22
Till: Lennart Lund; Minh Hon Chau
Kopia: opensaf-devel@lists.sourceforge.net; Canh Van Truong
Ämne: [PATCH 1/2] ntf: Limit the logger buffer [#2961]

When writing the notificaion fail with TRY_AGAIN in callback, the notificaion 
is pushed
again to the list. If this happens for long time, the list is going to be very 
big.
This cause NTFD take time to process writing all the notification in the list 
and
the request from NTFA come this time may be timeout.

The patch does:
   - Limit the logger buffer
   - Provide the env variable that user can set the value of the limit
   - Return TRY_AGAIN error in case the limit of buffer is reached and write 
all the
 notifications in the buffer to the log file. The current of notification 
isn't
 written to log file.
---
 src/ntf/README|  10 
 src/ntf/ntfd/NtfAdmin.cc  |  44 +-
 src/ntf/ntfd/NtfLogger.cc | 149 +++---
 src/ntf/ntfd/NtfLogger.h  |  11 +++-
 src/ntf/ntfd/ntfd.conf|  10 
 src/ntf/ntfd/ntfs.h   |   2 +
 6 files changed, 147 insertions(+), 79 deletions(-)

diff --git a/src/ntf/README b/src/ntf/README
index 6dd5173e1..5bf670647 100644
--- a/src/ntf/README
+++ b/src/ntf/README
@@ -233,6 +233,16 @@ NTFSV_ENV_CACHE_SIZE
 The size of the notification cache in the NTF server processes running on the 
Controller nodes.
 The default value is 1 notification.

+NTFSV_LOGGER_BUFFER_CAPACITY
+
+The logger buffer is used to store the notification if writing notification
+to log file fail. This variable is set for limit of logger buffer size in
+NTFD. If the logger buffer is full and NTFD receives new notification,
+the TRY_AGAIN  error is returned to user. The limit should be set with relevant
+value to avoid congestion in NTFD. Because if this value is set too big and
+writing notification is fail for long time, NTF has to write a big number of
+notifications whenever handling sending notification request and that will 
delay
+to handle other requests come to NTFD. The value of variable is from 10 to 
5000.

 for debug see DEBUG.

diff --git a/src/ntf/ntfd/NtfAdmin.cc b/src/ntf/ntfd/NtfAdmin.cc
index 2cb99457c..6c2d69b43 100644
--- a/src/ntf/ntfd/NtfAdmin.cc
+++ b/src/ntf/ntfd/NtfAdmin.cc
@@ -193,19 +193,32 @@ void NtfAdmin::processNotification(unsigned int clientId,
   notificationId, notificationType,
   (unsigned int)notificationMap.size());

-  // log the notification. Callback from SAF log will confirm later.
-  logger.log(notification, activeController());
-
-  /* send notification to standby */
-  sendNotificationUpdate(clientId, notification->getNotInfo());
+  if ((logger.isLoggerBufferFull() == true) &&
+  (logger.isAlarmNotification(notification) == true)) {
+NtfClient *client = getClient(clientId);
+MDS_DEST dest = client->getMdsDest();
+LOG_WA("The logger buffer is full. Check if there is issue in writing");
+if (activeController())
+  notfication_result_lib(SA_AIS_ERR_TRY_AGAIN, notificationId,
+ mdsCtxt, dest);
+  } else {
+/* send notification to standby */
+sendNotificationUpdate(clientId, notification->getNotInfo());

-  ClientMap::iterator pos;
-  for (pos = clientMap.begin(); pos != clientMap.end(); pos++) {
-NtfClient *client = pos->second;
-client->notificationReceived(clientId, notification, mdsCtxt);
+ClientMap::iterator pos;
+for (pos = clientMap.begin(); pos != clientMap.end(); pos++) {
+  NtfClient *client = pos->second;
+  client->notificationReceived(clientId, notification, mdsCtxt);
+}
   }

-  /* remove notification if sent to all subscribers and logged */
+  // Log the notification. Callback from SAF log will confirm later.
+  if (activeController())
+logger.log(notification);
+  // Add the notification to Reader list
+  logger.addNotificationToReaderList(notification);
+
+  // Remove the notification if it is sent to all subscribers and logged
   if (notification->isSubscriptionListEmpty() && notification->loggedOk()) {
 NotificationMap::iterator posNot;
 posNot = notificationMap.find(notificationId);
@@ -341,9 +354,9 @@ void NtfAdmin::notificationReceivedColdSync(
   TRACE_LEAVE();
 }
 /**
- * A cached notification is received in Cold Sync.
- * This cached notification will be marked as logged, and stored
- * only in NtfLogger class to serve the reader.
+ * A cached notifications are received in Cold Sync.
+ * This cached notifications are stored in NtfLogger
+ * class to serve the reader.
  *
  * @param clientId Node-wide unique id for the client who sent the
  * notification.
@@ -363,8 +376,7 @@ void NtfAdmin::cachedNotificationReceivedColdSync(
   TRACE_2("cached notification %u received", (unsigned

[devel] [PATCH 2/2] ntf: Update TRY_AGAIN error in ntfsend tool [#2961]

2019-01-09 Thread Canh Van Truong

ntfsend is blocked in case TRY_AGAIN error. It should do the retry in
limit time.
---
 src/ntf/tools/ntfclient.c |  75 ---
 src/ntf/tools/ntfsend.c   | 240 +++---
 2 files changed, 120 insertions(+), 195 deletions(-)

diff --git a/src/ntf/tools/ntfclient.c b/src/ntf/tools/ntfclient.c
index 472be4bc3..ce93bb164 100644
--- a/src/ntf/tools/ntfclient.c
+++ b/src/ntf/tools/ntfclient.c
@@ -21,10 +21,13 @@
 #include 
 #include 
 #include 
+#include "base/osaf_time.h"
 #include "ntf/tools/ntfclient.h"
 #include "ntf/tools/ntfconsumer.h"
 #include 
 
+const uint64_t kWaitTime = 10*1000;  // Wait for timeout is 10 seconds
+
 /* help functions for printouts */
 void ntfsvtools_exitIfFalse(const char *file, unsigned int line, int 
expression)
 {
@@ -1097,18 +1100,19 @@ SaAisErrorT ntftool_saNtfInitialize(SaNtfHandleT 
*ntfHandle,
const SaNtfCallbacksT *ntfCallbacks,
SaVersionT *version)
 {
-   int curRetryTime = 0;
SaAisErrorT rc;
+   struct timespec timeout_time;
SaVersionT ntf_version = *version;
+
+   osaf_set_millis_timeout(kWaitTime, _time);
do {
rc = saNtfInitialize(ntfHandle, ntfCallbacks, _version);
if (rc == SA_AIS_ERR_TRY_AGAIN) {
-   sleep(gl_apiRetry);
-   curRetryTime += gl_apiRetry;
+   osaf_nanosleep();
ntf_version = *version;
} else
break;
-   } while (curRetryTime < gl_apiTolerance);
+   } while (!osaf_is_timeout(_time));
 
return rc;
 }
@@ -1119,16 +1123,17 @@ SaAisErrorT ntftool_saNtfInitialize(SaNtfHandleT 
*ntfHandle,
 SaAisErrorT ntftool_saNtfDispatch(SaNtfHandleT ntfHandle,
  SaDispatchFlagsT dispatchFlags)
 {
-   int curRetryTime = 0;
SaAisErrorT rc;
+   struct timespec timeout_time;
+
+   osaf_set_millis_timeout(kWaitTime, _time);
do {
rc = saNtfDispatch(ntfHandle, dispatchFlags);
if (rc == SA_AIS_ERR_TRY_AGAIN) {
-   sleep(gl_apiRetry);
-   curRetryTime += gl_apiRetry;
+   osaf_nanosleep();
} else
break;
-   } while (curRetryTime < gl_apiTolerance);
+   } while (!osaf_is_timeout(_time));
 
return rc;
 }
@@ -1139,16 +1144,17 @@ SaAisErrorT ntftool_saNtfDispatch(SaNtfHandleT 
ntfHandle,
 SaAisErrorT
 ntftool_saNtfNotificationSend(SaNtfNotificationHandleT notificationHandle)
 {
-   int curRetryTime = 0;
SaAisErrorT rc;
+   struct timespec timeout_time;
+
+   osaf_set_millis_timeout(kWaitTime, _time);
do {
rc = saNtfNotificationSend(notificationHandle);
if (rc == SA_AIS_ERR_TRY_AGAIN) {
-   sleep(gl_apiRetry);
-   curRetryTime += gl_apiRetry;
+   osaf_nanosleep();
} else
break;
-   } while (curRetryTime < gl_apiTolerance);
+   } while (!osaf_is_timeout(_time));
 
return rc;
 }
@@ -1160,17 +1166,18 @@ SaAisErrorT ntftool_saNtfNotificationSubscribe(
 const SaNtfNotificationTypeFilterHandlesT *notificationFilterHandles,
 SaNtfSubscriptionIdT subscriptionId)
 {
-   int curRetryTime = 0;
SaAisErrorT rc;
+   struct timespec timeout_time;
+
+   osaf_set_millis_timeout(kWaitTime, _time);
do {
rc = saNtfNotificationSubscribe(notificationFilterHandles,
subscriptionId);
if (rc == SA_AIS_ERR_TRY_AGAIN) {
-   sleep(gl_apiRetry);
-   curRetryTime += gl_apiRetry;
+   osaf_nanosleep();
} else
break;
-   } while (curRetryTime < gl_apiTolerance);
+   } while (!osaf_is_timeout(_time));
 
return rc;
 }
@@ -1181,16 +1188,17 @@ SaAisErrorT ntftool_saNtfNotificationSubscribe(
 SaAisErrorT
 ntftool_saNtfNotificationUnsubscribe(SaNtfSubscriptionIdT subscriptionId)
 {
-   int curRetryTime = 0;
SaAisErrorT rc;
+   struct timespec timeout_time;
+
+   osaf_set_millis_timeout(kWaitTime, _time);
do {
rc = saNtfNotificationUnsubscribe(subscriptionId);
if (rc == SA_AIS_ERR_TRY_AGAIN) {
-   sleep(gl_apiRetry);
-   curRetryTime += gl_apiRetry;
+   osaf_nanosleep();
} else
break;
-   } while (curRetryTime < gl_apiTolerance);
+   } while (!osaf_is_timeout(_time));
 
return rc;
 }
@@ -1203,17 +1211,18 @@ SaAisErrorT ntftool_saNtfNotificationReadInitialize(
 const SaNtfNotificationTypeFilterHandlesT *notificationFilterHandles,

[devel] [PATCH 0/2] Review Request for ntf: Limit the logger buffer [#2961] V2

2019-01-09 Thread Canh Van Truong

Summary: ntf: Limit the logger buffer [#2961]
Review request for Ticket(s): 2961
Peer Reviewer(s): Lennart, Minh
Pull request to: Minh
Affected branch(es): develop, release
Development branch: ticket-2961
Base revision: 8679707a45014fc2cb964cd931126a3607a03aab
Personal repository: git://git.code.sf.net/u/canht32/review


Impacted area   Impact y/n

 Docsn
 Build systemn
 RPM/packaging   n
 Configuration files n
 Startup scripts n
 SAF servicesy
 OpenSAF servicesn
 Core libraries  n
 Samples n
 Tests   n
 Other   n


Comments (indicate scope for each "y" above):
-
*** EXPLAIN/COMMENT THE PATCH SERIES HERE ***

revision 2d825d47b278c76fb010fec5380e99b2e6d5f875
Author: Canh Van Truong 
Date:   Wed, 9 Jan 2019 15:51:41 +0700

ntf: Update TRY_AGAIN error in ntfsend tool [#2961]

ntfsend is blocked in case TRY_AGAIN error. It should do the retry in
limit time.



revision 19ca9bcf42b68044dfabe1bde3a2a992c0e41d88
Author: Canh Van Truong 
Date:   Wed, 9 Jan 2019 15:51:25 +0700

ntf: Limit the logger buffer [#2961]

When writing the notificaion fail with TRY_AGAIN in callback, the notificaion 
is pushed
again to the list. If this happens for long time, the list is going to be very 
big.
This cause NTFD take time to process writing all the notification in the list 
and
the request from NTFA come this time may be timeout.

The patch does:
   - Limit the logger buffer
   - Provide the env variable that user can set the value of the limit
   - Return TRY_AGAIN error in case the limit of buffer is reached and write 
all the
 notifications in the buffer to the log file. The current of notification 
isn't
 written to log file.



Complete diffstat:
--
 src/ntf/README|  10 ++
 src/ntf/ntfd/NtfAdmin.cc  |  44 +
 src/ntf/ntfd/NtfLogger.cc | 149 
 src/ntf/ntfd/NtfLogger.h  |  11 ++-
 src/ntf/ntfd/ntfd.conf|  10 ++
 src/ntf/ntfd/ntfs.h   |   2 +
 src/ntf/tools/ntfclient.c |  75 ---
 src/ntf/tools/ntfsend.c   | 240 +++---
 8 files changed, 267 insertions(+), 274 deletions(-)


Testing Commands:
-
*** LIST THE COMMAND LINE TOOLS/STEPS TO TEST YOUR CHANGES ***


Testing, Expected Results:
--
*** PASTE COMMAND OUTPUTS / TEST RESULTS ***


Conditions of Submission:
-
*** HOW MANY DAYS BEFORE PUSHING, CONSENSUS ETC ***


Arch  Built StartedLinux distro
---
mipsn  n
mips64  n  n
x86 n  n
x86_64  n  n
powerpc n  n
powerpc64   n  n


Reviewer Checklist:
---
[Submitters: make sure that your review doesn't trigger any checkmarks!]


Your checkin has not passed review because (see checked entries):

___ Your RR template is generally incomplete; it has too many blank entries
that need proper data filled in.

___ You have failed to nominate the proper persons for review and push.

___ Your patches do not have proper short+long header

___ You have grammar/spelling in your header that is unacceptable.

___ You have exceeded a sensible line length in your headers/comments/text.

___ You have failed to put in a proper Trac Ticket # into your commits.

___ You have incorrectly put/left internal data in your comments/files
(i.e. internal bug tracking tool IDs, product names etc)

___ You have not given any evidence of testing beyond basic build tests.
Demonstrate some level of runtime or other sanity testing.

___ You have ^M present in some of your files. These have to be removed.

___ You have needlessly changed whitespace or added whitespace crimes
like trailing spaces, or spaces before tabs.

___ You have mixed real technical changes with whitespace and other
cosmetic code cleanup changes. These have to be separate commits.

___ You need to refactor your submission into logical chunks; there is
too much content into a single commit.

___ You have extraneous garbage in your review (merge commits etc)

___ You have giant attachments which should never have been sent;
Instead you should place your content in a public tree to be pulled.

___ You have too many commits attached to an e-mail; resend as threaded
commits, or place in a public tree for a pull.

___ You have resent this content multiple times without a clear indication
of what has changed between each re-send.

___ You have failed to adequately and individually address all of the
comments and change requests that were proposed in the initial review.

___ You have a misconfigured ~/.gitconfig file (i.e. user.name, user.email etc)

___ Your computer have a badly configured date

[devel] [PATCH 1/2] ntf: Limit the logger buffer [#2961]

2019-01-09 Thread Canh Van Truong

When writing the notificaion fail with TRY_AGAIN in callback, the notificaion 
is pushed
again to the list. If this happens for long time, the list is going to be very 
big.
This cause NTFD take time to process writing all the notification in the list 
and
the request from NTFA come this time may be timeout.

The patch does:
   - Limit the logger buffer
   - Provide the env variable that user can set the value of the limit
   - Return TRY_AGAIN error in case the limit of buffer is reached and write 
all the
 notifications in the buffer to the log file. The current of notification 
isn't
 written to log file.
---
 src/ntf/README|  10 
 src/ntf/ntfd/NtfAdmin.cc  |  44 +-
 src/ntf/ntfd/NtfLogger.cc | 149 +++---
 src/ntf/ntfd/NtfLogger.h  |  11 +++-
 src/ntf/ntfd/ntfd.conf|  10 
 src/ntf/ntfd/ntfs.h   |   2 +
 6 files changed, 147 insertions(+), 79 deletions(-)

diff --git a/src/ntf/README b/src/ntf/README
index 6dd5173e1..5bf670647 100644
--- a/src/ntf/README
+++ b/src/ntf/README
@@ -233,6 +233,16 @@ NTFSV_ENV_CACHE_SIZE
 The size of the notification cache in the NTF server processes running on the 
Controller nodes.
 The default value is 1 notification.
 
+NTFSV_LOGGER_BUFFER_CAPACITY
+
+The logger buffer is used to store the notification if writing notification
+to log file fail. This variable is set for limit of logger buffer size in
+NTFD. If the logger buffer is full and NTFD receives new notification,
+the TRY_AGAIN  error is returned to user. The limit should be set with relevant
+value to avoid congestion in NTFD. Because if this value is set too big and
+writing notification is fail for long time, NTF has to write a big number of
+notifications whenever handling sending notification request and that will 
delay
+to handle other requests come to NTFD. The value of variable is from 10 to 
5000.
 
 for debug see DEBUG.
 
diff --git a/src/ntf/ntfd/NtfAdmin.cc b/src/ntf/ntfd/NtfAdmin.cc
index 2cb99457c..6c2d69b43 100644
--- a/src/ntf/ntfd/NtfAdmin.cc
+++ b/src/ntf/ntfd/NtfAdmin.cc
@@ -193,19 +193,32 @@ void NtfAdmin::processNotification(unsigned int clientId,
   notificationId, notificationType,
   (unsigned int)notificationMap.size());
 
-  // log the notification. Callback from SAF log will confirm later.
-  logger.log(notification, activeController());
-
-  /* send notification to standby */
-  sendNotificationUpdate(clientId, notification->getNotInfo());
+  if ((logger.isLoggerBufferFull() == true) &&
+  (logger.isAlarmNotification(notification) == true)) {
+NtfClient *client = getClient(clientId);
+MDS_DEST dest = client->getMdsDest();
+LOG_WA("The logger buffer is full. Check if there is issue in writing");
+if (activeController())
+  notfication_result_lib(SA_AIS_ERR_TRY_AGAIN, notificationId,
+ mdsCtxt, dest);
+  } else {
+/* send notification to standby */
+sendNotificationUpdate(clientId, notification->getNotInfo());
 
-  ClientMap::iterator pos;
-  for (pos = clientMap.begin(); pos != clientMap.end(); pos++) {
-NtfClient *client = pos->second;
-client->notificationReceived(clientId, notification, mdsCtxt);
+ClientMap::iterator pos;
+for (pos = clientMap.begin(); pos != clientMap.end(); pos++) {
+  NtfClient *client = pos->second;
+  client->notificationReceived(clientId, notification, mdsCtxt);
+}
   }
 
-  /* remove notification if sent to all subscribers and logged */
+  // Log the notification. Callback from SAF log will confirm later.
+  if (activeController())
+logger.log(notification);
+  // Add the notification to Reader list
+  logger.addNotificationToReaderList(notification);
+
+  // Remove the notification if it is sent to all subscribers and logged
   if (notification->isSubscriptionListEmpty() && notification->loggedOk()) {
 NotificationMap::iterator posNot;
 posNot = notificationMap.find(notificationId);
@@ -341,9 +354,9 @@ void NtfAdmin::notificationReceivedColdSync(
   TRACE_LEAVE();
 }
 /**
- * A cached notification is received in Cold Sync.
- * This cached notification will be marked as logged, and stored
- * only in NtfLogger class to serve the reader.
+ * A cached notifications are received in Cold Sync.
+ * This cached notifications are stored in NtfLogger
+ * class to serve the reader.
  *
  * @param clientId Node-wide unique id for the client who sent the
  * notification.
@@ -363,8 +376,7 @@ void NtfAdmin::cachedNotificationReceivedColdSync(
   TRACE_2("cached notification %u received", (unsigned int)notificationId);
   NtfSmartPtr notification(new NtfNotification(notificationId,
   notificationType, sendNotInfo));
-  notification->notificationLoggedConfirmed();
-  logger.log(notification, false);
+  logger.addNotificationToReaderList(notification);
   TRACE_LEAVE();
 }
 
@@ -706,7 +718,7 @@ void NtfAdmin::checkNotificationList() {
 
 if

Re: [devel] [PATCH 1/1] imm: allow empty-value attribute with default-tag persisted [#2985]

Re: [devel] [PATCH 1/2] ntf: Limit the logger buffer [#2961]

[devel] FW: [PATCH 1/2] ntf: Limit the logger buffer [#2961]

Re: [devel] [PATCH 2/2] ntf: Update TRY_AGAIN error in ntfsend tool [#2961]

Re: [devel] [PATCH 1/2] ntf: Limit the logger buffer [#2961]

[devel] [PATCH 2/2] ntf: Update TRY_AGAIN error in ntfsend tool [#2961]

[devel] [PATCH 0/2] Review Request for ntf: Limit the logger buffer [#2961] V2

[devel] [PATCH 1/2] ntf: Limit the logger buffer [#2961]

8 matches

Site Navigation

Mail list logo

Footer information