Summary: log: improve the resilience of log service [#3116]
Review request for Ticket(s): 3116
Peer Reviewer(s): Lennart, Gary, Minh
Pull request to: *** LIST THE PERSON WITH PUSH ACCESS HERE ***
Affected branch(es): develop
Development branch: ticket-3116
Base revision: 8e07c19aed63c249f4e7fa8470270d2de1a56046
Personal repository: git://git.code.sf.net/u/winhvu/review

--------------------------------
Impacted area       Impact y/n
--------------------------------
 Docs                    n
 Build system            n
 RPM/packaging           n
 Configuration files     n
 Startup scripts         n
 SAF services            y
 OpenSAF services        n
 Core libraries          n
 Samples                 n
 Tests                   n
 Other                   n

NOTE: Patch(es) contain lines longer than 80 characers

Comments (indicate scope for each "y" above):
---------------------------------------------
*** EXPLAIN/COMMENT THE PATCH SERIES HERE ***

revision 2643f8f829bb7ba638193df19cc6f20f86acb497
Author: Vu Minh Nguyen <vu.m.ngu...@dektech.com.au>
Date:   Thu, 28 Nov 2019 15:19:58 +0700

log: add test cases of improving the log resilience [#3116]

Adding 08 new test cases into 02 suites:
1) Suite 20 with 07 test cases, including:
- Test changing queue size & resilient timeout;
- Test if a write async is dropped if its timeout setting is overdue,
also verify if log server has kept the request in proper time.
- Test if getting write callback right away if the cache is full.
- Test if the cache is fully and correctly synced with standby.

2) Suite 21 with one test case:
Test if LOG agent notifies all lost invocation to log client.

As the suite 21 requires manual interaction, it is put into
'extended' tests. Only run with option '-e'.



revision 2a714a95f9f714684840047ae1e02d7c11bca32f
Author: Vu Minh Nguyen <vu.m.ngu...@dektech.com.au>
Date:   Thu, 28 Nov 2019 15:19:58 +0700

log: update README file for improvement of log resilience [#3116]



revision a103db0f56db32a1abe6f43eda7f06d48b54d469
Author: Vu Minh Nguyen <vu.m.ngu...@dektech.com.au>
Date:   Thu, 28 Nov 2019 15:19:58 +0700

saflogger: make timeout waiting for getting acknowledgment configurable [#3116]

Introducing a new option `-t second` or `--timeout=second` to let user input
his desired timeout of waiting for write async acknowledgment.

Default timeout is 20 seconds to keep saflogger backward compatible.



revision 54d46abd5c433ee884294ada6b277b04042f444f
Author: Vu Minh Nguyen <vu.m.ngu...@dektech.com.au>
Date:   Thu, 28 Nov 2019 15:19:58 +0700

log: notify all lost log records when cluster goes to headless [#3116]

This change introduces a light list keeping all invocations that not yet
get the acknowledgement from log server. If the server is disappeared
in case of headless, log agent will notify all lost invocations to log client
with error code SA_AIS_ERR_TRY_AGAIN.



revision 6260044621a2ecfad70bd5eeece3e811dc032188
Author: Vu Minh Nguyen <vu.m.ngu...@dektech.com.au>
Date:   Thu, 28 Nov 2019 15:19:58 +0700

log: improve the resilience of log service [#3116]

In order to improve resilience of OpenSAF LOG service when underlying
file system is unresponsive, a queue is introduced to hold async
write request up to an configurable time that is around 15 - 30 seconds.

The readiness of the I/O thread will periodically check, and if it turns
to ready state, the front element will go first. Returns SA_AIS_ERR_TRY_AGAIN
to client if the element stays in the queue longer than the setting time.

The queue capacity and the resilient time are configurable via the attributes:
`logMaxPendingWriteRequests` and `logResilienceTimeout`.

In default, this feature is disabled to keep log server backward compatible.



Added Files:
------------
 src/log/apitest/tet_saLogWriteLogAsync_cache.c
 src/log/logd/lgs_cache.cc
 src/log/logd/lgs_cache.h
 src/log/logd/lgs_mbcsv_cache.cc
 src/log/logd/lgs_mbcsv_cache.h


Complete diffstat:
------------------
 src/log/Makefile.am                            |  24 +-
 src/log/README                                 |  38 ++
 src/log/agent/lga_agent.cc                     |   2 +
 src/log/agent/lga_client.cc                    |   8 +-
 src/log/agent/lga_client.h                     |  34 ++
 src/log/apitest/logtest.c                      |   7 +
 src/log/apitest/logtest.h                      |   7 +-
 src/log/apitest/logutil.c                      |  14 +-
 src/log/apitest/tet_log_runtime_cfgobj.c       |   2 +-
 src/log/apitest/tet_saLogWriteLogAsync_cache.c | 648 +++++++++++++++++++++++++
 src/log/config/logsv_classes.xml               |  43 +-
 src/log/logd/lgs_cache.cc                      | 469 ++++++++++++++++++
 src/log/logd/lgs_cache.h                       | 287 +++++++++++
 src/log/logd/lgs_config.cc                     |  78 ++-
 src/log/logd/lgs_config.h                      |  10 +-
 src/log/logd/lgs_evt.cc                        | 161 ++----
 src/log/logd/lgs_evt.h                         |  10 +
 src/log/logd/lgs_file.cc                       |   8 +-
 src/log/logd/lgs_filehdl.cc                    |  58 +--
 src/log/logd/lgs_imm.cc                        |  40 +-
 src/log/logd/lgs_main.cc                       |  24 +-
 src/log/logd/lgs_mbcsv.cc                      | 447 +++++++++++++----
 src/log/logd/lgs_mbcsv.h                       |  19 +-
 src/log/logd/lgs_mbcsv_cache.cc                | 372 ++++++++++++++
 src/log/logd/lgs_mbcsv_cache.h                 | 110 +++++
 src/log/logd/lgs_mbcsv_v1.cc                   |   1 +
 src/log/logd/lgs_mbcsv_v2.cc                   |   2 +
 src/log/tools/saf_logger.c                     |  29 +-
 28 files changed, 2659 insertions(+), 293 deletions(-)


Testing Commands:
-----------------
- Run test suite #20
- Run 'extended' test suite #21 (require manually testing)

Testing, Expected Results:
--------------------------
All tests are passed


Conditions of Submission:
-------------------------
Ack from reviewers


Arch      Built     Started    Linux distro
-------------------------------------------
mips        n          n
mips64      n          n
x86         n          n
x86_64      n          n
powerpc     n          n
powerpc64   n          n


Reviewer Checklist:
-------------------
[Submitters: make sure that your review doesn't trigger any checkmarks!]


Your checkin has not passed review because (see checked entries):

___ Your RR template is generally incomplete; it has too many blank entries
    that need proper data filled in.

___ You have failed to nominate the proper persons for review and push.

___ Your patches do not have proper short+long header

___ You have grammar/spelling in your header that is unacceptable.

___ You have exceeded a sensible line length in your headers/comments/text.

___ You have failed to put in a proper Trac Ticket # into your commits.

___ You have incorrectly put/left internal data in your comments/files
    (i.e. internal bug tracking tool IDs, product names etc)

___ You have not given any evidence of testing beyond basic build tests.
    Demonstrate some level of runtime or other sanity testing.

___ You have ^M present in some of your files. These have to be removed.

___ You have needlessly changed whitespace or added whitespace crimes
    like trailing spaces, or spaces before tabs.

___ You have mixed real technical changes with whitespace and other
    cosmetic code cleanup changes. These have to be separate commits.

___ You need to refactor your submission into logical chunks; there is
    too much content into a single commit.

___ You have extraneous garbage in your review (merge commits etc)

___ You have giant attachments which should never have been sent;
    Instead you should place your content in a public tree to be pulled.

___ You have too many commits attached to an e-mail; resend as threaded
    commits, or place in a public tree for a pull.

___ You have resent this content multiple times without a clear indication
    of what has changed between each re-send.

___ You have failed to adequately and individually address all of the
    comments and change requests that were proposed in the initial review.

___ You have a misconfigured ~/.gitconfig file (i.e. user.name, user.email etc)

___ Your computer have a badly configured date and time; confusing the
    the threaded patch review.

___ Your changes affect IPC mechanism, and you don't present any results
    for in-service upgradability test.

___ Your changes affect user manual and documentation, your patch series
    do not contain the patch that updates the Doxygen manual.



_______________________________________________
Opensaf-devel mailing list
Opensaf-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/opensaf-devel

Reply via email to