Hi, A. I have got a comment that if writing a log record fails because of a timeout of the write request (write operation in file thread don't return and therefore the file thread is timed out) the answer (in the callback) shall be SA_AIS_ERR_TIMEOUT instead of SA_AIS_ERR_TRY_AGAIN. In all other error cases it shall be SA_AIS_ERR_TRY_AGAIN as in current implementation. If a client request to write a log record while the file thread is already hanging the answer shall be SA_AIS_ERR_TRY_AGAIN since no attempt to write is done in this case.
I think this would be correct behavior. In the first case an attempt to write is made and the log record may or may not be written depending on how/if the file system recovers. In the second case no attempt to write is done and we know that the log record never will be written. I intend to change the implementation according to this if there are no objections. B. Can file thread hanging because of a file operation e.g. write that does not return be permanent? I suspect there are situations where the file system can be "fixed" in such a way that this may happen. This means that some sort of recovery mechanism must be implemented. One way of doing this is to restart the file thread if it's still hanging after a significant time (what is a significant time? 10 min?). Note that after a restart the file thread will be in its initial state which means that the callback answer will be SA_AIS_ERR_TIMEOUT if the file system still is not working (if A. is implemented). Any better ideas? BR Lennart > -----Original Message----- > From: Lennart Lund [mailto:[email protected]] > Sent: den 9 juli 2013 14:49 > To: [email protected] > Cc: [email protected] > Subject: [devel] [PATCH 0 of 1] logsv: Fix hanging main thread when file i/o > don't return. [#9] > > Summary: logsv: Fix hanging main thread when file i/o don't return Review > request for Trac Ticket(s): #9 Peer Reviewer(s): Madhurika Koppula, (Anders > Widell, Hans Feldt) Pull request to: NA Affected branch(es): devel (4.4) > Development branch: <<IF ANY GIVE THE REPO URL>> > > > -------------------------------- > Impacted area Impact y/n > -------------------------------- > Docs n > Build system n > RPM/packaging n > Configuration files n > Startup scripts n > SAF services y > OpenSAF services n > Core libraries n > Samples n > Tests n > Other n > > > Comments (indicate scope for each "y" above): > --------------------------------------------- > In order to protect the log server "main thread" (MT) from hanging if a file > operation like write, mkdir etc. does not return, all such operations are done > in a separate "file thread" (FT). > Functions running in the "Main Thread" (MT) that needs file system > operations handle over the execution to the FT when file handling has to be > done. Execution is then given back to the MT again. If a file operation does > not return FT will hang but MT will time out the FT and resume. A timeout will > be handled as a file operation fail. > The MT can detect if the FT is hanging and new requests for file operations > will be "failed". > > Note: LEAP mailbox is not used for thread synchronization. Separate mail > discussion ongoing. > > changeset 63709a6e3e9862642e1d569065114d04e41bce2b > Author: Lennart Lund <[email protected]> > Date: Fri, 05 Jul 2013 14:48:41 +0200 > > logsv: Fix hanging main thread when file i/o don't return. > > - Fix review remarks and some findings from test > - Fix some findings found when using code analyze tool > - Cleanup of TRACE and LOG > - Add information for contributors/maintainers about file system > handling in > the Log-service README file > > > Removed Files: > -------------- > README_LOGENH > > > Complete diffstat: > ------------------ > README_LOGENH | 42 ------------------ > osaf/services/saf/logsv/README | 23 +++++++++ > osaf/services/saf/logsv/config/logd.conf | 2 +- > osaf/services/saf/logsv/lgs/lgs_evt.c | 6 +-- > osaf/services/saf/logsv/lgs/lgs_file.c | 56 +++++++---------------- > osaf/services/saf/logsv/lgs/lgs_filehdl.c | 190 > +++++++++++++++++++++++++++++++++++++++++++++++++--------------- > ----------------- > osaf/services/saf/logsv/lgs/lgs_imm.c | 113 > +++++++++++++++++++++++++++++++---------------- > osaf/services/saf/logsv/lgs/lgs_mbcsv.c | 24 +--------- > osaf/services/saf/logsv/lgs/lgs_mbcsv.h | 6 +- > osaf/services/saf/logsv/lgs/lgs_stream.c | 132 > ++++++++++++++++++++++++++++++-------------------------- > osaf/services/saf/logsv/lgs/lgs_stream.h | 4 +- > osaf/services/saf/logsv/lgs/lgs_util.c | 75 > ++++++++++++++++++----------- > -- > osaf/services/saf/logsv/lgs/lgs_util.h | 2 +- > tests/logsv/tet_LogOiOps.c | 6 ++ > 14 files changed, 357 insertions(+), 324 deletions(-) > > > Testing Commands: > ----------------- > 1. Regession test > > logtest > 2. Switch over test (using alarm stream) > > saflogger -l -s crit "alarm message 1" > > cat repl_opensaf/saflog/saLogAlarm_SOME_DATE.log > Printout containing "alarm message 1" > > immadm -o 7 safSi=SC-2N,safApp=OpenSAF saflogger -l -s crit "alarm > > message 2" > > cat repl_opensaf/saflog/saLogAlarm_SOME_DATE.log > Printout contaning "alarm message 1" and "alarm message 2" > 3. Redo tests after node start with simulated > unavailable filesystem for the log service > - Activate simulated unavailable file system by uncommenting > the LLD_DELAY_TST define in file lgs_file.c in the log server. > This means to "hang" the "file thread" for some tme during system > start. > - Rebuild the log server. > - Remove old log files in repl-opensaf/saflog/ > - Start the cluster with the rebuilt log server. > Note: The repl_opensaf/saflog directory is empty after system > start. The .cfg and .log files for alarm, notoify and system > that normally can be found is missing since they could not be > created during system start. However files for respective log > stream will be created when writing log records. > - Re-run test 1 and 2 > > > Testing, Expected Results: > -------------------------- > 1. Regression test with no fail. > 2. "alarm message 1" and "alarm message 1" found in the same file. > 3.1. Regression test with no fail. > 3.2. "alarm message 1" and "alarm message 1" found in the same file. > > > Conditions of Submission: > ------------------------- > Ack from reviewers. > > > Arch Built Started Linux distro > ------------------------------------------- > mips n n > mips64 n n > x86 n n > x86_64 n n > powerpc n n > powerpc64 n n > > > Reviewer Checklist: > ------------------- > [Submitters: make sure that your review doesn't trigger any checkmarks!] > > > Your checkin has not passed review because (see checked entries): > > ___ Your RR template is generally incomplete; it has too many blank entries > that need proper data filled in. > > ___ You have failed to nominate the proper persons for review and push. > > ___ Your patches do not have proper short+long header > > ___ You have grammar/spelling in your header that is unacceptable. > > ___ You have exceeded a sensible line length in your > headers/comments/text. > > ___ You have failed to put in a proper Trac Ticket # into your commits. > > ___ You have incorrectly put/left internal data in your comments/files > (i.e. internal bug tracking tool IDs, product names etc) > > ___ You have not given any evidence of testing beyond basic build tests. > Demonstrate some level of runtime or other sanity testing. > > ___ You have ^M present in some of your files. These have to be removed. > > ___ You have needlessly changed whitespace or added whitespace crimes > like trailing spaces, or spaces before tabs. > > ___ You have mixed real technical changes with whitespace and other > cosmetic code cleanup changes. These have to be separate commits. > > ___ You need to refactor your submission into logical chunks; there is > too much content into a single commit. > > ___ You have extraneous garbage in your review (merge commits etc) > > ___ You have giant attachments which should never have been sent; > Instead you should place your content in a public tree to be pulled. > > ___ You have too many commits attached to an e-mail; resend as threaded > commits, or place in a public tree for a pull. > > ___ You have resent this content multiple times without a clear indication > of what has changed between each re-send. > > ___ You have failed to adequately and individually address all of the > comments and change requests that were proposed in the initial review. > > ___ You have a misconfigured ~/.hgrc file (i.e. username, email etc) > > ___ Your computer have a badly configured date and time; confusing the > the threaded patch review. > > ___ Your changes affect IPC mechanism, and you don't present any results > for in-service upgradability test. > > ___ Your changes affect user manual and documentation, your patch series > do not contain the patch that updates the Doxygen manual. > > > ------------------------------------------------------------------------------ > See everything from the browser to the database with AppDynamics Get > end-to-end visibility with application monitoring from AppDynamics Isolate > bottlenecks and diagnose root cause in seconds. > Start your free trial of AppDynamics Pro today! > http://pubads.g.doubleclick.net/gampad/clk?id=48808831&iu=/4140/ostg.clk > trk > _______________________________________________ > Opensaf-devel mailing list > [email protected] > https://lists.sourceforge.net/lists/listinfo/opensaf-devel ------------------------------------------------------------------------------ See everything from the browser to the database with AppDynamics Get end-to-end visibility with application monitoring from AppDynamics Isolate bottlenecks and diagnose root cause in seconds. Start your free trial of AppDynamics Pro today! http://pubads.g.doubleclick.net/gampad/clk?id=48808831&iu=/4140/ostg.clktrk _______________________________________________ Opensaf-devel mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/opensaf-devel
