We see this in the traces leading up to the sqlite integrity violation:

CCB 45 creates an object with the empty DN (or possibly non printable DN):
Aug 21 15:39:56.186353 osafimmpbed [13438:immpbe_daemon.cc:1382] TR Begin PBE 
transaction for CCB 45 OK
Aug 21 15:39:56.186433 osafimmpbed [13438:immpbe_daemon.cc:0105] TR Create of 
object with DN: 

Then CCB 46 does the same and gets rejected by sqlite integrity violation
because of duplicate (empty or unprintable) on primary key:
Aug 21 15:39:56.350387 osafimmpbed [13438:immpbe_daemon.cc:1382] TR Begin PBE 
transaction for CCB 46 OK
Aug 21 15:39:56.350417 osafimmpbed [13438:immpbe_daemon.cc:0105] TR Create of 
object with DN: 
Aug 21 15:39:56.350525 osafimmpbed [13438:immpbe_dump.cc:2311] ER SQL 
statement('INSERT INTO objects (obj_id, class_id, dn, last_ccb) VALUES (?, ?, 
?, ?)') failed with error code: 19

Of course an empty or unprintable DN is not allowed by immsv and in fact
earlier when the create callback for ccb 45 reaches PBE it extrracts a 
correct DN:
Aug 21 15:39:55.950013 osafimmpbed [13438:immpbe_daemon.cc:1572] TR Extracted 
DN: attrName_Object4(16)

And for ccb 46:
Aug 21 15:39:55.971075 osafimmpbed [13438:immpbe_daemon.cc:1572] TR Extracted 
DN: attrName_Object5(16)


So somehow the heap gets corrupted in the PBE....

But this seems "easy" to reproduce so we should be able to find
what causes the corruption soon.



---

** [tickets:#1001] IMMSv is not honoring further CCB commits once SQL operation 
failure is reported by PBE**

**Status:** accepted
**Milestone:** 4.5.0
**Created:** Thu Aug 21, 2014 05:38 AM UTC by Sirisha Alla
**Last Updated:** Thu Aug 21, 2014 10:52 AM UTC
**Owner:** Anders Bjornerstedt

This issue is seen on SLES 64bit 4 node VMs. The changeset of OpenSAF used is 
5608 along with patches for #938,#994 and #997.

IMM applications are in progress when the issue is observed. During one of the 
CCB commit operations following is observed on the active controller:

Aug 21 10:15:44 SLES-64BIT-SLOT1 osafimmpbed: ER SQL statement('INSERT INTO 
objects (obj_id, class_id, dn, last_ccb) VALUES (?, ?, ?, ?)') failed with 
error code: 19
Aug 21 10:15:44 SLES-64BIT-SLOT1 osafimmnd[3645]: NO Invalid error reported 
implementer 'OpenSafImmPBE', Ccb 46 will be aborted
Aug 21 10:15:44 SLES-64BIT-SLOT1 osafimmnd[3645]: NO Ccb 46 ABORTED (exowner)
Aug 21 10:15:44 SLES-64BIT-SLOT1 osafimmpbed: WA Failed to find CCB object for 
2e/46

Once this error is seen further CCB commits(ccbApply()) fail with 
FAILED_OPERATION. Cluster becomes unusable for CCB commit operations.

Aug 21 10:15:50 SLES-64BIT-SLOT1 osafimmpbed: ER Failed to bind obj_id with 
error code: 21
Aug 21 10:15:50 SLES-64BIT-SLOT1 osafimmnd[3645]: NO Invalid error reported 
implementer 'OpenSafImmPBE', Ccb 47 will be aborted
Aug 21 10:15:50 SLES-64BIT-SLOT1 osafimmnd[3645]: NO Ccb 47 ABORTED (adminOwner)
Aug 21 10:15:50 SLES-64BIT-SLOT1 osafimmpbed: WA Failed to find CCB object for 
2f/47
Aug 21 10:15:52 SLES-64BIT-SLOT1 osafimmpbed: IN Create of class 
noDanglingPreconfigurationClass committing with ccbId:100000089
Aug 21 10:15:53 SLES-64BIT-SLOT1 osafimmnd[3645]: NO Create of class 
noDanglingPreconfigurationClass is PERSISTENT.
Aug 21 10:15:53 SLES-64BIT-SLOT1 osafimmpbed: ER Failed to bind obj_id with 
error code: 21
Aug 21 10:15:53 SLES-64BIT-SLOT1 osafimmnd[3645]: NO Invalid error reported 
implementer 'OpenSafImmPBE', Ccb 92 will be aborted
Aug 21 10:15:53 SLES-64BIT-SLOT1 osafimmnd[3645]: NO Ccb 92 ABORTED 
(noDanglingPreconfigurationClass)
Aug 21 10:15:53 SLES-64BIT-SLOT1 osafimmpbed: WA Failed to find CCB object for 
5c/92
Aug 21 10:15:53 SLES-64BIT-SLOT1 osafimmpbed: IN Delete of class 
noDanglingPreconfigurationClass committing with ccbId:10000008a
Aug 21 10:15:53 SLES-64BIT-SLOT1 osafimmnd[3645]: NO Delete of class 
noDanglingPreconfigurationClass is PERSISTENT.
Aug 21 10:15:53 SLES-64BIT-SLOT1 osafimmpbed: IN Create of class 
immsv_N0_Dangling_test01 committing with ccbId:10000008b
Aug 21 10:15:53 SLES-64BIT-SLOT1 osafimmnd[3645]: NO Create of class 
immsv_N0_Dangling_test01 is PERSISTENT.
Aug 21 10:15:53 SLES-64BIT-SLOT1 osafimmpbed: ER Failed to bind obj_id with 
error code: 21
Aug 21 10:15:53 SLES-64BIT-SLOT1 osafimmnd[3645]: NO Invalid error reported 
implementer 'OpenSafImmPBE', Ccb 98 will be aborted
Aug 21 10:15:53 SLES-64BIT-SLOT1 osafimmnd[3645]: NO Ccb 98 ABORTED 
(immsv_N0_Dangling_test01)
Aug 21 10:15:53 SLES-64BIT-SLOT1 osafimmpbed: WA Failed to find CCB object for 
62/98

This issue is reproducible. Traces are attached. IMMD traces are available and 
will be able to share them on request. I am not sure whether this is 
consequence after #1000 is observed. I will try to see if this can be 
reproduced independantly.


---

Sent from sourceforge.net because opensaf-tickets@lists.sourceforge.net is 
subscribed to https://sourceforge.net/p/opensaf/tickets/

To unsubscribe from further messages, a project admin can change settings at 
https://sourceforge.net/p/opensaf/admin/tickets/options.  Or, if this is a 
mailing list, you can unsubscribe from the mailing list.
------------------------------------------------------------------------------
Slashdot TV.  
Video for Nerds.  Stuff that matters.
http://tv.slashdot.org/
_______________________________________________
Opensaf-tickets mailing list
Opensaf-tickets@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/opensaf-tickets

Reply via email to