|
Hi Oliver, here is the stack trace, it happened again :( The interesting thing is that when i artificially slow down our throughoutput, it never happens. When i set it loose (with highest throughoutput) it usually happens. This stack trace is from i386 Linux 2.4, but we noticed the same on Sun Solaris HW. Btw, we made the needed changes to LockManager (to prevent lock accumulation is lock manager, a memory leak), coming a patch for review soon. The log output is a bit messy, it seems that when FRM declares DIRTY for database, and all my threads are dying in row.... it is ok. But why ERR_SYSTEM? Remember, that commit occures BEFORE the filename is sent over Queue, and every Stage is working in it's own workspace (eg. FRM is set to storeDir=/ips/store, workDir=/ips/work), and Stage1 works with files ONLY in "/Stage1", Stage2 in "/Stage2" accessed thru FRM... so, it is theoretically IMPOSSIBLE for two different stage/thread to work on the SAME file. Just to remeber: Stage1 reads using FRM from the inputWorkarea (eg. /stage1), process the file and spits it out to outputWorkarea (/stage2), the Stage2 gets the filename over Queue and reads the file from it's inputarea (/stage2) and writes to outputarea (/stage3) and so on... Real stages are: CRBExploder, InputReader, CDRProcessor, CRBPacker 2005-11-30 05:20:36.693 INFO [CRBPacker:Stage CRBPacker] - hu.siemens.tmd.CRBPackerTask: Tariff Switch CDR written into CRB file ipms_node1_20051130_042036_00001.cdr 2005-11-30 05:20:36.693 INFO [CRBPacker:Stage CRBPacker] - hu.siemens.tmd.CRBPackerTask: Tariff Switch CDR written into CRB file ipms_node1_20051130_042036_00001.cdr 2005-11-30 05:20:36.696 INFO [CRBPacker:Stage CRBPacker] - hu.siemens.tmd.CRBPackerTask: Wrote out a new CRB file: ipms_node1_20051130_042036_00001.cdr 2005-11-30 05:20:36.825 ERROR [InputReader:Stage InputReader] - hu.siemens.tmk.transaction.fs.FSTransactionalResource: Fatal error during critical commit/rollback of transaction 1133324406295_129327, setting database to dirty. org.apache.commons.transaction.file.ResourceManagerSystemException: 1133324406295_129327: Commit failed (ERR_SYSTEM) Caused by: java.io.FileNotFoundException: /users/barath/ips-test-1.2.14/consolidator/txfs/store/inputreader/20051115122953_87327151000_mdqs_20051115_122957_00000.cdr_0083_ber.cdr (No such file or directory) at java.io.FileOutputStream.open(Native Method) at java.io.FileOutputStream.<init>(FileOutputStream.java:179) at java.io.FileOutputStream.<init>(FileOutputStream.java:131) at org.apache.commons.transaction.util.FileHelper.copy(FileHelper.java:235) at org.apache.commons.transaction.util.FileHelper.moveRec(FileHelper.java:130) at org.apache.commons.transaction.util.FileHelper.moveRec(FileHelper.java:111) at org.apache.commons.transaction.util.FileHelper.moveRec(FileHelper.java:135) at org.apache.commons.transaction.util.FileHelper.moveRec(FileHelper.java:111) at org.apache.commons.transaction.file.FileResourceManager$TransactionContext.commit(FileResourceManager.java:1366) at org.apache.commons.transaction.file.FileResourceManager.commitTransaction(FileResourceManager.java:648) at hu.siemens.tmk.transaction.fs.FSManager.doCommit(FSManager.java:128) at hu.siemens.tmk.transaction.AbstractTransactionManager.commitOrRollback(AbstractTransactionManager.java:99) at hu.siemens.tmk.transaction.AbstractTransactionManager.commit(AbstractTransactionManager.java:79) at hu.siemens.tmk.AbstractBridgingTransactionalTask.commitTransaction(AbstractBridgingTransactionalTask.java:125) at hu.siemens.tmd.AbstractBridgingTMDTask.doRunBridged(AbstractBridgingTMDTask.java:105) at hu.siemens.tmk.AbstractBridgingTransactionalTask.doRun(AbstractBridgingTransactionalTask.java:75) at hu.siemens.tmk.AbstractTask.run(AbstractTask.java:47) at hu.siemens.tmk.InterconnectedSTStage.doTheWork(InterconnectedSTStage.java:40) at hu.siemens.tmk.AbstractStage.run(AbstractStage.java:80) at java.lang.Thread.run(Thread.java:595) at org.apache.commons.transaction.file.FileResourceManager$TransactionContext.commit(FileResourceManager.java:1368) at org.apache.commons.transaction.file.FileResourceManager.commitTransaction(FileResourceManager.java:648) at hu.siemens.tmk.transaction.fs.FSManager.doCommit(FSManager.java:128) at hu.siemens.tmk.transaction.AbstractTransactionManager.commitOrRollback(AbstractTransactionManager.java:99) at hu.siemens.tmk.transaction.AbstractTransactionManager.commit(AbstractTransactionManager.java:79) at hu.siemens.tmk.AbstractBridgingTransactionalTask.commitTransaction(AbstractBridgingTransactionalTask.java:125) at hu.siemens.tmd.AbstractBridgingTMDTask.doRunBridged(AbstractBridgingTMDTask.java:105) at hu.siemens.tmk.AbstractBridgingTransactionalTask.doRun(AbstractBridgingTransactionalTask.java:75) at hu.siemens.tmk.AbstractTask.run(AbstractTask.java:47) at hu.siemens.tmk.InterconnectedSTStage.doTheWork(InterconnectedSTStage.java:40) at hu.siemens.tmk.AbstractStage.run(AbstractStage.java:80) at java.lang.Thread.run(Thread.java:595) 2005-11-30 05:20:36.830 ERROR [InputReader:Stage InputReader] - hu.siemens.tmk.transaction.fs.FSManager: Exception during finishing transaction FSTx:1133324406295_129327 hu.siemens.tmk.transaction.TransactionException: FileResourceManager threw exception during commit. at hu.siemens.tmk.transaction.fs.FSManager.doCommit(FSManager.java:130) at hu.siemens.tmk.transaction.AbstractTransactionManager.commitOrRollback(AbstractTransactionManager.java:99) at hu.siemens.tmk.transaction.AbstractTransactionManager.commit(AbstractTransactionManager.java:79) at hu.siemens.tmk.AbstractBridgingTransactionalTask.commitTransaction(AbstractBridgingTransactionalTask.java:125) at hu.siemens.tmd.AbstractBridgingTMDTask.doRunBridged(AbstractBridgingTMDTask.java:105) at hu.siemens.tmk.AbstractBridgingTransactionalTask.doRun(AbstractBridgingTransactionalTask.java:75) at hu.siemens.tmk.AbstractTask.run(AbstractTask.java:47) at hu.siemens.tmk.InterconnectedSTStage.doTheWork(InterconnectedSTStage.java:40) at hu.siemens.tmk.AbstractStage.run(AbstractStage.java:80) at java.lang.Thread.run(Thread.java:595) Caused by: org.apache.commons.transaction.file.ResourceManagerSystemException: 1133324406295_129327: Commit failed (ERR_SYSTEM) Caused by: java.io.FileNotFoundException: /users/barath/ips-test-1.2.14/consolidator/txfs/store/inputreader/20051115122953_87327151000_mdqs_20051115_122957_00000.cdr_0083_ber.cdr (No such file or directory) at java.io.FileOutputStream.open(Native Method) at java.io.FileOutputStream.<init>(FileOutputStream.java:179) at java.io.FileOutputStream.<init>(FileOutputStream.java:131) at org.apache.commons.transaction.util.FileHelper.copy(FileHelper.java:235) at org.apache.commons.transaction.util.FileHelper.moveRec(FileHelper.java:130) at org.apache.commons.transaction.util.FileHelper.moveRec(FileHelper.java:111) at org.apache.commons.transaction.util.FileHelper.moveRec(FileHelper.java:135) at org.apache.commons.transaction.util.FileHelper.moveRec(FileHelper.java:111) at org.apache.commons.transaction.file.FileResourceManager$TransactionContext.commit(FileResourceManager.java:1366) at org.apache.commons.transaction.file.FileResourceManager.commitTransaction(FileResourceManager.java:648) at hu.siemens.tmk.transaction.fs.FSManager.doCommit(FSManager.java:128) at hu.siemens.tmk.transaction.AbstractTransactionManager.commitOrRollback(AbstractTransactionManager.java:99) at hu.siemens.tmk.transaction.AbstractTransactionManager.commit(AbstractTransactionManager.java:79) at hu.siemens.tmk.AbstractBridgingTransactionalTask.commitTransaction(AbstractBridgingTransactionalTask.java:125) at hu.siemens.tmd.AbstractBridgingTMDTask.doRunBridged(AbstractBridgingTMDTask.java:105) at hu.siemens.tmk.AbstractBridgingTransactionalTask.doRun(AbstractBridgingTransactionalTask.java:75) at hu.siemens.tmk.AbstractTask.run(AbstractTask.java:47) at hu.siemens.tmk.InterconnectedSTStage.doTheWork(InterconnectedSTStage.java:40) at hu.siemens.tmk.AbstractStage.run(AbstractStage.java:80) at java.lang.Thread.run(Thread.java:595) at org.apache.commons.transaction.file.FileResourceManager$TransactionContext.commit(FileResourceManager.java:1368) at org.apache.commons.transaction.file.FileResourceManager.commitTransaction(FileResourceManager.java:648) at hu.siemens.tmk.transaction.fs.FSManager.doCommit(FSManager.java:128) ... 9 more 2005-11-30 05:20:36.831 ERROR [InputReader:Stage InputReader] - hu.siemens.tmk.transaction.fs.FSManager: Exception during finishing transaction FSTx:1133324406295_129327 hu.siemens.tmk.transaction.TransactionException: FileResourceManager threw exception during rollback. at hu.siemens.tmk.transaction.fs.FSManager.doRollback(FSManager.java:148) at hu.siemens.tmk.transaction.AbstractTransactionManager.commitOrRollback(AbstractTransactionManager.java:102) at hu.siemens.tmk.transaction.AbstractTransactionManager.rollback(AbstractTransactionManager.java:84) at hu.siemens.tmk.AbstractBridgingTransactionalTask.rollbackTransaction(AbstractBridgingTransactionalTask.java:141) at hu.siemens.tmd.AbstractBridgingTMDTask.doRunBridged(AbstractBridgingTMDTask.java:108) at hu.siemens.tmk.AbstractBridgingTransactionalTask.doRun(AbstractBridgingTransactionalTask.java:75) at hu.siemens.tmk.AbstractTask.run(AbstractTask.java:47) at hu.siemens.tmk.InterconnectedSTStage.doTheWork(InterconnectedSTStage.java:40) at hu.siemens.tmk.AbstractStage.run(AbstractStage.java:80) at java.lang.Thread.run(Thread.java:595) Caused by: org.apache.commons.transaction.file.ResourceManagerSystemException: 1133324406295_129327: Database is set to dirty, this *may* mean it is corrupt. No modifications are allowed until a recovery run has been performed! (ERR_SYSTEM) at org.apache.commons.transaction.file.FileResourceManager.txInitialSaneCheckForWriting(FileResourceManager.java:932) at org.apache.commons.transaction.file.FileResourceManager.rollbackTransaction(FileResourceManager.java:600) at hu.siemens.tmk.transaction.fs.FSManager.doRollback(FSManager.java:146) ... 9 more 2005-11-30 05:20:36.831 ERROR [InputReader:Stage InputReader] - hu.siemens.tmk.InterconnectedSTStage: Stage InputReader throw Exception, exiting. hu.siemens.tmk.ExecutionException: hu.siemens.tmk.transaction.TransactionException: FileResourceManager threw exception during rollback. at hu.siemens.tmk.AbstractTask.run(AbstractTask.java:53) at hu.siemens.tmk.InterconnectedSTStage.doTheWork(InterconnectedSTStage.java:40) at hu.siemens.tmk.AbstractStage.run(AbstractStage.java:80) at java.lang.Thread.run(Thread.java:595) Caused by: hu.siemens.tmk.transaction.TransactionException: FileResourceManager threw exception during rollback. at hu.siemens.tmk.transaction.fs.FSManager.doRollback(FSManager.java:148) at hu.siemens.tmk.transaction.AbstractTransactionManager.commitOrRollback(AbstractTransactionManager.java:102) at hu.siemens.tmk.transaction.AbstractTransactionManager.rollback(AbstractTransactionManager.java:84) at hu.siemens.tmk.AbstractBridgingTransactionalTask.rollbackTransaction(AbstractBridgingTransactionalTask.java:141) at hu.siemens.tmd.AbstractBridgingTMDTask.doRunBridged(AbstractBridgingTMDTask.java:108) at hu.siemens.tmk.AbstractBridgingTransactionalTask.doRun(AbstractBridgingTransactionalTask.java:75) at hu.siemens.tmk.AbstractTask.run(AbstractTask.java:47) ... 3 more Caused by: org.apache.commons.transaction.file.ResourceManagerSystemException: 1133324406295_129327: Database is set to dirty, this *may* mean it is corrupt. No modifications are allowed until a recovery run has been performed! (ERR_SYSTEM) at org.apache.commons.transaction.file.FileResourceManager.txInitialSaneCheckForWriting(FileResourceManager.java:932) at org.apache.commons.transaction.file.FileResourceManager.rollbackTransaction(FileResourceManager.java:600) at hu.siemens.tmk.transaction.fs.FSManager.doRollback(FSManager.java:146) ... 9 more 2005-11-30 05:20:36.834 ERROR [TMKCmd:TMKCmd] - hu.siemens.tmk.TMKernel: Got ERROR from Stage InputReader 2005-11-30 05:20:36.834 ERROR [InputReader:Stage InputReader] - hu.siemens.tmk.transaction.fs.FSManager: Exception during finishing transaction FSTx:1133324406295_129327 hu.siemens.tmk.transaction.TransactionException: FileResourceManager threw exception during rollback. at hu.siemens.tmk.transaction.fs.FSManager.doRollback(FSManager.java:148) at hu.siemens.tmk.transaction.AbstractTransactionManager.commitOrRollback(AbstractTransactionManager.java:102) at hu.siemens.tmk.transaction.AbstractTransactionManager.rollback(AbstractTransactionManager.java:84) at hu.siemens.tmk.AbstractBridgingTransactionalTask.rollbackTransaction(AbstractBridgingTransactionalTask.java:141) at hu.siemens.tmd.AbstractBridgingTMDTask.doRunBridged(AbstractBridgingTMDTask.java:108) at hu.siemens.tmk.AbstractBridgingTransactionalTask.doRun(AbstractBridgingTransactionalTask.java:75) at hu.siemens.tmk.AbstractTask.run(AbstractTask.java:47) at hu.siemens.tmk.InterconnectedSTStage.doTheWork(InterconnectedSTStage.java:40) at hu.siemens.tmk.AbstractStage.run(AbstractStage.java:80) at java.lang.Thread.run(Thread.java:595) Caused by: org.apache.commons.transaction.file.ResourceManagerSystemException: 1133324406295_129327: Database is set to dirty, this *may* mean it is corrupt. No modifications are allowed until a recovery run has been performed! (ERR_SYSTEM) at org.apache.commons.transaction.file.FileResourceManager.txInitialSaneCheckForWriting(FileResourceManager.java:932) at org.apache.commons.transaction.file.FileResourceManager.rollbackTransaction(FileResourceManager.java:600) at hu.siemens.tmk.transaction.fs.FSManager.doRollback(FSManager.java:146) ... 9 more 2005-11-30 05:20:36.834 INFO [TMKCmd:TMKCmd] - hu.siemens.tmk.TMKernel: Alert [Internal kernel error] [Component Stage InputReader reported ERROR.] sent by Stage InputReader(hu.siemens.tmk.InterconnectedSTStage) 2005-11-30 05:20:36.834 ERROR [InputReader:Stage InputReader] - hu.siemens.tmk.InterconnectedSTStage: Stage InputReader throw Exception, exiting. hu.siemens.tmk.ExecutionException: hu.siemens.tmk.transaction.TransactionException: FileResourceManager threw exception during rollback. at hu.siemens.tmk.AbstractTask.run(AbstractTask.java:53) at hu.siemens.tmk.InterconnectedSTStage.doTheWork(InterconnectedSTStage.java:40) at hu.siemens.tmk.AbstractStage.run(AbstractStage.java:80) at java.lang.Thread.run(Thread.java:595) Caused by: hu.siemens.tmk.transaction.TransactionException: FileResourceManager threw exception during rollback. at hu.siemens.tmk.transaction.fs.FSManager.doRollback(FSManager.java:148) at hu.siemens.tmk.transaction.AbstractTransactionManager.commitOrRollback(AbstractTransactionManager.java:102) at hu.siemens.tmk.transaction.AbstractTransactionManager.rollback(AbstractTransactionManager.java:84) at hu.siemens.tmk.AbstractBridgingTransactionalTask.rollbackTransaction(AbstractBridgingTransactionalTask.java:141) at hu.siemens.tmd.AbstractBridgingTMDTask.doRunBridged(AbstractBridgingTMDTask.java:108) at hu.siemens.tmk.AbstractBridgingTransactionalTask.doRun(AbstractBridgingTransactionalTask.java:75) at hu.siemens.tmk.AbstractTask.run(AbstractTask.java:47) ... 3 more Caused by: org.apache.commons.transaction.file.ResourceManagerSystemException: 1133324406295_129327: Database is set to dirty, this *may* mean it is corrupt. No modifications are allowed until a recovery run has been performed! (ERR_SYSTEM) at org.apache.commons.transaction.file.FileResourceManager.txInitialSaneCheckForWriting(FileResourceManager.java:932) at org.apache.commons.transaction.file.FileResourceManager.rollbackTransaction(FileResourceManager.java:600) at hu.siemens.tmk.transaction.fs.FSManager.doRollback(FSManager.java:146) ... 9 more 2005-11-30 05:20:36.835 ERROR [InputReader:Stage InputReader] - hu.siemens.tmk.transaction.fs.FSManager: Exception during finishing transaction FSTx:1133324406295_129327 hu.siemens.tmk.transaction.TransactionException: FileResourceManager threw exception during rollback. at hu.siemens.tmk.transaction.fs.FSManager.doRollback(FSManager.java:148) at hu.siemens.tmk.transaction.AbstractTransactionManager.commitOrRollback(AbstractTransactionManager.java:102) at hu.siemens.tmk.transaction.AbstractTransactionManager.rollback(AbstractTransactionManager.java:84) at hu.siemens.tmk.AbstractBridgingTransactionalTask.rollbackTransaction(AbstractBridgingTransactionalTask.java:141) at hu.siemens.tmd.AbstractBridgingTMDTask.doRunBridged(AbstractBridgingTMDTask.java:108) at hu.siemens.tmk.AbstractBridgingTransactionalTask.doRun(AbstractBridgingTransactionalTask.java:75) at hu.siemens.tmk.AbstractTask.run(AbstractTask.java:47) at hu.siemens.tmk.InterconnectedSTStage.doTheWork(InterconnectedSTStage.java:40) at hu.siemens.tmk.AbstractStage.run(AbstractStage.java:80) at java.lang.Thread.run(Thread.java:595) Caused by: org.apache.commons.transaction.file.ResourceManagerSystemException: 1133324406295_129327: Database is set to dirty, this *may* mean it is corrupt. No modifications are allowed until a recovery run has been performed! (ERR_SYSTEM) at org.apache.commons.transaction.file.FileResourceManager.txInitialSaneCheckForWriting(FileResourceManager.java:932) at org.apache.commons.transaction.file.FileResourceManager.rollbackTransaction(FileResourceManager.java:600) at hu.siemens.tmk.transaction.fs.FSManager.doRollback(FSManager.java:146) ... 9 more 2005-11-30 05:20:36.836 ERROR [InputReader:Stage InputReader] - hu.siemens.tmk.InterconnectedSTStage: Stage InputReader throw Exception, exiting. hu.siemens.tmk.ExecutionException: hu.siemens.tmk.transaction.TransactionException: FileResourceManager threw exception during rollback. at hu.siemens.tmk.AbstractTask.run(AbstractTask.java:53) at hu.siemens.tmk.InterconnectedSTStage.doTheWork(InterconnectedSTStage.java:40) at hu.siemens.tmk.AbstractStage.run(AbstractStage.java:80) at java.lang.Thread.run(Thread.java:595) Caused by: hu.siemens.tmk.transaction.TransactionException: FileResourceManager threw exception during rollback. at hu.siemens.tmk.transaction.fs.FSManager.doRollback(FSManager.java:148) at hu.siemens.tmk.transaction.AbstractTransactionManager.commitOrRollback(AbstractTransactionManager.java:102) at hu.siemens.tmk.transaction.AbstractTransactionManager.rollback(AbstractTransactionManager.java:84) at hu.siemens.tmk.AbstractBridgingTransactionalTask.rollbackTransaction(AbstractBridgingTransactionalTask.java:141) at hu.siemens.tmd.AbstractBridgingTMDTask.doRunBridged(AbstractBridgingTMDTask.java:108) at hu.siemens.tmk.AbstractBridgingTransactionalTask.doRun(AbstractBridgingTransactionalTask.java:75) at hu.siemens.tmk.AbstractTask.run(AbstractTask.java:47) ... 3 more Caused by: org.apache.commons.transaction.file.ResourceManagerSystemException: 1133324406295_129327: Database is set to dirty, this *may* mean it is corrupt. No modifications are allowed until a recovery run has been performed! (ERR_SYSTEM) at org.apache.commons.transaction.file.FileResourceManager.txInitialSaneCheckForWriting(FileResourceManager.java:932) at org.apache.commons.transaction.file.FileResourceManager.rollbackTransaction(FileResourceManager.java:600) at hu.siemens.tmk.transaction.fs.FSManager.doRollback(FSManager.java:146) ... 9 more 2005-11-30 05:20:36.836 INFO [TMKCmd:TMKCmd] - hu.siemens.tmk.TMKernel: Initiating shutdown by call from Stage InputReader 2005-11-30 05:20:36.844 ERROR [TMKCmd:TMKCmd] - hu.siemens.tmk.TMKernel: Got ERROR from Stage InputReader 2005-11-30 05:20:36.845 INFO [TMKCmd:TMKCmd] - hu.siemens.tmk.TMKernel: Alert [Internal kernel error] [Component Stage InputReader reported ERROR.] sent by Stage InputReader(hu.siemens.tmk.InterconnectedSTStage) 2005-11-30 05:20:36.845 INFO [TMKCmd:TMKCmd] - hu.siemens.tmk.InterconnectedSTStage: Stopping Stage CRBExploder 2005-11-30 05:20:36.845 INFO [TMKCmd:TMKCmd] - hu.siemens.tmk.InterconnectedSTStage: Stopping Stage InputReader 2005-11-30 05:20:36.845 INFO [TMKCmd:TMKCmd] - hu.siemens.tmk.InterconnectedSTStage: Stopping Stage Algorithm 2005-11-30 05:20:36.845 INFO [TMKCmd:TMKCmd] - hu.siemens.tmk.InterconnectedSTStage: Stopping Stage CRBPacker thanx, ~t~ Oliver Zeigermann wrote: commit and not when you synchronize commit it is most likely that two commits clash in a way they should not.By the way synchronizing the commit takes a lot of possible concurrency from the manager as at most one tx can commit at it time in this case. |
--------------------------------------------------------------------- To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
