Hi
We are trying to create a coordinator workflow on oozie-2.3.2+27.19.But we
are facing a certain problem,if the number of
coordinator exceeding more than two.We are getting the following error
2012-06-08 18:39:19,315 INFO WfEndCommand:525 - USER[sukirti]
GROUP[users] TOKEN[] APP[java-main-logs]
JOB[0000004-120608174022095-oozie-oozi-W]
ACTION[0000004-120608174022095-oozie-oozi-W@java-node] XException -
connection logs from store conn 597321988, false 2012-06-08 18:39:19,317
ERROR ActionStartCommand:522 - USER[-] GROUP[-] TOKEN[-] APP[-] JOB[-]
ACTION[-] XException, org.apache.oozie.command.CommandException: E0607:
Other error in operation [getWorkflow], An optimistic lock violation was
detected when flushing object instance "A lock could not be obtained within
the time requested [java.lang.String]" to the data store. This indicates
that the object was concurrently modified in another transaction.
org.apache.oozie.command.CommandException: E0607: Other error in operation
[getWorkflow], An optimistic lock violation was detected when flushing
object instance "A lock could not be obtained within the time requested
[java.lang.String]" to the data store. This indicates that the object was
concurrently modified in another transaction. at
org.apache.oozie.command.Command.call(Command.java:259) at
org.apache.oozie.command.wf.ActionStartCommand.handleError(ActionStartComman
d.java:241) at
org.apache.oozie.command.wf.ActionStartCommand.call(ActionStartCommand.java:
108) at
org.apache.oozie.command.wf.ActionStartCommand.execute(ActionStartCommand.ja
va:251) at
org.apache.oozie.command.wf.ActionStartCommand.execute(ActionStartCommand.ja
va:47) at org.apache.oozie.command.Command.call(Command.java:202) at
org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQu
eueService.java:132) at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.ja
va:886) at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
08) at java.lang.Thread.run(Thread.java:662) Caused by:
org.apache.oozie.store.StoreException: E0607: Other error in operation
[getWorkflow], An optimistic lock violation was detected when flushing
object instance "A lock could not be obtained within the time requested
[java.lang.String]" to the data store. This indicates that the object was
concurrently modified in another transaction. at
org.apache.oozie.store.WorkflowStore.doOperation(WorkflowStore.java:823)
at
org.apache.oozie.store.WorkflowStore.getWorkflow(WorkflowStore.java:125)
at org.apache.oozie.command.wf.WfEndCommand.call(WfEndCommand.java:56)
at
org.apache.oozie.command.wf.WfEndCommand.call(WfEndCommand.java:42) at
org.apache.oozie.command.Command.execute(Command.java:586) at
org.apache.oozie.command.Command.call(Command.java:202) ... 9 more
Caused
by: <openjpa-1.2.1-r752877:753278 nonfatal store error>
org.apache.openjpa.persistence.OptimisticLockException: An optimistic lock
violation was detected when flushing object instance "A lock could not be
obtained within the time requested [java.lang.String]" to the data store.
This indicates that the object was concurrently modified in another
transaction. at
org.apache.openjpa.jdbc.sql.DBDictionary.narrow(DBDictionary.java:4228)
at
org.apache.openjpa.jdbc.sql.DBDictionary.newStoreException(DBDictionary.java
:4197) at
org.apache.openjpa.jdbc.sql.SQLExceptions.getStore(SQLExceptions.java:102)
at
org.apache.openjpa.jdbc.sql.SQLExceptions.getStore(SQLExceptions.java:88)
at
org.apache.openjpa.jdbc.sql.SQLExceptions.getStore(SQLExceptions.java:64)
at
org.apache.openjpa.jdbc.kernel.SelectResultObjectProvider.handleCheckedExcep
tion(SelectResultObjectProvider.java:155) at
org.apache.openjpa.lib.rop.EagerResultList.<init>(EagerResultList.java:40)
at org.apache.openjpa.kernel.QueryImpl.toResult(QueryImpl.java:1228)
at
org.apache.openjpa.kernel.QueryImpl.execute(QueryImpl.java:990) at
org.apache.openjpa.kernel.QueryImpl.execute(QueryImpl.java:848) at
org.apache.openjpa.kernel.QueryImpl.execute(QueryImpl.java:779) at
org.apache.openjpa.kernel.DelegatingQuery.execute(DelegatingQuery.java:525)
at org.apache.openjpa.persistence.QueryImpl.execute(QueryImpl.java:254)
at
org.apache.openjpa.persistence.QueryImpl.getResultList(QueryImpl.java:294)
at
org.apache.oozie.store.WorkflowStore.getWorkflowOnly(WorkflowStore.java:838)
at
org.apache.oozie.store.WorkflowStore.access$100(WorkflowStore.java:54)
at org.apache.oozie.store.WorkflowStore$2.call(WorkflowStore.java:128)
at
org.apache.oozie.store.WorkflowStore$2.call(WorkflowStore.java:125) at
org.apache.oozie.store.WorkflowStore.doOperation(WorkflowStore.java:808)
... 14 more Caused by: java.sql.SQLTransactionRollbackException: A lock
could not be obtained within the time requested at
org.apache.derby.impl.jdbc.SQLExceptionFactory40.getSQLException(Unknown
Source) at
org.apache.derby.impl.jdbc.Util.generateCsSQLException(Unknown
Source) at
org.apache.derby.impl.jdbc.TransactionResourceImpl.wrapInSQLException(Unknow
n Source) at
org.apache.derby.impl.jdbc.TransactionResourceImpl.handleException(Unknown
Source) at
org.apache.derby.impl.jdbc.EmbedConnection.handleException(Unknown Source)
at org.apache.derby.impl.jdbc.ConnectionChild.handleException(Unknown
Source) at
org.apache.derby.impl.jdbc.EmbedResultSet.closeOnTransactionError(Unknown
Source) at
org.apache.derby.impl.jdbc.EmbedResultSet.movePosition(Unknown
Source) at org.apache.derby.impl.jdbc.EmbedResultSet.next(Unknown
Source)
at
org.apache.commons.dbcp.DelegatingResultSet.next(DelegatingResultSet.java:20
7) at
org.apache.commons.dbcp.DelegatingResultSet.next(DelegatingResultSet.java:20
7) at
org.apache.openjpa.lib.jdbc.DelegatingResultSet.next(DelegatingResultSet.jav
a:106) at
org.apache.openjpa.jdbc.sql.ResultSetResult.nextInternal(ResultSetResult.jav
a:222) at
org.apache.openjpa.jdbc.sql.SelectImpl$SelectResult.nextInternal(SelectImpl.
java:2457) at
org.apache.openjpa.jdbc.sql.AbstractResult.next(AbstractResult.java:173)
at
org.apache.openjpa.jdbc.kernel.SelectResultObjectProvider.next(SelectResultO
bjectProvider.java:99) at
org.apache.openjpa.lib.rop.EagerResultList.<init>(EagerResultList.java:35)
... 26 more Caused by: java.sql.SQLException: A lock could not be
obtained
within the time requested at
org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(Unknown
Source) at
org.apache.derby.impl.jdbc.SQLExceptionFactory40.wrapArgsForTransportAcrossD
RDA(Unknown Source) ... 43 more Caused by: ERROR 40XL1: A lock could not
be obtained within the time requested at
org.apache.derby.iapi.error.StandardException.newException(Unknown Source)
at
org.apache.derby.impl.services.locks.ConcurrentLockSet.lockObject(Unknown
Source) at
org.apache.derby.impl.services.locks.ConcurrentLockSet.zeroDurationLockObjec
t(Unknown Source) at
org.apache.derby.impl.services.locks.AbstractPool.zeroDurationlockObject(Unk
nown Source) at
org.apache.derby.impl.services.locks.ConcurrentPool.zeroDurationlockObject(U
nknown Source) at
org.apache.derby.impl.store.raw.xact.RowLocking2nohold.lockRecordForRead(Unk
nown Source) at
org.apache.derby.impl.store.access.heap.HeapController.lockRow(Unknown
Source) at
org.apache.derby.impl.store.access.heap.HeapController.lockRow(Unknown
Source) at
org.apache.derby.impl.store.access.btree.index.B2IRowLocking3.lockRowOnPage(
Unknown Source) at
org.apache.derby.impl.store.access.btree.index.B2IRowLocking3._lockScanRow(U
nknown Source) at
org.apache.derby.impl.store.access.btree.index.B2IRowLockingRR.lockScanRow(U
nknown Source) at
org.apache.derby.impl.store.access.btree.BTreeForwardScan.fetchRows(Unknown
Source) at
org.apache.derby.impl.store.access.btree.BTreeScan.fetchNextGroup(Unknown
Source) at
org.apache.derby.impl.sql.execute.BulkTableScanResultSet.reloadArray(Unknown
Source) at
org.apache.derby.impl.sql.execute.BulkTableScanResultSet.getNextRowCore(Unkn
own Source) at
org.apache.derby.impl.sql.execute.IndexRowToBaseRowResultSet.getNextRowCore(
Unknown Source) at
org.apache.derby.impl.sql.execute.ProjectRestrictResultSet.getNextRowCore(Un
known Source) at
org.apache.derby.impl.sql.execute.ProjectRestrictResultSet.getNextRowCore(Un
known Source) at
org.apache.derby.impl.sql.execute.BasicNoPutResultSetImpl.getNextRow(Unknown
Source) ... 36 more 2012-06-08 18:39:19,318 INFO
ActionStartCommand:525
- USER[-] GROUP[-] TOKEN[-] APP[-] JOB[-] ACTION[-] XException - connection
logs from store conn 1304116788, false 2012-06-08 18:39:19,330 WARN
CallableQueueService$CallableWrapper:528 - USER[-] GROUP[-] TOKEN[-] APP[-]
JOB[-] ACTION[-] exception callable [action.start], E0607: Other error in
operation [getWorkflow], An optimistic lock violation was detected when
flushing object instance "A lock could not be obtained within the time
requested [java.lang.String]" to the data store. This indicates that the
object was concurrently modified in another transaction.
org.apache.oozie.command.CommandException: E0607: Other error in operation
[getWorkflow], An optimistic lock violation was detected when flushing
object instance "A lock could not be obtained within the time requested
[java.lang.String]" to the data store. This indicates that the object was
concurrently modified in another transaction. at
org.apache.oozie.command.Command.call(Command.java:259) at
org.apache.oozie.command.wf.ActionStartCommand.handleError(ActionStartComman
d.java:241) at
org.apache.oozie.command.wf.ActionStartCommand.call(ActionStartCommand.java:
108) at
org.apache.oozie.command.wf.ActionStartCommand.execute(ActionStartCommand.ja
va:251) at
org.apache.oozie.command.wf.ActionStartCommand.execute(ActionStartCommand.ja
va:47) at org.apache.oozie.command.Command.call(Command.java:202) at
org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQu
eueService.java:132) at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.ja
va:886) at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
08) at java.lang.Thread.run(Thread.java:662) Caused by:
org.apache.oozie.store.StoreException: E0607: Other error in operation
[getWorkflow], An optimistic lock violation was detected when flushing
object instance "A lock could not be obtained within the time requested
[java.lang.String]" to the data store. This indicates that the object was
concurrently modified in another transaction. at
org.apache.oozie.store.WorkflowStore.doOperation(WorkflowStore.java:823)
at
org.apache.oozie.store.WorkflowStore.getWorkflow(WorkflowStore.java:125)
at org.apache.oozie.command.wf.WfEndCommand.call(WfEndCommand.java:56)
at
org.apache.oozie.command.wf.WfEndCommand.call(WfEndCommand.java:42) at
org.apache.oozie.command.Command.execute(Command.java:586) at
org.apache.oozie.command.Command.call(Command.java:202) ... 9 more
Caused
by: <openjpa-1.2.1-r752877:753278 nonfatal store error>
org.apache.openjpa.persistence.OptimisticLockException: An optimistic lock
violation was detected when flushing object instance "A lock could not be
obtained within the time requested [java.lang.String]" to the data store.
This indicates that the object was concurrently modified in another
transaction. at
org.apache.openjpa.jdbc.sql.DBDictionary.narrow(DBDictionary.java:4228)
at
org.apache.openjpa.jdbc.sql.DBDictionary.newStoreException(DBDictionary.java
:4197) at
org.apache.openjpa.jdbc.sql.SQLExceptions.getStore(SQLExceptions.java:102)
at
org.apache.openjpa.jdbc.sql.SQLExceptions.getStore(SQLExceptions.java:88)
at
org.apache.openjpa.jdbc.sql.SQLExceptions.getStore(SQLExceptions.java:64)
at
org.apache.openjpa.jdbc.kernel.SelectResultObjectProvider.handleCheckedExcep
tion(SelectResultObjectProvider.java:155) at
org.apache.openjpa.lib.rop.EagerResultList.<init>(EagerResultList.java:40)
at org.apache.openjpa.kernel.QueryImpl.toResult(QueryImpl.java:1228)
at
org.apache.openjpa.kernel.QueryImpl.execute(QueryImpl.java:990) at
org.apache.openjpa.kernel.QueryImpl.execute(QueryImpl.java:848) at
org.apache.openjpa.kernel.QueryImpl.execute(QueryImpl.java:779) at
org.apache.openjpa.kernel.DelegatingQuery.execute(DelegatingQuery.java:525)
at org.apache.openjpa.persistence.QueryImpl.execute(QueryImpl.java:254)
at
org.apache.openjpa.persistence.QueryImpl.getResultList(QueryImpl.java:294)
at
org.apache.oozie.store.WorkflowStore.getWorkflowOnly(WorkflowStore.java:838)
at
org.apache.oozie.store.WorkflowStore.access$100(WorkflowStore.java:54)
at org.apache.oozie.store.WorkflowStore$2.call(WorkflowStore.java:128)
at
org.apache.oozie.store.WorkflowStore$2.call(WorkflowStore.java:125) at
org.apache.oozie.store.WorkflowStore.doOperation(WorkflowStore.java:808)
... 14 more Caused by: java.sql.SQLTransactionRollbackException: A lock
could not be obtained within the time requested at
org.apache.derby.impl.jdbc.SQLExceptionFactory40.getSQLException(Unknown
Source) at
org.apache.derby.impl.jdbc.Util.generateCsSQLException(Unknown
Source) at
org.apache.derby.impl.jdbc.TransactionResourceImpl.wrapInSQLException(Unknow
n Source) at
org.apache.derby.impl.jdbc.TransactionResourceImpl.handleException(Unknown
Source) at
org.apache.derby.impl.jdbc.EmbedConnection.handleException(Unknown Source)
at org.apache.derby.impl.jdbc.ConnectionChild.handleException(Unknown
Source) at
org.apache.derby.impl.jdbc.EmbedResultSet.closeOnTransactionError(Unknown
Source) at
org.apache.derby.impl.jdbc.EmbedResultSet.movePosition(Unknown
Source) at org.apache.derby.impl.jdbc.EmbedResultSet.next(Unknown
Source)
at
org.apache.commons.dbcp.DelegatingResultSet.next(DelegatingResultSet.java:20
7) at
org.apache.commons.dbcp.DelegatingResultSet.next(DelegatingResultSet.java:20
7) at
org.apache.openjpa.lib.jdbc.DelegatingResultSet.next(DelegatingResultSet.jav
a:106) at
org.apache.openjpa.jdbc.sql.ResultSetResult.nextInternal(ResultSetResult.jav
a:222) at
org.apache.openjpa.jdbc.sql.SelectImpl$SelectResult.nextInternal(SelectImpl.
java:2457) at
org.apache.openjpa.jdbc.sql.AbstractResult.next(AbstractResult.java:173)
at
org.apache.openjpa.jdbc.kernel.SelectResultObjectProvider.next(SelectResultO
bjectProvider.java:99) at
org.apache.openjpa.lib.rop.EagerResultList.<init>(EagerResultList.java:35)
... 26 more Caused by: java.sql.SQLException: A lock could not be
obtained
within the time requested at
org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(Unknown
Source) at
org.apache.derby.impl.jdbc.SQLExceptionFactory40.wrapArgsForTransportAcrossD
RDA(Unknown Source) ... 43 more Caused by: ERROR 40XL1: A lock could not
be obtained within the time requested at
org.apache.derby.iapi.error.StandardException.newException(Unknown Source)
at
org.apache.derby.impl.services.locks.ConcurrentLockSet.lockObject(Unknown
Source) at
org.apache.derby.impl.services.locks.ConcurrentLockSet.zeroDurationLockObjec
t(Unknown Source) at
org.apache.derby.impl.services.locks.AbstractPool.zeroDurationlockObject(Unk
nown Source) at
org.apache.derby.impl.services.locks.ConcurrentPool.zeroDurationlockObject(U
nknown Source) at
org.apache.derby.impl.store.raw.xact.RowLocking2nohold.lockRecordForRead(Unk
nown Source) at
org.apache.derby.impl.store.access.heap.HeapController.lockRow(Unknown
Source) at
org.apache.derby.impl.store.access.heap.HeapController.lockRow(Unknown
Source) at
org.apache.derby.impl.store.access.btree.index.B2IRowLocking3.lockRowOnPage(
Unknown Source) at
org.apache.derby.impl.store.access.btree.index.B2IRowLocking3._lockScanRow(U
nknown Source) at
org.apache.derby.impl.store.access.btree.index.B2IRowLockingRR.lockScanRow(U
nknown Source) at
org.apache.derby.impl.store.access.btree.BTreeForwardScan.fetchRows(Unknown
Source) at
org.apache.derby.impl.store.access.btree.BTreeScan.fetchNextGroup(Unknown
Source) at
org.apache.derby.impl.sql.execute.BulkTableScanResultSet.reloadArray(Unknown
Source) at
org.apache.derby.impl.sql.execute.BulkTableScanResultSet.getNextRowCore(Unkn
own Source) at
org.apache.derby.impl.sql.execute.IndexRowToBaseRowResultSet.getNextRowCore(
Unknown Source) at
org.apache.derby.impl.sql.execute.ProjectRestrictResultSet.getNextRowCore(Un
known Source) at
org.apache.derby.impl.sql.execute.ProjectRestrictResultSet.getNextRowCore(Un
known Source) at
org.apache.derby.impl.sql.execute.BasicNoPutResultSetImpl.getNextRow(Unknown
Source) ... 36 more 2012-06-08 18:39:19,334 WARN
ActionStartCommand:528
- USER[sukirti] GROUP[users] TOKEN[] APP[java-main-logs]
JOB[0000004-120608174022095-oozie-oozi-W] ACTION[-]
[***0000004-120608174022095-oozie-oozi-W@java-node***]In
call()....status=PREP 2012-06-08 18:39:19,343 WARN ActionStartCommand:528
- USER[sukirti] GROUP[users] TOKEN[] APP[java-main-logs]
JOB[0000004-120608174022095-oozie-oozi-W]
ACTION[0000004-120608174022095-oozie-oozi-W@java-node] ELException in
ActionStartCommand javax.servlet.jsp.el.ELException: variable [inputData]
cannot be resolved at
org.apache.oozie.util.ELEvaluator$Context.resolveVariable(ELEvaluator.java:1
02) at org.apache.commons.el.NamedValue.evaluate(NamedValue.java:124)
at
org.apache.commons.el.ExpressionString.evaluate(ExpressionString.java:114)
at
org.apache.commons.el.ExpressionEvaluatorImpl.evaluate(ExpressionEvaluatorIm
pl.java:274) at
org.apache.commons.el.ExpressionEvaluatorImpl.evaluate(ExpressionEvaluatorIm
pl.java:190) at
org.apache.oozie.util.ELEvaluator.evaluate(ELEvaluator.java:199) at
org.apache.oozie.command.wf.ActionStartCommand.call(ActionStartCommand.java:
94) at
org.apache.oozie.command.wf.ActionStartCommand.execute(ActionStartCommand.ja
va:251) at
org.apache.oozie.command.wf.ActionStartCommand.execute(ActionStartCommand.ja
va:47) at org.apache.oozie.command.Command.call(Command.java:202) at
org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQu
eueService.java:132) at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.ja
va:886) at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
08) at java.lang.Thread.run(Thread.java:662) 2012-06-08 18:39:19,343 WARN
ActionStartCommand:528 - USER[sukirti] GROUP[users] TOKEN[]
APP[java-main-logs] JOB[0000004-120608174022095-oozie-oozi-W]
ACTION[0000004-120608174022095-oozie-oozi-W@java-node] Failing Job due to
failed action [java-node] 2012-06-08 18:39:19,345 WARN
LiteWorkflowInstance:528 - USER[sukirti] GROUP[users] TOKEN[]
APP[java-main-logs] JOB[0000004-120608174022095-oozie-oozi-W]
ACTION[0000004-120608174022095-oozie-oozi-W@java-node] Workflow Failed.
Failing node [java-node]
Can you please help in finding the solution to this particular problem.
Thanks and RegardsSukirti Panigrahi
