Re: [Neo4j] neo4j - One of the node hangs at times
Please create a GitHub issue. You might also want to migrate to a more
recent version like 3.4.5 and possibly move away from HA to a causal
cluster.
On Fri, Aug 3, 2018 at 7:55 PM, wrote:
> One or other of our node in HA architecture get hung most of the times
> Our neo4j version is 3.2.9 .
>
> $ neo4j status
> Neo4j is not running
> $neo4j start
> ...
> Starting Neo4j.
> Started neo4j (pid 15565). It is available at http://127.0.0.1:7474/
> There may be a short delay until the server is ready.
> See /neo4j/lensdblog/neo4j.log for current status.
> $tail -20 neo4j.log
> at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.
> start(LifeSupport.java:434)
> ... 5 more
> Caused by: org.neo4j.kernel.lifecycle.LifecycleException: Component
> 'org.neo4j.causalclustering.core.state.CoreLife@f719071' was successfully
> initialized, but failed to start. Please see the attached cause exception
> "Unable to find transaction 1 in any of my logical logs: Couldn't find any
> log containing 1".
> at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.
> start(LifeSupport.java:444)
> at org.neo4j.kernel.lifecycle.LifeSupport.start(LifeSupport.
> java:107)
> at org.neo4j.kernel.impl.factory.GraphDatabaseFacadeFactory.
> initFacade(GraphDatabaseFacadeFactory.java:205)
> ... 10 more
> Caused by: java.lang.RuntimeException: org.neo4j.kernel.impl.
> transaction.log.NoSuchTransactionException: Unable to find transaction 1
> in any of my logical logs: Couldn't find any log containing 1
> at org.neo4j.causalclustering.core.state.machines.tx.
> LastCommittedIndexFinder.getLastCommittedIndex(
> LastCommittedIndexFinder.java:67)
> at org.neo4j.causalclustering.core.state.machines.tx.
> RecoverConsensusLogIndex.findLastAppliedIndex(
> RecoverConsensusLogIndex.java:48)
> at org.neo4j.causalclustering.core.state.machines.
> CoreStateMachines.installCommitProcess(CoreStateMachines.java:138)
> at org.neo4j.causalclustering.core.state.CoreLife.start(
> CoreLife.java:83)
> at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.
> start(LifeSupport.java:434)
> ... 12 more
> Caused by: org.neo4j.kernel.impl.transaction.log.NoSuchTransactionException:
> Unable to find transaction 1 in any of my logical logs: Couldn't find any
> log containing 1
> at org.neo4j.kernel.impl.transaction.log.
> PhysicalLogicalTransactionStore$LogVersionLocator.getLogPosition(
> PhysicalLogicalTransactionStore.java:223)
> at org.neo4j.kernel.impl.transaction.log.
> PhysicalLogicalTransactionStore.getTransactions(
> PhysicalLogicalTransactionStore.java:83)
> at org.neo4j.causalclustering.core.state.machines.tx.
> LastCommittedIndexFinder.getLastCommittedIndex(
> LastCommittedIndexFinder.java:57)
> ... 16 more
> 2018-08-03 17:50:29.302+ INFO Neo4j Server shutdown initiated by
> request
> $cat debug.log
> 2018-08-03 17:50:26.154+ INFO [o.n.k.i.a.i.IndexingService]
> IndexingService.init: indexes not specifically mentioned above are null
> 2018-08-03 17:50:26.206+ INFO [o.n.k.NeoStoreDataSource] No commits
> found after last check point (which is at LogPosition{logVersion=0,
> byteOffset=16})
> 2018-08-03 17:50:26.212+ INFO [o.n.k.i.a.i.IndexingService]
> IndexingService.start: indexes not specifically mentioned above are null
> 2018-08-03 17:50:26.216+ INFO [o.n.k.i.DatabaseHealth] Database health
> set to OK
> 2018-08-03 17:50:26.900+ INFO [o.n.c.c.c.RaftServer] RaftServer
> stopping and unbinding from 0.0.0.0:7000
> 2018-08-03 17:50:29.106+ WARN [o.n.c.c.c.RaftServer] Worker group not
> shutdown within 10 seconds.
> 2018-08-03 17:50:29.106+ INFO [o.n.c.c.c.l.p.PruningScheduler]
> PruningScheduler stopping
> 2018-08-03 17:50:29.107+ INFO [o.n.c.c.CatchUpClient] CatchUpClient
> stopping
> 2018-08-03 17:50:29.121+ INFO [o.n.c.d.HazelcastCoreTopologyService]
> HazelcastCoreTopologyService stopping and unbinding from 0.0.0.0:5000
> 2018-08-03 17:50:29.140+ INFO [o.n.k.i.DiagnosticsManager] ---
> STOPPING diagnostics START ---
> 2018-08-03 17:50:29.141+ INFO [o.n.k.i.DiagnosticsManager] ---
> STOPPING diagnostics END ---
> 2018-08-03 17:50:29.141+ INFO [o.n.k.i.f.GraphDatabaseFacadeFactory]
> Shutdown started
> 2018-08-03 17:50:29.150+ INFO [o.n.k.i.t.l.c.CheckPointerImpl] Check
> Pointing triggered by database shutdown [1]: Starting check pointing...
> 2018-08-03 17:50:29.150+ INFO [o.n.k.i.t.l.c.CheckPointerImpl] Check
> Pointing triggered by database shutdown [1]: Starting store flush...
> 2018-08-03 17:50:29.155+ INFO [o.n.k.i.t.l.c.CheckPointerImpl] Check
> Pointing triggered by database shutdown [1]: Store flush completed
> 2018-08-03 17:50:29.155+ INFO [o.n.k.i.t.l.c.CheckPointerImpl] Check
> Pointing triggered by database shutdown [1]: Starting appending check
> point entry into the tx log...
> 2018-08-03 17:50:29.169+ INFO
[Neo4j] neo4j - One of the node hangs at times
One or other of our node in HA architecture get hung most of the times
Our neo4j version is 3.2.9 .
$ neo4j status
Neo4j is not running
$neo4j start
...
Starting Neo4j.
Started neo4j (pid 15565). It is available at http://127.0.0.1:7474/
There may be a short delay until the server is ready.
See /neo4j/lensdblog/neo4j.log for current status.
$tail -20 neo4j.log
at
org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:434)
... 5 more
Caused by: org.neo4j.kernel.lifecycle.LifecycleException: Component
'org.neo4j.causalclustering.core.state.CoreLife@f719071' was successfully
initialized, but failed to start. Please see the attached cause exception
"Unable to find transaction 1 in any of my logical logs: Couldn't find any
log containing 1".
at
org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:444)
at
org.neo4j.kernel.lifecycle.LifeSupport.start(LifeSupport.java:107)
at
org.neo4j.kernel.impl.factory.GraphDatabaseFacadeFactory.initFacade(GraphDatabaseFacadeFactory.java:205)
... 10 more
Caused by: java.lang.RuntimeException:
org.neo4j.kernel.impl.transaction.log.NoSuchTransactionException: Unable to
find transaction 1 in any of my logical logs: Couldn't find any log
containing 1
at
org.neo4j.causalclustering.core.state.machines.tx.LastCommittedIndexFinder.getLastCommittedIndex(LastCommittedIndexFinder.java:67)
at
org.neo4j.causalclustering.core.state.machines.tx.RecoverConsensusLogIndex.findLastAppliedIndex(RecoverConsensusLogIndex.java:48)
at
org.neo4j.causalclustering.core.state.machines.CoreStateMachines.installCommitProcess(CoreStateMachines.java:138)
at
org.neo4j.causalclustering.core.state.CoreLife.start(CoreLife.java:83)
at
org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:434)
... 12 more
Caused by:
org.neo4j.kernel.impl.transaction.log.NoSuchTransactionException: Unable to
find transaction 1 in any of my logical logs: Couldn't find any log
containing 1
at
org.neo4j.kernel.impl.transaction.log.PhysicalLogicalTransactionStore$LogVersionLocator.getLogPosition(PhysicalLogicalTransactionStore.java:223)
at
org.neo4j.kernel.impl.transaction.log.PhysicalLogicalTransactionStore.getTransactions(PhysicalLogicalTransactionStore.java:83)
at
org.neo4j.causalclustering.core.state.machines.tx.LastCommittedIndexFinder.getLastCommittedIndex(LastCommittedIndexFinder.java:57)
... 16 more
2018-08-03 17:50:29.302+ INFO Neo4j Server shutdown initiated by
request
$cat debug.log
2018-08-03 17:50:26.154+ INFO [o.n.k.i.a.i.IndexingService]
IndexingService.init: indexes not specifically mentioned above are null
2018-08-03 17:50:26.206+ INFO [o.n.k.NeoStoreDataSource] No commits
found after last check point (which is at LogPosition{logVersion=0,
byteOffset=16})
2018-08-03 17:50:26.212+ INFO [o.n.k.i.a.i.IndexingService]
IndexingService.start: indexes not specifically mentioned above are null
2018-08-03 17:50:26.216+ INFO [o.n.k.i.DatabaseHealth] Database health
set to OK
2018-08-03 17:50:26.900+ INFO [o.n.c.c.c.RaftServer] RaftServer
stopping and unbinding from 0.0.0.0:7000
2018-08-03 17:50:29.106+ WARN [o.n.c.c.c.RaftServer] Worker group not
shutdown within 10 seconds.
2018-08-03 17:50:29.106+ INFO [o.n.c.c.c.l.p.PruningScheduler]
PruningScheduler stopping
2018-08-03 17:50:29.107+ INFO [o.n.c.c.CatchUpClient] CatchUpClient
stopping
2018-08-03 17:50:29.121+ INFO [o.n.c.d.HazelcastCoreTopologyService]
HazelcastCoreTopologyService stopping and unbinding from 0.0.0.0:5000
2018-08-03 17:50:29.140+ INFO [o.n.k.i.DiagnosticsManager] --- STOPPING
diagnostics START ---
2018-08-03 17:50:29.141+ INFO [o.n.k.i.DiagnosticsManager] --- STOPPING
diagnostics END ---
2018-08-03 17:50:29.141+ INFO [o.n.k.i.f.GraphDatabaseFacadeFactory]
Shutdown started
2018-08-03 17:50:29.150+ INFO [o.n.k.i.t.l.c.CheckPointerImpl] Check
Pointing triggered by database shutdown [1]: Starting check pointing...
2018-08-03 17:50:29.150+ INFO [o.n.k.i.t.l.c.CheckPointerImpl] Check
Pointing triggered by database shutdown [1]: Starting store flush...
2018-08-03 17:50:29.155+ INFO [o.n.k.i.t.l.c.CheckPointerImpl] Check
Pointing triggered by database shutdown [1]: Store flush completed
2018-08-03 17:50:29.155+ INFO [o.n.k.i.t.l.c.CheckPointerImpl] Check
Pointing triggered by database shutdown [1]: Starting appending check
point entry into the tx log...
2018-08-03 17:50:29.169+ INFO [o.n.k.i.t.l.c.CheckPointerImpl] Check
Pointing triggered by database shutdown [1]: Appending check point entry
into the tx log completed
2018-08-03 17:50:29.169+ INFO [o.n.k.i.t.l.c.CheckPointerImpl] Check
Pointing triggered by database shutdown [1]: Check pointing completed
2018-08-03 17:50:29.169+ INFO [o.n.k.i.t.l.p.LogPruningImpl] Log
