Just had something odd happen - my keyspace disappeared. I'm curious if anyone
has seen the same or may have an explanation.
I am using Titan 0.2.0 and Cassandra 1.1.6. I ran my application which creates
a graph and progressively builds it for a couple of hours. But before
completing, Cassandra seems to simply lose track of the keyspace,
"Recommender". The exception is that it doesn't exist. So I check in
cassandra-cli, and it also doesn't show the keyspace. The data directory and
column family directories are there, but are empty. Restarting cassandra
doesn't change anything.
Ron
Here is part of my application log and the Cassandra log from the relevant time:
2013-02-07 17:58:02,012 [pool-10-thread-26] INFO publisher.GraphPublisher -
RelationProcessor started fo
r items [47070, 34334, 34334, 34334, 42297, 34334, 34334]
2013-02-07 17:58:02,550 [pool-10-thread-36] ERROR publisher.GraphPublisher -
Unhandled error for account
7314766 order 17489735
com.thinkaurelius.titan.core.TitanException: Could not read from storage after
3 attempts
at
com.thinkaurelius.titan.graphdb.database.StandardTitanGraph.readException(StandardTitanGraph.j
ava:162)
at
com.thinkaurelius.titan.graphdb.database.StandardTitanGraph.appendResults(StandardTitanGraph.j
ava:607)
at
com.thinkaurelius.titan.graphdb.database.StandardTitanGraph.appendResults(StandardTitanGraph.j
ava:592)
at
com.thinkaurelius.titan.graphdb.database.StandardTitanGraph.queryForEntries(StandardTitanGraph
.java:557)
at
com.thinkaurelius.titan.graphdb.database.StandardTitanGraph.loadRelations(StandardTitanGraph.j
ava:322)
at
com.thinkaurelius.titan.graphdb.transaction.StandardPersistTitanTx.loadRelations(StandardPersi
stTitanTx.java:155)
at
com.thinkaurelius.titan.graphdb.vertices.AbstractTitanVertex.ensureLoadedEdges(AbstractTitanVe
rtex.java:88)
at
com.thinkaurelius.titan.graphdb.vertices.StandardTitanVertex.getRelations(StandardTitanVertex.
java:68)
at
com.thinkaurelius.titan.graphdb.query.SimpleAtomicQuery.propertyIterator(SimpleAtomicQuery.jav
a:461)
at
com.thinkaurelius.titan.graphdb.vertices.AbstractTitanVertex.getProperty(AbstractTitanVertex.j
ava:115)
at
com.thinkaurelius.titan.graphdb.vertices.AbstractTitanVertex.getProperty(AbstractTitanVertex.j
ava:127)
at
com.charityusa.publisher.GraphPublisher.updateRelation(GraphPublisher.java:206)
at
com.charityusa.publisher.GraphPublisher.access$100(GraphPublisher.java:55)
at
com.charityusa.publisher.GraphPublisher$OrderProcessor.call(GraphPublisher.java:320)
at
com.charityusa.publisher.GraphPublisher$OrderProcessor.call(GraphPublisher.java:266)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:722)
Caused by: com.thinkaurelius.titan.diskstorage.TemporaryStorageException:
Temporary failure in storage ba
ckend
at
com.thinkaurelius.titan.diskstorage.cassandra.astyanax.AstyanaxOrderedKeyColumnValueStore.getS
lice(AstyanaxOrderedKeyColumnValueStore.java:148)
at
com.thinkaurelius.titan.diskstorage.keycolumnvalue.BufferedKeyColumnValueStore.getSlice(Buffer
edKeyColumnValueStore.java:36)
at
com.thinkaurelius.titan.diskstorage.locking.consistentkey.ConsistentKeyLockStore.getSlice(Cons
istentKeyLockStore.java:82)
at
com.thinkaurelius.titan.graphdb.database.StandardTitanGraph.appendResults(StandardTitanGraph.j
ava:602)
... 18 more
Caused by: com.netflix.astyanax.connectionpool.exceptions.BadRequestException:
BadRequestException: [host
=cirrus1(10.1.5.44):9160, latency=0(0), attempts=1]
InvalidRequestException(why:Keyspace Recommender does
not exist)
at
com.netflix.astyanax.thrift.ThriftConverter.ToConnectionPoolException(ThriftConverter.java:159
)
at
com.netflix.astyanax.thrift.AbstractOperationImpl.execute(AbstractOperationImpl.java:60)
at
com.netflix.astyanax.thrift.ThriftColumnFamilyQueryImpl$1$2.execute(ThriftColumnFamilyQueryImp
l.java:196)
at
com.netflix.astyanax.thrift.ThriftColumnFamilyQueryImpl$1$2.execute(ThriftColumnFamilyQueryImp
l.java:188)
at
com.netflix.astyanax.thrift.ThriftSyncConnectionFactoryImpl$1.execute(ThriftSyncConnectionFact
oryImpl.java:131)
at
com.netflix.astyanax.connectionpool.impl.AbstractExecuteWithFailoverImpl.tryOperation(Abstract
ExecuteWithFailoverImpl.java:52)
at
com.netflix.astyanax.connectionpool.impl.AbstractHostPartitionConnectionPool.executeWithFailov
er(AbstractHostPartitionConnectionPool.java:229)
at
com.netflix.astyanax.thrift.ThriftColumnFamilyQueryImpl$1.execute(ThriftColumnFamilyQueryImpl.
java:186)
at
com.thinkaurelius.titan.diskstorage.cassandra.astyanax.AstyanaxOrderedKeyColumnValueStore.getS
lice(AstyanaxOrderedKeyColumnValueStore.java:145)
... 21 more
Caused by: InvalidRequestException(why:Keyspace Recommender does not exist)
at
org.apache.cassandra.thrift.Cassandra$get_slice_result.read(Cassandra.java:7280)
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:78)
at
org.apache.cassandra.thrift.Cassandra$Client.recv_get_slice(Cassandra.java:552)
at
org.apache.cassandra.thrift.Cassandra$Client.get_slice(Cassandra.java:536)
at
com.netflix.astyanax.thrift.ThriftColumnFamilyQueryImpl$1$2.internalExecute(ThriftColumnFamily
QueryImpl.java:201)
at
com.netflix.astyanax.thrift.ThriftColumnFamilyQueryImpl$1$2.internalExecute(ThriftColumnFamily
QueryImpl.java:188)
at
com.netflix.astyanax.thrift.AbstractOperationImpl.execute(AbstractOperationImpl.java:55)
... 28 more
Cassandra's log:
WARN [ScheduledTasks:1] 2013-02-07 17:46:24,935 StorageService.java (line
2855) Flushing CFS(Keyspace='R
ecommender', ColumnFamily='edgestore') to relieve memory pressure
INFO [ScheduledTasks:1] 2013-02-07 17:46:24,935 ColumnFamilyStore.java (line
659) Enqueuing flush of Mem
table-edgestore@2032656921(3689547/40713915 serialized/live bytes, 165613 ops)
INFO [FlushWriter:237] 2013-02-07 17:46:24,961 Memtable.java (line 264)
Writing Memtable-edgestore@20326
56921(3689547/40713915 serialized/live bytes, 165613 ops)
INFO [FlushWriter:237] 2013-02-07 17:46:29,798 Memtable.java (line 305)
Completed flushing /home/apollo/
cassandra/data/Recommender/edgestore/Recommender-edgestore-hf-7-Data.db
(2143580 bytes) for commitlog pos
ition ReplayPosition(segmentId=1360013782638, position=19121495)
INFO [MemoryMeter:1] 2013-02-07 17:49:39,624 Memtable.java (line 213)
CFS(Keyspace='Recommender', Column
Family='edgestore') liveRatio is 9.312177865096587 (just-counted was
9.312177865096587). calculation too
k 868ms for 11757 columns
INFO [Thrift:1977] 2013-02-07 17:58:01,292 MigrationManager.java (line 174)
Drop Keyspace 'Recommender'
INFO [MigrationStage:1] 2013-02-07 17:58:01,294 ColumnFamilyStore.java (line
659) Enqueuing flush of Mem
table-schema_keyspaces@937719474(0/0 serialized/live bytes, 1 ops)
INFO [FlushWriter:238] 2013-02-07 17:58:01,362 Memtable.java (line 264)
Writing Memtable-schema_keyspace
s@937719474(0/0 serialized/live bytes, 1 ops)
INFO [FlushWriter:238] 2013-02-07 17:58:01,404 Memtable.java (line 305)
Completed flushing /home/apollo/
cassandra/data/system/schema_keyspaces/system-schema_keyspaces-hf-96-Data.db
(61 bytes) for commitlog pos
ition ReplayPosition(segmentId=1360013782638, position=21758272)
INFO [MigrationStage:1] 2013-02-07 17:58:01,513 ColumnFamilyStore.java (line
659) Enqueuing flush of Mem
table-schema_columnfamilies@1656096012(0/0 serialized/live bytes, 1 ops)
INFO [FlushWriter:238] 2013-02-07 17:58:01,513 Memtable.java (line 264)
Writing Memtable-schema_columnfa
milies@1656096012(0/0 serialized/live bytes, 1 ops)
INFO [FlushWriter:238] 2013-02-07 17:58:01,519 Memtable.java (line 305)
Completed flushing /home/apollo/
cassandra/data/system/schema_columnfamilies/system-schema_columnfamilies-hf-302-Data.db
(61 bytes) for co
mmitlog position ReplayPosition(segmentId=1360013782638, position=21758674)
INFO [CompactionExecutor:442] 2013-02-07 17:58:01,531 CompactionTask.java
(line 109) Compacting [SSTable
Reader(path='/home/apollo/cassandra/data/system/schema_keyspaces/system-schema_keyspaces-hf-96-Data.db'),
SSTableReader(path='/home/apollo/cassandra/data/system/schema_keyspaces/system-schema_keyspaces-hf-95-Da
ta.db'),
SSTableReader(path='/home/apollo/cassandra/data/system/schema_keyspaces/system-schema_keyspaces-
hf-94-Data.db'),
SSTableReader(path='/home/apollo/cassandra/data/system/schema_keyspaces/system-schema_ke
yspaces-hf-93-Data.db')]
INFO [MigrationStage:1] 2013-02-07 17:58:01,596 ColumnFamilyStore.java (line
659) Enqueuing flush of Mem
table-schema_columns@1442995080(0/0 serialized/live bytes, 1 ops)
INFO [FlushWriter:238] 2013-02-07 17:58:01,597 Memtable.java (line 264)
Writing Memtable-schema_columns@
1442995080(0/0 serialized/live bytes, 1 ops)
INFO [FlushWriter:238] 2013-02-07 17:58:01,722 Memtable.java (line 305)
Completed flushing /home/apollo/
cassandra/data/system/schema_columns/system-schema_columns-hf-63-Data.db (61
bytes) for commitlog positio
n ReplayPosition(segmentId=1360013782638, position=21759634)
INFO [CompactionExecutor:442] 2013-02-07 17:58:01,746 CompactionTask.java
(line 221) Compacted to [/home
/apollo/cassandra/data/system/schema_keyspaces/system-schema_keyspaces-hf-97-Data.db,].
870 to 309 (~35%
of original) bytes for 2 keys at 0.001371MB/s. Time: 215ms.
INFO [MigrationStage:1] 2013-02-07 17:58:01,862 ColumnFamilyStore.java (line
659) Enqueuing flush of Mem
table-titan_ids@1923920972(2304/2880 serialized/live bytes, 36 ops)
INFO [FlushWriter:238] 2013-02-07 17:58:01,862 Memtable.java (line 264)
Writing Memtable-titan_ids@19239
20972(2304/2880 serialized/live bytes, 36 ops)
INFO [FlushWriter:238] 2013-02-07 17:58:01,914 Memtable.java (line 305)
Completed flushing /home/apollo/
cassandra/data/Recommender/titan_ids/Recommender-titan_ids-hf-1-Data.db (870
bytes) for commitlog positio
n ReplayPosition(segmentId=1360013782638, position=21759634)
INFO [MigrationStage:1] 2013-02-07 17:58:01,916 ColumnFamilyStore.java (line
659) Enqueuing flush of Mem
table-propertyindex_lock_@540683493(840/1050 serialized/live bytes, 28 ops)
INFO [FlushWriter:238] 2013-02-07 17:58:01,971 Memtable.java (line 264)
Writing Memtable-propertyindex_l
ock_@540683493(840/1050 serialized/live bytes, 28 ops)
INFO [FlushWriter:238] 2013-02-07 17:58:02,027 Memtable.java (line 305)
Completed flushing /home/apollo/
cassandra/data/Recommender/propertyindex_lock_/Recommender-propertyindex_lock_-hf-1-Data.db
(650 bytes) f
or commitlog position ReplayPosition(segmentId=1360013782638, position=21759634)
INFO [MigrationStage:1] 2013-02-07 17:58:02,047 ColumnFamilyStore.java (line
659) Enqueuing flush of Mem
table-edgestore@2005322361(1191757/13872316 serialized/live bytes, 53956 ops)
INFO [FlushWriter:238] 2013-02-07 17:58:02,050 Memtable.java (line 264)
Writing Memtable-edgestore@20053
22361(1191757/13872316 serialized/live bytes, 53956 ops)
INFO [FlushWriter:238] 2013-02-07 17:58:02,134 Memtable.java (line 305)
Completed flushing /home/apollo/
cassandra/data/Recommender/edgestore/Recommender-edgestore-hf-8-Data.db (720306
bytes) for commitlog posi
tion ReplayPosition(segmentId=1360013782638, position=21760036)
INFO [CompactionExecutor:443] 2013-02-07 17:58:02,135 CompactionTask.java
(line 109) Compacting [SSTable
Reader(path='/home/apollo/cassandra/data/Recommender/edgestore/Recommender-edgestore-hf-6-Data.db'),
SSTa
bleReader(path='/home/apollo/cassandra/data/Recommender/edgestore/Recommender-edgestore-hf-8-Data.db'),
S
STableReader(path='/home/apollo/cassandra/data/Recommender/edgestore/Recommender-edgestore-hf-5-Data.db')
,
SSTableReader(path='/home/apollo/cassandra/data/Recommender/edgestore/Recommender-edgestore-hf-7-Data.d
b')]
INFO [MigrationStage:1] 2013-02-07 17:58:02,160 ColumnFamilyStore.java (line
659) Enqueuing flush of Mem
table-propertyindex@1052449648(243223/8141246 serialized/live bytes, 8387 ops)
INFO [FlushWriter:238] 2013-02-07 17:58:02,160 Memtable.java (line 264)
Writing Memtable-propertyindex@1
052449648(243223/8141246 serialized/live bytes, 8387 ops)
INFO [FlushWriter:238] 2013-02-07 17:58:02,223 Memtable.java (line 305)
Completed flushing /home/apollo/
cassandra/data/Recommender/propertyindex/Recommender-propertyindex-hf-3-Data.db
(301191 bytes) for commit
log position ReplayPosition(segmentId=1360013782638, position=21760036)
INFO [CompactionExecutor:443] 2013-02-07 17:58:05,032 CompactionTask.java
(line 221) Compacted to [/home
/apollo/cassandra/data/Recommender/edgestore/Recommender-edgestore-hf-9-Data.db,].
17,110,084 to 16,336,
307 (~95% of original) bytes for 78,555 keys at 5.377810MB/s. Time: 2,897ms.
INFO [CompactionExecutor:444] 2013-02-07 21:36:21,419 AutoSavingCache.java
(line 268) Saved KeyCache (35
939 items) in 31 ms
INFO [CompactionExecutor:445] 2013-02-08 01:36:21,435 AutoSavingCache.java
(line 268) Saved KeyCache (35
939 items) in 37 ms
INFO [CompactionExecutor:446] 2013-02-08 05:36:21,447 AutoSavingCache.java
(line 268) Saved KeyCache (40
902 items) in 38 ms
INFO [CompactionExecutor:447] 2013-02-08 09:36:21,478 AutoSavingCache.java
(line 268) Saved KeyCache (40
902 items) in 59 ms
