[ https://issues.apache.org/jira/browse/CASSANDRA-4337?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Patrycjusz Matuszak updated CASSANDRA-4337: ------------------------------------------- Attachment: system-node1.log Cassandra server log from node 1 > Data insertion fails because of commitlog rename failure > -------------------------------------------------------- > > Key: CASSANDRA-4337 > URL: https://issues.apache.org/jira/browse/CASSANDRA-4337 > Project: Cassandra > Issue Type: Bug > Components: Core > Affects Versions: 1.1.1 > Environment: - Node 1: > Hardware: Intel Xeon 2.83 GHz (4 cores), 24GB RAM, Dell VIRTUAL DISK SCSI > 500GB > System: Windows Server 2008 R2 x64 > Java version: 7 update 4 x64 > - Node 2: > Hardware: Intel Xeon 2.83 GHz (4 cores), 8GB RAM, Dell VIRTUAL DISK SCSI > 500GB > System: Windows Server 2008 R2 x64 > Java version: 7 update 4 x64 > Reporter: Patrycjusz Matuszak > Attachments: system-node1.log, system-node2.log > > > h3. Configuration > Cassandra server configuration: > {noformat}heap size: 4 GB > seed_provider: > - class_name: org.apache.cassandra.locator.SimpleSeedProvider > parameters: > - seeds: "xxx.xxx.xxx.10,xxx.xxx.xxx.11" > listen_address: xxx.xxx.xxx.10 > rpc_address: 0.0.0.0 > rpc_port: 9160 > rpc_timeout_in_ms: 20000 > endpoint_snitch: PropertyFileSnitch{noformat} > cassandra-topology.properties > {noformat}xxx.xxx.xxx.10=datacenter1:rack1 > xxx.xxx.xxx.11=datacenter1:rack1 > default=datacenter1:rack1{noformat} > Ring configuration: > {noformat}Address DC Rack Status State Load > Effective-Ownership Token > > 85070591730234615865843651857942052864 > xxx.xxx.xxx.10 datacenter1 rack1 Up Normal 23,11 kB > 100,00% 0 > xxx.xxx.xxx.11 datacenter1 rack1 Up Normal 23,25 kB > 100,00% 85070591730234615865843651857942052864{noformat} > h3.Problem > I have ctreated keyspace and column family using CLI commands: > {noformat}create keyspace testks with placement_strategy = > 'org.apache.cassandra.locator.NetworkTopologyStrategy' and strategy_options = > {datacenter1:2}; > use testks; > create column family testcf;{noformat} > Then I started my Java application, which inserts 50 000 000 rows to created > column family using Hector client. Client is connected to node 1. > After about 30 seconds (160 000 rows were inserted) Cassandra server on node > 1 throws an exception: > {noformat}ERROR [COMMIT-LOG-ALLOCATOR] 2012-06-13 10:26:38,393 > AbstractCassandraDaemon.java (line 134) Exception in thread > Thread[COMMIT-LOG-ALLOCATOR,5,main] > java.io.IOError: java.io.IOException: Rename from > c:\apache-cassandra\storage\commitlog\CommitLog-7345742389552.log to > 7475933520374 failed > at > org.apache.cassandra.db.commitlog.CommitLogSegment.<init>(CommitLogSegment.java:127) > at > org.apache.cassandra.db.commitlog.CommitLogSegment.recycle(CommitLogSegment.java:204) > at > org.apache.cassandra.db.commitlog.CommitLogAllocator$2.run(CommitLogAllocator.java:166) > at > org.apache.cassandra.db.commitlog.CommitLogAllocator$1.runMayThrow(CommitLogAllocator.java:95) > at > org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30) > at java.lang.Thread.run(Thread.java:722) > Caused by: java.io.IOException: Rename from > c:\apache-cassandra\storage\commitlog\CommitLog-7345742389552.log to > 7475933520374 failed > at > org.apache.cassandra.db.commitlog.CommitLogSegment.<init>(CommitLogSegment.java:105) > ... 5 more{noformat} > > Then, few seconds later Cassandra server on node 2 throws the same exception: > {noformat}ERROR [COMMIT-LOG-ALLOCATOR] 2012-06-14 10:26:44,005 > AbstractCassandraDaemon.java (line 134) Exception in thread > Thread[COMMIT-LOG-ALLOCATOR,5,main] > java.io.IOError: java.io.IOException: Rename from > c:\apache-cassandra\storage\commitlog\CommitLog-7320337904033.log to > 7437675489307 failed > at > org.apache.cassandra.db.commitlog.CommitLogSegment.<init>(CommitLogSegment.java:127) > at > org.apache.cassandra.db.commitlog.CommitLogSegment.recycle(CommitLogSegment.java:204) > at > org.apache.cassandra.db.commitlog.CommitLogAllocator$2.run(CommitLogAllocator.java:166) > at > org.apache.cassandra.db.commitlog.CommitLogAllocator$1.runMayThrow(CommitLogAllocator.java:95) > at > org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30) > at java.lang.Thread.run(Unknown Source) > Caused by: java.io.IOException: Rename from > c:\apache-cassandra\storage\commitlog\CommitLog-7320337904033.log to > 7437675489307 failed > at > org.apache.cassandra.db.commitlog.CommitLogSegment.<init>(CommitLogSegment.java:105) > ... 5 more{noformat} > After that, my application cannot insert any more data. Hector gets > TimedOutException from Thrift: > {noformat}Thread-4 HConnectionManager.java 306 2012-06-14 10:12:56,034 > HConnectionManager operateWithFailover WARN %Could not fullfill > request on this host CassandraClient<172.22.1.36:9160-10> > Thread-4 HConnectionManager.java 307 2012-06-14 10:12:56,034 > HConnectionManager operateWithFailover WARN %Exception: > me.prettyprint.hector.api.exceptions.HTimedOutException: TimedOutException() > at > me.prettyprint.cassandra.service.ExceptionsTranslatorImpl.translate(ExceptionsTranslatorImpl.java:35) > at > me.prettyprint.cassandra.connection.HConnectionManager.operateWithFailover(HConnectionManager.java:264) > at > me.prettyprint.cassandra.model.ExecutingKeyspace.doExecuteOperation(ExecutingKeyspace.java:97) > at > me.prettyprint.cassandra.model.MutatorImpl.execute(MutatorImpl.java:243) > at > patrycjusz.nosqltest.db.cassandra.CassandraHectorDbAdapter.commitTransaction(CassandraDbAdapter.java:63) > at patrycjusz.nosqltest.DbTest.insertData(DbTest.java:459) > at patrycjusz.nosqltest.gui.InsertPanel.executeTask(NePanel.java:154) > at patrycjusz.nosqltest.gui.InsertPanel$1.run(NePanel.java:141) > at java.lang.Thread.run(Unknown Source) > Caused by: TimedOutException() > at > org.apache.cassandra.thrift.Cassandra$batch_mutate_result.read(Cassandra.java:20269) > at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:78) > at > org.apache.cassandra.thrift.Cassandra$Client.recv_batch_mutate(Cassandra.java:922) > at > org.apache.cassandra.thrift.Cassandra$Client.batch_mutate(Cassandra.java:908) > at > me.prettyprint.cassandra.model.MutatorImpl$3.execute(MutatorImpl.java:246) > at > me.prettyprint.cassandra.model.MutatorImpl$3.execute(MutatorImpl.java:243) > at > me.prettyprint.cassandra.service.Operation.executeAndSetResult(Operation.java:103) > at > me.prettyprint.cassandra.connection.HConnectionManager.operateWithFailover(HConnectionManager.java:258) > ... 8 more{noformat} -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira