[jira] [Commented] (CASSANDRA-4337) Data insertion fails because of commitlog rename failure
[ https://issues.apache.org/jira/browse/CASSANDRA-4337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13405929#comment-13405929 ] Patrycjusz Matuszak commented on CASSANDRA-4337: I've tested with attached patch and couldn't reproduce this bug. > Data insertion fails because of commitlog rename failure > > > Key: CASSANDRA-4337 > URL: https://issues.apache.org/jira/browse/CASSANDRA-4337 > Project: Cassandra > Issue Type: Bug > Components: Core >Affects Versions: 1.1.1 > Environment: - Node 1: >Hardware: Intel Xeon 2.83 GHz (4 cores), 24GB RAM, Dell VIRTUAL DISK SCSI > 500GB >System: Windows Server 2008 R2 x64 >Java version: 7 update 4 x64 > - Node 2: > Hardware: Intel Xeon 2.83 GHz (4 cores), 8GB RAM, Dell VIRTUAL DISK SCSI > 500GB > System: Windows Server 2008 R2 x64 > Java version: 7 update 4 x64 >Reporter: Patrycjusz Matuszak >Assignee: Jonathan Ellis > Labels: commitlog > Fix For: 1.1.3 > > Attachments: 4337-poc.txt, system-node1-stress-test.log, > system-node1.log, system-node2-stress-test.log, system-node2.log > > > h3. Configuration > Cassandra server configuration: > {noformat}heap size: 4 GB > seed_provider: > - class_name: org.apache.cassandra.locator.SimpleSeedProvider > parameters: > - seeds: "xxx.xxx.xxx.10,xxx.xxx.xxx.11" > listen_address: xxx.xxx.xxx.10 > rpc_address: 0.0.0.0 > rpc_port: 9160 > rpc_timeout_in_ms: 2 > endpoint_snitch: PropertyFileSnitch{noformat} > cassandra-topology.properties > {noformat}xxx.xxx.xxx.10=datacenter1:rack1 > xxx.xxx.xxx.11=datacenter1:rack1 > default=datacenter1:rack1{noformat} > Ring configuration: > {noformat}Address DC RackStatus State Load >Effective-Ownership Token > > 85070591730234615865843651857942052864 > xxx.xxx.xxx.10 datacenter1 rack1 Up Normal 23,11 kB > 100,00% 0 > xxx.xxx.xxx.11 datacenter1 rack1 Up Normal 23,25 kB > 100,00% 85070591730234615865843651857942052864{noformat} > h3.Problem > I have ctreated keyspace and column family using CLI commands: > {noformat}create keyspace testks with placement_strategy = > 'org.apache.cassandra.locator.NetworkTopologyStrategy' and strategy_options = > {datacenter1:2}; > use testks; > create column family testcf;{noformat} > Then I started my Java application, which inserts 50 000 000 rows to created > column family using Hector client. Client is connected to node 1. > After about 30 seconds (160 000 rows were inserted) Cassandra server on node > 1 throws an exception: > {noformat}ERROR [COMMIT-LOG-ALLOCATOR] 2012-06-13 10:26:38,393 > AbstractCassandraDaemon.java (line 134) Exception in thread > Thread[COMMIT-LOG-ALLOCATOR,5,main] > java.io.IOError: java.io.IOException: Rename from > c:\apache-cassandra\storage\commitlog\CommitLog-7345742389552.log to > 7475933520374 failed > at > org.apache.cassandra.db.commitlog.CommitLogSegment.(CommitLogSegment.java:127) > at > org.apache.cassandra.db.commitlog.CommitLogSegment.recycle(CommitLogSegment.java:204) > at > org.apache.cassandra.db.commitlog.CommitLogAllocator$2.run(CommitLogAllocator.java:166) > at > org.apache.cassandra.db.commitlog.CommitLogAllocator$1.runMayThrow(CommitLogAllocator.java:95) > at > org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30) > at java.lang.Thread.run(Thread.java:722) > Caused by: java.io.IOException: Rename from > c:\apache-cassandra\storage\commitlog\CommitLog-7345742389552.log to > 7475933520374 failed > at > org.apache.cassandra.db.commitlog.CommitLogSegment.(CommitLogSegment.java:105) > ... 5 more{noformat} > > Then, few seconds later Cassandra server on node 2 throws the same exception: > {noformat}ERROR [COMMIT-LOG-ALLOCATOR] 2012-06-14 10:26:44,005 > AbstractCassandraDaemon.java (line 134) Exception in thread > Thread[COMMIT-LOG-ALLOCATOR,5,main] > java.io.IOError: java.io.IOException: Rename from > c:\apache-cassandra\storage\commitlog\CommitLog-7320337904033.log to > 7437675489307 failed > at > org.apache.cassandra.db.commitlog.CommitLogSegment.(CommitLogSegment.java:127) > at > org.apache.cassandra.db.commitlog.CommitLogSegment.recycle(CommitLogSegment.java:204) > at > org.apache.cassandra.db.commitlog.CommitLogAllocator$2.run(CommitLogAllocator.java:166) > at > org.apache.cassandra.db.commitlog.CommitLogAllocator$1.runMayThrow(CommitLogAllocator.java:95) > at > org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30) > at java.lang.
[jira] [Updated] (CASSANDRA-4337) Data insertion fails because of commitlog rename failure
[ https://issues.apache.org/jira/browse/CASSANDRA-4337?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrycjusz Matuszak updated CASSANDRA-4337: --- Attachment: system-node2-stress-test.log system-node1-stress-test.log I've attached logs from test with Stress tool on Java 7 update 5. I had to run Stress tool twice to reproduce this bug. At first run no exception was thrown. > Data insertion fails because of commitlog rename failure > > > Key: CASSANDRA-4337 > URL: https://issues.apache.org/jira/browse/CASSANDRA-4337 > Project: Cassandra > Issue Type: Bug > Components: Core >Affects Versions: 1.1.1 > Environment: - Node 1: >Hardware: Intel Xeon 2.83 GHz (4 cores), 24GB RAM, Dell VIRTUAL DISK SCSI > 500GB >System: Windows Server 2008 R2 x64 >Java version: 7 update 4 x64 > - Node 2: > Hardware: Intel Xeon 2.83 GHz (4 cores), 8GB RAM, Dell VIRTUAL DISK SCSI > 500GB > System: Windows Server 2008 R2 x64 > Java version: 7 update 4 x64 >Reporter: Patrycjusz Matuszak > Labels: commitlog > Attachments: system-node1-stress-test.log, system-node1.log, > system-node2-stress-test.log, system-node2.log > > > h3. Configuration > Cassandra server configuration: > {noformat}heap size: 4 GB > seed_provider: > - class_name: org.apache.cassandra.locator.SimpleSeedProvider > parameters: > - seeds: "xxx.xxx.xxx.10,xxx.xxx.xxx.11" > listen_address: xxx.xxx.xxx.10 > rpc_address: 0.0.0.0 > rpc_port: 9160 > rpc_timeout_in_ms: 2 > endpoint_snitch: PropertyFileSnitch{noformat} > cassandra-topology.properties > {noformat}xxx.xxx.xxx.10=datacenter1:rack1 > xxx.xxx.xxx.11=datacenter1:rack1 > default=datacenter1:rack1{noformat} > Ring configuration: > {noformat}Address DC RackStatus State Load >Effective-Ownership Token > > 85070591730234615865843651857942052864 > xxx.xxx.xxx.10 datacenter1 rack1 Up Normal 23,11 kB > 100,00% 0 > xxx.xxx.xxx.11 datacenter1 rack1 Up Normal 23,25 kB > 100,00% 85070591730234615865843651857942052864{noformat} > h3.Problem > I have ctreated keyspace and column family using CLI commands: > {noformat}create keyspace testks with placement_strategy = > 'org.apache.cassandra.locator.NetworkTopologyStrategy' and strategy_options = > {datacenter1:2}; > use testks; > create column family testcf;{noformat} > Then I started my Java application, which inserts 50 000 000 rows to created > column family using Hector client. Client is connected to node 1. > After about 30 seconds (160 000 rows were inserted) Cassandra server on node > 1 throws an exception: > {noformat}ERROR [COMMIT-LOG-ALLOCATOR] 2012-06-13 10:26:38,393 > AbstractCassandraDaemon.java (line 134) Exception in thread > Thread[COMMIT-LOG-ALLOCATOR,5,main] > java.io.IOError: java.io.IOException: Rename from > c:\apache-cassandra\storage\commitlog\CommitLog-7345742389552.log to > 7475933520374 failed > at > org.apache.cassandra.db.commitlog.CommitLogSegment.(CommitLogSegment.java:127) > at > org.apache.cassandra.db.commitlog.CommitLogSegment.recycle(CommitLogSegment.java:204) > at > org.apache.cassandra.db.commitlog.CommitLogAllocator$2.run(CommitLogAllocator.java:166) > at > org.apache.cassandra.db.commitlog.CommitLogAllocator$1.runMayThrow(CommitLogAllocator.java:95) > at > org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30) > at java.lang.Thread.run(Thread.java:722) > Caused by: java.io.IOException: Rename from > c:\apache-cassandra\storage\commitlog\CommitLog-7345742389552.log to > 7475933520374 failed > at > org.apache.cassandra.db.commitlog.CommitLogSegment.(CommitLogSegment.java:105) > ... 5 more{noformat} > > Then, few seconds later Cassandra server on node 2 throws the same exception: > {noformat}ERROR [COMMIT-LOG-ALLOCATOR] 2012-06-14 10:26:44,005 > AbstractCassandraDaemon.java (line 134) Exception in thread > Thread[COMMIT-LOG-ALLOCATOR,5,main] > java.io.IOError: java.io.IOException: Rename from > c:\apache-cassandra\storage\commitlog\CommitLog-7320337904033.log to > 7437675489307 failed > at > org.apache.cassandra.db.commitlog.CommitLogSegment.(CommitLogSegment.java:127) > at > org.apache.cassandra.db.commitlog.CommitLogSegment.recycle(CommitLogSegment.java:204) > at > org.apache.cassandra.db.commitlog.CommitLogAllocator$2.run(CommitLogAllocator.java:166) > at > org.apache.cassandra.db.commitlog.CommitLogAllocator$1.runMayThrow(CommitLogAllocator.java:95) > at > org.apache.cassandra.utils.WrappedRunnable.run(W
[jira] [Commented] (CASSANDRA-4337) Data insertion fails because of commitlog rename failure
[ https://issues.apache.org/jira/browse/CASSANDRA-4337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13294834#comment-13294834 ] Patrycjusz Matuszak commented on CASSANDRA-4337: I don't think that it has anything to do with permissions. I'm running Cassandra as administrator. Last night I found a probable cause of the problem. I switched to Java 6 update 33 x64 on both machines and was able to perform all planed tests without any errors. It seems that this bug is related to Java version I used. Problem doesn't occur when using Java 6. > Data insertion fails because of commitlog rename failure > > > Key: CASSANDRA-4337 > URL: https://issues.apache.org/jira/browse/CASSANDRA-4337 > Project: Cassandra > Issue Type: Bug > Components: Core >Affects Versions: 1.1.1 > Environment: - Node 1: >Hardware: Intel Xeon 2.83 GHz (4 cores), 24GB RAM, Dell VIRTUAL DISK SCSI > 500GB >System: Windows Server 2008 R2 x64 >Java version: 7 update 4 x64 > - Node 2: > Hardware: Intel Xeon 2.83 GHz (4 cores), 8GB RAM, Dell VIRTUAL DISK SCSI > 500GB > System: Windows Server 2008 R2 x64 > Java version: 7 update 4 x64 >Reporter: Patrycjusz Matuszak > Labels: commitlog > Attachments: system-node1.log, system-node2.log > > > h3. Configuration > Cassandra server configuration: > {noformat}heap size: 4 GB > seed_provider: > - class_name: org.apache.cassandra.locator.SimpleSeedProvider > parameters: > - seeds: "xxx.xxx.xxx.10,xxx.xxx.xxx.11" > listen_address: xxx.xxx.xxx.10 > rpc_address: 0.0.0.0 > rpc_port: 9160 > rpc_timeout_in_ms: 2 > endpoint_snitch: PropertyFileSnitch{noformat} > cassandra-topology.properties > {noformat}xxx.xxx.xxx.10=datacenter1:rack1 > xxx.xxx.xxx.11=datacenter1:rack1 > default=datacenter1:rack1{noformat} > Ring configuration: > {noformat}Address DC RackStatus State Load >Effective-Ownership Token > > 85070591730234615865843651857942052864 > xxx.xxx.xxx.10 datacenter1 rack1 Up Normal 23,11 kB > 100,00% 0 > xxx.xxx.xxx.11 datacenter1 rack1 Up Normal 23,25 kB > 100,00% 85070591730234615865843651857942052864{noformat} > h3.Problem > I have ctreated keyspace and column family using CLI commands: > {noformat}create keyspace testks with placement_strategy = > 'org.apache.cassandra.locator.NetworkTopologyStrategy' and strategy_options = > {datacenter1:2}; > use testks; > create column family testcf;{noformat} > Then I started my Java application, which inserts 50 000 000 rows to created > column family using Hector client. Client is connected to node 1. > After about 30 seconds (160 000 rows were inserted) Cassandra server on node > 1 throws an exception: > {noformat}ERROR [COMMIT-LOG-ALLOCATOR] 2012-06-13 10:26:38,393 > AbstractCassandraDaemon.java (line 134) Exception in thread > Thread[COMMIT-LOG-ALLOCATOR,5,main] > java.io.IOError: java.io.IOException: Rename from > c:\apache-cassandra\storage\commitlog\CommitLog-7345742389552.log to > 7475933520374 failed > at > org.apache.cassandra.db.commitlog.CommitLogSegment.(CommitLogSegment.java:127) > at > org.apache.cassandra.db.commitlog.CommitLogSegment.recycle(CommitLogSegment.java:204) > at > org.apache.cassandra.db.commitlog.CommitLogAllocator$2.run(CommitLogAllocator.java:166) > at > org.apache.cassandra.db.commitlog.CommitLogAllocator$1.runMayThrow(CommitLogAllocator.java:95) > at > org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30) > at java.lang.Thread.run(Thread.java:722) > Caused by: java.io.IOException: Rename from > c:\apache-cassandra\storage\commitlog\CommitLog-7345742389552.log to > 7475933520374 failed > at > org.apache.cassandra.db.commitlog.CommitLogSegment.(CommitLogSegment.java:105) > ... 5 more{noformat} > > Then, few seconds later Cassandra server on node 2 throws the same exception: > {noformat}ERROR [COMMIT-LOG-ALLOCATOR] 2012-06-14 10:26:44,005 > AbstractCassandraDaemon.java (line 134) Exception in thread > Thread[COMMIT-LOG-ALLOCATOR,5,main] > java.io.IOError: java.io.IOException: Rename from > c:\apache-cassandra\storage\commitlog\CommitLog-7320337904033.log to > 7437675489307 failed > at > org.apache.cassandra.db.commitlog.CommitLogSegment.(CommitLogSegment.java:127) > at > org.apache.cassandra.db.commitlog.CommitLogSegment.recycle(CommitLogSegment.java:204) > at > org.apache.cassandra.db.commitlog.CommitLogAllocator$2.run(CommitLogAllocator.java:166) > at > org.apache.cassandra.db.commitlog.CommitLogAll
[jira] [Updated] (CASSANDRA-4337) Data insertion fails because of commitlog rename failure
[ https://issues.apache.org/jira/browse/CASSANDRA-4337?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrycjusz Matuszak updated CASSANDRA-4337: --- Description: h3. Configuration Cassandra server configuration: {noformat}heap size: 4 GB seed_provider: - class_name: org.apache.cassandra.locator.SimpleSeedProvider parameters: - seeds: "xxx.xxx.xxx.10,xxx.xxx.xxx.11" listen_address: xxx.xxx.xxx.10 rpc_address: 0.0.0.0 rpc_port: 9160 rpc_timeout_in_ms: 2 endpoint_snitch: PropertyFileSnitch{noformat} cassandra-topology.properties {noformat}xxx.xxx.xxx.10=datacenter1:rack1 xxx.xxx.xxx.11=datacenter1:rack1 default=datacenter1:rack1{noformat} Ring configuration: {noformat}Address DC RackStatus State Load Effective-Ownership Token 85070591730234615865843651857942052864 xxx.xxx.xxx.10 datacenter1 rack1 Up Normal 23,11 kB100,00% 0 xxx.xxx.xxx.11 datacenter1 rack1 Up Normal 23,25 kB100,00% 85070591730234615865843651857942052864{noformat} h3.Problem I have ctreated keyspace and column family using CLI commands: {noformat}create keyspace testks with placement_strategy = 'org.apache.cassandra.locator.NetworkTopologyStrategy' and strategy_options = {datacenter1:2}; use testks; create column family testcf;{noformat} Then I started my Java application, which inserts 50 000 000 rows to created column family using Hector client. Client is connected to node 1. After about 30 seconds (160 000 rows were inserted) Cassandra server on node 1 throws an exception: {noformat}ERROR [COMMIT-LOG-ALLOCATOR] 2012-06-13 10:26:38,393 AbstractCassandraDaemon.java (line 134) Exception in thread Thread[COMMIT-LOG-ALLOCATOR,5,main] java.io.IOError: java.io.IOException: Rename from c:\apache-cassandra\storage\commitlog\CommitLog-7345742389552.log to 7475933520374 failed at org.apache.cassandra.db.commitlog.CommitLogSegment.(CommitLogSegment.java:127) at org.apache.cassandra.db.commitlog.CommitLogSegment.recycle(CommitLogSegment.java:204) at org.apache.cassandra.db.commitlog.CommitLogAllocator$2.run(CommitLogAllocator.java:166) at org.apache.cassandra.db.commitlog.CommitLogAllocator$1.runMayThrow(CommitLogAllocator.java:95) at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30) at java.lang.Thread.run(Thread.java:722) Caused by: java.io.IOException: Rename from c:\apache-cassandra\storage\commitlog\CommitLog-7345742389552.log to 7475933520374 failed at org.apache.cassandra.db.commitlog.CommitLogSegment.(CommitLogSegment.java:105) ... 5 more{noformat} Then, few seconds later Cassandra server on node 2 throws the same exception: {noformat}ERROR [COMMIT-LOG-ALLOCATOR] 2012-06-14 10:26:44,005 AbstractCassandraDaemon.java (line 134) Exception in thread Thread[COMMIT-LOG-ALLOCATOR,5,main] java.io.IOError: java.io.IOException: Rename from c:\apache-cassandra\storage\commitlog\CommitLog-7320337904033.log to 7437675489307 failed at org.apache.cassandra.db.commitlog.CommitLogSegment.(CommitLogSegment.java:127) at org.apache.cassandra.db.commitlog.CommitLogSegment.recycle(CommitLogSegment.java:204) at org.apache.cassandra.db.commitlog.CommitLogAllocator$2.run(CommitLogAllocator.java:166) at org.apache.cassandra.db.commitlog.CommitLogAllocator$1.runMayThrow(CommitLogAllocator.java:95) at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30) at java.lang.Thread.run(Unknown Source) Caused by: java.io.IOException: Rename from c:\apache-cassandra\storage\commitlog\CommitLog-7320337904033.log to 7437675489307 failed at org.apache.cassandra.db.commitlog.CommitLogSegment.(CommitLogSegment.java:105) ... 5 more{noformat} After that, my application cannot insert any more data. Hector gets TimedOutException from Thrift: {noformat}Thread-4 HConnectionManager.java 306 2012-06-14 10:26:56,034 HConnectionManager operateWithFailover WARN%Could not fullfill request on this host CassandraClient Thread-4 HConnectionManager.java 307 2012-06-14 10:26:56,034 HConnectionManager operateWithFailover WARN%Exception: me.prettyprint.hector.api.exceptions.HTimedOutException: TimedOutException() at me.prettyprint.cassandra.service.ExceptionsTranslatorImpl.translate(ExceptionsTranslatorImpl.java:35) at me.prettyprint.cassandra.connection.HConnectionManager.operateWithFailover(HConnectionManager.java:264) at me.prettyprint.cassandra.model.ExecutingKeyspace.doExecuteOperation(ExecutingKeyspace.java:97) at me.prettyprint.cassandra.model.MutatorImpl.execute(MutatorImpl.java:243) at patrycjusz.nosqltest.db.cassandra.CassandraHect
[jira] [Updated] (CASSANDRA-4337) Data insertion fails because of commitlog rename failure
[ https://issues.apache.org/jira/browse/CASSANDRA-4337?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrycjusz Matuszak updated CASSANDRA-4337: --- Environment: - Node 1: Hardware: Intel Xeon 2.83 GHz (4 cores), 24GB RAM, Dell VIRTUAL DISK SCSI 500GB System: Windows Server 2008 R2 x64 Java version: 7 update 4 x64 - Node 2: Hardware: Intel Xeon 2.83 GHz (4 cores), 8GB RAM, Dell VIRTUAL DISK SCSI 500GB System: Windows Server 2008 R2 x64 Java version: 7 update 4 x64 was: - Node 1: Hardware: Intel Xeon 2.83 GHz (4 cores), 24GB RAM, Dell VIRTUAL DISK SCSI 500GB System: Windows Server 2008 R2 x64 Java version: 7 update 4 x64 - Node 2: Hardware: Intel Xeon 2.83 GHz (4 cores), 8GB RAM, Dell VIRTUAL DISK SCSI 500GB System: Windows Server 2008 R2 x64 Java version: 7 update 4 x64 > Data insertion fails because of commitlog rename failure > > > Key: CASSANDRA-4337 > URL: https://issues.apache.org/jira/browse/CASSANDRA-4337 > Project: Cassandra > Issue Type: Bug > Components: Core >Affects Versions: 1.1.1 > Environment: - Node 1: >Hardware: Intel Xeon 2.83 GHz (4 cores), 24GB RAM, Dell VIRTUAL DISK SCSI > 500GB >System: Windows Server 2008 R2 x64 >Java version: 7 update 4 x64 > - Node 2: > Hardware: Intel Xeon 2.83 GHz (4 cores), 8GB RAM, Dell VIRTUAL DISK SCSI > 500GB > System: Windows Server 2008 R2 x64 > Java version: 7 update 4 x64 >Reporter: Patrycjusz Matuszak > Attachments: system-node1.log, system-node2.log > > > h3. Configuration > Cassandra server configuration: > {noformat}heap size: 4 GB > seed_provider: > - class_name: org.apache.cassandra.locator.SimpleSeedProvider > parameters: > - seeds: "xxx.xxx.xxx.10,xxx.xxx.xxx.11" > listen_address: xxx.xxx.xxx.10 > rpc_address: 0.0.0.0 > rpc_port: 9160 > rpc_timeout_in_ms: 2 > endpoint_snitch: PropertyFileSnitch{noformat} > cassandra-topology.properties > {noformat}xxx.xxx.xxx.10=datacenter1:rack1 > xxx.xxx.xxx.11=datacenter1:rack1 > default=datacenter1:rack1{noformat} > Ring configuration: > {noformat}Address DC RackStatus State Load >Effective-Ownership Token > > 85070591730234615865843651857942052864 > xxx.xxx.xxx.10 datacenter1 rack1 Up Normal 23,11 kB > 100,00% 0 > xxx.xxx.xxx.11 datacenter1 rack1 Up Normal 23,25 kB > 100,00% 85070591730234615865843651857942052864{noformat} > h3.Problem > I have ctreated keyspace and column family using CLI commands: > {noformat}create keyspace testks with placement_strategy = > 'org.apache.cassandra.locator.NetworkTopologyStrategy' and strategy_options = > {datacenter1:2}; > use testks; > create column family testcf;{noformat} > Then I started my Java application, which inserts 50 000 000 rows to created > column family using Hector client. Client is connected to node 1. > After about 30 seconds (160 000 rows were inserted) Cassandra server on node > 1 throws an exception: > {noformat}ERROR [COMMIT-LOG-ALLOCATOR] 2012-06-13 10:26:38,393 > AbstractCassandraDaemon.java (line 134) Exception in thread > Thread[COMMIT-LOG-ALLOCATOR,5,main] > java.io.IOError: java.io.IOException: Rename from > c:\apache-cassandra\storage\commitlog\CommitLog-7345742389552.log to > 7475933520374 failed > at > org.apache.cassandra.db.commitlog.CommitLogSegment.(CommitLogSegment.java:127) > at > org.apache.cassandra.db.commitlog.CommitLogSegment.recycle(CommitLogSegment.java:204) > at > org.apache.cassandra.db.commitlog.CommitLogAllocator$2.run(CommitLogAllocator.java:166) > at > org.apache.cassandra.db.commitlog.CommitLogAllocator$1.runMayThrow(CommitLogAllocator.java:95) > at > org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30) > at java.lang.Thread.run(Thread.java:722) > Caused by: java.io.IOException: Rename from > c:\apache-cassandra\storage\commitlog\CommitLog-7345742389552.log to > 7475933520374 failed > at > org.apache.cassandra.db.commitlog.CommitLogSegment.(CommitLogSegment.java:105) > ... 5 more{noformat} > > Then, few seconds later Cassandra server on node 2 throws the same exception: > {noformat}ERROR [COMMIT-LOG-ALLOCATOR] 2012-06-14 10:26:44,005 > AbstractCassandraDaemon.java (line 134) Exception in thread > Thread[COMMIT-LOG-ALLOCATOR,5,main] > java.io.IOError: java.io.IOException: Rename from > c:\apache-cassandra\storage\commitlog\CommitLog-7320337904033.log to > 7437675489307 failed > at > org.apache.cassandra.db.commitlog.CommitLogSegment.(CommitLogSegment.java:127) > at > org.apache.cassandra.
[jira] [Comment Edited] (CASSANDRA-4337) Data insertion fails because of commitlog rename failure
[ https://issues.apache.org/jira/browse/CASSANDRA-4337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13294294#comment-13294294 ] Patrycjusz Matuszak edited comment on CASSANDRA-4337 at 6/13/12 9:38 AM: - Attached Cassandra server logs was (Author: patrycjusz): Cassandra server log from node 1 > Data insertion fails because of commitlog rename failure > > > Key: CASSANDRA-4337 > URL: https://issues.apache.org/jira/browse/CASSANDRA-4337 > Project: Cassandra > Issue Type: Bug > Components: Core >Affects Versions: 1.1.1 > Environment: - Node 1: >Hardware: Intel Xeon 2.83 GHz (4 cores), 24GB RAM, Dell VIRTUAL DISK SCSI > 500GB >System: Windows Server 2008 R2 x64 >Java version: 7 update 4 x64 > - Node 2: > Hardware: Intel Xeon 2.83 GHz (4 cores), 8GB RAM, Dell VIRTUAL DISK SCSI > 500GB > System: Windows Server 2008 R2 x64 > Java version: 7 update 4 x64 >Reporter: Patrycjusz Matuszak > Attachments: system-node1.log, system-node2.log > > > h3. Configuration > Cassandra server configuration: > {noformat}heap size: 4 GB > seed_provider: > - class_name: org.apache.cassandra.locator.SimpleSeedProvider > parameters: > - seeds: "xxx.xxx.xxx.10,xxx.xxx.xxx.11" > listen_address: xxx.xxx.xxx.10 > rpc_address: 0.0.0.0 > rpc_port: 9160 > rpc_timeout_in_ms: 2 > endpoint_snitch: PropertyFileSnitch{noformat} > cassandra-topology.properties > {noformat}xxx.xxx.xxx.10=datacenter1:rack1 > xxx.xxx.xxx.11=datacenter1:rack1 > default=datacenter1:rack1{noformat} > Ring configuration: > {noformat}Address DC RackStatus State Load >Effective-Ownership Token > > 85070591730234615865843651857942052864 > xxx.xxx.xxx.10 datacenter1 rack1 Up Normal 23,11 kB > 100,00% 0 > xxx.xxx.xxx.11 datacenter1 rack1 Up Normal 23,25 kB > 100,00% 85070591730234615865843651857942052864{noformat} > h3.Problem > I have ctreated keyspace and column family using CLI commands: > {noformat}create keyspace testks with placement_strategy = > 'org.apache.cassandra.locator.NetworkTopologyStrategy' and strategy_options = > {datacenter1:2}; > use testks; > create column family testcf;{noformat} > Then I started my Java application, which inserts 50 000 000 rows to created > column family using Hector client. Client is connected to node 1. > After about 30 seconds (160 000 rows were inserted) Cassandra server on node > 1 throws an exception: > {noformat}ERROR [COMMIT-LOG-ALLOCATOR] 2012-06-13 10:26:38,393 > AbstractCassandraDaemon.java (line 134) Exception in thread > Thread[COMMIT-LOG-ALLOCATOR,5,main] > java.io.IOError: java.io.IOException: Rename from > c:\apache-cassandra\storage\commitlog\CommitLog-7345742389552.log to > 7475933520374 failed > at > org.apache.cassandra.db.commitlog.CommitLogSegment.(CommitLogSegment.java:127) > at > org.apache.cassandra.db.commitlog.CommitLogSegment.recycle(CommitLogSegment.java:204) > at > org.apache.cassandra.db.commitlog.CommitLogAllocator$2.run(CommitLogAllocator.java:166) > at > org.apache.cassandra.db.commitlog.CommitLogAllocator$1.runMayThrow(CommitLogAllocator.java:95) > at > org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30) > at java.lang.Thread.run(Thread.java:722) > Caused by: java.io.IOException: Rename from > c:\apache-cassandra\storage\commitlog\CommitLog-7345742389552.log to > 7475933520374 failed > at > org.apache.cassandra.db.commitlog.CommitLogSegment.(CommitLogSegment.java:105) > ... 5 more{noformat} > > Then, few seconds later Cassandra server on node 2 throws the same exception: > {noformat}ERROR [COMMIT-LOG-ALLOCATOR] 2012-06-14 10:26:44,005 > AbstractCassandraDaemon.java (line 134) Exception in thread > Thread[COMMIT-LOG-ALLOCATOR,5,main] > java.io.IOError: java.io.IOException: Rename from > c:\apache-cassandra\storage\commitlog\CommitLog-7320337904033.log to > 7437675489307 failed > at > org.apache.cassandra.db.commitlog.CommitLogSegment.(CommitLogSegment.java:127) > at > org.apache.cassandra.db.commitlog.CommitLogSegment.recycle(CommitLogSegment.java:204) > at > org.apache.cassandra.db.commitlog.CommitLogAllocator$2.run(CommitLogAllocator.java:166) > at > org.apache.cassandra.db.commitlog.CommitLogAllocator$1.runMayThrow(CommitLogAllocator.java:95) > at > org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30) > at java.lang.Thread.run(Unknown Source) > Caused by: java.io.IOException: Rename from >
[jira] [Updated] (CASSANDRA-4337) Data insertion fails because of commitlog rename failure
[ https://issues.apache.org/jira/browse/CASSANDRA-4337?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrycjusz Matuszak updated CASSANDRA-4337: --- Comment: was deleted (was: Cassandra server log from node 2) > Data insertion fails because of commitlog rename failure > > > Key: CASSANDRA-4337 > URL: https://issues.apache.org/jira/browse/CASSANDRA-4337 > Project: Cassandra > Issue Type: Bug > Components: Core >Affects Versions: 1.1.1 > Environment: - Node 1: >Hardware: Intel Xeon 2.83 GHz (4 cores), 24GB RAM, Dell VIRTUAL DISK SCSI > 500GB >System: Windows Server 2008 R2 x64 >Java version: 7 update 4 x64 > - Node 2: > Hardware: Intel Xeon 2.83 GHz (4 cores), 8GB RAM, Dell VIRTUAL DISK SCSI > 500GB > System: Windows Server 2008 R2 x64 > Java version: 7 update 4 x64 >Reporter: Patrycjusz Matuszak > Attachments: system-node1.log, system-node2.log > > > h3. Configuration > Cassandra server configuration: > {noformat}heap size: 4 GB > seed_provider: > - class_name: org.apache.cassandra.locator.SimpleSeedProvider > parameters: > - seeds: "xxx.xxx.xxx.10,xxx.xxx.xxx.11" > listen_address: xxx.xxx.xxx.10 > rpc_address: 0.0.0.0 > rpc_port: 9160 > rpc_timeout_in_ms: 2 > endpoint_snitch: PropertyFileSnitch{noformat} > cassandra-topology.properties > {noformat}xxx.xxx.xxx.10=datacenter1:rack1 > xxx.xxx.xxx.11=datacenter1:rack1 > default=datacenter1:rack1{noformat} > Ring configuration: > {noformat}Address DC RackStatus State Load >Effective-Ownership Token > > 85070591730234615865843651857942052864 > xxx.xxx.xxx.10 datacenter1 rack1 Up Normal 23,11 kB > 100,00% 0 > xxx.xxx.xxx.11 datacenter1 rack1 Up Normal 23,25 kB > 100,00% 85070591730234615865843651857942052864{noformat} > h3.Problem > I have ctreated keyspace and column family using CLI commands: > {noformat}create keyspace testks with placement_strategy = > 'org.apache.cassandra.locator.NetworkTopologyStrategy' and strategy_options = > {datacenter1:2}; > use testks; > create column family testcf;{noformat} > Then I started my Java application, which inserts 50 000 000 rows to created > column family using Hector client. Client is connected to node 1. > After about 30 seconds (160 000 rows were inserted) Cassandra server on node > 1 throws an exception: > {noformat}ERROR [COMMIT-LOG-ALLOCATOR] 2012-06-13 10:26:38,393 > AbstractCassandraDaemon.java (line 134) Exception in thread > Thread[COMMIT-LOG-ALLOCATOR,5,main] > java.io.IOError: java.io.IOException: Rename from > c:\apache-cassandra\storage\commitlog\CommitLog-7345742389552.log to > 7475933520374 failed > at > org.apache.cassandra.db.commitlog.CommitLogSegment.(CommitLogSegment.java:127) > at > org.apache.cassandra.db.commitlog.CommitLogSegment.recycle(CommitLogSegment.java:204) > at > org.apache.cassandra.db.commitlog.CommitLogAllocator$2.run(CommitLogAllocator.java:166) > at > org.apache.cassandra.db.commitlog.CommitLogAllocator$1.runMayThrow(CommitLogAllocator.java:95) > at > org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30) > at java.lang.Thread.run(Thread.java:722) > Caused by: java.io.IOException: Rename from > c:\apache-cassandra\storage\commitlog\CommitLog-7345742389552.log to > 7475933520374 failed > at > org.apache.cassandra.db.commitlog.CommitLogSegment.(CommitLogSegment.java:105) > ... 5 more{noformat} > > Then, few seconds later Cassandra server on node 2 throws the same exception: > {noformat}ERROR [COMMIT-LOG-ALLOCATOR] 2012-06-14 10:26:44,005 > AbstractCassandraDaemon.java (line 134) Exception in thread > Thread[COMMIT-LOG-ALLOCATOR,5,main] > java.io.IOError: java.io.IOException: Rename from > c:\apache-cassandra\storage\commitlog\CommitLog-7320337904033.log to > 7437675489307 failed > at > org.apache.cassandra.db.commitlog.CommitLogSegment.(CommitLogSegment.java:127) > at > org.apache.cassandra.db.commitlog.CommitLogSegment.recycle(CommitLogSegment.java:204) > at > org.apache.cassandra.db.commitlog.CommitLogAllocator$2.run(CommitLogAllocator.java:166) > at > org.apache.cassandra.db.commitlog.CommitLogAllocator$1.runMayThrow(CommitLogAllocator.java:95) > at > org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30) > at java.lang.Thread.run(Unknown Source) > Caused by: java.io.IOException: Rename from > c:\apache-cassandra\storage\commitlog\CommitLog-7320337904033.log to > 7437675489307 failed > at > org.apache.cassandra.db.commitlog.CommitLogSegment.(CommitLogSeg
[jira] [Updated] (CASSANDRA-4337) Data insertion fails because of commitlog rename failure
[ https://issues.apache.org/jira/browse/CASSANDRA-4337?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrycjusz Matuszak updated CASSANDRA-4337: --- Attachment: system-node1.log Cassandra server log from node 1 > Data insertion fails because of commitlog rename failure > > > Key: CASSANDRA-4337 > URL: https://issues.apache.org/jira/browse/CASSANDRA-4337 > Project: Cassandra > Issue Type: Bug > Components: Core >Affects Versions: 1.1.1 > Environment: - Node 1: >Hardware: Intel Xeon 2.83 GHz (4 cores), 24GB RAM, Dell VIRTUAL DISK SCSI > 500GB >System: Windows Server 2008 R2 x64 >Java version: 7 update 4 x64 > - Node 2: > Hardware: Intel Xeon 2.83 GHz (4 cores), 8GB RAM, Dell VIRTUAL DISK SCSI > 500GB > System: Windows Server 2008 R2 x64 > Java version: 7 update 4 x64 >Reporter: Patrycjusz Matuszak > Attachments: system-node1.log, system-node2.log > > > h3. Configuration > Cassandra server configuration: > {noformat}heap size: 4 GB > seed_provider: > - class_name: org.apache.cassandra.locator.SimpleSeedProvider > parameters: > - seeds: "xxx.xxx.xxx.10,xxx.xxx.xxx.11" > listen_address: xxx.xxx.xxx.10 > rpc_address: 0.0.0.0 > rpc_port: 9160 > rpc_timeout_in_ms: 2 > endpoint_snitch: PropertyFileSnitch{noformat} > cassandra-topology.properties > {noformat}xxx.xxx.xxx.10=datacenter1:rack1 > xxx.xxx.xxx.11=datacenter1:rack1 > default=datacenter1:rack1{noformat} > Ring configuration: > {noformat}Address DC RackStatus State Load >Effective-Ownership Token > > 85070591730234615865843651857942052864 > xxx.xxx.xxx.10 datacenter1 rack1 Up Normal 23,11 kB > 100,00% 0 > xxx.xxx.xxx.11 datacenter1 rack1 Up Normal 23,25 kB > 100,00% 85070591730234615865843651857942052864{noformat} > h3.Problem > I have ctreated keyspace and column family using CLI commands: > {noformat}create keyspace testks with placement_strategy = > 'org.apache.cassandra.locator.NetworkTopologyStrategy' and strategy_options = > {datacenter1:2}; > use testks; > create column family testcf;{noformat} > Then I started my Java application, which inserts 50 000 000 rows to created > column family using Hector client. Client is connected to node 1. > After about 30 seconds (160 000 rows were inserted) Cassandra server on node > 1 throws an exception: > {noformat}ERROR [COMMIT-LOG-ALLOCATOR] 2012-06-13 10:26:38,393 > AbstractCassandraDaemon.java (line 134) Exception in thread > Thread[COMMIT-LOG-ALLOCATOR,5,main] > java.io.IOError: java.io.IOException: Rename from > c:\apache-cassandra\storage\commitlog\CommitLog-7345742389552.log to > 7475933520374 failed > at > org.apache.cassandra.db.commitlog.CommitLogSegment.(CommitLogSegment.java:127) > at > org.apache.cassandra.db.commitlog.CommitLogSegment.recycle(CommitLogSegment.java:204) > at > org.apache.cassandra.db.commitlog.CommitLogAllocator$2.run(CommitLogAllocator.java:166) > at > org.apache.cassandra.db.commitlog.CommitLogAllocator$1.runMayThrow(CommitLogAllocator.java:95) > at > org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30) > at java.lang.Thread.run(Thread.java:722) > Caused by: java.io.IOException: Rename from > c:\apache-cassandra\storage\commitlog\CommitLog-7345742389552.log to > 7475933520374 failed > at > org.apache.cassandra.db.commitlog.CommitLogSegment.(CommitLogSegment.java:105) > ... 5 more{noformat} > > Then, few seconds later Cassandra server on node 2 throws the same exception: > {noformat}ERROR [COMMIT-LOG-ALLOCATOR] 2012-06-14 10:26:44,005 > AbstractCassandraDaemon.java (line 134) Exception in thread > Thread[COMMIT-LOG-ALLOCATOR,5,main] > java.io.IOError: java.io.IOException: Rename from > c:\apache-cassandra\storage\commitlog\CommitLog-7320337904033.log to > 7437675489307 failed > at > org.apache.cassandra.db.commitlog.CommitLogSegment.(CommitLogSegment.java:127) > at > org.apache.cassandra.db.commitlog.CommitLogSegment.recycle(CommitLogSegment.java:204) > at > org.apache.cassandra.db.commitlog.CommitLogAllocator$2.run(CommitLogAllocator.java:166) > at > org.apache.cassandra.db.commitlog.CommitLogAllocator$1.runMayThrow(CommitLogAllocator.java:95) > at > org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30) > at java.lang.Thread.run(Unknown Source) > Caused by: java.io.IOException: Rename from > c:\apache-cassandra\storage\commitlog\CommitLog-7320337904033.log to > 7437675489307 failed > at > org.apache.cassandra.db.commitlog.CommitLogSegment.
[jira] [Updated] (CASSANDRA-4337) Data insertion fails because of commitlog rename failure
[ https://issues.apache.org/jira/browse/CASSANDRA-4337?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrycjusz Matuszak updated CASSANDRA-4337: --- Attachment: system-node2.log Cassandra server log from node 2 > Data insertion fails because of commitlog rename failure > > > Key: CASSANDRA-4337 > URL: https://issues.apache.org/jira/browse/CASSANDRA-4337 > Project: Cassandra > Issue Type: Bug > Components: Core >Affects Versions: 1.1.1 > Environment: - Node 1: >Hardware: Intel Xeon 2.83 GHz (4 cores), 24GB RAM, Dell VIRTUAL DISK SCSI > 500GB >System: Windows Server 2008 R2 x64 >Java version: 7 update 4 x64 > - Node 2: > Hardware: Intel Xeon 2.83 GHz (4 cores), 8GB RAM, Dell VIRTUAL DISK SCSI > 500GB > System: Windows Server 2008 R2 x64 > Java version: 7 update 4 x64 >Reporter: Patrycjusz Matuszak > Attachments: system-node1.log, system-node2.log > > > h3. Configuration > Cassandra server configuration: > {noformat}heap size: 4 GB > seed_provider: > - class_name: org.apache.cassandra.locator.SimpleSeedProvider > parameters: > - seeds: "xxx.xxx.xxx.10,xxx.xxx.xxx.11" > listen_address: xxx.xxx.xxx.10 > rpc_address: 0.0.0.0 > rpc_port: 9160 > rpc_timeout_in_ms: 2 > endpoint_snitch: PropertyFileSnitch{noformat} > cassandra-topology.properties > {noformat}xxx.xxx.xxx.10=datacenter1:rack1 > xxx.xxx.xxx.11=datacenter1:rack1 > default=datacenter1:rack1{noformat} > Ring configuration: > {noformat}Address DC RackStatus State Load >Effective-Ownership Token > > 85070591730234615865843651857942052864 > xxx.xxx.xxx.10 datacenter1 rack1 Up Normal 23,11 kB > 100,00% 0 > xxx.xxx.xxx.11 datacenter1 rack1 Up Normal 23,25 kB > 100,00% 85070591730234615865843651857942052864{noformat} > h3.Problem > I have ctreated keyspace and column family using CLI commands: > {noformat}create keyspace testks with placement_strategy = > 'org.apache.cassandra.locator.NetworkTopologyStrategy' and strategy_options = > {datacenter1:2}; > use testks; > create column family testcf;{noformat} > Then I started my Java application, which inserts 50 000 000 rows to created > column family using Hector client. Client is connected to node 1. > After about 30 seconds (160 000 rows were inserted) Cassandra server on node > 1 throws an exception: > {noformat}ERROR [COMMIT-LOG-ALLOCATOR] 2012-06-13 10:26:38,393 > AbstractCassandraDaemon.java (line 134) Exception in thread > Thread[COMMIT-LOG-ALLOCATOR,5,main] > java.io.IOError: java.io.IOException: Rename from > c:\apache-cassandra\storage\commitlog\CommitLog-7345742389552.log to > 7475933520374 failed > at > org.apache.cassandra.db.commitlog.CommitLogSegment.(CommitLogSegment.java:127) > at > org.apache.cassandra.db.commitlog.CommitLogSegment.recycle(CommitLogSegment.java:204) > at > org.apache.cassandra.db.commitlog.CommitLogAllocator$2.run(CommitLogAllocator.java:166) > at > org.apache.cassandra.db.commitlog.CommitLogAllocator$1.runMayThrow(CommitLogAllocator.java:95) > at > org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30) > at java.lang.Thread.run(Thread.java:722) > Caused by: java.io.IOException: Rename from > c:\apache-cassandra\storage\commitlog\CommitLog-7345742389552.log to > 7475933520374 failed > at > org.apache.cassandra.db.commitlog.CommitLogSegment.(CommitLogSegment.java:105) > ... 5 more{noformat} > > Then, few seconds later Cassandra server on node 2 throws the same exception: > {noformat}ERROR [COMMIT-LOG-ALLOCATOR] 2012-06-14 10:26:44,005 > AbstractCassandraDaemon.java (line 134) Exception in thread > Thread[COMMIT-LOG-ALLOCATOR,5,main] > java.io.IOError: java.io.IOException: Rename from > c:\apache-cassandra\storage\commitlog\CommitLog-7320337904033.log to > 7437675489307 failed > at > org.apache.cassandra.db.commitlog.CommitLogSegment.(CommitLogSegment.java:127) > at > org.apache.cassandra.db.commitlog.CommitLogSegment.recycle(CommitLogSegment.java:204) > at > org.apache.cassandra.db.commitlog.CommitLogAllocator$2.run(CommitLogAllocator.java:166) > at > org.apache.cassandra.db.commitlog.CommitLogAllocator$1.runMayThrow(CommitLogAllocator.java:95) > at > org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30) > at java.lang.Thread.run(Unknown Source) > Caused by: java.io.IOException: Rename from > c:\apache-cassandra\storage\commitlog\CommitLog-7320337904033.log to > 7437675489307 failed > at > org.apache.cassandra.db.commitlog.CommitLogSegment.
[jira] [Created] (CASSANDRA-4337) Data insertion fails because of commitlog rename failure
Patrycjusz Matuszak created CASSANDRA-4337: -- Summary: Data insertion fails because of commitlog rename failure Key: CASSANDRA-4337 URL: https://issues.apache.org/jira/browse/CASSANDRA-4337 Project: Cassandra Issue Type: Bug Components: Core Affects Versions: 1.1.1 Environment: - Node 1: Hardware: Intel Xeon 2.83 GHz (4 cores), 24GB RAM, Dell VIRTUAL DISK SCSI 500GB System: Windows Server 2008 R2 x64 Java version: 7 update 4 x64 - Node 2: Hardware: Intel Xeon 2.83 GHz (4 cores), 8GB RAM, Dell VIRTUAL DISK SCSI 500GB System: Windows Server 2008 R2 x64 Java version: 7 update 4 x64 Reporter: Patrycjusz Matuszak h3. Configuration Cassandra server configuration: {noformat}heap size: 4 GB seed_provider: - class_name: org.apache.cassandra.locator.SimpleSeedProvider parameters: - seeds: "xxx.xxx.xxx.10,xxx.xxx.xxx.11" listen_address: xxx.xxx.xxx.10 rpc_address: 0.0.0.0 rpc_port: 9160 rpc_timeout_in_ms: 2 endpoint_snitch: PropertyFileSnitch{noformat} cassandra-topology.properties {noformat}xxx.xxx.xxx.10=datacenter1:rack1 xxx.xxx.xxx.11=datacenter1:rack1 default=datacenter1:rack1{noformat} Ring configuration: {noformat}Address DC RackStatus State Load Effective-Ownership Token 85070591730234615865843651857942052864 xxx.xxx.xxx.10 datacenter1 rack1 Up Normal 23,11 kB100,00% 0 xxx.xxx.xxx.11 datacenter1 rack1 Up Normal 23,25 kB100,00% 85070591730234615865843651857942052864{noformat} h3.Problem I have ctreated keyspace and column family using CLI commands: {noformat}create keyspace testks with placement_strategy = 'org.apache.cassandra.locator.NetworkTopologyStrategy' and strategy_options = {datacenter1:2}; use testks; create column family testcf;{noformat} Then I started my Java application, which inserts 50 000 000 rows to created column family using Hector client. Client is connected to node 1. After about 30 seconds (160 000 rows were inserted) Cassandra server on node 1 throws an exception: {noformat}ERROR [COMMIT-LOG-ALLOCATOR] 2012-06-13 10:26:38,393 AbstractCassandraDaemon.java (line 134) Exception in thread Thread[COMMIT-LOG-ALLOCATOR,5,main] java.io.IOError: java.io.IOException: Rename from c:\apache-cassandra\storage\commitlog\CommitLog-7345742389552.log to 7475933520374 failed at org.apache.cassandra.db.commitlog.CommitLogSegment.(CommitLogSegment.java:127) at org.apache.cassandra.db.commitlog.CommitLogSegment.recycle(CommitLogSegment.java:204) at org.apache.cassandra.db.commitlog.CommitLogAllocator$2.run(CommitLogAllocator.java:166) at org.apache.cassandra.db.commitlog.CommitLogAllocator$1.runMayThrow(CommitLogAllocator.java:95) at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30) at java.lang.Thread.run(Thread.java:722) Caused by: java.io.IOException: Rename from c:\apache-cassandra\storage\commitlog\CommitLog-7345742389552.log to 7475933520374 failed at org.apache.cassandra.db.commitlog.CommitLogSegment.(CommitLogSegment.java:105) ... 5 more{noformat} Then, few seconds later Cassandra server on node 2 throws the same exception: {noformat}ERROR [COMMIT-LOG-ALLOCATOR] 2012-06-14 10:26:44,005 AbstractCassandraDaemon.java (line 134) Exception in thread Thread[COMMIT-LOG-ALLOCATOR,5,main] java.io.IOError: java.io.IOException: Rename from c:\apache-cassandra\storage\commitlog\CommitLog-7320337904033.log to 7437675489307 failed at org.apache.cassandra.db.commitlog.CommitLogSegment.(CommitLogSegment.java:127) at org.apache.cassandra.db.commitlog.CommitLogSegment.recycle(CommitLogSegment.java:204) at org.apache.cassandra.db.commitlog.CommitLogAllocator$2.run(CommitLogAllocator.java:166) at org.apache.cassandra.db.commitlog.CommitLogAllocator$1.runMayThrow(CommitLogAllocator.java:95) at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30) at java.lang.Thread.run(Unknown Source) Caused by: java.io.IOException: Rename from c:\apache-cassandra\storage\commitlog\CommitLog-7320337904033.log to 7437675489307 failed at org.apache.cassandra.db.commitlog.CommitLogSegment.(CommitLogSegment.java:105) ... 5 more{noformat} After that, my application cannot insert any more data. Hector gets TimedOutException from Thrift: {noformat}Thread-4 HConnectionManager.java 306 2012-06-14 10:12:56,034 HConnectionManager operateWithFailover WARN%Could not fullfill request on this host CassandraClient<172.22.1.36:9160-10> Thread-4 HConnectionManager.java 307 2012-06-14 10:12:56,034 HConnectionManager operateWithFailover W