[
https://issues.apache.org/jira/browse/CASSANDRA-4337?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Patrycjusz Matuszak updated CASSANDRA-4337:
-------------------------------------------
Description:
h3. Configuration
Cassandra server configuration:
{noformat}heap size: 4 GB
seed_provider:
- class_name: org.apache.cassandra.locator.SimpleSeedProvider
parameters:
- seeds: "xxx.xxx.xxx.10,xxx.xxx.xxx.11"
listen_address: xxx.xxx.xxx.10
rpc_address: 0.0.0.0
rpc_port: 9160
rpc_timeout_in_ms: 20000
endpoint_snitch: PropertyFileSnitch{noformat}
cassandra-topology.properties
{noformat}xxx.xxx.xxx.10=datacenter1:rack1
xxx.xxx.xxx.11=datacenter1:rack1
default=datacenter1:rack1{noformat}
Ring configuration:
{noformat}Address DC Rack Status State Load
Effective-Ownership Token
85070591730234615865843651857942052864
xxx.xxx.xxx.10 datacenter1 rack1 Up Normal 23,11 kB 100,00%
0
xxx.xxx.xxx.11 datacenter1 rack1 Up Normal 23,25 kB 100,00%
85070591730234615865843651857942052864{noformat}
h3.Problem
I have ctreated keyspace and column family using CLI commands:
{noformat}create keyspace testks with placement_strategy =
'org.apache.cassandra.locator.NetworkTopologyStrategy' and strategy_options =
{datacenter1:2};
use testks;
create column family testcf;{noformat}
Then I started my Java application, which inserts 50 000 000 rows to created
column family using Hector client. Client is connected to node 1.
After about 30 seconds (160 000 rows were inserted) Cassandra server on node 1
throws an exception:
{noformat}ERROR [COMMIT-LOG-ALLOCATOR] 2012-06-13 10:26:38,393
AbstractCassandraDaemon.java (line 134) Exception in thread
Thread[COMMIT-LOG-ALLOCATOR,5,main]
java.io.IOError: java.io.IOException: Rename from
c:\apache-cassandra\storage\commitlog\CommitLog-7345742389552.log to
7475933520374 failed
at
org.apache.cassandra.db.commitlog.CommitLogSegment.<init>(CommitLogSegment.java:127)
at
org.apache.cassandra.db.commitlog.CommitLogSegment.recycle(CommitLogSegment.java:204)
at
org.apache.cassandra.db.commitlog.CommitLogAllocator$2.run(CommitLogAllocator.java:166)
at
org.apache.cassandra.db.commitlog.CommitLogAllocator$1.runMayThrow(CommitLogAllocator.java:95)
at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
at java.lang.Thread.run(Thread.java:722)
Caused by: java.io.IOException: Rename from
c:\apache-cassandra\storage\commitlog\CommitLog-7345742389552.log to
7475933520374 failed
at
org.apache.cassandra.db.commitlog.CommitLogSegment.<init>(CommitLogSegment.java:105)
... 5 more{noformat}
Then, few seconds later Cassandra server on node 2 throws the same exception:
{noformat}ERROR [COMMIT-LOG-ALLOCATOR] 2012-06-14 10:26:44,005
AbstractCassandraDaemon.java (line 134) Exception in thread
Thread[COMMIT-LOG-ALLOCATOR,5,main]
java.io.IOError: java.io.IOException: Rename from
c:\apache-cassandra\storage\commitlog\CommitLog-7320337904033.log to
7437675489307 failed
at
org.apache.cassandra.db.commitlog.CommitLogSegment.<init>(CommitLogSegment.java:127)
at
org.apache.cassandra.db.commitlog.CommitLogSegment.recycle(CommitLogSegment.java:204)
at
org.apache.cassandra.db.commitlog.CommitLogAllocator$2.run(CommitLogAllocator.java:166)
at
org.apache.cassandra.db.commitlog.CommitLogAllocator$1.runMayThrow(CommitLogAllocator.java:95)
at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
at java.lang.Thread.run(Unknown Source)
Caused by: java.io.IOException: Rename from
c:\apache-cassandra\storage\commitlog\CommitLog-7320337904033.log to
7437675489307 failed
at
org.apache.cassandra.db.commitlog.CommitLogSegment.<init>(CommitLogSegment.java:105)
... 5 more{noformat}
After that, my application cannot insert any more data. Hector gets
TimedOutException from Thrift:
{noformat}Thread-4 HConnectionManager.java 306 2012-06-14 10:26:56,034
HConnectionManager operateWithFailover WARN %Could not fullfill request
on this host CassandraClient<xxx.xxx.xxx.10:9160-10>
Thread-4 HConnectionManager.java 307 2012-06-14 10:26:56,034 HConnectionManager
operateWithFailover WARN %Exception:
me.prettyprint.hector.api.exceptions.HTimedOutException: TimedOutException()
at
me.prettyprint.cassandra.service.ExceptionsTranslatorImpl.translate(ExceptionsTranslatorImpl.java:35)
at
me.prettyprint.cassandra.connection.HConnectionManager.operateWithFailover(HConnectionManager.java:264)
at
me.prettyprint.cassandra.model.ExecutingKeyspace.doExecuteOperation(ExecutingKeyspace.java:97)
at
me.prettyprint.cassandra.model.MutatorImpl.execute(MutatorImpl.java:243)
at
patrycjusz.nosqltest.db.cassandra.CassandraHectorDbAdapter.commitTransaction(CassandraDbAdapter.java:63)
at patrycjusz.nosqltest.DbTest.insertData(DbTest.java:459)
at patrycjusz.nosqltest.gui.InsertPanel.executeTask(NePanel.java:154)
at patrycjusz.nosqltest.gui.InsertPanel$1.run(NePanel.java:141)
at java.lang.Thread.run(Unknown Source)
Caused by: TimedOutException()
at
org.apache.cassandra.thrift.Cassandra$batch_mutate_result.read(Cassandra.java:20269)
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:78)
at
org.apache.cassandra.thrift.Cassandra$Client.recv_batch_mutate(Cassandra.java:922)
at
org.apache.cassandra.thrift.Cassandra$Client.batch_mutate(Cassandra.java:908)
at
me.prettyprint.cassandra.model.MutatorImpl$3.execute(MutatorImpl.java:246)
at
me.prettyprint.cassandra.model.MutatorImpl$3.execute(MutatorImpl.java:243)
at
me.prettyprint.cassandra.service.Operation.executeAndSetResult(Operation.java:103)
at
me.prettyprint.cassandra.connection.HConnectionManager.operateWithFailover(HConnectionManager.java:258)
... 8 more{noformat}
was:
h3. Configuration
Cassandra server configuration:
{noformat}heap size: 4 GB
seed_provider:
- class_name: org.apache.cassandra.locator.SimpleSeedProvider
parameters:
- seeds: "xxx.xxx.xxx.10,xxx.xxx.xxx.11"
listen_address: xxx.xxx.xxx.10
rpc_address: 0.0.0.0
rpc_port: 9160
rpc_timeout_in_ms: 20000
endpoint_snitch: PropertyFileSnitch{noformat}
cassandra-topology.properties
{noformat}xxx.xxx.xxx.10=datacenter1:rack1
xxx.xxx.xxx.11=datacenter1:rack1
default=datacenter1:rack1{noformat}
Ring configuration:
{noformat}Address DC Rack Status State Load
Effective-Ownership Token
85070591730234615865843651857942052864
xxx.xxx.xxx.10 datacenter1 rack1 Up Normal 23,11 kB 100,00%
0
xxx.xxx.xxx.11 datacenter1 rack1 Up Normal 23,25 kB 100,00%
85070591730234615865843651857942052864{noformat}
h3.Problem
I have ctreated keyspace and column family using CLI commands:
{noformat}create keyspace testks with placement_strategy =
'org.apache.cassandra.locator.NetworkTopologyStrategy' and strategy_options =
{datacenter1:2};
use testks;
create column family testcf;{noformat}
Then I started my Java application, which inserts 50 000 000 rows to created
column family using Hector client. Client is connected to node 1.
After about 30 seconds (160 000 rows were inserted) Cassandra server on node 1
throws an exception:
{noformat}ERROR [COMMIT-LOG-ALLOCATOR] 2012-06-13 10:26:38,393
AbstractCassandraDaemon.java (line 134) Exception in thread
Thread[COMMIT-LOG-ALLOCATOR,5,main]
java.io.IOError: java.io.IOException: Rename from
c:\apache-cassandra\storage\commitlog\CommitLog-7345742389552.log to
7475933520374 failed
at
org.apache.cassandra.db.commitlog.CommitLogSegment.<init>(CommitLogSegment.java:127)
at
org.apache.cassandra.db.commitlog.CommitLogSegment.recycle(CommitLogSegment.java:204)
at
org.apache.cassandra.db.commitlog.CommitLogAllocator$2.run(CommitLogAllocator.java:166)
at
org.apache.cassandra.db.commitlog.CommitLogAllocator$1.runMayThrow(CommitLogAllocator.java:95)
at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
at java.lang.Thread.run(Thread.java:722)
Caused by: java.io.IOException: Rename from
c:\apache-cassandra\storage\commitlog\CommitLog-7345742389552.log to
7475933520374 failed
at
org.apache.cassandra.db.commitlog.CommitLogSegment.<init>(CommitLogSegment.java:105)
... 5 more{noformat}
Then, few seconds later Cassandra server on node 2 throws the same exception:
{noformat}ERROR [COMMIT-LOG-ALLOCATOR] 2012-06-14 10:26:44,005
AbstractCassandraDaemon.java (line 134) Exception in thread
Thread[COMMIT-LOG-ALLOCATOR,5,main]
java.io.IOError: java.io.IOException: Rename from
c:\apache-cassandra\storage\commitlog\CommitLog-7320337904033.log to
7437675489307 failed
at
org.apache.cassandra.db.commitlog.CommitLogSegment.<init>(CommitLogSegment.java:127)
at
org.apache.cassandra.db.commitlog.CommitLogSegment.recycle(CommitLogSegment.java:204)
at
org.apache.cassandra.db.commitlog.CommitLogAllocator$2.run(CommitLogAllocator.java:166)
at
org.apache.cassandra.db.commitlog.CommitLogAllocator$1.runMayThrow(CommitLogAllocator.java:95)
at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
at java.lang.Thread.run(Unknown Source)
Caused by: java.io.IOException: Rename from
c:\apache-cassandra\storage\commitlog\CommitLog-7320337904033.log to
7437675489307 failed
at
org.apache.cassandra.db.commitlog.CommitLogSegment.<init>(CommitLogSegment.java:105)
... 5 more{noformat}
After that, my application cannot insert any more data. Hector gets
TimedOutException from Thrift:
{noformat}Thread-4 HConnectionManager.java 306 2012-06-14 10:12:56,034
HConnectionManager operateWithFailover WARN %Could not fullfill request
on this host CassandraClient<172.22.1.36:9160-10>
Thread-4 HConnectionManager.java 307 2012-06-14 10:12:56,034 HConnectionManager
operateWithFailover WARN %Exception:
me.prettyprint.hector.api.exceptions.HTimedOutException: TimedOutException()
at
me.prettyprint.cassandra.service.ExceptionsTranslatorImpl.translate(ExceptionsTranslatorImpl.java:35)
at
me.prettyprint.cassandra.connection.HConnectionManager.operateWithFailover(HConnectionManager.java:264)
at
me.prettyprint.cassandra.model.ExecutingKeyspace.doExecuteOperation(ExecutingKeyspace.java:97)
at
me.prettyprint.cassandra.model.MutatorImpl.execute(MutatorImpl.java:243)
at
patrycjusz.nosqltest.db.cassandra.CassandraHectorDbAdapter.commitTransaction(CassandraDbAdapter.java:63)
at patrycjusz.nosqltest.DbTest.insertData(DbTest.java:459)
at patrycjusz.nosqltest.gui.InsertPanel.executeTask(NePanel.java:154)
at patrycjusz.nosqltest.gui.InsertPanel$1.run(NePanel.java:141)
at java.lang.Thread.run(Unknown Source)
Caused by: TimedOutException()
at
org.apache.cassandra.thrift.Cassandra$batch_mutate_result.read(Cassandra.java:20269)
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:78)
at
org.apache.cassandra.thrift.Cassandra$Client.recv_batch_mutate(Cassandra.java:922)
at
org.apache.cassandra.thrift.Cassandra$Client.batch_mutate(Cassandra.java:908)
at
me.prettyprint.cassandra.model.MutatorImpl$3.execute(MutatorImpl.java:246)
at
me.prettyprint.cassandra.model.MutatorImpl$3.execute(MutatorImpl.java:243)
at
me.prettyprint.cassandra.service.Operation.executeAndSetResult(Operation.java:103)
at
me.prettyprint.cassandra.connection.HConnectionManager.operateWithFailover(HConnectionManager.java:258)
... 8 more{noformat}
> Data insertion fails because of commitlog rename failure
> --------------------------------------------------------
>
> Key: CASSANDRA-4337
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4337
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.1.1
> Environment: - Node 1:
> Hardware: Intel Xeon 2.83 GHz (4 cores), 24GB RAM, Dell VIRTUAL DISK SCSI
> 500GB
> System: Windows Server 2008 R2 x64
> Java version: 7 update 4 x64
> - Node 2:
> Hardware: Intel Xeon 2.83 GHz (4 cores), 8GB RAM, Dell VIRTUAL DISK SCSI
> 500GB
> System: Windows Server 2008 R2 x64
> Java version: 7 update 4 x64
> Reporter: Patrycjusz Matuszak
> Attachments: system-node1.log, system-node2.log
>
>
> h3. Configuration
> Cassandra server configuration:
> {noformat}heap size: 4 GB
> seed_provider:
> - class_name: org.apache.cassandra.locator.SimpleSeedProvider
> parameters:
> - seeds: "xxx.xxx.xxx.10,xxx.xxx.xxx.11"
> listen_address: xxx.xxx.xxx.10
> rpc_address: 0.0.0.0
> rpc_port: 9160
> rpc_timeout_in_ms: 20000
> endpoint_snitch: PropertyFileSnitch{noformat}
> cassandra-topology.properties
> {noformat}xxx.xxx.xxx.10=datacenter1:rack1
> xxx.xxx.xxx.11=datacenter1:rack1
> default=datacenter1:rack1{noformat}
> Ring configuration:
> {noformat}Address DC Rack Status State Load
> Effective-Ownership Token
>
> 85070591730234615865843651857942052864
> xxx.xxx.xxx.10 datacenter1 rack1 Up Normal 23,11 kB
> 100,00% 0
> xxx.xxx.xxx.11 datacenter1 rack1 Up Normal 23,25 kB
> 100,00% 85070591730234615865843651857942052864{noformat}
> h3.Problem
> I have ctreated keyspace and column family using CLI commands:
> {noformat}create keyspace testks with placement_strategy =
> 'org.apache.cassandra.locator.NetworkTopologyStrategy' and strategy_options =
> {datacenter1:2};
> use testks;
> create column family testcf;{noformat}
> Then I started my Java application, which inserts 50 000 000 rows to created
> column family using Hector client. Client is connected to node 1.
> After about 30 seconds (160 000 rows were inserted) Cassandra server on node
> 1 throws an exception:
> {noformat}ERROR [COMMIT-LOG-ALLOCATOR] 2012-06-13 10:26:38,393
> AbstractCassandraDaemon.java (line 134) Exception in thread
> Thread[COMMIT-LOG-ALLOCATOR,5,main]
> java.io.IOError: java.io.IOException: Rename from
> c:\apache-cassandra\storage\commitlog\CommitLog-7345742389552.log to
> 7475933520374 failed
> at
> org.apache.cassandra.db.commitlog.CommitLogSegment.<init>(CommitLogSegment.java:127)
> at
> org.apache.cassandra.db.commitlog.CommitLogSegment.recycle(CommitLogSegment.java:204)
> at
> org.apache.cassandra.db.commitlog.CommitLogAllocator$2.run(CommitLogAllocator.java:166)
> at
> org.apache.cassandra.db.commitlog.CommitLogAllocator$1.runMayThrow(CommitLogAllocator.java:95)
> at
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
> at java.lang.Thread.run(Thread.java:722)
> Caused by: java.io.IOException: Rename from
> c:\apache-cassandra\storage\commitlog\CommitLog-7345742389552.log to
> 7475933520374 failed
> at
> org.apache.cassandra.db.commitlog.CommitLogSegment.<init>(CommitLogSegment.java:105)
> ... 5 more{noformat}
>
> Then, few seconds later Cassandra server on node 2 throws the same exception:
> {noformat}ERROR [COMMIT-LOG-ALLOCATOR] 2012-06-14 10:26:44,005
> AbstractCassandraDaemon.java (line 134) Exception in thread
> Thread[COMMIT-LOG-ALLOCATOR,5,main]
> java.io.IOError: java.io.IOException: Rename from
> c:\apache-cassandra\storage\commitlog\CommitLog-7320337904033.log to
> 7437675489307 failed
> at
> org.apache.cassandra.db.commitlog.CommitLogSegment.<init>(CommitLogSegment.java:127)
> at
> org.apache.cassandra.db.commitlog.CommitLogSegment.recycle(CommitLogSegment.java:204)
> at
> org.apache.cassandra.db.commitlog.CommitLogAllocator$2.run(CommitLogAllocator.java:166)
> at
> org.apache.cassandra.db.commitlog.CommitLogAllocator$1.runMayThrow(CommitLogAllocator.java:95)
> at
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
> at java.lang.Thread.run(Unknown Source)
> Caused by: java.io.IOException: Rename from
> c:\apache-cassandra\storage\commitlog\CommitLog-7320337904033.log to
> 7437675489307 failed
> at
> org.apache.cassandra.db.commitlog.CommitLogSegment.<init>(CommitLogSegment.java:105)
> ... 5 more{noformat}
> After that, my application cannot insert any more data. Hector gets
> TimedOutException from Thrift:
> {noformat}Thread-4 HConnectionManager.java 306 2012-06-14 10:26:56,034
> HConnectionManager operateWithFailover WARN %Could not fullfill
> request on this host CassandraClient<xxx.xxx.xxx.10:9160-10>
> Thread-4 HConnectionManager.java 307 2012-06-14 10:26:56,034
> HConnectionManager operateWithFailover WARN %Exception:
> me.prettyprint.hector.api.exceptions.HTimedOutException: TimedOutException()
> at
> me.prettyprint.cassandra.service.ExceptionsTranslatorImpl.translate(ExceptionsTranslatorImpl.java:35)
> at
> me.prettyprint.cassandra.connection.HConnectionManager.operateWithFailover(HConnectionManager.java:264)
> at
> me.prettyprint.cassandra.model.ExecutingKeyspace.doExecuteOperation(ExecutingKeyspace.java:97)
> at
> me.prettyprint.cassandra.model.MutatorImpl.execute(MutatorImpl.java:243)
> at
> patrycjusz.nosqltest.db.cassandra.CassandraHectorDbAdapter.commitTransaction(CassandraDbAdapter.java:63)
> at patrycjusz.nosqltest.DbTest.insertData(DbTest.java:459)
> at patrycjusz.nosqltest.gui.InsertPanel.executeTask(NePanel.java:154)
> at patrycjusz.nosqltest.gui.InsertPanel$1.run(NePanel.java:141)
> at java.lang.Thread.run(Unknown Source)
> Caused by: TimedOutException()
> at
> org.apache.cassandra.thrift.Cassandra$batch_mutate_result.read(Cassandra.java:20269)
> at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:78)
> at
> org.apache.cassandra.thrift.Cassandra$Client.recv_batch_mutate(Cassandra.java:922)
> at
> org.apache.cassandra.thrift.Cassandra$Client.batch_mutate(Cassandra.java:908)
> at
> me.prettyprint.cassandra.model.MutatorImpl$3.execute(MutatorImpl.java:246)
> at
> me.prettyprint.cassandra.model.MutatorImpl$3.execute(MutatorImpl.java:243)
> at
> me.prettyprint.cassandra.service.Operation.executeAndSetResult(Operation.java:103)
> at
> me.prettyprint.cassandra.connection.HConnectionManager.operateWithFailover(HConnectionManager.java:258)
> ... 8 more{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
