[
https://issues.apache.org/jira/browse/CASSANDRA-4337?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jonathan Ellis resolved CASSANDRA-4337.
---------------------------------------
Resolution: Fixed
Reviewer: patrycjusz
I was going to try to make this super robust for JVMs that don't provide an
munmap cleaner, but I decided that since (a) it's extremely unlikely that
anyone runs such JVMs on windows, and (b) such JVMs work fine anyway on *nix
since you can rename w/o unmapping first, and (c) this doesn't make things
worse for such JVMs; it just fixes it for OpenJDK / Oracle JDK, I'll just
commit this as is.
(In any case, the "right" solution for such a posited JVM is to wrap its munmap
funcationality the way we did for Oracle's... such functionality has to exist
for the GC to clean up direct buffers.)
> Data insertion fails because of commitlog rename failure
> --------------------------------------------------------
>
> Key: CASSANDRA-4337
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4337
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.1.1
> Environment: - Node 1:
> Hardware: Intel Xeon 2.83 GHz (4 cores), 24GB RAM, Dell VIRTUAL DISK SCSI
> 500GB
> System: Windows Server 2008 R2 x64
> Java version: 7 update 4 x64
> - Node 2:
> Hardware: Intel Xeon 2.83 GHz (4 cores), 8GB RAM, Dell VIRTUAL DISK SCSI
> 500GB
> System: Windows Server 2008 R2 x64
> Java version: 7 update 4 x64
> Reporter: Patrycjusz Matuszak
> Assignee: Jonathan Ellis
> Labels: commitlog
> Fix For: 1.1.4
>
> Attachments: 4337-poc.txt, system-node1-stress-test.log,
> system-node1.log, system-node2-stress-test.log, system-node2.log
>
>
> h3. Configuration
> Cassandra server configuration:
> {noformat}heap size: 4 GB
> seed_provider:
> - class_name: org.apache.cassandra.locator.SimpleSeedProvider
> parameters:
> - seeds: "xxx.xxx.xxx.10,xxx.xxx.xxx.11"
> listen_address: xxx.xxx.xxx.10
> rpc_address: 0.0.0.0
> rpc_port: 9160
> rpc_timeout_in_ms: 20000
> endpoint_snitch: PropertyFileSnitch{noformat}
> cassandra-topology.properties
> {noformat}xxx.xxx.xxx.10=datacenter1:rack1
> xxx.xxx.xxx.11=datacenter1:rack1
> default=datacenter1:rack1{noformat}
> Ring configuration:
> {noformat}Address DC Rack Status State Load
> Effective-Ownership Token
>
> 85070591730234615865843651857942052864
> xxx.xxx.xxx.10 datacenter1 rack1 Up Normal 23,11 kB
> 100,00% 0
> xxx.xxx.xxx.11 datacenter1 rack1 Up Normal 23,25 kB
> 100,00% 85070591730234615865843651857942052864{noformat}
> h3.Problem
> I have ctreated keyspace and column family using CLI commands:
> {noformat}create keyspace testks with placement_strategy =
> 'org.apache.cassandra.locator.NetworkTopologyStrategy' and strategy_options =
> {datacenter1:2};
> use testks;
> create column family testcf;{noformat}
> Then I started my Java application, which inserts 50 000 000 rows to created
> column family using Hector client. Client is connected to node 1.
> After about 30 seconds (160 000 rows were inserted) Cassandra server on node
> 1 throws an exception:
> {noformat}ERROR [COMMIT-LOG-ALLOCATOR] 2012-06-13 10:26:38,393
> AbstractCassandraDaemon.java (line 134) Exception in thread
> Thread[COMMIT-LOG-ALLOCATOR,5,main]
> java.io.IOError: java.io.IOException: Rename from
> c:\apache-cassandra\storage\commitlog\CommitLog-7345742389552.log to
> 7475933520374 failed
> at
> org.apache.cassandra.db.commitlog.CommitLogSegment.<init>(CommitLogSegment.java:127)
> at
> org.apache.cassandra.db.commitlog.CommitLogSegment.recycle(CommitLogSegment.java:204)
> at
> org.apache.cassandra.db.commitlog.CommitLogAllocator$2.run(CommitLogAllocator.java:166)
> at
> org.apache.cassandra.db.commitlog.CommitLogAllocator$1.runMayThrow(CommitLogAllocator.java:95)
> at
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
> at java.lang.Thread.run(Thread.java:722)
> Caused by: java.io.IOException: Rename from
> c:\apache-cassandra\storage\commitlog\CommitLog-7345742389552.log to
> 7475933520374 failed
> at
> org.apache.cassandra.db.commitlog.CommitLogSegment.<init>(CommitLogSegment.java:105)
> ... 5 more{noformat}
>
> Then, few seconds later Cassandra server on node 2 throws the same exception:
> {noformat}ERROR [COMMIT-LOG-ALLOCATOR] 2012-06-14 10:26:44,005
> AbstractCassandraDaemon.java (line 134) Exception in thread
> Thread[COMMIT-LOG-ALLOCATOR,5,main]
> java.io.IOError: java.io.IOException: Rename from
> c:\apache-cassandra\storage\commitlog\CommitLog-7320337904033.log to
> 7437675489307 failed
> at
> org.apache.cassandra.db.commitlog.CommitLogSegment.<init>(CommitLogSegment.java:127)
> at
> org.apache.cassandra.db.commitlog.CommitLogSegment.recycle(CommitLogSegment.java:204)
> at
> org.apache.cassandra.db.commitlog.CommitLogAllocator$2.run(CommitLogAllocator.java:166)
> at
> org.apache.cassandra.db.commitlog.CommitLogAllocator$1.runMayThrow(CommitLogAllocator.java:95)
> at
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
> at java.lang.Thread.run(Unknown Source)
> Caused by: java.io.IOException: Rename from
> c:\apache-cassandra\storage\commitlog\CommitLog-7320337904033.log to
> 7437675489307 failed
> at
> org.apache.cassandra.db.commitlog.CommitLogSegment.<init>(CommitLogSegment.java:105)
> ... 5 more{noformat}
> After that, my application cannot insert any more data. Hector gets
> TimedOutException from Thrift:
> {noformat}Thread-4 HConnectionManager.java 306 2012-06-14 10:26:56,034
> HConnectionManager operateWithFailover WARN %Could not fullfill
> request on this host CassandraClient<xxx.xxx.xxx.10:9160-10>
> Thread-4 HConnectionManager.java 307 2012-06-14 10:26:56,034
> HConnectionManager operateWithFailover WARN %Exception:
> me.prettyprint.hector.api.exceptions.HTimedOutException: TimedOutException()
> at
> me.prettyprint.cassandra.service.ExceptionsTranslatorImpl.translate(ExceptionsTranslatorImpl.java:35)
> at
> me.prettyprint.cassandra.connection.HConnectionManager.operateWithFailover(HConnectionManager.java:264)
> at
> me.prettyprint.cassandra.model.ExecutingKeyspace.doExecuteOperation(ExecutingKeyspace.java:97)
> at
> me.prettyprint.cassandra.model.MutatorImpl.execute(MutatorImpl.java:243)
> at
> patrycjusz.nosqltest.db.cassandra.CassandraHectorDbAdapter.commitTransaction(CassandraDbAdapter.java:63)
> at patrycjusz.nosqltest.DbTest.insertData(DbTest.java:459)
> at patrycjusz.nosqltest.gui.InsertPanel.executeTask(NePanel.java:154)
> at patrycjusz.nosqltest.gui.InsertPanel$1.run(NePanel.java:141)
> at java.lang.Thread.run(Unknown Source)
> Caused by: TimedOutException()
> at
> org.apache.cassandra.thrift.Cassandra$batch_mutate_result.read(Cassandra.java:20269)
> at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:78)
> at
> org.apache.cassandra.thrift.Cassandra$Client.recv_batch_mutate(Cassandra.java:922)
> at
> org.apache.cassandra.thrift.Cassandra$Client.batch_mutate(Cassandra.java:908)
> at
> me.prettyprint.cassandra.model.MutatorImpl$3.execute(MutatorImpl.java:246)
> at
> me.prettyprint.cassandra.model.MutatorImpl$3.execute(MutatorImpl.java:243)
> at
> me.prettyprint.cassandra.service.Operation.executeAndSetResult(Operation.java:103)
> at
> me.prettyprint.cassandra.connection.HConnectionManager.operateWithFailover(HConnectionManager.java:258)
> ... 8 more{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
