Re: failed to delete commitlog, cassandra can't accept writes
On Tue, 10 Jul 2012 14:35:23 -0700, Frank Hsueh wrote: after reading the JIRA, I decided to use Java 6. It has nothing to do with the JDK. I can reproduce it with either JDK6 or JDK7 as well. anybody seen this before? is this related to 4337 ? It's exactly that. -h
Re: failed to delete commitlog, cassandra can't accept writes
I don't think it's related to 4337. There is an explicit close call just before the deletion attempt. Can you create a ticket on https://issues.apache.org/jira/browse/CASSANDRA with all of the information you've got here (including the full JVM vendor, version, build). Can you also check if the file it tries to delete exists ? (I assume it does, otherwise it would be a different error). Thanks for digging into this. - Aaron Morton Freelance Developer @aaronmorton http://www.thelastpickle.com On 11/07/2012, at 9:36 AM, Frank Hsueh wrote: oops; I missed log line: ERROR [COMMIT-LOG-ALLOCATOR] 2012-07-10 14:19:39,776 AbstractCassandraDaemon.java (line 134) Exception in thread Thread[COMMIT-LOG-ALLOCATOR,5,main] java.io.IOError: java.io.IOException: Failed to delete C:\var\lib\cassandra\commitlog\CommitLog-948695923996466.log at org.apache.cassandra.db.commitlog.CommitLogSegment.discard(CommitLogSegment.java:176) at org.apache.cassandra.db.commitlog.CommitLogAllocator$4.run(CommitLogAllocator.java:223) at org.apache.cassandra.db.commitlog.CommitLogAllocator$1.runMayThrow(CommitLogAllocator.java:95) at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30) at java.lang.Thread.run(Thread.java:662) Caused by: java.io.IOException: Failed to delete C:\var\lib\cassandra\commitlog\CommitLog-948695923996466.log at org.apache.cassandra.io.util.FileUtils.deleteWithConfirm(FileUtils.java:54) at org.apache.cassandra.db.commitlog.CommitLogSegment.discard(CommitLogSegment.java:172) ... 4 more On Tue, Jul 10, 2012 at 2:35 PM, Frank Hsueh frank.hs...@gmail.com wrote: after reading the JIRA, I decided to use Java 6. with Casandra 1.1.2 on Java 6 x64 on Win7 sp1 x64 (all latest versions), after a several minutes of sustained writes, I see: from system.log: java.io.IOError: java.io.IOException: Failed to delete C:\var\lib\cassandra\commitlog\CommitLog-948695923996466.log at org.apache.cassandra.db.commitlog.CommitLogSegment.discard(CommitLogSegment.java:176) at org.apache.cassandra.db.commitlog.CommitLogAllocator$4.run(CommitLogAllocator.java:223) at org.apache.cassandra.db.commitlog.CommitLogAllocator$1.runMayThrow(CommitLogAllocator.java:95) at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30) at java.lang.Thread.run(Thread.java:662) Caused by: java.io.IOException: Failed to delete C:\var\lib\cassandra\commitlog\CommitLog-948695923996466.log at org.apache.cassandra.io.util.FileUtils.deleteWithConfirm(FileUtils.java:54) at org.apache.cassandra.db.commitlog.CommitLogSegment.discard(CommitLogSegment.java:172) ... 4 more anybody seen this before? is this related to 4337 ? On Sat, Jul 7, 2012 at 6:36 PM, Frank Hsueh frank.hs...@gmail.com wrote: bug already reported: https://issues.apache.org/jira/browse/CASSANDRA-4337 On Sat, Jul 7, 2012 at 6:26 PM, Frank Hsueh frank.hs...@gmail.com wrote: Hi, I'm running Casandra 1.1.2 on Java 7 x64 on Win7 sp1 x64 (all latest versions). If it matters, I'm using a late version of Astyanax as my client. I'm using 4 threads to write a lot of data into a single CF. After several minutes of load (~ 30m at last incident), Cassandra stops accepting writes (client reports an OperationTimeoutException). I looked at the logs and I see on the Cassandra server: ERROR 18:00:42,807 Exception in thread Thread[COMMIT-LOG-ALLOCATOR,5,main] java.io.IOError: java.io.IOException: Rename from \var\lib\cassandra\commitlog\CommitLog-701533048437587.log to 703272597990002 failed at org.apache.cassandra.db.commitlog.CommitLogSegment.init(CommitLogSegment.java:127) at org.apache.cassandra.db.commitlog.CommitLogSegment.recycle(CommitLogSegment.java:204) at org.apache.cassandra.db.commitlog.CommitLogAllocator$2.run(CommitLogAllocator.java:166) at org.apache.cassandra.db.commitlog.CommitLogAllocator$1.runMayThrow(CommitLogAllocator.java:95) at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30) at java.lang.Thread.run(Thread.java:722) Caused by: java.io.IOException: Rename from \var\lib\cassandra\commitlog\CommitLog-701533048437587.log to 703272597990002 failed at org.apache.cassandra.db.commitlog.CommitLogSegment.init(CommitLogSegment.java:105) ... 5 more Anybody else seen this before ? -- Frank Hsueh | frank.hs...@gmail.com -- Frank Hsueh | frank.hs...@gmail.com -- Frank Hsueh | frank.hs...@gmail.com -- Frank Hsueh | frank.hs...@gmail.com
failed to delete commitlog, cassandra can't accept writes
after reading the JIRA, I decided to use Java 6. with Casandra 1.1.2 on Java 6 x64 on Win7 sp1 x64 (all latest versions), after a several minutes of sustained writes, I see: from system.log: java.io.IOError: java.io.IOException: Failed to delete C:\var\lib\cassandra\commitlog\CommitLog-948695923996466.log at org.apache.cassandra.db.commitlog.CommitLogSegment.discard(CommitLogSegment.java:176) at org.apache.cassandra.db.commitlog.CommitLogAllocator$4.run(CommitLogAllocator.java:223) at org.apache.cassandra.db.commitlog.CommitLogAllocator$1.runMayThrow(CommitLogAllocator.java:95) at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30) at java.lang.Thread.run(Thread.java:662) Caused by: java.io.IOException: Failed to delete C:\var\lib\cassandra\commitlog\CommitLog-948695923996466.log at org.apache.cassandra.io.util.FileUtils.deleteWithConfirm(FileUtils.java:54) at org.apache.cassandra.db.commitlog.CommitLogSegment.discard(CommitLogSegment.java:172) ... 4 more anybody seen this before? is this related to 4337 ? On Sat, Jul 7, 2012 at 6:36 PM, Frank Hsueh frank.hs...@gmail.com wrote: bug already reported: https://issues.apache.org/jira/browse/CASSANDRA-4337 On Sat, Jul 7, 2012 at 6:26 PM, Frank Hsueh frank.hs...@gmail.com wrote: Hi, I'm running Casandra 1.1.2 on Java 7 x64 on Win7 sp1 x64 (all latest versions). If it matters, I'm using a late version of Astyanax as my client. I'm using 4 threads to write a lot of data into a single CF. After several minutes of load (~ 30m at last incident), Cassandra stops accepting writes (client reports an OperationTimeoutException). I looked at the logs and I see on the Cassandra server: ERROR 18:00:42,807 Exception in thread Thread[COMMIT-LOG-ALLOCATOR,5,main] java.io.IOError: java.io.IOException: Rename from \var\lib\cassandra\commitlog\CommitLog-701533048437587.log to 703272597990002 failed at org.apache.cassandra.db.commitlog.CommitLogSegment.init(CommitLogSegment.java:127) at org.apache.cassandra.db.commitlog.CommitLogSegment.recycle(CommitLogSegment.java:204) at org.apache.cassandra.db.commitlog.CommitLogAllocator$2.run(CommitLogAllocator.java:166) at org.apache.cassandra.db.commitlog.CommitLogAllocator$1.runMayThrow(CommitLogAllocator.java:95) at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30) at java.lang.Thread.run(Thread.java:722) Caused by: java.io.IOException: Rename from \var\lib\cassandra\commitlog\CommitLog-701533048437587.log to 703272597990002 failed at org.apache.cassandra.db.commitlog.CommitLogSegment.init(CommitLogSegment.java:105) ... 5 more Anybody else seen this before ? -- Frank Hsueh | frank.hs...@gmail.com -- Frank Hsueh | frank.hs...@gmail.com -- Frank Hsueh | frank.hs...@gmail.com
Re: failed to delete commitlog, cassandra can't accept writes
oops; I missed log line: ERROR [COMMIT-LOG-ALLOCATOR] 2012-07-10 14:19:39,776 AbstractCassandraDaemon.java (line 134) Exception in thread Thread[COMMIT-LOG-ALLOCATOR,5,main] java.io.IOError: java.io.IOException: Failed to delete C:\var\lib\cassandra\commitlog\CommitLog-948695923996466.log at org.apache.cassandra.db.commitlog.CommitLogSegment.discard(CommitLogSegment.java:176) at org.apache.cassandra.db.commitlog.CommitLogAllocator$4.run(CommitLogAllocator.java:223) at org.apache.cassandra.db.commitlog.CommitLogAllocator$1.runMayThrow(CommitLogAllocator.java:95) at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30) at java.lang.Thread.run(Thread.java:662) Caused by: java.io.IOException: Failed to delete C:\var\lib\cassandra\commitlog\CommitLog-948695923996466.log at org.apache.cassandra.io.util.FileUtils.deleteWithConfirm(FileUtils.java:54) at org.apache.cassandra.db.commitlog.CommitLogSegment.discard(CommitLogSegment.java:172) ... 4 more On Tue, Jul 10, 2012 at 2:35 PM, Frank Hsueh frank.hs...@gmail.com wrote: after reading the JIRA, I decided to use Java 6. with Casandra 1.1.2 on Java 6 x64 on Win7 sp1 x64 (all latest versions), after a several minutes of sustained writes, I see: from system.log: java.io.IOError: java.io.IOException: Failed to delete C:\var\lib\cassandra\commitlog\CommitLog-948695923996466.log at org.apache.cassandra.db.commitlog.CommitLogSegment.discard(CommitLogSegment.java:176) at org.apache.cassandra.db.commitlog.CommitLogAllocator$4.run(CommitLogAllocator.java:223) at org.apache.cassandra.db.commitlog.CommitLogAllocator$1.runMayThrow(CommitLogAllocator.java:95) at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30) at java.lang.Thread.run(Thread.java:662) Caused by: java.io.IOException: Failed to delete C:\var\lib\cassandra\commitlog\CommitLog-948695923996466.log at org.apache.cassandra.io.util.FileUtils.deleteWithConfirm(FileUtils.java:54) at org.apache.cassandra.db.commitlog.CommitLogSegment.discard(CommitLogSegment.java:172) ... 4 more anybody seen this before? is this related to 4337 ? On Sat, Jul 7, 2012 at 6:36 PM, Frank Hsueh frank.hs...@gmail.com wrote: bug already reported: https://issues.apache.org/jira/browse/CASSANDRA-4337 On Sat, Jul 7, 2012 at 6:26 PM, Frank Hsueh frank.hs...@gmail.comwrote: Hi, I'm running Casandra 1.1.2 on Java 7 x64 on Win7 sp1 x64 (all latest versions). If it matters, I'm using a late version of Astyanax as my client. I'm using 4 threads to write a lot of data into a single CF. After several minutes of load (~ 30m at last incident), Cassandra stops accepting writes (client reports an OperationTimeoutException). I looked at the logs and I see on the Cassandra server: ERROR 18:00:42,807 Exception in thread Thread[COMMIT-LOG-ALLOCATOR,5,main] java.io.IOError: java.io.IOException: Rename from \var\lib\cassandra\commitlog\CommitLog-701533048437587.log to 703272597990002 failed at org.apache.cassandra.db.commitlog.CommitLogSegment.init(CommitLogSegment.java:127) at org.apache.cassandra.db.commitlog.CommitLogSegment.recycle(CommitLogSegment.java:204) at org.apache.cassandra.db.commitlog.CommitLogAllocator$2.run(CommitLogAllocator.java:166) at org.apache.cassandra.db.commitlog.CommitLogAllocator$1.runMayThrow(CommitLogAllocator.java:95) at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30) at java.lang.Thread.run(Thread.java:722) Caused by: java.io.IOException: Rename from \var\lib\cassandra\commitlog\CommitLog-701533048437587.log to 703272597990002 failed at org.apache.cassandra.db.commitlog.CommitLogSegment.init(CommitLogSegment.java:105) ... 5 more Anybody else seen this before ? -- Frank Hsueh | frank.hs...@gmail.com -- Frank Hsueh | frank.hs...@gmail.com -- Frank Hsueh | frank.hs...@gmail.com -- Frank Hsueh | frank.hs...@gmail.com
