Re: failed to delete commitlog, cassandra can't accept writes
On Tue, 10 Jul 2012 14:35:23 -0700, Frank Hsueh wrote: > after reading the JIRA, I decided to use Java 6. It has nothing to do with the JDK. I can reproduce it with either JDK6 or JDK7 as well. > anybody seen this before? is this related to 4337 ? It's exactly that. -h
Re: failed to delete commitlog, cassandra can't accept writes
I don't think it's related to 4337. There is an explicit close call just before the deletion attempt. Can you create a ticket on https://issues.apache.org/jira/browse/CASSANDRA with all of the information you've got here (including the full JVM vendor, version, build). Can you also check if the file it tries to delete exists ? (I assume it does, otherwise it would be a different error). Thanks for digging into this. - Aaron Morton Freelance Developer @aaronmorton http://www.thelastpickle.com On 11/07/2012, at 9:36 AM, Frank Hsueh wrote: > oops; I missed log line: > > > ERROR [COMMIT-LOG-ALLOCATOR] 2012-07-10 14:19:39,776 > AbstractCassandraDaemon.java (line 134) Exception in thread > Thread[COMMIT-LOG-ALLOCATOR,5,main] > java.io.IOError: java.io.IOException: Failed to delete > C:\var\lib\cassandra\commitlog\CommitLog-948695923996466.log > at > org.apache.cassandra.db.commitlog.CommitLogSegment.discard(CommitLogSegment.java:176) > at > org.apache.cassandra.db.commitlog.CommitLogAllocator$4.run(CommitLogAllocator.java:223) > at > org.apache.cassandra.db.commitlog.CommitLogAllocator$1.runMayThrow(CommitLogAllocator.java:95) > at > org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30) > at java.lang.Thread.run(Thread.java:662) > Caused by: java.io.IOException: Failed to delete > C:\var\lib\cassandra\commitlog\CommitLog-948695923996466.log > at > org.apache.cassandra.io.util.FileUtils.deleteWithConfirm(FileUtils.java:54) > at > org.apache.cassandra.db.commitlog.CommitLogSegment.discard(CommitLogSegment.java:172) > ... 4 more > > > > On Tue, Jul 10, 2012 at 2:35 PM, Frank Hsueh wrote: > after reading the JIRA, I decided to use Java 6. > > with Casandra 1.1.2 on Java 6 x64 on Win7 sp1 x64 (all latest versions), > after a several minutes of sustained writes, I see: > > from system.log: > > java.io.IOError: java.io.IOException: Failed to delete > C:\var\lib\cassandra\commitlog\CommitLog-948695923996466.log > at > org.apache.cassandra.db.commitlog.CommitLogSegment.discard(CommitLogSegment.java:176) > at > org.apache.cassandra.db.commitlog.CommitLogAllocator$4.run(CommitLogAllocator.java:223) > at > org.apache.cassandra.db.commitlog.CommitLogAllocator$1.runMayThrow(CommitLogAllocator.java:95) > at > org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30) > at java.lang.Thread.run(Thread.java:662) > Caused by: java.io.IOException: Failed to delete > C:\var\lib\cassandra\commitlog\CommitLog-948695923996466.log > at > org.apache.cassandra.io.util.FileUtils.deleteWithConfirm(FileUtils.java:54) > at > org.apache.cassandra.db.commitlog.CommitLogSegment.discard(CommitLogSegment.java:172) > ... 4 more > > > anybody seen this before? is this related to 4337 ? > > > > > On Sat, Jul 7, 2012 at 6:36 PM, Frank Hsueh wrote: > bug already reported: > > https://issues.apache.org/jira/browse/CASSANDRA-4337 > > > > On Sat, Jul 7, 2012 at 6:26 PM, Frank Hsueh wrote: > Hi, > > I'm running Casandra 1.1.2 on Java 7 x64 on Win7 sp1 x64 (all latest > versions). If it matters, I'm using a late version of Astyanax as my client. > > I'm using 4 threads to write a lot of data into a single CF. > > After several minutes of load (~ 30m at last incident), Cassandra stops > accepting writes (client reports an OperationTimeoutException). I looked at > the logs and I see on the Cassandra server: > > > ERROR 18:00:42,807 Exception in thread Thread[COMMIT-LOG-ALLOCATOR,5,main] > java.io.IOError: java.io.IOException: Rename from > \var\lib\cassandra\commitlog\CommitLog-701533048437587.log to 703272597990002 > failed > at > org.apache.cassandra.db.commitlog.CommitLogSegment.(CommitLogSegment.java:127) > at > org.apache.cassandra.db.commitlog.CommitLogSegment.recycle(CommitLogSegment.java:204) > at > org.apache.cassandra.db.commitlog.CommitLogAllocator$2.run(CommitLogAllocator.java:166) > at > org.apache.cassandra.db.commitlog.CommitLogAllocator$1.runMayThrow(CommitLogAllocator.java:95) > at > org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30) > at java.lang.Thread.run(Thread.java:722) > Caused by: java.io.IOException: Rename from > \var\lib\cassandra\commitlog\CommitLog-701533048437587.log to 703272597990002 > failed > at > org.apache.cassandra.db.commitlog.CommitLogSegment.(CommitLogSegment.java:105) > ... 5 more > > > Anybody else seen this before ? > > > -- > Frank Hsueh | frank.hs...@gmail.com > > > > -- > Frank Hsueh | frank.hs...@gmail.com > > > > -- > Frank Hsueh | frank.hs...@gmail.com > > > > -- > Frank Hsueh | frank.hs...@gmail.com
Re: failed to delete commitlog, cassandra can't accept writes
oops; I missed log line: ERROR [COMMIT-LOG-ALLOCATOR] 2012-07-10 14:19:39,776 AbstractCassandraDaemon.java (line 134) Exception in thread Thread[COMMIT-LOG-ALLOCATOR,5,main] java.io.IOError: java.io.IOException: Failed to delete C:\var\lib\cassandra\commitlog\CommitLog-948695923996466.log at org.apache.cassandra.db.commitlog.CommitLogSegment.discard(CommitLogSegment.java:176) at org.apache.cassandra.db.commitlog.CommitLogAllocator$4.run(CommitLogAllocator.java:223) at org.apache.cassandra.db.commitlog.CommitLogAllocator$1.runMayThrow(CommitLogAllocator.java:95) at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30) at java.lang.Thread.run(Thread.java:662) Caused by: java.io.IOException: Failed to delete C:\var\lib\cassandra\commitlog\CommitLog-948695923996466.log at org.apache.cassandra.io.util.FileUtils.deleteWithConfirm(FileUtils.java:54) at org.apache.cassandra.db.commitlog.CommitLogSegment.discard(CommitLogSegment.java:172) ... 4 more On Tue, Jul 10, 2012 at 2:35 PM, Frank Hsueh wrote: > after reading the JIRA, I decided to use Java 6. > > with Casandra 1.1.2 on Java 6 x64 on Win7 sp1 x64 (all latest versions), > after a several minutes of sustained writes, I see: > > from system.log: > > java.io.IOError: java.io.IOException: Failed to delete > C:\var\lib\cassandra\commitlog\CommitLog-948695923996466.log > at > org.apache.cassandra.db.commitlog.CommitLogSegment.discard(CommitLogSegment.java:176) > at > org.apache.cassandra.db.commitlog.CommitLogAllocator$4.run(CommitLogAllocator.java:223) > at > org.apache.cassandra.db.commitlog.CommitLogAllocator$1.runMayThrow(CommitLogAllocator.java:95) > at > org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30) > at java.lang.Thread.run(Thread.java:662) > Caused by: java.io.IOException: Failed to delete > C:\var\lib\cassandra\commitlog\CommitLog-948695923996466.log > at > org.apache.cassandra.io.util.FileUtils.deleteWithConfirm(FileUtils.java:54) > at > org.apache.cassandra.db.commitlog.CommitLogSegment.discard(CommitLogSegment.java:172) > ... 4 more > > > anybody seen this before? is this related to 4337 ? > > > > > On Sat, Jul 7, 2012 at 6:36 PM, Frank Hsueh wrote: > >> bug already reported: >> >> https://issues.apache.org/jira/browse/CASSANDRA-4337 >> >> >> >> On Sat, Jul 7, 2012 at 6:26 PM, Frank Hsueh wrote: >> >>> Hi, >>> >>> I'm running Casandra 1.1.2 on Java 7 x64 on Win7 sp1 x64 (all latest >>> versions). If it matters, I'm using a late version of Astyanax as my >>> client. >>> >>> I'm using 4 threads to write a lot of data into a single CF. >>> >>> After several minutes of load (~ 30m at last incident), Cassandra stops >>> accepting writes (client reports an OperationTimeoutException). I looked >>> at the logs and I see on the Cassandra server: >>> >>> >>> ERROR 18:00:42,807 Exception in thread >>> Thread[COMMIT-LOG-ALLOCATOR,5,main] >>> java.io.IOError: java.io.IOException: Rename from >>> \var\lib\cassandra\commitlog\CommitLog-701533048437587.log to >>> 703272597990002 failed >>> at >>> org.apache.cassandra.db.commitlog.CommitLogSegment.(CommitLogSegment.java:127) >>> at >>> org.apache.cassandra.db.commitlog.CommitLogSegment.recycle(CommitLogSegment.java:204) >>> at >>> org.apache.cassandra.db.commitlog.CommitLogAllocator$2.run(CommitLogAllocator.java:166) >>> at >>> org.apache.cassandra.db.commitlog.CommitLogAllocator$1.runMayThrow(CommitLogAllocator.java:95) >>> at >>> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30) >>> at java.lang.Thread.run(Thread.java:722) >>> Caused by: java.io.IOException: Rename from >>> \var\lib\cassandra\commitlog\CommitLog-701533048437587.log to >>> 703272597990002 failed >>> at >>> org.apache.cassandra.db.commitlog.CommitLogSegment.(CommitLogSegment.java:105) >>> ... 5 more >>> >>> >>> Anybody else seen this before ? >>> >>> >>> -- >>> Frank Hsueh | frank.hs...@gmail.com >>> >> >> >> >> -- >> Frank Hsueh | frank.hs...@gmail.com >> > > > > -- > Frank Hsueh | frank.hs...@gmail.com > -- Frank Hsueh | frank.hs...@gmail.com
failed to delete commitlog, cassandra can't accept writes
after reading the JIRA, I decided to use Java 6. with Casandra 1.1.2 on Java 6 x64 on Win7 sp1 x64 (all latest versions), after a several minutes of sustained writes, I see: from system.log: java.io.IOError: java.io.IOException: Failed to delete C:\var\lib\cassandra\commitlog\CommitLog-948695923996466.log at org.apache.cassandra.db.commitlog.CommitLogSegment.discard(CommitLogSegment.java:176) at org.apache.cassandra.db.commitlog.CommitLogAllocator$4.run(CommitLogAllocator.java:223) at org.apache.cassandra.db.commitlog.CommitLogAllocator$1.runMayThrow(CommitLogAllocator.java:95) at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30) at java.lang.Thread.run(Thread.java:662) Caused by: java.io.IOException: Failed to delete C:\var\lib\cassandra\commitlog\CommitLog-948695923996466.log at org.apache.cassandra.io.util.FileUtils.deleteWithConfirm(FileUtils.java:54) at org.apache.cassandra.db.commitlog.CommitLogSegment.discard(CommitLogSegment.java:172) ... 4 more anybody seen this before? is this related to 4337 ? On Sat, Jul 7, 2012 at 6:36 PM, Frank Hsueh wrote: > bug already reported: > > https://issues.apache.org/jira/browse/CASSANDRA-4337 > > > > On Sat, Jul 7, 2012 at 6:26 PM, Frank Hsueh wrote: > >> Hi, >> >> I'm running Casandra 1.1.2 on Java 7 x64 on Win7 sp1 x64 (all latest >> versions). If it matters, I'm using a late version of Astyanax as my >> client. >> >> I'm using 4 threads to write a lot of data into a single CF. >> >> After several minutes of load (~ 30m at last incident), Cassandra stops >> accepting writes (client reports an OperationTimeoutException). I looked >> at the logs and I see on the Cassandra server: >> >> >> ERROR 18:00:42,807 Exception in thread Thread[COMMIT-LOG-ALLOCATOR,5,main] >> java.io.IOError: java.io.IOException: Rename from >> \var\lib\cassandra\commitlog\CommitLog-701533048437587.log to >> 703272597990002 failed >> at >> org.apache.cassandra.db.commitlog.CommitLogSegment.(CommitLogSegment.java:127) >> at >> org.apache.cassandra.db.commitlog.CommitLogSegment.recycle(CommitLogSegment.java:204) >> at >> org.apache.cassandra.db.commitlog.CommitLogAllocator$2.run(CommitLogAllocator.java:166) >> at >> org.apache.cassandra.db.commitlog.CommitLogAllocator$1.runMayThrow(CommitLogAllocator.java:95) >> at >> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30) >> at java.lang.Thread.run(Thread.java:722) >> Caused by: java.io.IOException: Rename from >> \var\lib\cassandra\commitlog\CommitLog-701533048437587.log to >> 703272597990002 failed >> at >> org.apache.cassandra.db.commitlog.CommitLogSegment.(CommitLogSegment.java:105) >> ... 5 more >> >> >> Anybody else seen this before ? >> >> >> -- >> Frank Hsueh | frank.hs...@gmail.com >> > > > > -- > Frank Hsueh | frank.hs...@gmail.com > -- Frank Hsueh | frank.hs...@gmail.com
