[jira] [Commented] (CASSANDRA-7145) FileNotFoundException during compaction

2014-08-05 Thread Maxim Zakharenkov (JIRA)

[ 
https://issues.apache.org/jira/browse/CASSANDRA-7145?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14086556#comment-14086556
 ] 

Maxim Zakharenkov commented on CASSANDRA-7145:
--

Today I had same problem on my local machine with one cassandra node, default 
configuration (2.0.8 and 2.0.9 versions). It was quite easily repeatable. I 
just execute quite heavy benchmark which writes new data as fast as it can into 
a single table. When it is running, execute nodetool compact for that table. 
After some 2 or 3 attempts normally you will get the issue.

 FileNotFoundException during compaction
 ---

 Key: CASSANDRA-7145
 URL: https://issues.apache.org/jira/browse/CASSANDRA-7145
 Project: Cassandra
  Issue Type: Bug
 Environment: CentOS 6.3, Datastax Enterprise 4.0.1 (Cassandra 2.0.5), 
 Java 1.7.0_55
Reporter: PJ
 Attachments: compaction - FileNotFoundException.txt, repair - 
 RuntimeException.txt, startup - AssertionError.txt


 I can't finish any compaction because my nodes always throw a 
 FileNotFoundException. I've already tried the following but nothing helped:
 1. nodetool flush
 2. nodetool repair (ends with RuntimeException; see attachment)
 3. node restart (via dse cassandra-stop)
 Whenever I restart the nodes, another type of exception is logged (see 
 attachment) somewhere near the end of startup process. This particular 
 exception doesn't seem to be critical because the nodes still manage to 
 finish the startup and become online.
 I don't have specific steps to reproduce the problem that I'm experiencing 
 with compaction and repair. I'm in the middle of migrating 4.8 billion rows 
 from MySQL via SSTableLoader. 
 Some things that may or may not be relevant:
 1. I didn't drop and recreate the keyspace (so probably not related to 
 CASSANDRA-4857)
 2. I do the bulk-loading in batches of 1 to 20 millions rows. When a batch 
 reaches 100% total progress (i.e. starts to build secondary index), I kill 
 the sstableloader process and cancel the index build
 3. I restart the nodes occasionally. It's possible that there is an on-going 
 compaction during one of those restarts.
 Related StackOverflow question (mine): 
 http://stackoverflow.com/questions/23435847/filenotfoundexception-during-compaction



--
This message was sent by Atlassian JIRA
(v6.2#6252)


[jira] [Comment Edited] (CASSANDRA-7145) FileNotFoundException during compaction

2014-08-05 Thread Maxim Zakharenkov (JIRA)

[ 
https://issues.apache.org/jira/browse/CASSANDRA-7145?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14086556#comment-14086556
 ] 

Maxim Zakharenkov edited comment on CASSANDRA-7145 at 8/5/14 6:30 PM:
--

Today I had same problem on my local machine with one cassandra node, default 
configuration (2.0.8 and 2.0.9 versions). It was quite easily repeatable. I 
just executed quite heavy benchmark which writes new data as fast as it can 
into a single table. When it was running, I execute nodetool compact for that 
table. First time I got the issue immediately. After some 2 or 3 attempts I got 
the issue issue. Then later it was quite hard to repeat (maybe due to some file 
system caching, have no idea)


was (Author: maxim.zakharenkov):
Today I had same problem on my local machine with one cassandra node, default 
configuration (2.0.8 and 2.0.9 versions). It was quite easily repeatable. I 
just execute quite heavy benchmark which writes new data as fast as it can into 
a single table. When it is running, execute nodetool compact for that table. 
After some 2 or 3 attempts normally you will get the issue.

 FileNotFoundException during compaction
 ---

 Key: CASSANDRA-7145
 URL: https://issues.apache.org/jira/browse/CASSANDRA-7145
 Project: Cassandra
  Issue Type: Bug
 Environment: CentOS 6.3, Datastax Enterprise 4.0.1 (Cassandra 2.0.5), 
 Java 1.7.0_55
Reporter: PJ
 Attachments: compaction - FileNotFoundException.txt, repair - 
 RuntimeException.txt, startup - AssertionError.txt


 I can't finish any compaction because my nodes always throw a 
 FileNotFoundException. I've already tried the following but nothing helped:
 1. nodetool flush
 2. nodetool repair (ends with RuntimeException; see attachment)
 3. node restart (via dse cassandra-stop)
 Whenever I restart the nodes, another type of exception is logged (see 
 attachment) somewhere near the end of startup process. This particular 
 exception doesn't seem to be critical because the nodes still manage to 
 finish the startup and become online.
 I don't have specific steps to reproduce the problem that I'm experiencing 
 with compaction and repair. I'm in the middle of migrating 4.8 billion rows 
 from MySQL via SSTableLoader. 
 Some things that may or may not be relevant:
 1. I didn't drop and recreate the keyspace (so probably not related to 
 CASSANDRA-4857)
 2. I do the bulk-loading in batches of 1 to 20 millions rows. When a batch 
 reaches 100% total progress (i.e. starts to build secondary index), I kill 
 the sstableloader process and cancel the index build
 3. I restart the nodes occasionally. It's possible that there is an on-going 
 compaction during one of those restarts.
 Related StackOverflow question (mine): 
 http://stackoverflow.com/questions/23435847/filenotfoundexception-during-compaction



--
This message was sent by Atlassian JIRA
(v6.2#6252)