@Aaron This problem happens when you drop and recreate a keyspace with the same name and you do it very quickly. I have also filed a JIRA for it
https://issues.apache.org/jira/browse/CASSANDRA-5843 On Tue, Aug 6, 2013 at 10:31 AM, Keith Wright <kwri...@nanigans.com> wrote: > The file does not appear on disk and the permissions are definitely > correct. We have seen the file in snapshots. This is completely blocking > us from adding the new node. How can we recover? Just run repairs? > > Thanks > > From: Aaron Morton <aa...@thelastpickle.com> > Reply-To: "user@cassandra.apache.org" <user@cassandra.apache.org> > Date: Tuesday, August 6, 2013 4:06 AM > To: "user@cassandra.apache.org" <user@cassandra.apache.org> > Subject: Re: Unable to bootstrap node > > Caused by: java.io.FileNotFoundException: > /data/1/cassandra/data/rts/40301_feedProducts/rts-40301_feedProducts-ib-1-Data.db > (No such file or directory) > at java.io.RandomAccessFile.open(Native Method) > at java.io.RandomAccessFile.<init>(RandomAccessFile.java:233) > at > org.apache.cassandra.io.util.RandomAccessReader.<init>(RandomAccessReader.java:67) > at > org.apache.cassandra.io.compress.CompressedRandomAccessReader.<init> > > This is somewhat serous, specially if it's from the a bug in dropping > tables. Though I would expect that would show up for a lot of people. > > Does the file exist on disk? > Are the permissions correct ? > > IMHO you need to address this issue on the existing nodes before worrying > about the new node. > > Cheers > > ----------------- > Aaron Morton > Cassandra Consultant > New Zealand > > @aaronmorton > http://www.thelastpickle.com > > On 6/08/2013, at 1:25 PM, sankalp kohli <kohlisank...@gmail.com> wrote: > > Let me know if this fixes the problem? > > > On Mon, Aug 5, 2013 at 6:24 PM, sankalp kohli <kohlisank...@gmail.com>wrote: > >> So the problem is that when you dropped and recreated the table with the >> same name, some how the old CFStore object was not purged. So now there >> were two objects which caused same sstable to have 2 SSTableReader object. >> >> The fix is to find all nodes which is emitting this FileNotFound >> Exception and restart them. >> >> In your case, restart the node which is serving the data and emitting >> FileNotFound exception. >> >> Once this is up, again restart the bootstrapping node with bootstrap >> argument. Now it will successfully stream the data. >> >> >> On Mon, Aug 5, 2013 at 6:08 PM, Keith Wright <kwri...@nanigans.com>wrote: >> >>> >>> Yes we likely dropped and recreated tables. If we stop the sending node, >>> what will happen to the bootstrapping node? >>> >>> sankalp kohli <kohlisank...@gmail.com> wrote: >>> >>> >>> Hi, >>> The problem is that the node sending the stream is hitting this >>> FileNotFound exception. You need to restart this node and it should fix the >>> problem. >>> >>> Are you seeing lot of FileNotFoundExceptions? Did you do any schema >>> change recently? >>> >>> Sankalp >>> >>> >>> On Mon, Aug 5, 2013 at 5:39 PM, Keith Wright <kwri...@nanigans.com>wrote: >>> >>>> Hi all, >>>> >>>> I have been trying to bootstrap a new node into my 7 node 1.2.4 C* >>>> cluster with Vnodes RF3 with no luck. It gets close to completing and then >>>> the streaming just stalls with streaming at 99% from 1 or 2 nodes. >>>> Nodetool netstats shows the items that have yet to stream but the logs on >>>> the new node do not show any errors. I tried shutting down then node, >>>> clearing all data/commit logs/caches, and re-boot strapping with no luck. >>>> The nodes that are hanging sending the data only have the error below but >>>> that's related to compactions (see below) although it is one of the files >>>> that is waiting to be sent. I tried nodetool scrub on the column family >>>> with the missing item but got an error indicating it could not get a hard >>>> link. Any ideas? We were able to bootstrap one of the new nodes with no >>>> issues but this other one has been a real pain. Note that when the new >>>> node is joining the cluster, it does not appear in nodetool status. Is >>>> that expected? >>>> >>>> Thanks all, my next step is to try getting a new IP for this machine, >>>> my thought being that the cluster doesn't like me continuing to attempt to >>>> bootstrap the node repeatedly each time getting a new host id. >>>> >>>> [kwright@lxpcas008 ~]$ nodetool netstats | grep >>>> rts-40301_feedProducts-ib-1-Data.db >>>> rts: >>>> /data/1/cassandra/data/rts/40301_feedProducts/rts-40301_feedProducts-ib-1-Data.db >>>> sections=73 progress=0/1884669 - 0% >>>> >>>> ERROR [ReadStage:427] 2013-08-05 23:23:29,294 CassandraDaemon.java >>>> (line 174) Exception in thread Thread[ReadStage:427,5,main] >>>> java.lang.RuntimeException: java.io.FileNotFoundException: >>>> /data/1/cassandra/data/rts/40301_feedProducts/rts-40301_feedProducts-ib-1-Data.db >>>> (No such file or directory) >>>> at >>>> org.apache.cassandra.io.compress.CompressedRandomAccessReader.open(CompressedRandomAccessReader.java:46) >>>> at >>>> org.apache.cassandra.io.util.CompressedSegmentedFile.createReader(CompressedSegmentedFile.java:57) >>>> at >>>> org.apache.cassandra.io.util.PoolingSegmentedFile.getSegment(PoolingSegmentedFile.java:41) >>>> at >>>> org.apache.cassandra.io.sstable.SSTableReader.getFileDataInput(SSTableReader.java:976) >>>> at >>>> org.apache.cassandra.db.columniterator.SSTableNamesIterator.createFileDataInput(SSTableNamesIterator.java:98) >>>> at >>>> org.apache.cassandra.db.columniterator.SSTableNamesIterator.read(SSTableNamesIterator.java:117) >>>> at >>>> org.apache.cassandra.db.columniterator.SSTableNamesIterator.<init>(SSTableNamesIterator.java:64) >>>> at >>>> org.apache.cassandra.db.filter.NamesQueryFilter.getSSTableColumnIterator(NamesQueryFilter.java:81) >>>> at >>>> org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:68) >>>> at >>>> org.apache.cassandra.db.CollationController.collectTimeOrderedData(CollationController.java:133) >>>> at >>>> org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:65) >>>> at >>>> org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1357) >>>> at >>>> org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1214) >>>> at >>>> org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1126) >>>> at org.apache.cassandra.db.Table.getRow(Table.java:347) >>>> at >>>> org.apache.cassandra.db.SliceByNamesReadCommand.getRow(SliceByNamesReadCommand.java:64) >>>> at >>>> org.apache.cassandra.db.ReadVerbHandler.doVerb(ReadVerbHandler.java:44) >>>> at >>>> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:56) >>>> at >>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) >>>> at >>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) >>>> at java.lang.Thread.run(Thread.java:722) >>>> Caused by: java.io.FileNotFoundException: >>>> /data/1/cassandra/data/rts/40301_feedProducts/rts-40301_feedProducts-ib-1-Data.db >>>> (No such file or directory) >>>> at java.io.RandomAccessFile.open(Native Method) >>>> at java.io.RandomAccessFile.<init>(RandomAccessFile.java:233) >>>> at >>>> org.apache.cassandra.io.util.RandomAccessReader.<init>(RandomAccessReader.java:67) >>>> at >>>> org.apache.cassandra.io.compress.CompressedRandomAccessReader.<init>(CompressedRandomAccessReader.java:75) >>>> at >>>> org.apache.cassandra.io.compress.CompressedRandomAccessReader.open(CompressedRandomAccessReader.java:42) >>>> ... 20 more >>>> >>> >>> >> > >
