[no subject]
Hello all, The environment: I have a 6 node Cassandra cluster. On each node I have: - 32 G RAM - 24 G RAM for cassa - ~150 - 200 MB/s disk speed - tomcat 6 with axis2 webservice that uses the datastax java driver to make asynch reads / writes - replication factor for the keyspace is 3 All nodes in the same data center The clients that read / write are in the same datacenter so network is Gigabit. Writes are performed via exposed methods from Axis2 WS . The Cassandra Java driver uses the round robin load balancing policy so all the nodes in the cluster should be hit with write requests under heavy write or read load from multiple clients. I am monitoring all nodes with JConsole from another box. The problem: When wrinting to a particular column family, only 3 nodes have high CPU load ~ 80 - 99 %. The remaining 3 are at ~2 - 10 % CPU. During writes, reads timeout. I need more speed for both writes of reads. Due to the fact that 3 nodes barely have CPU activity leads me to think that the whole potential for C* is not touched. I am running out of ideas... If further details about the environment I can provide them. Thank you very much.
Re:
Thanks , Edward and David, Your contribution lead me to the conclusion. Unknown to me, the partition key had 1 value. So of course all info was stored on a single node. Having replication factor 3 lead to having 99 % CPU on 3 machines. Regarding RAM we have so much CPU power that I actualy run a tomcat server on each node and tomcat prepares data for queries, sends them, retrieves results, processes them and sends responses to clients. We are running 6 Intel® Xeon® E3-1270 v3 Quad-Core Haswell Hyper-Threading. So there is CPU power there to handle high java heap IMO. Thank you all for your prompt responses. On Thursday, March 13, 2014 1:38 AM, David McNelis wrote: Not knowing anything about your data structure (to expand on what Edward said), you could be running into something where you've got some hot keys that are getting the majority of writes during those heavily loads more specifically I might look for a single key that you're writing, since you're RF=3 and you have 3 nodes specifically that are causing problems. On Wed, Mar 12, 2014 at 7:28 PM, Russ Bradberry wrote: I wouldn't go above 8G unless you have a very powerful machine that can keep the GC pauses low. > >Sent from my iPhone > >On Mar 12, 2014, at 7:11 PM, Edward Capriolo wrote: > > >That is too much ram for cassandra make that 6g to 10g. >> >>The uneven perf could be because your requests do not shard evenly. >> >>On Wednesday, March 12, 2014, Batranut Bogdan wrote: >>> Hello all, >>> >>> The environment: >>> >>> I have a 6 node Cassandra cluster. On each node I have: >>> - 32 G RAM >>> - 24 G RAM for cassa >>> - ~150 - 200 MB/s disk speed >>> - tomcat 6 with axis2 webservice that uses the datastax java driver to make >>> asynch reads / writes >>> - replication factor for the keyspace is 3 >>> >>> All nodes in the same data center >>> The clients that read / write are in the same datacenter so network is >>> Gigabit. >>> >>> Writes are performed via exposed methods from Axis2 WS . The Cassandra Java >>> driver uses the round robin load balancing policy so all the nodes in the >>> cluster should be hit with write requests under heavy write or read load >>> from multiple clients. >>> >>> I am monitoring all nodes with JConsole from another box. >>> >>> The problem: >>> >>> When wrinting to a particular column family, only 3 nodes have high CPU load >>> ~ 80 - 99 %. The remaining 3 are at ~2 - 10 % CPU. During writes, reads >>> timeout. >>> >>> I need more speed for both writes of reads. Due to the fact that 3 nodes >>> barely have CPU activity leads me to think that the whole potential for C* >>> is not touched. >>> >>> I am running out of ideas... >>> >>> If further details about the environment I can provide them. >>> >>> >>> Thank you very much. >> >>-- >>Sorry this was sent from mobile. Will do less grammar and spell check than >>usual. >>
Cassandra slow on some reads
Hello all, Here is the environment: I have a 6 node Cassandra cluster. On each node I have: - 32 G RAM - 24 G RAM for cassa - ~150 - 200 MB/s disk speed - tomcat 6 with axis2 webservice that uses the datastax java driver to make asynch reads / writes - replication factor for the keyspace is 3 (I know that there is a lot of heap but I also have write heavy tasks and I want them to get into mem fast) . All nodes in the same data center The clients that read / write are in the same datacenter so network is Gigabit. The table structure is like this: PK(key String , timestam int, column1 string, col2 string) , list1 , list 2, list 3 . There are about 300 milions individual keys. There are about 100 timestamps for each key now, so the rows will get wider as time passes. I am using datastax java driver to query the cluster. I have ~450 queries that are like this: SELECT * FROM table where key = 'some string' and ts = some value; some value is close to present time. The problem: About 10 - 20 % of these queries take more than 5 seconds to execute, in fact, the majority of those take around 10 seconds. When investigating I saw that if I have a slow response and I redo the query it will finish in 8 - 10 MILIseconds like the rest of the queries that I have. I could not see using JConsole any spikes in CPU / memory when executing the queries. The rise in resource consumtion is very small on all nodes on the cluster. I expect such delays to be generated by a BIG increase in resource consumption. Any comments will be appreciated. Thank you.
Re: Cassandra slow on some reads
Ok will try to reduce heap and see what happens. Thanks guys. I' get back with conclusions.https://overview.mail.yahoo.com?.src=iOS";>Sent from Yahoo Mail for iPhone
Re: Cassandra slow on some reads
Well the problem still persists. Giving cassandra 12G of heap and having a look at the table I saw that caching='KEYS_ONLY' . Did not find how to disable caching for rows (I'm not sure if setting to 0 will disable it) On Friday, March 14, 2014 3:14 PM, Andras Szerdahelyi wrote: Is row cache enabled on this CF? Try disabling it. Seems like you might have a very wide row there. Can you grep for GCInspector in your Cassandra log? 24G might be a bit too much for the Cassandra JVM, bogging down GC, and not leaving much to page cache ( 32G -24G - Tomcat ). I don’t quite understand your reasoning here: > (I know that there is a lot of heap but I also have write heavy tasks and I >want them to get into mem fast) . So, I would try with default cassandra-env.sh JVM params too From: Batranut Bogdan Reply-To: "user@cassandra.apache.org" , Batranut Bogdan Date: Friday 14 March 2014 13:50 To: "user@cassandra.apache.org" Subject: Cassandra slow on some reads Hello all, Here is the environment: I have a 6 node Cassandra cluster. On each node I have: - 32 G RAM - 24 G RAM for cassa - ~150 - 200 MB/s disk speed - tomcat 6 with axis2 webservice that uses the datastax java driver to make asynch reads / writes - replication factor for the keyspace is 3 (I know that there is a lot of heap but I also have write heavy tasks and I want them to get into mem fast) . All nodes in the same data center The clients that read / write are in the same datacenter so network is Gigabit. The table structure is like this: PK(key String , timestam int, column1 string, col2 string) , list1 , list 2, list 3 . There are about 300 milions individual keys. There are about 100 timestamps for each key now, so the rows will get wider as time passes. I am using datastax java driver to query the cluster. I have ~450 queries that are like this: SELECT * FROM table where key = 'some string' and ts = some value; some value is close to present time. The problem: About 10 - 20 % of these queries take more than 5 seconds to execute, in fact, the majority of those take around 10 seconds. When investigating I saw that if I have a slow response and I redo the query it will finish in 8 - 10 MILIseconds like the rest of the queries that I have. I could not see using JConsole any spikes in CPU / memory when executing the queries. The rise in resource consumtion is very small on all nodes on the cluster. I expect such delays to be generated by a BIG increase in resource consumption. Any comments will be appreciated. Thank you.
Re: Cassandra slow on some reads
Hello, I can't go this way... this cf will be used for time ranges. On Friday, March 14, 2014 5:10 PM, "Laing, Michael" wrote: If you do not need to do range queries on your 'timestam' (ts) column - and if you can change your schema (big if...), then you could move 'timestam' into the partition key like this (using your notation): PK((key String , timestam int), column1 string, col2 string) , list1 , list 2, list 3 . Now the select query you showed should execute more consistently. But of course something else might break...! ml On Fri, Mar 14, 2014 at 8:50 AM, Batranut Bogdan wrote: Hello all, > > >Here is the environment: > > >I have a 6 node Cassandra cluster. On each node I have: >- 32 G RAM >- 24 G RAM for cassa >- ~150 - 200 MB/s disk speed >- tomcat 6 with axis2 webservice that uses the datastax java driver to make >asynch reads / writes >- replication factor for the keyspace is 3 > >(I know that there is a lot of heap but I also have write heavy tasks and I want them to get into mem fast) . > >All nodes in the same data center >The clients that read / write are in the same datacenter so network is Gigabit. > > > >The table structure is like this: PK(key String , timestam int, column1 >string, col2 string) , list1 , list 2, list 3 . >There are about 300 milions individual keys. >There are about 100 timestamps for each key now, so the rows will get wider as >time passes. > > >I am using datastax java driver to query the cluster. > > >I have ~450 queries that are like this: SELECT * FROM table where key = 'some >string' and ts = some value; some value is close to present time. > > >The problem: > > >About 10 - 20 % of these queries take more than 5 seconds to execute, in fact, >the majority of those take around 10 seconds. >When investigating I saw that if I have a slow response and I redo the query >it will finish in 8 - 10 MILIseconds like the rest of the queries that I have. >I could not see using JConsole any spikes in CPU / memory when executing the >queries. The rise in resource consumtion is very small on all nodes on the >cluster. I expect such delays to be generated by a BIG increase in resource >consumption. > > >Any comments will be appreciated. > > >Thank you. > > > > > > >
Re: Cassandra slow on some reads
Hello, Yes for the first query the server can be slow but no matter what is should not take 10 seconds to get one key from a column family. I can see this happening if I have times like 20 ms for some queries and let's say 300 ms for uncached ones. But I have responses that take from 8 to 20 ms (420 out of 450 queries) and 30 response that take from 5 to 13 seconds. It is this HUGE difference that bothers me. On Friday, March 14, 2014 9:18 PM, Manoj Khangaonkar wrote: > > > >I have ~450 queries that are like this: SELECT * FROM table where key = 'some >string' and ts = some value; some value is close to present time. > > >The problem: > > >About 10 - 20 % of these queries take more than 5 seconds to execute, in fact, >the majority of those take around 10 seconds. >When investigating I saw that if I have a slow response and I redo the query >it will finish in 8 - 10 MILIseconds like the rest of the queries that I have. > > > Most likely for the first slow query, the server has to go looking in SSTables on disk. 2nd fast response is returned from memory. I am not sure how much data each node is managing. But try adding more nodes , so that each node is managing fewer keys. More nodes with recommended heap size of 8G is better than fewer nodes with bigger heap size. regards -- http://khangaonkar.blogspot.com/
Re: Cassandra slow on some reads
Whooops On one of the nodes when running my tests I found an exception java FileNotFoundException : file -Data.db not found at org.apache.cassandra.io.compress.CompressedThrottledReader.open(CompressedThrottledReader.java:52 also got another one coming from java.io.RandomAccessFile.open(native method) java.io.RandomAccessFile.. (RandomAccesFile.java:241) org.apache.cassandra.io.util.RandomAccessReader.(RandomAccessReader.java:58) On Saturday, March 15, 2014 10:52 PM, Benedict Elliott Smith wrote: Post the server logs and traces of one of the lengthy queries? On 15 Mar 2014 20:49, "Batranut Bogdan" wrote: Hello, > > >Yes for the first query the server can be slow but no matter what is should >not take 10 seconds to get one key from a column family. I can see this >happening if I have times like 20 ms for some queries and let's say 300 ms for >uncached ones. But I have responses that take from 8 to 20 ms (420 out of 450 >queries) and 30 response that take from 5 to 13 seconds. It is this HUGE >difference that bothers me. > > > >On Friday, March 14, 2014 9:18 PM, Manoj Khangaonkar >wrote: > > >> >> >> >>I have ~450 queries that are like this: SELECT * FROM table where key = 'some >>string' and ts = some value; some value is close to present time. >> >> >>The problem: >> >> >>About 10 - 20 % of these queries take more than 5 seconds to execute, in >>fact, the majority of those take around 10 seconds. >>When investigating I saw that if I have a slow response and I redo the query >>it will finish in 8 - 10 MILIseconds like the rest of the queries that I >>have. >> >> >> > >Most likely for the first slow query, the server has to go looking in SSTables >on disk. 2nd fast response is returned from memory. I am not sure how much >data each node is managing. But try adding more nodes , so that each node is >managing fewer keys. More nodes with recommended heap size of 8G is better >than fewer nodes with bigger heap size. > > >regards > > >-- >http://khangaonkar.blogspot.com/ > >
Cassandra FileNotFoundException .db file not found
Hello, A investigation on one of the nodes reveals that I am missing several .db files. I got FileNotFoundException in the logs on one node in the cluster. How can I fix this? nodetool repair doesn't seem to work in this case. Thank you.
FileNotFoundException ...-Data.db
Hello all, I am reposting a question about missing files on one of the nodes of my cluster. After I first saw that I was missing a ...-Data.db file, I decomissioned the node, deleted the data and added it back into the cluster. Now I see again that I am missing some files. Do you have an ideea on how to fix this?
Re: FileNotFoundException ...-Data.db
I am running 2.0.6 and I use /etc/init.d/cassandra start / stop . Also before stopping I do : nodetool disablegossip nodetool disablethrift nodetool drain after that /etc/init.d/cassandra stop On Monday, March 24, 2014 9:48 PM, Robert Coli wrote: On Mon, Mar 24, 2014 at 9:41 AM, Batranut Bogdan wrote: I am reposting a question about missing files on one of the nodes of my cluster. > >After I first saw that I was missing a ...-Data.db file, I decomissioned the >node, deleted the data and added it back into the cluster. Now I see again >that I am missing some files. > > >Do you have an ideea on how to fix this? What version of Cassandra? How are you stop and starting cassandra? =Rob
Re: FileNotFoundException ...-Data.db
I believe that no matter how I stop cassandra, I should not be missing data. Even if compaction is in progress. As far as I can tell during compaction cassa reads from some files and creates a new temp file. Here I believe that it stores the info and after the new file is complete it should rename maybe do some internal work about changes and then then delete the original files. Now if a node just crashes for whatever reason, the initial data or the new data should be there. When running nodetool drain , for a while, I cannot write anything in the linux terminal. When the cursor is again available I assume that nodetool has finished execution. After that I stop cassandra. These missing files are on the same node that previously I decomissioned (due to the same problem), deleted all data and readded it to the cluster. So in this case all data should be recreated from the other nodes in the cluster. On Tuesday, March 25, 2014 9:53 PM, Duncan Sands wrote: Hi, On 25/03/14 19:30, Robert Coli wrote: > On Tue, Mar 25, 2014 at 5:36 AM, Batranut Bogdan <mailto:batra...@yahoo.com>> wrote: > > I am running 2.0.6 and I use /etc/init.d/cassandra start / stop . Also > before stopping I do : > nodetool disablegossip > nodetool disablethrift > nodetool drain > after that /etc/init.d/cassandra stop > > > This seems reasonable/best practice. Do you verify the drain has completed > before stopping? > > If I were you, I'd be looking for files with the name of the missing file in > logs previous to the restart. It is pretty unusual for SSTables or their > components to go missing. I've noticed that compaction can continue to run after the drain has completed, and restarting while in the middle of the compaction can then cause similar problems. Ciao, Duncan.
Re: FileNotFoundException ...-Data.db
Well grepping the logs for a file resulted in this: During startup that file was opened OK. During runtime the compactionexecutor tried to compact more files that included this one but threw and exception for another one that is first in that list. So I do not have more info. In one particular day I cleared the logs. But the file I am talking about opened ok during startup.(I have that in the log BATCHOPEN section) So not much more info on this. Any ideeas on what to do ? Decommission , delete the /data dir and readd to cluster? For that keyspace I have a replication factor of 3 so data should be available on other nodes. On Wednesday, March 26, 2014 12:36 AM, Robert Coli wrote: On Tue, Mar 25, 2014 at 3:24 PM, James Rothering wrote: What message would be on the log, or not on the log, what would you be grep-ing for? You'd be looking for the broken SSTable to appear in either a MemtableFlusher line or a compaction line. Also, why were you interested in startup procedure? Is there some "best practices: for that which could impact this? Startup procedure? I was asking details about shutdown procedure... ? =Rob
Compaction throrheling
Hello allCan someone please explain to me what compaction throtheling does ? I am reffering to the yaml parameter. I have changed it from the default 16 to 160 but i see no improvement. I have a cluster with hdds. I might be missing something ...Thankshttps://overview.mail.yahoo.com?.src=iOS";>Sent from Yahoo Mail for iPhone
Re: Load balancing issue with virtual nodes
Htop is not the only tool for this . Cassandra will hit io bottlnecks before cpu (on faster cpus) . A simple solution is to check the size of the data dir on the boxes. If you have aprox the same size then cassandra is wrinting in the whole cluster. Check how the data dir size changes when importing use iostat to see how loaded the hdds are. In my experience I rarely look at cpu load. Since i have hdds i/o is the thing that i keep an eye on. Also try secvential strings when inserting but this should behave aprox the same as for random ones.https://overview.mail.yahoo.com?.src=iOS";>Sent from Yahoo Mail for iPhone
Re: Compaction throrheling
Is this a setting that will have an impact only on fast cpu + ssd ? On Thursday, April 24, 2014 11:52 PM, Robert Coli wrote: On Thu, Apr 24, 2014 at 12:50 PM, Batranut Bogdan wrote: Can someone please explain to me what compaction throtheling does ? I am reffering to the yaml parameter. I have changed it from the default 16 to 160 but i see no improvement. I have a cluster with hdds. I might be missing something ... > It sets the maximum throughput used for compaction operations, across all compactions. If your system is i/o or CPU bound on compaction, raising it or setting it to 0 (unlimited) will have no effect. =Rob
Re: Compaction throrheling
I have CPU to spare but reached hdd limit. Well this might deserve it's own conversation thread but I did reach the limit of IO after after using a wide row and counter columnfamily ... def looks like this: a int , b int, c string , timestamp int, d counter, e counter f counter, g counter PK ((a,b,c) ts) After 200 mil PKs and 6 Timestamps for each of them my whole 8 core XEON, 32 Gb ram, 6 server cluster went into medium to high load (according to opscenter) and the hdds load was ~ 100% according to iostat. But that is something else. I asked the question since my cluster is not under heavy load and I see compactions done with 4 MBs / sec (for big files according to my observations) and compactions done with 40 MBs/s for smaller ones. I need C* to do compactions as fast as possible if the system is not under load. On Friday, April 25, 2014 12:06 AM, Robert Coli wrote: On Thu, Apr 24, 2014 at 1:54 PM, Batranut Bogdan wrote: Is this a setting that will have an impact only on fast cpu + ssd ? It's a setting that will only have impact if you have CPU or IO to spare. You don't need a fast CPU or SSD to meet those conditions. =Rob
Re: Load balancing issue with virtual nodes
I don't know about hector but the datastax java driver needs just one ip from the cluster and it will discover the rest of the nodes. Then by default it will do a round robin when sending requests. So if Hector does the same the patterb will againg appear. Did you look at the size of the dirs? That documentation is for C* 0.8. It's old. But depending on your boxes you might reach CPU bottleneck. Might want to google for write path in cassandra.. According to that, there is not much to do when writes come in... On Friday, April 25, 2014 12:00 AM, DuyHai Doan wrote: I did some experiments. Let's say we have node1 and node2 First, I configured Hector with node1 & node2 as hosts and I saw that only node1 has high CPU load To eliminate the "client connection" issue, I re-test with only node2 provided as host for Hector. Same pattern. CPU load is above 50% on node1 and below 10% on node2. It means that node2 is playing as coordinator and forward many write/read request to node1 Why did I look at CPU load and not iostat & al ? Because I have a very intensive write work load with read-only-once pattern. I've read here (http://www.datastax.com/docs/0.8/cluster_architecture/cluster_planning) that heavy write in C* is more CPU bound but maybe the info may be outdated and no longer true Regards Duy Hai DOAN On Thu, Apr 24, 2014 at 10:00 PM, Michael Shuler wrote: On 04/24/2014 10:29 AM, DuyHai Doan wrote: > > Client used = Hector 1.1-4 >> Default Load Balancing connection policy >> Both nodes addresses are provided to Hector so according to its >>connection policy, the client should switch alternatively between both nodes >> > OK, so is only one connection being established to one node for one bulk write operation? Or are multiple connections being made to both nodes and writes performed on both? > >-- >Michael >
Old tables still exist
Hello all, I have a question. I looked in the keyspace directory and saw that old dirs still exist after dropping the corresponding table. eg drop table t1 and in /data/keyspace I still see the t1 dir. Is there a way to get cassandra to remove these old and unused tables?
Re: Old tables still exist
Well this is not good at all... During the development of our entire system we generated huge tables (GB or TB) then had to drop them since we changed the definition. This takes up space. There is a constant write on the system so we want to optimize our cluster. Basicaly not expanding untill the real need is there. Thanks for the answer. On Thursday, May 1, 2014 1:06 AM, Robert Coli wrote: On Wed, Apr 30, 2014 at 3:00 PM, Batranut Bogdan wrote: Is there a way to get cassandra to remove these old and unused tables? Those directories have to exist after DROP because snapshots taken when you drop, and the snapshots are in a subdirectory of those directories. nodetool clearsnapshot Will clear the snapshots; I don't believe there is any way to make cassandra actually clean up the directories. If I were you I might file a trivial feature request in the Cassandra project JIRA. =Rob
cassandra snapshots
Hello all I have a big col family and I see that cassandra is taking snapshots for it. I do not have incremental enabled. What are the triggers that start the process of taking a snapshot? Is is automatic ? Thanks
Re: cassandra snapshots
Hello Robert, Neither of those actions were taken on that cf as far as I know. In that cf we only insert historical data. No deletes no drops / truncates. Thanks On Monday, May 5, 2014 10:50 PM, Robert Coli wrote: On Mon, May 5, 2014 at 12:48 PM, Batranut Bogdan wrote: I have a big col family and I see that cassandra is taking snapshots for it. I do not have incremental enabled. What are the triggers that start the process of taking a snapshot? Is is automatic ? It's automatic if you DROP or TRUNCATE a columnfamily or keyspace. =Rob
Cassandra counter column family performance
Hello all, I have a counter CF defined as pk text PRIMARY KEY, a counter, b counter, c counter, d counter After inserting a few million keys... 55 mil, the performance goes down the drain, 2-3 nodes in the cluster are on medium load, and when inserting batches of same lengths writes take longer and longer until the whole cluster becomes loaded and I get a lot of TExceptions... and the cluster becomes unresponsive. Did anyone have the same problem? Feel free to comment and share experiences about counter CF performance.
Couter column family performance problems
Hello all, I have a counter CF defined as pk text PRIMARY KEY, a counter, b counter, c counter, d counter After inserting a few million keys... 55 mil, the performance goes down the drain, 2-3 nodes in the cluster are on medium load, and when inserting batches of same lengths writes take longer and longer until the whole cluster becomes loaded and I get a lot of TExceptions... and the cluster becomes unresponsive. Did anyone have the same problem? Feel free to comment and share experiences about counter CF performance.
Cassandra select results differs
Hello all,
I have a CF
CREATE TABLE cf (
a text,
b int,
c int,
d int,
e int,
PRIMARY KEY (a)
) WITH
bloom_filter_fp_chance=0.01 AND
caching='KEYS_ONLY' AND
comment='' AND
dclocal_read_repair_chance=0.00 AND
gc_grace_seconds=864000 AND
index_interval=128 AND
read_repair_chance=0.10 AND
replicate_on_write='true' AND
populate_io_cache_on_flush='false' AND
default_time_to_live=0 AND
speculative_retry='99.0PERCENTILE' AND
memtable_flush_period_in_ms=0 AND
compaction={'class': 'SizeTieredCompactionStrategy'} AND
compression={'sstable_compression': 'LZ4Compressor'};
there are 3 rows there
when I do select * from cf;
I get random number of rows in results... like 0 , 2, or 3 rows.
What is the problem?
Re: Cassandra select results differs
I have cron jobs that repair every week. node 1 - monday , node 2 tuesday .
On Wednesday, July 23, 2014 7:52 PM, Russell Bradberry
wrote:
sounds like you may need to run a repair
On July 23, 2014 at 12:50:23 PM, Batranut Bogdan (batra...@yahoo.com) wrote:
Hello all,
I have a CF
CREATE TABLE
cf (
a
text,
b
int,
c
int,
d
int,
e
int,
PRIMARY
KEY (a)
) WITH
bloom_filter_fp_chance=0.01 AND
caching='KEYS_ONLY' AND
comment='' AND
dclocal_read_repair_chance=0.00 AND
gc_grace_seconds=864000 AND
index_interval=128 AND
read_repair_chance=0.10 AND
replicate_on_write='true' AND
populate_io_cache_on_flush='false' AND
default_time_to_live=0 AND
speculative_retry='99.0PERCENTILE' AND
memtable_flush_period_in_ms=0 AND
compaction={'class': 'SizeTieredCompactionStrategy'} AND
compression={'sstable_compression': 'LZ4Compressor'};
there are 3 rows there
when I do select * from cf;
I get random number of rows in results...
like 0 , 2, or 3 rows.
What is the problem?
Cassandra corrupt column family
Hello all, I have altered a table in cassandra and on one node it somehow got corrupted. I the changes did not propagate ok. Ran repair keyspace columnfamily... noting changed... Is there a way to repair this?
Cassandra COPY to CSV and DateTieredCompactionStrategy
Hello all, I have a few things that I need to understand. 1 . Here is the scenario: we have a HUGE cf where there are daily writes it is like a time series. Now we want to change the type of a column in primary key. What I think we can do is to export to csv, create the new table and write back the transformed data. But here is the catch... the constant writes in the cf. I assume that by the time the export finishes, new data will be inserted in the source cf. So is there a tool that will export data without having to stop the writes? 2. I have seen that there is a new compaction strategy: DTCS, that will better fit historical data. This compaction strategy will take into account writeTime() of an entry or will it be smart enough and detect that the column family is a time series and take into account those timestamps when creating the time windows? I am asking this since when we write to the cf, the time for a particular record is 00:00h of a given day, so basicaly all entries have the same timestamp value in the cf but of course different writeTime() .
Date Tiered Compaction Strategy and collections
Hello all, If one has a table like this:id text,ts timestampvalues list PK (id,ts) How will the DTCS work? I am asking this because the writeTime() function does not work on collections.
Key Cache Questions
Hello all,I just read that the default size of the Key cache is 100 MB. Is it stored in memory or disk?
Datastax Cassandra Java Driver executeAsynch question.
Hello all, In my implementation of the FutureCallBack<> interface in the onSuccess method, I put Thread.currentThread.getName(). What I saw was that there is a ThreadPool... That is all fine, but seems to me that the pool does not have that many threads. About 10 from my observations - I did not bother to get the exact number. Why? Well because when I query cassandra I have a list of about 6000 cf keys. I traverse the list and executeAsynch a prepared statement for each of these. I have seen that during the execution the driver is waiting for free threads to continue if all the threads in the pool are waiting for response from C*. How can I increase the number of threads that the driver uses to query Cassandra?
2 Cassandra nodes slow
Hello, I have a cluster of 6 C* nodes. All machines have the same hardware. I have noticed in opscenter that when I start reading a lot from the cluster 2 nodes have read latencies, but the rest do not have such high values. The replication factor for the keyspace is 3. Also those 2 nodes have latencies when measuring the CF Local Read Latency so I know that the reads on the main col family are slow on these 2 nodes. Partitioning is good since in the column family there are a few million possible values.Another thing is that I have stopped these 2 nodes and ran a few selections for 6000 rows those were executed in about 15 seconds. I have changed the keys so that I avoid caching as much as possible.Also I have done the same test when running with all nodes and the execution of the selection was abou 22 seconds Any ideeas? I am thinking on rebuilding the data on the nodes... Stop the node , delete the data folder and restart... but I don't know the consequences
Re: 2 Cassandra nodes slow
Hello, The data distribution is ok, as I sayd there area few milion distinct values for the keys. I am running repairs like this :Node 1st runs repair on 1 and 15th of the monthNode 3rd runs repair on 3 and 15th of the month and so on... On Tuesday, January 13, 2015 1:47 PM, Rahul Neelakantan wrote: Is the data distribution OK? Have you tried running repairs? Rahul On Jan 13, 2015, at 5:01 AM, Batranut Bogdan wrote: Hello, I have a cluster of 6 C* nodes. All machines have the same hardware. I have noticed in opscenter that when I start reading a lot from the cluster 2 nodes have read latencies, but the rest do not have such high values. The replication factor for the keyspace is 3. Also those 2 nodes have latencies when measuring the CF Local Read Latency so I know that the reads on the main col family are slow on these 2 nodes. Partitioning is good since in the column family there are a few million possible values.Another thing is that I have stopped these 2 nodes and ran a few selections for 6000 rows those were executed in about 15 seconds. I have changed the keys so that I avoid caching as much as possible.Also I have done the same test when running with all nodes and the execution of the selection was abou 22 seconds Any ideeas? I am thinking on rebuilding the data on the nodes... Stop the node , delete the data folder and restart... but I don't know the consequences
Nodetool clearsnapshot
I have read that snapshots are basicaly symlinks and they do not take that much space.Why if I run nodetool clearsnapshot it frees a lot of space? I am seeing GBs freed...
Re: Nodetool clearsnapshot
OK Thanks, But I also read that repair will take a snapshot. Due to the fact that I have Replication factor 3 for my keyspace, I run nodetool clearsnapshot to keep disk space use to a minimum. Will this impact my repair? On Tuesday, January 13, 2015 4:19 PM, Jan Kesten wrote: Hi, I have read that snapshots are basicaly symlinks and they do not take that much space. Why if I run nodetool clearsnapshot it frees a lot of space? I am seeing GBs freed... both together makes sense. Creating a snaphot just creates links for all files unter the snapshot directory. This is very fast and takes no space. But those links are hard links, not symbolic ones. After a while your running cluster will compact some of its sstables and writing it to a new one as deleting the old ones. Now for example you had SSTable1..4 and a snapshot with the links to those four after compaction you will have one active SSTable5 which is newly written and consumes space. The snapshot-linked ones are still there, still consuming their space. Only when this snapshot is cleared you get your disk space back. HTH, Jan
Re: Nodetool clearsnapshot
Got it, Thank you! On Tuesday, January 13, 2015 5:00 PM, Yuki Morishita wrote: Snapshot during repair is automatically cleared if repair succeeds. Unfortunately, you have to delete it manually if repair failed or stalled. On Tue, Jan 13, 2015 at 8:30 AM, Batranut Bogdan wrote: > OK Thanks, > > But I also read that repair will take a snapshot. Due to the fact that I > have Replication factor 3 for my keyspace, I run nodetool clearsnapshot to > keep disk space use to a minimum. Will this impact my repair? > > > On Tuesday, January 13, 2015 4:19 PM, Jan Kesten > wrote: > > > Hi, > > I have read that snapshots are basicaly symlinks and they do not take that > much space. > Why if I run nodetool clearsnapshot it frees a lot of space? I am seeing GBs > freed... > > > both together makes sense. Creating a snaphot just creates links for all > files unter the snapshot directory. This is very fast and takes no space. > But those links are hard links, not symbolic ones. > > After a while your running cluster will compact some of its sstables and > writing it to a new one as deleting the old ones. Now for example you had > SSTable1..4 and a snapshot with the links to those four after compaction you > will have one active SSTable5 which is newly written and consumes space. The > snapshot-linked ones are still there, still consuming their space. Only when > this snapshot is cleared you get your disk space back. > > HTH, > Jan > > > > -- Yuki Morishita t:yukim (http://twitter.com/yukim)
Opscenter served reads / second
Hello, Is there a metric that will show how many reads per second C* serves? Read requests shows how many requests are issued to cassandra, but I want to know how many the cluster can actualy serve .
Adding new node - OPSCenter problems
Hello all, I have added new 3 nodes to existing cluster. I must point out that I have copied the cassandra yaml file, from an existing node and just changed listen_addres per instructions here: Adding nodes to an existing cluster | DataStax Cassandra 2.0 Documentation | | | | | | | | | Adding nodes to an existing cluster | DataStax Cassandra 2.0 DocumentationSteps to add nodes when using virtual nodes. | | | | View on www.datastax.com | Preview by Yahoo | | | | | Installed datastax agents but in OpsCenter I see the new nodes in a different, empty Cluster. Also opscenter does not see the datacenter for the new nodes. They are all in the same datacenter, and the name of the Cluster is the same for all 9 nodes. Any ideeas?
Re: Adding new node - OPSCenter problems
Hello, nodetool status shows existing nodes as UN and the new 3 as UJ . What is strange is that in the Owns column for the new 3 nodes I have ? instead of a percentage value. What I see is all are in rack1 in the Rack column. On Wednesday, February 11, 2015 4:50 PM, Carlos Rolo wrote: Hello, What is the output of nodetool status? All nodes should appear, otherwise there is some configuration error. Regards, Carlos Juzarte RoloCassandra Consultant Pythian - Love your data rolo@pythian | Twitter: cjrolo | Linkedin: linkedin.com/in/carlosjuzarteroloTel: 1649www.pythian.com On Wed, Feb 11, 2015 at 3:46 PM, Batranut Bogdan wrote: Hello all, I have added new 3 nodes to existing cluster. I must point out that I have copied the cassandra yaml file, from an existing node and just changed listen_addres per instructions here: Adding nodes to an existing cluster | DataStax Cassandra 2.0 Documentation | | | | | | | | | Adding nodes to an existing cluster | DataStax Cassandra 2.0 DocumentationSteps to add nodes when using virtual nodes. | | | | View on www.datastax.com | Preview by Yahoo | | | | | Installed datastax agents but in OpsCenter I see the new nodes in a different, empty Cluster. Also opscenter does not see the datacenter for the new nodes. They are all in the same datacenter, and the name of the Cluster is the same for all 9 nodes. Any ideeas? --
Added new nodes to cluster but no streams
Hello, I have added new nodes to the existing cluster. In Opscenter I do not see any streams... I presume that the new nodes get the data from the rest of the cluster via streams. The existing cluster has TB magnitude, and space used in the new nodes is ~90 GB. I must admit that I have restarted the new nodes several times after adding them . Does this affect boostrap? AFAIK the new nodes should start loading a part of all the data in the existing cluster. Is this normal?
Re: Added new nodes to cluster but no streams
Hello, When adding a new node to the cluster I need to wait for each node to receive all the data from other nodes in the cluster or just wait a few minutes before I start each node? On Thursday, February 12, 2015 7:21 PM, Robert Coli wrote: On Thu, Feb 12, 2015 at 3:20 AM, Batranut Bogdan wrote: I have added new nodes to the existing cluster. In Opscenter I do not see any streams... I presume that the new nodes get the data from the rest of the cluster via streams. The existing cluster has TB magnitude, and space used in the new nodes is ~90 GB. I must admit that I have restarted the new nodes several times after adding them . Does this affect boostrap? AFAIK the new nodes should start loading a part of all the data in the existing cluster. If it stays like this for a while, it sounds like your bootstraps have hung. Note that in general you should add nodes one at a time, especially if you are in a version without the fix for CASSANDRA-2434, in theory adding multiple nodes "at once" might contribute to their bootstraps hanging. Stop cassandra on the joining nodes, wipe/move aside their data directories, and try again one at a time. =Rob
Re: Added new nodes to cluster but no streams
Got it, thank you very much. On Friday, February 13, 2015 4:04 PM, Jens Rantil wrote: Hi Bastranut, A few minutes between each node will do. Cheers,Jens On Fri, Feb 13, 2015 at 1:12 PM, Batranut Bogdan wrote: Hello, When adding a new node to the cluster I need to wait for each node to receive all the data from other nodes in the cluster or just wait a few minutes before I start each node? On Thursday, February 12, 2015 7:21 PM, Robert Coli wrote: On Thu, Feb 12, 2015 at 3:20 AM, Batranut Bogdan wrote: I have added new nodes to the existing cluster. In Opscenter I do not see any streams... I presume that the new nodes get the data from the rest of the cluster via streams. The existing cluster has TB magnitude, and space used in the new nodes is ~90 GB. I must admit that I have restarted the new nodes several times after adding them . Does this affect boostrap? AFAIK the new nodes should start loading a part of all the data in the existing cluster. If it stays like this for a while, it sounds like your bootstraps have hung. Note that in general you should add nodes one at a time, especially if you are in a version without the fix for CASSANDRA-2434, in theory adding multiple nodes "at once" might contribute to their bootstraps hanging. Stop cassandra on the joining nodes, wipe/move aside their data directories, and try again one at a time. =Rob -- Jens RantilBackend engineerTink AB Email: jens.rantil@tink.sePhone: +46 708 84 18 32Web: www.tink.se Facebook Linkedin Twitter
Adding new node to cluster
Hello all, I have an existing cluster. When adding a new node, I saw that Opscenter saw the node in an unknown cluster. In the yaml, the cluster name is the same. So i have stopped the node and added it's ip address in the list of seeds. Now Opscenter sees my node. But nodetool status now sees it as UN, instead of UJ when it first started. One other mension is that even if I stop the node, remove it's ip from the list of seeds, Opscenter sees the node in the known clustre but nodetool sees it as UN. I am not sure what the implications of adding a node's ip in it's seed list are and I think that for the existing nodes I have might done the same. Eg. started with it's ip in the seed list but after removing it and having to restart the nodes for whatever reason, I did not see any changes. Is my cluster ok, or what do I need to do to bring the cluster to a good state? Thank you.
Re: Adding new node to cluster
Hello, I know that UN is good, but what troubles me is the addition of the own node's ip in it's yaml seeds section. On Tuesday, February 17, 2015 3:40 PM, Reynald Bourtembourg wrote: Hi Bogdan In nodetool status: - UJ: means your node is Up and Joining - UN: means your node is Up and in Normal state UN in nodetool is good ;-) On 17/02/2015 13:56, Batranut Bogdan wrote: Hello all, I have an existing cluster. When adding a new node, I saw that Opscenter saw the node in an unknown cluster. In the yaml, the cluster name is the same. So i have stopped the node and added it's ip address in the list of seeds. Now Opscenter sees my node. But nodetool status now sees it as UN, instead of UJ when it first started. One other mension is that even if I stop the node, remove it's ip from the list of seeds, Opscenter sees the node in the known clustre but nodetool sees it as UN. I am not sure what the implications of adding a node's ip in it's seed list are and I think that for the existing nodes I have might done the same. Eg. started with it's ip in the seed list but after removing it and having to restart the nodes for whatever reason, I did not see any changes. Is my cluster ok, or what do I need to do to bring the cluster to a good state? Thank you.
Re: Adding new node to cluster
Hello,
I use SimpleSnitch. All the nodes are in the sane datacenter. Not sure if all
are in the same rack.
On Tuesday, February 17, 2015 8:53 PM, "sean_r_dur...@homedepot.com"
wrote:
#yiv3880239437 #yiv3880239437 -- _filtered #yiv3880239437
{font-family:Helvetica;panose-1:2 11 6 4 2 2 2 2 2 4;} _filtered #yiv3880239437
{font-family:Wingdings;panose-1:5 0 0 0 0 0 0 0 0 0;} _filtered #yiv3880239437
{font-family:Wingdings;panose-1:5 0 0 0 0 0 0 0 0 0;} _filtered #yiv3880239437
{font-family:Calibri;panose-1:2 15 5 2 2 2 4 3 2 4;} _filtered #yiv3880239437
{font-family:Tahoma;panose-1:2 11 6 4 3 5 4 4 2 4;}#yiv3880239437
#yiv3880239437 p.yiv3880239437MsoNormal, #yiv3880239437
li.yiv3880239437MsoNormal, #yiv3880239437 div.yiv3880239437MsoNormal
{margin:0in;margin-bottom:.0001pt;font-size:12.0pt;}#yiv3880239437 a:link,
#yiv3880239437 span.yiv3880239437MsoHyperlink
{color:blue;text-decoration:underline;}#yiv3880239437 a:visited, #yiv3880239437
span.yiv3880239437MsoHyperlinkFollowed
{color:purple;text-decoration:underline;}#yiv3880239437
span.yiv3880239437EmailStyle17 {color:#1F497D;}#yiv3880239437
.yiv3880239437MsoChpDefault {font-size:10.0pt;} _filtered #yiv3880239437
{margin:1.0in 1.0in 1.0in 1.0in;}#yiv3880239437 div.yiv3880239437WordSection1
{}#yiv3880239437 _filtered #yiv3880239437 {} _filtered #yiv3880239437
{font-family:Symbol;} _filtered #yiv3880239437 {} _filtered #yiv3880239437
{font-family:Wingdings;} _filtered #yiv3880239437 {font-family:Wingdings;}
_filtered #yiv3880239437 {font-family:Wingdings;} _filtered #yiv3880239437
{font-family:Wingdings;} _filtered #yiv3880239437 {font-family:Wingdings;}
_filtered #yiv3880239437 {font-family:Wingdings;} _filtered #yiv3880239437
{font-family:Wingdings;}#yiv3880239437 ol {margin-bottom:0in;}#yiv3880239437 ul
{margin-bottom:0in;}#yiv3880239437 What snitch are you using? You may need to
do some work on your topology file (or rackdc) to make sure you have the
topology you want. Also, it is possible you may need to restart OpsCenter
agents and/or your browser to see the nodes represented properly in OpsCenter.
Sean Durity – Cassandra Admin, Home Depot From: Batranut Bogdan
[mailto:batra...@yahoo.com]
Sent: Tuesday, February 17, 2015 10:20 AM
To: user@cassandra.apache.org; reynald.bourtembo...@esrf.fr
Subject: Re: Adding new node to cluster Hello, I know that UN is good,
but what troubles me is the addition of the own node's ip in it's yaml seeds
section. On Tuesday, February 17, 2015 3:40 PM, Reynald Bourtembourg
wrote: Hi Bogdan
In nodetool status:
- UJ: means your node is Up and Joining
- UN: means your node is Up and in Normal state
UN in nodetool is good ;-) On 17/02/2015 13:56, Batranut Bogdan wrote:
Hello all, I have an existing cluster. When adding a new node, I saw that
Opscenter saw the node in an unknown cluster. In the yaml, the cluster name is
the same. So i have stopped the node and added it's ip address in the list of
seeds. Now Opscenter sees my node. But nodetool status now sees it as UN,
instead of UJ when it first started. One other mension is that even if I stop
the node, remove it's ip from the list of seeds, Opscenter sees the node in the
known clustre but nodetool sees it as UN. I am not sure what the implications
of adding a node's ip in it's seed list are and I think that for the existing
nodes I have might done the same. Eg. started with it's ip in the seed list but
after removing it and having to restart the nodes for whatever reason, I did
not see any changes. Is my cluster ok, or what do I need to do to bring the
cluster to a good state? Thank you.
The information in this Internet Email is confidential and may be legally
privileged. It is intended solely for the addressee. Access to this Email by
anyone else is unauthorized. If you are not the intended recipient, any
disclosure, copying, distribution or any action taken or omitted to be taken in
reliance on it, is prohibited and may be unlawful. When addressed to our
clients any opinions or advice contained in this Email are subject to the terms
and conditions expressed in any applicable governing The Home Depot terms of
business or client engagement letter. The Home Depot disclaims all
responsibility and liability for the accuracy and content of this attachment
and for any damages or losses arising from any inaccuracies, errors, viruses,
e.g., worms, trojan horses, etc., or other items of a destructive nature, which
may be contained in this attachment and shall not be liable for direct,
indirect, consequential or special damages in connection with this e-mail
message or its attachment.
Re: Adding new node to cluster
Hello, I have decommissioned a node, deleted data,commitlog and saved caches, changed yaml file to not include self ip and started it. For some reason I do not fully understand, Opscenter says that the node is in an unknown datacenter. Nodetool says UJ but shows "?" in the Owns column. I have started the node yesterday. I still see streams towards this node so I can I assume that once it finishes joining, Opscenter will see it properly? I remember that I might have setup the entire initial cluster with own ips in the yaml file.To bring the cluster to a valid state, I assume that I have to decommission the nodes one by one, delete the data, and restart with correct yaml settings.Is this correct? Also a change of the cluster name would be nice.How can this be done with minimal impact? On Wednesday, February 18, 2015 12:56 AM, Eric Stevens wrote: > Seed nodes apparently don’t bootstrap That's right, if a node has itself in its own seeds list, it assumes it's a foundational member of the cluster, and it will join immediately with no bootstrap. If you've done this by accident, you should do nodetool decommission on that node, and when it's fully left the cluster, wipe its data directory, edit the yaml and remove it from the seeds list. On Tue, Feb 17, 2015 at 3:25 PM, wrote: SimpleSnitch is not rack aware. You would want to choose seed nodes and then not change them. Seed nodes apparently don’t bootstrap. All nodes need the same seeds in the yaml file. Here is more info: http://www.datastax.com/documentation/cassandra/2.0/cassandra/initialize/initializeSingleDS.html Sean Durity – Cassandra Admin, Big Data TeamTo engage the team,create a request From: Batranut Bogdan [mailto:batra...@yahoo.com] Sent: Tuesday, February 17, 2015 3:28 PM To: user@cassandra.apache.org; reynald.bourtembo...@esrf.fr Subject: Re: Adding new node to cluster Hello, I use SimpleSnitch. All the nodes are in the sane datacenter. Not sure if all are in the same rack. On Tuesday, February 17, 2015 8:53 PM, "sean_r_dur...@homedepot.com" wrote: What snitch are you using? You may need to do some work on your topology file (or rackdc) to make sure you have the topology you want. Also, it is possible you may need to restart OpsCenter agents and/or your browser to see the nodes represented properly in OpsCenter. Sean Durity – Cassandra Admin, Home Depot From: Batranut Bogdan [mailto:batra...@yahoo.com] Sent: Tuesday, February 17, 2015 10:20 AM To: user@cassandra.apache.org;reynald.bourtembo...@esrf.fr Subject: Re: Adding new node to cluster Hello, I know that UN is good, but what troubles me is the addition of the own node's ip in it's yaml seeds section. On Tuesday, February 17, 2015 3:40 PM, Reynald Bourtembourg wrote: Hi Bogdan In nodetool status: - UJ: means your node is Up and Joining - UN: means your node is Up and in Normal state UN in nodetool is good ;-) On 17/02/2015 13:56, Batranut Bogdan wrote: Hello all, I have an existing cluster. When adding a new node, I saw that Opscenter saw the node in an unknown cluster. In the yaml, the cluster name is the same. So i have stopped the node and added it's ip address in the list of seeds. Now Opscenter sees my node. But nodetool status now sees it as UN, instead of UJ when it first started. One other mension is that even if I stop the node, remove it's ip from the list of seeds, Opscenter sees the node in the known clustre but nodetool sees it as UN. I am not sure what the implications of adding a node's ip in it's seed list are and I think that for the existing nodes I have might done the same. Eg. started with it's ip in the seed list but after removing it and having to restart the nodes for whatever reason, I did not see any changes. Is my cluster ok, or what do I need to do to bring the cluster to a good state? Thank you. The information in this Internet Email is confidential and may be legally privileged. It is intended solely for the addressee. Access to this Email by anyone else is unauthorized. If you are not the intended recipient, any disclosure, copying, distribution or any action taken or omitted to be taken in reliance on it, is prohibited and may be unlawful. When addressed to our clients any opinions or advice contained in this Email are subject to the terms and conditions expressed in any applicable governing The Home Depot terms of business or client engagement letter. The Home Depot disclaims all responsibility and liability for the accuracy and content of this attachment and for any damages or losses arising from any inaccuracies, errors, viruses, e.g., worms, trojan horses, etc., or other items of a destructive nature, which may be contained in this attachment and shall not be liable for direct, indirect, consequential or special damages in connection with this e-mail message or its attachment. The informatio
FileNotFoundException
Hello all, One of my C* throws a big amount of exceptions like this: ERROR [ReadStage:792] 2015-02-24 10:43:54,183 CassandraDaemon.java (line 199) Exception in thread Thread[ReadStage:792,5,main]java.lang.RuntimeException: java.lang.RuntimeException: java.io.FileNotFoundException: /data/ranks/positions/ranks-positions-jb-174-Data.db (No such file or directory) at org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:2008) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:744)Caused by: java.lang.RuntimeException: java.io.FileNotFoundException: /data/ranks/positions/ranks-positions-jb-174-Data.db (No such file or directory) at org.apache.cassandra.io.compress.CompressedRandomAccessReader.open(CompressedRandomAccessReader.java:47) at org.apache.cassandra.io.util.CompressedPoolingSegmentedFile.createReader(CompressedPoolingSegmentedFile.java:48) at org.apache.cassandra.io.util.PoolingSegmentedFile.getSegment(PoolingSegmentedFile.java:39) at org.apache.cassandra.io.sstable.SSTableReader.getFileDataInput(SSTableReader.java:1239) at org.apache.cassandra.db.columniterator.IndexedSliceReader$IndexedBlockFetcher.getNextBlock(IndexedSliceReader.java:417) at org.apache.cassandra.db.columniterator.IndexedSliceReader$IndexedBlockFetcher.fetchMoreData(IndexedSliceReader.java:387) at org.apache.cassandra.db.columniterator.IndexedSliceReader.computeNext(IndexedSliceReader.java:145) at org.apache.cassandra.db.columniterator.IndexedSliceReader.computeNext(IndexedSliceReader.java:45) at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143) at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138) at org.apache.cassandra.db.columniterator.SSTableSliceIterator.hasNext(SSTableSliceIterator.java:82) at org.apache.cassandra.db.filter.QueryFilter$2.getNext(QueryFilter.java:157) at org.apache.cassandra.db.filter.QueryFilter$2.hasNext(QueryFilter.java:140) at org.apache.cassandra.utils.MergeIterator$Candidate.advance(MergeIterator.java:144) at org.apache.cassandra.utils.MergeIterator$ManyToOne.(MergeIterator.java:87) at org.apache.cassandra.utils.MergeIterator.get(MergeIterator.java:46) at org.apache.cassandra.db.filter.QueryFilter.collateColumns(QueryFilter.java:120) at org.apache.cassandra.db.filter.QueryFilter.collateOnDiskAtom(QueryFilter.java:80) at org.apache.cassandra.db.filter.QueryFilter.collateOnDiskAtom(QueryFilter.java:72) at org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:297) at org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:53) at org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1550) at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1379) at org.apache.cassandra.db.Keyspace.getRow(Keyspace.java:333) at org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:65) at org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:1439) at org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:2004) ... 3 moreCaused by: java.io.FileNotFoundException: /data/ranks/positions/ranks-positions-jb-174-Data.db (No such file or directory) at java.io.RandomAccessFile.open(Native Method) at java.io.RandomAccessFile.(RandomAccessFile.java:241) at org.apache.cassandra.io.util.RandomAccessReader.(RandomAccessReader.java:58) at org.apache.cassandra.io.compress.CompressedRandomAccessReader.(CompressedRandomAccessReader.java:76) at org.apache.cassandra.io.compress.CompressedRandomAccessReader.open(CompressedRandomAccessReader.java:43) ... 29 more I must say, that this node was started with cleared data / commit log and it was joining an existing cluster. I see a lot of these exceptions. Not sure what to do.
Fw: FileNotFoundException
Also I must add that grepping the logs for a particular file I see this: INFO [CompactionExecutor:19] 2015-02-24 10:44:35,618 CompactionTask.java (line 120) Compacting [SSTableReader(path='/data/ranks/positions/ranks-positions-jb-339-Data.db'), SSTableReader(path='/data/ranks/positions/ranks-positions-jb-354-Data.db'), SSTableReader(path='/data/ranks/positions/ranks-positions-jb-408-Data.db'), SSTableReader(path='/data/ranks/positions/ranks-positions-jb-286-Data.db'), SSTableReader(path='/data/ranks/positions/ranks-positions-jb-20-Data.db'), SSTableReader(path='/data/ranks/positions/ranks-positions-jb-127-Data.db'), SSTableReader(path='/data/ranks/positions/ranks-positions-jb-357-Data.db'), SSTableReader(path='/data/ranks/positions/ranks-positions-jb-257-Data.db'), SSTableReader(path='/data/ranks/positions/ranks-positions-jb-316-Data.db'), SSTableReader(path='/data/ranks/positions/ranks-positions-jb-41-Data.db'), SSTableReader(path='/data/ranks/positions/ranks-positions-jb-285-Data.db'), SSTableReader(path='/data/ranks/positions/ranks-positions-jb-338-Data.db'), SSTableReader(path='/data/ranks/positions/ranks-positions-jb-180-Data.db'), SSTableReader(path='/data/ranks/positions/ranks-positions-jb-398-Data.db'), SSTableReader(path='/data/ranks/positions/ranks-positions-jb-249-Data.db'), SSTableReader(path='/data/ranks/positions/ranks-positions-jb-284-Data.db'), SSTableReader(path='/data/ranks/positions/ranks-positions-jb-294-Data.db'), SSTableReader(path='/data/ranks/positions/ranks-positions-jb-248-Data.db'), SSTableReader(path='/data/ranks/positions/ranks-positions-jb-377-Data.db'), SSTableReader(path='/data/ranks/positions/ranks-positions-jb-395-Data.d ... also several entries like this in the log after grep.java.lang.RuntimeException: java.io.FileNotFoundException: /data/ranks/positions/ranks-positions-jb-41-Data.db (No such file or directory)Caused by: java.io.FileNotFoundException: /data/ranks/positions/ranks-positions-jb-41-Data.db (No such file or directory) I was grepping for jb-41-Data.db ... seems that this file does not exist for some reason. I must say that when I first added the node I included it's IP in the seeds list. Then I have decommissioned it, removed it's IP from the seed list, deleted all data / commit log / saved caches and started it. Since then I have not manualy deleted any files . Any ideeas? On Tuesday, February 24, 2015 11:46 AM, Batranut Bogdan wrote: Hello all, One of my C* throws a big amount of exceptions like this: ERROR [ReadStage:792] 2015-02-24 10:43:54,183 CassandraDaemon.java (line 199) Exception in thread Thread[ReadStage:792,5,main]java.lang.RuntimeException: java.lang.RuntimeException: java.io.FileNotFoundException: /data/ranks/positions/ranks-positions-jb-174-Data.db (No such file or directory) at org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:2008) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:744)Caused by: java.lang.RuntimeException: java.io.FileNotFoundException: /data/ranks/positions/ranks-positions-jb-174-Data.db (No such file or directory) at org.apache.cassandra.io.compress.CompressedRandomAccessReader.open(CompressedRandomAccessReader.java:47) at org.apache.cassandra.io.util.CompressedPoolingSegmentedFile.createReader(CompressedPoolingSegmentedFile.java:48) at org.apache.cassandra.io.util.PoolingSegmentedFile.getSegment(PoolingSegmentedFile.java:39) at org.apache.cassandra.io.sstable.SSTableReader.getFileDataInput(SSTableReader.java:1239) at org.apache.cassandra.db.columniterator.IndexedSliceReader$IndexedBlockFetcher.getNextBlock(IndexedSliceReader.java:417) at org.apache.cassandra.db.columniterator.IndexedSliceReader$IndexedBlockFetcher.fetchMoreData(IndexedSliceReader.java:387) at org.apache.cassandra.db.columniterator.IndexedSliceReader.computeNext(IndexedSliceReader.java:145) at org.apache.cassandra.db.columniterator.IndexedSliceReader.computeNext(IndexedSliceReader.java:45) at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143) at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138) at org.apache.cassandra.db.columniterator.SSTableSliceIterator.hasNext(SSTableSliceIterator.java:82) at org.apache.cassandra.db.filter.QueryFilter$2.getNext(QueryFilter.java:157) at org.apache.cassandra.db.filter.QueryFilter$2.hasNext(QueryFilter.java:140) at org.apache.cassandra.utils.MergeIterator$C
Node stuck in joining the ring
Hello all, I have a new node that I want to add to the ring. The problem is that nodetool says UJ I have left it for several days and the status has not changed. In Opscenter it is "seen" as in an unknown cluster. >From the time that I started it, it was streaming data and the data size is >5,9 TB. This is very strange since all other nodes in the cluster have about >3,3 TB of data. Also tonight I saw that it stopped getting streams and the >status in nodetool was still UJ. So I thought to decommission the node delete >the data and start again. Nodetool throws unsupported operation: local node is >not a member of the token ring yet. So I have just restarted the node. Now >streaming data begins again. At this rate, I'll run out of disk space on that >node.One ideea that comes to mind is to stop, clear the data and restart. But >I am not sure about the implications for that. Also, I have tried nodetool >join. I got: This node has already joined the ring. So nodetool status says UJ but nodetool join says otherwise, or am I not understanding someting here. Any ideeas?
Re: Node stuck in joining the ring
C* version 2.0.12 How do I resolve item 2) ? Just want to mention that when the node is stopped, nodetool status does not show it Down, it is missing from the list... Thanks for the support. On Thursday, February 26, 2015 2:52 AM, Robert Coli wrote: On Wed, Feb 25, 2015 at 3:38 PM, Batranut Bogdan wrote: I have a new node that I want to add to the ring. The problem is that nodetool says UJ I have left it for several days and the status has not changed. In Opscenter it is "seen" as in an unknown cluster. If I were you, I would do the following [1] : 1) stop the joining node2) make sure that the other nodes no longer see it joining3) wipe the joining node's data directory4) verify cluster name is correct in cassandra.yaml, and matches the other nodes5) re-join the node What version of Cassandra? =Rob[1] Which, jeesh, I should put into a "dealing with failed bootstrap" blog post one of these days...
Re: Node stuck in joining the ring
All the nodes have the same version. 2.0.12
Re: Node stuck in joining the ring
Hello Jan, Yes I do have ntp and it is in synch. On Thursday, February 26, 2015 11:49 AM, Jan Kesten wrote: Hi Batranut, apart from the other suggestions - do you have ntp running on all your cluster nodes and are times in sync? Jan
Re: Node stuck in joining the ring
No errors in the system.log file [root@cassa09 cassandra]# grep "ERROR" system.log[root@cassa09 cassandra]# Nothing. On Thursday, February 26, 2015 1:55 PM, mck wrote: Any errors in your log file? We saw something similar when bootstrap crashed when rebuilding secondary indexes. See CASSANDRA-8798 ~mck
