but the data size in the saved_cache are relatively small: will that cause the load problem?
ls -lh /cassandra/saved_caches/ total 32M -rw-r--r-- 1 cass cass 2.9M 2011-08-12 19:53 cass-CommentSortsCache-KeyCache -rw-r--r-- 1 cass cass 2.9M 2011-08-17 04:29 cass-CommentSortsCache-RowCache -rw-r--r-- 1 cass cass 2.7M 2011-08-12 18:50 cass-CommentVote-KeyCache -rw-r--r-- 1 cass cass 140K 2011-08-12 19:53 cass-device_images-KeyCache -rw-r--r-- 1 cass cass 33K 2011-08-12 18:51 cass-Hide-KeyCache -rw-r--r-- 1 cass cass 4.6M 2011-08-12 19:53 cass-images-KeyCache -rw-r--r-- 1 cass cass 2.6M 2011-08-12 19:53 cass-LinksByUrl-KeyCache -rw-r--r-- 1 cass cass 2.5M 2011-08-12 18:50 cass-LinkVote-KeyCache -rw-r--r-- 1 cass cass 7.5M 2011-08-12 18:50 cass-cache-KeyCache -rw-r--r-- 1 cass cass 3.7M 2011-08-12 21:51 cass-cache-RowCache -rw-r--r-- 1 cass cass 1.8M 2011-08-12 18:51 cass-Save-KeyCache -rw-r--r-- 1 cass cass 111K 2011-08-12 19:50 cass-SavesByAccount-KeyCache -rw-r--r-- 1 cass cass 864 2011-08-12 19:49 cass-VotesByDay-KeyCache -rw-r--r-- 1 cass cass 249K 2011-08-12 19:49 cass-VotesByLink-KeyCache -rw-r--r-- 1 cass cass 28 2011-08-14 12:50 system-HintsColumnFamily-KeyCache -rw-r--r-- 1 cass cass 5 2011-08-14 12:50 system-LocationInfo-KeyCache -rw-r--r-- 1 cass cass 54 2011-08-13 13:30 system-Migrations-KeyCache -rw-r--r-- 1 cass cass 76 2011-08-13 13:30 system-Schema-KeyCache On Wed, Aug 17, 2011 at 4:31 PM, aaron morton <aa...@thelastpickle.com> wrote: > If you have a node that cannot start up due to issues loading the saved cache > delete the files in the saved_cache directory before starting it. > > The settings to save the row and key cache are per CF. You can change them > with an update column family statement via the CLI when attached to any node. > You may then want to check the saved_caches directory and delete any files > that are left (not sure if they are automatically deleted). > > i would recommend: > - stop node 2 > - delete it's saved_cache > - make the schema change via another node > - startup node 2 > > Cheers > > ----------------- > Aaron Morton > Freelance Cassandra Developer > @aaronmorton > http://www.thelastpickle.com > > On 17/08/2011, at 2:59 PM, Yan Chunlu wrote: > >> does this need to be cluster wide? or I could just modify the caches >> on one node? since I could not connect to the node with >> cassandra-cli, it says "connection refused" >> >> >> [default@unknown] connect node2/9160; >> Exception connecting to node2/9160. Reason: Connection refused. >> >> >> so if I change the cache size via other nodes, how could node2 be >> notified the changing? kill cassandra and start it again could make >> it update the schema? >> >> >> >> On Wed, Aug 17, 2011 at 5:59 AM, Teijo Holzer <thol...@wetafx.co.nz> wrote: >>> Hi, >>> >>> yes, we saw exactly the same messages. We got rid of these by doing the >>> following: >>> >>> * Set all row & key caches in your CFs to 0 via cassandra-cli >>> * Kill Cassandra >>> * Remove all files in the saved_caches directory >>> * Start Cassandra >>> * Slowly bring back row & key caches (if desired, we left them off) >>> >>> Cheers, >>> >>> T. >>> >>> On 16/08/11 23:35, Yan Chunlu wrote: >>>> >>>> I saw alot slicequeryfilter things if changed the log level to DEBUG. >>>> just >>>> thought even bring up a new node will be faster than start the old >>>> one..... it >>>> is wired >>>> >>>> DEBUG [main] 2011-08-16 06:32:49,213 SliceQueryFilter.java (line 123) >>>> collecting 0 of 2147483647: 76616c7565:false:225@1313068845474382 >>>> DEBUG [main] 2011-08-16 06:32:49,245 SliceQueryFilter.java (line 123) >>>> collecting 0 of 2147483647: 76616c7565:false:453@1310999270198313 >>>> DEBUG [main] 2011-08-16 06:32:49,251 SliceQueryFilter.java (line 123) >>>> collecting 0 of 2147483647: 76616c7565:false:26@1313199902088827 >>>> DEBUG [main] 2011-08-16 06:32:49,576 SliceQueryFilter.java (line 123) >>>> collecting 0 of 2147483647: 76616c7565:false:157@1313097239332314 >>>> DEBUG [main] 2011-08-16 06:32:50,674 SliceQueryFilter.java (line 123) >>>> collecting 0 of 2147483647: 76616c7565:false:41729@1313190821826229 >>>> DEBUG [main] 2011-08-16 06:32:50,811 SliceQueryFilter.java (line 123) >>>> collecting 0 of 2147483647: 76616c7565:false:6@1313174157301203 >>>> DEBUG [main] 2011-08-16 06:32:50,867 SliceQueryFilter.java (line 123) >>>> collecting 0 of 2147483647: 76616c7565:false:98@1312011362250907 >>>> DEBUG [main] 2011-08-16 06:32:50,881 SliceQueryFilter.java (line 123) >>>> collecting 0 of 2147483647: 76616c7565:false:42@1313201711997005 >>>> DEBUG [main] 2011-08-16 06:32:50,910 SliceQueryFilter.java (line 123) >>>> collecting 0 of 2147483647: 76616c7565:false:96@1312939986190155 >>>> DEBUG [main] 2011-08-16 06:32:50,954 SliceQueryFilter.java (line 123) >>>> collecting 0 of 2147483647: 76616c7565:false:621@1313192538616112 >>>> >>>> >>>> >>>> On Tue, Aug 16, 2011 at 7:32 PM, Yan Chunlu <springri...@gmail.com >>>> <mailto:springri...@gmail.com>> wrote: >>>> >>>> but it seems the row cache is cluster wide, how will the change of row >>>> cache affect the read speed? >>>> >>>> >>>> On Mon, Aug 15, 2011 at 7:33 AM, Jonathan Ellis <jbel...@gmail.com >>>> <mailto:jbel...@gmail.com>> wrote: >>>> >>>> Or leave row cache enabled but disable cache saving (and remove the >>>> one already on disk). >>>> >>>> On Sun, Aug 14, 2011 at 5:05 PM, aaron morton >>>> <aa...@thelastpickle.com >>>> <mailto:aa...@thelastpickle.com>> wrote: >>>> > INFO [main] 2011-08-14 09:24:52,198 ColumnFamilyStore.java >>>> (line 547) >>>> > completed loading (1744370 ms; 200000 keys) row cache for >>>> COMMENT >>>> > >>>> > It's taking 29 minutes to load 200,000 rows in the row cache. >>>> Thats a >>>> > pretty big row cache, I would suggest reducing or disabling it. >>>> > Background >>>> >>>> http://www.datastax.com/dev/blog/maximizing-cache-benefit-with-cassandra >>>> > >>>> > and server can not afford the load then crashed. after come >>>> back, >>>> node 3 can >>>> > not return for more than 96 hours >>>> > >>>> > Crashed how ? >>>> > You may be seeing >>>> https://issues.apache.org/jira/browse/CASSANDRA-2280 >>>> > Watch nodetool compactionstats to see when the Merkle tree build >>>> finishes >>>> > and nodetool netstats to see which CF's are streaming. >>>> > Cheers >>>> > ----------------- >>>> > Aaron Morton >>>> > Freelance Cassandra Developer >>>> > @aaronmorton >>>> > http://www.thelastpickle.com >>>> > On 15 Aug 2011, at 04:23, Yan Chunlu wrote: >>>> > >>>> > >>>> > I got 3 nodes and RF=3, when I repairing ndoe3, it seems alot >>>> data >>>> > generated. and server can not afford the load then crashed. >>>> > after come back, node 3 can not return for more than 96 hours >>>> > >>>> > for 34GB data, the node 2 could restart and back online within 1 >>>> hour. >>>> > >>>> > I am not sure what's wrong with node3 and should I restart node >>>> 3 again? >>>> > thanks! >>>> > >>>> > Address Status State Load Owns Token >>>> > >>>> > 113427455640312821154458202477256070484 >>>> > node1 Up Normal 34.11 GB 33.33% 0 >>>> > node2 Up Normal 31.44 GB 33.33% >>>> > 56713727820156410577229101238628035242 >>>> > node3 Down Normal 177.55 GB 33.33% >>>> > 113427455640312821154458202477256070484 >>>> > >>>> > >>>> > the log shows it is still going on, not sure why it is so slow: >>>> > >>>> > >>>> > INFO [main] 2011-08-14 08:55:47,734 SSTableReader.java (line >>>> 154) >>>> Opening >>>> > /cassandra/data/COMMENT >>>> > INFO [main] 2011-08-14 08:55:47,828 ColumnFamilyStore.java >>>> (line 275) >>>> > reading saved cache /cassandra/saved_caches/COMMENT-RowCache >>>> > INFO [main] 2011-08-14 09:24:52,198 ColumnFamilyStore.java >>>> (line 547) >>>> > completed loading (1744370 ms; 200000 keys) row cache for >>>> COMMENT >>>> > INFO [main] 2011-08-14 09:24:52,299 ColumnFamilyStore.java >>>> (line 275) >>>> > reading saved cache /cassandra/saved_caches/COMMENT-RowCache >>>> > INFO [CompactionExecutor:1] 2011-08-14 10:24:55,480 >>>> CacheWriter.java (line >>>> > 96) Saved COMMENT-RowCache (200000 items) in 2535 ms >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> >>>> >>>> >>>> -- >>>> Jonathan Ellis >>>> Project Chair, Apache Cassandra >>>> co-founder of DataStax, the source for professional Cassandra >>>> support >>>> http://www.datastax.com >>>> >>>> >>>> >>> >>> > >
