Re: Hbase vs Cassandra
I think the links provided are general comparison. I will try to explain the difference as I implemented time series with both Cassandra and HBase. We migrated from Cassandra to HBase. Reasons: Scan is fast with HBase. It fits better with time series data model. Please look at opentsdb. Cassandra models it with large rows. Server side filtering. You can use to filter some of your time series data on the server side. Hbase has a better integration with hadoop in general. We had to write our own bulk loader using mapreduce for cassandra. hbase has already had a tool for that. There is a nice integration with flume and kite. High availability didnet matter for us. 10 secs down is fine for our use cases.HBase started to support eventually consistent reads. Cem Sent from my iPhone > On May 30, 2015, at 4:24 PM, Brady Gentile wrote: > > Hey Ajay, > > Here are some use cases from PlanetCassandra.org of companies who chose > Cassandra over HBase after evaluation, or migrated to Cassandra from HBase. > > The eComNext interview cited on the page touches on time-series data; if > you're interested in talking with Miguel Coira, I'd be happy to make that > connection. > > http://planetcassandra.org/hbase-to-cassandra-migration/ > > Brady Gentile > Program Manager > 480.735.1133 > > >> On May 29, 2015, at 12:09 PM, Ajay wrote: >> >> Hi, >> >> I need some info on Hbase vs Cassandra as a data store (in general plus >> specific to time series data). >> >> The comparison in the following helps: >> 1: features >> 2: deployment and monitoring >> 3: performance >> 4: anything else >> >> Thanks >> Ajay
Re: disk space issue
thanks for the answers! Cem On Wed, Oct 1, 2014 at 2:38 PM, Ken Hancock wrote: > *https://github.com/hancockks/cassandra-compact-cf > <https://github.com/hancockks/cassandra-compact-cf>* > > On Tue, Sep 30, 2014 at 5:49 PM, cem wrote: > >> Hi All, >> >> I have a 7 node cluster. One node ran out of disk space and others are >> around 80% disk utilization. >> The data has 10 days TTL but I think compaction wasn't fast enough to >> clean up the expired data. gc_grace value is set default. I have a >> replication factor of 3. Do you think that it may help if I delete all data >> for that node and run repair. Does node repair check the ttl value before >> retrieving data from other nodes? Do you have any other suggestions? >> >> Best Regards, >> Cem. >> > > > > -- > *Ken Hancock *| System Architect, Advanced Advertising > SeaChange International > 50 Nagog Park > Acton, Massachusetts 01720 > ken.hanc...@schange.com | www.schange.com | NASDAQ:SEAC > <http://www.schange.com/en-US/Company/InvestorRelations.aspx> > Office: +1 (978) 889-3329 | [image: Google Talk:] ken.hanc...@schange.com > | [image: Skype:]hancockks | [image: Yahoo IM:]hancockks[image: LinkedIn] > <http://www.linkedin.com/in/kenhancock> > > [image: SeaChange International] > <http://www.schange.com/>This e-mail and any attachments may contain > information which is SeaChange International confidential. The information > enclosed is intended only for the addressees herein and may not be copied > or forwarded without permission from SeaChange International. >
disk space issue
Hi All, I have a 7 node cluster. One node ran out of disk space and others are around 80% disk utilization. The data has 10 days TTL but I think compaction wasn't fast enough to clean up the expired data. gc_grace value is set default. I have a replication factor of 3. Do you think that it may help if I delete all data for that node and run repair. Does node repair check the ttl value before retrieving data from other nodes? Do you have any other suggestions? Best Regards, Cem.
Re: 2 nodes cassandra cluster raid10 or JBOD
Thanks! On Wed, Dec 11, 2013 at 9:37 PM, Aaron Morton wrote: > If you have two nodes, and RF 2, you will only be able to use eventual > consistency. If you want to have stronger consistency and some redundancy 3 > nodes is the minimum requirement. > > In the current setup, with only 2 nodes, I would use RAID 10 as it > requires less operator intervention and there is a chance of data loss with > RF 2. If a write (including the hints) is only committed to one node and > the disk on that node fails the write will be lost. > > Hope that helps. > > > - > Aaron Morton > New Zealand > @aaronmorton > > Co-Founder & Principal Consultant > Apache Cassandra Consulting > http://www.thelastpickle.com > > On 11/12/2013, at 9:33 pm, Veysel Taşçıoğlu > wrote: > > Hi, > > What about using JBOD and replication factor 2? > > Regards. > On 11 Dec 2013 02:03, "cem" wrote: > >> Hi all, >> >> I need to setup 2 nodes Cassandra cluster. I know that Datastax >> recommends using JBOD as a disk configuration and have replication for the >> redundancy. I was planning to use RAID 10 but using JBOD can save 50% disk >> space and increase the performance . But I am not sure I should use JBOD >> with 2 nodes cluster since there is a higher chance to lose 50% of our >> cluster compare to a larger cluster. I may prefer to have stronger nodes if >> I have limited number of nodes. >> >> >> What do you think about that? Is there anyone who has 2 nodes cluster? >> >> >> Best Regards, >> >> Cem >> > >
2 nodes cassandra cluster raid10 or JBOD
Hi all, I need to setup 2 nodes Cassandra cluster. I know that Datastax recommends using JBOD as a disk configuration and have replication for the redundancy. I was planning to use RAID 10 but using JBOD can save 50% disk space and increase the performance . But I am not sure I should use JBOD with 2 nodes cluster since there is a higher chance to lose 50% of our cluster compare to a larger cluster. I may prefer to have stronger nodes if I have limited number of nodes. What do you think about that? Is there anyone who has 2 nodes cluster? Best Regards, Cem
Re: Deleting Row Key
It will be deleted after a compaction. Sent from my iPhone On 05 Oct 2013, at 07:29, Sebastian Schmidt wrote: > Hi, > > per default, the key of a row is not deleted, if all columns were > deleted. I tried to figure out why, but I didn't find an answer, except > that it is 'intended'. Why is that intended? And is there a possibility > to manually delete the row key? > > Regards > Sebastian
Re: Cassandra Heap Size for data more than 1 TB
I think 512 is fine. Could you tell more about your traffic characteristics? Cem On Wed, Oct 2, 2013 at 4:32 PM, srmore wrote: > I changed my index_interval from 128 to index_interval: 128 to 512, does > it make sense to increase more than this ? > > > On Wed, Oct 2, 2013 at 9:30 AM, cem wrote: > >> Have a look to index_interval. >> >> Cem. >> >> >> On Wed, Oct 2, 2013 at 2:25 PM, srmore wrote: >> >>> The version of Cassandra I am using is 1.0.11, we are migrating to 1.2.X >>> though. We had tuned bloom filters (0.1) and AFAIK making it lower than >>> this won't matter. >>> >>> Thanks ! >>> >>> >>> On Tue, Oct 1, 2013 at 11:54 PM, Mohit Anchlia >>> wrote: >>> >>>> Which Cassandra version are you on? Essentially heap size is function >>>> of number of keys/metadata. In Cassandra 1.2 lot of the metadata like bloom >>>> filters were moved off heap. >>>> >>>> >>>> On Tue, Oct 1, 2013 at 9:34 PM, srmore wrote: >>>> >>>>> Does anyone know what would roughly be the heap size for cassandra >>>>> with 1TB of data ? We started with about 200 G and now on one of the nodes >>>>> we are already on 1 TB. We were using 8G of heap and that served us well >>>>> up >>>>> until we reached 700 G where we started seeing failures and nodes >>>>> flipping. >>>>> >>>>> With 1 TB of data the node refuses to come back due to lack of memory. >>>>> needless to say repairs and compactions takes a lot of time. We upped the >>>>> heap from 8 G to 12 G and suddenly everything started moving rapidly i.e. >>>>> the repair tasks and the compaction tasks. But soon (in about 9-10 hrs) we >>>>> started seeing the same symptoms as we were seeing with 8 G. >>>>> >>>>> So my question is how do I determine what is the optimal size of heap >>>>> for data around 1 TB ? >>>>> >>>>> Following are some of my JVM settings >>>>> >>>>> -Xms8G >>>>> -Xmx8G >>>>> -Xmn800m >>>>> -XX:NewSize=1200M >>>>> XX:MaxTenuringThreshold=2 >>>>> -XX:SurvivorRatio=4 >>>>> >>>>> Thanks ! >>>>> >>>> >>>> >>> >> >
Re: Cassandra Heap Size for data more than 1 TB
Have a look to index_interval. Cem. On Wed, Oct 2, 2013 at 2:25 PM, srmore wrote: > The version of Cassandra I am using is 1.0.11, we are migrating to 1.2.X > though. We had tuned bloom filters (0.1) and AFAIK making it lower than > this won't matter. > > Thanks ! > > > On Tue, Oct 1, 2013 at 11:54 PM, Mohit Anchlia wrote: > >> Which Cassandra version are you on? Essentially heap size is function of >> number of keys/metadata. In Cassandra 1.2 lot of the metadata like bloom >> filters were moved off heap. >> >> >> On Tue, Oct 1, 2013 at 9:34 PM, srmore wrote: >> >>> Does anyone know what would roughly be the heap size for cassandra with >>> 1TB of data ? We started with about 200 G and now on one of the nodes we >>> are already on 1 TB. We were using 8G of heap and that served us well up >>> until we reached 700 G where we started seeing failures and nodes flipping. >>> >>> With 1 TB of data the node refuses to come back due to lack of memory. >>> needless to say repairs and compactions takes a lot of time. We upped the >>> heap from 8 G to 12 G and suddenly everything started moving rapidly i.e. >>> the repair tasks and the compaction tasks. But soon (in about 9-10 hrs) we >>> started seeing the same symptoms as we were seeing with 8 G. >>> >>> So my question is how do I determine what is the optimal size of heap >>> for data around 1 TB ? >>> >>> Following are some of my JVM settings >>> >>> -Xms8G >>> -Xmx8G >>> -Xmn800m >>> -XX:NewSize=1200M >>> XX:MaxTenuringThreshold=2 >>> -XX:SurvivorRatio=4 >>> >>> Thanks ! >>> >> >> >
Re: How to contribute to C*?
Thanks Rob! On Thu, Sep 5, 2013 at 12:17 AM, Robert Coli wrote: > On Wed, Sep 4, 2013 at 2:48 PM, cem wrote: > >> I would prefer to have it on server side since it introduces too much >> complexity on client side and CF overheads. >> >> I would like to know how to contribute to C* code base? >> Am I going to open a ticket and assign to me? >> > > Open a JIRA, attach your patch and a description of why you want to > contribute it. If that doesn't get the response you're looking for, ask on > #cassandra-dev@. > > =Rob >
How to contribute to C*?
Hi All, I have a implementation for range partitioning on the client side. Each partition is a CF. It allows you to create and drop partitions. It is very useful if you have time series or event data that you want to store temporarily. You can use TTL for the same issue but it is not fast enough to clean data if you have high write traffic. You can run out of disk space. It is quite similar to range partitions in RDBMS. I used to use it with mysql to clean disk. I would prefer to have it on server side since it introduces too much complexity on client side and CF overheads. I would like to know how to contribute to C* code base? Am I going to open a ticket and assign to me? Can someone provide me the path? Thank you very much! Best Regards, Cem
Re: Periodical deletes and compaction strategy
Hi Alain, I solved the same issue by implementing a client that manages time range partitions. Each time range partition is a CF. Cem. On Mon, Aug 26, 2013 at 11:34 AM, Alain RODRIGUEZ wrote: > Hi, > > Any guidance on this topic would be appreciated :). > > > 2013/8/23 Alain RODRIGUEZ > >> Hi, >> >> I am currently using about 10 CF to store temporal data. Those data are >> growing pretty big (hundreds of GB when I actually only need information >> from the last month - i.e. about hundreds of MB). >> >> I am going to delete old (and useless) data, I cannot always use TTL >> since I have counters too. Yet I know that deletes are a bit tricky in >> Cassandra, due to the fact that they are distributed. >> >> I was wondering about the best way to keep high performance and get rid >> of tombstones easily. >> >> I was considering 2 ways to do it : >> >> - Major compaction on these 10 CF to force them to always keep fresh data >> only and remove tombstones >> - LCS to have more chance to get all parts of the row in one SSTable, >> allowing tombstones to be removed eventually. >> >> What would be the better option (i.e. what would be the impact of both >> solutions) ? >> Do you need more information about those CF to answer this question ? >> >> Any insight is welcome, as always. >> >> Alain >> > >
Re: maximum storage per node
I dont think it is a good idea to put multiple instance in same machine. You may lose multiple instances at the same time if the machine goes down. You can also specify multiple directories as storage in 1.2. I am also not sure boot-strapping will be a big problem since the number keys you will store is relatively small. Why didnt you partition your data according to time instead of using your own compactor? Cem On Fri, Jul 26, 2013 at 3:50 AM, sankalp kohli wrote: > Try putting multiple instances per machine with each instance mapped to > its own disk. This might not work with v-nodes > > > On Thu, Jul 25, 2013 at 9:04 AM, Pruner, Anne (Anne) wrote: > >> I actually wrote my own compactor that deals with this problem. >> >> ** ** >> >> Anne >> >> ** ** >> >> *From:* cem [mailto:cayiro...@gmail.com] >> *Sent:* Thursday, July 25, 2013 11:59 AM >> >> *To:* user@cassandra.apache.org >> *Subject:* Re: maximum storage per node >> >> ** ** >> >> You will suffer from long compactions if you are planning to get rid of >> from old records by TTL. >> >> ** ** >> >> Best Regards, >> >> Cem. >> >> ** ** >> >> On Thu, Jul 25, 2013 at 5:51 PM, Kanwar Sangha >> wrote: >> >> Issues with large data nodes would be – >> >> >> >> · Nodetool repair will be impossible to run >> >> · Your read i/o will suffer since you will almost always go to >> disk (each read will take 3 IOPS worst case) >> >> · Boot-straping the node in case of failures will take days/weeks >> >> >> >> >> >> >> *From:* Pruner, Anne (Anne) [mailto:pru...@avaya.com] >> *Sent:* 25 July 2013 10:45 >> *To:* user@cassandra.apache.org >> *Subject:* RE: maximum storage per node >> >> >> >> We’re storing fairly large files (about 1MB apiece) for a few months and >> then deleting the oldest to get more space to add new ones. We have large >> requirements (maybe up to 100 TB), so having a 1TB limit would be >> unworkable. >> >> >> >> What is the reason for the limit? Does something fail after that? >> >> >> >> If there are hardware issues, what’s recommended? >> >> >> >> BTW, we’re using Cassandra 1.2 >> >> **** >> >> Anne >> >> >> >> *From:* cem [mailto:cayiro...@gmail.com ] >> *Sent:* Thursday, July 25, 2013 11:41 AM >> *To:* user@cassandra.apache.org >> *Subject:* Re: maximum storage per node >> >> >> >> Between 500GB - 1TB is recommended. >> >> >> >> But it depends also your hardware, traffic characteristics and >> requirements. Can you give some details on that? >> >> >> >> Best Regards, >> >> Cem >> >> >> >> On Thu, Jul 25, 2013 at 5:35 PM, Pruner, Anne (Anne) >> wrote: >> >> Does anyone have opinions on the maximum amount of data reasonable to >> store on one Cassandra node? If there are limitations, what are the >> reasons for it? >> >> >> >> Thanks, >> >> Anne >> >> >> >> ** ** >> > >
Re: maximum storage per node
You will suffer from long compactions if you are planning to get rid of from old records by TTL. Best Regards, Cem. On Thu, Jul 25, 2013 at 5:51 PM, Kanwar Sangha wrote: > Issues with large data nodes would be – > > ** ** > > **· **Nodetool repair will be impossible to run > > **· **Your read i/o will suffer since you will almost always go > to disk (each read will take 3 IOPS worst case) > > **· **Boot-straping the node in case of failures will take > days/weeks > > ** ** > > ** ** > > *From:* Pruner, Anne (Anne) [mailto:pru...@avaya.com] > *Sent:* 25 July 2013 10:45 > *To:* user@cassandra.apache.org > *Subject:* RE: maximum storage per node > > ** ** > > We’re storing fairly large files (about 1MB apiece) for a few months and > then deleting the oldest to get more space to add new ones. We have large > requirements (maybe up to 100 TB), so having a 1TB limit would be > unworkable. > > ** ** > > What is the reason for the limit? Does something fail after that? > > ** ** > > If there are hardware issues, what’s recommended? > > ** ** > > BTW, we’re using Cassandra 1.2 > > ** ** > > Anne > > ** ** > > *From:* cem [mailto:cayiro...@gmail.com ] > *Sent:* Thursday, July 25, 2013 11:41 AM > *To:* user@cassandra.apache.org > *Subject:* Re: maximum storage per node > > ** ** > > Between 500GB - 1TB is recommended. > > ** ** > > But it depends also your hardware, traffic characteristics and > requirements. Can you give some details on that? > > ** ** > > Best Regards, > > Cem > > ** ** > > On Thu, Jul 25, 2013 at 5:35 PM, Pruner, Anne (Anne) > wrote: > > Does anyone have opinions on the maximum amount of data reasonable to > store on one Cassandra node? If there are limitations, what are the > reasons for it? > > > > Thanks, > > Anne > > ** ** >
Re: maximum storage per node
Between 500GB - 1TB is recommended. But it depends also your hardware, traffic characteristics and requirements. Can you give some details on that? Best Regards, Cem On Thu, Jul 25, 2013 at 5:35 PM, Pruner, Anne (Anne) wrote: > Does anyone have opinions on the maximum amount of data reasonable to > store on one Cassandra node? If there are limitations, what are the > reasons for it? > > ** ** > > Thanks, > > Anne >
Re: Compression ratio
Thank you very much! On Fri, Jul 12, 2013 at 5:59 PM, Yuki Morishita wrote: > it's compressed/original. > > > https://github.com/apache/cassandra/blob/cassandra-1.1.11/src/java/org/apache/cassandra/io/sstable/SSTableMetadata.java#L124 > > On Fri, Jul 12, 2013 at 10:02 AM, cem wrote: > > Hi All, > > > > Can anyone explain the compression ratio? > > > > Is it the "compressed data / original" or "original/ compressed" ? Or > > something else. > > > > thanks a lot. > > > > Best Regards, > > Cem > > > > -- > Yuki Morishita > t:yukim (http://twitter.com/yukim) >
Compression ratio
Hi All, Can anyone explain the compression ratio? Is it the "compressed data / original" or "original/ compressed" ? Or something else. thanks a lot. Best Regards, Cem
Dropped mutation messages
Hi All, I have a cluster of 5 nodes with C* 1.2.4. Each node has 4 disks 1 TB each. I see a lot of dropped messages after it stores 400 GB per disk. (1.6 TB per node). The recommendation was 500 GB max per node before 1.2. Datastax says that we can store terabytes of data per node with 1.2. http://www.datastax.com/docs/1.2/cluster_architecture/cluster_planning Do I need to enable anything to leverage from 1.2? Do you have any other advice? What should be the path to investigate this? Thanks in advance! Best Regards, Cem.
Re: Is there anyone who implemented time range partitions with column families?
Thank you very much for the fast answer. Does playORM use different column families for each partition in Cassandra? Cem On Wed, May 29, 2013 at 5:30 PM, Jeremy Powell wrote: > Cem, yes, you can do this with C*, though you have to handle the logic > yourself (other libraries might do this for you, seen the dev of playORM > discuss some things which might be similar). We use Astyanax > and programmatically create CFs based on a time period of our choosing that > makes sense for our system, programmatically drop CFs if/when they are > outside a certain time period (rather than using C*'s TTL), and write data > to the different CFs as needed. > > ~Jeremy > > On Wed, May 29, 2013 at 8:36 AM, cem wrote: > >> Hi All, >> >> I used time range partitions 5 years ago with MySQL to clean up data much >> faster. >> >> I had a big FACT table with time range partitions and it was very is to >> drop old partitions (with archiving) and do some saving on disk. >> >> Has anyone implemented such a thing in Cassandra? It would be great if we >> have that in Cassandra. >> >> Best Regards, >> Cem. >> > >
Is there anyone who implemented time range partitions with column families?
Hi All, I used time range partitions 5 years ago with MySQL to clean up data much faster. I had a big FACT table with time range partitions and it was very is to drop old partitions (with archiving) and do some saving on disk. Has anyone implemented such a thing in Cassandra? It would be great if we have that in Cassandra. Best Regards, Cem.
Re: data clean up problem
Thanks for the answers! Cem On Wed, May 29, 2013 at 1:26 AM, Robert Coli wrote: > On Tue, May 28, 2013 at 2:38 PM, Bryan Talbot > wrote: > > I think what you're asking for (efficient removal of TTL'd write-once > data) > > is already in the works but not until 2.0 it seems. > > If your entire dataset in a keyspace or column family is deleted every > [small time period], then maybe use TRUNCATE? > > Two caveats : > > 1) https://issues.apache.org/jira/browse/CASSANDRA-4655 - TRUNCATE > will leave undelivered hints untouched until 1.2.6 > 2) TRUNCATE will create a snapshot unless you (unsafely!!! > danger) disable this feature [1] > > Alternately, perhaps a datastore with immutable data files is not for you? > :D > > =Rob > [1] https://issues.apache.org/jira/browse/CASSANDRA-3710 >
Re: data clean up problem
Thanks for the answer. Sorry for the misunderstanding. I tried to say I don't send delete request from the client so it safe to set gc_grace to 0. TTL is used for data clean up. I am not running a manual compaction. I tried that ones but it took a lot of time finish and I will not have this amount of off-peek time in the production to run this. I even set the compaction throughput to unlimited and it didnt help that much. Disk size just keeps on growing but I know that there is enough space to store 1 day data. What do you think about time rage partitioning? Creating new column family for each partition and drop when you know that all records are expired. I have 5 nodes. Cem. On Tue, May 28, 2013 at 9:37 PM, Hiller, Dean wrote: > Also, how many nodes are you running? > > From: cem mailto:cayiro...@gmail.com>> > Reply-To: "user@cassandra.apache.org<mailto:user@cassandra.apache.org>" < > user@cassandra.apache.org<mailto:user@cassandra.apache.org>> > Date: Tuesday, May 28, 2013 1:17 PM > To: "user@cassandra.apache.org<mailto:user@cassandra.apache.org>" < > user@cassandra.apache.org<mailto:user@cassandra.apache.org>> > Subject: Re: data clean up problem > > Thanks for the answer but it is already set to 0 since I don't do any > delete. > > Cem > > > On Tue, May 28, 2013 at 9:03 PM, Edward Capriolo <mailto:edlinuxg...@gmail.com>> wrote: > You need to change the gc_grace time of the column family. It defaults to > 10 days. By default the tombstones will not go away for 10 days. > > > On Tue, May 28, 2013 at 2:46 PM, cem cayiro...@gmail.com>> wrote: > Hi Experts, > > > We have general problem about cleaning up data from the disk. I need to > free the disk space after retention period and the customer wants to > dimension the disk space base on that. > > After running multiple performance tests with TTL of 1 day we saw that the > compaction couldn't keep up with the request rate. Disks were getting full > after 3 days. There were also a lot of sstables that are older than 1 day > after 3 days. > > Things that we tried: > > -Change the compaction strategy to leveled. (helped a bit but not much) > > -Use big sstable size (10G) with leveled compaction to have more > aggressive compaction.(helped a bit but not much) > > -Upgrade Cassandra from 1.0 to 1.2 to use TTL histograms (didn't help at > all since it has key overlapping estimation algorithm that generates %100 > match. Although we don't have...) > > Our column family structure is like this: > > Event_data_cf: (we store event data. Event_id is randomly generated and > each event has attributes like location=london) > > row data > > event id data blob > > timeseries_cf: (key is the attribute that we want to index. It can be > location=london, we didnt use secondary indexes because the indexes are > dynamic.) > > row data > > index key time series of event id (event1_id, event2_id….) > > timeseries_inv_cf: (this is used for removing event by event row key. ) > > row data > > event id set of index keys > > Candidate Solution: Implementing time range partitions. > > Each partition will have column family set and will be managed by client. > > Suppose that you want to have 7 days retention period. Then you can > configure the partition size as 1 day and have 7 active partitions at any > time. Then you can drop inactive partitions (older that 7 days). Dropping > will immediate remove the data from the disk. (With proper Cassandra.yaml > configuration) > > Storing an event: > > Find the current partition p1 > > store to event_data to Event_data_cf_p1 > > store to indexes to timeseries_cff_p1 > > store to inverted indexes to timeseries_inv_cf_p1 > > > A time range query with an index: > > Find the all partitions belongs to that time range > > Do read starting from the first partition until you reach to limit > > . > > Could you please provide your comments and concerns ? > > Is there any other option that we can try? > > What do you think about the candidate solution? > > Does anyone have the same issue? How would you solve it in another way? > > > Thanks in advance! > > Cem > > >
Re: data clean up problem
Thanks for the answer but it is already set to 0 since I don't do any delete. Cem On Tue, May 28, 2013 at 9:03 PM, Edward Capriolo wrote: > You need to change the gc_grace time of the column family. It defaults to > 10 days. By default the tombstones will not go away for 10 days. > > > On Tue, May 28, 2013 at 2:46 PM, cem wrote: > >> Hi Experts, >> >> We have general problem about cleaning up data from the disk. I need to >> free the disk space after retention period and the customer wants to >> dimension the disk space base on that. >> >> After running multiple performance tests with TTL of 1 day we saw that >> the compaction couldn't keep up with the request rate. Disks were getting >> full after 3 days. There were also a lot of sstables that are older than 1 >> day after 3 days. >> >> Things that we tried: >> >> -Change the compaction strategy to leveled. (helped a bit but not much) >> >> -Use big sstable size (10G) with leveled compaction to have more >> aggressive compaction.(helped a bit but not much) >> >> -Upgrade Cassandra from 1.0 to 1.2 to use TTL histograms (didn't help at >> all since it has key overlapping estimation algorithm that generates %100 >> match. Although we don't have...) >> >> Our column family structure is like this: >> >> Event_data_cf: (we store event data. Event_id is randomly generated and >> each event has attributes like location=london) >> >> row data >> >> event id data blob >> >> timeseries_cf: (key is the attribute that we want to index. It can be >> location=london, we didnt use secondary indexes because the indexes are >> dynamic.) >> >> row data >> >> index key time series of event id (event1_id, event2_id….) >> >> timeseries_inv_cf: (this is used for removing event by event row key. ) >> >> row data >> >> event id set of index keys >> >> Candidate Solution: Implementing time range partitions. >> >> Each partition will have column family set and will be managed by client. >> >> Suppose that you want to have 7 days retention period. Then you can >> configure the partition size as 1 day and have 7 active partitions at any >> time. Then you can drop inactive partitions (older that 7 days). Dropping >> will immediate remove the data from the disk. (With proper Cassandra.yaml >> configuration) >> >> Storing an event: >> >> Find the current partition p1 >> >> store to event_data to Event_data_cf_p1 >> >> store to indexes to timeseries_cff_p1 >> >> store to inverted indexes to timeseries_inv_cf_p1 >> >> >> A time range query with an index: >> >> Find the all partitions belongs to that time range >> >> Do read starting from the first partition until you reach to limit >> >> . >> >> Could you please provide your comments and concerns ? >> >> Is there any other option that we can try? >> >> What do you think about the candidate solution? >> >> Does anyone have the same issue? How would you solve it in another way? >> >> >> Thanks in advance! >> >> Cem >> > >
data clean up problem
Hi Experts, We have general problem about cleaning up data from the disk. I need to free the disk space after retention period and the customer wants to dimension the disk space base on that. After running multiple performance tests with TTL of 1 day we saw that the compaction couldn't keep up with the request rate. Disks were getting full after 3 days. There were also a lot of sstables that are older than 1 day after 3 days. Things that we tried: -Change the compaction strategy to leveled. (helped a bit but not much) -Use big sstable size (10G) with leveled compaction to have more aggressive compaction.(helped a bit but not much) -Upgrade Cassandra from 1.0 to 1.2 to use TTL histograms (didn't help at all since it has key overlapping estimation algorithm that generates %100 match. Although we don't have...) Our column family structure is like this: Event_data_cf: (we store event data. Event_id is randomly generated and each event has attributes like location=london) row data event id data blob timeseries_cf: (key is the attribute that we want to index. It can be location=london, we didnt use secondary indexes because the indexes are dynamic.) row data index key time series of event id (event1_id, event2_id….) timeseries_inv_cf: (this is used for removing event by event row key. ) row data event id set of index keys Candidate Solution: Implementing time range partitions. Each partition will have column family set and will be managed by client. Suppose that you want to have 7 days retention period. Then you can configure the partition size as 1 day and have 7 active partitions at any time. Then you can drop inactive partitions (older that 7 days). Dropping will immediate remove the data from the disk. (With proper Cassandra.yaml configuration) Storing an event: Find the current partition p1 store to event_data to Event_data_cf_p1 store to indexes to timeseries_cff_p1 store to inverted indexes to timeseries_inv_cf_p1 A time range query with an index: Find the all partitions belongs to that time range Do read starting from the first partition until you reach to limit . Could you please provide your comments and concerns ? Is there any other option that we can try? What do you think about the candidate solution? Does anyone have the same issue? How would you solve it in another way? Thanks in advance! Cem
Re: Cassandra 1.2 TTL histogram problem
Thanks for the answer. It means that if we use randompartioner it will be very difficult to find a sstable without any overlap. Let me give you an example from my test. I have ~50 sstables in total and an sstable with droppable ratio 0.9. I use GUID for key and only insert (no update -delete) so I dont expect a key in different sstables. I put extra logging to AbstractCompactionStrategy to see the overlaps.size() and keys and remainingKeys: overlaps.size() is around 30, number of keys for that sstable is around 5 M and remainingKeys is always 0. Are you sure that it is a good idea to estimate remainingKeys like that? Best Regards, Cem On Wed, May 22, 2013 at 5:58 PM, Yuki Morishita wrote: > > Can method calculate non-overlapping keys as overlapping? > > Yes. > And randomized keys don't matter here since sstables are sorted by > "token" calculated from key by your partitioner, and the method uses > sstable's min/max token to estimate overlap. > > On Tue, May 21, 2013 at 4:43 PM, cem wrote: > > Thank you very much for the swift answer. > > > > I have one more question about the second part. Can method calculate > > non-overlapping keys as overlapping? I mean it uses max and min tokens > and > > column count. They can be very close to each other if random keys are > used. > > > > In my use case I generate a GUID for each key and send a single write > > request. > > > > Cem > > > > On Tue, May 21, 2013 at 11:13 PM, Yuki Morishita > wrote: > >> > >> > Why does Cassandra single table compaction skips the keys that are in > >> > the other sstables? > >> > >> because we don't want to resurrect deleted columns. Say, sstable A has > >> the column with timestamp 1, and sstable B has the same column which > >> deleted at timestamp 2. Then if we purge that column only from sstable > >> B, we would see the column with timestamp 1 again. > >> > >> > I also dont understand why we have this line in > worthDroppingTombstones > >> > method > >> > >> What the method is trying to do is to "guess" how many columns that > >> are not in the rows that don't overlap, without actually going through > >> every rows in the sstable. We have statistics like column count > >> histogram, min and max row token for every sstables, we use those in > >> the method to estimate how many columns the two sstables overlap. > >> You may have remainingColumnsRatio of 0 when the two sstables overlap > >> almost entirely. > >> > >> > >> On Tue, May 21, 2013 at 3:43 PM, cem wrote: > >> > Hi all, > >> > > >> > I have a question about ticket > >> > https://issues.apache.org/jira/browse/CASSANDRA-3442 > >> > > >> > Why does Cassandra single table compaction skips the keys that are in > >> > the > >> > other sstables? Please correct if I am wrong. > >> > > >> > I also dont understand why we have this line in > worthDroppingTombstones > >> > method: > >> > > >> > double remainingColumnsRatio = ((double) columns) / > >> > (sstable.getEstimatedColumnCount().count() * > >> > sstable.getEstimatedColumnCount().mean()); > >> > > >> > remainingColumnsRatio is always 0 in my case and the droppableRatio > is > >> > 0.9. Cassandra skips all sstables which are already expired. > >> > > >> > This line was introduced by > >> > https://issues.apache.org/jira/browse/CASSANDRA-4022. > >> > > >> > Best Regards, > >> > Cem > >> > >> > >> > >> -- > >> Yuki Morishita > >> t:yukim (http://twitter.com/yukim) > > > > > > > > -- > Yuki Morishita > t:yukim (http://twitter.com/yukim) >
Re: Cassandra 1.2 TTL histogram problem
Thank you very much for the swift answer. I have one more question about the second part. Can method calculate non-overlapping keys as overlapping? I mean it uses max and min tokens and column count. They can be very close to each other if random keys are used. In my use case I generate a GUID for each key and send a single write request. Cem On Tue, May 21, 2013 at 11:13 PM, Yuki Morishita wrote: > > Why does Cassandra single table compaction skips the keys that are in > the other sstables? > > because we don't want to resurrect deleted columns. Say, sstable A has > the column with timestamp 1, and sstable B has the same column which > deleted at timestamp 2. Then if we purge that column only from sstable > B, we would see the column with timestamp 1 again. > > > I also dont understand why we have this line in worthDroppingTombstones > method > > What the method is trying to do is to "guess" how many columns that > are not in the rows that don't overlap, without actually going through > every rows in the sstable. We have statistics like column count > histogram, min and max row token for every sstables, we use those in > the method to estimate how many columns the two sstables overlap. > You may have remainingColumnsRatio of 0 when the two sstables overlap > almost entirely. > > > On Tue, May 21, 2013 at 3:43 PM, cem wrote: > > Hi all, > > > > I have a question about ticket > > https://issues.apache.org/jira/browse/CASSANDRA-3442 > > > > Why does Cassandra single table compaction skips the keys that are in the > > other sstables? Please correct if I am wrong. > > > > I also dont understand why we have this line in worthDroppingTombstones > > method: > > > > double remainingColumnsRatio = ((double) columns) / > > (sstable.getEstimatedColumnCount().count() * > > sstable.getEstimatedColumnCount().mean()); > > > > remainingColumnsRatio is always 0 in my case and the droppableRatio is > > 0.9. Cassandra skips all sstables which are already expired. > > > > This line was introduced by > > https://issues.apache.org/jira/browse/CASSANDRA-4022. > > > > Best Regards, > > Cem > > > > -- > Yuki Morishita > t:yukim (http://twitter.com/yukim) >
Cassandra 1.2 TTL histogram problem
Hi all, I have a question about ticket https://issues.apache.org/jira/browse/CASSANDRA-3442 Why does Cassandra single table compaction skips the keys that are in the other sstables? Please correct if I am wrong. I also dont understand why we have this line in worthDroppingTombstones method: double remainingColumnsRatio = ((double) columns) / (sstable. getEstimatedColumnCount<http://grepcode.com/file/repo1.maven.org/maven2/org.apache.cassandra/cassandra-all/1.2.2/org/apache/cassandra/io/sstable/SSTableReader.java#SSTableReader.getEstimatedColumnCount%28%29> ().count<http://grepcode.com/file/repo1.maven.org/maven2/org.apache.cassandra/cassandra-all/1.2.2/org/apache/cassandra/utils/EstimatedHistogram.java#EstimatedHistogram.count%28%29> () * sstable.getEstimatedColumnCount<http://grepcode.com/file/repo1.maven.org/maven2/org.apache.cassandra/cassandra-all/1.2.2/org/apache/cassandra/io/sstable/SSTableReader.java#SSTableReader.getEstimatedColumnCount%28%29> ().mean<http://grepcode.com/file/repo1.maven.org/maven2/org.apache.cassandra/cassandra-all/1.2.2/org/apache/cassandra/utils/EstimatedHistogram.java#EstimatedHistogram.mean%28%29> ()); remainingColumnsRatio is always *0 *in my case and the droppableRatio is *0.9. Cassandra skips all sstables which are already expired.* This line was introduced by https://issues.apache.org/jira/browse/CASSANDRA-4022. Best Regards, Cem
Re: Linear scalability problems
What was the RF before adding nodes? Sent from my iPhone On 04 Apr 2013, at 15:12, Anand Somani wrote: > We are using a single process with multiple threads, will look at client side > delays. > > Thanks > > On Wed, Apr 3, 2013 at 9:30 AM, Tyler Hobbs wrote: > If I had to guess, I would say that your client is the bottleneck, not the > cluster. Are you inserting data with multiple threads or processes? > > > On Wed, Apr 3, 2013 at 8:49 AM, Anand Somani wrote: > Hi, > > I am running some tests trying to scale out our application from using a 3 > node cluster to 6 node cluster. The thing I observed is that when using a 3 > node cluster I was able to handle abt 41 req/second, so I added 3 more nodes > thinking it should close to double, but instead it only goes upto bat 47 > req/second!! I am doing something wrong and it is not obvious, so wanted some > help in what stats could/should I monitor to tell me things like if a node > has more requests or if the load distribution is not random enough? > > Note I am using direct thrift (old code base) and cassandra 1.1.6. The data > model is for storing blobs (split across columns) and has around 6 CF, RF=3 > and all operations are at quorum. Also at the end of the run nodetool ring > reports the same data size. > > Thanks > Anand > > > > -- > Tyler Hobbs > DataStax >
Re: Vnodes - HUNDRED of MapReduce jobs
Hi Alicia , Cassandra input format creates mappers as many as vnodes. It is a known issue. You need to lower the number of vnodes :( I have a simple solution for that and ready to write a patch. Should I create a ticket about that? I don't know the procedure about that. Regards, Cem On Thu, Mar 28, 2013 at 2:30 PM, Alicia Leong wrote: > Hi All, > > I have 3 nodes of Cassandra 1.2.3 & edited the cassandra.yaml for vnodes. > > When I execute a M/R job .. the console showed HUNDRED of Map tasks. > > May I know, is the normal since is vnodes? If yes, this have slow the M/R > job to finish/complete. > > > Thanks >
Re: cassandra performance
Hi, Could you provide some other details about your schema design and queries? It is very hard to tell anything. Regards, Cem On Sun, Mar 24, 2013 at 12:40 PM, dong.yajun wrote: > Hello, > > I'd suggest you to take look at the difference between Nosql and RDMS. > > Best, > > On Sun, Mar 24, 2013 at 5:15 PM, 张刚 wrote: > >> Hello, >> I am new to Cassandra.I do some test on a single machine. I install >> Cassandra with a binary tarball distribution. >> I create a CF to store the data that get from MySQL. The CF has the same >> fields as the table in MySQL. So it looks like a table. >> I do the same select from the CF in Cassandra and the table in MySQL,and >> I find the processing time of MySQL is better than Cassandra. >> So,I wander what are the advantages of Cassandra compare MySQL and how to >> improve the performance of Cassandra. >> Is this the right way to use Cassandra. >> >> > > > -- > *Ric Dong* > >
Re: virtual nodes + map reduce = too many mappers
Thanks Eric for the appreciation :) Default split size is 64K rows. ColumnFamilyInputFormat first collects all tokens and create a split for each. if you have 256 vnode for each node that it creates 256 splits even if you have no data at all. current split size will only work if you have a vnode that has more than 64K rows. Possible solution that came to my mind: We can simply extend ColumnFamilySplit by adding a list of token ranges instead of one. Than no need create mapper for each token. Each mapper can do multiple range queries. But I don't know how to combine the range queries because in the typical range query you need to set start and end token. But in the virtual nodes I realized that tokens are not continuous. Best Regards, Cem On Sun, Feb 17, 2013 at 2:47 AM, Edward Capriolo wrote: > Split size does not have to equal block size. > > > http://hadoop.apache.org/docs/r1.1.1/api/org/apache/hadoop/mapred/lib/CombineFileInputFormat.html > > An abstract InputFormat that returns CombineFileSplit's in > InputFormat.getSplits(JobConf, int) method. Splits are constructed > from the files under the input paths. A split cannot have files from > different pools. Each split returned may contain blocks from different > files. If a maxSplitSize is specified, then blocks on the same node > are combined to form a single split. Blocks that are left over are > then combined with other blocks in the same rack. If maxSplitSize is > not specified, then blocks from the same rack are combined in a single > split; no attempt is made to create node-local splits. If the > maxSplitSize is equal to the block size, then this class is similar to > the default spliting behaviour in Hadoop: each block is a locally > processed split. Subclasses implement > InputFormat.getRecordReader(InputSplit, JobConf, Reporter) to > construct RecordReader's for CombineFileSplit's. > > Hive offers a CombinedHiveInputFormat > > https://issues.apache.org/jira/browse/HIVE-74 > > Essentially Combined input formats rock hard. If you have a directory > with say 2000 files, you do not want 2000 splits, and then the > overhead of starting stopping 2000 mappers. > > If you enable CombineInputFormat you can tune mapred.split.size and > the number of mappers is based (mostly) on the input size. This gives > jobs that would create too many map tasks way more throughput, and > stops them from monopolizing the map slots on the cluster. > > It would seem like all the extra splits from the vnode change could be > combined back together. > > On Sat, Feb 16, 2013 at 8:21 PM, Jonathan Ellis wrote: > > Wouldn't you have more than 256 splits anyway, given a normal amount of > data? > > > > (Default split size is 64k rows.) > > > > On Fri, Feb 15, 2013 at 7:01 PM, Edward Capriolo > wrote: > >> Seems like the hadoop Input format should combine the splits that are > >> on the same node into the same map task, like Hadoop's > >> CombinedInputFormat can. I am not sure who recommends vnodes as the > >> default, because this is now the second problem (that I know of) of > >> this class where vnodes has extra overhead, > >> https://issues.apache.org/jira/browse/CASSANDRA-5161 > >> > >> This seems to be the standard operating practice in c* now, enable > >> things in the default configuration like new partitioners and newer > >> features like vnodes, even though they are not heavily tested in the > >> wild or well understood, then deal with fallout. > >> > >> > >> On Fri, Feb 15, 2013 at 11:52 AM, cem wrote: > >>> Hi All, > >>> > >>> I have just started to use virtual nodes. I set the number of nodes to > 256 > >>> as recommended. > >>> > >>> The problem that I have is when I run a mapreduce job it creates node > * 256 > >>> mappers. It creates node * 256 splits. this effects the performance > since > >>> the range queries have a lot of overhead. > >>> > >>> Any suggestion to improve the performance? It seems like I need to > lower the > >>> number of virtual nodes. > >>> > >>> Best Regards, > >>> Cem > >>> > >>> > > > > > > > > -- > > Jonathan Ellis > > Project Chair, Apache Cassandra > > co-founder, http://www.datastax.com > > @spyced >
virtual nodes + map reduce = too many mappers
Hi All, I have just started to use virtual nodes. I set the number of nodes to 256 as recommended. The problem that I have is when I run a mapreduce job it creates node * 256 mappers. It creates node * 256 splits. this effects the performance since the range queries have a lot of overhead. Any suggestion to improve the performance? It seems like I need to lower the number of virtual nodes. Best Regards, Cem
Counter column families (pending replicate on write stage tasks)
Hi All, We are currently facing a performance issue with counter column families. I see lots of pending ReplicateOnWriteStage tasks in tpstats. Then I disabled replicate_on_write. It helped a lot. I want to use like that but I am not sure how to use it. I have 3 node setup with replication factor 2. I have listed my actions when this option is disable. 1. Using consistency level ALL for read operations. Since each replica might have different values. 2. Running node repair every day to sync all counters + set read repair change to 1.0 Please comment, correct and add to that list. Best Regards, Cem
Re: All host pools Marked Down
It should retry but it doesn't. It is also clear that it delegates the retry to the client " *Retry burden pushed out to client* " you can also check Hector code. I wrote a separate service that retries when this exception occurs. I think you have a problem with your load balancer. Try to connect with telnet. Cem. On Tue, May 29, 2012 at 3:06 PM, Shubham Srivastava < shubham.srivast...@makemytrip.com> wrote: > My webapp connects to the LoadBalancer IP which has the actual nodes in > its pool. > > If there is by any chance a connection break then will hector not retry > to re-establish connection I guess it should retry every XX seconds based > on retryDownedHostsDelayInSeconds . > > Regards, > Shubham > -- > *From:* cem [cayiro...@gmail.com] > *Sent:* Tuesday, May 29, 2012 6:13 PM > *To:* user@cassandra.apache.org > *Subject:* Re: All host pools Marked Down > > Since all hosts are seem to be down, Hector will not do retry. There > should be at least one node up in a cluster. Make sure that you have a > proper connection from your webapps to your cluster. > > Cem. > > On Tue, May 29, 2012 at 1:46 PM, Shubham Srivastava < > shubham.srivast...@makemytrip.com> wrote: > >> Any takers on this. Hitting us badly right now. >> >> Regards, >> Shubham >> -- >> *From:* Shubham Srivastava >> *Sent:* Tuesday, May 29, 2012 12:55 PM >> *To:* user@cassandra.apache.org >> *Subject:* All host pools Marked Down >> >> I am getting this exception lot of times >> >> >> >> *me.prettyprint.hector.api.exceptions.HectorException: All host pools >> marked down. Retry burden pushed out to client.* >> >> * * >> >> What this causes is no data read/write from the ring from my WebApp. >> >> >> >> I have retries as 3 and can see that max retries 3 getting exhausted with >> the same error as above. >> >> >> >> Checked cfstats and tpstats nothing seem to be a problem. >> >> >> >> However through the logs I see lot of time taken in compactions like the >> below >> >> >> >> *INFO [CompactionExecutor:73] 2012-05-29 11:03:01,605 >> CompactionManager.java (line 608) Compacted to >> /opt/cassandra-data/data/LH/UserPrefrences-tmp-g-8906-Data.db. 36,986,932 >> to 36,961,554 (~99% of original) bytes for 132,743 keys. Time: 112,910ms. >> * >> >> >> >> The time taken here seems pretty high. Will this cause a pause or read >> timeout etc. >> >> >> >> I have the connection from my web app through a hardware loadbalancer . >> Cassandra version is 0.8.6 with multi-DC ring on 6 nodes each in one DC. >> >> CL:1 and RF:3. >> >> >> >> Memeory:8Gb heap -> 14Gb Server memory with 8Core CPU. >> >> >> >> How do I move ahead in this. >> >> >> >> *Shubham Srivastava* *|* Technical Lead - Technology Development >> >> +91 124 4910 548 | MakeMyTrip.com, 243 SP Infocity, Udyog Vihar Phase >> 1, Gurgaon, Haryana - 122 016, India >> >> [image: http://www.mailmktg.makemytrip.com/signature/images/bulb.gif]*What's >> new?* My Trip Rewards - An exclusive loyalty program for MakeMyTrip >> customers. <https://rewards.makemytrip.com/MTR> >> >> [image: >> http://www.mailmktg.makemytrip.com/signature/images/MMT-signature-footer-V4.gif]<http://www.makemytrip.com/> >> >> [image: >> http://www.mailmktg.makemytrip.com/signature/images/map-icon.gif]<http://www.makemytrip.com/support/gurgaon-travel-agent-office.php> >> *Office Map* >> >> [image: >> http://www.mailmktg.makemytrip.com/signature/images/facebook-icon.gif]<http://www.facebook.com/pages/MakeMyTrip-Deals/120740541030?ref=search&sid=10077980239.1422657277..1> >> *Facebook* >> >> [image: >> http://www.mailmktg.makemytrip.com/signature/images/twitter-icon.gif]<http://twitter.com/makemytripdeals> >> *Twitter* >> >> >> > > <><><><><>
Re: All host pools Marked Down
Since all hosts are seem to be down, Hector will not do retry. There should be at least one node up in a cluster. Make sure that you have a proper connection from your webapps to your cluster. Cem. On Tue, May 29, 2012 at 1:46 PM, Shubham Srivastava < shubham.srivast...@makemytrip.com> wrote: > Any takers on this. Hitting us badly right now. > > Regards, > Shubham > -- > *From:* Shubham Srivastava > *Sent:* Tuesday, May 29, 2012 12:55 PM > *To:* user@cassandra.apache.org > *Subject:* All host pools Marked Down > > I am getting this exception lot of times > > > > *me.prettyprint.hector.api.exceptions.HectorException: All host pools > marked down. Retry burden pushed out to client.* > > * * > > What this causes is no data read/write from the ring from my WebApp. > > > > I have retries as 3 and can see that max retries 3 getting exhausted with > the same error as above. > > > > Checked cfstats and tpstats nothing seem to be a problem. > > > > However through the logs I see lot of time taken in compactions like the > below > > > > *INFO [CompactionExecutor:73] 2012-05-29 11:03:01,605 > CompactionManager.java (line 608) Compacted to > /opt/cassandra-data/data/LH/UserPrefrences-tmp-g-8906-Data.db. 36,986,932 > to 36,961,554 (~99% of original) bytes for 132,743 keys. Time: 112,910ms. > * > > > > The time taken here seems pretty high. Will this cause a pause or read > timeout etc. > > > > I have the connection from my web app through a hardware loadbalancer . > Cassandra version is 0.8.6 with multi-DC ring on 6 nodes each in one DC. > > CL:1 and RF:3. > > > > Memeory:8Gb heap -> 14Gb Server memory with 8Core CPU. > > > > How do I move ahead in this. > > > > *Shubham Srivastava* *|* Technical Lead - Technology Development > > +91 124 4910 548 | MakeMyTrip.com, 243 SP Infocity, Udyog Vihar Phase > 1, Gurgaon, Haryana - 122 016, India > > [image: http://www.mailmktg.makemytrip.com/signature/images/bulb.gif]*What's > new?* My Trip Rewards - An exclusive loyalty program for MakeMyTrip > customers. <https://rewards.makemytrip.com/MTR> > > [image: > http://www.mailmktg.makemytrip.com/signature/images/MMT-signature-footer-V4.gif]<http://www.makemytrip.com/> > > [image: > http://www.mailmktg.makemytrip.com/signature/images/map-icon.gif]<http://www.makemytrip.com/support/gurgaon-travel-agent-office.php> > *Office Map* > > [image: > http://www.mailmktg.makemytrip.com/signature/images/facebook-icon.gif]<http://www.facebook.com/pages/MakeMyTrip-Deals/120740541030?ref=search&sid=10077980239.1422657277..1> > *Facebook* > > [image: > http://www.mailmktg.makemytrip.com/signature/images/twitter-icon.gif]<http://twitter.com/makemytripdeals> > *Twitter* > > > <><><><><>
Re: TTL 3 hours + GC grace 0
Thank you for the swift response. Cem. On Sun, Mar 11, 2012 at 11:03 PM, Peter Schuller < peter.schul...@infidyne.com> wrote: > > I am using TTL 3 hours and GC grace 0 for a CF. I have a normal CF that > has > > records with TTL 3 hours and I dont send any delete request. I just > wonder > > if using GC grace 0 will cause any problem except extra Memory/IO/network > > load. I know that gc grace is for not transferring deleted records after > a > > down node comes back. So I assumed that transferring expired records > will > > not cause any problem. > > > > Do you have any idea? Thank you! > > If you do not perform any deletes at all, a GC grace of 0 should be > fine. But if you don't, the GC grace should not really be relevant > either. So I suggest leaving GC grace high in case you do start doing > deletes. > > Columns with TTL:s will disappear regardless of GC grace. > > If you do decide to run with short GC grace, be aware of the > consequencues: > http://wiki.apache.org/cassandra/Operations#Dealing_with_the_consequences_of_nodetool_repair_not_running_within_GCGraceSeconds > > -- > / Peter Schuller (@scode, http://worldmodscode.wordpress.com) >
TTL 3 hours + GC grace 0
Hi All, I am using TTL 3 hours and GC grace 0 for a CF. I have a normal CF that has records with TTL 3 hours and I dont send any delete request. I just wonder if using GC grace 0 will cause any problem except extra Memory/IO/network load. I know that gc grace is for not transferring deleted records after a down node comes back. So I assumed that transferring expired records will not cause any problem. Do you have any idea? Thank you! Regards, Cem.
