Andrei, Oh, Monday mornings...Tb :)
On Mon, Nov 24, 2014 at 9:12 AM, Andrei Ivanov <aiva...@iponweb.net> wrote: > Nikolai, > > Are you sure about 1.26Gb? Like it doesn't look right - 5195 tables > with 256Mb table size... > > Andrei > > On Mon, Nov 24, 2014 at 5:09 PM, Nikolai Grigoriev <ngrigor...@gmail.com> > wrote: > > Jean-Armel, > > > > I have only two large tables, the rest is super-small. In the test > cluster > > of 15 nodes the largest table has about 110M rows. Its total size is > about > > 1,26Gb per node (total disk space used per node for that CF). It's got > about > > 5K sstables per node - the sstable size is 256Mb. cfstats on a "healthy" > > node look like this: > > > > Read Count: 8973748 > > Read Latency: 16.130059053251774 ms. > > Write Count: 32099455 > > Write Latency: 1.6124713938912671 ms. > > Pending Tasks: 0 > > Table: wm_contacts > > SSTable count: 5195 > > SSTables in each level: [27/4, 11/10, 104/100, 1053/1000, 4000, > 0, > > 0, 0, 0] > > Space used (live), bytes: 1266060391852 > > Space used (total), bytes: 1266144170869 > > SSTable Compression Ratio: 0.32604853410787327 > > Number of keys (estimate): 25696000 > > Memtable cell count: 71402 > > Memtable data size, bytes: 26938402 > > Memtable switch count: 9489 > > Local read count: 8973748 > > Local read latency: 17.696 ms > > Local write count: 32099471 > > Local write latency: 1.732 ms > > Pending tasks: 0 > > Bloom filter false positives: 32248 > > Bloom filter false ratio: 0.50685 > > Bloom filter space used, bytes: 20744432 > > Compacted partition minimum bytes: 104 > > Compacted partition maximum bytes: 3379391 > > Compacted partition mean bytes: 172660 > > Average live cells per slice (last five minutes): 495.0 > > Average tombstones per slice (last five minutes): 0.0 > > > > Another table of similar structure (same number of rows) is about 4x > times > > smaller. That table does not suffer from those issues - it compacts well > and > > efficiently. > > > > On Mon, Nov 24, 2014 at 2:30 AM, Jean-Armel Luce <jaluc...@gmail.com> > wrote: > >> > >> Hi Nikolai, > >> > >> Please could you clarify a little bit what you call "a large amount of > >> data" ? > >> > >> How many tables ? > >> How many rows in your largest table ? > >> How many GB in your largest table ? > >> How many GB per node ? > >> > >> Thanks. > >> > >> > >> > >> 2014-11-24 8:27 GMT+01:00 Jean-Armel Luce <jaluc...@gmail.com>: > >>> > >>> Hi Nikolai, > >>> > >>> Thanks for those informations. > >>> > >>> Please could you clarify a little bit what you call " > >>> > >>> 2014-11-24 4:37 GMT+01:00 Nikolai Grigoriev <ngrigor...@gmail.com>: > >>>> > >>>> Just to clarify - when I was talking about the large amount of data I > >>>> really meant large amount of data per node in a single CF (table). > LCS does > >>>> not seem to like it when it gets thousands of sstables (makes 4-5 > levels). > >>>> > >>>> When bootstraping a new node you'd better enable that option from > >>>> CASSANDRA-6621 (the one that disables STCS in L0). But it will still > be a > >>>> mess - I have a node that I have bootstrapped ~2 weeks ago. Initially > it had > >>>> 7,5K pending compactions, now it has almost stabilized ad 4,6K. Does > not go > >>>> down. Number of sstables at L0 is over 11K and it is slowly slowly > building > >>>> upper levels. Total number of sstables is 4x the normal amount. Now I > am not > >>>> entirely sure if this node will ever get back to normal life. And > believe me > >>>> - this is not because of I/O, I have SSDs everywhere and 16 physical > cores. > >>>> This machine is barely using 1-3 cores at most of the time. The > problem is > >>>> that allowing STCS fallback is not a good option either - it will > quickly > >>>> result in a few 200Gb+ sstables in my configuration and then these > sstables > >>>> will never be compacted. Plus, it will require close to 2x disk space > on > >>>> EVERY disk in my JBOD configuration...this will kill the node sooner > or > >>>> later. This is all because all sstables after bootstrap end at L0 and > then > >>>> the process slowly slowly moves them to other levels. If you have > write > >>>> traffic to that CF then the number of sstables and L0 will grow > quickly - > >>>> like it happens in my case now. > >>>> > >>>> Once something like > https://issues.apache.org/jira/browse/CASSANDRA-8301 > >>>> is implemented it may be better. > >>>> > >>>> > >>>> On Sun, Nov 23, 2014 at 4:53 AM, Andrei Ivanov <aiva...@iponweb.net> > >>>> wrote: > >>>>> > >>>>> Stephane, > >>>>> > >>>>> We are having a somewhat similar C* load profile. Hence some comments > >>>>> in addition Nikolai's answer. > >>>>> 1. Fallback to STCS - you can disable it actually > >>>>> 2. Based on our experience, if you have a lot of data per node, LCS > >>>>> may work just fine. That is, till the moment you decide to join > >>>>> another node - chances are that the newly added node will not be able > >>>>> to compact what it gets from old nodes. In your case, if you switch > >>>>> strategy the same thing may happen. This is all due to limitations > >>>>> mentioned by Nikolai. > >>>>> > >>>>> Andrei, > >>>>> > >>>>> > >>>>> On Sun, Nov 23, 2014 at 8:51 AM, Servando Muñoz G. <smg...@gmail.com > > > >>>>> wrote: > >>>>> > ABUSE > >>>>> > > >>>>> > > >>>>> > > >>>>> > YA NO QUIERO MAS MAILS SOY DE MEXICO > >>>>> > > >>>>> > > >>>>> > > >>>>> > De: Nikolai Grigoriev [mailto:ngrigor...@gmail.com] > >>>>> > Enviado el: sábado, 22 de noviembre de 2014 07:13 p. m. > >>>>> > Para: user@cassandra.apache.org > >>>>> > Asunto: Re: Compaction Strategy guidance > >>>>> > Importancia: Alta > >>>>> > > >>>>> > > >>>>> > > >>>>> > Stephane, > >>>>> > > >>>>> > As everything good, LCS comes at certain price. > >>>>> > > >>>>> > LCS will put most load on you I/O system (if you use spindles - you > >>>>> > may need > >>>>> > to be careful about that) and on CPU. Also LCS (by default) may > fall > >>>>> > back to > >>>>> > STCS if it is falling behind (which is very possible with heavy > >>>>> > writing > >>>>> > activity) and this will result in higher disk space usage. Also LCS > >>>>> > has > >>>>> > certain limitation I have discovered lately. Sometimes LCS may not > be > >>>>> > able > >>>>> > to use all your node's resources (algorithm limitations) and this > >>>>> > reduces > >>>>> > the overall compaction throughput. This may happen if you have a > >>>>> > large > >>>>> > column family with lots of data per node. STCS won't have this > >>>>> > limitation. > >>>>> > > >>>>> > > >>>>> > > >>>>> > By the way, the primary goal of LCS is to reduce the number of > >>>>> > sstables C* > >>>>> > has to look at to find your data. With LCS properly functioning > this > >>>>> > number > >>>>> > will be most likely between something like 1 and 3 for most of the > >>>>> > reads. > >>>>> > But if you do few reads and not concerned about the latency today, > >>>>> > most > >>>>> > likely LCS may only save you some disk space. > >>>>> > > >>>>> > > >>>>> > > >>>>> > On Sat, Nov 22, 2014 at 6:25 PM, Stephane Legay > >>>>> > <sle...@looplogic.com> > >>>>> > wrote: > >>>>> > > >>>>> > Hi there, > >>>>> > > >>>>> > > >>>>> > > >>>>> > use case: > >>>>> > > >>>>> > > >>>>> > > >>>>> > - Heavy write app, few reads. > >>>>> > > >>>>> > - Lots of updates of rows / columns. > >>>>> > > >>>>> > - Current performance is fine, for both writes and reads.. > >>>>> > > >>>>> > - Currently using SizedCompactionStrategy > >>>>> > > >>>>> > > >>>>> > > >>>>> > We're trying to limit the amount of storage used during compaction. > >>>>> > Should > >>>>> > we switch to LeveledCompactionStrategy? > >>>>> > > >>>>> > > >>>>> > > >>>>> > Thanks > >>>>> > > >>>>> > > >>>>> > > >>>>> > > >>>>> > -- > >>>>> > > >>>>> > Nikolai Grigoriev > >>>>> > >>>> > >>>> > >>>> > >>>> -- > >>>> Nikolai Grigoriev > >>>> > >>> > >> > > > > > > > > -- > > Nikolai Grigoriev > > > -- Nikolai Grigoriev (514) 772-5178
