How many columns are you inserting/querying per key? Could we see some example CQL statements for the insert/read workload?
If you are maxing out at 10 clients, something fishy is going on. In general, though, if you find that adding nodes causes performance to degrade I would suspect that you are querying data in one CQL statement that is spread over multiple partitions, and so extra work needs to be done cross-cluster to service your requests as more nodes are added. I would also consider what effect the file cache may be having on your workload, as it sounds small enough to fit in memory, so is likely a major determining factor for performance of your benchmark. As you try different client levels for the smaller cluster you may see improved performance as the data is pulled into file cache across test runs, and then when you build your larger cluster this is lost so performance appears to degrade (for instance). On Fri, Jul 18, 2014 at 12:25 PM, Diane Griffith <dfgriff...@gmail.com> wrote: > The column family schema is: > > CREATE TABLE IF NOT EXISTS foo (key text, col_name text, col_value text, > PRIMARY KEY(key, col_name)) > > where the key is a generated uuid and all keys were inserted in random > order but in the end we were compacting down to one sstable per node. > > So we were doing it this way to achieve dynamic columns. > > Thanks, > Diane > > On Fri, Jul 18, 2014 at 12:19 AM, Jack Krupansky <j...@basetechnology.com> > wrote: > >> Sorry I may have confused the discussion by mentioning tokens – I >> wasn’t intending to refer to vnodes or the num_tokens property, but merely >> referring to the token range of a node and that the partition key hashes to >> a token value. >> >> The main question is what you use for your primary key and whether you >> are using a small number of partition keys and a large number of clustering >> columns, or does each row have a unique partition key and no clustering >> columns. >> >> -- Jack Krupansky >> >> *From:* Diane Griffith <dfgriff...@gmail.com> >> *Sent:* Thursday, July 17, 2014 6:21 PM >> *To:* user <user@cassandra.apache.org> >> *Subject:* Re: horizontal query scaling issues follow on >> >> So do partitions equate to tokens/vnodes? >> >> If so we had configured all cluster nodes/vms with num_tokens: 256 >> instead of setting init_token and assigning ranges. I am still not getting >> why in Cassandra 2.0, I would assign my own ranges via init_token and this >> was based on the documentation and even this blog item >> <http://www.datastax.com/dev/blog/virtual-nodes-in-cassandra-1-2> that >> made it seem right for us to always configure our cluster vms with >> num_tokens: 256 in the cassandra.yaml file. >> >> Also in all testing, all vms were of equal sizing so one was not more >> powerful than another. >> >> I didn't think I was hitting an i/o wall on the client vm (separate vm) >> where we command line scripted our query call to the cassandra cluster. >> I can break the client call load across vms which I tried early on. Happy >> to verify that again though. >> >> So given that I was assuming the partitions were such that it wasn't a >> problem. Is that an incorrect assumption and something to dig into more? >> >> Thanks, >> Diane >> >> >> On Thu, Jul 17, 2014 at 3:01 PM, Jack Krupansky <j...@basetechnology.com> >> wrote: >> >>> How many partitions are you spreading those 18 million rows over? >>> That many rows in a single partition will not be a sweet spot for >>> Cassandra. It’s not exceeding any hard limit (2 billion), but some internal >>> operations may cache the partition rather than the logical row. >>> >>> And all those rows in a single partition would certainly not be a test >>> of “horizontal scaling” (adding nodes to handle more data – more token >>> values or partitions.) >>> >>> -- Jack Krupansky >>> >>> *From:* Diane Griffith <dfgriff...@gmail.com> >>> *Sent:* Thursday, July 17, 2014 1:33 PM >>> *To:* user <user@cassandra.apache.org> >>> *Subject:* horizontal query scaling issues follow on >>> >>> >>> This is a follow on re-post to clarify what we are trying to do, >>> providing information that was missing or not clear. >>> >>> >>> >>> Goal: Verify horizontal scaling for random non duplicating key reads >>> using the simplest configuration (or minimal configuration) possible. >>> >>> >>> >>> Background: >>> >>> A couple years ago we did similar performance testing with Cassandra for >>> both read and write performance and found excellent (essentially linear) >>> horizontal scalability. That project got put on hold. We are now moving >>> forward with an operational system and are having scaling problems. >>> >>> >>> >>> During the prior testing (3 years ago) we were using a much older >>> version of Cassandra (0.8 or older), the THRIFT API, and Amazon AWS rather >>> than OpenStack VMs. We are now using the latest Cassandra and the CQL >>> interface. We did try moving from OpenStack to AWS/EC2 but that did not >>> materially change our (poor) results. >>> >>> >>> >>> Test Procedure: >>> >>> - Inserted 54 million cells in 18 million rows (so 3 cells per row), >>> using randomly generated row keys. That was to be our data control for >>> the >>> test. >>> - Spawn a client on a different VM to query 100k rows and do that >>> for 100 reps. Each row key queried is drawn randomly from the set of >>> existing row keys, and then not re-used, so all 10 million row queries >>> use >>> a different (valid) row key. This test is a specific use case of our >>> system we are trying to show will scale >>> >>> Result: >>> >>> - 2 nodes performed better than 1 node test but 4 nodes showed >>> decreased performance over 2 nodes. So that did not show horizontal >>> scaling >>> >>> >>> >>> Notes: >>> >>> - We have replication factor set to 1 as we were trying to keep the >>> control test simple to prove out horizontal scaling. >>> - When we tried to add threading to see if it would help it had >>> interesting side behavior which did not prove out horizontal scaling. >>> - We are using CQL versus THRIFT API for Cassandra 2.0.6 >>> >>> >>> >>> >>> >>> Does anyone have any feedback that either threading or replication >>> factor is necessary to show horizontal scaling of Cassandra versus the >>> minimal way of just continue to add nodes to help throughput? >>> >>> >>> >>> Any suggestions of minimal configuration necessary to show scaling of >>> our query use case 100k requests for random non repeating keys constantly >>> coming in over a period of time? >>> >>> >>> Thanks, >>> >>> Diane >>> >> >> > >
