Martin Bligh created CASSANDRA-7103: ---------------------------------------
Summary: Very poor performance with simple setup Key: CASSANDRA-7103 URL: https://issues.apache.org/jira/browse/CASSANDRA-7103 Project: Cassandra Issue Type: Bug Components: Core Environment: Fedora 19 (also happens on Ubuntu), Cassandra 2.0.7. dsc standard install Reporter: Martin Bligh Single node (this is just development, 32GB 20 core server), single disk array. Create the following table: CREATE TABLE reut ( time_order bigint, time_start bigint, ack_us map<int, int>, gc_strategy map<text, int>, gc_strategy_symbol map<text, int>, gc_symbol map<text, int>, ge_strategy map<text, int>, ge_strategy_symbol map<text, int>, ge_symbol map<text, int>, go_strategy map<text, int>, go_strategy_symbol map<text, int>, go_symbol map<text, int>, message_type map<text, int>, PRIMARY KEY (time_order, time_start) ) WITH bloom_filter_fp_chance=0.010000 AND caching='KEYS_ONLY' AND comment='' AND dclocal_read_repair_chance=0.000000 AND gc_grace_seconds=864000 AND index_interval=128 AND read_repair_chance=0.100000 AND replicate_on_write='true' AND populate_io_cache_on_flush='false' AND default_time_to_live=0 AND speculative_retry='99.0PERCENTILE' AND memtable_flush_period_in_ms=0 AND compaction={'class': 'SizeTieredCompactionStrategy'} AND compression={}; Now I just insert data into it (using python driver, async insert, prepared insert statement). Each row only fills out one of the gc_*, go_*, or ge_* columns, and there's something like 20-100 entries per map column, occasionally 1000, but it's nothing huge. First run 685 inserts in 1.004860 seconds (681.687053 Operations/s). OK, not great, but that's fine. Now throw 50,000 rows at it. Now run the first run again, and it takes 53s to do the same insert of 685 rows - I'm getting about 10 rows per second. It's not IO bound - "iostat 1" shows quiescent for 9 seconds, then ~640KB write, then sleeps again - seems like the fflush sync. Run "nodetool flush" and performance goes back to as before!!!! Not sure why this gets so slow - I think it just builds huge commit logs and memtables, but never writes out to the data/ directory with sstables because I only have one table? That doesn't seem like a good situation. Worse ... if you let the python driver just throw stuff at it async (I think this allows up to 128 request if I understand the underlying protocol, then it gets so slow that a single write takes over 10s and times out). Seems to be some sort of synchronization problem in Java ... if I limit the concurrent async requests to the left column below, I get the number of seconds elapsed on the right: 1: 103 seconds 2: 63 seconds 8: 53 seconds 16: 53 seconds 32: 66 seconds 64: so slow it explodes in timeouts on write (over 10s each). I guess there's some thundering herd type locking issue in whatever Java primitive you are using to lock concurrent access to a single table. I know some of the Java concurrent.* stuff has this issue. So for the other tests above, I was limiting async writes to 16 pending. -- This message was sent by Atlassian JIRA (v6.2#6252)