Hi Mikhail,
Could you please provide :
- your cluster version/topology (number of nodes, cpu, ram available etc)
- what kind of underlying storage you are using
- cfstat using -H option cause I'm never sure I'm converting bytes=>GB
You are storing 1Tb per node, so long running compaction is not really a
surprise, you can play with concurrent compaction thread number, compaction
throughput to begin with
On 27 April 2018 at 16:59, Mikhail Tsaplin <tsmis...@gmail.com> wrote:
> Hi,
> I have a five nodes C* cluster suffering from a big number of pending
> compaction tasks: 1) 571; 2) 91; 3) 367; 4) 22; 5) 232
>
> Initially, it was holding one big table (table_a). With Spark, I read that
> table, extended its data and stored in a second table_b. After this
> copying/extending process the number of compaction tasks in the cluster has
> grown up. From nodetool cfstats (see output at the bottom): table_a has 20
> SSTables and table_b has 18219.
>
> As I understand table_b has a big SSTables number because data was
> transferred from one table to another within a short time and eventually
> this tables will be compacted. But now I have to read whole data from this
> table_b and send it to Elasticsearch. When Spark reads this table some
> Cassandra nodes are dying because of OOM.
>
> I think that when compaction will be completed - the Spark reading job
> will work fine.
>
> The question is how can I speed up compaction process, what if I will add
> another two nodes to cluster - will compaction finish faster? Or data will
> be copied to new nodes but compaction will continue on the original set of
> SSTables?
>
>
> *Nodetool cfstats output:
>
> Table: table_a
> SSTable count: 20
> Space used (live): 1064889308052
> Space used (total): 1064889308052
> Space used by snapshots (total): 0
> Off heap memory used (total): 1118106937
> SSTable Compression Ratio: 0.12564594959566894
> Number of keys (estimate): 56238959
> Memtable cell count: 76824
> Memtable data size: 115531402
> Memtable off heap memory used: 0
> Memtable switch count: 17
> Local read count: 0
> Local read latency: NaN ms
> Local write count: 77308
> Local write latency: 0.045 ms
> Pending flushes: 0
> Bloom filter false positives: 0
> Bloom filter false ratio: 0.00000
> Bloom filter space used: 120230328
> Bloom filter off heap memory used: 120230168
> Index summary off heap memory used: 2837249
> Compression metadata off heap memory used: 995039520
> Compacted partition minimum bytes: 1110
> Compacted partition maximum bytes: 52066354
> Compacted partition mean bytes: 133152
> Average live cells per slice (last five minutes): NaN
> Maximum live cells per slice (last five minutes): 0
> Average tombstones per slice (last five minutes): NaN
> Maximum tombstones per slice (last five minutes): 0
>
>
> nodetool cfstats table_b
> Keyspace: dump_es
> Read Count: 0
> Read Latency: NaN ms.
> Write Count: 0
> Write Latency: NaN ms.
> Pending Flushes: 0
> Table: table_b
> SSTable count: 18219
> Space used (live): 1316641151665
> Space used (total): 1316641151665
> Space used by snapshots (total): 0
> Off heap memory used (total): 3863604976
> SSTable Compression Ratio: 0.20387645535477916
> Number of keys (estimate): 712032622
> Memtable cell count: 0
> Memtable data size: 0
> Memtable off heap memory used: 0
> Memtable switch count: 0
> Local read count: 0
> Local read latency: NaN ms
> Local write count: 0
> Local write latency: NaN ms
> Pending flushes: 0
> Bloom filter false positives: 0
> Bloom filter false ratio: 0.00000
> Bloom filter space used: 2382971488
> Bloom filter off heap memory used: 2742320056
> Index summary off heap memory used: 371500752
> Compression metadata off heap memory used: 749784168
> Compacted partition minimum bytes: 771
> Compacted partition maximum bytes: 1629722
> Compacted partition mean bytes: 3555
> Average live cells per slice (last five minutes): 132.375
> Maximum live cells per slice (last five minutes): 149
> Average tombstones per slice (last five minutes): 1.0
> Maximum tombstones per slice (last five minutes): 1
>
>
> ------------------
>
>
> I logged CQL requests going from Spark and checked how one such request is
> performing - it fetches 8075rows, 59mb data in 155s (see below check output)
>
> $ date; echo 'SELECT "scan_id", "snapshot_id", "scan_doc", "snapshot_doc"
> FROM "dump_es"."table_b" WHERE token("scan_id") > 946122293981930504 AND
> token("scan_id") <= 946132293981
> 930504 ALLOW FILTERING;' | cqlsh --request-timeout=3600 | wc ; date
>
>
> Fri Apr 27 13:32:55 UTC 2018
> 8076 61191 59009831
> Fri Apr 27 13:35:30 UTC 2018
>
>
>
