Hi, I came across the following method ( https://github.com/apache/cassandra/blob/afd68abe60742c6deb6357ba4605268dfb3d06ea/src/java/org/apache/cassandra/service/StorageService.java#L5006-L5021). It seems data is evenly split across disks according to local token ranges.
It might be that data stored is not evenly spread across your partition key and thus the imbalance in disk usage. Regards, Akhil On Mon, Jun 12, 2017 at 4:59 PM, Erick Ramirez <flightc...@gmail.com> wrote: > That's the cause of the imbalance -- an excessively large sstable which > suggests to me that at some point you performed a manual major compaction > with nodetool compact. > > If the table is using STCS, there won't be other compaction partners in > the near future so you split the sstable manually with sstablesplit > (offline tool so requires C* on the node to be shutdown temporarily). > > > On Mon, Jun 12, 2017 at 1:25 PM, Xihui He <xihu...@gmail.com> wrote: > >> Hi Vladimir, >> >> The disks size are all the same, 1.8T as show in df, and only used by >> cassandra. >> It seems to me that maybe the compacted lagetst file is on data01 which >> uses 1.1T. >> >> Thanks, >> Xihui >> >> On 11 June 2017 at 17:26, Vladimir Yudovin <vla...@winguzone.com> wrote: >> >>> Hi, >>> >>> Do your disks have the same size? AFAK Cassandra distributes data with >>> proportion to disk size, i.e. keeps the same percent of busy space. >>> >>> Best regards, Vladimir Yudovin, >>> *Winguzone <https://winguzone.com?from=list> - Cloud Cassandra Hosting* >>> >>> >>> ---- On Wed, 07 Jun 2017 06:15:48 -0400 *Xihui He <xihu...@gmail.com >>> <xihu...@gmail.com>>* wrote ---- >>> >>> Dear All, >>> >>> We are using multiple disks per node and find the data is not evenly >>> distributed (data01 uses 1.1T, but data02 uses 353G). Is this expected? If >>> data01 becomes full, would the node be still writable? We are using 2.2.6. >>> >>> Thanks, >>> Xihui >>> >>> data_file_directories: >>> - /data00/cassandra >>> - /data01/cassandra >>> - /data02/cassandra >>> - /data03/cassandra >>> - /data04/cassandra >>> >>> df >>> /dev/sde1 1.8T 544G 1.2T 32% /data03 >>> /dev/sdc1 1.8T 1.1T 683G 61% /data01 >>> /dev/sdf1 1.8T 491G 1.3T 29% /data04 >>> /dev/sdd1 1.8T 353G 1.4T 21% /data02 >>> /dev/sdb1 1.8T 285G 1.5T 17% /data00 >>> >>> root@n9-016-015:~# du -sh /data01/cassandra/album_media_feature/* >>> 143M /data01/cassandra/album_media_feature/media_feature_blur-066 >>> e5700c41511e5beacf197ae340934 >>> 4.4G /data01/cassandra/album_media_feature/media_feature_c1-dbadf >>> 930c41411e5974743d3a691d887 >>> 56K /data01/cassandra/album_media_feature/media_feature_duplicat >>> e-09d4b380c41511e58501e9aa37be91a5 >>> 16K /data01/cassandra/album_media_feature/media_feature_emotion- >>> b8570470054d11e69fb88f073bab8267 >>> 240M /data01/cassandra/album_media_feature/media_feature_exposure >>> -f55449c0c41411e58f5c9b66773b60c3 >>> 649M /data01/cassandra/album_media_feature/media_feature_group-f8 >>> de0cc0c41411e5827b995f709095c8 >>> 22G /data01/cassandra/album_media_feature/media_feature_multi_cl >>> ass-cf3bb72006c511e69fb88f073bab8267 >>> 44K /data01/cassandra/album_media_feature/media_feature_pool5-11 >>> 85b200c41511e5b7d8757e25e34d67 >>> 15G /data01/cassandra/album_media_feature/media_feature_poster-f >>> cf45850c41411e597bb1507d1856305 >>> 8.0K /data01/cassandra/album_media_feature/media_feature_quality- >>> 155d9500c41511e5974743d3a691d887 >>> 17G /data01/cassandra/album_media_feature/media_feature_quality_ >>> rc-51babf50dba811e59fb88f073bab8267 >>> 8.7G /data01/cassandra/album_media_feature/media_feature_scene-00 >>> 8a5050c41511e59ebcc3582d286c8d >>> 8.0K /data01/cassandra/album_media_feature/media_region_features_ >>> v4-29a0cd10150611e6bd3e3f41faa2612a >>> 971G /data01/cassandra/album_media_feature/media_region_features_ >>> v5-1b805470a3d711e68121757e9ac51b7b >>> >>> root@n9-016-015:~# du -sh /data02/cassandra/album_media_feature/* >>> 1.6G /data02/cassandra/album_media_feature/media_feature_blur-066 >>> e5700c41511e5beacf197ae340934 >>> 44G /data02/cassandra/album_media_feature/media_feature_c1-dbadf >>> 930c41411e5974743d3a691d887 >>> 64K /data02/cassandra/album_media_feature/media_feature_duplicat >>> e-09d4b380c41511e58501e9aa37be91a5 >>> 75G /data02/cassandra/album_media_feature/media_feature_emotion- >>> b8570470054d11e69fb88f073bab8267 >>> 2.0G /data02/cassandra/album_media_feature/media_feature_exposure >>> -f55449c0c41411e58f5c9b66773b60c3 >>> 21G /data02/cassandra/album_media_feature/media_feature_group-f8 >>> de0cc0c41411e5827b995f709095c8 >>> 336M /data02/cassandra/album_media_feature/media_feature_multi_cl >>> ass-cf3bb72006c511e69fb88f073bab8267 >>> 44K /data02/cassandra/album_media_feature/media_feature_pool5-11 >>> 85b200c41511e5b7d8757e25e34d67 >>> 2.0G /data02/cassandra/album_media_feature/media_feature_poster-f >>> cf45850c41411e597bb1507d1856305 >>> 8.0K /data02/cassandra/album_media_feature/media_feature_quality- >>> 155d9500c41511e5974743d3a691d887 >>> 17G /data02/cassandra/album_media_feature/media_feature_quality_ >>> rc-51babf50dba811e59fb88f073bab8267 >>> 141M /data02/cassandra/album_media_feature/media_feature_scene-00 >>> 8a5050c41511e59ebcc3582d286c8d >>> 8.0K /data02/cassandra/album_media_feature/media_region_features_ >>> v4-29a0cd10150611e6bd3e3f41faa2612a >>> 93G /data02/cassandra/album_media_feature/media_region_features_ >>> v5-1b805470a3d711e68121757e9ac51b7b >>> >>> root@n9-016-015:~# du -sh /data03/cassandra/album_media_feature/* >>> 4.3G /data03/cassandra/album_media_feature/media_feature_blur-066 >>> e5700c41511e5beacf197ae340934 >>> 19G /data03/cassandra/album_media_feature/media_feature_c1-dbadf >>> 930c41411e5974743d3a691d887 >>> 72K /data03/cassandra/album_media_feature/media_feature_duplicat >>> e-09d4b380c41511e58501e9aa37be91a5 >>> 2.8G /data03/cassandra/album_media_feature/media_feature_emotion- >>> b8570470054d11e69fb88f073bab8267 >>> 105M /data03/cassandra/album_media_feature/media_feature_exposure >>> -f55449c0c41411e58f5c9b66773b60c3 >>> 15G /data03/cassandra/album_media_feature/media_feature_group-f8 >>> de0cc0c41411e5827b995f709095c8 >>> 23G /data03/cassandra/album_media_feature/media_feature_multi_cl >>> ass-cf3bb72006c511e69fb88f073bab8267 >>> 44K /data03/cassandra/album_media_feature/media_feature_pool5-11 >>> 85b200c41511e5b7d8757e25e34d67 >>> 17G /data03/cassandra/album_media_feature/media_feature_poster-f >>> cf45850c41411e597bb1507d1856305 >>> 8.0K /data03/cassandra/album_media_feature/media_feature_quality- >>> 155d9500c41511e5974743d3a691d887 >>> 294M /data03/cassandra/album_media_feature/media_feature_quality_ >>> rc-51babf50dba811e59fb88f073bab8267 >>> 5.1G /data03/cassandra/album_media_feature/media_feature_scene-00 >>> 8a5050c41511e59ebcc3582d286c8d >>> 8.0K /data03/cassandra/album_media_feature/media_region_features_ >>> v4-29a0cd10150611e6bd3e3f41faa2612a >>> 228G /data03/cassandra/album_media_feature/media_region_features_ >>> v5-1b805470a3d711e68121757e9ac51b7b >>> >>> >>> >> >
