I think you answered your own question, sort of.
When you expand a cluster, it copies the appropriate rows to the new
node(s) but doesn't automatically remove them from the old nodes. When you
ran cleanup on datacenter1, it cleared out those old extra copies. I would
suggest running a repair first for safety on datacenter2, then a "nodetool
cleanup" on those hosts.
Also run "nodetool snapshot" to make sure you don't have any old snapshots
sitting around taking up space.
On Wed, Nov 28, 2018 at 5:29 AM Eunsu Kim <eunsu.bil...@gmail.com> wrote:
> (I am sending the previous mail again because it seems that it has not
> been sent properly.)
>
> HI experts,
>
> I am running 2 datacenters each containing five nodes. (total 10 nodes,
> all 3.11.3)
>
> My data is stored one at each data center. (REPLICATION = { 'class' :
> 'org.apache.cassandra.locator.NetworkTopologyStrategy', 'datacenter1': '1'
> , 'datacenter2': '1’ })
>
> Most of my data have a short TTL(14days). The gc_grace_seconds value for
> all tables is also 600sec.
>
> I expect the two data centers to use the same size but datacenter2 is
> using more size. It seems that the datas of datacenter2 is rarely
> deleted. While the disk usage for datacenter1 remains constant, the disk
> usage for datacenter2 continues to grow.
>
> ——————
> Datacenter: datacenter1
> =======================
> Status=Up/Down
> |/ State=Normal/Leaving/Joining/Moving
> -- Address Load Tokens Owns (effective) Host ID
> Rack
> UN 10.61.58.228 925.48 GiB 256 21.5%
> 60d1bac8-b4d6-4e02-a05f-badee0bb36f5 rack1
> UN 10.61.58.167 840 GiB 256 20.0%
> a04fc77a-907f-490c-971c-4e1f964c7b14 rack1
> UN 10.61.75.86 1.13 TiB 256 19.3%
> 618c101b-036d-42e7-bf9f-2bcbd429cbd1 rack1
> UN 10.61.59.22 844.19 GiB 256 20.0%
> d8a4a165-13f0-4f4a-9278-4024730b8116 rack1
> UN 10.61.59.82 737.88 GiB 256 19.2%
> 054a4eb5-6d1c-46fa-b550-34da610da4e0 rack1
> Datacenter: datacenter2
> =======================
> Status=Up/Down
> |/ State=Normal/Leaving/Joining/Moving
> -- Address Load Tokens Owns (effective) Host ID
> Rack
> UN 10.42.6.120 1.11 TiB 256 18.6%
> 69f15be0-e5a1-474e-87cf-b063e6854402 rack1
> UN 10.42.5.207 1.17 TiB 256 20.0%
> f78bdce5-cb01-47e0-90b9-fcc31568e49e rack1
> UN 10.42.6.47 1.01 TiB 256 20.1%
> 3ff93b47-2c15-4e1a-a4ea-2596f26b4281 rack1
> UN 10.42.6.48 1007.67 GiB 256 20.4%
> 8cbbe76d-6496-403a-8b09-fe6812c9dea2 rack1
> UN 10.42.5.208 1.29 TiB 256 20.9%
> 4aa96c6a-6083-417f-a58a-ec847bcbfc7e rack1
> ------------------
>
> A few days ago, one node of datacenter1 broke down and replaced it, and I
> worked on rebuild, repair, and cleanup.
>
>
> What else can I do?
>
> Thank you in advance.
>
