Question #226895 on Graphite changed: https://answers.launchpad.net/graphite/+question/226895
Slawomir Skowron posted a new comment: I'am using graphite-carbon and whisper from 0.9.12. All hosts, and carbons have same configuration on all 3 machines in AWS. MAX_CACHE_SIZE = inf MAX_UPDATES_PER_SECOND = 5000 MAX_CREATES_PER_MINUTE = 1000 io-waits are from 10-12% on this SSD drives on all machines. But with MAX_CACHE_SIZE = 10000000 (or other big, or low number) all is droping down, and carbon-caches get 100% cpu and now metric not showing in web-app That's why it is cache_size is inf With this setup numbers in carbon-cache are: update_operations ~ 50k commited_points ~ 200k received_points ~ 200k avg update time = 0.002 sec Strange is that some carbon caches making 25-30k update operations and some of then only 10k. This carbon who's making 30k updates, flushing into drives more offten and queues/cache size drops to 0 for a while and than going up. This carbon have much less reads, and little more writes. All host makes in summary 10k iops (5k per carbon-cache) When i look @ graphs, then some metrics not catching time with late about 1-2 minutes (4 x 30sec updates) For carbon daemons minimal interval is 10 seconds, and for diamond 30 sec. Now whatever change i made in my carbon.conf for carbon-cache it's only worst. Other strange thing is that, when i run first time whole graphite, then when creates happen all carbon-caches working perfect, but when creates and data was filled on drives and updates catch received number of metric all goes down to 50k updates from 200k updates and now toping low values and it's not good :( -- You received this question notification because you are a member of graphite-dev, which is an answer contact for Graphite. _______________________________________________ Mailing list: https://launchpad.net/~graphite-dev Post to : [email protected] Unsubscribe : https://launchpad.net/~graphite-dev More help : https://help.launchpad.net/ListHelp
