New question #217341 on Graphite:
https://answers.launchpad.net/graphite/+question/217341
As part of a monitoring and metrics application we are using Carbon to store
the various statistics generated by Diamond collectors and the monitored
application. One of the decisions was that the entire application would run as
a single process with the various daemons running as threads. This would
hopefully help to make it perform the same on both Unix and Windows systems.
(We have made some small modifications to allow carbon to operate on windows,
but these changes don't affect the behaviour below.)
This all worked out fine in development and testing, with both carbon-cache and
carbon-aggregator seemingly running fine and accepting data from diamond
collectors, and graphite-web displaying the data appropriately. Unfortunately
after running like this for more than a few minutes CPU usage quickly spikes to
100% (or more with multiple CPUs). Memory usage does not grow and everything
seems to perform fine and work appropriately except for the high CPU usage.
This does not happen when no data (neither carbon instrumentation nor diamond)
is sent to carbon.
Is there any way to launch both daemons in a single python process? I know the
reactor object is global and that there can only ever be one, but are there any
other global conflicts between the two daemons that would cause this behaviour?
Are they perhaps sharing the same internal buffer and filling it exponentially
via an aggregator-cache-aggregator-cache loop?
The code I am using to launch both daemons is as follows:
-------------------------------------------
. . . (relevant import and setup here)
daemons = ['carbon-cache', 'carbon-aggregator'] # settings.CARBON_DAEMONS
print("Carbon Daemons = %s" % daemons)work
topsvc = service.MultiService()
for program in daemons:
twistd_options = ["--no_save", "--nodaemon", program]
if settings.CARBON_CONFIG != None:
twistd_options.append('--config='+settings.CARBON_CONFIG)
config = ServerOptions()
config.parseOptions(twistd_options)
config['originalname'] = program
plg = config.loadedPlugins[config.subCommand]
ser = plg.makeService(config.subOptions)
ser.setServiceParent(topsvc)
topsvc.startService()
from twisted.internet import reactor
reactor.run()
-------------------------------------------
This also behaves fine if I launch the daemons seperately using the same code
and simply changing the 'daemons' variable to ['carbon-cache'] and
['carbon-aggregator'] respectively.
We are using the 0.9.x branch.
--
You received this question notification because you are a member of
graphite-dev, which is an answer contact for Graphite.
_______________________________________________
Mailing list: https://launchpad.net/~graphite-dev
Post to : [email protected]
Unsubscribe : https://launchpad.net/~graphite-dev
More help : https://help.launchpad.net/ListHelp
