Hey Harold Those pull requests came after mine, see carbon#139 [1]. Nice catch in 698, in ours we took a different approach and loaded a custom finder within the database plugin [2] itself. Looking forward to comparing notes and helping where I can
-Jake [1]: https://github.com/graphite-project/carbon/pull/139 [2]: https://github.com/jfarrell/graphite-web/commit/af90af4d67a2b32d450d4559e06ae2b2fddcc4f1 On Tue, Oct 7, 2014 at 10:20 AM, Harald Kraemer < hkrae...@goodgamestudios.com> wrote: > Hi, > > am I looking at the right pull requests with graphite-project/carbon#210 > and #216? > > Quite interesting. Sadly, I don't think I could provide exactly that > python API with our existing storage input frontend. > > Just look out for the issues we fixed in > https://github.com/graphite-project/graphite-web/pull/698 :) > > - Harald > > 2014-10-07 14:35 GMT+02:00 Jake Farrell <jfarr...@apache.org>: > >> Hi Harald >> I have been working on a similar project which enables carbon to have a >> plugable backend storage system that leverages Apache Cassandra for >> storage. I opened pull requests in both carbon and graphite for the >> plugable backend portion and the Cassandra backend is still in the works. >> Your projects topic is something that I am very familiar with and all the >> mentioned related technologies. I would be happy to help as either a >> champion or a mentor for this project >> >> -Jake >> >> >> >> On Tue, Oct 7, 2014 at 4:59 AM, Harald Kraemer < >> hkrae...@goodgamestudios.com >> > wrote: >> >> > Hi, >> > >> > we have been allowed to open-source one of our company internal >> projects - >> > currently called Bifroest. Bifroest is a storage backend for >> graphite-web, >> > based on Apache Cassandra. I'm quite happy about this, and now I'm in >> the >> > process of finding the best options and means to do so. This mail isn't >> an >> > entire proposal yet, but I will try to stick at least to the major >> points >> > in a proposal. >> > >> > What does Bifroest do, and where does it come from. >> > >> > At GoodgameStudios, we used Munin for most of our monitoring, using a >> lot >> > of custom plugins for our servers and pushing 500 - 700 hosts around. >> > That's ambitious with munin and by now, the munin-master is not able to >> > take the stress anymore. >> > As such, we started to evaluate graphite, since graphite is the state of >> > the art larger scale monitoring solution. To start evaluating graphite, >> we >> > deployed graphite with a carbon backend on a virtual machine. Our senior >> > monitoring admin (which we didn't have back then) probably just had to >> > giggle a bit and doesn't know why - things didn't perform that well on a >> > virtual machine. It could handle the important data, but the system >> didn't >> > seem to scale that well. >> > An admin would have tossed hardware at this, SSD-Raids and all that, >> > naturally. But we are software engineers, not admins, thus we tossed >> > software at it (until we required hardware) :) >> > >> > Our intention was to have a graphite with data stored in a distributed >> > database. A distributed database would scale both in storage space and >> in >> > load the system can deal with. And it's all behind a well-defined >> > interface. That seemed like a nifty feature for a scalable monitoring >> > system. >> > Hence, we tried Cyanide, since Cyanide was just that. Tossed a lot of >> data >> > into Apache Cassandra, click on the metric tree and... well. Nothing >> > happened, since Cyanide figured that a "select *" across several 100k >> rows >> > is a grand idea. After that, we looked at InfluxDB, but at the time we >> > started developing this, InfluxDB didn't support data aggregation and >> > seemed to be in a very, very early stage of development. >> > >> > Thus, the first thought of bifroest was born: Why don't we take the good >> > parts of Cyanide, a solid distributed database, such as Apache >> Cassandra, >> > and the good parts of carbon and toss them in a big stew? >> > >> > That's what we did, and that's what we are currently deploying as our >> > productive monitoring system, graphite on bifroest as a frontend for >> apache >> > cassandra. >> > >> > Fun features of this system include: >> > - Existing graphite and most carbon apis: >> > -- Full support of the graphite rest API, since we are just a backend. >> > -- Support for the Plaintext Protocol of Carbon >> > -- Planned: An AMQP interface to handle globally distributed networks >> > - Neat things, which graphite could do as well: >> > -- A fast key cache >> > -- A fast value-cache, which is fed by the data collection to hit the >> > database as little as possible >> > - New things, Graphite+carbon+whisper cannot do: >> > -- On the fly adjustable retention levels. You don't have the space to >> > keep 6 weeks of 1m data? Just reduce it. Or increase it. Our system can >> do >> > that on the fly. >> > -- Currently in progress: On the fly addition of new retention levels. >> > Have an emergency and need data in greater resolution? Just add a >> retention >> > level with 1 datapoint / 5s, keep the full data history and tell your >> data >> > collection to collect more data and delete it later on again wiithout >> > losing data. >> > -- High fault tolerance. We are relying on cassandra for persistent >> > storage, and a properly deployed cassandra cluster with redundancy just >> > doesn't care. Add a new machine, tell everything to rebuild the cluster >> and >> > the frontend didn't even notice the outage. >> > >> > So, after this wall of text, there are two questions from me: >> > >> > a) is this project interesting enough for everyone? :) >> > b) Are there people who would volunteer to coach me and my team through >> the >> > proposal and the incubator? >> > >> > Regards, >> > Harald. >> > -- >> > >> > *Harald Krämer* >> > Server Developer (Profiling first) >> > *hkrae...@goodgamestudios.com <hkrae...@goodgamestudios.com>* >> > >> > Goodgame Studios >> > Theodorstr. 42-90, House 9 >> > 22761 Hamburg, Germany >> > Phone: +49 (0)40 219 880 -0 >> > *www.goodgamestudios.com <http://www.goodgamestudios.com>* >> > >> > Goodgame Studios is a branch of Altigi GmbH >> > Altigi GmbH, District court Hamburg, HRB 99869 >> > Board of directors: Dr. Kai Wawrzinek, Dr. Christian Wawrzinek, Fabian >> > Ritter >> > >> > > > > -- > > *Harald Krämer* > Server Developer (Profiling first) > *hkrae...@goodgamestudios.com <hkrae...@goodgamestudios.com>* > > Goodgame Studios > Theodorstr. 42-90, House 9 > 22761 Hamburg, Germany > Phone: +49 (0)40 219 880 -0 > *www.goodgamestudios.com <http://www.goodgamestudios.com>* > > Goodgame Studios is a branch of Altigi GmbH > Altigi GmbH, District court Hamburg, HRB 99869 > Board of directors: Dr. Kai Wawrzinek, Dr. Christian Wawrzinek, Fabian > Ritter >
