Yeah, you can create your own finder and matching reader and just add them to the list of finders in storage.py to start.
The Finder should expose the method find_nodes(query). This method should take a single parameter, a storage.FindQuery object and yield node.BranchNode objects and node.LeafNode objects. The creation of the LeafNode objects are where your Reader will be instantiated and should be whatever makes sense for your storage - for Whisper this is the files themselves. For RRD files, it's a single datasource within the file. If appropriate, it should take into account the requested interval. Multiple yields of LeafNodes will be merged based on the time intervals they contain (get_intervals() on the reader). Your Reader should expose get_intervals() and fetch(startTime, endTime). get_intervals should return an intervals.IntervalSet composed of the Intervals (just start and end timestamps) contained in the LeafNodes you'll be reading (whatever those may actually be). fetch will return a tuple like this: ( (startTime, endTime, step), [<list of data values>] ) - details about the data (actual timestamp of the data, not necessarily the requested time (due to retentions, rounding, etc) and the data itself. Adding the settings.py config seems reasonable. I'd try to add it to settings.py with logic to configure it as it already is from local_settings.py. Those who need the advanced functionality of adding their own readers can then add their own later on: STORAGE_FINDERS.append(...). Keep us updated on how it works out michael On Fri, Jun 15, 2012 at 5:44 PM, Jon Dugan <[email protected]> wrote: > Hello, > > I discussed this with Chris quite some time ago and he suggested that > I wait until the Ceres patch landed. Since it has landed I'd like to > reopen the question. > > What is the best way to link in a storage back end other than the > standard ones: Whisper, RRD and Ceres? > > It looks like there is a nod to this in webapp/graphite/storage.py > starting at line 149: > > 149 # Exposed Storage API > 150 finders = [ > 151 CeresFinder(settings.CERES_DIR), > 152 StandardFinder(settings.STANDARD_DIRS), > 153 ] > 154 STORE = Store(finders, hosts=settings.CLUSTER_SERVERS) > > How do I plugin my own finder? This seems like a place holder for a > storage API. Obviously I can modify the code but then I have to > track my own patches to Graphite. Which I have done for some time now > and it's a bit tiring. > > My suggestion would be to enumerate the finders in the settings file. > Something like this: > > STORAGE_FINDERS = [('graphite.finders.CeresFinder', CERES_DIR), > ('graphite.finders.StandardFinder', STANDARD_DIRS)] > > And then creating an instance of each finder called with the second > item of the tuple as an argument in storage.py. And passing that list > of finders to the Store. > > (IIRC there were some circular import issues if you just try to create > a Store instance in settings.py so this sidesteps that.) > > I'd be happy to create a patch that does something like this, but I > figured I'd get some input before I do it. > > Does this seem like a reasonable approach? Does someone else have > some other way of doing this already? > > Thanks, > > Jon > > _______________________________________________ > Mailing list: https://launchpad.net/~graphite-dev > Post to : [email protected] > Unsubscribe : https://launchpad.net/~graphite-dev > More help : https://help.launchpad.net/ListHelp _______________________________________________ Mailing list: https://launchpad.net/~graphite-dev Post to : [email protected] Unsubscribe : https://launchpad.net/~graphite-dev More help : https://help.launchpad.net/ListHelp
