Eric, Any additional conclusions to recursively querying your dependency graph? I just found SA, and have been wondering how feasible it would be to manage and query a graph structure somewhat similar to the one you've described. My problem is that I have a number of classes that build expensive data attributes, each of which is one step to producing many possible different solutions. The classes are organized so that all attributes built by that class depend on the same set of parameters, so that if I know the set of parameter values, I know that all of the data attributes built by that class are congruent. As each class instance performs one or more transformations, it needs to be able to retrieve the correct data structures, i.e. those that have been built with the same parameter values.
Currently I keep the data attributes in python shelves, which I plan on continue doing, as they are too large to stick in sql (numpy arrays of up to a Gb or two), and manage a half-baked sql db to track the class instances, their parameter values, and shelf file names. I want to use SA to manage the shelf file names and directories by mirroring the dependency structure of the classes. The problem is that there are a bunch of parameters, and the more dependent the classes become (deep children in a graph), the more parameter values are needed to exactly define the data attributes. In addition, I don't know the complete dependency structure until run- time, as some classes can depend on an arbitrary number of other classes (these other classes have different methods to produce the needed data). If SA could manage a dependency graph as I build the data attributes, I could then query it to check whether a needed data attribute with the right specifications already existed, and in which shelf in which directory (one shelf per class instance holding the keyed data attributes). Each class instance would not need to store the parameter values of all the class instances it depended on (there may be scores of such parameter values), but only a small subset directly used by its methods, as a query could traverse the graph and check which parameter values had been used at each step. I thought if I made a simple class structure that mirrored my working classes that all inherited from a class that would serve as nodes in the graph, that I might be able to reasonably query to see if a data structure had already been built with the correct parameter values. I guess this means that this class would be self-referential, and that I would be performing some recursive traversal for the query. Currently my graph would be less than ten nodes deep, occasionally wide (a class might require a thousand instances of another class), and speed is not much of an issue, as manipulating these big data attributes takes the vast majority of time/resources. Data integrity, however, does matter, as if I lose the parameter values used to build an object, it could become useless. I am being conservative by creating new classes to mirror the dependency structure, as I don't want to take the deep plunge yet (I have RAM issues and need to let the working classes die easily (maybe not a problem), plus I've overloaded __getattr__ to fetch data attributes from the shelves, and don't want to mess with this until I understand SA a bit more). Any lessons? I found your posts useful, though apparently from the crazed fringe... :) Rich --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "sqlalchemy" group. To post to this group, send email to sqlalchemy@googlegroups.com To unsubscribe from this group, send email to sqlalchemy+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/sqlalchemy?hl=en -~----------~----~----~----~------~----~------~--~---
