We devised our own simple Akubra implementation for exactly this kind of problem:
http://uvalib.github.io/twostore/ --- A. Soroka The University of Virginia Library On Apr 18, 2013, at 11:42 AM, Gary Phillips wrote: > Hello, > > I've spent some time looking at akubra and the HashPathIdMapper to get a feel > for how we would distribute our datastreamStore over multiple file systems. > The default configuration (##) creates 256 potential directories. Changing > that to # (or something like #/##) would give us 16 top level > sub-directories, which we could work with. However, there is another issue, > in that I don't see how we can easily predict how each of those directories > would grow, if I am understanding how the files are distributed across the > directories. > > I assume one solution might involve sym linking the top level directories > over to a few directories that each correspond to a mount point. > > Have other people tackled this particular problem (and what solutions did you > come up with) or found another way around distributing large amounts of > Fedora data over multiple mount points? Thanks in advance. > > > ------------------------------------------------------------------------------ > Precog is a next-generation analytics platform capable of advanced > analytics on semi-structured data. The platform includes APIs for building > apps and a phenomenal toolset for data science. Developers can use > our toolset for easy data analysis & visualization. Get a free account! > http://www2.precog.com/precogplatform/slashdotnewsletter_______________________________________________ > Fedora-commons-users mailing list > [email protected] > https://lists.sourceforge.net/lists/listinfo/fedora-commons-users ------------------------------------------------------------------------------ Precog is a next-generation analytics platform capable of advanced analytics on semi-structured data. The platform includes APIs for building apps and a phenomenal toolset for data science. Developers can use our toolset for easy data analysis & visualization. Get a free account! http://www2.precog.com/precogplatform/slashdotnewsletter _______________________________________________ Fedora-commons-users mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/fedora-commons-users
