Hi, I have mailed the list a few times about a specific application need, and keep coming back to the possibility of using Riak for our needs (Disco, Hadoop/Cassandra and other applications are also under consideration).
The requirement is that we need to be able to save large volumes of log data (specifically video/web access records) that are in ASCII format. We have clusters of client nodes generating this data, where each client will process in the order of 20,000 log records per second (probably about 5 client nodes per cluster). There is the need to do some pre-processing of that data for internal usage, but after that we need to save it in a system that is: redundant, supports multiple languages for access/queries, supports map-reduce and can scale. Obviously we can not save that amount of data as individual records, so the goal would be to save 1 seconds worth of records per node (that would be about 4MB of data per client per second). We would probably use LevelDB as the backend database. The pre-processing of the data would be done via a pre or post-commit hook. The question is would Riak or Luwak be the best choice for this need? I am erring to using Luwak, (if I recall correctly I can still use pre or post commit hooks and map-reduce type searches). Comments welcome Matt _______________________________________________ riak-users mailing list [email protected] http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
