The problem in this will be difference in storage and structure of data in various systems. I think saveAsNewAPIHadoopFile in PairRDDFunctions provides decent abstraction to write to any Hadoop supported output. A way to build a generic API to persist to different storage will be to create serializer for the output format and use the Hadoop Output API to write out to the storage.
*Founder & CEO, **Tuplejump, Inc.* ____________________________ www.tuplejump.com *The Data Engineering Platform* On Sun, Feb 23, 2014 at 6:43 PM, Azuryy Yu <azury...@gmail.com> wrote: > Hi Rohit, > > Cool! > currently RDD can only be persisted in memory or dist level(CACHE LEVEL), > but > I always want to add a general methods for RDD, which can persisted on any > storage, such as HDFS level, Cassandra, HBase etc. > > any thoughts here? > > > > On Sun, Feb 23, 2014 at 6:54 PM, Rohit Rai <ro...@tuplejump.com> wrote: > >> Checkout this link - >> http://tuplejump.github.io/calliope/transformers.html >> >> Feel free to mail me directly if you have any specific requirement, will >> help you write the transformer. >> >> *Founder & CEO, **Tuplejump, Inc.* >> ____________________________ >> www.tuplejump.com >> *The Data Engineering Platform* >> >> >> On Fri, Feb 21, 2014 at 11:35 PM, Liam Stewart <liam.stew...@gmail.com>wrote: >> >>> We're looking at putting spark and shark on top of some tables stored in >>> cassandra 2 and defined using cql3. Most of our columns are simple types >>> (ints, strings, bools, timestamps, ...) but some of our data falls >>> naturally into sets and maps - using those would be ideal if we can, but if >>> not we can flatten our schema. >>> >>> I'm wondering if anybody has had experience with cql3 containers and >>> spark - it seems like with calliope, we would have to provide an >>> unmarshaller that knows how to unpack containers to scala sets/maps? >>> >>> For shark, it looks like we could use >>> cash<https://github.com/tuplejump/cash/tree/master/cassandra-handler>as the >>> storage handler. Does cash support unpacking maps/sets? >>> >>> -- >>> Liam Stewart :: liam.stew...@gmail.com >>> >> >> >
