Thanks! On Thu, Oct 23, 2014 at 10:56 AM, Akshat Aranya <aara...@gmail.com> wrote:
> Yes, that is a downside of Spark's design in general. The only way to > share data across consumers of data is by having a separate entity that > owns the Spark context. That's the idea behind Ooyala's job server. The > driver is still a single point of failure; if you lose the driver process, > you lose all information about the RDDs. > On Oct 22, 2014 6:33 PM, "Hajime Takase" <placeofnomemor...@gmail.com> > wrote: > >> Interesting.I see the interface of IndexedRDD, which seems to be like >> key/value store of the certain SparkContext. >> https://github.com/apache/spark/pull/1297 >> But the different SparkContext won't let their IndexedRDD to be used by >> other ( I want to use multiple "driver" in my system)? >> >> >> >> On Thu, Oct 23, 2014 at 1:01 AM, Akshat Aranya <aara...@gmail.com> wrote: >> >>> Spark, in general, is good for iterating through an entire dataset again >>> and again. All operations are expressed in terms of iteration through all >>> the records of at least one partition. You may want to look at IndexedRDD ( >>> https://issues.apache.org/jira/browse/SPARK-2365) that aims to improve >>> point queries. In general though, Spark is unlikely to outperform KV >>> stores because of the nature of scheduling a job for every operation. >>> >>> On Wed, Oct 22, 2014 at 7:51 AM, Hajime Takase < >>> placeofnomemor...@gmail.com> wrote: >>> >>>> Hi, >>>> Is it possible to use Spark as clustered key/value store ( say, like >>>> redis-cluster or hazelcast)?Will it out perform in write/read or other >>>> operation? >>>> My main urge is to use same RDD from several different SparkContext >>>> without saving to disk or using spark-job server,but I'm curious if someone >>>> has already tried using Spark like key/value store. >>>> >>>> Thanks, >>>> >>>> Hajime >>>> >>>> >>>> >>> >>> >>
