bq. that is, key-value stores Please consider HBase for this purpose :-)
On Tue, Jul 14, 2015 at 5:55 PM, Tathagata Das <t...@databricks.com> wrote: > I do not recommend using IndexRDD for state management in Spark Streaming. > What it does not solve out-of-the-box is checkpointing of indexRDDs, which > important because long running streaming jobs can lead to infinite chain of > RDDs. Spark Streaming solves it for the updateStateByKey operation which > you can use, which gives state management capabilities. Though for most > flexible arbitrary look up of stuff, its better to use a dedicated system > that is designed and optimized for long term storage of data, that is, > key-value stores, databases, etc. > > On Tue, Jul 14, 2015 at 5:44 PM, Ted Yu <yuzhih...@gmail.com> wrote: > >> Please take a look at SPARK-2365 which is in progress. >> >> On Tue, Jul 14, 2015 at 5:18 PM, swetha <swethakasire...@gmail.com> >> wrote: >> >>> Hi, >>> >>> Is IndexedRDD available in Spark 1.4.0? We would like to use this in >>> Spark >>> Streaming to do lookups/updates/deletes in RDDs using keys by storing >>> them >>> as key/value pairs. >>> >>> Thanks, >>> Swetha >>> >>> >>> >>> -- >>> View this message in context: >>> http://apache-spark-user-list.1001560.n3.nabble.com/Is-IndexedRDD-available-in-Spark-1-4-0-tp23841.html >>> Sent from the Apache Spark User List mailing list archive at Nabble.com. >>> >>> --------------------------------------------------------------------- >>> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org >>> For additional commands, e-mail: user-h...@spark.apache.org >>> >>> >> >
