Re: How to lookup by a key in an RDD
I read about the IndexedRDD. Is the IndexedRDD join with another RDD that is not an IndexedRDD efficient? On Mon, Nov 2, 2015 at 9:56 PM, Deenar Toraskarwrote: > Swetha > > Currently IndexedRDD is an external library and not part of Spark Core. > You can use it by adding a dependency and pull it in. There are plans to > move it to Spark core tracked in > https://issues.apache.org/jira/browse/SPARK-2365. See > https://spark-summit.org/2015/events/indexedrdd-efficient-fine-grained-updates-for-rdds/ > and https://github.com/amplab/spark-indexedrdd > > *Think Reactive Ltd* > deenar.toras...@thinkreactive.co.uk > 07714140812 > > > > On 2 November 2015 at 23:29, Ted Yu wrote: > >> Please take a look at SPARK-2365 >> >> On Mon, Nov 2, 2015 at 3:25 PM, swetha kasireddy < >> swethakasire...@gmail.com> wrote: >> >>> Hi, >>> >>> Is Indexed RDDs released yet? >>> >>> Thanks, >>> Swetha >>> >>> On Sun, Nov 1, 2015 at 1:21 AM, Gylfi wrote: >>> Hi. You may want to look into Indexed RDDs https://github.com/amplab/spark-indexedrdd Regards, Gylfi. -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/How-to-lookup-by-a-key-in-an-RDD-tp25243p25247.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org >>> >> >
Re: How to lookup by a key in an RDD
Swetha Currently IndexedRDD is an external library and not part of Spark Core. You can use it by adding a dependency and pull it in. There are plans to move it to Spark core tracked in https://issues.apache.org/jira/browse/SPARK-2365. See https://spark-summit.org/2015/events/indexedrdd-efficient-fine-grained-updates-for-rdds/ and https://github.com/amplab/spark-indexedrdd *Think Reactive Ltd* deenar.toras...@thinkreactive.co.uk 07714140812 On 2 November 2015 at 23:29, Ted Yuwrote: > Please take a look at SPARK-2365 > > On Mon, Nov 2, 2015 at 3:25 PM, swetha kasireddy < > swethakasire...@gmail.com> wrote: > >> Hi, >> >> Is Indexed RDDs released yet? >> >> Thanks, >> Swetha >> >> On Sun, Nov 1, 2015 at 1:21 AM, Gylfi wrote: >> >>> Hi. >>> >>> You may want to look into Indexed RDDs >>> https://github.com/amplab/spark-indexedrdd >>> >>> Regards, >>> Gylfi. >>> >>> >>> >>> >>> >>> >>> -- >>> View this message in context: >>> http://apache-spark-user-list.1001560.n3.nabble.com/How-to-lookup-by-a-key-in-an-RDD-tp25243p25247.html >>> Sent from the Apache Spark User List mailing list archive at Nabble.com. >>> >>> - >>> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org >>> For additional commands, e-mail: user-h...@spark.apache.org >>> >>> >> >
Re: How to lookup by a key in an RDD
Hi, Is Indexed RDDs released yet? Thanks, Swetha On Sun, Nov 1, 2015 at 1:21 AM, Gylfiwrote: > Hi. > > You may want to look into Indexed RDDs > https://github.com/amplab/spark-indexedrdd > > Regards, > Gylfi. > > > > > > > -- > View this message in context: > http://apache-spark-user-list.1001560.n3.nabble.com/How-to-lookup-by-a-key-in-an-RDD-tp25243p25247.html > Sent from the Apache Spark User List mailing list archive at Nabble.com. > > - > To unsubscribe, e-mail: user-unsubscr...@spark.apache.org > For additional commands, e-mail: user-h...@spark.apache.org > >
Re: How to lookup by a key in an RDD
Please take a look at SPARK-2365 On Mon, Nov 2, 2015 at 3:25 PM, swetha kasireddywrote: > Hi, > > Is Indexed RDDs released yet? > > Thanks, > Swetha > > On Sun, Nov 1, 2015 at 1:21 AM, Gylfi wrote: > >> Hi. >> >> You may want to look into Indexed RDDs >> https://github.com/amplab/spark-indexedrdd >> >> Regards, >> Gylfi. >> >> >> >> >> >> >> -- >> View this message in context: >> http://apache-spark-user-list.1001560.n3.nabble.com/How-to-lookup-by-a-key-in-an-RDD-tp25243p25247.html >> Sent from the Apache Spark User List mailing list archive at Nabble.com. >> >> - >> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org >> For additional commands, e-mail: user-h...@spark.apache.org >> >> >
Re: How to lookup by a key in an RDD
Hi. You may want to look into Indexed RDDs https://github.com/amplab/spark-indexedrdd Regards, Gylfi. -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/How-to-lookup-by-a-key-in-an-RDD-tp25243p25247.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
How to lookup by a key in an RDD
Hi, I have a requirement wherein I have to load data from hdfs, build an RDD and then lookup by key to do some updates to the value and then save it back to hdfs. How to lookup for a value using a key in an RDD? Thanks, Swetha -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/How-to-lookup-by-a-key-in-an-RDD-tp25243.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
Re: How to lookup by a key in an RDD
Hi, Looking here for the lookup function might help you: https://spark.apache.org/docs/latest/api/scala/index.html#org.apache.spark.rdd.PairRDDFunctions Natu On Sat, Oct 31, 2015 at 6:04 PM, swethawrote: > Hi, > > I have a requirement wherein I have to load data from hdfs, build an RDD > and > then lookup by key to do some updates to the value and then save it back to > hdfs. How to lookup for a value using a key in an RDD? > > > Thanks, > Swetha > > > > -- > View this message in context: > http://apache-spark-user-list.1001560.n3.nabble.com/How-to-lookup-by-a-key-in-an-RDD-tp25243.html > Sent from the Apache Spark User List mailing list archive at Nabble.com. > > - > To unsubscribe, e-mail: user-unsubscr...@spark.apache.org > For additional commands, e-mail: user-h...@spark.apache.org > >