Re: How to lookup by a key in an RDD

[swetha kasireddy]

I read about the IndexedRDD. Is the IndexedRDD join with another RDD that
is not an IndexedRDD efficient?

On Mon, Nov 2, 2015 at 9:56 PM, Deenar Toraskar 
wrote:

> Swetha
>
> Currently IndexedRDD is an external library and not part of Spark Core.
> You can use it by adding a dependency and pull it in. There are plans to
> move it to Spark core tracked in
> https://issues.apache.org/jira/browse/SPARK-2365. See
> https://spark-summit.org/2015/events/indexedrdd-efficient-fine-grained-updates-for-rdds/
> and https://github.com/amplab/spark-indexedrdd
>
> *Think Reactive Ltd*
> deenar.toras...@thinkreactive.co.uk
> 07714140812
>
>
>
> On 2 November 2015 at 23:29, Ted Yu  wrote:
>
>> Please take a look at SPARK-2365
>>
>> On Mon, Nov 2, 2015 at 3:25 PM, swetha kasireddy <
>> swethakasire...@gmail.com> wrote:
>>
>>> Hi,
>>>
>>> Is Indexed RDDs released yet?
>>>
>>> Thanks,
>>> Swetha
>>>
>>> On Sun, Nov 1, 2015 at 1:21 AM, Gylfi  wrote:
>>>
 Hi.

 You may want to look into Indexed RDDs
 https://github.com/amplab/spark-indexedrdd

 Regards,
 Gylfi.






 --
 View this message in context:
 http://apache-spark-user-list.1001560.n3.nabble.com/How-to-lookup-by-a-key-in-an-RDD-tp25243p25247.html
 Sent from the Apache Spark User List mailing list archive at Nabble.com.

 -
 To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
 For additional commands, e-mail: user-h...@spark.apache.org


>>>
>>
>

Re: How to lookup by a key in an RDD

[Deenar Toraskar]

Swetha

Currently IndexedRDD is an external library and not part of Spark Core. You
can use it by adding a dependency and pull it in. There are plans to move
it to Spark core tracked in https://issues.apache.org/jira/browse/SPARK-2365.
See
https://spark-summit.org/2015/events/indexedrdd-efficient-fine-grained-updates-for-rdds/
and https://github.com/amplab/spark-indexedrdd

*Think Reactive Ltd*
deenar.toras...@thinkreactive.co.uk
07714140812



On 2 November 2015 at 23:29, Ted Yu  wrote:

> Please take a look at SPARK-2365
>
> On Mon, Nov 2, 2015 at 3:25 PM, swetha kasireddy <
> swethakasire...@gmail.com> wrote:
>
>> Hi,
>>
>> Is Indexed RDDs released yet?
>>
>> Thanks,
>> Swetha
>>
>> On Sun, Nov 1, 2015 at 1:21 AM, Gylfi  wrote:
>>
>>> Hi.
>>>
>>> You may want to look into Indexed RDDs
>>> https://github.com/amplab/spark-indexedrdd
>>>
>>> Regards,
>>> Gylfi.
>>>
>>>
>>>
>>>
>>>
>>>
>>> --
>>> View this message in context:
>>> http://apache-spark-user-list.1001560.n3.nabble.com/How-to-lookup-by-a-key-in-an-RDD-tp25243p25247.html
>>> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>>>
>>> -
>>> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
>>> For additional commands, e-mail: user-h...@spark.apache.org
>>>
>>>
>>
>

Re: How to lookup by a key in an RDD

[swetha kasireddy]

Hi,

Is Indexed RDDs released yet?

Thanks,
Swetha

On Sun, Nov 1, 2015 at 1:21 AM, Gylfi  wrote:

> Hi.
>
> You may want to look into Indexed RDDs
> https://github.com/amplab/spark-indexedrdd
>
> Regards,
> Gylfi.
>
>
>
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/How-to-lookup-by-a-key-in-an-RDD-tp25243p25247.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> -
> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
> For additional commands, e-mail: user-h...@spark.apache.org
>
>

Re: How to lookup by a key in an RDD

[Ted Yu]

Please take a look at SPARK-2365

On Mon, Nov 2, 2015 at 3:25 PM, swetha kasireddy 
wrote:

> Hi,
>
> Is Indexed RDDs released yet?
>
> Thanks,
> Swetha
>
> On Sun, Nov 1, 2015 at 1:21 AM, Gylfi  wrote:
>
>> Hi.
>>
>> You may want to look into Indexed RDDs
>> https://github.com/amplab/spark-indexedrdd
>>
>> Regards,
>> Gylfi.
>>
>>
>>
>>
>>
>>
>> --
>> View this message in context:
>> http://apache-spark-user-list.1001560.n3.nabble.com/How-to-lookup-by-a-key-in-an-RDD-tp25243p25247.html
>> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>>
>> -
>> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
>> For additional commands, e-mail: user-h...@spark.apache.org
>>
>>
>

Re: How to lookup by a key in an RDD

[Gylfi]

Hi. 

You may want to look into Indexed RDDs 
https://github.com/amplab/spark-indexedrdd

Regards, 
Gylfi. 






--
View this message in context: 
http://apache-spark-user-list.1001560.n3.nabble.com/How-to-lookup-by-a-key-in-an-RDD-tp25243p25247.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

-
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional commands, e-mail: user-h...@spark.apache.org

How to lookup by a key in an RDD

[swetha]

Hi,

I have a requirement wherein I have to load data from hdfs, build an RDD and
then lookup by key to do some updates to the value and then save it back to
hdfs. How to lookup for a value using a key in an RDD?


Thanks,
Swetha



--
View this message in context: 
http://apache-spark-user-list.1001560.n3.nabble.com/How-to-lookup-by-a-key-in-an-RDD-tp25243.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

-
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional commands, e-mail: user-h...@spark.apache.org

Re: How to lookup by a key in an RDD

[Natu Lauchande]

Hi,

Looking here for the lookup function might help you:

https://spark.apache.org/docs/latest/api/scala/index.html#org.apache.spark.rdd.PairRDDFunctions

Natu

On Sat, Oct 31, 2015 at 6:04 PM, swetha  wrote:

> Hi,
>
> I have a requirement wherein I have to load data from hdfs, build an RDD
> and
> then lookup by key to do some updates to the value and then save it back to
> hdfs. How to lookup for a value using a key in an RDD?
>
>
> Thanks,
> Swetha
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/How-to-lookup-by-a-key-in-an-RDD-tp25243.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> -
> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
> For additional commands, e-mail: user-h...@spark.apache.org
>
>