javascript:_e(%7B%7D,'cvml','user@spark.apache.org');
*Subject:* Re: How to share large resources like dictionaries while
processing data with Spark ?
Thanks so much, Yiannis, Olivier, Huang!
On Thu, Jun 4, 2015 at 6:44 PM, Yiannis Gkoufas johngou...@gmail.com
javascript:_e(%7B%7D,'cvml
[mailto:dgoldenberg...@gmail.com]
Sent: Friday, June 5, 2015 12:12 AM
To: Yiannis Gkoufas
Cc: Olivier Girardot; user@spark.apache.org
Subject: Re: How to share large resources like dictionaries while processing
data with Spark ?
Thanks so much, Yiannis, Olivier, Huang!
On Thu, Jun 4, 2015 at 6
@spark.apache.org'
Subject: RE: How to share large resources like dictionaries while processing
data with Spark ?
It is called Indexed RDD https://github.com/amplab/spark-indexedrdd
From: Dmitry Goldenberg [mailto:dgoldenberg...@gmail.com]
Sent: Friday, June 5, 2015 3:15 PM
To: Evo
: How to share large resources like dictionaries while
processing data with Spark ?
Thanks so much, Yiannis, Olivier, Huang!
On Thu, Jun 4, 2015 at 6:44 PM, Yiannis Gkoufas johngou...@gmail.com
javascript:_e(%7B%7D,'cvml','johngou...@gmail.com'); wrote:
Hi there,
I would recommend
Goldenberg [mailto:dgoldenberg...@gmail.com]
Sent: Friday, June 5, 2015 12:12 AM
To: Yiannis Gkoufas
Cc: Olivier Girardot; user@spark.apache.org
Subject: Re: How to share large resources like dictionaries while processing
data with Spark ?
Thanks so much, Yiannis, Olivier, Huang!
On Thu, Jun
:* Friday, June 5, 2015 12:12 AM
*To:* Yiannis Gkoufas
*Cc:* Olivier Girardot; user@spark.apache.org
*Subject:* Re: How to share large resources like dictionaries while
processing data with Spark ?
Thanks so much, Yiannis, Olivier, Huang!
On Thu, Jun 4, 2015 at 6:44 PM, Yiannis Gkoufas
: How to share large resources like dictionaries while processing
data with Spark ?
Would tachyon be appropriate here?
On Friday, June 5, 2015, Evo Eftimov evo.efti...@isecc.com wrote:
Oops, @Yiannis, sorry to be a party pooper but the Job Server is for Spark
Batch Jobs (besides anyone can put
across the RDD's?
I'm looking for a way to load such a resource once into the cluster memory
and have it be available throughout the lifecycle of a consumer...
Thanks.
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/How-to-share-large-resources-like
large resources like dictionaries while processing data
with Spark ?
We have some pipelines defined where sometimes we need to load potentially
large resources such as dictionaries.
What would be the best strategy for sharing such resources among the
transformations/actions within a consumer? Can
to load such a resource once into the cluster memory
and have it be available throughout the lifecycle of a consumer...
Thanks.
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/How-to-share-large-resources-like-dictionaries-while-processing-data-with-Spark
://apache-spark-user-list.1001560.n3.nabble.com/How-to-share-large-resources-like-dictionaries-while-processing-data-with-Spark-tp23162.html
Sent from the Apache Spark User List mailing list archive at Nabble.com
to load such a resource once into the cluster memory
and have it be available throughout the lifecycle of a consumer...
Thanks.
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/How-to-share-large-resources-like-dictionaries-while-processing-data-with-Spark
12 matches
Mail list logo
