Re: About Spark Multiple Shared Context with Spark 2.0

[Calvin Jia]

Hi,

Alluxio will allow you to share or cache data in-memory between different
Spark contexts by storing RDDs or Dataframes as a file in the Alluxio
system. The files can then be accessed by any Spark job like a file in any
other distributed storage system.

These two blogs do a good job of summarizing the end-to-end workflow of
using Alluxio to share RDDs
 or Dataframes
 between
Spark jobs.

Hope this helps,
Calvin

On Tue, Dec 13, 2016 at 3:42 AM, Chetan Khatri 
wrote:

> Hello Guys,
>
> What would be approach to accomplish Spark Multiple Shared Context without
> Alluxio and with with Alluxio , and what would be best practice to achieve
> parallelism and concurrency for spark jobs.
>
> Thanks.
>
> --
> Yours Aye,
> Chetan Khatri.
> M.+91 7 80574 <+91%207%2080574>
> Data Science Researcher
> INDIA
>
> ​​Statement of Confidentiality
> 
> The contents of this e-mail message and any attachments are confidential
> and are intended solely for addressee. The information may also be legally
> privileged. This transmission is sent in trust, for the sole purpose of
> delivery to the intended recipient. If you have received this transmission
> in error, any use, reproduction or dissemination of this transmission is
> strictly prohibited. If you are not the intended recipient, please
> immediately notify the sender by reply e-mail or phone and delete this
> message and its attachments, if any.​​
>

About Spark Multiple Shared Context with Spark 2.0

[Chetan Khatri]

Hello Guys,

What would be approach to accomplish Spark Multiple Shared Context without
Alluxio and with with Alluxio , and what would be best practice to achieve
parallelism and concurrency for spark jobs.

Thanks.

-- 
Yours Aye,
Chetan Khatri.
M.+91 7 80574
Data Science Researcher
INDIA

​​Statement of Confidentiality

The contents of this e-mail message and any attachments are confidential
and are intended solely for addressee. The information may also be legally
privileged. This transmission is sent in trust, for the sole purpose of
delivery to the intended recipient. If you have received this transmission
in error, any use, reproduction or dissemination of this transmission is
strictly prohibited. If you are not the intended recipient, please
immediately notify the sender by reply e-mail or phone and delete this
message and its attachments, if any.​​