Re: Follow up on Jira Issue 39549

[Sean Owen]

No, programs can't read information out of other processes's memory - this
is true of all software. A cached DataFrame is tied to a Spark
application, but many things could be running inside that app. You may be
thinking of something like, exposing some kind of service on the driver
that responds to requests and returns results out of memory. Sure you could
do that. People usually use a service that uses Spark as a SQL engine for
things like that.

On Fri, Jun 24, 2022 at 1:17 PM Chenyang Zhang  wrote:

> Thanks for response. I want to figure out is there a way to share data
> without writing it to disk because of performance issues.
>
>
>
> *Chenyang Zhang*
> Software Engineering Intern, Platform
> Redwood City, California
> 
> © 2022 C3.ai. Confidential Information.
>
> On Jun 24, 2022, at 11:15 AM, Sean Owen  wrote:
>
> *EXTERNAL EMAIL:* This email originated outside of our organization. Do
> not click links or open attachments unless you recognize the sender and
> know the content is safe.
> Spark is decoupled from storage. You can write data to any storage you
> like. Anything that can read that data, can read that data - Spark or not,
> different session or not. Temp views are specific to a session and do not
> store data. I think this is trivial and no problem at all, or else I'm not
> clear what you're asking.
>
> On Fri, Jun 24, 2022 at 1:03 PM Chenyang Zhang 
> wrote:
>
>> Hi,
>>
>>
>>
>> Thanks so much for the response from
>> https://issues.apache.org/jira/browse/SPARK-39549
>> .
>> I am curious what do you mean by write down to a table and read it from a
>> different Spark Application. Do you mean a table in a database or the Spark
>> to_table() api? Could I read the table created in different Spark Session
>> through to_table()? Is there any api I could use to achieve my objective? I
>> noticed there are some apis, e.g. createGlobalTempView. Thanks so much for
>> your help.
>>
>>
>>
>> Bets regards,
>>
>> Chenyang
>>
>>
>> *Chenyang Zhang*
>> Software Engineering Intern, Platform
>> Redwood City, California
>> 
>> 
>> © 2022 C3.ai. Confidential Information.
>>
>>
>>
>>
>>
>>
>>
>>
>>
>

Re: Follow up on Jira Issue 39549

[Chenyang Zhang]

Thanks for response. I want to figure out is there a way to share data without 
writing it to disk because of performance issues.



Chenyang Zhang
Software Engineering Intern, Platform
Redwood City, California

[cid:EnterpriseAI_Banner_1200_e6f8b810-93f3-44f1-b795-bb502b7a52ae.png]
© 2022 C3.ai. Confidential Information.

On Jun 24, 2022, at 11:15 AM, Sean Owen 
mailto:sro...@gmail.com>> wrote:

EXTERNAL EMAIL: This email originated outside of our organization. Do not click 
links or open attachments unless you recognize the sender and know the content 
is safe.
Spark is decoupled from storage. You can write data to any storage you like. 
Anything that can read that data, can read that data - Spark or not, different 
session or not. Temp views are specific to a session and do not store data. I 
think this is trivial and no problem at all, or else I'm not clear what you're 
asking.

On Fri, Jun 24, 2022 at 1:03 PM Chenyang Zhang 
mailto:chenyang.zh...@c3.ai>> wrote:
Hi,

Thanks so much for the response from 
https://issues.apache.org/jira/browse/SPARK-39549.
 I am curious what do you mean by write down to a table and read it from a 
different Spark Application. Do you mean a table in a database or the Spark 
to_table() api? Could I read the table created in different Spark Session 
through to_table()? Is there any api I could use to achieve my objective? I 
noticed there are some apis, e.g. createGlobalTempView. Thanks so much for your 
help.

Bets regards,
Chenyang


Chenyang Zhang
Software Engineering Intern, Platform
Redwood City, California


© 2022 C3.ai. Confidential Information.

Re: Follow up on Jira Issue 39549

[Sean Owen]

Spark is decoupled from storage. You can write data to any storage you
like. Anything that can read that data, can read that data - Spark or not,
different session or not. Temp views are specific to a session and do not
store data. I think this is trivial and no problem at all, or else I'm not
clear what you're asking.

On Fri, Jun 24, 2022 at 1:03 PM Chenyang Zhang  wrote:

> Hi,
>
>
>
> Thanks so much for the response from
> https://issues.apache.org/jira/browse/SPARK-39549. I am curious what do
> you mean by write down to a table and read it from a different Spark
> Application. Do you mean a table in a database or the Spark to_table() api?
> Could I read the table created in different Spark Session through
> to_table()? Is there any api I could use to achieve my objective? I noticed
> there are some apis, e.g. createGlobalTempView. Thanks so much for your
> help.
>
>
>
> Bets regards,
>
> Chenyang
>
>
> *Chenyang Zhang*
> Software Engineering Intern, Platform
> Redwood City, California
> 
> © 2022 C3.ai. Confidential Information.
>
>

Follow up on Jira Issue 39549

[Chenyang Zhang]

Hi,

Thanks so much for the response from 
https://issues.apache.org/jira/browse/SPARK-39549. I am curious what do you 
mean by write down to a table and read it from a different Spark Application. 
Do you mean a table in a database or the Spark to_table() api? Could I read the 
table created in different Spark Session through to_table()? Is there any api I 
could use to achieve my objective? I noticed there are some apis, e.g. 
createGlobalTempView. Thanks so much for your help.

Bets regards,
Chenyang


Chenyang Zhang
Software Engineering Intern, Platform
Redwood City, California

[cid:EnterpriseAI_Banner_1200_e6f8b810-93f3-44f1-b795-bb502b7a52ae.png]
(c) 2022 C3.ai. Confidential Information.