No, programs can't read information out of other processes's memory - this is true of all software. A cached DataFrame is tied to a Spark application, but many things could be running inside that app. You may be thinking of something like, exposing some kind of service on the driver that responds to requests and returns results out of memory. Sure you could do that. People usually use a service that uses Spark as a SQL engine for things like that.
On Fri, Jun 24, 2022 at 1:17 PM Chenyang Zhang <chenyang.zh...@c3.ai> wrote: > Thanks for response. I want to figure out is there a way to share data > without writing it to disk because of performance issues. > > > > *Chenyang Zhang* > Software Engineering Intern, Platform > Redwood City, California > <https://c3.ai/?utm_source=signature&utm_campaign=enterpriseai> > © 2022 C3.ai. Confidential Information. > > On Jun 24, 2022, at 11:15 AM, Sean Owen <sro...@gmail.com> wrote: > > *EXTERNAL EMAIL:* This email originated outside of our organization. Do > not click links or open attachments unless you recognize the sender and > know the content is safe. > Spark is decoupled from storage. You can write data to any storage you > like. Anything that can read that data, can read that data - Spark or not, > different session or not. Temp views are specific to a session and do not > store data. I think this is trivial and no problem at all, or else I'm not > clear what you're asking. > > On Fri, Jun 24, 2022 at 1:03 PM Chenyang Zhang <chenyang.zh...@c3.ai> > wrote: > >> Hi, >> >> >> >> Thanks so much for the response from >> https://issues.apache.org/jira/browse/SPARK-39549 >> <https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FSPARK-39549&data=05%7C01%7CChenyang.Zhang%40c3.ai%7C16869542c92f4af9188708da560d9886%7C53ad779a93e7485cba20ac8290d7252b%7C1%7C0%7C637916913636880755%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=sSW7Pi2vB9NTks0rCvRIZdO4ubJELlprARyXBO60upc%3D&reserved=0>. >> I am curious what do you mean by write down to a table and read it from a >> different Spark Application. Do you mean a table in a database or the Spark >> to_table() api? Could I read the table created in different Spark Session >> through to_table()? Is there any api I could use to achieve my objective? I >> noticed there are some apis, e.g. createGlobalTempView. Thanks so much for >> your help. >> >> >> >> Bets regards, >> >> Chenyang >> >> >> *Chenyang Zhang* >> Software Engineering Intern, Platform >> Redwood City, California >> <EnterpriseAI_Banner_1200_e6f8b810-93f3-44f1-b795-bb502b7a52ae.png> >> <https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fc3.ai%2F%3Futm_source%3Dsignature%26utm_campaign%3Denterpriseai&data=05%7C01%7CChenyang.Zhang%40c3.ai%7C16869542c92f4af9188708da560d9886%7C53ad779a93e7485cba20ac8290d7252b%7C1%7C0%7C637916913636880755%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=tACD7UJOVFHeyQhjH81dx2nJ5mo%2BdgaT%2F09YoqhCXyE%3D&reserved=0> >> © 2022 C3.ai. Confidential Information. >> >> >> >> >> >> >> >> >> >
