Spark memory is the sum of execution memory and storage memory. unpersist only removes the storage memory. Execution memory is there which is what is Spark all about.
HTH Dr Mich Talebzadeh LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>* http://talebzadehmich.wordpress.com *Disclaimer:* Use it at your own risk. Any and all responsibility for any loss, damage or destruction of data or any other property which may arise from relying on this email's technical content is explicitly disclaimed. The author will in no case be liable for any monetary damages arising from such loss, damage or destruction. On 31 August 2016 at 17:18, Rajani, Arpan <arpan.raj...@worldpay.com> wrote: > *Removing Data* > > > > Spark automatically monitors cache usage on each node and drops out old > data partitions in a least-recently-used (LRU) fashion. > > If you would like to manually remove an RDD instead of waiting for it to > fall out of the cache, use the *RDD.unpersist()* method. (Copied from > Documentation > <http://spark.apache.org/docs/latest/programming-guide.html#removing-data> > ) > > > > *From:* Cesar [mailto:ces...@gmail.com] > *Sent:* 31 August 2016 16:57 > *To:* user > *Subject:* releasing memory without stopping the spark context ? > > > > > > Is there a way to release all persisted RDD's/DataFrame's in Spark without > stopping the SparkContext ? > > > > > > Thanks a lot > > -- > > Cesar Flores > > > This e-mail and any attachments are confidential, intended only for the > addressee and may be privileged. If you have received this e-mail in error, > please notify the sender immediately and delete it. Any content that does not > relate to the business of Worldpay is personal to the sender and not > authorised or endorsed by Worldpay. Worldpay does not accept responsibility > for viruses or any loss or damage arising from transmission or access. > > Worldpay (UK) Limited (Company No: 07316500/ Financial Conduct Authority No: > 530923), Worldpay Limited (Company No:03424752 / Financial Conduct Authority > No: 504504), Worldpay AP Limited (Company No: 05593466 / Financial Conduct > Authority No: 502597). Registered Office: The Walbrook Building, 25 Walbrook, > London EC4N 8AF and authorised by the Financial Conduct Authority under the > Payment Service Regulations 2009 for the provision of payment services. > Worldpay (UK) Limited is authorised and regulated by the Financial Conduct > Authority for consumer credit activities. Worldpay B.V. (WPBV) has its > registered office in Amsterdam, the Netherlands (Handelsregister KvK no. > 60494344). WPBV holds a licence from and is included in the register kept by > De Nederlandsche Bank, which registration can be consulted through > www.dnb.nl. Worldpay, the logo and any associated brand names are trade marks > of the Worldpay group. > > > >
