Thanks a lot for the insight. Actually I have the exact same transformations for all the datasets, hence only 1 python code. Now, do you suggest that I run different spark-submit for all the different datasets given that I have the exact same transformations?
On Tue 5 Jun, 2018, 1:48 AM Jörn Franke [via Apache Spark User List], < ml+s1001560n32458...@n3.nabble.com> wrote: > Yes if they are independent with different transformations then I would > create a separate python program. Especially for big data processing > frameworks one should avoid to put everything in one big monotholic > applications. > > > On 4. Jun 2018, at 22:02, Shuporno Choudhury <[hidden email] > <http:///user/SendEmail.jtp?type=node&node=32458&i=0>> wrote: > > Hi, > > Thanks for the input. > I was trying to get the functionality first, hence I was using local mode. > I will be running on a cluster definitely but later. > > Sorry for my naivety, but can you please elaborate on the modularity > concept that you mentioned and how it will affect whatever I am already > doing? > Do you mean running a different spark-submit for each different dataset > when you say 'an independent python program for each process '? > > On Tue, 5 Jun 2018 at 01:12, Jörn Franke [via Apache Spark User List] <[hidden > email] <http:///user/SendEmail.jtp?type=node&node=32458&i=1>> wrote: > >> Why don’t you modularize your code and write for each process an >> independent python program that is submitted via Spark? >> >> Not sure though if Spark local make sense. If you don’t have a cluster >> then a normal python program can be much better. >> >> On 4. Jun 2018, at 21:37, Shuporno Choudhury <[hidden email] >> <http:///user/SendEmail.jtp?type=node&node=32455&i=0>> wrote: >> >> Hi everyone, >> I am trying to run a pyspark code on some data sets sequentially [basically >> 1. Read data into a dataframe 2.Perform some join/filter/aggregation 3. >> Write modified data in parquet format to a target location] >> Now, while running this pyspark code across *multiple independent data >> sets sequentially*, the memory usage from the previous data set doesn't >> seem to get released/cleared and hence spark's memory consumption (JVM >> memory consumption from Task Manager) keeps on increasing till it fails at >> some data set. >> So, is there a way to clear/remove dataframes that I know are not going >> to be used later? >> Basically, can I clear out some memory programmatically (in the pyspark >> code) when processing for a particular data set ends? >> At no point, I am caching any dataframe (so unpersist() is also not a >> solution). >> >> I am running spark using local[*] as master. There is a single >> SparkSession that is doing all the processing. >> If it is not possible to clear out memory, what can be a better approach >> for this problem? >> >> Can someone please help me with this and tell me if I am going wrong >> anywhere? >> >> --Thanks, >> Shuporno Choudhury >> >> >> >> ------------------------------ >> If you reply to this email, your message will be added to the discussion >> below: >> >> http://apache-spark-user-list.1001560.n3.nabble.com/PySpark-Releasing-memory-after-a-spark-job-is-finished-tp32454p32455.html >> To start a new topic under Apache Spark User List, email [hidden email] >> <http:///user/SendEmail.jtp?type=node&node=32458&i=2> >> To unsubscribe from Apache Spark User List, click here. >> NAML >> <http://apache-spark-user-list.1001560.n3.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml> >> > > > -- > --Thanks, > Shuporno Choudhury > > > > ------------------------------ > If you reply to this email, your message will be added to the discussion > below: > > http://apache-spark-user-list.1001560.n3.nabble.com/PySpark-Releasing-memory-after-a-spark-job-is-finished-tp32454p32458.html > To start a new topic under Apache Spark User List, email > ml+s1001560n1...@n3.nabble.com > To unsubscribe from Apache Spark User List, click here > <http://apache-spark-user-list.1001560.n3.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=1&code=c2h1cG9ybm8uY2hvdWRodXJ5QGdtYWlsLmNvbXwxfC0xODI0MTU0MzQ0> > . > NAML > <http://apache-spark-user-list.1001560.n3.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml> >
