Hello Rui, Sorry , What I meant was the resultant of the original dataframe to which a new column was added gives a new DataFrame.
Please check this for more https://spark.apache.org/docs/1.5.1/api/R/index.html Check for WithColumn Thanks, Vipul On 23 November 2015 at 12:42, Sun, Rui <rui....@intel.com> wrote: > Vipul, > > Not sure if I understand your question. DataFrame is immutable. You can't > update a DataFrame. > > Could you paste some log info for the OOM error? > > -----Original Message----- > From: vipulrai [mailto:vipulrai8...@gmail.com] > Sent: Friday, November 20, 2015 12:11 PM > To: user@spark.apache.org > Subject: SparkR DataFrame , Out of memory exception for very small file. > > Hi Users, > > I have a general doubt regarding DataFrames in SparkR. > > I am trying to read a file from Hive and it gets created as DataFrame. > > sqlContext <- sparkRHive.init(sc) > > #DF > sales <- read.df(sqlContext, "hdfs://sample.csv", header ='true', > source = "com.databricks.spark.csv", inferSchema='true') > > registerTempTable(sales,"Sales") > > Do I need to create a new DataFrame for every update to the DataFrame like > addition of new column or need to update the original sales DataFrame. > > sales1<- SparkR::sql(sqlContext,"Select a.* , 607 as C1 from Sales as a") > > > Please help me with this , as the orignal file is only 20MB but it throws > out of memory exception on a cluster of 4GB Master and Two workers of 4GB > each. > > Also, what is the logic with DataFrame do I need to register and drop > tempTable after every update?? > > Thanks, > Vipul > > > > -- > View this message in context: > http://apache-spark-user-list.1001560.n3.nabble.com/SparkR-DataFrame-Out-of-memory-exception-for-very-small-file-tp25435.html > Sent from the Apache Spark User List mailing list archive at Nabble.com. > > --------------------------------------------------------------------- > To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional > commands, e-mail: user-h...@spark.apache.org > > -- Regards, Vipul Rai www.vipulrai.me +91-8892598819 <http://in.linkedin.com/in/vipulrai/>
