the last try was without log2.cache() and still getting out of memory
I using the following conf, maybe help:
conf = (SparkConf()
.setAppName("LoadS3")
.set("spark.executor.memory", "13g")
.set("spark.driver.memory", "13g")
.set("spark.driver.maxResultSize","2g")
.set("spark.default.parallelism","200")
.set("spark.kryoserializer.buffer.mb","512"))
sc = SparkContext(conf=conf )
sqlContext = SQLContext(sc)
On Thu, Mar 26, 2015 at 2:29 PM, Davies Liu <dav...@databricks.com> wrote:
> Could you try to remove the line `log2.cache()` ?
>
> On Thu, Mar 26, 2015 at 10:02 AM, Eduardo Cusa
> <eduardo.c...@usmediaconsulting.com> wrote:
> > I running on ec2 :
> >
> > 1 Master : 4 CPU 15 GB RAM (2 GB swap)
> >
> > 2 Slaves 4 CPU 15 GB RAM
> >
> >
> > the uncompressed dataset size is 15 GB
> >
> >
> >
> >
> > On Thu, Mar 26, 2015 at 10:41 AM, Eduardo Cusa
> > <eduardo.c...@usmediaconsulting.com> wrote:
> >>
> >> Hi Davies, I upgrade to 1.3.0 and still getting Out of Memory.
> >>
> >> I ran the same code as before, I need to make any changes?
> >>
> >>
> >>
> >>
> >>
> >>
> >> On Wed, Mar 25, 2015 at 4:00 PM, Davies Liu <dav...@databricks.com>
> wrote:
> >>>
> >>> With batchSize = 1, I think it will become even worse.
> >>>
> >>> I'd suggest to go with 1.3, have a taste for the new DataFrame API.
> >>>
> >>> On Wed, Mar 25, 2015 at 11:49 AM, Eduardo Cusa
> >>> <eduardo.c...@usmediaconsulting.com> wrote:
> >>> > Hi Davies, I running 1.1.0.
> >>> >
> >>> > Now I'm following this thread that recommend use batchsize parameter
> =
> >>> > 1
> >>> >
> >>> >
> >>> >
> >>> >
> http://apache-spark-user-list.1001560.n3.nabble.com/pySpark-memory-usage-td3022.html
> >>> >
> >>> > if this does not work I will install 1.2.1 or 1.3
> >>> >
> >>> > Regards
> >>> >
> >>> >
> >>> >
> >>> >
> >>> >
> >>> >
> >>> > On Wed, Mar 25, 2015 at 3:39 PM, Davies Liu <dav...@databricks.com>
> >>> > wrote:
> >>> >>
> >>> >> What's the version of Spark you are running?
> >>> >>
> >>> >> There is a bug in SQL Python API [1], it's fixed in 1.2.1 and 1.3,
> >>> >>
> >>> >> [1] https://issues.apache.org/jira/browse/SPARK-6055
> >>> >>
> >>> >> On Wed, Mar 25, 2015 at 10:33 AM, Eduardo Cusa
> >>> >> <eduardo.c...@usmediaconsulting.com> wrote:
> >>> >> > Hi Guys, I running the following function with spark-submmit and
> de
> >>> >> > SO
> >>> >> > is
> >>> >> > killing my process :
> >>> >> >
> >>> >> >
> >>> >> > def getRdd(self,date,provider):
> >>> >> > path='s3n://'+AWS_BUCKET+'/'+date+'/*.log.gz'
> >>> >> > log2= self.sqlContext.jsonFile(path)
> >>> >> > log2.registerTempTable('log_test')
> >>> >> > log2.cache()
> >>> >>
> >>> >> You only visit the table once, cache does not help here.
> >>> >>
> >>> >> > out=self.sqlContext.sql("SELECT user, tax from log_test where
> >>> >> > provider =
> >>> >> > '"+provider+"'and country <> ''").map(lambda row: (row.user,
> >>> >> > row.tax))
> >>> >> > print "out1"
> >>> >> > return map((lambda (x,y): (x, list(y))),
> >>> >> > sorted(out.groupByKey(2000).collect()))
> >>> >>
> >>> >> 100 partitions (or less) will be enough for 2G dataset.
> >>> >>
> >>> >> >
> >>> >> >
> >>> >> > The input dataset has 57 zip files (2 GB)
> >>> >> >
> >>> >> > The same process with a smaller dataset completed successfully
> >>> >> >
> >>> >> > Any ideas to debug is welcome.
> >>> >> >
> >>> >> > Regards
> >>> >> > Eduardo
> >>> >> >
> >>> >> >
> >>> >
> >>> >
> >>
> >>
> >
>
