Re: how to set spark.executor.memory and heap size

[Arpit Tak]

Okk fine,

try like this , i tried and it works..
specify spark path also in constructor...
and also
export SPARK_JAVA_OPTS="-Xms300m -Xmx512m -XX:MaxPermSize=1g"

import org.apache.spark.SparkContext
    import org.apache.spark.SparkContext._
    object SimpleApp {
       def main(args: Array[String]) {
          val logFile = "/var/log/auth.log" // read any file.....
          val sc = new SparkContext("spark://localhost:7077", "Simple App",
"/home/ubuntu/spark-0.9.1-incubating/",
          List("target/scala-2.10/simple-project_2.10-2.0.jar"))
          val tr = sc.textFile(logFile).cache
          tr.take(100).foreach(println)

       }
    }

This will work....


On Thu, Apr 24, 2014 at 3:00 PM, wxhsdp <wxh...@gmail.com> wrote:

> hi arpit,
> on spark shell, i can read local file properly,
> but when i use sbt run, error occurs.
> the sbt error message is in the beginning of the thread
>
>
> Arpit Tak-2 wrote
> > Hi,
> >
> > You should be able to read it, file://or file:/// not even required for
> > reading locally , just path is enough..
> > what error message you getting on spark-shell while reading...
> > for local:
> >
> >
> > Also read the same from hdfs file also ...
> > put your README file there and read , it  works in both ways..
> > val a= sc.textFile("hdfs://localhost:54310/t/README.md")
> >
> > also, print stack message of your spark-shell...
> >
> >
> > On Thu, Apr 24, 2014 at 2:25 PM, wxhsdp &lt;
>
> > wxhsdp@
>
> > &gt; wrote:
> >
> >> thanks for your reply, adnan, i tried
> >> val logFile = "file:///home/wxhsdp/spark/example/standalone/README.md"
> >> i think there needs three left slash behind file:
> >>
> >> it's just the same as val logFile =
> >> "home/wxhsdp/spark/example/standalone/README.md"
> >> the error remains:(
> >>
> >>
> >>
> >> --
> >> View this message in context:
> >>
> http://apache-spark-user-list.1001560.n3.nabble.com/how-to-set-spark-executor-memory-and-heap-size-tp4719p4743.html
> >> Sent from the Apache Spark User List mailing list archive at Nabble.com.
> >>
>
>
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/how-to-set-spark-executor-memory-and-heap-size-tp4719p4745.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>