Spark: Why Standalone mode can not set Executor Number.

As far as I know, only yarn mode can set --num-executors, someone proved to set more number-execuotrs for will perform better than set only 1 or 2 executor with large mem and core. sett http://apache-spark-user-list.1001560.n3.nabble.com/executor-cores-vs-num-executors-td9878.html Why

Re: Spark: why need a masterLock when sending heartbeat to master

Thanks, I got it ! -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Spark-why-need-a-masterLock-when-sending-heartbeat-to-master-tp12256p12297.html Sent from the Apache Spark User List mailing list archive at Nabble.com.

Spark: why need a masterLock when sending heartbeat to master

I don't understand why worker need a master lock when sending heartbeat. Caused by master HA ? Who can explain this in detail? Thanks~ Please refer: http://stackoverflow.com/questions/25173219/why-does-the-spark-worker-actor-use-a-masterlock case SendHeartbeat = masterLock.synchronized {

Re: spark1.0.1 spark sql error java.lang.NoClassDefFoundError: Could not initialize class $line11.$read$

Hi, Yin Huai I test again with your snippet code. It works well in spark-1.0.1 Here is my code: val sqlContext = new org.apache.spark.sql.SQLContext(sc) case class Record(data_date: String, mobile: String, create_time: String) val mobile = Record(2014-07-20,1234567,2014-07-19)

Re: spark1.0.1 spark sql error java.lang.NoClassDefFoundError: Could not initialize class $line11.$read$

Hi,Kevin I tried it on spark1.0.0, it works fine. It's a bug in spark1.0.1 ... Thanks, Victor -- View this message in context:

Re: spark1.0.1 spark sql error java.lang.NoClassDefFoundError: Could not initialize class $line11.$read$

Hi, Michael I only modified the default hadoop version to 0.20.2-cdh3u5, and DEFAULT_HIVE=true in SparkBuild.scala. Then sbt/sbt assembly. I just run in the local standalone mode by using sbin/start-all.sh. Hadoop version is 0.20.2-cdh3u5. Then use spark-shell to execute the spark

Re: spark1.0.1 spark sql error java.lang.NoClassDefFoundError: Could not initialize class $line11.$read$

Hi,Svend Your reply is very helpful to me. I'll keep an eye on that ticket. And also... Cheers :) Best Regards, Victor -- View this message in context:

spark1.0.1 spark sql error java.lang.NoClassDefFoundError: Could not initialize class $line11.$read$

when I run a query to a hadoop file. mobile.registerAsTable(mobile) val count = sqlContext.sql(select count(1) from mobile) res5: org.apache.spark.sql.SchemaRDD = SchemaRDD[21] at RDD at SchemaRDD.scala:100 == Query Plan == ExistingRdd [data_date#0,mobile#1,create_time#2], MapPartitionsRDD[4] at

spark1.0.1 catalyst transform filter not push down

Hi, I encountered a weird problem in spark sql. I use sbt/sbt hive/console to go into the shell. I test the filter push down by using catalyst. scala val queryPlan = sql(select value from (select key,value from src)a where a.key=86 ) scala queryPlan.baseLogicalPlan res0:

Re: spark1.0.1 catalyst transform filter not push down

I use queryPlan.queryExecution.analyzed to get the logical plan. it works. And What you explained to me is very useful. Thank you very much. -- View this message in context: