Spark: Why Standalone mode can not set Executor Number.

As far as I know, only yarn mode can set --num-executors, someone proved to set more number-execuotrs for will perform better than set only 1 or 2 executor with large mem and core. sett http://apache-spark-user-list.1001560.n3.nabble.com/executor-cores-vs-num-executors-td9878.html Why Standalon

Re: Spark: why need a masterLock when sending heartbeat to master

Thanks, I got it ! -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Spark-why-need-a-masterLock-when-sending-heartbeat-to-master-tp12256p12297.html Sent from the Apache Spark User List mailing list archive at Nabble.com. -

Spark: why need a masterLock when sending heartbeat to master

I don't understand why worker need a master lock when sending heartbeat. Caused by master HA ? Who can explain this in detail? Thanks~ Please refer: http://stackoverflow.com/questions/25173219/why-does-the-spark-worker-actor-use-a-masterlock case SendHeartbeat => masterLock.synchronized {

Re: spark1.0.1 spark sql error java.lang.NoClassDefFoundError: Could not initialize class $line11.$read$

Hi, Yin Huai I test again with your snippet code. It works well in spark-1.0.1 Here is my code: val sqlContext = new org.apache.spark.sql.SQLContext(sc) case class Record(data_date: String, mobile: String, create_time: String) val mobile = Record("2014-07-20","1234567","2014-07-19

Re: spark1.0.1 spark sql error java.lang.NoClassDefFoundError: Could not initialize class $line11.$read$

Hi,Kevin I tried it on spark1.0.0, it works fine. It's a bug in spark1.0.1 ... Thanks, Victor -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/spark1-0-1-spark-sql-error-java-lang-NoClassDefFoundError-Could-not-initialize-class-line11-read-tp10135p102

Re: spark1.0.1 spark sql error java.lang.NoClassDefFoundError: Could not initialize class $line11.$read$

Hi, Michael I only modified the default hadoop version to 0.20.2-cdh3u5, and DEFAULT_HIVE=true in SparkBuild.scala. Then sbt/sbt assembly. I just run in the local standalone mode by using sbin/start-all.sh. Hadoop version is 0.20.2-cdh3u5. Then use spark-shell to execute the spark s

Re: spark1.0.1 spark sql error java.lang.NoClassDefFoundError: Could not initialize class $line11.$read$

Hi,Svend Your reply is very helpful to me. I'll keep an eye on that ticket. And also... Cheers :) Best Regards, Victor -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/spark1-0-1-spark-sql-error-java-lang-NoClassDefFoundError-Could-not-initialize

spark1.0.1 spark sql error java.lang.NoClassDefFoundError: Could not initialize class $line11.$read$

when I run a query to a hadoop file. mobile.registerAsTable("mobile") val count = sqlContext.sql("select count(1) from mobile") res5: org.apache.spark.sql.SchemaRDD = SchemaRDD[21] at RDD at SchemaRDD.scala:100 == Query Plan == ExistingRdd [data_date#0,mobile#1,create_time#2], MapPartitionsRDD[4]

Re: spark1.0.1 catalyst transform filter not push down

I use queryPlan.queryExecution.analyzed to get the logical plan. it works. And What you explained to me is very useful. Thank you very much. -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/spark1-0-1-catalyst-transform-filter-not-push-down-tp9599p9689.ht

spark1.0.1 catalyst transform filter not push down

Hi, I encountered a weird problem in spark sql. I use sbt/sbt hive/console to go into the shell. I test the filter push down by using catalyst. scala> val queryPlan = sql("select value from (select key,value from src)a where a.key=86 ") scala> queryPlan.baseLogicalPlan res0: org.apache.spark.sq

spark1.0 spark sql saveAsParquetFile Error

I've got an exception when run the example of spark sql -- Using Parquet When I call saveAsParquetFile method on people schemaRDD, throws an exception like below, I don't know why, can anyone help me with it ? Exception: parquet.io.ParquetDecodingException: Can not read value at 0 in block -1