As far as I know, only yarn mode can set --num-executors, someone proved to
set more number-execuotrs for will perform better than set only 1 or 2
executor with large mem and core. sett
http://apache-spark-user-list.1001560.n3.nabble.com/executor-cores-vs-num-executors-td9878.html
Why Standalon
Thanks, I got it !
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/Spark-why-need-a-masterLock-when-sending-heartbeat-to-master-tp12256p12297.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
-
I don't understand why worker need a master lock when sending heartbeat.
Caused by master HA ? Who can explain this in detail? Thanks~
Please refer:
http://stackoverflow.com/questions/25173219/why-does-the-spark-worker-actor-use-a-masterlock
case SendHeartbeat =>
masterLock.synchronized {
Hi, Yin Huai
I test again with your snippet code.
It works well in spark-1.0.1
Here is my code:
val sqlContext = new org.apache.spark.sql.SQLContext(sc)
case class Record(data_date: String, mobile: String, create_time: String)
val mobile = Record("2014-07-20","1234567","2014-07-19
Hi,Kevin
I tried it on spark1.0.0, it works fine.
It's a bug in spark1.0.1 ...
Thanks,
Victor
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/spark1-0-1-spark-sql-error-java-lang-NoClassDefFoundError-Could-not-initialize-class-line11-read-tp10135p102
Hi, Michael
I only modified the default hadoop version to 0.20.2-cdh3u5, and
DEFAULT_HIVE=true in SparkBuild.scala.
Then sbt/sbt assembly.
I just run in the local standalone mode by using sbin/start-all.sh.
Hadoop version is 0.20.2-cdh3u5.
Then use spark-shell to execute the spark s
Hi,Svend
Your reply is very helpful to me. I'll keep an eye on that ticket.
And also... Cheers :)
Best Regards,
Victor
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/spark1-0-1-spark-sql-error-java-lang-NoClassDefFoundError-Could-not-initialize
when I run a query to a hadoop file.
mobile.registerAsTable("mobile")
val count = sqlContext.sql("select count(1) from mobile")
res5: org.apache.spark.sql.SchemaRDD =
SchemaRDD[21] at RDD at SchemaRDD.scala:100
== Query Plan ==
ExistingRdd [data_date#0,mobile#1,create_time#2], MapPartitionsRDD[4]
I use queryPlan.queryExecution.analyzed to get the logical plan.
it works.
And What you explained to me is very useful.
Thank you very much.
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/spark1-0-1-catalyst-transform-filter-not-push-down-tp9599p9689.ht
Hi, I encountered a weird problem in spark sql.
I use sbt/sbt hive/console to go into the shell.
I test the filter push down by using catalyst.
scala> val queryPlan = sql("select value from (select key,value from src)a
where a.key=86 ")
scala> queryPlan.baseLogicalPlan
res0: org.apache.spark.sq
I've got an exception when run the example of spark sql -- Using Parquet
When I call saveAsParquetFile method on people schemaRDD, throws an
exception like below, I don't know why, can anyone help me with it ?
Exception:
parquet.io.ParquetDecodingException: Can not read value at 0 in block -1
11 matches
Mail list logo