Yeah, the problem was work around by adding config
spark.sql.codegen.wholeStage=false.
Thanks and Regards,
Tony
发件人: kathleen li
发送时间: 2018年10月8日 12:15
收件人: 阎志涛
抄送: user@spark.apache.org
主题: Re: 答复: Executor hang
https://jaceklaskowski.gitbooks.io/mastering-spark-sql/spark-sql-whole-stage
https://jaceklaskowski.gitbooks.io/mastering-spark-sql/spark-sql-whole-stage-codegen.html#spark.sql.codegen.wholeStage
Sent from my iPhone
> On Oct 7, 2018, at 3:21 PM, 阎志涛 wrote:
>
> It is not data skewed issue. The executor running more than 8 hours and I
> have to kill the executor. I doub
It is not data skewed issue. The executor running more than 8 hours and I have
to kill the executor. I doubt the problem is caused by Spark codegen but it
sound spark.sql.codegen.wholeStage=false can not work. I met another problem
with codegen which cause my spark job failed with following stac
