[jira] [Comment Edited] (SPARK-18750) spark should be able to control the number of executor and should not throw stack overslow
[ https://issues.apache.org/jira/browse/SPARK-18750?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15733892#comment-15733892 ] Yibing Shi edited comment on SPARK-18750 at 12/9/16 12:59 AM: -- [~srowen] The log says: {noformat} 16/11/29 15:49:11 WARN yarn.ApplicationMaster: Reporter thread fails 1 time(s) in a row. java.lang.StackOverflowError {noformat} It looks like that number of executors is too high for reporter thread to deal with. was (Author: yibing): [~srowen] The log says: {noformat} 16/11/29 15:49:11 WARN yarn.ApplicationMaster: Reporter thread fails 1 time(s) in a row. java.lang.StackOverflowError {noformat} The looks like that number of executors is too high for reporter thread to deal with. > spark should be able to control the number of executor and should not throw > stack overslow > -- > > Key: SPARK-18750 > URL: https://issues.apache.org/jira/browse/SPARK-18750 > Project: Spark > Issue Type: Bug >Reporter: Neerja Khattar > > When running Sql queries on large datasets. Job fails with stack overflow > warning and it shows it is requesting lots of executors. > Looks like there is no limit to number of executors or not even having an > upperbound based on yarn available resources. > 16/11/29 15:47:47 INFO impl.ContainerManagementProtocolProxy: Opening proxy : > bdtcstr61n5.svr.us.jpmchase.net:8041 > 16/11/29 15:47:47 INFO impl.ContainerManagementProtocolProxy: Opening proxy : > bdtcstr61n8.svr.us.jpmchase.net:8041 > 16/11/29 15:47:47 INFO impl.ContainerManagementProtocolProxy: Opening proxy : > bdtcstr61n2.svr.us.jpmchase.net:8041 > 16/11/29 15:47:47 INFO yarn.YarnAllocator: Driver requested a total number of > 32770 executor(s). > 16/11/29 15:47:47 INFO yarn.YarnAllocator: Will request 24576 executor > containers, each with 1 cores and 6758 MB memory including 614 MB overhead > 16/11/29 15:49:11 INFO yarn.YarnAllocator: Driver requested a total number of > 52902 executor(s). > 16/11/29 15:47:47 INFO impl.ContainerManagementProtocolProxy: Opening proxy : > bdtcstr61n5.svr.us.jpmchase.net:8041 > 16/11/29 15:47:47 INFO impl.ContainerManagementProtocolProxy: Opening proxy : > bdtcstr61n8.svr.us.jpmchase.net:8041 > 16/11/29 15:47:47 INFO impl.ContainerManagementProtocolProxy: Opening proxy : > bdtcstr61n2.svr.us.jpmchase.net:8041 > 16/11/29 15:47:47 INFO yarn.YarnAllocator: Driver requested a total number of > 32770 executor(s). > 16/11/29 15:47:47 INFO yarn.YarnAllocator: Will request 24576 executor > containers, each with 1 cores and 6758 MB memory including 614 MB overhead > 16/11/29 15:49:11 INFO yarn.YarnAllocator: Driver requested a total number of > 52902 executor(s). > 16/11/29 15:49:11 WARN yarn.ApplicationMaster: Reporter thread fails 1 > time(s) in a row. > java.lang.StackOverflowError > at scala.collection.immutable.HashMap.$plus(HashMap.scala:57) > at scala.collection.immutable.HashMap.$plus(HashMap.scala:36) > at scala.collection.mutable.MapBuilder.$plus$eq(MapBuilder.scala:28) > at scala.collection.mutable.MapBuilder.$plus$eq(MapBuilder.scala:24) > at > scala.collection.generic.Growable$$anonfun$$plus$plus$eq$1.apply(Growable.scala:48) > at > scala.collection.generic.Growable$$anonfun$$plus$plus$eq$1.apply(Growable.scala:48) > at > scala.collection.immutable.HashMap$HashMap1.foreach(HashMap.scala:224) > at > scala.collection.immutable.HashMap$HashTrieMap.foreach(HashMap.scala:403) > at > scala.collection.immutable.HashMap$HashTrieMap.foreach(HashMap.scala:403) > at > scala.collection.generic.Growable$class.$plus$plus$eq(Growable.scala:48) > at > scala.collection.mutable.MapBuilder.$plus$plus$eq(MapBuilder.scala:24) > at > scala.collection.TraversableLike$class.$plus$plus(TraversableLike.scala:156) > at > scala.collection.AbstractTraversable.$plus$plus(Traversable.scala:105) > at scala.collection.immutable.HashMap.$plus(HashMap.scala:60) > at scala.collection.immutable.Map$Map4.updated(Map.scala:172) > at scala.collection.immutable.Map$Map4.$plus(Map.scala:173) > at scala.collection.immutable.Map$Map4.$plus(Map.scala:158) > at scala.collection.mutable.MapBuilder.$plus$eq(MapBuilder.scala:28) > at scala.collection.mutable.MapBuilder.$plus$eq(MapBuilder.scala:24) > at > scala.collection.TraversableLike$$anonfun$filter$1.apply(TraversableLike.scala:264) > at > scala.collection.MapLike$MappedValues$$anonfun$foreach$3.apply(MapLike.scala:245) > at > scala.collection.MapLike$MappedValues$$anonfun$foreach$3.apply(MapLike.scala:245) > at > scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:772) > at > scala
[jira] [Commented] (SPARK-18750) spark should be able to control the number of executor and should not throw stack overslow
[ https://issues.apache.org/jira/browse/SPARK-18750?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15733892#comment-15733892 ] Yibing Shi commented on SPARK-18750: [~srowen] The log says: {noformat} 16/11/29 15:49:11 WARN yarn.ApplicationMaster: Reporter thread fails 1 time(s) in a row. java.lang.StackOverflowError {noformat} The looks like that number of executors is too high for reporter thread to deal with. > spark should be able to control the number of executor and should not throw > stack overslow > -- > > Key: SPARK-18750 > URL: https://issues.apache.org/jira/browse/SPARK-18750 > Project: Spark > Issue Type: Bug >Reporter: Neerja Khattar > > When running Sql queries on large datasets. Job fails with stack overflow > warning and it shows it is requesting lots of executors. > Looks like there is no limit to number of executors or not even having an > upperbound based on yarn available resources. > 16/11/29 15:47:47 INFO impl.ContainerManagementProtocolProxy: Opening proxy : > bdtcstr61n5.svr.us.jpmchase.net:8041 > 16/11/29 15:47:47 INFO impl.ContainerManagementProtocolProxy: Opening proxy : > bdtcstr61n8.svr.us.jpmchase.net:8041 > 16/11/29 15:47:47 INFO impl.ContainerManagementProtocolProxy: Opening proxy : > bdtcstr61n2.svr.us.jpmchase.net:8041 > 16/11/29 15:47:47 INFO yarn.YarnAllocator: Driver requested a total number of > 32770 executor(s). > 16/11/29 15:47:47 INFO yarn.YarnAllocator: Will request 24576 executor > containers, each with 1 cores and 6758 MB memory including 614 MB overhead > 16/11/29 15:49:11 INFO yarn.YarnAllocator: Driver requested a total number of > 52902 executor(s). > 16/11/29 15:47:47 INFO impl.ContainerManagementProtocolProxy: Opening proxy : > bdtcstr61n5.svr.us.jpmchase.net:8041 > 16/11/29 15:47:47 INFO impl.ContainerManagementProtocolProxy: Opening proxy : > bdtcstr61n8.svr.us.jpmchase.net:8041 > 16/11/29 15:47:47 INFO impl.ContainerManagementProtocolProxy: Opening proxy : > bdtcstr61n2.svr.us.jpmchase.net:8041 > 16/11/29 15:47:47 INFO yarn.YarnAllocator: Driver requested a total number of > 32770 executor(s). > 16/11/29 15:47:47 INFO yarn.YarnAllocator: Will request 24576 executor > containers, each with 1 cores and 6758 MB memory including 614 MB overhead > 16/11/29 15:49:11 INFO yarn.YarnAllocator: Driver requested a total number of > 52902 executor(s). > 16/11/29 15:49:11 WARN yarn.ApplicationMaster: Reporter thread fails 1 > time(s) in a row. > java.lang.StackOverflowError > at scala.collection.immutable.HashMap.$plus(HashMap.scala:57) > at scala.collection.immutable.HashMap.$plus(HashMap.scala:36) > at scala.collection.mutable.MapBuilder.$plus$eq(MapBuilder.scala:28) > at scala.collection.mutable.MapBuilder.$plus$eq(MapBuilder.scala:24) > at > scala.collection.generic.Growable$$anonfun$$plus$plus$eq$1.apply(Growable.scala:48) > at > scala.collection.generic.Growable$$anonfun$$plus$plus$eq$1.apply(Growable.scala:48) > at > scala.collection.immutable.HashMap$HashMap1.foreach(HashMap.scala:224) > at > scala.collection.immutable.HashMap$HashTrieMap.foreach(HashMap.scala:403) > at > scala.collection.immutable.HashMap$HashTrieMap.foreach(HashMap.scala:403) > at > scala.collection.generic.Growable$class.$plus$plus$eq(Growable.scala:48) > at > scala.collection.mutable.MapBuilder.$plus$plus$eq(MapBuilder.scala:24) > at > scala.collection.TraversableLike$class.$plus$plus(TraversableLike.scala:156) > at > scala.collection.AbstractTraversable.$plus$plus(Traversable.scala:105) > at scala.collection.immutable.HashMap.$plus(HashMap.scala:60) > at scala.collection.immutable.Map$Map4.updated(Map.scala:172) > at scala.collection.immutable.Map$Map4.$plus(Map.scala:173) > at scala.collection.immutable.Map$Map4.$plus(Map.scala:158) > at scala.collection.mutable.MapBuilder.$plus$eq(MapBuilder.scala:28) > at scala.collection.mutable.MapBuilder.$plus$eq(MapBuilder.scala:24) > at > scala.collection.TraversableLike$$anonfun$filter$1.apply(TraversableLike.scala:264) > at > scala.collection.MapLike$MappedValues$$anonfun$foreach$3.apply(MapLike.scala:245) > at > scala.collection.MapLike$MappedValues$$anonfun$foreach$3.apply(MapLike.scala:245) > at > scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:772) > at > scala.collection.MapLike$MappedValues$$anonfun$foreach$3.apply(MapLike.scala:245) > at > scala.collection.MapLike$MappedValues$$anonfun$foreach$3.apply(MapLike.scala:245) > at > scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:772) > at > scala.collection.MapLike$Mapped