RE: StackOverflow in Spark
I’ve found my problem. I’ve got a DAG with two consecutive “updateStateByKey” functions . When I only process (map/foreachRDD/JavaEsSpark) the state of the last “updateStateByKey” function, I get an stackoverflow after a while (too long linage). But when I also do some processing (foreachRDD/rdd.take) on the first “updatestatebykey”, then there is no problem. Does this make sense? Probably the “long linage” problem. But why should I have such a “linage problem” when Sparks claims to be a “abstract/high level” architecture? Why should I be worried about “long linage”? Its seems a contradiction with the abstract/high level (functional programming) approach when I have to know/consider how Spark doest it. Van: Rishabh Wadhawan [mailto:rishabh...@gmail.com] Verzonden: donderdag 2 juni 2016 06:06 Aan: Yash Sharma <yash...@gmail.com> CC: Ted Yu <yuzhih...@gmail.com>; Matthew Young <taige...@gmail.com>; Michel Hubert <mich...@phact.nl>; user@spark.apache.org Onderwerp: Re: StackOverflow in Spark Stackoverflow is generated when DAG is too log as there are many transformations in lot of iterations. Please use checkpointing to store the DAG and break the linage to get away from this stack overflow error. Look into checkpoint fuction. Thanks Hope it helps. Let me know if you need anymore help. On Jun 1, 2016, at 8:18 PM, Yash Sharma <yash...@gmail.com<mailto:yash...@gmail.com>> wrote: Not sure if its related, But I got a similar stack overflow error some time back while reading files and converting them to parquet. Stack trace- 16/06/02 02:23:54 INFO YarnAllocator: Driver requested a total number of 32769 executor(s). 16/06/02 02:23:54 INFO ExecutorAllocationManager: Requesting 16384 new executors because tasks are backlogged (new desired total will be 32769) 16/06/02 02:23:54 INFO YarnAllocator: Will request 24576 executor containers, each with 5 cores and 22528 MB memory including 2048 MB overhead 16/06/02 02:23:55 WARN ApplicationMaster: Reporter thread fails 2 time(s) in a row. at scala.collection.mutable.MapBuilder.$plus$eq(MapBuilder.scala:28) at scala.collection.mutable.MapBuilder.$plus$eq(MapBuilder.scala:24) at scala.collection.TraversableLike$$anonfun$filter$1.apply(TraversableLike.scala:264) at scala.collection.MapLike$MappedValues$$anonfun$foreach$3.apply(MapLike.scala:245) at scala.collection.MapLike$MappedValues$$anonfun$foreach$3.apply(MapLike.scala:245) at scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:772) at scala.collection.MapLike$MappedValues$$anonfun$foreach$3.apply(MapLike.scala:245) at scala.collection.MapLike$MappedValues$$anonfun$foreach$3.apply(MapLike.scala:245) java.lang.StackOverflowError On Thu, Jun 2, 2016 at 12:58 PM, Ted Yu <yuzhih...@gmail.com<mailto:yuzhih...@gmail.com>> wrote: Looking at Michel's stack trace, it seems to be different issue. On Jun 1, 2016, at 7:45 PM, Matthew Young <taige...@gmail.com<mailto:taige...@gmail.com>> wrote: Hi, It's related to the one fixed bug in Spark, jira ticket SPARK-6847<https://issues.apache.org/jira/browse/SPARK-6847> Matthew Yang On Wed, May 25, 2016 at 7:48 PM, Michel Hubert <mich...@phact.nl<mailto:mich...@phact.nl>> wrote: Hi, I have an Spark application which generates StackOverflowError exceptions after 30+ min. Anyone any ideas? 16/05/25 10:48:51 WARN scheduler.TaskSetManager: Lost task 0.0 in stage 55449.0 (TID 5584, host81440-cld.opentsp.com<http://host81440-cld.opentsp.com/>): java.lang.StackOverflowError ·at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1382) ·at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990) ·at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915) ·at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798) ·at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350) ·at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990) ·at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915) ·at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798) ·at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350) ·at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370) ·at scala.collection.immutable.$colon$colon.readObject(List.scala:362) ·at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source) ·at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ·at java.lang.reflect.Method.invoke(Method.java:606) ·at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1017) ·at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1893) ·at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798) ·at java.io.ObjectInputStream.readObject0(ObjectInputStre
submitMissingTasks - serialize throws StackOverflow exception
Hi, My Spark application throws stackoverflow exceptions after a while. The DAGScheduler function submitMissingTasks tries to serialize a Tuple (MapPartitionsRDD, EsSpark..saveToEs) which is handled with a recursive algorithm. The recursive algorithm is too deep and results in a stackoverflow exception. Should I just try to increase the heap size? Or will it just happen later in time? How can I fix this? With kind regards, michel [cid:image001.png@01D1B7F5.8E8C3980]
StackOverflow in Spark
Hi, I have an Spark application which generates StackOverflowError exceptions after 30+ min. Anyone any ideas? Seems like problems with deserialization of checkpoint data? 16/05/25 10:48:51 WARN scheduler.TaskSetManager: Lost task 0.0 in stage 55449.0 (TID 5584, host81440-cld.opentsp.com): java.lang.StackOverflowError *at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1382) *at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990) *at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915) *at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798) *at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350) *at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990) *at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915) *at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798) *at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350) *at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370) *at scala.collection.immutable.$colon$colon.readObject(List.scala:362) *at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source) *at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) *at java.lang.reflect.Method.invoke(Method.java:606) *at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1017) *at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1893) *at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798) *at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350) *at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990) *at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915) *at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798) *at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350) *at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990) *at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915) *at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798) *at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350) *at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370) *at scala.collection.immutable.$colon$colon.readObject(List.scala:362) at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source) Driver stacktrace: *at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1431) *at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1419) *at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1418) *at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59) *at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47) *at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1418) *at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:799) *at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:799) *at scala.Option.foreach(Option.scala:236) *at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:799) *at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:1640) *at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1599) *at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1588) *at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48) *at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:620) *at org.apache.spark.SparkContext.runJob(SparkContext.scala:1843) *at org.apache.spark.SparkContext.runJob(SparkContext.scala:1856) *at org.apache.spark.SparkContext.runJob(SparkContext.scala:1933) *at org.elasticsearch.spark.rdd.EsSpark$.saveToEs(EsSpark.scala:67) *at org.elasticsearch.spark.rdd.EsSpark$.saveToEs(EsSpark.scala:54) *at org.elasticsearch.spark.rdd.EsSpark$.saveJsonToEs(EsSpark.scala:90) *at org.elasticsearch.spark.rdd.api.java.JavaEsSpark$.saveJsonToEs(JavaEsSpark.scala:62) at org.elasticsearch.spark.rdd.api.java.JavaEsSpark.saveJsonToEs(JavaEsSpark.scala)
Kafka 0.9 and spark-streaming-kafka_2.10
Hi, I'm thinking of upgdrading our kafka cluster to 0.9. Will this be a problem for the Spark Streaming + Kafka Direct Approach Integration using artifact spark-streaming-kafka_2.10 (1.6.1)? groupId = org.apache.spark artifactId = spark-streaming-kafka_2.10 version = 1.6.1 Because the documentation states: Kafka: Spark Streaming 1.6.1 is compatible with Kafka 0.8.2.1. Thanks.
RE: run-example streaming.KafkaWordCount fails on CDH 5.7.0
We're running Kafka 0.8.2.2 Is that the problem, why? -Oorspronkelijk bericht- Van: Sean Owen [mailto:so...@cloudera.com] Verzonden: woensdag 4 mei 2016 10:41 Aan: Michel Hubert <mich...@phact.nl> CC: user@spark.apache.org Onderwerp: Re: run-example streaming.KafkaWordCount fails on CDH 5.7.0 Please try the CDH forums; this is the Spark list: http://community.cloudera.com/t5/Advanced-Analytics-Apache-Spark/bd-p/Spark Before you even do that, I can tell you to double check you're running Kafka 0.9. On Wed, May 4, 2016 at 9:29 AM, Michel Hubert <mich...@phact.nl> wrote: > > > Hi, > > > > After an upgrade to CDH 5.7.0 we have troubles with the Kafka to Spark > Streaming. > > > > The example jar doesn’t work: > > > > /opt/cloudera/parcels/CDH/lib/spark/bin/run-example > streaming.KafkaWordCount …. > > > > Attached is a log file. > > > > 16/05/04 10:06:23 WARN consumer.ConsumerFetcherThread: > [ConsumerFetcherThread-wordcount1_host81436-cld.domain.com-14623491821 > 72-13b9f2e7-0-2], Error in fetch > kafka.consumer.ConsumerFetcherThread$FetchRequest@47fb7e6d. > Possible cause: java.nio.BufferUnderflowException > > 16/05/04 10:06:24 WARN consumer.ConsumerFetcherThread: > [ConsumerFetcherThread-wordcount1_host81436-cld.domain.com-14623491821 > 72-13b9f2e7-0-1], Error in fetch > kafka.consumer.ConsumerFetcherThread$FetchRequest@73b9a762. > Possible cause: java.lang.IllegalArgumentException > > > > We have no problem running this from my development pc, only in > product op CDH 5.7.0 environment. > > > > > > Any ideas? > > > > Thanks, > > Michel > > > > > > - > To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For > additional commands, e-mail: user-h...@spark.apache.org
RE: Kafka exception in Apache Spark
This is production. Van: Mich Talebzadeh [mailto:mich.talebza...@gmail.com] Verzonden: dinsdag 26 april 2016 12:01 Aan: Michel Hubert <mich...@phact.nl> CC: user@spark.apache.org Onderwerp: Re: Kafka exception in Apache Spark Hi Michael, Is this production or test? Dr Mich Talebzadeh LinkedIn https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw http://talebzadehmich.wordpress.com<http://talebzadehmich.wordpress.com/> On 26 April 2016 at 09:07, Michel Hubert <mich...@phact.nl<mailto:mich...@phact.nl>> wrote: Hi, I use a Kafka direct stream approach. My Spark application was running ok. This morning we upgraded to CDH 5.7.0 And when I re-started my Spark application I get exceptions. It seems a problem with the direct stream approach. Any ideas how to fix this? User class threw exception: org.apache.spark.SparkException: Job aborted due to stage failure: Task 3 in stage 0.0 failed 4 times, most recent failure: Lost task 3.3 in stage 0.0 (TID 26, bfravicsvr81439-cld.opentsp.com<http://bfravicsvr81439-cld.opentsp.com>): java.lang.IllegalArgumentException at java.nio.Buffer.limit(Buffer.java:267) at kafka.api.FetchResponsePartitionData$.readFrom(FetchResponse.scala:38) at kafka.api.TopicData$$anonfun$1.apply(FetchResponse.scala:100) at kafka.api.TopicData$$anonfun$1.apply(FetchResponse.scala:98) at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244) at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244) at scala.collection.immutable.Range.foreach(Range.scala:141) at scala.collection.TraversableLike$class.map(TraversableLike.scala:244) at scala.collection.AbstractTraversable.map(Traversable.scala:105) at kafka.api.TopicData$.readFrom(FetchResponse.scala:98) at kafka.api.FetchResponse$$anonfun$4.apply(FetchResponse.scala:170) at kafka.api.FetchResponse$$anonfun$4.apply(FetchResponse.scala:169) at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:251) at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:251) at scala.collection.immutable.Range.foreach(Range.scala:141) at scala.collection.TraversableLike$class.flatMap(TraversableLike.scala:251) at scala.collection.AbstractTraversable.flatMap(Traversable.scala:105) at kafka.api.FetchResponse$.readFrom(FetchResponse.scala:169) at kafka.consumer.SimpleConsumer.fetch(SimpleConsumer.scala:135) at org.apache.spark.streaming.kafka.KafkaRDD$KafkaRDDIterator.fetchBatch(KafkaRDD.scala:192) at org.apache.spark.streaming.kafka.KafkaRDD$KafkaRDDIterator.getNext(KafkaRDD.scala:208) at org.apache.spark.util.NextIterator.hasNext(NextIterator.scala:73) at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327) at scala.collection.Iterator$$anon$14.hasNext(Iterator.scala:388) at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327) at org.apache.spark.shuffle.sort.BypassMergeSortShuffleWriter.write(BypassMergeSortShuffleWriter.java:126) at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:73) at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41) at org.apache.spark.scheduler.Task.run(Task.scala:89) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745)
Kafka exception in Apache Spark
Hi, I use a Kafka direct stream approach. My Spark application was running ok. This morning we upgraded to CDH 5.7.0 And when I re-started my Spark application I get exceptions. It seems a problem with the direct stream approach. Any ideas how to fix this? User class threw exception: org.apache.spark.SparkException: Job aborted due to stage failure: Task 3 in stage 0.0 failed 4 times, most recent failure: Lost task 3.3 in stage 0.0 (TID 26, bfravicsvr81439-cld.opentsp.com): java.lang.IllegalArgumentException at java.nio.Buffer.limit(Buffer.java:267) at kafka.api.FetchResponsePartitionData$.readFrom(FetchResponse.scala:38) at kafka.api.TopicData$$anonfun$1.apply(FetchResponse.scala:100) at kafka.api.TopicData$$anonfun$1.apply(FetchResponse.scala:98) at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244) at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244) at scala.collection.immutable.Range.foreach(Range.scala:141) at scala.collection.TraversableLike$class.map(TraversableLike.scala:244) at scala.collection.AbstractTraversable.map(Traversable.scala:105) at kafka.api.TopicData$.readFrom(FetchResponse.scala:98) at kafka.api.FetchResponse$$anonfun$4.apply(FetchResponse.scala:170) at kafka.api.FetchResponse$$anonfun$4.apply(FetchResponse.scala:169) at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:251) at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:251) at scala.collection.immutable.Range.foreach(Range.scala:141) at scala.collection.TraversableLike$class.flatMap(TraversableLike.scala:251) at scala.collection.AbstractTraversable.flatMap(Traversable.scala:105) at kafka.api.FetchResponse$.readFrom(FetchResponse.scala:169) at kafka.consumer.SimpleConsumer.fetch(SimpleConsumer.scala:135) at org.apache.spark.streaming.kafka.KafkaRDD$KafkaRDDIterator.fetchBatch(KafkaRDD.scala:192) at org.apache.spark.streaming.kafka.KafkaRDD$KafkaRDDIterator.getNext(KafkaRDD.scala:208) at org.apache.spark.util.NextIterator.hasNext(NextIterator.scala:73) at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327) at scala.collection.Iterator$$anon$14.hasNext(Iterator.scala:388) at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327) at org.apache.spark.shuffle.sort.BypassMergeSortShuffleWriter.write(BypassMergeSortShuffleWriter.java:126) at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:73) at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41) at org.apache.spark.scheduler.Task.run(Task.scala:89) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745)
RE: apache spark errors
No.
But I may be on to something.
I use Jedis to send data to Redis.
I used a ThreadLocal construct:
private static final ThreadLocal jedis = new ThreadLocal(){
@Override
protected Jedis initialValue()
{
return new Jedis("10.101.41.19",6379);
}
};
and then
.foreachRDD(new VoidFunction<JavaRDD>() {
public void call(JavaRDD rdd) throws Exception {
for (TopData t: rdd.take(top)) {
jedis …
}
May this resulted in a memory leak?
Van: Ted Yu [mailto:yuzhih...@gmail.com]
Verzonden: donderdag 24 maart 2016 15:15
Aan: Michel Hubert <mich...@phact.nl>
CC: user@spark.apache.org
Onderwerp: Re: apache spark errors
Do you have history server enabled ?
Posting your code snippet would help us understand your use case (and reproduce
the leak).
Thanks
On Thu, Mar 24, 2016 at 6:40 AM, Michel Hubert
<mich...@phact.nl<mailto:mich...@phact.nl>> wrote:
org.apache.spark
spark-core_2.10
1.6.1
org.apache.spark
spark-streaming_2.10
1.6.1
org.apache.spark
spark-streaming-kafka_2.10
1.6.1
org.elasticsearch
elasticsearch
2.2.0
org.apache.kafka
kafka_2.10
0.8.2.2
org.elasticsearch
elasticsearch-spark_2.10
2.2.0
redis.clients
jedis
2.8.0
jar
compile
How can I look at those tasks?
Van: Ted Yu [mailto:yuzhih...@gmail.com<mailto:yuzhih...@gmail.com>]
Verzonden: donderdag 24 maart 2016 14:33
Aan: Michel Hubert <mich...@phact.nl<mailto:mich...@phact.nl>>
CC: user@spark.apache.org<mailto:user@spark.apache.org>
Onderwerp: Re: apache spark errors
Which release of Spark are you using ?
Have you looked the tasks whose Ids were printed to see if there was more clue ?
Thanks
On Thu, Mar 24, 2016 at 6:12 AM, Michel Hubert
<mich...@phact.nl<mailto:mich...@phact.nl>> wrote:
HI,
I constantly get these errors:
0[Executor task launch worker-15] ERROR org.apache.spark.executor.Executor
- Managed memory leak detected; size = 6564500 bytes, TID = 38969
310002 [Executor task launch worker-12] ERROR
org.apache.spark.executor.Executor - Managed memory leak detected; size =
5523550 bytes, TID = 43270
318445 [Executor task launch worker-12] ERROR
org.apache.spark.executor.Executor - Managed memory leak detected; size =
6879566 bytes, TID = 43408
388893 [Executor task launch worker-12] ERROR
org.apache.spark.executor.Executor - Managed memory leak detected; size =
5572546 bytes, TID = 44382
418186 [Executor task launch worker-13] ERROR
org.apache.spark.executor.Executor - Managed memory leak detected; size =
5289222 bytes, TID = 44795
488421 [Executor task launch worker-4] ERROR org.apache.spark.executor.Executor
- Managed memory leak detected; size = 8738142 bytes, TID = 45769
619276 [Executor task launch worker-4] ERROR org.apache.spark.executor.Executor
- Managed memory leak detected; size = 5759312 bytes, TID = 47571
632275 [Executor task launch worker-12] ERROR
org.apache.spark.executor.Executor - Managed memory leak detected; size =
5602240 bytes, TID = 47709
644989 [Executor task launch worker-13] ERROR
org.apache.spark.executor.Executor - Managed memory leak detected; size =
5326260 bytes, TID = 47863
720701 [Executor task launch worker-12] ERROR
org.apache.spark.executor.Executor - Managed memory leak detected; size =
5399578 bytes, TID = 48959
1147961 [Executor task launch worker-16] ERROR
org.apache.spark.executor.Executor - Managed memory leak detected; size =
5251872 bytes, TID = 54922
How can I fix this?
With kind regard,
Michel
RE: apache spark errors
org.apache.spark spark-core_2.10 1.6.1 org.apache.spark spark-streaming_2.10 1.6.1 org.apache.spark spark-streaming-kafka_2.10 1.6.1 org.elasticsearch elasticsearch 2.2.0 org.apache.kafka kafka_2.10 0.8.2.2 org.elasticsearch elasticsearch-spark_2.10 2.2.0 redis.clients jedis 2.8.0 jar compile How can I look at those tasks? Van: Ted Yu [mailto:yuzhih...@gmail.com] Verzonden: donderdag 24 maart 2016 14:33 Aan: Michel Hubert <mich...@phact.nl> CC: user@spark.apache.org Onderwerp: Re: apache spark errors Which release of Spark are you using ? Have you looked the tasks whose Ids were printed to see if there was more clue ? Thanks On Thu, Mar 24, 2016 at 6:12 AM, Michel Hubert <mich...@phact.nl<mailto:mich...@phact.nl>> wrote: HI, I constantly get these errors: 0[Executor task launch worker-15] ERROR org.apache.spark.executor.Executor - Managed memory leak detected; size = 6564500 bytes, TID = 38969 310002 [Executor task launch worker-12] ERROR org.apache.spark.executor.Executor - Managed memory leak detected; size = 5523550 bytes, TID = 43270 318445 [Executor task launch worker-12] ERROR org.apache.spark.executor.Executor - Managed memory leak detected; size = 6879566 bytes, TID = 43408 388893 [Executor task launch worker-12] ERROR org.apache.spark.executor.Executor - Managed memory leak detected; size = 5572546 bytes, TID = 44382 418186 [Executor task launch worker-13] ERROR org.apache.spark.executor.Executor - Managed memory leak detected; size = 5289222 bytes, TID = 44795 488421 [Executor task launch worker-4] ERROR org.apache.spark.executor.Executor - Managed memory leak detected; size = 8738142 bytes, TID = 45769 619276 [Executor task launch worker-4] ERROR org.apache.spark.executor.Executor - Managed memory leak detected; size = 5759312 bytes, TID = 47571 632275 [Executor task launch worker-12] ERROR org.apache.spark.executor.Executor - Managed memory leak detected; size = 5602240 bytes, TID = 47709 644989 [Executor task launch worker-13] ERROR org.apache.spark.executor.Executor - Managed memory leak detected; size = 5326260 bytes, TID = 47863 720701 [Executor task launch worker-12] ERROR org.apache.spark.executor.Executor - Managed memory leak detected; size = 5399578 bytes, TID = 48959 1147961 [Executor task launch worker-16] ERROR org.apache.spark.executor.Executor - Managed memory leak detected; size = 5251872 bytes, TID = 54922 How can I fix this? With kind regard, Michel
apache spark errors
HI, I constantly get these errors: 0[Executor task launch worker-15] ERROR org.apache.spark.executor.Executor - Managed memory leak detected; size = 6564500 bytes, TID = 38969 310002 [Executor task launch worker-12] ERROR org.apache.spark.executor.Executor - Managed memory leak detected; size = 5523550 bytes, TID = 43270 318445 [Executor task launch worker-12] ERROR org.apache.spark.executor.Executor - Managed memory leak detected; size = 6879566 bytes, TID = 43408 388893 [Executor task launch worker-12] ERROR org.apache.spark.executor.Executor - Managed memory leak detected; size = 5572546 bytes, TID = 44382 418186 [Executor task launch worker-13] ERROR org.apache.spark.executor.Executor - Managed memory leak detected; size = 5289222 bytes, TID = 44795 488421 [Executor task launch worker-4] ERROR org.apache.spark.executor.Executor - Managed memory leak detected; size = 8738142 bytes, TID = 45769 619276 [Executor task launch worker-4] ERROR org.apache.spark.executor.Executor - Managed memory leak detected; size = 5759312 bytes, TID = 47571 632275 [Executor task launch worker-12] ERROR org.apache.spark.executor.Executor - Managed memory leak detected; size = 5602240 bytes, TID = 47709 644989 [Executor task launch worker-13] ERROR org.apache.spark.executor.Executor - Managed memory leak detected; size = 5326260 bytes, TID = 47863 720701 [Executor task launch worker-12] ERROR org.apache.spark.executor.Executor - Managed memory leak detected; size = 5399578 bytes, TID = 48959 1147961 [Executor task launch worker-16] ERROR org.apache.spark.executor.Executor - Managed memory leak detected; size = 5251872 bytes, TID = 54922 How can I fix this? With kind regard, Michel
Spark 1.6.0 op CDH 5.6.0
Hi, I'm trying to run a Spark 1.6.0 application on a CDH 5.6.0 cluster. How do I submit the uber-jar so it's totally self-reliant? With kind regards, Mitchel spark-submit --class TEST --master yarn-cluster ./uber-TEST-1.0-SNAPSHOT.jar Spark 1.6.1 Version: Cloudera Express 5.6.0 16/03/22 09:16:33 INFO yarn.ApplicationMaster: Unregistering ApplicationMaster with FAILED (diag message: User class threw exception: java.lang.NoSuchMethodError: org.apache.spark.streaming.api.java.JavaPairDStream.mapWithState(Lorg/apache/spark/streaming/StateSpec;)Lorg/apache/spark/streaming/api/java/JavaMapWithStateDStream;)
updateStateByKey schedule time
Hi, I want to implement a time-out mechanism in de updateStateByKey(...) routine. But is there a way the retrieve the time of the start of the batch corresponding to the call to my updateStateByKey routines? Suppose the streaming has build up some delay then a System.currentTimeMillis() will not be the time of the time the batch was scheduled. I want to retrieve the job/task schedule time of the batch for which my updateStateByKey(..) routine is called. Is this possible? With kind regards, Michel Hubert
spark streaming performance
Hi, I've developed a POC Spark Streaming application. But it seems to perform better on my development machine than on our cluster. I submit it to yarn on our cloudera cluster. But my first question is more detailed: In de application UI (:4040) I see in the streaming section that the batch processing took 6 sec. Then when I look at the stages I indeed see a stage with duration 5s. For example: 1678 map at LogonAnalysis.scala:215+details 2015/07/09 09:17:00 5 s 50/50 173.5 KB But when I look into the details of state 1678 it tells me the duration was 14 ms and the aggregated metrics by executor has 1.0s as Task Time. What is responsible for the gap between 14 ms, 1s and 5 sec? Details for Stage 1678 * Total task time across all tasks: 0.8 s * Shuffle write: 173.5 KB / 2031 Show additional metrics Summary Metrics for 50 Completed Tasks Metric Min 25th percentile Median 75th percentile Max Duration 14 ms 14 ms 15 ms 15 ms 24 ms GC Time 0 ms 0 ms 0 ms 0 ms 0 ms Shuffle Write Size / Records 2.6 KB / 28 3.1 KB / 35 3.5 KB / 42 3.9 KB / 46 4.4 KB / 53 Aggregated Metrics by Executor Executor ID Address Task Time Total Tasks Failed Tasks Succeeded Tasks Shuffle Write Size / Records 2 :44231 1.0 s 50 0 50 173.5 KB / 2031
RE: Breaking lineage and reducing stages in Spark Streaming
Hi,
I was just wondering how you generated to second image with the charts.
What product?
From: Anand Nalya [mailto:anand.na...@gmail.com]
Sent: donderdag 9 juli 2015 11:48
To: spark users
Subject: Breaking lineage and reducing stages in Spark Streaming
Hi,
I've an application in which an rdd is being updated with tuples coming from
RDDs in a DStream with following pattern.
dstream.foreachRDD(rdd = {
myRDD = myRDD.union(rdd.filter(myfilter)).reduceByKey(_+_)
})
I'm using cache() and checkpointin to cache results. Over the time, the lineage
of myRDD keeps increasing and stages in each batch of dstream keeps increasing,
even though all the earlier stages are skipped. When the number of stages grow
big enough, the overall delay due to scheduling delay starts increasing. The
processing time for each batch is still fixed.
Following figures illustrate the problem:
Job execution: https://i.imgur.com/GVHeXH3.png?1
[Image removed by sender.]
Delays: https://i.imgur.com/1DZHydw.png?1
[Image removed by sender.]
Is there some pattern that I can use to avoid this?
Regards,
Anand
