No.
But I may be on to something.
I use Jedis to send data to Redis.
I used a ThreadLocal construct:
private static final ThreadLocal<Jedis> jedis = new ThreadLocal<Jedis>(){
@Override
protected Jedis initialValue()
{
return new Jedis("10.101.41.19",6379);
}
};
and then
.foreachRDD(new VoidFunction<JavaRDD<TopData>>() {
public void call(JavaRDD<TopData> rdd) throws Exception {
for (TopData t: rdd.take(top)) {
jedis …
}
May this resulted in a memory leak?
Van: Ted Yu [mailto:yuzhih...@gmail.com]
Verzonden: donderdag 24 maart 2016 15:15
Aan: Michel Hubert <mich...@phact.nl>
CC: user@spark.apache.org
Onderwerp: Re: apache spark errors
Do you have history server enabled ?
Posting your code snippet would help us understand your use case (and reproduce
the leak).
Thanks
On Thu, Mar 24, 2016 at 6:40 AM, Michel Hubert
<mich...@phact.nl<mailto:mich...@phact.nl>> wrote:
<dependencies>
<dependency> <!-- Spark dependency -->
<groupId>org.apache.spark</groupId>
<artifactId>spark-core_2.10</artifactId>
<version>1.6.1</version>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-streaming_2.10</artifactId>
<version>1.6.1</version>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-streaming-kafka_2.10</artifactId>
<version>1.6.1</version>
</dependency>
<dependency>
<groupId>org.elasticsearch</groupId>
<artifactId>elasticsearch</artifactId>
<version>2.2.0</version>
</dependency>
<dependency>
<groupId>org.apache.kafka</groupId>
<artifactId>kafka_2.10</artifactId>
<version>0.8.2.2</version>
</dependency>
<dependency>
<groupId>org.elasticsearch</groupId>
<artifactId>elasticsearch-spark_2.10</artifactId>
<version>2.2.0</version>
</dependency>
<dependency>
<groupId>redis.clients</groupId>
<artifactId>jedis</artifactId>
<version>2.8.0</version>
<type>jar</type>
<scope>compile</scope>
</dependency>
</dependencies>
How can I look at those tasks?
Van: Ted Yu [mailto:yuzhih...@gmail.com<mailto:yuzhih...@gmail.com>]
Verzonden: donderdag 24 maart 2016 14:33
Aan: Michel Hubert <mich...@phact.nl<mailto:mich...@phact.nl>>
CC: user@spark.apache.org<mailto:user@spark.apache.org>
Onderwerp: Re: apache spark errors
Which release of Spark are you using ?
Have you looked the tasks whose Ids were printed to see if there was more clue ?
Thanks
On Thu, Mar 24, 2016 at 6:12 AM, Michel Hubert
<mich...@phact.nl<mailto:mich...@phact.nl>> wrote:
HI,
I constantly get these errors:
0 [Executor task launch worker-15] ERROR org.apache.spark.executor.Executor
- Managed memory leak detected; size = 6564500 bytes, TID = 38969
310002 [Executor task launch worker-12] ERROR
org.apache.spark.executor.Executor - Managed memory leak detected; size =
5523550 bytes, TID = 43270
318445 [Executor task launch worker-12] ERROR
org.apache.spark.executor.Executor - Managed memory leak detected; size =
6879566 bytes, TID = 43408
388893 [Executor task launch worker-12] ERROR
org.apache.spark.executor.Executor - Managed memory leak detected; size =
5572546 bytes, TID = 44382
418186 [Executor task launch worker-13] ERROR
org.apache.spark.executor.Executor - Managed memory leak detected; size =
5289222 bytes, TID = 44795
488421 [Executor task launch worker-4] ERROR org.apache.spark.executor.Executor
- Managed memory leak detected; size = 8738142 bytes, TID = 45769
619276 [Executor task launch worker-4] ERROR org.apache.spark.executor.Executor
- Managed memory leak detected; size = 5759312 bytes, TID = 47571
632275 [Executor task launch worker-12] ERROR
org.apache.spark.executor.Executor - Managed memory leak detected; size =
5602240 bytes, TID = 47709
644989 [Executor task launch worker-13] ERROR
org.apache.spark.executor.Executor - Managed memory leak detected; size =
5326260 bytes, TID = 47863
720701 [Executor task launch worker-12] ERROR
org.apache.spark.executor.Executor - Managed memory leak detected; size =
5399578 bytes, TID = 48959
1147961 [Executor task launch worker-16] ERROR
org.apache.spark.executor.Executor - Managed memory leak detected; size =
5251872 bytes, TID = 54922
How can I fix this?
With kind regard,
Michel
