Hi, Davies
The log shows that LogParser initializes and loads data once per executor,
thus I think singleton still works.
I change the code to
sc.textFile(inputPath)
.flatMap(line = LogParser.parseLine(line))
.foreach(_ = {})
to avoid shuffle IO, but it’s slower.
I thought it may be caused by
Is it possible to re-run your job with spark.eventLog.enabled to true, and
send the resulting logs to the list? Those have more per-task information
that can help diagnose this.
-Kay
On Wed, Jan 21, 2015 at 1:57 AM, Fengyun RAO raofeng...@gmail.com wrote:
btw: Shuffle Write(11 GB) mean 11 GB
On Tue, Jan 20, 2015 at 11:13 PM, Fengyun RAO raofeng...@gmail.com wrote:
the LogParser instance is not serializable, and thus cannot be a broadcast,
You could create a empty LogParser object (it's serializable), then
load the data
in executor lazily.
Could you add some logging to LogParser to
thanks, Sean.
I don't quite understand you have *more *partitions across *more *workers.
It's within the same cluster, and the same data, thus I think the same
partition, the same workers.
we switched from spark 1.1 to 1.2, then it's 3x slower.
(We upgrade from CDH 5.2.1 to CDH 5.3, hence
maybe you mean different spark-submit script?
we also use the same spark-submit script, thus the same memory, cores, etc
configuration.
2015-01-21 15:45 GMT+08:00 Sean Owen so...@cloudera.com:
I don't know of any reason to think the singleton pattern doesn't work or
works differently. I
web UI because itcan bepossibly related to my case.
Thanks
Kevin
--- Original Message ---
Sender : Fengyun RAOraofeng...@gmail.com
Date : 2015-01-21 17:41 (GMT+09:00)
Title : Re: spark 1.2 three times slower than spark 1.1, why?
maybe you mean different spark-submit script
I don't know how to debug distributed application, any tools or suggestion?
but from spark web UI,
the GC time (~0.1 s), Shuffle Write(11 GB) are similar for spark 1.1 and
1.2.
there are no Shuffle Read and Spill.
The only difference is Duration
DurationMin25th percentileMedian75th
* : Re: spark 1.2 three times slower than spark 1.1, why?
maybe you mean different spark-submit script?
we also use the same spark-submit script, thus the same memory, cores,
etc configuration.
2015-01-21 15:45 GMT+08:00 Sean Owen so...@cloudera.com:
I don't know of any reason to think
btw: Shuffle Write(11 GB) mean 11 GB per Executor, for each task, it's ~40
MB
2015-01-21 17:53 GMT+08:00 Fengyun RAO raofeng...@gmail.com:
I don't know how to debug distributed application, any tools or suggestion?
but from spark web UI,
the GC time (~0.1 s), Shuffle Write(11 GB) are
To force one instance per executor, you could explicitly subclass
FlatMapFunction and have it lazy-create your parser in the subclass
constructor. You might also want to try RDD#mapPartitions() (instead of
RDD#flatMap() if you want one instance per partition. This approach worked
well for me
Thanks, Paul,
I don’t understand how subclass FlatMapFunction helps, could you show a
sample code?
We need one instance per executor, not per partition, thus mapPartitions()
doesn’t help.
2015-01-21 16:07 GMT+08:00 Paul Wais paulw...@gmail.com:
To force one instance per executor, you could
the LogParser instance is not serializable, and thus cannot be a broadcast,
what’s worse, it contains an LRU cache, which is essential to the
performance, and we would like to share among all the tasks on the same
node.
If it is the case, what’s the recommended way to share a variable among all
I don't know of any reason to think the singleton pattern doesn't work or
works differently. I wonder if, for example, task scheduling is different
in 1.2 and you have more partitions across more workers and so are loading
more copies more slowly into your singletons.
On Jan 21, 2015 7:13 AM,
Currently we are migrating from spark 1.1 to spark 1.2, but found the
program 3x slower, with nothing else changed.
note: our program in spark 1.1 has successfully processed a whole year
data, quite stable.
the main script is as below
sc.textFile(inputPath)
.flatMap(line =
Maybe some change related to serialize the closure cause LogParser is
not a singleton any more, then it is initialized for every task.
Could you change it to a Broadcast?
On Tue, Jan 20, 2015 at 10:39 PM, Fengyun RAO raofeng...@gmail.com wrote:
Currently we are migrating from spark 1.1 to spark
15 matches
Mail list logo
