FWIW observed similar behavior in similar situation. Was able to work
around by forcefully committing one of the rdds right before the union
into cache, and forcing that by executing take(1). Nothing else ever
helped.
Seems like yet-undiscovered 1.2.x thing.
On Tue, Mar 17, 2015 at 4:21 PM, Eugen Cepoi <cepoi.eu...@gmail.com> wrote:
> Doing the reduceByKey without changing the number of partitions and then do
> a coalesce works.
> But the other version still hangs, without any information (while working
> with spark 1.1.1). The previous logs don't seem to be related to what
> happens.
> I don't think this is a memory issue as the GC time remains low and the
> shuffle read is small. My guess is that it might be related to a high number
> of initial partitions, but in that case shouldn't it fail for coalesce
> too...?
>
> Does anyone have an idea where to look at to find what the source of the
> problem is?
>
> Thanks,
> Eugen
>
> 2015-03-13 19:18 GMT+01:00 Eugen Cepoi <cepoi.eu...@gmail.com>:
>>
>> Hum increased it to 1024 but doesn't help still have the same problem :(
>>
>> 2015-03-13 18:28 GMT+01:00 Eugen Cepoi <cepoi.eu...@gmail.com>:
>>>
>>> The one by default 0.07 of executor memory. I'll try increasing it and
>>> post back the result.
>>>
>>> Thanks
>>>
>>> 2015-03-13 18:09 GMT+01:00 Ted Yu <yuzhih...@gmail.com>:
>>>>
>>>> Might be related: what's the value for
>>>> spark.yarn.executor.memoryOverhead ?
>>>>
>>>> See SPARK-6085
>>>>
>>>> Cheers
>>>>
>>>> On Fri, Mar 13, 2015 at 9:45 AM, Eugen Cepoi <cepoi.eu...@gmail.com>
>>>> wrote:
>>>>>
>>>>> Hi,
>>>>>
>>>>> I have a job that hangs after upgrading to spark 1.2.1 from 1.1.1.
>>>>> Strange thing, the exact same code does work (after upgrade) in the
>>>>> spark-shell. But this information might be misleading as it works with
>>>>> 1.1.1...
>>>>>
>>>>>
>>>>> The job takes as input two data sets:
>>>>> - rdd A of +170gb (with less it is hard to reproduce) and more than
>>>>> 11K partitions
>>>>> - rdd B of +100mb and 32 partitions
>>>>>
>>>>> I run it via EMR over YARN and use 4*m3.xlarge computing nodes. I am
>>>>> not sure the executor config is relevant here. Anyway I tried with
>>>>> multiple
>>>>> small executors with fewer ram and the inverse.
>>>>>
>>>>>
>>>>> The job basically does this:
>>>>> A.flatMap(...).union(B).keyBy(f).reduceByKey(..., 32).map(...).save
>>>>>
>>>>> After the flatMap rdd A size is much smaller similar to B.
>>>>>
>>>>> Configs I used to run this job:
>>>>>
>>>>> storage.memoryFraction: 0
>>>>> shuffle.memoryFraction: 0.5
>>>>>
>>>>> akka.timeout 500
>>>>> akka.frameSize 40
>>>>>
>>>>> // this one defines also the memory used by yarn master, but not sure
>>>>> if it needs to be important
>>>>> driver.memory 5g
>>>>> excutor.memory 4250m
>>>>>
>>>>> I have 7 executors with 2 cores.
>>>>>
>>>>> What happens:
>>>>> The job produces two stages: keyBy and save. The keyBy stage runs fine
>>>>> and produces a shuffle write of ~150mb. The save stage where the suffle
>>>>> read
>>>>> occurs hangs. Greater the initial dataset is more tasks hang.
>>>>>
>>>>> I did run it for much larger datasets with same config/cluster but
>>>>> without doing the union and it worked fine.
>>>>>
>>>>> Some more infos and logs:
>>>>>
>>>>> Amongst 4 nodes 1 finished all his tasks and the "running" ones are on
>>>>> the 3 other nodes. But not sure this is a good information (one node that
>>>>> completed all his work vs the others) as with some smaller dataset I
>>>>> manage
>>>>> to get only one hanging task.
>>>>>
>>>>> Here are the last parts of the executor logs that show some timeouts.
>>>>>
>>>>> An executor from node ip-10-182-98-220
>>>>>
>>>>> 15/03/13 15:43:10 INFO storage.ShuffleBlockFetcherIterator: Started 6
>>>>> remote fetches in 66 ms
>>>>> 15/03/13 15:58:44 WARN server.TransportChannelHandler: Exception in
>>>>> connection from /10.181.48.153:56806
>>>>> java.io.IOException: Connection timed out
>>>>>
>>>>>
>>>>> An executor from node ip-10-181-103-186
>>>>>
>>>>> 15/03/13 15:43:22 INFO storage.ShuffleBlockFetcherIterator: Started 6
>>>>> remote fetches in 20 ms
>>>>> 15/03/13 15:58:41 WARN server.TransportChannelHandler: Exception in
>>>>> connection from /10.182.98.220:38784
>>>>> java.io.IOException: Connection timed out
>>>>>
>>>>> An executor from node ip-10-181-48-153 (all the logs bellow belong this
>>>>> node)
>>>>>
>>>>> 15/03/13 15:43:24 INFO executor.Executor: Finished task 26.0 in stage
>>>>> 1.0 (TID 13860). 802 bytes result sent to driver
>>>>> 15/03/13 15:58:43 WARN server.TransportChannelHandler: Exception in
>>>>> connection from /10.181.103.186:46381
>>>>> java.io.IOException: Connection timed out
>>>>>
>>>>> Followed by many
>>>>>
>>>>> 15/03/13 15:58:43 ERROR server.TransportRequestHandler: Error sending
>>>>> result
>>>>> ChunkFetchSuccess{streamChunkId=StreamChunkId{streamId=2064203432016,
>>>>> chunkIndex=405},
>>>>> buffer=FileSegmentManagedBuffer{file=/mnt/var/lib/hadoop/tmp/nm-local-dir/usercache/hadoop/appcache/application_1426256247374_0002/spark-1659efcd-c6b6-4a12-894d-e869486d3d00/35/shuffle_0_9885_0.data,
>>>>> offset=8631, length=571}} to /10.181.103.186:46381; closing connection
>>>>> java.nio.channels.ClosedChannelException
>>>>>
>>>>> with last one being
>>>>>
>>>>> 15/03/13 15:58:43 ERROR server.TransportRequestHandler: Error sending
>>>>> result RpcResponse{requestId=7377187355282895939, response=[B@6fcd0014} to
>>>>> /10.181.103.186:46381; closing connection
>>>>> java.nio.channels.ClosedChannelException
>>>>>
>>>>>
>>>>> The executors from the node that finished his tasks doesn't show
>>>>> anything special.
>>>>>
>>>>> Note that I don't cache anything thus reduced the
>>>>> storage.memoryFraction to 0.
>>>>> I see some of those, but don't think they are related.
>>>>>
>>>>> 15/03/13 15:43:15 INFO storage.MemoryStore: Memory use = 0.0 B (blocks)
>>>>> + 0.0 B (scratch space shared across 0 thread(s)) = 0.0 B. Storage limit =
>>>>> 0.0 B.
>>>>>
>>>>>
>>>>> Sorry for the long email with maybe misleading infos, but I hope it
>>>>> might help to track down what happens and why it was working with spark
>>>>> 1.1.1.
>>>>>
>>>>> Thanks,
>>>>> Eugen
>>>>>
>>>>
>>>
>>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional commands, e-mail: user-h...@spark.apache.org
