Re: Spark Job Fails with Unknown Error writing to S3 from AWS EMR

[Shriraj Bhardwaj]

We faced this similar situation with jre 8u262 try reverting back...

On Thu, Jul 23, 2020, 5:18 AM koti reddy  wrote:

> Hi,
>
> Can someone help to resolve this issue?
> Thank you in advance.
>
> Error logs :
>
> java.io.EOFException: Unexpected EOF while trying to read response from server
>   at 
> org.apache.hadoop.hdfs.protocolPB.PBHelperClient.vintPrefixed(PBHelperClient.java:402)
>   at 
> org.apache.hadoop.hdfs.protocol.datatransfer.PipelineAck.readFields(PipelineAck.java:213)
>   at 
> org.apache.hadoop.hdfs.DataStreamer$ResponseProcessor.run(DataStreamer.java:1073)
> 20/07/22 22:43:37 WARN DataStreamer: Error Recovery for 
> BP-439833631-172.19.222.143-1595381416559:blk_1073742309_1498 in pipeline 
> [DatanodeInfoWithStorage[172.19.222.182:50010,DS-7783002b-d57a-43a3-9d91-9934e2d063f8,DISK],
>  
> DatanodeInfoWithStorage[172.19.223.27:50010,DS-1bc8cea7-9c28-4869-aada-55b7d0b0680c,DISK],
>  
> DatanodeInfoWithStorage[172.19.223.199:50010,DS-880d5121-16a2-465e-ad20-ca99f4287770,DISK]]:
>  datanode 
> 0(DatanodeInfoWithStorage[172.19.222.182:50010,DS-7783002b-d57a-43a3-9d91-9934e2d063f8,DISK])
>  is bad.
> 20/07/22 22:44:58 WARN DataStreamer: Exception for 
> BP-439833631-172.19.222.143-1595381416559:blk_1073742309_1499
> java.io.EOFException: Unexpected EOF while trying to read response from server
>   at 
> org.apache.hadoop.hdfs.protocolPB.PBHelperClient.vintPrefixed(PBHelperClient.java:402)
>   at 
> org.apache.hadoop.hdfs.protocol.datatransfer.PipelineAck.readFields(PipelineAck.java:213)
>   at 
> org.apache.hadoop.hdfs.DataStreamer$ResponseProcessor.run(DataStreamer.java:1073)
> 20/07/22 22:44:58 WARN DataStreamer: Error Recovery for 
> BP-439833631-172.19.222.143-1595381416559:blk_1073742309_1499 in pipeline 
> [DatanodeInfoWithStorage[172.19.223.27:50010,DS-1bc8cea7-9c28-4869-aada-55b7d0b0680c,DISK],
>  
> DatanodeInfoWithStorage[172.19.223.199:50010,DS-880d5121-16a2-465e-ad20-ca99f4287770,DISK],
>  
> DatanodeInfoWithStorage[172.19.222.180:50010,DS-40b1c81b-18d1-4d8d-ab49-11904f3dd23c,DISK]]:
>  datanode 
> 0(DatanodeInfoWithStorage[172.19.223.27:50010,DS-1bc8cea7-9c28-4869-aada-55b7d0b0680c,DISK])
>  is bad.
> 20/07/22 22:47:00 WARN DataStreamer: Exception for 
> BP-439833631-172.19.222.143-1595381416559:blk_1073742309_1500
> java.io.EOFException: Unexpected EOF while trying to read response from server
>   at 
> org.apache.hadoop.hdfs.protocolPB.PBHelperClient.vintPrefixed(PBHelperClient.java:402)
>   at 
> org.apache.hadoop.hdfs.protocol.datatransfer.PipelineAck.readFields(PipelineAck.java:213)
>   at 
> org.apache.hadoop.hdfs.DataStreamer$ResponseProcessor.run(DataStreamer.java:1073)
> 20/07/22 22:47:00 WARN DataStreamer: Error Recovery for 
> BP-439833631-172.19.222.143-1595381416559:blk_1073742309_1500 in pipeline 
> [DatanodeInfoWithStorage[172.19.223.199:50010,DS-880d5121-16a2-465e-ad20-ca99f4287770,DISK],
>  
> DatanodeInfoWithStorage[172.19.222.180:50010,DS-40b1c81b-18d1-4d8d-ab49-11904f3dd23c,DISK],
>  
> DatanodeInfoWithStorage[172.19.223.55:50010,DS-3e5dc677-cd1d-49fc-b50a-4b058ae298aa,DISK]]:
>  datanode 
> 0(DatanodeInfoWithStorage[172.19.223.199:50010,DS-880d5121-16a2-465e-ad20-ca99f4287770,DISK])
>  is bad.
> 20/07/22 22:47:03 INFO MemoryStore: Block broadcast_4 stored as values in 
> memory (estimated size 18.0 GB, free 24.5 GB)
>
> --
> Thanks,
> Koti Reddy Nusum,
> +1-(660) 541-5623.
>
>

Spark ShutdownHook through python jobs.

[Shriraj Bhardwaj]

Hi,
we have spark jobs written totally in python similar to repo
https://github.com/AlexIoannides/pyspark-example-project,
we are using spark-submit to submit the application in the local mode, but
want to send metrics when the job ends (on SIGTERM as well), to do so we
need something similar to shutdown hook in java/scala, can somebody help
how to do it ? I couldn't fine the shutdown hook interface in pyspark ?
-- 
Regards,
Shriraj Bhardwaj


*कर्मण्येवाधिकारस्ते मा फलेषु कदाचन। मा कर्मफलहेतुर्भूर्मा ते
सङ्गोऽस्त्वकर्मणि॥ *