Re: Spark Job Fails with Unknown Error writing to S3 from AWS EMR

2020-07-22 Thread Shriraj Bhardwaj
We faced this similar situation with jre 8u262 try reverting back...

On Thu, Jul 23, 2020, 5:18 AM koti reddy  wrote:

> Hi,
>
> Can someone help to resolve this issue?
> Thank you in advance.
>
> Error logs :
>
> java.io.EOFException: Unexpected EOF while trying to read response from server
>   at 
> org.apache.hadoop.hdfs.protocolPB.PBHelperClient.vintPrefixed(PBHelperClient.java:402)
>   at 
> org.apache.hadoop.hdfs.protocol.datatransfer.PipelineAck.readFields(PipelineAck.java:213)
>   at 
> org.apache.hadoop.hdfs.DataStreamer$ResponseProcessor.run(DataStreamer.java:1073)
> 20/07/22 22:43:37 WARN DataStreamer: Error Recovery for 
> BP-439833631-172.19.222.143-1595381416559:blk_1073742309_1498 in pipeline 
> [DatanodeInfoWithStorage[172.19.222.182:50010,DS-7783002b-d57a-43a3-9d91-9934e2d063f8,DISK],
>  
> DatanodeInfoWithStorage[172.19.223.27:50010,DS-1bc8cea7-9c28-4869-aada-55b7d0b0680c,DISK],
>  
> DatanodeInfoWithStorage[172.19.223.199:50010,DS-880d5121-16a2-465e-ad20-ca99f4287770,DISK]]:
>  datanode 
> 0(DatanodeInfoWithStorage[172.19.222.182:50010,DS-7783002b-d57a-43a3-9d91-9934e2d063f8,DISK])
>  is bad.
> 20/07/22 22:44:58 WARN DataStreamer: Exception for 
> BP-439833631-172.19.222.143-1595381416559:blk_1073742309_1499
> java.io.EOFException: Unexpected EOF while trying to read response from server
>   at 
> org.apache.hadoop.hdfs.protocolPB.PBHelperClient.vintPrefixed(PBHelperClient.java:402)
>   at 
> org.apache.hadoop.hdfs.protocol.datatransfer.PipelineAck.readFields(PipelineAck.java:213)
>   at 
> org.apache.hadoop.hdfs.DataStreamer$ResponseProcessor.run(DataStreamer.java:1073)
> 20/07/22 22:44:58 WARN DataStreamer: Error Recovery for 
> BP-439833631-172.19.222.143-1595381416559:blk_1073742309_1499 in pipeline 
> [DatanodeInfoWithStorage[172.19.223.27:50010,DS-1bc8cea7-9c28-4869-aada-55b7d0b0680c,DISK],
>  
> DatanodeInfoWithStorage[172.19.223.199:50010,DS-880d5121-16a2-465e-ad20-ca99f4287770,DISK],
>  
> DatanodeInfoWithStorage[172.19.222.180:50010,DS-40b1c81b-18d1-4d8d-ab49-11904f3dd23c,DISK]]:
>  datanode 
> 0(DatanodeInfoWithStorage[172.19.223.27:50010,DS-1bc8cea7-9c28-4869-aada-55b7d0b0680c,DISK])
>  is bad.
> 20/07/22 22:47:00 WARN DataStreamer: Exception for 
> BP-439833631-172.19.222.143-1595381416559:blk_1073742309_1500
> java.io.EOFException: Unexpected EOF while trying to read response from server
>   at 
> org.apache.hadoop.hdfs.protocolPB.PBHelperClient.vintPrefixed(PBHelperClient.java:402)
>   at 
> org.apache.hadoop.hdfs.protocol.datatransfer.PipelineAck.readFields(PipelineAck.java:213)
>   at 
> org.apache.hadoop.hdfs.DataStreamer$ResponseProcessor.run(DataStreamer.java:1073)
> 20/07/22 22:47:00 WARN DataStreamer: Error Recovery for 
> BP-439833631-172.19.222.143-1595381416559:blk_1073742309_1500 in pipeline 
> [DatanodeInfoWithStorage[172.19.223.199:50010,DS-880d5121-16a2-465e-ad20-ca99f4287770,DISK],
>  
> DatanodeInfoWithStorage[172.19.222.180:50010,DS-40b1c81b-18d1-4d8d-ab49-11904f3dd23c,DISK],
>  
> DatanodeInfoWithStorage[172.19.223.55:50010,DS-3e5dc677-cd1d-49fc-b50a-4b058ae298aa,DISK]]:
>  datanode 
> 0(DatanodeInfoWithStorage[172.19.223.199:50010,DS-880d5121-16a2-465e-ad20-ca99f4287770,DISK])
>  is bad.
> 20/07/22 22:47:03 INFO MemoryStore: Block broadcast_4 stored as values in 
> memory (estimated size 18.0 GB, free 24.5 GB)
>
> --
> Thanks,
> Koti Reddy Nusum,
> +1-(660) 541-5623.
>
>


Spark Job Fails with Unknown Error writing to S3 from AWS EMR

2020-07-22 Thread koti reddy
Hi,

Can someone help to resolve this issue?
Thank you in advance.

Error logs :

java.io.EOFException: Unexpected EOF while trying to read response from server
at 
org.apache.hadoop.hdfs.protocolPB.PBHelperClient.vintPrefixed(PBHelperClient.java:402)
at 
org.apache.hadoop.hdfs.protocol.datatransfer.PipelineAck.readFields(PipelineAck.java:213)
at 
org.apache.hadoop.hdfs.DataStreamer$ResponseProcessor.run(DataStreamer.java:1073)
20/07/22 22:43:37 WARN DataStreamer: Error Recovery for
BP-439833631-172.19.222.143-1595381416559:blk_1073742309_1498 in
pipeline 
[DatanodeInfoWithStorage[172.19.222.182:50010,DS-7783002b-d57a-43a3-9d91-9934e2d063f8,DISK],
DatanodeInfoWithStorage[172.19.223.27:50010,DS-1bc8cea7-9c28-4869-aada-55b7d0b0680c,DISK],
DatanodeInfoWithStorage[172.19.223.199:50010,DS-880d5121-16a2-465e-ad20-ca99f4287770,DISK]]:
datanode 
0(DatanodeInfoWithStorage[172.19.222.182:50010,DS-7783002b-d57a-43a3-9d91-9934e2d063f8,DISK])
is bad.
20/07/22 22:44:58 WARN DataStreamer: Exception for
BP-439833631-172.19.222.143-1595381416559:blk_1073742309_1499
java.io.EOFException: Unexpected EOF while trying to read response from server
at 
org.apache.hadoop.hdfs.protocolPB.PBHelperClient.vintPrefixed(PBHelperClient.java:402)
at 
org.apache.hadoop.hdfs.protocol.datatransfer.PipelineAck.readFields(PipelineAck.java:213)
at 
org.apache.hadoop.hdfs.DataStreamer$ResponseProcessor.run(DataStreamer.java:1073)
20/07/22 22:44:58 WARN DataStreamer: Error Recovery for
BP-439833631-172.19.222.143-1595381416559:blk_1073742309_1499 in
pipeline 
[DatanodeInfoWithStorage[172.19.223.27:50010,DS-1bc8cea7-9c28-4869-aada-55b7d0b0680c,DISK],
DatanodeInfoWithStorage[172.19.223.199:50010,DS-880d5121-16a2-465e-ad20-ca99f4287770,DISK],
DatanodeInfoWithStorage[172.19.222.180:50010,DS-40b1c81b-18d1-4d8d-ab49-11904f3dd23c,DISK]]:
datanode 
0(DatanodeInfoWithStorage[172.19.223.27:50010,DS-1bc8cea7-9c28-4869-aada-55b7d0b0680c,DISK])
is bad.
20/07/22 22:47:00 WARN DataStreamer: Exception for
BP-439833631-172.19.222.143-1595381416559:blk_1073742309_1500
java.io.EOFException: Unexpected EOF while trying to read response from server
at 
org.apache.hadoop.hdfs.protocolPB.PBHelperClient.vintPrefixed(PBHelperClient.java:402)
at 
org.apache.hadoop.hdfs.protocol.datatransfer.PipelineAck.readFields(PipelineAck.java:213)
at 
org.apache.hadoop.hdfs.DataStreamer$ResponseProcessor.run(DataStreamer.java:1073)
20/07/22 22:47:00 WARN DataStreamer: Error Recovery for
BP-439833631-172.19.222.143-1595381416559:blk_1073742309_1500 in
pipeline 
[DatanodeInfoWithStorage[172.19.223.199:50010,DS-880d5121-16a2-465e-ad20-ca99f4287770,DISK],
DatanodeInfoWithStorage[172.19.222.180:50010,DS-40b1c81b-18d1-4d8d-ab49-11904f3dd23c,DISK],
DatanodeInfoWithStorage[172.19.223.55:50010,DS-3e5dc677-cd1d-49fc-b50a-4b058ae298aa,DISK]]:
datanode 
0(DatanodeInfoWithStorage[172.19.223.199:50010,DS-880d5121-16a2-465e-ad20-ca99f4287770,DISK])
is bad.
20/07/22 22:47:03 INFO MemoryStore: Block broadcast_4 stored as values
in memory (estimated size 18.0 GB, free 24.5 GB)

-- 
Thanks,
Koti Reddy Nusum,
+1-(660) 541-5623.