Hi users, we have a bunch of pyspark jobs that are using S3 for loading / intermediate steps and final output of parquet files.
We're running into the following issues on a semi regular basis: * These are intermittent errors, IE we have about 300 jobs that run nightly... And a fairly random but small-ish percentage of them fail with the following classes of errors. *S3 write errors* > "ERROR Utils: Aborting task > com.amazonaws.services.s3.model.AmazonS3Exception: Status Code: 404, AWS > Service: Amazon S3, AWS Request ID: 2D3RP, AWS Error Code: null, AWS Error > Message: Not Found, S3 Extended Request ID: BlaBlahEtc=" > > "Py4JJavaError: An error occurred while calling o43.parquet. > : com.amazonaws.services.s3.model.MultiObjectDeleteException: Status Code: > 0, AWS Service: null, AWS Request ID: null, AWS Error Code: null, AWS Error > Message: One or more objects could not be deleted, S3 Extended Request ID: > null" *S3 Read Errors:* > [Stage 1:=================================================> (27 + 4) > / 31]17/05/10 16:25:23 ERROR Executor: Exception in task 10.0 in stage 1.0 > (TID 11) > java.net.SocketException: Connection reset > at java.net.SocketInputStream.read(SocketInputStream.java:196) > at java.net.SocketInputStream.read(SocketInputStream.java:122) > at sun.security.ssl.InputRecord.readFully(InputRecord.java:442) > at sun.security.ssl.InputRecord.readV3Record(InputRecord.java:554) > at sun.security.ssl.InputRecord.read(InputRecord.java:509) > at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:927) > at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:884) > at sun.security.ssl.AppInputStream.read(AppInputStream.java:102) > at > org.apache.http.impl.io.AbstractSessionInputBuffer.read(AbstractSessionInputBuffer.java:198) > at > org.apache.http.impl.io.ContentLengthInputStream.read(ContentLengthInputStream.java:178) > at > org.apache.http.impl.io.ContentLengthInputStream.read(ContentLengthInputStream.java:200) > at > org.apache.http.impl.io.ContentLengthInputStream.close(ContentLengthInputStream.java:103) > at > org.apache.http.conn.BasicManagedEntity.streamClosed(BasicManagedEntity.java:168) > at > org.apache.http.conn.EofSensorInputStream.checkClose(EofSensorInputStream.java:228) > at > org.apache.http.conn.EofSensorInputStream.close(EofSensorInputStream.java:174) > at java.io.FilterInputStream.close(FilterInputStream.java:181) > at java.io.FilterInputStream.close(FilterInputStream.java:181) > at java.io.FilterInputStream.close(FilterInputStream.java:181) > at java.io.FilterInputStream.close(FilterInputStream.java:181) > at com.amazonaws.services.s3.model.S3Object.close(S3Object.java:203) > at org.apache.hadoop.fs.s3a.S3AInputStream.close(S3AInputStream.java:187) We have literally tons of logs we can add but it would make the email unwieldy big. If it would be helpful I'll drop them in a pastebin or something. Our config is along the lines of: - spark-2.1.0-bin-hadoop2.7 - '--packages com.amazonaws:aws-java-sdk:1.10.34,org.apache.hadoop:hadoop-aws:2.6.0 pyspark-shell' Given the stack overflow / googling I've been doing I know we're not the only org with these issues but I haven't found a good set of solutions in those spaces yet. Thanks! Gary Lucas
