[jira] [Commented] (HADOOP-17338) Intermittent S3AInputStream failures: Premature end of Content-Length delimited message body etc
[ https://issues.apache.org/jira/browse/HADOOP-17338?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17309433#comment-17309433 ] Yongjun Zhang commented on HADOOP-17338: For the record, thanks [~ste...@apache.org] for merging the 2.10.x on 09/Feb/21 https://github.com/apache/hadoop/pull/2692 . Our platform has been free of these errors with the fix for some time. > Intermittent S3AInputStream failures: Premature end of Content-Length > delimited message body etc > > > Key: HADOOP-17338 > URL: https://issues.apache.org/jira/browse/HADOOP-17338 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.3.0 >Reporter: Yongjun Zhang >Assignee: Yongjun Zhang >Priority: Major > Labels: pull-request-available > Fix For: 3.3.1, 2.10.2 > > Attachments: HADOOP-17338.001.patch > > Time Spent: 5h 20m > Remaining Estimate: 0h > > We are seeing the following two kinds of intermittent exceptions when using > S3AInputSteam: > 1. > {code:java} > Caused by: com.amazonaws.thirdparty.apache.http.ConnectionClosedException: > Premature end of Content-Length delimited message body (expected: 156463674; > received: 150001089 > at > com.amazonaws.thirdparty.apache.http.impl.io.ContentLengthInputStream.read(ContentLengthInputStream.java:178) > at > com.amazonaws.thirdparty.apache.http.conn.EofSensorInputStream.read(EofSensorInputStream.java:135) > at > com.amazonaws.internal.SdkFilterInputStream.read(SdkFilterInputStream.java:82) > at com.amazonaws.event.ProgressInputStream.read(ProgressInputStream.java:180) > at > com.amazonaws.internal.SdkFilterInputStream.read(SdkFilterInputStream.java:82) > at > com.amazonaws.services.s3.internal.S3AbortableInputStream.read(S3AbortableInputStream.java:125) > at > com.amazonaws.internal.SdkFilterInputStream.read(SdkFilterInputStream.java:82) > at > com.amazonaws.internal.SdkFilterInputStream.read(SdkFilterInputStream.java:82) > at > com.amazonaws.internal.SdkFilterInputStream.read(SdkFilterInputStream.java:82) > at com.amazonaws.event.ProgressInputStream.read(ProgressInputStream.java:180) > at > com.amazonaws.internal.SdkFilterInputStream.read(SdkFilterInputStream.java:82) > at > com.amazonaws.util.LengthCheckInputStream.read(LengthCheckInputStream.java:107) > at > com.amazonaws.internal.SdkFilterInputStream.read(SdkFilterInputStream.java:82) > at org.apache.hadoop.fs.s3a.S3AInputStream.read(S3AInputStream.java:181) > at java.io.DataInputStream.readFully(DataInputStream.java:195) > at java.io.DataInputStream.readFully(DataInputStream.java:169) > at > org.apache.parquet.hadoop.ParquetFileReader$ConsecutiveChunkList.readAll(ParquetFileReader.java:779) > at > org.apache.parquet.hadoop.ParquetFileReader.readNextRowGroup(ParquetFileReader.java:511) > at > org.apache.parquet.hadoop.InternalParquetRecordReader.checkRead(InternalParquetRecordReader.java:130) > at > org.apache.parquet.hadoop.InternalParquetRecordReader.nextKeyValue(InternalParquetRecordReader.java:214) > at > org.apache.parquet.hadoop.ParquetRecordReader.nextKeyValue(ParquetRecordReader.java:227) > at > org.apache.hadoop.hive.ql.io.parquet.read.ParquetRecordReaderWrapper.next(ParquetRecordReaderWrapper.java:208) > at > org.apache.hadoop.hive.ql.io.parquet.read.ParquetRecordReaderWrapper.next(ParquetRecordReaderWrapper.java:63) > at > org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.doNext(HiveContextAwareRecordReader.java:350) > ... 15 more > {code} > 2. > {code:java} > Caused by: javax.net.ssl.SSLException: SSL peer shut down incorrectly > at sun.security.ssl.InputRecord.readV3Record(InputRecord.java:596) > at sun.security.ssl.InputRecord.read(InputRecord.java:532) > at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:990) > at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:948) > at sun.security.ssl.AppInputStream.read(AppInputStream.java:105) > at > com.amazonaws.thirdparty.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137) > at > com.amazonaws.thirdparty.apache.http.impl.io.SessionInputBufferImpl.read(SessionInputBufferImpl.java:198) > at > com.amazonaws.thirdparty.apache.http.impl.io.ContentLengthInputStream.read(ContentLengthInputStream.java:176) > at > com.amazonaws.thirdparty.apache.http.conn.EofSensorInputStream.read(EofSensorInputStream.java:135) > at > com.amazonaws.internal.SdkFilterInputStream.read(SdkFilterInputStream.java:82) > at com.amazonaws.event.ProgressInputStream.read(ProgressInputStream.java:180) > at > com.amazonaws.internal.SdkFilterInputStream.read(SdkFilterInputStream.java:82) > at > com.amazonaws.internal.SdkFilterInputStream.read(SdkFilterInputStream.java:82) >
[jira] [Commented] (HADOOP-17338) Intermittent S3AInputStream failures: Premature end of Content-Length delimited message body etc
[ https://issues.apache.org/jira/browse/HADOOP-17338?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17309155#comment-17309155 ] Yongjun Zhang commented on HADOOP-17338: Sorry [~ste...@apache.org] I missed your message above. Have you created a new jira? thanks. > Intermittent S3AInputStream failures: Premature end of Content-Length > delimited message body etc > > > Key: HADOOP-17338 > URL: https://issues.apache.org/jira/browse/HADOOP-17338 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.3.0 >Reporter: Yongjun Zhang >Assignee: Yongjun Zhang >Priority: Major > Labels: pull-request-available > Fix For: 3.3.1, 2.10.2 > > Attachments: HADOOP-17338.001.patch > > Time Spent: 5h 20m > Remaining Estimate: 0h > > We are seeing the following two kinds of intermittent exceptions when using > S3AInputSteam: > 1. > {code:java} > Caused by: com.amazonaws.thirdparty.apache.http.ConnectionClosedException: > Premature end of Content-Length delimited message body (expected: 156463674; > received: 150001089 > at > com.amazonaws.thirdparty.apache.http.impl.io.ContentLengthInputStream.read(ContentLengthInputStream.java:178) > at > com.amazonaws.thirdparty.apache.http.conn.EofSensorInputStream.read(EofSensorInputStream.java:135) > at > com.amazonaws.internal.SdkFilterInputStream.read(SdkFilterInputStream.java:82) > at com.amazonaws.event.ProgressInputStream.read(ProgressInputStream.java:180) > at > com.amazonaws.internal.SdkFilterInputStream.read(SdkFilterInputStream.java:82) > at > com.amazonaws.services.s3.internal.S3AbortableInputStream.read(S3AbortableInputStream.java:125) > at > com.amazonaws.internal.SdkFilterInputStream.read(SdkFilterInputStream.java:82) > at > com.amazonaws.internal.SdkFilterInputStream.read(SdkFilterInputStream.java:82) > at > com.amazonaws.internal.SdkFilterInputStream.read(SdkFilterInputStream.java:82) > at com.amazonaws.event.ProgressInputStream.read(ProgressInputStream.java:180) > at > com.amazonaws.internal.SdkFilterInputStream.read(SdkFilterInputStream.java:82) > at > com.amazonaws.util.LengthCheckInputStream.read(LengthCheckInputStream.java:107) > at > com.amazonaws.internal.SdkFilterInputStream.read(SdkFilterInputStream.java:82) > at org.apache.hadoop.fs.s3a.S3AInputStream.read(S3AInputStream.java:181) > at java.io.DataInputStream.readFully(DataInputStream.java:195) > at java.io.DataInputStream.readFully(DataInputStream.java:169) > at > org.apache.parquet.hadoop.ParquetFileReader$ConsecutiveChunkList.readAll(ParquetFileReader.java:779) > at > org.apache.parquet.hadoop.ParquetFileReader.readNextRowGroup(ParquetFileReader.java:511) > at > org.apache.parquet.hadoop.InternalParquetRecordReader.checkRead(InternalParquetRecordReader.java:130) > at > org.apache.parquet.hadoop.InternalParquetRecordReader.nextKeyValue(InternalParquetRecordReader.java:214) > at > org.apache.parquet.hadoop.ParquetRecordReader.nextKeyValue(ParquetRecordReader.java:227) > at > org.apache.hadoop.hive.ql.io.parquet.read.ParquetRecordReaderWrapper.next(ParquetRecordReaderWrapper.java:208) > at > org.apache.hadoop.hive.ql.io.parquet.read.ParquetRecordReaderWrapper.next(ParquetRecordReaderWrapper.java:63) > at > org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.doNext(HiveContextAwareRecordReader.java:350) > ... 15 more > {code} > 2. > {code:java} > Caused by: javax.net.ssl.SSLException: SSL peer shut down incorrectly > at sun.security.ssl.InputRecord.readV3Record(InputRecord.java:596) > at sun.security.ssl.InputRecord.read(InputRecord.java:532) > at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:990) > at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:948) > at sun.security.ssl.AppInputStream.read(AppInputStream.java:105) > at > com.amazonaws.thirdparty.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137) > at > com.amazonaws.thirdparty.apache.http.impl.io.SessionInputBufferImpl.read(SessionInputBufferImpl.java:198) > at > com.amazonaws.thirdparty.apache.http.impl.io.ContentLengthInputStream.read(ContentLengthInputStream.java:176) > at > com.amazonaws.thirdparty.apache.http.conn.EofSensorInputStream.read(EofSensorInputStream.java:135) > at > com.amazonaws.internal.SdkFilterInputStream.read(SdkFilterInputStream.java:82) > at com.amazonaws.event.ProgressInputStream.read(ProgressInputStream.java:180) > at > com.amazonaws.internal.SdkFilterInputStream.read(SdkFilterInputStream.java:82) > at > com.amazonaws.internal.SdkFilterInputStream.read(SdkFilterInputStream.java:82) > at > com.amazonaws.internal.SdkFilterInputStream.read(SdkFilterInputStream.java:82) > at
[jira] [Commented] (HADOOP-17338) Intermittent S3AInputStream failures: Premature end of Content-Length delimited message body etc
[ https://issues.apache.org/jira/browse/HADOOP-17338?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17259742#comment-17259742 ] Steve Loughran commented on HADOOP-17338: - now, I just got the error {{Data read has a different length than the expected: dataLength=0; expectedLength=43236817}} on a test run with network problems. These should be converted to EOFExceptions, so retried as well, shouldn't they? Will file a new JIRA > Intermittent S3AInputStream failures: Premature end of Content-Length > delimited message body etc > > > Key: HADOOP-17338 > URL: https://issues.apache.org/jira/browse/HADOOP-17338 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.3.0 >Reporter: Yongjun Zhang >Assignee: Yongjun Zhang >Priority: Major > Labels: pull-request-available > Fix For: 3.3.1 > > Attachments: HADOOP-17338.001.patch > > Time Spent: 4h 20m > Remaining Estimate: 0h > > We are seeing the following two kinds of intermittent exceptions when using > S3AInputSteam: > 1. > {code:java} > Caused by: com.amazonaws.thirdparty.apache.http.ConnectionClosedException: > Premature end of Content-Length delimited message body (expected: 156463674; > received: 150001089 > at > com.amazonaws.thirdparty.apache.http.impl.io.ContentLengthInputStream.read(ContentLengthInputStream.java:178) > at > com.amazonaws.thirdparty.apache.http.conn.EofSensorInputStream.read(EofSensorInputStream.java:135) > at > com.amazonaws.internal.SdkFilterInputStream.read(SdkFilterInputStream.java:82) > at com.amazonaws.event.ProgressInputStream.read(ProgressInputStream.java:180) > at > com.amazonaws.internal.SdkFilterInputStream.read(SdkFilterInputStream.java:82) > at > com.amazonaws.services.s3.internal.S3AbortableInputStream.read(S3AbortableInputStream.java:125) > at > com.amazonaws.internal.SdkFilterInputStream.read(SdkFilterInputStream.java:82) > at > com.amazonaws.internal.SdkFilterInputStream.read(SdkFilterInputStream.java:82) > at > com.amazonaws.internal.SdkFilterInputStream.read(SdkFilterInputStream.java:82) > at com.amazonaws.event.ProgressInputStream.read(ProgressInputStream.java:180) > at > com.amazonaws.internal.SdkFilterInputStream.read(SdkFilterInputStream.java:82) > at > com.amazonaws.util.LengthCheckInputStream.read(LengthCheckInputStream.java:107) > at > com.amazonaws.internal.SdkFilterInputStream.read(SdkFilterInputStream.java:82) > at org.apache.hadoop.fs.s3a.S3AInputStream.read(S3AInputStream.java:181) > at java.io.DataInputStream.readFully(DataInputStream.java:195) > at java.io.DataInputStream.readFully(DataInputStream.java:169) > at > org.apache.parquet.hadoop.ParquetFileReader$ConsecutiveChunkList.readAll(ParquetFileReader.java:779) > at > org.apache.parquet.hadoop.ParquetFileReader.readNextRowGroup(ParquetFileReader.java:511) > at > org.apache.parquet.hadoop.InternalParquetRecordReader.checkRead(InternalParquetRecordReader.java:130) > at > org.apache.parquet.hadoop.InternalParquetRecordReader.nextKeyValue(InternalParquetRecordReader.java:214) > at > org.apache.parquet.hadoop.ParquetRecordReader.nextKeyValue(ParquetRecordReader.java:227) > at > org.apache.hadoop.hive.ql.io.parquet.read.ParquetRecordReaderWrapper.next(ParquetRecordReaderWrapper.java:208) > at > org.apache.hadoop.hive.ql.io.parquet.read.ParquetRecordReaderWrapper.next(ParquetRecordReaderWrapper.java:63) > at > org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.doNext(HiveContextAwareRecordReader.java:350) > ... 15 more > {code} > 2. > {code:java} > Caused by: javax.net.ssl.SSLException: SSL peer shut down incorrectly > at sun.security.ssl.InputRecord.readV3Record(InputRecord.java:596) > at sun.security.ssl.InputRecord.read(InputRecord.java:532) > at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:990) > at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:948) > at sun.security.ssl.AppInputStream.read(AppInputStream.java:105) > at > com.amazonaws.thirdparty.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137) > at > com.amazonaws.thirdparty.apache.http.impl.io.SessionInputBufferImpl.read(SessionInputBufferImpl.java:198) > at > com.amazonaws.thirdparty.apache.http.impl.io.ContentLengthInputStream.read(ContentLengthInputStream.java:176) > at > com.amazonaws.thirdparty.apache.http.conn.EofSensorInputStream.read(EofSensorInputStream.java:135) > at > com.amazonaws.internal.SdkFilterInputStream.read(SdkFilterInputStream.java:82) > at com.amazonaws.event.ProgressInputStream.read(ProgressInputStream.java:180) > at > com.amazonaws.internal.SdkFilterInputStream.read(SdkFilterInputStream.java:82) > at >
[jira] [Commented] (HADOOP-17338) Intermittent S3AInputStream failures: Premature end of Content-Length delimited message body etc
[ https://issues.apache.org/jira/browse/HADOOP-17338?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17258363#comment-17258363 ] Yongjun Zhang commented on HADOOP-17338: Happy new year and many thanks again [~ste...@apache.org]! I will work out a 2.10.x version asap. > Intermittent S3AInputStream failures: Premature end of Content-Length > delimited message body etc > > > Key: HADOOP-17338 > URL: https://issues.apache.org/jira/browse/HADOOP-17338 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.3.0 >Reporter: Yongjun Zhang >Assignee: Yongjun Zhang >Priority: Major > Labels: pull-request-available > Fix For: 3.3.1 > > Attachments: HADOOP-17338.001.patch > > Time Spent: 4h > Remaining Estimate: 0h > > We are seeing the following two kinds of intermittent exceptions when using > S3AInputSteam: > 1. > {code:java} > Caused by: com.amazonaws.thirdparty.apache.http.ConnectionClosedException: > Premature end of Content-Length delimited message body (expected: 156463674; > received: 150001089 > at > com.amazonaws.thirdparty.apache.http.impl.io.ContentLengthInputStream.read(ContentLengthInputStream.java:178) > at > com.amazonaws.thirdparty.apache.http.conn.EofSensorInputStream.read(EofSensorInputStream.java:135) > at > com.amazonaws.internal.SdkFilterInputStream.read(SdkFilterInputStream.java:82) > at com.amazonaws.event.ProgressInputStream.read(ProgressInputStream.java:180) > at > com.amazonaws.internal.SdkFilterInputStream.read(SdkFilterInputStream.java:82) > at > com.amazonaws.services.s3.internal.S3AbortableInputStream.read(S3AbortableInputStream.java:125) > at > com.amazonaws.internal.SdkFilterInputStream.read(SdkFilterInputStream.java:82) > at > com.amazonaws.internal.SdkFilterInputStream.read(SdkFilterInputStream.java:82) > at > com.amazonaws.internal.SdkFilterInputStream.read(SdkFilterInputStream.java:82) > at com.amazonaws.event.ProgressInputStream.read(ProgressInputStream.java:180) > at > com.amazonaws.internal.SdkFilterInputStream.read(SdkFilterInputStream.java:82) > at > com.amazonaws.util.LengthCheckInputStream.read(LengthCheckInputStream.java:107) > at > com.amazonaws.internal.SdkFilterInputStream.read(SdkFilterInputStream.java:82) > at org.apache.hadoop.fs.s3a.S3AInputStream.read(S3AInputStream.java:181) > at java.io.DataInputStream.readFully(DataInputStream.java:195) > at java.io.DataInputStream.readFully(DataInputStream.java:169) > at > org.apache.parquet.hadoop.ParquetFileReader$ConsecutiveChunkList.readAll(ParquetFileReader.java:779) > at > org.apache.parquet.hadoop.ParquetFileReader.readNextRowGroup(ParquetFileReader.java:511) > at > org.apache.parquet.hadoop.InternalParquetRecordReader.checkRead(InternalParquetRecordReader.java:130) > at > org.apache.parquet.hadoop.InternalParquetRecordReader.nextKeyValue(InternalParquetRecordReader.java:214) > at > org.apache.parquet.hadoop.ParquetRecordReader.nextKeyValue(ParquetRecordReader.java:227) > at > org.apache.hadoop.hive.ql.io.parquet.read.ParquetRecordReaderWrapper.next(ParquetRecordReaderWrapper.java:208) > at > org.apache.hadoop.hive.ql.io.parquet.read.ParquetRecordReaderWrapper.next(ParquetRecordReaderWrapper.java:63) > at > org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.doNext(HiveContextAwareRecordReader.java:350) > ... 15 more > {code} > 2. > {code:java} > Caused by: javax.net.ssl.SSLException: SSL peer shut down incorrectly > at sun.security.ssl.InputRecord.readV3Record(InputRecord.java:596) > at sun.security.ssl.InputRecord.read(InputRecord.java:532) > at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:990) > at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:948) > at sun.security.ssl.AppInputStream.read(AppInputStream.java:105) > at > com.amazonaws.thirdparty.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137) > at > com.amazonaws.thirdparty.apache.http.impl.io.SessionInputBufferImpl.read(SessionInputBufferImpl.java:198) > at > com.amazonaws.thirdparty.apache.http.impl.io.ContentLengthInputStream.read(ContentLengthInputStream.java:176) > at > com.amazonaws.thirdparty.apache.http.conn.EofSensorInputStream.read(EofSensorInputStream.java:135) > at > com.amazonaws.internal.SdkFilterInputStream.read(SdkFilterInputStream.java:82) > at com.amazonaws.event.ProgressInputStream.read(ProgressInputStream.java:180) > at > com.amazonaws.internal.SdkFilterInputStream.read(SdkFilterInputStream.java:82) > at > com.amazonaws.internal.SdkFilterInputStream.read(SdkFilterInputStream.java:82) > at > com.amazonaws.internal.SdkFilterInputStream.read(SdkFilterInputStream.java:82) > at
[jira] [Commented] (HADOOP-17338) Intermittent S3AInputStream failures: Premature end of Content-Length delimited message body etc
[ https://issues.apache.org/jira/browse/HADOOP-17338?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17252170#comment-17252170 ] Steve Loughran commented on HADOOP-17338: - ok, merged it into trunk after doing a local test run; cherrypicked into 3.3. (FWIW the test run took 45 minutes and I was worried that there was a regression, but it was actually google chrome had malloced 15GB of ram so everything was swapping) > Intermittent S3AInputStream failures: Premature end of Content-Length > delimited message body etc > > > Key: HADOOP-17338 > URL: https://issues.apache.org/jira/browse/HADOOP-17338 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.3.0 >Reporter: Yongjun Zhang >Assignee: Yongjun Zhang >Priority: Major > Labels: pull-request-available > Fix For: 3.3.1 > > Attachments: HADOOP-17338.001.patch > > Time Spent: 3h 50m > Remaining Estimate: 0h > > We are seeing the following two kinds of intermittent exceptions when using > S3AInputSteam: > 1. > {code:java} > Caused by: com.amazonaws.thirdparty.apache.http.ConnectionClosedException: > Premature end of Content-Length delimited message body (expected: 156463674; > received: 150001089 > at > com.amazonaws.thirdparty.apache.http.impl.io.ContentLengthInputStream.read(ContentLengthInputStream.java:178) > at > com.amazonaws.thirdparty.apache.http.conn.EofSensorInputStream.read(EofSensorInputStream.java:135) > at > com.amazonaws.internal.SdkFilterInputStream.read(SdkFilterInputStream.java:82) > at com.amazonaws.event.ProgressInputStream.read(ProgressInputStream.java:180) > at > com.amazonaws.internal.SdkFilterInputStream.read(SdkFilterInputStream.java:82) > at > com.amazonaws.services.s3.internal.S3AbortableInputStream.read(S3AbortableInputStream.java:125) > at > com.amazonaws.internal.SdkFilterInputStream.read(SdkFilterInputStream.java:82) > at > com.amazonaws.internal.SdkFilterInputStream.read(SdkFilterInputStream.java:82) > at > com.amazonaws.internal.SdkFilterInputStream.read(SdkFilterInputStream.java:82) > at com.amazonaws.event.ProgressInputStream.read(ProgressInputStream.java:180) > at > com.amazonaws.internal.SdkFilterInputStream.read(SdkFilterInputStream.java:82) > at > com.amazonaws.util.LengthCheckInputStream.read(LengthCheckInputStream.java:107) > at > com.amazonaws.internal.SdkFilterInputStream.read(SdkFilterInputStream.java:82) > at org.apache.hadoop.fs.s3a.S3AInputStream.read(S3AInputStream.java:181) > at java.io.DataInputStream.readFully(DataInputStream.java:195) > at java.io.DataInputStream.readFully(DataInputStream.java:169) > at > org.apache.parquet.hadoop.ParquetFileReader$ConsecutiveChunkList.readAll(ParquetFileReader.java:779) > at > org.apache.parquet.hadoop.ParquetFileReader.readNextRowGroup(ParquetFileReader.java:511) > at > org.apache.parquet.hadoop.InternalParquetRecordReader.checkRead(InternalParquetRecordReader.java:130) > at > org.apache.parquet.hadoop.InternalParquetRecordReader.nextKeyValue(InternalParquetRecordReader.java:214) > at > org.apache.parquet.hadoop.ParquetRecordReader.nextKeyValue(ParquetRecordReader.java:227) > at > org.apache.hadoop.hive.ql.io.parquet.read.ParquetRecordReaderWrapper.next(ParquetRecordReaderWrapper.java:208) > at > org.apache.hadoop.hive.ql.io.parquet.read.ParquetRecordReaderWrapper.next(ParquetRecordReaderWrapper.java:63) > at > org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.doNext(HiveContextAwareRecordReader.java:350) > ... 15 more > {code} > 2. > {code:java} > Caused by: javax.net.ssl.SSLException: SSL peer shut down incorrectly > at sun.security.ssl.InputRecord.readV3Record(InputRecord.java:596) > at sun.security.ssl.InputRecord.read(InputRecord.java:532) > at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:990) > at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:948) > at sun.security.ssl.AppInputStream.read(AppInputStream.java:105) > at > com.amazonaws.thirdparty.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137) > at > com.amazonaws.thirdparty.apache.http.impl.io.SessionInputBufferImpl.read(SessionInputBufferImpl.java:198) > at > com.amazonaws.thirdparty.apache.http.impl.io.ContentLengthInputStream.read(ContentLengthInputStream.java:176) > at > com.amazonaws.thirdparty.apache.http.conn.EofSensorInputStream.read(EofSensorInputStream.java:135) > at > com.amazonaws.internal.SdkFilterInputStream.read(SdkFilterInputStream.java:82) > at com.amazonaws.event.ProgressInputStream.read(ProgressInputStream.java:180) > at > com.amazonaws.internal.SdkFilterInputStream.read(SdkFilterInputStream.java:82) > at >
[jira] [Commented] (HADOOP-17338) Intermittent S3AInputStream failures: Premature end of Content-Length delimited message body etc
[ https://issues.apache.org/jira/browse/HADOOP-17338?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17252071#comment-17252071 ] Yongjun Zhang commented on HADOOP-17338: Many thanks to [~ste...@apache.org] for reviewing and committing the PR! > Intermittent S3AInputStream failures: Premature end of Content-Length > delimited message body etc > > > Key: HADOOP-17338 > URL: https://issues.apache.org/jira/browse/HADOOP-17338 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.3.0 >Reporter: Yongjun Zhang >Assignee: Yongjun Zhang >Priority: Major > Labels: pull-request-available > Fix For: 3.3.1 > > Attachments: HADOOP-17338.001.patch > > Time Spent: 3h 50m > Remaining Estimate: 0h > > We are seeing the following two kinds of intermittent exceptions when using > S3AInputSteam: > 1. > {code:java} > Caused by: com.amazonaws.thirdparty.apache.http.ConnectionClosedException: > Premature end of Content-Length delimited message body (expected: 156463674; > received: 150001089 > at > com.amazonaws.thirdparty.apache.http.impl.io.ContentLengthInputStream.read(ContentLengthInputStream.java:178) > at > com.amazonaws.thirdparty.apache.http.conn.EofSensorInputStream.read(EofSensorInputStream.java:135) > at > com.amazonaws.internal.SdkFilterInputStream.read(SdkFilterInputStream.java:82) > at com.amazonaws.event.ProgressInputStream.read(ProgressInputStream.java:180) > at > com.amazonaws.internal.SdkFilterInputStream.read(SdkFilterInputStream.java:82) > at > com.amazonaws.services.s3.internal.S3AbortableInputStream.read(S3AbortableInputStream.java:125) > at > com.amazonaws.internal.SdkFilterInputStream.read(SdkFilterInputStream.java:82) > at > com.amazonaws.internal.SdkFilterInputStream.read(SdkFilterInputStream.java:82) > at > com.amazonaws.internal.SdkFilterInputStream.read(SdkFilterInputStream.java:82) > at com.amazonaws.event.ProgressInputStream.read(ProgressInputStream.java:180) > at > com.amazonaws.internal.SdkFilterInputStream.read(SdkFilterInputStream.java:82) > at > com.amazonaws.util.LengthCheckInputStream.read(LengthCheckInputStream.java:107) > at > com.amazonaws.internal.SdkFilterInputStream.read(SdkFilterInputStream.java:82) > at org.apache.hadoop.fs.s3a.S3AInputStream.read(S3AInputStream.java:181) > at java.io.DataInputStream.readFully(DataInputStream.java:195) > at java.io.DataInputStream.readFully(DataInputStream.java:169) > at > org.apache.parquet.hadoop.ParquetFileReader$ConsecutiveChunkList.readAll(ParquetFileReader.java:779) > at > org.apache.parquet.hadoop.ParquetFileReader.readNextRowGroup(ParquetFileReader.java:511) > at > org.apache.parquet.hadoop.InternalParquetRecordReader.checkRead(InternalParquetRecordReader.java:130) > at > org.apache.parquet.hadoop.InternalParquetRecordReader.nextKeyValue(InternalParquetRecordReader.java:214) > at > org.apache.parquet.hadoop.ParquetRecordReader.nextKeyValue(ParquetRecordReader.java:227) > at > org.apache.hadoop.hive.ql.io.parquet.read.ParquetRecordReaderWrapper.next(ParquetRecordReaderWrapper.java:208) > at > org.apache.hadoop.hive.ql.io.parquet.read.ParquetRecordReaderWrapper.next(ParquetRecordReaderWrapper.java:63) > at > org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.doNext(HiveContextAwareRecordReader.java:350) > ... 15 more > {code} > 2. > {code:java} > Caused by: javax.net.ssl.SSLException: SSL peer shut down incorrectly > at sun.security.ssl.InputRecord.readV3Record(InputRecord.java:596) > at sun.security.ssl.InputRecord.read(InputRecord.java:532) > at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:990) > at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:948) > at sun.security.ssl.AppInputStream.read(AppInputStream.java:105) > at > com.amazonaws.thirdparty.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137) > at > com.amazonaws.thirdparty.apache.http.impl.io.SessionInputBufferImpl.read(SessionInputBufferImpl.java:198) > at > com.amazonaws.thirdparty.apache.http.impl.io.ContentLengthInputStream.read(ContentLengthInputStream.java:176) > at > com.amazonaws.thirdparty.apache.http.conn.EofSensorInputStream.read(EofSensorInputStream.java:135) > at > com.amazonaws.internal.SdkFilterInputStream.read(SdkFilterInputStream.java:82) > at com.amazonaws.event.ProgressInputStream.read(ProgressInputStream.java:180) > at > com.amazonaws.internal.SdkFilterInputStream.read(SdkFilterInputStream.java:82) > at > com.amazonaws.internal.SdkFilterInputStream.read(SdkFilterInputStream.java:82) > at > com.amazonaws.internal.SdkFilterInputStream.read(SdkFilterInputStream.java:82) > at
[jira] [Commented] (HADOOP-17338) Intermittent S3AInputStream failures: Premature end of Content-Length delimited message body etc
[ https://issues.apache.org/jira/browse/HADOOP-17338?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17229470#comment-17229470 ] Yongjun Zhang commented on HADOOP-17338: HI [~ste...@apache.org], Sorry for the delay. I tried to address your comments and created a PR. See linked. Below are the changes I made: # add finally block in closeStream() to set object to null, when wrapperStreamis set to null # changed LOG.debug to LOG.warn, because I think it's worthwhile to see the error when it happens. Wonder if it was set to debug due to too many of these logs? # added try block around wrappedStream.abort(), report the exception there if any and swallow it there, to possibly address HADOOP-17312 as you suggested. Would you please help taking a look again? Once you are ok with this diff, I will follow up with the integration tests. Thanks. > Intermittent S3AInputStream failures: Premature end of Content-Length > delimited message body etc > > > Key: HADOOP-17338 > URL: https://issues.apache.org/jira/browse/HADOOP-17338 > Project: Hadoop Common > Issue Type: Bug > Components: fs/s3 >Affects Versions: 3.3.0 >Reporter: Yongjun Zhang >Assignee: Yongjun Zhang >Priority: Major > Labels: pull-request-available > Attachments: HADOOP-17338.001.patch > > Time Spent: 10m > Remaining Estimate: 0h > > We are seeing the following two kinds of intermittent exceptions when using > S3AInputSteam: > 1. > {code} > Caused by: com.amazonaws.thirdparty.apache.http.ConnectionClosedException: > Premature end of Content-Length delimited message body (expected: 156463674; > received: 150001089 > at > com.amazonaws.thirdparty.apache.http.impl.io.ContentLengthInputStream.read(ContentLengthInputStream.java:178) > at > com.amazonaws.thirdparty.apache.http.conn.EofSensorInputStream.read(EofSensorInputStream.java:135) > at > com.amazonaws.internal.SdkFilterInputStream.read(SdkFilterInputStream.java:82) > at com.amazonaws.event.ProgressInputStream.read(ProgressInputStream.java:180) > at > com.amazonaws.internal.SdkFilterInputStream.read(SdkFilterInputStream.java:82) > at > com.amazonaws.services.s3.internal.S3AbortableInputStream.read(S3AbortableInputStream.java:125) > at > com.amazonaws.internal.SdkFilterInputStream.read(SdkFilterInputStream.java:82) > at > com.amazonaws.internal.SdkFilterInputStream.read(SdkFilterInputStream.java:82) > at > com.amazonaws.internal.SdkFilterInputStream.read(SdkFilterInputStream.java:82) > at com.amazonaws.event.ProgressInputStream.read(ProgressInputStream.java:180) > at > com.amazonaws.internal.SdkFilterInputStream.read(SdkFilterInputStream.java:82) > at > com.amazonaws.util.LengthCheckInputStream.read(LengthCheckInputStream.java:107) > at > com.amazonaws.internal.SdkFilterInputStream.read(SdkFilterInputStream.java:82) > at org.apache.hadoop.fs.s3a.S3AInputStream.read(S3AInputStream.java:181) > at java.io.DataInputStream.readFully(DataInputStream.java:195) > at java.io.DataInputStream.readFully(DataInputStream.java:169) > at > org.apache.parquet.hadoop.ParquetFileReader$ConsecutiveChunkList.readAll(ParquetFileReader.java:779) > at > org.apache.parquet.hadoop.ParquetFileReader.readNextRowGroup(ParquetFileReader.java:511) > at > org.apache.parquet.hadoop.InternalParquetRecordReader.checkRead(InternalParquetRecordReader.java:130) > at > org.apache.parquet.hadoop.InternalParquetRecordReader.nextKeyValue(InternalParquetRecordReader.java:214) > at > org.apache.parquet.hadoop.ParquetRecordReader.nextKeyValue(ParquetRecordReader.java:227) > at > org.apache.hadoop.hive.ql.io.parquet.read.ParquetRecordReaderWrapper.next(ParquetRecordReaderWrapper.java:208) > at > org.apache.hadoop.hive.ql.io.parquet.read.ParquetRecordReaderWrapper.next(ParquetRecordReaderWrapper.java:63) > at > org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.doNext(HiveContextAwareRecordReader.java:350) > ... 15 more > {code} > 2. > {code} > Caused by: javax.net.ssl.SSLException: SSL peer shut down incorrectly > at sun.security.ssl.InputRecord.readV3Record(InputRecord.java:596) > at sun.security.ssl.InputRecord.read(InputRecord.java:532) > at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:990) > at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:948) > at sun.security.ssl.AppInputStream.read(AppInputStream.java:105) > at > com.amazonaws.thirdparty.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137) > at > com.amazonaws.thirdparty.apache.http.impl.io.SessionInputBufferImpl.read(SessionInputBufferImpl.java:198) > at > com.amazonaws.thirdparty.apache.http.impl.io.ContentLengthInputStream.read(ContentLengthInputStream.java:176) > at >
[jira] [Commented] (HADOOP-17338) Intermittent S3AInputStream failures: Premature end of Content-Length delimited message body etc
[ https://issues.apache.org/jira/browse/HADOOP-17338?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17225132#comment-17225132 ] Yongjun Zhang commented on HADOOP-17338: Many thanks for the nice feedback [~ste...@apache.org]! I will try to get back as soon as possible. > Intermittent S3AInputStream failures: Premature end of Content-Length > delimited message body etc > > > Key: HADOOP-17338 > URL: https://issues.apache.org/jira/browse/HADOOP-17338 > Project: Hadoop Common > Issue Type: Bug > Components: fs/s3 >Affects Versions: 3.3.0 >Reporter: Yongjun Zhang >Assignee: Yongjun Zhang >Priority: Major > Attachments: HADOOP-17338.001.patch > > > We are seeing the following two kinds of intermittent exceptions when using > S3AInputSteam: > 1. > {code} > Caused by: com.amazonaws.thirdparty.apache.http.ConnectionClosedException: > Premature end of Content-Length delimited message body (expected: 156463674; > received: 150001089 > at > com.amazonaws.thirdparty.apache.http.impl.io.ContentLengthInputStream.read(ContentLengthInputStream.java:178) > at > com.amazonaws.thirdparty.apache.http.conn.EofSensorInputStream.read(EofSensorInputStream.java:135) > at > com.amazonaws.internal.SdkFilterInputStream.read(SdkFilterInputStream.java:82) > at com.amazonaws.event.ProgressInputStream.read(ProgressInputStream.java:180) > at > com.amazonaws.internal.SdkFilterInputStream.read(SdkFilterInputStream.java:82) > at > com.amazonaws.services.s3.internal.S3AbortableInputStream.read(S3AbortableInputStream.java:125) > at > com.amazonaws.internal.SdkFilterInputStream.read(SdkFilterInputStream.java:82) > at > com.amazonaws.internal.SdkFilterInputStream.read(SdkFilterInputStream.java:82) > at > com.amazonaws.internal.SdkFilterInputStream.read(SdkFilterInputStream.java:82) > at com.amazonaws.event.ProgressInputStream.read(ProgressInputStream.java:180) > at > com.amazonaws.internal.SdkFilterInputStream.read(SdkFilterInputStream.java:82) > at > com.amazonaws.util.LengthCheckInputStream.read(LengthCheckInputStream.java:107) > at > com.amazonaws.internal.SdkFilterInputStream.read(SdkFilterInputStream.java:82) > at org.apache.hadoop.fs.s3a.S3AInputStream.read(S3AInputStream.java:181) > at java.io.DataInputStream.readFully(DataInputStream.java:195) > at java.io.DataInputStream.readFully(DataInputStream.java:169) > at > org.apache.parquet.hadoop.ParquetFileReader$ConsecutiveChunkList.readAll(ParquetFileReader.java:779) > at > org.apache.parquet.hadoop.ParquetFileReader.readNextRowGroup(ParquetFileReader.java:511) > at > org.apache.parquet.hadoop.InternalParquetRecordReader.checkRead(InternalParquetRecordReader.java:130) > at > org.apache.parquet.hadoop.InternalParquetRecordReader.nextKeyValue(InternalParquetRecordReader.java:214) > at > org.apache.parquet.hadoop.ParquetRecordReader.nextKeyValue(ParquetRecordReader.java:227) > at > org.apache.hadoop.hive.ql.io.parquet.read.ParquetRecordReaderWrapper.next(ParquetRecordReaderWrapper.java:208) > at > org.apache.hadoop.hive.ql.io.parquet.read.ParquetRecordReaderWrapper.next(ParquetRecordReaderWrapper.java:63) > at > org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.doNext(HiveContextAwareRecordReader.java:350) > ... 15 more > {code} > 2. > {code} > Caused by: javax.net.ssl.SSLException: SSL peer shut down incorrectly > at sun.security.ssl.InputRecord.readV3Record(InputRecord.java:596) > at sun.security.ssl.InputRecord.read(InputRecord.java:532) > at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:990) > at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:948) > at sun.security.ssl.AppInputStream.read(AppInputStream.java:105) > at > com.amazonaws.thirdparty.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137) > at > com.amazonaws.thirdparty.apache.http.impl.io.SessionInputBufferImpl.read(SessionInputBufferImpl.java:198) > at > com.amazonaws.thirdparty.apache.http.impl.io.ContentLengthInputStream.read(ContentLengthInputStream.java:176) > at > com.amazonaws.thirdparty.apache.http.conn.EofSensorInputStream.read(EofSensorInputStream.java:135) > at > com.amazonaws.internal.SdkFilterInputStream.read(SdkFilterInputStream.java:82) > at com.amazonaws.event.ProgressInputStream.read(ProgressInputStream.java:180) > at > com.amazonaws.internal.SdkFilterInputStream.read(SdkFilterInputStream.java:82) > at > com.amazonaws.internal.SdkFilterInputStream.read(SdkFilterInputStream.java:82) > at > com.amazonaws.internal.SdkFilterInputStream.read(SdkFilterInputStream.java:82) > at com.amazonaws.event.ProgressInputStream.read(ProgressInputStream.java:180) > at > com.amazonaws.internal.SdkFilterInputStream.read(SdkFilterInputStream.java:82) >
[jira] [Commented] (HADOOP-17338) Intermittent S3AInputStream failures: Premature end of Content-Length delimited message body etc
[ https://issues.apache.org/jira/browse/HADOOP-17338?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17224621#comment-17224621 ] Steve Loughran commented on HADOOP-17338: - well, this is interesting # can you stick it up on github as a PR, following the "submitter must run the integration tests and say where their bucket was" policy. # Re-opened HADOOP-17312 as the patch as is doesn't address it. What I want there is the S3A client to swallow _all_ exceptions surfacing in abort(). The patch here may address a common cause, but there may be others. # we should discard that object ref in abort(), just for completeness It makes sense to include a fix for HADOOP-17312 in here. > Intermittent S3AInputStream failures: Premature end of Content-Length > delimited message body etc > > > Key: HADOOP-17338 > URL: https://issues.apache.org/jira/browse/HADOOP-17338 > Project: Hadoop Common > Issue Type: Bug > Components: fs/s3 >Affects Versions: 3.3.0 >Reporter: Yongjun Zhang >Assignee: Yongjun Zhang >Priority: Major > Attachments: HADOOP-17338.001.patch > > > We are seeing the following two kinds of intermittent exceptions when using > S3AInputSteam: > 1. > {code} > Caused by: com.amazonaws.thirdparty.apache.http.ConnectionClosedException: > Premature end of Content-Length delimited message body (expected: 156463674; > received: 150001089 > at > com.amazonaws.thirdparty.apache.http.impl.io.ContentLengthInputStream.read(ContentLengthInputStream.java:178) > at > com.amazonaws.thirdparty.apache.http.conn.EofSensorInputStream.read(EofSensorInputStream.java:135) > at > com.amazonaws.internal.SdkFilterInputStream.read(SdkFilterInputStream.java:82) > at com.amazonaws.event.ProgressInputStream.read(ProgressInputStream.java:180) > at > com.amazonaws.internal.SdkFilterInputStream.read(SdkFilterInputStream.java:82) > at > com.amazonaws.services.s3.internal.S3AbortableInputStream.read(S3AbortableInputStream.java:125) > at > com.amazonaws.internal.SdkFilterInputStream.read(SdkFilterInputStream.java:82) > at > com.amazonaws.internal.SdkFilterInputStream.read(SdkFilterInputStream.java:82) > at > com.amazonaws.internal.SdkFilterInputStream.read(SdkFilterInputStream.java:82) > at com.amazonaws.event.ProgressInputStream.read(ProgressInputStream.java:180) > at > com.amazonaws.internal.SdkFilterInputStream.read(SdkFilterInputStream.java:82) > at > com.amazonaws.util.LengthCheckInputStream.read(LengthCheckInputStream.java:107) > at > com.amazonaws.internal.SdkFilterInputStream.read(SdkFilterInputStream.java:82) > at org.apache.hadoop.fs.s3a.S3AInputStream.read(S3AInputStream.java:181) > at java.io.DataInputStream.readFully(DataInputStream.java:195) > at java.io.DataInputStream.readFully(DataInputStream.java:169) > at > org.apache.parquet.hadoop.ParquetFileReader$ConsecutiveChunkList.readAll(ParquetFileReader.java:779) > at > org.apache.parquet.hadoop.ParquetFileReader.readNextRowGroup(ParquetFileReader.java:511) > at > org.apache.parquet.hadoop.InternalParquetRecordReader.checkRead(InternalParquetRecordReader.java:130) > at > org.apache.parquet.hadoop.InternalParquetRecordReader.nextKeyValue(InternalParquetRecordReader.java:214) > at > org.apache.parquet.hadoop.ParquetRecordReader.nextKeyValue(ParquetRecordReader.java:227) > at > org.apache.hadoop.hive.ql.io.parquet.read.ParquetRecordReaderWrapper.next(ParquetRecordReaderWrapper.java:208) > at > org.apache.hadoop.hive.ql.io.parquet.read.ParquetRecordReaderWrapper.next(ParquetRecordReaderWrapper.java:63) > at > org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.doNext(HiveContextAwareRecordReader.java:350) > ... 15 more > {code} > 2. > {code} > Caused by: javax.net.ssl.SSLException: SSL peer shut down incorrectly > at sun.security.ssl.InputRecord.readV3Record(InputRecord.java:596) > at sun.security.ssl.InputRecord.read(InputRecord.java:532) > at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:990) > at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:948) > at sun.security.ssl.AppInputStream.read(AppInputStream.java:105) > at > com.amazonaws.thirdparty.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137) > at > com.amazonaws.thirdparty.apache.http.impl.io.SessionInputBufferImpl.read(SessionInputBufferImpl.java:198) > at > com.amazonaws.thirdparty.apache.http.impl.io.ContentLengthInputStream.read(ContentLengthInputStream.java:176) > at > com.amazonaws.thirdparty.apache.http.conn.EofSensorInputStream.read(EofSensorInputStream.java:135) > at > com.amazonaws.internal.SdkFilterInputStream.read(SdkFilterInputStream.java:82) > at com.amazonaws.event.ProgressInputStream.read(ProgressInputStream.java:180) > at >