[jira] [Commented] (HADOOP-16921) NPE in s3a byte buffer block upload
[ https://issues.apache.org/jira/browse/HADOOP-16921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17090707#comment-17090707 ] Steve Loughran commented on HADOOP-16921: - happy for you to take up. * anything related to mark/position in the AWS SDK is usually caused by upload failures and retries, usually very intermittent and so hard o track down. * I'd start with adding some verifyOpen() checks on this and related methods, and add a check for the argument being null. Maybe a null pointer came back from the bytebuffer factory in some failure mode > NPE in s3a byte buffer block upload > --- > > Key: HADOOP-16921 > URL: https://issues.apache.org/jira/browse/HADOOP-16921 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.3.0 >Reporter: Steve Loughran >Priority: Minor > > NPE in s3a upload when fs.s3a.fast.upload.buffer = bytebuffer -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16921) NPE in s3a byte buffer block upload
[ https://issues.apache.org/jira/browse/HADOOP-16921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17090702#comment-17090702 ] Mukund Thakur commented on HADOOP-16921: Can I pick this up? Or you have already started working on this ? [~ste...@apache.org] > NPE in s3a byte buffer block upload > --- > > Key: HADOOP-16921 > URL: https://issues.apache.org/jira/browse/HADOOP-16921 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.3.0 >Reporter: Steve Loughran >Priority: Minor > > NPE in s3a upload when fs.s3a.fast.upload.buffer = bytebuffer -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16921) NPE in s3a byte buffer block upload
[
https://issues.apache.org/jira/browse/HADOOP-16921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17057746#comment-17057746
]
Steve Loughran commented on HADOOP-16921:
-
{code}
java.io.IOException: regular upload failed: java.lang.NullPointerException at
org.apache.hadoop.fs.s3a.S3AUtils.extractException(S3AUtils.java:338) at
org.apache.hadoop.fs.s3a.S3ABlockOutputStream.putObject(S3ABlockOutputStream.java:454)
at
org.apache.hadoop.fs.s3a.S3ABlockOutputStream.close(S3ABlockOutputStream.java:366)
at
org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:72)
at org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:101)
at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:70) at
org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:129) at
org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:415) at
org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:387) at
org.apache.hadoop.hive.common.FileUtils.copy(FileUtils.java:666) at
org.apache.hadoop.hive.common.FileUtils.copy(FileUtils.java:633) at
org.apache.hadoop.hive.ql.metadata.Hive.mvFile(Hive.java:4436) at
org.apache.hadoop.hive.ql.metadata.Hive.access$100(Hive.java:221) at
org.apache.hadoop.hive.ql.metadata.Hive$5.call(Hive.java:4296) ... 5 moreCaused
by: java.lang.NullPointerException at
org.apache.hadoop.fs.s3a.S3ADataBlocks$ByteBufferBlockFactory$ByteBufferBlock$ByteBufferInputStream.position(S3ADataBlocks.java:708)
at
org.apache.hadoop.fs.s3a.S3ADataBlocks$ByteBufferBlockFactory$ByteBufferBlock$ByteBufferInputStream.mark(S3ADataBlocks.java:721)
at
com.amazonaws.internal.SdkFilterInputStream.mark(SdkFilterInputStream.java:114)
at
com.amazonaws.internal.SdkFilterInputStream.mark(SdkFilterInputStream.java:114)
at
com.amazonaws.util.LengthCheckInputStream.mark(LengthCheckInputStream.java:116)
at
com.amazonaws.internal.SdkFilterInputStream.mark(SdkFilterInputStream.java:114)
at
com.amazonaws.internal.SdkFilterInputStream.mark(SdkFilterInputStream.java:114)
at
com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1067)
at
com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:770)
at
com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:744)
at
com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:726)
at
com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:686)
at
com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:668)
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:532) at
com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:512) at
com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4920) at
com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4866) at
com.amazonaws.services.s3.AmazonS3Client.access$300(AmazonS3Client.java:389) at
com.amazonaws.services.s3.AmazonS3Client$PutObjectStrategy.invokeServiceCall(AmazonS3Client.java:5800)
at
com.amazonaws.services.s3.AmazonS3Client.uploadObject(AmazonS3Client.java:1789)
at com.amazonaws.services.s3.AmazonS3Client.putObject(AmazonS3Client.java:1749)
at
org.apache.hadoop.fs.s3a.S3AFileSystem.putObjectDirect(S3AFileSystem.java:2136)
at
org.apache.hadoop.fs.s3a.WriteOperationHelper.lambda$putObject$5(WriteOperationHelper.java:462)
at org.apache.hadoop.fs.s3a.Invoker.once(Invoker.java:110) at
org.apache.hadoop.fs.s3a.Invoker.lambda$retry$4(Invoker.java:315) at
org.apache.hadoop.fs.s3a.Invoker.retryUntranslated(Invoker.java:407) at
org.apache.hadoop.fs.s3a.Invoker.retry(Invoker.java:311) at
org.apache.hadoop.fs.s3a.Invoker.retry(Invoker.java:286) at
org.apache.hadoop.fs.s3a.WriteOperationHelper.retry(WriteOperationHelper.java:150)
at
org.apache.hadoop.fs.s3a.WriteOperationHelper.putObject(WriteOperationHelper.java:460)
at
org.apache.hadoop.fs.s3a.S3ABlockOutputStream.lambda$putObject$0(S3ABlockOutputStream.java:439)
at
org.apache.hadoop.util.SemaphoredDelegatingExecutor$CallableWithPermitRelease.call(SemaphoredDelegatingExecutor.java:219)
at
org.apache.hadoop.util.SemaphoredDelegatingExecutor$CallableWithPermitRelease.call(SemaphoredDelegatingExecutor.java:219)
at
com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:108)
at
com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:41)
at
com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:77)
... 3 more
{code}
> NPE in s3a byte buffer block upload
> ---
>
> Key: HADOOP-16921
> URL: https://issues.apache.org/jira/browse/HADOOP-16921
> Project: Hadoop Common
> Issue Type: Sub-task
