[ https://issues.apache.org/jira/browse/HADOOP-18339?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17599017#comment-17599017 ]
ASF GitHub Bot commented on HADOOP-18339: ----------------------------------------- monthonk commented on code in PR #4669: URL: https://github.com/apache/hadoop/pull/4669#discussion_r960771439 ########## hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/impl/RequestFactoryImpl.java: ########## @@ -416,19 +419,22 @@ public PutObjectRequest newDirectoryMarkerRequest(String directory) { String key = directory.endsWith("/") ? directory : (directory + "/"); - // an input stream which is laways empty - final InputStream im = new InputStream() { + // an input stream which is always empty + final InputStream inputStream = new InputStream() { @Override public int read() throws IOException { return -1; } }; // preparation happens in here - final ObjectMetadata md = createObjectMetadata(0L, true); - md.setContentType(HeaderProcessing.CONTENT_TYPE_X_DIRECTORY); - PutObjectRequest putObjectRequest = - newPutObjectRequest(key, md, null, im); - return putObjectRequest; + final ObjectMetadata metadata = createObjectMetadata(0L, true); + metadata.setContentType(HeaderProcessing.CONTENT_TYPE_X_DIRECTORY); + + PutObjectRequest putObjectRequest = new PutObjectRequest(getBucket(), key, + inputStream, metadata); + setOptionalPutRequestParameters(putObjectRequest); + putObjectRequest.setCannedAcl(cannedACL); Review Comment: no, it was in `newPutObjectRequest(..)` before, but I extracted it here because calling that function will also set storage class to directory marker > S3A storage class option only picked up when buffering writes to disk > --------------------------------------------------------------------- > > Key: HADOOP-18339 > URL: https://issues.apache.org/jira/browse/HADOOP-18339 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 > Affects Versions: 3.3.9 > Reporter: Steve Loughran > Assignee: Monthon Klongklaew > Priority: Major > Labels: pull-request-available > Time Spent: 20m > Remaining Estimate: 0h > > when you switch s3a output stream buffering to heap or byte buffer, the > storage class option isn't added to the put request > {code} > <property> > <name>fs.s3a.fast.upload.buffer</name> > <value>bytebuffer</value> > </property> > {code} > and the ITestS3AStorageClass tests fail. > {code} > java.lang.AssertionError: [Storage class of object > s3a://stevel-london/test/testCreateAndCopyObjectWithStorageClassGlacier/file1] > > Expecting: > <null> > to be equal to: > <"glacier"> > ignoring case considerations > at > org.apache.hadoop.fs.s3a.ITestS3AStorageClass.assertObjectHasStorageClass(ITestS3AStorageClass.java:215) > at > org.apache.hadoop.fs.s3a.ITestS3AStorageClass.testCreateAndCopyObjectWithStorageClassGlacier(ITestS3AStorageClass.java:129) > {code} > we noticed this in a code review; the request factory only sets the option > when the source is a file, not memory. > proposed: parameterize the test suite on disk/byte buffer, then fix -- This message was sent by Atlassian Jira (v8.20.10#820010) --------------------------------------------------------------------- To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org