[ https://issues.apache.org/jira/browse/HADOOP-13122?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chris Nauroth updated HADOOP-13122: ----------------------------------- Attachment: HADOOP-13122.001.patch I'm attaching patch v001. This sets up the AWS SDK {{ClientConfiguration}} to prepend the Hadoop version number into the User-Agent. If a configuration property is specified, then it also prepends the property value in front of that. This information is additive, not a full replacement. The information included by the AWS SDK is still present. {{TestS3AConfiguration}} includes tests with and without the custom prefix. I refactored a bit to share code with other tests that need to use reflection to check AWS SDK internals. In addition to the unit tests, I ran manual testing live against S3, with the following in log4j.properties: {code} log4j.logger.org.apache.hadoop.fs.s3a=DEBUG log4j.logger.org.apache.http=DEBUG log4j.logger.org.apache.http.wire=ERROR {code} I could see the initialization log message announcing what it would use for the User-Agent. Then, the HTTP Components log messages showed that the User-Agent was in fact passing all the way through to the HTTP call: h3. Before Patch {code} hadoop fs -ls s3a://cnauroth-test-aws-s3a/ User-Agent: aws-sdk-java/1.10.6 Mac_OS_X/10.9.5 Java_HotSpot(TM)_64-Bit_Server_VM/24.65-b04/1.7.0_67 {code} h3. After Patch {code} hadoop fs -ls s3a://cnauroth-test-aws-s3a/ User-Agent: Hadoop 3.0.0-SNAPSHOT, aws-sdk-java/1.10.6 Mac_OS_X/10.9.5 Java_HotSpot(TM)_64-Bit_Server_VM/24.65-b04/1.7.0_67 {code} h3. After Patch/Custom Prefix: {code} hadoop fs -Dfs.s3a.user.agent.prefix=MyApp -ls s3a://cnauroth-test-aws-s3a/ User-Agent: MyApp, Hadoop 3.0.0-SNAPSHOT, aws-sdk-java/1.10.6 Mac_OS_X/10.9.5 Java_HotSpot(TM)_64-Bit_Server_VM/24.65-b04/1.7.0_67 {code} > Customize User-Agent header sent in HTTP requests by S3A. > --------------------------------------------------------- > > Key: HADOOP-13122 > URL: https://issues.apache.org/jira/browse/HADOOP-13122 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 > Reporter: Chris Nauroth > Assignee: Chris Nauroth > Attachments: HADOOP-13122.001.patch > > > S3A passes a User-Agent header to the S3 back-end. Right now, it uses the > default value set by the AWS SDK, so Hadoop HTTP traffic doesn't appear any > different from general AWS SDK traffic. If we customize the User-Agent > header, then it will enable better troubleshooting and analysis by AWS or > alternative providers of S3-like services. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org