[ https://issues.apache.org/jira/browse/HADOOP-17337?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Steve Loughran resolved HADOOP-17337. ------------------------------------- Release Note: the s3a filesystem will link against the unshaded AWS s3 SDK. Making an application's dependencies consistent with that SDK is left as exercise. Note: native openssl is not supported as a socket factory in unshaded deployments. Resolution: Fixed > S3A NetworkBinding has a runtime class dependency on a third-party shaded > class > ------------------------------------------------------------------------------- > > Key: HADOOP-17337 > URL: https://issues.apache.org/jira/browse/HADOOP-17337 > Project: Hadoop Common > Issue Type: Bug > Components: fs/s3 > Affects Versions: 3.3.0 > Reporter: Chris Wensel > Priority: Blocker > Labels: pull-request-available > Fix For: 3.3.1 > > Time Spent: 2h > Remaining Estimate: 0h > > The hadoop-aws library has a dependency on > 'com.amazonaws':aws-java-sdk-bundle' which in turn is a fat jar of all AWS > SDK libraries and shaded dependencies. > > This dependency is 181MB. > > Some applications using the S3AFilesystem may be sensitive to having a large > footprint. For example, building an application using Parquet and bundled > with Docker. > > Typically, in prior Hadoop versions, the bundle was replaced by the specific > AWS SDK dependencies, dropping the overall footprint. > > In 3.3 (and maybe prior versions) this strategy does not work because of the > following exception: > {{java.lang.NoClassDefFoundError: > com/amazonaws/thirdparty/apache/http/conn/socket/ConnectionSocketFactory}} > {{ at > org.apache.hadoop.fs.s3a.S3AUtils.initProtocolSettings(S3AUtils.java:1335)}} > {{ at > org.apache.hadoop.fs.s3a.S3AUtils.initConnectionSettings(S3AUtils.java:1290)}} > {{ at org.apache.hadoop.fs.s3a.S3AUtils.createAwsConf(S3AUtils.java:1247)}} > {{ at > org.apache.hadoop.fs.s3a.DefaultS3ClientFactory.createS3Client(DefaultS3ClientFactory.java:61)}} > {{ at > org.apache.hadoop.fs.s3a.S3AFileSystem.bindAWSClient(S3AFileSystem.java:644)}} > {{ at > org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:390)}} > {{ at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3414)}} > {{ at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:158)}} > {{ at > org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3474)}} > {{ at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3442)}} > {{ at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:524)}} > -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org