It looks like a number of doc pages can be improved by referencing some changes made recently.
With the inclusion of the needed jars for s3a with Drill, there is no longer a need to download jets3t [1]. In addition to setting your credentials, this option for allowing more concurrent connections (necessary to allow reads of wider parquet files) can also be set in this block instead of a core-site.xml file [2]. This config block can actually be used to set any filesystem properties. Some of these are custom to a particular filesystem like S3, but a number of them are used by a variety of implementations of the HDFS interface. Any properties like these [3] should be able to be set in this config block. [1] - https://drill.apache.org/blog/2014/12/09/running-sql-queries-on-amazon-s3/ [2] - https://drill.apache.org/docs/s3-storage-plugin/#quering-parquet-format-files-on-s3 [3] - https://hadoop.apache.org/docs/r2.7.2/hadoop-project-dist/hadoop-common/core-default.xml Jason Altekruse Software Engineer at Dremio Apache Drill Committer On Wed, Apr 20, 2016 at 9:52 AM, Abhishek Girish <abhishek.gir...@gmail.com> wrote: > Thanks Jason! I hadn't noticed the config property for S3. I tried this out > now, and feel it is a lot more easier now. > > And yes, we should definitely update the docs. There have been quite a few > threads related to S3 config. > > On Wed, Apr 20, 2016 at 8:19 AM, Jason Altekruse <ja...@dremio.com> wrote: > > > I don't believe there is any way in which a particular bucket has a > > property of being s3, s3n or s3a. As I understand it, this only change > the > > client library that is used to interface with S3. We have included the > jars > > necessary for s3a with Drill, which is the newest and most performant > > option available. > > > > I need to open a doc JIRA for this, but there is one way in which the s3 > > experience was improved recently to prevent the need to restart Drill to > > add your S3 credentials. When you create a connection to an S3 bucket, > you > > can now specify your credentials in a property named "config" in the > > storage plugin. This allows you to set any filesystem properties, which > we > > previously was only possible to set with a core-site.xml file on the > > classpath when starting Drill. > > > > Example: > > { > > "type": "file", > > "enabled": true, > > "connection": "s3a://address.of.your.bucket/", > > "config": { > > "fs.s3a.access.key": "<PUT YOUR ACCESS KEY HERE>", > > "fs.s3a.secret.key": "<PUT YOUR SECRET KEY HERE>" > > }, > > "workspaces": { > > "root": { > > "location": "/", > > "writable": false, > > "defaultInputFormat": null > > } > > }, > > "formats": { > > "psv": { > > "type": "text", > > "extensions": [ > > "tbl" > > ], > > "delimiter": "|" > > }, ... > > > > > > Jason Altekruse > > Software Engineer at Dremio > > Apache Drill Committer > > > > On Wed, Apr 20, 2016 at 7:40 AM, Nick Monetta <ni...@inrix.com> wrote: > > > > > Hi, > > > Does Drill v1.6 still support s3n connections or just s3a? > > > > > > I have a s3n S3 bucket that I'm trying to connect to and it will not > > work. > > > My config is: > > > > > > { > > > "type": "file", > > > "enabled": true, > > > "connection": "s3n://inrixprod-tapp/", > > > "workspaces": { > > > "root": { > > > "location": "/", > > > "writable": false, > > > "defaultInputFormat": null > > > }, > > > > > > Nick Monetta | INRIX |ni...@inrix.com |Movement Intelligence | > > > www.inrix.com | mobile +1 646-248-4105 | > > > > > > > > > > > >
