Created DRILL-4621 <https://issues.apache.org/jira/browse/DRILL-4621> to track doc change request.
Thanks, Bridget On Wed, Apr 20, 2016 at 10:10 AM, Jason Altekruse <[email protected]> wrote: > It looks like a number of doc pages can be improved by referencing some > changes made recently. > > With the inclusion of the needed jars for s3a with Drill, there is no > longer a need to download jets3t [1]. In addition to setting your > credentials, this option for allowing more concurrent connections > (necessary to allow reads of wider parquet files) can also be set in this > block instead of a core-site.xml file [2]. > > This config block can actually be used to set any filesystem properties. > Some of these are custom to a particular filesystem like S3, but a number > of them are used by a variety of implementations of the HDFS interface. Any > properties like these [3] should be able to be set in this config block. > > [1] - > https://drill.apache.org/blog/2014/12/09/running-sql-queries-on-amazon-s3/ > [2] - > > https://drill.apache.org/docs/s3-storage-plugin/#quering-parquet-format-files-on-s3 > [3] - > > https://hadoop.apache.org/docs/r2.7.2/hadoop-project-dist/hadoop-common/core-default.xml > > Jason Altekruse > Software Engineer at Dremio > Apache Drill Committer > > On Wed, Apr 20, 2016 at 9:52 AM, Abhishek Girish < > [email protected]> > wrote: > > > Thanks Jason! I hadn't noticed the config property for S3. I tried this > out > > now, and feel it is a lot more easier now. > > > > And yes, we should definitely update the docs. There have been quite a > few > > threads related to S3 config. > > > > On Wed, Apr 20, 2016 at 8:19 AM, Jason Altekruse <[email protected]> > wrote: > > > > > I don't believe there is any way in which a particular bucket has a > > > property of being s3, s3n or s3a. As I understand it, this only change > > the > > > client library that is used to interface with S3. We have included the > > jars > > > necessary for s3a with Drill, which is the newest and most performant > > > option available. > > > > > > I need to open a doc JIRA for this, but there is one way in which the > s3 > > > experience was improved recently to prevent the need to restart Drill > to > > > add your S3 credentials. When you create a connection to an S3 bucket, > > you > > > can now specify your credentials in a property named "config" in the > > > storage plugin. This allows you to set any filesystem properties, which > > we > > > previously was only possible to set with a core-site.xml file on the > > > classpath when starting Drill. > > > > > > Example: > > > { > > > "type": "file", > > > "enabled": true, > > > "connection": "s3a://address.of.your.bucket/", > > > "config": { > > > "fs.s3a.access.key": "<PUT YOUR ACCESS KEY HERE>", > > > "fs.s3a.secret.key": "<PUT YOUR SECRET KEY HERE>" > > > }, > > > "workspaces": { > > > "root": { > > > "location": "/", > > > "writable": false, > > > "defaultInputFormat": null > > > } > > > }, > > > "formats": { > > > "psv": { > > > "type": "text", > > > "extensions": [ > > > "tbl" > > > ], > > > "delimiter": "|" > > > }, ... > > > > > > > > > Jason Altekruse > > > Software Engineer at Dremio > > > Apache Drill Committer > > > > > > On Wed, Apr 20, 2016 at 7:40 AM, Nick Monetta <[email protected]> wrote: > > > > > > > Hi, > > > > Does Drill v1.6 still support s3n connections or just s3a? > > > > > > > > I have a s3n S3 bucket that I'm trying to connect to and it will not > > > work. > > > > My config is: > > > > > > > > { > > > > "type": "file", > > > > "enabled": true, > > > > "connection": "s3n://inrixprod-tapp/", > > > > "workspaces": { > > > > "root": { > > > > "location": "/", > > > > "writable": false, > > > > "defaultInputFormat": null > > > > }, > > > > > > > > Nick Monetta | INRIX |[email protected] |Movement Intelligence | > > > > www.inrix.com | mobile +1 646-248-4105 | > > > > > > > > > > > > > > > > > >
