Thanks Bridget, let me know if you need any other info from me or want me to review the changes.
Jason Altekruse Software Engineer at Dremio Apache Drill Committer On Wed, Apr 20, 2016 at 11:28 AM, Bridget Bevens <[email protected]> wrote: > Created DRILL-4621 <https://issues.apache.org/jira/browse/DRILL-4621> to > track doc change request. > > Thanks, > Bridget > > On Wed, Apr 20, 2016 at 10:10 AM, Jason Altekruse <[email protected]> > wrote: > > > It looks like a number of doc pages can be improved by referencing some > > changes made recently. > > > > With the inclusion of the needed jars for s3a with Drill, there is no > > longer a need to download jets3t [1]. In addition to setting your > > credentials, this option for allowing more concurrent connections > > (necessary to allow reads of wider parquet files) can also be set in this > > block instead of a core-site.xml file [2]. > > > > This config block can actually be used to set any filesystem properties. > > Some of these are custom to a particular filesystem like S3, but a number > > of them are used by a variety of implementations of the HDFS interface. > Any > > properties like these [3] should be able to be set in this config block. > > > > [1] - > > > https://drill.apache.org/blog/2014/12/09/running-sql-queries-on-amazon-s3/ > > [2] - > > > > > https://drill.apache.org/docs/s3-storage-plugin/#quering-parquet-format-files-on-s3 > > [3] - > > > > > https://hadoop.apache.org/docs/r2.7.2/hadoop-project-dist/hadoop-common/core-default.xml > > > > Jason Altekruse > > Software Engineer at Dremio > > Apache Drill Committer > > > > On Wed, Apr 20, 2016 at 9:52 AM, Abhishek Girish < > > [email protected]> > > wrote: > > > > > Thanks Jason! I hadn't noticed the config property for S3. I tried this > > out > > > now, and feel it is a lot more easier now. > > > > > > And yes, we should definitely update the docs. There have been quite a > > few > > > threads related to S3 config. > > > > > > On Wed, Apr 20, 2016 at 8:19 AM, Jason Altekruse <[email protected]> > > wrote: > > > > > > > I don't believe there is any way in which a particular bucket has a > > > > property of being s3, s3n or s3a. As I understand it, this only > change > > > the > > > > client library that is used to interface with S3. We have included > the > > > jars > > > > necessary for s3a with Drill, which is the newest and most performant > > > > option available. > > > > > > > > I need to open a doc JIRA for this, but there is one way in which the > > s3 > > > > experience was improved recently to prevent the need to restart Drill > > to > > > > add your S3 credentials. When you create a connection to an S3 > bucket, > > > you > > > > can now specify your credentials in a property named "config" in the > > > > storage plugin. This allows you to set any filesystem properties, > which > > > we > > > > previously was only possible to set with a core-site.xml file on the > > > > classpath when starting Drill. > > > > > > > > Example: > > > > { > > > > "type": "file", > > > > "enabled": true, > > > > "connection": "s3a://address.of.your.bucket/", > > > > "config": { > > > > "fs.s3a.access.key": "<PUT YOUR ACCESS KEY HERE>", > > > > "fs.s3a.secret.key": "<PUT YOUR SECRET KEY HERE>" > > > > }, > > > > "workspaces": { > > > > "root": { > > > > "location": "/", > > > > "writable": false, > > > > "defaultInputFormat": null > > > > } > > > > }, > > > > "formats": { > > > > "psv": { > > > > "type": "text", > > > > "extensions": [ > > > > "tbl" > > > > ], > > > > "delimiter": "|" > > > > }, ... > > > > > > > > > > > > Jason Altekruse > > > > Software Engineer at Dremio > > > > Apache Drill Committer > > > > > > > > On Wed, Apr 20, 2016 at 7:40 AM, Nick Monetta <[email protected]> > wrote: > > > > > > > > > Hi, > > > > > Does Drill v1.6 still support s3n connections or just s3a? > > > > > > > > > > I have a s3n S3 bucket that I'm trying to connect to and it will > not > > > > work. > > > > > My config is: > > > > > > > > > > { > > > > > "type": "file", > > > > > "enabled": true, > > > > > "connection": "s3n://inrixprod-tapp/", > > > > > "workspaces": { > > > > > "root": { > > > > > "location": "/", > > > > > "writable": false, > > > > > "defaultInputFormat": null > > > > > }, > > > > > > > > > > Nick Monetta | INRIX |[email protected] |Movement Intelligence | > > > > > www.inrix.com | mobile +1 646-248-4105 | > > > > > > > > > > > > > > > > > > > > > > > > >
