[
https://issues.apache.org/jira/browse/DRILL-6504?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16519855#comment-16519855
]
Bridget Bevens commented on DRILL-6504:
---------------------------------------
Hi [~paul-rogers],
I edited the content based on your feedback:
https://drill.apache.org/docs/s3-storage-plugin/
Hoping the flow's a bit better now. It seems that the three methods really
apply to how the AWS credentials are defined
because for all three methods, the user still has to configure the S3 bucket
name in the SP configuration. So, I moved the
three methods of configuration into the AWS credentials section. Hope that
works. If not, let me know.
One thing I'm not totally clear on is if the user is working in a Hadoop
environment with S3,
should the Hadoop core-site.xml file already be pre-configured with the access
keys? I made the assumption of yes. :-)
Again, if that's not true, let me know and I will change it.
Thanks,
Bridget
> Corrections to S3 storage doc pages
> -----------------------------------
>
> Key: DRILL-6504
> URL: https://issues.apache.org/jira/browse/DRILL-6504
> Project: Apache Drill
> Issue Type: Bug
> Components: Documentation
> Affects Versions: 1.13.0
> Reporter: Paul Rogers
> Assignee: Bridget Bevens
> Priority: Major
> Labels: doc-impacting
>
> [The documentation for S3
> storage|http://drill.apache.org/docs/s3-storage-plugin/] contains a number of
> minor errors.
> "using the S3a library."
> Change to "using the HDFS s3a library." (The library is provided via HDFS,
> not Drill.)
> ----
> "Drill's previous S3n interface"
> Change to "the older HDFS s3n library." (Again, S3 support is provided by
> HDFS.)
> ----
> "Starting with version 1.3.0"
> Can probably be removed, 1.3 was quite a long time ago.
> ----
> "To enable Drill's S3a support"
> Change to "To enable HDFS s3a support"
> ----
> Include a link to the HDFS S3 documentation:
> [https://hadoop.apache.org/docs/stable/hadoop-aws/tools/hadoop-aws/index.html]
> ----
> Refer to the S3a documentation link above. There are actually multiple ways
> to configure S3a:
> * In the storage plugin config (as is suggested by the shipped s3 example in
> the Drill storage page.)
> * Using core-site.xml}} as described in the docs.
> * Using environment variables set before running Drill or in {{drill-env.sh}}.
> * Maybe using the {{~/.aws/credentials}} directory? Have not tested this one.
> ----
> Since Drill does not use HDFS 3.x, Drill dues not support AWS temporary
> credentials as described in the S3a documentation.
> ----
> "edit the file conf/core-site.xml in your Drill install directory,"
> Change to "in the $DRILL_HOME/conf or $DRILL_SITE directory, rename
> core-site-example.xml to core-site.xml and ..."
> Note: once the file is renamed, it the user had $HADOOP_HOME on their path,
> Hadoop support will break because Drill will pull in the Drill version of
> core-site.xml rather than the Hadoop one. This will cause tools such as
> Drill-on-YARN to fail.
> In this situation, the user should make the changes in Hadoop's core-site.xml
> and should not create one for Drill. (In fact, if the user is using Hadoop
> and want to use S3 with Drill, they probably already had S3 support
> configured...)
> In Drill 1.13 (not sure when it was added), the default "s3" storage plugin
> lets the user define the access keys as storage plugin configuration
> properties:
> {code}
> "config": {
> "fs.s3a.access.key": "ID",
> "fs.s3a.secret.key": "SECRET"
> },
> {code}
> This approach is not very secure, but is probably OK when Drill has a single
> user (such as on a laptop.)
> ----
> When using the above approach, it appears that one must specify the endpoint:
> {code}
> "connection": "s3a://<bucket-name>/",
> "config": {
> "fs.s3a.access.key": "<key>",
> "fs.s3a.secret.key": "<key>",
> "fs.s3a.endpoint": "s3.us-west-1.amazonaws.com"
> },
> {code}
> I could not get the above to work using the pattern in the default S3 config:
> {code}
> connection: "s3a://my.bucket.location.com",
> {code}
> Using the endpoint is how all S3a examples I could find described the usage.
> ----
> A workable, semi-secure combination is:
> * Use the S3 storage plugin config to specify only the bucket.
> {code}
> "connection": "s3a://mybucket/",
> "config": {
> },
> {code}
> * Specify the credentials and endpoint in the {{core-site.xml}} file:
> {code}
> <configuration>
> <property>
> <name>fs.s3a.access.key</name>
> <value>ACCESS-KEY</value>
> </property>
> <property>
> <name>fs.s3a.secret.key</name>
> <value>SECRET-KEY</value>
> </property>
> <property>
> <name>fs.s3a.endpoint</name>
> <value>s3.REGION.amazonaws.com</value>
> </property>
> </configuration>
> {code}
> ----
> "Point your browser to http://:8047";
> Change to "http://<drill-host>:8047, where <drill-host> is a node on which
> Drill is running."
> "Note: on a single machine system, you'll need to run drill-embedded before
> you can access the web console site"
> The general rule is that Drill must be running, whether embedded, in
> server-mode on the local host, or in a cluster.
> ----
> "Duplicate the 'dfs' plugin."
> This is not necessary. If Drill is local (single server) then it is helpful
> to allow both local and S3 access. But, if Drill is deployed in a cluster,
> local file access is problematic. In short, make this section closer to the
> [HDFS storage|http://drill.apache.org/docs/file-system-storage-plugin/] page.
> Note also that in Drill 1.13, Drill ships with an "s3" storage configuration;
> the user need only enable it. No need to copy/paste the dfs plugin.
> ----
> "you can set this parameter in conf/core-site.xml file in your Drill install
> directory"
> Based on the comments above, change this to: "you can set this parameter in
> core-site.xml"
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
