subject:"Re\: Drill v1.6 and s3n connection"

Re: Drill v1.6 and s3n connection

2016-04-20 Thread Jason Altekruse

Thanks Bridget, let me know if you need any other info from me or want me
to review the changes.

Jason Altekruse
Software Engineer at Dremio
Apache Drill Committer

On Wed, Apr 20, 2016 at 11:28 AM, Bridget Bevens 
wrote:

> Created DRILL-4621  to
> track doc change request.
>
> Thanks,
> Bridget
>
> On Wed, Apr 20, 2016 at 10:10 AM, Jason Altekruse 
> wrote:
>
> > It looks like a number of doc pages can be improved by referencing some
> > changes made recently.
> >
> > With the inclusion of the needed jars for s3a with Drill, there is no
> > longer a need to download jets3t [1]. In addition to setting your
> > credentials, this option for allowing more concurrent connections
> > (necessary to allow reads of wider parquet files) can also be set in this
> > block instead of a core-site.xml file [2].
> >
> > This config block can actually be used to set any filesystem properties.
> > Some of these are custom to a particular filesystem like S3, but a number
> > of them are used by a variety of implementations of the HDFS interface.
> Any
> > properties like these [3] should be able to be set in this config block.
> >
> > [1] -
> >
> https://drill.apache.org/blog/2014/12/09/running-sql-queries-on-amazon-s3/
> > [2] -
> >
> >
> https://drill.apache.org/docs/s3-storage-plugin/#quering-parquet-format-files-on-s3
> > [3] -
> >
> >
> https://hadoop.apache.org/docs/r2.7.2/hadoop-project-dist/hadoop-common/core-default.xml
> >
> > Jason Altekruse
> > Software Engineer at Dremio
> > Apache Drill Committer
> >
> > On Wed, Apr 20, 2016 at 9:52 AM, Abhishek Girish <
> > abhishek.gir...@gmail.com>
> > wrote:
> >
> > > Thanks Jason! I hadn't noticed the config property for S3. I tried this
> > out
> > > now, and feel it is a lot more easier now.
> > >
> > > And yes, we should definitely update the docs. There have been quite a
> > few
> > > threads related to S3 config.
> > >
> > > On Wed, Apr 20, 2016 at 8:19 AM, Jason Altekruse 
> > wrote:
> > >
> > > > I don't believe there is any way in which a particular bucket has a
> > > > property of being s3, s3n or s3a. As I understand it, this only
> change
> > > the
> > > > client library that is used to interface with S3. We have included
> the
> > > jars
> > > > necessary for s3a with Drill, which is the newest and most performant
> > > > option available.
> > > >
> > > > I need to open a doc JIRA for this, but there is one way in which the
> > s3
> > > > experience was improved recently to prevent the need to restart Drill
> > to
> > > > add your S3 credentials. When you create a connection to an S3
> bucket,
> > > you
> > > > can now specify your credentials in a property named "config" in the
> > > > storage plugin. This allows you to set any filesystem properties,
> which
> > > we
> > > > previously was only possible to set with a core-site.xml file on the
> > > > classpath when starting Drill.
> > > >
> > > > Example:
> > > > {
> > > >   "type": "file",
> > > >   "enabled": true,
> > > >   "connection": "s3a://address.of.your.bucket/",
> > > >   "config": {
> > > > "fs.s3a.access.key": "",
> > > > "fs.s3a.secret.key": ""
> > > >   },
> > > >   "workspaces": {
> > > > "root": {
> > > >   "location": "/",
> > > >   "writable": false,
> > > >   "defaultInputFormat": null
> > > > }
> > > >   },
> > > >   "formats": {
> > > > "psv": {
> > > >   "type": "text",
> > > >   "extensions": [
> > > > "tbl"
> > > >   ],
> > > >   "delimiter": "|"
> > > > }, ...
> > > >
> > > >
> > > > Jason Altekruse
> > > > Software Engineer at Dremio
> > > > Apache Drill Committer
> > > >
> > > > On Wed, Apr 20, 2016 at 7:40 AM, Nick Monetta 
> wrote:
> > > >
> > > > > Hi,
> > > > > Does Drill v1.6 still support s3n connections or just s3a?
> > > > >
> > > > > I have a s3n S3 bucket that I'm trying to connect to and it will
> not
> > > > work.
> > > > > My config is:
> > > > >
> > > > > {
> > > > >   "type": "file",
> > > > >   "enabled": true,
> > > > >   "connection": "s3n://inrixprod-tapp/",
> > > > >   "workspaces": {
> > > > > "root": {
> > > > >   "location": "/",
> > > > >   "writable": false,
> > > > >   "defaultInputFormat": null
> > > > > },
> > > > >
> > > > > Nick Monetta | INRIX |ni...@inrix.com |Movement Intelligence |
> > > > > www.inrix.com  | mobile +1 646-248-4105 |
> > > > >
> > > > >
> > > > >
> > > >
> > >
> >
>

Re: Drill v1.6 and s3n connection

2016-04-20 Thread Bridget Bevens

Created DRILL-4621  to
track doc change request.

Thanks,
Bridget

On Wed, Apr 20, 2016 at 10:10 AM, Jason Altekruse  wrote:

> It looks like a number of doc pages can be improved by referencing some
> changes made recently.
>
> With the inclusion of the needed jars for s3a with Drill, there is no
> longer a need to download jets3t [1]. In addition to setting your
> credentials, this option for allowing more concurrent connections
> (necessary to allow reads of wider parquet files) can also be set in this
> block instead of a core-site.xml file [2].
>
> This config block can actually be used to set any filesystem properties.
> Some of these are custom to a particular filesystem like S3, but a number
> of them are used by a variety of implementations of the HDFS interface. Any
> properties like these [3] should be able to be set in this config block.
>
> [1] -
> https://drill.apache.org/blog/2014/12/09/running-sql-queries-on-amazon-s3/
> [2] -
>
> https://drill.apache.org/docs/s3-storage-plugin/#quering-parquet-format-files-on-s3
> [3] -
>
> https://hadoop.apache.org/docs/r2.7.2/hadoop-project-dist/hadoop-common/core-default.xml
>
> Jason Altekruse
> Software Engineer at Dremio
> Apache Drill Committer
>
> On Wed, Apr 20, 2016 at 9:52 AM, Abhishek Girish <
> abhishek.gir...@gmail.com>
> wrote:
>
> > Thanks Jason! I hadn't noticed the config property for S3. I tried this
> out
> > now, and feel it is a lot more easier now.
> >
> > And yes, we should definitely update the docs. There have been quite a
> few
> > threads related to S3 config.
> >
> > On Wed, Apr 20, 2016 at 8:19 AM, Jason Altekruse 
> wrote:
> >
> > > I don't believe there is any way in which a particular bucket has a
> > > property of being s3, s3n or s3a. As I understand it, this only change
> > the
> > > client library that is used to interface with S3. We have included the
> > jars
> > > necessary for s3a with Drill, which is the newest and most performant
> > > option available.
> > >
> > > I need to open a doc JIRA for this, but there is one way in which the
> s3
> > > experience was improved recently to prevent the need to restart Drill
> to
> > > add your S3 credentials. When you create a connection to an S3 bucket,
> > you
> > > can now specify your credentials in a property named "config" in the
> > > storage plugin. This allows you to set any filesystem properties, which
> > we
> > > previously was only possible to set with a core-site.xml file on the
> > > classpath when starting Drill.
> > >
> > > Example:
> > > {
> > >   "type": "file",
> > >   "enabled": true,
> > >   "connection": "s3a://address.of.your.bucket/",
> > >   "config": {
> > > "fs.s3a.access.key": "",
> > > "fs.s3a.secret.key": ""
> > >   },
> > >   "workspaces": {
> > > "root": {
> > >   "location": "/",
> > >   "writable": false,
> > >   "defaultInputFormat": null
> > > }
> > >   },
> > >   "formats": {
> > > "psv": {
> > >   "type": "text",
> > >   "extensions": [
> > > "tbl"
> > >   ],
> > >   "delimiter": "|"
> > > }, ...
> > >
> > >
> > > Jason Altekruse
> > > Software Engineer at Dremio
> > > Apache Drill Committer
> > >
> > > On Wed, Apr 20, 2016 at 7:40 AM, Nick Monetta  wrote:
> > >
> > > > Hi,
> > > > Does Drill v1.6 still support s3n connections or just s3a?
> > > >
> > > > I have a s3n S3 bucket that I'm trying to connect to and it will not
> > > work.
> > > > My config is:
> > > >
> > > > {
> > > >   "type": "file",
> > > >   "enabled": true,
> > > >   "connection": "s3n://inrixprod-tapp/",
> > > >   "workspaces": {
> > > > "root": {
> > > >   "location": "/",
> > > >   "writable": false,
> > > >   "defaultInputFormat": null
> > > > },
> > > >
> > > > Nick Monetta | INRIX |ni...@inrix.com |Movement Intelligence |
> > > > www.inrix.com  | mobile +1 646-248-4105 |
> > > >
> > > >
> > > >
> > >
> >
>

Re: Drill v1.6 and s3n connection

2016-04-20 Thread Jason Altekruse

It looks like a number of doc pages can be improved by referencing some
changes made recently.

With the inclusion of the needed jars for s3a with Drill, there is no
longer a need to download jets3t [1]. In addition to setting your
credentials, this option for allowing more concurrent connections
(necessary to allow reads of wider parquet files) can also be set in this
block instead of a core-site.xml file [2].

This config block can actually be used to set any filesystem properties.
Some of these are custom to a particular filesystem like S3, but a number
of them are used by a variety of implementations of the HDFS interface. Any
properties like these [3] should be able to be set in this config block.

[1] -
https://drill.apache.org/blog/2014/12/09/running-sql-queries-on-amazon-s3/
[2] -
https://drill.apache.org/docs/s3-storage-plugin/#quering-parquet-format-files-on-s3
[3] -
https://hadoop.apache.org/docs/r2.7.2/hadoop-project-dist/hadoop-common/core-default.xml

Jason Altekruse
Software Engineer at Dremio
Apache Drill Committer

On Wed, Apr 20, 2016 at 9:52 AM, Abhishek Girish 
wrote:

> Thanks Jason! I hadn't noticed the config property for S3. I tried this out
> now, and feel it is a lot more easier now.
>
> And yes, we should definitely update the docs. There have been quite a few
> threads related to S3 config.
>
> On Wed, Apr 20, 2016 at 8:19 AM, Jason Altekruse  wrote:
>
> > I don't believe there is any way in which a particular bucket has a
> > property of being s3, s3n or s3a. As I understand it, this only change
> the
> > client library that is used to interface with S3. We have included the
> jars
> > necessary for s3a with Drill, which is the newest and most performant
> > option available.
> >
> > I need to open a doc JIRA for this, but there is one way in which the s3
> > experience was improved recently to prevent the need to restart Drill to
> > add your S3 credentials. When you create a connection to an S3 bucket,
> you
> > can now specify your credentials in a property named "config" in the
> > storage plugin. This allows you to set any filesystem properties, which
> we
> > previously was only possible to set with a core-site.xml file on the
> > classpath when starting Drill.
> >
> > Example:
> > {
> >   "type": "file",
> >   "enabled": true,
> >   "connection": "s3a://address.of.your.bucket/",
> >   "config": {
> > "fs.s3a.access.key": "",
> > "fs.s3a.secret.key": ""
> >   },
> >   "workspaces": {
> > "root": {
> >   "location": "/",
> >   "writable": false,
> >   "defaultInputFormat": null
> > }
> >   },
> >   "formats": {
> > "psv": {
> >   "type": "text",
> >   "extensions": [
> > "tbl"
> >   ],
> >   "delimiter": "|"
> > }, ...
> >
> >
> > Jason Altekruse
> > Software Engineer at Dremio
> > Apache Drill Committer
> >
> > On Wed, Apr 20, 2016 at 7:40 AM, Nick Monetta  wrote:
> >
> > > Hi,
> > > Does Drill v1.6 still support s3n connections or just s3a?
> > >
> > > I have a s3n S3 bucket that I'm trying to connect to and it will not
> > work.
> > > My config is:
> > >
> > > {
> > >   "type": "file",
> > >   "enabled": true,
> > >   "connection": "s3n://inrixprod-tapp/",
> > >   "workspaces": {
> > > "root": {
> > >   "location": "/",
> > >   "writable": false,
> > >   "defaultInputFormat": null
> > > },
> > >
> > > Nick Monetta | INRIX |ni...@inrix.com |Movement Intelligence |
> > > www.inrix.com  | mobile +1 646-248-4105 |
> > >
> > >
> > >
> >
>

Re: Drill v1.6 and s3n connection

2016-04-20 Thread Abhishek Girish

Thanks Jason! I hadn't noticed the config property for S3. I tried this out
now, and feel it is a lot more easier now.

And yes, we should definitely update the docs. There have been quite a few
threads related to S3 config.

On Wed, Apr 20, 2016 at 8:19 AM, Jason Altekruse  wrote:

> I don't believe there is any way in which a particular bucket has a
> property of being s3, s3n or s3a. As I understand it, this only change the
> client library that is used to interface with S3. We have included the jars
> necessary for s3a with Drill, which is the newest and most performant
> option available.
>
> I need to open a doc JIRA for this, but there is one way in which the s3
> experience was improved recently to prevent the need to restart Drill to
> add your S3 credentials. When you create a connection to an S3 bucket, you
> can now specify your credentials in a property named "config" in the
> storage plugin. This allows you to set any filesystem properties, which we
> previously was only possible to set with a core-site.xml file on the
> classpath when starting Drill.
>
> Example:
> {
>   "type": "file",
>   "enabled": true,
>   "connection": "s3a://address.of.your.bucket/",
>   "config": {
> "fs.s3a.access.key": "",
> "fs.s3a.secret.key": ""
>   },
>   "workspaces": {
> "root": {
>   "location": "/",
>   "writable": false,
>   "defaultInputFormat": null
> }
>   },
>   "formats": {
> "psv": {
>   "type": "text",
>   "extensions": [
> "tbl"
>   ],
>   "delimiter": "|"
> }, ...
>
>
> Jason Altekruse
> Software Engineer at Dremio
> Apache Drill Committer
>
> On Wed, Apr 20, 2016 at 7:40 AM, Nick Monetta  wrote:
>
> > Hi,
> > Does Drill v1.6 still support s3n connections or just s3a?
> >
> > I have a s3n S3 bucket that I'm trying to connect to and it will not
> work.
> > My config is:
> >
> > {
> >   "type": "file",
> >   "enabled": true,
> >   "connection": "s3n://inrixprod-tapp/",
> >   "workspaces": {
> > "root": {
> >   "location": "/",
> >   "writable": false,
> >   "defaultInputFormat": null
> > },
> >
> > Nick Monetta | INRIX |ni...@inrix.com |Movement Intelligence |
> > www.inrix.com  | mobile +1 646-248-4105 |
> >
> >
> >
>

Re: Drill v1.6 and s3n connection

2016-04-20 Thread Oscar Morante


I think you need to set up jets3t if you want to use s3n.

On Wed, Apr 20, 2016 at 02:40:29PM +, Nick Monetta wrote:

Hi,
Does Drill v1.6 still support s3n connections or just s3a?

I have a s3n S3 bucket that I'm trying to connect to and it will not work. My 
config is:

{
 "type": "file",
 "enabled": true,
 "connection": "s3n://inrixprod-tapp/",
 "workspaces": {
   "root": {
 "location": "/",
 "writable": false,
 "defaultInputFormat": null
   },

Nick Monetta | INRIX |ni...@inrix.com |Movement Intelligence | www.inrix.com  | 
mobile +1 646-248-4105 |




--
Oscar Morante
"Self-education is, I firmly believe, the only kind of education there is."
 -- Isaac Asimov.


signature.asc
Description: Digital signature

Re: Drill v1.6 and s3n connection

2016-04-20 Thread Jason Altekruse

I don't believe there is any way in which a particular bucket has a
property of being s3, s3n or s3a. As I understand it, this only change the
client library that is used to interface with S3. We have included the jars
necessary for s3a with Drill, which is the newest and most performant
option available.

I need to open a doc JIRA for this, but there is one way in which the s3
experience was improved recently to prevent the need to restart Drill to
add your S3 credentials. When you create a connection to an S3 bucket, you
can now specify your credentials in a property named "config" in the
storage plugin. This allows you to set any filesystem properties, which we
previously was only possible to set with a core-site.xml file on the
classpath when starting Drill.

Example:
{
  "type": "file",
  "enabled": true,
  "connection": "s3a://address.of.your.bucket/",
  "config": {
"fs.s3a.access.key": "",
"fs.s3a.secret.key": ""
  },
  "workspaces": {
"root": {
  "location": "/",
  "writable": false,
  "defaultInputFormat": null
}
  },
  "formats": {
"psv": {
  "type": "text",
  "extensions": [
"tbl"
  ],
  "delimiter": "|"
}, ...

Jason Altekruse
Software Engineer at Dremio
Apache Drill Committer

On Wed, Apr 20, 2016 at 7:40 AM, Nick Monetta  wrote:

> Hi,
> Does Drill v1.6 still support s3n connections or just s3a?
>
> I have a s3n S3 bucket that I'm trying to connect to and it will not work.
> My config is:
>
> {
>   "type": "file",
>   "enabled": true,
>   "connection": "s3n://inrixprod-tapp/",
>   "workspaces": {
> "root": {
>   "location": "/",
>   "writable": false,
>   "defaultInputFormat": null
> },
>
> Nick Monetta | INRIX |ni...@inrix.com |Movement Intelligence |
> www.inrix.com  | mobile +1 646-248-4105 |
>
>
>

Re: Drill v1.6 and s3n connection

Re: Drill v1.6 and s3n connection

Re: Drill v1.6 and s3n connection

Re: Drill v1.6 and s3n connection

Re: Drill v1.6 and s3n connection

Re: Drill v1.6 and s3n connection

6 matches

Site Navigation

Mail list logo

Footer information