Can you try with following s3 config
{
"type": "file",
"enabled": true,
"connection": "s3a://bucket_name",
"config": {
"fs.s3a.connection.maximum": "10000",
"fs.s3a.access.key": "access_key",
"fs.s3a.secret.key": "secret_key",
"fs.s3a.buffer.dir": "/tmp",
"fs.s3a.multipart.size": "10485760",
"fs.s3a.multipart.threshold": "104857600"
},
"workspaces": {
"root": {
"location": "/",
"writable": false,
"defaultInputFormat": null
},
"tmp": {
"location": "/tmp",
"writable": true,
"defaultInputFormat": null
}
},
"formats": {
"psv": {
"type": "text",
"extensions": [
"tbl"
],
"delimiter": "|"
},
"csv": {
"type": "text",
"extensions": [
"csv"
],
"extractHeader": true,
"delimiter": ","
},
"tsv": {
"type": "text",
"extensions": [
"tsv"
],
"delimiter": "\t"
},
"parquet": {
"type": "parquet"
},
"json": {
"type": "json",
"extensions": [
"json"
]
},
"avro": {
"type": "avro"
},
"sequencefile": {
"type": "sequencefile",
"extensions": [
"seq"
]
},
"csvh": {
"type": "text",
"extensions": [
"csvh"
],
"extractHeader": true,
"delimiter": ","
}
}
}
On Fri, May 26, 2017 at 10:29 AM, Shuporno Choudhury <
shuporno.choudh...@manthan.com> wrote:
> Hi,
> Can someone at Drill help me with issue please?
>
> On Thu, May 25, 2017 at 1:33 PM, Shuporno Choudhury <
> shuporno.choudh...@manthan.com> wrote:
>
> > HI,
> >
> > I corrected the "show schemas" output by putting only "/" in the
> > "location" . Now it shows s3.tmp in the output.
> >
> > But, it has a weird problem.
> > The moment I add a folder to the location, eg: "/myfolder", then s3.tmp
> > vanishes from the "show schemas" output.
> >
> > Also, when I try to write into s3, I get the following error:
> >
> > Exception in thread "drill-executor-9" java.lang.UnsatisfiedLinkError:
> > org.apache.hadoop.io.nativeio.NativeIO$Windows.access0(
> > Ljava/lang/String;I)Z
> > at org.apache.hadoop.io.nativeio.NativeIO$Windows.access0(Native
> > Method)+--+
> > at org.apache.hadoop.io.nativeio.NativeIO$Windows.access(
> > NativeIO.java:609)
> >
> > This is only a snippet of the error associated with writing to s3
> >
> > On Thu, May 25, 2017 at 12:41 PM, Shuporno Choudhury <
> > shuporno.choudh...@manthan.com> wrote:
> >
> >> My s3 plugin info is as follows:
> >>
> >> {
> >> "type": "file",
> >> "enabled": true,
> >> "connection": "s3a://abcd",
> >> "config": {
> >> "fs.s3a.access.key": "abcd",
> >> "fs.s3a.secret.key": "abcd"
> >> },
> >> "workspaces": {
> >> "root": {
> >> "location": "/",
> >> "writable": false,
> >> "defaultInputFormat": null
> >> },
> >> "tmp": {
> >> "location": "/",
> >> "writable": *true*,
> >> "defaultInputFormat": "parquet"
> >> }
> >> }
> >>
> >>
> >> I have removed the info about the formats to keep the mail small.
> >> Also, I am using Dill on *Windows 10*
> >>
> >> On Mon, May 22, 2017 at 3:57 PM, Shuporno Choudhury <
> >> shuporno.choudh...@manthan.com> wrote:
> >>
> >>> Hi,
> >>>
> >>> Is it possible to write to a folder in an s3 bucket using the *s3.tmp*
> >>> workspace?
> >>> Whenever I try, it gives me the follwing error:
> >>>
> >>> *Error: VALIDATION ERROR: Schema [s3.tmp] is not valid with respect to
> >>> either root schema or current default schema.*
> >>> *Current default schema: s3.root*
> >>>
> >>> Also, s3.tmp doesn't appear while using the command "*show schemas*"
> >>> though the tmp workspace exists in the web console
> >>>
> >>> I am using Drill Version 1.10; embedded mode on my local system.
> >>>
> >>> However, I have no problem reading from an s3 bucket, the problem is
> >>> only writing to a s3 bucket.
> >>> --
> >>> Regards,
> >>> Shuporno Choudhury
> >>>
> >>
> >>
> >>
> >> --
> >> Regards,
> >> Shuporno Choudhury
> >>
> >
> >
> >
> > --
> > Regards,
> > Shuporno Choudhury
> >
>
>
>
> --
> Regards,
> Shuporno Choudhury
>
--
Nitin Pawar
