Yes, I am also using drill in embedded mode. Do you have at least read access to the bucket ? Otherwise, I do not think this will work.
Thanks Padma > On Oct 25, 2017, at 8:19 AM, Charles Givre <cgi...@gmail.com> wrote: > > Hi Padma, > I have been using drill in embedded mode. Would that make a difference? > Also, I’m wondering if Drill might be trying to use a blocked port for S3. > —C > > >> On Oct 24, 2017, at 2:00 PM, Padma Penumarthy <ppenumar...@mapr.com> wrote: >> >> Yes, I guess you need to have access to the bucket. >> Not sure how it will work otherwise. >> >> Thanks >> Padma >> >> >> On Oct 24, 2017, at 10:51 AM, Charles Givre >> <cgi...@gmail.com<mailto:cgi...@gmail.com>> wrote: >> >> Hi Padma, >> I’m wondering if the issue is that I only have access to a subfolder in the >> s3 bucket. IE: >> >> s3://bucket/folder1/folder2/folder3 <s3://bucket/folder1/folder2/folder3> >> >> I only have access to folder3. Might that be causing the issue? >> —C >> >> >> >> On Oct 24, 2017, at 13:49, Padma Penumarthy >> <ppenumar...@mapr.com<mailto:ppenumar...@mapr.com>> wrote: >> >> Charles, can you try exactly what I did. >> I did not do anything else other than enable the S3 plugin and change the >> plugin >> configuration like this. >> >> { >> "type": "file", >> "enabled": true, >> "connection": "s3a://<bucket-name>", >> "config": { >> "fs.s3a.access.key": “XXXX", >> "fs.s3a.secret.key": “YYYY" >> }, >> >> >> Thanks >> Padma >> >> >> On Oct 24, 2017, at 10:06 AM, Charles Givre >> <cgi...@gmail.com<mailto:cgi...@gmail.com> <mailto:cgi...@gmail.com>> wrote: >> >> Hi everyone and thank you for your help. I’m still not able to connect to >> S3. >> >> >> Here is the error I’m getting: >> >> 0: jdbc:drill:zk=local> use s3; >> Error: RESOURCE ERROR: Failed to create schema tree. >> >> >> [Error Id: 57c82d90-2166-4a37-94a0-1cfeb0cdc4b6 on >> charless-mbp-2.fios-router.home:31010] (state=,code=0) >> java.sql.SQLException: RESOURCE ERROR: Failed to create schema tree. >> >> >> [Error Id: 57c82d90-2166-4a37-94a0-1cfeb0cdc4b6 on >> charless-mbp-2.fios-router.home:31010] >> at >> org.apache.drill.jdbc.impl.DrillCursor.nextRowInternally(DrillCursor.java:489) >> at >> org.apache.drill.jdbc.impl.DrillCursor.loadInitialSchema(DrillCursor.java:561) >> at >> org.apache.drill.jdbc.impl.DrillResultSetImpl.execute(DrillResultSetImpl.java:1895) >> at >> org.apache.drill.jdbc.impl.DrillResultSetImpl.execute(DrillResultSetImpl.java:61) >> at >> org.apache.calcite.avatica.AvaticaConnection$1.execute(AvaticaConnection.java:473) >> at >> org.apache.drill.jdbc.impl.DrillMetaImpl.prepareAndExecute(DrillMetaImpl.java:1100) >> at >> org.apache.calcite.avatica.AvaticaConnection.prepareAndExecuteInternal(AvaticaConnection.java:477) >> at >> org.apache.drill.jdbc.impl.DrillConnectionImpl.prepareAndExecuteInternal(DrillConnectionImpl.java:181) >> at >> org.apache.calcite.avatica.AvaticaStatement.executeInternal(AvaticaStatement.java:109) >> at >> org.apache.calcite.avatica.AvaticaStatement.execute(AvaticaStatement.java:121) >> at >> org.apache.drill.jdbc.impl.DrillStatementImpl.execute(DrillStatementImpl.java:101) >> at sqlline.Commands.execute(Commands.java:841) >> at sqlline.Commands.sql(Commands.java:751) >> at sqlline.SqlLine.dispatch(SqlLine.java:746) >> at sqlline.SqlLine.begin(SqlLine.java:621) >> at sqlline.SqlLine.start(SqlLine.java:375) >> at sqlline.SqlLine.main(SqlLine.java:268) >> Caused by: org.apache.drill.common.exceptions.UserRemoteException: RESOURCE >> ERROR: Failed to create schema tree. >> >> >> [Error Id: 57c82d90-2166-4a37-94a0-1cfeb0cdc4b6 on >> charless-mbp-2.fios-router.home:31010] >> at >> org.apache.drill.exec.rpc.user.QueryResultHandler.resultArrived(QueryResultHandler.java:123) >> at org.apache.drill.exec.rpc.user.UserClient.handle(UserClient.java:368) >> at org.apache.drill.exec.rpc.user.UserClient.handle(UserClient.java:90) >> at org.apache.drill.exec.rpc.RpcBus$InboundHandler.decode(RpcBus.java:274) >> at org.apache.drill.exec.rpc.RpcBus$InboundHandler.decode(RpcBus.java:244) >> at >> io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:89) >> at >> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339) >> at >> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324) >> at >> io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:254) >> at >> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339) >> at >> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324) >> at >> io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) >> at >> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339) >> at >> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324) >> at >> io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:242) >> at >> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339) >> at >> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324) >> at >> io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86) >> at >> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339) >> at >> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324) >> at >> io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:847) >> at >> io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:131) >> at >> io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511) >> at >> io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468) >> at >> io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382) >> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354) >> at >> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111) >> at java.lang.Thread.run(Thread.java:745) >> 0: jdbc:drill:zk=local> >> >> >> Here is my conf.site <http://conf.site/> <http://conf.site/ >> <http://conf.site/>>.xml file: >> >> <property> >> <name>fs.s3.awsAccessKeyId</name> >> <value>XXX</value> >> </property> >> >> <property> >> <name>fs.s3.awsSecretAccessKey</name> >> <value> XXX </value> >> </property> >> <property> >> <name>fs.s3n.awsAccessKeyId</name> >> <value> XXX </value> >> </property> >> >> <property> >> <name>fs.s3n.awsSecretAccessKey</name> >> <value> XXX </value> >> </property> >> <property> >> <name>fs.s3a.awsAccessKeyId</name> >> <value> XXX </value> >> </property> >> >> <property> >> <name>fs.s3a.awsSecretAccessKey</name> >> <value> XXX </value> >> </property> >> >> And my config info: >> >> { >> "type": "file", >> "enabled": true, >> "connection": "s3://<my bucket>", >> "config": null, >> "workspaces": { >> "root": { >> "location": "/", >> "writable": false, >> "defaultInputFormat": null >> } >> }, >> >> I did copy jets3t-0.9.4.jar to the /jars/3rdparty path. Any debugging >> suggestions? >> —C >> >> >> On Oct 20, 2017, at 15:55, Arjun kr >> <arjun...@outlook.com<mailto:arjun...@outlook.com>> wrote: >> >> Hi Charles, >> >> >> I'm not aware of any such settings. As Padma mentioned in previous mail, It >> works fine for me by following instructions in >> https://drill.apache.org/docs/s3-storage-plugin/ . >> >> >> Thanks, >> >> >> Arjun >> >> >> ________________________________ >> From: Charles Givre <cgi...@gmail.com<mailto:cgi...@gmail.com>> >> Sent: Friday, October 20, 2017 11:48 PM >> To: user@drill.apache.org<mailto:user@drill.apache.org> >> Subject: Re: S3 Connection Issues >> >> Hi Arjun, >> Thanks for your help. Are there settings in S3 that would prevent Drill >> from connecting? I’ll try hdfs shell, but I am able to connect with the CLI >> tool. My hunch is that there is a permission not set correctly on S3 or >> I’m missing some config variable in Drill. >> — C >> >> >> On Oct 20, 2017, at 14:12, Arjun kr >> <arjun...@outlook.com<mailto:arjun...@outlook.com>> wrote: >> >> Hi Charles, >> >> >> Any chance you can test s3 connectivity with other tools like hdfs shell or >> hive in case you haven't tried already (and these tools available)? This may >> help to identify if it is Drill specific issue. >> >> >> For connecting via hdfs , you may try below command. >> >> >> hadoop fs -Dfs.s3a.access.key="XXXX" -Dfs.s3a.secret.key="YYYYY" -ls >> s3a://<bucket-name>/ >> >> >> Enable DEBUG logging if needed. >> >> >> export HADOOP_ROOT_LOGGER=hadoop.root.logger=DEBUG,console >> >> >> Thanks, >> >> >> Arjun >> >> >> ________________________________ >> From: Padma Penumarthy <ppenumar...@mapr.com<mailto:ppenumar...@mapr.com>> >> Sent: Friday, October 20, 2017 3:00 AM >> To: user@drill.apache.org<mailto:user@drill.apache.org> >> Subject: Re: S3 Connection Issues >> >> Hi Charles, >> >> I tried us-west-2 and it worked fine for me with drill built from latest >> source. >> I did not do anything special. >> Just enabled the S3 plugin and updated the plugin configuration like this. >> >> { >> "type": "file", >> "enabled": true, >> "connection": "s3a://<bucket-name>", >> "config": { >> "fs.s3a.access.key": “XXXX", >> "fs.s3a.secret.key": “YYYY" >> }, >> >> I am able to do show databases and also can query the parquet files I >> uploaded to the bucket. >> >> 0: jdbc:drill:zk=local> show databases; >> +---------------------+ >> | SCHEMA_NAME | >> +---------------------+ >> | INFORMATION_SCHEMA | >> | cp.default | >> | dfs.default | >> | dfs.root | >> | dfs.tmp | >> | s3.default | >> | s3.root | >> | sys | >> +---------------------+ >> 8 rows selected (2.892 seconds) >> >> >> Thanks >> Padma >> >> On Oct 18, 2017, at 9:18 PM, Charles Givre >> <cgi...@gmail.com<mailto:cgi...@gmail.com><mailto:cgi...@gmail.com>> wrote: >> >> Hi Padma, >> The bucket is is us-west-2. I also discovered that some of the variable >> names in the documentation on the main Drill site are incorrect. Do I need >> to specify the region in the configuration somewhere? >> >> As an update, after discovering that the variable names are incorrect and >> that I didn’t have Jets3t installed properly, I’m now getting the following >> error: >> >> jdbc:drill:zk=local> show databases; >> Error: RESOURCE ERROR: Failed to create schema tree. >> >> >> [Error Id: e6012aa2-c775-46b9-b3ee-0af7d0b0871d on >> charless-mbp-2.fios-router.home:31010] >> >> (org.apache.hadoop.fs.s3.S3Exception) org.jets3t.service.S3ServiceException: >> Service Error Message. -- ResponseCode: 403, ResponseStatus: Forbidden, XML >> Error Message: <?xml version="1.0" >> encoding="UTF-8"?><Error><Code>SignatureDoesNotMatch</Code><Message>The >> request signature we calculated does not match the signature you provided. >> Check your key and signing method.</Message></Error> >> org.apache.hadoop.fs.s3.Jets3tFileSystemStore.get():175 >> org.apache.hadoop.fs.s3.Jets3tFileSystemStore.retrieveINode():221 >> >> Thanks, >> — C >> >> >> On Oct 19, 2017, at 00:14, Padma Penumarthy >> <ppenumar...@mapr.com<mailto:ppenumar...@mapr.com><mailto:ppenumar...@mapr.com>> >> wrote: >> >> Which AWS region are you trying to connect to ? >> We have a problem connecting to regions which support only v4 signature >> since the version of hadoop we include in Drill is old. >> Last time I tried, using Hadoop 2.8.1 worked for me. >> >> Thanks >> Padma >> >> >> On Oct 18, 2017, at 8:14 PM, Charles Givre >> <cgi...@gmail.com<mailto:cgi...@gmail.com><mailto:cgi...@gmail.com>> wrote: >> >> Hello all, >> I’m trying to use Drill to query data in an S3 bucket and running into some >> issues which I can’t seem to fix. I followed the various instructions >> online to set up Drill with S3, and put my keys in both the conf-site.xml >> and in the plugin config, but every time I attempt to do anything I get the >> following errors: >> >> >> jdbc:drill:zk=local> show databases; >> Error: SYSTEM ERROR: AmazonS3Exception: Status Code: 403, AWS Service: >> Amazon S3, AWS Request ID: 56D1999BD1E62DEB, AWS Error Code: null, AWS Error >> Message: Forbidden >> >> >> [Error Id: 65d0bb52-a923-4e98-8ab1-65678169140e on >> charless-mbp-2.fios-router.home:31010] (state=,code=0) >> 0: jdbc:drill:zk=local> show databases; >> Error: SYSTEM ERROR: AmazonS3Exception: Status Code: 403, AWS Service: >> Amazon S3, AWS Request ID: 4D2CBA8D42A9ECA0, AWS Error Code: null, AWS Error >> Message: Forbidden >> >> >> [Error Id: 25a2d008-2f4d-4433-a809-b91ae063e61a on >> charless-mbp-2.fios-router.home:31010] (state=,code=0) >> 0: jdbc:drill:zk=local> show files in s3.root; >> Error: SYSTEM ERROR: AmazonS3Exception: Status Code: 403, AWS Service: >> Amazon S3, AWS Request ID: 2C635944EDE591F0, AWS Error Code: null, AWS Error >> Message: Forbidden >> >> >> [Error Id: 02e136f5-68c0-4b47-9175-a9935bda5e1c on >> charless-mbp-2.fios-router.home:31010] (state=,code=0) >> 0: jdbc:drill:zk=local> show schemas; >> Error: SYSTEM ERROR: AmazonS3Exception: Status Code: 403, AWS Service: >> Amazon S3, AWS Request ID: 646EB5B2EBCF7CD2, AWS Error Code: null, AWS Error >> Message: Forbidden >> >> >> [Error Id: 954aaffe-616a-4f40-9ba5-d4b7c04fe238 on >> charless-mbp-2.fios-router.home:31010] (state=,code=0) >> >> I have verified that the keys are correct but using the AWS CLI and >> downloaded some of the files, but I’m kind of at a loss as to how to debug. >> Any suggestions? >> Thanks in advance, >> — C >> >
