Re: Problem running queries on EMR - org.apache.drill.common.exceptions.UserRemoteException: SYSTEM ERROR: IllegalArgumentException: AWS Access Key ID

For large cluster I normally use clush (cluster shell), edit the config files on one node and then copy them to all using clush. --Andries > On Nov 16, 2015, at 6:41 PM, Mikhailau, Alex wrote: > > Thanks, guys. I had to ssh into each node and update core-site.xml on > every node. >

An extremely long time to run a query in EMR against S3 bucket of JSON files in GZ

Guys, I am trying to evaluate performance of a basic query – select count(*) from MY_TABLE I have 800 million records partitioned in S3 in subfolders by YEAR/DAY/HOUR in 14MB GZ JSON files I have a 2+1 node cluster m3.xlarge instance type set up in EMR. It is taking over 54 minutes to return

Re: Problem running queries on EMR - org.apache.drill.common.exceptions.UserRemoteException: SYSTEM ERROR: IllegalArgumentException: AWS Access Key ID

Thanks, guys. I had to ssh into each node and update core-site.xml on every node.

Re: Problem running queries on EMR - org.apache.drill.common.exceptions.UserRemoteException: SYSTEM ERROR: IllegalArgumentException: AWS Access Key ID

Make sure to use s3n and not just s3 in the connection of the storage plugin configuration. This changed a while back and have tripped some people up. --Andries > On Nov 16, 2015, at 3:21 PM, Mikhailau, Alex wrote: > > Guys, > > I keep getting the following exception: > > org.apache.drill.c

Re: Problem running queries on EMR - org.apache.drill.common.exceptions.UserRemoteException: SYSTEM ERROR: IllegalArgumentException: AWS Access Key ID

Hi Alex, Check out the "Setting up Apache Drill for use with S3" section of this article I wrote: http://www.dremio.com/blog/using-sql-to-interface-with-google-analytics-data-stored-on-amazon-s3/ It should handle what you're trying to do. In particular, make sure your storage plugin for s3 is set

Re: Proposal for Skipping Records

Agree with Julian. Users definitely should not have to interpret failure scenarios (i.e. warnings or errors) by having to look at Exceptions in the logs. On Mon, Nov 16, 2015 at 3:07 PM, Julian Hyde wrote: > Fair enough. > > Remember that end users don’t (in general) write Java functions and don

Problem running queries on EMR - org.apache.drill.common.exceptions.UserRemoteException: SYSTEM ERROR: IllegalArgumentException: AWS Access Key ID

Guys, I keep getting the following exception: org.apache.drill.common.exceptions.UserRemoteException: SYSTEM ERROR: IllegalArgumentException: AWS Access Key ID and Secret Access Key must be specified as the username or password (respectively) of a s3 URL, or by setting the fs.s3.awsAccessKeyId

Re: Proposal for Skipping Records

Fair enough. Remember that end users don’t (in general) write Java functions and don’t know what exceptions are. If your intent is to write a specification, you should describe in SQL terms what are error conditions for the built-in operators. > On Nov 16, 2015, at 3:01 PM, Sean Hsuan-Yi Chu w

Re: Proposal for Skipping Records

It is defined with respect to the behavior of function evaluation. If a function evaluation fails (throwing exceptions), we then considered something bad with the input record of this function. I agree people might have different beliefs on the definition. However, from the aspect of users' experi

Re: Proposal for Skipping Records

It would be useful if you could describe the different ways that a record can be “bad”. IIRC the SQL standard divides the conditions into errors and warnings. Examples of a warning would be a string column that is truncated because it is too large for a varchar(20), or numeric underflow when you

Proposal for Skipping Records

Hi all, We have worked on coming up a design document on this topic, which focuses on external design. Thanks Neeraja for summarizing a document as below: https://docs.google.com/document/d/1D4mDS-N722MZtkeYGSJbY-wUHG5E8IMT9rIMk1NHHGA/edit Please help take a look and offer some feedback.

Re: XML in Apache Drill

This is cool. I haven't taken a look yet but I will. Thanks! -- Jacques Nadeau CTO and Co-Founder, Dremio On Thu, Nov 12, 2015 at 2:35 AM, Magnus Pierre wrote: > Hello Drill Users, > A few weeks ago i had the pleasure of writing a small SAX parser for XML > that I was using in Storm to convert

Re: How to unsubscribe from the mail group ?

You can get digests; see https://mail-archives.apache.org/mod_mbox/drill-user/ > On Nov 15, 2015, at 6:24 PM, Kim Chew wrote: > > https://drill.apache.org/mailinglists/ > > On Sun, Nov 15, 2015 at 4:03 PM, ganesh wrote: > >> Hi, >> >>

can Drill also read zip files

I understand from the documentation, that Drill can read gzip compressed files. Is there a way to also read zip files instead? Tks. Uwe