Re: Drill CTAS to single file

Another way to do it is to let sqlline save the csv file for you, this way you won't have to worry about Drill's parallelization, but you might need to make slight changes to your storage plugin to properly read sqlline's csv files. For example, I have the following CTAS: create table e as select

Re: CTAS over empty file throws NPE

Chandan actually found the JIRA: https://issues.apache.org/jira/browse/DRILL-3539 On Thu, Oct 22, 2015 at 10:25 AM, Neeraja Rentachintala < nrentachint...@maprtech.com> wrote: > Hsuan > Is there is a JIRA for this? > > On Thu, Oct 22, 2015 at 10:11 AM, Hsuan Yi Chu > wrote: > > > Hi, > > This i

Re: Exception with CSV storage format : Repeated types are not supported

Hey Chandan, I assume 'parquetlogs' contain parquet files, right ? what is the schema of 'parquetlogs' ? does it contain repeated fields ? thanks On Tue, Oct 27, 2015 at 2:40 AM, chandan prakash wrote: > Hi everyone, > Can anyone help how to write CTAS query with storage format as CSV without

Re: Drill Query Error

Hi Sanjeev, are you running a single query or multiple queries concurrently ? Thanks On Thu, Oct 29, 2015 at 9:24 AM, Andries Engelbrecht < aengelbre...@maprtech.com> wrote: > Information that will be more helpful are answering questions like the > following. > > What DFS are you trying to conn

Re: Help with Troubleshooting dense error message

The error message "index: 9604, length: 4 (expected: range(0, 8192))" suggests an error happened when Drill tried to access a memory buffer (most likely while writing an int or float value) This may be a bug actually exposed by that particular data record. You can try enabling verbose error loggin

Re: Help with Troubleshooting dense error message

ure I set that to get a > full stack trace. > > John > > On Wed, Nov 4, 2015 at 12:13 PM, Abdel Hakim Deneche < > adene...@maprtech.com> > wrote: > > > The error message "index: 9604, length: 4 (expected: range(0, 8192))" > > suggests an error ha

Re: Help with Troubleshooting dense error message

d was the json as text option at the session level and > I was setting it on both the pre drillbit reboot and the post drillbit > reboot sessions (I need that to query the data). > > On Wed, Nov 4, 2015 at 12:46 PM, Abdel Hakim Deneche < > adene...@maprtech.com> > wrote: &

Re: Help with Troubleshooting dense error message

> > > https://issues.apache.org/jira/browse/DRILL-4006 > > > > > > On Wed, Nov 4, 2015 at 12:16 PM, John Omernik wrote: > > > > > I am on MapR's 1.2.1 Package. > > > > > > > > > > > > > > > On Wed, Nov 4, 20

Re: Infinite pending Bug ?

Hello Boris, What version of Drill are you using ? Thanks On Fri, Nov 13, 2015 at 8:33 AM, Hsuan Yi Chu wrote: > Do you know if it is stuck at planning? > > On Fri, Nov 13, 2015 at 8:03 AM, Boris Chmiel < > boris.chm...@yahoo.com.invalid> wrote: > > > Hello every one, > > I reach an infinite p

Re: Infinite pending Bug ?

ation to the create View2 query in the "trim" section > makes the select * query succeed (for example changing '' to ' '). I'm not > sure if this will help narrow down the issue though. > > -Abhishek > > On Fri, Nov 13, 2015 at 8:55 AM, Abdel Ha

Re: Bug in Drill 1.3 CSV - please confirm

Hi Uwe, I couldn't reproduce the issue using the 1.3 release! can you send me the dummy test file you created, to my email address (you can't send it to an apache mailing list). Thanks On Tue, Nov 24, 2015 at 3:03 AM, Geercken, Uwe wrote: > I have downloaded 1.3 and made a quick test of the ne

Re: CSV Reader on 1.3

Hey John, What do you get when you run "select * from sys.version" ? extractHeader is false by default, so you need to explicitly set it to true. can you post your storage plugin configuration ? Thanks On Tue, Dec 1, 2015 at 6:04 AM, John Omernik wrote: > Hey all, > > Per my comment on https

Re: CSV Reader on 1.3

eplying. > > Is there away to select from a csv directory with extract header for only > that query or table? (Options?) > > > > On Wed, Dec 2, 2015 at 11:56 AM, Abdel Hakim Deneche < > adene...@maprtech.com> > wrote: > > > Hey John,

Re: Drill Query Problem

Hi Nirav, can you give us more information to help reproduce this issue ? thanks On Fri, Dec 4, 2015 at 3:42 AM, Nirav Shah wrote: > Hello, > > I am getting below error while running big query. > > === > > Error: SYSTEM ERROR: CompileException: File > 'org.a

Re: Announcing new committer: Kristine Hahn

Congrats Kristine :D On Fri, Dec 4, 2015 at 9:36 AM, Sudheesh Katkam wrote: > Congratulations and welcome, Kris! > > > On Dec 4, 2015, at 9:19 AM, Jacques Nadeau wrote: > > > > The Apache Drill PMC is very pleased to announce Kristine Hahn as a new > > committer. > > > > Kris has worked tireles

Re: Issues selecting last column on csv file with Drill 1.4

Hello Stefan, Did you create the .csvh file on a Windows machine ? We have a known issue DRILL-3726 where Drill doesn't interpret CRLF properly. On Tue, Dec 29, 2015 at 9:44 AM, Stefan Sedich wrote: > Hi, > > I have the following csv file and I

Re: Issue in developing UDF

According to Drill documentation: http://drill.apache.org/docs/adding-custom-functions-to-drill/ You need to copy both class jar and source jar of your UDF to $DRILL_HOME/jars/3rdparty/ did you do it ? On Tue, Jan 5, 2016 at 11:58 PM, Nirav Shah wrote: > Hi , > > > I am trying to extract info

Re: Issue in developing UDF

> outValue.buffer = > > >> buffer.reallocIfNeeded(valueDecoded.length); > > >> > outValue.start = 0; > > >> > outValue.end = valueDecoded.length; > > >> > buffer.setBytes(0, valueDecoded); &

Re: Too many open files

Hi Ian, Can you open up a JIRA for this ? is it easy to reproduce ? Thanks On Mon, Jan 11, 2016 at 8:59 AM, Ian Maloney wrote: > Hi, > > I've been running a lot of queries via jdbc/drill. I have four drillbits, > but I could not get the zk jdbc URL to work so I used: > jdbc:drill:drillbit=a-bi

Re: JDBC Driver - Possible regression

Stefán, Please reopen the JIRA and add a comment describing what you are seeing. Thanks On Wed, Jan 20, 2016 at 4:34 AM, Stefán Baxter wrote: > Hi again, > > We have verified that the error exists on master:head (1.5-SNAPSHOT). > > Regards, > -Stefan > > On Wed, Jan 20, 2016 at 10:39 AM, Stef

Re: Drill ODBC: format number on excel looks like Text

Could this help ? http://superuser.com/questions/385511/easy-way-to-one-off-import-data-with-different-decimal-separator-in-excel The solution is a bit old though and newer version of Excel may not offer the option anymore. On Sat, Jan 23, 2016 at 3:11 PM, Paolo Spanevello wrote: > Dear Ted, >

Re: CTAS error with CSV data

Does a select * on the same data also fail ? On Tue, Jan 26, 2016 at 9:44 AM, Matt wrote: > Getting some errors when attempting to create Parquet files from CSV data, > and trying to determine if it is due to the format of the source data. > > Its a fairly simple format of > "datetime,key,key,ke

Re: CTAS error with CSV data

nnectionImpl > ~~~ > > Is that index a byte or line offset? > > > On 26 Jan 2016, at 12:55, Abdel Hakim Deneche wrote: > > Does a select * on the same data also fail ? >> >> On Tue, Jan 26, 2016 at 9:44 AM, Matt wrote: >> >> Getting some errors when

Re: CTAS error with CSV data

at sqlline.SqlLine.begin(SqlLine.java:621) > at sqlline.SqlLine.start(SqlLine.java:375) > at sqlline.SqlLine.main(SqlLine.java:268) > ~~~ > > It also looks like if I run the SELECT from a bash shell as "sqlline -u > ... -f test.sql 2>&1 > test

Re: DRILL 1.4 - newline in strings not supported

Hey Nicolas, what kind of queries are you running on your csv file ? On Sun, Jan 31, 2016 at 12:14 PM, Nicolas Paris wrote: > Hello, > > I am trying to import a csv containing large texts. They contains newline > character "\n". > Apache Drill conplains about that. There is a jira issue opened

Re: DRILL 1.4 - newline in strings not supported

) > Usually I used postgresql or monetdb in order to mine the texts, but I am > benchmarking/studying apache drill too. > > Thanks, > > > 2016-02-01 15:54 GMT+01:00 Abdel Hakim Deneche : > > > Hey Nicolas, > > > > what kind of queries are you running on your

Re: DRILL 1.4 - newline in strings not supported

ple files > > ​Does it means newlines are incompatible with drill's distributed calculus > ? > > Do you have a fair number of files?​ > ​I have one 30GB csv file. I don't know how many parquet file it could > create as process crashes because of newlines. > I can imagine app

Re: DRILL 1.4 - newline in strings not supported

Paris wrote: > Abdel, > > select * on my csv file fails as well > > Thanks > > 2016-02-01 17:16 GMT+01:00 Abdel Hakim Deneche : > > > When you run a select * on your csv file, does it succeed or fail ? > > > > On Mon, Feb 1, 2016 at 7:53 AM, Nicolas Paris >

Re: Avro reader - Possible regression in 1.5-SNAPSHOT

Hi Stefán, Can you open a JIRA for this, please ? Thanks On Tue, Feb 2, 2016 at 6:21 AM, Stefán Baxter wrote: > Hi, > > I can confirm that this same query+avro-files work in 1.4 so this is > probably a regression > > Regards, > -Stefan > > On Tue, Feb 2, 2016 at 1:59 PM, Stefán Baxter > wrot

Re: Avro reader - Possible regression in 1.5-SNAPSHOT

Thanks On Tue, Feb 2, 2016 at 9:03 AM, Stefán Baxter wrote: > https://issues.apache.org/jira/browse/DRILL-4339 > > On Tue, Feb 2, 2016 at 4:46 PM, Abdel Hakim Deneche > > wrote: > > > Hi Stefán, > > > > Can you open a JIRA for this, please ? > > >

Re: UDF - BooleanHolder

It's called BitHolder On Wed, Feb 3, 2016 at 3:12 PM, Nicolas Paris wrote: > Hello, > > Hello, > > I would like to create a user defined function that would return a boolean > value. > Use case would be : > > SELECT * FROM x WHERE MyFunction(); > > I haven't found any BooleanHolder in order to.

Re: Query Planning and Directory Pruning

Hey John, can you try an explain plan for both queries and see how much times it takes ? for example, for the first query you would run: *explain plan for* select count(1) from `data/2016-02-03`; It can also be helpful if you could share the query profiles for both queries. Thanks On Thu, Feb

Re: Dealing with files created in Windows

is dos2unix an option ? On Mon, Feb 8, 2016 at 9:56 AM, John Omernik wrote: > Are there any decent tricks for dealing with Windows based text files (that > use /r/n as the line ending rather than just /n) > > Right now my last field has /r showing up, and I'd like to not have that > there, I gue

Re: Query Planning and Directory Pruning

the query. Indicating to me the issue isn't in the plan that was > > created, but the actual planning process. (Let me know if you disagree or > > still need to see the plan, like I said, the actual plans were exactly > the > > same) > > > > > > John. >

Re: Source for drill's calcite?

You can find the r10 branch here: https://github.com/mapr/incubator-calcite/tree/DrillCalcite1.4.0 On Tue, Feb 9, 2016 at 8:00 AM, Jason Altekruse wrote: > I can't find the latest version either, but this is the r9 branch. I don't > think any very major changes happened in the last update (it's

expected behavior when using wild cards in table name

I have the following table tpch100/lineitem that contains 97 parquet files: tpch100/lineitem/part-m-0.parquet tpch100/lineitem/part-m-1.parquet tpch100/lineitem/part-m-2.parquet ... tpch100/lineitem/part-m-00096.parquet I can run the following queries: SELECT COUNT(*) FROM `tpch100/

Re: expected behavior when using wild cards in table name

d. > > On Thu, Feb 11, 2016 at 9:42 AM, Abdel Hakim Deneche < > adene...@maprtech.com> > wrote: > > > I have the following table tpch100/lineitem that contains 97 parquet > files: > > > > tpch100/lineitem/part-m-0.parquet > > tpch100/linei

Re: handling Date

Would EXTRACT help ? http://drill.apache.org/docs/date-time-functions-and-arithmetic/#extract On Fri, Feb 12, 2016 at 7:15 AM, Paolo Spanevello wrote: > Dear All, > > I have a field with a date like this: > > > Date > > 2016/01/16 09:44:28 UTC > I would like to split it in Date, Time and remove

Re: One single query for more files JSON

Yes, it should work. On Fri, Feb 12, 2016 at 7:31 AM, Paolo Spanevello wrote: > Dear All, > > Could i've a single query for more json files ? > > Example: > >- /user/folder1/file1.json >- /user/folder2/file2.json >- /user/folder3/file3.json > > Query: > SELECT * FROM /user/*/file*.js

Re: One single query for more files JSON

Of course, if the schema changes between files, this will most likely cause the query to fail On Fri, Feb 12, 2016 at 7:42 AM, Abdel Hakim Deneche wrote: > Yes, it should work. > > On Fri, Feb 12, 2016 at 7:31 AM, Paolo Spanevello > wrote: > >> Dear All, >> >&

Re: Drill Doc Question: Multi Tenant Clusters

No, it's the maximum number of threads each drillbit will be able to spawn for every major fragment of a query. If you run a query on a cluster of 32 core machines, and the query plan contains multiple major fragments, each major fragment will have "at most" 32 x 0.7= 23 minor fragments (or thread

Re: Drill Doc Question: Multi Tenant Clusters

re saying that we should be taking 1 > drill bit per node * 32 * 0.7 ... correct? > > Quote from the docs: > number of active drillbits (typically one per node) * number of cores per > node * 0.7 > > On Mon, Feb 15, 2016 at 11:15 AM, Abdel Hakim Deneche < > adene...@ma

Re: Drill Doc Question: Multi Tenant Clusters

so yes, you are correct, you should set it to 1 x 32 x 0.7 Btw, Drill should already have set this option to 32 x 0.7 On Mon, Feb 15, 2016 at 11:36 AM, Abdel Hakim Deneche wrote: > Don't be, it took me quite some time to figure out this one either =P > > the "number of activ

Re: Drill Doc Question: Multi Tenant Clusters

re nodes vs 24 core nodes for example) > > > > On Mon, Feb 15, 2016 at 1:37 PM, Abdel Hakim Deneche < > adene...@maprtech.com> > wrote: > > > so yes, you are correct, you should set it to 1 x 32 x 0.7 > > > > Btw, Drill should already have set this option t

Re: One single query for more files JSON

> > > Are you running on Windows? If so, perhaps this is DRILL-4305? > > > > > > -- Zelaine > > > > > > On Fri, Feb 12, 2016 at 8:00 AM, Paolo Spanevello < > paolosp...@gmail.com> > > > > > > wrote: > > > >

Re: One single query for more files JSON

"tsv" > ], > "delimiter": "\t" > }, > "parquet": { > "type": "parquet" > }, > "json": { > "type": "json" > }, > "avro&qu

Re: what am I missing?

To run Drill in distributed mode you need to have Zookeeper up and running. This shouldn't be too complicated, you can find more details here: https://zookeeper.apache.org/doc/r3.1.2/zookeeperStarted.html#sc_InstallingSingleMode On my Mac I used brew and it took care of everything. On Mon, Feb 2

Re: Drill join performance

Hello Dmitry, Welcome to Drill's community :) What version of Drill are you using ? Also, can you share the query profile of your query, it helps to show what taking most of the time. Thanks On Mon, Feb 22, 2016 at 10:54 AM, Dmitry Krivov wrote: > Hello > > I have load (as CTAS) into parquet-

Re: Date Format conversion

more precisely you can use TO_DATE . The following worked for me: TO_DATE('01/25/2016', 'MM/dd/') On Tue, Feb 23, 2016 at 10:48 AM, Neeraja Rentachintala < nrentachint...@maprtech.com> wrote: > Please refer to

Re: Drill error with large sort

Short answer: increase the value of planner.memory.max_query_memory_per_node, by default it's set to 2GB, try setting to 4 or even 8GB. This should get the query to pass. On Thu, Feb 25, 2016 at 5:24 PM, Jeff Maass wrote: > > If you are open to changing the query: > # try removing the functio

Re: Drill error with large sort

y/ On Thu, Feb 25, 2016 at 5:26 PM, Abdel Hakim Deneche wrote: > Short answer: > > increase the value of planner.memory.max_query_memory_per_node, by default > it's set to 2GB, try setting to 4 or even 8GB. This should get the query to > pass. > > On Thu, Feb 25

Re: The praises for Drill

Looking forward to reading the paper! On Fri, Feb 26, 2016 at 10:19 AM, Parth Chandra wrote: > Welcome back Edmon, and thanks for the praise :). Hope to see you on the > next hangout. > > On Thu, Feb 25, 2016 at 7:27 PM, Edmon Begoli wrote: > > > Hello fellow Driilers, > > > > I have been inact

Re: Drill with String Aggregation

You can always develop a User Defined Aggregate Function: http://drill.apache.org/docs/develop-custom-functions/ Thanks On Wed, Mar 9, 2016 at 12:29 AM, Bosung Seo wrote: > Hello, > > I found that Drill doesn't support string_agg function yet. > Is there another way to query as the string_agg

Re: Drill with String Aggregation

project. Wish lists, challenges, new > functionality can be developed openly, and maybe (depending on the answer > to my first question) some of these could even make it into the Drill > project itself. Almost like a developer preview of self contained > functions. > > What are the

Re: NumberFormatException with cast to double?

Looks like the COALESCE function is the source of the problem. Passing a double (0.0) instead of an int (0) as a second expression solved the problem for me: CAST(COALESCE(t_total, 0.0) AS double) On Fri, Mar 11, 2016 at 12:45 AM, Matt wrote: > ~~~ > 00-01 Project(date_tm=[CAST($23):TIMES

Question about Text Files documentation

In this documentation page: http://drill.apache.org/docs/text-files-csv-tsv-psv/ We can read the following: Using a distributed file system, such as HDFS, instead of a local file > system to query the files also improves performance because currently Drill > *does > not split* files on block sp

Re: unable to start Drill 1.6.0

Hi Shankar, mailing list doesn't allow attachments, can you post the file in some public place and share link ? Thanks On Thu, Mar 17, 2016 at 1:51 PM, Shankar Mane wrote: > I am not able to start drill 1.6.0. Please find the attached file for more > details. > > -- Abdelhakim Deneche Sof

Re: Drill join performance

One quick note here, I don't think partitioning LINEORDER table on LO_ORDERDATE would help this query. If you look at the query profile you will see that Drill is reading everything from LINEORDER. On Fri, Mar 18, 2016 at 7:57 AM, Dmitry Krivov wrote: > Just for info : > > After recreating table

Re: Question about Text Files documentation

MapRFS. > - Drill doesn't currently split files that are sourced from the local file > system. > > -- > Jacques Nadeau > CTO and Co-Founder, Dremio > > On Wed, Mar 16, 2016 at 4:02 AM, Abdel Hakim Deneche < > adene...@maprtech.com> > wrote: > > > In this

Re: unable to start Drill 1.6.0

Easiest fix when Drill fails to load a storage plugin is to delete the existing configurations. Deleting /tmp/drill/ should resolve this. I know this may not be practical in some cases, and other developers may give you a better solution. On Thu, Mar 17, 2016 at 2:13 PM, Shankar Mane wrote: > *

Re: unable to start Drill 1.6.0

"com.companyname.drill.*" > ] > } > } > */ > > *But DRILL GET SHUTDOWN on all nodes.* > > > > > *Please help me to resolved this issue. Or suggest any other way to invoke > my custome UDFs. * > > > > > > On Thu, Mar 17, 2016 at 6:50

Re: Code too large

This exception states that the code generated for the project is too big for that Java compiler. Can you share the query that caused this failure ? On Thu, Mar 24, 2016 at 1:27 PM, Edmon Begoli wrote: > Does anyone know what might be causing this exception: > > *Error: SYSTEM ERROR: CompileExcep

Re: IllegalStateException: Memory was leaked by query - Drill error on wide table, but OK on a narrow but longer table.

Hey Edmon, Can you search the logs for the errorId displayed with this error ? I've seen a similar issue when external sort fails to spill to disk (no space left on disk), it would leak memory and display this error message instead of the original issue (problem spilling to disk). I will open a J

Re: simple join failing for SQL server.

Hi Devender, Whenever you see such errors, it's Drill's internal memory accounting reporting a memory leak. This is "always" a bug, but you did fix so many of them(*) since 1.4 and we even improved the memory allocator in 1.5. Do you want to try again on the latest version and see if you still see

Re: simple join failing for SQL server.

"we" did fix so many of them. =P On Thu, Mar 31, 2016 at 8:12 AM, Abdel Hakim Deneche wrote: > Hi Devender, > > Whenever you see such errors, it's Drill's internal memory accounting > reporting a memory leak. This is "always" a bug, but you did fix

Re: simple join failing for SQL server.

> > Allocator(op:0:0:3:JdbcSubScan) 100/73728/569344/100 > > (res/actual/peak/limit) > > > > > > > > Regards, > > Devender > > > > On Thu, Mar 31, 2016 at 12:45 PM, Devender Yadav > > wrote: > > > >> Yes

Re: simple join failing for SQL server.

gt; > > Regards, > > Devender > > > > On Thu, Mar 31, 2016 at 12:43 PM, Abdel Hakim Deneche < > > adene...@maprtech.com> wrote: > > > >> "we" did fix so many of them. =P > >> > >> On Thu, Mar 31, 2016 at 8:12 AM, Abdel Hak

Re: How to modify connection timeout delay ?

Hi Damien, Like Jason said, we have a heartbeat mechanism that should've prevented this issue all together, so I'm interested to learn how this is happening. We've seen this happen many times but so far we were never able to reproduce it. Could you give us more details so we can reproduce the iss

Re: Reading Avro Arrays

pull requests are fine. You still need a JIRA though On Sun, Apr 3, 2016 at 8:03 AM, Johannes Schulte wrote: > I now extended the AvroFormatTest-Suite by two unit tests that show that > > * Flattening of primitive array works as expected > * Flattening of arrays of records does not work properly

Re: How to modify connection timeout delay ?

On Tue, Apr 5, 2016 at 7:59 AM, COUERON Damien (i-BP - MICROPOLE) < damien.coueron_s...@i-bp.fr> wrote: > Despite the log below, what kind of details are you interested in ? > > > > -Message d'origine- > De : Abdel Hakim Deneche [mailto:adene...@maprtech.com

Re: Apache Drill issues

one quick note here, we have a known issue when the foreman bit dies [1] but a fix is being reviewed and should be merged soon into master branch. Once this issue is resolved, the client will fail the query when the foreman dies. [1] https://issues.apache.org/jira/browse/DRILL-3743 On Tue, Apr 5,

Re: Where is this C++ API mentioned in Drill docs

I believe the source code is part of Drill distribution, in the following folder: contrib/native/client It also contains an example folder with a fairly good demonstration on how to use the API to submit queries to Drill. Thanks On Sat, Apr 9, 2016 at 2:23 PM, Devender Yadav wrote: > Hi All,

Re: problem running drill in 10minutes tutorial on macpro

Hey, Unfortunately, the Apache mailing list blocks attachments so we are not able to see the error message. If you want you can just copy past the error messages here, or share a link to the screenshots. Thanks On Wed, May 4, 2016 at 4:23 PM, Rita Kuo wrote: > Hi, > > I tried follow the Drill

Re: workspaces

I believe Drill stores storage plugins in different places when running in embedded mode vs distributed mode. Embedded mode uses local disk and distributed mode uses Zookeeper. On Fri, May 13, 2016 at 9:08 AM, Odin Guillermo Caudillo Gallegos < odin.guille...@gmail.com> wrote: > The plugins are w

Re: Queries and Timeout

Long running queries shouldn't timeout. This is most likely a bug. Is it reproducible ? Can you give more details about the query ? Thanks On Mon, May 9, 2016 at 12:30 PM, Subbu Srinivasan wrote: > What is the best way to implement queries that are long running? If queries > take a long > time

Re: CTAS Out of Memory

Stefan, Can you share the query profile for the query that seems to be running forever ? you won't find it on disk but you can append .json to the profile web url and save the file. Thanks On Fri, May 13, 2016 at 9:55 AM, Stefan Sedich wrote: > Zelaine, > > It does, I forgot about those ones,

Re: test

your test succeeded ;) On Tue, May 17, 2016 at 10:17 AM, Khurram Faraaz wrote: > test email > -- Abdelhakim Deneche Software Engineer Now Available - Free Hadoop On-Demand Training

Re: Issue with Queries Hanging

One question about the missing query profile: do you store the query profiles in the local file system or the distributed file system ? On Mon, May 23, 2016 at 9:31 AM, John Omernik wrote: > Hey all, this is separate, yet related issue to my other posts RE Parquet, > however, I thought I'd post

Re: [ANNOUNCE] New PMC Chair of Apache Drill

Congrats Parth ! On Wed, May 25, 2016 at 9:15 AM, Zelaine Fong wrote: > Congratulations, Parth. Looking forward to working with in your new role > :). > > -- Zelaine > > On Wed, May 25, 2016 at 9:02 AM, Jinfeng Ni wrote: > > > Big congratulations, Parth! > > > > Thank you, Jacques, for your co

Re: Reading and converting Parquet files intended for Impala

the new parquet reader, the complex reader, is disabled by default. You can enable it by setting the following option to true: store.parquet.use_new_reader On Sat, May 28, 2016 at 4:56 AM, John Omernik wrote: > I remember reading that drill uses two readers. One for certain cases ( I > think

Re: Reading GC Logs

My understanding (which is incomplete) is that both the "new reader" and "dictionary encoding" are not stable yet and can cause failures or worse, incorrect data. That's why they are disabled by default. The "Allocation Failure" means that the JVM had to run a Full GC because it couldn't allocate

Re: Hangout link?

Sorry about the delay, there you go: https://plus.google.com/hangouts/_/event/ci4rdiju8bv04a64efj5fedd0lc On Tue, May 31, 2016 at 9:57 AM, John Omernik wrote: > > -- Abdelhakim Deneche Software Engineer Now Available - Free Hadoop On-Demand Training

Re: Profiles Gone in Web UI: The great profile heist

are you storing the profiles in a local folder or in nfs ? On Tue, May 31, 2016 at 12:49 PM, John Omernik wrote: > I am scratching my head at this one... I made some minor changes to my > drill-env.sh to enable gclogging, and was using the profiles in the webui > just fine. Due to some previous

Re: Guidelines for planner.memory.max_query_memory_per_node

I don't know about any specific guidelines for this options, but what I know is that it only affects the sort operator, and it's related to direct memory not heap memory. On Wed, Jun 1, 2016 at 1:20 PM, John Omernik wrote: > I am reposting this question here as well. (I posted on the MapR Comm

Re: queries take over 2 min

sometimes, if you have an issue in one of your storage plugin it affects all queries even those not querying that specific plugin. Do you have any enable storage plugin that's causing issues ? On Wed, Jun 1, 2016 at 2:21 PM, Scott Kinney wrote: > i'm running queries on local json files and queri

Re: Guidelines for planner.memory.max_query_memory_per_node

ested in guidelines here, keeping it at 2GB with such beefy nodes > seems to be a waste. > > John > > On Wed, Jun 1, 2016 at 3:38 PM, Abdel Hakim Deneche > > wrote: > > > I don't know about any specific guidelines for this options, but what I > > know is tha

Re: How to specify Drill JDBC connection timeout or JDBC Query timeout

For connection timeout, there are configuration options that you can set in drill-override.conf that affect how much time the Drill client will try to connect to the server, but even then the client could actually block forever (I've seen it happen on an internal tool). Drill test framework has a

Re: CTAS on MySQL*

MySQL storage plugin is indeed read only, you cannot create a table on mysql through Drill. Thanks On Fri, Jun 3, 2016 at 10:05 AM, Shankar Mane wrote: > As we know, we can able to read MySQL data using drill mysql storage > plugin. But it seems it is Read Only and No Write permissions . > > Ca

Re: How to specify Drill JDBC connection timeout or JDBC Query timeout

api level timeout? > > Thanks, > Hao > > On Thu, Jun 2, 2016 at 6:42 PM, Abdel Hakim Deneche > > wrote: > > > For connection timeout, there are configuration options that you can set > in > > drill-override.conf that affect how much time the Drill client will try &

Re: Memory Settings for a Non-Sorted Failed Query

Running out of heap could also make a Drillbit become irresponsive, eventually it will die after printing the following message in it's drillbit.out: Unable to handle out of memory condition in FragmentExecutor You may want to check your drillbits' drillbit.out for such message On Mon, Jun 13, 2

Re: Drill Plugin Update

SkipFirstLine is an optional parameter with a default value of "false", so when the parameter "disappears" it's actually equivalent to setting it to false. On Tue, Jun 21, 2016 at 4:30 AM, Kumar Anil7/DEL/TCS wrote: > > Hi, > > > I am using drill 1.4.0 in MapR5.1 cluster. I am trying to update

Re: Information about ENQUEUED state in Drill

Most likely planing is taking longer to finish. Once it's done, it should move to either ENQUEUED if the queuing was enabled or RUNNING if it was disabled. One easy way to confirm if planing is indeed taking too long is to just run a "EXPLAIN PLAN FOR " and see how long it takes to finish. On Fri

Re: Parquet Block Size Detection

some answers inline: On Fri, Jul 1, 2016 at 10:56 AM, John Omernik wrote: > I looked at that, and both the meta and schema options didn't provide me > block size. > > I may be looking at parquet block size wrong, so let me toss out some > observations, and inferences I am making, and then others

Re: Parquet Block Size Detection

Just make sure you enable parquet metadata caching, otherwise the more files you have the more time Drill will spend reading the metadata from every single file. On Fri, Jul 1, 2016 at 11:17 AM, John Omernik wrote: > In addition > 7. Generally speaking, keeping number of files low, will help in

Re: array in json with mixed values (int and float)

This may help: https://mail-archives.apache.org/mod_mbox/drill-user/201605.mbox/%3ccakofcwrdwr2ytzk4a5dglboq0p0ygwtox6nrmsgrm6nskk3...@mail.gmail.com%3E On Fri, Jul 1, 2016 at 1:26 PM, Parth Chandra wrote: > If you mean the REST api, then yes, there is no session maintained unless > impersonati

Re: Initial Feed Back on 1.7.0 Release

answers inline. On Tue, Jul 5, 2016 at 8:39 AM, John Omernik wrote: > Working with the 1.7.0, the feature that I was very interested in was the > fixing of the Metadata Caching while using user impersonation. > > I have a large table, with a day directory that can contain up to 1000 > parquet fi

Re: Initial Feed Back on 1.7.0 Release

ng, you want to ensure that one days updates don't clobber another > > days) > > > > Just a thought on that. > > > > Yep, the incremental issue would come into play here. Are there any > design > > docs or JIRAs on the incremental updates to metadata? >

Re: Number of records per batch

hey Eric, Can you give more information about what you are trying to achieve ? Thanks On Tue, Jul 5, 2016 at 3:41 PM, Eric Fukuda wrote: > Hi, > > Does anyone know if there is a way to increase or specify the number of > records per batch manually? > > Thanks, > Eric > -- Abdelhakim Denec

Re: Number of records per batch

ebugger, there seem to be 4096 records in a > batch. Can this be 8192 or larger? > > On Tue, Jul 5, 2016 at 6:47 PM, Abdel Hakim Deneche > > wrote: > > > hey Eric, > > > > Can you give more information about what you are trying to achieve ? > > > >

Re: Number of records per batch

y restriction? It doesn't have to be a configuration > option. I don't mind changing and compiling the code. > > On Tue, Jul 5, 2016 at 8:55 PM, Abdel Hakim Deneche > > wrote: > > > Unfortunately I don't think there is way to do it. > > > > On Tue, Jul 5

  1   2   >