Re: Drill and Parquet - Best practices - part 1

2015-11-03 Thread Stefán Baxter
Hi again, Are incrimental timestamp values (long) being encoded in Parquet as incremental values? (This option in parquet to refrain from storing complete numbers and store only the delta between numbers to save space) Regards, -Stefan On Mon, Nov 2, 2015 at 5:54 PM, Stefán Baxter wrote: > Hi

Re: Unable to use Drill JDBC Driver: NoClassDefFoundError ( sys/PStoreProvider ) on startup.

2015-11-03 Thread Martin Senne
Hi Andries, hi all, great stuff, drillbit=localhost instead of zk=local did the trick. Could you elaborate a little more, on what the difference is or where to retrieve additional information / documentation? Thx Martin 2015-11-02 19:26 GMT+01:00 Andries Engelbrecht : > Looks like

Query Type: SQL | PHYSICAL | LOGICAL

2015-11-03 Thread Ascot Moss
Hi, On the Drill GUI Query page, there is radio button named "Query Type" with 3 options SQL | PHYSICAL | LOGICAL Any idea what PHYSICAL | LOGICAL mean here? Regards

Re: Query Type: SQL | PHYSICAL | LOGICAL

2015-11-03 Thread USC
SQL one means the SQL language. Usually, we use this one. Logical, physical plans are plans (think of it as scripts) for drill engine to run. They are usually generated by machine, so they are used mainly for development/experiment purpose. Sent from my iPhone > On Nov 3, 2015, at 5:52 AM, A

Re: Unable to use Drill JDBC Driver: NoClassDefFoundError ( sys/PStoreProvider ) on startup.

2015-11-03 Thread Andries Engelbrecht
Drill can run in 2 modes - Embedded and Distributed. In Distributed Mode it uses ZK for keeping track of all the Drillbits in a Drill cluster. When connecting to a a distributed cluster you can either connect direct to a single drill or use ZK to get a list if all drillbits and then get connect

Re: Drill Hangout Happening

2015-11-03 Thread Parth Chandra
Hangout at the following link : https://plus.google.com/hangouts/_/event/ci4rdiju8bv04a64efj5fedd0lc

Re: "ETL" Options for Drill

2015-11-03 Thread John Omernik
What are the thoughts for combining all the data in the individual directories? Is there an option to replace like in Hive? On Mon, Nov 2, 2015 at 9:27 AM, Andries Engelbrecht < aengelbre...@maprtech.com> wrote: > You are probably on the right track to create a workspace for querying one > or

Line Parsing Errors and Skipping

2015-11-03 Thread John Omernik
I am doing some "active" loading of data into json files on MapRFS. Basically I have feeds pulling from a message queue and outputting the JSON messages. I have a query that is doing aggregations on all the data that seem to work 90% of the time. The other 10%, I get this error: Error: DATA_REA

Re: Line Parsing Errors and Skipping

2015-11-03 Thread mark charts
Hi. I read your dilemma. Would a trap in program to handle this ERROR or Exception work for you in this case and address it by skip around the trouble? My guess is you have a timing condition gone astray somewhere and you need to assure all states are timed correctly. But what do I know. Good lu

Re: Line Parsing Errors and Skipping

2015-11-03 Thread Andries Engelbrecht
See DRILL-2424 and DRILL-1131 Incomplete records/files can cause issues, in Drill 1.2 hey have added the ability to ignore data files with a .prefix. Perhaps copy files in over NFS using a . prefix and then rename once copied on the DFS. I had the same issue with Flume data streaming in and inc

Re: Line Parsing Errors and Skipping

2015-11-03 Thread John Omernik
Well I have one program writing data via Python to MapRFS in a directory that Drill is reading, so yes, I have two different programs reading and writing data. What I am looking for here is knowing I may have this scenario where a read may occur before a write is complete, can I just have Drill ig

Re: Line Parsing Errors and Skipping

2015-11-03 Thread John Omernik
Great feature and this fixes my problem. All I do is in my python script when I open a file, it opens with the .prefix. When I "close" it I rename it without the . prefix. Easy fix. Thanks for the pointer Andries! John On Tue, Nov 3, 2015 at 1:52 PM, Andries Engelbrecht < aengelbre...@maprtech.c

Errors on JSON File on field not being read.

2015-11-03 Thread John Omernik
I have some JSON that includes a field that is a array type, and some of those fields are json nulls. As seen in my other recent post about views and the "Please set `store.json.all_text_mode` I am getting this error here. Like I put in my other post, I don't want to set this at a system level, n