Re: Drill & Caravel

The limit 0 form is highly to be preferred since that will give you information about columns for a view. This can be set in the dialect for SQL Alchemy. On Fri, May 13, 2016 at 11:03 AM, Neeraja Rentachintala < nrentachint...@maprtech.com> wrote: > Great, thanks John. I wil look forward for

Drill & Caravel

Great, thanks John. I wil look forward for an update on how drill queuing part goes : ) Btw with regards to metadata queries, Drill already supports metadata (both the limit 0 form and also show tables/show schemas which are served from the Information_schema). On Friday, May 13, 2016, John

Re: CTAS Out of Memory

Interesting, Wonder if it is related to the varchar issue Zelaine mentioned above, even with the specific columns specified the query plan shows a SELECT * being pushed to postgres, does the select not send down the specific columns? I will create another table with only the columns I want and

Re: CTAS Out of Memory

I am curious if this is a bug in the JDBC plugin. Can you try to change the output format to CSV? In that case we don't do any large buffering. Jason Altekruse Software Engineer at Dremio Apache Drill Committer On Fri, May 13, 2016 at 10:35 AM, Stefan Sedich wrote: >

Re: CTAS Out of Memory

Seems like it just ran out of memory again and was not hanging. I tried to append a limit 100 to the select query and it still runs out of memory, Just ran the CTAS against some other smaller tables and it works fine. I will play around with this some more on the weekend, I can only assume I am

Re: workspaces

Alternatively you can just create the storage plugin json file, delete the old one and post the new one using the REST API. See https://drill.apache.org/docs/rest-api/#storage > On May 13, 2016, at 10:05 AM, Vince Gonzalez

Re: workspaces

Ok i'll give it a try, thanks! 2016-05-13 12:05 GMT-05:00 Vince Gonzalez : > I have used a pipeline involving jq and curl to modify storage plugins via > the rest interface. The one below adds a workspace to the dfs plugin: > > curl -s localhost:8047/storage.json | jq

Re: workspaces

I have used a pipeline involving jq and curl to modify storage plugins via the rest interface. The one below adds a workspace to the dfs plugin: curl -s localhost:8047/storage.json | jq '.[] | select(.name == "dfs") | .config.workspaces |= . + { "nypdmvc": { "location":

Re: CTAS Out of Memory

Stefan, Can you share the query profile for the query that seems to be running forever ? you won't find it on disk but you can append .json to the profile web url and save the file. Thanks On Fri, May 13, 2016 at 9:55 AM, Stefan Sedich wrote: > Zelaine, > > It does, I

Re: CTAS Out of Memory

Zelaine, It does, I forgot about those ones, I will do a test where I filter those out and see how I go, in my test with a 12GB heap size it seemed to just sit there forever and not finish. Thanks On Fri, May 13, 2016 at 9:50 AM Zelaine Fong wrote: > Stefan, > > Does your

Re: Drill & Caravel

So with that Docker file, I got caravel working easily with test data (no drill yet) that will be weekend fun (and the pyodbc is already installed in the container, so now it's time to play!) So I started my docker image with: sudo docker run -it --rm --net=host

Re: workspaces

I have the restriction to not configure it via web console, so is there a way to configure them on the terminal? Cause in embed mode, i only create the files on the /tmp/ directory via terminal, also on the drill-override.conf file i use another path for the plugins (with

Re: CTAS Out of Memory

Stefan, Does your source data contain varchar columns? We've seen instances where Drill isn't as efficient as it can be when Parquet is dealing with variable length columns. -- Zelaine On Fri, May 13, 2016 at 9:26 AM, Stefan Sedich wrote: > Thanks for getting back to

Re: Queries and Timeout

Long running queries shouldn't timeout. This is most likely a bug. Is it reproducible ? Can you give more details about the query ? Thanks On Mon, May 9, 2016 at 12:30 PM, Subbu Srinivasan wrote: > What is the best way to implement queries that are long running? If

Re: CTAS Out of Memory

Thanks for getting back to me so fast! I was just playing with that now, went up to 8GB and still ran into it, trying to go higher to see if I can find the sweet spot, only got 16GB total RAM on this laptop :) Is this an expected amount of memory for not an overly huge table (16 million rows, 6

Re: workspaces

I believe Drill stores storage plugins in different places when running in embedded mode vs distributed mode. Embedded mode uses local disk and distributed mode uses Zookeeper. On Fri, May 13, 2016 at 9:08 AM, Odin Guillermo Caudillo Gallegos < odin.guille...@gmail.com> wrote: > The plugins are

Re: Drill & Caravel

So, without running this, but having it build successfully, this seems like a good place to start, it has caravel, and pyodbc all installed here. I will be playing more this weekend FROM ubuntu RUN apt-get update && apt-get install -y build-essential libssl-dev libffi-dev python-dev

Re: workspaces

You should start drill in distributed mode first and then configure the storage plugins. If you configure the storage plugins in embedded mode the information is stored in the tmp space instead of registered with ZK for the cluster to use. --Andries > On May 13, 2016, at 9:08 AM, Odin

Re: CTAS Out of Memory

I could not find anywhere this is mentioned in the docs, but it has come up a few times one the list. While we made a number of efforts to move our interactions with the Parquet library to the off-heap memory (which we use everywhere else in the engine during processing) the version of the writer

Re: workspaces

The plugins are working fine in the embbed mode, but when i start the drillbit on each server and connect via drill-conf i don't see them. Do i need to configure another parameter apart from the zookeeper servers in the drill-override.conf file? 2016-05-13 11:01 GMT-05:00 Andries Engelbrecht

CTAS Out of Memory

Just trying to do a CTAS on a postgres table, it is not huge and only has 16 odd million rows, I end up with an out of memory after a while. Unable to handle out of memory condition in FragmentExecutor. java.lang.OutOfMemoryError: GC overhead limit exceeded Is there a way to avoid this without

Re: workspaces

If Drill was correctly installed in distributed mode the storage plugin and workspaces will be used by the Drill cluster. Make sure the plugin and workspace was correctly configured and accepted. Are you using the WebUI or REST to configure the storage plugins? --Andries > On May 13, 2016, at

workspaces

Is there a way to configure workspaces on a distributed installation? Cause i only see the default plugin configuration but not the one that i created. Thanks

Re: Drill & Caravel

Lots of potential here. On May 13, 2016 9:42 AM, "Neeraja Rentachintala" < nrentachint...@maprtech.com> wrote: > Yes, the key thing is the SQL Alchemy layer. > I can see it more broadly being used than just Caravel. > > On Fri, May 13, 2016 at 6:13 AM, Erik Antelman > wrote:

Re: Drill & Caravel

Yes, the key thing is the SQL Alchemy layer. I can see it more broadly being used than just Caravel. On Fri, May 13, 2016 at 6:13 AM, Erik Antelman wrote: > Isn't this a matter of Drill<->SQLAlchemy. Such a support could likely > enable other frameworks. > > Would one think

Re: Drill & Caravel

Isn't this a matter of Drill<->SQLAlchemy. Such a support could likely enable other frameworks. Would one think that adaptation of SQLAlchemy to Drill is specific to Caravel? What subset of features from a RDBMS ORM is meaningfull, feasable and usefull to map to Drill. This sounds like a broad

Run multiple login

Hi Team, I have one data node and run one drillbit but if one person login in sqlline another person can not login on the same machine. Can you please suggest how can I enable this. Best Regards, ANKIT BEOHAR

Re: Drill & Caravel

I will be looking into this as well, thanks for sharing! On May 13, 2016 2:01 AM, "Nirav Shah" wrote: > I Hi Neeraja, > > I am interested in contributing if integration is not available. > Kindly let me know > > Regards, > Nirav > > On Thu, May 12, 2016 at 9:19 PM,

Re: Drill & Caravel

I Hi Neeraja, I am interested in contributing if integration is not available. Kindly let me know Regards, Nirav On Thu, May 12, 2016 at 9:19 PM, Neeraja Rentachintala < nrentachint...@maprtech.com> wrote: > Hi Folks > > Caravel is nice visualization tool recently open sourced by airbnb. Did >