[jira] [Created] (DRILL-7631) Updates to the Json Structure Parser

2020-03-09 Thread Paul Rogers (Jira)
Paul Rogers created DRILL-7631: -- Summary: Updates to the Json Structure Parser Key: DRILL-7631 URL: https://issues.apache.org/jira/browse/DRILL-7631 Project: Apache Drill Issue Type

Re: [DISCUSS] ExecConstants class refactoring

2020-03-02 Thread Paul Rogers
will never be fixed... Huh, but at least we're moving towards the goal :) Kind regards, Igor On Mon, Mar 2, 2020 at 8:01 PM Paul Rogers wrote: > Hi Igor, > > Great idea; I've been noticing that file has gotten excessively large. > > I wonder if we can split the file by topic ins

Re: [DISCUSS] ExecConstants class refactoring

2020-03-02 Thread Paul Rogers
Hi Igor, Great idea; I've been noticing that file has gotten excessively large. I wonder if we can split the file by topic instead of by the (often odd) naming hierarchy which has evolved over the years. For example, one file for internal server config options (thread counts, RPC stuff.)

Re: [GitHub] [drill] dobesv commented on issue #1994: DRILL-7203: Accept impersonation userName as form field & fix back button for query page

2020-02-28 Thread Paul Rogers
Oh my! This related to my change. We had a "format" method which build and threw the exception. I changed that to just build and return the exception, then I changed all callers to do the throwing. I also changed the method name to "create". The mystery is how the code passed the build and

Re: GitHub Actions Improvements?

2020-02-28 Thread Paul Rogers
Thanks for the suggestions. One more: probably not that helpful to the zillions of lines of download progress reports in the build log. Can we turn those off so it is easier to find the "high grade ore"? Thanks, - Paul On Friday, February 28, 2020, 12:45:39 PM PST, Dobes Vandermeer

Re: [GitHub] [drill] dobesv commented on issue #1994: DRILL-7203: Accept impersonation userName as form field & fix back button for query page

2020-02-28 Thread Paul Rogers
+1 Thanks, - Paul On Friday, February 28, 2020, 12:15:38 PM PST, GitBox wrote: dobesv commented on issue #1994: DRILL-7203: Accept impersonation userName as form field & fix back button for query page URL: https://github.com/apache/drill/pull/1994#issuecomment-592709083   Is

Re: [DISCUSS]: Proposed Agenda for Drill Hangout

2020-02-28 Thread Paul Rogers
If we have time, I'd like to outline the "SPI" project I'm working on. This project will create a "Service Provider Interface" for add-on code which follows Java practices used elsewhere, such as in the JDK, Presto etc. Another topic is our efforts to straighten out our data model as touched on

[jira] [Created] (DRILL-7613) Revise, harden the preliminary storage plugin upgrade facility

2020-02-26 Thread Paul Rogers (Jira)
Paul Rogers created DRILL-7613: -- Summary: Revise, harden the preliminary storage plugin upgrade facility Key: DRILL-7613 URL: https://issues.apache.org/jira/browse/DRILL-7613 Project: Apache Drill

[jira] [Created] (DRILL-7612) Modify the ExcelFormatConfig immutable

2020-02-26 Thread Paul Rogers (Jira)
Paul Rogers created DRILL-7612: -- Summary: Modify the ExcelFormatConfig immutable Key: DRILL-7612 URL: https://issues.apache.org/jira/browse/DRILL-7612 Project: Apache Drill Issue Type

[jira] [Created] (DRILL-7611) Minor improvements to the Option Manager

2020-02-26 Thread Paul Rogers (Jira)
Paul Rogers created DRILL-7611: -- Summary: Minor improvements to the Option Manager Key: DRILL-7611 URL: https://issues.apache.org/jira/browse/DRILL-7611 Project: Apache Drill Issue Type

[jira] [Created] (DRILL-7610) Allow user to specify table schema in Metastore

2020-02-26 Thread Paul Rogers (Jira)
Paul Rogers created DRILL-7610: -- Summary: Allow user to specify table schema in Metastore Key: DRILL-7610 URL: https://issues.apache.org/jira/browse/DRILL-7610 Project: Apache Drill Issue Type

[jira] [Created] (DRILL-7609) Display query in Web UI results page

2020-02-26 Thread Paul Rogers (Jira)
Paul Rogers created DRILL-7609: -- Summary: Display query in Web UI results page Key: DRILL-7609 URL: https://issues.apache.org/jira/browse/DRILL-7609 Project: Apache Drill Issue Type

[jira] [Created] (DRILL-7608) Web UI: Avoid wait UI for short queries

2020-02-26 Thread Paul Rogers (Jira)
Paul Rogers created DRILL-7608: -- Summary: Web UI: Avoid wait UI for short queries Key: DRILL-7608 URL: https://issues.apache.org/jira/browse/DRILL-7608 Project: Apache Drill Issue Type

[jira] [Created] (DRILL-7606) Support Hive client and JDBC APIs

2020-02-25 Thread Paul Rogers (Jira)
Paul Rogers created DRILL-7606: -- Summary: Support Hive client and JDBC APIs Key: DRILL-7606 URL: https://issues.apache.org/jira/browse/DRILL-7606 Project: Apache Drill Issue Type: Improvement

Re: Build / test cycle time improvement tips

2020-02-25 Thread Paul Rogers
. Thanks, - Paul [1] https://github.com/paul-rogers/drill/wiki/Launch-a-Drillbit-From-Eclipse On Tuesday, February 25, 2020, 10:53:28 AM PST, Dobes Vandermeer wrote: Hi, I've been experimenting with some improvements to the drill Web UI & REST API but I find that I spend a

[jira] [Created] (DRILL-7601) Shift column conversion to reader from scan framework

2020-02-24 Thread Paul Rogers (Jira)
Paul Rogers created DRILL-7601: -- Summary: Shift column conversion to reader from scan framework Key: DRILL-7601 URL: https://issues.apache.org/jira/browse/DRILL-7601 Project: Apache Drill Issue

[jira] [Created] (DRILL-7598) PostgreSQL-like functions for working with JSON

2020-02-23 Thread Paul Rogers (Jira)
Paul Rogers created DRILL-7598: -- Summary: PostgreSQL-like functions for working with JSON Key: DRILL-7598 URL: https://issues.apache.org/jira/browse/DRILL-7598 Project: Apache Drill Issue Type

[jira] [Created] (DRILL-7597) Read selected JSON colums as JSON text

2020-02-23 Thread Paul Rogers (Jira)
Paul Rogers created DRILL-7597: -- Summary: Read selected JSON colums as JSON text Key: DRILL-7597 URL: https://issues.apache.org/jira/browse/DRILL-7597 Project: Apache Drill Issue Type

Re: Drill Hangout Proposal

2020-02-20 Thread Paul Rogers
Hi All, Thanks much for thinking of us straggler PT folks. 7 AM is fine. I may not turn on the camera, however. As for the foreign language bit, you all speak (and write!) English so well we'd never know it was a second language unless you told us. I am always very impressed by your English

[jira] [Created] (DRILL-7593) Standardize local paths

2020-02-19 Thread Paul Rogers (Jira)
Paul Rogers created DRILL-7593: -- Summary: Standardize local paths Key: DRILL-7593 URL: https://issues.apache.org/jira/browse/DRILL-7593 Project: Apache Drill Issue Type: Improvement Affects

Re: [DISCUSS] Schema queries - solutions?

2020-02-17 Thread Paul Rogers
.apache.org/docs/describe/ Thanks, Igor On Tue, Feb 18, 2020 at 9:23 AM Paul Rogers wrote: > Hi All, > > Charles has a little PR,  #1978, that adds a convenient feature to his > HDF5 format reader: the ability to query the schema of the file. (It seems > that HDF5 is a bit like a zip

[DISCUSS] Schema queries - solutions?

2020-02-17 Thread Paul Rogers
Hi All, Charles has a little PR,  #1978, that adds a convenient feature to his HDF5 format reader: the ability to query the schema of the file. (It seems that HDF5 is a bit like a zip file: it contains a set of files. Unlike zip, each file is a data set with a schema.) Charles added a clever

[jira] [Created] (DRILL-7590) Refactor plugin registry

2020-02-17 Thread Paul Rogers (Jira)
Paul Rogers created DRILL-7590: -- Summary: Refactor plugin registry Key: DRILL-7590 URL: https://issues.apache.org/jira/browse/DRILL-7590 Project: Apache Drill Issue Type: Improvement

[jira] [Created] (DRILL-7583) Remove STOP status in favor of fail-fast

2020-02-13 Thread Paul Rogers (Jira)
Paul Rogers created DRILL-7583: -- Summary: Remove STOP status in favor of fail-fast Key: DRILL-7583 URL: https://issues.apache.org/jira/browse/DRILL-7583 Project: Apache Drill Issue Type

Re: Unit Tests for Drill Authenticators

2020-02-11 Thread Paul Rogers
Hi Charles, Thanks for pointing me to the PRs. Really great contributions! Very helpful as we create our Drillbit Docker support. I commented on DRILL-7573 about how we might test the htpasswd feature. There is lots of detail which, I hope, will help other contributors whose code needs

Re: DRILL-7575

2020-02-11 Thread Paul Rogers
Hi Charles, You are right. Wonderful to see multiple community contributions. My bad for confusing the two. Seems fine to commit this one; no reason to wait for a full test run. A quick check shows no references to this class or its name. Is this class found automatically based on its

Re: DRILL-7575

2020-02-11 Thread Paul Rogers
Hi Charles, Just did a review. Very handy improvement! Had a few minor comments; perhaps the author can address them before we do the merge. Whoever does the merge will want to do the "squash" with any revisions. Thanks, - Paul On Tuesday, February 11, 2020, 7:12:05 AM PST, Charles

[jira] [Created] (DRILL-7576) Fail fast in operators

2020-02-08 Thread Paul Rogers (Jira)
Paul Rogers created DRILL-7576: -- Summary: Fail fast in operators Key: DRILL-7576 URL: https://issues.apache.org/jira/browse/DRILL-7576 Project: Apache Drill Issue Type: Improvement Affects

[jira] [Created] (DRILL-7574) Generalize projection parser

2020-02-07 Thread Paul Rogers (Jira)
Paul Rogers created DRILL-7574: -- Summary: Generalize projection parser Key: DRILL-7574 URL: https://issues.apache.org/jira/browse/DRILL-7574 Project: Apache Drill Issue Type: Improvement

Re: [REVIEW]: Draft Board Report

2020-02-07 Thread Paul Rogers
Hi All, I agree on not mentioning Arrow integration; it is too early. As Igor says, it would be a big project, done over many releases. Arrow points out the need for other work that will occur shorter term. A big challenge now is that our "crazy complex type" feature does not really work.

Re: [REVIEW]: Draft Board Report

2020-02-07 Thread Paul Rogers
Hi Igor & Charles, Just my two cents. In years past we tried to do a release every three months using the "train model": those features that are ready are shipped, others wait for the next release. Not so bad if the delay is 3 months. Once the delay gets to be 6  months, everyone (rightly)

[jira] [Created] (DRILL-7572) JSON structure parser

2020-02-06 Thread Paul Rogers (Jira)
Paul Rogers created DRILL-7572: -- Summary: JSON structure parser Key: DRILL-7572 URL: https://issues.apache.org/jira/browse/DRILL-7572 Project: Apache Drill Issue Type: Improvement Affects

[jira] [Created] (DRILL-7567) Metastore enhancements

2020-02-03 Thread Paul Rogers (Jira)
Paul Rogers created DRILL-7567: -- Summary: Metastore enhancements Key: DRILL-7567 URL: https://issues.apache.org/jira/browse/DRILL-7567 Project: Apache Drill Issue Type: Improvement

[jira] [Created] (DRILL-7564) Revisit documentation structure

2020-02-02 Thread Paul Rogers (Jira)
Paul Rogers created DRILL-7564: -- Summary: Revisit documentation structure Key: DRILL-7564 URL: https://issues.apache.org/jira/browse/DRILL-7564 Project: Apache Drill Issue Type: Improvement

[jira] [Created] (DRILL-7563) Docker & Kubernetes Drill server container

2020-02-01 Thread Paul Rogers (Jira)
Paul Rogers created DRILL-7563: -- Summary: Docker & Kubernetes Drill server container Key: DRILL-7563 URL: https://issues.apache.org/jira/browse/DRILL-7563 Project: Apache Drill Issue

[jira] [Created] (DRILL-7560) Free leaked memory after failed unit tests

2020-01-30 Thread Paul Rogers (Jira)
Paul Rogers created DRILL-7560: -- Summary: Free leaked memory after failed unit tests Key: DRILL-7560 URL: https://issues.apache.org/jira/browse/DRILL-7560 Project: Apache Drill Issue Type

[jira] [Created] (DRILL-7559) Generalize provided schema handling for non-DFS plugins

2020-01-30 Thread Paul Rogers (Jira)
Paul Rogers created DRILL-7559: -- Summary: Generalize provided schema handling for non-DFS plugins Key: DRILL-7559 URL: https://issues.apache.org/jira/browse/DRILL-7559 Project: Apache Drill

[jira] [Created] (DRILL-7558) Generalize filter push-down planner phase

2020-01-30 Thread Paul Rogers (Jira)
Paul Rogers created DRILL-7558: -- Summary: Generalize filter push-down planner phase Key: DRILL-7558 URL: https://issues.apache.org/jira/browse/DRILL-7558 Project: Apache Drill Issue Type

[jira] [Created] (DRILL-7557) Revise "Base" storage plugin filter-push down listerner with a builder

2020-01-30 Thread Paul Rogers (Jira)
Paul Rogers created DRILL-7557: -- Summary: Revise "Base" storage plugin filter-push down listerner with a builder Key: DRILL-7557 URL: https://issues.apache.org/jira/browse/DRILL-7557 Proje

[jira] [Created] (DRILL-7556) Generalize the "Base" storage plugin filter push down mechanism

2020-01-30 Thread Paul Rogers (Jira)
Paul Rogers created DRILL-7556: -- Summary: Generalize the "Base" storage plugin filter push down mechanism Key: DRILL-7556 URL: https://issues.apache.org/jira/browse/DRILL-7556 Project: Ap

[jira] [Created] (DRILL-7555) Standardize Jackson ObjectMapper usage

2020-01-30 Thread Paul Rogers (Jira)
Paul Rogers created DRILL-7555: -- Summary: Standardize Jackson ObjectMapper usage Key: DRILL-7555 URL: https://issues.apache.org/jira/browse/DRILL-7555 Project: Apache Drill Issue Type

Re: UDF Ideas

2020-01-30 Thread Paul Rogers
esponse, Igor. I'll delve into those resources  > > -Andrew > > On Thu, Jan 30, 2020, 00:46 Igor Guzenko wrote: > >> Hello Andrew, >> >> I highly appreciate your interest in Drill. A very good place to start your >> journey is the GitHub wiki[1] created by Paul Ro

Re: [DISCUSS]: Thoughts

2020-01-30 Thread Paul Rogers
Hi All, Great discussion. Charles, thanks for starting it. Thanks Isabel for your insights. In my mind, question 4 (mission) is the most critical: what users do we want to serve? What problems should Drill solve for these users? We've touched on this topic in previous threads, perhaps we

Re: [ANNOUNCE] New PMC member: Bohdan Kazydub

2020-01-29 Thread Paul Rogers
Congratulations Bohdan, well deserved! - Paul On Wednesday, January 29, 2020, 09:41:21 AM PST, Arina Ielchiieva wrote: I am pleased to announce that Drill PMC invited Bohdan Kazydub to the PMC and he has accepted the invitation. Congratulations Bohdan and welcome! - Arina (on

Re: Apache Drill querying extremely slow

2020-01-28 Thread Paul Rogers
Hi Marc, I don't think any of the core team has used MinIO. Sounds like you are running Drill in Docker. So, the first question to others is: is anyone using Drill in Docker against plain old S3? If someone is, and does not hit the delay issues you describe, then we can narrow down the problem

Re: [DISCUSS] Format Plugin Interface

2020-01-26 Thread Paul Rogers
Hi Charles, Better APIs are always a good thing! The EVF ManagedReader interface has the minimum common denominator API: open, next (batch) and close. We can create extensions that provide more structure such as with your EasyEVFReader. For example: open() might: 1) fiddle with the

[jira] [Created] (DRILL-7553) Modernize type management

2020-01-26 Thread Paul Rogers (Jira)
Paul Rogers created DRILL-7553: -- Summary: Modernize type management Key: DRILL-7553 URL: https://issues.apache.org/jira/browse/DRILL-7553 Project: Apache Drill Issue Type: Improvement

Google F1 Query

2020-01-24 Thread Paul Rogers
Hi All, The Google F1 query engine [1] has many features similar to Drill and suggests many ideas that we might adopt. I believe F1 has been mentioned here before; the paper is worth a read if you've not yet done so. Like Drill, F1 derives from the original Dremel work [2] done at Google. Just

Re: Strange behavior with JOIN against correlated subquery

2020-01-23 Thread Paul Rogers
Hi Andy, When you have an array at the top level of your file, Drill treats the members as top-level tuples. Lateral is for the case where the data contains nested tuples. Try removing the LATERAL keyword. Thanks, - Paul On Thursday, January 23, 2020, 6:24:37 AM PST, Andy Grove

[jira] [Created] (DRILL-7545) Projection ambiguities in complex types

2020-01-21 Thread Paul Rogers (Jira)
Paul Rogers created DRILL-7545: -- Summary: Projection ambiguities in complex types Key: DRILL-7545 URL: https://issues.apache.org/jira/browse/DRILL-7545 Project: Apache Drill Issue Type: Bug

Re: DICT keys in projection

2020-01-21 Thread Paul Rogers
nts can be done. Hope, this answers your questions, do ask again if any clarification is needed. On Tue, Jan 21, 2020 at 12:11 AM Paul Rogers wrote: > Hi Bohdan, > > Thanks for your explanation.  My question comes from a little project I'm > working on to handle projection in EVF.

Re: DICT keys in projection

2020-01-20 Thread Paul Rogers
;key");`). In case when a key is an integer, schema path is the same as in case of array. Is this what you meant by "schema path parser does not recognize the syntax" or do you get an error? On Mon, Jan 20, 2020 at 5:16 AM Paul Rogers wrote: > Hi All, > > What did we decide

Re: Complex data types

2020-01-20 Thread Paul Rogers
ypes in a columnar way. I would like to understand more about the alternative. Could you please show, using examples, how new representation will work for complex columns? I'm interested in how to represent each row value for structs preserving low memory footprint and avoid an increase in networki

Complex data types

2020-01-19 Thread Paul Rogers
Hi All, We've been having a lively discussion about the pros and cons of Arrow. What are people's thoughts about another difficult topic: how we support complex types? Drill is a columnar engine. Columnar works best when data can be split into simple columns. The whole purpose of columnar is

Re: Connection Persistence (Cassandra Storage Plugin)

2020-01-19 Thread Paul Rogers
here is a link to my branch[1].  Feedback is definitely appreciated, but in no way is this ready for code review. Best, -- C [1]: https://github.com/cgivre/drill/tree/storage-cassandra <https://github.com/cgivre/drill/tree/storage-cassandra> > On Jan 17, 2020, at 5:37 PM, Paul Rogers wr

DICT keys in projection

2020-01-19 Thread Paul Rogers
Hi All, What did we decide to do about projecting DICT values? Drill allows us to project specific MAP members: SELECT myMap.x ... And, Drill allows projecting array members: SELECT myArray[3] ... I thought there was discussion of allowing Python-like syntax for projecting DICT values:

Re: Connection Persistence

2020-01-17 Thread Paul Rogers
Hi Charles, Poked around a bit. Turns out that the Cassandra client seems to work a bit differently than a JDBC client. From the JavaDoc page: "Session instances are thread-safe and usually a single instance is enough per application." Given this, you might be able to cache a single connection

Re: ScanBatchCreator

2020-01-17 Thread Paul Rogers
Hi Charles, Excellent question. The short answer is "no", but the longer answer is "we should fix this so the answer is yes." The reason the answer is "no" today is that Drill uses some odd magic to match up scan batch creators with plugins. In particular, the class of the second argument to

Re: Connection Persistence

2020-01-17 Thread Paul Rogers
Hi Charles, I've seen nothing like this in my travels through Drill code. My guess is that you'd have to create a connection pool. I'd also guess that connection pool implementations exist that could be reused. Drill is multi-threaded: any one Drillbit could be running many concurrent

Re: Apache Drill documentation updates

2020-01-14 Thread Paul Rogers
Hi Bridget, Thanks much for the update and for all your efforts on Drill documentation over the last several years. Thanks for squeezing in time to handle the release updates. Also, thanks for documenting how to update the documentation: the team should be able to keep information updated

Re: About integration of drill and arrow

2020-01-13 Thread Paul Rogers
Thanks Andy! Very helpful. You have hit on one of the questions that we've been wrestling with: which tools would consume Drill data as Arrow? More generally, what are the use cases for Arrow data interchange? Flight makes sense for transferring large data sets, such as in exchanges within a

Re: About integration of drill and arrow

2020-01-13 Thread Paul Rogers
Hi Andy & Charles, We've discussed two ways for Drill to interface to Arrow: either as an input or an output: Arrow Producer --> Drill --> Arrow Consumer Given how Drill works, the easiest of the two is to create a storage plugin to read from an Arrow Producer, perhaps using Arrow Flight

Re: About integration of drill and arrow

2020-01-13 Thread Paul Rogers
Hi Igor, Thanks much for volunteering to create some POCs for our various options! It is not entirely obvious what we want to test, so let's think about it a bit. We want to identify those areas that are either the biggest risk or benefit to performance. We want to do that without the cost of

[jira] [Created] (DRILL-7522) JSON reader (v1) omits null columns in SELECT *

2020-01-12 Thread Paul Rogers (Jira)
Paul Rogers created DRILL-7522: -- Summary: JSON reader (v1) omits null columns in SELECT * Key: DRILL-7522 URL: https://issues.apache.org/jira/browse/DRILL-7522 Project: Apache Drill Issue Type

Re: HTTP Plugin Revisions

2020-01-12 Thread Paul Rogers
Hi Charles, For cosmetic things like toString(), IMHO you can just leave them as they are for now. Thanks, - Paul On Sunday, January 12, 2020, 2:22:07 PM PST, Charles Givre wrote: Hey Paul, I'm about ready to push some revisions to the HTTP plugin, and I had a quick question. 

Re: About integration of drill and arrow

2020-01-12 Thread Paul Rogers
Hi All, As you've seen, I've been suggesting that we consider multiple choices for our internal data representation beyond just the current value vector layout and the "obvious" Arrow layout. And, that we consider out options based on where we see Drill adding value in the open source

Re: About integration of drill and arrow

2020-01-12 Thread Paul Rogers
Hi Volodymyr, You made a number of excellent points that we should remember as we continue our discussion. If I may paraphrase: 1. A conversion of our internal data layout will be complex. We can't expect to do it in a single step. Some readers may never convert. For a while, at least in a

Re: About integration of drill and arrow

2020-01-10 Thread Paul Rogers
won't have such problems because we will use own readers for most storage plugins. Although it will not matter at all if, as a result, we choose our own path of development. Thanks, Igor On Fri, Jan 10, 2020 at 9:17 PM Paul Rogers wrote: > Hi All, > > Glad to see the Arrow di

Re: About integration of drill and arrow

2020-01-10 Thread Paul Rogers
Hi Igor, You asked about the fixed-size block idea. This is the classic DB memory management mechanism: a "buffer pool" consisting of some number of fixed-size blocks. Memory allocation is simply a matter of grabbing a buffer from the pool. Freeing memory returns the buffer to the pool. Since

Re: About integration of drill and arrow

2020-01-10 Thread Paul Rogers
Hi All, Glad to see the Arrow discussion heating up and that it is causing us to ask deeper questions. Here I want to get a bit techie on everyone and highlight two potential memory management problems with Arrow. First: memory fragmentation. Recall that this is how we started on the EVF

Re: About integration of drill and arrow

2020-01-10 Thread Paul Rogers
Drill is slowly > dying and giving up to its competitors. > > NOTE: I want to encourage everyone to join the discussion and share vision > of what should be included in Drill 2.0 and what are strategic points we > want to achieve in the future. > > Kind regards, > Igor > > >

Re: About integration of drill and arrow

2020-01-10 Thread Paul Rogers
rill is slowly dying and giving up to its competitors. NOTE: I want to encourage everyone to join the discussion and share vision of what should be included in Drill 2.0 and what are strategic points we want to achieve in the future. Kind regards, Igor On Thu, Jan 9, 2020 at 10:12 PM Paul Rogers wr

Re: About integration of drill and arrow

2020-01-09 Thread Paul Rogers
development we could try their newest > features, like Flight which promises improved data transfers over the > network. > > Thanks, > Igor > On Wed, Jan 8, 2020 at 11:55 PM Paul Rogers > wrote: > > > Hi Igor, > > > > Before diving into design issues, it

Re: About integration of drill and arrow

2020-01-08 Thread Paul Rogers
if you can help us come up with a plan to solve these challenges! Thanks, - Paul [1] https://github.com/paul-rogers/drill-jig On Wednesday, January 8, 2020, 10:02:43 AM PST, Igor Guzenko wrote: Hello Paul, I totally agree that integrating Arrow by simply replacing Vectors usage

Re: HTTP Format Plugin Feedback

2020-01-08 Thread Paul Rogers
Hi Charles, Makes sense. Perhaps there is a middle ground. Perhaps the user creates a different plugin config for each API "group". That is, one for Google, another for Facebook, another for CatVideoTube. The config would include things like security tokens, REST format and so on. Then,

Re: About integration of drill and arrow

2020-01-08 Thread Paul Rogers
* org.apache.drill.exec.physical.rowSet - (Record batches management) org.apache.drill.exec.physical.resultSet - (Enhanced rowSet with memory mgmt) org.apache.drill.exec.physical.impl.scan - (Row set based scan) Thanks, Igor Guzenko On Mon, Dec 9, 2019 at 8:53 PM Paul Rogers wrote: > Hi All, > > Would be good to do so

Re: About integration of drill and arrow

2020-01-08 Thread Paul Rogers
Hi Igor, Before diving into design issues, it may be worthwhile to think about the premise: should Drill adopt Arrow as its internal memory layout? This is the question that the team has wrestled with since Arrow was launched. Arrow has three parts. Let's think about each. First is a direct

Re: About integration of drill and arrow

2020-01-08 Thread Paul Rogers
the team agrees, we can coordinate the incremental efforts done thus far.   Thanks, - Paul [1] https://github.com/paul-rogers/drill/wiki/BH-Future-Work On Wednesday, January 8, 2020, 10:02:43 AM PST, Igor Guzenko wrote: Hello Paul, I totally agree that integrating Arrow by simply

Re: Testing Storage Plugins

2020-01-08 Thread Paul Rogers
ork. Kind regards, Arina > On Jan 8, 2020, at 2:13 AM, Charles Givre wrote: > > Hi Paul, > In principle, I like the idea.  Currently, the main sticking point with > format and storage plugins is the protobuf.  But for that, they would be > completely “pluggable". > &g

Re: Testing Storage Plugins

2020-01-07 Thread Paul Rogers
Hi All, Wanted to chime in on this topic. We've long talked about the idea of building plugins separately from Drill itself; but have never had the resources to achieve this goal. Turns out Presto has a nice, simple way to build plugins separately from Presto itself. [1] If Drill were to

Re: Question about foreman restart

2020-01-07 Thread Paul Rogers
Hi Nitin, Thanks for letting us know about the OOM issues. These are serious and we should focus on finding the cause and fixing them. In general, it is the goal of the Drill project that Drill suffer no OOM errors on a cluster configured properly for your target workload. Thank you for

Re: HDF5 Format Plugin

2020-01-03 Thread Paul Rogers
Hi All, Perhaps we can wrap up the HDF5 and REST plugin PRs in their current form. Making small improvements seems to work better than trying to do too much in any one PR. Over time, we can use the V2 JSON reader in the REST plugin, but we should do so step-by-step after making sure the

[jira] [Created] (DRILL-7510) Incorrect String/number comparison with union types

2020-01-03 Thread Paul Rogers (Jira)
Paul Rogers created DRILL-7510: -- Summary: Incorrect String/number comparison with union types Key: DRILL-7510 URL: https://issues.apache.org/jira/browse/DRILL-7510 Project: Apache Drill Issue

[jira] [Created] (DRILL-7507) Convert fragment interrupts to exceptions

2020-01-01 Thread Paul Rogers (Jira)
Paul Rogers created DRILL-7507: -- Summary: Convert fragment interrupts to exceptions Key: DRILL-7507 URL: https://issues.apache.org/jira/browse/DRILL-7507 Project: Apache Drill Issue Type

[jira] [Created] (DRILL-7506) Simplify code gen error handling

2020-01-01 Thread Paul Rogers (Jira)
Paul Rogers created DRILL-7506: -- Summary: Simplify code gen error handling Key: DRILL-7506 URL: https://issues.apache.org/jira/browse/DRILL-7506 Project: Apache Drill Issue Type: Improvement

[jira] [Created] (DRILL-7503) Refactor project operator

2019-12-30 Thread Paul Rogers (Jira)
Paul Rogers created DRILL-7503: -- Summary: Refactor project operator Key: DRILL-7503 URL: https://issues.apache.org/jira/browse/DRILL-7503 Project: Apache Drill Issue Type: Improvement

Re: [ANNOUNCE] New Committer: Denys Ordynskiy

2019-12-30 Thread Paul Rogers
Congratulations Denys! - Paul On Monday, December 30, 2019, 4:25:49 AM PST, Arina Ielchiieva wrote: The Project Management Committee (PMC) for Apache Drill has invited Denys Ordynskiy to become a committer, and we are pleased to announce that he has accepted. Denys has been

[jira] [Created] (DRILL-7502) Incorrect/invalid codegen for typeof() with UNION

2019-12-30 Thread Paul Rogers (Jira)
Paul Rogers created DRILL-7502: -- Summary: Incorrect/invalid codegen for typeof() with UNION Key: DRILL-7502 URL: https://issues.apache.org/jira/browse/DRILL-7502 Project: Apache Drill Issue

[jira] [Resolved] (DRILL-7501) Drill 1.17 sqlTypeOf for a Map now reports STRUCT

2019-12-29 Thread Paul Rogers (Jira)
[ https://issues.apache.org/jira/browse/DRILL-7501?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Paul Rogers resolved DRILL-7501. Resolution: Won't Fix As explained on the dev list, the return value in this case was changed

[jira] [Resolved] (DRILL-5189) There's no documentation for the typeof() function

2019-12-29 Thread Paul Rogers (Jira)
[ https://issues.apache.org/jira/browse/DRILL-5189?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Paul Rogers resolved DRILL-5189. Resolution: Duplicate > There's no documentation for the typeof() funct

[jira] [Resolved] (DRILL-6362) typeof() lies about types

2019-12-29 Thread Paul Rogers (Jira)
[ https://issues.apache.org/jira/browse/DRILL-6362?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Paul Rogers resolved DRILL-6362. Resolution: Won't Fix > typeof() lies about ty

Re: [VOTE] Release Apache Drill 1.17.0 - RC2

2019-12-26 Thread Paul Rogers
Hi All, I'm late to the party (was distracted by a certain holiday). Downloaded the artifacts and ran though the examples in the first several sections of Chapter 8 of the Learning Apache Drill book. It turns out we have a regression in the sqlTypeOf() function. See DRILL-7499 and DRILL-7501.

[jira] [Created] (DRILL-7501) Drill 1.17 sqlTypeOf for a Map now reports STRUCT

2019-12-26 Thread Paul Rogers (Jira)
Paul Rogers created DRILL-7501: -- Summary: Drill 1.17 sqlTypeOf for a Map now reports STRUCT Key: DRILL-7501 URL: https://issues.apache.org/jira/browse/DRILL-7501 Project: Apache Drill Issue

[jira] [Created] (DRILL-7500) CTAS to JSON omits the final newline

2019-12-26 Thread Paul Rogers (Jira)
Paul Rogers created DRILL-7500: -- Summary: CTAS to JSON omits the final newline Key: DRILL-7500 URL: https://issues.apache.org/jira/browse/DRILL-7500 Project: Apache Drill Issue Type: Bug

[jira] [Created] (DRILL-7499) sqltypeof() function with an array returns "ARRAY", not type

2019-12-26 Thread Paul Rogers (Jira)
Paul Rogers created DRILL-7499: -- Summary: sqltypeof() function with an array returns "ARRAY", not type Key: DRILL-7499 URL: https://issues.apache.org/jira/browse/DRILL-7499 Project: Ap

[jira] [Created] (DRILL-7498) Allow the storage plugin editor window to be resizable

2019-12-26 Thread Paul Rogers (Jira)
Paul Rogers created DRILL-7498: -- Summary: Allow the storage plugin editor window to be resizable Key: DRILL-7498 URL: https://issues.apache.org/jira/browse/DRILL-7498 Project: Apache Drill

Re: [ANNOUNCE] New PMC member: Ihor Guzenko

2019-12-13 Thread Paul Rogers
Congratulations Ihor! - Paul On Friday, December 13, 2019, 05:39:14 AM PST, Volodymyr Vysotskyi wrote: I am pleased to announce that Drill PMC invited Ihor Guzenko to the PMC and he has accepted the invitation. Congratulations Ihor and welcome! - Vova (on behalf of Drill PMC)

[jira] [Created] (DRILL-7487) Retire unused OUT_OF_MEMORY iterator status

2019-12-12 Thread Paul Rogers (Jira)
Paul Rogers created DRILL-7487: -- Summary: Retire unused OUT_OF_MEMORY iterator status Key: DRILL-7487 URL: https://issues.apache.org/jira/browse/DRILL-7487 Project: Apache Drill Issue Type

[jira] [Resolved] (DRILL-5272) Text file reader is inefficient

2019-12-12 Thread Paul Rogers (Jira)
[ https://issues.apache.org/jira/browse/DRILL-5272?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Paul Rogers resolved DRILL-5272. Resolution: Fixed This issue was fixed when converting the text readers to use the result set

[jira] [Created] (DRILL-7486) Restructure row set reader builder

2019-12-12 Thread Paul Rogers (Jira)
Paul Rogers created DRILL-7486: -- Summary: Restructure row set reader builder Key: DRILL-7486 URL: https://issues.apache.org/jira/browse/DRILL-7486 Project: Apache Drill Issue Type: Improvement

<    1   2   3   4   5   6   7   8   9   10   >