Thank you Aman for organizing and to MapR for hosting! On Wed, Sep 20, 2017 at 11:12 AM, Aman Sinha <amansi...@apache.org> wrote:
> Thanks to all the folks who attended the hackathon - both local and remote. > For the remote attendees, you missed out on a good dinner :) > > We had a day of excellent discussion on several topics: Resource > management, operator level performance improvements, TPC-DS coverage, > metadata management, concurrency, usability and error handling, storage > plugins + rest APIs. It will take a couple of days to compile all the > notes and we will post them. > > Since the focus was more in-depth discussion rather than breadth, and 1 day > is clearly not adequate, some topics were left out. We can continue those > discussions on the dev list / hangout or if it can wait, possibly do it in > a future hackathon. > > -Aman > > On Fri, Sep 15, 2017 at 2:54 PM, Charles Givre <cgi...@gmail.com> wrote: > > > Hi Pritesh, > > What time do you think you’d want me to present? Also, should I make > some > > slides? > > Best, > > — C > > > > > On Sep 15, 2017, at 13:23, Pritesh Maker <pma...@mapr.com> wrote: > > > > > > Hi All > > > > > > We are looking forward to hosting the hackathon on Monday. Just a few > > updates on the logistics and agenda > > > > > > • We are expecting over 25 people attending the event – you can see the > > attendee list at the Eventbrite site - https://www.eventbrite.com/e/ > > drill-developer-day-sept-2017-registration-7478463285 > > > > > > • Breakfast will be served starting at 8:30AM – we would like to begin > > promptly at 9AM > > > > > > • The agenda has been updated to reflect the speakers (see the update > in > > the sheet - https://docs.google.com/spreadsheets/d/ > > 1PEpgmBNAaPcu9UhWmZ8yPYtXbUGqOAYwH87alWkpCic/edit#gid=0 ) > > > o Key Note & Introduction – Ted Dunning, Parth Chandra and Aman Sinha > > > o Community Contributions – Anil Kumar, John Omernik, Charles Givre and > > Ted Dunning > > > o Two tracks for technical design discussions – some topics have > initial > > thoughts for the topics and some will have open brainstorming discussions > > > o Once the discussions are concluded, we will have summaries presented > > and notes shared with the community > > > > > > • We will have a WebEx for the first two sessions. For the two tracks, > > we will either continue the WebEx or have Hangout links (will publish > them > > to the google sheet) > > > "JOIN WEBEX MEETING > > > https://mapr.webex.com/mapr/j.php?MTID=m9d39036e3953cce59ea81250c70c6 > c76 > > > Meeting number (access code): 806 111 950 > > > Meeting password: ApacheDrill" > > > > > > • For the attendees in person, we have made bookings for a dinner in > the > > evening - https://www.yelp.com/biz/chili-garden-restaurant-milpitas > > > > > > Looking forward to a fantastic day for the Apache Drill! community! > > > > > > Thanks, > > > Pritesh > > > > > > > > > > > > On 9/5/17, 10:47 PM, "Aman Sinha" <amansi...@apache.org> wrote: > > > > > > Here is the Eventbrite event for registration: > > > > > > https://www.eventbrite.com/e/drill-developer-day-sept-2017- > > registration-7478463285 > > > > > > Please register so we can plan for food and drinks appropriately. > > > > > > The link also contains a google doc link for the preliminary agenda > > and a > > > 'Topics' tab with volunteer sign-up column. Please add your name to > > the > > > area(s) of interest. > > > > > > Thanks and look forward to seeing you all ! > > > > > > -Aman > > > > > > On Wed, Aug 30, 2017 at 9:44 AM, Paul Rogers <prog...@mapr.com> > > wrote: > > > > > >> A partial list of Drill’s public APIs: > > >> > > >> IMHO, highest priority for Drill 2.0. > > >> > > >> > > >> * JDBC/ODBC drivers > > >> * Client (for JDBC/ODBC) + ODBC & JDBC > > >> * Client (for full Drill async, columnar) > > >> * Storage plugin > > >> * Format plugin > > >> * System/session options > > >> * Queueing (e.g. ZK-based queues) > > >> * Rest API > > >> * Resource Planning (e.g. max query memory per node) > > >> * Metadata access, storage (e.g. file system locations vs. a > > metastore) > > >> * Metadata files formats (Parquet, views, etc.) > > >> > > >> Lower priority for future releases: > > >> > > >> > > >> * Query Planning (e.g. Calcite rules) > > >> * Config options > > >> * SQL syntax, especially Drill extensions > > >> * UDF > > >> * Management (e.g. JMX, Rest API calls, etc.) > > >> * Drill File System (HDFS) > > >> * Web UI > > >> * Shell scripts > > >> > > >> There are certainly more. Please suggest those that are missing. I’ve > > >> taken a rough cut at which APIs need forward/backward compatibility > > first, > > >> in part based on those that are the “most public” and most likely to > > >> change. Others are important, but we can’t do them all at once. > > >> > > >> Thanks, > > >> > > >> - Paul > > >> > > >> On Aug 29, 2017, at 6:00 PM, Aman Sinha <amansi...@apache.org<mailto: > a > > >> mansi...@apache.org>> wrote: > > >> > > >> Hi Paul, > > >> certainly makes sense to have the API compatibility discussions during > > this > > >> hackathon. The 2.0 release may be a good checkpoint to introduce > > breaking > > >> changes necessitating changes to the ODBC/JDBC drivers and other > > external > > >> applications. As part of this exercise (not during the hackathon but > as > > a > > >> follow-up action), we also should clearly identify the "public" > > interfaces. > > >> > > >> > > >> I will add this to the agenda. > > >> > > >> thanks, > > >> -Aman > > >> > > >> On Tue, Aug 29, 2017 at 2:08 PM, Paul Rogers <prog...@mapr.com > <mailto: > > >> prog...@mapr.com>> wrote: > > >> > > >> Thanks Aman for organizing the Hackathon! > > >> > > >> The list included many good ideas for Drill 2.0. Some of those require > > >> changes to Drill’s “public” interfaces (file format, client protocol, > > SQL > > >> behavior, etc.) > > >> > > >> At present, Drill has no good mechanism to handle backward/forward > > >> compatibility at the API level. Protobuf versioning certainly helps, > but > > >> can’t completely solve semantic changes (where a field changes > meaning, > > or > > >> a non-Protobuf data chunk changes format.) As just one concrete > example, > > >> changing to Arrow will break pre-Arrow ODBC/JDBC drivers because class > > >> names and data formats will change. > > >> > > >> Perhaps we can prioritize, for the proposed 2.0 release, a one-time > set > > of > > >> breaking changes that introduce a versioning mechanism into our public > > >> APIs. Once these are in place, we can evolve the APIs in the future by > > >> following the newly-created versioning protocol. > > >> > > >> Without such a mechanism, we cannot support old & new clients in the > > same > > >> cluster. Nor can we support rolling upgrades. Of course, another > > solution > > >> is to get it right the second time, then freeze all APIs and agree to > > never > > >> again change them. Not sure we have sufficient access to a crystal > ball > > to > > >> predict everything we’d ever need in our APIs, however... > > >> > > >> Thanks, > > >> > > >> - Paul > > >> > > >> On Aug 24, 2017, at 8:39 AM, Aman Sinha <amansi...@apache.org<mailto: > a > > >> mansi...@apache.org>> wrote: > > >> > > >> Drill Developers, > > >> > > >> In order to kick-start the Drill 2.0 release discussions, I would > like > > >> to > > >> propose a Drill 2.0 (design) hackathon (a.k.a Drill Developer Day ™ J > > ). > > >> > > >> As I mentioned in the hangout on Tuesday, MapR has offered to host it > > on > > >> Sept 18th at their offices at 350 Holger Way, San Jose. Hope that > > works > > >> for most of you! > > >> > > >> The goal is to get the community together for a day-long technical > > >> discussion on key topics in preparation for a Drill 2.0 release as > well > > >> as > > >> potential improvements in upcoming 1.xx releases. Depending on the > > >> interest areas, we could form groups and have a volunteer lead each > > >> group. > > >> > > >> Based on prior discussions on the dev list, hangouts and existing > JIRAs, > > >> there is already a substantial set of topics and I have summarized a > few > > >> of > > >> them below. What other topics do folks want to talk about? Feel > free > > >> to > > >> respond to this thread and I will create a google doc to consolidate. > > >> Understandably, the list would be long but we will use the hackathon > to > > >> get > > >> a sense of a reasonable feature set for 1.xx and 2.0 releases. > > >> > > >> > > >> 1. Metadata management. > > >> > > >> 1a: Defining an abstraction layer for various types of metadata: > views, > > >> schema, statistics, security > > >> > > >> 1b: Underlying storage for metadata: what are the options and their > > >> trade-offs? > > >> > > >> - Hive metastore > > >> > > >> - Parquet metadata cache (parquet specific) > > >> > > >> - An embedded DBMS > > >> > > >> - A distributed key-value store > > >> > > >> - Others.. > > >> > > >> > > >> > > >> 2. Drill integration with Apache Arrow > > >> > > >> 2a: Evaluate the choices and tradeoffs > > >> > > >> > > >> > > >> 3. Resource management > > >> > > >> 3a: Memory limits per query > > >> > > >> 3b: Spilling > > >> > > >> 3c: Resource management with Drill on Yarn/Mesos/Kubernetes > > >> > > >> 3d: Local vs. global resource management > > >> > > >> 3e: Aligning with admission control/queueing > > >> > > >> > > >> > > >> 4. TPC-DS coverage and related planner/operator enhancements > > >> > > >> 4a: Additional set operations: INTERSECT, EXCEPT > > >> > > >> 4b: GROUPING SETS, ROLLUP, CUBE support > > >> > > >> 4c: Handling inequality joins and cartesian joins of non-scalar inputs > > >> (via Nested Loop Join) > > >> > > >> 4d: Remaining gaps in correlated subquery > > >> > > >> 4e: Statistics: Number of Distinct Values, Histograms > > >> > > >> > > >> > > >> 5. Schema handling > > >> > > >> 5a: Creation, management of schema > > >> > > >> 5b: Handling schema changes in certain common cases > > >> > > >> 5c: Schema-awareness > > >> > > >> 5d: Others TBD > > >> > > >> > > >> > > >> 6. Concurrency > > >> > > >> 6a: What are the bottlenecks to achieving higher concurrency > > >> > > >> 6b: Ideas to address these..e.g async execution ? > > >> > > >> > > >> > > >> 7. Storage plugins, REST APIs related enhancements > > >> > > >> <Topics TBD> > > >> > > >> > > >> > > >> 8. Performance improvements > > >> > > >> 8a: Filter pushdown > > >> > > >> 8b: Vectorized Parquet reader > > >> > > >> 8c: Code-gen improvements > > >> > > >> 8d: Others TBD > > >> > > >> > > >> > > >> > > > > > > > > > > >