Re: How to implement AbstractRecordWriter

2019-05-31 Thread Paul Rogers
Hi Nicolas, Yet another suggestion, FWIW. We already have tests for writing JSON and CSV. We also have tests for reading MapR DB. So, try making changes to those and seeing if you can get those to run. For example, create a test that reads a file in CSV, write it to JSON, read it as JSON and

Re: How to implement AbstractRecordWriter

2019-05-31 Thread Paul Rogers
Hi Nicolas, To address your last issue about the wide variety of ways we have to write tests... Yes, you are right that there is a wonderful variety of techniques that evolved over the life of the project. Unlike Spark, we do not enjoy an over-abundance of contributors, so we've pretty much

Re: How to implement AbstractRecordWriter

2019-05-31 Thread Nicolas A Perez
Is there a chance we can into a webex call at some point so someone can help me out with an initial test run? On Fri, May 31, 2019 at 19:38 Paul Rogers wrote: > Hi Nicolas, > > Regarding your point that plugins should be, well, plugins -- independent > of Drill code. Yes, that is true. But, no

Re: [DISCUSSION] DRILL-7097 Rename MapVector to StructVector

2019-05-31 Thread Paul Rogers
Ted, you found the simple, elegant solution, as usual! As it turns out, this project could be made far simpler if we just look at it from the right angle. Bear with me. Ted and Charles each have, on occasion, suggested the value of a correlated list, along with functions to zip/unzip values in

Re: [DISCUSSION] DRILL-7097 Rename MapVector to StructVector

2019-05-31 Thread Ted Dunning
Would it be possible to call the new structure a Dict (following Python's inspiration)? That would avoid the large disruption of renaming Map*. On Fri, May 31, 2019 at 10:10 AM Paul Rogers wrote: > Hi Igor, > > Thank you for finally addressing a long-running irritation: that the Drill > Map

Re: How to implement AbstractRecordWriter

2019-05-31 Thread Paul Rogers
Hi Nicolas, Regarding your point that plugins should be, well, plugins -- independent of Drill code. Yes, that is true. But, no one has invested the time to make it so. Doing so would require a clear, stable code API; an easy way to develop such code without the need for the "build jar, copy

Re: [DISCUSSION] DRILL-7097 Rename MapVector to StructVector

2019-05-31 Thread Paul Rogers
Hi Igor, Thank you for finally addressing a long-running irritation: that the Drill Map type is not a map, it is a tuple. Perhaps you can divide the discussion into three parts. 1. Renaming classes, enums and other items internal to the Drill source code. 2. Renaming classes that are part of

Re: How to implement AbstractRecordWriter

2019-05-31 Thread Nicolas A Perez
One of the issues I have is that I haven’t found a way to debug my tests from intelliJ. It continues to say that some constructs from other modules are missing. Also, I haven’t found *simple* examples of how to write *simple* tests. Every time i look at the existing code, the tests are done in a

Re: How to implement AbstractRecordWriter

2019-05-31 Thread Paul Rogers
Hi Nicolas, Charles outlined the choices quite well. Let's talk about your observation that you find it annoying to deal with the full Drill code. There may be some tricks here that can help you. As you know, I've been revising the text reader and the "EVF" (row set framework). Doing so

[GitHub] [drill] vvysotskyi commented on a change in pull request #1799: DRILL-7251: Read Hive array w/o nulls

2019-05-31 Thread GitBox
vvysotskyi commented on a change in pull request #1799: DRILL-7251: Read Hive array w/o nulls URL: https://github.com/apache/drill/pull/1799#discussion_r289394302 ## File path:

[GitHub] [drill] vvysotskyi commented on a change in pull request #1799: DRILL-7251: Read Hive array w/o nulls

2019-05-31 Thread GitBox
vvysotskyi commented on a change in pull request #1799: DRILL-7251: Read Hive array w/o nulls URL: https://github.com/apache/drill/pull/1799#discussion_r289409389 ## File path:

[GitHub] [drill] vvysotskyi commented on a change in pull request #1799: DRILL-7251: Read Hive array w/o nulls

2019-05-31 Thread GitBox
vvysotskyi commented on a change in pull request #1799: DRILL-7251: Read Hive array w/o nulls URL: https://github.com/apache/drill/pull/1799#discussion_r289435171 ## File path:

[GitHub] [drill] vvysotskyi commented on a change in pull request #1799: DRILL-7251: Read Hive array w/o nulls

2019-05-31 Thread GitBox
vvysotskyi commented on a change in pull request #1799: DRILL-7251: Read Hive array w/o nulls URL: https://github.com/apache/drill/pull/1799#discussion_r289420404 ## File path: contrib/storage-hive/core/src/main/java/org/apache/drill/exec/store/hive/readers/ReadersInitializer.java

[GitHub] [drill] vvysotskyi commented on a change in pull request #1799: DRILL-7251: Read Hive array w/o nulls

2019-05-31 Thread GitBox
vvysotskyi commented on a change in pull request #1799: DRILL-7251: Read Hive array w/o nulls URL: https://github.com/apache/drill/pull/1799#discussion_r289391195 ## File path:

[GitHub] [drill] vvysotskyi commented on a change in pull request #1799: DRILL-7251: Read Hive array w/o nulls

2019-05-31 Thread GitBox
vvysotskyi commented on a change in pull request #1799: DRILL-7251: Read Hive array w/o nulls URL: https://github.com/apache/drill/pull/1799#discussion_r289389523 ## File path:

[GitHub] [drill] vvysotskyi commented on a change in pull request #1799: DRILL-7251: Read Hive array w/o nulls

2019-05-31 Thread GitBox
vvysotskyi commented on a change in pull request #1799: DRILL-7251: Read Hive array w/o nulls URL: https://github.com/apache/drill/pull/1799#discussion_r289394136 ## File path:

[GitHub] [drill] vvysotskyi commented on a change in pull request #1799: DRILL-7251: Read Hive array w/o nulls

2019-05-31 Thread GitBox
vvysotskyi commented on a change in pull request #1799: DRILL-7251: Read Hive array w/o nulls URL: https://github.com/apache/drill/pull/1799#discussion_r289403764 ## File path:

[GitHub] [drill] vvysotskyi commented on a change in pull request #1799: DRILL-7251: Read Hive array w/o nulls

2019-05-31 Thread GitBox
vvysotskyi commented on a change in pull request #1799: DRILL-7251: Read Hive array w/o nulls URL: https://github.com/apache/drill/pull/1799#discussion_r289420813 ## File path: contrib/storage-hive/core/src/main/java/org/apache/drill/exec/store/hive/readers/ReadersInitializer.java

[GitHub] [drill] vvysotskyi commented on a change in pull request #1799: DRILL-7251: Read Hive array w/o nulls

2019-05-31 Thread GitBox
vvysotskyi commented on a change in pull request #1799: DRILL-7251: Read Hive array w/o nulls URL: https://github.com/apache/drill/pull/1799#discussion_r289393173 ## File path:

[GitHub] [drill] vvysotskyi commented on a change in pull request #1799: DRILL-7251: Read Hive array w/o nulls

2019-05-31 Thread GitBox
vvysotskyi commented on a change in pull request #1799: DRILL-7251: Read Hive array w/o nulls URL: https://github.com/apache/drill/pull/1799#discussion_r289393607 ## File path:

[GitHub] [drill] vvysotskyi commented on a change in pull request #1799: DRILL-7251: Read Hive array w/o nulls

2019-05-31 Thread GitBox
vvysotskyi commented on a change in pull request #1799: DRILL-7251: Read Hive array w/o nulls URL: https://github.com/apache/drill/pull/1799#discussion_r289390310 ## File path:

Re: How to implement AbstractRecordWriter

2019-05-31 Thread Nicolas A Perez
Let me see what I can do. Thanks. On Fri, May 31, 2019 at 2:17 PM Charles Givre wrote: > Hi Nicolas, > > You have two options: > 1. You can develop format plugins and UDFs in Drill by adding them to the > contrib/ folder and then test them with unit tests. Take a look at this PR > as an

Re: How to implement AbstractRecordWriter

2019-05-31 Thread Charles Givre
Hi Nicolas, You have two options: 1. You can develop format plugins and UDFs in Drill by adding them to the contrib/ folder and then test them with unit tests. Take a look at this PR as an example[1]. If you're intending to submit your work to Drill for inclusion, this would be my

Re: How to implement AbstractRecordWriter

2019-05-31 Thread Nicolas A Perez
Paul, Is it possible to develop my plugin outside of the drill code, let's say in my own repository and then package it and add it to the location where the plugins live? Does that work, too? I just find annoying to deal with the full drill code in order to develop a plugin. At the same time, I

[DISCUSSION] DRILL-7097 Rename MapVector to StructVector

2019-05-31 Thread Igor Guzenko
Hello Drillers, I'm working on the renaming of Map vector[1] and related stuff to make space for new canonical Map vector [2] [3]. I believe this renaming causes big impact on Drill and related client's code (ODBC/JDBC). So I'd like to be sure that this renaming is really necessary and everybody

[GitHub] [drill] ihuzenko opened a new pull request #1803: DRILL-7097: Rename MapVector to StructVector

2019-05-31 Thread GitBox
ihuzenko opened a new pull request #1803: DRILL-7097: Rename MapVector to StructVector URL: https://github.com/apache/drill/pull/1803 This is an automated message from the Apache Git Service. To respond to the message,