Re: [DISCUSS] Drill 1.19.0 release

[Laurent Goujon]

Hi,

I just sent the announcement email to the Apache mailing lists, and
this was the last item on the release document. I guess it means that
1.19 is officially released. Thank you everyone for your help!

If I missed anything, please let me know.

Cheers,

Laurent

On Fri, Jun 4, 2021 at 2:03 PM Laurent Goujon  wrote:
>
> So the vote for RC0 has been cancelled because of DRILL-7945. I will publish 
> a new RC with the following fixes:
> - https://issues.apache.org/jira/browse/DRILL-7937
> - https://issues.apache.org/jira/browse/DRILL-7940
> - https://issues.apache.org/jira/browse/DRILL-7945
> - https://issues.apache.org/jira/browse/DRILL-7946 (Security issue)
>
> There has been 2 issues open since RC0 that the requestor made "blocker"
> - https://issues.apache.org/jira/browse/DRILL-7948
> - https://issues.apache.org/jira/browse/DRILL-7949
>
> I do not plan to wait on a fix for those issues because they are not 
> regression compared to 1.18.0. One of the issue is actually something visible 
> only after DRILL-7937 was addressed (and this issue was not a regression 
> itself).
> Like stated in the RC0 email, I also do not intend to merge any of the other 
> junit/mockito/hamcrest version change: the junit change has actually new 
> comment, and all those patches do not have visible impact on the distribution 
> so I think they can also wait to be merged just after the release approval 
> (and I'll happy help get them merged).
>
> Laurent
>
> On Mon, May 31, 2021 at 9:17 PM Laurent Goujon  wrote:
>>
>> It looks like the parquet patch for DRILL-7934. I'll send an email to 
>> announce the start of the release process.
>>
>> On Sun, May 30, 2021 at 10:35 PM Laurent Goujon  wrote:
>>>
>>> Ok, let's do that!
>>>
>>> On Sat, May 29, 2021, 19:09 Charles Givre  wrote:
>>>>
>>>> Hi Laurent,
>>>> We got the CVEs sorted out, and at least one of the parquet bugs was 
>>>> sorted.  Maybe give it until Tuesday and if DRILL-7934 is merged, great, 
>>>> if not, we go without it and start preparing a release.
>>>> Does that work?
>>>> - C
>>>>
>>>> > On May 28, 2021, at 9:55 PM, luoc  wrote:
>>>> >
>>>> > Hi Laurent,
>>>> >  That's right. Thanks all for the contributions. As Charles said, We 
>>>> > plan to speed up the release frequency. I'm ready to post the [VOTE] 
>>>> > mail at the end of 1.19 release.
>>>> >
>>>> >> 在 2021年5月29日，01:55，Laurent Goujon  写道：
>>>> >>
>>>> >> Today's update: several changes related to the CVEs have been merged, 
>>>> >> along
>>>> >> with a bugfix for Parquet. Thanks to all of you who helped on those 
>>>> >> changes.
>>>> >> I believe there's only one Parquet change left for DRILL-7934:
>>>> >> <https://issues.apache.org/jira/browse/DRILL-7934> Charles, is this 
>>>> >> correct?
>>>> >>
>>>> >> Laurent
>>>> >>
>>>> >>> On Thu, May 27, 2021 at 10:48 AM Laurent Goujon  
>>>> >>> wrote:
>>>> >>>
>>>> >>> Some fixes/improvements were made to the codebase since the last 
>>>> >>> release,
>>>> >>> and sadly an official release is needed to pick up those changes. Ray 
>>>> >>> asked
>>>> >>> the community more than a month ago. More recently, other people have 
>>>> >>> been
>>>> >>> asking too on the user mailing list.
>>>> >>>
>>>> >>> Like I said, it might be okay to change the scope but what I'm asking 
>>>> >>> is a
>>>> >>> little help/transparency here because it looks like I'm chasing a 
>>>> >>> moving
>>>> >>> target. If we can clarify which new issues have to be part of the 
>>>> >>> release
>>>> >>> and why (depending on the severity), and how long we think it will 
>>>> >>> take,
>>>> >>> I'd hope we can have some constructive discussion.
>>>> >>>
>>>> >>> As for the dependencies change:
>>>> >>> - I actually wrote a pull request to address CVEs in both Hadoop and 
>>>> >>> Jetty
>>>> >>> - The Guava change will not address the most recent CVE. To a

[ANNOUNCE] Apache Drill 1.19.0 Released

[Laurent Goujon]

On behalf of the Apache Drill community, I am happy to announce the release
of Apache Drill 1.19.0.

Drill is an Apache open-source SQL query engine for Big Data exploration.
Drill is designed from the ground up to support high-performance analysis
on the semi-structured and rapidly evolving data coming from modern Big
Data applications, while still providing the familiarity and ecosystem of
ANSI SQL, the industry-standard query language. Drill provides
plug-and-play integration with existing Apache Hive and Apache HBase
deployments.

For information about Apache Drill, and to get involved, visit the project
website [1].

Total of 115 JIRA's are resolved in this release of Drill with following
new features and improvements [2]:

 - Cassandra Storage Plugin (DRILL-92)
 - Elasticsearch Storage Plugin (DRILL-3637)
 - XML Storage Plugin (DRILL-7823)
 - Splunk Storage Plugin (DRILL-7751)
 - Avro with schema registry support for Kafka (DRILL-5940)
 - Secure mechanism for specifying storage plugin credentials (DRILL-7855)
 - Linux ARM64 based system support (DRILL-7921)
 - Rowset based JSON reader (DRILL-6953)
 - Use streaming for REST JSON queries (DRILL-7733)
 - Several plugins have been converted to the Enhanced Vector Framework
(EVF)
   - Convert SequenceFiles to EVF (DRILL-7525)
   - Convert SysLog to EVF (DRILL-7532)
   - Convert Pcapng to EVF (DRILL-7533)
   - Convert HTTPD format plugin to EVF (DRILL-7534)
   - Convert Image Format to EVF (DRILL-7533)

For the full list please see release notes [3].

The binary and source artifacts are available here [4].

Thanks to everyone in the community who contributed to this release!

1. https://drill.apache.org/
2. https://drill.apache.org/blog/2021/06/10/drill-1.19-released/
3. https://drill.apache.org/docs/apache-drill-1-19-0-release-notes/
4. https://drill.apache.org/download/

Re: [Attn] Drill 1.19.0 release - master tree is frozen

[Laurent Goujon]

Master is now unfrozen. Thank you all for your understanding.

Laurent

On Tue, Jun 1, 2021 at 2:42 PM Laurent Goujon  wrote:

> Change has been merged and is part of the release candidate.
>
> On Tue, Jun 1, 2021 at 8:38 AM luoc  wrote:
>
>>
>> DRILL-7928, please merge it, thanks
>>
>> > 2021年6月1日 下午11:29，Laurent Goujon  写道：
>> >
>> > Technically, yes, it has started. Which PR? this wasn't send to the
>> mailing
>> > list
>> >
>> > On Mon, May 31, 2021 at 10:51 PM luoc  wrote:
>> >
>> >>
>> >> Has it started, Laurent? I want to merge the last PR now.
>> >>
>> >>> 2021年6月1日 下午12:25，Laurent Goujon  写道：
>> >>>
>> >>> Hi,
>> >>>
>> >>> In preparation for the 1.19.0 release, master tree is currently frozen
>> >>> until the release process is completed. For committers, until the
>> release
>> >>> is over and Drill version is changed to 1.20.0-SNAPSHOT, please do not
>> >> push
>> >>> any changes into Drill master.
>> >>>
>> >>> Cheers,
>> >>>
>> >>> Laurent
>> >>
>> >>
>>
>>

Re: [RESULT] [VOTE] Release Apache Drill 1.19.0 RC1

[Laurent Goujon]

Thanks Vova, I was able to update JIRA. Just waiting now on on the last
comments for the website update.

@Charles, yes, you can go ahead: master branch has been changed to
1.20.0-SNAPSHOT

On Thu, Jun 10, 2021 at 12:29 PM Charles Givre  wrote:

> Thank you all for all your work on what promises to be the best Drill
> release yet!
>
> @Laurent,
> Is the master branch unfrozen at this point?  Are we ok to merge the PRs
> that have been approved?
> Thanks!
>
> — C
>
>
>
> > On Jun 10, 2021, at 12:44 PM, Vova Vysotskyi  wrote:
> >
> > Hi Laurent,
> >
> > * I have provided you with administrator permissions in Apache Drill
> Jira, so you should be able to update the current release date and create a
> new release (and do some batch operations for tickets if required, like
> updating "Fix version", etc.).
> > * I have access to that Twitter account (but not only I), please send an
> announcement about the release in a private message to @ApacheDrill and
> I’ll post it.
> > * Done: https://reporter.apache.org/addrelease.html?drill
> >
> > Kind regards,
> > Volodymyr Vysotskyi
> >
> > On 2021/06/10 07:41:46, Laurent Goujon  wrote:
> >> Thanks Vova.>
> >>
> >> I checked that the docker image and the maven artifacts have been
> published.>
> >>
> >> I also created a pull request for updating the website:>
> >> https://github.com/apache/drill/pull/2257/commits>
> >> It contains the javadoc changes, but the commits are separated so it
> should>
> >> be easy for people to review the changes and the blog announcement.
> Please>
> >> let me know if I forgot an important feature.>
> >>
> >> Still looking at the release documentation (>
> >> https://github.com/apache/drill/blob/master/docs/dev/Release.md),
> there are>
> >> a couple of things it seems I won't be able to do but would expect a
> PMC to>
> >> do:>
> >> * Updating the current release date in JIRA and creating a new release>
> >> * Post the announcement on Twitter>
> >> * Post the release date on
> https://reporter.apache.org/addrelease.html?drill>
> >>
> >> As for the website update, can someone confirm that those instructions
> at>
> >>
> https://github.com/apache/drill/blob/gh-pages/README.md#uploading-to-the-apache-website-drill-committers-only>
>
> >> are still correct?>
> >>
> >> Laurent>
> >>
> >>
> >> On Wed, Jun 9, 2021 at 2:44 PM Vova Vysotskyi 
> wrote:>
> >>
> >>> Good news, that issue was resolved so I have published artifacts:>
> >>> https://dist.apache.org/repos/dist/release/drill/drill-1.19.0/.>
> >>>>
> >>> We will have to update the release instruction since now it would
> work>
> >>> only with moving artifacts from dist/dev to dist/release>
> >>>>
> >>> Kind regards,>
> >>> Volodymyr Vysotskyi>
> >>>>
> >>> On 2021/06/09 20:16:12, Ted Dunning  wrote:>
> >>>> Thanks.>>
> >>>>>
> >>>> I figured you would be ahead of me on this.>>
> >>>>>
> >>>> On Wed, Jun 9, 2021 at 12:27 PM  wrote:>>
> >>>>>
> >>>>> Hello Ted,>>
> >>>>>>>
> >>>>> Yes, initially I tried both options.>>
> >>>>> I have also left a comment on the ticket, hope it will be resolved>
> >>> soon.>>
> >>>>>>>
> >>>>> Kind regards,>>
> >>>>> Volodymyr Vysotskyi>>
> >>>>>>>
> >>>>> On 2021/06/09 19:04:02, Ted Dunning  wrote:>>
> >>>>>> Vova,>>>
> >>>>>>>>
> >>>>>> Gavin responded on INFRA-21981 to the effect that upload should go>
> >>> to>>
> >>>>> the>>>
> >>>>>> dev side and then svn mv should be used to move to the release>
> >>> side.>>>
> >>>>>>>>
> >>>>>> Is that what you tried to do?>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>> On Wed, Jun 9, 2021 at 10:25 AM  wrote:>>>
> >>>>>>>>
> >>>>>>> I have some issues, will deploy after>>>
> 

Re: [RESULT] [VOTE] Release Apache Drill 1.19.0 RC1

[Laurent Goujon]

Thanks Vova.

I checked that the docker image and the maven artifacts have been published.

I also created a pull request for updating the website:
https://github.com/apache/drill/pull/2257/commits
It contains the javadoc changes, but the commits are separated so it should
be easy for people to review the changes and the blog announcement. Please
let me know if I forgot an important feature.

Still looking at the release documentation (
https://github.com/apache/drill/blob/master/docs/dev/Release.md), there are
a couple of things it seems I won't be able to do but would expect a PMC to
do:
* Updating the current release date in JIRA and creating a new release
* Post the announcement on Twitter
* Post the release date on https://reporter.apache.org/addrelease.html?drill

As for the website update, can someone confirm that those instructions at
https://github.com/apache/drill/blob/gh-pages/README.md#uploading-to-the-apache-website-drill-committers-only
are still correct?

Laurent


On Wed, Jun 9, 2021 at 2:44 PM Vova Vysotskyi  wrote:

> Good news, that issue was resolved so I have published artifacts:
> https://dist.apache.org/repos/dist/release/drill/drill-1.19.0/.
>
> We will have to update the release instruction since now it would work
> only with moving artifacts from dist/dev to dist/release
>
> Kind regards,
> Volodymyr Vysotskyi
>
> On 2021/06/09 20:16:12, Ted Dunning  wrote:
> > Thanks.>
> >
> > I figured you would be ahead of me on this.>
> >
> > On Wed, Jun 9, 2021 at 12:27 PM  wrote:>
> >
> > > Hello Ted,>
> > >>
> > > Yes, initially I tried both options.>
> > > I have also left a comment on the ticket, hope it will be resolved
> soon.>
> > >>
> > > Kind regards,>
> > > Volodymyr Vysotskyi>
> > >>
> > > On 2021/06/09 19:04:02, Ted Dunning  wrote:>
> > > > Vova,>>
> > > >>
> > > > Gavin responded on INFRA-21981 to the effect that upload should go
> to>
> > > the>>
> > > > dev side and then svn mv should be used to move to the release
> side.>>
> > > >>
> > > > Is that what you tried to do?>>
> > > >>
> > > >>
> > > >>
> > > > On Wed, Jun 9, 2021 at 10:25 AM  wrote:>>
> > > >>
> > > > > I have some issues, will deploy after>>
> > > > > https://issues.apache.org/jira/browse/INFRA-21981 is fixed.>>
> > > > >>>
> > > > > On 2021/06/09 16:27:12, vo...@apache.org wrote:>>
> > > > > > Hello Laurent,>>>
> > > > > >>>
> > > > > > I’ll publish them later today.>>>
> > > > > >>>
> > > > > > Kind regards,>>>
> > > > > > Volodymyr Vysotskyi>>>
> > > > > >>>
> > > > > >>>
> > > > > > On 2021/06/09 04:39:50, Laurent Goujon 
> wrote: >>>
> > > > > > > Hi,> >>>
> > > > > > > >>>
> > > > > > > May I kindly ask for a PMC to push the RC1 artifacts to the
> dist>>
> > > > > repository> >>>
> > > > > > > per instructions at> >>>
> > > > > > >
> https://github.com/apache/drill/blob/master/docs/dev/Release.md?>>
> > > >>>
> > > > > > > >>>
> > > > > > > The artifacts are available at> >>>
> > > > > > > https://home.apache.org/~laurent/drill/releases/1.19.0/rc1/>
> >>>
> > > > > > > >>>
> > > > > > > Laurent> >>>
> > > > > > > >>>
> > > > > > > On Tue, Jun 8, 2021 at 9:36 PM Laurent Goujon <
> la...@dremio.com>>>
> > > > > wrote:> >>>
> > > > > > > >>>
> > > > > > > > Hi all,> >>>
> > > > > > > >> >>>
> > > > > > > > The vote passes. Thanks to everyone who has tested the
> release>>
> > > >>>
> > > > > > > >> >>>
> > > > > > > >>>
> > > > > > > >>>
> > > > > > > > candidate and given their comments and votes. Final tally:>
> >>>
> > > > > > > >> >>>
> > > > > > > > 3x +1 (binding): Laurent, Ted, Vova> >>>
> > > > > > > >> >>>
> > > > > > > > No 0s or -1s.> >>>
> > > > > > > >> >>>
> > > > > > > > I'll start the process for pushing the release artifacts
> and>
> > > send>>
> > > > > an> >>>
> > > > > > > > announcement once propagated.> >>>
> > > > > > > >> >>>
> > > > > > > > Kind regards,> >>>
> > > > > > > >> >>>
> > > > > > > > Laurent> >>>
> > > > > > > >> >>>
> > > > > > > >>>
> > > >>
> >

[jira] [Resolved] (DRILL-7854) When writing to S3 "WriteOperationHelper.newUploadPartRequest" throws "NoSuchMethodError"

[Laurent Goujon (Jira)]

 [ 
https://issues.apache.org/jira/browse/DRILL-7854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Laurent Goujon resolved DRILL-7854.
---
Resolution: Fixed

Should have been resolved when Guava version was updated to 30.0

> When writing to S3 "WriteOperationHelper.newUploadPartRequest" throws 
> "NoSuchMethodError"
> -
>
> Key: DRILL-7854
> URL: https://issues.apache.org/jira/browse/DRILL-7854
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Storage - Other
>Affects Versions: 1.18.0
> Environment: Standalone drill running in a docker environment on 
> x86_64 (centos7, alpine 3, debian buster)
>  
> Drill 1.18.0 was downloaded from an official mirror
>Reporter: Joshua Pedrick
>Priority: Major
> Fix For: 1.19.0
>
>
> When writing to S3(hosted on local Minio cluster) I am getting a 
> "NoSuchMethod" error. It appears to be related to the guava version included 
> with hadoop-3.2.1.
>  
> {code:java}
> 2021-02-01 21:39:17,753 [1fe78bd6-0525-196e-b3d2-25c294a604e2:frag:1:13] 
> ERROR o.a.d.e.p.i.p.ProjectRecordBatch - 
> ProjectRecordBatch[projector=Projector[vector2=null, 
> selectionVectorMode=NONE], hasRemainder=false, remainderIndex=0, 
> recordCount=0, 
> container=org.apache.drill.exec.record.VectorContainer@3efe0be4[recordCount = 
> 27884, schemaChanged = false, schema = BatchSchema [, ...]]2021-02-01 
> 21:39:17,754 [1fe78bd6-0525-196e-b3d2-25c294a604e2:frag:1:13] ERROR 
> o.a.d.e.physical.impl.BaseRootExec - Batch dump completed.2021-02-01 
> 21:39:17,754 [1fe78bd6-0525-196e-b3d2-25c294a604e2:frag:1:13] INFO  
> o.a.d.e.w.fragment.FragmentExecutor - 
> 1fe78bd6-0525-196e-b3d2-25c294a604e2:1:13: State change requested 
> CANCELLATION_REQUESTED --> FAILED2021-02-01 21:39:25,199 
> [1fe78bd6-0525-196e-b3d2-25c294a604e2:frag:1:13] ERROR 
> o.a.d.exec.server.BootStrapContext - 
> org.apache.drill.exec.work.WorkManager$WorkerBee$1.run() leaked an 
> exception.java.lang.NoSuchMethodError: 
> com/google/common/base/Preconditions.checkArgument(ZLjava/lang/String;Ljava/lang/Object;J)V
>  (loaded from  by sun.misc.Launcher$AppClassLoader@3156fd60) called 
> from class org.apache.hadoop.fs.s3a.WriteOperationHelper (loaded from 
> file:/opt/drill/jars/3rdparty/hadoop-aws-3.2.1.jar by 
> sun.misc.Launcher$AppClassLoader@3156fd60).at 
> org.apache.hadoop.fs.s3a.WriteOperationHelper.newUploadPartRequest(WriteOperationHelper.java:397)
> at 
> org.apache.hadoop.fs.s3a.S3ABlockOutputStream$MultiPartUpload.uploadBlockAsync(S3ABlockOutputStream.java:584)
> at 
> org.apache.hadoop.fs.s3a.S3ABlockOutputStream$MultiPartUpload.access$000(S3ABlockOutputStream.java:521)
>   at 
> org.apache.hadoop.fs.s3a.S3ABlockOutputStream.uploadCurrentBlock(S3ABlockOutputStream.java:314)
>   at 
> org.apache.hadoop.fs.s3a.S3ABlockOutputStream.write(S3ABlockOutputStream.java:292)
>at 
> org.apache.hadoop.fs.FSDataOutputStream$PositionCache.write(FSDataOutputStream.java:57)
>   at java.io.DataOutputStream.write(DataOutputStream.java:107)at 
> java.io.FilterOutputStream.write(FilterOutputStream.java:97) at 
> org.apache.parquet.hadoop.util.HadoopPositionOutputStream.write(HadoopPositionOutputStream.java:45)
>   at 
> org.apache.parquet.bytes.CapacityByteArrayOutputStream.writeToOutput(CapacityByteArrayOutputStream.java:234)
>  at 
> org.apache.parquet.bytes.CapacityByteArrayOutputStream.writeTo(CapacityByteArrayOutputStream.java:247)
>at 
> org.apache.parquet.bytes.BytesInput$CapacityBAOSBytesInput.writeAllTo(BytesInput.java:421)
>at 
> org.apache.parquet.hadoop.ParquetFileWriter.writeColumnChunk(ParquetFileWriter.java:620)
>  at 
> org.apache.parquet.hadoop.ParquetColumnChunkPageWriteStore$ColumnChunkPageWriter.writeToFileWriter(ParquetColumnChunkPageWriteStore.java:268)
> at 
> org.apache.parquet.hadoop.ParquetColumnChunkPageWriteStore.flushToFileWriter(ParquetColumnChunkPageWriteStore.java:89)
>at 
> org.apache.drill.exec.store.parquet.ParquetRecordWriter.flushParquetFileWriter(ParquetRecordWriter.java:737)
>  at 
> org.apache.drill.exec.store.parquet.ParquetRecordWriter.flush(ParquetRecordWriter.java:435)
>   at 
> org.apache.drill.exec.store.parquet.ParquetRecordWriter.cleanup(ParquetRecordWriter.java:703)
> at 
> org.apache.drill.exec.physical.impl.WriterRecordBatch.closeWriter(WriterRecordBatch.java:203)
> at 
> org.apache.drill.exec.physical.impl.WriterRecordBatch.close(WriterRecordBatch.java:221)
&

[jira] [Resolved] (DRILL-7914) Apache Drill 1.19.0 Release Activities

[Laurent Goujon (Jira)]

 [ 
https://issues.apache.org/jira/browse/DRILL-7914?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Laurent Goujon resolved DRILL-7914.
---
Resolution: Fixed

> Apache Drill 1.19.0 Release Activities
> --
>
> Key: DRILL-7914
> URL: https://issues.apache.org/jira/browse/DRILL-7914
> Project: Apache Drill
>  Issue Type: Task
>    Reporter: Laurent Goujon
>        Assignee: Laurent Goujon
>Priority: Major
> Fix For: 1.19.0
>
>
> Source - https://github.com/apache/drill/blob/master/docs/dev/Release.md
> 1.19.0 Dashboard: 
> https://issues.apache.org/jira/secure/Dashboard.jspa?selectPageId=12336127
> Kanban Board - 
> https://issues.apache.org/jira/secure/RapidBoard.jspa?rapidView=185



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Resolved] (DRILL-7921) Support the Linux ARM64 based system

[Laurent Goujon (Jira)]

 [ 
https://issues.apache.org/jira/browse/DRILL-7921?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Laurent Goujon resolved DRILL-7921.
---
Resolution: Fixed

> Support the Linux ARM64 based system
> 
>
> Key: DRILL-7921
> URL: https://issues.apache.org/jira/browse/DRILL-7921
> Project: Apache Drill
>  Issue Type: New Feature
>Reporter: Cong Luo
>Assignee: Martin Tzvetanov Grigorov
>Priority: Major
> Fix For: 1.19.0
>
>
> This is an umbrella Jira for support to running on Linux ARM64 based system.
>  # Build must be passed.
>  # All the JUnit test must be passed.
>  # Running stable on the Linux aarch64 machines.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [RESULT] [VOTE] Release Apache Drill 1.19.0 RC1

[Laurent Goujon]

Hi,

May I kindly ask for a PMC to push the RC1 artifacts to the dist repository
per instructions at
https://github.com/apache/drill/blob/master/docs/dev/Release.md?

The artifacts are available at
https://home.apache.org/~laurent/drill/releases/1.19.0/rc1/

Laurent

On Tue, Jun 8, 2021 at 9:36 PM Laurent Goujon  wrote:

> Hi all,
>
> The vote passes. Thanks to everyone who has tested the release
>


> candidate and given their comments and votes. Final tally:
>
> 3x +1 (binding): Laurent, Ted, Vova
>
> No 0s or -1s.
>
> I'll start the process for pushing the release artifacts and send an
> announcement once propagated.
>
> Kind regards,
>
> Laurent
>

[RESULT] [VOTE] Release Apache Drill 1.19.0 RC1

[Laurent Goujon]

Hi all,

The vote passes. Thanks to everyone who has tested the release candidate
and given their comments and votes. Final tally:

3x +1 (binding): Laurent, Ted, Vova

No 0s or -1s.

I'll start the process for pushing the release artifacts and send an
announcement once propagated.

Kind regards,

Laurent

[VOTE] Release Apache Drill 1.19.0 - RC1

[Laurent Goujon]

Hi all,

I'd like to propose the first release candidate (RC1) of Apache Drill,
version 1.19.0.
The release candidate covers a total of 109 resolved JIRAs [1]. Thanks
to everyone who contributed to this release.
The tarball artifacts are hosted at [2] and the maven artifacts are
hosted at [3].
This release candidate is based on commit
ad3f344ac21e0462aa82f51f648a21a0554cf368 located at [4].
Please download and try out the release.

The vote ends at 5 PM UTC (9 AM PDT, 7 PM EET, 10:30 PM IST), June 8, 2021.

[ ] +1
[ ] +0
[ ] -1
Here's my vote: +1
Laurent

[1] 
https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12313820&version=12348331
[2] https://home.apache.org/~laurent/drill/releases/1.19.0/rc1/

[3] https://repository.apache.org/content/repositories/orgapachedrill-1086

[4] https://github.com/laurentgo/drill/commits/drill-1.19.0

[jira] [Resolved] (DRILL-7946) Bump HttpClient from 4.5.12 to 4.5.13 for CVE-2020-13956

[Laurent Goujon (Jira)]

 [ 
https://issues.apache.org/jira/browse/DRILL-7946?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Laurent Goujon resolved DRILL-7946.
---
Resolution: Fixed

> Bump HttpClient from 4.5.12 to 4.5.13 for CVE-2020-13956
> 
>
> Key: DRILL-7946
> URL: https://issues.apache.org/jira/browse/DRILL-7946
> Project: Apache Drill
>  Issue Type: Improvement
>Reporter: Cong Luo
>Assignee: Cong Luo
>Priority: Major
> Fix For: 1.19.0
>
>
> Apache HttpClient versions prior to version 4.5.13 and 5.0.3 can misinterpret 
> malformed authority component in request URIs passed to the library as 
> java.net.URI object and pick the wrong target host for request execution.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [DISCUSS] Drill 1.19.0 release

[Laurent Goujon]

So the vote for RC0 has been cancelled because of DRILL-7945
<https://issues.apache.org/jira/browse/DRILL-7945>. I will publish a new RC
with the following fixes:
- https://issues.apache.org/jira/browse/DRILL-7937
- https://issues.apache.org/jira/browse/DRILL-7940
- https://issues.apache.org/jira/browse/DRILL-7945
- https://issues.apache.org/jira/browse/DRILL-7946 (Security issue)

There has been 2 issues open since RC0 that the requestor made "blocker"
- https://issues.apache.org/jira/browse/DRILL-7948
- https://issues.apache.org/jira/browse/DRILL-7949

I do not plan to wait on a fix for those issues because they are not
regression compared to 1.18.0. One of the issue is actually something
visible only after DRILL-7937 was addressed (and this issue was not a
regression itself).
Like stated in the RC0 email, I also do not intend to merge any of the
other junit/mockito/hamcrest version change: the junit change has actually
new comment, and all those patches do not have visible impact on the
distribution so I think they can also wait to be merged just after the
release approval (and I'll happy help get them merged).

Laurent

On Mon, May 31, 2021 at 9:17 PM Laurent Goujon  wrote:

> It looks like the parquet patch for DRILL-7934. I'll send an email to
> announce the start of the release process.
>
> On Sun, May 30, 2021 at 10:35 PM Laurent Goujon 
> wrote:
>
>> Ok, let's do that!
>>
>> On Sat, May 29, 2021, 19:09 Charles Givre  wrote:
>>
>>> Hi Laurent,
>>> We got the CVEs sorted out, and at least one of the parquet bugs was
>>> sorted.  Maybe give it until Tuesday and if DRILL-7934 is merged, great, if
>>> not, we go without it and start preparing a release.
>>> Does that work?
>>> - C
>>>
>>> > On May 28, 2021, at 9:55 PM, luoc  wrote:
>>> >
>>> > Hi Laurent,
>>> >  That's right. Thanks all for the contributions. As Charles said, We
>>> plan to speed up the release frequency. I'm ready to post the [VOTE] mail
>>> at the end of 1.19 release.
>>> >
>>> >> 在 2021年5月29日，01:55，Laurent Goujon  写道：
>>> >>
>>> >> Today's update: several changes related to the CVEs have been
>>> merged, along
>>> >> with a bugfix for Parquet. Thanks to all of you who helped on those
>>> changes.
>>> >> I believe there's only one Parquet change left for DRILL-7934:
>>> >> <https://issues.apache.org/jira/browse/DRILL-7934> Charles, is this
>>> correct?
>>> >>
>>> >> Laurent
>>> >>
>>> >>> On Thu, May 27, 2021 at 10:48 AM Laurent Goujon 
>>> wrote:
>>> >>>
>>> >>> Some fixes/improvements were made to the codebase since the last
>>> release,
>>> >>> and sadly an official release is needed to pick up those changes.
>>> Ray asked
>>> >>> the community more than a month ago. More recently, other people
>>> have been
>>> >>> asking too on the user mailing list.
>>> >>>
>>> >>> Like I said, it might be okay to change the scope but what I'm
>>> asking is a
>>> >>> little help/transparency here because it looks like I'm chasing a
>>> moving
>>> >>> target. If we can clarify which new issues have to be part of the
>>> release
>>> >>> and why (depending on the severity), and how long we think it will
>>> take,
>>> >>> I'd hope we can have some constructive discussion.
>>> >>>
>>> >>> As for the dependencies change:
>>> >>> - I actually wrote a pull request to address CVEs in both Hadoop and
>>> Jetty
>>> >>> - The Guava change will not address the most recent CVE. To address
>>> the
>>> >>> CVE, code must be changed, and it doesn't require a Guava update. The
>>> >>> change made to the Guava library was to deprecate the unsecure
>>> method... So
>>> >>> imho updating dependencies to address CVE without looking at the CVE
>>> itself
>>> >>> does not make things safer. So to address specifically the CVE, I
>>> opened a
>>> >>> new ticket (DRILL-7936 <
>>> https://issues.apache.org/jira/browse/DRILL-7936>)
>>> >>> and a pull request (https://github.com/apache/drill/pull/2240)
>>> >>>
>>> >>>
>>> >>>> On Thu, May 27, 2021 at 9:30 AM Charles Givre 
>>&g

[jira] [Resolved] (DRILL-7945) Unable to patch Guava classes

[Laurent Goujon (Jira)]

 [ 
https://issues.apache.org/jira/browse/DRILL-7945?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Laurent Goujon resolved DRILL-7945.
---
  Assignee: Laurent Goujon  (was: Vitalii Diravka)
Resolution: Fixed

> Unable to patch Guava classes
> -
>
> Key: DRILL-7945
> URL: https://issues.apache.org/jira/browse/DRILL-7945
> Project: Apache Drill
>  Issue Type: Bug
>Affects Versions: 1.19.0
>Reporter: Vova Vysotskyi
>    Assignee: Laurent Goujon
>Priority: Blocker
> Fix For: 1.19.0
>
>
> When starting Drill in embedded mode, logs have the following line:
> {noformat}
> 2021-06-03 21:59:07,711 [main] WARN  o.a.drill.common.util.GuavaPatcher - 
> Unable to patch Guava classes: [source error] 
> format(java.lang.String,java.lang.Object[]) not found in 
> com.google.common.base.Preconditions
> {noformat}
> Most likely it is a regression after DRILL-7904.
> The importance of this issue is that all 3rd party libraries that initially 
> relied on patched methods from {{Preconditions}} will stop working because of 
> method not found errors during the runtime.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [VOTE] Release Apache Drill 1.19.0 - RC0

[Laurent Goujon]

I think you missed my point:

- Not all bugs are equal and not all of them may cause to discard the
current evaluation of a release candidate. There are currently 1631 open
bugs for the project according to the Apache Drill JIRA: does it mean we
should wait for all those bugs to be fixed before we can release? What
makes this bug special compared to the others?
- Assuming the previous point is cleared and that the bug should indeed be
part of the release, it would be good to let the release manager handle it
or at least coordinate with them instead of doing it on your own. That's
usually what I've seen done in other projects and it seems to be a
reasonable thing to do as the release manager might be already deep in
patches merge and evaluation, and unexpected changes to the tree might
cause extra work which could have been avoided. Or just because that's the
respectful/polite thing to do...

On Thu, Jun 3, 2021 at 11:11 PM luoc  wrote:

> Laurent,
>   Thanks for doing this. RC0 is no longer eligible for the next step
> operation. It is a consensus that we cannot release a version with known
> issues (the pull request mark as `bug`). In fact, Drill's release process
> is not friendly, and we will put these discussion after the release. Now
> our focus is on preparing for RC1. BTW, You're doing great.
>
> > 2021年6月4日 下午1:20，Laurent Goujon  写道：
> >
> > You actually went ahead and merged those patches without waiting while I
> > was hoping we could get some consensus first :(
> >
> > Can I just ask you to please respect the effort I'm putting in following
> > what I think is the release process? If people think I'm not following
> the
> > proper steps or that I'm not doing a good job at doing it, I'll gladly
> > accept feedback and will do my best to address it, but going over me
> isn't
> > helping me or the future volunteers for the next releases which might be
> > also wondering what's the release process should be.
> > Meanwhile I'll wait to get a review for the DRILL-7945 patch fixing the
> > Guava regression, and hopefully I should be able to do another release
> > candidate tomorrow.
> >
> > Laurent
> >
> > On Thu, Jun 3, 2021 at 5:46 PM luoc  wrote:
> >
> >>
> >> The DRILL-7945 blocked the release. So, I'm ready to merge the
> DRILL-7937
> >> and DRILL-7940 for bugfix.
> >>
> >>> 在 2021年6月4日，01:15，Laurent Goujon  写道：
> >>>
> >>> Hey guys,
> >>>
> >>> Can we please stop changing the goal post again and again? The fact
> that
> >>> some of those pull requests are ready to merge should not be the sole
> >>> consideration when to do a next release candidate.
> >>>
> >>> I've been asking several times on this mailing list about what we want
> to
> >>> include or not, and we got an agreement several times about it, and
> >> several
> >>> times we are now having this conversation.
> >>> IMHO, I would not include DRILL-7941, DRILL-7942 and DRILL-7943: those
> >> are
> >>> new enhancements impacting Drill tests (not even the main product) and
> I
> >> do
> >>> not understand the rush in making them part of the release.
> Specifically
> >>> for the JUnit 5 update, I think the change is misleading because it
> looks
> >>> like it's only the introduction of JUnit5 in one test class and
> >> everything
> >>> else still uses JUnit 4, so I would hardly call it an upgrade...
> >>>
> >>> As for DRILL-7937 and DRILL-7940, the issues were open in the last 3
> days
> >>> ago, but they do not seem to be regressions since 1.18.0, just gaps in
> >> what
> >>> Drill provides. Personally since we are this deep in the release, I
> would
> >>> also skip these one too. But if people have more contexts on those,
> maybe
> >>> we can agree they should be merged?
> >>>
> >>> Laurent
> >>>
> >>>
> >>>> On Thu, Jun 3, 2021 at 6:10 AM Charles Givre 
> wrote:
> >>>>
> >>>> There are like 5 minor PRs that are approved and awaiting merge.  I'd
> >> vote
> >>>> that we include them.  Specifically:
> >>>>
> >>>> DRILL-7943: Update Hamcrest
> >>>> DRILL-7942: Update Mockito
> >>>> DRILL-7941: Update junit to 5.7.2
> >>>> DRILL-7937:  Parquet decimal error
> >>>> DRILL-7940: Fix Kafka Key
> >>>>
> >>>> These are all approved and can be merged.
> >>>>
> >>>> -- C
> >>>>
> >>>>>> On Jun 3, 2021, at 9:01 AM, luoc  wrote:
> >>>>>
> >>>>>
> >>>>> DRILL-7940, too
> >>>>>
> >>>>>> 在 2021年6月3日，19:57，Charles Givre  写道：
> >>>>>>
> >>>>>> -1 (Binding)
> >>>>>>
> >>>>>> I'd agree with Nick.  Drill-7937 should be included in this release.
> >>>>>> -- C
> >>>>>>
> >>>>>>> On Jun 2, 2021, at 9:25 AM, Nick Stenroos-Dam 
> >> wrote:
> >>>>>>>
> >>>>>>> Vote -1
> >>>>>>>
> >>>>>>> Can we please include  DRILL-7937
> >>>>>
> >>>>
> >>>>
> >>
> >>
>
>

Re: [VOTE] Release Apache Drill 1.19.0 - RC0

[Laurent Goujon]

Because of the regression introduced in 1.19.0 (DRILL-7945
<https://issues.apache.org/jira/browse/DRILL-7945>), the current vote is
being cancelled and a new release candidate will be produced. Meanwhile,
the master branch is still being frozen.

Laurent

On Tue, Jun 1, 2021 at 8:44 PM Prabhakar Bhosaale 
wrote:

> Vote +1
>
> On Wed, Jun 2, 2021 at 3:12 AM Laurent Goujon  wrote:
>
> > Hi all,
> >
> > I'd like to propose the first release candidate (RC0) of Apache Drill,
> > version 1.19.0.
> > The release candidate covers a total of 105 resolved JIRAs [1]. Thanks
> > to everyone who contributed to this release.
> > The tarball artifacts are hosted at [2] and the maven artifacts are
> > hosted at [3].
> > This release candidate is based on commit
> > ad3f344ac21e0462aa82f51f648a21a0554cf368 located at [4].
> > Please download and try out the release.
> >
> > The vote ends at 5 PM UTC (9 AM PDT, 7 PM EET, 10:30 PM IST), June 4,
> 2021.
> >
> > [ ] +1
> > [ ] +0
> > [ ] -1
> > Here's my vote: +1
> > Laurent
> >
> > [1]
> >
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12313820&version=12348331
> > [2] https://home.apache.org/~laurent/drill/releases/1.19.0/rc0/
> > [3]
> > https://repository.apache.org/content/repositories/orgapachedrill-1083/
> > [4] https://github.com/laurentgo/drill/commits/drill-1.19.0
> >
>

Re: [VOTE] Release Apache Drill 1.19.0 - RC0

[Laurent Goujon]

You actually went ahead and merged those patches without waiting while I
was hoping we could get some consensus first :(

Can I just ask you to please respect the effort I'm putting in following
what I think is the release process? If people think I'm not following the
proper steps or that I'm not doing a good job at doing it, I'll gladly
accept feedback and will do my best to address it, but going over me isn't
helping me or the future volunteers for the next releases which might be
also wondering what's the release process should be.
Meanwhile I'll wait to get a review for the DRILL-7945 patch fixing the
Guava regression, and hopefully I should be able to do another release
candidate tomorrow.

Laurent

On Thu, Jun 3, 2021 at 5:46 PM luoc  wrote:

>
> The DRILL-7945 blocked the release. So, I'm ready to merge the DRILL-7937
> and DRILL-7940 for bugfix.
>
> > 在 2021年6月4日，01:15，Laurent Goujon  写道：
> >
> > Hey guys,
> >
> > Can we please stop changing the goal post again and again? The fact that
> > some of those pull requests are ready to merge should not be the sole
> > consideration when to do a next release candidate.
> >
> > I've been asking several times on this mailing list about what we want to
> > include or not, and we got an agreement several times about it, and
> several
> > times we are now having this conversation.
> > IMHO, I would not include DRILL-7941, DRILL-7942 and DRILL-7943: those
> are
> > new enhancements impacting Drill tests (not even the main product) and I
> do
> > not understand the rush in making them part of the release. Specifically
> > for the JUnit 5 update, I think the change is misleading because it looks
> > like it's only the introduction of JUnit5 in one test class and
> everything
> > else still uses JUnit 4, so I would hardly call it an upgrade...
> >
> > As for DRILL-7937 and DRILL-7940, the issues were open in the last 3 days
> > ago, but they do not seem to be regressions since 1.18.0, just gaps in
> what
> > Drill provides. Personally since we are this deep in the release, I would
> > also skip these one too. But if people have more contexts on those, maybe
> > we can agree they should be merged?
> >
> > Laurent
> >
> >
> >> On Thu, Jun 3, 2021 at 6:10 AM Charles Givre  wrote:
> >>
> >> There are like 5 minor PRs that are approved and awaiting merge.  I'd
> vote
> >> that we include them.  Specifically:
> >>
> >> DRILL-7943: Update Hamcrest
> >> DRILL-7942: Update Mockito
> >> DRILL-7941: Update junit to 5.7.2
> >> DRILL-7937:  Parquet decimal error
> >> DRILL-7940: Fix Kafka Key
> >>
> >> These are all approved and can be merged.
> >>
> >> -- C
> >>
> >>>> On Jun 3, 2021, at 9:01 AM, luoc  wrote:
> >>>
> >>>
> >>> DRILL-7940, too
> >>>
> >>>> 在 2021年6月3日，19:57，Charles Givre  写道：
> >>>>
> >>>> -1 (Binding)
> >>>>
> >>>> I'd agree with Nick.  Drill-7937 should be included in this release.
> >>>> -- C
> >>>>
> >>>>> On Jun 2, 2021, at 9:25 AM, Nick Stenroos-Dam 
> wrote:
> >>>>>
> >>>>> Vote -1
> >>>>>
> >>>>> Can we please include  DRILL-7937
> >>>
> >>
> >>
>
>

Re: [VOTE] Release Apache Drill 1.19.0 - RC0

[Laurent Goujon]

Yes, this looks like a regression. I can try and help fix it.

On Thu, Jun 3, 2021 at 12:15 PM  wrote:

> Hi all,
>
> Thanks, Laurent for creating this release candidate!
>
> I have started verifying the release, but for now my vote is -1 (binding)
> because of the following issue: DRILL-7945 <
> https://issues.apache.org/jira/browse/DRILL-7945>
>
> It is definitely a regression and it might break some important Drill
> functionality so I'm considering it as a blocker for the release.
>
> Kind regards,
> Volodymyr Vysotskyi
>
>
> On 2021/06/03 17:14:37, Laurent Goujon  wrote:
> > Hey guys,>
> >
> > Can we please stop changing the goal post again and again? The fact
> that>
> > some of those pull requests are ready to merge should not be the sole>
> > consideration when to do a next release candidate.>
> >
> > I've been asking several times on this mailing list about what we want
> to>
> > include or not, and we got an agreement several times about it, and
> several>
> > times we are now having this conversation.>
> > IMHO, I would not include DRILL-7941, DRILL-7942 and DRILL-7943: those
> are>
> > new enhancements impacting Drill tests (not even the main product) and I
> do>
> > not understand the rush in making them part of the release.
> Specifically>
> > for the JUnit 5 update, I think the change is misleading because it
> looks>
> > like it's only the introduction of JUnit5 in one test class and
> everything>
> > else still uses JUnit 4, so I would hardly call it an upgrade...>
> >
> > As for DRILL-7937 and DRILL-7940, the issues were open in the last 3
> days>
> > ago, but they do not seem to be regressions since 1.18.0, just gaps in
> what>
> > Drill provides. Personally since we are this deep in the release, I
> would>
> > also skip these one too. But if people have more contexts on those,
> maybe>
> > we can agree they should be merged?>
> >
> > Laurent>
> >
> >
> > On Thu, Jun 3, 2021 at 6:10 AM Charles Givre  wrote:>
> >
> > > There are like 5 minor PRs that are approved and awaiting merge.  I'd
> vote>
> > > that we include them.  Specifically:>
> > >>
> > > DRILL-7943: Update Hamcrest>
> > > DRILL-7942: Update Mockito>
> > > DRILL-7941: Update junit to 5.7.2>
> > > DRILL-7937:  Parquet decimal error>
> > > DRILL-7940: Fix Kafka Key>
> > >>
> > > These are all approved and can be merged.>
> > >>
> > > -- C>
> > >>
> > > > On Jun 3, 2021, at 9:01 AM, luoc  wrote:>
> > > >>
> > > >>
> > > > DRILL-7940, too>
> > > >>
> > > >> 在 2021年6月3日，19:57，Charles Givre  写道：>
> > > >>>
> > > >> -1 (Binding)>
> > > >>>
> > > >> I'd agree with Nick.  Drill-7937 should be included in this
> release.>
> > > >> -- C>
> > > >>>
> > > >>> On Jun 2, 2021, at 9:25 AM, Nick Stenroos-Dam 
> wrote:>
> > > >>>>
> > > >>> Vote -1>
> > > >>>>
> > > >>> Can we please include  DRILL-7937>
> > > >>
> > >>
> > >>
> >

Re: [VOTE] Release Apache Drill 1.19.0 - RC0

[Laurent Goujon]

Hey guys,

Can we please stop changing the goal post again and again? The fact that
some of those pull requests are ready to merge should not be the sole
consideration when to do a next release candidate.

I've been asking several times on this mailing list about what we want to
include or not, and we got an agreement several times about it, and several
times we are now having this conversation.
IMHO, I would not include DRILL-7941, DRILL-7942 and DRILL-7943: those are
new enhancements impacting Drill tests (not even the main product) and I do
not understand the rush in making them part of the release. Specifically
for the JUnit 5 update, I think the change is misleading because it looks
like it's only the introduction of JUnit5 in one test class and everything
else still uses JUnit 4, so I would hardly call it an upgrade...

As for DRILL-7937 and DRILL-7940, the issues were open in the last 3 days
ago, but they do not seem to be regressions since 1.18.0, just gaps in what
Drill provides. Personally since we are this deep in the release, I would
also skip these one too. But if people have more contexts on those, maybe
we can agree they should be merged?

Laurent


On Thu, Jun 3, 2021 at 6:10 AM Charles Givre  wrote:

> There are like 5 minor PRs that are approved and awaiting merge.  I'd vote
> that we include them.  Specifically:
>
> DRILL-7943: Update Hamcrest
> DRILL-7942: Update Mockito
> DRILL-7941: Update junit to 5.7.2
> DRILL-7937:  Parquet decimal error
> DRILL-7940: Fix Kafka Key
>
> These are all approved and can be merged.
>
> -- C
>
> > On Jun 3, 2021, at 9:01 AM, luoc  wrote:
> >
> >
> > DRILL-7940, too
> >
> >> 在 2021年6月3日，19:57，Charles Givre  写道：
> >>
> >> -1 (Binding)
> >>
> >> I'd agree with Nick.  Drill-7937 should be included in this release.
> >> -- C
> >>
> >>> On Jun 2, 2021, at 9:25 AM, Nick Stenroos-Dam  wrote:
> >>>
> >>> Vote -1
> >>>
> >>> Can we please include  DRILL-7937
> >
>
>

Re: [Attn] Drill 1.19.0 release - master tree is frozen

[Laurent Goujon]

Change has been merged and is part of the release candidate.

On Tue, Jun 1, 2021 at 8:38 AM luoc  wrote:

>
> DRILL-7928, please merge it, thanks
>
> > 2021年6月1日 下午11:29，Laurent Goujon  写道：
> >
> > Technically, yes, it has started. Which PR? this wasn't send to the
> mailing
> > list
> >
> > On Mon, May 31, 2021 at 10:51 PM luoc  wrote:
> >
> >>
> >> Has it started, Laurent? I want to merge the last PR now.
> >>
> >>> 2021年6月1日 下午12:25，Laurent Goujon  写道：
> >>>
> >>> Hi,
> >>>
> >>> In preparation for the 1.19.0 release, master tree is currently frozen
> >>> until the release process is completed. For committers, until the
> release
> >>> is over and Drill version is changed to 1.20.0-SNAPSHOT, please do not
> >> push
> >>> any changes into Drill master.
> >>>
> >>> Cheers,
> >>>
> >>> Laurent
> >>
> >>
>
>

[VOTE] Release Apache Drill 1.19.0 - RC0

[Laurent Goujon]

Hi all,

I'd like to propose the first release candidate (RC0) of Apache Drill,
version 1.19.0.
The release candidate covers a total of 105 resolved JIRAs [1]. Thanks
to everyone who contributed to this release.
The tarball artifacts are hosted at [2] and the maven artifacts are
hosted at [3].
This release candidate is based on commit
ad3f344ac21e0462aa82f51f648a21a0554cf368 located at [4].
Please download and try out the release.

The vote ends at 5 PM UTC (9 AM PDT, 7 PM EET, 10:30 PM IST), June 4, 2021.

[ ] +1
[ ] +0
[ ] -1
Here's my vote: +1
Laurent

[1] 
https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12313820&version=12348331
[2] https://home.apache.org/~laurent/drill/releases/1.19.0/rc0/
[3] https://repository.apache.org/content/repositories/orgapachedrill-1083/
[4] https://github.com/laurentgo/drill/commits/drill-1.19.0

[jira] [Resolved] (DRILL-7795) Add option to native Drill Client to set TLS SNI property

[Laurent Goujon (Jira)]

 [ 
https://issues.apache.org/jira/browse/DRILL-7795?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Laurent Goujon resolved DRILL-7795.
---
Fix Version/s: 1.19.0
   Resolution: Fixed

> Add option to native Drill Client to set TLS SNI property
> -
>
> Key: DRILL-7795
> URL: https://issues.apache.org/jira/browse/DRILL-7795
> Project: Apache Drill
>  Issue Type: Improvement
>  Components: Client - C++
>Reporter: James Duong
>Priority: Major
> Fix For: 1.19.0
>
>
> Add an option to the C++ DrillClient library that sets the TLS SNI extension 
> to let you set the Server Name Indication TLS property. If no option for this 
> value is specified, choose a default based on the target host.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Resolved] (DRILL-6547) IllegalStateException: Tried to remove unmanaged buffer.

[Laurent Goujon (Jira)]

 [ 
https://issues.apache.org/jira/browse/DRILL-6547?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Laurent Goujon resolved DRILL-6547.
---
Fix Version/s: 1.19.0
   Resolution: Fixed

> IllegalStateException: Tried to remove unmanaged buffer.
> 
>
> Key: DRILL-6547
> URL: https://issues.apache.org/jira/browse/DRILL-6547
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Execution - Relational Operators
>Affects Versions: 1.14.0
>Reporter: Robert Hou
>Assignee: Pritesh Maker
>Priority: Major
> Fix For: 1.19.0
>
>
> This is the query:
> select * from (
> select Index, concat(BinaryValue, 'aaa') NewVarcharValue from (select * from 
> dfs.`/drill/testdata/batch_memory/alltypes_large_1MB.parquet`)) d where 
> d.Index = 1;
> This is the plan:
> {noformat}
> | 00-00Screen
> 00-01  Project(Index=[$0], NewVarcharValue=[$1])
> 00-02SelectionVectorRemover
> 00-03  Filter(condition=[=($0, 1)])
> 00-04Project(Index=[$0], NewVarcharValue=[CONCAT($1, 'aaa')])
> 00-05  Scan(groupscan=[ParquetGroupScan 
> [entries=[ReadEntryWithPath 
> [path=maprfs:///drill/testdata/batch_memory/alltypes_large_1MB.parquet]], 
> selectionRoot=maprfs:/drill/testdata/batch_memory/alltypes_large_1MB.parquet, 
> numFiles=1, numRowGroups=1, usedMetadataFile=false, columns=[`Index`, 
> `BinaryValue`]]])
> {noformat}
> Here is the stack trace from drillbit.log:
> {noformat}
> 2018-06-27 13:55:03,291 [24cc0659-30b7-b290-7fae-ecb1c1f15c05:frag:0:0] ERROR 
> o.a.d.e.w.fragment.FragmentExecutor - SYSTEM ERROR: IllegalStateException: 
> Tried to remove unmanaged buffer.
> Fragment 0:0
> [Error Id: bc1f2f72-c31b-4b9a-964f-96dec9e0f388 on qa-node186.qa.lab:31010]
> org.apache.drill.common.exceptions.UserException: SYSTEM ERROR: 
> IllegalStateException: Tried to remove unmanaged buffer.
> Fragment 0:0
> [Error Id: bc1f2f72-c31b-4b9a-964f-96dec9e0f388 on qa-node186.qa.lab:31010]
>   at 
> org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:633)
>  ~[drill-common-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
>   at 
> org.apache.drill.exec.work.fragment.FragmentExecutor.sendFinalState(FragmentExecutor.java:361)
>  [drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
>   at 
> org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:216)
>  [drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
>   at 
> org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:327)
>  [drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
>   at 
> org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38)
>  [drill-common-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>  [na:1.8.0_161]
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>  [na:1.8.0_161]
>   at java.lang.Thread.run(Thread.java:748) [na:1.8.0_161]
> Caused by: java.lang.IllegalStateException: Tried to remove unmanaged buffer.
>   at 
> org.apache.drill.exec.ops.BufferManagerImpl.replace(BufferManagerImpl.java:50)
>  ~[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
>   at io.netty.buffer.DrillBuf.reallocIfNeeded(DrillBuf.java:97) 
> ~[drill-memory-base-1.14.0-SNAPSHOT.jar:4.0.48.Final]
>   at 
> org.apache.drill.exec.test.generated.ProjectorGen4046.doEval(ProjectorTemplate.java:77)
>  ~[na:na]
>   at 
> org.apache.drill.exec.test.generated.ProjectorGen4046.projectRecords(ProjectorTemplate.java:67)
>  ~[na:na]
>   at 
> org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.doWork(ProjectRecordBatch.java:236)
>  ~[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
>   at 
> org.apache.drill.exec.record.AbstractUnaryRecordBatch.innerNext(AbstractUnaryRecordBatch.java:117)
>  ~[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
>   at 
> org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext(ProjectRecordBatch.java:147)
>  ~[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
>   at 
> org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:172)
>  ~[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
>   at 
> org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:119)
>  ~[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
>   at 
> org.apache.dri

Re: [Attn] Drill 1.19.0 release - master tree is frozen

[Laurent Goujon]

Technically, yes, it has started. Which PR? this wasn't send to the mailing
list

On Mon, May 31, 2021 at 10:51 PM luoc  wrote:

>
> Has it started, Laurent? I want to merge the last PR now.
>
> > 2021年6月1日 下午12:25，Laurent Goujon  写道：
> >
> > Hi,
> >
> > In preparation for the 1.19.0 release, master tree is currently frozen
> > until the release process is completed. For committers, until the release
> > is over and Drill version is changed to 1.20.0-SNAPSHOT, please do not
> push
> > any changes into Drill master.
> >
> > Cheers,
> >
> > Laurent
>
>

[Attn] Drill 1.19.0 release - master tree is frozen

[Laurent Goujon]

Hi,

In preparation for the 1.19.0 release, master tree is currently frozen
until the release process is completed. For committers, until the release
is over and Drill version is changed to 1.20.0-SNAPSHOT, please do not push
any changes into Drill master.

Cheers,

Laurent

Re: [DISCUSS] Drill 1.19.0 release

[Laurent Goujon]

It looks like the parquet patch for DRILL-7934. I'll send an email to
announce the start of the release process.

On Sun, May 30, 2021 at 10:35 PM Laurent Goujon  wrote:

> Ok, let's do that!
>
> On Sat, May 29, 2021, 19:09 Charles Givre  wrote:
>
>> Hi Laurent,
>> We got the CVEs sorted out, and at least one of the parquet bugs was
>> sorted.  Maybe give it until Tuesday and if DRILL-7934 is merged, great, if
>> not, we go without it and start preparing a release.
>> Does that work?
>> - C
>>
>> > On May 28, 2021, at 9:55 PM, luoc  wrote:
>> >
>> > Hi Laurent,
>> >  That's right. Thanks all for the contributions. As Charles said, We
>> plan to speed up the release frequency. I'm ready to post the [VOTE] mail
>> at the end of 1.19 release.
>> >
>> >> 在 2021年5月29日，01:55，Laurent Goujon  写道：
>> >>
>> >> Today's update: several changes related to the CVEs have been merged,
>> along
>> >> with a bugfix for Parquet. Thanks to all of you who helped on those
>> changes.
>> >> I believe there's only one Parquet change left for DRILL-7934:
>> >> <https://issues.apache.org/jira/browse/DRILL-7934> Charles, is this
>> correct?
>> >>
>> >> Laurent
>> >>
>> >>> On Thu, May 27, 2021 at 10:48 AM Laurent Goujon 
>> wrote:
>> >>>
>> >>> Some fixes/improvements were made to the codebase since the last
>> release,
>> >>> and sadly an official release is needed to pick up those changes. Ray
>> asked
>> >>> the community more than a month ago. More recently, other people have
>> been
>> >>> asking too on the user mailing list.
>> >>>
>> >>> Like I said, it might be okay to change the scope but what I'm asking
>> is a
>> >>> little help/transparency here because it looks like I'm chasing a
>> moving
>> >>> target. If we can clarify which new issues have to be part of the
>> release
>> >>> and why (depending on the severity), and how long we think it will
>> take,
>> >>> I'd hope we can have some constructive discussion.
>> >>>
>> >>> As for the dependencies change:
>> >>> - I actually wrote a pull request to address CVEs in both Hadoop and
>> Jetty
>> >>> - The Guava change will not address the most recent CVE. To address
>> the
>> >>> CVE, code must be changed, and it doesn't require a Guava update. The
>> >>> change made to the Guava library was to deprecate the unsecure
>> method... So
>> >>> imho updating dependencies to address CVE without looking at the CVE
>> itself
>> >>> does not make things safer. So to address specifically the CVE, I
>> opened a
>> >>> new ticket (DRILL-7936 <
>> https://issues.apache.org/jira/browse/DRILL-7936>)
>> >>> and a pull request (https://github.com/apache/drill/pull/2240)
>> >>>
>> >>>
>> >>>> On Thu, May 27, 2021 at 9:30 AM Charles Givre 
>> wrote:
>> >>>>
>> >>>> Hi Laurent,
>> >>>> I’m not sure what the rush is to get a release out.  I would much
>> rather
>> >>>> do a quality release than just get something out the door for the
>> sake of
>> >>>> getting something out the door.
>> >>>>
>> >>>> In reference to Drill-7934 (Parquet), DRILL-7919 I am personally not
>> in
>> >>>> favor of putting out a release with known bugs, especially when
>> these bugs
>> >>>> affect parts of Drill that are in active use, we don’t do releases
>> that
>> >>>> frequently, and there is a PR that is awaiting merge.
>> >>>>
>> >>>> I’m also not in favor of a release that has known issues with
>> >>>> dependencies, especially again when there are pending PRs that
>> address
>> >>>> these CVEs.  If we did more frequent releases (which we have
>> discussed and
>> >>>> hope to do going forward), then fine, but we’ve been averaging 2 a
>> year and
>> >>>> I’d hate for users to have to wait 6 months for these fixes.
>> >>>>
>> >>>> — C
>> >>>>
>> >>>>
>> >>>>
>> >>>>> On May 27, 2021, at 12:19 PM, Laurent Goujon 
>> &g

Re: [DISCUSS] Drill 1.19.0 release

[Laurent Goujon]

Ok, let's do that!

On Sat, May 29, 2021, 19:09 Charles Givre  wrote:

> Hi Laurent,
> We got the CVEs sorted out, and at least one of the parquet bugs was
> sorted.  Maybe give it until Tuesday and if DRILL-7934 is merged, great, if
> not, we go without it and start preparing a release.
> Does that work?
> - C
>
> > On May 28, 2021, at 9:55 PM, luoc  wrote:
> >
> > Hi Laurent,
> >  That's right. Thanks all for the contributions. As Charles said, We
> plan to speed up the release frequency. I'm ready to post the [VOTE] mail
> at the end of 1.19 release.
> >
> >> 在 2021年5月29日，01:55，Laurent Goujon  写道：
> >>
> >> Today's update: several changes related to the CVEs have been merged,
> along
> >> with a bugfix for Parquet. Thanks to all of you who helped on those
> changes.
> >> I believe there's only one Parquet change left for DRILL-7934:
> >> <https://issues.apache.org/jira/browse/DRILL-7934> Charles, is this
> correct?
> >>
> >> Laurent
> >>
> >>> On Thu, May 27, 2021 at 10:48 AM Laurent Goujon 
> wrote:
> >>>
> >>> Some fixes/improvements were made to the codebase since the last
> release,
> >>> and sadly an official release is needed to pick up those changes. Ray
> asked
> >>> the community more than a month ago. More recently, other people have
> been
> >>> asking too on the user mailing list.
> >>>
> >>> Like I said, it might be okay to change the scope but what I'm asking
> is a
> >>> little help/transparency here because it looks like I'm chasing a
> moving
> >>> target. If we can clarify which new issues have to be part of the
> release
> >>> and why (depending on the severity), and how long we think it will
> take,
> >>> I'd hope we can have some constructive discussion.
> >>>
> >>> As for the dependencies change:
> >>> - I actually wrote a pull request to address CVEs in both Hadoop and
> Jetty
> >>> - The Guava change will not address the most recent CVE. To address the
> >>> CVE, code must be changed, and it doesn't require a Guava update. The
> >>> change made to the Guava library was to deprecate the unsecure
> method... So
> >>> imho updating dependencies to address CVE without looking at the CVE
> itself
> >>> does not make things safer. So to address specifically the CVE, I
> opened a
> >>> new ticket (DRILL-7936 <
> https://issues.apache.org/jira/browse/DRILL-7936>)
> >>> and a pull request (https://github.com/apache/drill/pull/2240)
> >>>
> >>>
> >>>> On Thu, May 27, 2021 at 9:30 AM Charles Givre 
> wrote:
> >>>>
> >>>> Hi Laurent,
> >>>> I’m not sure what the rush is to get a release out.  I would much
> rather
> >>>> do a quality release than just get something out the door for the
> sake of
> >>>> getting something out the door.
> >>>>
> >>>> In reference to Drill-7934 (Parquet), DRILL-7919 I am personally not
> in
> >>>> favor of putting out a release with known bugs, especially when these
> bugs
> >>>> affect parts of Drill that are in active use, we don’t do releases
> that
> >>>> frequently, and there is a PR that is awaiting merge.
> >>>>
> >>>> I’m also not in favor of a release that has known issues with
> >>>> dependencies, especially again when there are pending PRs that address
> >>>> these CVEs.  If we did more frequent releases (which we have
> discussed and
> >>>> hope to do going forward), then fine, but we’ve been averaging 2 a
> year and
> >>>> I’d hate for users to have to wait 6 months for these fixes.
> >>>>
> >>>> — C
> >>>>
> >>>>
> >>>>
> >>>>> On May 27, 2021, at 12:19 PM, Laurent Goujon 
> >>>> wrote:
> >>>>>
> >>>>> Since I'm also a reviewer and that I see that the past comments I've
> >>>> been
> >>>>> addressed, and since I do not see another committer opposing the
> patch,
> >>>>> wouldn't I be able to give my +1 and that would clear that bar?
> >>>>>
> >>>>> As for the parquet issues, when we started the release discussion a
> >>>> month
> >>>>> ago, we agreed on a scope, an

Re: [DISCUSS] Drill 1.19.0 release

[Laurent Goujon]

Today's update: several changes related to the CVEs have been merged, along
with a bugfix for Parquet. Thanks to all of you who helped on those changes.
I believe there's only one Parquet change left for DRILL-7934:
<https://issues.apache.org/jira/browse/DRILL-7934> Charles, is this correct?

Laurent

On Thu, May 27, 2021 at 10:48 AM Laurent Goujon  wrote:

> Some fixes/improvements were made to the codebase since the last release,
> and sadly an official release is needed to pick up those changes. Ray asked
> the community more than a month ago. More recently, other people have been
> asking too on the user mailing list.
>
> Like I said, it might be okay to change the scope but what I'm asking is a
> little help/transparency here because it looks like I'm chasing a moving
> target. If we can clarify which new issues have to be part of the release
> and why (depending on the severity), and how long we think it will take,
> I'd hope we can have some constructive discussion.
>
> As for the dependencies change:
> - I actually wrote a pull request to address CVEs in both Hadoop and Jetty
> - The Guava change will not address the most recent CVE. To address the
> CVE, code must be changed, and it doesn't require a Guava update. The
> change made to the Guava library was to deprecate the unsecure method... So
> imho updating dependencies to address CVE without looking at the CVE itself
> does not make things safer. So to address specifically the CVE, I opened a
> new ticket (DRILL-7936 <https://issues.apache.org/jira/browse/DRILL-7936>)
> and a pull request (https://github.com/apache/drill/pull/2240)
>
>
> On Thu, May 27, 2021 at 9:30 AM Charles Givre  wrote:
>
>> Hi Laurent,
>> I’m not sure what the rush is to get a release out.  I would much rather
>> do a quality release than just get something out the door for the sake of
>> getting something out the door.
>>
>> In reference to Drill-7934 (Parquet), DRILL-7919 I am personally not in
>> favor of putting out a release with known bugs, especially when these bugs
>> affect parts of Drill that are in active use, we don’t do releases that
>> frequently, and there is a PR that is awaiting merge.
>>
>> I’m also not in favor of a release that has known issues with
>> dependencies, especially again when there are pending PRs that address
>> these CVEs.  If we did more frequent releases (which we have discussed and
>> hope to do going forward), then fine, but we’ve been averaging 2 a year and
>> I’d hate for users to have to wait 6 months for these fixes.
>>
>> — C
>>
>>
>>
>> > On May 27, 2021, at 12:19 PM, Laurent Goujon 
>> wrote:
>> >
>> > Since I'm also a reviewer and that I see that the past comments I've
>> been
>> > addressed, and since I do not see another committer opposing the patch,
>> > wouldn't I be able to give my +1 and that would clear that bar?
>> >
>> > As for the parquet issues, when we started the release discussion a
>> month
>> > ago, we agreed on a scope, and the parquet issues were not part of it. I
>> > understand that scope can change but can we discuss it in this thread
>> about
>> > why this release should include it vs wait on the next release? We need
>> to
>> > draw a line somewhere.
>> >
>> > Laurent
>> >
>> > On Thu, May 27, 2021 at 8:05 AM Charles Givre  wrote:
>> >
>> >> Laurent,
>> >> Per Apache policy, you need a +1 from a reviewer to merge a PR.  Unless
>> >> there is one, please do not merge.  I'll reach out to Vitalii to see
>> what
>> >> the current status is.   Also there are a few bug fixes for the Parquet
>> >> which Vova submitted which looks like we should include as well.
>> >> Best,
>> >> -- C
>> >>
>> >>> On May 27, 2021, at 11:01 AM, Laurent Goujon 
>> wrote:
>> >>>
>> >>> Sadly, I haven't heard from people regarding the patches. At the same
>> >> time,
>> >>> I think we held the window open for merging the changes for a very
>> long
>> >>> time. Unless there's objection, I'm planning to merge the Guava and
>> >>> Jetty/Hadoop pull requests later today, and doing the first RC for
>> Drill
>> >>> 1.19.0
>> >>>
>> >>> Here are the pull request links:
>> >>> * https://github.com/apache/drill/pull/2202
>> >>> * https://github.com/apache/drill/pull/2236
>> >>>
>> >&g

[jira] [Resolved] (DRILL-7936) Remove Guava Files#createTempDir usage

[Laurent Goujon (Jira)]

 [ 
https://issues.apache.org/jira/browse/DRILL-7936?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Laurent Goujon resolved DRILL-7936.
---
Fix Version/s: 1.19.0
 Reviewer: Charles Givre
   Resolution: Fixed

> Remove Guava Files#createTempDir usage
> --
>
> Key: DRILL-7936
> URL: https://issues.apache.org/jira/browse/DRILL-7936
> Project: Apache Drill
>  Issue Type: Bug
>    Reporter: Laurent Goujon
>        Assignee: Laurent Goujon
>Priority: Major
> Fix For: 1.19.0
>
>
> According to CVE-2020-8908, method Files#createTempDir has some security 
> issues, and usage has been deprecated in Guava 30.0.
> Usage of this function in Drill should be replaced with the native Java 
> version as recommended by the Guava team.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Resolved] (DRILL-7162) Apache Drill uses 3rd Party with Highest CVEs

[Laurent Goujon (Jira)]

 [ 
https://issues.apache.org/jira/browse/DRILL-7162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Laurent Goujon resolved DRILL-7162.
---
Fix Version/s: 1.19.0
   Resolution: Fixed

Apache Drill has been updated to the latest Jetty 9.4 version available. This 
should address all the CVEs on the list.

>  Apache Drill uses 3rd Party with Highest CVEs
> --
>
> Key: DRILL-7162
> URL: https://issues.apache.org/jira/browse/DRILL-7162
> Project: Apache Drill
>  Issue Type: Bug
>Affects Versions: 1.13.0, 1.14.0, 1.15.0
>Reporter: Ayush Sharma
>Priority: Major
> Fix For: 1.19.0
>
> Attachments: Jars.xlsx
>
>
> Apache Drill uses 3rd party libraries with almost 250+ CVEs.
> Most of the CVEs are in the older version of Jetty (9.1.x) whereas the 
> current version of Jetty is 9.4.x
> Also many of the other libraries are in EOF versions and the are not patched 
> even in the latest release.
> This creates an issue of security when we use it in production.
> We are able to replace many older version of libraries with the latest 
> versions with no CVEs , however many of them are not replaceable as it is and 
> would require some changes in the source code.
> The jetty version is of the highest priority and needs migration to 9.4.x 
> version immediately.
>  
> Please look into this issue at immediate priority as it compromises with the 
> security of the application utilizing Apache Drill.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Resolved] (DRILL-7135) Upgrade to Jetty 9.4

[Laurent Goujon (Jira)]

 [ 
https://issues.apache.org/jira/browse/DRILL-7135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Laurent Goujon resolved DRILL-7135.
---
Fix Version/s: 1.19.0
 Reviewer: Paul Rogers
 Assignee: Laurent Goujon
   Resolution: Fixed

> Upgrade to Jetty 9.4
> 
>
> Key: DRILL-7135
> URL: https://issues.apache.org/jira/browse/DRILL-7135
> Project: Apache Drill
>  Issue Type: Improvement
>Affects Versions: 1.15.0
>Reporter: Vitalii Diravka
>    Assignee: Laurent Goujon
>Priority: Minor
> Fix For: 1.19.0
>
>
> Initially DRILL-7051 updated Jetty to 9.4 version and DRILL-7081 updated 
> Jersey version to 2.28 version. These versions work fine for Drill with 
> Hadoop version below 3.0.
>  Starting from Hadoop 3.0 it uses 
> [org.eclipse.jetty|https://github.com/apache/hadoop/blob/branch-3.0/hadoop-project/pom.xml#L38]
>  9.3 version.
>  That's why it conflicts with newer Jetty&Jersey versions.
> Drill can update Jetty and Jersey versions after resolution HADOOP-14930 and 
> HBASE-19256.
>  Or alternatively these libs can be shaded in Drill, but there is no real 
> reason to do it nowadays.
> See details in 
> [#1681|https://github.com/apache/drill/pull/1681#discussion_r265904521] PR.
> _Notes_: 
> * For Jersey update it is necessary to add 
> org.glassfish.jersey.inject:jersey-hk2 in Drill to solve all compilation 
> failures.
> * See doc for Jetty update: 
> https://www.eclipse.org/jetty/documentation/9.4.x/upgrading-jetty.html



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Resolved] (DRILL-7932) Update Hadoop version to 3.2.2.

[Laurent Goujon (Jira)]

 [ 
https://issues.apache.org/jira/browse/DRILL-7932?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Laurent Goujon resolved DRILL-7932.
---
Fix Version/s: 1.19.0
 Reviewer: Cong Luo
   Resolution: Fixed

Update done as part of the fix for DRILL-7135

> Update Hadoop version to 3.2.2.
> ---
>
> Key: DRILL-7932
> URL: https://issues.apache.org/jira/browse/DRILL-7932
> Project: Apache Drill
>  Issue Type: Improvement
>    Reporter: Laurent Goujon
>        Assignee: Laurent Goujon
>Priority: Major
> Fix For: 1.19.0
>
>
> Update Hadoop version to the latest patch version for the 3.2.x branch.
> This should address possible security vulnerability 
> [https://nvd.nist.gov/vuln/detail/CVE-2020-9492|CVE-2020-9492]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [DISCUSS] Drill 1.19.0 release

[Laurent Goujon]

Some fixes/improvements were made to the codebase since the last release,
and sadly an official release is needed to pick up those changes. Ray asked
the community more than a month ago. More recently, other people have been
asking too on the user mailing list.

Like I said, it might be okay to change the scope but what I'm asking is a
little help/transparency here because it looks like I'm chasing a moving
target. If we can clarify which new issues have to be part of the release
and why (depending on the severity), and how long we think it will take,
I'd hope we can have some constructive discussion.

As for the dependencies change:
- I actually wrote a pull request to address CVEs in both Hadoop and Jetty
- The Guava change will not address the most recent CVE. To address the
CVE, code must be changed, and it doesn't require a Guava update. The
change made to the Guava library was to deprecate the unsecure method... So
imho updating dependencies to address CVE without looking at the CVE itself
does not make things safer. So to address specifically the CVE, I opened a
new ticket (DRILL-7936 <https://issues.apache.org/jira/browse/DRILL-7936>)
and a pull request (https://github.com/apache/drill/pull/2240)


On Thu, May 27, 2021 at 9:30 AM Charles Givre  wrote:

> Hi Laurent,
> I’m not sure what the rush is to get a release out.  I would much rather
> do a quality release than just get something out the door for the sake of
> getting something out the door.
>
> In reference to Drill-7934 (Parquet), DRILL-7919 I am personally not in
> favor of putting out a release with known bugs, especially when these bugs
> affect parts of Drill that are in active use, we don’t do releases that
> frequently, and there is a PR that is awaiting merge.
>
> I’m also not in favor of a release that has known issues with
> dependencies, especially again when there are pending PRs that address
> these CVEs.  If we did more frequent releases (which we have discussed and
> hope to do going forward), then fine, but we’ve been averaging 2 a year and
> I’d hate for users to have to wait 6 months for these fixes.
>
> — C
>
>
>
> > On May 27, 2021, at 12:19 PM, Laurent Goujon  wrote:
> >
> > Since I'm also a reviewer and that I see that the past comments I've been
> > addressed, and since I do not see another committer opposing the patch,
> > wouldn't I be able to give my +1 and that would clear that bar?
> >
> > As for the parquet issues, when we started the release discussion a month
> > ago, we agreed on a scope, and the parquet issues were not part of it. I
> > understand that scope can change but can we discuss it in this thread
> about
> > why this release should include it vs wait on the next release? We need
> to
> > draw a line somewhere.
> >
> > Laurent
> >
> > On Thu, May 27, 2021 at 8:05 AM Charles Givre  wrote:
> >
> >> Laurent,
> >> Per Apache policy, you need a +1 from a reviewer to merge a PR.  Unless
> >> there is one, please do not merge.  I'll reach out to Vitalii to see
> what
> >> the current status is.   Also there are a few bug fixes for the Parquet
> >> which Vova submitted which looks like we should include as well.
> >> Best,
> >> -- C
> >>
> >>> On May 27, 2021, at 11:01 AM, Laurent Goujon 
> wrote:
> >>>
> >>> Sadly, I haven't heard from people regarding the patches. At the same
> >> time,
> >>> I think we held the window open for merging the changes for a very long
> >>> time. Unless there's objection, I'm planning to merge the Guava and
> >>> Jetty/Hadoop pull requests later today, and doing the first RC for
> Drill
> >>> 1.19.0
> >>>
> >>> Here are the pull request links:
> >>> * https://github.com/apache/drill/pull/2202
> >>> * https://github.com/apache/drill/pull/2236
> >>>
> >>> Laurent
> >>>
> >>>
> >>> On Wed, May 26, 2021 at 11:59 AM Laurent Goujon 
> >> wrote:
> >>>
> >>>> After several retries, the Guava checks successfully passed:
> >>>> https://github.com/apache/drill/pull/2202
> >>>>
> >>>> Charles, can we proceed on merging your change?
> >>>>
> >>>> Laurent
> >>>>
> >>>> On Tue, May 25, 2021 at 10:24 PM Laurent Goujon 
> >>>> wrote:
> >>>>
> >>>>> Just an update. There's a patch for updating both Jetty and Hadoop
> (at
> >>>>> the same time) as those changes are co-dependent:
> >

[jira] [Created] (DRILL-7936) Remove Guava Files#createTempDir usage

[Laurent Goujon (Jira)]

Laurent Goujon created DRILL-7936:
-

 Summary: Remove Guava Files#createTempDir usage
 Key: DRILL-7936
 URL: https://issues.apache.org/jira/browse/DRILL-7936
 Project: Apache Drill
  Issue Type: Bug
Reporter: Laurent Goujon
Assignee: Laurent Goujon


According to CVE-2020-8908, method Files#createTempDir has some security 
issues, and usage has been deprecated in Guava 30.0.

Usage of this function in Drill should be replaced with the native Java version 
as recommended by the Guava team.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [DISCUSS] Drill 1.19.0 release

[Laurent Goujon]

Since I'm also a reviewer and that I see that the past comments I've been
addressed, and since I do not see another committer opposing the patch,
wouldn't I be able to give my +1 and that would clear that bar?

As for the parquet issues, when we started the release discussion a month
ago, we agreed on a scope, and the parquet issues were not part of it. I
understand that scope can change but can we discuss it in this thread about
why this release should include it vs wait on the next release? We need to
draw a line somewhere.

Laurent

On Thu, May 27, 2021 at 8:05 AM Charles Givre  wrote:

> Laurent,
> Per Apache policy, you need a +1 from a reviewer to merge a PR.  Unless
> there is one, please do not merge.  I'll reach out to Vitalii to see what
> the current status is.   Also there are a few bug fixes for the Parquet
> which Vova submitted which looks like we should include as well.
> Best,
> -- C
>
> > On May 27, 2021, at 11:01 AM, Laurent Goujon  wrote:
> >
> > Sadly, I haven't heard from people regarding the patches. At the same
> time,
> > I think we held the window open for merging the changes for a very long
> > time. Unless there's objection, I'm planning to merge the Guava and
> > Jetty/Hadoop pull requests later today, and doing the first RC for Drill
> > 1.19.0
> >
> > Here are the pull request links:
> > * https://github.com/apache/drill/pull/2202
> > * https://github.com/apache/drill/pull/2236
> >
> > Laurent
> >
> >
> > On Wed, May 26, 2021 at 11:59 AM Laurent Goujon 
> wrote:
> >
> >> After several retries, the Guava checks successfully passed:
> >> https://github.com/apache/drill/pull/2202
> >>
> >> Charles, can we proceed on merging your change?
> >>
> >> Laurent
> >>
> >> On Tue, May 25, 2021 at 10:24 PM Laurent Goujon 
> >> wrote:
> >>
> >>> Just an update. There's a patch for updating both Jetty and Hadoop (at
> >>> the same time) as those changes are co-dependent:
> >>> https://github.com/apache/drill/pull/2236
> >>>
> >>> As for the Guava patch, I'd be happy to help, but I'm not sure what's
> >>> left. As far as I can tell the shaded version of Guava has been
> updated,
> >>> but the build is failing. The security vulnerabilities for Guava are
> >>> moderate (and actually it seems a fix for CVE-2020-8908 would require a
> >>> code change instead of a Guava update.
> >>>
> >>> Since this has been almost a month since we started this release
> process,
> >>> I wonder if we still want to wait on this patch, or if we should move
> it to
> >>> the next release.
> >>>
> >>> Let me know what people think,
> >>>
> >>> On Tue, May 25, 2021 at 8:24 AM Laurent Goujon 
> >>> wrote:
> >>>
> >>>> Anything I can help with?
> >>>>
> >>>> On Tue, May 25, 2021 at 7:02 AM Charles Givre 
> wrote:
> >>>>
> >>>>> HI Laurent,
> >>>>> My apologies.  I said Junit, when I was meaning to say to the Guava
> PR (
> >>>>> https://github.com/apache/drill/pull/2202 <
> >>>>> https://github.com/apache/drill/pull/2202>).  I think this one is
> >>>>> almost done as well.
> >>>>> -- C
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>> On May 24, 2021, at 5:29 PM, Laurent Goujon 
> >>>>> wrote:
> >>>>>>
> >>>>>> Ok, I was hoping that some of the PRs could be merged, but if we are
> >>>>> in
> >>>>>> agreement, let's start the work :)
> >>>>>>
> >>>>>> On Sun, May 23, 2021 at 6:52 PM luoc  wrote:
> >>>>>>
> >>>>>>> Hi Charles,
> >>>>>>> All right, we'll be expecting the update.
> >>>>>>>
> >>>>>>>> 2021年5月24日 上午12:13，Charles Givre  写道：
> >>>>>>>>
> >>>>>>>> Hi Luoc,
> >>>>>>>> We still have a few PRs pending that we really should get into
> Drill
> >>>>>>> 1.19.  The main one is the junit upgrade.  There are a few critical
> >>>>> CVEs
> >>>>>>> associated with that, so I do think it is important to get that one
>

Re: [DISCUSS] Drill 1.19.0 release

[Laurent Goujon]

Sadly, I haven't heard from people regarding the patches. At the same time,
I think we held the window open for merging the changes for a very long
time. Unless there's objection, I'm planning to merge the Guava and
Jetty/Hadoop pull requests later today, and doing the first RC for Drill
1.19.0

Here are the pull request links:
* https://github.com/apache/drill/pull/2202
* https://github.com/apache/drill/pull/2236

Laurent


On Wed, May 26, 2021 at 11:59 AM Laurent Goujon  wrote:

> After several retries, the Guava checks successfully passed:
> https://github.com/apache/drill/pull/2202
>
> Charles, can we proceed on merging your change?
>
> Laurent
>
> On Tue, May 25, 2021 at 10:24 PM Laurent Goujon 
> wrote:
>
>> Just an update. There's a patch for updating both Jetty and Hadoop (at
>> the same time) as those changes are co-dependent:
>> https://github.com/apache/drill/pull/2236
>>
>> As for the Guava patch, I'd be happy to help, but I'm not sure what's
>> left. As far as I can tell the shaded version of Guava has been updated,
>> but the build is failing. The security vulnerabilities for Guava are
>> moderate (and actually it seems a fix for CVE-2020-8908 would require a
>> code change instead of a Guava update.
>>
>> Since this has been almost a month since we started this release process,
>> I wonder if we still want to wait on this patch, or if we should move it to
>> the next release.
>>
>> Let me know what people think,
>>
>> On Tue, May 25, 2021 at 8:24 AM Laurent Goujon 
>> wrote:
>>
>>> Anything I can help with?
>>>
>>> On Tue, May 25, 2021 at 7:02 AM Charles Givre  wrote:
>>>
>>>> HI Laurent,
>>>> My apologies.  I said Junit, when I was meaning to say to the Guava PR (
>>>> https://github.com/apache/drill/pull/2202 <
>>>> https://github.com/apache/drill/pull/2202>).  I think this one is
>>>> almost done as well.
>>>> -- C
>>>>
>>>>
>>>>
>>>>
>>>> > On May 24, 2021, at 5:29 PM, Laurent Goujon 
>>>> wrote:
>>>> >
>>>> > Ok, I was hoping that some of the PRs could be merged, but if we are
>>>> in
>>>> > agreement, let's start the work :)
>>>> >
>>>> > On Sun, May 23, 2021 at 6:52 PM luoc  wrote:
>>>> >
>>>> >> Hi Charles,
>>>> >>  All right, we'll be expecting the update.
>>>> >>
>>>> >>> 2021年5月24日 上午12:13，Charles Givre  写道：
>>>> >>>
>>>> >>> Hi Luoc,
>>>> >>> We still have a few PRs pending that we really should get into Drill
>>>> >> 1.19.  The main one is the junit upgrade.  There are a few critical
>>>> CVEs
>>>> >> associated with that, so I do think it is important to get that one
>>>> >> merged.  I think Vitalii will have that one done in short order.
>>>> >>> Best,
>>>> >>> -- C
>>>> >>>
>>>> >>>
>>>> >>>
>>>> >>>> On May 22, 2021, at 5:16 AM, luoc  wrote:
>>>> >>>>
>>>> >>>> Hi Laurent,
>>>> >>>> It’s time to do a release with 1.19.0.
>>>> >>>>
>>>> >>>>> 2021年5月19日 上午2:20，Vitalii Diravka  写道：
>>>> >>>>>
>>>> >>>>> Hi Laurent,
>>>> >>>>> DRILL-7871 requires additional time to be introduced and it is
>>>> better
>>>> >> to
>>>> >>>>> include it for the next release.
>>>> >>>>> DRILL-7904 is updated, I think it will be merged in a few days.
>>>> But it
>>>> >>>>> doesn't matter whether it is included in this release or in the
>>>> next
>>>> >> one.
>>>> >>>>>
>>>> >>>>> So we can plan to start the release process
>>>> >>>>>
>>>> >>>>>
>>>> >>>>> Kind regards
>>>> >>>>> Vitalii
>>>> >>>>>
>>>> >>>>>
>>>> >>>>> On Tue, May 11, 2021 at 7:52 PM Laurent Goujon <
>>>> laur...@dremio.com>
>>>> >> wrote:
>>>> >>>>>
>>>

Re: [DISCUSS] Drill 1.19.0 release

[Laurent Goujon]

After several retries, the Guava checks successfully passed:
https://github.com/apache/drill/pull/2202

Charles, can we proceed on merging your change?

Laurent

On Tue, May 25, 2021 at 10:24 PM Laurent Goujon  wrote:

> Just an update. There's a patch for updating both Jetty and Hadoop (at the
> same time) as those changes are co-dependent:
> https://github.com/apache/drill/pull/2236
>
> As for the Guava patch, I'd be happy to help, but I'm not sure what's
> left. As far as I can tell the shaded version of Guava has been updated,
> but the build is failing. The security vulnerabilities for Guava are
> moderate (and actually it seems a fix for CVE-2020-8908 would require a
> code change instead of a Guava update.
>
> Since this has been almost a month since we started this release process,
> I wonder if we still want to wait on this patch, or if we should move it to
> the next release.
>
> Let me know what people think,
>
> On Tue, May 25, 2021 at 8:24 AM Laurent Goujon  wrote:
>
>> Anything I can help with?
>>
>> On Tue, May 25, 2021 at 7:02 AM Charles Givre  wrote:
>>
>>> HI Laurent,
>>> My apologies.  I said Junit, when I was meaning to say to the Guava PR (
>>> https://github.com/apache/drill/pull/2202 <
>>> https://github.com/apache/drill/pull/2202>).  I think this one is
>>> almost done as well.
>>> -- C
>>>
>>>
>>>
>>>
>>> > On May 24, 2021, at 5:29 PM, Laurent Goujon 
>>> wrote:
>>> >
>>> > Ok, I was hoping that some of the PRs could be merged, but if we are in
>>> > agreement, let's start the work :)
>>> >
>>> > On Sun, May 23, 2021 at 6:52 PM luoc  wrote:
>>> >
>>> >> Hi Charles,
>>> >>  All right, we'll be expecting the update.
>>> >>
>>> >>> 2021年5月24日 上午12:13，Charles Givre  写道：
>>> >>>
>>> >>> Hi Luoc,
>>> >>> We still have a few PRs pending that we really should get into Drill
>>> >> 1.19.  The main one is the junit upgrade.  There are a few critical
>>> CVEs
>>> >> associated with that, so I do think it is important to get that one
>>> >> merged.  I think Vitalii will have that one done in short order.
>>> >>> Best,
>>> >>> -- C
>>> >>>
>>> >>>
>>> >>>
>>> >>>> On May 22, 2021, at 5:16 AM, luoc  wrote:
>>> >>>>
>>> >>>> Hi Laurent,
>>> >>>> It’s time to do a release with 1.19.0.
>>> >>>>
>>> >>>>> 2021年5月19日 上午2:20，Vitalii Diravka  写道：
>>> >>>>>
>>> >>>>> Hi Laurent,
>>> >>>>> DRILL-7871 requires additional time to be introduced and it is
>>> better
>>> >> to
>>> >>>>> include it for the next release.
>>> >>>>> DRILL-7904 is updated, I think it will be merged in a few days.
>>> But it
>>> >>>>> doesn't matter whether it is included in this release or in the
>>> next
>>> >> one.
>>> >>>>>
>>> >>>>> So we can plan to start the release process
>>> >>>>>
>>> >>>>>
>>> >>>>> Kind regards
>>> >>>>> Vitalii
>>> >>>>>
>>> >>>>>
>>> >>>>> On Tue, May 11, 2021 at 7:52 PM Laurent Goujon >> >
>>> >> wrote:
>>> >>>>>
>>> >>>>>> Thanks Vitalii
>>> >>>>>>
>>> >>>>>> On Tue, May 11, 2021 at 9:29 AM Vitalii Diravka <
>>> vita...@apache.org>
>>> >>>>>> wrote:
>>> >>>>>>
>>> >>>>>>> Hi Luoc!
>>> >>>>>>>
>>> >>>>>>> They are almost ready. I plan to update PR for them today.
>>> >>>>>>>
>>> >>>>>>> Kind regards
>>> >>>>>>> Vitalii
>>> >>>>>>>
>>> >>>>>>>
>>> >>>>>>> On Sat, May 8, 2021 at 5:26 PM luoc  wrote:
>>> >>>>>>>
>>> >>>>>>>> Hi Vitalii,
>>> >>>>>>>> Would you mi

Re: [DISCUSS] Drill 1.19.0 release

[Laurent Goujon]

Just an update. There's a patch for updating both Jetty and Hadoop (at the
same time) as those changes are co-dependent:
https://github.com/apache/drill/pull/2236

As for the Guava patch, I'd be happy to help, but I'm not sure what's left.
As far as I can tell the shaded version of Guava has been updated, but the
build is failing. The security vulnerabilities for Guava are moderate (and
actually it seems a fix for CVE-2020-8908 would require a code change
instead of a Guava update.

Since this has been almost a month since we started this release process, I
wonder if we still want to wait on this patch, or if we should move it to
the next release.

Let me know what people think,

On Tue, May 25, 2021 at 8:24 AM Laurent Goujon  wrote:

> Anything I can help with?
>
> On Tue, May 25, 2021 at 7:02 AM Charles Givre  wrote:
>
>> HI Laurent,
>> My apologies.  I said Junit, when I was meaning to say to the Guava PR (
>> https://github.com/apache/drill/pull/2202 <
>> https://github.com/apache/drill/pull/2202>).  I think this one is almost
>> done as well.
>> -- C
>>
>>
>>
>>
>> > On May 24, 2021, at 5:29 PM, Laurent Goujon  wrote:
>> >
>> > Ok, I was hoping that some of the PRs could be merged, but if we are in
>> > agreement, let's start the work :)
>> >
>> > On Sun, May 23, 2021 at 6:52 PM luoc  wrote:
>> >
>> >> Hi Charles,
>> >>  All right, we'll be expecting the update.
>> >>
>> >>> 2021年5月24日 上午12:13，Charles Givre  写道：
>> >>>
>> >>> Hi Luoc,
>> >>> We still have a few PRs pending that we really should get into Drill
>> >> 1.19.  The main one is the junit upgrade.  There are a few critical
>> CVEs
>> >> associated with that, so I do think it is important to get that one
>> >> merged.  I think Vitalii will have that one done in short order.
>> >>> Best,
>> >>> -- C
>> >>>
>> >>>
>> >>>
>> >>>> On May 22, 2021, at 5:16 AM, luoc  wrote:
>> >>>>
>> >>>> Hi Laurent,
>> >>>> It’s time to do a release with 1.19.0.
>> >>>>
>> >>>>> 2021年5月19日 上午2:20，Vitalii Diravka  写道：
>> >>>>>
>> >>>>> Hi Laurent,
>> >>>>> DRILL-7871 requires additional time to be introduced and it is
>> better
>> >> to
>> >>>>> include it for the next release.
>> >>>>> DRILL-7904 is updated, I think it will be merged in a few days. But
>> it
>> >>>>> doesn't matter whether it is included in this release or in the next
>> >> one.
>> >>>>>
>> >>>>> So we can plan to start the release process
>> >>>>>
>> >>>>>
>> >>>>> Kind regards
>> >>>>> Vitalii
>> >>>>>
>> >>>>>
>> >>>>> On Tue, May 11, 2021 at 7:52 PM Laurent Goujon 
>> >> wrote:
>> >>>>>
>> >>>>>> Thanks Vitalii
>> >>>>>>
>> >>>>>> On Tue, May 11, 2021 at 9:29 AM Vitalii Diravka <
>> vita...@apache.org>
>> >>>>>> wrote:
>> >>>>>>
>> >>>>>>> Hi Luoc!
>> >>>>>>>
>> >>>>>>> They are almost ready. I plan to update PR for them today.
>> >>>>>>>
>> >>>>>>> Kind regards
>> >>>>>>> Vitalii
>> >>>>>>>
>> >>>>>>>
>> >>>>>>> On Sat, May 8, 2021 at 5:26 PM luoc  wrote:
>> >>>>>>>
>> >>>>>>>> Hi Vitalii,
>> >>>>>>>> Would you mind sharing that... Is DRILL-7904 ready to review
>> again?
>> >>>>>>> And what’s
>> >>>>>>>> the status on the DRILL-7871? thanks
>> >>>>>>>>
>> >>>>>>>> 2021年5月4日 下午1:10，Ted Dunning  写道：
>> >>>>>>>>
>> >>>>>>>> Laurent,
>> >>>>>>>>
>> >>>>>>>> I don't have a stake here, so can't really comment about
>> specifics,
>> >> but
>> >>>>>>> the
>> >>>>>

Re: [DISCUSS] Drill 1.19.0 release

[Laurent Goujon]

Anything I can help with?

On Tue, May 25, 2021 at 7:02 AM Charles Givre  wrote:

> HI Laurent,
> My apologies.  I said Junit, when I was meaning to say to the Guava PR (
> https://github.com/apache/drill/pull/2202 <
> https://github.com/apache/drill/pull/2202>).  I think this one is almost
> done as well.
> -- C
>
>
>
>
> > On May 24, 2021, at 5:29 PM, Laurent Goujon  wrote:
> >
> > Ok, I was hoping that some of the PRs could be merged, but if we are in
> > agreement, let's start the work :)
> >
> > On Sun, May 23, 2021 at 6:52 PM luoc  wrote:
> >
> >> Hi Charles,
> >>  All right, we'll be expecting the update.
> >>
> >>> 2021年5月24日 上午12:13，Charles Givre  写道：
> >>>
> >>> Hi Luoc,
> >>> We still have a few PRs pending that we really should get into Drill
> >> 1.19.  The main one is the junit upgrade.  There are a few critical CVEs
> >> associated with that, so I do think it is important to get that one
> >> merged.  I think Vitalii will have that one done in short order.
> >>> Best,
> >>> -- C
> >>>
> >>>
> >>>
> >>>> On May 22, 2021, at 5:16 AM, luoc  wrote:
> >>>>
> >>>> Hi Laurent,
> >>>> It’s time to do a release with 1.19.0.
> >>>>
> >>>>> 2021年5月19日 上午2:20，Vitalii Diravka  写道：
> >>>>>
> >>>>> Hi Laurent,
> >>>>> DRILL-7871 requires additional time to be introduced and it is better
> >> to
> >>>>> include it for the next release.
> >>>>> DRILL-7904 is updated, I think it will be merged in a few days. But
> it
> >>>>> doesn't matter whether it is included in this release or in the next
> >> one.
> >>>>>
> >>>>> So we can plan to start the release process
> >>>>>
> >>>>>
> >>>>> Kind regards
> >>>>> Vitalii
> >>>>>
> >>>>>
> >>>>> On Tue, May 11, 2021 at 7:52 PM Laurent Goujon 
> >> wrote:
> >>>>>
> >>>>>> Thanks Vitalii
> >>>>>>
> >>>>>> On Tue, May 11, 2021 at 9:29 AM Vitalii Diravka  >
> >>>>>> wrote:
> >>>>>>
> >>>>>>> Hi Luoc!
> >>>>>>>
> >>>>>>> They are almost ready. I plan to update PR for them today.
> >>>>>>>
> >>>>>>> Kind regards
> >>>>>>> Vitalii
> >>>>>>>
> >>>>>>>
> >>>>>>> On Sat, May 8, 2021 at 5:26 PM luoc  wrote:
> >>>>>>>
> >>>>>>>> Hi Vitalii,
> >>>>>>>> Would you mind sharing that... Is DRILL-7904 ready to review
> again?
> >>>>>>> And what’s
> >>>>>>>> the status on the DRILL-7871? thanks
> >>>>>>>>
> >>>>>>>> 2021年5月4日 下午1:10，Ted Dunning  写道：
> >>>>>>>>
> >>>>>>>> Laurent,
> >>>>>>>>
> >>>>>>>> I don't have a stake here, so can't really comment about
> specifics,
> >> but
> >>>>>>> the
> >>>>>>>> process is looking good.
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> On Mon, May 3, 2021 at 9:23 PM Laurent Goujon  >
> >>>>>>> wrote:
> >>>>>>>>
> >>>>>>>> Thanks for all the answers
> >>>>>>>>
> >>>>>>>> So the issues I found based on the feedback are:
> >>>>>>>>
> >>>>>>>> - DRILL-7878: Fix LGTM Alerts
> >>>>>>>> <https://issues.apache.org/jira/browse/DRILL-7878>
> >>>>>>>> - DRILL-7871: StoragePluginStore instances for different users
> >>>>>>>> <https://issues.apache.org/jira/browse/DRILL-7871>
> >>>>>>>> - DRILL-7908: Fix GitHub Actions CI
> >>>>>>>> <https://issues.apache.org/jira/browse/DRILL-7908>
> >>>>>>>> - DRILL-7904: Update to 30-jre Guava version
> >>>>&

[jira] [Created] (DRILL-7932) Update Hadoop version to 3.2.2.

[Laurent Goujon (Jira)]

Laurent Goujon created DRILL-7932:
-

 Summary: Update Hadoop version to 3.2.2.
 Key: DRILL-7932
 URL: https://issues.apache.org/jira/browse/DRILL-7932
 Project: Apache Drill
  Issue Type: Improvement
Reporter: Laurent Goujon
Assignee: Laurent Goujon


Update Hadoop version to the latest patch version for the 3.2.x branch.

This should address possible security vulnerability 
[https://nvd.nist.gov/vuln/detail/CVE-2020-9492|CVE-2020-9492]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [DISCUSS] Drill 1.19.0 release

[Laurent Goujon]

The JUnit patch has been merged a couple of days ago, I do not see any
other open pull requests labeled with security.
I also ran the dependency check report too and since the report is quite
big, I've uploaded it to Google drive, and made it available at the
following location:
https://drive.google.com/file/d/1xzt0WMWG2hGxRyllcyaxz5TYKWOKA_Gd/view?usp=sharing
.
There are a couple of critical issues related to Hadoop and Hive, but also
for Jetty.  I can pull requests for the Hadoop and Jetty ones, the Hive
seems quite new and to be fixed in Hive 4.0

On Mon, May 24, 2021 at 2:30 PM Laurent Goujon  wrote:

> I might be able to help on the JUnit and CVE patches too too
>
> On Mon, May 24, 2021 at 2:29 PM Laurent Goujon  wrote:
>
>> Ok, I was hoping that some of the PRs could be merged, but if we are in
>> agreement, let's start the work :)
>>
>> On Sun, May 23, 2021 at 6:52 PM luoc  wrote:
>>
>>> Hi Charles,
>>>   All right, we'll be expecting the update.
>>>
>>> > 2021年5月24日 上午12:13，Charles Givre  写道：
>>> >
>>> > Hi Luoc,
>>> > We still have a few PRs pending that we really should get into Drill
>>> 1.19.  The main one is the junit upgrade.  There are a few critical CVEs
>>> associated with that, so I do think it is important to get that one
>>> merged.  I think Vitalii will have that one done in short order.
>>> > Best,
>>> > -- C
>>> >
>>> >
>>> >
>>> >> On May 22, 2021, at 5:16 AM, luoc  wrote:
>>> >>
>>> >> Hi Laurent,
>>> >> It’s time to do a release with 1.19.0.
>>> >>
>>> >>> 2021年5月19日 上午2:20，Vitalii Diravka  写道：
>>> >>>
>>> >>> Hi Laurent,
>>> >>> DRILL-7871 requires additional time to be introduced and it is
>>> better to
>>> >>> include it for the next release.
>>> >>> DRILL-7904 is updated, I think it will be merged in a few days. But
>>> it
>>> >>> doesn't matter whether it is included in this release or in the next
>>> one.
>>> >>>
>>> >>> So we can plan to start the release process
>>> >>>
>>> >>>
>>> >>> Kind regards
>>> >>> Vitalii
>>> >>>
>>> >>>
>>> >>> On Tue, May 11, 2021 at 7:52 PM Laurent Goujon 
>>> wrote:
>>> >>>
>>> >>>> Thanks Vitalii
>>> >>>>
>>> >>>> On Tue, May 11, 2021 at 9:29 AM Vitalii Diravka >> >
>>> >>>> wrote:
>>> >>>>
>>> >>>>> Hi Luoc!
>>> >>>>>
>>> >>>>> They are almost ready. I plan to update PR for them today.
>>> >>>>>
>>> >>>>> Kind regards
>>> >>>>> Vitalii
>>> >>>>>
>>> >>>>>
>>> >>>>> On Sat, May 8, 2021 at 5:26 PM luoc  wrote:
>>> >>>>>
>>> >>>>>> Hi Vitalii,
>>> >>>>>> Would you mind sharing that... Is DRILL-7904 ready to review
>>> again?
>>> >>>>> And what’s
>>> >>>>>> the status on the DRILL-7871? thanks
>>> >>>>>>
>>> >>>>>> 2021年5月4日 下午1:10，Ted Dunning  写道：
>>> >>>>>>
>>> >>>>>> Laurent,
>>> >>>>>>
>>> >>>>>> I don't have a stake here, so can't really comment about
>>> specifics, but
>>> >>>>> the
>>> >>>>>> process is looking good.
>>> >>>>>>
>>> >>>>>>
>>> >>>>>>
>>> >>>>>> On Mon, May 3, 2021 at 9:23 PM Laurent Goujon >> >
>>> >>>>> wrote:
>>> >>>>>>
>>> >>>>>> Thanks for all the answers
>>> >>>>>>
>>> >>>>>> So the issues I found based on the feedback are:
>>> >>>>>>
>>> >>>>>> - DRILL-7878: Fix LGTM Alerts
>>> >>>>>> <https://issues.apache.org/jira/browse/DRILL-7878>
>>> >>>>>> - DRILL-7871: StoragePluginStore instances for different users
>>> >>>>>> 

Re: Release and GPG key

[Laurent Goujon]

Yes, I was thinking of doing a zoom meeting where I would show proof of id
+ key id. Especially because of Covid, that seems the easiest option.

On Mon, May 24, 2021, 16:08 Ted Dunning  wrote:

> Laurent,
>
> The critical question here is how you can substantiate this key. IN person,
> with a government ID, this would be easy.
>
> Do you know a committer personally who could vouch for you? Would you be
> interested in having a video call where you can present some ID?
>
> On Mon, May 24, 2021 at 3:24 PM Laurent Goujon  wrote:
>
> > Hi,
> >
> > I opened a pull request to add my public GPG keys to the KEYS file at the
> > root of the project:
> > https://github.com/apache/drill/pull/2234
> >
> > Sadly this key is not part of the Web Of Trust, and I would need someone
> > part of it to validate my key. And also a PMC member to add it to the
> Drill
> > release SVN repository.
> >
> > Anybody interested?
> >
> > Laurent
> >
>

Release and GPG key

[Laurent Goujon]

Hi,

I opened a pull request to add my public GPG keys to the KEYS file at the
root of the project:
https://github.com/apache/drill/pull/2234

Sadly this key is not part of the Web Of Trust, and I would need someone
part of it to validate my key. And also a PMC member to add it to the Drill
release SVN repository.

Anybody interested?

Laurent

Re: [DISCUSS] Drill 1.19.0 release

[Laurent Goujon]

I might be able to help on the JUnit and CVE patches too too

On Mon, May 24, 2021 at 2:29 PM Laurent Goujon  wrote:

> Ok, I was hoping that some of the PRs could be merged, but if we are in
> agreement, let's start the work :)
>
> On Sun, May 23, 2021 at 6:52 PM luoc  wrote:
>
>> Hi Charles,
>>   All right, we'll be expecting the update.
>>
>> > 2021年5月24日 上午12:13，Charles Givre  写道：
>> >
>> > Hi Luoc,
>> > We still have a few PRs pending that we really should get into Drill
>> 1.19.  The main one is the junit upgrade.  There are a few critical CVEs
>> associated with that, so I do think it is important to get that one
>> merged.  I think Vitalii will have that one done in short order.
>> > Best,
>> > -- C
>> >
>> >
>> >
>> >> On May 22, 2021, at 5:16 AM, luoc  wrote:
>> >>
>> >> Hi Laurent,
>> >> It’s time to do a release with 1.19.0.
>> >>
>> >>> 2021年5月19日 上午2:20，Vitalii Diravka  写道：
>> >>>
>> >>> Hi Laurent,
>> >>> DRILL-7871 requires additional time to be introduced and it is better
>> to
>> >>> include it for the next release.
>> >>> DRILL-7904 is updated, I think it will be merged in a few days. But it
>> >>> doesn't matter whether it is included in this release or in the next
>> one.
>> >>>
>> >>> So we can plan to start the release process
>> >>>
>> >>>
>> >>> Kind regards
>> >>> Vitalii
>> >>>
>> >>>
>> >>> On Tue, May 11, 2021 at 7:52 PM Laurent Goujon 
>> wrote:
>> >>>
>> >>>> Thanks Vitalii
>> >>>>
>> >>>> On Tue, May 11, 2021 at 9:29 AM Vitalii Diravka 
>> >>>> wrote:
>> >>>>
>> >>>>> Hi Luoc!
>> >>>>>
>> >>>>> They are almost ready. I plan to update PR for them today.
>> >>>>>
>> >>>>> Kind regards
>> >>>>> Vitalii
>> >>>>>
>> >>>>>
>> >>>>> On Sat, May 8, 2021 at 5:26 PM luoc  wrote:
>> >>>>>
>> >>>>>> Hi Vitalii,
>> >>>>>> Would you mind sharing that... Is DRILL-7904 ready to review again?
>> >>>>> And what’s
>> >>>>>> the status on the DRILL-7871? thanks
>> >>>>>>
>> >>>>>> 2021年5月4日 下午1:10，Ted Dunning  写道：
>> >>>>>>
>> >>>>>> Laurent,
>> >>>>>>
>> >>>>>> I don't have a stake here, so can't really comment about
>> specifics, but
>> >>>>> the
>> >>>>>> process is looking good.
>> >>>>>>
>> >>>>>>
>> >>>>>>
>> >>>>>> On Mon, May 3, 2021 at 9:23 PM Laurent Goujon 
>> >>>>> wrote:
>> >>>>>>
>> >>>>>> Thanks for all the answers
>> >>>>>>
>> >>>>>> So the issues I found based on the feedback are:
>> >>>>>>
>> >>>>>> - DRILL-7878: Fix LGTM Alerts
>> >>>>>> <https://issues.apache.org/jira/browse/DRILL-7878>
>> >>>>>> - DRILL-7871: StoragePluginStore instances for different users
>> >>>>>> <https://issues.apache.org/jira/browse/DRILL-7871>
>> >>>>>> - DRILL-7908: Fix GitHub Actions CI
>> >>>>>> <https://issues.apache.org/jira/browse/DRILL-7908>
>> >>>>>> - DRILL-7904: Update to 30-jre Guava version
>> >>>>>> <https://issues.apache.org/jira/browse/DRILL-7904>
>> >>>>>> - DRILL-7826: Merge Pcap and Pcapng format plugin based on EVF
>> >>>>>> <https://issues.apache.org/jira/browse/DRILL-7826>
>> >>>>>>   - DRILL-7828: Refactor Pcap and Pcapng format plugin
>> >>>>>>   <https://issues.apache.org/jira/browse/DRILL-7828>
>> >>>>>> - DRILL-7910: Bumps commons-io from 2.4 to 2.7
>> >>>>>> <https://issues.apache.org/jira/browse/DRILL-7910>
>> >>>>>> - DRILL-7901: Bump junit from 4.12 to 4.13.1
>> >>>>

Re: [DISCUSS] Drill 1.19.0 release

[Laurent Goujon]

Ok, I was hoping that some of the PRs could be merged, but if we are in
agreement, let's start the work :)

On Sun, May 23, 2021 at 6:52 PM luoc  wrote:

> Hi Charles,
>   All right, we'll be expecting the update.
>
> > 2021年5月24日 上午12:13，Charles Givre  写道：
> >
> > Hi Luoc,
> > We still have a few PRs pending that we really should get into Drill
> 1.19.  The main one is the junit upgrade.  There are a few critical CVEs
> associated with that, so I do think it is important to get that one
> merged.  I think Vitalii will have that one done in short order.
> > Best,
> > -- C
> >
> >
> >
> >> On May 22, 2021, at 5:16 AM, luoc  wrote:
> >>
> >> Hi Laurent,
> >> It’s time to do a release with 1.19.0.
> >>
> >>> 2021年5月19日 上午2:20，Vitalii Diravka  写道：
> >>>
> >>> Hi Laurent,
> >>> DRILL-7871 requires additional time to be introduced and it is better
> to
> >>> include it for the next release.
> >>> DRILL-7904 is updated, I think it will be merged in a few days. But it
> >>> doesn't matter whether it is included in this release or in the next
> one.
> >>>
> >>> So we can plan to start the release process
> >>>
> >>>
> >>> Kind regards
> >>> Vitalii
> >>>
> >>>
> >>> On Tue, May 11, 2021 at 7:52 PM Laurent Goujon 
> wrote:
> >>>
> >>>> Thanks Vitalii
> >>>>
> >>>> On Tue, May 11, 2021 at 9:29 AM Vitalii Diravka 
> >>>> wrote:
> >>>>
> >>>>> Hi Luoc!
> >>>>>
> >>>>> They are almost ready. I plan to update PR for them today.
> >>>>>
> >>>>> Kind regards
> >>>>> Vitalii
> >>>>>
> >>>>>
> >>>>> On Sat, May 8, 2021 at 5:26 PM luoc  wrote:
> >>>>>
> >>>>>> Hi Vitalii,
> >>>>>> Would you mind sharing that... Is DRILL-7904 ready to review again?
> >>>>> And what’s
> >>>>>> the status on the DRILL-7871? thanks
> >>>>>>
> >>>>>> 2021年5月4日 下午1:10，Ted Dunning  写道：
> >>>>>>
> >>>>>> Laurent,
> >>>>>>
> >>>>>> I don't have a stake here, so can't really comment about specifics,
> but
> >>>>> the
> >>>>>> process is looking good.
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> On Mon, May 3, 2021 at 9:23 PM Laurent Goujon 
> >>>>> wrote:
> >>>>>>
> >>>>>> Thanks for all the answers
> >>>>>>
> >>>>>> So the issues I found based on the feedback are:
> >>>>>>
> >>>>>> - DRILL-7878: Fix LGTM Alerts
> >>>>>> <https://issues.apache.org/jira/browse/DRILL-7878>
> >>>>>> - DRILL-7871: StoragePluginStore instances for different users
> >>>>>> <https://issues.apache.org/jira/browse/DRILL-7871>
> >>>>>> - DRILL-7908: Fix GitHub Actions CI
> >>>>>> <https://issues.apache.org/jira/browse/DRILL-7908>
> >>>>>> - DRILL-7904: Update to 30-jre Guava version
> >>>>>> <https://issues.apache.org/jira/browse/DRILL-7904>
> >>>>>> - DRILL-7826: Merge Pcap and Pcapng format plugin based on EVF
> >>>>>> <https://issues.apache.org/jira/browse/DRILL-7826>
> >>>>>>   - DRILL-7828: Refactor Pcap and Pcapng format plugin
> >>>>>>   <https://issues.apache.org/jira/browse/DRILL-7828>
> >>>>>> - DRILL-7910: Bumps commons-io from 2.4 to 2.7
> >>>>>> <https://issues.apache.org/jira/browse/DRILL-7910>
> >>>>>> - DRILL-7901: Bump junit from 4.12 to 4.13.1
> >>>>>> <https://issues.apache.org/jira/browse/DRILL-7901>
> >>>>>>
> >>>>>> I wanted to propose Monday May 10th to do the first release
> candidate,
> >>>>> but
> >>>>>> I have some concerns about some of the changes which may not be
> ready
> >>>> by
> >>>>>> then considering they seem to involve some level of effort and are
> in
> >>>>> very
> >>>

Re: [DISCUSS] Drill 1.19.0 release

[Laurent Goujon]

Thanks Vitalii

On Tue, May 11, 2021 at 9:29 AM Vitalii Diravka  wrote:

> Hi Luoc!
>
> They are almost ready. I plan to update PR for them today.
>
> Kind regards
> Vitalii
>
>
> On Sat, May 8, 2021 at 5:26 PM luoc  wrote:
>
> > Hi Vitalii,
> >   Would you mind sharing that... Is DRILL-7904 ready to review again?
> And what’s
> > the status on the DRILL-7871? thanks
> >
> > 2021年5月4日 下午1:10，Ted Dunning  写道：
> >
> > Laurent,
> >
> > I don't have a stake here, so can't really comment about specifics, but
> the
> > process is looking good.
> >
> >
> >
> > On Mon, May 3, 2021 at 9:23 PM Laurent Goujon 
> wrote:
> >
> > Thanks for all the answers
> >
> > So the issues I found based on the feedback are:
> >
> >   - DRILL-7878: Fix LGTM Alerts
> >   <https://issues.apache.org/jira/browse/DRILL-7878>
> >   - DRILL-7871: StoragePluginStore instances for different users
> >   <https://issues.apache.org/jira/browse/DRILL-7871>
> >   - DRILL-7908: Fix GitHub Actions CI
> >   <https://issues.apache.org/jira/browse/DRILL-7908>
> >   - DRILL-7904: Update to 30-jre Guava version
> >   <https://issues.apache.org/jira/browse/DRILL-7904>
> >   - DRILL-7826: Merge Pcap and Pcapng format plugin based on EVF
> >   <https://issues.apache.org/jira/browse/DRILL-7826>
> >  - DRILL-7828: Refactor Pcap and Pcapng format plugin
> >  <https://issues.apache.org/jira/browse/DRILL-7828>
> >   - DRILL-7910: Bumps commons-io from 2.4 to 2.7
> >   <https://issues.apache.org/jira/browse/DRILL-7910>
> >   - DRILL-7901: Bump junit from 4.12 to 4.13.1
> >   <https://issues.apache.org/jira/browse/DRILL-7901>
> >
> > I wanted to propose Monday May 10th to do the first release candidate,
> but
> > I have some concerns about some of the changes which may not be ready by
> > then considering they seem to involve some level of effort and are in
> very
> > early stage: The LGTM alert changes and the StoragePluginStore model
> > change. JUnit version update might also become quite a large change if
> > instead of moving to 4.13.1, Drill is switching to JUnit5.
> >
> > What do people think?
> >
> > On Sat, Apr 24, 2021 at 1:00 PM Vitalii Diravka 
> > wrote:
> >
> > Hi Laurent,
> >
> > I want to include:
> > DRILL-7871 <https://issues.apache.org/jira/browse/DRILL-7871> (preparing
> > PR)
> > DRILL-7908 <https://issues.apache.org/jira/browse/DRILL-7908> (preparing
> > PR)
> > DRILL-7904 <https://issues.apache.org/jira/browse/DRILL-7904> (PR is
> > opened, in review)
> > DRILL-7828 <https://issues.apache.org/jira/browse/DRILL-7828> (PR is
> > opened, review is almost completed)
> >
> > All these tasks are expected to be completed in a week
> >
> > Kind regards
> > Vitalii
> >
> >
> > On Fri, Apr 23, 2021 at 9:25 PM Charles Givre  wrote:
> >
> > Hi Laurent,
> > We have a few PRs pending which I'd like to see in the next version
> >
> > which
> >
> > are:
> > 1.  The update(s) and bug fixes to the Mongo plugin.
> > 2.  There is an extended PR for bug fixes which clean up a lot of
> >
> > alerts
> >
> > generated by LGTM
> > 3.  There are a few other library updates which are pending.
> > 4.  We have some work which changes the access model around storage
> > plugins which would be good for this release
> > 5.  The PCAP/PCAP-NG consolidation is awaiting review.
> >
> > I think that's it.
> > -- C
> >
> > On Apr 22, 2021, at 12:33 PM, Laurent Goujon 
> >
> > wrote:
> >
> >
> > Hello everyone,
> >
> > It has been more than 6 months since the last release, and I believe
> >
> > this
> >
> > would be a good time to discuss the next one.
> >
> > As mentioned in a previous email thread, I am volunteering to be the
> > release manager, and I'm looking forward  working with the whole
> >
> > community
> >
> > to make another great release.
> >
> > We have around 80 changes in master since the last release, and there
> >
> > are
> >
> > several changes open for review too. It would be nice if people could
> >
> > reply
> >
> > to this email and share issues which should be part of that release,
> >
> > so
> >
> > we
> >
> > can decide on an initial cut-off date.
> >
> > Thanks in advance,
> >
> > Laurent
> >
> >
> >
> >
> >
> >
> >
>

Re: [DISCUSS] Drill 1.19.0 release

[Laurent Goujon]

Thanks for all the answers

So the issues I found based on the feedback are:

   - DRILL-7878: Fix LGTM Alerts
   <https://issues.apache.org/jira/browse/DRILL-7878>
   - DRILL-7871: StoragePluginStore instances for different users
   <https://issues.apache.org/jira/browse/DRILL-7871>
   - DRILL-7908: Fix GitHub Actions CI
   <https://issues.apache.org/jira/browse/DRILL-7908>
   - DRILL-7904: Update to 30-jre Guava version
   <https://issues.apache.org/jira/browse/DRILL-7904>
   - DRILL-7826: Merge Pcap and Pcapng format plugin based on EVF
   <https://issues.apache.org/jira/browse/DRILL-7826>
  - DRILL-7828: Refactor Pcap and Pcapng format plugin
  <https://issues.apache.org/jira/browse/DRILL-7828>
   - DRILL-7910: Bumps commons-io from 2.4 to 2.7
   <https://issues.apache.org/jira/browse/DRILL-7910>
   - DRILL-7901: Bump junit from 4.12 to 4.13.1
   <https://issues.apache.org/jira/browse/DRILL-7901>

I wanted to propose Monday May 10th to do the first release candidate, but
I have some concerns about some of the changes which may not be ready by
then considering they seem to involve some level of effort and are in very
early stage: The LGTM alert changes and the StoragePluginStore model
change. JUnit version update might also become quite a large change if
instead of moving to 4.13.1, Drill is switching to JUnit5.

What do people think?

On Sat, Apr 24, 2021 at 1:00 PM Vitalii Diravka  wrote:

> Hi Laurent,
>
> I want to include:
> DRILL-7871 <https://issues.apache.org/jira/browse/DRILL-7871> (preparing
> PR)
> DRILL-7908 <https://issues.apache.org/jira/browse/DRILL-7908> (preparing
> PR)
> DRILL-7904 <https://issues.apache.org/jira/browse/DRILL-7904> (PR is
> opened, in review)
> DRILL-7828 <https://issues.apache.org/jira/browse/DRILL-7828> (PR is
> opened, review is almost completed)
>
> All these tasks are expected to be completed in a week
>
> Kind regards
> Vitalii
>
>
> On Fri, Apr 23, 2021 at 9:25 PM Charles Givre  wrote:
>
> > Hi Laurent,
> > We have a few PRs pending which I'd like to see in the next version which
> > are:
> > 1.  The update(s) and bug fixes to the Mongo plugin.
> > 2.  There is an extended PR for bug fixes which clean up a lot of alerts
> > generated by LGTM
> > 3.  There are a few other library updates which are pending.
> > 4.  We have some work which changes the access model around storage
> > plugins which would be good for this release
> > 5.  The PCAP/PCAP-NG consolidation is awaiting review.
> >
> > I think that's it.
> > -- C
> >
> > > On Apr 22, 2021, at 12:33 PM, Laurent Goujon 
> wrote:
> > >
> > > Hello everyone,
> > >
> > > It has been more than 6 months since the last release, and I believe
> this
> > > would be a good time to discuss the next one.
> > >
> > > As mentioned in a previous email thread, I am volunteering to be the
> > > release manager, and I'm looking forward  working with the whole
> > community
> > > to make another great release.
> > >
> > > We have around 80 changes in master since the last release, and there
> are
> > > several changes open for review too. It would be nice if people could
> > reply
> > > to this email and share issues which should be part of that release, so
> > we
> > > can decide on an initial cut-off date.
> > >
> > > Thanks in advance,
> > >
> > > Laurent
> >
> >
>

[jira] [Created] (DRILL-7914) Apache Drill 1.19.0 Release Activities

[Laurent Goujon (Jira)]

Laurent Goujon created DRILL-7914:
-

 Summary: Apache Drill 1.19.0 Release Activities
 Key: DRILL-7914
 URL: https://issues.apache.org/jira/browse/DRILL-7914
 Project: Apache Drill
  Issue Type: Task
Reporter: Laurent Goujon
Assignee: Laurent Goujon


Source - https://github.com/apache/drill/blob/master/docs/dev/Release.md

1.19.0 Dashboard: 
https://issues.apache.org/jira/secure/Dashboard.jspa?selectPageId=12336127

Kanban Board - 
https://issues.apache.org/jira/secure/RapidBoard.jspa?rapidView=185




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[DISCUSS] Drill 1.19.0 release

[Laurent Goujon]

Hello everyone,

It has been more than 6 months since the last release, and I believe this
would be a good time to discuss the next one.

As mentioned in a previous email thread, I am volunteering to be the
release manager, and I'm looking forward  working with the whole community
to make another great release.

We have around 80 changes in master since the last release, and there are
several changes open for review too. It would be nice if people could reply
to this email and share issues which should be part of that release, so we
can decide on an initial cut-off date.

Thanks in advance,

Laurent

Re: Requesting a release

[Laurent Goujon]

Perfect!

On Mon, Apr 12, 2021 at 9:40 PM luoc  wrote:

> Hi,
>   Drill is a community-driven project, so we welcome the contributions for
> ever (in any way). there is a document about the release <
> https://github.com/apache/drill/blob/master/docs/dev/Release.md>.
>
> > 2021年4月13日 上午7:17，Ted Dunning  写道：
> >
> >
> > Laurent,
> >
> > There are definitely some steps that require privileges, but the
> management of the process doesn't require privileges.
> >
> > The key steps are a) developing consensus on the content of the release,
> b) building a release candidate and c) conducting the vote. After these,
> pushing the artifacts may require some mojo, but it is easy to get somebody
> to help. None of these other steps require more than a commit bit.
> >
> > Remember that the core point of a release is the community involvement,
> not the technical aspects of packaging.
> >
> > On 2021/04/12 21:45:22, Laurent Goujon  wrote:
> >> Hi Ted,
> >>
> >> I was led to believe that only a PMC member could perform some of the
> >> release tasks, but if not the case, I'm happy to volunteer for the next
> >> one. Since it would be my first release, is there any document detailing
> >> the list of tasks to be completed?
> >>
> >> On Mon, Apr 12, 2021 at 1:55 PM Ted Dunning 
> wrote:
> >>
> >>> Hey Ray,
> >>>
> >>> Any Drill committer should be able to act as a release manager.
> >>>
> >>> My guess is that you know several Drill committers at Dremio who might
> be
> >>> able to help with this.
> >>>
> >>>
> >>>
> >>> On Mon, Apr 12, 2021 at 12:00 PM Ray Lum  wrote:
> >>>
> >>>> Hi Drill community,
> >>>>
> >>>> Is there a process for requesting a release of the latest code
> currently
> >>> in
> >>>> master? I am keen on adopting some of the changes if they were in an
> >>>> official release.
> >>>>
> >>>> Thanks kindly,
> >>>> Ray
> >>>>
> >>>
> >>
>
>

Re: Requesting a release

[Laurent Goujon]

It's true what our contribution to Drill's dwelled a bit unfortunately as
we decided to take a different approach on some core aspects, but we are
still using the same User protocol for JDBC/ODBC connectivity, and we are
still contributing those changes to the Apache Drill project hopefully to
the benefit of both projects.

Laurent

On Mon, Apr 12, 2021 at 3:26 PM Charles Givre  wrote:

> Laurent,
> Thanks for volunteering.  There are a few PRs and work planned in the next
> month or so for the next release.  Additionally, there are some tests which
> must be run outside of the regular unit test framework that require a
> special setup.
>
> I have to ask as I am curious but what/why is Dremio still interested in
> Drill?  All the code is OSS, and you could just use what's already in
> github without it being "released".
> Best,
> - C
>
>
>
> > On Apr 12, 2021, at 5:45 PM, Laurent Goujon  wrote:
> >
> > Hi Ted,
> >
> > I was led to believe that only a PMC member could perform some of the
> > release tasks, but if not the case, I'm happy to volunteer for the next
> > one. Since it would be my first release, is there any document detailing
> > the list of tasks to be completed?
> >
> > On Mon, Apr 12, 2021 at 1:55 PM Ted Dunning 
> wrote:
> >
> >> Hey Ray,
> >>
> >> Any Drill committer should be able to act as a release manager.
> >>
> >> My guess is that you know several Drill committers at Dremio who might
> be
> >> able to help with this.
> >>
> >>
> >>
> >> On Mon, Apr 12, 2021 at 12:00 PM Ray Lum  wrote:
> >>
> >>> Hi Drill community,
> >>>
> >>> Is there a process for requesting a release of the latest code
> currently
> >> in
> >>> master? I am keen on adopting some of the changes if they were in an
> >>> official release.
> >>>
> >>> Thanks kindly,
> >>> Ray
> >>>
> >>
>
>

Re: Requesting a release

[Laurent Goujon]

Hi Ted,

I was led to believe that only a PMC member could perform some of the
release tasks, but if not the case, I'm happy to volunteer for the next
one. Since it would be my first release, is there any document detailing
the list of tasks to be completed?

On Mon, Apr 12, 2021 at 1:55 PM Ted Dunning  wrote:

> Hey Ray,
>
> Any Drill committer should be able to act as a release manager.
>
> My guess is that you know several Drill committers at Dremio who might be
> able to help with this.
>
>
>
> On Mon, Apr 12, 2021 at 12:00 PM Ray Lum  wrote:
>
> > Hi Drill community,
> >
> > Is there a process for requesting a release of the latest code currently
> in
> > master? I am keen on adopting some of the changes if they were in an
> > official release.
> >
> > Thanks kindly,
> > Ray
> >
>

[jira] [Resolved] (DRILL-7726) Boost requirement is incorrect

[Laurent Goujon (Jira)]

 [ 
https://issues.apache.org/jira/browse/DRILL-7726?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Laurent Goujon resolved DRILL-7726.
---
Fix Version/s: 1.18.0
   Resolution: Fixed

> Boost requirement is incorrect
> --
>
> Key: DRILL-7726
> URL: https://issues.apache.org/jira/browse/DRILL-7726
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Client - C++
>        Reporter: Laurent Goujon
>    Assignee: Laurent Goujon
>Priority: Major
> Fix For: 1.18.0
>
>
> Drill C++ connector documentation states that Boost 1.53 is required, but as 
> support for tlsv12 is present, Boost 1.54 is actually required. Trying to 
> compile with Boost 1.53 actually result in a compilation error for undefined 
> constant



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (DRILL-7727) Address Protobuf C++ warnings

[Laurent Goujon (Jira)]

Laurent Goujon created DRILL-7727:
-

 Summary: Address Protobuf C++ warnings
 Key: DRILL-7727
 URL: https://issues.apache.org/jira/browse/DRILL-7727
 Project: Apache Drill
  Issue Type: Test
  Components: Client - C++
Reporter: Laurent Goujon
Assignee: Laurent Goujon


Compiling C++ client with recent versions of protobuf (3.4.0 or higher) 
produces compilation warnings:
{noformat}
.../drill/contrib/native/client/src/clientlib/rpcMessage.cpp:208:32: warning: 
'ByteSize' is deprecated: Please use ByteSizeLong() instead 
[-Wdeprecated-declarations]
int header_length = header.ByteSize();
   ^
/usr/local/include/google/protobuf/message_lite.h:401:3: note: 'ByteSize' has 
been explicitly marked deprecated here
  PROTOBUF_DEPRECATED_MSG("Please use ByteSizeLong() instead")
{noformat}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (DRILL-7726) Boost requirement is incorrect

[Laurent Goujon (Jira)]

Laurent Goujon created DRILL-7726:
-

 Summary: Boost requirement is incorrect
 Key: DRILL-7726
 URL: https://issues.apache.org/jira/browse/DRILL-7726
 Project: Apache Drill
  Issue Type: Bug
  Components: Client - C++
Reporter: Laurent Goujon
Assignee: Laurent Goujon


Drill C++ connector documentation states that Boost 1.53 is required, but as 
support for tlsv12 is present, Boost 1.54 is actually required. Trying to 
compile with Boost 1.53 actually result in a compilation error for undefined 
constant



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: Drill SASL Forward Compatibility

[Laurent Goujon]

If encryption is not enabled, any MITM can intercept and inject traffic in
the session (meaning it could also do requests on behalf of the users
without the user noticing), even with Kerberos.

On Wed, Nov 1, 2017 at 2:13 PM, Sorabh Hamirwasia 
wrote:

> Parth,
>
> Your understanding is correct.
>
>
> I am also debating for 1(A) specially in context of security mechanisms
> like Kerberos which guarantees prevention from MITM during handshake.
>
>
> But with 1( B ) we are saying in case of Drill it's possible and fine
> since no data is compromised.
>
>
> Thanks,
> Sorabh
>
> 
> From: Parth Chandra 
> Sent: Wednesday, November 1, 2017 1:42:14 PM
> To: dev
> Subject: Re: Drill SASL Forward Compatibility
>
> I sort of lost track of the arguments in the thread. Is my understanding
> below correct ?
>
>
> 1) A handshake from a (1.12) client expecting authentication and encryption
> is intercepted by a rogue server. The server then responds with a success
> message and bypasses the auth and encryption for the session.
>
> 2) The client is now connected, but not to the server it wanted to connect
> to.
>
> 3) The rogue server can now feed any bogus response to the client.
>
>
> Question 1 - Is #3 a security issue?
>
>
> Answer 1 (A) - Yes. The handshake has been compromised. The client is no
> longer connected to an authentic server.
>
>
> Answer 1 (B) - No. There is no data that has been compromised. Just a
> client that has been misled.
>
>
>
> I believe this is a security issue. A rogue server can now feed invalid
> results to the client and that is not safe. Perhaps others with more
> experience on industrial grade security can chime in.
>
> Question 2 - If this is a security issue, is it severe enough to break
> forward compatibility?
>
> In general, I'm -1 on breaking backward compatibility and -0 on breaking
> forward compatibility. I believe it is a very desirable goal to maintain
> both backward and forward compatibility. However, forward compatibility
> cannot be guaranteed unless we bake it into the RPC protocol and design
> clients to be version and feature aware. This itself would be a breaking
> change and should be one of the goals for V2.
>
> In this case, I'm inclined to go with what Arina is suggesting.
>

Re: Drill SASL Forward Compatibility

[Laurent Goujon]

I have a parallel scenario:

- Scenario 1:
1) A handshake from a (1.12) client expecting authentication and encryption
is intercepted by a rogue server. The rogue server then responds first with
AUTH_REQUIRED, but authenticationMechanisms doesn't provide gssapi/kerberos
as a sasl mechanism. The client accepts another authentication mechanism,
and authenticate with the rogue server, the server send a SASL_SUCCESS
message and bypasses the auth and encryption for the session.

2) The client is now connected, but not to the server it wanted to connect
to.

3) The rogue server can now feed any bogus response to the client.

Question:
- is it the same situation as the scenario proposed by Parth where the
rogue server was directly sending SUCCESS?
I personally consider it is the same, and that is a potential security
issue, the same way phishing is a security issue, and can be used to misled
users

- is it possible for both these scenario to happen if client actually sets
DrillProperties.SASL_ENCRYPT to true, meaning it would only allow to
complete handshake if encryption is actually not set up properly?
Answer for this question is likely to be no since both C++ and Java clients
check if encryption has been set up, and will fail if not
*
https://github.com/apache/drill/blob/b0c4e0486d6d4620b04a1bb8198e959d433b4840/exec/java-exec/src/main/java/org/apache/drill/exec/rpc/user/UserClient.java#L238
*
https://github.com/apache/drill/blob/b0c4e0486d6d4620b04a1bb8198e959d433b4840/contrib/native/client/src/clientlib/drillClientImpl.cpp#L614

My suggestions would be:
- client should be able to select which SASL mechanisms to allow or not.
Client could choose mechanisms which only allow for mutual authentication.
If client requires encryption, client could only allow for mechanisms
allowing encryption. C++ client seems to support any mechanism registered
in SASL library. Java client has PLAIN and KERBEROS always enabled, and can
add other custom mechanims, but it is not possible to disable PLAIN!

- modify the check introduced with DRILL-5582 to check if client wants to
use PLAIN authentication. If client wants so, allow to connect it to older
servers. If not, disable PLAIN mechanism in SASL, and yes, make sure you go
through a SASL authentication.

Laurent

On Wed, Nov 1, 2017 at 1:42 PM, Parth Chandra  wrote:

> I sort of lost track of the arguments in the thread. Is my understanding
> below correct ?
>
>
> 1) A handshake from a (1.12) client expecting authentication and encryption
> is intercepted by a rogue server. The server then responds with a success
> message and bypasses the auth and encryption for the session.
>
> 2) The client is now connected, but not to the server it wanted to connect
> to.
>
> 3) The rogue server can now feed any bogus response to the client.
>
>
> Question 1 - Is #3 a security issue?
>
>
> Answer 1 (A) - Yes. The handshake has been compromised. The client is no
> longer connected to an authentic server.
>
>
> Answer 1 (B) - No. There is no data that has been compromised. Just a
> client that has been misled.
>
>
>
> I believe this is a security issue. A rogue server can now feed invalid
> results to the client and that is not safe. Perhaps others with more
> experience on industrial grade security can chime in.
>
> Question 2 - If this is a security issue, is it severe enough to break
> forward compatibility?
>
> In general, I'm -1 on breaking backward compatibility and -0 on breaking
> forward compatibility. I believe it is a very desirable goal to maintain
> both backward and forward compatibility. However, forward compatibility
> cannot be guaranteed unless we bake it into the RPC protocol and design
> clients to be version and feature aware. This itself would be a breaking
> change and should be one of the goals for V2.
>
> In this case, I'm inclined to go with what Arina is suggesting.
>

Re: Drill SASL Forward Compatibility

[Laurent Goujon]

My comments inline:

On Tue, Oct 31, 2017 at 6:11 PM, Sorabh Hamirwasia 
wrote:

> - if using Kerberos, things are a bit different: even if a MITM intercepts
>
> the token, only the server can decode it properly and set up encryption so
> that only client can use it (a proxy server would not be able to decode the
> traffic). So what you need to ensure is that you actually use Kerberos as
> the only authentication mechanism, and that the channel is encrypted (if
> channel is not encryted, see above). This is things you should do by
> configuring the client by not sending the password (no need to), only
> authorize Kerberos authentication, and verify that encryption is enabled
> (which is already done I believe).
>
>
> [Sorabh] - You are correct about the Kerberos preventing MITM which is
> what I mentioned in the last response. But this is guaranteed if client and
> server reach to the point of SASL handshake in their communication, since
> SASL handshake exchanges all the bits related to Kerberos protocol. Before
> that point is reached there are still few messages which is exchanged
> between DrillClient and Drillbit to detect whether server side needs
> authentication or not and what are supported mechanisms. This is where the
> security flaw can cause client to believe Authentication is successfully
> completed (even when Drillbit/DrillClient are authenticating using Kerberos
> with or without encryption). This is what patch for DRILL-5582 is trying to
> address.
>
>
You still haven't answered what is the issue/security risk for the client
here: sure the client didn't authenticate with the server, but at the same
time it didn't get access to the server either...

Also, it doesn't take very long to modify the rogue server to fake a SASL
authentication. So, now you are "authenticated", but still not to the right
server...



>
> As for the other issue at hand (compatibility between 1.11 client and 1.10
> server), I am not sure to understand the proposed fix: is the logic you
> proposed to be added to 1.10 server? why not simply add the missing enum
> value (and that's it! no more values after that!)?
>
>
> [Sorabh] - Just adding the missing enum value in 1.10 will not help since
> in future if any other enum value is introduced then the same issue will be
> seen. Moreover 1.10 doesn't support Encryption so that enum value should
> not be added in that release. Instead the fix is to treat the return value
> of UNKNOWN_SASL_SERVER while retrieving messages from future client as
> valid default value and take decision based on that.
>

But 1.10.1 could consider that a client supporting encryption also support
authentication? The code is already here in 1.10 btw:
https://github.com/apache/drill/blob/1.10.0/exec/java-exec/src/main/java/org/apache/drill/exec/rpc/user/UserServer.java#L297

Alternatively, you could just check that the field sasl_support is set and
not check the value alltogether. I'm not convinced you need to do some
extra logic around UNKNOWN_SASL_SERVER which would just keep people
confused (although it doesn't seem something you need to apply to 1.11 or
higher)



>
>
> Thanks,
> Sorabh
>
> 
> From: Laurent Goujon 
> Sent: Tuesday, October 31, 2017 5:42 PM
> To: dev
> Cc: Arina Lelchieva; sudhe...@apache.org
> Subject: Re: Drill SASL Forward Compatibility
>
> Regarding DRILL-5582 patch which broke compatibility with 1.9 version
> (which is less than a year old):
> I'm still not clear what we are trying to prevent here: stealing user
> credentials and/or data? connecting to a rogue server which could return
> corrupted data to the user? The patch gives a false sense of security
> because it prevents none of it:
> - if using password-based authentication, client is sending it in clear in
> the initial handshake so I guess it's already game over!
> - You could configure a client to not sent password over wire by choosing
> another authentication algorithm, BUT if there's no encryption of the
> channel, any data can be intercepted and rewritten. And of course, the
> rogue server could actually run queries on behalf of the user...
> - if using Kerberos, things are a bit different: even if a MITM intercepts
> the token, only the server can decode it properly and set up encryption so
> that only client can use it (a proxy server would not be able to decode the
> traffic). So what you need to ensure is that you actually use Kerberos as
> the only authentication mechanism, and that the channel is encrypted (if
> channel is not encryted, see above). This is things you should do by
> configuring the client by not sending the password (no need to), only
> authorize Kerberos authentication, and veri

Re: Drill SASL Forward Compatibility

[Laurent Goujon]

Regarding DRILL-5582 patch which broke compatibility with 1.9 version
(which is less than a year old):
I'm still not clear what we are trying to prevent here: stealing user
credentials and/or data? connecting to a rogue server which could return
corrupted data to the user? The patch gives a false sense of security
because it prevents none of it:
- if using password-based authentication, client is sending it in clear in
the initial handshake so I guess it's already game over!
- You could configure a client to not sent password over wire by choosing
another authentication algorithm, BUT if there's no encryption of the
channel, any data can be intercepted and rewritten. And of course, the
rogue server could actually run queries on behalf of the user...
- if using Kerberos, things are a bit different: even if a MITM intercepts
the token, only the server can decode it properly and set up encryption so
that only client can use it (a proxy server would not be able to decode the
traffic). So what you need to ensure is that you actually use Kerberos as
the only authentication mechanism, and that the channel is encrypted (if
channel is not encryted, see above). This is things you should do by
configuring the client by not sending the password (no need to), only
authorize Kerberos authentication, and verify that encryption is enabled
(which is already done I believe).

For comparison, HTTP protocol (using it since it is one of the most used
public protocol) has no issue with client sending an authentication header
to a remote server, without knowing based on the server response if
authentication happened.

As for the other issue at hand (compatibility between 1.11 client and 1.10
server), I am not sure to understand the proposed fix: is the logic you
proposed to be added to 1.10 server? why not simply add the missing enum
value (and that's it! no more values after that!)?

Laurent

On Tue, Oct 31, 2017 at 3:12 PM, Sorabh Hamirwasia 
wrote:

> Hi Laurent,
>
> We are preventing 2 cases here:
>
>   *   False positive for successful authentication. Even though MITM
> attack can happen after successful authentication, but client and server
> involved here has ensure successful authentication handshake has taken
> place. With the security flaw there can always be false positive on client
> side thinking authentication was successful even though that might not be
> the case.
>   *   False positive for successful handshake with encryption capability:
> In cases when server is properly configured to support encryption, MITM
> attack tweaking handshake response and making client to believe that after
> successful handshake all communications will be encrypted is again another
> bad false positive.
>
> IMHO Drill Client should be able to validate when server says that
> authentication is completed then it's actually completed, at least until
> the point SASL Handshake is initiated, rather than blindly trusting it.
> This is because if Drill client doesn't guarantees that, then it's making
> the support for protocol like Kerberos weaker which prevent's from any MITM
> attack at handshake level. Whereas mechanisms like PLAIN are still prone to
> MITM even during SASL handshake.
>
> As far as forward compatibility is concerned there are few things:
>
>   *   AFAIK DrillClient & Drillbit doesn't have any concept of supporting
> different RPC versions across releases.They are forced to talk on same RPC
> versions else connection will fail. Once we have that I think then we will
> be able to clearly justify or provide the matrix of backward and forward
> compatibility across releases.
>   *   We can put the check based on supported_methods to detect if server
> side supports SASL or not. But this is again just a work around not proper
> solution. With workaround there can still be similar security holes as
> PLAIN mechanism itself is prone to MITM. Given 1.9 is 3 releases behind
> now, not sure if we still want to support that combination.
>   *   At least for compatibility between Drill 1.11 client and Drill 1.10
> server, I think the fix should be made which is mentioned in first email of
> this thread.
>
> Thanks,
> Sorabh
>
> 
> From: Laurent Goujon 
> Sent: Tuesday, October 31, 2017 9:38:13 AM
> To: dev
> Cc: Arina Lelchieva; sudhe...@apache.org
> Subject: Re: Drill SASL Forward Compatibility
>
> See my answers inline.
>
> On Tue, Oct 31, 2017 at 1:40 AM, Sorabh Hamirwasia 
> wrote:
>
> > Hi Laurent,
> >
> > Thanks for pointing issue with <= 1.9 version Drillbit, I looked into
> > supported_methods field and it doesn't advertise SASL support using that
> > [1][2]. Do you mean that checking if supported_methods list is non-empty
> >

Re: Drill SASL Forward Compatibility

[Laurent Goujon]

See my answers inline.

On Tue, Oct 31, 2017 at 1:40 AM, Sorabh Hamirwasia 
wrote:

> Hi Laurent,
>
> Thanks for pointing issue with <= 1.9 version Drillbit, I looked into
> supported_methods field and it doesn't advertise SASL support using that
> [1][2]. Do you mean that checking if supported_methods list is non-empty
> should suffice since it was introduced in 1.10 ?
>

My bad, I thought SASL_MESSAGE was added to the list of supported methods,
but it's not. Alternatively you could check for authenticationMechanisms
which should be not empty if version >= 1.10 and authentication is turned
on.


>
>
> For security flaw let's consider an example. Let say client is connecting
> to a Drillbit with authentication (and or encryption) enabled for Kerberos
> mechanism. The message flow will happen something like below:
>
>
> Good Case:
>
>   *   DrillClient sends Handshake Request to Drillbit
>   *   Drillbit sends the response back to DrillClient with AUTH_REQD as
> status
>   *   DrillClient exchange SASL handshake messages with Drillbit.
>   *   Once handshake is successful DrillClient is connected to secure
> Drillbit.
>   *   App using DrillClient has actually established a connection to
> secure Drillbit with authentication (and or encryption) and can submit it's
> query.
>
> Bad Case:
>
>   *   Step 1 as above
>   *   Step 2 as above. This message was intercepted by MITM and status was
> changed to SUCCESS.
>   *   Without the recent change DrillClient will not initiate SASL
> handshake and will return connection successful.
>   *   App using DrillClient will think it has successfully connected to
> secure Drillbit which is NOT the case.
>

What are we preventing in the bad case? if this is credentials or data
interception, a MITM could simply act as proxy to intercept all of it since
traffic is not encrypted. If we were to prevent the loss of credentials,
the solution to avoid transmitting the credentials in clear in the first
place. For that, we don't need a protocol change but:
- disable plain authentication, and use something like Kerberos or
CRAM-MD5/SCRAM
- make sure the password is not sent in the initial handshake (if using
Kerberos, there should no credentials to send over in the first place)


>
> I think what you are pointing out is even in good case and in
> authentication only scenario, even if connection is successful, the
> messages between DrillClient and Drillbit can still be intercepted since
> they will be in plain text. The only way to avoid that is using encryption.
> But the fix was more of to avoid wrong behavior in that case too where
> connection should fail, instead of client just relying on server response.
>

The "wrong" behavior was what allowed for compatibility with older servers
in the original design...


>
> [1]: https://github.com/apache/drill/blob/1.11.0/exec/java-
> exec/src/main/java/org/apache/drill/exec/rpc/user/UserServer.java#L254
> [2]: https://github.com/apache/drill/blob/1.11.0/exec/java-
> exec/src/main/java/org/apache/drill/exec/rpc/user/UserRpcConfig.java#L89
>
>
> Thanks,
> Sorabh
>
> 
> From: Laurent Goujon 
> Sent: Monday, October 30, 2017 5:47 PM
> To: dev
> Cc: Arina Lelchieva; sudhe...@apache.org
> Subject: Re: Drill SASL Forward Compatibility
>
> Regarding DRILL-5582, I see that fix as a breakage of the work to maintain
> compatibility for an newer client to connect to a older version of the
> server. Or put it differently: current (master) client does not connect
> anymore to a server not supporting SASL (<=1.9). Note that the client could
> detect if the server supports SASL as it is advertised in the
> supported_methods field of the BitToUserHandshake, and it would restore
> compatibility, but it seems the fix was done in response to a potential
> security flaw (although I have to admin not sure what issue it does prevent
> since it is still possible for a MITM to intercept all traffic between a
> client and a server).
>
> Laurent
>
> On Mon, Oct 30, 2017 at 5:18 PM, Sorabh Hamirwasia 
> wrote:
>
> > Hi All,
> >
> > We recently added a check (as part of DRILL-5582<https://issues.
> > apache.org/jira/browse/DRILL-5582>) on DrillClient side to enforce that
> > if client showed intent for authentication and Drillbit say's it doesn't
> > require authentication then connection will fail with proper error
> message.
> >
> >
> > With this change we found a hidden issue w.r.t forward compatibility of
> > Drill. New clients running on 1.11+ authenticating to older Drillbit
> > running on 1.10 are treated as client running without any SASL support or
> > (<=1.9 ve

Re: Drill SASL Forward Compatibility

[Laurent Goujon]

Regarding DRILL-5582, I see that fix as a breakage of the work to maintain
compatibility for an newer client to connect to a older version of the
server. Or put it differently: current (master) client does not connect
anymore to a server not supporting SASL (<=1.9). Note that the client could
detect if the server supports SASL as it is advertised in the
supported_methods field of the BitToUserHandshake, and it would restore
compatibility, but it seems the fix was done in response to a potential
security flaw (although I have to admin not sure what issue it does prevent
since it is still possible for a MITM to intercept all traffic between a
client and a server).

Laurent

On Mon, Oct 30, 2017 at 5:18 PM, Sorabh Hamirwasia 
wrote:

> Hi All,
>
> We recently added a check (as part of DRILL-5582 apache.org/jira/browse/DRILL-5582>) on DrillClient side to enforce that
> if client showed intent for authentication and Drillbit say's it doesn't
> require authentication then connection will fail with proper error message.
>
>
> With this change we found a hidden issue w.r.t forward compatibility of
> Drill. New clients running on 1.11+ authenticating to older Drillbit
> running on 1.10 are treated as client running without any SASL support or
> (<=1.9 version). The root cause for this issue is as follows:
>
>
> In 1.10 a new field SASL_SUPPORT was introduced in Handshake message
> between DrillCilent and Drillbit. The supported values for that field are:
>
>
> Drill 1.10: [1]
>
>
> enum SASL_SUPPORT {
> UNKNOWN_SASL_SUPPORT = 0;
> SASL_AUTH = 1;
> }
>
>
> Drill 1.11/1.12: [2]
>
>
> enum SASL_SUPPORT {
> UNKNOWN_SASL_SUPPORT = 0;
> SASL_AUTH = 1;
> SASL_PRIVACY = 2;
> }
>
> A 1.11 client always has SASL_PRIVACY set in handshake. A 1.10 Drillbit
> getting the message interprets the value as UNKNOWN_SASL_SUPPORT [3]. In
> 1.10 Drillbit treats that value as an indication of older client < 1.10
> [4], and it will try to authenticate using the 1.9 mechanism and send the
> SUCCESS/FAILURE state in Handshake Response. The 1.12 DrillClient will fail
> the connection as it will expect AUTH_REQUIRED from Drillbit instead of
> SUCCESS/FAILURE.
>
>
> The fix for this issue can be to use only absence of field as indication
> of client < 1.10 and if the field is present but it evaluates to
> UNKNOWN_SASL_SUPPORT value then Drillbit should consider corresponding
> client to be of future version at least for authentication purpose and
> speak SASL protocol.
>
> We have to either back port above forward compatibility fix into 1.10 and
> 1.11 or just fix in current release and support forward compatibility post
> 1.12+.
>
>
> Arina/Sudheesh - Please suggest if the analysis and fix sounds good and
> what are the releases we should consider the fix for. Considering 1.12
> release is planned soon can we take the fix in 1.12 release ?
>
>
>
> [1]: https://github.com/apache/drill/blob/1.10.0/protocol/
> src/main/protobuf/User.proto#L70
>
> [2]: https://github.com/apache/drill/blob/1.11.0/protocol/
> src/main/protobuf/User.proto#L70
>
> [3]: http://androiddevblog.com/protocol-buffers-pitfall-
> adding-enum-values/
>
> [4]: https://github.com/apache/drill/blob/1.10.0/exec/java-
> exec/src/main/java/org/apache/drill/exec/rpc/user/UserServer.java#L297
>
>
> Thanks,
> Sorabh
>

Re: Using Tableau to connect to DB engines using Calcite's JDBC driver

[Laurent Goujon]

AFAIK Tableau uses Drill ODBC driver, not JDBC (although Tableau hinted at
some JDBC support at some point: https://community.tableau.com/ideas/4633).

it's technically feasible BUT the Drill protocol is very low level so the
adapter would have to use the Drill RPC protocol and represent its own data
as DrillBuf (which is column oriented, not row oriented). Also, Tableau
seems to optimize things a bit for a specific DB: it doesn't query the
driver for what is supported and lots of things are hardcoded, which means
either the DB would have to replicate Drill dialect of SQL, or the
adaptation layer would have to translate.

Laurent

On Tue, Sep 19, 2017 at 3:17 PM, Muhammad Gelbana 
wrote:

> Tableau supports Apache Drill JDBC driver, so you basically can use Drill
> as a data provider for Tableau.
>
> I'm asking if anyone implemented a Calcite adapter for some data engine and
> tested if Tableau would be able to connect to it as if it was Apache Drill
> ?
>
> It's like you connect to that adapter by configuring an Apache Drill
> connection to it, through Tableau.
>
> Because otherwise, that data engine will need to have an ODBC driver, which
> is clearly a pain in the neck if you Google enough. That's actually what
> I'm trying to do. I need to implement a Calcite adapter to support a data
> engine but supporting Tableau is essential to our customers and I'd be very
> happy if I can avoid going through the Calcite ODBC driver path.
>
> I apologize if this sounds like a Calcite question but I believe Drill
> developers who worked on the JDBC driver can give a good insight.
>
> If you ask me, I believe Drill is all about Calcite in distributed mode :D,
> this may very well be so sketchy point of view but I'm not experienced with
> Drill or Calcite myself.
>
> Hopefully I explained my self clearly.
>
> Thanks,
> Gelbana
>

[jira] [Created] (DRILL-5668) C++ connector crash when query error message is too long

[Laurent Goujon (JIRA)]

Laurent Goujon created DRILL-5668:
-

 Summary: C++ connector crash when query error message is too long
 Key: DRILL-5668
 URL: https://issues.apache.org/jira/browse/DRILL-5668
 Project: Apache Drill
  Issue Type: Bug
  Components: Client - C++
Affects Versions: 1.10.0
Reporter: Laurent Goujon
Assignee: Laurent Goujon
 Fix For: 1.11.0


The C++ connector crashes if the query sent to the server is invalid and the 
error message returned by the server is too long.

The cause of the crash is a call to a {{vsprintf}} function which takes a fixed 
sized char buffer, but doesn't take a size argument.

https://github.com/apache/drill/blob/master/contrib/native/client/src/clientlib/errmsgs.cpp#L80



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (DRILL-5369) Missing initialization for ServerMetaContext

[Laurent Goujon (JIRA)]

Laurent Goujon created DRILL-5369:
-

 Summary: Missing initialization for ServerMetaContext
 Key: DRILL-5369
 URL: https://issues.apache.org/jira/browse/DRILL-5369
 Project: Apache Drill
  Issue Type: Bug
  Components: Client - C++
Reporter: Laurent Goujon
Assignee: Laurent Goujon


{{ServerMetaContext}} is not initialized properly which might cause some 
unexpected issues (like getMetadata() returning before receiving the answer) in 
some cases.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (DRILL-5368) Memory leak in C++ server metadata handler

[Laurent Goujon (JIRA)]

Laurent Goujon created DRILL-5368:
-

 Summary: Memory leak in C++ server metadata handler
 Key: DRILL-5368
 URL: https://issues.apache.org/jira/browse/DRILL-5368
 Project: Apache Drill
  Issue Type: Bug
  Components: Client - C++
Affects Versions: 1.10.0
Reporter: Laurent Goujon
Assignee: Laurent Goujon
Priority: Minor


When receiving server metadata response, a protobuf ServerMetaResp object is 
dynamically allocated but never freed.

Since for this handler, there's no need to keep the instance attached to the 
handler (content is copied over by the MetaData class), a reference is enough 
and allocation can be done on the stack.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (DRILL-5311) C++ connector connect doesn't wait for handshake to complete

[Laurent Goujon (JIRA)]

Laurent Goujon created DRILL-5311:
-

 Summary: C++ connector connect doesn't wait for handshake to 
complete
 Key: DRILL-5311
 URL: https://issues.apache.org/jira/browse/DRILL-5311
 Project: Apache Drill
  Issue Type: Bug
  Components: Client - C++
Reporter: Laurent Goujon


The C++ connector connect methods returns okay as soon as the tcp connection is 
succesfully established between client and server, and the handshake message is 
sent. However it doesn't wait for handshake to have completed.

The consequence is that if handshake failed, the error is deferred to the first 
query, which might be unexpected by the application.

I believe that validateHanshake method in drillClientImpl should wait for the 
handshake to complete, as it seems a bit more saner...



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Re: Time for 1.10 release

[Laurent Goujon]

The C++ patches have been built under windows/Linux/macos, and sanity
checks have been done for all of them. For the escape character fix and the
cancel fix, those have been tested in production like environment for weeks.

Laurent

On Mar 1, 2017 13:10, "Jinfeng Ni"  wrote:

Hi Laurent,

Thanks for the update. For the C++ client commits (DRILL-5301,
DRILL-5221), did you guys run some sanity check to see if there is any
regression?  I know Parth used to build/test with the new C++ change.
Since he is not available this time, just wanna to check if you have
done similar things.

Thanks,

Jinfeng



On Wed, Mar 1, 2017 at 10:31 AM, Laurent Goujon  wrote:
> The following jiras have all been reviewed and are ready to commit:
> * DRILL-4994: Add back JDBC prepared statement for older servers
> * DRILL-4730: Update JDBC DatabaseMetaData implementation to use new
> Metadata APIs
> * DRILL-5301: Server metadata API
> * DRILL-5167: Send escape character for metadata queries
> * DRILL-5221: Send cancel message as soon as possible in C++ connector
>
> For DRILL-3510 (Add ANSI_QUOTES option so that Drill's SQL Parser will
> recognize ANSI_SQL identifiers), it's my understanding that Vitalii would
> wait for DRILL-5301 changes to be merged, and refactor his patch on top of
> it, and I guess it will be post 1.10
>
> Laurent
>
>
>
> On Mon, Feb 27, 2017 at 3:07 PM, Jinfeng Ni  wrote:
>
>> The proposed cutoff date is this Wednesday, March 1st.
>>
>> I normally will just @someone in the PRs, so that someone would get
>> notice of PR, and hopefully get it reviewed.
>>
>>
>>
>> On Mon, Feb 27, 2017 at 2:42 PM, Laurent Goujon 
>> wrote:
>> > Thanks for the heads up: I'll try to ping directly some committers to
>> > review my changes.
>> >
>> > For information, what's the official cutoff date?
>> >
>> > Laurent
>> >
>> > PS: it's too bad that people with expertise don't have time to review
my
>> > patches, since some of them have been open for weeks or even months
(e.g.
>> > DRILL-4730 was cut out of Drill 1.9). If some people have suggestions
on
>> > what I should have done differently, I would greatly appreciate the
>> > feedback.
>> >
>> > On Mon, Feb 27, 2017 at 2:18 PM, Jinfeng Ni  wrote:
>> >
>> >> Here are the current status for the pending PRs for 1.10.
>> >>
>> >> JIRAs have been merged into Apache master branches:
>> >>
>> >>   DRILL-4280
>> >>   DRILL-5275
>> >>   DRILL-5260
>> >>   DRILL-5273
>> >>   DRILL-5257
>> >>   DRILL-5255
>> >>   DRILL-5274
>> >>
>> >> JIRAs that are in "ready-to-commit" status:
>> >>   DRILL-5258
>> >>   DRILL-5034
>> >>   DRILL-4963
>> >>   DRILL-5252
>> >>
>> >> JIRAs that have review comments to address:
>> >>   DRILL-5266
>> >>   DRILL-5114
>> >>   DRILL-5284
>> >>
>> >> @Laurent,  for the JDBC PRs you want to merge, is it possible that you
>> >> may get someone to review the PRs? Since both Sudheesh and Parth
>> >> (people have expertise) are on travel currently, it probably is not
>> >> likely that one of them can review your PRs before the cutoff for
>> >> 1.10.
>> >>
>> >> Thanks,
>> >> Jinfeng
>> >>
>> >>
>> >>
>> >>
>> >> On Mon, Feb 27, 2017 at 9:48 AM, Laurent Goujon 
>> >> wrote:
>> >> > Hi Jinfeng,
>> >> >
>> >> > Thanks for volunteering:
>> >> >
>> >> > Please consider the following JIRAs (PRs already open):
>> >> > * DRILL-4994: Add back JDBC prepared statement for older servers
>> >> > * DRILL-4730: Update JDBC DatabaseMetaData implementation to use new
>> >> > Metadata APIs
>> >> > * DRILL-5301: Server metadata API
>> >> > * DRILL-5167: Send escape character for metadata queries
>> >> > * DRILL-5221: Send cancel message as soon as possible in C++
connector
>> >> > * DRILL-3510: Add ANSI_QUOTES option so that Drill's SQL Parser will
>> >> > recognize ANSI_SQL identifiers
>> >> >
>> >> >
>> >> >
>> >> > On Fri, Feb 24, 2017 at 5:22 AM, Arina Yelchiyeva <
>> >> > arina.yelchiy...@gmail.com> wrote:
>> >> >
>> >> >> Hi Jinfeng,
>>

Re: Time for 1.10 release

[Laurent Goujon]

The following jiras have all been reviewed and are ready to commit:
* DRILL-4994: Add back JDBC prepared statement for older servers
* DRILL-4730: Update JDBC DatabaseMetaData implementation to use new
Metadata APIs
* DRILL-5301: Server metadata API
* DRILL-5167: Send escape character for metadata queries
* DRILL-5221: Send cancel message as soon as possible in C++ connector

For DRILL-3510 (Add ANSI_QUOTES option so that Drill's SQL Parser will
recognize ANSI_SQL identifiers), it's my understanding that Vitalii would
wait for DRILL-5301 changes to be merged, and refactor his patch on top of
it, and I guess it will be post 1.10

Laurent



On Mon, Feb 27, 2017 at 3:07 PM, Jinfeng Ni  wrote:

> The proposed cutoff date is this Wednesday, March 1st.
>
> I normally will just @someone in the PRs, so that someone would get
> notice of PR, and hopefully get it reviewed.
>
>
>
> On Mon, Feb 27, 2017 at 2:42 PM, Laurent Goujon 
> wrote:
> > Thanks for the heads up: I'll try to ping directly some committers to
> > review my changes.
> >
> > For information, what's the official cutoff date?
> >
> > Laurent
> >
> > PS: it's too bad that people with expertise don't have time to review my
> > patches, since some of them have been open for weeks or even months (e.g.
> > DRILL-4730 was cut out of Drill 1.9). If some people have suggestions on
> > what I should have done differently, I would greatly appreciate the
> > feedback.
> >
> > On Mon, Feb 27, 2017 at 2:18 PM, Jinfeng Ni  wrote:
> >
> >> Here are the current status for the pending PRs for 1.10.
> >>
> >> JIRAs have been merged into Apache master branches:
> >>
> >>   DRILL-4280
> >>   DRILL-5275
> >>   DRILL-5260
> >>   DRILL-5273
> >>   DRILL-5257
> >>   DRILL-5255
> >>   DRILL-5274
> >>
> >> JIRAs that are in "ready-to-commit" status:
> >>   DRILL-5258
> >>   DRILL-5034
> >>   DRILL-4963
> >>   DRILL-5252
> >>
> >> JIRAs that have review comments to address:
> >>   DRILL-5266
> >>   DRILL-5114
> >>   DRILL-5284
> >>
> >> @Laurent,  for the JDBC PRs you want to merge, is it possible that you
> >> may get someone to review the PRs? Since both Sudheesh and Parth
> >> (people have expertise) are on travel currently, it probably is not
> >> likely that one of them can review your PRs before the cutoff for
> >> 1.10.
> >>
> >> Thanks,
> >> Jinfeng
> >>
> >>
> >>
> >>
> >> On Mon, Feb 27, 2017 at 9:48 AM, Laurent Goujon 
> >> wrote:
> >> > Hi Jinfeng,
> >> >
> >> > Thanks for volunteering:
> >> >
> >> > Please consider the following JIRAs (PRs already open):
> >> > * DRILL-4994: Add back JDBC prepared statement for older servers
> >> > * DRILL-4730: Update JDBC DatabaseMetaData implementation to use new
> >> > Metadata APIs
> >> > * DRILL-5301: Server metadata API
> >> > * DRILL-5167: Send escape character for metadata queries
> >> > * DRILL-5221: Send cancel message as soon as possible in C++ connector
> >> > * DRILL-3510: Add ANSI_QUOTES option so that Drill's SQL Parser will
> >> > recognize ANSI_SQL identifiers
> >> >
> >> >
> >> >
> >> > On Fri, Feb 24, 2017 at 5:22 AM, Arina Yelchiyeva <
> >> > arina.yelchiy...@gmail.com> wrote:
> >> >
> >> >> Hi Jinfeng,
> >> >>
> >> >> please also consider the following Jiras (PR are already open):
> >> >> * DRILL-4963: Issues when overloading Drill native functions with
> >> dynamic
> >> >> UDFs
> >> >> * DRILL-5255: Remove default temporary workspace check at drillbit
> >> start up
> >> >> * DRILL-5274: Exception thrown in Drillbit shutdown in UDF cleanup
> code
> >> >>
> >> >> Kind regards
> >> >> Arina
> >> >>
> >> >> On Thu, Feb 23, 2017 at 8:40 PM, Paul Rogers 
> wrote:
> >> >>
> >> >> > Hi Jinfeng,
> >> >> >
> >> >> > Thanks for volunteering!
> >> >> >
> >> >> > The following are working their way though the system:
> >> >> >
> >> >> > PRs outstanding:
> >> >> >
> >> >> > * DRILL-5275: Sort spill is slow due to r

Re: Time for 1.10 release

[Laurent Goujon]

Thanks for the heads up: I'll try to ping directly some committers to
review my changes.

For information, what's the official cutoff date?

Laurent

PS: it's too bad that people with expertise don't have time to review my
patches, since some of them have been open for weeks or even months (e.g.
DRILL-4730 was cut out of Drill 1.9). If some people have suggestions on
what I should have done differently, I would greatly appreciate the
feedback.

On Mon, Feb 27, 2017 at 2:18 PM, Jinfeng Ni  wrote:

> Here are the current status for the pending PRs for 1.10.
>
> JIRAs have been merged into Apache master branches:
>
>   DRILL-4280
>   DRILL-5275
>   DRILL-5260
>   DRILL-5273
>   DRILL-5257
>   DRILL-5255
>   DRILL-5274
>
> JIRAs that are in "ready-to-commit" status:
>   DRILL-5258
>   DRILL-5034
>   DRILL-4963
>   DRILL-5252
>
> JIRAs that have review comments to address:
>   DRILL-5266
>   DRILL-5114
>   DRILL-5284
>
> @Laurent,  for the JDBC PRs you want to merge, is it possible that you
> may get someone to review the PRs? Since both Sudheesh and Parth
> (people have expertise) are on travel currently, it probably is not
> likely that one of them can review your PRs before the cutoff for
> 1.10.
>
> Thanks,
> Jinfeng
>
>
>
>
> On Mon, Feb 27, 2017 at 9:48 AM, Laurent Goujon 
> wrote:
> > Hi Jinfeng,
> >
> > Thanks for volunteering:
> >
> > Please consider the following JIRAs (PRs already open):
> > * DRILL-4994: Add back JDBC prepared statement for older servers
> > * DRILL-4730: Update JDBC DatabaseMetaData implementation to use new
> > Metadata APIs
> > * DRILL-5301: Server metadata API
> > * DRILL-5167: Send escape character for metadata queries
> > * DRILL-5221: Send cancel message as soon as possible in C++ connector
> > * DRILL-3510: Add ANSI_QUOTES option so that Drill's SQL Parser will
> > recognize ANSI_SQL identifiers
> >
> >
> >
> > On Fri, Feb 24, 2017 at 5:22 AM, Arina Yelchiyeva <
> > arina.yelchiy...@gmail.com> wrote:
> >
> >> Hi Jinfeng,
> >>
> >> please also consider the following Jiras (PR are already open):
> >> * DRILL-4963: Issues when overloading Drill native functions with
> dynamic
> >> UDFs
> >> * DRILL-5255: Remove default temporary workspace check at drillbit
> start up
> >> * DRILL-5274: Exception thrown in Drillbit shutdown in UDF cleanup code
> >>
> >> Kind regards
> >> Arina
> >>
> >> On Thu, Feb 23, 2017 at 8:40 PM, Paul Rogers  wrote:
> >>
> >> > Hi Jinfeng,
> >> >
> >> > Thanks for volunteering!
> >> >
> >> > The following are working their way though the system:
> >> >
> >> > PRs outstanding:
> >> >
> >> > * DRILL-5275: Sort spill is slow due to repeated allocations
> >> > * DRILL-5260: Extend "Cluster Fixture" test framework
> >> > * DRILL-5258: Access mock data definition from SQL
> >> > * DRILL-5273: CompliantTextReader excessive memory use
> >> > * DRILL-5266: Parquet returns low-density batches
> >> > * DRILL-5257: Run-time control of query profiles
> >> >
> >> > The following community contribution has been approved and is ready to
> >> > commit:
> >> >
> >> > * DRILL-5252: Fix a condition that always returns true
> >> >
> >> > The following will be (re)issued today or tomorrow:
> >> >
> >> > * DRILL-5114: Rationalize use of Logback logging in unit tests
> >> > * DRILL-5284: Roll-up of final fixes for managed sort
> >> >
> >> > Thanks,
> >> >
> >> > - Paul
> >> >
> >> > > On Feb 23, 2017, at 9:57 AM, Sudheesh Katkam 
> wrote:
> >> > >
> >> > > I would like to include:
> >> > >
> >> > > + DRILL-4280: https://github.com/apache/drill/pull/578 (going
> through
> >> > the last round of comments)
> >> > >
> >> > > Thank you,
> >> > > Sudheesh
> >> > >
> >> > >> On Feb 22, 2017, at 11:16 PM, Jinfeng Ni  wrote:
> >> > >>
> >> > >> Hi Drillers,
> >> > >>
> >> > >> It has been almost 3 months since we release Drill 1.9. We have
> >> > >> resolved plenty of fixes and improvements (closed around 88 JIRAs
> >> > >> [1]). I propose that we start the 1.10 release process, and set
> >> > >> Wednesday 3/1 as the cutoff day for code checkin. After 3/1, we
> should
> >> > >> start build a release candidate.
> >> > >>
> >> > >> Please reply in this email thread if you have something near
> complete
> >> > >> and you would like to include in 1.10 release.
> >> > >>
> >> > >> I volunteer as the release manager, unless someone else come
> forward.
> >> > >>
> >> > >> Thanks,
> >> > >>
> >> > >> Jinfeng
> >> > >>
> >> > >> [1] https://issues.apache.org/jira/browse/DRILL/
> >> fixforversion/12338769
> >> > >
> >> >
> >> >
> >>
>

Re: Time for 1.10 release

[Laurent Goujon]

Hi Jinfeng,

Thanks for volunteering:

Please consider the following JIRAs (PRs already open):
* DRILL-4994: Add back JDBC prepared statement for older servers
* DRILL-4730: Update JDBC DatabaseMetaData implementation to use new
Metadata APIs
* DRILL-5301: Server metadata API
* DRILL-5167: Send escape character for metadata queries
* DRILL-5221: Send cancel message as soon as possible in C++ connector
* DRILL-3510: Add ANSI_QUOTES option so that Drill's SQL Parser will
recognize ANSI_SQL identifiers



On Fri, Feb 24, 2017 at 5:22 AM, Arina Yelchiyeva <
arina.yelchiy...@gmail.com> wrote:

> Hi Jinfeng,
>
> please also consider the following Jiras (PR are already open):
> * DRILL-4963: Issues when overloading Drill native functions with dynamic
> UDFs
> * DRILL-5255: Remove default temporary workspace check at drillbit start up
> * DRILL-5274: Exception thrown in Drillbit shutdown in UDF cleanup code
>
> Kind regards
> Arina
>
> On Thu, Feb 23, 2017 at 8:40 PM, Paul Rogers  wrote:
>
> > Hi Jinfeng,
> >
> > Thanks for volunteering!
> >
> > The following are working their way though the system:
> >
> > PRs outstanding:
> >
> > * DRILL-5275: Sort spill is slow due to repeated allocations
> > * DRILL-5260: Extend "Cluster Fixture" test framework
> > * DRILL-5258: Access mock data definition from SQL
> > * DRILL-5273: CompliantTextReader excessive memory use
> > * DRILL-5266: Parquet returns low-density batches
> > * DRILL-5257: Run-time control of query profiles
> >
> > The following community contribution has been approved and is ready to
> > commit:
> >
> > * DRILL-5252: Fix a condition that always returns true
> >
> > The following will be (re)issued today or tomorrow:
> >
> > * DRILL-5114: Rationalize use of Logback logging in unit tests
> > * DRILL-5284: Roll-up of final fixes for managed sort
> >
> > Thanks,
> >
> > - Paul
> >
> > > On Feb 23, 2017, at 9:57 AM, Sudheesh Katkam  wrote:
> > >
> > > I would like to include:
> > >
> > > + DRILL-4280: https://github.com/apache/drill/pull/578 (going through
> > the last round of comments)
> > >
> > > Thank you,
> > > Sudheesh
> > >
> > >> On Feb 22, 2017, at 11:16 PM, Jinfeng Ni  wrote:
> > >>
> > >> Hi Drillers,
> > >>
> > >> It has been almost 3 months since we release Drill 1.9. We have
> > >> resolved plenty of fixes and improvements (closed around 88 JIRAs
> > >> [1]). I propose that we start the 1.10 release process, and set
> > >> Wednesday 3/1 as the cutoff day for code checkin. After 3/1, we should
> > >> start build a release candidate.
> > >>
> > >> Please reply in this email thread if you have something near complete
> > >> and you would like to include in 1.10 release.
> > >>
> > >> I volunteer as the release manager, unless someone else come forward.
> > >>
> > >> Thanks,
> > >>
> > >> Jinfeng
> > >>
> > >> [1] https://issues.apache.org/jira/browse/DRILL/
> fixforversion/12338769
> > >
> >
> >
>

[jira] [Created] (DRILL-5301) Add server metadata API

[Laurent Goujon (JIRA)]

Laurent Goujon created DRILL-5301:
-

 Summary: Add server metadata API
 Key: DRILL-5301
 URL: https://issues.apache.org/jira/browse/DRILL-5301
 Project: Apache Drill
  Issue Type: Improvement
  Components:  Server, Client - C++, Client - Java, Client - JDBC, 
Client - ODBC
Reporter: Laurent Goujon


JDBC and ODBC clients exposes lots of metadata regarding server version and 
support of various parts of the SQL standard.

Currently the returned information is hardcoded in both clients/drivers which 
means that the infomation returned is support as of the client version, not the 
server version.

Instead, a new method should be provided to the clients to query the actual 
server support. Support on the client or the server should be optional (for 
example a client should not use this API if the server doesn't support it and 
fallback to default values).



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

C++ patches for review

[Laurent Goujon]

Hi,

I have a couple of open patches for the C++ connector:
- DRILL-5167: Send escape character for metadata queries
https://github.com/apache/drill/pull/712

- DRILL-5219: Relax user properties validation in C++ client
https://github.com/apache/drill/pull/727

- DRILL-5220: Provide API to set application/client names in C++ connector
https://github.com/apache/drill/pull/728

- DRILL-5221: Send cancel message as soon as possible in C++ connector
https://github.com/apache/drill/pull/733

If there are some committers who can have a look at those, it would be much
appreciated.

Laurent

[jira] [Created] (DRILL-5221) cancel message is delayed until queryid or data is received

[Laurent Goujon (JIRA)]

Laurent Goujon created DRILL-5221:
-

 Summary: cancel message is delayed until queryid or data is 
received
 Key: DRILL-5221
 URL: https://issues.apache.org/jira/browse/DRILL-5221
 Project: Apache Drill
  Issue Type: Improvement
  Components: Client - C++
Affects Versions: 1.9.0
Reporter: Laurent Goujon


When user is calling the cancel method of the C++ client, the client wait for a 
message from the server to reply back with a cancellation message.

In case of queries taking a long time to return their first batch, it means 
cancellation is taking the same amount of time to be effective, instead of 
cancelling right away the query (assuming the query id has already been 
received, which is generally the case).

It seems this was foreseen by [~vkorukanti] in his initial patch 
(https://github.com/vkorukanti/drill/commit/e0ef6349aac48de5828b6d725c2cf013905d18eb)
 but was omitted when I backported it post metadata changes.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (DRILL-5220) Add api to set application name in C++ connector

[Laurent Goujon (JIRA)]

Laurent Goujon created DRILL-5220:
-

 Summary: Add api to set application name in C++ connector
 Key: DRILL-5220
 URL: https://issues.apache.org/jira/browse/DRILL-5220
 Project: Apache Drill
  Issue Type: Improvement
  Components: Client - C++
Affects Versions: 1.8.0
Reporter: Laurent Goujon
Priority: Minor


There's no API for a C++ connector user to specify the name of the application, 
and to provide it to the server (optional field added in DRILL-4369)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (DRILL-5219) Remove DrillUserProperties filtering in C++ driver

[Laurent Goujon (JIRA)]

Laurent Goujon created DRILL-5219:
-

 Summary: Remove DrillUserProperties filtering in C++ driver
 Key: DRILL-5219
 URL: https://issues.apache.org/jira/browse/DRILL-5219
 Project: Apache Drill
  Issue Type: Bug
  Components: Client - C++
Reporter: Laurent Goujon
Priority: Minor


Unlike the Java client, the C++ connector filter out unknown Drill user 
properties:
https://github.com/apache/drill/blob/master/contrib/native/client/src/clientlib/drillClientImpl.cpp#L374

This prevents a client (like the ODBC driver) to pass extra properties to the 
server (like extra metainformation, or some specific behavior for a given 
software)




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Connect Drill to Google Cloud Storage

[Laurent Goujon]

Never tried myself, but that page might help:
https://cloud.google.com/hadoop/google-cloud-storage-connector. It explains
where to download the google connector and how to configure it.

After that, you should be able to configure the storage plugin to use gs://
url (vs s3://)

Laurent

On Tue, Jan 10, 2017 at 10:37 PM, Pawan Sharma 
wrote:

> Hi,
>
> Need your help on connecting Drill to Google cloud storage. I want to
> connect Apache Drill to google cloud storage buckets and fetch data from
> the file files stored in those buckets.
>
> I can specify access id and key in core-site.xml in order to connect to
> AWS. Is there a similar way to connect drill to google cloud.
> Thanks,
> Pawan
>
> Software Engineer
> Mastek Ltd.
>

[jira] [Created] (DRILL-5167) C++ connector does not set escape string for metadata search pattern

[Laurent Goujon (JIRA)]

Laurent Goujon created DRILL-5167:
-

 Summary: C++ connector does not set escape string for metadata 
search pattern
 Key: DRILL-5167
 URL: https://issues.apache.org/jira/browse/DRILL-5167
 Project: Apache Drill
  Issue Type: Bug
Reporter: Laurent Goujon
Priority: Minor


C++ connector does not set the escape string for search pattern when doing 
metadata operation (getCatalogs/getSchema/getTables/getColumns). It is assumed 
to be '\\' as returned by DrillMetadata::getSearchEscapeString(), but because 
this is not sent over the wire, the server will actually consider that there's 
no escape character, and might return different or no result compared to what 
has been requested.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [Discuss] Drill and Java 8

[Laurent Goujon]

I don't think that all unit tests pass in Java8. I fixed a couple of test
cases but I'm pretty sure some JDBC tests still fail because of the new
default methods added to the JDBC interface which are not overriden in
Avatica and Drill: https://issues.apache.org/jira/browse/DRILL-4333

I don't have strong opinions against or for dropping Java 7 support but as
far as I know, Oracle still support Java7 until 2022 for paying customers.
Redhat on the other end will end openjdk 7 support in June 2018. I guess
it's more of a matter of whatever users are still using or not (and looking
at Hortonworks/Cloudera/MapR, they still support Java7).

As for the mysterious issue in Eclipse regarding Java8, it's more of a
maven/eclipse integration issue than a Drill issue. The corresponding
Eclipse bug is https://bugs.eclipse.org/bugs/show_bug.cgi?id=432992 and I
work around it by adding the correct tools.jar in the classpath to match
the JRE version.

Laurent

On Mon, Nov 28, 2016 at 11:12 AM, Abhishek Girish  wrote:

> One of the reasons I remember not supporting Java 8 were unresolved unit
> test failures. I'm not sure if they still occur - did you get a chance to
> try a build with unit tests turned on?
>
> On Mon, Nov 28, 2016 at 10:50 AM, Paul Rogers 
> wrote:
>
> > Hi All,
> >
> > Drill still builds with Java 7. Many of us have upgraded to Java 8. (Java
> > 7 reached end-of-life in April of 2015.)
> >
> > For the most part, it works fine to have Java 8 as the default JDK, while
> > Drill builds with Java 7. However, it becomes a real problem when trying
> to
> > use Eclipse to run unit tests, especially those that include JMockit.
> >
> > When I used Java 8 for Eclipse, but Java 7 for the Drill build, unit
> tests
> > fail with mysterious errors. Forcing Eclipse to use Java 7 (or forcing
> > Drill to use Java 8) is the workaround. (Something about the way the
> JUnit
> > test runner works.)
> >
> > Have we considered upgrading Drill’s build to Java 8?
> >
> > Does Drill or Apache have obligations to support Java 7 even though
> Oracle
> > no longer does so?
> >
> > The workaround is to muck about until the Java versions match. A hassle,
> > but it works. It does, however take time that could be better spent on
> > other things.
> >
> > Thanks,
> >
> > - Paul
>

Re: Time for a 1.9 Release?

[Laurent Goujon]

I guess it's DRILL-4730 and not DRILL-4370

On Fri, Nov 4, 2016 at 6:23 PM, Sudheesh Katkam  wrote:

> Out of the 17 requested tickets, we resolved 13 over the week, and 4 have
> been deferred (DRILL-4280, DRILL-4858, DRILL-4370, DRILL-4706). Thank you
> everyone!
>
> I get will get the RC0 out on Monday.
>
> - Sudheesh
>
> On Fri, Nov 4, 2016 at 11:49 AM, Jinfeng Ni  wrote:
>
> > Agreed with Parth that we probably should start a separate thread to
> > discuss release version number after 1.9.0.
> >
> > I'll start a new thread to discuss that, and leave this thread for
> > drill 1.9.0 release matters.
> >
> >
> > On Thu, Nov 3, 2016 at 3:53 PM, Sudheesh Katkam 
> > wrote:
> > > Gentle reminder that all check-ins should be done by tomorrow. Please
> see
> > > the latest statuses of commits that we are targeting:
> > >
> > > https://docs.google.com/spreadsheets/d/1UJSXLrfUNZwUnx_
> > > JzkwAcXSxmcbG7meBDad6ZTxlSmw
> > >
> > > Thank you,
> > > Sudheesh
> > >
> > >
> > > On Tue, Nov 1, 2016 at 11:19 AM, Sudheesh Katkam 
> > > wrote:
> > >
> > >> The current list of candidate commits for the release is here:
> > >>
> > >> https://docs.google.com/spreadsheets/d/1UJSXLrfUNZwUnx_
> > >> JzkwAcXSxmcbG7meBDad6ZTxlSmw
> > >>
> > >>
> > >> On Mon, Oct 31, 2016 at 8:53 AM, Subbu Srinivasan <
> > ssriniva...@zscaler.com
> > >> > wrote:
> > >>
> > >>> +1.
> > >>>
> > >>> On Sun, Oct 30, 2016 at 10:23 PM, Paul Rogers 
> > >>> wrote:
> > >>>
> > >>> > For release numbers, 1.10 (then 1.11, 1.12, …) seems like a good
> > idea.
> > >>> >
> > >>> > At first it may seem odd to go to 1.10 from 1.9. Might people get
> > >>> confused
> > >>> > between 1.10 and 1.1.0? But, there is precedence. Tomcat’s latest
> > >>> 7-series
> > >>> > release is 7.0.72. Java is on 8u112. And so on.
> > >>> >
> > >>> > I like the idea of moving to 2.0 later when the team introduces a
> > major
> > >>> > change, rather than by default just because the numbers roll
> around.
> > For
> > >>> > example, Hadoop when to 2.x when YARN was introduced. Impala
> appears
> > to
> > >>> > have moved to 2.0 when they added Spill to disk for some (all?)
> > >>> operators.
> > >>> >
> > >>> > - Paul
> > >>> >
> > >>> > > On Oct 28, 2016, at 10:34 AM, Sudheesh Katkam <
> sudhe...@apache.org
> > >
> > >>> > wrote:
> > >>> > >
> > >>> > > Hi Drillers,
> > >>> > >
> > >>> > > We have a reasonable number of fixes and features since the last
> > >>> release
> > >>> > > [1]. Releasing itself takes a while; so I propose we start the
> 1.9
> > >>> > release
> > >>> > > process.
> > >>> > >
> > >>> > > I volunteer as the release manager, unless there are objections.
> > >>> > >
> > >>> > > We should also discuss what the release version number should be
> > after
> > >>> > 1.9.
> > >>> > >
> > >>> > > Thank you,
> > >>> > > Sudheesh
> > >>> > >
> > >>> > > [1] https://issues.apache.org/jira/browse/DRILL/fixforversion/
> > >>> 12337861
> > >>> >
> > >>> >
> > >>>
> > >>
> > >>
> >
>

[jira] [Created] (DRILL-4994) Prepared statement stopped working between 1.8.0 client and < 1.7.0 server

[Laurent Goujon (JIRA)]

Laurent Goujon created DRILL-4994:
-

 Summary: Prepared statement stopped working between 1.8.0 client 
and < 1.7.0 server
 Key: DRILL-4994
 URL: https://issues.apache.org/jira/browse/DRILL-4994
 Project: Apache Drill
  Issue Type: Bug
Reporter: Laurent Goujon


Older servers (pre-1.8.0) don't support the prepared statement rpc method, but 
the JDBC client doesn't check if it is available or not. The end result is that 
the statement is stuck as the server is not responding back.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Apache Drill Hangout Minutes - 11/1/16

[Laurent Goujon]

(+dev@drill.apache.org bcc u...@drill.apache.org)
other options:
- let the user decide between plain auth and sasl auth
- let the client could send a flag to announce support for SASL auth, and
do not send the credentials in the first handshake. An old server would
then send AUTH_FAILED (and drop the connection) but the client would retry
with the plain credentials, whereas the new server would send AUTH_REQUIRED
as expected.

Laurent

Laurent

On Wed, Nov 2, 2016 at 3:30 PM, Sudheesh Katkam  wrote:

> I am going to update the pull request so that both will be "ok".
>
> This implies that username/password credentials will be sent to the server
> twice, during handshake and during SASL exchange. And sending credentials
> through handshake will be deprecated (and removed in a future release).
>
> Thank you,
> Sudheesh
>
> On Wed, Nov 2, 2016 at 2:58 PM, Jacques Nadeau  wrote:
>
> > Since I'm not that close to DRILL-4280, I wanted to clarify expectation:
> >
> >
> > <1.9 Client  <==>  1.9 Server (ok)
> >  1.9 Client  <==> <1.9 Server (fails)
> >
> > Is that correct?
> >
> >
> >
> >
> >
> >
> > --
> > Jacques Nadeau
> > CTO and Co-Founder, Dremio
> >
> > On Tue, Nov 1, 2016 at 8:44 PM, Sudheesh Katkam 
> > wrote:
> >
> > > Hi Laurent,
> > >
> > > That's right; this was mentioned in the design document.
> > >
> > > I am piggybacking on previous changes that break the "newer clients
> > talking
> > > to older servers" compatibility. For example, as I understand, some
> > > resolved sub-tasks of DRILL-4714 [1] *implicitly* break this
> > compatibility;
> > > say the "newer" API that was introduced is used by an application which
> > is
> > > talking to an older server. The older server drops the connection,
> unable
> > > to handle the message.
> > >
> > > In DRILL-4280, there is an *explicit* break in that specific
> > compatibility,
> > > and the error message is much cleaner with a version mismatch message.
> > The
> > > difference is that the C++ client (unlike the Java client) checks for
> the
> > > server version as well, which make the compatibility break more
> visible.
> > >
> > > I am not sure about the plan of action in general about this
> > compatibility.
> > > However, I could work around the issue by advertising clients' SASL
> > > capability to the server. What do you think?
> > >
> > > Thank you,
> > > Sudheesh
> > >
> > > [1] https://issues.apache.org/jira/browse/DRILL-4714
> > >
> > > On Nov 1, 2016, at 7:49 PM, Laurent Goujon  wrote:
> > >
> > > Just for clarity, DRILL-4280 is a breaking-protocol change, so is the
> > plan
> > > to defer this change to a later release, or to defer bringing back
> > > compatibility between newer clients and older servers to a later
> release?
> > >
> > > Laurent
> > >
> > > On Tue, Nov 1, 2016 at 3:43 PM, Zelaine Fong 
> wrote:
> > >
> > > Oops, mistake in my notes.  For the second item, I meant DRILL-4280,
> not
> > > DRILL-1950.
> > >
> > > On Tue, Nov 1, 2016 at 3:40 PM, Zelaine Fong 
> wrote:
> > >
> > > Attendees: Paul, Padma, Sorabh, Boaz, Sudheesh, Vitalii, Roman, Dave O,
> > > Arina, Laurent, Kunal, Zelaine
> > >
> > > I had to leave the hangout at 10:30, so my notes only cover the
> > >
> > > discussion
> > >
> > > up till then.
> > >
> > > 1) Variable width decimal support - Dave O
> > >
> > > Currently Drill only supports fixed width byte array storage of
> decimals.
> > > Dave has submitted a pull request for DRILL-4834 to add support for
> > >
> > > storing
> > >
> > > decimals with variable width byte arrays.  Eventually, variable width
> can
> > > replace fixed width, but the pull request doesn't cover that.  Dave
> would
> > > like someone in the community to review his pull request.
> > >
> > > 2) 1.9 release - Sudheesh
> > >
> > > Sudheesh is collecting pull requests for the release.  Some have been
> > > reviewed and are waiting to be merged.  Sudheesh plans to commit a
> batch
> > > this Wed and another this Friday.  He's targeting having a release
> > > candidate build available next Monday.
> > >
> > > Laurent asked about Sudheesh's pull request for DRILL-1950.  He asked
> > > whether thought had been given to supporting newer Drill clients with
> > >
> > > older
> > >
> > > Drill servers.  Sudheesh indicated that doing this would entail a
> > >
> > > breaking
> > >
> > > change in the protocol, and the plan was to defer doing this for a
> later
> > > release where we may want to make other breaking changes like this.
> > >
> >
>

Re: Apache Drill Hangout Minutes - 11/1/16

[Laurent Goujon]

That's a good point that the introduction of new request types for prepared
statement and metadata introduced some incompatibility issues, and that's
why I added a client/server version exchange so that client can make smart
decisions about using these new methods or not (this is the approach used
by the ODBC driver, where the new calls will be used only if server
advertises 1.9.0 or newer). On Java side, direct users of DrillClient would
have to check the version as well, a thing the JDBC driver doesn't. That
said, the metadata change is not merged yet (and I will add the extra
logic), which is not the case regarding prepared statement support.

Also, I verified that the old server doesn't drop the connection: it's
actually dropping the request, and not responding to the client :( (best
would be to return an error message to say unsupported).

Laurent


On Tue, Nov 1, 2016 at 8:44 PM, Sudheesh Katkam  wrote:

> Hi Laurent,
>
> That's right; this was mentioned in the design document.
>
> I am piggybacking on previous changes that break the "newer clients talking
> to older servers" compatibility. For example, as I understand, some
> resolved sub-tasks of DRILL-4714 [1] *implicitly* break this compatibility;
> say the "newer" API that was introduced is used by an application which is
> talking to an older server. The older server drops the connection, unable
> to handle the message.
>
> In DRILL-4280, there is an *explicit* break in that specific compatibility,
> and the error message is much cleaner with a version mismatch message. The
> difference is that the C++ client (unlike the Java client) checks for the
> server version as well, which make the compatibility break more visible.
>
> I am not sure about the plan of action in general about this compatibility.
> However, I could work around the issue by advertising clients' SASL
> capability to the server. What do you think?
>
> Thank you,
> Sudheesh
>
> [1] https://issues.apache.org/jira/browse/DRILL-4714
>
> On Nov 1, 2016, at 7:49 PM, Laurent Goujon  wrote:
>
> Just for clarity, DRILL-4280 is a breaking-protocol change, so is the plan
> to defer this change to a later release, or to defer bringing back
> compatibility between newer clients and older servers to a later release?
>
> Laurent
>
> On Tue, Nov 1, 2016 at 3:43 PM, Zelaine Fong  wrote:
>
> Oops, mistake in my notes.  For the second item, I meant DRILL-4280, not
> DRILL-1950.
>
> On Tue, Nov 1, 2016 at 3:40 PM, Zelaine Fong  wrote:
>
> Attendees: Paul, Padma, Sorabh, Boaz, Sudheesh, Vitalii, Roman, Dave O,
> Arina, Laurent, Kunal, Zelaine
>
> I had to leave the hangout at 10:30, so my notes only cover the
>
> discussion
>
> up till then.
>
> 1) Variable width decimal support - Dave O
>
> Currently Drill only supports fixed width byte array storage of decimals.
> Dave has submitted a pull request for DRILL-4834 to add support for
>
> storing
>
> decimals with variable width byte arrays.  Eventually, variable width can
> replace fixed width, but the pull request doesn't cover that.  Dave would
> like someone in the community to review his pull request.
>
> 2) 1.9 release - Sudheesh
>
> Sudheesh is collecting pull requests for the release.  Some have been
> reviewed and are waiting to be merged.  Sudheesh plans to commit a batch
> this Wed and another this Friday.  He's targeting having a release
> candidate build available next Monday.
>
> Laurent asked about Sudheesh's pull request for DRILL-1950.  He asked
> whether thought had been given to supporting newer Drill clients with
>
> older
>
> Drill servers.  Sudheesh indicated that doing this would entail a
>
> breaking
>
> change in the protocol, and the plan was to defer doing this for a later
> release where we may want to make other breaking changes like this.
>

Re: Apache Drill Hangout Minutes - 11/1/16

[Laurent Goujon]

Just for clarity, DRILL-4280 is a breaking-protocol change, so is the plan
to defer this change to a later release, or to defer bringing back
compatibility between newer clients and older servers to a later release?

Laurent

On Tue, Nov 1, 2016 at 3:43 PM, Zelaine Fong  wrote:

> Oops, mistake in my notes.  For the second item, I meant DRILL-4280, not
> DRILL-1950.
>
> On Tue, Nov 1, 2016 at 3:40 PM, Zelaine Fong  wrote:
>
> > Attendees: Paul, Padma, Sorabh, Boaz, Sudheesh, Vitalii, Roman, Dave O,
> > Arina, Laurent, Kunal, Zelaine
> >
> > I had to leave the hangout at 10:30, so my notes only cover the
> discussion
> > up till then.
> >
> > 1) Variable width decimal support - Dave O
> >
> > Currently Drill only supports fixed width byte array storage of decimals.
> > Dave has submitted a pull request for DRILL-4834 to add support for
> storing
> > decimals with variable width byte arrays.  Eventually, variable width can
> > replace fixed width, but the pull request doesn't cover that.  Dave would
> > like someone in the community to review his pull request.
> >
> > 2) 1.9 release - Sudheesh
> >
> > Sudheesh is collecting pull requests for the release.  Some have been
> > reviewed and are waiting to be merged.  Sudheesh plans to commit a batch
> > this Wed and another this Friday.  He's targeting having a release
> > candidate build available next Monday.
> >
> > Laurent asked about Sudheesh's pull request for DRILL-1950.  He asked
> > whether thought had been given to supporting newer Drill clients with
> older
> > Drill servers.  Sudheesh indicated that doing this would entail a
> breaking
> > change in the protocol, and the plan was to defer doing this for a later
> > release where we may want to make other breaking changes like this.
> >
>

Re: [HANGOUT] Topics for 11/1/16

[Laurent Goujon]

Which hangout link are we using today?

On Tue, Nov 1, 2016 at 9:51 AM, Sudheesh Katkam  wrote:

> Zelaine, That list is from the other thread ("Time for a 1.9 Release?"),
> where there was no mention of PR#639.
>
> Charles, DRILL-3423 is already on the list.
>
> Thank you,
> Sudheesh
>
> On Tue, Nov 1, 2016 at 9:43 AM, Charles Givre  wrote:
>
> > I think DRILL-3423 HTTPD parser should be added as well. I'm sorry I'm
> > traveling at the moment and won't be able to join the hangout.
> >
> > Sent from my iPhone
> >
> > > On Nov 1, 2016, at 12:40, Zelaine Fong  wrote:
> > >
> > > Regarding [1], shouldn't https://github.com/apache/drill/pull/639 be
> > added
> > > to the list of pull requests in review?
> > >
> > > -- Zelaine
> > >
> > >
> > >> On Tue, Nov 1, 2016 at 9:34 AM, Sudheesh Katkam 
> > wrote:
> > >>
> > >> I would like to talk about the 1.9 release. The current list of
> pending
> > >> requests is at [1].
> > >>
> > >> If Arina joins the hangout, I would like to discuss about permissions
> on
> > >> the various areas setup for dynamic UDF support.
> > >>
> > >> Thank you,
> > >> Sudheesh
> > >>
> > >> [1]
> > >> https://docs.google.com/spreadsheets/d/1UJSXLrfUNZwUnx_JzkwA
> > >> cXSxmcbG7meBDad6ZTxlSmw
> > >>
> > >>
> > >> On Mon, Oct 31, 2016 at 10:03 AM, Dave Oshinsky <
> > doshin...@commvault.com>
> > >> wrote:
> > >>
> > >>> Hi Paul,
> > >>> Can we talk a bit about working to improve decimal type support?  My
> > >>> related PR is waiting for review:
> > >>> https://issues.apache.org/jira/browse/DRILL-4834
> > >>>
> > >>> Thanks,
> > >>> Dave Oshinsky
> > >>>
> > >>> -Original Message-
> > >>> From: Paul Rogers [mailto:prog...@maprtech.com]
> > >>> Sent: Monday, October 31, 2016 12:33 PM
> > >>> To: dev@drill.apache.org; u...@drill.apache.org
> > >>> Subject: [HANGOUT] Topics for 11/1/16
> > >>>
> > >>> Hi All,
> > >>>
> > >>> Our bi-weekly hangout is tomorrow (11/01/16, 10 AM PT). Please
> respond
> > >>> with suggested topics. We will also ask for additional topics at the
> > >>> beginning of the hangout.
> > >>>
> > >>> See you then,
> > >>>
> > >>> - Paul
> > >>>
> > >>>
> > >>> ***Legal Disclaimer
> ***
> > >>> "This communication may contain confidential and privileged material
> > for
> > >>> the
> > >>> sole use of the intended recipient. Any unauthorized review, use or
> > >>> distribution
> > >>> by others is strictly prohibited. If you have received the message by
> > >>> mistake,
> > >>> please advise the sender by reply email and delete the message. Thank
> > >> you."
> > >>> 
> **
> > >>>
> > >>>
> > >>
> >
>

Re: Time for a 1.9 Release?

[Laurent Goujon]

Hi,

I have several JIRAs regarding metadata I worked on recently and would like
to see resolved before 1.9:

- DRILL-1268: Add unit test to C++ native client
- DRILL-4853: Update C++ protobuf source files
- DRILL-4420: C++ API for metadata access and prepared statements
- DRILL-1996: Add cancel method to Drill C++ connector

Those changes are all part of PR #602 (
https://github.com/apache/drill/pull/602) and have been reviewed by Parth.

- DRILL-4730: Update JDBC DatabaseMetaData implementation to use new
Metadata APIs
This is PR #613 (https://github.com/apache/drill/pull/613), and is waiting
for some reviews

- DRILL-4945: Report INTERVAL exact type as column type name
This is PR #618 (https://github.com/apache/drill/pull/618), and is waiting
for some reviews

- DRILL-4969: Basic implementation of display size for column metadata
This is PR #632 (https://github.com/apache/drill/pull/632), and is waiting
for some reviews

Cheers,

Laurent

On Fri, Oct 28, 2016 at 5:14 PM, Parth Chandra 
wrote:

> +1 on doing a release.
>
> I'm hoping the following get a +1 :
>
> DRILL-4800 - Improve parquet reader performance
> DRILL-3423 - Add New HTTPD format plugin
> DRILL-4858 - REPEATED_COUNT on JSON containing an array of maps
>
> Specifically what did you want to discuss about the release number after
> 1.9?  Ordinarily you would just go to 2.0. The only reason for holding off
> on 2.0, that I can think of, is if you want to make breaking changes in the
> 2.0 release and those are not going to be ready for the next release cycle.
> Are any dev's planning on such breaking changes? If so we should discuss
> that (or any other reason we might have for deferring 2.0) in a separate
> thread?
> I'm +0 on any version number we chose.
>
>
>
> On Fri, Oct 28, 2016 at 4:53 PM, Sudheesh Katkam 
> wrote:
>
> > Let's aim for EOD next Friday (11/04/16) to get all changes in; I will
> try
> > to get RC0 out on Monday (11/07/16).
> >
> > Current list of commits:
> >
> > [Sudheesh]
> > + DRILL-4280: pull request being reviewed
> > https://github.com/apache/drill/pull/578
> >
> > [Jinfeng]
> > + DRILL-1950: pull request pending
> >
> > Any other pull requests that developers would like to get into the
> release?
> > Please post the status too.
> >
> > Thank you,
> > Sudheesh
> >
> > On Fri, Oct 28, 2016 at 11:35 AM, Jinfeng Ni  wrote:
> >
> > > +1
> > >
> > > I'm working on DRILL-1950 to support parquet row group level filter
> > > pruning. I plan to submit a pull request for code review in 1-2 days,
> > > hopefully.
> > >
> > >
> > >
> > > On Fri, Oct 28, 2016 at 11:04 AM, Aman Sinha 
> > wrote:
> > > > +1
> > > >
> > > > On Fri, Oct 28, 2016 at 10:34 AM, Sudheesh Katkam <
> sudhe...@apache.org
> > >
> > > > wrote:
> > > >
> > > >> Hi Drillers,
> > > >>
> > > >> We have a reasonable number of fixes and features since the last
> > release
> > > >> [1]. Releasing itself takes a while; so I propose we start the 1.9
> > > release
> > > >> process.
> > > >>
> > > >> I volunteer as the release manager, unless there are objections.
> > > >>
> > > >> We should also discuss what the release version number should be
> after
> > > 1.9.
> > > >>
> > > >> Thank you,
> > > >> Sudheesh
> > > >>
> > > >> [1] https://issues.apache.org/jira/browse/DRILL/
> > fixforversion/12337861
> > > >>
> > >
> >
>

[jira] [Created] (DRILL-4969) Have basic implementation for displaySize

[Laurent Goujon (JIRA)]

Laurent Goujon created DRILL-4969:
-

 Summary: Have basic implementation for displaySize
 Key: DRILL-4969
 URL: https://issues.apache.org/jira/browse/DRILL-4969
 Project: Apache Drill
  Issue Type: Sub-task
  Components: Metadata
Reporter: Laurent Goujon


display size is fixed to 10, but for most types display size is well defined as 
shown in the ODBC table:
https://msdn.microsoft.com/en-us/library/ms713974(v=vs.85).aspx



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (DRILL-4968) Add column size information to ColumnMetadata

[Laurent Goujon (JIRA)]

Laurent Goujon created DRILL-4968:
-

 Summary: Add column size information to ColumnMetadata
 Key: DRILL-4968
 URL: https://issues.apache.org/jira/browse/DRILL-4968
 Project: Apache Drill
  Issue Type: Bug
  Components: Metadata
Reporter: Laurent Goujon
Assignee: Laurent Goujon


Both ODBC and JDBC needs column size information for the column metadata. 
Instead of duplicating the logic between C++ and Java (and having to keep in 
them sync), column size should be computed on the server so that value is kept 
consistent across clients.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (DRILL-4945) Missing subtype information in metadata returned by prepared statement

[Laurent Goujon (JIRA)]

Laurent Goujon created DRILL-4945:
-

 Summary: Missing subtype information in metadata returned by 
prepared statement
 Key: DRILL-4945
 URL: https://issues.apache.org/jira/browse/DRILL-4945
 Project: Apache Drill
  Issue Type: Bug
Reporter: Laurent Goujon
Priority: Minor


Column metadata returned by prepared statement contains partial type 
information, especially for interval types. 

Currently it only returns "INTERVAL" instead of a more precise type like 
"INTERVAL MONTH" for example.

There's also no minor type, so the client can adjust the type itself.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [HANGOUT] Topics for 10/04/16

[Laurent Goujon]

d prepare apis the server exposes. It provides an improved BI user
> > > > experience. It also introduces unit tests in the C++ client,
> something
> > > that
> > > > was previously sorely missing.
> > > >
> > > >
> > > >
> > > > --
> > > > Jacques Nadeau
> > > > CTO and Co-Founder, Dremio
> > > >
> > > > On Tue, Oct 4, 2016 at 9:47 AM, Parth Chandra  >
> > > > wrote:
> > > >
> > > > > Hi guys,
> > > > >
> > > > >   I won't be able to join the hangout but it would be good to
> discuss
> > > the
> > > > > plan for the related backend changes.
> > > > >
> > > > >   As I mentioned before I would like to see a concrete proposal for
> > the
> > > > > backend that will accompany these changes. Without that, I feel
> there
> > > is
> > > > no
> > > > > point to adding so much new code.
> > > > >
> > > > > Thanks
> > > > >
> > > > > Parth
> > > > >
> > > > >
> > > > > On Mon, Oct 3, 2016 at 7:52 PM, Laurent Goujon  >
> > > > wrote:
> > > > >
> > > > > > Hi,
> > > > > >
> > > > > > I'm currently working on improving metadata support for both the
> > JDBC
> > > > > > driver and the C++ connector, more specifically the following
> > JIRAs:
> > > > > >
> > > > > > DRILL-4853: Update C++ protobuf source files
> > > > > > DRILL-4420: Server-side metadata and prepared-statement support
> for
> > > C++
> > > > > > connector
> > > > > > DRILL-4880: Support JDBC driver registration using ServiceLoader
> > > > > > DRILL-4925: Add tableType filter to GetTables metadata query
> > > > > > DRILL-4730: Update JDBC DatabaseMetaData implementation to use
> new
> > > > > Metadata
> > > > > > APIs
> > > > > >
> > > > > > I  already opened multiple pull requests for those (the list is
> > > > available
> > > > > > at https://github.com/apache/drill/pulls/laurentgo)
> > > > > >
> > > > > > I'm planning to join tomorrow hangout in case people have
> questions
> > > > about
> > > > > > those.
> > > > > >
> > > > > > Cheers,
> > > > > >
> > > > > > Laurent
> > > > > >
> > > > > > On Mon, Oct 3, 2016 at 10:28 AM, Subbu Srinivasan <
> > > > > ssriniva...@zscaler.com
> > > > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Can we close on https://github.com/apache/drill/pull/518 ?
> > > > > > >
> > > > > > > On Mon, Oct 3, 2016 at 10:27 AM, Sudheesh Katkam <
> > > > sudhe...@apache.org>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hi drillers,
> > > > > > > >
> > > > > > > > Our bi-weekly hangout is tomorrow (10/04/16, 10 AM PT). If
> you
> > > have
> > > > > any
> > > > > > > > suggestions for hangout topics, you can add them to this
> > thread.
> > > We
> > > > > > will
> > > > > > > > also ask around at the beginning of the hangout for topics.
> > > > > > > >
> > > > > > > > Thank you,
> > > > > > > > Sudheesh
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

[jira] [Created] (DRILL-4930) Metadata results are not sorted

[Laurent Goujon (JIRA)]

Laurent Goujon created DRILL-4930:
-

 Summary: Metadata results are not sorted
 Key: DRILL-4930
 URL: https://issues.apache.org/jira/browse/DRILL-4930
 Project: Apache Drill
  Issue Type: Bug
  Components: Metadata
Reporter: Laurent Goujon
Priority: Minor


According to JDBC and ODBC specs, metadata results should be ordered. 
Currently, results are unordered.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [HANGOUT] Topics for 10/04/16

[Laurent Goujon]

Hi,

I'm currently working on improving metadata support for both the JDBC
driver and the C++ connector, more specifically the following JIRAs:

DRILL-4853: Update C++ protobuf source files
DRILL-4420: Server-side metadata and prepared-statement support for C++
connector
DRILL-4880: Support JDBC driver registration using ServiceLoader
DRILL-4925: Add tableType filter to GetTables metadata query
DRILL-4730: Update JDBC DatabaseMetaData implementation to use new Metadata
APIs

I  already opened multiple pull requests for those (the list is available
at https://github.com/apache/drill/pulls/laurentgo)

I'm planning to join tomorrow hangout in case people have questions about
those.

Cheers,

Laurent

On Mon, Oct 3, 2016 at 10:28 AM, Subbu Srinivasan 
wrote:

> Can we close on https://github.com/apache/drill/pull/518 ?
>
> On Mon, Oct 3, 2016 at 10:27 AM, Sudheesh Katkam 
> wrote:
>
> > Hi drillers,
> >
> > Our bi-weekly hangout is tomorrow (10/04/16, 10 AM PT). If you have any
> > suggestions for hangout topics, you can add them to this thread. We will
> > also ask around at the beginning of the hangout for topics.
> >
> > Thank you,
> > Sudheesh
> >
>

[jira] [Created] (DRILL-4925) Add types filter to getTables metadata API

[Laurent Goujon (JIRA)]

Laurent Goujon created DRILL-4925:
-

 Summary: Add types filter to getTables metadata API
 Key: DRILL-4925
 URL: https://issues.apache.org/jira/browse/DRILL-4925
 Project: Apache Drill
  Issue Type: Bug
  Components: Metadata
Reporter: Laurent Goujon
Priority: Minor


Both ODBC and JDBC API have a types parameter to filter tables based on their 
types.

Metadata API should support it too to avoid connectors to have to do extra 
filtering on client-side.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Load JDBC service ServiceLoader

[Laurent Goujon]

That sounds like a good idea. The JDBC example has a
Class.forName("org.apache.drill.jdbc.Driver") statement to force driver
registration, but a ServiceLoader registration is pretty standard. Maybe
you should open a bug for it?

Laurent

On Tue, Sep 6, 2016 at 9:53 AM, Sudip Mukherjee 
wrote:

> Hi,
> I am trying out a java program where I want to load drill JDBC driver
> using ServiceLoader.load like all other jdbc drivers. But the code is
> failing to load org.apache.drill.jdbc.Driver class. One possible reason
> Is that the drill jdbc driver jar doesn't have a java.sql.Driver file
> under META_INF/services which is common for all the other driver jars.
>
> Can the drill jdbc driver jar come with the driver file?
>
> Thanks,
> Sudip
>

Re: WIP for prepared statement/metadata querying in C++ native client

[Laurent Goujon]

I'm  currently focusing on the client work, and making sure the C++ client
is not lagging behind the Java one. I personally haven't worked on backend
changes for prepared statements.

Laurent

On Mon, Aug 22, 2016 at 7:32 PM, Parth Chandra  wrote:

> Hi Laurent,
>
>   I'll take a look at this in the next few days.
>
>   On a related note, do you or Venki have a proposal for the backend
> changes (i.e actual implementation of prepare)? It would be a good idea to
> start a discussion on that.
>
> Parth
>
> On Mon, Aug 22, 2016 at 3:24 PM, Laurent Goujon 
> wrote:
>
> > Hi,
> >
> > I just started working on adding support for prepared statements and
> > metadata querying in the C++ Drill client. Hopefully, nobody else has
> > started working on this (The Drill jiras don't mention any activity on
> > this), but if it is not the case, let me know.
> >
> > My working branch is
> > https://github.com/laurentgo/drill/tree/laurent/improve-native-client.
> >
> > For now, I just have a basic interface API (
> > https://github.com/laurentgo/drill/commit/1f55a3e631cd97016b113b9d4bca07
> > b5e016a25e),
> > and it would be nice if people knowledgeable about the C++ client could
> > review it and give me some feedback. I'm also adding an actual initial
> > implementation in the coming days.
> >
> > Cheers,
> >
> > Laurent
> >
>

WIP for prepared statement/metadata querying in C++ native client

[Laurent Goujon]

Hi,

I just started working on adding support for prepared statements and
metadata querying in the C++ Drill client. Hopefully, nobody else has
started working on this (The Drill jiras don't mention any activity on
this), but if it is not the case, let me know.

My working branch is
https://github.com/laurentgo/drill/tree/laurent/improve-native-client.

For now, I just have a basic interface API (
https://github.com/laurentgo/drill/commit/1f55a3e631cd97016b113b9d4bca07b5e016a25e),
and it would be nice if people knowledgeable about the C++ client could
review it and give me some feedback. I'm also adding an actual initial
implementation in the coming days.

Cheers,

Laurent

Re: C++ protobuf files

[Laurent Goujon]

I just opened DRILL-4853 jira and PR
https://github.com/apache/drill/pull/571 for the protobuf source files
update.

Laurent

On Thu, Aug 18, 2016 at 9:49 AM, Laurent Goujon  wrote:

> Thanks Sékine,
>
> After fiddling around and with a little help from Venki (who told me about
> the CMakeList.txt files under src/protobuf), I was able to refresh the
> files using the following command in the build directory:
> $ make fixProtobufs cpProtobufs
>
> This should copy the refreshed files over to the c++ src directory.
>
> Laurent
>
> On Thu, Aug 18, 2016 at 1:01 AM, Sékine Coulibaly 
> wrote:
>
>> Laurent,
>>
>> I used this on  1.7 branch (commit 7800aebdca845466a8d912f0f59c00
>> cda093b01f)
>>  :
>>
>> PYTHONPATH="/home/scoulibaly/python/build/lib.linux-x86_64-2.7"
>> /home/scoulibaly/protobuf/src/protoc  --cpp_out="." --proto_path="."
>> *.proto
>>
>> Just update to your actual Python path and to your actual protoc binary
>> path.
>>
>> The output on my machine is as follow :
>> [libprotobuf WARNING google/protobuf/compiler/parser.cc:547] No syntax
>> specified for the proto file: BitControl.proto. Please use 'syntax =
>> "proto2";' or 'syntax = "proto3";' to specify a syntax version. (Defaulted
>> to proto2 syntax.)
>> [libprotobuf WARNING google/protobuf/compiler/parser.cc:547] No syntax
>> specified for the proto file: ExecutionProtos.proto. Please use 'syntax =
>> "proto2";' or 'syntax = "proto3";' to specify a syntax version. (Defaulted
>> to proto2 syntax.)
>> [libprotobuf WARNING google/protobuf/compiler/parser.cc:547] No syntax
>> specified for the proto file: Coordination.proto. Please use 'syntax =
>> "proto2";' or 'syntax = "proto3";' to specify a syntax version. (Defaulted
>> to proto2 syntax.)
>> [libprotobuf WARNING google/protobuf/compiler/parser.cc:547] No syntax
>> specified for the proto file: UserBitShared.proto. Please use 'syntax =
>> "proto2";' or 'syntax = "proto3";' to specify a syntax version. (Defaulted
>> to proto2 syntax.)
>> [libprotobuf WARNING google/protobuf/compiler/parser.cc:547] No syntax
>> specified for the proto file: Types.proto. Please use 'syntax = "proto2";'
>> or 'syntax = "proto3";' to specify a syntax version. (Defaulted to proto2
>> syntax.)
>> [libprotobuf WARNING google/protobuf/compiler/parser.cc:547] No syntax
>> specified for the proto file: SchemaDef.proto. Please use 'syntax =
>> "proto2";' or 'syntax = "proto3";' to specify a syntax version. (Defaulted
>> to proto2 syntax.)
>> [libprotobuf WARNING google/protobuf/compiler/parser.cc:547] No syntax
>> specified for the proto file: BitData.proto. Please use 'syntax =
>> "proto2";' or 'syntax = "proto3";' to specify a syntax version. (Defaulted
>> to proto2 syntax.)
>> BitData.proto: warning: Import Coordination.proto but not used.
>> BitData.proto: warning: Import ExecutionProtos.proto but not used.
>> [libprotobuf WARNING google/protobuf/compiler/parser.cc:547] No syntax
>> specified for the proto file: GeneralRPC.proto. Please use 'syntax =
>> "proto2";' or 'syntax = "proto3";' to specify a syntax version. (Defaulted
>> to proto2 syntax.)
>> GeneralRPC.proto: warning: Import Coordination.proto but not used.
>> [libprotobuf WARNING google/protobuf/compiler/parser.cc:547] No syntax
>> specified for the proto file: User.proto. Please use 'syntax = "proto2";'
>> or 'syntax = "proto3";' to specify a syntax version. (Defaulted to proto2
>> syntax.)
>> User.proto: warning: Import BitData.proto but not used.
>> User.proto: warning: Import ExecutionProtos.proto but not used.
>> User.proto: warning: Import SchemaDef.proto but not used.
>>
>> Hope this will help you start.
>>
>>
>> 2016-08-17 20:01 GMT+02:00 Laurent Goujon :
>>
>> > Hi,
>> >
>> > There's no instructions on how to generate the C++ protobuf files, and
>> they
>> > are currently out-of-sync with the definitions stored under protocol
>> > module.
>> >
>> > Does someone know how these files are generated? I'd like to update them
>> > and add some instructions at the same time.
>> >
>> > Cheers,
>> >
>> > Laurent
>> >
>>
>
>

[jira] [Created] (DRILL-4853) Update C++ protobuf source files

[Laurent Goujon (JIRA)]

Laurent Goujon created DRILL-4853:
-

 Summary: Update C++ protobuf source files
 Key: DRILL-4853
 URL: https://issues.apache.org/jira/browse/DRILL-4853
 Project: Apache Drill
  Issue Type: Task
  Components: Client - C++
Reporter: Laurent Goujon
Assignee: Laurent Goujon


C++ protobuf files have not been updated since january, missing changes for 
prepared statement and metadata querying support



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: C++ protobuf files

[Laurent Goujon]

Thanks Sékine,

After fiddling around and with a little help from Venki (who told me about
the CMakeList.txt files under src/protobuf), I was able to refresh the
files using the following command in the build directory:
$ make fixProtobufs cpProtobufs

This should copy the refreshed files over to the c++ src directory.

Laurent

On Thu, Aug 18, 2016 at 1:01 AM, Sékine Coulibaly 
wrote:

> Laurent,
>
> I used this on  1.7 branch (commit 7800aebdca845466a8d912f0f59c00
> cda093b01f)
>  :
>
> PYTHONPATH="/home/scoulibaly/python/build/lib.linux-x86_64-2.7"
> /home/scoulibaly/protobuf/src/protoc  --cpp_out="." --proto_path="."
> *.proto
>
> Just update to your actual Python path and to your actual protoc binary
> path.
>
> The output on my machine is as follow :
> [libprotobuf WARNING google/protobuf/compiler/parser.cc:547] No syntax
> specified for the proto file: BitControl.proto. Please use 'syntax =
> "proto2";' or 'syntax = "proto3";' to specify a syntax version. (Defaulted
> to proto2 syntax.)
> [libprotobuf WARNING google/protobuf/compiler/parser.cc:547] No syntax
> specified for the proto file: ExecutionProtos.proto. Please use 'syntax =
> "proto2";' or 'syntax = "proto3";' to specify a syntax version. (Defaulted
> to proto2 syntax.)
> [libprotobuf WARNING google/protobuf/compiler/parser.cc:547] No syntax
> specified for the proto file: Coordination.proto. Please use 'syntax =
> "proto2";' or 'syntax = "proto3";' to specify a syntax version. (Defaulted
> to proto2 syntax.)
> [libprotobuf WARNING google/protobuf/compiler/parser.cc:547] No syntax
> specified for the proto file: UserBitShared.proto. Please use 'syntax =
> "proto2";' or 'syntax = "proto3";' to specify a syntax version. (Defaulted
> to proto2 syntax.)
> [libprotobuf WARNING google/protobuf/compiler/parser.cc:547] No syntax
> specified for the proto file: Types.proto. Please use 'syntax = "proto2";'
> or 'syntax = "proto3";' to specify a syntax version. (Defaulted to proto2
> syntax.)
> [libprotobuf WARNING google/protobuf/compiler/parser.cc:547] No syntax
> specified for the proto file: SchemaDef.proto. Please use 'syntax =
> "proto2";' or 'syntax = "proto3";' to specify a syntax version. (Defaulted
> to proto2 syntax.)
> [libprotobuf WARNING google/protobuf/compiler/parser.cc:547] No syntax
> specified for the proto file: BitData.proto. Please use 'syntax =
> "proto2";' or 'syntax = "proto3";' to specify a syntax version. (Defaulted
> to proto2 syntax.)
> BitData.proto: warning: Import Coordination.proto but not used.
> BitData.proto: warning: Import ExecutionProtos.proto but not used.
> [libprotobuf WARNING google/protobuf/compiler/parser.cc:547] No syntax
> specified for the proto file: GeneralRPC.proto. Please use 'syntax =
> "proto2";' or 'syntax = "proto3";' to specify a syntax version. (Defaulted
> to proto2 syntax.)
> GeneralRPC.proto: warning: Import Coordination.proto but not used.
> [libprotobuf WARNING google/protobuf/compiler/parser.cc:547] No syntax
> specified for the proto file: User.proto. Please use 'syntax = "proto2";'
> or 'syntax = "proto3";' to specify a syntax version. (Defaulted to proto2
> syntax.)
> User.proto: warning: Import BitData.proto but not used.
> User.proto: warning: Import ExecutionProtos.proto but not used.
> User.proto: warning: Import SchemaDef.proto but not used.
>
> Hope this will help you start.
>
>
> 2016-08-17 20:01 GMT+02:00 Laurent Goujon :
>
> > Hi,
> >
> > There's no instructions on how to generate the C++ protobuf files, and
> they
> > are currently out-of-sync with the definitions stored under protocol
> > module.
> >
> > Does someone know how these files are generated? I'd like to update them
> > and add some instructions at the same time.
> >
> > Cheers,
> >
> > Laurent
> >
>

C++ protobuf files

[Laurent Goujon]

Hi,

There's no instructions on how to generate the C++ protobuf files, and they
are currently out-of-sync with the definitions stored under protocol module.

Does someone know how these files are generated? I'd like to update them
and add some instructions at the same time.

Cheers,

Laurent

Re: Including Drill and Hadoop jars in the same program

[Laurent Goujon]

I also believe Hadoop have some optional classloader to isolate hadoop
internal classpath from application classpath, like that guy:
https://hadoop.apache.org/docs/r2.7.2/api/org/apache/hadoop/util/ApplicationClassLoader.html

Laurent

On Wed, Apr 6, 2016 at 12:34 PM, Hanifi Gunes  wrote:

> Shading might be an option here.
>
> Thanks.
> -Hanifi
>
> 1: https://maven.apache.org/plugins/maven-shade-plugin/
>
>
> On Thu, Mar 31, 2016 at 6:04 PM, Paul Rogers  wrote:
>
> > Hi All,
> >
> > Here’s an obscure question for the expert developers…
> >
> > We’re developing the YARN Application Master (AM) for Drill. We’d like to
> > monitor Drill’s ZooKeeper (ZK) Drill-bit registrations. Since the ZK
> > entries are in Protobuf format, and Drill already has classes do to the
> > monitoring, the logical solution is just to use the Drill code in the AM.
> >
> > The problem is, the dependencies (such as Guava version) for Drill differ
> > from those of Hadoop. Simply including Drill and YARN libraries in the
> same
> > build trigger runtime failures due to Guava version incompatibilities.
> >
> > One solution is to create a custom class loader for the Drill classes,
> but
> > that introduces a different set of complexities.
> >
> > Suggestions?
> >
> > Thanks,
> >
> > - Paul
> >
> >
>