[GitHub] [drill] sohami commented on issue #1874: DRILL-7405: Avoiding download of TPC-H data

[GitBox]

sohami commented on issue #1874: DRILL-7405: Avoiding download of TPC-H data
URL: https://github.com/apache/drill/pull/1874#issuecomment-544024689
 
 
   Looks like these files were packaged as jar in Drill class path as an 
example data for users to run some exploratory queries. I think putting these 
files as part of source repo should be fine.
   
   @vvysotskyi : I think your main concern is related to the unit tests data 
files which are merged with the source files. I guess that was done to keep the 
test execution time lower otherwise ideally unit tests should use the in-memory 
data generator for it's use. May be we should come up with some policies which 
can dictate when is it fine to check in the test data file and when one should 
use in-memory data generator. 
   Also how does moving data files to a separate git repo will help here ?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Created] (DRILL-7411) DRILL 1.16

[Sorabh Hamirwasia (Jira)]

Sorabh Hamirwasia created DRILL-7411:


 Summary: DRILL 1.16
 Key: DRILL-7411
 URL: https://issues.apache.org/jira/browse/DRILL-7411
 Project: Apache Drill
  Issue Type: Sub-task
Reporter: Sorabh Hamirwasia


Design document for following features are added in this JIRA:

 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (DRILL-7410) Design Documents

[Sorabh Hamirwasia (Jira)]

Sorabh Hamirwasia created DRILL-7410:


 Summary: Design Documents
 Key: DRILL-7410
 URL: https://issues.apache.org/jira/browse/DRILL-7410
 Project: Apache Drill
  Issue Type: Task
Reporter: Sorabh Hamirwasia


This Jira is created to track the design documents available for all the 
features developed in Apache Drill. It serves as an index for easy access of 
these document for future reference.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [drill] Agirish commented on issue #1874: DRILL-7405: Avoiding download of TPC-H data

[GitBox]

Agirish commented on issue #1874: DRILL-7405: Avoiding download of TPC-H data
URL: https://github.com/apache/drill/pull/1874#issuecomment-543879126
 
 
   @vvysotskyi , I understand but I am not for using personal GitHub 
repositories. I don't think it's the best approach for an Apache project. The 
files are small enough ~3 MB. I'm not sure I agree that this would constitute 
as large. I think what we have here would be more straightforward.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

Re: Next Release?

[Charles Givre]

Hi Volodymyr, 
I'd like to get DRILL-7177 (https://github.com/apache/drill/pull/1749 
) into the next release, which is 
awaiting final review.  As a stretch goal, I'd like to get DRILL-4303 in as 
well, which shouldn't be too difficult either but is awaiting review. 

Is there any progress on DRILL-6540?  That does seem like an extremely 
important PR as well. 
-- C

> On Oct 18, 2019, at 12:27 PM, Volodymyr Vysotskyi  
> wrote:
> 
> Hi all,
> 
> I'm going to open PR for DRILL-7273
>  on the next week, it is
> the last thing blocking the release, sorry for one more delay.
> 
> I'm planning to start the release process preliminary on October, 28, so
> please let me know if there is something else that should be included in
> the release and make sure that features you are working on and want to
> include in the release are finished before that time.
> 
> Kind regards,
> Volodymyr Vysotskyi
> 
> 
> On Wed, Sep 25, 2019 at 11:30 AM Volodymyr Vysotskyi 
> wrote:
> 
>> Thanks for moving this topic forward, yes, I think mid-October is
>> achievable date.
>> I'll start a pre-release discussion when will wrap up major things for
>> metastore.
>> 
>> Kind regards,
>> Volodymyr Vysotskyi
>> 
>> 
>> On Mon, Sep 23, 2019 at 4:46 PM Charles Givre  wrote:
>> 
>>> That sounds good to me.  It seems to me that there are several PRs which
>>> are relatively simple and could be cleared off the board as well.
>>> -- C
>>> 
>>> 
 On Sep 23, 2019, at 8:41 AM, Arina Yelchiyeva <
>>> arina.yelchiy...@gmail.com> wrote:
 
 Metastore work was aimed to be included in this release, since delivery
>>> date was shifted due to larger scope of work than expected, we did not push
>>> for the release until it’s done but I think mid October is achievable due
>>> date. Volodymyr any thoughts?
 
 Kind regards,
 Arina
 
> On 23 Sep 2019, at 15:09, Charles Givre  wrote:
> 
> Hello All,
> I wanted to ask if we can start thinking about our next release?  I
>>> seem to recall that there was discussion around a new release around
>>> mid-September which clearly didn't happen. So... What if we were to shoot
>>> for mid-October?
> -- C
>>> 
>>> 

Re: Dependencies used by Drill contain known vulnerabilities

[Charles Givre]

Good question.  I'd suggest creating one, and if issues arise in the process, 
create a separate issue for that.  But it seems excessive to create separate 
issues for each update especially if all you are doing is updating a pom file.  
 That's just my .02.
-- C

> On Oct 18, 2019, at 11:59 AM, Bradley Parker  wrote:
> 
> Thank you Charles, will do. Is it more appropriate to open one JIRA for all 
> the packages or to break each package into a separate issue?
> 
> -Brad
> 
> -Charles Givre  wrote: -
> To: dev@drill.apache.org
> From: Charles Givre 
> Date: 10/17/2019 04:57PM
> Cc: Glen Bizeau , Sean Peppard 
> 
> Subject: [EXTERNAL] Re: Dependencies used by Drill contain known 
> vulnerabilities
> 
> HI Brad,
> Thanks for your interest in Drill.  Can you please create a JIRA 
> (issues.apache.org 
>   >) and start the discussion.  
> Thanks,
> -- C
> 
>> On Oct 17, 2019, at 3:40 PM, Bradley Parker  wrote:
>> 
>> Hello Apache Drill Devs,
>> 
>> We are looking to make use of Apache Drill for a project, as a member of our 
>> product security team I was asked to perform a dependency analysis of Drill. 
>> I identified 24 dependencies with known vulnerabilities using OWASP 
>> Dependency Scan.
>> 
>> 
>> I found this in the archives from two years ago 
>> https://urldefense.proofpoint.com/v2/url?u=http-3A__mail-2Darchives.apache.org_mod-5Fmbox_drill-2Ddev_201709.mbox_-253Cb4df2a35-2D121c-2D11a5-2Da666-2D4af7bd98b1db-40apache.org-253E&d=DwIFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=hsBDWgqUN16IByrh81JE1VQ3XJqGyuoBQmD8uAV4Rng&m=H9jTPsQZwIWD4ceIRB0dLwxapVuh3uL9ZJZE6101xLg&s=IqIyU0LrcQNtu_EpyatX56_ZawBUPxgU1my6Y721l48&e=
>>   
>> discussing the potential of integrating OWASP into the project.
>> 
>> 
>> Aside from Kafka [DRILL-6739] and Avro [DRILL-7302] I was unable to find 
>> mention
>> in Jira of updates to the remaining 22 libraries. Is it reasonable to assume 
>> there is no plan to upgrade at this time then?
>> 
>> 
>> I’m more than willing to step up and raise these and future dependency 
>> vulnerabilities I am aware of in Jira to get the discussions started. 
>> I think that is a good place to raise these security issues, and from there 
>> the 
>> community can discuss upgrading the affected dependencies, or rule them out 
>> as 
>> not applicable.
>> 
>> 
>> Thank you for your time,
>> -Brad
>> 
>> 
>> 
>> For reference, the list of vulnerabilities identified by the OWASP tool:
>> 
>> 
>> Package: avro-1.8.2
>> Should be: 1.9.0
>> Max CVE (CVSS): CVE-2018-10237 (5.9)
>> Complete CVE list: 
>> CVE-2018-10237
>> 
>> 
>> Package: commons-beanutils-1.9.2
>> Should be: 1.9.4
>> Max CVE (CVSS): CVE-2019-10086 (7.3)
>> Complete CVE list: 
>> CVE-2019-10086
>> 
>> 
>> Package: commons-beanutils-core-1.8.0
>> Should be: Moved to commons-beanutils
>> Max CVE (CVSS): CVE-2014-0114 (7.5)
>> Complete CVE list: 
>> CVE-2014-0114
>> 
>> 
>> Package: converter-jackson
>> Should be: 2.5.0
>> Max CVE (CVSS): CVE-2018-1000850 (7.5)
>> Complete CVE list: 
>> CVE-2018-1000850
>> 
>> 
>> Package: derby-10.10.2.0
>> Should be: 10.14.2.0
>> Max CVE (CVSS): CVE-2015-1832 (9.1)
>> Complete CVE list: 
>> CVE-2015-1832
>> CVE-2018-1313
>> 
>> 
>> Package: drill-hive-exec-shaded
>> Should be: New release needed with updated Guava
>> Max CVE (CVSS): CVE-2018-10237 (7.5)
>> Complete CVE list: 
>> CVE-2018-10237
>> 
>> 
>> Package: drill-java-exec
>> Should be: New release needed with updated JjQuery and Bootstrap
>> Max CVE (CVSS): CVE-2019-11358 (6.1)
>> Complete CVE list: 
>> CVE-2018-14040
>> CVE-2018-14041 
>> CVE-2018-14042
>> CVE-2019-8331
>> CVE-2019-11358
>> 
>> 
>> Package: drill-shaded-guava-23
>> Should be: New release needed with updated Guava
>> Max CVE (CVSS): CVE-2018-10237 (5.9)
>> Complete CVE list: 
>> CVE-2018-10237
>> 
>> 
>> Package: guava-19.0
>> Should be: 24.1.1
>> Max CVE (CVSS): CVE-2018-10237 (5.9)
>> Complete CVE list: 
>> CVE-2018-10237
>> 
>> 
>> Package: hadoop-yarn-common-2.7.4
>> Should be: 3.2.1
>> Max CVE (CVSS): CVE-2019-11358 (6.1)
>> Complete CVE list: 
>> CVE-2012-6708
>> CVE-2015-9251
>> CVE-2019-11358
>> CVE-2010-5312
>> CVE-2016-7103
>> 
>> 
>> Package: hbase-http-2.1.1.jar 
>> Should be: 2.1.4
>> Max CVE (CVSS): CVE-2019-0212 (7.5)
>> Complete CVE list: 
>> CVE-2019-0212
>> 
>> 
>> Package: httpclient-4.2.5.jar
>> Should be:  4.3.6
>> Max CVE (CVSS): CVE-2014-3577  (5.8)
>> Complete CVE list: 
>> CVE-2014-3577
>> CVE-2015-5262
>> 
>> 
>> Package: jackson-databind-2.9.5
>> Should be: 2.10.0
>> Max CVE (CVSS): CVE-2018-14721  (10)
>> Complete CVE list: 
>> CVE-2019-17267
>> CVE-2019-16943
>> CVE-2019-16942
>> CVE-2019-16335
>> CVE-2019-14540
>> CVE-2019-14439
>> CVE-2019-14379
>> CVE-2018-11307
>> CVE-2019-12384
>> CVE-2019-12814
>> CVE-2019-12086
>> C

Re: Next Release?

[Volodymyr Vysotskyi]

Hi all,

I'm going to open PR for DRILL-7273
 on the next week, it is
the last thing blocking the release, sorry for one more delay.

I'm planning to start the release process preliminary on October, 28, so
please let me know if there is something else that should be included in
the release and make sure that features you are working on and want to
include in the release are finished before that time.

Kind regards,
Volodymyr Vysotskyi


On Wed, Sep 25, 2019 at 11:30 AM Volodymyr Vysotskyi 
wrote:

> Thanks for moving this topic forward, yes, I think mid-October is
> achievable date.
> I'll start a pre-release discussion when will wrap up major things for
> metastore.
>
> Kind regards,
> Volodymyr Vysotskyi
>
>
> On Mon, Sep 23, 2019 at 4:46 PM Charles Givre  wrote:
>
>> That sounds good to me.  It seems to me that there are several PRs which
>> are relatively simple and could be cleared off the board as well.
>> -- C
>>
>>
>> > On Sep 23, 2019, at 8:41 AM, Arina Yelchiyeva <
>> arina.yelchiy...@gmail.com> wrote:
>> >
>> > Metastore work was aimed to be included in this release, since delivery
>> date was shifted due to larger scope of work than expected, we did not push
>> for the release until it’s done but I think mid October is achievable due
>> date. Volodymyr any thoughts?
>> >
>> > Kind regards,
>> > Arina
>> >
>> >> On 23 Sep 2019, at 15:09, Charles Givre  wrote:
>> >>
>> >> Hello All,
>> >> I wanted to ask if we can start thinking about our next release?  I
>> seem to recall that there was discussion around a new release around
>> mid-September which clearly didn't happen. So... What if we were to shoot
>> for mid-October?
>> >> -- C
>>
>>

RE: Dependencies used by Drill contain known vulnerabilities

[Bradley Parker]

Thank you Charles, will do. Is it more appropriate to open one JIRA for all the 
packages or to break each package into a separate issue?
 
-Brad

-Charles Givre  wrote: -
To: dev@drill.apache.org
From: Charles Givre 
Date: 10/17/2019 04:57PM
Cc: Glen Bizeau , Sean Peppard 
Subject: [EXTERNAL] Re: Dependencies used by Drill contain known vulnerabilities

HI Brad,
Thanks for your interest in Drill.  Can you please create a JIRA 
(issues.apache.org 
) and start the discussion.  
Thanks,
-- C

> On Oct 17, 2019, at 3:40 PM, Bradley Parker  wrote:
> 
> Hello Apache Drill Devs,
> 
> We are looking to make use of Apache Drill for a project, as a member of our 
> product security team I was asked to perform a dependency analysis of Drill. 
> I identified 24 dependencies with known vulnerabilities using OWASP 
> Dependency Scan.
> 
> 
> I found this in the archives from two years ago 
> https://urldefense.proofpoint.com/v2/url?u=http-3A__mail-2Darchives.apache.org_mod-5Fmbox_drill-2Ddev_201709.mbox_-253Cb4df2a35-2D121c-2D11a5-2Da666-2D4af7bd98b1db-40apache.org-253E&d=DwIFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=hsBDWgqUN16IByrh81JE1VQ3XJqGyuoBQmD8uAV4Rng&m=H9jTPsQZwIWD4ceIRB0dLwxapVuh3uL9ZJZE6101xLg&s=IqIyU0LrcQNtu_EpyatX56_ZawBUPxgU1my6Y721l48&e=
>   
> discussing the potential of integrating OWASP into the project.
> 
> 
> Aside from Kafka [DRILL-6739] and Avro [DRILL-7302] I was unable to find 
> mention
> in Jira of updates to the remaining 22 libraries. Is it reasonable to assume 
> there is no plan to upgrade at this time then?
> 
> 
> I’m more than willing to step up and raise these and future dependency 
> vulnerabilities I am aware of in Jira to get the discussions started. 
> I think that is a good place to raise these security issues, and from there 
> the 
> community can discuss upgrading the affected dependencies, or rule them out 
> as 
> not applicable.
> 
> 
> Thank you for your time,
> -Brad
> 
> 
> 
> For reference, the list of vulnerabilities identified by the OWASP tool:
> 
> 
> Package: avro-1.8.2
> Should be: 1.9.0
> Max CVE (CVSS): CVE-2018-10237 (5.9)
> Complete CVE list: 
> CVE-2018-10237
> 
> 
> Package: commons-beanutils-1.9.2
> Should be: 1.9.4
> Max CVE (CVSS): CVE-2019-10086 (7.3)
> Complete CVE list: 
> CVE-2019-10086
> 
> 
> Package: commons-beanutils-core-1.8.0
> Should be: Moved to commons-beanutils
> Max CVE (CVSS): CVE-2014-0114 (7.5)
> Complete CVE list: 
> CVE-2014-0114
> 
> 
> Package: converter-jackson
> Should be: 2.5.0
> Max CVE (CVSS): CVE-2018-1000850 (7.5)
> Complete CVE list: 
> CVE-2018-1000850
> 
> 
> Package: derby-10.10.2.0
> Should be: 10.14.2.0
> Max CVE (CVSS): CVE-2015-1832 (9.1)
> Complete CVE list: 
> CVE-2015-1832
> CVE-2018-1313
> 
> 
> Package: drill-hive-exec-shaded
> Should be: New release needed with updated Guava
> Max CVE (CVSS): CVE-2018-10237 (7.5)
> Complete CVE list: 
> CVE-2018-10237
> 
> 
> Package: drill-java-exec
> Should be: New release needed with updated JjQuery and Bootstrap
> Max CVE (CVSS): CVE-2019-11358 (6.1)
> Complete CVE list: 
> CVE-2018-14040
> CVE-2018-14041 
> CVE-2018-14042
> CVE-2019-8331
> CVE-2019-11358
> 
> 
> Package: drill-shaded-guava-23
> Should be: New release needed with updated Guava
> Max CVE (CVSS): CVE-2018-10237 (5.9)
> Complete CVE list: 
> CVE-2018-10237
> 
> 
> Package: guava-19.0
> Should be: 24.1.1
> Max CVE (CVSS): CVE-2018-10237 (5.9)
> Complete CVE list: 
> CVE-2018-10237
> 
> 
> Package: hadoop-yarn-common-2.7.4
> Should be: 3.2.1
> Max CVE (CVSS): CVE-2019-11358 (6.1)
> Complete CVE list: 
> CVE-2012-6708
> CVE-2015-9251
> CVE-2019-11358
> CVE-2010-5312
> CVE-2016-7103
> 
> 
> Package: hbase-http-2.1.1.jar 
> Should be: 2.1.4
> Max CVE (CVSS): CVE-2019-0212 (7.5)
> Complete CVE list: 
> CVE-2019-0212
> 
> 
> Package: httpclient-4.2.5.jar
> Should be:  4.3.6
> Max CVE (CVSS): CVE-2014-3577  (5.8)
> Complete CVE list: 
> CVE-2014-3577
> CVE-2015-5262
> 
> 
> Package: jackson-databind-2.9.5
> Should be: 2.10.0
> Max CVE (CVSS): CVE-2018-14721  (10)
> Complete CVE list: 
> CVE-2019-17267
> CVE-2019-16943
> CVE-2019-16942
> CVE-2019-16335
> CVE-2019-14540
> CVE-2019-14439
> CVE-2019-14379
> CVE-2018-11307
> CVE-2019-12384
> CVE-2019-12814
> CVE-2019-12086
> CVE-2018-12023
> CVE-2018-12022
> CVE-2018-19362
> CVE-2018-19361
> CVE-2018-19360
> CVE-2018-14721
> CVE-2018-14720
> CVE-2018-14719
> CVE-2018-14718
> CVE-2018-1000873
> 
> 
> Package: Kafka 0.11.0.1
> Should be: 2.1.0
> Max CVE (CVSS): CVE-2018-17196 (8.8)
> Complete CVE list: 
> CVE-2018-17196
> CVE-2018-1288
> CVE-2017-12610
> 
> 
> Package: kudu-client-1.3.0.jar 
> Should be: 1.10.0
> Only a partial fix, no fix for netty CVE-2019-16869 (7.5), kudu still needs 
> to 
> update their netty (this is not unexpec

[GitHub] [drill] arina-ielchiieva opened a new pull request #1875: DRILL-7401: Upgrade to SqlLine 1.9.0

[GitBox]

arina-ielchiieva opened a new pull request #1875: DRILL-7401: Upgrade to 
SqlLine 1.9.0
URL: https://github.com/apache/drill/pull/1875
 
 
   Jira - [DRILL-7401](https://issues.apache.org/jira/browse/DRILL-7401).


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [drill] arina-ielchiieva commented on issue #1875: DRILL-7401: Upgrade to SqlLine 1.9.0

[GitBox]

arina-ielchiieva commented on issue #1875: DRILL-7401: Upgrade to SqlLine 1.9.0
URL: https://github.com/apache/drill/pull/1875#issuecomment-543730392
 
 
   @vvysotskyi please review.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [drill] arina-ielchiieva commented on issue #1871: DRILL-7403: Validate batch checks, vector integretity in unit tests

[GitBox]

arina-ielchiieva commented on issue #1871: DRILL-7403: Validate batch checks, 
vector integretity in unit tests
URL: https://github.com/apache/drill/pull/1871#issuecomment-543689950
 
 
   LGTM, +1


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [drill] arina-ielchiieva commented on issue #1872: DRILL-7402: Suppress batch dumps for expected failures in tests

[GitBox]

arina-ielchiieva commented on issue #1872: DRILL-7402: Suppress batch dumps for 
expected failures in tests
URL: https://github.com/apache/drill/pull/1872#issuecomment-543688813
 
 
   +1, @paul-rogers thanks for making the changes.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Created] (DRILL-7409) Remove bigIntDictionary.parquet from project sources

[Vova Vysotskyi (Jira)]

Vova Vysotskyi created DRILL-7409:
-

 Summary: Remove bigIntDictionary.parquet from project sources
 Key: DRILL-7409
 URL: https://issues.apache.org/jira/browse/DRILL-7409
 Project: Apache Drill
  Issue Type: Task
  Components: Tools, Build & Test
Reporter: Vova Vysotskyi


{{bigIntDictionary.parquet}} file has size of 1.8M, but it is used in single 
unit test {{TestColumnReaderFactory.testBigIntWithDictionary}}. We should 
either move this test to a test-framework or recreate a smaller file that will 
still allow us to verify this case.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [drill] vvysotskyi commented on issue #1874: DRILL-7405: Avoiding download of TPC-H data

[GitBox]

vvysotskyi commented on issue #1874: DRILL-7405: Avoiding download of TPC-H data
URL: https://github.com/apache/drill/pull/1874#issuecomment-543633312
 
 
   I disagree that we should add so large files to the project sources.
   
   I have experimented some time ago with alternative solutions for this 
problem and one of the ideas was to create a new GitHub repository with these 
files, use JitPack for publishing archive and use `maven-dependency-plugin` in 
Drill to obtain and unpack files when the project is built.
   
   Here is a link to the repo with files: 
https://github.com/vvysotskyi/tpch-parquet and commit with changes in Drill: 
https://github.com/vvysotskyi/drill/commit/0635133bbd22945e7648791cfd6e2d146730b219.
   What do you think about this? You may use these changes if we will choose 
this approach.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services