[jira] [Commented] (DRILL-5432) Want a memory format for PCAP files
[
https://issues.apache.org/jira/browse/DRILL-5432?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16070954#comment-16070954
]
ASF GitHub Bot commented on DRILL-5432:
---
Github user tdunning commented on the issue:
https://github.com/apache/drill/pull/831
On Fri, Jun 30, 2017 at 5:34 PM, Paul Rogers
wrote:
> That only works if Drill has an autoloading capability that allows storage
> formats to be loaded and authenticated easily.
>
> Why? To get pcap in 1.11, one must install a new Drill version, which
> requires a restart. Assuming that pcap were a separate project, you'd
> install a new jar, and restart. Neither provides auto loading.
>
No. But if I have 1.11 already and decide that I want pcap (or any similar
data format), it would be nice if I could do the equivalent of pip (for
python) or install.packages("...") (for R) or mvn test (for Java) and get
whatever cool capability I like.
This may be described as "just install a new jar" but the convenience level
is a proven Big Deal (tm). It would even be possible to leverage Maven
central to make it happen, but there needs to be convenience sugar around
the process to make it consumable.
> Not clear on the authentication issue. If a plugin were a separate
> project, wouldn't that project have a way of certifying its jars?
>
Dunno. Not some random project.
If Drill requires it, then yes. And I am saying that Drill should require
resolution back to some level of trust.
> Want a memory format for PCAP files
> ---
>
> Key: DRILL-5432
> URL: https://issues.apache.org/jira/browse/DRILL-5432
> Project: Apache Drill
> Issue Type: New Feature
>Reporter: Ted Dunning
>
> PCAP files [1] are the de facto standard for storing network capture data. In
> security and protocol applications, it is very common to want to extract
> particular packets from a capture for further analysis.
> At a first level, it is desirable to query and filter by source and
> destination IP and port or by protocol. Beyond that, however, it would be
> very useful to be able to group packets by TCP session and eventually to look
> at packet contents. For now, however, the most critical requirement is that
> we should be able to scan captures at very high speed.
> I previously wrote a (kind of working) proof of concept for a PCAP decoder
> that did lazy deserialization and could traverse hundreds of MB of PCAP data
> per second per core. This compares to roughly 2-3 MB/s for widely available
> Apache-compatible open source PCAP decoders.
> This JIRA covers the integration and extension of that proof of concept as a
> Drill file format.
> Initial work is available at https://github.com/mapr-demos/drill-pcap-format
> [1] https://en.wikipedia.org/wiki/Pcap
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (DRILL-5420) all cores at 100% of all servers
[ https://issues.apache.org/jira/browse/DRILL-5420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16070953#comment-16070953 ] ASF GitHub Bot commented on DRILL-5420: --- Github user parthchandra commented on the issue: https://github.com/apache/drill/pull/862 LGTM. > all cores at 100% of all servers > > > Key: DRILL-5420 > URL: https://issues.apache.org/jira/browse/DRILL-5420 > Project: Apache Drill > Issue Type: Bug >Affects Versions: 1.10.0 > Environment: linux, cluster with 5 servers over hdfs/parquet >Reporter: Hugo Bellomusto >Assignee: Kunal Khatua > Labels: ready-to-commit > Fix For: 1.11.0 > > Attachments: 2709a36d-804a-261a-64e5-afa271e782f8.json > > > We have a drill cluster with five servers over hdfs/parquet. > Each machine have 8 cores. All cores get at 100% of use. > Each thread is looping in the while in line 314 in AsyncPageReader.java > inside clear() method. > https://github.com/apache/drill/blob/1.10.0/exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/columnreaders/AsyncPageReader.java#L314 > jstack -l 19255|grep -A 50 $(printf "%x" 29250) > "271d6262-ff19-ad24-af36-777bfe6c6375:frag:1:4" daemon prio=10 > tid=0x7f5b2adec800 nid=0x7242 runnable [0x7f5aa33e8000] >java.lang.Thread.State: RUNNABLE > at java.lang.Throwable.fillInStackTrace(Native Method) > at java.lang.Throwable.fillInStackTrace(Throwable.java:783) > - locked <0x0007374bfcb0> (a java.lang.InterruptedException) > at java.lang.Throwable.(Throwable.java:250) > at java.lang.Exception.(Exception.java:54) > at java.lang.InterruptedException.(InterruptedException.java:57) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireInterruptibly(AbstractQueuedSynchronizer.java:1219) > at > java.util.concurrent.locks.ReentrantLock.lockInterruptibly(ReentrantLock.java:340) > at > java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:439) > at > org.apache.drill.exec.store.parquet.columnreaders.AsyncPageReader.clear(AsyncPageReader.java:317) > at > org.apache.drill.exec.store.parquet.columnreaders.ColumnReader.clear(ColumnReader.java:140) > at > org.apache.drill.exec.store.parquet.columnreaders.ParquetRecordReader.close(ParquetRecordReader.java:632) > at > org.apache.drill.exec.physical.impl.ScanBatch.next(ScanBatch.java:183) > at > org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:119) > at > org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:109) > at > org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext(AbstractSingleRecordBatch.java:51) > at > org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext(ProjectRecordBatch.java:135) > at > org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:162) > at > org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:119) > at > org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:109) > at > org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext(AbstractSingleRecordBatch.java:51) > at > org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:162) > at > org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:119) > at > org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:109) > at > org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext(AbstractSingleRecordBatch.java:51) > at > org.apache.drill.exec.physical.impl.svremover.RemovingRecordBatch.innerNext(RemovingRecordBatch.java:93) > at > org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:162) > at > org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:119) > at > org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:109) > at > org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext(AbstractSingleRecordBatch.java:51) > at > org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext(ProjectRecordBatch.java:135) > at > org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:162) > at > org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:119) > at > org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:109) > at > org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext(AbstractSingleRecordBatch.java:51) > at > org.apache.drill.exec.physical.impl.limit.LimitRecordBatch.
[jira] [Commented] (DRILL-5432) Want a memory format for PCAP files
[ https://issues.apache.org/jira/browse/DRILL-5432?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16070901#comment-16070901 ] ASF GitHub Bot commented on DRILL-5432: --- Github user paul-rogers commented on the issue: https://github.com/apache/drill/pull/831 > That only works if Drill has an autoloading capability that allows storage > formats to be loaded and authenticated easily. Why? To get pcap in 1.11, one must install a new Drill version, which requires a restart. Assuming that pcap were a separate project, you'd install a new jar, and restart. Neither provides auto loading. Not clear on the authentication issue. If a plugin were a separate project, wouldn't that project have a way of certifying its jars? Understanding this will help us figure out how to handle the growing set of specialized storage plugins... > Want a memory format for PCAP files > --- > > Key: DRILL-5432 > URL: https://issues.apache.org/jira/browse/DRILL-5432 > Project: Apache Drill > Issue Type: New Feature >Reporter: Ted Dunning > > PCAP files [1] are the de facto standard for storing network capture data. In > security and protocol applications, it is very common to want to extract > particular packets from a capture for further analysis. > At a first level, it is desirable to query and filter by source and > destination IP and port or by protocol. Beyond that, however, it would be > very useful to be able to group packets by TCP session and eventually to look > at packet contents. For now, however, the most critical requirement is that > we should be able to scan captures at very high speed. > I previously wrote a (kind of working) proof of concept for a PCAP decoder > that did lazy deserialization and could traverse hundreds of MB of PCAP data > per second per core. This compares to roughly 2-3 MB/s for widely available > Apache-compatible open source PCAP decoders. > This JIRA covers the integration and extension of that proof of concept as a > Drill file format. > Initial work is available at https://github.com/mapr-demos/drill-pcap-format > [1] https://en.wikipedia.org/wiki/Pcap -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (DRILL-5644) Projection using a scalar subquery causes zero rows to be returned by statement
N Campbell created DRILL-5644: - Summary: Projection using a scalar subquery causes zero rows to be returned by statement Key: DRILL-5644 URL: https://issues.apache.org/jira/browse/DRILL-5644 Project: Apache Drill Issue Type: Bug Components: Server Affects Versions: 1.10.0 Environment: Drill 1.10 Reporter: N Campbell When C3 is included in the projection zero rows are returned versus the expected 3 rows select TJOIN1.RNUM, TJOIN1.C1, case when 10 in ( select C1 from ( values (1) ) T(C1) ) then 'yes' else 'no' end C3 from ( values ( 0, 10, 15), ( 1, 20, 25), ( 2, cast(NULL as integer), 50) ) TJOIN1 (RNUM, C1, C2) -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (DRILL-3640) Drill JDBC driver support Statement.setQueryTimeout(int)
[ https://issues.apache.org/jira/browse/DRILL-3640?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16070671#comment-16070671 ] ASF GitHub Bot commented on DRILL-3640: --- Github user kkhatua commented on the issue: https://github.com/apache/drill/pull/858 Wouldn't a query cancellation automatically interrupt those potentially blocking operations? I'm simply looking up whether the trigger was a timeout DrillStatement.isTimedOut() to decide if the returning exception is a SqlTimeoutExcp. The extra thread (though mostly sleeping) didn't seem like a huge cost considering that the timeout is at the granularity of seconds. I ran a small custom JDBC client application to test this theory. Running longevity and concurrency testing worked well when testing for timeouts. For timeouts as low as 1 second for queries that needed to do lot of back end processing before returning rows much later (like a massive join), I noticed that we caught underlying exceptions as the timeouts during the 'executeQuery' call itself. If I were to not go the route of issuing a Statement.cancel() but relying primarily on the DrillCursor, unless I put in some mechanism of constantly polling for the out-of-time state of the clock. I think then I also need to cancel then cancel all operations from the level of the DrillCursor in both directions (up to the Statement and down to the fragments). All this versus issuing a Statement.cancel() that effectively cancels all the operations down to the fragment level. I am now wondering whether I missied a corner case which will be addressed by doing the timeout in the DrillCursor? > Drill JDBC driver support Statement.setQueryTimeout(int) > > > Key: DRILL-3640 > URL: https://issues.apache.org/jira/browse/DRILL-3640 > Project: Apache Drill > Issue Type: New Feature > Components: Client - JDBC >Affects Versions: 1.2.0 >Reporter: Chun Chang >Assignee: Kunal Khatua > Fix For: 1.11.0 > > > It would be nice if we have this implemented. Run away queries can be > automatically canceled by setting the timeout. > java.sql.SQLFeatureNotSupportedException: Setting network timeout is not > supported. > at > org.apache.drill.jdbc.impl.DrillStatementImpl.setQueryTimeout(DrillStatementImpl.java:152) -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (DRILL-5591) non-ASCII characters in text file result in MalformedInputException
[
https://issues.apache.org/jira/browse/DRILL-5591?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16070669#comment-16070669
]
Victor Garcia commented on DRILL-5591:
--
I resolved the problem temporally using VIM editor:
vim +"set nobomb | set fenc=utf8 | x" filename.txt
Explanation part!
+ : Used by vim to directly enter command when opening a file. Usualy used to
open a file at a specific line: vim +14 file.txt
| : Separator of multiple commands (like ; in bash)
set nobomb : no utf-8 BOM
set fenc=utf8 : Set new encoding to utf-8 doc link
x : Save and close file
filename.txt : path to the file
" : qotes are here because of pipes. (otherwise bash will use them as bash pipe)
The command and the explaniation was taked from:
[https://stackoverflow.com/questions/64860/best-way-to-convert-text-files-between-character-sets]
Thank you Paul.
> non-ASCII characters in text file result in MalformedInputException
> ---
>
> Key: DRILL-5591
> URL: https://issues.apache.org/jira/browse/DRILL-5591
> Project: Apache Drill
> Issue Type: Bug
> Components: Storage - Text & CSV
>Affects Versions: 1.11.0
>Reporter: Khurram Faraaz
>
> I am on Drill 1.11.0 commit id: 874bf62
> To repro the issue:
> wget http://cfdisat.blob.core.windows.net/lco/l_RFC_2017_05_11_2.txt.gz
> gunzip l_RFC_2017_05_11_2.txt.gz
> hadoop fs -put l_RFC_2017_05_11_2.txt /tmp
> There are some non-ASCII characters at the beginning and end of the file used
> in the test.
> {noformat}
> [root@centos-01 drill_5590]# head l_RFC_2017_05_11_2.txt
> ����0���1��
> ��� ��RFC|SNCF|SUBCONTRATACION
> CUBB910321AC1|0|0
> CUBB9104187K9|0|0
> CUBB910709KD0|0|0
> CUBB910817CE8|0|0
> CUBB9111286YA|0|0
> CUBB920408J69|0|0
> {noformat}
> Failing query
> {noformat}
> 0: jdbc:drill:schema=dfs.tmp> select count(1) from `l_RFC_2017_05_11_2.txt` t
> where columns[0] like 'CUBA7706%';
> Error: SYSTEM ERROR: MalformedInputException: Input length = 1
> Fragment 0:0
> [Error Id: cdfa704c-0bc8-4791-95ae-d05b4c63beab on centos-01.qa.lab:31010]
> (state=,code=0)
> {noformat}
> Stack trace from drillbit.log
> {noformat}
> Caused by: java.lang.RuntimeException:
> java.nio.charset.MalformedInputException: Input length = 1
> at
> org.apache.drill.exec.expr.fn.impl.CharSequenceWrapper.decodeUT8(CharSequenceWrapper.java:185)
> ~[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
> at
> org.apache.drill.exec.expr.fn.impl.CharSequenceWrapper.setBuffer(CharSequenceWrapper.java:119)
> ~[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
> at
> org.apache.drill.exec.test.generated.FiltererGen15.doEval(FilterTemplate2.java:50)
> ~[na:na]
> at
> org.apache.drill.exec.test.generated.FiltererGen15.filterBatchNoSV(FilterTemplate2.java:100)
> ~[na:na]
> at
> org.apache.drill.exec.test.generated.FiltererGen15.filterBatch(FilterTemplate2.java:73)
> ~[na:na]
> at
> org.apache.drill.exec.physical.impl.filter.FilterRecordBatch.doWork(FilterRecordBatch.java:81)
> ~[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
> at
> org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext(AbstractSingleRecordBatch.java:93)
> ~[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
> at
> org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:162)
> ~[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
> at
> org.apache.drill.exec.physical.impl.validate.IteratorValidatorBatchIterator.next(IteratorValidatorBatchIterator.java:225)
> ~[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
> at
> org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:119)
> ~[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
> at
> org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:109)
> ~[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
> at
> org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext(AbstractSingleRecordBatch.java:51)
> ~[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
> at
> org.apache.drill.exec.physical.impl.svremover.RemovingRecordBatch.innerNext(RemovingRecordBatch.java:93)
> ~[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
> at
> org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:162)
> ~[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
> at
> org.apache.drill.exec.physical.impl.validate.IteratorValidatorBatchIterator.next(IteratorValidatorBatchIterator.java:225)
> ~[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
> at
> org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:119)
> ~[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
> at
> o
[jira] [Created] (DRILL-5643) Provide a way to configure excluded list of protocols and ciphers to be used by WebServer
Sorabh Hamirwasia created DRILL-5643:
Summary: Provide a way to configure excluded list of protocols and
ciphers to be used by WebServer
Key: DRILL-5643
URL: https://issues.apache.org/jira/browse/DRILL-5643
Project: Apache Drill
Issue Type: Improvement
Components: Web Server
Affects Versions: 1.11.0
Reporter: Sorabh Hamirwasia
Drill's WebServer uses the default protocol for TLS which is TLSv1 and default
list of cipher suites when SSL is enabled. This task is to add capability to
configure list of protocols / cipher to exclude from being used by WebServer.
*Supported Protocols:*
enabledProtocols = {ProtocolList@6589} "[SSLv2Hello, TLSv1, TLSv1.1, TLSv1.2]”
*Selected Protocol Version:*
protocolVersion = {ProtocolVersion@6566} "TLSv1"
*Cipher Suites:*
cipherSuites = {ArrayList@6755} size = 36
0 = {CipherSuite@6607} "TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384"
1 = {CipherSuite@6608} "TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384"
2 = {CipherSuite@6609} "TLS_RSA_WITH_AES_256_CBC_SHA256"
3 = {CipherSuite@6610} "TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA384"
4 = {CipherSuite@6611} "TLS_ECDH_RSA_WITH_AES_256_CBC_SHA384"
5 = {CipherSuite@6612} "TLS_DHE_RSA_WITH_AES_256_CBC_SHA256"
6 = {CipherSuite@6613} "TLS_DHE_DSS_WITH_AES_256_CBC_SHA256"
7 = {CipherSuite@6614} "TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA"
8 = {CipherSuite@6615} "TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA"
9 = {CipherSuite@6616} "TLS_RSA_WITH_AES_256_CBC_SHA"
10 = {CipherSuite@6617} "TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA"
11 = {CipherSuite@6618} "TLS_ECDH_RSA_WITH_AES_256_CBC_SHA"
12 = {CipherSuite@6619} "TLS_DHE_RSA_WITH_AES_256_CBC_SHA"
13 = {CipherSuite@6620} "TLS_DHE_DSS_WITH_AES_256_CBC_SHA"
14 = {CipherSuite@6621} "TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256"
15 = {CipherSuite@6622} "TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256"
16 = {CipherSuite@6623} "TLS_RSA_WITH_AES_128_CBC_SHA256"
17 = {CipherSuite@6624} "TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA256"
18 = {CipherSuite@6625} "TLS_ECDH_RSA_WITH_AES_128_CBC_SHA256"
19 = {CipherSuite@6626} "TLS_DHE_RSA_WITH_AES_128_CBC_SHA256"
20 = {CipherSuite@6627} "TLS_DHE_DSS_WITH_AES_128_CBC_SHA256"
21 = {CipherSuite@6628} "TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA"
22 = {CipherSuite@6629} "TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA"
23 = {CipherSuite@6630} "TLS_RSA_WITH_AES_128_CBC_SHA"
24 = {CipherSuite@6631} "TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA"
25 = {CipherSuite@6632} "TLS_ECDH_RSA_WITH_AES_128_CBC_SHA"
26 = {CipherSuite@6633} "TLS_DHE_RSA_WITH_AES_128_CBC_SHA"
27 = {CipherSuite@6634} "TLS_DHE_DSS_WITH_AES_128_CBC_SHA"
28 = {CipherSuite@6635} "TLS_ECDHE_ECDSA_WITH_3DES_EDE_CBC_SHA"
29 = {CipherSuite@6636} "TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA"
30 = {CipherSuite@6637} "SSL_RSA_WITH_3DES_EDE_CBC_SHA"
31 = {CipherSuite@6638} "TLS_ECDH_ECDSA_WITH_3DES_EDE_CBC_SHA"
32 = {CipherSuite@6639} "TLS_ECDH_RSA_WITH_3DES_EDE_CBC_SHA"
33 = {CipherSuite@6640} "SSL_DHE_RSA_WITH_3DES_EDE_CBC_SHA"
34 = {CipherSuite@6641} "SSL_DHE_DSS_WITH_3DES_EDE_CBC_SHA"
35 = {CipherSuite@6642} "TLS_EMPTY_RENEGOTIATION_INFO_SCSV"
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Created] (DRILL-5642) Error: VALIDATION ERROR: null
N Campbell created DRILL-5642: - Summary: Error: VALIDATION ERROR: null Key: DRILL-5642 URL: https://issues.apache.org/jira/browse/DRILL-5642 Project: Apache Drill Issue Type: Improvement Components: Server Affects Versions: 1.10.0 Environment: Drill 1.10 Reporter: N Campbell Following statement will fail Error: VALIDATION ERROR: null select TJOIN1.C1,TJOIN2.C2 from ( values ( 0, 10, 15), ( 1, 20, 25), ( 2, cast(NULL as integer), 50) ) TJOIN1 (RNUM, C1, C2) left outer join ( values ( 0, 10, 'BB'), ( 1, 15, 'DD'), ( 2, cast(NULL as integer), 'EE'), ( 3, 10, 'FF') ) TJOIN2 (RNUM, C1,C2) on ( TJOIN1.C1 = TJOIN2.C1 and TJOIN1.C2 < ( select count(*) from (values (1)) TX (RNUM) ) ) -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (DRILL-5641) Error: VALIDATION ERROR: class org.apache.calcite.sql.SqlNodeList
N Campbell created DRILL-5641: - Summary: Error: VALIDATION ERROR: class org.apache.calcite.sql.SqlNodeList Key: DRILL-5641 URL: https://issues.apache.org/jira/browse/DRILL-5641 Project: Apache Drill Issue Type: Improvement Components: Server Affects Versions: 1.10.0 Environment: Drill 1.10 Reporter: N Campbell Following statement will fail with Error: VALIDATION ERROR: class org.apache.calcite.sql.SqlNodeList: select TJOIN1.C1,TJOIN2.C2 from ( values ( 0, 10, 15), ( 1, 20, 25), ( 2, cast(NULL as integer), 50) ) TJOIN1 (RNUM, C1, C2) left outer join ( values ( 0, 10, 'BB'), ( 1, 15, 'DD'), ( 2, cast(NULL as integer), 'EE'), ( 3, 10, 'FF') ) TJOIN2 (RNUM, C1,C2) on TJOIN1.C1=TJOIN2.C1 and TJOIN1.C1 and TJOIN1.C1 >= ( (select sum(C1) from ( values ( 0, 10, 15), ( 1, 20, 25), ( 2, cast(NULL as integer), 50) ) TX (RNUM, C1, C2) ) /2 ) -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (DRILL-5640) NOT causes Error: SYSTEM ERROR: IllegalArgumentException: Invalid value for boolean: AA
[ https://issues.apache.org/jira/browse/DRILL-5640?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] N Campbell updated DRILL-5640: -- Issue Type: Bug (was: Improvement) > NOT causes Error: SYSTEM ERROR: IllegalArgumentException: Invalid > value for boolean: AA > --- > > Key: DRILL-5640 > URL: https://issues.apache.org/jira/browse/DRILL-5640 > Project: Apache Drill > Issue Type: Bug > Components: Server >Affects Versions: 1.10.0 > Environment: Drill 1.10 >Reporter: N Campbell > > Following statement will fail, will not fail when NOT removed > select TJOIN2.RNUM, TJOIN1.C1, TJOIN1.C2, TJOIN2.C2 as C2J2 from > ( > values > ( 0, 10, 15), > ( 1, 20, 25), > ( 2, cast(NULL as integer), 50) > ) TJOIN1 (RNUM, C1, C2) > inner join > ( > values ( 0, 10, 'BB'), > ( 1, 15, 'DD'), > ( 2, cast(NULL as integer), 'EE'), > ( 3, 10, 'FF') > ) TJOIN2 (RNUM, C1, C2) > on ( TJOIN1.C1 = TJOIN2.C1 and not TJOIN2.C2 = 'AA' ) > Error: SYSTEM ERROR: IllegalArgumentException: Invalid value for boolean: AA -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (DRILL-5640) NOT causes Error: SYSTEM ERROR: IllegalArgumentException: Invalid value for boolean: AA
N Campbell created DRILL-5640: - Summary: NOT causes Error: SYSTEM ERROR: IllegalArgumentException: Invalid value for boolean: AA Key: DRILL-5640 URL: https://issues.apache.org/jira/browse/DRILL-5640 Project: Apache Drill Issue Type: Improvement Components: Server Affects Versions: 1.10.0 Environment: Drill 1.10 Reporter: N Campbell Following statement will fail, will not fail when NOT removed select TJOIN2.RNUM, TJOIN1.C1, TJOIN1.C2, TJOIN2.C2 as C2J2 from ( values ( 0, 10, 15), ( 1, 20, 25), ( 2, cast(NULL as integer), 50) ) TJOIN1 (RNUM, C1, C2) inner join ( values ( 0, 10, 'BB'), ( 1, 15, 'DD'), ( 2, cast(NULL as integer), 'EE'), ( 3, 10, 'FF') ) TJOIN2 (RNUM, C1, C2) on ( TJOIN1.C1 = TJOIN2.C1 and not TJOIN2.C2 = 'AA' ) Error: SYSTEM ERROR: IllegalArgumentException: Invalid value for boolean: AA -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (DRILL-5432) Want a memory format for PCAP files
[ https://issues.apache.org/jira/browse/DRILL-5432?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16070590#comment-16070590 ] ASF GitHub Bot commented on DRILL-5432: --- Github user tdunning commented on the issue: https://github.com/apache/drill/pull/831 On Fri, Jun 30, 2017 at 9:55 AM, Paul Rogers wrote: > Later, once Drill provides the correct framework, I'd suggest that this > code move into a separate Github repo to be maintained by experts in pcap. > Frankly, most Drill developers are familiar with query engines, not pcap > (or other specialized formats.) > > That only works if Drill has an autoloading capability that allows storage formats to be loaded and authenticated easily. > Want a memory format for PCAP files > --- > > Key: DRILL-5432 > URL: https://issues.apache.org/jira/browse/DRILL-5432 > Project: Apache Drill > Issue Type: New Feature >Reporter: Ted Dunning > > PCAP files [1] are the de facto standard for storing network capture data. In > security and protocol applications, it is very common to want to extract > particular packets from a capture for further analysis. > At a first level, it is desirable to query and filter by source and > destination IP and port or by protocol. Beyond that, however, it would be > very useful to be able to group packets by TCP session and eventually to look > at packet contents. For now, however, the most critical requirement is that > we should be able to scan captures at very high speed. > I previously wrote a (kind of working) proof of concept for a PCAP decoder > that did lazy deserialization and could traverse hundreds of MB of PCAP data > per second per core. This compares to roughly 2-3 MB/s for widely available > Apache-compatible open source PCAP decoders. > This JIRA covers the integration and extension of that proof of concept as a > Drill file format. > Initial work is available at https://github.com/mapr-demos/drill-pcap-format > [1] https://en.wikipedia.org/wiki/Pcap -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (DRILL-5183) Drill doesn't seem to handle array values correctly in Parquet files
[
https://issues.apache.org/jira/browse/DRILL-5183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16070574#comment-16070574
]
Dave Kincaid commented on DRILL-5183:
-
Right. Spark is using the Parquet list type format that's explained here -
https://github.com/apache/parquet-format/blob/master/LogicalTypes.md
> Drill doesn't seem to handle array values correctly in Parquet files
>
>
> Key: DRILL-5183
> URL: https://issues.apache.org/jira/browse/DRILL-5183
> Project: Apache Drill
> Issue Type: Bug
>Reporter: Dave Kincaid
> Attachments: books.parquet
>
>
> It looks to me that Drill is not properly converting array values in Parquet
> records. I have created a simple example and will attach a simple Parquet
> file to this issue. If I write Parquet records using the Avro schema
> {code:title=Book.avsc}
> { "type": "record",
> "name": "Book",
> "fields": [
> { "name": "title", "type": "string" },
> { "name": "pages", "type": "int" },
> { "name": "authors", "type": {"type": "array", "items": "string"} }
> ]
> }
> {code}
> I write two records using this schema into the attached Parquet file and then
> simply run {{SELECT * FROM dfs.`books.parquet`}} I get the following result:
> ||title||pages||authors||
> |Physics of Waves|477|{"array":["William C. Elmore","Mark A. Heald"]}|
> |Foundations of Mathematical Analysis|428|{"array":["Richard
> Johnsonbaugh","W.E. Pfaffenberger"]}|
> You can see that the authors column seems to be a nested record with the name
> "array" instead of being a repeated value. If I change the SQL query to
> {{SELECT title,pages,t.authors.`array` FROM
> dfs.`/home/davek/src/drill-parquet-example/resources/books.parquet` t;}} then
> I get:
> ||title||pages||EXPR$2||
> |Physics of Waves|477|["William C. Elmore","Mark A. Heald"]|
> |Foundations of Mathematical Analysis|428|["Richard Johnsonbaugh","W.E.
> Pfaffenberger"]|
> and now that column behaves in Drill as a repeated values column.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (DRILL-5183) Drill doesn't seem to handle array values correctly in Parquet files
[
https://issues.apache.org/jira/browse/DRILL-5183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16070568#comment-16070568
]
Rahul Challapalli commented on DRILL-5183:
--
HmmThe one from spark looks nasty and I can't think of a good workaround as
well
> Drill doesn't seem to handle array values correctly in Parquet files
>
>
> Key: DRILL-5183
> URL: https://issues.apache.org/jira/browse/DRILL-5183
> Project: Apache Drill
> Issue Type: Bug
>Reporter: Dave Kincaid
> Attachments: books.parquet
>
>
> It looks to me that Drill is not properly converting array values in Parquet
> records. I have created a simple example and will attach a simple Parquet
> file to this issue. If I write Parquet records using the Avro schema
> {code:title=Book.avsc}
> { "type": "record",
> "name": "Book",
> "fields": [
> { "name": "title", "type": "string" },
> { "name": "pages", "type": "int" },
> { "name": "authors", "type": {"type": "array", "items": "string"} }
> ]
> }
> {code}
> I write two records using this schema into the attached Parquet file and then
> simply run {{SELECT * FROM dfs.`books.parquet`}} I get the following result:
> ||title||pages||authors||
> |Physics of Waves|477|{"array":["William C. Elmore","Mark A. Heald"]}|
> |Foundations of Mathematical Analysis|428|{"array":["Richard
> Johnsonbaugh","W.E. Pfaffenberger"]}|
> You can see that the authors column seems to be a nested record with the name
> "array" instead of being a repeated value. If I change the SQL query to
> {{SELECT title,pages,t.authors.`array` FROM
> dfs.`/home/davek/src/drill-parquet-example/resources/books.parquet` t;}} then
> I get:
> ||title||pages||EXPR$2||
> |Physics of Waves|477|["William C. Elmore","Mark A. Heald"]|
> |Foundations of Mathematical Analysis|428|["Richard Johnsonbaugh","W.E.
> Pfaffenberger"]|
> and now that column behaves in Drill as a repeated values column.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (DRILL-5183) Drill doesn't seem to handle array values correctly in Parquet files
[
https://issues.apache.org/jira/browse/DRILL-5183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16070547#comment-16070547
]
Rahul Challapalli commented on DRILL-5183:
--
Hmm.this is clearly a bug we should fix. Have you considered the below
workaround using a view
{code}
0: jdbc:drill:zk=10.10.100.190:5181> create view drill5183 as select d.title,
d.pages, d.authors.`array` authors from dfs.`/drill/testdata/books.parquet` d;
+---+-+
| ok | summary |
+---+-+
| true | View 'drill5183' created successfully in 'dfs.drillTestDir' schema |
+---+-+
1 row selected (0.403 seconds)
0: jdbc:drill:zk=10.10.100.190:5181> select * from drill5183;
+---+++
| title | pages |authors
|
+---+++
| Physics of Waves | 477| ["William C. Elmore","Mark
A. Heald"] |
| Foundations of Mathematical Analysis | 428| ["Richard
Johnsonbaugh","W.E. Pfaffenberger"] |
+---+++
2 rows selected (0.33 seconds)
{code}
> Drill doesn't seem to handle array values correctly in Parquet files
>
>
> Key: DRILL-5183
> URL: https://issues.apache.org/jira/browse/DRILL-5183
> Project: Apache Drill
> Issue Type: Bug
>Reporter: Dave Kincaid
> Attachments: books.parquet
>
>
> It looks to me that Drill is not properly converting array values in Parquet
> records. I have created a simple example and will attach a simple Parquet
> file to this issue. If I write Parquet records using the Avro schema
> {code:title=Book.avsc}
> { "type": "record",
> "name": "Book",
> "fields": [
> { "name": "title", "type": "string" },
> { "name": "pages", "type": "int" },
> { "name": "authors", "type": {"type": "array", "items": "string"} }
> ]
> }
> {code}
> I write two records using this schema into the attached Parquet file and then
> simply run {{SELECT * FROM dfs.`books.parquet`}} I get the following result:
> ||title||pages||authors||
> |Physics of Waves|477|{"array":["William C. Elmore","Mark A. Heald"]}|
> |Foundations of Mathematical Analysis|428|{"array":["Richard
> Johnsonbaugh","W.E. Pfaffenberger"]}|
> You can see that the authors column seems to be a nested record with the name
> "array" instead of being a repeated value. If I change the SQL query to
> {{SELECT title,pages,t.authors.`array` FROM
> dfs.`/home/davek/src/drill-parquet-example/resources/books.parquet` t;}} then
> I get:
> ||title||pages||EXPR$2||
> |Physics of Waves|477|["William C. Elmore","Mark A. Heald"]|
> |Foundations of Mathematical Analysis|428|["Richard Johnsonbaugh","W.E.
> Pfaffenberger"]|
> and now that column behaves in Drill as a repeated values column.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (DRILL-5420) all cores at 100% of all servers
[ https://issues.apache.org/jira/browse/DRILL-5420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kunal Khatua updated DRILL-5420: Labels: ready-to-commit (was: ) > all cores at 100% of all servers > > > Key: DRILL-5420 > URL: https://issues.apache.org/jira/browse/DRILL-5420 > Project: Apache Drill > Issue Type: Bug >Affects Versions: 1.10.0 > Environment: linux, cluster with 5 servers over hdfs/parquet >Reporter: Hugo Bellomusto >Assignee: Kunal Khatua > Labels: ready-to-commit > Fix For: 1.11.0 > > Attachments: 2709a36d-804a-261a-64e5-afa271e782f8.json > > > We have a drill cluster with five servers over hdfs/parquet. > Each machine have 8 cores. All cores get at 100% of use. > Each thread is looping in the while in line 314 in AsyncPageReader.java > inside clear() method. > https://github.com/apache/drill/blob/1.10.0/exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/columnreaders/AsyncPageReader.java#L314 > jstack -l 19255|grep -A 50 $(printf "%x" 29250) > "271d6262-ff19-ad24-af36-777bfe6c6375:frag:1:4" daemon prio=10 > tid=0x7f5b2adec800 nid=0x7242 runnable [0x7f5aa33e8000] >java.lang.Thread.State: RUNNABLE > at java.lang.Throwable.fillInStackTrace(Native Method) > at java.lang.Throwable.fillInStackTrace(Throwable.java:783) > - locked <0x0007374bfcb0> (a java.lang.InterruptedException) > at java.lang.Throwable.(Throwable.java:250) > at java.lang.Exception.(Exception.java:54) > at java.lang.InterruptedException.(InterruptedException.java:57) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireInterruptibly(AbstractQueuedSynchronizer.java:1219) > at > java.util.concurrent.locks.ReentrantLock.lockInterruptibly(ReentrantLock.java:340) > at > java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:439) > at > org.apache.drill.exec.store.parquet.columnreaders.AsyncPageReader.clear(AsyncPageReader.java:317) > at > org.apache.drill.exec.store.parquet.columnreaders.ColumnReader.clear(ColumnReader.java:140) > at > org.apache.drill.exec.store.parquet.columnreaders.ParquetRecordReader.close(ParquetRecordReader.java:632) > at > org.apache.drill.exec.physical.impl.ScanBatch.next(ScanBatch.java:183) > at > org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:119) > at > org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:109) > at > org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext(AbstractSingleRecordBatch.java:51) > at > org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext(ProjectRecordBatch.java:135) > at > org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:162) > at > org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:119) > at > org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:109) > at > org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext(AbstractSingleRecordBatch.java:51) > at > org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:162) > at > org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:119) > at > org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:109) > at > org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext(AbstractSingleRecordBatch.java:51) > at > org.apache.drill.exec.physical.impl.svremover.RemovingRecordBatch.innerNext(RemovingRecordBatch.java:93) > at > org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:162) > at > org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:119) > at > org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:109) > at > org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext(AbstractSingleRecordBatch.java:51) > at > org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext(ProjectRecordBatch.java:135) > at > org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:162) > at > org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:119) > at > org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:109) > at > org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext(AbstractSingleRecordBatch.java:51) > at > org.apache.drill.exec.physical.impl.limit.LimitRecordBatch.innerNext(LimitRecordBatch.java:115) > at > org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:162)
[jira] [Commented] (DRILL-3640) Drill JDBC driver support Statement.setQueryTimeout(int)
[ https://issues.apache.org/jira/browse/DRILL-3640?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16070517#comment-16070517 ] ASF GitHub Bot commented on DRILL-3640: --- Github user laurentgo commented on the issue: https://github.com/apache/drill/pull/858 As for System.currentTimeMillis(), maybe not as expensive as you think: if I remember correctly, it's a JVM intrinsic (no JNI cost), and if mapped to gettimeofday (which I believe is the case), it's a VSDO function on Linux, which means it's pretty cheap (no user/kernel context change). Compared to creating threads, and waiting on conditions, should be way cheaper. > Drill JDBC driver support Statement.setQueryTimeout(int) > > > Key: DRILL-3640 > URL: https://issues.apache.org/jira/browse/DRILL-3640 > Project: Apache Drill > Issue Type: New Feature > Components: Client - JDBC >Affects Versions: 1.2.0 >Reporter: Chun Chang >Assignee: Kunal Khatua > Fix For: 1.11.0 > > > It would be nice if we have this implemented. Run away queries can be > automatically canceled by setting the timeout. > java.sql.SQLFeatureNotSupportedException: Setting network timeout is not > supported. > at > org.apache.drill.jdbc.impl.DrillStatementImpl.setQueryTimeout(DrillStatementImpl.java:152) -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (DRILL-3640) Drill JDBC driver support Statement.setQueryTimeout(int)
[ https://issues.apache.org/jira/browse/DRILL-3640?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16070508#comment-16070508 ] ASF GitHub Bot commented on DRILL-3640: --- Github user laurentgo commented on the issue: https://github.com/apache/drill/pull/858 you should look at: - https://github.com/kkhatua/drill/blob/c51473859d1dd81cf70e857f729c3a8491b2834a/exec/jdbc/src/main/java/org/apache/drill/jdbc/impl/DrillCursor.java#L144 - https://github.com/kkhatua/drill/blob/c51473859d1dd81cf70e857f729c3a8491b2834a/exec/jdbc/src/main/java/org/apache/drill/jdbc/impl/DrillCursor.java#L226 - https://github.com/kkhatua/drill/blob/c51473859d1dd81cf70e857f729c3a8491b2834a/exec/jdbc/src/main/java/org/apache/drill/jdbc/impl/DrillCursor.java#L255 I believe those are the places where blocking operations are done, most of them don't have a timeout configured, but could be configured to have one (getNext is a bit special since it's waking up every 50ms to manage throttling) - https://github.com/kkhatua/drill/blob/c51473859d1dd81cf70e857f729c3a8491b2834a/exec/jdbc/src/main/java/org/apache/drill/jdbc/impl/DrillCursor.java#L548 - > Drill JDBC driver support Statement.setQueryTimeout(int) > > > Key: DRILL-3640 > URL: https://issues.apache.org/jira/browse/DRILL-3640 > Project: Apache Drill > Issue Type: New Feature > Components: Client - JDBC >Affects Versions: 1.2.0 >Reporter: Chun Chang >Assignee: Kunal Khatua > Fix For: 1.11.0 > > > It would be nice if we have this implemented. Run away queries can be > automatically canceled by setting the timeout. > java.sql.SQLFeatureNotSupportedException: Setting network timeout is not > supported. > at > org.apache.drill.jdbc.impl.DrillStatementImpl.setQueryTimeout(DrillStatementImpl.java:152) -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (DRILL-5432) Want a memory format for PCAP files
[ https://issues.apache.org/jira/browse/DRILL-5432?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16070503#comment-16070503 ] ASF GitHub Bot commented on DRILL-5432: --- Github user dmitriyHavrilovich commented on the issue: https://github.com/apache/drill/pull/831 This is really good. Will do this immediately. > Want a memory format for PCAP files > --- > > Key: DRILL-5432 > URL: https://issues.apache.org/jira/browse/DRILL-5432 > Project: Apache Drill > Issue Type: New Feature >Reporter: Ted Dunning > > PCAP files [1] are the de facto standard for storing network capture data. In > security and protocol applications, it is very common to want to extract > particular packets from a capture for further analysis. > At a first level, it is desirable to query and filter by source and > destination IP and port or by protocol. Beyond that, however, it would be > very useful to be able to group packets by TCP session and eventually to look > at packet contents. For now, however, the most critical requirement is that > we should be able to scan captures at very high speed. > I previously wrote a (kind of working) proof of concept for a PCAP decoder > that did lazy deserialization and could traverse hundreds of MB of PCAP data > per second per core. This compares to roughly 2-3 MB/s for widely available > Apache-compatible open source PCAP decoders. > This JIRA covers the integration and extension of that proof of concept as a > Drill file format. > Initial work is available at https://github.com/mapr-demos/drill-pcap-format > [1] https://en.wikipedia.org/wiki/Pcap -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (DRILL-3640) Drill JDBC driver support Statement.setQueryTimeout(int)
[
https://issues.apache.org/jira/browse/DRILL-3640?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16070492#comment-16070492
]
ASF GitHub Bot commented on DRILL-3640:
---
Github user kkhatua commented on the issue:
https://github.com/apache/drill/pull/858
@laurentgo Within the DrillCursor, the only place I could do such a check
was
https://github.com/kkhatua/drill/blob/c51473859d1dd81cf70e857f729c3a8491b2834a/exec/jdbc/src/main/java/org/apache/drill/jdbc/impl/DrillCursor.java#L582
(Hosting a fork of this PR in this repo:
https://github.com/kkhatua/drill/commits/altDrill3640 )
My JDBC client for performance testing has a similar mechanism for JDBC
drivers that don't support timeout, but allow for query cancellation by using a
cancelling-trigger thread to sleep until the timeout, before waking up
explicitly cancelling the query. I've simply replicated behaviour that in the
Drill JDBC package. Having a constant check on the time remaining using a
system call like {{System.currentTimeMillis()}} is actually expensive, which is
why I didn't want to have the DrillCursor contantly do that check before
throwing an exception. Can you point to me on which async framework should I be
looking at as well ?
> Drill JDBC driver support Statement.setQueryTimeout(int)
>
>
> Key: DRILL-3640
> URL: https://issues.apache.org/jira/browse/DRILL-3640
> Project: Apache Drill
> Issue Type: New Feature
> Components: Client - JDBC
>Affects Versions: 1.2.0
>Reporter: Chun Chang
>Assignee: Kunal Khatua
> Fix For: 1.11.0
>
>
> It would be nice if we have this implemented. Run away queries can be
> automatically canceled by setting the timeout.
> java.sql.SQLFeatureNotSupportedException: Setting network timeout is not
> supported.
> at
> org.apache.drill.jdbc.impl.DrillStatementImpl.setQueryTimeout(DrillStatementImpl.java:152)
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (DRILL-5636) Reduce external sort memory use when spilling
[ https://issues.apache.org/jira/browse/DRILL-5636?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Paul Rogers updated DRILL-5636: --- Description: The external sort spills data to disk under memory pressure. The sort code uses a generic mechanism to do the spilling: * Use a "priority queue copier" to copy sorted records into a new batch * Spill the new batch by writing the vectors for the newly-created batch The above works fine when memory is plentiful. But, under low-memory conditions, the intermediate copy can cause OOM errors. An improved algorithm is: * Priority queue copier works vector-by-vector * Serialize each vector to disk * Release its memory * Repeat for the next vector The advantages of the above: * Less intermediate memory use * Perhaps better CPU cache performance through greater locality (all writes happen to a single vector at a time, rather than row by row) * No change in disk format or disk write performance (because data is buffered prior to write anyway.) was: The external sort spills data to disk under memory pressure. The sort code uses a generic mechanism to do the spilling: * Use a "priority queue copier" to copy sorted records into a new batch * Spill the new batch by writing the vectors for the newly-created batch The above works fine when memory is plentiful. But, under low-memory conditions, the intermediate copy can cause OOM errors. An improved algorithm is: * Priority queue copier works vector-by-vector * Serialize each vector to disk * Release its memory * Repeat for the next vector The advantages of the above: * Less intermediate memory use * Perhaps better CPU cache performance through greater locality (all writes happen to a single vector at a time, rather than row by row) * No change in disk format or disk write performance (because data is buffered prior to write anyway.) * > Reduce external sort memory use when spilling > - > > Key: DRILL-5636 > URL: https://issues.apache.org/jira/browse/DRILL-5636 > Project: Apache Drill > Issue Type: Improvement >Affects Versions: 1.8.0 >Reporter: Paul Rogers >Assignee: Paul Rogers >Priority: Minor > > The external sort spills data to disk under memory pressure. The sort code > uses a generic mechanism to do the spilling: > * Use a "priority queue copier" to copy sorted records into a new batch > * Spill the new batch by writing the vectors for the newly-created batch > The above works fine when memory is plentiful. But, under low-memory > conditions, the intermediate copy can cause OOM errors. > An improved algorithm is: > * Priority queue copier works vector-by-vector > * Serialize each vector to disk > * Release its memory > * Repeat for the next vector > The advantages of the above: > * Less intermediate memory use > * Perhaps better CPU cache performance through greater locality (all writes > happen to a single vector at a time, rather than row by row) > * No change in disk format or disk write performance (because data is > buffered prior to write anyway.) -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (DRILL-5601) Rollup of External Sort memory management fixes
[ https://issues.apache.org/jira/browse/DRILL-5601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16070479#comment-16070479 ] ASF GitHub Bot commented on DRILL-5601: --- Github user paul-rogers commented on the issue: https://github.com/apache/drill/pull/860 # Additional Revisions The code contains an optimization for short queries. If the query has just one batch then the sort need not copy that batch to produce the output. Instead, the sort simply passes along the input batch along with an SV2. This commit fixes some issues with transferring that single, buffered, input batch to the sort’s output container. While reproducing problems found by QA, it turned out to be useful to be able to parse SQL statements from a file in the context of the test framework. Added test methods for this purpose. The “AllocationHelper” class turns out to throw an exception if asked to allocate a vector with zero elements. Since a zero-size can occasionally come from the record batch sizer (via the “smart allocation helper”, special code was added to handle this case. Backed out some debugging code that accidentally appeared in the original PR. With this set of improvements, the revised, managed sort has become more stable and reliable than the original version. As a result, this commit enables the managed sort by default. > Rollup of External Sort memory management fixes > --- > > Key: DRILL-5601 > URL: https://issues.apache.org/jira/browse/DRILL-5601 > Project: Apache Drill > Issue Type: Task >Affects Versions: 1.11.0 >Reporter: Paul Rogers >Assignee: Paul Rogers > Fix For: 1.11.0 > > > Rollup of a set of specific JIRA entries that all relate to the very > difficult problem of managing memory within Drill in order for the external > sort to stay within a memory budget. In general, the fixes relate to better > estimating memory used by the three ways that Drill allocates vector memory > (see DRILL-5522) and to predicting the size of vectors that the sort will > create, to avoid repeated realloc-copy cycles (see DRILL-5594). -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (DRILL-5517) Provide size-aware set operations in value vectors
[
https://issues.apache.org/jira/browse/DRILL-5517?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16070405#comment-16070405
]
ASF GitHub Bot commented on DRILL-5517:
---
Github user paul-rogers commented on a diff in the pull request:
https://github.com/apache/drill/pull/840#discussion_r125083942
--- Diff: exec/vector/src/main/codegen/templates/FixedValueVectors.java ---
@@ -546,238 +576,400 @@ public DateTime getObject(int index) {
public ${minor.javaType!type.javaType} getPrimitiveObject(int index) {
return get(index);
}
-
+
public void get(int index, ${minor.class}Holder holder){
<#if minor.class.startsWith("Decimal")>
holder.scale = getField().getScale();
holder.precision = getField().getPrecision();
- holder.value =
data.get${(minor.javaType!type.javaType)?cap_first}(index * ${type.width});
+ holder.value =
data.get${(minor.javaType!type.javaType)?cap_first}(index * VALUE_WIDTH);
}
public void get(int index, Nullable${minor.class}Holder holder){
holder.isSet = 1;
- holder.value =
data.get${(minor.javaType!type.javaType)?cap_first}(index * ${type.width});
+ holder.value =
data.get${(minor.javaType!type.javaType)?cap_first}(index * VALUE_WIDTH);
}
<#-- type.width -->
- }
-
- /**
- * ${minor.class}.Mutator implements a mutable vector of fixed width
values. Elements in the
- * vector are accessed by position from the logical start of the vector.
Values should be pushed
- * onto the vector sequentially, but may be randomly accessed.
- * The width of each element is ${type.width} byte(s)
- * The equivalent Java primitive is '${minor.javaType!type.javaType}'
- *
- * NB: this class is automatically generated from ValueVectorTypes.tdd
using FreeMarker.
- */
- public final class Mutator extends BaseDataValueVector.BaseMutator {
-
-private Mutator(){};
- /**
-* Set the element at the given index to the given value. Note that
widths smaller than
-* 32 bits are handled by the DrillBuf interface.
-*
-* @param index position of the bit to set
-* @param value value to set
-*/
+ }
+
+ /**
+ * ${minor.class}.Mutator implements a mutable vector of fixed width
values. Elements in the
+ * vector are accessed by position from the logical start of the vector.
Values should be pushed
+ * onto the vector sequentially, but may be randomly accessed.
+ *
+ * The width of each element is {@link #VALUE_WIDTH} (=
${type.width}) byte(s).
+ * The equivalent Java primitive is
'${minor.javaType!type.javaType}'
+ *
+ *
+ * NB: this class is automatically generated from ValueVectorTypes.tdd
using FreeMarker.
+ */
+ public final class Mutator extends BaseDataValueVector.BaseMutator {
+
+private Mutator() {};
+
+/**
+ * Set the element at the given index to the given value. Note that
widths smaller than
+ * 32 bits are handled by the DrillBuf interface.
+ *
+ * @param index position of the bit to set
+ * @param value value to set
+ */
+
<#if (type.width > 8)>
public void set(int index, <#if (type.width >
4)>${minor.javaType!type.javaType}<#else>int value) {
- data.setBytes(index * ${type.width}, value, 0, ${type.width});
+ data.setBytes(index * VALUE_WIDTH, value, 0, VALUE_WIDTH);
}
public void setSafe(int index, <#if (type.width >
4)>${minor.javaType!type.javaType}<#else>int value) {
while(index >= getValueCapacity()) {
reAlloc();
}
- data.setBytes(index * ${type.width}, value, 0, ${type.width});
+ data.setBytes(index * VALUE_WIDTH, value, 0, VALUE_WIDTH);
}
- <#if (minor.class == "Interval")>
-public void set(int index, int months, int days, int milliseconds){
- final int offsetIndex = index * ${type.width};
- data.setInt(offsetIndex, months);
- data.setInt((offsetIndex + ${minor.daysOffset}), days);
- data.setInt((offsetIndex + ${minor.millisecondsOffset}),
milliseconds);
+/**
+ * Set the value of a required or nullable vector. Enforces the value
+ * and size limits.
+ * @param index item to write
+ * @return true if the item was written, false if the index would
--- End diff --
Fixed here and several other places.
> Provide size-aware set operations in value vectors
> --
>
> Key: DRILL-5517
> URL: https://issues.apache.org/j
[jira] [Commented] (DRILL-5432) Want a memory format for PCAP files
[ https://issues.apache.org/jira/browse/DRILL-5432?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16070396#comment-16070396 ] ASF GitHub Bot commented on DRILL-5432: --- Github user paul-rogers commented on the issue: https://github.com/apache/drill/pull/831 Looking at the big picture, Drill should allow specialized plugins such as this one to exist as independent projects. Users should be able to download the plugin jar, add it to Drill, and go. As we've discussed, Drill has a bit of work before we get there. We can't hold up this work waiting for a better solution. So, please fix the two minor issues you identified. The code will then be ready for a final quick review and approval. Later, once Drill provides the correct framework, I'd suggest that this code move into a separate Github repo to be maintained by experts in pcap. Frankly, most Drill developers are familiar with query engines, not pcap (or other specialized formats.) The same is true, for example, of the "indexr" and TSDB plugins which are (slowly) working their way through the review process. Summary: please add the package-info file and the comments in utils. We can then give approval. Can we do this by, say, July 10? If so, we can likely get this PR into 1.11, if the Release Manager agrees. > Want a memory format for PCAP files > --- > > Key: DRILL-5432 > URL: https://issues.apache.org/jira/browse/DRILL-5432 > Project: Apache Drill > Issue Type: New Feature >Reporter: Ted Dunning > > PCAP files [1] are the de facto standard for storing network capture data. In > security and protocol applications, it is very common to want to extract > particular packets from a capture for further analysis. > At a first level, it is desirable to query and filter by source and > destination IP and port or by protocol. Beyond that, however, it would be > very useful to be able to group packets by TCP session and eventually to look > at packet contents. For now, however, the most critical requirement is that > we should be able to scan captures at very high speed. > I previously wrote a (kind of working) proof of concept for a PCAP decoder > that did lazy deserialization and could traverse hundreds of MB of PCAP data > per second per core. This compares to roughly 2-3 MB/s for widely available > Apache-compatible open source PCAP decoders. > This JIRA covers the integration and extension of that proof of concept as a > Drill file format. > Initial work is available at https://github.com/mapr-demos/drill-pcap-format > [1] https://en.wikipedia.org/wiki/Pcap -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (DRILL-5639) CAST( as CHAR(N) or VARCHAR(N)) ResultsetMetadata always returned as VARCHAR(65535)
[
https://issues.apache.org/jira/browse/DRILL-5639?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
N Campbell updated DRILL-5639:
--
Priority: Minor (was: Major)
> CAST( as CHAR(N) or VARCHAR(N)) ResultsetMetadata always returned as
> VARCHAR(65535)
> ---
>
> Key: DRILL-5639
> URL: https://issues.apache.org/jira/browse/DRILL-5639
> Project: Apache Drill
> Issue Type: Improvement
> Components: Client - JDBC
>Affects Versions: 1.10.0
> Environment: Statement is explicitly CASTING to a CHAR type with
> precision but result described as VARCHAR with maximum precision.
> select cast(C1 as CHAR(10)) from (values('ABC')) T (C1)
> select cast(C1 as VARCHAR(10)) from (values('ABC')) T (C1)
> ResultsetMetadata
> ColumnIndex getColumnName getColumnTypeName getPrecision
> getScaleisNullable getTableNamegetSchemaName
> getCatalogName getColumnClassName getColumnDisplaySize
> getColumnLabel getColumnType isAutoIncrement isCaseSensitive isCurrency
> isDefinitelyWritableisReadOnly isSearchableisSigned
> isWritable
> 1 EXPR$0 CHARACTER VARYING 65536 0 1
> DRILL java.lang.String65536 EXPR$0 12 false false false
> false truetruefalse false
>Reporter: N Campbell
>Priority: Minor
>
> While Drill states it internally uses VARCHAR type, unclear if Drill had
> considered/attempted to ensure it would project a CHAR(n) with blank padding.
> https://drill.apache.org/docs/supported-data-types/
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (DRILL-5639) CAST( as CHAR(N) or VARCHAR(N)) ResultsetMetadata always returned as VARCHAR(65535)
[
https://issues.apache.org/jira/browse/DRILL-5639?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
N Campbell updated DRILL-5639:
--
Description:
While Drill states it internally uses VARCHAR type, unclear if Drill had
considered/attempted to ensure it would project a CHAR(n) with blank padding.
https://drill.apache.org/docs/supported-data-types/
> CAST( as CHAR(N) or VARCHAR(N)) ResultsetMetadata always returned as
> VARCHAR(65535)
> ---
>
> Key: DRILL-5639
> URL: https://issues.apache.org/jira/browse/DRILL-5639
> Project: Apache Drill
> Issue Type: Improvement
> Components: Client - JDBC
>Affects Versions: 1.10.0
> Environment: Statement is explicitly CASTING to a CHAR type with
> precision but result described as VARCHAR with maximum precision.
> select cast(C1 as CHAR(10)) from (values('ABC')) T (C1)
> select cast(C1 as VARCHAR(10)) from (values('ABC')) T (C1)
> ResultsetMetadata
> ColumnIndex getColumnName getColumnTypeName getPrecision
> getScaleisNullable getTableNamegetSchemaName
> getCatalogName getColumnClassName getColumnDisplaySize
> getColumnLabel getColumnType isAutoIncrement isCaseSensitive isCurrency
> isDefinitelyWritableisReadOnly isSearchableisSigned
> isWritable
> 1 EXPR$0 CHARACTER VARYING 65536 0 1
> DRILL java.lang.String65536 EXPR$0 12 false false false
> false truetruefalse false
>Reporter: N Campbell
>
> While Drill states it internally uses VARCHAR type, unclear if Drill had
> considered/attempted to ensure it would project a CHAR(n) with blank padding.
> https://drill.apache.org/docs/supported-data-types/
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Created] (DRILL-5639) CAST( as CHAR(N) or VARCHAR(N)) ResultsetMetadata always returned as VARCHAR(65535)
N Campbell created DRILL-5639:
-
Summary: CAST( as CHAR(N) or VARCHAR(N)) ResultsetMetadata always
returned as VARCHAR(65535)
Key: DRILL-5639
URL: https://issues.apache.org/jira/browse/DRILL-5639
Project: Apache Drill
Issue Type: Improvement
Components: Client - JDBC
Affects Versions: 1.10.0
Environment: Statement is explicitly CASTING to a CHAR type with
precision but result described as VARCHAR with maximum precision.
select cast(C1 as CHAR(10)) from (values('ABC')) T (C1)
select cast(C1 as VARCHAR(10)) from (values('ABC')) T (C1)
ResultsetMetadata
ColumnIndex getColumnName getColumnTypeName getPrecision
getScaleisNullable getTableNamegetSchemaName getCatalogName
getColumnClassName getColumnDisplaySizegetColumnLabel getColumnType
isAutoIncrement isCaseSensitive isCurrency isDefinitelyWritable
isReadOnly isSearchableisSignedisWritable
1 EXPR$0 CHARACTER VARYING 65536 0 1
DRILL java.lang.String65536 EXPR$0 12 false false false
false truetruefalse false
Reporter: N Campbell
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Created] (DRILL-5638) Increase the precision of INTERVAL types to be larger than 2
N Campbell created DRILL-5638: - Summary: Increase the precision of INTERVAL types to be larger than 2 Key: DRILL-5638 URL: https://issues.apache.org/jira/browse/DRILL-5638 Project: Apache Drill Issue Type: Improvement Components: SQL Parser Affects Versions: 1.10.0 Environment: Drill 1.10 Reporter: N Campbell You cannot express an interval literal (DAY, HOUR, MINUTE, SECOND, ...) with a precision greater than 2. select INTERVAL '100' DAY from dfs.certp.TVERSION Error: VALIDATION ERROR: From line 1, column 8 to line 1, column 25: Interval field value 100 exceeds precision of DAY(2) field select INTERVAL '160' HOUR from dfs.certp.TVERSION Error: VALIDATION ERROR: From line 1, column 8 to line 1, column 26: Interval field value 160 exceeds precision of HOUR(2) field -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (DRILL-4720) MINDIR() and IMINDIR() functions return no results with metadata cache
[
https://issues.apache.org/jira/browse/DRILL-4720?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arina Ielchiieva updated DRILL-4720:
Fix Version/s: 1.11.0
> MINDIR() and IMINDIR() functions return no results with metadata cache
> --
>
> Key: DRILL-4720
> URL: https://issues.apache.org/jira/browse/DRILL-4720
> Project: Apache Drill
> Issue Type: Bug
> Components: Functions - Drill
>Affects Versions: 1.7.0
>Reporter: Krystal
>Assignee: Arina Ielchiieva
> Fix For: 1.11.0
>
>
> Parquet directories with meta data cache return 0 rows for MINDIR and IMINDIR
> functions.
> hadoop fs -ls /tmp/querylogs_4
> Found 6 items
> -rwxr-xr-x 3 mapr mapr 15406 2016-06-13 10:18
> /tmp/querylogs_4/.drill.parquet_metadata
> drwxr-xr-x - root root 4 2016-06-13 10:18 /tmp/querylogs_4/1985
> drwxr-xr-x - root root 3 2016-06-13 10:18 /tmp/querylogs_4/1999
> drwxr-xr-x - root root 3 2016-06-13 10:18 /tmp/querylogs_4/2005
> drwxr-xr-x - root root 4 2016-06-13 10:18 /tmp/querylogs_4/2014
> drwxr-xr-x - root root 6 2016-06-13 10:18 /tmp/querylogs_4/2016
> hadoop fs -ls /tmp/querylogs_4/1985
> Found 4 items
> -rwxr-xr-x 3 mapr mapr 3634 2016-06-13 10:18
> /tmp/querylogs_4/1985/.drill.parquet_metadata
> drwxr-xr-x - root root 2 2016-06-13 10:18 /tmp/querylogs_4/1985/Feb
> drwxr-xr-x - root root 2 2016-06-13 10:18 /tmp/querylogs_4/1985/apr
> drwxr-xr-x - root root 2 2016-06-13 10:18
> /tmp/querylogs_4/1985/jan
> SELECT * FROM `dfs.tmp`.`querylogs_4` WHERE dir0 =
> MINDIR('dfs.tmp','querylogs_4');
> +---+---+--+---++++---+---+---+
> | voter_id | name | age | registration | contributions | voterzone |
> date_time | dir0 | dir1 | dir2 |
> +---+---+--+---++++---+---+---+
> +---+---+--+---++++---+---+---+
> No rows selected (0.803 seconds)
> If the meta cache is removed, expected data is returned.
> Here is the physical plan:
> {code}
> 00-00Screen : rowType = RecordType(ANY *): rowcount = 3.75, cumulative
> cost = {54.125 rows, 169.125 cpu, 0.0 io, 0.0 network, 0.0 memory}, id =
> 664191
> 00-01 Project(*=[$0]) : rowType = RecordType(ANY *): rowcount = 3.75,
> cumulative cost = {53.75 rows, 168.75 cpu, 0.0 io, 0.0 network, 0.0 memory},
> id = 664190
> 00-02Project(T51¦¦*=[$0]) : rowType = RecordType(ANY T51¦¦*):
> rowcount = 3.75, cumulative cost = {53.75 rows, 168.75 cpu, 0.0 io, 0.0
> network, 0.0 memory}, id = 664189
> 00-03 SelectionVectorRemover : rowType = RecordType(ANY T51¦¦*, ANY
> dir0): rowcount = 3.75, cumulative cost = {53.75 rows, 168.75 cpu, 0.0 io,
> 0.0 network, 0.0 memory}, id = 664188
> 00-04Filter(condition=[=($1, '.drill.parquet_metadata')]) :
> rowType = RecordType(ANY T51¦¦*, ANY dir0): rowcount = 3.75, cumulative cost
> = {50.0 rows, 165.0 cpu, 0.0 io, 0.0 network, 0.0 memory}, id = 664187
> 00-05 Project(T51¦¦*=[$0], dir0=[$1]) : rowType = RecordType(ANY
> T51¦¦*, ANY dir0): rowcount = 25.0, cumulative cost = {25.0 rows, 50.0 cpu,
> 0.0 io, 0.0 network, 0.0 memory}, id = 664186
> 00-06Scan(groupscan=[ParquetGroupScan
> [entries=[ReadEntryWithPath
> [path=/tmp/querylogs_4/2005/May/voter25.parquet/0_0_0.parquet]],
> selectionRoot=/tmp/querylogs_4, numFiles=1, usedMetadataFile=true,
> columns=[`*`]]]) : rowType = (DrillRecordRow[*, dir0]): rowcount = 25.0,
> cumulative cost = {25.0 rows, 50.0 cpu, 0.0 io, 0.0 network, 0.0 memory}, id
> = 664185
> {code}
> Here is the plan for the same query against the same directory structure
> without meta data cache:
> {code}
> 00-00Screen : rowType = RecordType(ANY *): rowcount = 75.0, cumulative
> cost = {82.5 rows, 157.5 cpu, 0.0 io, 0.0 network, 0.0 memory}, id = 664312
> 00-01 Project(*=[$0]) : rowType = RecordType(ANY *): rowcount = 75.0,
> cumulative cost = {75.0 rows, 150.0 cpu, 0.0 io, 0.0 network, 0.0 memory}, id
> = 664311
> 00-02Project(*=[$0]) : rowType = RecordType(ANY *): rowcount = 75.0,
> cumulative cost = {75.0 rows, 150.0 cpu, 0.0 io, 0.0 network, 0.0 memory}, id
> = 664310
> 00-03 Scan(groupscan=[ParquetGroupScan [entries=[ReadEntryWithPath
> [path=maprfs:///tmp/querylogs_1/1985/Feb/voter10.parquet/0_0_0.parquet],
> ReadEntryWithPath
> [path=maprfs:///tmp/querylogs_1/1985/jan/voter5.parquet/0_0_0.parquet],
> ReadEntryWithPath
> [path=maprfs:///tmp/querylogs_1/1985/apr/voter65.parquet/0_0_0.parquet]],
> selectionRoot=maprfs:/tmp/querylogs_1, numFiles=3,
[jira] [Created] (DRILL-5637) Date arith using literals SYSTEM ERROR: AssertionError: Internal error: Conversion to relational algebra failed to preserve datatypes
N Campbell created DRILL-5637: - Summary: Date arith using literals SYSTEM ERROR: AssertionError: Internal error: Conversion to relational algebra failed to preserve datatypes Key: DRILL-5637 URL: https://issues.apache.org/jira/browse/DRILL-5637 Project: Apache Drill Issue Type: Bug Components: Server Affects Versions: 1.10.0 Environment: Drill 1.10 Reporter: N Campbell Use any file with one row in it. Submit the following statement to Drill which is performing interval arith on a date using literals. select DATE '2000-01-29' + INTERVAL -'1' YEAR from dfs.certp.TVERSION Error: SYSTEM ERROR: AssertionError: Internal error: Conversion to relational algebra failed to preserve datatypes: validated type: RecordType(TIMESTAMP(0) NOT NULL EXPR$0) NOT NULL converted type: RecordType(DATE NOT NULL EXPR$0) NOT NULL rel: LogicalProject(EXPR$0=[DATETIME_PLUS(2000-01-29, -12)]) LogicalTableScan(table=[[dfs, certp, TVERSION]]) -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (DRILL-4720) MINDIR() and IMINDIR() functions return no results with metadata cache
[
https://issues.apache.org/jira/browse/DRILL-4720?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16069961#comment-16069961
]
ASF GitHub Bot commented on DRILL-4720:
---
GitHub user arina-ielchiieva opened a pull request:
https://github.com/apache/drill/pull/864
DRILL-4720: Fix SchemaPartitionExplorer.getSubPartitions method
implementations to return only Drill file system directories
1. Added file system util helper classes to standardize list directory and
file statuses usage in Drill with appropriate unit tests.
2. Fixed SchemaPartitionExplorer.getSubPartitions method implementations to
return only directories that can be partitions according to Drill file system
rules (excluded all files and directories that start with dot or underscore).
3. Added unit test for directory explorers UDFs with and without metadata
cache presence.
4. Minor refactoring.
Details in Jira
[DRILL-4720](https://issues.apache.org/jira/browse/DRILL-4720).
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/arina-ielchiieva/drill DRILL-4720
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/drill/pull/864.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #864
commit 6d592373c740fd793ed6bbb3264b97b52e4b763b
Author: Arina Ielchiieva
Date: 2017-06-29T13:08:33Z
DRILL-4720: Fix SchemaPartitionExplorer.getSubPartitions method
implementations to return only Drill file system directories
1. Added file system util helper classes to standardize list directory and
file statuses usage in Drill with appropriate unit tests.
2. Fixed SchemaPartitionExplorer.getSubPartitions method implementations to
return only directories that can be partitions according to Drill file system
rules
(excluded all files and directories that start with dot or underscore).
3. Added unit test for directory explorers UDFs with and without metadata
cache presence.
4. Minor refactoring.
> MINDIR() and IMINDIR() functions return no results with metadata cache
> --
>
> Key: DRILL-4720
> URL: https://issues.apache.org/jira/browse/DRILL-4720
> Project: Apache Drill
> Issue Type: Bug
> Components: Functions - Drill
>Affects Versions: 1.7.0
>Reporter: Krystal
>Assignee: Arina Ielchiieva
>
> Parquet directories with meta data cache return 0 rows for MINDIR and IMINDIR
> functions.
> hadoop fs -ls /tmp/querylogs_4
> Found 6 items
> -rwxr-xr-x 3 mapr mapr 15406 2016-06-13 10:18
> /tmp/querylogs_4/.drill.parquet_metadata
> drwxr-xr-x - root root 4 2016-06-13 10:18 /tmp/querylogs_4/1985
> drwxr-xr-x - root root 3 2016-06-13 10:18 /tmp/querylogs_4/1999
> drwxr-xr-x - root root 3 2016-06-13 10:18 /tmp/querylogs_4/2005
> drwxr-xr-x - root root 4 2016-06-13 10:18 /tmp/querylogs_4/2014
> drwxr-xr-x - root root 6 2016-06-13 10:18 /tmp/querylogs_4/2016
> hadoop fs -ls /tmp/querylogs_4/1985
> Found 4 items
> -rwxr-xr-x 3 mapr mapr 3634 2016-06-13 10:18
> /tmp/querylogs_4/1985/.drill.parquet_metadata
> drwxr-xr-x - root root 2 2016-06-13 10:18 /tmp/querylogs_4/1985/Feb
> drwxr-xr-x - root root 2 2016-06-13 10:18 /tmp/querylogs_4/1985/apr
> drwxr-xr-x - root root 2 2016-06-13 10:18
> /tmp/querylogs_4/1985/jan
> SELECT * FROM `dfs.tmp`.`querylogs_4` WHERE dir0 =
> MINDIR('dfs.tmp','querylogs_4');
> +---+---+--+---++++---+---+---+
> | voter_id | name | age | registration | contributions | voterzone |
> date_time | dir0 | dir1 | dir2 |
> +---+---+--+---++++---+---+---+
> +---+---+--+---++++---+---+---+
> No rows selected (0.803 seconds)
> If the meta cache is removed, expected data is returned.
> Here is the physical plan:
> {code}
> 00-00Screen : rowType = RecordType(ANY *): rowcount = 3.75, cumulative
> cost = {54.125 rows, 169.125 cpu, 0.0 io, 0.0 network, 0.0 memory}, id =
> 664191
> 00-01 Project(*=[$0]) : rowType = RecordType(ANY *): rowcount = 3.75,
> cumulative cost = {53.75 rows, 168.75 cpu, 0.0 io, 0.0 network, 0.0 memory},
> id = 664190
> 00-02Project(T51¦¦*=[$0]) : rowType = RecordType(ANY T51¦¦*):
> rowcount = 3.75, cumulative cost = {53.75 rows, 168.75 cpu, 0.0 io, 0.0
> network, 0.0 memory}, id = 664189
> 00-03 SelectionVectorRemover : rowType = RecordType(ANY T51¦
[jira] [Comment Edited] (DRILL-4720) MINDIR() and IMINDIR() functions return no results with metadata cache
[
https://issues.apache.org/jira/browse/DRILL-4720?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16069951#comment-16069951
]
Arina Ielchiieva edited comment on DRILL-4720 at 6/30/17 11:40 AM:
---
In Drill code we frequently list directory and file statuses:
{{ShowFileHandler}} - lists all directories and files present in given path
(including hidden)
{{FileSelection}} - lists all files recursively in given path excluding files
and directories that start with dot or underscore
{{FileSystemSchemaFactory}} - lists all directories present in given path
excluding directories that start with dot or underscore
{{WorkspaceSchemaFactory}} - 1. lists all directories present in given path
excluding directories that start with dot or underscore 2. lists all files
recursively in given path excluding files and directories that start with dot
or underscore
{{FooterGatherer}} - lists all files in given path excluding files that start
with dot or underscore
{{Metadata}} - 1. lists all directories and files in given path excluding files
that start with dot or underscore 2. lists all files recursively in given path
excluding files and directories that start with dot or underscore
{{ParquetFormatPlugin}} - lists all files in given path excluding files that
start with dot or underscore
{{ParquetGroupScan}} - lists all files recursively in given path excluding
files that start with dot or underscore
{{LocalPersistentStore}} - lists all files in given path that end with
.sys.drill excluding files that start with dot or underscore
In many cases recursive search is implemented in each class, new instance of
Drill filter is created (though it can be shared) etc.
Some use {{DrillFileSystem.list(boolean recursive, Path... paths)}} where as I
have mentioned before logic is not obvious.
Common list status use cases:
1. List directories and / or files recursively or not.
2. List directories and / or files recursively or not applying drill file
system filter.
3. List directories and / or files recursively or not applying custom filter.
To standardize list statuses usage in Drill I suggest we add two new helper
classes which will hold all list status logic.
First one - {{FileSystemUtil}} which will have the following methods:
{{public static List listDirectories(final FileSystem fs, Path
path, boolean recursive, PathFilter... filters) throws IOException {}}
{{public static List listFiles(FileSystem fs, Path path, boolean
recursive, PathFilter... filters) throws IOException {}}
{{public static List listAll(FileSystem fs, Path path, boolean
recursive, PathFilter... filters) throws IOException {}}
We might add some other file system helper method later, that's why class name
is quite abstract. Developer will be able yo use these methods to list statuses
of directories, files or both, recursively or not and also will be able to
apply custom filters if needed.
Second one - {{DrillFileSystemUtil}} which will have the same methods as
{{FileSystemUtil}} but will also add Drill file system filter, so files and
folders that start with dot and underscore is excluded.
was (Author: arina):
In Drill code we frequently list directory and file statuses:
{{ShowFileHandler}} - lists all directories and files present in given path
(including hidden)
{{FileSelection}} - lists all files recursively in given path excluding files
and directories that start with dot or underscore
{{FileSystemSchemaFactory}} - lists all directories present in given path
excluding directories that start with dot or underscore
{{WorkspaceSchemaFactory}} - 1. lists all directories present in given path
excluding directories that start with dot or underscore 2. lists all files
recursively in given path excluding files and directories that start with dot
or underscore
{{FooterGatherer}} - lists all files in given path excluding files that start
with dot or underscore
{[Metadata}} - 1. lists all directories and files in given path excluding files
that start with dot or underscore 2. lists all files recursively in given path
excluding files and directories that start with dot or underscore
{{ParquetFormatPlugin}} - lists all files in given path excluding files that
start with dot or underscore
{{ParquetGroupScan}} - lists all files recursively in given path excluding
files that start with dot or underscore
{{LocalPersistentStore}} - lists all files in given path that end with
.sys.drill excluding files that start with dot or underscore
In many cases recursive search is implemented in each class, new instance of
Drill filter is created (though it can be shared) etc.
Some use {{DrillFileSystem.list(boolean recursive, Path... paths)}} where as I
have mentioned before logic is not obvious.
Common list status use cases:
1. List directories and / or files recursively or not.
2. List directories and / or files recursively or not a
[jira] [Commented] (DRILL-4720) MINDIR() and IMINDIR() functions return no results with metadata cache
[
https://issues.apache.org/jira/browse/DRILL-4720?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16069951#comment-16069951
]
Arina Ielchiieva commented on DRILL-4720:
-
In Drill code we frequently list directory and file statuses:
{{ShowFileHandler}} - lists all directories and files present in given path
(including hidden)
{{FileSelection}} - lists all files recursively in given path excluding files
and directories that start with dot or underscore
{{FileSystemSchemaFactory}} - lists all directories present in given path
excluding directories that start with dot or underscore
{{WorkspaceSchemaFactory}} - 1. lists all directories present in given path
excluding directories that start with dot or underscore 2. lists all files
recursively in given path excluding files and directories that start with dot
or underscore
{{FooterGatherer}} - lists all files in given path excluding files that start
with dot or underscore
{[Metadata}} - 1. lists all directories and files in given path excluding files
that start with dot or underscore 2. lists all files recursively in given path
excluding files and directories that start with dot or underscore
{{ParquetFormatPlugin}} - lists all files in given path excluding files that
start with dot or underscore
{{ParquetGroupScan}} - lists all files recursively in given path excluding
files that start with dot or underscore
{{LocalPersistentStore}} - lists all files in given path that end with
.sys.drill excluding files that start with dot or underscore
In many cases recursive search is implemented in each class, new instance of
Drill filter is created (though it can be shared) etc.
Some use {{DrillFileSystem.list(boolean recursive, Path... paths)}} where as I
have mentioned before logic is not obvious.
Common list status use cases:
1. List directories and / or files recursively or not.
2. List directories and / or files recursively or not applying drill file
system filter.
3. List directories and / or files recursively or not applying custom filter.
To standardize list statuses usage in Drill I suggest we add two new helper
classes which will hold all list status logic.
First one - {{FileSystemUtil}} which will have the following methods:
{{public static List listDirectories(final FileSystem fs, Path
path, boolean recursive, PathFilter... filters) throws IOException {}}
{{public static List listFiles(FileSystem fs, Path path, boolean
recursive, PathFilter... filters) throws IOException {}}
{{public static List listAll(FileSystem fs, Path path, boolean
recursive, PathFilter... filters) throws IOException {}}
We might add some other file system helper method later, that's why class name
is quite abstract. Developer will be able yo use these methods to list statuses
of directories, files or both, recursively or not and also will be able to
apply custom filters if needed.
Second one - {{DrillFileSystemUtil}} which will have the same methods as
{{FileSystemUtil}} but will also add Drill file system filter, so files and
folders that start with dot and underscore is excluded.
> MINDIR() and IMINDIR() functions return no results with metadata cache
> --
>
> Key: DRILL-4720
> URL: https://issues.apache.org/jira/browse/DRILL-4720
> Project: Apache Drill
> Issue Type: Bug
> Components: Functions - Drill
>Affects Versions: 1.7.0
>Reporter: Krystal
>Assignee: Arina Ielchiieva
>
> Parquet directories with meta data cache return 0 rows for MINDIR and IMINDIR
> functions.
> hadoop fs -ls /tmp/querylogs_4
> Found 6 items
> -rwxr-xr-x 3 mapr mapr 15406 2016-06-13 10:18
> /tmp/querylogs_4/.drill.parquet_metadata
> drwxr-xr-x - root root 4 2016-06-13 10:18 /tmp/querylogs_4/1985
> drwxr-xr-x - root root 3 2016-06-13 10:18 /tmp/querylogs_4/1999
> drwxr-xr-x - root root 3 2016-06-13 10:18 /tmp/querylogs_4/2005
> drwxr-xr-x - root root 4 2016-06-13 10:18 /tmp/querylogs_4/2014
> drwxr-xr-x - root root 6 2016-06-13 10:18 /tmp/querylogs_4/2016
> hadoop fs -ls /tmp/querylogs_4/1985
> Found 4 items
> -rwxr-xr-x 3 mapr mapr 3634 2016-06-13 10:18
> /tmp/querylogs_4/1985/.drill.parquet_metadata
> drwxr-xr-x - root root 2 2016-06-13 10:18 /tmp/querylogs_4/1985/Feb
> drwxr-xr-x - root root 2 2016-06-13 10:18 /tmp/querylogs_4/1985/apr
> drwxr-xr-x - root root 2 2016-06-13 10:18
> /tmp/querylogs_4/1985/jan
> SELECT * FROM `dfs.tmp`.`querylogs_4` WHERE dir0 =
> MINDIR('dfs.tmp','querylogs_4');
> +---+---+--+---++++---+---+---+
> | voter_id | name | age | registration | contributions | voterzone |
> date_time | dir0 | dir1 | dir2 |
> +---
[jira] [Updated] (DRILL-5629) large traceback if query attempts to use TABLESAMPLE
[
https://issues.apache.org/jira/browse/DRILL-5629?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
N Campbell updated DRILL-5629:
--
Summary: large traceback if query attempts to use TABLESAMPLE (was: large
traceback is query attempts to use TABLESAMPLE)
> large traceback if query attempts to use TABLESAMPLE
>
>
> Key: DRILL-5629
> URL: https://issues.apache.org/jira/browse/DRILL-5629
> Project: Apache Drill
> Issue Type: Bug
> Components: Server
>Affects Versions: 1.10.0
> Environment: Drill 1.10
> ORACLE 12C2
>Reporter: N Campbell
>Priority: Minor
>
> A plugin is defined to ORACLE
> Following statement is sent to DRILL which returns large exception response
> SELECT RNUM, C1 FROM certora.DBCERT.TOLAP TABLESAMPLE SYSTEM ( 2 )
> repeatable ( 5 )
> SYSTEM ERROR: CannotPlanException: Node
> [rel#939863:Subset#2.LOGICAL.ANY([]).[]] could not be implemented; planner
> state:
> Root: rel#939863:Subset#2.LOGICAL.ANY([]).[]
> Original rel:
> Sets:
> Set#0, type: RecordType(DECIMAL(38, 0) RNUM, CHAR(3) C1, CHAR(2) C2,
> DECIMAL(38, 0) C3, DECIMAL(38, 0) C4)
> rel#939858:Subset#0.JDBC.certora.ANY([]).[], best=rel#939844,
> importance=0.7291
>
> rel#939844:JdbcTableScan.JDBC.certora.ANY([]).[](table=[certora, DBCERT,
> TOLAP]), rowcount=100.0, cumulative cost={100.0 rows, 101.0 cpu, 0.0 io, 0.0
> network, 0.0 memory}
>
> rel#939894:AbstractConverter.JDBC.certora.ANY([]).[](input=rel#939893:Subset#0.LOGICAL.ANY([]).[],convention=JDBC.certora,DrillDistributionTraitDef=ANY([]),sort=[]),
> rowcount=100.0, cumulative cost={inf}
>
> rel#939900:AbstractConverter.JDBC.certora.ANY([]).[](input=rel#939899:Subset#0.ENUMERABLE.ANY([]).[],convention=JDBC.certora,DrillDistributionTraitDef=ANY([]),sort=[]),
> rowcount=100.0, cumulative cost={inf}
>
> rel#939908:AbstractConverter.JDBC.certora.ANY([]).[](input=rel#939907:Subset#0.PHYSICAL.ANY([]).[],convention=JDBC.certora,DrillDistributionTraitDef=ANY([]),sort=[]),
> rowcount=100.0, cumulative cost={inf}
> rel#939893:Subset#0.LOGICAL.ANY([]).[], best=rel#939892,
> importance=0.6561
>
> rel#939895:AbstractConverter.LOGICAL.ANY([]).[](input=rel#939858:Subset#0.JDBC.certora.ANY([]).[],convention=LOGICAL,DrillDistributionTraitDef=ANY([]),sort=[]),
> rowcount=100.0, cumulative cost={inf}
>
> rel#939892:JdbcDrel.LOGICAL.ANY([]).[](input=rel#939858:Subset#0.JDBC.certora.ANY([]).[]),
> rowcount=100.0, cumulative cost={200.0 rows, 201.0 cpu, 0.0 io, 0.0 network,
> 0.0 memory}
>
> rel#939901:AbstractConverter.LOGICAL.ANY([]).[](input=rel#939899:Subset#0.ENUMERABLE.ANY([]).[],convention=LOGICAL,DrillDistributionTraitDef=ANY([]),sort=[]),
> rowcount=100.0, cumulative cost={inf}
>
> rel#939909:AbstractConverter.LOGICAL.ANY([]).[](input=rel#939907:Subset#0.PHYSICAL.ANY([]).[],convention=LOGICAL,DrillDistributionTraitDef=ANY([]),sort=[]),
> rowcount=100.0, cumulative cost={inf}
> rel#939899:Subset#0.ENUMERABLE.ANY([]).[], best=rel#939898,
> importance=0.59049001
>
> rel#939902:AbstractConverter.ENUMERABLE.ANY([]).[](input=rel#939858:Subset#0.JDBC.certora.ANY([]).[],convention=ENUMERABLE,DrillDistributionTraitDef=ANY([]),sort=[]),
> rowcount=100.0, cumulative cost={inf}
>
> rel#939903:AbstractConverter.ENUMERABLE.ANY([]).[](input=rel#939893:Subset#0.LOGICAL.ANY([]).[],convention=ENUMERABLE,DrillDistributionTraitDef=ANY([]),sort=[]),
> rowcount=100.0, cumulative cost={inf}
>
> rel#939898:JdbcToEnumerableConverter.ENUMERABLE.ANY([]).[](input=rel#939858:Subset#0.JDBC.certora.ANY([]).[]),
> rowcount=100.0, cumulative cost={110.0 rows, 111.0 cpu, 0.0 io, 0.0 network,
> 0.0 memory}
>
> rel#939910:AbstractConverter.ENUMERABLE.ANY([]).[](input=rel#939907:Subset#0.PHYSICAL.ANY([]).[],convention=ENUMERABLE,DrillDistributionTraitDef=ANY([]),sort=[]),
> rowcount=100.0, cumulative cost={inf}
> rel#939907:Subset#0.PHYSICAL.ANY([]).[], best=rel#939906,
> importance=0.531441
>
> rel#939911:AbstractConverter.PHYSICAL.ANY([]).[](input=rel#939858:Subset#0.JDBC.certora.ANY([]).[],convention=PHYSICAL,DrillDistributionTraitDef=ANY([]),sort=[]),
> rowcount=100.0, cumulative cost={inf}
>
> rel#939912:AbstractConverter.PHYSICAL.ANY([]).[](input=rel#939893:Subset#0.LOGICAL.ANY([]).[],convention=PHYSICAL,DrillDistributionTraitDef=ANY([]),sort=[]),
> rowcount=100.0, cumulative cost={inf}
>
> rel#939913:AbstractConverter.PHYSICAL.ANY([]).[](input=rel#939899:Subset#0.ENUMERABLE.ANY([]).[],convention=PHYSICAL,DrillDistributionTraitDef=ANY([]),sort=[]),
> rowcount=100.0, cumulative cost={inf}
>
> rel#939906:Jdb
