[jira] [Commented] (DRILL-6032) Use RecordBatchSizer to estimate size of columns in HashAgg

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-6032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16379721#comment-16379721
 ] 

ASF GitHub Bot commented on DRILL-6032:
---

Github user ilooner commented on a diff in the pull request:

https://github.com/apache/drill/pull/1101#discussion_r171133616
  
--- Diff: exec/java-exec/src/main/resources/drill-module.conf ---
@@ -427,8 +427,8 @@ drill.exec.options: {
 exec.enable_union_type: false,
 exec.errors.verbose: false,
 exec.hashagg.mem_limit: 0,
-exec.hashagg.min_batches_per_partition: 2,
-exec.hashagg.num_partitions: 32,
+exec.hashagg.min_batches_per_partition: 1,
--- End diff --

@Ben-Zvi This setting controls the minimum number of batches kept in memory 
per partition. Making this larger will cause us to consume more memory. Making 
it smaller makes us consume less memory. Also in general the purpose of this PR 
was to make the memory calculations more precise and deterministic and it 
passes all regression tests.


> Use RecordBatchSizer to estimate size of columns in HashAgg
> ---
>
> Key: DRILL-6032
> URL: https://issues.apache.org/jira/browse/DRILL-6032
> Project: Apache Drill
>  Issue Type: Improvement
>Reporter: Timothy Farkas
>Assignee: Timothy Farkas
>Priority: Major
> Fix For: 1.13.0
>
>
> We need to use the RecordBatchSize to estimate the size of columns in the 
> Partition batches created by HashAgg.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (DRILL-6032) Use RecordBatchSizer to estimate size of columns in HashAgg

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-6032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16379694#comment-16379694
 ] 

ASF GitHub Bot commented on DRILL-6032:
---

Github user Ben-Zvi commented on a diff in the pull request:

https://github.com/apache/drill/pull/1101#discussion_r171130622
  
--- Diff: exec/java-exec/src/main/resources/drill-module.conf ---
@@ -427,8 +427,8 @@ drill.exec.options: {
 exec.enable_union_type: false,
 exec.errors.verbose: false,
 exec.hashagg.mem_limit: 0,
-exec.hashagg.min_batches_per_partition: 2,
-exec.hashagg.num_partitions: 32,
+exec.hashagg.min_batches_per_partition: 1,
--- End diff --

This option was meant to create a "slack". **1** is the lowest value - 
requiring only 1 batch per each partition, i.e., no slack; so that requires the 
memory computations to be more precise now !!



> Use RecordBatchSizer to estimate size of columns in HashAgg
> ---
>
> Key: DRILL-6032
> URL: https://issues.apache.org/jira/browse/DRILL-6032
> Project: Apache Drill
>  Issue Type: Improvement
>Reporter: Timothy Farkas
>Assignee: Timothy Farkas
>Priority: Major
> Fix For: 1.13.0
>
>
> We need to use the RecordBatchSize to estimate the size of columns in the 
> Partition batches created by HashAgg.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (DRILL-4708) connection closed unexpectedly

[Pritesh Maker (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-4708?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pritesh Maker resolved DRILL-4708.
--
Resolution: Duplicate

DRILL-5902

> connection closed unexpectedly
> --
>
> Key: DRILL-4708
> URL: https://issues.apache.org/jira/browse/DRILL-4708
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Execution - RPC
>Affects Versions: 1.7.0
>Reporter: Chun Chang
>Assignee: Vlad Rozov
>Priority: Critical
> Fix For: 1.13.0
>
> Attachments: data.tgz
>
>
> Running DRILL functional automation, we often see query failed randomly due 
> to the following unexpected connection close error.
> {noformat}
> Execution Failures:
> /root/drillAutomation/framework/framework/resources/Functional/ctas/ctas_flatten/10rows/filter5.q
> Query: 
> select * from dfs.ctas_flatten.`filter5_10rows_ctas`
> Failed with exception
> java.sql.SQLException: CONNECTION ERROR: Connection /10.10.100.171:36185 <--> 
> drillats4.qa.lab/10.10.100.174:31010 (user client) closed unexpectedly. 
> Drillbit down?
> [Error Id: 3d5dad8e-80d0-4c7f-9012-013bf01ce2b7 ]
>   at 
> org.apache.drill.jdbc.impl.DrillCursor.nextRowInternally(DrillCursor.java:247)
>   at org.apache.drill.jdbc.impl.DrillCursor.next(DrillCursor.java:321)
>   at 
> oadd.net.hydromatic.avatica.AvaticaResultSet.next(AvaticaResultSet.java:187)
>   at 
> org.apache.drill.jdbc.impl.DrillResultSetImpl.next(DrillResultSetImpl.java:172)
>   at 
> org.apache.drill.test.framework.DrillTestJdbc.executeQuery(DrillTestJdbc.java:210)
>   at 
> org.apache.drill.test.framework.DrillTestJdbc.run(DrillTestJdbc.java:99)
>   at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>   at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>   at java.lang.Thread.run(Thread.java:744)
> Caused by: oadd.org.apache.drill.common.exceptions.UserException: CONNECTION 
> ERROR: Connection /10.10.100.171:36185 <--> 
> drillats4.qa.lab/10.10.100.174:31010 (user client) closed unexpectedly. 
> Drillbit down?
> [Error Id: 3d5dad8e-80d0-4c7f-9012-013bf01ce2b7 ]
>   at 
> oadd.org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:543)
>   at 
> oadd.org.apache.drill.exec.rpc.user.QueryResultHandler$ChannelClosedHandler$1.operationComplete(QueryResultHandler.java:373)
>   at 
> oadd.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:680)
>   at 
> oadd.io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:603)
>   at 
> oadd.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:563)
>   at 
> oadd.io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:406)
>   at 
> oadd.io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:82)
>   at 
> oadd.io.netty.channel.AbstractChannel$CloseFuture.setClosed(AbstractChannel.java:943)
>   at 
> oadd.io.netty.channel.AbstractChannel$AbstractUnsafe.doClose0(AbstractChannel.java:592)
>   at 
> oadd.io.netty.channel.AbstractChannel$AbstractUnsafe.close(AbstractChannel.java:584)
>   at 
> oadd.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.closeOnRead(AbstractNioByteChannel.java:71)
>   at 
> oadd.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.handleReadException(AbstractNioByteChannel.java:89)
>   at 
> oadd.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:162)
>   at 
> oadd.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511)
>   at 
> oadd.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)
>   at 
> oadd.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)
>   at oadd.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)
>   at 
> oadd.io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
>   ... 1 more
> {noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (DRILL-4708) connection closed unexpectedly

[Pritesh Maker (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16379570#comment-16379570
 ] 

Pritesh Maker commented on DRILL-4708:
--

[~cch...@maprtech.com] DRILL-5902 is merged, please retest.

> connection closed unexpectedly
> --
>
> Key: DRILL-4708
> URL: https://issues.apache.org/jira/browse/DRILL-4708
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Execution - RPC
>Affects Versions: 1.7.0
>Reporter: Chun Chang
>Assignee: Vlad Rozov
>Priority: Critical
> Fix For: 1.13.0
>
> Attachments: data.tgz
>
>
> Running DRILL functional automation, we often see query failed randomly due 
> to the following unexpected connection close error.
> {noformat}
> Execution Failures:
> /root/drillAutomation/framework/framework/resources/Functional/ctas/ctas_flatten/10rows/filter5.q
> Query: 
> select * from dfs.ctas_flatten.`filter5_10rows_ctas`
> Failed with exception
> java.sql.SQLException: CONNECTION ERROR: Connection /10.10.100.171:36185 <--> 
> drillats4.qa.lab/10.10.100.174:31010 (user client) closed unexpectedly. 
> Drillbit down?
> [Error Id: 3d5dad8e-80d0-4c7f-9012-013bf01ce2b7 ]
>   at 
> org.apache.drill.jdbc.impl.DrillCursor.nextRowInternally(DrillCursor.java:247)
>   at org.apache.drill.jdbc.impl.DrillCursor.next(DrillCursor.java:321)
>   at 
> oadd.net.hydromatic.avatica.AvaticaResultSet.next(AvaticaResultSet.java:187)
>   at 
> org.apache.drill.jdbc.impl.DrillResultSetImpl.next(DrillResultSetImpl.java:172)
>   at 
> org.apache.drill.test.framework.DrillTestJdbc.executeQuery(DrillTestJdbc.java:210)
>   at 
> org.apache.drill.test.framework.DrillTestJdbc.run(DrillTestJdbc.java:99)
>   at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>   at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>   at java.lang.Thread.run(Thread.java:744)
> Caused by: oadd.org.apache.drill.common.exceptions.UserException: CONNECTION 
> ERROR: Connection /10.10.100.171:36185 <--> 
> drillats4.qa.lab/10.10.100.174:31010 (user client) closed unexpectedly. 
> Drillbit down?
> [Error Id: 3d5dad8e-80d0-4c7f-9012-013bf01ce2b7 ]
>   at 
> oadd.org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:543)
>   at 
> oadd.org.apache.drill.exec.rpc.user.QueryResultHandler$ChannelClosedHandler$1.operationComplete(QueryResultHandler.java:373)
>   at 
> oadd.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:680)
>   at 
> oadd.io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:603)
>   at 
> oadd.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:563)
>   at 
> oadd.io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:406)
>   at 
> oadd.io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:82)
>   at 
> oadd.io.netty.channel.AbstractChannel$CloseFuture.setClosed(AbstractChannel.java:943)
>   at 
> oadd.io.netty.channel.AbstractChannel$AbstractUnsafe.doClose0(AbstractChannel.java:592)
>   at 
> oadd.io.netty.channel.AbstractChannel$AbstractUnsafe.close(AbstractChannel.java:584)
>   at 
> oadd.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.closeOnRead(AbstractNioByteChannel.java:71)
>   at 
> oadd.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.handleReadException(AbstractNioByteChannel.java:89)
>   at 
> oadd.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:162)
>   at 
> oadd.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511)
>   at 
> oadd.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)
>   at 
> oadd.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)
>   at oadd.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)
>   at 
> oadd.io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
>   ... 1 more
> {noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (DRILL-4708) connection closed unexpectedly

[Pritesh Maker (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-4708?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pritesh Maker updated DRILL-4708:
-
Fix Version/s: 1.13.0

> connection closed unexpectedly
> --
>
> Key: DRILL-4708
> URL: https://issues.apache.org/jira/browse/DRILL-4708
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Execution - RPC
>Affects Versions: 1.7.0
>Reporter: Chun Chang
>Assignee: Vlad Rozov
>Priority: Critical
> Fix For: 1.13.0
>
> Attachments: data.tgz
>
>
> Running DRILL functional automation, we often see query failed randomly due 
> to the following unexpected connection close error.
> {noformat}
> Execution Failures:
> /root/drillAutomation/framework/framework/resources/Functional/ctas/ctas_flatten/10rows/filter5.q
> Query: 
> select * from dfs.ctas_flatten.`filter5_10rows_ctas`
> Failed with exception
> java.sql.SQLException: CONNECTION ERROR: Connection /10.10.100.171:36185 <--> 
> drillats4.qa.lab/10.10.100.174:31010 (user client) closed unexpectedly. 
> Drillbit down?
> [Error Id: 3d5dad8e-80d0-4c7f-9012-013bf01ce2b7 ]
>   at 
> org.apache.drill.jdbc.impl.DrillCursor.nextRowInternally(DrillCursor.java:247)
>   at org.apache.drill.jdbc.impl.DrillCursor.next(DrillCursor.java:321)
>   at 
> oadd.net.hydromatic.avatica.AvaticaResultSet.next(AvaticaResultSet.java:187)
>   at 
> org.apache.drill.jdbc.impl.DrillResultSetImpl.next(DrillResultSetImpl.java:172)
>   at 
> org.apache.drill.test.framework.DrillTestJdbc.executeQuery(DrillTestJdbc.java:210)
>   at 
> org.apache.drill.test.framework.DrillTestJdbc.run(DrillTestJdbc.java:99)
>   at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>   at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>   at java.lang.Thread.run(Thread.java:744)
> Caused by: oadd.org.apache.drill.common.exceptions.UserException: CONNECTION 
> ERROR: Connection /10.10.100.171:36185 <--> 
> drillats4.qa.lab/10.10.100.174:31010 (user client) closed unexpectedly. 
> Drillbit down?
> [Error Id: 3d5dad8e-80d0-4c7f-9012-013bf01ce2b7 ]
>   at 
> oadd.org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:543)
>   at 
> oadd.org.apache.drill.exec.rpc.user.QueryResultHandler$ChannelClosedHandler$1.operationComplete(QueryResultHandler.java:373)
>   at 
> oadd.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:680)
>   at 
> oadd.io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:603)
>   at 
> oadd.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:563)
>   at 
> oadd.io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:406)
>   at 
> oadd.io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:82)
>   at 
> oadd.io.netty.channel.AbstractChannel$CloseFuture.setClosed(AbstractChannel.java:943)
>   at 
> oadd.io.netty.channel.AbstractChannel$AbstractUnsafe.doClose0(AbstractChannel.java:592)
>   at 
> oadd.io.netty.channel.AbstractChannel$AbstractUnsafe.close(AbstractChannel.java:584)
>   at 
> oadd.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.closeOnRead(AbstractNioByteChannel.java:71)
>   at 
> oadd.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.handleReadException(AbstractNioByteChannel.java:89)
>   at 
> oadd.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:162)
>   at 
> oadd.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511)
>   at 
> oadd.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)
>   at 
> oadd.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)
>   at oadd.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)
>   at 
> oadd.io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
>   ... 1 more
> {noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (DRILL-6151) Fragment executors may terminate without sending final batch to a downstream causing query to hang

[Pritesh Maker (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-6151?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pritesh Maker updated DRILL-6151:
-
Fix Version/s: 1.13.0

> Fragment executors may terminate without sending final batch to a downstream 
> causing query to hang
> --
>
> Key: DRILL-6151
> URL: https://issues.apache.org/jira/browse/DRILL-6151
> Project: Apache Drill
>  Issue Type: Bug
>Reporter: Vlad Rozov
>Priority: Major
> Fix For: 1.13.0
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (DRILL-6151) Fragment executors may terminate without sending final batch to a downstream causing query to hang

[Pritesh Maker (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-6151?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pritesh Maker reassigned DRILL-6151:


Assignee: Vlad Rozov

> Fragment executors may terminate without sending final batch to a downstream 
> causing query to hang
> --
>
> Key: DRILL-6151
> URL: https://issues.apache.org/jira/browse/DRILL-6151
> Project: Apache Drill
>  Issue Type: Bug
>Reporter: Vlad Rozov
>Assignee: Vlad Rozov
>Priority: Major
> Fix For: 1.13.0
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (DRILL-6193) Latest Calcite optimized out join condition and cause "This query cannot be planned possibly due to either a cartesian join or an inequality join"

[Chunhui Shi (JIRA)]

Chunhui Shi created DRILL-6193:
--

 Summary: Latest Calcite optimized out join condition and cause 
"This query cannot be planned possibly due to either a cartesian join or an 
inequality join"
 Key: DRILL-6193
 URL: https://issues.apache.org/jira/browse/DRILL-6193
 Project: Apache Drill
  Issue Type: Bug
  Components: Query Planning  Optimization
Affects Versions: 1.13.0
Reporter: Chunhui Shi
Assignee: Hanumath Rao Maduri
 Fix For: 1.13.0


I got the same error on apache master's MapR profile on the tip(before Hive 
upgrade) and on changeset 9e944c97ee6f6c0d1705f09d531af35deed2e310, the last 
commit of Calcite upgrade with the failed query reported in functional test but 
now it is on parquet file:
 
{quote}SELECT L.L_QUANTITY, L.L_DISCOUNT, L.L_EXTENDEDPRICE, L.L_TAX
 
FROM cp.`tpch/lineitem.parquet` L, cp.`tpch/orders.parquet` O
WHERE cast(L.L_ORDERKEY as int) = cast(O.O_ORDERKEY as int) AND 
cast(L.L_LINENUMBER as int) = 7 AND cast(L.L_ORDERKEY as int) = 10208 AND 
cast(O.O_ORDERKEY as int) = 10208;
 {quote}
However, built Drill on commit ef0fafea214e866556fa39c902685d48a56001e1, the 
commit right before Calcite upgrade commits, the same query worked.

This was caused by latest Calcite simplified the predicates and during this 
process, "cast(L.L_ORDERKEY as int) = cast(O.O_ORDERKEY as int) " was 
considered redundant and was removed, so the logical plan of this query is 
getting an always true condition for Join:
{quote}DrillJoinRel(condition=[true], joinType=[inner])
{quote}
While in previous version we have 
{quote}DrillJoinRel(condition=[=($5, $0)], joinType=[inner])
{quote}
 

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (DRILL-5270) Improve loading of profiles listing in the WebUI

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5270?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16379407#comment-16379407
 ] 

ASF GitHub Bot commented on DRILL-5270:
---

Github user arina-ielchiieva commented on a diff in the pull request:

https://github.com/apache/drill/pull/755#discussion_r171084819
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/store/sys/store/LocalPersistentStore.java
 ---
@@ -112,23 +127,65 @@ public static DrillFileSystem 
getFileSystem(DrillConfig config, Path root) throw
 
   @Override
   public Iterator> getRange(int skip, int take) {
+//Marking currently seen modification time
+long currBasePathModified = 0L;
+try {
+  currBasePathModified = 
fs.getFileStatus(basePath).getModificationTime();
+} catch (IOException ioexcp) {
+  ioexcp.printStackTrace();
--- End diff --

Please do not use `printStackTrace()`


> Improve loading of profiles listing in the WebUI
> 
>
> Key: DRILL-5270
> URL: https://issues.apache.org/jira/browse/DRILL-5270
> Project: Apache Drill
>  Issue Type: Improvement
>  Components: Web Server
>Affects Versions: 1.9.0
>Reporter: Kunal Khatua
>Assignee: Kunal Khatua
>Priority: Major
> Fix For: 1.13.0
>
>
> Currently, as the number of profiles increase, we reload the same list of 
> profiles from the FS.
> An ideal improvement would be to detect if there are any new profiles and 
> only reload from the disk then. Otherwise, a cached list is sufficient.
> For a directory of 280K profiles, the load time is close to 6 seconds on a 32 
> core server. With the caching, we can get it down to as much as a few 
> milliseconds.
> To render the cache as invalid, we inspect the last modified time of the 
> directory to confirm whether a reload is needed. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (DRILL-5270) Improve loading of profiles listing in the WebUI

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5270?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16379410#comment-16379410
 ] 

ASF GitHub Bot commented on DRILL-5270:
---

Github user arina-ielchiieva commented on a diff in the pull request:

https://github.com/apache/drill/pull/755#discussion_r171084274
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/store/sys/store/LocalPersistentStore.java
 ---
@@ -61,16 +63,29 @@
   private final AutoCloseableLock readLock = new 
AutoCloseableLock(readWriteLock.readLock());
   private final AutoCloseableLock writeLock = new 
AutoCloseableLock(readWriteLock.writeLock());
 
+  //Provides a threshold above which we report the time to load
+  private static final long LISTTIME_THRESHOLD_MSEC = 2000L;
--- End diff --

`LISTTIME_THRESHOLD_MSEC` -> `LIST_TIME_THRESHOLD_MSEC`


> Improve loading of profiles listing in the WebUI
> 
>
> Key: DRILL-5270
> URL: https://issues.apache.org/jira/browse/DRILL-5270
> Project: Apache Drill
>  Issue Type: Improvement
>  Components: Web Server
>Affects Versions: 1.9.0
>Reporter: Kunal Khatua
>Assignee: Kunal Khatua
>Priority: Major
> Fix For: 1.13.0
>
>
> Currently, as the number of profiles increase, we reload the same list of 
> profiles from the FS.
> An ideal improvement would be to detect if there are any new profiles and 
> only reload from the disk then. Otherwise, a cached list is sufficient.
> For a directory of 280K profiles, the load time is close to 6 seconds on a 32 
> core server. With the caching, we can get it down to as much as a few 
> milliseconds.
> To render the cache as invalid, we inspect the last modified time of the 
> directory to confirm whether a reload is needed. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (DRILL-5270) Improve loading of profiles listing in the WebUI

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5270?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16379412#comment-16379412
 ] 

ASF GitHub Bot commented on DRILL-5270:
---

Github user arina-ielchiieva commented on a diff in the pull request:

https://github.com/apache/drill/pull/755#discussion_r171089532
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/store/sys/store/DrillSysFilePathFilter.java
 ---
@@ -0,0 +1,53 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.sys.store;
+
+import static org.apache.drill.exec.ExecConstants.DRILL_SYS_FILE_SUFFIX;
+
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.PathFilter;
+
+/**
+ * Filter for Drill System Files
+ */
+public class DrillSysFilePathFilter implements PathFilter {
--- End diff --

Please consider using `FileSystemUtil` which help to create filters. 
Passing custom filter is also possible.


> Improve loading of profiles listing in the WebUI
> 
>
> Key: DRILL-5270
> URL: https://issues.apache.org/jira/browse/DRILL-5270
> Project: Apache Drill
>  Issue Type: Improvement
>  Components: Web Server
>Affects Versions: 1.9.0
>Reporter: Kunal Khatua
>Assignee: Kunal Khatua
>Priority: Major
> Fix For: 1.13.0
>
>
> Currently, as the number of profiles increase, we reload the same list of 
> profiles from the FS.
> An ideal improvement would be to detect if there are any new profiles and 
> only reload from the disk then. Otherwise, a cached list is sufficient.
> For a directory of 280K profiles, the load time is close to 6 seconds on a 32 
> core server. With the caching, we can get it down to as much as a few 
> milliseconds.
> To render the cache as invalid, we inspect the last modified time of the 
> directory to confirm whether a reload is needed. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (DRILL-5270) Improve loading of profiles listing in the WebUI

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5270?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16379411#comment-16379411
 ] 

ASF GitHub Bot commented on DRILL-5270:
---

Github user arina-ielchiieva commented on a diff in the pull request:

https://github.com/apache/drill/pull/755#discussion_r171089128
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/store/sys/store/LocalPersistentStore.java
 ---
@@ -141,11 +198,33 @@ public static DrillFileSystem 
getFileSystem(DrillConfig config, Path root) throw
 }
   }
 
+  /**
+   * Add profile name to a TreeSet
+   * @param profileName
--- End diff --

Please do not leave `@param`, `@return` without description. IDE usually 
highlights them, asking to add description. 


> Improve loading of profiles listing in the WebUI
> 
>
> Key: DRILL-5270
> URL: https://issues.apache.org/jira/browse/DRILL-5270
> Project: Apache Drill
>  Issue Type: Improvement
>  Components: Web Server
>Affects Versions: 1.9.0
>Reporter: Kunal Khatua
>Assignee: Kunal Khatua
>Priority: Major
> Fix For: 1.13.0
>
>
> Currently, as the number of profiles increase, we reload the same list of 
> profiles from the FS.
> An ideal improvement would be to detect if there are any new profiles and 
> only reload from the disk then. Otherwise, a cached list is sufficient.
> For a directory of 280K profiles, the load time is close to 6 seconds on a 32 
> core server. With the caching, we can get it down to as much as a few 
> milliseconds.
> To render the cache as invalid, we inspect the last modified time of the 
> directory to confirm whether a reload is needed. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (DRILL-5270) Improve loading of profiles listing in the WebUI

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5270?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16379408#comment-16379408
 ] 

ASF GitHub Bot commented on DRILL-5270:
---

Github user arina-ielchiieva commented on a diff in the pull request:

https://github.com/apache/drill/pull/755#discussion_r171088565
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/store/sys/store/LocalPersistentStore.java
 ---
@@ -112,23 +127,65 @@ public static DrillFileSystem 
getFileSystem(DrillConfig config, Path root) throw
 
   @Override
   public Iterator> getRange(int skip, int take) {
+//Marking currently seen modification time
+long currBasePathModified = 0L;
+try {
+  currBasePathModified = 
fs.getFileStatus(basePath).getModificationTime();
+} catch (IOException ioexcp) {
+  ioexcp.printStackTrace();
+}
+
+//Acquiring lock to avoid reloading for request coming in before 
completion of profile read
--- End diff --

1. Before reading lock acquirement was enough, with your changes you modify 
class fields. Since many threads can access this method, you'll end up with 
raise conditions, also class fields can be cached by threads as well... I think 
design here should be reconsidered.
2. Guava library has several cache implementations. Can we leverage any of 
them instead of using tree set?

Pinging @vlad since he is working on DRILL-6053 which intends to make 
changes in the same class to avoid excessive locking to be aware of intended 
changes.



> Improve loading of profiles listing in the WebUI
> 
>
> Key: DRILL-5270
> URL: https://issues.apache.org/jira/browse/DRILL-5270
> Project: Apache Drill
>  Issue Type: Improvement
>  Components: Web Server
>Affects Versions: 1.9.0
>Reporter: Kunal Khatua
>Assignee: Kunal Khatua
>Priority: Major
> Fix For: 1.13.0
>
>
> Currently, as the number of profiles increase, we reload the same list of 
> profiles from the FS.
> An ideal improvement would be to detect if there are any new profiles and 
> only reload from the disk then. Otherwise, a cached list is sufficient.
> For a directory of 280K profiles, the load time is close to 6 seconds on a 32 
> core server. With the caching, we can get it down to as much as a few 
> milliseconds.
> To render the cache as invalid, we inspect the last modified time of the 
> directory to confirm whether a reload is needed. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (DRILL-5270) Improve loading of profiles listing in the WebUI

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5270?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16379409#comment-16379409
 ] 

ASF GitHub Bot commented on DRILL-5270:
---

Github user arina-ielchiieva commented on a diff in the pull request:

https://github.com/apache/drill/pull/755#discussion_r171084169
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/store/sys/store/LocalPersistentStore.java
 ---
@@ -61,16 +63,29 @@
   private final AutoCloseableLock readLock = new 
AutoCloseableLock(readWriteLock.readLock());
   private final AutoCloseableLock writeLock = new 
AutoCloseableLock(readWriteLock.writeLock());
 
+  //Provides a threshold above which we report the time to load
+  private static final long LISTTIME_THRESHOLD_MSEC = 2000L;
+
+  private static final int DrillSysFileExtSize = 
DRILL_SYS_FILE_SUFFIX.length();
--- End diff --

`DrillSysFileExtSize` -> `drillSysFileExtSize`


> Improve loading of profiles listing in the WebUI
> 
>
> Key: DRILL-5270
> URL: https://issues.apache.org/jira/browse/DRILL-5270
> Project: Apache Drill
>  Issue Type: Improvement
>  Components: Web Server
>Affects Versions: 1.9.0
>Reporter: Kunal Khatua
>Assignee: Kunal Khatua
>Priority: Major
> Fix For: 1.13.0
>
>
> Currently, as the number of profiles increase, we reload the same list of 
> profiles from the FS.
> An ideal improvement would be to detect if there are any new profiles and 
> only reload from the disk then. Otherwise, a cached list is sufficient.
> For a directory of 280K profiles, the load time is close to 6 seconds on a 32 
> core server. With the caching, we can get it down to as much as a few 
> milliseconds.
> To render the cache as invalid, we inspect the last modified time of the 
> directory to confirm whether a reload is needed. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (DRILL-5270) Improve loading of profiles listing in the WebUI

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5270?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16379406#comment-16379406
 ] 

ASF GitHub Bot commented on DRILL-5270:
---

Github user arina-ielchiieva commented on a diff in the pull request:

https://github.com/apache/drill/pull/755#discussion_r171082847
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/store/sys/store/DrillSysFilePathFilter.java
 ---
@@ -0,0 +1,53 @@
+/**
--- End diff --

Please use comment for the header, not javadoc.


> Improve loading of profiles listing in the WebUI
> 
>
> Key: DRILL-5270
> URL: https://issues.apache.org/jira/browse/DRILL-5270
> Project: Apache Drill
>  Issue Type: Improvement
>  Components: Web Server
>Affects Versions: 1.9.0
>Reporter: Kunal Khatua
>Assignee: Kunal Khatua
>Priority: Major
> Fix For: 1.13.0
>
>
> Currently, as the number of profiles increase, we reload the same list of 
> profiles from the FS.
> An ideal improvement would be to detect if there are any new profiles and 
> only reload from the disk then. Otherwise, a cached list is sufficient.
> For a directory of 280K profiles, the load time is close to 6 seconds on a 32 
> core server. With the caching, we can get it down to as much as a few 
> milliseconds.
> To render the cache as invalid, we inspect the last modified time of the 
> directory to confirm whether a reload is needed. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (DRILL-6099) Drill does not push limit past project (flatten) if it cannot be pushed into scan

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-6099?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16379395#comment-16379395
 ] 

ASF GitHub Bot commented on DRILL-6099:
---

Github user gparai commented on a diff in the pull request:

https://github.com/apache/drill/pull/1096#discussion_r171086924
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/planner/logical/DrillPushLimitToScanRule.java
 ---
@@ -55,18 +62,21 @@ public void onMatch(RelOptRuleCall call) {
 }
   };
 
-  public static DrillPushLimitToScanRule LIMIT_ON_PROJECT =
-  new DrillPushLimitToScanRule(
-  RelOptHelper.some(DrillLimitRel.class, RelOptHelper.some(
-  DrillProjectRel.class, 
RelOptHelper.any(DrillScanRel.class))),
-  "DrillPushLimitToScanRule_LimitOnProject") {
+  public static DrillPushLimitToScanRule LIMIT_ON_PROJECT = new 
DrillPushLimitToScanRule(
--- End diff --

There are many instances where we would have a PROJECT on top of the SCAN. 
The way the rule is refactored now the LIMIT_SCAN rule would not work unless we 
do LIMIT_PROJECT. Hence, these rules should go together for LIMIT_SCAN to work 
effectively. That is the reason I kept the rule here rather than creating a new 
rule.


> Drill does not push limit past project (flatten) if it cannot be pushed into 
> scan
> -
>
> Key: DRILL-6099
> URL: https://issues.apache.org/jira/browse/DRILL-6099
> Project: Apache Drill
>  Issue Type: Bug
>Affects Versions: 1.12.0
>Reporter: Gautam Kumar Parai
>Assignee: Gautam Kumar Parai
>Priority: Major
> Fix For: 1.13.0
>
>
> It would be useful to have pushdown occur past flatten(project). Here is an 
> example to illustrate the issue:
> {{explain plan without implementation for }}{{select name, 
> flatten(categories) as category from dfs.`/tmp/t_json_20` LIMIT 1;}}
> {{DrillScreenRel}}{{  }}
> {{  DrillLimitRel(fetch=[1])}}{{    }}
> {{    DrillProjectRel(name=[$0], category=[FLATTEN($1)])}}
> {{      DrillScanRel(table=[[dfs, /tmp/t_json_20]], groupscan=[EasyGroupScan 
> [selectionRoot=maprfs:/tmp/t_json_20, numFiles=1, columns=[`name`, 
> `categories`], files=[maprfs:///tmp/t_json_20/0_0_0.json]]])}}
> = 
> Content of 0_0_0.json
> =
> {
>   "name" : "Eric Goldberg, MD",
>   "categories" : [ "Doctors", "Health & Medical" ]
> } {
>   "name" : "Pine Cone Restaurant",
>   "categories" : [ "Restaurants" ]
> } {
>   "name" : "Deforest Family Restaurant",
>   "categories" : [ "American (Traditional)", "Restaurants" ]
> } {
>   "name" : "Culver's",
>   "categories" : [ "Food", "Ice Cream & Frozen Yogurt", "Fast Food", 
> "Restaurants" ]
> } {
>   "name" : "Chang Jiang Chinese Kitchen",
>   "categories" : [ "Chinese", "Restaurants" ]
> } 
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (DRILL-6099) Drill does not push limit past project (flatten) if it cannot be pushed into scan

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-6099?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16379390#comment-16379390
 ] 

ASF GitHub Bot commented on DRILL-6099:
---

Github user gparai commented on a diff in the pull request:

https://github.com/apache/drill/pull/1096#discussion_r171085780
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/planner/logical/DrillPushLimitToScanRule.java
 ---
@@ -55,18 +62,21 @@ public void onMatch(RelOptRuleCall call) {
 }
   };
 
-  public static DrillPushLimitToScanRule LIMIT_ON_PROJECT =
-  new DrillPushLimitToScanRule(
-  RelOptHelper.some(DrillLimitRel.class, RelOptHelper.some(
-  DrillProjectRel.class, 
RelOptHelper.any(DrillScanRel.class))),
-  "DrillPushLimitToScanRule_LimitOnProject") {
+  public static DrillPushLimitToScanRule LIMIT_ON_PROJECT = new 
DrillPushLimitToScanRule(
+  RelOptHelper.some(DrillLimitRel.class, 
RelOptHelper.any(DrillProjectRel.class)), 
"DrillPushLimitToScanRule_LimitOnProject") {
 @Override
 public boolean matches(RelOptRuleCall call) {
   DrillLimitRel limitRel = call.rel(0);
-  DrillScanRel scanRel = call.rel(2);
-  // For now only applies to Parquet. And pushdown only apply limit 
but not offset,
+  DrillProjectRel projectRel = call.rel(1);
+  // pushdown only apply limit but not offset,
   // so if getFetch() return null no need to run this rule.
-  if (scanRel.getGroupScan().supportsLimitPushdown() && 
(limitRel.getFetch() != null)) {
--- End diff --

We still have the LIMIT_ON_SCAN rule which does that check. This rule is 
changed from LIMIT_PROJECT_SCAN to LIMIT_PROJECT. The LIMIT_SCAN along with the 
LIMIT_PROJECT would work as the LIMIT_PROJECT_SCAN.


> Drill does not push limit past project (flatten) if it cannot be pushed into 
> scan
> -
>
> Key: DRILL-6099
> URL: https://issues.apache.org/jira/browse/DRILL-6099
> Project: Apache Drill
>  Issue Type: Bug
>Affects Versions: 1.12.0
>Reporter: Gautam Kumar Parai
>Assignee: Gautam Kumar Parai
>Priority: Major
> Fix For: 1.13.0
>
>
> It would be useful to have pushdown occur past flatten(project). Here is an 
> example to illustrate the issue:
> {{explain plan without implementation for }}{{select name, 
> flatten(categories) as category from dfs.`/tmp/t_json_20` LIMIT 1;}}
> {{DrillScreenRel}}{{  }}
> {{  DrillLimitRel(fetch=[1])}}{{    }}
> {{    DrillProjectRel(name=[$0], category=[FLATTEN($1)])}}
> {{      DrillScanRel(table=[[dfs, /tmp/t_json_20]], groupscan=[EasyGroupScan 
> [selectionRoot=maprfs:/tmp/t_json_20, numFiles=1, columns=[`name`, 
> `categories`], files=[maprfs:///tmp/t_json_20/0_0_0.json]]])}}
> = 
> Content of 0_0_0.json
> =
> {
>   "name" : "Eric Goldberg, MD",
>   "categories" : [ "Doctors", "Health & Medical" ]
> } {
>   "name" : "Pine Cone Restaurant",
>   "categories" : [ "Restaurants" ]
> } {
>   "name" : "Deforest Family Restaurant",
>   "categories" : [ "American (Traditional)", "Restaurants" ]
> } {
>   "name" : "Culver's",
>   "categories" : [ "Food", "Ice Cream & Frozen Yogurt", "Fast Food", 
> "Restaurants" ]
> } {
>   "name" : "Chang Jiang Chinese Kitchen",
>   "categories" : [ "Chinese", "Restaurants" ]
> } 
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (DRILL-6021) Show shutdown button when authentication is not enabled

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-6021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16379355#comment-16379355
 ] 

ASF GitHub Bot commented on DRILL-6021:
---

Github user arina-ielchiieva commented on a diff in the pull request:

https://github.com/apache/drill/pull/1127#discussion_r171075503
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/server/rest/DrillRoot.java 
---
@@ -411,6 +415,8 @@ public String getCurrentVersion() {
   public boolean shouldShowAdminInfo() { return shouldShowAdminInfo; }
 
   public QueueInfo queueInfo() { return queueInfo; }
+
+  public boolean authEnabled() { return  authEnabled; }
--- End diff --

Please rename `authEnabled` -> `isAuthEnabled`.


> Show shutdown button when authentication is not enabled
> ---
>
> Key: DRILL-6021
> URL: https://issues.apache.org/jira/browse/DRILL-6021
> Project: Apache Drill
>  Issue Type: Bug
>Affects Versions: 1.12.0
>Reporter: Arina Ielchiieva
>Assignee: Venkata Jyothsna Donapati
>Priority: Major
> Fix For: 1.13.0
>
>
> After DRILL-6017 {{shouldShowAdminInfo}} is used to decide if shutdown button 
> should be displayed on index page. But this option is set to true when 
> authentication is enabled and user is an admin. When authentication is not 
> enabled, user by default is admin. So with this fix without authentication, 
> shutdown button is absent but should be present.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (DRILL-6021) Show shutdown button when authentication is not enabled

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-6021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16379354#comment-16379354
 ] 

ASF GitHub Bot commented on DRILL-6021:
---

Github user arina-ielchiieva commented on a diff in the pull request:

https://github.com/apache/drill/pull/1127#discussion_r171075361
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/server/rest/DrillRoot.java 
---
@@ -355,14 +356,16 @@ public ClusterInfo(Collection drillbits,
  boolean userEncryption,
  boolean bitEncryption,
  boolean shouldShowAdminInfo,
- QueueInfo queueInfo) {
+ QueueInfo queueInfo,
+ boolean authEnabled ) {
--- End diff --

Please remove space after `authEnabled` and before `)`.


> Show shutdown button when authentication is not enabled
> ---
>
> Key: DRILL-6021
> URL: https://issues.apache.org/jira/browse/DRILL-6021
> Project: Apache Drill
>  Issue Type: Bug
>Affects Versions: 1.12.0
>Reporter: Arina Ielchiieva
>Assignee: Venkata Jyothsna Donapati
>Priority: Major
> Fix For: 1.13.0
>
>
> After DRILL-6017 {{shouldShowAdminInfo}} is used to decide if shutdown button 
> should be displayed on index page. But this option is set to true when 
> authentication is enabled and user is an admin. When authentication is not 
> enabled, user by default is admin. So with this fix without authentication, 
> shutdown button is absent but should be present.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (DRILL-6185) Error is displaying while accessing query profiles via the Web-UI

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-6185?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16379350#comment-16379350
 ] 

ASF GitHub Bot commented on DRILL-6185:
---

Github user arina-ielchiieva commented on a diff in the pull request:

https://github.com/apache/drill/pull/1137#discussion_r171071136
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/server/rest/profile/ProfileWrapper.java
 ---
@@ -330,10 +332,16 @@ public boolean isOnlyImpersonationEnabled() {
   //Generates operator names inferred from physical plan
   private void generateOpMap(String plan) {
 this.physicalOperatorMap = new HashMap();
--- End diff --

Please replace with diamonds -> `new HashMap<>();`


> Error is displaying while accessing query profiles via the Web-UI
> -
>
> Key: DRILL-6185
> URL: https://issues.apache.org/jira/browse/DRILL-6185
> Project: Apache Drill
>  Issue Type: Bug
>Affects Versions: 1.13.0
>Reporter: Anton Gozhiy
>Assignee: Kunal Khatua
>Priority: Blocker
> Fix For: 1.13.0
>
>
> *Steps:*
>  # Execute the following query:
> {code:sql}
> show schemas;
> {code}
> # On the Web-UI, go to the Profiles tab
> # Open the profile for the query you executed
> *Expected result:* You can access to the profile entry
> *Actual result:* Error is displayed:
> {code:json}
> {
>   "errorMessage" : "1"
> }
> {code}
> *Note:* This error doesn't happen with every query. For example, "select * 
> from system.version" can be accessed without error, while "show tables", "use 
> dfs", "alter sessions" etc end with this error.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (DRILL-6040) Need to add usage for graceful_stop to drillbit.sh

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-6040?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16379343#comment-16379343
 ] 

ASF GitHub Bot commented on DRILL-6040:
---

Github user arina-ielchiieva commented on the issue:

https://github.com/apache/drill/pull/1135
  
+1, LGTM.


> Need to add usage for graceful_stop to drillbit.sh
> --
>
> Key: DRILL-6040
> URL: https://issues.apache.org/jira/browse/DRILL-6040
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Execution - Flow
>Affects Versions: 1.13.0
>Reporter: Krystal
>Assignee: Venkata Jyothsna Donapati
>Priority: Major
>  Labels: ready-to-commit
> Fix For: 1.13.0
>
>
> git.commit.id.abbrev=eb0c403
> Usage for graceful_stop is missing from drillbit.sh.
> ./drillbit.sh
> Usage: drillbit.sh [--config|--site ] 
> (start|stop|status|restart|run) [args]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (DRILL-6040) Need to add usage for graceful_stop to drillbit.sh

[Arina Ielchiieva (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-6040?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Arina Ielchiieva updated DRILL-6040:

Labels: ready-to-commit  (was: )

> Need to add usage for graceful_stop to drillbit.sh
> --
>
> Key: DRILL-6040
> URL: https://issues.apache.org/jira/browse/DRILL-6040
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Execution - Flow
>Affects Versions: 1.13.0
>Reporter: Krystal
>Assignee: Venkata Jyothsna Donapati
>Priority: Major
>  Labels: ready-to-commit
> Fix For: 1.13.0
>
>
> git.commit.id.abbrev=eb0c403
> Usage for graceful_stop is missing from drillbit.sh.
> ./drillbit.sh
> Usage: drillbit.sh [--config|--site ] 
> (start|stop|status|restart|run) [args]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (DRILL-6185) Error is displaying while accessing query profiles via the Web-UI

[Kunal Khatua (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-6185?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16379278#comment-16379278
 ] 

Kunal Khatua commented on DRILL-6185:
-

[~arina] / [~amansinha100] (cc: [~parthc])

Can you please review this?

> Error is displaying while accessing query profiles via the Web-UI
> -
>
> Key: DRILL-6185
> URL: https://issues.apache.org/jira/browse/DRILL-6185
> Project: Apache Drill
>  Issue Type: Bug
>Affects Versions: 1.13.0
>Reporter: Anton Gozhiy
>Assignee: Kunal Khatua
>Priority: Blocker
> Fix For: 1.13.0
>
>
> *Steps:*
>  # Execute the following query:
> {code:sql}
> show schemas;
> {code}
> # On the Web-UI, go to the Profiles tab
> # Open the profile for the query you executed
> *Expected result:* You can access to the profile entry
> *Actual result:* Error is displayed:
> {code:json}
> {
>   "errorMessage" : "1"
> }
> {code}
> *Note:* This error doesn't happen with every query. For example, "select * 
> from system.version" can be accessed without error, while "show tables", "use 
> dfs", "alter sessions" etc end with this error.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (DRILL-6185) Error is displaying while accessing query profiles via the Web-UI

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-6185?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16379276#comment-16379276
 ] 

ASF GitHub Bot commented on DRILL-6185:
---

GitHub user kkhatua opened a pull request:

https://github.com/apache/drill/pull/1137

DRILL-6185: Fixed error while displaying system profiles via the Web-UI

The bug lies in the absence of a text plan for profiles related to system 
queries:
e.g.
```
use dfs.tmp;
show tables;
alter session reset all;
```
This addresses that by ensuring an empty string is substituted and the 
tokenization is done correctly.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kkhatua/drill DRILL-6185

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/drill/pull/1137.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1137


commit aea75f86a945ea901165458707019edebc0dd91a
Author: Kunal Khatua 
Date:   2018-02-27T20:10:06Z

DRILL-6185: Fixed error while displaying system profiles via the Web-UI

The bug lies in the absence of a text plan for profiles related to system 
queries:
e.g.
{code:sql}
use dfs.tmp;
show tables;
alter session reset all;
{code}
This addresses that by ensuring an empty string is substituted and the 
tokenization is done correctly.




> Error is displaying while accessing query profiles via the Web-UI
> -
>
> Key: DRILL-6185
> URL: https://issues.apache.org/jira/browse/DRILL-6185
> Project: Apache Drill
>  Issue Type: Bug
>Affects Versions: 1.13.0
>Reporter: Anton Gozhiy
>Assignee: Kunal Khatua
>Priority: Blocker
> Fix For: 1.13.0
>
>
> *Steps:*
>  # Execute the following query:
> {code:sql}
> show schemas;
> {code}
> # On the Web-UI, go to the Profiles tab
> # Open the profile for the query you executed
> *Expected result:* You can access to the profile entry
> *Actual result:* Error is displayed:
> {code:json}
> {
>   "errorMessage" : "1"
> }
> {code}
> *Note:* This error doesn't happen with every query. For example, "select * 
> from system.version" can be accessed without error, while "show tables", "use 
> dfs", "alter sessions" etc end with this error.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (DRILL-6192) Drill is vulnerable to CVE-2017-12197

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-6192?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16379237#comment-16379237
 ] 

ASF GitHub Bot commented on DRILL-6192:
---

GitHub user vladimirtkach opened a pull request:

https://github.com/apache/drill/pull/1136

DRILL-6192: Drill is vulnerable to CVE-2017-12197

Changed libpam4j version from 1.8-rev1 to 1.9-mapr

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/vladimirtkach/drill DRILL-6192

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/drill/pull/1136.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1136


commit 4d89ac6306923200340576991cb7593261d136d1
Author: vladimir tkach 
Date:   2018-02-27T18:25:28Z

DRILL-6192: Drill is vulnerable to CVE-2017-12197

Changed libpam4j version from 1.8-rev1 to 1.9-mapr




> Drill is vulnerable to CVE-2017-12197
> -
>
> Key: DRILL-6192
> URL: https://issues.apache.org/jira/browse/DRILL-6192
> Project: Apache Drill
>  Issue Type: Bug
>Reporter: Volodymyr Tkach
>Assignee: Volodymyr Tkach
>Priority: Major
>
> The current version of libpam4j bundled with MCS does not perform any 
> authorization check. Any user with valid password could access the cluster 
> even if the user account is disabled/password expired/'not allowed to access 
> the service(pam_access ..)' etc..



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (DRILL-6040) Need to add usage for graceful_stop to drillbit.sh

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-6040?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16379187#comment-16379187
 ] 

ASF GitHub Bot commented on DRILL-6040:
---

GitHub user dvjyothsna opened a pull request:

https://github.com/apache/drill/pull/1135

DRILL-6040: Added usage for graceful_stop in drillbit.sh



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/dvjyothsna/drill DRILL-6040

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/drill/pull/1135.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1135


commit 398cbc26a29b4fb4a2d2cbabffc2cc5b39857da9
Author: dvjyothsna 
Date:   2018-02-22T19:30:50Z

Added usage for graceful_stop in drillbit.sh




> Need to add usage for graceful_stop to drillbit.sh
> --
>
> Key: DRILL-6040
> URL: https://issues.apache.org/jira/browse/DRILL-6040
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Execution - Flow
>Affects Versions: 1.13.0
>Reporter: Krystal
>Assignee: Venkata Jyothsna Donapati
>Priority: Major
> Fix For: 1.13.0
>
>
> git.commit.id.abbrev=eb0c403
> Usage for graceful_stop is missing from drillbit.sh.
> ./drillbit.sh
> Usage: drillbit.sh [--config|--site ] 
> (start|stop|status|restart|run) [args]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (DRILL-6040) Need to add usage for graceful_stop to drillbit.sh

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-6040?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16379175#comment-16379175
 ] 

ASF GitHub Bot commented on DRILL-6040:
---

Github user dvjyothsna closed the pull request at:

https://github.com/apache/drill/pull/1128


> Need to add usage for graceful_stop to drillbit.sh
> --
>
> Key: DRILL-6040
> URL: https://issues.apache.org/jira/browse/DRILL-6040
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Execution - Flow
>Affects Versions: 1.13.0
>Reporter: Krystal
>Assignee: Venkata Jyothsna Donapati
>Priority: Major
> Fix For: 1.13.0
>
>
> git.commit.id.abbrev=eb0c403
> Usage for graceful_stop is missing from drillbit.sh.
> ./drillbit.sh
> Usage: drillbit.sh [--config|--site ] 
> (start|stop|status|restart|run) [args]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (DRILL-6023) Graceful shutdown improvements (umbrella jira)

[Venkata Jyothsna Donapati (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-6023?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16379164#comment-16379164
 ] 

Venkata Jyothsna Donapati commented on DRILL-6023:
--

I'm planning to address the following four jiras for 1.13:

DRILL-6044, DRILL-6040, DRILL-6039, DRILL-6021

> Graceful shutdown improvements (umbrella jira)
> --
>
> Key: DRILL-6023
> URL: https://issues.apache.org/jira/browse/DRILL-6023
> Project: Apache Drill
>  Issue Type: Improvement
>Affects Versions: 1.12.0
>Reporter: Arina Ielchiieva
>Assignee: Venkata Jyothsna Donapati
>Priority: Major
> Fix For: 1.14.0
>
>
> DRILL-4286 introduces graceful shutdown.
> This is umbrella Jira for currently known graceful shutdown improvements.
> All known Jiras are linked.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (DRILL-6153) Revised operator framework

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-6153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16379163#comment-16379163
 ] 

ASF GitHub Bot commented on DRILL-6153:
---

Github user paul-rogers commented on the issue:

https://github.com/apache/drill/pull/1121
  
@arina-ielchiieva, can you do a committer review of this one? 


> Revised operator framework
> --
>
> Key: DRILL-6153
> URL: https://issues.apache.org/jira/browse/DRILL-6153
> Project: Apache Drill
>  Issue Type: Improvement
>Reporter: Paul Rogers
>Assignee: Paul Rogers
>Priority: Major
> Fix For: 1.13.0
>
>
> Adds the core operator framework which is the foundation for the revised scan 
> operators. This is another incremental part of the batch sizing project.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (DRILL-6005) Fix TestGracefulShutdown tests to skip check for loopback address usage in distributed mode

[Pritesh Maker (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-6005?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pritesh Maker updated DRILL-6005:
-
Priority: Minor  (was: Major)

> Fix TestGracefulShutdown tests to skip check for loopback address usage in 
> distributed mode
> ---
>
> Key: DRILL-6005
> URL: https://issues.apache.org/jira/browse/DRILL-6005
> Project: Apache Drill
>  Issue Type: Bug
>Affects Versions: 1.12.0
>Reporter: Arina Ielchiieva
>Assignee: Venkata Jyothsna Donapati
>Priority: Minor
> Fix For: 1.14.0
>
>
> After DRILL-4286 changes some of the newly added unit tests fail with 
> {noformat}
> Drillbit is disallowed to bind to loopback address in distributed mode.
> {noformat}
> List of failed tests:
> {noformat}
> Tests in error: 
>   TestGracefulShutdown.testOnlineEndPoints:96 » IllegalState Cluster fixture 
> set...
>   TestGracefulShutdown.testStateChange:130 » IllegalState Cluster fixture 
> setup ...
>   TestGracefulShutdown.testRestApi:167 » IllegalState Cluster fixture setup 
> fail...
>   TestGracefulShutdown.testRestApiShutdown:207 » IllegalState Cluster fixture 
> se...
> {noformat}
> This can be fixed if {{/etc/hosts}} file is edited.
> Source - 
> https://stackoverflow.com/questions/40506221/how-to-start-drillbit-locally-in-distributed-mode
> Though these changes are required on production during running unit tests I 
> don't think this check should be enforced.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (DRILL-6005) Fix TestGracefulShutdown tests to skip check for loopback address usage in distributed mode

[Pritesh Maker (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-6005?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pritesh Maker updated DRILL-6005:
-
Fix Version/s: (was: 1.13.0)
   1.14.0

> Fix TestGracefulShutdown tests to skip check for loopback address usage in 
> distributed mode
> ---
>
> Key: DRILL-6005
> URL: https://issues.apache.org/jira/browse/DRILL-6005
> Project: Apache Drill
>  Issue Type: Bug
>Affects Versions: 1.12.0
>Reporter: Arina Ielchiieva
>Assignee: Venkata Jyothsna Donapati
>Priority: Minor
> Fix For: 1.14.0
>
>
> After DRILL-4286 changes some of the newly added unit tests fail with 
> {noformat}
> Drillbit is disallowed to bind to loopback address in distributed mode.
> {noformat}
> List of failed tests:
> {noformat}
> Tests in error: 
>   TestGracefulShutdown.testOnlineEndPoints:96 » IllegalState Cluster fixture 
> set...
>   TestGracefulShutdown.testStateChange:130 » IllegalState Cluster fixture 
> setup ...
>   TestGracefulShutdown.testRestApi:167 » IllegalState Cluster fixture setup 
> fail...
>   TestGracefulShutdown.testRestApiShutdown:207 » IllegalState Cluster fixture 
> se...
> {noformat}
> This can be fixed if {{/etc/hosts}} file is edited.
> Source - 
> https://stackoverflow.com/questions/40506221/how-to-start-drillbit-locally-in-distributed-mode
> Though these changes are required on production during running unit tests I 
> don't think this check should be enforced.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (DRILL-6023) Graceful shutdown improvements (umbrella jira)

[Pritesh Maker (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-6023?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pritesh Maker updated DRILL-6023:
-
Fix Version/s: (was: 1.13.0)
   1.14.0

> Graceful shutdown improvements (umbrella jira)
> --
>
> Key: DRILL-6023
> URL: https://issues.apache.org/jira/browse/DRILL-6023
> Project: Apache Drill
>  Issue Type: Improvement
>Affects Versions: 1.12.0
>Reporter: Arina Ielchiieva
>Assignee: Venkata Jyothsna Donapati
>Priority: Major
> Fix For: 1.14.0
>
>
> DRILL-4286 introduces graceful shutdown.
> This is umbrella Jira for currently known graceful shutdown improvements.
> All known Jiras are linked.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (DRILL-6023) Graceful shutdown improvements (umbrella jira)

[Pritesh Maker (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-6023?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pritesh Maker updated DRILL-6023:
-
Summary: Graceful shutdown improvements (umbrella jira)  (was: Graceful 
shutdown improvements)

> Graceful shutdown improvements (umbrella jira)
> --
>
> Key: DRILL-6023
> URL: https://issues.apache.org/jira/browse/DRILL-6023
> Project: Apache Drill
>  Issue Type: Improvement
>Affects Versions: 1.12.0
>Reporter: Arina Ielchiieva
>Assignee: Venkata Jyothsna Donapati
>Priority: Major
> Fix For: 1.14.0
>
>
> DRILL-4286 introduces graceful shutdown.
> This is umbrella Jira for currently known graceful shutdown improvements.
> All known Jiras are linked.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (DRILL-6010) Working drillbit showing as in QUIESCENT state

[Pritesh Maker (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-6010?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pritesh Maker updated DRILL-6010:
-
Fix Version/s: (was: 1.13.0)
   1.14.0

> Working drillbit showing as in QUIESCENT state
> --
>
> Key: DRILL-6010
> URL: https://issues.apache.org/jira/browse/DRILL-6010
> Project: Apache Drill
>  Issue Type: Bug
>Reporter: Arina Ielchiieva
>Assignee: Venkata Jyothsna Donapati
>Priority: Major
> Fix For: 1.14.0
>
> Attachments: online_vs_quiescent.JPG
>
>
> After DRILL-4286 once I got a situation that after running all functional 
> tests three drillbits were in ONLINE state, another one in QUIESCENT. Though 
> from the one in quiescent state I could run queries and so it was online. 
> drillbit.sh stop could not shutdown it and had to do kill -9 of the process 
> (online_vs_quiescent.JPG).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (DRILL-6044) Shutdown button does not work from WebUI

[Pritesh Maker (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-6044?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pritesh Maker updated DRILL-6044:
-
Priority: Critical  (was: Major)

> Shutdown button does not work from WebUI
> 
>
> Key: DRILL-6044
> URL: https://issues.apache.org/jira/browse/DRILL-6044
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Client - HTTP
>Affects Versions: 1.13.0
>Reporter: Krystal
>Assignee: Venkata Jyothsna Donapati
>Priority: Critical
> Fix For: 1.13.0
>
> Attachments: Screen Shot 2017-12-19 at 10.51.16 AM.png
>
>
> git.commit.id.abbrev=eb0c403
> Nothing happens when click on the SHUTDOWN button from the WebUI.  The 
> browser's debugger showed that the request failed due to access control 
> checks (see attached screen shot).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (DRILL-6022) Improve js part for graceful shutdown

[Pritesh Maker (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-6022?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pritesh Maker updated DRILL-6022:
-
Fix Version/s: (was: 1.13.0)
   1.14.0

> Improve js part for graceful shutdown
> -
>
> Key: DRILL-6022
> URL: https://issues.apache.org/jira/browse/DRILL-6022
> Project: Apache Drill
>  Issue Type: Bug
>Affects Versions: 1.12.0
>Reporter: Arina Ielchiieva
>Assignee: Venkata Jyothsna Donapati
>Priority: Major
> Fix For: 1.14.0
>
>
> DRILL-4286 introduces graceful shutdown but its js part needs improvement:
> a. ajax call do not handle errors, so when error occurs it is just swallowed.
> b. there are some unused and / or unnecessary variables usage
> c. shutdown functionality is disabled when user is not an admin but some 
> other ajax calls are still being executed, for example, port number, number 
> of queries, grace period. All that can be also can be disabled when user is 
> not an admin.
> d. there are many ajax calls which can be factored out in dedicated js file.
> Other fixes:
> a. all shutdown functionality reside in DrillRoot class, it can be factored 
> out in shutdown specific class where all shutdown functionality will be 
> allowed only for admin on class level, currently we marked in on the level 
> (see DRILL-6019).
> b. issue described in DRILL-6021.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (DRILL-6040) Need to add usage for graceful_stop to drillbit.sh

[Pritesh Maker (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-6040?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pritesh Maker updated DRILL-6040:
-
Reviewer: Arina Ielchiieva

> Need to add usage for graceful_stop to drillbit.sh
> --
>
> Key: DRILL-6040
> URL: https://issues.apache.org/jira/browse/DRILL-6040
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Execution - Flow
>Affects Versions: 1.13.0
>Reporter: Krystal
>Assignee: Venkata Jyothsna Donapati
>Priority: Major
> Fix For: 1.13.0
>
>
> git.commit.id.abbrev=eb0c403
> Usage for graceful_stop is missing from drillbit.sh.
> ./drillbit.sh
> Usage: drillbit.sh [--config|--site ] 
> (start|stop|status|restart|run) [args]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (DRILL-6099) Drill does not push limit past project (flatten) if it cannot be pushed into scan

[Gautam Kumar Parai (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-6099?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16379154#comment-16379154
 ] 

Gautam Kumar Parai commented on DRILL-6099:
---

[~priteshm] no I did not get a chance to address them yet. I will take a look.

> Drill does not push limit past project (flatten) if it cannot be pushed into 
> scan
> -
>
> Key: DRILL-6099
> URL: https://issues.apache.org/jira/browse/DRILL-6099
> Project: Apache Drill
>  Issue Type: Bug
>Affects Versions: 1.12.0
>Reporter: Gautam Kumar Parai
>Assignee: Gautam Kumar Parai
>Priority: Major
> Fix For: 1.13.0
>
>
> It would be useful to have pushdown occur past flatten(project). Here is an 
> example to illustrate the issue:
> {{explain plan without implementation for }}{{select name, 
> flatten(categories) as category from dfs.`/tmp/t_json_20` LIMIT 1;}}
> {{DrillScreenRel}}{{  }}
> {{  DrillLimitRel(fetch=[1])}}{{    }}
> {{    DrillProjectRel(name=[$0], category=[FLATTEN($1)])}}
> {{      DrillScanRel(table=[[dfs, /tmp/t_json_20]], groupscan=[EasyGroupScan 
> [selectionRoot=maprfs:/tmp/t_json_20, numFiles=1, columns=[`name`, 
> `categories`], files=[maprfs:///tmp/t_json_20/0_0_0.json]]])}}
> = 
> Content of 0_0_0.json
> =
> {
>   "name" : "Eric Goldberg, MD",
>   "categories" : [ "Doctors", "Health & Medical" ]
> } {
>   "name" : "Pine Cone Restaurant",
>   "categories" : [ "Restaurants" ]
> } {
>   "name" : "Deforest Family Restaurant",
>   "categories" : [ "American (Traditional)", "Restaurants" ]
> } {
>   "name" : "Culver's",
>   "categories" : [ "Food", "Ice Cream & Frozen Yogurt", "Fast Food", 
> "Restaurants" ]
> } {
>   "name" : "Chang Jiang Chinese Kitchen",
>   "categories" : [ "Chinese", "Restaurants" ]
> } 
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (DRILL-6116) MAXDIR - Unsupported operation exception // partition explorer interface?

[John Humphreys (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-6116?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16379152#comment-16379152
 ] 

John Humphreys commented on DRILL-6116:
---

[~kkhatua] - Running into this again.  Did you ever manage to take a peek?

> MAXDIR - Unsupported operation exception // partition explorer interface?
> -
>
> Key: DRILL-6116
> URL: https://issues.apache.org/jira/browse/DRILL-6116
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Client - JDBC
>Affects Versions: 1.10.0
>Reporter: John Humphreys
>Priority: Major
> Attachments: maxdir_fail.json, maxdir_success.json
>
>
> Using MAXDIR in drill seems to cause errors when trying to apply other 
> standard where clause filters. Am I doing something wrong here?
> **This Query Fails**
> SELECT epoch_hour, entity, `1_average`, `1_95p`
>  FROM dfs.`/path/drill-hour-final/`
>  where
>  dir0 = '2018' and dir1 = '01' and dir2 = '27'
>  and dir3 = MAXDIR('dfs', concat('/path/drill-hour-final/',
>  dir0, '/', dir1, '/', dir2, '/')) 
>  and entity = 1784928
> **With This Error**
> > SYSTEM ERROR: UnsupportedOperationException: The partition explorer
> > interface can only be used in functions that can be evaluated at
> > planning time. Make sure that the planner.enable_constant_folding
> > configuration option is set to true.
> > 
> > Fragment 1:7
> > 
> > [Error Id: d62c8227-db0e-4d31-9443-e570193d1010 on
> > psclxcpdevsys13.nomura.com:31010]
> **Wrapping it Makes it Work Without Error**
> select * from (
>  SELECT epoch_hour, entity, `1_average`, `1_95p`
>  FROM dfs.`/path/drill-hour-final/`
>  where
>  dir0 = '2018' and dir1 = '01' and dir2 = '27'
>  and dir3 = MAXDIR('dfs', concat('/path/drill-hour-final/',
>  dir0, '/', dir1, '/', dir2, '/'))
>  ) where entity = 1784928
> epoch_hour entity 1_average 1_95p
>  1517086800 1784928 5.3347222 8.82857142857143
>  1517014800 1784928 25.944984717977217 40.37453087409783



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (DRILL-6021) Show shutdown button when authentication is not enabled

[Pritesh Maker (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-6021?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pritesh Maker updated DRILL-6021:
-
Reviewer: Arina Ielchiieva

> Show shutdown button when authentication is not enabled
> ---
>
> Key: DRILL-6021
> URL: https://issues.apache.org/jira/browse/DRILL-6021
> Project: Apache Drill
>  Issue Type: Bug
>Affects Versions: 1.12.0
>Reporter: Arina Ielchiieva
>Assignee: Venkata Jyothsna Donapati
>Priority: Major
> Fix For: 1.13.0
>
>
> After DRILL-6017 {{shouldShowAdminInfo}} is used to decide if shutdown button 
> should be displayed on index page. But this option is set to true when 
> authentication is enabled and user is an admin. When authentication is not 
> enabled, user by default is admin. So with this fix without authentication, 
> shutdown button is absent but should be present.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (DRILL-6099) Drill does not push limit past project (flatten) if it cannot be pushed into scan

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-6099?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16379135#comment-16379135
 ] 

ASF GitHub Bot commented on DRILL-6099:
---

Github user priteshm commented on the issue:

https://github.com/apache/drill/pull/1096
  
@gparai did you get a chance to address @amansinha100 's comment for this 
PR?


> Drill does not push limit past project (flatten) if it cannot be pushed into 
> scan
> -
>
> Key: DRILL-6099
> URL: https://issues.apache.org/jira/browse/DRILL-6099
> Project: Apache Drill
>  Issue Type: Bug
>Affects Versions: 1.12.0
>Reporter: Gautam Kumar Parai
>Assignee: Gautam Kumar Parai
>Priority: Major
> Fix For: 1.13.0
>
>
> It would be useful to have pushdown occur past flatten(project). Here is an 
> example to illustrate the issue:
> {{explain plan without implementation for }}{{select name, 
> flatten(categories) as category from dfs.`/tmp/t_json_20` LIMIT 1;}}
> {{DrillScreenRel}}{{  }}
> {{  DrillLimitRel(fetch=[1])}}{{    }}
> {{    DrillProjectRel(name=[$0], category=[FLATTEN($1)])}}
> {{      DrillScanRel(table=[[dfs, /tmp/t_json_20]], groupscan=[EasyGroupScan 
> [selectionRoot=maprfs:/tmp/t_json_20, numFiles=1, columns=[`name`, 
> `categories`], files=[maprfs:///tmp/t_json_20/0_0_0.json]]])}}
> = 
> Content of 0_0_0.json
> =
> {
>   "name" : "Eric Goldberg, MD",
>   "categories" : [ "Doctors", "Health & Medical" ]
> } {
>   "name" : "Pine Cone Restaurant",
>   "categories" : [ "Restaurants" ]
> } {
>   "name" : "Deforest Family Restaurant",
>   "categories" : [ "American (Traditional)", "Restaurants" ]
> } {
>   "name" : "Culver's",
>   "categories" : [ "Food", "Ice Cream & Frozen Yogurt", "Fast Food", 
> "Restaurants" ]
> } {
>   "name" : "Chang Jiang Chinese Kitchen",
>   "categories" : [ "Chinese", "Restaurants" ]
> } 
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (DRILL-5365) FileNotFoundException when reading a parquet file

[Pritesh Maker (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-5365?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pritesh Maker updated DRILL-5365:
-
Fix Version/s: (was: 1.13.0)
   1.14.0

> FileNotFoundException when reading a parquet file
> -
>
> Key: DRILL-5365
> URL: https://issues.apache.org/jira/browse/DRILL-5365
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Storage - Hive
>Affects Versions: 1.10.0
>Reporter: Chun Chang
>Assignee: Chunhui Shi
>Priority: Major
> Fix For: 1.14.0
>
>
> The parquet file is generated through the following CTAS.
> To reproduce the issue: 1) two or more nodes cluster; 2) enable 
> impersonation; 3) set "fs.default.name": "file:///" in hive storage plugin; 
> 4) restart drillbits; 5) as a regular user, on node A, drop the table/file; 
> 6) ctas from a large enough hive table as source to recreate the table/file; 
> 7) query the table from node A should work; 8) query from node B as same user 
> should reproduce the issue.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (DRILL-6191) Need more information on TCP flags

[Pritesh Maker (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-6191?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pritesh Maker reassigned DRILL-6191:


Assignee: Ted Dunning

> Need more information on TCP flags
> --
>
> Key: DRILL-6191
> URL: https://issues.apache.org/jira/browse/DRILL-6191
> Project: Apache Drill
>  Issue Type: Bug
>Reporter: Ted Dunning
>Assignee: Ted Dunning
>Priority: Major
> Fix For: 1.13.0
>
>
>  
> This is a small fix based on input from Charles Givre



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (DRILL-5265) External Sort consumes more memory than allocated

[Pritesh Maker (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-5265?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pritesh Maker updated DRILL-5265:
-
Fix Version/s: (was: 1.13.0)

> External Sort consumes more memory than allocated
> -
>
> Key: DRILL-5265
> URL: https://issues.apache.org/jira/browse/DRILL-5265
> Project: Apache Drill
>  Issue Type: Bug
>Reporter: Rahul Challapalli
>Assignee: Paul Rogers
>Priority: Major
>
> git.commit.id.abbrev=300e934
> Based on the profile for the below query, the external sort has a peak memory 
> usage of ~126MB when only ~100MB was allocated 
> {code}
> alter session set `planner.memory.max_query_memory_per_node` = 104857600;
> alter session set `planner.width.max_per_node` = 1;
> select * from dfs.`/drill/testdata/md1362` order by c_email_address;
> {code}
> I attached the profile and the log files



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (DRILL-6190) Packets can be bigger than strictly legal

[Pritesh Maker (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-6190?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pritesh Maker reassigned DRILL-6190:


Assignee: Ted Dunning

> Packets can be bigger than strictly legal
> -
>
> Key: DRILL-6190
> URL: https://issues.apache.org/jira/browse/DRILL-6190
> Project: Apache Drill
>  Issue Type: Bug
>Reporter: Ted Dunning
>Assignee: Ted Dunning
>Priority: Major
> Fix For: 1.13.0
>
>
> Packets, especially those generated by malware, can be bigger than the legal 
> limit for IP. The fix is to leave 64kB padding in the buffers instead of 9kB.
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (DRILL-6190) Packets can be bigger than strictly legal

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-6190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16379125#comment-16379125
 ] 

ASF GitHub Bot commented on DRILL-6190:
---

Github user priteshm commented on the issue:

https://github.com/apache/drill/pull/1133
  
@parthchandra or @arina-ielchiieva  can you review this?


> Packets can be bigger than strictly legal
> -
>
> Key: DRILL-6190
> URL: https://issues.apache.org/jira/browse/DRILL-6190
> Project: Apache Drill
>  Issue Type: Bug
>Reporter: Ted Dunning
>Assignee: Ted Dunning
>Priority: Major
> Fix For: 1.13.0
>
>
> Packets, especially those generated by malware, can be bigger than the legal 
> limit for IP. The fix is to leave 64kB padding in the buffers instead of 9kB.
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (DRILL-6160) Limit batch size for streaming aggregate based on memory

[Pritesh Maker (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-6160?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pritesh Maker updated DRILL-6160:
-
Fix Version/s: (was: 1.13.0)
   1.14.0

> Limit batch size for streaming aggregate based on memory
> 
>
> Key: DRILL-6160
> URL: https://issues.apache.org/jira/browse/DRILL-6160
> Project: Apache Drill
>  Issue Type: Improvement
>  Components: Execution - Flow
>Affects Versions: 1.12.0
>Reporter: Padma Penumarthy
>Assignee: Padma Penumarthy
>Priority: Critical
> Fix For: 1.14.0
>
>
> Streaming aggregate has batch size of 32k rows. Change it to figure out 
> number of rows in the outgoing batch based on memory configured with the 
> option outputBatchSize and actual incoming batch size. Limit it to max of 64k 
> and min of 1. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (DRILL-6166) RecordBatchSizer does not handle hyper vectors

[Pritesh Maker (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-6166?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pritesh Maker updated DRILL-6166:
-
Fix Version/s: (was: 1.13.0)
   1.14.0

> RecordBatchSizer does not handle hyper vectors
> --
>
> Key: DRILL-6166
> URL: https://issues.apache.org/jira/browse/DRILL-6166
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Execution - Flow
>Affects Versions: 1.12.0
>Reporter: Padma Penumarthy
>Assignee: Padma Penumarthy
>Priority: Critical
> Fix For: 1.14.0
>
>
> RecordBatchSizer throws an exception when incoming batch has hyper vector.
> (java.lang.UnsupportedOperationException) null
>  org.apache.drill.exec.record.HyperVectorWrapper.getValueVector():61
>  org.apache.drill.exec.record.RecordBatchSizer.():346
>  org.apache.drill.exec.record.RecordBatchSizer.():311
>  
> org.apache.drill.exec.physical.impl.aggregate.StreamingAggBatch$StreamingAggregateMemoryManager.update():198
>  
> org.apache.drill.exec.physical.impl.aggregate.StreamingAggBatch.innerNext():328
>  org.apache.drill.exec.record.AbstractRecordBatch.next():164
>  
> org.apache.drill.exec.physical.impl.validate.IteratorValidatorBatchIterator.next():228
>  org.apache.drill.exec.physical.impl.BaseRootExec.next():105
>  
> org.apache.drill.exec.physical.impl.partitionsender.PartitionSenderRootExec.innerNext():155
>  org.apache.drill.exec.physical.impl.BaseRootExec.next():95
>  org.apache.drill.exec.work.fragment.FragmentExecutor$1.run():233
>  org.apache.drill.exec.work.fragment.FragmentExecutor$1.run():226
>  java.security.AccessController.doPrivileged():-2
>  javax.security.auth.Subject.doAs():422
>  org.apache.hadoop.security.UserGroupInformation.doAs():1657
>  org.apache.drill.exec.work.fragment.FragmentExecutor.run():226
>  org.apache.drill.common.SelfCleaningRunnable.run():38
>  java.util.concurrent.ThreadPoolExecutor.runWorker():1142
>  java.util.concurrent.ThreadPoolExecutor$Worker.run():617
>  java.lang.Thread.run():745



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (DRILL-6187) Exception in RPC communication between DataClient/ControlClient and respective servers when bit-to-bit security is on

[Pritesh Maker (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-6187?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pritesh Maker updated DRILL-6187:
-
Reviewer: Vlad Rozov

> Exception in RPC communication between DataClient/ControlClient and 
> respective servers when bit-to-bit security is on
> -
>
> Key: DRILL-6187
> URL: https://issues.apache.org/jira/browse/DRILL-6187
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Execution - RPC, Security
>Reporter: Sorabh Hamirwasia
>Assignee: Sorabh Hamirwasia
>Priority: Major
> Fix For: 1.13.0
>
>
>  
> {color:#00}Below is the summary of issue: {color}
>  
> {color:#00}*Scenario:*{color}
> {color:#00}It seems like first sendRecordBatch was sent to Foreman which 
> initiated the Authentication handshake. But before initiating handshake for 
> auth we establish a connection and store that in a registry. Now if in 
> parallel there is another recordBatch (by a different minor fragment running 
> on same Drillbit) to be sent then that will see the connection available in 
> registry and will initiate the send. Before the authentication is completed 
> this second request reached foreman and it throws below exception saying RPC 
> type 3 message is not allowed and closes the connection. This also fails the 
> authentication handshake which was in progress.{color}{color:#00} Here 
> the logs with details:{color}
> {color:#00} {color}
> {color:#00}*Forman received the SASL_START message from another 
> node:*{color}
> {color:#00}*_2018-02-21 18:43:30,759 
> [_*{color}{color:#00}_BitServer-4] TRACE 
> o.a.d.e.r.s.ServerAuthenticationHandler - Received SASL message SASL_START 
> from /10.10.100.161:35482_{color}
> {color:#00} {color}
> {color:#00}*Then around same time it received another message from client 
> of Rpc Type 3 which is for SendRecordBatch and fails since handshake is not 
> completed yet.*{color}
> {color:#00} {color}
> {color:#00}*_2018-02-21 18:43:30,762_*{color}{color:#00} 
> _[BitServer-4] ERROR o.a.d.exec.rpc.RpcExceptionHandler - Exception in RPC 
> communication.  Connection: /10.10.100.162:31012 <--> /__10.10.100.161:35482_ 
> _(data server).  Closing connection._{color}
> {color:#00}_io.netty.handler.codec.DecoderException: 
> org.apache.drill.exec.rpc.RpcException: Request of type 3 is not allowed 
> without authentication. Client on /__10.10.100.161:35482_ _must authenticate 
> before making requests. Connection dropped. [Details: Encryption: enabled , 
> MaxWrappedSize: 65536 , WrapSizeLimit: 0]_{color}
> {color:#00} {color}
> {color:#00}*Then client receives an channel closed exception:*{color}
> {color:#00} {color}
> {color:#00}*2018-02-21 18:43:30,764 [*{color}{color:#00}BitClient-4] 
> WARN  o.a.d.exec.rpc.RpcExceptionHandler - Exception occurred with closed 
> channel.  Connection: /_10.10.100.161:35482_ <--> _10.10.100.162:31012_ (data 
> client){color}
> {color:#00} {color}
> {color:#00}*and due to this it's initial command for authentication also 
> fails. Since there is channel closed exception above I will think that 
> triggered the failure of authentication request as well.*{color}
> {color:#00} {color}
> {color:#00}_Caused by: org.apache.drill.exec.rpc.RpcException: Command 
> failed while establishing connection.  Failure type AUTHENTICATION._{color}
> {color:#00}        _at 
> org.apache.drill.exec.rpc.RpcException.mapException(RpcException.java:67) 
> ~[drill-rpc-1.12.0-mapr.jar:1.12.0-mapr]_{color}
> {color:#00}        _at 
> org.apache.drill.exec.rpc.ListeningCommand.connectionFailed(ListeningCommand.java:66)
>  ~[drill-rpc-1.12.0-mapr.jar:1.12.0-mapr]_{color}
> {color:#00}        _at 
> org.apache.drill.exec.rpc.data.DataTunnel$SendBatchAsyncListen.connectionFailed(DataTunnel.java:166)
>  ~[drill-java-exec-1.12.0-mapr.jar:1.12.0-mapr]_{color}
> {color:#00}        _at 
> org.apache.drill.exec.rpc.data.DataClient$AuthenticationCommand.connectionSucceeded(DataClient.java:203)
>  ~[drill-java-exec-1.12.0-mapr.jar:1.12.0-mapr]_{color}
> {color:#00}        _at 
> org.apache.drill.exec.rpc.data.DataClient$AuthenticationCommand.connectionSucceeded(DataClient.java:147)
>  ~[drill-java-exec-1.12.0-mapr.jar:1.12.0-mapr]_{color}
> {color:#00}        _at 
> org.apache.drill.exec.rpc.ReconnectingConnection$ConnectionListeningFuture.waitAndRun(ReconnectingConnection.java:122)
>  ~[drill-rpc-1.12.0-mapr.jar:1.12.0-mapr]_{color}
> {color:#00}        _at 
> org.apache.drill.exec.rpc.ReconnectingConnection.runCommand(ReconnectingConnection.java:83)
>  ~[drill-rpc-1.12.0-mapr.jar:1.12.0-mapr]_{color}
> {color:#00}        _at 
> 

[jira] [Updated] (DRILL-6133) RecordBatchSizer throws IndexOutOfBounds Exception for union vector

[Pritesh Maker (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-6133?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pritesh Maker updated DRILL-6133:
-
Fix Version/s: (was: 1.13.0)
   1.14.0

> RecordBatchSizer throws IndexOutOfBounds Exception for union vector
> ---
>
> Key: DRILL-6133
> URL: https://issues.apache.org/jira/browse/DRILL-6133
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Execution - Flow
>Affects Versions: 1.12.0
>Reporter: Padma Penumarthy
>Assignee: Padma Penumarthy
>Priority: Minor
> Fix For: 1.14.0
>
>
> RecordBatchSizer throws IndexOutOfBoundsException when trying to get payload 
> byte count of union vector. 
> [Error Id: 430026a7-a963-40f1-bae2-1850649e8434 on 172.30.8.158:31013]
>  at 
> org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:633)
>  ~[classes/:na]
>  at 
> org.apache.drill.exec.work.fragment.FragmentExecutor.sendFinalState(FragmentExecutor.java:300)
>  [classes/:na]
>  at 
> org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:160)
>  [classes/:na]
>  at 
> org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:266)
>  [classes/:na]
>  at 
> org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38)
>  [classes/:na]
>  at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>  [na:1.8.0_45]
>  at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>  [na:1.8.0_45]
>  at java.lang.Thread.run(Thread.java:745) [na:1.8.0_45]
> Caused by: java.lang.IndexOutOfBoundsException: DrillBuf[2], udle: [1 0..0], 
> index: 4, length: 4 (expected: range(0, 0))
> DrillBuf[2], udle: [1 0..0]
>  at 
> org.apache.drill.exec.memory.BoundsChecking.checkIndex(BoundsChecking.java:80)
>  ~[classes/:na]
>  at 
> org.apache.drill.exec.memory.BoundsChecking.lengthCheck(BoundsChecking.java:86)
>  ~[classes/:na]
>  at io.netty.buffer.DrillBuf.chk(DrillBuf.java:114) ~[classes/:4.0.48.Final]
>  at io.netty.buffer.DrillBuf.getInt(DrillBuf.java:484) 
> ~[classes/:4.0.48.Final]
>  at 
> org.apache.drill.exec.vector.UInt4Vector$Accessor.get(UInt4Vector.java:432) 
> ~[classes/:na]
>  at 
> org.apache.drill.exec.vector.VarCharVector.getPayloadByteCount(VarCharVector.java:308)
>  ~[classes/:na]
>  at 
> org.apache.drill.exec.vector.NullableVarCharVector.getPayloadByteCount(NullableVarCharVector.java:256)
>  ~[classes/:na]
>  at 
> org.apache.drill.exec.vector.complex.AbstractMapVector.getPayloadByteCount(AbstractMapVector.java:303)
>  ~[classes/:na]
>  at 
> org.apache.drill.exec.vector.complex.UnionVector.getPayloadByteCount(UnionVector.java:574)
>  ~[classes/:na]
>  at 
> org.apache.drill.exec.physical.impl.spill.RecordBatchSizer$ColumnSize.(RecordBatchSizer.java:147)
>  ~[classes/:na]
>  at 
> org.apache.drill.exec.physical.impl.spill.RecordBatchSizer.measureColumn(RecordBatchSizer.java:403)
>  ~[classes/:na]
>  at 
> org.apache.drill.exec.physical.impl.spill.RecordBatchSizer.(RecordBatchSizer.java:350)
>  ~[classes/:na]
>  at 
> org.apache.drill.exec.physical.impl.spill.RecordBatchSizer.(RecordBatchSizer.java:320)
>  ~[classes/:na]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (DRILL-6147) Limit batch size for Flat Parquet Reader

[Pritesh Maker (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-6147?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pritesh Maker updated DRILL-6147:
-
Fix Version/s: (was: 1.13.0)
   1.14.0

> Limit batch size for Flat Parquet Reader
> 
>
> Key: DRILL-6147
> URL: https://issues.apache.org/jira/browse/DRILL-6147
> Project: Apache Drill
>  Issue Type: Improvement
>  Components: Storage - Parquet
>Reporter: salim achouche
>Assignee: salim achouche
>Priority: Major
> Fix For: 1.14.0
>
>
> The Parquet reader currently uses a hard-coded batch size limit (32k rows) 
> when creating scan batches; there is no parameter nor any logic for 
> controlling the amount of memory used. This enhancement will allow Drill to 
> take an extra input parameter to control direct memory usage.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (DRILL-6185) Error is displaying while accessing query profiles via the Web-UI

[Kunal Khatua (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-6185?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16379108#comment-16379108
 ] 

Kunal Khatua commented on DRILL-6185:
-

Already working on it. 

> Error is displaying while accessing query profiles via the Web-UI
> -
>
> Key: DRILL-6185
> URL: https://issues.apache.org/jira/browse/DRILL-6185
> Project: Apache Drill
>  Issue Type: Bug
>Affects Versions: 1.13.0
>Reporter: Anton Gozhiy
>Assignee: Kunal Khatua
>Priority: Blocker
> Fix For: 1.13.0
>
>
> *Steps:*
>  # Execute the following query:
> {code:sql}
> show schemas;
> {code}
> # On the Web-UI, go to the Profiles tab
> # Open the profile for the query you executed
> *Expected result:* You can access to the profile entry
> *Actual result:* Error is displayed:
> {code:json}
> {
>   "errorMessage" : "1"
> }
> {code}
> *Note:* This error doesn't happen with every query. For example, "select * 
> from system.version" can be accessed without error, while "show tables", "use 
> dfs", "alter sessions" etc end with this error.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (DRILL-6192) Drill is vulnerable to CVE-2017-12197

[Volodymyr Tkach (JIRA)]

Volodymyr Tkach created DRILL-6192:
--

 Summary: Drill is vulnerable to CVE-2017-12197
 Key: DRILL-6192
 URL: https://issues.apache.org/jira/browse/DRILL-6192
 Project: Apache Drill
  Issue Type: Bug
Reporter: Volodymyr Tkach
Assignee: Volodymyr Tkach


The current version of libpam4j bundled with MCS does not perform any 
authorization check. Any user with valid password could access the cluster even 
if the user account is disabled/password expired/'not allowed to access the 
service(pam_access ..)' etc..



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (DRILL-6174) Parquet pushdown planning improvements

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-6174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16378887#comment-16378887
 ] 

ASF GitHub Bot commented on DRILL-6174:
---

Github user arina-ielchiieva commented on the issue:

https://github.com/apache/drill/pull/1131
  
+1, LGTM.


> Parquet pushdown planning improvements
> --
>
> Key: DRILL-6174
> URL: https://issues.apache.org/jira/browse/DRILL-6174
> Project: Apache Drill
>  Issue Type: Improvement
>Affects Versions: 1.12.0
>Reporter: Arina Ielchiieva
>Assignee: Roman Kulyk
>Priority: Major
>  Labels: doc-impacting, ready-to-commit
> Fix For: 1.13.0
>
>
> Currently parquet pushdown planning has certain limitations 
> (https://drill.apache.org/docs/parquet-filter-pushdown/). This Jira aims to 
> fix some of them. List of improvements can be find below:
> 1. IS [NOT] NULL / TRUE / FALSE
> 2. Timestamp / date / time implicit / explicit casts
> {noformat}
> timestamp -> date
> timestamp -> varchar
> date -> timestamp
> date -> varchar
> time -> timestamp
> time -> date
> time -> varchar
> {noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (DRILL-6174) Parquet pushdown planning improvements

[Arina Ielchiieva (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-6174?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Arina Ielchiieva updated DRILL-6174:

Labels: doc-impacting ready-to-commit  (was: doc-impacting)

> Parquet pushdown planning improvements
> --
>
> Key: DRILL-6174
> URL: https://issues.apache.org/jira/browse/DRILL-6174
> Project: Apache Drill
>  Issue Type: Improvement
>Affects Versions: 1.12.0
>Reporter: Arina Ielchiieva
>Assignee: Roman Kulyk
>Priority: Major
>  Labels: doc-impacting, ready-to-commit
> Fix For: 1.13.0
>
>
> Currently parquet pushdown planning has certain limitations 
> (https://drill.apache.org/docs/parquet-filter-pushdown/). This Jira aims to 
> fix some of them. List of improvements can be find below:
> 1. IS [NOT] NULL / TRUE / FALSE
> 2. Timestamp / date / time implicit / explicit casts
> {noformat}
> timestamp -> date
> timestamp -> varchar
> date -> timestamp
> date -> varchar
> time -> timestamp
> time -> date
> time -> varchar
> {noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (DRILL-6191) Need more information on TCP flags

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-6191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16378839#comment-16378839
 ] 

ASF GitHub Bot commented on DRILL-6191:
---

Github user cgivre commented on the issue:

https://github.com/apache/drill/pull/1134
  
Thanks @tdunning for doing this! 


> Need more information on TCP flags
> --
>
> Key: DRILL-6191
> URL: https://issues.apache.org/jira/browse/DRILL-6191
> Project: Apache Drill
>  Issue Type: Bug
>Reporter: Ted Dunning
>Priority: Major
> Fix For: 1.13.0
>
>
>  
> This is a small fix based on input from Charles Givre



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (DRILL-6174) Parquet pushdown planning improvements

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-6174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16378832#comment-16378832
 ] 

ASF GitHub Bot commented on DRILL-6174:
---

Github user KulykRoman commented on a diff in the pull request:

https://github.com/apache/drill/pull/1131#discussion_r170971218
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/expr/stat/ParquetPredicatesHelper.java
 ---
@@ -0,0 +1,41 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * 
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * 
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.expr.stat;
+
+import org.apache.parquet.column.statistics.Statistics;
+
+/**
+ * Parquet predicates class helper for filter pushdown.
+ */
+public class ParquetPredicatesHelper {
+
+  /**
+   * Check does the column chunk's statistics have all nulls
+   */
+  static boolean isAllNulls(Statistics stat, long rowCount) {
--- End diff --

@arina-ielchiieva,
Done. 


> Parquet pushdown planning improvements
> --
>
> Key: DRILL-6174
> URL: https://issues.apache.org/jira/browse/DRILL-6174
> Project: Apache Drill
>  Issue Type: Improvement
>Affects Versions: 1.12.0
>Reporter: Arina Ielchiieva
>Assignee: Roman Kulyk
>Priority: Major
>  Labels: doc-impacting
> Fix For: 1.13.0
>
>
> Currently parquet pushdown planning has certain limitations 
> (https://drill.apache.org/docs/parquet-filter-pushdown/). This Jira aims to 
> fix some of them. List of improvements can be find below:
> 1. IS [NOT] NULL / TRUE / FALSE
> 2. Timestamp / date / time implicit / explicit casts
> {noformat}
> timestamp -> date
> timestamp -> varchar
> date -> timestamp
> date -> varchar
> time -> timestamp
> time -> date
> time -> varchar
> {noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (DRILL-6191) Need more information on TCP flags

[Ted Dunning (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-6191?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Dunning updated DRILL-6191:
---
Fix Version/s: 1.13.0

> Need more information on TCP flags
> --
>
> Key: DRILL-6191
> URL: https://issues.apache.org/jira/browse/DRILL-6191
> Project: Apache Drill
>  Issue Type: Bug
>Reporter: Ted Dunning
>Priority: Major
> Fix For: 1.13.0
>
>
>  
> This is a small fix based on input from Charles Givre



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (DRILL-6190) Packets can be bigger than strictly legal

[Ted Dunning (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-6190?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Dunning updated DRILL-6190:
---
Fix Version/s: 1.13.0

> Packets can be bigger than strictly legal
> -
>
> Key: DRILL-6190
> URL: https://issues.apache.org/jira/browse/DRILL-6190
> Project: Apache Drill
>  Issue Type: Bug
>Reporter: Ted Dunning
>Priority: Major
> Fix For: 1.13.0
>
>
> Packets, especially those generated by malware, can be bigger than the legal 
> limit for IP. The fix is to leave 64kB padding in the buffers instead of 9kB.
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (DRILL-6191) Need more information on TCP flags

[Ted Dunning (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-6191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16378816#comment-16378816
 ] 

Ted Dunning commented on DRILL-6191:


Created pull request for this

> Need more information on TCP flags
> --
>
> Key: DRILL-6191
> URL: https://issues.apache.org/jira/browse/DRILL-6191
> Project: Apache Drill
>  Issue Type: Bug
>Reporter: Ted Dunning
>Priority: Major
>
>  
> This is a small fix based on input from Charles Givre



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (DRILL-6190) Packets can be bigger than strictly legal

[Ted Dunning (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-6190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16378817#comment-16378817
 ] 

Ted Dunning commented on DRILL-6190:


Created pull request for this.

> Packets can be bigger than strictly legal
> -
>
> Key: DRILL-6190
> URL: https://issues.apache.org/jira/browse/DRILL-6190
> Project: Apache Drill
>  Issue Type: Bug
>Reporter: Ted Dunning
>Priority: Major
>
> Packets, especially those generated by malware, can be bigger than the legal 
> limit for IP. The fix is to leave 64kB padding in the buffers instead of 9kB.
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (DRILL-6191) Need more information on TCP flags

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-6191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16378814#comment-16378814
 ] 

ASF GitHub Bot commented on DRILL-6191:
---

GitHub user tdunning opened a pull request:

https://github.com/apache/drill/pull/1134

DRILL-6191 - Add acknowledgement sequence number and flags fields, details 
for flags



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/tdunning/drill DRILL-6191

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/drill/pull/1134.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1134


commit 86cf928cf4ca0dacaa359059c6cd88c488b3b9a0
Author: Ted Dunning 
Date:   2018-01-03T00:20:35Z

DRILL-6191 - Add acknowledgement sequence number and flags fields, details 
for flags




> Need more information on TCP flags
> --
>
> Key: DRILL-6191
> URL: https://issues.apache.org/jira/browse/DRILL-6191
> Project: Apache Drill
>  Issue Type: Bug
>Reporter: Ted Dunning
>Priority: Major
>
>  
> This is a small fix based on input from Charles Givre



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (DRILL-6190) Packets can be bigger than strictly legal

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-6190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16378813#comment-16378813
 ] 

ASF GitHub Bot commented on DRILL-6190:
---

GitHub user tdunning opened a pull request:

https://github.com/apache/drill/pull/1133

DRILL-6190 - Fix handling of packets longer than legally allowed.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/tdunning/drill DRILL-6190

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/drill/pull/1133.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1133


commit 678e17807a0a3f205d35febd6216e2088483ce54
Author: Ted Dunning 
Date:   2018-01-11T00:52:53Z

DRILL-6190 - Fix handling of packets longer than legally allowed.




> Packets can be bigger than strictly legal
> -
>
> Key: DRILL-6190
> URL: https://issues.apache.org/jira/browse/DRILL-6190
> Project: Apache Drill
>  Issue Type: Bug
>Reporter: Ted Dunning
>Priority: Major
>
> Packets, especially those generated by malware, can be bigger than the legal 
> limit for IP. The fix is to leave 64kB padding in the buffers instead of 9kB.
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (DRILL-6067) Add acknowledgement sequence number and flags to TCP fields

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-6067?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16378806#comment-16378806
 ] 

ASF GitHub Bot commented on DRILL-6067:
---

Github user tdunning closed the pull request at:

https://github.com/apache/drill/pull/1080


> Add acknowledgement sequence number and flags to TCP fields
> ---
>
> Key: DRILL-6067
> URL: https://issues.apache.org/jira/browse/DRILL-6067
> Project: Apache Drill
>  Issue Type: Bug
>Reporter: Ted Dunning
>Priority: Minor
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (DRILL-6174) Parquet pushdown planning improvements

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-6174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16378794#comment-16378794
 ] 

ASF GitHub Bot commented on DRILL-6174:
---

Github user arina-ielchiieva commented on a diff in the pull request:

https://github.com/apache/drill/pull/1131#discussion_r170964343
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/expr/stat/ParquetPredicatesHelper.java
 ---
@@ -0,0 +1,41 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * 
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * 
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.expr.stat;
+
+import org.apache.parquet.column.statistics.Statistics;
+
+/**
+ * Parquet predicates class helper for filter pushdown.
+ */
+public class ParquetPredicatesHelper {
+
+  /**
+   * Check does the column chunk's statistics have all nulls
+   */
+  static boolean isAllNulls(Statistics stat, long rowCount) {
--- End diff --

1. Please make helper method public as well. 
2. Please add `@param` and `@return` with description to method javadoc.
Please fix here and in method below.


> Parquet pushdown planning improvements
> --
>
> Key: DRILL-6174
> URL: https://issues.apache.org/jira/browse/DRILL-6174
> Project: Apache Drill
>  Issue Type: Improvement
>Affects Versions: 1.12.0
>Reporter: Arina Ielchiieva
>Assignee: Roman Kulyk
>Priority: Major
>  Labels: doc-impacting
> Fix For: 1.13.0
>
>
> Currently parquet pushdown planning has certain limitations 
> (https://drill.apache.org/docs/parquet-filter-pushdown/). This Jira aims to 
> fix some of them. List of improvements can be find below:
> 1. IS [NOT] NULL / TRUE / FALSE
> 2. Timestamp / date / time implicit / explicit casts
> {noformat}
> timestamp -> date
> timestamp -> varchar
> date -> timestamp
> date -> varchar
> time -> timestamp
> time -> date
> time -> varchar
> {noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (DRILL-6191) Need more information on TCP flags

[Ted Dunning (JIRA)]

Ted Dunning created DRILL-6191:
--

 Summary: Need more information on TCP flags
 Key: DRILL-6191
 URL: https://issues.apache.org/jira/browse/DRILL-6191
 Project: Apache Drill
  Issue Type: Bug
Reporter: Ted Dunning


 

This is a small fix based on input from Charles Givre



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (DRILL-6189) Security: passwords logging and file permisions

[Arina Ielchiieva (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-6189?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Arina Ielchiieva updated DRILL-6189:

Affects Version/s: 1.12.0

> Security: passwords logging and file permisions
> ---
>
> Key: DRILL-6189
> URL: https://issues.apache.org/jira/browse/DRILL-6189
> Project: Apache Drill
>  Issue Type: Bug
>Affects Versions: 1.12.0
>Reporter: Volodymyr Tkach
>Assignee: Volodymyr Tkach
>Priority: Major
>
> *Prerequisites:*
>  *1.* Log level is set to "all" in the conf/logback.xml:
> {code:xml}
> 
> 
> 
> 
> {code}
> *2.* PLAIN authentication mechanism is configured:
> {code:java}
>   security.user.auth: {
>   enabled: true,
>   packages += "org.apache.drill.exec.rpc.user.security",
>   impl: "pam",
>   pam_profiles: [ "sudo", "login" ]
>   }
> {code}
> *Steps:*
>  *1.* Start the drillbits
>  *2.* Connect by sqlline:
> {noformat}
> /opt/mapr/drill/drill-1.13.0/bin/sqlline -u "jdbc:drill:zk=node1:5181;" -n 
> user1 -p 
> {noformat}
> *Expected result:* Logs shouldn't contain clear-text passwords
> *Actual results:* During the drillbit startup or establishing connections via 
> the jdbc or odbc, the following lines appear in the drillbit.log:
> {noformat}
> properties {
> key: "password"
> value: ""
> }
> {noformat}
> Same thing happens with storage configuration data, everything, including 
> passwords is being logged to file.
> *Another issue:*
> Currently Drill config files has the permissions 0644:
> {noformat}
> -rw-r--r--. 1 mapr mapr 1081 Nov 16 14:42 core-site-example.xml
> -rwxr-xr-x. 1 mapr mapr 1807 Dec 19 11:55 distrib-env.sh
> -rw-r--r--. 1 mapr mapr 1424 Nov 16 14:42 distrib-env.sh.prejmx
> -rw-r--r--. 1 mapr mapr 1942 Nov 16 14:42 drill-am-log.xml
> -rw-r--r--. 1 mapr mapr 1279 Dec 19 11:55 drill-distrib.conf
> -rw-r--r--. 1 mapr mapr  117 Nov 16 14:50 drill-distrib-mem-qs.conf
> -rw-r--r--. 1 mapr mapr 6016 Nov 16 14:42 drill-env.sh
> -rw-r--r--. 1 mapr mapr 1855 Nov 16 14:50 drill-on-yarn.conf
> -rw-r--r--. 1 mapr mapr 6913 Nov 16 14:42 drill-on-yarn-example.conf
> -rw-r--r--. 1 mapr mapr 1135 Dec 19 11:55 drill-override.conf
> -rw-r--r--. 1 mapr mapr 7820 Nov 16 14:42 drill-override-example.conf
> -rw-r--r--. 1 mapr mapr 3136 Nov 16 14:42 logback.xml
> -rw-r--r--. 1 mapr mapr  668 Nov 16 14:51 warden.drill-bits.conf
> -rw-r--r--. 1 mapr mapr 1581 Nov 16 14:42 yarn-client-log.xml
> {noformat}
> As they may contain some sensitive information, like passwords or secret 
> keys, they cannot be viewable to everyone. So I suggest to reduce the 
> permissions at least to 0640.
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (DRILL-6190) Packets can be bigger than strictly legal

[Ted Dunning (JIRA)]

Ted Dunning created DRILL-6190:
--

 Summary: Packets can be bigger than strictly legal
 Key: DRILL-6190
 URL: https://issues.apache.org/jira/browse/DRILL-6190
 Project: Apache Drill
  Issue Type: Bug
Reporter: Ted Dunning


Packets, especially those generated by malware, can be bigger than the legal 
limit for IP. The fix is to leave 64kB padding in the buffers instead of 9kB.

 

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (DRILL-1491) Support for JDK 8

[Arina Ielchiieva (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-1491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16378695#comment-16378695
 ] 

Arina Ielchiieva commented on DRILL-1491:
-

Since all tests pass, we can move Drill from Java 1.7 to 1.8. We need to make 
changes in travis,yml, pom.xml, drill-config.sh etc.

> Support for JDK 8
> -
>
> Key: DRILL-1491
> URL: https://issues.apache.org/jira/browse/DRILL-1491
> Project: Apache Drill
>  Issue Type: Task
>  Components: Tools, Build  Test
>Reporter: Aditya Kishore
>Assignee: Volodymyr Tkach
>Priority: Blocker
> Fix For: 1.13.0
>
> Attachments: DRILL-1491.1.patch.txt
>
>
> This will be the umbrella JIRA used to track and fix issues with JDK 8 
> support.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (DRILL-1491) Support for JDK 8

[Volodymyr Tkach (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-1491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16378627#comment-16378627
 ] 

Volodymyr Tkach commented on DRILL-1491:


Unit tests succeed on jdk 8  with and without mapr profile.

> Support for JDK 8
> -
>
> Key: DRILL-1491
> URL: https://issues.apache.org/jira/browse/DRILL-1491
> Project: Apache Drill
>  Issue Type: Task
>  Components: Tools, Build  Test
>Reporter: Aditya Kishore
>Assignee: Volodymyr Tkach
>Priority: Blocker
> Fix For: 1.13.0
>
> Attachments: DRILL-1491.1.patch.txt
>
>
> This will be the umbrella JIRA used to track and fix issues with JDK 8 
> support.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (DRILL-6174) Parquet pushdown planning improvements

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-6174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16378607#comment-16378607
 ] 

ASF GitHub Bot commented on DRILL-6174:
---

Github user KulykRoman commented on the issue:

https://github.com/apache/drill/pull/1131
  
All changes were applied and all tests passed.


> Parquet pushdown planning improvements
> --
>
> Key: DRILL-6174
> URL: https://issues.apache.org/jira/browse/DRILL-6174
> Project: Apache Drill
>  Issue Type: Improvement
>Affects Versions: 1.12.0
>Reporter: Arina Ielchiieva
>Assignee: Roman Kulyk
>Priority: Major
>  Labels: doc-impacting
> Fix For: 1.13.0
>
>
> Currently parquet pushdown planning has certain limitations 
> (https://drill.apache.org/docs/parquet-filter-pushdown/). This Jira aims to 
> fix some of them. List of improvements can be find below:
> 1. IS [NOT] NULL / TRUE / FALSE
> 2. Timestamp / date / time implicit / explicit casts
> {noformat}
> timestamp -> date
> timestamp -> varchar
> date -> timestamp
> date -> varchar
> time -> timestamp
> time -> date
> time -> varchar
> {noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (DRILL-6188) Fix C++ client build on Centos 7 and OSX

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-6188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16378573#comment-16378573
 ] 

ASF GitHub Bot commented on DRILL-6188:
---

GitHub user parthchandra opened a pull request:

https://github.com/apache/drill/pull/1132

DRILL-6188: Fix C++ client build on Centos7, OS X

Removed unused method that didn't compile. 
Included iostream for declaration of std::cout

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/parthchandra/drill DRILL-6188

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/drill/pull/1132.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1132


commit bf1f90138502300649b0945fcfc2b1b109512b52
Author: Parth Chandra 
Date:   2018-02-27T12:01:51Z

DRILL-6188: Fix C++ client build on Centos7, OS X




> Fix C++ client build on Centos 7 and OSX 
> -
>
> Key: DRILL-6188
> URL: https://issues.apache.org/jira/browse/DRILL-6188
> Project: Apache Drill
>  Issue Type: Bug
>Reporter: Parth Chandra
>Priority: Major
>
> compile issue on CentOS 7:
> {quote}In file included from 
> /root/default/private-drill/contrib/native/client/src/clientlib/utils.cpp:22:0:
>  /root/default/private-drill/contrib/native/client/src/clientlib/logger.hpp: 
> In constructor 'Drill::Logger::Logger()':
>  
> /root/default/private-drill/contrib/native/client/src/clientlib/logger.hpp:38:29:
>  error: 'cout' is not a member of 'std'
>  m_pOutStream = ::cout;
>  ^
>  make[2]: *** [src/clientlib/CMakeFiles/drillClient.dir/utils.cpp.o] Error 1
>  make[1]: *** [src/clientlib/CMakeFiles/drillClient.dir/all] Error 2
>  make: *** [all] Error 2
> {quote}
> OSX - has this compile error:
> {quote}In file included from 
> /Users/mapr/private-drill/contrib/native/client/src/clientlib/drillClientImpl.cpp:34:
>  
> /Users/mapr/private-drill/contrib/native/client/src/clientlib/drillClientImpl.hpp:185:39:
>  error: 'm_bHasError' is a private member of 'Drill::DrillClientQueryHandle'
>  void setHasError(bool hasError)
> Unknown macro: \{ m_bHasError = hasError; }
> ^
>  
> /Users/mapr/private-drill/contrib/native/client/src/clientlib/drillClientImpl.hpp:158:10:
>  note: declared private here
>  bool m_bHasError;
>  ^
> {quote}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (DRILL-6188) Fix C++ client build on Centos 7 and OSX

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-6188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16378574#comment-16378574
 ] 

ASF GitHub Bot commented on DRILL-6188:
---

Github user parthchandra commented on the issue:

https://github.com/apache/drill/pull/1132
  
@pwong-mapr please review 


> Fix C++ client build on Centos 7 and OSX 
> -
>
> Key: DRILL-6188
> URL: https://issues.apache.org/jira/browse/DRILL-6188
> Project: Apache Drill
>  Issue Type: Bug
>Reporter: Parth Chandra
>Priority: Major
>
> compile issue on CentOS 7:
> {quote}In file included from 
> /root/default/private-drill/contrib/native/client/src/clientlib/utils.cpp:22:0:
>  /root/default/private-drill/contrib/native/client/src/clientlib/logger.hpp: 
> In constructor 'Drill::Logger::Logger()':
>  
> /root/default/private-drill/contrib/native/client/src/clientlib/logger.hpp:38:29:
>  error: 'cout' is not a member of 'std'
>  m_pOutStream = ::cout;
>  ^
>  make[2]: *** [src/clientlib/CMakeFiles/drillClient.dir/utils.cpp.o] Error 1
>  make[1]: *** [src/clientlib/CMakeFiles/drillClient.dir/all] Error 2
>  make: *** [all] Error 2
> {quote}
> OSX - has this compile error:
> {quote}In file included from 
> /Users/mapr/private-drill/contrib/native/client/src/clientlib/drillClientImpl.cpp:34:
>  
> /Users/mapr/private-drill/contrib/native/client/src/clientlib/drillClientImpl.hpp:185:39:
>  error: 'm_bHasError' is a private member of 'Drill::DrillClientQueryHandle'
>  void setHasError(bool hasError)
> Unknown macro: \{ m_bHasError = hasError; }
> ^
>  
> /Users/mapr/private-drill/contrib/native/client/src/clientlib/drillClientImpl.hpp:158:10:
>  note: declared private here
>  bool m_bHasError;
>  ^
> {quote}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (DRILL-6189) Security: passwords logging and file permisions

[Volodymyr Tkach (JIRA)]

Volodymyr Tkach created DRILL-6189:
--

 Summary: Security: passwords logging and file permisions
 Key: DRILL-6189
 URL: https://issues.apache.org/jira/browse/DRILL-6189
 Project: Apache Drill
  Issue Type: Bug
Reporter: Volodymyr Tkach
Assignee: Volodymyr Tkach


*Prerequisites:*
 *1.* Log level is set to "all" in the conf/logback.xml:
{code:xml}




{code}
*2.* PLAIN authentication mechanism is configured:
{code:java}
  security.user.auth: {
enabled: true,
packages += "org.apache.drill.exec.rpc.user.security",
impl: "pam",
pam_profiles: [ "sudo", "login" ]
  }
{code}
*Steps:*
 *1.* Start the drillbits
 *2.* Connect by sqlline:
{noformat}
/opt/mapr/drill/drill-1.13.0/bin/sqlline -u "jdbc:drill:zk=node1:5181;" -n 
user1 -p 
{noformat}
*Expected result:* Logs shouldn't contain clear-text passwords

*Actual results:* During the drillbit startup or establishing connections via 
the jdbc or odbc, the following lines appear in the drillbit.log:
{noformat}
properties {
key: "password"
value: ""
}
{noformat}
Same thing happens with storage configuration data, everything, including 
passwords is being logged to file.

*Another issue:*

Currently Drill config files has the permissions 0644:
{noformat}
-rw-r--r--. 1 mapr mapr 1081 Nov 16 14:42 core-site-example.xml
-rwxr-xr-x. 1 mapr mapr 1807 Dec 19 11:55 distrib-env.sh
-rw-r--r--. 1 mapr mapr 1424 Nov 16 14:42 distrib-env.sh.prejmx
-rw-r--r--. 1 mapr mapr 1942 Nov 16 14:42 drill-am-log.xml
-rw-r--r--. 1 mapr mapr 1279 Dec 19 11:55 drill-distrib.conf
-rw-r--r--. 1 mapr mapr  117 Nov 16 14:50 drill-distrib-mem-qs.conf
-rw-r--r--. 1 mapr mapr 6016 Nov 16 14:42 drill-env.sh
-rw-r--r--. 1 mapr mapr 1855 Nov 16 14:50 drill-on-yarn.conf
-rw-r--r--. 1 mapr mapr 6913 Nov 16 14:42 drill-on-yarn-example.conf
-rw-r--r--. 1 mapr mapr 1135 Dec 19 11:55 drill-override.conf
-rw-r--r--. 1 mapr mapr 7820 Nov 16 14:42 drill-override-example.conf
-rw-r--r--. 1 mapr mapr 3136 Nov 16 14:42 logback.xml
-rw-r--r--. 1 mapr mapr  668 Nov 16 14:51 warden.drill-bits.conf
-rw-r--r--. 1 mapr mapr 1581 Nov 16 14:42 yarn-client-log.xml
{noformat}

As they may contain some sensitive information, like passwords or secret keys, 
they cannot be viewable to everyone. So I suggest to reduce the permissions at 
least to 0640.

 

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (DRILL-6174) Parquet pushdown planning improvements

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-6174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16378493#comment-16378493
 ] 

ASF GitHub Bot commented on DRILL-6174:
---

Github user arina-ielchiieva commented on a diff in the pull request:

https://github.com/apache/drill/pull/1131#discussion_r170902942
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/expr/stat/ParquetPredicatesHelper.java
 ---
@@ -0,0 +1,36 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * 
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * 
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.expr.stat;
+
+import org.apache.parquet.column.statistics.Statistics;
+
+class ParquetPredicatesHelper {
--- End diff --

Please make public and add java doc for each helper method.


> Parquet pushdown planning improvements
> --
>
> Key: DRILL-6174
> URL: https://issues.apache.org/jira/browse/DRILL-6174
> Project: Apache Drill
>  Issue Type: Improvement
>Affects Versions: 1.12.0
>Reporter: Arina Ielchiieva
>Assignee: Roman Kulyk
>Priority: Major
>  Labels: doc-impacting
> Fix For: 1.13.0
>
>
> Currently parquet pushdown planning has certain limitations 
> (https://drill.apache.org/docs/parquet-filter-pushdown/). This Jira aims to 
> fix some of them. List of improvements can be find below:
> 1. IS [NOT] NULL / TRUE / FALSE
> 2. Timestamp / date / time implicit / explicit casts
> {noformat}
> timestamp -> date
> timestamp -> varchar
> date -> timestamp
> date -> varchar
> time -> timestamp
> time -> date
> time -> varchar
> {noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (DRILL-6174) Parquet pushdown planning improvements

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-6174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16378492#comment-16378492
 ] 

ASF GitHub Bot commented on DRILL-6174:
---

Github user arina-ielchiieva commented on a diff in the pull request:

https://github.com/apache/drill/pull/1131#discussion_r170902502
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/expr/stat/ParquetBooleanPredicates.java
 ---
@@ -0,0 +1,72 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to you under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.expr.stat;
+
+import org.apache.drill.common.expression.BooleanOperator;
+import org.apache.drill.common.expression.ExpressionPosition;
+import org.apache.drill.common.expression.LogicalExpression;
+import org.apache.drill.common.expression.visitors.ExprVisitor;
+
+import java.util.List;
+
+public abstract class ParquetBooleanPredicates {
--- End diff --

1. Please remove abstract.
2. Add javadoc.
Here and in other two similar classes.


> Parquet pushdown planning improvements
> --
>
> Key: DRILL-6174
> URL: https://issues.apache.org/jira/browse/DRILL-6174
> Project: Apache Drill
>  Issue Type: Improvement
>Affects Versions: 1.12.0
>Reporter: Arina Ielchiieva
>Assignee: Roman Kulyk
>Priority: Major
>  Labels: doc-impacting
> Fix For: 1.13.0
>
>
> Currently parquet pushdown planning has certain limitations 
> (https://drill.apache.org/docs/parquet-filter-pushdown/). This Jira aims to 
> fix some of them. List of improvements can be find below:
> 1. IS [NOT] NULL / TRUE / FALSE
> 2. Timestamp / date / time implicit / explicit casts
> {noformat}
> timestamp -> date
> timestamp -> varchar
> date -> timestamp
> date -> varchar
> time -> timestamp
> time -> date
> time -> varchar
> {noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (DRILL-6188) Fix C++ client build on Centos 7 and OSX

[Parth Chandra (JIRA)]

Parth Chandra created DRILL-6188:


 Summary: Fix C++ client build on Centos 7 and OSX 
 Key: DRILL-6188
 URL: https://issues.apache.org/jira/browse/DRILL-6188
 Project: Apache Drill
  Issue Type: Bug
Reporter: Parth Chandra


compile issue on CentOS 7:
{quote}In file included from 
/root/default/private-drill/contrib/native/client/src/clientlib/utils.cpp:22:0:
 /root/default/private-drill/contrib/native/client/src/clientlib/logger.hpp: In 
constructor 'Drill::Logger::Logger()':
 
/root/default/private-drill/contrib/native/client/src/clientlib/logger.hpp:38:29:
 error: 'cout' is not a member of 'std'
 m_pOutStream = ::cout;
 ^
 make[2]: *** [src/clientlib/CMakeFiles/drillClient.dir/utils.cpp.o] Error 1
 make[1]: *** [src/clientlib/CMakeFiles/drillClient.dir/all] Error 2
 make: *** [all] Error 2
{quote}
OSX - has this compile error:
{quote}In file included from 
/Users/mapr/private-drill/contrib/native/client/src/clientlib/drillClientImpl.cpp:34:
 
/Users/mapr/private-drill/contrib/native/client/src/clientlib/drillClientImpl.hpp:185:39:
 error: 'm_bHasError' is a private member of 'Drill::DrillClientQueryHandle'
 void setHasError(bool hasError)
Unknown macro: \{ m_bHasError = hasError; }
^
 
/Users/mapr/private-drill/contrib/native/client/src/clientlib/drillClientImpl.hpp:158:10:
 note: declared private here
 bool m_bHasError;
 ^
{quote}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)