[jira] [Updated] (DRILL-6234) Improve Documentation of VariableWidthVector Behavior
[
https://issues.apache.org/jira/browse/DRILL-6234?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Timothy Farkas updated DRILL-6234:
--
Labels: ready-to-commit (was: )
> Improve Documentation of VariableWidthVector Behavior
> -
>
> Key: DRILL-6234
> URL: https://issues.apache.org/jira/browse/DRILL-6234
> Project: Apache Drill
> Issue Type: Improvement
>Reporter: Timothy Farkas
>Assignee: Timothy Farkas
>Priority: Major
> Labels: ready-to-commit
> Fix For: 1.14.0
>
>
> Doing the following will throw an Index out of bounds exception.
> {code}
> final VarCharVector vector = new VarCharVector(field, allocator);
> vector.allocateNew();
> vector.getMutator().setValueCount(100);
> {code}
> The expected behavior is to resize the array appropriately. If an index is
> uninitialized you should not call get for that index.
> {code}
> at
> org.apache.drill.exec.memory.BoundsChecking.checkIndex(BoundsChecking.java:80)
> at
> org.apache.drill.exec.memory.BoundsChecking.lengthCheck(BoundsChecking.java:86)
> at io.netty.buffer.DrillBuf.chk(DrillBuf.java:114)
> at io.netty.buffer.DrillBuf.getInt(DrillBuf.java:484)
> at
> org.apache.drill.exec.vector.UInt4Vector$Accessor.get(UInt4Vector.java:432)
> at
> org.apache.drill.exec.vector.VarCharVector$Mutator.setValueCount(VarCharVector.java:729)
> at
> org.apache.drill.exec.vector.VarCharVectorTest.testExpandingNonEmptyVectorSetValueCount(VarCharVectorTest.java:97)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:68)
> at
> com.intellij.rt.execution.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:47)
> at
> com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:242)
> at
> com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:70)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (DRILL-4176) Dynamic Schema Discovery is not done in case of Drill- Hive
[
https://issues.apache.org/jira/browse/DRILL-4176?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vitalii Diravka reassigned DRILL-4176:
--
Assignee: Vitalii Diravka
> Dynamic Schema Discovery is not done in case of Drill- Hive
> ---
>
> Key: DRILL-4176
> URL: https://issues.apache.org/jira/browse/DRILL-4176
> Project: Apache Drill
> Issue Type: Bug
>Affects Versions: 1.3.0
>Reporter: Devender Yadav
>Assignee: Vitalii Diravka
>Priority: Major
> Fix For: Future
>
>
> I am using hive with drill.
> Storage Plugin info:
> {
> "type": "hive",
> "enabled": true,
> "configProps": {
> "hive.metastore.uris": "",
> "javax.jdo.option.ConnectionURL":
> "jdbc:mysql://localhost:3306/metastore_hive",
> "javax.jdo.option.ConnectionDriverName": "com.mysql.jdbc.Driver",
> "javax.jdo.option.ConnectionUserName": "root",
> "javax.jdo.option.ConnectionPassword": "root",
> "hive.metastore.warehouse.dir": "/user/hive/warehouse",
> "fs.default.name": "file:///",
> "hive.metastore.sasl.enabled": "false"
> }
> }
> It's working fine for querying and all.
> Then I wanted to check whether it automatically discover newly created tables
> in hive or not.
> I started drill in embedded mode and used a particular database in hive using
> use hive.testDB;
> Here testDB is a database in Hive with tables t1 & t2. Then I queried:
> show tables;
> It gave me table names
> t1
> t2
> I created a table t3 in hive and again fired show tables; in Drill. It's
> still showing t1 t2. After 5-10 min I fired show tables; again and it's
> showing t1 t2 t3.
> I think it should show t3 immediately after adding t3 in hive.
> What can be reason for this behavior and how drill is handling it internally?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (DRILL-4176) Dynamic Schema Discovery is not done in case of Drill- Hive
[
https://issues.apache.org/jira/browse/DRILL-4176?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vitalii Diravka reassigned DRILL-4176:
--
Assignee: (was: Vitalii Diravka)
> Dynamic Schema Discovery is not done in case of Drill- Hive
> ---
>
> Key: DRILL-4176
> URL: https://issues.apache.org/jira/browse/DRILL-4176
> Project: Apache Drill
> Issue Type: Bug
>Affects Versions: 1.3.0
>Reporter: Devender Yadav
>Priority: Major
> Fix For: Future
>
>
> I am using hive with drill.
> Storage Plugin info:
> {
> "type": "hive",
> "enabled": true,
> "configProps": {
> "hive.metastore.uris": "",
> "javax.jdo.option.ConnectionURL":
> "jdbc:mysql://localhost:3306/metastore_hive",
> "javax.jdo.option.ConnectionDriverName": "com.mysql.jdbc.Driver",
> "javax.jdo.option.ConnectionUserName": "root",
> "javax.jdo.option.ConnectionPassword": "root",
> "hive.metastore.warehouse.dir": "/user/hive/warehouse",
> "fs.default.name": "file:///",
> "hive.metastore.sasl.enabled": "false"
> }
> }
> It's working fine for querying and all.
> Then I wanted to check whether it automatically discover newly created tables
> in hive or not.
> I started drill in embedded mode and used a particular database in hive using
> use hive.testDB;
> Here testDB is a database in Hive with tables t1 & t2. Then I queried:
> show tables;
> It gave me table names
> t1
> t2
> I created a table t3 in hive and again fired show tables; in Drill. It's
> still showing t1 t2. After 5-10 min I fired show tables; again and it's
> showing t1 t2 t3.
> I think it should show t3 immediately after adding t3 in hive.
> What can be reason for this behavior and how drill is handling it internally?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (DRILL-6249) Add Markdown Docs for Unit Testing and Link to it in README.md
[ https://issues.apache.org/jira/browse/DRILL-6249?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Timothy Farkas updated DRILL-6249: -- Description: I am working on a presentation about how to use the unit testing utilities in Drill. Instead of writing the doc and having it be lost in Google Drive somewhere I am going to add a Markdown doc to the drill repo and link to it in the README.md. This is appropriate since these docs will only be used by developers, and the way we unit test will change as the code changes. So the unit testing docs should be kept in the same repo as the code. > Add Markdown Docs for Unit Testing and Link to it in README.md > -- > > Key: DRILL-6249 > URL: https://issues.apache.org/jira/browse/DRILL-6249 > Project: Apache Drill > Issue Type: Improvement >Reporter: Timothy Farkas >Assignee: Timothy Farkas >Priority: Major > > I am working on a presentation about how to use the unit testing utilities in > Drill. Instead of writing the doc and having it be lost in Google Drive > somewhere I am going to add a Markdown doc to the drill repo and link to it > in the README.md. This is appropriate since these docs will only be used by > developers, and the way we unit test will change as the code changes. So the > unit testing docs should be kept in the same repo as the code. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (DRILL-6249) Add Markdown Docs for Unit Testing and Link to it in README.md
[ https://issues.apache.org/jira/browse/DRILL-6249?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Timothy Farkas updated DRILL-6249: -- Description: I am working on a presentation about how to use the unit testing utilities in Drill. Instead of writing the doc and having it be lost in Google Drive somewhere I am going to add a Markdown doc to the drill repo and link to it in the README.md. This is appropriate since these docs will only be used by developers, and the way we unit test will change as the code changes. So the unit testing docs should be kept in the same repo as the code so it can be updated and kept in sync with the rest of Drill. (was: I am working on a presentation about how to use the unit testing utilities in Drill. Instead of writing the doc and having it be lost in Google Drive somewhere I am going to add a Markdown doc to the drill repo and link to it in the README.md. This is appropriate since these docs will only be used by developers, and the way we unit test will change as the code changes. So the unit testing docs should be kept in the same repo as the code.) > Add Markdown Docs for Unit Testing and Link to it in README.md > -- > > Key: DRILL-6249 > URL: https://issues.apache.org/jira/browse/DRILL-6249 > Project: Apache Drill > Issue Type: Improvement >Reporter: Timothy Farkas >Assignee: Timothy Farkas >Priority: Major > > I am working on a presentation about how to use the unit testing utilities in > Drill. Instead of writing the doc and having it be lost in Google Drive > somewhere I am going to add a Markdown doc to the drill repo and link to it > in the README.md. This is appropriate since these docs will only be used by > developers, and the way we unit test will change as the code changes. So the > unit testing docs should be kept in the same repo as the code so it can be > updated and kept in sync with the rest of Drill. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (DRILL-6249) Add Markdown Docs for Unit Testing and Link to it in README.md
Timothy Farkas created DRILL-6249: - Summary: Add Markdown Docs for Unit Testing and Link to it in README.md Key: DRILL-6249 URL: https://issues.apache.org/jira/browse/DRILL-6249 Project: Apache Drill Issue Type: Improvement Reporter: Timothy Farkas Assignee: Timothy Farkas -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (DRILL-6247) Minor fragments remain in CANCELLATION_REQUESTED state after query failure
[
https://issues.apache.org/jira/browse/DRILL-6247?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Khurram Faraaz updated DRILL-6247:
--
Attachment: 25569e98-10f9-2fe2-9dec-0a42f3ad45fa.sys.drill
> Minor fragments remain in CANCELLATION_REQUESTED state after query failure
> --
>
> Key: DRILL-6247
> URL: https://issues.apache.org/jira/browse/DRILL-6247
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Flow
>Affects Versions: 1.13.0
>Reporter: Khurram Faraaz
>Assignee: Vlad Rozov
>Priority: Major
> Attachments: 25569e98-10f9-2fe2-9dec-0a42f3ad45fa.sys.drill,
> cancellation_requested_march_14.png, drillbit_snippet.log,
> jstack_cancellation_requested.txt
>
>
> Once a query fails, in this case due to an OOM in RPC we see many minor
> fragments are reported to be in CANCELLATION_REQUESTED state on Web UI after
> query has failed. The problem is reproducible. drillbit.log for this failure
> and jstack output are attached here.
> To reproduce the problem on a 4 node cluster.
> alter system reset all;
> alter system set `planner.slice_target`=1;
> Failing query => SELECT * , FLATTEN(arr) FROM many_json_files;
> Drill 1.13.0-SNAPSHOT, commit id: 766315ea17377199897d685ab801edd38394fe01
> Stack trace from output of jstack, fragment 0:0 is reported to be in
> CANCELLATION_REQUESTED state on Drill Web UI
> jstack -l 13488 > jstack_DRILL_6235.txt
> {noformat}
> "25569e98-10f9-2fe2-9dec-0a42f3ad45fa:frag:0:0" #87 daemon prio=10 os_prio=0
> tid=0x7f9d01374360 nid=0x2ff5 waiting on condition [0x7f9cd5536000]
> java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x0007a388b300> (a
> org.apache.drill.exec.rpc.ResettableBarrier$InternalSynchronizer)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
> at
> org.apache.drill.exec.rpc.ResettableBarrier.await(ResettableBarrier.java:70)
> at
> org.apache.drill.exec.rpc.AbstractRemoteConnection$WriteManager.waitForWritable(AbstractRemoteConnection.java:114)
> at
> org.apache.drill.exec.rpc.AbstractRemoteConnection.blockOnNotWritable(AbstractRemoteConnection.java:76)
> at org.apache.drill.exec.rpc.RpcBus.send(RpcBus.java:108)
> at
> org.apache.drill.exec.rpc.user.UserServer$BitToUserConnection.sendData(UserServer.java:275)
> at
> org.apache.drill.exec.ops.AccountingUserConnection.sendData(AccountingUserConnection.java:42)
> at
> org.apache.drill.exec.physical.impl.ScreenCreator$ScreenRoot.innerNext(ScreenCreator.java:120)
> at
> org.apache.drill.exec.physical.impl.BaseRootExec.next(BaseRootExec.java:95)
> at
> org.apache.drill.exec.work.fragment.FragmentExecutor$1.run(FragmentExecutor.java:233)
> at
> org.apache.drill.exec.work.fragment.FragmentExecutor$1.run(FragmentExecutor.java:226)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1595)
> at
> org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:226)
> at
> org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> {noformat}
> Stack trace from drillbit.log
> {noformat}
> 2018-03-14 10:52:44,545 [25569e98-10f9-2fe2-9dec-0a42f3ad45fa:frag:1:49] INFO
> o.a.d.e.w.fragment.FragmentExecutor - User Error Occurred: One or more nodes
> ran out of memory while executing the query. (Failure allocating buffer.)
> org.apache.drill.common.exceptions.UserException: RESOURCE ERROR: One or more
> nodes ran out of memory while executing the query.
> Failure allocating buffer.
> [Error Id: b83884df-af31-411a-9b28-554c294a7357 ]
> at
> org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:633)
> ~[drill-common-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT]
> at
> org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:243)
> [drill-java-exec-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT]
> at
> org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38)
> [drill-common-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT]
> at
[jira] [Assigned] (DRILL-6248) Support pushdown into System Table
[
https://issues.apache.org/jira/browse/DRILL-6248?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pritesh Maker reassigned DRILL-6248:
Assignee: Arina Ielchiieva
> Support pushdown into System Table
> --
>
> Key: DRILL-6248
> URL: https://issues.apache.org/jira/browse/DRILL-6248
> Project: Apache Drill
> Issue Type: Improvement
>Reporter: Kunal Khatua
>Assignee: Arina Ielchiieva
>Priority: Minor
>
> When querying a profile store, we fetch all the records before applying a
> limit.
> {code:sql}
> select * from sys.profiles limit 1
> {code}
> For a test scenario with 120K+ profiles in the store, a
> {code:sql}
> select count(*) from sys.profiles
> {code}
> took 90 minutes.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (DRILL-6248) Support pushdown into System Table
Kunal Khatua created DRILL-6248:
---
Summary: Support pushdown into System Table
Key: DRILL-6248
URL: https://issues.apache.org/jira/browse/DRILL-6248
Project: Apache Drill
Issue Type: Improvement
Reporter: Kunal Khatua
When querying a profile store, we fetch all the records before applying a limit.
{code:sql}
select * from sys.profiles limit 1
{code}
For a test scenario with 120K+ profiles in the store, a
{code:sql}
select count(*) from sys.profiles
{code}
took 90 minutes.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (DRILL-6241) Saffron properties config has the excessive permissions
[
https://issues.apache.org/jira/browse/DRILL-6241?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pritesh Maker updated DRILL-6241:
-
Fix Version/s: 1.14.0
> Saffron properties config has the excessive permissions
>
>
> Key: DRILL-6241
> URL: https://issues.apache.org/jira/browse/DRILL-6241
> Project: Apache Drill
> Issue Type: Bug
>Affects Versions: 1.13.0
>Reporter: Volodymyr Tkach
>Assignee: Volodymyr Tkach
>Priority: Major
> Labels: ready-to-commit
> Fix For: 1.14.0
>
>
> Current Drill config permissions:
> {noformat}
> ls -al ./drill-1.13.0/conf/saffron.properties
> -rw-rw-r-- 1 mapr mapr 1191 Mar 12 09:36 saffron.properties
> {noformat}
> *Expected result:*
> It should have permissions 0640
> *Actual result:*
> It has the permissions 0664
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-6239) Add Build and License Badges to README.md
[ https://issues.apache.org/jira/browse/DRILL-6239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16399096#comment-16399096 ] ASF GitHub Bot commented on DRILL-6239: --- Github user ilooner commented on the issue: https://github.com/apache/drill/pull/1165 @vrozov Fixed bad link and added maven badge. Currently the maven badge points to drill's distribution pom. It's also interesting to note that only some of drill's artifacts are published on maven central. See https://search.maven.org/#search%7Cga%7C1%7Cg%3A%22org.apache.drill%22 > Add Build and License Badges to README.md > - > > Key: DRILL-6239 > URL: https://issues.apache.org/jira/browse/DRILL-6239 > Project: Apache Drill > Issue Type: Improvement >Reporter: Timothy Farkas >Assignee: Timothy Farkas >Priority: Major > Labels: ready-to-commit > Fix For: 1.14.0 > > > Other projects have pretty badges showing the build status and license on the > README.md page. We should have it too! -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-6239) Add Build and License Badges to README.md
[ https://issues.apache.org/jira/browse/DRILL-6239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16399087#comment-16399087 ] ASF GitHub Bot commented on DRILL-6239: --- Github user ilooner commented on a diff in the pull request: https://github.com/apache/drill/pull/1165#discussion_r174570587 --- Diff: README.md --- @@ -1,5 +1,8 @@ # Apache Drill +[](https://travis-ci.org/apache/drill) --- End diff -- Oh carp! Thanks for catching this. > Add Build and License Badges to README.md > - > > Key: DRILL-6239 > URL: https://issues.apache.org/jira/browse/DRILL-6239 > Project: Apache Drill > Issue Type: Improvement >Reporter: Timothy Farkas >Assignee: Timothy Farkas >Priority: Major > Labels: ready-to-commit > Fix For: 1.14.0 > > > Other projects have pretty badges showing the build status and license on the > README.md page. We should have it too! -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-6199) Filter push down doesn't work with more than one nested subqueries
[
https://issues.apache.org/jira/browse/DRILL-6199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16399078#comment-16399078
]
ASF GitHub Bot commented on DRILL-6199:
---
Github user HanumathRao commented on a diff in the pull request:
https://github.com/apache/drill/pull/1152#discussion_r174568699
--- Diff:
exec/java-exec/src/test/java/org/apache/drill/exec/store/parquet/TestPushDownAndPruningWithItemStar.java
---
@@ -180,4 +248,38 @@ public void testFilterPushDownMultipleConditions()
throws Exception {
.build();
}
+ @Test
+ public void testFilterPushDownWithSeveralNestedStarSubQueries() throws
Exception {
+String subQuery = String.format("select * from `%s`.`%s`",
DFS_TMP_SCHEMA, TABLE_NAME);
+String query = String.format("select * from (select * from (select *
from (%s))) where o_orderdate = date '1992-01-01'", subQuery);
+
+String[] expectedPlan = {"numFiles=1, numRowGroups=1,
usedMetadataFile=false, columns=\\[`\\*\\*`, `o_orderdate`\\]"};
+String[] excludedPlan = {};
+
+PlanTestBase.testPlanMatchingPatterns(query, expectedPlan,
excludedPlan);
+
+testBuilder()
+.sqlQuery(query)
+.unOrdered()
+.sqlBaselineQuery("select * from `%s`.`%s` where o_orderdate =
date '1992-01-01'", DFS_TMP_SCHEMA, TABLE_NAME)
+.build();
+ }
+
+ @Test
+ public void
testFilterPushDownWithSeveralNestedStarSubQueriesWithAdditionalColumns() throws
Exception {
+String subQuery = String.format("select * from `%s`.`%s`",
DFS_TMP_SCHEMA, TABLE_NAME);
+String query = String.format("select * from (select * from (select *,
o_orderdate from (%s))) where o_orderdate = date '1992-01-01'", subQuery);
--- End diff --
Is it better to use other column than o_orderdate in the inside subquery?
> Filter push down doesn't work with more than one nested subqueries
> --
>
> Key: DRILL-6199
> URL: https://issues.apache.org/jira/browse/DRILL-6199
> Project: Apache Drill
> Issue Type: Bug
>Affects Versions: 1.13.0
>Reporter: Anton Gozhiy
>Assignee: Arina Ielchiieva
>Priority: Major
> Fix For: 1.14.0
>
> Attachments: DRILL_6118_data_source.csv
>
>
> *Data set:*
> The data is generated used the attached file: *DRILL_6118_data_source.csv*
> Data gen commands:
> {code:sql}
> create table dfs.tmp.`DRILL_6118_parquet_partitioned_by_folders/d1` (c1, c2,
> c3, c4, c5) as select cast(columns[0] as int) c1, columns[1] c2, columns[2]
> c3, columns[3] c4, columns[4] c5 from dfs.tmp.`DRILL_6118_data_source.csv`
> where columns[0] in (1, 3);
> create table dfs.tmp.`DRILL_6118_parquet_partitioned_by_folders/d2` (c1, c2,
> c3, c4, c5) as select cast(columns[0] as int) c1, columns[1] c2, columns[2]
> c3, columns[3] c4, columns[4] c5 from dfs.tmp.`DRILL_6118_data_source.csv`
> where columns[0]=2;
> create table dfs.tmp.`DRILL_6118_parquet_partitioned_by_folders/d3` (c1, c2,
> c3, c4, c5) as select cast(columns[0] as int) c1, columns[1] c2, columns[2]
> c3, columns[3] c4, columns[4] c5 from dfs.tmp.`DRILL_6118_data_source.csv`
> where columns[0]>3;
> {code}
> *Steps:*
> # Execute the following query:
> {code:sql}
> explain plan for select * from (select * from (select * from
> dfs.tmp.`DRILL_6118_parquet_partitioned_by_folders`)) where c1<3
> {code}
> *Expected result:*
> numFiles=2, numRowGroups=2, only files from the folders d1 and d2 should be
> scanned.
> *Actual result:*
> Filter push down doesn't work:
> numFiles=3, numRowGroups=3, scanning from all files
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (DRILL-6247) Minor fragments remain in CANCELLATION_REQUESTED state after query failure
[
https://issues.apache.org/jira/browse/DRILL-6247?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pritesh Maker reassigned DRILL-6247:
Assignee: Vlad Rozov
> Minor fragments remain in CANCELLATION_REQUESTED state after query failure
> --
>
> Key: DRILL-6247
> URL: https://issues.apache.org/jira/browse/DRILL-6247
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Flow
>Affects Versions: 1.13.0
>Reporter: Khurram Faraaz
>Assignee: Vlad Rozov
>Priority: Major
> Attachments: cancellation_requested_march_14.png,
> drillbit_snippet.log, jstack_cancellation_requested.txt
>
>
> Once a query fails, in this case due to an OOM in RPC we see many minor
> fragments are reported to be in CANCELLATION_REQUESTED state on Web UI after
> query has failed. The problem is reproducible. drillbit.log for this failure
> and jstack output are attached here.
> To reproduce the problem on a 4 node cluster.
> alter system reset all;
> alter system set `planner.slice_target`=1;
> Failing query => SELECT * , FLATTEN(arr) FROM many_json_files;
> Drill 1.13.0-SNAPSHOT, commit id: 766315ea17377199897d685ab801edd38394fe01
> Stack trace from output of jstack, fragment 0:0 is reported to be in
> CANCELLATION_REQUESTED state on Drill Web UI
> jstack -l 13488 > jstack_DRILL_6235.txt
> {noformat}
> "25569e98-10f9-2fe2-9dec-0a42f3ad45fa:frag:0:0" #87 daemon prio=10 os_prio=0
> tid=0x7f9d01374360 nid=0x2ff5 waiting on condition [0x7f9cd5536000]
> java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x0007a388b300> (a
> org.apache.drill.exec.rpc.ResettableBarrier$InternalSynchronizer)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
> at
> org.apache.drill.exec.rpc.ResettableBarrier.await(ResettableBarrier.java:70)
> at
> org.apache.drill.exec.rpc.AbstractRemoteConnection$WriteManager.waitForWritable(AbstractRemoteConnection.java:114)
> at
> org.apache.drill.exec.rpc.AbstractRemoteConnection.blockOnNotWritable(AbstractRemoteConnection.java:76)
> at org.apache.drill.exec.rpc.RpcBus.send(RpcBus.java:108)
> at
> org.apache.drill.exec.rpc.user.UserServer$BitToUserConnection.sendData(UserServer.java:275)
> at
> org.apache.drill.exec.ops.AccountingUserConnection.sendData(AccountingUserConnection.java:42)
> at
> org.apache.drill.exec.physical.impl.ScreenCreator$ScreenRoot.innerNext(ScreenCreator.java:120)
> at
> org.apache.drill.exec.physical.impl.BaseRootExec.next(BaseRootExec.java:95)
> at
> org.apache.drill.exec.work.fragment.FragmentExecutor$1.run(FragmentExecutor.java:233)
> at
> org.apache.drill.exec.work.fragment.FragmentExecutor$1.run(FragmentExecutor.java:226)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1595)
> at
> org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:226)
> at
> org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> {noformat}
> Stack trace from drillbit.log
> {noformat}
> 2018-03-14 10:52:44,545 [25569e98-10f9-2fe2-9dec-0a42f3ad45fa:frag:1:49] INFO
> o.a.d.e.w.fragment.FragmentExecutor - User Error Occurred: One or more nodes
> ran out of memory while executing the query. (Failure allocating buffer.)
> org.apache.drill.common.exceptions.UserException: RESOURCE ERROR: One or more
> nodes ran out of memory while executing the query.
> Failure allocating buffer.
> [Error Id: b83884df-af31-411a-9b28-554c294a7357 ]
> at
> org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:633)
> ~[drill-common-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT]
> at
> org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:243)
> [drill-java-exec-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT]
> at
> org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38)
> [drill-common-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[jira] [Created] (DRILL-6247) Minor fragments remain in CANCELLATION_REQUESTED state after query failure
Khurram Faraaz created DRILL-6247:
-
Summary: Minor fragments remain in CANCELLATION_REQUESTED state
after query failure
Key: DRILL-6247
URL: https://issues.apache.org/jira/browse/DRILL-6247
Project: Apache Drill
Issue Type: Bug
Components: Execution - Flow
Affects Versions: 1.13.0
Reporter: Khurram Faraaz
Attachments: cancellation_requested_march_14.png,
drillbit_snippet.log, jstack_cancellation_requested.txt
Once a query fails, in this case due to an OOM in RPC we see many minor
fragments are reported to be in CANCELLATION_REQUESTED state on Web UI after
query has failed. The problem is reproducible. drillbit.log for this failure
and jstack output are attached here.
To reproduce the problem on a 4 node cluster.
alter system reset all;
alter system set `planner.slice_target`=1;
Failing query => SELECT * , FLATTEN(arr) FROM many_json_files;
Drill 1.13.0-SNAPSHOT, commit id: 766315ea17377199897d685ab801edd38394fe01
Stack trace from output of jstack, fragment 0:0 is reported to be in
CANCELLATION_REQUESTED state on Drill Web UI
jstack -l 13488 > jstack_DRILL_6235.txt
{noformat}
"25569e98-10f9-2fe2-9dec-0a42f3ad45fa:frag:0:0" #87 daemon prio=10 os_prio=0
tid=0x7f9d01374360 nid=0x2ff5 waiting on condition [0x7f9cd5536000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x0007a388b300> (a
org.apache.drill.exec.rpc.ResettableBarrier$InternalSynchronizer)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
at org.apache.drill.exec.rpc.ResettableBarrier.await(ResettableBarrier.java:70)
at
org.apache.drill.exec.rpc.AbstractRemoteConnection$WriteManager.waitForWritable(AbstractRemoteConnection.java:114)
at
org.apache.drill.exec.rpc.AbstractRemoteConnection.blockOnNotWritable(AbstractRemoteConnection.java:76)
at org.apache.drill.exec.rpc.RpcBus.send(RpcBus.java:108)
at
org.apache.drill.exec.rpc.user.UserServer$BitToUserConnection.sendData(UserServer.java:275)
at
org.apache.drill.exec.ops.AccountingUserConnection.sendData(AccountingUserConnection.java:42)
at
org.apache.drill.exec.physical.impl.ScreenCreator$ScreenRoot.innerNext(ScreenCreator.java:120)
at org.apache.drill.exec.physical.impl.BaseRootExec.next(BaseRootExec.java:95)
at
org.apache.drill.exec.work.fragment.FragmentExecutor$1.run(FragmentExecutor.java:233)
at
org.apache.drill.exec.work.fragment.FragmentExecutor$1.run(FragmentExecutor.java:226)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1595)
at
org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:226)
at
org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
{noformat}
Stack trace from drillbit.log
{noformat}
2018-03-14 10:52:44,545 [25569e98-10f9-2fe2-9dec-0a42f3ad45fa:frag:1:49] INFO
o.a.d.e.w.fragment.FragmentExecutor - User Error Occurred: One or more nodes
ran out of memory while executing the query. (Failure allocating buffer.)
org.apache.drill.common.exceptions.UserException: RESOURCE ERROR: One or more
nodes ran out of memory while executing the query.
Failure allocating buffer.
[Error Id: b83884df-af31-411a-9b28-554c294a7357 ]
at
org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:633)
~[drill-common-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT]
at
org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:243)
[drill-java-exec-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT]
at
org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38)
[drill-common-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[na:1.8.0_161]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[na:1.8.0_161]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0_161]
Caused by: org.apache.drill.exec.exception.OutOfMemoryException: Failure
allocating buffer.
at
io.netty.buffer.PooledByteBufAllocatorL.allocate(PooledByteBufAllocatorL.java:67)
~[drill-memory-base-1.13.0-SNAPSHOT.jar:4.0.48.Final]
at
org.apache.drill.exec.me
[jira] [Comment Edited] (DRILL-6242) Output format for nested date, time, timestamp values in an object hierarchy
[
https://issues.apache.org/jira/browse/DRILL-6242?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16399049#comment-16399049
]
Jiang Wu edited comment on DRILL-6242 at 3/14/18 6:25 PM:
--
I think there are two causes for the above:
1) When outputting hierarchical data, Drill places the data inside
JsonStringHashMap and JsonStringArrayList objects. Both of these classes use
their own private static ObjectMapper to serialize the content to JSON string
representation. However, the ObjectMapper does not have any configuration to
serialize time based data types to their logical value. Instead, by using the
default settings, the serialization will expose all the getters of the
org.joda.time.DataTime class.
2) When the data is retrieved from Drill vectors for date, time, or timestamp,
the code always use a org.joda.time.DateTime as the java object. See
NullableDateVector.java:429 as an example. The result is that regardless of
the underlying data being a date, a time, or a timestamp, the Java
representation is always DateTime. This is not quite correct as it loses the
actual type of the underlying data.
Possible fixes:
A) add a mixin for DateTime in 1) to output logical timestamp value. However,
this won't fix 2) and the result is that all date, time, and timestamp will
have the same output.
B) do not use DateTime directly for 2). Instead, introduce 3 subclasses of
DateTime for Drill Date, Drill Time, Drill Timestamp. These 3 subclass have
built-in ser-de to output the right JSON representation. This is similar to
how java.sql.Time, java.sql.Timestamp, java.sql.Date work. These 3 classes are
subclass of java.util.Date, but with built-in serde for the proper string
representation.
C) Can we use java.sql.Time, java.sql.Date, and java.sql.Timestamp for 2)
instead of DateTime?
was (Author: wu):
I think there are two causes for the above:
1) When outputting hierarchical data, Drill places the data inside
JsonStringHashMap and JsonStringArrayList objects. Both of these classes use
their own private static ObjectMapper to serialize the content to JSON string
representation. However, the ObjectMapper does not have any configuration to
serialize time based data types to their logical value. Instead, by using the
default settings, the serialization will expose all the getters of the
org.joda.time.DataTime class.
2) When the data is retrieved from Drill vectors for date, time, or timestamp,
the code always use a org.joda.time.DateTime as the java object. See
NullableDateVector.java:429 as an example. The result is that regardless of
the underlying data being a date, a time, or a timestamp, the Java
representation is always DateTime. This is not quite correct as it loses the
actual type of the underlying data.
Possible fixes:
A) add a mixin for DateTime in 1) to output logical timestamp value. However,
this won't fix 2) and the result is that all date, time, and timestamp will
have the same output.
B) do not use DateTime directly for 2). Instead, introduce 3 subclasses of
DateTime for Drill Date, Drill Time, Drill Timestamp. These 3 subclass have
built-in ser-de to output the right JSON representation. This is similar to
how java.sql.Time, java.sql.Timestamp, java.sql.Date work. These 3 classes are
subclass of java.util.Date, but with built-in serde for the proper string
representation.
> Output format for nested date, time, timestamp values in an object hierarchy
>
>
> Key: DRILL-6242
> URL: https://issues.apache.org/jira/browse/DRILL-6242
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Data Types
>Affects Versions: 1.12.0
>Reporter: Jiang Wu
>Priority: Major
>
> Some storages (mapr db, mongo db, etc.) have hierarchical objects that
> contain nested fields of date, time, timestamp types. When a query returns
> these objects, the output format for the nested date, time, timestamp, are
> showing the internal object (org.joda.time.DateTime), rather than the logical
> data value.
> For example. Suppose in MongoDB, we have a single object that looks like
> this:
> {code:java}
> > db.test.findOne();
> {
> "_id" : ObjectId("5aa8487d470dd39a635a12f5"),
> "name" : "orange",
> "context" : {
> "date" : ISODate("2018-03-13T21:52:54.940Z"),
> "user" : "jack"
> }
> }
> {code}
> Then connect Drill to the above MongoDB storage, and run the following query
> within Drill:
> {code:java}
> > select t.context.`date`, t.context from test t;
> ++-+
> | EXPR$0 | context |
> ++-+
> | 2018-03-13 |
> {"date":{"dayOfYear":72,"year":2018,"dayOfMonth":13,"dayOfWeek":2,"era":1,"millisOfDay":78774940,"weekOfWeekyear"
[jira] [Comment Edited] (DRILL-6242) Output format for nested date, time, timestamp values in an object hierarchy
[
https://issues.apache.org/jira/browse/DRILL-6242?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16399049#comment-16399049
]
Jiang Wu edited comment on DRILL-6242 at 3/14/18 6:24 PM:
--
I think there are two causes for the above:
1) When outputting hierarchical data, Drill places the data inside
JsonStringHashMap and JsonStringArrayList objects. Both of these classes use
their own private static ObjectMapper to serialize the content to JSON string
representation. However, the ObjectMapper does not have any configuration to
serialize time based data types to their logical value. Instead, by using the
default settings, the serialization will expose all the getters of the
org.joda.time.DataTime class.
2) When the data is retrieved from Drill vectors for date, time, or timestamp,
the code always use a org.joda.time.DateTime as the java object. See
NullableDateVector.java:429 as an example. The result is that regardless of
the underlying data being a date, a time, or a timestamp, the Java
representation is always DateTime. This is not quite correct as it loses the
actual type of the underlying data.
Possible fixes:
A) add a mixin for DateTime in 1) to output logical timestamp value. However,
this won't fix 2) and the result is that all date, time, and timestamp will
have the same output.
B) do not use DateTime directly for 2). Instead, introduce 3 subclasses of
DateTime for Drill Date, Drill Time, Drill Timestamp. These 3 subclass have
built-in ser-de to output the right JSON representation. This is similar to
how java.sql.Time, java.sql.Timestamp, java.sql.Date work. These 3 classes are
subclass of java.util.Date, but with built-in serde for the proper string
representation.
was (Author: wu):
I think there are two causes for the above:
1) When outputting hierarchical data, Drill places the data inside
JsonStringHashMap and JsonStringArrayList objects. Both of these classes use
their own private static ObjectMapper to serialize the content to JSON string
representation. However, the ObjectMapper does not have any configuration to
serialize time based data types to their logical value. Instead, by using the
default settings, the serialization will expose all the getters of the
org.joda.time.DataTime class.
2) When the data is retrieved from Drill vectors for date, time, or timestamp,
the code always use a org.joda.time.DateTime as the java object. See
NullableDateVector.java:429 as an example. The result is that regardless of
the underlying data being a date, a time, or a timestamp, the Java
representation is always DateTime. This is not quite correct as it loses the
actual type of the underlying data.
Possible fixes:
A) add a mixin for DateTime in 1) to output logical timestamp value. However,
this won't fix 2) and the result is that all date, time, and timestamp will
have the same output.
B) do not use DateTime directly for 2). Rather introduce 3 subclasses of
DateTime for Drill Date, Drill Time, Drill Timestamp. These 3 subclass have
built-in ser-de to output the right JSON representation. This is similar to
how java.sql.Time, java.sql.Timestamp, java.sql.Date work. These 3 classes are
subclass of java.util.Date, but with built-in serde for the proper string
representation.
> Output format for nested date, time, timestamp values in an object hierarchy
>
>
> Key: DRILL-6242
> URL: https://issues.apache.org/jira/browse/DRILL-6242
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Data Types
>Affects Versions: 1.12.0
>Reporter: Jiang Wu
>Priority: Major
>
> Some storages (mapr db, mongo db, etc.) have hierarchical objects that
> contain nested fields of date, time, timestamp types. When a query returns
> these objects, the output format for the nested date, time, timestamp, are
> showing the internal object (org.joda.time.DateTime), rather than the logical
> data value.
> For example. Suppose in MongoDB, we have a single object that looks like
> this:
> {code:java}
> > db.test.findOne();
> {
> "_id" : ObjectId("5aa8487d470dd39a635a12f5"),
> "name" : "orange",
> "context" : {
> "date" : ISODate("2018-03-13T21:52:54.940Z"),
> "user" : "jack"
> }
> }
> {code}
> Then connect Drill to the above MongoDB storage, and run the following query
> within Drill:
> {code:java}
> > select t.context.`date`, t.context from test t;
> ++-+
> | EXPR$0 | context |
> ++-+
> | 2018-03-13 |
> {"date":{"dayOfYear":72,"year":2018,"dayOfMonth":13,"dayOfWeek":2,"era":1,"millisOfDay":78774940,"weekOfWeekyear":11,"weekyear":2018,"monthOfYear":3,"yearOfEra":2018,"yearOfCentury":18,"centuryOfEra":20,"millisOf
[jira] [Comment Edited] (DRILL-6242) Output format for nested date, time, timestamp values in an object hierarchy
[
https://issues.apache.org/jira/browse/DRILL-6242?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16399049#comment-16399049
]
Jiang Wu edited comment on DRILL-6242 at 3/14/18 6:23 PM:
--
I think there are two causes for the above:
1) When outputting hierarchical data, Drill places the data inside
JsonStringHashMap and JsonStringArrayList objects. Both of these classes use
their own private static ObjectMapper to serialize the content to JSON string
representation. However, the ObjectMapper does not have any configuration to
serialize time based data types to their logical value. Instead, by using the
default settings, the serialization will expose all the getters of the
org.joda.time.DataTime class.
2) When the data is retrieved from Drill vectors for date, time, or timestamp,
the code always use a org.joda.time.DateTime as the java object. See
NullableDateVector.java:429 as an example. The result is that regardless of
the underlying data being a date, a time, or a timestamp, the Java
representation is always DateTime. This is not quite correct as it loses the
actual type of the underlying data.
Possible fixes:
A) add a mixin for DateTime in 1) to output logical timestamp value. However,
this won't fix 2) and the result is that all date, time, and timestamp will
have the same output.
B) do not use DateTime directly for 2). Rather introduce 3 subclasses of
DateTime for Drill Date, Drill Time, Drill Timestamp. These 3 subclass have
built-in ser-de to output the right JSON representation. This is similar to
how java.sql.Time, java.sql.Timestamp, java.sql.Date work. These 3 classes are
subclass of java.util.Date, but with built-in serde for the proper string
representation.
was (Author: wu):
I think there are two causes for the above:
1) When outputting hierarchical data, Drill places the data inside
JsonStringHashMap and JsonStringArrayList objects. Both of these classes use
their own private static ObjectMapper to serialize the content to JSON string
representation. However, the ObjectMapper does not have any configuration to
serialize time based data types to their logical value. Instead, by using the
default settings, the serialization will expose all the getters of the
org.joda.time.DataTime class.
2) When the data is retrieved from Drill vectors for date, time, or timestamp,
the code always use a org.joda.time.DateTime as the java object. See
NullableDateVector.java:429 as an example. The result is that regardless of
the underlying data being a date, a time, or a timestamp, the Java
representation is always DateTime. This is not quite correct as it loses the
actual type of the underlying data.
Possible fixes:
A) add a mixin for DateTime in 1) to output logical timestamp value. However,
this won't fix 2) and the result is that all date, time, and timestamp will
have the same output.
B) do not use DateTime directly from 2). Rather introduce 3 subclasses of
DateTime for Drill Date, Drill Time, Drill Timestamp. These 3 subclass have
built-in ser-de to output the right JSON representation. This is similar to
how java.sql.Time, java.sql.Timestamp, java.sql.Date work. These 3 classes are
subclass of java.util.Date, but with built-in serde for the proper string
representation.
> Output format for nested date, time, timestamp values in an object hierarchy
>
>
> Key: DRILL-6242
> URL: https://issues.apache.org/jira/browse/DRILL-6242
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Data Types
>Affects Versions: 1.12.0
>Reporter: Jiang Wu
>Priority: Major
>
> Some storages (mapr db, mongo db, etc.) have hierarchical objects that
> contain nested fields of date, time, timestamp types. When a query returns
> these objects, the output format for the nested date, time, timestamp, are
> showing the internal object (org.joda.time.DateTime), rather than the logical
> data value.
> For example. Suppose in MongoDB, we have a single object that looks like
> this:
> {code:java}
> > db.test.findOne();
> {
> "_id" : ObjectId("5aa8487d470dd39a635a12f5"),
> "name" : "orange",
> "context" : {
> "date" : ISODate("2018-03-13T21:52:54.940Z"),
> "user" : "jack"
> }
> }
> {code}
> Then connect Drill to the above MongoDB storage, and run the following query
> within Drill:
> {code:java}
> > select t.context.`date`, t.context from test t;
> ++-+
> | EXPR$0 | context |
> ++-+
> | 2018-03-13 |
> {"date":{"dayOfYear":72,"year":2018,"dayOfMonth":13,"dayOfWeek":2,"era":1,"millisOfDay":78774940,"weekOfWeekyear":11,"weekyear":2018,"monthOfYear":3,"yearOfEra":2018,"yearOfCentury":18,"centuryOfEra":20,"millisOfS
[jira] [Comment Edited] (DRILL-6242) Output format for nested date, time, timestamp values in an object hierarchy
[
https://issues.apache.org/jira/browse/DRILL-6242?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16399049#comment-16399049
]
Jiang Wu edited comment on DRILL-6242 at 3/14/18 6:23 PM:
--
I think there are two causes for the above:
1) When outputting hierarchical data, Drill places the data inside
JsonStringHashMap and JsonStringArrayList objects. Both of these classes use
their own private static ObjectMapper to serialize the content to JSON string
representation. However, the ObjectMapper does not have any configuration to
serialize time based data types to their logical value. Instead, by using the
default settings, the serialization will expose all the getters of the
org.joda.time.DataTime class.
2) When the data is retrieved from Drill vectors for date, time, or timestamp,
the code always use a org.joda.time.DateTime as the java object. See
NullableDateVector.java:429 as an example. The result is that regardless of
the underlying data being a date, a time, or a timestamp, the Java
representation is always DateTime. This is not quite correct as it loses the
actual type of the underlying data.
Possible fixes:
A) add a mixin for DateTime in 1) to output logical timestamp value. However,
this won't fix 2) and the result is that all date, time, and timestamp will
have the same output.
B) do not use DateTime directly from 2). Rather introduce 3 subclasses of
DateTime for Drill Date, Drill Time, Drill Timestamp. These 3 subclass have
built-in ser-de to output the right JSON representation. This is similar to
how java.sql.Time, java.sql.Timestamp, java.sql.Date work. These 3 classes are
subclass of java.util.Date, but with built-in serde for the proper string
representation.
was (Author: wu):
I think there are two causes for the above:
1) When outputting hierarchical data, Drill places the data inside
JsonStringHashMap and JsonStringArrayList objects. Both of these classes use
their own private static ObjectMapper to serialize the content to JSON string
representation. However, the ObjectMapper does not have any configuration to
serialize time based data types to their logical value. Instead, by using the
default settings, the serialization will expose all the getters of the
org.joda.time.DataTime class.
2) When the data is retrieved from Drill vectors for date, time, or timestamp,
the code always use a org.joda.time.DateTime as the java object. See
NullableDateVector.java:429 as an example. The result is that regardless of
the underlying data being a date, a time, or a timestamp, the Java
representation is always DateTime.
Possible fixes:
A) add a mixin for DateTime in 1) to output logical timestamp value. However,
this won't fix 2) and the result is that all date, time, and timestamp will
have the same output.
B) do not use DateTime directly from 2). Rather introduce 3 subclasses of
DateTime for Drill Date, Drill Time, Drill Timestamp. These 3 subclass have
built-in ser-de to output the right JSON representation. This is similar to
how java.sql.Time, java.sql.Timestamp, java.sql.Date work. These 3 classes are
subclass of java.util.Date, but with built-in serde for the proper string
representation.
> Output format for nested date, time, timestamp values in an object hierarchy
>
>
> Key: DRILL-6242
> URL: https://issues.apache.org/jira/browse/DRILL-6242
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Data Types
>Affects Versions: 1.12.0
>Reporter: Jiang Wu
>Priority: Major
>
> Some storages (mapr db, mongo db, etc.) have hierarchical objects that
> contain nested fields of date, time, timestamp types. When a query returns
> these objects, the output format for the nested date, time, timestamp, are
> showing the internal object (org.joda.time.DateTime), rather than the logical
> data value.
> For example. Suppose in MongoDB, we have a single object that looks like
> this:
> {code:java}
> > db.test.findOne();
> {
> "_id" : ObjectId("5aa8487d470dd39a635a12f5"),
> "name" : "orange",
> "context" : {
> "date" : ISODate("2018-03-13T21:52:54.940Z"),
> "user" : "jack"
> }
> }
> {code}
> Then connect Drill to the above MongoDB storage, and run the following query
> within Drill:
> {code:java}
> > select t.context.`date`, t.context from test t;
> ++-+
> | EXPR$0 | context |
> ++-+
> | 2018-03-13 |
> {"date":{"dayOfYear":72,"year":2018,"dayOfMonth":13,"dayOfWeek":2,"era":1,"millisOfDay":78774940,"weekOfWeekyear":11,"weekyear":2018,"monthOfYear":3,"yearOfEra":2018,"yearOfCentury":18,"centuryOfEra":20,"millisOfSecond":940,"secondOfMinute":54,"secondOfDay":78774,"minuteOfHour":52,"minuteOfD
[jira] [Commented] (DRILL-6242) Output format for nested date, time, timestamp values in an object hierarchy
[
https://issues.apache.org/jira/browse/DRILL-6242?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16399049#comment-16399049
]
Jiang Wu commented on DRILL-6242:
-
I think there are two causes for the above:
1) When outputting hierarchical data, Drill places the data inside
JsonStringHashMap and JsonStringArrayList objects. Both of these classes use
their own private static ObjectMapper to serialize the content to JSON string
representation. However, the ObjectMapper does not have any configuration to
serialize time based data types to their logical value. Instead, by using the
default settings, the serialization will expose all the getters of the
org.joda.time.DataTime class.
2) When the data is retrieved from Drill vectors for date, time, or timestamp,
the code always use a org.joda.time.DateTime as the java object. See
NullableDateVector.java:429 as an example. The result is that regardless of
the underlying data being a date, a time, or a timestamp, the Java
representation is always DateTime.
Possible fixes:
A) add a mixin for DateTime in 1) to output logical timestamp value. However,
this won't fix 2) and the result is that all date, time, and timestamp will
have the same output.
B) do not use DateTime directly from 2). Rather introduce 3 subclasses of
DateTime for Drill Date, Drill Time, Drill Timestamp. These 3 subclass have
built-in ser-de to output the right JSON representation. This is similar to
how java.sql.Time, java.sql.Timestamp, java.sql.Date work. These 3 classes are
subclass of java.util.Date, but with built-in serde for the proper string
representation.
> Output format for nested date, time, timestamp values in an object hierarchy
>
>
> Key: DRILL-6242
> URL: https://issues.apache.org/jira/browse/DRILL-6242
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Data Types
>Affects Versions: 1.12.0
>Reporter: Jiang Wu
>Priority: Major
>
> Some storages (mapr db, mongo db, etc.) have hierarchical objects that
> contain nested fields of date, time, timestamp types. When a query returns
> these objects, the output format for the nested date, time, timestamp, are
> showing the internal object (org.joda.time.DateTime), rather than the logical
> data value.
> For example. Suppose in MongoDB, we have a single object that looks like
> this:
> {code:java}
> > db.test.findOne();
> {
> "_id" : ObjectId("5aa8487d470dd39a635a12f5"),
> "name" : "orange",
> "context" : {
> "date" : ISODate("2018-03-13T21:52:54.940Z"),
> "user" : "jack"
> }
> }
> {code}
> Then connect Drill to the above MongoDB storage, and run the following query
> within Drill:
> {code:java}
> > select t.context.`date`, t.context from test t;
> ++-+
> | EXPR$0 | context |
> ++-+
> | 2018-03-13 |
> {"date":{"dayOfYear":72,"year":2018,"dayOfMonth":13,"dayOfWeek":2,"era":1,"millisOfDay":78774940,"weekOfWeekyear":11,"weekyear":2018,"monthOfYear":3,"yearOfEra":2018,"yearOfCentury":18,"centuryOfEra":20,"millisOfSecond":940,"secondOfMinute":54,"secondOfDay":78774,"minuteOfHour":52,"minuteOfDay":1312,"hourOfDay":21,"zone":{"fixed":true,"id":"UTC"},"millis":1520977974940,"chronology":{"zone":{"fixed":true,"id":"UTC"}},"afterNow":false,"beforeNow":true,"equalNow":false},"user":"jack"}
> |
> {code}
> We can see that from the above output, when the date field is retrieved as a
> top level column, Drill outputs a logical date value. But when the same
> field is within an object hierarchy, Drill outputs the internal object used
> to hold the date value.
> The expected output is the same display for whether the date field is shown
> as a top level column or when it is within an object hierarchy:
> {code:java}
> > select t.context.`date`, t.context from test t;
> ++-+
> | EXPR$0 | context |
> ++-+
> | 2018-03-13 | {"date":"2018-03-13","user":"jack"} |
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-6199) Filter push down doesn't work with more than one nested subqueries
[
https://issues.apache.org/jira/browse/DRILL-6199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16399011#comment-16399011
]
ASF GitHub Bot commented on DRILL-6199:
---
Github user HanumathRao commented on a diff in the pull request:
https://github.com/apache/drill/pull/1152#discussion_r174558288
--- Diff:
exec/java-exec/src/main/java/org/apache/drill/exec/planner/logical/DrillFilterItemStarReWriterRule.java
---
@@ -54,83 +44,189 @@
import static
org.apache.drill.exec.planner.logical.FieldsReWriterUtil.FieldsReWriter;
/**
- * Rule will transform filter -> project -> scan call with item star
fields in filter
- * into project -> filter -> project -> scan where item star fields are
pushed into scan
- * and replaced with actual field references.
+ * Rule will transform item star fields in filter and replaced with actual
field references.
*
* This will help partition pruning and push down rules to detect fields
that can be pruned or push downed.
* Item star operator appears when sub-select or cte with star are used as
source.
*/
-public class DrillFilterItemStarReWriterRule extends RelOptRule {
+public class DrillFilterItemStarReWriterRule {
- public static final DrillFilterItemStarReWriterRule INSTANCE = new
DrillFilterItemStarReWriterRule(
- RelOptHelper.some(Filter.class, RelOptHelper.some(Project.class,
RelOptHelper.any( TableScan.class))),
- "DrillFilterItemStarReWriterRule");
+ public static final DrillFilterItemStarReWriterRule.ProjectOnScan
PROJECT_ON_SCAN = new ProjectOnScan(
+ RelOptHelper.some(DrillProjectRel.class,
RelOptHelper.any(DrillScanRel.class)),
+ "DrillFilterItemStarReWriterRule.ProjectOnScan");
- private DrillFilterItemStarReWriterRule(RelOptRuleOperand operand,
String id) {
-super(operand, id);
- }
+ public static final DrillFilterItemStarReWriterRule.FilterOnScan
FILTER_ON_SCAN = new FilterOnScan(
+ RelOptHelper.some(DrillFilterRel.class,
RelOptHelper.any(DrillScanRel.class)),
+ "DrillFilterItemStarReWriterRule.FilterOnScan");
- @Override
- public void onMatch(RelOptRuleCall call) {
-Filter filterRel = call.rel(0);
-Project projectRel = call.rel(1);
-TableScan scanRel = call.rel(2);
+ public static final DrillFilterItemStarReWriterRule.FilterOnProject
FILTER_ON_PROJECT = new FilterOnProject(
+ RelOptHelper.some(DrillFilterRel.class,
RelOptHelper.some(DrillProjectRel.class, RelOptHelper.any(DrillScanRel.class))),
+ "DrillFilterItemStarReWriterRule.FilterOnProject");
-ItemStarFieldsVisitor itemStarFieldsVisitor = new
ItemStarFieldsVisitor(filterRel.getRowType().getFieldNames());
-filterRel.getCondition().accept(itemStarFieldsVisitor);
-// there are no item fields, no need to proceed further
-if (!itemStarFieldsVisitor.hasItemStarFields()) {
- return;
+ private static class ProjectOnScan extends RelOptRule {
+
+ProjectOnScan(RelOptRuleOperand operand, String id) {
+ super(operand, id);
}
-Map itemStarFields =
itemStarFieldsVisitor.getItemStarFields();
+@Override
+public boolean matches(RelOptRuleCall call) {
+ DrillScanRel scan = call.rel(1);
+ return scan.getGroupScan() instanceof ParquetGroupScan &&
super.matches(call);
+}
-// create new scan
-RelNode newScan = constructNewScan(scanRel, itemStarFields.keySet());
+@Override
+public void onMatch(RelOptRuleCall call) {
+ DrillProjectRel projectRel = call.rel(0);
+ DrillScanRel scanRel = call.rel(1);
+
+ ItemStarFieldsVisitor itemStarFieldsVisitor = new
ItemStarFieldsVisitor(scanRel.getRowType().getFieldNames());
+ List projects = projectRel.getProjects();
+ for (RexNode project : projects) {
+project.accept(itemStarFieldsVisitor);
+ }
-// combine original and new projects
-List newProjects = new ArrayList<>(projectRel.getProjects());
+ Map itemStarFields =
itemStarFieldsVisitor.getItemStarFields();
-// prepare node mapper to replace item star calls with new input field
references
-Map fieldMapper = new HashMap<>();
+ // if there are no item fields, no need to proceed further
+ if (itemStarFieldsVisitor.hasNoItemStarFields()) {
--- End diff --
Can this be moved to before getItemStarFields call?.
> Filter push down doesn't work with more than one nested subqueries
> --
>
> Key: DRILL-6199
> URL: https://issues.apache.org/jira/browse/DRILL-6199
> Project
[jira] [Commented] (DRILL-6199) Filter push down doesn't work with more than one nested subqueries
[
https://issues.apache.org/jira/browse/DRILL-6199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16399008#comment-16399008
]
ASF GitHub Bot commented on DRILL-6199:
---
Github user HanumathRao commented on a diff in the pull request:
https://github.com/apache/drill/pull/1152#discussion_r174558063
--- Diff:
exec/java-exec/src/main/java/org/apache/drill/exec/planner/logical/DrillFilterItemStarReWriterRule.java
---
@@ -54,83 +44,189 @@
import static
org.apache.drill.exec.planner.logical.FieldsReWriterUtil.FieldsReWriter;
/**
- * Rule will transform filter -> project -> scan call with item star
fields in filter
- * into project -> filter -> project -> scan where item star fields are
pushed into scan
- * and replaced with actual field references.
+ * Rule will transform item star fields in filter and replaced with actual
field references.
*
* This will help partition pruning and push down rules to detect fields
that can be pruned or push downed.
* Item star operator appears when sub-select or cte with star are used as
source.
*/
-public class DrillFilterItemStarReWriterRule extends RelOptRule {
+public class DrillFilterItemStarReWriterRule {
- public static final DrillFilterItemStarReWriterRule INSTANCE = new
DrillFilterItemStarReWriterRule(
- RelOptHelper.some(Filter.class, RelOptHelper.some(Project.class,
RelOptHelper.any( TableScan.class))),
- "DrillFilterItemStarReWriterRule");
+ public static final DrillFilterItemStarReWriterRule.ProjectOnScan
PROJECT_ON_SCAN = new ProjectOnScan(
+ RelOptHelper.some(DrillProjectRel.class,
RelOptHelper.any(DrillScanRel.class)),
+ "DrillFilterItemStarReWriterRule.ProjectOnScan");
- private DrillFilterItemStarReWriterRule(RelOptRuleOperand operand,
String id) {
-super(operand, id);
- }
+ public static final DrillFilterItemStarReWriterRule.FilterOnScan
FILTER_ON_SCAN = new FilterOnScan(
+ RelOptHelper.some(DrillFilterRel.class,
RelOptHelper.any(DrillScanRel.class)),
+ "DrillFilterItemStarReWriterRule.FilterOnScan");
- @Override
- public void onMatch(RelOptRuleCall call) {
-Filter filterRel = call.rel(0);
-Project projectRel = call.rel(1);
-TableScan scanRel = call.rel(2);
+ public static final DrillFilterItemStarReWriterRule.FilterOnProject
FILTER_ON_PROJECT = new FilterOnProject(
--- End diff --
Would it be good to rename this as FILTER_PROJECT_SCAN?
> Filter push down doesn't work with more than one nested subqueries
> --
>
> Key: DRILL-6199
> URL: https://issues.apache.org/jira/browse/DRILL-6199
> Project: Apache Drill
> Issue Type: Bug
>Affects Versions: 1.13.0
>Reporter: Anton Gozhiy
>Assignee: Arina Ielchiieva
>Priority: Major
> Fix For: 1.14.0
>
> Attachments: DRILL_6118_data_source.csv
>
>
> *Data set:*
> The data is generated used the attached file: *DRILL_6118_data_source.csv*
> Data gen commands:
> {code:sql}
> create table dfs.tmp.`DRILL_6118_parquet_partitioned_by_folders/d1` (c1, c2,
> c3, c4, c5) as select cast(columns[0] as int) c1, columns[1] c2, columns[2]
> c3, columns[3] c4, columns[4] c5 from dfs.tmp.`DRILL_6118_data_source.csv`
> where columns[0] in (1, 3);
> create table dfs.tmp.`DRILL_6118_parquet_partitioned_by_folders/d2` (c1, c2,
> c3, c4, c5) as select cast(columns[0] as int) c1, columns[1] c2, columns[2]
> c3, columns[3] c4, columns[4] c5 from dfs.tmp.`DRILL_6118_data_source.csv`
> where columns[0]=2;
> create table dfs.tmp.`DRILL_6118_parquet_partitioned_by_folders/d3` (c1, c2,
> c3, c4, c5) as select cast(columns[0] as int) c1, columns[1] c2, columns[2]
> c3, columns[3] c4, columns[4] c5 from dfs.tmp.`DRILL_6118_data_source.csv`
> where columns[0]>3;
> {code}
> *Steps:*
> # Execute the following query:
> {code:sql}
> explain plan for select * from (select * from (select * from
> dfs.tmp.`DRILL_6118_parquet_partitioned_by_folders`)) where c1<3
> {code}
> *Expected result:*
> numFiles=2, numRowGroups=2, only files from the folders d1 and d2 should be
> scanned.
> *Actual result:*
> Filter push down doesn't work:
> numFiles=3, numRowGroups=3, scanning from all files
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-6053) Avoid excessive locking in LocalPersistentStore
[
https://issues.apache.org/jira/browse/DRILL-6053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16398981#comment-16398981
]
ASF GitHub Bot commented on DRILL-6053:
---
Github user vrozov commented on a diff in the pull request:
https://github.com/apache/drill/pull/1163#discussion_r174551433
--- Diff: common/src/main/java/org/apache/drill/common/AutoCloseables.java
---
@@ -25,6 +25,11 @@
*/
public class AutoCloseables {
+ public interface Closeable extends AutoCloseable {
--- End diff --
It is on purpose:
- There is a minimal difference between Drill `Closeable` and Java
'Closeable/AutoCloseable`, so name reflects that.
- It won't be possible to use Drill `Closeable` in place of Java
`Closeable/AutoCloseable` in case `close()` throws checked exception.
- Drill `Closeable` is not a top level interface, so where it is necessary
to distinguish Drill `Closeable` from Java `Closeable` full name
`AutoClosables.Closeable` should be used.
> Avoid excessive locking in LocalPersistentStore
> ---
>
> Key: DRILL-6053
> URL: https://issues.apache.org/jira/browse/DRILL-6053
> Project: Apache Drill
> Issue Type: Improvement
>Reporter: Vlad Rozov
>Assignee: Vlad Rozov
>Priority: Major
> Fix For: 1.14.0
>
>
> When query profiles are written to LocalPersistentStore, the write is
> unnecessary serialized due to read/write lock that was introduced for
> versioned PersistentStore. Only versioned access needs to be protected by
> read/write lock.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-6244) Ability to shutdown cross domain Drillbits
[ https://issues.apache.org/jira/browse/DRILL-6244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16398944#comment-16398944 ] Venkata Jyothsna Donapati commented on DRILL-6244: -- Following document explains the issues and some of the proposed solutions to solve this problem [https://docs.google.com/document/d/1LozFqXSuVHsOZ31gFNfI39h2W3IeJG84j9-eW6cq92Q/edit?usp=sharing] > Ability to shutdown cross domain Drillbits > -- > > Key: DRILL-6244 > URL: https://issues.apache.org/jira/browse/DRILL-6244 > Project: Apache Drill > Issue Type: Improvement >Reporter: Venkata Jyothsna Donapati >Priority: Minor > > Currently Drill Web UI supports shutdown of current Drillbit and doesn't > provide an option to shutdown cross domain Drillbits. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (DRILL-6246) Build Failing in jdbc-all artifact
salim achouche created DRILL-6246:
-
Summary: Build Failing in jdbc-all artifact
Key: DRILL-6246
URL: https://issues.apache.org/jira/browse/DRILL-6246
Project: Apache Drill
Issue Type: Bug
Components: Client - JDBC
Affects Versions: 1.13.0
Reporter: salim achouche
Assignee: salim achouche
* {color:#00}It was noticed that the build was failing because of the
jdbc-all artifact{color}
* {color:#00}The maximum compressed jar size was set to 32MB but we are
currently creating a JAR a bit larger than 32MB {color}
* {color:#00}I compared apache drill-1.10.0, drill-1.12.0, and
drill-1.13.0 (on my MacOS){color}
* {color:#00}jdbc-all-1.10.0 jar size: 21MB{color}
* {color:#00}jdbc-all-1.12.0 jar size: 27MB{color}
* {color:#00}jdbc-all-1.13.0 jar size: 34MB (on Linux this size is roughly
32MB){color}
* {color:#00}Compared then in more details jdbc-all-1.12.0 and
jdbc-all-1.13.0{color}
* {color:#00}The bulk of the increase is attributed to the calcite
artifact{color}
* {color:#00}Used to be 2MB (uncompressed) and now 22MB
(uncompressed){color}
* {color:#00}It is likely an exclusion problem {color}
* {color:#00}The jdbc-all-1.12.0 version has only two top packages
calcite/avatica/utils and calcite/avatica/remote{color}
* {color:#00}The jdbc-all-1.13.0 includes new packages (within
calcite/avatica) metrics, proto, org/apache/, com/fasterxml, com/google{color}
{color:#00} {color}
{color:#00}I am planning to exclude these new sub-packages{color}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (DRILL-6244) Ability to shutdown cross domain Drillbits
[ https://issues.apache.org/jira/browse/DRILL-6244?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Venkata Jyothsna Donapati updated DRILL-6244: - Priority: Minor (was: Major) > Ability to shutdown cross domain Drillbits > -- > > Key: DRILL-6244 > URL: https://issues.apache.org/jira/browse/DRILL-6244 > Project: Apache Drill > Issue Type: Improvement >Reporter: Venkata Jyothsna Donapati >Priority: Minor > > Currently Drill Web UI supports shutdown of current Drillbit and doesn't > provide an option to shutdown cross domain Drillbits. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-6245) Clicking on anything redirects to main login page
[ https://issues.apache.org/jira/browse/DRILL-6245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16398936#comment-16398936 ] Venkata Jyothsna Donapati commented on DRILL-6245: -- This might the root cause: [https://stackoverflow.com/questions/2163828/reading-cookies-via-https-that-were-set-using-http] > Clicking on anything redirects to main login page > - > > Key: DRILL-6245 > URL: https://issues.apache.org/jira/browse/DRILL-6245 > Project: Apache Drill > Issue Type: Bug >Reporter: Venkata Jyothsna Donapati >Priority: Minor > > When the Drill Web UI is accessed using https and then by http protocol, the > Web UI is always trying to redirect to main login page if anything is clicked > on index page. However, this works fine if the cookies are cleared. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-5270) Improve loading of profiles listing in the WebUI
[ https://issues.apache.org/jira/browse/DRILL-5270?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16398934#comment-16398934 ] ASF GitHub Bot commented on DRILL-5270: --- Github user kkhatua commented on the issue: https://github.com/apache/drill/pull/755 Holding off to do a rebase once @vrozov 's PR #1163 (DRILL-6053) goes into Apache. > Improve loading of profiles listing in the WebUI > > > Key: DRILL-5270 > URL: https://issues.apache.org/jira/browse/DRILL-5270 > Project: Apache Drill > Issue Type: Improvement > Components: Web Server >Affects Versions: 1.9.0 >Reporter: Kunal Khatua >Assignee: Kunal Khatua >Priority: Major > Fix For: 1.14.0 > > > Currently, as the number of profiles increase, we reload the same list of > profiles from the FS. > An ideal improvement would be to detect if there are any new profiles and > only reload from the disk then. Otherwise, a cached list is sufficient. > For a directory of 280K profiles, the load time is close to 6 seconds on a 32 > core server. With the caching, we can get it down to as much as a few > milliseconds. > To render the cache as invalid, we inspect the last modified time of the > directory to confirm whether a reload is needed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (DRILL-6245) Clicking on anything redirects to main login page
Venkata Jyothsna Donapati created DRILL-6245: Summary: Clicking on anything redirects to main login page Key: DRILL-6245 URL: https://issues.apache.org/jira/browse/DRILL-6245 Project: Apache Drill Issue Type: Improvement Reporter: Venkata Jyothsna Donapati When the Drill Web UI is accessed using https and then by http protocol, the Web UI is always trying to redirect to main login page if anything is clicked on index page. However, this works fine if the cookies are cleared. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (DRILL-6245) Clicking on anything redirects to main login page
[ https://issues.apache.org/jira/browse/DRILL-6245?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Venkata Jyothsna Donapati updated DRILL-6245: - Issue Type: Bug (was: Improvement) > Clicking on anything redirects to main login page > - > > Key: DRILL-6245 > URL: https://issues.apache.org/jira/browse/DRILL-6245 > Project: Apache Drill > Issue Type: Bug >Reporter: Venkata Jyothsna Donapati >Priority: Minor > > When the Drill Web UI is accessed using https and then by http protocol, the > Web UI is always trying to redirect to main login page if anything is clicked > on index page. However, this works fine if the cookies are cleared. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-6053) Avoid excessive locking in LocalPersistentStore
[
https://issues.apache.org/jira/browse/DRILL-6053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16398927#comment-16398927
]
ASF GitHub Bot commented on DRILL-6053:
---
Github user arina-ielchiieva commented on a diff in the pull request:
https://github.com/apache/drill/pull/1163#discussion_r174541733
--- Diff:
exec/java-exec/src/main/java/org/apache/drill/exec/store/sys/Store.java ---
@@ -0,0 +1,54 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.drill.exec.store.sys;
+
+import java.util.Iterator;
+import java.util.Map;
+
+public interface Store extends AutoCloseable
+{
--- End diff --
Well, we definitely need to improve Drill checkstyle. Tim used to bring up
this topic on dev mailing list. For instance, there are many things we need to
check, like javadocs for headers etc.
> Avoid excessive locking in LocalPersistentStore
> ---
>
> Key: DRILL-6053
> URL: https://issues.apache.org/jira/browse/DRILL-6053
> Project: Apache Drill
> Issue Type: Improvement
>Reporter: Vlad Rozov
>Assignee: Vlad Rozov
>Priority: Major
> Fix For: 1.14.0
>
>
> When query profiles are written to LocalPersistentStore, the write is
> unnecessary serialized due to read/write lock that was introduced for
> versioned PersistentStore. Only versioned access needs to be protected by
> read/write lock.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (DRILL-6244) Ability to shutdown cross domain Drillbits
Venkata Jyothsna Donapati created DRILL-6244: Summary: Ability to shutdown cross domain Drillbits Key: DRILL-6244 URL: https://issues.apache.org/jira/browse/DRILL-6244 Project: Apache Drill Issue Type: Improvement Reporter: Venkata Jyothsna Donapati Currently Drill Web UI supports shutdown of current Drillbit and doesn't provide an option to shutdown cross domain Drillbits. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (DRILL-6243) Alert box to confirm shutdown of drillbit after clicking shutdown button
[ https://issues.apache.org/jira/browse/DRILL-6243?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Venkata Jyothsna Donapati updated DRILL-6243: - Summary: Alert box to confirm shutdown of drillbit after clicking shutdown button (was: Alert box to confirm shutdown of drillbit) > Alert box to confirm shutdown of drillbit after clicking shutdown button > - > > Key: DRILL-6243 > URL: https://issues.apache.org/jira/browse/DRILL-6243 > Project: Apache Drill > Issue Type: Improvement >Reporter: Venkata Jyothsna Donapati >Priority: Minor > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (DRILL-6243) Alert box to confirm shutdown of drillbit
Venkata Jyothsna Donapati created DRILL-6243: Summary: Alert box to confirm shutdown of drillbit Key: DRILL-6243 URL: https://issues.apache.org/jira/browse/DRILL-6243 Project: Apache Drill Issue Type: Improvement Reporter: Venkata Jyothsna Donapati -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-6053) Avoid excessive locking in LocalPersistentStore
[
https://issues.apache.org/jira/browse/DRILL-6053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16398890#comment-16398890
]
ASF GitHub Bot commented on DRILL-6053:
---
Github user vrozov commented on a diff in the pull request:
https://github.com/apache/drill/pull/1163#discussion_r174534739
--- Diff:
exec/java-exec/src/main/java/org/apache/drill/exec/store/sys/Store.java ---
@@ -0,0 +1,54 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.drill.exec.store.sys;
+
+import java.util.Iterator;
+import java.util.Map;
+
+public interface Store extends AutoCloseable
+{
--- End diff --
Will do. What is the reason it is not handled by checkstyle?
> Avoid excessive locking in LocalPersistentStore
> ---
>
> Key: DRILL-6053
> URL: https://issues.apache.org/jira/browse/DRILL-6053
> Project: Apache Drill
> Issue Type: Improvement
>Reporter: Vlad Rozov
>Assignee: Vlad Rozov
>Priority: Major
> Fix For: 1.14.0
>
>
> When query profiles are written to LocalPersistentStore, the write is
> unnecessary serialized due to read/write lock that was introduced for
> versioned PersistentStore. Only versioned access needs to be protected by
> read/write lock.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-6053) Avoid excessive locking in LocalPersistentStore
[ https://issues.apache.org/jira/browse/DRILL-6053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16398887#comment-16398887 ] ASF GitHub Bot commented on DRILL-6053: --- Github user vrozov commented on a diff in the pull request: https://github.com/apache/drill/pull/1163#discussion_r174533741 --- Diff: pom.xml --- @@ -259,7 +259,13 @@ org.apache.maven.plugins maven-checkstyle-plugin 2.12.1 - + + +com.puppycrawl.tools +checkstyle +5.9 + + --- End diff -- 1. thanks for catching 2. each version of `maven-checkstyle-plugin` is preconfigured with a specific version of checkstyle (2.12.1 uses 5.7) that can be overridden by specifying that dependency explicitly. > Avoid excessive locking in LocalPersistentStore > --- > > Key: DRILL-6053 > URL: https://issues.apache.org/jira/browse/DRILL-6053 > Project: Apache Drill > Issue Type: Improvement >Reporter: Vlad Rozov >Assignee: Vlad Rozov >Priority: Major > Fix For: 1.14.0 > > > When query profiles are written to LocalPersistentStore, the write is > unnecessary serialized due to read/write lock that was introduced for > versioned PersistentStore. Only versioned access needs to be protected by > read/write lock. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-6053) Avoid excessive locking in LocalPersistentStore
[
https://issues.apache.org/jira/browse/DRILL-6053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16398886#comment-16398886
]
ASF GitHub Bot commented on DRILL-6053:
---
Github user arina-ielchiieva commented on a diff in the pull request:
https://github.com/apache/drill/pull/1163#discussion_r174533520
--- Diff: common/src/main/java/org/apache/drill/common/AutoCloseables.java
---
@@ -25,6 +25,11 @@
*/
public class AutoCloseables {
+ public interface Closeable extends AutoCloseable {
--- End diff --
But java has already `Closeable` interface, that's confusing having custom
one with the same name.
> Avoid excessive locking in LocalPersistentStore
> ---
>
> Key: DRILL-6053
> URL: https://issues.apache.org/jira/browse/DRILL-6053
> Project: Apache Drill
> Issue Type: Improvement
>Reporter: Vlad Rozov
>Assignee: Vlad Rozov
>Priority: Major
> Fix For: 1.14.0
>
>
> When query profiles are written to LocalPersistentStore, the write is
> unnecessary serialized due to read/write lock that was introduced for
> versioned PersistentStore. Only versioned access needs to be protected by
> read/write lock.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-6053) Avoid excessive locking in LocalPersistentStore
[
https://issues.apache.org/jira/browse/DRILL-6053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16398871#comment-16398871
]
ASF GitHub Bot commented on DRILL-6053:
---
Github user vrozov commented on a diff in the pull request:
https://github.com/apache/drill/pull/1163#discussion_r174530802
--- Diff: common/src/main/java/org/apache/drill/common/AutoCloseables.java
---
@@ -25,6 +25,11 @@
*/
public class AutoCloseables {
+ public interface Closeable extends AutoCloseable {
--- End diff --
The change allows avoiding `catch` in try-with-resources when `close()`
does not throw an exception.
> Avoid excessive locking in LocalPersistentStore
> ---
>
> Key: DRILL-6053
> URL: https://issues.apache.org/jira/browse/DRILL-6053
> Project: Apache Drill
> Issue Type: Improvement
>Reporter: Vlad Rozov
>Assignee: Vlad Rozov
>Priority: Major
> Fix For: 1.14.0
>
>
> When query profiles are written to LocalPersistentStore, the write is
> unnecessary serialized due to read/write lock that was introduced for
> versioned PersistentStore. Only versioned access needs to be protected by
> read/write lock.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (DRILL-6242) Output format for nested date, time, timestamp values in an object hierarchy
Jiang Wu created DRILL-6242:
---
Summary: Output format for nested date, time, timestamp values in
an object hierarchy
Key: DRILL-6242
URL: https://issues.apache.org/jira/browse/DRILL-6242
Project: Apache Drill
Issue Type: Bug
Components: Execution - Data Types
Affects Versions: 1.12.0
Reporter: Jiang Wu
Some storages (mapr db, mongo db, etc.) have hierarchical objects that contain
nested fields of date, time, timestamp types. When a query returns these
objects, the output format for the nested date, time, timestamp, are showing
the internal object (org.joda.time.DateTime), rather than the logical data
value.
For example. Suppose in MongoDB, we have a single object that looks like this:
{code:java}
> db.test.findOne();
{
"_id" : ObjectId("5aa8487d470dd39a635a12f5"),
"name" : "orange",
"context" : {
"date" : ISODate("2018-03-13T21:52:54.940Z"),
"user" : "jack"
}
}
{code}
Then connect Drill to the above MongoDB storage, and run the following query
within Drill:
{code:java}
> select t.context.`date`, t.context from test t;
++-+
| EXPR$0 | context |
++-+
| 2018-03-13 |
{"date":{"dayOfYear":72,"year":2018,"dayOfMonth":13,"dayOfWeek":2,"era":1,"millisOfDay":78774940,"weekOfWeekyear":11,"weekyear":2018,"monthOfYear":3,"yearOfEra":2018,"yearOfCentury":18,"centuryOfEra":20,"millisOfSecond":940,"secondOfMinute":54,"secondOfDay":78774,"minuteOfHour":52,"minuteOfDay":1312,"hourOfDay":21,"zone":{"fixed":true,"id":"UTC"},"millis":1520977974940,"chronology":{"zone":{"fixed":true,"id":"UTC"}},"afterNow":false,"beforeNow":true,"equalNow":false},"user":"jack"}
|
{code}
We can see that from the above output, when the date field is retrieved as a
top level column, Drill outputs a logical date value. But when the same field
is within an object hierarchy, Drill outputs the internal object used to hold
the date value.
The expected output is the same display for whether the date field is shown as
a top level column or when it is within an object hierarchy:
{code:java}
> select t.context.`date`, t.context from test t;
++-+
| EXPR$0 | context |
++-+
| 2018-03-13 | {"date":"2018-03-13","user":"jack"} |
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-6145) Implement Hive MapR-DB JSON handler.
[
https://issues.apache.org/jira/browse/DRILL-6145?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16398709#comment-16398709
]
ASF GitHub Bot commented on DRILL-6145:
---
Github user vdiravka commented on a diff in the pull request:
https://github.com/apache/drill/pull/1158#discussion_r174489502
--- Diff: distribution/pom.xml ---
@@ -324,6 +324,14 @@
org.apache.hbase
hbase-protocol
+
--- End diff --
I have tried maven 3.5.3 version (I used 3.5.0 by default for couple of
month), but nothing is changed.
I have also tried the newest `maven-assembly-plugin`, but no luck too.
So I have added dependencies for `storage-hive/core` module for `mapr`
profile without removing them from `distribution` module. I believe that it is
better than adding a new module to storage-hive or adding extra assembly
plugins.
But to create a JIra to investigate it in future can be a good point here.
> Implement Hive MapR-DB JSON handler.
> -
>
> Key: DRILL-6145
> URL: https://issues.apache.org/jira/browse/DRILL-6145
> Project: Apache Drill
> Issue Type: Bug
> Components: Storage - Hive, Storage - MapRDB
>Affects Versions: 1.12.0
>Reporter: Vitalii Diravka
>Assignee: Vitalii Diravka
>Priority: Major
> Labels: doc-impacting
> Fix For: 1.14.0
>
>
> Similar to "hive-hbase-storage-handler" to support querying MapR-DB Hive's
> external tables it is necessary to add "hive-maprdb-json-handler".
> Use case:
> # Create a table MapR-DB JSON table:
> {code}
> _> mapr dbshell_
> _maprdb root:> create /tmp/table/json_ (make sure /tmp/table exists)
> {code}
> -- insert data
> {code}
> insert /tmp/table/json --value '\{"_id":"movie002" , "title":"Developers
> on the Edge", "studio":"Command Line Studios"}'
> insert /tmp/table/json --id movie003 --value '\{"title":"The Golden
> Master", "studio":"All-Nighter"}'
> {code}
> # Create a Hive external table:
> {code}
> hive> CREATE EXTERNAL TABLE mapr_db_json_hive_tbl (
> > movie_id string, title string, studio string)
> > STORED BY 'org.apache.hadoop.hive.maprdb.json.MapRDBJsonStorageHandler'
> > TBLPROPERTIES("maprdb.table.name" =
> "/tmp/table/json","maprdb.column.id" = "movie_id");
> {code}
>
> # Use hive schema to query this table via Drill:
> {code}
> 0: jdbc:drill:> select * from hive.mapr_db_json_hive_tbl;
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-6239) Add Build and License Badges to README.md
[ https://issues.apache.org/jira/browse/DRILL-6239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16398691#comment-16398691 ] ASF GitHub Bot commented on DRILL-6239: --- Github user vrozov commented on a diff in the pull request: https://github.com/apache/drill/pull/1165#discussion_r174483759 --- Diff: README.md --- @@ -1,5 +1,8 @@ # Apache Drill +[](https://travis-ci.org/apache/drill) --- End diff -- Please fix the link to point to drill (https://api.travis-ci.org/apache/drill.svg?branch=master). Consider adding maven central badge. > Add Build and License Badges to README.md > - > > Key: DRILL-6239 > URL: https://issues.apache.org/jira/browse/DRILL-6239 > Project: Apache Drill > Issue Type: Improvement >Reporter: Timothy Farkas >Assignee: Timothy Farkas >Priority: Major > Labels: ready-to-commit > Fix For: 1.14.0 > > > Other projects have pretty badges showing the build status and license on the > README.md page. We should have it too! -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-6016) Error reading INT96 created by Apache Spark
[ https://issues.apache.org/jira/browse/DRILL-6016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16398652#comment-16398652 ] ASF GitHub Bot commented on DRILL-6016: --- Github user vdiravka commented on the issue: https://github.com/apache/drill/pull/1166 @parthchandra I have compared meta of files from `TestParquetWriter.testImpalaParquetBinaryAsTimeStamp_DictChange` and the meta from Rahul's dataset and found that test case indeed makes a query from two parquet files: one is dictionary encoded and other isn't. But the dataMode of column is `Optional`, that's why `Nullable` column reader is used. Rahul's dataset contains `required` mode for INT96 column. This is a difference. Therefore other non-nullable column reader is necessary. But I believe we have some mess in names of that column readers. Maybe to make some refactoring would be a good point. What do you think? For example to remove `Dictionary` prefixes from nested classes, but to leave it for top class name. > Error reading INT96 created by Apache Spark > --- > > Key: DRILL-6016 > URL: https://issues.apache.org/jira/browse/DRILL-6016 > Project: Apache Drill > Issue Type: Bug > Environment: Drill 1.11 >Reporter: Rahul Raj >Priority: Major > Fix For: 1.14.0 > > > Hi, > I am getting the error - SYSTEM ERROR : ClassCastException: > org.apache.drill.exec.vector.TimeStampVector cannot be cast to > org.apache.drill.exec.vector.VariableWidthVector while trying to read a spark > INT96 datetime field on Drill 1.11 in spite of setting the property > store.parquet.reader.int96_as_timestamp to true. > I believe this was fixed in drill > 1.10(https://issues.apache.org/jira/browse/DRILL-4373). What could be wrong. > I have attached the dataset at > https://github.com/rajrahul/files/blob/master/result.tar.gz -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (DRILL-6224) The metrics' page has gauges reset to near zero values and does not seem to update
[
https://issues.apache.org/jira/browse/DRILL-6224?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arina Ielchiieva reassigned DRILL-6224:
---
Assignee: Arina Ielchiieva
> The metrics' page has gauges reset to near zero values and does not seem to
> update
> --
>
> Key: DRILL-6224
> URL: https://issues.apache.org/jira/browse/DRILL-6224
> Project: Apache Drill
> Issue Type: Bug
> Components: Web Server
>Affects Versions: 1.12.0
>Reporter: Kunal Khatua
>Assignee: Arina Ielchiieva
>Priority: Major
> Fix For: 1.14.0
>
>
> When viewing http://:8047/metrics#gauges
> The gauges reset to near zero values and does not seem to update.
> Tracing the server calls made, I see the following:
> {code:json}
> {version: "3.0.0", gauges: {G1-Old-Generation.count: {value: 0},
> G1-Old-Generation.time: {value: 0},…},…}
> counters :
> {drill.connections.rpc.control.encrypted: {count: 0},…}
> gauges :
> {G1-Old-Generation.count: {value: 0}, G1-Old-Generation.time: {value: 0},…}
> histograms : {,…}
> meters : {}
> timers : {}
> version : "3.0.0"
> {code}
> This looks incorrect and would explain why the metrics appear to be
> incomplete.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (DRILL-6224) The metrics' page has gauges reset to near zero values and does not seem to update
[
https://issues.apache.org/jira/browse/DRILL-6224?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arina Ielchiieva updated DRILL-6224:
Fix Version/s: 1.14.0
> The metrics' page has gauges reset to near zero values and does not seem to
> update
> --
>
> Key: DRILL-6224
> URL: https://issues.apache.org/jira/browse/DRILL-6224
> Project: Apache Drill
> Issue Type: Bug
> Components: Web Server
>Affects Versions: 1.12.0
>Reporter: Kunal Khatua
>Priority: Major
> Fix For: 1.14.0
>
>
> When viewing http://:8047/metrics#gauges
> The gauges reset to near zero values and does not seem to update.
> Tracing the server calls made, I see the following:
> {code:json}
> {version: "3.0.0", gauges: {G1-Old-Generation.count: {value: 0},
> G1-Old-Generation.time: {value: 0},…},…}
> counters :
> {drill.connections.rpc.control.encrypted: {count: 0},…}
> gauges :
> {G1-Old-Generation.count: {value: 0}, G1-Old-Generation.time: {value: 0},…}
> histograms : {,…}
> meters : {}
> timers : {}
> version : "3.0.0"
> {code}
> This looks incorrect and would explain why the metrics appear to be
> incomplete.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (DRILL-6224) The metrics' page has gauges reset to near zero values and does not seem to update
[
https://issues.apache.org/jira/browse/DRILL-6224?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arina Ielchiieva reassigned DRILL-6224:
---
Assignee: Kunal Khatua (was: Arina Ielchiieva)
> The metrics' page has gauges reset to near zero values and does not seem to
> update
> --
>
> Key: DRILL-6224
> URL: https://issues.apache.org/jira/browse/DRILL-6224
> Project: Apache Drill
> Issue Type: Bug
> Components: Web Server
>Affects Versions: 1.12.0
>Reporter: Kunal Khatua
>Assignee: Kunal Khatua
>Priority: Major
> Fix For: 1.14.0
>
>
> When viewing http://:8047/metrics#gauges
> The gauges reset to near zero values and does not seem to update.
> Tracing the server calls made, I see the following:
> {code:json}
> {version: "3.0.0", gauges: {G1-Old-Generation.count: {value: 0},
> G1-Old-Generation.time: {value: 0},…},…}
> counters :
> {drill.connections.rpc.control.encrypted: {count: 0},…}
> gauges :
> {G1-Old-Generation.count: {value: 0}, G1-Old-Generation.time: {value: 0},…}
> histograms : {,…}
> meters : {}
> timers : {}
> version : "3.0.0"
> {code}
> This looks incorrect and would explain why the metrics appear to be
> incomplete.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (DRILL-6224) The metrics' page has gauges reset to near zero values and does not seem to update
[
https://issues.apache.org/jira/browse/DRILL-6224?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arina Ielchiieva updated DRILL-6224:
Reviewer: Arina Ielchiieva
> The metrics' page has gauges reset to near zero values and does not seem to
> update
> --
>
> Key: DRILL-6224
> URL: https://issues.apache.org/jira/browse/DRILL-6224
> Project: Apache Drill
> Issue Type: Bug
> Components: Web Server
>Affects Versions: 1.12.0
>Reporter: Kunal Khatua
>Priority: Major
> Fix For: 1.14.0
>
>
> When viewing http://:8047/metrics#gauges
> The gauges reset to near zero values and does not seem to update.
> Tracing the server calls made, I see the following:
> {code:json}
> {version: "3.0.0", gauges: {G1-Old-Generation.count: {value: 0},
> G1-Old-Generation.time: {value: 0},…},…}
> counters :
> {drill.connections.rpc.control.encrypted: {count: 0},…}
> gauges :
> {G1-Old-Generation.count: {value: 0}, G1-Old-Generation.time: {value: 0},…}
> histograms : {,…}
> meters : {}
> timers : {}
> version : "3.0.0"
> {code}
> This looks incorrect and would explain why the metrics appear to be
> incomplete.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (DRILL-6005) Fix TestGracefulShutdown tests to skip check for loopback address usage in distributed mode
[
https://issues.apache.org/jira/browse/DRILL-6005?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arina Ielchiieva updated DRILL-6005:
Reviewer: Arina Ielchiieva
> Fix TestGracefulShutdown tests to skip check for loopback address usage in
> distributed mode
> ---
>
> Key: DRILL-6005
> URL: https://issues.apache.org/jira/browse/DRILL-6005
> Project: Apache Drill
> Issue Type: Improvement
>Affects Versions: 1.12.0
>Reporter: Arina Ielchiieva
>Assignee: Venkata Jyothsna Donapati
>Priority: Minor
> Labels: doc-impacting, ready-to-commit
> Fix For: 1.14.0
>
>
> After DRILL-4286 changes some of the newly added unit tests fail with
> {noformat}
> Drillbit is disallowed to bind to loopback address in distributed mode.
> {noformat}
> List of failed tests:
> {noformat}
> Tests in error:
> TestGracefulShutdown.testOnlineEndPoints:96 » IllegalState Cluster fixture
> set...
> TestGracefulShutdown.testStateChange:130 » IllegalState Cluster fixture
> setup ...
> TestGracefulShutdown.testRestApi:167 » IllegalState Cluster fixture setup
> fail...
> TestGracefulShutdown.testRestApiShutdown:207 » IllegalState Cluster fixture
> se...
> {noformat}
> This can be fixed if {{/etc/hosts}} file is edited.
> Source -
> https://stackoverflow.com/questions/40506221/how-to-start-drillbit-locally-in-distributed-mode
> Though these changes are required on production during running unit tests I
> don't think this check should be enforced.
> *For documentation*
> {{drill.exec.allow_loopback_address_binding}} -> Allow drillbit to bind to
> loopback address in distributed mode. Enabled only for testing purposes.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-6224) The metrics' page has gauges reset to near zero values and does not seem to update
[
https://issues.apache.org/jira/browse/DRILL-6224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16398522#comment-16398522
]
ASF GitHub Bot commented on DRILL-6224:
---
Github user arina-ielchiieva commented on a diff in the pull request:
https://github.com/apache/drill/pull/1160#discussion_r174445380
--- Diff: exec/java-exec/src/main/resources/rest/index.ftl ---
@@ -192,6 +199,8 @@
var port = getPortNum();
var timeout;
var size = $("#size").html();
+ reloadMemory();
+ setInterval(reloadMemory, refreshTime);
--- End diff --
In graceful shutdown `setTimeout` is used. Which one is better? Anayway
it's better to be consistent in scheduler usage. Plus we do why need two
schedulers? Can one work for both tasks? Plus if one drillbit is unavailable,
there is no need to check memory.
> The metrics' page has gauges reset to near zero values and does not seem to
> update
> --
>
> Key: DRILL-6224
> URL: https://issues.apache.org/jira/browse/DRILL-6224
> Project: Apache Drill
> Issue Type: Bug
> Components: Web Server
>Affects Versions: 1.12.0
>Reporter: Kunal Khatua
>Priority: Major
>
> When viewing http://:8047/metrics#gauges
> The gauges reset to near zero values and does not seem to update.
> Tracing the server calls made, I see the following:
> {code:json}
> {version: "3.0.0", gauges: {G1-Old-Generation.count: {value: 0},
> G1-Old-Generation.time: {value: 0},…},…}
> counters :
> {drill.connections.rpc.control.encrypted: {count: 0},…}
> gauges :
> {G1-Old-Generation.count: {value: 0}, G1-Old-Generation.time: {value: 0},…}
> histograms : {,…}
> meters : {}
> timers : {}
> version : "3.0.0"
> {code}
> This looks incorrect and would explain why the metrics appear to be
> incomplete.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-6224) The metrics' page has gauges reset to near zero values and does not seem to update
[
https://issues.apache.org/jira/browse/DRILL-6224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16398517#comment-16398517
]
ASF GitHub Bot commented on DRILL-6224:
---
Github user arina-ielchiieva commented on a diff in the pull request:
https://github.com/apache/drill/pull/1160#discussion_r174440528
--- Diff: exec/java-exec/src/main/resources/rest/index.ftl ---
@@ -291,6 +300,46 @@
var url = protocol + "//" + host + requestPath;
return url;
}
+
+ //Iterates through all the nodes for update
+ function reloadMemory () {
+ for (i = 1; i <= size; i++) {
+ var address =
$("#row-"+i).find("#address").contents().get(0).nodeValue.trim();
+ updateMemory(address, i);
+ }
+ }
+
+ //Update memory
+ function updateMemory(drillbit,idx) {
+var result = $.ajax({
+ type: 'GET',
+ url: "http://"+drillbit+":8047/status/metrics";,
+ dataType: "json",
+ complete: function(data) {
+var metrics = data.responseJSON['gauges'];
+var usedHeap = metrics['heap.used'].value;
+var maxHeap = metrics['heap.max'].value;
+var usedDirect = metrics['drill.allocator.root.used'].value;
+var peakDirect = metrics['drill.allocator.root.peak'].value;
+var heapUsage= computeMemUsage(usedHeap,maxHeap);
--- End diff --
Please remove unnecessary spaces here and in other file as well.
> The metrics' page has gauges reset to near zero values and does not seem to
> update
> --
>
> Key: DRILL-6224
> URL: https://issues.apache.org/jira/browse/DRILL-6224
> Project: Apache Drill
> Issue Type: Bug
> Components: Web Server
>Affects Versions: 1.12.0
>Reporter: Kunal Khatua
>Priority: Major
>
> When viewing http://:8047/metrics#gauges
> The gauges reset to near zero values and does not seem to update.
> Tracing the server calls made, I see the following:
> {code:json}
> {version: "3.0.0", gauges: {G1-Old-Generation.count: {value: 0},
> G1-Old-Generation.time: {value: 0},…},…}
> counters :
> {drill.connections.rpc.control.encrypted: {count: 0},…}
> gauges :
> {G1-Old-Generation.count: {value: 0}, G1-Old-Generation.time: {value: 0},…}
> histograms : {,…}
> meters : {}
> timers : {}
> version : "3.0.0"
> {code}
> This looks incorrect and would explain why the metrics appear to be
> incomplete.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-6224) The metrics' page has gauges reset to near zero values and does not seem to update
[
https://issues.apache.org/jira/browse/DRILL-6224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16398521#comment-16398521
]
ASF GitHub Bot commented on DRILL-6224:
---
Github user arina-ielchiieva commented on a diff in the pull request:
https://github.com/apache/drill/pull/1160#discussion_r17297
--- Diff: exec/java-exec/src/main/resources/rest/index.ftl ---
@@ -291,6 +300,46 @@
var url = protocol + "//" + host + requestPath;
return url;
}
+
+ //Iterates through all the nodes for update
+ function reloadMemory () {
+ for (i = 1; i <= size; i++) {
+ var address =
$("#row-"+i).find("#address").contents().get(0).nodeValue.trim();
+ updateMemory(address, i);
+ }
+ }
+
+ //Update memory
+ function updateMemory(drillbit,idx) {
+var result = $.ajax({
+ type: 'GET',
+ url: "http://"+drillbit+":8047/status/metrics";,
+ dataType: "json",
+ complete: function(data) {
+var metrics = data.responseJSON['gauges'];
+var usedHeap = metrics['heap.used'].value;
+var maxHeap = metrics['heap.max'].value;
+var usedDirect = metrics['drill.allocator.root.used'].value;
+var peakDirect = metrics['drill.allocator.root.peak'].value;
+var heapUsage= computeMemUsage(usedHeap,maxHeap);
+var directUsage = computeMemUsage(usedDirect,peakDirect);
+var heapElem =
document.getElementById("row-"+idx).getElementsByClassName("heap")[0];
+heapElem.innerHTML = heapUsage;
+var directElem =
document.getElementById("row-"+idx).getElementsByClassName("direct")[0];
+directElem.innerHTML = directUsage;
+}
+ });
+ }
+
+ //Compute Usage
+ function computeMemUsage(used, max) {
+var percent = 0;
+if ( max > 0)
+ var percent = Math.round((100 * used / max), 2);
+var usage = Math.round((used / 1073741824), 2) + "GB (" +
Math.max(0, percent) + "%)";
--- End diff --
This logic repeats in several files, so can create js function in separate
file and import it.
> The metrics' page has gauges reset to near zero values and does not seem to
> update
> --
>
> Key: DRILL-6224
> URL: https://issues.apache.org/jira/browse/DRILL-6224
> Project: Apache Drill
> Issue Type: Bug
> Components: Web Server
>Affects Versions: 1.12.0
>Reporter: Kunal Khatua
>Priority: Major
>
> When viewing http://:8047/metrics#gauges
> The gauges reset to near zero values and does not seem to update.
> Tracing the server calls made, I see the following:
> {code:json}
> {version: "3.0.0", gauges: {G1-Old-Generation.count: {value: 0},
> G1-Old-Generation.time: {value: 0},…},…}
> counters :
> {drill.connections.rpc.control.encrypted: {count: 0},…}
> gauges :
> {G1-Old-Generation.count: {value: 0}, G1-Old-Generation.time: {value: 0},…}
> histograms : {,…}
> meters : {}
> timers : {}
> version : "3.0.0"
> {code}
> This looks incorrect and would explain why the metrics appear to be
> incomplete.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-6224) The metrics' page has gauges reset to near zero values and does not seem to update
[
https://issues.apache.org/jira/browse/DRILL-6224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16398518#comment-16398518
]
ASF GitHub Bot commented on DRILL-6224:
---
Github user arina-ielchiieva commented on a diff in the pull request:
https://github.com/apache/drill/pull/1160#discussion_r17069
--- Diff: exec/java-exec/src/main/resources/rest/metrics/metrics.ftl ---
@@ -109,18 +114,23 @@
};
function updateBars(gauges) {
- $.each(["heap","non-heap","total"], function(i, key) {
+ $.each(["heap","non-heap","total","drill.allocator.root"],
function(i, key) {
var used= gauges[key + ".used"].value;
-var max = gauges[key + ".max"].value;
-var usage = round((used / 1073741824), 2) + "GB";
-var percent = round((used / max), 2);
-
-var styleVal = "width: " + percent + "%;"
-$("#" + key + "Usage").attr({
+var max;
+if (key.startsWith("drill.allocator")) {
--- End diff --
Using `key.startsWith("drill.allocator")` several times in this method is
not very elegant. Besides it he fact it is repeated action, I do not like such
checks in general. I guess that can be re-written.
> The metrics' page has gauges reset to near zero values and does not seem to
> update
> --
>
> Key: DRILL-6224
> URL: https://issues.apache.org/jira/browse/DRILL-6224
> Project: Apache Drill
> Issue Type: Bug
> Components: Web Server
>Affects Versions: 1.12.0
>Reporter: Kunal Khatua
>Priority: Major
>
> When viewing http://:8047/metrics#gauges
> The gauges reset to near zero values and does not seem to update.
> Tracing the server calls made, I see the following:
> {code:json}
> {version: "3.0.0", gauges: {G1-Old-Generation.count: {value: 0},
> G1-Old-Generation.time: {value: 0},…},…}
> counters :
> {drill.connections.rpc.control.encrypted: {count: 0},…}
> gauges :
> {G1-Old-Generation.count: {value: 0}, G1-Old-Generation.time: {value: 0},…}
> histograms : {,…}
> meters : {}
> timers : {}
> version : "3.0.0"
> {code}
> This looks incorrect and would explain why the metrics appear to be
> incomplete.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-6224) The metrics' page has gauges reset to near zero values and does not seem to update
[
https://issues.apache.org/jira/browse/DRILL-6224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16398520#comment-16398520
]
ASF GitHub Bot commented on DRILL-6224:
---
Github user arina-ielchiieva commented on a diff in the pull request:
https://github.com/apache/drill/pull/1160#discussion_r174440404
--- Diff: exec/java-exec/src/main/resources/rest/index.ftl ---
@@ -291,6 +300,46 @@
var url = protocol + "//" + host + requestPath;
return url;
}
+
+ //Iterates through all the nodes for update
+ function reloadMemory () {
+ for (i = 1; i <= size; i++) {
+ var address =
$("#row-"+i).find("#address").contents().get(0).nodeValue.trim();
+ updateMemory(address, i);
+ }
+ }
+
+ //Update memory
+ function updateMemory(drillbit,idx) {
+var result = $.ajax({
+ type: 'GET',
+ url: "http://"+drillbit+":8047/status/metrics";,
--- End diff --
You can not hard-code port, it can be changed using property. See how
graceful shutdown is checking drillbits statuses.
> The metrics' page has gauges reset to near zero values and does not seem to
> update
> --
>
> Key: DRILL-6224
> URL: https://issues.apache.org/jira/browse/DRILL-6224
> Project: Apache Drill
> Issue Type: Bug
> Components: Web Server
>Affects Versions: 1.12.0
>Reporter: Kunal Khatua
>Priority: Major
>
> When viewing http://:8047/metrics#gauges
> The gauges reset to near zero values and does not seem to update.
> Tracing the server calls made, I see the following:
> {code:json}
> {version: "3.0.0", gauges: {G1-Old-Generation.count: {value: 0},
> G1-Old-Generation.time: {value: 0},…},…}
> counters :
> {drill.connections.rpc.control.encrypted: {count: 0},…}
> gauges :
> {G1-Old-Generation.count: {value: 0}, G1-Old-Generation.time: {value: 0},…}
> histograms : {,…}
> meters : {}
> timers : {}
> version : "3.0.0"
> {code}
> This looks incorrect and would explain why the metrics appear to be
> incomplete.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-6224) The metrics' page has gauges reset to near zero values and does not seem to update
[
https://issues.apache.org/jira/browse/DRILL-6224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16398519#comment-16398519
]
ASF GitHub Bot commented on DRILL-6224:
---
Github user arina-ielchiieva commented on a diff in the pull request:
https://github.com/apache/drill/pull/1160#discussion_r174440089
--- Diff: exec/java-exec/src/main/resources/rest/index.ftl ---
@@ -53,11 +53,16 @@
#
Address
+ Heap Usage
+ Direct Usage
User Port
Control Port
Data Port
Version
Status
+ <#if model.shouldShowAdminInfo() || !model.isAuthEnabled() >
--- End diff --
On master we have different line `<#if (model.shouldShowAdminInfo() ||
!model.isAuthEnabled()) && drillbit.isCurrent() >`
> The metrics' page has gauges reset to near zero values and does not seem to
> update
> --
>
> Key: DRILL-6224
> URL: https://issues.apache.org/jira/browse/DRILL-6224
> Project: Apache Drill
> Issue Type: Bug
> Components: Web Server
>Affects Versions: 1.12.0
>Reporter: Kunal Khatua
>Priority: Major
>
> When viewing http://:8047/metrics#gauges
> The gauges reset to near zero values and does not seem to update.
> Tracing the server calls made, I see the following:
> {code:json}
> {version: "3.0.0", gauges: {G1-Old-Generation.count: {value: 0},
> G1-Old-Generation.time: {value: 0},…},…}
> counters :
> {drill.connections.rpc.control.encrypted: {count: 0},…}
> gauges :
> {G1-Old-Generation.count: {value: 0}, G1-Old-Generation.time: {value: 0},…}
> histograms : {,…}
> meters : {}
> timers : {}
> version : "3.0.0"
> {code}
> This looks incorrect and would explain why the metrics appear to be
> incomplete.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-6016) Error reading INT96 created by Apache Spark
[ https://issues.apache.org/jira/browse/DRILL-6016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16398508#comment-16398508 ] ASF GitHub Bot commented on DRILL-6016: --- Github user rajrahul commented on the issue: https://github.com/apache/drill/pull/1166 @parthchandra I will create a unit test with few time stamp fields. > Error reading INT96 created by Apache Spark > --- > > Key: DRILL-6016 > URL: https://issues.apache.org/jira/browse/DRILL-6016 > Project: Apache Drill > Issue Type: Bug > Environment: Drill 1.11 >Reporter: Rahul Raj >Priority: Major > Fix For: 1.14.0 > > > Hi, > I am getting the error - SYSTEM ERROR : ClassCastException: > org.apache.drill.exec.vector.TimeStampVector cannot be cast to > org.apache.drill.exec.vector.VariableWidthVector while trying to read a spark > INT96 datetime field on Drill 1.11 in spite of setting the property > store.parquet.reader.int96_as_timestamp to true. > I believe this was fixed in drill > 1.10(https://issues.apache.org/jira/browse/DRILL-4373). What could be wrong. > I have attached the dataset at > https://github.com/rajrahul/files/blob/master/result.tar.gz -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-6053) Avoid excessive locking in LocalPersistentStore
[
https://issues.apache.org/jira/browse/DRILL-6053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16398468#comment-16398468
]
ASF GitHub Bot commented on DRILL-6053:
---
Github user arina-ielchiieva commented on a diff in the pull request:
https://github.com/apache/drill/pull/1163#discussion_r174431372
--- Diff:
exec/java-exec/src/main/java/org/apache/drill/exec/store/sys/Store.java ---
@@ -0,0 +1,54 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.drill.exec.store.sys;
+
+import java.util.Iterator;
+import java.util.Map;
+
+public interface Store extends AutoCloseable
+{
--- End diff --
Please move to upper line.
> Avoid excessive locking in LocalPersistentStore
> ---
>
> Key: DRILL-6053
> URL: https://issues.apache.org/jira/browse/DRILL-6053
> Project: Apache Drill
> Issue Type: Improvement
>Reporter: Vlad Rozov
>Assignee: Vlad Rozov
>Priority: Major
> Fix For: 1.14.0
>
>
> When query profiles are written to LocalPersistentStore, the write is
> unnecessary serialized due to read/write lock that was introduced for
> versioned PersistentStore. Only versioned access needs to be protected by
> read/write lock.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-6053) Avoid excessive locking in LocalPersistentStore
[
https://issues.apache.org/jira/browse/DRILL-6053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16398469#comment-16398469
]
ASF GitHub Bot commented on DRILL-6053:
---
Github user arina-ielchiieva commented on a diff in the pull request:
https://github.com/apache/drill/pull/1163#discussion_r174432531
--- Diff:
exec/java-exec/src/main/java/org/apache/drill/exec/store/sys/store/VersionedDelegatingStore.java
---
@@ -0,0 +1,118 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.drill.exec.store.sys.store;
+
+import java.util.Iterator;
+import java.util.Map;
+import java.util.concurrent.locks.ReadWriteLock;
+import java.util.concurrent.locks.ReentrantReadWriteLock;
+
+import org.apache.drill.common.AutoCloseables.Closeable;
+import org.apache.drill.common.concurrent.AutoCloseableLock;
+import org.apache.drill.exec.exception.VersionMismatchException;
+import org.apache.drill.exec.store.sys.PersistentStore;
+import org.apache.drill.exec.store.sys.PersistentStoreMode;
+import org.apache.drill.exec.store.sys.VersionedPersistentStore;
+
+public class VersionedDelegatingStore implements
VersionedPersistentStore
+{
+ private final PersistentStore store;
+ private final ReadWriteLock readWriteLock;
+ private final AutoCloseableLock readLock;
+ private final AutoCloseableLock writeLock;
+ private int version;
+
+ public VersionedDelegatingStore(PersistentStore store) {
+this.store = store;
+readWriteLock = new ReentrantReadWriteLock();
+readLock = new AutoCloseableLock(readWriteLock.readLock());
+writeLock = new AutoCloseableLock(readWriteLock.writeLock());
+version = -1;
+ }
+
+ @Override
+ public PersistentStoreMode getMode()
+ {
--- End diff --
Please move to the upper line.
> Avoid excessive locking in LocalPersistentStore
> ---
>
> Key: DRILL-6053
> URL: https://issues.apache.org/jira/browse/DRILL-6053
> Project: Apache Drill
> Issue Type: Improvement
>Reporter: Vlad Rozov
>Assignee: Vlad Rozov
>Priority: Major
> Fix For: 1.14.0
>
>
> When query profiles are written to LocalPersistentStore, the write is
> unnecessary serialized due to read/write lock that was introduced for
> versioned PersistentStore. Only versioned access needs to be protected by
> read/write lock.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-6053) Avoid excessive locking in LocalPersistentStore
[
https://issues.apache.org/jira/browse/DRILL-6053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16398467#comment-16398467
]
ASF GitHub Bot commented on DRILL-6053:
---
Github user arina-ielchiieva commented on a diff in the pull request:
https://github.com/apache/drill/pull/1163#discussion_r174431507
--- Diff:
exec/java-exec/src/main/java/org/apache/drill/exec/store/sys/VersionedPersistentStore.java
---
@@ -0,0 +1,54 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.drill.exec.store.sys;
+
+import org.apache.drill.exec.store.sys.store.DataChangeVersion;
+
+public interface VersionedPersistentStore extends Store
+{
--- End diff --
Please move to upper line.
> Avoid excessive locking in LocalPersistentStore
> ---
>
> Key: DRILL-6053
> URL: https://issues.apache.org/jira/browse/DRILL-6053
> Project: Apache Drill
> Issue Type: Improvement
>Reporter: Vlad Rozov
>Assignee: Vlad Rozov
>Priority: Major
> Fix For: 1.14.0
>
>
> When query profiles are written to LocalPersistentStore, the write is
> unnecessary serialized due to read/write lock that was introduced for
> versioned PersistentStore. Only versioned access needs to be protected by
> read/write lock.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-6053) Avoid excessive locking in LocalPersistentStore
[
https://issues.apache.org/jira/browse/DRILL-6053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16398472#comment-16398472
]
ASF GitHub Bot commented on DRILL-6053:
---
Github user arina-ielchiieva commented on a diff in the pull request:
https://github.com/apache/drill/pull/1163#discussion_r174433799
--- Diff:
exec/java-exec/src/main/java/org/apache/drill/exec/store/sys/PersistentStoreProvider.java
---
@@ -33,7 +34,9 @@
* @param store value type
*/
PersistentStore getOrCreateStore(PersistentStoreConfig config)
throws StoreException;
-
+ default VersionedPersistentStore
getOrCreateVersionedStore(PersistentStoreConfig config) throws
StoreException {
--- End diff --
Please add javadoc here and in newly created classes as well.
> Avoid excessive locking in LocalPersistentStore
> ---
>
> Key: DRILL-6053
> URL: https://issues.apache.org/jira/browse/DRILL-6053
> Project: Apache Drill
> Issue Type: Improvement
>Reporter: Vlad Rozov
>Assignee: Vlad Rozov
>Priority: Major
> Fix For: 1.14.0
>
>
> When query profiles are written to LocalPersistentStore, the write is
> unnecessary serialized due to read/write lock that was introduced for
> versioned PersistentStore. Only versioned access needs to be protected by
> read/write lock.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-6053) Avoid excessive locking in LocalPersistentStore
[ https://issues.apache.org/jira/browse/DRILL-6053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16398470#comment-16398470 ] ASF GitHub Bot commented on DRILL-6053: --- Github user arina-ielchiieva commented on a diff in the pull request: https://github.com/apache/drill/pull/1163#discussion_r174433987 --- Diff: pom.xml --- @@ -259,7 +259,13 @@ org.apache.maven.plugins maven-checkstyle-plugin 2.12.1 - + + +com.puppycrawl.tools +checkstyle +5.9 + + --- End diff -- 1. Please `configuration` move to the next line. 2. You indicating that we upgrade dependency but in fact we added new? > Avoid excessive locking in LocalPersistentStore > --- > > Key: DRILL-6053 > URL: https://issues.apache.org/jira/browse/DRILL-6053 > Project: Apache Drill > Issue Type: Improvement >Reporter: Vlad Rozov >Assignee: Vlad Rozov >Priority: Major > Fix For: 1.14.0 > > > When query profiles are written to LocalPersistentStore, the write is > unnecessary serialized due to read/write lock that was introduced for > versioned PersistentStore. Only versioned access needs to be protected by > read/write lock. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-6053) Avoid excessive locking in LocalPersistentStore
[
https://issues.apache.org/jira/browse/DRILL-6053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16398471#comment-16398471
]
ASF GitHub Bot commented on DRILL-6053:
---
Github user arina-ielchiieva commented on a diff in the pull request:
https://github.com/apache/drill/pull/1163#discussion_r174433648
--- Diff: common/src/main/java/org/apache/drill/common/AutoCloseables.java
---
@@ -25,6 +25,11 @@
*/
public class AutoCloseables {
+ public interface Closeable extends AutoCloseable {
--- End diff --
Not sure about benefit of this change... Could you please explain.
> Avoid excessive locking in LocalPersistentStore
> ---
>
> Key: DRILL-6053
> URL: https://issues.apache.org/jira/browse/DRILL-6053
> Project: Apache Drill
> Issue Type: Improvement
>Reporter: Vlad Rozov
>Assignee: Vlad Rozov
>Priority: Major
> Fix For: 1.14.0
>
>
> When query profiles are written to LocalPersistentStore, the write is
> unnecessary serialized due to read/write lock that was introduced for
> versioned PersistentStore. Only versioned access needs to be protected by
> read/write lock.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-6016) Error reading INT96 created by Apache Spark
[ https://issues.apache.org/jira/browse/DRILL-6016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16398446#comment-16398446 ] ASF GitHub Bot commented on DRILL-6016: --- Github user parthchandra commented on the issue: https://github.com/apache/drill/pull/1166 @rajrahul this link is good. As expected, the int96 column is dictionary encoded. Is it possible for you to extract just a couple of records from this file and then use that for a unit test? see [TestParquetWriter.testImpalaParquetBinaryAsTimeStamp_DictChange](https://github.com/apache/drill/blob/master/exec/java-exec/src/test/java/org/apache/drill/exec/physical/impl/writer/TestParquetWriter.java#L784) @vdiravka TestParquetWriter.testImpalaParquetBinaryAsTimeStamp_DictChange also uses an int96 that is dictionary encoded. Any idea whether (and why) it might be going thru a different code path? > Error reading INT96 created by Apache Spark > --- > > Key: DRILL-6016 > URL: https://issues.apache.org/jira/browse/DRILL-6016 > Project: Apache Drill > Issue Type: Bug > Environment: Drill 1.11 >Reporter: Rahul Raj >Priority: Major > Fix For: 1.14.0 > > > Hi, > I am getting the error - SYSTEM ERROR : ClassCastException: > org.apache.drill.exec.vector.TimeStampVector cannot be cast to > org.apache.drill.exec.vector.VariableWidthVector while trying to read a spark > INT96 datetime field on Drill 1.11 in spite of setting the property > store.parquet.reader.int96_as_timestamp to true. > I believe this was fixed in drill > 1.10(https://issues.apache.org/jira/browse/DRILL-4373). What could be wrong. > I have attached the dataset at > https://github.com/rajrahul/files/blob/master/result.tar.gz -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-6199) Filter push down doesn't work with more than one nested subqueries
[
https://issues.apache.org/jira/browse/DRILL-6199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16398421#comment-16398421
]
ASF GitHub Bot commented on DRILL-6199:
---
Github user arina-ielchiieva commented on the issue:
https://github.com/apache/drill/pull/1152
@chunhui-shi yes, you are correct, we are trying to find item star columns
and push them into scan
This case if F is optional then we don't have filter and there will no
filter push down, only project push down can happen in this case. Such case is
covered in `testProjectIntoScanWithSeveralNestedStarSubQueries`.
Regarding additional columns in the outermost 'select' , could you please
give an example?
> Filter push down doesn't work with more than one nested subqueries
> --
>
> Key: DRILL-6199
> URL: https://issues.apache.org/jira/browse/DRILL-6199
> Project: Apache Drill
> Issue Type: Bug
>Affects Versions: 1.13.0
>Reporter: Anton Gozhiy
>Assignee: Arina Ielchiieva
>Priority: Major
> Fix For: 1.14.0
>
> Attachments: DRILL_6118_data_source.csv
>
>
> *Data set:*
> The data is generated used the attached file: *DRILL_6118_data_source.csv*
> Data gen commands:
> {code:sql}
> create table dfs.tmp.`DRILL_6118_parquet_partitioned_by_folders/d1` (c1, c2,
> c3, c4, c5) as select cast(columns[0] as int) c1, columns[1] c2, columns[2]
> c3, columns[3] c4, columns[4] c5 from dfs.tmp.`DRILL_6118_data_source.csv`
> where columns[0] in (1, 3);
> create table dfs.tmp.`DRILL_6118_parquet_partitioned_by_folders/d2` (c1, c2,
> c3, c4, c5) as select cast(columns[0] as int) c1, columns[1] c2, columns[2]
> c3, columns[3] c4, columns[4] c5 from dfs.tmp.`DRILL_6118_data_source.csv`
> where columns[0]=2;
> create table dfs.tmp.`DRILL_6118_parquet_partitioned_by_folders/d3` (c1, c2,
> c3, c4, c5) as select cast(columns[0] as int) c1, columns[1] c2, columns[2]
> c3, columns[3] c4, columns[4] c5 from dfs.tmp.`DRILL_6118_data_source.csv`
> where columns[0]>3;
> {code}
> *Steps:*
> # Execute the following query:
> {code:sql}
> explain plan for select * from (select * from (select * from
> dfs.tmp.`DRILL_6118_parquet_partitioned_by_folders`)) where c1<3
> {code}
> *Expected result:*
> numFiles=2, numRowGroups=2, only files from the folders d1 and d2 should be
> scanned.
> *Actual result:*
> Filter push down doesn't work:
> numFiles=3, numRowGroups=3, scanning from all files
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-6241) Saffron properties config has the excessive permissions
[
https://issues.apache.org/jira/browse/DRILL-6241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16398418#comment-16398418
]
ASF GitHub Bot commented on DRILL-6241:
---
Github user arina-ielchiieva commented on the issue:
https://github.com/apache/drill/pull/1167
+1
> Saffron properties config has the excessive permissions
>
>
> Key: DRILL-6241
> URL: https://issues.apache.org/jira/browse/DRILL-6241
> Project: Apache Drill
> Issue Type: Bug
>Affects Versions: 1.13.0
>Reporter: Volodymyr Tkach
>Assignee: Volodymyr Tkach
>Priority: Major
> Labels: ready-to-commit
>
> Current Drill config permissions:
> {noformat}
> ls -al ./drill-1.13.0/conf/saffron.properties
> -rw-rw-r-- 1 mapr mapr 1191 Mar 12 09:36 saffron.properties
> {noformat}
> *Expected result:*
> It should have permissions 0640
> *Actual result:*
> It has the permissions 0664
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (DRILL-6241) Saffron properties config has the excessive permissions
[
https://issues.apache.org/jira/browse/DRILL-6241?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arina Ielchiieva updated DRILL-6241:
Reviewer: Arina Ielchiieva
> Saffron properties config has the excessive permissions
>
>
> Key: DRILL-6241
> URL: https://issues.apache.org/jira/browse/DRILL-6241
> Project: Apache Drill
> Issue Type: Bug
>Affects Versions: 1.13.0
>Reporter: Volodymyr Tkach
>Assignee: Volodymyr Tkach
>Priority: Major
> Labels: ready-to-commit
>
> Current Drill config permissions:
> {noformat}
> ls -al ./drill-1.13.0/conf/saffron.properties
> -rw-rw-r-- 1 mapr mapr 1191 Mar 12 09:36 saffron.properties
> {noformat}
> *Expected result:*
> It should have permissions 0640
> *Actual result:*
> It has the permissions 0664
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (DRILL-6241) Saffron properties config has the excessive permissions
[
https://issues.apache.org/jira/browse/DRILL-6241?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arina Ielchiieva updated DRILL-6241:
Labels: ready-to-commit (was: )
> Saffron properties config has the excessive permissions
>
>
> Key: DRILL-6241
> URL: https://issues.apache.org/jira/browse/DRILL-6241
> Project: Apache Drill
> Issue Type: Bug
>Affects Versions: 1.13.0
>Reporter: Volodymyr Tkach
>Assignee: Volodymyr Tkach
>Priority: Major
> Labels: ready-to-commit
>
> Current Drill config permissions:
> {noformat}
> ls -al ./drill-1.13.0/conf/saffron.properties
> -rw-rw-r-- 1 mapr mapr 1191 Mar 12 09:36 saffron.properties
> {noformat}
> *Expected result:*
> It should have permissions 0640
> *Actual result:*
> It has the permissions 0664
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (DRILL-6241) Saffron properties config has the excessive permissions
[
https://issues.apache.org/jira/browse/DRILL-6241?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arina Ielchiieva updated DRILL-6241:
Affects Version/s: 1.13.0
> Saffron properties config has the excessive permissions
>
>
> Key: DRILL-6241
> URL: https://issues.apache.org/jira/browse/DRILL-6241
> Project: Apache Drill
> Issue Type: Bug
>Affects Versions: 1.13.0
>Reporter: Volodymyr Tkach
>Assignee: Volodymyr Tkach
>Priority: Major
> Labels: ready-to-commit
>
> Current Drill config permissions:
> {noformat}
> ls -al ./drill-1.13.0/conf/saffron.properties
> -rw-rw-r-- 1 mapr mapr 1191 Mar 12 09:36 saffron.properties
> {noformat}
> *Expected result:*
> It should have permissions 0640
> *Actual result:*
> It has the permissions 0664
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-6241) Saffron properties config has the excessive permissions
[
https://issues.apache.org/jira/browse/DRILL-6241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16398410#comment-16398410
]
ASF GitHub Bot commented on DRILL-6241:
---
GitHub user vladimirtkach opened a pull request:
https://github.com/apache/drill/pull/1167
DRILL-6241: Saffron properties config has the excessive permissions
changed saffron.properties permission to 640
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/vladimirtkach/drill DRILL-6241
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/drill/pull/1167.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1167
commit 7ac1359495dcbeb258b1e8ae8a11912a436edce0
Author: Vladimir Tkach
Date: 2018-03-14T11:21:43Z
DRILL-6241: Saffron properties config has the excessive permissions
changed saffron.properties permission to 640
> Saffron properties config has the excessive permissions
>
>
> Key: DRILL-6241
> URL: https://issues.apache.org/jira/browse/DRILL-6241
> Project: Apache Drill
> Issue Type: Bug
>Reporter: Volodymyr Tkach
>Assignee: Volodymyr Tkach
>Priority: Major
>
> Current Drill config permissions:
> {noformat}
> ls -al ./drill-1.13.0/conf/saffron.properties
> -rw-rw-r-- 1 mapr mapr 1191 Mar 12 09:36 saffron.properties
> {noformat}
> *Expected result:*
> It should have permissions 0640
> *Actual result:*
> It has the permissions 0664
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (DRILL-6241) Saffron properties config has the excessive permissions
Volodymyr Tkach created DRILL-6241:
--
Summary: Saffron properties config has the excessive permissions
Key: DRILL-6241
URL: https://issues.apache.org/jira/browse/DRILL-6241
Project: Apache Drill
Issue Type: Bug
Reporter: Volodymyr Tkach
Assignee: Volodymyr Tkach
Current Drill config permissions:
{noformat}
ls -al ./drill-1.13.0/conf/saffron.properties
-rw-rw-r-- 1 mapr mapr 1191 Mar 12 09:36 saffron.properties
{noformat}
*Expected result:*
It should have permissions 0640
*Actual result:*
It has the permissions 0664
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-6016) Error reading INT96 created by Apache Spark
[ https://issues.apache.org/jira/browse/DRILL-6016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16398379#comment-16398379 ] ASF GitHub Bot commented on DRILL-6016: --- Github user rajrahul commented on the issue: https://github.com/apache/drill/pull/1166 @parthchandra please use the link https://github.com/rajrahul/files/raw/master/result.tar.gz The files are present inside result/parquet/latest. > Error reading INT96 created by Apache Spark > --- > > Key: DRILL-6016 > URL: https://issues.apache.org/jira/browse/DRILL-6016 > Project: Apache Drill > Issue Type: Bug > Environment: Drill 1.11 >Reporter: Rahul Raj >Priority: Major > Fix For: 1.14.0 > > > Hi, > I am getting the error - SYSTEM ERROR : ClassCastException: > org.apache.drill.exec.vector.TimeStampVector cannot be cast to > org.apache.drill.exec.vector.VariableWidthVector while trying to read a spark > INT96 datetime field on Drill 1.11 in spite of setting the property > store.parquet.reader.int96_as_timestamp to true. > I believe this was fixed in drill > 1.10(https://issues.apache.org/jira/browse/DRILL-4373). What could be wrong. > I have attached the dataset at > https://github.com/rajrahul/files/blob/master/result.tar.gz -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (DRILL-6005) Fix TestGracefulShutdown tests to skip check for loopback address usage in distributed mode
[
https://issues.apache.org/jira/browse/DRILL-6005?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arina Ielchiieva updated DRILL-6005:
Labels: doc-impacting ready-to-commit (was: doc-impacting)
> Fix TestGracefulShutdown tests to skip check for loopback address usage in
> distributed mode
> ---
>
> Key: DRILL-6005
> URL: https://issues.apache.org/jira/browse/DRILL-6005
> Project: Apache Drill
> Issue Type: Improvement
>Affects Versions: 1.12.0
>Reporter: Arina Ielchiieva
>Assignee: Venkata Jyothsna Donapati
>Priority: Minor
> Labels: doc-impacting, ready-to-commit
> Fix For: 1.14.0
>
>
> After DRILL-4286 changes some of the newly added unit tests fail with
> {noformat}
> Drillbit is disallowed to bind to loopback address in distributed mode.
> {noformat}
> List of failed tests:
> {noformat}
> Tests in error:
> TestGracefulShutdown.testOnlineEndPoints:96 » IllegalState Cluster fixture
> set...
> TestGracefulShutdown.testStateChange:130 » IllegalState Cluster fixture
> setup ...
> TestGracefulShutdown.testRestApi:167 » IllegalState Cluster fixture setup
> fail...
> TestGracefulShutdown.testRestApiShutdown:207 » IllegalState Cluster fixture
> se...
> {noformat}
> This can be fixed if {{/etc/hosts}} file is edited.
> Source -
> https://stackoverflow.com/questions/40506221/how-to-start-drillbit-locally-in-distributed-mode
> Though these changes are required on production during running unit tests I
> don't think this check should be enforced.
> *For documentation*
> {{drill.exec.allow_loopback_address_binding}} -> Allow drillbit to bind to
> loopback address in distributed mode. Enabled only for testing purposes.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (DRILL-6005) Fix TestGracefulShutdown tests to skip check for loopback address usage in distributed mode
[
https://issues.apache.org/jira/browse/DRILL-6005?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arina Ielchiieva updated DRILL-6005:
Description:
After DRILL-4286 changes some of the newly added unit tests fail with
{noformat}
Drillbit is disallowed to bind to loopback address in distributed mode.
{noformat}
List of failed tests:
{noformat}
Tests in error:
TestGracefulShutdown.testOnlineEndPoints:96 » IllegalState Cluster fixture
set...
TestGracefulShutdown.testStateChange:130 » IllegalState Cluster fixture setup
...
TestGracefulShutdown.testRestApi:167 » IllegalState Cluster fixture setup
fail...
TestGracefulShutdown.testRestApiShutdown:207 » IllegalState Cluster fixture
se...
{noformat}
This can be fixed if {{/etc/hosts}} file is edited.
Source -
https://stackoverflow.com/questions/40506221/how-to-start-drillbit-locally-in-distributed-mode
Though these changes are required on production during running unit tests I
don't think this check should be enforced.
*For documentation*
{{drill.exec.allow_loopback_address_binding}} -> Allow drillbit to bind to
loopback address in distributed mode. Enabled only for testing purposes.
was:
After DRILL-4286 changes some of the newly added unit tests fail with
{noformat}
Drillbit is disallowed to bind to loopback address in distributed mode.
{noformat}
List of failed tests:
{noformat}
Tests in error:
TestGracefulShutdown.testOnlineEndPoints:96 » IllegalState Cluster fixture
set...
TestGracefulShutdown.testStateChange:130 » IllegalState Cluster fixture setup
...
TestGracefulShutdown.testRestApi:167 » IllegalState Cluster fixture setup
fail...
TestGracefulShutdown.testRestApiShutdown:207 » IllegalState Cluster fixture
se...
{noformat}
This can be fixed if {{/etc/hosts}} file is edited.
Source -
https://stackoverflow.com/questions/40506221/how-to-start-drillbit-locally-in-distributed-mode
Though these changes are required on production during running unit tests I
don't think this check should be enforced.
> Fix TestGracefulShutdown tests to skip check for loopback address usage in
> distributed mode
> ---
>
> Key: DRILL-6005
> URL: https://issues.apache.org/jira/browse/DRILL-6005
> Project: Apache Drill
> Issue Type: Improvement
>Affects Versions: 1.12.0
>Reporter: Arina Ielchiieva
>Assignee: Venkata Jyothsna Donapati
>Priority: Minor
> Labels: doc-impacting, ready-to-commit
> Fix For: 1.14.0
>
>
> After DRILL-4286 changes some of the newly added unit tests fail with
> {noformat}
> Drillbit is disallowed to bind to loopback address in distributed mode.
> {noformat}
> List of failed tests:
> {noformat}
> Tests in error:
> TestGracefulShutdown.testOnlineEndPoints:96 » IllegalState Cluster fixture
> set...
> TestGracefulShutdown.testStateChange:130 » IllegalState Cluster fixture
> setup ...
> TestGracefulShutdown.testRestApi:167 » IllegalState Cluster fixture setup
> fail...
> TestGracefulShutdown.testRestApiShutdown:207 » IllegalState Cluster fixture
> se...
> {noformat}
> This can be fixed if {{/etc/hosts}} file is edited.
> Source -
> https://stackoverflow.com/questions/40506221/how-to-start-drillbit-locally-in-distributed-mode
> Though these changes are required on production during running unit tests I
> don't think this check should be enforced.
> *For documentation*
> {{drill.exec.allow_loopback_address_binding}} -> Allow drillbit to bind to
> loopback address in distributed mode. Enabled only for testing purposes.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-6005) Fix TestGracefulShutdown tests to skip check for loopback address usage in distributed mode
[
https://issues.apache.org/jira/browse/DRILL-6005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16398369#comment-16398369
]
ASF GitHub Bot commented on DRILL-6005:
---
Github user arina-ielchiieva commented on the issue:
https://github.com/apache/drill/pull/1162
+1, thanks for making the changes.
> Fix TestGracefulShutdown tests to skip check for loopback address usage in
> distributed mode
> ---
>
> Key: DRILL-6005
> URL: https://issues.apache.org/jira/browse/DRILL-6005
> Project: Apache Drill
> Issue Type: Improvement
>Affects Versions: 1.12.0
>Reporter: Arina Ielchiieva
>Assignee: Venkata Jyothsna Donapati
>Priority: Minor
> Labels: doc-impacting, ready-to-commit
> Fix For: 1.14.0
>
>
> After DRILL-4286 changes some of the newly added unit tests fail with
> {noformat}
> Drillbit is disallowed to bind to loopback address in distributed mode.
> {noformat}
> List of failed tests:
> {noformat}
> Tests in error:
> TestGracefulShutdown.testOnlineEndPoints:96 » IllegalState Cluster fixture
> set...
> TestGracefulShutdown.testStateChange:130 » IllegalState Cluster fixture
> setup ...
> TestGracefulShutdown.testRestApi:167 » IllegalState Cluster fixture setup
> fail...
> TestGracefulShutdown.testRestApiShutdown:207 » IllegalState Cluster fixture
> se...
> {noformat}
> This can be fixed if {{/etc/hosts}} file is edited.
> Source -
> https://stackoverflow.com/questions/40506221/how-to-start-drillbit-locally-in-distributed-mode
> Though these changes are required on production during running unit tests I
> don't think this check should be enforced.
> *For documentation*
> {{drill.exec.allow_loopback_address_binding}} -> Allow drillbit to bind to
> loopback address in distributed mode. Enabled only for testing purposes.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (DRILL-6239) Add Build and License Badges to README.md
[ https://issues.apache.org/jira/browse/DRILL-6239?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arina Ielchiieva updated DRILL-6239: Labels: ready-to-commit (was: ) > Add Build and License Badges to README.md > - > > Key: DRILL-6239 > URL: https://issues.apache.org/jira/browse/DRILL-6239 > Project: Apache Drill > Issue Type: Improvement >Reporter: Timothy Farkas >Assignee: Timothy Farkas >Priority: Major > Labels: ready-to-commit > Fix For: 1.14.0 > > > Other projects have pretty badges showing the build status and license on the > README.md page. We should have it too! -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-6239) Add Build and License Badges to README.md
[ https://issues.apache.org/jira/browse/DRILL-6239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16398366#comment-16398366 ] ASF GitHub Bot commented on DRILL-6239: --- Github user arina-ielchiieva commented on the issue: https://github.com/apache/drill/pull/1165 +1 > Add Build and License Badges to README.md > - > > Key: DRILL-6239 > URL: https://issues.apache.org/jira/browse/DRILL-6239 > Project: Apache Drill > Issue Type: Improvement >Reporter: Timothy Farkas >Assignee: Timothy Farkas >Priority: Major > Labels: ready-to-commit > Fix For: 1.14.0 > > > Other projects have pretty badges showing the build status and license on the > README.md page. We should have it too! -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-6016) Error reading INT96 created by Apache Spark
[ https://issues.apache.org/jira/browse/DRILL-6016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16398344#comment-16398344 ] ASF GitHub Bot commented on DRILL-6016: --- Github user parthchandra commented on the issue: https://github.com/apache/drill/pull/1166 @rajrahul, thanks for submitting the patch. It looks good. I guess we missed dictionary encoded int96 timestamps (even though timestamps with nanosecond precision) are the one thing that should never, ever, be dictionary encoded! Just to make sure, I tried the use the sample file in DRILL-6016, but I could not even unzip it! Can you please check and see if the file is correct? WE can use that to create the unit test as well. > Error reading INT96 created by Apache Spark > --- > > Key: DRILL-6016 > URL: https://issues.apache.org/jira/browse/DRILL-6016 > Project: Apache Drill > Issue Type: Bug > Environment: Drill 1.11 >Reporter: Rahul Raj >Priority: Major > Fix For: 1.14.0 > > > Hi, > I am getting the error - SYSTEM ERROR : ClassCastException: > org.apache.drill.exec.vector.TimeStampVector cannot be cast to > org.apache.drill.exec.vector.VariableWidthVector while trying to read a spark > INT96 datetime field on Drill 1.11 in spite of setting the property > store.parquet.reader.int96_as_timestamp to true. > I believe this was fixed in drill > 1.10(https://issues.apache.org/jira/browse/DRILL-4373). What could be wrong. > I have attached the dataset at > https://github.com/rajrahul/files/blob/master/result.tar.gz -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-6016) Error reading INT96 created by Apache Spark
[ https://issues.apache.org/jira/browse/DRILL-6016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16398190#comment-16398190 ] ASF GitHub Bot commented on DRILL-6016: --- Github user rajrahul commented on the issue: https://github.com/apache/drill/pull/1166 @parthchandra @vdiravka I do not have a test case for this. I have manually verified the scenario with and without the patch. The sample input file is attached with https://issues.apache.org/jira/browse/DRILL-6016. > Error reading INT96 created by Apache Spark > --- > > Key: DRILL-6016 > URL: https://issues.apache.org/jira/browse/DRILL-6016 > Project: Apache Drill > Issue Type: Bug > Environment: Drill 1.11 >Reporter: Rahul Raj >Priority: Major > Fix For: 1.14.0 > > > Hi, > I am getting the error - SYSTEM ERROR : ClassCastException: > org.apache.drill.exec.vector.TimeStampVector cannot be cast to > org.apache.drill.exec.vector.VariableWidthVector while trying to read a spark > INT96 datetime field on Drill 1.11 in spite of setting the property > store.parquet.reader.int96_as_timestamp to true. > I believe this was fixed in drill > 1.10(https://issues.apache.org/jira/browse/DRILL-4373). What could be wrong. > I have attached the dataset at > https://github.com/rajrahul/files/blob/master/result.tar.gz -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-6016) Error reading INT96 created by Apache Spark
[ https://issues.apache.org/jira/browse/DRILL-6016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16398167#comment-16398167 ] ASF GitHub Bot commented on DRILL-6016: --- Github user priteshm commented on the issue: https://github.com/apache/drill/pull/1166 @parthchandra would you please review this? > Error reading INT96 created by Apache Spark > --- > > Key: DRILL-6016 > URL: https://issues.apache.org/jira/browse/DRILL-6016 > Project: Apache Drill > Issue Type: Bug > Environment: Drill 1.11 >Reporter: Rahul Raj >Priority: Major > Fix For: 1.14.0 > > > Hi, > I am getting the error - SYSTEM ERROR : ClassCastException: > org.apache.drill.exec.vector.TimeStampVector cannot be cast to > org.apache.drill.exec.vector.VariableWidthVector while trying to read a spark > INT96 datetime field on Drill 1.11 in spite of setting the property > store.parquet.reader.int96_as_timestamp to true. > I believe this was fixed in drill > 1.10(https://issues.apache.org/jira/browse/DRILL-4373). What could be wrong. > I have attached the dataset at > https://github.com/rajrahul/files/blob/master/result.tar.gz -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (DRILL-6016) Error reading INT96 created by Apache Spark
[ https://issues.apache.org/jira/browse/DRILL-6016?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pritesh Maker updated DRILL-6016: - Fix Version/s: 1.14.0 > Error reading INT96 created by Apache Spark > --- > > Key: DRILL-6016 > URL: https://issues.apache.org/jira/browse/DRILL-6016 > Project: Apache Drill > Issue Type: Bug > Environment: Drill 1.11 >Reporter: Rahul Raj >Priority: Major > Fix For: 1.14.0 > > > Hi, > I am getting the error - SYSTEM ERROR : ClassCastException: > org.apache.drill.exec.vector.TimeStampVector cannot be cast to > org.apache.drill.exec.vector.VariableWidthVector while trying to read a spark > INT96 datetime field on Drill 1.11 in spite of setting the property > store.parquet.reader.int96_as_timestamp to true. > I believe this was fixed in drill > 1.10(https://issues.apache.org/jira/browse/DRILL-4373). What could be wrong. > I have attached the dataset at > https://github.com/rajrahul/files/blob/master/result.tar.gz -- This message was sent by Atlassian JIRA (v7.6.3#76005)
