[jira] [Commented] (DRILL-6255) Drillbit while sending control message to itself creates a connection instead of submitting locally
[ https://issues.apache.org/jira/browse/DRILL-6255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16475307#comment-16475307 ] ASF GitHub Bot commented on DRILL-6255: --- ilooner commented on issue #1253: DRILL-6255: Drillbit while sending control message to itself creates … URL: https://github.com/apache/drill/pull/1253#issuecomment-389050187 @sohami Please fix Travis failures ``` Tests in error: TestLocalControlConnectionManager.setupForTest:101 WrongTypeOfReturnValue Con... TestLocalControlConnectionManager.setupForTest:101 MissingMethodInvocation wh... TestLocalControlConnectionManager.setupForTest:101 MissingMethodInvocation wh... TestLocalControlConnectionManager.setupForTest:101 MissingMethodInvocation wh... TestLocalControlConnectionManager.setupForTest:101 MissingMethodInvocation wh... TestLocalControlConnectionManager.setupForTest:101 MissingMethodInvocation wh... TestLocalControlConnectionManager.setupForTest:101 MissingMethodInvocation wh... TestLocalControlConnectionManager.setupForTest:101 MissingMethodInvocation wh... TestLocalControlConnectionManager.setupForTest:101 MissingMethodInvocation wh... TestLocalControlConnectionManager.setupForTest:101 MissingMethodInvocation wh... TestLocalControlConnectionManager.setupForTest:101 MissingMethodInvocation wh... TestLocalControlConnectionManager.setupForTest:101 MissingMethodInvocation wh... ``` This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Drillbit while sending control message to itself creates a connection instead > of submitting locally > --- > > Key: DRILL-6255 > URL: https://issues.apache.org/jira/browse/DRILL-6255 > Project: Apache Drill > Issue Type: Bug > Components: Execution - Flow >Affects Versions: 1.12.0 >Reporter: Sorabh Hamirwasia >Assignee: Sorabh Hamirwasia >Priority: Major > Fix For: 1.14.0 > > > With the new shutdown feature introduced in 1.12, there is a state introduced > in DrillbitEndpoint. Due to this the equality check happening > [here|https://github.com/apache/drill/blob/master/exec/java-exec/src/main/java/org/apache/drill/exec/work/foreman/FragmentsRunner.java#L256] > will result in false and hence the fragments supposed to be scheduled on > Foreman will be treated as remote fragments and a connection will be created > to schedule it. The equality check is false because localEndpoint state is > STARTUP whereas state in assigned Drillbit is ONLINE. > I guess now we should update the equality check to verify just for address > and control port to be same between assigned and local Drillbit endpoint. A > test can be added for this based on _sys.memory_ table since that guarantees > scheduling minor fragments on each Drillbit node. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-6356) batch sizing for union all
[ https://issues.apache.org/jira/browse/DRILL-6356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16475305#comment-16475305 ] ASF GitHub Bot commented on DRILL-6356: --- ilooner commented on issue #1255: DRILL-6356: batch sizing for union all URL: https://github.com/apache/drill/pull/1255#issuecomment-389049941 @ppadma It looks like the Travis build took 7 minutes longer than usual and timed out and failed. Could this change have introduced a performance regression? Would it be possible to do a quick benchmark? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > batch sizing for union all > -- > > Key: DRILL-6356 > URL: https://issues.apache.org/jira/browse/DRILL-6356 > Project: Apache Drill > Issue Type: Improvement > Components: Execution - Relational Operators >Affects Versions: 1.13.0 >Reporter: Padma Penumarthy >Assignee: Padma Penumarthy >Priority: Major > Fix For: 1.14.0 > > > batch sizing changes for union all operator -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-6353) Upgrade Parquet MR dependencies
[
https://issues.apache.org/jira/browse/DRILL-6353?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16475300#comment-16475300
]
ASF GitHub Bot commented on DRILL-6353:
---
ilooner commented on issue #1259: DRILL-6353: Upgrade Parquet MR dependencies
URL: https://github.com/apache/drill/pull/1259#issuecomment-389049203
@vrozov Please fix Travis failures
```
Tests in error:
TestCTASPartitionFilter.withoutDistribution » UserRemote SYSTEM ERROR:
Illegal...
TestCTASPartitionFilter>BaseTestQuery.closeClient:286 » Runtime Exception
whil...
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Upgrade Parquet MR dependencies
> ---
>
> Key: DRILL-6353
> URL: https://issues.apache.org/jira/browse/DRILL-6353
> Project: Apache Drill
> Issue Type: Task
>Reporter: Vlad Rozov
>Assignee: Vlad Rozov
>Priority: Major
> Fix For: 1.14.0
>
>
> Upgrade from a custom build {{1.8.1-drill-r0}} to Apache release {{1.10.0}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-6321) Lateral Join: Planning changes - enable submitting physical plan
[
https://issues.apache.org/jira/browse/DRILL-6321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16475242#comment-16475242
]
ASF GitHub Bot commented on DRILL-6321:
---
amansinha100 commented on a change in pull request #1258: DRILL-6321: Lateral
Join and Unnest - initial implementation for parser and planning
URL: https://github.com/apache/drill/pull/1258#discussion_r188153197
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/planner/common/DrillCorrelateRelBase.java
##
@@ -0,0 +1,55 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.planner.common;
+
+import org.apache.calcite.plan.RelOptCluster;
+import org.apache.calcite.plan.RelOptCost;
+import org.apache.calcite.plan.RelOptPlanner;
+import org.apache.calcite.plan.RelTraitSet;
+import org.apache.calcite.rel.RelNode;
+import org.apache.calcite.rel.core.Correlate;
+import org.apache.calcite.rel.core.CorrelationId;
+import org.apache.calcite.rel.metadata.RelMetadataQuery;
+import org.apache.calcite.sql.SemiJoinType;
+import org.apache.calcite.util.ImmutableBitSet;
+import org.apache.drill.exec.ExecConstants;
+import org.apache.drill.exec.planner.cost.DrillCostBase;
+import org.apache.drill.exec.planner.physical.PrelUtil;
+
+
+public abstract class DrillCorrelateRelBase extends Correlate implements
DrillRelNode {
+ public DrillCorrelateRelBase(RelOptCluster cluster, RelTraitSet traits,
RelNode left, RelNode right,
+ CorrelationId correlationId, ImmutableBitSet
requiredColumns, SemiJoinType semiJoinType) {
+super(cluster, traits, left, right, correlationId, requiredColumns,
semiJoinType);
+ }
+
+ @Override public RelOptCost computeSelfCost(RelOptPlanner planner,
+ RelMetadataQuery mq) {
+DrillCostBase.DrillCostFactory costFactory =
(DrillCostBase.DrillCostFactory) planner.getCostFactory();
+
+double rowCount = mq.getRowCount(this.getLeft());
+long fieldWidth = PrelUtil.getPlannerSettings(planner).getOptions()
+.getOption(ExecConstants.AVERAGE_FIELD_WIDTH_KEY).num_val;
+
+double rowSize = (this.getLeft().getRowType().getFieldList().size()) *
fieldWidth;
+
+double cpuCost = rowCount * rowSize * DrillCostBase.BASE_CPU_COST;
+double memCost = 0;
Review comment:
For accurate costing, we should have non-zero memCost here (since lateral
join uses memory), although It does not necessarily affect the plan that you
are generating.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Lateral Join: Planning changes - enable submitting physical plan
>
>
> Key: DRILL-6321
> URL: https://issues.apache.org/jira/browse/DRILL-6321
> Project: Apache Drill
> Issue Type: Task
>Reporter: Parth Chandra
>Assignee: Chunhui Shi
>Priority: Major
> Labels: ready-to-commit
> Fix For: 1.14.0
>
>
> Implement changes to enable submitting a physical plan containing lateral and
> unnest.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-6321) Lateral Join: Planning changes - enable submitting physical plan
[ https://issues.apache.org/jira/browse/DRILL-6321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16475241#comment-16475241 ] ASF GitHub Bot commented on DRILL-6321: --- amansinha100 commented on issue #1258: DRILL-6321: Lateral Join and Unnest - initial implementation for parser and planning URL: https://github.com/apache/drill/pull/1258#issuecomment-389029135 @chunhui-shi I have a few comments/questions mostly around costing. However, these are not blockers for the current PR. You can comment on them and address them in a follow-up if needed. Code changes overall LGTM. +1 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Lateral Join: Planning changes - enable submitting physical plan > > > Key: DRILL-6321 > URL: https://issues.apache.org/jira/browse/DRILL-6321 > Project: Apache Drill > Issue Type: Task >Reporter: Parth Chandra >Assignee: Chunhui Shi >Priority: Major > Labels: ready-to-commit > Fix For: 1.14.0 > > > Implement changes to enable submitting a physical plan containing lateral and > unnest. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-6321) Lateral Join: Planning changes - enable submitting physical plan
[
https://issues.apache.org/jira/browse/DRILL-6321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16475234#comment-16475234
]
ASF GitHub Bot commented on DRILL-6321:
---
amansinha100 commented on a change in pull request #1258: DRILL-6321: Lateral
Join and Unnest - initial implementation for parser and planning
URL: https://github.com/apache/drill/pull/1258#discussion_r188153197
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/planner/common/DrillCorrelateRelBase.java
##
@@ -0,0 +1,55 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.planner.common;
+
+import org.apache.calcite.plan.RelOptCluster;
+import org.apache.calcite.plan.RelOptCost;
+import org.apache.calcite.plan.RelOptPlanner;
+import org.apache.calcite.plan.RelTraitSet;
+import org.apache.calcite.rel.RelNode;
+import org.apache.calcite.rel.core.Correlate;
+import org.apache.calcite.rel.core.CorrelationId;
+import org.apache.calcite.rel.metadata.RelMetadataQuery;
+import org.apache.calcite.sql.SemiJoinType;
+import org.apache.calcite.util.ImmutableBitSet;
+import org.apache.drill.exec.ExecConstants;
+import org.apache.drill.exec.planner.cost.DrillCostBase;
+import org.apache.drill.exec.planner.physical.PrelUtil;
+
+
+public abstract class DrillCorrelateRelBase extends Correlate implements
DrillRelNode {
+ public DrillCorrelateRelBase(RelOptCluster cluster, RelTraitSet traits,
RelNode left, RelNode right,
+ CorrelationId correlationId, ImmutableBitSet
requiredColumns, SemiJoinType semiJoinType) {
+super(cluster, traits, left, right, correlationId, requiredColumns,
semiJoinType);
+ }
+
+ @Override public RelOptCost computeSelfCost(RelOptPlanner planner,
+ RelMetadataQuery mq) {
+DrillCostBase.DrillCostFactory costFactory =
(DrillCostBase.DrillCostFactory) planner.getCostFactory();
+
+double rowCount = mq.getRowCount(this.getLeft());
+long fieldWidth = PrelUtil.getPlannerSettings(planner).getOptions()
+.getOption(ExecConstants.AVERAGE_FIELD_WIDTH_KEY).num_val;
+
+double rowSize = (this.getLeft().getRowType().getFieldList().size()) *
fieldWidth;
+
+double cpuCost = rowCount * rowSize * DrillCostBase.BASE_CPU_COST;
+double memCost = 0;
Review comment:
For accurate costing, we should have non-zero memCost here (since lateral
join uses memory), although It does necessarily affect the plan that you are
generating.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Lateral Join: Planning changes - enable submitting physical plan
>
>
> Key: DRILL-6321
> URL: https://issues.apache.org/jira/browse/DRILL-6321
> Project: Apache Drill
> Issue Type: Task
>Reporter: Parth Chandra
>Assignee: Chunhui Shi
>Priority: Major
> Labels: ready-to-commit
> Fix For: 1.14.0
>
>
> Implement changes to enable submitting a physical plan containing lateral and
> unnest.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-6321) Lateral Join: Planning changes - enable submitting physical plan
[
https://issues.apache.org/jira/browse/DRILL-6321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16475233#comment-16475233
]
ASF GitHub Bot commented on DRILL-6321:
---
amansinha100 commented on a change in pull request #1258: DRILL-6321: Lateral
Join and Unnest - initial implementation for parser and planning
URL: https://github.com/apache/drill/pull/1258#discussion_r188150005
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/planner/common/DrillUnnestRelBase.java
##
@@ -0,0 +1,54 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.planner.common;
+
+import org.apache.calcite.plan.RelOptCluster;
+import org.apache.calcite.plan.RelOptCost;
+import org.apache.calcite.plan.RelOptPlanner;
+import org.apache.calcite.plan.RelTraitSet;
+import org.apache.calcite.rel.AbstractRelNode;
+import org.apache.calcite.rel.metadata.RelMetadataQuery;
+import org.apache.calcite.rex.RexNode;
+import org.apache.drill.exec.planner.cost.DrillCostBase;
+
+public class DrillUnnestRelBase extends AbstractRelNode implements
DrillRelNode {
+
+ final protected RexNode ref;
+
+ public DrillUnnestRelBase(RelOptCluster cluster, RelTraitSet traitSet,
RexNode ref) {
+super(cluster, traitSet);
+this.ref = ref;
+ }
+
+ @Override
+ public RelOptCost computeSelfCost(RelOptPlanner planner, RelMetadataQuery
mq) {
+
+double rowCount = mq.getRowCount(this);
+// Attribute small cost for projecting simple fields. In reality
projecting simple columns in not free and
Review comment:
Technically, the unnest cost would be proportional to the size of the array
being unnested but the array size is not known at planning time. What will the
EXPLAIN plan show for the estimated row count of the UNNEST ? will it always
be 1 or does Calcite use some average array size ? Note that the downstream
operators (such as filter, aggregate, sort and the Lateral Join itself) will
use this row count to determine their own cost (e.g hash aggregate vs streaming
agg). Have you thought about how to model this ?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Lateral Join: Planning changes - enable submitting physical plan
>
>
> Key: DRILL-6321
> URL: https://issues.apache.org/jira/browse/DRILL-6321
> Project: Apache Drill
> Issue Type: Task
>Reporter: Parth Chandra
>Assignee: Chunhui Shi
>Priority: Major
> Labels: ready-to-commit
> Fix For: 1.14.0
>
>
> Implement changes to enable submitting a physical plan containing lateral and
> unnest.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-6321) Lateral Join: Planning changes - enable submitting physical plan
[
https://issues.apache.org/jira/browse/DRILL-6321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16475235#comment-16475235
]
ASF GitHub Bot commented on DRILL-6321:
---
amansinha100 commented on a change in pull request #1258: DRILL-6321: Lateral
Join and Unnest - initial implementation for parser and planning
URL: https://github.com/apache/drill/pull/1258#discussion_r188150611
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/planner/physical/CorrelatePrule.java
##
@@ -0,0 +1,56 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.planner.physical;
+
+import org.apache.calcite.plan.RelOptRule;
+import org.apache.calcite.plan.RelOptRuleCall;
+import org.apache.calcite.plan.RelOptRuleOperand;
+import org.apache.calcite.plan.RelTraitSet;
+import org.apache.calcite.rel.RelNode;
+import org.apache.drill.exec.planner.logical.DrillCorrelateRel;
+import org.apache.drill.exec.planner.logical.RelOptHelper;
+
+public class CorrelatePrule extends Prule {
+ public static final RelOptRule INSTANCE = new
CorrelatePrule("Prel.CorrelatePrule",
+ RelOptHelper.any(DrillCorrelateRel.class));
+
+ private CorrelatePrule(String name, RelOptRuleOperand operand) {
+super(operand, name);
+ }
+
+ @Override
+ public void onMatch(RelOptRuleCall call) {
+final DrillCorrelateRel correlate = call.rel(0);
+final RelNode left = correlate.getLeft();
+final RelNode right = correlate.getRight();
+RelTraitSet traitsLeft = left.getTraitSet().plus(Prel.DRILL_PHYSICAL);
+RelTraitSet traitsRight = right.getTraitSet().plus(Prel.DRILL_PHYSICAL);
+
+RelTraitSet corrTraits =
traitsLeft.plus(DrillDistributionTrait.RANDOM_DISTRIBUTED);
Review comment:
Pls add a comment about why RANDOM_DISTRIBUTED trait is chosen for
correlate/lateral.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Lateral Join: Planning changes - enable submitting physical plan
>
>
> Key: DRILL-6321
> URL: https://issues.apache.org/jira/browse/DRILL-6321
> Project: Apache Drill
> Issue Type: Task
>Reporter: Parth Chandra
>Assignee: Chunhui Shi
>Priority: Major
> Labels: ready-to-commit
> Fix For: 1.14.0
>
>
> Implement changes to enable submitting a physical plan containing lateral and
> unnest.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-6321) Lateral Join: Planning changes - enable submitting physical plan
[
https://issues.apache.org/jira/browse/DRILL-6321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16475236#comment-16475236
]
ASF GitHub Bot commented on DRILL-6321:
---
amansinha100 commented on a change in pull request #1258: DRILL-6321: Lateral
Join and Unnest - initial implementation for parser and planning
URL: https://github.com/apache/drill/pull/1258#discussion_r188152596
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/planner/common/DrillCorrelateRelBase.java
##
@@ -0,0 +1,55 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.planner.common;
+
+import org.apache.calcite.plan.RelOptCluster;
+import org.apache.calcite.plan.RelOptCost;
+import org.apache.calcite.plan.RelOptPlanner;
+import org.apache.calcite.plan.RelTraitSet;
+import org.apache.calcite.rel.RelNode;
+import org.apache.calcite.rel.core.Correlate;
+import org.apache.calcite.rel.core.CorrelationId;
+import org.apache.calcite.rel.metadata.RelMetadataQuery;
+import org.apache.calcite.sql.SemiJoinType;
+import org.apache.calcite.util.ImmutableBitSet;
+import org.apache.drill.exec.ExecConstants;
+import org.apache.drill.exec.planner.cost.DrillCostBase;
+import org.apache.drill.exec.planner.physical.PrelUtil;
+
+
+public abstract class DrillCorrelateRelBase extends Correlate implements
DrillRelNode {
+ public DrillCorrelateRelBase(RelOptCluster cluster, RelTraitSet traits,
RelNode left, RelNode right,
+ CorrelationId correlationId, ImmutableBitSet
requiredColumns, SemiJoinType semiJoinType) {
+super(cluster, traits, left, right, correlationId, requiredColumns,
semiJoinType);
+ }
+
+ @Override public RelOptCost computeSelfCost(RelOptPlanner planner,
+ RelMetadataQuery mq) {
+DrillCostBase.DrillCostFactory costFactory =
(DrillCostBase.DrillCostFactory) planner.getCostFactory();
+
+double rowCount = mq.getRowCount(this.getLeft());
Review comment:
Since correlate (lateral join) is a cross join, the actual rowCount = left
row count x estimated right row count.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Lateral Join: Planning changes - enable submitting physical plan
>
>
> Key: DRILL-6321
> URL: https://issues.apache.org/jira/browse/DRILL-6321
> Project: Apache Drill
> Issue Type: Task
>Reporter: Parth Chandra
>Assignee: Chunhui Shi
>Priority: Major
> Labels: ready-to-commit
> Fix For: 1.14.0
>
>
> Implement changes to enable submitting a physical plan containing lateral and
> unnest.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-6353) Upgrade Parquet MR dependencies
[
https://issues.apache.org/jira/browse/DRILL-6353?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16475192#comment-16475192
]
ASF GitHub Bot commented on DRILL-6353:
---
vrozov commented on a change in pull request #1259: DRILL-6353: Upgrade Parquet
MR dependencies
URL: https://github.com/apache/drill/pull/1259#discussion_r188148618
##
File path: pom.xml
##
@@ -44,7 +44,7 @@
1.7.6
18.0
2
-1.8.1-drill-r0
+1.10.0
Review comment:
`1.8.3` as well as `1.8.1-drill-r0` are supposed to be a patch release on
top of `1.8.0`. Unfortunately `parquet-mr` does not properly follow semantic
versioning and have functional and API level changes in the patch releases. On
top of that, both `1.9.0` and `1.10.0` are not backward compatible with `1.8.0`
and/or `1.8.3`, so upgrade to `1.8.3` will not help us with the upgrade to
`1.10.0` later either or make it safer. I'd suggest to pay the price once.
Additionally, the latest parquet version may have a functionality to be used in
filter pushdown. @arina-ielchiieva what is your take?
Parquet libraries used by hive are shaded within `drill-hive-exec-shaded`,
so hive is guarded from the parquet-mr library upgrade.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Upgrade Parquet MR dependencies
> ---
>
> Key: DRILL-6353
> URL: https://issues.apache.org/jira/browse/DRILL-6353
> Project: Apache Drill
> Issue Type: Task
>Reporter: Vlad Rozov
>Assignee: Vlad Rozov
>Priority: Major
> Fix For: 1.14.0
>
>
> Upgrade from a custom build {{1.8.1-drill-r0}} to Apache release {{1.10.0}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-6321) Lateral Join: Planning changes - enable submitting physical plan
[ https://issues.apache.org/jira/browse/DRILL-6321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16475152#comment-16475152 ] ASF GitHub Bot commented on DRILL-6321: --- chunhui-shi commented on a change in pull request #1258: DRILL-6321: Lateral Join and Unnest - initial implementation for parser and planning URL: https://github.com/apache/drill/pull/1258#discussion_r188143508 ## File path: exec/java-exec/src/main/java/org/apache/drill/exec/planner/physical/PlannerSettings.java ## @@ -112,6 +112,12 @@ public static final EnumeratedStringValidator QUOTING_IDENTIFIERS = new EnumeratedStringValidator( QUOTING_IDENTIFIERS_KEY, Quoting.BACK_TICK.string, Quoting.DOUBLE_QUOTE.string, Quoting.BRACKET.string); + /* +"planner.enable_unnest_lateral" is to allow users to choose enable unnest+lateraljoin feature. + */ + public static final String ENABLE_UNNEST_LATERAL_KEY = "planner.enable_unnest_lateral"; Review comment: This is temporary or for test purpose This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Lateral Join: Planning changes - enable submitting physical plan > > > Key: DRILL-6321 > URL: https://issues.apache.org/jira/browse/DRILL-6321 > Project: Apache Drill > Issue Type: Task >Reporter: Parth Chandra >Assignee: Chunhui Shi >Priority: Major > Labels: ready-to-commit > Fix For: 1.14.0 > > > Implement changes to enable submitting a physical plan containing lateral and > unnest. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-6321) Lateral Join: Planning changes - enable submitting physical plan
[ https://issues.apache.org/jira/browse/DRILL-6321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16475126#comment-16475126 ] ASF GitHub Bot commented on DRILL-6321: --- amansinha100 commented on a change in pull request #1258: DRILL-6321: Lateral Join and Unnest - initial implementation for parser and planning URL: https://github.com/apache/drill/pull/1258#discussion_r188138431 ## File path: exec/java-exec/src/main/java/org/apache/drill/exec/planner/physical/PlannerSettings.java ## @@ -112,6 +112,12 @@ public static final EnumeratedStringValidator QUOTING_IDENTIFIERS = new EnumeratedStringValidator( QUOTING_IDENTIFIERS_KEY, Quoting.BACK_TICK.string, Quoting.DOUBLE_QUOTE.string, Quoting.BRACKET.string); + /* +"planner.enable_unnest_lateral" is to allow users to choose enable unnest+lateraljoin feature. + */ + public static final String ENABLE_UNNEST_LATERAL_KEY = "planner.enable_unnest_lateral"; Review comment: Do we need this option ? Or is this temporary ? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Lateral Join: Planning changes - enable submitting physical plan > > > Key: DRILL-6321 > URL: https://issues.apache.org/jira/browse/DRILL-6321 > Project: Apache Drill > Issue Type: Task >Reporter: Parth Chandra >Assignee: Chunhui Shi >Priority: Major > Labels: ready-to-commit > Fix For: 1.14.0 > > > Implement changes to enable submitting a physical plan containing lateral and > unnest. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-6373) Refactor the Result Set Loader to prepare for Union, List support
[
https://issues.apache.org/jira/browse/DRILL-6373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16475122#comment-16475122
]
ASF GitHub Bot commented on DRILL-6373:
---
paul-rogers commented on issue #1244: DRILL-6373: Refactor Result Set Loader
for Union, List support
URL: https://github.com/apache/drill/pull/1244#issuecomment-389006322
Thanks @ppadma!
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Refactor the Result Set Loader to prepare for Union, List support
> -
>
> Key: DRILL-6373
> URL: https://issues.apache.org/jira/browse/DRILL-6373
> Project: Apache Drill
> Issue Type: Improvement
>Affects Versions: 1.13.0
>Reporter: Paul Rogers
>Assignee: Paul Rogers
>Priority: Major
> Labels: ready-to-commit
> Fix For: 1.14.0
>
>
> As the next step in merging the "batch sizing" enhancements, refactor the
> {{ResultSetLoader}} and related classes to prepare for Union and List
> support. This fix follows the refactoring of the column accessors for the
> same purpose. Actual Union and List support is to follow in a separate PR.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (DRILL-6373) Refactor the Result Set Loader to prepare for Union, List support
[
https://issues.apache.org/jira/browse/DRILL-6373?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Paul Rogers updated DRILL-6373:
---
Labels: ready-to-commit (was: )
Reviewer: Padma Penumarthy
> Refactor the Result Set Loader to prepare for Union, List support
> -
>
> Key: DRILL-6373
> URL: https://issues.apache.org/jira/browse/DRILL-6373
> Project: Apache Drill
> Issue Type: Improvement
>Affects Versions: 1.13.0
>Reporter: Paul Rogers
>Assignee: Paul Rogers
>Priority: Major
> Labels: ready-to-commit
> Fix For: 1.14.0
>
>
> As the next step in merging the "batch sizing" enhancements, refactor the
> {{ResultSetLoader}} and related classes to prepare for Union and List
> support. This fix follows the refactoring of the column accessors for the
> same purpose. Actual Union and List support is to follow in a separate PR.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (DRILL-6415) Unit test TestGracefulShutdown.testRestApiShutdown times out
[
https://issues.apache.org/jira/browse/DRILL-6415?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Abhishek Girish updated DRILL-6415:
---
Summary: Unit test TestGracefulShutdown.testRestApiShutdown times out
(was: Unit test testRestApiShutdown timesout)
> Unit test TestGracefulShutdown.testRestApiShutdown times out
>
>
> Key: DRILL-6415
> URL: https://issues.apache.org/jira/browse/DRILL-6415
> Project: Apache Drill
> Issue Type: Bug
> Components: Tools, Build & Test
>Reporter: Abhishek Girish
>Priority: Major
>
> {code}
> 16:03:40.415 [main] ERROR org.apache.drill.TestReporter - Test Failed (d:
> -18.3 KiB(72.9 KiB), h: -335.3 MiB(1.3 GiB), nh: 1.1 MiB(335.9 MiB)):
> testRestApiShutdown(org.apache.drill.test.TestGracefulShutdown)
> org.junit.runners.model.TestTimedOutException: test timed out after 18
> milliseconds
> at sun.misc.Unsafe.park(Native Method) ~[na:1.8.0_161]
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> ~[na:1.8.0_161]
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitUninterruptibly(AbstractQueuedSynchronizer.java:1976)
> ~[na:1.8.0_161]
> at
> org.apache.drill.exec.work.WorkManager.waitToExit(WorkManager.java:203)
> ~[classes/:na]
> at org.apache.drill.exec.server.Drillbit.close(Drillbit.java:242)
> ~[classes/:na]
> at
> org.apache.drill.test.ClusterFixture.safeClose(ClusterFixture.java:454)
> ~[test-classes/:1.14.0-SNAPSHOT]
> at org.apache.drill.test.ClusterFixture.close(ClusterFixture.java:405)
> ~[test-classes/:1.14.0-SNAPSHOT]
> at
> org.apache.drill.test.TestGracefulShutdown.testRestApiShutdown(TestGracefulShutdown.java:294)
> ~[test-classes/:1.14.0-SNAPSHOT]
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> ~[na:1.8.0_161]
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> ~[na:1.8.0_161]
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> ~[na:1.8.0_161]
> at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_161]
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> ~[junit-4.12.jar:4.12]
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> ~[junit-4.12.jar:4.12]
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> ~[junit-4.12.jar:4.12]
> at
> mockit.integration.junit4.internal.JUnit4TestRunnerDecorator.executeTestMethod(JUnit4TestRunnerDecorator.java:154)
> ~[jmockit-1.39.jar:1.39]
> at
> mockit.integration.junit4.internal.JUnit4TestRunnerDecorator.invokeExplosively(JUnit4TestRunnerDecorator.java:70)
> ~[jmockit-1.39.jar:1.39]
> at
> mockit.integration.junit4.internal.FakeFrameworkMethod.invokeExplosively(FakeFrameworkMethod.java:34)
> ~[jmockit-1.39.jar:1.39]
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java)
> ~[junit-4.12.jar:4.12]
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> ~[junit-4.12.jar:4.12]
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> ~[junit-4.12.jar:4.12]
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> ~[junit-4.12.jar:4.12]
> at
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298)
> ~[junit-4.12.jar:4.12]
> at
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292)
> ~[junit-4.12.jar:4.12]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> ~[na:1.8.0_161]
> at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_161]
> {code}
> {code}
> testRestApiShutdown(org.apache.drill.test.TestGracefulShutdown) Time
> elapsed: 180.028 sec <<< ERROR!
> org.junit.runners.model.TestTimedOutException: test timed out after 18
> milliseconds
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitUninterruptibly(AbstractQueuedSynchronizer.java:1976)
> at
> org.apache.drill.exec.work.WorkManager.waitToExit(WorkManager.java:203)
> at org.apache.drill.exec.server.Drillbit.close(Drillbit.java:242)
> at
> org.apache.drill.test.ClusterFixture.safeClose(ClusterFixture.java:454)
> at org.apache.drill.test.ClusterFixture.close(ClusterFixture.java:405)
> at
> org.apache.drill.test.TestGracefulShutdown.testRestApiShutdown(TestGracefulShutdown.java:294)
[jira] [Created] (DRILL-6416) Unit test TestTpchDistributedConcurrent.testConcurrentQueries fails with AssertionError
Abhishek Girish created DRILL-6416:
--
Summary: Unit test
TestTpchDistributedConcurrent.testConcurrentQueries fails with AssertionError
Key: DRILL-6416
URL: https://issues.apache.org/jira/browse/DRILL-6416
Project: Apache Drill
Issue Type: Bug
Components: Tools, Build & Test
Reporter: Abhishek Girish
{code}
Running org.apache.drill.TestTpchDistributedConcurrent#testConcurrentQueries
16:38:21.784 [2505e212-b165-7812-5c91-0a407a213964:frag:3:1] ERROR
o.a.d.e.w.fragment.FragmentExecutor - SYSTEM ERROR: AssertionError
Fragment 3:1
[Error Id: 436120b6-5255-437e-af53-313e1c3207e0 on drillu1.qa.lab:31064]
org.apache.drill.common.exceptions.UserException: SYSTEM ERROR: AssertionError
Fragment 3:1
[Error Id: 436120b6-5255-437e-af53-313e1c3207e0 on drillu1.qa.lab:31064]
at
org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:633)
~[drill-common-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
at
org.apache.drill.exec.work.fragment.FragmentExecutor.sendFinalState(FragmentExecutor.java:359)
[classes/:na]
at
org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:214)
[classes/:na]
at
org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:325)
[classes/:na]
at
org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38)
[drill-common-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[na:1.8.0_161]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[na:1.8.0_161]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0_161]
Caused by: java.lang.RuntimeException: java.lang.AssertionError
at
org.apache.drill.common.DeferredException.addThrowable(DeferredException.java:101)
~[drill-common-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
at
org.apache.drill.exec.work.fragment.FragmentExecutor.fail(FragmentExecutor.java:471)
[classes/:na]
at
org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:313)
[classes/:na]
... 4 common frames omitted
Caused by: java.lang.AssertionError: null
at
org.apache.drill.exec.compile.sig.MappingSet.enterConstant(MappingSet.java:85)
~[classes/:na]
at
org.apache.drill.exec.expr.EvaluationVisitor$ConstantFilter.visitBooleanConstant(EvaluationVisitor.java:1376)
~[classes/:na]
at
org.apache.drill.exec.expr.EvaluationVisitor$CSEFilter.visitBooleanConstant(EvaluationVisitor.java:1043)
~[classes/:na]
at
org.apache.drill.exec.expr.EvaluationVisitor$CSEFilter.visitBooleanConstant(EvaluationVisitor.java:843)
~[classes/:na]
at
org.apache.drill.common.expression.ValueExpressions$BooleanExpression.accept(ValueExpressions.java:186)
~[drill-logical-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
at
org.apache.drill.exec.expr.EvaluationVisitor$EvalVisitor.visitReturnValueExpression(EvaluationVisitor.java:579)
~[classes/:na]
at
org.apache.drill.exec.expr.EvaluationVisitor$EvalVisitor.visitUnknown(EvaluationVisitor.java:342)
~[classes/:na]
at
org.apache.drill.exec.expr.EvaluationVisitor$ConstantFilter.visitUnknown(EvaluationVisitor.java:1399)
~[classes/:na]
at
org.apache.drill.exec.expr.EvaluationVisitor$CSEFilter.visitUnknown(EvaluationVisitor.java:1084)
~[classes/:na]
at
org.apache.drill.exec.expr.EvaluationVisitor$CSEFilter.visitUnknown(EvaluationVisitor.java:843)
~[classes/:na]
at
org.apache.drill.exec.physical.impl.filter.ReturnValueExpression.accept(ReturnValueExpression.java:56)
~[classes/:na]
at
org.apache.drill.exec.expr.EvaluationVisitor.addExpr(EvaluationVisitor.java:100)
~[classes/:na]
at
org.apache.drill.exec.expr.ClassGenerator.addExpr(ClassGenerator.java:334)
~[classes/:na]
at
org.apache.drill.exec.physical.impl.join.NestedLoopJoinBatch.setupWorker(NestedLoopJoinBatch.java:266)
~[classes/:na]
at
org.apache.drill.exec.physical.impl.join.NestedLoopJoinBatch.buildSchema(NestedLoopJoinBatch.java:384)
~[classes/:na]
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:144)
~[classes/:na]
at
org.apache.drill.exec.physical.impl.validate.IteratorValidatorBatchIterator.next(IteratorValidatorBatchIterator.java:229)
~[classes/:na]
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:118)
~[classes/:na]
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:108)
~[classes/:na]
at
org.apache.drill.exec.record.AbstractUnaryRecordBatch.innerNext(AbstractUnaryRecordBatch.java:63)
~[classes/:na]
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:164)
~[classes/:na]
[jira] [Created] (DRILL-6415) Unit test testRestApiShutdown timesout
Abhishek Girish created DRILL-6415:
--
Summary: Unit test testRestApiShutdown timesout
Key: DRILL-6415
URL: https://issues.apache.org/jira/browse/DRILL-6415
Project: Apache Drill
Issue Type: Bug
Components: Tools, Build & Test
Reporter: Abhishek Girish
{code}
16:03:40.415 [main] ERROR org.apache.drill.TestReporter - Test Failed (d: -18.3
KiB(72.9 KiB), h: -335.3 MiB(1.3 GiB), nh: 1.1 MiB(335.9 MiB)):
testRestApiShutdown(org.apache.drill.test.TestGracefulShutdown)
org.junit.runners.model.TestTimedOutException: test timed out after 18
milliseconds
at sun.misc.Unsafe.park(Native Method) ~[na:1.8.0_161]
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
~[na:1.8.0_161]
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitUninterruptibly(AbstractQueuedSynchronizer.java:1976)
~[na:1.8.0_161]
at
org.apache.drill.exec.work.WorkManager.waitToExit(WorkManager.java:203)
~[classes/:na]
at org.apache.drill.exec.server.Drillbit.close(Drillbit.java:242)
~[classes/:na]
at
org.apache.drill.test.ClusterFixture.safeClose(ClusterFixture.java:454)
~[test-classes/:1.14.0-SNAPSHOT]
at org.apache.drill.test.ClusterFixture.close(ClusterFixture.java:405)
~[test-classes/:1.14.0-SNAPSHOT]
at
org.apache.drill.test.TestGracefulShutdown.testRestApiShutdown(TestGracefulShutdown.java:294)
~[test-classes/:1.14.0-SNAPSHOT]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
~[na:1.8.0_161]
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
~[na:1.8.0_161]
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
~[na:1.8.0_161]
at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_161]
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
~[junit-4.12.jar:4.12]
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
~[junit-4.12.jar:4.12]
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
~[junit-4.12.jar:4.12]
at
mockit.integration.junit4.internal.JUnit4TestRunnerDecorator.executeTestMethod(JUnit4TestRunnerDecorator.java:154)
~[jmockit-1.39.jar:1.39]
at
mockit.integration.junit4.internal.JUnit4TestRunnerDecorator.invokeExplosively(JUnit4TestRunnerDecorator.java:70)
~[jmockit-1.39.jar:1.39]
at
mockit.integration.junit4.internal.FakeFrameworkMethod.invokeExplosively(FakeFrameworkMethod.java:34)
~[jmockit-1.39.jar:1.39]
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java)
~[junit-4.12.jar:4.12]
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
~[junit-4.12.jar:4.12]
at
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
~[junit-4.12.jar:4.12]
at
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
~[junit-4.12.jar:4.12]
at
org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298)
~[junit-4.12.jar:4.12]
at
org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292)
~[junit-4.12.jar:4.12]
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
~[na:1.8.0_161]
at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_161]
{code}
{code}
testRestApiShutdown(org.apache.drill.test.TestGracefulShutdown) Time elapsed:
180.028 sec <<< ERROR!
org.junit.runners.model.TestTimedOutException: test timed out after 18
milliseconds
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitUninterruptibly(AbstractQueuedSynchronizer.java:1976)
at
org.apache.drill.exec.work.WorkManager.waitToExit(WorkManager.java:203)
at org.apache.drill.exec.server.Drillbit.close(Drillbit.java:242)
at
org.apache.drill.test.ClusterFixture.safeClose(ClusterFixture.java:454)
at org.apache.drill.test.ClusterFixture.close(ClusterFixture.java:405)
at
org.apache.drill.test.TestGracefulShutdown.testRestApiShutdown(TestGracefulShutdown.java:294)
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-6255) Drillbit while sending control message to itself creates a connection instead of submitting locally
[ https://issues.apache.org/jira/browse/DRILL-6255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16475047#comment-16475047 ] ASF GitHub Bot commented on DRILL-6255: --- sohami commented on issue #1253: DRILL-6255: Drillbit while sending control message to itself creates … URL: https://github.com/apache/drill/pull/1253#issuecomment-388995280 Rebased on top of latest master This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Drillbit while sending control message to itself creates a connection instead > of submitting locally > --- > > Key: DRILL-6255 > URL: https://issues.apache.org/jira/browse/DRILL-6255 > Project: Apache Drill > Issue Type: Bug > Components: Execution - Flow >Affects Versions: 1.12.0 >Reporter: Sorabh Hamirwasia >Assignee: Sorabh Hamirwasia >Priority: Major > Fix For: 1.14.0 > > > With the new shutdown feature introduced in 1.12, there is a state introduced > in DrillbitEndpoint. Due to this the equality check happening > [here|https://github.com/apache/drill/blob/master/exec/java-exec/src/main/java/org/apache/drill/exec/work/foreman/FragmentsRunner.java#L256] > will result in false and hence the fragments supposed to be scheduled on > Foreman will be treated as remote fragments and a connection will be created > to schedule it. The equality check is false because localEndpoint state is > STARTUP whereas state in assigned Drillbit is ONLINE. > I guess now we should update the equality check to verify just for address > and control port to be same between assigned and local Drillbit endpoint. A > test can be added for this based on _sys.memory_ table since that guarantees > scheduling minor fragments on each Drillbit node. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-6255) Drillbit while sending control message to itself creates a connection instead of submitting locally
[
https://issues.apache.org/jira/browse/DRILL-6255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16475042#comment-16475042
]
ASF GitHub Bot commented on DRILL-6255:
---
sohami commented on a change in pull request #1253: DRILL-6255: Drillbit while
sending control message to itself creates …
URL: https://github.com/apache/drill/pull/1253#discussion_r188097298
##
File path:
exec/rpc/src/main/java/org/apache/drill/exec/rpc/FutureBitCommand.java
##
@@ -71,10 +73,15 @@ public void interrupted(final InterruptedException e) {
return parentFuture;
}
+ @Override
+ public RpcOutcomeListener getOutcomeListener() {
+return outcomeListener;
+ }
+
@Override
public void connectionFailed(FailureType type, Throwable t) {
settableFuture.setException(RpcException.mapException(
String.format("Command failed while establishing connection. Failure
type %s.", type), t));
}
-}
\ No newline at end of file
+}
Review comment:
Added intentionally as it's recommended.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Drillbit while sending control message to itself creates a connection instead
> of submitting locally
> ---
>
> Key: DRILL-6255
> URL: https://issues.apache.org/jira/browse/DRILL-6255
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Flow
>Affects Versions: 1.12.0
>Reporter: Sorabh Hamirwasia
>Assignee: Sorabh Hamirwasia
>Priority: Major
> Fix For: 1.14.0
>
>
> With the new shutdown feature introduced in 1.12, there is a state introduced
> in DrillbitEndpoint. Due to this the equality check happening
> [here|https://github.com/apache/drill/blob/master/exec/java-exec/src/main/java/org/apache/drill/exec/work/foreman/FragmentsRunner.java#L256]
> will result in false and hence the fragments supposed to be scheduled on
> Foreman will be treated as remote fragments and a connection will be created
> to schedule it. The equality check is false because localEndpoint state is
> STARTUP whereas state in assigned Drillbit is ONLINE.
> I guess now we should update the equality check to verify just for address
> and control port to be same between assigned and local Drillbit endpoint. A
> test can be added for this based on _sys.memory_ table since that guarantees
> scheduling minor fragments on each Drillbit node.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-6255) Drillbit while sending control message to itself creates a connection instead of submitting locally
[
https://issues.apache.org/jira/browse/DRILL-6255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16475040#comment-16475040
]
ASF GitHub Bot commented on DRILL-6255:
---
sohami commented on a change in pull request #1253: DRILL-6255: Drillbit while
sending control message to itself creates …
URL: https://github.com/apache/drill/pull/1253#discussion_r188090731
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/work/foreman/FragmentsRunner.java
##
@@ -180,44 +170,38 @@ private void setupNonRootFragments(final
Collection fragments) thr
foreman.getQueryManager().addFragmentStatusTracker(planFragment, false);
if (planFragment.getLeafFragment()) {
-updateFragmentCollection(planFragment, localDrillbitEndpoint,
localLeafFragmentList, remoteLeafFragmentMap);
+leafFragmentMap.put(planFragment.getAssignment(), planFragment);
} else {
-updateFragmentCollection(planFragment, localDrillbitEndpoint,
localIntFragmentList, remoteIntFragmentMap);
+intFragmentMap.put(planFragment.getAssignment(), planFragment);
}
}
/*
* We need to wait for the intermediates to be sent so that they'll be set
up by the time
- * the leaves start producing data. We'll use this latch to wait for the
responses.
+ * the leaves start producing data. We'll use this latch to wait for the
responses. All the local intermediate
+ * fragments are submitted locally without creating any actual control
connection to itself.
*
* However, in order not to hang the process if any of the RPC requests
fails, we always
* count down (see FragmentSubmitFailures), but we count the number of
failures so that we'll
* know if any submissions did fail.
*/
-scheduleRemoteIntermediateFragments(remoteIntFragmentMap);
-
-// Setup local intermediate fragments
-for (final PlanFragment fragment : localIntFragmentList) {
- startLocalFragment(fragment);
-}
+scheduleRemoteIntermediateFragments(intFragmentMap);
injector.injectChecked(foreman.getQueryContext().getExecutionControls(),
"send-fragments", ForemanException.class);
/*
* Send the remote (leaf) fragments; we don't wait for these. Any problems
will come in through
- * the regular sendListener event delivery.
+ * the regular sendListener event delivery˚. All the local leaf fragments
are submitted locally without creating
+ * any actual control connection to itself.
*/
-for (final DrillbitEndpoint ep : remoteLeafFragmentMap.keySet()) {
- sendRemoteFragments(ep, remoteLeafFragmentMap.get(ep), null, null);
-}
-
-// Setup local leaf fragments
-for (final PlanFragment fragment : localLeafFragmentList) {
- startLocalFragment(fragment);
+for (final DrillbitEndpoint ep : leafFragmentMap.keySet()) {
+ sendRemoteFragments(ep, leafFragmentMap.get(ep), null, null);
}
}
/**
- * Send all the remote fragments belonging to a single target drillbit in
one request.
+ * Send all the remote fragments belonging to a single target drillbit in
one request. If the assignment
Review comment:
renamed `scheduleRemoteIntermediateFragments` to
`scheduleIntermediateFragments` but kept `sendRemoteFragments` as is since it's
used to send intermediate fragments as well not just leaf fragments.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Drillbit while sending control message to itself creates a connection instead
> of submitting locally
> ---
>
> Key: DRILL-6255
> URL: https://issues.apache.org/jira/browse/DRILL-6255
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Flow
>Affects Versions: 1.12.0
>Reporter: Sorabh Hamirwasia
>Assignee: Sorabh Hamirwasia
>Priority: Major
> Fix For: 1.14.0
>
>
> With the new shutdown feature introduced in 1.12, there is a state introduced
> in DrillbitEndpoint. Due to this the equality check happening
> [here|https://github.com/apache/drill/blob/master/exec/java-exec/src/main/java/org/apache/drill/exec/work/foreman/FragmentsRunner.java#L256]
> will result in false and hence the fragments supposed to be scheduled on
> Foreman will be treated as remote fragments and a connection will be created
> to schedule it. The equality check is false because localEndpoint state is
> STARTUP whereas state in assigned Drillbit is ONLINE.
> I guess now we should upd
[jira] [Commented] (DRILL-6255) Drillbit while sending control message to itself creates a connection instead of submitting locally
[
https://issues.apache.org/jira/browse/DRILL-6255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16475046#comment-16475046
]
ASF GitHub Bot commented on DRILL-6255:
---
sohami commented on a change in pull request #1253: DRILL-6255: Drillbit while
sending control message to itself creates …
URL: https://github.com/apache/drill/pull/1253#discussion_r188096274
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/rpc/control/LocalControlConnectionManager.java
##
@@ -0,0 +1,236 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.rpc.control;
+
+import io.netty.buffer.ByteBuf;
+import io.netty.buffer.DrillBuf;
+import org.apache.drill.exec.exception.OutOfMemoryException;
+import org.apache.drill.exec.proto.BitControl;
+import org.apache.drill.exec.proto.CoordinationProtos.DrillbitEndpoint;
+import org.apache.drill.exec.proto.GeneralRPCProtos.Ack;
+import org.apache.drill.exec.proto.UserBitShared;
+import org.apache.drill.exec.rpc.BasicClient;
+import org.apache.drill.exec.rpc.Response;
+import org.apache.drill.exec.rpc.RpcCommand;
+import org.apache.drill.exec.rpc.RpcConstants;
+import org.apache.drill.exec.rpc.RpcException;
+import org.apache.drill.exec.rpc.RpcOutcomeListener;
+import org.apache.drill.exec.work.batch.ControlMessageHandler;
+
+public class LocalControlConnectionManager extends ControlConnectionManager {
+ private static final org.slf4j.Logger logger =
org.slf4j.LoggerFactory.getLogger(LocalControlConnectionManager.class);
+
+ private final ControlConnectionConfig config;
+
+ public LocalControlConnectionManager(ControlConnectionConfig config,
DrillbitEndpoint localEndpoint) {
+super(localEndpoint, localEndpoint);
+this.config = config;
+ }
+
+ @Override
+ protected BasicClient getNewClient() {
+throw new UnsupportedOperationException("LocalControlConnectionManager
doesn't support creating a control client");
+ }
+
+ @Override
+ public void runCommand(RpcCommand cmd) {
+final int rpcType = cmd.getRpcType().getNumber();
+final ControlMessageHandler messageHandler = config.getMessageHandler();
+
+if (RpcConstants.EXTRA_DEBUGGING) {
+ logger.debug("Received bit com message of type {} over local connection
manager", rpcType);
+}
+
+switch (rpcType) {
+
+ case BitControl.RpcType.REQ_CANCEL_FRAGMENT_VALUE: {
+final ControlTunnel.SignalFragment signalFragment =
((ControlTunnel.SignalFragment) cmd);
+final RpcOutcomeListener outcomeListener =
signalFragment.getOutcomeListener();
+final Ack ackResponse =
messageHandler.cancelFragment(signalFragment.getMessage());
+outcomeListener.success(ackResponse, null);
+break;
+ }
+
+ case BitControl.RpcType.REQ_CUSTOM_VALUE: {
+final ByteBuf[] dataBodies;
+final RpcOutcomeListener outcomeListener;
+
+if (cmd instanceof ControlTunnel.CustomMessageSender) {
+ dataBodies =
((ControlTunnel.CustomMessageSender)cmd).getDataBodies();
+ outcomeListener =
((ControlTunnel.CustomMessageSender)cmd).getOutcomeListener();
+} else if (cmd instanceof ControlTunnel.SyncCustomMessageSender) {
+ dataBodies =
((ControlTunnel.SyncCustomMessageSender)cmd).getDataBodies();
+ outcomeListener =
((ControlTunnel.SyncCustomMessageSender)cmd).getOutcomeListener();
+} else {
+ throw new UnsupportedOperationException("Unknown Custom Type control
message received");
+}
+
+DrillBuf reqDrillBuff;
+try {
+ reqDrillBuff = convertToByteBuf(dataBodies);
+} catch (Exception ex) {
+ outcomeListener.failed(new RpcException("Failed to allocate memory
while sending request in " +
+"LocalControlConnectionManager#convertToByteBuff", ex));
+ return;
+} finally {
+ releaseByteBuf(dataBodies);
+}
+
+try {
+ BitControl.CustomMessage message = (BitControl.CustomMessage)
cmd.getMessage();
+ final Response response =
messageHandler.getHandlerRegistry().handle(message, reqDrillBuff);
+
[jira] [Commented] (DRILL-6255) Drillbit while sending control message to itself creates a connection instead of submitting locally
[
https://issues.apache.org/jira/browse/DRILL-6255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16475045#comment-16475045
]
ASF GitHub Bot commented on DRILL-6255:
---
sohami commented on a change in pull request #1253: DRILL-6255: Drillbit while
sending control message to itself creates …
URL: https://github.com/apache/drill/pull/1253#discussion_r188096939
##
File path:
exec/java-exec/src/test/java/org/apache/drill/exec/rpc/control/TestLocalControlConnectionManager.java
##
@@ -0,0 +1,304 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.rpc.control;
+
+import io.netty.buffer.ByteBuf;
+import org.apache.drill.exec.proto.BitControl;
+import org.apache.drill.exec.proto.CoordinationProtos.DrillbitEndpoint;
+import org.apache.drill.exec.proto.ExecProtos;
+import org.apache.drill.exec.proto.GeneralRPCProtos;
+import org.apache.drill.exec.proto.UserBitShared;
+import org.apache.drill.exec.rpc.Acks;
+import org.apache.drill.exec.rpc.RpcException;
+import org.apache.drill.exec.rpc.RpcOutcomeListener;
+import org.apache.drill.exec.work.batch.ControlMessageHandler;
+import org.junit.Before;
+import org.junit.BeforeClass;
+import org.junit.Test;
+
+import java.util.concurrent.CountDownLatch;
+
+import static junit.framework.TestCase.fail;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertTrue;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+
+public class TestLocalControlConnectionManager {
+
+ private static final DrillbitEndpoint localEndpoint =
DrillbitEndpoint.newBuilder()
+.setAddress("10.0.0.1")
+.setControlPort(31011)
+.setState(DrillbitEndpoint.State.STARTUP)
+.build();
+
+ private static ControlConnectionConfig mockConfig;
+
+ private static ControlMessageHandler mockHandler;
+
+ private static ControlTunnel controlTunnel;
+
+ private CountDownLatch latch;
+
+ private final RpcOutcomeListener outcomeListener =
+new RpcOutcomeListener() {
+@Override
+public void failed(RpcException ex) {
+ throw new IllegalStateException(ex);
+}
+
+@Override
+public void success(GeneralRPCProtos.Ack value, ByteBuf buffer) {
+ if (value.getOk()) {
+latch.countDown();
+ } else {
+throw new IllegalStateException("Negative Ack received");
+ }
+}
+
+@Override
+public void interrupted(InterruptedException e) {
+
Review comment:
This is test specific implementation and am not testing the interrupted
scenario hence no action.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Drillbit while sending control message to itself creates a connection instead
> of submitting locally
> ---
>
> Key: DRILL-6255
> URL: https://issues.apache.org/jira/browse/DRILL-6255
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Flow
>Affects Versions: 1.12.0
>Reporter: Sorabh Hamirwasia
>Assignee: Sorabh Hamirwasia
>Priority: Major
> Fix For: 1.14.0
>
>
> With the new shutdown feature introduced in 1.12, there is a state introduced
> in DrillbitEndpoint. Due to this the equality check happening
> [here|https://github.com/apache/drill/blob/master/exec/java-exec/src/main/java/org/apache/drill/exec/work/foreman/FragmentsRunner.java#L256]
> will result in false and hence the fragments supposed to be scheduled on
> Foreman will be treated as remote fragments and a connection will be created
> to schedule it. The equality check is false because localEndpoint state is
> STARTUP whereas state in assigned Drillbit is ONLINE.
> I guess now we should update the equality check to verify just for address
>
[jira] [Commented] (DRILL-6255) Drillbit while sending control message to itself creates a connection instead of submitting locally
[
https://issues.apache.org/jira/browse/DRILL-6255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16475041#comment-16475041
]
ASF GitHub Bot commented on DRILL-6255:
---
sohami commented on a change in pull request #1253: DRILL-6255: Drillbit while
sending control message to itself creates …
URL: https://github.com/apache/drill/pull/1253#discussion_r188090319
##
File path:
exec/java-exec/src/test/java/org/apache/drill/exec/rpc/control/TestLocalControlConnectionManager.java
##
@@ -0,0 +1,304 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.rpc.control;
+
+import io.netty.buffer.ByteBuf;
+import org.apache.drill.exec.proto.BitControl;
+import org.apache.drill.exec.proto.CoordinationProtos.DrillbitEndpoint;
+import org.apache.drill.exec.proto.ExecProtos;
+import org.apache.drill.exec.proto.GeneralRPCProtos;
+import org.apache.drill.exec.proto.UserBitShared;
+import org.apache.drill.exec.rpc.Acks;
+import org.apache.drill.exec.rpc.RpcException;
+import org.apache.drill.exec.rpc.RpcOutcomeListener;
+import org.apache.drill.exec.work.batch.ControlMessageHandler;
+import org.junit.Before;
+import org.junit.BeforeClass;
+import org.junit.Test;
+
+import java.util.concurrent.CountDownLatch;
+
+import static junit.framework.TestCase.fail;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertTrue;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+
+public class TestLocalControlConnectionManager {
+
+ private static final DrillbitEndpoint localEndpoint =
DrillbitEndpoint.newBuilder()
+.setAddress("10.0.0.1")
+.setControlPort(31011)
+.setState(DrillbitEndpoint.State.STARTUP)
+.build();
+
+ private static ControlConnectionConfig mockConfig;
+
+ private static ControlMessageHandler mockHandler;
+
+ private static ControlTunnel controlTunnel;
+
+ private CountDownLatch latch;
+
+ private final RpcOutcomeListener outcomeListener =
+new RpcOutcomeListener() {
+@Override
+public void failed(RpcException ex) {
+ throw new IllegalStateException(ex);
+}
+
+@Override
+public void success(GeneralRPCProtos.Ack value, ByteBuf buffer) {
+ if (value.getOk()) {
+latch.countDown();
+ } else {
+throw new IllegalStateException("Negative Ack received");
+ }
+}
+
+@Override
+public void interrupted(InterruptedException e) {
+
+}
+ };
+
+ @BeforeClass
+ public static void setup() {
+mockConfig = mock(ControlConnectionConfig.class);
+final ConnectionManagerRegistry registry = new
ConnectionManagerRegistry(mockConfig);
+registry.setLocalEndpoint(localEndpoint);
+ControlConnectionManager manager =
registry.getConnectionManager(localEndpoint);
+assertTrue(manager instanceof LocalControlConnectionManager);
+controlTunnel = new ControlTunnel(manager);
+ }
+
+ @Before
+ public void setupForTest() {
+mockHandler = mock(ControlMessageHandler.class);
+when(mockConfig.getMessageHandler()).thenReturn(mockHandler);
+ }
+
+ /**
+ * Verify that SendFragmentStatus is handled correctly using ControlTunnel
with LocalControlConnectionManager
+ */
+ @Test
+ public void testLocalSendFragmentStatus_Success() {
+final UserBitShared.QueryId mockQueryId =
UserBitShared.QueryId.getDefaultInstance();
+final UserBitShared.QueryProfile mockProfile =
UserBitShared.QueryProfile.getDefaultInstance();
+try {
Review comment:
`checkedGet()` can throw an exception and it's not expected in these test
cases. So in `catch` block we are failing the tests.
But made the changes to remove those along with using `ExpectedException`
rule in JUnit
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Drillbit while sending control message to itself creates a connection instead
> of submitting
[jira] [Commented] (DRILL-6255) Drillbit while sending control message to itself creates a connection instead of submitting locally
[
https://issues.apache.org/jira/browse/DRILL-6255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16475044#comment-16475044
]
ASF GitHub Bot commented on DRILL-6255:
---
sohami commented on a change in pull request #1253: DRILL-6255: Drillbit while
sending control message to itself creates …
URL: https://github.com/apache/drill/pull/1253#discussion_r188096235
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/rpc/control/LocalControlConnectionManager.java
##
@@ -0,0 +1,236 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.rpc.control;
+
+import io.netty.buffer.ByteBuf;
+import io.netty.buffer.DrillBuf;
+import org.apache.drill.exec.exception.OutOfMemoryException;
+import org.apache.drill.exec.proto.BitControl;
+import org.apache.drill.exec.proto.CoordinationProtos.DrillbitEndpoint;
+import org.apache.drill.exec.proto.GeneralRPCProtos.Ack;
+import org.apache.drill.exec.proto.UserBitShared;
+import org.apache.drill.exec.rpc.BasicClient;
+import org.apache.drill.exec.rpc.Response;
+import org.apache.drill.exec.rpc.RpcCommand;
+import org.apache.drill.exec.rpc.RpcConstants;
+import org.apache.drill.exec.rpc.RpcException;
+import org.apache.drill.exec.rpc.RpcOutcomeListener;
+import org.apache.drill.exec.work.batch.ControlMessageHandler;
+
+public class LocalControlConnectionManager extends ControlConnectionManager {
+ private static final org.slf4j.Logger logger =
org.slf4j.LoggerFactory.getLogger(LocalControlConnectionManager.class);
+
+ private final ControlConnectionConfig config;
+
+ public LocalControlConnectionManager(ControlConnectionConfig config,
DrillbitEndpoint localEndpoint) {
+super(localEndpoint, localEndpoint);
+this.config = config;
+ }
+
+ @Override
+ protected BasicClient getNewClient() {
+throw new UnsupportedOperationException("LocalControlConnectionManager
doesn't support creating a control client");
+ }
+
+ @Override
+ public void runCommand(RpcCommand cmd) {
+final int rpcType = cmd.getRpcType().getNumber();
+final ControlMessageHandler messageHandler = config.getMessageHandler();
+
+if (RpcConstants.EXTRA_DEBUGGING) {
+ logger.debug("Received bit com message of type {} over local connection
manager", rpcType);
+}
+
+switch (rpcType) {
+
+ case BitControl.RpcType.REQ_CANCEL_FRAGMENT_VALUE: {
+final ControlTunnel.SignalFragment signalFragment =
((ControlTunnel.SignalFragment) cmd);
+final RpcOutcomeListener outcomeListener =
signalFragment.getOutcomeListener();
+final Ack ackResponse =
messageHandler.cancelFragment(signalFragment.getMessage());
+outcomeListener.success(ackResponse, null);
+break;
+ }
+
+ case BitControl.RpcType.REQ_CUSTOM_VALUE: {
+final ByteBuf[] dataBodies;
+final RpcOutcomeListener outcomeListener;
+
+if (cmd instanceof ControlTunnel.CustomMessageSender) {
+ dataBodies =
((ControlTunnel.CustomMessageSender)cmd).getDataBodies();
+ outcomeListener =
((ControlTunnel.CustomMessageSender)cmd).getOutcomeListener();
+} else if (cmd instanceof ControlTunnel.SyncCustomMessageSender) {
+ dataBodies =
((ControlTunnel.SyncCustomMessageSender)cmd).getDataBodies();
+ outcomeListener =
((ControlTunnel.SyncCustomMessageSender)cmd).getOutcomeListener();
+} else {
+ throw new UnsupportedOperationException("Unknown Custom Type control
message received");
+}
+
+DrillBuf reqDrillBuff;
+try {
+ reqDrillBuff = convertToByteBuf(dataBodies);
+} catch (Exception ex) {
+ outcomeListener.failed(new RpcException("Failed to allocate memory
while sending request in " +
+"LocalControlConnectionManager#convertToByteBuff", ex));
+ return;
+} finally {
+ releaseByteBuf(dataBodies);
+}
+
+try {
+ BitControl.CustomMessage message = (BitControl.CustomMessage)
cmd.getMessage();
+ final Response response =
messageHandler.getHandlerRegistry().handle(message, reqDrillBuff);
+
[jira] [Commented] (DRILL-6255) Drillbit while sending control message to itself creates a connection instead of submitting locally
[
https://issues.apache.org/jira/browse/DRILL-6255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16475043#comment-16475043
]
ASF GitHub Bot commented on DRILL-6255:
---
sohami commented on a change in pull request #1253: DRILL-6255: Drillbit while
sending control message to itself creates …
URL: https://github.com/apache/drill/pull/1253#discussion_r188097134
##
File path: exec/rpc/src/main/java/org/apache/drill/exec/rpc/RpcCommand.java
##
@@ -17,10 +17,16 @@
*/
package org.apache.drill.exec.rpc;
+import com.google.protobuf.Internal.EnumLite;
import com.google.protobuf.MessageLite;
public interface RpcCommand
extends RpcConnectionHandler{
Review comment:
Fixed
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Drillbit while sending control message to itself creates a connection instead
> of submitting locally
> ---
>
> Key: DRILL-6255
> URL: https://issues.apache.org/jira/browse/DRILL-6255
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Flow
>Affects Versions: 1.12.0
>Reporter: Sorabh Hamirwasia
>Assignee: Sorabh Hamirwasia
>Priority: Major
> Fix For: 1.14.0
>
>
> With the new shutdown feature introduced in 1.12, there is a state introduced
> in DrillbitEndpoint. Due to this the equality check happening
> [here|https://github.com/apache/drill/blob/master/exec/java-exec/src/main/java/org/apache/drill/exec/work/foreman/FragmentsRunner.java#L256]
> will result in false and hence the fragments supposed to be scheduled on
> Foreman will be treated as remote fragments and a connection will be created
> to schedule it. The equality check is false because localEndpoint state is
> STARTUP whereas state in assigned Drillbit is ONLINE.
> I guess now we should update the equality check to verify just for address
> and control port to be same between assigned and local Drillbit endpoint. A
> test can be added for this based on _sys.memory_ table since that guarantees
> scheduling minor fragments on each Drillbit node.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-6410) Memory leak in Parquet Reader during cancellation
[
https://issues.apache.org/jira/browse/DRILL-6410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16474998#comment-16474998
]
ASF GitHub Bot commented on DRILL-6410:
---
parthchandra commented on a change in pull request #1257: DRILL-6410: Fixed
memory leak in flat Parquet reader
URL: https://github.com/apache/drill/pull/1257#discussion_r188122320
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/columnreaders/AsyncPageReader.java
##
@@ -297,11 +297,11 @@ private void waitForExecutionResult() throws
InterruptedException, ExecutionExce
while (asyncPageRead != null && !asyncPageRead.isEmpty()) {
try {
Future f = asyncPageRead.poll();
-if(!f.isDone() && !f.isCancelled()){
+if(!f.isDone() && !f.isCancelled()) {
f.cancel(true);
-} else {
- f.get(1, TimeUnit.MILLISECONDS);
}
+// The framework guarantees a blocking version of FutureTask
cancellation. At this time we are
Review comment:
My $0.02 is to tend towards what @vrozov is suggesting. It would be
preferable to have one reusable implementation that works for both use cases.
I don't know if @sachouche needs the localStats in PartitionTask, nor, as he
mentioned, do you want to pay a price for non cancellation paths.
Is it possible to come up with a unified approach here?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Memory leak in Parquet Reader during cancellation
> -
>
> Key: DRILL-6410
> URL: https://issues.apache.org/jira/browse/DRILL-6410
> Project: Apache Drill
> Issue Type: Bug
> Components: Storage - Parquet
>Reporter: salim achouche
>Assignee: salim achouche
>Priority: Major
>
> Occasionally, a memory leak is observed within the flat Parquet reader when
> query cancellation is invoked.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-6373) Refactor the Result Set Loader to prepare for Union, List support
[
https://issues.apache.org/jira/browse/DRILL-6373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16474985#comment-16474985
]
ASF GitHub Bot commented on DRILL-6373:
---
ppadma commented on issue #1244: DRILL-6373: Refactor Result Set Loader for
Union, List support
URL: https://github.com/apache/drill/pull/1244#issuecomment-388986616
@paul-rogers LGTM. +1.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Refactor the Result Set Loader to prepare for Union, List support
> -
>
> Key: DRILL-6373
> URL: https://issues.apache.org/jira/browse/DRILL-6373
> Project: Apache Drill
> Issue Type: Improvement
>Affects Versions: 1.13.0
>Reporter: Paul Rogers
>Assignee: Paul Rogers
>Priority: Major
> Fix For: 1.14.0
>
>
> As the next step in merging the "batch sizing" enhancements, refactor the
> {{ResultSetLoader}} and related classes to prepare for Union and List
> support. This fix follows the refactoring of the column accessors for the
> same purpose. Actual Union and List support is to follow in a separate PR.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-6411) Make batch memory sizing logs uniform across all operators
[ https://issues.apache.org/jira/browse/DRILL-6411?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16474974#comment-16474974 ] ASF GitHub Bot commented on DRILL-6411: --- ppadma commented on issue #1260: DRILL-6411: Make batch memory sizing logs uniform across all operators URL: https://github.com/apache/drill/pull/1260#issuecomment-388982642 @Ben-Zvi Boaz, please review. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Make batch memory sizing logs uniform across all operators > -- > > Key: DRILL-6411 > URL: https://issues.apache.org/jira/browse/DRILL-6411 > Project: Apache Drill > Issue Type: Bug >Affects Versions: 1.13.0 >Reporter: Padma Penumarthy >Assignee: Padma Penumarthy >Priority: Major > Fix For: 1.14.0 > > > Make batch memory sizing logs uniform across all operators so QA can parse > and verify easily. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-6411) Make batch memory sizing logs uniform across all operators
[ https://issues.apache.org/jira/browse/DRILL-6411?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16474842#comment-16474842 ] ASF GitHub Bot commented on DRILL-6411: --- ppadma opened a new pull request #1260: DRILL-6411: Make batch memory sizing logs uniform across all operators URL: https://github.com/apache/drill/pull/1260 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Make batch memory sizing logs uniform across all operators > -- > > Key: DRILL-6411 > URL: https://issues.apache.org/jira/browse/DRILL-6411 > Project: Apache Drill > Issue Type: Bug >Affects Versions: 1.13.0 >Reporter: Padma Penumarthy >Assignee: Padma Penumarthy >Priority: Major > Fix For: 1.14.0 > > > Make batch memory sizing logs uniform across all operators so QA can parse > and verify easily. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (DRILL-6353) Upgrade Parquet MR dependencies
[
https://issues.apache.org/jira/browse/DRILL-6353?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pritesh Maker updated DRILL-6353:
-
Reviewer: Parth Chandra
> Upgrade Parquet MR dependencies
> ---
>
> Key: DRILL-6353
> URL: https://issues.apache.org/jira/browse/DRILL-6353
> Project: Apache Drill
> Issue Type: Task
>Reporter: Vlad Rozov
>Assignee: Vlad Rozov
>Priority: Major
> Fix For: 1.14.0
>
>
> Upgrade from a custom build {{1.8.1-drill-r0}} to Apache release {{1.10.0}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (DRILL-6353) Upgrade Parquet MR dependencies
[
https://issues.apache.org/jira/browse/DRILL-6353?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pritesh Maker updated DRILL-6353:
-
Fix Version/s: 1.14.0
> Upgrade Parquet MR dependencies
> ---
>
> Key: DRILL-6353
> URL: https://issues.apache.org/jira/browse/DRILL-6353
> Project: Apache Drill
> Issue Type: Task
>Reporter: Vlad Rozov
>Assignee: Vlad Rozov
>Priority: Major
> Fix For: 1.14.0
>
>
> Upgrade from a custom build {{1.8.1-drill-r0}} to Apache release {{1.10.0}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (DRILL-6348) Unordered Receiver does not report its memory usage
[ https://issues.apache.org/jira/browse/DRILL-6348?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pritesh Maker updated DRILL-6348: - Labels: (was: ready-to-commit) > Unordered Receiver does not report its memory usage > --- > > Key: DRILL-6348 > URL: https://issues.apache.org/jira/browse/DRILL-6348 > Project: Apache Drill > Issue Type: Task > Components: Execution - Flow >Reporter: salim achouche >Assignee: salim achouche >Priority: Major > Fix For: 1.14.0 > > > The Drill Profile functionality doesn't show any memory usage for the > Unordered Receiver operator. This is problematic when analyzing OOM > conditions since we cannot account for all of a query memory usage. This Jira > is to fix memory reporting for the Unordered Receiver operator. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-6356) batch sizing for union all
[ https://issues.apache.org/jira/browse/DRILL-6356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16474829#comment-16474829 ] ASF GitHub Bot commented on DRILL-6356: --- ppadma commented on issue #1255: DRILL-6356: batch sizing for union all URL: https://github.com/apache/drill/pull/1255#issuecomment-388970217 @Ben-Zvi Boaz, can you help review this PR ? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > batch sizing for union all > -- > > Key: DRILL-6356 > URL: https://issues.apache.org/jira/browse/DRILL-6356 > Project: Apache Drill > Issue Type: Improvement > Components: Execution - Relational Operators >Affects Versions: 1.13.0 >Reporter: Padma Penumarthy >Assignee: Padma Penumarthy >Priority: Major > Fix For: 1.14.0 > > > batch sizing changes for union all operator -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-6414) Allow Case Insensitive Comparisons in Drill
[ https://issues.apache.org/jira/browse/DRILL-6414?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16474792#comment-16474792 ] Robert Hou commented on DRILL-6414: --- Would this be enabled by a session or system option? And keep the current behavior as the default? > Allow Case Insensitive Comparisons in Drill > --- > > Key: DRILL-6414 > URL: https://issues.apache.org/jira/browse/DRILL-6414 > Project: Apache Drill > Issue Type: Improvement >Reporter: Volodymyr Tkach >Assignee: Volodymyr Tkach >Priority: Major > > Lets take a case: > Table t1 with column c1 having a value 'abc' > For a query like below drill should treat the comparison as case insensitive > and return 1 row. > This is not an issue but we need to know if we can set case insensitivity at > system level. > select * from t1 where c1 = 'ABC' -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (DRILL-6386) Disallow Unused Imports In Checkstyle
[ https://issues.apache.org/jira/browse/DRILL-6386?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Timothy Farkas resolved DRILL-6386. --- Resolution: Fixed > Disallow Unused Imports In Checkstyle > - > > Key: DRILL-6386 > URL: https://issues.apache.org/jira/browse/DRILL-6386 > Project: Apache Drill > Issue Type: Improvement >Reporter: Timothy Farkas >Assignee: Timothy Farkas >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-6353) Upgrade Parquet MR dependencies
[
https://issues.apache.org/jira/browse/DRILL-6353?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16474762#comment-16474762
]
ASF GitHub Bot commented on DRILL-6353:
---
parthchandra commented on a change in pull request #1259: DRILL-6353: Upgrade
Parquet MR dependencies
URL: https://github.com/apache/drill/pull/1259#discussion_r188088771
##
File path: pom.xml
##
@@ -44,7 +44,7 @@
1.7.6
18.0
2
-1.8.1-drill-r0
+1.10.0
Review comment:
1.10? Is it safer to upgrade to 1.8.3 and then test out 1.9/1.10 before
upgrading to it?
Also, with this change the Hive parquet version is 1.8.2. I wonder what
impact that might have on compatibility?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Upgrade Parquet MR dependencies
> ---
>
> Key: DRILL-6353
> URL: https://issues.apache.org/jira/browse/DRILL-6353
> Project: Apache Drill
> Issue Type: Task
>Reporter: Vlad Rozov
>Assignee: Vlad Rozov
>Priority: Major
>
> Upgrade from a custom build {{1.8.1-drill-r0}} to Apache release {{1.10.0}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-6242) Output format for nested date, time, timestamp values in an object hierarchy
[
https://issues.apache.org/jira/browse/DRILL-6242?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16474694#comment-16474694
]
ASF GitHub Bot commented on DRILL-6242:
---
jiang-wu closed pull request #1184: DRILL-6242 - Use
java.sql.[Date|Time|Timestamp] classes to hold value…
URL: https://github.com/apache/drill/pull/1184
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git
a/contrib/storage-hive/core/src/test/java/org/apache/drill/exec/fn/hive/TestInbuiltHiveUDFs.java
b/contrib/storage-hive/core/src/test/java/org/apache/drill/exec/fn/hive/TestInbuiltHiveUDFs.java
index d4e0b5cb9c..3ae6aee337 100644
---
a/contrib/storage-hive/core/src/test/java/org/apache/drill/exec/fn/hive/TestInbuiltHiveUDFs.java
+++
b/contrib/storage-hive/core/src/test/java/org/apache/drill/exec/fn/hive/TestInbuiltHiveUDFs.java
@@ -17,22 +17,24 @@
*/
package org.apache.drill.exec.fn.hive;
-import com.google.common.collect.Lists;
+import java.sql.Timestamp;
+import java.util.List;
+
import org.apache.commons.lang3.tuple.Pair;
import org.apache.drill.categories.HiveStorageTest;
-import org.apache.drill.test.QueryTestUtil;
-import org.apache.drill.test.TestBuilder;
import org.apache.drill.categories.SlowTest;
import org.apache.drill.common.expression.SchemaPath;
import org.apache.drill.common.types.TypeProtos;
import org.apache.drill.exec.compile.ClassTransformer;
import org.apache.drill.exec.hive.HiveTestBase;
import org.apache.drill.exec.server.options.OptionValue;
+import org.apache.drill.test.QueryTestUtil;
+import org.apache.drill.test.TestBuilder;
import org.joda.time.DateTime;
import org.junit.Test;
import org.junit.experimental.categories.Category;
-import java.util.List;
+import com.google.common.collect.Lists;
@Category({SlowTest.class, HiveStorageTest.class})
public class TestInbuiltHiveUDFs extends HiveTestBase {
@@ -169,7 +171,7 @@ public void testFromUTCTimestamp() throws Exception {
.sqlQuery("select from_utc_timestamp('1970-01-01 08:00:00','PST') as
PST_TIMESTAMP from (VALUES(1))")
.unOrdered()
.baselineColumns("PST_TIMESTAMP")
-.baselineValues(DateTime.parse("1970-01-01T00:00:00.0"))
+.baselineValues(new
Timestamp(DateTime.parse("1970-01-01T00:00:00.0").getMillis()))
.go();
}
@@ -179,7 +181,7 @@ public void testToUTCTimestamp() throws Exception {
.sqlQuery("select to_utc_timestamp('1970-01-01 00:00:00','PST') as
UTC_TIMESTAMP from (VALUES(1))")
.unOrdered()
.baselineColumns("UTC_TIMESTAMP")
-.baselineValues(DateTime.parse("1970-01-01T08:00:00.0"))
+.baselineValues(new
Timestamp(DateTime.parse("1970-01-01T08:00:00.0").getMillis()))
.go();
}
diff --git
a/contrib/storage-hive/core/src/test/java/org/apache/drill/exec/hive/TestHiveStorage.java
b/contrib/storage-hive/core/src/test/java/org/apache/drill/exec/hive/TestHiveStorage.java
index 4da22b6a3a..5a97bf7ea9 100644
---
a/contrib/storage-hive/core/src/test/java/org/apache/drill/exec/hive/TestHiveStorage.java
+++
b/contrib/storage-hive/core/src/test/java/org/apache/drill/exec/hive/TestHiveStorage.java
@@ -17,8 +17,19 @@
*/
package org.apache.drill.exec.hive;
-import com.google.common.collect.ImmutableMap;
-import com.google.common.collect.Maps;
+import static org.hamcrest.CoreMatchers.containsString;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertNotNull;
+import static org.junit.Assert.assertThat;
+import static org.junit.Assert.assertTrue;
+
+import java.math.BigDecimal;
+import java.sql.Date;
+import java.sql.Timestamp;
+import java.util.List;
+import java.util.Map;
+
import org.apache.drill.PlanTestBase;
import org.apache.drill.categories.HiveStorageTest;
import org.apache.drill.categories.SlowTest;
@@ -28,24 +39,13 @@
import org.apache.drill.exec.proto.UserProtos;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.hive.common.type.HiveVarchar;
-import org.joda.time.DateTime;
import org.junit.AfterClass;
import org.junit.BeforeClass;
import org.junit.Test;
import org.junit.experimental.categories.Category;
-import java.math.BigDecimal;
-import java.sql.Date;
-import java.sql.Timestamp;
-import java.util.List;
-import java.util.Map;
-
-import static org.hamcrest.CoreMatchers.containsString;
-import static org.junit.Assert.assertEquals;
-import static org.junit.Assert.assertFalse;
-import static org.junit.Assert.assertNotNull;
-import static org.junit.Assert.assertThat;
-import static org.junit.Assert.assertTrue;
+import com.google.common.collect.ImmutableMap;
+import com.google.common.collect.Maps;
[jira] [Commented] (DRILL-6242) Output format for nested date, time, timestamp values in an object hierarchy
[
https://issues.apache.org/jira/browse/DRILL-6242?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16474693#comment-16474693
]
ASF GitHub Bot commented on DRILL-6242:
---
jiang-wu commented on issue #1184: DRILL-6242 - Use
java.sql.[Date|Time|Timestamp] classes to hold value…
URL: https://github.com/apache/drill/pull/1184#issuecomment-388934630
Subsumed by https://github.com/apache/drill/pull/1247 which uses
java.time.Local{Date|Time|DateTime} rather than java.sql.{Date|Time|Timestamp}.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Output format for nested date, time, timestamp values in an object hierarchy
>
>
> Key: DRILL-6242
> URL: https://issues.apache.org/jira/browse/DRILL-6242
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Data Types
>Affects Versions: 1.12.0
>Reporter: Jiang Wu
>Assignee: Jiang Wu
>Priority: Major
> Labels: ready-to-commit
> Fix For: 1.14.0
>
>
> Some storages (mapr db, mongo db, etc.) have hierarchical objects that
> contain nested fields of date, time, timestamp types. When a query returns
> these objects, the output format for the nested date, time, timestamp, are
> showing the internal object (org.joda.time.DateTime), rather than the logical
> data value.
> For example. Suppose in MongoDB, we have a single object that looks like
> this:
> {code:java}
> > db.test.findOne();
> {
> "_id" : ObjectId("5aa8487d470dd39a635a12f5"),
> "name" : "orange",
> "context" : {
> "date" : ISODate("2018-03-13T21:52:54.940Z"),
> "user" : "jack"
> }
> }
> {code}
> Then connect Drill to the above MongoDB storage, and run the following query
> within Drill:
> {code:java}
> > select t.context.`date`, t.context from test t;
> ++-+
> | EXPR$0 | context |
> ++-+
> | 2018-03-13 |
> {"date":{"dayOfYear":72,"year":2018,"dayOfMonth":13,"dayOfWeek":2,"era":1,"millisOfDay":78774940,"weekOfWeekyear":11,"weekyear":2018,"monthOfYear":3,"yearOfEra":2018,"yearOfCentury":18,"centuryOfEra":20,"millisOfSecond":940,"secondOfMinute":54,"secondOfDay":78774,"minuteOfHour":52,"minuteOfDay":1312,"hourOfDay":21,"zone":{"fixed":true,"id":"UTC"},"millis":1520977974940,"chronology":{"zone":{"fixed":true,"id":"UTC"}},"afterNow":false,"beforeNow":true,"equalNow":false},"user":"jack"}
> |
> {code}
> We can see that from the above output, when the date field is retrieved as a
> top level column, Drill outputs a logical date value. But when the same
> field is within an object hierarchy, Drill outputs the internal object used
> to hold the date value.
> The expected output is the same display for whether the date field is shown
> as a top level column or when it is within an object hierarchy:
> {code:java}
> > select t.context.`date`, t.context from test t;
> ++-+
> | EXPR$0 | context |
> ++-+
> | 2018-03-13 | {"date":"2018-03-13","user":"jack"} |
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-5270) Improve loading of profiles listing in the WebUI
[
https://issues.apache.org/jira/browse/DRILL-5270?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16474658#comment-16474658
]
ASF GitHub Bot commented on DRILL-5270:
---
kkhatua commented on a change in pull request #1250: DRILL-5270: Improve
loading of profiles listing in the WebUI
URL: https://github.com/apache/drill/pull/1250#discussion_r188064161
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/store/sys/store/LocalPersistentStore.java
##
@@ -1,220 +1,377 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.drill.exec.store.sys.store;
-
-import static org.apache.drill.exec.ExecConstants.DRILL_SYS_FILE_SUFFIX;
-
-import java.io.File;
-import java.io.IOException;
-import java.io.InputStream;
-import java.io.OutputStream;
-import java.util.Collections;
-import java.util.Iterator;
-import java.util.List;
-import java.util.Map;
-import java.util.Map.Entry;
-
-import javax.annotation.Nullable;
-
-import org.apache.commons.io.IOUtils;
-import org.apache.drill.common.collections.ImmutableEntry;
-import org.apache.drill.common.config.DrillConfig;
-import org.apache.drill.exec.store.dfs.DrillFileSystem;
-import org.apache.drill.exec.util.DrillFileSystemUtil;
-import org.apache.drill.exec.store.sys.BasePersistentStore;
-import org.apache.drill.exec.store.sys.PersistentStoreConfig;
-import org.apache.drill.exec.store.sys.PersistentStoreMode;
-import org.apache.hadoop.conf.Configuration;
-import org.apache.hadoop.fs.FileStatus;
-import org.apache.hadoop.fs.FileSystem;
-import org.apache.hadoop.fs.Path;
-
-import com.google.common.base.Function;
-import com.google.common.base.Preconditions;
-import com.google.common.collect.Iterables;
-import com.google.common.collect.Lists;
-import org.apache.hadoop.fs.PathFilter;
-import org.slf4j.Logger;
-import org.slf4j.LoggerFactory;
-
-public class LocalPersistentStore extends BasePersistentStore {
- private static final Logger logger =
LoggerFactory.getLogger(LocalPersistentStore.class);
-
- private final Path basePath;
- private final PersistentStoreConfig config;
- private final DrillFileSystem fs;
-
- public LocalPersistentStore(DrillFileSystem fs, Path base,
PersistentStoreConfig config) {
-this.basePath = new Path(base, config.getName());
-this.config = config;
-this.fs = fs;
-try {
- mkdirs(getBasePath());
-} catch (IOException e) {
- throw new RuntimeException("Failure setting pstore configuration path.");
-}
- }
-
- protected Path getBasePath() {
-return basePath;
- }
-
- @Override
- public PersistentStoreMode getMode() {
-return PersistentStoreMode.PERSISTENT;
- }
-
- private void mkdirs(Path path) throws IOException {
-fs.mkdirs(path);
- }
-
- public static Path getLogDir() {
-String drillLogDir = System.getenv("DRILL_LOG_DIR");
-if (drillLogDir == null) {
- drillLogDir = System.getProperty("drill.log.dir");
-}
-if (drillLogDir == null) {
- drillLogDir = "/var/log/drill";
-}
-return new Path(new File(drillLogDir).getAbsoluteFile().toURI());
- }
-
- public static DrillFileSystem getFileSystem(DrillConfig config, Path root)
throws IOException {
-Path blobRoot = root == null ? getLogDir() : root;
-Configuration fsConf = new Configuration();
-if (blobRoot.toUri().getScheme() != null) {
- fsConf.set(FileSystem.FS_DEFAULT_NAME_KEY, blobRoot.toUri().toString());
-}
-
-
-DrillFileSystem fs = new DrillFileSystem(fsConf);
-fs.mkdirs(blobRoot);
-return fs;
- }
-
- @Override
- public Iterator> getRange(int skip, int take) {
-try {
- // list only files with sys file suffix
- PathFilter sysFileSuffixFilter = new PathFilter() {
-@Override
-public boolean accept(Path path) {
- return path.getName().endsWith(DRILL_SYS_FILE_SUFFIX);
-}
- };
-
- List fileStatuses = DrillFileSystemUtil.listFiles(fs,
basePath, false, sysFileSuffixFilter);
- if (fileStatuses.isEmpty()) {
-return Collections.emptyIterator();
- }
-
- List files = Lists.newArrayList();
- for (FileStatus st
[jira] [Commented] (DRILL-5270) Improve loading of profiles listing in the WebUI
[
https://issues.apache.org/jira/browse/DRILL-5270?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16474652#comment-16474652
]
ASF GitHub Bot commented on DRILL-5270:
---
kkhatua commented on a change in pull request #1250: DRILL-5270: Improve
loading of profiles listing in the WebUI
URL: https://github.com/apache/drill/pull/1250#discussion_r188063691
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/store/sys/store/LocalPersistentStore.java
##
@@ -1,220 +1,377 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.drill.exec.store.sys.store;
-
-import static org.apache.drill.exec.ExecConstants.DRILL_SYS_FILE_SUFFIX;
-
-import java.io.File;
-import java.io.IOException;
-import java.io.InputStream;
-import java.io.OutputStream;
-import java.util.Collections;
-import java.util.Iterator;
-import java.util.List;
-import java.util.Map;
-import java.util.Map.Entry;
-
-import javax.annotation.Nullable;
-
-import org.apache.commons.io.IOUtils;
-import org.apache.drill.common.collections.ImmutableEntry;
-import org.apache.drill.common.config.DrillConfig;
-import org.apache.drill.exec.store.dfs.DrillFileSystem;
-import org.apache.drill.exec.util.DrillFileSystemUtil;
-import org.apache.drill.exec.store.sys.BasePersistentStore;
-import org.apache.drill.exec.store.sys.PersistentStoreConfig;
-import org.apache.drill.exec.store.sys.PersistentStoreMode;
-import org.apache.hadoop.conf.Configuration;
-import org.apache.hadoop.fs.FileStatus;
-import org.apache.hadoop.fs.FileSystem;
-import org.apache.hadoop.fs.Path;
-
-import com.google.common.base.Function;
-import com.google.common.base.Preconditions;
-import com.google.common.collect.Iterables;
-import com.google.common.collect.Lists;
-import org.apache.hadoop.fs.PathFilter;
-import org.slf4j.Logger;
-import org.slf4j.LoggerFactory;
-
-public class LocalPersistentStore extends BasePersistentStore {
- private static final Logger logger =
LoggerFactory.getLogger(LocalPersistentStore.class);
-
- private final Path basePath;
- private final PersistentStoreConfig config;
- private final DrillFileSystem fs;
-
- public LocalPersistentStore(DrillFileSystem fs, Path base,
PersistentStoreConfig config) {
-this.basePath = new Path(base, config.getName());
-this.config = config;
-this.fs = fs;
-try {
- mkdirs(getBasePath());
-} catch (IOException e) {
- throw new RuntimeException("Failure setting pstore configuration path.");
-}
- }
-
- protected Path getBasePath() {
-return basePath;
- }
-
- @Override
- public PersistentStoreMode getMode() {
-return PersistentStoreMode.PERSISTENT;
- }
-
- private void mkdirs(Path path) throws IOException {
-fs.mkdirs(path);
- }
-
- public static Path getLogDir() {
-String drillLogDir = System.getenv("DRILL_LOG_DIR");
-if (drillLogDir == null) {
- drillLogDir = System.getProperty("drill.log.dir");
-}
-if (drillLogDir == null) {
- drillLogDir = "/var/log/drill";
-}
-return new Path(new File(drillLogDir).getAbsoluteFile().toURI());
- }
-
- public static DrillFileSystem getFileSystem(DrillConfig config, Path root)
throws IOException {
-Path blobRoot = root == null ? getLogDir() : root;
-Configuration fsConf = new Configuration();
-if (blobRoot.toUri().getScheme() != null) {
- fsConf.set(FileSystem.FS_DEFAULT_NAME_KEY, blobRoot.toUri().toString());
-}
-
-
-DrillFileSystem fs = new DrillFileSystem(fsConf);
-fs.mkdirs(blobRoot);
-return fs;
- }
-
- @Override
- public Iterator> getRange(int skip, int take) {
-try {
- // list only files with sys file suffix
- PathFilter sysFileSuffixFilter = new PathFilter() {
-@Override
-public boolean accept(Path path) {
- return path.getName().endsWith(DRILL_SYS_FILE_SUFFIX);
-}
- };
-
- List fileStatuses = DrillFileSystemUtil.listFiles(fs,
basePath, false, sysFileSuffixFilter);
- if (fileStatuses.isEmpty()) {
-return Collections.emptyIterator();
- }
-
- List files = Lists.newArrayList();
- for (FileStatus st
[jira] [Commented] (DRILL-6410) Memory leak in Parquet Reader during cancellation
[
https://issues.apache.org/jira/browse/DRILL-6410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16474646#comment-16474646
]
ASF GitHub Bot commented on DRILL-6410:
---
vrozov commented on a change in pull request #1257: DRILL-6410: Fixed memory
leak in flat Parquet reader
URL: https://github.com/apache/drill/pull/1257#discussion_r188062899
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/columnreaders/AsyncPageReader.java
##
@@ -297,11 +297,11 @@ private void waitForExecutionResult() throws
InterruptedException, ExecutionExce
while (asyncPageRead != null && !asyncPageRead.isEmpty()) {
try {
Future f = asyncPageRead.poll();
-if(!f.isDone() && !f.isCancelled()){
+if(!f.isDone() && !f.isCancelled()) {
f.cancel(true);
-} else {
- f.get(1, TimeUnit.MILLISECONDS);
}
+// The framework guarantees a blocking version of FutureTask
cancellation. At this time we are
Review comment:
I suggest to agree on a unified approach. Having two different
implementations for the same functionality is, IMO, undesirable.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Memory leak in Parquet Reader during cancellation
> -
>
> Key: DRILL-6410
> URL: https://issues.apache.org/jira/browse/DRILL-6410
> Project: Apache Drill
> Issue Type: Bug
> Components: Storage - Parquet
>Reporter: salim achouche
>Assignee: salim achouche
>Priority: Major
>
> Occasionally, a memory leak is observed within the flat Parquet reader when
> query cancellation is invoked.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-6410) Memory leak in Parquet Reader during cancellation
[
https://issues.apache.org/jira/browse/DRILL-6410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16474618#comment-16474618
]
ASF GitHub Bot commented on DRILL-6410:
---
sachouche commented on a change in pull request #1257: DRILL-6410: Fixed memory
leak in flat Parquet reader
URL: https://github.com/apache/drill/pull/1257#discussion_r188058597
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/columnreaders/AsyncPageReader.java
##
@@ -297,11 +297,11 @@ private void waitForExecutionResult() throws
InterruptedException, ExecutionExce
while (asyncPageRead != null && !asyncPageRead.isEmpty()) {
try {
Future f = asyncPageRead.poll();
-if(!f.isDone() && !f.isCancelled()){
+if(!f.isDone() && !f.isCancelled()) {
f.cancel(true);
-} else {
- f.get(1, TimeUnit.MILLISECONDS);
}
+// The framework guarantees a blocking version of FutureTask
cancellation. At this time we are
Review comment:
@vrozov
I have looked at that fix but decided against it as I felt the same logic
could be achieved with a simpler implementation especially that the performance
penalty is only incurred during the cancellation process.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Memory leak in Parquet Reader during cancellation
> -
>
> Key: DRILL-6410
> URL: https://issues.apache.org/jira/browse/DRILL-6410
> Project: Apache Drill
> Issue Type: Bug
> Components: Storage - Parquet
>Reporter: salim achouche
>Assignee: salim achouche
>Priority: Major
>
> Occasionally, a memory leak is observed within the flat Parquet reader when
> query cancellation is invoked.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-6410) Memory leak in Parquet Reader during cancellation
[ https://issues.apache.org/jira/browse/DRILL-6410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16474619#comment-16474619 ] ASF GitHub Bot commented on DRILL-6410: --- sachouche commented on a change in pull request #1257: DRILL-6410: Fixed memory leak in flat Parquet reader URL: https://github.com/apache/drill/pull/1257#discussion_r188058654 ## File path: exec/java-exec/src/main/java/org/apache/drill/exec/server/BootStrapContext.java ## @@ -43,10 +41,15 @@ import org.apache.drill.exec.rpc.security.AuthenticatorProvider; import org.apache.drill.exec.rpc.security.AuthenticatorProviderImpl; import org.apache.drill.exec.server.options.OptionDefinition; +import org.apache.drill.exec.util.ExecutorServiceUtil; import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.fs.CommonConfigurationKeys; import org.apache.hadoop.security.UserGroupInformation; +import com.codahale.metrics.MetricRegistry; Review comment: Sure. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Memory leak in Parquet Reader during cancellation > - > > Key: DRILL-6410 > URL: https://issues.apache.org/jira/browse/DRILL-6410 > Project: Apache Drill > Issue Type: Bug > Components: Storage - Parquet >Reporter: salim achouche >Assignee: salim achouche >Priority: Major > > Occasionally, a memory leak is observed within the flat Parquet reader when > query cancellation is invoked. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-6353) Upgrade Parquet MR dependencies
[
https://issues.apache.org/jira/browse/DRILL-6353?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16474605#comment-16474605
]
ASF GitHub Bot commented on DRILL-6353:
---
vrozov opened a new pull request #1259: DRILL-6353: Upgrade Parquet MR
dependencies
URL: https://github.com/apache/drill/pull/1259
@parthchandra Please review
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Upgrade Parquet MR dependencies
> ---
>
> Key: DRILL-6353
> URL: https://issues.apache.org/jira/browse/DRILL-6353
> Project: Apache Drill
> Issue Type: Task
>Reporter: Vlad Rozov
>Assignee: Vlad Rozov
>Priority: Major
>
> Upgrade from a custom build {{1.8.1-drill-r0}} to Apache release {{1.10.0}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-6410) Memory leak in Parquet Reader during cancellation
[ https://issues.apache.org/jira/browse/DRILL-6410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16474604#comment-16474604 ] ASF GitHub Bot commented on DRILL-6410: --- vrozov commented on a change in pull request #1257: DRILL-6410: Fixed memory leak in flat Parquet reader URL: https://github.com/apache/drill/pull/1257#discussion_r188056534 ## File path: exec/java-exec/src/main/java/org/apache/drill/exec/server/BootStrapContext.java ## @@ -43,10 +41,15 @@ import org.apache.drill.exec.rpc.security.AuthenticatorProvider; import org.apache.drill.exec.rpc.security.AuthenticatorProviderImpl; import org.apache.drill.exec.server.options.OptionDefinition; +import org.apache.drill.exec.util.ExecutorServiceUtil; import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.fs.CommonConfigurationKeys; import org.apache.hadoop.security.UserGroupInformation; +import com.codahale.metrics.MetricRegistry; Review comment: Please avoid import order change. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Memory leak in Parquet Reader during cancellation > - > > Key: DRILL-6410 > URL: https://issues.apache.org/jira/browse/DRILL-6410 > Project: Apache Drill > Issue Type: Bug > Components: Storage - Parquet >Reporter: salim achouche >Assignee: salim achouche >Priority: Major > > Occasionally, a memory leak is observed within the flat Parquet reader when > query cancellation is invoked. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-6410) Memory leak in Parquet Reader during cancellation
[
https://issues.apache.org/jira/browse/DRILL-6410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16474603#comment-16474603
]
ASF GitHub Bot commented on DRILL-6410:
---
vrozov commented on a change in pull request #1257: DRILL-6410: Fixed memory
leak in flat Parquet reader
URL: https://github.com/apache/drill/pull/1257#discussion_r188056305
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/columnreaders/AsyncPageReader.java
##
@@ -297,11 +297,11 @@ private void waitForExecutionResult() throws
InterruptedException, ExecutionExce
while (asyncPageRead != null && !asyncPageRead.isEmpty()) {
try {
Future f = asyncPageRead.poll();
-if(!f.isDone() && !f.isCancelled()){
+if(!f.isDone() && !f.isCancelled()) {
f.cancel(true);
-} else {
- f.get(1, TimeUnit.MILLISECONDS);
}
+// The framework guarantees a blocking version of FutureTask
cancellation. At this time we are
Review comment:
Please see `PartitionerTask` in `PartitionerDecorator`. The design goal for
the `PartionerTask` is exactly the same, make sure that the main thread will
not proceed till all tasks are either done or canceled. Currently,
`PartitionerTask` is a private class, but it should be possible to generalize
it.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Memory leak in Parquet Reader during cancellation
> -
>
> Key: DRILL-6410
> URL: https://issues.apache.org/jira/browse/DRILL-6410
> Project: Apache Drill
> Issue Type: Bug
> Components: Storage - Parquet
>Reporter: salim achouche
>Assignee: salim achouche
>Priority: Major
>
> Occasionally, a memory leak is observed within the flat Parquet reader when
> query cancellation is invoked.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-6414) Allow Case Insensitive Comparisons in Drill
[
https://issues.apache.org/jira/browse/DRILL-6414?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16474527#comment-16474527
]
Paul Rogers commented on DRILL-6414:
This is an important issue: for formats such as JSON, both {{a}} and {{A}} can
be column names:
{noformat}
{a: 10, A: "30"}
{noformat}
Why anyone would do such a twisted thing is beyond me, but the point is, it is
legal JSON.
The fun starts as this gets passed up the chain. When do we do case sensitive
vs. insensitive. How about the following:
{noformat}
SELECT A FROM `myFile.json`
{noformat}
Sure, that can be case sensitive. How about this:
{noformat}
SELECT CAST(a AS DECIMAL) AS a, CAST(A AS INT) AS A FROM `myFile.json`
{noformat}
We've just defined columns in Drill itself. Should those two columns have case
sensitive or case insensitive names? Should they follow the SQL rules (because
they were defined in SQL) or JSON rules (because I'm just using Drill's way of
expressing a schema)?
Then, in JDBC, when do we follow the case insensitive rule vs. the case
sensitive rule? How does the JDBC developer know which rule will apply?
Might we instead detect the case sensitivity issue in a JSON file, say, and
generate two columns rather than simply try to assign both {{a}} and {{A}} to
the same vector? Maybe call the first one "a" and the second one "A$2" or some
such. The user can then use Drill's schema rewrite mechanism (AKA casting and
projection) to create Drill-friendly names:
{noformat}
SELECT a AS littleA, A$2 AS bigA FROM `myFile.json`
{noformat}
The point is, we'll need to be creative to solve this one, we can't do it as a
simple "use a case sensitive comparison" bug fix.
> Allow Case Insensitive Comparisons in Drill
> ---
>
> Key: DRILL-6414
> URL: https://issues.apache.org/jira/browse/DRILL-6414
> Project: Apache Drill
> Issue Type: Improvement
>Reporter: Volodymyr Tkach
>Assignee: Volodymyr Tkach
>Priority: Major
>
> Lets take a case:
> Table t1 with column c1 having a value 'abc'
> For a query like below drill should treat the comparison as case insensitive
> and return 1 row.
> This is not an issue but we need to know if we can set case insensitivity at
> system level.
> select * from t1 where c1 = 'ABC'
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (DRILL-6348) Unordered Receiver does not report its memory usage
[ https://issues.apache.org/jira/browse/DRILL-6348?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pritesh Maker updated DRILL-6348: - Labels: ready-to-commit (was: ) > Unordered Receiver does not report its memory usage > --- > > Key: DRILL-6348 > URL: https://issues.apache.org/jira/browse/DRILL-6348 > Project: Apache Drill > Issue Type: Task > Components: Execution - Flow >Reporter: salim achouche >Assignee: salim achouche >Priority: Major > Labels: ready-to-commit > Fix For: 1.14.0 > > > The Drill Profile functionality doesn't show any memory usage for the > Unordered Receiver operator. This is problematic when analyzing OOM > conditions since we cannot account for all of a query memory usage. This Jira > is to fix memory reporting for the Unordered Receiver operator. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (DRILL-6348) Unordered Receiver does not report its memory usage
[ https://issues.apache.org/jira/browse/DRILL-6348?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pritesh Maker updated DRILL-6348: - Reviewer: Vlad Rozov (was: Parth Chandra) > Unordered Receiver does not report its memory usage > --- > > Key: DRILL-6348 > URL: https://issues.apache.org/jira/browse/DRILL-6348 > Project: Apache Drill > Issue Type: Task > Components: Execution - Flow >Reporter: salim achouche >Assignee: salim achouche >Priority: Major > Labels: ready-to-commit > Fix For: 1.14.0 > > > The Drill Profile functionality doesn't show any memory usage for the > Unordered Receiver operator. This is problematic when analyzing OOM > conditions since we cannot account for all of a query memory usage. This Jira > is to fix memory reporting for the Unordered Receiver operator. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (DRILL-6414) Allow Case Insensitive Comparisons in Drill
Volodymyr Tkach created DRILL-6414: -- Summary: Allow Case Insensitive Comparisons in Drill Key: DRILL-6414 URL: https://issues.apache.org/jira/browse/DRILL-6414 Project: Apache Drill Issue Type: Improvement Reporter: Volodymyr Tkach Assignee: Volodymyr Tkach Lets take a case: Table t1 with column c1 having a value 'abc' For a query like below drill should treat the comparison as case insensitive and return 1 row. This is not an issue but we need to know if we can set case insensitivity at system level. select * from t1 where c1 = 'ABC' -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-6321) Lateral Join: Planning changes - enable submitting physical plan
[ https://issues.apache.org/jira/browse/DRILL-6321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16474402#comment-16474402 ] ASF GitHub Bot commented on DRILL-6321: --- chunhui-shi opened a new pull request #1258: DRILL-6321: Lateral Join and Unnest - initial implementation for parser and planning URL: https://github.com/apache/drill/pull/1258 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Lateral Join: Planning changes - enable submitting physical plan > > > Key: DRILL-6321 > URL: https://issues.apache.org/jira/browse/DRILL-6321 > Project: Apache Drill > Issue Type: Task >Reporter: Parth Chandra >Assignee: Chunhui Shi >Priority: Major > Labels: ready-to-commit > Fix For: 1.14.0 > > > Implement changes to enable submitting a physical plan containing lateral and > unnest. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (DRILL-6336) Inconsistent method name.
[
https://issues.apache.org/jira/browse/DRILL-6336?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arina Ielchiieva closed DRILL-6336.
---
Resolution: Not A Bug
> Inconsistent method name.
> -
>
> Key: DRILL-6336
> URL: https://issues.apache.org/jira/browse/DRILL-6336
> Project: Apache Drill
> Issue Type: Improvement
>Reporter: KuiLIU
>Priority: Major
> Attachments: rename-method.patch
>
>
> The following method is named "append", but its body code is an method
> invocation "writer.print( s )". The method should be named "print".
> {code:java}
> public DebugStringBuilder append( String s ) {
> writer.print( s );
> return this;
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-6336) Inconsistent method name.
[
https://issues.apache.org/jira/browse/DRILL-6336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16474353#comment-16474353
]
ASF GitHub Bot commented on DRILL-6336:
---
BruceKuiLiu commented on issue #1235: DRILL-6336: Inconsistent method name.
URL: https://github.com/apache/drill/pull/1235#issuecomment-388856486
Agree.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Inconsistent method name.
> -
>
> Key: DRILL-6336
> URL: https://issues.apache.org/jira/browse/DRILL-6336
> Project: Apache Drill
> Issue Type: Improvement
>Reporter: KuiLIU
>Priority: Major
> Attachments: rename-method.patch
>
>
> The following method is named "append", but its body code is an method
> invocation "writer.print( s )". The method should be named "print".
> {code:java}
> public DebugStringBuilder append( String s ) {
> writer.print( s );
> return this;
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-6336) Inconsistent method name.
[
https://issues.apache.org/jira/browse/DRILL-6336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16474354#comment-16474354
]
ASF GitHub Bot commented on DRILL-6336:
---
BruceKuiLiu closed pull request #1235: DRILL-6336: Inconsistent method name.
URL: https://github.com/apache/drill/pull/1235
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git
a/exec/java-exec/src/main/java/org/apache/drill/exec/expr/ClassGenerator.java
b/exec/java-exec/src/main/java/org/apache/drill/exec/expr/ClassGenerator.java
index d4ee60ac2f..33b2963263 100644
---
a/exec/java-exec/src/main/java/org/apache/drill/exec/expr/ClassGenerator.java
+++
b/exec/java-exec/src/main/java/org/apache/drill/exec/expr/ClassGenerator.java
@@ -835,18 +835,18 @@ public boolean isRepeated() {
public String toString() {
DebugStringBuilder buf = new DebugStringBuilder(this);
if (isConstant()) {
-buf.append("const ");
+buf.print("const ");
}
- buf.append(holder.type().fullName())
-.append(" ")
-.append(holder.name())
-.append(", ")
-.append(type.getMode().name())
-.append(" ")
-.append(type.getMinorType().name())
-.append(", ");
+ buf.print(holder.type().fullName())
+.print(" ")
+.print(holder.name())
+.print(", ")
+.print(type.getMode().name())
+.print(" ")
+.print(type.getMinorType().name())
+.print(", ");
holder.generate(buf.formatter());
- buf.append(", ");
+ buf.print(", ");
value.generate(buf.formatter());
return buf.toString();
}
diff --git
a/exec/java-exec/src/main/java/org/apache/drill/exec/expr/DebugStringBuilder.java
b/exec/java-exec/src/main/java/org/apache/drill/exec/expr/DebugStringBuilder.java
index a9b0f229b0..01a901aeea 100644
---
a/exec/java-exec/src/main/java/org/apache/drill/exec/expr/DebugStringBuilder.java
+++
b/exec/java-exec/src/main/java/org/apache/drill/exec/expr/DebugStringBuilder.java
@@ -44,7 +44,7 @@ public DebugStringBuilder( Object obj ) {
fmt = new JFormatter( writer );
}
- public DebugStringBuilder append( String s ) {
+ public DebugStringBuilder print( String s ) {
writer.print( s );
return this;
}
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Inconsistent method name.
> -
>
> Key: DRILL-6336
> URL: https://issues.apache.org/jira/browse/DRILL-6336
> Project: Apache Drill
> Issue Type: Improvement
>Reporter: KuiLIU
>Priority: Major
> Attachments: rename-method.patch
>
>
> The following method is named "append", but its body code is an method
> invocation "writer.print( s )". The method should be named "print".
> {code:java}
> public DebugStringBuilder append( String s ) {
> writer.print( s );
> return this;
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-6336) Inconsistent method name.
[
https://issues.apache.org/jira/browse/DRILL-6336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16474352#comment-16474352
]
ASF GitHub Bot commented on DRILL-6336:
---
arina-ielchiieva commented on issue #1235: DRILL-6336: Inconsistent method name.
URL: https://github.com/apache/drill/pull/1235#issuecomment-388856337
@BruceKuiLiu so we can close the PR and Jira?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Inconsistent method name.
> -
>
> Key: DRILL-6336
> URL: https://issues.apache.org/jira/browse/DRILL-6336
> Project: Apache Drill
> Issue Type: Improvement
>Reporter: KuiLIU
>Priority: Major
> Attachments: rename-method.patch
>
>
> The following method is named "append", but its body code is an method
> invocation "writer.print( s )". The method should be named "print".
> {code:java}
> public DebugStringBuilder append( String s ) {
> writer.print( s );
> return this;
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-6336) Inconsistent method name.
[
https://issues.apache.org/jira/browse/DRILL-6336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16474347#comment-16474347
]
ASF GitHub Bot commented on DRILL-6336:
---
BruceKuiLiu commented on issue #1235: DRILL-6336: Inconsistent method name.
URL: https://github.com/apache/drill/pull/1235#issuecomment-388855662
The target of this class is to build a debug string for an object.
Change the class name may break this target.
I agree with @vrozov and @paul-rogers. Considering the class is a
_builder_, `append()` should be better than `print()`.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Inconsistent method name.
> -
>
> Key: DRILL-6336
> URL: https://issues.apache.org/jira/browse/DRILL-6336
> Project: Apache Drill
> Issue Type: Improvement
>Reporter: KuiLIU
>Priority: Major
> Attachments: rename-method.patch
>
>
> The following method is named "append", but its body code is an method
> invocation "writer.print( s )". The method should be named "print".
> {code:java}
> public DebugStringBuilder append( String s ) {
> writer.print( s );
> return this;
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-6336) Inconsistent method name.
[
https://issues.apache.org/jira/browse/DRILL-6336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16474338#comment-16474338
]
ASF GitHub Bot commented on DRILL-6336:
---
BruceKuiLiu opened a new pull request #1235: DRILL-6336: Inconsistent method
name.
URL: https://github.com/apache/drill/pull/1235
Change the method name "append" to "print" since its body code is an method
invocation "writer.print( s )". The method named "print" should be more clear.
The method name "append" is labeled as "@Deprecated" in case of breaking
backward compatibility.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Inconsistent method name.
> -
>
> Key: DRILL-6336
> URL: https://issues.apache.org/jira/browse/DRILL-6336
> Project: Apache Drill
> Issue Type: Improvement
>Reporter: KuiLIU
>Priority: Major
> Attachments: rename-method.patch
>
>
> The following method is named "append", but its body code is an method
> invocation "writer.print( s )". The method should be named "print".
> {code:java}
> public DebugStringBuilder append( String s ) {
> writer.print( s );
> return this;
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-6336) Inconsistent method name.
[
https://issues.apache.org/jira/browse/DRILL-6336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16474337#comment-16474337
]
ASF GitHub Bot commented on DRILL-6336:
---
BruceKuiLiu closed pull request #1235: DRILL-6336: Inconsistent method name.
URL: https://github.com/apache/drill/pull/1235
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git
a/exec/java-exec/src/main/java/org/apache/drill/exec/expr/ClassGenerator.java
b/exec/java-exec/src/main/java/org/apache/drill/exec/expr/ClassGenerator.java
index d4ee60ac2f..33b2963263 100644
---
a/exec/java-exec/src/main/java/org/apache/drill/exec/expr/ClassGenerator.java
+++
b/exec/java-exec/src/main/java/org/apache/drill/exec/expr/ClassGenerator.java
@@ -835,18 +835,18 @@ public boolean isRepeated() {
public String toString() {
DebugStringBuilder buf = new DebugStringBuilder(this);
if (isConstant()) {
-buf.append("const ");
+buf.print("const ");
}
- buf.append(holder.type().fullName())
-.append(" ")
-.append(holder.name())
-.append(", ")
-.append(type.getMode().name())
-.append(" ")
-.append(type.getMinorType().name())
-.append(", ");
+ buf.print(holder.type().fullName())
+.print(" ")
+.print(holder.name())
+.print(", ")
+.print(type.getMode().name())
+.print(" ")
+.print(type.getMinorType().name())
+.print(", ");
holder.generate(buf.formatter());
- buf.append(", ");
+ buf.print(", ");
value.generate(buf.formatter());
return buf.toString();
}
diff --git
a/exec/java-exec/src/main/java/org/apache/drill/exec/expr/DebugStringBuilder.java
b/exec/java-exec/src/main/java/org/apache/drill/exec/expr/DebugStringBuilder.java
index a9b0f229b0..01a901aeea 100644
---
a/exec/java-exec/src/main/java/org/apache/drill/exec/expr/DebugStringBuilder.java
+++
b/exec/java-exec/src/main/java/org/apache/drill/exec/expr/DebugStringBuilder.java
@@ -44,7 +44,7 @@ public DebugStringBuilder( Object obj ) {
fmt = new JFormatter( writer );
}
- public DebugStringBuilder append( String s ) {
+ public DebugStringBuilder print( String s ) {
writer.print( s );
return this;
}
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Inconsistent method name.
> -
>
> Key: DRILL-6336
> URL: https://issues.apache.org/jira/browse/DRILL-6336
> Project: Apache Drill
> Issue Type: Improvement
>Reporter: KuiLIU
>Priority: Major
> Attachments: rename-method.patch
>
>
> The following method is named "append", but its body code is an method
> invocation "writer.print( s )". The method should be named "print".
> {code:java}
> public DebugStringBuilder append( String s ) {
> writer.print( s );
> return this;
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-6336) Inconsistent method name.
[
https://issues.apache.org/jira/browse/DRILL-6336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16474325#comment-16474325
]
ASF GitHub Bot commented on DRILL-6336:
---
arina-ielchiieva commented on issue #1235: DRILL-6336: Inconsistent method name.
URL: https://github.com/apache/drill/pull/1235#issuecomment-388849574
@BruceKuiLiu could you please check if @vrozov suggestion is feasible?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Inconsistent method name.
> -
>
> Key: DRILL-6336
> URL: https://issues.apache.org/jira/browse/DRILL-6336
> Project: Apache Drill
> Issue Type: Improvement
>Reporter: KuiLIU
>Priority: Major
> Attachments: rename-method.patch
>
>
> The following method is named "append", but its body code is an method
> invocation "writer.print( s )". The method should be named "print".
> {code:java}
> public DebugStringBuilder append( String s ) {
> writer.print( s );
> return this;
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-5270) Improve loading of profiles listing in the WebUI
[
https://issues.apache.org/jira/browse/DRILL-5270?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16474316#comment-16474316
]
ASF GitHub Bot commented on DRILL-5270:
---
arina-ielchiieva commented on a change in pull request #1250: DRILL-5270:
Improve loading of profiles listing in the WebUI
URL: https://github.com/apache/drill/pull/1250#discussion_r187984321
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/store/sys/store/LocalPersistentStore.java
##
@@ -1,220 +1,377 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.drill.exec.store.sys.store;
-
-import static org.apache.drill.exec.ExecConstants.DRILL_SYS_FILE_SUFFIX;
-
-import java.io.File;
-import java.io.IOException;
-import java.io.InputStream;
-import java.io.OutputStream;
-import java.util.Collections;
-import java.util.Iterator;
-import java.util.List;
-import java.util.Map;
-import java.util.Map.Entry;
-
-import javax.annotation.Nullable;
-
-import org.apache.commons.io.IOUtils;
-import org.apache.drill.common.collections.ImmutableEntry;
-import org.apache.drill.common.config.DrillConfig;
-import org.apache.drill.exec.store.dfs.DrillFileSystem;
-import org.apache.drill.exec.util.DrillFileSystemUtil;
-import org.apache.drill.exec.store.sys.BasePersistentStore;
-import org.apache.drill.exec.store.sys.PersistentStoreConfig;
-import org.apache.drill.exec.store.sys.PersistentStoreMode;
-import org.apache.hadoop.conf.Configuration;
-import org.apache.hadoop.fs.FileStatus;
-import org.apache.hadoop.fs.FileSystem;
-import org.apache.hadoop.fs.Path;
-
-import com.google.common.base.Function;
-import com.google.common.base.Preconditions;
-import com.google.common.collect.Iterables;
-import com.google.common.collect.Lists;
-import org.apache.hadoop.fs.PathFilter;
-import org.slf4j.Logger;
-import org.slf4j.LoggerFactory;
-
-public class LocalPersistentStore extends BasePersistentStore {
- private static final Logger logger =
LoggerFactory.getLogger(LocalPersistentStore.class);
-
- private final Path basePath;
- private final PersistentStoreConfig config;
- private final DrillFileSystem fs;
-
- public LocalPersistentStore(DrillFileSystem fs, Path base,
PersistentStoreConfig config) {
-this.basePath = new Path(base, config.getName());
-this.config = config;
-this.fs = fs;
-try {
- mkdirs(getBasePath());
-} catch (IOException e) {
- throw new RuntimeException("Failure setting pstore configuration path.");
-}
- }
-
- protected Path getBasePath() {
-return basePath;
- }
-
- @Override
- public PersistentStoreMode getMode() {
-return PersistentStoreMode.PERSISTENT;
- }
-
- private void mkdirs(Path path) throws IOException {
-fs.mkdirs(path);
- }
-
- public static Path getLogDir() {
-String drillLogDir = System.getenv("DRILL_LOG_DIR");
-if (drillLogDir == null) {
- drillLogDir = System.getProperty("drill.log.dir");
-}
-if (drillLogDir == null) {
- drillLogDir = "/var/log/drill";
-}
-return new Path(new File(drillLogDir).getAbsoluteFile().toURI());
- }
-
- public static DrillFileSystem getFileSystem(DrillConfig config, Path root)
throws IOException {
-Path blobRoot = root == null ? getLogDir() : root;
-Configuration fsConf = new Configuration();
-if (blobRoot.toUri().getScheme() != null) {
- fsConf.set(FileSystem.FS_DEFAULT_NAME_KEY, blobRoot.toUri().toString());
-}
-
-
-DrillFileSystem fs = new DrillFileSystem(fsConf);
-fs.mkdirs(blobRoot);
-return fs;
- }
-
- @Override
- public Iterator> getRange(int skip, int take) {
-try {
- // list only files with sys file suffix
- PathFilter sysFileSuffixFilter = new PathFilter() {
-@Override
-public boolean accept(Path path) {
- return path.getName().endsWith(DRILL_SYS_FILE_SUFFIX);
-}
- };
-
- List fileStatuses = DrillFileSystemUtil.listFiles(fs,
basePath, false, sysFileSuffixFilter);
- if (fileStatuses.isEmpty()) {
-return Collections.emptyIterator();
- }
-
- List files = Lists.newArrayList();
- for (File
[jira] [Commented] (DRILL-5270) Improve loading of profiles listing in the WebUI
[
https://issues.apache.org/jira/browse/DRILL-5270?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16474315#comment-16474315
]
ASF GitHub Bot commented on DRILL-5270:
---
arina-ielchiieva commented on a change in pull request #1250: DRILL-5270:
Improve loading of profiles listing in the WebUI
URL: https://github.com/apache/drill/pull/1250#discussion_r187981874
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/store/sys/store/LocalPersistentStore.java
##
@@ -1,220 +1,377 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.drill.exec.store.sys.store;
-
-import static org.apache.drill.exec.ExecConstants.DRILL_SYS_FILE_SUFFIX;
-
-import java.io.File;
-import java.io.IOException;
-import java.io.InputStream;
-import java.io.OutputStream;
-import java.util.Collections;
-import java.util.Iterator;
-import java.util.List;
-import java.util.Map;
-import java.util.Map.Entry;
-
-import javax.annotation.Nullable;
-
-import org.apache.commons.io.IOUtils;
-import org.apache.drill.common.collections.ImmutableEntry;
-import org.apache.drill.common.config.DrillConfig;
-import org.apache.drill.exec.store.dfs.DrillFileSystem;
-import org.apache.drill.exec.util.DrillFileSystemUtil;
-import org.apache.drill.exec.store.sys.BasePersistentStore;
-import org.apache.drill.exec.store.sys.PersistentStoreConfig;
-import org.apache.drill.exec.store.sys.PersistentStoreMode;
-import org.apache.hadoop.conf.Configuration;
-import org.apache.hadoop.fs.FileStatus;
-import org.apache.hadoop.fs.FileSystem;
-import org.apache.hadoop.fs.Path;
-
-import com.google.common.base.Function;
-import com.google.common.base.Preconditions;
-import com.google.common.collect.Iterables;
-import com.google.common.collect.Lists;
-import org.apache.hadoop.fs.PathFilter;
-import org.slf4j.Logger;
-import org.slf4j.LoggerFactory;
-
-public class LocalPersistentStore extends BasePersistentStore {
- private static final Logger logger =
LoggerFactory.getLogger(LocalPersistentStore.class);
-
- private final Path basePath;
- private final PersistentStoreConfig config;
- private final DrillFileSystem fs;
-
- public LocalPersistentStore(DrillFileSystem fs, Path base,
PersistentStoreConfig config) {
-this.basePath = new Path(base, config.getName());
-this.config = config;
-this.fs = fs;
-try {
- mkdirs(getBasePath());
-} catch (IOException e) {
- throw new RuntimeException("Failure setting pstore configuration path.");
-}
- }
-
- protected Path getBasePath() {
-return basePath;
- }
-
- @Override
- public PersistentStoreMode getMode() {
-return PersistentStoreMode.PERSISTENT;
- }
-
- private void mkdirs(Path path) throws IOException {
-fs.mkdirs(path);
- }
-
- public static Path getLogDir() {
-String drillLogDir = System.getenv("DRILL_LOG_DIR");
-if (drillLogDir == null) {
- drillLogDir = System.getProperty("drill.log.dir");
-}
-if (drillLogDir == null) {
- drillLogDir = "/var/log/drill";
-}
-return new Path(new File(drillLogDir).getAbsoluteFile().toURI());
- }
-
- public static DrillFileSystem getFileSystem(DrillConfig config, Path root)
throws IOException {
-Path blobRoot = root == null ? getLogDir() : root;
-Configuration fsConf = new Configuration();
-if (blobRoot.toUri().getScheme() != null) {
- fsConf.set(FileSystem.FS_DEFAULT_NAME_KEY, blobRoot.toUri().toString());
-}
-
-
-DrillFileSystem fs = new DrillFileSystem(fsConf);
-fs.mkdirs(blobRoot);
-return fs;
- }
-
- @Override
- public Iterator> getRange(int skip, int take) {
-try {
- // list only files with sys file suffix
- PathFilter sysFileSuffixFilter = new PathFilter() {
-@Override
-public boolean accept(Path path) {
- return path.getName().endsWith(DRILL_SYS_FILE_SUFFIX);
-}
- };
-
- List fileStatuses = DrillFileSystemUtil.listFiles(fs,
basePath, false, sysFileSuffixFilter);
- if (fileStatuses.isEmpty()) {
-return Collections.emptyIterator();
- }
-
- List files = Lists.newArrayList();
- for (File
[jira] [Commented] (DRILL-5270) Improve loading of profiles listing in the WebUI
[
https://issues.apache.org/jira/browse/DRILL-5270?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16474320#comment-16474320
]
ASF GitHub Bot commented on DRILL-5270:
---
arina-ielchiieva commented on a change in pull request #1250: DRILL-5270:
Improve loading of profiles listing in the WebUI
URL: https://github.com/apache/drill/pull/1250#discussion_r187984786
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/store/sys/store/LocalPersistentStore.java
##
@@ -1,220 +1,377 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.drill.exec.store.sys.store;
-
-import static org.apache.drill.exec.ExecConstants.DRILL_SYS_FILE_SUFFIX;
-
-import java.io.File;
-import java.io.IOException;
-import java.io.InputStream;
-import java.io.OutputStream;
-import java.util.Collections;
-import java.util.Iterator;
-import java.util.List;
-import java.util.Map;
-import java.util.Map.Entry;
-
-import javax.annotation.Nullable;
-
-import org.apache.commons.io.IOUtils;
-import org.apache.drill.common.collections.ImmutableEntry;
-import org.apache.drill.common.config.DrillConfig;
-import org.apache.drill.exec.store.dfs.DrillFileSystem;
-import org.apache.drill.exec.util.DrillFileSystemUtil;
-import org.apache.drill.exec.store.sys.BasePersistentStore;
-import org.apache.drill.exec.store.sys.PersistentStoreConfig;
-import org.apache.drill.exec.store.sys.PersistentStoreMode;
-import org.apache.hadoop.conf.Configuration;
-import org.apache.hadoop.fs.FileStatus;
-import org.apache.hadoop.fs.FileSystem;
-import org.apache.hadoop.fs.Path;
-
-import com.google.common.base.Function;
-import com.google.common.base.Preconditions;
-import com.google.common.collect.Iterables;
-import com.google.common.collect.Lists;
-import org.apache.hadoop.fs.PathFilter;
-import org.slf4j.Logger;
-import org.slf4j.LoggerFactory;
-
-public class LocalPersistentStore extends BasePersistentStore {
- private static final Logger logger =
LoggerFactory.getLogger(LocalPersistentStore.class);
-
- private final Path basePath;
- private final PersistentStoreConfig config;
- private final DrillFileSystem fs;
-
- public LocalPersistentStore(DrillFileSystem fs, Path base,
PersistentStoreConfig config) {
-this.basePath = new Path(base, config.getName());
-this.config = config;
-this.fs = fs;
-try {
- mkdirs(getBasePath());
-} catch (IOException e) {
- throw new RuntimeException("Failure setting pstore configuration path.");
-}
- }
-
- protected Path getBasePath() {
-return basePath;
- }
-
- @Override
- public PersistentStoreMode getMode() {
-return PersistentStoreMode.PERSISTENT;
- }
-
- private void mkdirs(Path path) throws IOException {
-fs.mkdirs(path);
- }
-
- public static Path getLogDir() {
-String drillLogDir = System.getenv("DRILL_LOG_DIR");
-if (drillLogDir == null) {
- drillLogDir = System.getProperty("drill.log.dir");
-}
-if (drillLogDir == null) {
- drillLogDir = "/var/log/drill";
-}
-return new Path(new File(drillLogDir).getAbsoluteFile().toURI());
- }
-
- public static DrillFileSystem getFileSystem(DrillConfig config, Path root)
throws IOException {
-Path blobRoot = root == null ? getLogDir() : root;
-Configuration fsConf = new Configuration();
-if (blobRoot.toUri().getScheme() != null) {
- fsConf.set(FileSystem.FS_DEFAULT_NAME_KEY, blobRoot.toUri().toString());
-}
-
-
-DrillFileSystem fs = new DrillFileSystem(fsConf);
-fs.mkdirs(blobRoot);
-return fs;
- }
-
- @Override
- public Iterator> getRange(int skip, int take) {
-try {
- // list only files with sys file suffix
- PathFilter sysFileSuffixFilter = new PathFilter() {
-@Override
-public boolean accept(Path path) {
- return path.getName().endsWith(DRILL_SYS_FILE_SUFFIX);
-}
- };
-
- List fileStatuses = DrillFileSystemUtil.listFiles(fs,
basePath, false, sysFileSuffixFilter);
- if (fileStatuses.isEmpty()) {
-return Collections.emptyIterator();
- }
-
- List files = Lists.newArrayList();
- for (File
[jira] [Commented] (DRILL-5270) Improve loading of profiles listing in the WebUI
[
https://issues.apache.org/jira/browse/DRILL-5270?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16474319#comment-16474319
]
ASF GitHub Bot commented on DRILL-5270:
---
arina-ielchiieva commented on a change in pull request #1250: DRILL-5270:
Improve loading of profiles listing in the WebUI
URL: https://github.com/apache/drill/pull/1250#discussion_r187982481
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/store/sys/store/LocalPersistentStore.java
##
@@ -1,220 +1,377 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.drill.exec.store.sys.store;
-
-import static org.apache.drill.exec.ExecConstants.DRILL_SYS_FILE_SUFFIX;
-
-import java.io.File;
-import java.io.IOException;
-import java.io.InputStream;
-import java.io.OutputStream;
-import java.util.Collections;
-import java.util.Iterator;
-import java.util.List;
-import java.util.Map;
-import java.util.Map.Entry;
-
-import javax.annotation.Nullable;
-
-import org.apache.commons.io.IOUtils;
-import org.apache.drill.common.collections.ImmutableEntry;
-import org.apache.drill.common.config.DrillConfig;
-import org.apache.drill.exec.store.dfs.DrillFileSystem;
-import org.apache.drill.exec.util.DrillFileSystemUtil;
-import org.apache.drill.exec.store.sys.BasePersistentStore;
-import org.apache.drill.exec.store.sys.PersistentStoreConfig;
-import org.apache.drill.exec.store.sys.PersistentStoreMode;
-import org.apache.hadoop.conf.Configuration;
-import org.apache.hadoop.fs.FileStatus;
-import org.apache.hadoop.fs.FileSystem;
-import org.apache.hadoop.fs.Path;
-
-import com.google.common.base.Function;
-import com.google.common.base.Preconditions;
-import com.google.common.collect.Iterables;
-import com.google.common.collect.Lists;
-import org.apache.hadoop.fs.PathFilter;
-import org.slf4j.Logger;
-import org.slf4j.LoggerFactory;
-
-public class LocalPersistentStore extends BasePersistentStore {
- private static final Logger logger =
LoggerFactory.getLogger(LocalPersistentStore.class);
-
- private final Path basePath;
- private final PersistentStoreConfig config;
- private final DrillFileSystem fs;
-
- public LocalPersistentStore(DrillFileSystem fs, Path base,
PersistentStoreConfig config) {
-this.basePath = new Path(base, config.getName());
-this.config = config;
-this.fs = fs;
-try {
- mkdirs(getBasePath());
-} catch (IOException e) {
- throw new RuntimeException("Failure setting pstore configuration path.");
-}
- }
-
- protected Path getBasePath() {
-return basePath;
- }
-
- @Override
- public PersistentStoreMode getMode() {
-return PersistentStoreMode.PERSISTENT;
- }
-
- private void mkdirs(Path path) throws IOException {
-fs.mkdirs(path);
- }
-
- public static Path getLogDir() {
-String drillLogDir = System.getenv("DRILL_LOG_DIR");
-if (drillLogDir == null) {
- drillLogDir = System.getProperty("drill.log.dir");
-}
-if (drillLogDir == null) {
- drillLogDir = "/var/log/drill";
-}
-return new Path(new File(drillLogDir).getAbsoluteFile().toURI());
- }
-
- public static DrillFileSystem getFileSystem(DrillConfig config, Path root)
throws IOException {
-Path blobRoot = root == null ? getLogDir() : root;
-Configuration fsConf = new Configuration();
-if (blobRoot.toUri().getScheme() != null) {
- fsConf.set(FileSystem.FS_DEFAULT_NAME_KEY, blobRoot.toUri().toString());
-}
-
-
-DrillFileSystem fs = new DrillFileSystem(fsConf);
-fs.mkdirs(blobRoot);
-return fs;
- }
-
- @Override
- public Iterator> getRange(int skip, int take) {
-try {
- // list only files with sys file suffix
- PathFilter sysFileSuffixFilter = new PathFilter() {
-@Override
-public boolean accept(Path path) {
- return path.getName().endsWith(DRILL_SYS_FILE_SUFFIX);
-}
- };
-
- List fileStatuses = DrillFileSystemUtil.listFiles(fs,
basePath, false, sysFileSuffixFilter);
- if (fileStatuses.isEmpty()) {
-return Collections.emptyIterator();
- }
-
- List files = Lists.newArrayList();
- for (File
[jira] [Commented] (DRILL-5270) Improve loading of profiles listing in the WebUI
[
https://issues.apache.org/jira/browse/DRILL-5270?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16474318#comment-16474318
]
ASF GitHub Bot commented on DRILL-5270:
---
arina-ielchiieva commented on a change in pull request #1250: DRILL-5270:
Improve loading of profiles listing in the WebUI
URL: https://github.com/apache/drill/pull/1250#discussion_r187984531
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/store/sys/store/LocalPersistentStore.java
##
@@ -1,220 +1,377 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.drill.exec.store.sys.store;
-
-import static org.apache.drill.exec.ExecConstants.DRILL_SYS_FILE_SUFFIX;
-
-import java.io.File;
-import java.io.IOException;
-import java.io.InputStream;
-import java.io.OutputStream;
-import java.util.Collections;
-import java.util.Iterator;
-import java.util.List;
-import java.util.Map;
-import java.util.Map.Entry;
-
-import javax.annotation.Nullable;
-
-import org.apache.commons.io.IOUtils;
-import org.apache.drill.common.collections.ImmutableEntry;
-import org.apache.drill.common.config.DrillConfig;
-import org.apache.drill.exec.store.dfs.DrillFileSystem;
-import org.apache.drill.exec.util.DrillFileSystemUtil;
-import org.apache.drill.exec.store.sys.BasePersistentStore;
-import org.apache.drill.exec.store.sys.PersistentStoreConfig;
-import org.apache.drill.exec.store.sys.PersistentStoreMode;
-import org.apache.hadoop.conf.Configuration;
-import org.apache.hadoop.fs.FileStatus;
-import org.apache.hadoop.fs.FileSystem;
-import org.apache.hadoop.fs.Path;
-
-import com.google.common.base.Function;
-import com.google.common.base.Preconditions;
-import com.google.common.collect.Iterables;
-import com.google.common.collect.Lists;
-import org.apache.hadoop.fs.PathFilter;
-import org.slf4j.Logger;
-import org.slf4j.LoggerFactory;
-
-public class LocalPersistentStore extends BasePersistentStore {
- private static final Logger logger =
LoggerFactory.getLogger(LocalPersistentStore.class);
-
- private final Path basePath;
- private final PersistentStoreConfig config;
- private final DrillFileSystem fs;
-
- public LocalPersistentStore(DrillFileSystem fs, Path base,
PersistentStoreConfig config) {
-this.basePath = new Path(base, config.getName());
-this.config = config;
-this.fs = fs;
-try {
- mkdirs(getBasePath());
-} catch (IOException e) {
- throw new RuntimeException("Failure setting pstore configuration path.");
-}
- }
-
- protected Path getBasePath() {
-return basePath;
- }
-
- @Override
- public PersistentStoreMode getMode() {
-return PersistentStoreMode.PERSISTENT;
- }
-
- private void mkdirs(Path path) throws IOException {
-fs.mkdirs(path);
- }
-
- public static Path getLogDir() {
-String drillLogDir = System.getenv("DRILL_LOG_DIR");
-if (drillLogDir == null) {
- drillLogDir = System.getProperty("drill.log.dir");
-}
-if (drillLogDir == null) {
- drillLogDir = "/var/log/drill";
-}
-return new Path(new File(drillLogDir).getAbsoluteFile().toURI());
- }
-
- public static DrillFileSystem getFileSystem(DrillConfig config, Path root)
throws IOException {
-Path blobRoot = root == null ? getLogDir() : root;
-Configuration fsConf = new Configuration();
-if (blobRoot.toUri().getScheme() != null) {
- fsConf.set(FileSystem.FS_DEFAULT_NAME_KEY, blobRoot.toUri().toString());
-}
-
-
-DrillFileSystem fs = new DrillFileSystem(fsConf);
-fs.mkdirs(blobRoot);
-return fs;
- }
-
- @Override
- public Iterator> getRange(int skip, int take) {
-try {
- // list only files with sys file suffix
- PathFilter sysFileSuffixFilter = new PathFilter() {
-@Override
-public boolean accept(Path path) {
- return path.getName().endsWith(DRILL_SYS_FILE_SUFFIX);
-}
- };
-
- List fileStatuses = DrillFileSystemUtil.listFiles(fs,
basePath, false, sysFileSuffixFilter);
- if (fileStatuses.isEmpty()) {
-return Collections.emptyIterator();
- }
-
- List files = Lists.newArrayList();
- for (File
[jira] [Commented] (DRILL-5270) Improve loading of profiles listing in the WebUI
[
https://issues.apache.org/jira/browse/DRILL-5270?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16474313#comment-16474313
]
ASF GitHub Bot commented on DRILL-5270:
---
arina-ielchiieva commented on a change in pull request #1250: DRILL-5270:
Improve loading of profiles listing in the WebUI
URL: https://github.com/apache/drill/pull/1250#discussion_r187981315
##
File path:
exec/java-exec/src/test/java/org/apache/drill/exec/store/sys/TestProfileSet.java
##
@@ -0,0 +1,130 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.sys;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertTrue;
+
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Random;
+
+import org.apache.commons.lang3.StringUtils;
+import org.apache.drill.exec.store.sys.store.ProfileSet;
+import org.junit.BeforeClass;
+import org.junit.Test;
+
+/**
+ * Test the size-constrained ProfileSet for use in the webserver's '/profiles'
listing
+ */
+public class TestProfileSet {
+ private final static String PROFILE_PREFIX = "t35t-pr0fil3-";
+ static int initCapacity;
+ static int finalCapacity;
+ static int storeCount;
+ static Random rand;
+ static List masterList;
+
+ @BeforeClass
+ public static void setupProfileSet() {
+initCapacity = 50;
+finalCapacity = 70;
+storeCount = 100;
+rand = new Random();
+//Generating source list of storeCount # 'profiles'
+masterList = new LinkedList();
+for (int i = 0; i < storeCount; i++) {
+ masterList.add(PROFILE_PREFIX + StringUtils.leftPad(String.valueOf(i),
String.valueOf(storeCount).length(), '0'));
+}
+ }
+
+ @Test
+ public void testProfileOrder() throws Exception {
+//clone initial # profiles and verify via iterator.
+ProfileSet testSet = new ProfileSet(initCapacity);
+List srcList = new LinkedList(masterList);
+
+//Loading randomly
+for (int i = 0; i < initCapacity; i++) {
+ String poppedProfile =
testSet.add(srcList.remove(rand.nextInt(storeCount - i)));
+ assert (poppedProfile == null);
+ assertEquals(null, poppedProfile);
+}
+
+//Testing order
+String prevProfile = null;
+while (!testSet.isEmpty()) {
+ String currOldestProfile = testSet.removeOldest();
+ if (prevProfile != null) {
+assertTrue( prevProfile.compareTo(currOldestProfile) > 0 );
+ }
+ prevProfile = currOldestProfile;
+}
+ }
+
+ //Test if inserts exceeding capacity leads to eviction of oldest
+ @Test
+ public void testExcessInjection() throws Exception {
+//clone initial # profiles and verify via iterator.
+ProfileSet testSet = new ProfileSet(initCapacity);
+List srcList = new LinkedList(masterList);
+
+//Loading randomly
+for (int i = 0; i < initCapacity; i++) {
+ String poppedProfile =
testSet.add(srcList.remove(rand.nextInt(storeCount - i)));
+ assertEquals(null, poppedProfile);
+}
+
+//Testing Excess by looking at oldest popped
+for (int i = initCapacity; i < finalCapacity; i++) {
+ String toInsert = srcList.remove(rand.nextInt(storeCount - i));
+ String expectedToPop = ( toInsert.compareTo(testSet.getOldest()) > 0 ?
+ toInsert : testSet.getOldest() );
+
+ String oldestPoppedProfile = testSet.add(toInsert);
+ assertEquals(expectedToPop, oldestPoppedProfile);
+}
+
+assertEquals(initCapacity, testSet.size());
+ }
+
+ //Test if size internally resizes to final capacity with no evictions
+ @Test
+ public void testSetResize() throws Exception {
+//clone initial # profiles into a 700-capacity set.
+ProfileSet testSet = new ProfileSet(finalCapacity);
+List srcList = new LinkedList(masterList);
+
+//Loading randomly
+for (int i = 0; i < initCapacity; i++) {
+ String poppedProfile =
testSet.add(srcList.remove(rand.nextInt(storeCount - i)));
+ assertEquals(null, poppedProfile);
+}
+
+assert(testSet.size() == initCapacity);
Review comment:
Please use junit assertions in tests.
---
[jira] [Commented] (DRILL-5270) Improve loading of profiles listing in the WebUI
[
https://issues.apache.org/jira/browse/DRILL-5270?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16474314#comment-16474314
]
ASF GitHub Bot commented on DRILL-5270:
---
arina-ielchiieva commented on a change in pull request #1250: DRILL-5270:
Improve loading of profiles listing in the WebUI
URL: https://github.com/apache/drill/pull/1250#discussion_r187981529
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/store/sys/store/ProfileSet.java
##
@@ -0,0 +1,152 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.sys.store;
+
+import java.util.Iterator;
+import java.util.TreeSet;
+import java.util.concurrent.atomic.AtomicInteger;
+
+/**
+ * Wrapper around TreeSet to mimic a size-bound set ordered by name
(implicitly the profiles' age)
+ */
+public class ProfileSet implements Iterable {
+ private TreeSet store;
+ private int maxCapacity;
+ //Using a dedicated counter to avoid
+ private AtomicInteger size;
+
+ @SuppressWarnings("unused")
+ @Deprecated
+ private ProfileSet() {}
+
+ public ProfileSet(int capacity) {
+this.store = new TreeSet();
+this.maxCapacity = capacity;
+this.size = new AtomicInteger();
+ }
+
+ public int size() {
+return size.get();
+ }
+
+ /**
+ * Get max capacity of the profile set
+ * @return max capacity
+ */
+ public int capacity() {
+return maxCapacity;
+ }
+
+ /**
+ * Add a profile name to the set, while removing the oldest, if exceeding
capacity
+ * @param profile
+ * @return oldest profile
+ */
+ public String add(String profile) {
+return add(profile, false);
+ }
+
+ /**
+ * Add a profile name to the set, while removing the oldest or youngest,
based on flag
+ * @param profile
+ * @param retainOldest indicate retaining policy as oldest
+ * @return youngest/oldest profile
+ */
+ public String add(String profile, boolean retainOldest) {
+store.add(profile);
+if ( size.incrementAndGet() > maxCapacity ) {
Review comment:
Please remove spaces.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Improve loading of profiles listing in the WebUI
>
>
> Key: DRILL-5270
> URL: https://issues.apache.org/jira/browse/DRILL-5270
> Project: Apache Drill
> Issue Type: Improvement
> Components: Web Server
>Affects Versions: 1.9.0
>Reporter: Kunal Khatua
>Assignee: Kunal Khatua
>Priority: Major
> Fix For: 1.14.0
>
>
> Currently, as the number of profiles increase, we reload the same list of
> profiles from the FS.
> An ideal improvement would be to detect if there are any new profiles and
> only reload from the disk then. Otherwise, a cached list is sufficient.
> For a directory of 280K profiles, the load time is close to 6 seconds on a 32
> core server. With the caching, we can get it down to as much as a few
> milliseconds.
> To render the cache as invalid, we inspect the last modified time of the
> directory to confirm whether a reload is needed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-5270) Improve loading of profiles listing in the WebUI
[
https://issues.apache.org/jira/browse/DRILL-5270?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16474317#comment-16474317
]
ASF GitHub Bot commented on DRILL-5270:
---
arina-ielchiieva commented on a change in pull request #1250: DRILL-5270:
Improve loading of profiles listing in the WebUI
URL: https://github.com/apache/drill/pull/1250#discussion_r187983484
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/store/sys/store/LocalPersistentStore.java
##
@@ -1,220 +1,377 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.drill.exec.store.sys.store;
-
-import static org.apache.drill.exec.ExecConstants.DRILL_SYS_FILE_SUFFIX;
-
-import java.io.File;
-import java.io.IOException;
-import java.io.InputStream;
-import java.io.OutputStream;
-import java.util.Collections;
-import java.util.Iterator;
-import java.util.List;
-import java.util.Map;
-import java.util.Map.Entry;
-
-import javax.annotation.Nullable;
-
-import org.apache.commons.io.IOUtils;
-import org.apache.drill.common.collections.ImmutableEntry;
-import org.apache.drill.common.config.DrillConfig;
-import org.apache.drill.exec.store.dfs.DrillFileSystem;
-import org.apache.drill.exec.util.DrillFileSystemUtil;
-import org.apache.drill.exec.store.sys.BasePersistentStore;
-import org.apache.drill.exec.store.sys.PersistentStoreConfig;
-import org.apache.drill.exec.store.sys.PersistentStoreMode;
-import org.apache.hadoop.conf.Configuration;
-import org.apache.hadoop.fs.FileStatus;
-import org.apache.hadoop.fs.FileSystem;
-import org.apache.hadoop.fs.Path;
-
-import com.google.common.base.Function;
-import com.google.common.base.Preconditions;
-import com.google.common.collect.Iterables;
-import com.google.common.collect.Lists;
-import org.apache.hadoop.fs.PathFilter;
-import org.slf4j.Logger;
-import org.slf4j.LoggerFactory;
-
-public class LocalPersistentStore extends BasePersistentStore {
- private static final Logger logger =
LoggerFactory.getLogger(LocalPersistentStore.class);
-
- private final Path basePath;
- private final PersistentStoreConfig config;
- private final DrillFileSystem fs;
-
- public LocalPersistentStore(DrillFileSystem fs, Path base,
PersistentStoreConfig config) {
-this.basePath = new Path(base, config.getName());
-this.config = config;
-this.fs = fs;
-try {
- mkdirs(getBasePath());
-} catch (IOException e) {
- throw new RuntimeException("Failure setting pstore configuration path.");
-}
- }
-
- protected Path getBasePath() {
-return basePath;
- }
-
- @Override
- public PersistentStoreMode getMode() {
-return PersistentStoreMode.PERSISTENT;
- }
-
- private void mkdirs(Path path) throws IOException {
-fs.mkdirs(path);
- }
-
- public static Path getLogDir() {
-String drillLogDir = System.getenv("DRILL_LOG_DIR");
-if (drillLogDir == null) {
- drillLogDir = System.getProperty("drill.log.dir");
-}
-if (drillLogDir == null) {
- drillLogDir = "/var/log/drill";
-}
-return new Path(new File(drillLogDir).getAbsoluteFile().toURI());
- }
-
- public static DrillFileSystem getFileSystem(DrillConfig config, Path root)
throws IOException {
-Path blobRoot = root == null ? getLogDir() : root;
-Configuration fsConf = new Configuration();
-if (blobRoot.toUri().getScheme() != null) {
- fsConf.set(FileSystem.FS_DEFAULT_NAME_KEY, blobRoot.toUri().toString());
-}
-
-
-DrillFileSystem fs = new DrillFileSystem(fsConf);
-fs.mkdirs(blobRoot);
-return fs;
- }
-
- @Override
- public Iterator> getRange(int skip, int take) {
-try {
- // list only files with sys file suffix
- PathFilter sysFileSuffixFilter = new PathFilter() {
-@Override
-public boolean accept(Path path) {
- return path.getName().endsWith(DRILL_SYS_FILE_SUFFIX);
-}
- };
-
- List fileStatuses = DrillFileSystemUtil.listFiles(fs,
basePath, false, sysFileSuffixFilter);
- if (fileStatuses.isEmpty()) {
-return Collections.emptyIterator();
- }
-
- List files = Lists.newArrayList();
- for (File
[jira] [Commented] (DRILL-6255) Drillbit while sending control message to itself creates a connection instead of submitting locally
[
https://issues.apache.org/jira/browse/DRILL-6255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16474290#comment-16474290
]
ASF GitHub Bot commented on DRILL-6255:
---
arina-ielchiieva commented on a change in pull request #1253: DRILL-6255:
Drillbit while sending control message to itself creates …
URL: https://github.com/apache/drill/pull/1253#discussion_r187977640
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/rpc/control/LocalControlConnectionManager.java
##
@@ -0,0 +1,236 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.rpc.control;
+
+import io.netty.buffer.ByteBuf;
+import io.netty.buffer.DrillBuf;
+import org.apache.drill.exec.exception.OutOfMemoryException;
+import org.apache.drill.exec.proto.BitControl;
+import org.apache.drill.exec.proto.CoordinationProtos.DrillbitEndpoint;
+import org.apache.drill.exec.proto.GeneralRPCProtos.Ack;
+import org.apache.drill.exec.proto.UserBitShared;
+import org.apache.drill.exec.rpc.BasicClient;
+import org.apache.drill.exec.rpc.Response;
+import org.apache.drill.exec.rpc.RpcCommand;
+import org.apache.drill.exec.rpc.RpcConstants;
+import org.apache.drill.exec.rpc.RpcException;
+import org.apache.drill.exec.rpc.RpcOutcomeListener;
+import org.apache.drill.exec.work.batch.ControlMessageHandler;
+
+public class LocalControlConnectionManager extends ControlConnectionManager {
+ private static final org.slf4j.Logger logger =
org.slf4j.LoggerFactory.getLogger(LocalControlConnectionManager.class);
+
+ private final ControlConnectionConfig config;
+
+ public LocalControlConnectionManager(ControlConnectionConfig config,
DrillbitEndpoint localEndpoint) {
+super(localEndpoint, localEndpoint);
+this.config = config;
+ }
+
+ @Override
+ protected BasicClient getNewClient() {
+throw new UnsupportedOperationException("LocalControlConnectionManager
doesn't support creating a control client");
+ }
+
+ @Override
+ public void runCommand(RpcCommand cmd) {
+final int rpcType = cmd.getRpcType().getNumber();
+final ControlMessageHandler messageHandler = config.getMessageHandler();
+
+if (RpcConstants.EXTRA_DEBUGGING) {
+ logger.debug("Received bit com message of type {} over local connection
manager", rpcType);
+}
+
+switch (rpcType) {
+
+ case BitControl.RpcType.REQ_CANCEL_FRAGMENT_VALUE: {
+final ControlTunnel.SignalFragment signalFragment =
((ControlTunnel.SignalFragment) cmd);
+final RpcOutcomeListener outcomeListener =
signalFragment.getOutcomeListener();
+final Ack ackResponse =
messageHandler.cancelFragment(signalFragment.getMessage());
+outcomeListener.success(ackResponse, null);
+break;
+ }
+
+ case BitControl.RpcType.REQ_CUSTOM_VALUE: {
+final ByteBuf[] dataBodies;
+final RpcOutcomeListener outcomeListener;
+
+if (cmd instanceof ControlTunnel.CustomMessageSender) {
+ dataBodies =
((ControlTunnel.CustomMessageSender)cmd).getDataBodies();
+ outcomeListener =
((ControlTunnel.CustomMessageSender)cmd).getOutcomeListener();
+} else if (cmd instanceof ControlTunnel.SyncCustomMessageSender) {
+ dataBodies =
((ControlTunnel.SyncCustomMessageSender)cmd).getDataBodies();
+ outcomeListener =
((ControlTunnel.SyncCustomMessageSender)cmd).getOutcomeListener();
+} else {
+ throw new UnsupportedOperationException("Unknown Custom Type control
message received");
+}
+
+DrillBuf reqDrillBuff;
+try {
+ reqDrillBuff = convertToByteBuf(dataBodies);
+} catch (Exception ex) {
+ outcomeListener.failed(new RpcException("Failed to allocate memory
while sending request in " +
+"LocalControlConnectionManager#convertToByteBuff", ex));
+ return;
+} finally {
+ releaseByteBuf(dataBodies);
+}
+
+try {
+ BitControl.CustomMessage message = (BitControl.CustomMessage)
cmd.getMessage();
+ final Response response =
messageHandler.getHandlerRegistry().handle(message, reqDrillBuff
[jira] [Commented] (DRILL-6255) Drillbit while sending control message to itself creates a connection instead of submitting locally
[
https://issues.apache.org/jira/browse/DRILL-6255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16474292#comment-16474292
]
ASF GitHub Bot commented on DRILL-6255:
---
arina-ielchiieva commented on a change in pull request #1253: DRILL-6255:
Drillbit while sending control message to itself creates …
URL: https://github.com/apache/drill/pull/1253#discussion_r187952040
##
File path:
exec/rpc/src/main/java/org/apache/drill/exec/rpc/FutureBitCommand.java
##
@@ -71,10 +73,15 @@ public void interrupted(final InterruptedException e) {
return parentFuture;
}
+ @Override
+ public RpcOutcomeListener getOutcomeListener() {
+return outcomeListener;
+ }
+
@Override
public void connectionFailed(FailureType type, Throwable t) {
settableFuture.setException(RpcException.mapException(
String.format("Command failed while establishing connection. Failure
type %s.", type), t));
}
-}
\ No newline at end of file
+}
Review comment:
New line.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Drillbit while sending control message to itself creates a connection instead
> of submitting locally
> ---
>
> Key: DRILL-6255
> URL: https://issues.apache.org/jira/browse/DRILL-6255
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Flow
>Affects Versions: 1.12.0
>Reporter: Sorabh Hamirwasia
>Assignee: Sorabh Hamirwasia
>Priority: Major
> Fix For: 1.14.0
>
>
> With the new shutdown feature introduced in 1.12, there is a state introduced
> in DrillbitEndpoint. Due to this the equality check happening
> [here|https://github.com/apache/drill/blob/master/exec/java-exec/src/main/java/org/apache/drill/exec/work/foreman/FragmentsRunner.java#L256]
> will result in false and hence the fragments supposed to be scheduled on
> Foreman will be treated as remote fragments and a connection will be created
> to schedule it. The equality check is false because localEndpoint state is
> STARTUP whereas state in assigned Drillbit is ONLINE.
> I guess now we should update the equality check to verify just for address
> and control port to be same between assigned and local Drillbit endpoint. A
> test can be added for this based on _sys.memory_ table since that guarantees
> scheduling minor fragments on each Drillbit node.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-6255) Drillbit while sending control message to itself creates a connection instead of submitting locally
[
https://issues.apache.org/jira/browse/DRILL-6255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16474296#comment-16474296
]
ASF GitHub Bot commented on DRILL-6255:
---
arina-ielchiieva commented on a change in pull request #1253: DRILL-6255:
Drillbit while sending control message to itself creates …
URL: https://github.com/apache/drill/pull/1253#discussion_r187956460
##
File path:
exec/java-exec/src/test/java/org/apache/drill/exec/rpc/control/TestLocalControlConnectionManager.java
##
@@ -0,0 +1,304 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.rpc.control;
+
+import io.netty.buffer.ByteBuf;
+import org.apache.drill.exec.proto.BitControl;
+import org.apache.drill.exec.proto.CoordinationProtos.DrillbitEndpoint;
+import org.apache.drill.exec.proto.ExecProtos;
+import org.apache.drill.exec.proto.GeneralRPCProtos;
+import org.apache.drill.exec.proto.UserBitShared;
+import org.apache.drill.exec.rpc.Acks;
+import org.apache.drill.exec.rpc.RpcException;
+import org.apache.drill.exec.rpc.RpcOutcomeListener;
+import org.apache.drill.exec.work.batch.ControlMessageHandler;
+import org.junit.Before;
+import org.junit.BeforeClass;
+import org.junit.Test;
+
+import java.util.concurrent.CountDownLatch;
+
+import static junit.framework.TestCase.fail;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertTrue;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+
+public class TestLocalControlConnectionManager {
+
+ private static final DrillbitEndpoint localEndpoint =
DrillbitEndpoint.newBuilder()
+.setAddress("10.0.0.1")
+.setControlPort(31011)
+.setState(DrillbitEndpoint.State.STARTUP)
+.build();
+
+ private static ControlConnectionConfig mockConfig;
+
+ private static ControlMessageHandler mockHandler;
+
+ private static ControlTunnel controlTunnel;
+
+ private CountDownLatch latch;
+
+ private final RpcOutcomeListener outcomeListener =
+new RpcOutcomeListener() {
+@Override
+public void failed(RpcException ex) {
+ throw new IllegalStateException(ex);
+}
+
+@Override
+public void success(GeneralRPCProtos.Ack value, ByteBuf buffer) {
+ if (value.getOk()) {
+latch.countDown();
+ } else {
+throw new IllegalStateException("Negative Ack received");
+ }
+}
+
+@Override
+public void interrupted(InterruptedException e) {
+
+}
+ };
+
+ @BeforeClass
+ public static void setup() {
+mockConfig = mock(ControlConnectionConfig.class);
+final ConnectionManagerRegistry registry = new
ConnectionManagerRegistry(mockConfig);
+registry.setLocalEndpoint(localEndpoint);
+ControlConnectionManager manager =
registry.getConnectionManager(localEndpoint);
+assertTrue(manager instanceof LocalControlConnectionManager);
+controlTunnel = new ControlTunnel(manager);
+ }
+
+ @Before
+ public void setupForTest() {
+mockHandler = mock(ControlMessageHandler.class);
+when(mockConfig.getMessageHandler()).thenReturn(mockHandler);
+ }
+
+ /**
+ * Verify that SendFragmentStatus is handled correctly using ControlTunnel
with LocalControlConnectionManager
+ */
+ @Test
+ public void testLocalSendFragmentStatus_Success() {
+final UserBitShared.QueryId mockQueryId =
UserBitShared.QueryId.getDefaultInstance();
+final UserBitShared.QueryProfile mockProfile =
UserBitShared.QueryProfile.getDefaultInstance();
+try {
+
when(mockHandler.requestQueryStatus(mockQueryId)).thenReturn(mockProfile);
+ final UserBitShared.QueryProfile returnedProfile =
controlTunnel.requestQueryProfile(mockQueryId).checkedGet();
+ assertEquals(returnedProfile, mockProfile);
+} catch (Exception ex) {
+ fail();
+}
+ }
+
+ /**
+ * Verify that SendFragmentStatus failure scenario is handled correctly
using ControlTunnel with
+ * LocalControlConnectionManager
+ */
+ @Test
+ public void testLocalSendFragmentStatus_Failure() {
+final UserBitShared.QueryId mockQueryId =
UserBitShared.QueryId.getDefaultInstance();
+try {
+ when(moc
[jira] [Commented] (DRILL-6255) Drillbit while sending control message to itself creates a connection instead of submitting locally
[
https://issues.apache.org/jira/browse/DRILL-6255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16474289#comment-16474289
]
ASF GitHub Bot commented on DRILL-6255:
---
arina-ielchiieva commented on a change in pull request #1253: DRILL-6255:
Drillbit while sending control message to itself creates …
URL: https://github.com/apache/drill/pull/1253#discussion_r187950137
##
File path: exec/rpc/src/main/java/org/apache/drill/exec/rpc/RpcCommand.java
##
@@ -17,10 +17,16 @@
*/
package org.apache.drill.exec.rpc;
+import com.google.protobuf.Internal.EnumLite;
import com.google.protobuf.MessageLite;
public interface RpcCommand
extends RpcConnectionHandler{
- public abstract void connectionAvailable(C connection);
+ void connectionAvailable(C connection);
-}
\ No newline at end of file
+ EnumLite getRpcType();
+
+ MessageLite getMessage();
+
+ RpcOutcomeListener getOutcomeListener();
+}
Review comment:
New line.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Drillbit while sending control message to itself creates a connection instead
> of submitting locally
> ---
>
> Key: DRILL-6255
> URL: https://issues.apache.org/jira/browse/DRILL-6255
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Flow
>Affects Versions: 1.12.0
>Reporter: Sorabh Hamirwasia
>Assignee: Sorabh Hamirwasia
>Priority: Major
> Fix For: 1.14.0
>
>
> With the new shutdown feature introduced in 1.12, there is a state introduced
> in DrillbitEndpoint. Due to this the equality check happening
> [here|https://github.com/apache/drill/blob/master/exec/java-exec/src/main/java/org/apache/drill/exec/work/foreman/FragmentsRunner.java#L256]
> will result in false and hence the fragments supposed to be scheduled on
> Foreman will be treated as remote fragments and a connection will be created
> to schedule it. The equality check is false because localEndpoint state is
> STARTUP whereas state in assigned Drillbit is ONLINE.
> I guess now we should update the equality check to verify just for address
> and control port to be same between assigned and local Drillbit endpoint. A
> test can be added for this based on _sys.memory_ table since that guarantees
> scheduling minor fragments on each Drillbit node.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-6255) Drillbit while sending control message to itself creates a connection instead of submitting locally
[
https://issues.apache.org/jira/browse/DRILL-6255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16474298#comment-16474298
]
ASF GitHub Bot commented on DRILL-6255:
---
arina-ielchiieva commented on a change in pull request #1253: DRILL-6255:
Drillbit while sending control message to itself creates …
URL: https://github.com/apache/drill/pull/1253#discussion_r187971211
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/rpc/control/LocalControlConnectionManager.java
##
@@ -0,0 +1,236 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.rpc.control;
+
+import io.netty.buffer.ByteBuf;
+import io.netty.buffer.DrillBuf;
+import org.apache.drill.exec.exception.OutOfMemoryException;
+import org.apache.drill.exec.proto.BitControl;
+import org.apache.drill.exec.proto.CoordinationProtos.DrillbitEndpoint;
+import org.apache.drill.exec.proto.GeneralRPCProtos.Ack;
+import org.apache.drill.exec.proto.UserBitShared;
+import org.apache.drill.exec.rpc.BasicClient;
+import org.apache.drill.exec.rpc.Response;
+import org.apache.drill.exec.rpc.RpcCommand;
+import org.apache.drill.exec.rpc.RpcConstants;
+import org.apache.drill.exec.rpc.RpcException;
+import org.apache.drill.exec.rpc.RpcOutcomeListener;
+import org.apache.drill.exec.work.batch.ControlMessageHandler;
+
+public class LocalControlConnectionManager extends ControlConnectionManager {
+ private static final org.slf4j.Logger logger =
org.slf4j.LoggerFactory.getLogger(LocalControlConnectionManager.class);
+
+ private final ControlConnectionConfig config;
+
+ public LocalControlConnectionManager(ControlConnectionConfig config,
DrillbitEndpoint localEndpoint) {
+super(localEndpoint, localEndpoint);
+this.config = config;
+ }
+
+ @Override
+ protected BasicClient getNewClient() {
+throw new UnsupportedOperationException("LocalControlConnectionManager
doesn't support creating a control client");
+ }
+
+ @Override
+ public void runCommand(RpcCommand cmd) {
+final int rpcType = cmd.getRpcType().getNumber();
+final ControlMessageHandler messageHandler = config.getMessageHandler();
+
+if (RpcConstants.EXTRA_DEBUGGING) {
+ logger.debug("Received bit com message of type {} over local connection
manager", rpcType);
+}
+
+switch (rpcType) {
+
+ case BitControl.RpcType.REQ_CANCEL_FRAGMENT_VALUE: {
+final ControlTunnel.SignalFragment signalFragment =
((ControlTunnel.SignalFragment) cmd);
+final RpcOutcomeListener outcomeListener =
signalFragment.getOutcomeListener();
+final Ack ackResponse =
messageHandler.cancelFragment(signalFragment.getMessage());
+outcomeListener.success(ackResponse, null);
+break;
+ }
+
+ case BitControl.RpcType.REQ_CUSTOM_VALUE: {
+final ByteBuf[] dataBodies;
+final RpcOutcomeListener outcomeListener;
+
+if (cmd instanceof ControlTunnel.CustomMessageSender) {
+ dataBodies =
((ControlTunnel.CustomMessageSender)cmd).getDataBodies();
+ outcomeListener =
((ControlTunnel.CustomMessageSender)cmd).getOutcomeListener();
+} else if (cmd instanceof ControlTunnel.SyncCustomMessageSender) {
+ dataBodies =
((ControlTunnel.SyncCustomMessageSender)cmd).getDataBodies();
+ outcomeListener =
((ControlTunnel.SyncCustomMessageSender)cmd).getOutcomeListener();
+} else {
+ throw new UnsupportedOperationException("Unknown Custom Type control
message received");
+}
+
+DrillBuf reqDrillBuff;
+try {
+ reqDrillBuff = convertToByteBuf(dataBodies);
+} catch (Exception ex) {
+ outcomeListener.failed(new RpcException("Failed to allocate memory
while sending request in " +
+"LocalControlConnectionManager#convertToByteBuff", ex));
+ return;
+} finally {
+ releaseByteBuf(dataBodies);
+}
+
+try {
+ BitControl.CustomMessage message = (BitControl.CustomMessage)
cmd.getMessage();
+ final Response response =
messageHandler.getHandlerRegistry().handle(message, reqDrillBuff
[jira] [Commented] (DRILL-6255) Drillbit while sending control message to itself creates a connection instead of submitting locally
[
https://issues.apache.org/jira/browse/DRILL-6255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16474299#comment-16474299
]
ASF GitHub Bot commented on DRILL-6255:
---
arina-ielchiieva commented on a change in pull request #1253: DRILL-6255:
Drillbit while sending control message to itself creates …
URL: https://github.com/apache/drill/pull/1253#discussion_r187950092
##
File path: exec/rpc/src/main/java/org/apache/drill/exec/rpc/RpcCommand.java
##
@@ -17,10 +17,16 @@
*/
package org.apache.drill.exec.rpc;
+import com.google.protobuf.Internal.EnumLite;
import com.google.protobuf.MessageLite;
public interface RpcCommand
extends RpcConnectionHandler{
Review comment:
Space before `{`.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Drillbit while sending control message to itself creates a connection instead
> of submitting locally
> ---
>
> Key: DRILL-6255
> URL: https://issues.apache.org/jira/browse/DRILL-6255
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Flow
>Affects Versions: 1.12.0
>Reporter: Sorabh Hamirwasia
>Assignee: Sorabh Hamirwasia
>Priority: Major
> Fix For: 1.14.0
>
>
> With the new shutdown feature introduced in 1.12, there is a state introduced
> in DrillbitEndpoint. Due to this the equality check happening
> [here|https://github.com/apache/drill/blob/master/exec/java-exec/src/main/java/org/apache/drill/exec/work/foreman/FragmentsRunner.java#L256]
> will result in false and hence the fragments supposed to be scheduled on
> Foreman will be treated as remote fragments and a connection will be created
> to schedule it. The equality check is false because localEndpoint state is
> STARTUP whereas state in assigned Drillbit is ONLINE.
> I guess now we should update the equality check to verify just for address
> and control port to be same between assigned and local Drillbit endpoint. A
> test can be added for this based on _sys.memory_ table since that guarantees
> scheduling minor fragments on each Drillbit node.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-6255) Drillbit while sending control message to itself creates a connection instead of submitting locally
[
https://issues.apache.org/jira/browse/DRILL-6255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16474295#comment-16474295
]
ASF GitHub Bot commented on DRILL-6255:
---
arina-ielchiieva commented on a change in pull request #1253: DRILL-6255:
Drillbit while sending control message to itself creates …
URL: https://github.com/apache/drill/pull/1253#discussion_r187971417
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/rpc/control/LocalControlConnectionManager.java
##
@@ -0,0 +1,236 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.rpc.control;
+
+import io.netty.buffer.ByteBuf;
+import io.netty.buffer.DrillBuf;
+import org.apache.drill.exec.exception.OutOfMemoryException;
+import org.apache.drill.exec.proto.BitControl;
+import org.apache.drill.exec.proto.CoordinationProtos.DrillbitEndpoint;
+import org.apache.drill.exec.proto.GeneralRPCProtos.Ack;
+import org.apache.drill.exec.proto.UserBitShared;
+import org.apache.drill.exec.rpc.BasicClient;
+import org.apache.drill.exec.rpc.Response;
+import org.apache.drill.exec.rpc.RpcCommand;
+import org.apache.drill.exec.rpc.RpcConstants;
+import org.apache.drill.exec.rpc.RpcException;
+import org.apache.drill.exec.rpc.RpcOutcomeListener;
+import org.apache.drill.exec.work.batch.ControlMessageHandler;
+
+public class LocalControlConnectionManager extends ControlConnectionManager {
+ private static final org.slf4j.Logger logger =
org.slf4j.LoggerFactory.getLogger(LocalControlConnectionManager.class);
+
+ private final ControlConnectionConfig config;
+
+ public LocalControlConnectionManager(ControlConnectionConfig config,
DrillbitEndpoint localEndpoint) {
+super(localEndpoint, localEndpoint);
+this.config = config;
+ }
+
+ @Override
+ protected BasicClient getNewClient() {
+throw new UnsupportedOperationException("LocalControlConnectionManager
doesn't support creating a control client");
+ }
+
+ @Override
+ public void runCommand(RpcCommand cmd) {
+final int rpcType = cmd.getRpcType().getNumber();
+final ControlMessageHandler messageHandler = config.getMessageHandler();
+
+if (RpcConstants.EXTRA_DEBUGGING) {
+ logger.debug("Received bit com message of type {} over local connection
manager", rpcType);
+}
+
+switch (rpcType) {
+
+ case BitControl.RpcType.REQ_CANCEL_FRAGMENT_VALUE: {
+final ControlTunnel.SignalFragment signalFragment =
((ControlTunnel.SignalFragment) cmd);
+final RpcOutcomeListener outcomeListener =
signalFragment.getOutcomeListener();
+final Ack ackResponse =
messageHandler.cancelFragment(signalFragment.getMessage());
+outcomeListener.success(ackResponse, null);
+break;
+ }
+
+ case BitControl.RpcType.REQ_CUSTOM_VALUE: {
+final ByteBuf[] dataBodies;
+final RpcOutcomeListener outcomeListener;
+
+if (cmd instanceof ControlTunnel.CustomMessageSender) {
+ dataBodies =
((ControlTunnel.CustomMessageSender)cmd).getDataBodies();
+ outcomeListener =
((ControlTunnel.CustomMessageSender)cmd).getOutcomeListener();
+} else if (cmd instanceof ControlTunnel.SyncCustomMessageSender) {
+ dataBodies =
((ControlTunnel.SyncCustomMessageSender)cmd).getDataBodies();
+ outcomeListener =
((ControlTunnel.SyncCustomMessageSender)cmd).getOutcomeListener();
+} else {
+ throw new UnsupportedOperationException("Unknown Custom Type control
message received");
+}
+
+DrillBuf reqDrillBuff;
+try {
+ reqDrillBuff = convertToByteBuf(dataBodies);
+} catch (Exception ex) {
+ outcomeListener.failed(new RpcException("Failed to allocate memory
while sending request in " +
+"LocalControlConnectionManager#convertToByteBuff", ex));
+ return;
+} finally {
+ releaseByteBuf(dataBodies);
+}
+
+try {
+ BitControl.CustomMessage message = (BitControl.CustomMessage)
cmd.getMessage();
+ final Response response =
messageHandler.getHandlerRegistry().handle(message, reqDrillBuff
[jira] [Commented] (DRILL-6255) Drillbit while sending control message to itself creates a connection instead of submitting locally
[
https://issues.apache.org/jira/browse/DRILL-6255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16474294#comment-16474294
]
ASF GitHub Bot commented on DRILL-6255:
---
arina-ielchiieva commented on a change in pull request #1253: DRILL-6255:
Drillbit while sending control message to itself creates …
URL: https://github.com/apache/drill/pull/1253#discussion_r187965870
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/work/batch/ControlMessageHandler.java
##
@@ -254,6 +249,32 @@ private Ack receivingFragmentFinished(final
FinishedReceiver finishedReceiver) {
return Acks.OK;
}
+ public Ack requestFragmentStatus(FragmentStatus status) {
+bee.getContext().getWorkBus().statusUpdate( status);
+return Acks.OK;
+ }
+
+ public Ack requestQueryCancel(QueryId queryId) {
+return bee.cancelForeman(queryId, null) ? Acks.OK : Acks.FAIL;
+ }
+
+ public Ack initializeFragment(InitializeFragments fragments) throws
RpcException {
+final DrillbitContext drillbitContext = bee.getContext();
+for(int i = 0; i < fragments.getFragmentCount(); i++) {
Review comment:
Space after for.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Drillbit while sending control message to itself creates a connection instead
> of submitting locally
> ---
>
> Key: DRILL-6255
> URL: https://issues.apache.org/jira/browse/DRILL-6255
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Flow
>Affects Versions: 1.12.0
>Reporter: Sorabh Hamirwasia
>Assignee: Sorabh Hamirwasia
>Priority: Major
> Fix For: 1.14.0
>
>
> With the new shutdown feature introduced in 1.12, there is a state introduced
> in DrillbitEndpoint. Due to this the equality check happening
> [here|https://github.com/apache/drill/blob/master/exec/java-exec/src/main/java/org/apache/drill/exec/work/foreman/FragmentsRunner.java#L256]
> will result in false and hence the fragments supposed to be scheduled on
> Foreman will be treated as remote fragments and a connection will be created
> to schedule it. The equality check is false because localEndpoint state is
> STARTUP whereas state in assigned Drillbit is ONLINE.
> I guess now we should update the equality check to verify just for address
> and control port to be same between assigned and local Drillbit endpoint. A
> test can be added for this based on _sys.memory_ table since that guarantees
> scheduling minor fragments on each Drillbit node.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-6255) Drillbit while sending control message to itself creates a connection instead of submitting locally
[
https://issues.apache.org/jira/browse/DRILL-6255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16474293#comment-16474293
]
ASF GitHub Bot commented on DRILL-6255:
---
arina-ielchiieva commented on a change in pull request #1253: DRILL-6255:
Drillbit while sending control message to itself creates …
URL: https://github.com/apache/drill/pull/1253#discussion_r187963287
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/work/foreman/FragmentsRunner.java
##
@@ -180,44 +170,38 @@ private void setupNonRootFragments(final
Collection fragments) thr
foreman.getQueryManager().addFragmentStatusTracker(planFragment, false);
if (planFragment.getLeafFragment()) {
-updateFragmentCollection(planFragment, localDrillbitEndpoint,
localLeafFragmentList, remoteLeafFragmentMap);
+leafFragmentMap.put(planFragment.getAssignment(), planFragment);
} else {
-updateFragmentCollection(planFragment, localDrillbitEndpoint,
localIntFragmentList, remoteIntFragmentMap);
+intFragmentMap.put(planFragment.getAssignment(), planFragment);
}
}
/*
* We need to wait for the intermediates to be sent so that they'll be set
up by the time
- * the leaves start producing data. We'll use this latch to wait for the
responses.
+ * the leaves start producing data. We'll use this latch to wait for the
responses. All the local intermediate
+ * fragments are submitted locally without creating any actual control
connection to itself.
*
* However, in order not to hang the process if any of the RPC requests
fails, we always
* count down (see FragmentSubmitFailures), but we count the number of
failures so that we'll
* know if any submissions did fail.
*/
-scheduleRemoteIntermediateFragments(remoteIntFragmentMap);
-
-// Setup local intermediate fragments
-for (final PlanFragment fragment : localIntFragmentList) {
- startLocalFragment(fragment);
-}
+scheduleRemoteIntermediateFragments(intFragmentMap);
injector.injectChecked(foreman.getQueryContext().getExecutionControls(),
"send-fragments", ForemanException.class);
/*
* Send the remote (leaf) fragments; we don't wait for these. Any problems
will come in through
- * the regular sendListener event delivery.
+ * the regular sendListener event delivery˚. All the local leaf fragments
are submitted locally without creating
+ * any actual control connection to itself.
*/
-for (final DrillbitEndpoint ep : remoteLeafFragmentMap.keySet()) {
- sendRemoteFragments(ep, remoteLeafFragmentMap.get(ep), null, null);
-}
-
-// Setup local leaf fragments
-for (final PlanFragment fragment : localLeafFragmentList) {
- startLocalFragment(fragment);
+for (final DrillbitEndpoint ep : leafFragmentMap.keySet()) {
+ sendRemoteFragments(ep, leafFragmentMap.get(ep), null, null);
}
}
/**
- * Send all the remote fragments belonging to a single target drillbit in
one request.
+ * Send all the remote fragments belonging to a single target drillbit in
one request. If the assignment
Review comment:
I think we should consider renaming: `sendRemoteFragments` ->
`sendLeafFragments`, `scheduleRemoteIntermediateFragments` ->
`scheduleIntermediateFragments`.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Drillbit while sending control message to itself creates a connection instead
> of submitting locally
> ---
>
> Key: DRILL-6255
> URL: https://issues.apache.org/jira/browse/DRILL-6255
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Flow
>Affects Versions: 1.12.0
>Reporter: Sorabh Hamirwasia
>Assignee: Sorabh Hamirwasia
>Priority: Major
> Fix For: 1.14.0
>
>
> With the new shutdown feature introduced in 1.12, there is a state introduced
> in DrillbitEndpoint. Due to this the equality check happening
> [here|https://github.com/apache/drill/blob/master/exec/java-exec/src/main/java/org/apache/drill/exec/work/foreman/FragmentsRunner.java#L256]
> will result in false and hence the fragments supposed to be scheduled on
> Foreman will be treated as remote fragments and a connection will be created
> to schedule it. The equality check is false because localEndpoint state is
> STARTUP whereas state in assigned Drillbit is ONLINE.
> I guess now we should update the equality check to veri
[jira] [Commented] (DRILL-6255) Drillbit while sending control message to itself creates a connection instead of submitting locally
[
https://issues.apache.org/jira/browse/DRILL-6255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16474291#comment-16474291
]
ASF GitHub Bot commented on DRILL-6255:
---
arina-ielchiieva commented on a change in pull request #1253: DRILL-6255:
Drillbit while sending control message to itself creates …
URL: https://github.com/apache/drill/pull/1253#discussion_r187955446
##
File path:
exec/java-exec/src/test/java/org/apache/drill/exec/rpc/control/TestLocalControlConnectionManager.java
##
@@ -0,0 +1,304 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.rpc.control;
+
+import io.netty.buffer.ByteBuf;
+import org.apache.drill.exec.proto.BitControl;
+import org.apache.drill.exec.proto.CoordinationProtos.DrillbitEndpoint;
+import org.apache.drill.exec.proto.ExecProtos;
+import org.apache.drill.exec.proto.GeneralRPCProtos;
+import org.apache.drill.exec.proto.UserBitShared;
+import org.apache.drill.exec.rpc.Acks;
+import org.apache.drill.exec.rpc.RpcException;
+import org.apache.drill.exec.rpc.RpcOutcomeListener;
+import org.apache.drill.exec.work.batch.ControlMessageHandler;
+import org.junit.Before;
+import org.junit.BeforeClass;
+import org.junit.Test;
+
+import java.util.concurrent.CountDownLatch;
+
+import static junit.framework.TestCase.fail;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertTrue;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+
+public class TestLocalControlConnectionManager {
+
+ private static final DrillbitEndpoint localEndpoint =
DrillbitEndpoint.newBuilder()
+.setAddress("10.0.0.1")
+.setControlPort(31011)
+.setState(DrillbitEndpoint.State.STARTUP)
+.build();
+
+ private static ControlConnectionConfig mockConfig;
+
+ private static ControlMessageHandler mockHandler;
+
+ private static ControlTunnel controlTunnel;
+
+ private CountDownLatch latch;
+
+ private final RpcOutcomeListener outcomeListener =
+new RpcOutcomeListener() {
+@Override
+public void failed(RpcException ex) {
+ throw new IllegalStateException(ex);
+}
+
+@Override
+public void success(GeneralRPCProtos.Ack value, ByteBuf buffer) {
+ if (value.getOk()) {
+latch.countDown();
+ } else {
+throw new IllegalStateException("Negative Ack received");
+ }
+}
+
+@Override
+public void interrupted(InterruptedException e) {
+
Review comment:
Why there is no action on interrupted?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Drillbit while sending control message to itself creates a connection instead
> of submitting locally
> ---
>
> Key: DRILL-6255
> URL: https://issues.apache.org/jira/browse/DRILL-6255
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Flow
>Affects Versions: 1.12.0
>Reporter: Sorabh Hamirwasia
>Assignee: Sorabh Hamirwasia
>Priority: Major
> Fix For: 1.14.0
>
>
> With the new shutdown feature introduced in 1.12, there is a state introduced
> in DrillbitEndpoint. Due to this the equality check happening
> [here|https://github.com/apache/drill/blob/master/exec/java-exec/src/main/java/org/apache/drill/exec/work/foreman/FragmentsRunner.java#L256]
> will result in false and hence the fragments supposed to be scheduled on
> Foreman will be treated as remote fragments and a connection will be created
> to schedule it. The equality check is false because localEndpoint state is
> STARTUP whereas state in assigned Drillbit is ONLINE.
> I guess now we should update the equality check to verify just for address
> and control port to be same between assigned and
[jira] [Commented] (DRILL-6255) Drillbit while sending control message to itself creates a connection instead of submitting locally
[
https://issues.apache.org/jira/browse/DRILL-6255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16474297#comment-16474297
]
ASF GitHub Bot commented on DRILL-6255:
---
arina-ielchiieva commented on a change in pull request #1253: DRILL-6255:
Drillbit while sending control message to itself creates …
URL: https://github.com/apache/drill/pull/1253#discussion_r187955079
##
File path:
exec/java-exec/src/test/java/org/apache/drill/exec/rpc/control/TestLocalControlConnectionManager.java
##
@@ -0,0 +1,304 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.rpc.control;
+
+import io.netty.buffer.ByteBuf;
+import org.apache.drill.exec.proto.BitControl;
+import org.apache.drill.exec.proto.CoordinationProtos.DrillbitEndpoint;
+import org.apache.drill.exec.proto.ExecProtos;
+import org.apache.drill.exec.proto.GeneralRPCProtos;
+import org.apache.drill.exec.proto.UserBitShared;
+import org.apache.drill.exec.rpc.Acks;
+import org.apache.drill.exec.rpc.RpcException;
+import org.apache.drill.exec.rpc.RpcOutcomeListener;
+import org.apache.drill.exec.work.batch.ControlMessageHandler;
+import org.junit.Before;
+import org.junit.BeforeClass;
+import org.junit.Test;
+
+import java.util.concurrent.CountDownLatch;
+
+import static junit.framework.TestCase.fail;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertTrue;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+
+public class TestLocalControlConnectionManager {
+
+ private static final DrillbitEndpoint localEndpoint =
DrillbitEndpoint.newBuilder()
+.setAddress("10.0.0.1")
+.setControlPort(31011)
+.setState(DrillbitEndpoint.State.STARTUP)
+.build();
+
+ private static ControlConnectionConfig mockConfig;
+
+ private static ControlMessageHandler mockHandler;
+
+ private static ControlTunnel controlTunnel;
+
+ private CountDownLatch latch;
+
+ private final RpcOutcomeListener outcomeListener =
+new RpcOutcomeListener() {
+@Override
+public void failed(RpcException ex) {
+ throw new IllegalStateException(ex);
+}
+
+@Override
+public void success(GeneralRPCProtos.Ack value, ByteBuf buffer) {
+ if (value.getOk()) {
+latch.countDown();
+ } else {
+throw new IllegalStateException("Negative Ack received");
+ }
+}
+
+@Override
+public void interrupted(InterruptedException e) {
+
+}
+ };
+
+ @BeforeClass
+ public static void setup() {
+mockConfig = mock(ControlConnectionConfig.class);
+final ConnectionManagerRegistry registry = new
ConnectionManagerRegistry(mockConfig);
+registry.setLocalEndpoint(localEndpoint);
+ControlConnectionManager manager =
registry.getConnectionManager(localEndpoint);
+assertTrue(manager instanceof LocalControlConnectionManager);
+controlTunnel = new ControlTunnel(manager);
+ }
+
+ @Before
+ public void setupForTest() {
+mockHandler = mock(ControlMessageHandler.class);
+when(mockConfig.getMessageHandler()).thenReturn(mockHandler);
+ }
+
+ /**
+ * Verify that SendFragmentStatus is handled correctly using ControlTunnel
with LocalControlConnectionManager
+ */
+ @Test
+ public void testLocalSendFragmentStatus_Success() {
+final UserBitShared.QueryId mockQueryId =
UserBitShared.QueryId.getDefaultInstance();
+final UserBitShared.QueryProfile mockProfile =
UserBitShared.QueryProfile.getDefaultInstance();
+try {
Review comment:
Why do we need try catch here? And in similar tests below...
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Drillbit while sending control message to itself creates a connection instead
> of submitting locally
> ---
>
> Key: DRILL-6255
>
[jira] [Commented] (DRILL-6409) Query Failed: An Error Occurred
[ https://issues.apache.org/jira/browse/DRILL-6409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16474264#comment-16474264 ] JUDILSON JOSE DA COSTA JUNIOR commented on DRILL-6409: -- Version Drill 1.13.0 > Query Failed: An Error Occurred > --- > > Key: DRILL-6409 > URL: https://issues.apache.org/jira/browse/DRILL-6409 > Project: Apache Drill > Issue Type: Bug >Reporter: JUDILSON JOSE DA COSTA JUNIOR >Priority: Major > Attachments: Test.rar, erro_drill.png, > erro_drill_det_mongo-java-driver-3.2.0.png, > erro_drill_det_mongo-java-driver-3.7.0.png > > > h2. Query Failed: An Error Occurred > org.apache.drill.common.exceptions.UserRemoteException: INTERNAL_ERROR ERROR: > UnSupported Bson type: DECIMAL128 Fragment 0:0 [Error Id: > 35c951e0-3ce7-4232-8a90-1c146ecc749f on DSK-0244.eicon.com.br:31010] > Verison mongo-java-driver: 3.7.0 > Version mongo: 3.4 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-6409) Query Failed: An Error Occurred
[ https://issues.apache.org/jira/browse/DRILL-6409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16474265#comment-16474265 ] JUDILSON JOSE DA COSTA JUNIOR commented on DRILL-6409: -- Version Mongo 3.4.9 > Query Failed: An Error Occurred > --- > > Key: DRILL-6409 > URL: https://issues.apache.org/jira/browse/DRILL-6409 > Project: Apache Drill > Issue Type: Bug >Reporter: JUDILSON JOSE DA COSTA JUNIOR >Priority: Major > Attachments: Test.rar, erro_drill.png, > erro_drill_det_mongo-java-driver-3.2.0.png, > erro_drill_det_mongo-java-driver-3.7.0.png > > > h2. Query Failed: An Error Occurred > org.apache.drill.common.exceptions.UserRemoteException: INTERNAL_ERROR ERROR: > UnSupported Bson type: DECIMAL128 Fragment 0:0 [Error Id: > 35c951e0-3ce7-4232-8a90-1c146ecc749f on DSK-0244.eicon.com.br:31010] > Verison mongo-java-driver: 3.7.0 > Version mongo: 3.4 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-6409) Query Failed: An Error Occurred
[ https://issues.apache.org/jira/browse/DRILL-6409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16474263#comment-16474263 ] JUDILSON JOSE DA COSTA JUNIOR commented on DRILL-6409: -- Attachment query and print of the screens, I also put a backup of the collection > Query Failed: An Error Occurred > --- > > Key: DRILL-6409 > URL: https://issues.apache.org/jira/browse/DRILL-6409 > Project: Apache Drill > Issue Type: Bug >Reporter: JUDILSON JOSE DA COSTA JUNIOR >Priority: Major > Attachments: Test.rar, erro_drill.png, > erro_drill_det_mongo-java-driver-3.2.0.png, > erro_drill_det_mongo-java-driver-3.7.0.png > > > h2. Query Failed: An Error Occurred > org.apache.drill.common.exceptions.UserRemoteException: INTERNAL_ERROR ERROR: > UnSupported Bson type: DECIMAL128 Fragment 0:0 [Error Id: > 35c951e0-3ce7-4232-8a90-1c146ecc749f on DSK-0244.eicon.com.br:31010] > Verison mongo-java-driver: 3.7.0 > Version mongo: 3.4 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (DRILL-6409) Query Failed: An Error Occurred
[ https://issues.apache.org/jira/browse/DRILL-6409?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] JUDILSON JOSE DA COSTA JUNIOR updated DRILL-6409: - Attachment: Test.rar > Query Failed: An Error Occurred > --- > > Key: DRILL-6409 > URL: https://issues.apache.org/jira/browse/DRILL-6409 > Project: Apache Drill > Issue Type: Bug >Reporter: JUDILSON JOSE DA COSTA JUNIOR >Priority: Major > Attachments: Test.rar, erro_drill.png, > erro_drill_det_mongo-java-driver-3.2.0.png, > erro_drill_det_mongo-java-driver-3.7.0.png > > > h2. Query Failed: An Error Occurred > org.apache.drill.common.exceptions.UserRemoteException: INTERNAL_ERROR ERROR: > UnSupported Bson type: DECIMAL128 Fragment 0:0 [Error Id: > 35c951e0-3ce7-4232-8a90-1c146ecc749f on DSK-0244.eicon.com.br:31010] > Verison mongo-java-driver: 3.7.0 > Version mongo: 3.4 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (DRILL-6409) Query Failed: An Error Occurred
[ https://issues.apache.org/jira/browse/DRILL-6409?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] JUDILSON JOSE DA COSTA JUNIOR updated DRILL-6409: - Attachment: erro_drill_det_mongo-java-driver-3.7.0.png > Query Failed: An Error Occurred > --- > > Key: DRILL-6409 > URL: https://issues.apache.org/jira/browse/DRILL-6409 > Project: Apache Drill > Issue Type: Bug >Reporter: JUDILSON JOSE DA COSTA JUNIOR >Priority: Major > Attachments: erro_drill.png, > erro_drill_det_mongo-java-driver-3.2.0.png, > erro_drill_det_mongo-java-driver-3.7.0.png > > > h2. Query Failed: An Error Occurred > org.apache.drill.common.exceptions.UserRemoteException: INTERNAL_ERROR ERROR: > UnSupported Bson type: DECIMAL128 Fragment 0:0 [Error Id: > 35c951e0-3ce7-4232-8a90-1c146ecc749f on DSK-0244.eicon.com.br:31010] > Verison mongo-java-driver: 3.7.0 > Version mongo: 3.4 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (DRILL-6409) Query Failed: An Error Occurred
[ https://issues.apache.org/jira/browse/DRILL-6409?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] JUDILSON JOSE DA COSTA JUNIOR updated DRILL-6409: - Attachment: erro_drill_det_mongo-java-driver-3.2.0.png > Query Failed: An Error Occurred > --- > > Key: DRILL-6409 > URL: https://issues.apache.org/jira/browse/DRILL-6409 > Project: Apache Drill > Issue Type: Bug >Reporter: JUDILSON JOSE DA COSTA JUNIOR >Priority: Major > Attachments: erro_drill.png, > erro_drill_det_mongo-java-driver-3.2.0.png, > erro_drill_det_mongo-java-driver-3.7.0.png > > > h2. Query Failed: An Error Occurred > org.apache.drill.common.exceptions.UserRemoteException: INTERNAL_ERROR ERROR: > UnSupported Bson type: DECIMAL128 Fragment 0:0 [Error Id: > 35c951e0-3ce7-4232-8a90-1c146ecc749f on DSK-0244.eicon.com.br:31010] > Verison mongo-java-driver: 3.7.0 > Version mongo: 3.4 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (DRILL-6342) Parquet filter pushdown doesn't work in case of filtering fields inside arrays of complex fields
[
https://issues.apache.org/jira/browse/DRILL-6342?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anton Gozhiy closed DRILL-6342.
---
Verified with Drill version 1.14.0-SNAPSHOT, commit id:
24193b1b038a6315681a65c76a67034b64f71fc5
> Parquet filter pushdown doesn't work in case of filtering fields inside
> arrays of complex fields
>
>
> Key: DRILL-6342
> URL: https://issues.apache.org/jira/browse/DRILL-6342
> Project: Apache Drill
> Issue Type: Bug
>Affects Versions: 1.14.0
>Reporter: Anton Gozhiy
>Assignee: Arina Ielchiieva
>Priority: Major
> Labels: ready-to-commit
> Fix For: 1.14.0
>
> Attachments: Complex_data.tar.gz
>
>
> *Data:*
> Complex_data data set is attached
> *Query:*
> {code:sql}
> explain plan for select * from dfs.tmp.`Complex_data` t where
> t.list_of_complex_fields[2].nested_field is true
> {code}
> *Expected result:*
> numFiles=2
> Statistics of the file that should't be scanned:
> {noformat}
> list_of_complex_fields:
> .nested_field: BOOLEAN UNCOMPRESSED DO:0 FPO:497
> SZ:41/41/1.00 VC:3 ENC:PLAIN,RLE ST:[min: false, max: false, num_nulls: 0]
> {noformat}
> *Actual result:*
> numFiles=3
> I.e, filter pushdown is not work
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (DRILL-6413) Specific query returns an exception if filter a boolean column by "equals" operator
[
https://issues.apache.org/jira/browse/DRILL-6413?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anton Gozhiy updated DRILL-6413:
Description:
*Data:*
Use the attached dataset
*Query:*
{code:sql}
select *
from dfs.tmp.`Test_data`
where bool_col = true and part_col in ('Partition_two')
{code}
*Expected result:*
The query should return result normally
*Actual result:*
Exception happens:
{noformat}
Error: SYSTEM ERROR: ClassCastException:
org.apache.drill.common.expression.TypedFieldExpr cannot be cast to
org.apache.drill.exec.expr.stat.ParquetFilterPredicate
{noformat}
*Notes:*
It works OK if use "is" operator or if not use "*" in the select statement
was:
*Data:*
Use the attached dataset
*Query:*
select *
from dfs.tmp.`Test_data`
where bool_col = true and part_col in ('Partition_two')
*Expected result:*
The query should return result normally
*Actual result:*
Exception happens:
{noformat}
Error: SYSTEM ERROR: ClassCastException:
org.apache.drill.common.expression.TypedFieldExpr cannot be cast to
org.apache.drill.exec.expr.stat.ParquetFilterPredicate
{noformat}
*Notes:*
It works OK if use "is" operator or if not use "*" in the select statement
> Specific query returns an exception if filter a boolean column by "equals"
> operator
> ---
>
> Key: DRILL-6413
> URL: https://issues.apache.org/jira/browse/DRILL-6413
> Project: Apache Drill
> Issue Type: Bug
>Reporter: Anton Gozhiy
>Priority: Major
> Attachments: Test_data.tar.gz
>
>
> *Data:*
> Use the attached dataset
> *Query:*
> {code:sql}
> select *
> from dfs.tmp.`Test_data`
> where bool_col = true and part_col in ('Partition_two')
> {code}
> *Expected result:*
> The query should return result normally
> *Actual result:*
> Exception happens:
> {noformat}
> Error: SYSTEM ERROR: ClassCastException:
> org.apache.drill.common.expression.TypedFieldExpr cannot be cast to
> org.apache.drill.exec.expr.stat.ParquetFilterPredicate
> {noformat}
> *Notes:*
> It works OK if use "is" operator or if not use "*" in the select statement
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (DRILL-6413) Specific query returns an exception if filter a boolean column by "equals" operator
Anton Gozhiy created DRILL-6413:
---
Summary: Specific query returns an exception if filter a boolean
column by "equals" operator
Key: DRILL-6413
URL: https://issues.apache.org/jira/browse/DRILL-6413
Project: Apache Drill
Issue Type: Bug
Reporter: Anton Gozhiy
Attachments: Test_data.tar.gz
*Data:*
Use the attached dataset
*Query:*
select *
from dfs.tmp.`Test_data`
where bool_col = true and part_col in ('Partition_two')
*Expected result:*
The query should return result normally
*Actual result:*
Exception happens:
{noformat}
Error: SYSTEM ERROR: ClassCastException:
org.apache.drill.common.expression.TypedFieldExpr cannot be cast to
org.apache.drill.exec.expr.stat.ParquetFilterPredicate
{noformat}
*Notes:*
It works OK if use "is" operator or if not use "*" in the select statement
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
