[
https://issues.apache.org/jira/browse/DRILL-6340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16514647#comment-16514647
]
ASF GitHub Bot commented on DRILL-6340:
---------------------------------------
Ben-Zvi commented on a change in pull request #1302: DRILL-6340: Output Batch
Control in Project using the RecordBatchSizer
URL: https://github.com/apache/drill/pull/1302#discussion_r195890686
##########
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/project/OutputWidthVisitor.java
##########
@@ -0,0 +1,278 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.drill.exec.physical.impl.project;
+
+import com.google.common.base.Preconditions;
+import org.apache.drill.common.expression.FunctionHolderExpression;
+import org.apache.drill.common.expression.IfExpression;
+import org.apache.drill.common.expression.LogicalExpression;
+import org.apache.drill.common.expression.TypedNullConstant;
+import org.apache.drill.common.expression.ValueExpressions;
+import org.apache.drill.common.types.TypeProtos.MajorType;
+import org.apache.drill.common.types.Types;
+import org.apache.drill.exec.expr.AbstractExecExprVisitor;
+import org.apache.drill.exec.expr.DrillFuncHolderExpr;
+import org.apache.drill.exec.expr.ValueVectorReadExpression;
+import org.apache.drill.exec.expr.ValueVectorWriteExpression;
+import org.apache.drill.exec.expr.annotations.FunctionTemplate;
+import org.apache.drill.exec.expr.fn.DrillFuncHolder;
+import org.apache.drill.exec.expr.fn.output.OutputWidthCalculator;
+import
org.apache.drill.exec.physical.impl.project.OutputWidthExpression.FixedLenExpr;
+import
org.apache.drill.exec.physical.impl.project.OutputWidthExpression.FunctionCallExpr;
+import
org.apache.drill.exec.physical.impl.project.OutputWidthExpression.VarLenReadExpr;
+import
org.apache.drill.exec.physical.impl.project.OutputWidthExpression.IfElseWidthExpr;
+import
org.apache.drill.common.expression.ValueExpressions.VarDecimalExpression;
+import org.apache.drill.exec.record.RecordBatchSizer;
+import org.apache.drill.exec.record.TypedFieldId;
+
+import java.util.ArrayList;
+
+public class OutputWidthVisitor extends
AbstractExecExprVisitor<OutputWidthExpression, OutputWidthVisitorState,
+ RuntimeException> {
+
+ @Override
+ public OutputWidthExpression visitVarDecimalConstant(VarDecimalExpression
varDecimalExpression,
+
OutputWidthVisitorState state) throws RuntimeException {
+
Preconditions.checkArgument(varDecimalExpression.getMajorType().hasPrecision());
+ return new
FixedLenExpr(varDecimalExpression.getMajorType().getPrecision());
+ }
+
+
+ /**
+ *
+ * Records the {@link IfExpression} as a {@link IfElseWidthExpr}.
IfElseWidthExpr will be reduced to
+ * a {@link FixedLenExpr} by taking the max of the if-expr-width and the
else-expr-width.
+ *
+ * @param ifExpression
+ * @param state
+ * @return IfElseWidthExpr
+ * @throws RuntimeException
+ */
+ @Override
+ public OutputWidthExpression visitIfExpression(IfExpression ifExpression,
OutputWidthVisitorState state)
+ throws
RuntimeException {
+ IfExpression.IfCondition condition = ifExpression.ifCondition;
+ LogicalExpression ifExpr = condition.expression;
+ LogicalExpression elseExpr = ifExpression.elseExpression;
+
+ OutputWidthExpression ifWidthExpr = ifExpr.accept(this, state);
+ OutputWidthExpression elseWidthExpr = null;
+ if (elseExpr != null) {
+ elseWidthExpr = elseExpr.accept(this, state);
+ }
+ return new IfElseWidthExpr(ifWidthExpr, elseWidthExpr);
+ }
+
+ /**
+ * Handles a {@link FunctionHolderExpression}. Functions that produce
fixed-width output are trivially
+ * converted to a {@link FixedLenExpr}. For functions that produce
variable width output, the output width calculator
+ * annotation is looked-up and recorded in a {@link FunctionCallExpr}.
This calculator will later be used to convert
+ * the FunctionCallExpr to a {@link FixedLenExpr} expression
+ * @param holderExpr
+ * @param state
+ * @return FunctionCallExpr
+ * @throws RuntimeException
+ */
+ @Override
+ public OutputWidthExpression
visitFunctionHolderExpression(FunctionHolderExpression holderExpr,
+
OutputWidthVisitorState state) throws RuntimeException {
+ OutputWidthExpression fixedWidth =
getFixedLenExpr(holderExpr.getMajorType());
+ if (fixedWidth != null) { return fixedWidth; }
+ // Only Drill functions can be handled. Non-drill Functions, like
HiveFunctions
+ // will default to a fixed value
+ if (!(holderExpr instanceof DrillFuncHolderExpr)) {
+ // We currently only know how to handle DrillFuncs.
+ // Use a default if this is not a DrillFunc
+ return new
FixedLenExpr(OutputSizeEstimateConstants.NON_DRILL_FUNCTION_OUTPUT_SIZE_ESTIMATE);
+ }
+
+ final DrillFuncHolder holder = ((DrillFuncHolderExpr)
holderExpr).getHolder();
+
+ // If the user has provided a size estimate, use it
+ int estimate = holder.variableOuputSizeEstimate();
+ if (estimate !=
FunctionTemplate.VARIABLE_OUTPUT_SIZE_ESTIMATE_DEFAULT) {
+ return new FixedLenExpr(estimate);
+ }
+ // Use the calculator provided by the user or use the default
+ OutputWidthCalculator widthCalculator =
holder.getOutputWidthCalculator();
Review comment:
Could the output-width-calculator be uninitialized ? Or does the
FunctionTemplate always initializes it to the DEFAULT ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Output Batch Control in Project using the RecordBatchSizer
> ----------------------------------------------------------
>
> Key: DRILL-6340
> URL: https://issues.apache.org/jira/browse/DRILL-6340
> Project: Apache Drill
> Issue Type: Improvement
> Components: Execution - Relational Operators
> Reporter: Karthikeyan Manivannan
> Assignee: Karthikeyan Manivannan
> Priority: Major
> Labels: doc-impacting
> Fix For: 1.14.0
>
>
> This bug is for tracking the changes required to implement Output Batch
> Sizing in Project using the RecordBatchSizer. The challenge in doing this
> mainly lies in dealing with expressions that produce variable-length columns.
> The following doc talks about some of the design approaches for dealing with
> such variable-length columns.
> [https://docs.google.com/document/d/1h0WsQsen6xqqAyyYSrtiAniQpVZGmQNQqC1I2DJaxAA/edit?usp=sharing]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
