Re: Review Request 31107: Ability to make PartitionSender multithreaded - useful in case of LocalExchange being enabled, as it allows to deal with high volume of incoming data
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/31107/ --- (Updated March 6, 2015, 2:59 p.m.) Review request for drill, Chris Westin, Jacques Nadeau, Steven Phillips, and Venki Korukanti. Changes --- Added Unit tests Added number of threads as a metric to PartitionSender operator Fixed partitioners distribution algorithm to do even distribution between threads Bugs: DRILL-2210 https://issues.apache.org/jira/browse/DRILL-2210 Repository: drill-git Description --- In addition to description Fixed few classes that did not handle multithreading well Added/Changed some Stats behavior to allow stats merge from multiple threads, since again this class is not suitable to be used in multithreaded environment Introduced new decorator class to handle multi thrteading (or not) to minimize changes to ParitionSenderRootExec class Diffs (updated) - exec/java-exec/src/main/java/org/apache/drill/exec/compile/CodeCompiler.java 7cc350e exec/java-exec/src/main/java/org/apache/drill/exec/ops/FragmentContext.java e413921 exec/java-exec/src/main/java/org/apache/drill/exec/ops/OperatorStats.java 0e9da0e exec/java-exec/src/main/java/org/apache/drill/exec/physical/config/IteratorValidator.java 64cf7c5 exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/SendingAccountor.java 7af7b65 exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/partitionsender/PartitionSenderRootExec.java a23bd7a exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/partitionsender/Partitioner.java 5ed9c39 exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/partitionsender/PartitionerDecorator.java PRE-CREATION exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/partitionsender/PartitionerTemplate.java 71ffd41 exec/java-exec/src/main/java/org/apache/drill/exec/planner/fragment/Materializer.java 961b603 exec/java-exec/src/main/java/org/apache/drill/exec/planner/physical/PlannerSettings.java bbfbbcb exec/java-exec/src/main/java/org/apache/drill/exec/server/DrillbitContext.java 0fb10ff exec/java-exec/src/main/java/org/apache/drill/exec/server/options/SystemOptionManager.java 3d3e96f exec/java-exec/src/main/java/org/apache/drill/exec/work/WorkManager.java 99c6ab8 exec/java-exec/src/test/java/org/apache/drill/exec/physical/impl/TestOptiqPlans.java 478 exec/java-exec/src/test/java/org/apache/drill/exec/physical/impl/partitionsender/TestPartitionSender.java PRE-CREATION Diff: https://reviews.apache.org/r/31107/diff/ Testing --- Still need to provide Unit Tests. Functional tests are passing Performance tests were run and look promising for some queries Thanks, Yuliya Feldman
Re: Review Request 31707: DRILL-2207: New Union-All Implementation
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/31707/ --- (Updated March 7, 2015, 12:49 a.m.) Review request for drill, Aman Sinha and Jinfeng Ni. Changes --- fix Bugs: DRILL-2207 https://issues.apache.org/jira/browse/DRILL-2207 Repository: drill-git Description --- DRILL-2207: New Union-All Implementation Diffs (updated) - exec/java-exec/src/main/java/org/apache/drill/exec/expr/ExpressionTreeMaterializer.java 3565bf4 exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/union/UnionAllRecordBatch.java 99aec92 exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/union/UnionAller.java PRE-CREATION exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/union/UnionAllerTemplate.java PRE-CREATION exec/java-exec/src/main/java/org/apache/drill/exec/planner/common/DrillRelOptUtil.java 270462b exec/java-exec/src/main/java/org/apache/drill/exec/planner/logical/PreProcessLogicalRel.java 4c9d301 exec/java-exec/src/main/java/org/apache/drill/exec/planner/physical/visitor/FinalColumnReorderer.java 60a9e4b exec/java-exec/src/main/java/org/apache/drill/exec/planner/sql/parser/UnsupportedOperatorsVisitor.java dcd5ebf exec/java-exec/src/main/java/org/apache/drill/exec/record/VectorContainer.java baf74b1 exec/java-exec/src/main/java/org/apache/drill/exec/resolver/TypeCastRules.java f5b0de4 exec/java-exec/src/test/java/org/apache/drill/TestExampleQueries.java 225b21e exec/java-exec/src/test/java/org/apache/drill/TestUnionAll.java 36b062b exec/java-exec/src/test/resources/store/text/data/t.json PRE-CREATION exec/java-exec/src/test/resources/testframework/testExampleQueries/testAggregationOnUnionAllOperator/q1.tsv PRE-CREATION exec/java-exec/src/test/resources/testframework/testExampleQueries/testAggregationOnUnionAllOperator/q2.tsv PRE-CREATION exec/java-exec/src/test/resources/testframework/testUnionAllQueries/q1.tsv PRE-CREATION exec/java-exec/src/test/resources/testframework/testUnionAllQueries/q10.tsv PRE-CREATION exec/java-exec/src/test/resources/testframework/testUnionAllQueries/q11.tsv PRE-CREATION exec/java-exec/src/test/resources/testframework/testUnionAllQueries/q12.tsv PRE-CREATION exec/java-exec/src/test/resources/testframework/testUnionAllQueries/q13.tsv PRE-CREATION exec/java-exec/src/test/resources/testframework/testUnionAllQueries/q14.tsv PRE-CREATION exec/java-exec/src/test/resources/testframework/testUnionAllQueries/q15.tsv PRE-CREATION exec/java-exec/src/test/resources/testframework/testUnionAllQueries/q16.tsv PRE-CREATION exec/java-exec/src/test/resources/testframework/testUnionAllQueries/q17.tsv PRE-CREATION exec/java-exec/src/test/resources/testframework/testUnionAllQueries/q2.tsv PRE-CREATION exec/java-exec/src/test/resources/testframework/testUnionAllQueries/q3.tsv PRE-CREATION exec/java-exec/src/test/resources/testframework/testUnionAllQueries/q4.tsv PRE-CREATION exec/java-exec/src/test/resources/testframework/testUnionAllQueries/q5.tsv PRE-CREATION exec/java-exec/src/test/resources/testframework/testUnionAllQueries/q6.tsv PRE-CREATION exec/java-exec/src/test/resources/testframework/testUnionAllQueries/q6_1.tsv PRE-CREATION exec/java-exec/src/test/resources/testframework/testUnionAllQueries/q7.tsv PRE-CREATION exec/java-exec/src/test/resources/testframework/testUnionAllQueries/q8.tsv PRE-CREATION exec/java-exec/src/test/resources/testframework/testUnionAllQueries/q9.tsv PRE-CREATION Diff: https://reviews.apache.org/r/31707/diff/ Testing --- Design Doc can be found from: https://issues.apache.org/jira/browse/DRILL-2207 Unit, Customers, TPCH passed waiting for Functional... Thanks, Sean Hsuan-Yi Chu
Re: Review Request 31707: DRILL-2207: New Union-All Implementation
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/31707/ --- (Updated March 7, 2015, 2:28 a.m.) Review request for drill, Aman Sinha and Jinfeng Ni. Changes --- throw DrillRuntime exception if there is no appropriate implicit casting Bugs: DRILL-2207 https://issues.apache.org/jira/browse/DRILL-2207 Repository: drill-git Description --- DRILL-2207: New Union-All Implementation Diffs (updated) - exec/java-exec/src/main/java/org/apache/drill/exec/expr/ExpressionTreeMaterializer.java 3565bf4 exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/union/UnionAllRecordBatch.java 99aec92 exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/union/UnionAller.java PRE-CREATION exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/union/UnionAllerTemplate.java PRE-CREATION exec/java-exec/src/main/java/org/apache/drill/exec/planner/common/DrillRelOptUtil.java 270462b exec/java-exec/src/main/java/org/apache/drill/exec/planner/logical/PreProcessLogicalRel.java 4c9d301 exec/java-exec/src/main/java/org/apache/drill/exec/planner/physical/visitor/FinalColumnReorderer.java 60a9e4b exec/java-exec/src/main/java/org/apache/drill/exec/planner/sql/parser/UnsupportedOperatorsVisitor.java dcd5ebf exec/java-exec/src/main/java/org/apache/drill/exec/record/VectorContainer.java baf74b1 exec/java-exec/src/main/java/org/apache/drill/exec/resolver/TypeCastRules.java f5b0de4 exec/java-exec/src/test/java/org/apache/drill/TestExampleQueries.java 225b21e exec/java-exec/src/test/java/org/apache/drill/TestUnionAll.java 36b062b exec/java-exec/src/test/resources/store/text/data/t.json PRE-CREATION exec/java-exec/src/test/resources/testframework/testExampleQueries/testAggregationOnUnionAllOperator/q1.tsv PRE-CREATION exec/java-exec/src/test/resources/testframework/testExampleQueries/testAggregationOnUnionAllOperator/q2.tsv PRE-CREATION exec/java-exec/src/test/resources/testframework/testUnionAllQueries/q1.tsv PRE-CREATION exec/java-exec/src/test/resources/testframework/testUnionAllQueries/q10.tsv PRE-CREATION exec/java-exec/src/test/resources/testframework/testUnionAllQueries/q11.tsv PRE-CREATION exec/java-exec/src/test/resources/testframework/testUnionAllQueries/q12.tsv PRE-CREATION exec/java-exec/src/test/resources/testframework/testUnionAllQueries/q13.tsv PRE-CREATION exec/java-exec/src/test/resources/testframework/testUnionAllQueries/q14.tsv PRE-CREATION exec/java-exec/src/test/resources/testframework/testUnionAllQueries/q15.tsv PRE-CREATION exec/java-exec/src/test/resources/testframework/testUnionAllQueries/q16.tsv PRE-CREATION exec/java-exec/src/test/resources/testframework/testUnionAllQueries/q17.tsv PRE-CREATION exec/java-exec/src/test/resources/testframework/testUnionAllQueries/q2.tsv PRE-CREATION exec/java-exec/src/test/resources/testframework/testUnionAllQueries/q3.tsv PRE-CREATION exec/java-exec/src/test/resources/testframework/testUnionAllQueries/q4.tsv PRE-CREATION exec/java-exec/src/test/resources/testframework/testUnionAllQueries/q5.tsv PRE-CREATION exec/java-exec/src/test/resources/testframework/testUnionAllQueries/q6.tsv PRE-CREATION exec/java-exec/src/test/resources/testframework/testUnionAllQueries/q6_1.tsv PRE-CREATION exec/java-exec/src/test/resources/testframework/testUnionAllQueries/q7.tsv PRE-CREATION exec/java-exec/src/test/resources/testframework/testUnionAllQueries/q8.tsv PRE-CREATION exec/java-exec/src/test/resources/testframework/testUnionAllQueries/q9.tsv PRE-CREATION Diff: https://reviews.apache.org/r/31707/diff/ Testing --- Design Doc can be found from: https://issues.apache.org/jira/browse/DRILL-2207 Unit, Customers, TPCH passed waiting for Functional... Thanks, Sean Hsuan-Yi Chu
Re: Review Request 31707: DRILL-2207: New Union-All Implementation
On March 6, 2015, 10:45 p.m., Aman Sinha wrote: exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/union/UnionAllRecordBatch.java, line 62 https://reviews.apache.org/r/31707/diff/2/?file=885179#file885179line62 This should ideally be just UnionAll. Is an import missing ? Initially, there is another class (the Runtime Code Generation template for union all) also called unionall, so package name is used to differentiate; Now I change the former one to UnionAller. On March 6, 2015, 10:45 p.m., Aman Sinha wrote: exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/union/UnionAllRecordBatch.java, line 241 https://reviews.apache.org/r/31707/diff/2/?file=885179#file885179line241 Won't this convert to nullable even if the desired output type is non-nullable ? Also, I think in general most of this logic for deciding the casting should live outside of the UnionAll code in some common utility class such that it is used by multiple operators. Move this part of code to ExpressionTreeMaterializer; after all, addCastExpression() method has been in this this class. On March 6, 2015, 10:45 p.m., Aman Sinha wrote: exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/union/UnionAllRecordBatch.java, line 288 https://reviews.apache.org/r/31707/diff/2/?file=885179#file885179line288 Can you run some tests with TPCH SF1 such that we are testing with larger numbers of record batches ? Also test a case where left side has 0 rows, right side has non-zero. 1. [TPCH SF1] A bunch of tests were tested on TPCH SF1; Looked fine 2. [Either side with 0 rows] Worked as we expected; To sum up again, no matter which side has no record, the output schema always follows the left side On March 6, 2015, 10:45 p.m., Aman Sinha wrote: exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/union/UnionAllRecordBatch.java, line 361 https://reviews.apache.org/r/31707/diff/2/?file=885179#file885179line361 You should identify whether the schema change occurred on left or right input. Pls change the error to a clearer one, e.g Schema change detected in left/right input of Union-All. This is not currently supported. More appropriate exception message is given On March 6, 2015, 10:45 p.m., Aman Sinha wrote: exec/java-exec/src/main/java/org/apache/drill/exec/planner/logical/PreProcessLogicalRel.java, line 150 https://reviews.apache.org/r/31707/diff/2/?file=885182#file885182line150 Error message should be clearer: Union-all over schema-less tables must specify the columns explicitly More appropriate exception message is given On March 6, 2015, 10:45 p.m., Aman Sinha wrote: exec/java-exec/src/main/java/org/apache/drill/exec/planner/physical/visitor/FinalColumnReorderer.java, line 51 https://reviews.apache.org/r/31707/diff/2/?file=885183#file885183line51 Is this going to force a Project on the right side even if the column names, ordinals and number of columns are the same as the ones on the left side ? I think we should avoid passing a 'force' flag here. This field is actually necessary. I took it off by adding another if-statement before calling this method - Sean Hsuan-Yi --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/31707/#review75407 --- On March 7, 2015, 12:49 a.m., Sean Hsuan-Yi Chu wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/31707/ --- (Updated March 7, 2015, 12:49 a.m.) Review request for drill, Aman Sinha and Jinfeng Ni. Bugs: DRILL-2207 https://issues.apache.org/jira/browse/DRILL-2207 Repository: drill-git Description --- DRILL-2207: New Union-All Implementation Diffs - exec/java-exec/src/main/java/org/apache/drill/exec/expr/ExpressionTreeMaterializer.java 3565bf4 exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/union/UnionAllRecordBatch.java 99aec92 exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/union/UnionAller.java PRE-CREATION exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/union/UnionAllerTemplate.java PRE-CREATION exec/java-exec/src/main/java/org/apache/drill/exec/planner/common/DrillRelOptUtil.java 270462b exec/java-exec/src/main/java/org/apache/drill/exec/planner/logical/PreProcessLogicalRel.java 4c9d301 exec/java-exec/src/main/java/org/apache/drill/exec/planner/physical/visitor/FinalColumnReorderer.java 60a9e4b exec/java-exec/src/main/java/org/apache/drill/exec/planner/sql/parser/UnsupportedOperatorsVisitor.java dcd5ebf
Re: Review Request 31804: DRILL-2187: Broadcast join planning and shared data distribution
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/31804/#review75578 --- exec/java-exec/src/main/java/org/apache/drill/exec/planner/physical/JoinPruleBase.java https://reviews.apache.org/r/31804/#comment122766 Do we assume Left has larger dataset than Right, and hence always put Broadcast into Right side? The current join order optimization may not always have LEFT RIGHT. exec/java-exec/src/main/java/org/apache/drill/exec/planner/physical/SubsetTransformer.java https://reviews.apache.org/r/31804/#comment122765 If we are passing candidateSet to convertChild, seems to me the for-loop of getRelList() is not necessary. All we need to check is if the candidate has a Physial rel. If yes, then call convertChild() once, since the convertChild() will not take trait from rel node, but candidateSet only. - Jinfeng Ni On March 6, 2015, 11:05 a.m., Jacques Nadeau wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/31804/ --- (Updated March 6, 2015, 11:05 a.m.) Review request for drill, Aman Sinha and Jinfeng Ni. Repository: drill-git Description --- Update the RPC layer so that we can do multi-target broadcast sends. Update planning layer to correctly propagate and explore traits using subsets rather than inner children. Diffs - exec/java-exec/src/main/java/io/netty/buffer/DrillBuf.java 8e9d395 exec/java-exec/src/main/java/io/netty/buffer/FakeAllocator.java 3de0a75 exec/java-exec/src/main/java/org/apache/drill/exec/memory/Accountor.java 2b48ef0 exec/java-exec/src/main/java/org/apache/drill/exec/memory/BufferAllocator.java 83d9d1e exec/java-exec/src/main/java/org/apache/drill/exec/memory/TopLevelAllocator.java 67e1fdb exec/java-exec/src/main/java/org/apache/drill/exec/ops/FragmentContext.java e413921 exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/SingleSenderCreator.java 6db9f4a exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/broadcastsender/BroadcastSenderRootExec.java 22fa047 exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/partitionsender/PartitionSenderRootExec.java a23bd7a exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/partitionsender/PartitionerTemplate.java 71ffd41 exec/java-exec/src/main/java/org/apache/drill/exec/planner/logical/DrillRuleSets.java 3b7adca exec/java-exec/src/main/java/org/apache/drill/exec/planner/physical/BroadcastExchangePrel.java 0c76de4 exec/java-exec/src/main/java/org/apache/drill/exec/planner/physical/DrillDistributionTraitDef.java 6a1dbb7 exec/java-exec/src/main/java/org/apache/drill/exec/planner/physical/HashJoinPrule.java 433405a exec/java-exec/src/main/java/org/apache/drill/exec/planner/physical/JoinPruleBase.java afcbf71 exec/java-exec/src/main/java/org/apache/drill/exec/planner/physical/MergeJoinPrel.java b7e86e3 exec/java-exec/src/main/java/org/apache/drill/exec/planner/physical/MergeJoinPrule.java faffa63 exec/java-exec/src/main/java/org/apache/drill/exec/planner/physical/PlannerSettings.java bbfbbcb exec/java-exec/src/main/java/org/apache/drill/exec/planner/physical/ProjectPrule.java 72034ed exec/java-exec/src/main/java/org/apache/drill/exec/planner/physical/StreamAggPrule.java 89b133a exec/java-exec/src/main/java/org/apache/drill/exec/planner/physical/SubsetTransformer.java d4cd21f exec/java-exec/src/main/java/org/apache/drill/exec/proto/helper/QueryIdHelper.java bb12a22 exec/java-exec/src/main/java/org/apache/drill/exec/record/FragmentWritableBatch.java 3d06806 exec/java-exec/src/main/java/org/apache/drill/exec/record/RawFragmentBatch.java e1725e6 exec/java-exec/src/main/java/org/apache/drill/exec/rpc/RpcBus.java 96c9911 exec/java-exec/src/main/java/org/apache/drill/exec/rpc/data/AckSender.java PRE-CREATION exec/java-exec/src/main/java/org/apache/drill/exec/rpc/data/DataClient.java a9eb66f exec/java-exec/src/main/java/org/apache/drill/exec/rpc/data/DataConnectionCreator.java 197996d exec/java-exec/src/main/java/org/apache/drill/exec/rpc/data/DataConnectionManager.java b2ea855 exec/java-exec/src/main/java/org/apache/drill/exec/rpc/data/DataProtobufLengthDecoder.java 193b050 exec/java-exec/src/main/java/org/apache/drill/exec/rpc/data/DataResponseHandler.java d4a73c3 exec/java-exec/src/main/java/org/apache/drill/exec/rpc/data/DataResponseHandlerImpl.java 1fcb3e9 exec/java-exec/src/main/java/org/apache/drill/exec/rpc/data/DataRpcConfig.java b54841d
Re: Review Request 31707: DRILL-2207: New Union-All Implementation
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/31707/#review75407 --- Sending some initial comments. exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/union/UnionAllRecordBatch.java https://reviews.apache.org/r/31707/#comment122444 This should ideally be just UnionAll. Is an import missing ? exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/union/UnionAllRecordBatch.java https://reviews.apache.org/r/31707/#comment122737 Won't this convert to nullable even if the desired output type is non-nullable ? Also, I think in general most of this logic for deciding the casting should live outside of the UnionAll code in some common utility class such that it is used by multiple operators. exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/union/UnionAllRecordBatch.java https://reviews.apache.org/r/31707/#comment122743 Can you run some tests with TPCH SF1 such that we are testing with larger numbers of record batches ? Also test a case where left side has 0 rows, right side has non-zero. exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/union/UnionAllRecordBatch.java https://reviews.apache.org/r/31707/#comment122729 You should identify whether the schema change occurred on left or right input. Pls change the error to a clearer one, e.g Schema change detected in left/right input of Union-All. This is not currently supported. exec/java-exec/src/main/java/org/apache/drill/exec/planner/logical/PreProcessLogicalRel.java https://reviews.apache.org/r/31707/#comment122538 Error message should be clearer: Union-all over schema-less tables must specify the columns explicitly exec/java-exec/src/main/java/org/apache/drill/exec/planner/physical/visitor/FinalColumnReorderer.java https://reviews.apache.org/r/31707/#comment122728 Is this going to force a Project on the right side even if the column names, ordinals and number of columns are the same as the ones on the left side ? I think we should avoid passing a 'force' flag here. - Aman Sinha On March 5, 2015, 12:27 a.m., Sean Hsuan-Yi Chu wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/31707/ --- (Updated March 5, 2015, 12:27 a.m.) Review request for drill, Aman Sinha and Jinfeng Ni. Bugs: DRILL-2207 https://issues.apache.org/jira/browse/DRILL-2207 Repository: drill-git Description --- DRILL-2207: New Union-All Implementation Diffs - exec/java-exec/src/main/java/org/apache/drill/exec/expr/ExpressionTreeMaterializer.java 3565bf4 exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/union/UnionAll.java PRE-CREATION exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/union/UnionAllRecordBatch.java 99aec92 exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/union/UnionAllTemplate.java PRE-CREATION exec/java-exec/src/main/java/org/apache/drill/exec/planner/common/DrillRelOptUtil.java 270462b exec/java-exec/src/main/java/org/apache/drill/exec/planner/logical/PreProcessLogicalRel.java 4c9d301 exec/java-exec/src/main/java/org/apache/drill/exec/planner/physical/visitor/FinalColumnReorderer.java 60a9e4b exec/java-exec/src/main/java/org/apache/drill/exec/planner/sql/parser/UnsupportedOperatorsVisitor.java dcd5ebf exec/java-exec/src/main/java/org/apache/drill/exec/record/VectorContainer.java baf74b1 exec/java-exec/src/main/java/org/apache/drill/exec/resolver/TypeCastRules.java f5b0de4 exec/java-exec/src/test/java/org/apache/drill/TestExampleQueries.java 225b21e exec/java-exec/src/test/java/org/apache/drill/TestUnionAll.java 36b062b exec/java-exec/src/test/resources/store/text/data/t.json PRE-CREATION exec/java-exec/src/test/resources/testframework/TestUnionAll/q1.tsv PRE-CREATION exec/java-exec/src/test/resources/testframework/TestUnionAll/q10.tsv PRE-CREATION exec/java-exec/src/test/resources/testframework/TestUnionAll/q11.tsv PRE-CREATION exec/java-exec/src/test/resources/testframework/TestUnionAll/q12.tsv PRE-CREATION exec/java-exec/src/test/resources/testframework/TestUnionAll/q13.tsv PRE-CREATION exec/java-exec/src/test/resources/testframework/TestUnionAll/q14.tsv PRE-CREATION exec/java-exec/src/test/resources/testframework/TestUnionAll/q15.tsv PRE-CREATION exec/java-exec/src/test/resources/testframework/TestUnionAll/q2.tsv PRE-CREATION exec/java-exec/src/test/resources/testframework/TestUnionAll/q3.tsv PRE-CREATION exec/java-exec/src/test/resources/testframework/TestUnionAll/q4.tsv PRE-CREATION
Review Request 31804: DRILL-2187: Broadcast join planning and shared data distribution
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/31804/ --- Review request for drill and Aman Sinha. Repository: drill-git Description --- Update the RPC layer so that we can do multi-target broadcast sends. Update planning layer to correctly propagate and explore traits using subsets rather than inner children. Diffs - exec/java-exec/src/main/java/io/netty/buffer/DrillBuf.java 8e9d395 exec/java-exec/src/main/java/io/netty/buffer/FakeAllocator.java 3de0a75 exec/java-exec/src/main/java/org/apache/drill/exec/memory/Accountor.java 2b48ef0 exec/java-exec/src/main/java/org/apache/drill/exec/memory/BufferAllocator.java 83d9d1e exec/java-exec/src/main/java/org/apache/drill/exec/memory/TopLevelAllocator.java 67e1fdb exec/java-exec/src/main/java/org/apache/drill/exec/ops/FragmentContext.java e413921 exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/SingleSenderCreator.java 6db9f4a exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/broadcastsender/BroadcastSenderRootExec.java 22fa047 exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/partitionsender/PartitionSenderRootExec.java a23bd7a exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/partitionsender/PartitionerTemplate.java 71ffd41 exec/java-exec/src/main/java/org/apache/drill/exec/planner/logical/DrillRuleSets.java 3b7adca exec/java-exec/src/main/java/org/apache/drill/exec/planner/physical/BroadcastExchangePrel.java 0c76de4 exec/java-exec/src/main/java/org/apache/drill/exec/planner/physical/DrillDistributionTraitDef.java 6a1dbb7 exec/java-exec/src/main/java/org/apache/drill/exec/planner/physical/HashJoinPrule.java 433405a exec/java-exec/src/main/java/org/apache/drill/exec/planner/physical/JoinPruleBase.java afcbf71 exec/java-exec/src/main/java/org/apache/drill/exec/planner/physical/MergeJoinPrel.java b7e86e3 exec/java-exec/src/main/java/org/apache/drill/exec/planner/physical/MergeJoinPrule.java faffa63 exec/java-exec/src/main/java/org/apache/drill/exec/planner/physical/PlannerSettings.java bbfbbcb exec/java-exec/src/main/java/org/apache/drill/exec/planner/physical/ProjectPrule.java 72034ed exec/java-exec/src/main/java/org/apache/drill/exec/planner/physical/StreamAggPrule.java 89b133a exec/java-exec/src/main/java/org/apache/drill/exec/planner/physical/SubsetTransformer.java d4cd21f exec/java-exec/src/main/java/org/apache/drill/exec/proto/helper/QueryIdHelper.java bb12a22 exec/java-exec/src/main/java/org/apache/drill/exec/record/FragmentWritableBatch.java 3d06806 exec/java-exec/src/main/java/org/apache/drill/exec/record/RawFragmentBatch.java e1725e6 exec/java-exec/src/main/java/org/apache/drill/exec/rpc/RpcBus.java 96c9911 exec/java-exec/src/main/java/org/apache/drill/exec/rpc/data/AckSender.java PRE-CREATION exec/java-exec/src/main/java/org/apache/drill/exec/rpc/data/DataClient.java a9eb66f exec/java-exec/src/main/java/org/apache/drill/exec/rpc/data/DataConnectionCreator.java 197996d exec/java-exec/src/main/java/org/apache/drill/exec/rpc/data/DataConnectionManager.java b2ea855 exec/java-exec/src/main/java/org/apache/drill/exec/rpc/data/DataProtobufLengthDecoder.java 193b050 exec/java-exec/src/main/java/org/apache/drill/exec/rpc/data/DataResponseHandler.java d4a73c3 exec/java-exec/src/main/java/org/apache/drill/exec/rpc/data/DataResponseHandlerImpl.java 1fcb3e9 exec/java-exec/src/main/java/org/apache/drill/exec/rpc/data/DataRpcConfig.java b54841d exec/java-exec/src/main/java/org/apache/drill/exec/rpc/data/DataServer.java e88455b exec/java-exec/src/main/java/org/apache/drill/exec/server/options/SystemOptionManager.java 3d3e96f exec/java-exec/src/main/java/org/apache/drill/exec/work/batch/ResponseSenderQueue.java 141c434 exec/java-exec/src/main/java/org/apache/drill/exec/work/batch/SpoolingRawBatchBuffer.java 6ee93ab exec/java-exec/src/main/java/org/apache/drill/exec/work/batch/UnlimitedRawBatchBuffer.java 895918c exec/java-exec/src/test/java/org/apache/drill/exec/physical/impl/broadcastsender/TestBroadcast.java PRE-CREATION exec/java-exec/src/test/java/org/apache/drill/exec/server/TestBitRpc.java 7b771f5 exec/java-exec/src/test/java/org/apache/drill/exec/work/batch/TestUnlimitedBatchBuffer.java a710d21 exec/java-exec/src/test/resources/broadcast/customer/cust.json PRE-CREATION exec/java-exec/src/test/resources/broadcast/sales/f1.json PRE-CREATION exec/java-exec/src/test/resources/broadcast/sales/f2.json PRE-CREATION protocol/src/main/java/org/apache/drill/exec/proto/BitData.java 37e8a18 protocol/src/main/java/org/apache/drill/exec/proto/SchemaBitData.java f35f489
Re: JDBC data source
Regarding what you call “JDBC storage plugin”. I’d call it a JDBC adapter. Calcite has a JDBC adapter that tries to push down as much of the algebra tree to a foreign database (this depends upon what the database is capable of), then generates database-specific SQL for the tree that it had managed to push. Drill’s requirements are exactly the same so it seems crazy to not start with Calcite’s adapter. Julian On Mar 6, 2015, at 10:05 AM, Norris Lee norr...@simba.com wrote: Close but not quite. Alex and I are actually working on an ODBC driver. An open source JDBC driver already exists and is in active development. As for a JDBC storage plugin to source data from other JDBC drivers (eg. Of RDBMSes), it's been requested several times recently but I don't think anyone is working on it. If I recall correctly, some people expressed interest in working on it. Norris -Original Message- From: Yash Sharma [mailto:yash...@gmail.com] Sent: Friday, March 06, 2015 6:51 AM To: u...@drill.apache.org Subject: Re: JDBC data source Drill currently doesn't have a jdbc storage plugin. However it has come up in discussions couple of times. As discussed in last hangout Norris and Alexander are working on it and we can expect a patch soon. -Via mobile. Excuse brevity. On 6 Mar 2015 8:06 pm, Chevalier Julien j...@numerigraphe.com wrote: Yash, what you pointed out is a way to produce Drill output through a JDBC. I want Drill to consume data from JDBC. As Carol clearly explains what I need is a JDBC Storage Plugin. Precisely are they any JDBC Storage plugin available ? Best regards. Juju Le 6 mars 2015 à 15:18, Yash Sharma yash...@gmail.com a écrit : Hi Juju, check if this can help: https://cwiki.apache.org/confluence/display/DRILL/Using+JDBC+to+Access +Apache+Drill+from+SQuirreL Regards On Fri, Mar 6, 2015 at 7:13 PM, Chevalier Julien j...@numerigraphe.com wrote: Dear List I Wonder if Drill can read data from a RDBMS through a good old JDBC driver Examples are welcome. Best regards Juju
Re: Review Request 31567: 2060 - part 1 - refactor interpreted expression evaluation
On March 5, 2015, 2:55 a.m., Jinfeng Ni wrote: exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/interpreter/InterpreterEvaluator.java, line 320 https://reviews.apache.org/r/31567/diff/2/?file=885225#file885225line320 Do you have a plan to move setup() call into places such that setup() will be called once for each VectorAccessible input? In the code compile + evaluation model, doSetup() will be called per batch, in stead of per row. Jason Altekruse wrote: I have started working on a fix for this. Its a little complicated with setting constant inputs before the setup method is called. I'm trying to figure out the best way to share code with the rest of the input passing used in the EvalVisitor. Would you be okay with this being opened as an enhancement request to be merged a little later? Considering the current use of the interpreter this won't have an impact on any actual queries today. Jinfeng Ni wrote: Yes, I'm fine that you enhance this later in a new JIRA. Made a JIRA to track this, will post my intermediate work there shortly. https://issues.apache.org/jira/browse/DRILL-2395 - Jason --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/31567/#review75278 --- On March 6, 2015, 1:18 a.m., Jason Altekruse wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/31567/ --- (Updated March 6, 2015, 1:18 a.m.) Review request for drill, Aman Sinha, Jacques Nadeau, Jinfeng Ni, and Mehant Baid. Bugs: DRILL-2060 https://issues.apache.org/jira/browse/DRILL-2060 Repository: drill-git Description --- The interpreter was previously not used in normal execution, it was added with unit tests but never hooked up to an execution component. When trying to use it in the new constant folding issues I ran into build issues that are described in detail on the 2060 JIRA. I have created this to isolate the changes from the others in 2060 for review, but they are intended to be committed together. Diffs - exec/interpreter/pom.xml 20539a8 exec/interpreter/src/test/java/org/apache/drill/exec/expr/ExpressionInterpreterTest.java a94ef94 exec/java-exec/src/main/java/org/apache/drill/exec/expr/DrillFuncHolderExpr.java bc631b8 exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/DrillBooleanOPHolder.java 743598a exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/DrillDecimalAddFuncHolder.java 3871cd7 exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/DrillDecimalCastFuncHolder.java db49173 exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/DrillDecimalDivScaleFuncHolder.java 683a04f exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/DrillDecimalMaxScaleFuncHolder.java aa8e2b5 exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/DrillDecimalModScaleFuncHolder.java b5e754e exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/DrillDecimalSetScaleFuncHolder.java 47b8507 exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/DrillDecimalSumScaleFuncHolder.java cb8bfed exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/DrillDecimalZeroScaleFuncHolder.java 674fc87 exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/DrillSimpleFuncHolder.java ec284a7 exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/FunctionConverter.java 0127e6e exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/interpreter/DrillFuncInterpreter.java 3a83542 exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/interpreter/DrillSimpleFuncInterpreter.java e3696f0 exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/interpreter/InterpreterBuilder.java 3dac818 exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/interpreter/InterpreterEvaluator.java 0fe36cb exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/interpreter/InterpreterGenerator.java 6cede33 exec/java-exec/src/main/java/org/apache/drill/exec/vector/ValueHolderHelper.java 2f5bf6a exec/java-exec/src/test/java/org/apache/drill/exec/fn/interp/ExpressionInterpreterTest.java PRE-CREATION exec/java-exec/src/test/resources/functions/interp/test_input.csv PRE-CREATION exec/pom.xml e27e50b Diff: https://reviews.apache.org/r/31567/diff/ Testing --- Ran the previous interpreter test, with this and part 2 I am still finishing up some last debugging on a few type-specific test cases. Thanks, Jason Altekruse
[jira] [Created] (DRILL-2401) Value 0 for monthOfYear must be in the range [1,12]
Sudhakar Thota created DRILL-2401:
-
Summary: Value 0 for monthOfYear must be in the range [1,12]
Key: DRILL-2401
URL: https://issues.apache.org/jira/browse/DRILL-2401
Project: Apache Drill
Issue Type: Bug
Components: Functions - Drill
Affects Versions: 0.7.0
Environment: Sandbox, SE cluster
Reporter: Sudhakar Thota
Assignee: Daniel Barclay (Drill)
Priority: Minor
0: jdbc:drill:zk=10.10.15.10:5181,10.10.15.11 select cast(row_key as
varchar(20)) as order_id, cast(t.CF1.`clerk` as varchar(20)) as order_clerk,
cast(t.CF1.orderstatus as varchar(20)) as ord_stat,cast(t.CF1.orderdate as
date) ordrdt from m7thotas.orders t where cast(t.CF1.orderdate as date)
BETWEEN cast('1996-01-10' as date) AND cast('1996-01-11' as date) and
cast(t.CF1.`clerk` as varchar(20))cast('Clerk#00334' as varchar(20));
.
.
.
| 3592807| Clerk#00358 | O | 1996-01-11 |
| 6969024| Clerk#00185 | O | 1996-01-10 |
| 6973990| Clerk#00491 | O | 1996-01-10 |
| 6975170| Clerk#00493 | O | 1996-01-10 |
| 6975462| Clerk#00352 | O | 1996-01-11 |
| 6978503| Clerk#00482 | O | 1996-01-11 |
| 6979654| Clerk#00389 | O | 1996-01-10 |
| 5344355| Clerk#00653 | O | 1996-01-11 |
| 5351969| Clerk#00237 | O | 1996-01-10 |
| 5352003| Clerk#00756 | O | 1996-01-10 |
| 3608519| Clerk#00384 | O | 1996-01-10 |
Query failed: Query failed: Failure while running fragment., Value 0 for
monthOfYear must be in the range [1,12] [ 2b41018b-b35e-42a9-bd4e-0307e582b718
on se-node11.se.lab:31010 ]
[ 2b41018b-b35e-42a9-bd4e-0307e582b718 on se-node11.se.lab:31010 ]
java.lang.RuntimeException: java.sql.SQLException: Failure while executing
query.
at sqlline.SqlLine$IncrementalRows.hasNext(SqlLine.java:2514)
at sqlline.SqlLine$TableOutputFormat.print(SqlLine.java:2148)
at sqlline.SqlLine.print(SqlLine.java:1809)
at sqlline.SqlLine$Commands.execute(SqlLine.java:3766)
at sqlline.SqlLine$Commands.sql(SqlLine.java:3663)
at sqlline.SqlLine.dispatch(SqlLine.java:889)
at sqlline.SqlLine.begin(SqlLine.java:763)
at sqlline.SqlLine.start(SqlLine.java:498)
at sqlline.SqlLine.main(SqlLine.java:460)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
Re: Review Request 30754: DRILL-2143 - part 2 - remove record batch interface from UDF interface
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30754/#review75551 --- changes look fine to me. Jinfeng could you review the changes to the class InterpreterEvaluator - Mehant Baid On March 4, 2015, 11:52 p.m., Jason Altekruse wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30754/ --- (Updated March 4, 2015, 11:52 p.m.) Review request for drill, Aman Sinha, Jacques Nadeau, Jinfeng Ni, Mehant Baid, and Parth Chandra. Bugs: DRILL-2143 https://issues.apache.org/jira/browse/DRILL-2143 Repository: drill-git Description --- This patch fixes the breakage of removing the record batch from the setup method in the DrillFunc interface. It adds an injectable type to bring back the date functions and make the interpreted expression evaluation work with the new interface. Diffs - exec/interpreter/src/test/java/org/apache/drill/exec/expr/ExpressionInterpreterTest.java a94ef94 exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/DrillFuncHolder.java 279c428 exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/FunctionConverter.java 0127e6e exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/impl/DateTypeFunctions.java cc4be89 exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/impl/StringFunctions.java a3bc1de exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/interpreter/DrillSimpleFuncInterpreter.java e3696f0 exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/interpreter/InterpreterEvaluator.java 0fe36cb exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/interpreter/InterpreterGenerator.java 6cede33 exec/java-exec/src/main/java/org/apache/drill/exec/expr/holders/ValueHolder.java 5c2adc6 exec/java-exec/src/main/java/org/apache/drill/exec/expr/package-info.java PRE-CREATION exec/java-exec/src/main/java/org/apache/drill/exec/ops/FragmentContext.java e413921 exec/java-exec/src/main/java/org/apache/drill/exec/ops/QueryDateTimeInfo.java PRE-CREATION exec/java-exec/src/main/java/org/apache/drill/exec/ops/UdfUtilities.java PRE-CREATION Diff: https://reviews.apache.org/r/30754/diff/ Testing --- Almost all cluster tests are passing, recieved some failures that seem unrelated and unlikely cased by the changes, but are not reported as expected failures currently. Still need to run full unit tests again with these most recent changes. Thanks, Jason Altekruse
[jira] [Resolved] (DRILL-2381) write lexical structures section, JSON/Parquet reference fixes, updates to data types
[ https://issues.apache.org/jira/browse/DRILL-2381?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kristine Hahn resolved DRILL-2381. -- Resolution: Fixed Review Request 31780 committed. write lexical structures section, JSON/Parquet reference fixes, updates to data types - Key: DRILL-2381 URL: https://issues.apache.org/jira/browse/DRILL-2381 Project: Apache Drill Issue Type: Task Components: Documentation Affects Versions: 1.0.0 Reporter: Kristine Hahn Assignee: Kristine Hahn Fix For: 1.0.0 1. Write new lexical structure section. 2. Resolve DRILL 1920 (remove mention of bytea in 003-functions.md), other occurrences already removed. 2. Use table/column alias terms consistently, alias preferred over qualifier per Andries. 3. Make miscellaneous JSON ref clarifications. 4. Shorten/simplify Parquet ref example by using the default dfs.tmp workspace and removing unnecessary back ticks. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: JDBC data source
Agreed. As soon as we get back to calcite master I hope someone can pick this up. On Mar 6, 2015 10:28 AM, Julian Hyde julianh...@gmail.com wrote: Regarding what you call “JDBC storage plugin”. I’d call it a JDBC adapter. Calcite has a JDBC adapter that tries to push down as much of the algebra tree to a foreign database (this depends upon what the database is capable of), then generates database-specific SQL for the tree that it had managed to push. Drill’s requirements are exactly the same so it seems crazy to not start with Calcite’s adapter. Julian On Mar 6, 2015, at 10:05 AM, Norris Lee norr...@simba.com wrote: Close but not quite. Alex and I are actually working on an ODBC driver. An open source JDBC driver already exists and is in active development. As for a JDBC storage plugin to source data from other JDBC drivers (eg. Of RDBMSes), it's been requested several times recently but I don't think anyone is working on it. If I recall correctly, some people expressed interest in working on it. Norris -Original Message- From: Yash Sharma [mailto:yash...@gmail.com] Sent: Friday, March 06, 2015 6:51 AM To: u...@drill.apache.org Subject: Re: JDBC data source Drill currently doesn't have a jdbc storage plugin. However it has come up in discussions couple of times. As discussed in last hangout Norris and Alexander are working on it and we can expect a patch soon. -Via mobile. Excuse brevity. On 6 Mar 2015 8:06 pm, Chevalier Julien j...@numerigraphe.com wrote: Yash, what you pointed out is a way to produce Drill output through a JDBC. I want Drill to consume data from JDBC. As Carol clearly explains what I need is a JDBC Storage Plugin. Precisely are they any JDBC Storage plugin available ? Best regards. Juju Le 6 mars 2015 à 15:18, Yash Sharma yash...@gmail.com a écrit : Hi Juju, check if this can help: https://cwiki.apache.org/confluence/display/DRILL/Using+JDBC+to+Access +Apache+Drill+from+SQuirreL Regards On Fri, Mar 6, 2015 at 7:13 PM, Chevalier Julien j...@numerigraphe.com wrote: Dear List I Wonder if Drill can read data from a RDBMS through a good old JDBC driver Examples are welcome. Best regards Juju
