----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30466/ -----------------------------------------------------------
(Updated Feb. 9, 2015, 3:05 p.m.) Review request for drill, Chris Westin, Jacques Nadeau, and Steven Phillips. Repository: drill-git Description ------- In this patch LocalExchange contains only multiplexing exchange. Currently working on demultiplexing exchange (there are few failures in executions and currently debugging those). Sending this patch to get initial feedback. Brief overview of changes: 1. Traverse the PRel tree after all optimizations. Whenever a HashToRandomExchangePrel is encountered insert a MuxExchange before HashToRandomExchangePrel and DemuxExchange after HashToRandomExchangePrel. 2. Parallelization changes: i) Traverse the physical operator tree and divide it into Fragments. ii) Based on the affinity of the sending Exchange, set parallelization dependencies between fragments. iii) Start parallelizing from the leaf fragments (fragment that have no other fragments depending on them for parallelization info). Stats collection include collecting parallelization info (minWidth, maxWidth, affinityMap) and cost. 3. Change the Receiver to accept set of (minorFragmentId, DrillbitEndpoint) as sender list. This also involved few changes in DataCollector. 4. Change SingleSender to accept custom minorFragmentId instead of default minorFragmentId of 0 Diffs (updated) ----- contrib/storage-hbase/src/test/java/org/apache/drill/hbase/BaseHBaseTest.java 1152b7b contrib/storage-hive/core/src/test/java/org/apache/drill/exec/hive/HiveTestBase.java 7e3b6c8 exec/java-exec/src/main/java/org/apache/drill/exec/physical/EndpointAffinity.java df31f74 exec/java-exec/src/main/java/org/apache/drill/exec/physical/MinorFragmentEndpoint.java PRE-CREATION exec/java-exec/src/main/java/org/apache/drill/exec/physical/base/AbstractExchange.java 73280ea exec/java-exec/src/main/java/org/apache/drill/exec/physical/base/AbstractGroupScan.java 5d0d9bf exec/java-exec/src/main/java/org/apache/drill/exec/physical/base/AbstractReceiver.java f621a26 exec/java-exec/src/main/java/org/apache/drill/exec/physical/base/AbstractSender.java 53a0721 exec/java-exec/src/main/java/org/apache/drill/exec/physical/base/Exchange.java 7be7f20 exec/java-exec/src/main/java/org/apache/drill/exec/physical/base/PhysicalOperatorUtil.java dfcb113 exec/java-exec/src/main/java/org/apache/drill/exec/physical/base/Receiver.java 0c67770 exec/java-exec/src/main/java/org/apache/drill/exec/physical/base/Sender.java bbd1b2c exec/java-exec/src/main/java/org/apache/drill/exec/physical/config/BroadcastExchange.java 73a1d20 exec/java-exec/src/main/java/org/apache/drill/exec/physical/config/BroadcastSender.java 1827367 exec/java-exec/src/main/java/org/apache/drill/exec/physical/config/DeMuxExchange.java PRE-CREATION exec/java-exec/src/main/java/org/apache/drill/exec/physical/config/HashPartitionSender.java bdb1362 exec/java-exec/src/main/java/org/apache/drill/exec/physical/config/HashToMergeExchange.java f62d922 exec/java-exec/src/main/java/org/apache/drill/exec/physical/config/HashToRandomExchange.java fac374b exec/java-exec/src/main/java/org/apache/drill/exec/physical/config/MergingReceiverPOP.java f5dca1a exec/java-exec/src/main/java/org/apache/drill/exec/physical/config/MuxExchange.java PRE-CREATION exec/java-exec/src/main/java/org/apache/drill/exec/physical/config/OrderedPartitionExchange.java 8e1526a exec/java-exec/src/main/java/org/apache/drill/exec/physical/config/OrderedPartitionSender.java 0a2b9be exec/java-exec/src/main/java/org/apache/drill/exec/physical/config/RangeSender.java c8c8f43 exec/java-exec/src/main/java/org/apache/drill/exec/physical/config/Screen.java 58c8e29 exec/java-exec/src/main/java/org/apache/drill/exec/physical/config/SingleMergeExchange.java 2914112 exec/java-exec/src/main/java/org/apache/drill/exec/physical/config/SingleSender.java 4a11a51 exec/java-exec/src/main/java/org/apache/drill/exec/physical/config/UnionExchange.java cfc21ac exec/java-exec/src/main/java/org/apache/drill/exec/physical/config/UnorderedReceiver.java 3a4dd0e exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/SingleSenderCreator.java 6db9f4a exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/broadcastsender/BroadcastSenderRootExec.java 22fa047 exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/mergereceiver/MergingRecordBatch.java d78ba8e exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/partitionsender/PartitionSenderRootExec.java f09acaa exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/partitionsender/PartitionerTemplate.java 4292c09 exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/unorderedreceiver/UnorderedReceiverBatch.java 52b892e exec/java-exec/src/main/java/org/apache/drill/exec/planner/fragment/Fragment.java ac63bde exec/java-exec/src/main/java/org/apache/drill/exec/planner/fragment/MakeFragmentsVisitor.java 8756e5b exec/java-exec/src/main/java/org/apache/drill/exec/planner/fragment/Materializer.java 961b603 exec/java-exec/src/main/java/org/apache/drill/exec/planner/fragment/ParallelizationInfo.java PRE-CREATION exec/java-exec/src/main/java/org/apache/drill/exec/planner/fragment/PlanningSet.java 8cc6c85 exec/java-exec/src/main/java/org/apache/drill/exec/planner/fragment/SimpleParallelizer.java 434cdd4 exec/java-exec/src/main/java/org/apache/drill/exec/planner/fragment/Stats.java eda364b exec/java-exec/src/main/java/org/apache/drill/exec/planner/fragment/StatsCollector.java 41ff678 exec/java-exec/src/main/java/org/apache/drill/exec/planner/fragment/Wrapper.java 86b395e exec/java-exec/src/main/java/org/apache/drill/exec/planner/physical/DeMuxExchangePrel.java PRE-CREATION exec/java-exec/src/main/java/org/apache/drill/exec/planner/physical/MuxExchangePrel.java PRE-CREATION exec/java-exec/src/main/java/org/apache/drill/exec/planner/physical/PlannerSettings.java faa8546 exec/java-exec/src/main/java/org/apache/drill/exec/planner/physical/visitor/InsertLocalExchangeVisitor.java PRE-CREATION exec/java-exec/src/main/java/org/apache/drill/exec/planner/sql/handlers/DefaultSqlHandler.java 6b3d301 exec/java-exec/src/main/java/org/apache/drill/exec/server/options/SystemOptionManager.java f20627d exec/java-exec/src/main/java/org/apache/drill/exec/store/dfs/easy/EasyGroupScan.java 9902443 exec/java-exec/src/main/java/org/apache/drill/exec/store/direct/DirectGroupScan.java aa1609d exec/java-exec/src/main/java/org/apache/drill/exec/store/ischema/InfoSchemaGroupScan.java 8335ed9 exec/java-exec/src/main/java/org/apache/drill/exec/store/mock/MockGroupScanPOP.java 5736df8 exec/java-exec/src/main/java/org/apache/drill/exec/store/sys/SystemTableScan.java 053f5de exec/java-exec/src/main/java/org/apache/drill/exec/work/batch/AbstractDataCollector.java c83106c exec/java-exec/src/main/java/org/apache/drill/exec/work/batch/PartitionedCollector.java 5190d84 exec/java-exec/src/main/java/org/apache/drill/exec/work/foreman/Foreman.java 378e81a exec/java-exec/src/test/java/org/apache/drill/BaseTestQuery.java 17bcb79 exec/java-exec/src/test/java/org/apache/drill/PlanTestBase.java c52545d exec/java-exec/src/test/java/org/apache/drill/exec/compile/TestClassTransformation.java bb855c9 exec/java-exec/src/test/java/org/apache/drill/exec/physical/impl/TestLocalExchange.java PRE-CREATION exec/java-exec/src/test/java/org/apache/drill/exec/pop/PopUnitTestBase.java 9a32ff9 exec/java-exec/src/test/java/org/apache/drill/exec/pop/TestFragmentChecker.java 6349b76 exec/java-exec/src/test/java/org/apache/drill/exec/store/parquet/ParquetRecordReaderTest.java 219e66f Diff: https://reviews.apache.org/r/30466/diff/ Testing ------- Ran Functional and TPCH SF100 parquet verification tests. Thanks, Venki Korukanti
