Yingyi Bu has submitted this change and it was merged. Change subject: Supports flexible degree of parallelism. ......................................................................
Supports flexible degree of parallelism. Detailed list of changes include: - Add the IClusterInfoCollector interface and feed that to the compiler; - Add the number of cores into the heartbeat; - Add partition calculation in APIFramework; - Fix partitioning property requirements in binary operators such as Join and UnionAll; - Fix PushSubplanIntoGroupByRule for type propagation; - Fix InlineLeftNtsInSubplanJoinFlatteningVisitor for the returned top join operator; - Add unit test for partition calculation in APIFrameworkTest - Add integration test SqlppExecutionFullParallelismIT, SqlppExecutionLessParallelismIT, AqlExecutionFullParallelismIT, and AqlExecutionLessParallelismIT for different parallelism settings; - Refactor AQLExecutionTest and SqlppExecutionTest; - Separate one cluster state test out from AqlExecutionTest. Change-Id: If70faf52bed995c7098cc343f2fabd1b6c8d96e7 Reviewed-on: https://asterix-gerrit.ics.uci.edu/1398 Tested-by: Jenkins <[email protected]> BAD: Jenkins <[email protected]> Integration-Tests: Jenkins <[email protected]> Reviewed-by: Till Westmann <[email protected]> --- M asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/subplan/InlineLeftNtsInSubplanJoinFlatteningVisitor.java M asterixdb/asterix-algebra/src/main/java/org/apache/asterix/translator/IStatementExecutor.java M asterixdb/asterix-app/src/main/java/org/apache/asterix/api/common/APIFramework.java M asterixdb/asterix-app/src/main/java/org/apache/asterix/app/translator/QueryTranslator.java M asterixdb/asterix-app/src/main/resources/asterix-build-configuration.xml A asterixdb/asterix-app/src/main/resources/asterix-build-configuration2.xml A asterixdb/asterix-app/src/main/resources/asterix-build-configuration3.xml A asterixdb/asterix-app/src/test/java/org/apache/asterix/api/common/APIFrameworkTest.java M asterixdb/asterix-app/src/test/java/org/apache/asterix/api/http/servlet/ConnectorAPIServletTest.java M asterixdb/asterix-app/src/test/java/org/apache/asterix/api/http/servlet/VersionAPIServletTest.java A asterixdb/asterix-app/src/test/java/org/apache/asterix/test/runtime/AqlExecutionFullParallelismIT.java A asterixdb/asterix-app/src/test/java/org/apache/asterix/test/runtime/AqlExecutionLessParallelismIT.java A asterixdb/asterix-app/src/test/java/org/apache/asterix/test/runtime/AqlExecutionTest.java A asterixdb/asterix-app/src/test/java/org/apache/asterix/test/runtime/ClusterStateExecutionFullParallelismTest.java A asterixdb/asterix-app/src/test/java/org/apache/asterix/test/runtime/ClusterStateExecutionLessParallelismTest.java A asterixdb/asterix-app/src/test/java/org/apache/asterix/test/runtime/ClusterStateExecutionTest.java D asterixdb/asterix-app/src/test/java/org/apache/asterix/test/runtime/ExecutionTest.java M asterixdb/asterix-app/src/test/java/org/apache/asterix/test/runtime/ExecutionTestUtil.java A asterixdb/asterix-app/src/test/java/org/apache/asterix/test/runtime/LangExecutionUtil.java M asterixdb/asterix-app/src/test/java/org/apache/asterix/test/runtime/RepeatedTest.java A asterixdb/asterix-app/src/test/java/org/apache/asterix/test/runtime/SqlppExecutionFullParallelismIT.java A asterixdb/asterix-app/src/test/java/org/apache/asterix/test/runtime/SqlppExecutionLessParallelismIT.java M asterixdb/asterix-app/src/test/java/org/apache/asterix/test/runtime/SqlppExecutionTest.java M asterixdb/asterix-app/src/test/resources/optimizerts/results/nonpure/query-ASTERIXDB-1608.plan M asterixdb/asterix-app/src/test/resources/optimizerts/results/query-issue827-2.plan A asterixdb/asterix-app/src/test/resources/runtimets/cluster_state.xml A asterixdb/asterix-app/src/test/resources/runtimets/cluster_state_full.xml A asterixdb/asterix-app/src/test/resources/runtimets/cluster_state_less.xml A asterixdb/asterix-app/src/test/resources/runtimets/only_cluster_state.xml M asterixdb/asterix-app/src/test/resources/runtimets/queries/api/APIQueries.xml M asterixdb/asterix-app/src/test/resources/runtimets/queries/flwor/at00/at00.3.query.aql M asterixdb/asterix-app/src/test/resources/runtimets/queries/flwor/at00/at00.4.asyncdefer.aql M asterixdb/asterix-app/src/test/resources/runtimets/queries/flwor/at00/at00.5.async.aql M asterixdb/asterix-app/src/test/resources/runtimets/queries/flwor/at06/at06.3.query.aql M asterixdb/asterix-app/src/test/resources/runtimets/queries/open-closed/query-issue456/query-issue456.3.query.aql M asterixdb/asterix-app/src/test/resources/runtimets/queries/tpch-sql-like/query-issue601/query-issue601.3.query.aql M asterixdb/asterix-app/src/test/resources/runtimets/queries/tpch-sql-like/query-issue785-2/query-issue785-2.3.query.aql M asterixdb/asterix-app/src/test/resources/runtimets/queries/tpch-sql-like/query-issue785/query-issue785.3.query.aql M asterixdb/asterix-app/src/test/resources/runtimets/queries/tpch/query-issue601/query-issue601.3.query.aql M asterixdb/asterix-app/src/test/resources/runtimets/queries/tpch/query-issue785-2/query-issue785-2.3.query.aql M asterixdb/asterix-app/src/test/resources/runtimets/queries/tpch/query-issue785/query-issue785.3.query.aql M asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/flwor/at00/at00.3.query.sqlpp M asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/flwor/at00/at00.4.asyncdefer.sqlpp M asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/flwor/at00/at00.5.async.sqlpp M asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/flwor/at06/at06.3.query.sqlpp M asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/open-closed/query-issue456/query-issue456.3.query.sqlpp M asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/tinysocial/tinysocial-suite-open/group-by.26.query.sqlpp M asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/tpch-sql-like/query-issue601/query-issue601.3.query.sqlpp M asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/tpch-sql-like/query-issue785-2/query-issue785-2.3.query.sqlpp M asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/tpch-sql-like/query-issue785/query-issue785.3.query.sqlpp M asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/tpch-sql-sugar/query-issue601/query-issue601.3.query.sqlpp M asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/tpch-sql-sugar/query-issue785-2/query-issue785-2.3.query.sqlpp M asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/tpch-sql-sugar/query-issue785/query-issue785.3.query.sqlpp M asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/tpch-sql/query-issue601/query-issue601.3.query.sqlpp M asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/tpch-sql/query-issue785-2/query-issue785-2.3.query.sqlpp M asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/tpch-sql/query-issue785/query-issue785.3.query.sqlpp M asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/tpch-with-index/query-issue601/query-issue601.3.query.sqlpp M asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/tpch-with-index/query-issue785-2/query-issue785-2.3.query.sqlpp M asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/tpch-with-index/query-issue785/query-issue785.3.query.sqlpp M asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/tpch/query-issue601/query-issue601.3.query.sqlpp M asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/tpch/query-issue785-2/query-issue785-2.3.query.sqlpp M asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/tpch/query-issue785/query-issue785.3.query.sqlpp M asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/types/any-object/any-object.2.query.sqlpp M asterixdb/asterix-app/src/test/resources/runtimets/results/api/cluster_state_1/cluster_state_1.1.adm A asterixdb/asterix-app/src/test/resources/runtimets/results/api/cluster_state_1_full/cluster_state_1_full.1.adm A asterixdb/asterix-app/src/test/resources/runtimets/results/api/cluster_state_1_less/cluster_state_1_less.1.adm M asterixdb/asterix-app/src/test/resources/runtimets/results/flwor/at00/at00.1.adm M asterixdb/asterix-app/src/test/resources/runtimets/results/flwor/at00/at00.2.adm M asterixdb/asterix-app/src/test/resources/runtimets/results/flwor/at00/at00.3.adm M asterixdb/asterix-app/src/test/resources/runtimets/results/flwor/at06/at06.1.adm M asterixdb/asterix-app/src/test/resources/runtimets/results/tinysocial/tinysocial-suite-open/group-by.26.adm M asterixdb/asterix-app/src/test/resources/runtimets/results/tpch-sql-like/query-issue601/query-issue601.1.adm M asterixdb/asterix-app/src/test/resources/runtimets/results/tpch-sql-like/query-issue785-2/query-issue785-2.1.adm M asterixdb/asterix-app/src/test/resources/runtimets/results/tpch-sql-like/query-issue785/query-issue785.1.adm M asterixdb/asterix-app/src/test/resources/runtimets/results/tpch-sql-sugar/query-issue601/query-issue601.1.adm M asterixdb/asterix-app/src/test/resources/runtimets/results/tpch-sql-sugar/query-issue785-2/query-issue785-2.1.adm M asterixdb/asterix-app/src/test/resources/runtimets/results/tpch-sql-sugar/query-issue785/query-issue785.1.adm M asterixdb/asterix-app/src/test/resources/runtimets/results/tpch-sql/query-issue601/query-issue601.1.adm M asterixdb/asterix-app/src/test/resources/runtimets/results/tpch-sql/query-issue785-2/query-issue785-2.1.adm M asterixdb/asterix-app/src/test/resources/runtimets/results/tpch-sql/query-issue785/query-issue785.1.adm M asterixdb/asterix-app/src/test/resources/runtimets/results/tpch-with-index/query-issue601/query-issue601.1.adm M asterixdb/asterix-app/src/test/resources/runtimets/results/tpch-with-index/query-issue785-2/query-issue785-2.1.adm M asterixdb/asterix-app/src/test/resources/runtimets/results/tpch-with-index/query-issue785/query-issue785.1.adm M asterixdb/asterix-app/src/test/resources/runtimets/results/tpch/query-issue601/query-issue601.1.adm M asterixdb/asterix-app/src/test/resources/runtimets/results/tpch/query-issue785-2/query-issue785-2.1.adm M asterixdb/asterix-app/src/test/resources/runtimets/results/tpch/query-issue785/query-issue785.1.adm M asterixdb/asterix-app/src/test/resources/runtimets/results_parser_sqlpp/flwor/at00/at00.3.ast M asterixdb/asterix-app/src/test/resources/runtimets/results_parser_sqlpp/flwor/at00/at00.4.ast M asterixdb/asterix-app/src/test/resources/runtimets/results_parser_sqlpp/flwor/at00/at00.5.ast M asterixdb/asterix-app/src/test/resources/runtimets/results_parser_sqlpp/flwor/at06/at06.3.ast M asterixdb/asterix-app/src/test/resources/runtimets/results_parser_sqlpp/open-closed/query-issue456/query-issue456.3.ast M asterixdb/asterix-app/src/test/resources/runtimets/results_parser_sqlpp/tinysocial/tinysocial-suite-open/tinysocial-suite.26.ast M asterixdb/asterix-app/src/test/resources/runtimets/results_parser_sqlpp/tpch-sql-like/query-issue601/query-issue601.3.ast M asterixdb/asterix-app/src/test/resources/runtimets/results_parser_sqlpp/tpch-sql-like/query-issue785-2/query-issue785-2.3.ast M asterixdb/asterix-app/src/test/resources/runtimets/results_parser_sqlpp/tpch-sql-like/query-issue785/query-issue785.3.ast M asterixdb/asterix-app/src/test/resources/runtimets/results_parser_sqlpp/tpch/query-issue601/query-issue601.3.ast M asterixdb/asterix-app/src/test/resources/runtimets/results_parser_sqlpp/tpch/query-issue785-2/query-issue785-2.3.ast M asterixdb/asterix-app/src/test/resources/runtimets/results_parser_sqlpp/tpch/query-issue785/query-issue785.3.ast M asterixdb/asterix-common/src/main/java/org/apache/asterix/common/config/CompilerProperties.java M hyracks-fullstack/algebricks/algebricks-core/src/main/java/org/apache/hyracks/algebricks/core/algebra/operators/physical/AbstractHashJoinPOperator.java M hyracks-fullstack/algebricks/algebricks-core/src/main/java/org/apache/hyracks/algebricks/core/algebra/operators/physical/NestedLoopJoinPOperator.java M hyracks-fullstack/algebricks/algebricks-core/src/main/java/org/apache/hyracks/algebricks/core/algebra/operators/physical/UnionAllPOperator.java M hyracks-fullstack/algebricks/algebricks-core/src/main/java/org/apache/hyracks/algebricks/core/algebra/properties/StructuralPropertiesVector.java M hyracks-fullstack/algebricks/algebricks-core/src/main/java/org/apache/hyracks/algebricks/core/algebra/util/OperatorPropertiesUtil.java M hyracks-fullstack/algebricks/algebricks-rewriter/src/main/java/org/apache/hyracks/algebricks/rewriter/rules/subplan/PushSubplanIntoGroupByRule.java M hyracks-fullstack/hyracks/hyracks-api/src/main/java/org/apache/hyracks/api/client/HyracksConnection.java A hyracks-fullstack/hyracks/hyracks-api/src/main/java/org/apache/hyracks/api/client/IClusterInfoCollector.java M hyracks-fullstack/hyracks/hyracks-api/src/main/java/org/apache/hyracks/api/client/IHyracksClientConnection.java M hyracks-fullstack/hyracks/hyracks-api/src/main/java/org/apache/hyracks/api/client/NodeControllerInfo.java M hyracks-fullstack/hyracks/hyracks-control/hyracks-control-cc/src/main/java/org/apache/hyracks/control/cc/NodeControllerState.java M hyracks-fullstack/hyracks/hyracks-control/hyracks-control-cc/src/main/java/org/apache/hyracks/control/cc/work/GetNodeControllersInfoWork.java M hyracks-fullstack/hyracks/hyracks-control/hyracks-control-common/src/main/java/org/apache/hyracks/control/common/heartbeat/HeartbeatData.java M hyracks-fullstack/hyracks/hyracks-control/hyracks-control-nc/src/main/java/org/apache/hyracks/control/nc/NodeControllerService.java M hyracks-fullstack/hyracks/hyracks-dataflow-std/src/main/java/org/apache/hyracks/dataflow/std/union/UnionAllOperatorDescriptor.java M hyracks-fullstack/hyracks/hyracks-hdfs/hyracks-hdfs-core/src/test/java/org/apache/hyracks/hdfs/scheduler/SchedulerTest.java M hyracks-fullstack/hyracks/hyracks-hdfs/hyracks-hdfs-core/src/test/java/org/apache/hyracks/hdfs/utils/TestUtils.java 116 files changed, 3,771 insertions(+), 2,311 deletions(-) Approvals: Till Westmann: Looks good to me, approved Jenkins: Verified; No violations found; Verified Objections: Jenkins: Violations found -- To view, visit https://asterix-gerrit.ics.uci.edu/1398 To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-MessageType: merged Gerrit-Change-Id: If70faf52bed995c7098cc343f2fabd1b6c8d96e7 Gerrit-PatchSet: 13 Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Owner: Yingyi Bu <[email protected]> Gerrit-Reviewer: Jenkins <[email protected]> Gerrit-Reviewer: Michael Blow <[email protected]> Gerrit-Reviewer: Steven Jacobs <[email protected]> Gerrit-Reviewer: Till Westmann <[email protected]> Gerrit-Reviewer: Yingyi Bu <[email protected]> Gerrit-Reviewer: abdullah alamoudi <[email protected]>
