>From <[email protected]>: [email protected] has uploaded this change for review. ( https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/20744?usp=email )
Change subject: [WIP]Workload manager with configurable priority-based scheduling - user model changes: yes - storage format changes: no - interface changes: yes ...................................................................... [WIP]Workload manager with configurable priority-based scheduling - user model changes: yes - storage format changes: no - interface changes: yes - Grammar extensions (SQLPP.jj) for scheduler configuration statements - Statement classes for create, update, delete, enable/disable operations - Metadata entities and tuple translators for persistence - WorkloadManager and JobTypeManager for workload classification - Queue implementations: CompositeQueue, PriorityBasedQueue, DefaultJobQueue, MPLQueue, DedicatedJobQueue - CapacityControllerGuard for resource management - Integration with QueryTranslator and CCApplication - Unit tests for all core scheduler components Change-Id: I52748f8e62c29e3595898f765d00fd33320316c3 --- M asterixdb/NOTICE M asterixdb/asterix-algebra/src/main/java/org/apache/asterix/compiler/provider/SqlppCompilationProvider.java M asterixdb/asterix-algebra/src/main/java/org/apache/asterix/translator/IStatementExecutor.java M asterixdb/asterix-algebra/src/main/java/org/apache/asterix/translator/ResultMetadata.java M asterixdb/asterix-app/src/main/java/org/apache/asterix/api/http/server/NCQueryServiceServlet.java M asterixdb/asterix-app/src/main/java/org/apache/asterix/api/http/server/QueryResultApiServlet.java M asterixdb/asterix-app/src/main/java/org/apache/asterix/app/result/fields/MetricsPrinter.java M asterixdb/asterix-app/src/main/java/org/apache/asterix/app/translator/QueryTranslator.java M asterixdb/asterix-app/src/main/java/org/apache/asterix/hyracks/bootstrap/CCApplication.java M asterixdb/asterix-app/src/test/java/org/apache/asterix/api/common/AsterixHyracksIntegrationUtil.java A asterixdb/asterix-app/src/test/resources/cc-scheduler.conf A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/scheduler/dynamic-workload-config-syntax/dynamic-workload-config-syntax-1.1.ddl.sqlpp A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/scheduler/dynamic-workload-config-syntax/dynamic-workload-config-syntax-1.10.delete.sqlpp A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/scheduler/dynamic-workload-config-syntax/dynamic-workload-config-syntax-1.11.query.sqlpp A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/scheduler/dynamic-workload-config-syntax/dynamic-workload-config-syntax-1.12.ddl.sqlpp A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/scheduler/dynamic-workload-config-syntax/dynamic-workload-config-syntax-1.13.query.sqlpp A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/scheduler/dynamic-workload-config-syntax/dynamic-workload-config-syntax-1.14.enable.sqlpp A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/scheduler/dynamic-workload-config-syntax/dynamic-workload-config-syntax-1.15.query.sqlpp A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/scheduler/dynamic-workload-config-syntax/dynamic-workload-config-syntax-1.16.enable.sqlpp A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/scheduler/dynamic-workload-config-syntax/dynamic-workload-config-syntax-1.17.ddl.sqlpp A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/scheduler/dynamic-workload-config-syntax/dynamic-workload-config-syntax-1.18.query.sqlpp A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/scheduler/dynamic-workload-config-syntax/dynamic-workload-config-syntax-1.19.enable.sqlpp A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/scheduler/dynamic-workload-config-syntax/dynamic-workload-config-syntax-1.2.ddl.sqlpp A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/scheduler/dynamic-workload-config-syntax/dynamic-workload-config-syntax-1.20.query.sqlpp A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/scheduler/dynamic-workload-config-syntax/dynamic-workload-config-syntax-1.21.update.sqlpp A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/scheduler/dynamic-workload-config-syntax/dynamic-workload-config-syntax-1.22.query.sqlpp A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/scheduler/dynamic-workload-config-syntax/dynamic-workload-config-syntax-1.3.ddl.sqlpp A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/scheduler/dynamic-workload-config-syntax/dynamic-workload-config-syntax-1.4.query.sqlpp A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/scheduler/dynamic-workload-config-syntax/dynamic-workload-config-syntax-1.5.upsert.sqlpp A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/scheduler/dynamic-workload-config-syntax/dynamic-workload-config-syntax-1.6.upsert.sqlpp A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/scheduler/dynamic-workload-config-syntax/dynamic-workload-config-syntax-1.7.upsert.sqlpp A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/scheduler/dynamic-workload-config-syntax/dynamic-workload-config-syntax-1.8.query.sqlpp A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/scheduler/dynamic-workload-config-syntax/dynamic-workload-config-syntax-1.9.delete.sqlpp M asterixdb/asterix-common/src/main/java/org/apache/asterix/common/api/IMetadataLockManager.java M asterixdb/asterix-common/src/main/java/org/apache/asterix/common/config/CompilerProperties.java M asterixdb/asterix-common/src/main/java/org/apache/asterix/common/config/OptimizationConfUtil.java M asterixdb/asterix-common/src/main/java/org/apache/asterix/common/exceptions/ErrorCode.java M asterixdb/asterix-common/src/main/java/org/apache/asterix/common/metadata/IMetadataLockUtil.java M asterixdb/asterix-common/src/main/java/org/apache/asterix/common/metadata/MetadataConstants.java M asterixdb/asterix-common/src/main/resources/asx_errormsg/en.properties M asterixdb/asterix-lang-common/src/main/java/org/apache/asterix/lang/common/base/Statement.java A asterixdb/asterix-lang-common/src/main/java/org/apache/asterix/lang/common/statement/CreateSchedulerConfigStatement.java A asterixdb/asterix-lang-common/src/main/java/org/apache/asterix/lang/common/statement/DeleteQGroupStatement.java A asterixdb/asterix-lang-common/src/main/java/org/apache/asterix/lang/common/statement/EnableSchedulerStatement.java A asterixdb/asterix-lang-common/src/main/java/org/apache/asterix/lang/common/statement/SchedulerConfigDropStatement.java A asterixdb/asterix-lang-common/src/main/java/org/apache/asterix/lang/common/statement/UpdateSchedulerStatement.java A asterixdb/asterix-lang-common/src/main/java/org/apache/asterix/lang/common/statement/UpsertQGroupStatement.java A asterixdb/asterix-lang-common/src/main/java/org/apache/asterix/lang/common/util/SchedulerConfigUtil.java M asterixdb/asterix-lang-common/src/main/java/org/apache/asterix/lang/common/visitor/FormatPrintVisitor.java M asterixdb/asterix-lang-common/src/main/java/org/apache/asterix/lang/common/visitor/base/AbstractQueryExpressionVisitor.java M asterixdb/asterix-lang-common/src/main/java/org/apache/asterix/lang/common/visitor/base/ILangVisitor.java M asterixdb/asterix-lang-sqlpp/src/main/javacc/SQLPP.jj M asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/MetadataCache.java M asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/MetadataManager.java M asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/MetadataNode.java M asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/MetadataTransactionContext.java M asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/api/IMetadataManager.java M asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/api/IMetadataNode.java M asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/bootstrap/MetadataBootstrap.java M asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/bootstrap/MetadataIndexesProvider.java M asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/bootstrap/MetadataPrimaryIndexes.java M asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/bootstrap/MetadataRecordTypes.java A asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/bootstrap/SchedulerConfigRecordEntity.java A asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/bootstrap/SchedulerConfigStateEntity.java M asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/declared/MetadataManagerUtil.java M asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/declared/MetadataProvider.java A asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/entities/SchedulerConfigMetadataEntity.java M asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/entitytupletranslators/MetadataTupleTranslatorProvider.java A asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/entitytupletranslators/SchedulerConfigMetadataEntityTupleTranslator.java M asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/lock/MetadataLockKey.java M asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/lock/MetadataLockManager.java M asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/utils/MetadataLockUtil.java A asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/utils/SchedulerUtil.java M asterixdb/asterix-runtime/src/main/java/org/apache/asterix/runtime/job/resource/JobCapacityController.java A asterixdb/asterix-runtime/src/main/java/org/apache/asterix/runtime/scheduler/ISchedulerConfigDescriptor.java A asterixdb/asterix-runtime/src/main/java/org/apache/asterix/runtime/scheduler/SchedulerConfigRecordDescriptor.java A asterixdb/asterix-runtime/src/main/java/org/apache/asterix/runtime/scheduler/SchedulerConfigStateDescriptor.java M hyracks-fullstack/NOTICE M hyracks-fullstack/algebricks/algebricks-core/src/main/java/org/apache/hyracks/algebricks/core/config/AlgebricksConfig.java M hyracks-fullstack/algebricks/algebricks-core/src/main/java/org/apache/hyracks/algebricks/core/jobgen/impl/PlanCompiler.java M hyracks-fullstack/algebricks/algebricks-core/src/main/java/org/apache/hyracks/algebricks/core/rewriter/base/PhysicalOptimizationConfig.java M hyracks-fullstack/hyracks/hyracks-api/src/main/java/org/apache/hyracks/api/job/JobSpecification.java M hyracks-fullstack/hyracks/hyracks-api/src/main/java/org/apache/hyracks/api/job/resource/IJobCapacityController.java M hyracks-fullstack/hyracks/hyracks-control/hyracks-control-cc/src/main/java/org/apache/hyracks/control/cc/ClusterControllerService.java M hyracks-fullstack/hyracks/hyracks-control/hyracks-control-cc/src/main/java/org/apache/hyracks/control/cc/job/JobManager.java M hyracks-fullstack/hyracks/hyracks-control/hyracks-control-cc/src/main/java/org/apache/hyracks/control/cc/job/JobRun.java A hyracks-fullstack/hyracks/hyracks-control/hyracks-control-cc/src/main/java/org/apache/hyracks/control/cc/job/WorkloadManager.java A hyracks-fullstack/hyracks/hyracks-control/hyracks-control-cc/src/main/java/org/apache/hyracks/control/cc/scheduler/CapacityControllerGuard.java A hyracks-fullstack/hyracks/hyracks-control/hyracks-control-cc/src/main/java/org/apache/hyracks/control/cc/scheduler/CompositeQueue.java A hyracks-fullstack/hyracks/hyracks-control/hyracks-control-cc/src/main/java/org/apache/hyracks/control/cc/scheduler/DedicatedJobQueue.java A hyracks-fullstack/hyracks/hyracks-control/hyracks-control-cc/src/main/java/org/apache/hyracks/control/cc/scheduler/DefaultJobQueue.java A hyracks-fullstack/hyracks/hyracks-control/hyracks-control-cc/src/main/java/org/apache/hyracks/control/cc/scheduler/DeleteGroupInfo.java A hyracks-fullstack/hyracks/hyracks-control/hyracks-control-cc/src/main/java/org/apache/hyracks/control/cc/scheduler/EnableConfigInfo.java M hyracks-fullstack/hyracks/hyracks-control/hyracks-control-cc/src/main/java/org/apache/hyracks/control/cc/scheduler/FIFOJobQueue.java M hyracks-fullstack/hyracks/hyracks-control/hyracks-control-cc/src/main/java/org/apache/hyracks/control/cc/scheduler/IJobQueue.java A hyracks-fullstack/hyracks/hyracks-control/hyracks-control-cc/src/main/java/org/apache/hyracks/control/cc/scheduler/IJobTypeManager.java A hyracks-fullstack/hyracks/hyracks-control/hyracks-control-cc/src/main/java/org/apache/hyracks/control/cc/scheduler/IWorkloadConfigInfo.java A hyracks-fullstack/hyracks/hyracks-control/hyracks-control-cc/src/main/java/org/apache/hyracks/control/cc/scheduler/JobTypeManager.java A hyracks-fullstack/hyracks/hyracks-control/hyracks-control-cc/src/main/java/org/apache/hyracks/control/cc/scheduler/MPLQueue.java A hyracks-fullstack/hyracks/hyracks-control/hyracks-control-cc/src/main/java/org/apache/hyracks/control/cc/scheduler/PriorityBasedQueue.java A hyracks-fullstack/hyracks/hyracks-control/hyracks-control-cc/src/main/java/org/apache/hyracks/control/cc/scheduler/UpsertGroupInfo.java A hyracks-fullstack/hyracks/hyracks-control/hyracks-control-cc/src/main/java/org/apache/hyracks/control/cc/work/NotifyWorkloadConfigWork.java A hyracks-fullstack/hyracks/hyracks-control/hyracks-control-cc/src/test/java/org/apache/hyracks/control/cc/job/WorkloadManagerTest.java A hyracks-fullstack/hyracks/hyracks-control/hyracks-control-cc/src/test/java/org/apache/hyracks/control/cc/scheduler/DefaultJobQueueTest.java A hyracks-fullstack/hyracks/hyracks-control/hyracks-control-cc/src/test/java/org/apache/hyracks/control/cc/scheduler/JobRunBatchFactory.java A hyracks-fullstack/hyracks/hyracks-control/hyracks-control-cc/src/test/java/org/apache/hyracks/control/cc/scheduler/JobRunBuilder.java A hyracks-fullstack/hyracks/hyracks-control/hyracks-control-cc/src/test/java/org/apache/hyracks/control/cc/scheduler/JobRunFactory.java A hyracks-fullstack/hyracks/hyracks-control/hyracks-control-cc/src/test/java/org/apache/hyracks/control/cc/scheduler/PriorityBasedQueueTest.java A hyracks-fullstack/hyracks/hyracks-control/hyracks-control-cc/src/test/java/org/apache/hyracks/control/cc/scheduler/TestUtils.java M hyracks-fullstack/hyracks/hyracks-examples/hyracks-integration-tests/src/test/java/org/apache/hyracks/tests/integration/AbstractMultiNCIntegrationTest.java 110 files changed, 6,028 insertions(+), 104 deletions(-) git pull ssh://asterix-gerrit.ics.uci.edu:29418/asterixdb refs/changes/44/20744/1 -- To view, visit https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/20744?usp=email To unsubscribe, or for help writing mail filters, visit https://asterix-gerrit.ics.uci.edu/settings?usp=email Gerrit-MessageType: newchange Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Change-Id: I52748f8e62c29e3595898f765d00fd33320316c3 Gerrit-Change-Number: 20744 Gerrit-PatchSet: 1 Gerrit-Owner: [email protected]
