----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/73076/#review222588 -----------------------------------------------------------
common/src/main/java/org/apache/atlas/repository/Constants.java Lines 97 (patched) <https://reviews.apache.org/r/73076/#comment311699> register this new property - 'PENDING_PROPAGATIONS' as SET type in GraphBackedSearchIndexer ('createCommonVertexIndex') intg/src/main/java/org/apache/atlas/model/tasks/TaskDef.java Lines 56 (patched) <https://reviews.apache.org/r/73076/#comment311700> consider making 'parameters' of Map type. serialize to string during writing to vertex (TaskRegistry.createVertex()) and deserialize when reading from vertex (TaskRegistry.toTaskDef()). intg/src/main/java/org/apache/atlas/type/AtlasType.java Lines 147 (patched) <https://reviews.apache.org/r/73076/#comment311702> remove unused method - 'fromLinkedHashMap', also remove usunsed method AtlasJson.fromLinkedHashMap() repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphMapper.java Lines 2780 (patched) <https://reviews.apache.org/r/73076/#comment311701> for LIST/SET type attributes, use 'AtlasGraphUtilsV2.addEncodedProperty()' to add multi-values to vertex. Please review. for (String pendingPropagationGuid : classification.getPendingPropagations()) { AtlasGraphUtilsV2.addEncodedProperty(classificationVertex, PENDING_PROPAGATIONS, classification.getPendingPropagations()); } repository/src/main/java/org/apache/atlas/repository/store/graph/v2/tasks/ClassificationPropagationTasks.java Lines 30 (patched) <https://reviews.apache.org/r/73076/#comment311698> "classificationVertexIds" => "classificationVertexId" - Sarath Subramanian On Feb. 11, 2021, 11:12 a.m., Ashutosh Mestry wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/73076/ > ----------------------------------------------------------- > > (Updated Feb. 11, 2021, 11:12 a.m.) > > > Review request for atlas, Jayendra Parab, Madhan Neethiraj, Nikhil Bonte, > Nixon Rodrigues, and Sarath Subramanian. > > > Bugs: ATLAS-3919 > https://issues.apache.org/jira/browse/ATLAS-3919 > > > Repository: atlas > > > Description > ------- > > **Approach** > Model: _TaskDef_ > - Definition of task. > - Contains status of the task for querying. > > _AbstractTask_ > - Takes TaskDef as input and calls run. > - The concrete tasks has logic for doing real work. > > _TaskFactory_ > > - Creates tasks given task type. > - Enumerates all the tasks that are supported. > > _TaskManagement_ > > - Component/Service. Constructed after all existing components are > constructed. > - Provides a simple interface for adding tasks. > - Processes tasks that are in pending state. > - Supports HA mode, where the tasks are not executed on passive instance. > > _TaskManagement.Registry_ > > - CRUD operations with TaskDef vertex. > > _TaskManagement.Processor_ > - Provides mechanism for provides single queue for processing tasks in a way > that is non-blocking. > - Uses task type and the associated TaskFactory to construct concrete task. > > Changes to _EntityGraphMapper_ > There are 3 new methods added that use portion of existing classification > propagation logic and create a separate method. > > New class _EntityGraphMapperWithTasks_ acts as a proxy for EntityGraphMapper. > All calls that operate without tasks get channeled to the existing > implementation. The ones involving tasks are modified to use the new task > framework. > > **Concurrent Changes to Classifications** > Via REST APIs and operations from web UI, it is possible to make changes to > the same entity. This situation will be more pronounced in the new setup. > > The existing framework has > _GraphTransactionInterceptor.lockObjectAndReleasePostCommit_. This provides > ReentrantLock on an entity that is identified by the supplied GUID. This will > cause a lock on the entity and all the impacted vertices. This has potential > for adversely impacting scalability. > > **New Updates** > - Separate log file for tasks. > - TaskDef now has _startTime_ and _endTime_ to indicating start and end of > task. > - Refactored parameter passing during task creation. It now accepts Map > instead of Object array. Each task creates its own set of parameters. > - Reduced use of TASKS_ENABLE flag checks. > > > Diffs > ----- > > common/src/main/java/org/apache/atlas/repository/Constants.java 61abfcaca > distro/src/conf/atlas-log4j.xml 7df963eb2 > intg/src/main/java/org/apache/atlas/AtlasConfiguration.java 08d6c9d4a > intg/src/main/java/org/apache/atlas/model/instance/AtlasClassification.java > 1edf4eeaf > intg/src/main/java/org/apache/atlas/model/tasks/TaskDef.java PRE-CREATION > intg/src/main/java/org/apache/atlas/type/AtlasType.java b30b483e0 > intg/src/main/java/org/apache/atlas/utils/AtlasJson.java abeddf640 > intg/src/test/resources/atlas-application.properties 7e74d5107 > repository/src/main/java/org/apache/atlas/GraphTransactionInterceptor.java > 86b369fc9 > > repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityStoreV2.java > ce58e9aa4 > > repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphMapper.java > 2cfcc0bc1 > > repository/src/main/java/org/apache/atlas/repository/store/graph/v2/tasks/ClassificationPropagateTaskFactory.java > PRE-CREATION > > repository/src/main/java/org/apache/atlas/repository/store/graph/v2/tasks/ClassificationPropagationTasks.java > PRE-CREATION > > repository/src/main/java/org/apache/atlas/repository/store/graph/v2/tasks/ClassificationTask.java > PRE-CREATION > repository/src/main/java/org/apache/atlas/tasks/AbstractTask.java > PRE-CREATION > repository/src/main/java/org/apache/atlas/tasks/TaskExecutor.java > PRE-CREATION > repository/src/main/java/org/apache/atlas/tasks/TaskFactory.java > PRE-CREATION > repository/src/main/java/org/apache/atlas/tasks/TaskLogger.java > PRE-CREATION > repository/src/main/java/org/apache/atlas/tasks/TaskManagement.java > PRE-CREATION > repository/src/main/java/org/apache/atlas/tasks/TaskRegistry.java > PRE-CREATION > repository/src/main/java/org/apache/atlas/util/BeanUtilRepo.java > PRE-CREATION > > repository/src/test/java/org/apache/atlas/repository/tagpropagation/ClassificationPropagationTest.java > 84e9bfa04 > repository/src/test/java/org/apache/atlas/tasks/TaskExecutorTest.java > PRE-CREATION > repository/src/test/java/org/apache/atlas/tasks/TaskManagementTest.java > PRE-CREATION > repository/src/test/java/org/apache/atlas/tasks/TaskRegistryTest.java > PRE-CREATION > server-api/src/main/java/org/apache/atlas/RequestContext.java 7de3536f4 > > server-api/src/main/java/org/apache/atlas/listener/ActiveStateChangeHandler.java > bb7f8fcc4 > test-tools/src/main/resources/log4j.properties 4db0598ad > webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java > e8fc111a6 > webapp/src/test/java/org/apache/atlas/web/resources/AdminResourceTest.java > 77422b2a5 > > > Diff: https://reviews.apache.org/r/73076/diff/15/ > > > Testing > ------- > > **Unit tests** > New tests. > > **Manual tests** > In-progress. > > **Volume test** > Pending. > > **Pre-commit build** > https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/281/ > > > Thanks, > > Ashutosh Mestry > >
