Re: Review Request 73296: AtlasTasks: Elegant handling of Failover Scenarios

[Ashutosh Mestry via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73296/
---

(Updated April 26, 2021, 5:56 a.m.)


Review request for atlas, Jayendra Parab and Sarath Subramanian.


Changes
---

Updates include: Addressed review comments.


Bugs: ATLAS-4256 and ATLAS-4258
https://issues.apache.org/jira/browse/ATLAS-4256
https://issues.apache.org/jira/browse/ATLAS-4258


Repository: atlas


Description
---

**Background**
Please read bug description.

**Approach**
New: *TaskFactoryRegistry*: Registration process for individual factory.
Modified: *AdminResource* Removed the postConstruct calling of 
*queuePendingTasks*. This is now part of *TaskManagement*, which is better 
design.


Diffs (updated)
-

  
repository/src/main/java/org/apache/atlas/repository/store/graph/v1/DeleteHandlerV1.java
 20d5e6fab 
  
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasGraphUtilsV2.java
 8d4fdf388 
  
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphMapper.java
 d8ef32b32 
  
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/tasks/ClassificationPropagateTaskFactory.java
 6244b2d5d 
  
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/tasks/ClassificationPropagationTasks.java
 4fda34aca 
  
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/tasks/ClassificationTask.java
 369db087b 
  repository/src/main/java/org/apache/atlas/tasks/TaskFactoryRegistry.java 
PRE-CREATION 
  repository/src/main/java/org/apache/atlas/tasks/TaskManagement.java 264aa8cfb 
  
repository/src/test/java/org/apache/atlas/repository/tagpropagation/ClassificationPropagationWithTasksTest.java
 e309a7658 
  webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java 
714b400c8 


Diff: https://reviews.apache.org/r/73296/diff/5/

Changes: https://reviews.apache.org/r/73296/diff/4-5/


Testing
---

**Functional tests**
Manually simulated failover scenarios.

Steps:
- Start Atlas on a cluster with multiple instances.
- Start propagation task that takes few minutes to complete.
- Shutdown active instance.

Expected behavior: Task should be started from the active node.


Thanks,

Ashutosh Mestry

Re: Review Request 73249: ATLAS-4221 : Solr collection names should be configurable

[chaitali]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73249/
---

(Updated April 26, 2021, 5:41 a.m.)


Review request for atlas, Ashutosh Mestry, Jayendra Parab, Nixon Rodrigues, 
Pinal Shah, and Sarath Subramanian.


Bugs: ATLAS-4221
https://issues.apache.org/jira/browse/ATLAS-4221


Repository: atlas


Description
---

Currently in Atlas, the collection names are not configurable

We need to provide functionality where in names of the solr collections can be 
customized using property configuration


Diffs (updated)
-

  common/src/main/java/org/apache/atlas/repository/Constants.java 4df38a5e6 
  distro/src/bin/atlas_config.py 80b3bfb78 
  distro/src/bin/atlas_start.py 7cf35a92a 
  distro/src/conf/atlas-application.properties e06e74a26 
  intg/src/main/java/org/apache/atlas/AtlasConfiguration.java 05b078416 
  repository/src/test/java/org/apache/atlas/TestSolrConfigurableNames.java 
PRE-CREATION 
  repository/src/test/resources/atlas-application.properties PRE-CREATION 
  
tools/atlas-index-repair/src/main/java/org/apache/atlas/tools/RepairIndex.java 
37565188e 


Diff: https://reviews.apache.org/r/73249/diff/6/

Changes: https://reviews.apache.org/r/73249/diff/5-6/


Testing
---

mvn clean install
mvn clean package -Pdist,embedded-hbase-solr
atlas server up and running
Collections created with entities linked


Thanks,

chaitali

Re: Review Request 73296: AtlasTasks: Elegant handling of Failover Scenarios

[Sarath Subramanian]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73296/#review222863
---




repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasGraphUtilsV2.java
Lines 856 (patched)


removeItemToListProperty => removeItemFromListProperty



repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasGraphUtilsV2.java
Lines 860 (patched)


if list doesn't exist in edge (null), this will assign emptylist ([]) to 
the property.

we may need to distinguish between null and empty here.



repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasGraphUtilsV2.java
Lines 865 (patched)


line 853 saves to edge as a 'list' object and line 865 reads as String 
object. Consider using list object to read to and write from edge:

To write:
edge.setListProperty(property, list);

To read:
List list = AtlasGraphUtilsV2.getEncodedProperty(edge, property, 
List.class);


- Sarath Subramanian


On April 25, 2021, 8:29 p.m., Ashutosh Mestry wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73296/
> ---
> 
> (Updated April 25, 2021, 8:29 p.m.)
> 
> 
> Review request for atlas, Jayendra Parab and Sarath Subramanian.
> 
> 
> Bugs: ATLAS-4256 and ATLAS-4258
> https://issues.apache.org/jira/browse/ATLAS-4256
> https://issues.apache.org/jira/browse/ATLAS-4258
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> **Background**
> Please read bug description.
> 
> **Approach**
> New: *TaskFactoryRegistry*: Registration process for individual factory.
> Modified: *AdminResource* Removed the postConstruct calling of 
> *queuePendingTasks*. This is now part of *TaskManagement*, which is better 
> design.
> 
> 
> Diffs
> -
> 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v1/DeleteHandlerV1.java
>  20d5e6fab 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasGraphUtilsV2.java
>  8d4fdf388 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphMapper.java
>  d8ef32b32 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/tasks/ClassificationPropagateTaskFactory.java
>  6244b2d5d 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/tasks/ClassificationPropagationTasks.java
>  4fda34aca 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/tasks/ClassificationTask.java
>  369db087b 
>   repository/src/main/java/org/apache/atlas/tasks/TaskFactoryRegistry.java 
> PRE-CREATION 
>   repository/src/main/java/org/apache/atlas/tasks/TaskManagement.java 
> 264aa8cfb 
>   
> repository/src/test/java/org/apache/atlas/repository/tagpropagation/ClassificationPropagationWithTasksTest.java
>  e309a7658 
>   webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java 
> 714b400c8 
> 
> 
> Diff: https://reviews.apache.org/r/73296/diff/4/
> 
> 
> Testing
> ---
> 
> **Functional tests**
> Manually simulated failover scenarios.
> 
> Steps:
> - Start Atlas on a cluster with multiple instances.
> - Start propagation task that takes few minutes to complete.
> - Shutdown active instance.
> 
> Expected behavior: Task should be started from the active node.
> 
> 
> Thanks,
> 
> Ashutosh Mestry
> 
>

Re: Review Request 73249: ATLAS-4221 : Solr collection names should be configurable

[chaitali]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73249/
---

(Updated April 26, 2021, 4:51 a.m.)


Review request for atlas, Ashutosh Mestry, Jayendra Parab, Nixon Rodrigues, 
Pinal Shah, and Sarath Subramanian.


Bugs: ATLAS-4221
https://issues.apache.org/jira/browse/ATLAS-4221


Repository: atlas


Description
---

Currently in Atlas, the collection names are not configurable

We need to provide functionality where in names of the solr collections can be 
customized using property configuration


Diffs (updated)
-

  common/src/main/java/org/apache/atlas/repository/Constants.java 4df38a5e6 
  distro/src/bin/atlas_config.py 80b3bfb78 
  distro/src/bin/atlas_start.py 7cf35a92a 
  distro/src/conf/atlas-application.properties e06e74a26 
  intg/src/main/java/org/apache/atlas/AtlasConfiguration.java 05b078416 
  repository/src/test/java/org/apache/atlas/TestSolrConfigurableNames.java 
PRE-CREATION 
  repository/src/test/resources/atlas-application.properties PRE-CREATION 
  
tools/atlas-index-repair/src/main/java/org/apache/atlas/tools/RepairIndex.java 
37565188e 


Diff: https://reviews.apache.org/r/73249/diff/5/

Changes: https://reviews.apache.org/r/73249/diff/4-5/


Testing
---

mvn clean install
mvn clean package -Pdist,embedded-hbase-solr
atlas server up and running
Collections created with entities linked


Thanks,

chaitali

Re: Review Request 73296: AtlasTasks: Elegant handling of Failover Scenarios

[Ashutosh Mestry via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73296/
---

(Updated April 26, 2021, 3:29 a.m.)


Review request for atlas, Jayendra Parab and Sarath Subramanian.


Changes
---

Updates include: 
- Addressed review comments.
- Improved logic for handling storing of list in an edge.


Bugs: ATLAS-4256 and ATLAS-4258
https://issues.apache.org/jira/browse/ATLAS-4256
https://issues.apache.org/jira/browse/ATLAS-4258


Repository: atlas


Description
---

**Background**
Please read bug description.

**Approach**
New: *TaskFactoryRegistry*: Registration process for individual factory.
Modified: *AdminResource* Removed the postConstruct calling of 
*queuePendingTasks*. This is now part of *TaskManagement*, which is better 
design.


Diffs (updated)
-

  
repository/src/main/java/org/apache/atlas/repository/store/graph/v1/DeleteHandlerV1.java
 20d5e6fab 
  
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasGraphUtilsV2.java
 8d4fdf388 
  
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphMapper.java
 d8ef32b32 
  
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/tasks/ClassificationPropagateTaskFactory.java
 6244b2d5d 
  
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/tasks/ClassificationPropagationTasks.java
 4fda34aca 
  
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/tasks/ClassificationTask.java
 369db087b 
  repository/src/main/java/org/apache/atlas/tasks/TaskFactoryRegistry.java 
PRE-CREATION 
  repository/src/main/java/org/apache/atlas/tasks/TaskManagement.java 264aa8cfb 
  
repository/src/test/java/org/apache/atlas/repository/tagpropagation/ClassificationPropagationWithTasksTest.java
 e309a7658 
  webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java 
714b400c8 


Diff: https://reviews.apache.org/r/73296/diff/3/

Changes: https://reviews.apache.org/r/73296/diff/2-3/


Testing
---

**Functional tests**
Manually simulated failover scenarios.

Steps:
- Start Atlas on a cluster with multiple instances.
- Start propagation task that takes few minutes to complete.
- Shutdown active instance.

Expected behavior: Task should be started from the active node.


Thanks,

Ashutosh Mestry