Re: Review Request 73423: ATLAS-4338: Import Service: Support change notifiers for migration import

2021-07-11 Thread Ashutosh Mestry via Review Board 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73423/#review223211
---


Ship it!




Ship It!

- Ashutosh Mestry


On June 30, 2021, 11:07 a.m., Nikhil Bonte wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73423/
> ---
> 
> (Updated June 30, 2021, 11:07 a.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Jayendra Parab, madhan, Nixon 
> Rodrigues, Sarath Subramanian, and Sidharth Mishra.
> 
> 
> Bugs: ATLAS-4338
> https://issues.apache.org/jira/browse/ATLAS-4338
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Currently Migration import feature (MigrationImport.java) does not support 
> enabling notifiers (audits writer, notification to Atlas entities topic).
> 
> With this improvement, Atlas now will enable notifiers for Migration import, 
> based on a flag.
> 
>  
> 
> Approach:
> 
> * New option in AtlasImportRequest - sendNotifications (default : false)
> * MigrationImport will use the flag to select appropriate implementation 
> of IAtlasEntityChangeNotifier to create EntityCreationManager.
> * EntityCreationManager will then notify listeners.
> 
> 
> Diffs
> -
> 
>   intg/src/main/java/org/apache/atlas/model/impexp/AtlasImportRequest.java 
> 2c1870444 
>   
> repository/src/main/java/org/apache/atlas/repository/impexp/ImportService.java
>  1d29bf833 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityChangeNotifier.java
>  99be169f2 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/BulkImporterImpl.java
>  8e17fd410 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/bulkimport/GuidMutationResponsePair.java
>  PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/bulkimport/MigrationImport.java
>  d6f23d6e2 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/bulkimport/pc/EntityConsumer.java
>  b73988fd7 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/bulkimport/pc/EntityCreationManager.java
>  734add6d7 
> 
> 
> Diff: https://reviews.apache.org/r/73423/diff/3/
> 
> 
> Testing
> ---
> 
> * Tested Manually
> * Sample request
> {
> "options": {
> "migration": "true",
> "size": "100",
> "sendNotifications": "true",
> "batchSize": 500,
> "numWorkers": 40
> }
> }
> * took ~270 secs to import 12987 entities with notifiers enabled.
> * PC build : 
> https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/679/console
> 
> 
> Thanks,
> 
> Nikhil Bonte
> 
>

Re: Review Request 73423: ATLAS-4338: Import Service: Support change notifiers for migration import

2021-06-30 Thread Nikhil Bonte 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73423/
---

(Updated June 30, 2021, 4:37 p.m.)


Review request for atlas, Ashutosh Mestry, Jayendra Parab, madhan, Nixon 
Rodrigues, Sarath Subramanian, and Sidharth Mishra.


Changes
---

Addressed review comments


Bugs: ATLAS-4338
https://issues.apache.org/jira/browse/ATLAS-4338


Repository: atlas


Description
---

Currently Migration import feature (MigrationImport.java) does not support 
enabling notifiers (audits writer, notification to Atlas entities topic).

With this improvement, Atlas now will enable notifiers for Migration import, 
based on a flag.

 

Approach:

* New option in AtlasImportRequest - sendNotifications (default : false)
* MigrationImport will use the flag to select appropriate implementation of 
IAtlasEntityChangeNotifier to create EntityCreationManager.
* EntityCreationManager will then notify listeners.


Diffs (updated)
-

  intg/src/main/java/org/apache/atlas/model/impexp/AtlasImportRequest.java 
2c1870444 
  
repository/src/main/java/org/apache/atlas/repository/impexp/ImportService.java 
1d29bf833 
  
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityChangeNotifier.java
 99be169f2 
  
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/BulkImporterImpl.java
 8e17fd410 
  
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/bulkimport/GuidMutationResponsePair.java
 PRE-CREATION 
  
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/bulkimport/MigrationImport.java
 d6f23d6e2 
  
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/bulkimport/pc/EntityConsumer.java
 b73988fd7 
  
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/bulkimport/pc/EntityCreationManager.java
 734add6d7 


Diff: https://reviews.apache.org/r/73423/diff/3/

Changes: https://reviews.apache.org/r/73423/diff/2-3/


Testing
---

* Tested Manually
* Sample request
{
"options": {
"migration": "true",
"size": "100",
"sendNotifications": "true",
"batchSize": 500,
"numWorkers": 40
}
}
* took ~270 secs to import 12987 entities with notifiers enabled.
* PC build : 
https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/679/console


Thanks,

Nikhil Bonte

Re: Review Request 73423: ATLAS-4338: Import Service: Support change notifiers for migration import

2021-06-29 Thread Ashutosh Mestry via Review Board 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73423/#review223196
---




intg/src/main/java/org/apache/atlas/model/impexp/AtlasImportRequest.java
Lines 58 (patched)


Rename: OPTION_MIGRATION_WITH_AUDITS



intg/src/main/java/org/apache/atlas/model/impexp/AtlasImportRequest.java
Lines 170 (patched)


Rename: getOptionMigrationWithAudis



repository/src/main/java/org/apache/atlas/repository/store/graph/v2/bulkimport/BulkImportResultItem.java
Lines 23 (patched)


Rename: GuidResponsePair


- Ashutosh Mestry


On June 23, 2021, 2:21 p.m., Nikhil Bonte wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73423/
> ---
> 
> (Updated June 23, 2021, 2:21 p.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Jayendra Parab, madhan, Nixon 
> Rodrigues, Sarath Subramanian, and Sidharth Mishra.
> 
> 
> Bugs: ATLAS-4338
> https://issues.apache.org/jira/browse/ATLAS-4338
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Currently Migration import feature (MigrationImport.java) does not support 
> enabling notifiers (audits writer, notification to Atlas entities topic).
> 
> With this improvement, Atlas now will enable notifiers for Migration import, 
> based on a flag.
> 
>  
> 
> Approach:
> 
> * New option in AtlasImportRequest - sendNotifications (default : false)
> * MigrationImport will use the flag to select appropriate implementation 
> of IAtlasEntityChangeNotifier to create EntityCreationManager.
> * EntityCreationManager will then notify listeners.
> 
> 
> Diffs
> -
> 
>   intg/src/main/java/org/apache/atlas/model/impexp/AtlasImportRequest.java 
> 2c1870444 
>   
> repository/src/main/java/org/apache/atlas/repository/impexp/ImportService.java
>  1d29bf833 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityChangeNotifier.java
>  99be169f2 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/BulkImporterImpl.java
>  8e17fd410 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/bulkimport/BulkImportResultItem.java
>  PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/bulkimport/MigrationImport.java
>  d6f23d6e2 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/bulkimport/pc/EntityConsumer.java
>  b73988fd7 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/bulkimport/pc/EntityCreationManager.java
>  734add6d7 
> 
> 
> Diff: https://reviews.apache.org/r/73423/diff/2/
> 
> 
> Testing
> ---
> 
> * Tested Manually
> * Sample request
> {
> "options": {
> "migration": "true",
> "size": "100",
> "sendNotifications": "true",
> "batchSize": 500,
> "numWorkers": 40
> }
> }
> * took ~270 secs to import 12987 entities with notifiers enabled.
> * PC build : 
> https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/679/console
> 
> 
> Thanks,
> 
> Nikhil Bonte
> 
>

Re: Review Request 73423: ATLAS-4338: Import Service: Support change notifiers for migration import

2021-06-23 Thread Nikhil Bonte 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73423/
---

(Updated June 23, 2021, 7:51 p.m.)


Review request for atlas, Ashutosh Mestry, Jayendra Parab, madhan, Nixon 
Rodrigues, Sarath Subramanian, and Sidharth Mishra.


Changes
---

Addressed review comment


Bugs: ATLAS-4338
https://issues.apache.org/jira/browse/ATLAS-4338


Repository: atlas


Description
---

Currently Migration import feature (MigrationImport.java) does not support 
enabling notifiers (audits writer, notification to Atlas entities topic).

With this improvement, Atlas now will enable notifiers for Migration import, 
based on a flag.

 

Approach:

* New option in AtlasImportRequest - sendNotifications (default : false)
* MigrationImport will use the flag to select appropriate implementation of 
IAtlasEntityChangeNotifier to create EntityCreationManager.
* EntityCreationManager will then notify listeners.


Diffs (updated)
-

  intg/src/main/java/org/apache/atlas/model/impexp/AtlasImportRequest.java 
2c1870444 
  
repository/src/main/java/org/apache/atlas/repository/impexp/ImportService.java 
1d29bf833 
  
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityChangeNotifier.java
 99be169f2 
  
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/BulkImporterImpl.java
 8e17fd410 
  
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/bulkimport/BulkImportResultItem.java
 PRE-CREATION 
  
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/bulkimport/MigrationImport.java
 d6f23d6e2 
  
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/bulkimport/pc/EntityConsumer.java
 b73988fd7 
  
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/bulkimport/pc/EntityCreationManager.java
 734add6d7 


Diff: https://reviews.apache.org/r/73423/diff/2/

Changes: https://reviews.apache.org/r/73423/diff/1-2/


Testing
---

* Tested Manually
* Sample request
{
"options": {
"migration": "true",
"size": "100",
"sendNotifications": "true",
"batchSize": 500,
"numWorkers": 40
}
}
* took ~270 secs to import 12987 entities with notifiers enabled.
* PC build : 
https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/679/console


Thanks,

Nikhil Bonte

Re: Review Request 73423: ATLAS-4338: Import Service: Support change notifiers for migration import

2021-06-23 Thread Ashutosh Mestry via Review Board 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73423/#review223184
---




repository/src/main/java/org/apache/atlas/repository/store/graph/v2/bulkimport/pc/EntityConsumer.java
Lines 118 (patched)


Refactor: Move to separate file.


- Ashutosh Mestry


On June 21, 2021, 2:06 p.m., Nikhil Bonte wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73423/
> ---
> 
> (Updated June 21, 2021, 2:06 p.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Jayendra Parab, madhan, Nixon 
> Rodrigues, Sarath Subramanian, and Sidharth Mishra.
> 
> 
> Bugs: ATLAS-4338
> https://issues.apache.org/jira/browse/ATLAS-4338
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Currently Migration import feature (MigrationImport.java) does not support 
> enabling notifiers (audits writer, notification to Atlas entities topic).
> 
> With this improvement, Atlas now will enable notifiers for Migration import, 
> based on a flag.
> 
>  
> 
> Approach:
> 
> * New option in AtlasImportRequest - sendNotifications (default : false)
> * MigrationImport will use the flag to select appropriate implementation 
> of IAtlasEntityChangeNotifier to create EntityCreationManager.
> * EntityCreationManager will then notify listeners.
> 
> 
> Diffs
> -
> 
>   intg/src/main/java/org/apache/atlas/model/impexp/AtlasImportRequest.java 
> 2c1870444 
>   
> repository/src/main/java/org/apache/atlas/repository/impexp/ImportService.java
>  1d29bf833 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityChangeNotifier.java
>  99be169f2 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/BulkImporterImpl.java
>  8e17fd410 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/bulkimport/MigrationImport.java
>  d6f23d6e2 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/bulkimport/pc/EntityConsumer.java
>  b73988fd7 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/bulkimport/pc/EntityCreationManager.java
>  734add6d7 
> 
> 
> Diff: https://reviews.apache.org/r/73423/diff/1/
> 
> 
> Testing
> ---
> 
> * Tested Manually
> * Sample request
> {
> "options": {
> "migration": "true",
> "size": "100",
> "sendNotifications": "true",
> "batchSize": 500,
> "numWorkers": 40
> }
> }
> * took ~270 secs to import 12987 entities with notifiers enabled.
> * PC build : 
> https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/679/console
> 
> 
> Thanks,
> 
> Nikhil Bonte
> 
>

Re: Review Request 73423: ATLAS-4338: Import Service: Support change notifiers for migration import

2021-06-21 Thread Nikhil Bonte 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73423/
---

(Updated June 21, 2021, 7:36 p.m.)


Review request for atlas, Ashutosh Mestry, Jayendra Parab, madhan, Nixon 
Rodrigues, Sarath Subramanian, and Sidharth Mishra.


Changes
---

Updated PC build link


Bugs: ATLAS-4338
https://issues.apache.org/jira/browse/ATLAS-4338


Repository: atlas


Description
---

Currently Migration import feature (MigrationImport.java) does not support 
enabling notifiers (audits writer, notification to Atlas entities topic).

With this improvement, Atlas now will enable notifiers for Migration import, 
based on a flag.

 

Approach:

* New option in AtlasImportRequest - sendNotifications (default : false)
* MigrationImport will use the flag to select appropriate implementation of 
IAtlasEntityChangeNotifier to create EntityCreationManager.
* EntityCreationManager will then notify listeners.


Diffs
-

  intg/src/main/java/org/apache/atlas/model/impexp/AtlasImportRequest.java 
2c1870444 
  
repository/src/main/java/org/apache/atlas/repository/impexp/ImportService.java 
1d29bf833 
  
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityChangeNotifier.java
 99be169f2 
  
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/BulkImporterImpl.java
 8e17fd410 
  
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/bulkimport/MigrationImport.java
 d6f23d6e2 
  
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/bulkimport/pc/EntityConsumer.java
 b73988fd7 
  
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/bulkimport/pc/EntityCreationManager.java
 734add6d7 


Diff: https://reviews.apache.org/r/73423/diff/1/


Testing (updated)
---

* Tested Manually
* Sample request
{
"options": {
"migration": "true",
"size": "100",
"sendNotifications": "true",
"batchSize": 500,
"numWorkers": 40
}
}
* took ~270 secs to import 12987 entities with notifiers enabled.
* PC build : 
https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/679/console


Thanks,

Nikhil Bonte

Review Request 73423: ATLAS-4338: Import Service: Support change notifiers for migration import

2021-06-15 Thread Nikhil Bonte 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73423/
---

Review request for atlas, Ashutosh Mestry, Jayendra Parab, madhan, Nixon 
Rodrigues, Sarath Subramanian, and Sidharth Mishra.


Bugs: ATLAS-4338
https://issues.apache.org/jira/browse/ATLAS-4338


Repository: atlas


Description
---

Currently Migration import feature (MigrationImport.java) does not support 
enabling notifiers (audits writer, notification to Atlas entities topic).

With this improvement, Atlas now will enable notifiers for Migration import, 
based on a flag.

 

Approach:

* New option in AtlasImportRequest - sendNotifications (default : false)
* MigrationImport will use the flag to select appropriate implementation of 
IAtlasEntityChangeNotifier to create EntityCreationManager.
* EntityCreationManager will then notify listeners.


Diffs
-

  intg/src/main/java/org/apache/atlas/model/impexp/AtlasImportRequest.java 
2c1870444 
  
repository/src/main/java/org/apache/atlas/repository/impexp/ImportService.java 
1d29bf833 
  
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityChangeNotifier.java
 99be169f2 
  
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/BulkImporterImpl.java
 8e17fd410 
  
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/bulkimport/MigrationImport.java
 d6f23d6e2 
  
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/bulkimport/pc/EntityConsumer.java
 b73988fd7 
  
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/bulkimport/pc/EntityCreationManager.java
 734add6d7 


Diff: https://reviews.apache.org/r/73423/diff/1/


Testing
---

* Tested Manually
* Attached sample zip, sample request
{
"options": {
"migration": "true",
"size": "100",
"sendNotifications": "true",
"batchSize": 500,
"numWorkers": 40
}
}
* took ~270 secs to import 12987 entities with notifiers enabled.
* PC build :


File Attachments


largedb.zip
  
https://reviews.apache.org/media/uploaded/files/2021/06/15/c84f450b-c2b3-4cac-8b4a-76942a2c14e3__largedb.zip


Thanks,

Nikhil Bonte