Add me as a contributor for Atlas

[Nikhil Bonte]

Hi,

Please add me as a contributor for Atlas.
My jira id is nikhilbonte.
-- 
Regards
Nikhil P. Bonte

Re: Review Request 69179: Export: Connected: Adds Same Entity GUID to Creation Order Causing Failure to Import

[Nikhil Bonte]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69179/#review210087
---


Ship it!




Ship It!

- Nikhil Bonte


On Oct. 26, 2018, 5:26 a.m., Ashutosh Mestry wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/69179/
> ---
> 
> (Updated Oct. 26, 2018, 5:26 a.m.)
> 
> 
> Review request for atlas, Kapildeo Nayak, Madhan Neethiraj, and Nixon 
> Rodrigues.
> 
> 
> Bugs: ATLAS-2900
> https://issues.apache.org/jira/browse/ATLAS-2900
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> **Root Cause**
> The starting entity is a table. During navigation to discover more entities, 
> the entities that are discovered also discover table using column connection. 
> This causes table to be re-added to the creation order list.
> 
> **Approach**
> 
> Convert entity creation order to unique list.
> 
> **CURL**
> 
> ```
> curl -X POST -u admin:admin -H "Content-Type: application/json" -H 
> "Cache-Control: no-cache" -d @../docs/export-options.json 
> http://localhost:21000/api/atlas/admin/export > ../docs/hr-1.zip
> ```
> 
> 
> Diffs
> -
> 
>   
> repository/src/main/java/org/apache/atlas/repository/impexp/ExportService.java
>  5e972a2cd 
>   repository/src/main/java/org/apache/atlas/repository/util/UniqueList.java 
> eebbc4eff 
>   
> repository/src/test/java/org/apache/atlas/repository/impexp/ExportIncrementalTest.java
>  86afd7f43 
>   
> repository/src/test/java/org/apache/atlas/repository/impexp/ExportServiceTest.java
>  18e7c419f 
>   repository/src/test/resources/json/stocksDB-Entities/export-connected.json 
> PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/69179/diff/1/
> 
> 
> Testing
> ---
> 
> **Unit tests**
> - Additional tests.
> - Minor modification to existing test.
> 
> **Functional tests**
> Using CURL calls.
> 
> 
> Thanks,
> 
> Ashutosh Mestry
> 
>

Review Request 69456: ATLAS-2979 Added service type dropdown to filter entitydef type - server side changes

[Nikhil Bonte]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69456/
---

Review request for atlas, Madhan Neethiraj, Nixon Rodrigues, and Sarath 
Subramanian.


Bugs: ATLAS-2979
https://issues.apache.org/jira/browse/ATLAS-2979


Repository: atlas


Description
---

ATLAS-2979 Added service type dropdown to filter entitydef type - server side 
changes


Diffs
-

  intg/src/main/java/org/apache/atlas/model/typedef/AtlasTypeDefHeader.java 
b0edcc0 


Diff: https://reviews.apache.org/r/69456/diff/1/


Testing
---

serviceType attribute is added to each entity type header.

[{"guid":"f7d9df46-ca81-43ab-a302-d2ea007ca152","name":"hive_table_db","serviceType":"hive","category":"RELATIONSHIP"},]


Thanks,

Nikhil Bonte

Re: Review Request 69392: ATLAS-2810 - Add notifications from RelationshipStore

[Nikhil Bonte]

ifiedName":"db2@qn","replicatedFrom":[],"replicatedTo":[],"clusterName":"cluster0","location":null,"ownerType":null},"guid":-1},"referredEntities":{}}

hive_table
request body - 
{"entity":{"typeName":"hive_table","attributes":{"description":null,"name":"tbl2","owner":null,"qualifiedName":"tbl2@qn","replicatedFrom":[],"replicatedTo":[],"aliases":[],"columns":[],"comment":null,"createTime":154455300,"db":{"guid":"cea76431-af27-43ae-ab11-98a2ef989bb8","typeName":"hive_db"},"lastAccessTime":154455300,"partitionKeys":[],"retention":null,"sd":null,"tableType":null,"temporary":null,"viewExpandedText":null,"viewOriginalText":null},"guid":-1},"referredEntities":{}}

## Update relationship - 
API - http://localhost:21000/api/atlas/v2/relationship - PUT
request body - 
{"typeName":"hive_table_db","guid":"8a8b8dc2-a163-42f0-b706-01c30cb0a688","provenanceType":0,"end1":{"guid":"6da63233-d449-42d5-8449-b6ad243f6af2","typeName":"hive_table"},"end2":{"guid":"cea76431-af27-43ae-ab11-98a2ef989bb8","typeName":"hive_db"},"label":"__hive_table.db","propagateTags":"NONE","status":"ACTIVE","createdBy":"admin","updatedBy":"admin","createTime":1542630658904,"updateTime":1542630658904,"version":0,"propagatedClassifications":[],"blockedPropagatedClassifications":[]}

## Delete relationship -
API - 
http://localhost:21000/api/atlas/v2/relationship/guid/8a8b8dc2-a163-42f0-b706-01c30cb0a688
 - DELETE


Thanks,

Nikhil Bonte

Re: Review Request 69392: ATLAS-2810 - Add notifications from RelationshipStore

[Nikhil Bonte]

ifiedName":"db2@qn","replicatedFrom":[],"replicatedTo":[],"clusterName":"cluster0","location":null,"ownerType":null},"guid":-1},"referredEntities":{}}

hive_table
request body - 
{"entity":{"typeName":"hive_table","attributes":{"description":null,"name":"tbl2","owner":null,"qualifiedName":"tbl2@qn","replicatedFrom":[],"replicatedTo":[],"aliases":[],"columns":[],"comment":null,"createTime":154455300,"db":{"guid":"cea76431-af27-43ae-ab11-98a2ef989bb8","typeName":"hive_db"},"lastAccessTime":154455300,"partitionKeys":[],"retention":null,"sd":null,"tableType":null,"temporary":null,"viewExpandedText":null,"viewOriginalText":null},"guid":-1},"referredEntities":{}}

## Update relationship - 
API - http://localhost:21000/api/atlas/v2/relationship - PUT
request body - 
{"typeName":"hive_table_db","guid":"8a8b8dc2-a163-42f0-b706-01c30cb0a688","provenanceType":0,"end1":{"guid":"6da63233-d449-42d5-8449-b6ad243f6af2","typeName":"hive_table"},"end2":{"guid":"cea76431-af27-43ae-ab11-98a2ef989bb8","typeName":"hive_db"},"label":"__hive_table.db","propagateTags":"NONE","status":"ACTIVE","createdBy":"admin","updatedBy":"admin","createTime":1542630658904,"updateTime":1542630658904,"version":0,"propagatedClassifications":[],"blockedPropagatedClassifications":[]}

## Delete relationship -
API - 
http://localhost:21000/api/atlas/v2/relationship/guid/8a8b8dc2-a163-42f0-b706-01c30cb0a688
 - DELETE


Thanks,

Nikhil Bonte

Re: Review Request 69392: ATLAS-2810 - Add notifications from RelationshipStore

[Nikhil Bonte]

uot;:"db2","owner":null,"qualifiedName":"db2@qn","replicatedFrom":[],"replicatedTo":[],"clusterName":"cluster0","location":null,"ownerType":null},"guid":-1},"referredEntities":{}}

hive_table
request body - 
{"entity":{"typeName":"hive_table","attributes":{"description":null,"name":"tbl2","owner":null,"qualifiedName":"tbl2@qn","replicatedFrom":[],"replicatedTo":[],"aliases":[],"columns":[],"comment":null,"createTime":154455300,"db":{"guid":"cea76431-af27-43ae-ab11-98a2ef989bb8","typeName":"hive_db"},"lastAccessTime":154455300,"partitionKeys":[],"retention":null,"sd":null,"tableType":null,"temporary":null,"viewExpandedText":null,"viewOriginalText":null},"guid":-1},"referredEntities":{}}

## Update relationship - 
API - http://localhost:21000/api/atlas/v2/relationship - PUT
request body - 
{"typeName":"hive_table_db","guid":"8a8b8dc2-a163-42f0-b706-01c30cb0a688","provenanceType":0,"end1":{"guid":"6da63233-d449-42d5-8449-b6ad243f6af2","typeName":"hive_table"},"end2":{"guid":"cea76431-af27-43ae-ab11-98a2ef989bb8","typeName":"hive_db"},"label":"__hive_table.db","propagateTags":"NONE","status":"ACTIVE","createdBy":"admin","updatedBy":"admin","createTime":1542630658904,"updateTime":1542630658904,"version":0,"propagatedClassifications":[],"blockedPropagatedClassifications":[]}

## Delete relationship -
API - 
http://localhost:21000/api/atlas/v2/relationship/guid/8a8b8dc2-a163-42f0-b706-01c30cb0a688
 - DELETE


Thanks,

Nikhil Bonte

Re: Review Request 69392: ATLAS-2810 - Add notifications from RelationshipStore

[Nikhil Bonte]

uot;:"db2","owner":null,"qualifiedName":"db2@qn","replicatedFrom":[],"replicatedTo":[],"clusterName":"cluster0","location":null,"ownerType":null},"guid":-1},"referredEntities":{}}

hive_table
request body - 
{"entity":{"typeName":"hive_table","attributes":{"description":null,"name":"tbl2","owner":null,"qualifiedName":"tbl2@qn","replicatedFrom":[],"replicatedTo":[],"aliases":[],"columns":[],"comment":null,"createTime":154455300,"db":{"guid":"cea76431-af27-43ae-ab11-98a2ef989bb8","typeName":"hive_db"},"lastAccessTime":154455300,"partitionKeys":[],"retention":null,"sd":null,"tableType":null,"temporary":null,"viewExpandedText":null,"viewOriginalText":null},"guid":-1},"referredEntities":{}}

## Update relationship - 
API - http://localhost:21000/api/atlas/v2/relationship - PUT
request body - 
{"typeName":"hive_table_db","guid":"8a8b8dc2-a163-42f0-b706-01c30cb0a688","provenanceType":0,"end1":{"guid":"6da63233-d449-42d5-8449-b6ad243f6af2","typeName":"hive_table"},"end2":{"guid":"cea76431-af27-43ae-ab11-98a2ef989bb8","typeName":"hive_db"},"label":"__hive_table.db","propagateTags":"NONE","status":"ACTIVE","createdBy":"admin","updatedBy":"admin","createTime":1542630658904,"updateTime":1542630658904,"version":0,"propagatedClassifications":[],"blockedPropagatedClassifications":[]}

## Delete relationship -
API - 
http://localhost:21000/api/atlas/v2/relationship/guid/8a8b8dc2-a163-42f0-b706-01c30cb0a688
 - DELETE


Thanks,

Nikhil Bonte

Re: Review Request 69392: ATLAS-2810 - Add notifications from RelationshipStore

[Nikhil Bonte]

API - http://localhost:21000/api/atlas/v2/entity - POST

hive_db
request body - 
{"entity":{"typeName":"hive_db","attributes":{"description":null,"name":"db2","owner":null,"qualifiedName":"db2@qn","replicatedFrom":[],"replicatedTo":[],"clusterName":"cluster0","location":null,"ownerType":null},"guid":-1},"referredEntities":{}}

hive_table
request body - 
{"entity":{"typeName":"hive_table","attributes":{"description":null,"name":"tbl2","owner":null,"qualifiedName":"tbl2@qn","replicatedFrom":[],"replicatedTo":[],"aliases":[],"columns":[],"comment":null,"createTime":154455300,"db":{"guid":"cea76431-af27-43ae-ab11-98a2ef989bb8","typeName":"hive_db"},"lastAccessTime":154455300,"partitionKeys":[],"retention":null,"sd":null,"tableType":null,"temporary":null,"viewExpandedText":null,"viewOriginalText":null},"guid":-1},"referredEntities":{}}

## Update relationship - 
API - http://localhost:21000/api/atlas/v2/relationship - PUT
request body - 
{"typeName":"hive_table_db","guid":"8a8b8dc2-a163-42f0-b706-01c30cb0a688","provenanceType":0,"end1":{"guid":"6da63233-d449-42d5-8449-b6ad243f6af2","typeName":"hive_table"},"end2":{"guid":"cea76431-af27-43ae-ab11-98a2ef989bb8","typeName":"hive_db"},"label":"__hive_table.db","propagateTags":"NONE","status":"ACTIVE","createdBy":"admin","updatedBy":"admin","createTime":1542630658904,"updateTime":1542630658904,"version":0,"propagatedClassifications":[],"blockedPropagatedClassifications":[]}

## Delete relationship -
API - 
http://localhost:21000/api/atlas/v2/relationship/guid/8a8b8dc2-a163-42f0-b706-01c30cb0a688
 - DELETE


Thanks,

Nikhil Bonte

Re: Review Request 69392: ATLAS-2810 - Add notifications from RelationshipStore

[Nikhil Bonte]

> On Dec. 20, 2018, 7:42 p.m., Sarath Subramanian wrote:
> > conside including qualifiedName/unique attributes for end1 and end2 in the 
> > notification sent out.

done


- Nikhil


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69392/#review211448
---


On Dec. 20, 2018, 10:13 a.m., Nikhil Bonte wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/69392/
> ---
> 
> (Updated Dec. 20, 2018, 10:13 a.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Nixon Rodrigues, and Sarath 
> Subramanian.
> 
> 
> Bugs: ATLAS-2810
> https://issues.apache.org/jira/browse/ATLAS-2810
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Notifications for relationship mutations.
> This will add message to kafka topic when relationship is 
> added/updated/deleted.
> 
> Sample messages :
> 
> ADD
> {"version":{"version":"1.0.0","versionParts":[1]},"msgCompressionKind":"NONE","msgSplitIdx":1,"msgSplitCount":1,"msgSourceIP":"127.0.1.1","msgCreatedBy":"","msgCreationTime":1544787083448,"message":{"type":"ENTITY_NOTIFICATION_V2","relationship":{"typeName":"hive_table_db","guid":"e4949325-562a-43bf-a980-c0c562c8759e","status":"ACTIVE","label":"__hive_table.db","end1":{"guid":"6da63233-d449-42d5-8449-b6ad243f6af2","typeName":"hive_table","uniqueAttributes":{"qualifiedName":"tbl2@qn"}},"end2":{"guid":"cea76431-af27-43ae-ab11-98a2ef989bb8","typeName":"hive_db","uniqueAttributes":{"qualifiedName":"db2@qn"}}},"operationType":"RELATIONSHIP_ADD","eventTime":1544787072298}}
> 
> UPDATE
> {"version":{"version":"1.0.0","versionParts":[1]},"msgCompressionKind":"NONE","msgSplitIdx":1,"msgSplitCount":1,"msgSourceIP":"127.0.1.1","msgCreatedBy":"","msgCreationTime":1544787404692,"message":{"type":"ENTITY_NOTIFICATION_V2","relationship":{"typeName":"hive_table_db","guid":"e4949325-562a-43bf-a980-c0c562c8759e","status":"ACTIVE","label":"__hive_table.db","end1":{"guid":"6da63233-d449-42d5-8449-b6ad243f6af2","typeName":"hive_table","uniqueAttributes":{"qualifiedName":"tbl2@qn"}},"end2":{"guid":"cea76431-af27-43ae-ab11-98a2ef989bb8","typeName":"hive_db","uniqueAttributes":{"qualifiedName":"db2@qn"}}},"operationType":"RELATIONSHIP_UPDATE","eventTime":1544787404679}}
> 
> DELETE
> {"version":{"version":"1.0.0","versionParts":[1]},"msgCompressionKind":"NONE","msgSplitIdx":1,"msgSplitCount":1,"msgSourceIP":"127.0.1.1","msgCreatedBy":"","msgCreationTime":1544787460081,"message":{"type":"ENTITY_NOTIFICATION_V2","relationship":{"typeName":"hive_table_db","guid":"e4949325-562a-43bf-a980-c0c562c8759e","status":"DELETED","label":"__hive_table.db","end1":{"guid":"6da63233-d449-42d5-8449-b6ad243f6af2","typeName":"hive_table","uniqueAttributes":{"qualifiedName":"tbl2@qn"}},"end2":{"guid":"cea76431-af27-43ae-ab11-98a2ef989bb8","typeName":"hive_db","uniqueAttributes":{"qualifiedName":"db2@qn"}}},"operationType":"RELATIONSHIP_DELETE","eventTime":1544787459708}}
> 
> 
> Diffs
> -
> 
>   intg/src/main/java/org/apache/atlas/listener/EntityChangeListenerV2.java 
> cccf0d4 
>   
> intg/src/main/java/org/apache/atlas/model/instance/AtlasRelationshipHeader.java
>  PRE-CREATION 
>   
> intg/src/main/java/org/apache/atlas/model/notification/EntityNotification.java
>  1eae100 

Re: Review Request 69392: ATLAS-2810 - Add notifications from RelationshipStore

[Nikhil Bonte]

rom REST client(Postman).
1) Added/updated/deleted relationship by hitting REST APIs(see details below)
2) Verified message is sent to kafka using kafka-console-consumer
   ./kafka-console-consumer.sh --bootstrap-server localhost:9027 --topic 
ATLAS_ENTITIES --from-beginning


## Add relationship - create hive_db then create hive_table. This will create 
new relationship
API - http://localhost:21000/api/atlas/v2/entity - POST

hive_db
request body - 
{"entity":{"typeName":"hive_db","attributes":{"description":null,"name":"db2","owner":null,"qualifiedName":"db2@qn","replicatedFrom":[],"replicatedTo":[],"clusterName":"cluster0","location":null,"ownerType":null},"guid":-1},"referredEntities":{}}

hive_table
request body - 
{"entity":{"typeName":"hive_table","attributes":{"description":null,"name":"tbl2","owner":null,"qualifiedName":"tbl2@qn","replicatedFrom":[],"replicatedTo":[],"aliases":[],"columns":[],"comment":null,"createTime":154455300,"db":{"guid":"cea76431-af27-43ae-ab11-98a2ef989bb8","typeName":"hive_db"},"lastAccessTime":154455300,"partitionKeys":[],"retention":null,"sd":null,"tableType":null,"temporary":null,"viewExpandedText":null,"viewOriginalText":null},"guid":-1},"referredEntities":{}}

## Update relationship - 
API - http://localhost:21000/api/atlas/v2/relationship - PUT
request body - 
{"typeName":"hive_table_db","guid":"8a8b8dc2-a163-42f0-b706-01c30cb0a688","provenanceType":0,"end1":{"guid":"6da63233-d449-42d5-8449-b6ad243f6af2","typeName":"hive_table"},"end2":{"guid":"cea76431-af27-43ae-ab11-98a2ef989bb8","typeName":"hive_db"},"label":"__hive_table.db","propagateTags":"NONE","status":"ACTIVE","createdBy":"admin","updatedBy":"admin","createTime":1542630658904,"updateTime":1542630658904,"version":0,"propagatedClassifications":[],"blockedPropagatedClassifications":[]}

## Delete relationship -
API - 
http://localhost:21000/api/atlas/v2/relationship/guid/8a8b8dc2-a163-42f0-b706-01c30cb0a688
 - DELETE


Thanks,

Nikhil Bonte

not able assign apache jira

[Nikhil Bonte]

Hi,

I am not able to assign jira to myself.
My jira id - *nikhilbonte.*

-- 
Regards
Nikhil P. Bonte

Re: Review Request 69392: ATLAS-2810 - Add notifications from RelationshipStore

[Nikhil Bonte]

/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphRetriever.java
 066abc1 
  
webapp/src/main/java/org/apache/atlas/notification/EntityNotificationListenerV2.java
 ef9ebab 


Diff: https://reviews.apache.org/r/69392/diff/10/

Changes: https://reviews.apache.org/r/69392/diff/9-10/


Testing
---

Tested from REST client(Postman).
1) Added/updated/deleted relationship by hitting REST APIs(see details below)
2) Verified message is sent to kafka using kafka-console-consumer
   ./kafka-console-consumer.sh --bootstrap-server localhost:9027 --topic 
ATLAS_ENTITIES --from-beginning


## Add relationship - create hive_db then create hive_table. This will create 
new relationship
API - http://localhost:21000/api/atlas/v2/entity - POST

hive_db
request body - 
{"entity":{"typeName":"hive_db","attributes":{"description":null,"name":"db2","owner":null,"qualifiedName":"db2@qn","replicatedFrom":[],"replicatedTo":[],"clusterName":"cluster0","location":null,"ownerType":null},"guid":-1},"referredEntities":{}}

hive_table
request body - 
{"entity":{"typeName":"hive_table","attributes":{"description":null,"name":"tbl2","owner":null,"qualifiedName":"tbl2@qn","replicatedFrom":[],"replicatedTo":[],"aliases":[],"columns":[],"comment":null,"createTime":154455300,"db":{"guid":"cea76431-af27-43ae-ab11-98a2ef989bb8","typeName":"hive_db"},"lastAccessTime":154455300,"partitionKeys":[],"retention":null,"sd":null,"tableType":null,"temporary":null,"viewExpandedText":null,"viewOriginalText":null},"guid":-1},"referredEntities":{}}

## Update relationship - 
API - http://localhost:21000/api/atlas/v2/relationship - PUT
request body - 
{"typeName":"hive_table_db","guid":"8a8b8dc2-a163-42f0-b706-01c30cb0a688","provenanceType":0,"end1":{"guid":"6da63233-d449-42d5-8449-b6ad243f6af2","typeName":"hive_table"},"end2":{"guid":"cea76431-af27-43ae-ab11-98a2ef989bb8","typeName":"hive_db"},"label":"__hive_table.db","propagateTags":"NONE","status":"ACTIVE","createdBy":"admin","updatedBy":"admin","createTime":1542630658904,"updateTime":1542630658904,"version":0,"propagatedClassifications":[],"blockedPropagatedClassifications":[]}

## Delete relationship -
API - 
http://localhost:21000/api/atlas/v2/relationship/guid/8a8b8dc2-a163-42f0-b706-01c30cb0a688
 - DELETE


Thanks,

Nikhil Bonte

Re: Review Request 69392: ATLAS-2810 - Add notifications from RelationshipStore

[Nikhil Bonte]

ever.java
 066abc1 
  
webapp/src/main/java/org/apache/atlas/notification/EntityNotificationListenerV2.java
 ef9ebab 


Diff: https://reviews.apache.org/r/69392/diff/11/

Changes: https://reviews.apache.org/r/69392/diff/10-11/


Testing
---

Tested from REST client(Postman).
1) Added/updated/deleted relationship by hitting REST APIs(see details below)
2) Verified message is sent to kafka using kafka-console-consumer
   ./kafka-console-consumer.sh --bootstrap-server localhost:9027 --topic 
ATLAS_ENTITIES --from-beginning


## Add relationship - create hive_db then create hive_table. This will create 
new relationship
API - http://localhost:21000/api/atlas/v2/entity - POST

hive_db
request body - 
{"entity":{"typeName":"hive_db","attributes":{"description":null,"name":"db2","owner":null,"qualifiedName":"db2@qn","replicatedFrom":[],"replicatedTo":[],"clusterName":"cluster0","location":null,"ownerType":null},"guid":-1},"referredEntities":{}}

hive_table
request body - 
{"entity":{"typeName":"hive_table","attributes":{"description":null,"name":"tbl2","owner":null,"qualifiedName":"tbl2@qn","replicatedFrom":[],"replicatedTo":[],"aliases":[],"columns":[],"comment":null,"createTime":154455300,"db":{"guid":"cea76431-af27-43ae-ab11-98a2ef989bb8","typeName":"hive_db"},"lastAccessTime":154455300,"partitionKeys":[],"retention":null,"sd":null,"tableType":null,"temporary":null,"viewExpandedText":null,"viewOriginalText":null},"guid":-1},"referredEntities":{}}

## Update relationship - 
API - http://localhost:21000/api/atlas/v2/relationship - PUT
request body - 
{"typeName":"hive_table_db","guid":"8a8b8dc2-a163-42f0-b706-01c30cb0a688","provenanceType":0,"end1":{"guid":"6da63233-d449-42d5-8449-b6ad243f6af2","typeName":"hive_table"},"end2":{"guid":"cea76431-af27-43ae-ab11-98a2ef989bb8","typeName":"hive_db"},"label":"__hive_table.db","propagateTags":"NONE","status":"ACTIVE","createdBy":"admin","updatedBy":"admin","createTime":1542630658904,"updateTime":1542630658904,"version":0,"propagatedClassifications":[],"blockedPropagatedClassifications":[]}

## Delete relationship -
API - 
http://localhost:21000/api/atlas/v2/relationship/guid/8a8b8dc2-a163-42f0-b706-01c30cb0a688
 - DELETE


Thanks,

Nikhil Bonte

Re: [VOTE] Release Apache Atlas version 2.0.0 - rc0

[Nikhil Bonte]

+1 Release Apache Atlas 2.0.0 - rc0 release

- Built successfully from the source tar file
- Verified PGP signature
- Ran quick start and validated basic functionality.
- Verified following features:
- Atlas Index Repair tool for JanusGraph
- Update stats/metrics module to collect notification metrics
- Add notifications from RelationshipStore to ATLAS_ENTITITES kafka topic

Thanks
Nikhil P. Bonte

On Mon, Apr 22, 2019 at 11:32 AM Sarath Subramanian 
wrote:

> Apache Atlas 2.0.0 release candidate #0 is now available for vote within
> dev community.
>
> Links to the release artifacts are given below. Please review and vote.
>
> The vote will be open for at least 72 hours or until necessary votes are
> reached.
>   [ ] +1 Approve
>   [ ] +0 No opinion
>   [ ] -1 Disapprove (and reason why)
>
> List of all issues addressed in this release:
>
> https://issues.apache.org/jira/issues/?jql=project%20%3D%20ATLAS%20AND%20status%20%3D%20Resolved%20AND%20fixVersion%20%3D%202.0.0%20ORDER%20BY%20priority%20DESC%2C%20updated%20DESC
>
> Git tag for the release:
> https://github.com/apache/atlas/tree/release-2.0.0-rc0
> Sources for the release:
>
> https://dist.apache.org/repos/dist/dev/atlas/2.0.0-rc0/apache-atlas-2.0.0-sources.tar.gz
>
> Source release verification:
> PGP Signature:
>
> https://dist.apache.org/repos/dist/dev/atlas/2.0.0-rc0/apache-atlas-2.0.0-sources.tar.gz.asc
> SHA512
> 
> Hash:
>
> https://dist.apache.org/repos/dist/dev/atlas/2.0.0-rc0/apache-atlas-2.0.0-sources.tar.gz.sha512
> MD5
> 
> Hash:
>
> https://dist.apache.org/repos/dist/dev/atlas/2.0.0-rc0/apache-atlas-2.0.0-sources.tar.gz.md5
>
> Keys to verify the signature of the release artifacts are available at:
> https://dist.apache.org/repos/dist/release/atlas/KEYS
>
> New features/enhancements:
>   - Update component versions to use Hadoop 3, HBase 2 and Solr 7
>   - Update JanusGraph version to 0.3.1
>   - Soft-reference attribute implementation.
>   - Create patch framework to persist typedef patches applied to atlas
>   - Handle data patches in patch framework
>   - Export Process: Support for incremental export.
>   - Atlas Import Transform handler implementation
>   - Update authentication to support trusted proxy
>   - Atlas Index Repair tool for JanusGraph
>   - Update stats/metrics module to collect notification metrics
>   - Add filter in lineage to hide Process entity.
>   - Add service-type dropdown to filter entitydef type
>   - Add notifications from RelationshipStore to ATLAS_ENTITITES kafka
> topic.
>   - Added unique-attributes constraints at store-level
>   - Performance enhancement to avoid unnecessary lookup when creating new
> relationships
>   - Updated notification processing to support batch-commits
>   - Updated hook notification processing with option to ignore potentially
> incorrect hive_column_lineage
>   - Updated Hive hook to avoid duplicate column-lineage entities; also
> updated Atlas server to skip duplicate column-lineage entities
>   - Improved batch processing in notificaiton handler to avoid processing
> of an entity multiple times
>   - Add option to ignore/prune metadata for temporary/staging Hive tables
>   - Update metrics query to use index query instead of graph query for
> faster response
>   - UI: Show counts besides the Type and Classification dropdown list in
> search
>   - UI: show lineage information for process entities
>   - UI: Show entity specific icon for the lineage node
>   - UI: Add relationship table inside relationships tab
>
>
> Thanks,
> Sarath
> 
>


-- 
Regards
Nikhil P. Bonte

Re: [VOTE] Release Apache Atlas version 2.0.0 - rc1

[Nikhil Bonte]

+1 Release Apache 2.0.0 release candidate #1

- Built successfully from the source tar file
- Ran quick start and validated basic functionality.
- Verified following features:
- Atlas Index Repair tool for JanusGraph
- Update stats/metrics module to collect notification metrics
- Add notifications from RelationshipStore to ATLAS_ENTITITES kafka topic

Thanks
Nikhil P. Bonte

On Thu, May 2, 2019 at 1:24 PM Madhan Neethiraj  wrote:

> Sarath - thanks for the updated release candidate.
>
> +1 for Apache Atlas 2.0.0 release candidate #1
>
> Performed the following:
> - downloaded tar.gz and verified the signature
> - built and installed Atlas for embedded HBase, Solr profile
> - started Atlas with embedded HBase and Solr
> - ran QuickStart successfully
> - added/updated/deleted few 'Favorite Searches' - both basic and DSL
> - performed number of searches - both basic and DSL
> - created glossaries with number of terms, associated terms with entities,
> associated terms with classifications
> - verified classification propagations
>
> Thanks,
> Madhan
>
>
>
>
> On 4/29/19, 11:56 PM, "Sarath Subramanian"  wrote:
>
> Atlas team,
>
> Apache Atlas 2.0.0 Release Candidate #1 is now available for a vote
> within
> dev community.
>
> Following commits went into branch-2.0 since the last release candidate
> (rc0):
>
>  - ATLAS-3168: Fix Intermittent UT failure:
> KafkaNotificationTest.initNotificationService()
>  - ATLAS-3148: Implement Hive Metastore hook for Atlas
>  - ATLAS-3167: Exception while starting Atlas with Cassandra
>  - ATLAS-3160: Remove unthrown exception specified in
> AtlasRelationshipDef
> constructors
>  - ATLAS-3157: Add Integration tests for Hive metastore hook
>  - ATLAS-3174: UI - Exclude process or deleted entity from search
> drop-down
> if respective options are chosen
>  - ATLAS-3155: UI - IE-11 - Name entry not clickable
>  - ATLAS-3168: PatchFix: Unit test fixes and optimization.
>  - ATLAS-3168: PatchFix: Support for HA Mode
>  - ATLAS-3129: Fix SSL Truststore reloader leak from AtlasBaseClient
>  - ATLAS-3159: UI - Add Reset button for Lienage
>  - ATLAS-3154: UI - Lineage PNG export not working on Firefox
>  - ATLAS-3158: UI - Render schema tab using relationshipAttributes
> (Columns)
>  - ATLAS-3145: UI - Entity showing Active even if the
> relationshipStatus is
> Deleted.
>
> Links to the release artifacts are given below. Please review and vote.
>
> The vote will be open for at least 72 hours or until necessary votes
> are
> reached.
>   [ ] +1 approve
>   [ ] +0 no opinion
>   [ ] -1 disapprove (and reason why)
>
>
> Thanks,
> Sarath
>
> List of issues addressed in this release:
>
> https://issues.apache.org/jira/issues/?jql=project%20%3D%20ATLAS%20AND%20status%20%3D%20Resolved%20AND%20fixVersion%20%3D%202.0.0%20ORDER%20BY%20priority%20DESC%2C%20updated%20DESC
> Git tag for the release:
> https://github.com/apache/atlas/tree/release-2.0.0-rc1
> Sources for the release:
>
> https://dist.apache.org/repos/dist/dev/atlas/2.0.0-rc1/apache-atlas-2.0.0-sources.tar.gz
>
> Source release verification:
> PGP Signature:
>
> https://dist.apache.org/repos/dist/dev/atlas/2.0.0-rc1/apache-atlas-2.0.0-sources.tar.gz.asc
> SHA512 Hash:
>
> https://dist.apache.org/repos/dist/dev/atlas/2.0.0-rc1/apache-atlas-2.0.0-sources.tar.gz.sha512
> MD5 Hash:
>
> https://dist.apache.org/repos/dist/dev/atlas/2.0.0-rc1/apache-atlas-2.0.0-sources.tar.gz.md5
>
> Keys to verify the signature of the release artifacts are available at:
> https://dist.apache.org/repos/dist/release/atlas/KEYS
>
>
>
>

-- 
Regards
Nikhil P. Bonte

Re: [VOTE] Release Apache Atlas version 2.0.0 - rc2

[Nikhil Bonte]

+1 for Apache Atlas 2.0.0 Release Candidate #2

- Built successfully from the source tar file
- Verified PGP signature
- Ran quick start and validated basic functionality.
- Verified following features:
- Atlas Index Repair tool for JanusGraph
- Update stats/metrics module to collect notification metrics
- Add notifications from RelationshipStore to ATLAS_ENTITITES kafka topic

Thanks Sarath for the release candidate.

Regards
Nikhil P. Bonte

On Tue, May 7, 2019 at 3:08 AM Sarath Subramanian  wrote:

> Atlas team,
>
> Apache Atlas 2.0.0 Release Candidate #2 is now available for a vote within
> dev community.
>
> Following commits went into branch-2.0 since the last release candidate
> (rc1):
>  - ATLAS-3190: UI - Allow user to refresh metric data inside popup
>  - ATLAS-3155: UI - IE-11 - Name entry not clickable (#2)
>  - ATLAS-3188: Regression: Simple insert queries are being captured in
> ATLAS_HOOK topic
>  - ATLAS-3033: Skip hive temp table while getting Table object details from
> hiveContext.
>  - ATLAS-3187: columns attribute is empty for deleted hive_table entity
>
> Links to the release artifacts are given below. Please review and vote.
>
> The vote will be open for at least 72 hours or until necessary votes are
> reached.
>   [ ] +1 approve
>   [ ] +0 no opinion
>   [ ] -1 disapprove (and reason why)
>
>
> Thanks,
> Sarath
>
>
> List of issues addressed in this release:
>
> https://issues.apache.org/jira/issues/?jql=project%20%3D%20ATLAS%20AND%20status%20%3D%20Resolved%20AND%20fixVersion%20%3D%202.0.0%20ORDER%20BY%20updated%20DESC%2C%20priority%20DESC
>
> Git tag for the release:
> https://github.com/apache/atlas/tree/release-2.0.0-rc2
> Sources for the release:
>
> https://dist.apache.org/repos/dist/dev/atlas/2.0.0-rc2/apache-atlas-2.0.0-sources.tar.gz
>
> Source release verification:
> PGP Signature:
>
> https://dist.apache.org/repos/dist/dev/atlas/2.0.0-rc2/apache-atlas-2.0.0-sources.tar.gz.asc
> SHA512
> 
> Hash:
>
> https://dist.apache.org/repos/dist/dev/atlas/2.0.0-rc2/apache-atlas-2.0.0-sources.tar.gz.sha512
> MD5
> 
> Hash:
>
> https://dist.apache.org/repos/dist/dev/atlas/2.0.0-rc2/apache-atlas-2.0.0-sources.tar.gz.md5
>
> Keys to verify the signature of the release artifacts are available at:
> https://dist.apache.org/repos/dist/release/atlas/KEYS
>


-- 
Regards
Nikhil P. Bonte

Re: Review Request 70599: Import Service: Improvement to Handle Importing to Existing Entity

[Nikhil Bonte]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/70599/#review215122
---


Ship it!




Ship It!

- Nikhil Bonte


On May 8, 2019, 1:42 a.m., Ashutosh Mestry wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/70599/
> ---
> 
> (Updated May 8, 2019, 1:42 a.m.)
> 
> 
> Review request for atlas, Madhan Neethiraj, Nikhil Bonte, and Sarath 
> Subramanian.
> 
> 
> Bugs: ATLAS-3193
> https://issues.apache.org/jira/browse/ATLAS-3193
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> **Background**
> Please look at bug description.
> 
> **Approach**
> _ExportService_: Refactor to extract starting entity logic in a separate 
> reusable class.
> (New) _StartEntityFetchByExportRequest_: Extracted logic for fetching start 
> entity.
> _BulkImporterImpl_: Handle _AtlasSchemaViolation_ to update guid of existing 
> entity. Uses start entity logic (from _StartEntityFetchByExportRequest_ to 
> check and fetch existing entities).
> _GraphBackedSearchIndexer_: Additional property to index.
> 
> **CURL**
> 
> Import:
> ```
> curl -X POST -u admin:passW0rd -H "Content-Type: multipart/form-data" -H 
> "Cache-Control: no-cache" -F data=@./stocks.zip 
> http://localhost:21000/api/atlas/admin/import
> ```
> 
> 
> Diffs
> -
> 
>   common/src/main/java/org/apache/atlas/repository/Constants.java 5055a0426 
>   
> repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedSearchIndexer.java
>  1cfa202a9 
>   
> repository/src/main/java/org/apache/atlas/repository/impexp/ExportService.java
>  62298f9f1 
>   
> repository/src/main/java/org/apache/atlas/repository/impexp/ImportService.java
>  a52c3f986 
>   
> repository/src/main/java/org/apache/atlas/repository/impexp/StartEntityFetchByExportRequest.java
>  PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/BulkImporterImpl.java
>  dc8bd19df 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphRetriever.java
>  14f991697 
>   
> repository/src/test/java/org/apache/atlas/repository/impexp/ImportServiceTest.java
>  a1d6cef5c 
>   
> repository/src/test/java/org/apache/atlas/repository/impexp/StartEntityFetchByExportRequestTest.java
>  PRE-CREATION 
>   
> repository/src/test/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityStoreV2BulkImportPercentTest.java
>  e432d4e42 
> 
> 
> Diff: https://reviews.apache.org/r/70599/diff/4/
> 
> 
> Testing
> ---
> 
> **Unit tests**
> New tests added.
> 
> **Functional tests**
> Existing export and import flow via CURL calls.
> 
> **Pre-commit**
> https://builds.apache.org/view/A/view/Atlas/job/PreCommit-ATLAS-Build-Test/1081/
> 
> 
> Thanks,
> 
> Ashutosh Mestry
> 
>

Re: Atlas 1.2.0 release heads up

[Nikhil Bonte]

+1 for the 1.2.0 release.

On Wed, May 15, 2019 at 12:28 PM Umesh Padashetty
 wrote:

> +1 for 1.2.0 release. Thanks Nixon.
>
> On Wed, May 15, 2019 at 9:22 AM Keval Bhatt 
> wrote:
>
> > +1 for the 1.2.0 release
> >
> > On Wed, May 15, 2019, 01:52 Sarath Subramanian 
> wrote:
> >
> > > +1 for 1.2.0 release. Thanks Nixon for volunteering.
> > >
> > > On Tue, May 14, 2019 at 8:11 AM Madhan Neethiraj 
> > > wrote:
> > >
> > > > +1 for 1.2.0 release.
> > > >
> > > > Nixon - thanks for initiating 1.2.0 release and volunteering to be
> the
> > > > release manager.
> > > >
> > > > Here is the URL for list of resolved JIRAs:
> > > >
> > >
> >
> https://issues.apache.org/jira/issues/?jql=project%20%3D%20ATLAS%20AND%20resolution%20%3D%20Fixed%20AND%20fixVersion%20%3D%201.2.0%20ORDER%20BY%20key%20DESC
> > > >
> > > > Thanks,
> > > > Madhan
> > > >
> > > >
> > > > On 5/14/19, 7:03 AM, "Nixon Rodrigues" <
> > > > nixon.rodrig...@freestoneinfotech.com> wrote:
> > > >
> > > > Atlas dev community,
> > > >
> > > > Since Atlas 1.1.0 was released, there are around 210 plus commits
> > > > <
> > > >
> https://github.com/apache/atlas/compare/release-1.1.0-rc2...branch-1.0
> > >
> > > on
> > > > branch-1.0 and 158 resolved issues
> > > > <
> > > >
> > >
> >
> https://issues.apache.org/jira/browse/ATLAS-1070?jql=project%20%3D%20ATLAS%20AND%20resolution%20%3D%20Fixed%20AND%20fixVersion%20%3D%201.2.0%20ORDER%20BY%20key%20ASC
> > > > >
> > > > with fix version 1.2.0 after 1.1.0 release.
> > > > These bug fixes and improvements are important for stability and
> > for
> > > > other
> > > > dependent components that consume Atlas.
> > > >
> > > > I propose that we should do a 1.2.0 release from branch-1.0 and
> the
> > > > tentative release completion date would be 31st May 2019.
> > > >
> > > > I volunteer to be the release manager for Apache Atlas 1.2.0
> > release.
> > > >
> > > > Let me know your thoughts and concerns if any.
> > > >
> > > > Regards
> > > > Nixon Rodrigues
> > > >
> > > >
> > > >
> > > >
> > >
> >
>
>
> --
> Thanks,
> Umesh
>


-- 
Regards
Nikhil P. Bonte

Re: Review Request 70768: ATLAS-3256 Modify export API to process with relationshipAttributes

[Nikhil Bonte]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/70768/
---

(Updated May 31, 2019, 12:21 p.m.)


Review request for atlas, Ashutosh Mestry, madhan, Nixon Rodrigues, and Sarath 
Subramanian.


Bugs: ATLAS-3256
https://issues.apache.org/jira/browse/ATLAS-3256


Repository: atlas


Description
---

Please refer jira for description


Diffs
-

  
repository/src/main/java/org/apache/atlas/repository/impexp/EntitiesExtractor.java
 PRE-CREATION 
  
repository/src/main/java/org/apache/atlas/repository/impexp/ExportService.java 
11289ea 
  
repository/src/main/java/org/apache/atlas/repository/impexp/ExtractStrategy.java
 PRE-CREATION 
  
repository/src/main/java/org/apache/atlas/repository/impexp/IncrementalExportEntityProvider.java
 3a2a917 
  
repository/src/main/java/org/apache/atlas/repository/impexp/RelationshipAttributesExtractor.java
 PRE-CREATION 
  
repository/src/main/java/org/apache/atlas/repository/impexp/VertexExtractor.java
 PRE-CREATION 
  
repository/src/test/java/org/apache/atlas/repository/impexp/IncrementalExportEntityProviderTest.java
 85ed5f9 
  
repository/src/test/java/org/apache/atlas/repository/impexp/RelationshipAttributesExtractorTest.java
 PRE-CREATION 
  repository/src/test/resources/hive_db_lineage.zip PRE-CREATION 


Diff: https://reviews.apache.org/r/70768/diff/2/


Testing
---

Added UTs
Test cases are passing


Thanks,

Nikhil Bonte

Re: [VOTE] Release Apache Atlas version 1.2.0 - rc3

[Nikhil Bonte]

+1 Release Apache Atlas 1.2.0 release candidate #3

- Verified PGP signature.
- Built successfully from the source tar file
- Ran quick start and validated basic functionality.
- Performed few basic search.

Regards,
Nikhil P. Bonte

On Tue, Jun 4, 2019 at 12:19 PM Sarath Subramanian
 wrote:

> +1 for Apache Atlas 1.2.0-rc3
>
> - Validated Checksum and signature of binaries
> - No binaries found in source.
> - Build using embedded-hbase-solr profile and build was successful.
> - Able to run quickstart and view entities and lineage in atlas
> - Validated basic, advanced search, saved-search, classification
> propagation and metrics UI
>
>
> Thanks,
> Sarath
>
>
> On Mon, Jun 3, 2019 at 1:11 AM Nixon Rodrigues <
> nixon.rodrig...@freestoneinfotech.com> wrote:
>
> > Atlas team,
> >
> > Apache Atlas 1.2.0 release candidate #3 is now available for a vote
> within
> > dev community.
> >
> > Following commits went into branch-1.0 since the last release candidate
> > (rc2):
> >
> > ATLAS-3180:- New Bulk retrieval API with servicetype parameter.
> > ATLAS-3227:- Bulk delete does not delete the indexes properly.
> >
> > Links to the release artifacts are given below. Please review and vote.
> >
> > The vote will be open for at least 72 hours or until necessary votes are
> > reached.
> >   [ ] +1 approve
> >   [ ] +0 no opinion
> >   [ ] -1 disapprove (and reason why)
> >
> >
> > Thanks,
> > Nixon Rodrigues
> >
> >
> > List of issues addressed in this release:
> >
> >
> https://issues.apache.org/jira/browse/ATLAS-3201?jql=project%20%3D%20ATLAS%20AND%20resolution%20%3D%20Fixed%20AND%20fixVersion%20%3D%201.2.0%20ORDER%20BY%20key%20DESC
> >
> > Git tag for the release:
> > https://github.com/apache/atlas/releases/tag/release-1.2.0-rc3
> >
> > Sources for the release:
> >
> >
> https://dist.apache.org/repos/dist/dev/atlas/1.2.0-rc3/apache-atlas-1.2.0-sources.tar.gz
> >
> > Source release verification:
> >   PGP Signature:
> >
> >
> https://dist.apache.org/repos/dist/dev/atlas/1.2.0-rc3/apache-atlas-1.2.0-sources.tar.gz.asc
> >   MD5 Hash:
> >
> >
> https://dist.apache.org/repos/dist/dev/atlas/1.2.0-rc3/apache-atlas-1.2.0-sources.tar.gz.md5
> >   SHA512 Hash:
> >
> >
> https://dist.apache.org/repos/dist/dev/atlas/1.2.0-rc3/apache-atlas-1.2.0-sources.tar.gz.sha512
> >
> > Keys to verify the signature of the release artifacts are available at:
> > https://dist.apache.org/repos/dist/dev/atlas/KEYS
> >
>


-- 
Regards
Nikhil P. Bonte

Re: Review Request 70768: ATLAS-3256 Modify export API to process with relationshipAttributes

[Nikhil Bonte]

> On June 3, 2019, 4:54 a.m., Ashutosh Mestry wrote:
> > repository/src/main/java/org/apache/atlas/repository/impexp/RelationshipAttributesExtractor.java
> > Lines 64 (patched)
> > <https://reviews.apache.org/r/70768/diff/2/?file=2147440#file2147440line64>
> >
> > Won't this do fullFetch?

seperated logic to avoid confusion


- Nikhil


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/70768/#review215634
-------


On May 31, 2019, 12:21 p.m., Nikhil Bonte wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/70768/
> ---
> 
> (Updated May 31, 2019, 12:21 p.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, madhan, Nixon Rodrigues, and 
> Sarath Subramanian.
> 
> 
> Bugs: ATLAS-3256
> https://issues.apache.org/jira/browse/ATLAS-3256
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Please refer jira for description
> 
> Will add Performance improvement numbers soon.
> 
> 
> Diffs
> -
> 
>   
> repository/src/main/java/org/apache/atlas/repository/impexp/EntitiesExtractor.java
>  PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/repository/impexp/ExportService.java
>  11289ea 
>   
> repository/src/main/java/org/apache/atlas/repository/impexp/ExtractStrategy.java
>  PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/repository/impexp/IncrementalExportEntityProvider.java
>  3a2a917 
>   
> repository/src/main/java/org/apache/atlas/repository/impexp/RelationshipAttributesExtractor.java
>  PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/repository/impexp/VertexExtractor.java
>  PRE-CREATION 
>   
> repository/src/test/java/org/apache/atlas/repository/impexp/IncrementalExportEntityProviderTest.java
>  85ed5f9 
>   
> repository/src/test/java/org/apache/atlas/repository/impexp/RelationshipAttributesExtractorTest.java
>  PRE-CREATION 
>   repository/src/test/resources/hive_db_lineage.zip PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/70768/diff/3/
> 
> 
> Testing
> ---
> 
> Added UTs
> Test cases are passing
> 
> Manual testing approach:
> 1. exported data > data1.zip
> 2. cleaned hbase & solr
> 3. imported data1.zip & verified imported data
> 4. exported again > data2.zip
> 5. compared contents of data1.zip & data2.zip
> 
> 
> Thanks,
> 
> Nikhil Bonte
> 
>

Re: Review Request 70768: ATLAS-3256 Modify export API to process with relationshipAttributes

[Nikhil Bonte]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/70768/
---

(Updated June 14, 2019, 9:21 a.m.)


Review request for atlas, Ashutosh Mestry, madhan, Nixon Rodrigues, and Sarath 
Subramanian.


Changes
---

Added Perfomance improvement numbers


Bugs: ATLAS-3256
https://issues.apache.org/jira/browse/ATLAS-3256


Repository: atlas


Description (updated)
---

Please refer jira for description

Perfomance improvement numbers with new extract strategy:

##Baseline

Full export: #1

Number of entities: 12342
Duration:   93 secs


Full export: #2

Number of entities: 25703
Duration:   245 secs


##Optimization:

Full export: #1

Number of entities: 12342
Duration:   82 secs
Improvement:12%


Full export: #2

Number of entities: 25703
Duration:   185 secs
Improvement:24%


Diffs
-

  
repository/src/main/java/org/apache/atlas/repository/impexp/EntitiesExtractor.java
 PRE-CREATION 
  
repository/src/main/java/org/apache/atlas/repository/impexp/ExportService.java 
11289ea 
  
repository/src/main/java/org/apache/atlas/repository/impexp/ExtractStrategy.java
 PRE-CREATION 
  
repository/src/main/java/org/apache/atlas/repository/impexp/IncrementalExportEntityProvider.java
 3a2a917 
  
repository/src/main/java/org/apache/atlas/repository/impexp/RelationshipAttributesExtractor.java
 PRE-CREATION 
  
repository/src/main/java/org/apache/atlas/repository/impexp/VertexExtractor.java
 PRE-CREATION 
  
repository/src/test/java/org/apache/atlas/repository/impexp/IncrementalExportEntityProviderTest.java
 85ed5f9 
  
repository/src/test/java/org/apache/atlas/repository/impexp/RelationshipAttributesExtractorTest.java
 PRE-CREATION 
  repository/src/test/resources/hive_db_lineage.zip PRE-CREATION 


Diff: https://reviews.apache.org/r/70768/diff/3/


Testing
---

Added UTs
Test cases are passing

Manual testing approach:
1. exported data > data1.zip
2. cleaned hbase & solr
3. imported data1.zip & verified imported data
4. exported again > data2.zip
5. compared contents of data1.zip & data2.zip


Thanks,

Nikhil Bonte

Re: [VOTE] Release Apache Atlas version 0.8.4 - rc1

[Nikhil Bonte]

Keval,
Thanks for the 0.8.4 release
+1 for the release candidate #1

Verified the following:
- Verified PGP signature.
- Built successfully from the source tar file
- Ran quick start and validated basic functionality.
- Performed few basic search.

Regards,
Nikhil P. Bonte

On Fri, Jun 14, 2019 at 10:53 AM Binit Gutka <
binit.gu...@freestoneinfotech.com> wrote:

> Keval,
> Thanks for the 0.8.4 release
> +1 for the release candidate #1
> Verified the following:
>
> -Verified PGP signatures.
> -Built from the source file.
> -Ran quick start and validated basic functionality.
> -Performed basic search
> -Created entities.
> -Lineage export functionality tested.
> -Lineage Filter, Lineage Search, Lineage reset functionality tested.
>
>
> --
> *Thanks,*
> *Binit.*
>
> On Thu, Jun 13, 2019 at 4:40 PM Keval Bhatt  wrote:
>
> > Atlas team,
> >
> > Apache Atlas 0.8.4 release candidate #1 is now available for a vote
> within
> > dev community.
> >
> > Following commits went into branch-0.8 since the last release candidate
> > (rc0):
> >
> > ATLAS-3231: Lineage Icon updated.
> >
> > Links to the release artifacts are given below. Please review and vote.
> >
> > The vote will be open for at least 72 hours or until necessary votes are
> > reached.
> >   [ ] +1 approve
> >   [ ] +0 no opinion
> >   [ ] -1 disapprove (and reason why)
> >
> >
> > Thanks,
> > Keval Bhatt
> >
> >
> > List of issues addressed in this release:
> >
> >
> https://issues.apache.org/jira/issues/?jql=project%3DATLAS%20AND%20resolution%3DFixed%20AND%20fixVersion%3D0.8.4%20ORDER%20BY%20key%20DESC
> >
> > Git tag for the release:
> > https://github.com/apache/atlas/tree/release-0.8.4-rc1
> >
> > Sources for the release:
> >
> >
> https://dist.apache.org/repos/dist/dev/atlas/0.8.4-rc1/apache-atlas-0.8.4-sources.tar.gz
> >
> > Source release verification:
> >   PGP Signature:
> >
> >
> https://dist.apache.org/repos/dist/dev/atlas/0.8.4-rc1/apache-atlas-0.8.4-sources.tar.gz.asc
> >
> >   MD5 Hash:
> >
> >
> https://dist.apache.org/repos/dist/dev/atlas/0.8.4-rc1/apache-atlas-0.8.4-sources.tar.gz.md5
> >
> >   SHA512 Hash:
> >
> >
> https://dist.apache.org/repos/dist/dev/atlas/0.8.4-rc1/apache-atlas-0.8.4-sources.tar.gz.sha512
> >
> >
> > Keys to verify the signature of the release artifacts are available at:
> > https://dist.apache.org/repos/dist/dev/atlas/KEYS
> >
>
>
> --
> *Thanks,*
> *Binit.*
>


-- 
Regards
Nikhil P. Bonte

Re: Review Request 71025: Import Service: Support Concurrent Ingest

[Nikhil Bonte]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71025/#review216479
---


Ship it!




Ship It!

- Nikhil Bonte


On July 9, 2019, 4:42 p.m., Ashutosh Mestry wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/71025/
> ---
> 
> (Updated July 9, 2019, 4:42 p.m.)
> 
> 
> Review request for atlas, Madhan Neethiraj, Nikhil Bonte, Nixon Rodrigues, 
> and Sarath Subramanian.
> 
> 
> Bugs: ATLAS-3320
> https://issues.apache.org/jira/browse/ATLAS-3320
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> **Approach**
> - Use existing producer-consumer (PC) framework.
> - Modify _BulkImporterImpl_ to implement _WorkItemConsumer_.
> - Add support for configuring number of workers and batch size within 
> _AtlasImportRequest_.
> 
> _AtlasImportRequest_
> ```
> {
> "options": {
> "numWorkers": 8,
> "batchSize": 25
> }
> }
> ```
> 
> **CURL**
> ```
> curl -v -X POST -u admin:admin -H "Content-Type: multipart/form-data" -H 
> "Cache-Control: no-cache" -F request=@./import-options.json -F 
> data=@./Default-3-pre.zip http://localhost:21000/api/atlas/admin/import
> ```
> 
> 
> Diffs
> -
> 
>   
> graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraph.java
>  499e8d1af 
>   intg/src/main/java/org/apache/atlas/model/impexp/AtlasImportRequest.java 
> 0b3ede93f 
>   intg/src/main/resources/atlas-log4j.xml 4f74c2abb 
>   
> repository/src/main/java/org/apache/atlas/repository/impexp/ImportService.java
>  3ded79842 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/BulkImporterImpl.java
>  2f330c093 
> 
> 
> Diff: https://reviews.apache.org/r/71025/diff/1/
> 
> 
> Testing
> ---
> 
> **Unit tests**
> Existing tests.
> 
> **Functional tests**
> - Verified import for pre-1.0 and post-1.0 exported ZIP files.
> 
> **Volume tests**
> - Measure performance with large data.
> 
> +--+--+--++
> | File | Before   | After| Configuration  |
> +--+--+--++
> | smalldb  |   6 min  |3 min | Shards: 4, Threads: 8  |
> | (2.2 MB) |  |  ||
> +--+--+--++
> | largedb  |3 hrs |  20 mins | Shards: 4, Threads: 16 |
> | (40 MB)  |  |  ||
> +--+--+--++
> 
> 
> Thanks,
> 
> Ashutosh Mestry
> 
>

Re: Review Request 71046: ATLAS-3324 Export API - incremental export with hive_table

[Nikhil Bonte]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71046/
---

(Updated July 10, 2019, 2:14 p.m.)


Review request for atlas, Ashutosh Mestry, madhan, Nixon Rodrigues, and Sarath 
Subramanian.


Bugs: ATLAS-3324
https://issues.apache.org/jira/browse/ATLAS-3324


Repository: atlas


Description
---

Right now Incremental export is supported if it is for hive_db.
This patch will enable Incremental export with hive_table.
This will export hive_table & its connected entities provided qualifies 
changeMarker.


Diffs
-

  
repository/src/main/java/org/apache/atlas/repository/impexp/EntitiesExtractor.java
 15cb111 
  
repository/src/main/java/org/apache/atlas/repository/impexp/ExportService.java 
5055607 
  
repository/src/main/java/org/apache/atlas/repository/impexp/IncrementalExportEntityProvider.java
 256d9de 
  
repository/src/test/java/org/apache/atlas/repository/impexp/ExportIncrementalTest.java
 1d44081 


Diff: https://reviews.apache.org/r/71046/diff/2/


Testing
---

Added unit test.
Passing unit tests.
Verified Manually.
e.g. : 
{
   "itemsToExport":[
  {
 "typeName":"hive_table",
 "uniqueAttributes":{
"qualifiedName":"largedb.testtable_7@cluster0"
 }
  }

   ],
   "options":{
 "fetchType" : "incremental", 
 "changeMarker" : 1562741110005,
 "skipLineage": true
   }
}


Thanks,

Nikhil Bonte

Re: Review Request 71046: ATLAS-3324 Export API - incremental export with hive_table

[Nikhil Bonte]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71046/
---

(Updated July 16, 2019, 9:40 a.m.)


Review request for atlas, Ashutosh Mestry, madhan, Nixon Rodrigues, and Sarath 
Subramanian.


Changes
---

Addressed comment.


Bugs: ATLAS-3324
https://issues.apache.org/jira/browse/ATLAS-3324


Repository: atlas


Description
---

Right now Incremental export is supported if it is for hive_db.
This patch will enable Incremental export with hive_table.
This will export hive_table & its connected entities provided qualifies 
changeMarker.


Diffs (updated)
-

  
repository/src/main/java/org/apache/atlas/repository/impexp/EntitiesExtractor.java
 15cb111 
  
repository/src/main/java/org/apache/atlas/repository/impexp/ExportService.java 
5055607 
  
repository/src/main/java/org/apache/atlas/repository/impexp/IncrementalExportEntityProvider.java
 256d9de 
  
repository/src/test/java/org/apache/atlas/repository/impexp/ExportIncrementalTest.java
 1d44081 


Diff: https://reviews.apache.org/r/71046/diff/4/

Changes: https://reviews.apache.org/r/71046/diff/3-4/


Testing
---

Added unit test.
Passing unit tests.
Verified Manually.
e.g. : 
{
   "itemsToExport":[
  {
 "typeName":"hive_table",
 "uniqueAttributes":{
"qualifiedName":"largedb.testtable_7@cluster0"
 }
  }

   ],
   "options":{
 "fetchType" : "incremental", 
 "changeMarker" : 1562741110005,
 "skipLineage": true
   }
}


Thanks,

Nikhil Bonte

Review Request 71238: ATLAS-3358 - NPE when Atlas is trying to read from graphDB

[Nikhil Bonte]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71238/
---

Review request for atlas, Ashutosh Mestry, Madhan Neethiraj, Nixon Rodrigues, 
and Sarath Subramanian.


Bugs: ATLAS-3358
https://issues.apache.org/jira/browse/ATLAS-3358


Repository: atlas


Description
---

Reading vertex which has "__type.category" as null & Atlas fails to startup.
For detailed log please refer jira.


Diffs
-

  
repository/src/main/java/org/apache/atlas/repository/typestore/GraphBackedTypeStore.java
 1525d7b8e 


Diff: https://reviews.apache.org/r/71238/diff/1/


Testing
---


Thanks,

Nikhil Bonte

Re: Review Request 71270: Table-level Export & Import: Preserving REPL Additional Info

[Nikhil Bonte]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71270/#review217169
---


Ship it!




Ship It!

- Nikhil Bonte


On Aug. 11, 2019, 4:56 a.m., Ashutosh Mestry wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/71270/
> ---
> 
> (Updated Aug. 11, 2019, 4:56 a.m.)
> 
> 
> Review request for atlas, Madhan Neethiraj, Nikhil Bonte, Nixon Rodrigues, 
> and Sarath Subramanian.
> 
> 
> Bugs: ATLAS-3362
> https://issues.apache.org/jira/browse/ATLAS-3362
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> **Background**
> Existing logic for preserving REPL _additionalInfo_ uses the first guid in 
> the _exportOrder_ list. This works fine for database-level replication as DB 
> GUID is always available in the list.
> 
> In case of table-level replication the list of tables vary based on what 
> _changeMarker_ is applied. 
> 
> Thus, there needs to be a constant GUID against which this REPL additional 
> info can be preserved.
> 
> **Approach**
> - Get the first guid from the in the _exportOrder_. 
> - Fetch the entity using the guid.
> - If it is of type hive_db, use it.
> - If it is hive_table or hive_column, fetch the DB by constructing DB's 
> _qualifiedName_ from the entity's qualified name.
> 
> 
> Diffs
> -
> 
>   
> repository/src/main/java/org/apache/atlas/repository/impexp/AuditsWriter.java 
> 9bf30f116 
>   
> repository/src/test/java/org/apache/atlas/repository/impexp/ReplicationEntityAttributeTest.java
>  829390b76 
> 
> 
> Diff: https://reviews.apache.org/r/71270/diff/1/
> 
> 
> Testing
> ---
> 
> **Functional tests**
> Executed scenarios of table-level replication.
> 
> **Unit tests**
> Additional test.
> 
> 
> Thanks,
> 
> Ashutosh Mestry
> 
>

Re: Review Request 71406: ATLAS-3387 Consider X-FORWARDED-FOR header for getting end user IP address when connected with proxy

[Nikhil Bonte]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71406/
---

(Updated Aug. 29, 2019, 1:31 p.m.)


Review request for atlas, madhan, Nixon Rodrigues, and Sarath Subramanian.


Bugs: ATLAS-3387
https://issues.apache.org/jira/browse/ATLAS-3387


Repository: atlas


Description
---

Consider X-FORWARDED-FOR header for getting end user IP address when connected 
with proxy


Diffs
-

  
authorization/src/main/java/org/apache/atlas/authorize/AtlasAccessRequest.java 
b031f4c3e 
  
authorization/src/main/java/org/apache/atlas/authorize/AtlasAuthorizationUtils.java
 ac2f52525 
  server-api/src/main/java/org/apache/atlas/RequestContext.java d518609d3 
  webapp/src/main/java/org/apache/atlas/web/filters/AuditFilter.java 54f7330da 


Diff: https://reviews.apache.org/r/71406/diff/5/


Testing
---

Tested with simple authorizer. Need to test after ranger end changes.


Thanks,

Nikhil Bonte

Re: Review Request 71409: ATLAS-3395 Show overlay with description when Logout in case of trustedProxy/KnoxSSO

[Nikhil Bonte]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71409/
---

(Updated Aug. 29, 2019, 2:26 p.m.)


Review request for atlas, madhan, Nixon Rodrigues, and Sarath Subramanian.


Bugs: ATLAS-3395
https://issues.apache.org/jira/browse/ATLAS-3395


Repository: atlas


Description
---

When user logins in Atlas via TrustedProxy/KnoxSSO authentication method, the 
Atlas logout event does not functions as intended. Showing proper message on 
logout click will not cause any ambiguity to enduser.


Diffs
-

  dashboardv2/public/js/main.js 7b33da74d 
  dashboardv2/public/js/utils/Utils.js 5274ef34c 
  dashboardv2/public/js/views/site/Header.js b8ce4a8ad 
  
webapp/src/main/java/org/apache/atlas/web/filters/AtlasAuthenticationFilter.java
 f56aeb464 
  
webapp/src/main/java/org/apache/atlas/web/filters/AtlasKnoxSSOAuthenticationFilter.java
 8bac8c6f9 
  webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java 
e78fcb61a 
  
webapp/src/main/java/org/apache/atlas/web/security/AtlasAuthenticationProvider.java
 dff3d8d31 
  
webapp/src/main/java/org/apache/atlas/web/security/AtlasWebAuthenticationDetails.java
 PRE-CREATION 


Diff: https://reviews.apache.org/r/71409/diff/1/


Testing (updated)
---

Tested on simple environment. 
Testing on Trusted Proxy enabled cluster is pending.


Thanks,

Nikhil Bonte

Re: Review Request 71406: ATLAS-3387 Consider X-FORWARDED-FOR header for getting end user IP address when connected with proxy

[Nikhil Bonte]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71406/
---

(Updated Sept. 5, 2019, 10:23 a.m.)


Review request for atlas, madhan, Nixon Rodrigues, and Sarath Subramanian.


Changes
---

Addressed open issue.


Bugs: ATLAS-3387
https://issues.apache.org/jira/browse/ATLAS-3387


Repository: atlas


Description
---

Consider X-FORWARDED-FOR header for getting end user IP address when connected 
with proxy


Diffs (updated)
-

  
authorization/src/main/java/org/apache/atlas/authorize/AtlasAccessRequest.java 
b031f4c3e 
  
authorization/src/main/java/org/apache/atlas/authorize/AtlasAuthorizationUtils.java
 ac2f52525 
  server-api/src/main/java/org/apache/atlas/RequestContext.java d518609d3 
  webapp/src/main/java/org/apache/atlas/web/filters/AuditFilter.java 54f7330da 


Diff: https://reviews.apache.org/r/71406/diff/6/

Changes: https://reviews.apache.org/r/71406/diff/5-6/


Testing
---

Tested with simple authorizer. Need to test after ranger end changes.


Thanks,

Nikhil Bonte

Re: Review Request 71426: Import Service: Support for Backing Directory for Large Imports

[Nikhil Bonte]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71426/#review217604
---




docs/src/site/twiki/Import-API-Options.twiki
Lines 142 (patched)
<https://reviews.apache.org/r/71426/#comment304874>

Suggestion: Adding this property in "atlas-application.properties" template 
file as commented line can be helpful.


- Nikhil Bonte


On Sept. 5, 2019, 1:14 a.m., Ashutosh Mestry wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/71426/
> ---
> 
> (Updated Sept. 5, 2019, 1:14 a.m.)
> 
> 
> Review request for atlas, Madhan Neethiraj, Nikhil Bonte, Nixon Rodrigues, 
> and Sarath Subramanian.
> 
> 
> Bugs: ATLAS-3396
> https://issues.apache.org/jira/browse/ATLAS-3396
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> **Background**
> The approach adds another option to _ImportService_ to be able to optionally 
> use a directory on the server for storing data to be imported.
> 
> **Approach**
> This is a backward compatible.
> 
> - - New: _ZipSourceWithBackingDirectory_: Uses a temporary directory for 
> storing contents of the zip file.
> - Modified: _ImportService_ Now takes _InputStream_ as parameter instead of 
> _ZipSource_. 
> - Modified: _ImportRequest_ Additionl option takes _backingDirectory_ as 
> additional option. Specifying this will use the new 
> _ZipSourceWithBackingDirectory_ during import.
>  
> 
> **CURL**
> 
> Use pre-configured temporary directory, set _atlas.import.temporaryDirectory_ 
> in the applicatino properties.
> 
> _CURL_
> ```
> curl -v -X POST -u admin:admin -H "Content-Type: multipart/form-data" -H 
> "Cache-Control: no-cache" -F request=@../docs/import-options.json -F 
> data=@../docs/smalldb.zip http://localhost:21000/api/atlas/admin/import
> ```
> 
> **Documentation**
> Updated.
> 
> 
> Diffs
> -
> 
>   docs/src/site/twiki/Import-API-Options.twiki 4004e7013 
>   intg/src/main/java/org/apache/atlas/model/impexp/AtlasImportRequest.java 
> 6a9985642 
>   
> repository/src/main/java/org/apache/atlas/repository/impexp/ImportService.java
>  b5d8b7c39 
>   
> repository/src/main/java/org/apache/atlas/repository/impexp/ZipSourceWithBackingDirectory.java
>  PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v1/AtlasEntityStreamForImport.java
>  90ae15d1e 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v1/EntityImportStream.java
>  d4b6c5505 
>   
> repository/src/test/java/org/apache/atlas/repository/impexp/ExportIncrementalTest.java
>  a355297a8 
>   
> repository/src/test/java/org/apache/atlas/repository/impexp/ExportSkipLineageTest.java
>  eaf4602d5 
>   
> repository/src/test/java/org/apache/atlas/repository/impexp/ImportServiceTest.java
>  693a163f8 
>   
> repository/src/test/java/org/apache/atlas/repository/impexp/ReplicationEntityAttributeTest.java
>  868b732d3 
>   
> repository/src/test/java/org/apache/atlas/repository/impexp/ZipFileResourceTestUtils.java
>  a2a5f58dc 
>   webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java 
> 8417e7eb0 
> 
> 
> Diff: https://reviews.apache.org/r/71426/diff/4/
> 
> 
> Testing
> ---
> 
> **Unit tests**
> Updated to exercise the new implementation.
> 
> **Volume tests**
> Imported 4GB ZIP file that took 26 hours. Memory and CPU stayed in normal 
> range during this operation.
> 
> **Functional tests**
> Accuracy tests preformed on small size data.
> 
> **Pre-commit Build**
> https://builds.apache.org/view/A/view/Atlas/job/PreCommit-ATLAS-Build-Test/1380/
> 
> 
> Thanks,
> 
> Ashutosh Mestry
> 
>

Re: Review Request 71406: ATLAS-3387 Consider X-FORWARDED-FOR header for getting end user IP address when connected with proxy

[Nikhil Bonte]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71406/
---

(Updated Sept. 6, 2019, 10:10 a.m.)


Review request for atlas, madhan, Nixon Rodrigues, and Sarath Subramanian.


Changes
---

Addressed open issue.


Bugs: ATLAS-3387
https://issues.apache.org/jira/browse/ATLAS-3387


Repository: atlas


Description
---

Consider X-FORWARDED-FOR header for getting end user IP address when connected 
with proxy


Diffs (updated)
-

  
authorization/src/main/java/org/apache/atlas/authorize/AtlasAccessRequest.java 
b031f4c3e 
  
authorization/src/main/java/org/apache/atlas/authorize/AtlasAdminAccessRequest.java
 1782b3272 
  
authorization/src/main/java/org/apache/atlas/authorize/AtlasAuthorizationUtils.java
 ac2f52525 
  
authorization/src/main/java/org/apache/atlas/authorize/AtlasEntityAccessRequest.java
 07ff678ed 
  
authorization/src/main/java/org/apache/atlas/authorize/AtlasRelationshipAccessRequest.java
 d2da03c1a 
  
authorization/src/main/java/org/apache/atlas/authorize/AtlasSearchResultScrubRequest.java
 c908b28a7 
  
authorization/src/main/java/org/apache/atlas/authorize/AtlasTypeAccessRequest.java
 af38425fb 
  server-api/src/main/java/org/apache/atlas/RequestContext.java d518609d3 
  webapp/src/main/java/org/apache/atlas/web/filters/AuditFilter.java 54f7330da 


Diff: https://reviews.apache.org/r/71406/diff/7/

Changes: https://reviews.apache.org/r/71406/diff/6-7/


Testing
---

Tested with simple authorizer. Need to test after ranger end changes.


Thanks,

Nikhil Bonte

Re: Review Request 71406: ATLAS-3387 Consider X-FORWARDED-FOR header for getting end user IP address when connected with proxy

[Nikhil Bonte]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71406/
---

(Updated Sept. 9, 2019, 5:32 a.m.)


Review request for atlas, madhan, Nixon Rodrigues, and Sarath Subramanian.


Changes
---

Resolved merge issue.


Bugs: ATLAS-3387
https://issues.apache.org/jira/browse/ATLAS-3387


Repository: atlas


Description
---

Consider X-FORWARDED-FOR header for getting end user IP address when connected 
with proxy


Diffs (updated)
-

  
authorization/src/main/java/org/apache/atlas/authorize/AtlasAccessRequest.java 
b031f4c3e 
  
authorization/src/main/java/org/apache/atlas/authorize/AtlasAdminAccessRequest.java
 1782b3272 
  
authorization/src/main/java/org/apache/atlas/authorize/AtlasAuthorizationUtils.java
 ac2f52525 
  
authorization/src/main/java/org/apache/atlas/authorize/AtlasEntityAccessRequest.java
 07ff678ed 
  
authorization/src/main/java/org/apache/atlas/authorize/AtlasRelationshipAccessRequest.java
 d2da03c1a 
  
authorization/src/main/java/org/apache/atlas/authorize/AtlasSearchResultScrubRequest.java
 c908b28a7 
  
authorization/src/main/java/org/apache/atlas/authorize/AtlasTypeAccessRequest.java
 af38425fb 
  server-api/src/main/java/org/apache/atlas/RequestContext.java 0c3ba08df 
  webapp/src/main/java/org/apache/atlas/web/filters/AuditFilter.java e9c44b336 


Diff: https://reviews.apache.org/r/71406/diff/8/

Changes: https://reviews.apache.org/r/71406/diff/7-8/


Testing
---

Tested with simple authorizer. Need to test after ranger end changes.


Thanks,

Nikhil Bonte

Review Request 71600: ATLAS-3441 - Add JVM resource metrics for Atlas monitoring

[Nikhil Bonte]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71600/
---

Review request for atlas and Sarath Subramanian.


Bugs: ATLAS-3441
https://issues.apache.org/jira/browse/ATLAS-3441


Repository: atlas


Description
---

This will include important JVM attributes to existing metric REST API for 
monitoring.

JSON structure:
{
  "data": {
"general": {
...
},
"system": {
  "memory": {
"heapCommitted": "687865856",
"heapInit": "262144000",
"heapMax": "954728448",
"heapUsed": "398610416",
"memory_pool_usages": {
  "PS Eden Space": {
"committed": 217579520,
"init": 66060288,
"max": 302514176,
"used": 171862912
  },
  "PS Old Gen": {
"committed": 449314816,
"init": 175112192,
"max": 716177408,
"used": 205808048
  },
  "PS Survivor Space": {
"committed": 20971520,
"init": 10485760,
"max": 20971520,
"used": 20939456
  }
},
"nonHeapCommitted": "172163072",
"nonHeapInit": "2555904",
"nonHeapMax": "-1",
"nonHeapUsed": "168581552"
  },
  "os": {
"os.spec": "Linux, amd64, 4.15.0-51-generic",
"os.vcpus": "4"
  },
  "runtime": {
    "name": "OpenJDK 64-Bit Server VM",
"version": "1.8.0_212"
  }
}
  }
}


Diffs
-

  repository/src/main/java/org/apache/atlas/services/MetricsService.java 
2a65b5558 
  repository/src/main/java/org/apache/atlas/util/AtlasMetricJVMUtil.java 
PRE-CREATION 


Diff: https://reviews.apache.org/r/71600/diff/1/


Testing
---

Verified using API - GET /api/atlas/admin/metrics


Thanks,

Nikhil Bonte

Re: Review Request 71683: ATLAS-3489: Import API: missing entity in atlas after import

[Nikhil Bonte]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71683/#review218413
---


Ship it!




Ship It!

- Nikhil Bonte


On Oct. 26, 2019, 5:23 a.m., Ashutosh Mestry wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/71683/
> ---
> 
> (Updated Oct. 26, 2019, 5:23 a.m.)
> 
> 
> Review request for atlas, Madhan Neethiraj, Nikhil Bonte, Nixon Rodrigues, 
> and Sarath Subramanian.
> 
> 
> Bugs: ATLAS-3489
> https://issues.apache.org/jira/browse/ATLAS-3489
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Bug Details:
> 
> Input :
> * Consider case where input zip contains hive_table which has 2 
> hive_columns with same qualifiedName but   different 
> entityStatus.
> * One of the columns is ACTIVE while other is DELETED.
> 
> Expected Result :
> 
> Imported table must contain 2 columns with same qualifiedName but with 
> different entityStatus.
> 
> Actual Result :
> 
> Imported table has 1 column with guid of ACTIVE column from input zip & 
> entityStatus as DELETED.
> 
> 
> Problem Area :
> 
> * Problem is in 
> AtlasEntityStoreV2.preCreateOrUpdate().getResolvedEntityVertex(discoveryContext,
>  entity), #951.
> * This method fetches already resolved vertex for column with same 
> qualifiedName since both entities have same uniqueAttributes, where as it 
> should return null vertex & further new vertex should be created for entity 
> with different entityStatus.
> 
> Solution :
> 
> * Moved field 'entityStatus' from class AtlasRelatedObjectId to class 
> AtlasObjectId.
> * Changed constructor of AtlasObjectId used in 
> EntityGraphRetriever.toAtlasObjectId(AtlasEntity entity).
> 
> 
> Diffs
> -
> 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityStoreV2.java
>  5b3c27dff 
>   
> repository/src/test/java/org/apache/atlas/repository/impexp/ImportServiceTest.java
>  95f6ec3bb 
> 
> 
> Diff: https://reviews.apache.org/r/71683/diff/1/
> 
> 
> Testing
> ---
> 
> Added new UT -> 
> ImportServiceTest.testImportDuplicateColumnsWithDifferentStatus
> Verified all existing UTs locally with the patch.
> Tested manually.
> 
> 
> Thanks,
> 
> Ashutosh Mestry
> 
>

Re: Review Request 71795: DSL Queries: LIKE Clause with Special Characters

[Nikhil Bonte]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71795/#review218729
---


Ship it!




Ship It!

- Nikhil Bonte


On Nov. 20, 2019, 9:52 p.m., Ashutosh Mestry wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/71795/
> ---
> 
> (Updated Nov. 20, 2019, 9:52 p.m.)
> 
> 
> Review request for atlas, Nikhil Bonte, Nixon Rodrigues, and Sarath 
> Subramanian.
> 
> 
> Bugs: ATLAS-3520
> https://issues.apache.org/jira/browse/ATLAS-3520
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> **Background**
> Please refer to the bug description for details.
> 
> **Approach**
> - Detect special chacters in the right-hand-side of LIKE clause.
> - Use textRegex instead of textContainsRegex.
> 
> 
> Diffs
> -
> 
>   repository/src/main/java/org/apache/atlas/query/GremlinQueryComposer.java 
> 294dc00e5 
>   repository/src/test/java/org/apache/atlas/query/DSLQueriesTest.java 
> 724ae9f7a 
>   
> repository/src/test/java/org/apache/atlas/query/GremlinQueryComposerTest.java 
> ca32ffc99 
> 
> 
> Diff: https://reviews.apache.org/r/71795/diff/1/
> 
> 
> Testing
> ---
> 
> **Unit tests**
> - _DSLQueriesTest_ updated to handle this case.
> - _GremlinQueryComposerTest_ updated to handle this case.
> 
> **Functional tests**
> Via UI.
> 
> **Pre-commit**
> https://builds.apache.org/view/A/view/Atlas/job/PreCommit-ATLAS-Build-Test/1545/
> 
> 
> Thanks,
> 
> Ashutosh Mestry
> 
>

Re: Review Request 71813: ATLAS-3536 Difference in results due to tag's case in regex and non-regex search

[Nikhil Bonte]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71813/
---

(Updated Nov. 28, 2019, 11:18 a.m.)


Review request for atlas, Ashutosh Mestry, Le Ma, Madhan Neethiraj, Nixon 
Rodrigues, and Sarath Subramanian.


Bugs: ATLAS-3536
https://issues.apache.org/jira/browse/ATLAS-3536


Repository: atlas


Description
---

Basic search: Difference in results due to tag's case in regex and non-regex 
search


**Dataset**
Create 2 tags: TAG1, tag1

Create 2 hdfs_path entities:

  1. hdfs_path1 , associate it to TAG1
  2. hdfs_path2 , associate it to tag1
 

**Problem:**

Fire basic search with classificationName as:

tag1 -> returns only hdfs_path2
tag* -> returns hdfs_path1 and hdfs_path2 (should only return hdfs_path2) 

e.g.
curl -X POST -u username:password '{host}/api/atlas/v2/search/basic' -H 
'Accept: application/json, text/javascript, */*; q=0.01' -H 'Content-Type: 
application/json' --data-binary 
'{"excludeDeletedEntities":true,"includeSubClassifications":false,"includeSubTypes":false,"includeClassificationAttributes":true,"entityFilters":null,"tagFilters":null,"attributes":[],"limit":25,"offset":0,"classification":"tag*","termName":null}'
 --compressed


**Analysis:**

1. Querying "tag1" also returns hdfs_path1 from index query, the filtering in  
"SearchProcessor.filterWhiteSpaceClassification()" removes hdfs_path1 from 
results as it does not **contain**
 "TAG1" in its classification list.
 
2. However, querying "tag*", does not go through such filtering & request end 
up returning both entities tagged with "tag1" & "TAG1"


** Proposed Solution:**
1. When classification search contains "*" as classification, in this case the 
possible tags should be retrieved first & then from & fire index query using 
all retrieved classification names.

2. This will also result in returning entities tagged with sub classifications 
of classification queried, when SearchParameters.includeSubClassifications is 
set to true (this is not supported currently with wildcard search).


**Code changes:**

1. Added case of isWildCardSearch into SearchContext's constructor. Doing this 
will take care of subClassifications while forming index query. 

2. Removed handling of case isWildcardSearch in class 
ClassificationSearchProcessor's constructor. This is diverted into flow of 
registered classification search.


Index query comparison for "tag*"
  1. with current approach with includeSubClassifications as
 A. false ->  ($v$"__classificationNames":tag* OR 
$v$"__propagatedClassificationNames":tag*)
 B. true  ->  ($v$"__classificationNames":tag* OR 
$v$"__propagatedClassificationNames":tag*)
 
  2. with new approach with includeSubClassifications as
 A. false ->  ($v$"__classificationNames":(tag1) OR 
$v$"__propagatedClassificationNames":(tag1))
 B. true  ->  ($v$"__classificationNames":(tag1 sub_tag_lower) OR 
$v$"__propagatedClassificationNames":(tag1 sub_tag_lower))


Diffs
-

  intg/src/main/java/org/apache/atlas/type/AtlasTypeRegistry.java b071dc9d6 
  
repository/src/main/java/org/apache/atlas/discovery/ClassificationSearchProcessor.java
 c0a5a46dd 
  repository/src/main/java/org/apache/atlas/discovery/SearchContext.java 
353411363 
  
repository/src/test/java/org/apache/atlas/query/BasicClassificationSearchTest.java
 PRE-CREATION 
  repository/src/test/java/org/apache/atlas/query/BasicTestSetup.java 02f78b369 


Diff: https://reviews.apache.org/r/71813/diff/6/


Testing (updated)
---

PC build - https://builds.apache.org/job/PreCommit-ATLAS-Build-Test/1570/

Pending 
Add/update some UT to verify with propagatedClassification.


Thanks,

Nikhil Bonte

Re: Review Request 72039: ATLAS-3595 Migration import: support zip file for data migration

[Nikhil Bonte]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72039/
---

(Updated Jan. 27, 2020, 1:56 p.m.)


Review request for atlas, Ashutosh Mestry, Madhan Neethiraj, Nixon Rodrigues, 
and Sarath Subramanian.


Changes
---

Addressed review comments.


Bugs: ATLAS-3595
https://issues.apache.org/jira/browse/ATLAS-3595


Repository: atlas


Description
---

Currently Data migration service works with json files as input.

This improvement will enable Data Migration service to import data using a zip 
file.


Diffs (updated)
-

  common/src/main/java/org/apache/atlas/service/Services.java 7b36db5ff 
  
repository/src/main/java/org/apache/atlas/repository/migration/DataMigrationService.java
 170136195 
  
repository/src/main/java/org/apache/atlas/repository/migration/ZipMigrator.java 
PRE-CREATION 
  
repository/src/test/java/org/apache/atlas/repository/migration/MigrationBaseAsserts.java
 18e950e8c 


Diff: https://reviews.apache.org/r/72039/diff/5/

Changes: https://reviews.apache.org/r/72039/diff/4-5/


Testing
---

PC build - https://builds.apache.org/job/PreCommit-ATLAS-Build-Test/1611/


Thanks,

Nikhil Bonte

Re: Review Request 72039: ATLAS-3595 Migration import: support zip file for data migration

[Nikhil Bonte]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72039/
---

(Updated Jan. 31, 2020, 11:52 a.m.)


Review request for atlas, Ashutosh Mestry, Madhan Neethiraj, Nixon Rodrigues, 
and Sarath Subramanian.


Bugs: ATLAS-3595
https://issues.apache.org/jira/browse/ATLAS-3595


Repository: atlas


Description (updated)
---

Data migration service added under ATLAS-2461 works with json files as input.

This improvement will enable Data Migration service to import data using a zip 
file.

Approach:

* Property value for migration should contain full path of file, below is the 
example. 
* Data migration service will first check migration property if value refers to 
a zip file.
* If true, it will call ZipMigrator to import data in zip file.
* ZipMigrator internally uses existing ImportService to ingest zip.
* Audit service & Notification service will also be started as a part of zip 
import.


E.g. properties to enable zip data migration:

* atlas.migration.data.filename=/path/fileName.zip
* atlas.migration.mode.workers=5
* atlas.migration.mode.batch.size=20


Diffs
-

  common/src/main/java/org/apache/atlas/service/Services.java 7b36db5ff 
  
repository/src/main/java/org/apache/atlas/repository/migration/DataMigrationService.java
 170136195 
  
repository/src/main/java/org/apache/atlas/repository/migration/ZipMigrator.java 
PRE-CREATION 
  
repository/src/test/java/org/apache/atlas/repository/migration/MigrationBaseAsserts.java
 18e950e8c 


Diff: https://reviews.apache.org/r/72039/diff/6/


Testing
---

PC build - https://builds.apache.org/job/PreCommit-ATLAS-Build-Test/1614/


Thanks,

Nikhil Bonte

Re: Review Request 72039: ATLAS-3595, ATLAS-3603 Migration import: support zip file for data migration

[Nikhil Bonte]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72039/
---

(Updated Feb. 13, 2020, 1:02 p.m.)


Review request for atlas, Ashutosh Mestry, Madhan Neethiraj, Nixon Rodrigues, 
and Sarath Subramanian.


Changes
---

* Addressed review comments.
* Included ATLAS-3603.


Summary (updated)
-

ATLAS-3595, ATLAS-3603 Migration import: support zip file for data migration


Bugs: ATLAS-3595
https://issues.apache.org/jira/browse/ATLAS-3595


Repository: atlas


Description (updated)
---

## ATLAS-3595
Data migration service added under ATLAS-2461 works with json files as input.

This improvement will enable Data Migration service to import data using a zip 
file.

Approach:

* Property value for migration should contain full path of file, below is the 
example. 
* Data migration service will first check migration property if value refers to 
a zip file.
* If true, it will call ZipMigrationImporter to import data in zip file.
* ZipMigrationImporter internally uses existing ImportService to ingest zip.
* Audit service & Notification service will also be started as a part of zip 
import.


E.g. properties to enable zip data migration:

* atlas.migration.data.filename=/path/fileName.zip
* atlas.migration.mode.workers=5
* atlas.migration.mode.batch.size=20


## ATLAS-3603
Migration import should start when the data (zip) to be ingested is available 
at the pre-configured location.

Approach:

ZipMigrationImporter will first check if zip is available, accessible & 
completely written.

If yes, migration will start ingesting zip.
if not, will poll the location periodically. when existence is detected, import 
will proceed as usual.

It uses the file size comparison approach. Once modify event is received, file 
size is compared in an interval. If sizes match, it means file is available and 
not being written.


Diffs (updated)
-

  common/src/main/java/org/apache/atlas/service/Services.java 7b36db5ff 
  
repository/src/main/java/org/apache/atlas/repository/migration/DataMigrationService.java
 170136195 
  
repository/src/main/java/org/apache/atlas/repository/migration/FileWatcher.java 
PRE-CREATION 
  
repository/src/main/java/org/apache/atlas/repository/migration/ZipMigrationImporter.java
 PRE-CREATION 
  
repository/src/test/java/org/apache/atlas/repository/impexp/MigrationImportTest.java
 PRE-CREATION 


Diff: https://reviews.apache.org/r/72039/diff/7/

Changes: https://reviews.apache.org/r/72039/diff/6-7/


Testing
---

PC build - https://builds.apache.org/job/PreCommit-ATLAS-Build-Test/1614/


Thanks,

Nikhil Bonte

Re: Review Request 72177: Import Service: Support zipDirect Format

[Nikhil Bonte]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72177/#review219714
---


Ship it!




Ship It!

- Nikhil Bonte


On Feb. 28, 2020, 9:52 p.m., Ashutosh Mestry wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72177/
> ---
> 
> (Updated Feb. 28, 2020, 9:52 p.m.)
> 
> 
> Review request for atlas, Madhan Neethiraj, Nikhil Bonte, Nixon Rodrigues, 
> and Sarath Subramanian.
> 
> 
> Bugs: ATLAS-3641
> https://issues.apache.org/jira/browse/ATLAS-3641
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> **Approach**
> During genration entities are stores as 1 zip entry, _entities.json_.
> 
> New: _ZipSourceDirect_ contains a light-weight JSON parser to read the 
> _entities.json_. This parser drives _next_ and _hasNext_ methods.
> Modified: _AtlasImportRequest_ adds new key and value to specify import 
> format. 
> Modified: _ImportService_ supports handling of this format.
> 
> _import-options.json_
> ```
> {
> "options": {
>   "format": "zipDirect"
>  }
> }
> ```
> 
> 
> Diffs
> -
> 
>   intg/src/main/java/org/apache/atlas/model/impexp/AtlasImportRequest.java 
> 0b3ede93f 
>   intg/src/main/java/org/apache/atlas/utils/AtlasJson.java 1c13860cf 
>   
> repository/src/main/java/org/apache/atlas/repository/impexp/ImportService.java
>  27001e3a9 
>   
> repository/src/main/java/org/apache/atlas/repository/impexp/ZipExportFileNames.java
>  351b47536 
>   
> repository/src/main/java/org/apache/atlas/repository/impexp/ZipSourceDirect.java
>  PRE-CREATION 
>   
> repository/src/test/java/org/apache/atlas/repository/impexp/ImportServiceTest.java
>  c14850f43 
>   
> repository/src/test/java/org/apache/atlas/repository/impexp/ZipDirectTest.java
>  PRE-CREATION 
>   repository/src/test/resources/zip-direct-1.zip PRE-CREATION 
>   repository/src/test/resources/zip-direct-2.zip PRE-CREATION 
>   repository/src/test/resources/zip-direct-3.zip PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/72177/diff/1/
> 
> 
> Testing
> ---
> 
> **Unit tests**
> - Tests for reading this format.
> - _ImportServiceTest_ performs end-to-end test.
> 
> **Volume test**
> Files with this format lends itself to volume. File with 10 million entities 
> was performed.
> 
> **Pre-commit**
> https://builds.apache.org/view/A/view/Atlas/job/PreCommit-ATLAS-Build-Test/
> 
> 
> Thanks,
> 
> Ashutosh Mestry
> 
>

Re: Review Request 72177: Import Service: Support zipDirect Format

[Nikhil Bonte]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72177/#review219715
---



"Ship It" for the patch.
Please review PC build - 
https://builds.apache.org/job/PreCommit-ATLAS-Build-Test/1686/console
Issue while applying patch as it has zips, I suggest to push zips as separate 
commit.

- Nikhil Bonte


On Feb. 28, 2020, 9:52 p.m., Ashutosh Mestry wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72177/
> ---
> 
> (Updated Feb. 28, 2020, 9:52 p.m.)
> 
> 
> Review request for atlas, Madhan Neethiraj, Nikhil Bonte, Nixon Rodrigues, 
> and Sarath Subramanian.
> 
> 
> Bugs: ATLAS-3641
> https://issues.apache.org/jira/browse/ATLAS-3641
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> **Approach**
> During genration entities are stores as 1 zip entry, _entities.json_.
> 
> New: _ZipSourceDirect_ contains a light-weight JSON parser to read the 
> _entities.json_. This parser drives _next_ and _hasNext_ methods.
> Modified: _AtlasImportRequest_ adds new key and value to specify import 
> format. 
> Modified: _ImportService_ supports handling of this format.
> 
> _import-options.json_
> ```
> {
> "options": {
>   "format": "zipDirect"
>  }
> }
> ```
> 
> 
> Diffs
> -
> 
>   intg/src/main/java/org/apache/atlas/model/impexp/AtlasImportRequest.java 
> 0b3ede93f 
>   intg/src/main/java/org/apache/atlas/utils/AtlasJson.java 1c13860cf 
>   
> repository/src/main/java/org/apache/atlas/repository/impexp/ImportService.java
>  27001e3a9 
>   
> repository/src/main/java/org/apache/atlas/repository/impexp/ZipExportFileNames.java
>  351b47536 
>   
> repository/src/main/java/org/apache/atlas/repository/impexp/ZipSourceDirect.java
>  PRE-CREATION 
>   
> repository/src/test/java/org/apache/atlas/repository/impexp/ImportServiceTest.java
>  c14850f43 
>   
> repository/src/test/java/org/apache/atlas/repository/impexp/ZipDirectTest.java
>  PRE-CREATION 
>   repository/src/test/resources/zip-direct-1.zip PRE-CREATION 
>   repository/src/test/resources/zip-direct-2.zip PRE-CREATION 
>   repository/src/test/resources/zip-direct-3.zip PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/72177/diff/1/
> 
> 
> Testing
> ---
> 
> **Unit tests**
> - Tests for reading this format.
> - _ImportServiceTest_ performs end-to-end test.
> 
> **Volume test**
> Files with this format lends itself to volume. File with 10 million entities 
> was performed.
> 
> **Pre-commit**
> https://builds.apache.org/view/A/view/Atlas/job/PreCommit-ATLAS-Build-Test/
> 
> 
> Thanks,
> 
> Ashutosh Mestry
> 
>

Re: Review Request 72180: PC Framework: Add Status Reporting

[Nikhil Bonte]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72180/#review219716
---


Ship it!




Ship It!

- Nikhil Bonte


On Feb. 28, 2020, 10:56 p.m., Ashutosh Mestry wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72180/
> ---
> 
> (Updated Feb. 28, 2020, 10:56 p.m.)
> 
> 
> Review request for atlas, Madhan Neethiraj, Nikhil Bonte, Nixon Rodrigues, 
> and Sarath Subramanian.
> 
> 
> Bugs: ATLAS-3643
> https://issues.apache.org/jira/browse/ATLAS-3643
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> **Approach**
> New: _StatusReporter_: Generic class to be used with producer-consumer.
> 
> 
> Diffs
> -
> 
>   intg/src/main/java/org/apache/atlas/pc/StatusReporter.java PRE-CREATION 
>   intg/src/test/java/org/apache/atlas/pc/StatusReporterTest.java PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/72180/diff/1/
> 
> 
> Testing
> ---
> 
> **Unit tests**
> New test added that demonstrates usage.
> 
> 
> Thanks,
> 
> Ashutosh Mestry
> 
>

Re: Review Request 71025: Import Service: Support Concurrent Ingest

[Nikhil Bonte]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71025/#review219746
---


Ship it!




Ship It!

- Nikhil Bonte


On March 3, 2020, 5:13 a.m., Ashutosh Mestry wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/71025/
> ---
> 
> (Updated March 3, 2020, 5:13 a.m.)
> 
> 
> Review request for atlas, Madhan Neethiraj, Nikhil Bonte, Nixon Rodrigues, 
> and Sarath Subramanian.
> 
> 
> Bugs: ATLAS-3320
> https://issues.apache.org/jira/browse/ATLAS-3320
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> **Approach**
> - Use existing producer-consumer (PC) framework.
> - Modify _BulkImporterImpl_ to implement _WorkItemConsumer_.
> - Add support for configuring number of workers and batch size within 
> _AtlasImportRequest_.
> - Existing import implementation continues to function as before. This is 
> maintained for backward compatibility.
> - New implementation supports additional more memory efficient zip format 
> (_ZipDirect_). This drastically reduces memory requirement during import.
> - The new import strategy, _MigrationImport_ uses the _bulkLoading_ mode of 
> _JanusGraph_ thereby achieving high ingest rates.
> 
> _AtlasImportRequest_
> ```
> {
> "options": {
> "numWorkers": 8,
> "batchSize": 25
> }
> }
> ```
> Support for ZipDirect format:
> _AtlasImportRequest_
> ```
> {
> "options": {
> "numWorkers": 8,
> "batchSize": 25,
> "format": "zipDirect",
> "migration": "true"
> }
> }
> ```
> 
> 
> **CURL**
> ```
> curl -v -X POST -u admin:admin -H "Content-Type: multipart/form-data" -H 
> "Cache-Control: no-cache" -F request=@./import-options.json -F 
> data=@./Default-3-pre.zip http://localhost:21000/api/atlas/admin/import
> ```
> 
> 
> Diffs
> -
> 
>   
> graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraph.java
>  4acb371f1 
>   intg/src/main/java/org/apache/atlas/model/impexp/AtlasImportRequest.java 
> 3362bf158 
>   repository/src/main/java/org/apache/atlas/GraphTransactionInterceptor.java 
> bbe0dc5ba 
>   
> repository/src/main/java/org/apache/atlas/repository/impexp/AuditsWriter.java 
> 55990f780 
>   
> repository/src/main/java/org/apache/atlas/repository/impexp/ImportService.java
>  1964ade9a 
>   
> repository/src/main/java/org/apache/atlas/repository/impexp/ZipSourceDirect.java
>  cb5a7acd0 
>   
> repository/src/main/java/org/apache/atlas/repository/migration/ZipFileMigrationImporter.java
>  f552525a4 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/AtlasEntityStore.java
>  39ea3f82e 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityStoreV2.java
>  30f5e5a7c 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasRelationshipStoreV2.java
>  fdf117a25 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/BulkImporterImpl.java
>  54c32c5e8 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphMapper.java
>  2f3aad06b 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/bulkimport/ImportStrategy.java
>  PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/bulkimport/MigrationImport.java
>  PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/bulkimport/RegularImport.java
>  PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/bulkimport/pc/EntityConsumer.java
>  PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/bulkimport/pc/EntityConsumerBuilder.java
>  PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/bulkimport/pc/EntityCreationManager.java
>  PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/71025/diff/9/
> 
> 
> Testing
> ---
> 
> **Unit tests**
> Existing tests.
> 
> **Functional tests**
> - Verified import for pre-1.0 and post-1.0 exported ZIP files.
> 
> **Pre-commit**
> https://builds.apache.org/view/A/view/Atlas/job/PreCommit-A

Re: Review Request 72235: ZipFileMigrator: Automatic Resume During Migration

[Nikhil Bonte]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72235/#review219958
---


Ship it!




Ship It!

- Nikhil Bonte


On March 13, 2020, 9:56 p.m., Ashutosh Mestry wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72235/
> ---
> 
> (Updated March 13, 2020, 9:56 p.m.)
> 
> 
> Review request for atlas, Nikhil Bonte, Nixon Rodrigues, and Sarath 
> Subramanian.
> 
> 
> Bugs: ATLAS-3663
> https://issues.apache.org/jira/browse/ATLAS-3663
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> **Approach**
> *new* DataMigrationStatusService: Creates a vertex specific to the migration. 
> Manages the vertex during the lifetime of the migration. Uses _indexQuery_ 
> for quick retrieval.
> *modified* ZipFileMigrationImporter: Consume the new service. Displays 
> appropriate log messages.
> *modified* EntityCreationManager: To consume the new position returned by the 
> service.
> *modified* EntityConsumer: Update last known good position.
> 
> The position is maintained across server restarts. On successful completion, 
> the vertex is deleted.
> 
> **Status REST API**
> This API now reflects migration status:
> ```
> http://amestryb-2.gce.cloudera.com:31000/api/atlas/admin/status
> ```
> 
> Output:
> ```
> {"Status":"MIGRATING","MigrationStatus":{"startTime":1584135451790,"currentIndex":44584,"currentCounter":0,"totalCount":6211055,"name":"/tmp/atlas-data/file-to-import-atlas.zip"}}
> ```
> 
> 
> Diffs
> -
> 
>   intg/src/main/java/org/apache/atlas/model/impexp/AtlasImportRequest.java 
> 09dafdf24 
>   
> intg/src/main/java/org/apache/atlas/model/migration/MigrationImportStatus.java
>  PRE-CREATION 
>   intg/src/main/java/org/apache/atlas/pc/StatusReporter.java f84e8d00f 
>   intg/src/test/java/org/apache/atlas/pc/StatusReporterTest.java 3e5056267 
>   
> repository/src/main/java/org/apache/atlas/repository/impexp/ImportService.java
>  c18c4ab68 
>   
> repository/src/main/java/org/apache/atlas/repository/impexp/MigrationProgressService.java
>  54ae32a01 
>   
> repository/src/main/java/org/apache/atlas/repository/migration/DataMigrationStatusService.java
>  PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/repository/migration/ZipFileMigrationImporter.java
>  35a76ea3d 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/bulkimport/MigrationImport.java
>  4c912fdbe 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/bulkimport/pc/EntityConsumer.java
>  e8f4b02a0 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/bulkimport/pc/EntityCreationManager.java
>  16bb49e86 
>   
> repository/src/test/java/org/apache/atlas/repository/impexp/DataMigrationStatusServiceTest.java
>  PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/72235/diff/4/
> 
> 
> Testing
> ---
> 
> **Unit tests**
> New tests added.
> 
> **REST API**
> Verified.
> 
> **Pre-commit**
> https://builds.apache.org/view/A/view/Atlas/job/PreCommit-ATLAS-Build-Test/1736/
> 
> 
> Thanks,
> 
> Ashutosh Mestry
> 
>

Re: Apache-atlas-3.0.0-SNAPSHOT shows only 1 entity type

[Nikhil Bonte]

Hi Bandeep,

Adding property *atlas.ui.editable.entity.types=* *to the atlas
configuration file *atlas-application.properties *will help you.



On Wed, Mar 18, 2020 at 5:16 AM Bandeep Singh  wrote:

> Hi Guys,
> I am very new to Apache atlas and recently started learning about it.
> I have setup the latest Apache-atlas-3.0.0-SNAPSHOT from the github repo
> with embedded hbase and solr.
>
> When i start atlas,  go to UI and try to add an entity i see only 1 entity
> type "hdfs_path".
>
> Does anybody know what am i missing here ?
>
> Really appreciate any help on this !
>
> Thanks
>


-- 
Regards
Nikhil P. Bonte

Re: Review Request 72287: Edge Creation Improvements

[Nikhil Bonte]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72287/#review220180
---


Ship it!




Ship It!

- Nikhil Bonte


On March 30, 2020, 11:19 p.m., Ashutosh Mestry wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72287/
> ---
> 
> (Updated March 30, 2020, 11:19 p.m.)
> 
> 
> Review request for atlas, Madhan Neethiraj, Nikhil Bonte, Nixon Rodrigues, 
> and Sarath Subramanian.
> 
> 
> Bugs: ATLAS-3706
> https://issues.apache.org/jira/browse/ATLAS-3706
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> **Approach**
> 
> 1. Added Metrics to most of the methods in entity creation. (The patch does 
> not include the additional metrics added to additional places.)
> 2. Started importing large number of entities using the 
> _ZipFileMigrationImporter_.
> 3. Observed behavior of import over 24 hours. Observations included CPU 
> usage, memory usage and the import throughput using the _metric.log_.
> 4. Changes were added to the one at a time. Impact of the change was observed 
> for performance (via metric.log) and accuracy before next change was added.
> 
> **Observations**
> * Relationship creation took inordinately large amount of time under load. 
> The time was spent in _GraphHelper.getAdjacentEdgesByLabel_. This 
> implementation also caused memory build up of _AtlasEdge_ objects which 
> stayed in memory for long time. This had the secondary effect of slowing down 
> entity creation operations after about 6 hours (this duration differed with 
> node configuration).
> * _GraphHelper.getOrCreateEdge_ did a vertex to vertex comparison which is 
> time consuming.
> * _GraphBackedSearchIndexer_ edge label index. Majority of edge creation 
> operation included lookup by edge label.
> 
> **Configuration**
> Cluster: 3 node: 40 cores, 128 GB RAM, 1.5 TB of disk space.
> Atlas configuration: 32 GB RAM.
> 
> 
> Diffs
> -
> 
>   
> repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedSearchIndexer.java
>  647e3040c 
>   repository/src/main/java/org/apache/atlas/repository/graph/GraphHelper.java 
> 5ab9f4d13 
> 
> 
> Diff: https://reviews.apache.org/r/72287/diff/1/
> 
> 
> Testing
> ---
> 
> **Manual tests**
> (See above).
> Accuracy verification.
> 
> **Unit tests**
> Executed existing unit tests.
> 
> **Pre-commit build**
> https://builds.apache.org/view/A/view/Atlas/job/PreCommit-ATLAS-Build-Test/1776/
> 
> 
> Thanks,
> 
> Ashutosh Mestry
> 
>

Re: Review Request 72270: ATLAS-3689 Add audits entries when Business Metadata Attributes are added/updated/deleted to an entity

[Nikhil Bonte]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72270/#review220263
---




intg/src/main/java/org/apache/atlas/listener/EntityChangeListenerV2.java
Lines 185 (patched)
<https://reviews.apache.org/r/72270/#comment308604>

I suggest to have input as list of businessMetadataAttributes.
That way events can be put for all businessMetadataAttributes at one go 
instead putting for 1 attribute at a time.

Same may be done for case update, delete.


- Nikhil Bonte


On April 8, 2020, 2:30 p.m., Mandar Ambawane wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72270/
> ---
> 
> (Updated April 8, 2020, 2:30 p.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Jayendra Parab, Madhan Neethiraj, 
> Nixon Rodrigues, Sarath Subramanian, and Sidharth Mishra.
> 
> 
> Bugs: ATLAS-3689
> https://issues.apache.org/jira/browse/ATLAS-3689
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Add audits entries when Namespace Attributes are added/updated/deleted to an 
> entity
> 
> 
> Diffs
> -
> 
>   intg/src/main/java/org/apache/atlas/listener/EntityChangeListenerV2.java 
> 2394a12 
>   intg/src/main/java/org/apache/atlas/model/audit/EntityAuditEventV2.java 
> 9301e21 
>   
> repository/src/main/java/org/apache/atlas/repository/audit/AtlasAuditService.java
>  590f7a0 
>   
> repository/src/main/java/org/apache/atlas/repository/audit/EntityAuditListenerV2.java
>  cab4e1e 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityChangeNotifier.java
>  00c0114 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphMapper.java
>  0d5b631 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/IAtlasEntityChangeNotifier.java
>  c4dc5a1 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/bulkimport/EntityChangeNotifierNop.java
>  2943ea9 
>   
> webapp/src/main/java/org/apache/atlas/notification/EntityNotificationListenerV2.java
>  6d64fec 
> 
> 
> Diff: https://reviews.apache.org/r/72270/diff/3/
> 
> 
> Testing
> ---
> 
> Pre-commit: 
> https://builds.apache.org/job/PreCommit-ATLAS-Build-Test/1759/console
> 
> Basic testing is done.
> 
> 
> Thanks,
> 
> Mandar Ambawane
> 
>

Review Request 72569: ATLAS-3816: Backport- Basic search issue with IndexSerializer

[Nikhil Bonte]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72569/
---

Review request for atlas, Ashutosh Mestry, Jayendra Parab, madhan, Nixon 
Rodrigues, Sarath Subramanian, and Sidharth Mishra.


Bugs: ATLAS-3816
https://issues.apache.org/jira/browse/ATLAS-3816


Repository: atlas


Description
---

Direct index query on janusgraph throws
java.lang.IllegalArgumentException: Found reference to non-existant property 
key in query at position...

Please refer jira for more details.


Diffs
-

  common/src/main/java/org/apache/atlas/repository/Constants.java 4fa6dadfb 
  
graphdb/titan0/src/main/java/org/apache/atlas/repository/graphdb/titan0/Titan0Graph.java
 2408287bf 
  
graphdb/titan1/src/main/java/org/apache/atlas/repository/graphdb/titan1/Titan1Graph.java
 622c90019 
  
repository/src/main/java/org/apache/atlas/discovery/EntityDiscoveryService.java 
5de4db560 
  
repository/src/main/java/org/apache/atlas/discovery/FullTextSearchProcessor.java
 5ffd88b85 
  repository/src/main/java/org/apache/atlas/discovery/SearchProcessor.java 
fab707455 
  
repository/src/main/java/org/apache/atlas/discovery/graph/GraphBackedDiscoveryService.java
 e5e527fdf 
  
repository/src/main/java/org/apache/atlas/repository/store/graph/v1/AtlasGraphUtilsV1.java
 bf46c4722 
  
repository/src/test/java/org/apache/atlas/repository/graph/GraphRepoMapperScaleTest.java
 1f077ba06 


Diff: https://reviews.apache.org/r/72569/diff/1/


Testing
---

Manual testing for identified senarios.

PC build : 
https://builds.apache.org/job/PreCommit-ATLAS-Build-Test/1955/console 
(Finished: SUCCESS)


Thanks,

Nikhil Bonte

Re: Review Request 72569: ATLAS-3816: Backport- Basic search issue with IndexSerializer

[Nikhil Bonte]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72569/
---

(Updated June 8, 2020, 5:46 a.m.)


Review request for atlas, Ashutosh Mestry, Jayendra Parab, madhan, Nixon 
Rodrigues, Sarath Subramanian, and Sidharth Mishra.


Changes
---

Addressed review comments


Bugs: ATLAS-3816
https://issues.apache.org/jira/browse/ATLAS-3816


Repository: atlas


Description
---

Direct index query on janusgraph throws
java.lang.IllegalArgumentException: Found reference to non-existant property 
key in query at position...

Please refer jira for more details.


Diffs (updated)
-

  common/src/main/java/org/apache/atlas/repository/Constants.java 4fa6dadfb 
  
graphdb/titan0/src/main/java/org/apache/atlas/repository/graphdb/titan0/Titan0Graph.java
 2408287bf 
  
graphdb/titan1/src/main/java/org/apache/atlas/repository/graphdb/titan1/Titan1Graph.java
 622c90019 
  
repository/src/main/java/org/apache/atlas/discovery/EntityDiscoveryService.java 
5de4db560 
  
repository/src/main/java/org/apache/atlas/discovery/FullTextSearchProcessor.java
 5ffd88b85 
  repository/src/main/java/org/apache/atlas/discovery/SearchProcessor.java 
fab707455 
  
repository/src/main/java/org/apache/atlas/discovery/graph/GraphBackedDiscoveryService.java
 e5e527fdf 
  
repository/src/main/java/org/apache/atlas/repository/store/graph/v1/AtlasGraphUtilsV1.java
 bf46c4722 
  
repository/src/test/java/org/apache/atlas/repository/graph/GraphRepoMapperScaleTest.java
 1f077ba06 


Diff: https://reviews.apache.org/r/72569/diff/2/

Changes: https://reviews.apache.org/r/72569/diff/1-2/


Testing
---

Manual testing for identified scenarios.

PC build : 
https://builds.apache.org/job/PreCommit-ATLAS-Build-Test/1955/console 
(Finished: SUCCESS)


Thanks,

Nikhil Bonte

Re: Review Request 72595: ATLAS-3833 : Packaging for atlas index repair tool

[Nikhil Bonte]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72595/#review221011
---




tools/atlas-index-repair/README
Lines 1 (patched)
<https://reviews.apache.org/r/72595/#comment309785>

Need to add atlas-index-repair/pom.xml in modules list of root pom.xml


- Nikhil Bonte


On June 16, 2020, 1:44 p.m., chaitali wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72595/
> ---
> 
> (Updated June 16, 2020, 1:44 p.m.)
> 
> 
> Review request for atlas, Jayendra Parab, Madhan Neethiraj, Nixon Rodrigues, 
> and Sarath Subramanian.
> 
> 
> Bugs: ATLAS-3833
> https://issues.apache.org/jira/browse/ATLAS-3833
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Add packaging for apache atlas build for tools/atlas-index-repair
> 
> 
> Diffs
> -
> 
>   distro/pom.xml 7159b16cf 
>   distro/src/main/assemblies/atlas-repair-index-package.xml PRE-CREATION 
>   tools/atlas-index-repair/README PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/72595/diff/2/
> 
> 
> Testing
> ---
> 
> verified tools directory after packaging atlas
> 
> 
> Thanks,
> 
> chaitali
> 
>

Re: Review Request 72614: ATLAS-3856 Create new entity types for Apache Ozone

[Nikhil Bonte]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72614/
---

(Updated June 23, 2020, 5:45 a.m.)


Review request for atlas, Ashutosh Mestry, Jayendra Parab, madhan, Nixon 
Rodrigues, Sarath Subramanian, and Sidharth Mishra.


Bugs: ATLAS-3856
https://issues.apache.org/jira/browse/ATLAS-3856


Repository: atlas


Description
---

Add following ozone typeDefs to Atlas models

Ozone volume
Ozone bucket
Ozone key


Diffs
-

  addons/models/3000-Cloud/3050-ozone-typedefs.json PRE-CREATION 


Diff: https://reviews.apache.org/r/72614/diff/1/


Testing
---


Thanks,

Nikhil Bonte

Re: Review Request 72614: ATLAS-3856 Create new entity types for Apache Ozone

[Nikhil Bonte]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72614/
---

(Updated June 24, 2020, 2:07 p.m.)


Review request for atlas, Ashutosh Mestry, Jayendra Parab, madhan, Nixon 
Rodrigues, Sarath Subramanian, and Sidharth Mishra.


Changes
---

Addressed review comments


Bugs: ATLAS-3856
https://issues.apache.org/jira/browse/ATLAS-3856


Repository: atlas


Description (updated)
---

Add following ozone typeDefs to Atlas models

entityDefs:
Ozone volume
Ozone bucket
Ozone key

enumDefs:
ozone_storage_type
ozone_replication_type

structDefs:
keyLocation

relationshipDefs:
ozone_volume_buckets
ozone_bucket_keys


Diffs (updated)
-

  addons/models/3000-Cloud/3050-ozone-typedefs.json PRE-CREATION 


Diff: https://reviews.apache.org/r/72614/diff/2/

Changes: https://reviews.apache.org/r/72614/diff/1-2/


Testing
---


Thanks,

Nikhil Bonte

Re: Review Request 72614: ATLAS-3856 Create new entity types for Apache Ozone

[Nikhil Bonte]

> On June 24, 2020, 1:20 a.m., Sarath Subramanian wrote:
> > addons/models/3000-Cloud/3050-ozone-typedefs.json
> > Lines 28 (patched)
> > <https://reviews.apache.org/r/72614/diff/1/?file=2234884#file2234884line28>
> >
> > does "length" type needs to be of string type?

set as "long"


- Nikhil


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72614/#review221057
-------


On June 24, 2020, 2:07 p.m., Nikhil Bonte wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72614/
> ---
> 
> (Updated June 24, 2020, 2:07 p.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Jayendra Parab, madhan, Nixon 
> Rodrigues, Sarath Subramanian, and Sidharth Mishra.
> 
> 
> Bugs: ATLAS-3856
> https://issues.apache.org/jira/browse/ATLAS-3856
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Add following ozone typeDefs to Atlas models
> 
> entityDefs:
> Ozone volume
> Ozone bucket
> Ozone key
> 
> enumDefs:
> ozone_storage_type
> ozone_replication_type
> 
> structDefs:
> keyLocation
> 
> relationshipDefs:
> ozone_volume_buckets
> ozone_bucket_keys
> 
> 
> Diffs
> -----
> 
>   addons/models/3000-Cloud/3050-ozone-typedefs.json PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/72614/diff/2/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Nikhil Bonte
> 
>

Re: Review Request 72614: ATLAS-3856 Create new entity types for Apache Ozone

[Nikhil Bonte]

> On June 24, 2020, 1:20 a.m., Sarath Subramanian wrote:
> > addons/models/3000-Cloud/3050-ozone-typedefs.json
> > Lines 15 (patched)
> > <https://reviews.apache.org/r/72614/diff/1/?file=2234884#file2234884line15>
> >
> > "isOptional" is set to false for all attributes of "keyLocations". Can 
> > there be cases where you don't have all the information and only some.

I am not sure about this, in this case may I keep all attributes of 
"keyLocations" as "isOptional: true"?


- Nikhil


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72614/#review221057
-------


On June 24, 2020, 2:07 p.m., Nikhil Bonte wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72614/
> ---
> 
> (Updated June 24, 2020, 2:07 p.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Jayendra Parab, madhan, Nixon 
> Rodrigues, Sarath Subramanian, and Sidharth Mishra.
> 
> 
> Bugs: ATLAS-3856
> https://issues.apache.org/jira/browse/ATLAS-3856
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Add following ozone typeDefs to Atlas models
> 
> entityDefs:
> Ozone volume
> Ozone bucket
> Ozone key
> 
> enumDefs:
> ozone_storage_type
> ozone_replication_type
> 
> structDefs:
> keyLocation
> 
> relationshipDefs:
> ozone_volume_buckets
> ozone_bucket_keys
> 
> 
> Diffs
> -
> 
>   addons/models/3000-Cloud/3050-ozone-typedefs.json PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/72614/diff/2/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Nikhil Bonte
> 
>

Re: Review Request 72614: ATLAS-3856 Create new entity types for Apache Ozone

[Nikhil Bonte]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72614/
---

(Updated June 24, 2020, 3:07 p.m.)


Review request for atlas, Ashutosh Mestry, Jayendra Parab, madhan, Nixon 
Rodrigues, Sarath Subramanian, and Sidharth Mishra.


Changes
---

Addressed review Comments.


Bugs: ATLAS-3856
https://issues.apache.org/jira/browse/ATLAS-3856


Repository: atlas


Description (updated)
---

Add following ozone typeDefs to Atlas models

entityDefs:
ozone_volume
ozone_bucket
ozone_key

enumDefs:
ozone_storage_type
ozone_replication_type

structDefs:
ozone_key_location

relationshipDefs:
ozone_volume_buckets
ozone_bucket_keys


Diffs (updated)
-

  addons/models/3000-Cloud/3050-ozone-typedefs.json PRE-CREATION 


Diff: https://reviews.apache.org/r/72614/diff/3/

Changes: https://reviews.apache.org/r/72614/diff/2-3/


Testing
---


Thanks,

Nikhil Bonte

Review Request 72631: ATLAS-3871 Add unit tests to path extractor utility for s3, s3 v2, adls path entities

[Nikhil Bonte]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72631/
---

Review request for atlas, Ashutosh Mestry, Jayendra Parab, Madhan Neethiraj, 
Nixon Rodrigues, Sarath Subramanian, and Sidharth Mishra.


Bugs: ATLAS-3871
https://issues.apache.org/jira/browse/ATLAS-3871


Repository: atlas


Description
---

AtlasPathExtractorUtilTest added in ATLAS-3836 which covers unit tests for 
ozone & hdfs path entites.
 
This will improve AtlasPathExtractorUtilTest.java to cover s3, s3v2 & adls path 
entities.

Would add more tests for ozone as well.


Diffs
-

  common/src/test/java/org/apache/atlas/utils/AtlasPathExtractorUtilTest.java 
664bfb72b 


Diff: https://reviews.apache.org/r/72631/diff/1/


Testing
---


Thanks,

Nikhil Bonte

Re: Review Request 72631: ATLAS-3871 Add unit tests to path extractor utility for s3, s3 v2, adls path entities

[Nikhil Bonte]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72631/
---

(Updated June 30, 2020, 1:30 p.m.)


Review request for atlas, Ashutosh Mestry, Jayendra Parab, Madhan Neethiraj, 
Nixon Rodrigues, Sarath Subramanian, and Sidharth Mishra.


Changes
---

Refactorings


Bugs: ATLAS-3871
https://issues.apache.org/jira/browse/ATLAS-3871


Repository: atlas


Description
---

AtlasPathExtractorUtilTest added in ATLAS-3836 which covers unit tests for 
ozone & hdfs path entites.
 
This will improve AtlasPathExtractorUtilTest.java to cover s3, s3v2 & adls path 
entities.

Would add more tests for ozone as well.


Diffs (updated)
-

  common/src/test/java/org/apache/atlas/utils/AtlasPathExtractorUtilTest.java 
664bfb72b 


Diff: https://reviews.apache.org/r/72631/diff/2/

Changes: https://reviews.apache.org/r/72631/diff/1-2/


Testing
---


Thanks,

Nikhil Bonte

Re: Review Request 72631: ATLAS-3871 Add unit tests to path extractor utility for s3, s3 v2, adls path entities

[Nikhil Bonte]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72631/
---

(Updated June 30, 2020, 1:34 p.m.)


Review request for atlas, Ashutosh Mestry, Jayendra Parab, Madhan Neethiraj, 
Nixon Rodrigues, Sarath Subramanian, and Sidharth Mishra.


Changes
---

Added Pre commit build details


Bugs: ATLAS-3871
https://issues.apache.org/jira/browse/ATLAS-3871


Repository: atlas


Description
---

AtlasPathExtractorUtilTest added in ATLAS-3836 which covers unit tests for 
ozone & hdfs path entites.
 
This will improve AtlasPathExtractorUtilTest.java to cover s3, s3v2 & adls path 
entities.

Would add more tests for ozone as well.


Diffs
-

  common/src/test/java/org/apache/atlas/utils/AtlasPathExtractorUtilTest.java 
664bfb72b 


Diff: https://reviews.apache.org/r/72631/diff/2/


Testing (updated)
---

PC build https://builds.apache.org/job/PreCommit-ATLAS-Build-Test/2001/console 
(In Progress)


Thanks,

Nikhil Bonte

Re: Review Request 72631: ATLAS-3871 Add unit tests to path extractor utility for s3, s3 v2, adls path entities

[Nikhil Bonte]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72631/
---

(Updated July 1, 2020, 4:29 a.m.)


Review request for atlas, Ashutosh Mestry, Jayendra Parab, Madhan Neethiraj, 
Nixon Rodrigues, Sarath Subramanian, and Sidharth Mishra.


Changes
---

Updated pre-commit build status


Bugs: ATLAS-3871
https://issues.apache.org/jira/browse/ATLAS-3871


Repository: atlas


Description
---

AtlasPathExtractorUtilTest added in ATLAS-3836 which covers unit tests for 
ozone & hdfs path entites.
 
This will improve AtlasPathExtractorUtilTest.java to cover s3, s3v2 & adls path 
entities.

Would add more tests for ozone as well.


Diffs
-

  common/src/test/java/org/apache/atlas/utils/AtlasPathExtractorUtilTest.java 
664bfb72b 


Diff: https://reviews.apache.org/r/72631/diff/2/


Testing (updated)
---

PC build https://builds.apache.org/job/PreCommit-ATLAS-Build-Test/2001/console 
(Finished: SUCCESS)


Thanks,

Nikhil Bonte

Re: [VOTE] Release Apache Atlas version 2.1.0 - rc2

[Nikhil Bonte]

+1 for Apache Atlas 2.1.0 - rc2 release

Validated the following items.

   - Verified PGP signature.
   - Verified md5 and SHA512 digests.
   - Packaged with embedded-hbase-solr
   - Verified entity creation & tag associations using quick_start.py
   - Tested basic, quick & advanced search.
   - Tested tag dissociation

Thanks
Nikhil P. Bonte

On Mon, Jul 6, 2020 at 12:11 PM Ramesh Mani  wrote:

> +1 for Apache Atlas 2.1.0 rc2 release.
>
> - Build successfully with the source
>
> https://dist.apache.org/repos/dist/dev/atlas/2.1.0-rc2/apache-atlas-2.1.0-sources.tar.gz
> - Verified PGP signature.
> - Verified md5 and SHA512 digests.
>
> Thanks.
> Ramesh
>
> On Sat, Jul 4, 2020 at 11:50 PM Madhan Neethiraj 
> wrote:
>
> > Atlas team,
> >
> > Apache Atlas 2.1.0 rc2 is now available for vote within dev community.
> >
> > Links to the release artifacts are given below. Please review and vote.
> >
> > The vote will be open for at least 72 hours or until necessary votes are
> > reached.
> >   [ ] +1 Approve
> >   [ ] +0 No opinion
> >   [ ] -1 Disapprove (and reason why)
> >
> > Thanks,
> > Madhan
> >
> >
> > List of all issues addressed in this release:
> > https://issues.apache.org/jira/issues/?jql=project=ATLAS AND
> > status=Resolved AND fixVersion=2.1.0 ORDER BY key DESC
> >
> > Git tag for the release:
> > https://github.com/apache/atlas/tree/release-2.1.0-rc2
> > Sources for the release:
> >
> https://dist.apache.org/repos/dist/dev/atlas/2.1.0-rc2/apache-atlas-2.1.0-sources.tar.gz
> >
> > Source release verification:
> >   PGP Signature:
> >
> https://dist.apache.org/repos/dist/dev/atlas/2.1.0-rc2/apache-atlas-2.1.0-sources.tar.gz.asc
> >   SHA512 Hash:
> >
> https://dist.apache.org/repos/dist/dev/atlas/2.1.0-rc2/apache-atlas-2.1.0-sources.tar.gz.sha512
> >   MD5 Hash:
> >
> https://dist.apache.org/repos/dist/dev/atlas/2.1.0-rc2/apache-atlas-2.1.0-sources.tar.gz.md5
> >
> > Keys to verify the signature of the release artifacts are available at:
> > https://dist.apache.org/repos/dist/release/atlas/KEYS
> >
> > New features/enhancements:
> >   - Quick Search: provides a simpler search experience with type-ahead
> > suggestions
> >   - Business Metadata: enables augmenting entity-types with additional
> > attributes, search entities using these attributes
> >   - Labels: ability to add/remove labels on entities, and search entities
> > using labels
> >   - Custom Attributes: ability to add entity instance specific custom
> > attributes i.e. attributes not defined in entity-def or business metadata
> >   - Entity Purge: added REST APIs to purge deleted entities
> >
> > Enhancements:
> >   - Search: ability to find entities by more than one classification
> >   - Performance: improvements in lineage retrieval and
> > classification-propagation
> >   - Notification: ability to process notifications from multiple Kafka
> > topics
> >   - Hive Hook: tracks process-executions via hive_process_execution
> > entities
> >   - Hive Hook: captures DDL operations via hive_db_ddl and hive_table_ddl
> > entities
> >   - Notification: introduced shell entities to record references to
> > non-existing entities in notifications
> >   - Spark: added model to capture Spark entities, processes and
> > relationships
> >   - AWS S3: introduced updated model to capture AWS S3 entities and
> > relationships
> >   - ADLS-Gen2: introduced model to capture Azure Data Lake Storage Gen2
> > entities and relationships
> >   - Dependencies: JanusGraph 0.5.1, Tinkerpop 3.4.6, Spring Framework
> > 4.3.20
> >   - Authorization: updated to cover new features, like: business
> metadata,
> > labels, purge
> >   - UI: multiple UI improvements, including a beta UI
> >
> >
> >
>


-- 
Regards
Nikhil P. Bonte

Review Request 72652: ATLAS-3879 Ozone: ozone_key entity is directly created under ozone_bucket

[Nikhil Bonte]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72652/
---

Review request for atlas, Ashutosh Mestry, Jayendra Parab, madhan, Nixon 
Rodrigues, Sarath Subramanian, and Sidharth Mishra.


Repository: atlas


Description
---

The support for Apache Ozone was added in ATLAS-3836. Currently, ozone_key 
entity is getting created under ozone_bucket, but there is no support to create 
a hierarchy of parent ozone_key and child ozone_key.

e.g.

-> CREATE EXTERNAL TABLE sales (id int) row format delimited fields terminated 
by ' ' stored as textfile location 
'o3fs://bucket1.volume1.ozone1/sale1/q1/sales';

This query will create only 1 ozone_key directly under bucket "bucket1".

Created entities :

o3fs://volume1@cm (ozone_key)
o3fs://volume1.bucket1@cm (ozone_bucket)
o3fs://bucket1.volume1.ozone1/sale1/q1/sales@cm (ozone_key)

 

This Jira will improve implementation to support creating the hierarchy of 
ozone_keys.

With this, above query will result into following entites

o3fs://volume1@cm (ozone_key)
o3fs://volume1.bucket1@cm (ozone_bucket)
o3fs://bucket1.volume1.ozone1/sale1/@cm (ozone_key) child of ozone_bucket
o3fs://bucket1.volume1.ozone1/sale1/q1/@cm (ozone_key) child of ozone_key
o3fs://bucket1.volume1.ozone1/sale1/q1/sales/@cm (ozone_key) child of 
ozone_key


Diffs
-

  addons/models/3000-Cloud/3050-ozone-typedefs.json 0f0a5c90f 
  common/src/main/java/org/apache/atlas/utils/AtlasPathExtractorUtil.java 
c3276a891 
  common/src/test/java/org/apache/atlas/utils/AtlasPathExtractorUtilTest.java 
4abdca498 


Diff: https://reviews.apache.org/r/72652/diff/1/


Testing
---

UTs updated & verified.
Manually verified.
PC build:


Thanks,

Nikhil Bonte

Re: Review Request 72652: ATLAS-3879 Ozone: ozone_key entity is directly created under ozone_bucket

[Nikhil Bonte]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72652/
---

(Updated July 7, 2020, 3:29 p.m.)


Review request for atlas, Ashutosh Mestry, Jayendra Parab, madhan, Nixon 
Rodrigues, Sarath Subramanian, and Sidharth Mishra.


Changes
---

Added Pre-commit build link


Bugs: ATLAS-3879
https://issues.apache.org/jira/browse/ATLAS-3879


Repository: atlas


Description
---

The support for Apache Ozone was added in ATLAS-3836. Currently, ozone_key 
entity is getting created under ozone_bucket, but there is no support to create 
a hierarchy of parent ozone_key and child ozone_key.

e.g.

-> CREATE EXTERNAL TABLE sales (id int) row format delimited fields terminated 
by ' ' stored as textfile location 
'o3fs://bucket1.volume1.ozone1/sale1/q1/sales';

This query will create only 1 ozone_key directly under bucket "bucket1".

Created entities :

o3fs://volume1@cm (ozone_key)
o3fs://volume1.bucket1@cm (ozone_bucket)
o3fs://bucket1.volume1.ozone1/sale1/q1/sales@cm (ozone_key)

 

This Jira will improve implementation to support creating the hierarchy of 
ozone_keys.

With this, above query will result into following entites

o3fs://volume1@cm (ozone_key)
o3fs://volume1.bucket1@cm (ozone_bucket)
o3fs://bucket1.volume1.ozone1/sale1/@cm (ozone_key) child of ozone_bucket
o3fs://bucket1.volume1.ozone1/sale1/q1/@cm (ozone_key) child of ozone_key
o3fs://bucket1.volume1.ozone1/sale1/q1/sales/@cm (ozone_key) child of 
ozone_key


Diffs
-

  addons/models/3000-Cloud/3050-ozone-typedefs.json 0f0a5c90f 
  common/src/main/java/org/apache/atlas/utils/AtlasPathExtractorUtil.java 
c3276a891 
  common/src/test/java/org/apache/atlas/utils/AtlasPathExtractorUtilTest.java 
4abdca498 


Diff: https://reviews.apache.org/r/72652/diff/1/


Testing (updated)
---

UTs updated & verified.
Manually verified.

PC build: https://builds.apache.org/job/PreCommit-ATLAS-Build-Test/2026/console 
(IN PROGRESS)


Thanks,

Nikhil Bonte

Re: Review Request 72652: ATLAS-3879 Ozone: ozone_key entity is directly created under ozone_bucket

[Nikhil Bonte]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72652/
---

(Updated July 7, 2020, 5:43 p.m.)


Review request for atlas, Ashutosh Mestry, Jayendra Parab, madhan, Nixon 
Rodrigues, Sarath Subramanian, and Sidharth Mishra.


Changes
---

Addressed review comment - Removed trailing "/" from ozone_key qualifiedName


Bugs: ATLAS-3879
https://issues.apache.org/jira/browse/ATLAS-3879


Repository: atlas


Description
---

The support for Apache Ozone was added in ATLAS-3836. Currently, ozone_key 
entity is getting created under ozone_bucket, but there is no support to create 
a hierarchy of parent ozone_key and child ozone_key.

e.g.

-> CREATE EXTERNAL TABLE sales (id int) row format delimited fields terminated 
by ' ' stored as textfile location 
'o3fs://bucket1.volume1.ozone1/sale1/q1/sales';

This query will create only 1 ozone_key directly under bucket "bucket1".

Created entities :

o3fs://volume1@cm (ozone_key)
o3fs://volume1.bucket1@cm (ozone_bucket)
o3fs://bucket1.volume1.ozone1/sale1/q1/sales@cm (ozone_key)

 

This Jira will improve implementation to support creating the hierarchy of 
ozone_keys.

With this, above query will result into following entites

o3fs://volume1@cm (ozone_key)
o3fs://volume1.bucket1@cm (ozone_bucket)
o3fs://bucket1.volume1.ozone1/sale1/@cm (ozone_key) child of ozone_bucket
o3fs://bucket1.volume1.ozone1/sale1/q1/@cm (ozone_key) child of ozone_key
o3fs://bucket1.volume1.ozone1/sale1/q1/sales/@cm (ozone_key) child of 
ozone_key


Diffs (updated)
-

  addons/models/3000-Cloud/3050-ozone-typedefs.json 0f0a5c90f 
  common/src/main/java/org/apache/atlas/utils/AtlasPathExtractorUtil.java 
c3276a891 
  common/src/test/java/org/apache/atlas/utils/AtlasPathExtractorUtilTest.java 
4abdca498 


Diff: https://reviews.apache.org/r/72652/diff/2/

Changes: https://reviews.apache.org/r/72652/diff/1-2/


Testing
---

UTs updated & verified.
Manually verified.

PC build: https://builds.apache.org/job/PreCommit-ATLAS-Build-Test/2026/console 
(IN PROGRESS)


Thanks,

Nikhil Bonte

Re: Review Request 72666: Notification: Solution to Memory Build-up

[Nikhil Bonte]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72666/#review221192
---




intg/src/main/java/org/apache/atlas/utils/FixedBufferList.java
Lines 85 (patched)
<https://reviews.apache.org/r/72666/#comment309968>

Refactor to arrange methods such that public methods appear earlier to 
private methods.



webapp/src/main/java/org/apache/atlas/notification/EntityNotificationListenerV2.java
Lines 77 (patched)
<https://reviews.apache.org/r/72666/#comment309967>

NamedFixedBufferList: Does not have a mechanism to remove objects


- Nikhil Bonte


On July 9, 2020, 6:17 p.m., Ashutosh Mestry wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72666/
> ---
> 
> (Updated July 9, 2020, 6:17 p.m.)
> 
> 
> Review request for atlas, Madhan Neethiraj, Nikhil Bonte, Nixon Rodrigues, 
> and Sarath Subramanian.
> 
> 
> Bugs: ATLAS-3878
> https://issues.apache.org/jira/browse/ATLAS-3878
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> **Background**
> See JIRA for details.
> 
> *Analysis* Using memory profiling tools, it was observed that large number of 
> notification objects were created. These stayed in memory and later were 
> promoted to higher generation, thereby taking even longer to be collected.
> 
> **Approach**
> Using the fixed-buffer approach to address the problem of creating large 
> number of small objects.
> 
> New *FixedBufferList* This is an encapsulation over *ArrayList*. During 
> initial allocation, list is populated with default values. Features:
> - Setting of values to these pre-allocated objects is achieved by first doing 
> a *get* on the element and then assigning values to it.
> - *toList* fetches the sub-list from the encapsulating list. This uses the 
> state within the class to fetch the right length for the returning array.
> 
> New *NamedFixedBufferList* Maintains a per-thread *FixedBufferList*. This is 
> necessary since the list is now part class's state.
> Modified *EntityAuditListenerV2* Uses the new classes.
> Modifed *EntityNotificationListener* Uses the new classes.
> 
> **Verification**
> - Using the test setup, the memory usage was observed over a period of 24 
> hrs. 
> - Memory usage and object allocation was obvserved using memory profiler.
> 
> 
> Diffs
> -
> 
>   intg/src/main/java/org/apache/atlas/utils/FixedBufferList.java PRE-CREATION 
>   intg/src/main/java/org/apache/atlas/utils/NamedFixedBufferList.java 
> PRE-CREATION 
>   intg/src/test/java/org/apache/atlas/utils/FixedBufferListTest.java 
> PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/repository/audit/EntityAuditListenerV2.java
>  79527acfa 
>   
> webapp/src/main/java/org/apache/atlas/notification/EntityNotificationListenerV2.java
>  a677b315c 
> 
> 
> Diff: https://reviews.apache.org/r/72666/diff/2/
> 
> 
> Testing
> ---
> 
> **Unit testing**
> Unit tests added for the new classes.
> 
> **Volume testing**
> Setup:
> - Node: Threads 40, Core: 40, Allocated Memory: 12 GB
> - Multiple Kafka queues ingesting data.
> - Bulk entity creation using custom script ingesting 100M entities.
> 
> Memory usage stayed between 0 and 5% during the 24 hr period.
> 
> **Pre-commit**
> https://builds.apache.org/view/A/view/Atlas/job/PreCommit-ATLAS-Build-Test/2035/
> 
> 
> Thanks,
> 
> Ashutosh Mestry
> 
>

Re: [VOTE] Release Apache Atlas version 2.1.0 - rc3

[Nikhil Bonte]

+1 for Apache Atlas 2.1.0 - rc3 release

Validated the following items.

   - Verified PGP signature.
   - Verified md5 and SHA512 digests.
   - Packaged with embedded-hbase-solr
   - Verified entity creation & tag associations using quick_start.py
   - Tested basic, quick & advanced search.
   - Tested tag dissociation

Thanks
Nikhil P. Bonte

On Fri, Jul 10, 2020 at 10:29 AM Madhan Neethiraj  wrote:

> Atlas team,
>
> Apache Atlas 2.1.0 rc3, with following fixes since rc2, is now available
> for vote within dev community.
>   ATLAS-3877: Purged entity audit throws 404 Entity not found error
>   ATLAS-3882: Classic UI: Loader not disappearing while assigning category
> to term
>
> Links to the release artifacts are given below. Please review and vote.
>
> The vote will be open for at least 72 hours or until necessary votes are
> reached.
>   [ ] +1 Approve
>   [ ] +0 No opinion
>   [ ] -1 Disapprove (and reason why)
>
> Thanks,
> Madhan
>
>
> List of all issues addressed in this release:
> https://issues.apache.org/jira/issues/?jql=project=ATLAS AND
> status=Resolved AND fixVersion=2.1.0 ORDER BY key DESC
>
> Git tag for the release:
> https://github.com/apache/atlas/tree/release-2.1.0-rc3
> Sources for the release:
> https://dist.apache.org/repos/dist/dev/atlas/2.1.0-rc3/apache-atlas-2.1.0-sources.tar.gz
>
> Source release verification:
>   PGP Signature:
> https://dist.apache.org/repos/dist/dev/atlas/2.1.0-rc3/apache-atlas-2.1.0-sources.tar.gz.asc
>   SHA512 Hash:
> https://dist.apache.org/repos/dist/dev/atlas/2.1.0-rc3/apache-atlas-2.1.0-sources.tar.gz.sha512
>   MD5 Hash:
> https://dist.apache.org/repos/dist/dev/atlas/2.1.0-rc3/apache-atlas-2.1.0-sources.tar.gz.md5
>
> Keys to verify the signature of the release artifacts are available at:
> https://dist.apache.org/repos/dist/release/atlas/KEYS
>
> New features/enhancements:
>   - Quick Search: provides a simpler search experience with type-ahead
> suggestions
>   - Business Metadata: enables augmenting entity-types with additional
> attributes, search entities using these attributes
>   - Labels: ability to add/remove labels on entities, and search entities
> using labels
>   - Custom Attributes: ability to add entity instance specific custom
> attributes i.e. attributes not defined in entity-def or business metadata
>   - Entity Purge: added REST APIs to purge deleted entities
>
> Enhancements:
>   - Search: ability to find entities by more than one classification
>   - Performance: improvements in lineage retrieval and
> classification-propagation
>   - Notification: ability to process notifications from multiple Kafka
> topics
>   - Hive Hook: tracks process-executions via hive_process_execution
> entities
>   - Hive Hook: captures DDL operations via hive_db_ddl and hive_table_ddl
> entities
>   - Notification: introduced shell entities to record references to
> non-existing entities in notifications
>   - Spark: added model to capture Spark entities, processes and
> relationships
>   - AWS S3: introduced updated model to capture AWS S3 entities and
> relationships
>   - ADLS-Gen2: introduced model to capture Azure Data Lake Storage Gen2
> entities and relationships
>   - Dependencies: JanusGraph 0.5.1, Tinkerpop 3.4.6, Spring Framework
> 4.3.20
>   - Authorization: updated to cover new features, like: business metadata,
> labels, purge
>   - UI: multiple UI improvements, including a beta UI
>
>
>

-- 
Regards
Nikhil P. Bonte

Re: Review Request 72666: Notification: Solution to Memory Build-up

[Nikhil Bonte]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72666/#review221317
---


Ship it!




Ship It!

- Nikhil Bonte


On July 22, 2020, 4:16 p.m., Ashutosh Mestry wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72666/
> ---
> 
> (Updated July 22, 2020, 4:16 p.m.)
> 
> 
> Review request for atlas, Madhan Neethiraj, Nikhil Bonte, Nixon Rodrigues, 
> and Sarath Subramanian.
> 
> 
> Bugs: ATLAS-3878
> https://issues.apache.org/jira/browse/ATLAS-3878
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> **Background**
> See JIRA for details.
> 
> *Analysis* Using memory profiling tools, it was observed that large number of 
> notification objects were created. These stayed in memory and later were 
> promoted to higher generation, thereby taking even longer to be collected.
> 
> **Approach**
> Using the fixed-buffer approach to address the problem of creating large 
> number of small objects.
> 
> New *FixedBufferList* This is an encapsulation over *ArrayList*. During 
> initial allocation, list is populated with default values. Features:
> - Setting of values to these pre-allocated objects is achieved by first doing 
> a *get* on the element and then assigning values to it.
> - *toList* fetches the sub-list from the encapsulating list. This uses the 
> state within the class to fetch the right length for the returning array.
> 
> New *NamedFixedBufferList* Maintains a per-thread *FixedBufferList*. This is 
> necessary since the list is now part class's state.
> Modified *EntityAuditListenerV2* Uses the new classes.
> Modifed *EntityNotificationListener* Uses the new classes.
> 
> **Verification**
> - Using the test setup, the memory usage was observed over a period of 24 
> hrs. 
> - Memory usage and object allocation was obvserved using memory profiler.
> 
> 
> Diffs
> -
> 
>   intg/src/main/java/org/apache/atlas/AtlasConfiguration.java 2c007ca01 
>   intg/src/main/java/org/apache/atlas/utils/FixedBufferList.java PRE-CREATION 
>   intg/src/main/java/org/apache/atlas/utils/FixedBufferListAccessor.java 
> PRE-CREATION 
>   intg/src/test/java/org/apache/atlas/utils/FixedBufferListAccessorTest.java 
> PRE-CREATION 
>   intg/src/test/java/org/apache/atlas/utils/FixedBufferListTest.java 
> PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/repository/audit/EntityAuditListenerV2.java
>  79527acfa 
>   
> webapp/src/main/java/org/apache/atlas/notification/EntityNotificationListenerV2.java
>  a677b315c 
> 
> 
> Diff: https://reviews.apache.org/r/72666/diff/6/
> 
> 
> Testing
> ---
> 
> **Unit testing**
> Unit tests added for the new classes.
> 
> **Volume testing**
> Setup:
> - Node: Threads 40, Core: 40, Allocated Memory: 12 GB
> - Multiple Kafka queues ingesting data.
> - Bulk entity creation using custom script ingesting 100M entities.
> 
> Memory usage stayed between 0 and 5% during the 24 hr period.
> 
> With:
> - Workers: 64
> - Batch size: 50 (fewer elements in batch improve commit time and audit write 
> time).
> - Throughput: ~1.2 M entities per hour. Without out of memory error.
> 
> **Pre-commit**
> https://builds.apache.org/view/A/view/Atlas/job/PreCommit-ATLAS-Build-Test/2035/
> 
> 
> Thanks,
> 
> Ashutosh Mestry
> 
>

Re: [ANNOUNCE] New Committer: Nikhil Bonte

[Nikhil Bonte]

Thanks Madhan.

Thank you Apache Atlas PMC for allowing me to have committer privileges.
I will use them wisely.

Regards
Nikhil P Bonte

On Thu, Sep 17, 2020 at 6:19 AM Madhan Neethiraj  wrote:

> Atlas community,
>
>
>
> Apache Atlas PMC has asked Nikhil Bonte to become a committer and we are
> pleased to announce that he has accepted.
>
>
>
> Welcome Nikhil, great to have you on board.
>
>
>
> On behalf of Apache Atlas PMC,
>
>
>
> Madhan
>

Re: Review Request 72991: Export/Import: Add Support for BusinessMetadata Attributes

[Nikhil Bonte]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72991/#review222135
---




repository/src/main/java/org/apache/atlas/repository/impexp/ExportTypeProcessor.java
Lines 124 (patched)
<https://reviews.apache.org/r/72991/#comment311194>

suggestion: rename: 
addBusinessMetadataType -> addBusinessMetadataTypes


- Nikhil Bonte


On Oct. 27, 2020, 8:21 p.m., Ashutosh Mestry wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72991/
> ---
> 
> (Updated Oct. 27, 2020, 8:21 p.m.)
> 
> 
> Review request for atlas, Nikhil Bonte and Sarath Subramanian.
> 
> 
> Bugs: ATLAS-4006 and ATLAS-4007
> https://issues.apache.org/jira/browse/ATLAS-4006
> https://issues.apache.org/jira/browse/ATLAS-4007
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> **Background**
> Existing implementation does not support export and import of _Business 
> Metadata_ (BM) attributes. 
> 
> **Implementation**
> - Export: Updated existing flow to include detecting BM attributes and adding 
> the types to the exported payload.
> - Import: Updated _TypeAttributeDifference_ to include BM attribute detection 
> and update.
> 
> 
> Diffs
> -
> 
>   
> repository/src/main/java/org/apache/atlas/repository/impexp/ExportService.java
>  0491a853a 
>   
> repository/src/main/java/org/apache/atlas/repository/impexp/ExportTypeProcessor.java
>  664af9166 
>   
> repository/src/main/java/org/apache/atlas/repository/impexp/ImportTypeDefProcessor.java
>  b65807742 
>   
> repository/src/main/java/org/apache/atlas/repository/impexp/TypeAttributeDifference.java
>  0d78dbe57 
>   
> repository/src/test/java/org/apache/atlas/repository/impexp/TypeAttributeDifferenceTest.java
>  f9b7cd11f 
> 
> 
> Diff: https://reviews.apache.org/r/72991/diff/1/
> 
> 
> Testing
> ---
> 
> **Unit tests**
> New tests added.
> 
> **Functional testing**
> End-to-end tests. (Export-import-export).
> 
> **Pre-commit Build**
> https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/154/
> 
> 
> Thanks,
> 
> Ashutosh Mestry
> 
>

Re: Review Request 72991: Export/Import: Add Support for BusinessMetadata Attributes

[Nikhil Bonte]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72991/#review222136
---


Ship it!




Ship It!

- Nikhil Bonte


On Oct. 27, 2020, 8:21 p.m., Ashutosh Mestry wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72991/
> ---
> 
> (Updated Oct. 27, 2020, 8:21 p.m.)
> 
> 
> Review request for atlas, Nikhil Bonte and Sarath Subramanian.
> 
> 
> Bugs: ATLAS-4006 and ATLAS-4007
> https://issues.apache.org/jira/browse/ATLAS-4006
> https://issues.apache.org/jira/browse/ATLAS-4007
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> **Background**
> Existing implementation does not support export and import of _Business 
> Metadata_ (BM) attributes. 
> 
> **Implementation**
> - Export: Updated existing flow to include detecting BM attributes and adding 
> the types to the exported payload.
> - Import: Updated _TypeAttributeDifference_ to include BM attribute detection 
> and update.
> 
> 
> Diffs
> -
> 
>   
> repository/src/main/java/org/apache/atlas/repository/impexp/ExportService.java
>  0491a853a 
>   
> repository/src/main/java/org/apache/atlas/repository/impexp/ExportTypeProcessor.java
>  664af9166 
>   
> repository/src/main/java/org/apache/atlas/repository/impexp/ImportTypeDefProcessor.java
>  b65807742 
>   
> repository/src/main/java/org/apache/atlas/repository/impexp/TypeAttributeDifference.java
>  0d78dbe57 
>   
> repository/src/test/java/org/apache/atlas/repository/impexp/TypeAttributeDifferenceTest.java
>  f9b7cd11f 
> 
> 
> Diff: https://reviews.apache.org/r/72991/diff/1/
> 
> 
> Testing
> ---
> 
> **Unit tests**
> New tests added.
> 
> **Functional testing**
> End-to-end tests. (Export-import-export).
> 
> **Pre-commit Build**
> https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/154/
> 
> 
> Thanks,
> 
> Ashutosh Mestry
> 
>

Re: Review Request 72993: Cache getGuid and getStatus in GraphTransactionInterceptor

[Nikhil Bonte]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72993/
---

(Updated Oct. 28, 2020, 1:59 p.m.)


Review request for atlas, Ashutosh Mestry, Jayendra Parab, madhan, Nixon 
Rodrigues, Sarath Subramanian, and Sidharth Mishra.


Bugs: ATLAS-4008
https://issues.apache.org/jira/browse/ATLAS-4008


Repository: atlas


Description
---

Problem Statement:

GraphHelper.getGuid() & GraphHelper.getStatus() causes overhead when called 
multiple times in a same request.

 

Approach:

Implement a caching mechanism in GraphTransactionInterceptor to cache the 
following mappings:

vertexID -> guid

vertexID -> entityStatus


Diffs
-

  repository/src/main/java/org/apache/atlas/GraphTransactionInterceptor.java 
57e454a8e 
  repository/src/main/java/org/apache/atlas/repository/graph/GraphHelper.java 
ca46d0fdb 
  
repository/src/main/java/org/apache/atlas/repository/store/graph/v1/DeleteHandlerV1.java
 9360dd3c9 
  
repository/src/test/java/org/apache/atlas/repository/store/graph/v2/AtlasComplexAttributesTest.java
 ad5fa92bd 
  
repository/src/test/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityStoreV2Test.java
 b9cbef1b0 


Diff: https://reviews.apache.org/r/72993/diff/1/


Testing (updated)
---

PC build : 
https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/164/console
 - In progress


Thanks,

Nikhil Bonte

Re: Review Request 72993: Cache getGuid and getStatus in GraphTransactionInterceptor

[Nikhil Bonte]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72993/
---

(Updated Oct. 29, 2020, 9:34 a.m.)


Review request for atlas, Ashutosh Mestry, Jayendra Parab, madhan, Nixon 
Rodrigues, Sarath Subramanian, and Sidharth Mishra.


Changes
---

Addressed review comments.


Bugs: ATLAS-4008
https://issues.apache.org/jira/browse/ATLAS-4008


Repository: atlas


Description
---

Problem Statement:

GraphHelper.getGuid() & GraphHelper.getStatus() causes overhead when called 
multiple times in a same request.

 

Approach:

Implement a caching mechanism in GraphTransactionInterceptor to cache the 
following mappings:

vertexID -> guid

vertexID -> entityStatus


Diffs (updated)
-

  repository/src/main/java/org/apache/atlas/GraphTransactionInterceptor.java 
57e454a8e 
  repository/src/main/java/org/apache/atlas/repository/graph/GraphHelper.java 
ca46d0fdb 
  
repository/src/main/java/org/apache/atlas/repository/store/graph/v1/DeleteHandlerV1.java
 9360dd3c9 
  
repository/src/test/java/org/apache/atlas/repository/store/graph/v2/AtlasComplexAttributesTest.java
 ad5fa92bd 
  
repository/src/test/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityStoreV2Test.java
 b9cbef1b0 


Diff: https://reviews.apache.org/r/72993/diff/2/

Changes: https://reviews.apache.org/r/72993/diff/1-2/


Testing
---

PC build : 
https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/164/console
 - In progress


Thanks,

Nikhil Bonte

Re: Review Request 72993: Cache getGuid and getStatus in GraphTransactionInterceptor

[Nikhil Bonte]

> On Oct. 28, 2020, 4:11 p.m., Ashutosh Mestry wrote:
> > repository/src/test/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityStoreV2Test.java
> > Line 860 (original), 861 (patched)
> > <https://reviews.apache.org/r/72993/diff/1/?file=2241708#file2241708line861>
> >
> > fyi: Since we use different DI mechanisms in test and regular code, the 
> > annotation does not get applied, because of which the finally block within 
> > _GraphTransactionInterceptor_ does not get called, hence explict call.
> > 
> > The test executes fine since everything happens in same txn.

Thanks for the info & helping out to fix UTs.


- Nikhil


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72993/#review222139
-------


On Oct. 29, 2020, 9:34 a.m., Nikhil Bonte wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72993/
> ---
> 
> (Updated Oct. 29, 2020, 9:34 a.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Jayendra Parab, madhan, Nixon 
> Rodrigues, Sarath Subramanian, and Sidharth Mishra.
> 
> 
> Bugs: ATLAS-4008
> https://issues.apache.org/jira/browse/ATLAS-4008
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Problem Statement:
> 
> GraphHelper.getGuid() & GraphHelper.getStatus() causes overhead when called 
> multiple times in a same request.
> 
>  
> 
> Approach:
> 
> Implement a caching mechanism in GraphTransactionInterceptor to cache the 
> following mappings:
> 
> vertexID -> guid
> 
> vertexID -> entityStatus
> 
> 
> Diffs
> -
> 
>   repository/src/main/java/org/apache/atlas/GraphTransactionInterceptor.java 
> 57e454a8e 
>   repository/src/main/java/org/apache/atlas/repository/graph/GraphHelper.java 
> ca46d0fdb 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v1/DeleteHandlerV1.java
>  9360dd3c9 
>   
> repository/src/test/java/org/apache/atlas/repository/store/graph/v2/AtlasComplexAttributesTest.java
>  ad5fa92bd 
>   
> repository/src/test/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityStoreV2Test.java
>  b9cbef1b0 
> 
> 
> Diff: https://reviews.apache.org/r/72993/diff/2/
> 
> 
> Testing
> ---
> 
> PC build : 
> https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/164/console
>  - In progress
> 
> 
> Thanks,
> 
> Nikhil Bonte
> 
>

Re: Review Request 72991: Export/Import: Add Support for BusinessMetadata Attributes

[Nikhil Bonte]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72991/#review222187
---


Ship it!




Ship It!

- Nikhil Bonte


On Nov. 10, 2020, 12:21 a.m., Ashutosh Mestry wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72991/
> ---
> 
> (Updated Nov. 10, 2020, 12:21 a.m.)
> 
> 
> Review request for atlas, Nikhil Bonte and Sarath Subramanian.
> 
> 
> Bugs: ATLAS-4006 and ATLAS-4007
> https://issues.apache.org/jira/browse/ATLAS-4006
> https://issues.apache.org/jira/browse/ATLAS-4007
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> **Background**
> Existing implementation does not support export and import of _Business 
> Metadata_ (BM) attributes. 
> 
> **Implementation**
> - Export: Updated existing flow to include detecting BM attributes and adding 
> the types to the exported payload.
> - Import: Updated _TypeAttributeDifference_ to include BM attribute detection 
> and update.
> 
> 
> Diffs
> -
> 
>   
> repository/src/main/java/org/apache/atlas/repository/impexp/AtlasServerService.java
>  542106f36 
>   
> repository/src/main/java/org/apache/atlas/repository/impexp/ExportService.java
>  0491a853a 
>   
> repository/src/main/java/org/apache/atlas/repository/impexp/ExportTypeProcessor.java
>  664af9166 
>   
> repository/src/main/java/org/apache/atlas/repository/impexp/ImportTypeDefProcessor.java
>  b65807742 
>   
> repository/src/main/java/org/apache/atlas/repository/impexp/TypeAttributeDifference.java
>  0d78dbe57 
>   
> repository/src/test/java/org/apache/atlas/repository/impexp/TypeAttributeDifferenceTest.java
>  f9b7cd11f 
>   repository/src/test/resources/json/typesDef-bm.json PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/72991/diff/3/
> 
> 
> Testing
> ---
> 
> **Unit tests**
> New tests added.
> 
> **Functional testing**
> End-to-end tests. (Export-import-export).
> 
> **Pre-commit Build**
> https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/154/
> https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/179/
> 
> 
> File Attachments
> 
> 
> Stocks with BM
>   
> https://reviews.apache.org/media/uploaded/files/2020/10/29/8ea9511f-3b0e-4b27-96cb-e5f2f50ba75d__stocks-bm.zip
> 
> 
> Thanks,
> 
> Ashutosh Mestry
> 
>

Re: Review Request 73046: ATLAS-4055 Basic search doesn't work if the search text has special characters

[Nikhil Bonte]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73046/#review43
---


Ship it!




Ship It!

- Nikhil Bonte


On Nov. 26, 2020, 9:46 a.m., Jayendra Parab wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73046/
> ---
> 
> (Updated Nov. 26, 2020, 9:46 a.m.)
> 
> 
> Review request for atlas, Madhan Neethiraj, Nixon Rodrigues, and Sarath 
> Subramanian.
> 
> 
> Bugs: ATLAS-4055
> https://issues.apache.org/jira/browse/ATLAS-4055
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> If the search text has a special character (i.e, one of + - & | ! ( ) { } [ ] 
> ^ " ~ * ? : \ /), then we need to enclose the search text in double quotes. 
> This change is required in addition to the changes provided in ATLAS-3983
> 
> 
> Diffs
> -
> 
>   intg/src/main/java/org/apache/atlas/type/AtlasStructType.java 28d2e23c4 
> 
> 
> Diff: https://reviews.apache.org/r/73046/diff/2/
> 
> 
> Testing
> ---
> 
> - Created entities with all the special characters in the name and qualified 
> name attribute
> - Tried basic search with attributes to search the name and qualified name
> - The search result included the appropriate entity that was matching the 
> search criteria
> 
> 
> Thanks,
> 
> Jayendra Parab
> 
>

Re: Review Request 73057: ATLAS-3913 Swagger documentation for APIs

[Nikhil Bonte]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73057/
---

(Updated Dec. 2, 2020, 2:33 p.m.)


Review request for atlas, Ashutosh Mestry, Jayendra Parab, Madhan Neethiraj, 
Nixon Rodrigues, Sameer Shaikh, and Sarath Subramanian.


Changes
---

Updated reviewers details


Bugs: ATLAS-3913
https://issues.apache.org/jira/browse/ATLAS-3913


Repository: atlas


Description (updated)
---

Currently, swagger documentation is not available with packaging.
This change will allow atlas server deployment to host its REST API doc using 
Swagger.

## Approach:

* Update enuntiate plugin version from 2.11.1 to 2.13.2
* Download latest swagger-ui from 
https://github.com/swagger-api/swagger-ui/archive/v3.36.2.zip & extract.
* Move swagger-ui-3.36.2/dist/ to 
build-tools/src/main/resources/ui-dist
* Update swagger tag in enuntiate.xml
* Inside build-tools/src/main/resources/ui-dist :
* Create index.js
* Modify SwaggerUIBundle object in ui-dist/index.html to load index.js 
along with following:
1 modify url to load swagger.json
2 Add requestInterceptor
* Updated Header.js, UrlLinks.js, Header.html to add API documentation menu.


## How to navigate to the documentation page?

-> Once Atlas UI is loaded, goto Logged-in-User -> Help -> API Documentation 
(see attached screenshot for reference)


Diffs
-

  3party-licenses/normalize.css-LICENSE PRE-CREATION 
  3party-licenses/swagger-ui-LECENSE PRE-CREATION 
  build-tools/src/main/resources/enunciate.xml 8a9383a01 
  build-tools/src/main/resources/ui-dist/favicon-16x16.png PRE-CREATION 
  build-tools/src/main/resources/ui-dist/favicon-32x32.png PRE-CREATION 
  build-tools/src/main/resources/ui-dist/index.html PRE-CREATION 
  build-tools/src/main/resources/ui-dist/index.js PRE-CREATION 
  build-tools/src/main/resources/ui-dist/swagger-ui-bundle.js PRE-CREATION 
  build-tools/src/main/resources/ui-dist/swagger-ui-standalone-preset.js 
PRE-CREATION 
  build-tools/src/main/resources/ui-dist/swagger-ui.css PRE-CREATION 
  dashboardv2/public/js/templates/site/Header.html 97e124aaa 
  dashboardv2/public/js/utils/UrlLinks.js 6d3c07952 
  dashboardv2/public/js/views/site/Header.js 2d6b55d47 
  dashboardv3/public/js/templates/site/Header.html 6d5e6efac 
  dashboardv3/public/js/utils/UrlLinks.js 69e527442 
  dashboardv3/public/js/views/site/Header.js cce00c730 
  pom.xml ba9450c16 
  webapp/pom.xml fe5dc0bdb 


Diff: https://reviews.apache.org/r/73057/diff/2/


Testing (updated)
---

Tested few APIs (POST | GET| DELETE) manually.


File Attachments


How-to-navigate-to-the-documentation-page
  
https://reviews.apache.org/media/uploaded/files/2020/12/02/8fe22101-fbf8-4ab2-87d9-6bfee0949845__99964008-a62afe80-2db8-11eb-8ecd-cea617fe7f4a.png


Thanks,

Nikhil Bonte

Re: Review Request 73057: ATLAS-3913 Swagger documentation for APIs

[Nikhil Bonte]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73057/
---

(Updated Dec. 3, 2020, 12:29 p.m.)


Review request for atlas, Ashutosh Mestry, Jayendra Parab, Madhan Neethiraj, 
Nixon Rodrigues, Sameer Shaikh, and Sarath Subramanian.


Changes
---

* Disabled validation of swagger specs
* Changed the tab icon


Bugs: ATLAS-3913
https://issues.apache.org/jira/browse/ATLAS-3913


Repository: atlas


Description
---

Currently, swagger documentation is not available with packaging.
This change will allow atlas server deployment to host its REST API doc using 
Swagger.

## Approach:

* Update enuntiate plugin version from 2.11.1 to 2.13.2
* Download latest swagger-ui from 
https://github.com/swagger-api/swagger-ui/archive/v3.36.2.zip & extract.
* Move swagger-ui-3.36.2/dist/ to 
build-tools/src/main/resources/ui-dist
* Update swagger tag in enuntiate.xml
* Inside build-tools/src/main/resources/ui-dist :
* Create index.js
* Modify SwaggerUIBundle object in ui-dist/index.html to load index.js 
along with following:
1 modify url to load swagger.json
2 Add requestInterceptor
* Updated Header.js, UrlLinks.js, Header.html to add API documentation menu.


## How to navigate to the documentation page?

-> Once Atlas UI is loaded, goto Logged-in-User -> Help -> API Documentation 
(see attached screenshot for reference)


Diffs (updated)
-

  3party-licenses/normalize.css-LICENSE PRE-CREATION 
  3party-licenses/swagger-ui-LECENSE PRE-CREATION 
  build-tools/src/main/resources/enunciate.xml 8a9383a01 
  build-tools/src/main/resources/ui-dist/index.html PRE-CREATION 
  build-tools/src/main/resources/ui-dist/index.js PRE-CREATION 
  build-tools/src/main/resources/ui-dist/swagger-ui-bundle.js PRE-CREATION 
  build-tools/src/main/resources/ui-dist/swagger-ui-standalone-preset.js 
PRE-CREATION 
  build-tools/src/main/resources/ui-dist/swagger-ui.css PRE-CREATION 
  dashboardv2/public/js/templates/site/Header.html 97e124aaa 
  dashboardv2/public/js/utils/UrlLinks.js 6d3c07952 
  dashboardv2/public/js/views/site/Header.js 2d6b55d47 
  dashboardv3/public/js/templates/site/Header.html 6d5e6efac 
  dashboardv3/public/js/utils/UrlLinks.js 69e527442 
  dashboardv3/public/js/views/site/Header.js cce00c730 
  pom.xml ba9450c16 
  webapp/pom.xml fe5dc0bdb 


Diff: https://reviews.apache.org/r/73057/diff/3/

Changes: https://reviews.apache.org/r/73057/diff/2-3/


Testing
---

Tested few APIs (POST | GET| DELETE) manually.


File Attachments


How-to-navigate-to-the-documentation-page
  
https://reviews.apache.org/media/uploaded/files/2020/12/02/8fe22101-fbf8-4ab2-87d9-6bfee0949845__99964008-a62afe80-2db8-11eb-8ecd-cea617fe7f4a.png


Thanks,

Nikhil Bonte

[ANNOUNCE] Apache Atlas - Swagger REST API documentation

[Nikhil Bonte]

All,

The Swagger REST API documentation for Apache Atlas is now available.

It provides an interactive interface to try out all the REST APIs supported
by Atlas (types, entities, glossaries, lineages, relationships and search).

How to navigate to the API documentation page?
-> Once Atlas UI is loaded, Navigate to Logged-in-User -> Help -> API
Documentation (see attached screenshot for reference)

Apache Atlas will be updating its website to include details about this
REST API doc. Feedback and suggestions are welcome.


Thanks,
Nikhil P Bonte

Re: [ANNOUNCE] Apache Atlas - Swagger REST API documentation

[Nikhil Bonte]

Attaching screenshots...

On Mon, Dec 7, 2020 at 1:50 PM Nikhil Bonte  wrote:

> All,
>
> The Swagger REST API documentation for Apache Atlas is now available.
>
> It provides an interactive interface to try out all the REST APIs
> supported by Atlas (types, entities, glossaries, lineages, relationships
> and search).
>
> How to navigate to the API documentation page?
> -> Once Atlas UI is loaded, Navigate to Logged-in-User -> Help -> API
> Documentation (see attached screenshot for reference)
>
> Apache Atlas will be updating its website to include details about this
> REST API doc. Feedback and suggestions are welcome.
>
>
> Thanks,
> Nikhil P Bonte
>


-- 
Regards
Nikhil P. Bonte

Re: Review Request 73128: DSL: Using AtlasTraversal Instead of Gremlin

[Nikhil Bonte]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73128/#review222440
---




repository/src/main/java/org/apache/atlas/query/executors/TraversalBasedExecutor.java
Lines 80 (patched)
<https://reviews.apache.org/r/73128/#comment311497>

Remove TODO, this block gets executed. e.g. query :
hive_column groupby (name)


- Nikhil Bonte


On Jan. 11, 2021, 12:27 p.m., Ashutosh Mestry wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73128/
> ---
> 
> (Updated Jan. 11, 2021, 12:27 p.m.)
> 
> 
> Review request for atlas, Apoorv Naik, Nikhil Bonte, Nixon Rodrigues, Pinal 
> Shah, and Sarath Subramanian.
> 
> 
> Bugs: ATLAS-2932
> https://issues.apache.org/jira/browse/ATLAS-2932
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> **Approach**
> General Approach:
> - Ensure that Gremlin-based implementation continues to function. This will 
> ensure backward compatibility and will guarad against regressions or behavior 
> deviations, if any, are introduced with the new implementation.
> - Continue using existing _GremlinQueryComposer_. 
> - Translate the clauses to _AtlasGraphTraversal_.
> - Projects that were earlier handled in Gremlin query are now handled in a 
> separate Java class.
> 
> Details:
> - Modified: _GremlinQueryComposer_: Continue composing queries using existing 
> claues. Additions alone.
> - New: _DSLQueryExecutor_ Instantiates old or new executor based on flag.
> - Modifed: _EntityDiscoveryService_ uses the new exectors.
> - New _GremlinToTraversalTranslator_: Converts Gremlin clauses to 
> _AtlasGraphTraversal_.
> - Modified: _AtlasGraphTraversal_: Additional methods.
> - New: _SelectClauseProjections_: All queries with select operations.
> 
> **Credits**
> - Apoorv Naik (apoorvnaik): Original implementation.
> - Nikhil Bonte (nbonte): Continued on Apoorv's implementation and addressed 
> short-comings.
> 
> 
> Diffs
> -
> 
>   
> graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraph.java 
> c016f6340 
>   
> graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraphTraversal.java
>  881bb1e0f 
>   
> graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraph.java
>  0dd573b89 
>   
> graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraphTraversal.java
>  c33c4f4d0 
>   intg/src/main/java/org/apache/atlas/AtlasConfiguration.java ea9f26d47 
>   
> repository/src/main/java/org/apache/atlas/discovery/EntityDiscoveryService.java
>  01a6c303a 
>   repository/src/main/java/org/apache/atlas/query/AtlasDSL.java b8a744b35 
>   repository/src/main/java/org/apache/atlas/query/GremlinClause.java 
> 9704b0f9e 
>   repository/src/main/java/org/apache/atlas/query/GremlinClauseList.java 
> 9f30e4dc0 
>   repository/src/main/java/org/apache/atlas/query/GremlinQuery.java 531f7ae86 
>   repository/src/main/java/org/apache/atlas/query/GremlinQueryComposer.java 
> 249381059 
>   repository/src/main/java/org/apache/atlas/query/SelectClauseComposer.java 
> 969fcd2b1 
>   
> repository/src/main/java/org/apache/atlas/query/executors/DSLQueryExecutor.java
>  PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/query/executors/GremlinClauseToTraversalTranslator.java
>  PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/query/executors/ScriptEngineBasedExecutor.java
>  PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/query/executors/SelectClauseProjections.java
>  PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/query/executors/TraversalBasedExecutor.java
>  PRE-CREATION 
>   repository/src/test/java/org/apache/atlas/query/DSLQueriesTest.java 
> 3bb3b07bf 
> 
> 
> Diff: https://reviews.apache.org/r/73128/diff/2/
> 
> 
> Testing
> ---
> 
> **Unit Tests**
> - Additional tests.
> - _GremlinCompoerTest_ continues to validate the clause creation.
> - Improved validation for _minMaxCount_ tests.
> - Refactored: _TableValidator_ Validates output from a query with select 
> claues.
> 
> **Volume Tests**
> - Old implementation: 528 queries take over 10 mins to execute.
> - New implemeentations: 528 queries execute in 7 secs.
> 
> 
> Thanks,
> 
> Ashutosh Mestry
> 
>

Re: Review Request 73128: DSL: Using AtlasTraversal Instead of Gremlin

[Nikhil Bonte]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73128/#review222442
---




repository/src/test/java/org/apache/atlas/query/DSLQueriesTest.java
Lines 589-598 (original), 613-622 (patched)
<https://reviews.apache.org/r/73128/#comment311499>

enable & validate these queries


- Nikhil Bonte


On Jan. 11, 2021, 12:27 p.m., Ashutosh Mestry wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73128/
> ---
> 
> (Updated Jan. 11, 2021, 12:27 p.m.)
> 
> 
> Review request for atlas, Apoorv Naik, Nikhil Bonte, Nixon Rodrigues, Pinal 
> Shah, and Sarath Subramanian.
> 
> 
> Bugs: ATLAS-2932
> https://issues.apache.org/jira/browse/ATLAS-2932
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> **Approach**
> General Approach:
> - Ensure that Gremlin-based implementation continues to function. This will 
> ensure backward compatibility and will guarad against regressions or behavior 
> deviations, if any, are introduced with the new implementation.
> - Continue using existing _GremlinQueryComposer_. 
> - Translate the clauses to _AtlasGraphTraversal_.
> - Projects that were earlier handled in Gremlin query are now handled in a 
> separate Java class.
> 
> Details:
> - Modified: _GremlinQueryComposer_: Continue composing queries using existing 
> claues. Additions alone.
> - New: _DSLQueryExecutor_ Instantiates old or new executor based on flag.
> - Modifed: _EntityDiscoveryService_ uses the new exectors.
> - New _GremlinToTraversalTranslator_: Converts Gremlin clauses to 
> _AtlasGraphTraversal_.
> - Modified: _AtlasGraphTraversal_: Additional methods.
> - New: _SelectClauseProjections_: All queries with select operations.
> 
> **Credits**
> - Apoorv Naik (apoorvnaik): Original implementation.
> - Nikhil Bonte (nbonte): Continued on Apoorv's implementation and addressed 
> short-comings.
> 
> 
> Diffs
> -
> 
>   
> graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraph.java 
> c016f6340 
>   
> graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraphTraversal.java
>  881bb1e0f 
>   
> graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraph.java
>  0dd573b89 
>   
> graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraphTraversal.java
>  c33c4f4d0 
>   intg/src/main/java/org/apache/atlas/AtlasConfiguration.java ea9f26d47 
>   
> repository/src/main/java/org/apache/atlas/discovery/EntityDiscoveryService.java
>  01a6c303a 
>   repository/src/main/java/org/apache/atlas/query/AtlasDSL.java b8a744b35 
>   repository/src/main/java/org/apache/atlas/query/GremlinClause.java 
> 9704b0f9e 
>   repository/src/main/java/org/apache/atlas/query/GremlinClauseList.java 
> 9f30e4dc0 
>   repository/src/main/java/org/apache/atlas/query/GremlinQuery.java 531f7ae86 
>   repository/src/main/java/org/apache/atlas/query/GremlinQueryComposer.java 
> 249381059 
>   repository/src/main/java/org/apache/atlas/query/SelectClauseComposer.java 
> 969fcd2b1 
>   
> repository/src/main/java/org/apache/atlas/query/executors/DSLQueryExecutor.java
>  PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/query/executors/GremlinClauseToTraversalTranslator.java
>  PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/query/executors/ScriptEngineBasedExecutor.java
>  PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/query/executors/SelectClauseProjections.java
>  PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/query/executors/TraversalBasedExecutor.java
>  PRE-CREATION 
>   repository/src/test/java/org/apache/atlas/query/DSLQueriesTest.java 
> 3bb3b07bf 
> 
> 
> Diff: https://reviews.apache.org/r/73128/diff/2/
> 
> 
> Testing
> ---
> 
> **Unit Tests**
> - Additional tests.
> - _GremlinCompoerTest_ continues to validate the clause creation.
> - Improved validation for _minMaxCount_ tests.
> - Refactored: _TableValidator_ Validates output from a query with select 
> claues.
> 
> **Volume Tests**
> - Old implementation: 528 queries take over 10 mins to execute.
> - New implemeentations: 528 queries execute in 7 secs.
> 
> 
> Thanks,
> 
> Ashutosh Mestry
> 
>

Re: Review Request 73128: DSL: Using AtlasTraversal Instead of Gremlin

[Nikhil Bonte]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73128/#review222441
---




repository/src/main/java/org/apache/atlas/query/executors/TraversalBasedExecutor.java
Lines 83 (patched)
<https://reviews.apache.org/r/73128/#comment311498>

Need to handle limit & offset here.
Assume you have 30 columns
query: "Column groupby (name)" will return all the 30 columns
Same behaviour if limit is passed in query.


- Nikhil Bonte


On Jan. 11, 2021, 12:27 p.m., Ashutosh Mestry wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73128/
> ---
> 
> (Updated Jan. 11, 2021, 12:27 p.m.)
> 
> 
> Review request for atlas, Apoorv Naik, Nikhil Bonte, Nixon Rodrigues, Pinal 
> Shah, and Sarath Subramanian.
> 
> 
> Bugs: ATLAS-2932
> https://issues.apache.org/jira/browse/ATLAS-2932
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> **Approach**
> General Approach:
> - Ensure that Gremlin-based implementation continues to function. This will 
> ensure backward compatibility and will guarad against regressions or behavior 
> deviations, if any, are introduced with the new implementation.
> - Continue using existing _GremlinQueryComposer_. 
> - Translate the clauses to _AtlasGraphTraversal_.
> - Projects that were earlier handled in Gremlin query are now handled in a 
> separate Java class.
> 
> Details:
> - Modified: _GremlinQueryComposer_: Continue composing queries using existing 
> claues. Additions alone.
> - New: _DSLQueryExecutor_ Instantiates old or new executor based on flag.
> - Modifed: _EntityDiscoveryService_ uses the new exectors.
> - New _GremlinToTraversalTranslator_: Converts Gremlin clauses to 
> _AtlasGraphTraversal_.
> - Modified: _AtlasGraphTraversal_: Additional methods.
> - New: _SelectClauseProjections_: All queries with select operations.
> 
> **Credits**
> - Apoorv Naik (apoorvnaik): Original implementation.
> - Nikhil Bonte (nbonte): Continued on Apoorv's implementation and addressed 
> short-comings.
> 
> 
> Diffs
> -
> 
>   
> graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraph.java 
> c016f6340 
>   
> graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraphTraversal.java
>  881bb1e0f 
>   
> graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraph.java
>  0dd573b89 
>   
> graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraphTraversal.java
>  c33c4f4d0 
>   intg/src/main/java/org/apache/atlas/AtlasConfiguration.java ea9f26d47 
>   
> repository/src/main/java/org/apache/atlas/discovery/EntityDiscoveryService.java
>  01a6c303a 
>   repository/src/main/java/org/apache/atlas/query/AtlasDSL.java b8a744b35 
>   repository/src/main/java/org/apache/atlas/query/GremlinClause.java 
> 9704b0f9e 
>   repository/src/main/java/org/apache/atlas/query/GremlinClauseList.java 
> 9f30e4dc0 
>   repository/src/main/java/org/apache/atlas/query/GremlinQuery.java 531f7ae86 
>   repository/src/main/java/org/apache/atlas/query/GremlinQueryComposer.java 
> 249381059 
>   repository/src/main/java/org/apache/atlas/query/SelectClauseComposer.java 
> 969fcd2b1 
>   
> repository/src/main/java/org/apache/atlas/query/executors/DSLQueryExecutor.java
>  PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/query/executors/GremlinClauseToTraversalTranslator.java
>  PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/query/executors/ScriptEngineBasedExecutor.java
>  PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/query/executors/SelectClauseProjections.java
>  PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/query/executors/TraversalBasedExecutor.java
>  PRE-CREATION 
>   repository/src/test/java/org/apache/atlas/query/DSLQueriesTest.java 
> 3bb3b07bf 
> 
> 
> Diff: https://reviews.apache.org/r/73128/diff/2/
> 
> 
> Testing
> ---
> 
> **Unit Tests**
> - Additional tests.
> - _GremlinCompoerTest_ continues to validate the clause creation.
> - Improved validation for _minMaxCount_ tests.
> - Refactored: _TableValidator_ Validates output from a query with select 
> claues.
> 
> **Volume Tests**
> - Old implementation: 528 queries take over 10 mins to execute.
> - New implemeentations: 528 queries execute in 7 secs.
> 
> 
> Thanks,
> 
> Ashutosh Mestry
> 
>

Re: Review Request 73166: Clause Change for Glossary

[Nikhil Bonte]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73166/#review222543
---


Ship it!




Ship It!

- Nikhil Bonte


On Feb. 1, 2021, 10:50 p.m., Ashutosh Mestry wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73166/
> ---
> 
> (Updated Feb. 1, 2021, 10:50 p.m.)
> 
> 
> Review request for atlas, Nikhil Bonte, Pinal Shah, and Sarath Subramanian.
> 
> 
> Bugs: ATLAS-4109
> https://issues.apache.org/jira/browse/ATLAS-4109
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Replaced _or_ with _where_ for simplicity.
> 
> 
> Diffs
> -
> 
>   repository/src/main/java/org/apache/atlas/query/GremlinClause.java 
> b7e5330ce 
>   
> repository/src/main/java/org/apache/atlas/query/executors/GremlinClauseToTraversalTranslator.java
>  a7b05fd88 
>   
> repository/src/test/java/org/apache/atlas/query/GremlinQueryComposerTest.java 
> 0e8af186d 
> 
> 
> Diff: https://reviews.apache.org/r/73166/diff/1/
> 
> 
> Testing
> ---
> 
> **Unit tests**
> - Existing tests run.
> - Updated _GremlinQueryComposer_ to include the new clause.
> 
> **PC-build**
> https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/363/
> 
> 
> Thanks,
> 
> Ashutosh Mestry
> 
>

Re: Review Request 73205: ATLAS-4150 : __AtlasAuditEntry appearing in metrics window

[Nikhil Bonte]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73205/#review222640
---




0001-ATLAS-4150.patch
Lines 1 (patched)
<https://reviews.apache.org/r/73205/#comment311796>

This file should not be the part of patch.
0001-ATLAS-4150.patch


- Nikhil Bonte


On March 1, 2021, 9:58 a.m., Shraddha Pardeshi wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73205/
> ---
> 
> (Updated March 1, 2021, 9:58 a.m.)
> 
> 
> Review request for atlas, Jayendra Parab and Nixon Rodrigues.
> 
> 
> Bugs: ATLAS-4150
> https://issues.apache.org/jira/browse/ATLAS-4150
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> __Approach__:
> - Entries for AtlasAuditEntry are visible on metrics UI. 
> - But it is an system internal entity and should ideally not be visible in 
> the metrics UI.
> - So here skipping internal type from UI.
> 
> 
> Diffs
> -
> 
>   0001-ATLAS-4150.patch PRE-CREATION 
>   repository/src/main/java/org/apache/atlas/services/MetricsService.java 
> e721108d3 
>   repository/src/test/java/org/apache/atlas/services/MetricsServiceTest.java 
> bea8eb8b5 
> 
> 
> Diff: https://reviews.apache.org/r/73205/diff/1/
> 
> 
> Testing
> ---
> 
> __Unit tests__
> - Changes made in existing test.
> 
> __Manual tests__
> - Via web UI.
> 
> 
> Thanks,
> 
> Shraddha Pardeshi
> 
>

Re: Review Request 73239: HiveHook: HiveServer2 Messages Improvement

[Nikhil Bonte]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73239/#review222803
---


Ship it!





addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/utils/HiveDDLEntityFilter.java
Lines 246 (patched)
<https://reviews.apache.org/r/73239/#comment311983>

remove extra blank line



addons/hive-bridge/src/test/java/org/apache/atlas/hive/hook/utils/ActiveEntityFilterTest.java
Lines 204 (patched)
<https://reviews.apache.org/r/73239/#comment311984>

remove extra blank line



addons/hive-bridge/src/test/resources/atlas-application.properties
Lines 125 (patched)
<https://reviews.apache.org/r/73239/#comment311982>

Suggesion: rename 
atlas.hook.hive.ignore.ddl.operations  ->  
atlas.hook.hive.hs2.ignore.ddl.operations


- Nikhil Bonte


On April 10, 2021, 5:11 a.m., Ashutosh Mestry wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73239/
> ---
> 
> (Updated April 10, 2021, 5:11 a.m.)
> 
> 
> Review request for atlas, Madhan Neethiraj and Sarath Subramanian.
> 
> 
> Bugs: ATLAS-4204
> https://issues.apache.org/jira/browse/ATLAS-4204
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> **Background**
> Please see JIRA.
> 
> Terms and abbreviations used:
> - Database objects: Database, table, views, etc.
> - DDL: Data Definition Langugage. Database parlance. Operations performed on 
> a database server that result in creation of objects that hold data.
> - DML: Data Manipulation Language. Database parlance. Operations performed on 
> a database server that result in manipulation (add, update, delete) of data. 
> These are performed using database objects that are already created.
> 
> Hive hook is implemented by:
> - _HiveHook_
> - _HiveMetastoreImpl_
> 
> - _BaseHiveEvent_ has 2 methods:
> - getHiveMetastoreEntities: Called by HMS events.
> - getHiveEntities: Called by HS2 events.
> 
> **Approach**
> - Introduce a flag _skipDDL_ indicating if entities generated as part of DDL 
> operation should be skipped by a hook. This flag is set as a configuration 
> parameters: atlas.hive.hook.ignore.ddl.operations
> - Continue using HMS event processing unchanged.
> - New: _ActiveEntityFilter_: Initializes entity filter based on the skipDDL 
> setting.
> - New: _LineageOnlyFilter_: Retains entities that participate in lineage.
> - New: _PassthroughFilter_: Does not do any filtering.
> 
> 
> Diffs
> -
> 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/AtlasHiveHookContext.java
>  128647147 
>   addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java 
> 79e87c79d 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/events/BaseHiveEvent.java
>  7c269ce53 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/utils/ActiveEntityFilter.java
>  PRE-CREATION 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/utils/EntityFilter.java
>  PRE-CREATION 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/utils/HiveDDLEntityFilter.java
>  PRE-CREATION 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/utils/PassthroughFilter.java
>  PRE-CREATION 
>   
> addons/hive-bridge/src/test/java/org/apache/atlas/hive/hook/utils/ActiveEntityFilterTest.java
>  PRE-CREATION 
>   addons/hive-bridge/src/test/resources/atlas-application.properties 
> 898b69c99 
>   addons/hive-bridge/src/test/resources/json/hs2-alter-view-v2.json 
> PRE-CREATION 
>   addons/hive-bridge/src/test/resources/json/hs2-alter-view.json PRE-CREATION 
>   addons/hive-bridge/src/test/resources/json/hs2-create-db-v2.json 
> PRE-CREATION 
>   addons/hive-bridge/src/test/resources/json/hs2-create-db.json PRE-CREATION 
>   addons/hive-bridge/src/test/resources/json/hs2-create-process-v2.json 
> PRE-CREATION 
>   addons/hive-bridge/src/test/resources/json/hs2-create-process.json 
> PRE-CREATION 
>   addons/hive-bridge/src/test/resources/json/hs2-create-table-v2.json 
> PRE-CREATION 
>   addons/hive-bridge/src/test/resources/json/hs2-create-table.json 
> PRE-CREATION 
>   addons/hive-bridge/src/test/resources/json/hs2-drop-db-v2.json PRE-CREATION 
>   addons/hive-bridge/src/test/resources/json/hs2-drop-db.json PRE-CREATION 
>   addons/hive-bridge/src/test/resources/json/hs2-drop-table-v2.json 
> PRE-CREATION 
>   addons/hive-bridge/src/test/resources/json/hs2-drop-table.json PRE-CREATION 
>   addons/hive-br

Re: Review Request 73239: HiveHook: HiveServer2 Messages Improvement

[Nikhil Bonte]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73239/#review222808
---




addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/utils/ActiveEntityFilter.java
Lines 38 (patched)
<https://reviews.apache.org/r/73239/#comment311990>

update info log
atlas.hook.hive.ignore.ddl.operations  ->  
atlas.hook.hive.hs2.ignore.ddl.operations



addons/hive-bridge/src/test/resources/atlas-application.properties
Lines 125 (patched)
<https://reviews.apache.org/r/73239/#comment311991>

change property key
atlas.hook.hive.ignore.ddl.operations  ->  
atlas.hook.hive.hs2.ignore.ddl.operations


- Nikhil Bonte


On April 12, 2021, 10:01 p.m., Ashutosh Mestry wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73239/
> ---
> 
> (Updated April 12, 2021, 10:01 p.m.)
> 
> 
> Review request for atlas, Madhan Neethiraj and Sarath Subramanian.
> 
> 
> Bugs: ATLAS-4204
> https://issues.apache.org/jira/browse/ATLAS-4204
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> **Background**
> Please see JIRA.
> 
> Terms and abbreviations used:
> - Database objects: Database, table, views, etc.
> - DDL: Data Definition Langugage. Database parlance. Operations performed on 
> a database server that result in creation of objects that hold data.
> - DML: Data Manipulation Language. Database parlance. Operations performed on 
> a database server that result in manipulation (add, update, delete) of data. 
> These are performed using database objects that are already created.
> 
> Hive hook is implemented by:
> - _HiveHook_
> - _HiveMetastoreImpl_
> 
> - _BaseHiveEvent_ has 2 methods:
> - getHiveMetastoreEntities: Called by HMS events.
> - getHiveEntities: Called by HS2 events.
> 
> **Approach**
> - Introduce a flag _skipDDL_ indicating if entities generated as part of DDL 
> operation should be skipped by a hook. This flag is set as a configuration 
> parameters: atlas.hive.hook.ignore.ddl.operations
> - Continue using HMS event processing unchanged.
> - New: _ActiveEntityFilter_: Initializes entity filter based on the skipDDL 
> setting.
> - New: _LineageOnlyFilter_: Retains entities that participate in lineage.
> - New: _PassthroughFilter_: Does not do any filtering.
> 
> 
> Diffs
> -
> 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/AtlasHiveHookContext.java
>  128647147 
>   addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/HiveHook.java 
> 79e87c79d 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/events/BaseHiveEvent.java
>  7c269ce53 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/utils/ActiveEntityFilter.java
>  PRE-CREATION 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/utils/EntityFilter.java
>  PRE-CREATION 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/utils/HiveDDLEntityFilter.java
>  PRE-CREATION 
>   
> addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/utils/PassthroughFilter.java
>  PRE-CREATION 
>   
> addons/hive-bridge/src/test/java/org/apache/atlas/hive/hook/utils/ActiveEntityFilterTest.java
>  PRE-CREATION 
>   addons/hive-bridge/src/test/resources/atlas-application.properties 
> 898b69c99 
>   addons/hive-bridge/src/test/resources/json/hs2-alter-view-v2.json 
> PRE-CREATION 
>   addons/hive-bridge/src/test/resources/json/hs2-alter-view.json PRE-CREATION 
>   addons/hive-bridge/src/test/resources/json/hs2-create-db-v2.json 
> PRE-CREATION 
>   addons/hive-bridge/src/test/resources/json/hs2-create-db.json PRE-CREATION 
>   addons/hive-bridge/src/test/resources/json/hs2-create-process-v2.json 
> PRE-CREATION 
>   addons/hive-bridge/src/test/resources/json/hs2-create-process.json 
> PRE-CREATION 
>   addons/hive-bridge/src/test/resources/json/hs2-create-table-v2.json 
> PRE-CREATION 
>   addons/hive-bridge/src/test/resources/json/hs2-create-table.json 
> PRE-CREATION 
>   addons/hive-bridge/src/test/resources/json/hs2-drop-db-v2.json PRE-CREATION 
>   addons/hive-bridge/src/test/resources/json/hs2-drop-db.json PRE-CREATION 
>   addons/hive-bridge/src/test/resources/json/hs2-drop-table-v2.json 
> PRE-CREATION 
>   addons/hive-bridge/src/test/resources/json/hs2-drop-table.json PRE-CREATION 
>   addons/hive-bridge/src/test/resources/json/hs2-table-rename-v2.json 
> PRE-CREATION 
>   addons/hive-bridge/src/test/resources/json/hs2-

Re: Review Request 73258: ATLAS-4195: [Atlas: Audits] 2 entries are created in audits when "replace columns" command is executed in hive

[Nikhil Bonte]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73258/#review222825
---



Started a pre-commit - 
https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/497/

- Nikhil Bonte


On March 29, 2021, 11:29 p.m., Deep Singh wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73258/
> ---
> 
> (Updated March 29, 2021, 11:29 p.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Madhan Neethiraj, and Sarath 
> Subramanian.
> 
> 
> Bugs: ATLAS-4195
> https://issues.apache.org/jira/browse/ATLAS-4195
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> When columns for certain table are deleted or replaced, in graph we mark the 
> table relationship with columns(replaced or deleted) as inactive. When shuch 
> a table's definition is sent by Hivehook in some event(may be as refered 
> entity), an update audit is created for the table (irrespective of 
> differential audit feature being enableld or disabled). 
> 
> This is because in entity comparator, where we compare relationships of 
> stored entity with the relationships of incoming refered table entity, we do 
> not ignore the inactive relationships of stored entity. This was because 
> mapVertexToArray method of EntityGraphRetriever did not have the capability 
> to ignore AtlasEdges having inactive status.
> 
> 
> Diffs
> -
> 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityComparator.java
>  d962385c2 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphRetriever.java
>  8208d11c2 
> 
> 
> Diff: https://reviews.apache.org/r/73258/diff/1/
> 
> 
> Testing
> ---
> 
> Manual testing was done.
> 
> 
> Thanks,
> 
> Deep Singh
> 
>

Re: Review Request 73258: ATLAS-4195: [Atlas: Audits] 2 entries are created in audits when "replace columns" command is executed in hive

[Nikhil Bonte]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73258/#review222842
---


Ship it!




Code looks good to me.
Going ahead, I think it will be good to have UTs around this scenario.

- Nikhil Bonte


On March 29, 2021, 11:29 p.m., Deep Singh wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73258/
> ---
> 
> (Updated March 29, 2021, 11:29 p.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Madhan Neethiraj, and Sarath 
> Subramanian.
> 
> 
> Bugs: ATLAS-4195
> https://issues.apache.org/jira/browse/ATLAS-4195
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> When columns for certain table are deleted or replaced, in graph we mark the 
> table relationship with columns(replaced or deleted) as inactive. When shuch 
> a table's definition is sent by Hivehook in some event(may be as refered 
> entity), an update audit is created for the table (irrespective of 
> differential audit feature being enableld or disabled). 
> 
> This is because in entity comparator, where we compare relationships of 
> stored entity with the relationships of incoming refered table entity, we do 
> not ignore the inactive relationships of stored entity. This was because 
> mapVertexToArray method of EntityGraphRetriever did not have the capability 
> to ignore AtlasEdges having inactive status.
> 
> 
> Diffs
> -
> 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityComparator.java
>  d962385c2 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphRetriever.java
>  8208d11c2 
> 
> 
> Diff: https://reviews.apache.org/r/73258/diff/1/
> 
> 
> Testing
> ---
> 
> Manual testing was done.
> 
> 
> Thanks,
> 
> Deep Singh
> 
>

Re: Review Request 73290: ATLAS-4251: Upgrade luceneMatchVersion in Solr configs

[Nikhil Bonte]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73290/
---

(Updated April 21, 2021, 8:35 p.m.)


Review request for atlas, Ashutosh Mestry, Jayendra Parab, madhan, Nixon 
Rodrigues, Pinal Shah, and Sarath Subramanian.


Bugs: ATLAS-4251
https://issues.apache.org/jira/browse/ATLAS-4251


Repository: atlas


Description
---

Altas' Solr config currently has luceneMatchVersion as 5.0.0

This improvement will upgrade it to 8.4.1


Diffs
-

  dev-support/atlas-docker/config/solr/solrconfig.xml 21d19ef19 
  distro/src/conf/solr/solrconfig.xml 21d19ef19 
  repository/src/test/resources/solr/core-template/solrconfig.xml 9264f99d4 
  test-tools/src/main/resources/solr/core-template/solrconfig.xml 977ebc49c 


Diff: https://reviews.apache.org/r/73290/diff/1/


Testing (updated)
---

Manually Tested

Pre-commit build - 
https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/511/consoleFull
 (IN PROGRESS)


Thanks,

Nikhil Bonte

Re: Review Request 73258: ATLAS-4195: [Atlas: Audits] 2 entries are created in audits when "replace columns" command is executed in hive

[Nikhil Bonte]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73258/#review222847
---


Fix it, then Ship it!





repository/src/test/java/org/apache/atlas/repository/store/graph/v2/DifferentialAuditsTest.java
Lines 47 (patched)
<https://reviews.apache.org/r/73258/#comment312009>

DSLQueriesTest.class -> DifferentialAuditsTest.class


- Nikhil Bonte


On April 22, 2021, 9:10 a.m., Deep Singh wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73258/
> ---
> 
> (Updated April 22, 2021, 9:10 a.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Madhan Neethiraj, Nikhil Bonte, 
> and Sarath Subramanian.
> 
> 
> Bugs: ATLAS-4195
> https://issues.apache.org/jira/browse/ATLAS-4195
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> When columns for certain table are deleted or replaced, in graph we mark the 
> table relationship with columns(replaced or deleted) as inactive. When shuch 
> a table's definition is sent by Hivehook in some event(may be as refered 
> entity), an update audit is created for the table (irrespective of 
> differential audit feature being enableld or disabled). 
> 
> This is because in entity comparator, where we compare relationships of 
> stored entity with the relationships of incoming refered table entity, we do 
> not ignore the inactive relationships of stored entity. This was because 
> mapVertexToArray method of EntityGraphRetriever did not have the capability 
> to ignore AtlasEdges having inactive status.
> 
> 
> Diffs
> -
> 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityStoreV2.java
>  c13392000 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphRetriever.java
>  b79002334 
>   
> repository/src/test/java/org/apache/atlas/repository/store/graph/v2/DifferentialAuditsTest.java
>  PRE-CREATION 
>   repository/src/test/resources/json/entities/db01.json PRE-CREATION 
>   repository/src/test/resources/json/entities/tbl01-2cols.json PRE-CREATION 
>   repository/src/test/resources/json/entities/tbl01.json PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/73258/diff/2/
> 
> 
> Testing
> ---
> 
> Manual testing was done.
> Unit test Added.
> 
> 
> Thanks,
> 
> Deep Singh
> 
>

Re: Review Request 73302: ATLAS-4255 : Atlas - Upgrade netty to 4.1.61

[Nikhil Bonte]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73302/#review222875
---


Ship it!




Ship It!

- Nikhil Bonte


On April 26, 2021, 11:39 p.m., chaitali wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73302/
> ---
> 
> (Updated April 26, 2021, 11:39 p.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Jayendra Parab, Madhan Neethiraj, 
> Nixon Rodrigues, and Sarath Subramanian.
> 
> 
> Bugs: ATLAS-4255
> https://issues.apache.org/jira/browse/ATLAS-4255
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Currently  Atlas is pulling netty jar versions 4.1.47,4.1.51,4.1.23,4.1.48 
> 
> Upgrade it to latest 4.1.61-final
> Introduced a dependency as netty-bom which can take care of all transitive 
> dependencies for io.netty atlas is pulling
> 
> 
> Diffs
> -
> 
>   pom.xml 092ba09ce 
> 
> 
> Diff: https://reviews.apache.org/r/73302/diff/1/
> 
> 
> Testing
> ---
> 
> Atlas server up and running
> Created entities,glossaries and tags
> quickstart.py successful
> PC : 
> https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/522/console
> 
> 
> Thanks,
> 
> chaitali
> 
>

Re: Review Request 73329: Correlating Deleted Entities with Lineage

[Nikhil Bonte]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73329/#review223027
---


Ship it!




Ship It!

- Nikhil Bonte


On May 21, 2021, 3:48 a.m., Ashutosh Mestry wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73329/
> ---
> 
> (Updated May 21, 2021, 3:48 a.m.)
> 
> 
> Review request for atlas, Radhika Kundam and Sarath Subramanian.
> 
> 
> Bugs: ATLAS-4152
> https://issues.apache.org/jira/browse/ATLAS-4152
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> **Background**
> As part of ATLAS-4204, HS2 notifications send entity-lineage only (provided 
> the poperty is enabled).
> 
> When Spooling enabled the order of messages can potentially change. The 
> notification messages coming from HS2 and HMS may not be in the same order as 
> when they arrived with direct notification.
> 
> Problem: 
> Consider the sequence of arriving messages:
> 
> This is the sequence of messages for Entity 1 (C = create, U = update, D = 
> delete, L?x = Lineage of type 'x')
> No problem: C1, U1, L1x, L1y, D1
> Problem: C1, U1, D1, L1x, L1y
> 
> This implementation attempts to handle ths problem mentioned above. If the 
> above case is not handled, it will end up creating shell entities, since 
> deleted entities are not looked up as part of entity creation.
> 
> **Approach**
> Used bounded stream approach where an incoming stream of messages is bounded 
> with an indicator that it originates from spool. This helps makes localized 
> decisions on the incoming stream of messages.
> 
> High-level approach:
> - Messages when written to the spool are tagged with a timestamp.
> - Deleted entities are maintained in a cache.
> - Lineage-only message are checked if they refer to a deleted entity.
> - If they refer to deleted entity, they are stitched to the one present in 
> the cache only if it falls within the threshold.
> - Using step-climbing approach for locating right entity to stitch lineage to.
> 
> New: _EntityCorrelationsManager_: Uses message timestamp and cached entity 
> qualifiedName-GUID map.
> Modifed: _NotificationHookConsumer_ Uses the new class.
> New: _HiveDDLLineagePreprocess_ Uses entity-correlation to link to deleted 
> entities.
> Modified: _SpoolConfiguration_: Added new configuration to pause message 
> sending after destination is available: 
> _atlas.hook.spool.pause.before.send.sec_.
> In-memory lookup approach changed to persistent loookup.
> 
> 
> Diffs
> -
> 
>   common/src/main/java/org/apache/atlas/repository/Constants.java ffcec9743 
>   
> intg/src/main/java/org/apache/atlas/model/notification/AtlasNotificationMessage.java
>  810ba97c9 
>   notification/src/main/java/org/apache/atlas/kafka/AtlasKafkaConsumer.java 
> f7d9668ec 
>   notification/src/main/java/org/apache/atlas/kafka/AtlasKafkaMessage.java 
> 22bd79fdf 
>   notification/src/main/java/org/apache/atlas/kafka/KafkaNotification.java 
> 3d1b3ccf1 
>   
> notification/src/main/java/org/apache/atlas/notification/AtlasNotificationMessageDeserializer.java
>  3264e264c 
>   
> notification/src/main/java/org/apache/atlas/notification/NotificationInterface.java
>  edd8ed931 
>   
> notification/src/main/java/org/apache/atlas/notification/spool/AtlasFileSpool.java
>  2d7d19595 
>   
> notification/src/main/java/org/apache/atlas/notification/spool/Publisher.java 
> 22242c933 
>   
> notification/src/main/java/org/apache/atlas/notification/spool/SpoolConfiguration.java
>  a9a3a78cc 
>   notification/src/main/java/org/apache/atlas/notification/spool/Spooler.java 
> 2cacaaadc 
>   
> notification/src/test/java/org/apache/atlas/notification/AbstractNotificationTest.java
>  d7e4959f7 
>   
> notification/src/test/java/org/apache/atlas/notification/spool/AtlasFileSpoolTest.java
>  167efbecc 
>   
> repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedSearchIndexer.java
>  cc727c6ba 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/EntityCorrelationStore.java
>  PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasGraphUtilsV2.java
>  0a9470815 
>   
> repository/src/test/java/org/apache/atlas/repository/store/graph/v2/EntityCorrelationStoreTest.java
>  PRE-CREATION 
>   
> webapp/src/main/java/org/apache/atlas/notification/EntityCorrelationManager.java
>  PRE-CREATION 
>   
&

Re: Review Request 73375: Handle Spool Directory Creation by Different Users

[Nikhil Bonte]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73375/#review223071
---


Ship it!




Ship It!

- Nikhil Bonte


On May 26, 2021, 3:12 a.m., Ashutosh Mestry wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73375/
> ---
> 
> (Updated May 26, 2021, 3:12 a.m.)
> 
> 
> Review request for atlas, Nikhil Bonte, Nixon Rodrigues, and Sarath 
> Subramanian.
> 
> 
> Bugs: ATLAS-4306
> https://issues.apache.org/jira/browse/ATLAS-4306
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Please see bug description.
> 
> 
> Diffs
> -
> 
>   
> notification/src/main/java/org/apache/atlas/notification/spool/AtlasFileSpool.java
>  ea3128465 
>   
> notification/src/main/java/org/apache/atlas/notification/spool/IndexManagement.java
>  f018983a4 
>   
> notification/src/main/java/org/apache/atlas/notification/spool/SpoolConfiguration.java
>  76f05efea 
>   
> notification/src/main/java/org/apache/atlas/notification/spool/SpoolUtils.java
>  abbe33d24 
>   
> notification/src/test/java/org/apache/atlas/notification/spool/BaseTest.java 
> 304c82122 
> 
> 
> Diff: https://reviews.apache.org/r/73375/diff/1/
> 
> 
> Testing
> ---
> 
> **Functional tests**
> Existing workflows.
> 
> **Unit tests**
> None.
> 
> 
> Thanks,
> 
> Ashutosh Mestry
> 
>

Re: Review Request 73405: ATLAS-4302: Migrated Data: Process Entity Name not set to QualifiedName

[Nikhil Bonte]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73405/#review223104
---


Ship it!




Ship It!

- Nikhil Bonte


On June 4, 2021, 10:18 a.m., Ashutosh Mestry wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73405/
> ---
> 
> (Updated June 4, 2021, 10:18 a.m.)
> 
> 
> Review request for atlas, Nikhil Bonte, Nixon Rodrigues, and Sarath 
> Subramanian.
> 
> 
> Bugs: ATLAS-4302
> https://issues.apache.org/jira/browse/ATLAS-4302
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> **Approach**
> Java patch is optionally run when 
> ```atlas.process.name.update.patch
> 
> is set to true.
> 
> 
> Diffs
> -
> 
>   intg/src/main/java/org/apache/atlas/AtlasConfiguration.java e31afa89a 
>   
> repository/src/main/java/org/apache/atlas/repository/patches/AtlasPatchManager.java
>  fae28c440 
>   
> repository/src/main/java/org/apache/atlas/repository/patches/ProcessNamePatch.java
>  PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/73405/diff/2/
> 
> 
> Testing
> ---
> 
> **Functional test**
> - Validated with medium data set (5M entities).
> 
> **Volume test**
> - Validated improvement for a lineage entity with 1200 inputs and 1700 
> outputs. With this change, the _AtlasEntityHeader_ payload size reduces by 
> 90%.
> 
> 
> Thanks,
> 
> Ashutosh Mestry
> 
>

Review Request 73423: ATLAS-4338: Import Service: Support change notifiers for migration import

[Nikhil Bonte]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73423/
---

Review request for atlas, Ashutosh Mestry, Jayendra Parab, madhan, Nixon 
Rodrigues, Sarath Subramanian, and Sidharth Mishra.


Bugs: ATLAS-4338
https://issues.apache.org/jira/browse/ATLAS-4338


Repository: atlas


Description
---

Currently Migration import feature (MigrationImport.java) does not support 
enabling notifiers (audits writer, notification to Atlas entities topic).

With this improvement, Atlas now will enable notifiers for Migration import, 
based on a flag.

 

Approach:

* New option in AtlasImportRequest - sendNotifications (default : false)
* MigrationImport will use the flag to select appropriate implementation of 
IAtlasEntityChangeNotifier to create EntityCreationManager.
* EntityCreationManager will then notify listeners.


Diffs
-

  intg/src/main/java/org/apache/atlas/model/impexp/AtlasImportRequest.java 
2c1870444 
  
repository/src/main/java/org/apache/atlas/repository/impexp/ImportService.java 
1d29bf833 
  
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityChangeNotifier.java
 99be169f2 
  
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/BulkImporterImpl.java
 8e17fd410 
  
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/bulkimport/MigrationImport.java
 d6f23d6e2 
  
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/bulkimport/pc/EntityConsumer.java
 b73988fd7 
  
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/bulkimport/pc/EntityCreationManager.java
 734add6d7 


Diff: https://reviews.apache.org/r/73423/diff/1/


Testing
---

* Tested Manually
* Attached sample zip, sample request
{
"options": {
"migration": "true",
"size": "100",
"sendNotifications": "true",
"batchSize": 500,
"numWorkers": 40
}
}
* took ~270 secs to import 12987 entities with notifiers enabled.
* PC build :


File Attachments


largedb.zip
  
https://reviews.apache.org/media/uploaded/files/2021/06/15/c84f450b-c2b3-4cac-8b4a-76942a2c14e3__largedb.zip


Thanks,

Nikhil Bonte

Re: Review Request 73423: ATLAS-4338: Import Service: Support change notifiers for migration import

[Nikhil Bonte]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73423/
---

(Updated June 21, 2021, 7:36 p.m.)


Review request for atlas, Ashutosh Mestry, Jayendra Parab, madhan, Nixon 
Rodrigues, Sarath Subramanian, and Sidharth Mishra.


Changes
---

Updated PC build link


Bugs: ATLAS-4338
https://issues.apache.org/jira/browse/ATLAS-4338


Repository: atlas


Description
---

Currently Migration import feature (MigrationImport.java) does not support 
enabling notifiers (audits writer, notification to Atlas entities topic).

With this improvement, Atlas now will enable notifiers for Migration import, 
based on a flag.

 

Approach:

* New option in AtlasImportRequest - sendNotifications (default : false)
* MigrationImport will use the flag to select appropriate implementation of 
IAtlasEntityChangeNotifier to create EntityCreationManager.
* EntityCreationManager will then notify listeners.


Diffs
-

  intg/src/main/java/org/apache/atlas/model/impexp/AtlasImportRequest.java 
2c1870444 
  
repository/src/main/java/org/apache/atlas/repository/impexp/ImportService.java 
1d29bf833 
  
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityChangeNotifier.java
 99be169f2 
  
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/BulkImporterImpl.java
 8e17fd410 
  
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/bulkimport/MigrationImport.java
 d6f23d6e2 
  
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/bulkimport/pc/EntityConsumer.java
 b73988fd7 
  
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/bulkimport/pc/EntityCreationManager.java
 734add6d7 


Diff: https://reviews.apache.org/r/73423/diff/1/


Testing (updated)
---

* Tested Manually
* Sample request
{
"options": {
"migration": "true",
"size": "100",
"sendNotifications": "true",
"batchSize": 500,
"numWorkers": 40
}
}
* took ~270 secs to import 12987 entities with notifiers enabled.
* PC build : 
https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/679/console


Thanks,

Nikhil Bonte

Re: Review Request 73423: ATLAS-4338: Import Service: Support change notifiers for migration import

[Nikhil Bonte]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73423/
---

(Updated June 23, 2021, 7:51 p.m.)


Review request for atlas, Ashutosh Mestry, Jayendra Parab, madhan, Nixon 
Rodrigues, Sarath Subramanian, and Sidharth Mishra.


Changes
---

Addressed review comment


Bugs: ATLAS-4338
https://issues.apache.org/jira/browse/ATLAS-4338


Repository: atlas


Description
---

Currently Migration import feature (MigrationImport.java) does not support 
enabling notifiers (audits writer, notification to Atlas entities topic).

With this improvement, Atlas now will enable notifiers for Migration import, 
based on a flag.

 

Approach:

* New option in AtlasImportRequest - sendNotifications (default : false)
* MigrationImport will use the flag to select appropriate implementation of 
IAtlasEntityChangeNotifier to create EntityCreationManager.
* EntityCreationManager will then notify listeners.


Diffs (updated)
-

  intg/src/main/java/org/apache/atlas/model/impexp/AtlasImportRequest.java 
2c1870444 
  
repository/src/main/java/org/apache/atlas/repository/impexp/ImportService.java 
1d29bf833 
  
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityChangeNotifier.java
 99be169f2 
  
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/BulkImporterImpl.java
 8e17fd410 
  
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/bulkimport/BulkImportResultItem.java
 PRE-CREATION 
  
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/bulkimport/MigrationImport.java
 d6f23d6e2 
  
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/bulkimport/pc/EntityConsumer.java
 b73988fd7 
  
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/bulkimport/pc/EntityCreationManager.java
 734add6d7 


Diff: https://reviews.apache.org/r/73423/diff/2/

Changes: https://reviews.apache.org/r/73423/diff/1-2/


Testing
---

* Tested Manually
* Sample request
{
"options": {
"migration": "true",
"size": "100",
"sendNotifications": "true",
"batchSize": 500,
"numWorkers": 40
}
}
* took ~270 secs to import 12987 entities with notifiers enabled.
* PC build : 
https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/679/console


Thanks,

Nikhil Bonte

Re: Review Request 73423: ATLAS-4338: Import Service: Support change notifiers for migration import

[Nikhil Bonte]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73423/
---

(Updated June 30, 2021, 4:37 p.m.)


Review request for atlas, Ashutosh Mestry, Jayendra Parab, madhan, Nixon 
Rodrigues, Sarath Subramanian, and Sidharth Mishra.


Changes
---

Addressed review comments


Bugs: ATLAS-4338
https://issues.apache.org/jira/browse/ATLAS-4338


Repository: atlas


Description
---

Currently Migration import feature (MigrationImport.java) does not support 
enabling notifiers (audits writer, notification to Atlas entities topic).

With this improvement, Atlas now will enable notifiers for Migration import, 
based on a flag.

 

Approach:

* New option in AtlasImportRequest - sendNotifications (default : false)
* MigrationImport will use the flag to select appropriate implementation of 
IAtlasEntityChangeNotifier to create EntityCreationManager.
* EntityCreationManager will then notify listeners.


Diffs (updated)
-

  intg/src/main/java/org/apache/atlas/model/impexp/AtlasImportRequest.java 
2c1870444 
  
repository/src/main/java/org/apache/atlas/repository/impexp/ImportService.java 
1d29bf833 
  
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityChangeNotifier.java
 99be169f2 
  
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/BulkImporterImpl.java
 8e17fd410 
  
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/bulkimport/GuidMutationResponsePair.java
 PRE-CREATION 
  
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/bulkimport/MigrationImport.java
 d6f23d6e2 
  
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/bulkimport/pc/EntityConsumer.java
 b73988fd7 
  
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/bulkimport/pc/EntityCreationManager.java
 734add6d7 


Diff: https://reviews.apache.org/r/73423/diff/3/

Changes: https://reviews.apache.org/r/73423/diff/2-3/


Testing
---

* Tested Manually
* Sample request
{
"options": {
"migration": "true",
"size": "100",
"sendNotifications": "true",
"batchSize": 500,
"numWorkers": 40
}
}
* took ~270 secs to import 12987 entities with notifiers enabled.
* PC build : 
https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/679/console


Thanks,

Nikhil Bonte

Re: Review Request 73442: ATLAS-4150 Remove System Internal Entity Types from Metrics GET api

[Nikhil Bonte]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73442/#review223234
---


Fix it, then Ship it!




Fix & Ship


repository/src/main/java/org/apache/atlas/services/MetricsService.java
Lines 248 (patched)
<https://reviews.apache.org/r/73442/#comment312326>

CollectionUtils.isNotEmpty(entityDef.getSuperTypes())

I think this check is not required, please review & remove.


- Nikhil Bonte


On July 9, 2021, 1:11 p.m., Mandar Ambawane wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73442/
> ---
> 
> (Updated July 9, 2021, 1:11 p.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Jayendra Parab, Nixon Rodrigues, 
> and Sarath Subramanian.
> 
> 
> Bugs: ATLAS-4150
> https://issues.apache.org/jira/browse/ATLAS-4150
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> This patch checks all the Super Types of an Entity Type. If any of the Super 
> Types is __internal
> 
> Then it will not allow that Entity Type in Metrics api response.
> 
> 
> Diffs
> -
> 
>   repository/src/main/java/org/apache/atlas/services/MetricsService.java 
> e721108d3 
>   repository/src/test/java/org/apache/atlas/services/MetricsServiceTest.java 
> bea8eb8b5 
> 
> 
> Diff: https://reviews.apache.org/r/73442/diff/1/
> 
> 
> Testing
> ---
> 
> PreCommit: 
> https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/734/consoleFull
> 
> 
> Thanks,
> 
> Mandar Ambawane
> 
>