[jira] [Commented] (ATLAS-3545) NullPointerException while trying to delete classification

[Bolke de Bruin (Jira)]

[ 
https://issues.apache.org/jira/browse/ATLAS-3545?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16987628#comment-16987628
 ] 

Bolke de Bruin commented on ATLAS-3545:
---

Thanks [~sarath]. We will apply the patch and observe the behavior. Please note 
that this only removes the symptom from search/retrieval. On entity update an 
invalid propagating classification is still being evaluated. That problem is 
worse as it creates performance issues. The question thus also becomes, apart 
from the race condition, how to delete the invalid vertex?

> NullPointerException while trying to delete classification
> --
>
> Key: ATLAS-3545
> URL: https://issues.apache.org/jira/browse/ATLAS-3545
> Project: Atlas
>  Issue Type: Bug
>Affects Versions: trunk
>Reporter: Bolke de Bruin
>Assignee: Sarath Subramanian
>Priority: Critical
> Attachments: ATLAS-3545.001.patch, Screenshot 2019-11-30 at 
> 22.16.44.png, Screenshot 2019-11-30 at 22.28.00.png, Screenshot 2019-12-02 at 
> 21.43.29.png
>
>
> We see an issue where there is a NullPointerException while trying to delete 
> a classification that is propagating.
> It seems (stack trace is as of yet unavailable) that it is caused due to a 
> typeName being NULL in AtlasTypeRegistry.getType. The UI looks fishy too see 
> screenshots.
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (ATLAS-3545) NullPointerException while trying to delete classification

[Sarath Subramanian (Jira)]

[ 
https://issues.apache.org/jira/browse/ATLAS-3545?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16987617#comment-16987617
 ] 

Sarath Subramanian commented on ATLAS-3545:
---

[~bolke], while we continue to investigate the cause of corrupt classification 
vertex being created (due to race condition). Attached patch should avoid NPE 
and print the corrupt classification vertex id in the logs and continue entity 
search/retrieval. 

 

 

> NullPointerException while trying to delete classification
> --
>
> Key: ATLAS-3545
> URL: https://issues.apache.org/jira/browse/ATLAS-3545
> Project: Atlas
>  Issue Type: Bug
>Affects Versions: trunk
>Reporter: Bolke de Bruin
>Assignee: Sarath Subramanian
>Priority: Critical
> Attachments: ATLAS-3545.001.patch, Screenshot 2019-11-30 at 
> 22.16.44.png, Screenshot 2019-11-30 at 22.28.00.png, Screenshot 2019-12-02 at 
> 21.43.29.png
>
>
> We see an issue where there is a NullPointerException while trying to delete 
> a classification that is propagating.
> It seems (stack trace is as of yet unavailable) that it is caused due to a 
> typeName being NULL in AtlasTypeRegistry.getType. The UI looks fishy too see 
> screenshots.
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (ATLAS-3545) NullPointerException while trying to delete classification

[Sarath Subramanian (Jira)]

 [ 
https://issues.apache.org/jira/browse/ATLAS-3545?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sarath Subramanian updated ATLAS-3545:
--
Attachment: ATLAS-3545.001.patch

> NullPointerException while trying to delete classification
> --
>
> Key: ATLAS-3545
> URL: https://issues.apache.org/jira/browse/ATLAS-3545
> Project: Atlas
>  Issue Type: Bug
>Affects Versions: trunk
>Reporter: Bolke de Bruin
>Assignee: Sarath Subramanian
>Priority: Critical
> Attachments: ATLAS-3545.001.patch, Screenshot 2019-11-30 at 
> 22.16.44.png, Screenshot 2019-11-30 at 22.28.00.png, Screenshot 2019-12-02 at 
> 21.43.29.png
>
>
> We see an issue where there is a NullPointerException while trying to delete 
> a classification that is propagating.
> It seems (stack trace is as of yet unavailable) that it is caused due to a 
> typeName being NULL in AtlasTypeRegistry.getType. The UI looks fishy too see 
> screenshots.
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Review Request 71870: ATLAS-3545: NullPointerException while trying to delete classification

[Sarath Subramanian]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71870/
---

Review request for atlas, Ashutosh Mestry, Aadarsh Jajodia, Sridhar K, Le Ma, 
Madhan Neethiraj, Mandar Ambawane, Nikhil Bonte, Nixon Rodrigues, Sameer 
Shaikh, and Sidharth Mishra.


Bugs: ATLAS-3545
https://issues.apache.org/jira/browse/ATLAS-3545


Repository: atlas


Description
---

Corrupt/Null classification vertices is causing NPE during entity search and 
retrieval. Null checks have been added to avoid NPE and logging added to 
identify potential corrupt classification vertex.


Diffs
-

  
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphMapper.java
 a114d252e 
  
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphRetriever.java
 6b111732a 


Diff: https://reviews.apache.org/r/71870/diff/1/


Testing
---

Precommit: 
https://builds.apache.org/view/A/view/Atlas/job/PreCommit-ATLAS-Build-Test/1575/

Manually verified that null/corrupt classification vertices are ignored during 
search and entity retrieval.

Approproate WARN messages are printed in the log file:

*2019-12-03 23:26:45,243 WARN  - [pool-2-thread-10 - 
305383de-19f2-4f93-89db-3b54db870c3f:] ~ Ignoring invalid classification 
vertex: vertex[id=81932528] (EntityGraphRetriever:284)*


Thanks,

Sarath Subramanian

[jira] [Commented] (ATLAS-3545) NullPointerException while trying to delete classification

[Sarath Subramanian (Jira)]

[ 
https://issues.apache.org/jira/browse/ATLAS-3545?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16987301#comment-16987301
 ] 

Sarath Subramanian commented on ATLAS-3545:
---

[~bolke] thanks for the repro steps, this is useful. I will investigate this 
issue.

> NullPointerException while trying to delete classification
> --
>
> Key: ATLAS-3545
> URL: https://issues.apache.org/jira/browse/ATLAS-3545
> Project: Atlas
>  Issue Type: Bug
>Affects Versions: trunk
>Reporter: Bolke de Bruin
>Assignee: Sarath Subramanian
>Priority: Critical
> Attachments: Screenshot 2019-11-30 at 22.16.44.png, Screenshot 
> 2019-11-30 at 22.28.00.png, Screenshot 2019-12-02 at 21.43.29.png
>
>
> We see an issue where there is a NullPointerException while trying to delete 
> a classification that is propagating.
> It seems (stack trace is as of yet unavailable) that it is caused due to a 
> typeName being NULL in AtlasTypeRegistry.getType. The UI looks fishy too see 
> screenshots.
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Assigned] (ATLAS-3545) NullPointerException while trying to delete classification

[Sarath Subramanian (Jira)]

 [ 
https://issues.apache.org/jira/browse/ATLAS-3545?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sarath Subramanian reassigned ATLAS-3545:
-

Assignee: Sarath Subramanian

> NullPointerException while trying to delete classification
> --
>
> Key: ATLAS-3545
> URL: https://issues.apache.org/jira/browse/ATLAS-3545
> Project: Atlas
>  Issue Type: Bug
>Affects Versions: trunk
>Reporter: Bolke de Bruin
>Assignee: Sarath Subramanian
>Priority: Critical
> Attachments: Screenshot 2019-11-30 at 22.16.44.png, Screenshot 
> 2019-11-30 at 22.28.00.png, Screenshot 2019-12-02 at 21.43.29.png
>
>
> We see an issue where there is a NullPointerException while trying to delete 
> a classification that is propagating.
> It seems (stack trace is as of yet unavailable) that it is caused due to a 
> typeName being NULL in AtlasTypeRegistry.getType. The UI looks fishy too see 
> screenshots.
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: Review Request 71818: ATLAS-3534 Changes to allow namespaces to be added to an entity instance

[Sarath Subramanian]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71818/#review218895
---



Patch fails to apply in apache master. can you please rebase.

- Sarath Subramanian


On Nov. 25, 2019, 6:04 p.m., Aadarsh Jajodia wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/71818/
> ---
> 
> (Updated Nov. 25, 2019, 6:04 p.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Sridhar K, Le Ma, Madhan 
> Neethiraj, and Sarath Subramanian.
> 
> 
> Bugs: ATLAS-3534
> https://issues.apache.org/jira/browse/ATLAS-3534
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> The changes made in these include adding/updating namespaces to an entity 
> instance
> as part of create or update
> 
> 
> Diffs
> -
> 
>   common/src/main/java/org/apache/atlas/repository/Constants.java 
> 0b28243c47d9e025ecc38a05ce1a54efd71e6149 
>   intg/src/main/java/org/apache/atlas/AtlasErrorCode.java 
> 7a2aae2e9ae8174c5309164f3e41c940cbf3ddf8 
>   intg/src/main/java/org/apache/atlas/model/instance/AtlasEntity.java 
> 1b033b9ac0d413c04269b480562940fb5b305939 
>   intg/src/main/java/org/apache/atlas/type/AtlasEntityType.java 
> 557ef74a95c2a939b4b89cd1db8fa4c73d52dd51 
>   intg/src/main/java/org/apache/atlas/type/AtlasNamespaceType.java 
> PRE-CREATION 
>   repository/src/main/java/org/apache/atlas/repository/graph/GraphHelper.java 
> 1e7acf1e728f472a01a26fe4be69a11e21ca3e77 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityGraphDiscoveryV2.java
>  d3d9367fb158f2bafbcc44f3c25ee1a63c5c2aa7 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityStoreV2.java
>  c8e65ef15fcf38cb4b5f1d9f1f37cf1578c13b48 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphMapper.java
>  a114d252e7462a08c14ebef598ed13b6b43e247b 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphRetriever.java
>  6b111732a11b2aef7dbeb050087bde3c70fd21bd 
> 
> 
> Diff: https://reviews.apache.org/r/71818/diff/1/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Aadarsh Jajodia
> 
>

[jira] [Commented] (ATLAS-2985) Fix delete handlers and relationship store

[Sidharth Kumar Mishra (Jira)]

[ 
https://issues.apache.org/jira/browse/ATLAS-2985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16987220#comment-16987220
 ] 

Sidharth Kumar Mishra commented on ATLAS-2985:
--

Hi [~alollo],

This is been resolved as part of ATLAS-3477. There is one more follow up 
feature for auditing the purge which is being tracked as part of ATLAS-3518.

> Fix delete handlers and relationship store
> --
>
> Key: ATLAS-2985
> URL: https://issues.apache.org/jira/browse/ATLAS-2985
> Project: Atlas
>  Issue Type: Bug
>Reporter: Graham Wallis
>Assignee: Graham Wallis
>Priority: Critical
> Attachments: ATLAS-2985-2018-11-30.patch
>
>
> The problems this fixes are as follows:
>  * following a soft delete, a hard delete is not possible because the entity 
> or relationship will have status DELETED, and prior to this Jira and patch 
> the abstract delete handler skips the delete. 
>  * during a soft delete the modified-time, modified-by and version number 
> attributes on the entity or relationship should be updated, but 
> updateRelationship does not update system attributes - this Jira and patch 
> updates the system attributes that will need to be updated during a delete or 
> other update operations.
>  * also fixes a minor copy-paste debug logging error.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (ATLAS-3547) Don’t evaluate propagating classifications for non relevant update

[Bolke de Bruin (Jira)]

Bolke de Bruin created ATLAS-3547:
-

 Summary: Don’t evaluate propagating classifications for non 
relevant update
 Key: ATLAS-3547
 URL: https://issues.apache.org/jira/browse/ATLAS-3547
 Project: Atlas
  Issue Type: Bug
Affects Versions: trunk
Reporter: Bolke de Bruin


In case of an aws_s3_bucket with a propagating classification and with several 
thousands (200K+) objects we are observing that every addition of an object to 
this aws_s3_bucket triggers a re-evaluation of all entities with the 
propagating tag. This exponentially increases the time that it takes a message 
to be consumed if the bucket is growing in size.

We send a create or update message for an aws_s3_object by Kafka which has all 
relevant information inside the message (ie. Aws_s3_bucket, aws_s3_pseudodir, 
aws_s3_object) as the producer system is unaware if the relevant metadata for 
the bucket and pseudodir are already available. This triggers an evaluation of 
the classifications and all propagating classifications. This seems over eager.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (ATLAS-3545) NullPointerException while trying to delete classification

[Bolke de Bruin (Jira)]

[ 
https://issues.apache.org/jira/browse/ATLAS-3545?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16987118#comment-16987118
 ] 

Bolke de Bruin commented on ATLAS-3545:
---

[~ayubpathan] they were included above, but just to get it clear:

1. Add a propagating classification to an entity with a large container (e.g. 
aws_s3_bucket). 
2. Add the same propagating classification again before the UI shows the 
classification is applied
3. Observe 2 classifications have been applied with the same name
4. Remove one of the propagating classifications
5. A Try to remove the other one(s), this should fail
B Observe that the count in the UI basic search view is 1 or higher (e.g. 
it reads something like MANAGED (2) )
C Try to search for this classification with basic search, this should fail 
with an error message
D Try to read classification properties in the classification view, this 
should fail
E Observer that the propagating classification still is applied to derived 
entities (e.g. aws_s3_objects)

Obviously we are seeing multiple issues here (race condition, null pointer). 
Our goal is first to remove the “orphan” propagating classification

4. Observer

> NullPointerException while trying to delete classification
> --
>
> Key: ATLAS-3545
> URL: https://issues.apache.org/jira/browse/ATLAS-3545
> Project: Atlas
>  Issue Type: Bug
>Affects Versions: trunk
>Reporter: Bolke de Bruin
>Priority: Critical
> Attachments: Screenshot 2019-11-30 at 22.16.44.png, Screenshot 
> 2019-11-30 at 22.28.00.png, Screenshot 2019-12-02 at 21.43.29.png
>
>
> We see an issue where there is a NullPointerException while trying to delete 
> a classification that is propagating.
> It seems (stack trace is as of yet unavailable) that it is caused due to a 
> typeName being NULL in AtlasTypeRegistry.getType. The UI looks fishy too see 
> screenshots.
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (ATLAS-3545) NullPointerException while trying to delete classification

[Ayub Pathan (Jira)]

[ 
https://issues.apache.org/jira/browse/ATLAS-3545?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16987104#comment-16987104
 ] 

Ayub Pathan commented on ATLAS-3545:


[~bolke] Can you please add the repro steps here?

> NullPointerException while trying to delete classification
> --
>
> Key: ATLAS-3545
> URL: https://issues.apache.org/jira/browse/ATLAS-3545
> Project: Atlas
>  Issue Type: Bug
>Affects Versions: trunk
>Reporter: Bolke de Bruin
>Priority: Critical
> Attachments: Screenshot 2019-11-30 at 22.16.44.png, Screenshot 
> 2019-11-30 at 22.28.00.png, Screenshot 2019-12-02 at 21.43.29.png
>
>
> We see an issue where there is a NullPointerException while trying to delete 
> a classification that is propagating.
> It seems (stack trace is as of yet unavailable) that it is caused due to a 
> typeName being NULL in AtlasTypeRegistry.getType. The UI looks fishy too see 
> screenshots.
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: Review Request 71791: ATLAS-3518: Create Audit Framework for Atlas Purge Entity

[Sidharth Mishra]

> On Dec. 1, 2019, 4:37 p.m., Madhan Neethiraj wrote:
> > addons/models/-Area0/0010-base_model.json
> > Lines 4 (patched)
> > 
> >
> > atlas_operation => atlas_admin_operation

I think as you initially suggested to rename it to atlas_operation is more 
flexible than atlas_admin_operation as in future we may allow some more 
operation to be audited like other than admin operations.


- Sidharth


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71791/#review218865
---


On Nov. 30, 2019, 7:44 p.m., Sidharth Mishra wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/71791/
> ---
> 
> (Updated Nov. 30, 2019, 7:44 p.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Madhan Neethiraj, and Sarath 
> Subramanian.
> 
> 
> Bugs: ATLAS-3518
> https://issues.apache.org/jira/browse/ATLAS-3518
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Right now for purge, delete entity, etc. at Atlas we create audit entry at 
> HBase. User can go to entity and click on audit to see the audit information 
> for that particular entity. But if user purge one entity there will not be 
> any vertex at Janus graph and user will not able to get the purge audit which 
> is important for governance purpose.
> 
> As part of fix, implemented Atlas Audit Framework which is being used for 
> Purge operation and can be extended for other operations. In Future once we 
> change the AtlasAudit type as super type for Import export as well we can use 
> this framework for all Atlas operations. 
> 
> admin/audit rest has been added which will return all the Atlas audit 
> operations including import export at Atlas.
> 
> 
> Diffs
> -
> 
>   addons/models/-Area0/0010-base_model.json 2f5fdaf14 
>   client/client-v2/src/main/java/org/apache/atlas/AtlasClientV2.java 
> 8c0a6407c 
>   dashboardv2/public/js/utils/Enums.js 58542ec59 
>   dashboardv3/public/js/utils/Enums.js 58542ec59 
>   intg/src/main/java/org/apache/atlas/model/audit/AtlasAuditEntry.java 
> PRE-CREATION 
>   
> intg/src/main/java/org/apache/atlas/model/instance/EntityMutationResponse.java
>  b448d510b 
>   
> repository/src/main/java/org/apache/atlas/repository/audit/AtlasAuditService.java
>  PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/repository/ogm/AtlasAuditEntryDTO.java
>  PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v1/DeleteHandlerV1.java
>  621458406 
>   repository/src/test/java/org/apache/atlas/TestModules.java 94b08a187 
>   repository/src/test/java/org/apache/atlas/glossary/GlossaryServiceTest.java 
> be1698fb4 
>   repository/src/test/java/org/apache/atlas/query/BasicTestSetup.java 
> 02f78b369 
>   
> repository/src/test/java/org/apache/atlas/repository/audit/AdminPurgeTest.java
>  PRE-CREATION 
>   
> repository/src/test/java/org/apache/atlas/repository/audit/AtlasAuditServiceTest.java
>  PRE-CREATION 
>   
> repository/src/test/java/org/apache/atlas/repository/impexp/AtlasServerServiceTest.java
>  78865c569 
>   
> repository/src/test/java/org/apache/atlas/repository/impexp/ExportImportAuditServiceTest.java
>  ba7a8a04b 
>   
> repository/src/test/java/org/apache/atlas/repository/impexp/ExportImportTestBase.java
>  925b2a007 
>   
> repository/src/test/java/org/apache/atlas/repository/impexp/ExportIncrementalTest.java
>  4d438525a 
>   
> repository/src/test/java/org/apache/atlas/repository/impexp/ExportServiceTest.java
>  db8c6c83c 
>   
> repository/src/test/java/org/apache/atlas/repository/impexp/ExportSkipLineageTest.java
>  25e0a5362 
>   
> repository/src/test/java/org/apache/atlas/repository/impexp/HdfsPathEntityCreatorTest.java
>  1863b8d74 
>   
> repository/src/test/java/org/apache/atlas/repository/impexp/ImportReactivateTableTest.java
>  d0c06a107 
>   
> repository/src/test/java/org/apache/atlas/repository/impexp/ImportServiceTest.java
>  c14850f43 
>   
> repository/src/test/java/org/apache/atlas/repository/impexp/ImportTransformsShaperTest.java
>  78fdaca8d 
>   
> repository/src/test/java/org/apache/atlas/repository/impexp/IncrementalExportEntityProviderTest.java
>  42b63535b 
>   
> repository/src/test/java/org/apache/atlas/repository/impexp/RelationshipAttributesExtractorTest.java
>  a1b512fb9 
>   
> repository/src/test/java/org/apache/atlas/repository/impexp/ReplicationEntityAttributeTest.java
>  7a1ed18cb 
>   
> repository/src/test/java/org/apache/atlas/repository/impexp/StartEntityFetchByExportRequestTest.java
>  48b958271 
>   
> repository/src/test/java/org/apache/atlas/repository/impexp/TableReplicationRequestProcessorTest.java
>  c9bb11c56 
>   
> repository

Re: Review Request 71791: ATLAS-3518: Create Audit Framework for Atlas Purge Entity

[Sidharth Mishra]

> On Nov. 27, 2019, 12:39 a.m., Madhan Neethiraj wrote:
> > addons/models/-Area0/0010-base_model.json
> > Lines 442 (patched)
> > 
> >
> > Consider renaming 'clientHost' to 'clientId' - given this field is used 
> > to store AtlasServer name for import/export audit entries.
> > 
> > This rename should be applied to AtlasAuditEntry.clientHost as well.

I changed it to clientId as suggested, But please note this field contains 
Client IP Address. In future we may put host fqdn if available. In both the 
cases do you think client Id makes more sense?


> On Nov. 27, 2019, 12:39 a.m., Madhan Neethiraj wrote:
> > pom.xml
> > Line 695 (original), 695 (patched)
> > 
> >
> > I suggest a separate patch for library version changes.

This fix was to fix the precommit build and now the precommit is successful - 
https://builds.apache.org/view/A/view/Atlas/job/PreCommit-ATLAS-Build-Test/1559/.
 I am having separate patch for the lib version change and reverted the changes


- Sidharth


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71791/#review218808
---


On Nov. 30, 2019, 7:44 p.m., Sidharth Mishra wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/71791/
> ---
> 
> (Updated Nov. 30, 2019, 7:44 p.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Madhan Neethiraj, and Sarath 
> Subramanian.
> 
> 
> Bugs: ATLAS-3518
> https://issues.apache.org/jira/browse/ATLAS-3518
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Right now for purge, delete entity, etc. at Atlas we create audit entry at 
> HBase. User can go to entity and click on audit to see the audit information 
> for that particular entity. But if user purge one entity there will not be 
> any vertex at Janus graph and user will not able to get the purge audit which 
> is important for governance purpose.
> 
> As part of fix, implemented Atlas Audit Framework which is being used for 
> Purge operation and can be extended for other operations. In Future once we 
> change the AtlasAudit type as super type for Import export as well we can use 
> this framework for all Atlas operations. 
> 
> admin/audit rest has been added which will return all the Atlas audit 
> operations including import export at Atlas.
> 
> 
> Diffs
> -
> 
>   addons/models/-Area0/0010-base_model.json 2f5fdaf14 
>   client/client-v2/src/main/java/org/apache/atlas/AtlasClientV2.java 
> 8c0a6407c 
>   dashboardv2/public/js/utils/Enums.js 58542ec59 
>   dashboardv3/public/js/utils/Enums.js 58542ec59 
>   intg/src/main/java/org/apache/atlas/model/audit/AtlasAuditEntry.java 
> PRE-CREATION 
>   
> intg/src/main/java/org/apache/atlas/model/instance/EntityMutationResponse.java
>  b448d510b 
>   
> repository/src/main/java/org/apache/atlas/repository/audit/AtlasAuditService.java
>  PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/repository/ogm/AtlasAuditEntryDTO.java
>  PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v1/DeleteHandlerV1.java
>  621458406 
>   repository/src/test/java/org/apache/atlas/TestModules.java 94b08a187 
>   repository/src/test/java/org/apache/atlas/glossary/GlossaryServiceTest.java 
> be1698fb4 
>   repository/src/test/java/org/apache/atlas/query/BasicTestSetup.java 
> 02f78b369 
>   
> repository/src/test/java/org/apache/atlas/repository/audit/AdminPurgeTest.java
>  PRE-CREATION 
>   
> repository/src/test/java/org/apache/atlas/repository/audit/AtlasAuditServiceTest.java
>  PRE-CREATION 
>   
> repository/src/test/java/org/apache/atlas/repository/impexp/AtlasServerServiceTest.java
>  78865c569 
>   
> repository/src/test/java/org/apache/atlas/repository/impexp/ExportImportAuditServiceTest.java
>  ba7a8a04b 
>   
> repository/src/test/java/org/apache/atlas/repository/impexp/ExportImportTestBase.java
>  925b2a007 
>   
> repository/src/test/java/org/apache/atlas/repository/impexp/ExportIncrementalTest.java
>  4d438525a 
>   
> repository/src/test/java/org/apache/atlas/repository/impexp/ExportServiceTest.java
>  db8c6c83c 
>   
> repository/src/test/java/org/apache/atlas/repository/impexp/ExportSkipLineageTest.java
>  25e0a5362 
>   
> repository/src/test/java/org/apache/atlas/repository/impexp/HdfsPathEntityCreatorTest.java
>  1863b8d74 
>   
> repository/src/test/java/org/apache/atlas/repository/impexp/ImportReactivateTableTest.java
>  d0c06a107 
>   
> repository/src/test/java/org/apache/atlas/repository/impexp/ImportServiceTest.java
>  c14850f43 
>   
> repository/src/test/java/org/apache/atlas/reposit

Re: Review Request 71791: ATLAS-3518: Create Audit Framework for Atlas Purge Entity

[Sidharth Mishra]

- Sidharth


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71791/#review218808
---


On Nov. 30, 2019, 7:44 p.m., Sidharth Mishra wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/71791/
> ---
> 
> (Updated Nov. 30, 2019, 7:44 p.m.)
> 
> 
> Review request for atlas, Ashutosh Mestry, Madhan Neethiraj, and Sarath 
> Subramanian.
> 
> 
> Bugs: ATLAS-3518
> https://issues.apache.org/jira/browse/ATLAS-3518
> 
> 
> Repository: atlas
> 
> 
> Description
> ---
> 
> Right now for purge, delete entity, etc. at Atlas we create audit entry at 
> HBase. User can go to entity and click on audit to see the audit information 
> for that particular entity. But if user purge one entity there will not be 
> any vertex at Janus graph and user will not able to get the purge audit which 
> is important for governance purpose.
> 
> As part of fix, implemented Atlas Audit Framework which is being used for 
> Purge operation and can be extended for other operations. In Future once we 
> change the AtlasAudit type as super type for Import export as well we can use 
> this framework for all Atlas operations. 
> 
> admin/audit rest has been added which will return all the Atlas audit 
> operations including import export at Atlas.
> 
> 
> Diffs
> -
> 
>   addons/models/-Area0/0010-base_model.json 2f5fdaf14 
>   client/client-v2/src/main/java/org/apache/atlas/AtlasClientV2.java 
> 8c0a6407c 
>   dashboardv2/public/js/utils/Enums.js 58542ec59 
>   dashboardv3/public/js/utils/Enums.js 58542ec59 
>   intg/src/main/java/org/apache/atlas/model/audit/AtlasAuditEntry.java 
> PRE-CREATION 
>   
> intg/src/main/java/org/apache/atlas/model/instance/EntityMutationResponse.java
>  b448d510b 
>   
> repository/src/main/java/org/apache/atlas/repository/audit/AtlasAuditService.java
>  PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/repository/ogm/AtlasAuditEntryDTO.java
>  PRE-CREATION 
>   
> repository/src/main/java/org/apache/atlas/repository/store/graph/v1/DeleteHandlerV1.java
>  621458406 
>   repository/src/test/java/org/apache/atlas/TestModules.java 94b08a187 
>   repository/src/test/java/org/apache/atlas/glossary/GlossaryServiceTest.java 
> be1698fb4 
>   repository/src/test/java/org/apache/atlas/query/BasicTestSetup.java 
> 02f78b369 
>   
> repository/src/test/java/org/apache/atlas/repository/audit/AdminPurgeTest.java
>  PRE-CREATION 
>   
> repository/src/test/java/org/apache/atlas/repository/audit/AtlasAuditServiceTest.java
>  PRE-CREATION 
>   
> repository/src/test/java/org/apache/atlas/repository/impexp/AtlasServerServiceTest.java
>  78865c569 
>   
> repository/src/test/java/org/apache/atlas/repository/impexp/ExportImportAuditServiceTest.java
>  ba7a8a04b 
>   
> repository/src/test/java/org/apache/atlas/repository/impexp/ExportImportTestBase.java
>  925b2a007 
>   
> repository/src/test/java/org/apache/atlas/repository/impexp/ExportIncrementalTest.java
>  4d438525a 
>   
> repository/src/test/java/org/apache/atlas/repository/impexp/ExportServiceTest.java
>  db8c6c83c 
>   
> repository/src/test/java/org/apache/atlas/repository/impexp/ExportSkipLineageTest.java
>  25e0a5362 
>   
> repository/src/test/java/org/apache/atlas/repository/impexp/HdfsPathEntityCreatorTest.java
>  1863b8d74 
>   
> repository/src/test/java/org/apache/atlas/repository/impexp/ImportReactivateTableTest.java
>  d0c06a107 
>   
> repository/src/test/java/org/apache/atlas/repository/impexp/ImportServiceTest.java
>  c14850f43 
>   
> repository/src/test/java/org/apache/atlas/repository/impexp/ImportTransformsShaperTest.java
>  78fdaca8d 
>   
> repository/src/test/java/org/apache/atlas/repository/impexp/IncrementalExportEntityProviderTest.java
>  42b63535b 
>   
> repository/src/test/java/org/apache/atlas/repository/impexp/RelationshipAttributesExtractorTest.java
>  a1b512fb9 
>   
> repository/src/test/java/org/apache/atlas/repository/impexp/ReplicationEntityAttributeTest.java
>  7a1ed18cb 
>   
> repository/src/test/java/org/apache/atlas/repository/impexp/StartEntityFetchByExportRequestTest.java
>  48b958271 
>   
> repository/src/test/java/org/apache/atlas/repository/impexp/TableReplicationRequestProcessorTest.java
>  c9bb11c56 
>   
> repository/src/test/java/org/apache/atlas/repository/impexp/ZipFileResourceTestUtils.java
>  0ffc3d595 
>   
> repository/src/test/java/org/apache/atlas/repository/migration/MigrationBaseAsserts.java
>  18e950e8c 
>   
> repository/src/test/java/org/apache/atlas/repository/migration/RelationshipCacheGeneratorTest.java
>  050bda3ae 
>   
> repository/src/test/java/org/apache/atlas/repository/tagpropagation/ClassificationPropagationTest.java
>  6f9c05e7a 
>   
> repository/sr

[jira] [Updated] (ATLAS-3528) UI: Support basic search to search with System attributes only for entity

[Sameer Shaikh (Jira)]

 [ 
https://issues.apache.org/jira/browse/ATLAS-3528?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sameer Shaikh updated ATLAS-3528:
-
Attachment: (was: ATLAS-3528-v1.patch)

> UI: Support basic search  to search with System attributes only for entity
> --
>
> Key: ATLAS-3528
> URL: https://issues.apache.org/jira/browse/ATLAS-3528
> Project: Atlas
>  Issue Type: Improvement
>Reporter: Sameer Shaikh
>Assignee: Sameer Shaikh
>Priority: Major
> Fix For: 2.1.0, 3.0.0
>
> Attachments: ATLAS-3528-v2.patch
>
>
> Support basic search  to search with System attributes only for entity:
> __guid, __historicalGuids, __labels, __customAttributes, 
> __classificationsText, 
> __classificationNames, __propagatedClassificationNames, __isIncomplete



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (ATLAS-3528) UI: Support basic search to search with System attributes only for entity

[Sameer Shaikh (Jira)]

 [ 
https://issues.apache.org/jira/browse/ATLAS-3528?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sameer Shaikh updated ATLAS-3528:
-
Attachment: (was: ATLAS-3528.patch)

> UI: Support basic search  to search with System attributes only for entity
> --
>
> Key: ATLAS-3528
> URL: https://issues.apache.org/jira/browse/ATLAS-3528
> Project: Atlas
>  Issue Type: Improvement
>Reporter: Sameer Shaikh
>Assignee: Sameer Shaikh
>Priority: Major
> Fix For: 2.1.0, 3.0.0
>
> Attachments: ATLAS-3528-v2.patch
>
>
> Support basic search  to search with System attributes only for entity:
> __guid, __historicalGuids, __labels, __customAttributes, 
> __classificationsText, 
> __classificationNames, __propagatedClassificationNames, __isIncomplete



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (ATLAS-3528) UI: Support basic search to search with System attributes only for entity

[Sameer Shaikh (Jira)]

[ 
https://issues.apache.org/jira/browse/ATLAS-3528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16986893#comment-16986893
 ] 

Sameer Shaikh commented on ATLAS-3528:
--

[~kevalbhatt] I found new issue which add "ALL_CLASSIFICATION_TYPES" twice on 
UI. So I updated the patch "ATLAS-3528-v2.patch"

> UI: Support basic search  to search with System attributes only for entity
> --
>
> Key: ATLAS-3528
> URL: https://issues.apache.org/jira/browse/ATLAS-3528
> Project: Atlas
>  Issue Type: Improvement
>Reporter: Sameer Shaikh
>Assignee: Sameer Shaikh
>Priority: Major
> Fix For: 2.1.0, 3.0.0
>
> Attachments: ATLAS-3528-v2.patch
>
>
> Support basic search  to search with System attributes only for entity:
> __guid, __historicalGuids, __labels, __customAttributes, 
> __classificationsText, 
> __classificationNames, __propagatedClassificationNames, __isIncomplete



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (ATLAS-3528) UI: Support basic search to search with System attributes only for entity

[Sameer Shaikh (Jira)]

 [ 
https://issues.apache.org/jira/browse/ATLAS-3528?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sameer Shaikh updated ATLAS-3528:
-
Attachment: ATLAS-3528-v2.patch

> UI: Support basic search  to search with System attributes only for entity
> --
>
> Key: ATLAS-3528
> URL: https://issues.apache.org/jira/browse/ATLAS-3528
> Project: Atlas
>  Issue Type: Improvement
>Reporter: Sameer Shaikh
>Assignee: Sameer Shaikh
>Priority: Major
> Fix For: 2.1.0, 3.0.0
>
> Attachments: ATLAS-3528-v1.patch, ATLAS-3528-v2.patch, 
> ATLAS-3528.patch
>
>
> Support basic search  to search with System attributes only for entity:
> __guid, __historicalGuids, __labels, __customAttributes, 
> __classificationsText, 
> __classificationNames, __propagatedClassificationNames, __isIncomplete



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (ATLAS-3528) UI: Support basic search to search with System attributes only for entity

[Sameer Shaikh (Jira)]

[ 
https://issues.apache.org/jira/browse/ATLAS-3528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16986862#comment-16986862
 ] 

Sameer Shaikh commented on ATLAS-3528:
--

[~kevalbhatt]  I have updated the patch for the issue mention below: 

When we refresh the browser the values in the fields are unaffected except for 
the entity type dropdown which gets overwritten by the "*_ALL_ENTITY_TYPES".*

> UI: Support basic search  to search with System attributes only for entity
> --
>
> Key: ATLAS-3528
> URL: https://issues.apache.org/jira/browse/ATLAS-3528
> Project: Atlas
>  Issue Type: Improvement
>Reporter: Sameer Shaikh
>Assignee: Sameer Shaikh
>Priority: Major
> Fix For: 2.1.0, 3.0.0
>
> Attachments: ATLAS-3528-v1.patch, ATLAS-3528.patch
>
>
> Support basic search  to search with System attributes only for entity:
> __guid, __historicalGuids, __labels, __customAttributes, 
> __classificationsText, 
> __classificationNames, __propagatedClassificationNames, __isIncomplete



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (ATLAS-3528) UI: Support basic search to search with System attributes only for entity

[Sameer Shaikh (Jira)]

 [ 
https://issues.apache.org/jira/browse/ATLAS-3528?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sameer Shaikh updated ATLAS-3528:
-
Attachment: ATLAS-3528-v1.patch

> UI: Support basic search  to search with System attributes only for entity
> --
>
> Key: ATLAS-3528
> URL: https://issues.apache.org/jira/browse/ATLAS-3528
> Project: Atlas
>  Issue Type: Improvement
>Reporter: Sameer Shaikh
>Assignee: Sameer Shaikh
>Priority: Major
> Fix For: 2.1.0, 3.0.0
>
> Attachments: ATLAS-3528-v1.patch, ATLAS-3528.patch
>
>
> Support basic search  to search with System attributes only for entity:
> __guid, __historicalGuids, __labels, __customAttributes, 
> __classificationsText, 
> __classificationNames, __propagatedClassificationNames, __isIncomplete



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Assigned] (ATLAS-3254) Atlas entity with large array of refs causes performance issues for lineage

[Mayank Jain (Jira)]

 [ 
https://issues.apache.org/jira/browse/ATLAS-3254?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mayank Jain reassigned ATLAS-3254:
--

Assignee: Mayank Jain  (was: Sarath Subramanian)

> Atlas entity with large array of refs causes performance issues for lineage
> ---
>
> Key: ATLAS-3254
> URL: https://issues.apache.org/jira/browse/ATLAS-3254
> Project: Atlas
>  Issue Type: Bug
>  Components:  atlas-core, atlas-webui
>Affects Versions: 1.0.0, 2.0.0
>Reporter: Adam Rempter
>Assignee: Mayank Jain
>Priority: Major
>  Labels: performance
> Attachments: Screenshot 2019-11-28 at 21.18.44.png, 
> example_create_entities.json, rest_entity_get_pseudodir.json
>
>
> We use “aws_s3_pseudo_dir” type from 3020-aws_s3_typedefs.json model.
> It has following property: 
> "name":    "s3Objects",
> "typeName":    "array"
>  
> Now in AWS buckets you can have thousands of objects. This causes that 
> s3Objects array grows quite quickly, causing aws_s3_pseudo_dir entity Json to 
> rich easly few MBs.
>  
> Then we start seeing problems like:
>  * UI is dying on displaying entity properties or lineage
>  * Error in logs: audit record too long: entityType=aws_s3_pseudo_dir, 
> guid=24398271-6ba0-4db5-adfa-38e432dc55ce, size=1053931; maxSize=1048576. 
> entity attribute values not stored in audit (EntityAuditListenerV2:234)
>  * Some errors with write to HBase (java.lang.IllegalArgumentException: 
> KeyValue size too large, as workaround we set hbase.client.keyvalue.maxsize 
> param to 0)
>  * kafka consumer errors (we can of course set some parameters on consumer, 
> but I think it is just workaround)
> …
> Exception in NotificationHookConsumer (NotificationHookConsumer:332)
> org.apache.kafka.clients.consumer.CommitFailedException: Commit cannot be 
> completed since the group has already rebalanced and assigned the partitions 
> to another member. This means that the time between subsequen
> t calls to poll() was longer than the configured max.poll.interval.ms, which 
> typically implies that the poll loop is spending too much time message 
> processing. You can address this either by increasing the sessio
> n timeout or by reducing the maximum size of batches returned in poll() with 
> max.poll.records.
> …
> Specifying pseudo_dir is required for s3objects:
> name": "pseudoDirectory",
> "typeName": "aws_s3_pseudo_dir",
> "cardinality": "SINGLE",
> "isIndexable": false,
> *"isOptional": false,*
> "isUnique": false,
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (ATLAS-3531) Atlas website: download/release page missing releases

[ASF subversion and git services (Jira)]

[ 
https://issues.apache.org/jira/browse/ATLAS-3531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16986790#comment-16986790
 ] 

ASF subversion and git services commented on ATLAS-3531:


Commit 9b01c980ed5bf4e613f2018d264278ca86403b54 in atlas-website's branch 
refs/heads/asf-site from sameer79
[ https://gitbox.apache.org/repos/asf?p=atlas-website.git;h=9b01c98 ]

ATLAS-3531 : UI : Atlas website: download/release page missing releases.


> Atlas website: download/release page missing releases
> -
>
> Key: ATLAS-3531
> URL: https://issues.apache.org/jira/browse/ATLAS-3531
> Project: Atlas
>  Issue Type: Bug
>Reporter: Sameer Shaikh
>Assignee: Sameer Shaikh
>Priority: Major
> Attachments: ATLAS-3531.patch, old-version-html-changes.patch
>
>
> [Atlas website|http://atlas.apache.org/#/Downloads] doesn’t show the 
> following releases.
>  * 0.8.3
>  * 0.8.4
>  * 1.2.0
> Download pages from previous release websites (like 
> [2.0.0|http://atlas.apache.org/2.0.0/index.html],  
> [1.1.0|http://atlas.apache.org/1.1.0/index.html], …)  end up in ‘Not Found’.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (ATLAS-3531) Atlas website: download/release page missing releases

[ASF subversion and git services (Jira)]

[ 
https://issues.apache.org/jira/browse/ATLAS-3531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16986791#comment-16986791
 ] 

ASF subversion and git services commented on ATLAS-3531:


Commit 8ca94c8de23c2453385656862244b268ea6482fe in atlas's branch 
refs/heads/master from sameer79
[ https://gitbox.apache.org/repos/asf?p=atlas.git;h=8ca94c8 ]

ATLAS-3531 : UI : Atlas website: download/release page missing releases.


> Atlas website: download/release page missing releases
> -
>
> Key: ATLAS-3531
> URL: https://issues.apache.org/jira/browse/ATLAS-3531
> Project: Atlas
>  Issue Type: Bug
>Reporter: Sameer Shaikh
>Assignee: Sameer Shaikh
>Priority: Major
> Attachments: ATLAS-3531.patch, old-version-html-changes.patch
>
>
> [Atlas website|http://atlas.apache.org/#/Downloads] doesn’t show the 
> following releases.
>  * 0.8.3
>  * 0.8.4
>  * 1.2.0
> Download pages from previous release websites (like 
> [2.0.0|http://atlas.apache.org/2.0.0/index.html],  
> [1.1.0|http://atlas.apache.org/1.1.0/index.html], …)  end up in ‘Not Found’.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (ATLAS-3537) UI: Allow users to search for deleted & shell entities from statistics popup

[Keval Bhatt (Jira)]

 [ 
https://issues.apache.org/jira/browse/ATLAS-3537?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Keval Bhatt updated ATLAS-3537:
---
Attachment: ATLAS-3537-1.patch

> UI: Allow users to search for deleted & shell entities from statistics popup
> 
>
> Key: ATLAS-3537
> URL: https://issues.apache.org/jira/browse/ATLAS-3537
> Project: Atlas
>  Issue Type: Improvement
>  Components: atlas-webui
>Affects Versions: 2.0.0, 3.0.0
>Reporter: Keval Bhatt
>Assignee: Keval Bhatt
>Priority: Major
> Fix For: 2.1.0, 3.0.0
>
> Attachments: ATLAS-3537-1.patch, ATLAS-3537.patch, 
> image-2019-11-26-11-53-45-573.png
>
>
> Add hyperlinks for Deleted & Shell entities count which redirects to the 
> basic search page as per expected results
> !image-2019-11-26-11-53-45-573.png|width=498,height=323!



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (ATLAS-3531) Atlas website: download/release page missing releases

[Sameer Shaikh (Jira)]

 [ 
https://issues.apache.org/jira/browse/ATLAS-3531?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sameer Shaikh updated ATLAS-3531:
-
Attachment: old-version-html-changes.patch

> Atlas website: download/release page missing releases
> -
>
> Key: ATLAS-3531
> URL: https://issues.apache.org/jira/browse/ATLAS-3531
> Project: Atlas
>  Issue Type: Bug
>Reporter: Sameer Shaikh
>Assignee: Sameer Shaikh
>Priority: Major
> Attachments: ATLAS-3531.patch, old-version-html-changes.patch
>
>
> [Atlas website|http://atlas.apache.org/#/Downloads] doesn’t show the 
> following releases.
>  * 0.8.3
>  * 0.8.4
>  * 1.2.0
> Download pages from previous release websites (like 
> [2.0.0|http://atlas.apache.org/2.0.0/index.html],  
> [1.1.0|http://atlas.apache.org/1.1.0/index.html], …)  end up in ‘Not Found’.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (ATLAS-3531) Atlas website: download/release page missing releases

[Sameer Shaikh (Jira)]

 [ 
https://issues.apache.org/jira/browse/ATLAS-3531?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sameer Shaikh updated ATLAS-3531:
-
Attachment: ATLAS-3531.patch

> Atlas website: download/release page missing releases
> -
>
> Key: ATLAS-3531
> URL: https://issues.apache.org/jira/browse/ATLAS-3531
> Project: Atlas
>  Issue Type: Bug
>Reporter: Sameer Shaikh
>Assignee: Sameer Shaikh
>Priority: Major
> Attachments: ATLAS-3531.patch, old-version-html-changes.patch
>
>
> [Atlas website|http://atlas.apache.org/#/Downloads] doesn’t show the 
> following releases.
>  * 0.8.3
>  * 0.8.4
>  * 1.2.0
> Download pages from previous release websites (like 
> [2.0.0|http://atlas.apache.org/2.0.0/index.html],  
> [1.1.0|http://atlas.apache.org/1.1.0/index.html], …)  end up in ‘Not Found’.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (ATLAS-3528) UI: Support basic search to search with System attributes only for entity

[Sameer Shaikh (Jira)]

 [ 
https://issues.apache.org/jira/browse/ATLAS-3528?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sameer Shaikh updated ATLAS-3528:
-
Attachment: (was: ATLAS-3482-v1.patch)

> UI: Support basic search  to search with System attributes only for entity
> --
>
> Key: ATLAS-3528
> URL: https://issues.apache.org/jira/browse/ATLAS-3528
> Project: Atlas
>  Issue Type: Improvement
>Reporter: Sameer Shaikh
>Assignee: Sameer Shaikh
>Priority: Major
> Fix For: 2.1.0, 3.0.0
>
> Attachments: ATLAS-3528.patch
>
>
> Support basic search  to search with System attributes only for entity:
> __guid, __historicalGuids, __labels, __customAttributes, 
> __classificationsText, 
> __classificationNames, __propagatedClassificationNames, __isIncomplete



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (ATLAS-3528) UI: Support basic search to search with System attributes only for entity

[Sameer Shaikh (Jira)]

 [ 
https://issues.apache.org/jira/browse/ATLAS-3528?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sameer Shaikh updated ATLAS-3528:
-
Attachment: (was: ATLAS-3482-v2.patch)

> UI: Support basic search  to search with System attributes only for entity
> --
>
> Key: ATLAS-3528
> URL: https://issues.apache.org/jira/browse/ATLAS-3528
> Project: Atlas
>  Issue Type: Improvement
>Reporter: Sameer Shaikh
>Assignee: Sameer Shaikh
>Priority: Major
> Fix For: 2.1.0, 3.0.0
>
> Attachments: ATLAS-3528.patch
>
>
> Support basic search  to search with System attributes only for entity:
> __guid, __historicalGuids, __labels, __customAttributes, 
> __classificationsText, 
> __classificationNames, __propagatedClassificationNames, __isIncomplete



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (ATLAS-3528) UI: Support basic search to search with System attributes only for entity

[Sameer Shaikh (Jira)]

 [ 
https://issues.apache.org/jira/browse/ATLAS-3528?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sameer Shaikh updated ATLAS-3528:
-
Attachment: (was: ATLAS-3482.patch)

> UI: Support basic search  to search with System attributes only for entity
> --
>
> Key: ATLAS-3528
> URL: https://issues.apache.org/jira/browse/ATLAS-3528
> Project: Atlas
>  Issue Type: Improvement
>Reporter: Sameer Shaikh
>Assignee: Sameer Shaikh
>Priority: Major
> Fix For: 2.1.0, 3.0.0
>
> Attachments: ATLAS-3482-v1.patch, ATLAS-3482-v2.patch, 
> ATLAS-3528.patch
>
>
> Support basic search  to search with System attributes only for entity:
> __guid, __historicalGuids, __labels, __customAttributes, 
> __classificationsText, 
> __classificationNames, __propagatedClassificationNames, __isIncomplete



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Comment Edited] (ATLAS-3528) UI: Support basic search to search with System attributes only for entity

[Sameer Shaikh (Jira)]

[ 
https://issues.apache.org/jira/browse/ATLAS-3528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16986661#comment-16986661
 ] 

Sameer Shaikh edited comment on ATLAS-3528 at 12/3/19 8:31 AM:
---

[~kevalbhatt]  previously I have created the patch with the wrong Jira name 
"ATLAS-3482" The "Jira"name pointed to the backed change. Now I have updated  
ATLAS-3528.patch  


was (Author: sameer.shaikh):
[~kevalbhatt]  previously I have created the patch with the wrong Jira name 
"ATLAS-3482" The "Jira"name pointed to the backed change. Now I have updated to 
the current one  name "ATLAS-3528.patch".  

> UI: Support basic search  to search with System attributes only for entity
> --
>
> Key: ATLAS-3528
> URL: https://issues.apache.org/jira/browse/ATLAS-3528
> Project: Atlas
>  Issue Type: Improvement
>Reporter: Sameer Shaikh
>Assignee: Sameer Shaikh
>Priority: Major
> Fix For: 2.1.0, 3.0.0
>
> Attachments: ATLAS-3482-v1.patch, ATLAS-3482-v2.patch, 
> ATLAS-3528.patch
>
>
> Support basic search  to search with System attributes only for entity:
> __guid, __historicalGuids, __labels, __customAttributes, 
> __classificationsText, 
> __classificationNames, __propagatedClassificationNames, __isIncomplete



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (ATLAS-3305) Unable to scale atlas kafka consumers

[Adam Rempter (Jira)]

[ 
https://issues.apache.org/jira/browse/ATLAS-3305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16986690#comment-16986690
 ] 

Adam Rempter commented on ATLAS-3305:
-

[~madhan], [~sarath], could you guys have a look at patch I attached and 
suggest if we need any more improvements to it?

By default patch sets one consumer per topic - so the same like it is at the 
moment.

Maybe add more clear docs like [~bolke] suggested to ensure that admin is aware 
of how atlas will process hook messages?

 

Now Atlas supports multiple topics, so it could be another addition to improve 
performance of atlas consumer. Btw I think it relaxes consistency also...

And consistency can be guaranteed on producer side, eg:

[https://www.javaworld.com/article/3066873/big-data-messaging-with-kafka-part-2.html]

 

Thanks,
Adam

 

> Unable to scale atlas kafka consumers
> -
>
> Key: ATLAS-3305
> URL: https://issues.apache.org/jira/browse/ATLAS-3305
> Project: Atlas
>  Issue Type: Bug
>  Components:  atlas-core, atlas-intg
>Affects Versions: 1.1.0, 2.0.0
>Reporter: Adam Rempter
>Priority: Major
>  Labels: performance
> Attachments: ATLAS-3305_multiple_kafka_consumers.patch, 
> multiple_consumers_perf.png
>
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> We wanted to scale kafka consumers for atlas, as we are getting many lineage 
> messages and processing them just with one consumer is not enough. 
>  
> There is parameter atlas.notification.hook.numthreads to scale consumers in  
> NotificationHookConsumer.
> But the method:
>  
> notificationInterface.createConsumers(NotificationType.HOOK, numThreads)
>  
> is always returning one element list, which effectively always starts one 
> consumer
> List> consumers = 
> Collections.singletonList(kafkaConsumer);
>  
> Log incorrectly says that nuber of consumers has been created:
> LOG.info("<== KafkaNotification.createConsumers(notificationType={}, 
> numConsumers={}, autoCommitEnabled={})", notificationType, numConsumers, 
> autoCommitEnabled)



--
This message was sent by Atlassian Jira
(v8.3.4#803005)